CCDAK Confluent Certified Developer for Apache Kafka Certification Examination Questions and Answers

Unit Testing → MockProducer

Integration Testing → Testcontainers

Performance Testing → Trogdor

Mock Data Generation → Connect Datagen

MockProducer: Simulates a Kafka producer in unit tests (no real broker interaction).

Testcontainers: Spawns Kafka in Docker for real environment testing.

Trogdor: Kafka’s built-in performance load testing framework.

Connect Datagen: Creates sample source records for test and demo purposes.

From Kafka Developer Tools Guide:

“Kafka developers commonly use MockProducer for unit tests, Testcontainers for integration, and Trogdor for performance tests.”

To send failed records to a dead letter queue (DLQ), you must configure:

errors.tolerance=all: Tells the connector to not fail on errors but handle them (e.g., send to DLQ).

errors.deadletterqueue.topic.name=DLQ-Topic: Specifies the DLQ topic.

errors.deadletterqueue.context.headers.enable=true: Includes error context in message headers.

From Kafka Connect Error Handling Docs:

“Kafka Connect supports directing problematic records to a separate topic (DLQ) using errors.* configs. Headers can include failure metadata.”

Options D, E, F are related to logging, not DLQ behavior.

The batch.size config sets the maximum number of bytes to batch per partition before sending. This allows Kafka producers to amortize I/O and improve throughput.

From Kafka Producer Configuration Docs:

“batch.size is the maximum amount of data per partition the producer will batch before sending.”

buffer.memory sets total memory for the producer, not per-batch.

send.buffer.bytes is a TCP socket buffer, not a Kafka config.

max.block.ms controls blocking time, not size.

According to the official Apache Kafka Streams documentation, Kafka Streams relies on internal topics (such as changelog topics and repartition topics) to store state, task assignments, and application metadata. These topics are essential for fault tolerance, rebalancing, and state recovery.

When auto-topic creation is disabled, Kafka Streams cannot automatically create the required internal topics unless they already exist. In this situation, the Streams application will fail during startup with a fatal, non-retriable exception, indicating that the necessary internal topics could not be created.

Kafka Streams does not pause or wait for manual topic creation, nor does it operate without storing metadata. Instead, it explicitly requires these topics to exist or be creatable. While administrators can manually pre-create the required internal topics with the correct configuration, failure to do so causes the application to terminate.

Therefore, the correct and documented behavior is that the Kafka Streams application terminates with a non-retriable exception when auto-topic creation is disabled and required internal topics are missing.

According to the official Apache Kafka Connect REST API documentation, restarting a connector can be done at different granularities: connector-level, task-level, or both. By default, calling POST /connectors/{name}/restart only restarts the connector instance, not its tasks.

To restart the connector and all of its tasks, the request must explicitly include the query parameter includeTasks=true. This instructs Kafka Connect to restart the connector and all associated tasks, regardless of whether they are currently running or failed.

Option B (onlyFailed=true) limits the restart to only failed tasks, which does not meet the requirement to restart all tasks. Option C restarts only the connector instance, leaving tasks untouched. Option D restarts a single specific task and does not restart the connector itself.

Therefore, the correct and officially documented REST request to restart the connector instance and all tasks is:

POST /connectors/s3-connector/restart?includeTasks=true

This ensures a full restart of the connector and its execution context across the Kafka Connect cluster.

If a message key is not provided, Kafka’s default partitioner uses round-robin distribution across available partitions.

From Kafka Producer Design:

“If no key is provided, the default partitioner distributes messages round-robin to available partitions.”

A is incorrect — not all go to partition 0.

C is invalid — Kafka doesn’t group messages into one segment without a key.

D is false — ordering is not preserved across partitions without a key.

The Apache Kafka Streams documentation clearly distinguishes between the Kafka Streams DSL and the Processor API. While the DSL is designed for common stream processing operations such as filtering, mapping, joining, and aggregations, it does not support fully custom processing logic.

For use cases that require custom transformations, fine-grained control over record processing, access to headers, timestamps, state stores, and punctuation, Kafka Streams provides the Processor API. This API allows developers to implement custom Processor, Transformer, or ValueTransformer classes and explicitly define the processing topology.

Option A (TopologyTestDriver) is a testing utility, not a development API. Option C (DSL) is higher-level and not suitable for advanced custom logic. Option D does not exist in Kafka.

Therefore, the correct and officially supported approach for defining custom processors in Kafka Streams is to use the Processor API.

=========================

Low I/O thread activity with blocked throughput often indicates that producer callbacks are consuming too much time, causing the sender thread to block while waiting for onCompletion() to finish.

From Kafka Producer Performance Guide:

“Expensive logic in callbacks (e.g., I/O or complex computation) can block the sender thread, reducing throughput.”

Compression (A) may slightly impact CPU but not I/O thread usage.

Large batches (B) improve throughput if managed correctly.

A Layer 2 network issue (C) would lead to packet loss, not specifically low callback metrics.

The offset in the RecordMetadata object returned by the producer represents the position of the record in the partition — i.e., the sequential ID assigned by Kafka once the message is committed.

From Kafka Producer API Documentation:

“The offset is the position of the record in the partition. This is a unique, sequential number assigned by the broker.”

D refers to metadata.partition(), not offset().

B and C are unrelated to how Kafka handles committed offsets.

According to the Apache Kafka consumer and scalability documentation, parallelism in message processing is primarily determined by the number of partitions and the number of consumers within a consumer group. Each partition can be assigned to only one consumer at a time. In this scenario, the topic already has four partitions, but only two consumers are processing them, meaning only two partitions are actively consumed in parallel.

Since monitoring indicates that message processing is CPU-bound, the bottleneck is not Kafka I/O but application-side processing. Adding more consumers (up to the number of partitions) allows Kafka to rebalance the partitions and distribute them across more consumer instances, increasing CPU parallelism and reducing lag.

Option A (adding more partitions) can increase parallelism but is more intrusive and may require repartitioning considerations for keys and downstream systems. Option B increases batch size, which can worsen CPU pressure. Option D may reduce per-poll workload but does not increase parallel processing capacity.

Therefore, the correct and recommended action per Kafka documentation is to add more consumers to the consumer group.

=========================

The Apache Kafka Streams documentation defines a co-partitioning requirement for KStream–KTable and KStream–KStream joins. Both input topics must have the same number of partitions and the same key partitioning strategy.

One valid solution is to use a GlobalKTable (Option A). A GlobalKTable is fully replicated to every Kafka Streams instance, removing the co-partitioning requirement. This approach is recommended when the reference data is relatively small and changes infrequently.

Another valid solution is to repartition one topic so that both topics have the same number of partitions (Option B). Kafka Streams provides repartition topics specifically for this purpose, allowing proper KStream–KTable joins.

Option C does not resolve the partition mismatch, as increasing instances does not change partitioning. Option D is incorrect because Kafka Streams does not automatically repartition both topics for joins; repartitioning must be explicitly configured.

Therefore, the correct and officially supported solutions are using a GlobalKTable and explicitly repartitioning one topic.

=========================

With acks=all and min.insync.replicas=2, Kafka requires at least two in-sync replicas to acknowledge a write. If only one broker is alive, the condition fails, and NotEnoughReplicasException is thrown by the producer.

From Kafka Producer Exception Docs:

“NotEnoughReplicasException is thrown when the number of in-sync replicas is insufficient to satisfy acks=all with min.insync.replicas.”

NetworkException is generic and not raised here.

NotCoordinatorException is related to consumer group coordination.

NotLeaderForPartitionException is unrelated unless accessing an unassigned leader.

SMTs (Single Message Transforms) are lightweight transformations applied to individual messages as they pass through Kafka Connect.

Chaining SMTs: You can apply multiple SMTs in sequence by defining them in order in the connector config.

Masking or modifying fields is a common use case (e.g., redacting sensitive data).

From Kafka Connect Documentation:

“Single Message Transforms (SMTs) are applied to individual messages. You can chain multiple SMTs together.”

SMTs do not perform joins (B is incorrect).

Converters and SMTs are separate concerns (D is incorrect).

When using Kafka Connect with sink connectors (like S3 Sink), each task is assigned one or more topic partitions. The total number of partitions (6) is evenly distributed across the available tasks (2). Thus, each task will handle 3 partitions.

From Kafka Connect Documentation:

“Kafka Connect divides the topic partitions among available tasks. For example, a topic with six partitions and two tasks results in each task handling three partitions.”

To ensure exactly-once delivery and avoid duplicates even during retries:

enable.idempotence=true ensures deduplication on the broker

retries=2147483647 allows unlimited retries on retriable errors

max.in.flight.requests.per.connection=5 is the maximum value that preserves message order with idempotence

From Kafka Producer Config Docs:

“To achieve exactly-once semantics, set enable.idempotence=true, and max.in.flight.requests.per.connection ≤ 5.”

A is unrelated (consumer-side)

C disables retries

D disables idempotence, leading to duplicates

Log compaction ensures that Kafka retains at least the latest value per key in a topic. Compaction happens in the background and removes older records with the same key in inactive log segments, not immediately.

From the Kafka Documentation > Log Compaction:

“Kafka guarantees that the last message for each key will be retained in the log after compaction, even if earlier messages with the same key are deleted.”

So, D is correct. A is incorrect because compaction does not overwrite; it's a background process. B is incorrect—deletion is not immediate. C incorrectly suggests client-side hashmap behavior.

According to the official Apache Kafka Connect and Confluent JDBC Source Connector documentation, the correct connector class for a JDBC source connector is io.confluent.connect.jdbc.JdbcSourceConnector. This connector is used to read data from relational databases and publish each table as a Kafka topic.

To read all rows from a specific table, the configuration must include table.whitelist (or table.include.list in newer versions) with the table name, and a topic.prefix to determine the Kafka topic name. In this case, using topic.prefix=dbl- with table.whitelist=orders results in records being written to the dbl-orders topic, which matches the requirement.

Options A, B, and C are invalid because they reference a non-existent connector class (DdbcSourceConnector), contain unsupported properties such as topic.whitelist for a source connector, or include malformed/incorrect JDBC parameters. Additionally, blacklisting tables does not ensure that only the orders table is read.

Therefore, option D is the only configuration that is syntactically correct, uses the proper connector class, and aligns with the official Kafka Connect JDBC Source Connector documentation.

For SASL/PLAIN authentication over SSL, the correct combination is:

security.protocol=SASL_SSL

sasl.jaas.config=...PlainLoginModule required username=... password=...

From Kafka Security Documentation:

“For SASL/PLAIN over SSL, configure security.protocol=SASL_SSL and provide login module configuration using sasl.jaas.config.”

Options B and D are incorrect because they do not pair the correct protocol with the authentication method.

Correct Matches

path.format→Only valid for some Source Connectors

value.converter→Valid for all Source Connectors

input.path→Only valid for some Source Connectors

errors.retry.timeout→Valid for all Source Connectors

tasks.max→Valid for all Source Connectors

path.formatThis is a connector-specific configuration, commonly used by file-based or storage-based source connectors (for example, HDFS, S3, or FilePulse connectors). It isnot part of the Kafka Connect framework itself, so it is only valid for some source connectors.

value.converterThis is aworker-level configurationthat defines how record values are converted (JSON, Avro, Protobuf, etc.). It applies toall connectorsrunning on the worker unless overridden at the connector level.

input.pathThis configuration is specific to file-based source connectors (e.g., FileStreamSourceConnector). It isnot universally supported, so it is only valid for some source connectors.

errors.retry.timeoutThis is part of Kafka Connect’serror handling framework, introduced to support retries and tolerance of transient errors. It is ageneric Connect configurationand valid for all connectors.

tasks.maxThis is amandatory, generic connector configurationthat controls the maximum number of tasks a connector may create. It is valid forall source (and sink) connectors.

If the application is CPU-bound and lagging, adding more consumers to the group will allow better parallel processing, especially since the topic has 4 partitions, allowing up to 4 active consumers.

From Kafka Consumer Group Docs:

“Kafka achieves parallelism by distributing partitions across consumers in a group. Adding consumers helps reduce lag if partitions are underutilized.”

B may help but requires repartitioning and coordination.

C or D affects how much data is polled, not how fast it’s processed.

The ConsumerRebalanceListener lets you handle partition assignments and revocations during rebalances. This is critical for managing offsets, stateful processing, or external transactions.

From Kafka Consumer Rebalance Docs:

“Implementing ConsumerRebalanceListener allows your application to take action before and after partitions are reassigned.”

A is true: It lets your app react when partitions assigned to the consumer change.

B, C, and D are unrelated to consumer rebalancing directly.

To change the replication factor of an existing topic, you must perform a partition reassignment, because the replication factor is not directly modifiable via kafka-topics.sh. The kafka-reassign-partitions.sh tool is the official command-line utility to handle partition reassignments, including changing the replication factor.

This tool allows you to generate a reassignment JSON specifying new replica assignments per partition and then apply it.

From official documentation:

“To change the replication factor, use the kafka-reassign-partitions tool to assign the partitions to more or fewer brokers.”

Neither kafka-topics.sh nor kafka-reassign-topics.sh exists for this purpose, and kafka-recreate-topic.sh is not an official Kafka tool.

According to the official Apache Kafka consumer semantics documentation, at-most-once delivery means that records may be lost but must never be processed more than once. To achieve this guarantee, a consumer must commit offsets before processing records. If the application crashes after committing but before processing, messages are skipped rather than reprocessed, which satisfies at-most-once semantics.

Option B correctly implements this pattern. It disables automatic offset commits (enable.auto.commit=false) and explicitly calls commitAsync() immediately after polling and before processing records. This ensures that offsets are advanced regardless of whether processing succeeds, eliminating the possibility of duplicate processing.

Option C represents at-least-once semantics, because offsets are committed after processing, which can lead to duplicates on failure. Options A and D rely on automatic commits or incorrect ordering and do not provide strict at-most-once guarantees.

The Kafka documentation explicitly states that manual offset management with commits before processing is required for at-most-once delivery.

Therefore, Option B is the best and officially documented configuration and code skeleton to avoid duplicate message processing.

To isolate logs for a specific connector, you can configure a separate logger and appender in the Connect worker’s log4j.properties file, using the connector’s name as the logging context.

From Kafka Connect Logging Docs:

“Kafka Connect loggers use hierarchical logger names. You can configure per-connector log levels and output files by extending log4j.properties.”

A and C change verbosity but don’t separate logs.

D is false; targeted logging is possible.