The Complete Apache Kafka Developer Guide

A deep-dive reference covering core concepts, producer/consumer internals, exactly-once semantics, Kafka Streams, Kafka Connect, schema management, security, performance tuning, common design patterns, and anti-patterns to avoid.

---

Table of Contents

1. Core Architecture
2. Topics, Partitions & Replication
3. Producers
4. Consumers & Consumer Groups
5. Delivery Semantics & Exactly-Once
6. Serialization & Schema Registry
7. Kafka Streams
8. Kafka Connect
9. Topic Design & Partitioning Strategy
10. Performance Tuning
11. Monitoring & Observability
12. Security
13. Common Design Patterns
14. Anti-Patterns & Pitfalls
15. Operational Best Practices

---

1. Core Architecture

Kafka is a distributed, partitioned, replicated commit log service that behaves as a publish-subscribe messaging system at scale.

Key Components

• Broker: A single Kafka server that stores data and serves clients. A cluster is made of multiple brokers.
• Topic: A named, append-only log to which records are published. Topics are split into partitions.
• Partition: The unit of parallelism and ordering. Each partition is an ordered, immutable sequence of records identified by an offset.
• Producer: Publishes records to topics.
• Consumer: Subscribes to topics and processes the record stream.
• Consumer Group: A set of consumers cooperating to consume a topic; each partition is assigned to exactly one consumer within a group.
• ZooKeeper (legacy) / KRaft (modern): Cluster metadata and controller election. Since Kafka 3.x, KRaft mode removes the ZooKeeper dependency entirely — as of Kafka 4.0, ZooKeeper is fully removed. New deployments should always use KRaft.
• Controller: The broker responsible for partition leader election and metadata propagation.

Log Structure

Each partition is stored on disk as a sequence of segment files. Kafka never rewrites records in place — it’s purely append-only, which is what gives it sequential-I/O disk throughput comparable to network throughput.

/var/lib/kafka/data/my-topic-0/
  ├── 00000000000000000000.log
  ├── 00000000000000000000.index
  ├── 00000000000000000000.timeindex
  ├── 00000000000000452312.log
  └── ...

• .log — the actual records
• .index — offset → physical file position mapping (sparse index, binary search)
• .timeindex — timestamp → offset mapping (for time-based seeks)

Broker Roles in KRaft

• Controller nodes: manage metadata (the __cluster_metadata topic).
• Broker nodes: serve produce/fetch requests.
• A node can play both roles (process.roles=broker,controller) in small clusters, or be split for large ones.

---

2. Topics, Partitions & Replication

Partition Count

• Determines maximum parallelism (one consumer per partition per group at most).
• More partitions = more open file handles, more replication traffic, longer leader election, higher end-to-end latency for very small clusters.
• Rule of thumb: partitions = target_throughput / single_partition_throughput. Start conservative (6–12), you can only increase, never decrease, partition count on an existing topic without recreating it (and increasing breaks key-based ordering guarantees for old keys).

Replication Factor

• replication.factor=3 is the standard production default (tolerates 1 broker loss with min.insync.replicas=2).
• Each partition has one leader and N-1 followers. Only the leader serves reads/writes (unless using rack-aware follower fetching / KIP-392 for read replicas).

In-Sync Replicas (ISR)

• The ISR is the set of replicas fully caught up with the leader within replica.lag.time.max.ms.
• min.insync.replicas (topic/broker config) combined with producer acks=all defines your durability guarantee:

min.insync.replicas=2

acks=all

→ A write is acknowledged only when it has been replicated to at least min.insync.replicas replicas, guaranteeing no data loss as long as fewer than min.insync.replicas brokers fail simultaneously.

Key Topic Configs

# Retention
retention.ms=604800000          # 7 days (time-based)
retention.bytes=-1               # unlimited (size-based)

# Compaction
cleanup.policy=compact           # or "delete" or "compact,delete"
min.cleanable.dirty.ratio=0.5
segment.ms=604800000

# Durability
min.insync.replicas=2
unclean.leader.election.enable=false   # NEVER allow out-of-sync replicas to become leader

# Throughput
max.message.bytes=1048588

Log Compaction vs Deletion

• Delete: records are dropped after retention.ms/retention.bytes. Used for event streams (transient facts).
• Compact: Kafka retains at least the last known value for each key, forever. Used for changelog topics, state reconstruction, and as the backing store for Kafka Streams state stores / KTables. A null value (“tombstone”) deletes a key after delete.retention.ms.

---

3. Producers

Core Configuration

bootstrap.servers=broker1:9092,broker2:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

acks=all                       # all | 1 | 0
enable.idempotence=true        # default true since Kafka 3.0
retries=2147483647             # effectively infinite, bounded by delivery.timeout.ms
max.in.flight.requests.per.connection=5   # safe up to 5 with idempotence enabled
delivery.timeout.ms=120000
linger.ms=5                    # batch window, trade latency for throughput
batch.size=32768
compression.type=lz4           # lz4/zstd recommended over gzip/snappy
buffer.memory=33554432

acks Semantics

Value	Meaning	Durability	Latency
0	Fire-and-forget	None	Lowest
1	Leader ack only	Lost if leader fails before replication	Medium
all (-1)	All in-sync replicas ack	Strongest (with min.insync.replicas)	Highest

Idempotent Producer

Setting enable.idempotence=true assigns each producer a Producer ID (PID) and each message a sequence number per partition. The broker deduplicates retries, giving exactly-once per-partition, per-producer-session delivery without application code changes. This is the foundation for transactional producers.

Partitioning Strategy

// Default partitioner (Kafka >= 2.4): sticky partitioning when key is null
// improves batching vs old round-robin.

ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", customerId, payload); // key drives partition via hash

• Keyed records: partition = hash(key) % numPartitions (murmur2 hash) — guarantees ordering per key.
• Null key: sticky partitioner batches records to one partition until batch is full, then switches — maximizes throughput.
• Custom partitioner: implement Partitioner interface for business-specific routing (e.g., VIP customers to dedicated partitions).

Handling Backpressure & Errors

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        if (exception instanceof RetriableException) {
            // let the producer's internal retry handle it; log for visibility
        } else {
            // non-retriable: DLQ, alert, or fail fast
        }
    }
});

• Always use the async callback, never block on .get() in hot paths unless you truly need synchronous confirmation.
• Watch for RecordTooLargeException, TimeoutException, NotEnoughReplicasException.

Transactional Producer (Exactly-Once)

transactional.id=order-service-1
enable.idempotence=true

producer.initTransactions();
try {
    producer.beginTransaction();
    producer.send(record1);
    producer.send(record2);
    producer.sendOffsetsToTransaction(offsets, groupMetadata); // for consume-transform-produce
    producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
    producer.close(); // fatal, must restart
} catch (KafkaException e) {
    producer.abortTransaction();
}

• transactional.id must be stable and unique per logical producer instance (e.g., per partition/shard) so fencing works across restarts.
• Downstream consumers must set isolation.level=read_committed to skip uncommitted/aborted messages.

---

4. Consumers & Consumer Groups

Core Configuration

bootstrap.servers=broker1:9092,broker2:9092
group.id=order-processing-service
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

enable.auto.commit=false        # prefer manual commits for control over "at-least-once"
auto.offset.reset=earliest      # earliest | latest | none
max.poll.records=500
max.poll.interval.ms=300000     # time allowed between polls before considered dead
session.timeout.ms=45000
heartbeat.interval.ms=15000
fetch.min.bytes=1
fetch.max.wait.ms=500
isolation.level=read_committed  # if reading transactional topics
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Consumer Group Rebalancing

• Eager rebalancing (old default, RangeAssignor/RoundRobinAssignor): all consumers stop processing, revoke all partitions, then reassign (“stop-the-world”).
• Cooperative Sticky rebalancing (CooperativeStickyAssignor, recommended): only reassigns partitions that actually need to move, minimizing pause time — use this in modern deployments.
• Static membership (group.instance.id): avoids triggering a rebalance on transient restarts (e.g., rolling deploys, brief network blips) — critical for stateful consumers (Kafka Streams) with large local state.

group.instance.id=consumer-instance-7
session.timeout.ms=45000

Manual Offset Commit Patterns

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> record : records) {
        process(record);
    }
    consumer.commitSync(); // or commitAsync() with a callback for higher throughput
}

Commit-after-process gives at-least-once delivery (default and recommended for most systems). To get exactly-once processing effects, either:

1. Make processing idempotent (e.g., upsert by key), or
2. Use Kafka transactions in a consume-transform-produce pipeline (see §3), or
3. Store the offset and the processing result atomically in an external system (e.g., a DB transaction that writes both the business row and the offset).

Handling Poison Pills

try {
    process(record);
} catch (DeserializationException | UnrecoverableBusinessException e) {
    deadLetterProducer.send(new ProducerRecord<>("orders-dlq", record.key(), record.value()));
} 

Never let a single malformed record block the entire partition forever — route to a Dead Letter Queue (DLQ) with the original headers/metadata preserved for later replay.

Consumer Lag

Lag = (latest offset in partition) − (last committed offset by group). Monitor with kafka-consumer-groups.sh --describe --group X or via JMX/Burrow/Prometheus exporters. Sustained growing lag indicates the consumer can’t keep up — scale out (add consumers up to partition count), optimize processing, or increase partitions.

---

5. Delivery Semantics & Exactly-Once

Semantic	How to achieve	Notes
At-most-once	enable.auto.commit=true with commit before processing, or acks=0	Data loss possible
At-least-once	Commit after successful processing; acks=all	Default recommendation; duplicates possible on failure
Exactly-once	Idempotent producer + transactions + read_committed consumers, OR idempotent downstream sink	Highest complexity, use only when duplicates are truly unacceptable (e.g., financial transactions)

Exactly-Once Semantics (EOS) in Kafka is guaranteed within the Kafka ecosystem (producer → topic → consumer, including Kafka Streams). If your pipeline writes to an external system (DB, S3, REST API), you need either:

• An idempotent write (upsert on a natural/business key), or
• The transactional outbox pattern (§13), or
• Kafka Connect sink connectors with exactly-once support (e.g., JDBC sink using upserts, or connectors implementing the exactly-once protocol via EXACTLY_ONCE delivery guarantee, KIP-618).

---

6. Serialization & Schema Registry

Why Schema Registry

Avro/Protobuf/JSON Schema + Confluent (or Apicurio/Karapace) Schema Registry gives you:

• Centralized schema versioning
• Compatibility enforcement (backward/forward/full) before bad data reaches the cluster
• Compact binary encoding (Avro/Protobuf) — smaller messages, faster (de)serialization than JSON

Compatibility Modes

Mode	Consumers can read	Producers can write
BACKWARD	New schema reads old data	Old schema readers unaffected by new writes with new schema… consumer must upgrade first
FORWARD	Old schema reads new data	Producers can upgrade first
FULL	Both directions	Safest, most restrictive
NONE	No checks	Dangerous, avoid in production

key.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=http://schema-registry:8081

// Example Avro schema — always provide defaults for new fields (backward compatible)
{
  "type": "record",
  "name": "OrderCreated",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "amount", "type": "double"},
    {"name": "currency", "type": "string", "default": "USD"}
  ]
}

Rules of Thumb

• Never remove a required field without a compatible migration plan.
• Always add new fields with defaults.
• Use a naming strategy (TopicNameStrategy, RecordNameStrategy) that matches your subject organization — RecordNameStrategy allows multiple event types on one topic.

---

7. Kafka Streams

Kafka Streams is a client library for building stream-processing applications directly on top of Kafka — no separate cluster needed.

Core Abstractions

• KStream: unbounded record stream, each record is an independent event.
• KTable: changelog stream representing the latest value per key (a compacted, materialized view).
• GlobalKTable: fully replicated table on every instance — good for small reference/lookup data.

Example Topology

StreamsBuilder builder = new StreamsBuilder();

KStream<String, Order> orders = builder.stream("orders",
        Consumed.with(Serdes.String(), orderSerde));

KTable<String, Long> orderCountsByCustomer = orders
        .groupBy((key, order) -> order.getCustomerId(), Grouped.with(Serdes.String(), orderSerde))
        .count(Materialized.as("order-counts-store"));

orderCountsByCustomer.toStream().to("order-counts", Produced.with(Serdes.String(), Serdes.Long()));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Stream-Table Duality & Joins

// Stream-Stream join (windowed, both sides bounded in time)
orders.join(payments,
    (order, payment) -> enrich(order, payment),
    JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)));

// Stream-Table join (no window; table represents current state)
orders.join(customersTable, (order, customer) -> enrich(order, customer));

State Stores & Fault Tolerance

• State stores are backed by RocksDB locally and changelog topics (compacted) remotely — so state can always be rebuilt on failover.
• Use standby.replicas > 0 to keep hot copies on other instances for fast failover.

num.standby.replicas=1
state.dir=/data/kafka-streams
cache.max.bytes.buffering=10485760

Exactly-Once in Streams

processing.guarantee=exactly_once_v2

This wraps the read-process-write cycle in a Kafka transaction automatically — the recommended way to get EOS without hand-rolling transactional code.

Windowing

• Tumbling: fixed, non-overlapping windows.
• Hopping: fixed size, overlapping (advance < size).
• Sliding: window per record pair within a time difference.
• Session: dynamic, gap-based windows per key.

TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)); // always set grace period

---

8. Kafka Connect

A framework for scalable, fault-tolerant integration between Kafka and external systems — no custom producer/consumer code needed for common integrations.

Modes

• Standalone: single process, config in a file — good for dev/small use cases.
• Distributed: multiple workers form a cluster, configs stored in internal Kafka topics (connect-configs, connect-offsets, connect-status) — production standard, supports scaling and fault tolerance.

Source vs Sink Connectors

// Debezium (CDC) source connector example
{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "orders_db",
    "table.include.list": "public.orders",
    "topic.prefix": "cdc",
    "plugin.name": "pgoutput"
  }
}

// JDBC sink connector example
{
  "name": "orders-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "connection.url": "jdbc:postgresql://warehouse/db",
    "topics": "orders",
    "insert.mode": "upsert",
    "pk.mode": "record_key",
    "auto.create": "true"
  }
}

Single Message Transforms (SMTs)

Lightweight inline transformations without a separate stream processor:

"transforms": "maskPII,route",
"transforms.maskPII.type": "org.apache.kafka.connect.transforms.MaskField$Value",
"transforms.maskPII.fields": "ssn",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "(.*)",
"transforms.route.replacement": "prefixed-$1"

Error Handling (Dead Letter Queue for Connect)

errors.tolerance=all
errors.deadletterqueue.topic.name=connect-dlq
errors.deadletterqueue.context.headers.enable=true
errors.log.enable=true

---

9. Topic Design & Partitioning Strategy

Naming Conventions

<domain>.<entity>.<event-type>.v<version>
e.g. ecommerce.orders.created.v1
     billing.invoices.updated.v2

• Include a version suffix for schema evolution that breaks compatibility.
• Separate concerns: don’t mix unrelated event types on one topic unless intentional (e.g., CDC tables).

Choosing Keys

• Key by the entity you need ordering for (e.g., orderId, customerId).
• Avoid overly hot keys (a single huge customer skews one partition) — consider salting extremely hot keys (customerId#shard) if ordering across the whole entity isn’t required.

Multi-Tenancy

• Shared topics with tenant ID in the key/header + ACLs for read isolation, OR
• Per-tenant topics for hard isolation (higher operational overhead, but strong quotas and security boundaries).

---

10. Performance Tuning

Producer Throughput

• Increase linger.ms and batch.size to batch more per request.
• Use compression.type=lz4 or zstd (best compression ratio, low CPU cost).
• Increase buffer.memory if you see BufferExhaustedException.

Consumer Throughput

• Increase fetch.min.bytes and fetch.max.wait.ms to reduce request overhead (batch fetches).
• Increase max.poll.records cautiously — balance against max.poll.interval.ms.
• Scale consumers up to (but not beyond) the partition count — extra consumers beyond partition count sit idle.

Broker Tuning

num.network.threads=8
num.io.threads=16
num.replica.fetchers=4
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
log.flush.interval.messages=Long.MAX_VALUE   # rely on replication, not fsync, for durability

• Use page cache effectively: don’t set heap too large (4–6GB is typically enough); let the OS cache log segments.
• Use dedicated disks (avoid RAID 5/6 for write-heavy logs — RAID 10 or JBOD with replication is preferred).
• Rack awareness (broker.rack) spreads replicas across failure domains (AZs).

Sizing Checklist

• Disk throughput: sequential writes, so plain SSD/NVMe is usually not even necessary — but helps with random reads during catch-up/reprocessing.
• Network: replication traffic = replication.factor × produce throughput, budget accordingly.

---

11. Monitoring & Observability

Critical Broker Metrics (JMX)

Metric	Why it matters
UnderReplicatedPartitions	> 0 means data-loss risk; investigate immediately
OfflinePartitionsCount	> 0 means unavailability
ActiveControllerCount	Should be exactly 1 cluster-wide
RequestHandlerAvgIdlePercent	Low = broker CPU-bound, thread pool saturated
BytesInPerSec / BytesOutPerSec	Throughput trend
ISR shrink/expand rate	Frequent shrink = flaky replicas / network issues

Consumer Metrics

• records-lag-max / records-lag per partition
• consumer group state (Stable, Rebalancing, Dead)
• commit-latency-avg

Tooling

• Prometheus + JMX Exporter + Grafana — de facto open-source standard.
• Burrow — dedicated consumer-lag monitoring with SLA-style evaluation.
• Cruise Control — automated partition rebalancing and broker decommissioning.
• kcat (kafkacat) — CLI Swiss-army knife for quick produce/consume/inspect.

---

12. Security

Encryption

listeners=SSL://broker1:9093
ssl.keystore.location=/certs/kafka.server.keystore.jks
ssl.keystore.password=changeit
ssl.truststore.location=/certs/kafka.server.truststore.jks

Use TLS for all inter-broker and client-broker traffic in production (security.inter.broker.protocol=SSL).

Authentication

• SASL/SCRAM — username/password based, simple to operate.
• SASL/GSSAPI (Kerberos) — enterprise AD/LDAP integration.
• mTLS — mutual TLS certs as identity.
• OAUTHBEARER — modern token-based auth (OIDC integration).

Authorization (ACLs)

kafka-acls.sh --bootstrap-server broker:9092 \
  --add --allow-principal User:order-service \
  --operation Write --operation Read \
  --topic orders

• Apply least privilege: producers get Write only on their topics, consumers get Read + DescribeGroup scoped to their consumer group.
• Use --resource-pattern-type prefixed for team/domain-based topic prefixes to avoid per-topic ACL sprawl.

Data-Level

• Field-level encryption/tokenization for PII before it reaches Kafka if compliance requires it (Kafka-level encryption is transport/at-rest, not field-level).

---

13. Common Design Patterns

Transactional Outbox Pattern

Solves the “dual write” problem (DB write + Kafka publish must be atomic) without distributed transactions:

1. Application writes the business row and an outbox row in the same local DB transaction.
2. A CDC connector (e.g., Debezium) tails the outbox table and publishes to Kafka.
3. Outbox row is later purged.

This guarantees the event is published if and only if the DB transaction committed.

Event Sourcing

Store state changes as an ordered sequence of immutable events (Kafka topic = the source of truth), rebuild current state by replaying events into a materialized view (KTable / external DB).

CQRS (Command Query Responsibility Segregation)

Separate write model (command handlers publishing events) from read model (materialized views built via Kafka Streams/Connect consuming those events) — enables independently scaling reads/writes and multiple read-optimized projections from one event stream.

Saga Pattern (Choreography via Events)

Long-running business transactions across services coordinated purely through published/consumed events, each service reacting to the previous step’s event and publishing a compensating event on failure — avoids distributed 2PC transactions.

Change Data Capture (CDC)

Use Debezium/Connect to stream database row-level changes into Kafka topics in near real-time, decoupling downstream systems from direct DB access and enabling event-driven architectures on top of legacy databases.

Compacted Topic as a KV Store

Use cleanup.policy=compact topics as a durable, replicated, replayable key-value store for reference/config data — consumers rebuild an in-memory or RocksDB map by replaying from the beginning.

Dead Letter Queue (DLQ)

Route unprocessable messages to a separate topic with error metadata in headers, allowing the main pipeline to proceed and enabling manual/automated replay after investigation.

Fan-Out / Multiple Consumer Groups

A single topic can be consumed independently by many services, each with its own consumer group — Kafka retains messages for all groups (unlike traditional queues), enabling true pub-sub fan-out without republishing.

---

14. Anti-Patterns & Pitfalls

Anti-Pattern	Why it’s a problem	Fix
Using Kafka as a traditional task queue with per-message ack/nack	Kafka commits are offset-based, not per-message; doesn’t fit priority queues or ack-per-item semantics	Use RabbitMQ/SQS for classic queueing, or model carefully with DLQs
Too many tiny topics (per-customer topics at scale)	Broker metadata overhead, partition leader election overhead, ZK/KRaft metadata bloat	Use keyed multi-tenant topics with ACL isolation instead
Giant messages (> few MB)	Hurts throughput, memory, GC pressure	Store payload in blob storage (S3), publish a reference/pointer in Kafka
unclean.leader.election.enable=true in production	Silent data loss on leader failover	Keep false, invest in proper replication instead
Relying on message ordering across partitions	Kafka only guarantees order within a partition	Key records so related events land on the same partition
Auto-commit with slow/failing processing	Can commit offsets for records that failed to process, causing silent data loss	Manual commit after successful processing
Not setting max.poll.interval.ms correctly for slow consumers	Consumer gets kicked from group mid-processing, causing rebalance storms	Tune interval to realistic worst-case processing time, or offload heavy work asynchronously
Schema changes without a registry/compatibility checks	Breaks consumers silently in production	Enforce Schema Registry compatibility checks in CI/CD
Ignoring consumer lag alerts	Silent backlog growth until outage	Alert on lag trend, not just absolute value
Using Kafka for request-response RPC	Adds unnecessary latency/complexity vs. gRPC/REST	Use Kafka for async/event-driven flows, not synchronous calls
Not planning partition count upfront	Can’t decrease partitions later without recreating the topic	Model expected throughput/parallelism before topic creation

---

15. Operational Best Practices

• Capacity plan replication traffic, not just produce traffic — replication.factor=3 triples your write network/disk load.
• Automate rolling restarts with health checks per broker (wait for UnderReplicatedPartitions=0 before moving to next broker).
• Use quotas (producer_byte_rate, consumer_byte_rate, request_percentage) to prevent noisy-neighbor tenants from starving the cluster.
• Version-pin client libraries and test broker upgrades in staging first — Kafka maintains strong backward compatibility but subtle behavior changes do occur.
• Back up topic configs and ACLs as code (Terraform providers exist for Kafka topics/ACLs) — never manage production topics purely via ad-hoc CLI.
• Test disaster recovery: simulate broker loss, AZ loss, and verify min.insync.replicas + acks=all actually protect you.
• Document your event schemas and topic ownership — event-driven systems fail silently when ownership is unclear; treat topics as public APIs with contracts.
• Consider MirrorMaker 2 / Cluster Linking for cross-region replication, disaster recovery, or multi-cluster architectures.

---

Quick Reference: Recommended Production Defaults

# Topic
replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

# Producer
acks=all
enable.idempotence=true
compression.type=lz4
linger.ms=5

# Consumer
enable.auto.commit=false
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
isolation.level=read_committed

---

This guide reflects modern Kafka (3.x/4.x, KRaft-based) best practices as of early 2026. Always cross-check against the official Apache Kafka documentation for the exact version you’re running, since defaults and available configs evolve between releases.