Introduction

A while back we were sizing a Kafka cluster for a customer who wanted thirty days of retention on their event topics. The throughput was modest, maybe 40MB/s across the cluster, but thirty days of that with replication factor 3 works out at roughly 300TB of provisioned block storage. The brokers themselves needed a fraction of that CPU and memory. We were, in effect, buying a very expensive and very slow object store, and paying EBS prices for it.

Request charges are left out of the table since they are noise at this volume, a few dollars a month for the PUTs, though a full replay pulling tens of terabytes back out of S3 will show up on the bill.

That is a ninety per cent cut on the storage line, and it understates the saving because you also stop sizing instances around disk attachment limits and IOPS, so the brokers themselves get smaller.

That is the problem tiered storage solves, and as of Kafka 3.9 it is production ready rather than the early access curiosity it was in 3.6.

What tiered storage actually changes

Kafka's own documentation makes the observation that the design hangs on: Kafka data is mostly consumed in a streaming fashion using tail reads, which are served out of the page cache, while older data is read infrequently for backfill or recovery. Keeping all of it on local disk means every byte of cold data is paying for hot storage.

With tiered storage enabled, a broker still writes to its local log directory exactly as before. Once a segment is closed, the RemoteLogManager copies it, along with its offset and time indexes, to the remote tier. The local copy remains on disk to handle hot reads until local retention policies (local.retention.*) trigger its deletion. Even after the local copy is deleted, the segment remains part of the log and fully readable. A consumer requesting an offset beyond local retention is served transparently from the remote tier, with no client-side changes and no separate archive topic to manage.

The part we find most interesting operationally is not the storage bill. It is what happens to replication. When a broker fails or you add one, only the local tier has to be re-replicated. Instead of streaming 100TB across the network to rebuild a replica, you stream whatever your local retention window holds, which might be a few hours. Cluster expansion goes from a weekend job to a coffee break.

The two plugins you have to supply

This is where most people trip up on the first attempt. Apache Kafka defines two interfaces and ships a usable default for only one of them.

RemoteStorageManager handles the lifecycle of remote log segments and indexes. Kafka provides no default implementation. There is a LocalTieredStorage class in the test sources which is fine for a laptop demo and absolutely not for production. For real clusters you take the Aiven tiered-storage-for-apache-kafka plugin, which is Apache 2.0 licensed and supports S3, GCS and Azure Blob, or you use whatever your vendor bundles.

RemoteLogMetadataManager tracks which segments live remotely and at which offsets. Here Kafka does give you a default, TopicBasedRemoteLogMetadataManager, which keeps the metadata in an internal __remote_log_metadata topic.

Turning it on

The cluster level switch is a static broker property, so it needs a rolling restart. Something like this in server.properties. You will need to also update the KRaft controller.

# Enable the feature cluster wide
remote.log.storage.system.enable=true

# The plugin that talks to object storage
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/kafka/plugins/tiered-storage/*

# Metadata, using the built in topic based implementation
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.listener.name=PLAINTEXT

# Plugin specific settings, prefixed and passed straight through
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=acme-kafka-tiered
rsm.config.storage.s3.region=eu-west-2
rsm.config.chunk.size=4194304

# Throughput guards so archiving cannot starve the hot path
remote.log.manager.copy.max.bytes.per.second=104857600
remote.log.manager.fetch.max.bytes.per.second=104857600

The remote.log.metadata.manager.listener.name property is mandatory with the default metadata manager and is a common cause of brokers refusing to start. It has to name a listener that the brokers can use to talk to each other.

Those last two quota settings are worth setting from day one. Without them, the first topic you enable will happily saturate your network uplink backfilling months of segments into S3 while your producers wonder what happened.

Enabling it on a topic

Nothing is tiered until you say so per topic. For a new topic:

kafka-topics.sh --bootstrap-server broker1:9092 --create \
  --topic payments.events \
  --partitions 12 --replication-factor 3 \
  --config remote.storage.enable=true \
  --config local.retention.ms=21600000 \
  --config retention.ms=2592000000

That says: keep six hours of log history on local broker disks and maintain thirty days in total log retention. As soon as a segment closes (governed by segment.bytes or segment.ms), it is immediately copied to remote storage. The local copy stays on disk to handle hot reads until it hits six hours (local.retention.ms), at which point it is purged locally while remaining readable from remote storage. Finally, the segment is purged entirely at thirty days (retention.ms). The local disk footprint reduction is roughly 120 to 1 compared to keeping all thirty days on local disk.

For an existing topic the same properties go through kafka-configs.sh:

kafka-configs.sh --bootstrap-server broker1:9092 --alter \
  --entity-type topics --entity-name payments.events \
  --add-config 'remote.storage.enable=true,local.retention.bytes=53687091200'

If you leave local.retention.* unset, it defaults to -2, inheriting the value of retention.*. In this configuration, segments are copied to remote storage and remain on local disk for their entire lifetime. Data is duplicated across both tiers, defeating the primary goal of offloading data to save local disk space. To reclaim local storage, explicitly set local.retention.* to a smaller value than retention.*.

Turning it off is more awkward than turning it on, so plan it. You can set remote.log.copy.disable=true to freeze the remote log as read only, or remote.log.delete.on.disable=true to purge it. Disabling the feature cluster wide requires deleting every topic that uses it first.

What it costs you

Be honest about the trade. A consumer replaying from three weeks ago is now doing ranged GETs against object storage rather than reading sequential blocks out of page cache. Expect first byte latency in the tens or hundreds of milliseconds instead of microseconds, and expect a per request bill. Kafka also serves only one partition per fetch request when the data comes from the remote tier, so a heavily parallel replay is slower than the equivalent local read.

The other limits are worth knowing before you design around them. Compacted topics are not supported, which rules out most changelog and state store topics. Tiered storage administration needs clients on 3.0 or later. Topics created before 2.8.0 lack producer snapshots and cannot be tiered.

None of that changes the conclusion for high volume event topics with long retention, which is the case we keep meeting. For those we would turn it on, size local retention to comfortably cover your consumers' normal lag plus your worst case replay window, and let the cold data go to S3.

Where to start

Pick your noisiest topic, the one driving your disk sizing. Work out how far back your consumers realistically read on a bad day, double it, and make that your local.retention.ms. Enable it on a staging cluster first and watch RemoteCopyBytesPerSec and the copy lag metrics before you touch production, because a badly tuned rollout will show up as producer latency long before it shows up on the storage bill.