Skip to content
All guides

Network cost

Cross-Zone Traffic from Stateful Workloads: Kafka, Databases, Caches and Search

Kafka, PostgreSQL, Redis and Elasticsearch copy every write to other zones, and that traffic is worth paying for. Most of their reads cross zones too, and those you can cut: the config for each system, what it saves, and what each fix can break.

KT

Kubeadapt Team

Updated 6 min read
  • Kubernetes
  • Cross-AZ
  • Kafka
  • PostgreSQL
  • Redis
  • Elasticsearch
  • Strimzi
Cross-Zone Traffic from Stateful Workloads: Kafka, Databases, Caches and Search

Kafka, PostgreSQL, Redis and Elasticsearch copy every write to other zones so they survive losing one. That traffic crosses zones on purpose, and it is worth what it costs. Their reads usually cross too, to a leader or primary in another zone, even when a copy of the data sits in the reader's own zone. Those reads are the part you can cut.

Which traffic to keep and which to cut

TrafficExamplesCut it?
Replication of writesKafka followers, PostgreSQL standbys, Elasticsearch replica shardsNo. It protects you from losing a zone
Writes from clientsKafka producers, writes to a database primaryOnly where the system supports it, such as Kafka producers since 4.4
Reads from clientsKafka consumers, read queries, cache gets, searchesYes. A copy in the same zone can usually serve them

Managed services take part of this off the bill. Amazon MSK doesn't bill replication between brokers, so only client traffic adds to the data transfer line. ElastiCache bills only your application's side of a connection that crosses zones. Self-managed systems on Kubernetes pay for every byte. Azure stopped charging between zones in 2024; the cross-AZ data transfer guide covers the other clouds.

Kafka consumers: read from the replica in their zone

A Kafka consumer reads each partition from its leader, wherever that leader runs. With three brokers, one per zone, two of every three leaders sit in another zone. Yet with a replication factor of three, every broker holds a copy of every partition. Follower fetching, from KIP-392 and in Apache Kafka since 2.4.0, lets the consumer read from the copy in its own zone.

Where Kafka consumers read from

Three brokers, one per zone, with every partition copied to all three. Each dot is one fetch.

Animated diagram of three zones, each with two consumer pods and one broker. Reading from the leader, about two of every three fetches go to a broker in another zone; fetching from the closest replica, every fetch stays in its own zone.

It takes three settings: broker.rack on every broker, replica.selector.class set to RackAwareReplicaSelector, and client.rack on every consumer. With Strimzi 1.3.1, the rack section sets broker.rack from the node's zone label, and the selector goes in config:

yaml
1apiVersion: kafka.strimzi.io/v1
2kind: Kafka
3metadata:
4  name: my-cluster
5spec:
6  kafka:
7    rack:
8      type: topology-label
9      topologyKey: topology.kubernetes.io/zone
10    config:
11      replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector

Your own consumers need client.rack set to their zone. From Kubernetes 1.35, the PodTopologyLabels admission plugin copies the node's zone label onto each pod, so the Downward API can pass it in:

yaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: orders-consumer
5spec:
6  template:
7    spec:
8      containers:
9        - name: consumer
10          image: registry.example.com/orders-consumer:1.8.0
11          env:
12            - name: KAFKA_CLIENT_RACK
13              valueFrom:
14                fieldRef:
15                  fieldPath: metadata.labels['topology.kubernetes.io/zone']

Map it to client.rack in your application. On older clusters, read the node label in an init container.

What can go wrong

If client.rack is empty or doesn't match broker.rack exactly, the consumer reads from the leader without an error, and only the traffic shows it. Reads also cross again while a broker restarts or its copy falls behind. Follower fetching may add latency too, because followers hear about new messages after the leader.

Consumer placement now decides which broker serves the reads:

Share of reads each broker serves

Example: half the consumers run in us-east-1a, 30% in us-east-1b and 20% in us-east-1c. The dashed line is an even share for three brokers. Spreading consumers evenly over the zones keeps the brokers level.

Bar chart of three brokers. Reading from leaders spread evenly, each broker serves about a third of the reads. With follower fetching, the broker in us-east-1a serves half of them, the broker in us-east-1b 30% and the broker in us-east-1c 20%.

Kafka producers: rack-aware partitioning

Producers write each record to its partition leader, so they cross zones the same way. Apache Kafka 4.4.0 (October 2026) adds rack-aware partitioning from KIP-1123: the producer prefers partitions whose leader runs in its own zone.

properties
client.rack=us-east-1a
partitioner.rack.aware=true
# Applies to records the partitioner places itself: no key, or
# every record when partitioner.ignore.keys=true

It only applies to records the producer places itself: records without a key, or all records with partitioner.ignore.keys=true, which gives up keeping each key on one partition. It is a setting of the Java producer. KIP-1123 also warns that producers must be spread fairly evenly, because with producers in only two zones the third zone's partitions receive nothing.

What the two fixes save

Take an example cluster with one broker per zone and 10 TB a month produced, which three consumer groups each read in full. At the AWS rate of $0.02 per GB that crosses (us-east-1, November 2026), it pays for three flows:

FlowCost todayWith both fixes
Producers to partition leaders$133$67, if half the records have no key
Replication to two followers$400$400
Consumers reading from leaders$400about $0
Total per month$933$467

Follower fetching does most of the work, and replication doesn't move: it is what keeps a zone failure from losing data. Each consumer group added later without client.rack costs another $133 a month here.

Database reads from a replica in the same zone

Keep replication across zones and send read-only queries to an instance in the client's zone. With CloudNativePG 1.30, spread the instances over zones and add a read Service that prefers the same zone:

yaml
1apiVersion: postgresql.cnpg.io/v1
2kind: Cluster
3metadata:
4  name: orders-db
5spec:
6  instances: 3
7  affinity:
8    topologyKey: topology.kubernetes.io/zone
9    podAntiAffinityType: required
10  postgresql:
11    synchronous:
12      method: any
13      number: 1
14      dataDurability: required
15  managed:
16    services:
17      additional:
18        - selectorType: r
19          serviceTemplate:
20            metadata:
21              name: orders-db-r-local
22            spec:
23              trafficDistribution: PreferSameZone
24  storage:
25    size: 100Gi

The read Service uses selectorType: r, which includes the primary. The default -ro Service covers only replicas, so with three instances in three zones the primary's zone has no replica to read from. On Kubernetes 1.31 to 1.33, write PreferClose (see topology-aware routing).

Replicas lag the primary, so reads that must see the last write stay on the -rw Service. With dataDurability: required, writes pause when no synchronous standby is available; two standbys let you lose a zone without that.

Redis and Valkey: read from the replica in your zone

Put a replica in each zone and use a client that reads from the one in its own zone. Valkey GLIDE calls this AZ affinity. It relies on the availability-zone setting added in Valkey 8, which you set on each node when you run Valkey yourself.

python
1import os
2from glide import GlideClusterClientConfiguration, NodeAddress, ReadFrom
3
4client_config = GlideClusterClientConfiguration(
5    [NodeAddress("valkey-cluster.cache.svc", 6379)],
6    read_from=ReadFrom.AZ_AFFINITY,
7    client_az=os.environ["POD_ZONE"],  # topology.kubernetes.io/zone via the Downward API
8)

The zone the client passes must match the one the server reports. Replica reads can be slightly stale, so keep reads that must see the last write on the primary. For a cache behind a plain Service, trafficDistribution: PreferSameNode or PreferSameZone does the same job without client changes.

Elasticsearch and OpenSearch

Use shard allocation awareness so a shard and its replica never share a zone, and keep searches on a node in the client's zone. On Elastic Cloud on Kubernetes, zoneAwareness sets this up:

yaml
1apiVersion: elasticsearch.k8s.elastic.co/v1
2kind: Elasticsearch
3metadata:
4  name: logs
5spec:
6  version: 9.5.6
7  nodeSets:
8    - name: default
9      count: 6
10      zoneAwareness: {}

Turn it on for every NodeSet at once, since adding it restarts all of them. A client Service with trafficDistribution: PreferSameZone then keeps the first hop local. In OpenSearch, the search preference _local runs on local shard copies where it can, and its docs warn that this can create hotspots.

Volumes that pin a pod to its zone

An EBS volume only attaches to an instance in its own zone, so a pod that claims one is scheduled there. Disk traffic never crosses a zone. The pod can never move to follow its callers, though, and if its zone fails, it waits until the zone comes back.

WaitForFirstConsumer creates each volume only after its pod is scheduled:

yaml
1apiVersion: storage.k8s.io/v1
2kind: StorageClass
3metadata:
4  name: gp3-zonal
5provisioner: ebs.csi.aws.com
6volumeBindingMode: WaitForFirstConsumer
7parameters:
8  type: gp3
9  csi.storage.k8s.io/fstype: xfs

With the default Immediate mode, a volume can land in a zone where the pod's spread constraint won't let it run, and the pod never starts. Set the zone spread constraint on the StatefulSet before its first rollout, since changing it later doesn't move existing volumes.

Seeing which consumer or reader crosses zones

Kafka clients connect straight to broker pods, and replicas stream from their primary by pod address, so none of it passes through a Service. The bill never names the pod either. Finding the reader that crosses takes traffic measured per workload; measuring cross-AZ traffic per workload compares the methods.

Kubeadapt's network agent counts bytes at each pod's own network interface and keeps the real pod address on both ends. In Network Intelligence, Explorer lists each workload pair with its zones, bytes and cost, most expensive first, priced at AWS or Azure rates or your own rate card. A consumer group without client.rack stands out next to the groups already reading locally. For reads through a Service, Network Intelligence names the Services that would save money with topology-aware routing.

Kubeadapt doesn't change your cluster; your team makes the change. With Kubeadapt MCP in Claude Code, ask which workloads read from brokers in another zone and what each costs, and Claude Code can add client.rack to those manifests in a pull request you review. Afterwards, Explorer shows whether those pairs stopped crossing, and a cost spike alert on the namespace flags it if a new consumer group ships without client.rack.

Before you enable the network agent

The network agent requires a chart version later than 1.0.8. Do not enable it with chart version 1.0.8 or earlier on a cluster that enforces network policy. The agent docs list its requirements.

Install the agent with the quick start, or book a demo to see Network Intelligence on a running cluster.

Sources

  1. Amazon EC2 On-Demand Pricing: Data Transfer within the same AWS Region
  2. Amazon MSK pricing
  3. Amazon ElastiCache pricing
  4. KIP-392: Allow consumers to fetch from closest replica
  5. KAFKA-8443: Allow broker to select a preferred read replica for consumer (fix version 2.4.0)
  6. KIP-1123: Rack-aware partitioning for Kafka Producer
  7. KAFKA-19193: Rack-aware partitioning for Kafka Producer (fix version 4.4.0)
  8. Apache Kafka downloads (4.4.0 released October 22, 2026)
  9. Apache Kafka 4.4 producer configs (partitioner.rack.aware, client.rack)
  10. Apache Kafka 4.4 broker configs (broker.rack, replica.selector.class)
  11. Apache Kafka 4.4 consumer configs (client.rack)
  12. Strimzi 1.3.1: Configuring (rack awareness, consuming from the closest replicas)
  13. Kubernetes: Admission controllers (PodTopologyLabels)
  14. Kubernetes: Service traffic distribution
  15. CloudNativePG 1.30: Replication
  16. CloudNativePG 1.30: Service management
  17. CloudNativePG 1.30: Scheduling
  18. CloudNativePG 1.30: API reference (ServiceSelectorType)
  19. Valkey GLIDE: Specify a read strategy
  20. Valkey blog: AZ affinity strategy
  21. Elasticsearch: Shard allocation awareness
  22. Elastic Cloud on Kubernetes: Advanced Elasticsearch node scheduling (zoneAwareness)
  23. Elasticsearch: Search shard routing
  24. OpenSearch: Search shard routing
  25. Kubernetes: Storage classes (volume binding mode)
  26. Kubernetes: Running in multiple zones (storage access for zones)
  27. Amazon EBS volumes
  28. Amazon EBS CSI driver: StorageClass parameters and topologies