Micro-Batching: Latency vs Throughput in Streaming

What is micro-batching?#

Micro-batching is a stream-processing strategy that collects incoming events into small, time-bounded groups and processes each group as a mini batch job. Instead of handling one record at a time (true streaming) or waiting hours for a large batch, micro-batching finds the middle ground — typically processing data every few hundred milliseconds to a few seconds.

Micro-batch vs true streaming#

True streaming engines like Flink process each event as it arrives. Micro-batch engines like Spark Structured Streaming wait for a trigger interval, collect all events that arrived during that window, and process them together.

When micro-batching wins#

• High-throughput analytics — aggregating millions of events per second where sub-second latency is acceptable
• Cost-sensitive workloads — fewer task launches means less scheduler overhead
• Exactly-once requirements — batch-level commits simplify offset management
• Teams already using Spark — reuse existing Spark knowledge and infrastructure

When true streaming wins#

• Ultra-low latency — fraud detection, real-time bidding, alerting
• Event-driven architectures — where each event triggers immediate downstream action
• Complex event processing — pattern detection across event sequences

Spark Structured Streaming internals#

Spark Structured Streaming is the most widely deployed micro-batch engine. Under the hood, each micro-batch goes through these stages:

1. Trigger fires#

The trigger interval (default: as fast as possible) determines when the next micro-batch starts. Common configurations:

# Process as fast as possible (default)
query = df.writeStream.trigger(processingTime="0 seconds")

# Fixed interval
query = df.writeStream.trigger(processingTime="10 seconds")

# Process all available data once, then stop
query = df.writeStream.trigger(once=True)

# Available-now: process all available data in multiple batches, then stop
query = df.writeStream.trigger(availableNow=True)

2. Offset range computation#

The engine queries the source (Kafka, file system, etc.) for the latest available offsets. It computes the range: from the last committed offset to the latest available offset.

3. Micro-batch planning#

Spark creates a standard batch execution plan — logical plan, physical plan, and optimised DAG. The key insight is that all of Spark's catalyst optimisations apply to micro-batches too.

4. Execution#

The micro-batch runs exactly like a normal Spark batch job — tasks are distributed across executors, data is shuffled as needed, and results are computed.

5. Commit#

Once the batch completes, the engine commits the end offsets to the checkpoint location. This atomic commit is what enables exactly-once semantics.

Windowing strategies#

Windowing determines how events are grouped for aggregation. Micro-batch engines support all standard window types:

Tumbling windows#

Fixed-size, non-overlapping time intervals. Every event belongs to exactly one window.

from pyspark.sql.functions import window

df.groupBy(
    window("event_time", "5 minutes")
).agg({"value": "sum"})

Sliding windows#

Fixed-size windows that advance by a slide interval. Events can belong to multiple windows.

df.groupBy(
    window("event_time", "10 minutes", "2 minutes")
).agg({"value": "avg"})

Session windows#

Dynamic windows that close after a gap of inactivity. Useful for user session analysis.

df.groupBy(
    session_window("event_time", "30 minutes")
).agg({"page_views": "count"})

Late data and watermarks#

Real-world data arrives late. Watermarks tell the engine how long to wait for late data before finalising a window:

df.withWatermark("event_time", "10 minutes") \
  .groupBy(window("event_time", "5 minutes")) \
  .agg({"value": "sum"})

Events arriving more than 10 minutes late are dropped. This bounds the state the engine must maintain.

Exactly-once semantics in micro-batch#

Micro-batching simplifies exactly-once delivery through a three-part mechanism:

1. Source replay#

Sources like Kafka support offset-based replay. If a micro-batch fails mid-way, the engine re-reads from the last committed offset.

2. Deterministic processing#

Given the same input offsets, the engine produces the same output. No randomness, no external calls that could vary between retries.

3. Idempotent sinks#

The sink must handle duplicate writes gracefully. Strategies include:

• Transactional writes — write output and commit offsets in the same transaction
• Idempotent inserts — use unique keys so re-inserts are no-ops
• Two-phase commit — coordinate between the engine and the sink

Spark Structured Streaming achieves end-to-end exactly-once with its checkpoint mechanism combined with idempotent sinks like Delta Lake.

Latency tradeoffs#

The trigger interval is the primary latency knob:

The scheduling tax#

Each micro-batch incurs fixed overhead: query the source for new offsets, plan the batch, launch tasks, commit offsets. For a 100 ms trigger, this overhead can consume a significant fraction of the interval.

Adaptive batching#

Some systems adapt batch size dynamically. When load is high, batches grow larger (higher throughput). When load is low, batches shrink (lower latency). Spark's default trigger (processingTime="0 seconds") achieves this naturally — it starts the next batch as soon as the previous one finishes.

Micro-batching at scale#

Checkpoint management#

Checkpoints grow over time and must be cleaned. Strategies:

• Compaction — periodically merge small checkpoint files
• Retention policies — delete checkpoints older than a threshold
• External state stores — use RocksDB-backed state for large stateful operations

Monitoring key metrics#

• Batch duration vs trigger interval — if batches take longer than the trigger, you are falling behind
• Input rate vs processing rate — sustained imbalance means you need more resources
• State size — unbounded state growth indicates missing watermarks
• Scheduler delay — time between trigger fire and batch start

Scaling patterns#

• Vertical — increase executor memory and cores per micro-batch
• Horizontal — add more Kafka partitions and Spark executors
• Source parallelism — match the number of Spark partitions to source partitions

When to move beyond micro-batch#

Consider migrating to true streaming when:

1. You need consistent sub-100 ms latency
2. Your processing logic is inherently per-event (no aggregations)
3. You need complex event processing with temporal patterns
4. Your team has the expertise to manage Flink or Kafka Streams

For most analytics, monitoring, and ETL workloads, micro-batching delivers the right balance of simplicity, throughput, and latency.

---

Explore more system design guides at codelit.io/blog.