Table of Contents

Most teams building data systems face the same architectural question, and they answer it wrong. They assume real-time is always better. It’s not. Real-time systems are expensive, operationally fragile, and often solve problems that don’t actually need solving at millisecond precision. Batch processing, by contrast, is boring. It’s also reliable, cheap, and solves more problems than you think.

The choice between batch and real-time isn’t actually about latency alone. It’s about cost, operational burden, failure recovery, and whether your business problem actually requires sub-second answers. Most of the time, it doesn’t.

What Batch Processing Is (And Why It Still Wins)

Batch processing means collecting data over a window of time—minutes, hours, or days—and processing it all at once. You accumulate writes to a queue, a log, or a database table. Then, at a scheduled interval, you run a job that reads all that data, transforms it, and writes the results to a destination.

A concrete example: you’re running a SaaS platform with usage-based billing. Every API call writes an event to a Kafka topic or a database table. Once per hour, a batch job reads all events from the last hour, aggregates them by customer and feature, and updates the customer’s usage counters. At the end of the month, you run another batch job that reads all daily aggregates and generates invoices. This is simple, durable, and testable.

The operational advantages are real. Batch jobs are easy to debug because they process a bounded set of data. You can replay a job. You can test it with production data in a staging environment before running it live. Failures are isolated: if a batch job fails at 3 AM, you wake up, fix the bug, and re-run the job. No cascading failures through a distributed system. The cost is also low because you can run batch jobs on spare capacity—off-peak times, cheaper hardware, or cloud spot instances that cost 80% less than on-demand.

The trade-off is latency. If you process once per hour, the worst-case delay for a single event to affect your system is one hour. For billing systems, that’s acceptable. For user-facing analytics, it might be too slow. The key insight is this: for most business problems, an hour is fine. We’ve built systems where the business asked for “real-time” dashboards, but when we pushed back and asked “how fast do you actually need this?” the answer was always “end of day” or “every 15 minutes.” Real-time was never the actual requirement.

What Real-Time Actually Costs

Real-time processing means ingesting events and producing results with minimal latency—seconds or less, often milliseconds. To ship this, you need a message broker (Kafka, RabbitMQ, AWS Kinesis, Google Pub/Sub), a stream processor (Kafka Streams, Flink, Spark Streaming, or custom code), and stateful aggregation logic. You need to handle out-of-order events, late-arriving data, and windowing. You need monitoring because failure modes are subtle and fast.

The infrastructure cost is substantial. A Kafka cluster with three brokers, replication, and monitoring costs thousands per month. A Flink job running 24/7 on a Kubernetes cluster costs hundreds per month just to keep the cluster warm. If you’re processing low-volume data—say, 1000 events per second—you’re paying for infrastructure that’s 99% idle. With batch processing, you spin up compute only when you need it, process the data in minutes, and shut down.

The operational cost is even higher. Real-time systems have failure modes that batch systems don’t have. A Kafka broker can lag. A stream processor can deadlock or run out of memory and crash. Late-arriving events can cause incorrect results that you have to detect and fix retroactively. State management is hard: if your stream processor is aggregating data by customer ID, and a customer ID is mismatched or a user is deleted, the state becomes inconsistent. Debugging is harder because you can’t easily replay a subset of data through the system.

We built a real-time billing system for a client once. Events flowed into Kafka, a Flink job aggregated them by customer, and results were written to a data warehouse every 30 seconds. The cost was $8K per month in infrastructure. Three months in, we discovered that a bug in the aggregation logic had been overcounting events for customers who made multiple API calls in the same second. We couldn’t easily replay the data. We had to manually query Kafka’s log, reconstruct the correct counts, and patch the database. The fix took a week and cost the client $50K in manual refunds. With a batch job that ran once per hour, we would have caught the bug in testing and replayed the job in minutes.

Latency Trade-Offs: The Math That Matters

The latency difference between batch and real-time looks dramatic on paper. A real-time system can process an event in 100 milliseconds. A batch system processes once per hour. But the actual user experience depends on what you’re measuring.

Consider an e-commerce platform tracking inventory. When a customer buys an item, you want to update the inventory count. With batch processing, inventory updates run every 5 minutes. So the worst-case latency is 5 minutes: a customer might see an item as in-stock, add it to their cart, and then find it out-of-stock at checkout because another customer bought it in the last 5 minutes.

With real-time processing, you update inventory in Kafka immediately. The latency is 100-200 milliseconds. But—and this is critical—the system is now more complex. You need to handle: (1) duplicate events (what if the same purchase event is written to Kafka twice?), (2) out-of-order events (what if the “out of stock” message arrives before the “sold out” message?), and (3) consistency (what if a customer’s payment fails after inventory is decremented?). These problems are solvable, but they’re not free.

The real question is: does the 5-minute delay actually hurt your business? For most e-commerce use cases, the answer is no. Overselling happens, and you handle it with a “sorry, we sold out” email. You lose a few orders per month due to the batch window. Real-time processing might prevent 10 oversells per month but costs 10x as much to run and requires a team to maintain it. The economics don’t work.

There are cases where latency matters. High-frequency trading systems need microseconds. Ad targeting systems need to serve personalized ads in 50 milliseconds. But these are exceptions. Most systems can tolerate latency in the 5-minute to 1-hour range. If you’re building a real-time system because you think you need it, you probably don’t.

Operational Complexity and Failure Modes

Batch systems fail in predictable ways. A batch job either succeeds or fails. If it fails, you see an error in your logs, you fix the code, and you re-run the job. The data is bounded—you know exactly which events were processed and which weren’t. Recovery is straightforward: re-run the failed job on the same input data.

Real-time systems fail in ways that are harder to detect and fix. A stream processor can lag without crashing. If a Kafka consumer is slow, messages pile up in the topic, and the system falls further and further behind real-time. By the time you notice, you might be hours behind. Now you have to catch up, and during the catch-up period, the system is producing stale results.

State corruption is another failure mode. If your stream processor is maintaining state in memory (like a count of events by customer), and the process crashes, that state is lost. You can recover from a persistent state store (like RocksDB), but that adds complexity. And if the state store and the output topic get out of sync—which can happen if there’s a failure between writing state and writing output—you have inconsistent data that’s hard to fix.

Consider a real-time recommendation system. A stream processor ingests user behavior events and updates user profiles in real-time. The profiles are stored in Redis for fast lookup. If the stream processor crashes, Redis is stale. When it comes back online, it’s processing new events, but the user profiles are old. For a few minutes, recommendations are based on stale data. This is usually acceptable, but it’s a failure mode that batch systems don’t have.

We’ve seen teams invest months building real-time systems only to discover that the operational overhead isn’t worth the latency gain. One team built a real-time anomaly detection system using Kafka and Flink to detect fraudulent transactions in milliseconds. The system worked, but it required on-call engineers to monitor it 24/7. When Flink crashed (which happened about once per month), they had to wake up someone at 2 AM to fix it. After a year, they switched to batch processing every 15 minutes. Fraud detection latency went from milliseconds to 15 minutes. False negatives increased slightly, but the operational burden dropped by 80%, and the team’s quality of life improved significantly.

Hybrid Architectures: The Practical Middle Ground

The best systems don’t choose batch or real-time. They use both. You ingest events into a durable log (Kafka, AWS Kinesis, or even S3 with a Lambda trigger) in real-time. For user-facing features that need low latency, you run a stream processor that produces results to a cache or a fast database. For analytics and billing, you run batch jobs that process the same events periodically and produce more complete, auditable results.

A concrete example: an advertising platform. Real-time requirements: when an advertiser uploads a campaign, it should be live within 30 seconds. When an ad is clicked, the click should be counted immediately so the advertiser can see real-time metrics. Batch requirements: accurate billing based on clicks and impressions, fraud detection, and daily performance reports.

The hybrid approach: events (clicks, impressions) flow into Kafka. A Kafka Streams job aggregates clicks and impressions by campaign every 10 seconds and writes to Redis. Advertisers see near-real-time metrics from Redis. A separate batch job runs every hour, reads all events from Kafka since the last run, re-aggregates them with careful deduplication, and writes to the data warehouse. Billing and fraud detection run on the data warehouse data, which is durable and auditable.

This hybrid approach costs less than pure real-time because batch processing is cheap, and it’s more reliable because the batch job serves as a source of truth. If the real-time metrics in Redis get out of sync, the batch job fixes them. If Kafka loses a broker, the batch job can re-process the events when the cluster recovers. The real-time metrics are best-effort; the batch results are authoritative.

The key to making this work is idempotency. Every event should be processable multiple times without changing the result. With careful idempotency keys (which we’ve covered in depth in our post on idempotency keys for payment processing), you can run batch jobs that re-process the same events without creating duplicates. This lets you recompute results when you find a bug, without worrying about double-counting.

Decision Framework: How to Choose

Here’s the framework we use to decide between batch and real-time:

Choose batch if: The actual business requirement is latency measured in minutes or hours (not what the stakeholder said, but what they actually need). You don’t have the operational budget for 24/7 on-call. The data volume is moderate (less than 10K events per second). The computation is complex or requires external lookups. You need to be able to recompute results if you find a bug. Cost is a constraint.

Choose real-time if: Users directly interact with the result (like a recommendation or a personalized ad). The latency requirement is sub-second, and you’ve validated that the business actually needs it. You have the operational budget to maintain the infrastructure. Data volume is high enough that batch windows would be unacceptably large. The computation is simple (aggregation, filtering, basic transformations).

Choose hybrid if: You have both real-time and batch requirements. You want real-time metrics but need batch-based billing or analytics. You want to reduce operational risk by having a batch job as a source of truth.

One practical test: ask the stakeholder what they’d do if the system was 1 hour behind. If they say “that’s fine,” batch is acceptable. If they say “we’d lose money” or “users would complain,” dig deeper. What’s the actual cost? How many users are affected? Is it worth the infrastructure cost to prevent it? Usually, the answer is no.

A real example: we were hired to build a usage analytics dashboard for a SaaS platform. The stakeholder said “it needs to be real-time.” We pushed back and asked what real-time actually meant. They said “I want to see usage metrics update when I refresh the page.” We asked “how often do you refresh?” They said “maybe once per hour.” So they needed a 1-hour batch window, not real-time. We built a batch job that runs every 15 minutes, storing results in PostgreSQL. The cost was $200 per month. A real-time system would have cost $5K per month and required on-call coverage.

The hard truth: real-time is seductive. It sounds fast and modern. But most teams don’t actually need it. They need reliable, auditable, and cost-effective data processing. Batch systems deliver that. Real-time is a solution to a specific class of problems. If you’re not sure you have that problem, you probably don’t.

If you’re at a point where you’re designing a data pipeline or rebuilding an aging system and you’re not sure whether to invest in real-time infrastructure, it’s worth getting a senior engineer’s perspective on the actual latency requirements and the operational trade-offs. We take three engagements a quarter by application, and this kind of architectural decision is exactly what a Sprint engagement covers—a focused audit with a clear recommendation on batch vs. real-time, including cost modeling and risk assessment.