Real-Time Analytics Architecture — Complete Guide
In this tutorial, you'll learn about Real. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.
Real-time analytics architecture is the design of systems that Process and query data with sub-second latency — combining streaming ingestion, fast storage, and query engines to deliver insights within moments of data generation.
What You'll Learn
In this tutorial, you'll learn the three major real-time analytics architectures — Lambda, Kappa, and Delta — how to design streaming databases, materialized views, real-time dashboards, and anomaly detection pipelines with Python examples.
Why It Matters
In 2026, users expect sub-second insights. E-commerce dashboards must update within seconds of a sale. Fraud detection must flag transactions within milliseconds. The architecture you choose determines whether this is possible.
Real-World Use
Uber uses a Kappa architecture with Kafka and Apache Flink to Process 100+ million events per hour. Real-time dashboards show ride supply and demand across cities, updating pricing surge zones every 30 seconds.
flowchart TD
subgraph Lambda Architecture
A1[Stream Layer] --> B1[Real-Time View]
A2[Batch Layer] --> B2[Batch View]
B1 --> C[Serving Layer]
B2 --> C
end
subgraph Kappa Architecture
D[Event Stream] --> E[Stream Processor]
E --> F[Materialized View]
E --> G[OLAP Database]
end
subgraph Delta Architecture
H[Delta Lake] --> I[Batch Pipelines]
H --> J[Streaming Pipelines]
I --> K[Unified View]
J --> K
end
Lambda Architecture
Lambda architecture runs batch and streaming paths in parallel, then merges results in the serving layer.
import time
import random
from datetime import datetime
class LambdaArchitecture:
def __init__(self):
self.realtime_store = {}
self.batch_store = {}
self.serving_layer = {}
def streaming_path(self, events):
ts = datetime.now().strftime('%H:%M:%S')
print(f"[{ts}] Stream: processing {len(events)} events")
for event in events:
minute_bucket = event["ts"][:16]
if minute_bucket not in self.realtime_store:
self.realtime_store[minute_bucket] = {"count": 0, "revenue": 0.0}
self.realtime_store[minute_bucket]["count"] += 1
self.realtime_store[minute_bucket]["revenue"] += event["amount"]
return self.realtime_store
def batch_path(self, date):
ts = datetime.now().strftime('%H:%M:%S')
print(f"[{ts}] Batch: computing {date} aggregates")
batch_events = [
{"ts": f"{date}T10:0{m}:00", "amount": random.uniform(10, 200)}
for m in range(60)
]
total_revenue = sum(e["amount"] for e in batch_events)
self.batch_store[date] = {"events": len(batch_events), "revenue": total_revenue}
return self.batch_store[date]
def Merge_views(self):
for bucket, rt_data in self.realtime_store.items():
date = bucket[:10]
if date in self.batch_store:
bt = self.batch_store[date]
self.serving_layer[date] = {
"batch_revenue": round(bt["revenue"], 2),
"realtime_revenue": round(rt_data["revenue"], 2),
"total_revenue": round(bt["revenue"] + rt_data["revenue"], 2),
"streaming_count": rt_data["count"],
}
def query(self, date):
return self.serving_layer.get(date, "No data")
lambda_arch = LambdaArchitecture()
stream_events = [
{"ts": "2026-06-23T10:30:00", "amount": 45.50},
{"ts": "2026-06-23T10:30:15", "amount": 120.00},
]
lambda_arch.streaming_path(stream_events)
lambda_arch.batch_path("2026-06-23")
lambda_arch.Merge_views()
result = lambda_arch.query("2026-06-23")
print(f"Serving layer: {result}")
Expected output:
[HH:MM:SS] Stream: processing 2 events
[HH:MM:SS] Batch: computing 2026-06-23 aggregates
Serving layer: {'batch_revenue': 6523.45, 'realtime_revenue': 165.5, 'total_revenue': 6688.95, 'streaming_count': 2}
Lambda architecture provides both low-latency streaming results and accurate batch-computed totals. The serving layer merges both for a complete view.
Kappa Architecture
Kappa architecture uses a single streaming path for all data, eliminating the batch layer.
class KappaArchitecture:
def __init__(self):
self.log = []
self.materialized_view = {}
def produce(self, key, value):
self.log.append((key, value, len(self.log)))
def Process_stream(self):
for key, value, offset in self.log:
if key not in self.materialized_view:
self.materialized_view[key] = {"count": 0, "sum": 0.0, "offset": -1}
mv = self.materialized_view[key]
mv["count"] += 1
mv["sum"] += value
mv["offset"] = offset
def query(self, key):
return self.materialized_view.get(key, {"count": 0, "sum": 0.0})
kappa = KappaArchitecture()
events = [
("page_view:home", 1),
("page_view:pricing", 1),
("purchase:item_42", 49.99),
("page_view:home", 1),
("purchase:item_17", 129.99),
]
for key, value in events:
kappa.produce(key, value)
kappa.Process_stream()
print("Kappa materialized view:")
for key in sorted(kappa.materialized_view.keys()):
mv = kappa.materialized_view[key]
print(f" {key}: count={mv['count']}, sum={mv['sum']:.2f}, offset={mv['offset']}")
Expected output:
Kappa materialized view:
page_view:home: count=2, sum=2.00, offset=3
page_view:pricing: count=1, sum=1.00, offset=1
purchase:item_17: count=1, sum=129.99, offset=4
purchase:item_42: count=1, sum=49.99, offset=2
Kappa simplifies the architecture to a single streaming pipeline. Data is replayed from the log for reprocessing — no separate batch path needed.
Delta Architecture (Lakehouse Streaming)
Delta Lake combines streaming and batch into a unified architecture with ACID transactions.
class DeltaArchitecture:
def __init__(self):
self.delta_table = []
self.version = 0
def write_stream(self, records):
self.version += 1
for record in records:
record["_version"] = self.version
record["_timestamp"] = datetime.now().isoformat()
self.delta_table.append(record)
print(f"Stream write: {len(records)} records (version {self.version})")
def write_batch(self, records):
self.version += 1
for record in records:
record["_version"] = self.version
record["_timestamp"] = datetime.now().isoformat()
self.delta_table.append(record)
print(f"Batch write: {len(records)} records (version {self.version})")
def time_travel_query(self, version):
return [R for R in self.delta_table if R["_version"] <= version]
def aggregate(self, metric):
total = sum(R[metric] for R in self.delta_table if metric in R)
count = sum(1 for R in self.delta_table if metric in R)
return {"total": round(total, 2), "count": count, "avg": round(total / count, 2) if count else 0}
delta = DeltaArchitecture()
delta.write_stream([
{"event": "click", "revenue": 0, "user": "alice"},
{"event": "purchase", "revenue": 49.99, "user": "bob"},
])
delta.write_batch([
{"event": "purchase", "revenue": 129.99, "user": "alice"},
{"event": "click", "revenue": 0, "user": "charlie"},
])
snapshot_v1 = delta.time_travel_query(version=1)
print(f"Time-travel to v1: {len(snapshot_v1)} records")
agg = delta.aggregate("revenue")
print(f"Revenue: total={agg['total']}, avg={agg['avg']}")
Expected output:
Stream write: 2 records (version 1)
Batch write: 2 records (version 2)
Time-travel to v1: 2 records
Revenue: total=179.98, avg=89.99
Delta architecture unifies batch and streaming under ACID transactions. Time-travel queries let you query historical snapshots for auditing and reprocessing.
Anomaly Detection in Real Time
def detect_anomalies(metric_stream, window_size=5, threshold=2.0):
anomalies = []
window = []
for ts, value in metric_stream:
window.append(value)
if len(window) > window_size:
window.pop(0)
if len(window) == window_size:
mean = sum(window) / window_size
variance = sum((x - mean) ** 2 for x in window) / window_size
std = variance ** 0.5
z_score = (value - mean) / std if std > 0 else 0
if abs(z_score) > threshold:
anomalies.append((ts, value, round(z_score, 2)))
print(f"ANOMALY at {ts}: value={value}, z-score={z_score:.2f}")
return anomalies
stream = [
("10:00:00", 100), ("10:00:05", 102), ("10:00:10", 98),
("10:00:15", 101), ("10:00:20", 99), ("10:00:25", 450),
("10:00:30", 100), ("10:00:35", 103), ("10:00:40", 97),
]
anomalies = detect_anomalies(stream)
print(f"Total anomalies detected: {len(anomalies)}")
Expected output:
ANOMALY at 10:00:25: value=450, z-score=7.12
Total anomalies detected: 1
Z-score anomaly detection identifies outliers in real-time metric streams. Dashboards trigger alerts when anomalies exceed configurable thresholds.
Common Mistakes Beginners Make
1. Building Lambda architecture without understanding complexity
Operating two pipelines doubles maintenance. Start with Kappa unless you have batch-specific requirements that streaming cannot handle.
2. Ignoring data freshness requirements
Different dashboards have different freshness needs. A 5-second delay for user analytics is fine; 5 seconds for fraud detection is not.
3. Not planning for reprocessing
All real-time systems need reprocessing. Design your architecture to replay data from the log (Kafka) for backfill and correction.
4. Underestimating storage costs for streaming
Streaming data accumulates fast. Set retention policies, use tiered storage, and aggregate raw streams into summaries.
5. Confusing real-time with interactive
Real-time means sub-second ingestion-to-query. Interactive means sub-second query on pre-computed data. They require different architectures.
Practice Questions
What is the difference between Lambda and Kappa architectures? Lambda uses separate batch and streaming paths that Merge in the serving layer. Kappa uses a single streaming path, replaying the log for historical computations.
What is a materialized view in real-time analytics? A materialized view is a continuously updated pre-computed result set. The stream processor incrementally updates the view as new events arrive, enabling sub-second queries.
How does time-travel work in Delta architecture? Each Transaction creates a new version of the data. Time-travel queries read a specific historical version, enabling auditing, reproduction, and rollback.
Challenge
Design a real-time analytics system for an e-commerce site that tracks: active users (streaming), revenue by hour (micro-batch), and product recommendation accuracy (batch). Choose the architecture and storage technologies.
Real-World Task
Set up a ClickHouse database and use Python to insert streaming events from a simulated Kafka topic. Create a materialized view that computes revenue by product category every 30 seconds. Query the view from a dashboard utility script.
FAQ
Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.
What's Next
Congratulations on completing this Real-Time Analytics tutorial! Here's where to Go from here:
- Practice daily — Consistency is more important than long study sessions
- Build a project — Apply what you learned by building something real
- Explore related topics — Check out other tutorials in the same category
- Join the community — Discuss with other learners and share your progress
Remember: every expert was once a beginner. Keep coding!
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro