realtime-monitoring-system

Consider InfluxDB as our TSDB. According to the official InfluxDB documentation, a cluster typically handles 5K–100K field writes per second depending on hardware. Our 6K requests/sec each carry 6 metric fields, giving us 36K field writes/sec at steady state and potentially 108K at peak. This pushes even a well-provisioned InfluxDB cluster to its limits.

More critically:

• No buffer during DB downtime: If the TSDB goes down for maintenance or fails, all incoming metrics are lost.
• No stream processing hook: We need to evaluate alert rules as data flows through, not after it's persisted.
• Tight coupling: Every agent needs to know the TSDB connection details, handle retries, and deal with backpressure.

There are two fundamental approaches for getting metric data from agents to the collector:

Recommendation: Push model. For a monitoring system that requires near-real-time data with 10K+ servers, push is preferred because:

• Agents can batch and send asynchronously without waiting for polls.
• No central poll scheduler bottleneck.
• Works across firewalls and NATs without exposing server ports.
• Kafka naturally handles the firehose of push traffic.

However, Prometheus-style pull has advantages in environments where agent discovery is easy (Kubernetes service discovery) and where you want the collector to control the scrape rate. Many production systems use a hybrid: pull for Kubernetes pods, push for bare-metal servers.

Kafka topic design directly impacts parallelism and ordering guarantees:

Option 1: Partition by server_id — All metrics from a single server land in the same partition. This guarantees ordering per server (important if alert rules reference sequential data points). The stream processor can aggregate per-server metrics without cross-partition joins.

Option 2: Partition by metric name — All CPU metrics in one partition, all memory in another. This enables metric-specific processing pipelines but loses per-server ordering.

Option 3: Partition by region/environment tag — Coarse-grained. Useful for geographically distributed deployments where each region has its own processing pipeline.

Recommendation: Partition by server_id. With 10K servers and 32–64 Kafka partitions, each partition handles ~100–200 servers. This balances parallelism with per-server ordering. The partition key is a hash of server_id, so adding servers naturally distributes across partitions.

InfluxDB:

• Purpose-built TSDB with its own query language (InfluxQL / Flux).
• Excellent write performance (100K+ field writes/sec per node).
• Built-in retention policies and continuous queries.
• Limitations: Custom query language (learning curve), HA requires enterprise edition.

TimescaleDB:

• PostgreSQL extension — full SQL compatibility.
• Automatic time-based partitioning (hypertables).
• Leverages PostgreSQL ecosystem (pg_dump, replication, extensions).
• Better for teams already running PostgreSQL who want SQL familiarity.

Prometheus TSDB:

• Designed for Prometheus's pull-based model.
• Excellent for Kubernetes-native monitoring.
• Limited long-term storage — typically paired with Thanos or Cortex for durable storage.

Recommendation for this design: InfluxDB (or TimescaleDB if SQL is preferred). InfluxDB gives us the highest raw write throughput and has built-in downsampling, while TimescaleDB is a strong choice if the team values SQL and PostgreSQL tooling.

Partitioning:

Time-series data is naturally partitioned by time. Both InfluxDB and TimescaleDB support automatic time-based partitioning:

• Time-based shards (InfluxDB): Data is divided into "shard groups" by time interval (e.g., 1 day). Old shards can be dropped as a unit when the retention policy expires — this is O(1) file deletion, not row-by-row DELETE.
• Hypertable chunks (TimescaleDB): Similar concept, with automatic chunk creation.

For cross-server distribution, we can additionally partition by server_id hash or region tag to spread write load across multiple TSDB nodes.

Replication:

• Within a data center: synchronous replication (2 replicas) for strong consistency.
• Across data centers: asynchronous replication for disaster recovery.
• InfluxDB Enterprise provides built-in clustering. For OSS InfluxDB 2.x, we'd need to manage replication at the Kafka level (multiple consumer groups writing to separate DBs).

Monitoring data follows a clear access pattern: recent data is hot, old data is cold. We implement a three-tier storage strategy:

The Batch Aggregator (Apache Spark job, runs hourly) handles the transitions:

1. Read raw data older than 7 days from TSDB.
2. Compute 1-minute aggregates (avg, max, min, p99 for each metric).
3. Write aggregates to the warm tier (or a separate InfluxDB retention policy).
4. The TSDB's built-in retention policy automatically drops raw data older than 7 days.
5. After 30 days, another Spark job downsamples 1-minute data to 1-hour aggregates and archives to S3.

Cost comparison:

• InfluxDB Cloud: ~$1.44/GB/month
• Amazon S3 Standard: ~$0.023/GB/month
• Ratio: 63:1 — moving cold data to S3 cuts storage costs by 98%.

Raw alert events from Flink can be noisy — if CPU stays above 80% for an hour, Flink emits an alert event every 5 minutes (12 alerts). The Alert Service must:

1. Deduplication: Group alerts by (rule_id, server_id). If an alert is already open for this combination, don't send another notification.
2. Cooldown: After an alert triggers, suppress re-notifications for a configurable period (e.g., 15 minutes) unless severity escalates.
3. Escalation: If an alert remains open for N minutes without acknowledgment, escalate to the next level (e.g., from team Slack to on-call SMS to VP email).
4. Resolution: When the metric drops below the threshold for a full window period, the alert is automatically resolved and a "resolved" notification is sent.

State for deduplication and cooldown is stored in Redis (fast lookups by composite key) with TTL-based expiration.

Production monitoring systems support several rule types:

Threshold, rate-of-change, and absence rules can all be evaluated purely in the stream. Anomaly rules require enrichment from the TSDB (fetching baseline statistics), which adds latency but is still faster than polling.

An optimization: instead of writing every raw metric directly to the TSDB, the TSDB Writer consumer can perform in-memory pre-aggregation over a short window (e.g., 10 seconds):

• Collect all metrics arriving in a 10-second micro-batch.
• If multiple data points exist for the same (server_id, metric) within the window, aggregate them (e.g., take the latest value, or compute avg).
• Write the single aggregated value to the TSDB.

This can reduce TSDB write volume significantly if agents send metrics more frequently than the 10-second interval (e.g., sub-second application metrics), or if there's jitter causing multiple points to arrive in the same window.

With 10K servers and 6 metrics, the micro-batch holds at most 60K entries — easily fits in memory (< 10 MB with overhead).

The monitoring agent can also aggregate locally before sending:

• Metric batching: Instead of sending one metric at a time, the agent collects all 6 metrics and sends them in a single payload. This is already our design.
• Local pre-aggregation: For high-frequency metrics (e.g., per-request latency measured 1000 times/sec), the agent computes 10-second histograms and sends only the summary (avg, p50, p99, max, count) instead of 10,000 raw values.
• Delta compression: For slowly-changing metrics (disk_usage), the agent can send only the delta from the last reported value, reducing payload size.

However, be cautious: aggressive client-side aggregation can hide short-lived spikes that a 10-second average would smooth out.

Dashboard reads can spike when an incident triggers — all engineers load the dashboard simultaneously ("thundering herd" on reads). Strategies:

1. Query result caching (Redis): Cache the results of common queries (e.g., "last 1 hour CPU for server X") with a short TTL (30 seconds). Since monitoring data for a past time window is immutable, cache invalidation is time-based, not event-based.
2. Read replicas: Route dashboard queries to TSDB read replicas, keeping the primary for writes.
3. Pre-computed materialized views: For standard dashboard panels (e.g., "avg CPU across all production servers"), pre-compute the values using Flink and store in a fast K/V store.
4. Query fan-out: For cross-server aggregation queries, the Dashboard API can query multiple TSDB shards in parallel and merge results.

The SSE Gateway is a separate service that:

1. Subscribes to Kafka's server_metrics topic (using a dedicated consumer group).
2. Maintains a mapping of active SSE connections → which server_ids/metrics each client is subscribed to.
3. When a new metric arrives from Kafka, checks which SSE clients are interested and pushes the data point to them.

This avoids the dashboard needing to query the TSDB for live data. The dashboard receives data with sub-second latency after the agent sends it.

For historical data, the dashboard sends REST queries to the Dashboard API, which routes to the appropriate TSDB tier.

In practice, many teams use off-the-shelf visualization tools:

Grafana — The most popular open-source dashboarding tool. Connects to InfluxDB, TimescaleDB, Prometheus, and dozens of other data sources. Provides rich graphing, alerting, and annotation capabilities. Highly extensible with plugins.

Kibana — Best for log-centric monitoring (pairs with Elasticsearch). Excellent for full-text log search and visualization. Less optimal for pure metric dashboards.

Chronograf — InfluxData's companion tool. Tightly integrated with InfluxDB but narrower ecosystem than Grafana.

For an interview, mention that you'd use Grafana in production but the underlying architecture (how data gets to the dashboard) is what matters. Building a custom dashboard is rarely justified when Grafana exists.

If your monitoring system goes down during a production incident, you're flying blind. This creates a circular dependency: the tool that detects failures is itself a system that can fail. Staff candidates should discuss:

• Running a separate, minimal Prometheus instance that monitors only the monitoring pipeline itself.
• Heartbeat-based health checks: each component publishes a heartbeat — if a heartbeat is missed, a simple dead man's switch alerts via a completely independent channel (e.g., PagerDuty direct integration).
• Avoiding shared infrastructure between the primary monitoring pipeline and the self-monitoring pipeline.
• What metrics about the monitoring system matter: Kafka consumer lag, Flink checkpoint latency, TSDB write error rate, alert delivery success rate.

Evolving from an internal monitoring tool to a multi-tenant SaaS platform (like Datadog) introduces entirely new architectural concerns:

• Tenant isolation: Should each tenant get dedicated Kafka topics and TSDB databases, or share infrastructure with logical separation?
• Noisy neighbor: One customer sending 100× expected data volume could impact others. How do you implement per-tenant rate limiting without losing their data?
• Billing: Charge by metrics volume (data points/month), active time series, or queries? Each pricing model has different engineering implications.
• Data residency: EU customers may require their monitoring data to stay in EU regions.

Static threshold alerts (CPU > 80%) generate noise because the "normal" baseline varies: a server running a batch job at 3 AM might legitimately hit 95% CPU, while a web server at 50% during off-peak could be anomalous. Staff candidates should discuss:

• Statistical baselines: Compute per-server, per-metric, per-time-of-day baselines using historical data (e.g., 7-day rolling average with day-of-week seasonality).
• Z-score alerting: Alert when the current value deviates more than Nσ from the baseline.
• ML models: Train lightweight anomaly detection models (Isolation Forest, LSTM) on historical patterns and deploy them as Flink operators.
• Cold start problem: New servers have no baseline — how do you handle the first 7 days?
• Feedback loops: Allow engineers to mark alerts as false positives, feeding back into model training.