ticketmaster

Start with the simplest possible approach: a single service that handles all operations — search, reservation, payment — with a single relational database.

The Flow

1. User searches for events → direct database query
2. User selects a seat → UPDATE seat SET status = 'reserved' WHERE seat_id = X
3. User pays → synchronous call to payment provider
4. On payment success → UPDATE seat SET status = 'booked'

Why This Breaks

This design fails under three simultaneous pressures:

• Read contention: 100K concurrent users all querying the same event's seats. Every seat map request hits the database. At 100K QPS with complex seat map queries, the database connection pool is exhausted.
• Write races: 20,000 users trying to reserve the same popular seats. Without proper concurrency control, two users reserve the same seat → double booking.
• Synchronous payment: The reservation holds a database connection while waiting for Stripe/PayPal to respond (1-3 seconds). At 2,000 concurrent payments, that's 2,000 database connections held open doing nothing.

✅ Works for: Small venue with a few hundred seats and gradual ticket sales

❌ Fails at scale: Database is the bottleneck for both reads and writes. No caching, no queue, no separation of concerns. Double-booking possible without explicit locking. Synchronous payment blocks resources.

The fundamental insight: reads and writes have completely different characteristics and should be handled by different infrastructure.

• Reads (event search, seat availability): high volume, tolerant of slight staleness (a seat showing as 'available' for an extra second is acceptable)
• Writes (reservation, booking): low volume but must be strictly consistent (no double-booking)

Read Path: Search Service + Cache

The Search Service handles all read operations. Event data and venue information changes infrequently, so it's cached aggressively:

• Event catalog → cached with long TTL (hours)
• Seat availability → cached with short TTL (seconds) in a Seat Cache (Redis or Memcached)

When a user opens the seat map, the Search Service reads from the Seat Cache. The cache is updated when seats are reserved or released. This offloads 95%+ of read traffic from the database.

Write Path: Booking Service

The Booking Service handles all reservation and booking operations. When a user reserves a seat:

1. Booking Service receives the reservation request
2. Attempts to mark the seat as reserved in the database
3. Updates the Seat Cache to reflect the new status
4. Returns reservation details + payment link to the user

The Problem: Concurrent Write Races

Even with read/write separation, 20,000 concurrent booking attempts can still cause race conditions. Two users simultaneously read a seat as available, both attempt to reserve it, and both succeed — double booking.

✅ NFRs addressed: Read scalability via caching (burst traffic absorbed), separation of concerns

❌ Still missing: No ordering of write requests — race conditions on concurrent reservations. Synchronous payment still blocking. No reservation expiry mechanism. No waitlist.

The write race problem from Step 2 has a clean solution: serialize all reservation requests through an ordered message queue.

Instead of the Booking Service writing directly to the database, it pushes reservation requests into a message queue (Kafka, SQS, or RabbitMQ). A Queue Consumer processes requests one at a time in FIFO order.

The Reservation Flow

1. User clicks "Reserve" → Booking Service enqueues the request
2. Booking Service immediately returns to the user: "Your seat is being reserved..." (spinner on UI)
3. Queue Consumer picks up the request:
   • Checks if the seat is still available in the database
   • If yes: marks it as reserved, updates the cache, triggers payment flow
   • If no: rejects the request, notifies the user
4. The queue ensures first-come, first-served ordering — the first request enqueued wins

This eliminates write races by design: only one consumer processes reservations for a given seat at a time. The queue also acts as a buffer during traffic spikes — it absorbs burst writes that would otherwise overwhelm the database.

Asynchronous Payment

Payment is decoupled from the reservation. After a seat is reserved:

1. Booking Service sends a payment link to the user (via SSE/notification)
2. User completes payment through the external provider
3. Payment provider sends a webhook to our Payment Service
4. Payment Service records the transaction and notifies Booking Service
5. Booking Service transitions the seat from reserved to booked
6. A message is sent to the queue to update the Seat Cache

Why a Queue Instead of Database Locking?

Database-level locks (e.g., SELECT ... FOR UPDATE) also prevent double-booking, but they hold connections open under contention. With 20,000 concurrent requests for the same event, lock contention cascades into connection pool exhaustion and timeouts. The queue decouples admission from processing — the database sees a steady, manageable stream of writes regardless of how bursty the incoming traffic is.

The queue also provides natural fairness: requests are processed in arrival order. With database locking, the request that acquires the lock first wins — which may depend on network latency rather than user arrival time.

✅ NFRs addressed: No double-booking (serialized writes), first-come first-served ordering, async payment (no blocking resources), burst absorption

❌ Still missing: No reservation expiry — what if a user reserves but never pays? No notifications to users about reservation status. No waitlist. No protection against bots and unfair access.

The final architecture addresses the remaining gaps: reservation timeout, user notifications, waitlist management, and bot protection.

Reservation Expiry via Delayed Task Scheduler

When a seat is reserved, the system simultaneously schedules a delayed task (e.g., 2 minutes later). The Scheduler checks: is the seat now booked? If not, the reservation expired without payment — release it back to available.

Implementation options:

• Redis key with TTL: Set a key reservation:{id} with a 2-minute TTL. On expiry, a notification triggers the release logic.
• Delayed message queue: Send a message with a 2-minute delivery delay. When the consumer receives it, check the seat status.
• Scheduled task service: A cron-like service that polls for expired reservations every N seconds.

The delayed message queue approach is simplest and most reliable — it's idempotent (checking status before acting) and doesn't require polling.

Notification Service

The Booking Service needs to notify users about:

• Seat is being reserved — proceed to payment
• Payment successful — booking confirmed
• Reservation expired — seat released
• Waitlist offer — it's your turn to reserve

This is done through a Notification Service that consumes events from the message queue and pushes updates to the user via Server-Sent Events (SSE). SSE is ideal here: communication is server-to-client only, and it works over standard HTTP — no WebSocket upgrade needed.

Waitlist Implementation

When a user tries to reserve an already-reserved seat, they can join a waitlist. The waitlist is a FIFO queue per seat (Redis list is ideal):

• Join: LPUSH waitlist:{event_id}:{seat_id} {user_id}
• Dequeue: RPOP waitlist:{event_id}:{seat_id} → returns the longest-waiting user

When a reservation expires (user didn't pay), the Scheduler releases the seat and immediately dequeues the next waitlisted user, giving them a fresh reservation window.

Fair Access and Bot Protection

During high-demand events, bots can monopolize the booking flow. Protections include:

• Virtual waiting room: Before on-sale time, users enter a queue. At on-sale, users are admitted in random or first-come order at a controlled rate.
• CAPTCHA/verification: Challenge users before allowing reservation to filter automated traffic.
• Rate limiting: Per-user and per-IP request limits.
• Lottery system: For extremely high-demand events, randomly select users who get the opportunity to purchase (used by some real-world ticketing systems).
• Time-slot allocation: Spread buying activity across intervals to prevent simultaneous request tsunamis.

NFR Scorecard

Double-booking is the cardinal sin of a ticketing system. This deep dive explores three complementary approaches at different layers.

Layer 1: Application-Level Serialization (Queue)

The ordered message queue is the primary defense. All reservation requests for a given event are routed to the same queue partition (partitioned by event_id). The consumer processes them sequentially. When it encounters a request for a seat that's already reserved, it rejects the request — no race condition possible.

This works because the queue is ordered and the consumer is single-threaded per partition. For systems handling multiple events simultaneously, each event gets its own partition — parallelism across events, serialization within an event.

Layer 2: Database-Level Pessimistic Locking

As a second line of defense (defense in depth), the Queue Consumer uses a pessimistic lock when updating the seat:

The FOR UPDATE lock prevents any other transaction from modifying this row until the current transaction commits. Even if (due to a bug) two consumers process the same seat concurrently, the lock serializes them at the database level.

Layer 3: Optimistic Concurrency Control

An alternative or complementary approach is optimistic locking with a version field:

Optimistic locking is better for high-contention scenarios because it doesn't hold a lock while processing — it tries the update and handles failure. But for the ticketing use case where the queue already serializes writes, pessimistic locking in the Consumer is safe (no contention, since only one Consumer processes a given event's requests).

Cache-Database Consistency

A subtle edge case: the Seat Cache shows a seat as available but the database has it as reserved. Two users see it as available, both submit reservation requests. The queue serializes them — only the first succeeds. The second gets rejected and can join the waitlist.

The cache is eventually consistent by design. It's updated after each reservation and release, but there's always a small window (milliseconds to seconds) where the cache lags behind the database. This is acceptable because the cache is not the source of truth — the database is, and the queue ensures only valid reservations succeed.

When a seat is already reserved, the user shouldn't just see "unavailable" — they should have a way to queue up. If the current holder doesn't complete payment, the next person in line gets a shot.

Data Structure: Redis List per Seat

A Redis list provides O(1) push/pop operations and natural FIFO ordering:

Waitlist Lifecycle

1. User A reserves seat A-101 (status → reserved)
2. User B tries to reserve A-101 → seat is taken → offered to join waitlist
3. User B joins → LPUSH waitlist:evt:A-101 user_B (position 1)
4. User C joins → LPUSH waitlist:evt:A-101 user_C (position 2)
5. User A's reservation expires (didn't pay within 2 minutes)
6. Scheduler releases seat A-101 → status back to available
7. Scheduler dequeues User B → RPOP waitlist:evt:A-101
8. System creates a fresh reservation for User B with a new 2-minute window
9. User B receives SSE notification: "Seat A-101 is available — complete payment now!"

Edge Cases

• What if User B's app is closed when they're dequeued? Send a push notification and give them a slightly longer window (e.g., 5 minutes instead of 2).
• What if the waitlist is very long? Show the user their position and a realistic estimate ("unlikely" if they're position 50 for a single seat).
• What if the user no longer wants the seat? Allow them to leave the waitlist (LREM), which doesn't affect other users' positions.

When 10 million fans refresh the page at exactly 10:00 AM for a Taylor Swift concert with 80,000 seats, the system faces a thundering herd problem. Without protection, the first few hundred milliseconds determine everything — and bots are faster than humans.

Virtual Waiting Room

Before the on-sale time, users enter a virtual queue (like a checkout line). The system works like this:

1. Users arrive at the event page before on-sale time and click "Join Queue"
2. Each user gets a random queue position (lottery-based) or a first-come position (arrival-based)
3. At on-sale time, the system admits users in controlled batches (e.g., 1,000 users every 5 seconds)
4. Admitted users can browse available seats and make reservations
5. Users still in the queue see their position and estimated wait time

The waiting room is typically implemented as a separate lightweight service (or even a CDN-level gate) that sits in front of the API Gateway. It serves a static waiting page to queued users and only routes admitted users to the actual Booking Service.

Why Random Position?

Arrival-based ordering (first to click "Join" gets position 1) rewards fast internet connections and bots. A randomized lottery at on-sale time gives everyone who joined before on-sale an equal chance, regardless of when they clicked (as long as it was before the cutoff). This is fairer and simulates a physical venue where doors open simultaneously.

Bot Protection Stack

Time-Slot Allocation

For extremely high-demand events, split the available inventory across multiple time slots (e.g., 10:00 AM, 10:15 AM, 10:30 AM). Each slot opens a portion of seats. This spreads the traffic spike into three smaller peaks, each manageable by the system. Users are assigned a time slot when they join the waiting room.

Most events see modest traffic — a local theater show or a minor league game. But a Taylor Swift or Beyoncé concert can spike traffic 100× within seconds. If you provision for peak Taylor Swift traffic all the time, you're wasting 99% of your infrastructure budget most of the time.

Traffic Tiers

Classify events into traffic tiers based on expected demand:

Pre-Scaling for Mega Events

For known mega events, the system pre-scales before the on-sale time:

1. Cache warming: Pre-populate the Seat Cache with the entire venue layout before on-sale. No cache misses during the initial rush.
2. Queue partitioning: Create dedicated queue partitions for the mega event. Standard events share a common partition pool.
3. Read replica scaling: Spin up additional read replicas for the event's database shard 30 minutes before on-sale.
4. CDN pre-staging: Push static event assets (venue map, images, seat layouts) to CDN edge nodes in advance.
5. Auto-scaling triggers: Set aggressive auto-scaling rules that trigger on queue depth rather than CPU (queue depth reacts faster to traffic spikes).

Geographic Sharding

Events are naturally geographic — a concert in LA is only relevant to users in the western US (mostly). Shard the Seat DB by geographic region:

• North America West, North America East, Europe, Asia-Pacific
• Each region has its own Search Service, Seat Cache, and queue infrastructure
• Global events (e.g., World Cup finals) are handled by a dedicated global cluster

This contains blast radius — a traffic spike for an LA concert doesn't affect users browsing events in London.

Post-Event Cleanup

After on-sale completes (usually within 30-60 minutes), automatically scale down the dedicated infrastructure. Set an auto-cleanup timer to decommission extra replicas and cache entries after a cooling period.

The following topics contain open-ended architectural questions for staff+ conversations.