Getting Ready: Thread Pool

Every modern application framework has a thread pool at its core. When your web server handles 10,000 concurrent requests, it doesn't spawn 10,000 threads — it routes those requests through a fixed pool of worker threads that pull tasks from a shared queue. Java's ExecutorService, Python's concurrent.futures.ThreadPoolExecutor, Go's goroutine scheduler, and Node.js's libuv thread pool all implement this pattern.

The engineering motivation is simple but critical: thread creation is expensive. Creating a thread involves allocating a stack (typically 1 MB), registering with the OS scheduler, and initializing thread-local storage. If a web server spawns a new thread per request, it burns ~1 ms on thread setup alone — and at 10,000 req/s, that's 10 seconds of CPU time per second just creating threads. A pool of pre-created workers amortizes this cost to near zero.

This problem is a staple in concurrency interviews at companies like Google, Amazon, Netflix, and any company that builds backend systems. It tests whether you understand thread lifecycle management, producer-consumer synchronization, graceful shutdown semantics, and the subtle bugs that arise when multiple threads share mutable state. Unlike the Blocking Queue (DP33), which focuses on a single synchronized data structure, the Thread Pool integrates multiple concurrency concepts into a cohesive system.

In this design problem, you will:

• Design a complete thread pool with a fixed number of worker threads that pull tasks from a shared blocking queue
• Implement graceful shutdown — stop accepting new tasks, let in-flight tasks finish, then terminate workers cleanly
• Implement immediate shutdown — interrupt workers and discard pending tasks for emergency stops
• Build a Future/result mechanism so callers can retrieve the outcome of a submitted task
• Handle rejection policies — what happens when the queue is full and a new task arrives?
• Understand the producer-consumer pattern (Fundamentals S15) at the heart of every thread pool
• Compare Python and Java concurrency primitives — the same design uses threading.Condition in Python and ReentrantLock + Condition in Java, with significantly different idioms
• Create a UML class diagram showing the structural relationships between ThreadPool, Worker, Task, and Future

Before starting this problem, make sure you're comfortable with:

• Threads and Mutual Exclusion — creating threads, acquiring/releasing locks, the concept of critical sections (Fundamentals S14). Workers are threads; the task queue is a critical section.
• Condition Variables — how wait() and notify() coordinate threads without busy-waiting (Fundamentals S14). Workers wait on a condition when the queue is empty.
• Producer-Consumer Pattern — the foundational concurrency pattern where producers add items to a shared buffer and consumers remove them (Fundamentals S15). Task submitters are producers; workers are consumers.
• Thread-Safe Blocking Queue (DP33) — if you completed the previous problem, you already built the core data structure that the thread pool uses internally.
• OOP Composition — the pool owns its workers and its task queue (Fundamentals S4).

The Thread Pool builds directly on DP33 (Blocking Queue). If you haven't completed that problem, consider doing it first — the queue is the synchronization backbone of the pool.

We will design: A fixed-size thread pool that accepts callable tasks, executes them on worker threads, returns results via futures, handles graceful and immediate shutdown, and applies rejection policies when the queue is full.

We will NOT design: Dynamic pool resizing (min/max thread counts), work-stealing across multiple queues, scheduled/delayed task execution, distributed task queues, or priority-based task ordering. These are extensions you can explore after mastering the fixed pool.

Key simplifications:

• Fixed pool size — set at construction, never changes
• Bounded task queue — finite capacity, configurable at construction
• No task priorities — FIFO ordering from the queue
• No task dependencies — each task is independent

Why separate Python and Java tutorials? Concurrency primitives differ fundamentally between languages. Python uses threading.Lock, threading.Condition, and queue.Queue. Java uses ReentrantLock, Condition, volatile, and LinkedBlockingQueue. The design is the same; the implementation idioms are not. Each language gets its own dedicated code tutorial.

Before we design our thread pool, let's understand the mechanics that every production implementation shares:

The Core Loop:
Every worker thread runs the same infinite loop: (1) pull a task from the shared queue (blocking if empty), (2) execute the task, (3) store the result, (4) go back to step 1. When shutdown is signaled, the loop exits after the current task completes.

Task Submission:
The caller submits a Callable (a function that returns a value). The pool wraps it in a Future — a placeholder for the eventual result — enqueues it, and immediately returns the Future to the caller. The caller can later call future.get() which blocks until the result is available.

Shutdown Semantics:
Two modes exist in every production pool:

• Graceful (shutdown): Stop accepting new tasks. Workers finish their current task plus any remaining tasks in the queue. Then all threads terminate.
• Immediate (shutdownNow): Stop accepting new tasks. Workers finish their current task but discard everything left in the queue. Returns the discarded tasks so the caller can handle them.

Rejection Policies:
When the queue is full and a new task arrives, the pool must decide:

• Abort: Throw an exception — the caller handles the overload
• CallerRuns: Execute the task on the submitter's thread — back-pressure that slows the producer
• Discard: Silently drop the task — acceptable for non-critical work

Java's ThreadPoolExecutor implements all of this. Python's ThreadPoolExecutor is simpler (unbounded queue, no rejection policy). Our design will cover the full feature set.