prefork vs gevent Celery Pools for eDiscovery Batching

A Celery deployment that pins CPUs at 100% while throughput stays low, or one that idles the CPU while thousands of downloads crawl, is almost always running the wrong pool for its workload. This is the defining configuration decision in the async batch processing design stage: Celery’s prefork pool and its gevent pool are optimized for opposite bottlenecks, and eDiscovery pipelines contain both. Hashing, OCR, and text extraction are CPU-bound; pulling files from cloud storage, calling a metadata API, or writing to a remote index are I/O-bound. Running one pool for both wastes the resource the workload is actually starved of. This guide compares the two pools on the axes that govern eDiscovery throughput and delivers a routing strategy that matches each task class to the right pool.

Diagnostic Log Signatures

The wrong-pool symptom is a resource imbalance: saturated CPU with low throughput, or idle CPU with backlogged I/O. Capture it verbatim.

[INFO] Celery worker: pool=gevent, concurrency=500, queue=hashing
[WARN] cpu=99% throughput=0.4x expected; greenlets blocked on CPU-bound hashing
[INFO] Celery worker: pool=prefork, concurrency=8, queue=downloads
[WARN] cpu=6% queue_depth=48210 rising; 8 processes blocked on network I/O
[WARN] memory_rss climbing: prefork forks each holding a large payload in RAM
[CRITICAL] Pool mismatch: CPU work on gevent, I/O work on prefork

Symptom checklist — two or more indicate a pool mismatch:

• A gevent worker saturates CPU while its throughput is far below expectation (CPU-bound tasks starving the event loop).
• A prefork worker sits near-idle on CPU while an I/O-bound queue backs up (too few processes for concurrent waits).
• Increasing gevent concurrency does not increase throughput on a CPU-bound queue.
• Prefork memory grows with concurrency because each forked process holds a full payload.
• Throughput improves markedly when a queue is moved from one pool to the other.

How the Two Pools Differ

The prefork pool runs a set of OS processes, each executing one task at a time with true parallelism across CPU cores. Because each task gets its own process and its own Python interpreter, CPU-bound work — SHA-256 hashing, OCR, PDF rendering — runs in genuine parallel, unaffected by the Global Interpreter Lock. The cost is memory (each process is a full fork) and a low practical concurrency (roughly one process per core), so prefork is poor at holding thousands of concurrent waits.

The gevent pool runs many greenlets (lightweight cooperative coroutines) inside a single process, multiplexing I/O with an event loop. A greenlet that blocks on network I/O yields to others, so one process can hold thousands of concurrent in-flight downloads or API calls cheaply. But greenlets share one interpreter and one core: a CPU-bound task does not yield, so it blocks every other greenlet until it finishes, collapsing concurrency. gevent is excellent for I/O-bound fan-out and useless for CPU-bound crunching.

The eDiscovery reality is that a single pipeline has both kinds of work, so the answer is not “which pool” but “which pool for which queue.”

Routing Strategy

Split the pipeline by bottleneck and give each task class the pool it needs.

1. Classify each task by bottleneck. Hashing, OCR, extraction, and rendering are CPU-bound; storage fetches, metadata-API calls, and remote index writes are I/O-bound. Tag each task accordingly.
2. Dedicate queues and pools. Route CPU-bound tasks to a prefork worker with concurrency near the core count; route I/O-bound tasks to a gevent worker with high concurrency. Run them as separate worker deployments consuming separate queues.
3. Size each pool to its resource. Prefork concurrency ≈ available cores (leave headroom); gevent concurrency to the point where added greenlets stop improving throughput or the downstream service rate-limits.
4. Keep CPU work off the event loop. Never let a CPU-bound task land on the gevent worker, because one such task stalls every concurrent I/O operation sharing its process.

The split below routes the two task classes to their matched pools.

Task-routing implementation

The routine classifies a task by its bottleneck and returns the queue (and thus the pool) it should run on, plus a suggested concurrency for each pool. It compiles and runs on the standard library.

import os
import logging
from dataclasses import dataclass
from enum import Enum
from typing import Dict

logger = logging.getLogger("ediscovery.celery.route")


class Bottleneck(str, Enum):
    CPU = "cpu"
    IO = "io"


# Map each task name to the resource it is bound by. CPU-bound tasks must never
# land on the gevent pool, where one of them would stall every I/O greenlet.
TASK_BOTTLENECK: Dict[str, Bottleneck] = {
    "hash_document": Bottleneck.CPU,
    "ocr_page": Bottleneck.CPU,
    "render_image": Bottleneck.CPU,
    "extract_text": Bottleneck.CPU,
    "download_from_storage": Bottleneck.IO,
    "call_metadata_api": Bottleneck.IO,
    "write_to_index": Bottleneck.IO,
}

QUEUE_FOR = {Bottleneck.CPU: "cpu_prefork", Bottleneck.IO: "io_gevent"}


class RoutingError(Exception):
    pass


def route_task(task_name: str) -> str:
    """Return the queue a task belongs on. An unclassified task is a hard error
    rather than a default, so CPU work can never silently land on gevent."""
    try:
        bottleneck = TASK_BOTTLENECK[task_name]
    except KeyError as exc:
        raise RoutingError(f"unclassified task '{task_name}'; classify before routing") from exc
    return QUEUE_FOR[bottleneck]


def suggested_concurrency(pool: str, cores: int | None = None) -> int:
    """prefork ~ core count (true parallelism); gevent high (cheap waits)."""
    cores = cores or (os.cpu_count() or 4)
    if pool == "prefork":
        return max(1, cores - 1)          # leave a core for the OS/broker
    if pool == "gevent":
        return 500                        # tune up until the downstream rate-limits
    raise RoutingError(f"unknown pool '{pool}'")


# Celery routing configuration derived from the classification.
task_routes = {name: {"queue": QUEUE_FOR[b]} for name, b in TASK_BOTTLENECK.items()}

Verification Checklist

Confirm the pools match the workload before scaling out:

• CPU-bound tasks (hashing, OCR, rendering, extraction) run only on the prefork worker, sized near the core count.
• I/O-bound tasks (downloads, API calls, remote writes) run only on the gevent worker with high concurrency.
• No CPU-bound task can land on the gevent queue — unclassified tasks fail routing rather than defaulting.
• Prefork memory stays bounded because concurrency is core-limited, not inflated.
• Moving a queue between pools no longer improves throughput — each is on its correct pool.

Conclusion

The prefork and gevent pools are optimized for opposite bottlenecks, and an eDiscovery pipeline has both, so the throughput mistake is picking one pool for everything. Prefork gives GIL-free multi-core parallelism for hashing, OCR, and rendering but holds few concurrent waits; gevent holds thousands of cheap concurrent I/O operations but collapses the instant a CPU-bound task refuses to yield. Classify every task by its bottleneck, route CPU work to a core-sized prefork worker and I/O work to a high-concurrency gevent worker on separate queues, and never let CPU work touch the event loop. Matched this way, each pool is saturated on the resource its workload actually needs, and the pipeline’s total throughput is the sum of both instead of the bottleneck of one.

Frequently Asked Questions

Why does a CPU-bound task destroy gevent’s concurrency?

Because gevent’s concurrency depends on greenlets yielding at I/O boundaries. A greenlet running a CPU-bound task like hashing never hits an I/O wait, so it never yields — it holds the single shared interpreter and core for the full duration of the computation, and every other greenlet in that process is blocked until it finishes. One CPU-bound task therefore serializes what looked like thousands of concurrent operations. That is why CPU work must run on prefork, where each task has its own process.

Can I run one pool and just tune concurrency?

Not effectively, because the two workloads are bound by different resources. Turning up prefork concurrency past the core count mostly adds memory and context-switching, not throughput, for CPU work — and it still blocks a whole process per I/O wait. Turning up gevent concurrency helps I/O but does nothing for CPU-bound tasks, which do not yield. The resource each workload is starved of is different, so one pool cannot be tuned to serve both; separate pools on separate queues is the correct design.

How should I size the two pools?

Size prefork to roughly the core count, leaving a core or two for the OS and the broker, because its parallelism is bounded by cores. Size gevent much higher — hundreds to a few thousand greenlets — and raise it until throughput stops improving or the downstream service (cloud storage, an API) begins to rate-limit or error, which is the real ceiling for I/O concurrency. Monitor CPU utilization on the prefork worker and queue depth plus downstream latency on the gevent worker to find each pool’s sweet spot.

Related

• Async Batch Processing Design — the parent stage this pool routing configures.
• Implementing Celery for Async eDiscovery Batching — the Celery worker model these pools deploy.
• Cryptographic Hash Generation — a CPU-bound task class that belongs on prefork.
• Native File Ingestion Pipelines — the I/O-bound storage fetches that belong on gevent.

For the pools, see the Celery worker pool documentation and the gevent documentation.

Up one level: Async Batch Processing Design — the subsystem that scales processing across a worker pool.