When scaling data pipelines or batch processing queues, “concurrency” is often treated as the ultimate lever for velocity. The logic seems simple: if 10 concurrent threads complete a batch in 10 minutes, then 100 concurrent threads should finish it in 1 minute.

But in the real world of third-party API integrations, raw concurrency eventually hits a hard wall: Rate Limiting (HTTP 429).

This post walks through how we scaled our automated batch processing pipelines to 100+ concurrent workers and made them extremely stable by replacing a naive, static failover approach with an elegant, self-regulating flow-control algorithm.

The Problem: Naive Coarse-Grained Scaling

Initially, we used a simple parallel thread pool execution strategy. We managed a queue of thousands of independent data jobs and executed them by spawning subprocess CLI/API requests in parallel:

with ThreadPoolExecutor(max_workers=55) as executor:
    # Submit jobs in parallel

At 55 concurrency, this was stable and yielded an impressive throughput. However, we wanted to maximize velocity and push the limit further—scaling to 100 concurrency.

But under a naive scaling model, high concurrency creates a thundering herd. At 100 workers, multiple threads hit the API at the exact same millisecond. The cloud API gateway detects this spike, triggers a rate limit (HTTP 429 / Quota Exceeded), and returns a non-zero exit code.

If we handle this with a naive static failover—e.g. aborting the run and retrying the whole batch at 90 or 80 concurrency—we lose precious velocity, delete transient progress, and introduce unnecessary file cleanup and state tracking overhead.

The Solution: Fine-Grained Adaptive Flow Control

Instead of treating API limits as a hard barrier that crashes the pipeline, we modified our algorithm to build self-regulating resilience directly into the thread worker’s execution loop.

We implemented a fine-grained Exponential Backoff with Randomized Jitter retry mechanism inside each concurrent thread.

1. Exponential Backoff

When a CLI/API call fails due to rate limits, the individual thread doesn’t crash the entire runner. Instead, it enters a localized retry loop.

Each subsequent failure increases the delay exponentially:

Delay = Base Delay × 2(Attempt − 1)

With a base delay of 2.0 seconds, the worker waits 2.0s on the first retry, 4.0s on the second, 8.0s on the third, and 16.0s on the fourth. This stretches the machine-scale rapid requests into human-scale time, giving the remote API’s quota bucket ample time to naturally replenish.

2. Adding Randomized Jitter

If we had 20 threads hit the rate limit at the exact same moment, and they all slept for exactly 2.0 seconds, they would all retry at the exact same millisecond—triggering another massive rate-limiting spike.

To prevent this “thundering herd” problem, we introduce randomized jitter:

Delay = (Base Delay × 2(Attempt − 1)) + random(0.5, 1.5)

By adding a random floating-point offset between 0.5 and 1.5 seconds, the threads naturally de-synchronize. They spread their retries out over time, smoothly trickling back into the API as slots open up.

Implementation Pattern

Here is a robust python-style pattern of this worker loop:

import time
import random
import subprocess

def execute_job_with_backoff(job_args, max_retries=5, base_delay=2.0):
    for attempt in range(1, max_retries + 1):
        # Run subprocess CLI or API call
        res = subprocess.run(["tool-cli", "--args", job_args], capture_output=True, text=True)
        if res.returncode == 0:
            return res.stdout
        
        if attempt == max_retries:
            raise RuntimeError(f"Job failed after {max_retries} attempts: {res.stderr}")
        
        # Calculate backoff delay with random jitter
        delay = base_delay * (2 ** (attempt - 1)) + random.uniform(0.5, 1.5)
        
        print(f"Warning: Request failed. Retrying in {delay:.2f}s (attempt {attempt}/{max_retries})...")
        time.sleep(delay)

The Result: Self-Regulating Velocity

By shifting our optimization from the macro-level (coarse-grained concurrency scaling) to the micro-level (adaptive thread flow control), we achieved remarkable improvements:

  1. Massive Concurrency Support: We can now scale our thread pool to 100+ concurrent workers safely. If we hit transient API limit blocks, the threads automatically self-throttle, spread out, and recover.
  2. Zero Hard Crashes: Transient network hiccups, rate-limiting windows, and temporary API outages are resolved inline by the individual worker.
  3. Optimized Throughput: We cut the total completion time of our multi-thousand-job batch runs significantly—converting hours of processing into a fast, hands-off evening run.

Takeaway

True velocity in software engineering isn’t just about throwing more threads at a problem. It’s about designing your threads to be polite, adaptive citizens that can automatically pace themselves to match the natural limits of the systems they interact with. Sometimes, slowing down a thread for a few human-scale seconds is the fastest way to speed up the entire pipeline.

Related: Postgres Threshold Batching