Scheduling, Queuing, and Concurrency Control
Systems that include agents and tool-driven workflows inherit a basic truth from distributed systems: work arrives in bursts, capacity is finite, and variance dominates outcomes. If the system does not decide what gets processed, when it gets processed, and how much is allowed to run at once, the system will decide on its own, usually through failure.
Scheduling and queuing are not secondary infrastructure. They are the layer that turns model capability into predictable throughput. They determine whether a service degrades gracefully under load, whether costs stay bounded, and whether user experience remains stable when traffic spikes or downstream dependencies slow down.
Value WiFi 7 RouterTri-Band Gaming RouterTP-Link Tri-Band BE11000 Wi-Fi 7 Gaming Router Archer GE650
TP-Link Tri-Band BE11000 Wi-Fi 7 Gaming Router Archer GE650
A gaming-router recommendation that fits comparison posts aimed at buyers who want WiFi 7, multi-gig ports, and dedicated gaming features at a lower price than flagship models.
- Tri-band BE11000 WiFi 7
- 320MHz support
- 2 x 5G plus 3 x 2.5G ports
- Dedicated gaming tools
- RGB gaming design
Why it stands out
- More approachable price tier
- Strong gaming-focused networking pitch
- Useful comparison option next to premium routers
Things to know
- Not as extreme as flagship router options
- Software preferences vary by buyer
Why agents amplify queuing problems
Agentic workloads are spiky by construction.
- A single user request can fan out into multiple retrieval calls, tool calls, and follow-up reasoning steps.
- Latency is high variance because external systems vary: databases, APIs, file storage, and network.
- Retries are tempting because failures are common, but retries create positive feedback loops that amplify load.
Classic web workloads have variance, but agentic workloads make variance the norm. That shifts the priority from average performance to tail behavior and backpressure.
The difference between concurrency and throughput
Teams often raise concurrency to increase throughput and accidentally destroy both.
- Concurrency is how much work is running at the same time.
- Throughput is how much work is completed per unit time.
If downstream systems saturate, increasing concurrency increases queue time, contention, and failure rates. The result is lower throughput and worse tail latency.
A stable system chooses concurrency as a control variable, not as a default scaling trick.
Capacity as a first-class contract
Scheduling is easiest when capacity is explicit.
- Token budget per request
- Maximum tool calls per workflow
- Maximum concurrent workflows per tenant
- Maximum queue depth per class of work
When these are not explicit, the system ends up with hidden queues: thread pools, database connection limits, GPU batches, or API rate limits. Hidden queues are dangerous because they are hard to observe and impossible to govern.
Admission control and backpressure
Admission control is the act of deciding whether to accept new work. Backpressure is how the decision propagates upstream.
A disciplined approach uses layered gates.
- A global gate that protects total capacity
- Per-tenant gates that enforce fairness
- Per-workflow gates that prevent runaway tool loops
- Per-dependency gates that prevent retry storms when a downstream system is degraded
Graceful degradation is not “try less hard.” It is a planned response: reduce tool calls, shorten context, switch to cached results, or return partial answers with clear boundaries.
Queue design choices that matter
Different queues encode different guarantees. Choosing the wrong guarantee creates “mystery” incidents later.
| Queue Choice | What it optimizes | Common failure mode | When it fits |
|---|---|---|---|
| FIFO (first-in, first-out) | Simplicity, fairness by arrival time | Head-of-line blocking when slow jobs appear | Homogeneous jobs with similar runtimes |
| Priority queue | Protect critical traffic | Starvation of low-priority work | Mixed workloads with clear criticality tiers |
| Weighted fair queue | Tenant fairness | Complex tuning, hidden bias via mis-weighting | Multi-tenant systems with paid tiers |
| Shortest-job-first style | Lower mean latency | Large jobs wait too long | Workloads where runtime can be estimated |
| Separate queues by class | Isolation | Over-provisioning one class while another suffers | Tool-heavy vs tool-light flows, batch vs interactive |
Agentic systems often need multiple queues: interactive user requests, background indexing, evaluation jobs, and long-running workflows. Mixing them in one FIFO line creates tail latency and unpredictable user experience.
Scheduling across GPU and CPU layers
In AI stacks, scheduling is multi-layered.
- GPU scheduling and batching determine inference throughput and tail latency.
- CPU scheduling and I/O determine retrieval, parsing, and tool call latency.
- Network scheduling determines whether downstream calls bunch together and trigger rate limits.
A queue that feeds GPU inference should be aware of batch behavior. High batch sizes improve throughput but can hurt tail latency for interactive work. Many systems adopt a two-lane approach: low-latency lane with small batches and high-throughput lane for batch work.
Tool call concurrency and rate limits
Tool calls are the fastest way to turn a stable system into an unstable one. External systems enforce rate limits and connection caps. If concurrency is unconstrained, the agent loop becomes a distributed denial-of-service against your own dependencies.
A practical control strategy:
- Limit concurrent tool calls per workflow.
- Limit concurrent tool calls per tenant.
- Apply per-tool budgets: calls per minute, concurrency caps, and cost caps.
- Use circuit breakers: when a tool errors repeatedly, stop calling it and degrade.
The goal is not perfect success. The goal is bounded failure.
Timeouts, retries, and idempotency
Retries are necessary. They are also dangerous.
If a workflow retries blindly, it multiplies load at the worst time: when a dependency is already slow or failing. The corrective pattern is to make retries conditional and observable.
- Retry only idempotent operations.
- Use exponential backoff with jitter.
- Cap total retry budget per workflow.
- Prefer “fail fast + reschedule” over “pile on now.”
Idempotency keys and deduplication are essential when tool calls change state. Without them, retries become duplicate writes.
Fairness as a product decision
Fairness is not purely technical. It is a contract with users.
- Paid tiers should be protected during bursts.
- Background tasks should yield to interactive work.
- A single noisy tenant should not degrade the whole system.
The queue is where those decisions become enforceable. Without explicit fairness, the system tends to become unfair in the worst way: the most aggressive users take the most resources.
Observability: what to measure
Scheduling work without measuring the queue is how teams get surprised.
- Queue depth per class of work
- Time in queue (p50, p95, p99)
- Processing time (p50, p95, p99)
- Concurrency utilization per dependency
- Retry rates and retry causes
- Drop rates and degradation events
- Cost per request and cost per tenant
The most important measure is often time-in-queue. It is the signal that capacity assumptions are breaking.
Load shedding and graceful degradation
When capacity is exceeded, the system must choose what to drop. Dropping work is not failure when it is planned.
Approaches that tend to work:
- Reject low-priority traffic early with a clear response rather than letting it time out.
- Convert some work to asynchronous mode: accept the request, enqueue processing, and notify when done.
- Reduce retrieval depth or switch to cached context for degraded mode.
- Switch the system to read-only posture when write tools are too risky under stress.
The key is to design degraded modes that preserve trust. A smaller, honest answer is safer than a full, wrong one.
Concurrency control for multi-step workflows
Concurrency limits should account for the fact that a workflow can hold resources for a long time.
- Limit concurrent workflows, not only concurrent requests.
- Track work-in-progress by tenant and by workflow type.
- Separate “active” concurrency from “waiting” concurrency so humans do not block capacity.
- Cap total tool calls per workflow so loops cannot run indefinitely.
A stable system behaves like a well-run airport: schedules, gates, queues, and clear rules for delays.
Prioritization strategies that avoid starvation
Priority queues can protect critical traffic and still be fair.
- Use aging: priority increases over time so low-priority work eventually runs.
- Use quotas: guarantee minimum capacity to each class of work.
- Use burst credits: allow short spikes without permanently stealing resources.
- Use separate queues for heavy batch work so it cannot block interactive traffic.
Fairness is rarely “equal.” It is “predictable.”
The relationship between queue depth and tail latency
Queue depth is not just a measure of load. It is a predictor of user experience.
- When depth grows, time-in-queue grows faster than linearly.
- When time-in-queue grows, timeouts rise.
- When timeouts rise, retries rise.
- When retries rise, depth grows again.
Breaking the loop requires controlling admission and retries. Monitoring only average latency will miss the problem until it is severe.
The operational posture that survives peak days
Peak days are not the time to discover missing controls. A durable posture includes:
- Explicit budgets for cost and tool calls
- Circuit breakers for dependencies
- Concurrency caps per queue
- Canary releases for configuration changes that affect routing or retries
- Alerts on time-in-queue and retry rates
- A degraded mode that is safe and useful
These controls turn unpredictable demand into manageable demand.
A brief checklist for stability
- Concurrency limits exist at the workflow level and at the tool level.
- Time-in-queue is monitored and alerting is based on high percentiles.
- Retry budgets exist and circuit breakers prevent storms.
- Work classes are isolated so batch work cannot block interactive work.
- Degraded mode exists and is safe: reduced tools, reduced retrieval, cached responses.
More Study Resources
- Category hub
- Agents and Orchestration Overview
- Related
- Tool Error Handling: Retries, Fallbacks, Timeouts
- Workflow Orchestration Engines and Triggers
- Context Pruning and Relevance Maintenance
- Guardrails: Policies, Constraints, Refusal Boundaries
- Prompt and Policy Version Control
- Deployment Playbooks
- Tool Stack Spotlights
- AI Topics Index
- Glossary
Books by Drew Higgins
