Deadline-aware routing, explained
Roundhouse sits between your applications and every place a job could run — your own machines and cloud APIs — and continuously answers one question: what is the cheapest executor that still meets this deadline? On the roadmap, that list grows to include capacity rented from other providers — same router, same decision. This page walks through the mechanics.
Every job carries a priority and a deadline
A job is submitted with three facts: how important it is, when it must be done, and which capability tier it needs. The router evaluates every executor that could take it — each of your on-prem machines and each cloud model of sufficient tier — and estimates completion time and cost for each.
If your own hardware can finish before the deadline, the job runs there at $0 marginal cost. The cloud is never used as a default — only as the answer to a deadline your machines can’t meet.
Re-checked every 30 seconds, escalated before it’s late
Backlogs are caught early
Locally queued work is re-evaluated continuously — every 30 seconds. If the backlog grows faster than your machines are clearing it, jobs that would miss their deadlines are identified immediately.
Outages don’t become misses
A machine dropping offline is just another sweep result: its queued jobs are re-planned on the next pass, and any that are now at risk escalate to the cloud automatically.
Cheapest capable model wins
Escalation isn’t a panic button. The router picks the least expensive cloud model that satisfies the job’s capability tier, and records the estimated cost and completion time of every model it considered.
Route by required model class, not by wishful thinking
Jobs declare one of three capability tiers, and each application can require a different one:
- Basic — bulk classification, tagging, extraction. Small local models excel here.
- Standard — summarization, drafting, structured generation.
- Frontier — hard reasoning and high-stakes output that requires a frontier-class cloud model.
Tiers are routed independently: your nightly classification batch never competes for the same executor decision as your hardest analysis job.
Every token metered — even the free ones
Every execution is metered, including on-prem runs at $0. That’s what makes the headline number honest: the dashboard shows exactly what you spent versus what the same jobs would have cost in an all-cloud setup.
Savings aren’t a projection on a pricing page — they’re computed from your own metered workload, job by job.
Spend is capped by policy, not by hope
Set a daily cloud budget. When the cap is reached, jobs hold instead of silently overspending — they wait for local capacity or for the next budget window. Jobs that genuinely can’t wait can be flagged to override the cap.
ExplainabilityEvery routing decision shows its work
For each job, Roundhouse records which executors were considered, their estimated completion times and costs, and why the winner won. When someone asks “why did this run in the cloud?”, the answer is a record, not a shrug.
# decision record (illustrative) job summarize-invoices · tier: standard deadline 14:00 · priority: normal considered: ✓ workstation-2 (local) ETA 13:41 $0.00 selected — meets deadline at zero cost bedrock · standard ETA 13:07 $0.43 passed over — not needed for deadline anthropic · standard ETA 13:06 $0.51 passed over — not cheapest capable
Illustrative decision record — the same fields the dashboard shows for every routed job.
A pip install on your machines — nothing inbound, ever
Works with your local stack
The agent is a lightweight Python client, installed with pip, that speaks to any OpenAI-compatible local server: Ollama, vLLM, LM Studio, llama.cpp. If your machine can serve a model, it can take jobs.
Outbound HTTPS only
The agent pulls work with outbound HTTPS requests — zero inbound ports. It runs behind NAT and corporate firewalls with no network changes, no tunnels, no exceptions to file with security.
The cloud side
When a deadline demands escalation, jobs run on AWS Bedrock or the Anthropic API — today’s supported cloud providers — always on the cheapest model that satisfies the job’s tier.
Built for the mixed fleet
A GPU workstation under a desk, an on-prem cluster, and a cloud API are all just executors with different ETAs and costs. The router treats them uniformly — that’s the whole trick. That’s also why the planned capacity marketplace is a routing strategy, not a rewrite.
Keep your Converse code. Change two things.
Roundhouse exposes a synchronous, Converse-compatible endpoint. A team already calling the AWS Bedrock Converse API keeps its request and response marshalling code and changes exactly two things: the endpoint URL and the auth header. An afternoon, not a quarter.
# before — the Bedrock Converse call
import boto3
client = boto3.client("bedrock-runtime")
response = client.converse(
modelId="anthropic.claude-3-5-sonnet-v1:0",
messages=[{"role": "user",
"content": [{"text": "Summarize this invoice."}]}],
)
print(response["output"]["message"])
# after — new endpoint, new auth header import requests response = requests.post( "https://YOUR_ENDPOINT/api/v1/converse", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={"modelId": "anthropic.claude-3-5-sonnet-v1:0", "messages": [{"role": "user", "content": [{"text": "Summarize this invoice."}]}]}, timeout=120, # becomes the job's deadline ) print(response.json()["output"]["message"])
Shape compatibility, stated honestly: the endpoint speaks the Converse request/response format — it is not an emulation of the AWS wire protocol, so you point your HTTP client at the new URL rather than repointing an unmodified boto3 client. An OpenAI-compatible shape is planned next. Full request, response and error reference: view the API docs.
Your timeout becomes a deadline
Behind the endpoint the call is a real routed job, its deadline set from your request timeout — placed on your own hardware when it can answer in time, escalated to the cloud when it can’t.
Your modelId maps to a tier
The modelId you send selects a capability tier. The job may run on your own machines advertising that tier — that’s the savings — and cloud escalation uses the cheapest model that satisfies it.
Same shape back, fully metered
You get a Converse-shaped response, and the call is metered like every other job — it shows up in the same saved-versus-all-cloud dashboard from the first request.
Questions platform engineers ask
What do I need to run on my own machines?
Any machine that runs an OpenAI-compatible server — Ollama, vLLM, LM Studio, or llama.cpp. The worker agent is a lightweight Python client installed with pip that connects your local server to the router.
Do I need to open inbound ports or change my firewall?
No. The worker agent makes outbound HTTPS requests only — zero inbound ports. It works behind NAT and corporate firewalls without network changes.
What happens when a machine goes offline or a backlog builds up?
Locally queued work is re-checked every 30 seconds. The moment local capacity falls behind, at-risk jobs escalate automatically to the cheapest cloud model that can do the work — before their deadlines slip.
Which cloud providers are supported?
AWS Bedrock and the Anthropic API today. When a job escalates, the router picks the cheapest capable model available on the cloud side.
How do budget caps work?
You set a daily cloud budget. When the cap is reached, jobs hold rather than silently overspending. Individual jobs can be flagged to override the cap when they truly must run.
Can different applications require different model quality?
Yes. Every job declares a capability tier — basic, standard, or frontier — so bulk classification can run on a small local model while hard reasoning goes to a frontier cloud model, each routed independently.
How is the saved-versus-all-cloud number computed?
Every execution is metered, including on-prem runs at $0. The dashboard compares what you actually spent with what the same jobs would have cost on cloud APIs alone.
Can I use Roundhouse without any hardware of my own?
Yes. Today the router right-sizes your cloud usage — capability tiers keep bulk work off frontier-priced models, each job gets the cheapest cloud model that qualifies, daily budget caps hold spend, and every execution is metered and explained. On the roadmap is a capacity marketplace where providers’ spare hardware becomes another executor, used when its price beats the cloud baseline; the early-access waitlist is open.
Do I have to rewrite my Bedrock integration to migrate?
No. The synchronous endpoint is Converse-compatible: it accepts the same request and response shape as the Bedrock Converse API. Keep your marshalling code and change two things — the endpoint URL and the auth header, which becomes a Bearer API key. Migration is an afternoon, not a quarter.
If I send a Bedrock modelId, will that exact model run my job?
The modelId you send maps to a capability tier, and the router picks the cheapest executor that satisfies it — often your own hardware advertising that tier, at $0 marginal cost. When the job escalates to the cloud, it runs on the cheapest model that meets the tier. That substitution is the product working: same shape back, routed placement underneath.
Is Roundhouse available today?
It is in private beta, invitation only. Join the waitlist and tell us about your workloads — we invite organizations as capacity opens up.
See it against your own workload
Join the private-beta waitlist and tell us what you run and on what hardware — that’s exactly the information that gets you invited sooner.