AWS open-sources Infinity, a Rust agent runtime that fits about 75,000 idle AI agents in 8 GB

Amazon Web Services has released Infinity, an open-source Rust runtime designed to let very large numbers of AI agents wait on slow tools without holding processes, threads or network connections while they wait. The project was introduced on October 23, 2026 on the AWS Open Source Blog by Shadaj Laddad, research lead for the Hydro team at AWS, and is published under the Apache 2.0 license as part of the Hydro project, a collaboration between AWS and the Sky Computing Lab at UC Berkeley. AWS describes the release as an early research prototype rather than a managed service, and it is asking developers for feedback on the design.

The central claim is about density. According to AWS, an idle Infinity agent occupies roughly 100 KB of memory, a figure that grows linearly with conversation history, which means that about 75,000 agents fit in 8 GB, the memory of a Raspberry Pi 5. The project documentation gives the measurement behind that figure: about 103 KB per agent after 20 tool-calling turns, and 8.4 GB for 80,000 idle resident agents. The benchmark that AWS cites ran 80,000 agents through 3.2 million completions on a single thread of an AMD EPYC 9R14 processor, using in-memory stores and a simulated model rather than a real language model.

Infinity ships with three connected pieces: the runtime itself, a tool protocol called the Reactive Agent Protocol (RAP), and a coding agent called Infinity Code that is built on the runtime. The same agent code is meant to run either as a resident process on a laptop or as a stateless, event-driven deployment on AWS Lambda, for which the repository includes AWS CDK constructs.

··········

THE PROBLEM INFINITY ADDRESSES: AGENTS THAT BLOCK WHILE TOOLS RUN.

The motivation AWS gives is that agents spend much of their lifetime waiting. A coding agent waits for a build or a test suite, an operations agent waits for a deployment to finish, and a support agent may wait days for a human reply or an external approval. Most agent harnesses implement the agent loop as a sequence of blocking steps, so while a tool runs, the agent keeps a process and its memory allocated, and it generally cannot react to other events that arrive in the meantime.

That design is acceptable when a developer runs a handful of agents interactively, but it scales poorly when an organization wants thousands of long-lived agents that each spend most of their time idle. The cost is dominated by resources reserved for waiting rather than by model inference. Infinity's answer is to make the waiting state as small as possible: between turns, an agent is reduced to its conversation history in a store, with no task, no stack and no open connection, and it is reconstructed only when a new message arrives for it.

AWS frames the approach with an explicit analogy to asynchronous programming, saying the project does for agents what async did for threads. The comparison is technical rather than rhetorical, because the execution model is borrowed directly from how modern languages implement asynchronous functions.

··········

STACKLESS COROUTINES AND BOUNDED SLICES.

Infinity models an agent as a stackless coroutine, the same technique Rust uses to compile `async fn`. In a stackless coroutine, the state needed to resume execution is captured explicitly rather than kept on a live call stack. In Infinity's case, the conversation history plays the role of the continuation: everything the agent needs in order to continue is already contained in the messages it has exchanged.

Execution is divided into bounded units that the project calls slices. A slice loads the thread's history from a pluggable conversation store, runs one model completion, persists the output, dispatches at most one tool call, and then ends. A tool call therefore acts as an `await` point. The slice never blocks on the tool, and the tool's result comes back later as a new input message that triggers a fresh slice.

The architecture documentation breaks a slice into three phases. In the load phase, the runtime restores history from the ConversationStore and the set of already processed message IDs from the StateStore. In the preparation and completion phase, a `prepare_input` step deduplicates messages, drops messages addressed to closed threads, routes subscription events and appends whatever is actionable; if anything is actionable, `run_completion` streams a completion from the model provider. In the dispatch phase, `execute_action` runs the tool call, persists the remaining state and ends the slice. On Lambda the process exits at that point, while in an embedded runtime the worker task simply returns to waiting on its channel.

Not every tool has to yield. Tools that implement `Tool::execute_synchronous`, such as the built-in `spawn_thread`, run inline, write their result to history and loop back into another completion inside the same slice. Unknown tool names follow the same path with an injected error result, so a model that hallucinates a tool name receives feedback instead of stalling the thread.

··········

ONE INPUT QUEUE FOR EVERY KIND OF EVENT.

A defining design choice is that every reason to resume an agent arrives through the same channel. User text, tool results, subscription events, reports from child agents, OAuth challenges and timer wake-ups are all represented as a single `InputMessage` type, each carrying a `group_id` that identifies its conversation thread. Dequeuing a message launches a slice that reloads the corresponding state.

An Infinity agent system is a pool of agent threads that share stores, tools and a model. AWS compares the design to an actor system: each thread has its own input queue, threads share no mutable state, and messages for one thread are processed in order, one slice at a time, while different threads run in parallel. When a thread has nothing to do, its task exits, and it is restarted only when its next message arrives. Because the runtime schedules these threads much like an async executor schedules tasks, thousands of agents can run on a few operating-system threads.

The tool interface reflects the same principle. Tools implement a `Tool<ChannelSender>` trait with `name()`, `description()`, `parameters()` and an asynchronous `execute()` method. The `execute()` call starts the work and returns immediately; a separately spawned task later computes the result and sends it back to the input queue using the `thread_id` and tool call ID provided in the tool context. AWS states that this queue send is the tool's only interaction with the runtime, and the blog's example notes that a 100-millisecond lookup and a three-day lookup use identical tool and agent code. Because the sender is defined as a trait, the queue behind it can be in memory or durable.

........

Component	Role in Infinity
InputMessage	Single input type for text, tool results, events, timers
Input queue	Per-thread FIFO delivery of InputMessages
ConversationStore	Conversation history and compaction summaries
StateStore	Message IDs each thread has already processed
ModelProvider	Supplies completions through run_completion
Tools and RAP tools	Start work, return later through the queue

........

··········

DURABILITY, DEDUPLICATION AND ORDERING GUARANTEES.

Running agents on serverless infrastructure means accepting at-least-once message delivery, retries and processes that can disappear at any moment. Infinity handles this with several explicit mechanisms. Every input message and every completion has a stable ID, and `prepare_input` skips any ID the StateStore has already recorded for that thread, so a redelivered message is harmless rather than producing a duplicate turn.

Model output is also treated transactionally. Streamed tokens are written to a turn buffer and committed only when the turn ends, either because the completion finished or because it produced a tool call. If the stream fails mid-turn, the buffer is discarded and the completion is retried. When a turn ends in a tool call, the buffer is flushed before the tool is dispatched, which guarantees that the call is durably recorded before its result could possibly arrive.

Ordering is FIFO per `group_id`. On AWS this maps to an SQS FIFO queue with one `MessageGroupId` per thread, with Lambda processing one invocation per active group; in an embedded runtime it maps to one `mpsc` channel and one worker task per thread. Spawning a child agent creates a new message group, which is why parent and child threads can run in parallel without locks.

For long-running agents, the local driver triggers compaction at roughly three quarters of the model's context window. A dedicated compaction thread writes a summary to the ConversationStore, tagged with the history index it covers, and later slices load that summary plus only the messages that follow it. Children spawned before a compaction can still rebuild their inherited history. AWS acknowledges, however, that compaction for large, long-running agent trees has not yet been studied in depth.

··········

SUBSCRIPTIONS, SYNTHETIC TOOL CALLS AND CHILD AGENTS.

Infinity also tries to solve a problem that most agent frameworks leave open: how to deliver an event to a model when the event was not requested in the current turn. Chat-style model APIs expect each tool call to receive exactly one result, and they offer no native slot for "something happened while you were waiting." AWS says it rejected three obvious workarounds. Sending a second result for the same tool call breaks the one-result rule, presenting the event as a user message misattributes its origin, and inserting it into the system prompt breaks the ordering of events.

The runtime instead injects a synthetic assistant tool call named `receive_event__injected`, carrying the arguments `original_tool_name`, `original_tool_call_id` and `original_args`, followed by a tool result that contains the event. The `__injected` suffix marks calls created by the runtime; that name is never offered to the model, and the runtime rejects it if the model tries to call it. A tool's first result opens a subscription, and later messages for that call are tagged as events, which can wake idle agents exactly as any other message does. This is how Infinity streams shell output chunk by chunk or reports a background test failure to an agent that has moved on to other work.

The default handling of such events is called threaded. Each event goes to a temporary child thread that inherits the parent's context; the child can act, report to the parent, or close silently, leaving the parent's history untouched, and concurrent events each get their own child. The alternative, associative mode, appends the event directly to the subscribing thread's history, which AWS suggests for updates about an operation the agent is actively following, such as background test output.

Multi-agent structures reuse the same machinery. Every agent has a built-in `spawn_thread` tool that creates a child thread with its own input queue and behaves like a subscription: the call returns once the child exists, and the child's later reports arrive as events on that call. Children report with `report_to_parent` and terminate with `close_thread`, so no separate inter-agent messaging system is required.

··········

THE REACTIVE AGENT PROTOCOL AND MCP COMPATIBILITY.

RAP extends the non-blocking model across the network. Its specification uses HTTP as the only transport: the runtime sends a single POST request to the tool's registered endpoint, including the arguments and a callback URL, and the tool acknowledges immediately with HTTP 200 before processing the request asynchronously. Results, subscription events, OAuth authorization requests and user-choice prompts are later posted back to that callback URL, together with a group ID that identifies the conversation thread. Messages are JSON encoded in UTF-8.

Because no connection stays open, RAP tool servers can scale like ordinary web services and can deliver results to agents that currently have no running process at all. Tools are described in a toolset, a declarative JSON manifest containing the server's endpoint and an array of tool definitions with names, descriptions and JSON Schema arguments, fetched from a well-known discovery endpoint. The specification leaves authentication to implementations but recommends established mechanisms such as AWS SigV4, bearer tokens or mutual TLS, and says callback URLs should be authenticated to prevent message injection.

Infinity does not require tool authors to abandon the Model Context Protocol. The project documentation states that stdio and HTTP MCP servers connect in process and that existing MCP servers run unchanged through a compatibility layer, while the RAP specification is open for other runtimes and tool servers to implement.

··········

SERVERLESS DEPLOYMENT ON AWS LAMBDA.

In what the project calls step mode, the runtime keeps no tasks alive between calls. The platform invokes a `step` function for each batch of delivered messages, and the process may exit afterwards, which is why the documentation claims that an idle agent consumes no compute at all. The reference deployment combines several AWS services, each mapped to one of the runtime's requirements.

........

AWS service	Function in the reference deployment
SQS FIFO	Input queue, one MessageGroupId per thread
AWS Lambda	Runs one runtime step per delivery (ARM64 Rust binary)
Aurora DSQL	Stores conversation histories
DynamoDB	Stores deduplication state
SQS delays, EventBridge Scheduler	Implement sleep tools and timers
Amazon Bedrock	Model access, Anthropic models enabled

........

The serverless quickstart builds the stack with an `InfinityAgent` CDK construct and requires Node.js 20 or later, pnpm, AWS credentials and either cargo-lambda or Docker. Interaction happens through SQS rather than an HTTP frontend. The documentation's example for sending a first message to a deployed agent is the following:

aws sqs send-message \
  --queue-url "$INPUT_QUEUE_URL" \
  --message-group-id my-conversation \
  --message-deduplication-id "$(date +%s)" \
  --message-body '{
    "content": { "type": "text", "text": "Write a haiku about serverless agents" },
    "group_id": "my-conversation"
  }'

Replies are read from a separate output queue. The documentation notes that the Lambda runtime currently uses a fixed Claude Sonnet model through Bedrock, configured by a `MODEL_ID` setting, and it publishes no pricing estimates for running agents this way.

··········

INFINITY CODE AS A FIRST APPLICATION.

To show the runtime in a realistic workload, the project includes Infinity Code, a coding agent whose sessions run in a daemon so that the interface remains responsive even when the agent runs remotely. It runs builds and tests in the background with streamed output, edits code in parallel threads using stacked sandboxes, and presents each result as a diff for review. It can be extended with both RAP and MCP tools, and the documentation also describes background agents and a Slack bot built on the same foundation.

The project site illustrates the intended behavior with an example session in which the agent launched `cargo test --workspace`, continued editing release notes while the tests ran, waited 18 minutes without polling, then received a test failure event, fixed a failing refunds module and reran the tests. The example is a vendor demonstration rather than an independent evaluation, but it shows concretely what the event model is meant to enable.

··········

WHAT IS ESTABLISHED AND WHAT REMAINS OPEN.

The verifiable part of the release is substantial: the code is public under Apache 2.0, the execution model is documented down to the slice phases, deduplication and turn-buffer semantics, and the benchmark source is included in the repository. The density figures, roughly 100 KB per idle agent and 80,000 agents through 3.2 million completions on one CPU thread, are AWS's own measurements and should be read with their conditions in mind. They used in-memory stores and a simulated model, so they measure the runtime's overhead rather than end-to-end agent throughput, and AWS publishes no latency, cost-per-agent or real-model figures.

AWS is also explicit about the open research questions. Language models are trained mostly on synchronous tool traces, and it is not yet known how interleaved, asynchronous events affect their reasoning. Scheduling currently follows arrival order, with policies such as prioritizing user-facing threads still to be explored. Compaction for large agent trees is not yet studied, and the authenticity of events reaching internet-exposed agents remains unresolved within the stateless architecture.

Taken together, Infinity is best understood as a well-specified research runtime that moves the cost of agentic systems away from reserved waiting capacity and toward actual work, with a concrete serverless reference design on AWS. Whether models behave reliably when events arrive out of the familiar request-response rhythm, and whether the economics hold with real model calls at scale, are the questions that independent use will have to answer.

··········

FOLLOW US FOR MORE.

·····

DATA STUDIOS

·····

datastudios.org