A Thousand Sandboxes That Agree on What Random Means

In May 2024, JFrog reported a vulnerability in Vanna, a Python library that turns a question into SQL and then draws you a chart. The chart is the interesting part. Vanna asks the model to write the plotting code, and then it runs it. So if you can get your text into the question, you can get your Python into the interpreter.

The NVD entry is worth reading slowly, because of one phrase in it: passing external input to the library’s ask method with visualize set to True, which it says is the default behaviour, leads to remote code execution. Not a misconfiguration. Not a debug flag somebody left on. The documented, out-of-the-box way to use the thing.

That’s a small library and an old bug, and it would be easy to file under “well, don’t do that”. Except that the shape of it is now the shape of the entire industry. Your agent writes code. Your agent runs the code. Nobody read the code in between.

Nobody reviewed the code your agent just ran

Say your agent is debugging a data pipeline. It fetches a GitHub issue to see what the reporter said. The issue body contains, somewhere below the fold, a paragraph addressed to the model rather than to you. The model writes a Python snippet. The snippet runs, inside your sandbox, with whatever the sandbox happens to hold.

That is three ordinary steps and one supply of attacker-controlled text. Every part of it is working exactly as designed. This is the point Greshake and colleagues made when they named indirect prompt injection: the attacker doesn’t need an interface to your system, only a document your model is likely to read, and they showed that processing a retrieved prompt can act as arbitrary code execution.

The obvious reply is that the model should just not do that, and the measurements are not encouraging. AgentDojo put 97 realistic tasks and 629 security test cases in front of tool-using agents and found that existing injection attacks break some security properties but not all, which is the polite way of saying some of them work.¹¹The same paper’s less quoted finding is that the models also fail plenty of the tasks with nobody attacking them at all. Both halves matter here: you are defending a component that is neither reliably safe nor reliably correct.

I’ve written about why agent runs fail, and that post is about whether a step succeeds. This one is about a step that succeeds perfectly at the wrong thing. A timeout costs you a run. An injected os.environ dump costs you whatever was in the environment.

So the question is not whether the model can be tricked. It is what the trick reaches when it lands, which is entirely a property of the box you put the interpreter in.

Four rungs, and what each one actually buys

There are roughly four places to draw that boundary, and the honest comparison is not “which is most secure” but “what is the thing a break-out has to get through”.

The bottom rung is a plain process with a syscall filter. Here’s the number that ruins it, from the Firecracker team’s NSDI 2020 paper: a trivial Linux application needs 15 unique syscalls, while a typical Ubuntu 15.04 installation needed 224 syscalls and 52 unique ioctls to run without problems. Your allowlist sits somewhere in that gap. Where it sits is decided by what broke last week, which makes it a compatibility negotiation wearing a security decision’s clothes.

Containers add namespaces and cgroups on top, and I want to be careful here because containerised dev boxes were my job for a while. At 6sense I built the Podman-based environments our data engineers worked in, and they were genuinely good at the thing they were for: pinning a filesystem, capping memory, keeping one person’s Spark job from eating the host. What they were not was a boundary against code that is actively trying to leave, because the kernel on the other side of it is the host’s kernel.

Two runc escapes make that concrete. CVE-2019-5736 let a container overwrite the host runc binary through /proc/self/exe and take host root. Five years later, CVE-2024-21626 leaked a file descriptor so that a malicious image could get a process started with its working directory in the host filesystem. Both are patched. Both are 8.6 on the CVSS scale. Neither is the last one.

gVisor sits in the middle and is the most interesting rung, because it is explicit about its own trade. The Sentry reimplements the Linux system call surface in Go, and gVisor’s security documentation commits to the rule that makes it work: no system call is passed through directly to the host, and the Sentry itself is not permitted to open new files or create new sockets. The performance guide is equally blunt about what that costs. CPU-bound work pays nothing, because instructions execute natively. Syscall-bound work pays, and pays in proportion to how chatty it is.

The top rung is a microVM, and the argument for it in the Firecracker paper is one sentence long: virtualisation moves the security-critical interface from the OS boundary to a boundary supported in hardware and comparatively simpler software. The guest kernel gets to be fully featured precisely because it is treated as untrusted. And then Firecracker’s jailer wraps the VMM process itself in a seccomp-bpf profile of 24 allowed syscalls and 30 ioctls.

Put those two numbers next to each other: 224 syscalls for a Linux userland to be happy, 24 for the process that is emulating one.

If your agent writes arbitrary Python and runs it, that top rung is the one you want, and everything up to here is uncontroversial. The trouble starts when you multiply it by how often you need it.

Five seconds of nothing, forty times a run

Firecracker publishes a specification that it enforces in CI, and the headline figure is that it takes 125 ms or less to go from the InstanceStart API call to the guest’s /sbin/init.

Hold that number carefully, because it is a floor. /sbin/init is not a Python interpreter with your libraries imported. Everything your tool call actually needs happens after the 125 ms.

Now count tool calls. An agent debugging that pipeline will make forty in a run without anyone noticing: read a file, run a snippet, read the traceback, run a smaller snippet. Forty fresh microVMs at 125 ms is five seconds of pure boot, per run, before a single line of your code executes.

Nobody ships that. And the clearest evidence that nobody ships that is the Firecracker paper itself, describing Lambda: while the 125 ms boot times are fast, they write, they are not fast enough for the scale-up path, which is sometimes blocking user requests. AWS’s answer at the time was to keep a pool of pre-booted microVMs sitting around.

A pool works when you have Lambda’s scale to amortise it. For a per-tool-call sandbox with a tight isolation boundary, the pool is the wrong shape: you want a fresh box each time, and a pre-booted pool of fresh boxes is just paying the 125 ms earlier and in bulk.

So you do the thing everybody does. You boot once, get the interpreter warm and the libraries imported, and take a snapshot.

Boot once, restore a thousand times

The mechanism is genuinely elegant. When Firecracker loads a snapshot it doesn’t read the memory file into RAM. It makes a MAP_PRIVATE mapping of it, so pages arrive on demand as the guest touches them and writes go to copy-on-write anonymous memory. Starting a sandbox becomes mapping a file and resuming some vCPUs.

It is not free, and the honest accounting is worth a paragraph. Ustiugov and colleagues measured Firecracker-based snapshotting and found that a function started from a snapshot took 95% longer to execute, on average, than the same function memory-resident, because its state comes back from disk one page fault at a time. Their prefetching work cut cold-start delays 3.7x from that baseline. So restoring doesn’t delete the cost of starting. It moves it out of a boot sequence and into the first second of your code running, where it is much harder to see on a dashboard.

Fine. Slide the boot budget down in that figure and watch the run close anyway. The arithmetic works, the isolation boundary is still hardware, and you now have a per-tool-call sandbox you can actually afford.

You have also just made a thousand sandboxes that are identical in every respect, including the ones you were relying on to be different.

A thousand sandboxes that agree on what random means

Firecracker documents this itself, in a section of the snapshot docs headed Snapshot security and uniqueness, and it is the most quotable paragraph in the whole area. When a guest’s state is resumed from more than once, it says, “guest information assumed to be unique may in fact not be”, and the list it gives is: identifiers, random numbers and random number seeds, the guest OS entropy pool, and cryptographic tokens.

Then it gets blunter. Their third worked example is the one that describes what you have built: snapshot S, loaded into microVM B, loaded into microVM C, and so on down the page. The verdict is flat. In this example, they write, “we consider microVMs B and C insecure”.

The consequence is not abstract, and it is not about getrandom. Think about what your sandbox image has already done by the time you snapshot it. It has imported your libraries. It may hold an OpenSSL context, a warm HTTP session, a cached client credential, a boot_id, a machine ID. Every one of those is bytes in a memory image that you are about to stamp out a thousand copies of.

Here is the sentence from the same docs that people miss, the one that says which half of the problem the mitigations don’t touch: state other than the guest kernel entropy pool, such as unique identifiers, cached random numbers and cryptographic tokens, will still be replicated across multiple microVMs resumed from the same snapshot.

That emphasis is theirs, not mine.

Which invites the fair objection, because Firecracker does ship a fix for exactly this, and the fix is good.

What reseeding covers, and what it doesn’t

The mitigation is real and worth knowing. Firecracker implements VMGenID, a virtual device that exposes a cryptographically random 16-byte identifier and changes it whenever the VM shifts in time, which includes every resume. Linux has used it since 5.18 to reseed its internal CSPRNG, and the kernel comment is charmingly direct about the reasoning: the ID is unique but not secret, so the kernel doesn’t credit it as entropy, it just forces an immediate reseed. Firecracker always enables the device.

So getrandom() and /dev/urandom do diverge across clones. That is the good news, and it is most of what people mean when they say the uniqueness problem is solved.

It isn’t solved, for two reasons that the same docs state plainly.

The first is a race. Reseeding happens when the guest kernel handles the VMGenID notification, which leaves a window between the vCPUs resuming and the CSPRNG actually being reseeded. Firecracker’s entropy-for-clones doc recommends closing it by hand, with RNDADDENTROPY and RNDRESEEDCRNG ioctls before any of your code runs, on every kernel version including the ones with VMGenID.

The second is bigger, and it is the one that bites you. The kernel is not where most programs get their randomness. AWS puts it about as clearly as anyone could in the Lambda SnapStart docs: the internal state of every random number generator is preserved exactly as it was at snapshot time, along with any random bytes your application or its dependencies have already generated and stored in memory buffers. AWS reseeds the kernel RNG on restore, tells you to use a CSPRNG, and ships an open-source scanning tool to find Java code that assumes uniqueness. You don’t build a scanner for a problem you’ve fixed.

Now the smallest true thing in this whole post, which is four lines of CPython.

tempfile generates its eight-character random names from a cached generator, and it guards that generator against being shared, like this:

@property
def rng(self):
    cur_pid = _os.getpid()
    if cur_pid != getattr(self, '_rng_pid', None):
        self._rng = _Random()
        self._rng_pid = cur_pid
    return self._rng

That check is correct and careful. It is a fork guard: the PID is the one thing that reliably changes when a process splits in two, and without it a parent and child would carry the same Mersenne Twister state and pick the same “unpredictable” temp filenames, which is a classic local vulnerability.

A snapshot restore is not a fork. The PID comes back identical, because the PID was in the memory image. So the guard passes, _rng is reused, and every clone walks the same sequence of temp names in the same order, straight out of CPython 3.13.²²mkstemp still opens with O_EXCL, so two files can’t collide inside one filesystem. That isn’t the property being lost. The one being lost is what the docstring promises: that the names are unpredictable.

The random module has the same shape one level up: CPython builds a single module-level Random() instance at import and seeds it once. Import it before the snapshot and you have baked its state into the image. And /proc/sys/kernel/random/boot_id is written at boot and read-only afterwards, so Firecracker’s docs note that every clone implicitly reads the same value, and suggest bind-mounting over the file if that matters to you. Their other suggestion is to delete /var/lib/systemd/random-seed before snapshotting, because systemd helpfully saved one for you.

Randomness is the sharp one because it’s invisible. The other two things a restore rewinds are at least loud.

The clock and the network come back wrong too

A snapshot freezes the guest’s network configuration along with everything else, and Firecracker’s network-for-clones guide names the problem in its first paragraph: each guest resumes with the same configuration, and most importantly the same IP address. The workaround is a separate network namespace per clone plus iptables NAT, and the doc opens with a caution block saying it is an example to get you started and they don’t claim it is performant or secure. That is an unusually honest thing for a vendor doc to say, and it should tell you how much of this is left to you.

The main snapshot doc adds that network and vsock packet loss can be expected on resumed guests, and that the state of network connections is not guaranteed to survive. So any connection your sandbox image was holding open comes back as a socket that believes it is connected and isn’t.

Clocks are the same story. Firecracker describes a paused microVM’s memory as frozen in time, and a resume starts it again from exactly there. That is why the VMClock device exists: it carries a vm_generation_counter that changes on every restore, so a userspace process can notice it has been moved through time and rebuild whatever state depended on the old answer. The catch is in the doc’s own important-note block. The poll() and generation-counter support landed in Linux v7.0, and using it on anything older means applying backports. Which is a polite way of saying that on the kernel you are probably running, your userspace will not be told.

One more, because it’s the detail I’d least like to find out about later. Firecracker verifies a snapshot’s VM state file with a 64-bit CRC, and the docs are explicit that this is only a partial measure against accidental corruption. The threat model trusts snapshot files. So if you are pulling snapshots from a registry over the network, which is the natural design, you are fetching an executable memory image of your sandbox, and authenticating and encrypting it is your job, not the VMM’s.

What I’d do about it

I run Rovers, where agents take voice instructions and act on real systems, so the blast radius question isn’t academic for me. It’s the thing I think about before I think about accuracy. Here’s the list I actually work from.

• Decide what the sandbox holds, before you decide what it is. A microVM containing a production database credential is not a sandbox, it’s a very well isolated attacker. The strongest control in this entire post is that the sandbox has nothing worth stealing, and it costs nothing to implement.
• Pick the rung on reachability, not on vibes. If the code can’t reach the network and can’t reach a token, a container is fine and cheap. If it can reach either, you are one kernel CVE from the host and the microVM is the answer.
• Do the boot arithmetic before you choose. Tool calls times start time is a number you can compute today. If it doesn’t close, you are going to snapshot, so plan for the snapshot now rather than discovering it in a sprint.
• Snapshot before identity, not after. Take it after the kernel has booted and the imports are warm, but before anything mints a token, opens a TLS session, seeds a PRNG or writes a machine ID. Everything in that image is public to every clone.
• Reseed on resume, in userspace, deliberately. VMGenID handles the kernel. You handle OpenSSL, random, your UUID generator, your temp-name sequence and anything that cached bytes at import. Write it as an explicit post-restore hook, and test it by diffing two clones.
• Give each clone its own netns and its own clock check. Same IP by default, stale connections by default, stale wallclock by default. All three are recoverable and none are automatic.

None of this is exotic. It’s the same discipline as running any fleet of cloned machine images, which we’ve been doing since AMIs existed. What’s new is only that the thing inside the image is now writing its own code, from instructions that may have come from a stranger, forty times a run.

The old question was how fast you can start a sandbox. The better one is how many of them are currently agreeing on a number they each believe only they know.

Sources

Research

• Agache, A. and others. Firecracker: Lightweight Virtualization for Serverless Applications, 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20), 2020.
• Greshake, K. and others. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, arXiv:2302.12173, 2023.
• Debenedetti, E. and others. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents, arXiv:2406.13352, 2024.
• Ustiugov, D. and others. Benchmarking, Analysis, and Optimization of Serverless Function Snapshots, arXiv:2101.09355, 2021.

Documentation and advisories

• Firecracker. Snapshotting support, project documentation.
• Firecracker. Entropy for clones, project documentation.
• Firecracker. Network connectivity for clones, project documentation.
• Firecracker. Specification, project documentation.
• gVisor. Security Model and Performance Guide, project documentation.
• Amazon Web Services. Handling uniqueness with Lambda SnapStart, AWS Lambda Developer Guide.
• Python Software Foundation. Lib/tempfile.py, CPython 3.13.0.
• NIST National Vulnerability Database. CVE-2024-5565, CVE-2019-5736 and CVE-2024-21626.