Hermes in a Sandbox: Personal Agents Behind Anvil Connect
How I fenced each user's Hermes agent in an OpenShell sandbox behind Anvil Connect, kept its keys out of reach, and found where browser sign-in stops.
Sekou M. Doumbouya
Listen to this article
The views expressed here are my own and do not represent those of any current or former employer.
Part 5 of a six-part series on Anvil Connect.
On November 14, the front door I had been building since September would not open for an app. The Hermex iPhone app could not finish Connect’s browser sign-in. Connect’s whole model of identity is a browser: you sign in once through my own OIDC provider, and every published app accepts that session. A native app is not a browser, and the flow that made everything else feel like one sign-in had nowhere to land.
That edge only matters because of what sat behind the door by then. Each Hermes user now gets their own instance, and everyone but me gets it inside a sandbox, on the same machine my own work runs on.
An agent is not a chat window
Chat was the easy case. In the post on Open WebUI and Hindsight, each person’s environment was sandboxed so they could not accidentally destroy the AI workstation it runs on. A chat model answers. An agent acts.
Hermes is the personal agent I run. A personal agent is only useful if it can do things: call tools, hold keys, keep state. Every one of those is a path to the machine underneath it. When a chat goes wrong, the worst case is a bad answer. When an agent goes wrong on my AI workstation, the worst case is the workstation.
The groundwork happened without me in the room. Profile fencing for Hermes landed on September 27 and 28, and a native Hermes catalog landed on October 17 through 19. Background agents built both, with no session where I was driving.
One Hermes per person
On October 30 and 31, I moved Hermes off an older box and onto my AI workstation, behind Connect. The move came with a rule: a separate Hermes for every user. I get full access. Everyone else gets an OpenShell sandbox. OpenShell is the sandbox runtime I use to fence agents, and I validated the split with test users before calling it done.
The difference is the difference between handing someone your house keys and giving them a guest room with its own lock. The guest can make themselves at home. They cannot rewire the kitchen. The owner keeps the run of the house, including the right to break it.
The same week, a durable Pi went into OpenShell and showed up under Connect’s Agents. (Pi has its own post.) That is when the pattern stopped being about Hermes. OpenShell became the sandbox for every Connect user except me.
The native Hermes adapter, worked on from October 17 to November 7, runs Hermes inside a dedicated OpenShell instance. It also refuses one thing that sounds minor: it never replays a lost submit. If a request is lost in flight, the adapter does not quietly send it again. Retrying is a reflex for anyone who has built web services, and for a read it is harmless. For an agent, a retried request is a second action. A user who sees a failure can decide what to do. A user whose agent did the thing twice behind their back cannot.
The fence is not only for users. On November 10, Anvil’s validation of Pi and GLM had to run inside OpenShell, and 508 tests passed there. If the sandbox is where people live, the sandbox is where the tests have to pass.
The key it could read
On November 7, the real problem surfaced: Hermes read keys from its own profile file. Anything an agent can read, an agent can be talked into using or repeating, and a fence around the process does nothing if the secret sits inside the fence with it.
So the key had to move out of reach, to a tool service running under a separate OS account, one the Hermes profile cannot read. The agent asks the service to do the work. The service holds the key.
Anyone who has worked near finance knows this shape. The clerk who approves an invoice is not the clerk who cuts the check. Separation of duties is not a statement about anyone’s honesty. It is a statement about what one compromised seat can do. Here the seat is an OS account, and the agent’s seat cannot reach the secret.
I accepted Hermes as working on November 8.
Two failures on the way
Two things broke, and neither was exotic.
First, sign-in redirects came back as 502 errors from Cloudflare. In my setup Cloudflare is only the public front door, so a 502 there looks like everything is down when one route simply was never configured. The cause was a missing Connect rule. Connect routes only what it has been told to route. That is the point of a closed design, and this time it worked against me.
Second, a web UI in the path lacked an OIDC group claim. It knew who you were but not which group you belonged to.
Connect in an app
Before the edge showed up, I had already been pulling in the other direction. From October 12 to 17, I worked on an iOS app I described as “Anvil Connect in an app”: Home, Chats, Agents and Memory, one sign-in, and a Pi sandbox per user.
The idea is that the end user should not have to care how any of it works. In my words at the time, they “don’t need to know all these different details.” A person using it does not need to know that memory is Hindsight or which runtime fences their agent. They need a tab called Memory, a tab called Agents, and a sandbox of their own behind them.
A UX review of that work found 54 accessibility problems. That number is the honest measure of a first pass. It is a list, not a verdict, and a list is something you can work down.
Where the browser stops
Then November 14. The Hermex iPhone app could not complete Connect’s browser OIDC sign-in, so it got a separate password and a private Tailscale address.
Yes, Tailscale. The tool the origin post was about taking out of my family’s path is back in the picture. (I noticed.)
But not as the wall. Tailscale was never removed. It still sits beside Connect, and the Hermex app needs it. Browser users go through Connect. A native app that cannot carry a browser session gets a private address instead, beside Connect, not in front of it. Tailscale is excellent for exactly this: a known device, talking to a service I run, over a private network. It was a bad requirement for my wife. It is a good answer for a native client.
Connect’s identity model is a browser session through my own OIDC provider, with passkeys. That model is strong where it applies. It does not apply to every client.
What this does not solve
Hermex cannot do browser sign-in. It runs on a separate password and a private network address. That is a second identity path to maintain beside Connect, and two paths means two places for access to drift apart. One sign-in holds for browsers. It does not hold for everything.
OpenShell may be too heavy for a non-technical user. It works on my hardware because I run it. On November 18, thinking about a less technical Windows user setting this up, I called OpenShell “maybe a step too far” and leaned toward Docker plus an installer. A fence that someone has to learn before they can use the thing it protects is the old install tax under a new name.
Sandboxing lagged behind sign-in. Identity came first, and the fences behind it followed. For a stretch, the door was further along than the rooms it opened onto. That order is backwards. The fence should ship with the door, and that belongs in what I would change.
Fence by default, and draw the edge on purpose
The owner of a machine has the right to break it. Nobody else should be able to break it by accident, and nobody’s agent should be able to break it on their behalf. So fence by default for everyone but the owner, and keep the secrets outside the fence even for the agents you trust.
Then be honest about where your front door’s model of identity stops. Every identity system assumes something about the client, and somewhere that assumption runs out. Find that line before a user does, put the second door beside it, and write down which door is which.
Co-authored with AI, based on the author's working sessions, dictations, and notes.
Explore the source
fakoli/anvil-serving
This article discusses an open-source project. Star it, fork it, or open an issue.