Skip to main content
Open WebUI and Hindsight Behind Anvil Connect: Memory for Each User
ai-infrastructureidentityproductlocal-llmhomelab

Open WebUI and Hindsight Behind Anvil Connect: Memory for Each User

How I gave each Open WebUI user behind Anvil Connect their own Hindsight memory bank, role, rate limit and signed identity.

Sekou M. Doumbouya

Sekou M. Doumbouya

· 9 min read

Listen to this article

The views expressed here are my own and do not represent those of any current or former employer.

Part 4 of a six-part series on Anvil Connect.

On October 7, I typed a sentence no operator wants to type about their own hardware: “someone is using my system right now and I don’t know who it is”. I had built this for my wife and my friends. I had given them a way in, and I couldn’t tell who was using it.

Open WebUI was the thing I wanted them to use: a chat window, my models behind it, nothing to install. Connect got them signed in. Making the experience theirs, with their own memory, their own limits, and a system that knew who was on the other end, was the harder work.

A chat window was the easy part

Before Connect, getting them in was what I called “an incomplete integration story”, and even once they reached the page, their sessions lacked the right tools.

Open WebUI sign-in came first, on September 13, and it did not go quietly. On September 19, Connect’s access checks went on top of it, and the gateway decided who got in. Around the same time, new models started appearing for every user automatically, so when I added a model, it showed up in their list too. Plugins like web search came along for everyone.

Then came a rough edge: a broken page, produced when a cached copy of the app met a session that had already expired. Small bug. It was also the first sign of a pattern that ran through the whole fall. Sign-in is a moment. Being a user is a state, and states go stale.

Maturity came in levels

Looking back, I described what happened next as compounding levels of maturity into the product until it was useful. No single feature made it theirs. The levels stacked.

  1. Entitlement: which resources each person is connected to, and at what level.
  2. Permissions: their sessions get the right tools, not just a login.
  3. Their own memory: a Hindsight bank per person.
  4. A sandbox, so a chat session couldn’t accidentally damage the machine it runs on.
  5. Knowing the system is in use, without reading anyone’s sessions.
  6. Model health in Grafana.
  7. A rate limit, made possible by per-user tokens.

Entitlement is the foundation, and my definition is plain: a user connected to which resources, and at which level of access. Permissions make that definition true for every session. The sandbox exists for a reason I can state without embarrassment: I fenced their Open WebUI environment so they couldn’t accidentally destroy the AI workstation it runs on. None of this assumes malice.

The system has to know who it’s serving

The October 7 question led to signed identity forwarding. When Open WebUI sends a request to the router in front of my models, it now attaches a short-lived signed token naming the person, in a header called X-OpenWebUI-User-Jwt. The router checks the signature and rejects the request if the token is missing, duplicated or expired. Usage gets recorded against that person, not against Open WebUI as one big anonymous caller.

The difference from a raw header is not cosmetic. A raw header is a claim, and a claim is only as trustworthy as everything on the path that could have written it. A signed token is checked, not believed. It is not magic, either. Open WebUI and the router share the signing key, so the signature proves the request came from something holding that key, and the key is what I have to protect.

Six days later, on October 13, Connect took over each person’s Open WebUI role. An admin grant in Connect makes them an admin there, a member grant makes them an ordinary user, and no grant leaves them pending, which Open WebUI won’t let in. No admin power comes from a global group. Two places deciding who is an admin is just a disagreement that hasn’t happened yet. Now there’s one place to change it.

Memory that starts with the account

Hindsight is my memory service. I stood it up between September 27 and 29 from my own product requirements doc, then migrated my own memory into it from Codex, Claude, Pi, Hermes and OpenClaw. That migration was not graceful. It crawled through 238 chunks over many hours and lost one fragment along the way. One lost fragment is small, but it is a loss, and it belongs in the record instead of being rounded to zero.

The real milestone came on October 10: per-user banks. A Hindsight bank is now created with each user, and its key is set up in Open WebUI for them. Nobody enters a key. I wanted memory “offered as part of my services,” the same way a phone plan comes with voicemail instead of a setup guide for voicemail.

That’s the line between access and ownership. With a shared memory, a friend would be talking to a system that remembers me. With their own bank, it remembers them.

What broke on the way

Memory extraction calls timed out past 240 seconds. Extraction is how Hindsight turns a conversation into memory. For me, a timeout is a log line. For someone who doesn’t know what extraction is, it’s a system that quietly forgets.

A review found that model-name normalization could slip past a key’s model grant. A key allowed for one model could reach another through a differently spelled name. The grant looked right on paper. The check compared names, and names had more than one spelling. Authorization that runs on strings has to agree with itself about what the strings mean.

A fixed output cap gave way to a per-model limit. One number for every model was the lazy answer, and different models need different limits.

Then October 19. Chat was blocked with request accounting unavailable once the usage log hit its 64 megabyte cap. Accounting had become something chat depended on, and I had never said so out loud.

Hearing the server is not monitoring

The usage tracking came from a specific discomfort. I didn’t want to see their sessions. I wanted to know whether the system was being used. I would hear the server spinning up, but I couldn’t tell what was using it or what was doing what. Fan noise was my first observability stack. It had excellent uptime and no labels.

So the stats go to Grafana, and what I watch there is whether the models are performing well, not what anyone asked them. Per-user tokens make the next level possible: a rate limit, so a runaway can only hit so many sessions per minute. That protects them from a loop they didn’t mean to start, and it protects the box from all of us.

My friends don’t have to know what a memory bank is to have one.

Where it stops

Accounting is a dependency of chat now. When accounting fails, chat stops. That’s the cost of making usage real instead of best effort, and October 19 is what it looks like when the bill comes due. The system fails closed rather than serving requests it can’t account for, and that is not free.

The old accounting gaps were not backfilled. I could have gone back and filled them, and I chose not to. The usage history has a seam in it.

Per-user isolation lagged behind sign-in. Open WebUI sign-in came on September 13, but signed identity came only after my October 7 question, per-user banks on October 10 and roles from Connect October 13. For that stretch, people could get in before the system was fully ready to treat them as distinct people. The post on Hermes in a sandbox picks up where browser sign-in stops being enough.

Access is not ownership

Access lets someone use what you built. Ownership means the system is theirs: their memory, their limits, a system that knows who they are without reading what they say. Access is the easy part. If the people you build for can get in but the system can’t tell them apart, you haven’t shared your work with them yet. You’ve lent it.

0 claps

Enjoyed this? Tap to applaud, hold to really applaud.

Share this post
LinkedInXHacker NewsBluesky

Co-authored with AI, based on the author's working sessions, dictations, and notes.

Explore the source

fakoli/anvil-serving

This article discusses an open-source project. Star it, fork it, or open an issue.