Creating an AI Streamer That Remembers Previous Visits — Designing Memory and Multi-Streaming States

📝 Originally published (in Japanese) at forge.workstyle.tech.

I'm building an AI avatar stream that runs unattended. No human broadcaster — the avatar reacts to comments, makes small talk, and carries the show through to the sign-off. Of all the walls you have to get over to make that work, this post covers two problems that both come down to state.

The first is remembering viewers across streams. Whether the avatar can say "oh, you were here last time too" makes a surprisingly large difference to the experience.

The second is where to keep the state during a stream. The renderer (the headless browser that produces the video) crashes. And if the avatar redoes its greeting every time it crashes and reconnects, viewers see a stream that introduces itself once a minute.

Push on either one and you arrive at the same question: which state, at what granularity, held where? I'll go through them in order, and I'll be candid about both the design decisions and the bugs I hit along the way.

Whose memory is it?

Start with viewer memory. The first thing I had to decide was the unit at which memory is shared.

I went with per character. The reason: the entity a viewer forms a relationship with isn't the "show," it's the character. If that same character is hosting a different show and remembers you from before, that matches how it intuitively ought to feel.

In implementation terms, the scope for memory lookups is keyed by the character's identifier.

scope = broadcast-char:{characterId}

This scope design sets up a bug I hit later.

When to extract memories

I made the process that extracts "things worth remembering" from conversation a two-stage affair.

• Incremental extraction: on every conversational turn, update a short memory about that viewer
• Final extraction: when the conversation with that viewer ends (they disconnect), consolidate everything

My first attempt used only final extraction, and it failed. Streams get cut off. The renderer crashes, the server restarts, the viewer leaves without saying anything. A design that "processes everything when it's over" leaves you with nothing when the end never arrives.

Accumulate incrementally, tidy up when the end comes. And if the end never comes, the incremental part is still there. This instinct — don't assume there will be an ending — shows up again in exactly the same shape in the state design below.

Bug 1: batch extraction produces memories that belong to no one

In the first implementation, I passed the whole conversation log to the extraction step at once. The resulting memories came back with no viewer identifier attached, and got saved as memories shared across the character.

Which means what Alice said could get referenced in a conversation with Bob. On a live stream, the avatar starts mixing in someone else's business — that's real harm.

The fix was to run the extraction inside the utterance turn, in the context of the target viewer. "Who is this memory about" is carried as the execution context, not as an input to the extraction.

Data that belongs to someone — like a memory — should be created in that person's context from the start, not associated with them after the fact.

Bug 2: the guest identifier didn't include the scope

This was the cleanest bug of the bunch.

Viewers are anonymous guests with no registered account. To create an internal record for them, I generated a placeholder email address.

guest+{displayName}@guest.invalid

Email addresses carry a unique constraint. And that string contains no scope — nothing indicating which character's viewer this is.

Here's what happens.

"Taro" shows up on Character A's stream
  → guest+Taro@guest.invalid is created           ✓

"Taro" shows up on Character B's stream
  → tries to create guest+Taro@guest.invalid
  → unique constraint collision → error           ✗

It breaks the moment a same-named viewer appears in a different scope. Display names are whatever viewers choose, so collisions should have been the design assumption.

Worse, the error surfaced in an unhelpful way: "only viewer processing fails, and only on certain streams." It took me a while to trace it back to the cause.

The fix is just to include the scope in the identifier.

guest+{scope}+{displayName}@guest.invalid

The lesson: if data has a scope, its unique key must include that scope. It sounds obvious, but this is exactly the obvious thing that slips when you're generating placeholder or dummy values — because the real data (an email address) genuinely is globally unique, and the stand-in isn't.

Also: "same-named users in different scopes" will never, ever happen in your test data. It only showed up once real viewers arrived. Which is another way of saying unique constraints lie to you until real data arrives.

The broadcaster who says "nice to meet you" on every reconnect

Once viewer memory was working, the next thing that started standing out was state loss.

Baseline reality: in an AI avatar livestream, the renderer (a headless browser) crashes. The GPU host gets flaky, the browser crashes, the video encoding process dies. Recovery — reload and reconnect — was in there from the beginning.

And here's what that produced.

On every recovery, the avatar redoes its greeting: "Good evening! Let's get started."

From the viewer's side, that's a stream that starts introducing itself once a minute. A wonderfully ill-timed failure mode: the better the recovery logic works, the more obvious the symptom.

What went wrong

The cause was clear: the page owned the "have I greeted yet?" flag.

page loads
  → connects
  → hasn't greeted yet (the variable is at its initial value)
  → greets

State in the page's memory disappears on reload. Obviously. And yet that's where I'd put "what has happened during this stream."

For the same reason, all of this was being lost too:

• The conversation so far
• Which comments had already been answered
• How much time was left in the show

The memory-extraction failure above was "assume there's an ending and you're left with nothing" — this is its twin. This time it's "put something you can't afford to lose in a place that isn't designed to break."

The fix: put state in a server-side external store

I changed the approach. The page holds nothing.

[stream state]      Redis snapshot (one per stream run)
     ↑ updates
[dialogue server]   generates utterances, updates state
     ↓ WebSocket
[page]              display only. On connect, receives the current state and renders it

The page became a "display-only client." On connect it receives a snapshot and builds the screen from it. The page itself remembers nothing, and doesn't need to.

The greeting decision moved server-side too, naturally.

a connection arrives
  → look at the snapshot
  → greeted flag is set    → don't greet, resume from where we were
  → flag isn't set         → greet, set the flag

Now the page can crash and reconnect any number of times, and the greeting happens exactly once.

What goes in the snapshot

Put in too much and it gets heavy; put in too little and you get inconsistencies after recovery. Here's roughly what's in there:

• Show progress state (has it greeted, has it entered the sign-off)
• Recent conversation history (used to generate utterances)
• Identifiers of events already responded to
• A short memory per viewer

That last one — the short per-viewer memory — is where this connects back to the character-scoped memory from the first half. The canonical memory lives in the persistent store, but the slice needed during the stream rides along in the snapshot, so the avatar can still say "you were here last time too" after a recovery.

What's not in there: video frames, the audio itself, UI animation state. Those are all "just redo them after recovery" things, so there's no point holding them as state.

The test is: is this needed to resume from where we left off? Appearance can be rebuilt; context can't.

Verifying that the size doesn't grow forever

With this kind of "data that keeps being updated for the whole stream," the scary failure is unbounded growth. Naively appending the entire conversation history will eat all your memory on a long stream.

I ran a two-hour continuous test to check.

Because history is capped to the most recent entries, it stops at a ceiling. Decide at design time whether there's a bound, then confirm it by measurement. One without the other isn't enough — the design can have a bound while the implementation grows somewhere else.

Side benefits of moving state out

Pushing state into an external store produced a few advantages I hadn't planned on.

1. Recovery is much easier to test

Just manually reload the page and you can watch the recovery behavior. Before, I had to kill the process and wait for it to come back up.

2. You can watch mid-stream

Since display-only clients are unlimited, I can peek at the live stream state from another browser. Debugging got dramatically easier.

3. Renderers became disposable

This is the big one. Because the page holds no state, a brutally simple recovery works: if the GPU host is flaky, throw the whole thing away and re-acquire on a different host. With state living in the page, that option didn't exist.

On privacy

Remembering what a viewer said and bringing it up next time is good as an experience, but it needs care.

Here's where I've drawn the line:

• Only remember what the person themselves wrote in the public stream chat
• Retain a short summary, not the raw log of what they said
• Keep the scope closed to the character, and don't repurpose it for anything else

Even for remarks in a public setting, some people find being remembered uncomfortable. At minimum, being able to explain what is remembered is the builder's responsibility. Keeping only short summaries in the snapshot isn't just a size concern — it lines up with this line too.

Wrapping up

Viewer memory and stream state look like separate topics, but they converged on the same design principles.

• Viewer memory belongs at the character level (not per show, not per run)
• Extraction is two-stage: incremental + on-end. "Consolidate when it's over" alone leaves nothing when the end never comes
• Create memories in someone's context from the start. Associate them after the fact and they end up belonging to no one
• If data has a scope, put the scope in its unique key. Placeholder values are where this slips most. Design assuming display names collide
• Keep state in the client (the page) and every reload restarts from "first time" — and the better your recovery works, the more visible the symptom
• Put state in an external store and make the page a disposable, display-only client
• The test for what goes in the snapshot: is it needed to resume after recovery? Appearance can be rebuilt
• Confirm size bounds by measurement, not just by design (6.2KB plateau over two hours)
• Moving state out buys you testability, observability, and the freedom to throw individual instances away

The thread running through all of it: don't put important state in a place that breaks, or a place that assumes an ending. Sometimes reconsidering where state lives is faster than hardening your recovery logic, and reconsidering "whose context is this created in" is faster than trying to re-associate memories after the fact. Both are things I learned from bugs that only surfaced once real viewers showed up. Test data doesn't lie to you like that.