Kataribe
Voice AI that talks with Japanese seniors for a few minutes a day on Gemini Live, and turns their memories into a life story their family can read, hear and ask more about. Built in four weeks for Google Cloud's AI Builder Cup 2026, and kept as a static archive since judging ended.
Highlights
- Live through judgingAnyone could try it on their own life, in English, until Nov 10. The site now keeps the demo and its family page.
- 36 of 36 consent checksA story reaches the family only on the person's own words of agreement. A no, a hesitation or no reply keeps it private.
- Every quote checked 3 waysWord for word in the transcript, in the right script, and heard in the recording. A quote that fails one is dropped.
- Session drops tracedSessions dying at two minutes were the client's own 20 s ping timeout, not the server. With it off, one ran 391 s.
Overview
A voice AI that talks with an elderly person in Japanese for a few minutes a day, and turns what they remember into a life story their family can read and hear. A kataribe (語り部) is a traditional keeper of oral history, the person who remembers the stories and passes them on.
The voice is only the channel; the product is the loop. Each conversation becomes stories, people and places on a life timeline, the gaps in that timeline plan the next conversation, and relatives can send questions the interviewer asks next time. Nothing reaches the family until the person says it can.
I built it in four weeks for Google Cloud’s AI Builder Cup 2026, and wrote all of the code myself; my wife helped with the Japanese and with advice along the way. It was submitted with a live site where judges could try it on their own life, the demo video above and a pitch deck. It did not make the top 50, so the service is shut down, and kataribe.kaseovo.com keeps the demo and its family page as a static archive.
Week 1
Sep 22-28
The whole loop
- Japanese voice in the browser, and the live eval harnessSep 22
- Conversations become stories; session two recalls session oneSep 23
- Speaks first, asks before sharing, the family asks backSep 25
Weeks 2-3
Sep 29-Oct 12
Deployed
- Pitch deck content and demo shot listSep 29
- Cloud Run in Tokyo, with Firestore and Cloud StorageOct 10
- Quotes and consent checked against the recordingOct 11
Week 4
Oct 13-18
Ready to judge
- Start page: judges try it on their own lifeOct 13
- kataribe.kaseovo.com, the family page as a bookOct 14
- Demo video, deck and release evalOct 18
- SubmittedOct 18
Judging
Oct 19-Nov 10
Judged, then archived
- Live site kept up, checked every morning
- Code frozen at the submitted commit
- Not in the top 50
- Shut down, kept as a static archiveNov 10
Problem
- Remembering takes time. Someone reaching for a sixty-year-old memory pauses for seconds. The Live API ends a turn after 800 ms of silence by default, so the interviewer would talk over the memory it asked for.
- A memoir cannot hold a plausible guess. The model hears Shigeru and writes 佐藤繁 when the man is 佐藤茂. Same sound, wrong grandfather, in the one document a family keeps.
- Sharing is the senior’s call. Nothing reaches a relative without her agreement, and a relative must not be able to tell what she declined.
- The bugs that matter only show up live. Configs that passed every offline check died on a live socket, and the 5-10 minute sessions the product needs are longer than any quick test.
- Judges arrive cold, in English. The product is Japanese and built for one senior over many sessions. A judge speaks English, talks for two minutes, and must never hear another judge’s childhood.
Approach
Built as vertical slices: each one ends with something that runs and can be shown, never a half-wired layer waiting on the next.
One rule runs through the code: the AI never speaks for the person. A quote is stored only if it is found word for word in the transcript and heard in the recording, consent counts only as her own words of agreement, and facts we already know (her name, birthplace and family) are handed to the model rather than transcribed.
Behaviour is measured against the live API over several runs and reported as n of n, because a single good run proves little. French and English exist so the whole loop can be exercised without a Japanese speaker in the room, and English is also what judges use, but only a Japanese session decides whether the product works.
What I built
Japanese voice in the browserDoneLive Japanese voice in the browser, on a token that locks the interviewer's prompt and tuning.
GoalRun a real-time Japanese voice conversation in the browser on Gemini Live, without letting the browser decide who the interviewer is or how long it waits.

The backend mints a single-use ephemeral token with the whole session config locked in: prompt, voice, pause length and transcription. The browser streams 16 kHz microphone audio straight to Gemini with it and cannot rewrite any of those fields. Each session is saved as a stereo WAV (senior left, interviewer right), and a saved one can be replayed in place of the microphone to compare two prompts on the same words.
How long the interviewer waits through silence, measured against synthesized speech:
| Patience | Pause | Result |
|---|---|---|
| 3500 ms | 1.5 s | waited |
| 3500 ms | 2.5 s | cut in |
| 6000 ms | 2.5 s | waited |
| 6000 ms | 4.0 s | waited |
The gap the model sees is longer than the pause itself, so the default is 5000 ms.
Testing against the live APIDoneA live suite streams real audio at the API; the release run passed 15 of 15 browser tests and a six-minute session.
GoalTest what only appears on a live socket: transcription of names, waiting through a pause, staying in the right language, and sessions as long as the product needs.
Two layers, split by cost:
make check, on every push: lint, types, config shape, endpoint validation and the resampler maths. Offline and fast.make eval, by hand before a release and after any change to how the interviewer speaks or listens: synthesized Japanese streamed at the real API in real time, through the same token path the browser uses. It checks the transcript, the patience setting and the reply’s rules, graded partly by code and partly by a model judging against the prompt itself.
make spar goes further and puts the interviewer in a live conversation
with an AI playing the person being interviewed. Both sides are real audio,
so pauses and turn-taking are genuine. Each persona presses one rule:
| Persona | Behaviour | The interviewer must |
|---|---|---|
haru | Pauses two or three seconds mid-memory | not fill the silence |
shigeru | Refuses to discuss the war | drop it |
kimiko | Drifts to another subject | follow rather than steer back |
The release run, before submitting:
| Suite | Result |
|---|---|
| Browser tests, a judge’s whole path included | 15 of 15 |
| Six-minute session in a real browser | no drop |
| Live API checks | 38 of 40, both failures traced and fixed, then 6 of 6 |
Sounding like a conversationDoneThe interviewer greets first, and 3 of 6 sparring turns now react without asking anything.
GoalHave the interviewer open the conversation itself, and stop answering every turn with the same acknowledgement followed by a question.
A sparring run showed exactly what a tester had reported: stiff and relentlessly interrogative. The prompt said to keep acknowledgements short, but never said a turn could be only an acknowledgement, so it always asked.
| Before | After |
|---|---|
| そうでしたか。…もう少しお聞かせいただけますか。 | それは立派な柿ですね。 |
| そうですか。…どんなお料理にされるのですか。 | まあ、それは愛らしいですね。 |
| Same acknowledgement, then a question, every turn | 6 turns, 3 with no question, acknowledgements varied |
Three rules went into the prompt in all three languages: a turn may end without a question, an acknowledgement is never reused, and a reaction says what the story did to you rather than repeating it. When a session opens, a short stage direction (“they have just sat down, speak to them gently”) makes the interviewer greet first, so nobody has to work out that they should start.
From conversation to life storyDoneEvery quote is matched to the transcript and heard in the recording; session two recalls session one.
GoalTurn each finished conversation into stories on a life timeline, find what is missing, and open the next session from it, without the model putting words in the person's mouth.
Finished session
stereo recording + both transcripts
Stories
title, life stage, people, places, her quotes
Gaps
thin life stages, people never described
Avoid list
topics she declined, never raised again
Next session prompt
story so far, planned questions, what to avoid
The model is asked for the person’s own words, and every quote then passes three checks before it is stored:
- Word for word in the transcript, ignoring only spacing and punctuation.
- Written in her language’s script. A Korean particle inside a Japanese quote once passed the first check, because the transcript had it too.
- Heard in the recording. Gemini Flash listens to the clip the family would hear and confirms the words. It caught the Live API writing 私は as 私を, unprompted, in a browser test.
A quote that fails any of them is dropped rather than kept: a memoir is the one place a plausible paraphrase must not end up. Stories land on six life stages, from childhood to later life, and the next session opens with a word about the last one.
Consent before sharingDoneA story is shared only on her own words of agreement, found in the transcript: 36 of 36 runs.
GoalLet a story reach the family only when the person agreed to share it, and keep every other story off the family page entirely.
The interviewer asks once, at the close: it names today’s stories in a few
words and asks whether the family may read them, and whether one should stay
hers. Extraction returns the words she used to agree, and verify checks them
against the transcript and the recording exactly as it checks quotes. No
words, or words that are not there, and the story stays private.
Twelve cases, measured against the real model, three runs each:
| What she said | Result |
|---|---|
| A clear yes | shared |
| A no, or a hesitation: 「うーん、そうですねえ……どうしましょうかねえ」 | private, never asked again |
| Yes to one story, no to another, in one sentence | each one as she said |
| 「あの話は家族に見せないで」, about a story from last time | taken back |
| No reply, or never asked | private, asked again at the next close |
| All twelve cases | 36 of 36 |
Private stories are filtered out in the query, so there is no route by which one arrives and is merely not rendered. A browser test checks that not a word of one reaches the page, and the page does not even show a count of them.

The family asks backDoneA relative's question opens the next session by name, and her answer links back if she shared it.
GoalLet a relative send a question the interviewer asks next time, saying who it is from, and bring the answer back to them without revealing anything she kept private.

The opening of the next session, with the family question in it:
こんにちは、田中ハルさん。前回伺ったお父様の織物の工場のお話を、今もよく覚えています。今日はお孫さんのゆかさんが、結婚式のときのお着物のことを知りたがっていらっしゃいましたよ。どんなお着物でしたか?
The rules around it:
- Who is asking is the point. The interviewer always says who the question is from, so the form asks for the relation as well as the name (孫のゆか).
- At most two per session, oldest first. It is her story, not a queue.
- Nothing she refused is asked, even by family. Each question is screened against the avoid list before it is stored, and again whenever she refuses something new. A held question looks exactly like one still waiting.
- The answer is linked, not copied. If she kept it private, the question still reads as waiting, the same as if she declined.
Against the real models: screening 3 of 3, reporting which questions were answered 9 of 9.
Try it on your own lifeDoneA judge talks about their own life in English and reads it on their own family page in about three minutes.
GoalLet a judge who speaks no Japanese run the whole loop on their own life in a few minutes: talk, decide what to share, and read the result, without ever meeting another judge's stories.

What the start page does for a stranger:
- A persona of their own. Memory is per person, so with one shared persona the second judge would be greeted with the first judge’s childhood. Try it asks for a name and makes a visitor whose link is 128 random bits. Talk again, and it remembers.
- A path in the corner. Watch the demo, read its family page, see a family question, talk, read your own page. Each step ticks itself off, and Go takes you there from any page.
- A close that fits two minutes. At two minutes the interviewer asks whether to go on. When they stop, it asks one thing, alone in its turn: may the family read today’s story. Keep it private on screen beats a spoken yes; Share never beats a spoken no.
- A read-only example. The demo’s family page shows the question form, but nothing is sent from it, so judges never read each other’s questions.
- A cap. 30 visitor conversations a day at about $0.10 each, so the worst case over the whole evaluation stays under $60.
Shipped and submittedDoneSubmitted with a live Cloud Run URL, a 3:48 demo video and a pitch deck, and kept up through judging.
GoalMeet every submission rule of the AI Builder Cup by 2026-10-18, and keep the live site working for every judge until judging ends on 2026-11-06.
| The rule asks for | What we submitted |
|---|---|
| A live URL on Google Cloud | kataribe.kaseovo.com, on Cloud Run in Tokyo with Firestore and Cloud Storage |
| A public 3-4 minute video showing it working | 3:48 on YouTube: two Japanese conversations with English subtitles, the family page they became, and my own turn on the live site |
| A PDF deck with architecture and business case | 15 slides, built by a python-pptx script on the contest’s template, so the wording is versioned with the code |
| A public repository, in English | kaseovo/kataribe, the submitted commit tagged; Japanese prompts carry English glosses |
Six of the deck’s slides:
Kept running through the evaluation:
- One instance stayed warm, since a cold start took 4.6 s.
- Watched by an uptime check, error emails and a budget alert, plus
make smokeevery morning: a judge’s whole path on the live service, because a health check cannot see a Gemini key that stopped working. - Frozen. The submitted commit was what ran; only an outage got a fix.
Starting a session
- Chose
- A big Talk button
- Instead of
- On-device presence detection
- Why
- Lighting and angle decide whether face detection works on camera, and a judge watching someone talk to a device sees a companion app. The family loop is what makes this different.
The demo grandmother
- Chose
- A made-up person, voiced by Gemini text-to-speech
- Instead of
- A live take with someone playing her
- Why
- The interviewer asks what an invented person cannot answer. Her family page is still the real pipeline run on both conversations, and the video says she is made up.
Working language
- Chose
- Keep testing in French and English
- Instead of
- Wait for a native Japanese review
- Why
- They exercise most of the behaviour and do not block on a speaker being free. What only Japanese can show stays unjudged until the review happens.
Shut down, kept as an archiveDoneA static copy on S3 and CloudFront for cents a month; the Google Cloud project and its Gemini key deleted.
GoalStop paying for Google Cloud and Gemini once judging was over, and keep the page that shows what Kataribe does, with nothing left to talk to.
The pages read everything through the API, so the archive keeps the API’s answers rather than the API. Before the shutdown, the demo’s family page, its English and her seven quote clips were saved from the live service into the repository. A CloudFront Function routes each path to them, and one build flag takes out every way to talk.
| Path | Until the shutdown | In the archive |
|---|---|---|
/, /family/{id} | the React app, from Cloud Run | the same app, built with VITE_ARCHIVE=1 |
/api/family/{id} | Firestore, only the stories she shared | {id}.json, saved before the shutdown |
A quote’s /audio | her turn, cut from the recording in Cloud Storage | audio.wav, saved the same way |
/talk/{id}, /console | a conversation, the team’s console | back to the start page |
/api/family/{id} and /api/family/{id}/translations cannot both be files on
disk, so the function adds .json to each API path. Then the Gemini key, the
budget, billing and the project went, and every visitor’s recording with
them, as the start page promised.
Tech stack
What each piece of the stack is actually doing, and where to look for more:
Languages
- Python
The backend, the processing pipeline, the live test harness and the script that builds the pitch deck, on 3.13 with uv.
- TypeScript
The frontend, which talks to Gemini Live directly through @google/genai on the token the backend minted.
Frameworks
AI/ML
- Gemini
gemini-3.8-live for the voice, with native audio and built-in transcription. gemini-3.8-flash with structured output for extraction, screening, planning, judging and listening back to quotes.
Tooling
- pytest
Offline tests on every push, plus a live marker for tests that stream real audio at the API.
- Playwright
Browser tests with the microphone replaced by a recorded sample: a judge's whole path, and a six-minute endurance run.
Topics
- Google Cloud
Through judging: Cloud Run in Tokyo serving the whole app from one image, Firestore and Cloud Storage behind the storage port, Cloud Build for deploys, and uptime checks, Error Reporting and a budget alert to watch it.
- AWS
The archive since the shutdown: S3 behind CloudFront, with a CloudFront Function routing each page and API path to the files saved from the service.
- SQLite
The store in development, with recordings on local disk. The same port runs on Firestore and Cloud Storage once deployed.
- GitHub Actions
Lint, types and offline tests on every push. The live evals run by hand before a release, since they run in real time and cost Gemini credit.
Future improvements
- Native Japanese review. The demo was voiced by AI and judges talked in English, so prompt changes have gone unreviewed in the language that ships. Whether a light restatement like 〜なんですね reads as warm aizuchi or as parroting is a native speaker’s call.
- Names heard only aloud. A name outside the profile can still be misheard, and once even the person’s own name, after they introduced themselves. The interviewer should hold to the profile’s name over what it hears, and a relative could fix a spelling in one tap.
- Exact audio for each quote. The Live API accepts
word_timestampand returns no word offsets, so a quote’s clip runs from one interviewer turn to the next. A transcription pass over the stored recording would make it exact. - Presence detection. The tablet noticing someone sitting nearby and inviting them, instead of a button.
- Sharing beyond the family. The start page already marks it Next: the teller choosing a friend, a care worker or a local archive as readers.
- Notifications. Relatives have to come back to see whether an answer landed.





