ProjectsVoice AI

Kataribe

Voice AI that talks with Japanese seniors for a few minutes a day on Gemini Live, and turns their memories into a life story their family can read, hear and ask more about. Built in four weeks for Google Cloud's AI Builder Cup 2026, and kept as a static archive since judging ended.

Highlights

Overview

The 3:48 demo: two Japanese conversations with English subtitles, the family page Kataribe wrote from them, and my own turn on the live site in English. The grandmother is made up and every voice is AI. Watch on YouTube ↗

A voice AI that talks with an elderly person in Japanese for a few minutes a day, and turns what they remember into a life story their family can read and hear. A kataribe (語り部) is a traditional keeper of oral history, the person who remembers the stories and passes them on.

The voice is only the channel; the product is the loop. Each conversation becomes stories, people and places on a life timeline, the gaps in that timeline plan the next conversation, and relatives can send questions the interviewer asks next time. Nothing reaches the family until the person says it can.

I built it in four weeks for Google Cloud’s AI Builder Cup 2026, and wrote all of the code myself; my wife helped with the Japanese and with advice along the way. It was submitted with a live site where judges could try it on their own life, the demo video above and a pitch deck. It did not make the top 50, so the service is shut down, and kataribe.kaseovo.com keeps the demo and its family page as a static archive.

  1. Week 1

    Sep 22-28

    The whole loop

    • Japanese voice in the browser, and the live eval harnessSep 22
    • Conversations become stories; session two recalls session oneSep 23
    • Speaks first, asks before sharing, the family asks backSep 25
  2. Weeks 2-3

    Sep 29-Oct 12

    Deployed

    • Pitch deck content and demo shot listSep 29
    • Cloud Run in Tokyo, with Firestore and Cloud StorageOct 10
    • Quotes and consent checked against the recordingOct 11
  3. Week 4

    Oct 13-18

    Ready to judge

    • Start page: judges try it on their own lifeOct 13
    • kataribe.kaseovo.com, the family page as a bookOct 14
    • Demo video, deck and release evalOct 18
    • SubmittedOct 18
  4. Judging

    Oct 19-Nov 10

    Judged, then archived

    • Live site kept up, checked every morning
    • Code frozen at the submitted commit
    • Not in the top 50
    • Shut down, kept as a static archiveNov 10

Problem

  • Remembering takes time. Someone reaching for a sixty-year-old memory pauses for seconds. The Live API ends a turn after 800 ms of silence by default, so the interviewer would talk over the memory it asked for.
  • A memoir cannot hold a plausible guess. The model hears Shigeru and writes 佐藤繁 when the man is 佐藤茂. Same sound, wrong grandfather, in the one document a family keeps.
  • Sharing is the senior’s call. Nothing reaches a relative without her agreement, and a relative must not be able to tell what she declined.
  • The bugs that matter only show up live. Configs that passed every offline check died on a live socket, and the 5-10 minute sessions the product needs are longer than any quick test.
  • Judges arrive cold, in English. The product is Japanese and built for one senior over many sessions. A judge speaks English, talks for two minutes, and must never hear another judge’s childhood.

Approach

Built as vertical slices: each one ends with something that runs and can be shown, never a half-wired layer waiting on the next.

One rule runs through the code: the AI never speaks for the person. A quote is stored only if it is found word for word in the transcript and heard in the recording, consent counts only as her own words of agreement, and facts we already know (her name, birthplace and family) are handed to the model rather than transcribed.

Behaviour is measured against the live API over several runs and reported as n of n, because a single good run proves little. French and English exist so the whole loop can be exercised without a Japanese speaker in the room, and English is also what judges use, but only a Japanese session decides whether the product works.

What I built

Japanese voice in the browserDoneLive Japanese voice in the browser, on a token that locks the interviewer's prompt and tuning.

GoalRun a real-time Japanese voice conversation in the browser on Gemini Live, without letting the browser decide who the interviewer is or how long it waits.

Interview tuning console: a senior picker, a voice picker set to Sulafat, a pause-before-replying slider at 5000 ms with measured guidance, end-of-speech sensitivity set to LOW, an input file picker for replaying a session, and a Start conversation button
The tuning console. Pause length and voice are chosen here, validated by the backend and locked into the session's token.

The backend mints a single-use ephemeral token with the whole session config locked in: prompt, voice, pause length and transcription. The browser streams 16 kHz microphone audio straight to Gemini with it and cannot rewrite any of those fields. Each session is saved as a stereo WAV (senior left, interviewer right), and a saved one can be replayed in place of the microphone to compare two prompts on the same words.

How long the interviewer waits through silence, measured against synthesized speech:

PatiencePauseResult
3500 ms1.5 swaited
3500 ms2.5 scut in
6000 ms2.5 swaited
6000 ms4.0 swaited

The gap the model sees is longer than the pause itself, so the default is 5000 ms.

Testing against the live APIDoneA live suite streams real audio at the API; the release run passed 15 of 15 browser tests and a six-minute session.

GoalTest what only appears on a live socket: transcription of names, waiting through a pause, staying in the right language, and sessions as long as the product needs.

Two layers, split by cost:

  • make check, on every push: lint, types, config shape, endpoint validation and the resampler maths. Offline and fast.
  • make eval, by hand before a release and after any change to how the interviewer speaks or listens: synthesized Japanese streamed at the real API in real time, through the same token path the browser uses. It checks the transcript, the patience setting and the reply’s rules, graded partly by code and partly by a model judging against the prompt itself.

make spar goes further and puts the interviewer in a live conversation with an AI playing the person being interviewed. Both sides are real audio, so pauses and turn-taking are genuine. Each persona presses one rule:

PersonaBehaviourThe interviewer must
haruPauses two or three seconds mid-memorynot fill the silence
shigeruRefuses to discuss the wardrop it
kimikoDrifts to another subjectfollow rather than steer back

The release run, before submitting:

SuiteResult
Browser tests, a judge’s whole path included15 of 15
Six-minute session in a real browserno drop
Live API checks38 of 40, both failures traced and fixed, then 6 of 6
Sounding like a conversationDoneThe interviewer greets first, and 3 of 6 sparring turns now react without asking anything.

GoalHave the interviewer open the conversation itself, and stop answering every turn with the same acknowledgement followed by a question.

A sparring run showed exactly what a tester had reported: stiff and relentlessly interrogative. The prompt said to keep acknowledgements short, but never said a turn could be only an acknowledgement, so it always asked.

BeforeAfter
そうでしたか。…もう少しお聞かせいただけますか。それは立派な柿ですね。
そうですか。…どんなお料理にされるのですか。まあ、それは愛らしいですね。
Same acknowledgement, then a question, every turn6 turns, 3 with no question, acknowledgements varied

Three rules went into the prompt in all three languages: a turn may end without a question, an acknowledgement is never reused, and a reaction says what the story did to you rather than repeating it. When a session opens, a short stage direction (“they have just sat down, speak to them gently”) makes the interviewer greet first, so nobody has to work out that they should start.

From conversation to life storyDoneEvery quote is matched to the transcript and heard in the recording; session two recalls session one.

GoalTurn each finished conversation into stories on a life timeline, find what is missing, and open the next session from it, without the model putting words in the person's mouth.

Finished session

stereo recording + both transcripts

Gemini Flash, structured output, then verified in code

Stories

title, life stage, people, places, her quotes

Gaps

thin life stages, people never described

Avoid list

topics she declined, never raised again

planned and written into the next token

Next session prompt

story so far, planned questions, what to avoid

The model is asked for the person’s own words, and every quote then passes three checks before it is stored:

  1. Word for word in the transcript, ignoring only spacing and punctuation.
  2. Written in her language’s script. A Korean particle inside a Japanese quote once passed the first check, because the transcript had it too.
  3. Heard in the recording. Gemini Flash listens to the clip the family would hear and confirms the words. It caught the Live API writing 私は as 私を, unprompted, in a browser test.

A quote that fails any of them is dropped rather than kept: a memoir is the one place a plausible paraphrase must not end up. Stories land on six life stages, from childhood to later life, and the next session opens with a word about the last one.

The family asks backDoneA relative's question opens the next session by name, and her answer links back if she shared it.

GoalLet a relative send a question the interviewer asks next time, saying who it is from, and bring the answer back to them without revealing anything she kept private.

Questions from the family tab: a question from granddaughter Yuka, Grandma, did you ever jump off the Shinbashi bridge?, stamped Answered with a link to the story, and a question from daughter Miho about her yukata at the Gujo Odori, stamped Waiting
Two questions from the demo's family. One is answered and links to the story it led to; the other waits for the next conversation.

The opening of the next session, with the family question in it:

こんにちは、田中ハルさん。前回伺ったお父様の織物の工場のお話を、今もよく覚えています。今日はお孫さんのゆかさんが、結婚式のときのお着物のことを知りたがっていらっしゃいましたよ。どんなお着物でしたか?

The rules around it:

  • Who is asking is the point. The interviewer always says who the question is from, so the form asks for the relation as well as the name (孫のゆか).
  • At most two per session, oldest first. It is her story, not a queue.
  • Nothing she refused is asked, even by family. Each question is screened against the avoid list before it is stored, and again whenever she refuses something new. A held question looks exactly like one still waiting.
  • The answer is linked, not copied. If she kept it private, the question still reads as waiting, the same as if she declined.

Against the real models: screening 3 of 3, reporting which questions were answered 9 of 9.

Try it on your own lifeDoneA judge talks about their own life in English and reads it on their own family page in about three minutes.

GoalLet a judge who speaks no Japanese run the whole loop on their own life in a few minutes: talk, decide what to share, and read the result, without ever meeting another judge's stories.

The Kataribe start page: the headline Every grandparent has stories nobody has asked about yet, buttons Watch it, See a life story and Try it on your own life, the embedded demo video, a Who it's for section with tabs For them, For their family and For their town marked Next, and a Your path through Kataribe card listing five steps with Go buttons
The start page as judges saw it, with the demo video and the judge's path in the corner.

What the start page does for a stranger:

  • A persona of their own. Memory is per person, so with one shared persona the second judge would be greeted with the first judge’s childhood. Try it asks for a name and makes a visitor whose link is 128 random bits. Talk again, and it remembers.
  • A path in the corner. Watch the demo, read its family page, see a family question, talk, read your own page. Each step ticks itself off, and Go takes you there from any page.
  • A close that fits two minutes. At two minutes the interviewer asks whether to go on. When they stop, it asks one thing, alone in its turn: may the family read today’s story. Keep it private on screen beats a spoken yes; Share never beats a spoken no.
  • A read-only example. The demo’s family page shows the question form, but nothing is sent from it, so judges never read each other’s questions.
  • A cap. 30 visitor conversations a day at about $0.10 each, so the worst case over the whole evaluation stays under $60.
Shipped and submittedDoneSubmitted with a live Cloud Run URL, a 3:48 demo video and a pitch deck, and kept up through judging.

GoalMeet every submission rule of the AI Builder Cup by 2026-10-18, and keep the live site working for every judge until judging ends on 2026-11-06.

The rule asks forWhat we submitted
A live URL on Google Cloudkataribe.kaseovo.com, on Cloud Run in Tokyo with Firestore and Cloud Storage
A public 3-4 minute video showing it working3:48 on YouTube: two Japanese conversations with English subtitles, the family page they became, and my own turn on the live site
A PDF deck with architecture and business case15 slides, built by a python-pptx script on the contest’s template, so the wording is versioned with the code
A public repository, in Englishkaseovo/kataribe, the submitted commit tagged; Japanese prompts carry English glosses

Six of the deck’s slides:

Kept running through the evaluation:

  • One instance stayed warm, since a cold start took 4.6 s.
  • Watched by an uptime check, error emails and a budget alert, plus make smoke every morning: a judge’s whole path on the live service, because a health check cannot see a Gemini key that stopped working.
  • Frozen. The submitted commit was what ran; only an outage got a fix.

Starting a session

Chose
A big Talk button
Instead of
On-device presence detection
Why
Lighting and angle decide whether face detection works on camera, and a judge watching someone talk to a device sees a companion app. The family loop is what makes this different.

The demo grandmother

Chose
A made-up person, voiced by Gemini text-to-speech
Instead of
A live take with someone playing her
Why
The interviewer asks what an invented person cannot answer. Her family page is still the real pipeline run on both conversations, and the video says she is made up.

Working language

Chose
Keep testing in French and English
Instead of
Wait for a native Japanese review
Why
They exercise most of the behaviour and do not block on a speaker being free. What only Japanese can show stays unjudged until the review happens.
Shut down, kept as an archiveDoneA static copy on S3 and CloudFront for cents a month; the Google Cloud project and its Gemini key deleted.

GoalStop paying for Google Cloud and Gemini once judging was over, and keep the page that shows what Kataribe does, with nothing left to talk to.

The pages read everything through the API, so the archive keeps the API’s answers rather than the API. Before the shutdown, the demo’s family page, its English and her seven quote clips were saved from the live service into the repository. A CloudFront Function routes each path to them, and one build flag takes out every way to talk.

PathUntil the shutdownIn the archive
/, /family/{id}the React app, from Cloud Runthe same app, built with VITE_ARCHIVE=1
/api/family/{id}Firestore, only the stories she shared{id}.json, saved before the shutdown
A quote’s /audioher turn, cut from the recording in Cloud Storageaudio.wav, saved the same way
/talk/{id}, /consolea conversation, the team’s consoleback to the start page

/api/family/{id} and /api/family/{id}/translations cannot both be files on disk, so the function adds .json to each API path. Then the Gemini key, the budget, billing and the project went, and every visitor’s recording with them, as the start page promised.

Tech stack

What each piece of the stack is actually doing, and where to look for more:

Languages

  • Python

    The backend, the processing pipeline, the live test harness and the script that builds the pitch deck, on 3.13 with uv.

  • TypeScript

    The frontend, which talks to Gemini Live directly through @google/genai on the token the backend minted.

Frameworks

  • FastAPI

    Seniors, visitors, sessions, recordings and the family view. It also mints the single-use token each conversation runs on.

  • React

    One Vite app for the start page, the tuning console and the family page. An audio worklet resamples the microphone to 16 kHz.

AI/ML

  • Gemini

    gemini-3.8-live for the voice, with native audio and built-in transcription. gemini-3.8-flash with structured output for extraction, screening, planning, judging and listening back to quotes.

Tooling

  • pytest

    Offline tests on every push, plus a live marker for tests that stream real audio at the API.

  • Playwright

    Browser tests with the microphone replaced by a recorded sample: a judge's whole path, and a six-minute endurance run.

Topics

  • Google Cloud

    Through judging: Cloud Run in Tokyo serving the whole app from one image, Firestore and Cloud Storage behind the storage port, Cloud Build for deploys, and uptime checks, Error Reporting and a budget alert to watch it.

  • AWS

    The archive since the shutdown: S3 behind CloudFront, with a CloudFront Function routing each page and API path to the files saved from the service.

  • SQLite

    The store in development, with recordings on local disk. The same port runs on Firestore and Cloud Storage once deployed.

  • GitHub Actions

    Lint, types and offline tests on every push. The live evals run by hand before a release, since they run in real time and cost Gemini credit.

Future improvements

  • Native Japanese review. The demo was voiced by AI and judges talked in English, so prompt changes have gone unreviewed in the language that ships. Whether a light restatement like 〜なんですね reads as warm aizuchi or as parroting is a native speaker’s call.
  • Names heard only aloud. A name outside the profile can still be misheard, and once even the person’s own name, after they introduced themselves. The interviewer should hold to the profile’s name over what it hears, and a relative could fix a spelling in one tap.
  • Exact audio for each quote. The Live API accepts word_timestamp and returns no word offsets, so a quote’s clip runs from one interviewer turn to the next. A transcription pass over the stored recording would make it exact.
  • Presence detection. The tablet noticing someone sitting nearby and inviting them, instead of a button.
  • Sharing beyond the family. The start page already marks it Next: the teller choosing a friend, a care worker or a local archive as readers.
  • Notifications. Relatives have to come back to see whether an answer landed.