VideoEdit

Core pipeline

The v1 slice end to end: upload the CapCut export, PySceneDetect (content detection, threshold ~40) splits it into scenes, adjacent scenes get merged into logical sections in the UI, sections containing speech are marked dialogue, dish context (name/ingredients/steps/description) is entered once per video, Gemini proposes captions for every cooking section in one coherent pass, dialogue sections get transcribed, and everything is editable in the UI before exporting SRTs.

Dialogue sections get a hybrid transcription rather than a single Gemini or whisper pass: three forced-language faster-whisper passes for word-level timestamps (fixed at en/fr/ja regardless of the profile’s own output languages, purely to help Gemini tell code-switched speech apart), then Gemini reads the audio plus all three passes and produces one merged, code-switched transcript aligned to the whisper timings:

CapCut direct-write

The bigger payoff over plain SRT export: write_captions() (capcut_captions.py) writes styled primary (top, black-on-white per-line highlight) and secondary (bottom, white-with-shadow) text tracks straight into the CapCut draft linked to a project, so an SRT re-import and manual restyling per video isn’t needed at all. None of this is documented anywhere. Every rule below came from inspecting a real CAPCUT_PC/9.1.0-beta4 draft on disk and testing against a live install.

Profiles & multilingual

Originally a single global settings singleton (one style source, one set of prompts, a fixed fr-primary/en-secondary pair). Profiles replace that with a named preset picked once per project at creation time, so the same install can cover more than one language pair, style, or prompt setup. Useful for a second show, or just a different caption tone.

Balance Audio Levels

Voiceover clips can come out louder or quieter than their neighbors depending on how close to the mic a given take was. capcut_audio.py’s analyze()/apply() fix that directly inside the linked CapCut draft: measure each eligible segment’s integrated loudness, push it toward a target, write the gain into the segment’s existing volume field. No re-encoding, no touching the original media.

Flagged clips stay selectable, just default-unchecked (normal clips default-checked). Checking one is an explicit override: it writes the uncapped wanted gain instead of the near-useless capped one. Otherwise “selecting” a flagged clip would do nothing audible. Each clip gets an inline before/after audio preview (one <audio> element with a tab toggle swapping its src, rather than two full-width players, which ate too much space in a long clip list) so the fix can be heard before it’s committed.

CapCut position & wrap calibration

CapCut lets a wrapped caption’s extra lines overflow past its declared single-line box without moving the box’s own anchor. That means there’s no anchor-position signal to derive a wrapped caption’s true per-line growth from. Getting primary/secondary caption pairs to sit correctly regardless of how many lines either one wraps to needed a different source of truth than the box’s own reported position.

A caption being written picks its bucket’s literal transform_x/transform_y when the reference draft had a real example of it, and falls back to a formula (primary at its base position, secondary one gap below plus a tunable extra-gap fraction per extra line) only for a bucket the reference draft never actually demonstrated.

One manual step remains by design: this tool can’t drive CapCut itself, so “Create calibration project” still requires importing the placeholder video into CapCut once, dragging each of the 4 sections’ boxes (and style, since the same draft doubles as the style source) to taste, then picking that draft as the profile’s CapCut style source as usual.

Bulk CSV editing

Editing captions one field at a time in the UI is fine for a few corrections, but slow for a full-video pass or for handing captions to someone else (a spreadsheet, ChatGPT) to edit in bulk. captions_bulk.py exports/imports one CSV per section-kind (cooking/dialogue/custom), scoped to the editor card it came from.

Caption chat & history

The original “generate captions for all sections” button was one-shot: call it, captions land immediately, no memory of what was just proposed. A bad batch was expensive to walk back and impossible to steer mid-generation. It’s replaced by a global caption chat, a persistent, project-wide conversation that proposes captions for every non-locked cooking section per turn and never auto-applies, plus a general caption/transcript history system that every kind of write, manual edit, chat apply, translate, or CSV import, now feeds into.

Proposals from the global chat render as a batch review list, checkbox per section, current text against proposed, with one “Apply N selected” button, mirroring the Balance Audio Levels preview/apply pattern instead of per-section “Use this” buttons. Applied state tracks per section rather than for the whole batch, so undoing one applied section, or adding one you’d originally skipped, doesn’t mean redoing the rest.

Reverting a snapshot now records the value it’s about to overwrite first, under a dedicated “revert” source, so undoing a revert is just reverting again instead of having nothing to go back to. Every history preview, the per-line popover, the section history panel, the global-chat review list, renders as a word-level diff against the section’s current live value rather than plain text, so a preview answers “what would actually change right now,” not just “what did this used to say.”

Caption examples & video hook

Two related caption-quality additions: real-caption few-shot examples pulled from past CapCut projects, and a dedicated “hook” section kind for the video’s opening line.

Example projects (Profile.example_projects): pick CapCut drafts in Settings whose captions are already good, and every one of their real captions gets extracted, reusing the same top-two-text-tracks-by-position heuristic as style extraction, as a candidate few-shot example, individually flagged for whether it’s a recipe-step example, a hook example, both, or neither. Gemini gets whichever ones are flagged on as real past outputs to imitate the voice of, kept separate from the style prompt the same way per-language grammar notes already were.

Hook is a fourth SectionKind alongside cooking/dialogue/custom: the video’s opening line, attention-grabbing rather than an instruction, marked the same click-to-cycle way as the other three, with no cardinality enforcement. It reuses the exact caption editor and chat UI cooking sections use, CaptionEditor gained a kind prop rather than a separate HookEditor component, since the two are otherwise identical: same history buttons, same translate, same split-into-lines. A hook can still be a short sequence (a question, then a reveal), so chat_about_hook keeps the same single-or-split proposal schema as the per-section cooking chat, just with hook-specific wording and its own style prompt and keyframe/dish-context toggles. Hook sections are automatically excluded from the global caption chat, which already hard-filters to cooking sections, and get their own step between Captions and Transcripts.

CapCut configuration check

A read-only step that flags captions already written into the linked CapCut draft whose position and style disagree about which role, primary or secondary, they actually belong to. Built after a real incident: a caption copy-pasted from the primary track into the secondary track’s slot, text edited, but never restyled to match.

The check reads every text segment currently in the draft’s draft_info.json fresh off disk, not just what the app itself wrote, since the failure mode it’s built for happens through edits made directly in CapCut, outside the app’s own bookkeeping. Each segment gets classified two ways, independently: by position role (its transform closer to the primary or secondary preset’s own anchor, a coarser check than the real per-wrap-bucket positions the direct-write path uses, since telling primary from secondary apart doesn’t need bucket-level precision) and by style role (exact-field match count against both presets, with fields that legitimately vary per wrap bucket or that CapCut sets to its own factory defaults excluded, so they don’t dilute the match).