A video recording of an hour and a half whose sound could not be heard. With Claude as technical guide it was repaired in six steps: sound back, a clearer picture, intelligible speech, and Dutch subtitles that were first made with speech recognition and then improved with the script.
Each step made a new file next to the previous one, so there was always a way back. The large video file stayed on the home PC; Claude worked in a cloud workspace and fetched only what each step needed through the link with the PC.
A 14.1 GB MP4 straight from the camera: 1080p picture at 24 frames per second, and a single MP3 audio track that could not be heard on playback. The sound was recorded with the camera's own microphone, at a distance. As a result speech, music and room noise are mixed in the same track, and voices are quiet compared with everything else.
Sound is a mixture of vibrations at different frequencies, measured in hertz (Hz): the number of vibrations per second. Low tones, such as a hum or a bass guitar, vibrate tens of times per second; high tones, such as an 's' or the hiss of noise, thousands of times. A spectrum shows how much energy sits in each part of that range.
To see where the voice is and where the noise is, a two-minute test clip of dialogue was measured at two kinds of moments: while someone is speaking, and in the pauses in between. What remains in the pauses is the background noise.
First ffprobe showed what is in the file: which tracks, which codec, how long, and whether the audio track actually contains a signal. Then FFmpeg repaired the track and put it back with the picture, without touching the picture itself. A cross-correlation between old and new sound showed that picture and sound line up to the millisecond.
Exposure and sharpness were improved with FFmpeg's classic filters and re-encoded with x264. Deliberately no generative AI: it invents detail, and faces must stay exactly as they are. After encoding every frame was compared with the original (PSNR) to check that no frame was dropped or shifted.
On a two-minute test clip three variants were first compared side by side, to listen before the whole video was processed: a speech filter (equaliser), the speech filter with an AI noise filter, and a full AI chain. Then a stronger version of the speech filter was added, and that became the choice. After that, dynamic normalisation (dynaudnorm) evens out quiet and loud passages, and a limiter prevents distortion.
An equaliser changes the balance between high and low, not the ratio of voice to noise at the same pitch. The AI noise filter did remove 15 to 20 dB more noise, as the chart below shows, but after listening the choice still fell on the stronger speech filter.
A voice detector (Silero VAD) cuts the sound into stretches with speech. Each stretch goes through Whisper large-v3, with Whisper turbo as a second opinion, via sherpa-onnx on the cloud workspace's processor. The raw text then gets a clean-up pass: repetitions and invented sentences removed, lines of at most 42 characters, two lines per subtitle and enough time to read.
The script (a Word file) was read in and laid line by line against the recognised text, with an alignment that keeps the order. The rule: what can be heard comes first. Only where recognition produced nonsense or half a sentence did the text come from the script. Of 416 compared passages, 54 stayed the same, 46 got a small correction, 234 came largely from the script and 82 were dropped because there was no speech. Sung lines the recognition missed were placed from the script onto the singing.
Every subtitle was laid against the sound again. Four speech recognisers take part: Whisper large-v3 and turbo, NVIDIA Parakeet with word timings, and Meta Omnilingual. The last one gives the probability of every letter for each fiftieth of a second, so the subtitle text can be placed on the audio letter by letter (forced alignment). For sung passages UVR-MDX separated the vocals from the music, and the lines were placed on the phrasing of the vocal track. The script was laid line by line against the subtitles once more to find missing passages.
How far from the spoken sentence did a subtitle start, before and after the check? Measured on 115 sentences the speech recogniser understood well. Because the same recogniser also helped with the timing, this is an indication rather than an independent test.
| Round | Subtitles | What it added |
|---|---|---|
| Speech recognition | 696 | A first version, with errors where the sound was poor |
| Script | 913 | Unclear passages from the script, 210 lines added |
| Check | 1,063 | 142 lines added, 66 corrected, about 370 re-timed |
What the tools and techniques from the steps above do, briefly explained. Everything except Claude is open source and ran in the cloud workspace or on the home PC, without paid services.