scher.nl
Personal project · November 2026

Video repair with Claude

A video recording of an hour and a half whose sound could not be heard. With Claude as technical guide it was repaired in six steps: sound back, a clearer picture, intelligible speech, and Dutch subtitles that were first made with speech recognition and then improved with the script.

Length
1:32:42
Subtitles
1,063
On time (± 0.5 s)
96%
File
14.1 → 4.9 GB
At a glance

Six steps, each building on the last

Each step made a new file next to the previous one, so there was always a way back. The large video file stayed on the home PC; Claude worked in a cloud workspace and fetched only what each step needed through the link with the PC.

Original 14.1 GBSoundPictureSpeechSubtitlesScriptCheckVideo + .srt
Starting point

The file

A 14.1 GB MP4 straight from the camera: 1080p picture at 24 frames per second, and a single MP3 audio track that could not be heard on playback. The sound was recorded with the camera's own microphone, at a distance. As a result speech, music and room noise are mixed in the same track, and voices are quiet compared with everything else.

Background

Sound in low and high frequencies

Sound is a mixture of vibrations at different frequencies, measured in hertz (Hz): the number of vibrations per second. Low tones, such as a hum or a bass guitar, vibrate tens of times per second; high tones, such as an 's' or the hiss of noise, thousands of times. A spectrum shows how much energy sits in each part of that range.

To see where the voice is and where the noise is, a two-minute test clip of dialogue was measured at two kinds of moments: while someone is speaking, and in the pauses in between. What remains in the pauses is the background noise.

Spectrum of speech and of background noise in the test clip Line chart per third of an octave from 50 hertz to 16 kilohertz. Below 150 hertz speech and pauses are equally loud: only rumble lives there. Between 250 and 650 hertz sits most of the voice energy, about 12 to 17 decibels above the background noise. Between 2 and 5 kilohertz, where the consonants are, the voice is 12 to 17 decibels weaker than in the vowels. Above 10 kilohertz there is mostly noise. rumble vowels consonants hiss and noise 0 -10 -20 -30 -40 -50 50 100 250 500 1k 2k 5k 10k Hz dB during speech in the pauses (background)
Done

Step by step

  1. 1

    Getting the sound back

    ClaudeFFmpeg

    First ffprobe showed what is in the file: which tracks, which codec, how long, and whether the audio track actually contains a signal. Then FFmpeg repaired the track and put it back with the picture, without touching the picture itself. A cross-correlation between old and new sound showed that picture and sound line up to the millisecond.

    BeforePicture without audible sound
    AfterSound audible, picture untouched
    ffprobeffmpegcross-correlation
  2. 2

    Improving the picture without generative AI

    ClaudeFFmpeg

    Exposure and sharpness were improved with FFmpeg's classic filters and re-encoded with x264. Deliberately no generative AI: it invents detail, and faces must stay exactly as they are. After encoding every frame was compared with the original (PSNR) to check that no frame was dropped or shifted.

    Before14.1 GB, as the camera made it
    After4.9 GB, brighter and sharper, frame-accurate
    FFmpeg filterslibx264PSNR
  3. 3

    Making speech intelligible

    ClaudeFFmpeg

    On a two-minute test clip three variants were first compared side by side, to listen before the whole video was processed: a speech filter (equaliser), the speech filter with an AI noise filter, and a full AI chain. Then a stronger version of the speech filter was added, and that became the choice. After that, dynamic normalisation (dynaudnorm) evens out quiet and loud passages, and a limiter prevents distortion.

    What the speech filter did per frequency Line chart of the gain per third of an octave, measured as the difference from the original at equal overall loudness. The stronger speech filter lowers everything below 150 hertz by 5 to 31 decibels, leaves 300 to 1000 hertz almost unchanged and boosts 2 to 5 kilohertz by 8 to 11 decibels. 15 5 -5 -15 -25 -35 50 100 250 500 1k 2k 5k 10k Hz dB louder quieter speech filter stronger speech filter (chosen)

    An equaliser changes the balance between high and low, not the ratio of voice to noise at the same pitch. The AI noise filter did remove 15 to 20 dB more noise, as the chart below shows, but after listening the choice still fell on the stronger speech filter.

    Gap between voice and background per frequency, for three variants Line chart of the difference in decibels between speech moments and pauses. The original and the speech filter almost overlap, at around 10 to 17 decibels between 250 hertz and 8 kilohertz. With the AI noise filter the gap is 25 to 35 decibels. 40 30 20 10 0 50 100 250 500 1k 2k 5k 10k Hz dB original stronger speech filter speech filter + AI noise filter
    BeforeVoices small and muffled under music and room
    AfterRumble removed, consonants 8 to 11 dB forward
    equalizerdynaudnormalimiter
  4. 4

    Subtitles with speech recognition

    ClaudeWhisper

    A voice detector (Silero VAD) cuts the sound into stretches with speech. Each stretch goes through Whisper large-v3, with Whisper turbo as a second opinion, via sherpa-onnx on the cloud workspace's processor. The raw text then gets a clean-up pass: repetitions and invented sentences removed, lines of at most 42 characters, two lines per subtitle and enough time to read.

    BeforeNo subtitles
    After696 subtitles, with errors where the sound is poor
    Silero VADsherpa-onnxWhisper large-v3Whisper turbo
  5. 5

    Improving with the script

    ClaudeRapidFuzz

    The script (a Word file) was read in and laid line by line against the recognised text, with an alignment that keeps the order. The rule: what can be heard comes first. Only where recognition produced nonsense or half a sentence did the text come from the script. Of 416 compared passages, 54 stayed the same, 46 got a small correction, 234 came largely from the script and 82 were dropped because there was no speech. Sung lines the recognition missed were placed from the script onto the singing.

    Before696 subtitles from speech recognition
    After913 subtitles, 210 lines added
    docxRapidFuzzDP alignment
  6. 6

    Full check of text and timing

    ClaudeForced alignment

    Every subtitle was laid against the sound again. Four speech recognisers take part: Whisper large-v3 and turbo, NVIDIA Parakeet with word timings, and Meta Omnilingual. The last one gives the probability of every letter for each fiftieth of a second, so the subtitle text can be placed on the audio letter by letter (forced alignment). For sung passages UVR-MDX separated the vocals from the music, and the lines were placed on the phrasing of the vocal track. The script was laid line by line against the subtitles once more to find missing passages.

    Before60% within 0.5 s of the speech
    After96% within 0.5 s; 142 lines added, 66 corrected
    NVIDIA ParakeetMeta OmnilingualUVR-MDX-Net
Result

Subtitles that keep pace

How far from the spoken sentence did a subtitle start, before and after the check? Measured on 115 sentences the speech recogniser understood well. Because the same recogniser also helped with the timing, this is an indication rather than an independent test.

Offset of the subtitle start, before and after the check Bar chart of 115 sentences. Before the check: 37 within 0.25 seconds, 32 between 0.25 and 0.5, 23 between 0.5 and 1, 17 between 1 and 2 and 6 more than 2 seconds off. After the check: 105, 5, 5, 0 and 0. before after < 0.25 s 37 105 0.25–0.5 s 32 5 0.5–1 s 23 5 1–2 s 17 0 > 2 s 6 0

Subtitles in three rounds

RoundSubtitlesWhat it added
Speech recognition696A first version, with errors where the sound was poor
Script913Unclear passages from the script, 210 lines added
Check1,063142 lines added, 66 corrected, about 370 re-timed
Tools

Tools and concepts explained

What the tools and techniques from the steps above do, briefly explained. Everything except Claude is open source and ran in the cloud workspace or on the home PC, without paid services.

Video

  • FFmpeg and ffprobe. Open-source command-line tooling that can read, edit and write almost any audio and video format. ffprobe is its measuring instrument: it shows which tracks a file contains, with which codec, how long and at what quality. Every picture and sound edit in the final video is an FFmpeg filter.
  • H.264 and x264. H.264 is the most widely used standard for compressing video: instead of storing every frame in full, it mainly stores what changes compared with the neighbouring frames. x264 is the open-source encoder that does this. Re-encoding brought the file down to about a third of the original.
  • PSNR. Peak Signal-to-Noise Ratio: a measure in decibels of how much a frame differs from a reference frame. The higher, the more alike; two identical frames give an infinite value. By comparing every new frame with the frame at the same position in the original, a shifted or missing frame stands out at once: the PSNR drops sharply.

Sound

  • Decibel and hertz. Hertz (Hz) is the number of vibrations per second, so the pitch; 1 kHz is 1000 Hz. Decibel (dB) is a logarithmic measure of strength: +6 dB is twice the amplitude, and +10 dB sounds roughly twice as loud. The charts divide the range into thirds of an octave: an octave is a doubling of frequency, and each band is a third of one.
  • Equaliser. A filter that makes each frequency range louder or quieter, like the bass and treble controls on a stereo but much finer. Here: the rumble below 150 Hz removed, the vowels roughly unchanged, the consonants around 2 to 5 kHz brought forward.
  • Dynamic normalisation (dynaudnorm). An FFmpeg filter that adjusts the volume per short time window: quiet passages go up, loud passages go down. That way you don't have to keep turning the volume knob between a whispering voice and a loud song.
  • Limiter (alimiter). The last step in the sound chain: it pushes peaks below a fixed ceiling, so the amplified sound never goes over the maximum and starts to crackle.
  • Cross-correlation. A calculation that slides two sound signals over each other step by step and checks at which shift they match best. That shift is the delay between them, down to the millisecond. This is how the new sound was checked to line up exactly with the picture.
  • AI noise filter (DeepFilterNet). A neural network that estimates, per frequency and per moment, how much of the sound is voice and how much is noise, and suppresses the noise. It removes much more than an equaliser, but can also make a voice sound slightly unnatural. It was tested, but not chosen.
  • UVR-MDX-Net. A neural network from Ultimate Vocal Remover that splits a mix into vocals and accompaniment. It works on the spectrogram, a picture of the sound over time and frequency, and estimates for each point how much of it belongs to the voice.

Speech to text

  • Silero VAD. Voice Activity Detection: a small neural network of a few megabytes that gives, for every piece of sound of about 30 milliseconds, the probability that someone is speaking. It was used to cut the sound into stretches with speech, so speech recognition doesn't waste time on music and silence and cannot invent text there.
  • Whisper large-v3 and turbo. Speech recognition from OpenAI, trained on millions of hours of audio in nearly a hundred languages. An encoder turns 30 seconds of sound at a time into a series of features, and a decoder writes the text for it. Large-v3 is the most accurate version, with over 1.5 billion parameters; turbo has a much smaller decoder and is therefore several times faster, a little less precise. Known weakness: with music or silence Whisper sometimes repeats itself or invents sentences.
  • sherpa-onnx and ONNX. ONNX is an open format for storing a trained neural network, independent of the program it was made in. sherpa-onnx is an open-source library that runs speech models in that format, even without a graphics card. The models were used in their int8 version: the numbers in the model are rounded to 8 bits, which makes them smaller and faster.
  • NVIDIA Parakeet. A speech recogniser from NVIDIA with 600 million parameters, for 25 European languages including Dutch. For every word piece it also predicts how long it lasts, giving a start and end time per word. That made it the second yardstick for the timing.
  • Meta Omnilingual and CTC. A speech recogniser from Meta for more than 1,600 languages. The version used works with CTC (Connectionist Temporal Classification): for every fiftieth of a second the model gives the probability of every letter, plus the probability of 'no new letter'. Exactly those letter probabilities make forced alignment possible.

Text and timing

  • Forced alignment. In ordinary speech recognition the text is unknown. In forced alignment the text is already known, namely the subtitle, and the only question is when each letter sounds. An algorithm (Viterbi) searches the letter probabilities from Omnilingual for the most likely path that runs through the letters in the right order. Between subtitles the path may pause freely, so silences and music don't shift anything.
  • RapidFuzz and Levenshtein distance. The Levenshtein distance counts how many letters you have to add, remove or replace to turn one word into another. RapidFuzz calculates it very quickly. That is how the script recognises that the Dutch 'musiek' and 'muziek' are almost the same, and how the script can be laid next to recognised text that contains errors.
  • Alignment with dynamic programming. A method for laying two sequences, such as the lines from the script and the recognised sentences, on top of each other so that the order is right and the total match is as large as possible. Lines that were skipped or added in the recording are recognised as a gap instead of shifting everything after them.
  • SRT. The simplest subtitle format: a text file with, for each subtitle, a number, a start and end time to the millisecond, and one or two lines of text. Almost every video player and TV reads it alongside the video file.

Way of working

  • Claude (Cowork). Claude worked in its own cloud workspace with the link to the folder on the PC. That is where the plan, all scripts, the measurements and the checks were made, and where the trade-offs were put forward: which variant, which rules, what to do when in doubt. The heavy steps ran in the cloud, and the result came back to the folder on the PC.
Ground rules

What was agreed up front

Learned

What turned out to be hard