@remotion/whisper-webgpuv4.0.518

Transcribe audio locally in the browser using timestamped Whisper models and WebGPU through Transformers.js, and convert the result to @remotion/captions.

Installation

Remotion CLI

npx remotion add @remotion/whisper-webgpu @huggingface/transformers

npm

npm i --save-exact @remotion/whisper-webgpu@4.0.526 @huggingface/transformers@4.3.0

This assumes you are currently using v4.0.526 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

bun

pnpm i @remotion/whisper-webgpu@4.0.526 @huggingface/transformers@4.3.0

This assumes you are currently using v4.0.526 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

pnpm

bun i @remotion/whisper-webgpu@4.0.526 @huggingface/transformers@4.3.0

This assumes you are currently using v4.0.526 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

yarn

yarn --exact add @remotion/whisper-webgpu@4.0.526 @huggingface/transformers@4.3.0

This assumes you are currently using v4.0.526 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

Example

transcribe.ts
import {clearStaleModels, resampleTo16Khz, toCaptions, transcribe} from '@remotion/whisper-webgpu';

await clearStaleModels();

const file = new File([], 'audio.wav');
const channelWaveform = await resampleTo16Khz({file});

const transcription = await transcribe({
  channelWaveform,
  model: 'small.en',
  language: 'en',
});

const {captions} = toCaptions({whisperWebGpuOutput: transcription});
console.log(captions);

The model is downloaded on the first call and cached by the browser. Call loadWhisperModel() first to display model download progress.

Model hosting

We noticed that hosting the models on R2 leads to faster downloading than through Hugging Face. Therefore, we mirrored the models to remotion.media, keeping them byte-identical.

APIs

canUseWhisperWebGpu()

Check whether WebGPU is available

getAvailableModels()

List timestamped models and their download sizes

clearStaleModels()

Remove models discontinued by newer versions

isWhisperModelCached()

Check whether a model is downloaded

loadWhisperModel()

Download and initialize a model

removeWhisperModel()

Remove a model from the browser cache

transcribe()

Transcribe a waveform with word-level timestamps

toCaptions()

Convert a transcription to Remotion captions

resampleTo16Khz()

Decode and resample browser audio

disposeWhisperModel()

Release model memory

Requirements

WebGPU requires HTTPS in production or localhost during development. Use canUseWhisperWebGpu() to check for a usable adapter.

License

MIT