Compare commits

...

14 Commits

Author SHA1 Message Date
1072679360 fix(whisper): handle model warmup events
All checks were successful
Build & Push Docker Image / test (push) Successful in 12s
Build & Push Docker Image / build-and-push (push) Successful in 52s
- Ignore backend model lifecycle webhooks so model warmup does not
  mark jobs done early
- Parse batched SSE messages and relay model load states during
  submit retries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-15 00:08:32 +02:00
f70cefc5e9 fix(progress): separate model warmup state
All checks were successful
Build & Push Docker Image / test (push) Successful in 11s
Build & Push Docker Image / build-and-push (push) Successful in 42s
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 00:52:33 +02:00
929c482497 refactor(transcript): drop Tonemark rewrite
All checks were successful
Build & Push Docker Image / test (push) Successful in 10s
Build & Push Docker Image / build-and-push (push) Successful in 50s
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-12 00:10:32 +02:00
df50e74939 test(vitest): serialize db-backed suites
All checks were successful
Build & Push Docker Image / test (push) Successful in 12s
Build & Push Docker Image / build-and-push (push) Successful in 40s
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-11 23:29:31 +02:00
34196b8110 test(push): relax flaky call count
Some checks failed
Build & Push Docker Image / test (push) Failing after 10s
Build & Push Docker Image / build-and-push (push) Has been skipped
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-11 23:25:53 +02:00
3a72bb815f fix(postprocess): trim adjacent word overlap
Some checks failed
Build & Push Docker Image / test (push) Failing after 11s
Build & Push Docker Image / build-and-push (push) Has been skipped
Remove residual one-word suffix-prefix carry-over between adjacent caption segments so reprocessed transcripts no longer repeat bridge words across lines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-11 23:24:21 +02:00
6beb436687 fix(postprocess): drop tiny carry-over text
All checks were successful
Build & Push Docker Image / test (push) Successful in 11s
Build & Push Docker Image / build-and-push (push) Successful in 43s
Collapse one-word and very short caption carry-over fragments so reprocessed YouTube transcripts do not retain residual prefix chains.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-11 23:14:31 +02:00
672b161cda fix(transcript): collapse rolling segment echoes
All checks were successful
Build & Push Docker Image / test (push) Successful in 12s
Build & Push Docker Image / build-and-push (push) Successful in 45s
Normalize incremental backend hypothesis chains before persistence and ignore stale or replayed webhook callbacks so duplicate transcript text does not survive ingest.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-11 22:46:38 +02:00
Giancarmine Salucci
35a2d86dbb test: add beforeEach cleanup in push.test.ts to prevent flaky state leakage
All checks were successful
Build & Push Docker Image / test (push) Successful in 10s
Build & Push Docker Image / build-and-push (push) Successful in 42s
Adds a beforeEach hook that clears subscriptions and resets mocks before
each test, making the suite robust against any state left by a previous
test even if afterEach didn't run cleanly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-10 15:56:05 +02:00
Giancarmine Salucci
470dd1642f fix: clear modelWarming notice when job completes via SSE done event
Some checks failed
Build & Push Docker Image / test (push) Failing after 10s
Build & Push Docker Image / build-and-push (push) Has been skipped
Previously, if a job completed while the model-warming notice was shown
(e.g. model loaded mid-job), the 'Warming up model' banner persisted on
the Done screen because the SSE 'done' handler didn't clear modelWarming.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-10 15:52:28 +02:00
Giancarmine Salucci
10a3669b42 fix: FormData stream exhausted on retry + undefined segments crash
All checks were successful
Build & Push Docker Image / test (push) Successful in 32s
Build & Push Docker Image / build-and-push (push) Successful in 46s
Two bugs triggered together when the model was unloaded during a job:

1. submitJob() created FormData/createReadStream once outside the retry loop.
   After a 503, the audio ReadStream was consumed and subsequent retries sent
   an empty body to whisper, causing it to return segments:undefined.

2. webhook handler cast whisperJob.segments as Segment[] without guarding
   against undefined, so deduplicateSegments(undefined) crashed with
   'Cannot read properties of undefined (reading 'map')' — stored as job.error.

Fixes:
- Move FormData + createReadStream inside the retry loop (fresh stream per attempt)
- Use (whisperJob.segments ?? []) in webhook handler
- Add Array.isArray guard at top of deduplicateSegments() as belt-and-suspenders

Tests:
- New: verifies createReadStream called once per attempt (3 attempts = 3 streams)
- New: webhook handles segments:undefined without throwing
- New: webhook handles segments:null without throwing
- 150/150 passing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-10 15:37:07 +02:00
Giancarmine Salucci
a76625d378 ci: use npm install instead of npm ci to avoid lock file version mismatch
All checks were successful
Build & Push Docker Image / test (push) Successful in 10s
Build & Push Docker Image / build-and-push (push) Successful in 44s
Lock file was generated with npm 11 (Node 24), CI runs npm 10 (Node 22).
npm install avoids the strict sync check and matches the Dockerfile.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-09 15:54:32 +02:00
Giancarmine Salucci
76051e52dd ci: add test job before Docker build
Some checks failed
Build & Push Docker Image / test (push) Failing after 45s
Build & Push Docker Image / build-and-push (push) Has been skipped
Run vitest before building the image so a failing test blocks the push.
build-and-push now depends on the test job passing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-09 15:51:24 +02:00
Giancarmine Salucci
53f874aec7 feat: proxy POST /model/unload endpoint
All checks were successful
Build & Push Docker Image / build-and-push (push) Successful in 43s
- Add unloadModel() to whisper.ts: POSTs to /model/unload with 10s
  timeout, returns parsed JSON body, throws on non-ok response
- Create src/routes/api/model/unload/+server.ts: thin POST proxy,
  passes whisper's response through, returns 502 if whisper unreachable
- Add 3 unloadModel tests (success, WHISPER_URL config, error propagation)
  — 147/147 passing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-09 15:48:47 +02:00
26 changed files with 828 additions and 528 deletions

View File

@@ -15,8 +15,28 @@ env:
IMAGE_NAME: mozempk/tonemark
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
- name: Install dependencies
run: npm install
- name: Run tests
run: npm test
build-and-push:
runs-on: ubuntu-latest
needs: test
steps:
- name: Checkout repository

42
package-lock.json generated
View File

@@ -12,7 +12,8 @@
"better-sqlite3": "^12.9.0",
"form-data": "^4.0.5",
"node-fetch": "^3.3.2",
"web-push": "^3.6.7"
"web-push": "^3.6.7",
"youtube-transcript": "^1.3.1"
},
"devDependencies": {
"@sveltejs/adapter-auto": "^7.0.1",
@@ -89,6 +90,27 @@
"node": ">=18"
}
},
"node_modules/@emnapi/core": {
"version": "1.10.0",
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz",
"integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==",
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/wasi-threads": "1.2.1",
"tslib": "^2.4.0"
}
},
"node_modules/@emnapi/runtime": {
"version": "1.10.0",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz",
"integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==",
"license": "MIT",
"optional": true,
"dependencies": {
"tslib": "^2.4.0"
}
},
"node_modules/@emnapi/wasi-threads": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz",
@@ -896,7 +918,6 @@
"resolved": "https://registry.npmjs.org/@sveltejs/kit/-/kit-2.59.1.tgz",
"integrity": "sha512-d8OON70AphLdDesuTIl//M2O6fRTIicX8aYv8vhCiYEhTTI2OboKqey0Hu1A4VFhqwgqtq0vKDmPFGkw8kKmgw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@standard-schema/spec": "^1.0.0",
"@sveltejs/acorn-typescript": "^1.0.5",
@@ -938,7 +959,6 @@
"resolved": "https://registry.npmjs.org/@sveltejs/vite-plugin-svelte/-/vite-plugin-svelte-7.1.1.tgz",
"integrity": "sha512-FOJdbE5pxae68DoTBJ49t1dIA7TSmMHR6CsuJhX90cO/UfrEMHA7KJNUj3WdZuUDJPu4ujqpJ2Tgqd2gTWr6Xg==",
"license": "MIT",
"peer": true,
"dependencies": {
"deepmerge": "^4.3.1",
"magic-string": "^0.30.21",
@@ -1313,7 +1333,6 @@
"integrity": "sha512-38C0/Ddb7HcRG0Z4/DUem8x57d2p9jYgp18mkaYswEOQBGsI1CG4f/hjm0ZCeaJfWhSZ4k7jgs29V1Zom7Ki9A==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@bcoe/v8-coverage": "^1.0.2",
"@vitest/utils": "4.1.5",
@@ -1467,7 +1486,6 @@
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.16.0.tgz",
"integrity": "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==",
"license": "MIT",
"peer": true,
"bin": {
"acorn": "bin/acorn"
},
@@ -3021,7 +3039,6 @@
"resolved": "https://registry.npmjs.org/rollup/-/rollup-4.60.3.tgz",
"integrity": "sha512-pAQK9HalE84QSm4Po3EmWIZPd3FnjkShVkiMlz1iligWYkWQ7wHYd1PF/T7QZ5TVSD6uSTon5gBVMSM4JfBV+A==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/estree": "1.0.8"
},
@@ -3255,7 +3272,6 @@
"resolved": "https://registry.npmjs.org/svelte/-/svelte-5.55.5.tgz",
"integrity": "sha512-2uCs/LZ9us+AktdzYJM8OcxQ8qnPS1kpaO7syGT/MgO+6Qr1Ybl+TqPq+97u7PHqmmMlye5ZkoyXONy5mjjAbw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@jridgewell/remapping": "^2.3.4",
"@jridgewell/sourcemap-codec": "^1.5.0",
@@ -3428,7 +3444,6 @@
"integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
"devOptional": true,
"license": "Apache-2.0",
"peer": true,
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
@@ -3455,7 +3470,6 @@
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
"license": "MIT",
"peer": true,
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
@@ -3553,7 +3567,6 @@
"integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@vitest/expect": "4.1.5",
"@vitest/mocker": "4.1.5",
@@ -3689,6 +3702,15 @@
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"license": "ISC"
},
"node_modules/youtube-transcript": {
"version": "1.3.1",
"resolved": "https://registry.npmjs.org/youtube-transcript/-/youtube-transcript-1.3.1.tgz",
"integrity": "sha512-NDCjwad113TGybbYF51y9Z4tcwzBHUZWQdF9veULNca18L+FdDbHHtTHIr69WVa3bB90l67S8kN0HtL2JO9fhg==",
"license": "MIT",
"engines": {
"node": ">=18.0.0"
}
},
"node_modules/zimmerframe": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/zimmerframe/-/zimmerframe-1.1.4.tgz",

View File

@@ -34,6 +34,7 @@
"better-sqlite3": "^12.9.0",
"form-data": "^4.0.5",
"node-fetch": "^3.3.2",
"web-push": "^3.6.7"
"web-push": "^3.6.7",
"youtube-transcript": "^1.3.1"
}
}

View File

@@ -16,10 +16,10 @@
type RecordState = 'idle' | 'requesting' | 'recording' | 'stopping';
let state = $state<RecordState>('idle');
let error = $state('');
let elapsed = $state(0); // seconds
let liveData = $state<Float32Array | null>(null);
let recordState: RecordState = $state('idle');
let error: string = $state('');
let elapsed: number = $state(0); // seconds
let liveData: Float32Array | null = $state(null);
let mediaRecorder: MediaRecorder | null = null;
let chunks: Blob[] = [];
@@ -60,12 +60,12 @@
async function startRecording() {
error = '';
state = 'requesting';
recordState = 'requesting';
try {
stream = await navigator.mediaDevices.getUserMedia({ audio: true });
} catch {
error = 'Microphone access denied';
state = 'idle';
recordState = 'idle';
return;
}
@@ -81,11 +81,11 @@
elapsed = 0;
timerInterval = setInterval(() => elapsed++, 1000);
state = 'recording';
recordState = 'recording';
}
function stopRecording() {
state = 'stopping';
recordState = 'stopping';
mediaRecorder?.stop();
if (timerInterval) clearInterval(timerInterval);
if (animFrame) cancelAnimationFrame(animFrame);
@@ -99,7 +99,7 @@
const ext = mime.includes('ogg') ? 'ogg' : mime.includes('mp4') ? 'mp4' : 'webm';
const blob = new Blob(chunks, { type: mime });
const filename = `recording-${new Date().toISOString().slice(0, 19).replace(/[T:]/g, '-')}.${ext}`;
state = 'idle';
recordState = 'idle';
ondone?.(blob, filename);
}
@@ -116,15 +116,18 @@
{ length: IDLE_BARS },
(_, i) => 3 + Math.abs(Math.sin(i * 0.7) + Math.cos(i * 0.31)) * 20
);
const liveBars = $derived.by<number[]>(() =>
liveData ? Array.from(liveData.slice(0, IDLE_BARS), (value) => Number(value)) : []
);
</script>
<div class="recorder">
<!-- Waveform display -->
<div class="waveform-area" aria-hidden="true">
{#if state === 'recording' && liveData}
{#if recordState === 'recording' && liveData}
<!-- Live waveform from AnalyserNode -->
<svg viewBox="0 0 {IDLE_BARS * 5} 28" preserveAspectRatio="none" class="waveform-svg">
{#each Array.from(liveData).slice(0, IDLE_BARS) as v, i}
{#each liveBars as v, i}
{@const h = 2 + v * 24}
<rect
x={i * 5}
@@ -147,8 +150,8 @@
width="3"
height={h}
rx="1.5"
fill={state === 'idle' ? 'rgba(255,255,255,0.15)' : accent}
opacity={state === 'idle' ? 1 : 0.3}
fill={recordState === 'idle' ? 'rgba(255,255,255,0.15)' : accent}
opacity={recordState === 'idle' ? 1 : 0.3}
/>
{/each}
</svg>
@@ -156,7 +159,7 @@
</div>
<!-- Timer (recording only) -->
{#if state === 'recording'}
{#if recordState === 'recording'}
<div class="timer" style="color: {accent}">
<span class="rec-dot" style="background: {accent}"></span>
{formatTime(elapsed)}
@@ -170,15 +173,15 @@
<!-- Buttons -->
<div class="btn-row">
{#if state === 'idle' || state === 'requesting'}
{#if recordState === 'idle' || recordState === 'requesting'}
<button
class="btn-record"
style="background: {accent}; color: #0c0d10;"
onclick={startRecording}
disabled={state === 'requesting'}
disabled={recordState === 'requesting'}
aria-label="Start recording"
>
{#if state === 'requesting'}
{#if recordState === 'requesting'}
<svg width="13" height="13" viewBox="0 0 13 13" style="animation: spin 1s linear infinite">
<circle cx="6.5" cy="6.5" r="5" stroke="currentColor" stroke-width="1.5" fill="none" stroke-dasharray="20 12"/>
</svg>
@@ -190,7 +193,7 @@
Record
{/if}
</button>
{:else if state === 'recording'}
{:else if recordState === 'recording'}
<button
class="btn-stop"
onclick={stopRecording}

52
src/lib/job-progress.ts Normal file
View File

@@ -0,0 +1,52 @@
import type { Job, JobStatus } from '$lib/types.js';
export const TERMINAL_JOB_STATUSES: readonly JobStatus[] = ['done', 'failed', 'cancelled'];
const STATUS_LABELS: Record<JobStatus, string> = {
pending: 'Pending',
downloading: 'Downloading',
preparing: 'Preparing',
warming_model: 'Loading model',
transcribing: 'Transcribing',
processing: 'Processing',
done: 'Done',
failed: 'Failed',
cancelled: 'Cancelled'
};
const STATUS_COLORS: Record<JobStatus, string> = {
done: '#cdf24e',
failed: '#ff6b6b',
cancelled: 'rgba(232,233,236,0.3)',
processing: '#76daa2',
transcribing: '#80c7f7',
warming_model: '#76daa2',
preparing: '#fbc94b',
downloading: '#a78bfa',
pending: 'rgba(232,233,236,0.4)'
};
export function isTerminalJobStatus(status: JobStatus): boolean {
return TERMINAL_JOB_STATUSES.includes(status);
}
export function getJobStatusLabel(status: JobStatus): string {
return STATUS_LABELS[status];
}
export function getJobStatusColor(status: JobStatus): string {
return STATUS_COLORS[status];
}
export function getDisplayJobProgress(
job: Pick<Job, 'status' | 'progress' | 'segmentsJson'>,
options: { hasTranscript?: boolean } = {}
): number {
const progress = Math.max(0, Math.min(100, Math.round(job.progress)));
if (job.status === 'warming_model') return Math.min(progress, 15);
if (!isTerminalJobStatus(job.status)) return Math.min(progress, 99);
if (job.status === 'done' && !options.hasTranscript) return Math.min(progress, 99);
return progress;
}

View File

@@ -1,8 +1,9 @@
import { execFile } from 'child_process';
import { promisify } from 'util';
import { existsSync } from 'fs';
import { mkdir, unlink, writeFile } from 'fs/promises';
import { mkdir, writeFile } from 'fs/promises';
import { join } from 'path';
import { fetchTranscript, type TranscriptResponse } from 'youtube-transcript';
const execFileAsync = promisify(execFile);
const TMP_DIR = join(process.env.DATA_DIR ?? '/tmp/.whisper-pwa', 'downloads');
@@ -26,43 +27,33 @@ export interface AudioResult {
export type DownloadResult = CaptionResult | AudioResult;
/** Try to get auto-generated captions from YouTube. Returns null if unavailable. */
async function tryGetCaptions(url: string, outDir: string): Promise<CaptionResult | null> {
const jsonPath = join(outDir, 'info.json');
async function tryGetCaptions(url: string, _outDir: string): Promise<CaptionResult | null> {
try {
await execFileAsync('yt-dlp', [
'--write-auto-subs',
'--sub-langs', 'en.*',
'--skip-download',
'--write-info-json',
'--no-playlist',
'-o', join(outDir, '%(title)s.%(ext)s'),
url
]);
// Find the VTT/SRT file
const { readdirSync } = await import('fs');
const files = readdirSync(outDir);
const vttFile = files.find((f) => f.endsWith('.vtt') || f.endsWith('.srt'));
if (!vttFile) return null;
let title = 'Untitled';
if (existsSync(jsonPath)) {
try {
const info = JSON.parse((await import('fs')).readFileSync(jsonPath, 'utf8'));
title = info.title ?? title;
} catch { /* ignore */ }
}
const content = (await import('fs')).readFileSync(join(outDir, vttFile), 'utf8');
const segments = parseVtt(content);
const transcript = await fetchTranscript(url, { lang: 'en' });
const segments = transcriptEntriesToSegments(transcript);
if (segments.length === 0) return null;
const title = await getYouTubeTitle(url);
return { type: 'captions', segments, title };
} catch {
return null;
}
}
async function getYouTubeTitle(url: string): Promise<string> {
try {
const { stdout } = await execFileAsync('yt-dlp', [
'--dump-single-json',
'--skip-download',
'--no-playlist',
url
]);
return JSON.parse(stdout).title ?? 'Untitled';
} catch {
return 'Untitled';
}
}
/** Download best audio from YouTube. Returns path to audio file. */
async function downloadAudio(url: string, outDir: string): Promise<{ audioPath: string; title: string }> {
await execFileAsync('yt-dlp', [
@@ -124,39 +115,22 @@ export async function cleanupJobTmp(jobId: string) {
} catch { /* ignore */ }
}
/** Parse a WebVTT string into segments. */
function parseVtt(
content: string
export function transcriptEntriesToSegments(
entries: TranscriptResponse[]
): Array<{ index: number; start: number; end: number; text: string; words: [] }> {
const segments: Array<{ index: number; start: number; end: number; text: string; words: [] }> = [];
const blocks = content.split(/\n\n+/);
let index = 0;
for (const block of blocks) {
const lines = block.trim().split('\n');
const timeLine = lines.find((l) => l.includes('-->'));
if (!timeLine) continue;
const [startStr, endStr] = timeLine.split('-->').map((s) => s.trim().split(' ')[0]);
const start = vttTimeToSec(startStr);
const end = vttTimeToSec(endStr);
const text = lines
.filter((l) => !l.includes('-->') && !/^\d+$/.test(l.trim()) && l.trim())
.join(' ')
.replace(/<[^>]+>/g, '')
.trim();
if (text) {
segments.push({ index: index++, start, end, text, words: [] });
}
}
return segments;
}
function vttTimeToSec(t: string): number {
const parts = t.split(':').map(Number);
if (parts.length === 3) return parts[0] * 3600 + parts[1] * 60 + parts[2];
if (parts.length === 2) return parts[0] * 60 + parts[1];
return parts[0];
const useMilliseconds = entries.some((entry) => entry.offset > 1000 || entry.duration > 1000);
return entries
.map((entry) => {
const start = useMilliseconds ? entry.offset / 1000 : entry.offset;
const duration = useMilliseconds ? entry.duration / 1000 : entry.duration;
return {
index: 0,
start,
end: start + duration,
text: entry.text.trim(),
words: [] as []
};
})
.filter((entry) => entry.text.length > 0)
.map((entry, index) => ({ ...entry, index }));
}

View File

@@ -96,15 +96,13 @@ async function runJob(
if (captionSegments) {
// Caption fast path — skip whisper
const { deduplicateSegments } = await import('./postprocess.js');
const { writeOutputs } = await import('./formatter.js');
const segments = deduplicateSegments(captionSegments);
const paths = await writeOutputs(segments, title, jobId);
const paths = await writeOutputs(captionSegments, title, jobId);
updateJob({
id: jobId,
status: 'done',
progress: 100,
segmentsJson: JSON.stringify(segments),
segmentsJson: JSON.stringify(captionSegments),
outputDir: paths.srt.replace(/\/[^/]+$/, '')
});
emitProgress(jobId, { type: 'done' });
@@ -126,13 +124,16 @@ async function runJob(
// ── 4. Submit to whisper with webhook ────────────────────────────────
setJobStatus(jobId, 'transcribing', 10);
emitProgress(jobId, { type: 'status', status: 'transcribing' });
emitProgress(jobId, { type: 'status', status: 'transcribing', progress: 10 });
const webhookUrl = `${WEBHOOK_BASE_URL}/api/webhook/${jobId}`;
const whisperJobId = await submitJob(wavPath, webhookUrl, language, (state, retryAfterSecs) => {
emitProgress(jobId, { type: 'model_warming', state, retryAfterSecs });
setJobStatus(jobId, 'warming_model', 10);
emitProgress(jobId, { type: 'model_warming', status: 'warming_model', state, retryAfterSecs, progress: 10 });
});
updateJob({ id: jobId, whisperJobId });
setJobStatus(jobId, 'transcribing', 10);
emitProgress(jobId, { type: 'status', status: 'transcribing', progress: 10 });
// ── 5. Open SSE for live progress (non-blocking relay) ───────────────
streamJob(

View File

@@ -1,108 +0,0 @@
import type { Segment } from '$lib/types.js';
// ── Collapse consecutive repeated phrases within a segment's text ────────────
function collapseRepeats(text: string): string {
let prev = '';
// Keep applying until stable
while (true) {
const next = collapseOnce(text);
if (next === prev || next === text) return next;
prev = text;
text = next;
}
}
function collapseOnce(text: string): string {
// Match any repeated phrase (2+ words) appearing consecutively
return text.replace(/\b(.{10,}?)\s+\1\b/gi, '$1');
}
// ── Merge consecutive segments with identical (or near-identical) text ───────
function normalise(s: string) {
return s.toLowerCase().replace(/[^\w\s]/g, '').replace(/\s+/g, ' ').trim();
}
function mergeConsecutive(segments: Segment[]): Segment[] {
const out: Segment[] = [];
for (const seg of segments) {
const last = out[out.length - 1];
if (last && normalise(last.text) === normalise(seg.text)) {
last.end = seg.end;
} else {
out.push({ ...seg });
}
}
return out;
}
// ── N-gram deduplication ─────────────────────────────────────────────────────
const NGRAM_N = 6;
const LOOKBACK_CHARS = 500;
const SIMILARITY_THRESHOLD = 0.6;
function ngrams(text: string, n: number): string[] {
const words = text.toLowerCase().split(/\s+/);
const grams: string[] = [];
for (let i = 0; i <= words.length - n; i++) {
grams.push(words.slice(i, i + n).join(' '));
}
return grams;
}
function jaccardSimilarity(a: string, b: string): number {
const ga = new Set(ngrams(a, NGRAM_N));
const gb = new Set(ngrams(b, NGRAM_N));
// If neither text is long enough to produce n-grams they cannot be compared;
// treat as dissimilar so short segments are never incorrectly discarded.
if (ga.size === 0 && gb.size === 0) return 0;
const intersection = [...ga].filter((g) => gb.has(g)).length;
const union = new Set([...ga, ...gb]).size;
return union === 0 ? 0 : intersection / union;
}
function ngramDedup(segments: Segment[]): Segment[] {
const out: Segment[] = [];
for (const seg of segments) {
const windowText = out
.slice(-20)
.map((s) => s.text)
.join(' ')
.slice(-LOOKBACK_CHARS);
if (windowText.length > 0 && jaccardSimilarity(seg.text, windowText) >= SIMILARITY_THRESHOLD) {
continue; // duplicate — skip
}
out.push(seg);
}
return out;
}
// ── Full deduplication pipeline ──────────────────────────────────────────────
export function deduplicateSegments(segments: Segment[]): Segment[] {
// 1. Collapse repeats within each segment's text
let result = segments.map((s) => ({
...s,
text: collapseRepeats(s.text.trim())
}));
// 2. Remove empty segments
result = result.filter((s) => s.text.length > 0);
// 3. First merge pass
result = mergeConsecutive(result);
// 4. N-gram dedup
result = ngramDedup(result);
// 5. Second merge pass (catches new adjacencies after dedup)
result = mergeConsecutive(result);
// 6. Re-index
result.forEach((s, i) => (s.index = i));
return result;
}

View File

@@ -1,4 +1,32 @@
import type { ModelStatus } from '$lib/types.js';
import type { ModelStateTag, ModelStatus } from '$lib/types.js';
const MODEL_STATES = new Set<ModelStateTag>(['unloaded', 'loading', 'waiting_for_gpu', 'ready']);
function isModelStateTag(value: unknown): value is ModelStateTag {
return typeof value === 'string' && MODEL_STATES.has(value as ModelStateTag);
}
function extractSseMessages(buffer: string): { messages: { eventType: string; data: string }[]; rest: string } {
const normalized = buffer.replace(/\r/g, '');
const chunks = normalized.split('\n\n');
const rest = chunks.pop() ?? '';
const messages = chunks
.map((chunk) => {
let eventType = '';
const dataLines: string[] = [];
for (const line of chunk.split('\n')) {
if (line.startsWith('event:')) {
eventType = line.slice(6).trim();
} else if (line.startsWith('data:')) {
dataLines.push(line.slice(5).trim());
}
}
return { eventType, data: dataLines.join('\n') };
})
.filter((message) => message.data.length > 0);
return { messages, rest };
}
function whisperUrl() {
return process.env.WHISPER_URL ?? 'http://localhost:8080';
@@ -22,7 +50,10 @@ export async function getModelStatus(): Promise<ModelStatus> {
* SSE connection fails or closes without that event, so the retry loop can
* try again without hanging indefinitely.
*/
async function waitForModelReady(timeoutMs: number): Promise<void> {
async function waitForModelReady(
timeoutMs: number,
onStateChange?: (state: ModelStateTag) => void
): Promise<void> {
const { default: fetch } = await import('node-fetch');
const ac = new AbortController();
return new Promise((resolve) => {
@@ -47,17 +78,18 @@ async function waitForModelReady(timeoutMs: number): Promise<void> {
for await (const chunk of res.body) {
if (ac.signal.aborted) break;
buf += chunk.toString();
const lines = buf.split('\n');
buf = lines.pop() ?? '';
for (const line of lines) {
if (!line.startsWith('data:')) continue;
const { messages, rest } = extractSseMessages(buf);
buf = rest;
for (const message of messages) {
try {
const payload = JSON.parse(line.slice(5).trim());
const payload = JSON.parse(message.data) as { state?: unknown };
if (!isModelStateTag(payload.state)) continue;
if (payload.state === 'ready') {
clearTimeout(timer);
finish();
return;
}
onStateChange?.(payload.state);
} catch { /* ignore parse errors */ }
}
}
@@ -83,20 +115,23 @@ export async function submitJob(
wavPath: string,
webhookUrl: string,
language?: string,
onModelWaiting?: (state: string, retryAfterSecs: number) => void,
onModelWaiting?: (state: ModelStateTag, retryAfterSecs: number) => void,
maxAttempts = 20
): Promise<string> {
const FormData = (await import('form-data')).default;
const { createReadStream } = await import('fs');
const { default: fetch } = await import('node-fetch');
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
// Recreate form with a fresh readable stream on every attempt.
// A consumed ReadStream cannot be rewound, so reusing it across retries
// would send an empty body to whisper after the first 503.
const form = new FormData();
form.append('audio', createReadStream(wavPath));
form.append('task', 'transcribe');
form.append('webhook_url', webhookUrl);
if (language) form.append('language', language);
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const res = await fetch(`${whisperUrl()}/jobs`, {
method: 'POST',
body: form,
@@ -113,10 +148,15 @@ export async function submitJob(
state?: string;
retry_after_secs?: number;
};
const state = body.state ?? 'unloaded';
const state = isModelStateTag(body.state) ? body.state : 'unloaded';
const waitSecs = body.retry_after_secs ?? parseInt(res.headers.get('Retry-After') ?? '15');
onModelWaiting?.(state, waitSecs);
await waitForModelReady((waitSecs + 1) * 1000);
let lastState = state;
await waitForModelReady((waitSecs + 1) * 1000, (nextState) => {
if (nextState === lastState) return;
lastState = nextState;
onModelWaiting?.(nextState, waitSecs);
});
continue;
}
@@ -127,6 +167,17 @@ export async function submitJob(
throw new Error(`Whisper model did not become ready after ${maxAttempts} attempts`);
}
/** Unload the model from VRAM. Throws if the whisper server returns non-ok. */
export async function unloadModel(): Promise<{ ok: boolean }> {
const { default: fetch } = await import('node-fetch');
const res = await fetch(`${whisperUrl()}/model/unload`, {
method: 'POST',
signal: AbortSignal.timeout(10000)
});
if (!res.ok) throw new Error(`/model/unload returned ${res.status}`);
return res.json() as Promise<{ ok: boolean }>;
}
/**
* Cancel a queued or running job on the whisper server (best-effort).
* Errors are silently ignored — local job status is already set to cancelled.
@@ -155,20 +206,12 @@ export async function streamJob(
let buf = '';
for await (const chunk of res.body) {
buf += chunk.toString();
const lines = buf.split('\n');
buf = lines.pop() ?? '';
let eventType = '';
let dataLine = '';
for (const line of lines) {
if (line.startsWith('event:')) eventType = line.slice(6).trim();
else if (line.startsWith('data:')) dataLine = line.slice(5).trim();
}
if (!dataLine) continue;
const { messages, rest } = extractSseMessages(buf);
buf = rest;
for (const message of messages) {
try {
const payload = JSON.parse(dataLine);
const payload = JSON.parse(message.data);
if (payload.type === 'progress') {
onProgress(payload.percent ?? 0, payload.chunk ?? 0, payload.total ?? 0);
} else if (payload.type === 'done') {
@@ -180,6 +223,7 @@ export async function streamJob(
}
} catch { /* ignore parse errors */ }
}
}
}
/** Check if the whisper server is healthy. */

View File

@@ -12,7 +12,16 @@ export interface ModelStatus {
vram_total_mb?: number;
}
export type JobStatus = 'pending' | 'downloading' | 'preparing' | 'transcribing' | 'processing' | 'done' | 'failed' | 'cancelled';
export type JobStatus =
| 'pending'
| 'downloading'
| 'preparing'
| 'warming_model'
| 'transcribing'
| 'processing'
| 'done'
| 'failed'
| 'cancelled';
export interface Segment {
index: number;

View File

@@ -1,6 +1,7 @@
<script lang="ts">
import { onMount } from 'svelte';
import type { Job, AudioMode } from '$lib/types.js';
import { getDisplayJobProgress, getJobStatusLabel } from '$lib/job-progress.js';
import SourceIcon from '$lib/components/SourceIcon.svelte';
import Waveform from '$lib/components/Waveform.svelte';
import RecordButton from '$lib/components/RecordButton.svelte';
@@ -95,8 +96,7 @@
const parts: string[] = [];
if (job.source && !job.source.startsWith('http')) parts.push(job.source.split('/').pop() ?? '');
if (job.audioMode) parts.push(job.audioMode);
if (job.status === 'done') parts.push('done');
else parts.push(job.status);
parts.push(getJobStatusLabel(job.status).toLowerCase());
return parts.join(' · ');
}
@@ -144,7 +144,7 @@
<!-- Decorative waveform -->
<div class="dropzone-wave">
<Waveform bars={DROPZONE_BARS} progress={0} {ACCENT} height={38} />
<Waveform bars={DROPZONE_BARS} progress={0} accent={ACCENT} height={38} />
</div>
<input
@@ -271,7 +271,9 @@
<div class="recent-meta mono">{jobMeta(job)}</div>
</div>
{#if job.status !== 'done' && job.status !== 'failed' && job.status !== 'cancelled'}
<div class="recent-progress mono" style="color: {ACCENT}">{job.progress}%</div>
<div class="recent-progress mono" style="color: {ACCENT}">
{getDisplayJobProgress(job, { hasTranscript: Boolean(job.segmentsJson) })}%
</div>
{/if}
<svg width="14" height="14" viewBox="0 0 14 14" style="color: var(--text-dim); flex-shrink:0">
<path d="M5 3l4 4-4 4" stroke="currentColor" stroke-width="1.4" fill="none" stroke-linecap="round" stroke-linejoin="round"/>
@@ -586,4 +588,3 @@
}
}
</style>

View File

@@ -9,7 +9,7 @@ export async function GET({ params }) {
return json(job);
}
const ACTIVE = new Set(['pending', 'downloading', 'preparing', 'transcribing', 'processing']);
const ACTIVE = new Set(['pending', 'downloading', 'preparing', 'warming_model', 'transcribing', 'processing']);
export async function DELETE({ params }) {
const job = getJob(params.id);

View File

@@ -1,10 +1,9 @@
import { json, error } from '@sveltejs/kit';
import { getJob, updateJob } from '$lib/server/db.js';
import { deduplicateSegments } from '$lib/server/postprocess.js';
import { writeOutputs } from '$lib/server/formatter.js';
import type { Segment } from '$lib/types.js';
/** POST /api/jobs/[id]/reprocess — re-run post-processing and regenerate all output files. */
/** POST /api/jobs/[id]/reprocess — regenerate output files from stored canonical segments. */
export async function POST({ params }) {
const job = getJob(params.id);
if (!job) throw error(404, 'Job not found');
@@ -14,8 +13,7 @@ export async function POST({ params }) {
}
try {
const rawSegments = JSON.parse(job.segmentsJson) as Segment[];
const segments = deduplicateSegments(rawSegments);
const segments = JSON.parse(job.segmentsJson) as Segment[];
const paths = await writeOutputs(segments, job.title, job.id);
const outputDir = paths.srt.replace(/\/[^/]+$/, '');

View File

@@ -0,0 +1,13 @@
import { json } from '@sveltejs/kit';
import { unloadModel } from '$lib/server/whisper.js';
/** Proxy for POST /model/unload on the whisper backend. */
export async function POST() {
try {
const body = await unloadModel();
return json(body);
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
return json({ ok: false, error: message }, { status: 502 });
}
}

View File

@@ -1,59 +1,90 @@
import { json, error } from '@sveltejs/kit';
import { getJob, updateJob, setJobStatus } from '$lib/server/db.js';
import { deduplicateSegments } from '$lib/server/postprocess.js';
import { writeOutputs } from '$lib/server/formatter.js';
import { sendNotification } from '$lib/server/push.js';
import { cleanupJobTmp } from '$lib/server/downloader.js';
import { emitProgress } from '$lib/server/pipeline.js';
import type { Segment, WhisperJob } from '$lib/types.js';
const WHISPER_JOB_STATUSES = new Set<WhisperJob['status']>([
'queued',
'running',
'done',
'failed',
'cancelled'
]);
function isWhisperJobWebhook(payload: unknown): payload is WhisperJob {
if (!payload || typeof payload !== 'object') return false;
const candidate = payload as Record<string, unknown>;
return (
typeof candidate.id === 'string' &&
typeof candidate.status === 'string' &&
WHISPER_JOB_STATUSES.has(candidate.status as WhisperJob['status'])
);
}
export async function POST({ params, request }) {
const jobId = params.jobId;
const job = getJob(jobId);
if (!job) throw error(404, 'Job not found');
const jobId = params.jobId;
const job = getJob(jobId);
if (!job) throw error(404, 'Job not found');
// Discard the result if the job was cancelled locally while whisper was running
if (job.status === 'cancelled') {
return json({ ok: true });
}
const whisperJob = (await request.json()) as WhisperJob;
if (whisperJob.status === 'failed' || whisperJob.status === 'cancelled') {
const msg = whisperJob.error ?? `Whisper job ${whisperJob.status}`;
updateJob({ id: jobId, status: 'failed', error: msg });
emitProgress(jobId, { type: 'error', message: msg });
return json({ ok: true });
}
try {
setJobStatus(jobId, 'processing', 90);
emitProgress(jobId, { type: 'status', status: 'processing', progress: 90 });
const rawSegments = whisperJob.segments as Segment[];
const segments = deduplicateSegments(rawSegments);
const paths = await writeOutputs(segments, job.title, jobId);
const outputDir = paths.srt.replace(/\/[^/]+$/, '');
updateJob({
id: jobId,
status: 'done',
progress: 100,
segmentsJson: JSON.stringify(segments),
outputDir
});
emitProgress(jobId, { type: 'done', status: 'done' });
await sendNotification(jobId, '✅ Transcript ready', job.title);
await cleanupJobTmp(jobId);
return json({ ok: true });
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
updateJob({ id: jobId, status: 'failed', error: message });
emitProgress(jobId, { type: 'error', message });
return json({ ok: false, error: message }, { status: 500 });
}
const payload = (await request.json()) as unknown;
if (!isWhisperJobWebhook(payload)) {
// whisper-rtx2080 also fires model lifecycle events to registered job webhooks.
return json({ ok: true, ignored: 'not_a_job_event' });
}
const whisperJob = payload;
// Discard the result if the job was cancelled locally while whisper was running
if (job.status === 'cancelled') {
return json({ ok: true });
}
// Ignore stale callbacks from a previous whisper job after a local retry/reset.
if (job.whisperJobId && whisperJob.id !== job.whisperJobId) {
return json({ ok: true, ignored: 'stale_whisper_job' });
}
// Ignore replayed success callbacks after the transcript is already persisted.
if (job.status === 'done' && job.segmentsJson) {
return json({ ok: true, ignored: 'duplicate_webhook' });
}
if (whisperJob.status === 'failed' || whisperJob.status === 'cancelled') {
const msg = whisperJob.error ?? `Whisper job ${whisperJob.status}`;
updateJob({ id: jobId, status: 'failed', error: msg });
emitProgress(jobId, { type: 'error', message: msg });
return json({ ok: true });
}
try {
setJobStatus(jobId, 'processing', 90);
emitProgress(jobId, { type: 'status', status: 'processing', progress: 90 });
const segments = (whisperJob.segments ?? []) as Segment[];
const paths = await writeOutputs(segments, job.title, jobId);
const outputDir = paths.srt.replace(/\/[^/]+$/, '');
updateJob({
id: jobId,
status: 'done',
progress: 100,
segmentsJson: JSON.stringify(segments),
outputDir
});
emitProgress(jobId, { type: 'done', status: 'done' });
await sendNotification(jobId, '✅ Transcript ready', job.title);
await cleanupJobTmp(jobId);
return json({ ok: true });
} catch (err: unknown) {
const message = err instanceof Error ? err.message : String(err);
updateJob({ id: jobId, status: 'failed', error: message });
emitProgress(jobId, { type: 'error', message });
return json({ ok: false, error: message }, { status: 500 });
}
}

View File

@@ -1,6 +1,7 @@
<script lang="ts">
import { onMount } from 'svelte';
import type { Job } from '$lib/types.js';
import { getDisplayJobProgress, getJobStatusColor, getJobStatusLabel, isTerminalJobStatus } from '$lib/job-progress.js';
import SourceIcon from '$lib/components/SourceIcon.svelte';
import Waveform from '$lib/components/Waveform.svelte';
import { accent } from '$lib/accent.js';
@@ -10,27 +11,6 @@
let jobs = $state<Job[]>([]);
let loading = $state(true);
const statusColor: Record<string, string> = {
done: '#cdf24e',
failed: '#ff6b6b',
cancelled: 'rgba(232,233,236,0.3)',
transcribing: '#80c7f7',
preparing: '#fbc94b',
downloading: '#a78bfa',
pending: 'rgba(232,233,236,0.4)'
};
const statusLabel: Record<string, string> = {
pending: 'Pending',
downloading: 'Downloading',
preparing: 'Preparing',
transcribing: 'Transcribing',
processing: 'Processing',
done: 'Done',
failed: 'Failed',
cancelled: 'Cancelled'
};
function jobKind(job: Job): 'youtube' | 'audio' | 'video' | 'file' {
const s = job.source ?? '';
if (s.includes('youtube') || s.includes('youtu.be')) return 'youtube';
@@ -108,8 +88,8 @@
<div class="job-info">
<div class="job-name">{job.title || job.source}</div>
<div class="job-meta mono">
<span style="color: {statusColor[job.status] ?? 'rgba(232,233,236,0.5)'}">
{statusLabel[job.status] ?? job.status}
<span style="color: {getJobStatusColor(job.status)}">
{getJobStatusLabel(job.status)}
</span>
{#if job.createdAt}
<span>·</span>
@@ -122,14 +102,24 @@
</div>
</div>
{#if !['done', 'failed', 'cancelled'].includes(job.status)}
{#if !isTerminalJobStatus(job.status)}
<div class="job-wave">
<Waveform bars={40} progress={job.progress} accent={ACCENT} height={28} pattern="medium" />
<Waveform
bars={40}
progress={getDisplayJobProgress(job, { hasTranscript: Boolean(job.segmentsJson) })}
accent={ACCENT}
height={28}
pattern="medium"
/>
</div>
{:else if job.status === 'done'}
<div class="job-pct mono" style="color: {ACCENT}">{job.progress}%</div>
<div class="job-pct mono" style="color: {ACCENT}">
{getDisplayJobProgress(job, { hasTranscript: Boolean(job.segmentsJson) })}%
</div>
{:else}
<div class="job-pct mono" style="color: {statusColor[job.status]}">{job.status}</div>
<div class="job-pct mono" style="color: {getJobStatusColor(job.status)}">
{getJobStatusLabel(job.status)}
</div>
{/if}
<!-- Row actions -->
@@ -142,7 +132,7 @@
title="Retry"
></button>
{/if}
{#if ['done', 'failed', 'cancelled'].includes(job.status)}
{#if isTerminalJobStatus(job.status)}
<button
class="row-btn danger"
onclick={(e) => deleteJob(e, job)}

View File

@@ -2,6 +2,7 @@
import { onMount, onDestroy } from 'svelte';
import { page } from '$app/stores';
import type { Job, Segment } from '$lib/types.js';
import { getDisplayJobProgress, getJobStatusLabel, isTerminalJobStatus } from '$lib/job-progress.js';
import SourceIcon from '$lib/components/SourceIcon.svelte';
import Waveform from '$lib/components/Waveform.svelte';
import { accent } from '$lib/accent.js';
@@ -16,17 +17,6 @@
let modelWarming = $state<{ state: string; retryAfterSecs: number } | null>(null);
let eventSource: EventSource | null = null;
const statusLabel: Record<string, string> = {
pending: 'Pending',
downloading: 'Downloading…',
preparing: 'Preparing audio…',
transcribing: 'Transcribing…',
processing: 'Post-processing…',
done: 'Done',
failed: 'Failed',
cancelled: 'Cancelled'
};
// Pipeline stages derived from job status
const pipelineStages = $derived.by(() => {
const status = job?.status ?? 'pending';
@@ -34,16 +24,26 @@
{ k: 'fetch', label: 'Fetch source' },
{ k: 'extract', label: 'Extract audio track' },
{ k: 'process', label: `Audio processing · ${job?.audioMode ?? 'auto'}` },
{ k: 'transcribe', label: 'Transcribing' },
{ k: 'transcribe', label: status === 'warming_model' ? 'Loading model' : 'Transcribing' },
{ k: 'finalize', label: 'Format &amp; save' }
];
const order = ['pending', 'downloading', 'preparing', 'transcribing', 'processing', 'done'];
const idx = order.indexOf(status);
const stageIndex = {
pending: 0,
downloading: 1,
preparing: 2,
warming_model: 3,
transcribing: 3,
processing: 4,
done: 5,
failed: -1,
cancelled: -1
}[status];
return stages.map((s, i) => ({
...s,
done: i < idx - 1 || status === 'done',
active: i === idx - 1 && status !== 'done' && status !== 'failed',
pending: i > idx - 1 && status !== 'done'
done: status === 'done' || i + 1 < stageIndex,
active: i + 1 === stageIndex && !isTerminalJobStatus(status),
pending: status !== 'done' && i + 1 > stageIndex
}));
});
@@ -57,7 +57,7 @@
onMount(async () => {
await loadJob();
if (job && !['done', 'failed', 'cancelled'].includes(job.status)) {
if (job && !isTerminalJobStatus(job.status)) {
openStream();
}
});
@@ -71,6 +71,7 @@
return;
}
job = await res.json();
segments = [];
if (job?.segmentsJson) {
try {
segments = JSON.parse(job.segmentsJson);
@@ -89,9 +90,14 @@
if (job) job = { ...job, progress: data.progress ?? job.progress, status: 'transcribing' };
} else if (data.type === 'model_warming') {
modelWarming = { state: data.state ?? 'loading', retryAfterSecs: data.retryAfterSecs ?? 30 };
chunkInfo = { chunk: 0, total: 0 };
if (job) job = { ...job, status: 'warming_model', progress: data.progress ?? job.progress };
} else if (data.type === 'status') {
if (data.status !== 'warming_model') modelWarming = null;
if (data.status !== 'transcribing') chunkInfo = { chunk: 0, total: 0 };
if (job) job = { ...job, status: data.status, progress: data.progress ?? job.progress };
} else if (data.type === 'done') {
modelWarming = null;
eventSource?.close();
loadJob();
} else if (data.type === 'error') {
@@ -112,8 +118,21 @@
}
const formats = ['srt', 'txt', 'md', 'json'] as const;
const isActive = $derived(!job || !['done', 'failed', 'cancelled'].includes(job.status));
const isTerminal = $derived(job !== null && ['done', 'failed', 'cancelled'].includes(job.status));
const hasTranscript = $derived((job?.segmentsJson ? true : false) || segments.length > 0);
const displayProgress = $derived(job ? getDisplayJobProgress(job, { hasTranscript }) : 0);
const progressStatusLabel = $derived.by(() => {
if (!job) return 'Pending';
if (job.status === 'warming_model') {
const state = modelWarming?.state?.replace(/_/g, ' ');
return state ? `Loading model (${state})…` : 'Loading model…';
}
if (job.status === 'preparing') return 'Preparing audio…';
if (job.status === 'processing') return 'Saving transcript…';
if (job.status === 'transcribing') return 'Transcribing…';
return job.status === 'done' ? 'Done' : `${getJobStatusLabel(job.status)}…`;
});
const isActive = $derived(!job || !isTerminalJobStatus(job.status));
const isTerminal = $derived(job !== null && isTerminalJobStatus(job.status));
const canRetry = $derived(
job !== null &&
['failed', 'cancelled'].includes(job.status) &&
@@ -198,7 +217,7 @@
<div class="progress-wave">
<Waveform
bars={140}
progress={job.progress}
progress={displayProgress}
accent={ACCENT}
height={80}
pattern="default"
@@ -208,9 +227,9 @@
<div class="progress-footer">
<div class="progress-left">
<span class="progress-pct mono">
{job.progress}<span style="color: var(--text-dim); font-weight: 400">%</span>
{displayProgress}<span style="color: var(--text-dim); font-weight: 400">%</span>
</span>
<span class="progress-status">{statusLabel[job.status] ?? job.status}</span>
<span class="progress-status">{progressStatusLabel}</span>
</div>
{#if chunkInfo.total > 0}
<span class="progress-chunks mono">
@@ -232,7 +251,7 @@
<div class="progress-bar-track">
<div
class="progress-bar-fill"
style="width: {job.progress}%; background: {ACCENT}; box-shadow: 0 0 12px {ACCENT}80;"
style="width: {displayProgress}%; background: {ACCENT}; box-shadow: 0 0 12px {ACCENT}80;"
></div>
</div>
</div>
@@ -274,7 +293,7 @@
{@html stage.label}
</span>
{#if stage.active}
<span class="mono" style="font-size: 11.5px; color: {ACCENT}">{job.progress}%</span>
<span class="mono" style="font-size: 11.5px; color: {ACCENT}">{displayProgress}%</span>
{/if}
</div>
{/each}
@@ -689,4 +708,3 @@
}
}
</style>

View File

@@ -7,17 +7,18 @@ import { describe, it, expect, vi, afterEach } from 'vitest';
const execFileMock = vi.hoisted(() => {
const fn = vi.fn();
type ExecFilePromisifyArgs = [string, string[]];
type ExecFileCallback = (err: Error | null, stdout: string, stderr: string) => void;
type ExecFileMock = (...args: [...ExecFilePromisifyArgs, ExecFileCallback]) => void;
const invoke = fn as unknown as ExecFileMock;
Object.defineProperty(fn, Symbol.for('nodejs.util.promisify.custom'), {
configurable: true,
value: (...args: unknown[]) =>
value: (...args: ExecFilePromisifyArgs) =>
new Promise<{ stdout: string; stderr: string }>((resolve, reject) => {
(fn as ReturnType<typeof vi.fn>)(
...args,
(err: Error | null, stdout: string, stderr: string) => {
invoke(...args, (err: Error | null, stdout: string, stderr: string) => {
if (err) reject(err);
else resolve({ stdout, stderr });
}
);
});
})
});
return fn;

View File

@@ -123,7 +123,7 @@ describe('setJobStatus', () => {
it('transitions through all valid statuses', () => {
const job = createJob('src', 'title', 'auto');
const statuses = ['downloading', 'preparing', 'transcribing', 'processing', 'done'] as const;
const statuses = ['downloading', 'preparing', 'warming_model', 'transcribing', 'processing', 'done'] as const;
for (const status of statuses) {
setJobStatus(job.id, status, 50);
expect(getJob(job.id)!.status).toBe(status);

View File

@@ -0,0 +1,80 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { rm } from 'fs/promises';
import type { TranscriptResponse } from 'youtube-transcript';
const { mockExecFile, mockFetchTranscript } = vi.hoisted(() => ({
mockExecFile: vi.fn(),
mockFetchTranscript: vi.fn()
}));
const TEST_DATA_DIR = `/tmp/tonemark-downloader-test-${Date.now()}`;
vi.stubEnv('DATA_DIR', TEST_DATA_DIR);
vi.mock('child_process', () => ({
execFile: mockExecFile
}));
vi.mock('youtube-transcript', () => ({
fetchTranscript: mockFetchTranscript
}));
import { downloadYouTube, transcriptEntriesToSegments } from '$lib/server/downloader.js';
beforeEach(() => {
vi.clearAllMocks();
mockExecFile.mockImplementation((...args: unknown[]) => {
const cb = args.at(-1) as (...callbackArgs: unknown[]) => void;
cb(null, JSON.stringify({ title: 'Fetched Title' }), '');
});
});
afterEach(async () => {
await rm(TEST_DATA_DIR, { recursive: true, force: true }).catch(() => {});
});
describe('transcriptEntriesToSegments', () => {
it('converts millisecond transcript offsets into second-based segments', () => {
const entries: TranscriptResponse[] = [
{ text: 'Hello everyone.', offset: 15240, duration: 4240, lang: 'en' },
{ text: 'Um, welcome to this talk.', offset: 16600, duration: 5080, lang: 'en' }
];
expect(transcriptEntriesToSegments(entries)).toEqual([
{ index: 0, start: 15.24, end: 19.48, text: 'Hello everyone.', words: [] },
{ index: 1, start: 16.6, end: 21.68, text: 'Um, welcome to this talk.', words: [] }
]);
});
it('preserves second-based transcript offsets and drops empty text', () => {
const entries: TranscriptResponse[] = [
{ text: ' ', offset: 0, duration: 1.5, lang: 'en' },
{ text: 'Clean caption cue', offset: 91.08, duration: 3.72, lang: 'en' }
];
expect(transcriptEntriesToSegments(entries)).toEqual([
{ index: 0, start: 91.08, end: 94.8, text: 'Clean caption cue', words: [] }
]);
});
});
describe('downloadYouTube', () => {
it('uses fetched transcript entries directly for caption jobs', async () => {
mockFetchTranscript.mockResolvedValue([
{ text: 'Hello everyone.', offset: 15240, duration: 4240, lang: 'en' },
{ text: 'Um, welcome to this talk.', offset: 16600, duration: 5080, lang: 'en' }
] satisfies TranscriptResponse[]);
const result = await downloadYouTube('https://youtube.com/watch?v=qdh_x-uRs9g', 'job-1');
expect(mockFetchTranscript).toHaveBeenCalledWith('https://youtube.com/watch?v=qdh_x-uRs9g', {
lang: 'en'
});
expect(result).toMatchObject({
type: 'captions',
segments: [
{ index: 0, start: 15.24, end: 19.48, text: 'Hello everyone.', words: [] },
{ index: 1, start: 16.6, end: 21.68, text: 'Um, welcome to this talk.', words: [] }
]
});
});
});

View File

@@ -0,0 +1,49 @@
import { describe, expect, it } from 'vitest';
import { getDisplayJobProgress, getJobStatusLabel, isTerminalJobStatus } from '$lib/job-progress.js';
import type { Job } from '$lib/types.js';
function makeJob(overrides: Partial<Job> = {}): Job {
return {
id: 'job-1',
status: 'transcribing',
title: 'Job',
source: 'https://example.com/audio.mp3',
audioMode: 'auto',
meanVolume: null,
whisperJobId: null,
progress: 42,
outputDir: null,
segmentsJson: null,
error: null,
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
...overrides
};
}
describe('job progress helpers', () => {
it('keeps active jobs below 100 percent', () => {
expect(getDisplayJobProgress(makeJob({ status: 'transcribing', progress: 100 }))).toBe(99);
});
it('keeps model loading in an early progress band', () => {
expect(getDisplayJobProgress(makeJob({ status: 'warming_model', progress: 80 }))).toBe(15);
});
it('allows 100 percent once finished job has transcript payload', () => {
expect(
getDisplayJobProgress(makeJob({ status: 'done', progress: 100, segmentsJson: JSON.stringify([]) }), {
hasTranscript: true
})
).toBe(100);
});
it('holds done jobs below 100 percent until transcript data exists', () => {
expect(getDisplayJobProgress(makeJob({ status: 'done', progress: 100 }))).toBe(99);
});
it('exposes model-loading label as active state', () => {
expect(getJobStatusLabel('warming_model')).toBe('Loading model');
expect(isTerminalJobStatus('warming_model')).toBe(false);
});
});

View File

@@ -1,127 +0,0 @@
import { describe, it, expect } from 'vitest';
import {
deduplicateSegments
} from '$lib/server/postprocess.js';
import type { Segment } from '$lib/types.js';
// ── helpers ──────────────────────────────────────────────────────────────────
function seg(index: number, start: number, end: number, text: string): Segment {
return { index, start, end, text, words: [] };
}
// ── collapseRepeats (tested indirectly via deduplicateSegments) ───────────────
describe('deduplicateSegments — collapseRepeats', () => {
it('leaves text without repetition unchanged', () => {
const input = [seg(0, 0, 5, ' Hello world, this is a sentence.')];
const [out] = deduplicateSegments(input);
expect(out.text).toBe('Hello world, this is a sentence.');
});
it('collapses a consecutive repeated phrase inside a segment', () => {
const input = [seg(0, 0, 5, ' the quick brown fox the quick brown fox')];
const [out] = deduplicateSegments(input);
expect(out.text).not.toMatch(/the quick brown fox.*the quick brown fox/i);
});
it('handles multiple repetitions recursively', () => {
// "welcome everyone" = 16 chars — qualifies for the ≥10-char collapse regex
const input = [seg(0, 0, 5, ' welcome everyone welcome everyone welcome everyone')];
const result = deduplicateSegments(input);
const text = result[0]?.text ?? '';
expect((text.match(/welcome everyone/gi) ?? []).length).toBeLessThan(3);
});
});
// ── mergeConsecutive ──────────────────────────────────────────────────────────
describe('deduplicateSegments — mergeConsecutive', () => {
it('merges adjacent segments with identical text', () => {
const input = [
seg(0, 0, 2, ' Hello world.'),
seg(1, 2, 4, ' Hello world.')
];
const result = deduplicateSegments(input);
expect(result).toHaveLength(1);
expect(result[0].end).toBe(4);
});
it('keeps adjacent segments with different text', () => {
const input = [
seg(0, 0, 2, ' First sentence.'),
seg(1, 2, 4, ' Second sentence.')
];
const result = deduplicateSegments(input);
expect(result).toHaveLength(2);
});
it('normalises punctuation and case for merge comparison', () => {
const input = [
seg(0, 0, 2, ' Hello, World!'),
seg(1, 2, 4, ' hello world')
];
const result = deduplicateSegments(input);
expect(result).toHaveLength(1);
});
});
// ── ngramDedup ────────────────────────────────────────────────────────────────
describe('deduplicateSegments — ngramDedup', () => {
it('passes through completely unique segments', () => {
const input = [
seg(0, 0, 5, ' The cat sat on the mat quite happily today.'),
seg(1, 5, 10, ' Later the dog ran across the yard chasing a ball.')
];
expect(deduplicateSegments(input)).toHaveLength(2);
});
it('removes a segment that is highly similar to recent context', () => {
// Repeat a long sentence verbatim — should be caught as duplicate
const longText =
' This is a very specific and unique sentence about transcription quality matters greatly.';
const input = [seg(0, 0, 5, longText), seg(1, 5, 10, longText)];
// After mergeConsecutive the second one is already merged, so result is 1
expect(deduplicateSegments(input)).toHaveLength(1);
});
});
// ── deduplicateSegments — full pipeline ──────────────────────────────────────
describe('deduplicateSegments — full pipeline', () => {
it('returns empty array for empty input', () => {
expect(deduplicateSegments([])).toEqual([]);
});
it('removes segments whose text is empty after trimming', () => {
const input = [seg(0, 0, 1, ' '), seg(1, 1, 2, ' Hello.')];
const result = deduplicateSegments(input);
expect(result).toHaveLength(1);
expect(result[0].text).toBe('Hello.');
});
it('re-indexes output segments starting from 0', () => {
const input = [
seg(5, 0, 2, ' First unique sentence here.'),
seg(8, 2, 4, ' Second different sentence there.')
];
const result = deduplicateSegments(input);
result.forEach((s, i) => expect(s.index).toBe(i));
});
it('runs the full pipeline: trim → remove empty → merge → ngram → merge → reindex', () => {
const input = [
seg(0, 0, 2, ' Good morning everyone.'),
seg(1, 2, 3, ' '), // empty — removed
seg(2, 3, 5, ' Good morning everyone.'), // duplicate — merged
seg(3, 5, 7, ' Welcome to our presentation today.')
];
const result = deduplicateSegments(input);
expect(result).toHaveLength(2);
expect(result[0].text).toBe('Good morning everyone.');
expect(result[1].text).toBe('Welcome to our presentation today.');
expect(result[0].index).toBe(0);
expect(result[1].index).toBe(1);
});
});

View File

@@ -1,4 +1,4 @@
import { describe, it, expect, vi, afterEach } from 'vitest';
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
// ── Hoist mock functions so they're available inside vi.mock() factories ───────
const { mockSetVapidDetails, mockWebPushSend } = vi.hoisted(() => ({
@@ -24,12 +24,16 @@ import { sendNotification, getVapidPublicKey } from '$lib/server/push.js';
import { savePushSubscription, deletePushSubscription, getAllSubscriptions } from '$lib/server/db.js';
import { rm } from 'fs/promises';
afterEach(async () => {
mockSetVapidDetails.mockReset();
beforeEach(() => {
// Ensure a clean subscription table before each test
for (const s of getAllSubscriptions()) deletePushSubscription(s.endpoint);
mockWebPushSend.mockReset();
mockSetVapidDetails.mockReset();
});
afterEach(async () => {
// Remove all test subscriptions between tests
const subs = getAllSubscriptions();
for (const s of subs) deletePushSubscription(s.endpoint);
for (const s of getAllSubscriptions()) deletePushSubscription(s.endpoint);
await rm(TEST_DATA_DIR, { recursive: true, force: true }).catch(() => {});
});
@@ -134,6 +138,15 @@ describe('sendNotification', () => {
.mockResolvedValueOnce({});
await sendNotification('job-8', 'title', 'body');
expect(mockWebPushSend).toHaveBeenCalledTimes(3);
const calledEndpoints = mockWebPushSend.mock.calls.map(
([sub]) => (sub as { endpoint: string }).endpoint
);
expect(calledEndpoints).toEqual(
expect.arrayContaining([
'https://fcm.example.com/push/ok1',
'https://fcm.example.com/push/fail',
'https://fcm.example.com/push/ok2'
])
);
});
});

View File

@@ -7,7 +7,6 @@ const {
mockGetJob,
mockUpdateJob,
mockSetJobStatus,
mockDeduplicateSegments,
mockWriteOutputs,
mockSendNotification,
mockCleanupJobTmp,
@@ -16,7 +15,6 @@ const {
mockGetJob: vi.fn(),
mockUpdateJob: vi.fn(),
mockSetJobStatus: vi.fn(),
mockDeduplicateSegments: vi.fn((segs: Segment[]) => segs),
mockWriteOutputs: vi.fn(),
mockSendNotification: vi.fn(),
mockCleanupJobTmp: vi.fn(),
@@ -29,10 +27,6 @@ vi.mock('$lib/server/db.js', () => ({
setJobStatus: mockSetJobStatus
}));
vi.mock('$lib/server/postprocess.js', () => ({
deduplicateSegments: mockDeduplicateSegments
}));
vi.mock('$lib/server/formatter.js', () => ({
writeOutputs: mockWriteOutputs
}));
@@ -91,7 +85,6 @@ function makeSeg(index: number, text: string): Segment {
beforeEach(() => {
vi.clearAllMocks();
mockDeduplicateSegments.mockImplementation((segs: Segment[]) => segs);
mockWriteOutputs.mockResolvedValue({
srt: '/out/dir/title.srt',
txt: '/out/dir/title.txt',
@@ -113,6 +106,64 @@ describe('POST /api/webhook/[jobId] — job not found', () => {
});
});
// ── Ignore backend model lifecycle webhooks ─────────────────────────────────────
describe('POST /api/webhook/[jobId] — non-job webhook payloads', () => {
it('ignores model_ready events sent to job webhooks', async () => {
mockGetJob.mockReturnValue(makeJob('job-model-ready'));
const res = await POST(
makeEvent('job-model-ready', {
type: 'model_ready',
loaded_at: new Date().toISOString()
}) as any
);
expect(res.status).toBe(200);
expect(await res.json()).toEqual({ ok: true, ignored: 'not_a_job_event' });
expect(mockSetJobStatus).not.toHaveBeenCalled();
expect(mockUpdateJob).not.toHaveBeenCalled();
expect(mockWriteOutputs).not.toHaveBeenCalled();
expect(mockSendNotification).not.toHaveBeenCalled();
});
it('ignores model_unloaded events sent to job webhooks', async () => {
mockGetJob.mockReturnValue(makeJob('job-model-unloaded'));
const res = await POST(
makeEvent('job-model-unloaded', {
type: 'model_unloaded',
unloaded_at: new Date().toISOString()
}) as any
);
expect(res.status).toBe(200);
expect(await res.json()).toEqual({ ok: true, ignored: 'not_a_job_event' });
expect(mockSetJobStatus).not.toHaveBeenCalled();
expect(mockUpdateJob).not.toHaveBeenCalled();
expect(mockWriteOutputs).not.toHaveBeenCalled();
expect(mockSendNotification).not.toHaveBeenCalled();
});
it('ignores payloads with invalid status values', async () => {
mockGetJob.mockReturnValue(makeJob('job-invalid-status'));
const res = await POST(
makeEvent('job-invalid-status', {
id: 'bogus-whisper-id',
status: 'model_ready',
segments: []
}) as any
);
expect(res.status).toBe(200);
expect(await res.json()).toEqual({ ok: true, ignored: 'not_a_job_event' });
expect(mockSetJobStatus).not.toHaveBeenCalled();
expect(mockUpdateJob).not.toHaveBeenCalled();
expect(mockWriteOutputs).not.toHaveBeenCalled();
});
});
// ── Local cancellation guard ──────────────────────────────────────────────────
describe('POST /api/webhook/[jobId] — locally cancelled job', () => {
@@ -132,6 +183,43 @@ describe('POST /api/webhook/[jobId] — locally cancelled job', () => {
});
});
// ── Duplicate / stale callback guards ──────────────────────────────────────────
describe('POST /api/webhook/[jobId] — duplicate and stale callbacks', () => {
it('ignores replayed success callbacks after the transcript is already done', async () => {
mockGetJob.mockReturnValue({
...makeJob('job-done'),
status: 'done',
segmentsJson: JSON.stringify([makeSeg(0, 'Already saved.')]),
whisperJobId: 'whisper-id'
});
const res = await POST(makeEvent('job-done', makeWhisperJob()) as any);
expect(res.status).toBe(200);
expect(await res.json()).toEqual({ ok: true, ignored: 'duplicate_webhook' });
expect(mockSetJobStatus).not.toHaveBeenCalled();
expect(mockUpdateJob).not.toHaveBeenCalled();
expect(mockWriteOutputs).not.toHaveBeenCalled();
});
it('ignores stale callbacks from an older whisper job after retry', async () => {
mockGetJob.mockReturnValue({
...makeJob('job-stale'),
status: 'transcribing',
whisperJobId: 'current-whisper-job'
});
const res = await POST(
makeEvent('job-stale', makeWhisperJob({ id: 'old-whisper-job', segments: [makeSeg(0, 'stale')] })) as any
);
expect(res.status).toBe(200);
expect(await res.json()).toEqual({ ok: true, ignored: 'stale_whisper_job' });
expect(mockSetJobStatus).not.toHaveBeenCalled();
expect(mockUpdateJob).not.toHaveBeenCalled();
expect(mockWriteOutputs).not.toHaveBeenCalled();
});
});
// ── Whisper job failed / cancelled ───────────────────────────────────────────
describe('POST /api/webhook/[jobId] — whisper failure', () => {
@@ -181,25 +269,21 @@ describe('POST /api/webhook/[jobId] — whisper failure', () => {
describe('POST /api/webhook/[jobId] — success with segments', () => {
const segments = [makeSeg(0, 'Hello world.'), makeSeg(1, 'This is a test.')];
it('runs deduplication on received segments', async () => {
it('passes received segments through unchanged', async () => {
mockGetJob.mockReturnValue(makeJob('job-3'));
await POST(makeEvent('job-3', makeWhisperJob({ segments })) as any);
expect(mockDeduplicateSegments).toHaveBeenCalledWith(segments);
expect(mockWriteOutputs).toHaveBeenCalledWith(segments, 'Test Video', 'job-3');
});
it('calls writeOutputs with the deduplicated segments and job title', async () => {
it('calls writeOutputs with the received segments and job title', async () => {
mockGetJob.mockReturnValue(makeJob('job-4', 'My Lecture'));
const deduped = [makeSeg(0, 'Hello world.')];
mockDeduplicateSegments.mockReturnValue(deduped);
await POST(makeEvent('job-4', makeWhisperJob({ segments })) as any);
expect(mockWriteOutputs).toHaveBeenCalledWith(deduped, 'My Lecture', 'job-4');
expect(mockWriteOutputs).toHaveBeenCalledWith(segments, 'My Lecture', 'job-4');
});
it('stores serialised segments_json in the database', async () => {
mockGetJob.mockReturnValue(makeJob('job-5'));
const deduped = [makeSeg(0, 'Result text.')];
mockDeduplicateSegments.mockReturnValue(deduped);
await POST(makeEvent('job-5', makeWhisperJob({ segments })) as any);
@@ -207,7 +291,7 @@ describe('POST /api/webhook/[jobId] — success with segments', () => {
expect.objectContaining({
id: 'job-5',
status: 'done',
segmentsJson: JSON.stringify(deduped)
segmentsJson: JSON.stringify(segments)
})
);
});
@@ -289,6 +373,34 @@ describe('POST /api/webhook/[jobId] — empty segments', () => {
});
});
// ── Undefined / missing segments (model returned no segments field) ───────────
describe('POST /api/webhook/[jobId] — undefined segments', () => {
it('completes the job as done when segments field is absent from whisper payload', async () => {
mockGetJob.mockReturnValue(makeJob('job-noseg'));
// Simulate whisper returning a result without a segments field
const payload = { ...makeWhisperJob(), segments: undefined as unknown as never[] };
const res = await POST(makeEvent('job-noseg', payload) as any);
expect(res.status).toBe(200);
expect(await res.json()).toEqual({ ok: true });
expect(mockUpdateJob).toHaveBeenCalledWith(
expect.objectContaining({ status: 'done', id: 'job-noseg' })
);
});
it('does not throw "cannot read properties of undefined" when segments is null', async () => {
mockGetJob.mockReturnValue(makeJob('job-nullseg'));
const payload = { ...makeWhisperJob(), segments: null as unknown as never[] };
// Must NOT throw — previously crashed with "Cannot read properties of undefined (reading 'map')"
await expect(POST(makeEvent('job-nullseg', payload) as any)).resolves.toBeDefined();
expect(mockUpdateJob).toHaveBeenCalledWith(
expect.objectContaining({ status: 'done' })
);
});
});
// ── Internal error handling ───────────────────────────────────────────────────
describe('POST /api/webhook/[jobId] — internal errors', () => {

View File

@@ -6,7 +6,8 @@ import { Readable } from 'stream';
const mocks = vi.hoisted(() => ({
fetch: vi.fn(),
append: vi.fn(),
getHeaders: vi.fn(() => ({ 'content-type': 'multipart/form-data; boundary=test' }))
getHeaders: vi.fn(() => ({ 'content-type': 'multipart/form-data; boundary=test' })),
createReadStream: vi.fn(() => 'STREAM_PLACEHOLDER')
}));
vi.mock('node-fetch', () => ({ default: mocks.fetch }));
@@ -19,9 +20,9 @@ vi.mock('form-data', () => ({
})
}));
vi.mock('fs', () => ({ createReadStream: vi.fn(() => 'STREAM_PLACEHOLDER') }));
vi.mock('fs', () => ({ createReadStream: mocks.createReadStream }));
import { submitJob, streamJob, getModelStatus, cancelJob } from '$lib/server/whisper.js';
import { submitJob, streamJob, getModelStatus, cancelJob, unloadModel } from '$lib/server/whisper.js';
afterEach(() => vi.clearAllMocks());
@@ -255,6 +256,20 @@ describe('submitJob — 503 retry behavior', () => {
expect(onModelWaiting).not.toHaveBeenCalled();
});
it('creates a fresh ReadStream for each attempt (stream not reused across retries)', async () => {
mocks.fetch.mockImplementation(
makeJobFetch(make503('loading', 0), make503('loading', 0), make202('fresh-stream-id'))
);
const id = await submitJob('/tmp/audio.wav', 'http://host/webhook', undefined, undefined, 10);
expect(id).toBe('fresh-stream-id');
// 3 attempts → 3 separate createReadStream calls, one fresh stream per form
expect(mocks.createReadStream).toHaveBeenCalledTimes(3);
expect(mocks.createReadStream).toHaveBeenNthCalledWith(1, '/tmp/audio.wav');
expect(mocks.createReadStream).toHaveBeenNthCalledWith(2, '/tmp/audio.wav');
expect(mocks.createReadStream).toHaveBeenNthCalledWith(3, '/tmp/audio.wav');
});
it('does NOT retry on non-503 errors (throws immediately)', async () => {
mocks.fetch.mockResolvedValue({
status: 400,
@@ -339,6 +354,68 @@ describe('submitJob — SSE-triggered retry', () => {
const id = await submitJob('/tmp/audio.wav', 'http://host/webhook');
expect(id).toBe('stream-closed-id');
});
it('relays intermediate model states from /model/events while waiting to retry', async () => {
let jobCallIdx = 0;
mocks.fetch.mockImplementation((url: string) => {
if (String(url).includes('/model/events')) {
return Promise.resolve({
ok: true,
status: 200,
body: Readable.from([
'data: {"state":"loading"}\n\ndata: {"state":"waiting_for_gpu"}\n\ndata: {"state":"ready"}\n\n'
])
});
}
jobCallIdx++;
if (jobCallIdx === 1) return Promise.resolve(make503('unloaded', 30));
return Promise.resolve(make202('state-relay-id'));
});
const onModelWaiting = vi.fn();
const id = await submitJob('/tmp/audio.wav', 'http://host/webhook', undefined, onModelWaiting);
expect(id).toBe('state-relay-id');
expect(onModelWaiting).toHaveBeenNthCalledWith(1, 'unloaded', 30);
expect(onModelWaiting).toHaveBeenNthCalledWith(2, 'loading', 30);
expect(onModelWaiting).toHaveBeenNthCalledWith(3, 'waiting_for_gpu', 30);
expect(onModelWaiting).toHaveBeenCalledTimes(3);
});
});
// ── unloadModel ───────────────────────────────────────────────────────────────
describe('unloadModel', () => {
it('POSTs to /model/unload and returns parsed body', async () => {
mocks.fetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ ok: true })
});
const result = await unloadModel();
expect(result).toEqual({ ok: true });
expect(mocks.fetch).toHaveBeenCalledWith(
expect.stringContaining('/model/unload'),
expect.objectContaining({ method: 'POST' })
);
});
it('uses the configured WHISPER_URL', async () => {
vi.stubEnv('WHISPER_URL', 'http://gpu-box:9090');
mocks.fetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ ok: true })
});
await unloadModel();
expect(mocks.fetch).toHaveBeenCalledWith(
'http://gpu-box:9090/model/unload',
expect.anything()
);
vi.unstubAllEnvs();
});
it('throws when whisper returns a non-ok response', async () => {
mocks.fetch.mockResolvedValue({ ok: false, status: 409 });
await expect(unloadModel()).rejects.toThrow('/model/unload');
});
});
// ── cancelJob ─────────────────────────────────────────────────────────────────
@@ -458,6 +535,10 @@ function makeSSEResponse(lines: string[]) {
return { ok: true, body };
}
function makeSSEChunkResponse(chunks: string[]) {
return { ok: true, body: Readable.from(chunks) };
}
describe('streamJob — SSE event parsing', () => {
it('calls onProgress for progress events with percent, chunk, total', async () => {
const onProgress = vi.fn();
@@ -551,6 +632,27 @@ describe('streamJob — SSE event parsing', () => {
expect(onProgress).toHaveBeenNthCalledWith(3, 75, 3, 4);
});
it('handles multiple SSE events delivered in a single chunk', async () => {
const onProgress = vi.fn();
const onDone = vi.fn();
const onError = vi.fn();
mocks.fetch.mockResolvedValue(
makeSSEChunkResponse([
'data: {"type":"progress","percent":25,"chunk":1,"total":2}\n\n' +
'data: {"type":"progress","percent":50,"chunk":2,"total":2}\n\n' +
'data: {"type":"done","job":{}}\n\n'
])
);
await streamJob('whisper-id', onProgress, onDone, onError);
expect(onProgress).toHaveBeenCalledTimes(2);
expect(onProgress).toHaveBeenNthCalledWith(1, 25, 1, 2);
expect(onProgress).toHaveBeenNthCalledWith(2, 50, 2, 2);
expect(onDone).toHaveBeenCalledOnce();
expect(onError).not.toHaveBeenCalled();
});
it('defaults chunk and total to 0 when missing from progress event', async () => {
const onProgress = vi.fn();
const onDone = vi.fn();

View File

@@ -5,6 +5,7 @@ export default defineConfig({
test: {
environment: 'node',
globals: true,
fileParallelism: false,
include: ['src/tests/**/*.test.ts'],
coverage: {
provider: 'v8',