Create src/lib/server/data-dir.ts as single source of truth for DATA_DIR
using db.ts's $HOME-based convention. All four modules now import from it
instead of defining their own inline default (some resolving to /tmp/.whisper-pwa).
Changes:
- src/lib/server/data-dir.ts — new file, exports DATA_DIR const
- db.ts — import DATA_DIR from data-dir.ts (same logic, same path)
- audio.ts — import DATA_DIR, replace inline process.env.DATA_DIR ?? '/tmp/...'
- downloader.ts — import DATA_DIR, replace inline '/tmp/.whisper-pwa'
- remux.ts — import DATA_DIR, replace inline process.env.DATA_DIR ?? '/tmp/...'
- tests/data-dir.test.ts — 10 tests covering default, custom, and module integrity
- tests/retry-pipeline.test.ts — fix stale hardcoded /tmp/.whisper-pwa/uploads
path; compute parent dir from getUploadPath() dynamically
Test pass rate: 217/220 (3 pre-existing failures unrelated to DATA_DIR)
- 2 D5 downloader tests need yt-dlp (not available in test env)
- All other tests green, including the new data-dir.test.ts (10/10)
When youtube-transcript fetchTranscript() returns empty segments array
(parse succeeded but zero captions), treat as fallback trigger instead of
returning empty transcript.
- tryGetCaptions(): return CaptionResult with empty segments (not null)
when segments array is empty. Distinguishes 'no captions available'
(null) from 'captions exist but empty' (CaptionResult with [])
- downloadYouTube(): check captions.segments.length > 0 before taking
fast path. Empty segments fall through to audio download + whisper
- Tests: empty array fallback, whitespace-only entries fallback,
non-empty segments still use fast path
Add health preflight check in runJob() after ensureWhisperRunning()
and before submitJob(). Uses existing getHealth() from whisper.ts
with retryWithBackoff() for transient failures.
- Healthy whisper (status 'ok') -> job proceeds to submission
- Unreachable whisper -> job fails early with 'whisper health check failed: ...'
- Non-ok status (e.g. 'starting') -> job fails early with actual status
- getHealth() 5000ms timeout preserved, no new HTTP client
Tests: health-preflight.test.ts covers error message contracts
and pipeline integration (YouTube + upload source)
retryJob() now wraps the file read in a fire-and-forget async IIFE using
fs/promises.readFile instead of readFileSync. When the original upload
file has been cleaned up (common after failure), the IIFE catches the
ENOENT error and marks the job as 'failed' with a 'retry failed:' prefix
asynchronously — the endpoint returns 200 immediately.
Includes 4 new integration tests:
- Missing file returns immediately (no throw), job async-fails
- ENOENT message propagated in job.error
- 404 on unknown job still works (unchanged contract)
- File present resets job to pending (runJob triggered)
- Remove http-only guard in retry endpoint (src/routes/api/jobs/[id]/retry/+server.ts)
- Extend pipeline.ts retryJob() to handle upload-source jobs:
read file from persistent storage, pass buffer to runJob
- Change saveUploadedFile() to save to DATA_DIR/uploads/{jobId}/
(survives cleanupJobTmp, available for retry on failure)
- Add getUploadPath(), cleanupUploadDir() to downloader.ts
- Remove cleanupFiles(rawAudioPath) from catch block in runJob()
— cleanupJobTmp handles TMP_DIR for YouTube, uploads survive
- Add cleanupUploadDir() on webhook success path
- Add retry-endpoint.test.ts with 9 tests covering:
- 404 for unknown job
- 409 for non-retryable status (done, pending)
- 200 for cancelled (retryable)
- 200 for failed YouTube job (existing behavior)
- 200 for failed upload-source job (new behavior, .webm and .mp3)
- Negative: retryJob not called on 404/409
- Add downloader.test.ts tests for saveUploadedFile persistent path,
getUploadPath, cleanupUploadDir, isolation, and non-existent cleanup
- Fix webhook.test.ts mock to include cleanupUploadDir
All 189 tests pass across 12 test files.
- Ignore backend model lifecycle webhooks so model warmup does not
mark jobs done early
- Parse batched SSE messages and relay model load states during
submit retries
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove residual one-word suffix-prefix carry-over between adjacent caption segments so reprocessed transcripts no longer repeat bridge words across lines.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collapse one-word and very short caption carry-over fragments so reprocessed YouTube transcripts do not retain residual prefix chains.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Normalize incremental backend hypothesis chains before persistence and ignore stale or replayed webhook callbacks so duplicate transcript text does not survive ingest.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two bugs triggered together when the model was unloaded during a job:
1. submitJob() created FormData/createReadStream once outside the retry loop.
After a 503, the audio ReadStream was consumed and subsequent retries sent
an empty body to whisper, causing it to return segments:undefined.
2. webhook handler cast whisperJob.segments as Segment[] without guarding
against undefined, so deduplicateSegments(undefined) crashed with
'Cannot read properties of undefined (reading 'map')' — stored as job.error.
Fixes:
- Move FormData + createReadStream inside the retry loop (fresh stream per attempt)
- Use (whisperJob.segments ?? []) in webhook handler
- Add Array.isArray guard at top of deduplicateSegments() as belt-and-suspenders
Tests:
- New: verifies createReadStream called once per attempt (3 attempts = 3 streams)
- New: webhook handles segments:undefined without throwing
- New: webhook handles segments:null without throwing
- 150/150 passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add cancelJob() to whisper.ts: sends DELETE /jobs/:id to the whisper
server (best-effort, errors silently ignored)
- DELETE /api/jobs/[id] now calls cancelJob() when cancelling an active
job that has a whisperJobId, stopping GPU use immediately
- Webhook handler guards against locally-cancelled jobs: returns ok early
so whisper's late completion cannot overwrite cancelled status or send
a phantom 'Transcript ready' notification
- Replace blind sleep(Retry-After + 1s) in submitJob() with
waitForModelReady(): subscribes to /model/events SSE and proceeds as
soon as state:ready arrives; falls back to the Retry-After timeout if
SSE is unreachable or closes without model_ready
- Refactor retry tests to use URL-aware makeJobFetch() helper; add 7 new
tests (3 SSE-triggered retry, 3 cancelJob, 1 webhook cancelled-guard)
— 144/144 passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- whisper.ts: add getModelStatus(); fix submitJob() to retry on 503 using
Retry-After header instead of throwing; optional onModelWaiting callback
lets the pipeline surface model state to the UI during the wait
- pipeline.ts: pass onModelWaiting callback → emits model_warming SSE event
so the job detail page can show 'Warming up model…' while waiting
- types.ts: add ModelStateTag union and ModelStatus interface
- api/model/status: GET route proxies whisper /model/status (falls back to
{state:'unloaded'} if whisper unreachable)
- api/model/events: GET route relays whisper SSE stream to the browser;
AbortController tied to request.signal cleans up on disconnect
- layout.svelte: status pill is now live — initial fetch + EventSource on
/api/model/events; dot colour + label reflect real model state with a
pulsing animation while loading or waiting_for_gpu
- jobs/[id]/+page.svelte: handle model_warming event type → show a yellow
'Warming up model…' sub-label with spinner inside the progress card
- whisper.test.ts: update submitJob mocks to status:202 to match real API
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tonemark is a SvelteKit PWA for transcribing YouTube videos, audio
and video files, and microphone recordings using a local Whisper backend.
Features:
- Dark glassmorphic UI with electric-lime accent (5 switchable themes)
- Rail nav (desktop) / tab bar (mobile) layout
- Drop zone, YouTube URL input, and live audio recording inputs
- Audio mode waveform cards (none / standard / aggressive / auto)
- Real-time transcription progress with animated waveform
- Job queue with SSE streaming updates
- Push notifications on job completion
- PWA with native SvelteKit service worker
- SRT / TXT / MD / JSON transcript downloads
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Giancarmine Salucci
mozempkandCopilot