Instagram's GraphQL API silently truncates captions WITHOUT '….' markers.
Both DWWxiymssxE (393 chars full, 327 from API) and DXT73izCBoH
(744+ chars full, cut mid-sentence) were affected.
Remove the GraphQL-interception shortcut entirely. Always use DOM
extraction (HTML Section) which clicks '… more' to get the complete text.
The intercepted GraphQL caption is kept only as emergency fallback if
all DOM strategies fail.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
If the GraphQL-intercepted caption ends with '….' (Instagram's truncation
marker), skip it and fall through to HTML Section extraction which clicks
the '… more' button in the DOM to get the complete, untruncated caption.
Previously the 327-char truncated caption for DWWxiymssxE was returned
immediately, causing the LLM to say 'no recipe' even though the full
description had all ingredients and steps.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Always extract the full caption via Playwright (browser sees the
untruncated text). yt-dlp runs in parallel only to get the thumbnail
CDN URL quickly; its result for the description is discarded.
This eliminates the truncation problem at the source without needing
a fallback heuristic.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When yt-dlp returns a caption ending with the truncation marker '….'
(GraphQL API caps the text), automatically retry with the Playwright
extractor, which intercepts the full caption from live GraphQL network
traffic.
Falls back gracefully to the partial yt-dlp caption if Playwright fails.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instagram truncates long captions server-side (ends with '…').
When yt-dlp returns a truncated caption, automatically fall back to
the Playwright extractor which runs JS in a real browser and can
click the 'more' button to expand the full caption.
Falls back gracefully: if Playwright fails, the truncated text is
still used rather than failing the whole extraction.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously cookies.txt was only regenerated when auth.json was newer. But yt-dlp
overwrites cookies.txt during extraction with its own header ('generated by yt-dlp')
and potentially fewer/different cookies, losing the sessionid from auth.json.
Fix: remove mtime comparison — always regenerate cookies.txt from auth.json on each
extraction call. This ensures the full session cookie set is always present.
Also remove the now-unused statSync import.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- layout.css: add button.ic-btn-reset rule so all icon buttons
(bell, back, close, retry, etc.) get proper background:none reset
instead of browser-default white/grey appearance in dark mode
- instagram-extractor.ts: auto-convert secrets/auth.json
(Playwright storage format) to Netscape cookies.txt at runtime
whenever auth.json is newer; ensures sessionid and all Instagram
session cookies are passed to yt-dlp, fixing empty media response
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Increase max_tokens from 10 to 1024 for detection so thinking
models have room to reason. Also fall back to reasoning_content
if content is empty, since some local models (e.g. Gemma 4
thinking variants) put their answer there.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add instagram-extractor.ts: yt-dlp subprocess backend for Instagram
caption extraction. No in-process browser state, maintained against
Instagram frontend churn, supports cookies.txt for auth-walled reels.
- Add feature flag EXTRACTOR_BACKEND (ytdlp|playwright) in QueueProcessor
so the old Playwright path remains available as fallback.
- Add 9 unit tests and 2 live-network integration tests for the new extractor.
- Dockerfile: install yt-dlp via pip3 alongside existing Chromium deps.
- docker-compose: expose EXTRACTOR_BACKEND env var (default: ytdlp).
Also in this commit:
- LLM: configurable per-request timeout via LLM_REQUEST_TIMEOUT_MS (default 120s);
set maxRetries=0 to surface errors immediately; llama-swap /running health probe.
- QueueProcessor: thread progress callback through parser phase.
- LlmHealthIndicator: surface llama-swap loaded-model name.
- Logging: improve error serialization in queue-processor tests.
- .env.example: document llama-swap endpoint and model options.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Exported cleanText() and extractFromDOM() for unit testing
- Fixed metadata prefix regex to handle optional quotes
- Created comprehensive unit tests with mocked Playwright Page (15 tests, 12ms)
- All 275 tests passing
- Fixed NodeJS.Timer → NodeJS.Timeout in scheduler.ts line 13
- Fixed NodeJS.Timer[] → NodeJS.Timeout[] in fixtures.ts line 151
- Resolves TypeScript compile errors from iteration 0 review
- All 260 tests passing, build succeeds with no errors
- Fix InvalidCharacterError in push notifications with proper VAPID key validation
- Add attractive PWA install prompt component with cross-browser support
- Make notification settings always visible regardless of queue status
- Implement PWA install manager with user engagement detection
- Use SvelteKit navigation APIs instead of browser history API
- Add comprehensive error handling and logging
- Include cross-browser compatibility and responsive design
- Add development tooling improvements
Fixes push notification bugs and significantly improves PWA user experience
with modern, accessible interface components and proper error handling.
Complete implementation of fixes for queue processing, SSE connection display, service worker installation, and failing tests.
Key Changes:
- Fix queue processor startup with proper import and subscription mechanism
- Implement centralized API error handling middleware for proper HTTP status codes
- Enhance service worker configuration for PWA compliance and reliability
- Fix SSE connection display with reactive state management
- Add comprehensive test coverage and health check endpoints
Results:
- All 169 tests now passing (previously 16 failing)
- Queue items process immediately from pending to success/error states
- Real-time SSE connection status with auto-reconnection logic
- Proper PWA functionality with working service worker registration
- API endpoints return correct HTTP status codes (400/404/409) instead of 500 errors
This resolves the critical issues preventing core app functionality and enables proper production deployment.
- Create validateInstagramUrl utility using URL constructor
- Replace regex-based validation with hostname and protocol checks
- Support posts, reels, IGTV, and URLs with query parameters
- Add comprehensive unit tests (22 tests, all passing)
- Add integration tests for new URL formats
- Update API documentation with supported URL formats
Closes: #RelaxInstagramUrlValidation
- Fix EventSource is not defined error in queue dashboard
- Add browser guards for all EventSource usage
- Replace static constants (EventSource.OPEN/CLOSED) with numeric values
- Fix setInterval SSR violation in LLM health indicator
- Replace $effect anti-pattern with onMount in share page
- Add comprehensive SvelteKit SSR best practices documentation
- Add SSR audit and testing verification
All changes follow SvelteKit best practices and are verified against
official documentation. Production build succeeds with no SSR errors.
Closes: FixEventSourceSSR
See: docs/outcomes/FixEventSourceSSR.md
- Implement strict HTTP 200 validation (reject all other status codes)
- Add content-type validation (must be image/*)
- Add 10-second timeout protection with AbortController
- Thread progressCallback through all fetchImageAsBase64 calls
- Add detailed logging for each validation failure scenario
- Report validation failures via SSE progress callbacks
Unit tests:
- Add comprehensive test coverage for all validation scenarios
- Test HTTP status codes (200, 404, 403, 500, etc.)
- Test content-type validation (image/* vs text/html, etc.)
- Test timeout behavior with AbortController
- Test error handling (network errors, DNS, SSL, etc.)
- Test progress callback reporting
Integration tests:
- Add tests for complete extraction flow with URL failures
- Test fallback chain behavior (meta tags → poster → Instagram data → screenshot)
- Test real-world scenarios (redirects, query params, different post types)
Documentation:
- Enhanced JSDoc with validation criteria
- Added examples showing fallback behavior
- Documented all failure scenarios and their handling
All tests passing ✅
- Remove unreliable URL pass-through strategy (image_url field)
- Always download and upload images as File objects
- Get MIME type from HTTP response headers for URLs
- Use File constructor (not just Blob) for proper multipart metadata
- Add comprehensive error logging with headers and file metadata
- Simplify to single reliable upload path
Fixes 400 'Upload a valid image' error caused by Blob not providing
proper filename/MIME metadata in multipart form data.
User's Tandoor instance uses Bearer token authentication (likely JWT)
rather than Django REST Framework's Token authentication.
Reverts authentication from 'Token' back to 'Bearer' to fix 403 error:
'Authentication credentials were not provided.'
- Fix authentication header from 'Bearer' to 'Token' (DRF TokenAuth)
- Implement three-strategy upload system:
1. URL pass-through for direct URLs (most efficient)
2. Base64 data URL conversion for screenshots
3. Fallback blob upload for any other format
- Add comprehensive error handling with response details
- Add detailed logging for debugging upload strategies
- Document thumbnail formats in extractThumbnailStealth()
Fixes#30 - Tandoor image upload 400 Bad Request error
Based on Tandoor source code analysis (cookbook/views/api.py):
- RecipeImageSerializer accepts 'image_url' field for server-side download
- Uses Token authentication, not Bearer
- Supports multipart file upload with proper MIME types
- Update RECIPE_EXTRACTION_PROMPT to v2.1
- Remove instruction to number steps sequentially
- Update OUTPUT FORMAT and both few-shot examples
- Remove 'All steps numbered sequentially' from quality checklist
- Update fallback parser system prompt in parseRecipeWithStandardCompletion
- Frontend <ol> element already handles auto-numbering
- Tandoor integration unaffected (uses array index for step numbers)
Fixes double-numbering bug where steps appeared as '1. 1. Step text'
All 34 tests passing
Implementation follows execution plan in docs/plans/RemoveStepNumberPrefixes.md
Documented in docs/outcomes/RemoveStepNumberPrefixes.md
- Add progressCallback parameter to extractFromEmbeddedJSON and extractFromDOM
- Pass onProgress callback from extractWithStrategies to all strategies
- Fix legacy strategy to use correct callback variable name
- Verify extractViaGraphQL correctly returns null thumbnail
This fixes ReferenceError that was preventing all extraction methods from working.
All extraction strategies now properly emit thumbnail progress events via SSE.
Closes: FixProgressCallbackUndefinedErrors
- Fix critical await bug in extract-stream endpoint
- Add comprehensive logging to LLM and parser modules
- Implement fallback to standard completion for incompatible models
- Create enhanced v2.0 prompts with social media handling and few-shot examples
- Add LLM health check endpoint
- Decompose share page into 6 focused Svelte 5 snippets
Resolves LM Studio integration issues and improves code maintainability
