This commit is contained in:
Giancarmine Salucci
2026-02-18 01:21:44 +01:00
parent 54321fd7c9
commit 49bccf8f15
84 changed files with 14474 additions and 13925 deletions

View File

@@ -3,10 +3,7 @@
"singleQuote": true,
"trailingComma": "none",
"printWidth": 100,
"plugins": [
"prettier-plugin-svelte",
"prettier-plugin-tailwindcss"
],
"plugins": ["prettier-plugin-svelte", "prettier-plugin-tailwindcss"],
"overrides": [
{
"files": "*.svelte",

View File

@@ -5,6 +5,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
## 🚀 Features
### Core Functionality
- **Async Queue Processing**: Fire-and-forget recipe extraction with background processing
- **Real-time Updates**: Server-Sent Events for live progress tracking
- **Push Notifications**: Background notifications when recipes complete
@@ -13,6 +14,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
- **PWA Support**: Installable Progressive Web App with offline capabilities
### User Experience
- **Queue Dashboard**: Monitor all recipe extractions in real-time
- **Share Integration**: Browser share target for easy URL submission
- **Responsive Design**: Works on desktop, tablet, and mobile
@@ -20,6 +22,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
- **Progress Tracking**: Visual progress through extraction phases
### Technical Architecture
- **SvelteKit Frontend**: Modern reactive UI with TypeScript
- **Hexagonal Architecture**: Clean separation of concerns
- **In-Memory Queue**: High-performance processing with configurable concurrency
@@ -29,6 +32,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
## 📋 API Endpoints
### Queue Management
- `POST /api/queue` - Enqueue Instagram URL for processing
- `GET /api/queue` - List queue items with filtering and pagination
- `GET /api/queue/{id}` - Get specific queue item details
@@ -36,17 +40,20 @@ A modern web application that extracts recipes from Instagram posts and saves th
- `GET /api/queue/stream` - Server-Sent Events for real-time updates
### Push Notifications
- `POST /api/notifications/subscribe` - Subscribe to push notifications
- `DELETE /api/notifications/subscribe` - Unsubscribe from notifications
- `GET /api/notifications/vapid-key` - Get VAPID public key
### Legacy Endpoints (Deprecated)
- ~~`POST /api/extract`~~ - Use `/api/queue` instead
- ~~`GET /api/extract-stream`~~ - Use `/api/queue/stream` instead
## 🛠 Development Setup
### Prerequisites
- Node.js 18+
- npm or pnpm
- Tandoor Recipe Manager instance (optional)
@@ -79,6 +86,7 @@ open https://localhost:5173
```
The app runs on HTTPS by default for:
- Service worker support (required for PWA)
- Push notifications
- Browser share target API
@@ -89,6 +97,7 @@ The app runs on HTTPS by default for:
The application uses HTTPS in development with SSL certificates signed by an external Caddy CA container. The current certificate is valid until **December 20, 2035** (10 years).
**Certificate Information:**
- Location: `.ssl/` directory
- CA Certificate: `.ssl/root.crt` (already trusted on the system)
- Server Certificate: `.ssl/localhost.crt`
@@ -97,18 +106,21 @@ The application uses HTTPS in development with SSL certificates signed by an ext
Since the Caddy CA is already trusted on the system, the certificate should work without additional trust steps. If you encounter browser warnings:
**Linux (Ubuntu/Debian):**
```bash
sudo cp .ssl/root.crt /usr/local/share/ca-certificates/caddy-local.crt
sudo update-ca-certificates
```
**Chrome/Chromium:**
1. Go to `chrome://settings/certificates`
2. Click "Authorities" → "Import"
3. Select `.ssl/root.crt`
4. Check "Trust this certificate for identifying websites"
**Checking Certificate Expiration:**
```bash
openssl x509 -enddate -noout -in .ssl/localhost.crt
```
@@ -220,6 +232,7 @@ To enable web push notifications:
## 🏗 Architecture Overview
### Queue System
```
User submits URL → Queue Manager → Queue Processor
@@ -257,6 +270,7 @@ npm run test:watch
```
Test Coverage:
- **138 total tests** covering all major components
- Queue Manager: 28 tests
- Queue Processor: 5 integration tests
@@ -279,11 +293,13 @@ npm run preview
### Deployment
The app is built as a Node.js application with the following outputs:
- `/.svelte-kit/output/server/` - Server bundle
- `/.svelte-kit/output/client/` - Static assets
- `/build/` - Adapter output
Deploy the server bundle with:
```bash
node build/index.js
```
@@ -307,6 +323,7 @@ CMD ["node", "build"]
The app was migrated from a synchronous extraction system to an async queue-based system:
**Before (Synchronous)**:
- User waited for entire extraction process to complete
- No progress tracking during processing
- No retry capability for failures
@@ -314,6 +331,7 @@ The app was migrated from a synchronous extraction system to an async queue-base
- Limited error handling
**After (Async Queue)**:
- Fire-and-forget: submit URL and redirect immediately
- Real-time progress tracking via SSE
- Comprehensive retry system for failures
@@ -324,12 +342,14 @@ The app was migrated from a synchronous extraction system to an async queue-base
### API Migration
**Old Synchronous Endpoints** (Deprecated):
```bash
POST /api/extract # Submit URL and wait for completion
GET /api/extract-stream # Long-polling for progress
```
**New Queue Endpoints**:
```bash
POST /api/queue # Submit URL, get queue ID immediately
GET /api/queue # List all queue items
@@ -351,6 +371,7 @@ If migrating from the old system:
### Backward Compatibility
The legacy endpoints are still available but deprecated:
- They will return `410 Gone` status with migration instructions
- Support will be removed in a future version
- All new development should use the queue endpoints
@@ -383,4 +404,3 @@ This project is licensed under the MIT License - see the LICENSE file for detail
- [Tandoor Recipe Manager](https://docs.tandoor.dev/) - Recipe management system
- [Workbox](https://developers.google.com/web/tools/workbox) - PWA capabilities
- [fastq](https://github.com/mcollina/fastq) - High-performance queue processing

View File

@@ -4,7 +4,7 @@ services:
container_name: insta-recipe
network_mode: host
ports:
- "3000:3000"
- '3000:3000'
environment:
# LLM Configuration (Required)
- OPENAI_BASE_URL=${OPENAI_BASE_URL}
@@ -40,7 +40,13 @@ services:
- ./secrets:/app/secrets
restart: unless-stopped
healthcheck:
test: ["CMD", "node", "-e", "fetch('http://localhost:3000/api/health').then(r => r.ok ? process.exit(0) : process.exit(1)).catch(() => process.exit(1))"]
test:
[
'CMD',
'node',
'-e',
"fetch('http://localhost:3000/api/health').then(r => r.ok ? process.exit(0) : process.exit(1)).catch(() => process.exit(1))"
]
interval: 30s
timeout: 10s
retries: 3

View File

@@ -2,7 +2,7 @@ services:
app:
build: .
ports:
- "5173:5173"
- '5173:5173'
environment:
- PLAYWRIGHT_WS_ENDPOINT=ws://playwright-service:3000
- OPENAI_BASE_URL=http://ollama:11434/v1
@@ -18,7 +18,7 @@ services:
playwright-service:
build: ./playwright-service
ipc: host
ports: ["3000:3000"]
ports: ['3000:3000']
environment:
- DISPLAY=:99
security_opt:
@@ -26,7 +26,7 @@ services:
ollama:
image: ollama/ollama:latest
ports: ["11434:11434"]
ports: ['11434:11434']
volumes:
- ollama_data:/root/.ollama

View File

@@ -5,6 +5,7 @@ This document describes the InstaRecipe API endpoints for the async queue-based
## Base URL
All API endpoints are relative to your InstaRecipe instance:
```
https://your-instarecipe-instance.com/api
```
@@ -25,11 +26,14 @@ All endpoints return standardized error responses:
{
"error": "Error type",
"message": "Human-readable error message",
"details": { /* Additional error context */ }
"details": {
/* Additional error context */
}
}
```
HTTP status codes follow REST conventions:
- `200` - Success
- `201` - Created
- `400` - Bad Request (invalid input)
@@ -45,6 +49,7 @@ HTTP status codes follow REST conventions:
Enqueue an Instagram URL for async processing.
**Request:**
```json
{
"url": "https://instagram.com/p/abc123"
@@ -52,6 +57,7 @@ Enqueue an Instagram URL for async processing.
```
**Supported URL Formats:**
- Posts: `https://instagram.com/p/{post-id}`
- Posts (www): `https://www.instagram.com/p/{post-id}`
- Reels: `https://instagram.com/reel/{reel-id}`
@@ -59,12 +65,14 @@ Enqueue an Instagram URL for async processing.
- With query parameters: `https://instagram.com/reel/{reel-id}?utm_source=share`
**URL Requirements:**
- Must use HTTPS protocol
- Hostname must be `instagram.com` or `www.instagram.com`
- Any Instagram path is accepted (posts, reels, IGTV, etc.)
- Query parameters and hash fragments are allowed
**Examples:**
```json
// Post URL
{ "url": "https://instagram.com/p/ABC123" }
@@ -77,6 +85,7 @@ Enqueue an Instagram URL for async processing.
```
**Response (201 Created):**
```json
{
"id": "550e8400-e29b-41d4-a716-446655440000",
@@ -105,6 +114,7 @@ Enqueue an Instagram URL for async processing.
```
**Errors:**
- `400` - Invalid URL format (not a valid URL)
- `400` - URL must use HTTPS protocol
- `400` - URL must be from instagram.com domain
@@ -115,6 +125,7 @@ Enqueue an Instagram URL for async processing.
List queue items with optional filtering, pagination, and sorting.
**Query Parameters:**
- `status` (optional): Filter by status (`pending`, `in_progress`, `success`, `error`, `unhealthy`)
- `limit` (optional): Number of items to return (default: 50, max: 100)
- `offset` (optional): Number of items to skip (default: 0)
@@ -122,6 +133,7 @@ List queue items with optional filtering, pagination, and sorting.
- `order` (optional): Sort order (`asc`, `desc`) (default: `desc`)
**Examples:**
```bash
GET /api/queue # All items
GET /api/queue?status=error # Failed items only
@@ -130,6 +142,7 @@ GET /api/queue?sort=status&order=asc # Sort by status
```
**Response (200 OK):**
```json
{
"items": [
@@ -199,12 +212,14 @@ GET /api/queue?sort=status&order=asc # Sort by status
Get details for a specific queue item.
**Path Parameters:**
- `id`: Queue item UUID
**Response (200 OK):**
Returns the same queue item structure as in the list response.
**Errors:**
- `400` - Invalid UUID format
- `404` - Queue item not found
@@ -213,9 +228,11 @@ Returns the same queue item structure as in the list response.
Retry a failed queue item. Only items with `error` or `unhealthy` status can be retried.
**Path Parameters:**
- `id`: Queue item UUID
**Response (200 OK):**
```json
{
"success": true,
@@ -229,6 +246,7 @@ Retry a failed queue item. Only items with `error` or `unhealthy` status can be
```
**Errors:**
- `400` - Invalid UUID format
- `404` - Queue item not found
- `409` - Cannot retry item with current status (only `error` and `unhealthy` can be retried)
@@ -240,10 +258,12 @@ Retry a failed queue item. Only items with `error` or `unhealthy` status can be
Server-Sent Events (SSE) endpoint for real-time queue updates.
**Query Parameters:**
- `itemId` (optional): Filter updates for specific item
- `status` (optional): Filter updates by status
**Headers:**
```
Accept: text/event-stream
Cache-Control: no-cache
@@ -253,14 +273,18 @@ Cache-Control: no-cache
SSE stream with the following event types:
#### connection
Sent when connection is established:
```
event: connection
data: {"message": "Connected to queue stream", "timestamp": "2024-12-21T10:30:00Z"}
```
#### queue-update
Sent when queue item status changes:
```
event: queue-update
data: {
@@ -279,7 +303,9 @@ data: {
```
#### ping
Keep-alive ping sent every 30 seconds:
```
event: ping
data: {"timestamp": "2024-12-21T10:30:30Z"}
@@ -288,6 +314,7 @@ data: {"timestamp": "2024-12-21T10:30:30Z"}
**Usage Examples:**
**JavaScript:**
```javascript
const eventSource = new EventSource('/api/queue/stream');
@@ -312,6 +339,7 @@ eventSource.onerror = (error) => {
```
**curl:**
```bash
curl -N -H "Accept: text/event-stream" \
"https://your-instance.com/api/queue/stream?itemId=550e8400-e29b-41d4-a716-446655440000"
@@ -324,6 +352,7 @@ curl -N -H "Accept: text/event-stream" \
Get the VAPID public key required for push notification subscriptions.
**Response (200 OK):**
```json
{
"publicKey": "BDummyPublicKeyForDevelopment...",
@@ -336,6 +365,7 @@ Get the VAPID public key required for push notification subscriptions.
Subscribe to push notifications for queue processing updates.
**Request:**
```json
{
"subscription": {
@@ -350,6 +380,7 @@ Subscribe to push notifications for queue processing updates.
```
**Response (200 OK):**
```json
{
"success": true,
@@ -359,6 +390,7 @@ Subscribe to push notifications for queue processing updates.
```
**Errors:**
- `400` - Invalid subscription object or missing clientId
### DELETE /api/notifications/subscribe
@@ -366,6 +398,7 @@ Subscribe to push notifications for queue processing updates.
Unsubscribe from push notifications.
**Request:**
```json
{
"clientId": "unique-client-identifier"
@@ -373,6 +406,7 @@ Unsubscribe from push notifications.
```
**Response (200 OK):**
```json
{
"success": true,
@@ -390,6 +424,7 @@ Unsubscribe from push notifications.
This synchronous extraction endpoint is deprecated. Use `POST /api/queue` instead.
**Migration:**
```javascript
// ❌ Old synchronous approach
const response = await fetch('/api/extract', {
@@ -413,6 +448,7 @@ const queueItem = await response.json(); // Immediate response
This SSE endpoint is deprecated. Use `POST /api/queue` + `GET /api/queue/stream` instead.
**Migration:**
```javascript
// ❌ Old approach
const response = await fetch('/api/extract-stream', {
@@ -485,7 +521,8 @@ interface Recipe {
keywords?: string[]; // Recipe tags
image?: string; // Image URL
nutrition?: { // Nutritional information
nutrition?: {
// Nutritional information
calories?: number;
protein?: number;
carbs?: number;
@@ -552,7 +589,6 @@ async function processInstagramUrl(url) {
reject(error);
};
});
} catch (error) {
console.error('Processing failed:', error);
throw error;
@@ -561,13 +597,13 @@ async function processInstagramUrl(url) {
// Usage
processInstagramUrl('https://instagram.com/p/abc123')
.then(results => {
.then((results) => {
console.log('Recipe extracted:', results.recipe);
if (results.tandoorUrl) {
console.log('Uploaded to Tandoor:', results.tandoorUrl);
}
})
.catch(error => {
.catch((error) => {
console.error('Extraction failed:', error.message);
});
```

View File

@@ -91,21 +91,27 @@ insta-recipe/
## Key Directories
### `/src/lib/server/`
Server-side business logic following Hexagonal Architecture principles. Contains domain logic, adapters for external systems (Instagram, Tandoor, LLM), and port definitions.
### `/src/lib/client/`
Client-side utilities for PWA features (push notifications, install prompts, service worker messaging).
### `/src/routes/api/`
RESTful API endpoints implemented as SvelteKit server routes. Each directory contains `+server.ts` files exporting HTTP verb handlers.
### `/src/routes/share/`
Share target page allowing users to share Instagram URLs directly from their browser or mobile apps.
### `/src/lib/server/queue/`
Queue management system with in-memory storage, processor workers, and type definitions.
### `/docs/`
Comprehensive documentation including plans, outcomes, API specs, and migration guides.
---
@@ -113,33 +119,43 @@ Comprehensive documentation including plans, outcomes, API specs, and migration
## Design Patterns
### Singleton Pattern
Used for shared service instances:
- `QueueManager` (`queueManager` exported instance)
- `QueueProcessor` (`queueProcessor` exported instance)
- `PushNotificationService` (`pushNotificationService` exported instance)
- `ServiceWorkerMessageHandler` (`serviceWorkerMessageHandler` exported instance)
### Factory Pattern
Used for creating configured instances:
- `createLLM()` - Creates OpenAI client with environment configuration
- `createBrowserContext()` - Creates Playwright browser context with options
- `initializeBrowser()` - Initializes Chromium browser instance
### Observer Pattern
Implemented in QueueManager for real-time updates:
- Subscribers receive notifications on queue item changes
- Server-Sent Events (SSE) stream queue updates to clients
- Push notifications notify users of completion events
### Adapter Pattern (Hexagonal Architecture)
External systems accessed via adapters:
- **Instagram Adapter**: `extraction.ts` - Extracts content via Playwright
- **LLM Adapter**: `llm.ts`, `parser.ts` - Recipe parsing via OpenAI
- **Tandoor Adapter**: `tandoor.ts` - Recipe management system integration
- **Browser Adapter**: `browser.ts` - Playwright browser automation
### Strategy Pattern
Multiple extraction strategies with fallback:
1. Embedded JSON extraction
2. DOM selector extraction
3. GraphQL API extraction
@@ -150,28 +166,34 @@ Multiple extraction strategies with fallback:
## Key Components
### Queue Management System
**Location**: `src/lib/server/queue/`
Three-phase processing pipeline:
1. **Extraction Phase**: Extract text and thumbnail from Instagram
2. **Parsing Phase**: Parse recipe using LLM
3. **Uploading Phase**: Upload to Tandoor (if enabled)
**Components**:
- `QueueManager`: In-memory FIFO queue with CRUD operations
- `QueueProcessor`: Worker that processes items with configurable concurrency
- `types.ts`: Comprehensive type definitions for queue items and updates
### API Layer
**Location**: `src/routes/api/`
RESTful endpoints for:
- Queue operations (`POST /api/queue`, `GET /api/queue`, `GET /api/queue/[id]`)
- Real-time updates (`GET /api/queue/stream` - SSE)
- Push notifications (`POST /api/notifications/subscribe`)
- Health checks (`GET /api/health`, `GET /api/llm-health`)
### Client-Side Services
**Location**: `src/lib/client/`
- **PushNotificationManager**: Manages Web Push API subscriptions
@@ -179,14 +201,17 @@ RESTful endpoints for:
- **ServiceWorkerMessageHandler**: Processes service worker messages
### Instagram Extraction
**Location**: `src/lib/server/extraction.ts`
Multi-method extraction with intelligent fallback:
- Progress callbacks for real-time feedback
- Retry logic with configurable attempts
- Thumbnail extraction and validation
### LLM Integration
**Location**: `src/lib/server/parser.ts`, `src/lib/server/llm.ts`
- Recipe detection endpoint
@@ -198,6 +223,7 @@ Multi-method extraction with intelligent fallback:
## Dependencies
### Production Dependencies
- **@types/uuid** (^10.0.0) - UUID type definitions
- **date-fns** (^4.1.0) - Date utility library
- **openai** (^4.20.0) - OpenAI API client
@@ -206,6 +232,7 @@ Multi-method extraction with intelligent fallback:
- **zod** (^3.23.0) - Schema validation
### Development Dependencies
- **@sveltejs/kit** (^2.48.5) - SvelteKit framework
- **@sveltejs/adapter-node** (^5.4.0) - Node.js adapter
- **svelte** (^5.43.8) - Svelte 5 framework
@@ -223,12 +250,14 @@ Multi-method extraction with intelligent fallback:
## Module Organization
### SvelteKit Path Aliases
- `$lib``src/lib/`
- `$lib/*``src/lib/*`
- `$app/*` → SvelteKit app imports
- `$env/dynamic/private` → Environment variables (server-side)
### Directory Structure Conventions
- **Server-only code**: `src/lib/server/` (not bundled to client)
- **Client-only code**: `src/lib/client/` (not executed on server)
- **Shared code**: `src/lib/` (available to both)
@@ -240,6 +269,7 @@ Multi-method extraction with intelligent fallback:
## Data Flow
### Recipe Extraction Flow
```
User submits URL
@@ -261,6 +291,7 @@ SSE updates notify client
```
### Real-time Updates Flow
```
Client connects to GET /api/queue/stream (SSE)
@@ -274,6 +305,7 @@ Client updates UI reactively
```
### Push Notification Flow
```
Client requests permission
@@ -295,37 +327,44 @@ Notification displayed to user
## Build System
### Build Command
```bash
npm run build
```
Generates production-ready build in `build/` directory using:
- Vite for bundling
- `@sveltejs/adapter-node` for Node.js deployment
- TypeScript compilation
- SvelteKit prerendering and optimization
### Test Command
```bash
npm test
```
Runs test suite using Vitest with two projects:
1. **Server tests**: Node environment for server-side code
2. **Client tests**: Playwright browser for Svelte components
### Development Server
```bash
npm run dev
```
Starts Vite dev server with:
- HTTPS enabled (certificates in `.ssl/`)
- Hot module replacement
- TypeScript checking
- File watching
### Linting & Formatting
```bash
npm run lint # ESLint + Prettier check
npm run format # Prettier write
@@ -336,19 +375,24 @@ npm run format # Prettier write
## Deployment
### Docker Deployment
Dockerfile includes:
- Node.js 22 Alpine base image
- Playwright Chromium installation
- Production build
- Port 3000 exposure
Run with:
```bash
docker-compose up
```
### Environment Variables
Required configuration:
- `OPENAI_API_KEY` - LLM API access
- `TANDOOR_URL` - Tandoor instance URL (optional)
- `TANDOOR_TOKEN` - Tandoor API token (optional)
@@ -360,13 +404,16 @@ Required configuration:
## Testing Architecture
### Test Categories
1. **Unit Tests**: Individual function testing
2. **Integration Tests**: Multi-component workflows
3. **API Tests**: Endpoint behavior validation
4. **Browser Tests**: Svelte component rendering
### Test Coverage
138 tests covering:
- Queue management operations
- Instagram URL validation
- SSE streaming
@@ -375,6 +422,7 @@ Required configuration:
- Notification service
### Test Configuration
- **Server tests**: Node environment with mocked dependencies
- **Client tests**: Playwright Chromium browser with Svelte testing library
@@ -383,15 +431,18 @@ Required configuration:
## Security Considerations
### SSL/TLS
- Development uses local SSL certificates signed by external Caddy CA
- Certificates stored in `.ssl/` (git-ignored)
- Required for PWA features (Service Worker, Push API)
### Authentication
- Basic auth for scheduled tasks (username/password from environment)
- Tandoor integration uses bearer token authentication
### Input Validation
- Instagram URL validation with regex patterns
- Zod schema validation for API payloads
- Error handling with custom error classes

View File

@@ -19,12 +19,14 @@
### Files & Directories
#### SvelteKit Route Files
- Route pages: `+page.svelte`
- Route servers: `+server.ts`
- Route layouts: `+layout.svelte`
- Type definitions: `$types.ts` (auto-generated)
**Example:**
```
src/routes/api/queue/
├── [id]/
@@ -37,19 +39,23 @@ src/routes/api/queue/
```
#### Library Files
- **PascalCase** for classes and managers: `QueueManager.ts`, `PushNotificationService.ts`
- **kebab-case** for utilities and configs: `tandoor-config.ts`, `instagram-url.ts`
- **lowercase** for general modules: `browser.ts`, `extraction.ts`, `parser.ts`
**Examples from codebase:**
- `src/lib/server/queue/QueueManager.ts`
- `src/lib/server/tandoor-config.ts`
- `src/lib/client/PushNotificationManager.ts`
#### Test Files
Pattern: `<name>.spec.ts` or `<name>.test.ts`
**Examples:**
- `queue-manager.spec.ts`
- `instagram-url-validation.spec.ts`
- `page.svelte.spec.ts`
@@ -57,10 +63,12 @@ Pattern: `<name>.spec.ts` or `<name>.test.ts`
### Variables & Functions
#### Variables
- **camelCase** for local variables and parameters
- **SCREAMING_SNAKE_CASE** for constants
**Examples:**
```typescript
// From QueueManager.ts
private items: Map<string, QueueItem> = new Map();
@@ -76,10 +84,12 @@ const unsubscribe = queueManager.subscribe(callback);
```
#### Functions
- **camelCase** for function names
- **Descriptive action verbs**: `enqueue`, `extractRecipe`, `uploadRecipeImage`
**Examples:**
```typescript
// From QueueManager.ts
enqueue(url: string): QueueItem { ... }
@@ -99,11 +109,13 @@ export async function extractRecipe(text: string): Promise<Recipe> { ... }
### Types & Interfaces
#### Interfaces & Types
- **PascalCase** for interface names
- Prefix with `I` is **NOT** used
- Exported types use `export type` or `export interface`
**Examples:**
```typescript
// From queue/types.ts
export interface QueueItem {
@@ -121,11 +133,7 @@ export interface QueueStatusUpdate {
// ...
}
export type QueueItemStatus =
| 'pending'
| 'in_progress'
| 'completed'
| 'failed';
export type QueueItemStatus = 'pending' | 'in_progress' | 'completed' | 'failed';
// From extraction.ts
export interface ExtractedContent {
@@ -137,16 +145,18 @@ export type ProgressCallback = (event: ProgressEvent) => void;
```
#### Zod Schemas
- **PascalCase** with `Schema` suffix
- Inferred types without suffix
**Examples:**
```typescript
// From parser.ts
const RecipeSchema = z.object({
name: z.string(),
description: z.string(),
servings: z.number(),
servings: z.number()
// ...
});
@@ -163,10 +173,12 @@ export type TandoorRecipe = z.infer<typeof TandoorRecipeSchema>;
### Classes
#### Class Names
- **PascalCase** for class names
- Descriptive suffixes: `Manager`, `Service`, `Processor`, `Handler`
**Examples:**
```typescript
// From QueueManager.ts
export class QueueManager {
@@ -188,6 +200,7 @@ class PushNotificationService {
```
#### Singleton Export Pattern
```typescript
// Class definition
export class QueueManager {
@@ -203,6 +216,7 @@ export const queueManager = new QueueManager();
## Indentation & Formatting
### General Rules
- **Indentation:** 2 spaces (enforced by Prettier)
- **No tabs**
- **Max line length:** 100 characters (soft limit, not enforced)
@@ -213,6 +227,7 @@ export const queueManager = new QueueManager();
### Code Examples
#### Function Declarations
```typescript
// From QueueManager.ts
enqueue(url: string): QueueItem {
@@ -241,6 +256,7 @@ enqueue(url: string): QueueItem {
```
#### Async Functions
```typescript
// From extraction.ts
export async function extractTextAndThumbnail(
@@ -261,6 +277,7 @@ export async function extractTextAndThumbnail(
```
#### Object Destructuring
```typescript
// From route handlers
export const POST: RequestHandler = async ({ request }) => {
@@ -279,12 +296,14 @@ export const GET: RequestHandler = async ({ params }) => {
## Import Patterns
### Import Order
1. External dependencies (Node.js built-ins, npm packages)
2. SvelteKit imports (`$lib`, `$app`, `$env`)
3. Relative imports (`./ `, `../`)
4. Type imports (separate from value imports when beneficial)
**Example:**
```typescript
// From QueueProcessor.ts
@@ -307,6 +326,7 @@ import type { QueueItem } from './types';
### Import Styles
#### Named Imports (Preferred)
```typescript
import { json } from '@sveltejs/kit';
import { queueManager } from '$lib/server/queue/QueueManager';
@@ -314,12 +334,14 @@ import { validateInstagramUrl } from '$lib/server/validation/instagram-url';
```
#### Type-Only Imports
```typescript
import type { RequestHandler } from './$types';
import type { QueueItem, QueueItemStatus } from './types';
```
#### Default Imports
```typescript
import OpenAI from 'openai';
import fs from 'fs';
@@ -329,6 +351,7 @@ import path from 'path';
### Export Patterns
#### Named Exports (Preferred)
```typescript
// Export functions
export async function extractRecipe(text: string): Promise<Recipe> { ... }
@@ -345,6 +368,7 @@ export interface QueueItem { ... }
```
#### Singleton Pattern Export
```typescript
// Define class
export class QueueManager { ... }
@@ -358,10 +382,12 @@ export const queueManager = new QueueManager();
## Comments & Documentation
### JSDoc Style
Used extensively for public APIs and exported functions.
**Function Documentation:**
```typescript
````typescript
/**
* Add URL to processing queue
*
@@ -377,10 +403,11 @@ Used extensively for public APIs and exported functions.
enqueue(url: string): QueueItem {
// Implementation
}
```
````
**Class Documentation:**
```typescript
````typescript
/**
* Singleton queue manager for processing Instagram URLs
*
@@ -402,9 +429,10 @@ enqueue(url: string): QueueItem {
export class QueueManager {
// Implementation
}
```
````
**Module-Level Documentation:**
```typescript
/**
* Queue Manager - Core queue operations and event management
@@ -421,19 +449,21 @@ export class QueueManager {
### Inline Comments
#### Single-line Comments
```typescript
// Set restrictive permissions
fs.chmodSync(authFile, 0o600);
// FIFO order - get oldest pending item
const pendingItems = Array.from(this.items.values())
.filter(item => item.status === 'pending');
const pendingItems = Array.from(this.items.values()).filter((item) => item.status === 'pending');
```
#### Block Comments (Avoided)
Single-line comments preferred. Block comments used only for large comment blocks or temporarily disabling code during development.
### TODO Comments
```typescript
// TODO: Add retry logic with exponential backoff
// FIXME: Handle race condition when multiple workers dequeue
@@ -446,6 +476,7 @@ Single-line comments preferred. Block comments used only for large comment block
### Type Safety
#### Strict Mode Enabled
```json
// tsconfig.json
{
@@ -457,6 +488,7 @@ Single-line comments preferred. Block comments used only for large comment block
```
#### Type Annotations
```typescript
// Explicit return types for public functions
export async function extractRecipe(text: string): Promise<Recipe> { ... }
@@ -469,28 +501,17 @@ const items = queueManager.getAll(); // Type inferred
```
### Union Types
```typescript
export type QueueItemStatus =
| 'pending'
| 'in_progress'
| 'completed'
| 'failed';
export type QueueItemStatus = 'pending' | 'in_progress' | 'completed' | 'failed';
export type ProcessingPhase =
| 'extraction'
| 'parsing'
| 'uploading';
export type ProcessingPhase = 'extraction' | 'parsing' | 'uploading';
export type ProgressEventType =
| 'status'
| 'method'
| 'retry'
| 'error'
| 'thumbnail'
| 'complete';
export type ProgressEventType = 'status' | 'method' | 'retry' | 'error' | 'thumbnail' | 'complete';
```
### Generics
```typescript
// Generic function
async function fetchFromTandoor<T>(
@@ -508,6 +529,7 @@ async function fetchFromTandoor<T>(
### Runes (Reactivity)
#### $state (Reactive Variables)
```svelte
<script lang="ts">
let count = $state(0);
@@ -516,6 +538,7 @@ async function fetchFromTandoor<T>(
```
#### $props (Component Props)
```svelte
<script lang="ts">
let {
@@ -533,6 +556,7 @@ async function fetchFromTandoor<T>(
```
#### $derived (Computed Values)
```svelte
<script lang="ts">
let count = $state(0);
@@ -541,6 +565,7 @@ async function fetchFromTandoor<T>(
```
#### $effect (Side Effects)
```svelte
<script lang="ts">
let url = $state('');
@@ -552,6 +577,7 @@ async function fetchFromTandoor<T>(
```
### Component Structure
```svelte
<script lang="ts">
// Imports
@@ -593,6 +619,7 @@ async function fetchFromTandoor<T>(
## Error Handling
### Custom Error Classes
```typescript
// From api/errors.ts
export class ValidationError extends Error {
@@ -618,6 +645,7 @@ export class ConflictError extends Error {
```
### Try-Catch Pattern
```typescript
export const POST: RequestHandler = async ({ request }) => {
try {
@@ -629,7 +657,6 @@ export const POST: RequestHandler = async ({ request }) => {
const item = queueManager.enqueue(url);
return json(item, { status: 201 });
} catch (error) {
return handleApiError(error);
}
@@ -641,6 +668,7 @@ export const POST: RequestHandler = async ({ request }) => {
## Linting Configuration
### ESLint
**Config:** `eslint.config.js`
- Base: `@eslint/js` recommended
@@ -649,6 +677,7 @@ export const POST: RequestHandler = async ({ request }) => {
- Formatting: `eslint-config-prettier`
**Rules:**
```javascript
{
rules: {
@@ -658,6 +687,7 @@ export const POST: RequestHandler = async ({ request }) => {
```
### Prettier
**Config:** `.prettierrc`
```json
@@ -675,6 +705,7 @@ export const POST: RequestHandler = async ({ request }) => {
## Testing Conventions
### Test Structure
```typescript
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
@@ -701,6 +732,7 @@ describe('QueueManager', () => {
```
### Mock Pattern
```typescript
vi.mock('$lib/server/extraction', () => ({
extractTextAndThumbnail: vi.fn().mockResolvedValue({
@@ -715,6 +747,7 @@ vi.mock('$lib/server/extraction', () => ({
## File Headers
### Module Documentation Pattern
Every major module includes a header comment:
```typescript
@@ -730,6 +763,7 @@ Every major module includes a header comment:
```
**Example:**
```typescript
/**
* Queue Manager - Core queue operations and event management
@@ -748,6 +782,7 @@ Every major module includes a header comment:
## Additional Conventions
### Environment Variables
```typescript
import { env } from '$env/dynamic/private';
@@ -756,19 +791,24 @@ const tandoorUrl = env.TANDOOR_URL || null;
```
### Date Handling
ISO8601 strings throughout the application:
```typescript
const now = new Date().toISOString();
// Output: "2026-02-15T12:30:45.123Z"
```
### Null vs Undefined
- `null`: Intentional absence of value
- `undefined`: Not yet initialized or optional parameters
- Prefer `null` for API responses and data structures
### Async/Await
Always preferred over Promise chains:
```typescript
// Preferred
async function fetchData() {
@@ -780,8 +820,8 @@ async function fetchData() {
// Avoid
function fetchData() {
return fetch(url)
.then(response => response.json())
.then(data => data);
.then((response) => response.json())
.then((data) => data);
}
```

File diff suppressed because it is too large Load Diff

View File

@@ -11,6 +11,7 @@ The migration transformed InstaRecipe from a blocking, synchronous extraction sy
### Architecture Transformation
**Before: Synchronous System**
```
User Request → Direct Processing → Response (wait 30-60s)
↓ ↓ ↓
@@ -18,6 +19,7 @@ User Request → Direct Processing → Response (wait 30-60s)
```
**After: Async Queue System**
```
User Request → Queue Item Created → Immediate Response
↓ ↓ ↓
@@ -60,6 +62,7 @@ User Request → Queue Item Created → Immediate Response
### New Endpoints
#### Queue Management
```typescript
// Enqueue URL for processing
POST /api/queue
@@ -84,6 +87,7 @@ Events: connection, queue-update, ping
```
#### Push Notifications
```typescript
// Subscribe to push notifications
POST /api/notifications/subscribe
@@ -101,11 +105,11 @@ These endpoints are marked for removal and should not be used in new code:
```typescript
// ❌ DEPRECATED: Synchronous extraction
POST /api/extract
POST / api / extract;
// 👉 Use: POST /api/queue
// ❌ DEPRECATED: Long-polling progress
GET /api/extract-stream
GET / api / extract - stream;
// 👉 Use: GET /api/queue/stream
```
@@ -167,6 +171,7 @@ interface QueueStatusUpdate {
### For Frontend Applications
1. **Replace Synchronous Calls**
```typescript
// ❌ Old synchronous approach
const response = await fetch('/api/extract', {
@@ -187,6 +192,7 @@ interface QueueStatusUpdate {
```
2. **Add Real-time Updates**
```typescript
// Setup Server-Sent Events for progress tracking
const eventSource = new EventSource(`/api/queue/stream?itemId=${itemId}`);
@@ -222,6 +228,7 @@ interface QueueStatusUpdate {
### For Backend Integrations
1. **Update API Calls**
```python
# ❌ Old synchronous API
response = requests.post('/api/extract', json={'url': url})
@@ -240,6 +247,7 @@ interface QueueStatusUpdate {
```
2. **Implement SSE Client** (Python example)
```python
import sseclient
@@ -314,18 +322,21 @@ npm test queue-sse
## Performance Considerations
### Before Migration
- **Blocking Operations**: Each request blocked a server thread
- **Single Processing**: One extraction at a time
- **No Progress**: Users waited without feedback
- **Memory Usage**: High memory usage during long operations
### After Migration
- **Non-blocking**: Requests return immediately
- **Concurrent Processing**: Multiple extractions in parallel
- **Real-time Feedback**: Live progress updates
- **Efficient Memory**: Event-driven, minimal memory footprint
### Performance Metrics
- **Response Time**: 50ms (queue) vs 30-60s (synchronous)
- **Throughput**: 2x concurrent processing vs 1x sequential
- **User Experience**: Immediate feedback vs long waiting
@@ -336,11 +347,13 @@ npm test queue-sse
If issues arise, the system can be rolled back by:
1. **Disable Queue Processing**
```env
QUEUE_PROCESSING_ENABLED=false
```
2. **Re-enable Legacy Endpoints** (if preserved)
```typescript
// Temporary fallback to synchronous processing
app.post('/api/extract', legacyExtractHandler);

View File

@@ -3,6 +3,7 @@
This guide documents SSR-safe patterns and anti-patterns for InstaRecipe development. All developers should follow these guidelines to prevent SSR errors.
## Table of Contents
- [Core Principle](#core-principle)
- [Browser API Detection](#browser-api-detection)
- [Lifecycle Hooks](#lifecycle-hooks)
@@ -18,6 +19,7 @@ This guide documents SSR-safe patterns and anti-patterns for InstaRecipe develop
**SvelteKit renders components on both the server and client.** Any browser-only APIs must be guarded or used in browser-only contexts.
### Browser-Only APIs (Require Guards)
- `window.*`
- `document.*`
- `localStorage`, `sessionStorage`
@@ -72,6 +74,7 @@ if (browser) {
### `onMount` - Browser-Only Lifecycle
**Use `onMount` for:**
- Browser API initialization
- Timer setup (`setInterval`, `setTimeout`)
- Event listener registration
@@ -160,11 +163,13 @@ onMount(() => {
```
**When to use `$effect`:**
- Synchronizing derived state
- DOM manipulation (with browser guard)
- Reactive cleanup
**When NOT to use `$effect`:**
- Initialization (use `onMount`)
- API calls on mount (use `onMount`)
- Timer setup (use `onMount`)
@@ -189,11 +194,13 @@ if (browser && eventSource?.readyState === EventSource.OPEN)
```
**EventSource States:**
- `EventSource.CONNECTING = 0`
- `EventSource.OPEN = 1`
- `EventSource.CLOSED = 2`
**WebSocket States:**
- `WebSocket.CONNECTING = 0`
- `WebSocket.OPEN = 1`
- `WebSocket.CLOSING = 2`
@@ -276,6 +283,7 @@ export class PushNotificationManager {
```
**Why it's good:**
- Guards all browser API access
- Early returns prevent unnecessary code execution during SSR
- Defensive programming with null checks
@@ -327,6 +335,7 @@ export class PushNotificationManager {
```
**Why it's good:**
- Uses `onMount` instead of `$effect` for initialization
- Timer setup in browser-only context
- Proper cleanup with return function
@@ -408,6 +417,7 @@ Watch server console for errors during build and preview.
### 2. Check for Hydration Warnings
Open browser DevTools console and look for:
- "Hydration failed"
- "The server response doesn't match the client content"
@@ -422,6 +432,7 @@ grep -r "navigator\." src/routes --include="*.svelte"
```
Then verify each usage is either:
- In an event handler (safe)
- In `onMount` (safe)
- Guarded with `if (browser)` (safe)

View File

@@ -21,16 +21,14 @@ export default defineConfig(
languageOptions: {
globals: { ...globals.browser, ...globals.node }
},
rules: { // typescript-eslint strongly recommend that you do not use the no-undef lint rule on TypeScript projects.
rules: {
// typescript-eslint strongly recommend that you do not use the no-undef lint rule on TypeScript projects.
// see: https://typescript-eslint.io/troubleshooting/faqs/eslint/#i-get-errors-from-the-no-undef-rule-about-global-variables-not-being-defined-even-though-there-are-no-typescript-errors
"no-undef": 'off' }
'no-undef': 'off'
}
},
{
files: [
'**/*.svelte',
'**/*.svelte.ts',
'**/*.svelte.js'
],
files: ['**/*.svelte', '**/*.svelte.ts', '**/*.svelte.js'],
languageOptions: {
parserOptions: {
projectService: true,

View File

@@ -15,20 +15,20 @@ export default defineConfig({
reporter: 'list',
use: {
baseURL: 'http://localhost:5173',
trace: 'on-first-retry',
trace: 'on-first-retry'
},
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
use: { ...devices['Desktop Chrome'] }
}
],
webServer: {
command: 'npm run dev',
url: 'http://localhost:5173',
reuseExistingServer: !process.env.CI,
timeout: 120000,
},
timeout: 120000
}
});

View File

@@ -50,7 +50,7 @@ async function generateFaviconIco() {
console.log('✓ All validation checks passed');
}
generateFaviconIco().catch(err => {
generateFaviconIco().catch((err) => {
console.error('Error generating favicon.ico:', err);
process.exit(1);
});

View File

@@ -54,7 +54,7 @@ async function generateFavicon() {
console.log('✓ All validation checks passed');
}
generateFavicon().catch(err => {
generateFavicon().catch((err) => {
console.error('Error generating favicon:', err);
process.exit(1);
});

View File

@@ -3,7 +3,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link rel="manifest" href="/manifest.json">
<link rel="manifest" href="/manifest.json" />
%sveltekit.head%
</head>
<body data-sveltekit-preload-data="hover">

View File

@@ -101,7 +101,7 @@ export class PWAInstallManager {
callback(this.canInstall());
return () => {
this.listeners = this.listeners.filter(cb => cb !== callback);
this.listeners = this.listeners.filter((cb) => cb !== callback);
};
}
@@ -109,7 +109,7 @@ export class PWAInstallManager {
* Notify all listeners of state change
*/
private notifyListeners(canInstall: boolean): void {
this.listeners.forEach(callback => {
this.listeners.forEach((callback) => {
try {
callback(canInstall);
} catch (error) {

View File

@@ -67,7 +67,7 @@ class PushNotificationManager {
callback(this.state); // Send initial state
return () => {
this.listeners = this.listeners.filter(cb => cb !== callback);
this.listeners = this.listeners.filter((cb) => cb !== callback);
};
}
@@ -90,11 +90,8 @@ class PushNotificationManager {
return;
}
this.state.supported = (
'serviceWorker' in navigator &&
'PushManager' in window &&
'Notification' in window
);
this.state.supported =
'serviceWorker' in navigator && 'PushManager' in window && 'Notification' in window;
this.state.permission = this.state.supported ? Notification.permission : 'denied';
}
@@ -164,7 +161,7 @@ class PushNotificationManager {
* Subscribe to push notifications
*/
async subscribe(): Promise<boolean> {
if (!await this.requestPermission()) {
if (!(await this.requestPermission())) {
return false;
}
@@ -215,7 +212,6 @@ class PushNotificationManager {
console.log('[PushManager] Successfully subscribed to push notifications');
return true;
} catch (error) {
console.error('[PushManager] Subscription failed:', error);
this.state.error = 'Failed to subscribe to notifications';
@@ -265,7 +261,6 @@ class PushNotificationManager {
console.log('[PushManager] Successfully unsubscribed from push notifications');
return true;
} catch (error) {
console.error('[PushManager] Unsubscription failed:', error);
this.state.error = 'Failed to unsubscribe from notifications';
@@ -331,10 +326,8 @@ class PushNotificationManager {
try {
// Add proper padding
const padding = '='.repeat((4 - cleanKey.length % 4) % 4);
const base64 = (cleanKey + padding)
.replace(/-/g, '+')
.replace(/_/g, '/');
const padding = '='.repeat((4 - (cleanKey.length % 4)) % 4);
const base64 = (cleanKey + padding).replace(/-/g, '+').replace(/_/g, '/');
// Validate base64 format before decoding
const base64Regex = /^[A-Za-z0-9+\/]*={0,2}$/;
@@ -351,7 +344,6 @@ class PushNotificationManager {
console.log(`[PushManager] Successfully decoded VAPID key (${outputArray.length} bytes)`);
return outputArray;
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
console.error('[PushManager] Failed to decode VAPID key:', error, 'Key:', cleanKey);
@@ -363,7 +355,7 @@ class PushNotificationManager {
* Notify all listeners of state change
*/
private notifyListeners(): void {
this.listeners.forEach(callback => {
this.listeners.forEach((callback) => {
try {
callback({ ...this.state });
} catch (error) {

View File

@@ -5,7 +5,7 @@
* and coordinates with the main application.
*/
import { pushState } from "$app/navigation";
import { pushState } from '$app/navigation';
interface ServiceWorkerMessage {
type: string;

View File

@@ -29,36 +29,46 @@ export function handleApiError(error: unknown): Response {
// Handle known error types with specific status codes
if (error instanceof ValidationError) {
return json({
return json(
{
message: error.message,
type: 'validation_error'
}, { status: 400 });
},
{ status: 400 }
);
}
if (error instanceof NotFoundError) {
return json({
return json(
{
message: error.message,
type: 'not_found_error'
}, { status: 404 });
},
{ status: 404 }
);
}
if (error instanceof ConflictError) {
return json({
return json(
{
message: error.message,
type: 'conflict_error'
}, { status: 409 });
},
{ status: 409 }
);
}
// Handle generic errors
const message = error instanceof Error ? error.message : 'Unknown error occurred';
// Don't expose internal error details in production
const publicMessage = process.env.NODE_ENV === 'production'
? 'Internal server error'
: message;
const publicMessage = process.env.NODE_ENV === 'production' ? 'Internal server error' : message;
return json({
return json(
{
message: publicMessage,
type: 'server_error'
}, { status: 500 });
},
{ status: 500 }
);
}

View File

@@ -9,7 +9,14 @@ export interface ExtractedContent {
thumbnail: string | null;
}
export type ExtractionMethod = 'embedded-json' | 'internal-state' | 'html-section' | 'dom-selector' | 'graphql-api' | 'graphql-intercept' | 'legacy';
export type ExtractionMethod =
| 'embedded-json'
| 'internal-state'
| 'html-section'
| 'dom-selector'
| 'graphql-api'
| 'graphql-intercept'
| 'legacy';
type CaptionCandidate = {
element: Element;
@@ -192,7 +199,146 @@ function extractShortcode(url: string): string | undefined {
}
/**
* Clean extracted text
* Recipe keywords used for caption scoring
*/
const RECIPE_KEYWORDS = [
'ingredienti',
'procedimento',
'preparazione',
'ricetta',
'recipe',
'instructions'
];
/**
* Timeout configuration constants (in milliseconds)
*/
const TIMEOUTS = {
CONTENT_LOAD: 1500,
MORE_BUTTON_VISIBILITY: 1000,
CAPTION_EXPANSION: 3000,
MORE_BUTTON_VISIBILITY_DOM: 500,
MORE_BUTTON_CLICK: 800,
PAGE_LOAD: 10000,
NETWORK_SETTLE: 2000,
ARTICLE_SELECTOR: 5000,
GRAPHQL_WAIT: 1000,
PAGE_NAVIGATION: 30000,
ANTI_DETECTION_MIN: 1000,
ANTI_DETECTION_MAX: 3000
} as const;
/**
* Try to expand truncated caption by clicking "more" button in HTML section method
*/
async function tryExpandCaptionInHTMLSection(page: Page): Promise<void> {
console.log('[Extractor] Looking for "more" button in primary post container...');
try {
await page.waitForTimeout(TIMEOUTS.CONTENT_LOAD);
const mainContainer = page.locator('article, main, [role="main"]').first();
const containerExists = (await mainContainer.count()) > 0;
if (!containerExists) {
console.log('[Extractor] No main container found');
return;
}
console.log('[Extractor] Found main post container, searching for "more" button...');
const morePatterns = [
{
locator: mainContainer.locator('span').filter({ hasText: /\.\.\.\s*more/i }),
desc: "span with '...more'"
},
{
locator: mainContainer.locator('span').filter({ hasText: /…\s*more/i }),
desc: "span with '… more'"
},
{
locator: mainContainer.locator('div[role="button"]').filter({ hasText: /more/i }),
desc: "button with 'more'"
},
{
locator: mainContainer.locator('span[role="button"]').filter({ hasText: /more/i }),
desc: "span button with 'more'"
}
];
for (const pattern of morePatterns) {
const count = await pattern.locator.count();
console.log(`[Extractor] Checking ${pattern.desc}: found ${count}`);
if (count === 0) continue;
const firstMore = pattern.locator.first();
try {
if (await firstMore.isVisible({ timeout: TIMEOUTS.MORE_BUTTON_VISIBILITY })) {
const text = await firstMore.textContent();
console.log(`[Extractor] Found visible "more": "${text}"`);
await firstMore.click();
console.log('[Extractor] Clicked "more" - waiting for expansion...');
await page.waitForTimeout(TIMEOUTS.CAPTION_EXPANSION);
console.log('[Extractor] Caption expansion complete');
break;
}
} catch (e) {
console.log(`[Extractor] ${pattern.desc} not clickable: ${e}`);
}
}
console.log('[Extractor] Finished "more" button expansion attempt');
} catch (e) {
console.log(`[Extractor] Error while trying to expand caption: ${e}`);
}
}
/**
* Try to expand truncated caption by clicking "more" button in DOM method
*/
async function tryExpandCaptionInDOM(page: Page): Promise<void> {
const moreButtonSelectors = [
'article button:has-text("more")',
'article button:has-text("More")',
'article button:has-text("… more")',
'article span[role="button"]:has-text("more")',
'article [role="button"]:has-text("more")',
'article div[role="button"]:has-text("more")',
'xpath=//article//span[contains(text(), "more")]/..',
'xpath=//article//button[contains(., "more")]'
];
const maxExpandAttempts = 3;
let expandAttempts = 0;
while (expandAttempts < maxExpandAttempts) {
try {
let clicked = false;
for (const selector of moreButtonSelectors) {
try {
const button = page.locator(selector).first();
if (await button.isVisible({ timeout: TIMEOUTS.MORE_BUTTON_VISIBILITY_DOM })) {
await button.click();
await page.waitForTimeout(TIMEOUTS.MORE_BUTTON_CLICK);
console.log(`[Extractor] Clicked "more" button with selector: ${selector}`);
clicked = true;
expandAttempts++;
break;
}
} catch (e) {
// Try next selector
}
}
if (!clicked) break;
} catch (e) {
break;
}
}
}
/**
* Clean up extracted text - removes HTML tags, decodes entities, cleans whitespace
*/
export function cleanText(text: string): string {
let cleaned = text;
@@ -292,7 +438,9 @@ async function extractFromEmbeddedJSON(
}
// Try __additionalDataLoaded pattern
const additionalDataMatch = content.match(/window\.__additionalDataLoaded\([^,]+,\s*(\{.+?\})\);/s);
const additionalDataMatch = content.match(
/window\.__additionalDataLoaded\([^,]+,\s*(\{.+?\})\);/s
);
if (additionalDataMatch) {
console.log(`[Extractor] Found __additionalDataLoaded in script ${i}`);
try {
@@ -309,7 +457,9 @@ async function extractFromEmbeddedJSON(
// Try to find any large JSON with caption data (new Instagram format)
if ((content.includes('"caption"') || content.includes('"text"')) && content.length > 10000) {
console.log(`[Extractor] Attempting to extract from large JSON in script ${i} (length: ${content.length})`);
console.log(
`[Extractor] Attempting to extract from large JSON in script ${i} (length: ${content.length})`
);
try {
// Try to parse as direct JSON
const jsonData = JSON.parse(content);
@@ -317,7 +467,9 @@ async function extractFromEmbeddedJSON(
// Try deep search first
const deepResult = deepSearchForCaption(jsonData);
if (deepResult && deepResult.bodyText && deepResult.bodyText.length > 130) {
console.log(`[Extractor] Deep search in JSON found caption: ${deepResult.bodyText.length} chars`);
console.log(
`[Extractor] Deep search in JSON found caption: ${deepResult.bodyText.length} chars`
);
const thumbnail = await extractThumbnailStealth(page, progressCallback);
return { ...deepResult, thumbnail };
}
@@ -325,7 +477,9 @@ async function extractFromEmbeddedJSON(
// Try standard parsing
const result = parseInstagramData(jsonData);
if (result && result.bodyText && result.bodyText.length > 130) {
console.log(`[Extractor] Successfully extracted from JSON, text length: ${result.bodyText.length}`);
console.log(
`[Extractor] Successfully extracted from JSON, text length: ${result.bodyText.length}`
);
const thumbnail = await extractThumbnailStealth(page, progressCallback);
return { ...result, thumbnail };
}
@@ -336,7 +490,7 @@ async function extractFromEmbeddedJSON(
const patterns = [
/"caption"\s*:\s*\{\s*"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"/, // Escaped quotes
/"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"\s*,?\s*"pk"/, // text field near pk
/"edge_media_to_caption"\s*:\s*\{\s*"edges"\s*:\s*\[\s*\{\s*"node"\s*:\s*\{\s*"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"/,
/"edge_media_to_caption"\s*:\s*\{\s*"edges"\s*:\s*\[\s*\{\s*"node"\s*:\s*\{\s*"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"/
];
for (const pattern of patterns) {
@@ -347,11 +501,15 @@ async function extractFromEmbeddedJSON(
const captionText = rawText
.replace(/\\n/g, '\n')
.replace(/\\"/g, '"')
.replace(/\\u([0-9a-fA-F]{4})/g, (_, code) => String.fromCharCode(parseInt(code, 16)))
.replace(/\\u([0-9a-fA-F]{4})/g, (_, code) =>
String.fromCharCode(parseInt(code, 16))
)
.replace(/\\\\/g, '\\');
if (captionText.length > 130) {
console.log(`[Extractor] Extracted caption from regex pattern, length: ${captionText.length}`);
console.log(
`[Extractor] Extracted caption from regex pattern, length: ${captionText.length}`
);
const thumbnail = await extractThumbnailStealth(page, progressCallback);
return { bodyText: cleanText(captionText), thumbnail };
}
@@ -448,7 +606,9 @@ export async function extractFromHTMLSection(
const currentShortcode = extractShortcode(currentUrl);
console.log(`[Extractor] Current page URL: ${currentUrl}`);
console.log(`[Extractor] Target shortcode: ${targetShortcode}, Current shortcode: ${currentShortcode}`);
console.log(
`[Extractor] Target shortcode: ${targetShortcode}, Current shortcode: ${currentShortcode}`
);
if (targetShortcode && currentShortcode !== targetShortcode) {
console.log(`[Extractor] URL mismatch: expected ${targetShortcode}, got ${currentShortcode}`);
@@ -458,62 +618,13 @@ export async function extractFromHTMLSection(
console.log(`[Extractor] Confirmed on correct post: ${currentShortcode}`);
// Wait for network to settle
await page.waitForLoadState('domcontentloaded', { timeout: 10000 });
await page.waitForTimeout(2000);
await page.waitForLoadState('domcontentloaded', { timeout: TIMEOUTS.PAGE_LOAD });
await page.waitForTimeout(TIMEOUTS.NETWORK_SETTLE);
// Try to expand truncated caption by clicking "more" button
// STRATEGY: Since we're already on the correct page (URL validated above),
// the FIRST article/main post container should be our target post.
// Instagram uses JS routing so links don't have shortcodes in hrefs.
console.log('[Extractor] Looking for "more" button in primary post container...');
try {
// Wait for content to load
await page.waitForTimeout(1500);
// Find the MAIN post container - should be the first article or main content area
const mainContainer = page.locator('article, main, [role="main"]').first();
const containerExists = await mainContainer.count() > 0;
if (containerExists) {
console.log('[Extractor] Found main post container, searching for "more" button...');
// Try different patterns for the "more" button within the main container
const morePatterns = [
{ locator: mainContainer.locator('span').filter({ hasText: /\.\.\.\s*more/i }), desc: "span with '...more'" },
{ locator: mainContainer.locator('span').filter({ hasText: /…\s*more/i }), desc: "span with '… more'" },
{ locator: mainContainer.locator('div[role="button"]').filter({ hasText: /more/i }), desc: "button with 'more'" },
{ locator: mainContainer.locator('span[role="button"]').filter({ hasText: /more/i }), desc: "span button with 'more'" }
];
for (const pattern of morePatterns) {
const count = await pattern.locator.count();
console.log(`[Extractor] Checking ${pattern.desc}: found ${count}`);
if (count > 0) {
const firstMore = pattern.locator.first();
try {
if (await firstMore.isVisible({ timeout: 1000 })) {
const text = await firstMore.textContent();
console.log(`[Extractor] Found visible "more": "${text}"`);
await firstMore.click();
console.log('[Extractor] Clicked "more" - waiting for expansion...');
await page.waitForTimeout(3000);
console.log('[Extractor] Caption expansion complete');
break; // Success!
}
} catch (e) {
console.log(`[Extractor] ${pattern.desc} not clickable: ${e}`);
}
}
}
} else {
console.log('[Extractor] No main container found');
}
console.log('[Extractor] Finished "more" button expansion attempt');
} catch (e) {
console.log(`[Extractor] Error while trying to expand caption: ${e}`);
}
await tryExpandCaptionInHTMLSection(page);
console.log('[Extractor] Extracting caption using intelligent span detection...');
@@ -538,9 +649,10 @@ export async function extractFromHTMLSection(
// If we found links to the post, search for spans within those link ancestors
const searchRoots: Element[] = [];
if (postLinks.length > 0) {
postLinks.forEach(link => {
postLinks.forEach((link) => {
// Get the article or section container for this post
let container = link.closest('article') || link.closest('section') || link.closest('[role="main"]');
let container =
link.closest('article') || link.closest('section') || link.closest('[role="main"]');
if (container && !searchRoots.includes(container)) {
searchRoots.push(container);
console.log(`[Extractor] Found container for target post`);
@@ -555,8 +667,8 @@ export async function extractFromHTMLSection(
}
const spans: HTMLElement[] = [];
searchRoots.forEach(root => {
root.querySelectorAll('span').forEach(span => spans.push(span as HTMLElement));
searchRoots.forEach((root) => {
root.querySelectorAll('span').forEach((span) => spans.push(span as HTMLElement));
});
console.log(`[Extractor] Searching ${spans.length} spans for recipe content`);
@@ -584,7 +696,7 @@ export async function extractFromHTMLSection(
score += brCount * 100; // Massive weight for line breaks
// Check for recipe keywords (strong indicator)
const hasKeywords = recipeKeywords.some(keyword => text.includes(keyword));
const hasKeywords = recipeKeywords.some((keyword) => text.includes(keyword));
if (hasKeywords) {
score += 500; // Huge boost for recipe keywords
}
@@ -616,7 +728,9 @@ export async function extractFromHTMLSection(
// Update best candidate
if (score > 0 && (!bestCandidate || score > bestCandidate.score)) {
console.log(`[Extractor] New best: score=${score}, len=${text.length}, br=${brCount}, links=${linkCount}, preview="${text.substring(0, 80)}..."`);
console.log(
`[Extractor] New best: score=${score}, len=${text.length}, br=${brCount}, links=${linkCount}, preview="${text.substring(0, 80)}..."`
);
bestCandidate = {
element: span,
text: span.textContent || '',
@@ -638,7 +752,9 @@ export async function extractFromHTMLSection(
// Explicit type assertion (safe after null guard)
const candidate: CaptionCandidate = bestCandidate;
console.log(`[Extractor] Final caption candidate: score=${candidate.score}, length=${candidate.text.length}`);
console.log(
`[Extractor] Final caption candidate: score=${candidate.score}, length=${candidate.text.length}`
);
// Extract text from the best candidate
// Use innerHTML to preserve <br> tags, which will be converted to newlines in cleanText
@@ -698,15 +814,15 @@ export async function extractFromDOM(
try {
// Give Instagram more time to load dynamic content
console.log('[Extractor] Waiting for network idle...');
await page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {
await page.waitForLoadState('networkidle', { timeout: TIMEOUTS.PAGE_LOAD }).catch(() => {
console.log('[Extractor] Network idle timeout, continuing anyway');
});
// Try to wait for article content
await page.waitForSelector('article', { timeout: 5000 }).catch(() => {});
await page.waitForSelector('article', { timeout: TIMEOUTS.ARTICLE_SELECTOR }).catch(() => {});
// Additional wait for dynamic content
await page.waitForTimeout(2000);
await page.waitForTimeout(TIMEOUTS.NETWORK_SETTLE);
// Try to intercept GraphQL responses
let graphqlCaption: string | null = null;
@@ -715,11 +831,12 @@ export async function extractFromDOM(
if (url.includes('graphql') || url.includes('api/v1')) {
try {
const json = await response.json();
// Try to find caption in the response
const captionData = extractCaptionFromGraphQL(json);
if (captionData && captionData.length > 130) {
graphqlCaption = captionData;
console.log(`[Extractor] Intercepted GraphQL response with ${captionData.length} chars`);
console.log(
`[Extractor] Intercepted GraphQL response with ${captionData.length} chars`
);
}
} catch (e) {
// Not JSON or parsing failed
@@ -727,54 +844,15 @@ export async function extractFromDOM(
}
});
// Wait a bit for any GraphQL requests to complete
await page.waitForTimeout(1000);
await page.waitForTimeout(TIMEOUTS.GRAPHQL_WAIT);
if (graphqlCaption) {
const thumbnail = await extractThumbnailStealth(page, progressCallback);
return { bodyText: cleanText(graphqlCaption), thumbnail };
}
// First, try to expand truncated captions by clicking "more" button
// Try multiple times with different selectors
let expandAttempts = 0;
const maxExpandAttempts = 3;
while (expandAttempts < maxExpandAttempts) {
try {
const moreButtonSelectors = [
'article button:has-text("more")',
'article button:has-text("More")',
'article button:has-text("… more")',
'article span[role="button"]:has-text("more")',
'article [role="button"]:has-text("more")',
'article div[role="button"]:has-text("more")',
'xpath=//article//span[contains(text(), "more")]/..',
'xpath=//article//button[contains(., "more")]'
];
let clicked = false;
for (const selector of moreButtonSelectors) {
try {
const button = page.locator(selector).first();
if (await button.isVisible({ timeout: 500 })) {
await button.click();
await page.waitForTimeout(800);
console.log(`[Extractor] Clicked "more" button with selector: ${selector}`);
clicked = true;
expandAttempts++;
break;
}
} catch (e) {
// Try next selector
}
}
if (!clicked) break; // No more buttons found
} catch (e) {
break;
}
}
// Try to expand truncated captions by clicking "more" button
await tryExpandCaptionInDOM(page);
const captionText = await page.evaluate(() => {
// First check og:description for comparison
@@ -787,7 +865,9 @@ export async function extractFromDOM(
// SMART APPROACH: Find the truncated text first, then look for full version nearby
// Look for text that ends with "..." or "… more"
const allSpans = Array.from(document.querySelectorAll('article span, article div, article h1'));
const allSpans = Array.from(
document.querySelectorAll('article span, article div, article h1')
);
let longestText = '';
let matchedElement = null;
@@ -809,9 +889,12 @@ export async function extractFromDOM(
}
// Strategy 2: Look in data attributes
const elementsWithData = Array.from(document.querySelectorAll('[data-caption], [data-text], [data-content]'));
const elementsWithData = Array.from(
document.querySelectorAll('[data-caption], [data-text], [data-content]')
);
for (const el of elementsWithData) {
const dataCaption = el.getAttribute('data-caption') ||
const dataCaption =
el.getAttribute('data-caption') ||
el.getAttribute('data-text') ||
el.getAttribute('data-content');
if (dataCaption && dataCaption.length > longestText.length) {
@@ -821,7 +904,11 @@ export async function extractFromDOM(
}
// Strategy 3: Look for hidden/collapsed content
const hiddenElements = Array.from(document.querySelectorAll('[style*="display: none"], [style*="display:none"], .collapsed, [aria-hidden="true"]'));
const hiddenElements = Array.from(
document.querySelectorAll(
'[style*="display: none"], [style*="display:none"], .collapsed, [aria-hidden="true"]'
)
);
for (const el of hiddenElements) {
const text = el.textContent?.trim() || '';
if (text.length > longestText.length && text.length > 200) {
@@ -838,7 +925,9 @@ export async function extractFromDOM(
const parentText = parent.textContent?.trim() || '';
if (parentText.length > longestText.length) {
longestText = parentText;
console.log(`[Extractor] Found fuller text in parent element: ${parentText.length} chars`);
console.log(
`[Extractor] Found fuller text in parent element: ${parentText.length} chars`
);
}
}
@@ -864,7 +953,10 @@ export async function extractFromDOM(
// Fallback to og:description
if (metaDesc) {
const content = ogContent;
const cleanedContent = content.replace(/^\d+K?\s+likes,\s+\d+\s+comments\s+-\s+[\w.]+\s+on\s+[^:]+:\s*["']?/, '');
const cleanedContent = content.replace(
/^\d+K?\s+likes,\s+\d+\s+comments\s+-\s+[\w.]+\s+on\s+[^:]+:\s*["']?/,
''
);
console.log('[Extractor] DOM selector fallback: og:description (with metadata cleanup)');
return cleanedContent;
}
@@ -1021,11 +1113,15 @@ async function extractFromInternalState(
}
if (result && result.bodyText && result.bodyText.length > 130) {
console.log(`[Extractor] Successfully extracted from ${stateData.key}, length: ${result.bodyText.length}`);
console.log(
`[Extractor] Successfully extracted from ${stateData.key}, length: ${result.bodyText.length}`
);
const thumbnail = await extractThumbnailStealth(page, progressCallback);
return { ...result, thumbnail };
} else if (result?.bodyText) {
console.log(`[Extractor] Found text in ${stateData.key} but it's truncated (${result.bodyText.length} chars)`);
console.log(
`[Extractor] Found text in ${stateData.key} but it's truncated (${result.bodyText.length} chars)`
);
}
} catch (e) {
console.log(`[Extractor] Failed to parse ${stateData.key}:`, e);
@@ -1042,7 +1138,11 @@ async function extractFromInternalState(
/**
* Deep search for caption text in any nested object structure
*/
function deepSearchForCaption(obj: any, maxDepth = 10, currentDepth = 0): Omit<ExtractedContent, 'thumbnail'> | null {
function deepSearchForCaption(
obj: any,
maxDepth = 10,
currentDepth = 0
): Omit<ExtractedContent, 'thumbnail'> | null {
if (currentDepth > maxDepth || !obj || typeof obj !== 'object') {
return null;
}
@@ -1208,7 +1308,8 @@ export async function extractTextAndThumbnail(
timestamp: new Date().toISOString()
});
return withRetry(async () => {
return withRetry(
async () => {
const authPath = resolveAuthPath();
const context = await createBrowserContext(authPath);
const page = await context.newPage();
@@ -1227,13 +1328,19 @@ export async function extractTextAndThumbnail(
page.on('response', async (response) => {
try {
const responseUrl = response.url();
if (responseUrl.includes('graphql') || responseUrl.includes('api/v1') || responseUrl.includes('/web/')) {
if (
responseUrl.includes('graphql') ||
responseUrl.includes('api/v1') ||
responseUrl.includes('/web/')
) {
try {
const json = await response.json();
const captionData = extractCaptionFromGraphQL(json, expectedShortcode ?? undefined);
if (captionData && captionData.length > 130) {
interceptedCaption = captionData;
console.log(`[Extractor] ✓ Intercepted GraphQL with full caption: ${captionData.length} chars (shortcode verified)`);
console.log(
`[Extractor] ✓ Intercepted GraphQL with full caption: ${captionData.length} chars (shortcode verified)`
);
}
} catch (e) {
// Not JSON or parse error, skip
@@ -1309,7 +1416,10 @@ export async function extractTextAndThumbnail(
await page.close();
await context.close();
}
}, DEFAULT_RETRY_CONFIG, onProgress);
},
DEFAULT_RETRY_CONFIG,
onProgress
);
}
/**

View File

@@ -131,15 +131,19 @@ class PushNotificationService {
},
payload,
{
TTL: 60 * 60 * 24, // 24 hours
TTL: 60 * 60 * 24 // 24 hours
}
);
console.log(`[PushService] ✓ Sent notification to ${subscription.endpoint.substring(0, 50)}...`);
console.log(
`[PushService] ✓ Sent notification to ${subscription.endpoint.substring(0, 50)}...`
);
} catch (error) {
// Check if subscription is expired/invalid
if ((error as any).statusCode === 410) {
console.warn(`[PushService] Subscription expired: ${subscription.endpoint.substring(0, 50)}...`);
console.warn(
`[PushService] Subscription expired: ${subscription.endpoint.substring(0, 50)}...`
);
throw new Error('Subscription expired');
}

View File

@@ -8,13 +8,15 @@ const RecipeSchema = z.object({
name: z.string(),
servings: z.number().nullable(),
description: z.string().nullable(),
ingredients: z.array(
ingredients: z
.array(
z.object({
item: z.string(),
amount: z.string(),
unit: z.string()
})
).nullable(),
)
.nullable(),
steps: z.array(z.string()).nullable(),
image: z.string().nullable().optional()
});
@@ -59,9 +61,9 @@ export async function detectRecipe(text: string): Promise<boolean> {
// Check if this is a model-related error
const errorMessage = (e as Error).message || '';
const isModelError = errorMessage.includes('400') &&
(errorMessage.toLowerCase().includes('model') ||
errorMessage.toLowerCase().includes('load'));
const isModelError =
errorMessage.includes('400') &&
(errorMessage.toLowerCase().includes('model') || errorMessage.toLowerCase().includes('load'));
if (isModelError) {
const { model } = createLLM();
@@ -116,9 +118,9 @@ export async function parseRecipe(text: string): Promise<Recipe> {
// Check if this is a model-related error
const errorMessage = (e as Error).message || '';
const isModelError = errorMessage.includes('400') &&
(errorMessage.toLowerCase().includes('model') ||
errorMessage.toLowerCase().includes('load'));
const isModelError =
errorMessage.includes('400') &&
(errorMessage.toLowerCase().includes('model') || errorMessage.toLowerCase().includes('load'));
if (isModelError) {
const { model } = createLLM();
@@ -129,8 +131,10 @@ export async function parseRecipe(text: string): Promise<Recipe> {
}
// If structured output fails, try standard completion
if ((e as any).message?.includes('response_format') ||
(e as any).message?.includes('structured output')) {
if (
(e as any).message?.includes('response_format') ||
(e as any).message?.includes('structured output')
) {
console.warn('[LLM] Falling back to standard completion');
return await parseRecipeWithStandardCompletion(text);
}

View File

@@ -142,11 +142,7 @@ export class QueueManager {
* });
* ```
*/
updateStatus(
itemId: string,
status: QueueItemStatus,
data?: any
): void {
updateStatus(itemId: string, status: QueueItemStatus, data?: any): void {
const item = this.items.get(itemId);
if (!item) return;
@@ -163,7 +159,7 @@ export class QueueManager {
}
// Update phases array
const phaseIndex = item.phases.findIndex(p => p.name === data.phase);
const phaseIndex = item.phases.findIndex((p) => p.name === data.phase);
if (phaseIndex >= 0) {
// Mark previous phases as completed
for (let i = 0; i < phaseIndex; i++) {
@@ -181,7 +177,7 @@ export class QueueManager {
if (status === 'success') {
item.completedAt = now;
// Mark all phases as completed
item.phases.forEach(phase => {
item.phases.forEach((phase) => {
if (phase.status !== 'completed') {
phase.status = 'completed';
phase.completedAt = now;
@@ -193,7 +189,7 @@ export class QueueManager {
item.completedAt = now;
// Mark current phase as error
if (item.currentPhase) {
const phaseIndex = item.phases.findIndex(p => p.name === item.currentPhase);
const phaseIndex = item.phases.findIndex((p) => p.name === item.currentPhase);
if (phaseIndex >= 0) {
item.phases[phaseIndex].status = 'error';
item.phases[phaseIndex].error = data?.error?.message;
@@ -202,7 +198,12 @@ export class QueueManager {
}
// Wrap results in results object
if (data?.extractedText || data?.thumbnail !== undefined || data?.recipe || data?.tandoorRecipeId) {
if (
data?.extractedText ||
data?.thumbnail !== undefined ||
data?.recipe ||
data?.tandoorRecipeId
) {
if (!item.results) {
item.results = {};
}

View File

@@ -115,12 +115,15 @@ export class QueueProcessor {
if (!item) break;
this.activeWorkers++;
console.log(`[QueueProcessor] Starting item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`);
console.log(
`[QueueProcessor] Starting item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`
);
this.processItem(item)
.finally(() => {
this.processItem(item).finally(() => {
this.activeWorkers--;
console.log(`[QueueProcessor] Finished item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`);
console.log(
`[QueueProcessor] Finished item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`
);
// Try to process next item immediately
setTimeout(() => this.processNextBatch(), 0);
});
@@ -164,7 +167,6 @@ export class QueueProcessor {
// Send push notification
await this.sendPushNotification(item, 'success');
} catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
const recoverable = this.isRecoverableError(error);
@@ -393,7 +395,7 @@ export class QueueProcessor {
'fetch failed'
];
return recoverablePatterns.some(pattern => message.includes(pattern));
return recoverablePatterns.some((pattern) => message.includes(pattern));
}
/**

View File

@@ -29,7 +29,9 @@ export const queueConfig = {
/** Web Push notification settings */
push: {
vapidPublicKey: env.VAPID_PUBLIC_KEY || 'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
vapidPublicKey:
env.VAPID_PUBLIC_KEY ||
'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
vapidPrivateKey: env.VAPID_PRIVATE_KEY || 'JwxI_KcsBcehYcTOufMcbVWJjCq1QbH5FJmSyQuG680',
vapidEmail: env.VAPID_EMAIL || 'mailto:admin@example.com'
}

View File

@@ -15,12 +15,7 @@ import type { ProgressEvent } from '$lib/server/extraction';
* - unhealthy: Recoverable error occurred, can be retried
* - error: Non-recoverable error occurred
*/
export type QueueItemStatus =
| 'pending'
| 'in_progress'
| 'success'
| 'unhealthy'
| 'error';
export type QueueItemStatus = 'pending' | 'in_progress' | 'success' | 'unhealthy' | 'error';
/**
* Processing phases for queue items
@@ -28,10 +23,7 @@ export type QueueItemStatus =
* - parsing: Parsing recipe from extracted text
* - uploading: Uploading recipe to Tandoor
*/
export type ProcessingPhase =
| 'extraction'
| 'parsing'
| 'uploading';
export type ProcessingPhase = 'extraction' | 'parsing' | 'uploading';
/**
* Phase progress information

View File

@@ -73,7 +73,9 @@ async function renewInstagramAuth(): Promise<boolean> {
const authPath = resolveAuthPath();
if (!fs.existsSync(authPath)) {
console.warn('[Scheduler] No existing auth.json found. Run gen-auth.js first to set up initial authentication.');
console.warn(
'[Scheduler] No existing auth.json found. Run gen-auth.js first to set up initial authentication.'
);
return false;
}
@@ -115,7 +117,9 @@ async function renewInstagramAuth(): Promise<boolean> {
await context.storageState({ path: authPath });
state.lastRenewalTime = Date.now();
console.log(`[Scheduler] Instagram authentication renewed successfully at ${new Date().toISOString()}`);
console.log(
`[Scheduler] Instagram authentication renewed successfully at ${new Date().toISOString()}`
);
console.log(`[Scheduler] Auth state updated at: ${authPath}`);
return true;
@@ -140,7 +144,9 @@ export async function startScheduler(): Promise<void> {
const config = getConfig();
if (!config.enabled) {
console.log('[Scheduler] Authentication scheduler is disabled (set AUTH_SCHEDULER_ENABLED=true to enable)');
console.log(
'[Scheduler] Authentication scheduler is disabled (set AUTH_SCHEDULER_ENABLED=true to enable)'
);
return;
}
@@ -151,7 +157,9 @@ export async function startScheduler(): Promise<void> {
const intervalMs = config.intervalMinutes * 60 * 1000;
console.log(`[Scheduler] Starting authentication scheduler with ${config.intervalMinutes}min interval`);
console.log(
`[Scheduler] Starting authentication scheduler with ${config.intervalMinutes}min interval`
);
// Schedule periodic renewals
state.intervalId = setInterval(async () => {

View File

@@ -15,38 +15,48 @@ export const TandoorRecipeSchema = z.object({
prep_time: z.string().optional(),
cook_time: z.string().optional(),
waiting_time: z.string().optional(),
steps: z.array(
steps: z
.array(
z.object({
step: z.number(),
instruction: z.string(),
ingredients: z.array(
ingredients: z
.array(
z.object({
food: z.object({
id: z.number(),
name: z.string()
}),
unit: z.object({
unit: z
.object({
id: z.number(),
name: z.string()
}).nullable(),
})
.nullable(),
amount: z.number(),
note: z.string().optional()
})
).optional()
)
.optional()
})
).optional(),
ingredients: z.array(
)
.optional(),
ingredients: z
.array(
z.object({
food: z.object({
name: z.string()
}),
unit: z.object({
unit: z
.object({
name: z.string()
}).nullable(),
})
.nullable(),
amount: z.number(),
note: z.string().optional()
})
).optional()
)
.optional()
});
export type TandoorRecipe = z.infer<typeof TandoorRecipeSchema>;
@@ -104,11 +114,11 @@ interface TandoorRecipeDTO {
*/
async function fetchFromTandoor<T>(
url: string,
options: Partial<RequestInit> = { method: 'GET' },
options: Partial<RequestInit> = { method: 'GET' }
): Promise<{ ok: boolean; data?: T; error?: string }> {
const headers = new Headers({
'Content-Type': 'application/json',
'Accept': 'application/json',
Accept: 'application/json',
Authorization: `Bearer ${tandoorConfig.token}`
});
@@ -153,8 +163,6 @@ async function fetchFromTandoor<T>(
}
}
/**
* Partitions ingredients across steps by distributing them evenly
* When step association is unknown, this spreads ingredients proportionally
@@ -181,7 +189,10 @@ function partitionIngredientsAcrossSteps(
partitions[index % stepCount].push(ingredient);
});
console.debug(`Partitioned ${ingredients.length} ingredients across ${stepCount} steps:`, partitions);
console.debug(
`Partitioned ${ingredients.length} ingredients across ${stepCount} steps:`,
partitions
);
return partitions;
}
@@ -224,10 +235,7 @@ function parseAmount(amountStr: string): number | null {
*/
function buildTandoorRecipeDTO(recipe: ExtractedRecipe): TandoorRecipeDTO {
const stepCount = recipe.steps?.length || 1;
const ingredientPartitions = partitionIngredientsAcrossSteps(
recipe.ingredients || [],
stepCount
);
const ingredientPartitions = partitionIngredientsAcrossSteps(recipe.ingredients || [], stepCount);
const steps: TandoorRecipeDTO['steps'] = (recipe.steps || []).map((instruction, index) => {
// Map ingredients, converting unparseable amounts to 1 q.b.
@@ -235,7 +243,9 @@ function buildTandoorRecipeDTO(recipe: ExtractedRecipe): TandoorRecipeDTO {
const amount = parseAmount(ing.amount);
if (amount === null) {
console.debug(`Converting ingredient with unparseable amount to 1 q.b.: ${ing.item} (${ing.amount})`);
console.debug(
`Converting ingredient with unparseable amount to 1 q.b.: ${ing.item} (${ing.amount})`
);
return {
food: {
name: ing.item
@@ -297,14 +307,11 @@ export async function uploadRecipeWithIngredientsDTO(
console.debug('Uploading recipe with ingredients DTO:', recipeDTO);
// Call the API with the DTO
const recipeResult = await fetchFromTandoor<{ id: number }>(
`/api/recipe/`,
{
const recipeResult = await fetchFromTandoor<{ id: number }>(`/api/recipe/`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(recipeDTO)
}
);
});
if (!recipeResult.ok || !recipeResult.data) {
console.error('Recipe creation failed:', recipeResult.error);
@@ -397,7 +404,9 @@ export async function uploadRecipeImage(
}
console.log(`[Tandoor Upload] Recipe ID: ${recipeId}`);
console.log(`[Tandoor Upload] Image type: ${isDirectUrl(imageUrl) ? 'URL' : isDataUrl(imageUrl) ? 'Base64' : 'Unknown'}`);
console.log(
`[Tandoor Upload] Image type: ${isDirectUrl(imageUrl) ? 'URL' : isDataUrl(imageUrl) ? 'Base64' : 'Unknown'}`
);
console.log(`[Tandoor Upload] Image source: ${imageUrl.substring(0, 100)}...`);
let buffer: Buffer;
@@ -462,26 +471,27 @@ export async function uploadRecipeImage(
formData.append('image', file);
console.log('[Tandoor Upload] Uploading to Tandoor...');
const uploadResponse = await fetch(
`${tandoorConfig.serverUrl}/api/recipe/${recipeId}/image/`,
{
const uploadResponse = await fetch(`${tandoorConfig.serverUrl}/api/recipe/${recipeId}/image/`, {
method: 'PUT',
headers: {
'Authorization': `Bearer ${token}`
Authorization: `Bearer ${token}`
// DO NOT set Content-Type - let fetch set it with boundary
},
body: formData
}
);
});
if (!uploadResponse.ok) {
const errorText = await uploadResponse.text().catch(() => uploadResponse.statusText);
const responseHeaders = JSON.stringify(Object.fromEntries(uploadResponse.headers.entries()));
console.error(`[Tandoor Upload] Failed: ${uploadResponse.status} ${uploadResponse.statusText}`);
console.error(
`[Tandoor Upload] Failed: ${uploadResponse.status} ${uploadResponse.statusText}`
);
console.error(`[Tandoor Upload] Response headers: ${responseHeaders}`);
console.error(`[Tandoor Upload] Response body: ${errorText.substring(0, 500)}`);
console.error(`[Tandoor Upload] File metadata: ${filename}, ${file.size} bytes, ${file.type}`);
console.error(
`[Tandoor Upload] File metadata: ${filename}, ${file.size} bytes, ${file.type}`
);
return {
success: false,

View File

@@ -18,11 +18,11 @@ export const GET = async () => {
// Get current queue items by status
const allItems = queueManager.getAll();
const statusCounts = {
pending: allItems.filter(item => item.status === 'pending').length,
in_progress: allItems.filter(item => item.status === 'in_progress').length,
success: allItems.filter(item => item.status === 'success').length,
error: allItems.filter(item => item.status === 'error').length,
unhealthy: allItems.filter(item => item.status === 'unhealthy').length
pending: allItems.filter((item) => item.status === 'pending').length,
in_progress: allItems.filter((item) => item.status === 'in_progress').length,
success: allItems.filter((item) => item.status === 'success').length,
error: allItems.filter((item) => item.status === 'error').length,
unhealthy: allItems.filter((item) => item.status === 'unhealthy').length
};
const stats = {
@@ -51,11 +51,14 @@ export const GET = async () => {
} catch (error) {
console.error('[Health Check] Error retrieving health status:', error);
return json({
return json(
{
timestamp: new Date().toISOString(),
status: 'unhealthy',
error: error instanceof Error ? error.message : 'Unknown error',
uptime: process.uptime()
}, { status: 500 });
},
{ status: 500 }
);
}
};

View File

@@ -15,16 +15,22 @@ export async function GET() {
message: 'LLM service is accessible'
});
} else {
return json({
return json(
{
status: 'unhealthy',
message: 'LLM service is not accessible'
}, { status: 503 });
},
{ status: 503 }
);
}
} catch (error) {
const errorMessage = error instanceof Error ? error.message : 'Unknown error';
return json({
return json(
{
status: 'error',
message: errorMessage
}, { status: 500 });
},
{ status: 500 }
);
}
}

View File

@@ -32,17 +32,11 @@ export const POST: RequestHandler = async ({ request }) => {
// Validate required fields
if (!subscription || !subscription.endpoint || !subscription.keys) {
return json(
{ error: 'Invalid subscription object' },
{ status: 400 }
);
return json({ error: 'Invalid subscription object' }, { status: 400 });
}
if (!clientId || typeof clientId !== 'string') {
return json(
{ error: 'Client ID is required' },
{ status: 400 }
);
return json({ error: 'Client ID is required' }, { status: 400 });
}
// Subscribe client
@@ -61,13 +55,9 @@ export const POST: RequestHandler = async ({ request }) => {
message: 'Successfully subscribed to push notifications',
subscriptionCount: pushNotificationService.getSubscriptionCount()
});
} catch (error) {
console.error('[NotificationAPI] Subscription error:', error);
return json(
{ error: 'Failed to subscribe to notifications' },
{ status: 500 }
);
return json({ error: 'Failed to subscribe to notifications' }, { status: 500 });
}
};
@@ -86,10 +76,7 @@ export const DELETE: RequestHandler = async ({ request }) => {
const { clientId } = await request.json();
if (!clientId || typeof clientId !== 'string') {
return json(
{ error: 'Client ID is required' },
{ status: 400 }
);
return json({ error: 'Client ID is required' }, { status: 400 });
}
// Unsubscribe client
@@ -102,12 +89,8 @@ export const DELETE: RequestHandler = async ({ request }) => {
message: 'Successfully unsubscribed from push notifications',
subscriptionCount: pushNotificationService.getSubscriptionCount()
});
} catch (error) {
console.error('[NotificationAPI] Unsubscription error:', error);
return json(
{ error: 'Failed to unsubscribe from notifications' },
{ status: 500 }
);
return json({ error: 'Failed to unsubscribe from notifications' }, { status: 500 });
}
};

View File

@@ -69,13 +69,11 @@ export const POST: RequestHandler = async ({ request }) => {
message: `Test ${type} notification sent`,
subscriberCount: pushNotificationService.getSubscriptionCount()
});
} catch (error) {
console.error('[NotificationTestAPI] Error sending test notification:',
error instanceof Error ? error.message : String(error));
return json(
{ error: 'Failed to send test notification' },
{ status: 500 }
console.error(
'[NotificationTestAPI] Error sending test notification:',
error instanceof Error ? error.message : String(error)
);
return json({ error: 'Failed to send test notification' }, { status: 500 });
}
};

View File

@@ -25,22 +25,15 @@ export const GET: RequestHandler = async () => {
const publicKey = pushNotificationService.getPublicVapidKey();
if (!publicKey) {
return json(
{ error: 'VAPID public key not configured' },
{ status: 503 }
);
return json({ error: 'VAPID public key not configured' }, { status: 503 });
}
return json({
publicKey,
applicationServerKey: publicKey // Alias for compatibility
});
} catch (error) {
console.error('[NotificationAPI] VAPID key error:', error);
return json(
{ error: 'Failed to get VAPID public key' },
{ status: 500 }
);
return json({ error: 'Failed to get VAPID public key' }, { status: 500 });
}
};

View File

@@ -60,7 +60,6 @@ export const POST: RequestHandler = async ({ request }) => {
status: queueItem.status,
enqueuedAt: queueItem.enqueuedAt
});
} catch (error) {
return handleApiError(error);
}
@@ -122,7 +121,7 @@ export const GET: RequestHandler = async ({ url }) => {
// Apply status filter
if (statusFilter) {
items = items.filter(item => item.status === statusFilter);
items = items.filter((item) => item.status === statusFilter);
}
// Sort by enqueued time (newest first)
@@ -130,7 +129,7 @@ export const GET: RequestHandler = async ({ url }) => {
// Apply pagination
const paginatedItems = items.slice(offset, offset + limit);
const hasMore = (offset + limit) < items.length;
const hasMore = offset + limit < items.length;
return json({
items: paginatedItems,
@@ -142,7 +141,6 @@ export const GET: RequestHandler = async ({ url }) => {
count: paginatedItems.length
}
});
} catch (error) {
return handleApiError(error);
}

View File

@@ -42,7 +42,6 @@ export const GET: RequestHandler = async ({ params }) => {
// Return full item details
return json(queueItem);
} catch (error) {
return handleApiError(error);
}
@@ -78,9 +77,7 @@ export const DELETE: RequestHandler = async ({ params }) => {
// Prevent deletion of in-progress items
if (existingItem.status === 'in_progress') {
throw new ConflictError(
'Cannot delete item that is currently being processed'
);
throw new ConflictError('Cannot delete item that is currently being processed');
}
// Remove the item
@@ -90,7 +87,6 @@ export const DELETE: RequestHandler = async ({ params }) => {
success,
message: 'Queue item removed successfully'
});
} catch (error) {
return handleApiError(error);
}

View File

@@ -63,7 +63,6 @@ export const POST: RequestHandler = async ({ params }) => {
item: updatedItem,
message: 'Queue item has been reset and will be reprocessed'
});
} catch (error) {
return handleApiError(error);
}

View File

@@ -108,10 +108,10 @@ export const GET: RequestHandler = async ({ url, request }) => {
// Apply filters
if (itemIdFilter) {
filteredItems = currentItems.filter(item => item.id === itemIdFilter);
filteredItems = currentItems.filter((item) => item.id === itemIdFilter);
}
if (statusFilter) {
filteredItems = filteredItems.filter(item => item.status === statusFilter);
filteredItems = filteredItems.filter((item) => item.status === statusFilter);
}
// Send initial state for each matching item

View File

@@ -40,4 +40,4 @@ export const POST: RequestHandler = async ({ request }) => {
{ status: 500 }
);
}
}
};

View File

@@ -169,9 +169,7 @@ self.addEventListener('push', (event) => {
const title = data.title || getNotificationTitle(data.type, data);
event.waitUntil(
self.registration.showNotification(title, options)
);
event.waitUntil(self.registration.showNotification(title, options));
});
// Handle notification clicks
@@ -195,8 +193,7 @@ self.addEventListener('notificationclick', (event) => {
}
event.waitUntil(
clients.matchAll({ type: 'window', includeUncontrolled: true })
.then((clientsList) => {
clients.matchAll({ type: 'window', includeUncontrolled: true }).then((clientsList) => {
// Check if there's already a window/tab open
for (const client of clientsList) {
if (client.url.includes(self.location.origin) && 'focus' in client) {

View File

@@ -29,8 +29,8 @@ describe('extraction.ts logging', () => {
expect(calls.length).toBeGreaterThan(0);
// Verify at least one call has the expected format
const errorCall = calls.find((call: any[]) =>
call[0]?.match(/\[.*\]/) && call[1] !== undefined
const errorCall = calls.find(
(call: any[]) => call[0]?.match(/\[.*\]/) && call[1] !== undefined
);
expect(errorCall).toBeDefined();
expect(errorCall[0]).toMatch(/\[.*\]/); // Has prefix like [Retry], [Extractor], etc
@@ -49,14 +49,11 @@ describe('extraction.ts logging', () => {
}
// Check all console.warn and console.error calls
const allCalls = [
...consoleWarnSpy.mock.calls,
...consoleErrorSpy.mock.calls
];
const allCalls = [...consoleWarnSpy.mock.calls, ...consoleErrorSpy.mock.calls];
const errorCalls = allCalls
.map(call => call.join(' '))
.filter(msg => msg.includes('[object Object]'));
.map((call) => call.join(' '))
.filter((msg) => msg.includes('[object Object]'));
expect(errorCalls).toHaveLength(0);
});
@@ -78,9 +75,7 @@ describe('extraction.ts logging', () => {
// Call real logError
logger.logError('[Test] Real test', mockError);
const output = consoleErrorSpy.mock.calls
.map(call => call.join(' '))
.join(' ');
const output = consoleErrorSpy.mock.calls.map((call) => call.join(' ')).join(' ');
// Should not contain [object Object]
expect(output).not.toContain('[object Object]');

View File

@@ -31,7 +31,8 @@ describe('Instagram Caption Extraction E2E', () => {
const page = await context.newPage();
try {
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
const testUrl =
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
console.log('[DEBUG] Navigating to:', testUrl);
await page.goto(testUrl, { waitUntil: 'domcontentloaded' });
@@ -71,7 +72,6 @@ describe('Instagram Caption Extraction E2E', () => {
console.log(`\n[DEBUG] Shortcode appears ${htmlMatches} times in page HTML`);
expect(true).toBe(true);
} finally {
await page.close();
await context.close();
@@ -84,7 +84,8 @@ describe('Instagram Caption Extraction E2E', () => {
const page = await context.newPage();
try {
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
const testUrl =
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
console.log('[DEBUG] Navigating to:', testUrl);
await page.goto(testUrl, { waitUntil: 'domcontentloaded' });
@@ -96,7 +97,10 @@ describe('Instagram Caption Extraction E2E', () => {
// Try to find and click "more" button
console.log('[DEBUG] Looking for "more" button...');
const moreElements = await page.locator('span, div, button').filter({ hasText: /more/i }).all();
const moreElements = await page
.locator('span, div, button')
.filter({ hasText: /more/i })
.all();
console.log(`[DEBUG] Found ${moreElements.length} elements with "more"`);
for (let i = 0; i < Math.min(moreElements.length, 10); i++) {
@@ -126,7 +130,7 @@ describe('Instagram Caption Extraction E2E', () => {
const spanData = await page.evaluate(() => {
const spans = Array.from(document.querySelectorAll('span'));
return spans
.filter(s => (s.textContent || '').length > 30)
.filter((s) => (s.textContent || '').length > 30)
.map((s, idx) => ({
index: idx,
text: (s.textContent || '').substring(0, 200),
@@ -139,13 +143,14 @@ describe('Instagram Caption Extraction E2E', () => {
});
console.log('[DEBUG] Top spans by LENGTH after click attempt:');
spanData.slice(0, 5).forEach(span => {
console.log(` [${span.index}] BR:${span.brCount} Links:${span.linkCount} Len:${span.length}`);
spanData.slice(0, 5).forEach((span) => {
console.log(
` [${span.index}] BR:${span.brCount} Links:${span.linkCount} Len:${span.length}`
);
console.log(` Text: "${span.text}"`);
});
expect(true).toBe(true); // Dummy assertion
} finally {
await page.close();
await context.close();
@@ -156,7 +161,8 @@ describe('Instagram Caption Extraction E2E', () => {
// Instagram's current anti-scraping measures make full extraction difficult
// This test validates that we try all available methods
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
const testUrl =
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
const result = await extractTextAndThumbnail(testUrl);
@@ -191,7 +197,8 @@ describe('Instagram Caption Extraction E2E', () => {
}, 30000);
it('should handle extraction attempt and return truncated text gracefully', async () => {
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
const testUrl =
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
const result = await extractTextAndThumbnail(testUrl);

View File

@@ -91,7 +91,8 @@ describe('extractFromDOM() with mocked og:description', () => {
it('should remove metadata prefix from og:description fallback', async () => {
// Exact fixture from context_compact.yaml
const ogContent = '16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
const ogContent =
'16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
const mockPage = createMockPage(ogContent);
@@ -104,7 +105,8 @@ describe('extractFromDOM() with mocked og:description', () => {
});
it('should remove opening quote after metadata prefix', async () => {
const ogContent = '16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
const ogContent =
'16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
const mockPage = createMockPage(ogContent);
@@ -168,7 +170,8 @@ describe('Integration: Full extraction flow', () => {
it('should extract, clean metadata prefix, remove quotes, and clean hashtags', async () => {
// Simulating what the browser's page.evaluate() would return AFTER cleaning metadata
// (the browser regex already strips the metadata prefix and quotes)
const browserCleanedContent = 'La cacio e pepe infallibile di Luciano Monosilio 🍝 #cacio #pepe #recipe';
const browserCleanedContent =
'La cacio e pepe infallibile di Luciano Monosilio 🍝 #cacio #pepe #recipe';
const mockPage = createMockPage(browserCleanedContent);
@@ -197,7 +200,8 @@ describe('Integration: Full extraction flow', () => {
it('should handle full real-world caption with multiline content', async () => {
// Browser has already cleaned metadata, only hashtags remain
const browserCleanedContent = 'La cacio e pepe\n\nIngredients:\n- Pasta\n- Cheese\n\n#recipe #pasta';
const browserCleanedContent =
'La cacio e pepe\n\nIngredients:\n- Pasta\n- Cheese\n\n#recipe #pasta';
const mockPage = createMockPage(browserCleanedContent);

View File

@@ -76,9 +76,6 @@ describe('llm.ts logging', () => {
await checkModelAvailability('test-model');
expect(logErrorSpy).toHaveBeenCalledWith(
'[LLM] Model availability check failed',
complexError
);
expect(logErrorSpy).toHaveBeenCalledWith('[LLM] Model availability check failed', complexError);
});
});

View File

@@ -149,10 +149,7 @@ describe('logger utilities', () => {
logObject('[Test]', obj);
expect(consoleLogSpy).toHaveBeenCalledWith(
'[Test]',
expect.stringContaining('[Circular]')
);
expect(consoleLogSpy).toHaveBeenCalledWith('[Test]', expect.stringContaining('[Circular]'));
});
});
});

View File

@@ -20,7 +20,9 @@ describe('POST /api/notifications/test', () => {
// Spy on pushNotificationService methods
sendNotificationSpy = vi.spyOn(pushNotificationService, 'sendNotification').mockResolvedValue();
getSubscriptionCountSpy = vi.spyOn(pushNotificationService, 'getSubscriptionCount').mockReturnValue(2);
getSubscriptionCountSpy = vi
.spyOn(pushNotificationService, 'getSubscriptionCount')
.mockReturnValue(2);
});
test('should validate notification type - reject invalid type', async () => {
@@ -179,7 +181,7 @@ describe('POST /api/notifications/test', () => {
const call1 = sendNotificationSpy.mock.calls[0][0];
// Wait a bit to ensure different timestamp
await new Promise(resolve => setTimeout(resolve, 2));
await new Promise((resolve) => setTimeout(resolve, 2));
await POST({ request: request2 } as any);
const call2 = sendNotificationSpy.mock.calls[1][0];

View File

@@ -47,10 +47,7 @@ describe('parser.ts logging', () => {
// Expected to throw
}
expect(logErrorSpy).toHaveBeenCalledWith(
'[LLM] Recipe detection error',
expect.any(Error)
);
expect(logErrorSpy).toHaveBeenCalledWith('[LLM] Recipe detection error', expect.any(Error));
});
test('parseRecipe should use logError on failure', async () => {
@@ -60,10 +57,7 @@ describe('parser.ts logging', () => {
// Expected to throw
}
expect(logErrorSpy).toHaveBeenCalledWith(
'[LLM] Recipe parsing error',
expect.any(Error)
);
expect(logErrorSpy).toHaveBeenCalledWith('[LLM] Recipe parsing error', expect.any(Error));
});
test('should not log stack trace separately', async () => {
@@ -73,8 +67,9 @@ describe('parser.ts logging', () => {
// Expected to throw
}
const stackCalls = consoleErrorSpy.mock.calls
.filter((call: any) => call[0]?.includes('Stack trace'));
const stackCalls = consoleErrorSpy.mock.calls.filter((call: any) =>
call[0]?.includes('Stack trace')
);
expect(stackCalls).toHaveLength(0);
});

View File

@@ -98,11 +98,9 @@ describe('PushNotificationService web-push integration', () => {
body: 'Progress update'
});
expect(webpush.sendNotification).toHaveBeenCalledWith(
expect.any(Object),
expect.any(String),
{ TTL: 60 * 60 * 24 }
);
expect(webpush.sendNotification).toHaveBeenCalledWith(expect.any(Object), expect.any(String), {
TTL: 60 * 60 * 24
});
});
test('should serialize notification data as JSON', async () => {
@@ -165,7 +163,8 @@ describe('PushNotificationService web-push integration', () => {
test('should log endpoint prefix only (privacy)', async () => {
const consoleSpy = vi.spyOn(console, 'log');
const longEndpoint = 'https://push.example.com/very-long-endpoint-with-secret-tokens-that-should-not-be-fully-logged-12345678901234567890';
const longEndpoint =
'https://push.example.com/very-long-endpoint-with-secret-tokens-that-should-not-be-fully-logged-12345678901234567890';
const mockSubscription = {
endpoint: longEndpoint,
keys: { p256dh: 'test-key', auth: 'test-auth' }
@@ -182,7 +181,7 @@ describe('PushNotificationService web-push integration', () => {
// Find the log call with endpoint
const endpointLogCall = consoleSpy.mock.calls.find(
call => typeof call[0] === 'string' && call[0].includes('Sent notification to')
(call) => typeof call[0] === 'string' && call[0].includes('Sent notification to')
);
expect(endpointLogCall).toBeTruthy();

View File

@@ -49,10 +49,12 @@ test.describe('Push Notifications E2E', () => {
const subscription = await page.evaluate(async () => {
const registration = await navigator.serviceWorker.ready;
const sub = await registration.pushManager.getSubscription();
return sub ? {
return sub
? {
endpoint: sub.endpoint,
hasKeys: !!(sub as any).keys
} : null;
}
: null;
});
expect(subscription).not.toBeNull();

View File

@@ -13,12 +13,12 @@ import { POST as retryPOST } from '../routes/api/queue/[id]/retry/+server.js';
describe('Queue API Endpoints', () => {
beforeEach(() => {
// Clear queue between tests
queueManager.getAll().forEach(item => queueManager.remove(item.id));
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
});
afterEach(() => {
// Clean up after tests
queueManager.getAll().forEach(item => queueManager.remove(item.id));
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
});
describe('POST /api/queue', () => {
@@ -26,7 +26,7 @@ describe('Queue API Endpoints', () => {
const request = new Request('http://localhost/api/queue', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({
url: 'https://instagram.com/p/ABC123'
@@ -52,7 +52,7 @@ describe('Queue API Endpoints', () => {
const request = new Request('http://localhost/api/queue', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({
url: 'https://www.instagram.com/p/XYZ789'
@@ -98,7 +98,9 @@ describe('Queue API Endpoints', () => {
const response = await queuePOST({ request } as any);
expect(response.status).toBe(200);
const data = await response.json();
expect(data.url).toBe('https://www.instagram.com/reel/DSevV5CDcNm/?utm_source=ig_web_copy_link');
expect(data.url).toBe(
'https://www.instagram.com/reel/DSevV5CDcNm/?utm_source=ig_web_copy_link'
);
});
it('should accept Instagram IGTV URLs', async () => {
@@ -135,17 +137,13 @@ describe('Queue API Endpoints', () => {
});
it('should reject invalid Instagram URL formats', async () => {
const invalidUrls = [
'https://facebook.com/post/123',
'not-a-url',
'https://other-site.com'
];
const invalidUrls = ['https://facebook.com/post/123', 'not-a-url', 'https://other-site.com'];
for (const url of invalidUrls) {
const request = new Request('http://localhost/api/queue', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({ url })
});
@@ -199,7 +197,7 @@ describe('Queue API Endpoints', () => {
const request = new Request('http://localhost/api/queue', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({})
});
@@ -219,7 +217,7 @@ describe('Queue API Endpoints', () => {
const request = new Request('http://localhost/api/queue', {
method: 'POST',
headers: {
'Content-Type': 'text/plain',
'Content-Type': 'text/plain'
},
body: 'not json'
});
@@ -321,10 +319,14 @@ describe('Queue API Endpoints', () => {
let response = await queueGET({ request, url } as any);
expect(response.status).toBe(400);
let data = await response.json();
expect(data.message).toBe('Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error');
expect(data.message).toBe(
'Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error'
);
} catch (err: any) {
expect(err.status).toBe(400);
expect(err.body.message).toBe('Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error');
expect(err.body.message).toBe(
'Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error'
);
}
// Invalid limit (negative)
@@ -485,10 +487,14 @@ describe('Queue API Endpoints', () => {
} as any);
expect(response.status).toBe(409);
const data = await response.json();
expect(data.message).toBe("Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried.");
expect(data.message).toBe(
"Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried."
);
} catch (err: any) {
expect(err.status).toBe(409);
expect(err.body.message).toBe("Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried.");
expect(err.body.message).toBe(
"Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried."
);
}
});

View File

@@ -29,10 +29,7 @@ describe('QueueManager logging', () => {
// Enqueue an item (this will notify subscribers)
manager.enqueue('https://instagram.com/p/test123');
expect(logErrorSpy).toHaveBeenCalledWith(
'[QueueManager] Subscriber error',
expect.any(Error)
);
expect(logErrorSpy).toHaveBeenCalledWith('[QueueManager] Subscriber error', expect.any(Error));
});
test('should serialize complex error objects', () => {
@@ -49,10 +46,7 @@ describe('QueueManager logging', () => {
manager.subscribe(failingCallback);
manager.enqueue('https://instagram.com/p/test456');
expect(logErrorSpy).toHaveBeenCalledWith(
'[QueueManager] Subscriber error',
complexError
);
expect(logErrorSpy).toHaveBeenCalledWith('[QueueManager] Subscriber error', complexError);
});
test('should not prevent other subscribers from being notified on error', () => {
@@ -74,12 +68,9 @@ describe('QueueManager logging', () => {
expect(successCallback).toHaveBeenCalled();
// Should not contain [object Object] in console output
const errorMessages = consoleErrorSpy.mock.calls
.map(call => call.join(' '));
const errorMessages = consoleErrorSpy.mock.calls.map((call) => call.join(' '));
const hasObjectObject = errorMessages.some(msg =>
msg.includes('[object Object]')
);
const hasObjectObject = errorMessages.some((msg) => msg.includes('[object Object]'));
expect(hasObjectObject).toBe(false);
});

View File

@@ -22,7 +22,6 @@ import * as extraction from '$lib/server/extraction';
import { queueProcessor } from '$lib/server/queue/QueueProcessor';
describe('QueueProcessor logging', () => {
let consoleErrorSpy: any;
beforeEach(async () => {
@@ -31,13 +30,13 @@ describe('QueueProcessor logging', () => {
// Clear queue
const items = queueManager.getAll();
items.forEach(item => queueManager.remove(item.id));
items.forEach((item) => queueManager.remove(item.id));
// Setup console.error spy
consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
// Give time for cleanup
await new Promise(resolve => setTimeout(resolve, 100));
await new Promise((resolve) => setTimeout(resolve, 100));
});
afterEach(() => {
@@ -59,34 +58,37 @@ describe('QueueProcessor logging', () => {
queueProcessor.start();
// Wait for error status
await vi.waitFor(() => {
await vi.waitFor(
() => {
const updated = queueManager.get(item.id);
return updated?.status === 'error' || updated?.status === 'unhealthy';
}, { timeout: 5000 });
},
{ timeout: 5000 }
);
// Stop processor
queueProcessor.stop();
// Wait a bit for all logs to finish
await new Promise(resolve => setTimeout(resolve, 100));
await new Promise((resolve) => setTimeout(resolve, 100));
// Check that console.error doesn't contain [object Object]
const allCalls = consoleErrorSpy.mock.calls.map((call: any[]) =>
call.map(arg => {
call
.map((arg) => {
if (arg && typeof arg === 'object' && arg.message) {
return arg.message; // Handle Error objects
}
return String(arg);
}).join(' ')
})
.join(' ')
);
const hasObjectObject = allCalls.some((msg: string) => msg.includes('[object Object]'));
expect(hasObjectObject).toBe(false);
// Verify QueueProcessor logs are present
const queueProcessorLogs = allCalls.filter((msg: string) =>
msg.includes('[QueueProcessor]')
);
const queueProcessorLogs = allCalls.filter((msg: string) => msg.includes('[QueueProcessor]'));
expect(queueProcessorLogs.length).toBeGreaterThan(0);
});

View File

@@ -27,7 +27,8 @@ vi.mock('$lib/server/queue/config', () => ({
serverUrl: 'http://localhost:8080'
},
push: {
vapidPublicKey: 'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
vapidPublicKey:
'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
vapidPrivateKey: 'JwxI_KcsBcehYcTOufMcbVWJjCq1QbH5FJmSyQuG680',
vapidEmail: 'mailto:test@example.com'
}
@@ -72,7 +73,7 @@ import '$lib/server/queue/QueueProcessor';
describe('QueueProcessor Integration Tests', () => {
beforeEach(async () => {
// Clear queue
queueManager.getAll().forEach(item => queueManager.remove(item.id));
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
// Reset mocks and their implementations
vi.resetAllMocks();
@@ -191,9 +192,7 @@ describe('QueueProcessor Integration Tests', () => {
}, 10000);
it('should handle extraction errors', async () => {
vi.mocked(extractTextAndThumbnail).mockRejectedValue(
new Error('Network timeout')
);
vi.mocked(extractTextAndThumbnail).mockRejectedValue(new Error('Network timeout'));
const item = queueManager.enqueue('https://instagram.com/p/error');
@@ -249,7 +248,7 @@ describe('QueueProcessor Integration Tests', () => {
await new Promise((resolve) => setTimeout(resolve, 150));
const items = queueManager.getAll();
const inProgress = items.filter(i => i.status === 'in_progress');
const inProgress = items.filter((i) => i.status === 'in_progress');
// With concurrency=2, should have max 2 in progress at once
expect(inProgress.length).toBeLessThanOrEqual(2);
@@ -258,7 +257,7 @@ describe('QueueProcessor Integration Tests', () => {
await new Promise((resolve) => setTimeout(resolve, 2000));
const final = queueManager.getAll();
const completed = final.filter(i => i.status === 'success');
const completed = final.filter((i) => i.status === 'success');
// All 3 should eventually complete
expect(completed.length).toBe(3);

View File

@@ -11,12 +11,12 @@ import { GET as streamGET } from '../routes/api/queue/stream/+server.js';
describe('Queue SSE Stream Endpoint', () => {
beforeEach(() => {
// Clear queue between tests
queueManager.getAll().forEach(item => queueManager.remove(item.id));
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
});
afterEach(() => {
// Clean up after tests
queueManager.getAll().forEach(item => queueManager.remove(item.id));
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
});
describe('GET /api/queue/stream', () => {

View File

@@ -112,7 +112,7 @@ describe('Frontend SSE Parser', () => {
'embedded-json': '📦',
'dom-selector': '🎯',
'graphql-api': '🔌',
'legacy': '📄'
legacy: '📄'
};
return method ? icons[method] || '⚙️' : '⚙️';
};

View File

@@ -24,18 +24,13 @@ describe('tandoor logging', () => {
name: 'Test Recipe',
servings: 4,
description: 'Test description',
ingredients: [
{ item: 'Flour', amount: '2', unit: 'cups' }
],
ingredients: [{ item: 'Flour', amount: '2', unit: 'cups' }],
steps: ['Mix ingredients']
};
await uploadRecipeWithIngredientsDTO(recipe);
expect(logErrorSpy).toHaveBeenCalledWith(
'[Tandoor] Fetch error',
expect.any(Error)
);
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor] Fetch error', expect.any(Error));
});
test('should use logError on API error response', async () => {
@@ -80,10 +75,7 @@ describe('tandoor logging', () => {
await uploadRecipeWithIngredientsDTO(recipe);
expect(logErrorSpy).toHaveBeenCalledWith(
'[Tandoor] Fetch error',
expect.any(Error)
);
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor] Fetch error', expect.any(Error));
});
test('should use logError on image upload failure', async () => {
@@ -93,10 +85,7 @@ describe('tandoor logging', () => {
const result = await uploadRecipeImage(123, 'https://example.com/image.jpg');
expect(result.success).toBe(false);
expect(logErrorSpy).toHaveBeenCalledWith(
'[Tandoor Upload] Exception',
error
);
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor Upload] Exception', error);
});
test('should use logError instead of manual error logging', async () => {
@@ -112,10 +101,7 @@ describe('tandoor logging', () => {
});
// Verify logError was called (which handles stack trace serialization)
expect(logErrorSpy).toHaveBeenCalledWith(
'[Tandoor] Fetch error',
error
);
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor] Fetch error', error);
// logError itself logs stack traces, which is expected behavior
// The key is that tandoor.ts uses logError instead of manual logging

View File

@@ -12,15 +12,15 @@ export default defineConfig({
watch: {
ignored: ['**/debug_page.txt', '**/.ssl/**', '**/docs/**', '**/secrets/**']
},
https: fs.existsSync('./.ssl/localhost.key') && fs.existsSync('./.ssl/localhost.crt')
https:
fs.existsSync('./.ssl/localhost.key') && fs.existsSync('./.ssl/localhost.crt')
? {
key: fs.readFileSync('./.ssl/localhost.key'),
cert: fs.readFileSync('./.ssl/localhost.crt')
}
: undefined
},
plugins: [
tailwindcss(), sveltekit()],
plugins: [tailwindcss(), sveltekit()],
test: {
expect: { requireAssertions: true },
projects: [