simplify
This commit is contained in:
@@ -3,10 +3,7 @@
|
||||
"singleQuote": true,
|
||||
"trailingComma": "none",
|
||||
"printWidth": 100,
|
||||
"plugins": [
|
||||
"prettier-plugin-svelte",
|
||||
"prettier-plugin-tailwindcss"
|
||||
],
|
||||
"plugins": ["prettier-plugin-svelte", "prettier-plugin-tailwindcss"],
|
||||
"overrides": [
|
||||
{
|
||||
"files": "*.svelte",
|
||||
|
||||
22
README.md
22
README.md
@@ -5,6 +5,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
## 🚀 Features
|
||||
|
||||
### Core Functionality
|
||||
|
||||
- **Async Queue Processing**: Fire-and-forget recipe extraction with background processing
|
||||
- **Real-time Updates**: Server-Sent Events for live progress tracking
|
||||
- **Push Notifications**: Background notifications when recipes complete
|
||||
@@ -13,6 +14,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
- **PWA Support**: Installable Progressive Web App with offline capabilities
|
||||
|
||||
### User Experience
|
||||
|
||||
- **Queue Dashboard**: Monitor all recipe extractions in real-time
|
||||
- **Share Integration**: Browser share target for easy URL submission
|
||||
- **Responsive Design**: Works on desktop, tablet, and mobile
|
||||
@@ -20,6 +22,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
- **Progress Tracking**: Visual progress through extraction phases
|
||||
|
||||
### Technical Architecture
|
||||
|
||||
- **SvelteKit Frontend**: Modern reactive UI with TypeScript
|
||||
- **Hexagonal Architecture**: Clean separation of concerns
|
||||
- **In-Memory Queue**: High-performance processing with configurable concurrency
|
||||
@@ -29,6 +32,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
## 📋 API Endpoints
|
||||
|
||||
### Queue Management
|
||||
|
||||
- `POST /api/queue` - Enqueue Instagram URL for processing
|
||||
- `GET /api/queue` - List queue items with filtering and pagination
|
||||
- `GET /api/queue/{id}` - Get specific queue item details
|
||||
@@ -36,17 +40,20 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
- `GET /api/queue/stream` - Server-Sent Events for real-time updates
|
||||
|
||||
### Push Notifications
|
||||
|
||||
- `POST /api/notifications/subscribe` - Subscribe to push notifications
|
||||
- `DELETE /api/notifications/subscribe` - Unsubscribe from notifications
|
||||
- `GET /api/notifications/vapid-key` - Get VAPID public key
|
||||
|
||||
### Legacy Endpoints (Deprecated)
|
||||
|
||||
- ~~`POST /api/extract`~~ - Use `/api/queue` instead
|
||||
- ~~`GET /api/extract-stream`~~ - Use `/api/queue/stream` instead
|
||||
|
||||
## 🛠 Development Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js 18+
|
||||
- npm or pnpm
|
||||
- Tandoor Recipe Manager instance (optional)
|
||||
@@ -79,6 +86,7 @@ open https://localhost:5173
|
||||
```
|
||||
|
||||
The app runs on HTTPS by default for:
|
||||
|
||||
- Service worker support (required for PWA)
|
||||
- Push notifications
|
||||
- Browser share target API
|
||||
@@ -89,6 +97,7 @@ The app runs on HTTPS by default for:
|
||||
The application uses HTTPS in development with SSL certificates signed by an external Caddy CA container. The current certificate is valid until **December 20, 2035** (10 years).
|
||||
|
||||
**Certificate Information:**
|
||||
|
||||
- Location: `.ssl/` directory
|
||||
- CA Certificate: `.ssl/root.crt` (already trusted on the system)
|
||||
- Server Certificate: `.ssl/localhost.crt`
|
||||
@@ -97,18 +106,21 @@ The application uses HTTPS in development with SSL certificates signed by an ext
|
||||
Since the Caddy CA is already trusted on the system, the certificate should work without additional trust steps. If you encounter browser warnings:
|
||||
|
||||
**Linux (Ubuntu/Debian):**
|
||||
|
||||
```bash
|
||||
sudo cp .ssl/root.crt /usr/local/share/ca-certificates/caddy-local.crt
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
**Chrome/Chromium:**
|
||||
|
||||
1. Go to `chrome://settings/certificates`
|
||||
2. Click "Authorities" → "Import"
|
||||
3. Select `.ssl/root.crt`
|
||||
4. Check "Trust this certificate for identifying websites"
|
||||
|
||||
**Checking Certificate Expiration:**
|
||||
|
||||
```bash
|
||||
openssl x509 -enddate -noout -in .ssl/localhost.crt
|
||||
```
|
||||
@@ -220,6 +232,7 @@ To enable web push notifications:
|
||||
## 🏗 Architecture Overview
|
||||
|
||||
### Queue System
|
||||
|
||||
```
|
||||
User submits URL → Queue Manager → Queue Processor
|
||||
↓
|
||||
@@ -257,6 +270,7 @@ npm run test:watch
|
||||
```
|
||||
|
||||
Test Coverage:
|
||||
|
||||
- **138 total tests** covering all major components
|
||||
- Queue Manager: 28 tests
|
||||
- Queue Processor: 5 integration tests
|
||||
@@ -279,11 +293,13 @@ npm run preview
|
||||
### Deployment
|
||||
|
||||
The app is built as a Node.js application with the following outputs:
|
||||
|
||||
- `/.svelte-kit/output/server/` - Server bundle
|
||||
- `/.svelte-kit/output/client/` - Static assets
|
||||
- `/build/` - Adapter output
|
||||
|
||||
Deploy the server bundle with:
|
||||
|
||||
```bash
|
||||
node build/index.js
|
||||
```
|
||||
@@ -307,6 +323,7 @@ CMD ["node", "build"]
|
||||
The app was migrated from a synchronous extraction system to an async queue-based system:
|
||||
|
||||
**Before (Synchronous)**:
|
||||
|
||||
- User waited for entire extraction process to complete
|
||||
- No progress tracking during processing
|
||||
- No retry capability for failures
|
||||
@@ -314,6 +331,7 @@ The app was migrated from a synchronous extraction system to an async queue-base
|
||||
- Limited error handling
|
||||
|
||||
**After (Async Queue)**:
|
||||
|
||||
- Fire-and-forget: submit URL and redirect immediately
|
||||
- Real-time progress tracking via SSE
|
||||
- Comprehensive retry system for failures
|
||||
@@ -324,12 +342,14 @@ The app was migrated from a synchronous extraction system to an async queue-base
|
||||
### API Migration
|
||||
|
||||
**Old Synchronous Endpoints** (Deprecated):
|
||||
|
||||
```bash
|
||||
POST /api/extract # Submit URL and wait for completion
|
||||
GET /api/extract-stream # Long-polling for progress
|
||||
```
|
||||
|
||||
**New Queue Endpoints**:
|
||||
|
||||
```bash
|
||||
POST /api/queue # Submit URL, get queue ID immediately
|
||||
GET /api/queue # List all queue items
|
||||
@@ -351,6 +371,7 @@ If migrating from the old system:
|
||||
### Backward Compatibility
|
||||
|
||||
The legacy endpoints are still available but deprecated:
|
||||
|
||||
- They will return `410 Gone` status with migration instructions
|
||||
- Support will be removed in a future version
|
||||
- All new development should use the queue endpoints
|
||||
@@ -383,4 +404,3 @@ This project is licensed under the MIT License - see the LICENSE file for detail
|
||||
- [Tandoor Recipe Manager](https://docs.tandoor.dev/) - Recipe management system
|
||||
- [Workbox](https://developers.google.com/web/tools/workbox) - PWA capabilities
|
||||
- [fastq](https://github.com/mcollina/fastq) - High-performance queue processing
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ services:
|
||||
container_name: insta-recipe
|
||||
network_mode: host
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- '3000:3000'
|
||||
environment:
|
||||
# LLM Configuration (Required)
|
||||
- OPENAI_BASE_URL=${OPENAI_BASE_URL}
|
||||
@@ -40,7 +40,13 @@ services:
|
||||
- ./secrets:/app/secrets
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "node", "-e", "fetch('http://localhost:3000/api/health').then(r => r.ok ? process.exit(0) : process.exit(1)).catch(() => process.exit(1))"]
|
||||
test:
|
||||
[
|
||||
'CMD',
|
||||
'node',
|
||||
'-e',
|
||||
"fetch('http://localhost:3000/api/health').then(r => r.ok ? process.exit(0) : process.exit(1)).catch(() => process.exit(1))"
|
||||
]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
@@ -2,7 +2,7 @@ services:
|
||||
app:
|
||||
build: .
|
||||
ports:
|
||||
- "5173:5173"
|
||||
- '5173:5173'
|
||||
environment:
|
||||
- PLAYWRIGHT_WS_ENDPOINT=ws://playwright-service:3000
|
||||
- OPENAI_BASE_URL=http://ollama:11434/v1
|
||||
@@ -18,7 +18,7 @@ services:
|
||||
playwright-service:
|
||||
build: ./playwright-service
|
||||
ipc: host
|
||||
ports: ["3000:3000"]
|
||||
ports: ['3000:3000']
|
||||
environment:
|
||||
- DISPLAY=:99
|
||||
security_opt:
|
||||
@@ -26,7 +26,7 @@ services:
|
||||
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
ports: ["11434:11434"]
|
||||
ports: ['11434:11434']
|
||||
volumes:
|
||||
- ollama_data:/root/.ollama
|
||||
|
||||
|
||||
46
docs/API.md
46
docs/API.md
@@ -5,6 +5,7 @@ This document describes the InstaRecipe API endpoints for the async queue-based
|
||||
## Base URL
|
||||
|
||||
All API endpoints are relative to your InstaRecipe instance:
|
||||
|
||||
```
|
||||
https://your-instarecipe-instance.com/api
|
||||
```
|
||||
@@ -25,11 +26,14 @@ All endpoints return standardized error responses:
|
||||
{
|
||||
"error": "Error type",
|
||||
"message": "Human-readable error message",
|
||||
"details": { /* Additional error context */ }
|
||||
"details": {
|
||||
/* Additional error context */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
HTTP status codes follow REST conventions:
|
||||
|
||||
- `200` - Success
|
||||
- `201` - Created
|
||||
- `400` - Bad Request (invalid input)
|
||||
@@ -45,6 +49,7 @@ HTTP status codes follow REST conventions:
|
||||
Enqueue an Instagram URL for async processing.
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"url": "https://instagram.com/p/abc123"
|
||||
@@ -52,6 +57,7 @@ Enqueue an Instagram URL for async processing.
|
||||
```
|
||||
|
||||
**Supported URL Formats:**
|
||||
|
||||
- Posts: `https://instagram.com/p/{post-id}`
|
||||
- Posts (www): `https://www.instagram.com/p/{post-id}`
|
||||
- Reels: `https://instagram.com/reel/{reel-id}`
|
||||
@@ -59,12 +65,14 @@ Enqueue an Instagram URL for async processing.
|
||||
- With query parameters: `https://instagram.com/reel/{reel-id}?utm_source=share`
|
||||
|
||||
**URL Requirements:**
|
||||
|
||||
- Must use HTTPS protocol
|
||||
- Hostname must be `instagram.com` or `www.instagram.com`
|
||||
- Any Instagram path is accepted (posts, reels, IGTV, etc.)
|
||||
- Query parameters and hash fragments are allowed
|
||||
|
||||
**Examples:**
|
||||
|
||||
```json
|
||||
// Post URL
|
||||
{ "url": "https://instagram.com/p/ABC123" }
|
||||
@@ -77,6 +85,7 @@ Enqueue an Instagram URL for async processing.
|
||||
```
|
||||
|
||||
**Response (201 Created):**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
@@ -105,6 +114,7 @@ Enqueue an Instagram URL for async processing.
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
|
||||
- `400` - Invalid URL format (not a valid URL)
|
||||
- `400` - URL must use HTTPS protocol
|
||||
- `400` - URL must be from instagram.com domain
|
||||
@@ -115,6 +125,7 @@ Enqueue an Instagram URL for async processing.
|
||||
List queue items with optional filtering, pagination, and sorting.
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
- `status` (optional): Filter by status (`pending`, `in_progress`, `success`, `error`, `unhealthy`)
|
||||
- `limit` (optional): Number of items to return (default: 50, max: 100)
|
||||
- `offset` (optional): Number of items to skip (default: 0)
|
||||
@@ -122,6 +133,7 @@ List queue items with optional filtering, pagination, and sorting.
|
||||
- `order` (optional): Sort order (`asc`, `desc`) (default: `desc`)
|
||||
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
GET /api/queue # All items
|
||||
GET /api/queue?status=error # Failed items only
|
||||
@@ -130,6 +142,7 @@ GET /api/queue?sort=status&order=asc # Sort by status
|
||||
```
|
||||
|
||||
**Response (200 OK):**
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
@@ -199,12 +212,14 @@ GET /api/queue?sort=status&order=asc # Sort by status
|
||||
Get details for a specific queue item.
|
||||
|
||||
**Path Parameters:**
|
||||
|
||||
- `id`: Queue item UUID
|
||||
|
||||
**Response (200 OK):**
|
||||
Returns the same queue item structure as in the list response.
|
||||
|
||||
**Errors:**
|
||||
|
||||
- `400` - Invalid UUID format
|
||||
- `404` - Queue item not found
|
||||
|
||||
@@ -213,9 +228,11 @@ Returns the same queue item structure as in the list response.
|
||||
Retry a failed queue item. Only items with `error` or `unhealthy` status can be retried.
|
||||
|
||||
**Path Parameters:**
|
||||
|
||||
- `id`: Queue item UUID
|
||||
|
||||
**Response (200 OK):**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@@ -229,6 +246,7 @@ Retry a failed queue item. Only items with `error` or `unhealthy` status can be
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
|
||||
- `400` - Invalid UUID format
|
||||
- `404` - Queue item not found
|
||||
- `409` - Cannot retry item with current status (only `error` and `unhealthy` can be retried)
|
||||
@@ -240,10 +258,12 @@ Retry a failed queue item. Only items with `error` or `unhealthy` status can be
|
||||
Server-Sent Events (SSE) endpoint for real-time queue updates.
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
- `itemId` (optional): Filter updates for specific item
|
||||
- `status` (optional): Filter updates by status
|
||||
|
||||
**Headers:**
|
||||
|
||||
```
|
||||
Accept: text/event-stream
|
||||
Cache-Control: no-cache
|
||||
@@ -253,14 +273,18 @@ Cache-Control: no-cache
|
||||
SSE stream with the following event types:
|
||||
|
||||
#### connection
|
||||
|
||||
Sent when connection is established:
|
||||
|
||||
```
|
||||
event: connection
|
||||
data: {"message": "Connected to queue stream", "timestamp": "2024-12-21T10:30:00Z"}
|
||||
```
|
||||
|
||||
#### queue-update
|
||||
|
||||
Sent when queue item status changes:
|
||||
|
||||
```
|
||||
event: queue-update
|
||||
data: {
|
||||
@@ -279,7 +303,9 @@ data: {
|
||||
```
|
||||
|
||||
#### ping
|
||||
|
||||
Keep-alive ping sent every 30 seconds:
|
||||
|
||||
```
|
||||
event: ping
|
||||
data: {"timestamp": "2024-12-21T10:30:30Z"}
|
||||
@@ -288,6 +314,7 @@ data: {"timestamp": "2024-12-21T10:30:30Z"}
|
||||
**Usage Examples:**
|
||||
|
||||
**JavaScript:**
|
||||
|
||||
```javascript
|
||||
const eventSource = new EventSource('/api/queue/stream');
|
||||
|
||||
@@ -312,6 +339,7 @@ eventSource.onerror = (error) => {
|
||||
```
|
||||
|
||||
**curl:**
|
||||
|
||||
```bash
|
||||
curl -N -H "Accept: text/event-stream" \
|
||||
"https://your-instance.com/api/queue/stream?itemId=550e8400-e29b-41d4-a716-446655440000"
|
||||
@@ -324,6 +352,7 @@ curl -N -H "Accept: text/event-stream" \
|
||||
Get the VAPID public key required for push notification subscriptions.
|
||||
|
||||
**Response (200 OK):**
|
||||
|
||||
```json
|
||||
{
|
||||
"publicKey": "BDummyPublicKeyForDevelopment...",
|
||||
@@ -336,6 +365,7 @@ Get the VAPID public key required for push notification subscriptions.
|
||||
Subscribe to push notifications for queue processing updates.
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"subscription": {
|
||||
@@ -350,6 +380,7 @@ Subscribe to push notifications for queue processing updates.
|
||||
```
|
||||
|
||||
**Response (200 OK):**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@@ -359,6 +390,7 @@ Subscribe to push notifications for queue processing updates.
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
|
||||
- `400` - Invalid subscription object or missing clientId
|
||||
|
||||
### DELETE /api/notifications/subscribe
|
||||
@@ -366,6 +398,7 @@ Subscribe to push notifications for queue processing updates.
|
||||
Unsubscribe from push notifications.
|
||||
|
||||
**Request:**
|
||||
|
||||
```json
|
||||
{
|
||||
"clientId": "unique-client-identifier"
|
||||
@@ -373,6 +406,7 @@ Unsubscribe from push notifications.
|
||||
```
|
||||
|
||||
**Response (200 OK):**
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
@@ -390,6 +424,7 @@ Unsubscribe from push notifications.
|
||||
This synchronous extraction endpoint is deprecated. Use `POST /api/queue` instead.
|
||||
|
||||
**Migration:**
|
||||
|
||||
```javascript
|
||||
// ❌ Old synchronous approach
|
||||
const response = await fetch('/api/extract', {
|
||||
@@ -413,6 +448,7 @@ const queueItem = await response.json(); // Immediate response
|
||||
This SSE endpoint is deprecated. Use `POST /api/queue` + `GET /api/queue/stream` instead.
|
||||
|
||||
**Migration:**
|
||||
|
||||
```javascript
|
||||
// ❌ Old approach
|
||||
const response = await fetch('/api/extract-stream', {
|
||||
@@ -485,7 +521,8 @@ interface Recipe {
|
||||
|
||||
keywords?: string[]; // Recipe tags
|
||||
image?: string; // Image URL
|
||||
nutrition?: { // Nutritional information
|
||||
nutrition?: {
|
||||
// Nutritional information
|
||||
calories?: number;
|
||||
protein?: number;
|
||||
carbs?: number;
|
||||
@@ -552,7 +589,6 @@ async function processInstagramUrl(url) {
|
||||
reject(error);
|
||||
};
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Processing failed:', error);
|
||||
throw error;
|
||||
@@ -561,13 +597,13 @@ async function processInstagramUrl(url) {
|
||||
|
||||
// Usage
|
||||
processInstagramUrl('https://instagram.com/p/abc123')
|
||||
.then(results => {
|
||||
.then((results) => {
|
||||
console.log('Recipe extracted:', results.recipe);
|
||||
if (results.tandoorUrl) {
|
||||
console.log('Uploaded to Tandoor:', results.tandoorUrl);
|
||||
}
|
||||
})
|
||||
.catch(error => {
|
||||
.catch((error) => {
|
||||
console.error('Extraction failed:', error.message);
|
||||
});
|
||||
```
|
||||
|
||||
@@ -91,21 +91,27 @@ insta-recipe/
|
||||
## Key Directories
|
||||
|
||||
### `/src/lib/server/`
|
||||
|
||||
Server-side business logic following Hexagonal Architecture principles. Contains domain logic, adapters for external systems (Instagram, Tandoor, LLM), and port definitions.
|
||||
|
||||
### `/src/lib/client/`
|
||||
|
||||
Client-side utilities for PWA features (push notifications, install prompts, service worker messaging).
|
||||
|
||||
### `/src/routes/api/`
|
||||
|
||||
RESTful API endpoints implemented as SvelteKit server routes. Each directory contains `+server.ts` files exporting HTTP verb handlers.
|
||||
|
||||
### `/src/routes/share/`
|
||||
|
||||
Share target page allowing users to share Instagram URLs directly from their browser or mobile apps.
|
||||
|
||||
### `/src/lib/server/queue/`
|
||||
|
||||
Queue management system with in-memory storage, processor workers, and type definitions.
|
||||
|
||||
### `/docs/`
|
||||
|
||||
Comprehensive documentation including plans, outcomes, API specs, and migration guides.
|
||||
|
||||
---
|
||||
@@ -113,33 +119,43 @@ Comprehensive documentation including plans, outcomes, API specs, and migration
|
||||
## Design Patterns
|
||||
|
||||
### Singleton Pattern
|
||||
|
||||
Used for shared service instances:
|
||||
|
||||
- `QueueManager` (`queueManager` exported instance)
|
||||
- `QueueProcessor` (`queueProcessor` exported instance)
|
||||
- `PushNotificationService` (`pushNotificationService` exported instance)
|
||||
- `ServiceWorkerMessageHandler` (`serviceWorkerMessageHandler` exported instance)
|
||||
|
||||
### Factory Pattern
|
||||
|
||||
Used for creating configured instances:
|
||||
|
||||
- `createLLM()` - Creates OpenAI client with environment configuration
|
||||
- `createBrowserContext()` - Creates Playwright browser context with options
|
||||
- `initializeBrowser()` - Initializes Chromium browser instance
|
||||
|
||||
### Observer Pattern
|
||||
|
||||
Implemented in QueueManager for real-time updates:
|
||||
|
||||
- Subscribers receive notifications on queue item changes
|
||||
- Server-Sent Events (SSE) stream queue updates to clients
|
||||
- Push notifications notify users of completion events
|
||||
|
||||
### Adapter Pattern (Hexagonal Architecture)
|
||||
|
||||
External systems accessed via adapters:
|
||||
|
||||
- **Instagram Adapter**: `extraction.ts` - Extracts content via Playwright
|
||||
- **LLM Adapter**: `llm.ts`, `parser.ts` - Recipe parsing via OpenAI
|
||||
- **Tandoor Adapter**: `tandoor.ts` - Recipe management system integration
|
||||
- **Browser Adapter**: `browser.ts` - Playwright browser automation
|
||||
|
||||
### Strategy Pattern
|
||||
|
||||
Multiple extraction strategies with fallback:
|
||||
|
||||
1. Embedded JSON extraction
|
||||
2. DOM selector extraction
|
||||
3. GraphQL API extraction
|
||||
@@ -150,28 +166,34 @@ Multiple extraction strategies with fallback:
|
||||
## Key Components
|
||||
|
||||
### Queue Management System
|
||||
|
||||
**Location**: `src/lib/server/queue/`
|
||||
|
||||
Three-phase processing pipeline:
|
||||
|
||||
1. **Extraction Phase**: Extract text and thumbnail from Instagram
|
||||
2. **Parsing Phase**: Parse recipe using LLM
|
||||
3. **Uploading Phase**: Upload to Tandoor (if enabled)
|
||||
|
||||
**Components**:
|
||||
|
||||
- `QueueManager`: In-memory FIFO queue with CRUD operations
|
||||
- `QueueProcessor`: Worker that processes items with configurable concurrency
|
||||
- `types.ts`: Comprehensive type definitions for queue items and updates
|
||||
|
||||
### API Layer
|
||||
|
||||
**Location**: `src/routes/api/`
|
||||
|
||||
RESTful endpoints for:
|
||||
|
||||
- Queue operations (`POST /api/queue`, `GET /api/queue`, `GET /api/queue/[id]`)
|
||||
- Real-time updates (`GET /api/queue/stream` - SSE)
|
||||
- Push notifications (`POST /api/notifications/subscribe`)
|
||||
- Health checks (`GET /api/health`, `GET /api/llm-health`)
|
||||
|
||||
### Client-Side Services
|
||||
|
||||
**Location**: `src/lib/client/`
|
||||
|
||||
- **PushNotificationManager**: Manages Web Push API subscriptions
|
||||
@@ -179,14 +201,17 @@ RESTful endpoints for:
|
||||
- **ServiceWorkerMessageHandler**: Processes service worker messages
|
||||
|
||||
### Instagram Extraction
|
||||
|
||||
**Location**: `src/lib/server/extraction.ts`
|
||||
|
||||
Multi-method extraction with intelligent fallback:
|
||||
|
||||
- Progress callbacks for real-time feedback
|
||||
- Retry logic with configurable attempts
|
||||
- Thumbnail extraction and validation
|
||||
|
||||
### LLM Integration
|
||||
|
||||
**Location**: `src/lib/server/parser.ts`, `src/lib/server/llm.ts`
|
||||
|
||||
- Recipe detection endpoint
|
||||
@@ -198,6 +223,7 @@ Multi-method extraction with intelligent fallback:
|
||||
## Dependencies
|
||||
|
||||
### Production Dependencies
|
||||
|
||||
- **@types/uuid** (^10.0.0) - UUID type definitions
|
||||
- **date-fns** (^4.1.0) - Date utility library
|
||||
- **openai** (^4.20.0) - OpenAI API client
|
||||
@@ -206,6 +232,7 @@ Multi-method extraction with intelligent fallback:
|
||||
- **zod** (^3.23.0) - Schema validation
|
||||
|
||||
### Development Dependencies
|
||||
|
||||
- **@sveltejs/kit** (^2.48.5) - SvelteKit framework
|
||||
- **@sveltejs/adapter-node** (^5.4.0) - Node.js adapter
|
||||
- **svelte** (^5.43.8) - Svelte 5 framework
|
||||
@@ -223,12 +250,14 @@ Multi-method extraction with intelligent fallback:
|
||||
## Module Organization
|
||||
|
||||
### SvelteKit Path Aliases
|
||||
|
||||
- `$lib` → `src/lib/`
|
||||
- `$lib/*` → `src/lib/*`
|
||||
- `$app/*` → SvelteKit app imports
|
||||
- `$env/dynamic/private` → Environment variables (server-side)
|
||||
|
||||
### Directory Structure Conventions
|
||||
|
||||
- **Server-only code**: `src/lib/server/` (not bundled to client)
|
||||
- **Client-only code**: `src/lib/client/` (not executed on server)
|
||||
- **Shared code**: `src/lib/` (available to both)
|
||||
@@ -240,6 +269,7 @@ Multi-method extraction with intelligent fallback:
|
||||
## Data Flow
|
||||
|
||||
### Recipe Extraction Flow
|
||||
|
||||
```
|
||||
User submits URL
|
||||
↓
|
||||
@@ -261,6 +291,7 @@ SSE updates notify client
|
||||
```
|
||||
|
||||
### Real-time Updates Flow
|
||||
|
||||
```
|
||||
Client connects to GET /api/queue/stream (SSE)
|
||||
↓
|
||||
@@ -274,6 +305,7 @@ Client updates UI reactively
|
||||
```
|
||||
|
||||
### Push Notification Flow
|
||||
|
||||
```
|
||||
Client requests permission
|
||||
↓
|
||||
@@ -295,37 +327,44 @@ Notification displayed to user
|
||||
## Build System
|
||||
|
||||
### Build Command
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
Generates production-ready build in `build/` directory using:
|
||||
|
||||
- Vite for bundling
|
||||
- `@sveltejs/adapter-node` for Node.js deployment
|
||||
- TypeScript compilation
|
||||
- SvelteKit prerendering and optimization
|
||||
|
||||
### Test Command
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
Runs test suite using Vitest with two projects:
|
||||
|
||||
1. **Server tests**: Node environment for server-side code
|
||||
2. **Client tests**: Playwright browser for Svelte components
|
||||
|
||||
### Development Server
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Starts Vite dev server with:
|
||||
|
||||
- HTTPS enabled (certificates in `.ssl/`)
|
||||
- Hot module replacement
|
||||
- TypeScript checking
|
||||
- File watching
|
||||
|
||||
### Linting & Formatting
|
||||
|
||||
```bash
|
||||
npm run lint # ESLint + Prettier check
|
||||
npm run format # Prettier write
|
||||
@@ -336,19 +375,24 @@ npm run format # Prettier write
|
||||
## Deployment
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
Dockerfile includes:
|
||||
|
||||
- Node.js 22 Alpine base image
|
||||
- Playwright Chromium installation
|
||||
- Production build
|
||||
- Port 3000 exposure
|
||||
|
||||
Run with:
|
||||
|
||||
```bash
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Required configuration:
|
||||
|
||||
- `OPENAI_API_KEY` - LLM API access
|
||||
- `TANDOOR_URL` - Tandoor instance URL (optional)
|
||||
- `TANDOOR_TOKEN` - Tandoor API token (optional)
|
||||
@@ -360,13 +404,16 @@ Required configuration:
|
||||
## Testing Architecture
|
||||
|
||||
### Test Categories
|
||||
|
||||
1. **Unit Tests**: Individual function testing
|
||||
2. **Integration Tests**: Multi-component workflows
|
||||
3. **API Tests**: Endpoint behavior validation
|
||||
4. **Browser Tests**: Svelte component rendering
|
||||
|
||||
### Test Coverage
|
||||
|
||||
138 tests covering:
|
||||
|
||||
- Queue management operations
|
||||
- Instagram URL validation
|
||||
- SSE streaming
|
||||
@@ -375,6 +422,7 @@ Required configuration:
|
||||
- Notification service
|
||||
|
||||
### Test Configuration
|
||||
|
||||
- **Server tests**: Node environment with mocked dependencies
|
||||
- **Client tests**: Playwright Chromium browser with Svelte testing library
|
||||
|
||||
@@ -383,15 +431,18 @@ Required configuration:
|
||||
## Security Considerations
|
||||
|
||||
### SSL/TLS
|
||||
|
||||
- Development uses local SSL certificates signed by external Caddy CA
|
||||
- Certificates stored in `.ssl/` (git-ignored)
|
||||
- Required for PWA features (Service Worker, Push API)
|
||||
|
||||
### Authentication
|
||||
|
||||
- Basic auth for scheduled tasks (username/password from environment)
|
||||
- Tandoor integration uses bearer token authentication
|
||||
|
||||
### Input Validation
|
||||
|
||||
- Instagram URL validation with regex patterns
|
||||
- Zod schema validation for API payloads
|
||||
- Error handling with custom error classes
|
||||
|
||||
@@ -19,12 +19,14 @@
|
||||
### Files & Directories
|
||||
|
||||
#### SvelteKit Route Files
|
||||
|
||||
- Route pages: `+page.svelte`
|
||||
- Route servers: `+server.ts`
|
||||
- Route layouts: `+layout.svelte`
|
||||
- Type definitions: `$types.ts` (auto-generated)
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
src/routes/api/queue/
|
||||
├── [id]/
|
||||
@@ -37,19 +39,23 @@ src/routes/api/queue/
|
||||
```
|
||||
|
||||
#### Library Files
|
||||
|
||||
- **PascalCase** for classes and managers: `QueueManager.ts`, `PushNotificationService.ts`
|
||||
- **kebab-case** for utilities and configs: `tandoor-config.ts`, `instagram-url.ts`
|
||||
- **lowercase** for general modules: `browser.ts`, `extraction.ts`, `parser.ts`
|
||||
|
||||
**Examples from codebase:**
|
||||
|
||||
- `src/lib/server/queue/QueueManager.ts`
|
||||
- `src/lib/server/tandoor-config.ts`
|
||||
- `src/lib/client/PushNotificationManager.ts`
|
||||
|
||||
#### Test Files
|
||||
|
||||
Pattern: `<name>.spec.ts` or `<name>.test.ts`
|
||||
|
||||
**Examples:**
|
||||
|
||||
- `queue-manager.spec.ts`
|
||||
- `instagram-url-validation.spec.ts`
|
||||
- `page.svelte.spec.ts`
|
||||
@@ -57,10 +63,12 @@ Pattern: `<name>.spec.ts` or `<name>.test.ts`
|
||||
### Variables & Functions
|
||||
|
||||
#### Variables
|
||||
|
||||
- **camelCase** for local variables and parameters
|
||||
- **SCREAMING_SNAKE_CASE** for constants
|
||||
|
||||
**Examples:**
|
||||
|
||||
```typescript
|
||||
// From QueueManager.ts
|
||||
private items: Map<string, QueueItem> = new Map();
|
||||
@@ -76,10 +84,12 @@ const unsubscribe = queueManager.subscribe(callback);
|
||||
```
|
||||
|
||||
#### Functions
|
||||
|
||||
- **camelCase** for function names
|
||||
- **Descriptive action verbs**: `enqueue`, `extractRecipe`, `uploadRecipeImage`
|
||||
|
||||
**Examples:**
|
||||
|
||||
```typescript
|
||||
// From QueueManager.ts
|
||||
enqueue(url: string): QueueItem { ... }
|
||||
@@ -99,11 +109,13 @@ export async function extractRecipe(text: string): Promise<Recipe> { ... }
|
||||
### Types & Interfaces
|
||||
|
||||
#### Interfaces & Types
|
||||
|
||||
- **PascalCase** for interface names
|
||||
- Prefix with `I` is **NOT** used
|
||||
- Exported types use `export type` or `export interface`
|
||||
|
||||
**Examples:**
|
||||
|
||||
```typescript
|
||||
// From queue/types.ts
|
||||
export interface QueueItem {
|
||||
@@ -121,11 +133,7 @@ export interface QueueStatusUpdate {
|
||||
// ...
|
||||
}
|
||||
|
||||
export type QueueItemStatus =
|
||||
| 'pending'
|
||||
| 'in_progress'
|
||||
| 'completed'
|
||||
| 'failed';
|
||||
export type QueueItemStatus = 'pending' | 'in_progress' | 'completed' | 'failed';
|
||||
|
||||
// From extraction.ts
|
||||
export interface ExtractedContent {
|
||||
@@ -137,16 +145,18 @@ export type ProgressCallback = (event: ProgressEvent) => void;
|
||||
```
|
||||
|
||||
#### Zod Schemas
|
||||
|
||||
- **PascalCase** with `Schema` suffix
|
||||
- Inferred types without suffix
|
||||
|
||||
**Examples:**
|
||||
|
||||
```typescript
|
||||
// From parser.ts
|
||||
const RecipeSchema = z.object({
|
||||
name: z.string(),
|
||||
description: z.string(),
|
||||
servings: z.number(),
|
||||
servings: z.number()
|
||||
// ...
|
||||
});
|
||||
|
||||
@@ -163,10 +173,12 @@ export type TandoorRecipe = z.infer<typeof TandoorRecipeSchema>;
|
||||
### Classes
|
||||
|
||||
#### Class Names
|
||||
|
||||
- **PascalCase** for class names
|
||||
- Descriptive suffixes: `Manager`, `Service`, `Processor`, `Handler`
|
||||
|
||||
**Examples:**
|
||||
|
||||
```typescript
|
||||
// From QueueManager.ts
|
||||
export class QueueManager {
|
||||
@@ -188,6 +200,7 @@ class PushNotificationService {
|
||||
```
|
||||
|
||||
#### Singleton Export Pattern
|
||||
|
||||
```typescript
|
||||
// Class definition
|
||||
export class QueueManager {
|
||||
@@ -203,6 +216,7 @@ export const queueManager = new QueueManager();
|
||||
## Indentation & Formatting
|
||||
|
||||
### General Rules
|
||||
|
||||
- **Indentation:** 2 spaces (enforced by Prettier)
|
||||
- **No tabs**
|
||||
- **Max line length:** 100 characters (soft limit, not enforced)
|
||||
@@ -213,6 +227,7 @@ export const queueManager = new QueueManager();
|
||||
### Code Examples
|
||||
|
||||
#### Function Declarations
|
||||
|
||||
```typescript
|
||||
// From QueueManager.ts
|
||||
enqueue(url: string): QueueItem {
|
||||
@@ -241,6 +256,7 @@ enqueue(url: string): QueueItem {
|
||||
```
|
||||
|
||||
#### Async Functions
|
||||
|
||||
```typescript
|
||||
// From extraction.ts
|
||||
export async function extractTextAndThumbnail(
|
||||
@@ -261,6 +277,7 @@ export async function extractTextAndThumbnail(
|
||||
```
|
||||
|
||||
#### Object Destructuring
|
||||
|
||||
```typescript
|
||||
// From route handlers
|
||||
export const POST: RequestHandler = async ({ request }) => {
|
||||
@@ -279,12 +296,14 @@ export const GET: RequestHandler = async ({ params }) => {
|
||||
## Import Patterns
|
||||
|
||||
### Import Order
|
||||
|
||||
1. External dependencies (Node.js built-ins, npm packages)
|
||||
2. SvelteKit imports (`$lib`, `$app`, `$env`)
|
||||
3. Relative imports (`./ `, `../`)
|
||||
4. Type imports (separate from value imports when beneficial)
|
||||
|
||||
**Example:**
|
||||
|
||||
```typescript
|
||||
// From QueueProcessor.ts
|
||||
|
||||
@@ -307,6 +326,7 @@ import type { QueueItem } from './types';
|
||||
### Import Styles
|
||||
|
||||
#### Named Imports (Preferred)
|
||||
|
||||
```typescript
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { queueManager } from '$lib/server/queue/QueueManager';
|
||||
@@ -314,12 +334,14 @@ import { validateInstagramUrl } from '$lib/server/validation/instagram-url';
|
||||
```
|
||||
|
||||
#### Type-Only Imports
|
||||
|
||||
```typescript
|
||||
import type { RequestHandler } from './$types';
|
||||
import type { QueueItem, QueueItemStatus } from './types';
|
||||
```
|
||||
|
||||
#### Default Imports
|
||||
|
||||
```typescript
|
||||
import OpenAI from 'openai';
|
||||
import fs from 'fs';
|
||||
@@ -329,6 +351,7 @@ import path from 'path';
|
||||
### Export Patterns
|
||||
|
||||
#### Named Exports (Preferred)
|
||||
|
||||
```typescript
|
||||
// Export functions
|
||||
export async function extractRecipe(text: string): Promise<Recipe> { ... }
|
||||
@@ -345,6 +368,7 @@ export interface QueueItem { ... }
|
||||
```
|
||||
|
||||
#### Singleton Pattern Export
|
||||
|
||||
```typescript
|
||||
// Define class
|
||||
export class QueueManager { ... }
|
||||
@@ -358,10 +382,12 @@ export const queueManager = new QueueManager();
|
||||
## Comments & Documentation
|
||||
|
||||
### JSDoc Style
|
||||
|
||||
Used extensively for public APIs and exported functions.
|
||||
|
||||
**Function Documentation:**
|
||||
```typescript
|
||||
|
||||
````typescript
|
||||
/**
|
||||
* Add URL to processing queue
|
||||
*
|
||||
@@ -377,10 +403,11 @@ Used extensively for public APIs and exported functions.
|
||||
enqueue(url: string): QueueItem {
|
||||
// Implementation
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
**Class Documentation:**
|
||||
```typescript
|
||||
|
||||
````typescript
|
||||
/**
|
||||
* Singleton queue manager for processing Instagram URLs
|
||||
*
|
||||
@@ -402,9 +429,10 @@ enqueue(url: string): QueueItem {
|
||||
export class QueueManager {
|
||||
// Implementation
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
**Module-Level Documentation:**
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Queue Manager - Core queue operations and event management
|
||||
@@ -421,19 +449,21 @@ export class QueueManager {
|
||||
### Inline Comments
|
||||
|
||||
#### Single-line Comments
|
||||
|
||||
```typescript
|
||||
// Set restrictive permissions
|
||||
fs.chmodSync(authFile, 0o600);
|
||||
|
||||
// FIFO order - get oldest pending item
|
||||
const pendingItems = Array.from(this.items.values())
|
||||
.filter(item => item.status === 'pending');
|
||||
const pendingItems = Array.from(this.items.values()).filter((item) => item.status === 'pending');
|
||||
```
|
||||
|
||||
#### Block Comments (Avoided)
|
||||
|
||||
Single-line comments preferred. Block comments used only for large comment blocks or temporarily disabling code during development.
|
||||
|
||||
### TODO Comments
|
||||
|
||||
```typescript
|
||||
// TODO: Add retry logic with exponential backoff
|
||||
// FIXME: Handle race condition when multiple workers dequeue
|
||||
@@ -446,6 +476,7 @@ Single-line comments preferred. Block comments used only for large comment block
|
||||
### Type Safety
|
||||
|
||||
#### Strict Mode Enabled
|
||||
|
||||
```json
|
||||
// tsconfig.json
|
||||
{
|
||||
@@ -457,6 +488,7 @@ Single-line comments preferred. Block comments used only for large comment block
|
||||
```
|
||||
|
||||
#### Type Annotations
|
||||
|
||||
```typescript
|
||||
// Explicit return types for public functions
|
||||
export async function extractRecipe(text: string): Promise<Recipe> { ... }
|
||||
@@ -469,28 +501,17 @@ const items = queueManager.getAll(); // Type inferred
|
||||
```
|
||||
|
||||
### Union Types
|
||||
|
||||
```typescript
|
||||
export type QueueItemStatus =
|
||||
| 'pending'
|
||||
| 'in_progress'
|
||||
| 'completed'
|
||||
| 'failed';
|
||||
export type QueueItemStatus = 'pending' | 'in_progress' | 'completed' | 'failed';
|
||||
|
||||
export type ProcessingPhase =
|
||||
| 'extraction'
|
||||
| 'parsing'
|
||||
| 'uploading';
|
||||
export type ProcessingPhase = 'extraction' | 'parsing' | 'uploading';
|
||||
|
||||
export type ProgressEventType =
|
||||
| 'status'
|
||||
| 'method'
|
||||
| 'retry'
|
||||
| 'error'
|
||||
| 'thumbnail'
|
||||
| 'complete';
|
||||
export type ProgressEventType = 'status' | 'method' | 'retry' | 'error' | 'thumbnail' | 'complete';
|
||||
```
|
||||
|
||||
### Generics
|
||||
|
||||
```typescript
|
||||
// Generic function
|
||||
async function fetchFromTandoor<T>(
|
||||
@@ -508,6 +529,7 @@ async function fetchFromTandoor<T>(
|
||||
### Runes (Reactivity)
|
||||
|
||||
#### $state (Reactive Variables)
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let count = $state(0);
|
||||
@@ -516,6 +538,7 @@ async function fetchFromTandoor<T>(
|
||||
```
|
||||
|
||||
#### $props (Component Props)
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let {
|
||||
@@ -533,6 +556,7 @@ async function fetchFromTandoor<T>(
|
||||
```
|
||||
|
||||
#### $derived (Computed Values)
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let count = $state(0);
|
||||
@@ -541,6 +565,7 @@ async function fetchFromTandoor<T>(
|
||||
```
|
||||
|
||||
#### $effect (Side Effects)
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let url = $state('');
|
||||
@@ -552,6 +577,7 @@ async function fetchFromTandoor<T>(
|
||||
```
|
||||
|
||||
### Component Structure
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
// Imports
|
||||
@@ -593,6 +619,7 @@ async function fetchFromTandoor<T>(
|
||||
## Error Handling
|
||||
|
||||
### Custom Error Classes
|
||||
|
||||
```typescript
|
||||
// From api/errors.ts
|
||||
export class ValidationError extends Error {
|
||||
@@ -618,6 +645,7 @@ export class ConflictError extends Error {
|
||||
```
|
||||
|
||||
### Try-Catch Pattern
|
||||
|
||||
```typescript
|
||||
export const POST: RequestHandler = async ({ request }) => {
|
||||
try {
|
||||
@@ -629,7 +657,6 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
|
||||
const item = queueManager.enqueue(url);
|
||||
return json(item, { status: 201 });
|
||||
|
||||
} catch (error) {
|
||||
return handleApiError(error);
|
||||
}
|
||||
@@ -641,6 +668,7 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
## Linting Configuration
|
||||
|
||||
### ESLint
|
||||
|
||||
**Config:** `eslint.config.js`
|
||||
|
||||
- Base: `@eslint/js` recommended
|
||||
@@ -649,6 +677,7 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
- Formatting: `eslint-config-prettier`
|
||||
|
||||
**Rules:**
|
||||
|
||||
```javascript
|
||||
{
|
||||
rules: {
|
||||
@@ -658,6 +687,7 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
```
|
||||
|
||||
### Prettier
|
||||
|
||||
**Config:** `.prettierrc`
|
||||
|
||||
```json
|
||||
@@ -675,6 +705,7 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
## Testing Conventions
|
||||
|
||||
### Test Structure
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
@@ -701,6 +732,7 @@ describe('QueueManager', () => {
|
||||
```
|
||||
|
||||
### Mock Pattern
|
||||
|
||||
```typescript
|
||||
vi.mock('$lib/server/extraction', () => ({
|
||||
extractTextAndThumbnail: vi.fn().mockResolvedValue({
|
||||
@@ -715,6 +747,7 @@ vi.mock('$lib/server/extraction', () => ({
|
||||
## File Headers
|
||||
|
||||
### Module Documentation Pattern
|
||||
|
||||
Every major module includes a header comment:
|
||||
|
||||
```typescript
|
||||
@@ -730,6 +763,7 @@ Every major module includes a header comment:
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Queue Manager - Core queue operations and event management
|
||||
@@ -748,6 +782,7 @@ Every major module includes a header comment:
|
||||
## Additional Conventions
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```typescript
|
||||
import { env } from '$env/dynamic/private';
|
||||
|
||||
@@ -756,19 +791,24 @@ const tandoorUrl = env.TANDOOR_URL || null;
|
||||
```
|
||||
|
||||
### Date Handling
|
||||
|
||||
ISO8601 strings throughout the application:
|
||||
|
||||
```typescript
|
||||
const now = new Date().toISOString();
|
||||
// Output: "2026-02-15T12:30:45.123Z"
|
||||
```
|
||||
|
||||
### Null vs Undefined
|
||||
|
||||
- `null`: Intentional absence of value
|
||||
- `undefined`: Not yet initialized or optional parameters
|
||||
- Prefer `null` for API responses and data structures
|
||||
|
||||
### Async/Await
|
||||
|
||||
Always preferred over Promise chains:
|
||||
|
||||
```typescript
|
||||
// Preferred
|
||||
async function fetchData() {
|
||||
@@ -780,8 +820,8 @@ async function fetchData() {
|
||||
// Avoid
|
||||
function fetchData() {
|
||||
return fetch(url)
|
||||
.then(response => response.json())
|
||||
.then(data => data);
|
||||
.then((response) => response.json())
|
||||
.then((data) => data);
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
377
docs/FINDINGS.md
377
docs/FINDINGS.md
File diff suppressed because it is too large
Load Diff
@@ -11,6 +11,7 @@ The migration transformed InstaRecipe from a blocking, synchronous extraction sy
|
||||
### Architecture Transformation
|
||||
|
||||
**Before: Synchronous System**
|
||||
|
||||
```
|
||||
User Request → Direct Processing → Response (wait 30-60s)
|
||||
↓ ↓ ↓
|
||||
@@ -18,6 +19,7 @@ User Request → Direct Processing → Response (wait 30-60s)
|
||||
```
|
||||
|
||||
**After: Async Queue System**
|
||||
|
||||
```
|
||||
User Request → Queue Item Created → Immediate Response
|
||||
↓ ↓ ↓
|
||||
@@ -60,6 +62,7 @@ User Request → Queue Item Created → Immediate Response
|
||||
### New Endpoints
|
||||
|
||||
#### Queue Management
|
||||
|
||||
```typescript
|
||||
// Enqueue URL for processing
|
||||
POST /api/queue
|
||||
@@ -84,6 +87,7 @@ Events: connection, queue-update, ping
|
||||
```
|
||||
|
||||
#### Push Notifications
|
||||
|
||||
```typescript
|
||||
// Subscribe to push notifications
|
||||
POST /api/notifications/subscribe
|
||||
@@ -101,11 +105,11 @@ These endpoints are marked for removal and should not be used in new code:
|
||||
|
||||
```typescript
|
||||
// ❌ DEPRECATED: Synchronous extraction
|
||||
POST /api/extract
|
||||
POST / api / extract;
|
||||
// 👉 Use: POST /api/queue
|
||||
|
||||
// ❌ DEPRECATED: Long-polling progress
|
||||
GET /api/extract-stream
|
||||
GET / api / extract - stream;
|
||||
// 👉 Use: GET /api/queue/stream
|
||||
```
|
||||
|
||||
@@ -167,6 +171,7 @@ interface QueueStatusUpdate {
|
||||
### For Frontend Applications
|
||||
|
||||
1. **Replace Synchronous Calls**
|
||||
|
||||
```typescript
|
||||
// ❌ Old synchronous approach
|
||||
const response = await fetch('/api/extract', {
|
||||
@@ -187,6 +192,7 @@ interface QueueStatusUpdate {
|
||||
```
|
||||
|
||||
2. **Add Real-time Updates**
|
||||
|
||||
```typescript
|
||||
// Setup Server-Sent Events for progress tracking
|
||||
const eventSource = new EventSource(`/api/queue/stream?itemId=${itemId}`);
|
||||
@@ -222,6 +228,7 @@ interface QueueStatusUpdate {
|
||||
### For Backend Integrations
|
||||
|
||||
1. **Update API Calls**
|
||||
|
||||
```python
|
||||
# ❌ Old synchronous API
|
||||
response = requests.post('/api/extract', json={'url': url})
|
||||
@@ -240,6 +247,7 @@ interface QueueStatusUpdate {
|
||||
```
|
||||
|
||||
2. **Implement SSE Client** (Python example)
|
||||
|
||||
```python
|
||||
import sseclient
|
||||
|
||||
@@ -314,18 +322,21 @@ npm test queue-sse
|
||||
## Performance Considerations
|
||||
|
||||
### Before Migration
|
||||
|
||||
- **Blocking Operations**: Each request blocked a server thread
|
||||
- **Single Processing**: One extraction at a time
|
||||
- **No Progress**: Users waited without feedback
|
||||
- **Memory Usage**: High memory usage during long operations
|
||||
|
||||
### After Migration
|
||||
|
||||
- **Non-blocking**: Requests return immediately
|
||||
- **Concurrent Processing**: Multiple extractions in parallel
|
||||
- **Real-time Feedback**: Live progress updates
|
||||
- **Efficient Memory**: Event-driven, minimal memory footprint
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
- **Response Time**: 50ms (queue) vs 30-60s (synchronous)
|
||||
- **Throughput**: 2x concurrent processing vs 1x sequential
|
||||
- **User Experience**: Immediate feedback vs long waiting
|
||||
@@ -336,11 +347,13 @@ npm test queue-sse
|
||||
If issues arise, the system can be rolled back by:
|
||||
|
||||
1. **Disable Queue Processing**
|
||||
|
||||
```env
|
||||
QUEUE_PROCESSING_ENABLED=false
|
||||
```
|
||||
|
||||
2. **Re-enable Legacy Endpoints** (if preserved)
|
||||
|
||||
```typescript
|
||||
// Temporary fallback to synchronous processing
|
||||
app.post('/api/extract', legacyExtractHandler);
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
This guide documents SSR-safe patterns and anti-patterns for InstaRecipe development. All developers should follow these guidelines to prevent SSR errors.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Core Principle](#core-principle)
|
||||
- [Browser API Detection](#browser-api-detection)
|
||||
- [Lifecycle Hooks](#lifecycle-hooks)
|
||||
@@ -18,6 +19,7 @@ This guide documents SSR-safe patterns and anti-patterns for InstaRecipe develop
|
||||
**SvelteKit renders components on both the server and client.** Any browser-only APIs must be guarded or used in browser-only contexts.
|
||||
|
||||
### Browser-Only APIs (Require Guards)
|
||||
|
||||
- `window.*`
|
||||
- `document.*`
|
||||
- `localStorage`, `sessionStorage`
|
||||
@@ -72,6 +74,7 @@ if (browser) {
|
||||
### `onMount` - Browser-Only Lifecycle
|
||||
|
||||
**Use `onMount` for:**
|
||||
|
||||
- Browser API initialization
|
||||
- Timer setup (`setInterval`, `setTimeout`)
|
||||
- Event listener registration
|
||||
@@ -160,11 +163,13 @@ onMount(() => {
|
||||
```
|
||||
|
||||
**When to use `$effect`:**
|
||||
|
||||
- Synchronizing derived state
|
||||
- DOM manipulation (with browser guard)
|
||||
- Reactive cleanup
|
||||
|
||||
**When NOT to use `$effect`:**
|
||||
|
||||
- Initialization (use `onMount`)
|
||||
- API calls on mount (use `onMount`)
|
||||
- Timer setup (use `onMount`)
|
||||
@@ -189,11 +194,13 @@ if (browser && eventSource?.readyState === EventSource.OPEN)
|
||||
```
|
||||
|
||||
**EventSource States:**
|
||||
|
||||
- `EventSource.CONNECTING = 0`
|
||||
- `EventSource.OPEN = 1`
|
||||
- `EventSource.CLOSED = 2`
|
||||
|
||||
**WebSocket States:**
|
||||
|
||||
- `WebSocket.CONNECTING = 0`
|
||||
- `WebSocket.OPEN = 1`
|
||||
- `WebSocket.CLOSING = 2`
|
||||
@@ -276,6 +283,7 @@ export class PushNotificationManager {
|
||||
```
|
||||
|
||||
**Why it's good:**
|
||||
|
||||
- Guards all browser API access
|
||||
- Early returns prevent unnecessary code execution during SSR
|
||||
- Defensive programming with null checks
|
||||
@@ -327,6 +335,7 @@ export class PushNotificationManager {
|
||||
```
|
||||
|
||||
**Why it's good:**
|
||||
|
||||
- Uses `onMount` instead of `$effect` for initialization
|
||||
- Timer setup in browser-only context
|
||||
- Proper cleanup with return function
|
||||
@@ -408,6 +417,7 @@ Watch server console for errors during build and preview.
|
||||
### 2. Check for Hydration Warnings
|
||||
|
||||
Open browser DevTools console and look for:
|
||||
|
||||
- "Hydration failed"
|
||||
- "The server response doesn't match the client content"
|
||||
|
||||
@@ -422,6 +432,7 @@ grep -r "navigator\." src/routes --include="*.svelte"
|
||||
```
|
||||
|
||||
Then verify each usage is either:
|
||||
|
||||
- In an event handler (safe)
|
||||
- In `onMount` (safe)
|
||||
- Guarded with `if (browser)` (safe)
|
||||
|
||||
@@ -21,16 +21,14 @@ export default defineConfig(
|
||||
languageOptions: {
|
||||
globals: { ...globals.browser, ...globals.node }
|
||||
},
|
||||
rules: { // typescript-eslint strongly recommend that you do not use the no-undef lint rule on TypeScript projects.
|
||||
rules: {
|
||||
// typescript-eslint strongly recommend that you do not use the no-undef lint rule on TypeScript projects.
|
||||
// see: https://typescript-eslint.io/troubleshooting/faqs/eslint/#i-get-errors-from-the-no-undef-rule-about-global-variables-not-being-defined-even-though-there-are-no-typescript-errors
|
||||
"no-undef": 'off' }
|
||||
'no-undef': 'off'
|
||||
}
|
||||
},
|
||||
{
|
||||
files: [
|
||||
'**/*.svelte',
|
||||
'**/*.svelte.ts',
|
||||
'**/*.svelte.js'
|
||||
],
|
||||
files: ['**/*.svelte', '**/*.svelte.ts', '**/*.svelte.js'],
|
||||
languageOptions: {
|
||||
parserOptions: {
|
||||
projectService: true,
|
||||
|
||||
@@ -15,20 +15,20 @@ export default defineConfig({
|
||||
reporter: 'list',
|
||||
use: {
|
||||
baseURL: 'http://localhost:5173',
|
||||
trace: 'on-first-retry',
|
||||
trace: 'on-first-retry'
|
||||
},
|
||||
|
||||
projects: [
|
||||
{
|
||||
name: 'chromium',
|
||||
use: { ...devices['Desktop Chrome'] },
|
||||
},
|
||||
use: { ...devices['Desktop Chrome'] }
|
||||
}
|
||||
],
|
||||
|
||||
webServer: {
|
||||
command: 'npm run dev',
|
||||
url: 'http://localhost:5173',
|
||||
reuseExistingServer: !process.env.CI,
|
||||
timeout: 120000,
|
||||
},
|
||||
timeout: 120000
|
||||
}
|
||||
});
|
||||
|
||||
@@ -50,7 +50,7 @@ async function generateFaviconIco() {
|
||||
console.log('✓ All validation checks passed');
|
||||
}
|
||||
|
||||
generateFaviconIco().catch(err => {
|
||||
generateFaviconIco().catch((err) => {
|
||||
console.error('Error generating favicon.ico:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
@@ -54,7 +54,7 @@ async function generateFavicon() {
|
||||
console.log('✓ All validation checks passed');
|
||||
}
|
||||
|
||||
generateFavicon().catch(err => {
|
||||
generateFavicon().catch((err) => {
|
||||
console.error('Error generating favicon:', err);
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="manifest" href="/manifest.json" />
|
||||
%sveltekit.head%
|
||||
</head>
|
||||
<body data-sveltekit-preload-data="hover">
|
||||
|
||||
@@ -101,7 +101,7 @@ export class PWAInstallManager {
|
||||
callback(this.canInstall());
|
||||
|
||||
return () => {
|
||||
this.listeners = this.listeners.filter(cb => cb !== callback);
|
||||
this.listeners = this.listeners.filter((cb) => cb !== callback);
|
||||
};
|
||||
}
|
||||
|
||||
@@ -109,7 +109,7 @@ export class PWAInstallManager {
|
||||
* Notify all listeners of state change
|
||||
*/
|
||||
private notifyListeners(canInstall: boolean): void {
|
||||
this.listeners.forEach(callback => {
|
||||
this.listeners.forEach((callback) => {
|
||||
try {
|
||||
callback(canInstall);
|
||||
} catch (error) {
|
||||
|
||||
@@ -67,7 +67,7 @@ class PushNotificationManager {
|
||||
callback(this.state); // Send initial state
|
||||
|
||||
return () => {
|
||||
this.listeners = this.listeners.filter(cb => cb !== callback);
|
||||
this.listeners = this.listeners.filter((cb) => cb !== callback);
|
||||
};
|
||||
}
|
||||
|
||||
@@ -90,11 +90,8 @@ class PushNotificationManager {
|
||||
return;
|
||||
}
|
||||
|
||||
this.state.supported = (
|
||||
'serviceWorker' in navigator &&
|
||||
'PushManager' in window &&
|
||||
'Notification' in window
|
||||
);
|
||||
this.state.supported =
|
||||
'serviceWorker' in navigator && 'PushManager' in window && 'Notification' in window;
|
||||
|
||||
this.state.permission = this.state.supported ? Notification.permission : 'denied';
|
||||
}
|
||||
@@ -164,7 +161,7 @@ class PushNotificationManager {
|
||||
* Subscribe to push notifications
|
||||
*/
|
||||
async subscribe(): Promise<boolean> {
|
||||
if (!await this.requestPermission()) {
|
||||
if (!(await this.requestPermission())) {
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -215,7 +212,6 @@ class PushNotificationManager {
|
||||
|
||||
console.log('[PushManager] Successfully subscribed to push notifications');
|
||||
return true;
|
||||
|
||||
} catch (error) {
|
||||
console.error('[PushManager] Subscription failed:', error);
|
||||
this.state.error = 'Failed to subscribe to notifications';
|
||||
@@ -265,7 +261,6 @@ class PushNotificationManager {
|
||||
|
||||
console.log('[PushManager] Successfully unsubscribed from push notifications');
|
||||
return true;
|
||||
|
||||
} catch (error) {
|
||||
console.error('[PushManager] Unsubscription failed:', error);
|
||||
this.state.error = 'Failed to unsubscribe from notifications';
|
||||
@@ -331,10 +326,8 @@ class PushNotificationManager {
|
||||
|
||||
try {
|
||||
// Add proper padding
|
||||
const padding = '='.repeat((4 - cleanKey.length % 4) % 4);
|
||||
const base64 = (cleanKey + padding)
|
||||
.replace(/-/g, '+')
|
||||
.replace(/_/g, '/');
|
||||
const padding = '='.repeat((4 - (cleanKey.length % 4)) % 4);
|
||||
const base64 = (cleanKey + padding).replace(/-/g, '+').replace(/_/g, '/');
|
||||
|
||||
// Validate base64 format before decoding
|
||||
const base64Regex = /^[A-Za-z0-9+\/]*={0,2}$/;
|
||||
@@ -351,7 +344,6 @@ class PushNotificationManager {
|
||||
|
||||
console.log(`[PushManager] Successfully decoded VAPID key (${outputArray.length} bytes)`);
|
||||
return outputArray;
|
||||
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
console.error('[PushManager] Failed to decode VAPID key:', error, 'Key:', cleanKey);
|
||||
@@ -363,7 +355,7 @@ class PushNotificationManager {
|
||||
* Notify all listeners of state change
|
||||
*/
|
||||
private notifyListeners(): void {
|
||||
this.listeners.forEach(callback => {
|
||||
this.listeners.forEach((callback) => {
|
||||
try {
|
||||
callback({ ...this.state });
|
||||
} catch (error) {
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
* and coordinates with the main application.
|
||||
*/
|
||||
|
||||
import { pushState } from "$app/navigation";
|
||||
import { pushState } from '$app/navigation';
|
||||
|
||||
interface ServiceWorkerMessage {
|
||||
type: string;
|
||||
|
||||
@@ -29,36 +29,46 @@ export function handleApiError(error: unknown): Response {
|
||||
|
||||
// Handle known error types with specific status codes
|
||||
if (error instanceof ValidationError) {
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
message: error.message,
|
||||
type: 'validation_error'
|
||||
}, { status: 400 });
|
||||
},
|
||||
{ status: 400 }
|
||||
);
|
||||
}
|
||||
|
||||
if (error instanceof NotFoundError) {
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
message: error.message,
|
||||
type: 'not_found_error'
|
||||
}, { status: 404 });
|
||||
},
|
||||
{ status: 404 }
|
||||
);
|
||||
}
|
||||
|
||||
if (error instanceof ConflictError) {
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
message: error.message,
|
||||
type: 'conflict_error'
|
||||
}, { status: 409 });
|
||||
},
|
||||
{ status: 409 }
|
||||
);
|
||||
}
|
||||
|
||||
// Handle generic errors
|
||||
const message = error instanceof Error ? error.message : 'Unknown error occurred';
|
||||
|
||||
// Don't expose internal error details in production
|
||||
const publicMessage = process.env.NODE_ENV === 'production'
|
||||
? 'Internal server error'
|
||||
: message;
|
||||
const publicMessage = process.env.NODE_ENV === 'production' ? 'Internal server error' : message;
|
||||
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
message: publicMessage,
|
||||
type: 'server_error'
|
||||
}, { status: 500 });
|
||||
},
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
@@ -9,7 +9,14 @@ export interface ExtractedContent {
|
||||
thumbnail: string | null;
|
||||
}
|
||||
|
||||
export type ExtractionMethod = 'embedded-json' | 'internal-state' | 'html-section' | 'dom-selector' | 'graphql-api' | 'graphql-intercept' | 'legacy';
|
||||
export type ExtractionMethod =
|
||||
| 'embedded-json'
|
||||
| 'internal-state'
|
||||
| 'html-section'
|
||||
| 'dom-selector'
|
||||
| 'graphql-api'
|
||||
| 'graphql-intercept'
|
||||
| 'legacy';
|
||||
|
||||
type CaptionCandidate = {
|
||||
element: Element;
|
||||
@@ -192,7 +199,146 @@ function extractShortcode(url: string): string | undefined {
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean extracted text
|
||||
* Recipe keywords used for caption scoring
|
||||
*/
|
||||
const RECIPE_KEYWORDS = [
|
||||
'ingredienti',
|
||||
'procedimento',
|
||||
'preparazione',
|
||||
'ricetta',
|
||||
'recipe',
|
||||
'instructions'
|
||||
];
|
||||
|
||||
/**
|
||||
* Timeout configuration constants (in milliseconds)
|
||||
*/
|
||||
const TIMEOUTS = {
|
||||
CONTENT_LOAD: 1500,
|
||||
MORE_BUTTON_VISIBILITY: 1000,
|
||||
CAPTION_EXPANSION: 3000,
|
||||
MORE_BUTTON_VISIBILITY_DOM: 500,
|
||||
MORE_BUTTON_CLICK: 800,
|
||||
PAGE_LOAD: 10000,
|
||||
NETWORK_SETTLE: 2000,
|
||||
ARTICLE_SELECTOR: 5000,
|
||||
GRAPHQL_WAIT: 1000,
|
||||
PAGE_NAVIGATION: 30000,
|
||||
ANTI_DETECTION_MIN: 1000,
|
||||
ANTI_DETECTION_MAX: 3000
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* Try to expand truncated caption by clicking "more" button in HTML section method
|
||||
*/
|
||||
async function tryExpandCaptionInHTMLSection(page: Page): Promise<void> {
|
||||
console.log('[Extractor] Looking for "more" button in primary post container...');
|
||||
try {
|
||||
await page.waitForTimeout(TIMEOUTS.CONTENT_LOAD);
|
||||
|
||||
const mainContainer = page.locator('article, main, [role="main"]').first();
|
||||
const containerExists = (await mainContainer.count()) > 0;
|
||||
|
||||
if (!containerExists) {
|
||||
console.log('[Extractor] No main container found');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log('[Extractor] Found main post container, searching for "more" button...');
|
||||
|
||||
const morePatterns = [
|
||||
{
|
||||
locator: mainContainer.locator('span').filter({ hasText: /\.\.\.\s*more/i }),
|
||||
desc: "span with '...more'"
|
||||
},
|
||||
{
|
||||
locator: mainContainer.locator('span').filter({ hasText: /…\s*more/i }),
|
||||
desc: "span with '… more'"
|
||||
},
|
||||
{
|
||||
locator: mainContainer.locator('div[role="button"]').filter({ hasText: /more/i }),
|
||||
desc: "button with 'more'"
|
||||
},
|
||||
{
|
||||
locator: mainContainer.locator('span[role="button"]').filter({ hasText: /more/i }),
|
||||
desc: "span button with 'more'"
|
||||
}
|
||||
];
|
||||
|
||||
for (const pattern of morePatterns) {
|
||||
const count = await pattern.locator.count();
|
||||
console.log(`[Extractor] Checking ${pattern.desc}: found ${count}`);
|
||||
|
||||
if (count === 0) continue;
|
||||
|
||||
const firstMore = pattern.locator.first();
|
||||
try {
|
||||
if (await firstMore.isVisible({ timeout: TIMEOUTS.MORE_BUTTON_VISIBILITY })) {
|
||||
const text = await firstMore.textContent();
|
||||
console.log(`[Extractor] Found visible "more": "${text}"`);
|
||||
await firstMore.click();
|
||||
console.log('[Extractor] Clicked "more" - waiting for expansion...');
|
||||
await page.waitForTimeout(TIMEOUTS.CAPTION_EXPANSION);
|
||||
console.log('[Extractor] Caption expansion complete');
|
||||
break;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(`[Extractor] ${pattern.desc} not clickable: ${e}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('[Extractor] Finished "more" button expansion attempt');
|
||||
} catch (e) {
|
||||
console.log(`[Extractor] Error while trying to expand caption: ${e}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Try to expand truncated caption by clicking "more" button in DOM method
|
||||
*/
|
||||
async function tryExpandCaptionInDOM(page: Page): Promise<void> {
|
||||
const moreButtonSelectors = [
|
||||
'article button:has-text("more")',
|
||||
'article button:has-text("More")',
|
||||
'article button:has-text("… more")',
|
||||
'article span[role="button"]:has-text("more")',
|
||||
'article [role="button"]:has-text("more")',
|
||||
'article div[role="button"]:has-text("more")',
|
||||
'xpath=//article//span[contains(text(), "more")]/..',
|
||||
'xpath=//article//button[contains(., "more")]'
|
||||
];
|
||||
|
||||
const maxExpandAttempts = 3;
|
||||
let expandAttempts = 0;
|
||||
|
||||
while (expandAttempts < maxExpandAttempts) {
|
||||
try {
|
||||
let clicked = false;
|
||||
for (const selector of moreButtonSelectors) {
|
||||
try {
|
||||
const button = page.locator(selector).first();
|
||||
if (await button.isVisible({ timeout: TIMEOUTS.MORE_BUTTON_VISIBILITY_DOM })) {
|
||||
await button.click();
|
||||
await page.waitForTimeout(TIMEOUTS.MORE_BUTTON_CLICK);
|
||||
console.log(`[Extractor] Clicked "more" button with selector: ${selector}`);
|
||||
clicked = true;
|
||||
expandAttempts++;
|
||||
break;
|
||||
}
|
||||
} catch (e) {
|
||||
// Try next selector
|
||||
}
|
||||
}
|
||||
|
||||
if (!clicked) break;
|
||||
} catch (e) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean up extracted text - removes HTML tags, decodes entities, cleans whitespace
|
||||
*/
|
||||
export function cleanText(text: string): string {
|
||||
let cleaned = text;
|
||||
@@ -292,7 +438,9 @@ async function extractFromEmbeddedJSON(
|
||||
}
|
||||
|
||||
// Try __additionalDataLoaded pattern
|
||||
const additionalDataMatch = content.match(/window\.__additionalDataLoaded\([^,]+,\s*(\{.+?\})\);/s);
|
||||
const additionalDataMatch = content.match(
|
||||
/window\.__additionalDataLoaded\([^,]+,\s*(\{.+?\})\);/s
|
||||
);
|
||||
if (additionalDataMatch) {
|
||||
console.log(`[Extractor] Found __additionalDataLoaded in script ${i}`);
|
||||
try {
|
||||
@@ -309,7 +457,9 @@ async function extractFromEmbeddedJSON(
|
||||
|
||||
// Try to find any large JSON with caption data (new Instagram format)
|
||||
if ((content.includes('"caption"') || content.includes('"text"')) && content.length > 10000) {
|
||||
console.log(`[Extractor] Attempting to extract from large JSON in script ${i} (length: ${content.length})`);
|
||||
console.log(
|
||||
`[Extractor] Attempting to extract from large JSON in script ${i} (length: ${content.length})`
|
||||
);
|
||||
try {
|
||||
// Try to parse as direct JSON
|
||||
const jsonData = JSON.parse(content);
|
||||
@@ -317,7 +467,9 @@ async function extractFromEmbeddedJSON(
|
||||
// Try deep search first
|
||||
const deepResult = deepSearchForCaption(jsonData);
|
||||
if (deepResult && deepResult.bodyText && deepResult.bodyText.length > 130) {
|
||||
console.log(`[Extractor] Deep search in JSON found caption: ${deepResult.bodyText.length} chars`);
|
||||
console.log(
|
||||
`[Extractor] Deep search in JSON found caption: ${deepResult.bodyText.length} chars`
|
||||
);
|
||||
const thumbnail = await extractThumbnailStealth(page, progressCallback);
|
||||
return { ...deepResult, thumbnail };
|
||||
}
|
||||
@@ -325,7 +477,9 @@ async function extractFromEmbeddedJSON(
|
||||
// Try standard parsing
|
||||
const result = parseInstagramData(jsonData);
|
||||
if (result && result.bodyText && result.bodyText.length > 130) {
|
||||
console.log(`[Extractor] Successfully extracted from JSON, text length: ${result.bodyText.length}`);
|
||||
console.log(
|
||||
`[Extractor] Successfully extracted from JSON, text length: ${result.bodyText.length}`
|
||||
);
|
||||
const thumbnail = await extractThumbnailStealth(page, progressCallback);
|
||||
return { ...result, thumbnail };
|
||||
}
|
||||
@@ -336,7 +490,7 @@ async function extractFromEmbeddedJSON(
|
||||
const patterns = [
|
||||
/"caption"\s*:\s*\{\s*"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"/, // Escaped quotes
|
||||
/"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"\s*,?\s*"pk"/, // text field near pk
|
||||
/"edge_media_to_caption"\s*:\s*\{\s*"edges"\s*:\s*\[\s*\{\s*"node"\s*:\s*\{\s*"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"/,
|
||||
/"edge_media_to_caption"\s*:\s*\{\s*"edges"\s*:\s*\[\s*\{\s*"node"\s*:\s*\{\s*"text"\s*:\s*"([^"\\]*(\\.[^"\\]*)*)"/
|
||||
];
|
||||
|
||||
for (const pattern of patterns) {
|
||||
@@ -347,11 +501,15 @@ async function extractFromEmbeddedJSON(
|
||||
const captionText = rawText
|
||||
.replace(/\\n/g, '\n')
|
||||
.replace(/\\"/g, '"')
|
||||
.replace(/\\u([0-9a-fA-F]{4})/g, (_, code) => String.fromCharCode(parseInt(code, 16)))
|
||||
.replace(/\\u([0-9a-fA-F]{4})/g, (_, code) =>
|
||||
String.fromCharCode(parseInt(code, 16))
|
||||
)
|
||||
.replace(/\\\\/g, '\\');
|
||||
|
||||
if (captionText.length > 130) {
|
||||
console.log(`[Extractor] Extracted caption from regex pattern, length: ${captionText.length}`);
|
||||
console.log(
|
||||
`[Extractor] Extracted caption from regex pattern, length: ${captionText.length}`
|
||||
);
|
||||
const thumbnail = await extractThumbnailStealth(page, progressCallback);
|
||||
return { bodyText: cleanText(captionText), thumbnail };
|
||||
}
|
||||
@@ -448,7 +606,9 @@ export async function extractFromHTMLSection(
|
||||
const currentShortcode = extractShortcode(currentUrl);
|
||||
|
||||
console.log(`[Extractor] Current page URL: ${currentUrl}`);
|
||||
console.log(`[Extractor] Target shortcode: ${targetShortcode}, Current shortcode: ${currentShortcode}`);
|
||||
console.log(
|
||||
`[Extractor] Target shortcode: ${targetShortcode}, Current shortcode: ${currentShortcode}`
|
||||
);
|
||||
|
||||
if (targetShortcode && currentShortcode !== targetShortcode) {
|
||||
console.log(`[Extractor] URL mismatch: expected ${targetShortcode}, got ${currentShortcode}`);
|
||||
@@ -458,62 +618,13 @@ export async function extractFromHTMLSection(
|
||||
console.log(`[Extractor] Confirmed on correct post: ${currentShortcode}`);
|
||||
|
||||
// Wait for network to settle
|
||||
await page.waitForLoadState('domcontentloaded', { timeout: 10000 });
|
||||
await page.waitForTimeout(2000);
|
||||
await page.waitForLoadState('domcontentloaded', { timeout: TIMEOUTS.PAGE_LOAD });
|
||||
await page.waitForTimeout(TIMEOUTS.NETWORK_SETTLE);
|
||||
|
||||
//Try to expand truncated caption by clicking "more" button
|
||||
// Try to expand truncated caption by clicking "more" button
|
||||
// STRATEGY: Since we're already on the correct page (URL validated above),
|
||||
// the FIRST article/main post container should be our target post.
|
||||
// Instagram uses JS routing so links don't have shortcodes in hrefs.
|
||||
console.log('[Extractor] Looking for "more" button in primary post container...');
|
||||
try {
|
||||
// Wait for content to load
|
||||
await page.waitForTimeout(1500);
|
||||
|
||||
// Find the MAIN post container - should be the first article or main content area
|
||||
const mainContainer = page.locator('article, main, [role="main"]').first();
|
||||
const containerExists = await mainContainer.count() > 0;
|
||||
|
||||
if (containerExists) {
|
||||
console.log('[Extractor] Found main post container, searching for "more" button...');
|
||||
|
||||
// Try different patterns for the "more" button within the main container
|
||||
const morePatterns = [
|
||||
{ locator: mainContainer.locator('span').filter({ hasText: /\.\.\.\s*more/i }), desc: "span with '...more'" },
|
||||
{ locator: mainContainer.locator('span').filter({ hasText: /…\s*more/i }), desc: "span with '… more'" },
|
||||
{ locator: mainContainer.locator('div[role="button"]').filter({ hasText: /more/i }), desc: "button with 'more'" },
|
||||
{ locator: mainContainer.locator('span[role="button"]').filter({ hasText: /more/i }), desc: "span button with 'more'" }
|
||||
];
|
||||
|
||||
for (const pattern of morePatterns) {
|
||||
const count = await pattern.locator.count();
|
||||
console.log(`[Extractor] Checking ${pattern.desc}: found ${count}`);
|
||||
|
||||
if (count > 0) {
|
||||
const firstMore = pattern.locator.first();
|
||||
try {
|
||||
if (await firstMore.isVisible({ timeout: 1000 })) {
|
||||
const text = await firstMore.textContent();
|
||||
console.log(`[Extractor] Found visible "more": "${text}"`);
|
||||
await firstMore.click();
|
||||
console.log('[Extractor] Clicked "more" - waiting for expansion...');
|
||||
await page.waitForTimeout(3000);
|
||||
console.log('[Extractor] Caption expansion complete');
|
||||
break; // Success!
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(`[Extractor] ${pattern.desc} not clickable: ${e}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.log('[Extractor] No main container found');
|
||||
}
|
||||
|
||||
console.log('[Extractor] Finished "more" button expansion attempt');
|
||||
} catch (e) {
|
||||
console.log(`[Extractor] Error while trying to expand caption: ${e}`);
|
||||
}
|
||||
await tryExpandCaptionInHTMLSection(page);
|
||||
|
||||
console.log('[Extractor] Extracting caption using intelligent span detection...');
|
||||
|
||||
@@ -538,9 +649,10 @@ export async function extractFromHTMLSection(
|
||||
// If we found links to the post, search for spans within those link ancestors
|
||||
const searchRoots: Element[] = [];
|
||||
if (postLinks.length > 0) {
|
||||
postLinks.forEach(link => {
|
||||
postLinks.forEach((link) => {
|
||||
// Get the article or section container for this post
|
||||
let container = link.closest('article') || link.closest('section') || link.closest('[role="main"]');
|
||||
let container =
|
||||
link.closest('article') || link.closest('section') || link.closest('[role="main"]');
|
||||
if (container && !searchRoots.includes(container)) {
|
||||
searchRoots.push(container);
|
||||
console.log(`[Extractor] Found container for target post`);
|
||||
@@ -555,8 +667,8 @@ export async function extractFromHTMLSection(
|
||||
}
|
||||
|
||||
const spans: HTMLElement[] = [];
|
||||
searchRoots.forEach(root => {
|
||||
root.querySelectorAll('span').forEach(span => spans.push(span as HTMLElement));
|
||||
searchRoots.forEach((root) => {
|
||||
root.querySelectorAll('span').forEach((span) => spans.push(span as HTMLElement));
|
||||
});
|
||||
|
||||
console.log(`[Extractor] Searching ${spans.length} spans for recipe content`);
|
||||
@@ -584,7 +696,7 @@ export async function extractFromHTMLSection(
|
||||
score += brCount * 100; // Massive weight for line breaks
|
||||
|
||||
// Check for recipe keywords (strong indicator)
|
||||
const hasKeywords = recipeKeywords.some(keyword => text.includes(keyword));
|
||||
const hasKeywords = recipeKeywords.some((keyword) => text.includes(keyword));
|
||||
if (hasKeywords) {
|
||||
score += 500; // Huge boost for recipe keywords
|
||||
}
|
||||
@@ -616,7 +728,9 @@ export async function extractFromHTMLSection(
|
||||
|
||||
// Update best candidate
|
||||
if (score > 0 && (!bestCandidate || score > bestCandidate.score)) {
|
||||
console.log(`[Extractor] New best: score=${score}, len=${text.length}, br=${brCount}, links=${linkCount}, preview="${text.substring(0, 80)}..."`);
|
||||
console.log(
|
||||
`[Extractor] New best: score=${score}, len=${text.length}, br=${brCount}, links=${linkCount}, preview="${text.substring(0, 80)}..."`
|
||||
);
|
||||
bestCandidate = {
|
||||
element: span,
|
||||
text: span.textContent || '',
|
||||
@@ -638,7 +752,9 @@ export async function extractFromHTMLSection(
|
||||
// Explicit type assertion (safe after null guard)
|
||||
const candidate: CaptionCandidate = bestCandidate;
|
||||
|
||||
console.log(`[Extractor] Final caption candidate: score=${candidate.score}, length=${candidate.text.length}`);
|
||||
console.log(
|
||||
`[Extractor] Final caption candidate: score=${candidate.score}, length=${candidate.text.length}`
|
||||
);
|
||||
|
||||
// Extract text from the best candidate
|
||||
// Use innerHTML to preserve <br> tags, which will be converted to newlines in cleanText
|
||||
@@ -698,15 +814,15 @@ export async function extractFromDOM(
|
||||
try {
|
||||
// Give Instagram more time to load dynamic content
|
||||
console.log('[Extractor] Waiting for network idle...');
|
||||
await page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {
|
||||
await page.waitForLoadState('networkidle', { timeout: TIMEOUTS.PAGE_LOAD }).catch(() => {
|
||||
console.log('[Extractor] Network idle timeout, continuing anyway');
|
||||
});
|
||||
|
||||
// Try to wait for article content
|
||||
await page.waitForSelector('article', { timeout: 5000 }).catch(() => {});
|
||||
await page.waitForSelector('article', { timeout: TIMEOUTS.ARTICLE_SELECTOR }).catch(() => {});
|
||||
|
||||
// Additional wait for dynamic content
|
||||
await page.waitForTimeout(2000);
|
||||
await page.waitForTimeout(TIMEOUTS.NETWORK_SETTLE);
|
||||
|
||||
// Try to intercept GraphQL responses
|
||||
let graphqlCaption: string | null = null;
|
||||
@@ -715,11 +831,12 @@ export async function extractFromDOM(
|
||||
if (url.includes('graphql') || url.includes('api/v1')) {
|
||||
try {
|
||||
const json = await response.json();
|
||||
// Try to find caption in the response
|
||||
const captionData = extractCaptionFromGraphQL(json);
|
||||
if (captionData && captionData.length > 130) {
|
||||
graphqlCaption = captionData;
|
||||
console.log(`[Extractor] Intercepted GraphQL response with ${captionData.length} chars`);
|
||||
console.log(
|
||||
`[Extractor] Intercepted GraphQL response with ${captionData.length} chars`
|
||||
);
|
||||
}
|
||||
} catch (e) {
|
||||
// Not JSON or parsing failed
|
||||
@@ -727,54 +844,15 @@ export async function extractFromDOM(
|
||||
}
|
||||
});
|
||||
|
||||
// Wait a bit for any GraphQL requests to complete
|
||||
await page.waitForTimeout(1000);
|
||||
await page.waitForTimeout(TIMEOUTS.GRAPHQL_WAIT);
|
||||
|
||||
if (graphqlCaption) {
|
||||
const thumbnail = await extractThumbnailStealth(page, progressCallback);
|
||||
return { bodyText: cleanText(graphqlCaption), thumbnail };
|
||||
}
|
||||
|
||||
// First, try to expand truncated captions by clicking "more" button
|
||||
// Try multiple times with different selectors
|
||||
let expandAttempts = 0;
|
||||
const maxExpandAttempts = 3;
|
||||
|
||||
while (expandAttempts < maxExpandAttempts) {
|
||||
try {
|
||||
const moreButtonSelectors = [
|
||||
'article button:has-text("more")',
|
||||
'article button:has-text("More")',
|
||||
'article button:has-text("… more")',
|
||||
'article span[role="button"]:has-text("more")',
|
||||
'article [role="button"]:has-text("more")',
|
||||
'article div[role="button"]:has-text("more")',
|
||||
'xpath=//article//span[contains(text(), "more")]/..',
|
||||
'xpath=//article//button[contains(., "more")]'
|
||||
];
|
||||
|
||||
let clicked = false;
|
||||
for (const selector of moreButtonSelectors) {
|
||||
try {
|
||||
const button = page.locator(selector).first();
|
||||
if (await button.isVisible({ timeout: 500 })) {
|
||||
await button.click();
|
||||
await page.waitForTimeout(800);
|
||||
console.log(`[Extractor] Clicked "more" button with selector: ${selector}`);
|
||||
clicked = true;
|
||||
expandAttempts++;
|
||||
break;
|
||||
}
|
||||
} catch (e) {
|
||||
// Try next selector
|
||||
}
|
||||
}
|
||||
|
||||
if (!clicked) break; // No more buttons found
|
||||
} catch (e) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Try to expand truncated captions by clicking "more" button
|
||||
await tryExpandCaptionInDOM(page);
|
||||
|
||||
const captionText = await page.evaluate(() => {
|
||||
// First check og:description for comparison
|
||||
@@ -787,7 +865,9 @@ export async function extractFromDOM(
|
||||
|
||||
// SMART APPROACH: Find the truncated text first, then look for full version nearby
|
||||
// Look for text that ends with "..." or "… more"
|
||||
const allSpans = Array.from(document.querySelectorAll('article span, article div, article h1'));
|
||||
const allSpans = Array.from(
|
||||
document.querySelectorAll('article span, article div, article h1')
|
||||
);
|
||||
|
||||
let longestText = '';
|
||||
let matchedElement = null;
|
||||
@@ -809,9 +889,12 @@ export async function extractFromDOM(
|
||||
}
|
||||
|
||||
// Strategy 2: Look in data attributes
|
||||
const elementsWithData = Array.from(document.querySelectorAll('[data-caption], [data-text], [data-content]'));
|
||||
const elementsWithData = Array.from(
|
||||
document.querySelectorAll('[data-caption], [data-text], [data-content]')
|
||||
);
|
||||
for (const el of elementsWithData) {
|
||||
const dataCaption = el.getAttribute('data-caption') ||
|
||||
const dataCaption =
|
||||
el.getAttribute('data-caption') ||
|
||||
el.getAttribute('data-text') ||
|
||||
el.getAttribute('data-content');
|
||||
if (dataCaption && dataCaption.length > longestText.length) {
|
||||
@@ -821,7 +904,11 @@ export async function extractFromDOM(
|
||||
}
|
||||
|
||||
// Strategy 3: Look for hidden/collapsed content
|
||||
const hiddenElements = Array.from(document.querySelectorAll('[style*="display: none"], [style*="display:none"], .collapsed, [aria-hidden="true"]'));
|
||||
const hiddenElements = Array.from(
|
||||
document.querySelectorAll(
|
||||
'[style*="display: none"], [style*="display:none"], .collapsed, [aria-hidden="true"]'
|
||||
)
|
||||
);
|
||||
for (const el of hiddenElements) {
|
||||
const text = el.textContent?.trim() || '';
|
||||
if (text.length > longestText.length && text.length > 200) {
|
||||
@@ -838,7 +925,9 @@ export async function extractFromDOM(
|
||||
const parentText = parent.textContent?.trim() || '';
|
||||
if (parentText.length > longestText.length) {
|
||||
longestText = parentText;
|
||||
console.log(`[Extractor] Found fuller text in parent element: ${parentText.length} chars`);
|
||||
console.log(
|
||||
`[Extractor] Found fuller text in parent element: ${parentText.length} chars`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -864,7 +953,10 @@ export async function extractFromDOM(
|
||||
// Fallback to og:description
|
||||
if (metaDesc) {
|
||||
const content = ogContent;
|
||||
const cleanedContent = content.replace(/^\d+K?\s+likes,\s+\d+\s+comments\s+-\s+[\w.]+\s+on\s+[^:]+:\s*["']?/, '');
|
||||
const cleanedContent = content.replace(
|
||||
/^\d+K?\s+likes,\s+\d+\s+comments\s+-\s+[\w.]+\s+on\s+[^:]+:\s*["']?/,
|
||||
''
|
||||
);
|
||||
console.log('[Extractor] DOM selector fallback: og:description (with metadata cleanup)');
|
||||
return cleanedContent;
|
||||
}
|
||||
@@ -1021,11 +1113,15 @@ async function extractFromInternalState(
|
||||
}
|
||||
|
||||
if (result && result.bodyText && result.bodyText.length > 130) {
|
||||
console.log(`[Extractor] Successfully extracted from ${stateData.key}, length: ${result.bodyText.length}`);
|
||||
console.log(
|
||||
`[Extractor] Successfully extracted from ${stateData.key}, length: ${result.bodyText.length}`
|
||||
);
|
||||
const thumbnail = await extractThumbnailStealth(page, progressCallback);
|
||||
return { ...result, thumbnail };
|
||||
} else if (result?.bodyText) {
|
||||
console.log(`[Extractor] Found text in ${stateData.key} but it's truncated (${result.bodyText.length} chars)`);
|
||||
console.log(
|
||||
`[Extractor] Found text in ${stateData.key} but it's truncated (${result.bodyText.length} chars)`
|
||||
);
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(`[Extractor] Failed to parse ${stateData.key}:`, e);
|
||||
@@ -1042,7 +1138,11 @@ async function extractFromInternalState(
|
||||
/**
|
||||
* Deep search for caption text in any nested object structure
|
||||
*/
|
||||
function deepSearchForCaption(obj: any, maxDepth = 10, currentDepth = 0): Omit<ExtractedContent, 'thumbnail'> | null {
|
||||
function deepSearchForCaption(
|
||||
obj: any,
|
||||
maxDepth = 10,
|
||||
currentDepth = 0
|
||||
): Omit<ExtractedContent, 'thumbnail'> | null {
|
||||
if (currentDepth > maxDepth || !obj || typeof obj !== 'object') {
|
||||
return null;
|
||||
}
|
||||
@@ -1208,7 +1308,8 @@ export async function extractTextAndThumbnail(
|
||||
timestamp: new Date().toISOString()
|
||||
});
|
||||
|
||||
return withRetry(async () => {
|
||||
return withRetry(
|
||||
async () => {
|
||||
const authPath = resolveAuthPath();
|
||||
const context = await createBrowserContext(authPath);
|
||||
const page = await context.newPage();
|
||||
@@ -1227,13 +1328,19 @@ export async function extractTextAndThumbnail(
|
||||
page.on('response', async (response) => {
|
||||
try {
|
||||
const responseUrl = response.url();
|
||||
if (responseUrl.includes('graphql') || responseUrl.includes('api/v1') || responseUrl.includes('/web/')) {
|
||||
if (
|
||||
responseUrl.includes('graphql') ||
|
||||
responseUrl.includes('api/v1') ||
|
||||
responseUrl.includes('/web/')
|
||||
) {
|
||||
try {
|
||||
const json = await response.json();
|
||||
const captionData = extractCaptionFromGraphQL(json, expectedShortcode ?? undefined);
|
||||
if (captionData && captionData.length > 130) {
|
||||
interceptedCaption = captionData;
|
||||
console.log(`[Extractor] ✓ Intercepted GraphQL with full caption: ${captionData.length} chars (shortcode verified)`);
|
||||
console.log(
|
||||
`[Extractor] ✓ Intercepted GraphQL with full caption: ${captionData.length} chars (shortcode verified)`
|
||||
);
|
||||
}
|
||||
} catch (e) {
|
||||
// Not JSON or parse error, skip
|
||||
@@ -1309,7 +1416,10 @@ export async function extractTextAndThumbnail(
|
||||
await page.close();
|
||||
await context.close();
|
||||
}
|
||||
}, DEFAULT_RETRY_CONFIG, onProgress);
|
||||
},
|
||||
DEFAULT_RETRY_CONFIG,
|
||||
onProgress
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -131,15 +131,19 @@ class PushNotificationService {
|
||||
},
|
||||
payload,
|
||||
{
|
||||
TTL: 60 * 60 * 24, // 24 hours
|
||||
TTL: 60 * 60 * 24 // 24 hours
|
||||
}
|
||||
);
|
||||
|
||||
console.log(`[PushService] ✓ Sent notification to ${subscription.endpoint.substring(0, 50)}...`);
|
||||
console.log(
|
||||
`[PushService] ✓ Sent notification to ${subscription.endpoint.substring(0, 50)}...`
|
||||
);
|
||||
} catch (error) {
|
||||
// Check if subscription is expired/invalid
|
||||
if ((error as any).statusCode === 410) {
|
||||
console.warn(`[PushService] Subscription expired: ${subscription.endpoint.substring(0, 50)}...`);
|
||||
console.warn(
|
||||
`[PushService] Subscription expired: ${subscription.endpoint.substring(0, 50)}...`
|
||||
);
|
||||
throw new Error('Subscription expired');
|
||||
}
|
||||
|
||||
|
||||
@@ -8,13 +8,15 @@ const RecipeSchema = z.object({
|
||||
name: z.string(),
|
||||
servings: z.number().nullable(),
|
||||
description: z.string().nullable(),
|
||||
ingredients: z.array(
|
||||
ingredients: z
|
||||
.array(
|
||||
z.object({
|
||||
item: z.string(),
|
||||
amount: z.string(),
|
||||
unit: z.string()
|
||||
})
|
||||
).nullable(),
|
||||
)
|
||||
.nullable(),
|
||||
steps: z.array(z.string()).nullable(),
|
||||
image: z.string().nullable().optional()
|
||||
});
|
||||
@@ -59,9 +61,9 @@ export async function detectRecipe(text: string): Promise<boolean> {
|
||||
|
||||
// Check if this is a model-related error
|
||||
const errorMessage = (e as Error).message || '';
|
||||
const isModelError = errorMessage.includes('400') &&
|
||||
(errorMessage.toLowerCase().includes('model') ||
|
||||
errorMessage.toLowerCase().includes('load'));
|
||||
const isModelError =
|
||||
errorMessage.includes('400') &&
|
||||
(errorMessage.toLowerCase().includes('model') || errorMessage.toLowerCase().includes('load'));
|
||||
|
||||
if (isModelError) {
|
||||
const { model } = createLLM();
|
||||
@@ -116,9 +118,9 @@ export async function parseRecipe(text: string): Promise<Recipe> {
|
||||
|
||||
// Check if this is a model-related error
|
||||
const errorMessage = (e as Error).message || '';
|
||||
const isModelError = errorMessage.includes('400') &&
|
||||
(errorMessage.toLowerCase().includes('model') ||
|
||||
errorMessage.toLowerCase().includes('load'));
|
||||
const isModelError =
|
||||
errorMessage.includes('400') &&
|
||||
(errorMessage.toLowerCase().includes('model') || errorMessage.toLowerCase().includes('load'));
|
||||
|
||||
if (isModelError) {
|
||||
const { model } = createLLM();
|
||||
@@ -129,8 +131,10 @@ export async function parseRecipe(text: string): Promise<Recipe> {
|
||||
}
|
||||
|
||||
// If structured output fails, try standard completion
|
||||
if ((e as any).message?.includes('response_format') ||
|
||||
(e as any).message?.includes('structured output')) {
|
||||
if (
|
||||
(e as any).message?.includes('response_format') ||
|
||||
(e as any).message?.includes('structured output')
|
||||
) {
|
||||
console.warn('[LLM] Falling back to standard completion');
|
||||
return await parseRecipeWithStandardCompletion(text);
|
||||
}
|
||||
|
||||
@@ -142,11 +142,7 @@ export class QueueManager {
|
||||
* });
|
||||
* ```
|
||||
*/
|
||||
updateStatus(
|
||||
itemId: string,
|
||||
status: QueueItemStatus,
|
||||
data?: any
|
||||
): void {
|
||||
updateStatus(itemId: string, status: QueueItemStatus, data?: any): void {
|
||||
const item = this.items.get(itemId);
|
||||
if (!item) return;
|
||||
|
||||
@@ -163,7 +159,7 @@ export class QueueManager {
|
||||
}
|
||||
|
||||
// Update phases array
|
||||
const phaseIndex = item.phases.findIndex(p => p.name === data.phase);
|
||||
const phaseIndex = item.phases.findIndex((p) => p.name === data.phase);
|
||||
if (phaseIndex >= 0) {
|
||||
// Mark previous phases as completed
|
||||
for (let i = 0; i < phaseIndex; i++) {
|
||||
@@ -181,7 +177,7 @@ export class QueueManager {
|
||||
if (status === 'success') {
|
||||
item.completedAt = now;
|
||||
// Mark all phases as completed
|
||||
item.phases.forEach(phase => {
|
||||
item.phases.forEach((phase) => {
|
||||
if (phase.status !== 'completed') {
|
||||
phase.status = 'completed';
|
||||
phase.completedAt = now;
|
||||
@@ -193,7 +189,7 @@ export class QueueManager {
|
||||
item.completedAt = now;
|
||||
// Mark current phase as error
|
||||
if (item.currentPhase) {
|
||||
const phaseIndex = item.phases.findIndex(p => p.name === item.currentPhase);
|
||||
const phaseIndex = item.phases.findIndex((p) => p.name === item.currentPhase);
|
||||
if (phaseIndex >= 0) {
|
||||
item.phases[phaseIndex].status = 'error';
|
||||
item.phases[phaseIndex].error = data?.error?.message;
|
||||
@@ -202,7 +198,12 @@ export class QueueManager {
|
||||
}
|
||||
|
||||
// Wrap results in results object
|
||||
if (data?.extractedText || data?.thumbnail !== undefined || data?.recipe || data?.tandoorRecipeId) {
|
||||
if (
|
||||
data?.extractedText ||
|
||||
data?.thumbnail !== undefined ||
|
||||
data?.recipe ||
|
||||
data?.tandoorRecipeId
|
||||
) {
|
||||
if (!item.results) {
|
||||
item.results = {};
|
||||
}
|
||||
|
||||
@@ -115,12 +115,15 @@ export class QueueProcessor {
|
||||
if (!item) break;
|
||||
|
||||
this.activeWorkers++;
|
||||
console.log(`[QueueProcessor] Starting item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`);
|
||||
console.log(
|
||||
`[QueueProcessor] Starting item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`
|
||||
);
|
||||
|
||||
this.processItem(item)
|
||||
.finally(() => {
|
||||
this.processItem(item).finally(() => {
|
||||
this.activeWorkers--;
|
||||
console.log(`[QueueProcessor] Finished item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`);
|
||||
console.log(
|
||||
`[QueueProcessor] Finished item ${item.id} (${this.activeWorkers}/${this.concurrency} active)`
|
||||
);
|
||||
// Try to process next item immediately
|
||||
setTimeout(() => this.processNextBatch(), 0);
|
||||
});
|
||||
@@ -164,7 +167,6 @@ export class QueueProcessor {
|
||||
|
||||
// Send push notification
|
||||
await this.sendPushNotification(item, 'success');
|
||||
|
||||
} catch (error) {
|
||||
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
|
||||
const recoverable = this.isRecoverableError(error);
|
||||
@@ -393,7 +395,7 @@ export class QueueProcessor {
|
||||
'fetch failed'
|
||||
];
|
||||
|
||||
return recoverablePatterns.some(pattern => message.includes(pattern));
|
||||
return recoverablePatterns.some((pattern) => message.includes(pattern));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -29,7 +29,9 @@ export const queueConfig = {
|
||||
|
||||
/** Web Push notification settings */
|
||||
push: {
|
||||
vapidPublicKey: env.VAPID_PUBLIC_KEY || 'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
|
||||
vapidPublicKey:
|
||||
env.VAPID_PUBLIC_KEY ||
|
||||
'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
|
||||
vapidPrivateKey: env.VAPID_PRIVATE_KEY || 'JwxI_KcsBcehYcTOufMcbVWJjCq1QbH5FJmSyQuG680',
|
||||
vapidEmail: env.VAPID_EMAIL || 'mailto:admin@example.com'
|
||||
}
|
||||
|
||||
@@ -15,12 +15,7 @@ import type { ProgressEvent } from '$lib/server/extraction';
|
||||
* - unhealthy: Recoverable error occurred, can be retried
|
||||
* - error: Non-recoverable error occurred
|
||||
*/
|
||||
export type QueueItemStatus =
|
||||
| 'pending'
|
||||
| 'in_progress'
|
||||
| 'success'
|
||||
| 'unhealthy'
|
||||
| 'error';
|
||||
export type QueueItemStatus = 'pending' | 'in_progress' | 'success' | 'unhealthy' | 'error';
|
||||
|
||||
/**
|
||||
* Processing phases for queue items
|
||||
@@ -28,10 +23,7 @@ export type QueueItemStatus =
|
||||
* - parsing: Parsing recipe from extracted text
|
||||
* - uploading: Uploading recipe to Tandoor
|
||||
*/
|
||||
export type ProcessingPhase =
|
||||
| 'extraction'
|
||||
| 'parsing'
|
||||
| 'uploading';
|
||||
export type ProcessingPhase = 'extraction' | 'parsing' | 'uploading';
|
||||
|
||||
/**
|
||||
* Phase progress information
|
||||
|
||||
@@ -73,7 +73,9 @@ async function renewInstagramAuth(): Promise<boolean> {
|
||||
const authPath = resolveAuthPath();
|
||||
|
||||
if (!fs.existsSync(authPath)) {
|
||||
console.warn('[Scheduler] No existing auth.json found. Run gen-auth.js first to set up initial authentication.');
|
||||
console.warn(
|
||||
'[Scheduler] No existing auth.json found. Run gen-auth.js first to set up initial authentication.'
|
||||
);
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -115,7 +117,9 @@ async function renewInstagramAuth(): Promise<boolean> {
|
||||
await context.storageState({ path: authPath });
|
||||
|
||||
state.lastRenewalTime = Date.now();
|
||||
console.log(`[Scheduler] Instagram authentication renewed successfully at ${new Date().toISOString()}`);
|
||||
console.log(
|
||||
`[Scheduler] Instagram authentication renewed successfully at ${new Date().toISOString()}`
|
||||
);
|
||||
console.log(`[Scheduler] Auth state updated at: ${authPath}`);
|
||||
|
||||
return true;
|
||||
@@ -140,7 +144,9 @@ export async function startScheduler(): Promise<void> {
|
||||
const config = getConfig();
|
||||
|
||||
if (!config.enabled) {
|
||||
console.log('[Scheduler] Authentication scheduler is disabled (set AUTH_SCHEDULER_ENABLED=true to enable)');
|
||||
console.log(
|
||||
'[Scheduler] Authentication scheduler is disabled (set AUTH_SCHEDULER_ENABLED=true to enable)'
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -151,7 +157,9 @@ export async function startScheduler(): Promise<void> {
|
||||
|
||||
const intervalMs = config.intervalMinutes * 60 * 1000;
|
||||
|
||||
console.log(`[Scheduler] Starting authentication scheduler with ${config.intervalMinutes}min interval`);
|
||||
console.log(
|
||||
`[Scheduler] Starting authentication scheduler with ${config.intervalMinutes}min interval`
|
||||
);
|
||||
|
||||
// Schedule periodic renewals
|
||||
state.intervalId = setInterval(async () => {
|
||||
|
||||
@@ -15,38 +15,48 @@ export const TandoorRecipeSchema = z.object({
|
||||
prep_time: z.string().optional(),
|
||||
cook_time: z.string().optional(),
|
||||
waiting_time: z.string().optional(),
|
||||
steps: z.array(
|
||||
steps: z
|
||||
.array(
|
||||
z.object({
|
||||
step: z.number(),
|
||||
instruction: z.string(),
|
||||
ingredients: z.array(
|
||||
ingredients: z
|
||||
.array(
|
||||
z.object({
|
||||
food: z.object({
|
||||
id: z.number(),
|
||||
name: z.string()
|
||||
}),
|
||||
unit: z.object({
|
||||
unit: z
|
||||
.object({
|
||||
id: z.number(),
|
||||
name: z.string()
|
||||
}).nullable(),
|
||||
})
|
||||
.nullable(),
|
||||
amount: z.number(),
|
||||
note: z.string().optional()
|
||||
})
|
||||
).optional()
|
||||
)
|
||||
.optional()
|
||||
})
|
||||
).optional(),
|
||||
ingredients: z.array(
|
||||
)
|
||||
.optional(),
|
||||
ingredients: z
|
||||
.array(
|
||||
z.object({
|
||||
food: z.object({
|
||||
name: z.string()
|
||||
}),
|
||||
unit: z.object({
|
||||
unit: z
|
||||
.object({
|
||||
name: z.string()
|
||||
}).nullable(),
|
||||
})
|
||||
.nullable(),
|
||||
amount: z.number(),
|
||||
note: z.string().optional()
|
||||
})
|
||||
).optional()
|
||||
)
|
||||
.optional()
|
||||
});
|
||||
|
||||
export type TandoorRecipe = z.infer<typeof TandoorRecipeSchema>;
|
||||
@@ -104,11 +114,11 @@ interface TandoorRecipeDTO {
|
||||
*/
|
||||
async function fetchFromTandoor<T>(
|
||||
url: string,
|
||||
options: Partial<RequestInit> = { method: 'GET' },
|
||||
options: Partial<RequestInit> = { method: 'GET' }
|
||||
): Promise<{ ok: boolean; data?: T; error?: string }> {
|
||||
const headers = new Headers({
|
||||
'Content-Type': 'application/json',
|
||||
'Accept': 'application/json',
|
||||
Accept: 'application/json',
|
||||
Authorization: `Bearer ${tandoorConfig.token}`
|
||||
});
|
||||
|
||||
@@ -153,8 +163,6 @@ async function fetchFromTandoor<T>(
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* Partitions ingredients across steps by distributing them evenly
|
||||
* When step association is unknown, this spreads ingredients proportionally
|
||||
@@ -181,7 +189,10 @@ function partitionIngredientsAcrossSteps(
|
||||
partitions[index % stepCount].push(ingredient);
|
||||
});
|
||||
|
||||
console.debug(`Partitioned ${ingredients.length} ingredients across ${stepCount} steps:`, partitions);
|
||||
console.debug(
|
||||
`Partitioned ${ingredients.length} ingredients across ${stepCount} steps:`,
|
||||
partitions
|
||||
);
|
||||
return partitions;
|
||||
}
|
||||
|
||||
@@ -224,10 +235,7 @@ function parseAmount(amountStr: string): number | null {
|
||||
*/
|
||||
function buildTandoorRecipeDTO(recipe: ExtractedRecipe): TandoorRecipeDTO {
|
||||
const stepCount = recipe.steps?.length || 1;
|
||||
const ingredientPartitions = partitionIngredientsAcrossSteps(
|
||||
recipe.ingredients || [],
|
||||
stepCount
|
||||
);
|
||||
const ingredientPartitions = partitionIngredientsAcrossSteps(recipe.ingredients || [], stepCount);
|
||||
|
||||
const steps: TandoorRecipeDTO['steps'] = (recipe.steps || []).map((instruction, index) => {
|
||||
// Map ingredients, converting unparseable amounts to 1 q.b.
|
||||
@@ -235,7 +243,9 @@ function buildTandoorRecipeDTO(recipe: ExtractedRecipe): TandoorRecipeDTO {
|
||||
const amount = parseAmount(ing.amount);
|
||||
|
||||
if (amount === null) {
|
||||
console.debug(`Converting ingredient with unparseable amount to 1 q.b.: ${ing.item} (${ing.amount})`);
|
||||
console.debug(
|
||||
`Converting ingredient with unparseable amount to 1 q.b.: ${ing.item} (${ing.amount})`
|
||||
);
|
||||
return {
|
||||
food: {
|
||||
name: ing.item
|
||||
@@ -297,14 +307,11 @@ export async function uploadRecipeWithIngredientsDTO(
|
||||
console.debug('Uploading recipe with ingredients DTO:', recipeDTO);
|
||||
|
||||
// Call the API with the DTO
|
||||
const recipeResult = await fetchFromTandoor<{ id: number }>(
|
||||
`/api/recipe/`,
|
||||
{
|
||||
const recipeResult = await fetchFromTandoor<{ id: number }>(`/api/recipe/`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(recipeDTO)
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
if (!recipeResult.ok || !recipeResult.data) {
|
||||
console.error('Recipe creation failed:', recipeResult.error);
|
||||
@@ -397,7 +404,9 @@ export async function uploadRecipeImage(
|
||||
}
|
||||
|
||||
console.log(`[Tandoor Upload] Recipe ID: ${recipeId}`);
|
||||
console.log(`[Tandoor Upload] Image type: ${isDirectUrl(imageUrl) ? 'URL' : isDataUrl(imageUrl) ? 'Base64' : 'Unknown'}`);
|
||||
console.log(
|
||||
`[Tandoor Upload] Image type: ${isDirectUrl(imageUrl) ? 'URL' : isDataUrl(imageUrl) ? 'Base64' : 'Unknown'}`
|
||||
);
|
||||
console.log(`[Tandoor Upload] Image source: ${imageUrl.substring(0, 100)}...`);
|
||||
|
||||
let buffer: Buffer;
|
||||
@@ -462,26 +471,27 @@ export async function uploadRecipeImage(
|
||||
formData.append('image', file);
|
||||
|
||||
console.log('[Tandoor Upload] Uploading to Tandoor...');
|
||||
const uploadResponse = await fetch(
|
||||
`${tandoorConfig.serverUrl}/api/recipe/${recipeId}/image/`,
|
||||
{
|
||||
const uploadResponse = await fetch(`${tandoorConfig.serverUrl}/api/recipe/${recipeId}/image/`, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Authorization': `Bearer ${token}`
|
||||
Authorization: `Bearer ${token}`
|
||||
// DO NOT set Content-Type - let fetch set it with boundary
|
||||
},
|
||||
body: formData
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
if (!uploadResponse.ok) {
|
||||
const errorText = await uploadResponse.text().catch(() => uploadResponse.statusText);
|
||||
const responseHeaders = JSON.stringify(Object.fromEntries(uploadResponse.headers.entries()));
|
||||
|
||||
console.error(`[Tandoor Upload] Failed: ${uploadResponse.status} ${uploadResponse.statusText}`);
|
||||
console.error(
|
||||
`[Tandoor Upload] Failed: ${uploadResponse.status} ${uploadResponse.statusText}`
|
||||
);
|
||||
console.error(`[Tandoor Upload] Response headers: ${responseHeaders}`);
|
||||
console.error(`[Tandoor Upload] Response body: ${errorText.substring(0, 500)}`);
|
||||
console.error(`[Tandoor Upload] File metadata: ${filename}, ${file.size} bytes, ${file.type}`);
|
||||
console.error(
|
||||
`[Tandoor Upload] File metadata: ${filename}, ${file.size} bytes, ${file.type}`
|
||||
);
|
||||
|
||||
return {
|
||||
success: false,
|
||||
|
||||
@@ -18,11 +18,11 @@ export const GET = async () => {
|
||||
// Get current queue items by status
|
||||
const allItems = queueManager.getAll();
|
||||
const statusCounts = {
|
||||
pending: allItems.filter(item => item.status === 'pending').length,
|
||||
in_progress: allItems.filter(item => item.status === 'in_progress').length,
|
||||
success: allItems.filter(item => item.status === 'success').length,
|
||||
error: allItems.filter(item => item.status === 'error').length,
|
||||
unhealthy: allItems.filter(item => item.status === 'unhealthy').length
|
||||
pending: allItems.filter((item) => item.status === 'pending').length,
|
||||
in_progress: allItems.filter((item) => item.status === 'in_progress').length,
|
||||
success: allItems.filter((item) => item.status === 'success').length,
|
||||
error: allItems.filter((item) => item.status === 'error').length,
|
||||
unhealthy: allItems.filter((item) => item.status === 'unhealthy').length
|
||||
};
|
||||
|
||||
const stats = {
|
||||
@@ -51,11 +51,14 @@ export const GET = async () => {
|
||||
} catch (error) {
|
||||
console.error('[Health Check] Error retrieving health status:', error);
|
||||
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
timestamp: new Date().toISOString(),
|
||||
status: 'unhealthy',
|
||||
error: error instanceof Error ? error.message : 'Unknown error',
|
||||
uptime: process.uptime()
|
||||
}, { status: 500 });
|
||||
},
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
};
|
||||
@@ -15,16 +15,22 @@ export async function GET() {
|
||||
message: 'LLM service is accessible'
|
||||
});
|
||||
} else {
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
status: 'unhealthy',
|
||||
message: 'LLM service is not accessible'
|
||||
}, { status: 503 });
|
||||
},
|
||||
{ status: 503 }
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : 'Unknown error';
|
||||
return json({
|
||||
return json(
|
||||
{
|
||||
status: 'error',
|
||||
message: errorMessage
|
||||
}, { status: 500 });
|
||||
},
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,17 +32,11 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
|
||||
// Validate required fields
|
||||
if (!subscription || !subscription.endpoint || !subscription.keys) {
|
||||
return json(
|
||||
{ error: 'Invalid subscription object' },
|
||||
{ status: 400 }
|
||||
);
|
||||
return json({ error: 'Invalid subscription object' }, { status: 400 });
|
||||
}
|
||||
|
||||
if (!clientId || typeof clientId !== 'string') {
|
||||
return json(
|
||||
{ error: 'Client ID is required' },
|
||||
{ status: 400 }
|
||||
);
|
||||
return json({ error: 'Client ID is required' }, { status: 400 });
|
||||
}
|
||||
|
||||
// Subscribe client
|
||||
@@ -61,13 +55,9 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
message: 'Successfully subscribed to push notifications',
|
||||
subscriptionCount: pushNotificationService.getSubscriptionCount()
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('[NotificationAPI] Subscription error:', error);
|
||||
return json(
|
||||
{ error: 'Failed to subscribe to notifications' },
|
||||
{ status: 500 }
|
||||
);
|
||||
return json({ error: 'Failed to subscribe to notifications' }, { status: 500 });
|
||||
}
|
||||
};
|
||||
|
||||
@@ -86,10 +76,7 @@ export const DELETE: RequestHandler = async ({ request }) => {
|
||||
const { clientId } = await request.json();
|
||||
|
||||
if (!clientId || typeof clientId !== 'string') {
|
||||
return json(
|
||||
{ error: 'Client ID is required' },
|
||||
{ status: 400 }
|
||||
);
|
||||
return json({ error: 'Client ID is required' }, { status: 400 });
|
||||
}
|
||||
|
||||
// Unsubscribe client
|
||||
@@ -102,12 +89,8 @@ export const DELETE: RequestHandler = async ({ request }) => {
|
||||
message: 'Successfully unsubscribed from push notifications',
|
||||
subscriptionCount: pushNotificationService.getSubscriptionCount()
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('[NotificationAPI] Unsubscription error:', error);
|
||||
return json(
|
||||
{ error: 'Failed to unsubscribe from notifications' },
|
||||
{ status: 500 }
|
||||
);
|
||||
return json({ error: 'Failed to unsubscribe from notifications' }, { status: 500 });
|
||||
}
|
||||
};
|
||||
@@ -69,13 +69,11 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
message: `Test ${type} notification sent`,
|
||||
subscriberCount: pushNotificationService.getSubscriptionCount()
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('[NotificationTestAPI] Error sending test notification:',
|
||||
error instanceof Error ? error.message : String(error));
|
||||
return json(
|
||||
{ error: 'Failed to send test notification' },
|
||||
{ status: 500 }
|
||||
console.error(
|
||||
'[NotificationTestAPI] Error sending test notification:',
|
||||
error instanceof Error ? error.message : String(error)
|
||||
);
|
||||
return json({ error: 'Failed to send test notification' }, { status: 500 });
|
||||
}
|
||||
};
|
||||
|
||||
@@ -25,22 +25,15 @@ export const GET: RequestHandler = async () => {
|
||||
const publicKey = pushNotificationService.getPublicVapidKey();
|
||||
|
||||
if (!publicKey) {
|
||||
return json(
|
||||
{ error: 'VAPID public key not configured' },
|
||||
{ status: 503 }
|
||||
);
|
||||
return json({ error: 'VAPID public key not configured' }, { status: 503 });
|
||||
}
|
||||
|
||||
return json({
|
||||
publicKey,
|
||||
applicationServerKey: publicKey // Alias for compatibility
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('[NotificationAPI] VAPID key error:', error);
|
||||
return json(
|
||||
{ error: 'Failed to get VAPID public key' },
|
||||
{ status: 500 }
|
||||
);
|
||||
return json({ error: 'Failed to get VAPID public key' }, { status: 500 });
|
||||
}
|
||||
};
|
||||
@@ -60,7 +60,6 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
status: queueItem.status,
|
||||
enqueuedAt: queueItem.enqueuedAt
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
return handleApiError(error);
|
||||
}
|
||||
@@ -122,7 +121,7 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
|
||||
// Apply status filter
|
||||
if (statusFilter) {
|
||||
items = items.filter(item => item.status === statusFilter);
|
||||
items = items.filter((item) => item.status === statusFilter);
|
||||
}
|
||||
|
||||
// Sort by enqueued time (newest first)
|
||||
@@ -130,7 +129,7 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
|
||||
// Apply pagination
|
||||
const paginatedItems = items.slice(offset, offset + limit);
|
||||
const hasMore = (offset + limit) < items.length;
|
||||
const hasMore = offset + limit < items.length;
|
||||
|
||||
return json({
|
||||
items: paginatedItems,
|
||||
@@ -142,7 +141,6 @@ export const GET: RequestHandler = async ({ url }) => {
|
||||
count: paginatedItems.length
|
||||
}
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
return handleApiError(error);
|
||||
}
|
||||
|
||||
@@ -42,7 +42,6 @@ export const GET: RequestHandler = async ({ params }) => {
|
||||
|
||||
// Return full item details
|
||||
return json(queueItem);
|
||||
|
||||
} catch (error) {
|
||||
return handleApiError(error);
|
||||
}
|
||||
@@ -78,9 +77,7 @@ export const DELETE: RequestHandler = async ({ params }) => {
|
||||
|
||||
// Prevent deletion of in-progress items
|
||||
if (existingItem.status === 'in_progress') {
|
||||
throw new ConflictError(
|
||||
'Cannot delete item that is currently being processed'
|
||||
);
|
||||
throw new ConflictError('Cannot delete item that is currently being processed');
|
||||
}
|
||||
|
||||
// Remove the item
|
||||
@@ -90,7 +87,6 @@ export const DELETE: RequestHandler = async ({ params }) => {
|
||||
success,
|
||||
message: 'Queue item removed successfully'
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
return handleApiError(error);
|
||||
}
|
||||
|
||||
@@ -63,7 +63,6 @@ export const POST: RequestHandler = async ({ params }) => {
|
||||
item: updatedItem,
|
||||
message: 'Queue item has been reset and will be reprocessed'
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
return handleApiError(error);
|
||||
}
|
||||
|
||||
@@ -108,10 +108,10 @@ export const GET: RequestHandler = async ({ url, request }) => {
|
||||
|
||||
// Apply filters
|
||||
if (itemIdFilter) {
|
||||
filteredItems = currentItems.filter(item => item.id === itemIdFilter);
|
||||
filteredItems = currentItems.filter((item) => item.id === itemIdFilter);
|
||||
}
|
||||
if (statusFilter) {
|
||||
filteredItems = filteredItems.filter(item => item.status === statusFilter);
|
||||
filteredItems = filteredItems.filter((item) => item.status === statusFilter);
|
||||
}
|
||||
|
||||
// Send initial state for each matching item
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import {tandoorConfig} from '$lib/server/tandoor-config';
|
||||
import { tandoorConfig } from '$lib/server/tandoor-config';
|
||||
export async function GET() {
|
||||
return json({...tandoorConfig, token: ''});
|
||||
return json({ ...tandoorConfig, token: '' });
|
||||
}
|
||||
|
||||
@@ -40,4 +40,4 @@ export const POST: RequestHandler = async ({ request }) => {
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@@ -169,9 +169,7 @@ self.addEventListener('push', (event) => {
|
||||
|
||||
const title = data.title || getNotificationTitle(data.type, data);
|
||||
|
||||
event.waitUntil(
|
||||
self.registration.showNotification(title, options)
|
||||
);
|
||||
event.waitUntil(self.registration.showNotification(title, options));
|
||||
});
|
||||
|
||||
// Handle notification clicks
|
||||
@@ -195,8 +193,7 @@ self.addEventListener('notificationclick', (event) => {
|
||||
}
|
||||
|
||||
event.waitUntil(
|
||||
clients.matchAll({ type: 'window', includeUncontrolled: true })
|
||||
.then((clientsList) => {
|
||||
clients.matchAll({ type: 'window', includeUncontrolled: true }).then((clientsList) => {
|
||||
// Check if there's already a window/tab open
|
||||
for (const client of clientsList) {
|
||||
if (client.url.includes(self.location.origin) && 'focus' in client) {
|
||||
|
||||
@@ -29,8 +29,8 @@ describe('extraction.ts logging', () => {
|
||||
expect(calls.length).toBeGreaterThan(0);
|
||||
|
||||
// Verify at least one call has the expected format
|
||||
const errorCall = calls.find((call: any[]) =>
|
||||
call[0]?.match(/\[.*\]/) && call[1] !== undefined
|
||||
const errorCall = calls.find(
|
||||
(call: any[]) => call[0]?.match(/\[.*\]/) && call[1] !== undefined
|
||||
);
|
||||
expect(errorCall).toBeDefined();
|
||||
expect(errorCall[0]).toMatch(/\[.*\]/); // Has prefix like [Retry], [Extractor], etc
|
||||
@@ -49,14 +49,11 @@ describe('extraction.ts logging', () => {
|
||||
}
|
||||
|
||||
// Check all console.warn and console.error calls
|
||||
const allCalls = [
|
||||
...consoleWarnSpy.mock.calls,
|
||||
...consoleErrorSpy.mock.calls
|
||||
];
|
||||
const allCalls = [...consoleWarnSpy.mock.calls, ...consoleErrorSpy.mock.calls];
|
||||
|
||||
const errorCalls = allCalls
|
||||
.map(call => call.join(' '))
|
||||
.filter(msg => msg.includes('[object Object]'));
|
||||
.map((call) => call.join(' '))
|
||||
.filter((msg) => msg.includes('[object Object]'));
|
||||
|
||||
expect(errorCalls).toHaveLength(0);
|
||||
});
|
||||
@@ -78,9 +75,7 @@ describe('extraction.ts logging', () => {
|
||||
// Call real logError
|
||||
logger.logError('[Test] Real test', mockError);
|
||||
|
||||
const output = consoleErrorSpy.mock.calls
|
||||
.map(call => call.join(' '))
|
||||
.join(' ');
|
||||
const output = consoleErrorSpy.mock.calls.map((call) => call.join(' ')).join(' ');
|
||||
|
||||
// Should not contain [object Object]
|
||||
expect(output).not.toContain('[object Object]');
|
||||
|
||||
@@ -31,7 +31,8 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
const page = await context.newPage();
|
||||
|
||||
try {
|
||||
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
const testUrl =
|
||||
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
console.log('[DEBUG] Navigating to:', testUrl);
|
||||
|
||||
await page.goto(testUrl, { waitUntil: 'domcontentloaded' });
|
||||
@@ -71,7 +72,6 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
console.log(`\n[DEBUG] Shortcode appears ${htmlMatches} times in page HTML`);
|
||||
|
||||
expect(true).toBe(true);
|
||||
|
||||
} finally {
|
||||
await page.close();
|
||||
await context.close();
|
||||
@@ -84,7 +84,8 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
const page = await context.newPage();
|
||||
|
||||
try {
|
||||
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
const testUrl =
|
||||
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
console.log('[DEBUG] Navigating to:', testUrl);
|
||||
|
||||
await page.goto(testUrl, { waitUntil: 'domcontentloaded' });
|
||||
@@ -96,7 +97,10 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
|
||||
// Try to find and click "more" button
|
||||
console.log('[DEBUG] Looking for "more" button...');
|
||||
const moreElements = await page.locator('span, div, button').filter({ hasText: /more/i }).all();
|
||||
const moreElements = await page
|
||||
.locator('span, div, button')
|
||||
.filter({ hasText: /more/i })
|
||||
.all();
|
||||
console.log(`[DEBUG] Found ${moreElements.length} elements with "more"`);
|
||||
|
||||
for (let i = 0; i < Math.min(moreElements.length, 10); i++) {
|
||||
@@ -126,7 +130,7 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
const spanData = await page.evaluate(() => {
|
||||
const spans = Array.from(document.querySelectorAll('span'));
|
||||
return spans
|
||||
.filter(s => (s.textContent || '').length > 30)
|
||||
.filter((s) => (s.textContent || '').length > 30)
|
||||
.map((s, idx) => ({
|
||||
index: idx,
|
||||
text: (s.textContent || '').substring(0, 200),
|
||||
@@ -139,13 +143,14 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
});
|
||||
|
||||
console.log('[DEBUG] Top spans by LENGTH after click attempt:');
|
||||
spanData.slice(0, 5).forEach(span => {
|
||||
console.log(` [${span.index}] BR:${span.brCount} Links:${span.linkCount} Len:${span.length}`);
|
||||
spanData.slice(0, 5).forEach((span) => {
|
||||
console.log(
|
||||
` [${span.index}] BR:${span.brCount} Links:${span.linkCount} Len:${span.length}`
|
||||
);
|
||||
console.log(` Text: "${span.text}"`);
|
||||
});
|
||||
|
||||
expect(true).toBe(true); // Dummy assertion
|
||||
|
||||
} finally {
|
||||
await page.close();
|
||||
await context.close();
|
||||
@@ -156,7 +161,8 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
// Instagram's current anti-scraping measures make full extraction difficult
|
||||
// This test validates that we try all available methods
|
||||
|
||||
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
const testUrl =
|
||||
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
|
||||
const result = await extractTextAndThumbnail(testUrl);
|
||||
|
||||
@@ -191,7 +197,8 @@ describe('Instagram Caption Extraction E2E', () => {
|
||||
}, 30000);
|
||||
|
||||
it('should handle extraction attempt and return truncated text gracefully', async () => {
|
||||
const testUrl = 'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
const testUrl =
|
||||
'https://www.instagram.com/reel/DP6oN7JCEo8/?utm_source=ig_web_button_share_sheet';
|
||||
|
||||
const result = await extractTextAndThumbnail(testUrl);
|
||||
|
||||
|
||||
@@ -91,7 +91,8 @@ describe('extractFromDOM() with mocked og:description', () => {
|
||||
|
||||
it('should remove metadata prefix from og:description fallback', async () => {
|
||||
// Exact fixture from context_compact.yaml
|
||||
const ogContent = '16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
|
||||
const ogContent =
|
||||
'16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
|
||||
|
||||
const mockPage = createMockPage(ogContent);
|
||||
|
||||
@@ -104,7 +105,8 @@ describe('extractFromDOM() with mocked og:description', () => {
|
||||
});
|
||||
|
||||
it('should remove opening quote after metadata prefix', async () => {
|
||||
const ogContent = '16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
|
||||
const ogContent =
|
||||
'16K likes, 325 comments - chef.antonio.la.cava on October 17, 2025: "La cacio e pepe infallibile di Luciano Monosilio 🍝';
|
||||
|
||||
const mockPage = createMockPage(ogContent);
|
||||
|
||||
@@ -168,7 +170,8 @@ describe('Integration: Full extraction flow', () => {
|
||||
it('should extract, clean metadata prefix, remove quotes, and clean hashtags', async () => {
|
||||
// Simulating what the browser's page.evaluate() would return AFTER cleaning metadata
|
||||
// (the browser regex already strips the metadata prefix and quotes)
|
||||
const browserCleanedContent = 'La cacio e pepe infallibile di Luciano Monosilio 🍝 #cacio #pepe #recipe';
|
||||
const browserCleanedContent =
|
||||
'La cacio e pepe infallibile di Luciano Monosilio 🍝 #cacio #pepe #recipe';
|
||||
|
||||
const mockPage = createMockPage(browserCleanedContent);
|
||||
|
||||
@@ -197,7 +200,8 @@ describe('Integration: Full extraction flow', () => {
|
||||
|
||||
it('should handle full real-world caption with multiline content', async () => {
|
||||
// Browser has already cleaned metadata, only hashtags remain
|
||||
const browserCleanedContent = 'La cacio e pepe\n\nIngredients:\n- Pasta\n- Cheese\n\n#recipe #pasta';
|
||||
const browserCleanedContent =
|
||||
'La cacio e pepe\n\nIngredients:\n- Pasta\n- Cheese\n\n#recipe #pasta';
|
||||
|
||||
const mockPage = createMockPage(browserCleanedContent);
|
||||
|
||||
|
||||
@@ -76,9 +76,6 @@ describe('llm.ts logging', () => {
|
||||
|
||||
await checkModelAvailability('test-model');
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[LLM] Model availability check failed',
|
||||
complexError
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[LLM] Model availability check failed', complexError);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -149,10 +149,7 @@ describe('logger utilities', () => {
|
||||
|
||||
logObject('[Test]', obj);
|
||||
|
||||
expect(consoleLogSpy).toHaveBeenCalledWith(
|
||||
'[Test]',
|
||||
expect.stringContaining('[Circular]')
|
||||
);
|
||||
expect(consoleLogSpy).toHaveBeenCalledWith('[Test]', expect.stringContaining('[Circular]'));
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -20,7 +20,9 @@ describe('POST /api/notifications/test', () => {
|
||||
|
||||
// Spy on pushNotificationService methods
|
||||
sendNotificationSpy = vi.spyOn(pushNotificationService, 'sendNotification').mockResolvedValue();
|
||||
getSubscriptionCountSpy = vi.spyOn(pushNotificationService, 'getSubscriptionCount').mockReturnValue(2);
|
||||
getSubscriptionCountSpy = vi
|
||||
.spyOn(pushNotificationService, 'getSubscriptionCount')
|
||||
.mockReturnValue(2);
|
||||
});
|
||||
|
||||
test('should validate notification type - reject invalid type', async () => {
|
||||
@@ -179,7 +181,7 @@ describe('POST /api/notifications/test', () => {
|
||||
const call1 = sendNotificationSpy.mock.calls[0][0];
|
||||
|
||||
// Wait a bit to ensure different timestamp
|
||||
await new Promise(resolve => setTimeout(resolve, 2));
|
||||
await new Promise((resolve) => setTimeout(resolve, 2));
|
||||
|
||||
await POST({ request: request2 } as any);
|
||||
const call2 = sendNotificationSpy.mock.calls[1][0];
|
||||
|
||||
@@ -47,10 +47,7 @@ describe('parser.ts logging', () => {
|
||||
// Expected to throw
|
||||
}
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[LLM] Recipe detection error',
|
||||
expect.any(Error)
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[LLM] Recipe detection error', expect.any(Error));
|
||||
});
|
||||
|
||||
test('parseRecipe should use logError on failure', async () => {
|
||||
@@ -60,10 +57,7 @@ describe('parser.ts logging', () => {
|
||||
// Expected to throw
|
||||
}
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[LLM] Recipe parsing error',
|
||||
expect.any(Error)
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[LLM] Recipe parsing error', expect.any(Error));
|
||||
});
|
||||
|
||||
test('should not log stack trace separately', async () => {
|
||||
@@ -73,8 +67,9 @@ describe('parser.ts logging', () => {
|
||||
// Expected to throw
|
||||
}
|
||||
|
||||
const stackCalls = consoleErrorSpy.mock.calls
|
||||
.filter((call: any) => call[0]?.includes('Stack trace'));
|
||||
const stackCalls = consoleErrorSpy.mock.calls.filter((call: any) =>
|
||||
call[0]?.includes('Stack trace')
|
||||
);
|
||||
|
||||
expect(stackCalls).toHaveLength(0);
|
||||
});
|
||||
|
||||
@@ -98,11 +98,9 @@ describe('PushNotificationService web-push integration', () => {
|
||||
body: 'Progress update'
|
||||
});
|
||||
|
||||
expect(webpush.sendNotification).toHaveBeenCalledWith(
|
||||
expect.any(Object),
|
||||
expect.any(String),
|
||||
{ TTL: 60 * 60 * 24 }
|
||||
);
|
||||
expect(webpush.sendNotification).toHaveBeenCalledWith(expect.any(Object), expect.any(String), {
|
||||
TTL: 60 * 60 * 24
|
||||
});
|
||||
});
|
||||
|
||||
test('should serialize notification data as JSON', async () => {
|
||||
@@ -165,7 +163,8 @@ describe('PushNotificationService web-push integration', () => {
|
||||
test('should log endpoint prefix only (privacy)', async () => {
|
||||
const consoleSpy = vi.spyOn(console, 'log');
|
||||
|
||||
const longEndpoint = 'https://push.example.com/very-long-endpoint-with-secret-tokens-that-should-not-be-fully-logged-12345678901234567890';
|
||||
const longEndpoint =
|
||||
'https://push.example.com/very-long-endpoint-with-secret-tokens-that-should-not-be-fully-logged-12345678901234567890';
|
||||
const mockSubscription = {
|
||||
endpoint: longEndpoint,
|
||||
keys: { p256dh: 'test-key', auth: 'test-auth' }
|
||||
@@ -182,7 +181,7 @@ describe('PushNotificationService web-push integration', () => {
|
||||
|
||||
// Find the log call with endpoint
|
||||
const endpointLogCall = consoleSpy.mock.calls.find(
|
||||
call => typeof call[0] === 'string' && call[0].includes('Sent notification to')
|
||||
(call) => typeof call[0] === 'string' && call[0].includes('Sent notification to')
|
||||
);
|
||||
|
||||
expect(endpointLogCall).toBeTruthy();
|
||||
|
||||
@@ -49,10 +49,12 @@ test.describe('Push Notifications E2E', () => {
|
||||
const subscription = await page.evaluate(async () => {
|
||||
const registration = await navigator.serviceWorker.ready;
|
||||
const sub = await registration.pushManager.getSubscription();
|
||||
return sub ? {
|
||||
return sub
|
||||
? {
|
||||
endpoint: sub.endpoint,
|
||||
hasKeys: !!(sub as any).keys
|
||||
} : null;
|
||||
}
|
||||
: null;
|
||||
});
|
||||
|
||||
expect(subscription).not.toBeNull();
|
||||
|
||||
@@ -13,12 +13,12 @@ import { POST as retryPOST } from '../routes/api/queue/[id]/retry/+server.js';
|
||||
describe('Queue API Endpoints', () => {
|
||||
beforeEach(() => {
|
||||
// Clear queue between tests
|
||||
queueManager.getAll().forEach(item => queueManager.remove(item.id));
|
||||
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
// Clean up after tests
|
||||
queueManager.getAll().forEach(item => queueManager.remove(item.id));
|
||||
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
|
||||
});
|
||||
|
||||
describe('POST /api/queue', () => {
|
||||
@@ -26,7 +26,7 @@ describe('Queue API Endpoints', () => {
|
||||
const request = new Request('http://localhost/api/queue', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
url: 'https://instagram.com/p/ABC123'
|
||||
@@ -52,7 +52,7 @@ describe('Queue API Endpoints', () => {
|
||||
const request = new Request('http://localhost/api/queue', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
url: 'https://www.instagram.com/p/XYZ789'
|
||||
@@ -98,7 +98,9 @@ describe('Queue API Endpoints', () => {
|
||||
const response = await queuePOST({ request } as any);
|
||||
expect(response.status).toBe(200);
|
||||
const data = await response.json();
|
||||
expect(data.url).toBe('https://www.instagram.com/reel/DSevV5CDcNm/?utm_source=ig_web_copy_link');
|
||||
expect(data.url).toBe(
|
||||
'https://www.instagram.com/reel/DSevV5CDcNm/?utm_source=ig_web_copy_link'
|
||||
);
|
||||
});
|
||||
|
||||
it('should accept Instagram IGTV URLs', async () => {
|
||||
@@ -135,17 +137,13 @@ describe('Queue API Endpoints', () => {
|
||||
});
|
||||
|
||||
it('should reject invalid Instagram URL formats', async () => {
|
||||
const invalidUrls = [
|
||||
'https://facebook.com/post/123',
|
||||
'not-a-url',
|
||||
'https://other-site.com'
|
||||
];
|
||||
const invalidUrls = ['https://facebook.com/post/123', 'not-a-url', 'https://other-site.com'];
|
||||
|
||||
for (const url of invalidUrls) {
|
||||
const request = new Request('http://localhost/api/queue', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({ url })
|
||||
});
|
||||
@@ -199,7 +197,7 @@ describe('Queue API Endpoints', () => {
|
||||
const request = new Request('http://localhost/api/queue', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({})
|
||||
});
|
||||
@@ -219,7 +217,7 @@ describe('Queue API Endpoints', () => {
|
||||
const request = new Request('http://localhost/api/queue', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'text/plain',
|
||||
'Content-Type': 'text/plain'
|
||||
},
|
||||
body: 'not json'
|
||||
});
|
||||
@@ -321,10 +319,14 @@ describe('Queue API Endpoints', () => {
|
||||
let response = await queueGET({ request, url } as any);
|
||||
expect(response.status).toBe(400);
|
||||
let data = await response.json();
|
||||
expect(data.message).toBe('Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error');
|
||||
expect(data.message).toBe(
|
||||
'Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error'
|
||||
);
|
||||
} catch (err: any) {
|
||||
expect(err.status).toBe(400);
|
||||
expect(err.body.message).toBe('Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error');
|
||||
expect(err.body.message).toBe(
|
||||
'Invalid status filter. Must be one of: pending, in_progress, success, unhealthy, error'
|
||||
);
|
||||
}
|
||||
|
||||
// Invalid limit (negative)
|
||||
@@ -485,10 +487,14 @@ describe('Queue API Endpoints', () => {
|
||||
} as any);
|
||||
expect(response.status).toBe(409);
|
||||
const data = await response.json();
|
||||
expect(data.message).toBe("Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried.");
|
||||
expect(data.message).toBe(
|
||||
"Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried."
|
||||
);
|
||||
} catch (err: any) {
|
||||
expect(err.status).toBe(409);
|
||||
expect(err.body.message).toBe("Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried.");
|
||||
expect(err.body.message).toBe(
|
||||
"Cannot retry item with status 'pending'. Only 'error' and 'unhealthy' items can be retried."
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -29,10 +29,7 @@ describe('QueueManager logging', () => {
|
||||
// Enqueue an item (this will notify subscribers)
|
||||
manager.enqueue('https://instagram.com/p/test123');
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[QueueManager] Subscriber error',
|
||||
expect.any(Error)
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[QueueManager] Subscriber error', expect.any(Error));
|
||||
});
|
||||
|
||||
test('should serialize complex error objects', () => {
|
||||
@@ -49,10 +46,7 @@ describe('QueueManager logging', () => {
|
||||
manager.subscribe(failingCallback);
|
||||
manager.enqueue('https://instagram.com/p/test456');
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[QueueManager] Subscriber error',
|
||||
complexError
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[QueueManager] Subscriber error', complexError);
|
||||
});
|
||||
|
||||
test('should not prevent other subscribers from being notified on error', () => {
|
||||
@@ -74,12 +68,9 @@ describe('QueueManager logging', () => {
|
||||
expect(successCallback).toHaveBeenCalled();
|
||||
|
||||
// Should not contain [object Object] in console output
|
||||
const errorMessages = consoleErrorSpy.mock.calls
|
||||
.map(call => call.join(' '));
|
||||
const errorMessages = consoleErrorSpy.mock.calls.map((call) => call.join(' '));
|
||||
|
||||
const hasObjectObject = errorMessages.some(msg =>
|
||||
msg.includes('[object Object]')
|
||||
);
|
||||
const hasObjectObject = errorMessages.some((msg) => msg.includes('[object Object]'));
|
||||
|
||||
expect(hasObjectObject).toBe(false);
|
||||
});
|
||||
|
||||
@@ -22,7 +22,6 @@ import * as extraction from '$lib/server/extraction';
|
||||
import { queueProcessor } from '$lib/server/queue/QueueProcessor';
|
||||
|
||||
describe('QueueProcessor logging', () => {
|
||||
|
||||
let consoleErrorSpy: any;
|
||||
|
||||
beforeEach(async () => {
|
||||
@@ -31,13 +30,13 @@ describe('QueueProcessor logging', () => {
|
||||
|
||||
// Clear queue
|
||||
const items = queueManager.getAll();
|
||||
items.forEach(item => queueManager.remove(item.id));
|
||||
items.forEach((item) => queueManager.remove(item.id));
|
||||
|
||||
// Setup console.error spy
|
||||
consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
// Give time for cleanup
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
@@ -59,34 +58,37 @@ describe('QueueProcessor logging', () => {
|
||||
queueProcessor.start();
|
||||
|
||||
// Wait for error status
|
||||
await vi.waitFor(() => {
|
||||
await vi.waitFor(
|
||||
() => {
|
||||
const updated = queueManager.get(item.id);
|
||||
return updated?.status === 'error' || updated?.status === 'unhealthy';
|
||||
}, { timeout: 5000 });
|
||||
},
|
||||
{ timeout: 5000 }
|
||||
);
|
||||
|
||||
// Stop processor
|
||||
queueProcessor.stop();
|
||||
|
||||
// Wait a bit for all logs to finish
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
|
||||
// Check that console.error doesn't contain [object Object]
|
||||
const allCalls = consoleErrorSpy.mock.calls.map((call: any[]) =>
|
||||
call.map(arg => {
|
||||
call
|
||||
.map((arg) => {
|
||||
if (arg && typeof arg === 'object' && arg.message) {
|
||||
return arg.message; // Handle Error objects
|
||||
}
|
||||
return String(arg);
|
||||
}).join(' ')
|
||||
})
|
||||
.join(' ')
|
||||
);
|
||||
|
||||
const hasObjectObject = allCalls.some((msg: string) => msg.includes('[object Object]'));
|
||||
expect(hasObjectObject).toBe(false);
|
||||
|
||||
// Verify QueueProcessor logs are present
|
||||
const queueProcessorLogs = allCalls.filter((msg: string) =>
|
||||
msg.includes('[QueueProcessor]')
|
||||
);
|
||||
const queueProcessorLogs = allCalls.filter((msg: string) => msg.includes('[QueueProcessor]'));
|
||||
|
||||
expect(queueProcessorLogs.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
@@ -27,7 +27,8 @@ vi.mock('$lib/server/queue/config', () => ({
|
||||
serverUrl: 'http://localhost:8080'
|
||||
},
|
||||
push: {
|
||||
vapidPublicKey: 'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
|
||||
vapidPublicKey:
|
||||
'BNextdcB_fQ0BVvyGioM5L8Tf9vKQjs-WnF-rUbnU8MdWIZQYfggIHxBnW21I-lq_0HykLCdMpYj8d5joavWdxQ',
|
||||
vapidPrivateKey: 'JwxI_KcsBcehYcTOufMcbVWJjCq1QbH5FJmSyQuG680',
|
||||
vapidEmail: 'mailto:test@example.com'
|
||||
}
|
||||
@@ -72,7 +73,7 @@ import '$lib/server/queue/QueueProcessor';
|
||||
describe('QueueProcessor Integration Tests', () => {
|
||||
beforeEach(async () => {
|
||||
// Clear queue
|
||||
queueManager.getAll().forEach(item => queueManager.remove(item.id));
|
||||
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
|
||||
|
||||
// Reset mocks and their implementations
|
||||
vi.resetAllMocks();
|
||||
@@ -191,9 +192,7 @@ describe('QueueProcessor Integration Tests', () => {
|
||||
}, 10000);
|
||||
|
||||
it('should handle extraction errors', async () => {
|
||||
vi.mocked(extractTextAndThumbnail).mockRejectedValue(
|
||||
new Error('Network timeout')
|
||||
);
|
||||
vi.mocked(extractTextAndThumbnail).mockRejectedValue(new Error('Network timeout'));
|
||||
|
||||
const item = queueManager.enqueue('https://instagram.com/p/error');
|
||||
|
||||
@@ -249,7 +248,7 @@ describe('QueueProcessor Integration Tests', () => {
|
||||
await new Promise((resolve) => setTimeout(resolve, 150));
|
||||
|
||||
const items = queueManager.getAll();
|
||||
const inProgress = items.filter(i => i.status === 'in_progress');
|
||||
const inProgress = items.filter((i) => i.status === 'in_progress');
|
||||
|
||||
// With concurrency=2, should have max 2 in progress at once
|
||||
expect(inProgress.length).toBeLessThanOrEqual(2);
|
||||
@@ -258,7 +257,7 @@ describe('QueueProcessor Integration Tests', () => {
|
||||
await new Promise((resolve) => setTimeout(resolve, 2000));
|
||||
|
||||
const final = queueManager.getAll();
|
||||
const completed = final.filter(i => i.status === 'success');
|
||||
const completed = final.filter((i) => i.status === 'success');
|
||||
|
||||
// All 3 should eventually complete
|
||||
expect(completed.length).toBe(3);
|
||||
|
||||
@@ -11,12 +11,12 @@ import { GET as streamGET } from '../routes/api/queue/stream/+server.js';
|
||||
describe('Queue SSE Stream Endpoint', () => {
|
||||
beforeEach(() => {
|
||||
// Clear queue between tests
|
||||
queueManager.getAll().forEach(item => queueManager.remove(item.id));
|
||||
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
// Clean up after tests
|
||||
queueManager.getAll().forEach(item => queueManager.remove(item.id));
|
||||
queueManager.getAll().forEach((item) => queueManager.remove(item.id));
|
||||
});
|
||||
|
||||
describe('GET /api/queue/stream', () => {
|
||||
|
||||
@@ -112,7 +112,7 @@ describe('Frontend SSE Parser', () => {
|
||||
'embedded-json': '📦',
|
||||
'dom-selector': '🎯',
|
||||
'graphql-api': '🔌',
|
||||
'legacy': '📄'
|
||||
legacy: '📄'
|
||||
};
|
||||
return method ? icons[method] || '⚙️' : '⚙️';
|
||||
};
|
||||
|
||||
@@ -24,18 +24,13 @@ describe('tandoor logging', () => {
|
||||
name: 'Test Recipe',
|
||||
servings: 4,
|
||||
description: 'Test description',
|
||||
ingredients: [
|
||||
{ item: 'Flour', amount: '2', unit: 'cups' }
|
||||
],
|
||||
ingredients: [{ item: 'Flour', amount: '2', unit: 'cups' }],
|
||||
steps: ['Mix ingredients']
|
||||
};
|
||||
|
||||
await uploadRecipeWithIngredientsDTO(recipe);
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[Tandoor] Fetch error',
|
||||
expect.any(Error)
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor] Fetch error', expect.any(Error));
|
||||
});
|
||||
|
||||
test('should use logError on API error response', async () => {
|
||||
@@ -80,10 +75,7 @@ describe('tandoor logging', () => {
|
||||
|
||||
await uploadRecipeWithIngredientsDTO(recipe);
|
||||
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[Tandoor] Fetch error',
|
||||
expect.any(Error)
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor] Fetch error', expect.any(Error));
|
||||
});
|
||||
|
||||
test('should use logError on image upload failure', async () => {
|
||||
@@ -93,10 +85,7 @@ describe('tandoor logging', () => {
|
||||
const result = await uploadRecipeImage(123, 'https://example.com/image.jpg');
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[Tandoor Upload] Exception',
|
||||
error
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor Upload] Exception', error);
|
||||
});
|
||||
|
||||
test('should use logError instead of manual error logging', async () => {
|
||||
@@ -112,10 +101,7 @@ describe('tandoor logging', () => {
|
||||
});
|
||||
|
||||
// Verify logError was called (which handles stack trace serialization)
|
||||
expect(logErrorSpy).toHaveBeenCalledWith(
|
||||
'[Tandoor] Fetch error',
|
||||
error
|
||||
);
|
||||
expect(logErrorSpy).toHaveBeenCalledWith('[Tandoor] Fetch error', error);
|
||||
|
||||
// logError itself logs stack traces, which is expected behavior
|
||||
// The key is that tandoor.ts uses logError instead of manual logging
|
||||
|
||||
@@ -12,15 +12,15 @@ export default defineConfig({
|
||||
watch: {
|
||||
ignored: ['**/debug_page.txt', '**/.ssl/**', '**/docs/**', '**/secrets/**']
|
||||
},
|
||||
https: fs.existsSync('./.ssl/localhost.key') && fs.existsSync('./.ssl/localhost.crt')
|
||||
https:
|
||||
fs.existsSync('./.ssl/localhost.key') && fs.existsSync('./.ssl/localhost.crt')
|
||||
? {
|
||||
key: fs.readFileSync('./.ssl/localhost.key'),
|
||||
cert: fs.readFileSync('./.ssl/localhost.crt')
|
||||
}
|
||||
: undefined
|
||||
},
|
||||
plugins: [
|
||||
tailwindcss(), sveltekit()],
|
||||
plugins: [tailwindcss(), sveltekit()],
|
||||
test: {
|
||||
expect: { requireAssertions: true },
|
||||
projects: [
|
||||
|
||||
Reference in New Issue
Block a user