This commit is contained in:
Giancarmine Salucci
2026-02-18 01:21:44 +01:00
parent 54321fd7c9
commit 49bccf8f15
84 changed files with 14474 additions and 13925 deletions

View File

@@ -5,6 +5,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
## 🚀 Features
### Core Functionality
- **Async Queue Processing**: Fire-and-forget recipe extraction with background processing
- **Real-time Updates**: Server-Sent Events for live progress tracking
- **Push Notifications**: Background notifications when recipes complete
@@ -13,6 +14,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
- **PWA Support**: Installable Progressive Web App with offline capabilities
### User Experience
- **Queue Dashboard**: Monitor all recipe extractions in real-time
- **Share Integration**: Browser share target for easy URL submission
- **Responsive Design**: Works on desktop, tablet, and mobile
@@ -20,6 +22,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
- **Progress Tracking**: Visual progress through extraction phases
### Technical Architecture
- **SvelteKit Frontend**: Modern reactive UI with TypeScript
- **Hexagonal Architecture**: Clean separation of concerns
- **In-Memory Queue**: High-performance processing with configurable concurrency
@@ -29,6 +32,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
## 📋 API Endpoints
### Queue Management
- `POST /api/queue` - Enqueue Instagram URL for processing
- `GET /api/queue` - List queue items with filtering and pagination
- `GET /api/queue/{id}` - Get specific queue item details
@@ -36,18 +40,21 @@ A modern web application that extracts recipes from Instagram posts and saves th
- `GET /api/queue/stream` - Server-Sent Events for real-time updates
### Push Notifications
- `POST /api/notifications/subscribe` - Subscribe to push notifications
- `DELETE /api/notifications/subscribe` - Unsubscribe from notifications
- `GET /api/notifications/vapid-key` - Get VAPID public key
### Legacy Endpoints (Deprecated)
- ~~`POST /api/extract`~~ - Use `/api/queue` instead
- ~~`GET /api/extract-stream`~~ - Use `/api/queue/stream` instead
## 🛠 Development Setup
### Prerequisites
- Node.js 18+
- Node.js 18+
- npm or pnpm
- Tandoor Recipe Manager instance (optional)
- LLM API access (OpenAI, Anthropic, or local)
@@ -79,6 +86,7 @@ open https://localhost:5173
```
The app runs on HTTPS by default for:
- Service worker support (required for PWA)
- Push notifications
- Browser share target API
@@ -89,6 +97,7 @@ The app runs on HTTPS by default for:
The application uses HTTPS in development with SSL certificates signed by an external Caddy CA container. The current certificate is valid until **December 20, 2035** (10 years).
**Certificate Information:**
- Location: `.ssl/` directory
- CA Certificate: `.ssl/root.crt` (already trusted on the system)
- Server Certificate: `.ssl/localhost.crt`
@@ -97,18 +106,21 @@ The application uses HTTPS in development with SSL certificates signed by an ext
Since the Caddy CA is already trusted on the system, the certificate should work without additional trust steps. If you encounter browser warnings:
**Linux (Ubuntu/Debian):**
```bash
sudo cp .ssl/root.crt /usr/local/share/ca-certificates/caddy-local.crt
sudo update-ca-certificates
```
**Chrome/Chromium:**
1. Go to `chrome://settings/certificates`
2. Click "Authorities" → "Import"
3. Select `.ssl/root.crt`
4. Check "Trust this certificate for identifying websites"
**Checking Certificate Expiration:**
```bash
openssl x509 -enddate -noout -in .ssl/localhost.crt
```
@@ -220,6 +232,7 @@ To enable web push notifications:
## 🏗 Architecture Overview
### Queue System
```
User submits URL → Queue Manager → Queue Processor
@@ -231,7 +244,7 @@ User submits URL → Queue Manager → Queue Processor
### Processing Pipeline
1. **Extraction Phase**: Browser automation extracts text and images
2. **Parsing Phase**: LLM converts text to structured recipe data
2. **Parsing Phase**: LLM converts text to structured recipe data
3. **Upload Phase**: Automatic upload to Tandoor (if configured)
Each phase tracks progress and can fail independently with proper error handling.
@@ -247,9 +260,9 @@ Each phase tracks progress and can fail independently with proper error handling
# Run all tests
npm test
# Run specific test suites
# Run specific test suites
npm run test:unit # Unit tests only
npm run test:client # Browser tests only
npm run test:client # Browser tests only
npm run test:server # Server tests only
# Run tests in watch mode
@@ -257,9 +270,10 @@ npm run test:watch
```
Test Coverage:
- **138 total tests** covering all major components
- Queue Manager: 28 tests
- Queue Processor: 5 integration tests
- Queue Processor: 5 integration tests
- API Endpoints: 17 tests
- SSE Streaming: 6 tests
- Frontend Components: Browser tests
@@ -279,11 +293,13 @@ npm run preview
### Deployment
The app is built as a Node.js application with the following outputs:
- `/.svelte-kit/output/server/` - Server bundle
- `/.svelte-kit/output/client/` - Static assets
- `/.svelte-kit/output/client/` - Static assets
- `/build/` - Adapter output
Deploy the server bundle with:
```bash
node build/index.js
```
@@ -307,13 +323,15 @@ CMD ["node", "build"]
The app was migrated from a synchronous extraction system to an async queue-based system:
**Before (Synchronous)**:
- User waited for entire extraction process to complete
- No progress tracking during processing
- No retry capability for failures
- No retry capability for failures
- Single-threaded processing
- Limited error handling
**After (Async Queue)**:
- Fire-and-forget: submit URL and redirect immediately
- Real-time progress tracking via SSE
- Comprehensive retry system for failures
@@ -324,16 +342,18 @@ The app was migrated from a synchronous extraction system to an async queue-base
### API Migration
**Old Synchronous Endpoints** (Deprecated):
```bash
POST /api/extract # Submit URL and wait for completion
GET /api/extract-stream # Long-polling for progress
```
**New Queue Endpoints**:
```bash
POST /api/queue # Submit URL, get queue ID immediately
GET /api/queue # List all queue items
GET /api/queue/{id} # Get specific item status
GET /api/queue/{id} # Get specific item status
POST /api/queue/{id}/retry # Retry failed items
GET /api/queue/stream # Real-time SSE updates
```
@@ -344,13 +364,14 @@ If migrating from the old system:
1. **Update Client Code**: Replace `/api/extract` calls with `/api/queue`
2. **Handle Async Responses**: Process queue ID instead of waiting for completion
3. **Add Progress Tracking**: Implement SSE listeners for real-time updates
3. **Add Progress Tracking**: Implement SSE listeners for real-time updates
4. **Update Error Handling**: Handle new error classification system
5. **Add Retry Logic**: Implement retry functionality for failed items
### Backward Compatibility
The legacy endpoints are still available but deprecated:
- They will return `410 Gone` status with migration instructions
- Support will be removed in a future version
- All new development should use the queue endpoints
@@ -383,4 +404,3 @@ This project is licensed under the MIT License - see the LICENSE file for detail
- [Tandoor Recipe Manager](https://docs.tandoor.dev/) - Recipe management system
- [Workbox](https://developers.google.com/web/tools/workbox) - PWA capabilities
- [fastq](https://github.com/mcollina/fastq) - High-performance queue processing