simplify
This commit is contained in:
40
README.md
40
README.md
@@ -5,6 +5,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
## 🚀 Features
|
||||
|
||||
### Core Functionality
|
||||
|
||||
- **Async Queue Processing**: Fire-and-forget recipe extraction with background processing
|
||||
- **Real-time Updates**: Server-Sent Events for live progress tracking
|
||||
- **Push Notifications**: Background notifications when recipes complete
|
||||
@@ -13,6 +14,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
- **PWA Support**: Installable Progressive Web App with offline capabilities
|
||||
|
||||
### User Experience
|
||||
|
||||
- **Queue Dashboard**: Monitor all recipe extractions in real-time
|
||||
- **Share Integration**: Browser share target for easy URL submission
|
||||
- **Responsive Design**: Works on desktop, tablet, and mobile
|
||||
@@ -20,6 +22,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
- **Progress Tracking**: Visual progress through extraction phases
|
||||
|
||||
### Technical Architecture
|
||||
|
||||
- **SvelteKit Frontend**: Modern reactive UI with TypeScript
|
||||
- **Hexagonal Architecture**: Clean separation of concerns
|
||||
- **In-Memory Queue**: High-performance processing with configurable concurrency
|
||||
@@ -29,6 +32,7 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
## 📋 API Endpoints
|
||||
|
||||
### Queue Management
|
||||
|
||||
- `POST /api/queue` - Enqueue Instagram URL for processing
|
||||
- `GET /api/queue` - List queue items with filtering and pagination
|
||||
- `GET /api/queue/{id}` - Get specific queue item details
|
||||
@@ -36,18 +40,21 @@ A modern web application that extracts recipes from Instagram posts and saves th
|
||||
- `GET /api/queue/stream` - Server-Sent Events for real-time updates
|
||||
|
||||
### Push Notifications
|
||||
|
||||
- `POST /api/notifications/subscribe` - Subscribe to push notifications
|
||||
- `DELETE /api/notifications/subscribe` - Unsubscribe from notifications
|
||||
- `GET /api/notifications/vapid-key` - Get VAPID public key
|
||||
|
||||
### Legacy Endpoints (Deprecated)
|
||||
|
||||
- ~~`POST /api/extract`~~ - Use `/api/queue` instead
|
||||
- ~~`GET /api/extract-stream`~~ - Use `/api/queue/stream` instead
|
||||
|
||||
## 🛠 Development Setup
|
||||
|
||||
### Prerequisites
|
||||
- Node.js 18+
|
||||
|
||||
- Node.js 18+
|
||||
- npm or pnpm
|
||||
- Tandoor Recipe Manager instance (optional)
|
||||
- LLM API access (OpenAI, Anthropic, or local)
|
||||
@@ -79,6 +86,7 @@ open https://localhost:5173
|
||||
```
|
||||
|
||||
The app runs on HTTPS by default for:
|
||||
|
||||
- Service worker support (required for PWA)
|
||||
- Push notifications
|
||||
- Browser share target API
|
||||
@@ -89,6 +97,7 @@ The app runs on HTTPS by default for:
|
||||
The application uses HTTPS in development with SSL certificates signed by an external Caddy CA container. The current certificate is valid until **December 20, 2035** (10 years).
|
||||
|
||||
**Certificate Information:**
|
||||
|
||||
- Location: `.ssl/` directory
|
||||
- CA Certificate: `.ssl/root.crt` (already trusted on the system)
|
||||
- Server Certificate: `.ssl/localhost.crt`
|
||||
@@ -97,18 +106,21 @@ The application uses HTTPS in development with SSL certificates signed by an ext
|
||||
Since the Caddy CA is already trusted on the system, the certificate should work without additional trust steps. If you encounter browser warnings:
|
||||
|
||||
**Linux (Ubuntu/Debian):**
|
||||
|
||||
```bash
|
||||
sudo cp .ssl/root.crt /usr/local/share/ca-certificates/caddy-local.crt
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
**Chrome/Chromium:**
|
||||
|
||||
1. Go to `chrome://settings/certificates`
|
||||
2. Click "Authorities" → "Import"
|
||||
3. Select `.ssl/root.crt`
|
||||
4. Check "Trust this certificate for identifying websites"
|
||||
|
||||
**Checking Certificate Expiration:**
|
||||
|
||||
```bash
|
||||
openssl x509 -enddate -noout -in .ssl/localhost.crt
|
||||
```
|
||||
@@ -220,6 +232,7 @@ To enable web push notifications:
|
||||
## 🏗 Architecture Overview
|
||||
|
||||
### Queue System
|
||||
|
||||
```
|
||||
User submits URL → Queue Manager → Queue Processor
|
||||
↓
|
||||
@@ -231,7 +244,7 @@ User submits URL → Queue Manager → Queue Processor
|
||||
### Processing Pipeline
|
||||
|
||||
1. **Extraction Phase**: Browser automation extracts text and images
|
||||
2. **Parsing Phase**: LLM converts text to structured recipe data
|
||||
2. **Parsing Phase**: LLM converts text to structured recipe data
|
||||
3. **Upload Phase**: Automatic upload to Tandoor (if configured)
|
||||
|
||||
Each phase tracks progress and can fail independently with proper error handling.
|
||||
@@ -247,9 +260,9 @@ Each phase tracks progress and can fail independently with proper error handling
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run specific test suites
|
||||
# Run specific test suites
|
||||
npm run test:unit # Unit tests only
|
||||
npm run test:client # Browser tests only
|
||||
npm run test:client # Browser tests only
|
||||
npm run test:server # Server tests only
|
||||
|
||||
# Run tests in watch mode
|
||||
@@ -257,9 +270,10 @@ npm run test:watch
|
||||
```
|
||||
|
||||
Test Coverage:
|
||||
|
||||
- **138 total tests** covering all major components
|
||||
- Queue Manager: 28 tests
|
||||
- Queue Processor: 5 integration tests
|
||||
- Queue Processor: 5 integration tests
|
||||
- API Endpoints: 17 tests
|
||||
- SSE Streaming: 6 tests
|
||||
- Frontend Components: Browser tests
|
||||
@@ -279,11 +293,13 @@ npm run preview
|
||||
### Deployment
|
||||
|
||||
The app is built as a Node.js application with the following outputs:
|
||||
|
||||
- `/.svelte-kit/output/server/` - Server bundle
|
||||
- `/.svelte-kit/output/client/` - Static assets
|
||||
- `/.svelte-kit/output/client/` - Static assets
|
||||
- `/build/` - Adapter output
|
||||
|
||||
Deploy the server bundle with:
|
||||
|
||||
```bash
|
||||
node build/index.js
|
||||
```
|
||||
@@ -307,13 +323,15 @@ CMD ["node", "build"]
|
||||
The app was migrated from a synchronous extraction system to an async queue-based system:
|
||||
|
||||
**Before (Synchronous)**:
|
||||
|
||||
- User waited for entire extraction process to complete
|
||||
- No progress tracking during processing
|
||||
- No retry capability for failures
|
||||
- No retry capability for failures
|
||||
- Single-threaded processing
|
||||
- Limited error handling
|
||||
|
||||
**After (Async Queue)**:
|
||||
|
||||
- Fire-and-forget: submit URL and redirect immediately
|
||||
- Real-time progress tracking via SSE
|
||||
- Comprehensive retry system for failures
|
||||
@@ -324,16 +342,18 @@ The app was migrated from a synchronous extraction system to an async queue-base
|
||||
### API Migration
|
||||
|
||||
**Old Synchronous Endpoints** (Deprecated):
|
||||
|
||||
```bash
|
||||
POST /api/extract # Submit URL and wait for completion
|
||||
GET /api/extract-stream # Long-polling for progress
|
||||
```
|
||||
|
||||
**New Queue Endpoints**:
|
||||
|
||||
```bash
|
||||
POST /api/queue # Submit URL, get queue ID immediately
|
||||
GET /api/queue # List all queue items
|
||||
GET /api/queue/{id} # Get specific item status
|
||||
GET /api/queue/{id} # Get specific item status
|
||||
POST /api/queue/{id}/retry # Retry failed items
|
||||
GET /api/queue/stream # Real-time SSE updates
|
||||
```
|
||||
@@ -344,13 +364,14 @@ If migrating from the old system:
|
||||
|
||||
1. **Update Client Code**: Replace `/api/extract` calls with `/api/queue`
|
||||
2. **Handle Async Responses**: Process queue ID instead of waiting for completion
|
||||
3. **Add Progress Tracking**: Implement SSE listeners for real-time updates
|
||||
3. **Add Progress Tracking**: Implement SSE listeners for real-time updates
|
||||
4. **Update Error Handling**: Handle new error classification system
|
||||
5. **Add Retry Logic**: Implement retry functionality for failed items
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
The legacy endpoints are still available but deprecated:
|
||||
|
||||
- They will return `410 Gone` status with migration instructions
|
||||
- Support will be removed in a future version
|
||||
- All new development should use the queue endpoints
|
||||
@@ -383,4 +404,3 @@ This project is licensed under the MIT License - see the LICENSE file for detail
|
||||
- [Tandoor Recipe Manager](https://docs.tandoor.dev/) - Recipe management system
|
||||
- [Workbox](https://developers.google.com/web/tools/workbox) - PWA capabilities
|
||||
- [fastq](https://github.com/mcollina/fastq) - High-performance queue processing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user