fix(ssr): resolve EventSource SSR violations and implement best practices

- Fix EventSource is not defined error in queue dashboard - Add browser guards for all EventSource usage - Replace static constants (EventSource.OPEN/CLOSED) with numeric values - Fix setInterval SSR violation in LLM health indicator - Replace $effect anti-pattern with onMount in share page - Add comprehensive SvelteKit SSR best practices documentation - Add SSR audit and testing verification All changes follow SvelteKit best practices and are verified against official documentation. Production build succeeds with no SSR errors. Closes: FixEventSourceSSR See: docs/outcomes/FixEventSourceSSR.md
2025-12-22 03:00:29 +01:00
parent 35d6f6e40a
commit 8545744bb1
47 changed files with 12827 additions and 363 deletions
--- a/README.md
+++ b/README.md
@@ -1,58 +1,386 @@
-# sv
+# InstaRecipe - Async Instagram Recipe Extractor

-Everything you need to build a Svelte project, powered by [`sv`](https://github.com/sveltejs/cli).
+A modern web application that extracts recipes from Instagram posts and saves them to Tandoor Recipe Manager using an async queue-based processing system.

-## Creating a project
+## 🚀 Features

-If you're seeing this, you've probably already done this step. Congrats!
+### Core Functionality
+- **Async Queue Processing**: Fire-and-forget recipe extraction with background processing
+- **Real-time Updates**: Server-Sent Events for live progress tracking
+- **Push Notifications**: Background notifications when recipes complete
+- **Instagram Integration**: Extract recipes from Instagram posts and stories
+- **Tandoor Integration**: Automatic upload to Tandoor Recipe Manager
+- **PWA Support**: Installable Progressive Web App with offline capabilities

-```sh
-# create a new project in the current directory
-npx sv create
+### User Experience
+- **Queue Dashboard**: Monitor all recipe extractions in real-time
+- **Share Integration**: Browser share target for easy URL submission
+- **Responsive Design**: Works on desktop, tablet, and mobile
+- **Error Recovery**: Retry failed extractions with one click
+- **Progress Tracking**: Visual progress through extraction phases

-# create a new project in my-app
-npx sv create my-app
+### Technical Architecture
+- **SvelteKit Frontend**: Modern reactive UI with TypeScript
+- **Hexagonal Architecture**: Clean separation of concerns
+- **In-Memory Queue**: High-performance processing with configurable concurrency
+- **Three-Phase Pipeline**: Extraction → Parsing → Uploading
+- **Comprehensive Testing**: 138 tests covering all components
+
+## 📋 API Endpoints
+
+### Queue Management
+- `POST /api/queue` - Enqueue Instagram URL for processing
+- `GET /api/queue` - List queue items with filtering and pagination
+- `GET /api/queue/{id}` - Get specific queue item details
+- `POST /api/queue/{id}/retry` - Retry failed item
+- `GET /api/queue/stream` - Server-Sent Events for real-time updates
+
+### Push Notifications
+- `POST /api/notifications/subscribe` - Subscribe to push notifications
+- `DELETE /api/notifications/subscribe` - Unsubscribe from notifications
+- `GET /api/notifications/vapid-key` - Get VAPID public key
+
+### Legacy Endpoints (Deprecated)
+- ~~`POST /api/extract`~~ - Use `/api/queue` instead
+- ~~`GET /api/extract-stream`~~ - Use `/api/queue/stream` instead
+
+## 🛠 Development Setup
+
+### Prerequisites
+- Node.js 18+ 
+- npm or pnpm
+- Tandoor Recipe Manager instance (optional)
+- LLM API access (OpenAI, Anthropic, or local)
+
+### Installation
+
+```bash
+# Clone the repository
+git clone <repository-url>
+cd insta-recipe
+
+# Install dependencies
+npm install
+
+# Copy environment template
+cp .env.example .env
+
+# Configure your environment variables (see Configuration section)
 ```

-## Developing
+### Local Development

-Once you've created a project and installed dependencies with `npm install` (or `pnpm install` or `yarn`), start a development server:
-
-```sh
+```bash
+# Start development server with HTTPS
 npm run dev

-# or start the server and open the app in a new browser tab
-npm run dev -- --open
+# Open in browser (certificate must be trusted)
+open https://localhost:5173
 ```

-## Building
+The app runs on HTTPS by default for:
+- Service worker support (required for PWA)
+- Push notifications
+- Browser share target API
+- Instagram cookie handling

-To create a production version of your app:
+### SSL Certificate Setup

-```sh
+The application uses HTTPS in development with SSL certificates signed by an external Caddy CA container. The current certificate is valid until **December 20, 2035** (10 years).
+
+**Certificate Information:**
+- Location: `.ssl/` directory
+- CA Certificate: `.ssl/root.crt` (already trusted on the system)
+- Server Certificate: `.ssl/localhost.crt`
+- Server Private Key: `.ssl/localhost.key`
+
+Since the Caddy CA is already trusted on the system, the certificate should work without additional trust steps. If you encounter browser warnings:
+
+**Linux (Ubuntu/Debian):**
+```bash
+sudo cp .ssl/root.crt /usr/local/share/ca-certificates/caddy-local.crt
+sudo update-ca-certificates
+```
+
+**Chrome/Chromium:**
+1. Go to `chrome://settings/certificates`
+2. Click "Authorities" → "Import"
+3. Select `.ssl/root.crt`
+4. Check "Trust this certificate for identifying websites"
+
+**Checking Certificate Expiration:**
+```bash
+openssl x509 -enddate -noout -in .ssl/localhost.crt
+```
+
+**Regenerating the Certificate (if needed):**
+
+If the certificate expires or needs to be regenerated:
+
+```bash
+# Identify the Caddy container (usually named caddy-local)
+CADDY_CONTAINER="caddy-local"
+
+# Copy Caddy's CA certificate and private key
+docker cp $CADDY_CONTAINER:/data/caddy/pki/authorities/local/root.crt .ssl/root.crt
+docker cp $CADDY_CONTAINER:/data/caddy/pki/authorities/local/root.key .ssl/caddy-ca.key
+
+# Generate new server private key
+openssl genrsa -out .ssl/localhost.key 2048
+
+# Generate Certificate Signing Request (CSR)
+openssl req -new \
+  -key .ssl/localhost.key \
+  -out .ssl/localhost.csr \
+  -subj "/O=Caddy Local Authority/CN=localhost"
+
+# Create OpenSSL config for Subject Alternative Names
+cat > .ssl/localhost.ext << 'EOF'
+authorityKeyIdentifier=keyid,issuer
+basicConstraints=CA:FALSE
+keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
+extendedKeyUsage = serverAuth
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = localhost
+DNS.2 = *.localhost
+IP.1 = 127.0.0.1
+IP.2 = ::1
+EOF
+
+# Sign certificate with Caddy's CA (10 years = 3650 days)
+openssl x509 -req \
+  -in .ssl/localhost.csr \
+  -CA .ssl/root.crt \
+  -CAkey .ssl/caddy-ca.key \
+  -CAcreateserial \
+  -out .ssl/localhost.crt \
+  -days 3650 \
+  -sha256 \
+  -extfile .ssl/localhost.ext
+
+# Cleanup temporary files and set permissions
+rm .ssl/localhost.csr .ssl/localhost.ext .ssl/caddy-ca.key .ssl/root.srl
+chmod 600 .ssl/localhost.key
+chmod 644 .ssl/localhost.crt .ssl/root.crt
+
+# Verify the certificate
+openssl verify -CAfile .ssl/root.crt .ssl/localhost.crt
+```
+
+## ⚙️ Configuration
+
+### Environment Variables
+
+Create a `.env` file with the following variables:
+
+```env
+# LLM Configuration
+LLM_API_BASE_URL=https://api.openai.com/v1
+LLM_API_KEY=your-api-key
+LLM_MODEL=gpt-4o-mini
+
+# Tandoor Integration (optional)
+TANDOOR_BASE_URL=https://your-tandoor.com
+TANDOOR_API_KEY=your-tandoor-token
+
+# Queue Processing
+QUEUE_CONCURRENCY=2
+QUEUE_TIMEOUT_MS=30000
+
+# Push Notifications (optional)
+VAPID_PUBLIC_KEY=your-vapid-public-key
+VAPID_PRIVATE_KEY=your-vapid-private-key
+
+# Instagram Authentication (optional)
+AUTH_SCHEDULER_ENABLED=true
+AUTH_SCHEDULER_INTERVAL_MINUTES=720
+```
+
+### Tandoor Setup
+
+To automatically upload extracted recipes to Tandoor:
+
+1. Create an API token in your Tandoor instance
+2. Set `TANDOOR_BASE_URL` and `TANDOOR_API_KEY` in `.env`
+3. Recipes will be automatically uploaded after successful extraction
+
+### Push Notifications
+
+To enable web push notifications:
+
+1. Generate VAPID keys:
+   ```bash
+   npx web-push generate-vapid-keys
+   ```
+2. Set `VAPID_PUBLIC_KEY` and `VAPID_PRIVATE_KEY` in `.env`
+3. Users can enable notifications in the dashboard settings
+
+## 🏗 Architecture Overview
+
+### Queue System
+```
+User submits URL → Queue Manager → Queue Processor
+                                        ↓
+                 Extraction Phase ← → Parsing Phase ← → Upload Phase
+                                        ↓
+                 Push Notifications ← → SSE Updates ← → Dashboard Updates
+```
+
+### Processing Pipeline
+
+1. **Extraction Phase**: Browser automation extracts text and images
+2. **Parsing Phase**: LLM converts text to structured recipe data  
+3. **Upload Phase**: Automatic upload to Tandoor (if configured)
+
+Each phase tracks progress and can fail independently with proper error handling.
+
+### Error Classification
+
+- **Recoverable Errors** (`unhealthy`): Temporary issues, can be retried
+- **Non-recoverable Errors** (`error`): Invalid URLs, parsing failures, etc.
+
+## 🧪 Testing
+
+```bash
+# Run all tests
+npm test
+
+# Run specific test suites  
+npm run test:unit    # Unit tests only
+npm run test:client  # Browser tests only  
+npm run test:server  # Server tests only
+
+# Run tests in watch mode
+npm run test:watch
+```
+
+Test Coverage:
+- **138 total tests** covering all major components
+- Queue Manager: 28 tests
+- Queue Processor: 5 integration tests  
+- API Endpoints: 17 tests
+- SSE Streaming: 6 tests
+- Frontend Components: Browser tests
+
+## 📦 Building & Deployment
+
+### Production Build
+
+```bash
+# Build for production
 npm run build
+
+# Preview production build locally
+npm run preview
 ```

-You can preview the production build with `npm run preview`.
+### Deployment

-> To deploy your app, you may need to install an [adapter](https://svelte.dev/docs/kit/adapters) for your target environment.
+The app is built as a Node.js application with the following outputs:
+- `/.svelte-kit/output/server/` - Server bundle
+- `/.svelte-kit/output/client/` - Static assets  
+- `/build/` - Adapter output

-## Local SSL Development
+Deploy the server bundle with:
+```bash
+node build/index.js
+```

-This project uses HTTPS for local development. The certificates are generated using a local Caddy instance.
+### Docker Deployment

-To trust the local CA and avoid browser warnings:
+```dockerfile
+FROM node:18-alpine
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production
+COPY build ./build
+EXPOSE 3000
+CMD ["node", "build"]
+```

-1.  **Linux (Ubuntu/Debian):**
-    ```bash
-    sudo cp .ssl/root.crt /usr/local/share/ca-certificates/caddy-local.crt
-    sudo update-ca-certificates
-    ```
+## 🔄 Migration from Synchronous System

-2.  **Chrome/Chromium:**
-    You might need to import the authority in Chrome settings:
-    - Go to `chrome://settings/certificates`
-    - Click "Authorities" -> "Import"
-    - Select `.ssl/root.crt`
-    - Check "Trust this certificate for identifying websites"
+### What Changed
+
+The app was migrated from a synchronous extraction system to an async queue-based system:
+
+**Before (Synchronous)**:
+- User waited for entire extraction process to complete
+- No progress tracking during processing
+- No retry capability for failures  
+- Single-threaded processing
+- Limited error handling
+
+**After (Async Queue)**:
+- Fire-and-forget: submit URL and redirect immediately
+- Real-time progress tracking via SSE
+- Comprehensive retry system for failures
+- Concurrent processing (configurable)
+- Detailed error classification and reporting
+- Push notifications for background updates
+
+### API Migration
+
+**Old Synchronous Endpoints** (Deprecated):
+```bash
+POST /api/extract          # Submit URL and wait for completion
+GET /api/extract-stream    # Long-polling for progress
+```
+
+**New Queue Endpoints**:
+```bash
+POST /api/queue           # Submit URL, get queue ID immediately
+GET /api/queue           # List all queue items
+GET /api/queue/{id}      # Get specific item status  
+POST /api/queue/{id}/retry  # Retry failed items
+GET /api/queue/stream    # Real-time SSE updates
+```
+
+### Migration Steps
+
+If migrating from the old system:
+
+1. **Update Client Code**: Replace `/api/extract` calls with `/api/queue`
+2. **Handle Async Responses**: Process queue ID instead of waiting for completion
+3. **Add Progress Tracking**: Implement SSE listeners for real-time updates  
+4. **Update Error Handling**: Handle new error classification system
+5. **Add Retry Logic**: Implement retry functionality for failed items
+
+### Backward Compatibility
+
+The legacy endpoints are still available but deprecated:
+- They will return `410 Gone` status with migration instructions
+- Support will be removed in a future version
+- All new development should use the queue endpoints
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Make your changes with tests
+4. Run the test suite (`npm test`)
+5. Commit your changes (`git commit -m 'Add amazing feature'`)
+6. Push to the branch (`git push origin feature/amazing-feature`)
+7. Open a Pull Request
+
+### Development Guidelines
+
+- Follow TypeScript strict mode
+- Add tests for all new functionality
+- Use the existing architecture patterns (Hexagonal Architecture)
+- Update documentation for API changes
+- Ensure PWA functionality remains intact
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+## 🙏 Acknowledgments
+
+- [SvelteKit](https://kit.svelte.dev/) - Application framework
+- [Tandoor Recipe Manager](https://docs.tandoor.dev/) - Recipe management system
+- [Workbox](https://developers.google.com/web/tools/workbox) - PWA capabilities
+- [fastq](https://github.com/mcollina/fastq) - High-performance queue processing