mozempk/insta-recipe
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

simplify

Browse Source
This commit is contained in:
Giancarmine Salucci
2026-02-18 01:21:44 +01:00
parent 54321fd7c9
commit 49bccf8f15
84 changed files with 14474 additions and 13925 deletions
Download Patch File Download Diff File Expand all files Collapse all files

135
docs/MIGRATION.md
View File

@@ -11,6 +11,7 @@ The migration transformed InstaRecipe from a blocking, synchronous extraction sy
### Architecture Transformation
**Before: Synchronous System**
```
User Request → Direct Processing → Response (wait 30-60s)
↓ ↓ ↓
@@ -18,6 +19,7 @@ User Request → Direct Processing → Response (wait 30-60s)
```
**After: Async Queue System**
```
User Request → Queue Item Created → Immediate Response
↓ ↓ ↓
@@ -60,6 +62,7 @@ User Request → Queue Item Created → Immediate Response
### New Endpoints
#### Queue Management
```typescript
// Enqueue URL for processing
POST /api/queue
@@ -84,6 +87,7 @@ Events: connection, queue-update, ping
```
#### Push Notifications
```typescript
// Subscribe to push notifications
POST /api/notifications/subscribe
@@ -101,11 +105,11 @@ These endpoints are marked for removal and should not be used in new code:
```typescript
// ❌ DEPRECATED: Synchronous extraction
POST /api/extract
POST / api / extract;
// 👉 Use: POST /api/queue
// ❌ DEPRECATED: Long-polling progress
GET /api/extract-stream
GET / api / extract - stream;
// 👉 Use: GET /api/queue/stream
```
@@ -117,33 +121,33 @@ New queue items follow this structure:
```typescript
interface QueueItem {
id: string; // UUID v4
url: string; // Instagram URL
status: 'pending' | 'in_progress' | 'success' | 'error' | 'unhealthy';
// Processing phases with individual progress
phases: Array<{
name: 'extraction' | 'parsing' | 'uploading';
status: 'pending' | 'in_progress' | 'completed' | 'error';
startedAt?: string;
completedAt?: string;
progress?: number; // 0-100
}>;
// Results (populated on success)
results?: {
recipe?: Recipe; // Extracted recipe data
tandoorUrl?: string; // Link to uploaded recipe
extractedText?: string; // Raw extracted text
thumbnail?: string; // Image URL
};
// Error information
error?: string;
// Timestamps
createdAt: string;
updatedAt: string;
id: string; // UUID v4
url: string; // Instagram URL
status: 'pending' | 'in_progress' | 'success' | 'error' | 'unhealthy';
// Processing phases with individual progress
phases: Array<{
name: 'extraction' | 'parsing' | 'uploading';
status: 'pending' | 'in_progress' | 'completed' | 'error';
startedAt?: string;
completedAt?: string;
progress?: number; // 0-100
}>;
// Results (populated on success)
results?: {
recipe?: Recipe; // Extracted recipe data
tandoorUrl?: string; // Link to uploaded recipe
extractedText?: string; // Raw extracted text
thumbnail?: string; // Image URL
};
// Error information
error?: string;
// Timestamps
createdAt: string;
updatedAt: string;
}
```
@@ -167,33 +171,35 @@ interface QueueStatusUpdate {
### For Frontend Applications
1. **Replace Synchronous Calls**
```typescript
// ❌ Old synchronous approach
const response = await fetch('/api/extract', {
method: 'POST',
body: JSON.stringify({ url })
method: 'POST',
body: JSON.stringify({ url })
});
const result = await response.json(); // Wait 30-60 seconds
// ✅ New async queue approach
const response = await fetch('/api/queue', {
method: 'POST',
body: JSON.stringify({ url })
method: 'POST',
body: JSON.stringify({ url })
});
const queueItem = await response.json(); // Immediate response
// Navigate to dashboard for real-time updates
window.location.href = `/?highlight=${queueItem.id}`;
```
2. **Add Real-time Updates**
```typescript
// Setup Server-Sent Events for progress tracking
const eventSource = new EventSource(`/api/queue/stream?itemId=${itemId}`);
eventSource.addEventListener('queue-update', (event) => {
const update = JSON.parse(event.data);
updateUI(update);
const update = JSON.parse(event.data);
updateUI(update);
});
```
@@ -201,36 +207,37 @@ interface QueueStatusUpdate {
```typescript
// Handle different queue statuses
switch (item.status) {
case 'pending':
showPendingState();
break;
case 'in_progress':
showProgressBar(item.phases);
break;
case 'success':
showResults(item.results);
break;
case 'error':
showErrorWithRetry(item.error, item.id);
break;
case 'unhealthy':
showRetryableError(item.error, item.id);
break;
case 'pending':
showPendingState();
break;
case 'in_progress':
showProgressBar(item.phases);
break;
case 'success':
showResults(item.results);
break;
case 'error':
showErrorWithRetry(item.error, item.id);
break;
case 'unhealthy':
showRetryableError(item.error, item.id);
break;
}
```
### For Backend Integrations
1. **Update API Calls**
```python
# ❌ Old synchronous API
response = requests.post('/api/extract', json={'url': url})
# This would block for 30-60 seconds
# ✅ New async queue API
response = requests.post('/api/queue', json={'url': url})
queue_item = response.json()
# Poll or use SSE for updates
while True:
item = requests.get(f'/api/queue/{queue_item["id"]}').json()
@@ -240,9 +247,10 @@ interface QueueStatusUpdate {
```
2. **Implement SSE Client** (Python example)
```python
import sseclient
def listen_to_queue_updates(item_id):
messages = sseclient.SSEClient(f'/api/queue/stream?itemId={item_id}')
for msg in messages:
@@ -266,7 +274,7 @@ QUEUE_TIMEOUT_MS=30000 # Processing timeout
QUEUE_RETRY_ATTEMPTS=3 # Maximum retry attempts
# Push notification settings (optional)
VAPID_PUBLIC_KEY=BDummyPublicKey...
VAPID_PUBLIC_KEY=BDummyPublicKey...
VAPID_PRIVATE_KEY=DummyPrivateKey...
# Existing LLM and Tandoor settings remain the same
@@ -306,7 +314,7 @@ npm test
# Test specific components
npm test queue-manager
npm test queue-processor
npm test queue-processor
npm test queue-api
npm test queue-sse
```
@@ -314,18 +322,21 @@ npm test queue-sse
## Performance Considerations
### Before Migration
- **Blocking Operations**: Each request blocked a server thread
- **Single Processing**: One extraction at a time
- **No Progress**: Users waited without feedback
- **Memory Usage**: High memory usage during long operations
### After Migration
### After Migration
- **Non-blocking**: Requests return immediately
- **Concurrent Processing**: Multiple extractions in parallel
- **Real-time Feedback**: Live progress updates
- **Efficient Memory**: Event-driven, minimal memory footprint
### Performance Metrics
- **Response Time**: 50ms (queue) vs 30-60s (synchronous)
- **Throughput**: 2x concurrent processing vs 1x sequential
- **User Experience**: Immediate feedback vs long waiting
@@ -336,11 +347,13 @@ npm test queue-sse
If issues arise, the system can be rolled back by:
1. **Disable Queue Processing**
```env
QUEUE_PROCESSING_ENABLED=false
```
2. **Re-enable Legacy Endpoints** (if preserved)
```typescript
// Temporary fallback to synchronous processing
app.post('/api/extract', legacyExtractHandler);
@@ -389,10 +402,10 @@ curl -X POST https://localhost:5173/api/notifications/vapid-key
The migration to an async queue system represents a significant architectural improvement that provides:
- **Better User Experience**: Immediate responses and real-time progress
- **Improved Reliability**: Error recovery and retry mechanisms
- **Improved Reliability**: Error recovery and retry mechanisms
- **Enhanced Performance**: Concurrent processing and resource efficiency
- **Modern Features**: Push notifications and PWA capabilities
The new system maintains backward compatibility during the transition period while providing a clear migration path for all integrations.
For questions or issues during migration, refer to the comprehensive test suite and documentation, or open an issue in the project repository.
For questions or issues during migration, refer to the comprehensive test suite and documentation, or open an issue in the project repository.