fix: resolve critical app functionality issues

Complete implementation of fixes for queue processing, SSE connection display, service worker installation, and failing tests. Key Changes: - Fix queue processor startup with proper import and subscription mechanism - Implement centralized API error handling middleware for proper HTTP status codes - Enhance service worker configuration for PWA compliance and reliability - Fix SSE connection display with reactive state management - Add comprehensive test coverage and health check endpoints Results: - All 169 tests now passing (previously 16 failing) - Queue items process immediately from pending to success/error states - Real-time SSE connection status with auto-reconnection logic - Proper PWA functionality with working service worker registration - API endpoints return correct HTTP status codes (400/404/409) instead of 500 errors This resolves the critical issues preventing core app functionality and enables proper production deployment.
2025-12-22 04:27:59 +01:00
parent b60f96a75e
commit 93aa25a31c
25 changed files with 3243 additions and 559 deletions
--- a/docs/plans/FixCriticalAppFunctionalityIssues.md
+++ b/docs/plans/FixCriticalAppFunctionalityIssues.md
@@ -0,0 +1,611 @@
+# Execution Plan: Fix Critical App Functionality Issues
+
+**OUTCOME_NAME:** FixCriticalAppFunctionalityIssues
+
+**Created:** 22 December 2025
+
+**Problem Statement:** The insta-recipe application has four critical issues preventing core functionality: (1) Queued items never start processing despite the queue system being implemented, (2) Frontend SSE connection status display never updates properly, (3) Service worker never gets installed due to registration conflicts, and (4) Multiple tests are failing with incorrect error status codes and queue processing timeouts. These interconnected issues are preventing the app from functioning as intended and frustrating users who expect a working queue processing system with real-time updates and PWA functionality.
+
+---
+
+## Current State Analysis
+
+### Issue 1: Queue Processing System Not Starting
+**Symptoms:**
+- Items are successfully enqueued but remain in 'pending' status indefinitely
+- Queue processor integration tests failing with items never progressing to 'success'
+- No processing activity visible in logs despite QueueProcessor being implemented
+
+**Evidence:**
+- Test failure: `expect(updated?.status).toBe('success')` but received `'pending'`
+- QueueProcessor has auto-start code: `queueProcessor.start()` on module import
+- But processor may not be imported anywhere in the running application
+
+### Issue 2: SSE Connection Status Never Updates
+**Symptoms:**
+- EventSource connection may be working but UI never shows "Live updates"
+- Connection status indicator remains "Disconnected" even when SSE is functional
+- Real-time queue updates may not be displaying properly in frontend
+
+**Evidence:**
+- EventSource implementation exists with proper browser guards
+- SSE endpoint `/api/queue/stream` implemented and functional
+- Connection status logic: `{eventSource?.readyState === 1 ? 'Live updates' : 'Disconnected'}`
+
+### Issue 3: Service Worker Never Installs
+**Symptoms:**
+- PWA functionality broken due to service worker registration failures
+- Multiple attempts to fix workbox initialization haven't resolved the core issue
+- Conflicts between SvelteKit and vite-pwa service worker registration
+
+**Evidence:**
+- Multiple registerSW.js files indicating conflicting registration attempts
+- Service worker has comprehensive error handling but still fails to register
+- SvelteKit service worker not properly disabled in configuration
+
+### Issue 4: Test Suite Failures (16 tests failing)
+**Symptoms:**
+- Queue API endpoints returning 500 status codes instead of expected 400/404/409
+- Queue processor integration tests timing out waiting for processing
+- Error handling not working correctly across API endpoints
+
+**Evidence:**
+- API tests: `expected 400 to be 500` pattern across multiple endpoints
+- Queue processor tests: Items remain 'pending' instead of being processed
+- Suggests both API error handling and queue processing are broken
+
+---
+
+## Root Cause Analysis
+
+### Primary Issue: QueueProcessor Not Starting in Production
+The QueueProcessor module exports a singleton that auto-starts on import, but it appears not to be imported anywhere in the running application. The tests import it directly, but the actual app may not be triggering the import, leaving the processor dormant.
+
+### Secondary Issue: API Error Handling Middleware Missing
+All queue API endpoints are throwing unhandled exceptions instead of returning proper HTTP status codes. This suggests missing or broken error handling middleware that should catch validation errors and return appropriate 400/404/409 responses.
+
+### Tertiary Issue: Service Worker Registration Conflicts
+SvelteKit's built-in service worker registration is still active while vite-pwa is also trying to register a service worker, creating conflicts that prevent either from working correctly.
+
+### Quaternary Issue: SSE Reactivity Problems
+The EventSource connection status may be working but not triggering reactive updates in the Svelte component, leaving the UI in a stale state.
+
+---
+
+## Solution Architecture
+
+### Hexagonal Architecture Mapping
+
+```
+┌─────────────────────────────────────────────────┐
+│        Primary Adapters (Inbound)               │
+│  - Queue API Endpoints: Process requests        │
+│  - Queue Dashboard: Display status              │
+│  - Service Worker: PWA functionality            │
+└─────────────────┬───────────────────────────────┘
+                  │
+┌─────────────────┴───────────────────────────────┐
+│           Domain (Core)                          │
+│  ┌────────────────────────────────────────────┐ │
+│  │  QueueManager (Port)                       │ │
+│  │  - Manages queue state and subscribers     │ │
+│  │  QueueProcessor (Domain Service)           │ │
+│  │  - Orchestrates processing workflow       │ │
+│  │  Error Handling (Domain Logic)             │ │
+│  │  - Validates inputs and manages errors    │ │
+│  └────────────────────────────────────────────┘ │
+└─────────────────┬───────────────────────────────┘
+                  │
+┌─────────────────┴───────────────────────────────┐
+│       Secondary Adapters (Outbound)             │
+│  - Extraction Service: Browser automation       │
+│  - Parser Service: LLM recipe extraction        │
+│  - Tandoor Service: Recipe upload               │
+│  - Push Notification Service: User alerts       │
+└─────────────────────────────────────────────────┘
+```
+
+### Technical Design
+
+#### Fix 1: Queue Processor Startup
+Ensure QueueProcessor is imported during app initialization by adding explicit import to server hooks, guaranteeing the auto-start code executes when the server starts.
+
+#### Fix 2: API Error Handling Middleware
+Implement comprehensive error handling that catches different exception types and returns proper HTTP status codes based on the error category (validation → 400, not found → 404, conflict → 409).
+
+#### Fix 3: Service Worker Registration
+Completely disable SvelteKit's service worker and ensure only vite-pwa handles registration, with proper workbox manifest injection for development and production environments.
+
+#### Fix 4: SSE Connection Reactivity  
+Add explicit reactivity triggers and connection state management to ensure UI updates when EventSource connection status changes.
+
+---
+
+## Story Breakdown
+
+### Story 1: Fix Queue Processor Startup and Import
+
+**Priority:** Critical  
+**Dependencies:** None  
+**Estimated Effort:** Small (1-2 hours)
+
+**Objective:** Ensure QueueProcessor starts when the application starts, enabling automatic processing of queued items.
+
+**Root Cause:** QueueProcessor singleton auto-starts on module import, but the module isn't imported anywhere in the running application, leaving the processor dormant despite being fully implemented.
+
+**Tasks:**
+1. Add explicit QueueProcessor import to `src/hooks.server.ts`
+2. Add startup logging to confirm processor initialization
+3. Add health check endpoint to verify processor status
+4. Test that queued items are processed automatically after restart
+5. Verify all existing queue processor tests pass
+
+**Technical Implementation:**
+
+```typescript
+// src/hooks.server.ts
+import './lib/server/queue/QueueProcessor'; // Trigger auto-start
+
+export const handle: Handle = async ({ event, resolve }) => {
+  // Existing handle logic
+};
+```
+
+**Acceptance Criteria:**
+- ✅ QueueProcessor starts automatically when application starts
+- ✅ Queued items transition from 'pending' to 'in_progress' to 'success'/'error'
+- ✅ Server logs show processor initialization and activity
+- ✅ All queue processor integration tests pass
+- ✅ Processing works in both development and production environments
+
+**Files:**
+- `src/hooks.server.ts` (modify - add import)
+- `src/routes/api/health/+server.ts` (new - health check endpoint)
+
+---
+
+### Story 2: Implement Comprehensive API Error Handling
+
+**Priority:** Critical  
+**Dependencies:** None  
+**Estimated Effort:** Medium (2-3 hours)
+
+**Objective:** Fix API endpoints to return proper HTTP status codes instead of 500 errors, enabling correct error handling and test validation.
+
+**Root Cause:** API endpoints are throwing unhandled exceptions that result in generic 500 responses instead of specific error status codes. Missing error handling middleware to catch and classify different error types.
+
+**Tasks:**
+1. Create error handling middleware for API endpoints
+2. Add validation error classification (400 for bad input)
+3. Add not found error handling (404 for missing resources)
+4. Add conflict error handling (409 for invalid state operations)
+5. Update all queue API endpoints to use proper error handling
+6. Test all API endpoints return correct status codes
+7. Verify all 12 failing API tests now pass
+
+**Technical Implementation:**
+
+```typescript
+// src/lib/server/api/errorHandler.ts
+export function handleApiError(error: unknown): Response {
+  if (error instanceof ValidationError) {
+    return json({ message: error.message }, { status: 400 });
+  }
+  if (error instanceof NotFoundError) {
+    return json({ message: error.message }, { status: 404 });
+  }
+  if (error instanceof ConflictError) {
+    return json({ message: error.message }, { status: 409 });
+  }
+  // Generic 500 for unexpected errors
+  console.error('Unhandled API error:', error);
+  return json({ message: 'Internal server error' }, { status: 500 });
+}
+
+// Usage in API endpoints:
+export const POST: RequestHandler = async ({ request }) => {
+  try {
+    // API logic
+  } catch (error) {
+    return handleApiError(error);
+  }
+};
+```
+
+**Acceptance Criteria:**
+- ✅ All queue API endpoints return correct HTTP status codes
+- ✅ Validation errors return 400 with descriptive messages
+- ✅ Not found errors return 404 with appropriate messages
+- ✅ Conflict errors return 409 with state information
+- ✅ All 12 failing API tests now pass
+- ✅ Error responses include helpful error messages
+- ✅ Unexpected errors still log to server console
+
+**Files:**
+- `src/lib/server/api/errorHandler.ts` (new)
+- `src/lib/server/api/errors.ts` (new - error classes)
+- `src/routes/api/queue/+server.ts` (modify)
+- `src/routes/api/queue/[id]/+server.ts` (modify)
+- `src/routes/api/queue/[id]/retry/+server.ts` (modify)
+
+---
+
+### Story 3: Resolve Service Worker Registration Conflicts
+
+**Priority:** High  
+**Dependencies:** None  
+**Estimated Effort:** Medium (2-3 hours)
+
+**Objective:** Fix service worker registration by eliminating conflicts and ensuring proper PWA functionality.
+
+**Root Cause:** SvelteKit's built-in service worker registration is still active while vite-pwa is also trying to register a service worker, creating conflicts. Additionally, workbox manifest injection may not be working correctly in the build process.
+
+**Tasks:**
+1. Disable SvelteKit service worker registration in `svelte.config.js`
+2. Verify vite-pwa configuration for proper workbox manifest injection
+3. Ensure service worker TypeScript compilation produces valid code
+4. Add environment-specific service worker behavior
+5. Test service worker registration in both development and production
+6. Verify PWA functionality (installation, offline, push notifications)
+7. Clean up conflicting registerSW.js files
+
+**Technical Implementation:**
+
+```typescript
+// svelte.config.js
+const config = {
+  kit: {
+    adapter: adapter(),
+    serviceWorker: {
+      register: false // Disable SvelteKit's service worker
+    }
+  }
+};
+
+// vite.config.ts - enhanced configuration
+SvelteKitPWA({
+  strategies: 'injectManifest',
+  filename: 'service-worker.ts',
+  mode: process.env.NODE_ENV === 'production' ? 'production' : 'development',
+  injectManifest: {
+    swSrc: 'src/service-worker.ts',
+    swDest: 'service-worker.js',
+    injectionPoint: 'self.__WB_MANIFEST'
+  },
+  workbox: {
+    globPatterns: ['client/**/*.{js,css,ico,png,svg,webp,woff,woff2}'],
+    cleanupOutdatedCaches: true
+  },
+  devOptions: {
+    enabled: true,
+    suppressWarnings: true,
+    type: 'module'
+  }
+});
+```
+
+**Acceptance Criteria:**
+- ✅ Service worker registers successfully without evaluation errors
+- ✅ Only one service worker registration visible in browser DevTools
+- ✅ Workbox precaching initializes correctly with proper manifest
+- ✅ Push notifications continue to work as before
+- ✅ PWA installation prompt appears correctly
+- ✅ Service worker works in both development and production
+- ✅ No service worker-related console errors
+
+**Files:**
+- `svelte.config.js` (modify - disable service worker registration)
+- `vite.config.ts` (modify - enhance vite-pwa config)
+- `src/service-worker.ts` (modify - add better error handling)
+
+---
+
+### Story 4: Fix SSE Connection Status Display
+
+**Priority:** Medium  
+**Dependencies:** Story 1 (for queue updates to flow)  
+**Estimated Effort:** Small (1-2 hours)
+
+**Objective:** Ensure frontend correctly displays SSE connection status and receives real-time updates from the queue processing system.
+
+**Root Cause:** EventSource connection may be working but not triggering reactive updates in the Svelte component, or event handling is not properly updating the UI state.
+
+**Tasks:**
+1. Debug EventSource connection establishment and event flow
+2. Add explicit reactivity triggers for connection status changes
+3. Enhance SSE event handling with better error recovery
+4. Add connection status indicators with proper state management
+5. Test real-time queue updates display correctly
+6. Add reconnection logic for dropped connections
+7. Verify connection status updates in real-time
+
+**Technical Implementation:**
+
+```svelte
+<!-- src/routes/+page.svelte -->
+<script lang="ts">
+  import { browser } from '$app/environment';
+  import { onMount, onDestroy } from 'svelte';
+  
+  let eventSource = $state<EventSource | null>(null);
+  let connectionStatus = $state<'connecting' | 'connected' | 'disconnected'>('disconnected');
+  let lastPing = $state<string | null>(null);
+
+  function startSSEConnection() {
+    if (!browser) return;
+    
+    connectionStatus = 'connecting';
+    
+    try {
+      eventSource = new EventSource('/api/queue/stream');
+      
+      eventSource.addEventListener('open', () => {
+        console.log('[SSE] Connection opened');
+        connectionStatus = 'connected';
+      });
+      
+      eventSource.addEventListener('connection', (event) => {
+        const data = JSON.parse(event.data);
+        console.log('[SSE] Connection confirmed:', data.message);
+        connectionStatus = 'connected';
+      });
+      
+      eventSource.addEventListener('ping', (event) => {
+        const data = JSON.parse(event.data);
+        lastPing = data.timestamp;
+        console.log('[SSE] Keep-alive ping received');
+      });
+      
+      eventSource.addEventListener('error', (event) => {
+        console.error('[SSE] Connection error:', event);
+        connectionStatus = 'disconnected';
+        // Reconnect after delay
+        setTimeout(() => {
+          if (eventSource?.readyState === 2) { // CLOSED
+            startSSEConnection();
+          }
+        }, 5000);
+      });
+      
+    } catch (error) {
+      console.error('[SSE] Failed to start connection:', error);
+      connectionStatus = 'disconnected';
+    }
+  }
+</script>
+
+<!-- Connection status indicator -->
+<div class="flex items-center space-x-2">
+  <div class="w-2 h-2 rounded-full {
+    connectionStatus === 'connected' ? 'bg-green-400' : 
+    connectionStatus === 'connecting' ? 'bg-yellow-400' :
+    'bg-red-400'
+  }"></div>
+  <span class="text-sm text-gray-600">
+    {connectionStatus === 'connected' ? 'Live updates' : 
+     connectionStatus === 'connecting' ? 'Connecting...' :
+     'Disconnected'}
+  </span>
+  {#if lastPing}
+    <span class="text-xs text-gray-400">
+      Last ping: {new Date(lastPing).toLocaleTimeString()}
+    </span>
+  {/if}
+</div>
+```
+
+**Acceptance Criteria:**
+- ✅ Connection status indicator shows correct state (connecting/connected/disconnected)
+- ✅ Status updates in real-time when connection state changes
+- ✅ Queue updates display immediately when items are processed
+- ✅ Connection automatically reconnects after temporary disconnections
+- ✅ Keep-alive pings maintain connection stability
+- ✅ Error states are handled gracefully with user feedback
+- ✅ Connection works reliably across page reloads and browser sessions
+
+**Files:**
+- `src/routes/+page.svelte` (modify - enhance connection status display)
+- `src/routes/api/queue/stream/+server.ts` (modify - improve event reliability)
+
+---
+
+## Testing Strategy
+
+### Unit Tests
+**QueueProcessor Import Test:**
+```typescript
+describe('QueueProcessor Startup', () => {
+  it('should auto-start when hooks.server.ts is imported');
+  it('should log startup confirmation');
+  it('should begin processing queued items immediately');
+});
+```
+
+**API Error Handling Tests:**
+```typescript
+describe('API Error Handling', () => {
+  it('should return 400 for validation errors');
+  it('should return 404 for not found errors');
+  it('should return 409 for conflict errors');
+  it('should include descriptive error messages');
+});
+```
+
+### Integration Tests
+**End-to-End Queue Processing:**
+```typescript
+describe('Complete Queue Flow', () => {
+  it('should enqueue item via API');
+  it('should auto-process item through all phases');
+  it('should send SSE updates during processing');
+  it('should complete with success status');
+});
+```
+
+**Service Worker Installation:**
+```typescript
+describe('PWA Functionality', () => {
+  it('should register service worker without errors');
+  it('should support push notifications');
+  it('should enable offline functionality');
+});
+```
+
+### Acceptance Testing
+1. **Manual Queue Testing:**
+   - Submit Instagram URL via share page
+   - Verify item appears in queue dashboard
+   - Confirm processing starts automatically
+   - Verify real-time status updates
+   - Check final recipe extraction
+
+2. **SSE Connection Testing:**
+   - Load queue dashboard
+   - Verify connection status shows "Live updates"
+   - Submit new item in another tab
+   - Confirm real-time update in dashboard
+   - Test reconnection after network interruption
+
+3. **Service Worker Testing:**
+   - Clear all service workers in DevTools
+   - Reload application
+   - Verify single service worker registration
+   - Test push notification functionality
+   - Verify PWA installation prompt
+
+4. **Cross-Browser Testing:**
+   - Test in Chrome, Firefox, Safari, Edge
+   - Verify all functionality works consistently
+   - Check for browser-specific issues
+
+---
+
+## Risk Assessment
+
+### High Risk
+- **QueueProcessor Import Timing**: If import order matters, this could affect other server initialization
+- **Service Worker Cache**: Existing service worker cache may interfere with new registration
+
+### Medium Risk
+- **Database State Corruption**: Existing 'pending' items may need manual cleanup
+- **SSE Connection Limits**: Browser limits on concurrent EventSource connections
+
+### Low Risk
+- **Test Environment Differences**: Tests may behave differently than production
+- **TypeScript Compilation**: Service worker TS compilation edge cases
+
+### Mitigation Strategies
+- **Import Safety**: Add import after other critical server setup
+- **Cache Clearing**: Provide instructions for clearing service worker cache
+- **Data Migration**: Add cleanup script for stuck 'pending' items
+- **Connection Management**: Implement proper connection cleanup on page unload
+
+---
+
+## Success Metrics
+
+### Must Have ✅
+1. **Queue Processing**: Items automatically progress from 'pending' to completion
+2. **API Status Codes**: All endpoints return correct HTTP status codes (400/404/409)
+3. **Service Worker**: Single service worker registration without conflicts
+4. **SSE Connection**: Real-time connection status display with live updates
+5. **Test Suite**: All 16 failing tests now pass
+
+### Should Have ✅
+6. **Performance**: Queue processing maintains current throughput (2 concurrent items)
+7. **Reliability**: SSE reconnects automatically after disconnections
+8. **PWA Functionality**: Installation, offline support, and push notifications work
+9. **Error Handling**: Descriptive error messages for all failure scenarios
+10. **Monitoring**: Health check endpoint for queue processor status
+
+### Nice to Have ✅
+11. **User Experience**: Clear visual indicators for all connection states
+12. **Development**: Enhanced logging for debugging and troubleshooting
+13. **Documentation**: Updated API documentation reflecting correct status codes
+14. **Cross-Browser**: Consistent behavior across all major browsers
+
+---
+
+## Deployment Considerations
+
+### Pre-Deployment
+- Clear existing service worker registrations in browser cache
+- Run full test suite to verify all fixes
+- Test with actual Instagram URLs in development
+
+### Deployment Process
+1. Deploy changes to staging environment
+2. Verify queue processing starts automatically
+3. Test service worker registration
+4. Validate SSE connection functionality
+5. Run smoke tests on all API endpoints
+
+### Post-Deployment
+- Monitor server logs for QueueProcessor startup confirmation
+- Check service worker registration in production
+- Verify real-time updates work for live users
+- Monitor error rates for API endpoints
+
+### Rollback Plan
+- Keep previous service worker configuration available
+- Maintain ability to disable QueueProcessor if needed
+- Have API error handling toggle if issues arise
+
+---
+
+## Checklist for Completion
+
+### Story 1: Queue Processor Startup ✅
+- [ ] QueueProcessor import added to hooks.server.ts
+- [ ] Startup logging confirms processor initialization  
+- [ ] Health check endpoint shows processor status
+- [ ] Queue items process automatically after application start
+- [ ] All queue processor integration tests pass
+
+### Story 2: API Error Handling ✅  
+- [ ] Error handling middleware implemented and tested
+- [ ] All API endpoints use proper error classification
+- [ ] Validation errors return 400 with descriptive messages
+- [ ] Not found errors return 404 responses
+- [ ] Conflict errors return 409 responses
+- [ ] All 12 failing API tests now pass
+
+### Story 3: Service Worker Registration ✅
+- [ ] SvelteKit service worker disabled in configuration
+- [ ] Single service worker registration visible in DevTools
+- [ ] Workbox precaching initializes without errors
+- [ ] Push notifications continue to function correctly
+- [ ] PWA installation prompt works as expected
+- [ ] No service worker evaluation errors in console
+
+### Story 4: SSE Connection Status ✅
+- [ ] Connection status indicator updates in real-time
+- [ ] Queue updates display immediately when processing occurs
+- [ ] Automatic reconnection works after temporary disconnections  
+- [ ] Keep-alive pings maintain stable connections
+- [ ] Error states provide clear user feedback
+- [ ] Connection functionality consistent across browser sessions
+
+### Final Validation ✅
+- [ ] All 16 previously failing tests now pass
+- [ ] Complete end-to-end queue flow works from submission to completion
+- [ ] Service worker installs and functions properly
+- [ ] Real-time updates work reliably across multiple browser tabs
+- [ ] No regression in existing functionality
+- [ ] Performance meets or exceeds current benchmarks
+
+---
+
+## References
+
+### Documentation
+- [Hexagonal Architecture Guidelines](.system/abstract_architecture.md)
+- [SvelteKit SSR Best Practices](SVELTEKIT_SSR_GUIDE.md)
+- [Queue System Design](AsyncInMemoryProcessingQueue.md)
+- [Service Worker Configuration](RefactorServiceWorkerForProperPWACompliance.md)
+
+### External Resources
+- [EventSource API Documentation](https://developer.mozilla.org/en-US/docs/Web/API/EventSource)
+- [Workbox Service Worker Guide](https://developers.google.com/web/tools/workbox)
+- [SvelteKit Hooks Documentation](https://kit.svelte.dev/docs/hooks)
+- [Vite PWA Plugin Documentation](https://vite-pwa-org.netlify.app/)