Get instant notifications for new articles, trending topics, and email digests with WebSocket-powered real-time updates

Real-Time Feed Monitoring & Alerts

Phase 3B Implementation - Get instant notifications for new articles, trending topics, and customizable email digests.

Overview

The real-time monitoring system provides:

Live Notifications: WebSocket-powered instant alerts for new articles
Trending Detection: Z-score analysis for identifying hot topics
Email Digests: Customizable daily/weekly digest subscriptions
Feed Follows: Subscribe to specific feeds for targeted notifications
Smart Bundling: Automatic notification grouping to prevent spam

Architecture

Components

Feed Poller (polling.py):
- Periodic feed fetching with retry logic
- Article deduplication via GUID
- Response time tracking
Notification Manager (notifications.py):
- Notification creation and bundling
- WebSocket broadcasting
- User preference filtering
Trending Detector (trending.py):
- Z-score statistical analysis
- Baseline calculation (mean/std dev)
- Representative article selection
Digest Manager (digests.py):
- HTML email generation
- Cron-based scheduling
- SMTP delivery
WebSocket Server (websocket_server.py):
- Socket.IO real-time server
- User authentication and rooms
- Event broadcasting
Scheduler (scheduler.py):
- APScheduler background jobs
- 4 periodic tasks (polling, trending, digests, cleanup)

Getting Started

1. Start Monitoring Server

# Start backend monitoring (WebSocket + scheduler)
uv run aiwebfeeds monitor start

# Output:
# ✓ Background scheduler started
# ✓ WebSocket server started on port 8000
#
# Scheduled Jobs:
#   poll_feeds       | Every 15 min   | Poll all active feeds
#   detect_trending  | Every 1 hour   | Z-score trend detection
#   send_digests     | Every minute   | Check for due email digests
#   cleanup_notifications | Daily 3:00 AM | Delete old notifications

2. Follow Feeds

# Get your user ID from browser localStorage
# (automatically generated on first visit)

# Follow a feed to receive notifications
uv run aiwebfeeds monitor follow <user-id> <feed-id>

# Example:
uv run aiwebfeeds monitor follow a1b2c3d4-... ai-news

# List your follows
uv run aiwebfeeds monitor list-follows <user-id>

# Unfollow
uv run aiwebfeeds monitor unfollow <user-id> <feed-id>

3. Frontend Integration

import { useState } from "react";
import { NotificationBell, NotificationCenter, FollowButton, TrendingTopics } from "@/components/notifications";

export default function Page() {
  const [showNotifications, setShowNotifications] = useState(false);

  return (
    <div>
      {/* Header with notification bell */}
      <header className="flex items-center justify-between">
        <h1>AI Web Feeds</h1>
        <NotificationBell onOpenCenter={() => setShowNotifications(true)} />
      </header>

      {/* Notification panel */}
      <NotificationCenter isOpen={showNotifications} onClose={() => setShowNotifications(false)} />

      {/* Feed page */}
      <div className="grid grid-cols-3 gap-6">
        <div className="col-span-2">
          <h2>AI News Feed</h2>

          {/* Follow button */}
          <FollowButton feedId="ai-news" variant="compact" />

          {/* Articles... */}
        </div>

        {/* Sidebar */}
        <aside>
          <TrendingTopics limit={5} />
        </aside>
      </div>
    </div>
  );
}

Features

Real-Time Notifications

Instant WebSocket alerts for:

New Articles: Individual notifications for each new article (below bundle threshold)
Bundled Updates: Single notification for multiple articles (>3 in 5 minutes)
Trending Topics: Alerts when topics exceed Z-score threshold (>2.0)
System Alerts: Important system messages

Notification Types:

type NotificationType =
  | "new_article" // Single new article
  | "trending_topic" // Hot topic alert
  | "feed_updated" // Multiple articles (bundled)
  | "system_alert"; // System message

Notification Bundling

Prevents notification spam with smart bundling:

IF articles_count >= threshold (default: 3)
AND within_window (default: 5 minutes)
THEN send_bundled_notification()
ELSE send_individual_notifications()

Configuration (.env):

AIWF_NOTIFICATION_BUNDLE_THRESHOLD=3
AIWF_NOTIFICATION_BUNDLE_WINDOW_SECONDS=300

Z-score statistical analysis:

Algorithm:

Baseline Calculation: Mean & StdDev of article counts over N days (default: 3)
Current Period: Article counts in last 1 hour
Z-Score: (current - baseline_mean) / baseline_std
Threshold: Alert if Z-score > 2.0 AND articles > 5

Formula:

Z = \frac{X - \mu}{\sigma}

Where:

$X$ = Current article count
$\mu$ = Baseline mean
$\sigma$ = Baseline standard deviation

Configuration:

AIWF_TRENDING_BASELINE_DAYS=3
AIWF_TRENDING_Z_SCORE_THRESHOLD=2.0
AIWF_TRENDING_MIN_ARTICLES=5
AIWF_TRENDING_UPDATE_INTERVAL_HOURS=1

Email Digests

Customizable email summaries:

Schedule Types:

Daily: Every day at 9:00 AM
Weekly: Every Monday at 9:00 AM
Custom: Cron expression (e.g., 0 9 * * *)

Configuration:

# SMTP Settings
AIWF_SMTP_HOST=localhost
AIWF_SMTP_PORT=25
AIWF_SMTP_USER=
AIWF_SMTP_PASSWORD=
AIWF_SMTP_FROM=noreply@aiwebfeeds.com

# Digest Settings
AIWF_DIGEST_MAX_ARTICLES=20

API Usage:

// Subscribe to daily digest
await fetch("/api/digests", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    user_id: "user-uuid",
    email: "user@example.com",
    schedule_type: "daily",
    schedule_cron: "0 9 * * *",
    timezone: "America/New_York",
  }),
});

Feed Follows

User-feed relationships for notification targeting:

// Follow a feed
await fetch("/api/follows", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    user_id: "user-uuid",
    feed_id: "ai-news",
  }),
});

// Get followed feeds
const response = await fetch(`/api/follows?user_id=${userId}`);
const { follows } = await response.json();

API Reference

REST Endpoints

Notifications

GET /api/notifications

List user notifications.

// Query params
?user_id=<uuid>&unread_only=true&limit=50

// Response
{
  "user_id": "...",
  "notifications": [...],
  "count": 10
}

PATCH /api/notifications/:id

Mark notification as read or dismissed.

// Body
{
  "action": "mark_read" | "dismiss"
}

Follows

GET /api/follows

List followed feeds.

?user_id=<uuid>

POST /api/follows

Follow a feed.

{
  "user_id": "...",
  "feed_id": "..."
}

DELETE /api/follows

Unfollow a feed.

?user_id=<uuid>&feed_id=<id>

GET /api/trending

Get current trending topics.

?limit=10

// Response
{
  "trending": [
    {
      "topic_id": "artificial-intelligence",
      "z_score": 2.5,
      "article_count": 50,
      "rank": 1
    }
  ]
}

Preferences

GET /api/preferences

Get notification preferences.

POST /api/preferences

Set notification preferences.

{
  "user_id": "...",
  "feed_id": "..." | null,  // null for global
  "delivery_method": "websocket" | "email" | "in_app",
  "frequency": "instant" | "hourly" | "daily" | "weekly" | "off",
  "quiet_hours_start": "22:00",
  "quiet_hours_end": "08:00"
}

Digests

GET /api/digests

Get digest subscriptions.

POST /api/digests

Create digest subscription.

DELETE /api/digests

Unsubscribe from digests.

WebSocket Protocol

Connection:

import { io } from "socket.io-client";

const socket = io("http://localhost:8000");

// Authenticate
socket.emit("authenticate", { user_id: "user-uuid" });

Events:

// Incoming
socket.on("notification", (data: Notification) => {
  console.log("New notification:", data);
});

socket.on("trending_alert", (data: TrendingAlert) => {
  console.log("Trending:", data.topic_id);
});

socket.on("notifications_history", (data: { notifications: Notification[] }) => {
  console.log("History:", data.notifications);
});

// Outgoing
socket.emit("mark_read", { notification_id: 123 });
socket.emit("dismiss", { notification_id: 123 });

Configuration

Environment Variables

# WebSocket Server
AIWF_WEBSOCKET_PORT=8000
AIWF_WEBSOCKET_CORS_ORIGINS=http://localhost:3000,https://aiwebfeeds.com
NEXT_PUBLIC_WEBSOCKET_URL=http://localhost:8000

# Feed Polling
AIWF_FEED_POLL_INTERVAL_MIN=15
AIWF_FEED_POLL_TIMEOUT=30
AIWF_FEED_POLL_MAX_CONCURRENT=10

# Notifications
AIWF_NOTIFICATION_RETENTION_DAYS=7
AIWF_NOTIFICATION_BUNDLE_THRESHOLD=3
AIWF_NOTIFICATION_BUNDLE_WINDOW_SECONDS=300

# Trending Detection
AIWF_TRENDING_BASELINE_DAYS=3
AIWF_TRENDING_Z_SCORE_THRESHOLD=2.0
AIWF_TRENDING_MIN_ARTICLES=5
AIWF_TRENDING_UPDATE_INTERVAL_HOURS=1

# Email Digests
AIWF_SMTP_HOST=localhost
AIWF_SMTP_PORT=25
AIWF_SMTP_USER=
AIWF_SMTP_PASSWORD=
AIWF_SMTP_FROM=noreply@aiwebfeeds.com
AIWF_DIGEST_MAX_ARTICLES=20

Database Schema

7 New Tables:

feed_entries - Article metadata from polling
feed_poll_jobs - Polling job tracking
notifications - User notifications
user_feed_follows - Feed follow relationships
trending_topics - Trending topic calculations
notification_preferences - User preferences
email_digests - Digest subscriptions

See Database Architecture for full schema.

Components

NotificationBell

Bell icon with unread count badge.

<NotificationBell onOpenCenter={() => setShowCenter(true)} className="..." />

Props:

onOpenCenter: Callback when bell is clicked
className: Additional CSS classes

NotificationCenter

Slide-in notification panel.

<NotificationCenter isOpen={isOpen} onClose={() => setIsOpen(false)} className="..." />

Features:

All/Unread filter tabs
Mark read, dismiss, view actions
Time-ago relative timestamps
Type-specific icons and colors

FollowButton

Toggle feed follow status.

<FollowButton
  feedId="ai-news"
  variant="default" | "compact"
  initialFollowing={false}
  onFollowChange={(following) => console.log(following)}
/>

Variants:

default: Full button with icons and text
compact: Small button for inline use

TrendingTopics

Display top trending topics.

<TrendingTopics limit={5} className="..." />

CLI Commands

Monitor

Start server:

uv run aiwebfeeds monitor start [--port 8000]

Check status:

uv run aiwebfeeds monitor status

Stop server:

# Use Ctrl+C to stop

Follows

Follow a feed:

uv run aiwebfeeds monitor follow <user-id> <feed-id>

Unfollow a feed:

uv run aiwebfeeds monitor unfollow <user-id> <feed-id>

List follows:

uv run aiwebfeeds monitor list-follows <user-id>

Testing

Run the test suite:

cd tests
uv run pytest tests/packages/test_polling.py -v
uv run pytest tests/packages/test_notifications.py -v
uv run pytest tests/packages/test_scheduler.py -v

Coverage:

uv run pytest --cov=ai_web_feeds --cov-report=html

Troubleshooting

WebSocket Connection Issues

Problem: Frontend can't connect to WebSocket server.

Solution:

Check server is running: uv run aiwebfeeds monitor status
Verify CORS origins in .env: AIWF_WEBSOCKET_CORS_ORIGINS
Check browser console for connection errors
Ensure NEXT_PUBLIC_WEBSOCKET_URL matches server URL

No Notifications Received

Problem: Following feeds but not getting notifications.

Solution:

Verify feed is being polled: Check scheduler status
Confirm follow relationship: aiwebfeeds monitor list-follows <user-id>
Check notification creation: Query database for recent notifications
Verify WebSocket authentication: Browser should emit authenticate event

Problem: Trending topics list is stale.

Solution:

Check scheduler: aiwebfeeds monitor status
Verify trending job next run time
Ensure sufficient baseline data (>= 3 days of articles)
Check Z-score threshold configuration

Email Digests Not Sending

Problem: Digest subscriptions created but emails not delivered.

Solution:

Verify SMTP configuration in .env
Check digest schedule (cron expression)
Test SMTP connection manually
Check scheduler logs for digest job errors

Performance

Optimization Tips

Feed Polling:
- Adjust interval based on feed update frequency
- Use AIWF_FEED_POLL_MAX_CONCURRENT to limit concurrent requests
- Monitor response times in feed_poll_jobs table
WebSocket:
- Enable connection pooling for high traffic
- Use Redis adapter for multi-server deployments
- Monitor active connections
Database:
- Enable WAL mode for SQLite (default)
- Add indexes on frequently queried columns
- Cleanup old notifications regularly
Trending Detection:
- Cache baseline calculations
- Reduce baseline period for faster computation
- Limit to top N topics

Security

User Identity

Anonymous identification via localStorage UUID
No authentication required (Phase 3B MVP)
Migration path to user accounts planned (Phase 3A)

WebSocket

CORS validation on connection
User ID authentication required
Room-based message targeting

API

Rate limiting (planned)
Input validation on all endpoints
SQL injection prevention via SQLModel

Roadmap

Phase 3A: User Accounts

Email/password authentication
User profile management
Account migration for existing localStorage users

Phase 3C: Community Curation

Feed ratings and reviews
User-submitted feeds
Collaborative filtering

Phase 3D: Advanced AI

Content summarization
Sentiment analysis
Smart article recommendations

Resources

Implemented: October 2025 · Version: Phase 3B

Real-Time Feed Monitoring

On this page