Getting Started

Complete guide to installing and using AI Web Feeds - from setup to your first analytics report

This guide will help you get started with AI Web Feeds, from installation to your first analytics report.

Prerequisites

Before you begin, ensure you have:

Python 3.13 or higher
pip (Python package manager)
Git (for cloning the repository)

Installation

Clone the Repository

git clone https://github.com/wyattowalsh/ai-web-feeds.git
cd ai-web-feeds

Choose Setup Method

Use the automated setup script:

chmod +x setup-enhanced-features.sh
./setup-enhanced-features.sh

This will:

Install the core library
Install the CLI
Create the data directory
Initialize the database

If you prefer manual installation:

# Install core library
cd packages/ai_web_feeds
pip install -e .

# Install CLI
cd ../../apps/cli
pip install -e .

# Create data directory and initialize database
cd ../..
mkdir -p data
python3 -c "
from ai_web_feeds.storage import DatabaseManager
db = DatabaseManager('sqlite:///data/aiwebfeeds.db')
db.create_db_and_tables()
"

Quick Start

1. Add Your First Feeds

The project comes with sample feeds in data/feeds.yaml. You can:

# The repository includes curated AI/ML feeds
cat data/feeds.yaml

Edit data/feeds.yaml:

sources:
  - id: "my-custom-feed"
    feed: "https://example.com/feed.xml"
    title: "My Custom Feed"
    source_type: "blog"
    topics: ["machine-learning"]
    tags: ["custom"]

2. Enrich Feed Metadata

Process feeds and extract metadata:

ai-web-feeds enrich all

This will:

Validate feed URLs
Discover feeds for sites without direct feed URLs
Detect feed formats
Generate quality scores

3. Fetch Feed Content

Download and analyze feed content:

# Fetch all feeds
ai-web-feeds fetch all

# Or fetch a specific feed
ai-web-feeds fetch one huggingface-blog --metadata

4. View Analytics

Explore your feed data:

# Overview dashboard
ai-web-feeds analytics overview

# View distributions
ai-web-feeds analytics distributions

# Check quality metrics
ai-web-feeds analytics quality

# See publishing trends
ai-web-feeds analytics trends --days 30

5. Check Feed Health

Monitor individual feed performance:

ai-web-feeds analytics health huggingface-blog

6. Generate Reports

Export comprehensive analytics:

# Generate JSON report
ai-web-feeds analytics report --output data/analytics-report.json

# Generate OPML for feed readers
ai-web-feeds opml generate --output data/feeds.opml

Common Workflows

Adding New Feeds

Find a feed URL - Most blogs have RSS/Atom feeds
Add to feeds.yaml:

- id: "new-feed-id"
  feed: "https://example.com/feed.xml"
  title: "Feed Title"
  source_type: "blog"
  topics: ["nlp", "computer-vision"]

Enrich and validate:

ai-web-feeds enrich one new-feed-id

Fetch content:

ai-web-feeds fetch one new-feed-id

Platform-Specific Feeds

For known platforms, you can use just the site URL:

# Reddit subreddit
- id: "reddit-machinelearning"
  site: "https://reddit.com/r/MachineLearning"
  discover: true
  platform_config:
    reddit:
      sort: "hot" # or "top", "new"

# YouTube channel
- id: "youtube-channel"
  site: "https://youtube.com/channel/UC123..."
  discover: true

# GitHub repository
- id: "github-repo"
  site: "https://github.com/owner/repo"
  discover: true
  platform_config:
    github:
      feed_type: "releases" # or "commits", "tags"

See the Platform Integrations guide for more details.

Monitoring Feed Health

Set up regular health checks:

# Check all feeds
ai-web-feeds fetch all --verified-only

# Review performance
ai-web-feeds analytics performance --days 7

# Check specific feed health
ai-web-feeds analytics health <feed-id>

Exporting Data

Export to different formats:

# OPML for feed readers
ai-web-feeds opml generate --output feeds.opml

# Categorized OPML
ai-web-feeds opml categorize --output feeds-categorized.opml

# JSON analytics report
ai-web-feeds analytics report --output analytics.json

Configuration

Database Configuration

By default, SQLite is used. To use a different database:

ai-web-feeds fetch all --database "postgresql://user:pass@localhost/dbname"

Or set an environment variable:

export AI_WEB_FEEDS_DB="postgresql://user:pass@localhost/dbname"

Fetch Configuration

Create a config file (e.g., .env):

AI_WEB_FEEDS_TIMEOUT=30
AI_WEB_FEEDS_USER_AGENT="MyApp/1.0"

Troubleshooting

Common Issues

Feed Not Found Error

# Verify feed exists in database
ai-web-feeds stats show

# Check feeds.yaml for typos
cat data/feeds.yaml | grep "id:"

Fetch Failures

# Check error details
ai-web-feeds analytics performance

# Test single feed with verbose output
ai-web-feeds fetch one <feed-id> --metadata

Database Issues

# Reinitialize database
rm data/aiwebfeeds.db
python3 -c "
from ai_web_feeds.storage import DatabaseManager
db = DatabaseManager('sqlite:///data/aiwebfeeds.db')
db.create_db_and_tables()
"

Next Steps

Explore Analytics - Try different analytics commands to understand your feed data
Set Up Automation - Create cron jobs for regular fetching
Customize Feeds - Add feeds relevant to your interests
Build Integrations - Use the Python API to integrate with other tools
Contribute - Share your curated feeds with the community

Example Session

Here's a complete example workflow:

# 1. Install
./setup-enhanced-features.sh

# 2. Check what's in the database
ai-web-feeds stats show

# 3. Enrich existing feeds
ai-web-feeds enrich all

# 4. Fetch content
ai-web-feeds fetch all --limit 10

# 5. View analytics
ai-web-feeds analytics overview
ai-web-feeds analytics distributions
ai-web-feeds analytics quality

# 6. Check a specific feed
ai-web-feeds analytics health huggingface-blog

# 7. Generate reports
ai-web-feeds analytics report --output report.json
ai-web-feeds opml generate --output feeds.opml

# 8. View trends
ai-web-feeds analytics trends --days 30

What's Next?

Now that you're set up, you can:

Add more feeds to your collection
Set up automated fetching (cron jobs)
Build custom analytics scripts
Integrate with other tools
Contribute back to the project

Resources

Feature Overview - Comprehensive feature list
CLI Reference - Command-line interface guide
Python API - Programmatic usage
Analytics Guide - Advanced analytics

Happy feed aggregating! 🚀

Getting Started

On this page