Add a Feed or Source

Quick path: Use the /add-source Claude Code skill for guided, automated source addition. It handles RSS discovery, selector inspection, YAML generation, and safe sync. See .claude/skills/add-source/SKILL.md.

Sources are defined in config/sources.yaml, seeded into PostgreSQL, and used by the scheduler and collectors. Runtime always reads from the database; YAML is only used for seeding and manual syncs.

Precedence rules

• Database values win at runtime.
• On startup, YAML seeding only runs if fewer than 5 sources exist (see Source Configuration).
• Manual sync: ./run_cli.sh sync-sources overwrites database entries with YAML values unless --no-remove is set.

Steps to add a source

1. Edit config/sources.yaml and add an entry with a unique id.

   YAML

   - id: "vmray_blog"
     name: "VMRay Blog"
     url: "https://www.vmray.com/blog/"
     rss_url: "https://www.vmray.com/blog/feed/"
     check_frequency: 1800
     active: true
     config:
       allow: ["vmray.com"]
       title_filter_keywords: ["webinar", "training", "careers"]
       rss_only: false
   

   Keep allow, post_url_regex, and title_filter_keywords consistent with existing entries to avoid scraping noise.

2. Sync YAML to the database. Use --new-only to insert new sources without overwriting existing DB configs (e.g. lookback_days, check_frequency):

   Bash

   ./run_cli.sh sync-sources --config config/sources.yaml --no-remove --new-only
   

   Without --new-only, sync will overwrite existing sources with YAML values. Use plain --no-remove only when you intentionally want to reset existing configs to YAML defaults.

3. Verify the source is active:

   Bash

   # quick breakdown via health endpoint
   curl -s http://localhost:8001/api/health/ingestion | jq '.ingestion.source_breakdown[] | {source_name, articles_count}'
   

   The Sources page in the UI also reflects the new entry and its enablement state.

Notes

• To reset the database to YAML defaults, rerun sync without --no-remove (overwrites DB rows not present in YAML).
• Set active: false to keep a source defined but disabled; collectors skip inactive sources at runtime.
• The Celery Beat periodic task check-all-sources-every-30min (fixed in src/worker/celery_app.py, not docker-compose.yml) sweeps all active sources every 30 minutes; it does not currently gate on each source's check_frequency. check_frequency (seconds, default 14400 / 4 hours) is still stored per source, exposed via the API/UI, and used by source scheduling helpers.
• If a new source starts failing repeatedly, check the Sources page in the UI for error details and manually update the source config as needed.

Last updated: 2026-07-05 Last reviewed: 2026-09-01