Backup And Restore¶
Backup and Restore System¶
Comprehensive guide for Huntable CTI Studio backup and restore operations, including database-only backups, full system backups, and automated backup configuration.
Table of Contents¶
- Quick Start
- System Overview
- Database-Only Backups
- Full System Backups
- Automated Backups
- Restore Procedures
- Disaster Recovery
- Retention & Pruning
- Monitoring & Troubleshooting
- Best Practices
- Migration from Database-Only to Full System
- Cross-Machine Backup Restore
Quick Start¶
Create Backup¶
# Full system backup (recommended)
./scripts/backup_restore.sh create
# Database-only backup
./scripts/backup_restore.sh db-create
Restore System¶
# Restore full system
./scripts/backup_restore.sh restore system_backup_20251010_103000
# Restore database only
./scripts/backup_restore.sh db-restore cti_scraper_backup_20250907_134653.sql.gz
List Backups¶
# List all backups
./scripts/backup_restore.sh list
# List database-only backups
./scripts/backup_restore.sh db-list
System Overview¶
Backup Types¶
| Type | Components | Use Case | Speed |
|---|---|---|---|
| Database-Only | PostgreSQL dump | Quick DB snapshot | Fast (~1-2 min) |
| Full System | Database + Models + Config + Outputs + Logs | Complete recovery | Slow (~5-10 min) |
Components Backed Up¶
Database-Only Backup¶
- PostgreSQL database dump (compressed) - includes ALL tables
- ml_model_versions: ML model version history with evaluation metrics (accuracy, precision, recall, F1 scores)
- agentic_workflow_config: Agent workflow configuration (thresholds, agent models, prompts)
- agent_prompt_versions: Agent prompt version history
- app_settings: Application settings (user preferences, API keys, LLM configurations)
- sources: Source configurations (enabled/disabled status, lookback_days, check_frequency, RSS URLs, config JSON)
- source_checks: Source check history and health metrics
- All other application tables
- Metadata JSON file (includes ml_model_versions count for verification)
Full System Backup¶
- Database - PostgreSQL dump with metadata
- ML Models - Trained models (
models/directory) - Configuration - Source configs and settings (
config/directory) config/sources.yaml- Source definitions (⚠️ Can overwrite database on sync)- Other configuration files
- Training Data - User feedback and training datasets (
outputs/directory) - Logs - Application logs (
logs/directory)
Docker volumes are not backed up
Full system backups intentionally do not archive Docker volumes. PostgreSQL state is captured by the SQL dump, which is the supported database backup and restore mechanism. Redis only stores ephemeral queue state and is not restored from backup.
✅ Source Configuration: Database values are the source of truth. config/sources.yaml is only used for brand new builds (< 5 sources). After restore, database settings are automatically preserved. See Source Configuration Precedence for details.
Backup Structure¶
Database-Only:
backups/
├── cti_scraper_backup_20250907_134653.sql.gz
└── cti_scraper_backup_20250907_134653.json
Full System:
backups/system_backup_20251010_103000/
├── database.sql.gz
├── metadata.json
├── models/
├── config/
├── outputs/
└── logs/
Database-Only Backups¶
Features¶
- Automatic Compression: gzip compression (~70-80% reduction)
- Metadata Tracking: JSON metadata with database stats (including ml_model_versions count)
- Timestamped Filenames:
cti_scraper_backup_YYYYMMDD_HHMMSS.sql.gz - Docker Integration: Verifies the
cti_postgrescontainer is running and runspg_dump/psqlagainst it directly - Safety Checks: Pre-restore validation and snapshots
- Complete Table Coverage: All tables backed up, including ml_model_versions for ML metric history
- Restore Verification: Automatically verifies ml_model_versions count matches backup metadata
- pgvector Extension: Automatically enables pgvector extension during restore (required for Sigma similarity search with vector embeddings)
Commands¶
Create Backup¶
# Database-only backup (recommended)
./scripts/backup_restore.sh db-create
# Via CLI
./run_cli.sh backup create
Backup directory
The backup directory is set via the BACKUP_DIR environment variable in .env. Default: ./backups/.
List Backups¶
Restore Database¶
# Basic restore
./scripts/backup_restore.sh db-restore cti_scraper_backup_20250907_134653.sql.gz
# Force restore without confirmation
python3 scripts/restore_database.py backup.sql.gz --force
# Skip pre-restore snapshot
python3 scripts/restore_database.py backup.sql.gz --no-snapshot
--force skips only the interactive confirmation. It still creates the pre-restore
snapshot; --no-snapshot is the explicit opt-out for both command-line and uploaded restores.
Metadata Format¶
{
"database_size": "15.2 MB",
"table_stats": "table_name | inserts | updates | deletes",
"ml_model_versions_count": "5",
"backup_timestamp": "2025-09-07T13:46:53.394736",
"postgres_version": "15-alpine"
}
Note: The ml_model_versions_count field is used during restore verification to ensure ML model metric history is correctly restored.
Full System Backups¶
Architecture¶
Full system backups provide complete system recovery capability, backing up all critical components for disaster recovery.
Metadata Format¶
{
"timestamp": "2025-10-10T10:30:00",
"version": "2.0",
"backup_name": "system_backup_20251010_103000",
"components": {
"database": {
"filename": "database_20251010_103000.sql.gz",
"size_mb": 45.2,
"checksum": "sha256..."
},
"models": {
"files": 1,
"size_mb": 0.5
},
"config": {
"files": 1,
"size_mb": 0.04
}
},
"total_size_mb": 93.04
}
Commands¶
Create Backup¶
# Full system backup (default)
./scripts/backup_restore.sh create
# Database-only
./scripts/backup_restore.sh create --type database
# Compatibility option: accepted, but currently follows the full-system path and still includes the database
./scripts/backup_restore.sh create --type files
# Custom options
./scripts/backup_restore.sh create --no-compress --no-verify
List Backups¶
# List all backups
./scripts/backup_restore.sh list
# Show detailed information
./scripts/backup_restore.sh list --show-details
Verify Backup¶
# Basic verification
./scripts/backup_restore.sh verify system_backup_20251010_103000
# With database restore test
./scripts/backup_restore.sh verify system_backup_20251010_103000 --test-restore
Verification fails (overall_valid: false) when the backup has no metadata.json,
when the metadata is unparseable, or when the database dump is missing, empty, or
does not match its recorded SHA-256 checksum. A backup whose pg_dump step failed
records only errors in its database component and is rejected on that basis.
Missing non-database components are reported as warnings, not failures.
Restore System¶
# Full system restore
./scripts/backup_restore.sh restore system_backup_20251010_103000
# Selective restore
./scripts/backup_restore.sh restore system_backup_20251010_103000 --components database,models
# Dry run
./scripts/backup_restore.sh restore system_backup_20251010_103000 --dry-run
CLI Integration¶
# Create backup
python3 -m src.cli.main backup create
# List backups
python3 -m src.cli.main backup list
# Restore system
python3 -m src.cli.main backup restore system_backup_20251010_103000
# Verify backup
python3 -m src.cli.main backup verify system_backup_20251010_103000
# Show statistics
python3 -m src.cli.main backup stats
Automated Backups¶
Default Configuration¶
- Daily Backup: 2:00 AM (configurable)
- Weekly Cleanup: 3:00 AM on Sundays (configurable)
- Retention Policy: 7 daily + 4 weekly + 3 monthly backups
- Max Size: 50 GB total backup storage
- Backup Type: Full system backup
Setup Methods¶
Automated Setup (Recommended)¶
# Full setup with automated backups
./setup.sh
# Setup without automated backups
./setup.sh --no-backups
# Setup with custom backup time
./setup.sh --backup-time 1:30
Manual Setup¶
# Setup with default settings
./scripts/setup_automated_backups.sh
# Setup with custom settings
./scripts/setup_automated_backups.sh --backup-time 1:30 --daily 10 --weekly 5
# Setup with custom retention policy
./scripts/setup_automated_backups.sh --daily 7 --weekly 4 --monthly 3 --max-size-gb 100
Configuration Options¶
Backup Timing¶
# Custom backup time (24-hour format)
./scripts/setup_automated_backups.sh --backup-time 1:30
# Custom cleanup time
./scripts/setup_automated_backups.sh --cleanup-time 4:00
Retention Policy¶
# Custom retention policy
./scripts/setup_automated_backups.sh --daily 10 --weekly 5 --monthly 2
# Size-based retention
./scripts/setup_automated_backups.sh --max-size-gb 100
Backup Directory¶
Management Commands¶
Check Status¶
This shows: - Whether automated backups are configured - Recent backup activity - Backup statistics and size - Recent log entries
Uninstall¶
Cron Job Details¶
The automated backup system creates two cron jobs:
Daily Backup Job:
# Runs daily at 2:00 AM
0 2 * * * cd /path/to/Huntable-CTI-Studio && ./scripts/backup_restore.sh create --type full >> logs/backup.log 2>&1
Weekly Cleanup Job:
# Runs weekly on Sundays at 3:00 AM
0 3 * * 0 cd /path/to/Huntable-CTI-Studio && ./scripts/backup_restore.sh prune --daily 7 --weekly 4 --monthly 3 --max-size-gb 50 --force >> logs/backup.log 2>&1
Monitoring Logs¶
Restore Procedures¶
Database-Only Restore¶
Safety Features¶
- Validates backup file format before restore
- Creates pre-restore snapshots by default
- Confirms restore operation (unless
--forceis used) - Verifies database connection after restore
Restore Steps¶
# 1. List available backups
./scripts/backup_restore.sh db-list
# 2. Restore from specific backup
# Note: pgvector extension is automatically enabled during restore for Sigma similarity search
./scripts/backup_restore.sh db-restore cti_scraper_backup_20250907_134653.sql.gz
# 3. Verify database
docker exec -it cti_postgres psql -U cti_user -d cti_scraper -c "SELECT COUNT(*) FROM articles;"
# 4. Verify pgvector extension (for Sigma features)
docker exec -it cti_postgres psql -U cti_user -d cti_scraper -c "SELECT * FROM pg_extension WHERE extname = 'vector';"
Restore from Snapshot¶
If a restore fails, restore from pre-restore snapshot:
# List snapshot files
ls backups/pre_restore_snapshot_*.sql
# Restore from snapshot
python3 scripts/restore_database.py backups/pre_restore_snapshot_20250907_140201.sql --force
Legacy Restore Scripts¶
Two legacy database restore scripts exist alongside the primary restore_database.py (v1) used by the UI and backup_restore.sh:
-
restore_database_v2.py: Used by the Settings UI's restore-from-file upload endpoint (POST /api/backup/restore-from-file). Offers the same safety features as v1 (pre-restore snapshot, rollback, app-container pausing,extract_psql_errorsguard) in a class-based implementation. Currently live in that one path. -
restore_database_v3.py: NOT wired into any user-facing path. Uses directpsqlwith--single-transactionplusON_ERROR_STOP=on(no Docker shell wrapper, atomic restore, reliable exit codes). Its sole unique value is a post-restorecheck_source_attribution_integrity()check that warns if the restored database's article canonical_url / source-url mismatch count exceeds a known baseline. See the file header for a full gap analysis. If consolidation is ever pursued, extract that check into_restore_common.pyand remove v3 entirely.
Full System Restore¶
Selective Component Restore¶
# Database only
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components database
# ML models only
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components models
# Config files only
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components config
# Multiple components
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components database,models,config
Verify After Restore¶
# Verify database
docker exec -it cti_postgres psql -U cti_user -d cti_scraper -c "SELECT COUNT(*) FROM articles;"
# Verify models
ls -la models/
python3 -c "import pickle; pickle.load(open('models/content_filter.pkl', 'rb'))"
# Verify config
cat config/sources.yaml
# Verify containers
docker compose ps
Disaster Recovery¶
RTO/RPO Targets¶
- Recovery Time Objective (RTO): < 30 minutes
- Recovery Point Objective (RPO): < 24 hours (daily backups)
Disaster Recovery Playbook¶
1. Full System Recovery¶
# 1. Stop all services
docker compose down
# 2. Restore from latest backup
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS
# 3. Start services
docker compose up -d
# 4. Verify system health
./scripts/backup_restore.sh verify system_backup_YYYYMMDD_HHMMSS
2. Database-Only Recovery¶
# 1. Restore database component only
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components database
# 2. Verify database
docker exec -it cti_postgres psql -U cti_user -d cti_scraper -c "SELECT COUNT(*) FROM articles;"
3. ML Model Recovery¶
# 1. Restore models component only
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components models
# 2. Verify models
ls -la models/
python3 -c "
from src.utils.content_filter import ContentFilter
cf = ContentFilter()
print('Model loaded' if cf.load_model() else 'Model missing')
"
If
models/is empty after restore: the backup was taken from an instance where the model file was also absent (e.g. a previous clean install). Theml_model_versionstable will contain version history, but the pkl artifact is gone. Seed a new baseline model from the bundled fixtures:Then annotate articles and retrain via MLOps → Retrain to incorporate your data. Version history from before the restore is preserved in the database and will appear in the ML performance charts.
Model rollback and the
backups/models/fallback: thePOST /api/model/rollback/{version_id}endpoint (MLModelVersionManager._resolve_artifact_path) checksmodels/first, then automatically falls back tobackups/models/<filename>if the primary pkl has been deleted. This means rollback returns 200 (not 422) even whenmodels/has been wiped, as long as the artifact was mirrored to the backup location. Full system backups copy model files intobackups/system_backup_.../models/; thebackups/models/mirror is populated separately by the training pipeline.
Recovery Testing¶
Pre-Production Testing¶
# 1. Create test backup
./scripts/backup_restore.sh create --type full
# 2. Verify backup integrity
./scripts/backup_restore.sh verify system_backup_YYYYMMDD_HHMMSS --test-restore
# 3. Test selective restore
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --components database --dry-run
Production Validation¶
- Monthly full restore test: Restore to isolated environment
- Quarterly disaster recovery drill: Simulate complete system failure
Retention & Pruning¶
Default Retention Policy¶
- Daily: Keep last 7 daily backups
- Weekly: Keep last 4 weekly backups
- Monthly: Keep last 3 monthly backups
- Size Limit: Maximum 50 GB total backup size
The size limit never deletes the newest backup. If a single backup exceeds
--max-size-gb, pruning keeps it and prints a warning rather than emptying the
retention set. Pruning runs unattended with --force, and being over the limit
is recoverable while having zero backups is not.
Pruning Commands¶
# Show what would be pruned (dry run)
./scripts/backup_restore.sh prune --dry-run
# Custom retention policy
./scripts/backup_restore.sh prune --daily 5 --weekly 2 --monthly 1 --max-size-gb 30
# Force prune without confirmation
./scripts/backup_restore.sh prune --force
Manual Cleanup¶
# List old backups
./scripts/backup_restore.sh list
# Remove specific backup
rm -rf backups/system_backup_20251001_103000
# Remove database-only backup
rm -f backups/cti_scraper_backup_20250901_*.{sql.gz,json}
Monitoring & Troubleshooting¶
Health Checks¶
# Check backup status
./scripts/backup_restore.sh list
# Show backup statistics (not a backup_restore.sh subcommand; use the CLI)
python3 -m src.cli.main backup stats
# Verify latest backup
LATEST_BACKUP=$(./scripts/backup_restore.sh list | grep "system_backup_" | head -1 | awk '{print $2}')
./scripts/backup_restore.sh verify $LATEST_BACKUP
Common Issues¶
1. Container Not Running¶
Error: PostgreSQL container 'cti_postgres' is not running!
Solution:
2. Permission Denied¶
Error: Permission denied
Solution:
3. Disk Space Issues¶
Error: No space left on device
Solution:
# Check disk space
df -h
# Clean up old backups
./scripts/backup_restore.sh prune --force
# Remove Docker resources
docker system prune -a
4. Backup Creation Fails¶
Solutions:
- Check Docker containers: docker compose ps
- Verify disk space: df -h
- Check file permissions: ls -la scripts/
- Review logs: tail -50 logs/backup.log
5. Restore Fails¶
Solutions: - Verify backup integrity first - Check Docker container status - Ensure sufficient disk space - Try selective component restore
If restore verification fails with could not resize shared memory segment, raise the Postgres container shared-memory allocation and recreate the container:
POSTGRES_SHM_SIZE=2g docker compose up -d --force-recreate postgres
./scripts/backup_restore.sh verify system_backup_YYYYMMDD_HHMMSS --test-restore
6. Checksum Validation Fails¶
Solutions: - Recreate backup from source - Check for file corruption - Verify backup storage integrity
7. Cron Jobs Not Running¶
Solutions:
- Check cron service: systemctl status cron
- Verify cron jobs: crontab -l
- Check cron logs: journalctl -u cron
Debug Commands¶
# Check Docker containers
docker ps
# Check Docker volumes
docker volume ls
# Check disk space
df -h
# Check backup directory
ls -la backups/
# Check cron jobs
crontab -l
# View backup logs
tail -50 logs/backup.log
# Test backup manually
./scripts/backup_restore.sh create --type full
Log Monitoring¶
# View recent backup logs
tail -f logs/backup.log
# Search for errors
grep -i error logs/backup.log
# Search for specific backup
grep "system_backup_20251010" logs/backup.log
Best Practices¶
Regular Backups¶
- Create backups before major updates
- Schedule regular automated backups (daily recommended)
- Keep multiple backup versions
- Test restore procedures regularly
Backup Management¶
- Monitor backup directory size
- Clean up old backups periodically with retention policies
- Verify backup integrity regularly
- Store backups in secure locations
Safety Measures¶
- Always create snapshots before restore
- Test restore in non-production environment first
- Use
--dry-runfor testing restore operations - Keep legacy backups for rollback options
Security¶
- Backup files contain sensitive data
- Use appropriate file permissions (600 or 640)
- Consider encryption for long-term storage
- Implement secure backup transfer methods
Performance¶
- Schedule backups during low-usage periods (default: 2:00 AM)
- Use compression for large backups (enabled by default)
- Monitor backup duration and optimize as needed
- Consider incremental backups for very large datasets
Regular Maintenance Tasks¶
Weekly: - Check backup status and logs - Verify backup integrity - Monitor storage usage
Monthly: - Test restore procedures - Review retention policies - Update documentation
Quarterly: - Disaster recovery drill - Performance optimization review - Security audit
Migration from Database-Only to Full System¶
Why Migrate?¶
Full system backups provide: - Complete system recovery (Database + ML models + config + volumes) - Better organization with metadata - Enhanced verification with checksum validation - Selective component restore - Automated retention management
Migration Steps¶
# 1. Create first comprehensive backup
./scripts/backup_restore.sh create
# 2. Verify backup
./scripts/backup_restore.sh verify system_backup_YYYYMMDD_HHMMSS
# 3. Test restore
./scripts/backup_restore.sh restore system_backup_YYYYMMDD_HHMMSS --dry-run
# 4. Setup automated backups
./scripts/setup_automated_backups.sh
Legacy Command Mapping¶
| Legacy Command | New Command | Notes |
|---|---|---|
./scripts/backup_restore.sh create |
./scripts/backup_restore.sh db-create |
Database-only backup |
./scripts/backup_restore.sh list |
./scripts/backup_restore.sh db-list |
Database-only backups |
./scripts/backup_restore.sh restore |
./scripts/backup_restore.sh db-restore |
Database-only restore |
Cross-Machine Backup Restore¶
Overview¶
Backups created on one computer can be restored on another computer. The backup format is portable and doesn't contain machine-specific paths.
Prerequisites¶
-
Docker containers running on the target machine:
-
Backup file/directory copied to the target machine
-
Docker Compose stack available on the target machine
Restore Methods¶
Method 1: Copy Backup to Default Location¶
-
Copy backup directory to the
backups/folder: -
Restore using backup name:
Method 2: Restore from Absolute Path¶
If the backup is in a different location:
Method 3: Via Web UI¶
- Copy backup directory to
backups/folder - Open Settings → Backup Actions
- Click "Restore from Backup"
- Select the backup from the list
What Gets Restored¶
Portable Components (Cross-Machine Safe)¶
✅ Database - SQL dump (fully portable)
✅ Models - ML model files (.pkl, .joblib, etc.)
✅ Config - Configuration files (YAML, JSON)
✅ Outputs - Training data and generated content
✅ Logs - Application logs
Docker Volumes¶
Docker volumes are not restored by the system restore script. Use the database component for PostgreSQL recovery. Redis volume contents are treated as disposable queue/cache state.
Important Considerations¶
1. Database Credentials¶
The restore script uses default credentials. If your target machine has different credentials:
Option A: Set environment variables before restore:
export POSTGRES_PASSWORD=your_password
python3 scripts/restore_system.py system_backup_20260121_113825
Option B: Edit scripts/restore_system.py DB_CONFIG section temporarily
2. Docker Container Names¶
The restore script expects:
- cti_postgres (PostgreSQL container)
- cti_redis (Redis container)
If your containers have different names, update scripts/restore_system.py or use docker compose which sets these names automatically.
3. Selective Restore¶
You can restore only specific components:
# Restore database and models only
python3 scripts/restore_system.py system_backup_20260121_113825 \
--components database,models
# Restore file-backed components
python3 scripts/restore_system.py system_backup_20260121_113825 \
--components database,models,config,outputs,logs
4. Backup Format Compatibility¶
- Version 1.0 backups: ✅ Supported
- Version 2.0 backups: ✅ Supported
- Older formats: May require migration
Step-by-Step Example¶
Source Machine (Computer A)¶
# Create backup
cd /path/to/Huntable-CTI-Studio
python3 scripts/backup_system.py
# Backup created: backups/system_backup_20260121_113825/
Transfer Backup¶
# Option 1: SCP
scp -r backups/system_backup_20260121_113825 user@target:/path/to/Huntable-CTI-Studio/backups/
# Option 2: USB drive
# Copy entire backup directory to USB, then copy to target machine
# Option 3: Network share
# Copy to shared network location, then copy from target machine
Target Machine (Computer B)¶
# 1. Ensure Docker containers are running
cd /path/to/Huntable-CTI-Studio
docker compose up -d
# 2. Verify backup is in place
ls -la backups/system_backup_20260121_113825/
# 3. Restore (with snapshot for safety)
python3 scripts/restore_system.py system_backup_20260121_113825
# 4. Verify restore
# Check web UI or run verification
Troubleshooting¶
"Backup directory not found"¶
Solution: Use absolute path:
"PostgreSQL container not running"¶
Solution: Start containers:
"Database restore failed: connection refused"¶
Solution: Check container name and credentials:
"Docker volume restore is not supported"¶
Solution: Restore the database component from the SQL dump. Legacy backups may contain docker_volume_* metadata, but those components are skipped because Docker volume archives are not a supported restore mechanism.
Verification Failures¶
If verification fails for empty directories (outputs/logs), this is normal - empty directories are valid. The restore succeeded.
Best Practices¶
-
Test restore first: Use
--dry-runto verify backup structure: -
Create snapshot: Always create a snapshot before restore (default behavior):
-
Verify backup integrity: Check backup before transferring:
-
Selective restore: Restore components incrementally if needed:
Security Notes¶
- Backups may contain sensitive data (API keys, database passwords)
- Transfer backups securely (encrypted connection, secure storage)
- Verify backup source before restoring
- Consider encrypting backups for cross-machine transfer
Summary¶
✅ Cross-machine restore is fully supported
✅ Backups are portable (no machine-specific paths)
✅ Selective restore available (choose components)
✅ Web UI and CLI both support cross-machine restore
The only requirement is that Docker containers are running with the expected names on the target machine.
Last updated: 2026-08-13 Last reviewed: 2026-09-01