Configuration

Environment Variables

Note

All environment variables must be prefixed with the environment name:

Development: DEV_ prefix (e.g., DEV_DATABASE_URL)
Test: TEST_ prefix (e.g., TEST_DATABASE_URL)
Production: PROD_ prefix (e.g., PROD_DATABASE_URL)

Development Environment

# Monitoring (optional)
DEV_LOGFIRE_TOKEN=your_logfire_token_here
DEV_LOGFIRE_CONSOLE=false

# Notifications (optional)
DEV_SLACK_WEBHOOK_URL=your_slack_webhook_url_here

# Features
DEV_WATCHER_AUTO_CREATE_ANOMALY_DETECTION_RULES=false
DEV_PROFILING_ENABLED=true

Production Environment

# Database (Dev has Docker Compose)
PROD_DATABASE_URL=postgresql+asyncpg://user:password@prod-db:5432/watcher_prod

# Redis (Dev has Docker Compose)
PROD_REDIS_URL=redis://prod-redis:6379/1

# Monitoring
PROD_LOGFIRE_TOKEN=your_production_logfire_token
PROD_LOGFIRE_CONSOLE=false

# Notifications
PROD_SLACK_WEBHOOK_URL=your_production_slack_webhook

# Features
PROD_WATCHER_AUTO_CREATE_ANOMALY_DETECTION_RULES=true
PROD_PROFILING_ENABLED=false

Database Configuration

Connection Pool Default Settings

Pool Size 20 connections
Max Overflow 10 additional connections
Pool Timeout 30 seconds
Pool Recycle 3600 seconds

Redis Configuration

Connection Default Settings

Host Redis server hostname
Port Redis port (default: 6379)
Database Redis database number (default: 1)

Celery Default Configuration

Broker Redis URL for message broker
Backend Redis URL for result backend
Task Serializer JSON
Result Serializer JSON
Accept Content JSON
Time Zone UTC

Monitoring Configuration

Logfire Integration

Logfire provides comprehensive observability for your Watcher instance:

Application Metrics Request/response times, error rates
Database Queries Query performance and slow query detection
Background Tasks Celery worker execution logging and task tracking

Setup:

Create a Logfire account at https://logfire.pydantic.dev (Free tier is 10 million calls per month)
Get your token from the Logfire dashboard
Set the LOGFIRE_TOKEN environment variable
Restart your application

Slack Notifications

Configure Slack webhooks for real-time alerts:

Create a Slack app in your workspace
Add an Incoming Webhook to your app
Copy the webhook URL
Set the SLACK_WEBHOOK_URL environment variable

Alert Types:

Queue Monitoring: Celery queue depth alerts
Anomaly Detection: Statistical anomaly alerts
Timeliness: Pipeline execution timeliness alerts
Freshness: DML operation freshness alerts

Feature Flags

Auto-Create Anomaly Detection Rules

When enabled, Watcher automatically creates anomaly detection rules for new pipelines:

WATCHER_AUTO_CREATE_ANOMALY_DETECTION_RULES=true

Profiling

Enable application profiling for performance analysis. Enabling this allows you to profile any API endpoint by adding ?profile=true to the URL.

PROFILING_ENABLED=true