Skip to content

configuration.md

Latest commit

 

History

History
204 lines (142 loc) · 10.4 KB

configuration.md

File metadata and controls

204 lines (142 loc) · 10.4 KB
Raw

Configuration & API Reference

All configuration lives in SourceMonitor.configure—the install generator creates config/initializers/source_monitor.rb with sensible defaults and inline documentation. This guide expands on every namespace so you know which knobs to turn in production.

SourceMonitor.configure do |config|
  # customize settings here
end

Restart your application whenever you change these settings. The engine reloads model extensions automatically when the block runs, but background processes need a restart to pick up queue name or adapter changes.

Asset Bundling

  • SourceMonitor ships with npm-based bundling via cssbundling-rails and jsbundling-rails; follow .ai/engine-asset-configuration.md:11-113 when adjusting dependency versions or adding new entrypoints.
  • Keep bundled outputs under the namespaced directories (app/assets/builds/source_monitor, images/source_monitor, svgs/source_monitor) so host apps avoid collisions. See .ai/engine-asset-configuration.md:32-44 for the layout recommendations.
  • Use rbenv exec bundle exec rake app:source_monitor:assets:build (or npm run build) after tweaking Tailwind/Stimulus code to refresh the builds, and rely on test/dummy/bin/dev to watch build:css:watch + build:js:watch during development.
  • Sprockets hosts receive automatic precompile coverage; if a host prefers manifest-based precompilation, adapt the approach documented in .ai/engine-asset-configuration.md:114-143.

Queue & Worker Settings

  • config.queue_namespace – prefix applied to queue names ("source_monitor" by default)
  • config.fetch_queue_name / config.scrape_queue_name – base queue names before the host's ActiveJob.queue_name_prefix is applied
  • config.fetch_queue_concurrency / config.scrape_queue_concurrency – advisory values Solid Queue uses for per-queue limits
  • config.maintenance_queue_name – queue name for maintenance jobs ("source_monitor_maintenance" by default)
  • config.maintenance_queue_concurrency – advisory concurrency for the maintenance queue (default 1)
  • config.queue_name_for(:fetch | :scrape | :maintenance) – helper that respects the host's queue prefix

Use the helpers exposed on SourceMonitor:

SourceMonitor.queue_name(:fetch)         # => "source_monitor_fetch"
SourceMonitor.queue_name(:maintenance)   # => "source_monitor_maintenance"
SourceMonitor.queue_concurrency(:scrape) # => 2

Job Metrics & Mission Control

  • config.job_metrics_enabled – toggles Solid Queue metrics collection for the dashboard cards (default true)
  • config.mission_control_enabled – surfaces the Mission Control link on the dashboard when true
  • config.mission_control_dashboard_path – host route helper or callable returning the Mission Control path/URL; left blank by default

The helper SourceMonitor.mission_control_dashboard_path performs a routing check so the dashboard only renders links that resolve.

HTTP Client Settings

config.http maps directly onto Faraday's middleware options.

  • timeout – total request timeout in seconds (default 15)
  • open_timeout – connection open timeout in seconds (5)
  • max_redirects – maximum redirects to follow (5)
  • user_agent – defaults to Mozilla/5.0 (compatible; SourceMonitor/<version>) (browser-like to avoid bot-blocking)
  • proxy – hash or URL to configure proxy usage
  • headers – hash (or callables) merged into every request
  • retry_max, retry_interval, retry_interval_randomness, retry_backoff_factor, retry_statuses – mapped to faraday-retry

Fetching Behaviour

config.fetching controls adaptive scheduling.

  • min_interval_minutes / max_interval_minutes – enforce floor/ceiling for automatic schedule adjustments (defaults: 5 and 1440)
  • increase_factor / decrease_factor – multipliers when a source trends slow/fast
  • failure_increase_factor – multiplier applied on consecutive failures
  • jitter_percent – random jitter applied to next fetch time (0.1 = ±10%)
  • scheduler_batch_size – max sources picked up per scheduler run (default 25, was 100)
  • stale_timeout_minutes – minutes before a source stuck in "fetching" is reset (default 5, was 10)

Retention Defaults

config.retention sets global defaults that sources inherit when their fields are blank.

  • items_retention_days – prune items older than this many days (nil = retain forever)
  • max_items – keep only the most recent N items (nil = unlimited)
  • strategy – :destroy or :soft_delete (defaults to :destroy in the configuration class; the installer comments demonstrate :soft_delete)

The retention pruner runs after every successful fetch and inside nightly cleanup jobs.

Scraper Registry

Register adapters that inherit from SourceMonitor::Scrapers::Base:

config.scrapers.register(:readability, SourceMonitor::Scrapers::Readability)
config.scrapers.register(:custom, "MyApp::Scrapers::Premium" )

Adapters receive merged settings (default -> source -> invocation), and must return a SourceMonitor::Scrapers::Result object. Use config.scrapers.unregister(:custom) to remove overrides.

Scraping Settings

config.scraping controls scrape concurrency and recommendations.

  • max_in_flight_per_source – max concurrent scrape jobs per source (nil = unlimited, default nil)
  • max_bulk_batch_size – max items per bulk scrape enqueue (default 100)
  • scrape_recommendation_threshold – minimum average feed word count below which a source is recommended for scraping (default 200)

Events & Item Processors

Respond to lifecycle events without monkey patching:

config.events.after_item_created do |event|
  Analytics.track_new_item(event.item, source: event.source)
end

config.events.after_fetch_completed do |event|
  Rails.logger.info("Feed #{event.source.name} finished with #{event.status}")
end

config.events.register_item_processor ->(context) {
  SearchIndexer.index(context.item)
}

Event structs expose item, source, entry, result, status, and occurred_at. Item processors run after events and receive an ItemProcessorContext with the same shape.

Model Extensions

config.models lets host apps customise engine models at load time.

  • config.models.table_name_prefix – override the default sourcemon_ prefix
  • config.models.source.include_concern "MyApp::SourceMonitor::SourceExtensions" – mix in concerns before models load
  • config.models.source.validate :ensure_metadata_rules – register validations (blocks or symbols)
  • Equivalent hooks exist for items, fetch logs, scrape logs, and item content via config.models.item, etc.

The engine reloads model extensions whenever configuration runs so code reloading in development continues to work.

Realtime Settings

config.realtime governs Action Cable transport.

  • config.realtime.adapter – one of :solid_cable, :redis, or :async
  • config.realtime.redis_url – optional Redis URL when using the Redis adapter
  • config.realtime.solid_cable – yields options: polling_interval, message_retention, autotrim, use_skip_locked, trim_batch_size, connects_to (hash for multi-database setups), silence_polling

Call config.realtime.action_cable_config if you need a full hash for environment-specific cable.yml generation.

Authentication Helpers

Fail-closed by default. SourceMonitor denies access to every engine route (returning 403 Forbidden) unless you configure an authentication or authorization handler. This prevents the engine's create/update/delete/enqueue routes from being public by accident.

Protect the dashboard with host-specific auth in one place:

config.authentication.authenticate_with :authenticate_admin!
config.authentication.authorize_with ->(controller) {
  controller.current_user&.feature_enabled?(:source_monitor)
}
config.authentication.current_user_method = :current_user
config.authentication.user_signed_in_method = :user_signed_in?

Handlers can be symbols (invoked on the controller) or callables. Return false or raise to deny access. As soon as either handler is configured, the handler decides access and the fail-closed guard no longer applies.

Open access opt-in (non-production)

For local demos or sandboxes where engine routes are deliberately public, you can explicitly opt out of the fail-closed guard:

config.authentication.open_access = true # default: false

This is intended for non-production/demo environments only. Configuring a handler always takes precedence over this flag.

Health Model

config.health tunes automatic pause/resume heuristics.

  • window_size – number of fetch attempts to evaluate (default 20)
  • healthy_threshold – ratio that drives the "working" badge
  • auto_pause_threshold / auto_resume_threshold – percentages that trigger automatic toggling
  • auto_pause_cooldown_minutes – grace period before re-enabling a source

Helper APIs

  • SourceMonitor.configure – run-time configuration entry point
  • SourceMonitor.reset_configuration! – revert to defaults (useful in tests)
  • SourceMonitor.events – direct access to the events registry
  • SourceMonitor.queue_name(role) / SourceMonitor.queue_concurrency(role) – convenience helpers
  • SourceMonitor::Metrics.snapshot – inspect counters/gauges (great for health checks)

Environment Variables

The engine honours several environment variables out of the box:

  • SOLID_QUEUE_SKIP_RECURRING – skip loading config/recurring.yml
  • SOLID_QUEUE_RECURRING_SCHEDULE_FILE – alternative schedule file path
  • SOFT_DELETE / SOURCE_IDS / SOURCE_ID – overrides for item cleanup rake tasks
  • FETCH_LOG_DAYS / SCRAPE_LOG_DAYS – retention windows for log cleanup
  • WINDOW_MINUTES – time window (minutes) for stagger_fetch_times rake task (default 10)
  • SOURCE_MONITOR_FETCH_CONCURRENCY – override fetch queue concurrency in solid_queue.yml
  • SOURCE_MONITOR_SCRAPE_CONCURRENCY – override scrape queue concurrency in solid_queue.yml
  • SOURCE_MONITOR_MAINTENANCE_CONCURRENCY – override maintenance queue concurrency in solid_queue.yml

After Changing Configuration

  1. Restart web and worker processes so Solid Queue picks up new queue names/adapters.
  2. Re-run npm run build (engine root) and bin/rails assets:precompile if you adjust engine assets per .ai/engine-asset-configuration.md:11-188.
  3. Keep a regression test per configuration extension—for example, ensure custom validations are exercised in MiniTest.