Solid Queue in Rails 8: Advanced Tuning and Pitfalls (Part 2)

Ruby on Rails • Thursday, Feb 20, 2025

Dive into Solid Queue’s advanced features like priorities, concurrency controls and scheduled jobs while learning how to avoid common mistakes.

Solid Queue may seem simple at first glance—it stores jobs in your database and processes them with worker processes—but it includes a number of advanced features that mirror or even surpass what you might be used to from Sidekiq. This post picks up where we left off in the introduction by exploring how to tune Solid Queue for high‑throughput workloads, configure job priorities and schedules and avoid some of the pitfalls I encountered while migrating a production system.

Configuring concurrency and queues

The worker process you start with bin/rails solid_queue:work can be tuned with command‑line options and YAML configuration files. By default one worker will poll the database and execute jobs sequentially. For most applications this is fine, but if you enqueue many jobs you’ll want multiple workers or threads. You can specify the number of worker threads like so:

bin/rails solid_queue:work --threads=5

Each thread polls the database separately and locks a job with FOR UPDATE SKIP LOCKED. This means up to five jobs can run concurrently in a single worker process. For higher concurrency you can run multiple worker processes; each will compete for jobs but the database locks prevent double processing.

For more granular control you can define a config/queues.yml file that tells Solid Queue how many threads to allocate per queue. Here’s an example that defines three queues with different concurrency:

development:
  workers:
    - queues: [default]
      concurrency: 3
    - queues: [mailers]
      concurrency: 1
    - queues: [critical]
      concurrency: 5

With this configuration, when you run bin/rails solid_queue:work Solid Queue will spawn three worker threads for the default queue, one for the mailers queue and five for the critical queue. You can also specify processes instead of threads by running multiple instances of the worker command. This YAML file lives in your repository and can be checked in, making concurrency configuration explicit rather than hidden in deployment scripts.

Another important tuning parameter is visibility timeout, which determines how long a job can be in the processing state before another worker is allowed to retry it. If your jobs involve long‑running tasks such as video encoding or external API calls, set the timeout accordingly via the SOLID_QUEUE_VISIBILITY_TIMEOUT environment variable (default is 30 minutes). This prevents stuck jobs from blocking the queue indefinitely.

Priorities and queue ordering

Solid Queue supports two methods of prioritizing work: numeric priorities and queue order. When enqueuing a job you can assign a priority between ‑10 and 10, with lower numbers indicating higher priority. For example:

class SendReceiptJob < ApplicationJob
  queue_as :default
  priority -5
 
  def perform(order_id)
    # send a confirmation email
  end
end

This job will jump ahead of other default queue jobs with a higher (less negative) priority. If you omit a priority, it defaults to 0. Numeric priority is great when different jobs share the same queue but have varying importance.

Queue ordering is the other lever. Workers will process queues in the order you list them in the queues.yml file. If you want your critical queue processed before default, place it first:

development:
  workers:
    - queues: [critical, default, mailers]
      concurrency: 5

Solid Queue will exhaust jobs in critical before moving to default. Combined with numeric priorities you can ensure the most important work always runs first. Remember that each worker process (or thread) polls only the queues assigned to it, so list them carefully.

Scheduled and recurring jobs

Beyond immediate execution, Solid Queue supports scheduled and recurring jobs. To schedule a job in the future, pass a timestamp to set:

# Schedule a job one hour from now
ReportGenerationJob.set(wait_until: 1.hour.from_now).perform_later(user_id)

When you schedule a job, Solid Queue writes a record with a scheduled_at timestamp. Worker threads ignore these jobs until their scheduled time arrives. This feature can replace cron tasks for many use cases.

Recurring jobs are defined at the queue configuration level using cron syntax. You create a recurring_jobs section in your queues.yml file like this:

production:
  recurring_jobs:
    weekly_cleanup:
      job: CleanupJob
      cron: '0 3 * * 0'   # Runs every Sunday at 3 AM
      queue: default
      priority: 5
    hourly_stats:
      job: StatsJob
      cron: '0 * * * *'   # Runs at the top of every hour
      queue: default

Solid Queue will evaluate these cron expressions and enqueue the corresponding jobs at the scheduled times. Recurring jobs are especially useful for routine maintenance tasks like clearing temp files or sending daily summaries. There’s no need for an external scheduler such as cron or a separate gem like Whenever.

Monitoring and mission control

Processing jobs in your database has one hidden benefit: you can inspect and manage them using standard SQL queries. You can see enqueued jobs by running SELECT * FROM active_jobs WHERE finished_at IS NULL, filter by queue or priority, and even delete orphaned jobs manually if needed. However, manually poking the database gets old quickly, so Solid Queue ships with a simple web UI called Mission Control. To use it, mount the engine in your routes:

# config/routes.rb
Rails.application.routes.draw do
  mount SolidQueue::MissionControl => '/solid_queue'
  # other routes...
end

Navigate to /solid_queue in your browser and you’ll see dashboards of enqueued, running and failed jobs. You can retry failed jobs, delete them or dig into their arguments. In my experience this UI has been invaluable when debugging issues in production—it provides the same visibility you get from commercial queue dashboards without an external dependency.

Common pitfalls and tips

Despite its convenience, Solid Queue introduces some challenges. Here are a few pitfalls I’ve encountered and how to avoid them:

Database connections: Each worker thread uses its own Active Record connection. If you start too many threads, you may exhaust your connection pool and starve your web requests. Ensure your config/database.yml pool size is large enough and monitor connection usage in production.
Long‑running jobs: Because jobs share the same database as your main app, a job that runs for minutes or hours can hold locks or block other queries. Break long jobs into smaller units or offload them to external services when possible.
Migrations: The active_jobs table needs appropriate indexes (e.g., on queue and priority) for performance. When you upgrade Solid Queue, read the release notes for migration changes and run them promptly.
Mixed adapters: If you plan to use Solid Queue in development and Redis-backed Sidekiq in production, be aware that Active Job uses different serializer options. Jobs serialized for one adapter may not deserialize in the other. Stick to one queue backend across environments to avoid surprises.
Testing: Write unit tests for your jobs and integration tests to ensure they enqueue correctly. Use the SolidQueue::Testing.inline! helper to run jobs synchronously in tests.