How to Use Delayed::Job in Ruby Applications

Delayed::Job – Database-backed Job Queue for Ruby | Randomize Blog

Mastering Delayed::Job – Database-backed Job Queue for Ruby

๐Ÿ”1. Fundamentals & Core Concepts

Delayed::Job is a database-backed job queue for Ruby that provides a simple and reliable way to handle background job processing. It stores jobs in your application’s database and processes them using worker processes.

What is Delayed::Job?

Delayed::Job is a background job processing system that allows you to move time-consuming tasks out of your main application flow. It stores jobs in your database and processes them using worker processes, making your application more responsive and scalable without requiring additional infrastructure like Redis.

โœ… Pros

  • No additional infrastructure needed (uses your database)
  • Simple setup and configuration
  • ACID compliant (database transactions)
  • Easy to monitor and debug
  • Built-in retry mechanism
  • Process-based concurrency (better fault isolation)
  • Mature and battle-tested

โŒ Cons

  • Lower performance than Redis-based solutions
  • Database becomes a bottleneck
  • Higher memory usage (process-based)
  • Limited advanced features
  • No built-in scheduling
  • Can impact database performance

๐Ÿ”„ Alternatives

  • Sidekiq: Higher performance, Redis-backed
  • Resque: Redis-backed, excellent monitoring
  • Que: PostgreSQL-based, ACID compliant
  • ActiveJob: Rails abstraction layer

Why Use Delayed::Job?

  • โœ… To perform tasks asynchronously (emails, notifications)
  • โœ… To avoid blocking the main web request/response cycle
  • โœ… To increase app responsiveness and user experience
  • โœ… To retry failed jobs automatically
  • โœ… To improve scalability and performance
  • โœ… When you want to avoid additional infrastructure
  • โœ… When you need ACID compliance for job processing

Delayed::Job Architecture

๐Ÿ”„ How Delayed::Job Works:

  1. Job Creation: Jobs are enqueued to database table
  2. Queue Storage: Jobs stored in delayed_jobs table
  3. Worker Processes: Separate processes poll database
  4. Job Execution: Workers pick up and execute jobs
  5. Result Handling: Success/failure logged to database

Key Components

  • Database: Your application’s database for job storage
  • Workers: Separate processes that execute jobs
  • delayed_jobs table: Database table that holds pending jobs
  • Jobs: Ruby objects that implement perform method
  • ActiveRecord: ORM for database interactions

โš™๏ธ2. Installation & Setup

Prerequisites

  • Ruby: 2.0 or higher
  • Rails: 4.0 or higher (optional but recommended)
  • Database: Any database supported by ActiveRecord
  • ActiveRecord: For database interactions

Installation Steps

1. Add Delayed::Job to Gemfile

# Gemfile
gem 'delayed_job'
gem 'delayed_job_active_record'  # For ActiveRecord backend
gem 'delayed_job_web'            # Optional: for Web UI

2. Install Gems

bundle install

3. Generate Migration

# Generate the delayed_jobs table
rails generate delayed_job:install

# This creates a migration file like:
# db/migrate/YYYYMMDDHHMMSS_create_delayed_jobs.rb

4. Run Migration

rails db:migrate

5. Create Delayed::Job Configuration

# config/initializers/delayed_job.rb
Delayed::Job.destroy_failed_jobs = false
Delayed::Job.max_attempts = 3
Delayed::Job.max_run_time = 5.minutes

# Optional: Configure logging
Delayed::Job.logger = Rails.logger

# Optional: Configure queue names
Delayed::Job.queue_name = 'default'

Rails Integration

Add to Application

# config/application.rb
module YourApp
  class Application < Rails::Application
    # ... existing code ...
    
    # Configure Delayed::Job as ActiveJob backend
    config.active_job.queue_adapter = :delayed_job
  end
end

Create Procfile

# Procfile
web: bundle exec rails server
worker: bundle exec rake jobs:work

Database Schema

Delayed Jobs Table Structure

# The migration creates a table with these columns:
create_table :delayed_jobs, force: true do |t|
  t.integer  :priority, default: 0, null: false
  t.integer  :attempts, default: 0, null: false
  t.text     :handler,                 null: false
  t.text     :last_error
  t.datetime :run_at
  t.datetime :locked_at
  t.datetime :failed_at
  t.string   :locked_by
  t.string   :queue
  t.datetime :created_at
  t.datetime :updated_at
end

add_index :delayed_jobs, [:priority, :run_at], name: 'delayed_jobs_priority'

Environment Setup

Environment Variables

# .env
DATABASE_URL=postgresql://localhost/your_app
RAILS_ENV=development
QUEUE=default,high,low

Development Setup

# Start Rails server
rails server

# Start Delayed::Job worker (in separate terminal)
bundle exec rake jobs:work

# Or with specific queue
bundle exec rake jobs:work QUEUE=high,default

Verification

Test Database Connection

# In Rails console
rails console

# Test database connection
ActiveRecord::Base.connection.execute("SELECT 1")
# Should return a result

Test Job Enqueue

# Create a test job
class TestJob
  def perform
    puts "Test job executed!"
  end
end

# Enqueue the job
Delayed::Job.enqueue(TestJob.new)
# Should return a Delayed::Job instance

Common Setup Issues

Issue: Migration failed
Solution: Check database connection and run rails db:migrate:status
Issue: Worker not processing jobs
Solution: Ensure worker is started: bundle exec rake jobs:work
Issue: Jobs stuck in database
Solution: Check worker processes: ps aux | grep delayed_job

๐Ÿš€3. Basic Usage & Workers

Creating Jobs

Basic Job Structure

class EmailJob
  def perform(user_id, message)
    user = User.find(user_id)
    UserMailer.notification(user, message).deliver_now
  end
end

Job with Error Handling

class DataProcessingJob
  def perform(data_id)
    data = Data.find(data_id)
    
    begin
      # Process the data
      data.process!
      data.update(status: 'processed')
    rescue => e
      data.update(status: 'failed', error: e.message)
      raise e  # Re-raise to trigger retry
    end
  end
end

Job with Priority

class HighPriorityJob
  def perform
    # High priority work
  end
end

# Enqueue with high priority
Delayed::Job.enqueue(HighPriorityJob.new, priority: 10)

Enqueueing Jobs

Basic Enqueue

# Enqueue immediately
Delayed::Job.enqueue(EmailJob.new(user.id, "Hello!"))

# Enqueue with delay
Delayed::Job.enqueue(EmailJob.new(user.id, "Hello!"), run_at: 1.hour.from_now)

# Enqueue with priority
Delayed::Job.enqueue(EmailJob.new(user.id, "Hello!"), priority: 5)

Using ActiveJob

class EmailJob < ApplicationJob
  queue_as :default
  
  def perform(user_id, message)
    user = User.find(user_id)
    UserMailer.notification(user, message).deliver_now
  end
end

# Enqueue using ActiveJob
EmailJob.perform_later(user.id, "Hello!")
EmailJob.set(wait: 1.hour).perform_later(user.id, "Hello!")

Worker Management

Starting Workers

# Start worker for all queues
bundle exec rake jobs:work

# Start worker for specific queues
bundle exec rake jobs:work QUEUE=high,default

# Start multiple workers
bundle exec rake jobs:work QUEUE=high,default COUNT=3

# Start worker in background
nohup bundle exec rake jobs:work > worker.log 2>&1 &

Worker Configuration

# config/initializers/delayed_job.rb
Delayed::Job.destroy_failed_jobs = false
Delayed::Job.max_attempts = 3
Delayed::Job.max_run_time = 5.minutes
Delayed::Job.sleep_delay = 5
Delayed::Job.read_ahead = 5

# Configure logging
Delayed::Job.logger = Rails.logger
Delayed::Job.logger.level = Logger::INFO

Job Lifecycle

๐Ÿ”„ Job Processing Flow:

  1. Enqueue: Job added to delayed_jobs table
  2. Wait: Job waits until run_at time
  3. Lock: Worker locks job for processing
  4. Execute: Worker calls perform method
  5. Complete: Job marked as completed or failed
  6. Retry: Failed jobs retried based on configuration

Error Handling

Retry Configuration

# Global retry settings
Delayed::Job.max_attempts = 3
Delayed::Job.destroy_failed_jobs = false

# Per-job retry settings
class RetryableJob
  def perform
    # Job logic
  end
  
  def max_attempts
    5  # Override global setting
  end
end

Custom Error Handling

class RobustJob
  def perform
    begin
      # Main job logic
      process_data
    rescue NetworkError => e
      # Retry network errors
      raise e
    rescue ValidationError => e
      # Don't retry validation errors
      log_error(e)
      return
    end
  end
  
  def max_attempts
    3
  end
  
  private
  
  def log_error(error)
    Rails.logger.error("Job failed: #{error.message}")
  end
end

Monitoring Workers

Check Worker Status

# Check running workers
ps aux | grep delayed_job

# Check job counts
Delayed::Job.count
Delayed::Job.where(failed_at: nil).count
Delayed::Job.where.not(failed_at: nil).count

# Check queue lengths
Delayed::Job.group(:queue).count

Worker Health Check

# Check if workers are processing
Delayed::Job.where('locked_at > ?', 10.minutes.ago).count

# Check for stuck jobs
Delayed::Job.where('locked_at < ?', 1.hour.ago).count

# Check failed jobs
Delayed::Job.where.not(failed_at: nil).count

๐Ÿ“‹4. Queue Management & Priority

Queue Concepts

Delayed::Job uses a single table for all jobs but supports queue names and priorities to organize job processing. Jobs are processed in order of priority (lower numbers = higher priority) and then by creation time.

Queue Configuration

Setting Queue Names

# Enqueue to specific queue
Delayed::Job.enqueue(EmailJob.new, queue: 'emails')

# Using ActiveJob
class EmailJob < ApplicationJob
  queue_as :emails
end

# Enqueue with queue
EmailJob.set(queue: 'high_priority').perform_later

Priority Levels

# High priority (processed first)
Delayed::Job.enqueue(Job.new, priority: 0)

# Normal priority
Delayed::Job.enqueue(Job.new, priority: 10)

# Low priority (processed last)
Delayed::Job.enqueue(Job.new, priority: 20)

Worker Queue Assignment

Processing Specific Queues

# Process all queues
bundle exec rake jobs:work

# Process specific queues
bundle exec rake jobs:work QUEUE=high,default

# Process single queue
bundle exec rake jobs:work QUEUE=emails

# Process multiple queues with priority
bundle exec rake jobs:work QUEUE=high,default,low

Queue Monitoring

# Check queue lengths
Delayed::Job.group(:queue).count

# Check jobs by priority
Delayed::Job.group(:priority).count

# Check failed jobs by queue
Delayed::Job.where.not(failed_at: nil).group(:queue).count

# Check stuck jobs by queue
Delayed::Job.where('locked_at < ?', 1.hour.ago).group(:queue).count

Advanced Queue Management

Queue-specific Workers

# Start dedicated email worker
bundle exec rake jobs:work QUEUE=emails COUNT=2

# Start high-priority worker
bundle exec rake jobs:work QUEUE=high COUNT=1

# Start background worker
bundle exec rake jobs:work QUEUE=default,low COUNT=3

Queue Cleanup

# Clean up old completed jobs
Delayed::Job.where('created_at < ?', 30.days.ago).delete_all

# Clean up failed jobs
Delayed::Job.where.not(failed_at: nil).delete_all

# Clean up stuck jobs
Delayed::Job.where('locked_at < ?', 1.hour.ago).update_all(locked_at: nil, locked_by: nil)

Best Practices

๐Ÿ’ก Queue Management Tips:

  • Use descriptive queue names (emails, notifications, reports)
  • Set appropriate priorities for different job types
  • Monitor queue lengths and processing times
  • Use dedicated workers for critical queues
  • Implement proper error handling and retries
  • Clean up old jobs regularly

๐Ÿ”ง5. Monitoring & Web UI

Built-in Monitoring

Database Queries

# Check overall job status
Delayed::Job.count
Delayed::Job.where(failed_at: nil).count
Delayed::Job.where.not(failed_at: nil).count

# Check queue status
Delayed::Job.group(:queue).count

# Check priority distribution
Delayed::Job.group(:priority).count

# Check processing status
Delayed::Job.where.not(locked_at: nil).count
Delayed::Job.where('locked_at < ?', 10.minutes.ago).count

Worker Status

# Check running workers
ps aux | grep delayed_job

# Check worker processes
pgrep -f "delayed_job"

# Check worker logs
tail -f log/delayed_job.log

Web UI Setup

Install delayed_job_web

# Add to Gemfile
gem 'delayed_job_web'

# Install
bundle install

Configure Routes

# config/routes.rb
Rails.application.routes.draw do
  # ... existing routes ...
  
  # Mount the web UI
  mount DelayedJobWeb => '/delayed_job'
end

Access Web UI

# Start Rails server
rails server

# Access web UI at:
# http://localhost:3000/delayed_job

Custom Monitoring

Health Check Endpoint

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  def delayed_job
    stats = {
      total_jobs: Delayed::Job.count,
      pending_jobs: Delayed::Job.where(failed_at: nil).count,
      failed_jobs: Delayed::Job.where.not(failed_at: nil).count,
      running_workers: `ps aux | grep delayed_job | grep -v grep | wc -l`.strip.to_i
    }
    
    render json: stats
  end
end

Monitoring Dashboard

# app/controllers/admin/dashboard_controller.rb
class Admin::DashboardController < ApplicationController
  def delayed_job
    @stats = {
      total_jobs: Delayed::Job.count,
      pending_jobs: Delayed::Job.where(failed_at: nil).count,
      failed_jobs: Delayed::Job.where.not(failed_at: nil).count,
      queue_stats: Delayed::Job.group(:queue).count,
      priority_stats: Delayed::Job.group(:priority).count
    }
  end
end

Logging Configuration

Configure Logging

# config/initializers/delayed_job.rb
Delayed::Job.logger = Rails.logger
Delayed::Job.logger.level = Logger::INFO

# Custom log format
Delayed::Job.logger.formatter = proc do |severity, datetime, progname, msg|
  "[#{datetime}] #{severity}: #{msg}\n"
end

Log Levels

# Development - verbose logging
Delayed::Job.logger.level = Logger::DEBUG

# Production - minimal logging
Delayed::Job.logger.level = Logger::WARN

# Custom logging
Delayed::Job.logger.info("Worker started")
Delayed::Job.logger.error("Job failed: #{error.message}")

Alerting

Failed Job Alerts

# config/initializers/delayed_job.rb
Delayed::Job.after_failure do |job, error|
  # Send alert
  AlertService.notify("Job failed: #{job.id} - #{error.message}")
end

Queue Monitoring

# Check queue health
def check_queue_health
  failed_count = Delayed::Job.where.not(failed_at: nil).count
  stuck_count = Delayed::Job.where('locked_at < ?', 1.hour.ago).count
  
  if failed_count > 100
    AlertService.notify("High failed job count: #{failed_count}")
  end
  
    if stuck_count > 50
    AlertService.notify("Jobs stuck: #{stuck_count}")
  end
end

๐Ÿ“Š6. Advanced Features

Job Scheduling

Delayed Execution

# Execute after 1 hour
Delayed::Job.enqueue(Job.new, run_at: 1.hour.from_now)

# Execute at specific time
Delayed::Job.enqueue(Job.new, run_at: Time.parse('2024-01-01 09:00:00'))

# Execute every day at 9 AM
class DailyJob
  def perform
    # Job logic
  end
  
  def self.schedule_daily
    Delayed::Job.enqueue(new, run_at: Time.current.beginning_of_day + 9.hours)
  end
end

Recurring Jobs

# Simple recurring job
class RecurringJob
  def perform
    # Job logic
    
    # Schedule next execution
    Delayed::Job.enqueue(self.class.new, run_at: 1.hour.from_now)
  end
end

# Start recurring job
Delayed::Job.enqueue(RecurringJob.new, run_at: 1.hour.from_now)

Job Serialization

Complex Objects

# Jobs with complex parameters
class ComplexJob
  def initialize(user, options = {})
    @user = user
    @options = options
  end
  
  def perform
    # Access instance variables
    user = @user
    options = @options
    
    # Job logic
  end
end

# Enqueue with complex object
user = User.find(1)
Delayed::Job.enqueue(ComplexJob.new(user, {priority: 'high'}))

ActiveRecord Objects

# Jobs with ActiveRecord objects
class UserNotificationJob
  def initialize(user_id)
    @user_id = user_id
  end
  
  def perform
    user = User.find(@user_id)
    # Job logic with user
  end
end

# Enqueue with user ID
Delayed::Job.enqueue(UserNotificationJob.new(user.id))

Custom Job Classes

Job with Custom Methods

class AdvancedJob
  def perform
    # Main job logic
    process_data
  end
  
  def max_attempts
    5  # Custom retry attempts
  end
  
  def max_run_time
    10.minutes  # Custom timeout
  end
  
  def queue_name
    'high_priority'  # Custom queue
  end
  
  private
  
  def process_data
    # Complex processing logic
  end
end

Job with Callbacks

class CallbackJob
  def perform
    # Job logic
  end
  
  def before(job)
    Rails.logger.info("Starting job #{job.id}")
  end
  
  def after(job)
    Rails.logger.info("Completed job #{job.id}")
  end
  
  def error(job, exception)
    Rails.logger.error("Job #{job.id} failed: #{exception.message}")
  end
end

Performance Optimization

Database Optimization

# Add database indexes
add_index :delayed_jobs, [:priority, :run_at]
add_index :delayed_jobs, [:queue, :priority, :run_at]
add_index :delayed_jobs, [:locked_at, :locked_by]
add_index :delayed_jobs, [:failed_at]

# Configure read_ahead
Delayed::Job.read_ahead = 10

# Configure sleep_delay
Delayed::Job.sleep_delay = 5

Worker Optimization

# Start multiple workers
bundle exec rake jobs:work COUNT=4

# Use different queues for different workers
# Worker 1: High priority jobs
bundle exec rake jobs:work QUEUE=high COUNT=2

# Worker 2: Background jobs
bundle exec rake jobs:work QUEUE=default,low COUNT=3

Integration with ActiveJob

ActiveJob Configuration

# config/application.rb
config.active_job.queue_adapter = :delayed_job

# Use ActiveJob features
class EmailJob < ApplicationJob
  queue_as :emails
  
  def perform(user_id, message)
    # Job logic
  end
end

# Enqueue with ActiveJob
EmailJob.perform_later(user.id, "Hello!")
EmailJob.set(wait: 1.hour).perform_later(user.id, "Hello!")
EmailJob.set(queue: 'high').perform_later(user.id, "Hello!")

Custom ActiveJob Adapter

# lib/active_job/queue_adapters/delayed_job_adapter.rb
module ActiveJob
  module QueueAdapters
    class DelayedJobAdapter
      def enqueue(job)
        Delayed::Job.enqueue(job, queue: job.queue_name)
      end
      
      def enqueue_at(job, timestamp)
        Delayed::Job.enqueue(job, run_at: Time.at(timestamp))
      end
    end
  end
end

๐Ÿ”ฅ7. Troubleshooting & Best Practices

Common Issues

Jobs Not Processing

Problem: Jobs are enqueued but not being processed
Diagnosis: Check if workers are running
Solution: Start workers with bundle exec rake jobs:work

Jobs Stuck in Database

Problem: Jobs are locked but not being processed
Diagnosis: Check for crashed workers
Solution: Unlock stuck jobs: Delayed::Job.where('locked_at < ?', 1.hour.ago).update_all(locked_at: nil, locked_by: nil)

High Memory Usage

Problem: Workers consuming too much memory
Diagnosis: Check for memory leaks in jobs
Solution: Restart workers periodically or use process monitoring

Performance Issues

Slow Job Processing

# Check for slow jobs
Delayed::Job.where('locked_at > ?', 10.minutes.ago).count

# Optimize database queries
add_index :delayed_jobs, [:priority, :run_at]

# Increase worker count
bundle exec rake jobs:work COUNT=4

# Use dedicated workers for slow jobs
bundle exec rake jobs:work QUEUE=slow_jobs COUNT=2

Database Bottleneck

# Monitor database performance
Delayed::Job.where('created_at > ?', 1.hour.ago).count

# Optimize read_ahead setting
Delayed::Job.read_ahead = 5

# Use connection pooling
Delayed::Job.connection_pool = ActiveRecord::Base.connection_pool

Monitoring & Debugging

Job Debugging

# Check job details
job = Delayed::Job.find(job_id)
puts job.handler
puts job.last_error
puts job.attempts

# Check job payload
job_object = YAML.load(job.handler)
puts job_object.class
puts job_object.instance_variables

Worker Debugging

# Check worker logs
tail -f log/delayed_job.log

# Check worker processes
ps aux | grep delayed_job

# Check worker status
Delayed::Job.where.not(locked_at: nil).group(:locked_by).count

Best Practices

๐Ÿ’ก Production Best Practices:

  • Use process monitoring (systemd, monit, supervisord)
  • Implement proper logging and monitoring
  • Set up alerts for failed jobs and stuck workers
  • Use database indexes for better performance
  • Implement job cleanup strategies
  • Test job failure scenarios
  • Use appropriate queue names and priorities
  • Monitor database performance impact

Security Considerations

Job Security

# Validate job parameters
class SecureJob
  def initialize(user_id, data)
    @user_id = user_id
    @data = data
  end
  
  def perform
    # Validate inputs
    user = User.find(@user_id)
    raise "Unauthorized" unless user.can_perform_action?
    
    # Process with validated data
    process_secure_data(@data)
  end
  
  private
  
  def process_secure_data(data)
    # Sanitize and process data
  end
end

Worker Security

# Run workers with limited permissions
# Use dedicated user for workers
sudo -u delayed_job_user bundle exec rake jobs:work

# Set up firewall rules
# Limit database access for workers

# Use environment-specific configurations
Delayed::Job.logger.level = Logger::WARN if Rails.env.production?

๐Ÿšจ8. Interview Questions & Answers

Basic Questions

Q: What is Delayed::Job and how does it work?
A: Delayed::Job is a database-backed job queue for Ruby that stores jobs in a database table and processes them using worker processes. Jobs are serialized and stored in the delayed_jobs table, and workers poll the database for pending jobs to execute.
Q: What are the advantages of using Delayed::Job over Redis-based solutions?
A: No additional infrastructure needed, ACID compliance, simple setup, easy debugging, and no external dependencies. It uses your existing database.
Q: How do you enqueue a job with Delayed::Job?
A: Delayed::Job.enqueue(JobClass.new, priority: 10, run_at: 1.hour.from_now, queue: 'emails')

Intermediate Questions

Q: How does Delayed::Job handle job failures and retries?
A: Failed jobs are marked with failed_at timestamp and last_error. Jobs are retried based on max_attempts configuration. You can configure global or per-job retry settings.
Q: What's the difference between priority and queue in Delayed::Job?
A: Priority determines processing order (lower numbers = higher priority). Queue names are used for organizing jobs and can be used to route jobs to specific workers.
Q: How do you monitor Delayed::Job in production?
A: Use database queries to check job counts, failed jobs, and queue lengths. Use delayed_job_web for web UI. Monitor worker processes and set up alerts for failures.

Advanced Questions

Q: How would you optimize Delayed::Job for high throughput?
A: Add database indexes, increase worker count, use dedicated workers for different queues, optimize read_ahead and sleep_delay settings, and implement proper job cleanup.
Q: What are the potential bottlenecks when using Delayed::Job?
A: Database becomes the bottleneck, limited concurrency due to process-based workers, potential for database locks, and slower performance compared to Redis-based solutions.
Q: How do you handle job serialization with complex objects?
A: Use YAML serialization for complex objects, pass IDs instead of full ActiveRecord objects, implement custom serialization for large objects, and be careful with object references.

System Design Questions

Q: When would you choose Delayed::Job over Sidekiq or Resque?
A: When you want to avoid additional infrastructure, need ACID compliance, have simple job requirements, or are working in an environment where Redis is not available.
Q: How would you design a job processing system that can handle millions of jobs?
A: Use database partitioning, implement job batching, use multiple worker pools, implement job prioritization, use database clustering, and implement proper monitoring and alerting.

โ“9. Real-World Case Studies

E-commerce Platform

Challenge

A large e-commerce platform needed to process order confirmations, inventory updates, and customer notifications without blocking the main application.

Solution

# Order processing jobs
class OrderConfirmationJob
  def perform(order_id)
    order = Order.find(order_id)
    OrderMailer.confirmation(order).deliver_now
    InventoryService.update_stock(order)
  end
end

class InventoryUpdateJob
  def perform(product_id)
    product = Product.find(product_id)
    product.update_stock_levels
    product.notify_low_stock if product.low_stock?
  end
end

# Queue configuration
bundle exec rake jobs:work QUEUE=high COUNT=2      # Order confirmations
bundle exec rake jobs:work QUEUE=default COUNT=3   # General processing
bundle exec rake jobs:work QUEUE=low COUNT=1       # Background tasks

Results

  • Reduced order processing time from 5 seconds to 200ms
  • Handled 10,000+ orders per day
  • Improved customer experience with immediate order confirmation
  • Reliable inventory management with automatic retries

Content Management System

Challenge

A CMS needed to process large file uploads, generate thumbnails, and send notifications without impacting user experience.

Solution

# File processing jobs
class FileProcessingJob
  def perform(file_id)
    file = FileUpload.find(file_id)
    
    # Generate thumbnails
    file.generate_thumbnails
    
    # Update file status
    file.update(status: 'processed')
    
    # Notify user
    NotificationJob.perform_later(file.user_id, "File processed")
  end
end

class NotificationJob
  def perform(user_id, message)
    user = User.find(user_id)
    UserMailer.notification(user, message).deliver_now
  end
end

# Worker configuration
bundle exec rake jobs:work QUEUE=file_processing COUNT=3
bundle exec rake jobs:work QUEUE=notifications COUNT=2

Results

  • Processed 5,000+ files per day
  • Reduced upload processing time by 80%
  • Improved system reliability with automatic retries
  • Scalable solution for growing user base

Analytics Platform

Challenge

An analytics platform needed to process large datasets, generate reports, and send scheduled notifications without impacting real-time data collection.

Solution

# Analytics processing jobs
class DataProcessingJob
  def perform(dataset_id)
    dataset = Dataset.find(dataset_id)
    
    # Process data
    results = AnalyticsService.process(dataset)
    
    # Store results
    dataset.update(results: results, status: 'completed')
    
    # Schedule report generation
    ReportGenerationJob.perform_later(dataset.id)
  end
end

class ReportGenerationJob
  def perform(dataset_id)
    dataset = Dataset.find(dataset_id)
    report = ReportService.generate(dataset)
    
    # Send report
    ReportMailer.send_report(dataset.user, report).deliver_now
  end
end

# Scheduled jobs
class ScheduledReportJob
  def perform
    # Generate weekly reports
    User.find_each do |user|
      WeeklyReportJob.perform_later(user.id)
    end
    
    # Schedule next run
    Delayed::Job.enqueue(self.class.new, run_at: 1.week.from_now)
  end
end

Results

  • Processed 1TB+ of data daily
  • Generated 10,000+ reports per week
  • 99.9% uptime for real-time analytics
  • Automated report delivery system

๐Ÿข10. Reference & Commands

Configuration Options

Global Configuration

# config/initializers/delayed_job.rb
Delayed::Job.destroy_failed_jobs = false
Delayed::Job.max_attempts = 3
Delayed::Job.max_run_time = 5.minutes
Delayed::Job.sleep_delay = 5
Delayed::Job.read_ahead = 5
Delayed::Job.logger = Rails.logger

Environment Variables

# .env
QUEUE=default,high,low
COUNT=3
RAILS_ENV=production
DATABASE_URL=postgresql://localhost/app_production

Rake Commands

Worker Management

# Start workers
bundle exec rake jobs:work
bundle exec rake jobs:work QUEUE=high,default
bundle exec rake jobs:work QUEUE=* COUNT=4

# Stop workers
pkill -f "delayed_job"

# Restart workers
pkill -f "delayed_job" && bundle exec rake jobs:work

Database Commands

# Generate migration
rails generate delayed_job:install

# Run migration
rails db:migrate

# Check migration status
rails db:migrate:status

Database Schema

Delayed Jobs Table

create_table :delayed_jobs, force: true do |t|
  t.integer  :priority, default: 0, null: false
  t.integer  :attempts, default: 0, null: false
  t.text     :handler,                 null: false
  t.text     :last_error
  t.datetime :run_at
  t.datetime :locked_at
  t.datetime :failed_at
  t.string   :locked_by
  t.string   :queue
  t.datetime :created_at
  t.datetime :updated_at
end

add_index :delayed_jobs, [:priority, :run_at], name: 'delayed_jobs_priority'

Monitoring Queries

Job Statistics

# Overall statistics
Delayed::Job.count
Delayed::Job.where(failed_at: nil).count
Delayed::Job.where.not(failed_at: nil).count

# Queue statistics
Delayed::Job.group(:queue).count
Delayed::Job.group(:priority).count

# Worker statistics
Delayed::Job.where.not(locked_at: nil).group(:locked_by).count
Delayed::Job.where('locked_at < ?', 10.minutes.ago).count

Health Checks

# Check for stuck jobs
Delayed::Job.where('locked_at < ?', 1.hour.ago).count

# Check for old failed jobs
Delayed::Job.where('failed_at < ?', 7.days.ago).count

# Check queue health
Delayed::Job.group(:queue).count.each do |queue, count|
  puts "#{queue}: #{count} jobs"
end

Useful Gems

Core Gems

  • delayed_job: Core job processing
  • delayed_job_active_record: ActiveRecord backend
  • delayed_job_web: Web monitoring interface

Related Gems

  • activejob: Rails job abstraction
  • sidekiq: Alternative Redis-based solution
  • resque: Alternative Redis-based solution
  • que: PostgreSQL-based alternative

Resources

Documentation

  • Official Documentation: https://github.com/collectiveidea/delayed_job
  • Rails Guide: https://guides.rubyonrails.org/active_job_basics.html
  • API Documentation: https://www.rubydoc.info/gems/delayed_job

Community

  • GitHub Issues: https://github.com/collectiveidea/delayed_job/issues
  • Stack Overflow: Search for "delayed_job"
  • Rails Forum: https://discuss.rubyonrails.org/

๐Ÿ“‹11. Commands & Concepts Reference Table

Core Commands

CommandDescriptionUsage
rails generate delayed_job:installGenerate migration for delayed_jobs tableInitial setup
rails db:migrateRun migration to create delayed_jobs tableAfter generating migration
bundle exec rake jobs:workStart worker to process jobsProduction/development
bundle exec rake jobs:work QUEUE=highStart worker for specific queueQueue-specific processing
bundle exec rake jobs:work COUNT=3Start multiple workersHigh throughput
pkill -f "delayed_job"Stop all delayed_job workersMaintenance/restart
ps aux | grep delayed_jobCheck running workersMonitoring

Ruby/Rails Commands

CommandDescriptionUsage
Delayed::Job.enqueue(Job.new)Enqueue job immediatelyBasic job enqueueing
Delayed::Job.enqueue(Job.new, run_at: 1.hour.from_now)Enqueue job with delayScheduled jobs
Delayed::Job.enqueue(Job.new, priority: 10)Enqueue job with priorityPriority processing
Delayed::Job.enqueue(Job.new, queue: 'emails')Enqueue job to specific queueQueue management
Delayed::Job.countCount total jobsMonitoring
Delayed::Job.where(failed_at: nil).countCount pending jobsQueue monitoring
Delayed::Job.where.not(failed_at: nil).countCount failed jobsError monitoring
Delayed::Job.group(:queue).countCount jobs by queueQueue analysis

Database Commands

CommandDescriptionUsage
rails db:migrate:statusCheck migration statusSetup verification
rails db:rollbackRollback last migrationDevelopment
rails consoleOpen Rails consoleDebugging/testing
ActiveRecord::Base.connection.execute("SELECT 1")Test database connectionConnection testing

Monitoring Commands

CommandDescriptionUsage
tail -f log/delayed_job.logMonitor worker logsReal-time monitoring
pgrep -f "delayed_job"Find worker processesProcess monitoring
Delayed::Job.where('locked_at < ?', 1.hour.ago).countCheck for stuck jobsHealth monitoring
Delayed::Job.where.not(locked_at: nil).group(:locked_by).countCheck worker distributionLoad balancing

Core Concepts

ConceptDescriptionUsage
JobRuby object with perform method that contains the work to be doneBackground task execution
WorkerProcess that polls database and executes jobsJob processing
QueueNamed group for organizing jobs (stored in queue column)Job organization
PriorityNumeric value determining job processing order (lower = higher priority)Job ordering
HandlerYAML-serialized job object stored in databaseJob persistence
LockingMechanism to prevent multiple workers from processing same jobConcurrency control
RetryAutomatic re-execution of failed jobs based on max_attemptsError handling
SerializationConverting job objects to YAML for database storageJob persistence

Configuration Options

OptionDescriptionDefault
max_attemptsMaximum number of retry attempts for failed jobs3
max_run_timeMaximum time a job can run before being killed5 minutes
sleep_delaySeconds to sleep when no jobs are available5
read_aheadNumber of jobs to read from database at once5
destroy_failed_jobsWhether to delete jobs after max_attemptsfalse
loggerLogger instance for worker outputRails.logger

Database Schema Fields

FieldTypeDescription
idintegerPrimary key, unique job identifier
priorityintegerJob priority (lower = higher priority)
attemptsintegerNumber of execution attempts
handlertextYAML-serialized job object
last_errortextLast error message if job failed
run_atdatetimeWhen job should be executed
locked_atdatetimeWhen job was locked by worker
locked_bystringWorker process identifier
failed_atdatetimeWhen job permanently failed
queuestringQueue name for job organization
created_atdatetimeWhen job was created
updated_atdatetimeWhen job was last updated

Environment Variables

VariableDescriptionExample
QUEUEComma-separated list of queues to processhigh,default,low
COUNTNumber of worker processes to start4
RAILS_ENVRails environment (development, production, etc.)production
DATABASE_URLDatabase connection URLpostgresql://localhost/app_production
REDIS_URLRedis connection URL (if using Redis for other features)redis://localhost:6379/0

Common Job Patterns

PatternDescriptionUse Case
Email JobSend emails asynchronouslyUser notifications, marketing emails
Data ProcessingProcess large datasets in backgroundAnalytics, reports, data imports
File ProcessingHandle file uploads and processingImage resizing, document processing
API CallsMake external API calls asynchronouslyThird-party integrations, webhooks
Cleanup JobsPeriodic cleanup of old dataDatabase maintenance, log cleanup
Scheduled JobsExecute jobs at specific timesDaily reports, maintenance tasks

Learn more aboutย Rails
Learn more aboutย Active Job
Learn more aboutย DevOps

Scroll to Top