Skip to content

Sidekiq + Parallel performance issues #328

Open
@joaopmrod

Description

I've been using Sidekiq + Parallel (https://github.com/grosser/parallel) together for a couple of years. Everything worked fine in in_processes paralellization. I would get basically the same performance when I ran Job.perform_now in the console and Job.perform_later.

This year I've upgraded my application to Ruby 3.0.2p107, Rails 6.1.6, Sidekiq 6.4.2 and Parallel 1.22 and I now get 1/10-20 the performance when the job runs in Sidekiq with perform_later.

class TestJob < ApplicationJob
  def perform
    Parallel.map(Model.all, in_processes: 5, progress: "Map") do |obj|
      heavy_task(obj)
    end
  end
end

The last time I could get an equal performance I was using Ruby 2.7.1, Rails 6.1.4,Sidekiq 6.2.1 and Parallel 1.20.

Extra info: peform_now in the console has the same performance in the new and old setup.

From my debug I think the 5 processes are running but I doubt they are running in parallel. I see that thing get slower and slower over the execution time.

Any ideia about what can be causing that?

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions