Configures ActiveJob with solid_queue gem #1415

rahul1990gupta · 2025-01-04T04:08:37Z

Overview

This PR closely follows the recommendation for setting up active job from the rails official video: https://www.youtube.com/watch?v=ReyKfb12EVU

This PR

Install foreman for running the web server as well as job server.
Installs solid_queue for sqlite3-based active job backend.
Installs mission_control for monitoring jobs in production.
Adds admin controller to access the jobs UI via basic authentication.
Remove TimeAgoInWords from ApplicationHelper because it is already provided by rails and it was having conflict with mission_controls method calling rails's time_ago_in_words which has different signature than the one defined here. https://github.com/rails/mission_control-jobs/blob/ca09138a06d0322a4a39a63ebf253ba0eb56bd7c/app/helpers/mission_control/jobs/dates_helper.rb#L17
Adds a Daily Maintenance Job as a first example of an Active Job.

Testing

3.3.1 :001 > DailyMaintenanceJob.perform_later
  TRANSACTION (0.0ms)  BEGIN immediate TRANSACTION
  SolidQueue::Job Create (0.4ms)  INSERT INTO "solid_queue_jobs" ("queue_name", "class_name", "arguments", "priority", "active_job_id", "scheduled_at", "finished_at", "concurrency_key", "created_at", "updated_at") VALUES ('default', 'DailyMaintenanceJob', '{"job_class":"DailyMaintenanceJob","job_id":"92cc254c-a881-454c-a8e8-0f99094c242d","provider_job_id":null,"queue_name":"default","priority":null,"arguments":[],"executions":0,"exception_executions":{},"locale":"en","timezone":"Central Time (US \u0026 Canada)","enqueued_at":"2025-01-04T09:45:16.261443263Z","scheduled_at":"2025-01-04T09:45:16.260483255Z"}', 0, '92cc254c-a881-454c-a8e8-0f99094c242d', '2025-01-04 09:45:16.260483', NULL, NULL, '2025-01-04 09:45:16.272599', '2025-01-04 09:45:16.272599') RETURNING "id"
  TRANSACTION (0.8ms)  SAVEPOINT active_record_1
  SolidQueue::Job Load (0.1ms)  SELECT "solid_queue_jobs".* FROM "solid_queue_jobs" WHERE "solid_queue_jobs"."id" = 2 LIMIT 1
  SolidQueue::ReadyExecution Create (0.1ms)  INSERT INTO "solid_queue_ready_executions" ("job_id", "queue_name", "priority", "created_at") VALUES (2, 'default', 0, '2025-01-04 09:45:16.295201') RETURNING "id"
  TRANSACTION (0.0ms)  RELEASE SAVEPOINT active_record_1
  TRANSACTION (0.1ms)  COMMIT TRANSACTION
Enqueued DailyMaintenanceJob (Job ID: 92cc254c-a881-454c-a8e8-0f99094c242d) to SolidQueue(default)
↳ (Lobsters):1:in `<top (required)>'
 => 
#<DailyMaintenanceJob:0x0000708e9dce4f90
 @_halted_callback_hook_called=nil,
 @arguments=[],
 @exception_executions={},
 @executions=0,
 @job_id="92cc254c-a881-454c-a8e8-0f99094c242d",
 @priority=nil,
 @provider_job_id=2,
 @queue_name="default",
 @scheduled_at=2025-01-04 03:45:16.260483255 CST -06:00,
 @successfully_enqueued=true,
 @timezone="Central Time (US & Canada)">

Deployment playbook

rails db:prepare will generate the required database and corresponding tables.

Next Steps.

This will lay the foundation for #1299 and other async jobs.

app/jobs/daily_maintenance_job.rb

pushcx · 2025-01-04T17:44:13Z

config/routes.rb

@@ -263,6 +263,9 @@
    resources :reparents, only: [:new, :create]
  end

+  # only admin should be able to access it.. right ?


Let's allow any moderator to use it. It'd be good to give them insight into the kinds of partial outages that people might ask about when jobs fail.

Right now, I have configured basic_http_authentication with credentials stored in Rails credentials. Integrating with our native authentication workflow may require some more work.

I found a tutorial for basic auth. The interesting thing was setting config.mission_control.jobs.base_controller_class - if you set that to our Mod::ModeratorController (from app/controllers/mod/moderator_controller.rb, be careful, there are several controllers with very similar names) would it use our authentication?

That would be very convenient for the moderators. But, it will require more
work. For example, ModController inherits from ApplicationController which
has n+1 query check middleware, that will cause problem with
mission-control gem.

My guess is if we make a new JobModcontroller class inheriting from
Actioncontroller::Base, maybe we can make it work.

Something like this is working.
rahul1990gupta#2

another option is to extract the authentication module into a concern and use it for ApplicationController and JobsModController
rahul1990gupta#3

A Concern sounds like the exact right tool for the job, please extract one to share authentication between the mission control and mod controllers. Thanks for figuring out our options here.

pushcx · 2025-01-04T17:47:42Z

I know you haven't requested a review yet, but I am excited to see you starting this. I appreciate you taking on all these features related to Rails 8!

The Rails implementation of time_ago_in_words doesn't produce the same output as our helper and I don't want to change that very familiar UI, so please rename our helper to how_long_ago.

pushcx · 2025-01-06T21:13:24Z

lib/time_ago_in_words.rb

@@ -1,7 +1,7 @@
 # typed: false

 module TimeAgoInWords
-  def time_ago_in_words(time)
+  def how_long_ago(time)


It's probably already on your to-do list, but: there are a lot more calls to time_ago_in_words in the views to update, and please rename time_ago_in_words_label to how_long_ago_label to match so it's clear what that helper is doing.

caius · 2025-01-07T01:18:57Z

app/jobs/expire_old_ribbons_job.rb

@@ -0,0 +1,7 @@
+class ExpireOldRibbonsJob < ApplicationJob
+  queue_as :default


We should probably use queues based on SLOs, https://alexis.bernard.io/blog/2023-10-15-background-job-queues-and-priorities-may-be-the-wrong-path.html is a half decent summary of doing so (and links off to Ben Sheldon of GoodJob maintainer, and Nate Berkopec of Speedshop … notoriety?)

I've seen this in production at scale and agree.

…d_queue

rahul1990gupta · 2025-01-07T10:27:19Z

What do you think about using rails's time_ago_in_words function here ?
This is what it outputs

      #   distance_of_time_in_words(from_time, from_time + 50.minutes)                                # => about 1 hour
      #   distance_of_time_in_words(from_time, 50.minutes.from_now)                                   # => about 1 hour
      #   distance_of_time_in_words(from_time, from_time + 15.seconds)                                # => less than a minute
      #   distance_of_time_in_words(from_time, from_time + 15.seconds, include_seconds: true)         # => less than 20 seconds
      #   distance_of_time_in_words(from_time, 3.years.from_now)                                      # => about 3 years
      #   distance_of_time_in_words(from_time, from_time + 60.hours)                                  # => 3 days
      #   distance_of_time_in_words(from_time, from_time + 45.seconds, include_seconds: true)         # => less than a minute
      #   distance_of_time_in_words(from_time, from_time - 45.seconds, include_seconds: true)         # => less than a minute
      #   distance_of_time_in_words(from_time, 76.seconds.from_now)                                   # => 1 minute
      #   distance_of_time_in_words(from_time, from_time + 1.year + 3.days)                           # => about 1 year
      #   distance_of_time_in_words(from_time, from_time + 3.years + 6.months)                        # => over 3 years
      #   distance_of_time_in_words(from_time, from_time + 4.years + 9.days + 30.minutes + 5.seconds) # => about 4 years

I know it looks more verbose than our implementation of "how_long_ago", but it can be customized using engligh locale file. here is an example from official reference

  datetime:
    distance_in_words:
      about_x_hours:
        one: about %{count} hour
        other: about %{count} hours
      about_x_months:
        one: about %{count} month
        other: about %{count} months
      about_x_years:
        one: about %{count} year
        other: about %{count} years
      almost_x_years:
        one: almost %{count} year
        other: almost %{count} years
      half_a_minute: half a minute
      less_than_x_seconds:
        one: less than %{count} second
        other: less than %{count} seconds

Another reason for using rails's method is their support of localization. and that is important for us because lobsters codebase is used to power a live french hacking website and many more. And staying close to rails's feature will take it other mods to launch their local version.

      def distance_of_time_in_words(from_time, to_time = 0, options = {})
        options = {
          scope: :'datetime.distance_in_words'
        }.merge!(options)

        from_time = normalize_distance_of_time_argument_to_time(from_time)
        to_time = normalize_distance_of_time_argument_to_time(to_time)
        from_time, to_time = to_time, from_time if from_time > to_time
        distance_in_minutes = ((to_time - from_time) / 60.0).round
        distance_in_seconds = (to_time - from_time).round

        I18n.with_options locale: options[:locale], scope: options[:scope] do |locale|
          case distance_in_minutes
          when 0..1
            return distance_in_minutes == 0 ?
                   locale.t(:less_than_x_minutes, count: 1) :
                   locale.t(:x_minutes, count: distance_in_minutes) unless options[:include_seconds]

rahul1990gupta · 2025-01-07T12:33:28Z

This PR is starting to extend its initial scope. So, I have taken out the two unrelated components

time_ago_in_words upgrade/refactor Replaces custom implementation of time_ago_in_words with rails #1423
authenticator concern Extract authentication concerns #1424

rahul1990gupta added 11 commits January 4, 2025 09:36

adds solid_queue gem

6753c3b

adds active job with solid queue

f285954

adds solid_queue with mission_control

41a90e7

makes soem changes

1210c64

adds time_ago_in_words back

8e30bf4

adds time_ago in words

3b7e085

passes lint and all tests

cc7f392

fix linting issues

5b4646a

adds credentials and authentication

39fde15

adds credentials

2f0f6a9

adds default value

78995e7

rahul1990gupta marked this pull request as ready for review January 4, 2025 09:35

rahul1990gupta changed the title ~~adds solid_queue gem~~ Configures ActiveJob with solid_queue gem Jan 4, 2025

removes specs

88d8a18

pushcx reviewed Jan 4, 2025

View reviewed changes

app/jobs/daily_maintenance_job.rb Outdated Show resolved Hide resolved

pushcx reviewed Jan 4, 2025

View reviewed changes

rahul1990gupta added 5 commits January 5, 2025 09:41

fix conflict

652670a

time_ago_in_words -> how_along_ago

2e02b26

split daily_maintenance into two jobs

6c493d6

adds job_mod_controller

ed9dd05

extracts authentication into a seperate concern

9425751

pushcx reviewed Jan 6, 2025

View reviewed changes

caius reviewed Jan 7, 2025

View reviewed changes

rahul1990gupta added 3 commits January 7, 2025 15:29

resolves conflict

30fa3e7

resolves conflict

66d48df

Merge branch 'feature/active_job_with_native_auth2' into feature/soli…

50c703b

…d_queue

rahul1990gupta mentioned this pull request Jan 7, 2025

Replaces custom implementation of time_ago_in_words with rails #1423

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configures ActiveJob with solid_queue gem #1415

Configures ActiveJob with solid_queue gem #1415

rahul1990gupta commented Jan 4, 2025 •

edited

Loading

pushcx Jan 4, 2025

rahul1990gupta Jan 5, 2025

pushcx Jan 5, 2025

rahul1990gupta Jan 6, 2025

rahul1990gupta Jan 6, 2025

rahul1990gupta Jan 6, 2025

pushcx Jan 6, 2025

pushcx commented Jan 4, 2025

pushcx Jan 6, 2025

caius Jan 7, 2025

pushcx Jan 7, 2025

rahul1990gupta commented Jan 7, 2025

rahul1990gupta commented Jan 7, 2025 •

edited

Loading

		@@ -0,0 +1,7 @@
		class ExpireOldRibbonsJob < ApplicationJob
		queue_as :default

Configures ActiveJob with solid_queue gem #1415

Are you sure you want to change the base?

Configures ActiveJob with solid_queue gem #1415

Conversation

rahul1990gupta commented Jan 4, 2025 • edited Loading

Overview

Testing

Deployment playbook

Next Steps.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pushcx commented Jan 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahul1990gupta commented Jan 7, 2025

rahul1990gupta commented Jan 7, 2025 • edited Loading

rahul1990gupta commented Jan 4, 2025 •

edited

Loading

rahul1990gupta commented Jan 7, 2025 •

edited

Loading