Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/rack attack cache #1406

Merged
merged 12 commits into from
Jan 4, 2025
Merged

Feature/rack attack cache #1406

merged 12 commits into from
Jan 4, 2025

Conversation

rahul1990gupta
Copy link
Contributor

@rahul1990gupta rahul1990gupta commented Dec 23, 2024 •
edited
Loading

This PR relates to the issue outlined here. #1401
which requires using sqlite3 based solid cache for rack_attack middleware.

This

  • Sets cache for all rails caching needs. This aligns well with the long-term direction of moving all application caching needs to SolidCache. Upgrade to Rails 8 #1366
  • Expiry policy is set to 2.weeks for all keys and are taken care of by solid cache through max_age in cache.yml

Steps for installation

  • bundle install will install solid_cache and sqlite3 geme.
  • rails solid_cache:install will create config/cache.yml and generate cache_schema.rb file.
  • rails db:prepare will generate the required tables.

Testing

3.3.1 :001 > Rails.cache.write("key2", "value2")
  SolidCache::Entry Upsert (16.4ms)  INSERT INTO "solid_cache_entries" ("key","value","key_hash","byte_size","created_at") VALUES (x'646576656c6f706d656e743a6b657932', x'0004085b0649220b76616c756532063a064554', -4426413592192554962, 175, STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ON CONFLICT ("key_hash") DO UPDATE SET "key"=excluded."key","value"=excluded."value","byte_size"=excluded."byte_size" RETURNING "id"
 => true 
3.3.1 :002 > Rails.cache.read("key2")
  SolidCache::Entry Load (0.7ms)  SELECT "solid_cache_entries"."key", "solid_cache_entries"."value" FROM "solid_cache_entries" WHERE "solid_cache_entries"."key_hash" IN (-4426413592192554962)
 => "value2" 
3.3.1 :003 > exit

local server logs

  SolidCache::Entry Upsert (0.4ms)  INSERT INTO "solid_cache_entries" ("key","value","key_hash","byte_size","created_at") VALUES (x'646576656c6f706d656e743a7261636b3a3a61747461636b3a313733343935323037323a7261746520313a3a3a31', x'0004085b0769086616313733343935323037342e353433383735', -7631808399991365527, 212, STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ON CONFLICT ("key_hash") DO UPDATE SET "key"=excluded."key","value"=excluded."value","byte_size"=excluded."byte_size" RETURNING "id"

$ sqlite3 storage/development_cache.sqlite3 
SQLite version 3.45.3 2024-04-15 13:34:05
Enter ".help" for usage hints.
sqlite> select * from solid_cache_entries;
id|key|value|created_at|key_hash|byte_size
1|development:my_key||2024-12-23 10:47:10.240|-1551528455229127965|179
2|development:key2||2024-12-23 10:54:43.568|-4426413592192554962|175
3|development:rack::attack:1734951945:rate 1:::1||2024-12-23 11:05:45.640|4597245492007402599|213
4|development:rack::attack:28915865:rate 2:::1||2024-12-23 11:05:45.676|-781566585054514272|210
5|development:rack::attack:2891586:rate 3:::1||2024-12-23 11:05:45.681|-8118890843378714208|210
7|development:rack::attack:1734951946:rate 1:::1||2024-12-23 11:05:46.791|-3001282264902824331|213
13|development:rack::attack:1734952057:rate 1:::1||2024-12-23 11:07:37.242|2159563249450111939|213
14|development:rack::attack:28915867:rate 2:::1||2024-12-23 11:07:37.247|-3117416797329245765|211
20|development:rack::attack:1734952060:rate 1:::1||2024-12-23 11:07:40.479|2694537370982936877|213
30|development:rack::attack:1734952063:rate 1:::1||2024-12-23 11:07:43.882|-521309589565167187|211
34|development:rack::attack:1734952064:rate 1:::1||2024-12-23 11:07:44.854|-3811503927996024528|212
40|__solid_cache_entry_size_moving_average_estimates|3923|2024-12-23 11:07:44.883|6706543775222517821|193
41|development:rack::attack:1734952067:rate 1:::1||2024-12-23 11:07:47.844|-3327124487150915279|213
51|development:rack::attack:1734952072:rate 1:::1||2024-12-23 11:07:52.544|-7631808399991365527|212
61|development:rack::attack:1734952440:rate 1:::1||2024-12-23 11:14:00.258|2356006652687021707|213
62|development:rack::attack:28915874:rate 2:::1||2024-12-23 11:14:00.286|4183512604904441246|209
63|development:rack::attack:2891587:rate 3:::1||2024-12-23 11:14:00.296|1668920627156930690|209
64|development:user:54:unread_replies||2024-12-23 11:14:01.108|8259605808144776374|201
65|development:rack::attack:1734952441:rate 1:::1||2024-12-23 11:14:01.201|-1826250249955248891|213

Deployment playbook

  • rails db:prepare will generate the required tables

Next Steps

  • More testing: since this affects all traffic. It will be nice to test in in a staging environment before deploying it to production.

@rahul1990gupta rahul1990gupta marked this pull request as ready for review December 23, 2024 11:28
@pushcx
Copy link
Member

pushcx commented Dec 26, 2024

Sorry this took me a little time to respond to, I was sick and then traveling for the holiday.

Thank you for this and your other PRs. I’ll give them a full review on my next office hours stream, you’re welcome to drop in to discuss them or just wait for comments.

@rahul1990gupta
Copy link
Contributor Author

Thanks for your response @pushcx . No rush.
Hope that you get well soon and enjoy some time off during these holidays.

I would have loved to drop in the office hours and chat, but that would be 1:30 am for me. We can communicate asynchronously and take it from there.

pushcx added a commit to lobsters/lobsters-ansible that referenced this pull request Dec 31, 2024
@pushcx
add cache dir for lobsters/lobsters#1406
9c015fb
pushcx
pushcx requested changes Dec 31, 2024
View reviewed changes
Copy link
Member

@pushcx pushcx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks great. I appreciate you taking on the setup tasks and, yeah, I'll give it a run before I put it into production.

Gemfile Outdated Show resolved Hide resolved
config/database.yml.sample Outdated Show resolved Hide resolved
config/cache.yml Outdated Show resolved Hide resolved
pushcx
pushcx reviewed Dec 31, 2024
View reviewed changes
config/database.yml.sample Outdated Show resolved Hide resolved
pushcx
pushcx reviewed Dec 31, 2024
View reviewed changes
config/database.yml.sample Outdated Show resolved Hide resolved
@rahul1990gupta rahul1990gupta requested a review from pushcx January 2, 2025 05:52
@pushcx
Copy link
Member

pushcx commented Jan 2, 2025 •
edited
Loading

Looks like the build just needs a cache setup ("The cache database is not configured for the test environment.") and this is ready to merge! I suppose this would have to be in .github/workflows/check.ymlwhich usesenv` to provide the mariadb config; I can't guess what it would look like for SolidCache.

@rahul1990gupta
Copy link
Contributor Author

Yeah. This will need some hit and trial. Will see what can be done.

@rahul1990gupta
Copy link
Contributor Author

For this PR to go further either

  • CI will need to triggered autiomatically on push (without manual triggering)
  • I setup my own runner server to run the workflow and iterate there.

@pushcx
Copy link
Member

pushcx commented Jan 4, 2025

Now that your other PR is merged, the build runs automatically for you.

@rahul1990gupta rahul1990gupta mentioned this pull request Jan 4, 2025
Open
@pushcx pushcx merged commit 4b35876 into lobsters:master Jan 4, 2025
4 checks passed
@pushcx
Copy link
Member

pushcx commented Jan 4, 2025

Wonderful! Thank you for setting up this feature and working through all the config + build setup. I know that can get really fiddly and tedious.

@pushcx pushcx mentioned this pull request Jan 4, 2025
Closed
@rahul1990gupta
Copy link
Contributor Author

Thank you, @pushcx for being patient with me through these set of PRs. I love that how you care about moderation on the platform, and spent so much effort grooming issues and reviewing PRs. This definitely makes contributing approachable for a first-time open source contributor like me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pushcx pushcx pushcx approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rahul1990gupta @pushcx