Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip redundant queries when include_data is disabled #47

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,9 @@ class UserSerializer < BaseSerializer
# The previous one is a shorthand for the following lines:
lazy_relationship :blog_posts, loader: AmsLazyRelationships::Loaders::Association.new("User", :blog_posts)
has_many :blog_posts, serializer: BlogPostSerializer do |serializer|
serializer.lazy_blog_posts
# non-proc custom finder will work as well, but it can produce redundant sql
# queries, please see [Example 2: Modifying the relationship before rendering](#example-2-modifying-the-relationship-before-rendering)
-> { serializer.lazy_blog_posts }
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can still be called without using a proc, right?
Maybe we could keep the old version for simplicity and just mention the proc form in the include_data example you added in https://github.com/Bajena/ams_lazy_relationships/pull/47/files#diff-04c6e90faac2675aa89e2176d2eec7d8R134 ?

Suggested change
-> { serializer.lazy_blog_posts }
serializer.lazy_blog_posts

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can still be called without using a proc, right?

sure, Reflection monkey-patch will forward non-proc value as it is.
But I believe that in readme is better to propose the best possible sample by default.
Just imagine:

  1. engineer_1 adds non-proc custom finders all over the services & serializers;
  2. engineer_2 turns ActiveModel::Serializer.config.include_data_default = false globally
  3. local naive tests will assert successfully that json response is free from data, but
  4. production servers will continue to fail due to OOM and that will be quite tricky to discover why

I think we need to leave -> { serializer.lazy_blog_posts } as a very default sample, and add a small hint that serializer.lazy_blog_posts will work as well but can produce redundant queries.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to leave -> { serializer.lazy_blog_posts } as a very default sample, and add a small hint that serializer.lazy_blog_posts will work as well but can produce redundant queries.

Ok, I must admit that your explanation makes sense to me. Let's leave proc, but also let's mention the non-proc version as you suggested 👍

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

roger that! )

end

lazy_has_one :poro_model, loader: AmsLazyRelationships::Loaders::Direct.new(:poro_model) { |object| PoroModel.new(object) }
Expand Down Expand Up @@ -116,6 +118,16 @@ end
#### Example 2: Modifying the relationship before rendering
Sometimes it may happen that you need to process the relationship before rendering, e.g. decorate the records. In this case the gem provides a special method (in our case `lazy_comments`) for each defined relationship. Check out the example - we'll decorate every comment before serializing:

```ruby
class BlogPostSerializer < BaseSerializer
lazy_has_many :comments do |serializer|
-> { serializer.lazy_comments.map(&:decorate) }
end
end
```

Despite the fact that non-block custom finder such as

```ruby
class BlogPostSerializer < BaseSerializer
lazy_has_many :comments do |serializer|
Expand All @@ -124,6 +136,29 @@ class BlogPostSerializer < BaseSerializer
end
```

will work still, it's better to implement it in a form of lambda, in order to avoid redundant SQL queries when `include_data` AMS setting appears to be `false`:

```ruby
class BlogPostSerializer < BaseSerializer
lazy_has_many :comments do |serializer|
include_data :if_sideloaded
-> { serializer.lazy_comments.map(&:decorate) }
end
end
```

Feel free to skip custom lazy finder for association if your goal is just to define `include_data` setting and/or to specify some links and metas:

```ruby
class BlogPostSerializer < BaseSerializer
lazy_has_many :comments do
include_data :if_sideloaded
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't it supposed to be:

Suggested change
include_data :if_sideloaded
include_data false

?

Copy link
Collaborator Author

@stokarenko stokarenko Jan 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:if_sideloaded is the most complex example, this way data will be included for explicit ?include=association and rejected without it.

In second case we need to suppress any kind of finders but we need to keep the finder alive for the time when first case will happen.

i.e.

  lazy_has_many :comments do
    include_data false
    nil
  end

is completely ok and ams_lazy_relationship will behave like that without this PR.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks for explanation - so this will work correctly, without N+1s, both when the resource is included and when it isn't 👍

link :self, 'a link'
meta name: 'Dan Brown'
end
end
```

#### Example 3: Introducing loader classes
Under the hood ams_lazy_relationships uses special loader classes to batch load the relationships. By default the gem uses serializer class names and relationship names to instantiate correct loaders, but it may happen that e.g. your serializer's class name doesn't match the model name (e.g. your model's name is `BlogPost` but the serializer's name is `PostSerializer`).

Expand Down
1 change: 1 addition & 0 deletions lib/ams_lazy_relationships.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
require "active_model_serializers"

require "ams_lazy_relationships/version"
require "ams_lazy_relationships/extensions"
require "ams_lazy_relationships/loaders"
require "ams_lazy_relationships/core"

Expand Down
19 changes: 14 additions & 5 deletions lib/ams_lazy_relationships/core/relationship_wrapper_methods.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ def define_relationship_wrapper_methods
define_singleton_method(
"lazy_#{relationship_type}"
) do |relationship_name, options = {}, &block|
send(:define_lazy_association, relationship_type, relationship_name, options, block)
define_lazy_association(relationship_type, relationship_name, options, block)
end
end
end
Expand All @@ -29,11 +29,20 @@ def define_lazy_association(type, name, options, block)

real_relationship_options = options.except(*lazy_relationship_option_keys)

block ||= lambda do |serializer|
serializer.public_send("lazy_#{name}")
end
public_send(type, name.to_sym, real_relationship_options) do |serializer|
block_value = instance_exec(serializer, &block) if block

public_send(type, name.to_sym, real_relationship_options, &block)
if block && block_value != :nil
# respect the custom finder for lazy association
# @see https://github.com/rails-api/active_model_serializers/blob/v0.10.10/lib/active_model/serializer/reflection.rb#L165-L168
block_value
else
# provide default lazy association finder in a form of lambda,
# in order to play nice with possible `include_data` setting.
# @see lib/ams_lazy_relationships/extensions/reflection.rb
serializer.method("lazy_#{name}")
end
end

lazy_relationship(name, options.slice(*lazy_relationship_option_keys))
end
Expand Down
8 changes: 8 additions & 0 deletions lib/ams_lazy_relationships/extensions.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# frozen_string_literal: true

require "ams_lazy_relationships/extensions/reflection"

module AmsLazyRelationships
module Extensions
end
end
71 changes: 71 additions & 0 deletions lib/ams_lazy_relationships/extensions/reflection.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# frozen_string_literal: true

# There is a general problem inside AMS related to custom association finder
# combined with `include_data` setting:
#
# class BlogPostSerializer < BaseSerializer
# belongs_to :category do
# include_data :if_sideloaded
# object.categories.last
# end
# end
#
# The problem is that `belongs_to` block will be fully evaluated each time for
# each object, and only after that AMS is able to take into account
# `include_data` mode -
# https://github.com/rails-api/active_model_serializers/blob/v0.10.10/lib/active_model/serializer/reflection.rb#L162-L163
#
# def value(serializer, include_slice)
# # ...
# block_value = instance_exec(serializer, &block) if block
# return unless include_data?(include_slice)
# # ...
# end
#
# That causing redundant (and so huge potentially!) SQL queries and AR objects
# allocation when `include_data` appears to be `false` but `belongs_to` block
# defines instant (not a kind of AR::Relation) custom association finder.
#
# Described problem is a very specific use case for pure AMS applications.
# The bad news is that `ams_lazy_relationships` always utilizes the
# association block -
# https://github.com/Bajena/ams_lazy_relationships/blob/v0.2.0/lib/ams_lazy_relationships/core/relationship_wrapper_methods.rb#L32-L36
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd avoid adding links to specific versions in the code. Let's get rid of lines 32-44 - they may quickly get out of date.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is exactly why I pinned v0.2.0 tag here ))

#
# def define_lazy_association(type, name, options, block)
# #...
# block ||= lambda do |serializer|
# serializer.public_send("lazy_#{name}")
# end
#
# public_send(type, name.to_sym, real_relationship_options, &block)
# #...
# end
#
# This way we break `include_data` optimizations for the host application.
#
# In order to overcome that we are forced to monkey-patch
# `AmsLazyRelationships::Extensions::Reflection#value` method and make it to be
# ready for Proc returned by association block. This way we will use a kind of
#
# block ||= lambda do |serializer|
# -> { serializer.public_send("lazy_#{name}") }
# end
#
# as association block, then AMS will evaluate it, get the value of `include_data`
# setting, make a decision do we need to continue with that association, if so -
# will finally evaluate the proc with lazy relationship inside it.

module AmsLazyRelationships
module Extensions
module Reflection
def value(*)
case (block_value = super)
when Proc, Method then block_value.call
else block_value
end
end
end
end
end

::ActiveModel::Serializer::Reflection.prepend AmsLazyRelationships::Extensions::Reflection
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd consider applying this patch only to AMS versions >= 0.10.3.
Nobody using lower versions of AMS is probably going to return lambdas from their association blocks anyways as they won't benefit from that anyhow.

Copy link
Collaborator Author

@stokarenko stokarenko Jan 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought a lot about that from the beginning, but this way we need to apply the same condition in default define_lazy_association block.

Even worse, readme description will be more complex as well since for lower AMS versions the proc-like custom finder will instantly fail. I.e. we propose the snippet of code which will crush on AMS version still accepted by gemspec file..

As an option, we can move all include_data stuff in separate lib file. require 'ams_lazy_relationships/efficient_finders' will engage monkey-patch and proc-like default finder, without said require nothing will be applied.

How you think?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an option, we can move all include_data stuff in separate lib file.

Notice that this way ams_lazy_relationships will still silently produce a huge problems to consumers due to cancelled effect from include_data optimizations in sense of DB hits and memory usage.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, you're right, doing it the way I suggested generates more problems than it solves. Let's keep it then. But in such case let's make sure that all AMS versions are tested for proc evaluation (specs are skipped for lower AMS versions in https://github.com/Bajena/ams_lazy_relationships/pull/47/files#diff-6e72b60a2fb8b930cc9741785114a948R871).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are tested already due to the new default define_lazy_association block, but I'll add an accented test with proc-like custom finder )

143 changes: 143 additions & 0 deletions spec/core_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -866,4 +866,147 @@ class UserSerializer < BaseTestSerializer
end
end
end

describe 'include_data AMS setting' do
shared_examples 'lazy loader when custom finder is specified' do
let(:adapter_class) { json_api_adapter_class }
let(:includes) { ['blog_posts'] }
let(:blog_post_data) { json.dig(:data, :relationships, :blog_posts, :data) }

it 'loads the association' do
expect { json }
.to make_database_queries(count: 1, matching: 'blog_posts')

expect(blog_post_data).to be_present
end
end

context 'proc-like custom finder' do
let(:level0_serializer_class) do
module Serializer14
class User1Serializer < BaseTestSerializer
lazy_has_many :blog_posts do |serializer|
-> { serializer.lazy_blog_posts }
end
end
end

Serializer14::User1Serializer
end

include_examples 'lazy loader when custom finder is specified'
end

context 'non-proc custom finder' do
let(:level0_serializer_class) do
module Serializer14
class User2Serializer < BaseTestSerializer
lazy_has_many :blog_posts do |serializer|
serializer.lazy_blog_posts
end
end
end

Serializer14::User2Serializer
end

include_examples 'lazy loader when custom finder is specified'
end

next unless AMS_VERSION >= Gem::Version.new("0.10.3")

shared_examples 'lazy loader when include_data option is set' do
let(:adapter_class) { json_api_adapter_class }
let(:includes) { ['blog_posts'] }
let(:category_data) { json.dig(:included, 0, :relationships, :category, :data) }

it 'does not fire unnecessary SQL query' do
expect { json }
.to make_database_queries(count: 1, matching: 'blog_posts')
.and make_database_queries(count: 0, matching: 'categories')

expect(category_data).to be_nil
end

context 'when sideloaded' do
let(:includes) { ['blog_posts.category'] }

it 'fires single SQL query' do
expect { json }
.to make_database_queries(count: 1, matching: 'blog_posts')
.and make_database_queries(count: 1, matching: 'categories')

expect(category_data).to be_present
end
end
end

context 'include_data disabled globally' do
let(:level0_serializer_class) do
module Serializer15
class BlogPost1Serializer < BaseTestSerializer
lazy_belongs_to :category
end

class User1Serializer < BaseTestSerializer
lazy_has_many :blog_posts, serializer: BlogPost1Serializer
end
end

Serializer15::User1Serializer
end

around do |example|
backup = ActiveModel::Serializer.config.include_data_default
ActiveModel::Serializer.config.include_data_default = :if_sideloaded

example.run

ActiveModel::Serializer.config.include_data_default = backup
end

include_examples 'lazy loader when include_data option is set'
end

context 'include_data disabled locally with custom finder' do
let(:level0_serializer_class) do
module Serializer15
class BlogPost2Serializer < BaseTestSerializer
lazy_belongs_to :category do |serializer|
include_data :if_sideloaded
-> { serializer.lazy_category }
end
end

class User2Serializer < BaseTestSerializer
lazy_has_many :blog_posts, serializer: BlogPost2Serializer
end
end

Serializer15::User2Serializer
end

include_examples 'lazy loader when include_data option is set'
end

context 'include_data disabled locally without custom finder' do
let(:level0_serializer_class) do
module Serializer15
class BlogPost3Serializer < BaseTestSerializer
lazy_belongs_to :category do
include_data :if_sideloaded
end
end

class User3Serializer < BaseTestSerializer
lazy_has_many :blog_posts, serializer: BlogPost3Serializer
end
end

Serializer15::User3Serializer
end

include_examples 'lazy loader when include_data option is set'
end
end
end