Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make caching easier for non-memory caches. #345

Merged
merged 2 commits into from
Dec 27, 2013
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 41 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,27 +226,63 @@ Typhoeus includes built in support for caching. In the following example, if the

```ruby
class Cache
attr_accessor :memory

def initialize
@memory = {}
end

def get(request)
memory[request]
@memory[request]
end

def set(request, response)
memory[request] = response
@memory[request] = response
end
end

Typhoeus::Config.cache = Cache.new

Typhoeus.get("www.example.com") == Typhoeus.get("www.example.com")
Typhoeus.get("www.example.com").cached?
#=> false
Typhoeus.get("www.example.com").cached?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. In addition to pointing out #cached?, I changed it because it's not actually true for non-memory cachers that serialize and then deserialize the response object.

#=> true
```

For use with [Dalli](https://github.com/mperham/dalli):

```ruby
class Cache
def initialize
@client = Dalli::Client.new
end

def get(request)
@client.get(request.cache_key)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you use cache_key instead of hash? cache_key can be arbitrarily long and nasty. Think unicode :).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While that's true, Dalli handles this internally by truncating and using
MD5 (I think it's MD5 at least). #cache_key is preferable because it is
more unique.

Also take a look at this in terms of using CRC32:
http://stackoverflow.com/questions/14210298/probability-of-collision-crc32

It's fine for a ruby Hash since hash equivalence is not enough for a match,
but for memcached type caching where it's purely key equivalence, that's
pretty terrible. Even just 30k cached URLs would result in a 10%(!) chance
of collision.

On Fri, Dec 27, 2013 at 3:54 PM, Hans Hasselberg
notifications@github.comwrote:

In README.md:

#=> true


+For use with [Dalli](https://github.com/mperham/dalli):
+
+```ruby
+class Cache
+  def initialize
+    @client = Dalli::Client.new
+  end
+
+  def get(request)
+    @client.get(request.cache_key)

Why would you use cache_key instead of hash? cache_key can be arbitrarily
long and nasty. Think unicode :).


Reply to this email directly or view it on GitHubhttps://github.com//pull/345/files#r8572583
.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. Sounds good.

end

def set(request, response)
@client.set(request.cache_key, response)
end
end

Typhoeus::Config.cache = Cache.new
```

For use with Rails:

```ruby
class Cache
def get(request)
Rails.cache.read(request)
end

def set(request, response)
Rails.cache.write(request, response)
end
end

Typhoeus::Config.cache = Cache.new
```

### Direct Stubbing

Hydra allows you to stub out specific urls and patterns to avoid hitting
Expand Down
21 changes: 20 additions & 1 deletion lib/typhoeus/request.rb
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,15 @@ def eql?(other)
#
# @api private
def hash
Zlib.crc32 "#{self.class.name}#{base_url}#{options}"
Zlib.crc32 cache_key
end

# Returns a cache key for use with caching methods that required a string
# for a key. Will get used by ActiveSupport::Cache stores automatically.
#
# @return [ String ] The cache key.
def cache_key
"#{self.class.name}#{base_url}#{hashable_string_for(options)}"
end

# Mimics libcurls POST body generation. This is not accurate, but good
Expand Down Expand Up @@ -187,6 +195,17 @@ def fuzzy_hash_eql?(left, right)
end
end

def hashable_string_for(obj)
case obj
when Hash
hashable_string_for(obj.sort_by {|sub_obj| sub_obj.first.to_s})
when Array
obj.map {|sub_obj| hashable_string_for(sub_obj)}.to_s
else
obj.to_s
end
end

# Sets default header and verbose when turned on.
def set_defaults
if @options[:headers]
Expand Down
30 changes: 26 additions & 4 deletions spec/typhoeus/request_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -143,15 +143,37 @@

describe "#hash" do
context "when request.eql?(other)" do
let(:other) { Typhoeus::Request.new(base_url, options) }
context "when different order" do
let(:other_options) {
{:headers => { 'User-Agent' => "Fubar" }, :verbose => true }
}
let(:other) { Typhoeus::Request.new(base_url, other_options)}

it "has same hashes" do
expect(request.hash).to eq(other.hash)
end
end

it "has same hashes" do
expect(request.hash).to eq(other.hash)
context "when same order" do
let(:other) { Typhoeus::Request.new(base_url, options) }

it "has same hashes" do
expect(request.hash).to eq(other.hash)
end
end

context "when hashes with different orders are contained in arrays" do
let(:request) { Typhoeus::Request.new(base_url, :params => [{:b => 2, :a => 1}]) }
let(:other) { Typhoeus::Request.new(base_url, :params => [{:a => 1, :b => 2}]) }
it "has different hashes" do
expect(request.hash).to eq(other.hash)
end
end
end

context "when not request.eql?(other)" do
let(:other) { Typhoeus::Request.new("base_url", {}) }
let(:request) { Typhoeus::Request.new(base_url, :params => {:foo => 'bar'}) }
let(:other) { Typhoeus::Request.new(base_url, :params => {:foo => 'baz'}) }

it "has different hashes" do
expect(request.hash).to_not eq(other.hash)
Expand Down