Hey @Intrepidd. This actually looks related to #713. Adding a random query string to the URL results in the expected behavior:

.**"text/html; charset=utf-8"
340
."text/html; charset=utf-8"
340
."text/html; charset=utf-8"
340
."text/html; charset=utf-8"
340

Note that I added some extra debugging locally to identify the content-type. Without the randomness in the query string, the content-type is empty.

I'm not sure if this is a Rails or webkit bug. However, I've seen unexpected cache behavior in the browser as well. This makes me think Rails may be misbehaving.

Disabling the browser cache resolves this issue. I need to write some additional tests but I'll commit a fix for this soon.

Thanks!

Fixed in master. Thanks for reproducing this!

Yes ! This fixed so many weird bugs in my specs

Would this be possible to release capybara-webkit 1.4.2 ? It would help a lot as bundler deals weirdly with native extensions when using git

It's me again ! Have you considered releasing 1.4.2 with this fix ?

Yes! There is another pull request that I'm wrapping up at the moment, but once that's out we'll release this. Thanks for your patience.

capybara-webkit 1.5.0 has been released, which includes the fix for this issue.

Thank you for your work ! This tool is becoming greater and greater.

@mhoran I'm still seeing this issue on 1.5.1 for remote sites. How do disable the cache for webkit?

@jaredjenkins there's an update to this, but it's currently unreleased. Can you try installing the master branch?

@jferris and @mhoran - As @jaredjenkins described, I'm still seeing this issue for remote sites (in particular lvh.me which uses the localhost loopback)

Any thoughts on if this is still a legitimate issue? Here's an example.

Thanks!

This is an app that makes heavy use of subdomains (which are managed by the apartment gem).

The request invite page is just a simple form with a name field, email field, and a submit button.

require "spec_helper"

feature "Remote Webkit" do
  around(:each) do |example|
    with_subdomain("TacoCorp") { example.run }
  end

  before(:each) { visit request_invite_path }

  describe "Example with Javascript", js: true do
    it "warns the user when email is not filled out" do
      expect { fill_in :email, with: "" }.not_to raise_error
    end
  end

  describe "Example without Javascript" do
    it "warns the user when email is not filled out" do
      expect { fill_in :email, with: "" }.not_to raise_error
    end
  end
end

The with_subdomain helper just runs any given tests within the context of that subdomain, through use of the remote lvh.me endpoint.

def with_subdomain(subdomain)
  saved_values = [Capybara.app_host, Capybara.always_include_port]
  host = "http://#{subdomain}.lvh.me"

  begin
    set_host(host, true)
    Apartment::Tenant.switch(subdomain) { yield }
  ensure
    set_host(*saved_values)
  end
end

def set_host(host, include_port)
  Capybara.app_host = host
  Capybara.always_include_port = include_port
end

This fails for the JS test

> rspec spec/features/foo_spec.rb
    F

    Failures:

      1) Request an Invite user clicks Request Invite button user has not properly filled out the fields warns the user when email is not filled out
         Failure/Error: expect { fill_in :email, with: "" }.not_to raise_error
           expected no Exception, got #<Capybara::ElementNotFound: Unable to find field :email> with backtrace:
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/capybara-2.4.4/lib/capybara/node/finders.rb:41:in `block in find'
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/capybara-2.4.4/lib/capybara/node/base.rb:84:in `synchronize'
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/capybara-2.4.4/lib/capybara/node/finders.rb:30:in `find'
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/capybara-2.4.4/lib/capybara/node/actions.rb:56:in `fill_in'
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/capybara-2.4.4/lib/capybara/session.rb:676:in `block (2 levels) in <class:Session>'
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/capybara-2.4.4/lib/capybara/dsl.rb:51:in `block (2 levels) in <module:DSL>'
             # ./spec/features/auth/request_invite_spec.rb:29:in `block (5 levels) in <top (required)>'
             # ./spec/features/auth/request_invite_spec.rb:29:in `block (4 levels) in <top (required)>'
             # ./spec/features/auth/request_invite_spec.rb:5:in `block (3 levels) in <top (required)>'
             # ./spec/support/features/subdomain_helpers.rb:9:in `block in with_subdomain'
             # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/apartment-1.0.2/lib/apartment/adapters/abstract_adapter.rb:98:in `switch'
             # ./spec/support/features/subdomain_helpers.rb:9:in `with_subdomain'
             # ./spec/features/auth/request_invite_spec.rb:5:in `block (2 levels) in <top (required)>'
             # ./spec/support/analytics.rb:4:in `block (2 levels) in <top (required)>'
         # ./spec/features/auth/request_invite_spec.rb:29:in `block (4 levels) in <top (required)>'
         # ./spec/features/auth/request_invite_spec.rb:5:in `block (3 levels) in <top (required)>'
         # ./spec/support/features/subdomain_helpers.rb:9:in `block in with_subdomain'
         # /Users/abhchand/.rvm/gems/ruby-2.2.2@gb/gems/apartment-1.0.2/lib/apartment/adapters/abstract_adapter.rb:98:in `switch'
         # ./spec/support/features/subdomain_helpers.rb:9:in `with_subdomain'
         # ./spec/features/auth/request_invite_spec.rb:5:in `block (2 levels) in <top (required)>'
         # ./spec/support/analytics.rb:4:in `block (2 levels) in <top (required)>'

    Finished in 3.36 seconds (files took 4.28 seconds to load)
    1 example, 1 failure

    Failed examples:

    rspec ./spec/features/auth/request_invite_spec.rb:28 # Request an Invite user clicks Request Invite button user has not properly filled out the fields warns the user when email is not filled out

    Randomized with seed 30910

I'm thinking this is likely related to #841 (which is in turn related to this original issue.) However, we added a compatibility spec against Selenium and to the best of my knowledge, our behavior is the same.

@mhoran - Thanks for the quick response.

Wanted to clarify the following -

we added a compatibility spec against Selenium and to the best of my knowledge, our behavior is the same.

That sounds like capybara-webkit has a test suite comparing it to Selenium's functionality, with no differences being observed?

If so is there any fix/workaround for the above behavior? If I remove any dependence on subdomains and just test with the default, all errors go away, so I suspect it's definitely due to the remote host.

Thanks again.

Hey @abhchand (apologies for the late response). You are correct; we have not noted any differences between capybara-webkit and Selenium when it comes to caching behavior.

One workaround would be to add a "cache-bust" query parameter to the URL. This should force a reload, but is not ideal. You may also be able to visit the URL, and I suspect (based on prior experience) that the second attempt may load as expected.

This bug still exists. I'm using a Could you please reopen it?

Reduction: https://github.com/adamhooper/capybara-webkit-bug-724

It seems, in my reduction at least, that the issue is that Capybara-Webkit is sending If-Modified-Since header. My tests take <1s to run, so Rack::Static response with 304.

IMO this is still a bug. Considering its purpose, capybara-webkit ought to figure out the correct body content when it receives a 304, or it ought to avoid 304s by not sending If-Modified-Since. But for those using Rack::Static, there's a workaround. You can use a bit of middleware:

class AvoidRackStatic304Responses
  def initialize(app)
    @app = app
  end

  def call(env)
    env['HTTP_IF_MODIFIED_SINCE'] = nil
    @app.call(env)
  end
end

Then use it in your Rack config, above your use(Rack::Static) lines:

use(AvoidRackStatic304Responses) # https://github.com/thoughtbot/capybara-webkit/issues/724

(As you can see, I'm not using Rails.)

Thanks for digging into this. I was pretty sure it had to do with If-Modified-Since, and we previously attempted to disable the cache because of that. However, our previous approach didn't work, and introduced other bugs. Perhaps we can find another way.

To me, the most obvious fix is to create a new browser instance per test. I'm not sure of whether a 304 response ought to have an empty body or not, but I am sure that the clicks in one test shouldn't affect the next test.

Of course, that'll make things slow. Slow is better than wrong, though. After that come hacky speedups, like the obligatory config.let_all_tests_share_a_global_state_for_speedups_and_breakage = true option (false by default).

To me, the most obvious fix is to create a new browser instance per test.

We create new instances of WebPage, but much of Qt's configuration and caching is stored outside of instance state, so I believe we'd have to start a new process for each test to get a totally clean slate. I'm guessing that would be unacceptable from a performance perspective. Maybe using a new network manager or something else would be enough to clear some of the shared state we're seeing.

It may be possible to use fork after initial configuration is loaded, but I'm not sure if there's a cross-platform solution there.

Shouldn't optimization come second, after implementation?

It looks like QAbstractNetworkCache has a clear() slot. Perhaps we could call that in NetworkAccessManager::reset(). I thought I had tried that before, but perhaps my repro was incorrect.

It's also possible the If-Modified-Since data is stored outside the NetworkAccessManager and restarting the process is the only way around this.

Shouldn't optimization come second, after implementation?

Restarting the process may be so slow that implementation becomes a moot point, because nobody would use the driver. Unfortunately, it's difficult to focus on 100% accuracy in a virtual browser, because the surface area is so huge and it's not always completely clear what "correct" means in various scenarios. The best we've been able to shoot for is reasonable accuracy and reasonable speed.

It looks like QAbstractNetworkCache has a clear() slot. Perhaps we could call that in NetworkAccessManager::reset(). I thought I had tried that before, but perhaps my repro was incorrect.

I think we actually decided not to clear the cache in between tests, because it caused Rails to build assets for every scenario or something?

It's also possible the If-Modified-Since data is stored outside the NetworkAccessManager and restarting the process is the only way around this.

Crazy idea: would we salt the etag or something with a unique ID per test run to make sure caches are discarded for new sessions?

I'll try to put together some more unit tests related to caching this week and see where it goes from there.

I think we actually decided not to clear the cache in between tests, because it caused Rails to build assets for every scenario or something?

We had previously disabled the cache completely, which resulted in issues between steps in a single test run. However, I don't think I ever found a way to only clear the cache at the end of the test (on reset), which should have been fine. With a name like clear(), you'd think I would have come across that method, but maybe I was looking in the wrong place.

Crazy idea: would we salt the etag or something with a unique ID per test run to make sure caches are discarded for new sessions?

I'm not sure salting the etag would resolve the issue. The real trouble is when the browser is reset, the "Last-Modified" time is cached by the NetworkAccessManager. I vaguely remember being unable to clear that value when I tried, but I also didn't have a reliable repro.

I'll try to put together some more unit tests related to caching this week and see where it goes from there.

The repro provided by @adamhooper works great.

I am coming here from #841. I can confirm from my own logs that capybara-webkit is caching redirects between cucumber scenarios:

Capybara: 2.4.4
capybara-webkit: 1.8.0
Qt: 5.4.2
WebKit: 538.1
QtWebKit: 5.4.2

Cleaned webkit_debug output:

Finished "CurrentUrl()" with response "Success(about:blank)"
Wrote response true "about:blank"
    <step/>
Received "Visit(http://127.0.0.1:56746/foobar)"
Started "Visit(http://127.0.0.1:56746/foobar)"
Load started
"Visit(http://127.0.0.1:56746/foobar)" started page load
Finished "Visit(http://127.0.0.1:56746/foobar)" with response "Success()"
Page finished with true
Load finished
Page load from command finished
Wrote response true ""
Received "Evaluate(window.max_concurrent_ajax;)"
Started "Evaluate(window.max_concurrent_ajax;)"
Finished "Evaluate(window.max_concurrent_ajax;)" with response "Success(null)"
Wrote response true "null"
Received "Evaluate(window.client_resized_images;)"
Started "Evaluate(window.client_resized_images;)"
Finished "Evaluate(window.client_resized_images;)" with response "Success(null)"
Wrote response true "null"
Received "CurrentUrl()"
Started "CurrentUrl()"
Finished "CurrentUrl()" with response "Success(http://127.0.0.1:56746/test_for_redirects.html)"
Wrote response true "http://127.0.0.1:56746/test_for_redirects.html"

Here we can see that CurrentUrl changed to test_for_redirects without the browser making a request. The previous scenarios have returned a 302 for the /foobar.

Received "Visit(http://127.0.0.1:56746/foobar)"
Started "Visit(http://127.0.0.1:56746/foobar)"
Load started
"Visit(http://127.0.0.1:56746/foobar)" started page load
Started request to "http://127.0.0.1:56746/foobar"
Finished "Visit(http://127.0.0.1:56746/foobar)" with response "Success()"
Started request to "http://127.0.0.1:56746/test_for_redirects.html"
Received 302 from "http://127.0.0.1:56746/foobar"
Received 200 from "http://127.0.0.1:56746/test_for_redirects.html"
Page finished with true
Load finished
Page load from command finished
Wrote response true ""

Hi guys, thanks for keeping the conversations going here. Does it make sense to provide an option to disable browser caching entirely? In our specs, we sometimes want to test if it runs as expected without browser caching. Currently, I have to roll back to before aeafb90, which is 1.5.2. Thanks!

Seems like even on 1.5.2 with cache entirely disabled, it still caches requests unreliably with Qt5 (sometimes it does, sometimes not). I will see if I can reproduce this issue, but wonder if this is caching behavior is just unreliable with Qt5? Thanks!

Seems like even on 1.5.2 with cache entirely disabled, it still caches requests unreliably with Qt5 (sometimes it does, sometimes not).

QtWebKit definitely has some internal caches that we're not clearing. We haven't figured out yet if those are at the instance level, class level, or process level. We're also not sure if there are public methods for clearing them, or if the caches are within the Qt, QtWebKit, or WebKit layers. It's odd that they cache inconsistently. It could have something to do with the request order or an async issue in how we clear our caches.

I will see if I can reproduce this issue, but wonder if this is caching behavior is just unreliable with Qt5?

I've hard a hard time figuring out the caching behavior, but it seems consistent whenever I identify a particular behavior.

Thanks, @jferris. It's good to know the caching behavior is consistent. I think we can get around with it in the specs given its consistency. Appreciated your response! 👍

We just figured out a workaround for this issue this morning (for rails). We have a single focused test run that is getting a spurious 304s due to etag headers (not last-modified). Adding the following line to config/environments/test.rb removes all etags in our tests and fixes this issue for us.

# inside of your config block
config.middleware.delete Rack::ETag