Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crashes with "Abort trap 6" for rails + thin site #698

Closed
ansonhoyt opened this issue Mar 15, 2016 · 15 comments
Closed

Crashes with "Abort trap 6" for rails + thin site #698

ansonhoyt opened this issue Mar 15, 2016 · 15 comments
Milestone

Comments

@ansonhoyt
Copy link

After upgrading from eventmachine 1.0.9.1 to 1.2.0, my rails site (running on thin) exits with Abort trap: 6 on the first request from my browser.

I'm not familiar with eventmachine. It's a dependency of thin 1.6.4:

# Gemfile.lock
    thin (1.6.4)
      daemons (~> 1.0, >= 1.0.9)
      eventmachine (~> 1.0, >= 1.0.4)
      rack (~> 1.0)

When I can downgrade to eventmachine 1.0.9.1 and I no longer get an error. I can work around by locking my Gemfile to 1.0.9.1, but letting you know in case this is an eventmachine bug. I'd be happy to add more details, just let me know what would be helpful.

Details

  • Mac OS X El Capitan 10.11.3
  • ruby 2.3.0 via rvm 1.26.11
@ansonhoyt ansonhoyt changed the title Abort trap 6 for Rails site Abort trap 6 for rails + thin site Mar 15, 2016
@ansonhoyt ansonhoyt changed the title Abort trap 6 for rails + thin site Crashes with "Abort trap 6" for rails + thin site Mar 15, 2016
@sodabrew
Copy link
Member

sodabrew commented Mar 15, 2016 via email

@wlipa
Copy link

wlipa commented Mar 15, 2016

It's happening for me with:

Ruby: 2.2.4 (2015-12-16 patchlevel 230) [x86_64-darwin15]
OS X El Capitan 10.11.3

As far as ssl being enabled, I'm not sure, but I didn't take any special action to enable it. I'm just using the default configuration with Rails.

@sodabrew
Copy link
Member

sodabrew commented Mar 15, 2016 via email

@sodabrew
Copy link
Member

I am not able to reproduce this locally with the same versions of OS X El Capitan and Ruby 2.2.4, with a a minimal sinatra + thin + eventmachine application. I can run apachebench for thousands of requests without a problem.

Could you help by looking for a backtrace in the Console application? Look for something like this:

os x console ruby segfault

@thedanbob
Copy link

I'm seeing the same problem. It seems to be triggered specifically by trying to download assets from ::1 rather than 127.0.0.1. For example, I see this in my Guard output:

Started GET "/assets/app.self-1b17a4fe335eed11dc555a6eda47eb5e2f93a4e922a261e4efc8f69a3fe95372.css?body=1" for ::1 at 2016-03-15 13:20:32 -0500
sh: line 1: 85988 Abort trap: 6           rails server -e development --pid "/Users/drarnfield/projects/warehouse/tmp/pids/development.pid" -b localhost -p 3000

I can also reproduce it by running rails server and hitting a page with the Chrome dev tools open (to disable asset caching). A workaround is to start the server with the host specified as 127.0.0.1.

Here's a Console log of the crash: backtrace.txt

Edit: I should probably mention I'm also on OS X 10.11.3, and this project uses ruby 2.1.5

@ansonhoyt
Copy link
Author

I'm not using SSL locally, so I'm opening http://localhost:3000 in Chrome.

Attaching a copy of the last crash from Console:
ruby_2016-03-15-111511_ahoyt-2.crash.zip

@wlipa
Copy link

wlipa commented Mar 15, 2016

Here's the backtrace from my console.

backtrace.txt

@sodabrew
Copy link
Member

IPv6 is the culprit - I was able to reproduce locally. The failure without line numbers is

2   libsystem_c.dylib               0x00007fff9472c787 __abort + 145
3   libsystem_c.dylib               0x00007fff9472d066 __stack_chk_fail + 200
4   rubyeventmachine.bundle         0x00000001036760fa AcceptorDescriptor::Read() + 792
5   rubyeventmachine.bundle         0x00000001036796f3 EventMachine_t::_RunSelectOnce() + 495
6   rubyeventmachine.bundle         0x0000000103678fc4 EventMachine_t::RunOnce() + 120
7   rubyeventmachine.bundle         0x0000000103678f41 EventMachine_t::Run() + 17
8   rubyeventmachine.bundle         0x000000010367fe69 t_run_machine(unsigned long) + 9

Next up I'll need to get line numbers to confirm the exact location, but my hunch is here:

    struct sockaddr_in pin;
    socklen_t addrlen = sizeof (pin);
    int accept_count = EventMachine_t::GetSimultaneousAcceptCount();

    for (int i=0; i < accept_count; i++) {
#if defined(HAVE_CONST_SOCK_CLOEXEC) && defined(HAVE_ACCEPT4)
        SOCKET sd = accept4 (GetSocket(), (struct sockaddr*)&pin, &addrlen, SOCK_CLOEXEC);
        if (sd == INVALID_SOCKET) {
            // We may be running in a kernel where
            // SOCK_CLOEXEC is not supported - fall back:
            sd = accept (GetSocket(), (struct sockaddr*)&pin, &addrlen);
        }
#else
        SOCKET sd = accept (GetSocket(), (struct sockaddr*)&pin, &addrlen);
#endif

At a glance, this looks correct; using sockaddr_in is the right way to fit either IPv4 or IPv6 structures on the stack. I'll need to investigate further as time permits this week.

@sodabrew sodabrew added this to the v1.2.1 milestone Mar 15, 2016
@ansonhoyt
Copy link
Author

Glad you were able to find the (likely) culprit so quickly! Will look forward to 1.2.1 to re-test.

@sodabrew
Copy link
Member

Wait, no it's not the right struct! sockaddr_in only fits an IPv4 address. I should be using either sockaddr_in6 or sockaddr_storage to be sure it is large enough.

I even reminded myself of this a few months ago!
https://twitter.com/sodabrew/status/658413526466170881

Refreshed myself on RFC 2553 today
sizeof(struct sockaddr_in) == 16
sizeof(struct sockaddr_in6) == 28
sizeof(struct sockaddr_storage) == 128

@sodabrew
Copy link
Member

I just confirmed that #699 resolves the problem. I'll push an updated gem to rubygems.org tonight. Test/confirm my results by adding to your Gemfile:

gem 'eventmachine', github: 'sodabrew/eventmachine', ref: '5adf325ef197177547fed2a440ff6ce9c3dc53e1'

Sidenote: need to use sockaddr_in6 because this struct gets memcpy-ied onto the ReturnAddress class member which is presently defined as sockaddr_in6.

@wlipa
Copy link

wlipa commented Mar 15, 2016

Fix worked for me. Thanks!

@thedanbob
Copy link

Same here, thanks!

@ansonhoyt
Copy link
Author

Worked for me too.

@sodabrew
Copy link
Member

EventMachine 1.2.0.1 is posted: https://rubygems.org/gems/eventmachine/versions/1.2.0.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants