Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 10 Apr 2015 23:18:26 -0400
From:      Philip Gollucci <pgollucci@gocurb.com>
To:        ruby@freebsd.org, ports@freebsd.org
Cc:        Matthew Vincenz <mvincenz@taximagic.com>, Scott Sanders <jssanders@gmail.com>, Stephen Garlick <sgarlick@taximagic.com>
Subject:   OPENSSL_ia32cap is my worst night mare
Message-ID:  <CADDV2kcb7828huyhV=vO8P5RMPHr2zC3Uy4qYOw1Y812e38aVA@mail.gmail.com>

next in thread | raw e-mail | index | archive | help
All,

This definitely needs an official bug report, I'm just not sure where.

So for those of you that don't know I work at a ruby on rails shop and we
also use AWS.  We're just finishing our 10.x update.  So take it as a given
all is well on 9.x.

   - Spawn cperciva official AMI for FreeBSD 10.1-RELEASE
   - Build and install ruby with ruby-install (gcc or clang).  As far as I
   can tell any version will do, but we're using 1.9.3-p551, 2.1.4, 2.2.1
   - deploy a ruby on rails app with bundler/unicorn.  any version of
   bundler/unicorn seems to cause it.
   - During unicorn start if you hook lib/libc/stdlib/getenv.c [patched
   file attached].  You'll see an ENV request for OPENSSL_ia32cap.  This
   CORRUPTS the ENV.  SMACK MY FACE AND PUNCH ME THEN HURT ME THEN CRY.
   - Once corrupted the next thing in *that* process/sub-process to SET an
   ENV var or rebuild the ENV will trigger it.

static const char CorruptEnvFindMsg[] = "environment corrupt; unable to
> find ";


which obviously causes the process to exit.

So as far as I can tell this is 100% reproducible.  I must be the only
person running rails with unicorn in aws on fbsd.

So the good news is if you do

> export OPENSSL_ia32cap=' ';

You can hack around it.

I will provide any other information on demand. I'll even make you an AMI
if you want. Do note, that I can do a full buildworld, kernel, install, run
chef, install pkgs, run ruby and nothing breaks.

$ sudo kill -USR2 `cat pid of unicorn master`

is a good way to cause it.

The error msg is very misleading, I promise you BUNDLE_BIN_PATH has nothing
to do with it which is the ENV its trying to dick with but its CORRUPT.

If you apply my getenv.c file you'll see pointers/memory go everywhere.

reaped #<Process::Status: pid 97758 exit 0> worker=unknown
> executing ["/u/apps/rc/shared/vendor_bundle/ruby/1.9.1/bin/unicorn", "-E",
> "dev", "-c", "/usr/local/etc/unicorn/rc.rb", "-D",
> {23=>#<Kgio::TCPServer:fd 23>}] (in /u/apps/rc/releases/20150405023358)
> ruby: environment corrupt; missing value for BUNDLE_B~A^B
> /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler/gem_path_manipulation.rb:5:in
> `[]=': Bad address - setenv (Errno::EFAULT)
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler/gem_path_manipulation.rb:5:in
> `preserve_gem_path'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler.rb:12:in
> `<module:Bundler>'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler.rb:11:in
> `<top (required)>'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in
> `require'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in
> `require'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler/setup.rb:4:in
> `<top (required)>'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in
> `require'
>         from
> /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in
> `require'


That *only* took 5 8 hour days of pain.


-- 
*PHILIP M. GOLLUCCI*
Sr. Director of IT | *Curb*
*w.* 703-579-6947
*m.* 703-336-9354
@gocurb <http://twitter.com/gocurb>;

Enter code* p6magic* for $10 off your first ride



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADDV2kcb7828huyhV=vO8P5RMPHr2zC3Uy4qYOw1Y812e38aVA>