From owner-freebsd-ruby@FreeBSD.ORG Sat Apr 11 03:18:34 2015 Return-Path: Delivered-To: ruby@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F73FEB5 for ; Sat, 11 Apr 2015 03:18:34 +0000 (UTC) Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D53DAF92 for ; Sat, 11 Apr 2015 03:18:33 +0000 (UTC) Received: by wiax7 with SMTP id x7so27461456wia.0 for ; Fri, 10 Apr 2015 20:18:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to:cc :content-type; bh=h9s6giee5IJX0pteAT0TntX2ehK8P+EqQwAly5uJ8b0=; b=LGmEZNHglc1J0bgJBauZj1INpJLKoAbSPtPuWm9/MfRDBspTTrXcVZnl+Fgo1Dg21V Cw8dXOlN9KtpyrUb3dbE8D+7ZDzylrLZlT3wkrtpJTZP+zJEWPUEYOD3f35jH0OmwlAk xNzpZ9niu/rt4Q2cP4BgV6546GLmohrX47/oM9xvLmRC+urjrLKvPyCipYyaT0zGP4Li z6qw+zhqIOp9gxtkiMwmFWMz/0ji3APe9nDbXNFxSDhgDqpU+tFtxVJ9W1chHNOJgoi2 zslWzyxgVgTf+3kuXNBHQVm/mlLlVbSzLTUTV02/0U46jm5NFaPEwdsM6L68655GZarr I06A== X-Gm-Message-State: ALoCoQn+/bx1rV8sQOEbSrVvdNcnWjG4XMMtG+Iopttkca89BAuIDRYroTDtGnkwdigGc+GHqK2W MIME-Version: 1.0 X-Received: by 10.180.87.101 with SMTP id w5mr2897045wiz.18.1428722306607; Fri, 10 Apr 2015 20:18:26 -0700 (PDT) Received: by 10.194.176.131 with HTTP; Fri, 10 Apr 2015 20:18:26 -0700 (PDT) Date: Fri, 10 Apr 2015 23:18:26 -0400 Message-ID: Subject: OPENSSL_ia32cap is my worst night mare From: Philip Gollucci To: ruby@freebsd.org, ports@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Matthew Vincenz , Scott Sanders , Stephen Garlick X-BeenThere: freebsd-ruby@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD-specific Ruby discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2015 03:18:34 -0000 All, This definitely needs an official bug report, I'm just not sure where. So for those of you that don't know I work at a ruby on rails shop and we also use AWS. We're just finishing our 10.x update. So take it as a given all is well on 9.x. - Spawn cperciva official AMI for FreeBSD 10.1-RELEASE - Build and install ruby with ruby-install (gcc or clang). As far as I can tell any version will do, but we're using 1.9.3-p551, 2.1.4, 2.2.1 - deploy a ruby on rails app with bundler/unicorn. any version of bundler/unicorn seems to cause it. - During unicorn start if you hook lib/libc/stdlib/getenv.c [patched file attached]. You'll see an ENV request for OPENSSL_ia32cap. This CORRUPTS the ENV. SMACK MY FACE AND PUNCH ME THEN HURT ME THEN CRY. - Once corrupted the next thing in *that* process/sub-process to SET an ENV var or rebuild the ENV will trigger it. static const char CorruptEnvFindMsg[] = "environment corrupt; unable to > find "; which obviously causes the process to exit. So as far as I can tell this is 100% reproducible. I must be the only person running rails with unicorn in aws on fbsd. So the good news is if you do > export OPENSSL_ia32cap=' '; You can hack around it. I will provide any other information on demand. I'll even make you an AMI if you want. Do note, that I can do a full buildworld, kernel, install, run chef, install pkgs, run ruby and nothing breaks. $ sudo kill -USR2 `cat pid of unicorn master` is a good way to cause it. The error msg is very misleading, I promise you BUNDLE_BIN_PATH has nothing to do with it which is the ENV its trying to dick with but its CORRUPT. If you apply my getenv.c file you'll see pointers/memory go everywhere. reaped # worker=unknown > executing ["/u/apps/rc/shared/vendor_bundle/ruby/1.9.1/bin/unicorn", "-E", > "dev", "-c", "/usr/local/etc/unicorn/rc.rb", "-D", > {23=>#}] (in /u/apps/rc/releases/20150405023358) > ruby: environment corrupt; missing value for BUNDLE_B~A^B > /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler/gem_path_manipulation.rb:5:in > `[]=': Bad address - setenv (Errno::EFAULT) > from > /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler/gem_path_manipulation.rb:5:in > `preserve_gem_path' > from > /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler.rb:12:in > `' > from > /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler.rb:11:in > `' > from > /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in > `require' > from > /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in > `require' > from > /opt/rubies/1.9.3-p551/lib/ruby/gems/1.9.1/gems/bundler-1.9.2/lib/bundler/setup.rb:4:in > `' > from > /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in > `require' > from > /opt/rubies/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in > `require' That *only* took 5 8 hour days of pain. -- *PHILIP M. GOLLUCCI* Sr. Director of IT | *Curb* *w.* 703-579-6947 *m.* 703-336-9354 @gocurb Enter code* p6magic* for $10 off your first ride