From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 8 16:43:58 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA8B2BA3; Tue, 8 Apr 2014 16:43:58 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6887A16A1; Tue, 8 Apr 2014 16:43:58 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.8/8.14.8) with ESMTP id s38Ghr9V016934; Tue, 8 Apr 2014 19:43:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua s38Ghr9V016934 Received: (from kostik@localhost) by tom.home (8.14.8/8.14.8/Submit) id s38GhrOL016933; Tue, 8 Apr 2014 19:43:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Apr 2014 19:43:53 +0300 From: Konstantin Belousov To: Karl Pielorz Subject: Re: Stuck CLOSED sockets / sshd / zombies... Message-ID: <20140408164353.GB21331@kib.kiev.ua> References: <3FE645E9723756F22EF901AE@Mail-PC.tdx.co.uk> <201404071148.10157.jhb@freebsd.org> <9647C5438B5CD4A3058AB1A2@Mail-PC.tdx.co.uk> <201404080936.30651.jhb@freebsd.org> <63EFBCBD259A410BB4D71742@Mail-PC.tdx.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XqFDY9bHNWRmuMQr" Content-Disposition: inline In-Reply-To: <63EFBCBD259A410BB4D71742@Mail-PC.tdx.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2014 16:43:58 -0000 --XqFDY9bHNWRmuMQr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Apr 08, 2014 at 04:33:03PM +0100, Karl Pielorz wrote: >=20 >=20 > --On 08 April 2014 09:36 -0400 John Baldwin wrote: >=20 > > Humm, it needs to build new ones with debug symbols. If it doesn't, > > you'll need to do 'make clean' before the other makes. I think you > > should be fine to do that (make clean then the command above) for those > > directories and restart gdb without having to restart your sshd. Please > > also add '/usr/src/libexec/rtld-elf' to the list of directories where y= ou > > do this. Then do 'detach' in gdb, exit gdb and restart it. >=20 > Ok, it hit an issue with libc (complained it couldn't find yp.h - but I= =20 > fixed that) and it compiled up OK then. The rest all recompiled / install= ed=20 > OK, and I included the rtld-elf code. >=20 > > Hmmm, that is useful even though the debug symbols aren't there. Please > > do the rebuilds I asked for above and re-attach gdb and get 'bt' again. >=20 > Ok, that now nets: >=20 > " > ... > [Switching to LWP 100218] > 0x00000008038ea89c in __error () from /lib/libthr.so.3 > (gdb) bt > #0 0x00000008038ea89c in __error () from /lib/libthr.so.3 > #1 0x00000008038e104f in __thr_rwlock_rdlock (rwlock=3D0x803af9480,=20 > flags=3D, tsp=3D) > at /usr/src/lib/libthr/thread/thr_umtx.c:277 > #2 0x00000008038e821c in _thr_rtld_rlock_acquire (lock=3D0x803af9480) at= =20 > atomic.h:143 > #3 0x000000080064f9a2 in rlock_acquire (lock=3D0x80085fe00,=20 > lockstate=3D0x7fffffffba68) at /usr/src/libexec/rtld-elf/rtld_lock.c:197 > #4 0x00000008006498c9 in _rtld_bind (obj=3D0x800662000, reloff=3D13008) = at=20 > /usr/src/libexec/rtld-elf/rtld.c:675 > #5 0x00000008006470cd in _rtld_bind_start () at=20 > /usr/src/libexec/rtld-elf/amd64/rtld_start.S:99 > #6 0x0000000000000246 in ?? () > #7 0x0000000000000000 in ?? () > " The following patch might allow to see the backtrace beyond the binder entry point. You might also have better luck with the gdb from ports. diff --git a/libexec/rtld-elf/amd64/rtld_start.S b/libexec/rtld-elf/amd64/r= tld_start.S index da3d156..54ef468 100644 --- a/libexec/rtld-elf/amd64/rtld_start.S +++ b/libexec/rtld-elf/amd64/rtld_start.S @@ -79,17 +79,39 @@ .globl _rtld_bind_start .type _rtld_bind_start,@function _rtld_bind_start: + .cfi_startproc + .cfi_adjust_cfa_offset 16 subq $8,%rsp + .cfi_adjust_cfa_offset 8 pushfq # Save rflags + .cfi_adjust_cfa_offset 8 pushq %rax # Save %rax + .cfi_adjust_cfa_offset 8 + .cfi_offset %rax,-24 pushq %rdx # Save %rdx + .cfi_adjust_cfa_offset 8 + .cfi_offset %rdx,-32 pushq %rcx # Save %rcx + .cfi_adjust_cfa_offset 8 + .cfi_offset %rcx,-40 pushq %rsi # Save %rsi + .cfi_adjust_cfa_offset 8 + .cfi_offset %rsi,-48 pushq %rdi # Save %rdi + .cfi_adjust_cfa_offset 8 + .cfi_offset %rdi,-56 pushq %r8 # Save %r8 + .cfi_adjust_cfa_offset 8 + .cfi_offset %r8,-64 pushq %r9 # Save %r9 + .cfi_adjust_cfa_offset 8 + .cfi_offset %r9,-72 pushq %r10 # Save %r10 + .cfi_adjust_cfa_offset 8 + .cfi_offset %r10,-80 pushq %r11 # Save %r11 + .cfi_adjust_cfa_offset 8 + .cfi_offset %r11,-88 =20 movq 0x58(%rsp),%rdi # Fetch obj argument movq 0x60(%rsp),%rsi # Fetch reloff argument @@ -101,16 +123,37 @@ _rtld_bind_start: =20 movq %rax,0x60(%rsp) # Store target over reloff argument popq %r11 # Restore %r11 + .cfi_adjust_cfa_offset -8 + .cfi_restore %r11 popq %r10 # Restore %r10 + .cfi_adjust_cfa_offset -8 + .cfi_restore %r10 popq %r9 # Restore %r9 + .cfi_adjust_cfa_offset -8 + .cfi_restore %r9 popq %r8 # Restore %r8 + .cfi_adjust_cfa_offset -8 + .cfi_restore %r8 popq %rdi # Restore %rdi + .cfi_adjust_cfa_offset -8 + .cfi_restore %rdi popq %rsi # Restore %rsi + .cfi_adjust_cfa_offset -8 + .cfi_restore %rsi popq %rcx # Restore %rcx + .cfi_adjust_cfa_offset -8 + .cfi_restore %rcx popq %rdx # Restore %rdx + .cfi_adjust_cfa_offset -8 + .cfi_restore %rdx popq %rax # Restore %rax + .cfi_adjust_cfa_offset -8 + .cfi_restore %rax popfq # Restore rflags + .cfi_adjust_cfa_offset -8 leaq 16(%rsp),%rsp # Discard spare, obj, do not change rflags ret # "Return" to target address + .cfi_endproc + .size _rtld_bind_start, . - _rtld_bind_start =20 .section .note.GNU-stack,"",%progbits --XqFDY9bHNWRmuMQr Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTRCdIAAoJEJDCuSvBvK1BpZoP/iGleVm0BPoTgjShOOtbe0EY kL5RjB7dxt2xP7G8Pajgh5uwt15lhgXSEYcCgMxZnNWVCQHkOdSRV/N45Prq52Xf XJ5ieVdGt88ftoET3VXdHibkZ1pP6X4rKszGaaBhyOjk6SIuxKIgUtCVfXeNH0r/ h5CmR6CgmhcFERWK34GfwrZQquITaAUJgoP9G0nv1pzOnn+ADQ3Cw2jSUS2KFoFa N/Dt5wcwelq0mEKv4ldnqd+5EhKhoqxt7k0r7+6op5z/m9vCMex060j7wHJ6TpBV a81s//OGy7vTXT7GxxCis8UNJJpcC1RKMrCYNL6CAGJqx8gmfKHBS5Kd8Jk00ky4 rcI71vznb8DaGspd8XahB2GKWpdW3SSdtLDDVKFcRTOk82smIjM+78oqwYM1dmsm jz0HkO0zsiZzLFoVRvEyd2+inGNPm7IxYMe2n+QPKvLOeieSxuYRoF7E7K8QJzzj GvXegO036p+wa+O/Eg1kMgchX8lkO7eYHbsvIG9B12JQZyb3U5MlVCsb2p2CtqUH bI7OOug/YkxAy0IK/JPDqLkloNMciKPoAUFgXpRLRnLFYgDIp3aABQIySWbWy4TB ZWOLQAwDcs3ftQ60g4aQkjHU5XmABAfXq/RrVmD/uS2LsW3MyQRY/c3hVoyGVGWA jzB+/2zm0NDLAMX91adm =8Rv4 -----END PGP SIGNATURE----- --XqFDY9bHNWRmuMQr--