From owner-freebsd-stable@FreeBSD.ORG Wed Oct 21 10:59:08 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB8BD106566C for ; Wed, 21 Oct 2009 10:59:08 +0000 (UTC) (envelope-from sergey.aleynikov@gmail.com) Received: from mail-ew0-f209.google.com (mail-ew0-f209.google.com [209.85.219.209]) by mx1.freebsd.org (Postfix) with ESMTP id 37B748FC26 for ; Wed, 21 Oct 2009 10:59:08 +0000 (UTC) Received: by ewy5 with SMTP id 5so3157614ewy.36 for ; Wed, 21 Oct 2009 03:59:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=RDBssILwmVczv4Q/GMfts60+me4SZUWmqgvpcGv9omE=; b=SL+vufVwOXLKPcaxJXTCGcrOt48cOUNIVi9VBbKHb8ztRHZyosX2jzFLHnwMb4Bifk hzyrfL0/C2uw+n7Iz0GWPu4uqm9h6RrbEDMnGbOPnCnv5D0xArN4jlykhXHjNatuy2JF cr7xxT1tUyCgwEiNh9RG2ZdpuEFl5D9QnUlcc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=AxTqu/qelvvNsF48bsisXXWLV8RAbrxicTVjGgDwWUxhjukyz4M3/4Y0QcMKo1ntSn gqOpK3WvS/ob9vo4dfgT4NKKNh7Bt0aPnnSXeCTcbRgjYjmccFzz0g7LW31hm76wPZUP y1JQYMptL+8gA3Kif/xZzxoKpOS98fgyI5JVg= MIME-Version: 1.0 Received: by 10.216.90.203 with SMTP id e53mr2699086wef.86.1256121410030; Wed, 21 Oct 2009 03:36:50 -0700 (PDT) Date: Wed, 21 Oct 2009 19:36:50 +0900 Message-ID: From: Sergey Aleynikov To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Processes stuck in *unp_mtx state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2009 10:59:08 -0000 I'm running FreeBSD 7.1-RELEASE and have application that spawnes some children and (heavily) communicates between them using pipes. But under load parent and one child both get stuck in unp_mtx state, they can't be killed (killing them puts whole system on hang). This is reproducable (i've written client emulation that causes this situation after ~5 min run). Everything else keeps running, but trying to access these stuck porcesses puts whole system on hang, so i have to perform reboot. Panic is not invoked, so i've set up ddb panic autocollecting script and had manually put system into panic. Is this a known issue in 7.1, so upgrade to 7.2 would help? I list below some info about system+ddb output: begom% uname -a FreeBSD begom.com 7.1-RELEASE FreeBSD 7.1-RELEASE #1: Mon Sep 14 07:37:56 MSD 2009 inferno@begom.com:/usr/obj/usr/src/release-7.1/sys/BEGOM i386 db:0:kdb.enter.panic> show pcpu cpuid = 1 curthread = 0x84d00d20: pid 1 "init" curpcb = 0xa940bd90 fpcurthread = none idlethread = 0x84d00af0: pid 11 "idle: cpu1" APIC ID = 1 currentldt = 0x50 db:0:kdb.enter.panic> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0x850e08c0: pid 1078 "mysqld" curpcb = 0xae5d4d90 fpcurthread = none idlethread = 0x84d008c0: pid 12 "idle: cpu0" APIC ID = 0 currentldt = 0x50 cpuid = 1 curthread = 0x84d00d20: pid 1 "init" curpcb = 0xa940bd90 fpcurthread = none idlethread = 0x84d00af0: pid 11 "idle: cpu1" APIC ID = 1 currentldt = 0x50 db:0:kdb.enter.panic> ps pid ppid pgrp uid state wmesg wchan cmd ... 89782 89764 89764 1013 L *unp_mtx 0x850b9b40 perl5.8.8 89764 1 89764 1013 Ls *unp_mtx 0x856a70f0 perl5.8.8 db:0:kdb.enter.panic> alltrace ... Tracing command perl5.8.8 pid 89764 tid 100434 td 0x85672690 sched_switch(85672690,0,1,23f76982,201f8b,...) at sched_switch+0x43b mi_switch(1,0,80831849,2d9,85bf2000,...) at mi_switch+0x146 turnstile_wait(856a70f0,85bf2000,0,8675d690,87087690,...) at turnstile_wait+0x2ec _mtx_lock_sleep(8675d720,85672690,0,0,0,...) at _mtx_lock_sleep+0x10e uipc_peeraddr(8bb431a0,ae6cac70,85672690,ae6cac80,86b74b94,...) at uipc_peeraddr+0xc5 kern_getpeername(85672690,11,ae6cac70,ae6cac6c,100,...) at kern_getpeername+0x60 getpeername(85672690,ae6cacfc,c,16,ae6cad2c,...) at getpeername+0x52 syscall(ae6cad38) at syscall+0x335 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (31, FreeBSD ELF32, getpeername), eip = 0x282b88db, esp = 0x7fbfeb5c, ebp = 0x7fbfeba8 --- Tracing command perl5.8.8 pid 89782 tid 100248 td 0x85bf2000 sched_switch(85bf2000,0,1,23f73c9c,201f8b,...) at sched_switch+0x43b mi_switch(1,0,80831849,2d9,805d7233,...) at mi_switch+0x146 turnstile_wait(850b9b40,85672690,0,87087690,8675d690,...) at turnstile_wait+0x2ec _mtx_lock_sleep(87087720,85bf2000,0,0,0,...) at _mtx_lock_sleep+0x10e uipc_peeraddr(8f1a8d00,ae3f4c70,ae3f4c44,4aded50c,84ff5da8,...) at uipc_peeraddr+0xc5 kern_getpeername(85bf2000,6,ae3f4c70,ae3f4c6c,100,...) at kern_getpeername+0x60 getpeername(85bf2000,ae3f4cfc,c,16,ae3f4d2c,...) at getpeername+0x52 syscall(ae3f4d38) at syscall+0x335 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (31, FreeBSD ELF32, getpeername), eip = 0x282b88db, esp = 0x7fbfeb5c, ebp = 0x7fbfeba8 --- db:0:kdb.enter.panic> show locks No such command db:0:kdb.enter.panic> show alllocks No such command db:0:kdb.enter.panic> show lockedvnods Locked vnodes options CONFIG_AUTOGENERATED ident BEGOM machine i386 cpu I686_CPU makeoptions DEBUG=-g options ATA_STATIC_ID options SMP options AUDIT options STOP_NMI options ADAPTIVE_GIANT options KBD_INSTALL_CDEV options _KPOSIX_PRIORITY_SCHEDULING options SYSVSEM options SYSVMSG options SYSVSHM options KTRACE options SCSI_DELAY=5000 options COMPAT_FREEBSD6 options COMPAT_FREEBSD5 options COMPAT_FREEBSD4 options COMPAT_43TTY options GEOM_LABEL options GEOM_PART_GPT options PSEUDOFS options PROCFS options CD9660 options MD_ROOT options UFS_GJOURNAL options UFS_DIRHASH options UFS_ACL options SOFTUPDATES options FFS options INET options PREEMPTION options SCHED_ULE options KVA_PAGES=512 options DEVICE_POLLING options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL options INCLUDE_CONFIG_FILE options ALT_BREAK_TO_DEBUGGER options KDB_UNATTENDED options DDB options KDB options KSE options GEOM_MBR options GEOM_BSD options ISAPNP Loaded kernel modules are: linux, acpi, aio (but i've seen this both with and without aio). (part from fstat) game perl5.8.8 89764 root / 2 drwxr-xr-x 512 r game perl5.8.8 89764 wd /home 15191040 drwxr-x--x 3072 r game perl5.8.8 89764 text /usr 615835 -rwxr-xr-x 29935 r game perl5.8.8 89764 0 /dev 117 crw--w---- ttyp1 rw game perl5.8.8 89764 1 /dev 117 crw--w---- ttyp1 rw game perl5.8.8 89764 2 /dev 117 crw--w---- ttyp1 rw game perl5.8.8 89764 3* local dgram 86a8e3f0 <-> 852dc0a8 game perl5.8.8 89764 4 /var 329768 -rw-r--r-- 6 rw game perl5.8.8 89764 5 /usr 733593 -r--r--r-- 6611 r game perl5.8.8 89764 7* local stream 8678cc78 <-> 8579bc78 game perl5.8.8 89764 8* local stream 8579b000 <-> 8579bbd0 game perl5.8.8 89764 9* local stream 8678c000 <-> 86e07150 game perl5.8.8 89764 10* local stream 86e07888 <-> 86e07d20 game perl5.8.8 89764 11* local stream 865ca0a8 <-> 875d5a80 game perl5.8.8 89764 12* local stream 8678c0a8 <-> 865ca738 game perl5.8.8 89764 13* local stream 8736b540 <-> 8678c690 game perl5.8.8 89764 14* local stream 8579bdc8 <-> 8736bbd0 game perl5.8.8 89764 15* local stream 85a5a498 <-> 875d5498 game perl5.8.8 89764 16* local stream 85c2a3f0 <-> 85c2a348 game perl5.8.8 89764 17* local stream 87087690 <-> 8675d690 game perl5.8.8 89764 18* local stream 85a5a888 <-> 865ca2a0 game perl5.8.8 89764 19* local stream 852c79d8 <-> 8528e3f0 game perl5.8.8 89764 20* internet stream tcp 8d7d9570 game perl5.8.8 89764 21* internet stream tcp 85e021d0 game perl5.8.8 89764 22* internet stream tcp 899f71d0 game perl5.8.8 89764 23* internet stream tcp 8e8bd1d0 game perl5.8.8 89764 24* internet stream tcp 8c137000 game perl5.8.8 89764 25* internet stream tcp 8d39b3a0 game perl5.8.8 89764 26* internet stream tcp 85bc0000 game perl5.8.8 89764 27* internet stream tcp 8c78e570 game perl5.8.8 89764 28* internet stream tcp 8a7fa910 game perl5.8.8 89764 29* internet stream tcp 8dd6bcb0 game perl5.8.8 89764 30* internet stream tcp 8e801570 game perl5.8.8 89764 31* internet stream tcp 86635cb0 game perl5.8.8 89764 32* internet stream tcp 8cd50740 game perl5.8.8 89764 33* internet stream tcp 86656000