From owner-freebsd-bugs@FreeBSD.ORG Fri Dec 26 15:50:03 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A27161065674 for ; Fri, 26 Dec 2008 15:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 80D628FC2A for ; Fri, 26 Dec 2008 15:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mBQFo3ef038174 for ; Fri, 26 Dec 2008 15:50:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mBQFo3FN038173; Fri, 26 Dec 2008 15:50:03 GMT (envelope-from gnats) Resent-Date: Fri, 26 Dec 2008 15:50:03 GMT Resent-Message-Id: <200812261550.mBQFo3FN038173@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, pluknet Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BEE1106564A for ; Fri, 26 Dec 2008 15:47:05 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 3C53D8FC0C for ; Fri, 26 Dec 2008 15:47:05 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id mBQFl59E077188 for ; Fri, 26 Dec 2008 15:47:05 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id mBQFl4ew077187; Fri, 26 Dec 2008 15:47:04 GMT (envelope-from nobody) Message-Id: <200812261547.mBQFl4ew077187@www.freebsd.org> Date: Fri, 26 Dec 2008 15:47:04 GMT From: pluknet To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/129956: Threadened process stuck in "vmopar" state, other in "ufs" later. X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Dec 2008 15:50:03 -0000 >Number: 129956 >Category: kern >Synopsis: Threadened process stuck in "vmopar" state, other in "ufs" later. >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Dec 26 15:50:03 UTC 2008 >Closed-Date: >Last-Modified: >Originator: pluknet >Release: 6.4-RELEASE-p1 i386 >Organization: >Environment: All seen on FreeBSD 6.4-RELEASE-p1, 6.3-RELEASE-p7, 6.2-RELEASE-p8 kernels built on 6.2-RELEASE-p8 world, i386. >Description: Custom kernel with options QUOTA enabled. Problem seen with Apache Server version: Apache/2.2.11 (Unix) built from sources: It was created by configure, which was generated by GNU Autoconf 2.63. Invocation command line was $ ./configure --prefix=/home/aaa301/mybin/ --enable-mods-shared=all --with-mpm =worker Apache uses mod_php, which in turn uses xcache module tuned on in php.ini. >From php.ini: '======== extension="xcache.so" xcache.size=64M xcache.count=8 xcache.slot=64K xcache.var_size=64M xcache.var_count=8 xcache.var_slots=64K xcache.mmap_path=/tmp/xcache '=========== [Pay your attention, xcache uses mmap(), see below]. Apache is running on UFS2 under normal user permissions (aaa301 here) with his exceed hard and soft quotas. Disk quotas for user aaa301 (uid 1000): Filesystem usage quota limit grace files quota limit grace /home 890120* 532480 532480 7days 56233 0 0 After issuing the `kill -9 2313` command the process became stuck in vmopar state forever. aaa301 2313 0.0 0.0 0 8 ?? DE 3:10PM 0:00.01 /home/aaa301/myb vmopar A bit later various processes begin to stuck in "ufs" state: quotacheck, quota, sync, and many others. [root ~]# sync load: 0.25 cmd: sync 1315 [ufs] 0.00u 0.00s 0% 464k backtrace of process that stucks in vmopar: db> bt 1385 Tracing pid 1385 tid 100181 td 0xc6c19960 sched_switch(c6c19960,0,1) at sched_switch+0x15b mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c2954ec8,c0a3d0a0,0,c094c4eb,211,...) at sleepq_switch+0xc1 sleepq_wait(c2954ec8,0,c096897c,709,c096897c,...) at sleepq_wait+0x46 msleep(c2954ec8,c0ab8e80,244,c0968ca9,0,c9030444,0,c0968eb0,200,c2954ec8,82) at msleep+0x279 vm_page_sleep_if_busy(c2954ec8,1,c0968ca9) at vm_page_sleep_if_busy+0x7c vm_object_page_remove(c9030444,4,0,8000,0,0) at vm_object_page_remove+0xf9 vnode_pager_setsize(c903c000,4000,0) at vnode_pager_setsize+0xbd ffs_write(f734a78c) at ffs_write+0x264 VOP_WRITE_APV(c0a09b00,f734a78c) at VOP_WRITE_APV+0x112 vnode_pager_generic_putpages(c903c000,f734a8d0,9000,5,f734a860,...) at vnode_pag er_generic_putpages+0x1ef vop_stdputpages(f734a814) at vop_stdputpages+0x1a VOP_PUTPAGES_APV(c0a09b00,f734a814) at VOP_PUTPAGES_APV+0x8c vnode_pager_putpages(c9030444,f734a8d0,9,5,f734a860) at vnode_pager_putpages+0x7 e vm_pageout_flush(f734a8d0,9,5,0,0,...) at vm_pageout_flush+0x112 vm_object_page_collect_flush(c9030444,c29505a8,251,5,4a,...) at vm_object_page_c ollect_flush+0x2a0 vm_object_page_clean(c9030444,0,0,0,0,...) at vm_object_page_clean+0x184 vm_object_terminate(c9030444) at vm_object_terminate+0x60 vnode_destroy_vobject(c903c000,c6973500,f734aab8,c6c19960,0,...) at vnode_destro y_vobject+0x39 ufs_reclaim(f734aab8) at ufs_reclaim+0x46 VOP_RECLAIM_APV(c0a09b00,f734aab8) at VOP_RECLAIM_APV+0x7e vgonel(c903c000) at vgonel+0x12d vrecycle(c903c000,c6c19960) at vrecycle+0x38 ufs_inactive(f734ab40) at ufs_inactive+0x2af VOP_INACTIVE_APV(c0a09b00,f734ab40) at VOP_INACTIVE_APV+0x7e vinactive(c903c000,c6c19960) at vinactive+0x72 vrele(c903c000,c9030444,0,c096897c,1a2,...) at vrele+0x14a vm_object_vndeallocate(c9030444) at vm_object_vndeallocate+0xc0 vm_object_deallocate(c9030444,c9030444,0,c0968016,8e7) at vm_object_deallocate+0 xb3 vm_map_entry_delete(c722a000,c7194bf4,f734ac20,c081ca37,c722a000,...) at vm_map_ entry_delete+0x130 vm_map_delete(c722a000,0,bfc00000) at vm_map_delete+0x18f vmspace_exit(c6c19960,c0a4bde0,0,c09463ea,125,...) at vmspace_exit+0xd5 exit1(c6c19960,9,2831a4b4,c6c19960,c7234000,...) at exit1+0x496 sigexit(c6c19960,9,c7234aa8,0,c09499bc,...) at sigexit+0xdf postsig(9) at postsig+0x160 ast(f734ad38) at ast+0x35e doreti_ast() at doreti_ast+0x17 db> show alllock Process 1385 (httpd) thread 0xc6c19960 (100181) exclusive sx user map r = 0 (0xc722a044) locked @ /usr/src/sys_uvmem_uip.6.2_RELEASE/vm/vm_map.c:307 >How-To-Repeat: Start apache and try to kill -9 all httpd processes. >Fix: Perhaps the problem is in mmap vs threads interaction. Switching off use of mmap() by xcache in php.ini workarounds the problem. >Release-Note: >Audit-Trail: >Unformatted: