From owner-freebsd-stable@FreeBSD.ORG Fri Dec 13 20:57:50 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62D10DB6 for ; Fri, 13 Dec 2013 20:57:50 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 39AEF1CCD for ; Fri, 13 Dec 2013 20:57:50 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id E3AA2B9DB; Fri, 13 Dec 2013 15:57:48 -0500 (EST) From: John Baldwin To: freebsd-stable@freebsd.org, eugene@imedia.ru Subject: Re: 9.2-RELEASE + memcached = page fault Date: Fri, 13 Dec 2013 15:54:22 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <201312091924.11590@badger> <20131209180532.GA24975@x2.osted.lan> <201312101515.34305@badger> In-Reply-To: <201312101515.34305@badger> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201312131554.22671.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 13 Dec 2013 15:57:49 -0500 (EST) Cc: Peter Holm X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Dec 2013 20:57:50 -0000 On Tuesday, December 10, 2013 6:15:34 am Eugene Mitrofanov wrote: > Hello > > On Monday 09 December 2013, Peter Holm wrote: > > On Mon, Dec 09, 2013 at 07:24:11PM +0400, Eugene Mitrofanov wrote: > > > Hello > > > > > > after upgrading to 9.2-release i can see periodical page faults. advises welcome. > > > > > > examples: > > > > > > 1. > > > > > > Fatal trap 12: page fault while in kernel mode > > > cpuid = 5; apic id = 22 > > > fault virtual address = 0x18 > > > fault code = supervisor read data, page not present > > > instruction pointer = 0x20:0xffffffff8051095c > > > stack pointer = 0x28:0xffffff88a9d77800 > > > frame pointer = 0x28:0xffffff88a9d77810 > > > code segment = base 0x0, limit 0xfffff, type 0x1b > > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > > processor eflags = interrupt enabled, resume, IOPL = 0 > > > current process = 1704 (memcached) > > > trap number = 12 > > > panic: page fault > > > cpuid = 5 > > > Uptime: 2d23h33m33s > > > > > > 2. > > > > > > Unread portion of the kernel message buffer: > > > stack pointer = 0x28:0xffffff88a9b70800 > > > frame pointer = 0x28:0xffffff88a9b70810 > > > code segment = base 0x0, limit 0xfffff, type 0x1b > > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > > processor eflags = interrupt enabled, resume, IOPL = 0 > > > current process = 2023 (memcached) > > > trap number = 12 > > > panic: page fault > > > cpuid = 4 > > > Uptime: 1d10h59m7s > > > > > > good luck > > > > It would be nice with a stack trace. Check out: > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers- handbook/kerneldebug-online-ddb.html > > > > Yes of course. You may find it below. > > 1st: > > #0 doadump (textdump=) at pcpu.h:234 > 234 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=) at pcpu.h:234 > #1 0xffffffff80523556 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:449 > #2 0xffffffff80523a3c in panic (fmt=0x1
) > at /usr/src/sys/kern/kern_shutdown.c:637 > #3 0xffffffff80792b9d in trap_fatal (frame=0xfffffe003e20d000, > eva=) at /usr/src/sys/amd64/amd64/trap.c:879 > #4 0xffffffff80792eb8 in trap_pfault (frame=0xffffff88a9d77750, usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:795 > #5 0xffffffff8079323c in trap (frame=0xffffff88a9d77750) > at /usr/src/sys/amd64/amd64/trap.c:463 > #6 0xffffffff8077cb5f in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:232 > #7 0xffffffff8051095c in _mtx_trylock (m=0x0, opts=0, > file=, line=) > at /usr/src/sys/kern/kern_mutex.c:295 > #8 0xffffffff8076cbae in vm_page_cowsetup (m=0xfffffe08529ecc08) > at /usr/src/sys/vm/vm_page.c:2849 > #9 0xffffffff8058465c in socow_setup (m0=0xfffffe02997b6100, > uio=0xffffff88a9d779e0) at /usr/src/sys/kern/uipc_cow.c:130 > #10 0xffffffff80592465 in sosend_copyin (uio=0xffffff88a9d779e0, > retmp=0xffffff88a9d77960, atomic=0, space=0xffffff88a9d77970, flags=0) > at /usr/src/sys/kern/uipc_socket.c:991 > #11 0xffffffff80592798 in sosend_generic (so=0xfffffe00575abd48, > addr=0xfffffe013f50d950, uio=0xffffff88a9d779e0, top=0x0, control=0x0, > flags=0, td=0xfffffe003e238920) at /usr/src/sys/kern/uipc_socket.c:1329 > #12 0xffffffff805946e3 in kern_sendit (td=0xfffffe003e238920, s=32, > mp=0xffffff88a9d77ab0, flags=0, control=0x0, segflg=) > at /usr/src/sys/kern/uipc_syscalls.c:811 > #13 0xffffffff8059499c in sendit (td=0xfffffe003e238920, s=32, > mp=0xffffff88a9d77ab0, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:739 > #14 0xffffffff80594a27 in sys_sendmsg (td=0xfffffe003e238920, > uap=0xffffff88a9d77bb0) at /usr/src/sys/kern/uipc_syscalls.c:944 > #15 0xffffffff80792490 in amd64_syscall (td=0xfffffe003e238920, traced=0) > at subr_syscall.c:135 > #16 0xffffffff8077ce47 in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:391 > #17 0x0000000800d3fe2c in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > 2nd: > > #0 doadump (textdump=) at pcpu.h:234 > 234 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=) at pcpu.h:234 > #1 0xffffffff80523556 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:449 > #2 0xffffffff80523a3c in panic (fmt=0x1
) > at /usr/src/sys/kern/kern_shutdown.c:637 > #3 0xffffffff80792b9d in trap_fatal (frame=0xfffffe0010664950, > eva=) at /usr/src/sys/amd64/amd64/trap.c:879 > #4 0xffffffff80792eb8 in trap_pfault (frame=0xffffff88a9b70750, usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:795 > #5 0xffffffff8079323c in trap (frame=0xffffff88a9b70750) > at /usr/src/sys/amd64/amd64/trap.c:463 > #6 0xffffffff8077cb5f in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:232 > #7 0xffffffff8051095c in _mtx_trylock (m=0x0, opts=0, > file=, line=) > at /usr/src/sys/kern/kern_mutex.c:295 > #8 0xffffffff8076cbae in vm_page_cowsetup (m=0xfffffe0850e13900) > at /usr/src/sys/vm/vm_page.c:2849 > #9 0xffffffff8058465c in socow_setup (m0=0xfffffe00543fe300, > uio=0xffffff88a9b709e0) at /usr/src/sys/kern/uipc_cow.c:130 > #10 0xffffffff80592465 in sosend_copyin (uio=0xffffff88a9b709e0, > retmp=0xffffff88a9b70960, atomic=0, space=0xffffff88a9b70970, flags=0) > at /usr/src/sys/kern/uipc_socket.c:991 > #11 0xffffffff80592798 in sosend_generic (so=0xfffffe02ad39caa0, > addr=0xfffffe02a3fda520, uio=0xffffff88a9b709e0, top=0x0, control=0x0, > flags=0, td=0xfffffe013301a000) at /usr/src/sys/kern/uipc_socket.c:1329 > #12 0xffffffff805946e3 in kern_sendit (td=0xfffffe013301a000, s=21, > mp=0xffffff88a9b70ab0, flags=0, control=0x0, segflg=) > at /usr/src/sys/kern/uipc_syscalls.c:811 > #13 0xffffffff8059499c in sendit (td=0xfffffe013301a000, s=21, > mp=0xffffff88a9b70ab0, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:739 > #14 0xffffffff80594a27 in sys_sendmsg (td=0xfffffe013301a000, > uap=0xffffff88a9b70bb0) at /usr/src/sys/kern/uipc_syscalls.c:944 > #15 0xffffffff80792490 in amd64_syscall (td=0xfffffe013301a000, traced=0) > at subr_syscall.c:135 > #16 0xffffffff8077ce47 in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:391 > #17 0x0000000800d3fe2c in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) Do you have ZEROCOPY_SOCKETS enabled? If so, turn it off. -- John Baldwin