From owner-freebsd-current@freebsd.org Fri Mar 4 12:53:36 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37E4B9DAD65 for ; Fri, 4 Mar 2016 12:53:36 +0000 (UTC) (envelope-from satan@ukr.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 28AE5EBC for ; Fri, 4 Mar 2016 12:53:36 +0000 (UTC) (envelope-from satan@ukr.net) Received: by mailman.ysv.freebsd.org (Postfix) id 251639DAD64; Fri, 4 Mar 2016 12:53:36 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0ABB29DAD61 for ; Fri, 4 Mar 2016 12:53:36 +0000 (UTC) (envelope-from satan@ukr.net) Received: from hell.ukr.net (hell.ukr.net [212.42.67.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C51D0EB6 for ; Fri, 4 Mar 2016 12:53:35 +0000 (UTC) (envelope-from satan@ukr.net) Received: from satan by hell.ukr.net with local ID 1abp2M-0006Y7-2B for current@freebsd.org; Fri, 04 Mar 2016 14:40:54 +0200 Date: Fri, 4 Mar 2016 14:40:54 +0200 From: Vitalij Satanivskij To: current@freebsd.org Subject: CURRENT r296381 panic in vn_sendfile (/usr/src/sys/kern/kern_sendfile.c:833) Message-ID: <20160304124053.GA25071@hell.ukr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 12:53:36 -0000 Hello. I get kernel panic on high loaded server with messages savecore: reboot after panic: vn_sendfile: mlen 326 space -20 hdrlen 326 # kgdb kernel.debug /var/crash/vmcore.0 Unread portion of the kernel message buffer: panic: vn_sendfile: mlen 326 space -20 hdrlen 326 cpuid = 5 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe20206314f0 vpanic() at vpanic+0x182/frame 0xfffffe2020631570 kassert_panic() at kassert_panic+0x126/frame 0xfffffe20206315e0 vn_sendfile() at vn_sendfile+0x14ca/frame 0xfffffe2020631900 sys_sendfile() at sys_sendfile+0x11e/frame 0xfffffe20206319a0 amd64_syscall() at amd64_syscall+0x2db/frame 0xfffffe2020631ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2020631ab0 --- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x801ef062a, rsp = 0x7fffffffd8d8, rbp = 0x7fffffffe1d0 --- KDB: enter: panic Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/carp.ko...Reading symbols from /usr/lib/debug//boot/kernel/carp.ko.debug...done. done. Loaded symbols for /boot/kernel/carp.ko Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done. done. Loaded symbols for /boot/kernel/ums.ko Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done. done. Loaded symbols for /boot/kernel/tmpfs.ko #0 doadump (textdump=0) at pcpu.h:221 221 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) bt #0 doadump (textdump=0) at pcpu.h:221 #1 0xffffffff80384a0b in db_dump (dummy=, dummy2=false, dummy3=0, dummy4=0x0) at /usr/src/sys/ddb/db_command.c:533 #2 0xffffffff803847fe in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440 #3 0xffffffff80384594 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 #4 0xffffffff8038702b in db_trap (type=, code=0) at /usr/src/sys/ddb/db_main.c:251 #5 0xffffffff80a656e3 in kdb_trap (type=3, code=0, tf=) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff80ea1298 in trap (frame=0xfffffe2020631420) at /usr/src/sys/amd64/amd64/trap.c:556 #7 0xffffffff80e81a77 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:234 #8 0xffffffff80a64dcb in kdb_enter (why=0xffffffff813b6c2f "panic", msg=0x80
) at cpufunc.h:63 #9 0xffffffff80a27b5f in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:750 #10 0xffffffff80a279b6 in kassert_panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:647 #11 0xffffffff80a25efa in vn_sendfile (fp=, sockfd=1619, hdr_uio=, trl_uio=0x0, offset=0, nbytes=, sent=, flags=, kflags=, td=0xa8) at /usr/src/sys/kern/kern_sendfile.c:833 #12 0xffffffff80a2641e in sys_sendfile (td=0xfffff80253593000, uap=0xfffffe2020631a40) at file.h:382 #13 0xffffffff80ea214b in amd64_syscall (td=0xfffff80253593000, traced=0) at subr_syscall.c:135 #14 0xffffffff80e81d5b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:394 #15 0x0000000801ef062a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) list *0xffffffff80a25efa 0xffffffff80a25efa is in vn_sendfile (/usr/src/sys/kern/kern_sendfile.c:833). 828 free(sfio, M_TEMP); 829 goto done; 830 } 831 832 /* Add the buffer chain to the socket buffer. */ 833 KASSERT(m_length(m, NULL) == space + hdrlen, 834 ("%s: mlen %u space %d hdrlen %d", 835 __func__, m_length(m, NULL), space, hdrlen)); 836 837 CURVNET_SET(so->so_vnet); System have 128Gb memory zfs as FS DB's worked on it and web pages served by this server. core saved. panic periodicaly repeted (few hours -- up to few days) Before this, old current (about two year old CURRENT ) work on this server without crashes. Can anybody point me to way of more complex problem diagnostic or any other useful things Thank you.