From owner-freebsd-stable@freebsd.org Wed Jul 22 10:56:26 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6ECF9A7477 for ; Wed, 22 Jul 2015 10:56:26 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: from smtp.new-ukraine.org (smtp.new-ukraine.org [148.251.53.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.new-ukraine.org", Issuer "smtp.new-ukraine.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 567061A9C for ; Wed, 22 Jul 2015 10:56:25 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: on behalf of honored client by smtp.new-ukraine.org with ESMTP id t6MArwkU070270 for on Wed, 22 Jul 2015 13:54:00 +0300 (EEST) Message-ID: <20150722135353.70269@smtp.new-ukraine.org> Date: Wed, 22 Jul 2015 13:53:53 +0300 From: "Zeus Panchenko" To: cc: Subject: [panic] 10.1-RELEASE-p14 #4 r285499M sporadic reboot Organization: I.B.S. LLC Reply-To: "Zeus Panchenko" X-Attribution: zeus Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAFVBMVEWxsbGdnZ3U1NQTExN cXFzx8fG/v7+f8hyWAAACXUlEQVQ4jUWSwXYiIRBFi4yyhtjtWpmRdTL0ZC3TJOukDa6Rc+T/P2F eFepwtFvr8upVFVDua8mLWw6La4VIKTuMdAPOebdU55sQs3n/D1xFFPFGVGh4AHKttr5K0bS6g7N ZCge7qpVLB+f1Z2WAj2OKXwIWt/bXpdXSiu8KXbviWkHxF5td9+lg2e3xlI2SCvatK8YLfHyh9lw 15yrad8Va5eXg4Llr7QmAaC+dL9sDt9iad/DX3OKvLMBf+dm0A0QuMrTvYIevSik1IaSVvgjIHt5 lSCG2ynNRpEcBZ8cgDWk+Ns99qzsYYV3MZoppWzGtYlTO9+meG6m/g92iNO9LfQB2JZsMpoJs7QG ku2KtabRK0bZRwDLyBDvwlxTm6ZlP7qyOqLcfqtLexpDSB4M0H3I/PQy1emvjjzgK+A0LmMKl6Lq zlqzh0VGAw440F6MJd8cY0nI7wiF/fVIBGY7UNCAXy6DmfYGCLLI0wtDbVcDUMqtJLmAhLqODQAe riERAxXJ1/QYGpa0ymqyytpKC19MNXHjvFmEsfcHIrncFR4xdbYWgmfEGLCcZokpGbGj1egMR+6M 1BkNX1pDdhPcOXpAnAeLQUwQLYepgQoZVNGS61yaE8CYA7gYAcWKzwGstACY2HTFvvOwk4FXAG/a mKHni/EcA/GkOk7I0IK7UMIf3+SahU8/FJdiE7KcuWdM3MFocUDEEIX9LfJoo4xV5tnNKc3jJuSs SZWgnnhepgU1zN4Hii18yW4RwDX52CXUtk0Hqz6cHOIUkWaX8fDcB+J7y1y2xDHwjv/8Buu8Ekz6 7tXQAAAAASUVORK5CYII= X-Mailer: MH-E 8.3.1; GNU Mailutils 2.99.98; GNU Emacs 24.3.1 MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable X-NewUkraine-Agent: mailfromd (7.99.92) X-NewUkraine-URL: https://mail.prozora-kraina.org/smtp.html X-NewUkraine-VirStat: NO X-NewUkraine-VirScan: ScanPE, ScanELF, ScanOLE2, ScanMail, PhishingSignatures, ScanHTML, ScanPDF X-NewUkraine-SpamStat: NO X-NewUkraine-SpamScore: -1.600 of 3.500 X-NewUkraine-SpamKeys: AWL,BAYES_00,NO_RECEIVED,NO_RELAYS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2015 10:56:27 -0000 =2D----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, on important production server I've started to experience weird regular (once a day-two) sporadic reboots after last upgrade (2015-07-17), and would appreciate if someone could help me to analyze further and fix the issue. Collected crash information below full logs are available upon request. Thank you! backtrace: =2D ---[ quotation start ]------------------------------------------- fo02.ibs dumped core - see /var/crash/vmcore.2 Wed Jul 22 08:01:15 EEST 2015 FreeBSD fo02.ibs 10.1-RELEASE-p14 FreeBSD 10.1-RELEASE-p14 #4 r285499M: Fri= Jul 17 22:45:54 EEST 2015 root@fo02.ibs:/usr/obj/usr/src/sys/FO02.2015= 0307233500 amd64 panic: page fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid =3D 3; apic id =3D 06 fault virtual address =3D 0xffff80403c82b740 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80730e9c stack pointer =3D 0x28:0xfffffe000023d790 frame pointer =3D 0x28:0xfffffe000023d7a0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 12 (irq283: igb3:que 3) trap number =3D 12 panic: page fault cpuid =3D 3 KDB: stack backtrace: #0 0xffffffff804fe970 at kdb_backtrace+0x60 #1 0xffffffff804c52a5 at panic+0x155 #2 0xffffffff8073f1cf at trap_fatal+0x38f #3 0xffffffff8073f4e8 at trap_pfault+0x308 #4 0xffffffff8073eb4a at trap+0x47a #5 0xffffffff80724cb2 at calltrap+0x8 #6 0xffffffff807910cb at bounce_bus_dmamap_load_buffer+0x1bb #7 0xffffffff804f9c82 at bus_dmamap_load_mbuf_sg+0x72 #8 0xffffffff810c37fe at igb_refresh_mbufs+0x19e #9 0xffffffff810c3603 at igb_rxeof+0x683 #10 0xffffffff810c3ab4 at igb_msix_que+0xe4 #11 0xffffffff80498e1b at intr_event_execute_handlers+0xab #12 0xffffffff80499266 at ithread_loop+0x96 #13 0xffffffff80496a3a at fork_exit+0x9a #14 0xffffffff807251ee at fork_trampoline+0xe Uptime: 1d10h23m19s Dumping 1191 out of 8136 MB:..2%..11%..21%..31%..41%..52%..61%..72%..81%..9= 2% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/pf.ko.symbols...done. Loaded symbols for /boot/kernel/pf.ko.symbols Reading symbols from /boot/kernel/if_igb.ko.symbols...done. Loaded symbols for /boot/kernel/if_igb.ko.symbols Reading symbols from /boot/kernel/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel/coretemp.ko.symbols Reading symbols from /boot/kernel/aesni.ko.symbols...done. Loaded symbols for /boot/kernel/aesni.ko.symbols Reading symbols from /boot/kernel/crypto.ko.symbols...done. Loaded symbols for /boot/kernel/crypto.ko.symbols Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. Loaded symbols for /boot/kernel/cryptodev.ko.symbols Reading symbols from /boot/kernel/pflog.ko.symbols...done. Loaded symbols for /boot/kernel/pflog.ko.symbols Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. Loaded symbols for /boot/kernel/ichsmb.ko.symbols Reading symbols from /boot/kernel/smbus.ko.symbols...done. Loaded symbols for /boot/kernel/smbus.ko.symbols Reading symbols from /boot/kernel/if_bridge.ko.symbols...done. Loaded symbols for /boot/kernel/if_bridge.ko.symbols Reading symbols from /boot/kernel/bridgestp.ko.symbols...done. Loaded symbols for /boot/kernel/bridgestp.ko.symbols Reading symbols from /boot/kernel/ums.ko.symbols...done. Loaded symbols for /boot/kernel/ums.ko.symbols #0 doadump (textdump=3D) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=3D) at pcpu.h:219 #1 0xffffffff804c4f22 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff804c52e4 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff8073f1cf in trap_fatal (frame=3D,=20 eva=3D) at /usr/src/sys/amd64/amd64/trap.c:865 #4 0xffffffff8073f4e8 in trap_pfault (frame=3D0xfffffe000023d6e0,=20 usermode=3D) at /usr/src/sys/amd64/amd64/trap.c:676 #5 0xffffffff8073eb4a in trap (frame=3D0xfffffe000023d6e0) at /usr/src/sys/amd64/amd64/trap.c:440 #6 0xffffffff80724cb2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff80730e9c in pmap_kextract (va=3D729286318152418877) at /usr/src/sys/amd64/amd64/pmap.c:639 #8 0xffffffff807910cb in bounce_bus_dmamap_load_buffer ( dmat=3D0xfffff800034c1800, map=3D0xffffffff80be01e8,=20 buf=3D, buflen=3D,=20 pmap=3D0xffffffff80c52ce0, flags=3D,=20 segs=3D) at /usr/src/sys/x86/x86/busdma_bounce.c:6= 90 #9 0xffffffff804f9c82 in bus_dmamap_load_mbuf_sg (dmat=3D0xfffff800034c180= 0,=20 map=3D0x0, m0=3D, segs=3D0xfffffe000023d8b0,=20 nsegs=3D0xfffffe000023d8ac, flags=3D) at /usr/src/sys/kern/subr_bus_dma.c:110 #10 0xffffffff810c37fe in igb_refresh_mbufs (rxr=3D0xfffff8000349f330,=20 limit=3D2101) at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:4144 #11 0xffffffff810c3603 in igb_rxeof (count=3D) at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:4998 #12 0xffffffff810c3ab4 in igb_msix_que (arg=3D0xfffff800034a8b38) at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:1621 #13 0xffffffff80498e1b in intr_event_execute_handlers ( p=3D, ie=3D0xfffff80003473c00) at /usr/src/sys/kern/kern_intr.c:1263 #14 0xffffffff80499266 in ithread_loop (arg=3D0xfffff800034a53e0) at /usr/src/sys/kern/kern_intr.c:1276 #15 0xffffffff80496a3a in fork_exit ( callout=3D0xffffffff804991d0 , arg=3D0xfffff800034a53e0,= =20 frame=3D0xfffffe000023dac0) at /usr/src/sys/kern/kern_fork.c:996 #16 0xffffffff807251ee in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #17 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb)=20 =2D ---[ quotation end ]------------------------------------------- #> uname -a FreeBSD 10.1-RELEASE-p14 #4 r285499M amd64 #> dmesg ... FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz (3100.06-MHz K8-class CPU) ... real memory =3D 8589934592 (8192 MB) avail memory =3D 8245338112 (7863 MB) ... pcib4: irq 18 at device 28.2 on pci0 pci4: on pcib4 igb0: port 0xd000-0x= d01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18 at device 0.0 o= n pci4 igb0: Using MSIX interrupts with 5 vectors igb0: Ethernet address: 00:11:22:33:44:51 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 pcib5: irq 19 at device 28.3 on pci0 pci5: on pcib5 igb1: port 0xc000-0x= c01f mem 0xf7300000-0xf737ffff,0xf7380000-0xf7383fff irq 19 at device 0.0 o= n pci5 igb1: Using MSIX interrupts with 5 vectors igb1: Ethernet address: 00:11:22:33:44:52 igb1: Bound queue 0 to cpu 0 igb1: Bound queue 1 to cpu 1 igb1: Bound queue 2 to cpu 2 igb1: Bound queue 3 to cpu 3 pcib6: irq 18 at device 28.6 on pci0 pci6: on pcib6 igb2: port 0xb000-0x= b01f mem 0xf7200000-0xf727ffff,0xf7280000-0xf7283fff irq 18 at device 0.0 o= n pci6 igb2: Using MSIX interrupts with 5 vectors igb2: Ethernet address: 00:11:22:33:44:53 igb2: Bound queue 0 to cpu 0 igb2: Bound queue 1 to cpu 1 igb2: Bound queue 2 to cpu 2 igb2: Bound queue 3 to cpu 3 pcib7: irq 19 at device 28.7 on pci0 pci7: on pcib7 igb3: port 0xa000-0x= a01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 19 at device 0.0 o= n pci7 igb3: Using MSIX interrupts with 5 vectors igb3: Ethernet address: 00:11:22:33:44:54 igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 ... #> dmidecode ... System Information Manufacturer: Supermicro Product Name: X10SLM+-LN4F ... #> pciconf -lv ... igb0@pci0:4:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet igb1@pci0:5:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet igb2@pci0:6:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet igb3@pci0:7:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet ... =2D --=20 Zeus V. Panchenko jid:zeus@im.ibs.dn.ua IT Dpt., I.B.S. LLC GMT+2 (EET) =2D----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlWvdkEACgkQr3jpPg/3oyraCwCg8OZFNErBWJXmjSHLm+Qm8ruL IdcAoLG5kxdQpRjv1lPGN3UO27vFY7+H =3DahV7 =2D----END PGP SIGNATURE-----