From owner-freebsd-questions@freebsd.org Mon May 29 09:20:48 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3AC9CF6788 for ; Mon, 29 May 2017 09:20:48 +0000 (UTC) (envelope-from raimo+freebsd@erix.ericsson.se) Received: from sesbmg23.ericsson.net (sesbmg23.ericsson.net [193.180.251.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5508F724A8 for ; Mon, 29 May 2017 09:20:47 +0000 (UTC) (envelope-from raimo+freebsd@erix.ericsson.se) X-AuditID: c1b4fb25-73a9f9a0000055fe-4d-592be7ebec85 Received: from ESESSHC014.ericsson.se (Unknown_Domain [153.88.183.60]) by sesbmg23.ericsson.net (Symantec Mail Security) with SMTP id C7.F5.22014.BE7EB295; Mon, 29 May 2017 11:20:44 +0200 (CEST) Received: from duper.otp.ericsson.se (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.62) with Microsoft SMTP Server id 14.3.339.0; Mon, 29 May 2017 11:20:43 +0200 Received: from duper.otp.ericsson.se (localhost [127.0.0.1]) by duper.otp.ericsson.se (8.15.2/8.15.2) with ESMTP id v4T9KhkU091538 for ; Mon, 29 May 2017 11:20:43 +0200 (CEST) (envelope-from raimo+freebsd@erix.otp.ericsson.se) Received: (from raimo@localhost) by duper.otp.ericsson.se (8.15.2/8.15.2/Submit) id v4T9KhbY091537 for freebsd-questions@freebsd.org; Mon, 29 May 2017 11:20:43 +0200 (CEST) (envelope-from raimo+freebsd@erix.otp.ericsson.se) X-Authentication-Warning: duper.otp.ericsson.se: raimo set sender to raimo+freebsd@erix.ericsson.se using -f Date: Mon, 29 May 2017 11:20:43 +0200 From: Raimo Niskanen To: Subject: Advice on kernel panics Message-ID: <20170529092043.GA89682@erix.ericsson.se> Mail-Followup-To: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline User-Agent: Mutt/1.7.2 (2016-11-26) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrMLMWRmVeSWpSXmKPExsUyM2K7je6b59qRBvfPMVm8/LqJxYHRY8an +SwBjFFcNimpOZllqUX6dglcGeceP2YtmCRVsWDqdLYGxnnCXYycHBICJhLvrvaxg9hCAkcY JU5/Du9i5AKyNzJK7JuzmgXCaWeSmHxtP1sXIweQ85RR4r4aRHOOxNtPR5lAbBYBVYlZ064w gthsAqYSjT/PsILYIgLKEv+uXWQGsYUFFCTWf17PBmLzAtVsbXjDAmHrS3xc8YoVwhaUODnz CVicWUBHYsHuT2BrmQWkJZb/4wAxRUFGfmWFOFlbYsKbA6wTGAVnIWmehaR5FkLzAkbmVYyi xanFSbnpRsZ6qUWZycXF+Xl6eaklmxiBQXlwy2/VHYyX3zgeYhTgYFTi4f10QjtSiDWxrLgy 9xCjBAezkgjv7cdAId6UxMqq1KL8+KLSnNTiQ4zSHCxK4ryO+y5ECAmkJ5akZqemFqQWwWSZ ODilGhhldmufMWR8/2tnwO7sGJ/aL74rzrz9tvXo/KXn3uWuzOb61lEcIjxRZa/zcwEONqG4 O2YS9xf8Stn6XignYd29okPeJjx+Hj+EpzcsLui5Nakx9+NHv9VXUiILOp2iFr+XyP2u4rWv Z9bkk/xdByd92Mi/+MvvurMX32kzhZ8SOpeQpeC+6ORSJZbijERDLeai4kQALAfSOkYCAAA= X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 May 2017 09:20:48 -0000 Hello list. I have a server that panics about every 3 days and need some advice on how to handle that. It currently has 7 dumps in /var/crash/, head of the latest core.txt.4 looks like this: ======= sasquatch.otp.ericsson.se dumped core - see /var/crash/vmcore.4 Mon May 29 03:15:32 CEST 2017 FreeBSD sasquatch.otp.ericsson.se 10.3-RELEASE-p18 FreeBSD 10.3-RELEASE-p18 #0: Tue Apr 11 10:31:00 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 panic: page fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x0 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff809fb017 stack pointer = 0x28:0xfffffe04673a18c0 frame pointer = 0x28:0xfffffe04673a1900 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 18 (syncer) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff8098e7e0 at kdb_backtrace+0x60 #1 0xffffffff809514b6 at vpanic+0x126 #2 0xffffffff80951383 at panic+0x43 #3 0xffffffff80d5646b at trap_fatal+0x36b #4 0xffffffff80d5676d at trap_pfault+0x2ed #5 0xffffffff80d55dea at trap+0x47a #6 0xffffffff80d3bdb2 at calltrap+0x8 #7 0xffffffff809f9b23 at vfs_msync+0x203 #8 0xffffffff809fb858 at sync_fsync+0x108 #9 0xffffffff80e81ed7 at VOP_FSYNC_APV+0xa7 #10 0xffffffff809fc27b at sched_sync+0x3ab #11 0xffffffff8091a93a at fork_exit+0x9a #12 0xffffffff80d3c2ee at fork_trampoline+0xe Uptime: 2d19h53m15s ======= What sticks out later in core.txt.4 is the fstat section that contains a lot of errors, but I can not tell if that is just a secondary symptom... Looks like this: ======= fstat fstat: can't read file 1 at 0x200007fffffffff fstat: can't read file 2 at 0x4000000001fffff fstat: can't read znode_phys at 0x1 fstat: can't read znode_phys at 0x1 fstat: can't read znode_phys at 0x1 : USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root sed 78401 root - - error - root sed 78401 wd - - error - root sed 78401 text - - error - root sed 78401 0* pipe fffff8001800f000 <-> fffff8001800f160 0 rw root grep 78400 root - - error - root grep 78400 wd - - error - root grep 78400 text - - error - : ======= To me the other core.txt.? files does not look exactly the same. All have an fstat section with many errors, though. Does anyone have some advice on how to proceed? -- / Raimo Niskanen, Erlang/OTP, Ericsson AB