From owner-freebsd-hackers@freebsd.org Fri Jun 16 19:14:38 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 341B2D87469; Fri, 16 Jun 2017 19:14:38 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C923682FA; Fri, 16 Jun 2017 19:14:38 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.15.2/8.15.2) with ESMTPS id v5GJEa9Q001246 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 16 Jun 2017 12:14:36 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.15.2/8.15.2/Submit) id v5GJEade001245; Fri, 16 Jun 2017 12:14:36 -0700 (PDT) (envelope-from sgk) Date: Fri, 16 Jun 2017 12:14:36 -0700 From: Steve Kargl To: freebsd-current@freebsd.org, freebsd-hackers@freebsd.org Subject: panic: handle_written_inodeblock: Invalid link count 57368 Message-ID: <20170616191436.GA1189@troutmask.apl.washington.edu> Reply-To: sgk@troutmask.apl.washington.edu MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jun 2017 19:14:38 -0000 I grabbed a spare USB 2 TB hard drive yesterday. Put a GPT scheme on the drive and then used newfs to create 1 large UFS2 partition with softupdates and with journalling enabled. I then some 150 GB of data to drive. The drive in question is ugen0.3: at usbus0 umass0 on uhub8 umass0: on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x4000 umass0:9:0: Attached to scbus9 da0 at umass-sim0 bus 0 scbus9 target 0 lun 0 da0: Fixed Direct Access SPC-4 SCSI device da0: Serial Number 575835314134334136333937 da0: 40.000MB/s transfers da0: 1907697MB (3906963456 512 byte sectors) da0: quirks=0x2 I just got the indicated panic while trying to do ls in a directory with 100+ files. From the panic, I have core.txt.7, info.7, and vmcore.7. info.7 contains Dump header from device: /dev/ada0p3 Architecture: amd64 Architecture Version: 2 Dump Length: 1150738432 Blocksize: 512 Dumptime: Fri Jun 16 11:47:07 2017 Hostname: troutmask.apl.washington.edu Magic: FreeBSD Kernel Dump Version String: FreeBSD 12.0-CURRENT #0 r318546: Fri May 19 12:51:04 PDT 2017 kargl@troutmask.apl.washington.edu:/data/obj/usr/src/sys/SPEW Panic String: handle_written_inodeblock: Invalid link count 57368 for inodedep 0xfffff800a56c4000 Dump Parity: 2911432532 Bounds: 7 Dump Status: good Leading portion of core.txt.7 is Unread portion of the kernel message buffer: panic: handle_written_inodeblock: Invalid link count 57368 for inodedep 0xfffff800a56c4000 cpuid = 3 time = 1497638827 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe022e2c40b0 vpanic() at vpanic+0x19c/frame 0xfffffe022e2c4130 panic() at panic+0x43/frame 0xfffffe022e2c4190 handle_written_inodeblock() at handle_written_inodeblock+0x7c9/frame 0xfffffe022e2c41e0 softdep_disk_write_complete() at softdep_disk_write_complete+0x1b4/frame 0xfffffe022e2c4240 bufdone_finish() at bufdone_finish+0x34/frame 0xfffffe022e2c42b0 bufdone() at bufdone+0x45/frame 0xfffffe022e2c42d0 g_io_deliver() at g_io_deliver+0x234/frame 0xfffffe022e2c4330 g_io_deliver() at g_io_deliver+0x234/frame 0xfffffe022e2c4390 g_disk_done() at g_disk_done+0x10d/frame 0xfffffe022e2c43e0 dadone() at dadone+0x1e21/frame 0xfffffe022e2c4960 xpt_done_process() at xpt_done_process+0x5d6/frame 0xfffffe022e2c49a0 xpt_done_td() at xpt_done_td+0x166/frame 0xfffffe022e2c49f0 fork_exit() at fork_exit+0x75/frame 0xfffffe022e2c4a30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe022e2c4a30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 27d22h21m17s (da0:umass-sim0:0:0:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da0:umass-sim0:0:0:0): CAM status: Resource Unavailable (da0:umass-sim0:0:0:0): Error 5, Retries exhausted (da0:umass-sim0:0:0:0): Synchronize cache failed Dumping 1097 out of 8142 MB:..2%..11%..21%..31%..41%..52%..62%..72%..81%..91% Reading symbols from /boot/kernel/radeonkms.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkms.ko.debug...done. done. Reading symbols from /boot/kernel/drm2.ko...Reading symbols from /usr/lib/debug//boot/kernel/drm2.ko.debug...done. done. Reading symbols from /boot/kernel/agp.ko...Reading symbols from /usr/lib/debug//boot/kernel/agp.ko.debug...done. done. Reading symbols from /boot/kernel/radeonkmsfw_CAICOS_pfp.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_CAICOS_pfp.ko.debug...done. done. Reading symbols from /boot/kernel/radeonkmsfw_CAICOS_me.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_CAICOS_me.ko.debug...done. done. Reading symbols from /boot/kernel/radeonkmsfw_BTC_rlc.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_BTC_rlc.ko.debug...done. done. Reading symbols from /boot/kernel/radeonkmsfw_CAICOS_mc.ko...Reading symbols from /usr/lib/debug//boot/kernel/radeonkmsfw_CAICOS_mc.ko.debug...done. done. __curthread () at ./machine/pcpu.h:232 232 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) #0 __curthread () at ./machine/pcpu.h:232 #1 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:318 #2 0xffffffff8058649b in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:386 #3 0xffffffff80586916 in vpanic (fmt=, ap=0xfffffe022e2c4170) at /usr/src/sys/kern/kern_shutdown.c:779 #4 0xffffffff80586733 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:710 #5 0xffffffff8079c1b9 in handle_written_inodeblock ( inodedep=0xfffff800a56c4000, bp=0xfffffe01f01b7a58, flags=) at /usr/src/sys/ufs/ffs/ffs_softdep.c:11535 #6 0xffffffff80782514 in softdep_disk_write_complete (bp=0xfffffe01f01b7a58) at /usr/src/sys/ufs/ffs/ffs_softdep.c:11043 #7 0xffffffff8061d2b4 in buf_complete (bp=0xfffffe01f01b7a58) at /usr/src/sys/sys/buf.h:422 #8 bufdone_finish (bp=0xfffffe01f01b7a58) at /usr/src/sys/kern/vfs_bio.c:4045 #9 0xffffffff8061d1b5 in bufdone (bp=0xfffffe01f01b7a58) at /usr/src/sys/kern/vfs_bio.c:4033 #10 0xffffffff8050df14 in g_io_deliver (bp=0xfffff80115d71bc0, error=) at /usr/src/sys/geom/geom_io.c:738 #11 0xffffffff8050df14 in g_io_deliver (bp=0xfffff801fd3b32f0, error=) at /usr/src/sys/geom/geom_io.c:738 #12 0xffffffff8050b77d in g_disk_done (bp=0xfffff8007613c2f0) at /usr/src/sys/geom/geom_disk.c:256 #13 0xffffffff802ef1b1 in dadone (periph=, done_ccb=0xfffff80068599800) at /usr/src/sys/cam/scsi/scsi_da.c:4216 #14 0xffffffff80292936 in xpt_done_process (ccb_h=0xfffff80068599800) at /usr/src/sys/cam/cam_xpt.c:5453 #15 0xffffffff80294ab6 in xpt_done_td ( arg=0xffffffff80bf9300 ) at /usr/src/sys/cam/cam_xpt.c:5480 #16 0xffffffff80553ff5 in fork_exit ( callout=0xffffffff80294950 , arg=0xffffffff80bf9300 , frame=0xfffffe022e2c4a40) at /usr/src/sys/kern/kern_fork.c:1038 #17 core.txt.7 and vmcore.7 can be made available for the asking. -- Steve 20170425 https://www.youtube.com/watch?v=VWUpyCsUKR4 20161221 https://www.youtube.com/watch?v=IbCHE-hONow