Date: Tue, 07 Oct 2014 18:48:51 -0700 (PDT) From: "Nick Sivo" <nick@ycombinator.com> To: freebsd-questions@freebsd.org Subject: Next Steps to Debug ZFS Hang? Message-ID: <1412732931033.813626ca@Nodemailer>
next in thread | raw e-mail | index | archive | help
Hello, I've been having trouble with ZFS on my server. For the most part it works = splendidly, but occasionally I'll experience permanent hangs. For example, right now on one of my ZFS filesystems (the others are fine), = I can read, write, and stat files, but if I run ls in any directory, ls and= the terminal will hang. CTRL-C, and kill -9 can't kill it: In top: =C2=A0 PID USERNAME=C2=A0 =C2=A0 THR PRI NICE =C2=A0 SIZE=C2=A0 =C2=A0 RES = STATE =C2=A0 C =C2=A0 TIME =C2=A0 WCPU COMMAND =C2=A05868 nsivo =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 20=C2=A0 =C2=A0 0 = 14456K=C2=A0 1016K zfs =C2=A0 =C2=A0 0 =C2=A0 0:00=C2=A0 0.00% ls In ps: USER=C2=A0 =C2=A0 =C2=A0 PID=C2=A0 %CPU %MEM =C2=A0 =C2=A0 VSZ =C2=A0 = =C2=A0 RSS TT=C2=A0 STAT STARTED=C2=A0 =C2=A0 =C2=A0 =C2=A0 TIME COMMAND nsivo=C2=A0 =C2=A0 5868 =C2=A0 0.0=C2=A0 0.0 =C2=A0 14456=C2=A0 =C2=A0 = 1016=C2=A0 2- D+=C2=A0 =C2=A0 2:35PM =C2=A0 =C2=A0 0:00.00 ls Eventually the entire system hangs, and can't be shutdown cleanly. What are the next steps to debug this=3F I'm a software developer, but am = not familiar with kernel debugging. Is there a way to discover in which = syscall ls is stuck=3F Ideally without requiring a crash dump=3F Thanks for reading, Nick -Nick From owner-freebsd-questions@FreeBSD.ORG Wed Oct 8 02:54:59 2014 Return-Path: <owner-freebsd-questions@FreeBSD.ORG> Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0403E170 for <freebsd-questions@freebsd.org>; Wed, 8 Oct 2014 02:54:59 +0000 (UTC) Received: from fly.hiwaay.net (fly.hiwaay.net [216.180.54.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5EF080D for <freebsd-questions@freebsd.org>; Wed, 8 Oct 2014 02:54:58 +0000 (UTC) Received: from kabini1.local (rbn1-216-180-19-6.adsl.hiwaay.net [216.180.19.6]) (authenticated bits=0) by fly.hiwaay.net (8.13.8/8.13.8/fly) with ESMTP id s982suZK018037 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for <freebsd-questions@freebsd.org>; Tue, 7 Oct 2014 21:54:56 -0500 Message-ID: <5434A8F7.1090507@hiwaay.net> Date: Tue, 07 Oct 2014 22:01:11 -0500 From: "William A. Mahaffey III" <wam@hiwaay.net> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "FreeBSD Questions !!!!" <freebsd-questions@freebsd.org> Subject: oddball syslog entries .... Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions <freebsd-questions.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>; List-Post: <mailto:freebsd-questions@freebsd.org> List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 08 Oct 2014 02:54:59 -0000 Over the last couple of days I am seeing some odd (to me) entries in my messages file: Oct 2 09:32:18 kabini1 pkg: linux-f10-gtk2-2.14.7_5 deinstalled Oct 2 09:32:18 kabini1 pkg: linux-f10-pango-1.28.3_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-cairo-1.8.0_3 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-xorg-libs-7.4_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-fontconfig-2.6.0_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-tiff-3.8.2 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-atk-1.24.0_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-expat-2.0.1_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-jpeg-6b deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-png-1.2.37_2 deinstalled Oct 2 09:32:21 kabini1 pkg: linux_base-f10-10_7 deinstalled Oct 2 09:35:47 kabini1 pkg-static: linux_base-c6-6.5_1 installed Oct 2 09:39:50 kabini1 pkg-static: linux-c6-atk-1.30.0 installed Oct 2 09:39:51 kabini1 pkg-static: linux-c6-expat-2.0.1 installed Oct 2 09:39:52 kabini1 pkg-static: linux-c6-fontconfig-2.8.0 installed Oct 2 09:39:53 kabini1 pkg-static: linux-c6-png-1.2.49 installed Oct 2 09:39:55 kabini1 pkg-static: linux-c6-xorg-libs-7.4 installed Oct 2 09:39:56 kabini1 pkg-static: linux-c6-cairo-1.8.8 installed Oct 2 09:39:58 kabini1 pkg-static: linux-c6-openssl-1.0.1e installed Oct 2 09:39:59 kabini1 pkg-static: linux-c6-libssh2-1.4.2 installed Oct 2 09:40:00 kabini1 pkg-static: linux-c6-openldap-2.4.23 installed Oct 2 09:40:01 kabini1 pkg-static: linux-c6-curl-7.19.7 installed Oct 2 09:40:04 kabini1 pkg-static: linux-c6-jpeg-1.2.1 installed Oct 2 09:40:05 kabini1 pkg-static: linux-c6-pango-1.28.1 installed Oct 2 09:40:06 kabini1 pkg-static: linux-c6-tiff-3.9.4 installed Oct 2 09:40:07 kabini1 pkg-static: linux-c6-gtk2-2.20.1 installed Oct 2 09:40:09 kabini1 pkg-static: linux-c6-openssl-compat-0.9.8e installed Oct 2 09:40:10 kabini1 pkg-static: linux-c6-cyrus-sasl2-2.1.23 installed Oct 2 09:40:11 kabini1 pkg-static: linux-c6-nspr-4.10.0 installed Oct 2 09:40:18 kabini1 pkg-static: linux-c6-sqlite-3.6.20 installed Oct 2 09:40:19 kabini1 pkg-static: linux-c6-nss-3.16.1 installed Oct 2 09:40:20 kabini1 pkg-static: linux-c6-flashplugin-11.2r202.406 installed Oct 5 11:30:22 kabini1 kernel: Limiting closed port RST response from 276 to 200 packets/sec Oct 5 11:30:24 kabini1 kernel: Limiting closed port RST response from 239 to 200 packets/sec Oct 5 11:30:25 kabini1 kernel: Limiting closed port RST response from 280 to 200 packets/sec Oct 5 11:30:26 kabini1 kernel: Limiting closed port RST response from 319 to 200 packets/sec Oct 7 10:41:25 kabini1 kernel: Limiting closed port RST response from 276 to 200 packets/sec Oct 7 10:41:26 kabini1 kernel: Limiting closed port RST response from 239 to 200 packets/sec Oct 7 10:41:27 kabini1 kernel: Limiting closed port RST response from 280 to 200 packets/sec Oct 7 10:41:29 kabini1 kernel: Limiting closed port RST response from 319 to 200 packets/sec Oct 7 14:59:41 kabini1 kernel: Limiting closed port RST response from 253 to 200 packets/sec Oct 7 14:59:42 kabini1 kernel: Limiting closed port RST response from 233 to 200 packets/sec Oct 7 14:59:44 kabini1 kernel: Limiting closed port RST response from 265 to 200 packets/sec Oct 7 14:59:45 kabini1 kernel: Limiting closed port RST response from 295 to 200 packets/sec Oct 7 14:59:47 kabini1 kernel: Limiting closed port RST response from 324 to 200 packets/sec Oct 7 15:03:18 kabini1 kernel: Limiting closed port RST response from 253 to 200 packets/sec Oct 7 15:03:20 kabini1 kernel: Limiting closed port RST response from 233 to 200 packets/sec Oct 7 15:03:21 kabini1 kernel: Limiting closed port RST response from 265 to 200 packets/sec Oct 7 15:03:22 kabini1 kernel: Limiting closed port RST response from 295 to 200 packets/sec Oct 7 15:03:24 kabini1 kernel: Limiting closed port RST response from 324 to 200 packets/sec The stuff from Oct 2 is irrelevant, included for completeness/context. The lines about 'Limiting closed port ....' are puzzling to me. Where are they coming from ? Problem or chatter ? Enquiring minds wanna know ;-) .... TIA for any clues .... -- William A. Mahaffey III ---------------------------------------------------------------------- "The M1 Garand is without doubt the finest implement of war ever devised by man." -- Gen. George S. Patton Jr.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1412732931033.813626ca>