Date: Tue, 07 Oct 2014 18:48:51 -0700 (PDT) From: "Nick Sivo" <nick@ycombinator.com> To: freebsd-questions@freebsd.org Subject: Next Steps to Debug ZFS Hang? Message-ID: <1412732931033.813626ca@Nodemailer>
index | next in thread | raw e-mail
Hello, I've been having trouble with ZFS on my server. For the most part it works splendidly, but occasionally I'll experience permanent hangs. For example, right now on one of my ZFS filesystems (the others are fine), I can read, write, and stat files, but if I run ls in any directory, ls and the terminal will hang. CTRL-C, and kill -9 can't kill it: In top: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 5868 nsivo 1 20 0 14456K 1016K zfs 0 0:00 0.00% ls In ps: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND nsivo 5868 0.0 0.0 14456 1016 2- D+ 2:35PM 0:00.00 ls Eventually the entire system hangs, and can't be shutdown cleanly. What are the next steps to debug this? I'm a software developer, but am not familiar with kernel debugging. Is there a way to discover in which syscall ls is stuck? Ideally without requiring a crash dump? Thanks for reading, Nick -Nick From owner-freebsd-questions@FreeBSD.ORG Wed Oct 8 02:54:59 2014 Return-Path: <owner-freebsd-questions@FreeBSD.ORG> Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0403E170 for <freebsd-questions@freebsd.org>; Wed, 8 Oct 2014 02:54:59 +0000 (UTC) Received: from fly.hiwaay.net (fly.hiwaay.net [216.180.54.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5EF080D for <freebsd-questions@freebsd.org>; Wed, 8 Oct 2014 02:54:58 +0000 (UTC) Received: from kabini1.local (rbn1-216-180-19-6.adsl.hiwaay.net [216.180.19.6]) (authenticated bits=0) by fly.hiwaay.net (8.13.8/8.13.8/fly) with ESMTP id s982suZK018037 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits8 verify=NO) for <freebsd-questions@freebsd.org>; Tue, 7 Oct 2014 21:54:56 -0500 Message-ID: <5434A8F7.1090507@hiwaay.net> Date: Tue, 07 Oct 2014 22:01:11 -0500 From: "William A. Mahaffey III" <wam@hiwaay.net> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "FreeBSD Questions !!!!" <freebsd-questions@freebsd.org> Subject: oddball syslog entries .... Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions <freebsd-questions.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>; List-Post: <mailto:freebsd-questions@freebsd.org> List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 08 Oct 2014 02:54:59 -0000 Over the last couple of days I am seeing some odd (to me) entries in my messages file: Oct 2 09:32:18 kabini1 pkg: linux-f10-gtk2-2.14.7_5 deinstalled Oct 2 09:32:18 kabini1 pkg: linux-f10-pango-1.28.3_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-cairo-1.8.0_3 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-xorg-libs-7.4_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-fontconfig-2.6.0_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-tiff-3.8.2 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-atk-1.24.0_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-expat-2.0.1_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-jpeg-6b deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-png-1.2.37_2 deinstalled Oct 2 09:32:21 kabini1 pkg: linux_base-f10-10_7 deinstalled Oct 2 09:35:47 kabini1 pkg-static: linux_base-c6-6.5_1 installed Oct 2 09:39:50 kabini1 pkg-static: linux-c6-atk-1.30.0 installed Oct 2 09:39:51 kabini1 pkg-static: linux-c6-expat-2.0.1 installed Oct 2 09:39:52 kabini1 pkg-static: linux-c6-fontconfig-2.8.0 installed Oct 2 09:39:53 kabini1 pkg-static: linux-c6-png-1.2.49 installed Oct 2 09:39:55 kabini1 pkg-static: linux-c6-xorg-libs-7.4 installed Oct 2 09:39:56 kabini1 pkg-static: linux-c6-cairo-1.8.8 installed Oct 2 09:39:58 kabini1 pkg-static: linux-c6-openssl-1.0.1e installed Oct 2 09:39:59 kabini1 pkg-static: linux-c6-libssh2-1.4.2 installed Oct 2 09:40:00 kabini1 pkg-static: linux-c6-openldap-2.4.23 installed Oct 2 09:40:01 kabini1 pkg-static: linux-c6-curl-7.19.7 installed Oct 2 09:40:04 kabini1 pkg-static: linux-c6-jpeg-1.2.1 installed Oct 2 09:40:05 kabini1 pkg-static: linux-c6-pango-1.28.1 installed Oct 2 09:40:06 kabini1 pkg-static: linux-c6-tiff-3.9.4 installed Oct 2 09:40:07 kabini1 pkg-static: linux-c6-gtk2-2.20.1 installed Oct 2 09:40:09 kabini1 pkg-static: linux-c6-openssl-compat-0.9.8e installed Oct 2 09:40:10 kabini1 pkg-static: linux-c6-cyrus-sasl2-2.1.23 installed Oct 2 09:40:11 kabini1 pkg-static: linux-c6-nspr-4.10.0 installed Oct 2 09:40:18 kabini1 pkg-static: linux-c6-sqlite-3.6.20 installed Oct 2 09:40:19 kabini1 pkg-static: linux-c6-nss-3.16.1 installed Oct 2 09:40:20 kabini1 pkg-static: linux-c6-flashplugin-11.2r202.406 installed Oct 5 11:30:22 kabini1 kernel: Limiting closed port RST response from 276 to 200 packets/sec Oct 5 11:30:24 kabini1 kernel: Limiting closed port RST response from 239 to 200 packets/sec Oct 5 11:30:25 kabini1 kernel: Limiting closed port RST response from 280 to 200 packets/sec Oct 5 11:30:26 kabini1 kernel: Limiting closed port RST response from 319 to 200 packets/sec Oct 7 10:41:25 kabini1 kernel: Limiting closed port RST response from 276 to 200 packets/sec Oct 7 10:41:26 kabini1 kernel: Limiting closed port RST response from 239 to 200 packets/sec Oct 7 10:41:27 kabini1 kernel: Limiting closed port RST response from 280 to 200 packets/sec Oct 7 10:41:29 kabini1 kernel: Limiting closed port RST response from 319 to 200 packets/sec Oct 7 14:59:41 kabini1 kernel: Limiting closed port RST response from 253 to 200 packets/sec Oct 7 14:59:42 kabini1 kernel: Limiting closed port RST response from 233 to 200 packets/sec Oct 7 14:59:44 kabini1 kernel: Limiting closed port RST response from 265 to 200 packets/sec Oct 7 14:59:45 kabini1 kernel: Limiting closed port RST response from 295 to 200 packets/sec Oct 7 14:59:47 kabini1 kernel: Limiting closed port RST response from 324 to 200 packets/sec Oct 7 15:03:18 kabini1 kernel: Limiting closed port RST response from 253 to 200 packets/sec Oct 7 15:03:20 kabini1 kernel: Limiting closed port RST response from 233 to 200 packets/sec Oct 7 15:03:21 kabini1 kernel: Limiting closed port RST response from 265 to 200 packets/sec Oct 7 15:03:22 kabini1 kernel: Limiting closed port RST response from 295 to 200 packets/sec Oct 7 15:03:24 kabini1 kernel: Limiting closed port RST response from 324 to 200 packets/sec The stuff from Oct 2 is irrelevant, included for completeness/context. The lines about 'Limiting closed port ....' are puzzling to me. Where are they coming from ? Problem or chatter ? Enquiring minds wanna know ;-) .... TIA for any clues .... -- William A. Mahaffey III ---------------------------------------------------------------------- "The M1 Garand is without doubt the finest implement of war ever devised by man." -- Gen. George S. Patton Jr.help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1412732931033.813626ca>