From owner-freebsd-questions@FreeBSD.ORG Wed Oct 8 01:48:53 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8957CBB8 for ; Wed, 8 Oct 2014 01:48:53 +0000 (UTC) Received: from mail-vc0-x234.google.com (mail-vc0-x234.google.com [IPv6:2607:f8b0:400c:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B0BEF76 for ; Wed, 8 Oct 2014 01:48:53 +0000 (UTC) Received: by mail-vc0-f180.google.com with SMTP id le20so5938122vcb.11 for ; Tue, 07 Oct 2014 18:48:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ycombinator.com; s=google; h=date:mime-version:message-id:from:to:subject:content-type; bh=bl6vN/OwMnup4Uy98PvJyXVAeFHbBQoJ7DWqoKN99xI=; b=i8X0GjWQEIFMySLVnqeUwoJLaZMANqlgU2EEANgzwUrx0t/AlFyzKc2E2keADtvDbo TYV58kgqz8LjWWobBWiJesCMbXh9lG/TEKOhA+1bv1kmU2OwZ/iB7YWy2cZlWEGiJ3TG AIWPmf/WimfjzK87TXkNEYm4NNAz2rgMeSDb4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:mime-version:message-id:from:to:subject :content-type; bh=bl6vN/OwMnup4Uy98PvJyXVAeFHbBQoJ7DWqoKN99xI=; b=d1g1vkZeiu/kVU8gqKhJsPT/FHoM0Q8IprllUwKoC/6tNK+IMBmcl0AO0DVfLdyj0K E7SM+K58GFPbx5aRSmiSD0wDspkp3q9AVItrzsY65wiZUh4bdoIBu9rBJLTgucCFvVC4 x2qqJqxre8/00t3YdinjijUiiGHqg3guX8CFQ6h6fllxyZ6Rjubm7HBGsi0o2Vz8tPiC d4JbBAXN6ZkJGzsRE25FKwcHgqXbytgiUJVhZjLg89CQvsPmU6FsX/M4TWE6RTIrecBT n6NJqLXsbkJrjYBi2bba0atC2yVOtgPQ/jxr3NvLUvXNlkRlUl5XTvRRMUitFL5na+49 9oGg== X-Gm-Message-State: ALoCoQkZ0eQj6xaTxrix14RkWRcf1ky3VfWpTvR8TwMM02QABwjb3q23/bcF5KQdzlv/l4OmdzsP X-Received: by 10.52.1.69 with SMTP id 5mr1347502vdk.86.1412732932263; Tue, 07 Oct 2014 18:48:52 -0700 (PDT) Received: from hedwig-35.prd.orcali.com (ec2-54-85-253-134.compute-1.amazonaws.com. [54.85.253.134]) by mx.google.com with ESMTPSA id n7sm6477279vdi.27.2014.10.07.18.48.51 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 07 Oct 2014 18:48:51 -0700 (PDT) Date: Tue, 07 Oct 2014 18:48:51 -0700 (PDT) X-Google-Original-Date: Wed, 08 Oct 2014 01:48:51 GMT MIME-Version: 1.0 X-Mailer: Nodemailer (0.5.0; +http://www.nodemailer.com/) Message-Id: <1412732931033.813626ca@Nodemailer> X-Orchestra-Oid: ED69F7B8-1734-40B2-BAEF-7BFB1F11FA3C X-Orchestra-Sig: ff4fd7a681b01de9fda898cf09b440ccc8abd1c8 X-Orchestra-Thrid: 380643E9-885F-4793-A609-B809C37F88EE X-Orchestra-Thrid-Sig: e08933f784f1cf639017239417199d91757ee807 X-Orchestra-Account: 528163f51839e26a114bb1b287609bd1929bd590 From: "Nick Sivo" To: freebsd-questions@freebsd.org Subject: Next Steps to Debug ZFS Hang? Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2014 01:48:53 -0000 Hello, I've been having trouble with ZFS on my server. For the most part it works = splendidly, but occasionally I'll experience permanent hangs. For example, right now on one of my ZFS filesystems (the others are fine), = I can read, write, and stat files, but if I run ls in any directory, ls and= the terminal will hang. CTRL-C, and kill -9 can't kill it: In top: =C2=A0 PID USERNAME=C2=A0 =C2=A0 THR PRI NICE =C2=A0 SIZE=C2=A0 =C2=A0 RES = STATE =C2=A0 C =C2=A0 TIME =C2=A0 WCPU COMMAND =C2=A05868 nsivo =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 20=C2=A0 =C2=A0 0 = 14456K=C2=A0 1016K zfs =C2=A0 =C2=A0 0 =C2=A0 0:00=C2=A0 0.00% ls In ps: USER=C2=A0 =C2=A0 =C2=A0 PID=C2=A0 %CPU %MEM =C2=A0 =C2=A0 VSZ =C2=A0 = =C2=A0 RSS TT=C2=A0 STAT STARTED=C2=A0 =C2=A0 =C2=A0 =C2=A0 TIME COMMAND nsivo=C2=A0 =C2=A0 5868 =C2=A0 0.0=C2=A0 0.0 =C2=A0 14456=C2=A0 =C2=A0 = 1016=C2=A0 2- D+=C2=A0 =C2=A0 2:35PM =C2=A0 =C2=A0 0:00.00 ls Eventually the entire system hangs, and can't be shutdown cleanly. What are the next steps to debug this=3F I'm a software developer, but am = not familiar with kernel debugging. Is there a way to discover in which = syscall ls is stuck=3F Ideally without requiring a crash dump=3F Thanks for reading, Nick -Nick From owner-freebsd-questions@FreeBSD.ORG Wed Oct 8 02:54:59 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0403E170 for ; Wed, 8 Oct 2014 02:54:59 +0000 (UTC) Received: from fly.hiwaay.net (fly.hiwaay.net [216.180.54.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5EF080D for ; Wed, 8 Oct 2014 02:54:58 +0000 (UTC) Received: from kabini1.local (rbn1-216-180-19-6.adsl.hiwaay.net [216.180.19.6]) (authenticated bits=0) by fly.hiwaay.net (8.13.8/8.13.8/fly) with ESMTP id s982suZK018037 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 7 Oct 2014 21:54:56 -0500 Message-ID: <5434A8F7.1090507@hiwaay.net> Date: Tue, 07 Oct 2014 22:01:11 -0500 From: "William A. Mahaffey III" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "FreeBSD Questions !!!!" Subject: oddball syslog entries .... Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2014 02:54:59 -0000 Over the last couple of days I am seeing some odd (to me) entries in my messages file: Oct 2 09:32:18 kabini1 pkg: linux-f10-gtk2-2.14.7_5 deinstalled Oct 2 09:32:18 kabini1 pkg: linux-f10-pango-1.28.3_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-cairo-1.8.0_3 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-xorg-libs-7.4_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-fontconfig-2.6.0_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-tiff-3.8.2 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-atk-1.24.0_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-expat-2.0.1_1 deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-jpeg-6b deinstalled Oct 2 09:32:19 kabini1 pkg: linux-f10-png-1.2.37_2 deinstalled Oct 2 09:32:21 kabini1 pkg: linux_base-f10-10_7 deinstalled Oct 2 09:35:47 kabini1 pkg-static: linux_base-c6-6.5_1 installed Oct 2 09:39:50 kabini1 pkg-static: linux-c6-atk-1.30.0 installed Oct 2 09:39:51 kabini1 pkg-static: linux-c6-expat-2.0.1 installed Oct 2 09:39:52 kabini1 pkg-static: linux-c6-fontconfig-2.8.0 installed Oct 2 09:39:53 kabini1 pkg-static: linux-c6-png-1.2.49 installed Oct 2 09:39:55 kabini1 pkg-static: linux-c6-xorg-libs-7.4 installed Oct 2 09:39:56 kabini1 pkg-static: linux-c6-cairo-1.8.8 installed Oct 2 09:39:58 kabini1 pkg-static: linux-c6-openssl-1.0.1e installed Oct 2 09:39:59 kabini1 pkg-static: linux-c6-libssh2-1.4.2 installed Oct 2 09:40:00 kabini1 pkg-static: linux-c6-openldap-2.4.23 installed Oct 2 09:40:01 kabini1 pkg-static: linux-c6-curl-7.19.7 installed Oct 2 09:40:04 kabini1 pkg-static: linux-c6-jpeg-1.2.1 installed Oct 2 09:40:05 kabini1 pkg-static: linux-c6-pango-1.28.1 installed Oct 2 09:40:06 kabini1 pkg-static: linux-c6-tiff-3.9.4 installed Oct 2 09:40:07 kabini1 pkg-static: linux-c6-gtk2-2.20.1 installed Oct 2 09:40:09 kabini1 pkg-static: linux-c6-openssl-compat-0.9.8e installed Oct 2 09:40:10 kabini1 pkg-static: linux-c6-cyrus-sasl2-2.1.23 installed Oct 2 09:40:11 kabini1 pkg-static: linux-c6-nspr-4.10.0 installed Oct 2 09:40:18 kabini1 pkg-static: linux-c6-sqlite-3.6.20 installed Oct 2 09:40:19 kabini1 pkg-static: linux-c6-nss-3.16.1 installed Oct 2 09:40:20 kabini1 pkg-static: linux-c6-flashplugin-11.2r202.406 installed Oct 5 11:30:22 kabini1 kernel: Limiting closed port RST response from 276 to 200 packets/sec Oct 5 11:30:24 kabini1 kernel: Limiting closed port RST response from 239 to 200 packets/sec Oct 5 11:30:25 kabini1 kernel: Limiting closed port RST response from 280 to 200 packets/sec Oct 5 11:30:26 kabini1 kernel: Limiting closed port RST response from 319 to 200 packets/sec Oct 7 10:41:25 kabini1 kernel: Limiting closed port RST response from 276 to 200 packets/sec Oct 7 10:41:26 kabini1 kernel: Limiting closed port RST response from 239 to 200 packets/sec Oct 7 10:41:27 kabini1 kernel: Limiting closed port RST response from 280 to 200 packets/sec Oct 7 10:41:29 kabini1 kernel: Limiting closed port RST response from 319 to 200 packets/sec Oct 7 14:59:41 kabini1 kernel: Limiting closed port RST response from 253 to 200 packets/sec Oct 7 14:59:42 kabini1 kernel: Limiting closed port RST response from 233 to 200 packets/sec Oct 7 14:59:44 kabini1 kernel: Limiting closed port RST response from 265 to 200 packets/sec Oct 7 14:59:45 kabini1 kernel: Limiting closed port RST response from 295 to 200 packets/sec Oct 7 14:59:47 kabini1 kernel: Limiting closed port RST response from 324 to 200 packets/sec Oct 7 15:03:18 kabini1 kernel: Limiting closed port RST response from 253 to 200 packets/sec Oct 7 15:03:20 kabini1 kernel: Limiting closed port RST response from 233 to 200 packets/sec Oct 7 15:03:21 kabini1 kernel: Limiting closed port RST response from 265 to 200 packets/sec Oct 7 15:03:22 kabini1 kernel: Limiting closed port RST response from 295 to 200 packets/sec Oct 7 15:03:24 kabini1 kernel: Limiting closed port RST response from 324 to 200 packets/sec The stuff from Oct 2 is irrelevant, included for completeness/context. The lines about 'Limiting closed port ....' are puzzling to me. Where are they coming from ? Problem or chatter ? Enquiring minds wanna know ;-) .... TIA for any clues .... -- William A. Mahaffey III ---------------------------------------------------------------------- "The M1 Garand is without doubt the finest implement of war ever devised by man." -- Gen. George S. Patton Jr.