From owner-freebsd-fs@freebsd.org Tue May 17 01:51:28 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7908B3E164 for ; Tue, 17 May 2016 01:51:28 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9986719AF for ; Tue, 17 May 2016 01:51:28 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id u4H1pQeY008619; Mon, 16 May 2016 20:51:27 -0500 (CDT) Date: Mon, 16 May 2016 20:51:26 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Rainer Duffner cc: FreeBSD Filesystems Subject: Re: zfs receive stalls whole system In-Reply-To: <0C2233A9-C64A-4773-ABA5-C0BCA0D037F0@ultra-secure.de> Message-ID: References: <0C2233A9-C64A-4773-ABA5-C0BCA0D037F0@ultra-secure.de> User-Agent: Alpine 2.20 (GSO 67 2015-01-07) MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Mon, 16 May 2016 20:51:27 -0500 (CDT) Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 01:51:28 -0000 On Tue, 17 May 2016, Rainer Duffner wrote: > > It’s very difficult (well, impossible) to debug, because the system > totally hangs and doesn’t accept any keypresses. > > Would a ZIL help in this case? > I always thought that NFS was the only thing that did SYNC writes… This sounds like a hardware or driver problem. A dedicated ZIL won't help a system which entirely hangs. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Tue May 17 03:59:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34572B3EDD1 for ; Tue, 17 May 2016 03:59:57 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from connect.ultra-secure.de (connect.ultra-secure.de [88.198.71.201]) by mx1.freebsd.org (Postfix) with ESMTP id 91AB4115F for ; Tue, 17 May 2016 03:59:56 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (Haraka outbound); Tue, 17 May 2016 05:59:54 +0200 Authentication-Results: connect.ultra-secure.de; iprev=pass; auth=pass (plain); spf=none smtp.mailfrom=ultra-secure.de Received-SPF: None (connect.ultra-secure.de: domain of ultra-secure.de does not designate 217.71.83.52 as permitted sender) receiver=connect.ultra-secure.de; identity=mailfrom; client-ip=217.71.83.52; helo=[192.168.1.200]; envelope-from= Received: from [192.168.1.200] (217-071-083-052.ip-tech.ch [217.71.83.52]) by connect.ultra-secure.de (Haraka/2.6.2-toaster) with ESMTPSA id DBC21CD3-3F3C-4C27-B0CE-EA8A86995E60.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=AES256-SHA verify=NO); Tue, 17 May 2016 05:59:47 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: zfs receive stalls whole system From: Rainer Duffner In-Reply-To: Date: Tue, 17 May 2016 05:59:45 +0200 Cc: FreeBSD Filesystems Content-Transfer-Encoding: quoted-printable Message-Id: <3E271E07-F60E-4181-B8B0-9ED2CFCDF5A0@ultra-secure.de> References: <0C2233A9-C64A-4773-ABA5-C0BCA0D037F0@ultra-secure.de> To: Bob Friesenhahn X-Mailer: Apple Mail (2.3124) X-Haraka-GeoIP: EU, CH, 451km X-Haraka-ASN: 24951 X-Haraka-GeoIP-Received: X-Haraka-ASN: 24951 217.71.80.0/20 X-Haraka-ASN-CYMRU: asn=24951 net=217.71.80.0/20 country=CH assignor=ripencc date=2003-08-07 X-Haraka-FCrDNS: 217-071-083-052.ip-tech.ch X-Haraka-p0f: os="Mac OS X " link_type="DSL" distance=13 total_conn=1 shared_ip=N X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on spamassassin X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1 X-Haraka-Karma: score: 6, good: 170, bad: 0, connections: 330, history: 170, asn_score: 104, asn_connections: 115, asn_good: 104, asn_bad: 0, pass:all_good, asn, asn_all_good, relaying X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 03:59:57 -0000 > Am 17.05.2016 um 03:51 schrieb Bob Friesenhahn = : >=20 > On Tue, 17 May 2016, Rainer Duffner wrote: >>=20 >> It=E2=80=99s very difficult (well, impossible) to debug, because the = system totally hangs and doesn=E2=80=99t accept any keypresses. >>=20 >> Would a ZIL help in this case? >> I always thought that NFS was the only thing that did SYNC writes=E2=80= =A6 >=20 > This sounds like a hardware or driver problem. A dedicated ZIL won't = help a system which entirely hangs. When I rebuilt these systems, I started with the 2nd one, the = standby-system. I zfs sent 5 or 6T worth of data from the original system to it and it = was very fast. I got 600 MBit flat out of it. Then, I made that system master while I rebuilt the other one. When I synced back, I got maybe 500MBit on the zfs sends. And I started to see these stalls on sending updates. Could this be a problem: (nfs2-prod ) 1 # sysctl -a |grep mps |grep = "driver_version\|firmware_version" dev.mps.2.driver_version: 20.00.00.00-fbsd dev.mps.2.firmware_version: 15.10.01.00 dev.mps.1.driver_version: 20.00.00.00-fbsd dev.mps.1.firmware_version: 13.10.53.00 dev.mps.0.driver_version: 20.00.00.00-fbsd dev.mps.0.firmware_version: 13.10.53.00 As per this thread: = https://forums.freenas.org/index.php?threads/9-3-1-update-with-alert-firmw= are-version-16-does-not-match-driver-version-20-for-dev-mps0.36536/page-4 I will have to ask HP for a newer firmware then=E2=80=A6