From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 01:09:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FF9F106566C for ; Sun, 28 Feb 2010 01:09:17 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id E1B4D8FC15 for ; Sun, 28 Feb 2010 01:09:15 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 8371C45DD8; Sun, 28 Feb 2010 02:09:14 +0100 (CET) Received: from localhost (shell.garage.wheel.pl [10.0.4.3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id E609E45684; Sun, 28 Feb 2010 02:09:08 +0100 (CET) Date: Sun, 28 Feb 2010 02:09:06 +0100 From: Pawel Jakub Dawidek To: Dimitry Andric Message-ID: <20100228010906.GD2586@garage.freebsd.pl> References: <4B7FD4D2.7040802@andric.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MPkR1dXiUZqK+927" Content-Disposition: inline In-Reply-To: <4B7FD4D2.7040802@andric.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: zfs on 4k sector disks X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 01:09:17 -0000 --MPkR1dXiUZqK+927 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Feb 20, 2010 at 01:25:54PM +0100, Dimitry Andric wrote: > On 2010-02-20 10:47, mailinglists wrote: > >This "Advanced Format" basically means the sectors size on disk is > >changed to 4k in stead of 512k. >=20 > Please note those disks still seem to expose 512B sectors to any OS; the > 4kiB sectors are only used internally. And there seems to be no jumper > to "fix" this behaviour... Unaligned writes can cause multiple > read-write-modify operations, which are most likely bad for performance. You can "fix" that with: # gnop create -S 4096 /dev/ad0 This command will create /dev/ad0.nop provider, which will expose 4kB sector (see diskinfo(8)). Unfortunately gnop won't survive reboot, as it doesn't use any metadata. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --MPkR1dXiUZqK+927 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuJwjEACgkQForvXbEpPzQ7SQCfbE40UkwzUdQtI6UwpcLKRxAE hjIAoJ+zRgrlQYIq23CUYy961nesw7+O =SN+L -----END PGP SIGNATURE----- --MPkR1dXiUZqK+927-- From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 05:40:54 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD784106566B; Sun, 28 Feb 2010 05:40:54 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f191.google.com (mail-iw0-f191.google.com [209.85.223.191]) by mx1.freebsd.org (Postfix) with ESMTP id 98D458FC08; Sun, 28 Feb 2010 05:40:53 +0000 (UTC) Received: by iwn29 with SMTP id 29so721866iwn.27 for ; Sat, 27 Feb 2010 21:40:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=/G5gucsMHx6ECBVhcjXb2b13/N3E1Ds42kIoK9zg6V8=; b=B81FZGlFc6GmtgMlix/TZUwZjFWXGjXSBpPxAL7Lu4botEtW/FabSA8toB/efZl3Ik XDNnfa7mOVMU504UZ/jh3lZhXlZ+sG+Pl5UPq4PrQ+aG23saTyq9vMr8Ew7vXRi9XWoy +nG1g7/514F1sFukdFAmsNGtoxZLF/MrSCJ9w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=sbQodAY5ntYlwCu2s6lJSpj6GyoPKBwhgCt7emJ8943EMXDx4i4rcCtRyDAL8oVFRJ AUnty42ZtraHQpwHb8xJYjN3hw/fEiVlMOYqAmgUTZGr6/a/KfNBUrridQuCYMyOqL0u oS6zN5+V3fDTT4wvW6nTCCM0fT1P47wcTGrSQ= MIME-Version: 1.0 Received: by 10.231.151.197 with SMTP id d5mr1480136ibw.73.1267335648334; Sat, 27 Feb 2010 21:40:48 -0800 (PST) In-Reply-To: <20100227204410.GB2586@garage.freebsd.pl> References: <20100227204410.GB2586@garage.freebsd.pl> Date: Sat, 27 Feb 2010 21:40:48 -0800 Message-ID: From: Freddie Cash To: Pawel Jakub Dawidek Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: fs@freebsd.org Subject: Re: HAST: file name consistency X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 05:40:54 -0000 On Sat, Feb 27, 2010 at 12:44 PM, Pawel Jakub Dawidek wrote: > Does that make sense for you? > > Yeah, that makes sense. Thanks. :) So far, in testing via VirtualBox, things are working quite nicely using HAST to mirror a zpool. Very nicely done!! Now I just have to wait for our fibre link between data centres to go live to test it on real hardware. :D -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 12:11:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF4D2106564A; Sun, 28 Feb 2010 12:11:47 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 89B018FC0C; Sun, 28 Feb 2010 12:11:47 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1Nlhzr-0004ci-3W; Sun, 28 Feb 2010 14:11:43 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Jeremy Chadwick In-reply-to: <20100227220310.GA65110@icarus.home.lan> References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Jeremy Chadwick message dated "Sat, 27 Feb 2010 14:03:10 -0800." Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Sun, 28 Feb 2010 14:11:42 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , Eirik =?iso-8859-1?Q?=D8verby?= , rwatson@freebsd.org Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 12:11:48 -0000 > On Sat, Feb 27, 2010 at 10:53:00PM +0100, Willem Jan Withagen wrote: > > On 27-2-2010 21:32, Eirik =D8verby wrote: > > >I've had a discussion with some folks on this for a while. I can eas= il=3Dy > > >reproduce this situation by mounting a FreeBSD ZFS filesystem via > > >NFS-UDP from an OpenBSD machine. Telling the OpenBSD machine to use = TC=3DP > > >instead of UDP makes the problem go away. > > > > > >Other FreeBSD systems mounting the same share, either using UDP or T= CP=3D, > > >does not cause the problem to show up. > > > > > >A patch was suggested by Rick Macklem, but that did not solve the is= su=3De: > > >http://lists.freebsd.org/pipermail/freebsd-current/2009-December/014= 18=3D1.html> >=20 > > I concur. > > Everything in my network is now on TCP, and there is no mbuf leakage.= > > I just don't get over the 5500 mark, no matter what I throw at it. > >=20 > > I do feel that TCP is not as well performing on a local net with Linu= x, > > hence the choice for UDP. But TCP is workable as next best. >=20 > I'm pulling in Robert Watson, who has some familiarity with the UDP > stack/code in FreeBSD. I'm not sure he'll be a sufficient source of > knowledge for this specific issue since it appears (?) to be specific t= o > NFS; Rick Macklem would be a better choice, but as reported, he's MIA. >=20 > Robert, are you aware of any changes or implementation issues which > might cause excessive (read: leaking) mbuf use under UDP-based NFS? Do= > you know of a way folks could determine the source of the leak, either > via DDB or while the system is live? I have been runing some tests in a controlled environment. server and client are both 64bit Xeon/X5550 =40 2.67GHz with 16Gb of mem= ory FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads the client is runing latest 8.0 stable the load is created by runing 'make -j32 buildworld' and sleeping 150 sec= . in between runs, this is the straight line you will see in the graphs. Both the src and obj directories are NFS mounted from the server, regular= UFS. when server is running 7.2-stable no leakage is seen. see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-7.2.= ps when server is runing 8.0-stable see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-8.0.= ps you can see that udp is leaking=21 cheers, danny ps: I think the subject should be changed again, removing zfs ... From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 12:21:32 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B02151065672; Sun, 28 Feb 2010 12:21:32 +0000 (UTC) (envelope-from rwatson@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6E2758FC14; Sun, 28 Feb 2010 12:21:32 +0000 (UTC) Received: from [192.168.2.102] (host86-181-230-35.range86-181.btcentralplus.com [86.181.230.35]) by cyrus.watson.org (Postfix) with ESMTPSA id 9F06646B17; Sun, 28 Feb 2010 07:21:30 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: Date: Sun, 28 Feb 2010 12:21:28 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> To: Daniel Braniss X-Mailer: Apple Mail (2.1077) Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?iso-8859-1?Q?Eirik_=D8verby?= , Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 12:21:32 -0000 On Feb 28, 2010, at 12:11 PM, Daniel Braniss wrote: >> I'm pulling in Robert Watson, who has some familiarity with the UDP >> stack/code in FreeBSD. I'm not sure he'll be a sufficient source of >> knowledge for this specific issue since it appears (?) to be specific = to >> NFS; Rick Macklem would be a better choice, but as reported, he's = MIA. >>=20 >> Robert, are you aware of any changes or implementation issues which >> might cause excessive (read: leaking) mbuf use under UDP-based NFS? = Do >> you know of a way folks could determine the source of the leak, = either >> via DDB or while the system is live? >=20 > I have been runing some tests in a controlled environment. >=20 > server and client are both 64bit Xeon/X5550 @ 2.67GHz with 16Gb of = memory > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads >=20 > the client is runing latest 8.0 stable > the load is created by runing 'make -j32 buildworld' and sleeping 150 = sec. > in between runs, this is the straight line you will see in the graphs. > Both the src and obj directories are NFS mounted from the server, = regular UFS. >=20 > when server is running 7.2-stable no leakage is seen. > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/{tcp,udp}-7.2.ps > when server is runing 8.0-stable > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/{tcp,udp}-8.0.ps > you can see that udp is leaking! >=20 > cheers, > danny > ps: I think the subject should be changed again, removing zfs ... This type of problem (occurs with one client but not another) is almost = always the result of the access pattern of a particular client = triggering a specific (and perhaps single) bug in error-handling. For = example, we might not be properly freeing the received request when = generating an EPERM in an edge case. The hard bit is identifying which = it is. If it's reproducible with UDP, then usually the process is: - Build a minimal test case to trigger the problem -- ideally with as = little complexity as possible. - Run netstat -m at the beginning of the test and the end of the test on = the server to count the number of leaked mbufs - Run wireshark throughout the test - Walk the wireshark trace looking for some error that occurs at about = the same or slightly lower number of times then the number of mbufs = leaked - Iterate, narrowing the test case until it's either obvious exactly = what's going on, or you've identified a relatively constrained code path = and can just spot the bug by reading the code It's almost certainly one or a small number of very specific RPCs that = are triggering it -- maybe OpenBSD does an extra lookup, or stat, or = something, on a name that may not exist anymore, or does it sooner than = the other clients. Hard to say, other than to wave hands at the = possibilities. And it may well be we're looking at two bugs: Danny may see one bug, = perhaps triggered by a race condition, but it may be different from the = OpenBSD client-triggered bug (to be clear: it's definitely a FreeBSD = bug, although we might only see it when an OpenBSD client is used = because perhaps OpenBSD also has a bug or feature). Robert= From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 14:52:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C96A8106564A; Sun, 28 Feb 2010 14:52:47 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 736698FC14; Sun, 28 Feb 2010 14:52:47 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1NlkVg-00068M-Qk; Sun, 28 Feb 2010 16:52:44 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: "Robert N. M. Watson" In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to "Robert N. M. Watson" message dated "Sun, 28 Feb 2010 12:21:28 +0000." Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Sun, 28 Feb 2010 16:52:44 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?iso-8859-1?Q?Eirik_=D8verby?= , Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was: mbuf leakage with nfs/zfs) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 14:52:47 -0000 > > On Feb 28, 2010, at 12:11 PM, Daniel Braniss wrote: >=20 > >> I'm pulling in Robert Watson, who has some familiarity with the UDP > >> stack/code in FreeBSD. I'm not sure he'll be a sufficient source of= > >> knowledge for this specific issue since it appears (?) to be specifi= c =3Dto > >> NFS; Rick Macklem would be a better choice, but as reported, he's = =3DMIA. > >>=20 > >> Robert, are you aware of any changes or implementation issues which > >> might cause excessive (read: leaking) mbuf use under UDP-based NFS? = =3DDo > >> you know of a way folks could determine the source of the leak, =3De= ither > >> via DDB or while the system is live? > >=20 > > I have been runing some tests in a controlled environment. > > > > server and client are both 64bit Xeon/X5550 =40 2.67GHz with 16G= b of =3Dmemory > > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads > > > > the client is runing latest 8.0 stable > > the load is created by runing 'make -j32 buildworld' and sleeping 150= =3Dsec. > > in between runs, this is the straight line you will see in the graphs= . > > Both the src and obj directories are NFS mounted from the server, =3D= regular UFS. > > > > when server is running 7.2-stable no leakage is seen. > > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-7= .2.ps > > when server is runing 8.0-stable > > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/=7Btcp,udp=7D-8= .0.ps > > you can see that udp is leaking=21 > > > > cheers, > > danny > > ps: I think the subject should be changed again, removing zfs ... > > This type of problem (occurs with one client but not another) is almo= st =3Dalways the result of the access pattern of a particular client =3Dt= riggering a specific (and perhaps single) bug in error-handling. For =3De= xample, we might not be properly freeing the received request when =3Dgen= erating an EPERM in an edge case. The hard bit is identifying which =3Dit= is. If it's reproducible with UDP, then usually the process is: > > - Build a minimal test case to trigger the problem -- ideally with as= =3Dlittle complexity as possible. > - Run netstat -m at the beginning of the test and the end of the test o= n =3Dthe server to count the number of leaked mbufs > - Run wireshark throughout the test > - Walk the wireshark trace looking for some error that occurs at about = =3Dthe same or slightly lower number of times then the number of mbufs = =3Dleaked > - Iterate, narrowing the test case until it's either obvious exactly = =3Dwhat's going on, or you've identified a relatively constrained code pa= th =3Dand can just spot the bug by reading the code > > It's almost certainly one or a small number of very specific RPCs tha= t =3Dare triggering it -- maybe OpenBSD does an extra lookup, or stat, or= =3Dsomething, on a name that may not exist anymore, or does it sooner th= an =3Dthe other clients. Hard to say, other than to wave hands at the =3D= possibilities. > > And it may well be we're looking at two bugs: Danny may see one bug, = =3Dperhaps triggered by a race condition, but it may be different from th= e =3DOpenBSD client-triggered bug (to be clear: it's definitely a FreeBSD= =3Dbug, although we might only see it when an OpenBSD client is used =3D= because perhaps OpenBSD also has a bug or feature). > > Robert=3D well, I have further reduced the problem, it happens with NFS/UDP writes.= i'll try the wireshark road, but i'm very rusty with RPC, the other road = is to check the changes, my oldest is from late october (RC2) where it's happen= ing,=20 while Gerrit tried 8-pre from November and worked, so it will be fun trying to nail it down :-) cheers, danny From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 14:56:08 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2379106566B; Sun, 28 Feb 2010 14:56:08 +0000 (UTC) (envelope-from rwatson@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id A251F8FC23; Sun, 28 Feb 2010 14:56:08 +0000 (UTC) Received: from [192.168.2.102] (host86-181-230-35.range86-181.btcentralplus.com [86.181.230.35]) by cyrus.watson.org (Postfix) with ESMTPSA id 348D146B29; Sun, 28 Feb 2010 09:56:07 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: Date: Sun, 28 Feb 2010 14:56:05 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> To: Daniel Braniss X-Mailer: Apple Mail (2.1077) Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?iso-8859-1?Q?Eirik_=D8verby?= , Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was: mbuf leakage with nfs/zfs) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 14:56:08 -0000 On Feb 28, 2010, at 2:52 PM, Daniel Braniss wrote: > well, I have further reduced the problem, it happens with NFS/UDP = writes. > i'll try the wireshark road, but i'm very rusty with RPC, the other = road is to > check the changes, my oldest is from late october (RC2) where it's = happening,=20 > while > Gerrit tried 8-pre from November and worked, so it will be fun > trying to nail it down :-) Fortunately, Wireshark actually has quite a good NFS RPC decoder -- it = will tell you what operation appears, what the arguments are, interpret = NFS error codes, etc. In fact, it's an excellent way to learn about = NFS... Robert= From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 17:56:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C887106566C; Sun, 28 Feb 2010 17:56:29 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id D1C198FC2E; Sun, 28 Feb 2010 17:56:28 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id o1SHtlWD029132; Sun, 28 Feb 2010 18:55:51 +0100 Received: from pmp.uni-hannover.de (theq.pmp.uni-hannover.de [130.75.117.4]) by www.pmp.uni-hannover.de (Postfix) with SMTP id AE49524; Sun, 28 Feb 2010 18:55:47 +0100 (CET) Date: Sun, 28 Feb 2010 18:55:46 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: "Robert N. M. Watson" Message-Id: <20100228185546.f9fefa97.gerrit@pmp.uni-hannover.de> In-Reply-To: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.4.2 (GTK+ 2.10.12; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.388399, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2010.2.28.174226 Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , Eirik =?ISO-8859-1?Q?=D8verby?= , Daniel Braniss , Jeremy Chadwick Subject: Re: mbuf leakage with nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 17:56:29 -0000 On Sun, 28 Feb 2010 12:21:28 +0000 "Robert N. M. Watson" wrote about Re: mbuf leakage with nfs/zfs? : RNMW> It's almost certainly one or a small number of very specific RPCs RNMW> that are triggering it -- maybe OpenBSD does an extra lookup, or RNMW> stat, or something, on a name that may not exist anymore, or does it RNMW> sooner than the other clients. Hard to say, other than to wave hands RNMW> at the possibilities. RNMW> RNMW> And it may well be we're looking at two bugs: Danny may see one bug, RNMW> perhaps triggered by a race condition, but it may be different from RNMW> the OpenBSD client-triggered bug (to be clear: it's definitely a RNMW> FreeBSD bug, although we might only see it when an OpenBSD client is RNMW> used because perhaps OpenBSD also has a bug or feature). In my case it is the Linux client causing the problems (cannot tell yet if it is only with udp, but I would think so). If I understand Daniel correctly his latest testes were performed with FreeBSD client and udp. So it may very well be a generel issue with udp?! Would this help narrowing down the problem? cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Sun Feb 28 17:58:20 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD3E8106566B; Sun, 28 Feb 2010 17:58:20 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id 3D9FF8FC1C; Sun, 28 Feb 2010 17:58:19 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id o1SHvhX6029177; Sun, 28 Feb 2010 18:57:44 +0100 Received: from pmp.uni-hannover.de (theq.pmp.uni-hannover.de [130.75.117.4]) by www.pmp.uni-hannover.de (Postfix) with SMTP id A275B24; Sun, 28 Feb 2010 18:57:43 +0100 (CET) Date: Sun, 28 Feb 2010 18:57:43 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Daniel Braniss Message-Id: <20100228185743.1afcc712.gerrit@pmp.uni-hannover.de> In-Reply-To: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.4.2 (GTK+ 2.10.12; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.388399, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2010.2.28.174226 Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , "Robert N. M. Watson" , Eirik =?ISO-8859-1?Q?=D8verby?= , Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was: mbuf leakage with nfs/zfs) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 17:58:20 -0000 On Sun, 28 Feb 2010 16:52:44 +0200 Daniel Braniss wrote about Re: mbuf leakage with nfs/udp (was: mbuf leakage with nfs/zfs): DB> well, I have further reduced the problem, it happens with NFS/UDP DB> writes. i'll try the wireshark road, but i'm very rusty with RPC, the DB> other road is to check the changes, my oldest is from late october DB> (RC2) where it's happening, while DB> Gerrit tried 8-pre from November and worked, so it will be fun DB> trying to nail it down :-) I already withdrew from this position yesterday, because the 8-PRE server I have does not have udp clients, only tcp. So I cannot tell (yet) wether it is affected by the leakage or not. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 05:24:46 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40FD7106566B; Mon, 1 Mar 2010 05:24:46 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id 706298FC1D; Mon, 1 Mar 2010 05:24:45 +0000 (UTC) Received: by fxm23 with SMTP id 23so320556fxm.3 for ; Sun, 28 Feb 2010 21:24:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=MUtVXr/4sPD/BOFV/lUJSDL6Vw4mW58RxgAw6ns8yCA=; b=wQ/YBDHXZjueud9GzmfBPzC9L1C7ubOCqPCUfhIXFqU+V2JRRHmmM3BRjQtuJtgZfO SRtBy485bT9WvW+x2Rf9zcH23AOCuCYELPe9Xa3mVNyfYxd8SUaGniDh3f9dQouVNceU 6Q5oE6IVTMmT62fcbtmMWfHa6NuhUhs5b0xfQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=oQIgfojMumU1/WN4FFYg7Rk7fWnW6on4aCRek8sEW320BCI7v575mO0LX5YsoOtpFp umHmmCH2SKyWEosePTVVXmbxPshhPPdP9vusN2kOhGcFOMY5tpnaCwTxpwTX/4lJ7hxy i0lRCfNZp8zY04P5N8vvu5dR6BfAbwk4s03jI= Received: by 10.223.36.92 with SMTP id s28mr4410813fad.28.1267421075677; Sun, 28 Feb 2010 21:24:35 -0800 (PST) Received: from mbp-gige.totalterror.net (93-152-151-19.ddns.onlinedirect.bg [93.152.151.19]) by mx.google.com with ESMTPS id 18sm4669095fks.5.2010.02.28.21.24.34 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 28 Feb 2010 21:24:34 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <20100227204715.GC2586@garage.freebsd.pl> Date: Mon, 1 Mar 2010 07:24:31 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <26CCAE99-B056-4754-9324-2945D1643C78@gmail.com> <272C3933-72C9-41AF-AD3A-605A7C57FB61@gmail.com> <20100227204715.GC2586@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@freebsd.org Subject: Re: should zfs care so much about device name changes X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 05:24:46 -0000 On 27 Feb, 2010, at 22:47 , Pawel Jakub Dawidek wrote: > On Wed, Feb 24, 2010 at 03:58:58PM +0200, Nikolay Denev wrote: >>=20 >> On Feb 24, 2010, at 11:53 AM, Nikolay Denev wrote: >>=20 >>> Hello, >>>=20 >>> I wanted to test the new option ATA_CAM, but that would require a = boot from another media (USB Drive/CD) and a zpool export/import >>> to update the vdev names. (actually it's shown as "path" in the = zpool.cache file) because otherwise the system would refuse to = open/mount >>> the pool. >>> But is that really necessary given the fact that all the devices are = here, and have matching GUIDS to those in the zpool.cache file? >>> Shouldnt ZFS just import the pool?=20 >>>=20 >>> In the current state what would one have to do to for example test = ATA_CAM on a remote machine, where export/import from a rescue >>> media is not possible? >>>=20 >>> Thanks, >>> Niki >>=20 >>=20 >> I have now looked at = sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c and it seems = that when a vdev isn't found by pathname it >> is being searched by guid. But it doesn't seem to be the case here. = The loader sees the pool and boots the kernel, but then the kernel does = not see >> the pool. Any ideas? >=20 > Try adding vfs.zfs.debug=3D1 to /boot/loader.conf. This will hopefully > show what's going on. >=20 > --=20 > Pawel Jakub Dawidek http://www.wheel.pl > pjd@FreeBSD.org http://www.FreeBSD.org > FreeBSD committer Am I Evil? Yes, I Am! Sorry for the noise, it seems like it's not zfs's fault, the machine = just doesn't see any disks because I added ATA_CAM but missed atacore and atapci. I had to capture the boot screen with my camera because my USB keyboard = does not work on rootfs mount prompt and I was not able scroll up to see what's = the cause, but that's another problem that deserves another thread not in -fs. Thanks, Niki From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 05:58:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42156106566B for ; Mon, 1 Mar 2010 05:58:26 +0000 (UTC) (envelope-from james-freebsd-fs2@jrv.org) Received: from mail.jrv.org (adsl-70-243-84-13.dsl.austtx.swbell.net [70.243.84.13]) by mx1.freebsd.org (Postfix) with ESMTP id CC2738FC1A for ; Mon, 1 Mar 2010 05:58:25 +0000 (UTC) Received: from kremvax.housenet.jrv (kremvax.housenet.jrv [192.168.3.124]) by mail.jrv.org (8.14.3/8.14.3) with ESMTP id o215wORe011114 for ; Sun, 28 Feb 2010 23:58:24 -0600 (CST) (envelope-from james-freebsd-fs2@jrv.org) Authentication-Results: mail.jrv.org; domainkeys=pass (testing) header.from=james-freebsd-fs2@jrv.org DomainKey-Signature: a=rsa-sha1; s=enigma; d=jrv.org; c=nofws; q=dns; h=message-id:date:from:user-agent:mime-version:to:subject: content-type:content-transfer-encoding; b=FcW7LMp1TajMr4pY0kfL8aV3zZVN7wgf7cxr2VqNbKWJlgSW7noSdZQyA2sFK4s+a OydHEsdO/jhTrLdv6atxiyfacqOKY3c1q6OgbkywcEWopB9N7rDAjNkE+iJ/+TXUb7M 4o8tN6patZkK5FC07CehTcpqkekv0B7zgSeRcZM= Message-ID: <4B8B5780.2050601@jrv.org> Date: Sun, 28 Feb 2010 23:58:24 -0600 From: "James R. Van Artsdalen" User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: freebsd-fs Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: [zfs] attach by name/uuid still attaches wrong device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 05:58:26 -0000 FreeBSD bigtex.housenet.jrv 9.0-CURRENT FreeBSD 9.0-CURRENT #2 r200727M: Tue Dec 22 23:25:56 CST 2009 james@bigtex.housenet.jrv:/usr/obj/usr/src/sys/BIGTEX amd64 It appears the zfs/vdev_geom.c can still attach to the wrong device in some cases. Note in the zpool status output how ada10 appears in two different vdevs. What happened is that a disk failed completely (scbus3 target 3) and is no longer detected by the driver. At boot time: 1. ZFS fails to attach by path and UUID, since what was at ada11 is now at ada10 and has a different UUID. 2. ZFS fails to attach by UUID since that UUID is on a dead drive and can no longer be found anywhere. 3. ZFS then attaches by path blindly, even though that drive is in a different part of the pool and has a different UUID. I don't think it's possible to do this right in vdev_geom.c: there's no way to guess what is intended without a hint from higher ZFS layers as to which drives should be found and which are new. The best fixes I can think of are to expose drives by serial number in GEOM, or perhaps as a fall-back expose names that are geographic locations, i.e., "/dev/scbus0/target3/lun0". # zpool status pool: bigtex state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAME STATE READ WRITE CKSUM bigtex DEGRADED 0 0 0 mirror ONLINE 0 0 0 ada6 ONLINE 0 0 0 ada13 ONLINE 0 0 0 mirror ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada11 ONLINE 0 0 0 mirror ONLINE 0 0 0 gptid/dbb5f9fd-5e40-11de-bef4-001aa01b0286 ONLINE 0 0 0 ada2p7 ONLINE 0 0 0 mirror ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada14 ONLINE 0 0 0 mirror ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada10 ONLINE 0 0 0 mirror ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada12 ONLINE 0 0 0 mirror ONLINE 0 0 0 ada9 ONLINE 0 0 0 ada15 ONLINE 0 0 0 mirror DEGRADED 0 0 0 ada10 FAULTED 10 754K 0 corrupted data ada16 ONLINE 0 0 0 errors: No known data errors # camcontrol devlist at scbus0 target 0 lun 0 (ada2,pass6) at scbus0 target 1 lun 0 (ada3,pass7) at scbus0 target 2 lun 0 (ada4,pass8) at scbus0 target 3 lun 0 (ada5,pass9) at scbus0 target 15 lun 0 (pass0,pmp0) at scbus3 target 0 lun 0 (ada6,pass10) at scbus3 target 1 lun 0 (ada7,pass11) at scbus3 target 2 lun 0 (ada9,pass13) at scbus3 target 15 lun 0 (pass1,pmp1) at scbus4 target 0 lun 0 (ada8,pass12) at scbus4 target 1 lun 0 (ada10,pass14) at scbus4 target 2 lun 0 (ada11,pass15) at scbus4 target 3 lun 0 (ada12,pass16) at scbus4 target 15 lun 0 (pass2,pmp2) at scbus7 target 0 lun 0 (ada13,pass17) at scbus7 target 1 lun 0 (ada14,pass18) at scbus7 target 2 lun 0 (ada15,pass19) at scbus7 target 3 lun 0 (ada16,pass20) at scbus7 target 15 lun 0 (pass3,pmp3) at scbus8 target 0 lun 0 (pass4,ada0) at scbus11 target 0 lun 0 (pass5,ada1) # grep ada10 /var/run/dmesg.boot vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_open_by_path:466[1]: Found provider by name /dev/ada10. vdev_geom_attach:112[1]: Attaching to ada10. vdev_geom_attach:138[1]: Found consumer for ada10. vdev_geom_attach:157[1]: Used existing consumer for ada10. vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_detach:173[1]: Closing access to ada10. vdev_geom_open_by_path:477[1]: guid mismatch for provider /dev/ada10: 3665972767133355802 != 12768899409278570370. vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_open_by_path:466[1]: Found provider by name /dev/ada10. vdev_geom_attach:112[1]: Attaching to ada10. vdev_geom_attach:138[1]: Found consumer for ada10. vdev_geom_attach:157[1]: Used existing consumer for ada10. vdev_geom_detach:173[1]: Closing access to ada10. vdev_geom_detach:173[1]: Closing access to ada10. vdev_geom_detach:177[1]: Destroyed consumer to ada10. vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_attach:112[1]: Attaching to ada10. vdev_geom_attach:153[1]: Created consumer for ada10. vdev_geom_open_by_guid:446[1]: Attach by guid [12768899409278570370] succeeded, provider /dev/ada10. vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_open_by_path:466[1]: Found provider by name /dev/ada10. vdev_geom_attach:112[1]: Attaching to ada10. vdev_geom_attach:138[1]: Found consumer for ada10. vdev_geom_attach:157[1]: Used existing consumer for ada10. vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_detach:173[1]: Closing access to ada10. vdev_geom_open_by_path:477[1]: guid mismatch for provider /dev/ada10: 3665972767133355802 != 12768899409278570370. vdev_geom_read_guid:301[1]: Reading guid from ada10... vdev_geom_read_guid:339[1]: guid for ada10 is 12768899409278570370 vdev_geom_open_by_path:466[1]: Found provider by name /dev/ada10. vdev_geom_attach:112[1]: Attaching to ada10. vdev_geom_attach:138[1]: Found consumer for ada10. vdev_geom_attach:157[1]: Used existing consumer for ada10. vdev_geom_detach:173[1]: Closing access to ada10. # From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 11:06:59 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE06E10656A6 for ; Mon, 1 Mar 2010 11:06:59 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B08388FC0A for ; Mon, 1 Mar 2010 11:06:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o21B6xOM017775 for ; Mon, 1 Mar 2010 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o21B6xAG017773 for freebsd-fs@FreeBSD.org; Mon, 1 Mar 2010 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 1 Mar 2010 11:06:59 GMT Message-Id: <201003011106.o21B6xAG017773@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 11:06:59 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/144330 fs [nfs] mbuf leakage in nfsd with zfs o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o bin/144214 fs zfsboot fails on gang block after upgrade to zfs v14 o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o kern/143345 fs [ext2fs] [patch] extfs minor header cleanups to better o kern/143343 fs [zfs] bug in sunlink flag on directories o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142924 fs [ext2fs] [patch] Small cleanup for the inode struct in o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142872 fs [zfs] ZFS ZVOL Lockmgr Deadlock o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142594 fs [zfs] Modification time reset to 1 Jan 1970 after fsyn o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs/ufs Lock order reversal o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141718 fs [zfs] [panic] kernel panic when 'zfs rename' is used o o kern/141685 fs [zfs] zfs corruption on adaptec 5805 raid controller o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141257 fs [gvinum] No puedo crear RAID5 por SW con gvinum o kern/141177 fs [zfs] fsync() on FIFO causes panic() on zfs o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140682 fs [netgraph] [panic] random panic in netgraph o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140433 fs [zfs] [panic] panic while replaying ZIL after crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138524 fs [msdosfs] disks and usb flashes/cards with Russian lab o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 164 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 11:50:44 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A004F106566C; Mon, 1 Mar 2010 11:50:44 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id 2E1308FC21; Mon, 1 Mar 2010 11:50:44 +0000 (UTC) Received: from localhost (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 19F47153433; Mon, 1 Mar 2010 12:50:43 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by localhost (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RNVEs56HfWuG; Mon, 1 Mar 2010 12:50:41 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id 0EA0B15342F; Mon, 1 Mar 2010 12:50:41 +0100 (CET) Message-ID: <4B8BAA80.7060401@digiware.nl> Date: Mon, 01 Mar 2010 12:52:32 +0100 From: Willem Jan Withagen Organization: Digiware User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Gerrit_K=FChn?= References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> <20100228185546.f9fefa97.gerrit@pmp.uni-hannover.de> In-Reply-To: <20100228185546.f9fefa97.gerrit@pmp.uni-hannover.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: stable@freebsd.org, freebsd-fs@freebsd.org, "Robert N. M. Watson" , =?ISO-8859-1?Q?Eirik_=D8verby?= , Daniel Braniss , Jeremy Chadwick Subject: Re: mbuf leakage with nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 11:50:44 -0000 On 28-2-2010 18:55, Gerrit Kühn wrote: > On Sun, 28 Feb 2010 12:21:28 +0000 "Robert N. M. Watson" > wrote about Re: mbuf leakage with nfs/zfs? : > > RNMW> It's almost certainly one or a small number of very specific RPCs > RNMW> that are triggering it -- maybe OpenBSD does an extra lookup, or > RNMW> stat, or something, on a name that may not exist anymore, or does it > RNMW> sooner than the other clients. Hard to say, other than to wave hands > RNMW> at the possibilities. > RNMW> > RNMW> And it may well be we're looking at two bugs: Danny may see one bug, > RNMW> perhaps triggered by a race condition, but it may be different from > RNMW> the OpenBSD client-triggered bug (to be clear: it's definitely a > RNMW> FreeBSD bug, although we might only see it when an OpenBSD client is > RNMW> used because perhaps OpenBSD also has a bug or feature). > > In my case it is the Linux client causing the problems (cannot tell yet if > it is only with udp, but I would think so). If I understand Daniel > correctly his latest testes were performed with FreeBSD client and udp. So > it may very well be a generel issue with udp?! Would this help narrowing > down the problem? I'm off 'till thursday. At which time I'm willing to run more tests. Got plenty of boxes here. Both FreeBSD and Linux. And otherwise will boot more in VirtualBox. --WjW From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 13:27:18 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B6F51065675; Mon, 1 Mar 2010 13:27:18 +0000 (UTC) (envelope-from joerg@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 62A268FC19; Mon, 1 Mar 2010 13:27:18 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o21DRIPv044326; Mon, 1 Mar 2010 13:27:18 GMT (envelope-from joerg@freefall.freebsd.org) Received: (from joerg@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o21DRIRG044322; Mon, 1 Mar 2010 14:27:18 +0100 (MET) (envelope-from joerg) Date: Mon, 1 Mar 2010 14:27:18 +0100 (MET) Message-Id: <201003011327.o21DRIRG044322@freefall.freebsd.org> To: tuxmen84@gmail.com, joerg@FreeBSD.org, freebsd-fs@FreeBSD.org From: joerg@FreeBSD.org Cc: Subject: Re: kern/141257: [gvinum] No puedo crear RAID5 por SW con gvinum X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 13:27:18 -0000 Synopsis: [gvinum] No puedo crear RAID5 por SW con gvinum State-Changed-From-To: open->closed State-Changed-By: joerg State-Changed-When: Mon Mar 1 14:23:19 MET 2010 State-Changed-Why: In order to create a RAID-5 (or striped) plex, you have to also provide the stripe size: volume ... plex org raid5 256k sd ... (Please submit bug reports in English, as this allows more people to understand it. I hope my poor Spanish knowledge was good enough to understand your actual problem.) http://www.freebsd.org/cgi/query-pr.cgi?pr=141257 From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 15:33:40 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A9DE1065670; Mon, 1 Mar 2010 15:33:40 +0000 (UTC) (envelope-from gerrit@pmp.uni-hannover.de) Received: from mrelay1.uni-hannover.de (mrelay1.uni-hannover.de [130.75.2.106]) by mx1.freebsd.org (Postfix) with ESMTP id AE0428FC13; Mon, 1 Mar 2010 15:33:39 +0000 (UTC) Received: from www.pmp.uni-hannover.de (www.pmp.uni-hannover.de [130.75.117.2]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id o21FWM8L006737; Mon, 1 Mar 2010 16:32:26 +0100 Received: from pmp.uni-hannover.de (arc.pmp.uni-hannover.de [130.75.117.1]) by www.pmp.uni-hannover.de (Postfix) with SMTP id 64E244F; Mon, 1 Mar 2010 16:32:22 +0100 (CET) Date: Mon, 1 Mar 2010 16:32:22 +0100 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Willem Jan Withagen Message-Id: <20100301163222.47d0b9f6.gerrit@pmp.uni-hannover.de> In-Reply-To: <4B8BAA80.7060401@digiware.nl> References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> <20100228185546.f9fefa97.gerrit@pmp.uni-hannover.de> <4B8BAA80.7060401@digiware.nl> Organization: Albert-Einstein-Institut (MPI =?ISO-8859-1?Q?f=FCr?= Gravitationsphysik & IGP =?ISO-8859-1?Q?Universit=E4t?= Hannover) X-Mailer: Sylpheed 2.7.1 (GTK+ 2.18.4; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PMX-Version: 5.5.9.388399, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2010.3.1.152129 Cc: stable@freebsd.org, freebsd-fs@freebsd.org, "Robert N. M. Watson" , Eirik =?ISO-8859-1?Q?=D8verby?= , Daniel Braniss , Jeremy Chadwick Subject: Re: mbuf leakage with nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 15:33:40 -0000 On Mon, 01 Mar 2010 12:52:32 +0100 Willem Jan Withagen wrote about Re: mbuf leakage with nfs: WJW> > In my case it is the Linux client causing the problems (cannot tell WJW> > yet if it is only with udp, but I would think so). If I understand WJW> > Daniel correctly his latest testes were performed with FreeBSD WJW> > client and udp. So it may very well be a generel issue with udp?! WJW> > Would this help narrowing down the problem? WJW> WJW> I'm off 'till thursday. WJW> At which time I'm willing to run more tests. Got plenty of boxes here. WJW> Both FreeBSD and Linux. And otherwise will boot more in VirtualBox. I finally too an axe and restarted nfsd without "-u". Now my mbuf usage is flat as it should be. I guess some people using computers with udp mounts will complian, but this can be fixed easily by converting their connections to tcp. However, I am still interested in having the issue fixed, so I will be following the thread and contribute if possible. cu Gerrit From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 18:37:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90808106566C; Mon, 1 Mar 2010 18:37:22 +0000 (UTC) (envelope-from jamesbrandongooch@gmail.com) Received: from mail-pz0-f197.google.com (mail-pz0-f197.google.com [209.85.222.197]) by mx1.freebsd.org (Postfix) with ESMTP id 28D098FC18; Mon, 1 Mar 2010 18:37:21 +0000 (UTC) Received: by pzk36 with SMTP id 36so63360pzk.8 for ; Mon, 01 Mar 2010 10:37:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=qb7HEq0DzLL5aj502ZUQFFRufU+cwvIFHA66gdQEaWk=; b=Wd1hT6AZ9UISfgXu0DQxwm/AMGGK4/G0hYxHHCGQASWTU0RzHcx3TPh/F9Wp6HKPTA dlWu+d9GUjiBT+rnB0AOM/KLOpTT0YnE7C6dlVe0R7U6a6lT3ZZq2P+JEAZwCSax8nYg xkEm3xg++7MvKMNgE8WW4R++DT+6u6frjhiPs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=RtCJ9WHwXKbyE7g5IUSOXH4l1GTrCKyDYdoSoMmeLxxr13oBbEH4H4htUvD9uEMjGF T9jwOS4xSeshSqUQq553d4D+8dbCieCGrYpcUdm1ok8eR/WlyZqJnbEe8b51QwiXt7G3 WH2HlaYpATlPoQivTkYTRCKArq/jdhkoUmfBk= MIME-Version: 1.0 Received: by 10.142.7.29 with SMTP id 29mr2785851wfg.192.1267468637788; Mon, 01 Mar 2010 10:37:17 -0800 (PST) In-Reply-To: <3bbf2fe11002201419v52b249ccg8d82c8ae747cf318@mail.gmail.com> References: <179b97fb1001270941m2d8e9c8au20abc798c16b9c11@mail.gmail.com> <3bbf2fe11002201419v52b249ccg8d82c8ae747cf318@mail.gmail.com> Date: Mon, 1 Mar 2010 12:37:17 -0600 Message-ID: <179b97fb1003011037w4828ca1fj8de0cb5da35dfc3f@mail.gmail.com> From: Brandon Gooch To: Attilio Rao Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, freebsd-emulation@freebsd.org, FreeBSD Current , stable-list freebsd Subject: Re: ZFS and sh(1) panic: spin lock [lock addr] (smp rendezvous) held by [sh(1) proc tid] too long X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 18:37:22 -0000 On Sat, Feb 20, 2010 at 4:19 PM, Attilio Rao wrote: > 2010/1/27 Brandon Gooch : >> The machine, a Dell Optiplex 755, has been locking up recently. The >> situation usually occurs while using VirtualBox (running a 64-bit >> Windows 7 instance) and doing anything else in another xterm (such as >> rebuilding a port). =A0I've been unable to reliably reproduce it (I'm in >> an X session and the machine will not panic "properly"). >> >> However, while rebuilding Xorg today at ttyv0 and runnning >> VBoxHeadless on ttyv1, I managed to trigger what I believe is the >> lockup. >> >> I've attached a textdump in hopes that someone may be able to take a >> look and provide clues or instruction on debugging this. > > I think that jhb@ saw a similar problem while working on nVidia driver > or the like. > Not sure if he made any progress to debug this. > The situation has improved slightly, although attempting to run two VirtualBox guests at the same time inevitably leads to a lock-up. I've just taken to running one at a time. Not ideal, but until more debugging can be done, it's the only option I have. I ran into this using nvidia and radeon both. I can't really find a pattern, but I do see it when Windows is trying to draw a new window, or dim the screen when UAC kicks in... BTW, anyone know how to get a good dump when running Xorg? I'm not sure I've ever been able to, even when I panic on something non related to X or video drivers. -Brandon From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 20:02:14 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E125106566B for ; Mon, 1 Mar 2010 20:02:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f183.google.com (mail-iw0-f183.google.com [209.85.223.183]) by mx1.freebsd.org (Postfix) with ESMTP id DD0358FC17 for ; Mon, 1 Mar 2010 20:02:13 +0000 (UTC) Received: by iwn13 with SMTP id 13so3254239iwn.14 for ; Mon, 01 Mar 2010 12:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=QcfcwY5e2K3ihRRqZioUAfjZTxHjPt9yD/FXWkfg/B0=; b=Gc51axVTreVnllwGxvHV+1/URZxl+Odra3u/uTRyzZsrGxSkT4KNtaVsiFBYd4CdQp cfEmNPNiJw9Gn6M8W/ZjAZXCUQrIIS0PRb0Vlgp/Zq18gZ6nj4926imRyIupEJjt1GvD x8iMtfQsMwt6N0GY2Rg3OzTx1nSO3s+A80zNo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=RvLhysrU925/T/bybwsp2BR9UYvRws0su1BdQ69MyGRmzWGfTruaadXla6+PRDlPOV t3VaNlUvcdFyyCkEF9CsMyRy1r7pdsXrF85di5cKkjla+PCj4M9VOzp6MZX0UiZJu/N0 d4ZNuqAYsvnlA4vY2B5wc1ClfBdYn8U5G3fls= MIME-Version: 1.0 Received: by 10.231.170.136 with SMTP id d8mr148062ibz.17.1267473435166; Mon, 01 Mar 2010 11:57:15 -0800 (PST) Date: Mon, 1 Mar 2010 11:57:15 -0800 Message-ID: From: Freddie Cash To: fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: HAST, ucarp, and ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 20:02:14 -0000 Perhaps it's just a misunderstanding on my part of the layering involved, but I'm having an issue with the sample ucarp_up.sh script on the HAST wiki page. Here's the test setup that I have: hast1: glabel 4x 2 GB virtual disks (label/disk01 --> label/disk04) hast.conf create 4 resources (disk01 --> disk04, using the glabelled disks) zpool create hapool raidz1 hast/disk01 .. hast/disk04 hast2: glabel 4x 2 GB virtual disks (label/disk01 --> label/disk04) hast.conf create 4 resources (disk01 --> disk04) So far so good. On hast1, I have a working ZFS pool, I can create data, filessytems, etc, and watch network traffic as it syncs to hast2. I can manually down hast1 and switch hast2 to "primary" and import the hapool. I can create data, filesystems, etc. And I can manually bring hast1 online and set it to secondary, and watch it sync back. Where I'm stuck is how to modify the ucarp_up.sh script to work with multiple hast resources. Do I just edit it to handle each of the 4 hast resources in turn, or am I missing something simple, like that there should only be a single hast resource? I'm guess it's a simple "edit the script to suit my setup" issue, but wanted to double-check. The production server I want to use this with has 24 harddrives in it, configured into multiple raidz2 vdevs, as part of a single ZFS pool. Which will mean 24 separate hast resources, if I understand things correctly. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 20:43:30 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DB8E106566B for ; Mon, 1 Mar 2010 20:43:30 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 3DD248FC17 for ; Mon, 1 Mar 2010 20:43:28 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 527E945EEB; Mon, 1 Mar 2010 21:43:26 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 39EBE45EC0; Mon, 1 Mar 2010 21:43:20 +0100 (CET) Date: Mon, 1 Mar 2010 21:43:16 +0100 From: Pawel Jakub Dawidek To: Freddie Cash Message-ID: <20100301204315.GA1946@garage.freebsd.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2oS5YaxWCcQjTEyO" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: fs@freebsd.org Subject: Re: HAST, ucarp, and ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 20:43:30 -0000 --2oS5YaxWCcQjTEyO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 01, 2010 at 11:57:15AM -0800, Freddie Cash wrote: > Perhaps it's just a misunderstanding on my part of the layering involved, > but I'm having an issue with the sample ucarp_up.sh script on the HAST wi= ki > page. >=20 > Here's the test setup that I have: > hast1: > glabel 4x 2 GB virtual disks (label/disk01 --> label/disk04) > hast.conf create 4 resources (disk01 --> disk04, using the glabelled > disks) > zpool create hapool raidz1 hast/disk01 .. hast/disk04 >=20 > hast2: > glabel 4x 2 GB virtual disks (label/disk01 --> label/disk04) > hast.conf create 4 resources (disk01 --> disk04) >=20 > So far so good. On hast1, I have a working ZFS pool, I can create data, > filessytems, etc, and watch network traffic as it syncs to hast2. >=20 > I can manually down hast1 and switch hast2 to "primary" and import the > hapool. I can create data, filesystems, etc. And I can manually bring > hast1 online and set it to secondary, and watch it sync back. >=20 > Where I'm stuck is how to modify the ucarp_up.sh script to work with > multiple hast resources. Do I just edit it to handle each of the 4 hast > resources in turn, or am I missing something simple, like that there shou= ld > only be a single hast resource? I'm guess it's a simple "edit the script= to > suit my setup" issue, but wanted to double-check. The scripts in share/examples/hast/ are well... just examples and unfortunately support only one resource. You have to build/modify the scripts on your own (and maybe share the results?). Note that most of hastctl(8) subcommands accept 'all' as resource name, which means that operation should be performed on all configured resources. > The production server I want to use this with has 24 harddrives in it, > configured into multiple raidz2 vdevs, as part of a single ZFS pool. Whi= ch > will mean 24 separate hast resources, if I understand things correctly. You do understand it correctly. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --2oS5YaxWCcQjTEyO Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuMJuMACgkQForvXbEpPzRXdQCfcbnozYRuK3rXfW/onzPB2pMw jsMAoJQnGdqMHns3bOqxbn7Rte/pDFsm =jJ9m -----END PGP SIGNATURE----- --2oS5YaxWCcQjTEyO-- From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 22:08:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCB6B106566B; Mon, 1 Mar 2010 22:08:57 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 59BB08FC0A; Mon, 1 Mar 2010 22:08:57 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAFjJi0uDaFvK/2dsb2JhbACbB3O9RYR7BIMX X-IronPort-AV: E=Sophos;i="4.49,562,1262581200"; d="scan'208";a="67378000" Received: from fraser.cs.uoguelph.ca ([131.104.91.202]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 01 Mar 2010 17:08:33 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 53400109C327; Mon, 1 Mar 2010 17:08:56 -0500 (EST) X-Virus-Scanned: amavisd-new at fraser.cs.uoguelph.ca Received: from fraser.cs.uoguelph.ca ([127.0.0.1]) by localhost (fraser.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kW9fRQgs545g; Mon, 1 Mar 2010 17:08:55 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id A4B31109C34B; Mon, 1 Mar 2010 17:08:55 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o21ML2Z27999; Mon, 1 Mar 2010 17:21:02 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Mon, 1 Mar 2010 17:21:02 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Jeremy Chadwick In-Reply-To: <20100227220310.GA65110@icarus.home.lan> Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 22:08:58 -0000 On Sat, 27 Feb 2010, Jeremy Chadwick wrote: >> I concur. >> Everything in my network is now on TCP, and there is no mbuf leakage. >> I just don't get over the 5500 mark, no matter what I throw at it. >> >> I do feel that TCP is not as well performing on a local net with Linux, >> hence the choice for UDP. But TCP is workable as next best. > > NFS; Rick Macklem would be a better choice, but as reported, he's MIA. > Not exactly MIA, but only able to read email from time to time at this point. I don't know when I'll be able to do more than that. So, it does sound like it is UDP specific. Robert mentioned one scenario, which was an infrequently executed code path that is being tickled and it has a missing m_freem(). One thing someone could try is switching to the experimental nfs server ("-e" on both mountd and nfsd) and see if the leak goes away. If it does go away, it is almost certainly the above in the regular nfs server code. If it doesn't go away, the problem is more likely in the krpc or the generic udp code. (When I looked at svc_dg.c, I could only spot one possible leak and you've already determined that patch doesn't help. The other big difference when using udp on the FreeBSD8 krpc is the reply cache code. I seem to recall it's an lru cache with a fixed upper bound, but it might be broken and leaking. If you change the server to set sp_rcache = NULL in the initialization function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay cache. You wouldn't want to run this way in production, but it would determine if the leak is in it. Change the 3 lines in nfsrv_init() to: nfsrv_pool->sp_rcache = NULL; nfsrv_pool->sp_assign = NULL; nfsrv_pool->sp_done = NULL; and I think the krpc replay cache will be disabled. Good luck with it and please report back if you get to try the above. I'll get back to committing etc one of these days, rick From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 22:15:18 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8707D106564A; Mon, 1 Mar 2010 22:15:18 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f183.google.com (mail-iw0-f183.google.com [209.85.223.183]) by mx1.freebsd.org (Postfix) with ESMTP id 40C308FC13; Mon, 1 Mar 2010 22:15:17 +0000 (UTC) Received: by iwn13 with SMTP id 13so3401212iwn.14 for ; Mon, 01 Mar 2010 14:15:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=C3Ov3SQJac60dgCcm8l7iK18Ho6FPJdXXwmRJ7dcql8=; b=PEdObT4NFOa+QrRA0ahJLbezHvEllp3012/HZtxKPX+VQBgX+Gd9f2gDugdn52l/PN jq/XpoEq6t5bd5mxJe4v4eiURb9f/li/2yeQBviIr1dJNSw3CmNLyTjhV+0NFsiL9KiD M/ESaWJfEFRXQkeXLQMNIZ6XsKxLuYzSarStc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Q5OIzXtjmoj/VifRnpqsC1fP8S1jKK6E3C9zuIiA3ynepYFgkTaVjCbjh364sXU8bz 4+HPUuTvmi0qiP2c77DOc8qFrVY+PKBW/XjpmTknObVJ6NRS9USg2UeKfKI3X/FZA1BM 33QxSaCHFksWd94IynAqrSaFs9+4e90cmk5GU= MIME-Version: 1.0 Received: by 10.231.145.5 with SMTP id b5mr88379ibv.70.1267481712889; Mon, 01 Mar 2010 14:15:12 -0800 (PST) In-Reply-To: <20100301204315.GA1946@garage.freebsd.pl> References: <20100301204315.GA1946@garage.freebsd.pl> Date: Mon, 1 Mar 2010 14:15:12 -0800 Message-ID: From: Freddie Cash To: Pawel Jakub Dawidek Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: fs@freebsd.org Subject: Re: HAST, ucarp, and ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 22:15:18 -0000 On Mon, Mar 1, 2010 at 12:43 PM, Pawel Jakub Dawidek wrote: > On Mon, Mar 01, 2010 at 11:57:15AM -0800, Freddie Cash wrote: > > The scripts in share/examples/hast/ are well... just examples and > unfortunately support only one resource. You have to build/modify the > scripts on your own (and maybe share the results?). > Note that most of hastctl(8) subcommands accept 'all' as resource name, > which means that operation should be performed on all configured > resources. > > I think I've got things sorted out. Heavily modified the scripts to work with the setup I have (just ZFS, multiple resources, etc). Everything is working ... except ucarp itself. Just hangs the box when enabling ucarp, even without any of the extra HAST-related stuff. :( I think it's a VirtualBox issue, though. I'll keep playing. At least the HAST side of things is working. :) > > The production server I want to use this with has 24 harddrives in it, > > configured into multiple raidz2 vdevs, as part of a single ZFS pool. > Which > > will mean 24 separate hast resources, if I understand things correctly. > > You do understand it correctly. > > Good to know. Thanks. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Mon Mar 1 23:30:28 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4CD0E1065672 for ; Mon, 1 Mar 2010 23:30:28 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.27]) by mx1.freebsd.org (Postfix) with ESMTP id D174B8FC08 for ; Mon, 1 Mar 2010 23:30:27 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id d26so331833eyd.3 for ; Mon, 01 Mar 2010 15:30:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:subject :message-id:user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint :mime-version:content-type; bh=qagCdIc6zvECERl3SsyCZGXekEO3p8xWIETBPAoMgDs=; b=qg/rh/JJwfMj3hnvYoyK7+qfndlOgiXXqhcuuBb7I3DmgIDwnVsKf+MBU4GrOn8TPr ZOC/g5MZS0M2Afom7U5KZSHj2PBQIJo+wnixKuPzkAAbnDkCnzHZyoYJuOBz3LexrVmd 1vAE3Sc7R3RDLrCuLPo2a+D6q0bK2Z2eEyYP0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:subject:message-id:user-agent:x-openpgp-key-id :x-openpgp-key-fingerprint:mime-version:content-type; b=OmvBdAoAHmbORp1XYjMP7W9pooqpvJq50/kT7BUYEEEwgt9UhkVm75W7jKUfTOI5xp 4OEzjz3xWIWqJaYx+g6+wnDdcYGG9Oa4sD3fj0+jHOlzR3RUeh4MW73qBAlKP2QMC0e3 vCce9Fzyi4yvVS4+49PBvRC/u5P5aH830gueE= Received: by 10.213.109.212 with SMTP id k20mr3765687ebp.32.1267486221309; Mon, 01 Mar 2010 15:30:21 -0800 (PST) Received: from centel.dataix.local (ppp-23.120.dialinfree.com [209.172.23.120]) by mx.google.com with ESMTPS id 15sm2671090ewy.8.2010.03.01.15.30.17 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 01 Mar 2010 15:30:20 -0800 (PST) Sender: "J. Hellenthal" Date: Mon, 1 Mar 2010 18:29:44 -0500 From: jhell To: FreeBSD Filesystems Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: [ZFS] sharenfs stable/7 # !!! DO NOT EDIT THIS FILE MANUALLY !!! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 23:30:28 -0000 Hi FS, After exporting some filesystems from zfs via sharenfs I decided to turn the shares back off after I was done with them on another machine. To my surprise the datasets that I had shared previously were still shared. FreeBSD 7.3-STABLE #0 r204486 Sun Feb 28 19:42:46 EST 2010 Process: # This command adds the share through nfs in /etc/zfs/exports zfs set sharenfs='-maproot=0 disbatch' exports/svn bla bla bla after using them for a little while: # This command should turn off the nfs share and remove the line from # /etc/zfs/exports and signal mountd or whatever. zfs set sharenfs=off exports/svn Low and behold this line is never removed the share and continues even through a reboot all while sharenfs=off on the dataset. No no other datasets are shared at this time. The line in /etc/zfs/exports changes just fine when changing the options via "zfs set sharenfs". I also removed /etc/zfs/exports and repeated the process. It still continued. Should a PR be filed ?, Can someone take a look at this ?. Relevant man page section below sig. -- jhell sharenfs=on | off | opts Controls whether the file system is shared via NFS, and what options are used. A file system with a"sharenfs" property of "off" is managed through traditional tools such as share(1M), unshare(1M), and dfstab(4). Otherwise, the file system is automati- cally shared and unshared with the "zfs share" and "zfs unshare" commands. If the property is set to "on", the share(1M) command is invoked with no options. Otherwise, the share(1M) command is invoked with options equivalent to the contents of this property. When the "sharenfs" property is changed for a dataset, the dataset and any children inheriting the property are re-shared with the new options, only if the property was previously "off", or if they were shared before the property was changed. If the new property is "off", the file systems are unshared. From owner-freebsd-fs@FreeBSD.ORG Tue Mar 2 00:15:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74ECB106566B for ; Tue, 2 Mar 2010 00:15:33 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id D57968FC12 for ; Tue, 2 Mar 2010 00:15:32 +0000 (UTC) Received: from mail.cicely.de ([10.1.1.37]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id o220F2u0064303 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 2 Mar 2010 01:15:02 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by mail.cicely.de (8.14.3/8.14.3) with ESMTP id o220ExBp020900 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 2 Mar 2010 01:14:59 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id o220ExUF072719; Tue, 2 Mar 2010 01:14:59 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id o220Exrq072718; Tue, 2 Mar 2010 01:14:59 +0100 (CET) (envelope-from ticso) Date: Tue, 2 Mar 2010 01:14:59 +0100 From: Bernd Walter To: jhell Message-ID: <20100302001459.GT58319@cicely7.cicely.de> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on spamd.cicely.de Cc: FreeBSD Filesystems Subject: Re: [ZFS] sharenfs stable/7 # !!! DO NOT EDIT THIS FILE MANUALLY !!! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 00:15:33 -0000 On Mon, Mar 01, 2010 at 06:29:44PM -0500, jhell wrote: > > Hi FS, > > After exporting some filesystems from zfs via sharenfs I decided to turn > the shares back off after I was done with them on another machine. To my > surprise the datasets that I had shared previously were still shared. > > FreeBSD 7.3-STABLE #0 r204486 Sun Feb 28 19:42:46 EST 2010 > > Process: > > # This command adds the share through nfs in /etc/zfs/exports > zfs set sharenfs='-maproot=0 disbatch' exports/svn > > bla bla bla after using them for a little while: > > # This command should turn off the nfs share and remove the line from > # /etc/zfs/exports and signal mountd or whatever. > zfs set sharenfs=off exports/svn > > Low and behold this line is never removed the share and continues even > through a reboot all while sharenfs=off on the dataset. No no other > datasets are shared at this time. > > The line in /etc/zfs/exports changes just fine when changing the options > via "zfs set sharenfs". > > I also removed /etc/zfs/exports and repeated the process. It still > continued. > > Should a PR be filed ?, Can someone take a look at this ?. This reminds me that my mountd complains about a zfs-export line. The given filesystem doesn't exist anymore. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. From owner-freebsd-fs@FreeBSD.ORG Tue Mar 2 00:41:07 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16095106564A; Tue, 2 Mar 2010 00:41:07 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id DFF838FC14; Tue, 2 Mar 2010 00:41:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o220f6Zb022994; Tue, 2 Mar 2010 00:41:06 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o220f6Z3022990; Tue, 2 Mar 2010 00:41:06 GMT (envelope-from linimon) Date: Tue, 2 Mar 2010 00:41:06 GMT Message-Id: <201003020041.o220f6Z3022990@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144402: [zfs] [panic] panic at zfs_znode_dmu_init: existing znode for dbuf X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 00:41:07 -0000 Old Synopsis: [zfs] panic at zfs_znode_dmu_init: existing znode for dbuf New Synopsis: [zfs] [panic] panic at zfs_znode_dmu_init: existing znode for dbuf Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Tue Mar 2 00:40:49 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=144402 From owner-freebsd-fs@FreeBSD.ORG Tue Mar 2 09:38:40 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 385B61065674 for ; Tue, 2 Mar 2010 09:38:40 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id D47998FC1C for ; Tue, 2 Mar 2010 09:38:38 +0000 (UTC) Received: from outgoing.leidinger.net (pD9E2D987.dip.t-dialin.net [217.226.217.135]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 65FB18451BE; Tue, 2 Mar 2010 10:38:30 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 1C80011BDFC; Tue, 2 Mar 2010 10:38:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1267522707; bh=c0ijb3CMC92ukdvzv+w31QZhw+otQWOwsqv3cyjPyiY=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=dUl4kQ7BJmkWaOr05w+Ex208D+Au/k1a48Ukn743Uu6Hrvr811rqTt6r7SHtX/wSt 1+2NBF55NCSLhSRc+LVwXjEdVXDo7rbhrQDdMnh36g4z1XEczwc6PYDCFuIroZR/TT AMWQNXD16ZVIJO401O79A11v1Nbb8jk/gukxMxhK2CD5RGJfkfc/fdxUg2tsUjQjDc E2VyJNFr6+daOVOAAFXNrU0BwoMGrGbTCz4otwSYAnMlyUra6JSoLETFj7OVC51f4u uuBISOKus+pVluufJHTNzRjvBgN+U+ilCmfxVrOPhj+fOxQ6Cq+1QRwXxRb+6O9RpI KUUWfJgDMhJmg== Received: (from www@localhost) by webmail.leidinger.net (8.14.3/8.13.8/Submit) id o229cQNg060891; Tue, 2 Mar 2010 10:38:26 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Tue, 02 Mar 2010 10:38:26 +0100 Message-ID: <20100302103826.14273mzlwp38550k@webmail.leidinger.net> Date: Tue, 02 Mar 2010 10:38:26 +0100 From: Alexander Leidinger To: "James R. Van Artsdalen" References: <4B8B5780.2050601@jrv.org> In-Reply-To: <4B8B5780.2050601@jrv.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 65FB18451BE.04AB3 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.44, required 6, autolearn=disabled, ALL_TRUSTED -1.44, DKIM_SIGNED 0.00, DKIM_VERIFIED -0.00) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1268127513.70265@FMu3rzQEndISrabJg4DwJw X-EBL-Spam-Status: No Cc: freebsd-fs Subject: Re: [zfs] attach by name/uuid still attaches wrong device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 09:38:40 -0000 Quoting "James R. Van Artsdalen" (from Sun, 28 Feb 2010 23:58:24 -0600): > I don't think it's possible to do this right in vdev_geom.c: there's no > way to guess what is intended without a hint from higher ZFS layers as > to which drives should be found and which are new. There is a way: do not attach blindly. Bye, Alexander. -- Chocolate Chip. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Tue Mar 2 10:25:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA1A8106566B; Tue, 2 Mar 2010 10:25:24 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 8B7B78FC08; Tue, 2 Mar 2010 10:25:24 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1NmPHy-0009jy-Dj; Tue, 02 Mar 2010 12:25:18 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Rick Macklem In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Rick Macklem message dated "Mon, 01 Mar 2010 17:21:02 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 02 Mar 2010 12:25:17 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 10:25:25 -0000 > > > On Sat, 27 Feb 2010, Jeremy Chadwick wrote: > > >> I concur. > >> Everything in my network is now on TCP, and there is no mbuf leakage. > >> I just don't get over the 5500 mark, no matter what I throw at it. > >> > >> I do feel that TCP is not as well performing on a local net with Linux, > >> hence the choice for UDP. But TCP is workable as next best. > > > > NFS; Rick Macklem would be a better choice, but as reported, he's MIA. > > > > Not exactly MIA, but only able to read email from time to time at this > point. I don't know when I'll be able to do more than that. > > So, it does sound like it is UDP specific. Robert mentioned one scenario, > which was an infrequently executed code path that is being tickled and it > has a missing m_freem(). > > One thing someone could try is switching to the experimental nfs server > ("-e" on both mountd and nfsd) and see if the leak goes away. If it does > go away, it is almost certainly the above in the regular nfs server code. > runing with the experimental nfs server all is ok! (at least I can't see any mbuf leakage :-) so now that we can assume that the problem is in NFS/UDP writes via classic nfsserver, where to look? > If it doesn't go away, the problem is more likely in the krpc or the > generic udp code. (When I looked at svc_dg.c, I could only spot one > possible leak and you've already determined that patch doesn't help. > The other big difference when using udp on the FreeBSD8 krpc is the > reply cache code. I seem to recall it's an lru cache with a fixed upper > bound, but it might be broken and leaking. > > If you change the server to set sp_rcache = NULL in the initialization > function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay > cache. You wouldn't want to run this way in production, but it would > determine if the leak is in it. > > Change the 3 lines in nfsrv_init() to: > nfsrv_pool->sp_rcache = NULL; > nfsrv_pool->sp_assign = NULL; > nfsrv_pool->sp_done = NULL; > > and I think the krpc replay cache will be disabled. > > Good luck with it and please report back if you get to try the above. > > I'll get back to committing etc one of these days, rick just keep sending insights/pointers and enjoy life danny From owner-freebsd-fs@FreeBSD.ORG Tue Mar 2 13:09:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89C72106566C for ; Tue, 2 Mar 2010 13:09:37 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-qy0-f194.google.com (mail-qy0-f194.google.com [209.85.221.194]) by mx1.freebsd.org (Postfix) with ESMTP id 3B5468FC15 for ; Tue, 2 Mar 2010 13:09:36 +0000 (UTC) Received: by qyk32 with SMTP id 32so91497qyk.28 for ; Tue, 02 Mar 2010 05:09:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=tg3T74lAaHEZysKVkdG/lxDVHtpo41ooAwha74WG8gw=; b=rEE9KT4FBLzYcOUo+HsuHXaf3TK2U1lxEfFIXhfMhslTSyH1XWqt/yH5+YkgxgymDu 3FIaFA+agMeE6I0leBDCxQtNQZ1vYBwA8YkHMnZ8YTINBdegCSguCxUfgx297nuE0LIK atmwKj6aWyR9eW/c38p9ZZbwuH7XJwqTDPikY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=tjLOIpOd0rKUOdzF4imGoum/HWnwL9q9RIN0VrnMgCCC69C2NZxAvTLHm1mMfFt24j Ght/5S8HgJxVn+X06bHMJNnsw0FT9ktLliCFJzfr+Onm2ilDTx/kzDH/vi0xRPQ6w95e DNVOxqURYkmdxagX6jOwuGaUBUOjvxeFx+fNA= Received: by 10.224.27.34 with SMTP id g34mr3264111qac.19.1267535370857; Tue, 02 Mar 2010 05:09:30 -0800 (PST) Received: from centel.dataix.local (ppp-22.138.dialinfree.com [209.172.22.138]) by mx.google.com with ESMTPS id 6sm8615273qwd.24.2010.03.02.05.09.26 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 02 Mar 2010 05:09:29 -0800 (PST) Sender: "J. Hellenthal" Date: Tue, 2 Mar 2010 08:08:31 -0500 From: jhell To: ticso@cicely.de In-Reply-To: <20100302001459.GT58319@cicely7.cicely.de> Message-ID: References: <20100302001459.GT58319@cicely7.cicely.de> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Filesystems Subject: Re: [ZFS] sharenfs stable/7 # !!! DO NOT EDIT THIS FILE MANUALLY !!! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 13:09:37 -0000 On Mon, 1 Mar 2010 19:14, ticso@ wrote: > On Mon, Mar 01, 2010 at 06:29:44PM -0500, jhell wrote: >> >> Hi FS, >> >> After exporting some filesystems from zfs via sharenfs I decided to turn >> the shares back off after I was done with them on another machine. To my >> surprise the datasets that I had shared previously were still shared. >> >> FreeBSD 7.3-STABLE #0 r204486 Sun Feb 28 19:42:46 EST 2010 >> >> Process: >> >> # This command adds the share through nfs in /etc/zfs/exports >> zfs set sharenfs='-maproot=0 disbatch' exports/svn >> >> bla bla bla after using them for a little while: >> >> # This command should turn off the nfs share and remove the line from >> # /etc/zfs/exports and signal mountd or whatever. >> zfs set sharenfs=off exports/svn >> >> Low and behold this line is never removed the share and continues even >> through a reboot all while sharenfs=off on the dataset. No no other >> datasets are shared at this time. >> >> The line in /etc/zfs/exports changes just fine when changing the options >> via "zfs set sharenfs". >> >> I also removed /etc/zfs/exports and repeated the process. It still >> continued. >> >> Should a PR be filed ?, Can someone take a look at this ?. > > This reminds me that my mountd complains about a zfs-export line. > The given filesystem doesn't exist anymore. > > Yeah this one stumped me right away when I went back to double check whether the mount was still available and found out it still existed in the zfs/exports file. Somewhere between the upgrade to v13 from v6 this was broken because I very clearly remember this functionality working. I haven't checked yet whether setting sharenfs on multiple fs's and then manually editing the exports file would do yet but I suppose its not going to be good. Hope this is fixed SOON. -- jhell From owner-freebsd-fs@FreeBSD.ORG Wed Mar 3 00:40:34 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECAF5106564A; Wed, 3 Mar 2010 00:40:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 651BF8FC14; Wed, 3 Mar 2010 00:40:33 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEADA/jUuDaFvK/2dsb2JhbACbDHO+PYR7BIMXix8 X-IronPort-AV: E=Sophos;i="4.49,570,1262581200"; d="scan'208";a="67549251" Received: from fraser.cs.uoguelph.ca ([131.104.91.202]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 02 Mar 2010 19:40:32 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 3D3BF109C2BF; Tue, 2 Mar 2010 19:40:32 -0500 (EST) X-Virus-Scanned: amavisd-new at fraser.cs.uoguelph.ca Received: from fraser.cs.uoguelph.ca ([127.0.0.1]) by localhost (fraser.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x0jwkDDWoWZl; Tue, 2 Mar 2010 19:40:31 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id AC603109C285; Tue, 2 Mar 2010 19:40:31 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o230qes05191; Tue, 2 Mar 2010 19:52:40 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 2 Mar 2010 19:52:40 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2010 00:40:34 -0000 On Tue, 2 Mar 2010, Daniel Braniss wrote: > runing with the experimental nfs server all is ok! > (at least I can't see any mbuf leakage :-) > > so now that we can assume that the problem is in NFS/UDP writes via > classic nfsserver, where to look? > It might also be the krpc reply cache, since the experimental server isn't using it (nfsv4 requires a rather twisted reply cache and it was easier to just use that one for nfsv2,3 for the experimental server, as well). >> If it doesn't go away, the problem is more likely in the krpc or the >> generic udp code. (When I looked at svc_dg.c, I could only spot one >> possible leak and you've already determined that patch doesn't help. >> The other big difference when using udp on the FreeBSD8 krpc is the >> reply cache code. I seem to recall it's an lru cache with a fixed upper >> bound, but it might be broken and leaking. >> >> If you change the server to set sp_rcache = NULL in the initialization >> function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay >> cache. You wouldn't want to run this way in production, but it would >> determine if the leak is in it. >> >> Change the 3 lines in nfsrv_init() to: >> nfsrv_pool->sp_rcache = NULL; >> nfsrv_pool->sp_assign = NULL; >> nfsrv_pool->sp_done = NULL; >> >> and I think the krpc replay cache will be disabled. >> If someone gets a chance to try the above (not in production mode:-), it will determine if the problem is in the reply cache or the nfs server's write code. >> Good luck with it and please report back if you get to try the above. >> Thanks for trying the experimental server. It is getting narrowed down, due to everyone's work on it. rick From owner-freebsd-fs@FreeBSD.ORG Wed Mar 3 08:57:43 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51C8E106566C; Wed, 3 Mar 2010 08:57:43 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id E508A8FC0C; Wed, 3 Mar 2010 08:57:42 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1NmkOe-000PSY-JT; Wed, 03 Mar 2010 10:57:36 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Rick Macklem In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Rick Macklem message dated "Tue, 02 Mar 2010 19:52:40 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 03 Mar 2010 10:57:35 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was mbuf leakage with nfs/zfs?) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2010 08:57:43 -0000 > > > On Tue, 2 Mar 2010, Daniel Braniss wrote: > > > runing with the experimental nfs server all is ok! > > (at least I can't see any mbuf leakage :-) > > > > so now that we can assume that the problem is in NFS/UDP writes via > > classic nfsserver, where to look? > > > > It might also be the krpc reply cache, since the experimental server > isn't using it (nfsv4 requires a rather twisted reply cache and it was > easier to just use that one for nfsv2,3 for the experimental server, > as well). > > >> If it doesn't go away, the problem is more likely in the krpc or the > >> generic udp code. (When I looked at svc_dg.c, I could only spot one > >> possible leak and you've already determined that patch doesn't help. > >> The other big difference when using udp on the FreeBSD8 krpc is the > >> reply cache code. I seem to recall it's an lru cache with a fixed upper > >> bound, but it might be broken and leaking. > >> > >> If you change the server to set sp_rcache = NULL in the initialization > >> function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay > >> cache. You wouldn't want to run this way in production, but it would > >> determine if the leak is in it. > >> > >> Change the 3 lines in nfsrv_init() to: > >> nfsrv_pool->sp_rcache = NULL; > >> nfsrv_pool->sp_assign = NULL; > >> nfsrv_pool->sp_done = NULL; > >> > >> and I think the krpc replay cache will be disabled. > >> > > If someone gets a chance to try the above (not in production mode:-), > it will determine if the problem is in the reply cache or the nfs server's > write code. > >> Good luck with it and please report back if you get to try the above. > >> > > Thanks for trying the experimental server. It is getting narrowed down, > due to everyone's work on it. > disabling the krpc reply cache does it, no visible damage. Somehow this reminds me of my old 1970 beetle, parts would fall off but it would continue working :-) where to go from here? danny From owner-freebsd-fs@FreeBSD.ORG Wed Mar 3 13:21:46 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEA471065673; Wed, 3 Mar 2010 13:21:46 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C55088FC19; Wed, 3 Mar 2010 13:21:46 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o23DLkkd073145; Wed, 3 Mar 2010 13:21:46 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o23DLkTR073141; Wed, 3 Mar 2010 13:21:46 GMT (envelope-from linimon) Date: Wed, 3 Mar 2010 13:21:46 GMT Message-Id: <201003031321.o23DLkTR073141@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144447: [zfs] sharenfs fsunshare() & fsshare_main() non functiional. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2010 13:21:47 -0000 Old Synopsis: [ZFS] sharenfs fsunshare() & fsshare_main() non functiional. New Synopsis: [zfs] sharenfs fsunshare() & fsshare_main() non functiional. Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Wed Mar 3 13:20:45 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=144447 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 3 17:13:39 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C3E8106564A for ; Wed, 3 Mar 2010 17:13:39 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 34F718FC14 for ; Wed, 3 Mar 2010 17:13:38 +0000 (UTC) Received: by pwj1 with SMTP id 1so1119522pwj.13 for ; Wed, 03 Mar 2010 09:13:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=Kfg6VanoE9EsenYy9PFBvIEFieVWAh0UGRPgd1ONXb8=; b=DymTrXHQ7SkC8w/7DX8F6+h1lqG8L2rJqgeWLMQ5Aj2sgTL3TAectqcaUEGp/UGpO7 1pP43Y82BSC8Wdoa9iuqfMlh09lcz/z1ay9seS8qByjkX1kGhtUlRy4IRZwG0OPXn+3v +oEfmHf3yR3W+NCBHn/78/gTRr41JEKwP7fm4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=OYaChLipPmlpF5emxMBLqMwUbTIJKZYTktNqO4v1uR1S9hsPPGH4Y4eWZ2tN67RPY/ 6/Ii/3LXbiQzv5kUIZtTOlY/F3p5XAJULSTks2V/pdX+Sallk1M5FylCmnXZ2V5XixzZ xAJnz1QhXKWtTy74aJjom6FXJQw71E+d4NxzU= MIME-Version: 1.0 Received: by 10.140.57.15 with SMTP id f15mr1036325rva.262.1267636412882; Wed, 03 Mar 2010 09:13:32 -0800 (PST) Date: Wed, 3 Mar 2010 09:13:32 -0800 Message-ID: From: Freddie Cash To: fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: PoC: ZFS fail-over with HAST + carp(4) + devd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Mar 2010 17:13:39 -0000 [Not sure if this should go to just fs@ or possibly current@ as well. I'll start with just fs@.] Thought I'd pass this along. It's a proof-of-concept setup I've been using to test HAST fail-over of a ZFS pool, using devd and carp(4). The original impetus for doing this was that ucarp doesn't work (for me) within a VirtualBox VM. Just hangs the VM. And, I prefer to use FreeBSD base tools whenever possible, so I thought I'd try and get it to work with carp(4). I know this isn't perfect as it (currently) relies on a "magic constant" and doesn't cover all the possible failure modes, but thought I'd pass it along to get your input, comments, criticisms, suggestions, etc. With a bit more work, it could be generalised a bit more to, for example, pull the resources list from /etc/hast.conf, and to work with non-ZFS setups. Perhaps someday it could be useful an an example in the HAST samples/ directory.?. With this setup, I can pull the plug on carp0 on the master node, and the hast devices and ZFS pool fail-over to the slave. And if I pull the plug on carp0 on the slave, everything fails over to the master again. And it works nicely with carp preempt enabled on the master node. Add the following stanzas to /etc/devd.conf: notify 10 { match "system" "IFNET"; match "subsystem" "carp0"; match "type" "LINK_UP"; action "/usr/local/bin/carp-hast-switch master"; }; notify 10 { match "system" "IFNET"; match "subsystem" "carp0"; match "type" "LINK_DOWN"; action "/usr/local/bin/carp-hast-switch slave"; }; Contents of /usr/local/bin/carp-hast-switch: #!/bin/sh # The names of the HAST resources, as listed in hast.conf resources="disk01 disk02 disk03 disk04" # The name of the ZFS pool built on top of HAST resources pool="hapool" case "$1" in master) logger -p local0.debug -t hast "Switching to primary provider for ${resources}." sleep 30 # Wait for any "hastd secondary" processes to stop for disk in ${resources}; do while $( pgrep -lf "hastd: ${disk} \(secondary\)" > /dev/null 2>&1 ); do sleep 1 done # Switch role for each disk hastctl role primary ${disk} if [ $? -ne 0 ]; then logger -p local0.debug -t hast "Unable to change role to primary for resource ${disk}." exit 1 fi done # Wait for the /dev/hast/* devices to appear for disk in ${resources}; do for I in $( jot 60 ); do [ -c "/dev/hast/${disk}" ] && break sleep 0.5 done if [ ! -c "/dev/hast/${disk}" ]; then logger -p local0.debug -t hast "GEOM provider /dev/hast/${disk} did not appear." exit 1 fi done logger -p local0.debug -t hast "Role for HAST resources ${resources} switched to primary." # Import the ZFS pool; has to be done forcibly due to hostid issues zpool import -f -d /dev/hast ${pool} 2>&1 if [ $? -ne 0 ]; then logger -p local0.debug -t hast "ZFS pool import for ${hapool} failed." exit 1 fi logger -p local0.debug -t hast "ZFS pool ${pool} imported." ;; slave) logger -p local0.debug -t hast "Switching to secondary provider for ${resources}." # Export the ZFS pool; has to be done forcibly in case the hast resources have already switched zpool export -f ${pool} 2>&1 if [ $? -ne 0 ]; then logger -p local0.debug -t hast "Unable to export the pool ${pool}." exit 1 fi # Switch roles for the HAST resources for disk in ${resources}; do hastctl role secondary ${disk} 2>&1 if [ $? -ne 0 ]; then logger -p local0.debug -t hast "Unable to switch role to secondary for resource ${disk}." exit 1 fi logger -p local0.debug -t hast "Role switched to secondary for resource ${disk}." done ;; esac -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 00:27:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA3461065674; Thu, 4 Mar 2010 00:27:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 389EE8FC17; Thu, 4 Mar 2010 00:27:35 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAKuMjkuDaFvK/2dsb2JhbACbEHO2TIR8BIMXiyA X-IronPort-AV: E=Sophos;i="4.49,577,1262581200"; d="scan'208";a="67449791" Received: from fraser.cs.uoguelph.ca ([131.104.91.202]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 03 Mar 2010 19:27:35 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id 4F3B0109C271; Wed, 3 Mar 2010 19:27:35 -0500 (EST) X-Virus-Scanned: amavisd-new at fraser.cs.uoguelph.ca Received: from fraser.cs.uoguelph.ca ([127.0.0.1]) by localhost (fraser.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wKpzilAhByqo; Wed, 3 Mar 2010 19:27:34 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by fraser.cs.uoguelph.ca (Postfix) with ESMTP id CF528109C25D; Wed, 3 Mar 2010 19:27:34 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o240dku29980; Wed, 3 Mar 2010 19:39:47 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Wed, 3 Mar 2010 19:39:46 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was mbuf leakage with nfs/zfs?) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 00:27:36 -0000 On Wed, 3 Mar 2010, Daniel Braniss wrote: > disabling the krpc reply cache does it, no visible damage. Somehow > this reminds me of my old 1970 beetle, parts would fall off but it would > continue working :-) > where to go from here? > Ok, so it sounds like the leak is in the krpc reply cache code, if I understand this? (ie. you are running the regular server with the reply cache disabled and the UDP client mounts aren't causing the leak.) Good work on tracking this down! I guess the next step is to look through the code for the leak. I'll do that someday, but if anyone else is inspired to do so, they are more than welcome.:-) Thanks for working through this, rick From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 00:31:45 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CC13106566B for ; Thu, 4 Mar 2010 00:31:45 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id B20CC8FC12 for ; Thu, 4 Mar 2010 00:31:44 +0000 (UTC) Received: by gya1 with SMTP id 1so1079894gya.13 for ; Wed, 03 Mar 2010 16:31:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=/gdTHo0wmp0Pgtb90127h+lChjvd4iFkY7cAtyb1dPg=; b=XFHwWrLIF3sUtv4/rr9QTp2xXTGWJMTQdf94u/28R157NNqgcgAv3gV6EC4UzeqbSA c3GYGlhEBOlEG5EKXirwlapedDRuDRbI0UWrMMCsvc9BNpaL5LrvzwUoopZO844Jo2in qe59U19WKMO1tLu7LMnVPYocS49KwxPtmWFME= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=MUbYCJwP8z4UfYnw2kq9VkYksB/A/E1Dt4ESrvoDlsBfEvqt8c48QgEH26yX61rDqq cw7jb/tHyrc+g1SDb4x2STyg7yJfvrl8f0lJ8XQ8+ByXJF3k9r+QhLsS6fdCnpnNWz6H iEhgONJYhqeWlllBvGEvnl7hDTrZnzP0kLbbA= MIME-Version: 1.0 Received: by 10.91.98.18 with SMTP id a18mr2062075agm.55.1267662700642; Wed, 03 Mar 2010 16:31:40 -0800 (PST) Date: Wed, 3 Mar 2010 16:31:40 -0800 Message-ID: From: Freddie Cash To: fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: HAST: split-brain -- how to force one side to become primary? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 00:31:45 -0000 According to the wiki, when a split-brain situation arises, I should be able to stop hastd on one side, write changes to the /dev/hast/* providers on the primary to increment the localcnt valye, and then bring up the secondary hastd. The locacnt/remotecnt values will be different, and everything will start to re-sync. However, this doesn't seem to work. Or, maybe I'm not doing things right to make it work. Or maybe I've completely misunderstood how it all works. (Nah, that can never happen. roll-eyes) :) /dev/hast/* is used to form a raidz1 vdev as part of pool "hapool". There's a single 1 GB zvol created, that is exported via iSCSI (net/istgt). I can mount the iSCSI disk on a Linux client, partition it, format it using XFS, and write data to it. Using only hast2 node as primary, I've written out 10 MB of new data, and verified that the data is there via "zfs list" on hast2, and multiple mount/unmount cycles on the client. Yet localcnt never increments beyond 1 (remotecnt is 0). Is there a way to forcibly increment localcnt on one node, so that bringing up hastd on the other node will correctly come up as secondary, and start a sync? Or, do I have to manually re-do the HAST setup on one side? Or zero out the base/physical disk underneath HAST? /etc/hast.conf (only listen line is different between nodes): # Global section control /var/run/hastctl listen 172.20.0.1 replication memsync # Resource section resource disk01 { on hast1 { local /dev/label/disk01 remote 172.20.0.2 } on hast2 { local /dev/label/disk01 remote 172.20.0.1 } } resource disk02 { on hast1 { local /dev/label/disk02 remote 172.20.0.2 } on hast2 { local /dev/label/disk02 remote 172.20.0.1 } } resource disk03 { on hast1 { local /dev/label/disk03 remote 172.20.0.2 } on hast2 { local /dev/label/disk03 remote 172.20.0.1 } } resource disk04 { on hast1 { local /dev/label/disk04 remote 172.20.0.2 } on hast2 { local /dev/label/disk04 remote 172.20.0.1 } } hastctl dump on hast1: resource: disk01 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 1224151284752404553 localcnt: 1 remotecnt: 0 prevrole: primary resource: disk02 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 10884849062207686761 localcnt: 1 remotecnt: 0 prevrole: primary resource: disk03 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 14443609578994823508 localcnt: 1 remotecnt: 0 prevrole: primary resource: disk04 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 1365498106518463540 localcnt: 1 remotecnt: 0 prevrole: primary hastctl dump on hast2: resource: disk01 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 1224151284752404553 localcnt: 1 remotecnt: 0 prevrole: primary resource: disk02 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 10884849062207686761 localcnt: 1 remotecnt: 0 prevrole: primary resource: disk03 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 14443609578994823508 localcnt: 1 remotecnt: 0 prevrole: primary resource: disk04 datasize: 2147478528 extentsize: 2097152 keepdirty: 64 localoff: 4608 resuid: 1365498106518463540 localcnt: 1 remotecnt: 0 prevrole: primary -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 04:09:26 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3563E106566B; Thu, 4 Mar 2010 04:09:26 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0C3428FC14; Thu, 4 Mar 2010 04:09:26 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o2449PF0022155; Thu, 4 Mar 2010 04:09:25 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o2449Pmk022151; Thu, 4 Mar 2010 04:09:25 GMT (envelope-from linimon) Date: Thu, 4 Mar 2010 04:09:25 GMT Message-Id: <201003040409.o2449Pmk022151@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144415: [zfs] [panic] kernel panics on boot after zfs crash X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 04:09:26 -0000 Old Synopsis: kernel panics on boot New Synopsis: [zfs] [panic] kernel panics on boot after zfs crash Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Mar 4 04:07:03 UTC 2010 Responsible-Changed-Why: Fix up synopsis and assign. http://www.freebsd.org/cgi/query-pr.cgi?pr=144415 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 04:12:07 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A426106564A; Thu, 4 Mar 2010 04:12:07 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 212458FC19; Thu, 4 Mar 2010 04:12:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o244C7b2030239; Thu, 4 Mar 2010 04:12:07 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o244C6Ye030235; Thu, 4 Mar 2010 04:12:07 GMT (envelope-from linimon) Date: Thu, 4 Mar 2010 04:12:07 GMT Message-Id: <201003040412.o244C6Ye030235@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144416: [panic] Kernel panic on online filesystem optimization X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 04:12:07 -0000 Old Synopsis: Kernel panic on online filesystem optimization New Synopsis: [panic] Kernel panic on online filesystem optimization Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Mar 4 04:09:51 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=144416 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 07:03:44 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 990BE1065674; Thu, 4 Mar 2010 07:03:44 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 385448FC1A; Thu, 4 Mar 2010 07:03:43 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1Nn55u-000Ik6-9C; Thu, 04 Mar 2010 09:03:38 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Rick Macklem In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Rick Macklem message dated "Wed, 03 Mar 2010 19:39:46 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 04 Mar 2010 09:03:37 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was mbuf leakage with nfs/zfs?) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 07:03:44 -0000 > > > On Wed, 3 Mar 2010, Daniel Braniss wrote: > > > disabling the krpc reply cache does it, no visible damage. Somehow > > this reminds me of my old 1970 beetle, parts would fall off but it would > > continue working :-) > > where to go from here? > > > Ok, so it sounds like the leak is in the krpc reply cache code, if I > understand this? (ie. you are running the regular server with the reply > cache disabled and the UDP client mounts aren't causing the leak.) correct. The interesting side effect, is that I can't see any negative issues when disabling the cash. > > Good work on tracking this down! > it was a coordinated efford :-) > I guess the next step is to look through the code for the leak. I'll > do that someday, but if anyone else is inspired to do so, they are > more than welcome.:-) > > Thanks for working through this, rick thank you! I have a vested interest in having this fixed, on the other hand nfsd seems ok, I have been running it now on a semi production server and it's holding up quiet nicely, the cache seems not up to expectations: store-mg-03# nfsstat -se Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 48176764 262687 12582599 19732 4225907 9186574 780793 818837 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 7623 160 27753 59551 59552 118216 0 1992779 Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf 0 979005 19 0 1644267 0 0 0 Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock 0 0 0 0 0 0 0 0 LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH 0 0 0 0 0 0 0 0 Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create 0 0 0 0 0 0 Server: Retfailed Faults Clients 0 0 0 OpenOwner Opens LockOwner Locks Delegs 0 0 0 0 0 Server Cache Stats: Inprog Idem Non-idem Misses CacheSize TCPPeak 307 0 297 80943198 0 0 danny From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 07:49:40 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD7AA106566B for ; Thu, 4 Mar 2010 07:49:40 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id 493448FC1C for ; Thu, 4 Mar 2010 07:49:39 +0000 (UTC) Received: by fxm23 with SMTP id 23so973261fxm.3 for ; Wed, 03 Mar 2010 23:49:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=Oz3tlAzZxzarjQDk09nVVlR/AN6uxOgsNU817oqsuIw=; b=dmkotipfnZR5wOexd/rfHPSleqqV3Bq95eejXFTglHb5DOsxfiN+nVjPw49SotB1tb DnTbDbHwFjS/Y5TXk29rMGFTgiR7+0BQrUtJuvfUqKR/jDnbDw0F0NALdBB7ikUUTLPy qj8BKbWedfBbOS+tzwI2Kh2AbKiV9GV8HVIDg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=nug8jgAkM1SaWcSkXD9NEB734lYvdbg2TT2JQRlz4wiT965FkP8GDhkQPyDpT0jkcp ztzVIHDfO18gtCziskW8SGIY1saAOGb+lHxqc5hvHfkAG7+Je4S89GMAb1wSf7vxIu2V vV1ZeXYglipLwJkTPu7qD0RNl3f44ED0eL4tk= Received: by 10.102.16.15 with SMTP id 15mr858388mup.38.1267687643900; Wed, 03 Mar 2010 23:27:23 -0800 (PST) Received: from [10.32.23.105] ([195.34.111.178]) by mx.google.com with ESMTPS id t10sm1284180muh.59.2010.03.03.23.27.22 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 03 Mar 2010 23:27:22 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: Date: Thu, 4 Mar 2010 09:27:19 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <7C1E347F-CD36-4F54-814B-90BE4EB67C75@gmail.com> References: To: Freddie Cash X-Mailer: Apple Mail (2.1077) Cc: fs@freebsd.org Subject: Re: PoC: ZFS fail-over with HAST + carp(4) + devd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 07:49:40 -0000 On Mar 3, 2010, at 7:13 PM, Freddie Cash wrote: > [Not sure if this should go to just fs@ or possibly current@ as well. = I'll > start with just fs@.] >=20 > Thought I'd pass this along. It's a proof-of-concept setup I've been = using > to test HAST fail-over of a ZFS pool, using devd and carp(4). The = original > impetus for doing this was that ucarp doesn't work (for me) within a > VirtualBox VM. Just hangs the VM. And, I prefer to use FreeBSD base = tools > whenever possible, so I thought I'd try and get it to work with = carp(4). >=20 > I know this isn't perfect as it (currently) relies on a "magic = constant" and > doesn't cover all the possible failure modes, but thought I'd pass it = along > to get your input, comments, criticisms, suggestions, etc. With a bit = more > work, it could be generalised a bit more to, for example, pull the = resources > list from /etc/hast.conf, and to work with non-ZFS setups. Perhaps = someday > it could be useful an an example in the HAST samples/ directory.?. >=20 > With this setup, I can pull the plug on carp0 on the master node, and = the > hast devices and ZFS pool fail-over to the slave. And if I pull the = plug on > carp0 on the slave, everything fails over to the master again. And it = works > nicely with carp preempt enabled on the master node. >=20 >=20 > Add the following stanzas to /etc/devd.conf: > notify 10 { > match "system" "IFNET"; > match "subsystem" "carp0"; > match "type" "LINK_UP"; > action "/usr/local/bin/carp-hast-switch = master"; > }; >=20 > notify 10 { > match "system" "IFNET"; > match "subsystem" "carp0"; > match "type" "LINK_DOWN"; > action "/usr/local/bin/carp-hast-switch = slave"; > }; >=20 >=20 Hi, It's a bit offtopic for fs-, but I'm just curious what FreeBSD version = are you using with this? My experience with 7.* is that LINK_UP/DOWN events are not generated for = carp interfaces, and I have just tested this in my devd.conf and it does nothing. I guess = this is fixed in 8 or -CURRENT? Regards, Niki Denev= From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 08:12:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11340106564A for ; Thu, 4 Mar 2010 08:12:22 +0000 (UTC) (envelope-from spam_zfs@hessmann.de) Received: from mail.vbcl.de (lists2.holidaycheck.de [82.135.108.108]) by mx1.freebsd.org (Postfix) with ESMTP id 9907F8FC1A for ; Thu, 4 Mar 2010 08:12:21 +0000 (UTC) Received: from ppp-88-217-1-172.dynamic.mnet-online.de ([88.217.1.172] helo=[192.168.0.62]) by mail.vbcl.de with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1Nn5p3-0002ec-Py for freebsd-fs@freebsd.org; Thu, 04 Mar 2010 08:50:18 +0100 Message-Id: <15662C97-CCB2-480A-838A-22EFF2922210@hessmann.de> From: =?ISO-8859-1?Q?Christian_He=DFmann?= To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Thu, 4 Mar 2010 08:50:15 +0100 X-Mailer: Apple Mail (2.936) X-Antivirus-Scanned: Clean Subject: ZFS RAID: Disk fails while repacing another disk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 08:12:22 -0000 Hello guys, first, I have to apologize to the people who've already read this on the freebsd Webforum or the Opensolaris ZFS mailinglist, but I just heard about this mailinglist from Bob in the Opensolaris ZFS ML and thought I'd give it another go here before doing anything drastic. I have a ZFS pool comprised of two 3-disk RAIDs which I've recently moved from OS X to FreeBSD (8 stable). One harddisk failed last weekend with lots of shouting, SMART messages and even a kernel panic. I attached a new disk and started the replacement. Unfortunately, about 20% into the replacement, a second disk in the same RAID showed signs of misbehaviour by giving me read errors. The resilvering did finish, though, and it left me with only three broken files according to zpool status: [root@camelot /]# zpool status -v tank pool: tank state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 10h42m with 136 errors on Tue Mar 2 07:55:05 2010 config: NAME STATE READ WRITE CKSUM tank DEGRADED 137 0 0 raidz1 ONLINE 0 0 0 ad17p2 ONLINE 0 0 0 ad18p2 ONLINE 0 0 0 ad20p2 ONLINE 0 0 0 raidz1 DEGRADED 326 0 0 replacing DEGRADED 0 0 0 ad16p2 OFFLINE 2 169K 6 ad4p2 ONLINE 0 0 0 839G resilvered ad14p2 ONLINE 0 0 0 5.33G resilvered ad15p2 ONLINE 418 0 0 5.33G resilvered errors: Permanent errors have been detected in the following files: tank/DVD:<0x9cd> tank/DVD@20100222225100:/Memento.m4v tank/DVD@20100222225100:/Payback.m4v tank/DVD@20100222225100:/TheManWhoWasntThere.m4v I have the feeling the problems on ad15p2 are related to a cable issue, since it doesn't have any SMART errors, is quite a new drive (3 months old) and was IMHO sufficiently "burned in" by repeatedly filling it to the brim and checking the contents (via ZFS). So I'd like to switch off the server, replace the cable and do a scrub afterwards to make sure it doesn't produce additional errors. Unfortunately, although it says the resilvering completed, I can't detach ad16p2 (the first faulted disk) from the system: [root@camelot /]# zpool detach tank ad16p2 cannot detach ad16p2: no valid replicas I know that it still says 'replacing', but I don't have any activity on the drives, so I have to assume that it either stopped and can't restart or it has finished but somehow doesn't know it. To be honest, I don't know how to proceed now. It feels like my system is in a very unstable state right now, with a replacement not yet finished and errors on two drives in one RAID.Z1. I deleted the files affected, but have about 20 snapshots of this filesystem and think these files are in most of them since they're quite old. So, what should I do now? Delete all snapshots? Move all other files from this filesystem to a new filesystem and destroy the old filesystem? Try to export and import the pool? Is it even safe to reboot the machine right now? So far, the answers I got are either: Booting is the last resort, be careful or (especially from the FreeBSD side): Yes, boot it and scrub it, that's the usual way we're doing things. I think what I'm looking for here is that no one shouts out: STOP, DON'T REBOOT! Yes, I have and I will again copy the data to other disks (at least the important files, unfortunately I don't have 7TB of disks lying around) and I won't blame ZFS for doing what it's supposed to do, but it would be nice if I didn't have to start from scratch. Although then I would be a lot smarter and go for RAID.Z2, believe me... Thank you. Christian From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 08:33:37 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44140106566C for ; Thu, 4 Mar 2010 08:33:37 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.156]) by mx1.freebsd.org (Postfix) with ESMTP id CE0588FC1C for ; Thu, 4 Mar 2010 08:33:35 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 22so534403fge.13 for ; Thu, 04 Mar 2010 00:33:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=EmWhEQ7BlftJL6hM4Xf6WxRrd3Ir5D1mZC7rsJKPGKs=; b=PGToHGp+9JcKYQtrMhO8tnEM4AjfjO17xbXe8j6iqJwuGFBn5ACnuezgaNTdN8IjB+ CZ61YxSRILeUOaZjMEeBf0EpDF/Mp9L2eZvrud7kR66HqjuIOq6D86hZMvDOAYQt9uIF yZoZPKdqHOT5ShoB7yjWycBMjwD8hzoqebays= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=K9yan5YzXuz9ZWcugJu2KeJkTcLSWp0cpbUsXlTgIINzI3j47/fZfrX8ck2PMo6GXI Pzj9z7l420e4UGR9AQ0KDvYxtE+EnXsBd/Cf9n2iBRdotN9QynmikZcmiSs4irOEWB5N 7ZNhnB8qVPgEBP+LUQwsRh8c1lqORDxz+gGZo= MIME-Version: 1.0 Received: by 10.102.169.26 with SMTP id r26mr1505116mue.27.1267690171948; Thu, 04 Mar 2010 00:09:31 -0800 (PST) In-Reply-To: <7C1E347F-CD36-4F54-814B-90BE4EB67C75@gmail.com> References: <7C1E347F-CD36-4F54-814B-90BE4EB67C75@gmail.com> Date: Thu, 4 Mar 2010 11:09:31 +0300 Message-ID: From: pluknet To: Nikolay Denev Content-Type: text/plain; charset=ISO-8859-1 Cc: fs@freebsd.org Subject: Re: PoC: ZFS fail-over with HAST + carp(4) + devd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 08:33:37 -0000 On 4 March 2010 10:27, Nikolay Denev wrote: > > Hi, > > It's a bit offtopic for fs-, but I'm just curious what FreeBSD version are you using with this? > My experience with 7.* is that LINK_UP/DOWN events are not generated for carp interfaces, > and I have just tested this in my devd.conf and it does nothing. I guess this is fixed in 8 or -CURRENT? Let me answer on this. This change is in ip_carp.c v1.60, which was never MFC'ed to 7. -- wbr, pluknet From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 12:59:20 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D19E4106564A for ; Thu, 4 Mar 2010 12:59:20 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7F68C8FC1F for ; Thu, 4 Mar 2010 12:59:20 +0000 (UTC) Received: by vws14 with SMTP id 14so1301446vws.13 for ; Thu, 04 Mar 2010 04:59:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:cc:subject :in-reply-to:message-id:references:user-agent:x-openpgp-key-id :x-openpgp-key-fingerprint:mime-version:content-type; bh=lewo2Iu9bZ9ntuC16+Xa5QEV0rvlx5ufb1a893bIdzk=; b=l0UniG2ziTxWtwvkXOVI4hTHeB6AUStWxDP8sCon4CL8nBVol+MveXkLnDkMKHB2F+ JLlSFFREOdBiBen3TXm/GPNs230FJh7jubu/9MLO7Il+x+ZFlIMaMfIHc5I8gT9sw1gi HFA/nKzH9uou5BHqiwbs28SsdA7hAXXRqv5N0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=RgwLqZLYe9KPhsAGWjIfUiMOE3uIhmgSyxQjkV9Io7iEEIxmzffqN++We4dXQGm4il 9hiL3BPSzXq+8GQVGU3hzD+DWt68V5KLnJZEXLBgsCHwOwHpWkqx5kDN0HD4Du1qfL/C U5fcAkpemvm0Od80i7i9qd+ivjOLCRYATJ7QQ= Received: by 10.220.122.193 with SMTP id m1mr936596vcr.154.1267707554285; Thu, 04 Mar 2010 04:59:14 -0800 (PST) Received: from centel.dataix.local (ppp-21.216.dialinfree.com [209.172.21.216]) by mx.google.com with ESMTPS id 23sm1294179vws.6.2010.03.04.04.59.10 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 04 Mar 2010 04:59:12 -0800 (PST) Sender: "J. Hellenthal" Date: Thu, 4 Mar 2010 07:56:45 -0500 From: jhell cc: FreeBSD Filesystems In-Reply-To: <20100302001459.GT58319@cicely7.cicely.de> Message-ID: References: <20100302001459.GT58319@cicely7.cicely.de> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Re: [ZFS] sharenfs stable/7 # !!! DO NOT EDIT THIS FILE MANUALLY !!! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 12:59:20 -0000 On Mon, 1 Mar 2010 19:14, ticso@ wrote: > On Mon, Mar 01, 2010 at 06:29:44PM -0500, jhell wrote: >> >> Hi FS, >> >> After exporting some filesystems from zfs via sharenfs I decided to turn >> the shares back off after I was done with them on another machine. To my >> surprise the datasets that I had shared previously were still shared. >> >> FreeBSD 7.3-STABLE #0 r204486 Sun Feb 28 19:42:46 EST 2010 >> >> Process: >> >> # This command adds the share through nfs in /etc/zfs/exports >> zfs set sharenfs='-maproot=0 disbatch' exports/svn >> >> bla bla bla after using them for a little while: >> >> # This command should turn off the nfs share and remove the line from >> # /etc/zfs/exports and signal mountd or whatever. >> zfs set sharenfs=off exports/svn >> >> Low and behold this line is never removed the share and continues even >> through a reboot all while sharenfs=off on the dataset. No no other >> datasets are shared at this time. >> >> The line in /etc/zfs/exports changes just fine when changing the options >> via "zfs set sharenfs". >> >> I also removed /etc/zfs/exports and repeated the process. It still >> continued. >> >> Should a PR be filed ?, Can someone take a look at this ?. > > This reminds me that my mountd complains about a zfs-export line. > The given filesystem doesn't exist anymore. > > This is now has a PR, http://www.freebsd.org/cgi/query-pr.cgi?pr=144447 -- jhell From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 14:09:21 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63CD2106566C; Thu, 4 Mar 2010 14:09:21 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3BD058FC1C; Thu, 4 Mar 2010 14:09:21 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o24E9LH3073078; Thu, 4 Mar 2010 14:09:21 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o24E9LZJ073074; Thu, 4 Mar 2010 14:09:21 GMT (envelope-from linimon) Date: Thu, 4 Mar 2010 14:09:21 GMT Message-Id: <201003041409.o24E9LZJ073074@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144458: [nfs] [patch] nfsd fails as a kld X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 14:09:21 -0000 Old Synopsis: [patch] nfsd fails as a kld New Synopsis: [nfs] [patch] nfsd fails as a kld Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Mar 4 14:08:46 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=144458 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 15:42:40 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E2AF106566B for ; Thu, 4 Mar 2010 15:42:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f183.google.com (mail-iw0-f183.google.com [209.85.223.183]) by mx1.freebsd.org (Postfix) with ESMTP id E9BC48FC1D for ; Thu, 4 Mar 2010 15:42:39 +0000 (UTC) Received: by iwn13 with SMTP id 13so2037323iwn.14 for ; Thu, 04 Mar 2010 07:42:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=Sj42RuOpbJHMNVqFozCjxbOMvrN3auauZa04OvLwtrc=; b=viAhVTfqbL1kI+RNoMclXyGUS68zulzYySTyI64HTTJCbicqiBo5fNe3TAvS94YA7d Jy6e8hh7+1Ex7Hwrg8PhMHUJoteRl/mdwgzjX6X10QTFxyO35dYte+pXM0r5aZCzkFsH yT2aY7FF2catkonn/vwAJgvf5X+Jmkgi3VMlo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=WgPjWkW7TVHyDw8kLj7e54kxMWLda48TOED1bptVskoxo5TJ/add3KQhUjGqnp9uDb yuHzYbSsDRYyocg5heJakBb0mDPp2JG1wfgkO8Hjv4wTU2BP+/cIaWeMyJUn33hvzrCn pVueW/o2BkmKRxzizN98F6v3fe7EMOSFwWlUg= MIME-Version: 1.0 Received: by 10.231.160.205 with SMTP id o13mr745616ibx.13.1267717352672; Thu, 04 Mar 2010 07:42:32 -0800 (PST) In-Reply-To: References: <7C1E347F-CD36-4F54-814B-90BE4EB67C75@gmail.com> Date: Thu, 4 Mar 2010 07:42:32 -0800 Message-ID: From: Freddie Cash To: fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: PoC: ZFS fail-over with HAST + carp(4) + devd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 15:42:40 -0000 On Thu, Mar 4, 2010 at 12:09 AM, pluknet wrote: > On 4 March 2010 10:27, Nikolay Denev wrote: > > It's a bit offtopic for fs-, but I'm just curious what FreeBSD version > are you using with this? > > My experience with 7.* is that LINK_UP/DOWN events are not generated for > carp interfaces, > > and I have just tested this in my devd.conf and it does nothing. I guess > this is fixed in 8 or -CURRENT? > > Let me answer on this. This change is in ip_carp.c v1.60, which was > never MFC'ed to 7. > I'm running 9-CURRENT from shortly after the initial HAST commit. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 21:05:19 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A905C106564A for ; Thu, 4 Mar 2010 21:05:19 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id DEDA08FC0C for ; Thu, 4 Mar 2010 21:05:18 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 16DD145E49; Thu, 4 Mar 2010 22:05:16 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id C81B745CDD; Thu, 4 Mar 2010 22:05:10 +0100 (CET) Date: Thu, 4 Mar 2010 22:05:08 +0100 From: Pawel Jakub Dawidek To: Freddie Cash Message-ID: <20100304210508.GA1816@garage.freebsd.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tThc/1wpZn/ma/RB" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: fs@freebsd.org Subject: Re: HAST: split-brain -- how to force one side to become primary? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 21:05:19 -0000 --tThc/1wpZn/ma/RB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 03, 2010 at 04:31:40PM -0800, Freddie Cash wrote: > According to the wiki, when a split-brain situation arises, I should be a= ble > to stop hastd on one side, write changes to the /dev/hast/* providers on = the > primary to increment the localcnt valye, and then bring up the secondary > hastd. The locacnt/remotecnt values will be different, and everything wi= ll > start to re-sync. >=20 > However, this doesn't seem to work. Or, maybe I'm not doing things right= to > make it work. Or maybe I've completely misunderstood how it all works. > (Nah, that can never happen. roll-eyes) :) On split-brain, when you decide that you don't need data on one of the nodes you simply do the following on this node: # hastctl role init # hastctl create # hastctl role secondary --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --tThc/1wpZn/ma/RB Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuQIIQACgkQForvXbEpPzT6UQCffhCXvk4qdlCV46iABfmbNPtV RyYAn2pg990Uh6zhVtNXXKTWCopifoco =nu6r -----END PGP SIGNATURE----- --tThc/1wpZn/ma/RB-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 23:31:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47698106566C; Thu, 4 Mar 2010 23:31:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id AFEAD8FC0A; Thu, 4 Mar 2010 23:31:00 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAPzRj0uDaFvG/2dsb2JhbACbRXO3bYJegh4EgxeLIA X-IronPort-AV: E=Sophos;i="4.49,583,1262581200"; d="scan'208";a="67839054" Received: from amazon.cs.uoguelph.ca ([131.104.91.198]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 04 Mar 2010 18:30:58 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id CBD9A350004; Thu, 4 Mar 2010 18:30:58 -0500 (EST) X-Virus-Scanned: amavisd-new at amazon.cs.uoguelph.ca Received: from amazon.cs.uoguelph.ca ([127.0.0.1]) by localhost (amazon.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NBn9SSsV4-Dp; Thu, 4 Mar 2010 18:30:57 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id 2CF7E210248; Thu, 4 Mar 2010 18:30:57 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o24NhAt07105; Thu, 4 Mar 2010 18:43:10 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Thu, 4 Mar 2010 18:43:10 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/udp (was mbuf leakage with nfs/zfs?) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 23:31:01 -0000 On Thu, 4 Mar 2010, Daniel Braniss wrote: > > correct. The interesting side effect, is that I can't see any negative > issues when disabling the cash. If the client retries a non-idempotent RPC, the server will do it again, which can result in data corruption. This is likely to happen infrequently, but with potentially nasty results. (The paper that describes this was given at a late 1980s Usenix by Chet J. His name is in a comment somewhere, I think. I won't dare to try and spell it.:-) > seems ok, I have been running it now on a semi production server and > it's holding up quiet nicely, the cache seems not up to expectations: > > store-mg-03# nfsstat -se > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 48176764 262687 12582599 19732 4225907 9186574 780793 818837 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 7623 160 27753 59551 59552 118216 0 1992779 > Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf > 0 979005 19 0 1644267 0 0 0 > Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock > 0 0 0 0 0 0 0 0 > LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH > 0 0 0 0 0 0 0 0 > Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create > 0 0 0 0 0 0 > Server: > Retfailed Faults Clients > 0 0 0 > OpenOwner Opens LockOwner Locks Delegs > 0 0 0 0 0 > Server Cache Stats: > Inprog Idem Non-idem Misses CacheSize TCPPeak > 307 0 297 80943198 0 0 > If you are referring to the high miss rate, that is normal and to be expected. It's the 297 Non-idempotent hits that could have caused data corruption without the cache. When there is a hit, the RPC reply comes from the cache, so that the RPC isn't performed again on the server. (Some/many of these are not harmful. For example, a retried Remove simply fails with ENOENT, but others...) Glad to hear that the experimental server is working ok for you, rick From owner-freebsd-fs@FreeBSD.ORG Thu Mar 4 23:40:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 402651065672 for ; Thu, 4 Mar 2010 23:40:03 +0000 (UTC) (envelope-from me@lexasoft.ru) Received: from relay.wahome.ru (relay.wahome.ru [95.211.21.141]) by mx1.freebsd.org (Postfix) with ESMTP id 0A0958FC16 for ; Thu, 4 Mar 2010 23:40:02 +0000 (UTC) Received: from mmx.lexasoft.ru (mmx.lexasoft.ru [92.241.160.6]) by relay.wahome.ru (Postfix) with ESMTP id 3653A6B218A for ; Fri, 5 Mar 2010 02:17:36 +0300 (MSK) Received: from [10.100.0.2] (petrovich-telecom-gw.wahome.ru [77.91.225.38]) by mmx.lexasoft.ru (Postfix) with ESMTPSA id 88CB428491; Fri, 5 Mar 2010 02:20:11 +0300 (MSK) From: Alexey Tarasov Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Fri, 5 Mar 2010 02:20:09 +0300 Message-Id: To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1077) X-Mailer: Apple Mail (2.1077) Cc: Subject: Strange ZFS behaviour with transmission X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2010 23:40:03 -0000 Hello. I have a machine with 4x1.5 Tb SATA disks (WDC WD15EARS). HDD's are in = AHCI mode (ada). I use root-on-ZFS with GPT and RAIDZ. I have the following problem: when I run transmission torrent client on = this server, the FS works very slow, all processes hangs in on of the = zfs, zio, tx states. According to gstat disk load changes very fast from 0 to 100%. After = some activity of all disks there is 2-3 seconds of 100% activity of the = 4-th HDD. During this activity FS is inaccessible. I have already = replaced this disk with new one and there were no effect. Is this a bug or some wrong configuration? /boot/loader.conf: zfs_load=3D"YES" ahci_load=3D"YES" vm.kmem_size=3D2G vm.kmem_size_max=3D2G vfs.root.mountfrom=3D"zfs:zroot" vfs.zfs.prefetch_disable=3D1 zpool status pool: zroot state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 raidz1 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 gpt/disk3 ONLINE 0 0 0 FreeBSD st4 8.0-STABLE FreeBSD 8.0-STABLE #0: Mon Mar 1 17:23:20 UTC = 2010 root@:/usr/obj/usr/src/sys/GENERIC amd64 -- Alexey Tarasov From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 00:02:04 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D06E106566B; Fri, 5 Mar 2010 00:02:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 820FB8FC0C; Fri, 5 Mar 2010 00:02:02 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAAXZj0uDaFvJ/2dsb2JhbACbRXO3SoR8BIMXiyA X-IronPort-AV: E=Sophos;i="4.49,583,1262581200"; d="scan'208";a="67841886" Received: from ganges.cs.uoguelph.ca ([131.104.91.201]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 04 Mar 2010 19:02:01 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by ganges.cs.uoguelph.ca (Postfix) with ESMTP id 7E3E9FB8027; Thu, 4 Mar 2010 19:02:01 -0500 (EST) X-Virus-Scanned: amavisd-new at ganges.cs.uoguelph.ca Received: from ganges.cs.uoguelph.ca ([127.0.0.1]) by localhost (ganges.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lwJGdca4mjRw; Thu, 4 Mar 2010 19:02:00 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by ganges.cs.uoguelph.ca (Postfix) with ESMTP id 97B06FB801F; Thu, 4 Mar 2010 19:02:00 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o250EFs10576; Thu, 4 Mar 2010 19:14:15 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Thu, 4 Mar 2010 19:14:15 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 00:02:04 -0000 On Tue, 2 Mar 2010, Daniel Braniss wrote: > > just keep sending insights/pointers and enjoy life > You could try this patch for sys/rpc/replay.c. Completely untested and just typed into email (so don't give it to "patch", just edit the file). - try adding these 2 lines just before the end of replay_setreply() in sys/rpc/replay.c: - } + } else if (m) + m_freem(m); mtx_unlock(&rc->rc_lock); } It's the only place I can see in replay.c that might leak, rick From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 03:56:10 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 426EB1065670 for ; Fri, 5 Mar 2010 03:56:10 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f173.google.com (mail-iw0-f173.google.com [209.85.223.173]) by mx1.freebsd.org (Postfix) with ESMTP id 094B88FC12 for ; Fri, 5 Mar 2010 03:56:09 +0000 (UTC) Received: by iwn3 with SMTP id 3so2403606iwn.13 for ; Thu, 04 Mar 2010 19:56:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=UJ0UiJuIOXDYJVwQoXwhhUwPoCuMPiTDAQFdBg5qIs0=; b=gEUbksir2LMMIewipgyeo9Kb4Gbtl40q3lz4LlugcWE+cZK29RfNcnAMnjRIvNi0xg 3dpAYqllnOFePgrPfaMxFzGlVUrMeN0k4aH4OlA0TZS/Z3WhK5ZXNnI4z/uAmyVTLYuE N+dqi+CzIL6PnN3NUN8A/Aq6o18Qlt2xCuI2M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=uerTVx5zWu8f7U+7e/TtkqeDDUWpOqHLxFOChaEKVPmXoZswqJYPj+OQlkxIXO54lc z3XGOm5NC5WoY82T4Z2T2kStLKEcx/9veuR9Ci48fFMw7XgnqZ+5l1mV+Ql3WoG5CS+m PAb0D3AA31WP1Oz3KuEUcgfXEDVLmBkqz5/M8= MIME-Version: 1.0 Received: by 10.231.150.74 with SMTP id x10mr1274456ibv.97.1267761367157; Thu, 04 Mar 2010 19:56:07 -0800 (PST) In-Reply-To: <20100304210508.GA1816@garage.freebsd.pl> References: <20100304210508.GA1816@garage.freebsd.pl> Date: Thu, 4 Mar 2010 19:56:06 -0800 Message-ID: From: Freddie Cash To: fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: HAST: split-brain -- how to force one side to become primary? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 03:56:10 -0000 On Thu, Mar 4, 2010 at 1:05 PM, Pawel Jakub Dawidek wrote: > On Wed, Mar 03, 2010 at 04:31:40PM -0800, Freddie Cash wrote: > > According to the wiki, when a split-brain situation arises, I should be > able > > to stop hastd on one side, write changes to the /dev/hast/* providers on > the > > primary to increment the localcnt valye, and then bring up the secondary > > hastd. The locacnt/remotecnt values will be different, and everything > will > > start to re-sync. > > > > However, this doesn't seem to work. Or, maybe I'm not doing things right > to > > make it work. Or maybe I've completely misunderstood how it all works. > > (Nah, that can never happen. roll-eyes) :) > > On split-brain, when you decide that you don't need data on one of the > nodes you simply do the following on this node: > > # hastctl role init > # hastctl create > # hastctl role secondary > > I figured it would be something simple like that. Thanks for the pointers. I'll give this a try in a couple of weeks, when we're back from vacation. :) -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 07:45:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17C4B1065670; Fri, 5 Mar 2010 07:45:48 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id B4A5D8FC0A; Fri, 5 Mar 2010 07:45:47 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1NnSEE-000BdD-7i; Fri, 05 Mar 2010 09:45:46 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Rick Macklem In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Rick Macklem message dated "Thu, 04 Mar 2010 19:14:15 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 05 Mar 2010 09:45:45 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 07:45:48 -0000 > > > On Tue, 2 Mar 2010, Daniel Braniss wrote: > > > > > just keep sending insights/pointers and enjoy life > > > > > You could try this patch for sys/rpc/replay.c. Completely untested and > just typed into email (so don't give it to "patch", just edit the file). > > - try adding these 2 lines just before the end of replay_setreply() in > sys/rpc/replay.c: > > - } > + } else if (m) > + m_freem(m); > mtx_unlock(&rc->rc_lock); > } > > It's the only place I can see in replay.c that might leak, rick > this is what I did: --- a/sys/rpc/replay.c Mon Mar 01 18:29:54 2010 +0200 +++ b/sys/rpc/replay.c Fri Mar 05 09:24:17 2010 +0200 @@ -243,6 +243,9 @@ rce->rce_repbody = m; if (m) rc->rc_size += m_length(m, NULL); + } else if (m) { + printf("free m=%p ...\n", m); + m_freem(m); } mtx_unlock(&rc->rc_lock); } but it didn't help, it's not triggered Thanks for the explanation on the cache, things are begining to make sense. If I understand, the reason for this cache is to prevent re-applying an already performed rpc, which could lead to data corruption btw, the list of CCs is rather big, so if anyone feels he rather be removed, please let me know. cheers, danny From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 09:16:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3542F106566B for ; Fri, 5 Mar 2010 09:16:01 +0000 (UTC) (envelope-from me@lexasoft.ru) Received: from relay.wahome.ru (relay.wahome.ru [95.211.21.141]) by mx1.freebsd.org (Postfix) with ESMTP id F0DBA8FC23 for ; Fri, 5 Mar 2010 09:16:00 +0000 (UTC) Received: from mmx.lexasoft.ru (mmx.lexasoft.ru [92.241.160.6]) by relay.wahome.ru (Postfix) with ESMTP id 5E0526B214C; Fri, 5 Mar 2010 12:13:23 +0300 (MSK) Received: from [10.100.0.2] (petrovich-telecom-gw.wahome.ru [77.91.225.38]) by mmx.lexasoft.ru (Postfix) with ESMTPSA id D983228491; Fri, 5 Mar 2010 12:15:58 +0300 (MSK) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=utf-8 From: Alexey Tarasov In-Reply-To: <20100305090143.GC29609@tf.uni-kiel.de> Date: Fri, 5 Mar 2010 12:15:57 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <0355438D-6F29-4676-BCD4-A7D8C576B749@lexasoft.ru> References: <20100305090143.GC29609@tf.uni-kiel.de> To: Michael Firnau X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@freebsd.org Subject: Re: Strange ZFS behaviour with transmission X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 09:16:01 -0000 Hello. Thank you for your reply. There are patches against rtorrent, not transmission. =3D) On 05.03.2010, at 12:01, Michael Firnau wrote: >=20 >=20 > Hi, >=20 > did you check this: >=20 > http://forums.freebsd.org/showthread.php?t=3D9935&highlight=3Drtorrent >=20 > especially posting #61 / #62? >=20 > There is a patch against the torrent client. >=20 > Hope this helps. >=20 > Greetings, >=20 > Mike -- Alexey Tarasov (\__/)=20 (=3D'.'=3D)=20 E[: | | | | :]=D0=97=20 (")_(") From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 14:15:35 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D86D1065675 for ; Fri, 5 Mar 2010 14:15:35 +0000 (UTC) (envelope-from mgamsjager@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id 209288FC1C for ; Fri, 5 Mar 2010 14:15:34 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 22so219118fge.13 for ; Fri, 05 Mar 2010 06:15:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=/L87mCwA6IB3f19B5s4lKFmKQvMi/iCFONk3S0aJ9j4=; b=d7X6Nfe4YIuC8/G9UF1o0aCVKu1W4Z2GLPRUjNVu88G+j75iznWPl3Yp3pPKNAhbij y1s/iKCBs7FNBNLf8aqbuySPBs66BLPYtd7kvfvSCrjRgxJwekOLfJcMqSLw3J3LmxSh acJc9mWchlRZwh8eSC9q87+w6ae4e0906IQ1Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=NkQeqo/hcXK1mjQCTxpPLu0LYd7OPl/qWpTPoknGQAQ0YHan/zJJkF2DlLjJPebf4x ZUAH7+XrZllyFgRhs/11sFkrpbIA2oaP8UCXA8YGixOz7vKKxjUjUEorDLckYdKYmgJo MZHP7Rhd0GxWAfTfAjl1LSgxWEfj4hSW8GmwA= MIME-Version: 1.0 Received: by 10.87.48.11 with SMTP id a11mr2124417fgk.36.1267798526944; Fri, 05 Mar 2010 06:15:26 -0800 (PST) In-Reply-To: <0355438D-6F29-4676-BCD4-A7D8C576B749@lexasoft.ru> References: <20100305090143.GC29609@tf.uni-kiel.de> <0355438D-6F29-4676-BCD4-A7D8C576B749@lexasoft.ru> From: Matthias Gamsjager Date: Fri, 5 Mar 2010 15:15:06 +0100 Message-ID: <585602e11003050615i63530075h179dec485bddb21d@mail.gmail.com> To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable Subject: Re: Strange ZFS behaviour with transmission X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 14:15:35 -0000 I do see this sometimes with my zfs-raid10 but after a few min the whole set of disk seems to get in line with each other and speed is back to normal. eg. 3 disk are writing with full speed and one disk is lagging behind. still don't know what causes this but afaik it does't have to do anything with transmission. On Fri, Mar 5, 2010 at 10:15 AM, Alexey Tarasov wrote: > Hello. > > Thank you for your reply. > There are patches against rtorrent, not transmission. =3D) > > On 05.03.2010, at 12:01, Michael Firnau wrote: > >> >> >> Hi, >> >> did you check this: >> >> http://forums.freebsd.org/showthread.php?t=3D9935&highlight=3Drtorrent >> >> especially posting #61 / #62? >> >> There is a patch against the torrent client. >> >> Hope this helps. >> >> Greetings, >> >> =9A =9A =9A Mike > > -- > Alexey Tarasov > > (\__/) > (=3D'.'=3D) > E[: | | | | :]=FA > (")_(") > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 14:57:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45AE51065670 for ; Fri, 5 Mar 2010 14:57:33 +0000 (UTC) (envelope-from me@lexasoft.ru) Received: from relay.wahome.ru (relay.wahome.ru [95.211.21.141]) by mx1.freebsd.org (Postfix) with ESMTP id 095468FC20 for ; Fri, 5 Mar 2010 14:57:32 +0000 (UTC) Received: from mmx.lexasoft.ru (mmx.lexasoft.ru [92.241.160.6]) by relay.wahome.ru (Postfix) with ESMTP id DA6A56B2193; Fri, 5 Mar 2010 17:54:54 +0300 (MSK) Received: from [10.100.0.2] (petrovich-telecom-gw.wahome.ru [77.91.225.38]) by mmx.lexasoft.ru (Postfix) with ESMTPSA id C99FE28490; Fri, 5 Mar 2010 17:57:30 +0300 (MSK) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=koi8-r From: Alexey Tarasov In-Reply-To: <585602e11003050615i63530075h179dec485bddb21d@mail.gmail.com> Date: Fri, 5 Mar 2010 17:57:29 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <21C2B0E0-62DE-4B7B-A95D-6939175849FF@lexasoft.ru> References: <20100305090143.GC29609@tf.uni-kiel.de> <0355438D-6F29-4676-BCD4-A7D8C576B749@lexasoft.ru> <585602e11003050615i63530075h179dec485bddb21d@mail.gmail.com> To: Matthias Gamsjager , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1077) Cc: Subject: Re: Strange ZFS behaviour with transmission X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 14:57:33 -0000 On 05.03.2010, at 17:15, Matthias Gamsjager wrote: >=20 > still don't know what causes this but afaik it does't have to do > anything with transmission. >=20 Yes, I know that the problem is not in transmission, but transmission = generates some huge IO load, that shows some bottleneck in ZFS. -- Alexey Tarasov (\__/)=20 (=3D'.'=3D)=20 E[: | | | | :]=FA=20 (")_(") From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 19:14:40 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE8EA106564A; Fri, 5 Mar 2010 19:14:40 +0000 (UTC) (envelope-from brucec@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A50BA8FC13; Fri, 5 Mar 2010 19:14:40 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o25JEeB5097062; Fri, 5 Mar 2010 19:14:40 GMT (envelope-from brucec@freefall.freebsd.org) Received: (from brucec@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o25JEeBs097058; Fri, 5 Mar 2010 19:14:40 GMT (envelope-from brucec) Date: Fri, 5 Mar 2010 19:14:40 GMT Message-Id: <201003051914.o25JEeBs097058@freefall.freebsd.org> To: brucec@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: brucec@FreeBSD.org Cc: Subject: Re: kern/33464: soft update inconsistencies after system crash X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 19:14:40 -0000 Synopsis: soft update inconsistencies after system crash Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: brucec Responsible-Changed-When: Fri Mar 5 19:14:21 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=33464 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 5 21:49:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A507C106566B; Fri, 5 Mar 2010 21:49:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1BB968FC1E; Fri, 5 Mar 2010 21:49:44 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAN8KkUuDaFvG/2dsb2JhbACbSXO3CIR3BIMXiyM X-IronPort-AV: E=Sophos;i="4.49,589,1262581200"; d="scan'208";a="67972920" Received: from amazon.cs.uoguelph.ca ([131.104.91.198]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 05 Mar 2010 16:49:44 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id 28A5E35000A; Fri, 5 Mar 2010 16:49:44 -0500 (EST) X-Virus-Scanned: amavisd-new at amazon.cs.uoguelph.ca Received: from amazon.cs.uoguelph.ca ([127.0.0.1]) by localhost (amazon.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FUOKkYKroi93; Fri, 5 Mar 2010 16:49:42 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id 6D38E350005; Fri, 5 Mar 2010 16:49:42 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o25M1xj13862; Fri, 5 Mar 2010 17:01:59 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 5 Mar 2010 17:01:59 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 21:49:45 -0000 On Fri, 5 Mar 2010, Daniel Braniss wrote: >> >> >> On Tue, 2 Mar 2010, Daniel Braniss wrote: >> >>> >>> just keep sending insights/pointers and enjoy life >>> >> >> >> You could try this patch for sys/rpc/replay.c. Completely untested and >> just typed into email (so don't give it to "patch", just edit the file). >> >> - try adding these 2 lines just before the end of replay_setreply() in >> sys/rpc/replay.c: >> >> - } >> + } else if (m) >> + m_freem(m); >> mtx_unlock(&rc->rc_lock); >> } >> >> It's the only place I can see in replay.c that might leak, rick >> > this is what I did: > --- a/sys/rpc/replay.c Mon Mar 01 18:29:54 2010 +0200 > +++ b/sys/rpc/replay.c Fri Mar 05 09:24:17 2010 +0200 > @@ -243,6 +243,9 @@ > rce->rce_repbody = m; > if (m) > rc->rc_size += m_length(m, NULL); > + } else if (m) { > + printf("free m=%p ...\n", m); > + m_freem(m); > } > mtx_unlock(&rc->rc_lock); > } > > but it didn't help, it's not triggered > Hmm, well that's the only place I could see in replay.c that could leak (and it's a pretty straightforward piece of code). This is getting interesting. Just to confirm where we currently are... - replay cache disabled --> no leak - replay cache enabled (with or without the above patch) --> leak I'll take another look, but I doubt the leak is in replay.c so... maybe a reply from the cache is somehow handled incorrectly and that causes the leak elsewhere? (Just a random hunch at this point.) > Thanks for the explanation on the cache, things are begining to make sense. > If I understand, the reason for this cache is to prevent re-applying an > already performed rpc, which could lead to data corruption > Yep, you've got it. It is basically a bandaid for the poor transport semantics provided by UDP. Having fun with this one. Thanks for the help, rick From owner-freebsd-fs@FreeBSD.ORG Sat Mar 6 07:10:04 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 945F3106566C; Sat, 6 Mar 2010 07:10:04 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 3CB9F8FC08; Sat, 6 Mar 2010 07:10:03 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1Nno98-000OLZ-SI; Sat, 06 Mar 2010 09:09:58 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: Rick Macklem In-reply-to: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> Comments: In-reply-to Rick Macklem message dated "Fri, 05 Mar 2010 17:01:59 -0500." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 06 Mar 2010 09:09:57 +0200 From: Daniel Braniss Message-ID: Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Mar 2010 07:10:04 -0000 [...] > > but it didn't help, it's not triggered > > > > Hmm, well that's the only place I could see in replay.c that could leak > (and it's a pretty straightforward piece of code). This is getting > interesting. Just to confirm where we currently are... > > - replay cache disabled --> no leak > - replay cache enabled (with or without the above patch) --> leak > yes and yes. > I'll take another look, but I doubt the leak is in replay.c so... maybe > a reply from the cache is somehow handled incorrectly and that causes the > leak elsewhere? (Just a random hunch at this point.) > it works ok in 7.2, so it would be interesting to compare changes ... > > Thanks for the explanation on the cache, things are begining to make sense. > > If I understand, the reason for this cache is to prevent re-applying an > > already performed rpc, which could lead to data corruption > > > > Yep, you've got it. It is basically a bandaid for the poor transport > semantics provided by UDP. > > Having fun with this one. Thanks for the help, rick > I'm glad :-) danny From owner-freebsd-fs@FreeBSD.ORG Sat Mar 6 08:26:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B2EA106566C for ; Sat, 6 Mar 2010 08:26:18 +0000 (UTC) (envelope-from journey@onpuls.com) Received: from dd17836.kasserver.com (dd17836.kasserver.com [85.13.138.126]) by mx1.freebsd.org (Postfix) with ESMTP id 206F48FC0C for ; Sat, 6 Mar 2010 08:26:17 +0000 (UTC) Received: from versatel.modem (i59F71A15.versanet.de [89.247.26.21]) by dd17836.kasserver.com (Postfix) with ESMTP id 55C8718415EA7 for ; Sat, 6 Mar 2010 09:04:17 +0100 (CET) From: Morrison Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Sat, 6 Mar 2010 09:04:15 +0100 Message-Id: <9EB2D14A-ABED-4DEC-A88F-09B0AB32905E@onpuls.com> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1077) X-Mailer: Apple Mail (2.1077) Subject: ZFS Boot problem after hard shutdown while resilvering RAIDZ1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Mar 2010 08:26:18 -0000 Hello folks. Here are my specs: FreeBSD Current 8.0 (amd64) 3x 2TB HDD combined to a RAIDZ1 GPT Root is running on ZFS, I used the following Guideline = (http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1) Here is the problem: I had to replace a drive, but chose by accident a spare which hat some = strange malfunction as well. I used the command:=20 zpool replace zroot gpt/disk1 =20 zpool status=20 gave me the information that resilvering would take 2500h (increasing). After a couple of MB were resilvered (very slowly), I decided to shut = the system down safely, which didn't work, so I powered it off by = holding the power switch. I detached the malfunctioned spare drive and tried to start the system = with the two HDDs, but now I am getting ZFS: can only boot from disk, mirror or raidz vdevs ZFS: i/o error - all block copies unavailable ZFS: can't read object set for dataset lld Can't find root filesystem - giving up ZFS: unexpected object set type lld ZFS: unexpected object set type lld FreeBSD/i386 boot Default: zroot:/boot/kernel/kernel boot: ZFS: unexpected object set type lld FreeBSD/i386 boot Default: zroot:/boot/kernel/kernel boot:_ I used my LiveFS USB and tried with Fixit, after loading opensolaris.ko = and zfs.ko: zpool import -f zroot After that, all HDDs started making noises as if they try to = heal/restore them self. But all commands I entered then, gave me errors = like: /libexec/ld-elf.so.1: Shared object "libavl.so.2" not found, = required by "zpool" Even sImple commands like "ls" produce simular messages. I waited one night until the HDDs went silent and rebooted, but without = any luck. Question: Is there any way of getting this system up again? =09 Regards, Morrison= From owner-freebsd-fs@FreeBSD.ORG Sat Mar 6 09:33:16 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2D2B1065670 for ; Sat, 6 Mar 2010 09:33:16 +0000 (UTC) (envelope-from me@lexasoft.ru) Received: from relay.wahome.ru (relay.wahome.ru [95.211.21.141]) by mx1.freebsd.org (Postfix) with ESMTP id AA8E68FC17 for ; Sat, 6 Mar 2010 09:33:16 +0000 (UTC) Received: from mmx.lexasoft.ru (mmx.lexasoft.ru [92.241.160.6]) by relay.wahome.ru (Postfix) with ESMTP id 279306B2158; Sat, 6 Mar 2010 12:30:37 +0300 (MSK) Received: from [10.100.0.2] (petrovich-telecom-gw.wahome.ru [77.91.225.38]) by mmx.lexasoft.ru (Postfix) with ESMTPSA id F27D828490; Sat, 6 Mar 2010 12:33:12 +0300 (MSK) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Apple Message framework v1077) From: Alexey Tarasov In-Reply-To: <9EB2D14A-ABED-4DEC-A88F-09B0AB32905E@onpuls.com> Date: Sat, 6 Mar 2010 12:33:10 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: References: <9EB2D14A-ABED-4DEC-A88F-09B0AB32905E@onpuls.com> To: Morrison , freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1077) Cc: Subject: Re: ZFS Boot problem after hard shutdown while resilvering RAIDZ1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Mar 2010 09:33:17 -0000 Hello. I have the same problem. You can do three things: 1) do nothing and = wait, because ZFS pool starts resilvering 2) set LD_LIBRARY_PATH to = /zroot and use all utilites 3) Exit from fixit after import and enter it = again. ZFS will be mounted and shell will be functional. On 06.03.2010, at 11:04, Morrison wrote: > Hello folks. >=20 > Here are my specs: > FreeBSD Current 8.0 (amd64) > 3x 2TB HDD combined to a RAIDZ1 > GPT > Root is running on ZFS, > I used the following Guideline = (http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1) >=20 >=20 > Here is the problem: >=20 > I had to replace a drive, but chose by accident a spare which hat some = strange malfunction as well. > I used the command:=20 > zpool replace zroot gpt/disk1 >=20 > zpool status=20 > gave me the information that resilvering would take 2500h = (increasing). > After a couple of MB were resilvered (very slowly), I decided to shut = the system down safely, which didn't work, so I powered it off by = holding the power switch. > I detached the malfunctioned spare drive and tried to start the system = with the two HDDs, but now I am getting >=20 > ZFS: can only boot from disk, mirror or raidz vdevs > ZFS: i/o error - all block copies unavailable > ZFS: can't read object set for dataset lld > Can't find root filesystem - giving up > ZFS: unexpected object set type lld > ZFS: unexpected object set type lld >=20 > FreeBSD/i386 boot > Default: zroot:/boot/kernel/kernel > boot: > ZFS: unexpected object set type lld >=20 > FreeBSD/i386 boot > Default: zroot:/boot/kernel/kernel > boot:_ >=20 > I used my LiveFS USB and tried with Fixit, after loading = opensolaris.ko and zfs.ko: > zpool import -f zroot >=20 > After that, all HDDs started making noises as if they try to = heal/restore them self. But all commands I entered then, gave me errors = like: > /libexec/ld-elf.so.1: Shared object "libavl.so.2" not found, = required by "zpool" > Even sImple commands like "ls" produce simular messages. > I waited one night until the HDDs went silent and rebooted, but = without any luck. >=20 >=20 > Question: > Is there any way of getting this system up again? > =09 >=20 > Regards, > Morrison_______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Alexey Tarasov (\__/)=20 (=3D'.'=3D)=20 E[: | | | | :]=D0=97=20 (")_(")