From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 02:22:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EEC6106566C for ; Sun, 9 Jan 2011 02:22:06 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id E8B978FC0A for ; Sun, 9 Jan 2011 02:22:05 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.13.8+Sun/8.13.8) with ESMTP id p092M4mW007455; Sat, 8 Jan 2011 20:22:04 -0600 (CST) Date: Sat, 8 Jan 2011 20:22:04 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Ivan Voras In-Reply-To: Message-ID: References: User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Sat, 08 Jan 2011 20:22:05 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 02:22:06 -0000 On Sun, 9 Jan 2011, Ivan Voras wrote: > Phoronix is not exactly the epitome of correct benchmarking but it appears > that they've at least recently found out about the existence of error bars. Yes, but you go read the site anyway. > In summary: very varied results, some are expected (low parallel write small > IOPS for FreeBSD), some are not (apparently the BSDs have a monopoly on > high-performance gzip :) ). But overall, pretty good relative results for > FreeBSD, better than earlier. Most of the results don't seem very believable to me. The filesystems behave quite differently in terms of the filesystem coherency they assure. Filesystem caching can also be quite different. There is no telling what these various benchmarks are actually measuring. The only slightly trustable benchmark I saw was the time to extract the Linux kernel, and that has to be taken with a grain of salt since it is not clear what would be on disk if one was to pull the power plug immediately when the extraction was claimed to be done. In some cases, possibly almost nothing. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 02:44:21 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7235E106564A for ; Sun, 9 Jan 2011 02:44:21 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 3D4698FC08 for ; Sun, 9 Jan 2011 02:44:20 +0000 (UTC) Received: from omta09.emeryville.ca.mail.comcast.net ([76.96.30.20]) by qmta01.emeryville.ca.mail.comcast.net with comcast id t2AU1f0040S2fkCA1EkLwb; Sun, 09 Jan 2011 02:44:20 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta09.emeryville.ca.mail.comcast.net with comcast id tEkK1f00A2tehsa8VEkKYM; Sun, 09 Jan 2011 02:44:20 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4550D9B427; Sat, 8 Jan 2011 18:44:19 -0800 (PST) Date: Sat, 8 Jan 2011 18:44:19 -0800 From: Jeremy Chadwick To: Bob Friesenhahn Message-ID: <20110109024419.GA25374@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 02:44:21 -0000 On Sat, Jan 08, 2011 at 08:22:04PM -0600, Bob Friesenhahn wrote: > On Sun, 9 Jan 2011, Ivan Voras wrote: > > >Phoronix is not exactly the epitome of correct benchmarking but it > >appears that they've at least recently found out about the > >existence of error bars. > > Yes, but you go read the site anyway. Which in turn makes me wonder why no one from the FreeBSD community (developer, contributor, or user) is offering to help the Phoronix folks "become more familiar" with FreeBSD and/or help track down or explain whatever "issues" they encounter, rather than just constantly ragging on them. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 02:50:43 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFC8A106566B; Sun, 9 Jan 2011 02:50:43 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 17EA08FC08; Sun, 9 Jan 2011 02:50:42 +0000 (UTC) Received: by fxm16 with SMTP id 16so17988313fxm.13 for ; Sat, 08 Jan 2011 18:50:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=YT7XggNIfcxNg23goluOU7B7Cprs8qWVJTCeoUCVytE=; b=ZeusRbdak2kH10sqs6GQsj0ziHHZG5eZnE+zphQxd/5FsRXbueJJIllzmGbssGuBzF UlOCz/BZQQFLl5s878YTJoOooKhQ3rUJr767uRU4E9jcxktY6dLyH+R6DFTW5S9rBf30 AbHo3U6FuUaXmRG5dlvnIWE3HY+vgLSZ8BgOU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=YRIeT7PYgUUWjwLOeXI5+zPb1eDk9rUoVvy9M1WsfGddl3Pjpg2ScRMnn7W1uHSIo2 5N7aROElVu0Vk9lh0X/TfrADod8H5UEJvx6wJTRLTXXApW1gTOwZ4wLcae489BqkaIi/ 4w23e80Qe/7K5fnAgtcftnvNKUyiHwNSheeW0= MIME-Version: 1.0 Received: by 10.223.116.1 with SMTP id k1mr5447493faq.51.1294541442009; Sat, 08 Jan 2011 18:50:42 -0800 (PST) Received: by 10.223.114.4 with HTTP; Sat, 8 Jan 2011 18:50:41 -0800 (PST) In-Reply-To: <20110109024419.GA25374@icarus.home.lan> References: <20110109024419.GA25374@icarus.home.lan> Date: Sat, 8 Jan 2011 20:50:41 -0600 Message-ID: From: Adam Vande More To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 02:50:43 -0000 On Sat, Jan 8, 2011 at 8:44 PM, Jeremy Chadwick wrote: > On Sat, Jan 08, 2011 at 08:22:04PM -0600, Bob Friesenhahn wrote: > > On Sun, 9 Jan 2011, Ivan Voras wrote: > > > > >Phoronix is not exactly the epitome of correct benchmarking but it > > >appears that they've at least recently found out about the > > >existence of error bars. > > > > Yes, but you go read the site anyway. > > Which in turn makes me wonder why no one from the FreeBSD community > (developer, contributor, or user) is offering to help the Phoronix folks > "become more familiar" with FreeBSD and/or help track down or explain > whatever "issues" they encounter, rather than just constantly ragging > on them. > They have, me and others(at least one with @freebsd address). Phoronix testers don't respond, sometimes trying to incorporate suggestions except they are still quite sloppily done. Phoronix has enough man power and skills to provide more reasonable and fairly structured tests and choose not too, so that is why they get ragged on. -- Adam Vande More From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 02:54:00 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98F58106564A for ; Sun, 9 Jan 2011 02:54:00 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2DC8B8FC0A for ; Sun, 9 Jan 2011 02:53:59 +0000 (UTC) Received: by fxm16 with SMTP id 16so17989056fxm.13 for ; Sat, 08 Jan 2011 18:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=yjG0k6UZM8j2ZUCqhKfIIsKj5hvxCoJJ9VqbBfOqY7Q=; b=Sou0Ob4SMeCyL1vCVRANfNO2+u+Fd/iNeQkF33hpsUZEJ530mo9ntpLZd+4ubD2/wP nA4FUnelzUrOEkXZIBLSODY0LYdTdz2lEvtsd4mXGP5fMIpHa1PyquX6NSDBhSryca0s 5BxckrZbLqn+0T3BuKtdlP7/TPJKKPPdogZkI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=JaWmO916rOIcPkA1P4HRHpMDYMZ2vnfpwbfJvePITyz8WCfsZUuBtG297Bus3O4Z5u uhY4uk2SbupkL3cN/5fQMd6G4kIHJEj5dboc7zUQBCtP5/5H510+ILWSN4fyvba8Ghdx QV3K+/q4tXirdUdSUtEX2zBiCYNKy647FDniE= MIME-Version: 1.0 Received: by 10.223.95.202 with SMTP id e10mr3998964fan.32.1294540208419; Sat, 08 Jan 2011 18:30:08 -0800 (PST) Received: by 10.223.114.4 with HTTP; Sat, 8 Jan 2011 18:30:08 -0800 (PST) In-Reply-To: References: Date: Sat, 8 Jan 2011 20:30:08 -0600 Message-ID: From: Adam Vande More To: Ivan Voras Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 02:54:00 -0000 On Sat, Jan 8, 2011 at 5:35 PM, Ivan Voras wrote: > In summary: very varied results, some are expected (low parallel write > small IOPS for FreeBSD), some are not (apparently the BSDs have a monopoly > on high-performance gzip :) ). > I think might have read the graph backwards or I'm misinterpreting your statement. The BSD's have a monopoly on low performance gzip meaning they aren't as fast as the Linux counterparts. That's not much a fs benchmark though as it's mostly a cpu bound operation and the various improvements to gcc and other bits allows Linux to clearly beat out the BSD's. -- Adam Vande More From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 03:20:05 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A0151065670 for ; Sun, 9 Jan 2011 03:20:05 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 160138FC12 for ; Sun, 9 Jan 2011 03:20:04 +0000 (UTC) Received: by qwj9 with SMTP id 9so18169243qwj.13 for ; Sat, 08 Jan 2011 19:20:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:sender:received :in-reply-to:references:from:date:x-google-sender-auth:message-id :subject:to:cc:content-type:content-transfer-encoding; bh=AjrGmwxmhmbIRecW9OT+JI/uWUsjJuGi9xfd3F9EpCc=; b=nMrDNgfG9SOyoEVG2YQEv/Y8shAU54OZ+dl48dLlb5LwdkuZdDIYEgCwDAEVOV1U3r yfzR1JbWLJDIJVoByFvSXFZqdmW4GT30rDUG7QBlmbt2xJmveyqfufyADM7qk5fqVgQD XgQOBvbJ624i7fkbD5z+v83aa/sV5MwtxM96Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=l3+AlYPmeIWvTMpKj3lSKNf1yncoSXp8pJvdqjMqciPqRi8PDtsMULIcGhueR2/ITX W8+gK3/xspKcKGTsaZj237//Yoo1MrGRONSyGnNEiDmVL1Z828/01aHPGzsXzvW5237A gswB8bCN0U92dNbuDrLYZNaFTdgKdWpcFLeVU= Received: by 10.229.190.204 with SMTP id dj12mr4734217qcb.101.1294541812974; Sat, 08 Jan 2011 18:56:52 -0800 (PST) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.229.44.70 with HTTP; Sat, 8 Jan 2011 18:56:12 -0800 (PST) In-Reply-To: References: From: Ivan Voras Date: Sun, 9 Jan 2011 03:56:12 +0100 X-Google-Sender-Auth: qzdAOT89_O8K3aBWgRdKSb9R5HE Message-ID: To: Adam Vande More Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 03:20:05 -0000 On 9 January 2011 03:30, Adam Vande More wrote: > On Sat, Jan 8, 2011 at 5:35 PM, Ivan Voras wrote: >> >> In summary: very varied results, some are expected (low parallel write >> small IOPS for FreeBSD), some are not (apparently the BSDs have a monopo= ly >> on high-performance gzip :) ). > > I think might have read the graph backwards or I'm misinterpreting your > statement.=C2=A0 The BSD's have a monopoly on low performance gzip meanin= g they > aren't as fast as the Linux counterparts. You're right. Which is interesting - it isn't a (simple) compiler difference since DragonflyBSD has gcc 4.4 also. > That's not much a fs benchmark Yes, they keep ignoring that remark. From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 03:44:31 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE3A6106566C for ; Sun, 9 Jan 2011 03:44:31 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [76.96.30.80]) by mx1.freebsd.org (Postfix) with ESMTP id A79128FC13 for ; Sun, 9 Jan 2011 03:44:31 +0000 (UTC) Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27]) by qmta08.emeryville.ca.mail.comcast.net with comcast id swNk1f0040b6N64A8FkWjB; Sun, 09 Jan 2011 03:44:30 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta03.emeryville.ca.mail.comcast.net with comcast id tFkV1f00L2tehsa8PFkVea; Sun, 09 Jan 2011 03:44:30 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5E6DD9B427; Sat, 8 Jan 2011 19:44:29 -0800 (PST) Date: Sat, 8 Jan 2011 19:44:29 -0800 From: Jeremy Chadwick To: Ivan Voras Message-ID: <20110109034429.GA26564@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 03:44:31 -0000 On Sun, Jan 09, 2011 at 03:56:12AM +0100, Ivan Voras wrote: > On 9 January 2011 03:30, Adam Vande More wrote: > > On Sat, Jan 8, 2011 at 5:35 PM, Ivan Voras wrote: > >> > >> In summary: very varied results, some are expected (low parallel write > >> small IOPS for FreeBSD), some are not (apparently the BSDs have a monopoly > >> on high-performance gzip :) ). > > > > I think might have read the graph backwards or I'm misinterpreting your > > statement.  The BSD's have a monopoly on low performance gzip meaning they > > aren't as fast as the Linux counterparts. > > You're right. Which is interesting - it isn't a (simple) compiler > difference since DragonflyBSD has gcc 4.4 also. > > > That's not much a fs benchmark > > Yes, they keep ignoring that remark. I agree it's not much of a filesystem benchmark but more of a "usability" benchmark. I actually see the legitimacy of using it as a benchmark point, but shouldn't be classified as a filesystem bench. Has anyone actually taken the time to profile gzip and libz on BSD to find out where it spends most of its time compared to Linux? It could be something as simple as a write() operation with a buffer size that isn't optimal (compared to what the underlying filesystem has), or equally some badly-sized buffers or operations/logic used within libz. And also remember, FreeBSD's gzip is not pure GNU gzip (see man page). All I'm trying to say here is that the effort to determine why BSD's gzip behaves worse than on Linux is something the community or developers should investigate -- if anything, maybe it's good Phoronix indirectly brought this to light. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 09:00:55 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0952C106564A; Sun, 9 Jan 2011 09:00:55 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 01C728FC13; Sun, 9 Jan 2011 09:00:53 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 737A2704FAF; Sun, 9 Jan 2011 10:00:52 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 10.4131] X-CRM114-CacheID: sfid-20110109_10005_8B1EC28F X-CRM114-Status: Good ( pR: 10.4131 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D297943.1040507@fsn.hu> Date: Sun, 09 Jan 2011 10:00:51 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Martin Matuska References: <4D0A09AF.3040005@FreeBSD.org> In-Reply-To: <4D0A09AF.3040005@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 09:00:55 -0000 On 12/16/2010 01:44 PM, Martin Matuska wrote: > Hi everyone, > > following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am > providing a ZFSv28 testing patch for 8-STABLE. > > Link to the patch: > > http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz > I've got an IO hang with dedup enabled (not sure it's related, I've started to rewrite all data on pool, which makes a heavy load): The processes are in various states: 65747 1001 1 54 10 28620K 24360K tx->tx 0 6:58 0.00% cvsup 80383 1001 1 54 10 40616K 30196K select 1 5:38 0.00% rsync 1501 www 1 44 0 7304K 2504K zio->i 0 2:09 0.00% nginx 1479 www 1 44 0 7304K 2416K zio->i 1 2:03 0.00% nginx 1477 www 1 44 0 7304K 2664K zio->i 0 2:02 0.00% nginx 1487 www 1 44 0 7304K 2376K zio->i 0 1:40 0.00% nginx 1490 www 1 44 0 7304K 1852K zfs 0 1:30 0.00% nginx 1486 www 1 44 0 7304K 2400K zfsvfs 1 1:05 0.00% nginx And everything which wants to touch the pool is/becomes dead. Procstat says about one process: # procstat -k 1497 PID TID COMM TDNAME KSTACK 1497 100257 nginx - mi_switch sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV null_lock VOP_LOCK1_APV _vn_lock nullfs_root lookup namei vn_open_cred kern_openat syscallenter syscall Xfast_syscall From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 11:49:30 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 852B31065696; Sun, 9 Jan 2011 11:49:30 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 7A58D8FC1B; Sun, 9 Jan 2011 11:49:28 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id E95DA70458C; Sun, 9 Jan 2011 12:49:27 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 13.6024] X-CRM114-CacheID: sfid-20110109_12492_927F0F69 X-CRM114-Status: Good ( pR: 13.6024 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D29A0C7.8050002@fsn.hu> Date: Sun, 09 Jan 2011 12:49:27 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Martin Matuska References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> In-Reply-To: <4D297943.1040507@fsn.hu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 11:49:30 -0000 On 01/09/2011 10:00 AM, Attila Nagy wrote: > On 12/16/2010 01:44 PM, Martin Matuska wrote: >> Hi everyone, >> >> following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am >> providing a ZFSv28 testing patch for 8-STABLE. >> >> Link to the patch: >> >> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz >> >> > I've got an IO hang with dedup enabled (not sure it's related, I've > started to rewrite all data on pool, which makes a heavy load): > > The processes are in various states: > 65747 1001 1 54 10 28620K 24360K tx->tx 0 6:58 0.00% cvsup > 80383 1001 1 54 10 40616K 30196K select 1 5:38 0.00% rsync > 1501 www 1 44 0 7304K 2504K zio->i 0 2:09 0.00% nginx > 1479 www 1 44 0 7304K 2416K zio->i 1 2:03 0.00% nginx > 1477 www 1 44 0 7304K 2664K zio->i 0 2:02 0.00% nginx > 1487 www 1 44 0 7304K 2376K zio->i 0 1:40 0.00% nginx > 1490 www 1 44 0 7304K 1852K zfs 0 1:30 0.00% nginx > 1486 www 1 44 0 7304K 2400K zfsvfs 1 1:05 0.00% nginx > > And everything which wants to touch the pool is/becomes dead. > > Procstat says about one process: > # procstat -k 1497 > PID TID COMM TDNAME KSTACK > 1497 100257 nginx - mi_switch sleepq_wait > __lockmgr_args vop_stdlock VOP_LOCK1_APV null_lock VOP_LOCK1_APV > _vn_lock nullfs_root lookup namei vn_open_cred kern_openat > syscallenter syscall Xfast_syscall No, it's not related. One of the disks in the RAIDZ2 pool went bad: (da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 (da4:arcmsr0:0:4:0): CAM status: SCSI Status Error (da4:arcmsr0:0:4:0): SCSI status: Check Condition (da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) and it seems it froze the whole zpool. Removing the disk by hand solved the problem. I've seen this previously on other machines with ciss. I wonder why ZFS didn't throw it out of the pool. From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 11:52:59 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BA7B106566B; Sun, 9 Jan 2011 11:52:59 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 5E2E38FC22; Sun, 9 Jan 2011 11:52:57 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id D0E1C7045D3; Sun, 9 Jan 2011 12:52:56 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 15.4350] X-CRM114-CacheID: sfid-20110109_12525_6A621F56 X-CRM114-Status: Good ( pR: 15.4350 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D29A198.4070107@fsn.hu> Date: Sun, 09 Jan 2011 12:52:56 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Artem Belevich References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 11:52:59 -0000 On 01/01/2011 08:09 PM, Artem Belevich wrote: > On Sat, Jan 1, 2011 at 10:18 AM, Attila Nagy wrote: >> What I see: >> - increased CPU load >> - decreased L2 ARC hit rate, decreased SSD (ad[46]), therefore increased >> hard disk load (IOPS graph) >> > ... >> Any ideas on what could cause these? I haven't upgraded the pool version and >> nothing was changed in the pool or in the file system. > The fact that L2 ARC is full does not mean that it contains the right > data. Initial L2ARC warm up happens at a much higher rate than the > rate L2ARC is updated after it's been filled initially. Even > accelerated warm-up took almost a day in your case. In order for L2ARC > to warm up properly you may have to wait quite a bit longer. My guess > is that it should slowly improve over the next few days as data goes > through L2ARC and those bits that are hit more often take residence > there. The larger your data set, the longer it will take for L2ARC to > catch the right data. > > Do you have similar graphs from pre-patch system just after reboot? I > suspect that it may show similarly abysmal L2ARC hit rates initially, > too. > > I've finally found the time to read the v28 patch and figured out the problem: vfs.zfs.l2arc_noprefetch was changed to 1, so it doesn't use the prefetched data on the L2ARC devices. This is a major hit in my case. Enabling this again restored the previous hit rates and lowered the load on the hard disks significantly. From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 12:18:03 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D58421065674 for ; Sun, 9 Jan 2011 12:18:03 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.westchester.pa.mail.comcast.net (qmta02.westchester.pa.mail.comcast.net [76.96.62.24]) by mx1.freebsd.org (Postfix) with ESMTP id 823538FC15 for ; Sun, 9 Jan 2011 12:18:03 +0000 (UTC) Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20]) by qmta02.westchester.pa.mail.comcast.net with comcast id tQAq1f0080SCNGk52QJ3sD; Sun, 09 Jan 2011 12:18:03 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta09.westchester.pa.mail.comcast.net with comcast id tQJ21f0022tehsa3VQJ2Mx; Sun, 09 Jan 2011 12:18:03 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id E62F39B427; Sun, 9 Jan 2011 04:18:00 -0800 (PST) Date: Sun, 9 Jan 2011 04:18:00 -0800 From: Jeremy Chadwick To: Attila Nagy Message-ID: <20110109121800.GA37231@icarus.home.lan> References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> <4D29A0C7.8050002@fsn.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D29A0C7.8050002@fsn.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Martin Matuska Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 12:18:03 -0000 On Sun, Jan 09, 2011 at 12:49:27PM +0100, Attila Nagy wrote: > On 01/09/2011 10:00 AM, Attila Nagy wrote: > > On 12/16/2010 01:44 PM, Martin Matuska wrote: > >>Hi everyone, > >> > >>following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am > >>providing a ZFSv28 testing patch for 8-STABLE. > >> > >>Link to the patch: > >> > >>http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz > >> > >> > >I've got an IO hang with dedup enabled (not sure it's related, > >I've started to rewrite all data on pool, which makes a heavy > >load): > > > >The processes are in various states: > >65747 1001 1 54 10 28620K 24360K tx->tx 0 6:58 0.00% cvsup > >80383 1001 1 54 10 40616K 30196K select 1 5:38 0.00% rsync > > 1501 www 1 44 0 7304K 2504K zio->i 0 2:09 0.00% nginx > > 1479 www 1 44 0 7304K 2416K zio->i 1 2:03 0.00% nginx > > 1477 www 1 44 0 7304K 2664K zio->i 0 2:02 0.00% nginx > > 1487 www 1 44 0 7304K 2376K zio->i 0 1:40 0.00% nginx > > 1490 www 1 44 0 7304K 1852K zfs 0 1:30 0.00% nginx > > 1486 www 1 44 0 7304K 2400K zfsvfs 1 1:05 0.00% nginx > > > >And everything which wants to touch the pool is/becomes dead. > > > >Procstat says about one process: > ># procstat -k 1497 > > PID TID COMM TDNAME KSTACK > > 1497 100257 nginx - mi_switch > >sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV null_lock > >VOP_LOCK1_APV _vn_lock nullfs_root lookup namei vn_open_cred > >kern_openat syscallenter syscall Xfast_syscall > No, it's not related. One of the disks in the RAIDZ2 pool went bad: > (da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 > (da4:arcmsr0:0:4:0): CAM status: SCSI Status Error > (da4:arcmsr0:0:4:0): SCSI status: Check Condition > (da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered > read error) > and it seems it froze the whole zpool. Removing the disk by hand > solved the problem. > I've seen this previously on other machines with ciss. > I wonder why ZFS didn't throw it out of the pool. Hold on a minute. An unrecoverable read error does not necessarily mean the drive is bad, it could mean that the individual LBA that was attempted to be read resulted in ASC 0x11 (MEDIUM ERROR) (e.g. a bad block was encountered). I would check SMART stats on the disk (since these are probably SATA given use of arcmsr(4)) and provide those. *That* will tell you if the disk is bad. I'll help you decode the attributes values if you provide them. My understanding is that a single LBA read failure should not warrant ZFS marking the disk UNAVAIL in the pool. It should have incremented the READ error counter and that's it. Did you receive a *single* error for the disk and then things went catatonic? If the entire system got wedged (a soft wedge, e.g. kernel is still alive but nothing's happening in userland), that could be a different problem -- either with ZFS or arcmsr(4). Does ZFS have some sort of timeout value internal to itself where it will literally mark a disk UNAVAIL in the case that repeated I/O transactions takes "too long"? What is its error recovery methodology? Speaking strictly about Solaris 10 and ZFS: I have seen many, many times a system "soft wedge" after repeated I/O errors (read or write) are spewed out on the console for a single SATA disk (via AHCI), but only when the disk is used as a sole root filesystem disk (no mirror/raidz). My impression is that ZFS isn't the problem in this scenario. In most cases, post-mortem debugging on my part shows that disks encountered some CRC errors (indicating cabling issues, etc.), sometimes as few as 2, but "something else" went crazy -- or possibly ZFS couldn't mark the disk UNAVAIL (if it has that logic) because it's a single disk associated with root. Hardware in this scenario are Hitachi SATA disks with an ICH ESB2 controller, software is Solaris 10 (Generic_142901-06) with ZFS v15. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 12:42:17 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23C081065674; Sun, 9 Jan 2011 12:42:17 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id A86118FC0C; Sun, 9 Jan 2011 12:42:15 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 267D7704876; Sun, 9 Jan 2011 13:42:14 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 31.0131] X-CRM114-CacheID: sfid-20110109_13421_CEB29D0A X-CRM114-Status: Good ( pR: 31.0131 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D29AD25.2030501@fsn.hu> Date: Sun, 09 Jan 2011 13:42:13 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> <4D29A0C7.8050002@fsn.hu> <20110109121800.GA37231@icarus.home.lan> In-Reply-To: <20110109121800.GA37231@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Martin Matuska Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 12:42:17 -0000 On 01/09/2011 01:18 PM, Jeremy Chadwick wrote: > On Sun, Jan 09, 2011 at 12:49:27PM +0100, Attila Nagy wrote: >> On 01/09/2011 10:00 AM, Attila Nagy wrote: >>> On 12/16/2010 01:44 PM, Martin Matuska wrote: >>>> Hi everyone, >>>> >>>> following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am >>>> providing a ZFSv28 testing patch for 8-STABLE. >>>> >>>> Link to the patch: >>>> >>>> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz >>>> >>>> >>> I've got an IO hang with dedup enabled (not sure it's related, >>> I've started to rewrite all data on pool, which makes a heavy >>> load): >>> >>> The processes are in various states: >>> 65747 1001 1 54 10 28620K 24360K tx->tx 0 6:58 0.00% cvsup >>> 80383 1001 1 54 10 40616K 30196K select 1 5:38 0.00% rsync >>> 1501 www 1 44 0 7304K 2504K zio->i 0 2:09 0.00% nginx >>> 1479 www 1 44 0 7304K 2416K zio->i 1 2:03 0.00% nginx >>> 1477 www 1 44 0 7304K 2664K zio->i 0 2:02 0.00% nginx >>> 1487 www 1 44 0 7304K 2376K zio->i 0 1:40 0.00% nginx >>> 1490 www 1 44 0 7304K 1852K zfs 0 1:30 0.00% nginx >>> 1486 www 1 44 0 7304K 2400K zfsvfs 1 1:05 0.00% nginx >>> >>> And everything which wants to touch the pool is/becomes dead. >>> >>> Procstat says about one process: >>> # procstat -k 1497 >>> PID TID COMM TDNAME KSTACK >>> 1497 100257 nginx - mi_switch >>> sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV null_lock >>> VOP_LOCK1_APV _vn_lock nullfs_root lookup namei vn_open_cred >>> kern_openat syscallenter syscall Xfast_syscall >> No, it's not related. One of the disks in the RAIDZ2 pool went bad: >> (da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 >> (da4:arcmsr0:0:4:0): CAM status: SCSI Status Error >> (da4:arcmsr0:0:4:0): SCSI status: Check Condition >> (da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered >> read error) >> and it seems it froze the whole zpool. Removing the disk by hand >> solved the problem. >> I've seen this previously on other machines with ciss. >> I wonder why ZFS didn't throw it out of the pool. > Hold on a minute. An unrecoverable read error does not necessarily mean > the drive is bad, it could mean that the individual LBA that was > attempted to be read resulted in ASC 0x11 (MEDIUM ERROR) (e.g. a bad > block was encountered). I would check SMART stats on the disk (since > these are probably SATA given use of arcmsr(4)) and provide those. > *That* will tell you if the disk is bad. I'll help you decode the > attributes values if you provide them. You are right, and I gave incorrect information. There are a lot more errors for that disk in the logs, and the zpool was frozen. I tried to offline the given disk. That helped in the ciss case, where the symptom is the same, or something similar, like there is no IO for ages, then something small and nothing for long seconds/minutes, and there are no errors logged. zpool status reported no errors, and the dmesg was clear too. There I could find the bad disk by watching gstat output and there I saw when the very small amount of IO was done, there was one disk with response times well above a second, while the others responded quickly. There the zpool offline helped. Here not, the command just got hang, like everything else. So what I did then: got into the areca-cli and searched for errors. One disk was set to failed and it seemed to be the cause. I've removed it (and did a camcontrol rescan, but I'm not sure it was necessary or not), and suddenly the zpool offline finished and everything went back to normal. But there are two controllers in the system and now I see that the above disk is on ctrl 1, while the one I have removed is on ctrl 2. I was misleaded by their same position. So now I have an offlined disk (which produces read errors, but I couldn't see them in the zpool output) and another, which is shown as failed in the RAID controller and got removed by hand (and solved the situation): NAME STATE READ WRITE CKSUM data DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 label/disk20-01 ONLINE 0 0 0 label/disk20-02 ONLINE 0 0 0 label/disk20-03 ONLINE 0 0 0 label/disk20-04 ONLINE 0 0 0 label/disk20-05 OFFLINE 0 0 0 label/disk20-06 ONLINE 0 0 0 label/disk20-07 ONLINE 0 0 0 label/disk20-08 ONLINE 0 0 0 label/disk20-09 ONLINE 0 0 0 label/disk20-10 ONLINE 0 0 0 label/disk20-11 ONLINE 0 0 0 label/disk20-12 ONLINE 0 0 0 raidz2-1 DEGRADED 0 0 0 label/disk21-01 ONLINE 0 0 0 label/disk21-02 ONLINE 0 0 0 label/disk21-03 ONLINE 0 0 0 label/disk21-04 ONLINE 0 0 0 label/disk21-05 REMOVED 0 0 0 label/disk21-06 ONLINE 0 0 0 label/disk21-07 ONLINE 0 0 0 label/disk21-08 ONLINE 0 0 0 label/disk21-09 ONLINE 0 0 0 label/disk21-10 ONLINE 0 0 0 label/disk21-11 ONLINE 0 0 0 label/disk21-12 ONLINE 0 0 0 cache ad4s2 ONLINE 0 0 0 ad6s2 ONLINE 0 0 0 > My understanding is that a single LBA read failure should not warrant > ZFS marking the disk UNAVAIL in the pool. It should have incremented > the READ error counter and that's it. Did you receive a *single* error > for the disk and then things went catatonic? No, there are more entries in the logs, but only for disk 20-05, which I zpool offlined without success and nothing about disk 21-05, which I have removed by hand in the controller, which marked it as failed. As you can see, all counters are zero, I haven't cleared them. > If the entire system got wedged (a soft wedge, e.g. kernel is still > alive but nothing's happening in userland), that could be a different > problem -- either with ZFS or arcmsr(4). Does ZFS have some sort of > timeout value internal to itself where it will literally mark a disk > UNAVAIL in the case that repeated I/O transactions takes "too long"? > What is its error recovery methodology? No, everything was fine, because the system is on UFS. So only the zpool was dead, I could log into the machine. That's what I'm asking too, because I saw some similar errors (see above) on different machines, and from that I would think there is no forced timeout, this can make the system livelock like this. I use ZFS mostly on RAID controllers, but if I remember correctly I could see exactly the same problem on an X4540 too (which has a plain SAS controller), gstat helped there too, by showing which disk has a very long response time. > Speaking strictly about Solaris 10 and ZFS: I have seen many, many times > a system "soft wedge" after repeated I/O errors (read or write) are > spewed out on the console for a single SATA disk (via AHCI), but only > when the disk is used as a sole root filesystem disk (no mirror/raidz). > My impression is that ZFS isn't the problem in this scenario. In most > cases, post-mortem debugging on my part shows that disks encountered > some CRC errors (indicating cabling issues, etc.), sometimes as few as > 2, but "something else" went crazy -- or possibly ZFS couldn't mark the > disk UNAVAIL (if it has that logic) because it's a single disk > associated with root. Hardware in this scenario are Hitachi SATA disks > with an ICH ESB2 controller, software is Solaris 10 (Generic_142901-06) > with ZFS v15. > I don't think it's cabling, it's the disks. I could repair all these errors by replacing the disks with a new one, and these are hot swap drives, so I wouldn't think about physical contact errors. From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 12:54:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 021BB106566B; Sun, 9 Jan 2011 12:54:04 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 771228FC13; Sun, 9 Jan 2011 12:54:03 +0000 (UTC) Received: by qyk36 with SMTP id 36so18107733qyk.13 for ; Sun, 09 Jan 2011 04:54:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=bVxi646sCt6eWAFASlMHwz0m2lkBuEw/d/PAATnLj7o=; b=BNx/gMGYssz3Szu7ekDko4hzKMfPT/hljMrWtZSLINvQ8V8L2cxly/A1QJplRzZwl1 /CUC0DxZpTdFfR9VeVY7Wo+hipUwi4SEF7eVCUEIBqGhmJkf/vjpRWedYvlyige2E7j9 CQBOkeNfFJLeA38cx2D265UepbOkWAZ+kF5v0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=pwwb3sZTvxuB/JZdTePbYESZMUNd89hRCd8wg+bQ30miy4nhglFW29wlD646gxQqQ8 5Q0uYgbl5S82br3OVV5tM8g2/10uz4qPzmHhyHu0+jSQFM8tPMQ+D68xj3Fo5qZ2DXXp Z2NxbpxXxPIrkhokauWFGqxIZ2SR2N3IO04Hk= MIME-Version: 1.0 Received: by 10.229.246.79 with SMTP id lx15mr24578675qcb.25.1294575755846; Sun, 09 Jan 2011 04:22:35 -0800 (PST) Received: by 10.229.230.5 with HTTP; Sun, 9 Jan 2011 04:22:35 -0800 (PST) In-Reply-To: <20110109121800.GA37231@icarus.home.lan> References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> <4D29A0C7.8050002@fsn.hu> <20110109121800.GA37231@icarus.home.lan> Date: Sun, 9 Jan 2011 07:22:35 -0500 Message-ID: From: Rich To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 12:54:04 -0000 Once upon a time, this was a known problem with the arcmsr driver not correctly interacting with ZFS, resulting in this behavior. Since I'm presuming that the arcmsr driver update which was intended to fix this behavior (in my case, at least) is in your nightly build, it's probably worth pinging the arcmsr driver maintainer about this. - Rich On Sun, Jan 9, 2011 at 7:18 AM, Jeremy Chadwick wrote: > On Sun, Jan 09, 2011 at 12:49:27PM +0100, Attila Nagy wrote: >> =A0On 01/09/2011 10:00 AM, Attila Nagy wrote: >> > On 12/16/2010 01:44 PM, Martin Matuska wrote: >> >>Hi everyone, >> >> >> >>following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I = am >> >>providing a ZFSv28 testing patch for 8-STABLE. >> >> >> >>Link to the patch: >> >> >> >>http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215= .patch.xz >> >> >> >> >> >I've got an IO hang with dedup enabled (not sure it's related, >> >I've started to rewrite all data on pool, which makes a heavy >> >load): >> > >> >The processes are in various states: >> >65747 =A0 1001 =A0 =A0 =A01 =A054 =A0 10 28620K 24360K tx->tx =A00 =A0 = 6:58 =A00.00% cvsup >> >80383 =A0 1001 =A0 =A0 =A01 =A054 =A0 10 40616K 30196K select =A01 =A0 = 5:38 =A00.00% rsync >> > 1501 www =A0 =A0 =A0 =A0 1 =A044 =A0 =A00 =A07304K =A02504K zio->i =A0= 0 =A0 2:09 =A00.00% nginx >> > 1479 www =A0 =A0 =A0 =A0 1 =A044 =A0 =A00 =A07304K =A02416K zio->i =A0= 1 =A0 2:03 =A00.00% nginx >> > 1477 www =A0 =A0 =A0 =A0 1 =A044 =A0 =A00 =A07304K =A02664K zio->i =A0= 0 =A0 2:02 =A00.00% nginx >> > 1487 www =A0 =A0 =A0 =A0 1 =A044 =A0 =A00 =A07304K =A02376K zio->i =A0= 0 =A0 1:40 =A00.00% nginx >> > 1490 www =A0 =A0 =A0 =A0 1 =A044 =A0 =A00 =A07304K =A01852K zfs =A0 = =A0 0 =A0 1:30 =A00.00% nginx >> > 1486 www =A0 =A0 =A0 =A0 1 =A044 =A0 =A00 =A07304K =A02400K zfsvfs =A0= 1 =A0 1:05 =A00.00% nginx >> > >> >And everything which wants to touch the pool is/becomes dead. >> > >> >Procstat says about one process: >> ># procstat -k 1497 >> > =A0PID =A0 =A0TID COMM =A0 =A0 =A0 =A0 =A0 =A0 TDNAME =A0 =A0 =A0 =A0 = =A0 KSTACK >> > 1497 100257 nginx =A0 =A0 =A0 =A0 =A0 =A0- =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0mi_switch >> >sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV null_lock >> >VOP_LOCK1_APV _vn_lock nullfs_root lookup namei vn_open_cred >> >kern_openat syscallenter syscall Xfast_syscall >> No, it's not related. One of the disks in the RAIDZ2 pool went bad: >> (da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 >> (da4:arcmsr0:0:4:0): CAM status: SCSI Status Error >> (da4:arcmsr0:0:4:0): SCSI status: Check Condition >> (da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered >> read error) >> and it seems it froze the whole zpool. Removing the disk by hand >> solved the problem. >> I've seen this previously on other machines with ciss. >> I wonder why ZFS didn't throw it out of the pool. > > Hold on a minute. =A0An unrecoverable read error does not necessarily mea= n > the drive is bad, it could mean that the individual LBA that was > attempted to be read resulted in ASC 0x11 (MEDIUM ERROR) (e.g. a bad > block was encountered). =A0I would check SMART stats on the disk (since > these are probably SATA given use of arcmsr(4)) and provide those. > *That* will tell you if the disk is bad. =A0I'll help you decode the > attributes values if you provide them. > > My understanding is that a single LBA read failure should not warrant > ZFS marking the disk UNAVAIL in the pool. =A0It should have incremented > the READ error counter and that's it. =A0Did you receive a *single* error > for the disk and then things went catatonic? > > If the entire system got wedged (a soft wedge, e.g. kernel is still > alive but nothing's happening in userland), that could be a different > problem -- either with ZFS or arcmsr(4). =A0Does ZFS have some sort of > timeout value internal to itself where it will literally mark a disk > UNAVAIL in the case that repeated I/O transactions takes "too long"? > What is its error recovery methodology? > > Speaking strictly about Solaris 10 and ZFS: I have seen many, many times > a system "soft wedge" after repeated I/O errors (read or write) are > spewed out on the console for a single SATA disk (via AHCI), but only > when the disk is used as a sole root filesystem disk (no mirror/raidz). > My impression is that ZFS isn't the problem in this scenario. =A0In most > cases, post-mortem debugging on my part shows that disks encountered > some CRC errors (indicating cabling issues, etc.), sometimes as few as > 2, but "something else" went crazy -- or possibly ZFS couldn't mark the > disk UNAVAIL (if it has that logic) because it's a single disk > associated with root. =A0Hardware in this scenario are Hitachi SATA disks > with an ICH ESB2 controller, software is Solaris 10 (Generic_142901-06) > with ZFS v15. > > -- > | Jeremy Chadwick =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 jdc@parodius.com | > | Parodius Networking =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 http://= www.parodius.com/ | > | UNIX Systems Administrator =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Mountain = View, CA, USA | > | Making life hard for others since 1977. =A0 =A0 =A0 =A0 =A0 =A0 =A0 PGP= 4BD6C0CB | > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 13:27:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F7C5106566B for ; Sun, 9 Jan 2011 13:27:06 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 5F2F08FC15 for ; Sun, 9 Jan 2011 13:27:06 +0000 (UTC) Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by qmta06.emeryville.ca.mail.comcast.net with comcast id tRDA1f0040cQ2SLA6RT5bU; Sun, 09 Jan 2011 13:27:05 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta10.emeryville.ca.mail.comcast.net with comcast id tRT41f00J2tehsa8WRT4qX; Sun, 09 Jan 2011 13:27:05 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7525F9B427; Sun, 9 Jan 2011 05:27:04 -0800 (PST) Date: Sun, 9 Jan 2011 05:27:04 -0800 From: Jeremy Chadwick To: Attila Nagy Message-ID: <20110109132704.GA38278@icarus.home.lan> References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> <4D29A0C7.8050002@fsn.hu> <20110109121800.GA37231@icarus.home.lan> <4D29AD25.2030501@fsn.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D29AD25.2030501@fsn.hu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Martin Matuska Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 13:27:06 -0000 On Sun, Jan 09, 2011 at 01:42:13PM +0100, Attila Nagy wrote: > On 01/09/2011 01:18 PM, Jeremy Chadwick wrote: > >On Sun, Jan 09, 2011 at 12:49:27PM +0100, Attila Nagy wrote: > >> On 01/09/2011 10:00 AM, Attila Nagy wrote: > >>>On 12/16/2010 01:44 PM, Martin Matuska wrote: > >>>>Hi everyone, > >>>> > >>>>following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am > >>>>providing a ZFSv28 testing patch for 8-STABLE. > >>>> > >>>>Link to the patch: > >>>> > >>>>http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz > >>>> > >>>> > >>>I've got an IO hang with dedup enabled (not sure it's related, > >>>I've started to rewrite all data on pool, which makes a heavy > >>>load): > >>> > >>>The processes are in various states: > >>>65747 1001 1 54 10 28620K 24360K tx->tx 0 6:58 0.00% cvsup > >>>80383 1001 1 54 10 40616K 30196K select 1 5:38 0.00% rsync > >>>1501 www 1 44 0 7304K 2504K zio->i 0 2:09 0.00% nginx > >>>1479 www 1 44 0 7304K 2416K zio->i 1 2:03 0.00% nginx > >>>1477 www 1 44 0 7304K 2664K zio->i 0 2:02 0.00% nginx > >>>1487 www 1 44 0 7304K 2376K zio->i 0 1:40 0.00% nginx > >>>1490 www 1 44 0 7304K 1852K zfs 0 1:30 0.00% nginx > >>>1486 www 1 44 0 7304K 2400K zfsvfs 1 1:05 0.00% nginx > >>> > >>>And everything which wants to touch the pool is/becomes dead. > >>> > >>>Procstat says about one process: > >>># procstat -k 1497 > >>> PID TID COMM TDNAME KSTACK > >>>1497 100257 nginx - mi_switch > >>>sleepq_wait __lockmgr_args vop_stdlock VOP_LOCK1_APV null_lock > >>>VOP_LOCK1_APV _vn_lock nullfs_root lookup namei vn_open_cred > >>>kern_openat syscallenter syscall Xfast_syscall > >>No, it's not related. One of the disks in the RAIDZ2 pool went bad: > >>(da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 > >>(da4:arcmsr0:0:4:0): CAM status: SCSI Status Error > >>(da4:arcmsr0:0:4:0): SCSI status: Check Condition > >>(da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered > >>read error) > >>and it seems it froze the whole zpool. Removing the disk by hand > >>solved the problem. > >>I've seen this previously on other machines with ciss. > >>I wonder why ZFS didn't throw it out of the pool. > >Hold on a minute. An unrecoverable read error does not necessarily mean > >the drive is bad, it could mean that the individual LBA that was > >attempted to be read resulted in ASC 0x11 (MEDIUM ERROR) (e.g. a bad > >block was encountered). I would check SMART stats on the disk (since > >these are probably SATA given use of arcmsr(4)) and provide those. > >*That* will tell you if the disk is bad. I'll help you decode the > >attributes values if you provide them. > You are right, and I gave incorrect information. There are a lot > more errors for that disk in the logs, and the zpool was frozen. > I tried to offline the given disk. That helped in the ciss case, > where the symptom is the same, or something similar, like there is > no IO for ages, then something small and nothing for long > seconds/minutes, and there are no errors logged. zpool status > reported no errors, and the dmesg was clear too. > There I could find the bad disk by watching gstat output and there I > saw when the very small amount of IO was done, there was one disk > with response times well above a second, while the others responded > quickly. > There the zpool offline helped. Here not, the command just got hang, > like everything else. > So what I did then: got into the areca-cli and searched for errors. > One disk was set to failed and it seemed to be the cause. I've > removed it (and did a camcontrol rescan, but I'm not sure it was > necessary or not), and suddenly the zpool offline finished and > everything went back to normal. > But there are two controllers in the system and now I see that the > above disk is on ctrl 1, while the one I have removed is on ctrl 2. > I was misleaded by their same position. So now I have an offlined > disk (which produces read errors, but I couldn't see them in the > zpool output) and another, which is shown as failed in the RAID > controller and got removed by hand (and solved the situation): > NAME STATE READ WRITE CKSUM > data DEGRADED 0 0 0 > raidz2-0 DEGRADED 0 0 0 > label/disk20-01 ONLINE 0 0 0 > label/disk20-02 ONLINE 0 0 0 > label/disk20-03 ONLINE 0 0 0 > label/disk20-04 ONLINE 0 0 0 > label/disk20-05 OFFLINE 0 0 0 > label/disk20-06 ONLINE 0 0 0 > label/disk20-07 ONLINE 0 0 0 > label/disk20-08 ONLINE 0 0 0 > label/disk20-09 ONLINE 0 0 0 > label/disk20-10 ONLINE 0 0 0 > label/disk20-11 ONLINE 0 0 0 > label/disk20-12 ONLINE 0 0 0 > raidz2-1 DEGRADED 0 0 0 > label/disk21-01 ONLINE 0 0 0 > label/disk21-02 ONLINE 0 0 0 > label/disk21-03 ONLINE 0 0 0 > label/disk21-04 ONLINE 0 0 0 > label/disk21-05 REMOVED 0 0 0 > label/disk21-06 ONLINE 0 0 0 > label/disk21-07 ONLINE 0 0 0 > label/disk21-08 ONLINE 0 0 0 > label/disk21-09 ONLINE 0 0 0 > label/disk21-10 ONLINE 0 0 0 > label/disk21-11 ONLINE 0 0 0 > label/disk21-12 ONLINE 0 0 0 > cache > ad4s2 ONLINE 0 0 0 > ad6s2 ONLINE 0 0 0 With regards to arcmsr(4) and this behaviour: I received a mail from Rich (which doesn't appear to have hit the freebsd-stable list yet, probably due to greylisting; I'm not sure if you received a copy of his mail yet), which indicates the wedging when the disks are on arcmsr(4) is a bug with arcmsr(4). One more thing: > I don't think it's cabling, it's the disks. I could repair all these > errors by replacing the disks with a new one, and these are hot swap > drives, so I wouldn't think about physical contact errors. You're probably right, but I'll share a story that hopefully you won't have the "pleasure" of experiencing. :-) Newer model servers at my workplace are SATA and backed by hot-swap enclosures. On occasion, a box will report disk I/O errors for LBAs which are scattered all across the platters (meaning they aren't even within 1,000,000 LBAs of one another). This causes our NOC to open a ticket stating the problem is a "bad disk". The prognosis is completely wrong (see below), and what ends up happening is our datacenter guys replace the disk only to find that the situation starts happening again within a few days. Meaning: even more customer impact + wasted efforts/time because everyone assumes an I/O error means a disk error. So why was the prognosis wrong? In these situations, I found SMART Attribute 199 (UDMA_CRC_Error_Count) has incremented numerous times (hundreds) during the issue. This attribute represents the number of times ATA command blocks between the disk and the controller didn't have matching CRCs, which is usually caused by a hardware issue. A few CRC errors during a system's lifetime is more than acceptable, but numbers in the hundreds or thousands isn't. The problem turned out to be a SATA backplane that had gone bad (there are some basic ASICs on those things) or SATA cabling between the backplane and the motherboard SATA ports was faulty. More than a few times, we found that the raw copper on the SATA cables had been exposed in a couple spots, a result of the vendor running the cable through the chassis and scraping it against sharp metal. "Oops". Disk troubleshooting is somewhat of an art, or at least thats what I'm starting to feel after doing it for so long. I like having concise explanations for why a storage device reports an error; blindly assuming it's the disk is a bad habit to get into. There are many pieces of hardware, and just as much firmware and software, between a disk and an operating system. Familiarising yourself with it all takes a lot of time (months/years), and *every situation is different*. Can I diagnose every situation that comes across my desk? Hell no. There are many which I simply can't determine the cause of, and in those situations I recommend everything get replaced (sans RAM and CPU). Have I recommended that only to have the same situation happen again? Yes, and I'm lead to believe chassis temperature, humidity, or PSU/voltage problems are responsible. Don't let SSDs fool you either; already in the past year I've seen literally 14 or 15 cases of people having their SSDs go bad, including 4 or 5 of our X25-Ms at work. When they die, it's pretty catastrophic; I haven't seen a "oh my SSD is acting wonky" situation yet, it's either all or nothing. YMMV. I do love using X25-Ms for OS disks on FreeBSD though. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Jan 9 17:26:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7FC6F106566B for ; Sun, 9 Jan 2011 17:26:07 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 4781F8FC0C for ; Sun, 9 Jan 2011 17:26:07 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.13.8+Sun/8.13.8) with ESMTP id p09HQ6eC018498; Sun, 9 Jan 2011 11:26:06 -0600 (CST) Date: Sun, 9 Jan 2011 11:26:06 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Jeremy Chadwick In-Reply-To: <20110109024419.GA25374@icarus.home.lan> Message-ID: References: <20110109024419.GA25374@icarus.home.lan> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Sun, 09 Jan 2011 11:26:06 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: Another Phoronix article X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jan 2011 17:26:07 -0000 On Sat, 8 Jan 2011, Jeremy Chadwick wrote: > > Which in turn makes me wonder why no one from the FreeBSD community > (developer, contributor, or user) is offering to help the Phoronix folks > "become more familiar" with FreeBSD and/or help track down or explain > whatever "issues" they encounter, rather than just constantly ragging > on them. Phoronix is an advertising-funded site. It does whatever will bring more eyeballs to the site. Sometimes doing the wrong thing or making wrong assumptions brings more eyeballs to the site so there is little incentive for improvement. If you look through various article comments and the forums you will find plenty of advisements (which are generally ignored for the next go-around). I feel free to say this even though Phoronix often uses my own software as part of its benchmarks. :-) Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 01:24:50 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DDC22106566B; Mon, 10 Jan 2011 01:24:50 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B504F8FC14; Mon, 10 Jan 2011 01:24:50 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0A1Ooun044474; Mon, 10 Jan 2011 01:24:50 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0A1Ooe6044470; Mon, 10 Jan 2011 01:24:50 GMT (envelope-from linimon) Date: Mon, 10 Jan 2011 01:24:50 GMT Message-Id: <201101100124.p0A1Ooe6044470@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/153680: [xfs] 8.1 failing to mount XFS partitions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 01:24:51 -0000 Old Synopsis: 8.1 failing to mount XFS partitions New Synopsis: [xfs] 8.1 failing to mount XFS partitions Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Mon Jan 10 01:24:36 UTC 2011 Responsible-Changed-Why: reclassify and assign. http://www.freebsd.org/cgi/query-pr.cgi?pr=153680 From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 09:09:30 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E25A106566B; Mon, 10 Jan 2011 09:09:30 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mx2.wheel.pl (grom.wheel.pl [91.121.70.66]) by mx1.freebsd.org (Postfix) with ESMTP id CF6CB8FC13; Mon, 10 Jan 2011 09:09:29 +0000 (UTC) Received: from mx1.whl (wheel-vpn [9.1.1.6]) by mx2.wheel.pl (Postfix) with ESMTP id BFFE32B2; Mon, 10 Jan 2011 10:02:11 +0100 (CET) Received: from localhost (pdawidek.whl [10.0.1.1]) by mx1.whl (Postfix) with ESMTPSA id 043E22EC; Mon, 10 Jan 2011 10:00:52 +0100 (CET) Date: Mon, 10 Jan 2011 10:02:05 +0100 From: Pawel Jakub Dawidek To: Attila Nagy Message-ID: <20110110090205.GB1744@garage.freebsd.pl> References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> <4D29A0C7.8050002@fsn.hu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IrhDeMKUP4DT/M7F" Content-Disposition: inline In-Reply-To: <4D29A0C7.8050002@fsn.hu> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Martin Matuska Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 09:09:30 -0000 --IrhDeMKUP4DT/M7F Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 09, 2011 at 12:49:27PM +0100, Attila Nagy wrote: > No, it's not related. One of the disks in the RAIDZ2 pool went bad: > (da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 > (da4:arcmsr0:0:4:0): CAM status: SCSI Status Error > (da4:arcmsr0:0:4:0): SCSI status: Check Condition > (da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read= =20 > error) > and it seems it froze the whole zpool. Removing the disk by hand solved= =20 > the problem. > I've seen this previously on other machines with ciss. > I wonder why ZFS didn't throw it out of the pool. Such hangs happen when I/O never returns. ZFS doesn't timeout I/O requests on its own, this is driver's responsibility. It is still strange that the driver didn't pass I/O error up to ZFS or it might as well be ZFS bug, but I don't think so. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --IrhDeMKUP4DT/M7F Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0qywwACgkQForvXbEpPzSojwCffanIRz1LN1MTmc/Jf3qur4cG M70An36Qn84voWMQ8pBF3Cc4KPDU13gw =o2Bn -----END PGP SIGNATURE----- --IrhDeMKUP4DT/M7F-- From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 09:14:30 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8A14106564A; Mon, 10 Jan 2011 09:14:30 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mx2.wheel.pl (grom.wheel.pl [91.121.70.66]) by mx1.freebsd.org (Postfix) with ESMTP id 2E02A8FC13; Mon, 10 Jan 2011 09:14:29 +0000 (UTC) Received: from mx1.whl (wheel-vpn [9.1.1.6]) by mx2.wheel.pl (Postfix) with ESMTP id 5BB812AF; Mon, 10 Jan 2011 09:58:02 +0100 (CET) Received: from localhost (pdawidek.whl [10.0.1.1]) by mx1.whl (Postfix) with ESMTPSA id 7B91B2EA; Mon, 10 Jan 2011 09:56:42 +0100 (CET) Date: Mon, 10 Jan 2011 09:57:56 +0100 From: Pawel Jakub Dawidek To: Attila Nagy Message-ID: <20110110085756.GA1744@garage.freebsd.pl> References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D29A198.4070107@fsn.hu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SLDf9lqlvOQaIe6s" Content-Disposition: inline In-Reply-To: <4D29A198.4070107@fsn.hu> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 09:14:30 -0000 --SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 09, 2011 at 12:52:56PM +0100, Attila Nagy wrote: [...] > I've finally found the time to read the v28 patch and figured out the=20 > problem: vfs.zfs.l2arc_noprefetch was changed to 1, so it doesn't use=20 > the prefetched data on the L2ARC devices. > This is a major hit in my case. Enabling this again restored the=20 > previous hit rates and lowered the load on the hard disks significantly. Well, not storing prefetched data on L2ARC vdevs is the default is Solaris. For some reason it was changed by kmacy@ in r205231. Not sure why and we can't ask him now, I'm afraid. I just sent an e-mail to Brendan Gregg from Oracle who originally implemented L2ARC in ZFS why this is turned off by default. Once I get answer we can think about turning it on again. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --SLDf9lqlvOQaIe6s Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0qyhMACgkQForvXbEpPzT7qACeM1nQ4x8dyLpt81dkOLtwBeCn H4wAn3qFwe2SESI5WQ3JMPjreqx5krnb =Knhp -----END PGP SIGNATURE----- --SLDf9lqlvOQaIe6s-- From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 11:07:02 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DCA3106564A for ; Mon, 10 Jan 2011 11:07:02 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7143F8FC2A for ; Mon, 10 Jan 2011 11:07:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0AB722c001755 for ; Mon, 10 Jan 2011 11:07:02 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0AB71xi001753 for freebsd-fs@FreeBSD.org; Mon, 10 Jan 2011 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Jan 2011 11:07:01 GMT Message-Id: <201101101107.p0AB71xi001753@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 11:07:02 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153584 fs [ext2fs] [patch] Performance fix and cleanups for BSD o kern/153552 fs [zfs] zfsboot from 8.2-RC1 freeze at boot time o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152605 fs [ufs] [panic] handle_workitem_freeblocks: inode 18 blo o kern/152488 fs [tmpfs] [patch] mtime of file updated when only inode o kern/152079 fs [msdosfs] [patch] Small cleanups from the other NetBSD o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o kern/151845 fs [smbfs] [patch] smbfs should be upgraded to support Un o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/150207 fs zpool(1): zpool import -d /dev tries to open weird dev o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149022 fs [hang] File system operations hangs with suspfs state o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o bin/148296 fs [zfs] [loader] [patch] Very slow probe in /usr/src/sys o kern/148204 fs [nfs] UDP NFS causes overload o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147790 fs [zfs] zfs set acl(mode|inherit) fails on existing zfs o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/146375 fs [nfs] [patch] Typos in macro variables names in sys/fs s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c o kern/144458 fs [nfs] [patch] nfsd fails as a kld p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [ffs] [snapshot] System crashes when manipulat o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o bin/94635 fs snapinfo(8)/libufs only works for disk-backed filesyst o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 217 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 11:57:05 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0831F1065693; Mon, 10 Jan 2011 11:57:05 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 980B28FC1E; Mon, 10 Jan 2011 11:57:04 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id BBA8B45CA0; Mon, 10 Jan 2011 12:57:01 +0100 (CET) Received: from localhost (pdawidek.whl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id B645E45685; Mon, 10 Jan 2011 12:56:56 +0100 (CET) Date: Mon, 10 Jan 2011 12:56:51 +0100 From: Pawel Jakub Dawidek To: Krzysztof Dajka Message-ID: <20110110115651.GH1744@garage.freebsd.pl> References: <4D0A09AF.3040005@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="U3BNvdZEnlJXqmh+" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 11:57:05 -0000 --U3BNvdZEnlJXqmh+ Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 18, 2010 at 10:00:11AM +0100, Krzysztof Dajka wrote: > Hi, > I applied patch against evening 2010-12-16 STABLE. I did what Martin aske= d: >=20 > On Thu, Dec 16, 2010 at 1:44 PM, Martin Matuska wrote: > > =A0 =A0# cd /usr/src > > =A0 =A0# fetch > > http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.= patch.xz > > =A0 =A0# xz -d stable-8-zfsv28-20101215.patch.xz > > =A0 =A0# patch -E -p0 < stable-8-zfsv28-20101215.patch > > =A0 =A0# rm sys/cddl/compat/opensolaris/sys/sysmacros.h > > > Patch applied cleanly. >=20 > #make buildworld > #make buildkernel > #make installkernel > Reboot into single user mode. > #mergemaster -p > #make installworld > #mergemaster > Reboot. >=20 >=20 > Rebooting with old world and new kernel went fine. But after reboot > with new world I got: > ZFS: zfs_alloc()/zfs_free() mismatch > Just before loading kernel modules, after that my system hangs. Could you tell me more about you pool configuration? 'zpool status' output might be helpful. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --U3BNvdZEnlJXqmh+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0q9AMACgkQForvXbEpPzTyLgCfRI+ljotibqDlGghSUXzlqggY 2r0AoIzlrlGkJP4Toc41fGOnMJ97uYby =5RaX -----END PGP SIGNATURE----- --U3BNvdZEnlJXqmh+-- From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 17:23:48 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E00DC106567A; Mon, 10 Jan 2011 17:23:48 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id CD4E38FC18; Mon, 10 Jan 2011 17:23:47 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 4624270614E; Mon, 10 Jan 2011 18:23:45 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 15.8155] X-CRM114-CacheID: sfid-20110110_18234_6D48A357 X-CRM114-Status: Good ( pR: 15.8155 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D2B40A0.20008@fsn.hu> Date: Mon, 10 Jan 2011 18:23:44 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4D0A09AF.3040005@FreeBSD.org> <4D297943.1040507@fsn.hu> <4D29A0C7.8050002@fsn.hu> <20110110090205.GB1744@garage.freebsd.pl> In-Reply-To: <20110110090205.GB1744@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Martin Matuska Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 17:23:49 -0000 On 01/10/2011 10:02 AM, Pawel Jakub Dawidek wrote: > On Sun, Jan 09, 2011 at 12:49:27PM +0100, Attila Nagy wrote: >> No, it's not related. One of the disks in the RAIDZ2 pool went bad: >> (da4:arcmsr0:0:4:0): READ(6). CDB: 8 0 2 10 10 0 >> (da4:arcmsr0:0:4:0): CAM status: SCSI Status Error >> (da4:arcmsr0:0:4:0): SCSI status: Check Condition >> (da4:arcmsr0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read >> error) >> and it seems it froze the whole zpool. Removing the disk by hand solved >> the problem. >> I've seen this previously on other machines with ciss. >> I wonder why ZFS didn't throw it out of the pool. > Such hangs happen when I/O never returns. ZFS doesn't timeout I/O > requests on its own, this is driver's responsibility. It is still > strange that the driver didn't pass I/O error up to ZFS or it might as > well be ZFS bug, but I don't think so. > Indeed, it may to be a controller/driver bug. The newly released (last december) firmware says something about a similar problem. I've upgraded, we'll see whether it will help next time a drive goes awry. I've only seen these errors in dmesg, not in zpool status, there everything was clear (all zeroes). BTW, I've swapped those bad drives (da4, which reported the above errors, and da16, which didn't reported anything to the OS, it was just plain bad according to the controller firmware -and after its deletion, I could offline da4, so it seems it's the real cause, see my previous e-mail), and zpool replaced first da4, but after some seconds of thinking all IO on all disks deceased. After waiting some minutes, it was still the same, so I've rebooted. Then I noticed that a scrub is going on, so I stopped it. Then the zpool replace da4 went fine, it started to resilver the disk. But another zpool replace (for da16) causes the same error: some seconds of IO, then nothing and it stuck in that. Has anybody tried replacing two drives simultaneously with the zfs v28 patch? (this is a stripe of two raidz2s and da4 and da16 are in different raidz2) From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 17:30:42 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71AC21065672; Mon, 10 Jan 2011 17:30:42 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id BCB428FC12; Mon, 10 Jan 2011 17:30:41 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 2717C7061F7; Mon, 10 Jan 2011 18:30:40 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 14.1618] X-CRM114-CacheID: sfid-20110110_18303_780BD314 X-CRM114-Status: Good ( pR: 14.1618 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D2B423F.2000403@fsn.hu> Date: Mon, 10 Jan 2011 18:30:39 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D29A198.4070107@fsn.hu> <20110110085756.GA1744@garage.freebsd.pl> In-Reply-To: <20110110085756.GA1744@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 17:30:42 -0000 On 01/10/2011 09:57 AM, Pawel Jakub Dawidek wrote: > On Sun, Jan 09, 2011 at 12:52:56PM +0100, Attila Nagy wrote: > [...] >> I've finally found the time to read the v28 patch and figured out the >> problem: vfs.zfs.l2arc_noprefetch was changed to 1, so it doesn't use >> the prefetched data on the L2ARC devices. >> This is a major hit in my case. Enabling this again restored the >> previous hit rates and lowered the load on the hard disks significantly. > Well, not storing prefetched data on L2ARC vdevs is the default is > Solaris. For some reason it was changed by kmacy@ in r205231. Not sure > why and we can't ask him now, I'm afraid. I just sent an e-mail to What happened to him? > Brendan Gregg from Oracle who originally implemented L2ARC in ZFS why > this is turned off by default. Once I get answer we can think about > turning it on again. > I think it makes some sense as a stupid form of preferring random IO in the L2ARC instead of sequential. But if I rely on auto tuning and let prefetch enabled, even a busy mailserver will prefetch a lot of blocks and I think that's a fine example of random IO (also, it makes the system unusable, but that's another story). Having this choice is good, and in this case enabling this makes sense for me. I don't know any reasons about why you wouldn't use all of your L2ARC space (apart from sparing the quickly wearing out flash space and move disk heads instead), but I'm sure Brendan made this choice with a good reason. If you get an answer, please tell us. :) Thanks, From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 18:18:34 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 319FA106566C for ; Mon, 10 Jan 2011 18:18:34 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id EB83E8FC12 for ; Mon, 10 Jan 2011 18:18:33 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.13.8+Sun/8.13.8) with ESMTP id p0AIIWdd010674; Mon, 10 Jan 2011 12:18:32 -0600 (CST) Date: Mon, 10 Jan 2011 12:18:32 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Attila Nagy In-Reply-To: <4D2B423F.2000403@fsn.hu> Message-ID: References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D29A198.4070107@fsn.hu> <20110110085756.GA1744@garage.freebsd.pl> <4D2B423F.2000403@fsn.hu> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 10 Jan 2011 12:18:32 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 18:18:34 -0000 On Mon, 10 Jan 2011, Attila Nagy wrote: > > Having this choice is good, and in this case enabling this makes sense for > me. I don't know any reasons about why you wouldn't use all of your L2ARC > space (apart from sparing the quickly wearing out flash space and move disk > heads instead), but I'm sure Brendan made this choice with a good reason. > If you get an answer, please tell us. :) Consider that your L2ARC device might be bandwidth limited to 100-240MB/second while your main storage is capable of 1000MB or 2000MB second sequential data transfer. This a good reason to not simply put all data (which does not fit in normal ARC) into L2ARC. ARC is supposed to be adaptive ... Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 20:44:37 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7AFB51065675 for ; Mon, 10 Jan 2011 20:44:37 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 334C38FC18 for ; Mon, 10 Jan 2011 20:44:36 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 97D33706E08; Mon, 10 Jan 2011 21:44:35 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 12.3386] X-CRM114-CacheID: sfid-20110110_21443_94D71404 X-CRM114-Status: Good ( pR: 12.3386 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D2B6FB2.2000506@fsn.hu> Date: Mon, 10 Jan 2011 21:44:34 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Bob Friesenhahn References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D29A198.4070107@fsn.hu> <20110110085756.GA1744@garage.freebsd.pl> <4D2B423F.2000403@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 20:44:37 -0000 On 01/10/2011 07:18 PM, Bob Friesenhahn wrote: > On Mon, 10 Jan 2011, Attila Nagy wrote: >> >> Having this choice is good, and in this case enabling this makes >> sense for me. I don't know any reasons about why you wouldn't use all >> of your L2ARC space (apart from sparing the quickly wearing out flash >> space and move disk heads instead), but I'm sure Brendan made this >> choice with a good reason. >> If you get an answer, please tell us. :) > > Consider that your L2ARC device might be bandwidth limited to > 100-240MB/second while your main storage is capable of 1000MB or > 2000MB second sequential data transfer. This a good reason to not > simply put all data (which does not fit in normal ARC) into L2ARC. ARC > is supposed to be adaptive ... > > Bob Well, yes, that's a valid point for having such a switch, so the user can decide which is better for him. For me, for this use case, the two SSDs can serve more bandwidth than the 24 SATA disks, so it's definitely worth it. Those disks can do 60-70 MBps sequentially, and all file transfers are sequential here. There are just many of them, so this becomes semi-random IO. Caching in the L2ARC helps a lot here. From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 21:42:27 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD750106564A; Mon, 10 Jan 2011 21:42:27 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id A8DA58FC12; Mon, 10 Jan 2011 21:42:26 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 0ECED7071B6; Mon, 10 Jan 2011 22:42:25 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 15.7484] X-CRM114-CacheID: sfid-20110110_22422_0F9498C7 X-CRM114-Status: Good ( pR: 15.7484 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D2B7D40.4000604@fsn.hu> Date: Mon, 10 Jan 2011 22:42:24 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Martin Matuska References: <4D0A09AF.3040005@FreeBSD.org> In-Reply-To: <4D0A09AF.3040005@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 21:42:27 -0000 On 12/16/2010 01:44 PM, Martin Matuska wrote: > Hi everyone, > > following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am > providing a ZFSv28 testing patch for 8-STABLE. > > Link to the patch: > > http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz > > Link to mfsBSD ISO files for testing (i386 and amd64): > http://mfsbsd.vx.sk/iso/zfs-v28/8.2-beta-zfsv28-amd64.iso > http://mfsbsd.vx.sk/iso/zfs-v28/8.2-beta-zfsv28-i386.iso > > The root password for the ISO files: "mfsroot" > The ISO files work on real systems and in virtualbox. > They conatin a full install of FreeBSD 8.2-PRERELEASE with ZFS v28, > simply use the provided "zfsinstall" script. > > The patch is against FreeBSD 8-STABLE as of 2010-12-15. > > When applying the patch be sure to use correct options for patch(1) > and make sure the file sys/cddl/compat/opensolaris/sys/sysmacros.h gets > deleted: > > # cd /usr/src > # fetch > http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz > # xz -d stable-8-zfsv28-20101215.patch.xz > # patch -E -p0< stable-8-zfsv28-20101215.patch > # rm sys/cddl/compat/opensolaris/sys/sysmacros.h I've just got a panic: http://people.fsn.hu/~bra/freebsd/20110101-zfsv28-fbsd/IMAGE_006.jpg The panic line for google: panic: solaris assert: task->ost_magic == TASKQ_MAGIC, file: /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c, line: 150 I hope this is enough for debugging, if it's not yet otherwise known. If not, I will try to catch it againt and make a dump. Thanks, From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 22:35:52 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 004AB106564A for ; Mon, 10 Jan 2011 22:35:51 +0000 (UTC) (envelope-from stef@memberwebs.com) Received: from thewalter.net (thewalter.net [94.75.203.97]) by mx1.freebsd.org (Postfix) with ESMTP id 93C1D8FC08 for ; Mon, 10 Jan 2011 22:35:51 +0000 (UTC) Received: from [172.17.182.217] (adsl-75-12-188-41.dsl.pltn13.sbcglobal.net [75.12.188.41]) by thewalter.net (Postfix) with ESMTPA id 2AC4D36122 for ; Mon, 10 Jan 2011 22:35:49 +0000 (UTC) Message-ID: <4D2B89C4.8080300@memberwebs.com> Date: Mon, 10 Jan 2011 14:35:48 -0800 From: Stef Walter User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4D2B8616.4000503@memberwebs.com> In-Reply-To: <4D2B8616.4000503@memberwebs.com> Content-Type: multipart/mixed; boundary="------------080206050804010103080703" Subject: Re: 'zfs list' does abort in make_dataset_handle X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 22:35:52 -0000 This is a multi-part message in MIME format. --------------080206050804010103080703 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 01/10/2011 02:20 PM, Stef Walter wrote: > After a failed zfs receive, zfs list now aborts in make_dataset_handle() > in libzfs.so.2. The following patch suppresses the problem, and may give a clear indication of how the problem manifests. Cheers, Stef --------------080206050804010103080703 Content-Type: text/x-patch; name="zfs-abort-bad-head-type.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="zfs-abort-bad-head-type.patch" --- cddl/contrib/opensolaris/lib/libzfs/common/libzfs_dataset.c.orig 2011-01-10 22:29:45.000000000 +0000 +++ cddl/contrib/opensolaris/lib/libzfs/common/libzfs_dataset.c 2011-01-10 22:32:11.000000000 +0000 @@ -454,6 +454,10 @@ else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZFS) zhp->zfs_head_type = ZFS_TYPE_FILESYSTEM; - else - abort(); + else { + fprintf (stderr, "WARNING: bad head type: %s\n", zhp->zfs_name); + free(zhp); + errno = ENOENT; + return (NULL); + } if (zhp->zfs_dmustats.dds_is_snapshot) --------------080206050804010103080703-- From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 22:37:01 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28B08106564A for ; Mon, 10 Jan 2011 22:37:01 +0000 (UTC) (envelope-from stef@memberwebs.com) Received: from thewalter.net (thewalter.net [94.75.203.97]) by mx1.freebsd.org (Postfix) with ESMTP id E7F7D8FC12 for ; Mon, 10 Jan 2011 22:37:00 +0000 (UTC) Received: from [172.17.182.217] (adsl-75-12-188-41.dsl.pltn13.sbcglobal.net [75.12.188.41]) by thewalter.net (Postfix) with ESMTPA id 62C7F360EC for ; Mon, 10 Jan 2011 22:20:08 +0000 (UTC) Message-ID: <4D2B8616.4000503@memberwebs.com> Date: Mon, 10 Jan 2011 14:20:06 -0800 From: Stef Walter User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: 'zfs list' does abort in make_dataset_handle X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 22:37:01 -0000 After a failed zfs receive, zfs list now aborts in make_dataset_handle() in libzfs.so.2. # zfs list Abort trap: 6 (core dumped) The backtrace is: #0 0x0000000800fe63cc in kill () from /lib/libc.so.7 #1 0x0000000800fe51cb in abort () from /lib/libc.so.7 #2 0x000000080067400f in make_dataset_handle () from /lib/libzfs.so.2 #3 0x0000000800674346 in zfs_iter_filesystems () from /lib/libzfs.so.2 #4 0x000000000040ad51 in ?? () #5 0x0000000800674354 in zfs_iter_filesystems () from /lib/libzfs.so.2 #6 0x000000000040ad51 in ?? () #7 0x000000080066642c in zfs_iter_root () from /lib/libzfs.so.2 Running on: FreeBSD m8.xxxx.xxx 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Sat Jan 8 19:26:12 UTC 2011 sam@:/usr/obj/usr/src/sys/RACK8 amd64 'zfs scrub' did not fix the error. 'zfs create' and other zfs commands still work, as do access to the various zfs file systems on this system. After poking around a bit, I find that in make_dataset_handle() zhp->zfs_dmustats.dds_type is equal to 0 (and this is the cause of the abort). Additionally zhp->zfs_dmustats.dds_inconsistent is not TRUE. Any ideas? Thanks in advance! Cheers, Stef From owner-freebsd-fs@FreeBSD.ORG Mon Jan 10 23:15:49 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1994D1065679 for ; Mon, 10 Jan 2011 23:15:49 +0000 (UTC) (envelope-from ivan@brawley.id.au) Received: from hash.brawley.id.au (unknown [IPv6:2001:44b8:60:f200::1:33]) by mx1.freebsd.org (Postfix) with ESMTP id 79FAF8FC1D for ; Mon, 10 Jan 2011 23:15:48 +0000 (UTC) Received: from [IPv6:2001:44b8:60:f200:226:18ff:fe66:2614] (unknown [IPv6:2001:44b8:60:f200:226:18ff:fe66:2614]) by hash.brawley.id.au (Postfix) with ESMTP id BA6A461C53 for ; Tue, 11 Jan 2011 09:45:45 +1030 (CST) Message-ID: <4D2B9321.9070306@brawley.id.au> Date: Tue, 11 Jan 2011 09:45:45 +1030 From: Ivan Brawley User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.11) Gecko/20100806 Lightning/1.0b2pre Thunderbird/3.0.6 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.96.5 at hash X-Virus-Status: Clean Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: 20110110115651.GH1744@garage.freebsd.pl List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jan 2011 23:15:49 -0000 On 10/Jan/2011, Pawel Jakub Dawidek wrote: > On Sat, Dec 18, 2010 at 10:00:11AM +0100, Krzysztof Dajka wrote: > > Patch applied cleanly. > >=20 > > #make buildworld > > #make buildkernel > > #make installkernel > > Reboot into single user mode. > > #mergemaster -p > > #make installworld > > #mergemaster > > Reboot. > > > > > > Rebooting with old world and new kernel went fine. But after reboot > > with new world I got: > > ZFS: zfs_alloc()/zfs_free() mismatch > > Just before loading kernel modules, after that my system hangs. > > Could you tell me more about you pool configuration? > 'zpool status' output might be helpful. I too got hit by the same thing with my workstation. My configuration is 3 x 1.5TB SATA drives in a RAIDZ set and I have a SSD split into two to use as L2ARC and ZIL (yeah, I know, one SSD only, all the fun and games should it die... That's what backups are for :-). # zpool status bang_raidz pool: bang_raidz state: ONLINE scan: scrub repaired 0 in 7h55m with 0 errors on Sat Jan 1 16:16:13 2011 config: NAME STATE READ WRITE CKSUM bang_raidz ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 ada2p3 ONLINE 0 0 0 ada3p3 ONLINE 0 0 0 logs ada4p2 ONLINE 0 0 0 cache ada4p3 ONLINE 0 0 0 errors: No known data errors I've tried upgrading the pool to v28 without luck. I can import the pool if I boot from alternate media (which is what I'm currently doing). Doing some debugging, the problem seems to be when zfs_free() is called by sys/boot/zfs/zfsimpl.c:zio_read(). I've been trying to replicate the problem with VirtualBox but without luck thus far. Another interesting bit is: # zdb bang_raidz zdb: can't open 'bang_raidz': No such file or directory The system was originally at 8.1R and had it's zpool upgraded to along the way to v15 before this as well (just as another data point). ivan. -- Ivan Brawley From owner-freebsd-fs@FreeBSD.ORG Tue Jan 11 01:45:47 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DB241065673; Tue, 11 Jan 2011 01:45:47 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 345938FC22; Tue, 11 Jan 2011 01:45:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0B1jlQh000900; Tue, 11 Jan 2011 01:45:47 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0B1jldh000896; Tue, 11 Jan 2011 01:45:47 GMT (envelope-from linimon) Date: Tue, 11 Jan 2011 01:45:47 GMT Message-Id: <201101110145.p0B1jldh000896@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/153847: [nfs] [panic] Kernel panic from incorrect m_free in nfs_getattr X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jan 2011 01:45:47 -0000 Old Synopsis: Kernel panic from incorrect m_free in nfs_getattr New Synopsis: [nfs] [panic] Kernel panic from incorrect m_free in nfs_getattr Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Tue Jan 11 01:45:24 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=153847 From owner-freebsd-fs@FreeBSD.ORG Tue Jan 11 21:46:33 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5BC610656A5 for ; Tue, 11 Jan 2011 21:46:33 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 4971A8FC17 for ; Tue, 11 Jan 2011 21:46:33 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 9CD3A45CA6; Tue, 11 Jan 2011 22:46:31 +0100 (CET) Received: from localhost (89-73-192-49.dynamic.chello.pl [89.73.192.49]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 55F3C45C9B; Tue, 11 Jan 2011 22:46:26 +0100 (CET) Date: Tue, 11 Jan 2011 22:46:19 +0100 From: Pawel Jakub Dawidek To: Stef Walter Message-ID: <20110111214619.GF1812@garage.freebsd.pl> References: <4D2B8616.4000503@memberwebs.com> <4D2B89C4.8080300@memberwebs.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3siQDZowHQqNOShm" Content-Disposition: inline In-Reply-To: <4D2B89C4.8080300@memberwebs.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: 'zfs list' does abort in make_dataset_handle X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jan 2011 21:46:33 -0000 --3siQDZowHQqNOShm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 10, 2011 at 02:35:48PM -0800, Stef Walter wrote: > On 01/10/2011 02:20 PM, Stef Walter wrote: > > After a failed zfs receive, zfs list now aborts in make_dataset_handle() > > in libzfs.so.2. >=20 > The following patch suppresses the problem, and may give a clear > indication of how the problem manifests. I'd more threat such situation as if dds_inconsistent would be true and destroy/rollback offending dataset. Could you try something like that? --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --3siQDZowHQqNOShm Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0sz6sACgkQForvXbEpPzRsewCeNBerJ5mH2h1njrZxWXhew0QH 0C0AoKhhYqWHSL2x2UL4EI/C8XraOgLk =vFb/ -----END PGP SIGNATURE----- --3siQDZowHQqNOShm-- From owner-freebsd-fs@FreeBSD.ORG Wed Jan 12 01:52:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0277A106564A; Wed, 12 Jan 2011 01:52:09 +0000 (UTC) (envelope-from stef@memberwebs.com) Received: from thewalter.net (thewalter.net [94.75.203.97]) by mx1.freebsd.org (Postfix) with ESMTP id BB5608FC17; Wed, 12 Jan 2011 01:52:08 +0000 (UTC) Received: from [172.17.182.217] (adsl-75-12-188-41.dsl.pltn13.sbcglobal.net [75.12.188.41]) by thewalter.net (Postfix) with ESMTPA id 358823660A; Wed, 12 Jan 2011 01:52:05 +0000 (UTC) Message-ID: <4D2D0941.40002@memberwebs.com> Date: Tue, 11 Jan 2011 17:52:01 -0800 From: Stef Walter User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4D2B8616.4000503@memberwebs.com> <4D2B89C4.8080300@memberwebs.com> <20110111214619.GF1812@garage.freebsd.pl> In-Reply-To: <20110111214619.GF1812@garage.freebsd.pl> Content-Type: multipart/mixed; boundary="------------060706040903030701070909" Cc: freebsd-fs@freebsd.org Subject: Re: 'zfs list' does abort in make_dataset_handle X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jan 2011 01:52:09 -0000 This is a multi-part message in MIME format. --------------060706040903030701070909 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 01/11/2011 01:46 PM, Pawel Jakub Dawidek wrote: > On Mon, Jan 10, 2011 at 02:35:48PM -0800, Stef Walter wrote: >> On 01/10/2011 02:20 PM, Stef Walter wrote: >>> After a failed zfs receive, zfs list now aborts in make_dataset_handle() >>> in libzfs.so.2. >> >> The following patch suppresses the problem, and may give a clear >> indication of how the problem manifests. > > I'd more threat such situation as if dds_inconsistent would be true and > destroy/rollback offending dataset. Could you try something like that? Sounds like a good plan. Attached is a patch which does that. Cheers, Stef --------------060706040903030701070909 Content-Type: text/x-patch; name="zfs-data-set-bad-head-type-2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="zfs-data-set-bad-head-type-2.patch" --- cddl/contrib/opensolaris/lib/libzfs/common/libzfs_dataset.c.orig 2011-01-10 22:29:45.000000000 +0000 +++ cddl/contrib/opensolaris/lib/libzfs/common/libzfs_dataset.c 2011-01-12 01:49:13.000000000 +0000 @@ -404,4 +404,24 @@ } + /* + * We've managed to open the dataset and gather statistics. Determine + * the high-level type. + */ + if (zhp->zfs_dmustats.dds_type == DMU_OST_ZVOL) + zhp->zfs_head_type = ZFS_TYPE_VOLUME; + else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZFS) + zhp->zfs_head_type = ZFS_TYPE_FILESYSTEM; + else + zhp->zfs_dmustats.dds_inconsistent = 1; + + if (zhp->zfs_dmustats.dds_is_snapshot) + zhp->zfs_type = ZFS_TYPE_SNAPSHOT; + else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZVOL) + zhp->zfs_type = ZFS_TYPE_VOLUME; + else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZFS) + zhp->zfs_type = ZFS_TYPE_FILESYSTEM; + else + zhp->zfs_dmustats.dds_inconsistent = 1; + if (zhp->zfs_dmustats.dds_inconsistent) { zfs_cmd_t zc = { 0 }; @@ -446,24 +466,4 @@ } - /* - * We've managed to open the dataset and gather statistics. Determine - * the high-level type. - */ - if (zhp->zfs_dmustats.dds_type == DMU_OST_ZVOL) - zhp->zfs_head_type = ZFS_TYPE_VOLUME; - else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZFS) - zhp->zfs_head_type = ZFS_TYPE_FILESYSTEM; - else - abort(); - - if (zhp->zfs_dmustats.dds_is_snapshot) - zhp->zfs_type = ZFS_TYPE_SNAPSHOT; - else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZVOL) - zhp->zfs_type = ZFS_TYPE_VOLUME; - else if (zhp->zfs_dmustats.dds_type == DMU_OST_ZFS) - zhp->zfs_type = ZFS_TYPE_FILESYSTEM; - else - abort(); /* we should never see any other types */ - zhp->zfs_hdl->libzfs_log_str = logstr; zhp->zpool_hdl = zpool_handle(zhp); --------------060706040903030701070909-- From owner-freebsd-fs@FreeBSD.ORG Wed Jan 12 07:22:28 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93F14106564A for ; Wed, 12 Jan 2011 07:22:28 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 3EB008FC15 for ; Wed, 12 Jan 2011 07:22:27 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 7986845C89; Wed, 12 Jan 2011 08:22:26 +0100 (CET) Received: from localhost (89-73-192-49.dynamic.chello.pl [89.73.192.49]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 897154569A; Wed, 12 Jan 2011 08:22:21 +0100 (CET) Date: Wed, 12 Jan 2011 08:22:13 +0100 From: Pawel Jakub Dawidek To: Stef Walter Message-ID: <20110112072213.GG1812@garage.freebsd.pl> References: <4D2B8616.4000503@memberwebs.com> <4D2B89C4.8080300@memberwebs.com> <20110111214619.GF1812@garage.freebsd.pl> <4D2D0941.40002@memberwebs.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XvKFcGCOAo53UbWW" Content-Disposition: inline In-Reply-To: <4D2D0941.40002@memberwebs.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: 'zfs list' does abort in make_dataset_handle X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jan 2011 07:22:28 -0000 --XvKFcGCOAo53UbWW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 11, 2011 at 05:52:01PM -0800, Stef Walter wrote: > On 01/11/2011 01:46 PM, Pawel Jakub Dawidek wrote: > > On Mon, Jan 10, 2011 at 02:35:48PM -0800, Stef Walter wrote: > >> On 01/10/2011 02:20 PM, Stef Walter wrote: > >>> After a failed zfs receive, zfs list now aborts in make_dataset_handl= e() > >>> in libzfs.so.2. > >> > >> The following patch suppresses the problem, and may give a clear > >> indication of how the problem manifests. > >=20 > > I'd more threat such situation as if dds_inconsistent would be true and > > destroy/rollback offending dataset. Could you try something like that? >=20 > Sounds like a good plan. Attached is a patch which does that. But does it fix your problem? --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --XvKFcGCOAo53UbWW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEUEARECAAYFAk0tVqQACgkQForvXbEpPzQJVwCY22Vr2Jw5ksyHnNxNFrHBTnkQ PQCg7wOweyivBFU/8Usv0h8ybmkpF58= =ajIW -----END PGP SIGNATURE----- --XvKFcGCOAo53UbWW-- From owner-freebsd-fs@FreeBSD.ORG Wed Jan 12 08:13:55 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A524106564A for ; Wed, 12 Jan 2011 08:13:55 +0000 (UTC) (envelope-from marcus@blazingdot.com) Received: from marklar.blazingdot.com (marklar.blazingdot.com [207.154.84.83]) by mx1.freebsd.org (Postfix) with SMTP id C7B248FC13 for ; Wed, 12 Jan 2011 08:13:54 +0000 (UTC) Received: (qmail 55010 invoked by uid 503); 12 Jan 2011 07:47:13 -0000 Date: Tue, 11 Jan 2011 23:47:13 -0800 From: Marcus Reid To: Attila Nagy Message-ID: <20110112074713.GB52770@blazingdot.com> References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D29A198.4070107@fsn.hu> <20110110085756.GA1744@garage.freebsd.pl> <4D2B423F.2000403@fsn.hu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D2B423F.2000403@fsn.hu> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.6i Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, Pawel Jakub Dawidek Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jan 2011 08:13:55 -0000 On Mon, Jan 10, 2011 at 06:30:39PM +0100, Attila Nagy wrote: > >why and we can't ask him now, I'm afraid. I just sent an e-mail to > > What happened to him? Oops, I was thinking of something else. http://valleywag.gawker.com/383763/freebsd-developer-kip-macy-arrested-for-tormenting-tenants Marcus From owner-freebsd-fs@FreeBSD.ORG Wed Jan 12 10:23:25 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 088D0106564A for ; Wed, 12 Jan 2011 10:23:25 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: from mail.bytecamp.net (mail.bytecamp.net [212.204.60.9]) by mx1.freebsd.org (Postfix) with ESMTP id 365B38FC1E for ; Wed, 12 Jan 2011 10:23:23 +0000 (UTC) Received: (qmail 78468 invoked by uid 89); 12 Jan 2011 10:56:43 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with CAMELLIA256-SHA encrypted SMTP; 12 Jan 2011 10:56:43 +0100 Message-ID: <4D2D7ADB.3090902@bytecamp.net> Date: Wed, 12 Jan 2011 10:56:43 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: nfs and dropped udp datagrams X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jan 2011 10:23:25 -0000 I wonder how big kern.ipc.maxsockbuf shall be tuned on a busy NFS-Server. In the last time, I often see clients loosing connection to their mounted directories. Furthermore, I see increasing counts for upd datagrams "dropped due to full socket buffers" in netstat -s -p udp. The mbuf situation does not seem to be the reason for the lost connections, vmstat -z shows 0 failures in the mbuf section. Are there any other tunables which could prevent loss of connection to the server? What is a reasonable value for maxsockbuf? With kind regards, Robert Schulze From owner-freebsd-fs@FreeBSD.ORG Wed Jan 12 16:18:41 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C47471065670; Wed, 12 Jan 2011 16:18:41 +0000 (UTC) (envelope-from stef@memberwebs.com) Received: from thewalter.net (thewalter.net [94.75.203.97]) by mx1.freebsd.org (Postfix) with ESMTP id 8BD3C8FC1B; Wed, 12 Jan 2011 16:18:41 +0000 (UTC) Received: from [172.17.182.217] (adsl-75-12-188-41.dsl.pltn13.sbcglobal.net [75.12.188.41]) by thewalter.net (Postfix) with ESMTPA id 7D63E36817; Wed, 12 Jan 2011 16:18:39 +0000 (UTC) Message-ID: <4D2DD45C.7050406@memberwebs.com> Date: Wed, 12 Jan 2011 08:18:36 -0800 From: Stef Walter User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4D2B8616.4000503@memberwebs.com> <4D2B89C4.8080300@memberwebs.com> <20110111214619.GF1812@garage.freebsd.pl> <4D2D0941.40002@memberwebs.com> <20110112072213.GG1812@garage.freebsd.pl> In-Reply-To: <20110112072213.GG1812@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: 'zfs list' does abort in make_dataset_handle X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jan 2011 16:18:41 -0000 On 01/11/2011 11:22 PM, Pawel Jakub Dawidek wrote: > On Tue, Jan 11, 2011 at 05:52:01PM -0800, Stef Walter wrote: >> On 01/11/2011 01:46 PM, Pawel Jakub Dawidek wrote: >>> On Mon, Jan 10, 2011 at 02:35:48PM -0800, Stef Walter wrote: >>>> On 01/10/2011 02:20 PM, Stef Walter wrote: >>>>> After a failed zfs receive, zfs list now aborts in make_dataset_handle() >>>>> in libzfs.so.2. >>>> >>>> The following patch suppresses the problem, and may give a clear >>>> indication of how the problem manifests. >>> >>> I'd more threat such situation as if dds_inconsistent would be true and >>> destroy/rollback offending dataset. Could you try something like that? >> >> Sounds like a good plan. Attached is a patch which does that. > > But does it fix your problem? Yup, it does. Stef From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 00:58:32 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4F57106566B for ; Thu, 13 Jan 2011 00:58:32 +0000 (UTC) (envelope-from freebsd-list@nikazam.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id AD6EA8FC08 for ; Thu, 13 Jan 2011 00:58:32 +0000 (UTC) Received: by qwj9 with SMTP id 9so1227867qwj.13 for ; Wed, 12 Jan 2011 16:58:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.224.28.140 with SMTP id m12mr1517226qac.107.1294879011576; Wed, 12 Jan 2011 16:36:51 -0800 (PST) Received: by 10.224.220.76 with HTTP; Wed, 12 Jan 2011 16:36:51 -0800 (PST) Date: Thu, 13 Jan 2011 00:36:51 +0000 Message-ID: From: Nik Azam To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 00:58:33 -0000 Hi all, For the past few weeks, I noticed that the amount of memory reported in top (sum of active, inact, wired, cache buf and free) keeps decreasing as the uptime increases. I can't pinpoint to when I first noticed this, as I have updated the system a few times just in case this has been fixed. The box has 8GB of RAM, running 8.2-PRERELEASE r216693. The ZFS pool consists of two RAIDZ1 with 5 disks each and 50GB L2 ARC. Root filesystem is on UFS on separate disk. I have samba, netatalk, istgt, mysql and apache running on the box. After booting up, top reports I have 8GB of memory as expected. After moving files around in the ZFS pool, the ARC grows to 4GB (the maximum size I set). Now the box has been up for 10 days, and the sum of memory reported in top is about 4GB (3GB wired). ARC size is about 2GB. I tried stopping all the services running, but this didn't help. When unmounting the ZFS pool (via /etc/rc.d/zfs stop), top reports that I have 8GB again. Throughout the 10 days period, there is no user activity at all to the UFS filesystem. Is there anything sinister happening here? Or is it just top is playing tricks on me? Thanks, Nik From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 03:33:27 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C56D11065672 for ; Thu, 13 Jan 2011 03:33:27 +0000 (UTC) (envelope-from cforgeron@acsi.ca) Received: from mta03.eastlink.ca (mta03.eastlink.ca [24.224.136.9]) by mx1.freebsd.org (Postfix) with ESMTP id 8F1148FC16 for ; Thu, 13 Jan 2011 03:33:27 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from ip02.eastlink.ca ([unknown] [24.222.39.20]) by mta03.eastlink.ca (Sun Java(tm) System Messaging Server 7.3-11.01 64bit (built Sep 1 2009)) with ESMTP id <0LEX00D1IXTNX4H0@mta03.eastlink.ca>; Wed, 12 Jan 2011 23:03:23 -0400 (AST) X-CMAE-Score: 0 X-CMAE-Analysis: v=1.1 cv=8reSTVRqS4Rq5Xx4Jai9N41eZpHz3D5gSX5rA0od4mg= c=1 sm=1 a=kj9zAlcOel0A:10 a=vNmoZVmIAAAA:8 a=sQ0fBdVQjJLqKWLw8ZoA:9 a=-9Ce0oEXxbJjTgpgn0YA:7 a=gdUC-0uo-COgpyxG29wr4OLNjRgA:4 a=CjuIK1q_8ugA:10 a=m_Qjd_tMtKcA:10 a=gA6IeH5FQcgA:10 a=NWVoK91CQyQA:10 a=Y4g+zi6NJtbRuBVJrbSZ6Q==:117 Received: from blk-222-10-85.eastlink.ca (HELO server7.acsi.ca) ([24.222.10.85]) by ip02.eastlink.ca with ESMTP; Wed, 12 Jan 2011 23:03:23 -0400 Received: from server7.acsi.ca ([192.168.9.7]) by server7.acsi.ca ([192.168.9.7]) with mapi; Wed, 12 Jan 2011 23:03:20 -0400 From: Chris Forgeron To: "freebsd-fs@freebsd.org" , "freebsd-current@freebsd.org" Date: Wed, 12 Jan 2011 23:03:19 -0400 Thread-topic: My ZFS v28 Testing Experience Thread-index: AcueQ3BAVz5TAZ89SmeN6hBSyIf4KQUeGZLA Message-id: References: <20101213214556.GC2038@garage.freebsd.pl> In-reply-to: Accept-Language: en-US Content-language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Cc: Subject: My ZFS v28 Testing Experience X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 03:33:27 -0000 I've been testing out the v28 patch code for a month now, and I've yet to report any real issues other than what is mentioned below. I'll detail some of the things I've tested, hopefully the stability of v28 in FreeBSD will convince others to give it a try so the final release of v28 will be as solid as possible. I've been using FreeBSD 9.0-CURRENT as of Dec 12th, and 8.2PRE as of Dec 16th What's worked well: - I've made and destroyed small raidz's (3-5 disks), large 26 disk raid-10's, and a large 20 disk raid-50. - I've upgraded from v15, zfs 4, no issues on the different arrays noted above - I've confirmed that a v15 or v28 pool will import into Solaris 11 Express, and vice versa, with the exception about dual log or cache devices noted below. - I've run many TB of data through the ZFS storage via benchmarks from my VM's connected via NFS, to simple copies inside the same pool, or copies from one pool to another. - I've tested pretty much every compression level, and changing them as I tweak my setup and try to find the best blend. - I've added and subtracted many a log and cache device, some in failed states from hot-removals, and the pools always stayed intact. Issues: - Import of pools with multiple cache or log devices. (May be a very minor point) A v28 pool created in Solaris 11 Express with 2 or more log devices, or 2 or more cache devices won't import in FreeBSD 9. This also applies to a pool that is created in FreeBSD, is imported in Solaris to have the 2 log devices added there, then exported and attempted to be imported back in FreeBSD. No errors, zpool import just hangs forever. If I reboot into Solaris, import the pool, remove the dual devices, then reboot into FreeBSD, I can then import the pool without issue. A single cache, or log device will import just fine. Unfortunately I deleted my witness-enabled FreeBSD-9 drive, so I can't easily fire it back up to give more debug info. I'm hoping some kind soul will attempt this type of transaction and report more detail to the list. Note - I just decided to try adding 2 cache devices to a raidz pool in FreeBSD, export, and then importing, all without rebooting. That seems to work. BUT - As soon as you try to reboot FreeBSD with this pool staying active, it hangs on boot. Booting into Solaris, removing the 2 cache devices, then booting back into FreeBSD then works. Something is kept in memory between exporting then importing that allows this to work. - Speed. (More of an issue, but what do we do?) Wow, it's much slower than Solaris 11 Express for transactions. I do understand that Solaris will have a slight advantage over any port of ZFS. All of my speed tests are made with a kernel without debug, and yes, these are -CURRENT and -PRE releases, but the speed difference is very large. At first, I thought it may be more of an issue with the ix0/Intel X520DA2 10Gbe drivers that I'm using, since the bulk of my tests are over NFS (I'm going to use this as a SAN via NFS, so I test in that environment). But - I did a raw cp command from one pool to another of several TB. I executed the same command under FreeBSD as I did under Solaris 11 Express. When executed in FreeBSD, the copy took 36 hours. With a fresh destination pool of the same settings/compression/etc under Solaris, the copy took 7.5 hours. Here's a quick breakdown of the difference in speed I'm seeing between Solaris 11 Express and FreeBSD. The test is Performance Test 6.1 on a Windows 2003 server, connected via NFS to the FreeBSD or Solaris box. More details are here: http://christopher-technicalmusings.blogspot.com/2011/01/solaris-11-express-faster-speed-for-san.html Solaris 11 Express svn_151a 903 MBs - Fileserver 466 MBs - Webserver 53 MBs - Workstation 201 MBs - Database FreeBSD-9.0 Current @ Dec 12th 2010 w/v28 Patch, all Debug off 95 MBs - Fileserver 60 MBs - Webserver 30 MBs - Workstation 32 MBs - Database Massive difference as you can see. Same machine, different boot drives. That's a real 903 MBs on the Solaris side as well - No cache devices or ZIL in place, just a basic raidz 5 disk pool. I've tried many a tweak to get these speeds up higher. The old v15 could hit mid 400's for the Fileserver test with zil_disable on, but that's no longer an option for v28 pools. I should compile my various test results into a summary and make a separate blog entry for those who care, as I also fiddled with vfs.nfsrv.async with little luck. I took great care to make sure the ZFS details were the same across the tests. 9 is faster than 8.2 for speed by a small amount. Between v28 pools and v15 pools there is speed degradation on both 8.2 and 9, but nothing as big as the difference between Solaris and FreeBSD. I haven't benchmarked OpenSolaris or any type of Solaris older than 11, so I'm not sure if this is a recent speed boost from the Solaris camp, or if it's always been there. As always, I'm delighted about the work that PJD and others have poured into ZFS on FreeBSD. I'm not knocking this implementation, just pointing out some points that may not be as well known. I am continuing to play with Solaris 11 Express, but I'm keeping my pools at v28 so I can boot into either OS to test. From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 10:37:11 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D801106566B for ; Thu, 13 Jan 2011 10:37:11 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id EBF6B8FC0C for ; Thu, 13 Jan 2011 10:37:10 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 8E5B045C8A; Thu, 13 Jan 2011 11:37:09 +0100 (CET) Received: from localhost (pdawidek.whl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 9DEE945684; Thu, 13 Jan 2011 11:37:04 +0100 (CET) Date: Thu, 13 Jan 2011 11:36:57 +0100 From: Pawel Jakub Dawidek To: Nik Azam Message-ID: <20110113103657.GC1723@garage.freebsd.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WfZ7S8PLGjBY9Voh" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 10:37:11 -0000 --WfZ7S8PLGjBY9Voh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 13, 2011 at 12:36:51AM +0000, Nik Azam wrote: > Hi all, >=20 > For the past few weeks, I noticed that the amount of memory reported in t= op > (sum of active, inact, wired, cache buf and free) keeps decreasing as the > uptime increases. I can't pinpoint to when I first noticed this, as I have > updated the system a few times just in case this has been fixed. >=20 > The box has 8GB of RAM, running 8.2-PRERELEASE r216693. The ZFS pool > consists of two RAIDZ1 with 5 disks each and 50GB L2 ARC. Root filesystem= is > on UFS on separate disk. I have samba, netatalk, istgt, mysql and apache > running on the box. >=20 > After booting up, top reports I have 8GB of memory as expected. After mov= ing > files around in the ZFS pool, the ARC grows to 4GB (the maximum size I se= t). > Now the box has been up for 10 days, and the sum of memory reported in top > is about 4GB (3GB wired). ARC size is about 2GB. I tried stopping all the > services running, but this didn't help. When unmounting the ZFS pool (via > /etc/rc.d/zfs stop), top reports that I have 8GB again. Throughout the 10 > days period, there is no user activity at all to the UFS filesystem. >=20 > Is there anything sinister happening here? Or is it just top is playing > tricks on me? The easiest way to check for memory leaks is to export the pool and unload zfs.ko. If the leaks exist, they will be reported on the system console. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --WfZ7S8PLGjBY9Voh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0u1ckACgkQForvXbEpPzTvqQCfZnvv3sUW0uQM+DPi+P4SDTvp vuUAnRmKLacklj3OcVEaZCwdYMKqDG2F =L57n -----END PGP SIGNATURE----- --WfZ7S8PLGjBY9Voh-- From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 11:04:03 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C79B1065672 for ; Thu, 13 Jan 2011 11:04:03 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0E6368FC13 for ; Thu, 13 Jan 2011 11:04:02 +0000 (UTC) Received: by qyk36 with SMTP id 36so1564871qyk.13 for ; Thu, 13 Jan 2011 03:04:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:date:x-google-sender-auth :message-id:subject:from:to:content-type; bh=sbMUfxohHMXkF5RJrvWhSp2V/DjmMFCxSRnLGJ7dBeI=; b=lprR+pQzx25zSGMLs/BfOc02Lw2xTVmCqV8T2WTfMAW2LU8oeeUT21r5xqPi9dWhcH 40JspT3gPDTaLaN2AbeH4w7EhNaATMUtUcNKDWqn+XEBTB3++ZTzRPX7Y4vFaaDX8qGd OX7HXEqnELqnXDQ6Fz/hz9HsbNo8W/Z8dZFW0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=QK8LLdZXJXI6fGrzt1u3Sp33zmgNAkrTzEqWnke2eTwf5ouj6oJDfDa7cOHxLv/eCT WETdIvoIhpWC1gWrd+FFnVJp/WhgYm0KWV7kHIEleUgzBe+jNZmtt2BPCXmmI9z/6iXr SAzPkekLGRxVlwLQaxwZn0Lqpm7pV6eJBy0os= MIME-Version: 1.0 Received: by 10.229.240.66 with SMTP id kz2mr1837828qcb.233.1294915238056; Thu, 13 Jan 2011 02:40:38 -0800 (PST) Sender: pluknet@gmail.com Received: by 10.229.102.87 with HTTP; Thu, 13 Jan 2011 02:40:37 -0800 (PST) Date: Thu, 13 Jan 2011 13:40:37 +0300 X-Google-Sender-Auth: aHFHoIrhaLu0CT8xlRe6-bAsGTg Message-ID: From: Sergey Kandaurov To: freebsd-fs@freebsd.org Content-Type: multipart/mixed; boundary=0016363b8848c755820499b7f2a2 Subject: Minor change in ufs_quota.c uprintf() fmt X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 11:04:03 -0000 --0016363b8848c755820499b7f2a2 Content-Type: text/plain; charset=ISO-8859-1 Hello. I found a minor issue in sys/ufs/ufs_quota.c: here uprintf() suboptimally formats a quota error message with "%s", while it is a hardcoded null-terminated character string. The purpose of my change is to embed the quota message into uprintf() fmt itself, thus it will look a bit more correct. While here, I fixed whitespaces around there (tab -> 4 spaces). I'm going to check it in if nobody objects. Please, see attached. -- wbr, pluknet --0016363b8848c755820499b7f2a2 Content-Type: application/octet-stream; name="ufs_quota.diff" Content-Disposition: attachment; filename="ufs_quota.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_givj1aym0 SW5kZXg6IHN5cy91ZnMvdWZzL3Vmc19xdW90YS5jCj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHN5cy91ZnMvdWZz L3Vmc19xdW90YS5jCShyZXZpc2lvbiAyMTczMDgpCisrKyBzeXMvdWZzL3Vmcy91ZnNfcXVvdGEu Ywkod29ya2luZyBjb3B5KQpAQCAtMjM4LDkgKzIzOCw5IEBACiAJCWRxLT5kcV9mbGFncyB8PSBE UV9NT0Q7CiAJCURRSV9VTkxPQ0soZHEpOwogCQlpZiAod2FybikKLQkJCXVwcmludGYoIlxuJXM6 IHdhcm5pbmcsICVzICVzXG4iLAotCQkJCUlUT1YoaXApLT52X21vdW50LT5tbnRfc3RhdC5mX21u dG9ubmFtZSwKLQkJCQlxdW90YXR5cGVzW2ldLCAiZGlzayBxdW90YSBleGNlZWRlZCIpOworCQkJ dXByaW50ZigiXG4lczogd2FybmluZywgJXMgZGlzayBxdW90YSBleGNlZWRlZFxuIiwKKwkJCSAg ICBJVE9WKGlwKS0+dl9tb3VudC0+bW50X3N0YXQuZl9tbnRvbm5hbWUsCisJCQkgICAgcXVvdGF0 eXBlc1tpXSk7CiAJfQogCXJldHVybiAoMCk7CiB9CkBAIC0yODksMTAgKzI4OSwxMCBAQAogCQkJ ICAgIGlwLT5pX3VpZCA9PSBjcmVkLT5jcl91aWQpIHsKIAkJCQlkcS0+ZHFfZmxhZ3MgfD0gRFFf QkxLUzsKIAkJCQlEUUlfVU5MT0NLKGRxKTsKLQkJCQl1cHJpbnRmKCJcbiVzOiB3cml0ZSBmYWls ZWQsICVzICVzXG4iLAorCQkJCXVwcmludGYoIlxuJXM6IHdyaXRlIGZhaWxlZCwgJXMgIgorCQkJ CSAgICAiZGlzayBxdW90YSBleGNlZWRlZCBmb3IgdG9vIGxvbmdcbiIsCiAJCQkJICAgIElUT1Yo aXApLT52X21vdW50LT5tbnRfc3RhdC5mX21udG9ubmFtZSwKLQkJCQkgICAgcXVvdGF0eXBlc1t0 eXBlXSwKLQkJCQkgICAgImRpc2sgcXVvdGEgZXhjZWVkZWQgZm9yIHRvbyBsb25nIik7CisJCQkJ ICAgIHF1b3RhdHlwZXNbdHlwZV0pOwogCQkJCXJldHVybiAoRURRVU9UKTsKIAkJCX0KIAkJCURR SV9VTkxPQ0soZHEpOwpAQCAtMzg0LDkgKzM4NCw5IEBACiAJCWRxLT5kcV9mbGFncyB8PSBEUV9N T0Q7CiAJCURRSV9VTkxPQ0soZHEpOwogCQlpZiAod2FybikKLQkJCXVwcmludGYoIlxuJXM6IHdh cm5pbmcsICVzICVzXG4iLAotCQkJCUlUT1YoaXApLT52X21vdW50LT5tbnRfc3RhdC5mX21udG9u bmFtZSwKLQkJCQlxdW90YXR5cGVzW2ldLCAiaW5vZGUgcXVvdGEgZXhjZWVkZWQiKTsKKwkJCXVw cmludGYoIlxuJXM6IHdhcm5pbmcsICVzIGlub2RlIHF1b3RhIGV4Y2VlZGVkXG4iLAorCQkJICAg IElUT1YoaXApLT52X21vdW50LT5tbnRfc3RhdC5mX21udG9ubmFtZSwKKwkJCSAgICBxdW90YXR5 cGVzW2ldKTsKIAl9CiAJcmV0dXJuICgwKTsKIH0KQEAgLTQzNCwxMCArNDM0LDEwIEBACiAJCQkg ICAgaXAtPmlfdWlkID09IGNyZWQtPmNyX3VpZCkgewogCQkJCWRxLT5kcV9mbGFncyB8PSBEUV9J Tk9EUzsKIAkJCQlEUUlfVU5MT0NLKGRxKTsKLQkJCQl1cHJpbnRmKCJcbiVzOiB3cml0ZSBmYWls ZWQsICVzICVzXG4iLAotCQkJCQlJVE9WKGlwKS0+dl9tb3VudC0+bW50X3N0YXQuZl9tbnRvbm5h bWUsCi0JCQkJCXF1b3RhdHlwZXNbdHlwZV0sCi0JCQkJCSJpbm9kZSBxdW90YSBleGNlZWRlZCBm b3IgdG9vIGxvbmciKTsKKwkJCQl1cHJpbnRmKCJcbiVzOiB3cml0ZSBmYWlsZWQsICVzICIKKwkJ CQkgICAgImlub2RlIHF1b3RhIGV4Y2VlZGVkIGZvciB0b28gbG9uZ1xuIiwKKwkJCQkgICAgSVRP VihpcCktPnZfbW91bnQtPm1udF9zdGF0LmZfbW50b25uYW1lLAorCQkJCSAgICBxdW90YXR5cGVz W3R5cGVdKTsKIAkJCQlyZXR1cm4gKEVEUVVPVCk7CiAJCQl9CiAJCQlEUUlfVU5MT0NLKGRxKTsK --0016363b8848c755820499b7f2a2-- From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 12:19:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04A0C106566B for ; Thu, 13 Jan 2011 12:19:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id B6C478FC0A for ; Thu, 13 Jan 2011 12:19:25 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAAt8Lk2DaFvO/2dsb2JhbACECKExrmqODYEhgzd0BIRohig X-IronPort-AV: E=Sophos;i="4.60,317,1291611600"; d="scan'208";a="105228820" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 13 Jan 2011 07:19:24 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id B798DB3F2D; Thu, 13 Jan 2011 07:19:24 -0500 (EST) Date: Thu, 13 Jan 2011 07:19:24 -0500 (EST) From: Rick Macklem To: Robert Schulze Message-ID: <213698831.150959.1294921164719.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4D2D7ADB.3090902@bytecamp.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: nfs and dropped udp datagrams X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 12:19:26 -0000 > I wonder how big kern.ipc.maxsockbuf shall be tuned on a busy > NFS-Server. In the last time, I often see clients loosing connection > to > their mounted directories. Furthermore, I see increasing counts for > upd datagrams "dropped due to full socket buffers" in netstat -s -p > udp. > > The mbuf situation does not seem to be the reason for the lost > connections, vmstat -z shows 0 failures in the mbuf section. > > Are there any other tunables which could prevent loss of connection to > the server? What is a reasonable value for maxsockbuf? > Prior to r213756 the kernel rpc didn't check for the return value from soreserve() so, if maxsockbuf wasn't large enough or the rsize/wsize was greater than 8K, it failed and the krpc didn't notice. However, if it fails, then sesend() drops messages and that causes grief. I'd just double it on all systems (clients and server), then double it again if you still see problems. rick From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 13:26:10 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D41B61065674 for ; Thu, 13 Jan 2011 13:26:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 8DDD48FC1B for ; Thu, 13 Jan 2011 13:26:10 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAHOMLk2DaFvO/2dsb2JhbACECKExrmaOC4Ehgzd0BIRohig X-IronPort-AV: E=Sophos;i="4.60,317,1291611600"; d="scan'208";a="105233900" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 13 Jan 2011 08:26:09 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 78CEFB3F52; Thu, 13 Jan 2011 08:26:09 -0500 (EST) Date: Thu, 13 Jan 2011 08:26:09 -0500 (EST) From: Rick Macklem To: Robert Schulze Message-ID: <1544383922.152247.1294925169439.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <213698831.150959.1294921164719.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: nfs and dropped udp datagrams X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 13:26:10 -0000 > > I wonder how big kern.ipc.maxsockbuf shall be tuned on a busy > > NFS-Server. In the last time, I often see clients loosing connection > > to > > their mounted directories. Furthermore, I see increasing counts for > > upd datagrams "dropped due to full socket buffers" in netstat -s -p > > udp. > > > > The mbuf situation does not seem to be the reason for the lost > > connections, vmstat -z shows 0 failures in the mbuf section. > > > > Are there any other tunables which could prevent loss of connection > > to > > the server? What is a reasonable value for maxsockbuf? > > > Prior to r213756 the kernel rpc didn't check for the return value from > soreserve() so, if maxsockbuf wasn't large enough or the rsize/wsize > was > greater than 8K, it failed and the krpc didn't notice. However, if it > fails, then sesend() drops messages and that causes grief. > > I'd just double it on all systems (clients and server), then double it > again if you still see problems. > Oh, and for a change to kern.ipc.maxsockbuf to take effect for udp on a server, you need to kill off/restart the nfsds. (The soreserver() is done when the socket is added for servicing and that happens on startup for UDP.) And I think you want to crank up the value on the clients as well, especiallly if your client(s) are pre-r213756 FreeBSD8 ones. rick From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 18:03:56 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA4F5106564A for ; Thu, 13 Jan 2011 18:03:56 +0000 (UTC) (envelope-from kvsbinsol@gmail.com) Received: from mail-yi0-f48.google.com (mail-yi0-f48.google.com [209.85.218.48]) by mx1.freebsd.org (Postfix) with ESMTP id 8A42A8FC15 for ; Thu, 13 Jan 2011 18:03:56 +0000 (UTC) Received: by yib17 with SMTP id 17so907318yib.35 for ; Thu, 13 Jan 2011 10:03:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=fZr+eu5dApJ8NVZyLkK8g+jxAcmA/351+IuSHk5YDQ8=; b=CZisjsHfslJ5Z9dgyd4L/ZoJ0oV0wYsElKgaBSa0m2DCZr9pzmtxBTicaCerPJuhS7 HHRm++SbIEih12d87Ng7f9oS8Fem95m8RGGpmnnRRsgZB2Ne5aHhWXZX38hEnCTCGmWf fJa8aVw6+Ln1gAaqazhLrU/hiUFfjH9yGO2A0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=lOjueEYCqGeivlL7XcKFobC3Fhpl1gT5DK5dJexTZyDqkBcvfzc0fLlSkK+/4wvk/s H6RcvA1ZltPOAjzE9kT0Atrx/4jqQcDJPQ1ZupOrwVzWYjy/TpzcI4ZT+zbRWMKkEqrr yIGTcDC+t/i1xwsr9gd7aEwrGKRTRmA+ezIMc= MIME-Version: 1.0 Received: by 10.151.49.1 with SMTP id b1mr4022278ybk.418.1294940203023; Thu, 13 Jan 2011 09:36:43 -0800 (PST) Received: by 10.151.82.17 with HTTP; Thu, 13 Jan 2011 09:36:42 -0800 (PST) Date: Thu, 13 Jan 2011 18:36:42 +0100 Message-ID: From: Kenneth Vestergaard To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: Booting from a ZFS pool exported on another system. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 18:03:56 -0000 Hello. I'm trying to create some VirtualBox-images with a ZFS root filesystem, but I can't seem to boot from it. I just get a "No ZFS pools located, can't boot". If I take the generated image, and boot it with an mfsbsd-CD, import the pool, and reboot, it works fine, so I'm guessing zfsboot doesn't like it when the pool has been exported? I can't find a way to build an image-file with a pool, without having to export it from the buildsystem when I'm done, so I'm in a bit of a catch-22 here. Is this a bug in zfsboot, or simply just the way things work? -- Best Regards Kenneth Vestergaard From owner-freebsd-fs@FreeBSD.ORG Thu Jan 13 22:31:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CB7D1065674; Thu, 13 Jan 2011 22:31:45 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id E6B678FC16; Thu, 13 Jan 2011 22:31:43 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 7629745C9C; Thu, 13 Jan 2011 23:31:41 +0100 (CET) Received: from localhost (89-73-192-49.dynamic.chello.pl [89.73.192.49]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 2834C45C8A; Thu, 13 Jan 2011 23:31:35 +0100 (CET) Date: Thu, 13 Jan 2011 23:31:25 +0100 From: Pawel Jakub Dawidek To: Chris Forgeron Message-ID: <20110113223125.GA2330@garage.freebsd.pl> References: <20101213214556.GC2038@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0F1p//8PRICkK4MW" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: "freebsd-fs@freebsd.org" , "freebsd-current@freebsd.org" Subject: Re: My ZFS v28 Testing Experience X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2011 22:31:45 -0000 --0F1p//8PRICkK4MW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 12, 2011 at 11:03:19PM -0400, Chris Forgeron wrote: > I've been testing out the v28 patch code for a month now, and I've yet to= report any real issues other than what is mentioned below.=20 >=20 > I'll detail some of the things I've tested, hopefully the stability of v2= 8 in FreeBSD will convince others to give it a try so the final release of = v28 will be as solid as possible. >=20 > I've been using FreeBSD 9.0-CURRENT as of Dec 12th, and 8.2PRE as of Dec = 16th >=20 > What's worked well: >=20 > - I've made and destroyed small raidz's (3-5 disks), large 26 disk raid-1= 0's, and a large 20 disk raid-50. > - I've upgraded from v15, zfs 4, no issues on the different arrays noted = above > - I've confirmed that a v15 or v28 pool will import into Solaris 11 Expre= ss, and vice versa, with the exception about dual log or cache devices note= d below.=20 > - I've run many TB of data through the ZFS storage via benchmarks from my= VM's connected via NFS, to simple copies inside the same pool, or copies f= rom one pool to another.=20 > - I've tested pretty much every compression level, and changing them as I= tweak my setup and try to find the best blend. > - I've added and subtracted many a log and cache device, some in failed s= tates from hot-removals, and the pools always stayed intact. Thank you very much for all your testing, that's really a valuable contribution. I'll be happy to work with you on tracking down the bottleneck in ZFSv28. > Issues: >=20 > - Import of pools with multiple cache or log devices. (May be a very mino= r point) >=20 > A v28 pool created in Solaris 11 Express with 2 or more log devices, or 2= or more cache devices won't import in FreeBSD 9. This also applies to a po= ol that is created in FreeBSD, is imported in Solaris to have the 2 log dev= ices added there, then exported and attempted to be imported back in FreeBS= D. No errors, zpool import just hangs forever. If I reboot into Solaris, im= port the pool, remove the dual devices, then reboot into FreeBSD, I can the= n import the pool without issue. A single cache, or log device will import = just fine. Unfortunately I deleted my witness-enabled FreeBSD-9 drive, so I= can't easily fire it back up to give more debug info. I'm hoping some kind= soul will attempt this type of transaction and report more detail to the l= ist. >=20 > Note - I just decided to try adding 2 cache devices to a raidz pool in Fr= eeBSD, export, and then importing, all without rebooting. That seems to wor= k. BUT - As soon as you try to reboot FreeBSD with this pool staying active= , it hangs on boot. Booting into Solaris, removing the 2 cache devices, the= n booting back into FreeBSD then works. Something is kept in memory between= exporting then importing that allows this to work. =20 Unfortunately I'm unable to reproduce this. It works for me with 2 cache and 2 log vdevs. I tried to reboot, etc. My test exactly looks like this: # zpool create tank raidz ada0 ada1 # zpool add tank cache ada0 ada1 # zpool export tank # kldunload zfs # zpool import tank # reboot > - Speed. (More of an issue, but what do we do?) >=20 > Wow, it's much slower than Solaris 11 Express for transactions. I do unde= rstand that Solaris will have a slight advantage over any port of ZFS. All = of my speed tests are made with a kernel without debug, and yes, these are = -CURRENT and -PRE releases, but the speed difference is very large. Before we go any further could you please confirm that you commented out this line in sys/modules/zfs/Makefile: CFLAGS+=3D-DDEBUG=3D1 This turns all kind of ZFS debugging and slows it down a lot, but for the correctness testing is invaluable. This will be turned off once we import ZFS into FreeBSD-CURRENT. BTW. In my testing Solaris 11 Express is much, much slower than FreeBSD/ZFSv28. And by much I mean two or more times in some tests. I was wondering if they have some debug turned on in Express. > At first, I thought it may be more of an issue with the ix0/Intel X520DA2= 10Gbe drivers that I'm using, since the bulk of my tests are over NFS (I'm= going to use this as a SAN via NFS, so I test in that environment).=20 >=20 > But - I did a raw cp command from one pool to another of several TB. I ex= ecuted the same command under FreeBSD as I did under Solaris 11 Express. Wh= en executed in FreeBSD, the copy took 36 hours. With a fresh destination po= ol of the same settings/compression/etc under Solaris, the copy took 7.5 ho= urs.=20 When you turn off compression (because it turns all-zero blocks into holes) you can test it by simply: # dd if=3D/dev/zero of=3D//zero bs=3D1m --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --0F1p//8PRICkK4MW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0vfT0ACgkQForvXbEpPzTvxwCgib/g/1XuwWzSXj325r/keAwA sHMAn2hW/6V3HJU2mFd3YKdvARFy0xv3 =sy15 -----END PGP SIGNATURE----- --0F1p//8PRICkK4MW-- From owner-freebsd-fs@FreeBSD.ORG Fri Jan 14 08:37:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C06B3106564A; Fri, 14 Jan 2011 08:37:06 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id 4A0248FC0A; Fri, 14 Jan 2011 08:37:05 +0000 (UTC) Received: from mail07.syd.optusnet.com.au (mail07.syd.optusnet.com.au [211.29.132.188]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p0E6aB9L030193; Fri, 14 Jan 2011 17:36:11 +1100 Received: from c122-106-165-206.carlnfd1.nsw.optusnet.com.au (c122-106-165-206.carlnfd1.nsw.optusnet.com.au [122.106.165.206]) by mail07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p0E6a7Tq016920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 14 Jan 2011 17:36:08 +1100 Date: Fri, 14 Jan 2011 17:36:07 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Sergey Kandaurov In-Reply-To: Message-ID: <20110114170710.R28039@besplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: Minor change in ufs_quota.c uprintf() fmt X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jan 2011 08:37:06 -0000 On Thu, 13 Jan 2011, Sergey Kandaurov wrote: > I found a minor issue in sys/ufs/ufs_quota.c: here uprintf() > suboptimally formats a quota error message with "%s", > while it is a hardcoded null-terminated character string. This is mckusick@ style to avoid lines longer than 80 characters without using unportable C90 string concatenation or (maybe) the style bug of using C90 string concatenation. It is common in ffs code. It is used a lot even in ffs_softdep.c, which is about 10 years newer than C90. There it seems to be used mainly for function names. Those have an even larger style bug associated with them. The C99 __func__ is too often used instead of a literal function name. This unportability and style bug is not used in ffs_softdep.c, but the larger unportability __FUNCTION__ is used in about 1/3 of the panics in compatibility cruft at the start of ffs_softdep.c (__FUNCTION__ is the 20 year old gcc spelling of __func__, which was obsoleted by C99 11 years ago except it is not quite compatible). Of course, the lines with this style bug were not written by mckusick@ :-). > The purpose of my change is to embed the quota message > into uprintf() fmt itself, thus it will look a bit more correct. > While here, I fixed whitespaces around there (tab -> 4 spaces). These style bugs were also not writen by mckusick :-). Originally, the %s was needed to split the line according to the style rules because the statement was more indented so there was no space for the literal string, and the continuation indent was 4 spaces. The continuation indent was broken on 14-Mar-2007 in the same commit that changed the statement indentation and missed changing the %s. > I'm going to check it in if nobody objects. Please, see attached. % Index: sys/ufs/ufs/ufs_quota.c % =================================================================== % --- sys/ufs/ufs/ufs_quota.c (revision 217308) % +++ sys/ufs/ufs/ufs_quota.c (working copy) % @@ -238,9 +238,9 @@ % dq->dq_flags |= DQ_MOD; % DQI_UNLOCK(dq); % if (warn) % - uprintf("\n%s: warning, %s %s\n", % - ITOV(ip)->v_mount->mnt_stat.f_mntonname, % - quotatypes[i], "disk quota exceeded"); % + uprintf("\n%s: warning, %s disk quota exceeded\n", % + ITOV(ip)->v_mount->mnt_stat.f_mntonname, % + quotatypes[i]); % } % return (0); % } Correct, since the literal now fits. % @@ -289,10 +289,10 @@ % ip->i_uid == cred->cr_uid) { % dq->dq_flags |= DQ_BLKS; % DQI_UNLOCK(dq); % - uprintf("\n%s: write failed, %s %s\n", % + uprintf("\n%s: write failed, %s " % + "disk quota exceeded for too long\n", % ITOV(ip)->v_mount->mnt_stat.f_mntonname, % - quotatypes[type], % - "disk quota exceeded for too long"); % + quotatypes[type]); % return (EDQUOT); % } % DQI_UNLOCK(dq); Just inconsistent with nearby line splitting style. % @@ -384,9 +384,9 @@ % dq->dq_flags |= DQ_MOD; % DQI_UNLOCK(dq); % if (warn) % - uprintf("\n%s: warning, %s %s\n", % - ITOV(ip)->v_mount->mnt_stat.f_mntonname, % - quotatypes[i], "inode quota exceeded"); % + uprintf("\n%s: warning, %s inode quota exceeded\n", % + ITOV(ip)->v_mount->mnt_stat.f_mntonname, % + quotatypes[i]); % } % return (0); % } Correct. I didn't check the history of this style bug. % @@ -434,10 +434,10 @@ % ip->i_uid == cred->cr_uid) { % dq->dq_flags |= DQ_INODS; % DQI_UNLOCK(dq); % - uprintf("\n%s: write failed, %s %s\n", % - ITOV(ip)->v_mount->mnt_stat.f_mntonname, % - quotatypes[type], % - "inode quota exceeded for too long"); % + uprintf("\n%s: write failed, %s " % + "inode quota exceeded for too long\n", % + ITOV(ip)->v_mount->mnt_stat.f_mntonname, % + quotatypes[type]); % return (EDQUOT); % } % DQI_UNLOCK(dq); Fixes the indentation, but inconsistent with nearby line splitting style. I don't care about pre-C90 not supporting string concatenation, but don't like using this concatenation to split strings except at newlines. It obfuscates the source code in a similar way to moving literals to an arg that is decoded by %s, except slightly less :-). %s is unfortunately necessary for non-literals. Bruce From owner-freebsd-fs@FreeBSD.ORG Fri Jan 14 11:47:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8511106564A for ; Fri, 14 Jan 2011 11:47:07 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1b.sarenet.es (proxypop1b.sarenet.es [194.30.0.104]) by mx1.freebsd.org (Postfix) with ESMTP id E763F8FC16 for ; Fri, 14 Jan 2011 11:47:06 +0000 (UTC) Received: from [172.16.1.55] (ssglan.sare.net [192.148.167.100]) by proxypop1b.sarenet.es (Postfix) with ESMTP id 11F6E5FBC for ; Fri, 14 Jan 2011 12:27:34 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Fri, 14 Jan 2011 12:27:34 +0100 Message-Id: <94E63B7E-C6A1-44EA-A83A-8DEAA60A352A@sarenet.es> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) Subject: ZFS v28 and the infamous deadlock X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jan 2011 11:47:08 -0000 Hello, Long ago I reported a deadlock condition. I reached it by sending = incremental snapshots to a second machine while doing heavy read access = on the target dataset. (I include my original posting below for = reference). I've tried the same with the new ZFS v28 and it's been able to run for = some hours without deadlock (hooray!) but it has crashed with a panic. = However, I was running both FreeBSD systems under VMWare Fusion on my = laptop, and the memory was scarce for them, so it could be related to a = memory shortage. I'm going to set up a couple of servers and try it anyway. At least = seems there's an important change.=20 Pawel, did you try to reproduce the infamous deadlock with v28? Borja. (original message follows) Trying. I started my typical test: Machine 1 doing a make buildworld on a dataset with src and obj on it. Machine 1 replicating incremental snapshots of the dataset to machine 2. Machine 2 running some "tar cf - . | ( cd /pool/anotherdataset && tar xf = - )" from the dataset being replicated, ie, doing read operations on the = target dataset. This time, with all those debug options, there was no deadlock, but = almost an instant trap entering DDB. Unfortunately, I tried to capture = the output of "alltrace", etc using the capture option. But couldn't = come back to the system to read it. Any ideas? I'm using VMWare Fusion to run FreeBSD for these tests and = seems I'm out of luck, I don't see any console output mechanism. When rebooting I was greeted by some LORs lock order reversal: 1st 0xffffff000286c2e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:549 2nd 0xffffff000286b0d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:1173 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff0002864e70 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode_sync.c:381 2nd 0xffffff00026e5140 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff000286c058 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1116 2nd 0xffffff0002591a38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002905638 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock = (spa->spa_sync_bplist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000293bcc8 zfs (zfs) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/gfs.c:437 2nd 0xffffff0002916310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff00025e4078 zp->z_name_lock (zp->z_name_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002916210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:1027 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002861958 buf->b_lock (buf->b_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/arc.c:2509 2nd 0xffffff0002956430 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:421 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_read_done() at dbuf_read_done+0x3b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x6e zap_cursor_retrieve() at zap_cursor_retrieve+0x1bc zfs_unlinked_drain() at zfs_unlinked_drain+0xd8 zfsvfs_setup() at zfsvfs_setup+0xfa zfs_mount() at zfs_mount+0x7df vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000284000) = 0.012686807 s ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based = forwarding disabled, default to deny, logging disabled lock order reversal: 1st 0xffffff800a2f5d08 bufwait (bufwait) @ = /pool/newsrc/src/sys/kern/vfs_bio.c:2559 2nd 0xffffff0002c69800 dirhash (dirhash) @ = /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x88b ufs_mkdir() at ufs_mkdir+0x623 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x80072bb0c, rsp =3D = 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff000286ab88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff0002f6b0d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:606 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_hold_impl() at dnode_hold_impl+0x184 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff00028842e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1724 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_rele() at dbuf_rele+0x2d dnode_hold_impl() at dnode_hold_impl+0x20f dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff00027d5d40 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:687 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_setdirty() at dnode_setdirty+0xbc dbuf_dirty() at dbuf_dirty+0x516 bplist_enqueue() at bplist_enqueue+0xbd dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002c84938 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff0002f6b7b0 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:543 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dbuf_read() at dbuf_read+0x2ad dbuf_will_dirty() at dbuf_will_dirty+0x53 dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x159 dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000291a210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:917 2nd 0xffffff00027450f8 ds->ds_rwlock (ds->ds_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dsl_dataset.c:2864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a dmu_recv_end() at dmu_recv_end+0x94 zfs_ioc_recv() at zfs_ioc_recv+0x29d zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:94 2nd 0xffffff0002f6b330 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1518 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x58 bplist_close() at bplist_close+0x37 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:94 2nd 0xffffffff81152650 h->hash_mutexes[i] (h->hash_mutexes[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:191 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x111 bplist_close() at bplist_close+0x37 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000291a250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:1317 2nd 0xffffff000291a310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:966 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_rezget() at zfs_rezget+0x4a zfs_resume_fs() at zfs_resume_fs+0x158 zfs_ioc_recv() at zfs_ioc_recv+0x2b4 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- #=20 Borja. From owner-freebsd-fs@FreeBSD.ORG Fri Jan 14 13:40:07 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E583D1065693; Fri, 14 Jan 2011 13:40:07 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B26978FC13; Fri, 14 Jan 2011 13:40:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p0EDe7jX078880; Fri, 14 Jan 2011 13:40:07 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p0EDe7ES078876; Fri, 14 Jan 2011 13:40:07 GMT (envelope-from jh) Date: Fri, 14 Jan 2011 13:40:07 GMT Message-Id: <201101141340.p0EDe7ES078876@freefall.freebsd.org> To: sdrhodus@gmail.com, jh@FreeBSD.org, freebsd-fs@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/152605: [ufs] [panic] handle_workitem_freeblocks: inode 18 block count 84 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jan 2011 13:40:08 -0000 Synopsis: [ufs] [panic] handle_workitem_freeblocks: inode 18 block count 84 State-Changed-From-To: open->closed State-Changed-By: jh State-Changed-When: Fri Jan 14 13:40:07 UTC 2011 State-Changed-Why: The panic was changed back to printf in r215950. http://www.freebsd.org/cgi/query-pr.cgi?pr=152605 From owner-freebsd-fs@FreeBSD.ORG Fri Jan 14 13:57:55 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6386A106566C for ; Fri, 14 Jan 2011 13:57:55 +0000 (UTC) (envelope-from znek@mulle-kybernetik.com) Received: from muller.mulle-kybernetik.com (port-212-202-151-204.static.qsc.de [212.202.151.204]) by mx1.freebsd.org (Postfix) with ESMTP id 953408FC12 for ; Fri, 14 Jan 2011 13:57:54 +0000 (UTC) Received: (qmail 40453 invoked from network); 14 Jan 2011 13:31:12 -0000 Received: from unknown (HELO zoidberg.z.net) (znek@78.94.79.22) by mail.mulle-kybernetik.com with AES128-SHA encrypted SMTP; 14 Jan 2011 13:31:12 -0000 From: =?iso-8859-1?Q?Marcus_M=FCller?= Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Fri, 14 Jan 2011 14:31:11 +0100 Message-Id: <37C1E643-C7A9-4061-8316-281819AC947E@mulle-kybernetik.com> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) Subject: Multiple ZFS pools and booting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jan 2011 13:57:55 -0000 Hi all, I have a single harddrive with GPT partitioning: root@muller:(~)# gpart show =3D> 34 234441581 ad10 GPT (112G) 34 128 1 freebsd-boot (64K) 162 8388608 2 freebsd-swap (4.0G) 8388770 10485760 3 freebsd-zfs (5.0G) 18874530 10485760 4 freebsd-zfs (5.0G) 29360290 102540662 5 freebsd-zfs (49G) 131900952 102540662 6 freebsd-zfs (49G) 234441614 1 - free - (512B) ad10p3/ad10p4 (tank) and ad10p5/ad10p6 (muller) are two mirror zpools. = The root filesystem currently resides on tank. I wanted to migrate the root filesystem from tank to muller by changing = the mountpoints accordingly and resetting the bootfs zpool propery on = tank like this: root@muller:(~)# zpool get bootfs muller NAME PROPERTY VALUE SOURCE muller bootfs muller/roots/8-current local root@muller:(~)# zpool set bootfs=3D tank root@muller:(~)# zpool get bootfs tank NAME PROPERTY VALUE SOURCE tank bootfs - default But when I reboot, BTX loader tries to access tank:/boot/kernel/kernel. Why does the loader care about tank at all, after I "removed" the bootfs = property? Do I have to export tank before I reboot? How do I tell the loader to just care about muller for booting? Thanks for any help in advance, Marcus --=20 Marcus Mueller . . . crack-admin/coder ;-) Mulle kybernetiK . http://www.mulle-kybernetik.com Current projects: http://www.mulle-kybernetik.com/znek/ From owner-freebsd-fs@FreeBSD.ORG Fri Jan 14 18:35:02 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C91B5106564A; Fri, 14 Jan 2011 18:35:02 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [64.81.247.49]) by mx1.freebsd.org (Postfix) with ESMTP id 8AF7A8FC15; Fri, 14 Jan 2011 18:35:02 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id p0EIZ1G1049852; Fri, 14 Jan 2011 10:35:02 -0800 (PST) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201101141835.p0EIZ1G1049852@chez.mckusick.com> To: Bruce Evans In-reply-to: <20110114170710.R28039@besplex.bde.org> Date: Fri, 14 Jan 2011 10:35:01 -0800 From: Kirk McKusick Cc: freebsd-fs@freebsd.org, Sergey Kandaurov Subject: Re: Minor change in ufs_quota.c uprintf() fmt X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jan 2011 18:35:02 -0000 I appreciate your detailed analysis of these changes :-) Overall I think that the proposed changes do improve the readability of the code and given that the string concatenation does meet the C99 standard (as opposed to being a gcc'ism) can be expected to be reasonably portable at this time. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Sat Jan 15 00:44:38 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5224A106566C for ; Sat, 15 Jan 2011 00:44:38 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from fep28.mx.upcmail.net (fep28.mx.upcmail.net [62.179.121.48]) by mx1.freebsd.org (Postfix) with ESMTP id 8D0258FC1A for ; Sat, 15 Jan 2011 00:44:37 +0000 (UTC) Received: from edge02.upcmail.net ([192.168.13.237]) by viefep16-int.chello.at (InterMail vM.8.01.02.02 201-2260-120-106-20100312) with ESMTP id <20110115002656.IFNR9043.viefep16-int.chello.at@edge02.upcmail.net> for ; Sat, 15 Jan 2011 01:26:56 +0100 Received: from pinky ([213.46.23.80]) by edge02.upcmail.net with edge id vcSu1f01o1jgp3H02cSwGK; Sat, 15 Jan 2011 01:26:56 +0100 X-SourceIP: 213.46.23.80 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org References: <37C1E643-C7A9-4061-8316-281819AC947E@mulle-kybernetik.com> Date: Sat, 15 Jan 2011 01:26:55 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <37C1E643-C7A9-4061-8316-281819AC947E@mulle-kybernetik.com> User-Agent: Opera Mail/11.00 (Win32) X-Cloudmark-Analysis: v=1.1 cv=vUpxTctd+kpWCBtSXXIkt5ll4Z8E5Qu9nLREXC/hfIo= c=1 sm=0 a=sH-B-3ndls8A:10 a=bgpUlknNv7MA:10 a=IkcTkHD0fZMA:10 a=AHmt4OI5AAAA:8 a=99pkGPs_ccut02i47DwA:9 a=v1dAn5t6yF1ZFUAQ_OuF28B54i8A:4 a=QEXdDO2ut3YA:10 a=btb5l0_vzsgA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 Subject: Re: Multiple ZFS pools and booting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jan 2011 00:44:38 -0000 Shouldn't you set the boot device in /boot/loader.conf? I have this line: vfs.root.mountfrom="zfs:zroot" Ronald. On Fri, 14 Jan 2011 14:31:11 +0100, Marcus Müller wrote: > Hi all, > > I have a single harddrive with GPT partitioning: > > root@muller:(~)# gpart show > => 34 234441581 ad10 GPT (112G) > 34 128 1 freebsd-boot (64K) > 162 8388608 2 freebsd-swap (4.0G) > 8388770 10485760 3 freebsd-zfs (5.0G) > 18874530 10485760 4 freebsd-zfs (5.0G) > 29360290 102540662 5 freebsd-zfs (49G) > 131900952 102540662 6 freebsd-zfs (49G) > 234441614 1 - free - (512B) > > ad10p3/ad10p4 (tank) and ad10p5/ad10p6 (muller) are two mirror zpools. > The root filesystem currently resides on tank. > > I wanted to migrate the root filesystem from tank to muller by changing > the mountpoints accordingly and resetting the bootfs zpool propery on > tank like this: > > root@muller:(~)# zpool get bootfs muller > NAME PROPERTY VALUE SOURCE > muller bootfs muller/roots/8-current local > root@muller:(~)# zpool set bootfs= tank > root@muller:(~)# zpool get bootfs tank > NAME PROPERTY VALUE SOURCE > tank bootfs - default > > But when I reboot, BTX loader tries to access tank:/boot/kernel/kernel. > > Why does the loader care about tank at all, after I "removed" the bootfs > property? > Do I have to export tank before I reboot? > How do I tell the loader to just care about muller for booting? > > Thanks for any help in advance, > > Marcus From owner-freebsd-fs@FreeBSD.ORG Sat Jan 15 09:27:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 821351065670 for ; Sat, 15 Jan 2011 09:27:45 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 2DDC38FC08 for ; Sat, 15 Jan 2011 09:27:44 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id D981145C9C; Sat, 15 Jan 2011 10:27:43 +0100 (CET) Received: from localhost (89-73-192-49.dynamic.chello.pl [89.73.192.49]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id D434B45E49; Sat, 15 Jan 2011 10:27:38 +0100 (CET) Date: Sat, 15 Jan 2011 10:27:29 +0100 From: Pawel Jakub Dawidek To: Kenneth Vestergaard Message-ID: <20110115092729.GB5335@garage.freebsd.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dc+cDN39EJAMEtIO" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Booting from a ZFS pool exported on another system. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jan 2011 09:27:45 -0000 --dc+cDN39EJAMEtIO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 13, 2011 at 06:36:42PM +0100, Kenneth Vestergaard wrote: > Hello. >=20 > I'm trying to create some VirtualBox-images with a ZFS root filesystem, b= ut > I can't seem to boot from it. I just get a "No ZFS pools located, can't > boot". >=20 > If I take the generated image, and boot it with an mfsbsd-CD, import the > pool, and reboot, it works fine, so I'm guessing zfsboot doesn't like it > when the pool has been exported? I can't find a way to build an image-file > with a pool, without having to export it from the buildsystem when I'm > done, so I'm in a bit of a catch-22 here. >=20 > Is this a bug in zfsboot, or simply just the way things work? This was implemented that way, but I don't like this behaviour too. I was beaten by this in the past myself. I just looked at OpenSolaris code and they only skip destroyed pools, which is better than what we do. Could you try the following patch: http://people.freebsd.org/~pjd/patches/zfsimpl.c.patch --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --dc+cDN39EJAMEtIO Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0xaIEACgkQForvXbEpPzQ6lACeJ+FxOxtBv6EJvI4N6LYKFmiN jiUAniYaYawZ+EdztLHuHPLDvC0aylQx =Dzhh -----END PGP SIGNATURE----- --dc+cDN39EJAMEtIO-- From owner-freebsd-fs@FreeBSD.ORG Sat Jan 15 11:23:18 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7EBDB106564A for ; Sat, 15 Jan 2011 11:23:18 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 2B9E08FC14 for ; Sat, 15 Jan 2011 11:23:17 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 51DD64569A; Sat, 15 Jan 2011 12:23:16 +0100 (CET) Received: from localhost (89-73-192-49.dynamic.chello.pl [89.73.192.49]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id F058945684; Sat, 15 Jan 2011 12:23:13 +0100 (CET) Date: Sat, 15 Jan 2011 12:23:04 +0100 From: Pawel Jakub Dawidek To: freebsd-fs@freebsd.org Message-ID: <20110115112304.GC5335@garage.freebsd.pl> References: <4D2B9321.9070306@brawley.id.au> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1ccMZA6j1vT5UqiK" Content-Disposition: inline In-Reply-To: <4D2B9321.9070306@brawley.id.au> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 Cc: Krzysztof Dajka Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jan 2011 11:23:18 -0000 --1ccMZA6j1vT5UqiK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 11, 2011 at 09:45:45AM +1030, Ivan Brawley wrote: > On 10/Jan/2011, Pawel Jakub Dawidek wrote: > > On Sat, Dec 18, 2010 at 10:00:11AM +0100, Krzysztof Dajka wrote: > > > Rebooting with old world and new kernel went fine. But after reboot > > > with new world I got: > > > ZFS: zfs_alloc()/zfs_free() mismatch > > > Just before loading kernel modules, after that my system hangs. > > > > Could you tell me more about you pool configuration? > > 'zpool status' output might be helpful. >=20 > I too got hit by the same thing with my workstation. >=20 > My configuration is 3 x 1.5TB SATA drives in a RAIDZ set and I have a=20 > SSD split into two to use as L2ARC and ZIL (yeah, I know, one SSD only,= =20 > all the fun and games should it die... That's what backups are for :-). >=20 > # zpool status bang_raidz > pool: bang_raidz > state: ONLINE > scan: scrub repaired 0 in 7h55m with 0 errors on Sat Jan 1 16:16:13 2011 > config: >=20 > NAME STATE READ WRITE CKSUM > bang_raidz ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > ada1p3 ONLINE 0 0 0 > ada2p3 ONLINE 0 0 0 > ada3p3 ONLINE 0 0 0 > logs > ada4p2 ONLINE 0 0 0 > cache > ada4p3 ONLINE 0 0 0 >=20 > errors: No known data errors >=20 > I've tried upgrading the pool to v28 without luck. I can import the pool= =20 > if I boot from alternate media (which is what I'm currently doing). >=20 > Doing some debugging, the problem seems to be when zfs_free() is called= =20 > by sys/boot/zfs/zfsimpl.c:zio_read(). I think I found it: http://people.freebsd.org/~pjd/patches/zfssubr.c.patch I was allocating memory instead of freeing it:) Could you try it? --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --1ccMZA6j1vT5UqiK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk0xg5gACgkQForvXbEpPzQi/wCgvZV2x6xH7gluvwWTDovDfxT9 7vIAn1mBedrqwpK79baSwB9Ck1NJbrC0 =dA9j -----END PGP SIGNATURE----- --1ccMZA6j1vT5UqiK-- From owner-freebsd-fs@FreeBSD.ORG Sat Jan 15 12:47:20 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FAD11065674; Sat, 15 Jan 2011 12:47:20 +0000 (UTC) (envelope-from kvsbinsol@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id DB98E8FC0A; Sat, 15 Jan 2011 12:47:19 +0000 (UTC) Received: by gwj21 with SMTP id 21so1492358gwj.13 for ; Sat, 15 Jan 2011 04:47:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Ju9aa/Ex7CdQ0Z4AEeH2qUBWcWiifLVNnQaHVjg2bTc=; b=h2C0db8HSKdPJ8AZJrNja3jhfJvzcMWvMG/TY36IemHj0DI/t3b51x2w9e+nSF/Kw3 le0BugXsBYyjE4Cbl+LLnwvD54/y3ghIrkIVOR0Rrw08dKFlSS7mdsRYA7gHPm/1kh8/ 0UGdtuBdO+1wGutUIg1W1qRVH3CnpOXs1JbY8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=Lq+SEx+FaTB+IMe9e4dxR+Kv1rILl69jhguqwOwFXOYX/kRE0LpzTcIj0bM5LCC4tr ARB+FbrTvYCSWr3SY0wwdpffFbynyx6qtd5xzJFbSldCcjUhfHjDD4wolAnsT1cC5hwd N8H+Lqi07CuOBcRMWx1CokKIMQ5/NtHMUgsO0= MIME-Version: 1.0 Received: by 10.151.147.9 with SMTP id z9mr2540143ybn.247.1295095638944; Sat, 15 Jan 2011 04:47:18 -0800 (PST) Received: by 10.151.82.17 with HTTP; Sat, 15 Jan 2011 04:47:18 -0800 (PST) In-Reply-To: <20110115092729.GB5335@garage.freebsd.pl> References: <20110115092729.GB5335@garage.freebsd.pl> Date: Sat, 15 Jan 2011 13:47:18 +0100 Message-ID: From: Kenneth Vestergaard To: Pawel Jakub Dawidek Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Booting from a ZFS pool exported on another system. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jan 2011 12:47:20 -0000 On Sat, Jan 15, 2011 at 10:27 AM, Pawel Jakub Dawidek wro= te: >> Is this a bug in zfsboot, or simply just the way things work? > > This was implemented that way, but I don't like this behaviour too. > I was beaten by this in the past myself. I just looked at OpenSolaris > code and they only skip destroyed pools, which is better than what we > do. Could you try the following patch: > > =C2=A0 =C2=A0 =C2=A0 =C2=A0http://people.freebsd.org/~pjd/patches/zfsimpl= .c.patch How'd that work when it comes to mounting the root pool? At that point, it'= d still be exported, so I'm guessing something would barf? I got my image booting by simply creating the image before exporting the po= ol. I did a 'zfs unmount imgroot' first, but I don't know if I can be sure that= all buffers are synced. Nonetheless, it worked. --=20 Kenneth Vestergaard From owner-freebsd-fs@FreeBSD.ORG Sat Jan 15 14:00:58 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEB661065675; Sat, 15 Jan 2011 14:00:58 +0000 (UTC) (envelope-from alteriks@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id 7D9E78FC1C; Sat, 15 Jan 2011 14:00:58 +0000 (UTC) Received: by qyk8 with SMTP id 8so357158qyk.13 for ; Sat, 15 Jan 2011 06:00:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=llfMsQlQDtwZvzDyd3DQlEWYVZgI0gLOGdI/GGtTXhw=; b=oWa9r373S7gG4uk9rSaW8xITIJRtgHMl05dPB2My1nMMTwWHvsnyY+gIuRnTChg//q /70YwePXjSld3GiTvgMBZvauPPROEDr6n4l6qkMiUw3MuHp+lA1mAFp9nSHbnKJiL29H yHNKBFNS4XUBDcZSt47PLzbRwILCCqUtQCwm4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=rClJ0wMxpuLTUctHfrwksGVp/Y4RVAmPjDeWVQf1wL5vfEZiFsRpBfiQXC9zM/z48G nuQoWC2eF4Xa3urKc79RWGpqQCVLL/2CvoDXZkHvXm1dZD3nYvztpLbnuwAHbHiw9jUw bxpLkOTTJdJtc4ELBpGve6DQFPOvLrzXaI//4= Received: by 10.224.73.137 with SMTP id q9mr1916649qaj.53.1295100057768; Sat, 15 Jan 2011 06:00:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.87.4 with HTTP; Sat, 15 Jan 2011 06:00:37 -0800 (PST) In-Reply-To: <20110115112304.GC5335@garage.freebsd.pl> References: <4D2B9321.9070306@brawley.id.au> <20110115112304.GC5335@garage.freebsd.pl> From: Krzysztof Dajka Date: Sat, 15 Jan 2011 14:00:37 +0000 Message-ID: To: Pawel Jakub Dawidek Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jan 2011 14:00:58 -0000 On Sat, Jan 15, 2011 at 11:23 AM, Pawel Jakub Dawidek wro= te: > On Tue, Jan 11, 2011 at 09:45:45AM +1030, Ivan Brawley wrote: >> On 10/Jan/2011, Pawel Jakub Dawidek wrote: >> > On Sat, Dec 18, 2010 at 10:00:11AM +0100, Krzysztof Dajka wrote: >> > > Rebooting with old world and new kernel went fine. But after reboot >> > > with new world I got: >> > > ZFS: zfs_alloc()/zfs_free() mismatch >> > > Just before loading kernel modules, after that my system hangs. >> > >> > Could you tell me more about you pool configuration? >> > 'zpool status' output might be helpful. >> At that time, I had 4x500GB hdd in raidz1. Nothing fancy. I won't post zpool status, because I created two zfs mirrors as I'm building new NAS at home. > > I think I found it: > > =C2=A0 =C2=A0 =C2=A0 =C2=A0http://people.freebsd.org/~pjd/patches/zfssubr= .c.patch > > I was allocating memory instead of freeing it:) > Could you try it? > I will try new world on zfs mirror first without zfssubr.c.patch to see if only raidz is affected. Will post results later today.