From owner-freebsd-fs@freebsd.org Sun Sep 4 09:46:05 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97B11A92B40 for ; Sun, 4 Sep 2016 09:46:05 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 31FB96CA30 for ; Sun, 4 Sep 2016 09:46:05 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x243.google.com with SMTP id l65so1109039wmf.3 for ; Sun, 04 Sep 2016 02:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=QjXRP4MA4x2gImgkZ21DJUPfgE17nSPRw7cUcXD0rbQ=; b=X9O52eBE6YMdSDsMulbSy4ARSGRyqHTLCvi62GaEPzeU160SWtAqLUmIup/KGgjLhB qo7JDdAl6yV3+ukvQNpH6jzWbC2IpQYu+9uQa09GKL1sloZhS5KOvtS/QvvaipCDPYIZ /R6FiPWLFYv1dlfSu8rGBHUdPtkVxDi7OEjh5BlEqgXbT1xMpNpDAbCiOf/zYLHc5nYT L8TIj4ZmXcgvwJJP3jL+o6MAIwOaHnnExbdubdn98cMkErDrQQyQ+TPqpfcOnia29NRr uaBbQQyrdcxO1QciU+4VWZo94X2w9hGq3umUsCJ2t0AToM6YXfNhLtpi0/3ibMF/gVTT 1t+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=QjXRP4MA4x2gImgkZ21DJUPfgE17nSPRw7cUcXD0rbQ=; b=F8+0Mh07S5ppqx8wuOPGxPaOZ46PPRpcSFSkZXSKYsKrw+xaY10J70REvZOEXx79ti 6ze9L0TUHScqx6ppnRgQNRAWlt+mTog39b85nix5Pnh+14gjWma+lheBXOnDNlXv4Xwf xlyL/k8u0iQ9rpnkb8Ja+luvtQBriAI2UUlwmD7hzX9hvksUtX2Ojn3b6f/dcyUBm1Ya ArJ9i5iIPABFbc+d+Y2FzTp4cUMHjlZEmeU88ttaKkv8NXzH2P5SoMy8oTlzolY7n4uF 1OPUhG51sMBrxyZxgX84i53LIZZe4QzNeXQ86x1iiaH/hJLM1zg+BzkY4Wp5gu4jA6Y/ BEMg== X-Gm-Message-State: AE9vXwOilyfWq3MQJMdqhFnb9GK62Jh301iCUuIo3gGXY/DQ5iPMmXJaQFR4B9z8svl6Wg== X-Received: by 10.28.25.71 with SMTP id 68mr10017074wmz.19.1472981309889; Sun, 04 Sep 2016 02:28:29 -0700 (PDT) Received: from [172.20.10.6] ([80.12.63.182]) by smtp.gmail.com with ESMTPSA id z18sm13327732wmz.6.2016.09.04.02.28.28 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 04 Sep 2016 02:28:29 -0700 (PDT) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON In-Reply-To: <1472914773423.63807@kuleuven.be> Date: Sun, 4 Sep 2016 11:28:26 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> References: <1472914773423.63807@kuleuven.be> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2016 09:46:05 -0000 Juergen & Bram, Thank you for your feedback. I then investigated further and think I found the root cause. No issue with refquota in my zroot pool containing (in this example) = 300.000 inodes used. However, refquota is terribly slow in my data pool containing around = 12.000.000 inodes used. I then created 12.000.000 empty file in my zroot pool, in a test = dataset. I put a refquota on this dataset and created a dd file to fulfil empty = space. And around the limit, it began to stall... I then created an empty dataset in the same pool, refquota is even slow = in this dataset having no inode used. The root cause seems then to be the total number of inodes used in the = pool... Some numbers : Time to fulfil 512MB with quota : 17s Time to fulfil 512MB with refquota : 3m35s Very strange. Do you experience the same thing ? Thank you again, Ben > On 03 Sep 2016, at 16:59, Bram Vandoren = wrote: >=20 > I encountered the same problem over NFS. I didn't manage to reproduce = it not using NFS. I think the userquota property works without any = problem though. >=20 > Cheers, > Bram. > On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter = wrote: >=20 > cant confirm this, works like a charm without difference to normal = quota > setting From owner-freebsd-fs@freebsd.org Sun Sep 4 11:42:33 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F8B9A9DE11 for ; Sun, 4 Sep 2016 11:42:33 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 42334B15 for ; Sun, 4 Sep 2016 11:42:33 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id E7AEC4C4C85D; Sun, 4 Sep 2016 13:42:24 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JobY1Iyy0ean; Sun, 4 Sep 2016 13:42:22 +0200 (CEST) Received: from ox-groupware-node01.internetx.de (ox.internetx.com [85.236.36.83]) by mx1.internetx.com (Postfix) with ESMTP id AF8654C4C85C; Sun, 4 Sep 2016 13:42:22 +0200 (CEST) Received: from ox-groupware-node01.internetx.de (localhost [127.0.0.1]) by ox-groupware-node01.internetx.de (Postfix) with ESMTP id 98CC4A1213E; Sun, 4 Sep 2016 13:42:22 +0200 (CEST) Date: Sun, 4 Sep 2016 13:42:22 +0200 (CEST) From: InterNetX - Juergen Gotteswinter To: Ben RUBSON , FreeBSD FS Message-ID: <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> In-Reply-To: <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> Subject: Re: [ZFS] refquota is very slow ! MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v7.8.0-Rev27 Organization: InterNetX GmbH X-Originating-Client: com.openexchange.ox.gui.dhtml X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2016 11:42:33 -0000 Did you try the same in a single local disk based pool? And pls post output of zfs get all, zdb & zpool get all > Ben RUBSON hat am 4. September 2016 um 11:28 > geschrieben: > > > Juergen & Bram, > > Thank you for your feedback. > > I then investigated further and think I found the root cause. > > No issue with refquota in my zroot pool containing (in this example) 300.000 > inodes used. > > However, refquota is terribly slow in my data pool containing around > 12.000.000 inodes used. > > I then created 12.000.000 empty file in my zroot pool, in a test dataset. > I put a refquota on this dataset and created a dd file to fulfil empty space. > And around the limit, it began to stall... > I then created an empty dataset in the same pool, refquota is even slow in > this dataset having no inode used. > The root cause seems then to be the total number of inodes used in the pool... > > Some numbers : > Time to fulfil 512MB with quota : 17s > Time to fulfil 512MB with refquota : 3m35s > > Very strange. > > Do you experience the same thing ? > > Thank you again, > > Ben > > > On 03 Sep 2016, at 16:59, Bram Vandoren wrote: > > > > I encountered the same problem over NFS. I didn't manage to reproduce it not > > using NFS. I think the userquota property works without any problem though. > > > > Cheers, > > Bram. > > > On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter > > wrote: > > > > cant confirm this, works like a charm without difference to normal quota > > setting > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sun Sep 4 15:16:31 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65788A9DD82 for ; Sun, 4 Sep 2016 15:16:31 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B576313 for ; Sun, 4 Sep 2016 15:16:31 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22f.google.com with SMTP id 1so99835907wmz.1 for ; Sun, 04 Sep 2016 08:16:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=OE83a4Lj1AgfBekmoz93Tc6XrQH0rUNzu+FF+tl6TQg=; b=qmhTv6B8nSrzcLOcS2qIQoQchPu85hcoOsG0vxMFAm08EI3I/EPqwuYmcCzigWD9M5 1TdJogOrL8Td3ViwNkSVXmRx3wQPbyuBus+uyiK186R5eY5C40NVja0v1KnlEu3c5OQ9 R1vOL4/5iV31t1OwJSGmQrC7w7s/rw7ArP7jCp6UpyGbgj2KYG7hHqlqud8RUwlEBSbG l+BNnuT+XyI5JWrHf4MIRB4uS15P9HObET061qnNThLZSvfq6SJD2XDJ++7WcvhPNm0l 7X2SaWRVWo5Hj8LrYEpVg1A7XAgz/oT3wNTqHlf76aQ0cqDRqXDO3ux7MW9QMrxaahWP RzVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=OE83a4Lj1AgfBekmoz93Tc6XrQH0rUNzu+FF+tl6TQg=; b=fLivlqCNeCiIM+aihX2sZwrqDwrI6aPLGssZDJvWxHEdXpmm8fMvUnPI5kLNMx1OQo ZIPez+RxYPKeY+BgB6lIFdRxfj7uH9vxmC+3O/TBCQVnwbRO8VkBpI+rLs+1N0crw3kh Pj/HF3GmtuJvd3E+bhW2lYrEUjAaqvp+gorK5aDd9vA8VtfupBkPLfdzv2bcXnjamvR0 Ccb2PiX5zNsGWir1DuiSVdbQjUMH4ZdZvyV+h+h8ktCak7Tl9zEGkaaAsYtdh0mIEM6M QAKGkLAb3oaLHX/JdM5WqJr2wYNJ8pM9AmVRxNS0el7KInYS1YpcjeM6KL+UW/mfNI5Y vYxw== X-Gm-Message-State: AE9vXwMjOUPZbYD5Lk3/+7rsLxjLp9svx5ZCouViClT7EF38zkpHDO3lB861P3X9fNIaTQ== X-Received: by 10.194.175.231 with SMTP id cd7mr16773603wjc.77.1473002189399; Sun, 04 Sep 2016 08:16:29 -0700 (PDT) Received: from [172.20.10.6] ([80.12.43.144]) by smtp.gmail.com with ESMTPSA id bc5sm15135484wjb.37.2016.09.04.08.16.27 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 04 Sep 2016 08:16:28 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON X-Priority: 3 In-Reply-To: <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> Date: Sun, 4 Sep 2016 17:16:23 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2016 15:16:31 -0000 Same kind of results with a single local (SSD) disk based pool, refquota = takes much more time than quota around the limit. Here is the output for this single disk based pool : zfs get all : http://pastebin.com/raw/TScgy0ps zdb : http://pastebin.com/raw/BxmQ4xNx zpool get all : http://pastebin.com/raw/XugMbydy Thank you ! Ben > On 04 Sep 2016, at 13:42, InterNetX - Juergen Gotteswinter = wrote: >=20 > Did you try the same in a single local disk based pool? And pls post = output of > zfs get all, zdb & zpool get all >=20 >=20 >> Ben RUBSON hat am 4. September 2016 um 11:28 >> geschrieben: >>=20 >>=20 >> Juergen & Bram, >>=20 >> Thank you for your feedback. >>=20 >> I then investigated further and think I found the root cause. >>=20 >> No issue with refquota in my zroot pool containing (in this example) = 300.000 >> inodes used. >>=20 >> However, refquota is terribly slow in my data pool containing around >> 12.000.000 inodes used. >>=20 >> I then created 12.000.000 empty file in my zroot pool, in a test = dataset. >> I put a refquota on this dataset and created a dd file to fulfil = empty space. >> And around the limit, it began to stall... >> I then created an empty dataset in the same pool, refquota is even = slow in >> this dataset having no inode used. >> The root cause seems then to be the total number of inodes used in = the pool... >>=20 >> Some numbers : >> Time to fulfil 512MB with quota : 17s >> Time to fulfil 512MB with refquota : 3m35s >>=20 >> Very strange. >>=20 >> Do you experience the same thing ? >>=20 >> Thank you again, >>=20 >> Ben >>=20 >>> On 03 Sep 2016, at 16:59, Bram Vandoren = wrote: >>>=20 >>> I encountered the same problem over NFS. I didn't manage to = reproduce it not >>> using NFS. I think the userquota property works without any problem = though. >>>=20 >>> Cheers, >>> Bram. >>=20 >>> On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter >>> wrote: >>>=20 >>> cant confirm this, works like a charm without difference to normal = quota >>> setting >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sun Sep 4 18:11:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C08CCB71F4D for ; Sun, 4 Sep 2016 18:11:40 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EB6683F for ; Sun, 4 Sep 2016 18:11:40 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: by mail-qk0-x234.google.com with SMTP id z190so173628635qkc.0 for ; Sun, 04 Sep 2016 11:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kraus-haus-org.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=IAIfkPXT99wrPrIrRDamVK4ZMTI0oFGl6cXiC8ksxOg=; b=ELV4ybHudF0nEpAMi6g4Uc4DjkOy17fNoNzdm4Jglfa73RElhEXOcqjybol/RG5PNq paBUfv8m4Vorzd7+rwI9P/UeLVNWreBh+ZAH0pb/IptKi3OXBlrrH0gKlaOurUySP0yT vcAVqDP4LJA8nsXSAzZcGAsjUw35ccXzt3GjNNLWJfTyHS6uhSsZb+ygMRN9ajVh7HSl Az+C+q0CC3HdUKXwWV04nHXRNnmFiMkdJllh1u8VtIyGAUHoLNzFgVDT/NdKJHaH0scK +DRD/T4OFsc5Ma6LE1c+SmrrbBSm774TtCdThlORcUqediFEdnLXZtmtQajPOKEg/eEH FP+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=IAIfkPXT99wrPrIrRDamVK4ZMTI0oFGl6cXiC8ksxOg=; b=TyDcvSNOF3fQQoV+5/KOSZRL8NrgWUaqEqdTsj1QA6cASvHoe9OIRl+pQw6KEkcP/j DhDQojmEVO/p+agG91ib2BByM9H2GUmg/duZS1YO8gFowBz7PwxnSAXBlaSpUnUvgyOc GDV8nRp1n6tWT1cLuSRUo0kRnk8/CXhieq5e/IhAg8n/sytVzYYwirYTbwXJBFqC/jFd HwjEPRRlQHCIAnLRNcyLMcgIlS8s+4ZCy5d8Hc1mMP2DFjFD6aZ2mQDb/ETCDRLfbAct qEXbE/Lv1Wc47RGr9pgFx/lh7s6R/m+qv3DZFn6WXRB7qnjksUASM4/A2HrT6ef441sI 6OGg== X-Gm-Message-State: AE9vXwNvoZKtdpn6VdYOwrjJvM0YfdDttAvZcuDnF1abnL2aqBtGPi5IyuqYLw7o45n3tw== X-Received: by 10.55.156.129 with SMTP id f123mr11869063qke.148.1473012699552; Sun, 04 Sep 2016 11:11:39 -0700 (PDT) Received: from [192.168.2.133] (pool-100-4-209-221.albyny.fios.verizon.net. [100.4.209.221]) by smtp.gmail.com with ESMTPSA id l187sm12495081qkc.8.2016.09.04.11.11.38 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 04 Sep 2016 11:11:38 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: phantom snapshots From: Paul Kraus In-Reply-To: Date: Sun, 4 Sep 2016 14:11:37 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <8B542627-1FBA-469D-BAAE-60C9BB615326@kraus-haus.org> References: <58816ff0-3ab2-cbae-3d50-c4d5e89d9773@kateley.com> <57C9D1FB.6040904@b1t.name> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2016 18:11:40 -0000 > On Sep 2, 2016, at 3:41 PM, Linda Kateley = wrote: >=20 > How did you remove the files in the snapshot? I keep trying but it is = read-only? Can I remount it read-write? Snapshots are by design read only copies of the data. When you create a ZFS snapshot you are (functionally) copying the = Uberblock for the dataset and preventing any slabs used by it from being = added to the free list. This is partly why ZFS snapshots have little to = no performance penalty. When you clone a snapshot you are effectively making a read-write copy = of it. So to delete files from a snapshot you destroy the snapshot itself. If = the snapshot has already been destroyed, then you _may_ be running into = the delayed snapshot destruction implemented a bunch of years ago for = some really good reasons. If a system rebooted in the midst of a = snapshot destruction, the reboot would not complete until the snapshot = destruction was completely done. For large snapshots this could be hours = (or even days). So the zfs destroy command returns once the snapshot has been = scheduled to be destroyed, not after the snapshot has been completely = destroyed. How big are these datasets and snapshots ? From owner-freebsd-fs@freebsd.org Sun Sep 4 21:00:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34238B7168E for ; Sun, 4 Sep 2016 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A0DA7A6 for ; Sun, 4 Sep 2016 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u84L01qB021693 for ; Sun, 4 Sep 2016 21:00:19 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201609042100.u84L01qB021693@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 04 Sep 2016 21:00:19 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2016 21:00:19 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 203419 | solaris assert: (dn->dn_phys->dn_nlevels == 0 && Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Sep 5 08:04:37 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02B0CB71DC0 for ; Mon, 5 Sep 2016 08:04:37 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2D5AC0B for ; Mon, 5 Sep 2016 08:04:36 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id C602A4C4C886; Mon, 5 Sep 2016 10:04:32 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m9lguncHJJ2K; Mon, 5 Sep 2016 10:04:28 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id D35CE45FC107; Mon, 5 Sep 2016 10:04:28 +0200 (CEST) Reply-To: juergen.gotteswinter@internetx.com Subject: Re: [ZFS] refquota is very slow ! References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> To: Ben RUBSON , FreeBSD FS From: InterNetX - Juergen Gotteswinter Organization: InterNetX GmbH Message-ID: Date: Mon, 5 Sep 2016 10:04:28 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 08:04:37 -0000 any special reason for disabling secondarycache and limiting the primarycache to metadata? does it change something when you revert it to default? compression -> lz4, even if its not compressable, it wont hurt you probably got several smaller performance issues which end up in this mess all together. Am 04.09.2016 um 17:16 schrieb Ben RUBSON: > Same kind of results with a single local (SSD) disk based pool, refquota takes much more time than quota around the limit. > > Here is the output for this single disk based pool : > zfs get all : http://pastebin.com/raw/TScgy0ps > zdb : http://pastebin.com/raw/BxmQ4xNx > zpool get all : http://pastebin.com/raw/XugMbydy > > Thank you ! > > Ben > > >> On 04 Sep 2016, at 13:42, InterNetX - Juergen Gotteswinter wrote: >> >> Did you try the same in a single local disk based pool? And pls post output of >> zfs get all, zdb & zpool get all >> >> >>> Ben RUBSON hat am 4. September 2016 um 11:28 >>> geschrieben: >>> >>> >>> Juergen & Bram, >>> >>> Thank you for your feedback. >>> >>> I then investigated further and think I found the root cause. >>> >>> No issue with refquota in my zroot pool containing (in this example) 300.000 >>> inodes used. >>> >>> However, refquota is terribly slow in my data pool containing around >>> 12.000.000 inodes used. >>> >>> I then created 12.000.000 empty file in my zroot pool, in a test dataset. >>> I put a refquota on this dataset and created a dd file to fulfil empty space. >>> And around the limit, it began to stall... >>> I then created an empty dataset in the same pool, refquota is even slow in >>> this dataset having no inode used. >>> The root cause seems then to be the total number of inodes used in the pool... >>> >>> Some numbers : >>> Time to fulfil 512MB with quota : 17s >>> Time to fulfil 512MB with refquota : 3m35s >>> >>> Very strange. >>> >>> Do you experience the same thing ? >>> >>> Thank you again, >>> >>> Ben >>> >>>> On 03 Sep 2016, at 16:59, Bram Vandoren wrote: >>>> >>>> I encountered the same problem over NFS. I didn't manage to reproduce it not >>>> using NFS. I think the userquota property works without any problem though. >>>> >>>> Cheers, >>>> Bram. >>> >>>> On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter >>>> wrote: >>>> >>>> cant confirm this, works like a charm without difference to normal quota >>>> setting >>> >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Sep 5 10:01:06 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 429D1B96D90 for ; Mon, 5 Sep 2016 10:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 31F92334 for ; Mon, 5 Sep 2016 10:01:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u85A15r6000271 for ; Mon, 5 Sep 2016 10:01:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212318] sysutils/fusefs-libs: fails to unmount due to a uniq dev node Date: Mon, 05 Sep 2016 10:01:05 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: m.r.sopacua@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-ports-bugs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 10:01:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212318 Melvyn Sopacua changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.r.sopacua@gmail.com --- Comment #1 from Melvyn Sopacua --- Operation not permitted hints at the cause: are you unmounting the device as root or as the same user that mounted the node? I have no issues unmounting multiple fuse devices on 10.3. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Sep 5 14:45:22 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17844B96449 for ; Mon, 5 Sep 2016 14:45:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 079481C8E for ; Mon, 5 Sep 2016 14:45:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u85EjJYv048911 for ; Mon, 5 Sep 2016 14:45:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194513] zfs recv hangs in state kmem arena Date: Mon, 05 Sep 2016 14:45:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: girgen@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 14:45:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194513 --- Comment #20 from Palle Girgensohn --- Hi, I just saw this problem again. zfs recv hang in kmem arena. This was a very common problem for us in that past, but after upgrading to = 10.3 and getting SSD:s for ZIL and L2Arc, this problem has drastically diminishe= d. Apparently, it still happens occasionally, but it is rare. FreeBSD 10.3 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Sep 5 17:20:33 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5661DB96D24 for ; Mon, 5 Sep 2016 17:20:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 45B5FC83 for ; Mon, 5 Sep 2016 17:20:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u85HKXSG066857 for ; Mon, 5 Sep 2016 17:20:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212318] sysutils/fusefs-libs: fails to unmount due to a uniq dev node Date: Mon, 05 Sep 2016 17:20:33 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ben.rubson@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-ports-bugs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 17:20:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212318 --- Comment #2 from Ben RUBSON --- Thank you for your feedback Melvyn. Every mount/unmount is done by users themselves, not by root. Look, in my example, when user3 tries to unmount, fuse tries to unmount FS = of another user (user2 in my example). I think fuse tries to unmount the last one mounted. I think the bug is due to the fact that all FSs share the same /dev/fuse no= de. Do you have several fuse dev nodes on 10.3, one per fuse FS mounted ? (/dev/fuse0, /dev/fuse1, dev/fuse2...) --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Sep 5 18:30:25 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9ADA4B9616B for ; Mon, 5 Sep 2016 18:30:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1FC2BDC9 for ; Mon, 5 Sep 2016 18:30:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id b187so30610188wme.1 for ; Mon, 05 Sep 2016 11:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=0aw2cOVVamznw3mq6Cg2NBuQXTDCxdHylLPqfWkeln4=; b=gDyr/HIyiwkVzBodhr3Ehg0xGOGNoN2usTZ7x2u8vRTBvCxMxOe9TMv1lLBHpd3rZo isqG4KkfQu4Fc23nyeHzdEonOXsbDAEPxCqaJjT9PEorOc4YtRL/fi6/SCa8s6sMLGXW 2pyyuXZ6opkCjVw+tzeAjXzezGFbMiW7wNxof7CkMbbggPUogWkpk6ZbzWAxvTpXQqwS YmLFIRqQAxy64VclNu10pBfnCUsID2+8JKUNXatARucnkz1ume13rRwDNDF8ZqO2RkUC WqliDALM2MYpS4M7ctcWzBnJAcGkJFGRlLphWRBZU7ofDNVg1Kk6C4uam17nZYnzDaUi MA+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=0aw2cOVVamznw3mq6Cg2NBuQXTDCxdHylLPqfWkeln4=; b=NsQ8LeLkGGiuehoZ5seqcW0pYqntzCnq/ueHr/64HYD0406SQpxp6+9pIvki9FEPj3 ThC8P3g3OJC5lbFqSyg56LBuw/EEFR0f9ru7HQmy7JNWUF+AYa3JZZkLxYLY7Cy2LO3Z 0KMnhta3x9pwAphYyUHfautA6mavuNSWPze3+dLcCFPW4r7tLOI8AKB5bTJOdCrmth5V OaVaGsWZzcxk/0ujD2q5kZyPFCMccuFaarVvMY/vwNzAxXp5ddUg9goFy6HQ1Xd1xW9p eKA3WA0uYOb2Aauey9Sb38nLp4fBEgr+io5TnlIAtMVuG0jNW3hDEbcFzVGgn3+PbonR +L3A== X-Gm-Message-State: AE9vXwMQZwr3zJ3fNOOH7qxymq4rrRE7I9nfYnQZ/BkCDiumXluS8imzJNWX+2B2P3TEnA== X-Received: by 10.28.125.80 with SMTP id y77mr17195498wmc.25.1473100223320; Mon, 05 Sep 2016 11:30:23 -0700 (PDT) Received: from [172.20.10.6] ([80.12.63.231]) by smtp.gmail.com with ESMTPSA id g184sm22220342wme.15.2016.09.05.11.30.21 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 05 Sep 2016 11:30:22 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON In-Reply-To: Date: Mon, 5 Sep 2016 20:30:19 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 18:30:25 -0000 Yes, I want to dedicate my ARC & L2ARC to my data pool (which anyway = suffers from the same bug...). I followed your advices and created a whole new SSD test pool, let all = options by default, and added lz4 compression. I then create 20.000.000 empty files in this test pool. Then : # zfs set quota=3D7.5G refquota=3Dnone test/test # dd if=3D/dev/random of=3D/test/test/bf dd: bf: Disc quota exceeded 1047553+0 records in 1047552+0 records out 536346624 bytes transferred in 16.769513 secs (31983434 bytes/sec) # rm /test/test/bf # zfs set quota=3Dnone refquota=3D7.5G test/test # dd if=3D/dev/random of=3D/test/test/bf dd: bf: Disc quota exceeded 1047520+0 records in 1047519+0 records out 536329728 bytes transferred in 215.986582 secs (2483162 bytes/sec) # rm /test/test/bf Additional tests give the same results. 6MB before the limit, write IOs are done very slowly and it takes = minutes to fulfil these remaining MB. How many files in the pool you performed this test in Juergen ? Ben > On 05 Sep 2016, at 10:04, InterNetX - Juergen Gotteswinter = wrote: >=20 > any special reason for disabling secondarycache and limiting the > primarycache to metadata? does it change something when you revert it = to > default? >=20 > compression -> lz4, even if its not compressable, it wont hurt >=20 > you probably got several smaller performance issues which end up in = this > mess all together. >=20 > Am 04.09.2016 um 17:16 schrieb Ben RUBSON: >> Same kind of results with a single local (SSD) disk based pool, = refquota takes much more time than quota around the limit. >>=20 >> Here is the output for this single disk based pool : >> zfs get all : http://pastebin.com/raw/TScgy0ps >> zdb : http://pastebin.com/raw/BxmQ4xNx >> zpool get all : http://pastebin.com/raw/XugMbydy >>=20 >> Thank you ! >>=20 >> Ben >>=20 >>=20 >>> On 04 Sep 2016, at 13:42, InterNetX - Juergen Gotteswinter = wrote: >>>=20 >>> Did you try the same in a single local disk based pool? And pls post = output of >>> zfs get all, zdb & zpool get all >>>=20 >>>=20 >>>> Ben RUBSON hat am 4. September 2016 um 11:28 >>>> geschrieben: >>>>=20 >>>>=20 >>>> Juergen & Bram, >>>>=20 >>>> Thank you for your feedback. >>>>=20 >>>> I then investigated further and think I found the root cause. >>>>=20 >>>> No issue with refquota in my zroot pool containing (in this = example) 300.000 >>>> inodes used. >>>>=20 >>>> However, refquota is terribly slow in my data pool containing = around >>>> 12.000.000 inodes used. >>>>=20 >>>> I then created 12.000.000 empty file in my zroot pool, in a test = dataset. >>>> I put a refquota on this dataset and created a dd file to fulfil = empty space. >>>> And around the limit, it began to stall... >>>> I then created an empty dataset in the same pool, refquota is even = slow in >>>> this dataset having no inode used. >>>> The root cause seems then to be the total number of inodes used in = the pool... >>>>=20 >>>> Some numbers : >>>> Time to fulfil 512MB with quota : 17s >>>> Time to fulfil 512MB with refquota : 3m35s >>>>=20 >>>> Very strange. >>>>=20 >>>> Do you experience the same thing ? >>>>=20 >>>> Thank you again, >>>>=20 >>>> Ben >>>>=20 >>>>> On 03 Sep 2016, at 16:59, Bram Vandoren = wrote: >>>>>=20 >>>>> I encountered the same problem over NFS. I didn't manage to = reproduce it not >>>>> using NFS. I think the userquota property works without any = problem though. >>>>>=20 >>>>> Cheers, >>>>> Bram. >>>>=20 >>>>> On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter >>>>> wrote: >>>>>=20 >>>>> cant confirm this, works like a charm without difference to normal = quota >>>>> setting From owner-freebsd-fs@freebsd.org Tue Sep 6 07:45:05 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 858B3B96FA4 for ; Tue, 6 Sep 2016 07:45:05 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34276DCA for ; Tue, 6 Sep 2016 07:45:04 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id B30E34C4C7BE; Tue, 6 Sep 2016 09:44:56 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KZDT+g0ZZ-t6; Tue, 6 Sep 2016 09:44:54 +0200 (CEST) Received: from [192.168.100.26] (pizza.internetx.de [62.116.129.3]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.internetx.com (Postfix) with ESMTPSA id 64FBC45FC10C; Tue, 6 Sep 2016 09:44:54 +0200 (CEST) Reply-To: juergen.gotteswinter@internetx.com Subject: Re: [ZFS] refquota is very slow ! References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> To: Ben RUBSON , FreeBSD FS From: InterNetX - Juergen Gotteswinter Organization: InterNetX GmbH Message-ID: <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> Date: Tue, 6 Sep 2016 09:44:53 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2016 07:45:05 -0000 i usually use bonnie, fio, or iozone to benchmark the pool io. i dont see the part where 20k files get generated, what you do is writing a single file in a stream from a slow random device until quota kicks in. even dd isnt a reliable benchmarking tool, to get the raw hardware throughput a conv=sync whould be a start to not get just the results of caching. if you could install fio, i can do comparable benchmarks on a single local flash device and spinning rust. fio -bs=8192 -runtime=300 -iodepth 24 -filename test.bin -direct=1 -ioengine=sync or simply download a tarball from kernel.org and time tar -xvzf kernel.tar.gz Am 05.09.2016 um 20:30 schrieb Ben RUBSON: > Yes, I want to dedicate my ARC & L2ARC to my data pool (which anyway suffers from the same bug...). > > I followed your advices and created a whole new SSD test pool, let all options by default, and added lz4 compression. > I then create 20.000.000 empty files in this test pool. > > Then : > > # zfs set quota=7.5G refquota=none test/test > # dd if=/dev/random of=/test/test/bf > dd: bf: Disc quota exceeded > 1047553+0 records in > 1047552+0 records out > 536346624 bytes transferred in 16.769513 secs (31983434 bytes/sec) > # rm /test/test/bf > > # zfs set quota=none refquota=7.5G test/test > # dd if=/dev/random of=/test/test/bf > dd: bf: Disc quota exceeded > 1047520+0 records in > 1047519+0 records out > 536329728 bytes transferred in 215.986582 secs (2483162 bytes/sec) > # rm /test/test/bf > > Additional tests give the same results. > 6MB before the limit, write IOs are done very slowly and it takes minutes to fulfil these remaining MB. > > How many files in the pool you performed this test in Juergen ? > > Ben > >> On 05 Sep 2016, at 10:04, InterNetX - Juergen Gotteswinter wrote: >> >> any special reason for disabling secondarycache and limiting the >> primarycache to metadata? does it change something when you revert it to >> default? >> >> compression -> lz4, even if its not compressable, it wont hurt >> >> you probably got several smaller performance issues which end up in this >> mess all together. >> >> Am 04.09.2016 um 17:16 schrieb Ben RUBSON: >>> Same kind of results with a single local (SSD) disk based pool, refquota takes much more time than quota around the limit. >>> >>> Here is the output for this single disk based pool : >>> zfs get all : http://pastebin.com/raw/TScgy0ps >>> zdb : http://pastebin.com/raw/BxmQ4xNx >>> zpool get all : http://pastebin.com/raw/XugMbydy >>> >>> Thank you ! >>> >>> Ben >>> >>> >>>> On 04 Sep 2016, at 13:42, InterNetX - Juergen Gotteswinter wrote: >>>> >>>> Did you try the same in a single local disk based pool? And pls post output of >>>> zfs get all, zdb & zpool get all >>>> >>>> >>>>> Ben RUBSON hat am 4. September 2016 um 11:28 >>>>> geschrieben: >>>>> >>>>> >>>>> Juergen & Bram, >>>>> >>>>> Thank you for your feedback. >>>>> >>>>> I then investigated further and think I found the root cause. >>>>> >>>>> No issue with refquota in my zroot pool containing (in this example) 300.000 >>>>> inodes used. >>>>> >>>>> However, refquota is terribly slow in my data pool containing around >>>>> 12.000.000 inodes used. >>>>> >>>>> I then created 12.000.000 empty file in my zroot pool, in a test dataset. >>>>> I put a refquota on this dataset and created a dd file to fulfil empty space. >>>>> And around the limit, it began to stall... >>>>> I then created an empty dataset in the same pool, refquota is even slow in >>>>> this dataset having no inode used. >>>>> The root cause seems then to be the total number of inodes used in the pool... >>>>> >>>>> Some numbers : >>>>> Time to fulfil 512MB with quota : 17s >>>>> Time to fulfil 512MB with refquota : 3m35s >>>>> >>>>> Very strange. >>>>> >>>>> Do you experience the same thing ? >>>>> >>>>> Thank you again, >>>>> >>>>> Ben >>>>> >>>>>> On 03 Sep 2016, at 16:59, Bram Vandoren wrote: >>>>>> >>>>>> I encountered the same problem over NFS. I didn't manage to reproduce it not >>>>>> using NFS. I think the userquota property works without any problem though. >>>>>> >>>>>> Cheers, >>>>>> Bram. >>>>> >>>>>> On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter >>>>>> wrote: >>>>>> >>>>>> cant confirm this, works like a charm without difference to normal quota >>>>>> setting > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Sep 6 17:13:17 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FED1BC68D3 for ; Tue, 6 Sep 2016 17:13:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F50A8EB for ; Tue, 6 Sep 2016 17:13:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u86HDHPL001265 for ; Tue, 6 Sep 2016 17:13:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Tue, 06 Sep 2016 17:13:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2016 17:13:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |regression --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Sep 6 18:08:49 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DAAF6BC63F6 for ; Tue, 6 Sep 2016 18:08:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA2BD6D2 for ; Tue, 6 Sep 2016 18:08:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u86I8nO1028233 for ; Tue, 6 Sep 2016 18:08:49 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212323] tests/sys/acl/01:main fails due to changes in NFSv4 ACL behavior on ^/head Date: Tue, 06 Sep 2016 18:08:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2016 18:08:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212323 --- Comment #1 from Edward Tomasz Napierala --- It was caused by one of (relatively) recent commits to ZFS. I've commented= on this on src-commits@. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Sep 6 20:01:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EC8DBC7398 for ; Tue, 6 Sep 2016 20:01:42 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 95472754 for ; Tue, 6 Sep 2016 20:01:42 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from [127.0.0.1] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 1F601195A for ; Tue, 6 Sep 2016 20:01:41 +0000 (UTC) (envelope-from rm@FreeBSD.org) To: freebsd-fs@freebsd.org From: Ruslan Makhmatkhanov Subject: ZFS-8000-8A: assistance needed Message-ID: Date: Tue, 6 Sep 2016 23:00:22 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2016 20:01:42 -0000 Hello, I've got something new here and just not sure where to start on solving that. It's on 10.2-RELEASE-p7 amd64. """ root:~ # zpool status -xv pool: storage_ssd state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 0h26m with 5 errors on Tue Aug 23 00:40:24 2016 config: NAME STATE READ WRITE CKSUM storage_ssd ONLINE 0 0 59.3K mirror-0 ONLINE 0 0 0 gpt/drive-06 ONLINE 0 0 0 gpt/drive-07 ONLINE 0 0 9 mirror-1 ONLINE 0 0 119K gpt/drive-08 ONLINE 0 0 119K gpt/drive-09 ONLINE 0 0 119K cache mfid5 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <0x1bd0a>:<0x8> <0x31f23>:<0x8> /storage_ssd/f262f6ebaf5011e39ca7047d7bb28f4a/disk /storage_ssd/7ba3f661fa9811e3bd9d047d7bb28f4a/disk /storage_ssd/2751d305ecba11e3aef0047d7bb28f4a/disk /storage_ssd/6aa805bd22e911e4b470047d7bb28f4a/disk """ The pool looks ok, if I understand correctly, but we have a slowdown in Xen VM's, that are using these disks via iSCSI. So can please anybody explain what exactly that mean? 1. Am I right that we have a hardware failure that lead to data corruption? If so, how to identify failed disk(s) and how it is possible that data is corrupted on zfs mirror? Is there anything I can do to recover except restoring from backup? 2. What first and second damaged "files" are and why they are shown like that? I have this in /var/log/messages, but to me it looks like iSCSI message, that's spring up when accessing damaged files: """ kernel: (1:32:0/28): WRITE command returned errno 122 """ Manual zpool scrub was tried on this pool to not avail. The pool capacity is only 66% full. Thanks for any hints in advance. -- Regards, Ruslan T.O.S. Of Reality From owner-freebsd-fs@freebsd.org Wed Sep 7 08:29:13 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF948B71144 for ; Wed, 7 Sep 2016 08:29:13 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu1176c.smtpx.saremail.com (cu1176c.smtpx.saremail.com [195.16.148.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6271933 for ; Wed, 7 Sep 2016 08:29:12 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPSA id DE8579DC492; Wed, 7 Sep 2016 10:21:32 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Recommended HBA for ZFS, contemporary From: Borja Marcos In-Reply-To: Date: Wed, 7 Sep 2016 10:21:32 +0200 Cc: Mike , freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <306ccab3-4c78-751a-2258-54701a64ab4b@internetx.com> <41DE4DAA-B5E1-4BB9-AF54-780236A901EF@gmail.com> To: Dmitry Morozovsky X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:29:14 -0000 > On 22 Aug 2016, at 20:36, Dmitry Morozovsky wrote: >=20 > On Mon, 22 Aug 2016, Mike wrote: >=20 >>>> LSI SAS 2008 Based HBA, there are dozens of OEM available for = Budget Prices >>>=20 >>> With the IT firmware. >>=20 >> What's the significance of "IT firmware"? >=20 > ability to export raw disk device withou unnatural intelligence of a = controller=20 > to mangle with data :) That=E2=80=99s a serious problem with other HBAs, which are designed as = =E2=80=9CRAID=E2=80=9D cards. That means you need some effort to have them give you direct, CAM access to the disks. The LSI2008 with IR firmware, however, is a different beast. It=E2=80=99s = an HBA with an add-on RAID thingy. Provided you don=E2=80=99t create logical volumes on it, the mps(4) driver will = just make it behave like a simple HBA, which is excellent. Although some claim that there is a performance difference between IR = and IT firmware on the same card, according to the helpful LSI guys in freebsd-scsi the difference should be = negligible.=20 Anyway in case you want to cross flash from IR to IT it=E2=80=99s = possible, although unsupported by LSI. However, sas2flash checks wether the card is running IT or IR firmware and it will abort = cross flashing attempts, except that at least one old version does not abort and lets you proceed. The procedure is described in a rather religious manner (religious = because they don=E2=80=99t say that a particular version is needed, nor why) here: http://forums.overclockers.com.au/showthread.php?t=3D1045376 If I remember well I tried a LSI utility from the same date more or = less, and it worked as well. More recent versions of sas2flash will fail, however. Cheers, Borja.= From owner-freebsd-fs@freebsd.org Wed Sep 7 08:44:29 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A470AB7177C for ; Wed, 7 Sep 2016 08:44:29 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41B2BF5B for ; Wed, 7 Sep 2016 08:44:29 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id i204so73827427wma.0 for ; Wed, 07 Sep 2016 01:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=L2Hpln2Zr0od6RuONKjJAexT++7tMzg9q0nj3bY8Cno=; b=NUEn8RlfwJD5vmPRlKc0Fwe1LqwySgumNSVgbSOokrLOOC+2G+nzmUO4406WZ2Y1lV JmB8cdx6NtyqAwTk3m8O1BM5xWrzJhO7ja/hRW16hmDhZ1+YsXXIu/1iqM72Agb1hiun 60Nu18y8hnrVHHZ4kDTnIRVa806x5rbPrLdQ/n7Nbf+fMuc9fXtnZlVOBc6m62ZhZPhl 1HhIby3zAbEcXLyBvUZvxdi0i5Cn0atcRCp9wKQ7j7nCK7Ft7TwELLPNv4u1DqjS0SgW ZwumSojgZevLPsr7DQn815AmiO2aKFgbIfVt8WrnhjdDb3LwwfUElBHPmnHdP+3+/d77 uFVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=L2Hpln2Zr0od6RuONKjJAexT++7tMzg9q0nj3bY8Cno=; b=Pg+oAMLYcLNuxl9OaAa7bhjrTH6do1nz9hnpB/C5iIYk4c0auUT7V5VV+qBFS60dAI zepEbsaXxn8AzzJhvP+WBLcvl/NCFRUqDcUdeS7SXEuzy8GUhwXo1fgrc4UJLHIMafx9 cNOoU2VfW6lPzOdrKVrQ7hP+iyyxC8VqYnKr5xLDbrYyC0bPI4U0BnUaoPr1+RmcYISb zJE7iYsU/POqjkybrXWGFAxXOETzbEdwEkAedH9AvcNorW6WuoXHRzkoGMIJpEaQ/T/+ 2wTVsCEsmJagMubCsTkpDhMzyOYeQ4pKqbelbEcTxAeLsmFhjAfsjJCP3qdKqnKQN3eM TMJg== X-Gm-Message-State: AE9vXwNUS6kNsk4D7HaKoc6ItcoYzk/akUxzOtcERhwJvHMrQZr/c1V3dcBqRDQXKdpbbw== X-Received: by 10.28.212.140 with SMTP id l134mr2575375wmg.15.1473237867483; Wed, 07 Sep 2016 01:44:27 -0700 (PDT) Received: from [172.20.10.6] ([80.12.59.77]) by smtp.gmail.com with ESMTPSA id ka5sm36921512wjb.7.2016.09.07.01.44.26 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 07 Sep 2016 01:44:26 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON In-Reply-To: <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> Date: Wed, 7 Sep 2016 10:44:23 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:44:29 -0000 OK ! Well, Juergen you put me on the right way. I investigated further and further and found that refquota slowness is = not due to the number of files but to the remaining free space. Let me explain this. Let's create a pool # zpool create -O atime=3Doff test /dev/... And do some fio tests with quota on it : # zfs set quota=3D500M refquota=3Dnone test/test ; cd /test/test # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D499M WRITE: io=3D510976KB, aggrb=3D805955KB/s, minb=3D805955KB/s, = maxb=3D805955KB/s, mint=3D634msec, maxt=3D634msec OK, very fast (nice SSD), even if we have filled the space up to the = quota. Now, let's do some fio tests with refquota : # zfs set quota=3Dnone refquota=3D500M test/test ; cd /test/test # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D499M WRITE: io=3D510976KB, aggrb=3D1679KB/s, minb=3D1679KB/s, = maxb=3D1679KB/s, mint=3D304177msec, maxt=3D304177msec (!!!) # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D490M WRITE: io=3D501760KB, aggrb=3D44010KB/s, minb=3D44010KB/s, = maxb=3D44010KB/s, mint=3D11401msec, maxt=3D11401msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D480M WRITE: io=3D491520KB, aggrb=3D92774KB/s, minb=3D92774KB/s, = maxb=3D92774KB/s, mint=3D5298msec, maxt=3D5298msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D450M WRITE: io=3D460800KB, aggrb=3D169038KB/s, minb=3D169038KB/s, = maxb=3D169038KB/s, mint=3D2726msec, maxt=3D2726msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D400M WRITE: io=3D409600KB, aggrb=3D215126KB/s, minb=3D215126KB/s, = maxb=3D215126KB/s, mint=3D1904msec, maxt=3D1904msec So, refquota is very very slow when filled space is near the limit, as = if it had some (huge) overhead for each block written. Now let's do some filling tests with dd, first with quota : # zfs set quota=3D500M refquota=3Dnone test/test # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D64k 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D8K 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D4K 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D1K 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D512 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) And now the same filling tests with refquota : # zfs set quota=3Dnone refquota=3D500M test/test # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D64k 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D8K 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D4K 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D1K 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D512 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) OK so here, filling remaining space with small blocks, there are of = course more write operations than with big blocks, refquota overhead is = then really impacting.=20 I think that all these tests reveal that refquota has some overhead = compared to quota to compute a block write when used space is near the = limit. Do you agree ? Question would be then, why this overhead, and could it be reduced (as = with quota) ? (certainly a dev question here) Thank you again, Ben From owner-freebsd-fs@freebsd.org Wed Sep 7 08:49:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10387B7182B for ; Wed, 7 Sep 2016 08:49:19 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 733F410B for ; Wed, 7 Sep 2016 08:49:18 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id C1BB245FC104; Wed, 7 Sep 2016 10:49:09 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 04-rj857161O; Wed, 7 Sep 2016 10:49:07 +0200 (CEST) Received: from ox-groupware-node02.internetx.de (ox.internetx.com [85.236.36.83]) by mx1.internetx.com (Postfix) with ESMTP id 0F13745FC0FF; Wed, 7 Sep 2016 10:49:07 +0200 (CEST) Received: from ox-groupware-node02.internetx.de (localhost [127.0.0.1]) by ox-groupware-node02.internetx.de (Postfix) with ESMTP id ED7481C4409B; Wed, 7 Sep 2016 10:49:06 +0200 (CEST) Date: Wed, 7 Sep 2016 10:49:06 +0200 (CEST) From: InterNetX - Juergen Gotteswinter To: Ben RUBSON , FreeBSD FS Message-ID: <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> In-Reply-To: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> Subject: Re: [ZFS] refquota is very slow ! MIME-Version: 1.0 X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v7.8.0-Rev27 Organization: InterNetX GmbH X-Originating-Client: com.openexchange.ox.gui.dhtml Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:49:19 -0000 > Ben RUBSON hat am 7. September 2016 um 10:44 > geschrieben: > > > > OK ! > Well, Juergen you put me on the right way. > I investigated further and further and found that refquota slowness is not due > to the number of files but to the remaining free space. > Let me explain this. > > Let's create a pool > # zpool create -O atime=off test /dev/... > > And do some fio tests with quota on it : > # zfs set quota=500M refquota=none test/test ; cd /test/test > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=499M > WRITE: io=510976KB, aggrb=805955KB/s, minb=805955KB/s, maxb=805955KB/s, > mint=634msec, maxt=634msec > > OK, very fast (nice SSD), even if we have filled the space up to the quota. > > Now, let's do some fio tests with refquota : > # zfs set quota=none refquota=500M test/test ; cd /test/test > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=499M > WRITE: io=510976KB, aggrb=1679KB/s, minb=1679KB/s, maxb=1679KB/s, > mint=304177msec, maxt=304177msec (!!!) > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=490M > WRITE: io=501760KB, aggrb=44010KB/s, minb=44010KB/s, maxb=44010KB/s, > mint=11401msec, maxt=11401msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=480M > WRITE: io=491520KB, aggrb=92774KB/s, minb=92774KB/s, maxb=92774KB/s, > mint=5298msec, maxt=5298msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=450M > WRITE: io=460800KB, aggrb=169038KB/s, minb=169038KB/s, maxb=169038KB/s, > mint=2726msec, maxt=2726msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=400M > WRITE: io=409600KB, aggrb=215126KB/s, minb=215126KB/s, maxb=215126KB/s, > mint=1904msec, maxt=1904msec > > So, refquota is very very slow when filled space is near the limit, as if it > had some (huge) overhead for each block written. > > Now let's do some filling tests with dd, first with quota : > # zfs set quota=500M refquota=none test/test > # dd if=/dev/random of=/test/test/bf conv=sync bs=64k > 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=8K > 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=4K > 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=1K > 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=512 > 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) > > And now the same filling tests with refquota : > # zfs set quota=none refquota=500M test/test > # dd if=/dev/random of=/test/test/bf conv=sync bs=64k > 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=8K > 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=4K > 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=1K > 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=512 > 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) > > OK so here, filling remaining space with small blocks, there are of course > more write operations than with big blocks, refquota overhead is then really > impacting. thats a general problem which you will also see on nearly full pools, try to no exceed 90% better 80-85%, its the nature of cow fs that this effect occours > > I think that all these tests reveal that refquota has some overhead compared > to quota to compute a block write when used space is near the limit. > Do you agree ? > > Question would be then, why this overhead, and could it be reduced (as with > quota) ? > (certainly a dev question here) > > Thank you again, > > Ben > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 7 08:53:39 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 742CAB71C69 for ; Wed, 7 Sep 2016 08:53:39 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 03181B5D for ; Wed, 7 Sep 2016 08:53:39 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id i204so74222224wma.0 for ; Wed, 07 Sep 2016 01:53:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:message-id:references:to; bh=eDTvCkGaCvtDAuZ3U+dM2nk9yp/S/Zc+680mcHCtdeA=; b=aPoUE5yY50eeVhBw9ipiGsfPvT02A3Uab4cyrAkxrbi3fHnvW8dcu8B+aTDo23M9wr nva27LExXied9EnmxkrYEKJVvJ92FqBV38m1rGhXo0ZV8zssNmpEwSoVgs4YIif0SDRG pN/4mtRg+exmxYFq0Wa31lZyiKpZ0R1lpdWdbLhFKxaTGUiJ8qHM+Mye0/8BuM0ntwX0 eMHi5b1iYcj86cEgAeid1Pn9RnndW8FENap913R7yI+jqS32QYYxN224YTGI3EK26XV1 hlQtBD4gUrzqENiu3hX+KdUt5hWIVUFGp5JPGGnMbtbz2cQY6MOp8VFt+pLz8sihl1UC FQfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :message-id:references:to; bh=eDTvCkGaCvtDAuZ3U+dM2nk9yp/S/Zc+680mcHCtdeA=; b=f82a3XoVwVHcccyyx4HohYj9m4NpuW1fUIy31yrGs5gGMdx6o29YsXDgrd3m749cpW 6XaggAona0pZq3xR88UCW1c2Ttsttn4UIzZE7Y2m7YcLidWXrEUVELlUUU2nR6ZAjqA2 f5hcUWwwVbblIZhq69GHX2Py/Eknm3ZNp2MUqAgoEa/G0p1zvUVp5lQM0aR/3NXsih4y p4TEWf5TqSd2mBWVg9NOyaUWeKUxWTf+mej2r14j2SS1uqPgA8fOr9PK8g+nWDhfvU2Q Q3+pyiGlKUFJpYpW60CRaOsDRySd12eRL4UMOTiaixkQcy8VbpA/ASZ8n/UGm2bp3z2E KOew== X-Gm-Message-State: AE9vXwPHXpsYzqUpZEytwrdFPhsD5M2PprzWhHC30MoTwqp8BNzh4JbTy0hKu6rUBQT98w== X-Received: by 10.28.209.71 with SMTP id i68mr2579802wmg.97.1473238417184; Wed, 07 Sep 2016 01:53:37 -0700 (PDT) Received: from [172.20.10.6] ([80.12.59.77]) by smtp.gmail.com with ESMTPSA id k2sm3039601wmg.23.2016.09.07.01.53.36 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 07 Sep 2016 01:53:36 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON X-Priority: 3 In-Reply-To: <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> Date: Wed, 7 Sep 2016 10:53:37 +0200 Message-Id: <67817F5E-BB61-4DFF-A8E7-B1A190C7CD31@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:53:39 -0000 > On 07 Sep 2016, at 10:49, InterNetX - Juergen Gotteswinter = wrote: >=20 > > Ben RUBSON > hat = am 7. September 2016 um 10:44 geschrieben: >=20 > > OK so here, filling remaining space with small blocks, there are of = course more write operations than with big blocks, refquota overhead is = then really impacting. > =20 > thats a general problem which you will also see on nearly full pools, = try to no exceed 90% better 80-85%, its the nature of cow fs that this = effect occours But why quota is not impacted then ? (ref)quota can be used to restrict users, so we should expect filling up = to the (ref)quota. Of course I agree that pool usage should not exceed 80%. Thank you, Ben From owner-freebsd-fs@freebsd.org Wed Sep 7 09:39:08 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECAA5B96A79 for ; Wed, 7 Sep 2016 09:39:08 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE406251 for ; Wed, 7 Sep 2016 09:39:08 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id B34E3284F6; Wed, 7 Sep 2016 11:38:58 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 0990C284CE; Wed, 7 Sep 2016 11:38:58 +0200 (CEST) Message-ID: <57CFE031.90308@quip.cz> Date: Wed, 07 Sep 2016 11:38:57 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Ben RUBSON , FreeBSD FS Subject: Re: [ZFS] refquota is very slow ! References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> In-Reply-To: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 09:39:09 -0000 Ben RUBSON wrote on 09/07/2016 10:44: > > OK ! > Well, Juergen you put me on the right way. > I investigated further and further and found that refquota slowness is not due to the number of files but to the remaining free space. > Let me explain this. I don't have a bookmark so I can't send you a direct link but it was mentioned before. refquota is similar to full pool in this case, so if you are close to or over 80% full, you will notice this slowdown. You can try to search the web for details. Miroslav Lachman From owner-freebsd-fs@freebsd.org Wed Sep 7 09:55:21 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32D50B96E50 for ; Wed, 7 Sep 2016 09:55:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 22408B07 for ; Wed, 7 Sep 2016 09:55:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u879tL50086543 for ; Wed, 7 Sep 2016 09:55:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 211939] ZFS does not correctly import cache and spares by label Date: Wed, 07 Sep 2016 09:55:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: ben.rubson@gmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Feedback Timeout X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 09:55:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211939 Ben RUBSON changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Feedback Timeout --- Comment #2 from Ben RUBSON --- If I'm the only one interested in this bug report, it can then be closed, a= s I now run FreeBSD 11. Feel free then to re-open it if needed. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Sep 7 18:41:07 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21B15BD0F6A for ; Wed, 7 Sep 2016 18:41:07 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A6B47164F for ; Wed, 7 Sep 2016 18:41:06 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id w12so48741415wmf.0 for ; Wed, 07 Sep 2016 11:41:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=HFMvOQk6vdaZXlqxHvbnQRa8yuLjql2+HQ5fHui2YBU=; b=u0JsIh/tglxLQQ6QltOtB7g2CL3zuuvog7dLUO6D08Nc9d55ttH9JWtc5aVM1kqU1I aeBoUVdjzuU4JOdSBOzNjSSfjWayJc89XEupLMh2AUUTzHzwprTzYTWsCzp8+SjCgWQW 2tGFpKzcRscCyfAQtiuSCdqTuLn4a/ZcLV8F54ysmJVcDT07Nm3PKHlSCVHQvieNJk50 wclNgLCIz0yOAs0s7cOGlS3u5qR2n+SKGv6epH0tpxOuWljV7bL9xOPLZMyflSYUOvzD uKinQ74h7oM6UVzoBERZvXJdZUZ+YcI80R3F80nw3G/LaRIQCzH8xf0O26EIF8K4Z5wq QwSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=HFMvOQk6vdaZXlqxHvbnQRa8yuLjql2+HQ5fHui2YBU=; b=JO13rlWr7iAnLI9EtThtAQbhqm2f1bTunR8L4q7l77Jh6jM7Pcgctw3U9qfRODXBDq rM/R2Ly08P3h/OIs1MPO6SsrDWlvphh//uQSY4X13UsDaFrTg6qThftYfSzFsFgHr6ku ru0w6U+tmC7RyGQ/NRqkFNSZM8ZgCMLIvAHH3RokiONU6n5gIg0boaVCwwpenCoXWmmL Q+Zte6xRn5NjWvCT6FrC5KX5UB7J9funqGZ3/GShpRZk7dx77QkJr7rJuTIk8Vfk9HxA CHg0Dy1aXaBUvhk4RFNwOnStz7ikcn5PlFrPl82CUwP3y7iy01IynwvbAUN2W0p0lhkD Fu+Q== X-Gm-Message-State: AE9vXwPsYhsv0dldiSZXyrRmQHIch13IS/D0XF3ZbSDGOYNkwQqptHUSthKhA3mQK/FSTw== X-Received: by 10.194.0.73 with SMTP id 9mr16370906wjc.123.1473273664916; Wed, 07 Sep 2016 11:41:04 -0700 (PDT) Received: from [172.20.10.6] ([80.12.63.77]) by smtp.gmail.com with ESMTPSA id t65sm717948wmt.15.2016.09.07.11.41.03 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 07 Sep 2016 11:41:04 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: HAST + ZFS + NFS + CARP From: Ben RUBSON In-Reply-To: <62938BA3-67EC-466E-8A0E-E8B6FA544D74@gmail.com> Date: Wed, 7 Sep 2016 20:41:02 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <61283600-A41A-4A8A-92F9-7FAFF54DD175@ixsystems.com> <20160704183643.GI41276@mordor.lan> <20160704193131.GJ41276@mordor.lan> <20160811091016.GI70364@mordor.lan> <1AA52221-9B04-4CF6-97A3-D2C2B330B7F9@sarenet.es> <20160811101539.GM70364@mordor.lan> <20160811110235.GN70364@mordor.lan> <20160811114919.GP70364@mordor.lan> <62938BA3-67EC-466E-8A0E-E8B6FA544D74@gmail.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 18:41:07 -0000 > On 03 Sep 2016, at 15:22, Ben RUBSON wrote: >=20 >> On 11 Aug 2016, at 13:49, Julien Cigar wrote: >>=20 >> BTW any idea if some sort of checksum for payload is made in the = iSCSI >> protocol? >=20 > There is ! > Interesting paper related to this : > = https://www.anixter.com/content/dam/Suppliers/viavi/White%20Paper/Understa= nding%20sCASI%20Digest.pdf Some throughput considerations : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212454= From owner-freebsd-fs@freebsd.org Thu Sep 8 07:45:40 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36FA1BD12F3 for ; Thu, 8 Sep 2016 07:45:40 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from host202-129-static.10-188-b.business.telecomitalia.it (host202-129-static.10-188-b.business.telecomitalia.it [188.10.129.202]) by mx1.freebsd.org (Postfix) with ESMTP id 9F2A8AD5; Thu, 8 Sep 2016 07:45:38 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from [192.168.0.60] (unknown [192.168.0.60]) by host202-129-static.10-188-b.business.telecomitalia.it (Postfix) with ESMTP id 5384712AD8F; Thu, 8 Sep 2016 09:39:59 +0200 (CEST) Subject: Re: ZFS-8000-8A: assistance needed To: Ruslan Makhmatkhanov References: From: Maurizio Vairani Cc: freebsd-fs@freebsd.org Message-ID: Date: Thu, 8 Sep 2016 09:39:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 07:45:40 -0000 Hi Ruslan, Il 06/09/2016 22:00, Ruslan Makhmatkhanov ha scritto: > Hello, > > I've got something new here and just not sure where to start on > solving that. It's on 10.2-RELEASE-p7 amd64. > > """ > root:~ # zpool status -xv > pool: storage_ssd > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://illumos.org/msg/ZFS-8000-8A > scan: scrub repaired 0 in 0h26m with 5 errors on Tue Aug 23 00:40:24 > 2016 > config: > > NAME STATE READ WRITE CKSUM > storage_ssd ONLINE 0 0 59.3K > mirror-0 ONLINE 0 0 0 > gpt/drive-06 ONLINE 0 0 0 > gpt/drive-07 ONLINE 0 0 9 > mirror-1 ONLINE 0 0 119K > gpt/drive-08 ONLINE 0 0 119K > gpt/drive-09 ONLINE 0 0 119K > cache > mfid5 ONLINE 0 0 0 > mfid6 ONLINE 0 0 0 > > errors: Permanent errors have been detected in the following files: > > <0x1bd0a>:<0x8> > <0x31f23>:<0x8> > /storage_ssd/f262f6ebaf5011e39ca7047d7bb28f4a/disk > /storage_ssd/7ba3f661fa9811e3bd9d047d7bb28f4a/disk > /storage_ssd/2751d305ecba11e3aef0047d7bb28f4a/disk > /storage_ssd/6aa805bd22e911e4b470047d7bb28f4a/disk > """ > > The pool looks ok, if I understand correctly, but we have a slowdown > in Xen VM's, that are using these disks via iSCSI. So can please > anybody explain what exactly that mean? The OS retries the read and/or write operation and you notice a slowdown. > > 1. Am I right that we have a hardware failure that lead to data > corruption? Yes. > If so, how to identify failed disk(s) The disks containing gpt/drive-07, the disk with gpt/drive-08 and the disk with gpt/drive-09. With smartctl you can read the smart status of the disks for more info. I use smartd with HDDs and SSDs and it, usually, warns me about a failing disk before zfs. > and how it is possible that data is corrupted on zfs mirror? If in both disks the sectors with the same data are damaged. > Is there anything I can do to recover except restoring from backup? Probably no, but you can check the iSCSI disk in the Xen VM if it is usable. > > 2. What first and second damaged "files" are and why they are shown > like that? ZFS metadata. > > I have this in /var/log/messages, but to me it looks like iSCSI > message, that's spring up when accessing damaged files: > > """ > kernel: (1:32:0/28): WRITE command returned errno 122 > """ Probably in /var/log/messages you can read messages like this: Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): CAM status: ATA Status Error Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC ) Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): RES: 51 40 e8 0f a6 40 44 00 00 08 00 Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): Error 5, Retries exhausted In this message the /dev/ada3 HDD is failing. > Manual zpool scrub was tried on this pool to not avail. The pool > capacity is only 66% full. > > Thanks for any hints in advance. > Maurizio From owner-freebsd-fs@freebsd.org Thu Sep 8 11:47:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D136EBD0FF3 for ; Thu, 8 Sep 2016 11:47:27 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id C5098B9B; Thu, 8 Sep 2016 11:47:27 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from [127.0.0.1] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id E3E7D14C9; Thu, 8 Sep 2016 11:47:26 +0000 (UTC) (envelope-from rm@FreeBSD.org) Subject: Re: ZFS-8000-8A: assistance needed To: Maurizio Vairani , Ruslan Makhmatkhanov References: Cc: freebsd-fs@freebsd.org From: Ruslan Makhmatkhanov Message-ID: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> Date: Thu, 8 Sep 2016 14:46:04 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 11:47:27 -0000 Maurizio, much thanks for your answers. It made some things clearer, but some are still isn't. Please see my comments inline: Maurizio Vairani wrote on 09/08/2016 10:39: > Hi Ruslan, > > Il 06/09/2016 22:00, Ruslan Makhmatkhanov ha scritto: >> Hello, >> >> I've got something new here and just not sure where to start on >> solving that. It's on 10.2-RELEASE-p7 amd64. >> >> """ >> root:~ # zpool status -xv >> pool: storage_ssd >> state: ONLINE >> status: One or more devices has experienced an error resulting in data >> corruption. Applications may be affected. >> action: Restore the file in question if possible. Otherwise restore the >> entire pool from backup. >> see: http://illumos.org/msg/ZFS-8000-8A >> scan: scrub repaired 0 in 0h26m with 5 errors on Tue Aug 23 00:40:24 >> 2016 >> config: >> >> NAME STATE READ WRITE CKSUM >> storage_ssd ONLINE 0 0 59.3K >> mirror-0 ONLINE 0 0 0 >> gpt/drive-06 ONLINE 0 0 0 >> gpt/drive-07 ONLINE 0 0 9 >> mirror-1 ONLINE 0 0 119K >> gpt/drive-08 ONLINE 0 0 119K >> gpt/drive-09 ONLINE 0 0 119K >> cache >> mfid5 ONLINE 0 0 0 >> mfid6 ONLINE 0 0 0 >> >> errors: Permanent errors have been detected in the following files: >> >> <0x1bd0a>:<0x8> >> <0x31f23>:<0x8> >> /storage_ssd/f262f6ebaf5011e39ca7047d7bb28f4a/disk >> /storage_ssd/7ba3f661fa9811e3bd9d047d7bb28f4a/disk >> /storage_ssd/2751d305ecba11e3aef0047d7bb28f4a/disk >> /storage_ssd/6aa805bd22e911e4b470047d7bb28f4a/disk >> """ >> >> The pool looks ok, if I understand correctly, but we have a slowdown >> in Xen VM's, that are using these disks via iSCSI. So can please >> anybody explain what exactly that mean? > The OS retries the read and/or write operation and you notice a slowdown. >> >> 1. Am I right that we have a hardware failure that lead to data >> corruption? > Yes. >> If so, how to identify failed disk(s) > The disks containing gpt/drive-07, the disk with gpt/drive-08 and the > disk with gpt/drive-09. With smartctl you can read the smart status of > the disks for more info. I use smartd with HDDs and SSDs and it, > usually, warns me about a failing disk before zfs. I checked the disks with that labels and all of them seems ok: gpart show excerpt: """ Geom name: mfid2 1. Name: mfid2p1 label: drive-07 Geom name: mfid3 1. Name: mfid3p1 label: drive-08 Geom name: mfid4 1. Name: mfid4p1 label: drive-09 """ I checked them all with smartctl -a -d sat /dev/pass2[3,4] and got """ SMART Status not supported: ATA return descriptor not supported by controller firmware SMART overall-health self-assessment test result: PASSED """ And they also seems ok in mfiutil: root:~ # mfiutil show volumes mfi0 Volumes: Id Size Level Stripe State Cache Name mfid0 ( 838G) RAID-0 256K OPTIMAL Disabled mfid1 ( 838G) RAID-0 256K OPTIMAL Disabled mfid2 ( 476G) RAID-0 256K OPTIMAL Disabled mfid3 ( 476G) RAID-0 256K OPTIMAL Disabled mfid4 ( 476G) RAID-0 256K OPTIMAL Disabled mfid5 ( 167G) RAID-0 64K OPTIMAL Writes mfid6 ( 167G) RAID-0 64K OPTIMAL Writes mfid7 ( 476G) RAID-0 64K OPTIMAL Writes I only wasn't able to read the smart status of mfid0/mfid1 (they are SCSI-6 drives according to mfiutil): smartctl -d scsi -a /dev/pass0[1] Standard Inquiry (36 bytes) failed [Input/output error] Retrying with a 64 byte Standard Inquiry Standard Inquiry (64 bytes) failed [Input/output error] But I'm not sure if I'm using correct type in -d switch (tried all of them). This drives doesn't seems participate in that zpool, but I'm interested in checking them too. So, I'm interested how you detect that particular drives (with gpt labels drive-07, drive-08 and drive-09) are failing? >> and how it is possible that data is corrupted on zfs mirror? > If in both disks the sectors with the same data are damaged. >> Is there anything I can do to recover except restoring from backup? > Probably no, but you can check the iSCSI disk in the Xen VM if it is > usable. It is usable, but the system is very slow. I'll try to read it with dd to check if there are any troubles. >> >> 2. What first and second damaged "files" are and why they are shown >> like that? > ZFS metadata. >> >> I have this in /var/log/messages, but to me it looks like iSCSI >> message, that's spring up when accessing damaged files: >> >> """ >> kernel: (1:32:0/28): WRITE command returned errno 122 >> """ > Probably in /var/log/messages you can read messages like this: > Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): CAM status: > ATA Status Error > Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): ATA status: > 51 (DRDY SERV ERR), error: 40 (UNC ) > Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): RES: 51 40 e8 > 0f a6 40 44 00 00 08 00 > Aug 27 03:02:19 clover-nas2 kernel: (ada3:ahcich15:0:0:0): Error 5, > Retries exhausted > > In this message the /dev/ada3 HDD is failing. Actually, I have no messages like that. Only that "kernel: (1:32:0/28): WRITE command returned errno 122" with no drive indication. Anyway if disk is failing shouldn't I see indication of that in mfiutil drives status and zpool status itself (I confused with that these disks are still listed ONLINE while they are failing)? I also checked logs in VM's itself - there are no complains about disk I/O failures... >> Manual zpool scrub was tried on this pool to not avail. The pool >> capacity is only 66% full. >> >> Thanks for any hints in advance. >> > Maurizio > > > -- Regards, Ruslan T.O.S. Of Reality From owner-freebsd-fs@freebsd.org Thu Sep 8 12:16:20 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DF18BD03FB for ; Thu, 8 Sep 2016 12:16:20 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6943E63A; Thu, 8 Sep 2016 12:16:20 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.87 (FreeBSD)) (envelope-from ) id 1bhyFd-000EtB-PP; Thu, 08 Sep 2016 13:16:18 +0100 Date: Thu, 8 Sep 2016 13:16:17 +0100 From: Gary Palmer To: Ruslan Makhmatkhanov Cc: Maurizio Vairani , freebsd-fs@freebsd.org Subject: Re: ZFS-8000-8A: assistance needed Message-ID: <20160908121617.GA42278@in-addr.com> References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 12:16:20 -0000 On Thu, Sep 08, 2016 at 02:46:04PM +0300, Ruslan Makhmatkhanov wrote: > Maurizio Vairani wrote on 09/08/2016 10:39: > > Hi Ruslan, > > > > Il 06/09/2016 22:00, Ruslan Makhmatkhanov ha scritto: > >> NAME STATE READ WRITE CKSUM > >> storage_ssd ONLINE 0 0 59.3K > >> mirror-0 ONLINE 0 0 0 > >> gpt/drive-06 ONLINE 0 0 0 > >> gpt/drive-07 ONLINE 0 0 9 > >> mirror-1 ONLINE 0 0 119K > >> gpt/drive-08 ONLINE 0 0 119K > >> gpt/drive-09 ONLINE 0 0 119K > >> cache > >> mfid5 ONLINE 0 0 0 > >> mfid6 ONLINE 0 0 0 > >> > So, I'm interested how you detect that particular drives (with gpt > labels drive-07, drive-08 and drive-09) are failing? The CKSUM errors from the zpool status output. They are indicative of underlying problems. gpt/drive-07 has a low count so it's possibly not a major problem, but gpt/drive-08 and gpt/drive-09 both have a significant number. I find it a bit curious that both drives in the mirror have similar number of errors. Possibly indicating data pathing errors. I presume you are using ECC memory? Regards, Gary From owner-freebsd-fs@freebsd.org Thu Sep 8 12:37:14 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 976C2BD0CDF for ; Thu, 8 Sep 2016 12:37:14 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 8ABCA83C; Thu, 8 Sep 2016 12:37:14 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from [127.0.0.1] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 757F81247; Thu, 8 Sep 2016 12:37:13 +0000 (UTC) (envelope-from rm@FreeBSD.org) Subject: Re: ZFS-8000-8A: assistance needed To: Gary Palmer , Ruslan Makhmatkhanov References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> <20160908121617.GA42278@in-addr.com> Cc: Maurizio Vairani , freebsd-fs@freebsd.org From: Ruslan Makhmatkhanov Message-ID: <75465e13-7992-4172-ab40-51b2e462a400@FreeBSD.org> Date: Thu, 8 Sep 2016 15:35:50 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160908121617.GA42278@in-addr.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 12:37:14 -0000 Gary Palmer wrote on 09/08/2016 15:16: > On Thu, Sep 08, 2016 at 02:46:04PM +0300, Ruslan Makhmatkhanov wrote: >> Maurizio Vairani wrote on 09/08/2016 10:39: >>> Hi Ruslan, >>> >>> Il 06/09/2016 22:00, Ruslan Makhmatkhanov ha scritto: >>>> NAME STATE READ WRITE CKSUM >>>> storage_ssd ONLINE 0 0 59.3K >>>> mirror-0 ONLINE 0 0 0 >>>> gpt/drive-06 ONLINE 0 0 0 >>>> gpt/drive-07 ONLINE 0 0 9 >>>> mirror-1 ONLINE 0 0 119K >>>> gpt/drive-08 ONLINE 0 0 119K >>>> gpt/drive-09 ONLINE 0 0 119K >>>> cache >>>> mfid5 ONLINE 0 0 0 >>>> mfid6 ONLINE 0 0 0 >>>> > >> So, I'm interested how you detect that particular drives (with gpt >> labels drive-07, drive-08 and drive-09) are failing? > > The CKSUM errors from the zpool status output. They are > indicative of underlying problems. gpt/drive-07 has a low count > so it's possibly not a major problem, but gpt/drive-08 and > gpt/drive-09 both have a significant number. I find it a bit > curious that both drives in the mirror have similar number of > errors. Possibly indicating data pathing errors. Ah, ok, thanks. I wasn't able to find meaning of CKSUM field description in zpool manpage, so didn't got it at the time. Not sure about data pathing errors though. > > I presume you are using ECC memory? Yes, it's ECC. -- Regards, Ruslan T.O.S. Of Reality From owner-freebsd-fs@freebsd.org Thu Sep 8 12:57:14 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43B33BD154E for ; Thu, 8 Sep 2016 12:57:14 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from host202-129-static.10-188-b.business.telecomitalia.it (host202-129-static.10-188-b.business.telecomitalia.it [188.10.129.202]) by mx1.freebsd.org (Postfix) with ESMTP id E7D1C9DB for ; Thu, 8 Sep 2016 12:57:13 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from [192.168.0.60] (MAURIZIO-PC [192.168.0.60]) by host202-129-static.10-188-b.business.telecomitalia.it (Postfix) with ESMTP id 789DF12A36E for ; Thu, 8 Sep 2016 14:57:12 +0200 (CEST) From: Maurizio Vairani Subject: Re: ZFS-8000-8A: assistance needed To: freebsd-fs@freebsd.org References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> Message-ID: <8b12fa0b-155d-c67c-7a0d-ce10dbe1b21d@cloverinformatica.it> Date: Thu, 8 Sep 2016 14:57:12 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 12:57:14 -0000 Il 08/09/2016 13:46, Ruslan Makhmatkhanov ha scritto: > Maurizio, much thanks for your answers. It made some things clearer, > but some are still isn't. Please see my comments inline: > > I checked the disks with that labels and all of them seems ok: > > gpart show excerpt: > """ > Geom name: mfid2 > 1. Name: mfid2p1 > label: drive-07 > Geom name: mfid3 > 1. Name: mfid3p1 > label: drive-08 > Geom name: mfid4 > 1. Name: mfid4p1 > label: drive-09 > """ > > I checked them all with smartctl -a -d sat /dev/pass2[3,4] and got > > """ > SMART Status not supported: ATA return descriptor not supported by > controller firmware > SMART overall-health self-assessment test result: PASSED > """ > > And they also seems ok in mfiutil: > root:~ # mfiutil show volumes > mfi0 Volumes: > Id Size Level Stripe State Cache Name > mfid0 ( 838G) RAID-0 256K OPTIMAL Disabled > mfid1 ( 838G) RAID-0 256K OPTIMAL Disabled > mfid2 ( 476G) RAID-0 256K OPTIMAL Disabled > mfid3 ( 476G) RAID-0 256K OPTIMAL Disabled > mfid4 ( 476G) RAID-0 256K OPTIMAL Disabled > mfid5 ( 167G) RAID-0 64K OPTIMAL Writes > mfid6 ( 167G) RAID-0 64K OPTIMAL Writes > mfid7 ( 476G) RAID-0 64K OPTIMAL Writes > > I only wasn't able to read the smart status of mfid0/mfid1 (they are > SCSI-6 drives according to mfiutil): > > smartctl -d scsi -a /dev/pass0[1] > Standard Inquiry (36 bytes) failed [Input/output error] > Retrying with a 64 byte Standard Inquiry > Standard Inquiry (64 bytes) failed [Input/output error] > > But I'm not sure if I'm using correct type in -d switch (tried all of > them). This drives doesn't seems participate in that zpool, but I'm > interested in checking them too. > I don't know the LSI MegaRAID SAS controller but according to https://www.smartmontools.org/wiki/Supported_RAID-Controllers seems it is supported in FreeBSD. But what is the output of the command "smartctl --scan" ? Regards, Maurizio From owner-freebsd-fs@freebsd.org Thu Sep 8 13:24:30 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F856BD1CE3 for ; Thu, 8 Sep 2016 13:24:30 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 145BB1B21; Thu, 8 Sep 2016 13:24:30 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.87 (FreeBSD)) (envelope-from ) id 1bhzJb-000F0U-Ba; Thu, 08 Sep 2016 14:24:27 +0100 Date: Thu, 8 Sep 2016 14:24:27 +0100 From: Gary Palmer To: Ruslan Makhmatkhanov Cc: Maurizio Vairani , freebsd-fs@freebsd.org Subject: Re: ZFS-8000-8A: assistance needed Message-ID: <20160908132427.GB42278@in-addr.com> References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> <20160908121617.GA42278@in-addr.com> <75465e13-7992-4172-ab40-51b2e462a400@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <75465e13-7992-4172-ab40-51b2e462a400@FreeBSD.org> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 13:24:30 -0000 On Thu, Sep 08, 2016 at 03:35:50PM +0300, Ruslan Makhmatkhanov wrote: > Gary Palmer wrote on 09/08/2016 15:16: > > On Thu, Sep 08, 2016 at 02:46:04PM +0300, Ruslan Makhmatkhanov wrote: > >> Maurizio Vairani wrote on 09/08/2016 10:39: > >>> Hi Ruslan, > >>> > >>> Il 06/09/2016 22:00, Ruslan Makhmatkhanov ha scritto: > >>>> NAME STATE READ WRITE CKSUM > >>>> storage_ssd ONLINE 0 0 59.3K > >>>> mirror-0 ONLINE 0 0 0 > >>>> gpt/drive-06 ONLINE 0 0 0 > >>>> gpt/drive-07 ONLINE 0 0 9 > >>>> mirror-1 ONLINE 0 0 119K > >>>> gpt/drive-08 ONLINE 0 0 119K > >>>> gpt/drive-09 ONLINE 0 0 119K > >>>> cache > >>>> mfid5 ONLINE 0 0 0 > >>>> mfid6 ONLINE 0 0 0 > >>>> > > > >> So, I'm interested how you detect that particular drives (with gpt > >> labels drive-07, drive-08 and drive-09) are failing? > > > > The CKSUM errors from the zpool status output. They are > > indicative of underlying problems. gpt/drive-07 has a low count > > so it's possibly not a major problem, but gpt/drive-08 and > > gpt/drive-09 both have a significant number. I find it a bit > > curious that both drives in the mirror have similar number of > > errors. Possibly indicating data pathing errors. > > Ah, ok, thanks. I wasn't able to find meaning of CKSUM field description > in zpool manpage, so didn't got it at the time. Not sure about data > pathing errors though. I mean something in the data path for both drives that is common to those two drives and not the others. If they're all attached to the same controller, maybe they're on a different SAS port or cable or something to the other drives. Regards, Gary > > > > I presume you are using ECC memory? > > Yes, it's ECC. > > > -- > Regards, > Ruslan > > T.O.S. Of Reality > From owner-freebsd-fs@freebsd.org Thu Sep 8 14:01:05 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1A86ABD0545 for ; Thu, 8 Sep 2016 14:01:05 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from host202-129-static.10-188-b.business.telecomitalia.it (host202-129-static.10-188-b.business.telecomitalia.it [188.10.129.202]) by mx1.freebsd.org (Postfix) with ESMTP id 7074ADC4; Thu, 8 Sep 2016 14:01:04 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from [192.168.0.60] (MAURIZIO-PC [192.168.0.60]) by host202-129-static.10-188-b.business.telecomitalia.it (Postfix) with ESMTP id 53ED312A485; Thu, 8 Sep 2016 16:01:02 +0200 (CEST) Subject: Re: ZFS-8000-8A: assistance needed To: Ruslan Makhmatkhanov References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> <98bfee78-5440-a105-829b-47a7790889cf@cloverinformatica.it> Cc: freebsd-fs@freebsd.org From: Maurizio Vairani Message-ID: Date: Thu, 8 Sep 2016 16:01:01 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 14:01:05 -0000 Il 08/09/2016 15:16, Ruslan Makhmatkhanov ha scritto: > Maurizio Vairani wrote on 09/08/2016 15:55: >> Il 08/09/2016 13:46, Ruslan Makhmatkhanov ha scritto: >>> Maurizio, much thanks for your answers. It made some things clearer, >>> but some are still isn't. Please see my comments inline: >>> >>> I checked the disks with that labels and all of them seems ok: >>> >>> gpart show excerpt: >>> """ >>> Geom name: mfid2 >>> 1. Name: mfid2p1 >>> label: drive-07 >>> Geom name: mfid3 >>> 1. Name: mfid3p1 >>> label: drive-08 >>> Geom name: mfid4 >>> 1. Name: mfid4p1 >>> label: drive-09 >>> """ >>> >>> I checked them all with smartctl -a -d sat /dev/pass2[3,4] and got >>> >>> """ >>> SMART Status not supported: ATA return descriptor not supported by >>> controller firmware >>> SMART overall-health self-assessment test result: PASSED >>> """ >>> >>> And they also seems ok in mfiutil: >>> root:~ # mfiutil show volumes >>> mfi0 Volumes: >>> Id Size Level Stripe State Cache Name >>> mfid0 ( 838G) RAID-0 256K OPTIMAL Disabled >>> mfid1 ( 838G) RAID-0 256K OPTIMAL Disabled >>> mfid2 ( 476G) RAID-0 256K OPTIMAL Disabled >>> mfid3 ( 476G) RAID-0 256K OPTIMAL Disabled >>> mfid4 ( 476G) RAID-0 256K OPTIMAL Disabled >>> mfid5 ( 167G) RAID-0 64K OPTIMAL Writes >>> mfid6 ( 167G) RAID-0 64K OPTIMAL Writes >>> mfid7 ( 476G) RAID-0 64K OPTIMAL Writes >>> >>> I only wasn't able to read the smart status of mfid0/mfid1 (they are >>> SCSI-6 drives according to mfiutil): >>> >>> smartctl -d scsi -a /dev/pass0[1] >>> Standard Inquiry (36 bytes) failed [Input/output error] >>> Retrying with a 64 byte Standard Inquiry >>> Standard Inquiry (64 bytes) failed [Input/output error] >>> >>> But I'm not sure if I'm using correct type in -d switch (tried all of >>> them). This drives doesn't seems participate in that zpool, but I'm >>> interested in checking them too. >>> >> I don't know the LSI MegaRAID SAS controller but according to >> https://www.smartmontools.org/wiki/Supported_RAID-Controllers >> seems it is supported in FreeBSD. But what is the output of the command >> "smartctl --scan" ? >> >> Regards, >> Maurizio > > Sure, it is supported. > > root@:/var/log# smartctl --scan > /dev/ses0 -d atacam # /dev/ses0, ATA device > /dev/pass1 -d scsi # /dev/pass1, SCSI device > /dev/pass2 -d scsi # /dev/pass2, SCSI device > /dev/pass3 -d scsi # /dev/pass3, SCSI device > /dev/pass4 -d scsi # /dev/pass4, SCSI device > /dev/pass5 -d scsi # /dev/pass5, SCSI device > /dev/pass6 -d scsi # /dev/pass6, SCSI device > /dev/pass7 -d scsi # /dev/pass7, SCSI device > /dev/pass8 -d scsi # /dev/pass8, SCSI device > > While > root@:/var/log# ls -1 /dev/ | grep pass > pass0 > pass1 > pass2 > pass3 > pass4 > pass5 > pass6 > pass7 > pass8 > > root@:/var/log# ls -1 /dev/| grep mfid > mfid0 > mfid0p1 > mfid0p2 > mfid0p3 > mfid1 > mfid1p1 > mfid1p2 > mfid1p3 > mfid2 > mfid2p1 > mfid3 > mfid3p1 > mfid4 > mfid4p1 > mfid5 > mfid6 > mfid7 > mfid7p1 > > So it looks like mfid0 != pass0. And actually pass1 is an interface > for mfid0? Right? > Yes, seems to me too. But, unfortunately, "mrsas (4) does not allow smartctl to access S.M.A.R.T data from physical drive hidden under /dev/daX" bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212211 Maurizio From owner-freebsd-fs@freebsd.org Thu Sep 8 14:56:58 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 965BEBD1E86 for ; Thu, 8 Sep 2016 14:56:58 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2ED661B1 for ; Thu, 8 Sep 2016 14:56:58 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from vader9.bultmann.eu (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id 69F37105F0 for ; Thu, 8 Sep 2016 16:56:47 +0200 (CEST) Subject: Re: ZFS-8000-8A: assistance needed To: freebsd-fs@freebsd.org References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> <98bfee78-5440-a105-829b-47a7790889cf@cloverinformatica.it> From: Jan Bramkamp Message-ID: Date: Thu, 8 Sep 2016 16:56:46 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 14:56:58 -0000 On 08/09/16 16:01, Maurizio Vairani wrote: > Il 08/09/2016 15:16, Ruslan Makhmatkhanov ha scritto: >> Maurizio Vairani wrote on 09/08/2016 15:55: >>> Il 08/09/2016 13:46, Ruslan Makhmatkhanov ha scritto: >>>> Maurizio, much thanks for your answers. It made some things clearer, >>>> but some are still isn't. Please see my comments inline: >>>> >>>> I checked the disks with that labels and all of them seems ok: >>>> >>>> gpart show excerpt: >>>> """ >>>> Geom name: mfid2 >>>> 1. Name: mfid2p1 >>>> label: drive-07 >>>> Geom name: mfid3 >>>> 1. Name: mfid3p1 >>>> label: drive-08 >>>> Geom name: mfid4 >>>> 1. Name: mfid4p1 >>>> label: drive-09 >>>> """ >>>> >>>> I checked them all with smartctl -a -d sat /dev/pass2[3,4] and got >>>> >>>> """ >>>> SMART Status not supported: ATA return descriptor not supported by >>>> controller firmware >>>> SMART overall-health self-assessment test result: PASSED >>>> """ >>>> >>>> And they also seems ok in mfiutil: >>>> root:~ # mfiutil show volumes >>>> mfi0 Volumes: >>>> Id Size Level Stripe State Cache Name >>>> mfid0 ( 838G) RAID-0 256K OPTIMAL Disabled >>>> mfid1 ( 838G) RAID-0 256K OPTIMAL Disabled >>>> mfid2 ( 476G) RAID-0 256K OPTIMAL Disabled >>>> mfid3 ( 476G) RAID-0 256K OPTIMAL Disabled >>>> mfid4 ( 476G) RAID-0 256K OPTIMAL Disabled >>>> mfid5 ( 167G) RAID-0 64K OPTIMAL Writes >>>> mfid6 ( 167G) RAID-0 64K OPTIMAL Writes >>>> mfid7 ( 476G) RAID-0 64K OPTIMAL Writes >>>> >>>> I only wasn't able to read the smart status of mfid0/mfid1 (they are >>>> SCSI-6 drives according to mfiutil): >>>> >>>> smartctl -d scsi -a /dev/pass0[1] >>>> Standard Inquiry (36 bytes) failed [Input/output error] >>>> Retrying with a 64 byte Standard Inquiry >>>> Standard Inquiry (64 bytes) failed [Input/output error] >>>> >>>> But I'm not sure if I'm using correct type in -d switch (tried all of >>>> them). This drives doesn't seems participate in that zpool, but I'm >>>> interested in checking them too. >>>> >>> I don't know the LSI MegaRAID SAS controller but according to >>> https://www.smartmontools.org/wiki/Supported_RAID-Controllers >>> seems it is supported in FreeBSD. But what is the output of the command >>> "smartctl --scan" ? >>> >>> Regards, >>> Maurizio >> >> Sure, it is supported. >> >> root@:/var/log# smartctl --scan >> /dev/ses0 -d atacam # /dev/ses0, ATA device >> /dev/pass1 -d scsi # /dev/pass1, SCSI device >> /dev/pass2 -d scsi # /dev/pass2, SCSI device >> /dev/pass3 -d scsi # /dev/pass3, SCSI device >> /dev/pass4 -d scsi # /dev/pass4, SCSI device >> /dev/pass5 -d scsi # /dev/pass5, SCSI device >> /dev/pass6 -d scsi # /dev/pass6, SCSI device >> /dev/pass7 -d scsi # /dev/pass7, SCSI device >> /dev/pass8 -d scsi # /dev/pass8, SCSI device >> >> While >> root@:/var/log# ls -1 /dev/ | grep pass >> pass0 >> pass1 >> pass2 >> pass3 >> pass4 >> pass5 >> pass6 >> pass7 >> pass8 >> >> root@:/var/log# ls -1 /dev/| grep mfid >> mfid0 >> mfid0p1 >> mfid0p2 >> mfid0p3 >> mfid1 >> mfid1p1 >> mfid1p2 >> mfid1p3 >> mfid2 >> mfid2p1 >> mfid3 >> mfid3p1 >> mfid4 >> mfid4p1 >> mfid5 >> mfid6 >> mfid7 >> mfid7p1 >> >> So it looks like mfid0 != pass0. And actually pass1 is an interface >> for mfid0? Right? >> > Yes, seems to me too. > > But, unfortunately, "mrsas (4) does not allow smartctl to access > S.M.A.R.T data from physical drive hidden under /dev/daX" He's using the older mfi driver instead of the mrsas driver, because otherwise the volumes would show up as da* instead of mfid*. From owner-freebsd-fs@freebsd.org Thu Sep 8 16:35:47 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F320BD2AE1 for ; Thu, 8 Sep 2016 16:35:47 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 83F85D6F; Thu, 8 Sep 2016 16:35:47 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from [127.0.0.1] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 9D67F1C14; Thu, 8 Sep 2016 16:35:46 +0000 (UTC) (envelope-from rm@FreeBSD.org) Subject: Re: ZFS-8000-8A: assistance needed To: Maurizio Vairani , Ruslan Makhmatkhanov References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> <98bfee78-5440-a105-829b-47a7790889cf@cloverinformatica.it> Cc: freebsd-fs@freebsd.org From: Ruslan Makhmatkhanov Message-ID: <352e1cbf-1daa-f78c-cb0b-0752e36f8928@FreeBSD.org> Date: Thu, 8 Sep 2016 19:34:23 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 16:35:47 -0000 Maurizio Vairani wrote on 09/08/2016 17:01: > Il 08/09/2016 15:16, Ruslan Makhmatkhanov ha scritto: >> Maurizio Vairani wrote on 09/08/2016 15:55: >>> Il 08/09/2016 13:46, Ruslan Makhmatkhanov ha scritto: >>>> Maurizio, much thanks for your answers. It made some things clearer, >>>> but some are still isn't. Please see my comments inline: >>>> >>>> I checked the disks with that labels and all of them seems ok: >>>> >>>> gpart show excerpt: >>>> """ >>>> Geom name: mfid2 >>>> 1. Name: mfid2p1 >>>> label: drive-07 >>>> Geom name: mfid3 >>>> 1. Name: mfid3p1 >>>> label: drive-08 >>>> Geom name: mfid4 >>>> 1. Name: mfid4p1 >>>> label: drive-09 >>>> """ >>>> >>>> I checked them all with smartctl -a -d sat /dev/pass2[3,4] and got >>>> >>>> """ >>>> SMART Status not supported: ATA return descriptor not supported by >>>> controller firmware >>>> SMART overall-health self-assessment test result: PASSED >>>> """ >>>> >>>> And they also seems ok in mfiutil: >>>> root:~ # mfiutil show volumes >>>> mfi0 Volumes: >>>> Id Size Level Stripe State Cache Name >>>> mfid0 ( 838G) RAID-0 256K OPTIMAL Disabled >>>> mfid1 ( 838G) RAID-0 256K OPTIMAL Disabled >>>> mfid2 ( 476G) RAID-0 256K OPTIMAL Disabled >>>> mfid3 ( 476G) RAID-0 256K OPTIMAL Disabled >>>> mfid4 ( 476G) RAID-0 256K OPTIMAL Disabled >>>> mfid5 ( 167G) RAID-0 64K OPTIMAL Writes >>>> mfid6 ( 167G) RAID-0 64K OPTIMAL Writes >>>> mfid7 ( 476G) RAID-0 64K OPTIMAL Writes >>>> >>>> I only wasn't able to read the smart status of mfid0/mfid1 (they are >>>> SCSI-6 drives according to mfiutil): >>>> >>>> smartctl -d scsi -a /dev/pass0[1] >>>> Standard Inquiry (36 bytes) failed [Input/output error] >>>> Retrying with a 64 byte Standard Inquiry >>>> Standard Inquiry (64 bytes) failed [Input/output error] >>>> >>>> But I'm not sure if I'm using correct type in -d switch (tried all of >>>> them). This drives doesn't seems participate in that zpool, but I'm >>>> interested in checking them too. >>>> >>> I don't know the LSI MegaRAID SAS controller but according to >>> https://www.smartmontools.org/wiki/Supported_RAID-Controllers >>> seems it is supported in FreeBSD. But what is the output of the command >>> "smartctl --scan" ? >>> >>> Regards, >>> Maurizio >> >> Sure, it is supported. >> [...] >> >> So it looks like mfid0 != pass0. And actually pass1 is an interface >> for mfid0? Right? >> > Yes, seems to me too. > > But, unfortunately, "mrsas (4) does not allow smartctl to access > S.M.A.R.T data from physical drive hidden under /dev/daX" > > bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212211 > > Maurizio This makes sense now. Thank you! We'll try check the controller port and cables as suggested by gpalmer@ because error rates on both drives are equal and this is strange. And if this doesn't make a difference, then replace the drives. -- Regards, Ruslan T.O.S. Of Reality From owner-freebsd-fs@freebsd.org Thu Sep 8 16:37:01 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB98ABD2B25 for ; Thu, 8 Sep 2016 16:37:01 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id CDA85DFD; Thu, 8 Sep 2016 16:37:01 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from [127.0.0.1] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id B2F791C77; Thu, 8 Sep 2016 16:37:00 +0000 (UTC) (envelope-from rm@FreeBSD.org) Subject: Re: ZFS-8000-8A: assistance needed To: Gary Palmer , Ruslan Makhmatkhanov References: <11f2a660-304d-4a50-bab8-ec2a2737b3a4@FreeBSD.org> <20160908121617.GA42278@in-addr.com> <75465e13-7992-4172-ab40-51b2e462a400@FreeBSD.org> <20160908132427.GB42278@in-addr.com> Cc: Maurizio Vairani , freebsd-fs@freebsd.org From: Ruslan Makhmatkhanov Message-ID: <51014b52-b521-7539-d7f5-9b265a85cd52@FreeBSD.org> Date: Thu, 8 Sep 2016 19:35:37 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160908132427.GB42278@in-addr.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Sep 2016 16:37:01 -0000 Gary Palmer wrote on 09/08/2016 16:24: > On Thu, Sep 08, 2016 at 03:35:50PM +0300, Ruslan Makhmatkhanov wrote: >> Gary Palmer wrote on 09/08/2016 15:16: >>> On Thu, Sep 08, 2016 at 02:46:04PM +0300, Ruslan Makhmatkhanov wrote: >>>> Maurizio Vairani wrote on 09/08/2016 10:39: >>>>> Hi Ruslan, >>>>> >>>>> Il 06/09/2016 22:00, Ruslan Makhmatkhanov ha scritto: >>>>>> NAME STATE READ WRITE CKSUM >>>>>> storage_ssd ONLINE 0 0 59.3K >>>>>> mirror-0 ONLINE 0 0 0 >>>>>> gpt/drive-06 ONLINE 0 0 0 >>>>>> gpt/drive-07 ONLINE 0 0 9 >>>>>> mirror-1 ONLINE 0 0 119K >>>>>> gpt/drive-08 ONLINE 0 0 119K >>>>>> gpt/drive-09 ONLINE 0 0 119K >>>>>> cache >>>>>> mfid5 ONLINE 0 0 0 >>>>>> mfid6 ONLINE 0 0 0 >>>>>> >>> >>>> So, I'm interested how you detect that particular drives (with gpt >>>> labels drive-07, drive-08 and drive-09) are failing? >>> >>> The CKSUM errors from the zpool status output. They are >>> indicative of underlying problems. gpt/drive-07 has a low count >>> so it's possibly not a major problem, but gpt/drive-08 and >>> gpt/drive-09 both have a significant number. I find it a bit >>> curious that both drives in the mirror have similar number of >>> errors. Possibly indicating data pathing errors. >> >> Ah, ok, thanks. I wasn't able to find meaning of CKSUM field description >> in zpool manpage, so didn't got it at the time. Not sure about data >> pathing errors though. > > I mean something in the data path for both drives that is common > to those two drives and not the others. If they're all attached to > the same controller, maybe they're on a different SAS port or cable > or something to the other drives. > > Regards, > > Gary We'll check that. Thank you, Gary! -- Regards, Ruslan T.O.S. Of Reality From owner-freebsd-fs@freebsd.org Sat Sep 10 08:58:27 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B36DBD2396 for ; Sat, 10 Sep 2016 08:58:27 +0000 (UTC) (envelope-from c.pilka@asconix.com) Received: from aibo.runbox.com (aibo.runbox.com [91.220.196.211]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52697EFE for ; Sat, 10 Sep 2016 08:58:26 +0000 (UTC) (envelope-from c.pilka@asconix.com) Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from ) id 1bie7F-0000pA-0u for freebsd-fs@freebsd.org; Sat, 10 Sep 2016 10:58:25 +0200 Received: from cm-84.211.200.201.getinternet.no ([84.211.200.201] helo=houdini.fritz.box) by mailfront11.runbox.com with esmtpsa (uid:865152 ) (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1bie7D-0005Lu-8j for freebsd-fs@freebsd.org; Sat, 10 Sep 2016 10:58:23 +0200 From: Christoph Pilka Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Server with 40 physical cores, 48 NVMe disks, feel free to test it Message-Id: Date: Sat, 10 Sep 2016 10:58:22 +0200 To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Sep 2016 08:58:27 -0000 Hi, we've just been granted a short-term loan of a server from Supermicro = with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of = mucking about, we managed to get 11-RC running. A couple of things are = preventing the system from being terribly useful: - We have to use hw.nvme.force_intx=3D1 for the server to boot If we don't, it panics around the 9th NVMe drive with "panic: couldn't = find an APIC vector for IRQ...". Increasing hw.nvme.min_cpus_per_ioq = brings it further, but it still panics later in the NVMe = enumeration/init. hw.nvme.per_cpu_io_queues=3D0 causes it to panic later = (I suspect during ixl init - the box has 4x10gb ethernet ports). - zfskern seems to be the limiting factor when doing ~40 parallel "dd = if=3D/dev/zer of=3D bs=3D1m" on a zpool stripe of all 48 drives. = Each drive shows ~30% utilization (gstat), I can do ~14GB/sec write and = 16 read. - direct writing to the NVMe devices (dd from /dev/zero) gives about = 550MB/sec and ~91% utilization per device=20 Obviously, the first item is the most troublesome. The rest is based on = entirely synthetic testing and may have little or no actual impact on = the server's usability or fitness for our purposes.=20 There is nothing but sshd running on the server, and if anyone wants to = play around you'll have IPMI access (remote kvm, virtual media, power) = and root. Any takers? //Chris=