From owner-freebsd-fs@freebsd.org Sun Apr 12 07:18:54 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8CC302B50F0 for ; Sun, 12 Apr 2020 07:18:54 +0000 (UTC) (envelope-from ipluta@wp.pl) Received: from mx4.wp.pl (mx4.wp.pl [212.77.101.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490NQj1qB0z3G1s for ; Sun, 12 Apr 2020 07:18:52 +0000 (UTC) (envelope-from ipluta@wp.pl) Received: (wp-smtpd smtp.wp.pl 11477 invoked from network); 12 Apr 2020 09:18:49 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wp.pl; s=1024a; t=1586675929; bh=Rt8BonfFGO142Q2qLTXQcKqa7rDH9tyGPVgEn7P5UiA=; h=Subject:To:Cc:From; b=WlYatngPlGSbe5TmT8E1oZ4Ub5k31hpGR5lHFrhZpaNRU3lq8SrKUOG0W0siyLk9u 4u1kh/btwUtuumM/3nFmlBq4dtOwQlUnb2LGOhdMOO4J3/mT79RpetMvy5xtyhSvin 1SVJH0BG74Qsm9Dltx/zs/8FV9YzMK+UtwWqlkAM= Received: from aazf8.neoplus.adsl.tpnet.pl (HELO [10.0.0.81]) (ipluta@wp.pl@[83.6.143.8]) (envelope-sender ) by smtp.wp.pl (WP-SMTPD) with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP for ; 12 Apr 2020 09:18:49 +0200 Subject: Re: ZFS server has gone crazy slow To: Peter Eriksson , freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> From: "Ireneusz Pluta/wp.pl" Message-ID: <708dabd7-838c-2b34-87dc-2151536fa8bc@wp.pl> Date: Sun, 12 Apr 2020 09:17:10 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: pl X-WP-MailID: 5eb99756f7a0d3df55abd89d53606ee7 X-WP-AV: skaner antywirusowy Poczty Wirtualnej Polski X-WP-SPAM: NO 0000000 [kUOE] X-Rspamd-Queue-Id: 490NQj1qB0z3G1s X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=wp.pl header.s=1024a header.b=WlYatngP; dmarc=pass (policy=none) header.from=wp.pl; spf=pass (mx1.freebsd.org: domain of ipluta@wp.pl designates 212.77.101.12 as permitted sender) smtp.mailfrom=ipluta@wp.pl X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[wp.pl:s=1024a]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:212.77.96.0/19]; FREEMAIL_FROM(0.00)[wp.pl]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; DWL_DNSWL_NONE(0.00)[wp.pl.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[wp.pl:+]; DMARC_POLICY_ALLOW(-0.50)[wp.pl,none]; RCVD_IN_DNSWL_NONE(0.00)[12.101.77.212.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[wp.pl]; ASN(0.00)[asn:12827, ipnet:212.77.101.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (-8.20), ipnet: 212.77.101.0/24(-3.98), asn: 12827(-2.87), country: PL(0.06)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 07:18:54 -0000 W dniu 2020-04-11 o 23:24, Peter Eriksson pisze: > (We have modified our rc.d startup scripts so we do the “zfs mount -a” part in the background so the rest of the system doesn’t have to wait for it (so we can get a ssh login prompt and login and check the progress of the filesystem mounts. (And then “nfs exports”, nfsd and samba also waits in the background for that to finish before starting up. A bit of a hack but it works (a real parallel server manager with dependencies would have been better would you share these modifications? > :-) From owner-freebsd-fs@freebsd.org Sun Apr 12 11:26:25 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 836882BAA8E for ; Sun, 12 Apr 2020 11:26:25 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 490TwJ30hyz410n for ; Sun, 12 Apr 2020 11:26:24 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f169.google.com with SMTP id m8so6158650lji.1 for ; Sun, 12 Apr 2020 04:26:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=LHjBG1R8im3w7L4yD/xOjxbruHa32JHNSxqhJevk/Js=; b=cj+jpsGc0ndf5zhwMW3BUB+E7nsGJn3Cf5wLerZXt+n4WKSuIJsGm05lWXoHLagwCt RmubE2CR4wuKNaW6Xx2SogyX9sevYeaP1bDPuMiRBOaVfW5ar654bXAszLcgJPxeHhrV jhEKmo3BDIupl54C1w8Su4Bdw3Ktgtz8Mx7QSfi1LpMUcsCL5R2rqAF0o7wDTjOZMmoZ pRY44u1FG6nRq6c9KU7MrG1bMobCHso1i0yKgoRdvdLEIIMhG/XmeIkpvEHNxpZ35zQM iMVVNB1qTtV8P+IRo69N2blhUMIxmrbe4cQgj4oumtz3nAPPF1ikMftmo7aP6RtF3JsG Oilw== X-Gm-Message-State: AGi0PubNULtSjWXnoKX9kyMnesDmSgLXMaUDlSVQeed8/VGahdYbURTI Kd18XO6egyeV8riYu0JZSJYZJc5HH6Q= X-Google-Smtp-Source: APiQypLecpK3WaL+evIoWvpYqowQPMUmWwvYvwpE+qvKsnsoqymnOC7UmiuBJHA8Bzu326jWknOVaA== X-Received: by 2002:a2e:a362:: with SMTP id i2mr7845052ljn.52.1586690781986; Sun, 12 Apr 2020 04:26:21 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id p23sm5755187lfc.95.2020.04.12.04.26.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Apr 2020 04:26:21 -0700 (PDT) Subject: Re: ZFS server has gone crazy slow To: Peter Eriksson , freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> Date: Sun, 12 Apr 2020 14:26:20 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 490TwJ30hyz410n X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.25 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[169.208.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.25)[ip: (-0.39), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[169.208.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:26:25 -0000 On 12/04/2020 00:24, Peter Eriksson wrote: > Another fun thing that might happen is if you reboot your server and happen > to have a lot of queued up writes in the ZIL (for example if you did a “zfs > destroy -d -r POOL@snapshots” (deferred(background) destroys of snapshots) > and do a hard reboot while it’s busy it will “write out” those queued > transactions at filesystem mount time during the boot sequence Just nitpicking on two bits of incorrect information here. First, zfs destroy never uses ZIL. Never. ZIL is used only for ZPL operations like file writes, renames, removes, etc. The things that you can do with Posix system calls (~ VFS KPI). Second, zfs destroy -d is not a background destroy. It is a deferred destroy. That means that either the destroy is done immediately if a snapshot has no holds which means no user holds and no clones. Or the destroy is postponed until holds are gone, that is, the last clone or the last user hold is removed. Note, however, that unless you have a very ancient pool version destroying a snapshot means that the snapshot object is removed and all blocks belonging to the snapshot are queued for freeing. Their actual freeing is done asynchronously ("in background") and can be spread over multiple TXG periods. That's done regardless of whether -d was used. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sun Apr 12 11:30:30 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AF0762BABC2 for ; Sun, 12 Apr 2020 11:30:30 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490V0z08yQz415D for ; Sun, 12 Apr 2020 11:30:26 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id B6C1F40010 for ; Sun, 12 Apr 2020 13:30:21 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 98ED940015; Sun, 12 Apr 2020 13:30:21 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842] (unknown [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id E736540010; Sun, 12 Apr 2020 13:30:19 +0200 (CEST) From: Peter Eriksson Message-Id: <92901912-F3AA-4C44-BF6E-3F4E0F996BFF@lysator.liu.se> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS server has gone crazy slow Date: Sun, 12 Apr 2020 13:30:17 +0200 In-Reply-To: <708dabd7-838c-2b34-87dc-2151536fa8bc@wp.pl> To: freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <708dabd7-838c-2b34-87dc-2151536fa8bc@wp.pl> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 490V0z08yQz415D X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-3.32 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-1.02)[ip: (-3.21), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.85), country: EU(-0.01)]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; FREEMAIL_CC(0.00)[wp.pl]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:30:30 -0000 Sure You can find my stuff at my Github page: https://github.com/ptrrkssn/freebsd-stuff = The things needed are the /etc/rc.d scripts (zfs, nfsd, samba_server, = cold) scripts in =E2=80=9Crc.d=E2=80=9D and the =E2=80=9Czfs-mountall=E2=80= =9D in =E2=80=9Cscripts=E2=80=9D that goes into /sbin. What they basically do are use a helper function =E2=80=9Cwaitfor=E2=80=9D= to wait for a =E2=80=9C${name}.done=E2=80=9D file to appear in /var/run = and then just=20 (waitfor zfs ; run_rc_command =E2=80=9C$1=E2=80=9D) & Instead of just the =E2=80=9Crun_rc_command =E2=80=9C$1=E2=80=9D. And = then we have the scripts create those =E2=80=9C${name}.done=E2=80=9D = files when they have started their daemons. Not foolproof (should check that the daemon really is successfully = running and more before signalling =E2=80=9Cdone=E2=80=9D). scripts/zfs-mountall=20 Mounts all filesystems under a specific DATASET sequentially. Basically = what a =E2=80=9Czfs mount -r=E2=80=9D would have done if that option had = existed. rc.d/zfs Executes /sbin/zfs-mountall to make sure all filesystems in the = =E2=80=98zroot=E2=80=99 (or whatever dataset / is mounted from) is = mounted first. This is needed to make sure all important filesystems are = mounted (like various /var filesystems etc). Then basically does: ( rm -f /var/run/zfs*.done zfs mount -va >/var/log/zfs/mount.log && touch = /var/run/zfs-mount.done zfs share -a >/var/log/zfs/share.log && touch /var/run/zfs.done ) & rc.d/mountd, rc.d/nfsd, rc.d/ctld & rc.d/samba_server all contains a = function: waitfor() { FIRST=3Dyes while [ ! -f "/var/run/$1.done" ]; do if [ $FIRST =3D yes ]; then echo "${name}: Waiting for '$1' to finish..." FIRST=3Dno fi sleep 1 done } It probably should go into /etc/rc.subr or somewhere else but=E2=80=A6 = =E2=80=9Cwaitfor=E2=80=9D basically is loop and waits for the = =E2=80=9C${name}.done=E2=80=9D file to appear in /var/run. rc.d/mountd then ends with this instead of just: =E2=80=98 = run_rc_command =E2=80=9C$1=E2=80=9D =E2=80=98 case "$1" in faststart) rm -f "/var/run/${name}.done" (waitfor zfs ; run_rc_command "$1" ; touch = "/var/run/${name}.done") & ;; *) run_rc_command "$1" ;; esac rc.d/nfsd ends with: case "$1" in faststart) (waitfor "mountd" ; run_rc_command "$1") & ;; *) run_rc_command "$1" ;; esac (And similar for cold) And rc.d/samba_server has a bit more modifications (we only want = =E2=80=9Csmbd=E2=80=9D to delay it=E2=80=99s startup, not winbindd or = numbed) so does this in the middle of the script. Also just waits for = the =E2=80=9Czfs mount -a=E2=80=9D to be finished, doesn=E2=80=99t have = to wait for =E2=80=9Czfs share -a=E2=80=9D (NFS)): # Daemon should be enabled and running = =20 if ( [ -n "${rcvar}" ] && checkyesno "${rcvar}" ) || [ -n = "$force_run" ]; then if [ "${rc_arg}" =3D "start" -a "${name}" =3D "smbd" ]; then (waitfor "zfs-mount" ; run_rc_command = "${_rc_prefix}${rc_arg}" ${rc_extra_args}) & else run_rc_command "${_rc_prefix}${rc_arg}" ${rc_extra_args} # If any of the commands failed, take it as a global = result =20 result=3D$((${result} || $?)) fi fi =20 - Peter > On 12 Apr 2020, at 09:17, Ireneusz Pluta/wp.pl wrote: >=20 > W dniu 2020-04-11 o 23:24, Peter Eriksson pisze: >> (We have modified our rc.d startup scripts so we do the =E2=80=9Czfs = mount -a=E2=80=9D part in the background so the rest of the system = doesn=E2=80=99t have to wait for it (so we can get a ssh login prompt = and login and check the progress of the filesystem mounts. (And then = =E2=80=9Cnfs exports=E2=80=9D, nfsd and samba also waits in the = background for that to finish before starting up. A bit of a hack but it = works (a real parallel server manager with dependencies would have been = better > would you share these modifications? >=20 >> :-) From owner-freebsd-fs@freebsd.org Sun Apr 12 11:46:33 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46D652BB1B5 for ; Sun, 12 Apr 2020 11:46:33 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490VMY0Fn6z41pJ; Sun, 12 Apr 2020 11:46:32 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 6A98A40010; Sun, 12 Apr 2020 13:46:29 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 5712840015; Sun, 12 Apr 2020 13:46:29 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842] (unknown [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id D156E40010; Sun, 12 Apr 2020 13:46:28 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS server has gone crazy slow From: Peter Eriksson In-Reply-To: <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> Date: Sun, 12 Apr 2020 13:46:28 +0200 Cc: Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: <747B75C0-73D7-42B2-9910-9E16FCAE23C4@lysator.liu.se> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> To: freebsd-fs X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 490VMY0Fn6z41pJ X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:46:33 -0000 You are probably right.=20 However - we have seen (thru experimentation :-) that =E2=80=9Czfs = destroy -d=E2=80=9D for recursive snapshot destruction on many = filesystems (recursively) seemed to allow it to be done much faster (Ie = the command finished much quicker) on our servers. But it also meant = that a lot of I/O seemed to be happening quite some time after the last = =E2=80=9Czfs destroy -d=E2=80=9D command was issued (and a really long = time when there were near-quota-full filesystems). No clones or =E2=80=9Cu= ser holds=E2=80=9D in use here as far as I know. Why that is I don=E2=80=99= t know. With =E2=80=9Czfs destroy=E2=80=9D (no =E2=80=9C-d=E2=80=9D) = things seems to be much more synchronous. We=E2=80=99ve stopped using =E2=80=9C-d=E2=80=9D now since we=E2=80=99d = rather not have that type of I/O load be happening during daytime and we = had some issues with some nightly snapshot cleanup jobs not finishing in = time. Anyway, the =E2=80=9Cseems to be writing out a lot of queued up ZIL = data=E2=80=9D at =E2=80=9Czfs mount -a=E2=80=9D time was definitely a = real problem - it mounted most of the filesystems pretty quickly but = then was =E2=80=9Cextremely slow=E2=80=9D for a couple of them (and was = causing a lot of I/O). Like 4-6 hours. Luckily that one was one of our = backup servers and during a time when the only one it frustrated was = me=E2=80=A6 I=E2=80=99d hate that to happen for one of the frontend = (NFS/SMB-serving) servers during office hours :-) - Peter > On 12 Apr 2020, at 13:26, Andriy Gapon wrote: >=20 >=20 > On 12/04/2020 00:24, Peter Eriksson wrote: >> Another fun thing that might happen is if you reboot your server and = happen >> to have a lot of queued up writes in the ZIL (for example if you did = a =E2=80=9Czfs >> destroy -d -r POOL@snapshots=E2=80=9D (deferred(background) destroys = of snapshots) >> and do a hard reboot while it=E2=80=99s busy it will =E2=80=9Cwrite = out=E2=80=9D those queued >> transactions at filesystem mount time during the boot sequence >=20 > Just nitpicking on two bits of incorrect information here. > First, zfs destroy never uses ZIL. Never. ZIL is used only for ZPL = operations > like file writes, renames, removes, etc. The things that you can do = with Posix > system calls (~ VFS KPI). >=20 > Second, zfs destroy -d is not a background destroy. It is a deferred = destroy. > That means that either the destroy is done immediately if a snapshot = has no > holds which means no user holds and no clones. Or the destroy is = postponed > until holds are gone, that is, the last clone or the last user hold is = removed. >=20 > Note, however, that unless you have a very ancient pool version = destroying a > snapshot means that the snapshot object is removed and all blocks = belonging to > the snapshot are queued for freeing. Their actual freeing is done > asynchronously ("in background") and can be spread over multiple TXG = periods. > That's done regardless of whether -d was used. >=20 > --=20 > Andriy Gapon From owner-freebsd-fs@freebsd.org Sun Apr 12 11:54:19 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AEFEF2BB43B for ; Sun, 12 Apr 2020 11:54:19 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490VXW3jMHz427x; Sun, 12 Apr 2020 11:54:19 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id A33F640010; Sun, 12 Apr 2020 13:54:17 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 9141E40015; Sun, 12 Apr 2020 13:54:17 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842] (unknown [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 3D1C440010; Sun, 12 Apr 2020 13:54:17 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS server has gone crazy slow From: Peter Eriksson In-Reply-To: <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> Date: Sun, 12 Apr 2020 13:54:16 +0200 Cc: Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> To: freebsd-fs X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 490VXW3jMHz427x X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:54:19 -0000 Hmm. By =E2=80=9Cancient=E2=80=9D pool version - are you thinking of the = one used during FreeBSD 11.0-11.1 days? That was what we were using at = the time when we tested =E2=80=9Czfs destroy -r foo@bar=E2=80=9D vs = =E2=80=9Czfs destroy -rd foo@bar=E2=80=9D and found that =E2=80=9C-d=E2=80= =9D was much quicker. (We=E2=80=99re now at 11.3 and 12.1 and that whole snapshot cleaning = script is completely rewritten and uses a custom =E2=80=9Czfs=E2=80=9D = binary instead that can look at custom user properties to decided which = snapshots to delete and not and is much more efficient in general.) - Peter > Note, however, that unless you have a very ancient pool version = destroying a > snapshot means that the snapshot object is removed and all blocks = belonging to > the snapshot are queued for freeing. Their actual freeing is done > asynchronously ("in background") and can be spread over multiple TXG = periods. > That's done regardless of whether -d was used. >=20 > --=20 > Andriy Gapon From owner-freebsd-fs@freebsd.org Sun Apr 12 11:56:06 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 81E4D2BB4E7 for ; Sun, 12 Apr 2020 11:56:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 490VZY30kYz42CC for ; Sun, 12 Apr 2020 11:56:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f172.google.com with SMTP id q22so6203076ljg.0 for ; Sun, 12 Apr 2020 04:56:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=vGjbMGb0BuR4s0Sf5F+7P44R3s4CFVlXZc0CufS3/XA=; b=i/Tv1nMUUzcILDfwE7oW2+o8K5PvJ7BJbLwdlMNMg0kAYfmuulJDB7JTGlPla5oWGm 34alc+05zeHHzV7HFEvF5XfhoWgUQ/OvkKGptF+Lpd7ffr9BTFbczkezYSzCy0Ja7C9c iyEjz6kn3R1RcVeHG/GZXE6FYKV4xpimGp0bubINttpBbeIQ98Lu7N2/QsgoJcKFfik4 vzG3R3vZzLis+kFUG9bjMz4nL8I2tMtWHilXnBAUKm7XehPlOM6PvCpUCU9JPVWo2Ndm ttAOrx1iVUyg6yp24uH8R7c0nnzlGf7+4It4ZkY3uUyymD7/fS9RJj4HJzyYX4JuDggD i6Qg== X-Gm-Message-State: AGi0PubGpSR/NJfTZevRfFh6z4ZcMICbnML8ydrjF/HiEvsY5Ph+OwEs jIkdI1/heivXV2Ww/J5RF8mzfvS9zVs= X-Google-Smtp-Source: APiQypJQUh9qY04oOhJaE6F63WQJ+GW8taTI+suESxi+15Y5llBu57jewSyTj6+tHAgZWXlknYNgRA== X-Received: by 2002:a2e:4942:: with SMTP id b2mr8095794ljd.135.1586692563167; Sun, 12 Apr 2020 04:56:03 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id l12sm2783190lfp.35.2020.04.12.04.56.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Apr 2020 04:56:02 -0700 (PDT) Subject: Re: ZFS server has gone crazy slow To: Peter Eriksson , freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> <747B75C0-73D7-42B2-9910-9E16FCAE23C4@lysator.liu.se> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <2f49fe01-50dc-2e72-8cd6-beb6f0fbb621@FreeBSD.org> Date: Sun, 12 Apr 2020 14:56:01 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <747B75C0-73D7-42B2-9910-9E16FCAE23C4@lysator.liu.se> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 490VZY30kYz42CC X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.40 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[172.208.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.40)[ip: (-1.13), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[172.208.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:56:06 -0000 On 12/04/2020 14:46, Peter Eriksson wrote: > You are probably right. > > However - we have seen (thru experimentation :-) that “zfs destroy -d” for > recursive snapshot destruction on many filesystems (recursively) seemed to > allow it to be done much faster (Ie the command finished much quicker) on our > servers. But it also meant that a lot of I/O seemed to be happening quite > some time after the last “zfs destroy -d” command was issued (and a really > long time when there were near-quota-full filesystems). No clones or “user > holds” in use here as far as I know. Why that is I don’t know. With “zfs > destroy” (no “-d”) things seems to be much more synchronous. > > We’ve stopped using “-d” now since we’d rather not have that type of I/O load > be happening during daytime and we had some issues with some nightly snapshot > cleanup jobs not finishing in time. I think that you want to re-test zfs destroy vs destroy -d and do that rigorously. I am not sure how to explain what you saw as it cannot be explained by how destroy -d actually differs from plain destroy. Maybe it it was a cold vs warmed up (with respect to the ARC) system, maybe something else, but certainly not -d if you do not have user holds and clones. Just in case, here is the only place in the code where 'defer' actually makes difference. dsl_destroy_snapshot_sync_impl: if (defer && (ds->ds_userrefs > 0 || dsl_dataset_phys(ds)->ds_num_children > 1)) { ASSERT(spa_version(dp->dp_spa) >= SPA_VERSION_USERREFS); dmu_buf_will_dirty(ds->ds_dbuf, tx); dsl_dataset_phys(ds)->ds_flags |= DS_FLAG_DEFER_DESTROY; spa_history_log_internal_ds(ds, "defer_destroy", tx, ""); return; } > Anyway, the “seems to be writing out a lot of queued up ZIL data” at “zfs > mount -a” time was definitely a real problem - it mounted most of the > filesystems pretty quickly but then was “extremely slow” for a couple of them > (and was causing a lot of I/O). Like 4-6 hours. Luckily that one was one of > our backup servers and during a time when the only one it frustrated was me… > I’d hate that to happen for one of the frontend (NFS/SMB-serving) servers > during office hours :-) I don't doubt that, I just tried to explain that whatever was in ZIL could not come from zfs destroy. It was something else. >> On 12 Apr 2020, at 13:26, Andriy Gapon wrote: >> >> >> On 12/04/2020 00:24, Peter Eriksson wrote: >>> Another fun thing that might happen is if you reboot your server and >>> happen to have a lot of queued up writes in the ZIL (for example if you >>> did a “zfs destroy -d -r POOL@snapshots” (deferred(background) destroys >>> of snapshots) and do a hard reboot while it’s busy it will “write out” >>> those queued transactions at filesystem mount time during the boot >>> sequence >> >> Just nitpicking on two bits of incorrect information here. First, zfs >> destroy never uses ZIL. Never. ZIL is used only for ZPL operations like >> file writes, renames, removes, etc. The things that you can do with Posix >> system calls (~ VFS KPI). >> >> Second, zfs destroy -d is not a background destroy. It is a deferred >> destroy. That means that either the destroy is done immediately if a >> snapshot has no holds which means no user holds and no clones. Or the >> destroy is postponed until holds are gone, that is, the last clone or the >> last user hold is removed. >> >> Note, however, that unless you have a very ancient pool version destroying >> a snapshot means that the snapshot object is removed and all blocks >> belonging to the snapshot are queued for freeing. Their actual freeing is >> done asynchronously ("in background") and can be spread over multiple TXG >> periods. That's done regardless of whether -d was used. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sun Apr 12 11:57:44 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 72BFF2BB6A6 for ; Sun, 12 Apr 2020 11:57:44 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 490VcR2dzTz42Lp for ; Sun, 12 Apr 2020 11:57:43 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f179.google.com with SMTP id u15so1644911ljd.3 for ; Sun, 12 Apr 2020 04:57:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=j0ERp1XqF78bm24k0AdTU/F5oMpp+AKsx6tgI6BTlZ4=; b=jq9Oqh5CHVykXo6MbPHh1UUewHagC1RjYHAJSgAOXWs6x9Q8ceaKizU8q2TlxETpCH Es8iZYJCaPMFEb96M7UsF3MdYLKIIA+Qiq7eSqq75jmLdpHa+UB8WjzlQN4/QoOrHO7A hu/ucZwnt9o28WiljqEyMeQjdbYaoq5x8HkZ/xCrdK5zGYz4/kvv3/Mg7BJfq3N4kejr PZKyoys8AeJP7Zial4LHx+/iz2U6BrcaysaM8OO+QayEFs+5RuYLQJz5sdoEKBByXBYL WX/o/xxuY0s7YaZTuckbkcxNXP5bqLgZKd3Hj0ALvYzDq7ctTNhhrJ/PSr6Ihm2bhfKp Oaaw== X-Gm-Message-State: AGi0PuYypwIVIkOR6wYNYZ40QaAtXuqlqlkbqEaaBMBtNX7PounsAILW hrXzDFl1yDV5g4YMzxKgAnIMBjfrn2Y= X-Google-Smtp-Source: APiQypI+4pxv0/mH4xs3AbmX6gvX4gx/Y53VG60aWKztz+fbQ2REpMiSxSbOhLxubC2R3hizaQRwyw== X-Received: by 2002:a2e:b888:: with SMTP id r8mr8068150ljp.128.1586692661371; Sun, 12 Apr 2020 04:57:41 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id a3sm5874579lfi.5.2020.04.12.04.57.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Apr 2020 04:57:40 -0700 (PDT) Subject: Re: ZFS server has gone crazy slow To: Peter Eriksson , freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Sun, 12 Apr 2020 14:57:39 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 490VcR2dzTz42Lp X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.179 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.40 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[179.208.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.40)[ip: (-1.14), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:57:44 -0000 On 12/04/2020 14:54, Peter Eriksson wrote: > Hmm. By “ancient” pool version - are you thinking of the one used during > FreeBSD 11.0-11.1 days? That was what we were using at the time when we > tested “zfs destroy -r foo@bar” vs “zfs destroy -rd foo@bar” and found that > “-d” was much quicker. > > (We’re now at 11.3 and 12.1 and that whole snapshot cleaning script is > completely rewritten and uses a custom “zfs” binary instead that can look at > custom user properties to decided which snapshots to delete and not and is > much more efficient in general.) I think even earlier. Before v26. >> Note, however, that unless you have a very ancient pool version destroying >> a snapshot means that the snapshot object is removed and all blocks >> belonging to the snapshot are queued for freeing. Their actual freeing is >> done asynchronously ("in background") and can be spread over multiple TXG >> periods. That's done regardless of whether -d was used. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Sun Apr 12 19:26:44 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BDACF2C68AB for ; Sun, 12 Apr 2020 19:26:44 +0000 (UTC) (envelope-from daryl@isletech.net) Received: from mail.isletech.net (mail.isletech.net [IPv6:2001:470:1d:c44::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490hZW6QKgz4XkG for ; Sun, 12 Apr 2020 19:26:43 +0000 (UTC) (envelope-from daryl@isletech.net) From: Daryl Richards Subject: zpool status question To: freebsd-fs Message-ID: <7582e197-f18f-7137-fe9d-a21aabc4c989@isletech.net> Date: Sun, 12 Apr 2020 15:26:42 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 490hZW6QKgz4XkG X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(0.00)[isletech.net:s=isle2017]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[isletech.net:-]; DMARC_POLICY_ALLOW(0.00)[isletech.net,quarantine]; DMARC_POLICY_ALLOW_WITH_FAILURES(-0.50)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-1.66)[ipnet: 2001:470::/32(-4.66), asn: 6939(-3.60), country: US(-0.05)]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 19:26:44 -0000 Hello list, I recently needed to add more storage to my pool and to get around the ashift problem of my old pool I made a new one on new drives and did a send->receive to copy my data. Everything went fine with the copy, but now when I do a zpool status, it takes about 15 minutes to show the status of the new pool. It takes about 5 minutes to show the first line (name of pool, then top level vdev), and then about 3 minutes per device after that. (4 drive raidz1). Everything shows ONLINE when done, no errors or timeouts show up in dmesg, nothing. It just takes a long time to display status.. Getting status of the other pools (zroot, and old array) is instantaneous. I'm just being overly cautious before I export the old array and import the new one as my main one.. I'm on FreeBSD 11.3-RELEASE Any ideas? Thanks! From owner-freebsd-fs@freebsd.org Wed Apr 15 06:03:15 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 23E032AC2E8 for ; Wed, 15 Apr 2020 06:03:15 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "anubis.delphij.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492Bc16C0Jz4KnW for ; Wed, 15 Apr 2020 06:03:13 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from odin.corp.delphij.net (unknown [IPv6:2601:646:8600:58ba:e8a7:7c50:1605:edb4]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 566B848846; Tue, 14 Apr 2020 23:03:12 -0700 (PDT) To: freebsd-fs Reply-To: d@delphij.net From: Xin Li Autocrypt: addr=delphij@delphij.net; prefer-encrypt=mutual; keydata= mQINBFuSR4oBEACvvEgwRIHs6IcSP/yaDtySF78Ji3rP29qdiQsxhMsOtvtffdbS56VApIWO UFb3/iN2gA8HwLvrmjijN0HEoLVX7na1WARmxRYzQMtApsZIUTtx7hnUYlsi2F5odZa6CDW9 a954DLRzYxiUwYDcu5Zjl9bglK1H8e/N9uC0Vuigr4teWfh86brzOyf819QzwFVYfMIK4ihw QGwMvTzbyVuCFy+LENkmcVYni70oQy6rZ5ktSuYbuOFvu7inRRfhSWPHziV7k+bW88sJ7xhv lBlegcnhkSudWX2M8tZ3MO1PJOcyys0CJlsBY5Weiog2lIPi05h/E9pZ9mc1Vud17iqDaL6w RaggOUhuPfDGCdO5ro82W4BZGeQMRnRF5Ntk+t2ShIH4nn3xRLV0E5nziCiKlgiMqOrz/ZTL QTVbHrCuiwD+fSK14y0oHbkOLYTYLlgh1JbwfY2Ty7elOYiWzyeJ7sJh2dF91NSEneWIOys3 mBpuvtU3nSzzTvAB48VV+Nbg1CpIOgNlPjj7uhIum/Z/VjUaJEyaLpTIRh0MVJVcbP7hXSqZ NA35EEZZVnWEOYdycm4CmEdeNPWkrAf2Ya77iR5VLGypwMlsUMQPh+sKVWDD38M8stFGBBNm d01Hi74Bsq5hKan654dOqMt5eYklrVj0ucMzFQtus7oE502UswARAQABtBxYaW4gTEkgPGRl bHBoaWpAZGVscGhpai5uZXQ+iQJUBBMBCgA+FiEEceNg5NEMZIki80nQQHl/fJX0g08FAluS R/YCGwMFCQmuhAAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQQHl/fJX0g0+2Og//bWpE F2V5/M5l6YW1T8oLcT9rIOH6oq9M0LMNRgFeiNNnilGIeeIgtOGBRueG4CZiZAvsRPJkrO70 1R2SrdkCIvwGUzUAxx1NfBWb+vgm4fgkW/MotGonceM5v0qfSKKXasWvDctkK28aG+IoQzmi FjXNW4+ju4zeQFYwD4ZDWqw9MqO0hVb24uW3dxtQhbfmOLgJ/PEDMQaFuANbW1c+iR0BQA3D Go/EeMY4kpN8on6Aqt/S/4JVltudfQ9OXdjQsC7netSaB9K3mHGt9aKAAB7RzlRY00DKkYS/ /eQwLzGPmK7yX13M68mMDjBs6mIR8t/E1S5OdBNhHRPNPlEbwugR4KaiCsN5yqzJoSV99fKY z2VyxjWPaG8yhHE+jmKUgIBKTfFUQEfkriQR4EASoeJ+soaMTiFDBij1Zw5n3ndLRFMB1ZCl fZLER36mAgW4m4kP83TWnDiJLxOxSOxifV8HpTFjff902H85cybg9KMwrfPDr6W19GGk5Vo1 fkza5krRMGbKWb7+74Evusi0ZxJLIOFwp5Y8eVqUMZaAD3f1ZX1M3pgXOp20QgAy+2KvMHij rLa4q+tMGRzYYD1BnFVSVdXAX5VOoTmHBcDz67DkuRwk2Byp1sgd407oEOmSwrNJlKS0TPCm xUJ2fdSQF+1/MMSRfee49vtMvz7cOrC5Ag0EW5JHigEQANiBmIFAfRNH3nzYNWC0yC+tfx3z sUwAsH1VaBM/cTib+yKtbBOSIlXWjJZWX3MHwoI/1LeGghB2mxkkX1L0pJ/vj1eXNR+sFZ32 0pYcl61Fxg/5fioG4QDTM4i3i7NR5PxDnc6UVaynSlII93DedRhZ1ROtdn4vyMgzsDiqhbL7 BthDOt5KxjqdRk4qRPSw7BovEqZLOcG5IJtf/zZUzRbM7SBljEbOAfekDGx1Br+RrYSD7/Ef Pwwzou9T8315IpBpIHyQF/dZNk3iFiB9Ed5CA71ZRYV5YoLWE9lL0j9kxOLQ5vHnX3mVq7QZ Bc7nzwZ6UhQgYmrG5+RWvuiPpGwvDRIsugJUGXucYkAQh5kuNblmkwpv6u9rNMjCNbzAylOa qdogra5EW+RUSbRz0b4iIr8nnZeAlh7BihCe7JjOwbDjoBEEEtSfVc4hD/LENqpcYVrChphf aOLB9YIXhnVDTVvMc9OklWT/81HzAaDQqOQCzEfY92199Ct9/CwRoQ2OpO8TO5+8A7b9Nb33 nmxMn09mb48ruRacMrfHxCWbgU4w9SEfbip4GcS5wGG6yTC+hw55Iwnnwus40NrJ0GEr8a4r cdsLbkvlyoNHB8ZGgyJ4aFCQ1V4qE1BnlTk7Z8BYBUkJM1odPSkVvHpCnMUjVpJ3hEOC+73Z YH1dh7lZABEBAAGJAjwEGAEKACYWIQRx42Dk0QxkiSLzSdBAeX98lfSDTwUCW5JHigIbDAUJ Ca6EAAAKCRBAeX98lfSDTz8DEACMh3poeUb+gWNF4RWFZuLteZVo0+E1JLYXQkmtrRBLXviP +Qy0pXyFAVxLM4hNIBoIDYfK9BcwrBYf7AwSKrH0GiNwFpgHCkbZd6qoZy2gB+adTnCpVCTJ KJetsH/8awkrChJWMK0ckGf3EeWMPvawG7kW7FBz70NYEZ0pOMiaEZNVtzD3wwbYWUiDFYth 83XGglOExg+1ShTW5XjQPRrdyJAO+aUW4o3lVjfyUJXMgI4rmhMiLVm06GuNrbpKIF0s+4Vd jQAjhrDQjfoXi9CkfsA/cONseuHNv1JGj3RqHiqHJq1dbrpodXp925zGDAnUGxCOBPoFopAH gVzR89GTut059GpwqsddZmU6y7rqifuam/ekJ+QRwc16vgt7pHqCrTY8WPxRZr2UpFU1wlTo COdeiFep1gq1F9jzFjJnoMaAdmC6k7bgAA+RQusOgIhJL0jIej7DoAHxmxFFCfRy+lDtpXwF gQ8HMvzHI65QWmQnMo7s6SQH/ZH5s1yR6SJq8+3lDz+dCuT42qJVqIPVvxd10LW0FNN+t7HF eLadU6ekSgD13/EYMYXlvNHkw7dAItSDxIzgRyykLz0bCU9xwNWoS4Z43+ifF9anJ+uR0ltW El1j++h6ZrD3LLuCgJIt1so0m49GzdcSpOI7LCwMlacyvafiEyjUn+tSNDsnfw== Subject: zpool question -- resilvering doesn't fully check on-disk data for corruption? Message-ID: Date: Tue, 14 Apr 2020 23:03:08 -0700 User-Agent: Thunderbird MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="5at5vN4LOJzYAeJKxIDmIji6TLzmlVUQ9" X-Rspamd-Queue-Id: 492Bc16C0Jz4KnW X-Spamd-Bar: ------- X-Spamd-Result: default: False [-7.73 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[d@delphij.net]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[delphij.net:+]; DMARC_POLICY_ALLOW(-0.50)[delphij.net,reject]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-3.64)[ip: (-9.91), ipnet: 2001:470::/32(-4.66), asn: 6939(-3.60), country: US(-0.05)]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[delphij.net:s=m7e2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 06:03:15 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --5at5vN4LOJzYAeJKxIDmIji6TLzmlVUQ9 Content-Type: multipart/mixed; boundary="qEs9RJ3lMWSyQCZXanM756zZTqWxxBQni"; protected-headers="v1" From: Xin Li Reply-To: d@delphij.net To: freebsd-fs Message-ID: Subject: zpool question -- resilvering doesn't fully check on-disk data for corruption? --qEs9RJ3lMWSyQCZXanM756zZTqWxxBQni Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi, I have recently seen a bad drive on my home storage server. The bad drive had some timeouts occasionally that would cause the CAM subsystem to kick it off eventually, like: (ada1:ahcich11:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada1:ahcich11:0:0:0): CAM status: Command timeout (ada1:ahcich11:0:0:0): Retrying command, 0 more tries remain ada1 at ahcich11 bus 0 scbus11 target 0 lun 0 ada1: s/n WD-WMC4E0090978 detached (ada1:ahcich11:0:0:0): Periph destroyed When this happens, a full 'camcontrol reset all' and 'camcontrol rescan all' would bring it back, and ZFS would correctly start a resilvering process as expected. After the resilvering, zpool would detect several checksum errors (also expected). As a precautional measure, I usually would start another zpool scrub to check data integration again when this happens. To my surprise, in the last few times when that drive was timing out, the zpool scrub would also find some checksum errors and correct these (the drive is in a RAID-Z pool). A second run of 'zpool scrub' after that would no longer be able to find any checksum errors. I initially thought that is probably because there were some bad blocks on the bad hard drive and didn't pay much attention as I already ordered a new hard drive to replace it, but when the new drive arrived, I have initiated a 'zpool replace' with both bad and new drive attached (which will start a resilver too; I didn't perform a zpool scrub the last time when the timeout happens because the scrub was very slow and I feared that I might end up causing more damage to the bad drive before the new drive arrived). When the new drive arrived, however, to my surprise, the zpool scrub after the replacement resilver have detected new checksum errors on the newly attached drive. Is this expected? (My understanding is that both resilver and scrub would read all data from a RAID-Z pool, therefore checking checksums for all blocks, and for replacing, so checksum errors shouldn't really happen for the new drive, because the written data was already checksummed? The system is equipped with ECC RAM, etc.; I know there is a possibility that the disk controller or the disk itself may still introduce bit flips, etc. if I'm really unlucky, but if that's the case I think I should have seen errors more often...) Cheers, --qEs9RJ3lMWSyQCZXanM756zZTqWxxBQni-- --5at5vN4LOJzYAeJKxIDmIji6TLzmlVUQ9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.2.20 (Darwin) iQIzBAEBCgAdFiEEceNg5NEMZIki80nQQHl/fJX0g08FAl6Wo5wACgkQQHl/fJX0 g0/b9w//U0Rv/w30CCdAqdH97RCZ0bnRBXJlq+XOFc2+eNymXcE6mIY29IYLbuPI h0dozLeNZGPjm7iRUhBibhRSwdK5/wD5E1+AVoBeUUyTv9bCEf1flORkZiz/zDMB AZ3cWotX6udqmOQFJn2Cu+cMft6NauEM3WYOnFT1BMmdReetgcGGY0WX1Pheoq8g 3RXZyecMm448vCOU3Syw78nTbSH6YPv0aTDr5GiSH5MD79E7EyzSLr/vhWmUe78G 9ec7NsDZhy/W8SF4KBTnas3N+wuAOlSHy3AGnLTzqLvhIUIFOxYi+UxxBtngz4v4 NjI8u9pURis4TF/NM86dZDBAoWaAQiAtuC7knGPc6ITmelveuIobWos73F8F3w81 NLguXAbC4ssN91MjYvAxhnv68u3nrzKWRF6QmAPPO7pEaQgMlKS0akbg1LrbQT6m ARheG9XTAM2hR9wsSN0yN07EOOFGkXraXC2asTBtZ5vtaOKDir/DdytTYB5yVB38 opdEJlZ6ClwvNjGP8+zHBeh4bHMXsMeD9mrymiNfAkZL9+R0ocqwwa/LS5c9+jTI HubDh79nPUOC3G6sRztVGxnP+zBHMucjgyY/SlhzncdULepkXLO7uOnT2i1SpxCS dXcefYjBowMzIvsHQAChO4ijPC1xnlEvkBAvv/cl0/JqE+ILBSA= =r7Jw -----END PGP SIGNATURE----- --5at5vN4LOJzYAeJKxIDmIji6TLzmlVUQ9-- From owner-freebsd-fs@freebsd.org Wed Apr 15 06:28:06 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A0A852ACCEB for ; Wed, 15 Apr 2020 06:28:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492C8j6VnSz4MBw for ; Wed, 15 Apr 2020 06:28:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f177.google.com with SMTP id q22so2446437ljg.0 for ; Tue, 14 Apr 2020 23:28:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=1ab98qNtGxzc3kVttZjXqAqY7uZvVGsV4HK2CgF8knY=; b=CkqW5pA0Jo99CvwjYyj1GJVhZ4q2SlEro6zb9dqKgyqUByahEm4SXA3jx4Uybh/Gxk 9yK7GnXPINCxZWM/nZ8iOraXHjzj19As55kdWYwBInzp3e8CKmVE2ryOoVKBKSpNszFs 0+utOw6QHjONt5VZM3R/a9UJB4oSp7JhpuoOR8Fv/jyWA73M5FcKYeC/FZDvE063s/Mf P79nMePKyLZrLtqtPhye0HFd81tgPFTD8AAUpCP7WSOLgg39IJcUnyrVI6Tb/68/VKSp nI5rKS+aM+4fLejYoVCx2V0k3EIlZQHapAwtQM9sMG/73JVis1uG4QsD52ow7dYBqnCU QsqQ== X-Gm-Message-State: AGi0PuYZBLkBHQ2cD+AhjYUrV4h9nYQDrCaXLXqQJff49w/Ads5wispC RpKDzLdp90iE5Ox6e/Gg1LwyCnpmLQ4= X-Google-Smtp-Source: APiQypLi/TV3OfQZ6lcfW3bUTrf4SDCeprE6m2RoAoktvNZqNUTXEajMKPwEGPO/WdoaCsXEvt2xRQ== X-Received: by 2002:a2e:593:: with SMTP id 141mr2129095ljf.271.1586932084040; Tue, 14 Apr 2020 23:28:04 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id t19sm11575162lfl.53.2020.04.14.23.28.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 14 Apr 2020 23:28:03 -0700 (PDT) Subject: Re: zpool question -- resilvering doesn't fully check on-disk data for corruption? To: d@delphij.net, freebsd-fs References: From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <9173c2f9-57be-b390-1941-0673ebc5ba9a@FreeBSD.org> Date: Wed, 15 Apr 2020 09:28:02 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 492C8j6VnSz4MBw X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.177 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-1.37 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; SUBJECT_ENDS_QUESTION(1.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[177.208.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.37)[ip: (-0.99), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; RWL_MAILSPIKE_POSSIBLE(0.00)[177.208.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 06:28:06 -0000 On 15/04/2020 09:03, Xin Li via freebsd-fs wrote: > My understanding is that both resilver and scrub > would read all data from a RAID-Z pool Resilver differs from scrub in that it checks only blocks corresponding to periods in dirty time log (DTL). That is, if a disks becomes unavailable and then comes back, a resilver will read only those blocks on other disks that are necessary to write any missing blocks on the flaky disk. If a disk gets replaced then a resilver reads all blocks on remaining disks that are necessary to re-create data on the replaced disk. A scrub always reads all used blocks. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Wed Apr 15 09:30:27 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CF1882B0AB4 for ; Wed, 15 Apr 2020 09:30:27 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 492HC62y3vz4Xwx for ; Wed, 15 Apr 2020 09:30:26 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Date: Wed, 15 Apr 2020 11:30:24 +0200 (CEST) From: Ronald Klop To: Daryl Richards Cc: freebsd-fs Message-ID: <1045079731.1.1586943024582@localhost> In-Reply-To: <7582e197-f18f-7137-fe9d-a21aabc4c989@isletech.net> References: <7582e197-f18f-7137-fe9d-a21aabc4c989@isletech.net> Subject: Re: zpool status question MIME-Version: 1.0 X-Mailer: Realworks (503.2641.c9b12d36d36) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Queue-Id: 492HC62y3vz4Xwx X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ronald-lists@klop.ws designates 194.109.157.24 as permitted sender) smtp.mailfrom=ronald-lists@klop.ws X-Spamd-Result: default: False [-0.69 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.888,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; NEURAL_HAM_LONG(-0.97)[-0.969,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[klop.ws]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[24.157.109.194.list.dnswl.org : 127.0.15.0]; HAS_X_PRIO_THREE(0.00)[3]; IP_SCORE(-0.03)[ipnet: 194.109.0.0/16(-0.15), asn: 3265(-0.04), country: NL(0.03)]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; MIME_TRACE(0.00)[0:+,1:+,2:~] Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 09:30:28 -0000 Hi, Are you sure the new pool is not almost full. That creates a significant performance drop. Otherwise look at "zpool iostat" and "gstat" to see what the disks are doing. Ronald. Van: Daryl Richards via freebsd-fs Datum: zondag, 12 april 2020 21:26 Aan: freebsd-fs Onderwerp: zpool status question > > Hello list, > > I recently needed to add more storage to my pool and to get around the ashift problem of my old pool I made a new one on new drives and did a send->receive to copy my data. Everything went fine with the copy, but now when I do a zpool status, it takes about 15 minutes to show the status of the new pool. It takes about 5 minutes to show the first line (name of pool, then top level vdev), and then about 3 minutes per device after that. (4 drive raidz1). Everything shows ONLINE when done, no errors or timeouts show up in dmesg, nothing. It just takes a long time to display status.. Getting status of the other pools (zroot, and old array) is instantaneous. > > I'm just being overly cautious before I export the old array and import the new one as my main one.. I'm on FreeBSD 11.3-RELEASE > > Any ideas? Thanks! > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > >