From owner-freebsd-fs@freebsd.org Sun Apr 12 11:56:06 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 81E4D2BB4E7 for ; Sun, 12 Apr 2020 11:56:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 490VZY30kYz42CC for ; Sun, 12 Apr 2020 11:56:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f172.google.com with SMTP id q22so6203076ljg.0 for ; Sun, 12 Apr 2020 04:56:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=vGjbMGb0BuR4s0Sf5F+7P44R3s4CFVlXZc0CufS3/XA=; b=i/Tv1nMUUzcILDfwE7oW2+o8K5PvJ7BJbLwdlMNMg0kAYfmuulJDB7JTGlPla5oWGm 34alc+05zeHHzV7HFEvF5XfhoWgUQ/OvkKGptF+Lpd7ffr9BTFbczkezYSzCy0Ja7C9c iyEjz6kn3R1RcVeHG/GZXE6FYKV4xpimGp0bubINttpBbeIQ98Lu7N2/QsgoJcKFfik4 vzG3R3vZzLis+kFUG9bjMz4nL8I2tMtWHilXnBAUKm7XehPlOM6PvCpUCU9JPVWo2Ndm ttAOrx1iVUyg6yp24uH8R7c0nnzlGf7+4It4ZkY3uUyymD7/fS9RJj4HJzyYX4JuDggD i6Qg== X-Gm-Message-State: AGi0PubGpSR/NJfTZevRfFh6z4ZcMICbnML8ydrjF/HiEvsY5Ph+OwEs jIkdI1/heivXV2Ww/J5RF8mzfvS9zVs= X-Google-Smtp-Source: APiQypJQUh9qY04oOhJaE6F63WQJ+GW8taTI+suESxi+15Y5llBu57jewSyTj6+tHAgZWXlknYNgRA== X-Received: by 2002:a2e:4942:: with SMTP id b2mr8095794ljd.135.1586692563167; Sun, 12 Apr 2020 04:56:03 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id l12sm2783190lfp.35.2020.04.12.04.56.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Apr 2020 04:56:02 -0700 (PDT) Subject: Re: ZFS server has gone crazy slow To: Peter Eriksson , freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> <575c01de-b503-f4f9-2f13-f57f428f53ec@FreeBSD.org> <747B75C0-73D7-42B2-9910-9E16FCAE23C4@lysator.liu.se> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <2f49fe01-50dc-2e72-8cd6-beb6f0fbb621@FreeBSD.org> Date: Sun, 12 Apr 2020 14:56:01 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <747B75C0-73D7-42B2-9910-9E16FCAE23C4@lysator.liu.se> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 490VZY30kYz42CC X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.40 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[172.208.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.40)[ip: (-1.13), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[172.208.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 11:56:06 -0000 On 12/04/2020 14:46, Peter Eriksson wrote: > You are probably right. > > However - we have seen (thru experimentation :-) that “zfs destroy -d” for > recursive snapshot destruction on many filesystems (recursively) seemed to > allow it to be done much faster (Ie the command finished much quicker) on our > servers. But it also meant that a lot of I/O seemed to be happening quite > some time after the last “zfs destroy -d” command was issued (and a really > long time when there were near-quota-full filesystems). No clones or “user > holds” in use here as far as I know. Why that is I don’t know. With “zfs > destroy” (no “-d”) things seems to be much more synchronous. > > We’ve stopped using “-d” now since we’d rather not have that type of I/O load > be happening during daytime and we had some issues with some nightly snapshot > cleanup jobs not finishing in time. I think that you want to re-test zfs destroy vs destroy -d and do that rigorously. I am not sure how to explain what you saw as it cannot be explained by how destroy -d actually differs from plain destroy. Maybe it it was a cold vs warmed up (with respect to the ARC) system, maybe something else, but certainly not -d if you do not have user holds and clones. Just in case, here is the only place in the code where 'defer' actually makes difference. dsl_destroy_snapshot_sync_impl: if (defer && (ds->ds_userrefs > 0 || dsl_dataset_phys(ds)->ds_num_children > 1)) { ASSERT(spa_version(dp->dp_spa) >= SPA_VERSION_USERREFS); dmu_buf_will_dirty(ds->ds_dbuf, tx); dsl_dataset_phys(ds)->ds_flags |= DS_FLAG_DEFER_DESTROY; spa_history_log_internal_ds(ds, "defer_destroy", tx, ""); return; } > Anyway, the “seems to be writing out a lot of queued up ZIL data” at “zfs > mount -a” time was definitely a real problem - it mounted most of the > filesystems pretty quickly but then was “extremely slow” for a couple of them > (and was causing a lot of I/O). Like 4-6 hours. Luckily that one was one of > our backup servers and during a time when the only one it frustrated was me… > I’d hate that to happen for one of the frontend (NFS/SMB-serving) servers > during office hours :-) I don't doubt that, I just tried to explain that whatever was in ZIL could not come from zfs destroy. It was something else. >> On 12 Apr 2020, at 13:26, Andriy Gapon wrote: >> >> >> On 12/04/2020 00:24, Peter Eriksson wrote: >>> Another fun thing that might happen is if you reboot your server and >>> happen to have a lot of queued up writes in the ZIL (for example if you >>> did a “zfs destroy -d -r POOL@snapshots” (deferred(background) destroys >>> of snapshots) and do a hard reboot while it’s busy it will “write out” >>> those queued transactions at filesystem mount time during the boot >>> sequence >> >> Just nitpicking on two bits of incorrect information here. First, zfs >> destroy never uses ZIL. Never. ZIL is used only for ZPL operations like >> file writes, renames, removes, etc. The things that you can do with Posix >> system calls (~ VFS KPI). >> >> Second, zfs destroy -d is not a background destroy. It is a deferred >> destroy. That means that either the destroy is done immediately if a >> snapshot has no holds which means no user holds and no clones. Or the >> destroy is postponed until holds are gone, that is, the last clone or the >> last user hold is removed. >> >> Note, however, that unless you have a very ancient pool version destroying >> a snapshot means that the snapshot object is removed and all blocks >> belonging to the snapshot are queued for freeing. Their actual freeing is >> done asynchronously ("in background") and can be spread over multiple TXG >> periods. That's done regardless of whether -d was used. -- Andriy Gapon