From nobody Sun Dec 11 16:23:01 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NVVRZ2mFgz4jcZn for ; Sun, 11 Dec 2022 16:23:06 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NVVRY5HBpz3H0J for ; Sun, 11 Dec 2022 16:23:05 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qv1-xf34.google.com with SMTP id c14so6730790qvq.0 for ; Sun, 11 Dec 2022 08:23:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc:subject:date:message-id:reply-to; bh=hj8bBurKqXBsRstuhP9P2yDYK/gG0s+S4ie0unxVKS0=; b=BizJu49DfjlVnOeJ/U0GcisjLWeHhLUEPQZK9qZPG1SUiAcui4Q00+HshHxxd4zNjm AC7SQbzkx8Mmql8960PGWOUVGMe3QkV0+Z2/taFqqJ+Purqq1ZaivdXF+rgPkHVQdlVc daT/Z7Zs95D9lQBXP+0zt9eHTEe7uw+S7icSafq5jCOh/gXKuOYSON8JSbRI5tS3tA57 T+veUCEkj1vgsIusJ4wyhLJCBKXfOAjsiRZBhCUTjFxldq017GoExbE+kt8CGk3XarU0 I7DMFDoYF464syzFtNWUvARG3EbyvvYpcylPUbVQr5tPqg4NkepvkX+aDeNtZsqfXAEj cCYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hj8bBurKqXBsRstuhP9P2yDYK/gG0s+S4ie0unxVKS0=; b=Wk046sIh780GzAcZMFSsjlHTggfaHwdp758KksP5D2HGAthxNqb/Zvfcf8M22vKCaJ acVBimc67q5psjQbFgodiH8y423vbjo507YlspqWhTpur7u7EoK6SfrzgBPP7FDU6CzB cFt/7coV2JwUFmCoC50Q25LKj9ztOUK1LofFPwMgKFV3G00NQ2zvGxJHIYWf3dE/DCmr ZySND2TEPaNFt53WWYaYUIYQTFmXYmIpE7rHFSgeSalUceb3ZUwdEoV9myxeNbTBX3BJ XFkk3nMwpWaHtKPOwcwO2Uu0H1tLZe/HuwTcN92OR0fz31DeLtrh/OFHctbqTeu89F2K oeIw== X-Gm-Message-State: ANoB5plrExTaVKwb4NAjzyONvr5cLteicqWlAnoRe8MVcnAE6PhS9fBk wF8+/0jUrSGgvIKEfZ7K0sbbKx8TnhE= X-Google-Smtp-Source: AA0mqf7qmB4PhMVXiCa18OMYvi5boecEY0OtroM6gWykUDg/mL9GGsw675pEsJzI2eU59czGP+qa6g== X-Received: by 2002:ad4:4e0b:0:b0:4c6:fdd0:df7d with SMTP id dl11-20020ad44e0b000000b004c6fdd0df7dmr18083066qvb.27.1670775784956; Sun, 11 Dec 2022 08:23:04 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id ay42-20020a05620a17aa00b006ef1a8f1b81sm4158357qkb.5.2022.12.11.08.23.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 08:23:03 -0800 (PST) Date: Sun, 11 Dec 2022 11:23:01 -0500 From: Mark Johnston To: Warner Losh Cc: Artem Kuchin , FreeBSD FS Subject: Re: Everchanging bytes at the end of mirror disks Message-ID: References: <85c5a64c-915e-d790-e617-c94f3fb7cd9a@gmail.com> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4NVVRY5HBpz3H0J X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sun, Dec 11, 2022 at 08:08:55AM -0700, Warner Losh wrote: > On Sun, Dec 11, 2022 at 1:45 AM Artem Kuchin > wrote: > > > 11.12.2022 11:22, Warner Losh пишет: > > > > > > > > On Sat, Dec 10, 2022, 11:52 PM Artem Kuchin > > wrote: > > > >> Hello! > >> > >> I am writing a small utility for myseld and part of it is comparing > >> gmirror disks. After running some tests i realized that some bytes at > >> the very end of disks are constantly changing. > >> > > > > The last sector has metadata about the mirror and about the mirror > > element. It's this latter data that differs. > > > > > > Thank you for reply. Then there are several question > > > > 1) Last SECTOR is not always 512KB or is it? Do i need to get block size > > from diskinfo and subtract its size from disk size? > > > > diskinfo(1) will tell you, it's returned with the DIOCGSECTORSIZE ioctl. In particular, it's much more likely to be 512B than 512KB. > > 2) Why its content is changing so often? On every write? How often? The > > only place to look for description is the gmirror sources? > > > When a mirror breaks (that is, writes can happen to one side but not the > other), we need to know right away which side is the more current one. The > gmirror does this by modifying the metadata to record how many writes have > happened to each mirror member (one reason that write is so expensive). Hmm, gmirror doesn't work like this as far as I know. The main source of updates to the metadata blocks is to set the "clean" flag: when more than g_mirror_idletime (default 5) seconds elapse without any writes to the mirror, all components are marked clean. If the system crashes when a mirror is clean, gmirror can skip having to synchronize the mirror components upon the next boot. gmirror does maintain a couple of generation counters ("syncid" and "genid") used when picking the most current disk after an unclean shutdown. They're only updated in fairly rare cases though, like after an I/O error or as part of administrative commands such as adding or removing mirror components. Definitely not after every regular write. When synchronizing mirror components, gmirror does periodically write the synchronization offset to the metadata sector, but this only happens once every 5 seconds. > > It does not look good to me, but maybe i am wrong? Also, does it mean no > go for gmirror on ssd? > > No. It's fine. +1 > All SSDs in the past 15-20 years have wear leveling (and > nearly all for an additional 10 years before that). It's quite hard to wear > out a device by repeated writing to one sector. You effectively have to > write the same amount of data you would if you were writing to multiple > sectors. SSDs are rated in 'drive writes per day': how many times you can > write to all the sectors of a drive, every day, for the warranty period of > the device. This is between 0.3 and 5 typically (though exceptions exist). > Any extra writes will be several orders of magnitude below this threshold > for all but the most insane write patterns (eg write all the odd sectors, > randomly, then write all the even sectors randomly, repeatedly). And if you > are doing an insane amount of writing, you likely wouldn't be using > gmirror.... It at most doubles the traffic to the drive, but if you have a > 64k block size to UFS, you'd typically see only a few percent increase. So > unless you are writing your data to the drives at rates approaching the > endurance limit of the drive, this extra write won't be an issue.[*] > > Warner > > [*] It would theoretically be helpful,though, if gmirror could add an extra > N sectors to match the underlying physical hardware page sizes, but the > experiments I've done I've not been able to see a speed increase....