From owner-freebsd-fs@freebsd.org Wed Oct 14 16:06:54 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8825A43D2B7 for ; Wed, 14 Oct 2020 16:06:54 +0000 (UTC) (envelope-from ngrechikhin@yandex.ru) Received: from forward100o.mail.yandex.net (forward100o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::600]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBHNY4wnpz4CQg for ; Wed, 14 Oct 2020 16:06:53 +0000 (UTC) (envelope-from ngrechikhin@yandex.ru) Received: from forward103q.mail.yandex.net (forward103q.mail.yandex.net [IPv6:2a02:6b8:c0e:50:0:640:b21c:d009]) by forward100o.mail.yandex.net (Yandex) with ESMTP id 19CD34AC0552 for ; Wed, 14 Oct 2020 19:06:43 +0300 (MSK) Received: from mxback7q.mail.yandex.net (mxback7q.mail.yandex.net [IPv6:2a02:6b8:c0e:41:0:640:cbbf:d618]) by forward103q.mail.yandex.net (Yandex) with ESMTP id 15B9B61E0002 for ; Wed, 14 Oct 2020 19:06:43 +0300 (MSK) Received: from localhost (localhost [::1]) by mxback7q.mail.yandex.net (mxback/Yandex) with ESMTP id cCsnA9lry7-6gj0UPiP; Wed, 14 Oct 2020 19:06:42 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1602691602; bh=9vdfJCxMlcVkL4CDMvTOzr+mj6A3M9o9K9lPpJ6plNU=; h=Message-Id:Date:Subject:To:From; b=fTvS0T7zLTxGHdDUTuzQ1U7VonGsYXN14akdpiDNtSci38CbVRJVzlw4XvYDeDRqf gw5y5/NuePEdy2mUSSc50i6SmQF8t9L2R949KQO/+6P6ftApETwiFdWYZH0I286rZ0 1RpZQaxeNizDblLNoAbm0s9OlMvcK1tZTb1/Srt0= Received: by vla1-d0568624e250.qloud-c.yandex.net with HTTP; Wed, 14 Oct 2020 19:06:42 +0300 From: =?utf-8?B?0JPRgNC10YfQuNGF0LjQvSDQndC40LrQuNGC0LA=?= To: "freebsd-fs@freebsd.org" Subject: fsck on big UFS filesystems (>100TB) X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Wed, 14 Oct 2020 19:06:42 +0300 Message-Id: <1352561602691577@mail.yandex.ru> X-Rspamd-Queue-Id: 4CBHNY4wnpz4CQg X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=fTvS0T7z; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of ngrechikhin@yandex.ru designates 2a02:6b8:0:1a2d::600 as permitted sender) smtp.mailfrom=ngrechikhin@yandex.ru X-Spamd-Result: default: False [-3.29 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.001]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:0:1000::/52:c]; FREEMAIL_FROM(0.00)[yandex.ru]; NEURAL_HAM_LONG(-1.02)[-1.019]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[yandex.ru:+]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; NEURAL_HAM_SHORT(-0.57)[-0.566]; MIME_HTML_ONLY(0.20)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:~]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim] MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 16:06:54 -0000 From owner-freebsd-fs@freebsd.org Wed Oct 14 16:19:47 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B529543DA38 for ; Wed, 14 Oct 2020 16:19:47 +0000 (UTC) (envelope-from ngrechikhin@yandex.ru) Received: from forward101j.mail.yandex.net (forward101j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBHgQ5snyz4Clx for ; Wed, 14 Oct 2020 16:19:45 +0000 (UTC) (envelope-from ngrechikhin@yandex.ru) Received: from forward100q.mail.yandex.net (forward100q.mail.yandex.net [IPv6:2a02:6b8:c0e:4b:0:640:4012:bb97]) by forward101j.mail.yandex.net (Yandex) with ESMTP id 3F34C1BE0503 for ; Wed, 14 Oct 2020 19:19:37 +0300 (MSK) Received: from mxback4q.mail.yandex.net (mxback4q.mail.yandex.net [IPv6:2a02:6b8:c0e:6d:0:640:ed15:d2bd]) by forward100q.mail.yandex.net (Yandex) with ESMTP id 39A487080004 for ; Wed, 14 Oct 2020 19:19:37 +0300 (MSK) Received: from localhost (localhost [::1]) by mxback4q.mail.yandex.net (mxback/Yandex) with ESMTP id o8cAe9rkiL-JaXGXGRw; Wed, 14 Oct 2020 19:19:36 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1602692376; bh=V/9DptWJITLdVLbxdvpxPd/tdqFF2xUcE1QSWMQZjXs=; h=Message-Id:Date:Subject:To:From; b=pd516KrJBhCtvqhxxohhItfMLAXgsJAmyp97pX/UnS+hboaKjhnNyF3ST1UcndJk8 sLXqGVSvpmYdFA6iF5jhp6QUhPk+6K3zc6fw2GHVkvfz8fFcrlHTrVEsuSL3Xw9Fg3 CjrwAgwNawJdWg+vEfTZF0hI1iEGK3nK2xqRxZHs= Received: by vla5-0c926b1cfc74.qloud-c.yandex.net with HTTP; Wed, 14 Oct 2020 19:19:36 +0300 From: Nikita Grechikhin To: "freebsd-fs@freebsd.org" Subject: fsck on big UFS filesystems (>100TB) MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Wed, 14 Oct 2020 19:19:36 +0300 Message-Id: <1720721602692234@mail.yandex.ru> Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Rspamd-Queue-Id: 4CBHgQ5snyz4Clx X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=pd516KrJ; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of ngrechikhin@yandex.ru designates 2a02:6b8:0:801:2::101 as permitted sender) smtp.mailfrom=ngrechikhin@yandex.ru X-Spamd-Result: default: False [-3.62 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.993]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[yandex.ru]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.02)[-1.024]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:0::/52]; DKIM_TRACE(0.00)[yandex.ru:+]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; NEURAL_HAM_SHORT(-0.60)[-0.598]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 16:19:47 -0000 Hi, everyone. I have several servers with large UFS partitions (>100TB). Physically it is RAID-6 on MegaRaid 9361. If i try to run fsck on them, i get same error on all servers: Code: ** Phase 5 - Check Cyl groups fsck_ffs: inoinfo: inumber 18446744071562087424 out of range inumber is exact same on all servers. tested on clean and drity filesystems with same result. And without succsessful run of fsck i can't remove dirty flag from filesystem This is a fresh install of FreeBSD 12.1 and filesystem created in 12.1 from scratch with default options. This is file storage for videoproduction. So, there is not too much files and they is relatively big. Code: # find . -type f -print | wc -l 251485 # find . -type d -print | wc -l 48351 Maybe i doing something wrong? Code: # uname -a FreeBSD *** 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC amd64 # gpart show mfid3 => 40 250031243184 mfid3 GPT (116T) 40 250031243184 1 freebsd-ufs (116T) # df -i /mnt/data2 Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/mfid3p1 116T 15T 91T 14% 300k 3.9G 0% /mnt/data2 # tunefs -p /dev/mfid3p1 tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) enabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 8336 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) # dumpfs /dev/mfid3p1 | head -30 magic 19540119 (UFS2) time Sat Oct 10 04:46:19 2020 superblock location 65536 id [ 5eab6d97 34710b41 ] ncg 149965 size 31253905398 blocks 31006762468 bsize 32768 shift 15 mask 0xffff8000 fsize 4096 shift 12 mask 0xfffff000 frag 8 shift 3 fsbtodb 3 minfree 8% optim time symlinklen 120 maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 nbfree 3369882144 ndir 48351 nifree 3915586242 nffree 36903 bpg 26051 fpg 208408 ipg 26112 unrefs 0 nindir 4096 inopb 128 maxfilesize 2252349704110079 sbsize 4096 cgsize 32768 csaddr 1672 cssize 2400256 sblkno 24 cblkno 32 iblkno 40 dblkno 1672 cgrotor 80650 fmod 0 ronly 0 clean 0 metaspace 8336 avgfpdir 64 avgfilesize 16384 flags soft-updates+journal check hashes cylinder-groups fsmnt /mnt/data2 volname swuid 0 providersize 31253905398 cs[].cs_(nbfree,ndir,nifree,nffree): (12392,3,26098,5) (11514,2,26104,4) (1034,4,26097,3) (5089,2,25977,7) (17633,1,26110,5) (992,7,26085,5) (14608,3,26098,5) (4047,3,26099,5) (17634,1,26110,5) (17636,1,26110,5) (13027,3,26099,0) (1089,3,26031,3) (13310,1,26106,1) (17632,1,26110,5) (8562,3,26098,5) (8185,1,26102,2) (14592,2,26107,3) (13173,2,26086,5) (13138,2,26020,2) (17687,1,26065,0) (13328,1,26016,7) (14754,1,26023,6) (13742,2,25995,1) (17825,1,26073,0) (6824,2,25958,4) (16919,2,26054,4) (1663,3,26098,5) (14690,3,26070,4) (2495,3,26099,4) (17636,3,26108,1) (10462,3,26099,3) (17631,1,26110,5) -- WBR, Nikita Grechikhin From owner-freebsd-fs@freebsd.org Wed Oct 14 16:55:16 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF7E643E2FB for ; Wed, 14 Oct 2020 16:55:16 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBJSN63dkz4Frw; Wed, 14 Oct 2020 16:55:16 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MBP-WLAN.fritz.box (p200300cd5f0bbc007cc4f02f434627f0.dip0.t-ipconnect.de [IPv6:2003:cd:5f0b:bc00:7cc4:f02f:4346:27f0]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4D40621E5F; Wed, 14 Oct 2020 16:55:16 +0000 (UTC) (envelope-from se@freebsd.org) To: Nikita Grechikhin References: <1720721602692234@mail.yandex.ru> From: Stefan Esser Cc: "freebsd-fs@freebsd.org" Subject: Re: fsck on big UFS filesystems (>100TB) Message-ID: <1998de69-b381-4258-2654-47fd439c402c@freebsd.org> Date: Wed, 14 Oct 2020 18:55:11 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.3.2 MIME-Version: 1.0 In-Reply-To: <1720721602692234@mail.yandex.ru> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="CgrS5hA1IFED7gHU3rE4aIJ2RrCPFWJNZ" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 16:55:17 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --CgrS5hA1IFED7gHU3rE4aIJ2RrCPFWJNZ Content-Type: multipart/mixed; boundary="asO6qngYmXXwN2XkiOWjTKLToCBKpDr54"; protected-headers="v1" From: Stefan Esser To: Nikita Grechikhin Cc: "freebsd-fs@freebsd.org" Message-ID: <1998de69-b381-4258-2654-47fd439c402c@freebsd.org> Subject: Re: fsck on big UFS filesystems (>100TB) References: <1720721602692234@mail.yandex.ru> In-Reply-To: <1720721602692234@mail.yandex.ru> --asO6qngYmXXwN2XkiOWjTKLToCBKpDr54 Content-Type: multipart/mixed; boundary="------------7DBCB3634E8ECB88FE3E57A5" Content-Language: de-DE This is a multi-part message in MIME format. --------------7DBCB3634E8ECB88FE3E57A5 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Am 14.10.20 um 18:19 schrieb Nikita Grechikhin: > Hi, everyone. > =20 > I have several servers with large UFS partitions (>100TB). > Physically it is RAID-6 on MegaRaid 9361. >=20 > If i try to run fsck on them, i get same error on all servers: > =20 > Code: >=20 > ** Phase 5 - Check Cyl groups > fsck_ffs: inoinfo: inumber 18446744071562087424 out of range This number is near minus 2^31 - without having looked at the code this smells like an integer overflow. > inumber is exact same on all servers. > tested on clean and drity filesystems with same result. >=20 > And without succsessful run of fsck i can't remove dirty flag from file= system > =20 > This is a fresh install of FreeBSD 12.1 and filesystem created in 12.1 = from scratch with default options. The default options assume an average file size of just a few KB, and I severely doubt you are going to fill 100 TB with small files. > This is file storage for videoproduction. So, there is not too much fil= es and they is relatively big. > =20 > Code: >=20 > # find . -type f -print | wc -l > 251485 > =20 > # find . -type d -print | wc -l > 48351 What fraction of the storage is filled? You may need to recreate the file system with an appropriate value if "-i", the number of bytes per inode. If your files are generally in the order of 1 GB, then you should try -i 100000000 (i.e. 100 mio to allow for up to 1 million files to have some slack). > Maybe i doing something wrong? > =20 > Code: >=20 > # uname -a > FreeBSD *** 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC amd64 >=20 > # gpart show mfid3 > =3D> 40 250031243184 mfid3 GPT (116T) > 40 250031243184 1 freebsd-ufs (116T) >=20 > # df -i /mnt/data2 > Filesystem Size Used Avail Capacity iused ifree %iused Mount= ed on > /dev/mfid3p1 116T 15T 91T 14% 300k 3.9G 0% /mnt/= data2 Yes, way too many inodes (4G) - you are wasting space and run into the fsck issue due to them ... Regards, STefan --------------7DBCB3634E8ECB88FE3E57A5-- --asO6qngYmXXwN2XkiOWjTKLToCBKpDr54-- --CgrS5hA1IFED7gHU3rE4aIJ2RrCPFWJNZ Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAl+HLW8FAwAAAAAACgkQR+u171r99UQm 8wf+OCfMxzdm0oQo0Z1Z7IKyQPKzs8T6Mf/KKywyjYUKZO9sxDZnpUpMMCTgvcVmV/tST9ETdbNL n7y/3IxSFLjQp0jocO2OnInJnbjFjEaNrYYfktwFiTFkJWacPCUSmo9Uxbt0UDzMObnFAxjgOQm7 YNnuYJdiCncLZk9+mJJ/9cBPbEEUlU44Pzed9IQvdmFtQiqU62eJBod08626gT/KAQpPV6AJtX9/ VFP1LBg71eV+mzEEvEA1OpEotm2c4lMZgN7rkNkfZnNkvTE3daAtUVo6BsjOulGT3QH6/a0zSc2/ TkB5O2zZiy1SYW1JfcvM/m/cjRebqccSJUupDFojdQ== =B0Iq -----END PGP SIGNATURE----- --CgrS5hA1IFED7gHU3rE4aIJ2RrCPFWJNZ--