Date: Wed, 20 Dec 2006 09:43:25 -0600 From: Eric Anderson <anderson@centtech.com> To: Arone Silimantia <aronesimi@yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: quotas safe on >2 TB filesystems in 6.1-RELEASE ? Message-ID: <45895A1D.2010105@centtech.com> In-Reply-To: <626700.55454.qm@web58612.mail.re3.yahoo.com> References: <626700.55454.qm@web58612.mail.re3.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/20/06 09:15, Arone Silimantia wrote: > Eric, > > Thanks for your comments and help - your posts on this > list are much appreciated. Comments in line below: > > > --- Eric Anderson <anderson@centtech.com> wrote: > >> With 9TB without >> any journaling, you might run into problems if you >> crash and need to >> fsck - the number of files you could have on the >> file system could well >> require more memory/time than you have available. > > > Hmmm...the time required is more dependent on inodes > than on size of data / size of files, right ? > > My 9 TB dataset uses about 36 million inodes. > > Any comments on that number ? Large ? Pedestrian ? > Typical ? Sounds like you have a lot of larger files (~250k per file on average possibly), which helps the fsck times. 36 million inodes should be fsck'able with enough memory (maybe ~3gb ish? That's a wild guess). I have two 10Tb file systems, one has 180million inodes. I don't attempt to fsck it, because it would take a very long time, and I might possibly run out of memory (I have 8Gb of memory). I use GJOURNAL on these, and am very happy about it (thanks Pawel!). df snippet: Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/label/vol10-data.journal 9925732858 8780187028 351487202 96% 180847683 1102147515 14% /vol10 /dev/label/vol11-data.journal 9925732858 2598987846 6532686384 28% 49422705 1233572493 4% /vol11 I also have several 2Tb partitions (set up prior to gjournal being available), that are *FULL*, and each have about 25-45million inodes on them. Those fsck in about 4-7hours each, using between 1Gb and 3Gb of memory to do so. > > I am hoping that that could be fsck'd (modern hitachi > SATA drives, raid-6, 3ware) in 48 hours ... or am I > way off ? Depends on the drives really, and maybe caching. The geom_cache module (still beta probably, and not in the src tree currently I don't believe) is said to improve fsck times. 48 hours is a long time, and I *think* it should complete within that time frame, but you'd really have to test it to be sure. >>> But I do absolutely need to run quotas (both user >> and >>> group) on this 9 GB array. I also need to >> successfully >>> use all quota admin tools (repquota, edquota, >> quota, >>> etc.) >>> >>> Can I get an assurance that this is totally safe, >>> sane, and fit to run in a mission critical, data >>> critical environment ? Anyone doing it currently >> ? >>> Any comments or warnings of _any kind_ much >> appreciated. >> >> I don't think anyone will say 'I promise it will >> work' of course, but I >> would start by using the latest 6-STABLE source >> since there have been >> quite a number of updates to file system related >> code since 6.1. > > > Ok, but all of the CLI tools (edquota, repquota, > quota, quotacheck, quotaon) are all known-good for > "bigdisk" ? > > And there is no known "quotas just don't work with > bigdisk" problems ? > > I was hoping someone out there was running quotas with > 6.1-RELEASE on a >2TB filesystem and could report favorably... I'm not certain. There were some bugs in quotas, that recently were fixed (Kris Kennaway I think reported them and saw the fixes into the tree), and prior to that I saw those consistently, so I stopped using them. I haven't tried since the fixes have been in place, and the fixes (if I recall correctly) had to do with background fsck (softupdates maybe) and not the size of the disk. You could try this in a mock-up environment, by creating a sparse file and use it with mdconfig, then newfs it, enabling quotas, mount, and then use a script to create massive amounts (36million-ish) files of about 200k-ish in size, with random users, in a similar fashion as your real data, and see if all goes well. You can also try your fsck that way too. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology An undefined problem has an infinite number of solutions. ------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45895A1D.2010105>