Date: Sun, 12 Jul 2015 13:02:21 -0700 From: Don whY <Don.whY@gmx.com> To: freebsd-hackers@freebsd.org Subject: Re: format/newfs larger external consumer drives Message-ID: <55A2C7CD.7080005@gmx.com> In-Reply-To: <CAA3ZYrDQwL4PGrg5gFa1mhoTB0VSdedAahKxj2-YzzaC327zUA@mail.gmail.com> References: <CAA3ZYrDQwL4PGrg5gFa1mhoTB0VSdedAahKxj2-YzzaC327zUA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 7/11/2015 10:53 AM, Dieter BSD wrote: >> When building a filesystem (FFS) on these 1/2/3/4TB external USB >> drives, any suggestions? E.g., changing block sizes? Cut into >> multiple slices to make fsck's job easier (as well as *recovery*)? > > If the average filesize will be large, use large block/frag sizes. > I use 64 KiB / 8 KiB. And reduce the number of inodes. I reduce > inodes as much as newfs allows and there are still way too many. Can you think of an algorithmic way to express this? I.e., you don't want blocks to get *too* large as you risk greater losses in "partial fragments", etc. Likewise, you don't want to run out of inodes. I'm trying to come up with a relatively simple set of guidelines so folks can look at volume size, expected number of files, typical file sizes and "risk" of loss/cost of recovery (when the medium *is* unceremoniously dismounted) > These 2 changes waste less space to filesystem overhead, and fsck > runs a lot faster. If the disk has 4 KiB sectors, have the frag > size be at least 4 KiB, and have partitions start at a multiple > of 4 KiB. In light of your below comment, that may be problematic. > I recommend soft updates. When I started using su I magically stopped > losing files. <frown> I can't say I've ever "lost files" on a *BSD box (though there was an issue on a real early FBSD release with one of the SCSI drives that I was using -- mounting it would promptly scramble the superblock?) I am concerned with the fact that users can so easily/carelessly "unplug" a USB device without the proper incantations beforehand. of course, *their* mistake is seen as a "product design flaw"! :-/ > Have newfs label the filesystem(s). Then use /dev/ufs/mylabel > in /etc/fstab and the filesystems will get mounted on the right > places regardless of the disk being da0 or da1 or whatever. Ha! That's worth knowing! I was simply going to actively probe volume labels and add a second level of indirection to the mounts as a "fixup". > Swap can be labeled with glabel(8). > > Multiple slices? Consider the size of whatever media you'll > be using for backups. Consider making a read-only partition, > as it will not require fscking after a crash. Consider gpt. > If you decide on one big filesystem you don't need any partitioning, > just newfs the raw drive. I don't think backups are an issue. When was the last time you "backed up" your TiVo? STB? etc. The trick is to avoid the *need* for a backup -- despite the users' tendencies to forget to Do The Right Thing. The "demo app" that I'm working on is a sort of (low performance) NAS built on SBC's and external drives. So, the "backup" is already an inherent part of the design. > USB specific stuff: There is an off by 1 sector problem, which will > bite you if you switch a drive between using the sata-usb bridge > and connecting the drive directly to a sata controller. I had to Ouch! I've not seen that with PATA-USB bridges. OTOH, I tend not to pull a drive *from* an external enclosure but, rather, rely on the external enclosures to provide portability. E.g., easier to move 500G of files from machineA to machineB by physically moving the volume containing them! [Imagine machineA dies or is taken out of service] Having said that, though, I have an "internal" volume with FFS from a NetBSD box that I will be examining with an external USB bridge (I dislike having to open machine cases as access is usually not very convenient, here). I will note if I encounter any problems in doing so. > do a fair bit of hacking on the kernel to deal with this. (No, > I don't have a nice clean patch to offer, sorry. Besides, I just > plastered over the problem rather than hunting down and fixing the > root cause, which is what really needs to be done.) I don't recall > if it is off by one physical sector, or one logical sector. On a > 4 KiB drive it *might* mess up having things be on 4K multiples. > Word is that the disk manufacturers have changed the interface > between the bridge and the drive on recent external drives, making it > difficult-to-impossible to use the drive connected directly to a sata > controller. :-( <frown> > Some people believe that drives are binned, and the less-wonderful > drives go into the externals. This could explain externals being > less expensive, despite having more stuff (enclosure, bridge, > wall-wart, cables). Given how high the failure rate of internal > drives is, I hate to think how bad the externals must be. I've actually only had bad experiences with laptop drives. And, typically in cases where I was trying to use the laptop as a sort of desktop -- on 24/7/365 and the poor little drives just get tired! :> My external USB drives have typically had their original drives removed prior to my acquiring them (rescues). So, I can stuff a drive of my choosing inside. The "demo app" will try to use the large multi-TB drives of which I have little long-term experience. OTOH, the usage model is "fire it up, pull off whichever files you need, then spin everything down"... until the next time you might need to retrieve an ISO (a week later?) > I have yet to see a [ps]ata-to-usb bridge that allows turning > the disk's write cache off. Having the write cache turned off > is kinda important. If anyone knows of a bridge that does allow > turning the write cache off I'd love to hear about it, > >> Of course, anything external means the user could insert/remove it at >> will -- which complicates how the system will deal with the resource. > > If the drive disappears with filesystem(s) mounted. the kernel might > very well panic. There was a discussion of this problem recently. > I thought that FUSE was suggested as a possible solution, but I > can't find the discussion. This problem is not limited to users > disconnecting usb drives without unmounting them. The problem > happens all by itself with internal drives, as the drive, port > multiplier, controller, or device driver decides to go out to lunch, > and the kernel panics. This happens *far* too often, and *kills* > reliability. We really need a solution for this. I think its hard to back-port these sorts of things. Much easier to consider the possibility of failure when initially designing the system, interfaces, etc. Of course, for systems that are reasonably open-ended, predicting the future is a skill better applied to financial futures!! :>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55A2C7CD.7080005>