From owner-freebsd-questions@freebsd.org Fri Apr 13 23:06:49 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FEB3F8A167 for ; Fri, 13 Apr 2018 23:06:49 +0000 (UTC) (envelope-from freebsd@edvax.de) Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.126.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.kundenserver.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7884380E55 for ; Fri, 13 Apr 2018 23:06:48 +0000 (UTC) (envelope-from freebsd@edvax.de) Received: from r56.edvax.de ([92.195.30.67]) by mrelayeu.kundenserver.de (mreue001 [212.227.15.167]) with ESMTPA (Nemesis) id 0MGsGP-1fBCEW3O7t-00DX6b; Sat, 14 Apr 2018 01:06:37 +0200 Date: Sat, 14 Apr 2018 01:06:36 +0200 From: Polytropon To: Arthur Chance Cc: "Ronald F. Guilmette" , freebsd-questions@freebsd.org Subject: Re: Two questions --- SSD block sizes and buffering Message-Id: <20180414010636.18b4e568.freebsd@edvax.de> In-Reply-To: <1cc45af6-bd4b-3854-4d37-8e9343786ce6@qeng-ho.org> References: <23423.1523502187@segfault.tristatelogic.com> <20180412210610.bafc713b.freebsd@edvax.de> <1cc45af6-bd4b-3854-4d37-8e9343786ce6@qeng-ho.org> Reply-To: Polytropon Organization: EDVAX X-Mailer: Sylpheed 3.1.1 (GTK+ 2.24.5; i386-portbld-freebsd8.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:7FOolnKzxKTrOPYZx8QURtNxYBVxxPNjzQIgASJX8yafwYbdXyb cdpvOQs8tU9jnjmM4VYkjv6XTeG1D1hUOEniRnJq1hwwrTUrxW5jU/rP0YLdZXNwty+esBJ tnU0J3XsRCrT2OkwhuYZZ3ZdlGL0umitq3kYEU91pztPuno0NMYuZsnRGYyPjwezS2WQ4z4 aYxfeJ8OwxejoYElrd/lA== X-UI-Out-Filterresults: notjunk:1;V01:K0:VKyD59/bzEA=:R5t/3YjPpu9mvIljvKXFr5 C0loBIxPOmy+FluKMeQ+mw/XKnktwEiJgQrepIy1Pq+Rn4HZTJc/lHg7aMNZoOza8ybHovp8f gOlrD2MAKiXZkBnh6I8Pd10fJVy+/7UOh4fplld5bkMpsT1s2Wjw8YwAZPaaTTYJ2wjaGzdEn hQ+R0sc8Hz01ZdrnsYqgCw+MerqcC6A1xOZIe2NKzvvYXRjMhu4GK7b5vZT3N1pXZtN/GkHbn llKul3SCAelvR4WX5w59t6ICHUOFfEa6erkOHkNA1iBzK63lyelMBu53LoCfDJmEFIUgwsqX3 j5GnX0tJrvLs/cLdL/ceE5GFgmA4rmtKy0EqKmEGgcjdU9qmMIRanvg/iMkcimbQu4K83pzpc oOh7OZuaY73u8l70IhgTKtHzPpjK9vGxKE/RwkqkWt5ZXDacIb0cy+NJrUTa8LZJGOb5Zro4i njVWeEgz5Gm6hmdi08AdjOSqX30V6a4Cr2ZSHWlfo1jpDB37xLImsbDc28yZvRazo+08cGfah 3O9CaC/bj5PXmlS3MmUn6lGN7/Hy2ClU+/TEWQgJ41vzgJlyK1w6nyTLmGVfpW3Sc2g9+EkUr UpHcrkmxy1URwyUhVPO5+aeVksyuga4uNWyI72AqOG5J4UYwX7x9/wV8gySceRfuNlIllGUFo mPz0E0SeWUNT7N00mGm6yIGObKhynS9FYTk6loNIVF5MFEdc+I2G4XTfp7RNtNb/GcYhG6DPd qoW3qd5ORgL26PAeCfYKn4eF74orKhSBnAz+zXMI0VopCPtspGlpl1qkvnA= X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Apr 2018 23:06:49 -0000 On Fri, 13 Apr 2018 08:27:14 +0100, Arthur Chance wrote: > On 12/04/2018 20:06, Polytropon wrote: > > On Wed, 11 Apr 2018 20:03:07 -0700, Ronald F. Guilmette wrote: > >> > >> Two rather simple questions: > >> > >> 1) I don't know much, generally speaking, but I have certainly read that > >> the underyling hardware of flash memory products (which I guess > >> includes both USB sticks and also SSDs) all effectively have > >> "physical" block sizes on the order of 128 KiB. > >> > >> So anyway, I'm just curious to know to what extent, if any, FreeBSD, > >> when running with, on, or from any such (flash-memory based) mass > >> storage device does things specifically with these larger physical > >> block sizes (of flash memory) in mind. > > > > No, it just works(TM). :-) > > > > > > > >> Does FreeBSD automagically sense that it is dealing with an SSD, and > >> does it then adjust the way it operates on the relevant filesystem(s) > >> accordingly, for maximal performance? Or must the system administrator > >> tweek something explicitly (e.g. tunefs parameters, or perhaps newfs > >> parameters) in order to get optimal performance on/with an SSD? > > > > The fragment size is to be applied by the system administrator > > when initializing the disk, depending on the block size of the > > device. For example, newfs allows you to do so: > > > > # newfs -m 0 -i 16384 -b 16384 -f 4069 -L somelabel \ > > -t enable -n enable -U /dev/ada0a > > > > Note the -f flag; from "man newfs": > > > > -f frag-size > > The fragment size of the file system in bytes. It must be a > > power of two ranging in value between blocksize/8 and blocksize. > > The default is 2048 bytes. > > Which revision of FBSD are you running? I'm on 10 (must upgrade) and man > newfs says > > -f frag-size > The fragment size of the file system in bytes. It must be a > power of two ranging in value between blocksize/8 and blocksize. > The default is 4096 bytes. > > so it's been fixed for 4k disks. Yes, this has been taken from a 8.x system I was just logged in to. You know, one of those systems that run and run and run and run and run and run... obeying the "never touch a running system" rule. :-) You are correct, 4k became the default frag size, so adjusting it manually is only needed for specific cases now. > > But as mentioned in many cases: Deviating from the default > > values should be done for good and educated reasons, and > > depending on actual requirements. > > > > The example above uses a 4k frag size as typically used > > with newer hard disks. Additionally, slices, partitions > > and labels should be adjusted to match block size multiples > > for better performance. > > It's worth noting that by default the first GPT partition (which is > usually the boot partition) starts at block 34, which isn't a multiple > of 4k. I always set it to start at block 40 to align with 4k disks and > start the second partition at block 2048 so it's 1 MB aligned. Fully agree. Using 1M as factor makes it easy to align the partitions. > >> 2) I have written some modest C programs that output lots and lots of > >> very short little tidbits of data, one per line. In some cases these > >> programs output millions of lines. > >> > >> For reasons that I can explain, these programs explicitly call setvbuf() > >> on stdout during startup, and they set the buffering type for stdout > >> to _IOLBF (i.e. line buffering). > >> > >> My question is just this: Assme that one of these programs is called > >> "xyz". Now, if I run the program thusly: > >> > >> xyz > xyz.output > >> > >> i.e. so that stdout is redirected to a file, will there be one actual > >> write to disk for each and every line that is written to stdout by xyz? > >> In other words, will my act of explicitly setting line buffering (for > >> stdout) in a case like this cause the xyz program to beat the living > >> hell out of my disk drive? > > > > Probably not. The actual write operationg are being issued > > somewhere "down the line" through the file system driver > > down to the disk driver. Even a "sync" command issued by > > the OS will not _immediately_ cause the drive to act. > > write(2) calls, which underlie stdio, add the data to the disk block > image in the VM cache. Unless your machine is under extreme memory > pressure or you call fsync or sync the buffers eventually get written > out by a kernel task. See man syncer for details. > > >> I hope not, but I'd like to know if it will, or know why it won't, if > >> it won't. > > > > Isn't all output buffered, per default? > > There is a magic open flag that prevents it, but that's for those doing > raw block io to devices. Interesting pointer, thanks! I think I found it in "man 2 open". :-) -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ...