Date: Thu, 9 Jan 2020 20:42:39 +0100 (CET) From: Wojciech Puchar <wojtek@puchar.net> To: Warner Losh <imp@bsdimp.com> Cc: Gary Jennejohn <gljennjohn@gmail.com>, Wojciech Puchar <wojtek@puchar.net>, Hans Petter Selasky <hps@selasky.org>, Rick Macklem <rmacklem@uoguelph.ca>, Conrad Meyer <cem@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Konstantin Belousov <kostikbel@gmail.com> Subject: Re: maximum MAXBSIZE Message-ID: <alpine.BSF.2.20.2001092042310.18186@puchar.net> In-Reply-To: <CANCZdfqwMFGV3DAsR1skbjrZL2WgSMOR5L=zpscjGCSSTKS7Bg@mail.gmail.com> References: <alpine.BSF.2.20.2001072210410.21107@puchar.net> <d79078c4-f1cb-93b9-ee6e-f689936c1e01@selasky.org> <YQBPR0101MB1427EEDE94AA6E34B49C3C09DD3F0@YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM> <20200108105136.0d54ebce@ernst.home> <alpine.BSF.2.20.2001081452360.44533@puchar.net> <20200108141810.GX23031@kib.kiev.ua> <CAG6CVpUrGyov12nQSKhofCPw5fAiXgDGChxf3-aFu1fKpirJTQ@mail.gmail.com> <alpine.BSF.2.20.2001091057420.96836@puchar.net> <CANCZdfokuE%2BKheFvSnx7M4he9Drx31xLj8o_GKUGJqKk32Oj7g@mail.gmail.com> <alpine.BSF.2.20.2001091520120.10661@puchar.net> <20200109164519.33fc7478@ernst.home> <CANCZdfqwMFGV3DAsR1skbjrZL2WgSMOR5L=zpscjGCSSTKS7Bg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> > POLA (principle of least amazement). I certainly don't need a MAXPHYS set > to 4MB on my desktop machine. Not everyone using FreeBSD is running > servers with large amounts of memory and disk storage. > > It's a trivial change if it's beneficial in a certain use scenario. The > decision should be left up to the user. > > > And if you change MAXPHYS, you'll also want to bump the insanely small runningbuf limits. And there may be a few other parameters of course i did > I've not reported that we set that are critical too :) > > Warner > > > > > On Thu, 9 Jan 2020, Warner Losh wrote: > > > > > Netflix runs our entire network at MAXPHYS=8MB since we're doing huge reads off HDD. > > > Warner > > > > > > > > > On Thu, Jan 9, 2020 at 2:58 AM Wojciech Puchar <wojtek@puchar.net> wrote: > > > 2MB MAXPHYS was what i have set for over 3 years without problems. > > > > > > On Wed, 8 Jan 2020, Conrad Meyer wrote: > > > > > > > Bufs are dynamically allocated from uma now, and perhaps a middle ground BSIZE is worth considering? Would > 1MB and 2kB > > > bufs (1kB > > > > 32-bit) be awful? > > > > > > > > Cheers, > > > > Conrad__ > > > > > > > > On Wed, Jan 8, 2020 at 06:18 Konstantin Belousov <kostikbel@gmail.com> wrote: > > > >__ __ __ __On Wed, Jan 08, 2020 at 02:52:57PM +0100, Wojciech Puchar wrote: > > > >__ __ __ __> sorry i made a mistake - i change MAXPHYS not MAXBSIZE. > > > >__ __ __ __> > > > >__ __ __ __> 16MB works for now without problems > > > >__ __ __ __MAXPHYS 16MB means that sizeof(struct buf) is around 32K (16K on 32bit). > > > > > > > >__ __ __ __> > > > >__ __ __ __> On Wed, 8 Jan 2020, Gary Jennejohn wrote: > > > >__ __ __ __> > > > >__ __ __ __> > On Tue, 7 Jan 2020 22:47:54 +0000 > > > >__ __ __ __> > Rick Macklem <rmacklem@uoguelph.ca> wrote: > > > >__ __ __ __> > > > > >__ __ __ __> > > Hans Petter Selasky wrote: > > > >__ __ __ __> > > > On 2020-01-07 22:12, Wojciech Puchar wrote: > > > >__ __ __ __> > > > > default MAXBSIZE is 128kB. badly low for todays magnetic disks. > > > >__ __ __ __> > > > > > > > >__ __ __ __> > > > > i have it set to 2MB on all computers that have magnetic disks. Great > > > >__ __ __ __> > > > > improvement with large files. especially when more than one are > > > >__ __ __ __> > > > > read/wrote in parallel. And no problems experienced > > > >__ __ __ __> > > > > > > > >__ __ __ __> > > > > But for optimal performance MAXBSIZE should be transfered in few times > > > >__ __ __ __> > > > > longer than average seek time. todays disk do 200-250MB/s so 2MB is > > > >__ __ __ __> > > > > transfered below 10ms. > > > >__ __ __ __> > > > > > > > >__ __ __ __> > > > > 8-16MB seems like good choice. is there any reason not to set it that high? > > > >__ __ __ __> > > > > > > >__ __ __ __> > > > Old disk may not support it, especially USB 1.0/2.0 disks. > > > >__ __ __ __> > > I also thought it was limited to MAXPHYS, but maybe I'm only thinking of the NFS > > > >__ __ __ __> > > specific case? > > > >__ __ __ __> > > > > > >__ __ __ __> > > > > >__ __ __ __> > There's a comment in param.h that it should not exceed MAXPHYS to be > > > >__ __ __ __> > on the safe side.__ How old that comment is I can't say and that may > > > >__ __ __ __> > not be the case today. > > > >__ __ __ __> > > > > >__ __ __ __> > MAXBSIZE is only 64KiB in my param.h. > > > >__ __ __ __> > > > > >__ __ __ __> > I have to agree with HPS.__ There are many old bridge-chips still in > > > >__ __ __ __> > use and problems with a large MAXBSIZE might occur.__ It's certainly > > > >__ __ __ __> > not uncommon to see capacity limitations - I have a docking station > > > >__ __ __ __> > which can't see more than 3TB. > > > >__ __ __ __> > > > > >__ __ __ __> > -- > > > >__ __ __ __> > Gary Jennejohn > > > >__ __ __ __> > > > > >__ __ __ __> > > > > >__ __ __ __> _______________________________________________ > > > >__ __ __ __> freebsd-hackers@freebsd.org mailing list > > > >__ __ __ __> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > >__ __ __ __> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > >__ __ __ _________________________________________________ > > > >__ __ __ __freebsd-hackers@freebsd.org mailing list > > > >__ __ __ __https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > >__ __ __ __To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > > > > > > > > > > _______________________________________________ > > > freebsd-hackers@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > > > > > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > -- > Gary Jennejohn > > > From owner-freebsd-hackers@freebsd.org Thu Jan 9 20:28:24 2020 Return-Path: <owner-freebsd-hackers@freebsd.org> Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 003F21F8173 for <freebsd-hackers@mailman.nyi.freebsd.org>; Thu, 9 Jan 2020 20:28:24 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47tyP25S5Dz4f72 for <freebsd-hackers@freebsd.org>; Thu, 9 Jan 2020 20:28:22 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 009KS9Wk010554 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 9 Jan 2020 20:28:12 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: wojtek@puchar.net Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 009KS7v1097115 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 10 Jan 2020 03:28:07 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: maximum MAXBSIZE To: Wojciech Puchar <wojtek@puchar.net>, Gary Jennejohn <gljennjohn@gmail.com> References: <alpine.BSF.2.20.2001072210410.21107@puchar.net> <d79078c4-f1cb-93b9-ee6e-f689936c1e01@selasky.org> <YQBPR0101MB1427EEDE94AA6E34B49C3C09DD3F0@YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM> <20200108105136.0d54ebce@ernst.home> <alpine.BSF.2.20.2001081452360.44533@puchar.net> <20200108141810.GX23031@kib.kiev.ua> <CAG6CVpUrGyov12nQSKhofCPw5fAiXgDGChxf3-aFu1fKpirJTQ@mail.gmail.com> <alpine.BSF.2.20.2001091057420.96836@puchar.net> <CANCZdfokuE+KheFvSnx7M4he9Drx31xLj8o_GKUGJqKk32Oj7g@mail.gmail.com> <alpine.BSF.2.20.2001091520120.10661@puchar.net> <20200109164519.33fc7478@ernst.home> <alpine.BSF.2.20.2001092041430.18186@puchar.net> Cc: Hans Petter Selasky <hps@selasky.org>, Rick Macklem <rmacklem@uoguelph.ca>, Conrad Meyer <cem@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Konstantin Belousov <kostikbel@gmail.com> From: Eugene Grosbein <eugen@grosbein.net> Message-ID: <e8eb77d5-9a9f-cc69-8dc2-7d5f6b274798@grosbein.net> Date: Fri, 10 Jan 2020 03:28:02 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <alpine.BSF.2.20.2001092041430.18186@puchar.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47tyP25S5Dz4f72 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.84 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; IP_SCORE(-1.74)[ip: (-4.75), ipnet: 2a01:4f8::/29(-2.45), asn: 24940(-1.50), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD <freebsd-hackers.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-hackers>, <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>; List-Post: <mailto:freebsd-hackers@freebsd.org> List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 09 Jan 2020 20:28:24 -0000 10.01.2020 2:42, Wojciech Puchar wrote: >> POLA (principle of least amazement). I certainly don't need a MAXPHYS set >> to 4MB on my desktop machine. Not everyone using FreeBSD is running > > i do if i have HDD - because i want good performance when say copying big file. I can tell you one real-life story about one of my FreeBSD servers. One of its task is collecting thousands of integer packet counters from many switches over SNMP to store them in thousands RRD files and draw graphs on-demand later. Due to high parallelism of custom scripts collecting values, it successfully collects data from 2500+ switches within one minute obtaining tens of counters per SNMP request storing data to temporary plain files before moving data to RRDs. There is another independent process scanning temporary files and using rrdtool to insert data to RRDs. With untuned 8.4-STABLE, that second process took less than one minute to process 2500+ RRD files. After 8.4 went EoL the server was upgraded to FreeBSD 9 and the second process suddenly became 6 times slower and UFS went much less responsible here. It took not much time for me to discover that default value for sysctl vfs.read_max changed from 8 blocks for FreeeBSD 8 to 64 block for FreeBSD 9, so read-ahead of UFS boosted reading database files eight times consuming I/O bandwidth and cache. The problem is, opening an RRD file for update and reading its header does NOT mean reading so many blocks would be of any use, so it just wasted time and memory. I lowered vfs.read_max back to old default 8 blocks. The server has UFS block size equal to 16K, and 8*16K=128K=MAXPHYS, so performance boosted 6 times and returned to previous normal state. The server then was updated several times more but still keeps vfs.read_max=8 in its /etc/sysctl.conf and runs just fine. Not each yoghurt is equally good for everyone.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.20.2001092042310.18186>