Date: Tue, 28 Apr 2009 01:59:43 +0200 From: Roland Smith <rsmith@xs4all.nl> To: Ghirai <ghirai@ghirai.com> Cc: freebsd-questions@freebsd.org Subject: Re: quick vfs tuning Message-ID: <20090427235943.GA35509@slackbox.xs4all.nl> In-Reply-To: <20090427191824.25e415e4.ghirai@ghirai.com> References: <20090427191824.25e415e4.ghirai@ghirai.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--y0ulUmNC+osPPQO6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 27, 2009 at 07:18:24PM +0300, Ghirai wrote: > Hi, >=20 > I'm running a RAID1 setup with gmirror and geli (AES-128) on top of > that. > While searching for ways to improve read performance, i found some > posts (on kerneltrap i think) about vfs.max_read. >=20 > The author suggested that increasing the default value of 8 to 16 > resulted in increased read speed, and that increasing it further > resulted in no noticeable performance gain. >=20 > Results are below. >=20 > Starting with vfs.read_max=3D32: >=20 > triton# dd if=3Da.iso of=3D/dev/null bs=3D3M > 1129+1 records in > 1129+1 records out > 3554287616 bytes transferred in 176.825898 secs (20100492 bytes/sec) >=20 > triton# sysctl vfs.read_max=3D64 > vfs.read_max: 32 -> 64 >=20 > triton# dd if=3Da.iso of=3D/dev/null bs=3D3M > 1129+1 records in > 1129+1 records out > 3554287616 bytes transferred in 162.943189 secs (21813048 bytes/sec) >=20 > triton# sysctl vfs.read_max=3D128 > vfs.read_max: 64 -> 128 >=20 > triton# dd if=3Da.iso of=3D/dev/null bs=3D3M > 1129+1 records in > 1129+1 records out > 3554287616 bytes transferred in 149.313994 secs (23804116 bytes/sec) >=20 > triton# sysctl vfs.read_max=3D256 > vfs.read_max: 128 -> 256 >=20 > triton# dd if=3Da.iso of=3D/dev/null bs=3D3M > 1129+1 records in > 1129+1 records out > 3554287616 bytes transferred in 150.466241 secs (23621828 bytes/sec) >=20 > Here is seems to have hit a wall. Going a bit down to 192 results in > almost exactly the same numbers, so the best value seems to be 128. > As i read, vfs.read_max means 'cluster read-ahead max block count'. > Does it read ahead the stuff into some memory? If so, can that memory > size be increased via sysctl? IIRC, if it gets a read request, it reads vfs.read_max extra clusters into the vfs cache, to improve subsequent reads. This won't do much if you're reading a lot of small files scattered around the disk. =20 > Does the improvement in performance have to do with my particular setup > (gmirror+geli)? In my experience, gmirror is slow (see below). If you have multiple cores, = geli isn't much of an issue. On a single-core machine it can become a bottleneck. > I thought i'd share the results and maybe get a discussion going in > this direction. >=20 > Test was done on a pair of SATA300 HDs spinning at 7200rmp (which are > seen as SATA150 by the OS for some reason; i couldn't fix it from the > BIOS, so it must be the mobo), and 7.1-RELEASE, i386. It doesn't matter much if your disk is seen as SATA 1.5 Gbit/s or 3 Gbit/s. A current rotating harddisk cannot max out a SATA 1.5 Gbit/s connection, see [http://en.wikipedia.org/wiki/Serial_ATA] (A flash-based drive can, though). - Intel ICH7 SATA 3Gbit/s controller - WDC WD5001ABYS-01YNA0 (500,107,862,016 bytes) - FreeBSD 7.2-PRERELEASE amd64 - no mirroring or encryption on this partition. My results: sysctl vfs.read_max=3D8 dd if=3D/tmp/var-0-20090426.dump of=3D/dev/null bs=3D3M 69+1 records in 69+1 records out 217405440 bytes transferred in 2.762058 secs (78,711,395 bytes/sec) (I added the commas to the bytes/sec figure for readability) Try it again: dd if=3D/tmp/var-0-20090426.dump of=3D/dev/null bs=3D3M 69+1 records in 69+1 records out 217405440 bytes transferred in 0.119592 secs (1,817,893,575 bytes/sec) This large figure on the second try is probably an effect of the disk's and/or vfs cache! All following reads are done after another huge file was read to try and eliminate cache effect. sysctl vfs.read_max=3D16 dd if=3D/tmp/usr-0-20090426.dump.bz2 of=3D/dev/null bs=3D3M 728+1 records in 728+1 records out 2292555598 bytes transferred in 29.368194 secs (78,062,532 bytes/sec) sysctl vfs.read_max=3D32 dd if=3D/tmp/root-0-20090426.dump of=3D/dev/null bs=3D3M 32+1 records in 32+1 records out 101068800 bytes transferred in 1.276318 secs (79,187,799 bytes/sec) sysctl vfs.read_max=3D64 dd if=3D/tmp/usr-0-20090426.dump of=3D/dev/null bs=3D3M 1753+1 records in 1753+1 records out 5516308480 bytes transferred in 70.226765 secs (78,549,944 bytes/sec) sysctl vfs.read_max=3D128 dd if=3D/tmp/usr-0-20090426.dump of=3D/dev/null bs=3D3M 1753+1 records in 1753+1 records out 5516308480 bytes transferred in 71.032365 secs (77,659,085 bytes/sec) So, for large reads not much difference. vfs.read_max=3D32 looks best. Let's try a smaller block size. sysctl vfs.read_max=3D8 dd if=3D/tmp/root-0-20090426.dump of=3D/dev/null bs=3D256k 385+1 records in 385+1 records out 101068800 bytes transferred in 1.391538 secs (72,631,008 bytes/sec) sysctl vfs.read_max=3D16 dd if=3D/tmp/usr-0-20090426.dump.bz2 of=3D/dev/null bs=3D256k 8745+1 records in 8745+1 records out 2292555598 bytes transferred in 29.736135 secs (77,096,623 bytes/sec) sysctl vfs.read_max=3D32 dd if=3D/tmp/var-0-20090426.dump of=3D/dev/null bs=3D256k 829+1 records in 829+1 records out 217405440 bytes transferred in 2.753552 secs (78,954,544 bytes/sec) sysctl vfs.read_max=3D64 dd if=3D/tmp/usr-0-20090426.dump of=3D/dev/null bs=3D256k 21043+1 records in 21043+1 records out 5516308480 bytes transferred in 71.165780 secs (77,513,497 bytes/sec) sysctl vfs.read_max=3D256 dd if=3D/tmp/var-0-20090426.dump of=3D/dev/null bs=3D256k 829+1 records in 829+1 records out 217405440 bytes transferred in 2.751325 secs (79,018,447 bytes/sec) So for this partition, vfs.read_max=3D32 seems to be optimal. Negligable CP= U load. Now, reading from a GELI encrypted partition: sysctl vfs.read_max=3D32 dd if=3Dusr-0-20090426.dump.bz2 of=3D/dev/null bs=3D256k 6572+1 records in 6572+1 records out 1722903675 bytes transferred in 22.951109 secs (75,068,428 bytes/sec) CPU load on Core2 Quad Q9300 is hovering at around 25-30%. sysctl vfs.read_max=3D64 dd if=3Dfilm.avi of=3D/dev/null bs=3D256k 2703+1 records in 2703+1 records out 708665574 bytes transferred in 9.892170 secs (71,639,042 bytes/sec) sysctl vfs.read_max=3D256 dd if=3Dfilm2.avi of=3D/dev/null bs=3D256k 3057+1 records in 3057+1 records out 801435148 bytes transferred in 11.061225 secs (72,454,466 bytes/sec) Again, vfs.read_max=3D32 seems about right. I dropped gmirror in favor of running an rsync to the second disk at night because gmirror is kinda slow. I saw the same performance as you did with the combination of gmirror and geli. Roland --=20 R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) --y0ulUmNC+osPPQO6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (FreeBSD) iEYEARECAAYFAkn2Ru8ACgkQEnfvsMMhpyU2IgCgkPWfsM5wo4tHYKHqBER00R7O XYUAn2PQX908uErSvTenm235+Z7IMITc =SVfL -----END PGP SIGNATURE----- --y0ulUmNC+osPPQO6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090427235943.GA35509>