Date: Sun, 6 Mar 2011 01:04:55 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Joshua Boyd <boydjd@jbip.net> Cc: freebsd-fs@freebsd.org Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE Message-ID: <20110306090455.GA87055@icarus.home.lan> In-Reply-To: <AANLkTimrdnxsxUpmZnr3=w5J8_46ZM91crEfY6c_ZR4z@mail.gmail.com> References: <1299232133.18671.3.camel@pc286.embl.fr> <20110304100517.GA23249@icarus.home.lan> <AANLkTikQiTi25TR6uDD2umRZQrOL8YZzEC960oWf4wax@mail.gmail.com> <20110304105608.GA23887@icarus.home.lan> <AANLkTimrdnxsxUpmZnr3=w5J8_46ZM91crEfY6c_ZR4z@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Mar 06, 2011 at 02:59:43AM -0500, Joshua Boyd wrote:
> On Fri, Mar 4, 2011 at 5:56 AM, Jeremy Chadwick
> <freebsd@jdc.parodius.com> wrote:
> > If you get better performance -- really, truly, honestly -- with
> > prefetch enabled on your system, then I strongly recommend you keep it
> > enabled. ?However, for what it's worth (probably not much), this is the
> > first I've ever heard of a FreeBSD system performing better with
> > prefetch enabled.
>
> I just recently turned it on after having it turned off for a long
> time ... my speeds went from ~300MB/s to 600+MB/s in bonnie++. This is
> a dual core AM3 system with 8GB of ram, and 15 disks in a striped
> raidz configuration (3 sets striped).
Here are some numbers for you. This is from a 8.2-STABLE (RELENG_8)
system built Thu Feb 24 22:06:45 PST 2011, type amd64.
/boot/loader.conf
======================
ahci_load="yes"
vm.kmem_size="8192M"
vfs.zfs.arc_max="6144M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
/etc/sysctl.conf
======================
kern.maxvnodes=250000
vfs.zfs.txg.write_limit_override=1073741824
ZFS details
======================
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
data 2.72T 780G 1.96T 28% ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 519G 1.28T 28.0K none
data/backups 519G 1.28T 519G /backups
data/home 7.19M 1.28T 7.19M /home
# zpool status data
pool: data
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
Pool and filesystem attributes are all defaults (ZFS v15),
except for mountpoint.
Disk details
======================
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD1001FALS-00U9B0 05.00K05> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD1001FALS-00U9B0 05.00K05> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
Benchmark commands
======================
shutdown -r now
<wait for machine to come back up>
cd /backups && bonnie++ -s 16g -u root
Benchmark results #1
======================
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
XXXXX 16G 92 99 127026 42 62634 18 244 99 177522 22 207.0 5
Latency 144ms 1277ms 1616ms 47845us 111ms 639ms
Version 1.96 ------Sequential Create------ --------Random Create--------
XXXXX -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 18945 96 +++++ +++ 17552 98 18975 94 +++++ +++ 18286 98
Latency 18018us 85us 131us 17862us 38us 80us
Benchmark results #2
======================
Results with only one change made to /boot/loader.conf:
vfs.zfs.prefetch_disable="0"
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
XXXXX 16G 95 99 126656 40 74678 24 229 97 187918 26 219.8 5
Latency 159ms 1813ms 2506ms 330ms 308ms 657ms
Version 1.96 ------Sequential Create------ --------Random Create--------
XXXXX -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20140 96 +++++ +++ 16814 96 10431 54 +++++ +++ 14846 95
Latency 21010us 158us 1242us 146ms 721us 11662us
--
| Jeremy Chadwick jdc@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110306090455.GA87055>