Date: Wed, 11 Nov 2009 17:44:51 -0600 From: "Sam Fourman Jr." <sfourman@gmail.com> To: Dan Nelson <dnelson@allantgroup.com> Cc: freebsd-current@freebsd.org, Ivan Voras <ivoras@freebsd.org> Subject: Re: Help ZFS FreeBSD 8.0 RC2 Write performance issue Message-ID: <11167f520911111544n2960fbe3w94e58452a95737f9@mail.gmail.com> In-Reply-To: <20091111232627.GK89052@dan.emsphone.com> References: <11167f520911111050j36dd94far667c81e6f5c18e69@mail.gmail.com> <hdf23m$b6n$1@ger.gmane.org> <20091111204903.GI89052@dan.emsphone.com> <11167f520911111326v13bb442bt36e853afbecdf834@mail.gmail.com> <20091111232627.GK89052@dan.emsphone.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Nov 11, 2009 at 5:26 PM, Dan Nelson <dnelson@allantgroup.com> wrote= : > In the last episode (Nov 11), Sam Fourman Jr. said: >> On Wed, Nov 11, 2009 at 2:49 PM, Dan Nelson <dnelson@allantgroup.com> wr= ote: >> > In the last episode (Nov 11), Ivan Voras said: >> >> Sam Fourman Jr. wrote: >> >> > I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is >> >> > acting weird on writes. =A0=A0I get ~150mbit writes idk if this is = good >> >> > or not? =A0=A0but it paused for a few seconds every once and awhile= . >> >> >> >> You didn't give any "iostat" statistics - I suspect that if you >> >> correlate ifstat and iostat output that you will see that network >> >> "pauses" happen during spikes in IO. =A0You should check for this and >> >> post your results. >> > >> > Yes, iostat would be useful here. =A0"iostat -zxC 2" will give you >> > per-disk stats plus CPU usage every 2 seconds (CPU may be a factor if >> > you have compression enabled). >> > >> > On a Solaris box I admin, setting zfs_write_limit_override helped >> > stuttering while doing heavy writes. =A0=A0It's not exported on FreeBS= D, but >> > it should be easy to add it as a RW sysctl; it lives in dsl_pool.c and >> > can be tweaked at runtime. =A0=A0Start big and tune it down so each wr= ite >> > burst takes under a second; it looks like you're writing solid for >> > around 6-8 seconds now. =A0=A0The number will vary depending on your d= isk >> > speed and how much ARC you have. >> >> here are some iostats for you. I do not believe I have compression enabl= ed >> am I mistaken? =A0isn'y SATA2 300MB/s? =A0and I am doing ~6MB/s per disk= ? =A0I >> built this machine with 4GB of memory because I thought ZFS would like i= t. >> now maybe a re(4) interface isnt the best choice. =A0if that is the prob= lem >> here I can change it. =A0We spent ~$800 on a hardware RAID card thinking >> that it would help performance >> >> Why is it that with sftp we do not see the pauses in Network transfer? > > Your service times below are worrying (both for the NFS and sftp cases); > anything above 50ms for a sustained period is usually a problem. =A0You > mention hardware RAID, but I see 6 da# devices; are these just hooked up = in > passthrough mode, or is each da device backed by multiple SATA disks? =A0= What > kind of write caching do you have enabled on the RAID? =A0Can you disable= it? > It sort of looks like zfs is bursting more data to disk than the RAID car= d > has RAM for, and it's spending multiple seconds just trying to recover. > It's also odd that you're seeing queue depths up to 70 on those disks whe= n > zfs by default should only do 35 (sysctl vfs.zfs.vdev.max_pending). > > NFS has data consistency guarantees that require it to flush to disk more > often than your sftp connection, so that may explain why the disk behavio= ur > is different. > here is some sysctl output (i believe my max pending is 35) # sysctl vfs.zfs vfs.zfs.arc_meta_limit: 67108864 vfs.zfs.arc_meta_used: 26956528 vfs.zfs.mdcomp_disable: 0 vfs.zfs.arc_min: 33554432 vfs.zfs.arc_max: 268435456 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 1 vfs.zfs.recover: 0 vfs.zfs.txg.synctime: 5 vfs.zfs.txg.timeout: 30 vfs.zfs.scrub_limit: 10 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 35 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.version.zpl: 3 vfs.zfs.version.vdev_boot: 1 vfs.zfs.version.spa: 13 vfs.zfs.version.dmu_backup_stream: 1 vfs.zfs.version.dmu_backup_header: 2 vfs.zfs.version.acl: 1 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 I am using this ARECA ARC-1130 in JBOD Mode We have write cache disabled on the controller. arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable) > mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 22 at device 14.0 on = pci4 ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.47 2009-06-25 arcmsr0: [ITHREAD] The motherboard in this machine is Asus M4A785-M http://usa.asus.com/product.aspx?P_ID=3Def0qgvMIwOUagAVl&templete=3D2 if my hardware setup is the trouble, do I just need to start from scratch and select and build a better machine? in the end I was looking to get the fastest writes I can. I am just confused on where I went wrong with this machine. Sam Fourman Jr.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?11167f520911111544n2960fbe3w94e58452a95737f9>