From owner-freebsd-current@FreeBSD.ORG Wed Nov 11 23:44:52 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B67C106566B; Wed, 11 Nov 2009 23:44:52 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com [209.85.160.44]) by mx1.freebsd.org (Postfix) with ESMTP id DC4D88FC14; Wed, 11 Nov 2009 23:44:51 +0000 (UTC) Received: by pwj15 with SMTP id 15so1095024pwj.3 for ; Wed, 11 Nov 2009 15:44:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=+Lmb2Ck72HaYgC5GBjVm+VXaCLPblrV11iXRC+nc3Go=; b=G5zN8/I8b0veysyYH7dXmmKhYIrrdp/joe0fJ1x9rwiXW6h2tzKrXKjbxgVX13UIjE hao/2F2VTVh+CgKEVjHW73cMSEj5fAWW89wUMF1QfwUmtsMnZKDHHgwlq9TzbB4+AfOU rgUImnuf1tMPHeqMNNB8jX6RJ3H+M2Ozjk0pI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=ojDzg27tRMhmehk/AW+znivWbYfdeKDzLj2xtJPMWLYHMnaYTmaHnYvcoDDoNt0QiY jog0jQg79SRvCFoCGBrD3A8DCiUEAKk1TpGpVZYDVA/82Slc/7Ndrq8QyKJ0ExfCz15R j4DScOV7WP2hgGMPg7FL1CWV5iTQY+T/xr9Fg= MIME-Version: 1.0 Received: by 10.142.55.20 with SMTP id d20mr224291wfa.37.1257983091226; Wed, 11 Nov 2009 15:44:51 -0800 (PST) In-Reply-To: <20091111232627.GK89052@dan.emsphone.com> References: <11167f520911111050j36dd94far667c81e6f5c18e69@mail.gmail.com> <20091111204903.GI89052@dan.emsphone.com> <11167f520911111326v13bb442bt36e853afbecdf834@mail.gmail.com> <20091111232627.GK89052@dan.emsphone.com> Date: Wed, 11 Nov 2009 17:44:51 -0600 Message-ID: <11167f520911111544n2960fbe3w94e58452a95737f9@mail.gmail.com> From: "Sam Fourman Jr." To: Dan Nelson Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current@freebsd.org, Ivan Voras Subject: Re: Help ZFS FreeBSD 8.0 RC2 Write performance issue X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Nov 2009 23:44:52 -0000 On Wed, Nov 11, 2009 at 5:26 PM, Dan Nelson wrote= : > In the last episode (Nov 11), Sam Fourman Jr. said: >> On Wed, Nov 11, 2009 at 2:49 PM, Dan Nelson wr= ote: >> > In the last episode (Nov 11), Ivan Voras said: >> >> Sam Fourman Jr. wrote: >> >> > I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is >> >> > acting weird on writes. =A0=A0I get ~150mbit writes idk if this is = good >> >> > or not? =A0=A0but it paused for a few seconds every once and awhile= . >> >> >> >> You didn't give any "iostat" statistics - I suspect that if you >> >> correlate ifstat and iostat output that you will see that network >> >> "pauses" happen during spikes in IO. =A0You should check for this and >> >> post your results. >> > >> > Yes, iostat would be useful here. =A0"iostat -zxC 2" will give you >> > per-disk stats plus CPU usage every 2 seconds (CPU may be a factor if >> > you have compression enabled). >> > >> > On a Solaris box I admin, setting zfs_write_limit_override helped >> > stuttering while doing heavy writes. =A0=A0It's not exported on FreeBS= D, but >> > it should be easy to add it as a RW sysctl; it lives in dsl_pool.c and >> > can be tweaked at runtime. =A0=A0Start big and tune it down so each wr= ite >> > burst takes under a second; it looks like you're writing solid for >> > around 6-8 seconds now. =A0=A0The number will vary depending on your d= isk >> > speed and how much ARC you have. >> >> here are some iostats for you. I do not believe I have compression enabl= ed >> am I mistaken? =A0isn'y SATA2 300MB/s? =A0and I am doing ~6MB/s per disk= ? =A0I >> built this machine with 4GB of memory because I thought ZFS would like i= t. >> now maybe a re(4) interface isnt the best choice. =A0if that is the prob= lem >> here I can change it. =A0We spent ~$800 on a hardware RAID card thinking >> that it would help performance >> >> Why is it that with sftp we do not see the pauses in Network transfer? > > Your service times below are worrying (both for the NFS and sftp cases); > anything above 50ms for a sustained period is usually a problem. =A0You > mention hardware RAID, but I see 6 da# devices; are these just hooked up = in > passthrough mode, or is each da device backed by multiple SATA disks? =A0= What > kind of write caching do you have enabled on the RAID? =A0Can you disable= it? > It sort of looks like zfs is bursting more data to disk than the RAID car= d > has RAM for, and it's spending multiple seconds just trying to recover. > It's also odd that you're seeing queue depths up to 70 on those disks whe= n > zfs by default should only do 35 (sysctl vfs.zfs.vdev.max_pending). > > NFS has data consistency guarantees that require it to flush to disk more > often than your sftp connection, so that may explain why the disk behavio= ur > is different. > here is some sysctl output (i believe my max pending is 35) # sysctl vfs.zfs vfs.zfs.arc_meta_limit: 67108864 vfs.zfs.arc_meta_used: 26956528 vfs.zfs.mdcomp_disable: 0 vfs.zfs.arc_min: 33554432 vfs.zfs.arc_max: 268435456 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 1 vfs.zfs.recover: 0 vfs.zfs.txg.synctime: 5 vfs.zfs.txg.timeout: 30 vfs.zfs.scrub_limit: 10 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 35 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.version.zpl: 3 vfs.zfs.version.vdev_boot: 1 vfs.zfs.version.spa: 13 vfs.zfs.version.dmu_backup_stream: 1 vfs.zfs.version.dmu_backup_header: 2 vfs.zfs.version.acl: 1 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 I am using this ARECA ARC-1130 in JBOD Mode We have write cache disabled on the controller. arcmsr0: mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 22 at device 14.0 on = pci4 ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.47 2009-06-25 arcmsr0: [ITHREAD] The motherboard in this machine is Asus M4A785-M http://usa.asus.com/product.aspx?P_ID=3Def0qgvMIwOUagAVl&templete=3D2 if my hardware setup is the trouble, do I just need to start from scratch and select and build a better machine? in the end I was looking to get the fastest writes I can. I am just confused on where I went wrong with this machine. Sam Fourman Jr.