From owner-freebsd-questions@freebsd.org Wed Oct 11 14:41:14 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D367E2DE22 for ; Wed, 11 Oct 2017 14:41:14 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 639226A80E for ; Wed, 11 Oct 2017 14:41:14 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: by mail-it0-x22c.google.com with SMTP id k70so10741674itk.0 for ; Wed, 11 Oct 2017 07:41:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=wL/PC9oalEeOeW0w4O2PnEy28XvFJXT2OH3c/FYSHvo=; b=e1B3xWrXzckcaUHQ4wwKK8dOeMYqlVsgMCA9VtMn5rvFIen9Quzpn7J1TZcqtQHsGQ +hC0XBIqW35btHhiTC9AbteEDxKWLZR4i3Y98ArPO9xSDa8/tRHzhnno/beMpMGAhuGf B5u0nEtkBrYykfwVyQQp4v0DqzCjrxCHbQgmf//H0EX2wF8/Tsz+j6kFc2X3mZWhm8vV Qn4mqWGzPtS2mJ5GRN8KdxBfCQTDIw47MdYTZtw1MWbWhMqj7vu0dz6u7kJTeHMbUuwV eR/eDdw7lpSI+p4Jc/hCT1qjsrjE2639iFJrPFwirvx3RZgcX79QxeS/rzkGGUZCipip sxew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=wL/PC9oalEeOeW0w4O2PnEy28XvFJXT2OH3c/FYSHvo=; b=f9N85bZKrxfwNzCkimsQHXrkyWflQkDrqT/NgUOB40LPUB/OmOXawjRFYv2ewcKgTt d/Y9ComhPDYpQ9kJVnyJbR+AtDCtxiW1v4jWlFz3b5xUkvYRu0KPRy44JOz7sNLyC5eo 1EqVTUe4lgbTkzFYXmf8MfYfDB3foh1xbuOXw755r5ORD701rkwvesOdKXW0ZpejgKFC 16HAY8lx0EJksG2hTRMEjHT7M7NAc1lninN9OnKojIEMtPBogU2W5UG5lMf41FboLH39 DlqeiM5+78XNPfHnomfwXMhxDk3kXtJWIr65dmRYZ4rpU9pSNETzZie3zmtueam/MAzJ IwVw== X-Gm-Message-State: AMCzsaWFqq8n3yLDIO8PraLqeU2vpBj0qcCGt9bkjnGyPPsqOkTbDsvx vRfm6qBR2nAKrlecVZDhKBUTm8JmwHyMCiypuXcj5w== X-Google-Smtp-Source: AOwi7QAvtD4ZulWZI0hkIe/Sxx9tSmG3Q8a46U2chbQaDRc0g19cYvwKocYxTqRxjnDCebV/dF99frZEWF3hoe2V6Dc= X-Received: by 10.36.176.70 with SMTP id b6mr24676634itj.26.1507732873730; Wed, 11 Oct 2017 07:41:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.201 with HTTP; Wed, 11 Oct 2017 07:41:13 -0700 (PDT) In-Reply-To: <20171011130512.GE24374@apple.rat.burntout.org> References: <20171011130512.GE24374@apple.rat.burntout.org> From: Adam Vande More Date: Wed, 11 Oct 2017 09:41:13 -0500 Message-ID: Subject: Re: FreeBSD ZFS file server with SSD HDD To: Kate Dawson Cc: FreeBSD Questions Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Oct 2017 14:41:14 -0000 On Wed, Oct 11, 2017 at 8:05 AM, Kate Dawson wrote: > Hi, > > Currently running a FreeBSD NFS server with a zpool comprising > > 12 x 1TB hard disk drives are arranged as pairs of mirrors in a strip set > ( RAID 10 ) > > An additional 2x 960GB SSD added. These two SSD are partitioned with a > small partition begin used for a ZIL log, and larger partion arranged for > L2ARC cache. > > Additionally the host has 64GB RAM and 16 CPU cores (AMD Opteron 2Ghz) > > A dataset from the pool is exported via NFS to a number of Debian > Gnu/Linux hosts running a xen hypervisor. These run several disk image > based virtual machines > > In general use, the FreeBSD NFS host sees very little read IO, which is to > expected > as the RAM cache and L2ARC are designed to minimise the amount of read > load > on the disks. > > However we're starting to see high load ( mostly IO WAIT ) on the Linux > virtualisation hosts, and virtual machines - with kernel timeouts > occurring resulting in crashes and instability. > > I believe this may be due to the limited number of random write IOPS > available > on the zpool NFS export. > > I can get sequential writes and reads to and from the NFS server at > speeds that approach the maximum the network provides ( currently 1Gb/s > + Jumbo Frames, and I could increase this by bonding multiple interfaces > together. ) > > However day to day usage does not show network utilisation anywhere near > this maximum. > > If I look at the output of `zpool iostat -v tank 1 ` I see that every > five seconds or so, the numner of write operation go to > 2k > > I think this shows that the I'm hitting the limit that the spinning disk > can provide in this workload. > I doubt that is the cause. It is more likely you have vfs.zfs.txg.timeout set to the default. Have you tried any other zfs or nfs tuning? If so, please share those details. Does gstat reveal anything useful? -- Adam