From owner-freebsd-fs@FreeBSD.ORG Wed May 8 22:02:30 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8F936FD9 for ; Wed, 8 May 2013 22:02:30 +0000 (UTC) (envelope-from brendan.gregg@joyent.com) Received: from mail-pb0-x232.google.com (mail-pb0-x232.google.com [IPv6:2607:f8b0:400e:c01::232]) by mx1.freebsd.org (Postfix) with ESMTP id 6D0F0F8C for ; Wed, 8 May 2013 22:02:30 +0000 (UTC) Received: by mail-pb0-f50.google.com with SMTP id um15so1522014pbc.9 for ; Wed, 08 May 2013 15:02:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:x-gm-message-state; bh=rrIBAA//Uaq3SBYTb0M7NL1s8GRcP8DVQ4wpLf4Dslg=; b=RYpG413+S7WcZ2GpMUZVR1ajp6idrqO7RlNpz1SdfXo/9BL5FbHedAvLBs26scyEc1 MkxufggQN2ECvz0sByn1PnraMJn2S2TWnOL3QxMhf8vS7ADPR+fGP+bX2Rn6sUhg+yXO hIX14gkZ4uRLCMbobACeA24n9udBUi9pH5nMJYMRBgupCS9CCnoo9mULt1zfv0Ta9wze 2+XfFv2uXtqtjkzyc3THr2hImdEvgqNhUNAfpuQvwmbnXJ7xCUQBziqlC5LEHzlLyIKu 236oLmhRYaQ5ZNC3+ucZBoHhYbbmzERdnyWMO3EHbgpZ6xRfO0scdB4j+0c6Gygrm3C9 SnDw== MIME-Version: 1.0 X-Received: by 10.68.11.164 with SMTP id r4mr9781197pbb.15.1368050550195; Wed, 08 May 2013 15:02:30 -0700 (PDT) Received: by 10.68.66.168 with HTTP; Wed, 8 May 2013 15:02:30 -0700 (PDT) In-Reply-To: References: Date: Wed, 8 May 2013 15:02:30 -0700 Message-ID: Subject: Re: Strange slowdown when cache devices enabled in ZFS From: Brendan Gregg To: Freddie Cash X-Gm-Message-State: ALoCoQn6V9Boqk82Cudza6DzM0ygN1XRKd0GI6JkH5l5//bjp6mII8gwwgStKlvUIbyMI9G98VoR Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 May 2013 22:02:30 -0000 On Wed, May 8, 2013 at 2:45 PM, Freddie Cash wrote: > On Wed, May 8, 2013 at 2:35 PM, Brendan Gregg wrote: > >> Freddie Cash wrote (Mon Apr 29 16:01:55 UTC 2013): >> | >> | The following settings in /etc/sysctl.conf prevent the "stalls" >> completely, >> [...] >> To feed at 160 Mbytes/sec, with an 8 Kbyte recsize, you'll need at least >> 20,000 random read disk IOPS. How many spindles does that take? A lot. Do >> you have a lot? >> >> > 45x 2 TB SATA harddrives, configured in raidz2 vdevs of 6 disks each for a > total of 7 vdevs (with a few spare disks). With 2x SSD for log+OS and 2x > SSD for cache. > What's the max random read rate? I'd expect (7 vdevs, modern disks) it to be something like 1,000. What is your recsize? (or if it is tiny files, then average size?). On the other hand, if it's caching streaming workloads, then do those 2 SSDs outperform 45 spindles? If you are getting 120 Mbytes/sec warmup, then I'm guessing it's either a 128 Kbyte recsize random reads, or sequential. Brendan > With plans to expand that out with another 45-disk JBOD next summer-ish > (2014) > > With the settings above, I get 120 MBps of writes to the L2ARC until each > SSD is over 90% full (boot), then it settles around 5-10 MBps while > receiving snapshots from the other 3 servers. > > I guess I could change the settings to make the _boost 100-odd MBps and > leave the _max at the default. I'll play with the l2arc_write_* settings > to see if that makes a difference with l2arc_norw enabled. > > -- > Freddie Cash > fjwcash@gmail.com > -- Brendan Gregg, Joyent http://dtrace.org/blogs/brendan