From owner-freebsd-fs@FreeBSD.ORG Thu Mar 27 10:06:03 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9442A242 for ; Thu, 27 Mar 2014 10:06:03 +0000 (UTC) Received: from mail-qa0-x229.google.com (mail-qa0-x229.google.com [IPv6:2607:f8b0:400d:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 53FF69F1 for ; Thu, 27 Mar 2014 10:06:03 +0000 (UTC) Received: by mail-qa0-f41.google.com with SMTP id j5so3552234qaq.14 for ; Thu, 27 Mar 2014 03:06:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UCWPa3sAkRz1AC26tCDTReXnfo5BDuPHS7TToRybsvA=; b=X71r/sLQLuOy8kj4w0rJ+8AB3ImxUyzCx8f+pmc+C7Ba1B3BTUJqDoRj9PPwRvmmJi Vwcf3Rssv1CcJA8ZS65ob4rtIVW8okUJ/2SqBdApvP84d1v9jDNbZ7RKiBR8EKb9avQf nsYz1hh1Fp/cGFa5qma2Un37o6z01RIE3ZpYZrLNivnLXvZJvIAa3L2L81kgyByKoHXF VHIDWyoRXrZ2NeK5F9gfJcmuat0UR26y+IJrmJHwhIiPctvQyhmoyCue9dcT9CGrhjFT P+ipwOpjdZDqpzqnHHwl/EnZE3c9bvW7tkgs6r9/u7mFqMF9GnDeKNvoosjT8pUkPo/V fCSQ== MIME-Version: 1.0 X-Received: by 10.140.90.80 with SMTP id w74mr859092qgd.96.1395914762467; Thu, 27 Mar 2014 03:06:02 -0700 (PDT) Received: by 10.96.143.37 with HTTP; Thu, 27 Mar 2014 03:06:02 -0700 (PDT) In-Reply-To: References: Date: Thu, 27 Mar 2014 11:06:02 +0100 Message-ID: Subject: Re: zfs l2arc warmup From: Joar Jegleim To: Ronald Klop Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 10:06:03 -0000 Hi, thnx for your input. The current setup : 2 X HP Proliant DL380 G7, 2xXeon (six core)@2667Mhz, 144GB DDR3 @1333Mhz (ecc, registered) Each server has an external shelf with 20 1TB SATA ('sas midline') 7200RPM disks The shelf is connected via a Smart Array P410i with 1GB cache . The second server is failover, I use zfs send/receive for replication ( with mbuffer). I had HAST in there for a couple months but got cold feet after some problems + I hate to have an expensive server just 'sitting there'. We will in near future start serving jpeg's from both servers which is a setup I like a lot more. I've setup 20 single disk 'raid 0' logical disks in that P410i, and built zfs mirror over those 20 disks ( raid 10) which give me about ~9TB of storage. For the record, I initially used an LSI SAS 9207-4i4e SGL HBA to connect the external shelf, but after some testing I realized I got more performance out of the P410i with cache enabled. I have dual power supplies as well as ups and want performance with the risk it involves. At the moment I have 2x Intel 520 480GB ssd's for l2arc, the plan is to add 2 more ssd's to get ~ 2TB for l2arc, and add a small ssd for log/zil The pool got default recordsize (128k), atime=off and I've set compression to lz4 and got a compressratio of 1.18x . I've set the following sysctl's related to zfs: # 100GB vfs.zfs.arc_max=107374182400 # used to be 5(default) trying 1 vfs.zfs.txg.timeout="1" # this to work with the raid ctrl cache vfs.zfs.cache_flush_disable=1 vfs.zfs.write_limit_shift=9 vfs.zfs.txg.synctime_ms=200 # L2ARC tuning # Maximum number of bytes written to l2arc per feed # 8MB/s (actuall=vfs.zfs.l2arc_write_max*vfs.zfs.l2arc_feed_min_ms) # so 8MB every 200ms = 40MB/s vfs.zfs.l2arc_write_max=8388608 # Mostly only relevant at the first few hours after boot # write_boost, speed to fill l2arc until it is filled (after boot) # 70MB/s, same rule applys, multiply by 5 vfs.zfs.l2arc_write_boost=73400320 # Not sure vfs.zfs.l2arc_headroom=2 # l2arc feeding period vfs.zfs.l2arc_feed_secs=1 # minimum l2arc feeding period vfs.zfs.l2arc_feed_min_ms=200 # control whether streaming data is cached or not vfs.zfs.l2arc_noprefetch=1 # control whether feed_min_ms is used or not vfs.zfs.l2arc_feed_again=1 # no read and write at the same time vfs.zfs.l2arc_norw=1 > 2TB of l2arc? > Why don't you put your data on SSD's, get rid of the l2arc and buy some > extra RAM instead. > Than you don't need any warm-up. I'm considering this option, but today I have ~10TB of storage, and need space for future growth + I like the idea that the l2arc may die and I'll loose performance, not my data. + I reckon I'd have to use a lot more expensive ssd's if I was to use them for main datastore, as l2arc I can use cheaper ssd's. Those intel 520's can deliver ~50 000 iops, and I need iops, not necessarily bandwidth. At least that's my understandig of this. Open for input ! :) On 27 March 2014 10:02, Ronald Klop wrote: > On Thu, 27 Mar 2014 08:50:06 +0100, Joar Jegleim > wrote: > >> Hi list ! >> >> I struggling to get a clear understanding of how the l2arc get warm ( >> zfs). >> It's a FreeBSD 9.2-RELEASE server. >> >> From various forum I've come up with this which I have in my >> /boot/loader.conf >> # L2ARC tuning >> # Maximum number of bytes written to l2arc per feed >> # 8MB (actuall=vfs.zfs.l2arc_write_max*(1000 / vfs.zfs.l2arc_feed_min_ms)) >> # so 8MB every 200ms = 40MB/s >> vfs.zfs.l2arc_write_max=8388608 >> # Mostly only relevant at the first few hours after boot >> # write_boost, speed to fill l2arc until it is filled (after boot) >> # 70MB, same rule applys, multiply by 5 = 350MB/s >> vfs.zfs.l2arc_write_boost=73400320 >> # Not sure >> vfs.zfs.l2arc_headroom=2 >> # l2arc feeding period >> vfs.zfs.l2arc_feed_secs=1 >> # minimum l2arc feeding period >> vfs.zfs.l2arc_feed_min_ms=200 >> # control whether streaming data is cached or not >> vfs.zfs.l2arc_noprefetch=1 >> # control whether feed_min_ms is used or not >> vfs.zfs.l2arc_feed_again=1 >> # no read and write at the same time >> vfs.zfs.l2arc_norw=1 >> >> But what I really wonder is how does the l2arc get warmed up ? >> I'm thinking of 2 scenarios: >> >> a.: when arc is full, stuff that evict from arc is put over in l2arc, >> that means that files in the fs that are never accessed will never end >> up in l2arc, right ? >> >> b.: zfs run through fs in the background and fill up the l2arc for any >> file, regardless if it has been accessed or not ( this is the >> 'feature' I'd like ) >> >> I suspect scenario a is what really happens, and if so, how does >> people warmup the l2arc manually (?) >> I figured that if I rsync everything from the pool I want to be >> cache'ed, it will fill up the l2arc for me, which I'm doing right now. >> But it takes 3-4 days to rsync the whole pool . >> >> Is this how 'you' do it to warmup the l2arc, or am I missing something ? >> >> The thing is with this particular pool is that it serves somewhere >> between 20 -> 30 million jpegs for a website. The front page of the >> site will for every reload present a mosaic of about 36 jpegs, and the >> jpegs are completely randomly fetched from the pool. >> I don't know what jpegs will be fetched at any given time, so I'm >> installing about 2TB of l2arc ( the pool is about 1.6TB today) and I >> want the whole pool to be available from the l2arc . >> >> >> Any input on my 'rsync solution' to warmup the l2arc is much appreciated >> :) > > > > 2TB of l2arc? > Why don't you put your data on SSD's, get rid of the l2arc and buy some > extra RAM instead. > Than you don't need any warm-up. > > For future questions, please provide more details about your setup. What are > disks, what ssds, how much RAM. How is your pool configured? Mirror, raidz, > ... Things like that. > > Ronald. -- ---------------------- Joar Jegleim Homepage: http://cosmicb.no Linkedin: http://no.linkedin.com/in/joarjegleim fb: http://www.facebook.com/joar.jegleim AKA: CosmicB @Freenode ----------------------