From owner-freebsd-fs@FreeBSD.ORG Thu Mar 27 20:40:15 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04060719 for ; Thu, 27 Mar 2014 20:40:15 +0000 (UTC) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CDCE1AA for ; Thu, 27 Mar 2014 20:40:14 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id bs8so3581045wib.17 for ; Thu, 27 Mar 2014 13:40:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=IoqIzcQFVUCHDgkooKldidxaagDMkc5joKss6M4I1R8=; b=a5D75JpjkdmlBZ+mURDRY+Jilvi8aO5PWbkXvt0Cx1XMdNdz/RSw0QMlZStrb864nr VxCe0FIuS2dN5PGc+03VyFhOBxnwYX+3u79ucQ30ch2LmxLSzqGoPfKFx3LdmM2SnmBl tmdNhzmGQT1IqqLX8CBxesd4bDqKmgxEG65l3Ggs2cyPeHmGww2c3cxhIGqBzR9cZlj3 OgVYJFngVdUkZatEWazm1K5gK+VFdfXnSK+8913KpyNKU/8p+gx+MeFpQR0/9VQ15VaC aZsnLizEgy9P8tGOnMcHwZRnajAQRqyfIkOI7A6+VHp/UybjutNH3iCt05A3sHKn/3uo ib9g== MIME-Version: 1.0 X-Received: by 10.180.10.66 with SMTP id g2mr16023417wib.5.1395952812035; Thu, 27 Mar 2014 13:40:12 -0700 (PDT) Received: by 10.216.146.195 with HTTP; Thu, 27 Mar 2014 13:40:11 -0700 (PDT) In-Reply-To: References: Date: Thu, 27 Mar 2014 21:40:11 +0100 Message-ID: Subject: Re: zfs l2arc warmup From: Joar Jegleim To: Bob Friesenhahn Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 20:40:15 -0000 Appreciate your input, and I've talked to our devs about that and requested them to make some finit subset of the jpegs that rotate every night. Actually the whole web application is being rewritten, I won't have anything like that until august at best, which certainly isn't bad . When I get that kind of feature maybe I no longer need an l2arc to cover the whole dataset. On 27 March 2014 15:26, Bob Friesenhahn wrote: > On Thu, 27 Mar 2014, Joar Jegleim wrote: >> >> Is this how 'you' do it to warmup the l2arc, or am I missing something ? >> >> The thing is with this particular pool is that it serves somewhere >> between 20 -> 30 million jpegs for a website. The front page of the >> site will for every reload present a mosaic of about 36 jpegs, and the >> jpegs are completely randomly fetched from the pool. >> I don't know what jpegs will be fetched at any given time, so I'm >> installing about 2TB of l2arc ( the pool is about 1.6TB today) and I >> want the whole pool to be available from the l2arc . > > > Your usage pattern is the opposite of what the ARC is supposed to do. The > ARC is supposed to keep most-often accessed data in memory (or retired to > L2ARC) based on access patterns. > > It does not seem necessary for your mosaic to be truely random across > 20 -> 30 million jpegs. Random across 1000 jpegs which are circulated > in time would produce a similar effect. > > The application building your web page mosiac can manage which files will be > included in the mosaic and achieve the same effect as a huge cache by always > building the mosiac from a known subset of files. The 1000 jpegs used for > the mosaics can be cycled over time from a random selection, with old ones > being removed. This approach assures that in-memory caching is effective > since the same files will be requested many times by many clients. > > Changing the problem from an OS-oriented one to an application-oriented one > (better algorithm) gives you more control and better efficiency. > > Bob > -- > Bob Friesenhahn > bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ -- ---------------------- Joar Jegleim Homepage: http://cosmicb.no Linkedin: http://no.linkedin.com/in/joarjegleim fb: http://www.facebook.com/joar.jegleim AKA: CosmicB @Freenode ----------------------