From owner-freebsd-current@freebsd.org Wed Sep 5 20:14:55 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E79CFFBC8B for ; Wed, 5 Sep 2018 20:14:55 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6CD37180A; Wed, 5 Sep 2018 20:14:54 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: by mail-lf1-x131.google.com with SMTP id z11-v6so7082179lff.9; Wed, 05 Sep 2018 13:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sqCvDFhkhh9LKzvVEgONsMrU8vl2aglrx+oji0cvTKQ=; b=NtC2p1bN4nwOv+7oKd5n+IpSFut4q9cbfXGkfEPufwWgWdunGL23agHzkicAQ1sEXX er3iL3YiDol7/o6Uq/k0IzuMuce/mAicStzs/uP/KCJodssqyXoq40GZUkn664+sCWyo FYbZRJ+4qmBgv7KQXCuCN70iABpPwUSgXpfc+lIbZ6nCKNUWEObDU3aqqmnezi3kGLpO CjqOyResbuzXqoHAhF6mU/dx5sxrn34LTmKurZ+WOWz3BMKGPSA5FNhdJ5OtbXK2nJYI KMtjAHB1zlDnSADmbE1k496UbWKyAPnNoXpMN6wohMhQTF+cE0ZOAKgE0MlLmvFhkhxi syNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sqCvDFhkhh9LKzvVEgONsMrU8vl2aglrx+oji0cvTKQ=; b=ipESBMOEQVASzK0uF6nx+41WuWC3vl94yk+Rsx1g8SNvHqwaUcWMbeOHseBZ0FNSx5 ye36/eBBeo8NfuAiOw5ZNYU9LcHWXk5xysuxaxDzBxJhGsBYSCuNV7So6SK0k8HuQ5SB m07Zs2e5KiKrfAKhDdiLjoyMt+lYCbnUMLqfnWKFiWaGvY3g8SAdcFGHWSByZVKGvpuj OZ0crxB+AwMTdNXpm8PoYAdgvfLKVXxEEGX6SvH4/Y63rWj8PL7+eFmOHRLIq2E4TaIc VXn/KgCOdPWQR3cHSlRt2Bh0SuhdGxb4cnlx3W10prSkVtOvmorxuEiAMMHK4RKTerNl OaqQ== X-Gm-Message-State: APzg51ABd5XI22ozLI5+X74yEXEGkaR3FmYwgTv8xF8Sl+68fuwSfntS EcTyOsOKSE4zcJYanBDG2NLe47HD6/xeCyckItBg5UVQ X-Google-Smtp-Source: ANB0VdYwq6gEn9Ft/N0J+1mO/Et4cXMGy7A6ffpUjKRj6pXIpFB90WQK/p0OUAt0bdMqsUllrVPOAPOz633RqhfGo5U= X-Received: by 2002:a19:6d12:: with SMTP id i18-v6mr19206747lfc.72.1536178493140; Wed, 05 Sep 2018 13:14:53 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Subbsd Date: Wed, 5 Sep 2018 23:15:03 +0300 Message-ID: Subject: Re: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 To: allanjude@freebsd.org Cc: freebsd-current Current Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 20:14:55 -0000 On Wed, Sep 5, 2018 at 5:58 PM Allan Jude wrote: > > On 2018-09-05 10:04, Subbsd wrote: > > Hi, > > > > I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12 > > to latest revision (r338466 the moment) and related to ARC. > > > > I can not say which revision was before except that the newver.sh > > pointed to ALPHA3. > > > > Problems are observed if you try to limit ARC. In my case: > > > > vfs.zfs.arc_max="128M" > > > > I know that this is very small. However, for two years with this there > > were no problems. > > > > When i send SIGINFO to process which is currently working with ZFS, i > > see "arc_reclaim_waiters_cv": > > > > e.g when i type: > > > > /bin/csh > > > > I have time (~5 seconds) to press several times 'ctrl+t' before csh is executed: > > > > load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.41r 0.00u 0.00s 0% 3512k > > load: 0.70 cmd: csh 5935 [zio->io_cv] 1.69r 0.00u 0.00s 0% 3512k > > load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.98r 0.00u 0.01s 0% 3512k > > load: 0.73 cmd: csh 5935 [arc_reclaim_waiters_cv] 2.19r 0.00u 0.01s 0% 4156k > > > > same story with find or any other commans: > > > > load: 0.34 cmd: find 5993 [zio->io_cv] 0.99r 0.00u 0.00s 0% 2676k > > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.13r 0.00u 0.00s 0% 2676k > > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.25r 0.00u 0.00s 0% 2680k > > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.38r 0.00u 0.00s 0% 2684k > > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.51r 0.00u 0.00s 0% 2704k > > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.64r 0.00u 0.00s 0% 2716k > > load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.78r 0.00u 0.00s 0% 2760k > > > > this problem goes away after increasing vfs.zfs.arc_max > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > > > Previously, ZFS was not actually able to evict enough dnodes to keep > your arc_max under 128MB, it would have been much higher based on the > number of open files you had. A recent improvement from upstream ZFS > (r337653 and r337660) was pulled in that fixed this, so setting an > arc_max of 128MB is much more effective now, and that is causing the > side effect of "actually doing what you asked it to do", in this case, > what you are asking is a bit silly. If you have a working set that is > greater than 128MB, and you ask ZFS to use less than that, it'll have to > constantly try to reclaim memory to keep under that very low bar. > > -- > Allan Jude > Hi, Thanks for comments. Mark was right when he pointed to r338416 ( https://svnweb.freebsd.org/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=338416&r2=338415&pathrev=338416 ). Commenting aggsum_value returns normal speed regardless of the rest of the new code from upstream. I would like to repeat that the speed with these two lines is not just slow, but _INCREDIBLY_ slow! Probably, this should be written in the relevant documentation for FreeBSD 12+