From owner-freebsd-fs@freebsd.org Wed Sep 5 14:05:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5182FF1130; Wed, 5 Sep 2018 14:05:09 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1761E825EE; Wed, 5 Sep 2018 14:05:09 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: by mail-lf1-x134.google.com with SMTP id h64-v6so6086792lfi.10; Wed, 05 Sep 2018 07:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=SYiJ4UFamiWU0/ZAzUyHRat24KWTcNJKf+KMjcoJU3k=; b=EOacOsNI2wJJ5cOqGHdbg4wD93WtHDGWAetSfCk4C2gwZOBziixzhYT8B3tu2m2WAT kmxRLgcSG/GnCVrEs9CxC8aaLku3aoCWhR+uBx9QkWnthz6cpdIA7+k6Pvi4rUvA0ZvL uV51LBH7L5ZieM3c7HoobuqB+bT1wk4P2jlizn11Giyr8oy9S0Klj9xLqn1Z/NpGyfjK MBbn7K1omnlAbB4GAsADLWj/pcniBRIhgwcgRvxbNdtk1jXROOigdbGzmtW5BaCTjZyc unN0AQkUELJiC21GWuBKqzSoWW+rslt6U6X/2ggrUpmFxuxEwcuzL8yqok/RQFL2TU3v M2hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=SYiJ4UFamiWU0/ZAzUyHRat24KWTcNJKf+KMjcoJU3k=; b=MmZw9OY97jVXiWoVaAWoeRuATGDdYmwlwz/2XnEvi2I6MqbqJMU6+Qv6NbHBxad6bt BBH3lEPhJtdgGSd5yStumgK+HlY4aDam40ut1oTw79rW6dS55srCfCQDkChT9O6sO6QJ E3UbbFQ5jsHqv8NbDR6C2dHvXt+SxAiaUgZb0IaQonNI9fr5xhpPcqqXOFGs0wPMWLQ6 sO4Ild5U7o+fakDtBzDmQEeJuLpxGjLyRaIBeh2NgdpqiGLdcg48vgekxPTw9feeT1pl sZR9E03HghA/kNssSAasPqTZhwws2poqaAurwIq6iuToD/6g6GWzVdLKG9JNF+Aff7hD D9dQ== X-Gm-Message-State: APzg51BeGT6idtImQFPsesPbz7ktaQ8Yu0AGuOgWVi3rIyTBu7xkv5/r oWU7LZeDuT3wAiGSFRtdpTe9hk/H39sMg5cnGmGVhc8h X-Google-Smtp-Source: ANB0Vdbb7e7nhRo22QxFjbaSeEZBXlkKfHQ9EQXAY8qbbj0Xr3AiEtDiBmwD6Hcf5QDesM21XAsHnoUI9KuKkn90M/Q= X-Received: by 2002:a19:a915:: with SMTP id s21-v6mr24625868lfe.92.1536156307583; Wed, 05 Sep 2018 07:05:07 -0700 (PDT) MIME-Version: 1.0 From: Subbsd Date: Wed, 5 Sep 2018 17:04:56 +0300 Message-ID: Subject: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 To: freebsd-current Current , freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 14:05:09 -0000 Hi, I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12 to latest revision (r338466 the moment) and related to ARC. I can not say which revision was before except that the newver.sh pointed to ALPHA3. Problems are observed if you try to limit ARC. In my case: vfs.zfs.arc_max="128M" I know that this is very small. However, for two years with this there were no problems. When i send SIGINFO to process which is currently working with ZFS, i see "arc_reclaim_waiters_cv": e.g when i type: /bin/csh I have time (~5 seconds) to press several times 'ctrl+t' before csh is executed: load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.41r 0.00u 0.00s 0% 3512k load: 0.70 cmd: csh 5935 [zio->io_cv] 1.69r 0.00u 0.00s 0% 3512k load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.98r 0.00u 0.01s 0% 3512k load: 0.73 cmd: csh 5935 [arc_reclaim_waiters_cv] 2.19r 0.00u 0.01s 0% 4156k same story with find or any other commans: load: 0.34 cmd: find 5993 [zio->io_cv] 0.99r 0.00u 0.00s 0% 2676k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.13r 0.00u 0.00s 0% 2676k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.25r 0.00u 0.00s 0% 2680k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.38r 0.00u 0.00s 0% 2684k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.51r 0.00u 0.00s 0% 2704k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.64r 0.00u 0.00s 0% 2716k load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.78r 0.00u 0.00s 0% 2760k this problem goes away after increasing vfs.zfs.arc_max