From owner-freebsd-current@freebsd.org Sun Mar 11 23:19:11 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 692ECF2F33B for ; Sun, 11 Mar 2018 23:19:11 +0000 (UTC) (envelope-from tom.b.rushworth@gmail.com) Received: from mail-pg0-x236.google.com (mail-pg0-x236.google.com [IPv6:2607:f8b0:400e:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E796584673 for ; Sun, 11 Mar 2018 23:19:10 +0000 (UTC) (envelope-from tom.b.rushworth@gmail.com) Received: by mail-pg0-x236.google.com with SMTP id a15so1413594pgn.5 for ; Sun, 11 Mar 2018 16:19:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=reply-to:subject:references:to:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=w+hsSo7BLS/ntG5y20MxPdQByf10tYYQ+8Xo5rHQUYE=; b=NmnCrcyt3GrNPcaKbXGfaX1FjFeN/4tMR90ir5TD/kgVgzmmxv7Yo6RvV0TEcNECZU pIZ+wnXkXQKZ/JNjJ2guUzhUQDMF2RQNv/ENX+2P+o/fKZcyLuAPcXLylbABZBrW7P// KT/n8JL1dIlfUEOBmfJbKby6G8Q0BEcZYFnrBhg4BSAaqe0GhsD6rKDXyEP/YcQlr7Dk 67HSgYnF1PlcStGuGgJaWVVtV+nJFQtn8Gn26br4dCFkDg5/uty0aw6kwO7azXtXO569 +R207OZR6g9iWffAKoPWbElvsxBF/R5wHm64WJWKmDSKBD2JKE+Mj7NZSou1mF3Iebdl oYeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:references:to:from:message-id :date:user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=w+hsSo7BLS/ntG5y20MxPdQByf10tYYQ+8Xo5rHQUYE=; b=MCYmNctid38gjIaANnIeOFWf4iwb4lM5ObA6uin6tj4Z9epskqQnpXcL+8iXvODjTq XDXCs0RAdazcedOyxziZBWV3HqziWYHk8bBZR9EtDsyoa3fK0LD3Fw0wG8lLhTmaI6wl TaiExmRxoPfphENNgfL2BPUw5AIQSMrz3XW/xQV9PY/VLRlOH326MBKPxHo/IH5oOG6p fCROBhne4BRi17VWbPoiED9+XKM1+x2/1h7ytY+ITw3yiTkSdzSvTQZ1aNNVKLqZiAmb Ent2yu3gT+UIH85k17upwNXpJ8BTmbXK2JZxaI44ws8jumIytgW5SV+u1yMWECKOqymx kUHQ== X-Gm-Message-State: AElRT7HZtUSj2CzQRXimp+imxRONHK7b6PqgEIG7XulS0/PdM+cmT2mH kgAdcJsOkIcmHOSS2a0iNAE71A== X-Google-Smtp-Source: AG47ELsOH4KnDbSfni2Qg2u7moRmqjCZx7iqbx0Zjh+Tfm4nexI4oISah+CeG2pCzPG17vIqH0fdpQ== X-Received: by 10.99.125.75 with SMTP id m11mr309908pgn.391.1520810349855; Sun, 11 Mar 2018 16:19:09 -0700 (PDT) Received: from kelvin.mannynkapy.net ([207.102.139.9]) by smtp.gmail.com with ESMTPSA id c184sm13525824pfc.0.2018.03.11.16.19.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 11 Mar 2018 16:19:09 -0700 (PDT) Reply-To: tbr@acm.org Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT References: <20180306173455.oacyqlbib4sbafqd@ler-imac.lerctr.org> <201803061816.w26IGaW5050053@pdx.rh.CN85.dnsmgr.net> <20180306193645.vv3ogqrhauivf2tr@ler-imac.lerctr.org> <20180306221554.uyshbzbboai62rdf@dx240.localdomain> <20180307103911.GA72239@kloomba> <20180311004737.3441dbf9@thor.intern.walstatt.dynvpn.de> To: freebsd-current@freebsd.org From: Tom Rushworth Message-ID: Date: Sun, 11 Mar 2018 16:19:05 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Mar 2018 23:19:11 -0000 Hi All, On 11/03/2018 13:43, Jeff Roberson wrote: [snip] > > Hi Folks, > > This could be my fault from recent NUMA and concurrency related work. I > did touch some of the arc back-pressure mechanisms. First, I would like > to identify whether the wired memory is in the buffer cache. Can those > of you that have a repro look at sysctl vfs.bufspace and tell me if that > accounts for the bulk of your wired memory usage? I'm wondering if a > job ran that pulled in all of the bufs from your root disk and filled up > the buffer cache which doesn't have a back-pressure mechanism. Then arc > didn't respond appropriately to lower its usage. > > Also, if you could try going back to r328953 or r326346 and let me know > if the problem exists in either. That would be very helpful. If anyone > is willing to debug this with me contact me directly and I will send > some test patches or debugging info after you have done the above steps. > > Thank you for the reports. > > Jeff [snip] I'm seeing this on 11.1 stable r330126 with 32G of memory. I have two physical storage devices (one SSD, one HD) each a separate ZFS pool and I can reproduce this fairly easily and quickly with: cp -r The directory being copied has about 25G (from du -sg), I end up with 16G wired after starting with less than 1G. After the copy: sysctl vfs.bufspace --> 0 Out of curiosity I copied it back the other way and drove the wired memory to 26G during the copy falling back to 24G once the copy finished, with vfs.bufspace at 0. I'm not really in a good position to roll back to r328953 (or anything much earlier), my graphics HW (i915) needs something pretty recent. I am running a custom kernel (I dropped a lot of the newtwork interfaces), so if you need more info I'm willing to help, as long as you explain what you need in short words :). (I'm not very familiar with FreeBSD kernel work or sysadmin.) Regards, -- Tom Rushworth