From owner-freebsd-hackers@freebsd.org Mon Nov 19 14:40:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F0EE1105492 for ; Mon, 19 Nov 2018 14:40:04 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F64C77A17; Mon, 19 Nov 2018 14:40:03 +0000 (UTC) (envelope-from sblachmann@gmail.com) Received: by mail-ot1-x332.google.com with SMTP id a11so24169506otr.10; Mon, 19 Nov 2018 06:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=XfiBqUVncJFse1tDHMD0T4nyp9VP77uipHHTQ9L+5WE=; b=c7S331aZ8J4674ZZMNhQoVk432VkmTodS3QHzii2pRwh7m1TZtJyByhwajo3KR2iee y5hWcdbEVR0GFHGHWZiQEuqppp04cMmn7GrT9b+KyQBkHdAYEghVW8ENg8t0CPYomTVB +ytmPO2YSdqEtyL9stjd4bbjxuZQuABHaZGbOF7POTYgkUxjTn7b1X342iw9lZm3KQSG regdbRIJUv5i+xgidSBVlSSaHflBNHM20JlUxEz81pGiwkSGTa2CO2Ur6AMR88g9rzml F/iJ8jwn4W5DTiG3skr9u3ilvVNfnmZBDTVYDOyYo8Lt5lgkkaXTBf53WMjNd8ZA0sRv 3q4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=XfiBqUVncJFse1tDHMD0T4nyp9VP77uipHHTQ9L+5WE=; b=VMm1YTp+ZlKS+OvbAVTWIDEuxy72NO1iiHvRBe9UJZc369qlYeSWMnlB8MiFIt7klw U7207GeOfWPi6z/ZF7IcY9uU2cmMkNRCq9fEJoAH0Pm9Rh9NcCKN63U1h2QcHeSyurry Dfd2etU1qlxzE0vbDuu3/ovFqL4NNsTjrjCGWy1G814IP2JxbfBbDB7eBBXNsOk9xHp4 R3zzPJFGkVG69k8uQV4q+YigCEwZP1vOUoQFHGJ4ghyb48XUGAoEpbQpWnwcnTGHdmB8 T747Mafg97NOZvcwLqOB6uOBQy0lhiE4LpdHSB60txhAky0lwtWSEyTgM8/qwZilk7jo C48w== X-Gm-Message-State: AGRZ1gIbOiOeq4vhX7NoYNYbTMnElzhLPZ3TiYdcxKKDqKMVgl1Nk50x yCCiJlJ+WFFCQPRmjf5PGz3tFjlceIe2cH2uQYWIltut X-Google-Smtp-Source: AJdET5cnMn6SsgNIWPa88nOsnbycpV/mwW2D1adM/T9D3BXhjr185Rj+BSN9iBw3jJCj21IOnenxLVCvnpxQgTgJfIA= X-Received: by 2002:a9d:775a:: with SMTP id t26mr12450952otl.32.1542638402717; Mon, 19 Nov 2018 06:40:02 -0800 (PST) MIME-Version: 1.0 Received: by 2002:ac9:3052:0:0:0:0:0 with HTTP; Mon, 19 Nov 2018 06:40:01 -0800 (PST) In-Reply-To: References: <201811182205.wAIM543W036241@slippy.cwsent.com> From: Stefan Blachmann Date: Mon, 19 Nov 2018 15:40:01 +0100 Message-ID: Subject: Re: 13-CURRENT: several GB swap being used despite plenty of free RAM To: Wojciech Puchar Cc: Cy Schubert , Mark Millard , Rebecca Cran , freebsd-hackers Hackers , Mark Johnston , Ian Lepore Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 8F64C77A17 X-Spamd-Result: default: False [-3.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[2.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.95)[-0.953,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.85)[ipnet: 2607:f8b0::/32(-2.49), asn: 15169(-1.69), country: US(-0.09)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 14:40:04 -0000 On 11/19/18, Wojciech Puchar wrote: >> can go a long way. Having said that, it's been a while since I've had >> to do this. The updates made to ZFS ARC and NUMA have allowed me to >> rely on the algorithms baked into FreeBSD. My 8 GB systems have >> performed rather well. > > 8GB is LOT LOT LOT of memory. > This might indeed be a cue. Actually, I *never* had such annoying swap lag issues, until I replaced my old computers (8GB) with newer ones (24-48GB). With this increased memory, the ZFS ARC issue became apparent, causing massive problems when ARC used more than 40 of the 48GB, starving programs from memory, but this can be tuned easily. And the "preemptive swapping" issue became apparent, too. Letting programs idle for some time caused swapping out. This led to a very bad user experience... often when one changes to another desktop/virtual screen to another program, there is a delay of swapping in, which in some cases could last for seconds when hundreds of megabytes were swapped in again, in spite of some tens of gigabytes of memory that never have been used since system boot. > and my servers that have LOTS of httpd servers each for one webpage which are usually rarely visited. I wondered why my test server had a delayed response time the first few times I call a page from it after having it idled for some time. To me it looks like that the threads apache keeps cached in memory for fast response times got swapped out and the server response is slow until all these have become "memory resident" again. As the bad response times due to unnecessary preemptive swapping would induce Google downranking for sites that aren't visited sufficiently frequently enough to keep them from being swapped out, I consider this a very very serious problem. Disabling swap (either totally or for the affected programs/users/jails using rctl) and taking care that memory never runs out was the only way I was able to fix these issues. And I find very interesting that this seems to occur *only* on machines with *much* memory. The complaints about this swapping behavior I saw on the forums *always* regarded machines with at least 48GB of RAM. (I am not sure whether I saw reports regarding 32GB machines, so I would suggest investigating the issue on machines >=48GB)