From owner-freebsd-current@freebsd.org Sun Apr 4 11:51:39 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A7555D4AD4 for ; Sun, 4 Apr 2021 11:51:39 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FCsZf4HdFz4rK6; Sun, 4 Apr 2021 11:51:38 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lj1-x22f.google.com with SMTP id s17so10058809ljc.5; Sun, 04 Apr 2021 04:51:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=7OK4tdujczpMg4qonQb1VvK1xBz1eA/We0c+I9XwNZA=; b=iWicm5EbASDko5F0PfDnchGboM5LdyL4IwpPDDKYKetyL5uGLV86oS9v5t5dEJvXhs wdGlO67+kwtJBxgu3LCTjggOLRv4gnmMFPIAa64LhqpQSrUEDFSsNzbabj68062eJg97 51UtVT+ZaYOQgLNRKtKrh+NY+n3NxIqWJ8LGSVl7Tc0jaP/7EYGuK5msaJsHNegDG+xO gGY4HKJbu5QugHytmj6a0miZmvuOm2VwTxOOVWYdfCdnEYzy8ze1euh0FCRtIboW6AUL ZSd2qYKjbpWs3Rm8TY6BLya5NmqRHSFkfv17k2PWVyg11aMNFYWnWs2pZ3PjOnUlhxFk NfpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=7OK4tdujczpMg4qonQb1VvK1xBz1eA/We0c+I9XwNZA=; b=GBxERxPXLklFlT1fW6qxVCojMGKKd3bb4MvweohxBNLLFM0Orp1NQ/O9ToQlQjwnYj 3uRUnugb6udKS1UPh7JZdPDQzqhr4QWjvd3E9zO0qk1Zpk+VtVmdZG07e6WqXBwQv2zc e4I3YYK0LN/OpsPSk/IYUeJRr05vLvZkZsrQQVef/be2ylgKX2xMmw/9sRg45mnrH5/c vFTJ3uDTRNCjiD9W8VjYK6IY5CFB/3tv523lDJ4uVXYFWt+TC9H2Bxr0SutH4ClPnJTz RQ/fmmaBRf5uXmwBmhEulupKbxw7s+dOSAu88Z6KMY+H0IzJGfLgKmWwrpvAHTZPQArk iQgg== X-Gm-Message-State: AOAM532LAEbVlhudk6I1y7pb0TX5t8KWy9Yk2L8Fzun5l7JuPmyKcAIs 3eTpiXA4yRa3omsW4FjYnlQOyOtZ+0I1HbsfqpA= X-Google-Smtp-Source: ABdhPJw9D1gfJoJKROGeZ5Va39y8U1JQ0Bcni3m5OGTwIB+iBVKWmuw7lLP8oVyyIyQDtDGpOI5zItQy5HRhgY/tpo0= X-Received: by 2002:a2e:575c:: with SMTP id r28mr13338766ljd.347.1617537096715; Sun, 04 Apr 2021 04:51:36 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:b54e:0:0:0:0:0 with HTTP; Sun, 4 Apr 2021 04:51:36 -0700 (PDT) In-Reply-To: <81671.1617432659@critter.freebsd.dk> References: <58bea0f0-5c3d-4263-ebee-f939a7e169e9@freebsd.org> <494d4aab-487b-83c9-03f3-10cf470081c5@freebsd.org> <81671.1617432659@critter.freebsd.dk> From: Mateusz Guzik Date: Sun, 4 Apr 2021 13:51:36 +0200 Message-ID: Subject: Re: [SOLVED] Re: Strange behavior after running under high load To: Poul-Henning Kamp Cc: Stefan Esser , FreeBSD CURRENT Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4FCsZf4HdFz4rK6 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=iWicm5Eb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::22f as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::22f:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::22f:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22f:from]; NEURAL_HAM_SHORT(-0.99)[-0.992]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Apr 2021 11:51:39 -0000 On 4/3/21, Poul-Henning Kamp wrote: > -------- > Mateusz Guzik writes: > >> It is high because of this: >> msleep(&vnlruproc_sig, &vnode_list_mtx, PVFS, "vlruwk", >> hz); >> >> i.e. it literally sleeps for 1 second. > > Before the line looked like that, it slept on "lbolt" aka "lightning > bolt" which was woken once a second. > > The calculations which come up with those "constants" have always > been utterly bogus math, not quite "square-root of shoe-size > times sun-angle in Patagonia", but close. > > The original heuristic came from university environments with tons of > students doing assignments and nethack behind VT102 terminals, on > filesystems where files only seldom grew past 100KB, so it made sense > to scale number of vnodes to how much RAM was in the system, because > that also scaled the size of the buffer-cache. > > With a merged VM buffer-cache, whatever validity that heuristic had > was lost, and we tweaked the bogomath in various ways until it > seemed to mostly work, trusting the users for which it did not, to > tweak things themselves. > > Please dont tweak the Finagle Constants again. > > Rip all that crap out and come up with something fundamentally better. > Some level of pacing is probably useful to control total memory use -- there can be A LOT of memory tied up in mere fact that vnode is fully cached. imo the thing to do is to come up with some watermarks to be revisited every 1-2 years and to change the behavior when they get exceeded -- try to whack some stuff but in face of trouble just go ahead and alloc without sleep 1. Should the load spike sort itself out, vnlru will slowly get things down to the watermark. If the watermark is too low, maybe it can autotune. Bottom line is that even with the current idea of limiting preferred total vnode count, the corner case behavior can be drastically better suffering SOME perf loss from recycling vnodes, but not sleeping for a second for every single one. I think the notion of 'struct vnode' being a separately allocated object is not very useful and it comes with complexity (and happens to suffer from several bugs). That said, the easiest and safest thing to do in the meantime is to bump the limit. Perhaps the sleep can be whacked as it is which would largely sort it out. -- Mateusz Guzik