From nobody Fri Nov 24 22:00:00 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ScTRz1vTpz51jkN for ; Fri, 24 Nov 2023 22:00:15 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ScTRx1190z4bjr for ; Fri, 24 Nov 2023 22:00:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=x++i0phA; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::632) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-a04196fc957so346184166b.2 for ; Fri, 24 Nov 2023 14:00:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700863212; x=1701468012; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=brfeNwqDzgbWvqpwKEG2I1t/AkvMr64L/3XDX7CcKEc=; b=x++i0phAFyBWWRl+jJw7Z+74OJhp5BMnFoHz1X5k0ls/4raty5bfFIf9Xff6c3EINO bttywkBc/5GMtf5ddexvZxvJZQAKIIMiQ/r1+b7HueZofpGaMsfIBcxc5z33b4m1vO0K i3jYWR0nkfrI9lToA3HYq/MEIKKRE3WuDGjJ5X+Qu2/fl1GD8KHO3h1wQSHPr5iOBvkj WcPyCTCFRTaVHQ2Ehnkm3KoXZbNwGTJBN0PNH37EevPNGTiQH9tbGjTNlJ3fvGU6ywo8 z4R5NEjN6QOYIG89B9G9Wno7AWSPcmi+WQyKHgqoa9HZqhqutIyEUZ3jzC8oz8qmL8Tc BKkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700863212; x=1701468012; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=brfeNwqDzgbWvqpwKEG2I1t/AkvMr64L/3XDX7CcKEc=; b=PuqRv6oJqACogn76p8HMpFBSe5tye6rctvbSvwY2n/vsZE4XLr0fPfQgd7WC1fBdI+ KX+47JFfkRqiX5+ZYBbKoRJzHljjk6rEv2xhn7jBhq7eXgitsza4XoHxAmCbhV+M/DHX WUt3irzozQizvoGmWwSJQs23kLfXZw0SWSLqihHerL9Aq9wRgRV6OEcpzw/D885ZHU4t 3ZEyHozbZEF/Y6LLRnZe62uLH+z69OUQDf2usxu2u6VRHors95CM/KlMvhUTOpw3WHO2 DnBojS2cQEhfuxTC8upmvfS4jyUya3icAF9T/+z1Q8GONzGsSgWVZe6Yb6OZl7MejN/r pgzQ== X-Gm-Message-State: AOJu0YwkzKXlF7cSIqc/ZRv3jNfghxWIjZUHZtQ81lFNozoEHBtFhpan Tj0syXElHS+ZFpWySyNjtPghFWGOWEny2wXkfCf/qAZOr7wV6GM9ohw= X-Google-Smtp-Source: AGHT+IHy7vrGjWu1bzOXfDakdYLqZuR3kP4XFl+JKxRYed/zIynIP4vnSyAwn/Ea3EFUfVcWcpnO3BGC4is5mLPHekw= X-Received: by 2002:a17:906:66c1:b0:9ff:77c2:e67f with SMTP id k1-20020a17090666c100b009ff77c2e67fmr3454254ejp.33.1700863211495; Fri, 24 Nov 2023 14:00:11 -0800 (PST) List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 24 Nov 2023 15:00:00 -0700 Message-ID: Subject: Re: Time to remove sccs tags To: Brooks Davis Cc: "freebsd-arch@freebsd.org" Content-Type: multipart/alternative; boundary="000000000000887243060aed1393" X-Spamd-Result: default: False [-2.96 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.957]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; TO_DN_EQ_ADDR_SOME(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::632:from]; DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com] X-Rspamd-Queue-Id: 4ScTRx1190z4bjr X-Spamd-Bar: -- --000000000000887243060aed1393 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable OK. I've pushed a branch to github for commenting. https://github.com/bsdimp/freebsd/tree/sccs I've broken it up in a couple of ways. First, one commit per directory for the SCCS removal. I did this for Brooks to see if that's what he wanted. Then I have one final sccs removal commit for the whole tree for what I had to do by hand. I can recreate the first 12 by script (plus two fixups) Then I have a 'remove all the copyright strings prefixed by @(#) that were ifdef'd out' commit which also did minor touchup of files. I did this mostly by hand since there was just a smidge too much variation. And then a whole lot of automatic removal of sys/defs.h that had filtered through the prior removal of FreeBSD, sccs, and copyrights. This is 100% scripted with me spot checking the diffs for sanity. I'm looking for feedback on the changes, of course, but also on whether I should fold back the last two commits into the first dozen or so done by directory (total of about 35 commits)... Or should I break them up by directory as well. I'm leaning towards folding back all but the manual changes and doing it all per directory (so 12 commits plus the one manial). Once I get initial feedback, I'll either go with my lean (if there's none) or adjust course as needed. I plan to MFC these to stable/14 on a best effort basis (eg, if it doesn't apply, I'll skip that chunk, but otherwise git cherry-pick -x). I have no plans ot merge this back to 13. diffstat says for this whole series: 7893 files changed, 125 insertions(+), 18138 deletions(-) I'm plowing through a tinderbox build right now, but amd64 builds both world and kernel. Since changes like this tend to go stale quickly, I wanted to do these two steps in parallel (any breakage is trivially different, usually adding back a sys/cdefs.h include). Finally, I have a number of sys/cdefs.h changes queued as well. These are cleaning up that file. I plan on batching it with this set of changes so that there's only one 'rebuild everything' event for most people... That is, unless the sccs series turns into a long discussion... Comments? Warner On Tue, Nov 21, 2023 at 11:03=E2=80=AFAM Brooks Davis = wrote: > On Tue, Nov 21, 2023 at 09:12:48AM -0700, Warner Losh wrote: > > It's been 30 odd years since the last csrg release. They are no more. > > > > At this point I think we can safely remove the few sccs tags that remai= n > in > > the tree. The data will be there in git if we ever need it. > > > > Comments? > > Yes please. The history is there in there repo(s) and there's so much > blind copying where the context is entirely lost. > > From my perspective it would be nice to have fewer commits per-subtree > than with $FreeBSD removal. In recent spelunking I've found that for low > traffic sub-trees I'm seeing a full screen (~50 lines for me) or more of > those > logs before getting to commits with content changes due to inconsistent > formatting leading to multiple removal commits. > > -- Brooks > --000000000000887243060aed1393 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
OK. I've pushed a branch=C2=A0to github for comme= nting.

=
I've broken it up in a couple of ways.

First, one commit per directory=C2=A0for the SCCS removal. I did th= is for Brooks to see if that's what he wanted.

Then I have one final sccs removal commit for the whole tree for what I ha= d to do by hand. I can recreate the first 12 by script (plus two fixups)

Then I have a 'remove all the copyright strings = prefixed by=C2=A0@(#) that were ifdef'd=C2=A0out' commit which also= did minor touchup of files. I did this mostly by hand since there was just= a smidge too much variation.

And then a whole lot= of automatic removal of sys/defs.h that had filtered through the prior rem= oval of FreeBSD, sccs, and copyrights. This is 100% scripted with me spot c= hecking the diffs for sanity.

I'm looking for = feedback on the changes, of course, but also on whether I should fold back = the last two commits into the first dozen or so done by directory=C2=A0(tot= al of about 35 commits)...=C2=A0 Or should I break them up by directory as = well. I'm leaning towards folding back all but the manual changes and d= oing it all per directory (so 12 commits plus the one manial).
Once I get initial feedback, I'll either go with my lean (= if there's none) or adjust course as needed.

I= plan to MFC these to stable/14 on a best effort basis (eg, if it doesn'= ;t apply, I'll skip that chunk, but otherwise git cherry-pick -x). I ha= ve no plans ot merge this back to 13.

diffstat say= s for this whole series:
=C2=A07893 files changed, 125 insertions= (+), 18138 deletions(-)

I'm plowing throug= h a tinderbox build right now, but amd64 builds both world and kernel. Sinc= e changes like this tend to go stale quickly, I wanted to do these two step= s in parallel (any breakage is trivially different, usually adding back a s= ys/cdefs.h include).

Finally, I have a number of s= ys/cdefs.h changes queued as well. These are cleaning up that file. I plan = on batching it with this set of changes so that there's only one 'r= ebuild everything' event for most people...=C2=A0 That is, unless the s= ccs series turns into a long discussion...

Comment= s?

Warner

=
On Tue, Nov 21, 2023 at 11:03=E2=80= =AFAM Brooks Davis <brooks@freebsd= .org> wrote:
On Tue, Nov 21, 2023 at 09:12:48AM -0700, Warner Losh wrote:
> It's been 30 odd years since the last csrg release. They are no mo= re.
>
> At this point I think we can safely remove the few sccs tags that rema= in in
> the tree. The data will be there in git if we ever need it.
>
> Comments?

Yes please.=C2=A0 The history is there in there repo(s) and there's so = much
blind copying where the context is entirely lost.

>From my perspective it would be nice to have fewer commits per-subtree
than with $FreeBSD removal.=C2=A0 In recent spelunking I've found that = for low
traffic sub-trees I'm seeing a full screen (~50 lines for me) or more o= f those
logs before getting to commits with content changes due to inconsistent
formatting leading to multiple removal commits.

-- Brooks
--000000000000887243060aed1393-- From nobody Wed Dec 13 01:25:55 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sqd922RcSz53cdf for ; Wed, 13 Dec 2023 01:25:58 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail2.karels.net (mail2.karels.net [3.19.118.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "freebsd", Issuer "freebsd" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sqd913ywMz4ShW for ; Wed, 13 Dec 2023 01:25:57 +0000 (UTC) (envelope-from mike@karels.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=karels.net header.s=mail2 header.b="arYI//7b"; spf=pass (mx1.freebsd.org: domain of mike@karels.net designates 3.19.118.201 as permitted sender) smtp.mailfrom=mike@karels.net; dmarc=none Received: from mail2.karels.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by mail2.karels.net (8.17.1/8.17.1) with ESMTP id 3BD1PuZf033098 for ; Tue, 12 Dec 2023 19:25:56 -0600 (CST) (envelope-from mike@karels.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=karels.net; s=mail2; t=1702430756; bh=UR19XMw01y3ZmApDnXxFpokWXft5eEEYT54ykP/WFLk=; h=From:To:Subject:Date; b=arYI//7bW51BC8ltgJhtTrwMZPj+ZLKa5f5gCKUM9BPVMn/2Ogez+JG3qZTyOFQgc uwhv/Zzzkz1gaAcxtlDZg+8zrVst71T/UjrynfdDtfVmwi8rA8l4x2LuvQLt9fG8M0 /KwXQEVPbh859EG7qtK9/3MKO9tZe3tn9Sau1alMkc4ZuERJUQQOfEzkkfwuIAMf0n i8ejJ1CHSL2D913eug3Fwc3Dho7rjptQIyzeaQiiQyHpQ4EqdQ9PxPNNsrBvTuBCI0 2e2ggMlB+8TIksb3icETR7mNv8HHpaH2Zt/7JZD4D++orFqgHZe0xP6PEJ1Jyizasz EpFN2xfHXxVPg== Received: from [10.0.2.130] ([73.62.165.147]) by mail2.karels.net with ESMTPSA id XEJvEyQIeWVIgQAAs/W3XQ (envelope-from ) for ; Tue, 12 Dec 2023 19:25:56 -0600 From: Mike Karels To: freebsd-arch Subject: tmpfs is overly aggressive on memory usage Date: Tue, 12 Dec 2023 19:25:55 -0600 X-Mailer: MailMate (1.14r5964) Message-ID: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[karels.net:s=mail2]; R_SPF_ALLOW(-0.20)[+ip4:3.19.118.201]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[karels.net:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; TO_DN_ALL(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.16.0.0/14, country:US]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[karels.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Sqd913ywMz4ShW X-Spamd-Bar: --- I have been looking at tmpfs and doing some experiments. I had noticed that tmpfs defaults to using a memory limit of "available amount of memory (including main memory and swap space)", which seemed overly optimistic. It isn't as bad as I first thought, as it is looking at current free memory and swap space. However, tests writing a file until something fails caused all of swap space to be exhausted, and processes started getting killed. In my test, this started with a large memory hog, but then killed root shells, nfsd, etc. This seems bad, and the system was screwed up enough that there was no way to reboot it short of a reset. One part of the problem was that with a default size, tmpfs enforced the limit when creating a file but not when writing it. I think this is an outright bug, and I have a proposed fix in https://reviews.freebsd.org/D43010. It has the disadvantage that it can fail a write with ENOSPC due to a transient memory load spike. The memory limit is aggressive enough that the system is very close to memory exhaustion, though, as well as swap space exhaustion (assuming there is swap). However, the limit is still high enough that continuous writes were likely to cause processes to be killed and even system hangs. I decided to try increasing the memory reserve; currently tmpfs has a memory reserve threshold of 4 MB. I ended up with a default reserve based on a percentage of memory plus swap, allowing 95% of available space to be used. This works fairly well in my tests, failing writes when the system is close to the edge and paging fairly heavily, but without killing processes. However, other memory load was essentially constant; these changes cannot predict changes in other memory demand, just react to the current situation. This change is in https://reviews.freebsd.org/D43011. I'm interested in any feedback, in particular other approaches that might be more general. Mike