From nobody Mon Jul 5 16:32:00 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5B8287CE92D for ; Mon, 5 Jul 2021 16:32:03 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GJWRk5l8kz4j94; Mon, 5 Jul 2021 16:32:02 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: by mail-lj1-x229.google.com with SMTP id w11so25438638ljh.0; Mon, 05 Jul 2021 09:32:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=wwCo42yd/6YII45ZPiAxfcMuQXFGwkJzgHoyUADHtrQ=; b=eKvIqFoMZSdBuI5gftLX3C9Dhj89/JeKJRxGBr63Wo91G3aCD6sR8I9ZgPuVirCAB2 wJGL2FNPkJS3tLSqMK9ab16zSLxXYP6g4eML4Sd+UbS/tgXbfv6cJNE+/XSFi7wAO5oG YTs6IT+6jqz9z/3ltb2QGPdKKICPROczPTC0RTsMV0bzcfrqQK3eRd0uVRIHee7L3Wjf SJrYZ2gcjdM8Ulv1ivZVb7rOc95CmIBG2w+Wkk4ChzyvQYsaMBtOqaGkZl39UgR4simf YyHlrnHApmMlgnzRuFscF3L2tlSyrch5Fy7zPjUBA6JvMVQUfzPTkeXk2h0WWVC+vq4H wzVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=wwCo42yd/6YII45ZPiAxfcMuQXFGwkJzgHoyUADHtrQ=; b=kaH5EkOvydleluOxgx4TxNKmP2QWfjWE7Pdh1y64hFp+vg6GCqEWeVe6eFIj0HPaqq 3o86EajE4+qVU61bwQdVsHl6PFr3lNeLyqIpp3aA9zDt1Oy0FjkXmOmZ9CWQI0LnE+zd eySvRkrmgQyrZRzyQGOs8A7Z0kRgwMHZ8XWZ+gtWXxweIfaf6W4Uj9C9L5r5y3OJ9zAo 9/0Fz2uHwN1QUt3Y30c9ni9yAd7nwpBPNfcdnMVwSdlk8QLen6vwLJvTWF4Mm6PXWulC vtbl/vR3I/16yMVDT+EsPVtFtUGVmsllBwNyXQ93SFbdVb1ja7c1XeCKWGSOsOuh1Goy z4ug== X-Gm-Message-State: AOAM531eP5pp3SbxKHb1kK6VsJSnTz2g5ZG7663bjycxpnOEzj2PqeOQ Aarylg5W0nVVeGX9lXCFT6Q= X-Google-Smtp-Source: ABdhPJwRuGk4/hK2kPhiJDfty9mCPfZGCR3cE7g2ZlxWVnFnlCiRyk5ID4mUIky3gDhJAXCp9WGldQ== X-Received: by 2002:a2e:a4ae:: with SMTP id g14mr11954355ljm.354.1625502721258; Mon, 05 Jul 2021 09:32:01 -0700 (PDT) Received: from [10.42.0.5] ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id c13sm1413254ljf.47.2021.07.05.09.32.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Jul 2021 09:32:00 -0700 (PDT) From: Vitaliy Gusev Message-Id: <2390FA9B-319E-45D4-BEA7-10878E43AD4B@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_283679B0-317F-482E-9F4E-E3C4F0D00D24" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Re: madvise(MADV_FREE) doesn't work in some cases? Date: Mon, 5 Jul 2021 19:32:00 +0300 In-Reply-To: Cc: freebsd-hackers@freebsd.org, gljennjohn@gmail.com, Mark Johnston To: kostikbel@gmail.com References: <20210703065420.6dbafb5f@ernst.home> X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4GJWRk5l8kz4j94 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eKvIqFoM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of gusevvitaliy@gmail.com designates 2a00:1450:4864:20::229 as permitted sender) smtp.mailfrom=gusevvitaliy@gmail.com X-Spamd-Result: default: False [-1.78 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::229:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RECEIVED_SPAMHAUS_PBL(0.00)[188.187.60.230:received]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.28)[-0.277]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::229:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::229:from]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-ThisMailContainsUnwantedMimeParts: Y --Apple-Mail=_283679B0-317F-482E-9F4E-E3C4F0D00D24 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, > > Does it mean madvise() doesn't work well in FreeBSD or test does = something wrong? >=20 > Your program does not exactly what you described above. There is a = generic > race to consume memory, and some specific details about madvise(2) on = FreeBSD. >=20 > >=46rom the code, you do: > - mmap anonymous private region > - fork > - both child and parent start touching the mmaped region. >=20 > Two processes race to consume 1/2 of RAM on your system. If one of > them happen to execute faster then another, you do get to the case = where > one of them does madvise(). But it could be that processes execute in > lockstep, and try to eat all the memory before going to madvise(). > Did you excluded this case? I believe I did all things right. You can see sleeps that serialise = execution. To check again I modified test and added time printing and = use MADV_DONTNEED: Here is source http://cpp.sh/2rd4f I=E2=80=99ve run:=20 $ ./mmapfork 2300 mmap 0x801000000 pid 40628 end 0x890c00000 len 0x8fc00000 pid 40628 pid 40629 40629: [1625500831] touch 40629: [1625500832] sleep before madvise 40629: [1625500833] madvise 40629: [1625500834] Press enter to exit 40628: [1625500845] touch 40628: [1625500846] sleep before madvise 40628: [1625500851] madvise 40628: [1625500852] Press enter to exit And you can see that child started running in 11 seconds after parent = had already called madvise() for all scope of touched memory. And finally in dmesg: pid 40629 (mmapfork), jid 0, uid 1001, was killed: out of swap space So the same result as I wrote in the first email. > Now, about the specific of madvise(MADV_FREE) on FreeBSD. Due to the = way > CoW is implemented with the shadow chain of objects, we cannot drop = the > top of the shadow chain, otherwise instead of returning zeroed pages = next > time, we would return content back in the time. It was relatively = recent > discovery, see bf5661f4a1af6931ec4b6, PR 240061. >=20 Thanks, I will look at it. > To explain it in simplified form, when there is potential old content > under the CoW copy for the mapping, we cannot drop CoW-ed pages. This > is the motivation why madvise(MADV_FREE) does nothing for your = program. > When you run two instances without fork, there is no previous content > and no Cow, so madvise() can safely remove the pages from the object, > and on the next access they are zero-filled. Do I understand right, that it should work with MADV_DONTNEED? But = =E2=80=9Cdontneed" variant doesn=E2=80=99t work.=20 >=20 > You can read more details in the referenced commit, as well as some = musings > about way to make it somewhat better. >=20 > I must say, that trying to allocated 1/2 + 1/2 of RAM this way, on a = system > without swap, is the way to ask for troubles anyway. I=E2=80=99ve just notify that other operation systems work well with = that, whereas FreeBSD has troubles. Probably something in madvise() is = not finished ? ---- Vitaliy Gusev --Apple-Mail=_283679B0-317F-482E-9F4E-E3C4F0D00D24--