From nobody Sun Aug 7 17:57:50 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M16W86Dtqz4YYkM for ; Sun, 7 Aug 2022 17:57:56 +0000 (UTC) (envelope-from satanist+freebsd@bureaucracy.de) Received: from smtp1.bureaucracy.de (smtp1.bureaucracy.de [IPv6:2a01:138:9000:bc42::15e4:2d4d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp1.bureaucracy.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M16W74cntz3wfR for ; Sun, 7 Aug 2022 17:57:55 +0000 (UTC) (envelope-from satanist+freebsd@bureaucracy.de) Received: from localhost (p200300e5b71d26001027fa3a9bef1d75.dip0.t-ipconnect.de [2003:e5:b71d:2600:1027:fa3a:9bef:1d75]) by smtp (OpenSMTPD) with ESMTPSA id c8d7149e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Sun, 7 Aug 2022 17:57:52 +0000 (UTC) Date: Sun, 7 Aug 2022 19:57:50 +0200 From: Philipp Takacs To: questions@freebsd.org Subject: Re: zfs and git upload-pack Message-ID: <20220807195750.0233e2f3@bureaucracy.de> In-Reply-To: References: <20220807102839.7c69f387@bureaucracy.de> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4M16W74cntz3wfR X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of satanist+freebsd@bureaucracy.de has no SPF policy when checking 2a01:138:9000:bc42::15e4:2d4d) smtp.mailfrom=satanist+freebsd@bureaucracy.de X-Spamd-Result: default: False [-2.10 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15598, ipnet:2a01:138::/32, country:DE]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[questions@freebsd.org]; DMARC_NA(0.00)[bureaucracy.de]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: N On Sun, 7 Aug 2022 09:54:41 -0700 David Christensen wrote: > On 8/7/22 01:28, Philipp wrote: > > Hi all > > > > I host a quite uncommon git repository mostly out of binary files. I > > have the problem every time this repo is cloned the host allocate > > memory and going to swap. This leads to the host being unusable and > > need to force rebooted. > > > > The repo is stored on a zfs and nullmounted in a jail to run the git > > service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and 4GB > > swap. > > > > What I have noticed is that the biggest memory consumtion is from > > mmap() a pack file. For the given repo this has the size of 6,7G. I > > suspect this file is mapped in memory but not correctly > > handled/unmaped (by the kernel) when not enough memory is available. > > > > I have tested some options to solve/workaround this issue: > > > > * limit the zfs ARC size in loader.conf > > * zfs set primarycache none for the dataset > > * limit datasize, memoryuse and vmemoryuse via login.conf > > * limit git packedGitLimit > > > > None of them have solved the issue. > > > > On my last test I have noticed that the memory is not freed after > > the git commands are stopped. To free the memory I had to unmount > > the zfs dataset: > > > > satanist@confusion:~ $ freecolor -om > > total used free shared buffers > > cached Mem: 3923 3881 41 0 > > 0 0 Swap: 4096 456 3639 > > satanist@confusion:~ $ sudo zfs umount zroot/data/git/bigrepository > > satanist@confusion:~ $ freecolor -om > > total used free shared buffers > > cached Mem: 3923 278 3645 0 > > 0 0 Swap: 4096 456 3639 > > > > Any idea how to debug this or what else I could try solve this? > > > > Philipp > > > Add memory; the more, the better. Can you borrow me your credit card? Jokes aside. this repo gets cloned a few times a month. Currently the Host dies because one client try to clone this repo. This should be work with 4GB memory. Philipp