From nobody Sun Aug 7 18:12:20 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M16r73466z4Ybs3 for ; Sun, 7 Aug 2022 18:12:39 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [IPv6:2001:470:0:19b::b869:801b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M16r64qXtz3yJn for ; Sun, 7 Aug 2022 18:12:38 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1659895956; bh=PbwCqaleR0LULBJaxF7XPvaacBO7Jdvpe3dEgqr5gUI=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=MJNuasd13taUjMqiXkrun9gOiRixM6rAENRpqWy/I6P4tYBawC8gQiM10a/O+nrrJ ytdTdqEfpuxr7L+6rWo8t4iw4f1CKDs2ig8a4BmSmAI5/sdD7v0XbNKQhlOvAvQ7TR i/zcjVrgDXgvoEBklyLgYNtxcZ6tJboFMx6L9s/zGhkd/LlQ9sE4pFFQurIGpNrzrY gMXfCLr+8G77s2mJw7hS55fsZ6W4GCKjxVtsWH75CIVme1BAo/ihi9hoijWSzR85w2 pEv4VshMUTReRd1rHyQLcp01j48U7NbcZPBy+WkGEIdjf2/KSoiYKeFlOg821ADzOe NHG17vy3CRpi/dYeII1T3paxbn8+8xZWY20W/lTgbpXEurq6ix7x1Ey/kxjC6sSEzq yisEKrJYwTyJ+i2CkC30WokOyTRvrPNy6wugrzqqgYUq5WMpU5b4bBlxq2jvGWO4yd YH5eBwsmPeJhqspWsacl6n8gyS0rQBXjkZsxniLZ1Mqv8g5PAfOxuGzIJEhYr1lTBI H2UHy3HQOh26jE2i2GEo7KpS1rqUgHE2qKmwMZcvrx2lCBh4tcTW4AH10ZvegddYhN cHegUPXqFbAGZv4Jll+x8mZQ23Xov6qIHXwCdOx2jTgR5uFQ7UMecCcJGf3ofJhas+ T73/SItdEuFvIRWPRoyueF8w= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Sun, 7 Aug 2022 11:12:36 -0700 Message-ID: <348470bf-0f11-f7b3-e782-881c3f864ffb@holgerdanske.com> Date: Sun, 7 Aug 2022 11:12:20 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Subject: Re: zfs and git upload-pack Content-Language: en-US To: questions@freebsd.org References: <20220807102839.7c69f387@bureaucracy.de> <20220807195750.0233e2f3@bureaucracy.de> From: David Christensen In-Reply-To: <20220807195750.0233e2f3@bureaucracy.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4M16r64qXtz3yJn X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=MJNuasd1; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 2001:470:0:19b::b869:801b as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; R_SPF_ALLOW(-0.20)[+a:november.he.net]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[holgerdanske.com:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/7/22 10:57, Philipp Takacs wrote: > On Sun, 7 Aug 2022 09:54:41 -0700 > David Christensen wrote: > >> On 8/7/22 01:28, Philipp wrote: >>> Hi all >>> >>> I host a quite uncommon git repository mostly out of binary files. I >>> have the problem every time this repo is cloned the host allocate >>> memory and going to swap. This leads to the host being unusable and >>> need to force rebooted. >>> >>> The repo is stored on a zfs and nullmounted in a jail to run the git >>> service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and 4GB >>> swap. >>> >>> What I have noticed is that the biggest memory consumtion is from >>> mmap() a pack file. For the given repo this has the size of 6,7G. I >>> suspect this file is mapped in memory but not correctly >>> handled/unmaped (by the kernel) when not enough memory is available. >>> >>> I have tested some options to solve/workaround this issue: >>> >>> * limit the zfs ARC size in loader.conf >>> * zfs set primarycache none for the dataset >>> * limit datasize, memoryuse and vmemoryuse via login.conf >>> * limit git packedGitLimit >>> >>> None of them have solved the issue. >>> >>> On my last test I have noticed that the memory is not freed after >>> the git commands are stopped. To free the memory I had to unmount >>> the zfs dataset: >>> >>> satanist@confusion:~ $ freecolor -om >>> total used free shared buffers >>> cached Mem: 3923 3881 41 0 >>> 0 0 Swap: 4096 456 3639 >>> satanist@confusion:~ $ sudo zfs umount zroot/data/git/bigrepository >>> satanist@confusion:~ $ freecolor -om >>> total used free shared buffers >>> cached Mem: 3923 278 3645 0 >>> 0 0 Swap: 4096 456 3639 >>> >>> Any idea how to debug this or what else I could try solve this? >>> >>> Philipp >> >> >> Add memory; the more, the better. > > Can you borrow me your credit card? > > Jokes aside. this repo gets cloned a few times a month. Currently the > Host dies because one client try to clone this repo. This should be work > with 4GB memory. Please post console sessions that demonstrate cloning without failure and cloning with failure. David