From nobody Mon Aug 8 03:52:55 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M1Mk21HShz4Yn0V for ; Mon, 8 Aug 2022 03:53:14 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M1Mk11Bltz47Tr for ; Mon, 8 Aug 2022 03:53:13 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1659930784; bh=oHVYO7sZ5E3dxUVtM2sTXxKF89QA5Fjh3YTqdFQHhFw=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=KRdNT32ajFS8sTjJEGlJiFBumkrRKMwGj4a1anYx3n3yFzxKxjDafxMd01rq4dbBC jrz+Axyw2WSUZxnoPRDHLfhwE8IWB6I25RdVKFxiMsV5hthS31GQ0ubKGEqC9cVSX6 TPOxWZB/dkabEep99yrQdIGYCAsZ+TMTMUudUlBSES76TEmoxee3jE/ckKPaByslTV pPQvYgUXlgfuHHt+lMsgy0n/3/3d7ZCxzJHtGYBUYmtSsTS24/8X7RVeMibg0CCiJY euep2okaSFRoZ5AIW/2b7dCBX4uT5Wvinhydt1QdwsTw5IKBPfMT4k/kbtVCLPkCKn D9tHG5NiHtVEgJX8hyB6EXlBLKzZeSzvvPQEkdswc0mIYWtZqlDkl4k6S6TNosrT/u Gp/bJ0cFVFeI4rf0v+riOYHApA5VhKAOSQhnerbneCs371lcVUfRHypg+9Dcklpu8U trc+gRM6Yt3q+FYq7lvtlJ9XfIyJoGcFG/8iHMkuOUcm5YHYZb88MdDbcuOXMSX+MG MvSseVAGFkeWG0PKTUjEbwVWfysvfFtjOIHxm2YMp48BK8MfzIpczO9iTK+Dkgf1Od z011oUaLBDjoWiM8rwjtdHE1kt07wsxEru4PQ33Nxo006nqqs0idrlEIEJYh8loRjB 8yfQORBd4VZQpOkeQb1V+v1Y= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Sun, 7 Aug 2022 20:53:04 -0700 Message-ID: <62ced745-9db4-021f-ae0a-fdb4aba03a13@holgerdanske.com> Date: Sun, 7 Aug 2022 20:52:55 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Subject: Re: zfs and git upload-pack Content-Language: en-US To: questions@freebsd.org References: <20220807102839.7c69f387@bureaucracy.de> <20220807195750.0233e2f3@bureaucracy.de> <348470bf-0f11-f7b3-e782-881c3f864ffb@holgerdanske.com> <20220807211348.401ee1c3@bureaucracy.de> From: David Christensen In-Reply-To: <20220807211348.401ee1c3@bureaucracy.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4M1Mk11Bltz47Tr X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=KRdNT32a; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[holgerdanske.com:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/7/22 12:13, Philipp Takacs wrote: > On Sun, 7 Aug 2022 11:12:20 -0700 > David Christensen wrote: > >> On 8/7/22 10:57, Philipp Takacs wrote: >>> On Sun, 7 Aug 2022 09:54:41 -0700 >>> David Christensen wrote: >>> >>>> On 8/7/22 01:28, Philipp wrote: >>>>> Hi all >>>>> >>>>> I host a quite uncommon git repository mostly out of binary >>>>> files. I have the problem every time this repo is cloned the host >>>>> allocate memory and going to swap. This leads to the host being >>>>> unusable and need to force rebooted. >>>>> >>>>> The repo is stored on a zfs and nullmounted in a jail to run the >>>>> git service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and >>>>> 4GB swap. >>>>> >>>>> What I have noticed is that the biggest memory consumtion is from >>>>> mmap() a pack file. For the given repo this has the size of 6,7G. >>>>> I suspect this file is mapped in memory but not correctly >>>>> handled/unmaped (by the kernel) when not enough memory is >>>>> available. >>>>> >>>>> I have tested some options to solve/workaround this issue: >>>>> >>>>> * limit the zfs ARC size in loader.conf >>>>> * zfs set primarycache none for the dataset >>>>> * limit datasize, memoryuse and vmemoryuse via login.conf >>>>> * limit git packedGitLimit >>>>> >>>>> None of them have solved the issue. I would restore them to previous values. >>> this repo gets cloned a few times a month. Currently >>> the Host dies because one client try to clone this repo. What happens if the clone is attempted by a different user on the same workstation? What happens if the clone is attempted from another workstation? >> Please post console sessions that demonstrate cloning without failure >> and cloning with failure. > > Not sure what you mean. Please post client console sessions that demonstrate correct operation and failed operation. It may be helpful to enable verbose and/or debug output. If the output is large, reduce by hand, filter with grep(1), and/or put it into files, compress, upload, and post the URL's. > This is a server, a client connect with a > git client over ssh and use git-upload-pack https://git-scm.com/docs/git-upload-pack > to receive the content of > the repo. The communication of the git client and git-upload-pack works > with stdin/stdout. I can give the logs of my git authorization handler > (inside jail): What file? > The last line mean the clone was finished[0]. But at this time > everything else on the host was unusable. Here the corresponding content > of /var/log/messages: > Between 12:00 and 14:00 the server started to be slow and running > ssh/mosh session stopped working. Starting new sessions over ssh was > not possible. The root login at 14:38 was me over ipmi try to somehow > get the the host working again. But I could login and only run top then > this session was also unusable. I have then restarted the server over > ipmi. It looks like the server is getting overloaded with incoming TCP packets, the client is closing connections, the client is timing out when reconnecting, etc.. Near the end, I see jails being killed. I do not see reasons why. Perhaps there are clues in other logs. Perhaps you can increase the logging verbosity of the Git and/or SSH services to obtain clues. Start the following command in a terminal on the server to monitor ZFS disk activity (press Ctrl+C to exit): # zpool iostat -v 60 Start the following command in another terminal on the server to monitor CPU and/or IO activity (press 'm' to switch between the two) (press 'q' to exit): # top -S -s 60 David