Date: Sun, 7 Aug 2022 20:52:55 -0700 From: David Christensen <dpchrist@holgerdanske.com> To: questions@freebsd.org Subject: Re: zfs and git upload-pack Message-ID: <62ced745-9db4-021f-ae0a-fdb4aba03a13@holgerdanske.com> In-Reply-To: <20220807211348.401ee1c3@bureaucracy.de> References: <20220807102839.7c69f387@bureaucracy.de> <a94ec43d-8ae8-fd20-a5a9-0e1e67750107@holgerdanske.com> <20220807195750.0233e2f3@bureaucracy.de> <348470bf-0f11-f7b3-e782-881c3f864ffb@holgerdanske.com> <20220807211348.401ee1c3@bureaucracy.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/7/22 12:13, Philipp Takacs wrote: > On Sun, 7 Aug 2022 11:12:20 -0700 > David Christensen <dpchrist@holgerdanske.com> wrote: > >> On 8/7/22 10:57, Philipp Takacs wrote: >>> On Sun, 7 Aug 2022 09:54:41 -0700 >>> David Christensen <dpchrist@holgerdanske.com> wrote: >>> >>>> On 8/7/22 01:28, Philipp wrote: >>>>> Hi all >>>>> >>>>> I host a quite uncommon git repository mostly out of binary >>>>> files. I have the problem every time this repo is cloned the host >>>>> allocate memory and going to swap. This leads to the host being >>>>> unusable and need to force rebooted. >>>>> >>>>> The repo is stored on a zfs and nullmounted in a jail to run the >>>>> git service over ssh. The host is a FreeBSD 13.1 with 4GB RAM and >>>>> 4GB swap. >>>>> >>>>> What I have noticed is that the biggest memory consumtion is from >>>>> mmap() a pack file. For the given repo this has the size of 6,7G. >>>>> I suspect this file is mapped in memory but not correctly >>>>> handled/unmaped (by the kernel) when not enough memory is >>>>> available. >>>>> >>>>> I have tested some options to solve/workaround this issue: >>>>> >>>>> * limit the zfs ARC size in loader.conf >>>>> * zfs set primarycache none for the dataset >>>>> * limit datasize, memoryuse and vmemoryuse via login.conf >>>>> * limit git packedGitLimit >>>>> >>>>> None of them have solved the issue. I would restore them to previous values. >>> this repo gets cloned a few times a month. Currently >>> the Host dies because one client try to clone this repo. What happens if the clone is attempted by a different user on the same workstation? What happens if the clone is attempted from another workstation? >> Please post console sessions that demonstrate cloning without failure >> and cloning with failure. > > Not sure what you mean. Please post client console sessions that demonstrate correct operation and failed operation. It may be helpful to enable verbose and/or debug output. If the output is large, reduce by hand, filter with grep(1), and/or put it into files, compress, upload, and post the URL's. > This is a server, a client connect with a > git client over ssh and use git-upload-pack https://git-scm.com/docs/git-upload-pack > to receive the content of > the repo. The communication of the git client and git-upload-pack works > with stdin/stdout. I can give the logs of my git authorization handler > (inside jail): <snip> What file? > The last line mean the clone was finished[0]. But at this time > everything else on the host was unusable. Here the corresponding content > of /var/log/messages: <snip> > Between 12:00 and 14:00 the server started to be slow and running > ssh/mosh session stopped working. Starting new sessions over ssh was > not possible. The root login at 14:38 was me over ipmi try to somehow > get the the host working again. But I could login and only run top then > this session was also unusable. I have then restarted the server over > ipmi. It looks like the server is getting overloaded with incoming TCP packets, the client is closing connections, the client is timing out when reconnecting, etc.. Near the end, I see jails being killed. I do not see reasons why. Perhaps there are clues in other logs. Perhaps you can increase the logging verbosity of the Git and/or SSH services to obtain clues. Start the following command in a terminal on the server to monitor ZFS disk activity (press Ctrl+C to exit): # zpool iostat -v 60 Start the following command in another terminal on the server to monitor CPU and/or IO activity (press 'm' to switch between the two) (press 'q' to exit): # top -S -s 60 David
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?62ced745-9db4-021f-ae0a-fdb4aba03a13>