From owner-freebsd-git@freebsd.org Tue Dec 1 09:52:24 2020 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6B5B4A3DD9 for ; Tue, 1 Dec 2020 09:52:24 +0000 (UTC) (envelope-from david@isnic.is) Received: from mx01.isnic.is (mx01.isnic.is [IPv6:2001:67c:6c:58::133]) by mx1.freebsd.org (Postfix) with ESMTP id 4ClcpJ0H8fz3Gyw; Tue, 1 Dec 2020 09:52:23 +0000 (UTC) (envelope-from david@isnic.is) Received: from localhost (wg-client01.isnic.is [185.93.159.98]) by mx01.isnic.is (Postfix) with ESMTPS id A36FB20186; Tue, 1 Dec 2020 09:52:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isnic.is; s=20200921; t=1606816340; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=zl84DIC00ohmo+ZOe1X9+9ehv2kE2VwjiWr2lQcw5bw=; b=vq2x1GDtHMKRjjU9VrqOC24L0ZHTmHhtzZbcNElzxHe7iK6k8MvlfUKIhm+u0kV/8FGXu1 MMkzsAFtm/0oFohkpOgPtGpNbPdz+RYdtkvn71C0ErsKjmRn/FgN6oDAK4N2lXODKVQNro 2nHgaKPSE8TlOzJCWxwepWHQq5LtyE1Kv1ODb/r0ZplEUCC4L1S77YsfefzbgBsBLUMU1Z q8orsrpzh+/bTs1qJw/eZF44VS7NyLKUj3Lp3PZzpB8XNQ8EdyQaUhhNLYGVwiviA+ktFU XIPfqjatYJKVgo33CTXZ9Rpr5Zp9Tafo0lMCHYE2rwKsHxjXPnsm6xBHe8YRSg== Date: Tue, 1 Dec 2020 09:52:18 +0000 From: =?iso-8859-1?B?RGF27fA=?= Steinn Geirsson To: Ulrich =?iso-8859-1?Q?Sp=F6rlein?= Cc: Li-Wen Hsu , freebsd-git@freebsd.org Subject: Re: 504 errors from cgit-beta Message-ID: <20201201095218.GC6221@mail> References: <20201112155659.GQ913@mail> <20201113.032709.2108746957258946268.yasu@utahime.org> <20201130150642.GB6221@mail> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="sXc4Kmr5FA7axrvy" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: 4ClcpJ0H8fz3Gyw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=isnic.is header.s=20200921 header.b=vq2x1GDt; dmarc=none; spf=pass (mx1.freebsd.org: domain of david@isnic.is designates 2001:67c:6c:58::133 as permitted sender) smtp.mailfrom=david@isnic.is X-Spamd-Result: default: False [-3.87 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:67c:6c:58::133:from]; R_DKIM_ALLOW(-0.20)[isnic.is:s=20200921]; FREEFALL_USER(0.00)[david]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2001:67c:6c::/48:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[isnic.is]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[2001:67c:6c:58::133:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[isnic.is:+]; NEURAL_HAM_SHORT(-0.98)[-0.977]; NEURAL_HAM_LONG(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_MIXED_CHARSET(1.11)[subject]; ASN(0.00)[asn:1850, ipnet:2001:67c:6c::/48, country:EU]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-git] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Dec 2020 09:52:24 -0000 --sXc4Kmr5FA7axrvy Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 30, 2020 at 10:24:50PM +0100, Ulrich Sp=F6rlein wrote: > On Mon, 2020-11-30 at 15:06:42 +0000, Dav=ED=F0 Steinn Geirsson wrote: > > On Fri, Nov 13, 2020 at 05:33:12PM +0800, Li-Wen Hsu wrote: > > > On Fri, Nov 13, 2020 at 2:28 AM Yasuhiro KIMURA wr= ote: > > > > > > > > From: Dav=ED=F0 Steinn Geirsson > > > > Subject: 504 errors from cgit-beta > > > > Date: Thu, 12 Nov 2020 15:56:59 +0000 > > > > > > > > > We are getting frequent 504 errors when running `git fetch` again= st an > > > > > existing checkout of `ports.git` from https://cgit-beta.freebsd.o= rg/ports.git: > > > > > > > > > > ``` > > > > > $ git fetch cgit-beta > > > > > error: RPC failed; HTTP 504 curl 22 The requested URL returned er= ror: 504 > > > > > fatal: the remote end hung up unexpectedly > > > > > ``` > > > > > > > > I experienced same error when accessing Emacs git remository with > > > > HTTPS. Following is bug report that I submitted to report the issue. > > > > > > > > https://savannah.nongnu.org/support/?110322 > > > > > > > > As you can see, site administrator fixed the issue by icreasing > > > > `fastcgi_read_timeout` and `proxy_read_timeout` parameters of > > > > nginx. Since cgit-beta also uses nginx this may also fix your > > > > error. In my case, however, access always failed and never > > > > succeeded. So cause may be different from the one of my case. > > >=20 > > > Thanks, I have checked this, indeed some requests' handlers don't have > > > a long enough timeout setting and I've relaxed them. Hope this solves > > > some people's issues. Please check it again, and if it still fails for > > > you, we might need to have more information to debug. > >=20 > > This problem disappeared after your changes, but as of this weekend see= ms > > to be happening again: > >=20 > > user@ssh:~/foo/ports$ git fetch -v cgit-beta > > POST git-upload-pack (gzip 3272 to 1703 bytes) > > POST git-upload-pack (gzip 2577 to 1354 bytes) > > error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 5= 04 Gateway Time-out > > fatal: the remote end hung up unexpectedly > >=20 > > Is it possible some web server config got overwritten during the last > > batch of changes? >=20 > This is most definitely fallout from the commit hashes changing. That mea= ns > your client will upload basically all the hashes or packs for the server = to > compare what it does and does not have. >=20 > What is your up/downstream bandwidth situation like? Could you try some m= ore > tracing as outlined here: https://stackoverflow.com/questions/27442134/gi= t-fetch-hangs-on-git-upload-pack > What sort of custom work do you have in there (branches, etc)? I'm curious > to find out a way to reset this non-destructively ... and I have an idea. Up/downstream should be good. Speedtests show ~100-160Mbit/s in both directions. Cloning a repo from cgit-beta.freebsd.org I see 7.75 MiB/s. The checkout I was working from had two branches: `upstream` which is a 1:1 clone of the state of the `main` branch on cgit-beta, and `main` which is the same but also has a couple of local ports in commits that get rebased on top of `upstream` when it's updated. When this error occurred I was on the `upstream` branch. This was a manual test, but normally the same update-then-rebase process happens as part of a CI job which was also failing. It seems this is fixed now, as the last 4 runs of the CI job were successful (first successful run was at 18:42 UTC). If I see a similar error again I'll follow the linked steps and send a more detailed trace. Thanks for your assistance, -Dav=ED=F0 >=20 > Li-Wen, we need to further up the timeouts here, freebsd repos are big, > after all. >=20 > Cheers > Uli --sXc4Kmr5FA7axrvy Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEvylfYbt7o3c60Grm/+HlKLuPmJoFAl/GElIACgkQ/+HlKLuP mJoKvQf/XmzIXD2KQeqEmOuwpyWYvmzKzZGuGgcarqVWBbbvWgtbQ01nTDZ6d9PF 6Fzf737iyiXW01awpcC5BUdZtsJT7ItLYqHSnJ1O2lQFZZkUKwxhlA231yZIR3wn 2ZzRozkAAlBKyGqpV+QZ8U7296eUU9qZU1ugGcgiVqCZvFhiVbaFMxJZE3J5fqfN fSl22R/pb0lJmjb4Wzl5U9gYrDz/zeuuRifNq5xB4abpnS4TpWyHgQU1u/jMl0/p OjWkc9kYCTjwYaacsVXX8Z5adtYcag3ZxdIRmwXw77Vu5LO6jFBWkqUdWMJ6FnqQ M4m9yBClvxATx8GJNzbwV9knBm0tjw== =26C5 -----END PGP SIGNATURE----- --sXc4Kmr5FA7axrvy--