From owner-freebsd-git@freebsd.org Tue Dec 1 15:17:25 2020 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 126784AC106 for ; Tue, 1 Dec 2020 15:17:25 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Clm1J731nz3w7b; Tue, 1 Dec 2020 15:17:24 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-oi1-f170.google.com with SMTP id v202so2069528oia.9; Tue, 01 Dec 2020 07:17:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=32EeU+GH8eiGN/d3fmV5Fe4QgzOXxrtdutM8Fr2mGkQ=; b=trV5f1UZyAfNHt9fqiYCikUqtpFPQ+vJG2l6egaFyKvJGT/ZlQVoCjYic5UUqvixyx LrBB/Jdev1DpaeIuksHJ3DeTE6BUnrjaoCqehHN8mbBiAI4m85FGgi1Zj9Hh0NppqT2/ zTfp3gF/MJDlN/dWDlZkUgmEkZhMVbST3fiQN1eNYVxAM+MQbLtuEX1sk6tD7jN9RZfq yrP1xokqNQfuI3GnLyKDmeLrCcUA3ov4ArPK8lLMF50+7xxUkbntxVd7Fo4tE0/jR34x 1IiOUN2WbhGZlMKyEJafyXbJJXBvIsiX2I1jxgn+oy6tC9pVT3IqlFRAtlWLFvAKpI7X cxUg== X-Gm-Message-State: AOAM533GdQYU76UACcrhxvKs52d8j/x/jTFz98M0aiGNH8gsToxGEfJx GrjqhbS//LlMc835XQAoKA7G5lf4oeqTQD6kJm6/8Hrw X-Google-Smtp-Source: ABdhPJyzZAmq5s3AVS/y/mhjaJQst1/ie8eAYWVoXidDa8DGINgHpQS0wEr4mJdJXqkI02Rj/JFJwhw32Oh8XBZIXlU= X-Received: by 2002:a54:4d8f:: with SMTP id y15mr2066219oix.150.1606835843435; Tue, 01 Dec 2020 07:17:23 -0800 (PST) MIME-Version: 1.0 References: <20201112155659.GQ913@mail> <20201113.032709.2108746957258946268.yasu@utahime.org> <20201130150642.GB6221@mail> <20201201095218.GC6221@mail> In-Reply-To: <20201201095218.GC6221@mail> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Tue, 1 Dec 2020 16:17:12 +0100 Message-ID: Subject: Re: 504 errors from cgit-beta To: =?UTF-8?Q?Dav=C3=AD=C3=B0_Steinn_Geirsson?= Cc: Li-Wen Hsu , freebsd-git@freebsd.org X-Rspamd-Queue-Id: 4Clm1J731nz3w7b X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Dec 2020 15:17:25 -0000 On Tue, Dec 1, 2020 at 10:52 AM Dav=C3=AD=C3=B0 Steinn Geirsson wrote: > On Mon, Nov 30, 2020 at 10:24:50PM +0100, Ulrich Sp=C3=B6rlein wrote: > > On Mon, 2020-11-30 at 15:06:42 +0000, Dav=C3=AD=C3=B0 Steinn Geirsson w= rote: > > > On Fri, Nov 13, 2020 at 05:33:12PM +0800, Li-Wen Hsu wrote: > > > > On Fri, Nov 13, 2020 at 2:28 AM Yasuhiro KIMURA > wrote: > > > > > > > > > > From: Dav=C3=AD=C3=B0 Steinn Geirsson > > > > > Subject: 504 errors from cgit-beta > > > > > Date: Thu, 12 Nov 2020 15:56:59 +0000 > > > > > > > > > > > We are getting frequent 504 errors when running `git fetch` > against an > > > > > > existing checkout of `ports.git` from > https://cgit-beta.freebsd.org/ports.git: > > > > > > > > > > > > ``` > > > > > > $ git fetch cgit-beta > > > > > > error: RPC failed; HTTP 504 curl 22 The requested URL returned > error: 504 > > > > > > fatal: the remote end hung up unexpectedly > > > > > > ``` > > > > > > > > > > I experienced same error when accessing Emacs git remository with > > > > > HTTPS. Following is bug report that I submitted to report the > issue. > > > > > > > > > > https://savannah.nongnu.org/support/?110322 > > > > > > > > > > As you can see, site administrator fixed the issue by icreasing > > > > > `fastcgi_read_timeout` and `proxy_read_timeout` parameters of > > > > > nginx. Since cgit-beta also uses nginx this may also fix your > > > > > error. In my case, however, access always failed and never > > > > > succeeded. So cause may be different from the one of my case. > > > > > > > > Thanks, I have checked this, indeed some requests' handlers don't > have > > > > a long enough timeout setting and I've relaxed them. Hope this > solves > > > > some people's issues. Please check it again, and if it still fails > for > > > > you, we might need to have more information to debug. > > > > > > This problem disappeared after your changes, but as of this weekend > seems > > > to be happening again: > > > > > > user@ssh:~/foo/ports$ git fetch -v cgit-beta > > > POST git-upload-pack (gzip 3272 to 1703 bytes) > > > POST git-upload-pack (gzip 2577 to 1354 bytes) > > > error: RPC failed; HTTP 504 curl 22 The requested URL returned error: > 504 Gateway Time-out > > > fatal: the remote end hung up unexpectedly > > > > > > Is it possible some web server config got overwritten during the last > > > batch of changes? > > > > This is most definitely fallout from the commit hashes changing. That > means > > your client will upload basically all the hashes or packs for the serve= r > to > > compare what it does and does not have. > > > > What is your up/downstream bandwidth situation like? Could you try some > more > > tracing as outlined here: > https://stackoverflow.com/questions/27442134/git-fetch-hangs-on-git-uploa= d-pack > > What sort of custom work do you have in there (branches, etc)? I'm > curious > > to find out a way to reset this non-destructively ... and I have an ide= a. > > Up/downstream should be good. Speedtests show ~100-160Mbit/s in both > directions. Cloning a repo from cgit-beta.freebsd.org I see 7.75 MiB/s. > > The checkout I was working from had two branches: `upstream` which is > a 1:1 clone of the state of the `main` branch on cgit-beta, and `main` > which is the same but also has a couple of local ports in commits that > get rebased on top of `upstream` when it's updated. When this error > occurred I was on the `upstream` branch. > > This was a manual test, but normally the same update-then-rebase process > happens as part of a CI job which was also failing. > > It seems this is fixed now, as the last 4 runs of the CI job were > successful > (first successful run was at 18:42 UTC). If I see a similar error again > I'll > follow the linked steps and send a more detailed trace. > > I saw that we had tons of loose objects in src and doc (but not in ports) and I gc'ed them today around UTC 9:00 or so. Maybe ports did auto-gc yesterday? I have some fixes to how we push things into the repo that might (or might not) reduce the number of loose objects we end up with. I'm puzzled that doc of all places would result in loose objects. For src this is expected due to the elaborate re-writes I'm doing post-conversion. Hmmm