From owner-freebsd-current@freebsd.org Fri Dec 25 22:11:36 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7B7814CB5D3 for ; Fri, 25 Dec 2020 22:11:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D2h4762Vlz4RfY for ; Fri, 25 Dec 2020 22:11:35 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qv1-xf30.google.com with SMTP id et9so2569986qvb.10 for ; Fri, 25 Dec 2020 14:11:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=cDoSol0P888vSPZcqWwMRefWZ16G3k7Vw6L1Yj5luw4=; b=amrGzDZkEU51iuc2GaGdSTqozHhlprQOhNPZOxPy8cobvDlj+ox5RWTTlJgfqF6pgN f2vReVEobEiWzkDrF49i7aPmCWSev+QENwgG45meYjOh7D34vgzuksB9Rq4FJn6w+hJV hCLrLMzVm3OF1OB3c9ifllD5GtZz49Gd5FbWsZ3ZGsopiEmNo/nrPap0Uoimm8mRoMBm rL5jhpo0ujKMaaEMP9zc5F5Kmt/drL8QCj56WAmTq6z/CgkmOlMuTFc7BhkhqQqMpomx rZqVDcv1AwPzBKZu4gz9lWzriPpu+PpNqVn9rlUezW1vYOLccMieubJGy778m8I9vNrW tMNQ== X-Gm-Message-State: AOAM533b98A992GXOuXTJOXLoXI5pLNyLhdqQqKyE6gnAMc/KyAmgDav 4tzMA+WsdbVN9Afeyw7EhMDaz12U5xpAx8gF6+Lyp90CeWPDlg== X-Google-Smtp-Source: ABdhPJwKE0jU0JCRmiK+GRnFWtZam2WSNnfU+bzyHSBa6k1g6VULRecQ7pzzG8QhrwAknVDZyfXBVhzAKAYZAZ+bffE= X-Received: by 2002:ad4:5b82:: with SMTP id 2mr37225762qvp.28.1608934294501; Fri, 25 Dec 2020 14:11:34 -0800 (PST) MIME-Version: 1.0 References: <20201223143545.Wf_Ww%steffen@sdaoden.eu> <20201225214041.jVKMU%steffen@sdaoden.eu> In-Reply-To: From: Warner Losh Date: Fri, 25 Dec 2020 15:11:23 -0700 Message-ID: Subject: Re: src: continued use of Subversion for getting updates To: FreeBSD Current X-Rspamd-Queue-Id: 4D2h4762Vlz4RfY X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::f30:from:127.0.2.255]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f30:from]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::f30:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; MAILMAN_DEST(0.00)[freebsd-current]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Dec 2020 22:11:36 -0000 [[ stupid gmail sent this too fast :( ]] On Fri, Dec 25, 2020 at 3:07 PM Warner Losh wrote: > > > On Fri, Dec 25, 2020 at 2:41 PM Steffen Nurpmeso > wrote: > >> Ulrich Sp=C3=B6rlein wrote in >> : >> |On Wed, 2020-12-23 at 15:35:45 +0100, Steffen Nurpmeso wrote: >> |>Jeffrey Bouquet wrote in >> |> : >> |>|On Wed, 23 Dec 2020 11:13:07 +0100, Johan Hendriks >> > |>|om> wrote: >> |>|> On 23/12/2020 09:49, Warner Losh wrote: >> |>|>> On Wed, Dec 23, 2020, 1:48 AM Graham Perrin < >> grahamperrin@gmail.com> \ >> |>|>> wrote: >> |> ... >> |>|> First of all a big thank you for all your time and effort you and >> all >> |>|> the other people put in this tremendous task. >> |> >> |>Yes, it is great to have FreeBSD as a stable git repository now, >> ... >> |>I really dislike that vendor imports have been tagged. Because >> |>there is only one tag namespace you cannot avoid getting all this >> |>cruft. I mean, it is too late now, but one could have used >> |>per-vendor import branches and step them via "git rm -rf '*' && >> |>tar -xf newball && git add . && git commit bla" or whatever, and >> |>then join them in. It does not matter for those who have all the >> | >> |That's basically what was done? I don't understand what you're saying >> |here ... >> >> Well, cgit-beta did not have had all these tags if i recall >> correctly, did it? I mean it has been two months or so since >> i last had it because "git fetch" bailed here due to the errors >> that i have reported, and fetching more than a gigabyte for >> brand-new fetches devastates here. >> > > It had them, but not under the refs/head/vendor space but under the > refs/vendor space. > > The multiple gigabyte fetch is because we changed the hashes two or three > times in the last few months. > > But i _think_ all the tags below refs/tags/vendor/ like >> >> vendor/wpa/2.9 >> vendor/wpa_supplicant/0.3.8 >> vendor/wpa_supplicant/0.5.10 >> vendor/wpa_supplicant/0.5.11 >> vendor/x86emu/4.6 >> vendor/xe/1.13 >> >> etc. did not exist in cgit-beta? I surely would have said >> something once comments have been requested, wouldn't i? >> > > They did exist. They were under refs/vendor rather than refs/head/vendor > though. > > >> The thing is if i do >> >> #?0|kent:free-src.git$ git ls-remote|wc -l >> From https://git.freebsd.org/src.git >> 6814 >> >> This is a tremendous amount of head references that need to be >> compared. >> >> #?0|kent:free-src.git$ cd ../net-src.git/ >> #?0|kent:net-src.git$ git ls-remote|wc -l >> From https://github.com/NetBSD/src.git >> 609 >> #?0|kent:net-src.git$ cd ../open-src.git/ >> #?0|kent:open-src.git$ git ls-remote|wc -l >> From https://github.com/openbsd/src.git >> > > 34 > > These are both artificial repos, that are export-only views. > #?0|kent:open-src.git$ cd ../dfly-src.git/ >> #?0|kent:dfly-src.git$ git ls-remote|wc -l >> From git://git.dragonflybsd.org/dragonfly.git >> 349 >> > > You can twek > You can tweak the default refs that you pull to pull less. > Linux is a little bit more >> >> #?0|kent:linux.git$ git ls-remote|wc -l >> From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ >> 7019 >> >> but this specific repository tracks all release candidates and all >> the update releases, which alone go into the hundreds. >> >> Anyhow, the difference is the number of local references that >> effectively have to be compared against remote heads. For example >> my local Linux repo is this: >> >> [remote "origin"] >> url =3D https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.g= it/ >> fetch =3D +refs/heads/linux-4.19.y:refs/remotes/origin/linux-4.19.y >> fetch =3D +refs/heads/linux-5.10.y:refs/remotes/origin/linux-5.10.y >> fetch =3D +refs/heads/master:refs/remotes/origin/master >> >> And if i "sr" (show-ref) i get: >> >> #?0|kent:linux.git$ git sr|wc -l >> 925 >> >> which is a lot due to Linux update policy, but it only relates to >> the project itself, there is not one reference of what is >> effectively vendor data like for example >> >> Merge tag 'wireless-drivers-next-2020-12-12' of >> git:// >> git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next >> >> may it be in refs/tags or anywhere else, whereas with FreeBSD and >> >> [remote "origin"] >> url =3D https://git.freebsd.org/src.git >> fetch =3D +refs/heads/releng/5.5:refs/remotes/origin/releng/5.5 >> fetch =3D +refs/heads/releng/6.4:refs/remotes/origin/releng/6.4 >> fetch =3D +refs/heads/releng/7.4:refs/remotes/origin/releng/7.4 >> fetch =3D +refs/heads/releng/8.4:refs/remotes/origin/releng/8.4 >> fetch =3D +refs/heads/releng/9.3:refs/remotes/origin/releng/9.3 >> fetch =3D +refs/heads/releng/10.3:refs/remotes/origin/releng/10.4 >> fetch =3D +refs/heads/releng/11.4:refs/remotes/origin/releng/11.4 >> fetch =3D +refs/heads/releng/12.2:refs/remotes/origin/releng/12.2 >> fetch =3D +refs/heads/stable/12:refs/remotes/origin/stable/12 >> fetch =3D +refs/heads/main:refs/remotes/origin/main >> >> there is >> >> #?0|kent:free-src.git$ git sr|wc -l >> 2137 >> >> but if i go for "the real" FreeBSD itself it is just >> >> #?0|kent:free-src.git$ git sr | grep -v vendor | wc -l >> 19 >> > You might be happier tracking on github, once we start pushing there as the vendor branches won't be published there. > namely >> >> a62107ed19a1095158f454132a3b6ec536a4de7c refs/remotes/origin/main >> 8411c9ac24aabdbbde468778b35da58dc3c15178 refs/remotes/origin/releng/10= .4 >> 4adbf1f6686ffdc4c0555a018e14ec63b5981534 refs/remotes/origin/releng/11= .4 >> 2120d07af09cb830873554ba5405c5d3e51b41cc refs/remotes/origin/releng/12= .2 >> d4c364b3dcf6dfe4e20c70d31912aad7e6219c14 refs/remotes/origin/releng/5.= 5 >> 0df48150179039ce25e09b48c85aa39bffdbb4c4 refs/remotes/origin/releng/6.= 4 >> 1c7fe7463d0204f08806b9da6d5d50fb7f75c5ad refs/remotes/origin/releng/7.= 4 >> 554a1309dbd71bb59ed4e97ea5e0bc829dad0f04 refs/remotes/origin/releng/8.= 4 >> b06b7e647fe7b4eefbd29369cf0c24e1902bf72a refs/remotes/origin/releng/9.= 3 >> f4d0bc6aa6b90cbb0ea6cb993d9a10e36f5f4a4c refs/remotes/origin/stable/12 >> aed278eb8da79777d64e06bbc94ee0a018885477 refs/tags/release/10.3.0 >> 20dbcb9a99d5c4b8ceb4897f27c53ab9fb853167 refs/tags/release/11.4.0 >> 906434f95e83f03e0ac62d2544dad1656b5df1e9 refs/tags/release/12.2.0 >> 1185954e6d623ada4ef32f25687ecd5e5c003545 refs/tags/release/5.5.0 >> 44b4768239e96f6d6ba8a43cb2542e1bc83dec94 refs/tags/release/6.4.0 >> c31c07d1a9283bd28bb00ce519dfca69e529f452 refs/tags/release/7.4.0 >> 174c77559c8cc317c743876f55f02bec233735f4 refs/tags/release/8.1.0 >> 45cf4d6a59488ea4113bde749bb61de33882cc46 refs/tags/release/8.4.0 >> 1fb916120b12fdc14c7f93aa5f14849db0efa166 refs/tags/release/9.3.0 >> >> and thus >> >> #?0|kent:free-src.git$ git sr | grep vendor | wc -l >> 2118 >> >> Which is a pity since all these references will be checked during >> "git fetch" unless i am mistaken. > > Yes. So far it's been doing it quite quickly for me, but I'm decently connected... |>repository, but you decided to loose one of the strengts of git, >> |>selective tracking. Also i think it causes updates to require >> |>more network traffic, i see this with the repos i have at >> |>repo.or.cz, the one with few heads/tags is minimal, the other >> |>requires hundreds of kilobytes just for the check that happens >> |>many times a day. All these references have to be compared each >> |>and every time. I think. On the other hand, a few years back >> |>i accidentally "heard" a discussion about improving the network >> |>protocol and that "head" reference matching, iirc, so it may >> |>change in the future. >> | >> |That's a valid point, we debated whether to keep vendor tags and >> decided >> |for now to replicate what we have in SVN. We can still delete all the >> |vendor tags on the main repo anytime we want ... >> >> I personally would track that in the commit message of the import >> on the vendor branch that anyway exists(!), and then when merging >> this into the mainline, but not create a real tag in the tag >> namespace. Also the backups/ and such, because why? >> > We need tags to keep track of what's been done, and to revert and do other management things with vendor imports. Warner