From owner-freebsd-git@freebsd.org Mon Sep 23 20:38:53 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 21DB3FF66D for ; Mon, 23 Sep 2019 20:38:53 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cbkz5wkrz4VYS for ; Mon, 23 Sep 2019 20:38:51 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-ot1-x344.google.com with SMTP id c10so13350659otd.9 for ; Mon, 23 Sep 2019 13:38:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+Cq2E703QpIqA8nyNPBLMdzrifC0a69n6hDbcfOCP3Q=; b=eQGdXeXnHuSG5OHsJGnoGuCLVMNXhconygTVd1UliR4MnuzEbD66/xZk9KjnfyzXvY M6naZxZId9xFGdgjKakZfvCBD5DgzAJwdskU+CjejzapHbqyVgxRRKS0sXMa7k/5mGgu KY2FcCxzUkWvTLCZiqDbaTI3sabvsnPsjlxanIlNCaWVLPcsmJohL6Jfpz49bq3O8OLt NOoykZvxxikF8RbEk1K6wssG0a0WKo5eRrfuOx88OnsYN/1HIFLgHUlvUYZYilfAnY4+ PhyjrpN/NAs/UqHMi/zF4vcUD0MZ9gU7LRpZnh7ry7MUD165FF+TP7T5Yoxx9795IysA SoVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+Cq2E703QpIqA8nyNPBLMdzrifC0a69n6hDbcfOCP3Q=; b=kzhbXJPeXyqacaPDtIoXPJRWlErp/47RA1HV7B5hRuo2Vq3c/eIlo98XR5H2VkPubs w7cZNTIgelwIV5X7XK5hGYXk9CoheuGC/CcLqsicfiIN7I6rDVjDXYx6bz75e8QQrrYs kynDtxDt4rM3Mo5Cl5uMeuYg29EcrIEJLYnJ9KObCZXKbNKa1/VkFNLQhu9l6GLfSIY7 KtgFAj4SigwqyBoBs/CgN2OG34rA8v+oQgQap8f5q8tMqoaJkrSs4G09BIcdir4oCXeU eBlJPvFFBsv6c+Ruw0T5Km6DhGFI8aYtCwh8YnEJ0hfglrdl9+2zg8oGMgvhAtC4eFHB gf4Q== X-Gm-Message-State: APjAAAVDU3ausulWx2D4xDBWatj4BwR/1uuDi4Y2SyendEPp6wa7c8TH X2+cPSeGKDPpNwBrVUXPWljGCu+IseEH+2xizqI= X-Google-Smtp-Source: APXvYqxHRozgvFoQToBynRQ4FSxbYOKeJQOjL61TRD1n2I8DnvoJVr9xVyazZuwLAoKmhU0YF4wfvq8GSqJ8OS4602U= X-Received: by 2002:a9d:3b02:: with SMTP id z2mr112872otb.71.1569271130542; Mon, 23 Sep 2019 13:38:50 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> In-Reply-To: <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Mon, 23 Sep 2019 22:38:36 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: Shawn Webb Cc: Sean Chittenden , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46cbkz5wkrz4VYS X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eQGdXeXn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::344 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (3.06), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 20:38:53 -0000 Am Mo., 23. Sept. 2019 um 20:51 Uhr schrieb Shawn Webb : > > On Mon, Sep 23, 2019 at 08:42:10PM +0200, Ulrich Sp??rlein wrote: > > Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > > : > > > > > > Hey Ulrich, > > > > > > I appreciate your hard work in maintaining the git mirror. Work like > > > this can sometimes go unthanked. I want to take a moment to show > > > appreciation for you and the FreeBSD project in maintaining the git > > > mirror. > > > > > > I do have a few concerns with what was stated in your email. I've > > > written my concerns inline. I hope this discussion is a positive one, > > > wherein upstream and downstream can effectively come to a conclusion. > > > > > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > > > : > > > > >> > > > > >> Please note however, that more "garbage" metadata escaped from S= VN into > > > > >> github, meaning 3rd parties have a hard time re-running the conv= ersion and > > > > >> making sure that it matches SVN down to the metadata (i.e. times= tamps). > > > > >> > > > > >> Eventually, this will have to be re-rolled and a new "master" br= anch will > > > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > > > > > > > Wait, what? Can you elaborate? > > > > > > > > > > Discussion of a force-push to github has occurred a few times and= been explicitly ruled out because most of our corporate citizens use githu= b to integrate changes from FreeBSD. Rerolling master was universally reje= cted when we socialized wanting to do this due to the level of disruption t= his would cause. The feedback was that this would be a high-cost, low-valu= e operation. In the tradeoffs of purity vs pragmatism, pragmatism wins eve= ry time (that is the FreeBSD way). > > > > > > > > > > -sc > > > > > > > > > > > > This is not just about pragmatism and the disruption it would cause= is > > > > vastly overblown by people who don't seem to know much about the gi= t > > > > storage model. > > > > > > > > There *is* garbage metadata in the published version on github, the= re > > > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > > > forever, and the cost of switching from 1 published branch to anoth= er > > > > is literally: > > > > > > > > - git diff origin/broken_master mybranch > mybranch.patch > > > > - git checkout -b fixed_branch origin/fixed_master > > > > - patch < mybranch.patch > > > > > > Such a workflow breaks historical accuracy. Instead of `git annotate` > > > showing the history properly, it's now based on an "epoch commit". > > > Sure such a commit brings the branch to a working condition, but at > > > the cost of history. > > > > Is there really that much value in having "git blame" work in that envi= ronment? > > My mental model is of short-lived branches that get upstreamed, so I'm = biased > > towards this not being all that big of a problem (for some at least). > > > > > > > > > > It should also be possible to merge both broken and fixed master in= to > > > > your branch (at the exact same SVN revision in time) and then you c= an > > > > follow fixed_master going forward. You'll schlepp around double the > > > > commit history, but not tree objects. > > > > If you want to retain history, you can upstream the changes prior t= o > > > > the switch > > > > > > I so wish that were possible for certain downstream projects. We're > > > unable to upstream the majority of our work. To argue "upstream your > > > work and you won't be affected" is to choose an argument that does no= t > > > reflect the reality of a growing portion of FreeBSD's downstream > > > consumers: the inability to work effectively with upstream. > > > > :/ > > > > I'm 80% sure that you can just merge both branches and things will be f= ine > > (though the exact incantation will surely be black magic). I'd love to > > try this on > > an actual repo though, I don't have the time to craft some test repo to= verify > > this assumption, and then find out that other repos are different). > > HardenedBSD's github repo has existed since 2013, with branches > stemming from that work existing still today. Perhaps HardenedBSD is > somewhat in a special case: we aim to provide the BSD community with a > clean-room reimplementation of publicly-documented parts of the > grsecurity patchset. > > With FreeBSD not taking the same approach, we will have very > long-lived branches. For example, our hardened/current/master branch > follows FreeBSD's HEAD and syncs every six hours. Meaning, we maintain > our patches, resolving whatever few merge conflicts arrive. The > hardened/current/master branch was created so many years ago, I've > forgotten when it was actually created (perhaps in 2013?) > > Though HardenedBSD's cause for existence may be a special case, this > problem can be viewed in a general fashion. I'm confident HardenedBSD > is not alone in facing issues of these types. > > Thanks, What I don't understand is how a security focused project can trust a random source for the svn2git conversion. I could have planted a bunch of backdoors and then come up with some SVN metadata corruption conspiracy as to why the commit hashes are different. Why would you trust me? HardenedBSD of all people should be running the converter themselves and check that the content really matches what is in SVN (which it currently doesn't for metadata). Cheers, Uli