From owner-freebsd-git@freebsd.org Sun Sep 22 12:59:02 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7AB8D121D53 for ; Sun, 22 Sep 2019 12:59:02 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46bnZs4P6Fz3HCK for ; Sun, 22 Sep 2019 12:59:01 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f53.google.com with SMTP id u8so5393293iom.5 for ; Sun, 22 Sep 2019 05:59:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=vstuO6hOBhJ8qEMoOhG1yC37tdztPDsv+YHQt0LKRyg=; b=C10VWVJYRVbLFPs88zdbmIJAFrNEG7p0yZbo91N3P8uOKiK7uzXfmv8xyV4dwLD43q nEgkFbIas9g9rTU973saLL40Er0OVKGsI3nlaOZ0jOa9Btei4jOvBylHAhwmWrmPTg/S IdhaLpelncVmh502MQfCKGD5+lJRIHhanHlDbKEFFjWzZ2ZIXcQpGIFvZfG4piQwnp99 eP4J+j8SIwQ62UXPSGU//XjOzYu14hzvXWt2mx4CMtfPo8js6f3qsTGMq4xKvcZGW37Q /O2YI1wKkJNQK4x3jX5TxehLPanyZwcruQM6SvF0z2LTWUOYd8u4laS3+C/9bOKmRJIt N1HQ== X-Gm-Message-State: APjAAAUYCzC/0v252chL5J9lBUij47L98uh0d9udH+Q4ZqF1LyxqCePp bz1vJD18Fol8cV9u8QcxBiZ8vky/h56CyHqLDUE= X-Google-Smtp-Source: APXvYqwVEJSRjyC6oefmw5+gL5t6yjzXOgm4mY7F/bm3su5wwRquW2xklYLqjUhaP0YbTQfVc1AkVv1O+DRZXi5nTGo= X-Received: by 2002:a6b:4a19:: with SMTP id w25mr22235060iob.115.1569157140209; Sun, 22 Sep 2019 05:59:00 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Sun, 22 Sep 2019 08:58:48 -0400 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46bnZs4P6Fz3HCK X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.53 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-3.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[53.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.70)[ip: (-2.94), ipnet: 209.85.128.0/17(-3.30), asn: 15169(-2.20), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[53.166.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2019 12:59:02 -0000 On Sat, 21 Sep 2019 at 04:28, Ulrich Sp=C3=B6rlein w= rote: > > Good news and bad news, we were able to roll back the converter jail and > doc and ports are basically working again (but not turned on live yet), b= ut > the src (or base) conversion is now diverging from what is published on > github. This will take some time to figure out still. If we have to investigate things here, one additional thing we ought to take a look at is the group of stable/X commits that show up in head. One example of these is r306097 / df422cbea30e, which copied a file from stable/10 to restore a file that had been svn rm'd in head. Effectively it was a reverse MFC. From owner-freebsd-git@freebsd.org Sun Sep 22 20:49:16 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 16DA312BBC0 for ; Sun, 22 Sep 2019 20:49:16 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46c01R26dtz4Ckh for ; Sun, 22 Sep 2019 20:49:15 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-ot1-x32b.google.com with SMTP id 21so10367941otj.11 for ; Sun, 22 Sep 2019 13:49:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=2KQOu9x/V/mYNpGnQHOZKdp+ZtuIvnDteg0maVeanzs=; b=F4ivKPFidT8jh80801hqWKzzWzMgCUj8qnt+oEWe2ESXFcNc8jNU9ar0mk6M9JA+GW aSOwmUFTq0fHvP3pgbZ/n+5XrZyPCQcIIGAae/DbM3CzfNXhpjru+kJRGKGI9ceMlNAA f2/sCsC/2PxyyGnjCz7gzrpWwun/42SRTO1Oo7Td6zTZvrwfvgZXSCn8i2/tzC12wqve Jl8TxENcP6iWAy+VbR+IPvpPdxAU5EhAk0xNLCARa9iwWBQEGOZ2bZ+eBT0uFhy3eqxo lKcmwVc1KU4CxDS/iqvBKGcB9EQ7723M0tz/lRLarNeBrlm0q68myUqrv3yjg1JXjAJW xCWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=2KQOu9x/V/mYNpGnQHOZKdp+ZtuIvnDteg0maVeanzs=; b=qZeq8A5w8e0YLc6F69qA+mAJHH+Y/4XpOrb5v38/ClSttPjrvj436kpV1S8zthktad BITBID3hwc85F2JKACsOU007Ki6QFycSr9uBsDhHHTrspiUK4yoPV66ZgU5OdpI+88Ge rccncK6YitCqWyvsaPYQnKpWIqXCUAFPkEJYHZTorHKNw+hP3XN9s6hlHKj/Xm412RPo ZVfrcNjML4buXwbOy/j/dweAv/IJkPt6MohIjBxK/5QAK7hrUAEGR39by7cgUpWTaGk5 lZiWhs66bYUqorFybqf7QvWcj8Eei1mQo/6+qlu0Ql5aOj9cL2j/ucly3DI45vuCdyIo k07g== X-Gm-Message-State: APjAAAUxgZOejvmQD/WcliAAL0tDuzEKoVhcJ611tzozGN88k5YiofyH bcW7cyGY7cr/oAQ2hOkOceNLc6MimpH58bNVc9FS9A== X-Google-Smtp-Source: APXvYqxzNPzdpWR4J5wNHDTzWDDPWAs1sFX0EGXEas/OAc8sKpsqim8iRWM92az+aY5+McJ4ffubTEbMuYvtj1lohJQ= X-Received: by 2002:a9d:6014:: with SMTP id h20mr18952407otj.207.1569185353765; Sun, 22 Sep 2019 13:49:13 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sun, 22 Sep 2019 22:49:02 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46c01R26dtz4Ckh X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=F4ivKPFi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::32b as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; TO_DN_NONE(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(0.00)[ip: (-8.10), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[b.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2019 20:49:16 -0000 Good news all around, this should now be back in a fully working order. Please note however, that more "garbage" metadata escaped from SVN into github, meaning 3rd parties have a hard time re-running the conversion and making sure that it matches SVN down to the metadata (i.e. timestamps). Eventually, this will have to be re-rolled and a new "master" branch will be force-pushed into github. There's no timeline for this yet. Regards Uli Am Sa., 21. Sept. 2019 um 10:28 Uhr schrieb Ulrich Sp=C3=B6rlein < uspoerlein@gmail.com>: > Good news and bad news, we were able to roll back the converter jail and > doc and ports are basically working again (but not turned on live yet), b= ut > the src (or base) conversion is now diverging from what is published on > github. This will take some time to figure out still. > > Sorry again for the inconvenience, let me know if you have further > questions. > > Regards > Uli > > Am Do., 19. Sept. 2019 um 09:01 Uhr schrieb Ulrich Sp=C3=B6rlein < > uspoerlein@gmail.com>: > >> Folks, >> the converter is currently crashing and thus not making progress on at >> least src and ports repos. I'm also unavailable most of the next 2 days,= so >> this might take well into the weekend to get fixed. >> >> Wish me luck! >> Uli >> > From owner-freebsd-git@freebsd.org Sun Sep 22 21:16:25 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9AF212D26C for ; Sun, 22 Sep 2019 21:16:25 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46c0cn09xtz4GN4 for ; Sun, 22 Sep 2019 21:16:24 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f43.google.com with SMTP id z19so6542289ior.0 for ; Sun, 22 Sep 2019 14:16:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=eUZDxK+s+/EZ4S0/q4m1/klVn4r4TYmmJYLEXQodawo=; b=WGqg0rBu2Ae1/XWoUfvj+nN1HUVP5L+ZNKF9o0aIFshVTQcvxNS84RSYON7/+hI2pH 3L01b7cfJKBx5F8sZSKZUY5DJ8ZB/+x6IEX8JnVc8w4EYVjsqHeOM+uOFjuBZxgt7dOR eAJWKJFXyzWPoPGbiVaaiQuwQaWbOm/tSloVXqTFF8fDr/1CBWpL4OsHhqiGWRwW2KFX vNfhiGVph+NVd50BJzij0iqTemA1feuG6NXqOm4Kf4dRXRDdsCTS4Qi9tzIEOm4SChbH +t+a9br4bPxinft5876kC7UIBXaRtaYpODnERjwp0GD02D1IP6QGjzSvVsDqN9Cqm29h WYAQ== X-Gm-Message-State: APjAAAUuZd8a+4Dlm6E7XDURP43cOlsOqjPyj6qows9EbpoyUVTqwTrp /q+MjYCDETDdm48XaDheqYe774G82LPfS7JUs94= X-Google-Smtp-Source: APXvYqxXuq1JFOP90ujIRSE0Jk9AD3SUziQ66+SJJ/o+7KhURXBtx9AMQWhzocEH1SOS7NefmBTTtMcuNoAivZUqeOw= X-Received: by 2002:a02:ba12:: with SMTP id z18mr10364631jan.16.1569186983285; Sun, 22 Sep 2019 14:16:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Sun, 22 Sep 2019 17:16:11 -0400 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46c0cn09xtz4GN4 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.43 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-4.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[43.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.07)[ip: (-4.80), ipnet: 209.85.128.0/17(-3.30), asn: 15169(-2.20), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[43.166.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2019 21:16:25 -0000 On Sun, 22 Sep 2019 at 16:49, Ulrich Sp=C3=B6rlein w= rote: > > Good news all around, this should now be back in a fully working order. Thank you for all the effort on getting this fixed. From owner-freebsd-git@freebsd.org Mon Sep 23 17:51:53 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF091FB38E for ; Mon, 23 Sep 2019 17:51:53 +0000 (UTC) (envelope-from sean@chittenden.org) Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cX2J23nkz4J80 for ; Mon, 23 Sep 2019 17:51:51 +0000 (UTC) (envelope-from sean@chittenden.org) Received: by mail-lj1-x22c.google.com with SMTP id f5so14604734ljg.8 for ; Mon, 23 Sep 2019 10:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chittenden.org; s=8eb15515; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8AX0YsfyauvaMPISoppKM4GMzlVIrPX4XRrIKt/kICc=; b=rF0gwb5iJhi3eJac0hAG1KefriMSPAOscJfLbR1O9EUjqKo/6nU6fTvJ5Ne5JqsAmb UlI4H2Xc7d518SJRWNCOrYRSEMJwXdw0EScdhIDKFhCyXrzMp672TGrmOeOUmPxpScN6 ZXp1WHpBpl97o2mrFE598pfcU/sfdq4G5nNkWBnuU+OVz39UpYzCrA46etaXmXHJL+qI 7BV+nlxdTCjuqOvSEDojmNr+vn1uKEsoQ62EcfkFbDAjgWT7D+ndgSbq53PixuP1XLyh oguUKWYWBKQQdNUQDorXK32+CJ5rhBZ42XX5zIEEZksMam7opeUGI644GE3tir+ONeP4 gvIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8AX0YsfyauvaMPISoppKM4GMzlVIrPX4XRrIKt/kICc=; b=LlUw/NmMmCwqmfXCdwM2rXXduLM21+KekOWL+BcVIAD0fumX51uuSPA52kalOlu/HC s5xxnynxsyXrpHYR9QMV4aH/8r5gecdE2EMhUlkb3W3vdbnLhGnx3RXfwoHugDvcJUAZ Jjo2JWSG2yZSRKxVlOP+iL6QLEWGeL/3SdYvYi0BHzo50gDefKnLAy059We3Vp5ULKqt +zkOGFHZ+jrn4N0qMnmA5vFIPlh2p0X8l92YyCwWH2DtQoAyr/Qnlb+VDq/8fFCjRckm nh662qWOoakKN7Uch8ycM0OlaC0M03Vts1Ak1C4lOUanPTmW026d77dTwRYxz6R1Sp+D xyVA== X-Gm-Message-State: APjAAAWQA0spw5h4UKZkrivpASf6ifeyvZAWR9T/MDY7cXCm+6hGU+fS uP7Ks94xmf9UtgZkrv7Is98vflS1Dy2j5PqAGgxmPdqx X-Google-Smtp-Source: APXvYqzBR/ewXWw77jhuy8tYreoSfJFBh/PG0S//2pO5lFXYIGlEbSGQ4K+rZfQISJoLf4Naxh/eg+gfNJdAp0mPGR8= X-Received: by 2002:a2e:1614:: with SMTP id w20mr342972ljd.159.1569261109753; Mon, 23 Sep 2019 10:51:49 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Sean Chittenden Date: Mon, 23 Sep 2019 10:51:33 -0700 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46cX2J23nkz4J80 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chittenden.org header.s=8eb15515 header.b=rF0gwb5i; dmarc=pass (policy=none) header.from=chittenden.org; spf=pass (mx1.freebsd.org: domain of sean@chittenden.org designates 2a00:1450:4864:20::22c as permitted sender) smtp.mailfrom=sean@chittenden.org X-Spamd-Result: default: False [-5.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[chittenden.org:s=8eb15515]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[chittenden.org:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[c.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[chittenden.org,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.93)[ip: (-9.47), ipnet: 2a00:1450::/32(-2.92), asn: 15169(-2.20), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 17:51:54 -0000 > > Please note however, that more "garbage" metadata escaped from SVN into > github, meaning 3rd parties have a hard time re-running the conversion and > making sure that it matches SVN down to the metadata (i.e. timestamps). > > Eventually, this will have to be re-rolled and a new "master" branch will > be force-pushed into github. There's no timeline for this yet. > Wait, what? Can you elaborate? Discussion of a force-push to github has occurred a few times and been explicitly ruled out because most of our corporate citizens use github to integrate changes from FreeBSD. Rerolling master was universally rejected when we socialized wanting to do this due to the level of disruption this would cause. The feedback was that this would be a high-cost, low-value operation. In the tradeoffs of purity vs pragmatism, pragmatism wins every time (that is the FreeBSD way). -sc From owner-freebsd-git@freebsd.org Mon Sep 23 18:16:42 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 33898FBB80 for ; Mon, 23 Sep 2019 18:16:42 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cXZx2yyGz4KtJ for ; Mon, 23 Sep 2019 18:16:41 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-ot1-x342.google.com with SMTP id g25so13030067otl.0 for ; Mon, 23 Sep 2019 11:16:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=MYjt/BsYyQLprhfhXl3bS3lTBzb6n5OhvpeMWWSqdM8=; b=JRemSZ+XCvm/+j5iFNyqzwkiKDtEqzHcs2h92eXYfZEWClFZ3+NTB6otB7iTPDvg63 G/uTuT0Ox5Tic7aUX3RFeeBO46RXKATUGoOWXuIfj4U9FnAwInuzjF1KARtQS3nRt8hV XvgNfXkJPkR8npia9UseJ0P0YumsvMgGwZ86DtcBeB7pl84cj9FeuElQjghTBL9wd5o1 jgLIuN47vP+lNygrh/NQWIWpm5t0l0Yo68O++EVK2MfPei4Jxc9WBQb1/kAHaX4bvsks VawTfMK12X0gA06C1GedXLny73EgkAppSz/URbYzYn6VD+4FoqzUSSplIG+JO8qnAt1l 8hZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=MYjt/BsYyQLprhfhXl3bS3lTBzb6n5OhvpeMWWSqdM8=; b=nBodG+OFCru2QIFReFCzSfErjYyMq3MW3Oc9B+hDv2BwbX73+q4aPMlKWxjNzTYmiK 03UfSP/JnqJL+PXyvyyh6nuwjzCo+Q0Ir0Wscbo9k183iAtqUkraWVdtvCxVV2f/a3bF qPSTpDv9z3Sqv/Q3AkLq8/mI4udDQUHwkpt5I2toovWrZrMU/9zJs/vVmDP/d5ytt+sG NUqbiGBmvFHZDEx2NE8CZ2WyO7PoCEIFTdk0TR/C34lZG6GohR8Mvt+H68uxNthKbh5J nlI5cRNJhNDXXMFfTPgIAeZCL4BfTBEJFlin4dGK0wk4A0WhW+q+M1vRXkml2rfG9p98 pkgw== X-Gm-Message-State: APjAAAUYuoTRxorO3uiGvk+guSvxlQugATFwv2/o5tAUBMvo0SYMi5VK kGVjx7Z94nfDDmryReeHdaOomNQ3hq3+2XIN8ag= X-Google-Smtp-Source: APXvYqwsUvdUur4OGxbvGuOxpTGHBB1jRXf/UjQh7x2xeC7maSDYHCDnh/Gn05wmFbvMG8yNzOkJmUx+9IPR963wDQg= X-Received: by 2002:a05:6830:1e2b:: with SMTP id t11mr993875otr.119.1569262600015; Mon, 23 Sep 2019 11:16:40 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Mon, 23 Sep 2019 20:16:25 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: Sean Chittenden Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46cXZx2yyGz4KtJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=JRemSZ+X; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::342 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; IP_SCORE(0.00)[ip: (3.09), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 18:16:42 -0000 Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden : >> >> Please note however, that more "garbage" metadata escaped from SVN into >> github, meaning 3rd parties have a hard time re-running the conversion a= nd >> making sure that it matches SVN down to the metadata (i.e. timestamps). >> >> Eventually, this will have to be re-rolled and a new "master" branch wil= l >> be force-pushed into github. There's no timeline for this yet. > > > Wait, what? Can you elaborate? > > Discussion of a force-push to github has occurred a few times and been ex= plicitly ruled out because most of our corporate citizens use github to int= egrate changes from FreeBSD. Rerolling master was universally rejected whe= n we socialized wanting to do this due to the level of disruption this woul= d cause. The feedback was that this would be a high-cost, low-value operat= ion. In the tradeoffs of purity vs pragmatism, pragmatism wins every time = (that is the FreeBSD way). > > -sc This is not just about pragmatism and the disruption it would cause is vastly overblown by people who don't seem to know much about the git storage model. There *is* garbage metadata in the published version on github, there *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since forever, and the cost of switching from 1 published branch to another is literally: - git diff origin/broken_master mybranch > mybranch.patch - git checkout -b fixed_branch origin/fixed_master - patch < mybranch.patch It should also be possible to merge both broken and fixed master into your branch (at the exact same SVN revision in time) and then you can follow fixed_master going forward. You'll schlepp around double the commit history, but not tree objects. If you want to retain history, you can upstream the changes prior to the switch, or do something more elaborate with git filter, maybe. This should be no surprise to anyone who's been reading up on the conversion workflow or following the various email threads. It is currently impossible to re-create the published repo w/o doing elaborate SVN metadata surgery. That is no state to be in, it is not pragmatism. hth Uli From owner-freebsd-git@freebsd.org Mon Sep 23 18:34:28 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 14B07FC410 for ; Mon, 23 Sep 2019 18:34:28 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cXzR0rDgz4MLx for ; Mon, 23 Sep 2019 18:34:26 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x842.google.com with SMTP id m15so18396398qtq.2 for ; Mon, 23 Sep 2019 11:34:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=eqX4Uy5JhAs6IjLqwkIwvWrH49AyZjjZbFe6qhfh9MY=; b=ZtwyfzWyFvl5MIajyKsp0ksfnuxyLiWje1HlJmN7jCCdGknHy9Mhxxfm0DEhILCy1y 1ClYOyumLYF+sJ5flI3yREjYlDAV7xXm8zeFY6NnABrNqax8zoGZUsoy0TOvsj2mivs6 T6RO1b+OdB0O20coedJJGIzqm7nL6xNABcs5EXzkqyQozNDxbrHs/RrzTvEo+v6rseN2 avtvHYJ/DxGTzsoooDIvwkzbPd3wTHB3hqPb+akIbkr84LdUgxvP0xwDLp+nUj+Uouw2 QOFqomM9edLmesIAdQs428AUNCywMkQOw4BypGYi9UUThFsd81pEpScN4Z52CsDB1SZv RF7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=eqX4Uy5JhAs6IjLqwkIwvWrH49AyZjjZbFe6qhfh9MY=; b=ndjVLnMp8TvQBHpZQsKl3NWULeOzmtMsmIsoD/qtmS4AQS7coLly9s2xd5MoADs2bW F73SMXH6RmvqPDIoycDM26p9V4ZStNO9SmyarFefTBIBR8UcltmC4MmGcyy6E9XXTK6W LqLxinQnQd7G983407RN1YRYIeFxUmtaf3xZy0hZI9g/l1rBD/YJIXYmIyBaWSf+WwcY Jz/d2NNXnpj9iVcg3chVaqyx2liX9+ci4RQVVq74AoZveDokeA0cGtwkG5Rp22TyYhkj A1rrWkgI8HiYCKyuXLOBIqIzMC2J4q25YYRRCXQEhk6nS5MxrEw1F5DJLssY+vFNrhri 4h6Q== X-Gm-Message-State: APjAAAWgO6X/5k1sdIM2B73CCsEpsnHNilIXVAc+5ezm6YzLv8N4ATZl ERRYQewzMI2in0iQcAZT7bdsug== X-Google-Smtp-Source: APXvYqwQXOIWV+HPpnMHvmibZLKfLHTX93jey8EAr8jNF1crTdYIEBOcAx2NaK4kjwNRF+3HdVc6qg== X-Received: by 2002:ac8:643:: with SMTP id e3mr1517765qth.184.1569263665930; Mon, 23 Sep 2019 11:34:25 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.120]) by smtp.gmail.com with ESMTPSA id z5sm5118890qkl.101.2019.09.23.11.34.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2019 11:34:25 -0700 (PDT) Date: Mon, 23 Sep 2019 14:34:24 -0400 From: Shawn Webb To: Ulrich Sp??rlein Cc: Sean Chittenden , freebsd-git@freebsd.org Subject: Re: Service disruption: git converter currently down Message-ID: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ndzuirdqly4wdq5z" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD r352488+2a20025064d4-c272097(hardened/current/master) HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0xFF2E67A277F8E1FA User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 46cXzR0rDgz4MLx X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=ZtwyfzWy; dmarc=none; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::842 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-4.66 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; RCVD_IN_DNSWL_NONE(0.00)[2.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.56)[ip: (2.11), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 18:34:28 -0000 --ndzuirdqly4wdq5z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hey Ulrich, I appreciate your hard work in maintaining the git mirror. Work like this can sometimes go unthanked. I want to take a moment to show appreciation for you and the FreeBSD project in maintaining the git mirror. I do have a few concerns with what was stated in your email. I've written my concerns inline. I hope this discussion is a positive one, wherein upstream and downstream can effectively come to a conclusion. On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > : > >> > >> Please note however, that more "garbage" metadata escaped from SVN into > >> github, meaning 3rd parties have a hard time re-running the conversion= and > >> making sure that it matches SVN down to the metadata (i.e. timestamps). > >> > >> Eventually, this will have to be re-rolled and a new "master" branch w= ill > >> be force-pushed into github. There's no timeline for this yet. > > > > > > Wait, what? Can you elaborate? > > > > Discussion of a force-push to github has occurred a few times and been = explicitly ruled out because most of our corporate citizens use github to i= ntegrate changes from FreeBSD. Rerolling master was universally rejected w= hen we socialized wanting to do this due to the level of disruption this wo= uld cause. The feedback was that this would be a high-cost, low-value oper= ation. In the tradeoffs of purity vs pragmatism, pragmatism wins every tim= e (that is the FreeBSD way). > > > > -sc >=20 >=20 > This is not just about pragmatism and the disruption it would cause is > vastly overblown by people who don't seem to know much about the git > storage model. >=20 > There *is* garbage metadata in the published version on github, there > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > forever, and the cost of switching from 1 published branch to another > is literally: >=20 > - git diff origin/broken_master mybranch > mybranch.patch > - git checkout -b fixed_branch origin/fixed_master > - patch < mybranch.patch Such a workflow breaks historical accuracy. Instead of `git annotate` showing the history properly, it's now based on an "epoch commit". Sure such a commit brings the branch to a working condition, but at the cost of history. >=20 > It should also be possible to merge both broken and fixed master into > your branch (at the exact same SVN revision in time) and then you can > follow fixed_master going forward. You'll schlepp around double the > commit history, but not tree objects. > If you want to retain history, you can upstream the changes prior to > the switch I so wish that were possible for certain downstream projects. We're unable to upstream the majority of our work. To argue "upstream your work and you won't be affected" is to choose an argument that does not reflect the reality of a growing portion of FreeBSD's downstream consumers: the inability to work effectively with upstream. Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0xFF2E67A277F8E1FA GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2 --ndzuirdqly4wdq5z Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAl2JECsACgkQ/y5nonf4 4foaRg/9EpwjCx65K5HRnjLT4Ml169J8xVsTtZNQ+CWdrDA5yTACQ8R+YRqZO7hz IQS1IdYSV5tDmLQLYMbXeUpUyn1hknXywSiN2+fm+ZgGH/nulhEucomiAh/eG/PW 0gw1hFPSbx1CVV2wttVSbmpCun8ggsgVviGUv/ZLmeM9sU0BJi6O8OHi80l2+qdF m2kO25Kx2hKgrbECNeg6z2P0NMyswoZdY25fzVVMEUK77n9sUpyb8KpGpvN1pxRd 967PtG0teVxlTCn8zTaOB13TCMv4HWzefnnv2+74SA8otObPsJpGr52C36lhIBpS /h652fyGM+0dBzoFe+Fqrr4oGWm6F3uUlqQrXDsCjF5cmISfthGf1YtwdtQJ0y4B sCdGY4FzrR7yVcR7iFosTg0uqs3hfYOzA0vW90DPhyVi30ofvdBn2hn7a7V5HbVv QQrLxmrqQcVwtIFAGQsIJqEz/oPSjA+VCAOKUGGX+HgC5KNZqEpiYe6URd5MW+wh tXaXY5+TCcfKiCOrnVNT62rK9Sy/w2fQv7LlCltOZMabkvXKdfFsJc0WFbpWa6DU C5nq41e9nA2ZuOTXNPUpEtCTwBrIsCk37HC/CApGI/x3UHXKxoztbClNwchIWYFL ncJSLsMBYaKljAp63fhg+qKWJFJwSNVDHqzEG1ouiFNfucE4gm0= =gzjI -----END PGP SIGNATURE----- --ndzuirdqly4wdq5z-- From owner-freebsd-git@freebsd.org Mon Sep 23 18:36:30 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B0FBDFC4C2 for ; Mon, 23 Sep 2019 18:36:30 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from fc.opsec.eu (fc.opsec.eu [IPv6:2001:14f8:200:4::4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46cY1p2nKlz4MP0 for ; Mon, 23 Sep 2019 18:36:30 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from pi by fc.opsec.eu with local (Exim 4.92.2 (FreeBSD)) (envelope-from ) id 1iCTCE-000NlQ-Cn; Mon, 23 Sep 2019 20:36:26 +0200 Date: Mon, 23 Sep 2019 20:36:26 +0200 From: Kurt Jaeger To: Shawn Webb Cc: freebsd-git@freebsd.org Subject: Re: Service disruption: git converter currently down Message-ID: <20190923183626.GR60644@fc.opsec.eu> References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> X-Rspamd-Queue-Id: 46cY1p2nKlz4MP0 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.90 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.91)[-0.906,0]; ASN(0.00)[asn:12502, ipnet:2001:14f8::/32, country:DE]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 18:36:30 -0000 Hi! > I so wish that were possible for certain downstream projects. We're > unable to upstream the majority of our work. To argue "upstream your > work and you won't be affected" is to choose an argument that does not > reflect the reality of a growing portion of FreeBSD's downstream > consumers: the inability to work effectively with upstream. Thanks for mentioning this. What exactly is the cause of that inability ? -- pi@FreeBSD.org +49 171 3101372 One year to go ! From owner-freebsd-git@freebsd.org Mon Sep 23 18:42:27 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B4E0EFC72E for ; Mon, 23 Sep 2019 18:42:27 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-oi1-x244.google.com (mail-oi1-x244.google.com [IPv6:2607:f8b0:4864:20::244]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cY8f5h2Hz4Mjn for ; Mon, 23 Sep 2019 18:42:26 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-oi1-x244.google.com with SMTP id w144so8520442oia.6 for ; Mon, 23 Sep 2019 11:42:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=uAcamWXF9RYfzTwS8kWzk+Zsn88KlEiruxUzo2h0UOs=; b=V0FuuSBJfXV6ZwlOgZSUltEEijauBwS2h5jfJT9DfojHYJE0hlm2NhNWtyQL74iAuk phR+WnHigB9BOIf7fdoTfr6ezXCmMkU4z3RkUftXsa6UpD9Z5Iki7UJ+XN4lToGVE+xW LtSAI0wrm209/Bg8NJaUv0zj1M3byYtLFa+bAq5BP3mOS/dzhaTkYeN4jHNc3keGfe1H e2w/IsDt+Nu2S6t+vXdeNAfp4UMpk255QXkTgCMWEnIdgV2Z7pjfK2mQAeoVHpBM8JvM ryHcxmjK72gIFRov+tgHNG90y9idacuCsV5tlzRxmgTlayk/RTYgZoo3Mr3bx5vjW+a5 M5Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=uAcamWXF9RYfzTwS8kWzk+Zsn88KlEiruxUzo2h0UOs=; b=nzeDXKbYaRJ+y71JHyooEQlbaf8VmZljkBEDavzv1j3Nyn4cgRSkbXeq/JUmGosDDF BJKFZAHxHDFnXUx3S+R9lUOdjAntGOXhtui3+HCw8YBwwiq+t/Iky/UpVqOO8A5TwJJr xm8hGF1gcdDLCCC/3DMOx0K2I6SpnCKTdiGfBlQ5TEAjye/Ds9CYLnmN6nUDsv3HHm1n OgZ0A1dueJorTr3ieuvoUvvNtU49SNdLT+/a+PZqpud/LDk3yRwbFM0sSZs9XfRbck33 kRHOsxjwTWDURIu4f4Ym/rmHLTDgok/eyciy3FxJA1jo56bP2c+lt0FZcVIHxYss9/cN sTJw== X-Gm-Message-State: APjAAAUPiE+Z+DW8Dy6gLfIhCaj5CcOLSQNng/gUfDoD14Yfav8M04TH VFygEqrJ/57EG3vDAjiqE2CpHp6OvYIe8JuuU7w= X-Google-Smtp-Source: APXvYqzPNyq82H1ApdNf7JlaFYX9SSBW5HPcEEkN8L+NB0Qx+6QxXqNYIwVRJ7s2tuROtDgnZKRqv7IDHPevejcXx/I= X-Received: by 2002:aca:2819:: with SMTP id 25mr1319683oix.42.1569264145274; Mon, 23 Sep 2019 11:42:25 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Mon, 23 Sep 2019 20:42:10 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: Shawn Webb Cc: Sean Chittenden , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46cY8f5h2Hz4Mjn X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=V0FuuSBJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::244 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (2.17), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 18:42:27 -0000 Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb : > > Hey Ulrich, > > I appreciate your hard work in maintaining the git mirror. Work like > this can sometimes go unthanked. I want to take a moment to show > appreciation for you and the FreeBSD project in maintaining the git > mirror. > > I do have a few concerns with what was stated in your email. I've > written my concerns inline. I hope this discussion is a positive one, > wherein upstream and downstream can effectively come to a conclusion. > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > : > > >> > > >> Please note however, that more "garbage" metadata escaped from SVN i= nto > > >> github, meaning 3rd parties have a hard time re-running the conversi= on and > > >> making sure that it matches SVN down to the metadata (i.e. timestamp= s). > > >> > > >> Eventually, this will have to be re-rolled and a new "master" branch= will > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > Wait, what? Can you elaborate? > > > > > > Discussion of a force-push to github has occurred a few times and bee= n explicitly ruled out because most of our corporate citizens use github to= integrate changes from FreeBSD. Rerolling master was universally rejected= when we socialized wanting to do this due to the level of disruption this = would cause. The feedback was that this would be a high-cost, low-value op= eration. In the tradeoffs of purity vs pragmatism, pragmatism wins every t= ime (that is the FreeBSD way). > > > > > > -sc > > > > > > This is not just about pragmatism and the disruption it would cause is > > vastly overblown by people who don't seem to know much about the git > > storage model. > > > > There *is* garbage metadata in the published version on github, there > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > forever, and the cost of switching from 1 published branch to another > > is literally: > > > > - git diff origin/broken_master mybranch > mybranch.patch > > - git checkout -b fixed_branch origin/fixed_master > > - patch < mybranch.patch > > Such a workflow breaks historical accuracy. Instead of `git annotate` > showing the history properly, it's now based on an "epoch commit". > Sure such a commit brings the branch to a working condition, but at > the cost of history. Is there really that much value in having "git blame" work in that environm= ent? My mental model is of short-lived branches that get upstreamed, so I'm bias= ed towards this not being all that big of a problem (for some at least). > > > > It should also be possible to merge both broken and fixed master into > > your branch (at the exact same SVN revision in time) and then you can > > follow fixed_master going forward. You'll schlepp around double the > > commit history, but not tree objects. > > If you want to retain history, you can upstream the changes prior to > > the switch > > I so wish that were possible for certain downstream projects. We're > unable to upstream the majority of our work. To argue "upstream your > work and you won't be affected" is to choose an argument that does not > reflect the reality of a growing portion of FreeBSD's downstream > consumers: the inability to work effectively with upstream. :/ I'm 80% sure that you can just merge both branches and things will be fine (though the exact incantation will surely be black magic). I'd love to try this on an actual repo though, I don't have the time to craft some test repo to ver= ify this assumption, and then find out that other repos are different). Also, I'll be offline in the coming weeks, so don't expect immediate respon= ses from me going forward. I already had to spent most of my weekend to patch things up and had to cancel other plans. Cheers, Uli From owner-freebsd-git@freebsd.org Mon Sep 23 18:51:01 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6162AFC7E6 for ; Mon, 23 Sep 2019 18:51:01 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cYLX3nttz4Mt9 for ; Mon, 23 Sep 2019 18:51:00 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82e.google.com with SMTP id d2so18459620qtr.4 for ; Mon, 23 Sep 2019 11:51:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sreoeKHU50Ysmyp/QpSfcHKSeAaLfLQeZloxVDSmU3A=; b=0QsF4VXA+vE3AWXoLle3kjlZ9qAttuPvEcQLUihiu0qqEAawWNuHRAUJjqvt9FTlZk lKO+9OMXEqnDjcWn6nkslPU29ZtAGGDZEEl/YcCFbRl8V/9LPHHbBbLD9HieYwNomeal DP9pnbo6IblP111SPcCkhfJiXNfTBMUbd88A51hQpQVsWd9W4uD+T8eLPOS2VuZlSrZo Tm2sIUb9mWVhzZrBThgKFXx4UK+JLmjVCISOUI1j0RIzcpV7pXn+OAXRLuliTP+dAu+D AF9HP/eWmfwSeQ/awlsQZB9QHcc2sBpQx7TdDBMvNsh8tXO9WC3BzErhtcIRHKNI/lQ1 dFFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sreoeKHU50Ysmyp/QpSfcHKSeAaLfLQeZloxVDSmU3A=; b=W7r1w5h4QXSAK/DBO6AoJmpIoAnrYaWtCCm6AKJ/sfaGbi8OMEdHhUPAb2d0FQSks7 DJZJLBmD7tkYrdNTkRsuNqySNXLydWQe9T440BwUbqOYToW3rLbVgZ6/nSPySwzI2a04 IC0UrViljs8D7Y9HCqoXfyDg4lul9kKuPxibJ2mQn+6+vT8Pd+L4EiS5mAqWZCbYbtAr 9qqzbisEQtKZHjs1c2YpVqv7+jebRar+Ad5e3qI/HnZhv+9FY1KKLsI4GUECj45HyW3A p5VfnKKYJDZRBp03r7I8r4cWyTc+98ju12Pb35dfmZTenkc/svRWNyxlW1q2O6pjo3Cn ejWA== X-Gm-Message-State: APjAAAXrHRGD6tgHgHxXJx4TcsmcAlHp5CIfdggpGm76bG3bmvCYndvn pEv9BYex76Ty7/7T8Kllk3YHhW8oMb6WVlwaSoZMgMLQ7Fo= X-Google-Smtp-Source: APXvYqx7RafWeMXCkIKy2Y10RY2dKZoBsl6Qtnpnt4dhPof+9+DCPFe2Zz7unz56gQpNLtX67ElMXiUZJSsYVQj/Yi8= X-Received: by 2002:ac8:1208:: with SMTP id x8mr1531662qti.32.1569264659149; Mon, 23 Sep 2019 11:50:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Mon, 23 Sep 2019 20:50:44 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: Sean Chittenden , freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46cYLX3nttz4Mt9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=0QsF4VXA; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::82e) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-4.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[e.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.84)[ip: (-9.31), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 18:51:01 -0000 On Mon, Sep 23, 2019, 8:16 PM Ulrich Sp=C3=B6rlein w= rote: > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > : > >> > >> Please note however, that more "garbage" metadata escaped from SVN int= o > >> github, meaning 3rd parties have a hard time re-running the conversion > and > >> making sure that it matches SVN down to the metadata (i.e. timestamps)= . > >> > >> Eventually, this will have to be re-rolled and a new "master" branch > will > >> be force-pushed into github. There's no timeline for this yet. > > > > > > Wait, what? Can you elaborate? > > > > Discussion of a force-push to github has occurred a few times and been > explicitly ruled out because most of our corporate citizens use github to > integrate changes from FreeBSD. Rerolling master was universally rejecte= d > when we socialized wanting to do this due to the level of disruption this > would cause. The feedback was that this would be a high-cost, low-value > operation. In the tradeoffs of purity vs pragmatism, pragmatism wins eve= ry > time (that is the FreeBSD way). > > > > -sc > > > This is not just about pragmatism and the disruption it would cause is > vastly overblown by people who don't seem to know much about the git > storage model. > > There *is* garbage metadata in the published version on github, there > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > forever, and the cost of switching from 1 published branch to another > is literally: > > - git diff origin/broken_master mybranch > mybranch.patch > - git checkout -b fixed_branch origin/fixed_master > - patch < mybranch.patch > Not a viable option. We have hundreds of commits in our tree at work and this would destroy the history there. A crazy set of rebase might work, but that's tricky too and would need to be scripted. It's possible, but a lot of pain for what most people will regard as zero real gain. It should also be possible to merge both broken and fixed master into > your branch (at the exact same SVN revision in time) and then you can > follow fixed_master going forward. You'll schlepp around double the > commit history, but not tree objects. > If you want to retain history, you can upstream the changes prior to > the switch, or do something more elaborate with git filter, maybe. > This should be no surprise to anyone who's been reading up on the > conversion workflow or following the various email threads. > We are long past this warning being enforceable. This would hit about 100 branches I have in progress. It is currently impossible to re-create the published repo w/o doing > elaborate SVN metadata surgery. That is no state to be in, it is not > pragmatism. > We know this. The pragmatism is that we don't really need to be that pure. Warner hth > Uli > _______________________________________________ > freebsd-git@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-git > To unsubscribe, send any mail to "freebsd-git-unsubscribe@freebsd.org" > From owner-freebsd-git@freebsd.org Mon Sep 23 18:51:16 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AF2A8FC932 for ; Mon, 23 Sep 2019 18:51:16 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x841.google.com (mail-qt1-x841.google.com [IPv6:2607:f8b0:4864:20::841]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cYLq4Mhjz4Mw5 for ; Mon, 23 Sep 2019 18:51:15 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x841.google.com with SMTP id o12so18466030qtf.3 for ; Mon, 23 Sep 2019 11:51:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=egX4S2MMjGk7foAaHDXxCyQOxX+9/bEZ9MZuOWBkggE=; b=W+NxHgaPMt/XFaFHMelPLg7W7Ms10WKScwOA1Ae9zLnEVu3WoEK7q3kq4wx94pBldo 5v4ON3Dlc7mmiNEmUihL5VV0a8yPg6rEK4JlTTQ/kRnDmexUEyklYIr4yUd7MMSK18fx Iu4RxC6B7Fm9AEkl+itUW3S5no5b3vf4GTiz2htnld77SAZon0mehR4vSzcGdm7dodIq 8+Ut8ytlEVxrKQXPHrw9L7d7oRoGTEckXqbrahZbo1JTX1V5MmQ34qnZsiBrYQTW3R4/ LjcetFO4xZGS/wol7v+IWlwD47SjOgluwM5W8S0Nvw8DrOr5drDxI7wCIC1LHfk0OuDs zbGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=egX4S2MMjGk7foAaHDXxCyQOxX+9/bEZ9MZuOWBkggE=; b=RrfD0dlxG1vVCpOJfAEbe1VKHO7o1M5VcVl14K47LMFn1NFIlS7lZ7zDTSv5YtPVKC 7iNSh/HMvPuklbfhzf73azH8NU8TCJgeAmKGsq9eJ2cIY5epsEotcam2oc9NyEuI+Kw8 oT7gu2mz9zXvxfJK6uYm3bNUXeNizEagSIIUYxCgLjt+OJ+xup6pvvwJUS2xycJ24BV3 ZxPB0dj3QPY+f+4XOmNhhRZXnMDWyvFv5e/RyvE9E0YnTz1g/baIKqvwq3tLL+fyTSNj SNU4TfnryHhx9PrySJ8gNpwIXOXR9YB2yOtvcmIH2TswrKdXLSY6vA52ZagzwbXvTc1s y1/A== X-Gm-Message-State: APjAAAXgaot5U9FsoGMz4GV2222ep45ap2yGIMD9lqNHpiq3pLgm21tc XXbTbayAVGws9XLbSnsUogh0sSBW7+w= X-Google-Smtp-Source: APXvYqyZBFn0yld6wBf+AZ2+6gOBr6577QZZKdFGh/l9XAC3DDNCQS99ngwum4bQdI2EC3LCro54Kw== X-Received: by 2002:a0c:ea92:: with SMTP id d18mr696164qvp.112.1569264674582; Mon, 23 Sep 2019 11:51:14 -0700 (PDT) Received: from mutt-hbsd ([63.88.83.120]) by smtp.gmail.com with ESMTPSA id z200sm6096124qkb.5.2019.09.23.11.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2019 11:51:13 -0700 (PDT) Date: Mon, 23 Sep 2019 14:51:13 -0400 From: Shawn Webb To: Ulrich Sp??rlein Cc: Sean Chittenden , freebsd-git@freebsd.org Subject: Re: Service disruption: git converter currently down Message-ID: <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yuhiezjj5qjvvmb7" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD r352488+2a20025064d4-c272097(hardened/current/master) HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0xFF2E67A277F8E1FA User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 46cYLq4Mhjz4Mw5 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=W+NxHgaP; dmarc=none; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::841 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-4.58 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; RCVD_IN_DNSWL_NONE(0.00)[1.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.48)[ip: (2.49), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 18:51:16 -0000 --yuhiezjj5qjvvmb7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 23, 2019 at 08:42:10PM +0200, Ulrich Sp??rlein wrote: > Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > : > > > > Hey Ulrich, > > > > I appreciate your hard work in maintaining the git mirror. Work like > > this can sometimes go unthanked. I want to take a moment to show > > appreciation for you and the FreeBSD project in maintaining the git > > mirror. > > > > I do have a few concerns with what was stated in your email. I've > > written my concerns inline. I hope this discussion is a positive one, > > wherein upstream and downstream can effectively come to a conclusion. > > > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > > : > > > >> > > > >> Please note however, that more "garbage" metadata escaped from SVN= into > > > >> github, meaning 3rd parties have a hard time re-running the conver= sion and > > > >> making sure that it matches SVN down to the metadata (i.e. timesta= mps). > > > >> > > > >> Eventually, this will have to be re-rolled and a new "master" bran= ch will > > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > > > > Wait, what? Can you elaborate? > > > > > > > > Discussion of a force-push to github has occurred a few times and b= een explicitly ruled out because most of our corporate citizens use github = to integrate changes from FreeBSD. Rerolling master was universally reject= ed when we socialized wanting to do this due to the level of disruption thi= s would cause. The feedback was that this would be a high-cost, low-value = operation. In the tradeoffs of purity vs pragmatism, pragmatism wins every= time (that is the FreeBSD way). > > > > > > > > -sc > > > > > > > > > This is not just about pragmatism and the disruption it would cause is > > > vastly overblown by people who don't seem to know much about the git > > > storage model. > > > > > > There *is* garbage metadata in the published version on github, there > > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > > forever, and the cost of switching from 1 published branch to another > > > is literally: > > > > > > - git diff origin/broken_master mybranch > mybranch.patch > > > - git checkout -b fixed_branch origin/fixed_master > > > - patch < mybranch.patch > > > > Such a workflow breaks historical accuracy. Instead of `git annotate` > > showing the history properly, it's now based on an "epoch commit". > > Sure such a commit brings the branch to a working condition, but at > > the cost of history. >=20 > Is there really that much value in having "git blame" work in that enviro= nment? > My mental model is of short-lived branches that get upstreamed, so I'm bi= ased > towards this not being all that big of a problem (for some at least). >=20 > > > > > > It should also be possible to merge both broken and fixed master into > > > your branch (at the exact same SVN revision in time) and then you can > > > follow fixed_master going forward. You'll schlepp around double the > > > commit history, but not tree objects. > > > If you want to retain history, you can upstream the changes prior to > > > the switch > > > > I so wish that were possible for certain downstream projects. We're > > unable to upstream the majority of our work. To argue "upstream your > > work and you won't be affected" is to choose an argument that does not > > reflect the reality of a growing portion of FreeBSD's downstream > > consumers: the inability to work effectively with upstream. >=20 > :/ >=20 > I'm 80% sure that you can just merge both branches and things will be fine > (though the exact incantation will surely be black magic). I'd love to > try this on > an actual repo though, I don't have the time to craft some test repo to v= erify > this assumption, and then find out that other repos are different). HardenedBSD's github repo has existed since 2013, with branches stemming from that work existing still today. Perhaps HardenedBSD is somewhat in a special case: we aim to provide the BSD community with a clean-room reimplementation of publicly-documented parts of the grsecurity patchset. With FreeBSD not taking the same approach, we will have very long-lived branches. For example, our hardened/current/master branch follows FreeBSD's HEAD and syncs every six hours. Meaning, we maintain our patches, resolving whatever few merge conflicts arrive. The hardened/current/master branch was created so many years ago, I've forgotten when it was actually created (perhaps in 2013?) Though HardenedBSD's cause for existence may be a special case, this problem can be viewed in a general fashion. I'm confident HardenedBSD is not alone in facing issues of these types. Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0xFF2E67A277F8E1FA GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2 --yuhiezjj5qjvvmb7 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAl2JFBkACgkQ/y5nonf4 4fo53Q/8DGpBwo2owUaJJSoGfVPpymG/gsucxtNdDa6XPC1+43Dggport4To+XcU kjPLrSaq8Z3LWVwsRNFdPn1D6+XnwYYupwJwZCheWe9CoV1Z0COsLJQRnVwRkkTu o0chDU62yd1dcRUbXc/48m7SE/2V/WrZpqj9/9grKGecx9if/PotcHL/+R/0Wa2W 5zCKURXcHYmUXvqCj3EzbD0Q7GfEm+ZfiZQgdYiZEP5y0J4xUP4Lf5VgBItCWA3Q 7gG8DDQwC4/LIYWZRm0gg+/94/cn2yUXzMYRJyBBbXitJVCnZjFYf8XsPdtaO53e Tx3Z71Ya9A5uuZYgKRSw41ndGkRBTCrbo/VgNxEm297a0AHJaUXA1s2LxgikEEWU fbAW8tR0CM3bB6e5XlbjxPtRPCev63MQpAA3yi8lUxEPf33jHLuNSkudJORjvzR9 TKQAumjyn2UgrcAqbJ+udnVOxhRhR+Sx0E4tVFPYfUGQr5mEj27P9VUKHU+7SerA XduPaOeT8zBhRhV7RVGaklnqoePg1XVUMJf7w75g2YovqdZlUEBue1AUUXz1Rr/p n+4rmMYcCpXfdYruORzcuJSRDfViGJtKN3Bk2k2Mt3EoVWTDyP5eeGumwOFzn1Cm v5z71J4aGFAOaukxSY2JiJu1i0PnssUKBOKgZEUcX2ctUSHphog= =sBvH -----END PGP SIGNATURE----- --yuhiezjj5qjvvmb7-- From owner-freebsd-git@freebsd.org Mon Sep 23 19:06:49 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2D440FCC90 for ; Mon, 23 Sep 2019 19:06:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cYhm25Qnz4Nmk for ; Mon, 23 Sep 2019 19:06:48 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x729.google.com with SMTP id w2so16647171qkf.2 for ; Mon, 23 Sep 2019 12:06:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JIrxGILvx5c4JQ/aHTffwRUAasbBURGzgIc5IsZbGJM=; b=mverWhamvr2yVbAFfWOsoon/QcFqUWr0KHH/PiBKfzhL0VqrnylgVsA7WjKJk1oKGA WLFs3w8XN4z7+3HW7YOlEzt8opQpNSELeMpbRbznz92ILa8gxhgBDZsv3yqy7O0VWne7 LzlA2nGKfWUa43Vru7BTcjVXeWIPNCG7KLxJF9v3stV9WLm91BHTwIJw3UFbyuQ8DnJ9 rTbzMUIzvXt0n2ZoQVLaDqtDUr9P5jIIrxZ0PFkqp1CiyJhPM4oGtIRA4DQiAQfjvSAu +Fcg3DgWmGTDgvCCzfO6QE6jIuvX3UR/LG9kjdUbOVxhOHYe/9fdz27dELhug3BMF5zu pyQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JIrxGILvx5c4JQ/aHTffwRUAasbBURGzgIc5IsZbGJM=; b=q9K0pl5vBOGans9CIYQFF6uxLv/umhQYBrYyJtMbLcizRYBUKPUBTr3lZyqCB8hm5U YFnikSIHSJAQgHwSrbVL+LmA0TfCLeY5Nu2JTphtv7ZbwrYa6ZpPIlJSTDCfytr6SKJE 0FdA99a1ZxLmPC9TNnJAKsfVlSYNs9xoCxpwFshUFMwXqDofeb+Nfb2B2Da0SLLdcyhS KallyS3P9Zwvz2JI3kflfqWdwRe65KCp4Oc49KD0IHMb8QqjaPOKYW3hoJJEiXodRbjw 7qXCLNonRtpfGgTI4pg7Bt90LZbCU+MM+DoDBpHQTYeXL8YFJulw1fDwvJiF2l8he33g 9BHQ== X-Gm-Message-State: APjAAAUrgXXV4hb4OL1q9NUOnKh7a7WwN2xFIH1lkVIt7ibOytmAbRsy UNkulUYfKDFgGD21fZ/SqtirxUKJgi2HgJHL5ZCU0IJlP8U= X-Google-Smtp-Source: APXvYqz/AGoeHl8tpVxNZGCS1w8cXovXVa+dudc610Ks9orZaOtn6nxs18vjreh+nvlVzZ5ONH81rxqOp92jo5qopgQ= X-Received: by 2002:a05:620a:6af:: with SMTP id i15mr1537873qkh.380.1569265606994; Mon, 23 Sep 2019 12:06:46 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: From: Warner Losh Date: Mon, 23 Sep 2019 21:06:34 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: Shawn Webb , freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46cYhm25Qnz4Nmk X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=mverWham; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::729) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-4.83 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[9.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.83)[ip: (-9.28), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 19:06:49 -0000 On Mon, Sep 23, 2019, 8:42 PM Ulrich Sp=C3=B6rlein w= rote: > Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > : > > > > Hey Ulrich, > > > > I appreciate your hard work in maintaining the git mirror. Work like > > this can sometimes go unthanked. I want to take a moment to show > > appreciation for you and the FreeBSD project in maintaining the git > > mirror. > > > > I do have a few concerns with what was stated in your email. I've > > written my concerns inline. I hope this discussion is a positive one, > > wherein upstream and downstream can effectively come to a conclusion. > > > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > > : > > > >> > > > >> Please note however, that more "garbage" metadata escaped from SVN > into > > > >> github, meaning 3rd parties have a hard time re-running the > conversion and > > > >> making sure that it matches SVN down to the metadata (i.e. > timestamps). > > > >> > > > >> Eventually, this will have to be re-rolled and a new "master" > branch will > > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > > > > Wait, what? Can you elaborate? > > > > > > > > Discussion of a force-push to github has occurred a few times and > been explicitly ruled out because most of our corporate citizens use gith= ub > to integrate changes from FreeBSD. Rerolling master was universally > rejected when we socialized wanting to do this due to the level of > disruption this would cause. The feedback was that this would be a > high-cost, low-value operation. In the tradeoffs of purity vs pragmatism= , > pragmatism wins every time (that is the FreeBSD way). > > > > > > > > -sc > > > > > > > > > This is not just about pragmatism and the disruption it would cause i= s > > > vastly overblown by people who don't seem to know much about the git > > > storage model. > > > > > > There *is* garbage metadata in the published version on github, there > > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > > forever, and the cost of switching from 1 published branch to another > > > is literally: > > > > > > - git diff origin/broken_master mybranch > mybranch.patch > > > - git checkout -b fixed_branch origin/fixed_master > > > - patch < mybranch.patch > > > > Such a workflow breaks historical accuracy. Instead of `git annotate` > > showing the history properly, it's now based on an "epoch commit". > > Sure such a commit brings the branch to a working condition, but at > > the cost of history. > > Is there really that much value in having "git blame" work in that > environment? > My mental model is of short-lived branches that get upstreamed, so I'm > biased > towards this not being all that big of a problem (for some at least). > Yes. There absolutely is. > > > > > It should also be possible to merge both broken and fixed master into > > > your branch (at the exact same SVN revision in time) and then you can > > > follow fixed_master going forward. You'll schlepp around double the > > > commit history, but not tree objects. > > > If you want to retain history, you can upstream the changes prior to > > > the switch > > > > I so wish that were possible for certain downstream projects. We're > > unable to upstream the majority of our work. To argue "upstream your > > work and you won't be affected" is to choose an argument that does not > > reflect the reality of a growing portion of FreeBSD's downstream > > consumers: the inability to work effectively with upstream. > > :/ > > I'm 80% sure that you can just merge both branches and things will be fin= e > (though the exact incantation will surely be black magic). I'd love to > try this on > an actual repo though, I don't have the time to craft some test repo to > verify > this assumption, and then find out that other repos are different). > One might be able to do this, but this is really quite a lot to ask. I've done a lot of git rebase in connection with qemu. The hard part us knowing hash X is for rev r123 in branch 1, but hash Y In branch 2. Once you can automate that, for various mappings, a script to rebase old to new becomes possible which will lessen the impact, but not eliminate it. Also, I'll be offline in the coming weeks, so don't expect immediate > responses > from me going forward. I already had to spent most of my weekend to patch > things up and had to cancel other plans. > I think it is safe to view this discussion as just don't force push yet as we are in no position to mitigate the large disruption this would cause. If you aren't force pushing master, there is no time urgency to solve this. Finally, and most importantly, thank you for all your time and efforts here. They are much appreciated. Any quibbling over future plans shouldn't detract from that. Warner Cheers, > Uli > _______________________________________________ > freebsd-git@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-git > To unsubscribe, send any mail to "freebsd-git-unsubscribe@freebsd.org" > From owner-freebsd-git@freebsd.org Mon Sep 23 20:38:53 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 21DB3FF66D for ; Mon, 23 Sep 2019 20:38:53 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cbkz5wkrz4VYS for ; Mon, 23 Sep 2019 20:38:51 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-ot1-x344.google.com with SMTP id c10so13350659otd.9 for ; Mon, 23 Sep 2019 13:38:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+Cq2E703QpIqA8nyNPBLMdzrifC0a69n6hDbcfOCP3Q=; b=eQGdXeXnHuSG5OHsJGnoGuCLVMNXhconygTVd1UliR4MnuzEbD66/xZk9KjnfyzXvY M6naZxZId9xFGdgjKakZfvCBD5DgzAJwdskU+CjejzapHbqyVgxRRKS0sXMa7k/5mGgu KY2FcCxzUkWvTLCZiqDbaTI3sabvsnPsjlxanIlNCaWVLPcsmJohL6Jfpz49bq3O8OLt NOoykZvxxikF8RbEk1K6wssG0a0WKo5eRrfuOx88OnsYN/1HIFLgHUlvUYZYilfAnY4+ PhyjrpN/NAs/UqHMi/zF4vcUD0MZ9gU7LRpZnh7ry7MUD165FF+TP7T5Yoxx9795IysA SoVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+Cq2E703QpIqA8nyNPBLMdzrifC0a69n6hDbcfOCP3Q=; b=kzhbXJPeXyqacaPDtIoXPJRWlErp/47RA1HV7B5hRuo2Vq3c/eIlo98XR5H2VkPubs w7cZNTIgelwIV5X7XK5hGYXk9CoheuGC/CcLqsicfiIN7I6rDVjDXYx6bz75e8QQrrYs kynDtxDt4rM3Mo5Cl5uMeuYg29EcrIEJLYnJ9KObCZXKbNKa1/VkFNLQhu9l6GLfSIY7 KtgFAj4SigwqyBoBs/CgN2OG34rA8v+oQgQap8f5q8tMqoaJkrSs4G09BIcdir4oCXeU eBlJPvFFBsv6c+Ruw0T5Km6DhGFI8aYtCwh8YnEJ0hfglrdl9+2zg8oGMgvhAtC4eFHB gf4Q== X-Gm-Message-State: APjAAAVDU3ausulWx2D4xDBWatj4BwR/1uuDi4Y2SyendEPp6wa7c8TH X2+cPSeGKDPpNwBrVUXPWljGCu+IseEH+2xizqI= X-Google-Smtp-Source: APXvYqxHRozgvFoQToBynRQ4FSxbYOKeJQOjL61TRD1n2I8DnvoJVr9xVyazZuwLAoKmhU0YF4wfvq8GSqJ8OS4602U= X-Received: by 2002:a9d:3b02:: with SMTP id z2mr112872otb.71.1569271130542; Mon, 23 Sep 2019 13:38:50 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> In-Reply-To: <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Mon, 23 Sep 2019 22:38:36 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: Shawn Webb Cc: Sean Chittenden , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46cbkz5wkrz4VYS X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eQGdXeXn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::344 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (3.06), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 20:38:53 -0000 Am Mo., 23. Sept. 2019 um 20:51 Uhr schrieb Shawn Webb : > > On Mon, Sep 23, 2019 at 08:42:10PM +0200, Ulrich Sp??rlein wrote: > > Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > > : > > > > > > Hey Ulrich, > > > > > > I appreciate your hard work in maintaining the git mirror. Work like > > > this can sometimes go unthanked. I want to take a moment to show > > > appreciation for you and the FreeBSD project in maintaining the git > > > mirror. > > > > > > I do have a few concerns with what was stated in your email. I've > > > written my concerns inline. I hope this discussion is a positive one, > > > wherein upstream and downstream can effectively come to a conclusion. > > > > > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > > > : > > > > >> > > > > >> Please note however, that more "garbage" metadata escaped from S= VN into > > > > >> github, meaning 3rd parties have a hard time re-running the conv= ersion and > > > > >> making sure that it matches SVN down to the metadata (i.e. times= tamps). > > > > >> > > > > >> Eventually, this will have to be re-rolled and a new "master" br= anch will > > > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > > > > > > > Wait, what? Can you elaborate? > > > > > > > > > > Discussion of a force-push to github has occurred a few times and= been explicitly ruled out because most of our corporate citizens use githu= b to integrate changes from FreeBSD. Rerolling master was universally reje= cted when we socialized wanting to do this due to the level of disruption t= his would cause. The feedback was that this would be a high-cost, low-valu= e operation. In the tradeoffs of purity vs pragmatism, pragmatism wins eve= ry time (that is the FreeBSD way). > > > > > > > > > > -sc > > > > > > > > > > > > This is not just about pragmatism and the disruption it would cause= is > > > > vastly overblown by people who don't seem to know much about the gi= t > > > > storage model. > > > > > > > > There *is* garbage metadata in the published version on github, the= re > > > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > > > forever, and the cost of switching from 1 published branch to anoth= er > > > > is literally: > > > > > > > > - git diff origin/broken_master mybranch > mybranch.patch > > > > - git checkout -b fixed_branch origin/fixed_master > > > > - patch < mybranch.patch > > > > > > Such a workflow breaks historical accuracy. Instead of `git annotate` > > > showing the history properly, it's now based on an "epoch commit". > > > Sure such a commit brings the branch to a working condition, but at > > > the cost of history. > > > > Is there really that much value in having "git blame" work in that envi= ronment? > > My mental model is of short-lived branches that get upstreamed, so I'm = biased > > towards this not being all that big of a problem (for some at least). > > > > > > > > > > It should also be possible to merge both broken and fixed master in= to > > > > your branch (at the exact same SVN revision in time) and then you c= an > > > > follow fixed_master going forward. You'll schlepp around double the > > > > commit history, but not tree objects. > > > > If you want to retain history, you can upstream the changes prior t= o > > > > the switch > > > > > > I so wish that were possible for certain downstream projects. We're > > > unable to upstream the majority of our work. To argue "upstream your > > > work and you won't be affected" is to choose an argument that does no= t > > > reflect the reality of a growing portion of FreeBSD's downstream > > > consumers: the inability to work effectively with upstream. > > > > :/ > > > > I'm 80% sure that you can just merge both branches and things will be f= ine > > (though the exact incantation will surely be black magic). I'd love to > > try this on > > an actual repo though, I don't have the time to craft some test repo to= verify > > this assumption, and then find out that other repos are different). > > HardenedBSD's github repo has existed since 2013, with branches > stemming from that work existing still today. Perhaps HardenedBSD is > somewhat in a special case: we aim to provide the BSD community with a > clean-room reimplementation of publicly-documented parts of the > grsecurity patchset. > > With FreeBSD not taking the same approach, we will have very > long-lived branches. For example, our hardened/current/master branch > follows FreeBSD's HEAD and syncs every six hours. Meaning, we maintain > our patches, resolving whatever few merge conflicts arrive. The > hardened/current/master branch was created so many years ago, I've > forgotten when it was actually created (perhaps in 2013?) > > Though HardenedBSD's cause for existence may be a special case, this > problem can be viewed in a general fashion. I'm confident HardenedBSD > is not alone in facing issues of these types. > > Thanks, What I don't understand is how a security focused project can trust a random source for the svn2git conversion. I could have planted a bunch of backdoors and then come up with some SVN metadata corruption conspiracy as to why the commit hashes are different. Why would you trust me? HardenedBSD of all people should be running the converter themselves and check that the content really matches what is in SVN (which it currently doesn't for metadata). Cheers, Uli From owner-freebsd-git@freebsd.org Mon Sep 23 20:44:18 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7E162FF8F0 for ; Mon, 23 Sep 2019 20:44:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cbsF6c4vz4W46 for ; Mon, 23 Sep 2019 20:44:17 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x72b.google.com with SMTP id u186so16937248qkc.5 for ; Mon, 23 Sep 2019 13:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wNYiBp+rr5XZj5oNVrHHUyDTm5DbGPxpvwZPtHhqXEA=; b=BAv2jWEZMbE/zwn8igPJEm5stw4tsrCU335TWQby6wdkJc+vMPK9iv/XM3EfeuKHaI RQBOsDjfwooeTCSMvK0Yc9Uco395aiPKExdTpwr2or10JA6cCz1aVSAN7CeWw/Ivx/Kh LsXOq4VhV+8+89CDX+C2qwDzqL9y0535Yr7+iHz7Ew96DgYwwSLTa2ewRhX+oEmGfPqN K5vmzqvmjc8F3VvgNp9OWVm+Y/fZejhzfka3cKoZMy7pVj6TlB3KF5WB91zKgZeq6mpK rf3SuND1BmlZrwyUmFtKhRGSCc0So1j/dzmWxW4ueE7YOdWb95K9aaYdkwfuFD76nOAX iF8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wNYiBp+rr5XZj5oNVrHHUyDTm5DbGPxpvwZPtHhqXEA=; b=unUrFNX2jxDN0nWrAbHs7UTKhvv6qYnF8U6RgVj9K+XL6e8MA6MKi26JnqrHYE/nx0 guWjDkF5jzjO+35IUmjEkoQhKVF/TBWarjUCHvU+EGg1P1jdRXoaOqqemag5KvLt/vFY MgSvwQCPjefZuqYnYIHCd2OteJaaAHy98MNmhzbPvOJFmgsSgjptstzW2aoQd+X83NRc //12DXOXx2YGkbbmMmXPl7Doq5aXBwJaNHHWzvXOO2W9ExQJtU5S3lw/WoiGtiv6zCm4 M+5ddgkAQNcVQZr/IelOQ4VWPaPDQgWQZIKq0GV26vGpjxrfJYCQ2GrBvZj983pACNE5 iM+w== X-Gm-Message-State: APjAAAUVhS8GzChayZ4ElRulkgTm9m673qfUEWrofwiHmdz4pPxxnaaB RIERIEIDwg4A/WFjkX2PeRpZd3JT02ZGxhLCZIvk/J6u51Q= X-Google-Smtp-Source: APXvYqwmO3FYuZJ5MRjSjU1cP1DBiEI3MbrOzpwfXp7EZhVdretsVTE1VMWVZhpszudz/1xWVDkG/qCo1CL22+qM57g= X-Received: by 2002:a37:7bc7:: with SMTP id w190mr2019524qkc.215.1569271456565; Mon, 23 Sep 2019 13:44:16 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> In-Reply-To: From: Warner Losh Date: Mon, 23 Sep 2019 22:44:04 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: Shawn Webb , freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46cbsF6c4vz4W46 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=BAv2jWEZ; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::72b) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-4.87 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[b.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.87)[ip: (-9.46), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 20:44:18 -0000 On Mon, Sep 23, 2019, 10:38 PM Ulrich Sp=C3=B6rlein = wrote: > Am Mo., 23. Sept. 2019 um 20:51 Uhr schrieb Shawn Webb > : > > > > On Mon, Sep 23, 2019 at 08:42:10PM +0200, Ulrich Sp??rlein wrote: > > > Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > > > : > > > > > > > > Hey Ulrich, > > > > > > > > I appreciate your hard work in maintaining the git mirror. Work lik= e > > > > this can sometimes go unthanked. I want to take a moment to show > > > > appreciation for you and the FreeBSD project in maintaining the git > > > > mirror. > > > > > > > > I do have a few concerns with what was stated in your email. I've > > > > written my concerns inline. I hope this discussion is a positive on= e, > > > > wherein upstream and downstream can effectively come to a conclusio= n. > > > > > > > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > > > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > > > > : > > > > > >> > > > > > >> Please note however, that more "garbage" metadata escaped from > SVN into > > > > > >> github, meaning 3rd parties have a hard time re-running the > conversion and > > > > > >> making sure that it matches SVN down to the metadata (i.e. > timestamps). > > > > > >> > > > > > >> Eventually, this will have to be re-rolled and a new "master" > branch will > > > > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > > > > > > > > > > Wait, what? Can you elaborate? > > > > > > > > > > > > Discussion of a force-push to github has occurred a few times > and been explicitly ruled out because most of our corporate citizens use > github to integrate changes from FreeBSD. Rerolling master was universal= ly > rejected when we socialized wanting to do this due to the level of > disruption this would cause. The feedback was that this would be a > high-cost, low-value operation. In the tradeoffs of purity vs pragmatism= , > pragmatism wins every time (that is the FreeBSD way). > > > > > > > > > > > > -sc > > > > > > > > > > > > > > > This is not just about pragmatism and the disruption it would > cause is > > > > > vastly overblown by people who don't seem to know much about the > git > > > > > storage model. > > > > > > > > > > There *is* garbage metadata in the published version on github, > there > > > > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > > > > forever, and the cost of switching from 1 published branch to > another > > > > > is literally: > > > > > > > > > > - git diff origin/broken_master mybranch > mybranch.patch > > > > > - git checkout -b fixed_branch origin/fixed_master > > > > > - patch < mybranch.patch > > > > > > > > Such a workflow breaks historical accuracy. Instead of `git annotat= e` > > > > showing the history properly, it's now based on an "epoch commit". > > > > Sure such a commit brings the branch to a working condition, but at > > > > the cost of history. > > > > > > Is there really that much value in having "git blame" work in that > environment? > > > My mental model is of short-lived branches that get upstreamed, so I'= m > biased > > > towards this not being all that big of a problem (for some at least). > > > > > > > > > > > > > It should also be possible to merge both broken and fixed master > into > > > > > your branch (at the exact same SVN revision in time) and then you > can > > > > > follow fixed_master going forward. You'll schlepp around double t= he > > > > > commit history, but not tree objects. > > > > > If you want to retain history, you can upstream the changes prior > to > > > > > the switch > > > > > > > > I so wish that were possible for certain downstream projects. We're > > > > unable to upstream the majority of our work. To argue "upstream you= r > > > > work and you won't be affected" is to choose an argument that does > not > > > > reflect the reality of a growing portion of FreeBSD's downstream > > > > consumers: the inability to work effectively with upstream. > > > > > > :/ > > > > > > I'm 80% sure that you can just merge both branches and things will be > fine > > > (though the exact incantation will surely be black magic). I'd love t= o > > > try this on > > > an actual repo though, I don't have the time to craft some test repo > to verify > > > this assumption, and then find out that other repos are different). > > > > HardenedBSD's github repo has existed since 2013, with branches > > stemming from that work existing still today. Perhaps HardenedBSD is > > somewhat in a special case: we aim to provide the BSD community with a > > clean-room reimplementation of publicly-documented parts of the > > grsecurity patchset. > > > > With FreeBSD not taking the same approach, we will have very > > long-lived branches. For example, our hardened/current/master branch > > follows FreeBSD's HEAD and syncs every six hours. Meaning, we maintain > > our patches, resolving whatever few merge conflicts arrive. The > > hardened/current/master branch was created so many years ago, I've > > forgotten when it was actually created (perhaps in 2013?) > > > > Though HardenedBSD's cause for existence may be a special case, this > > problem can be viewed in a general fashion. I'm confident HardenedBSD > > is not alone in facing issues of these types. > > > > Thanks, > > What I don't understand is how a security focused project can trust a > random source for the svn2git conversion. I could have planted a bunch > of backdoors and then come up with some SVN metadata corruption > conspiracy as to why the commit hashes are different. Why would you > trust me > The metadata has been independently audited. otherwise I'd be worried... Git use has taken off to such an extent we'd need a well tested easy button to do migrations that has good messaging and clear, easy to follow instructions that are as automated as possible. Warner HardenedBSD of all people should be running the converter themselves > and check that the content really matches what is in SVN (which it > currently doesn't for metadata). > > Cheers, > Uli > _______________________________________________ > freebsd-git@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-git > To unsubscribe, send any mail to "freebsd-git-unsubscribe@freebsd.org" > From owner-freebsd-git@freebsd.org Mon Sep 23 21:09:56 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AC3C312122C for ; Mon, 23 Sep 2019 21:09:56 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46ccQq2z9Sz4XtQ for ; Mon, 23 Sep 2019 21:09:55 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-ot1-x341.google.com with SMTP id f21so13413285otl.13 for ; Mon, 23 Sep 2019 14:09:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Zz/pWADifwVGe0Z6GOtM7tcSFSPgGi13pHzDneXGFDU=; b=XFqfzL/l/UR0zcK8ZJB+GkwE1Lpt3WsoYUwssPaaOvVXhpmcgedstS4aiv3MII322r eH8+mgmkjWT8psQRnfhQ9PWFFoBGJkFNKeWixa5Z9myPSlnU7A7DaFeNqNQjRptHdmL8 deuqe25D9pfz/u7taFsb3LMFPAylbdcapk4RjC9NZHQW1GbHO8iKN+ffoxqzysB3nwSY J4D67jOyi6I84zxyww6v+v7XLWhfJw6TAma+l6yxSuU6KCbIcGqnqFBDL5EBvPLfwGiv 53HmGr+Sv1eZD/N+49y4i+nJR8mNC92CHAclah1EhwBE2f7fZHL8Qg0bd7V7XXqi0isS uHbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Zz/pWADifwVGe0Z6GOtM7tcSFSPgGi13pHzDneXGFDU=; b=KurLkdvA1YPAhA190jKj6E8XjgX0/rXD+myeahLWufKGfGGNm/ADXkhAwJgltm6HW8 2ONM1n+jLNgXJna+0hev/ikfFpVIVpBhTL76Tdk/TGRI+dqnnle3ZY7uslrMpMLmt4xD VSa0Fc6Xs+kg7oUd/pHwY3FQmqeMwfoB0fBoKQEpqp/0ZW5LolDFZzEtk+OuHVgB98Fo HuPBNGGAL9XHMGfXDENPBm6GSW7gx5oBzUHKvu2OyZZxRUmyPctGtvl8+ok3XFPx5HuX 7JbdzAnCxoX2ZGmMJ+kUmo8RcGfzH2NqIAq769oeAfRlDI4Gsi7xov49qRZb/G3hEvts IYmw== X-Gm-Message-State: APjAAAWrOTigFML/oWzTNWY7RJe1H0LzwFt0p8rTaA+kQOp337WgYBAN PQxoY9VawNOE31o0QNXohbx5sPXv21CYfjfRP+oXZyO2i/0= X-Google-Smtp-Source: APXvYqzaKLHT1zvbALWRs0OBhuO0+e9DM0lSn7oloQKQMMmkP5+26pByt9X6LUDOKwQLPuYdUZy3k6UsGfxBwxc8QW4= X-Received: by 2002:a9d:be4:: with SMTP id 91mr224510oth.111.1569272994088; Mon, 23 Sep 2019 14:09:54 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Mon, 23 Sep 2019 23:09:39 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: Warner Losh Cc: Shawn Webb , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46ccQq2z9Sz4XtQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=XFqfzL/l; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 2607:f8b0:4864:20::341 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[1.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (3.06), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 21:09:56 -0000 Am Mo., 23. Sept. 2019 um 21:06 Uhr schrieb Warner Losh : > > > > On Mon, Sep 23, 2019, 8:42 PM Ulrich Sp=C3=B6rlein = wrote: >> >> Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb >> : >> > >> > Hey Ulrich, >> > >> > I appreciate your hard work in maintaining the git mirror. Work like >> > this can sometimes go unthanked. I want to take a moment to show >> > appreciation for you and the FreeBSD project in maintaining the git >> > mirror. >> > >> > I do have a few concerns with what was stated in your email. I've >> > written my concerns inline. I hope this discussion is a positive one, >> > wherein upstream and downstream can effectively come to a conclusion. >> > >> > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: >> > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden >> > > : >> > > >> >> > > >> Please note however, that more "garbage" metadata escaped from SV= N into >> > > >> github, meaning 3rd parties have a hard time re-running the conve= rsion and >> > > >> making sure that it matches SVN down to the metadata (i.e. timest= amps). >> > > >> >> > > >> Eventually, this will have to be re-rolled and a new "master" bra= nch will >> > > >> be force-pushed into github. There's no timeline for this yet. >> > > > >> > > > >> > > > Wait, what? Can you elaborate? >> > > > >> > > > Discussion of a force-push to github has occurred a few times and = been explicitly ruled out because most of our corporate citizens use github= to integrate changes from FreeBSD. Rerolling master was universally rejec= ted when we socialized wanting to do this due to the level of disruption th= is would cause. The feedback was that this would be a high-cost, low-value= operation. In the tradeoffs of purity vs pragmatism, pragmatism wins ever= y time (that is the FreeBSD way). >> > > > >> > > > -sc >> > > >> > > >> > > This is not just about pragmatism and the disruption it would cause = is >> > > vastly overblown by people who don't seem to know much about the git >> > > storage model. >> > > >> > > There *is* garbage metadata in the published version on github, ther= e >> > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since >> > > forever, and the cost of switching from 1 published branch to anothe= r >> > > is literally: >> > > >> > > - git diff origin/broken_master mybranch > mybranch.patch >> > > - git checkout -b fixed_branch origin/fixed_master >> > > - patch < mybranch.patch >> > >> > Such a workflow breaks historical accuracy. Instead of `git annotate` >> > showing the history properly, it's now based on an "epoch commit". >> > Sure such a commit brings the branch to a working condition, but at >> > the cost of history. >> >> Is there really that much value in having "git blame" work in that envir= onment? >> My mental model is of short-lived branches that get upstreamed, so I'm b= iased >> towards this not being all that big of a problem (for some at least). > > > Yes. There absolutely is. > >> > > >> > > It should also be possible to merge both broken and fixed master int= o >> > > your branch (at the exact same SVN revision in time) and then you ca= n >> > > follow fixed_master going forward. You'll schlepp around double the >> > > commit history, but not tree objects. >> > > If you want to retain history, you can upstream the changes prior to >> > > the switch >> > >> > I so wish that were possible for certain downstream projects. We're >> > unable to upstream the majority of our work. To argue "upstream your >> > work and you won't be affected" is to choose an argument that does not >> > reflect the reality of a growing portion of FreeBSD's downstream >> > consumers: the inability to work effectively with upstream. >> >> :/ >> >> I'm 80% sure that you can just merge both branches and things will be fi= ne >> (though the exact incantation will surely be black magic). I'd love to >> try this on >> an actual repo though, I don't have the time to craft some test repo to = verify >> this assumption, and then find out that other repos are different). > > > One might be able to do this, but this is really quite a lot to ask. I've= done a lot of git rebase in connection with qemu. The hard part us knowing= hash X is for rev r123 in branch 1, but hash Y In branch 2. Once you can a= utomate that, for various mappings, a script to rebase old to new becomes p= ossible which will lessen the impact, but not eliminate it. Hmm, I think that's the trivial part. All commits record the SVN revision in the notes after all. See: % git show commit 2e105280ca193f7bafe103652bb1249704ba25f6 (HEAD -> master) Author: bdrewery Date: 2015-10-15 20:46:34 +0000 Remove unneeded libg++ reference that came in with r267511 based on a removed comment. Sponsored by: EMC / Isilon Storage Division Notes: svn path=3D/head/; revision=3D289388 So all that is left is matching up 2 notes in 2 branches (or 2 tree objects in 2 branches, actually). What's the tree on a specific branch: % git log -1 --format=3D%T master 9c96d76028084fe6b8077292fc428388a22f07f0 Search for it on a different branch (using the same master here, I don't have a repo handy with 2 different but identical branches) % git rev-list master | while read hash; do case `git log -1 --format=3D%T $hash` in 9c96d76028084fe6b8077292fc428388a22f07f0) echo "found commit $hash also pointing to tree 9c96d76028084fe6b8077292fc428388a22f07f0";; esac; done found commit 2e105280ca193f7bafe103652bb1249704ba25f6 also pointing to tree 9c96d76028084fe6b8077292fc428388a22f07f0 Except you can probably write that quicker also, maybe just dump all commit =E2=86=92 tree mappings and do something with that. As I said, it should be straightforward as long as you know a thing or two about git. > >> Also, I'll be offline in the coming weeks, so don't expect immediate res= ponses >> from me going forward. I already had to spent most of my weekend to patc= h >> things up and had to cancel other plans. > > > I think it is safe to view this discussion as just don't force push yet a= s we are in no position to mitigate the large disruption this would cause. = If you aren't force pushing master, there is no time urgency to solve this. > > Finally, and most importantly, thank you for all your time and efforts he= re. They are much appreciated. Any quibbling over future plans shouldn't de= tract from that. > > Warner There will not be a surprise force push if I can help it. That's what I just sacrificed a weekend for, after all :( Should (!!) the project ever decide to abandon SVN, then I would suggest to roll that into a re-do of the conversion, and also only start the conversion with the first commit to SVN, because the cvs2svn conversion also has a bunch of garbage in there, and we should rather not claim that those commits actually every happened in the way and form they are currently recorded in SVN. hth Uli From owner-freebsd-git@freebsd.org Mon Sep 23 21:16:21 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 592531214A7 for ; Mon, 23 Sep 2019 21:16:21 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46ccZD496Sz4YJL for ; Mon, 23 Sep 2019 21:16:20 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk1-x741.google.com with SMTP id x134so17090554qkb.0 for ; Mon, 23 Sep 2019 14:16:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=4LPcoqQ1nJh11e2+i+81ht4p9862Rl+JD3troD4pHY4=; b=fiTiQNy5KZefU7toy9AZld9LqtXOf6Of7XrFyt7RemSAivJ580QNJbnO9D1dXa54en EcNa1PLXyYB4e+H9YYTAwSIOP6foKWbZN35g/sUUUiXaJL1yQzocu/K9dnMRwxLD+1C3 VmgOI16WGpdJrlD0Kd/nawfTzlyG829AZ8LgCSvYziDQUgqhrHwAmAjBQppqnbv8mitK u/PB11lYzd0S/h/squwho8sp6YQ1S564C2G7TUhLo9Nc7ImpHZb524nl3Ur+4vFQBDI/ TbVP+Ju/dz74LSRx0221zop3t5zZybNrPYstFpDB5otpxTI7uENAcN5/FPrtRNgdqwwA h1Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=4LPcoqQ1nJh11e2+i+81ht4p9862Rl+JD3troD4pHY4=; b=K5kka38zjtkT8l0M4Xd0MiFue4dM4/araouyncOIIJ5GoMCl0MksY/e9pMkUw9aSgI ReDrKXPslCwKQkqlSJK1IRzGLHXOSD7OXQXEUTJn+Czc+lifb2BNJBV7acUMOTOlWHGe W6Kfurz5Xc/M0tNMucguNOBpeIK/BzYo+zersu0zmgKBy25VnaLEcf5usou+rx2mmJT5 T1pP7WsWYUUbnE0HVko27wIwlnH4pC6tPUlrxC7vLI1vLhLetf+81hFYzJm1cCmSWMnu OM+y4AVEtPXF9VofA8YCmhrEUQtqaUzxcqNYjcjjGbG+4cHfg6HocR3GfUC/7GAckp9A Ku+Q== X-Gm-Message-State: APjAAAUeV0pP4RGtJYpreLINZrYBCstZR8wwJ9USdcGgelRsI53eSkYA SRgaxOW3PIDoZs/3hYMICcTW8Q== X-Google-Smtp-Source: APXvYqwYyvsbHk1xzfjkITRDJfQxTHHveZstHwtpL7ze1rpHCQC3YoeI27NrH3dtWpTVGildgcjI8A== X-Received: by 2002:a37:6787:: with SMTP id b129mr2217309qkc.60.1569273379366; Mon, 23 Sep 2019 14:16:19 -0700 (PDT) Received: from mutt-hbsd (pool-100-16-231-224.bltmmd.fios.verizon.net. [100.16.231.224]) by smtp.gmail.com with ESMTPSA id t63sm6011522qkf.48.2019.09.23.14.16.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2019 14:16:18 -0700 (PDT) Date: Mon, 23 Sep 2019 17:16:17 -0400 From: Shawn Webb To: Ulrich Sp??rlein Cc: Sean Chittenden , freebsd-git@freebsd.org Subject: Re: Service disruption: git converter currently down Message-ID: <20190923211617.ifajkj2pwss346ub@mutt-hbsd> References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> <20190923185113.dyvxxn36gvj4dtu5@mutt-hbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="czzanpxa67by6knx" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT-HBSD FreeBSD 13.0-CURRENT-HBSD r352488+2a20025064d4-c272097(hardened/current/master) HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0xFF2E67A277F8E1FA User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 46ccZD496Sz4YJL X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=fiTiQNy5; dmarc=none; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::741 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-4.61 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; RCVD_IN_DNSWL_NONE(0.00)[1.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.51)[ip: (2.34), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; RECEIVED_SPAMHAUS_PBL(0.00)[224.231.16.100.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 21:16:21 -0000 --czzanpxa67by6knx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 23, 2019 at 10:38:36PM +0200, Ulrich Sp??rlein wrote: > Am Mo., 23. Sept. 2019 um 20:51 Uhr schrieb Shawn Webb > : > > > > On Mon, Sep 23, 2019 at 08:42:10PM +0200, Ulrich Sp??rlein wrote: > > > Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > > > : > > > > > > > > Hey Ulrich, > > > > > > > > I appreciate your hard work in maintaining the git mirror. Work like > > > > this can sometimes go unthanked. I want to take a moment to show > > > > appreciation for you and the FreeBSD project in maintaining the git > > > > mirror. > > > > > > > > I do have a few concerns with what was stated in your email. I've > > > > written my concerns inline. I hope this discussion is a positive on= e, > > > > wherein upstream and downstream can effectively come to a conclusio= n. > > > > > > > > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > > > > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > > > > > : > > > > > >> > > > > > >> Please note however, that more "garbage" metadata escaped from= SVN into > > > > > >> github, meaning 3rd parties have a hard time re-running the co= nversion and > > > > > >> making sure that it matches SVN down to the metadata (i.e. tim= estamps). > > > > > >> > > > > > >> Eventually, this will have to be re-rolled and a new "master" = branch will > > > > > >> be force-pushed into github. There's no timeline for this yet. > > > > > > > > > > > > > > > > > > Wait, what? Can you elaborate? > > > > > > > > > > > > Discussion of a force-push to github has occurred a few times a= nd been explicitly ruled out because most of our corporate citizens use git= hub to integrate changes from FreeBSD. Rerolling master was universally re= jected when we socialized wanting to do this due to the level of disruption= this would cause. The feedback was that this would be a high-cost, low-va= lue operation. In the tradeoffs of purity vs pragmatism, pragmatism wins e= very time (that is the FreeBSD way). > > > > > > > > > > > > -sc > > > > > > > > > > > > > > > This is not just about pragmatism and the disruption it would cau= se is > > > > > vastly overblown by people who don't seem to know much about the = git > > > > > storage model. > > > > > > > > > > There *is* garbage metadata in the published version on github, t= here > > > > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > > > > > forever, and the cost of switching from 1 published branch to ano= ther > > > > > is literally: > > > > > > > > > > - git diff origin/broken_master mybranch > mybranch.patch > > > > > - git checkout -b fixed_branch origin/fixed_master > > > > > - patch < mybranch.patch > > > > > > > > Such a workflow breaks historical accuracy. Instead of `git annotat= e` > > > > showing the history properly, it's now based on an "epoch commit". > > > > Sure such a commit brings the branch to a working condition, but at > > > > the cost of history. > > > > > > Is there really that much value in having "git blame" work in that en= vironment? > > > My mental model is of short-lived branches that get upstreamed, so I'= m biased > > > towards this not being all that big of a problem (for some at least). > > > > > > > > > > > > > It should also be possible to merge both broken and fixed master = into > > > > > your branch (at the exact same SVN revision in time) and then you= can > > > > > follow fixed_master going forward. You'll schlepp around double t= he > > > > > commit history, but not tree objects. > > > > > If you want to retain history, you can upstream the changes prior= to > > > > > the switch > > > > > > > > I so wish that were possible for certain downstream projects. We're > > > > unable to upstream the majority of our work. To argue "upstream your > > > > work and you won't be affected" is to choose an argument that does = not > > > > reflect the reality of a growing portion of FreeBSD's downstream > > > > consumers: the inability to work effectively with upstream. > > > > > > :/ > > > > > > I'm 80% sure that you can just merge both branches and things will be= fine > > > (though the exact incantation will surely be black magic). I'd love to > > > try this on > > > an actual repo though, I don't have the time to craft some test repo = to verify > > > this assumption, and then find out that other repos are different). > > > > HardenedBSD's github repo has existed since 2013, with branches > > stemming from that work existing still today. Perhaps HardenedBSD is > > somewhat in a special case: we aim to provide the BSD community with a > > clean-room reimplementation of publicly-documented parts of the > > grsecurity patchset. > > > > With FreeBSD not taking the same approach, we will have very > > long-lived branches. For example, our hardened/current/master branch > > follows FreeBSD's HEAD and syncs every six hours. Meaning, we maintain > > our patches, resolving whatever few merge conflicts arrive. The > > hardened/current/master branch was created so many years ago, I've > > forgotten when it was actually created (perhaps in 2013?) > > > > Though HardenedBSD's cause for existence may be a special case, this > > problem can be viewed in a general fashion. I'm confident HardenedBSD > > is not alone in facing issues of these types. > > > > Thanks, >=20 > What I don't understand is how a security focused project can trust a > random source for the svn2git conversion. I could have planted a bunch > of backdoors and then come up with some SVN metadata corruption > conspiracy as to why the commit hashes are different. Why would you > trust me? >=20 > HardenedBSD of all people should be running the converter themselves > and check that the content really matches what is in SVN (which it > currently doesn't for metadata). There's far easier ways for upstream to introduce code into downstream's infrastructure. Just adding `nc -l -p 1234 -e /bin/sh` to a Makefile would do it. Or introducing a vulnerability into telnet in 2019. The possibilities are limited to the imagination and the number of XKCD-obeying butterflies one has on-hand on any given Monday. We at HardenedBSD have to trust FreeBSD in many (uncomfortable) ways. These trust issues are shared amongst all derivatives of FreeBSD, regardless of underlying version control system. Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0xFF2E67A277F8E1FA GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9 3633 C85B 0AF8 AB23 0FB2 --czzanpxa67by6knx Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAl2JNhoACgkQ/y5nonf4 4fqlDBAAglY5L6xmE3j3o0XKvaQbBesNsDgySAm8uwGWd/5FYKzLuCjlJL989auI +0qxn+waRykOzXxdmyOK7sZh2pbMi3TAxZKekzT9kw+dBx+o77gYvw5ajC+kn9DE 4b857ohbMXxxRyXPWsiEJKC2sMgd4uald2Wc2TG44xtfcDB0+6fL6YJCeDLGLezz fiXU1EqpBEFw05J9qYH9YF3txvQPLoxf/guBaKN8cJBYp4dR4eGNrhagiDaE39o7 UwUSpvQmSuFIOUBTF4cC661iSy0PUbLhq/Y5HzYVA2INdfsvzmjszE1Nl2QRk96D qDHSMe1KwZLqvyosWV3MhnnKi62kIJsEIjY+uLJjne/3/uMoOcbQ8UthOzxpTuJF 79l+or2nQNBzCNZw5Dc84X5r6Lxz3DGH2s6bG7r7g1JXfAXSjBE+i9xdTfyN0smc 0Ab8rpj5aq6yRSUWOQY2UCeDHj1BbsSfNcNpzM146BnSLZkW6TSfqw+kn3AIdN1G NKvkPkHNrOhMyYdRqJaYekt/QsHRsa0aXVn5l+QC7DbBX0M3SdaAOJbObABsZnul 5chn24BBzh85Ui6OeFbTOLv3CQOkmWxUhQaqHPKo44vdF4AVHt6om1qfJyoBbSlw p3GbBKLJfmfnqaTFnAVEuMivgCFWkB1q7C/rXYkrbxBCq8WMMIM= =yc8P -----END PGP SIGNATURE----- --czzanpxa67by6knx-- From owner-freebsd-git@freebsd.org Mon Sep 23 21:40:41 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B5E27121BB2 for ; Mon, 23 Sep 2019 21:40:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46cd6J27L5z4Zf1 for ; Mon, 23 Sep 2019 21:40:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x835.google.com with SMTP id n7so19045277qtb.6 for ; Mon, 23 Sep 2019 14:40:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZFdLQ7dS+sUW+9X2ENRQ0bJJo0/Pe3X4kuz51f7+PNw=; b=nUPIo4kNmEkqtXJEWD4BCFphN8npz9tMbdX7YPU3wFbSrscj+kKjl9n/5XaLpUAnCS eo4U8IJT2U7klb3wpgMBCQwvdMNl9AyCISYrKxv2XVITsucDIErRpR9miDWHjXRjQ339 fElpvgJpqYM+RG1ldWuCzvAwAkqKmgGRgKZ8pozpz1lMk4K7n9PfDSOpGNl9EzLMw69S Qxo/MSnddaW0L2bBlVVwL+EdBLOsdKquoXZLw3gOrbAyuxkba3ge6R+Hr4OE82zTbORo dcSQTS2PaLfM2eLi/Dihy8gIdrqh2gzfKaCfgdhNSEKt/TIkNxne2ibAMSUCGJd5CtWa lRcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZFdLQ7dS+sUW+9X2ENRQ0bJJo0/Pe3X4kuz51f7+PNw=; b=QYVBFamgQOPcmTNv54xtcL0iJUbUldRgljmcvLuDzbGqDvSABwAeBctRq8j5h2Wngy R0eaqwz793BHaKKdgBvqhr1ld4oajSTBQjvxaBJ081NPbTdAoTOPBLMu3aON613YbnrX mYdSwCnL/IUCnpbUUeg1l/8h2rxJPSeYqFqOzp385VOlgJcgIY1Gry0eLHpFKglJtwnZ SWOM7X6OZvYk0fD4WDkPbV3gq7jzw2v6sY+ZzVLCG/dP4et+l8OXXxjWnhWGzMQ72iaT NyarDezXKdBvvsEMM4gHDCAFOCFmuBv/KmoyZrv+plGOhFteFJX6pMUvUKPOooksjGB9 cYNQ== X-Gm-Message-State: APjAAAW/E8Twu2Y8Zd87vEOMotccpONXvkhxcq1mTj34GiS/tEHwFCvy Jz0Dv7KFoYQ8lBkLHjLkwmIHCfJe8dadndCD5FIvYQ== X-Google-Smtp-Source: APXvYqzlsC82lU+7ue8WPw7M6lHkdK09nNK0xJ4W2lpe9Aww1/LvD4JDI9no3BMCE8dgRzH8awIQhAlMFpM12L3UMyc= X-Received: by 2002:ac8:2bca:: with SMTP id n10mr2378195qtn.242.1569274838533; Mon, 23 Sep 2019 14:40:38 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: From: Warner Losh Date: Mon, 23 Sep 2019 23:40:25 +0200 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: Shawn Webb , freebsd-git@freebsd.org X-Rspamd-Queue-Id: 46cd6J27L5z4Zf1 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=nUPIo4kN; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::835) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.86 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[5.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.86)[ip: (-9.39), ipnet: 2607:f8b0::/32(-2.64), asn: 15169(-2.20), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 21:40:41 -0000 On Mon, Sep 23, 2019, 11:09 PM Ulrich Sp=C3=B6rlein = wrote: > Am Mo., 23. Sept. 2019 um 21:06 Uhr schrieb Warner Losh : > > > > > > > > On Mon, Sep 23, 2019, 8:42 PM Ulrich Sp=C3=B6rlein > wrote: > >> > >> Am Mo., 23. Sept. 2019 um 20:34 Uhr schrieb Shawn Webb > >> : > >> > > >> > Hey Ulrich, > >> > > >> > I appreciate your hard work in maintaining the git mirror. Work like > >> > this can sometimes go unthanked. I want to take a moment to show > >> > appreciation for you and the FreeBSD project in maintaining the git > >> > mirror. > >> > > >> > I do have a few concerns with what was stated in your email. I've > >> > written my concerns inline. I hope this discussion is a positive one= , > >> > wherein upstream and downstream can effectively come to a conclusion= . > >> > > >> > On Mon, Sep 23, 2019 at 08:16:25PM +0200, Ulrich Sp??rlein wrote: > >> > > Am Mo., 23. Sept. 2019 um 19:51 Uhr schrieb Sean Chittenden > >> > > : > >> > > >> > >> > > >> Please note however, that more "garbage" metadata escaped from > SVN into > >> > > >> github, meaning 3rd parties have a hard time re-running the > conversion and > >> > > >> making sure that it matches SVN down to the metadata (i.e. > timestamps). > >> > > >> > >> > > >> Eventually, this will have to be re-rolled and a new "master" > branch will > >> > > >> be force-pushed into github. There's no timeline for this yet. > >> > > > > >> > > > > >> > > > Wait, what? Can you elaborate? > >> > > > > >> > > > Discussion of a force-push to github has occurred a few times an= d > been explicitly ruled out because most of our corporate citizens use gith= ub > to integrate changes from FreeBSD. Rerolling master was universally > rejected when we socialized wanting to do this due to the level of > disruption this would cause. The feedback was that this would be a > high-cost, low-value operation. In the tradeoffs of purity vs pragmatism= , > pragmatism wins every time (that is the FreeBSD way). > >> > > > > >> > > > -sc > >> > > > >> > > > >> > > This is not just about pragmatism and the disruption it would caus= e > is > >> > > vastly overblown by people who don't seem to know much about the g= it > >> > > storage model. > >> > > > >> > > There *is* garbage metadata in the published version on github, > there > >> > > *is* a disclaimer on https://wiki.freebsd.org/GitWorkflow since > >> > > forever, and the cost of switching from 1 published branch to > another > >> > > is literally: > >> > > > >> > > - git diff origin/broken_master mybranch > mybranch.patch > >> > > - git checkout -b fixed_branch origin/fixed_master > >> > > - patch < mybranch.patch > >> > > >> > Such a workflow breaks historical accuracy. Instead of `git annotate= ` > >> > showing the history properly, it's now based on an "epoch commit". > >> > Sure such a commit brings the branch to a working condition, but at > >> > the cost of history. > >> > >> Is there really that much value in having "git blame" work in that > environment? > >> My mental model is of short-lived branches that get upstreamed, so I'm > biased > >> towards this not being all that big of a problem (for some at least). > > > > > > Yes. There absolutely is. > > > >> > > > >> > > It should also be possible to merge both broken and fixed master > into > >> > > your branch (at the exact same SVN revision in time) and then you > can > >> > > follow fixed_master going forward. You'll schlepp around double th= e > >> > > commit history, but not tree objects. > >> > > If you want to retain history, you can upstream the changes prior = to > >> > > the switch > >> > > >> > I so wish that were possible for certain downstream projects. We're > >> > unable to upstream the majority of our work. To argue "upstream your > >> > work and you won't be affected" is to choose an argument that does n= ot > >> > reflect the reality of a growing portion of FreeBSD's downstream > >> > consumers: the inability to work effectively with upstream. > >> > >> :/ > >> > >> I'm 80% sure that you can just merge both branches and things will be > fine > >> (though the exact incantation will surely be black magic). I'd love to > >> try this on > >> an actual repo though, I don't have the time to craft some test repo t= o > verify > >> this assumption, and then find out that other repos are different). > > > > > > One might be able to do this, but this is really quite a lot to ask. > I've done a lot of git rebase in connection with qemu. The hard part us > knowing hash X is for rev r123 in branch 1, but hash Y In branch 2. Once > you can automate that, for various mappings, a script to rebase old to ne= w > becomes possible which will lessen the impact, but not eliminate it. > > Hmm, I think that's the trivial part. All commits record the SVN > revision in the notes after all. See: > > % git show > commit 2e105280ca193f7bafe103652bb1249704ba25f6 (HEAD -> master) > Author: bdrewery > Date: 2015-10-15 20:46:34 +0000 > > Remove unneeded libg++ reference that came in with r267511 based > on a removed > comment. > > Sponsored by: EMC / Isilon Storage Division > > Notes: > svn path=3D/head/; revision=3D289388 > > > So all that is left is matching up 2 notes in 2 branches (or 2 tree > objects in 2 branches, actually). > > What's the tree on a specific branch: > % git log -1 --format=3D%T master > 9c96d76028084fe6b8077292fc428388a22f07f0 > > Search for it on a different branch (using the same master here, I > don't have a repo handy with 2 different but identical branches) > % git rev-list master | while read hash; do case `git log -1 > --format=3D%T $hash` in 9c96d76028084fe6b8077292fc428388a22f07f0) echo > "found commit $hash also pointing to tree > 9c96d76028084fe6b8077292fc428388a22f07f0";; esac; done > found commit 2e105280ca193f7bafe103652bb1249704ba25f6 also pointing to > tree 9c96d76028084fe6b8077292fc428388a22f07f0 > > Except you can probably write that quicker also, maybe just dump all > commit =E2=86=92 tree mappings and do something with that. > > As I said, it should be straightforward as long as you know a thing or > two about git. > And I'm saying that it needs to be simple and easy to convert. I know a lot about git and the above is neither simple nor easy. We need a bulletproof script to rebase current branches into the new branch. For maximum safety, this should be at identical branch points (the less safe rebasing to the new branch is, I'll grant, relatively easy). So, the above is a good sketch to get started, but no where near sufficient given our current user base. And I've not even thought through the implications of people that forked from our repo on github and what the change means to them (or if it is really all the same problem). I'm willing to stick my hand up here to be involved in any automation. I have a enough to loose in WIP work alone to be motivated that this process just work... > > >> Also, I'll be offline in the coming weeks, so don't expect immediate > responses > >> from me going forward. I already had to spent most of my weekend to > patch > >> things up and had to cancel other plans. > > > > > > I think it is safe to view this discussion as just don't force push yet > as we are in no position to mitigate the large disruption this would caus= e. > If you aren't force pushing master, there is no time urgency to solve thi= s. > > > > Finally, and most importantly, thank you for all your time and efforts > here. They are much appreciated. Any quibbling over future plans shouldn'= t > detract from that. > > > > Warner > > There will not be a surprise force push if I can help it. That's what > I just sacrificed a weekend for, after all :( > And we all owe you big for that. Thank you! Should (!!) the project ever decide to abandon SVN, then I would > suggest to roll that into a re-do of the > conversion, and also only start the conversion with the first commit > to SVN, because the cvs2svn conversion > also has a bunch of garbage in there, and we should rather not claim > that those commits actually every happened > in the way and form they are currently recorded in SVN. > Rerolling has issues. Not rerolling also has issues. We should collectively decide on which set of issues we want. Warner hth > Uli > From owner-freebsd-git@freebsd.org Tue Sep 24 13:48:25 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ED975127588 for ; Tue, 24 Sep 2019 13:48:25 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46d2Zx0w1Xz4V41 for ; Tue, 24 Sep 2019 13:48:24 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-lf1-f42.google.com with SMTP id w6so1473028lfl.2 for ; Tue, 24 Sep 2019 06:48:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=I7Zbd424UkQnRyvvwXPfelbbJpGPF9O3Xd9QSbKcY/U=; b=g2w9BcYqQH/6ecEt4hmX9sYlALmD1Lu74qbBYJxlvvpkTDfa9TbIjmUFcSewd6jq5t Yvk+rR/r9ufZEcY4M3R9ePNLm6rd0bu9F1HbD5Xq4TtDuyxpy/jdvb74UM0bkMUxPTBi Tg95Zg126RG6ahL49L1+DMwsw+Us7pxJqULaKZE5ccNIRAyhmHmMH/Arux7kMbuf6J2c /RTd+USxpGmqTfa5wjhbrvGxBvNXDIv2BV/8my2ShY19SZ5oRenlirLPFhh09n99OkuC QNXmb0c3isxC+5bs2zu7Rhnb/H5A47hwnVU/S4d/HP4WO/v1yjoqIQSdPJihNplPZlvF AN1g== X-Gm-Message-State: APjAAAV2CX+ksUFeFlg48kixmp8CC0j7HQvcQxImaJ6ULXkkaGCm3jru PaSQUlg5UQixAt1MqMWCphyxCC/vh4Qd3pv3I0kCKsSocgI= X-Google-Smtp-Source: APXvYqxI/ipng8hgQ94KDgdQnD9sxdEXy+PWaDVc5/ftT2FY0qAawcs8t7/AfEpWj7IpD3fiTw/ao7CIXx7/HouqCqs= X-Received: by 2002:ac2:48ba:: with SMTP id u26mr2000297lfg.164.1569332903251; Tue, 24 Sep 2019 06:48:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Tue, 24 Sep 2019 09:48:10 -0400 Message-ID: Subject: Re: Service disruption: git converter currently down To: Sean Chittenden Cc: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46d2Zx0w1Xz4V41 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-3.21 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[42.167.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.22)[ip: (-0.53), ipnet: 209.85.128.0/17(-3.30), asn: 15169(-2.20), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[42.167.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2019 13:48:26 -0000 On Mon, 23 Sep 2019 at 13:51, Sean Chittenden wrote: > > > > > Please note however, that more "garbage" metadata escaped from SVN into > > github, meaning 3rd parties have a hard time re-running the conversion and > > making sure that it matches SVN down to the metadata (i.e. timestamps). > > > Discussion of a force-push to github has occurred a few times and been > explicitly ruled out because most of our corporate citizens use github to > integrate changes from FreeBSD. Unfortunately it's not just a few instances of bogus metadata (commit timestamps and authors), there are also a cases where the git conversion misrepresents the operation performed and makes for confusing history. One example of this is r306097 / df422cbea30e. A file that had been removed in head needed to be restored, and in SVN it was done by copying from the stable/10 branch. The git converter interpreted that as a merge, and then grafted the post-branch stable/10 history into head as a result. That is why we have a bit over 9000 stable/10 commits showing up in 'git log origin/master'. From owner-freebsd-git@freebsd.org Tue Sep 24 14:03:14 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ECFA4127960 for ; Tue, 24 Sep 2019 14:03:14 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46d2w16N4vz4Vp5 for ; Tue, 24 Sep 2019 14:03:13 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f46.google.com with SMTP id z19so4735984ior.0 for ; Tue, 24 Sep 2019 07:03:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=MplZFWS1jLMKWKfE2HyefWyI/sgjjWG1EZkls6+u7So=; b=tuDOyc0UzP04HGweAG/gOg2hmPejfbOSUKsyTsgbhaEwc1haOG69+GCU9gdbtFFfad 3YBNMav7nVQauHYhZZPfx7ErGQIbn2fN2L21D04vs7YEUwPiqjyPAzb4wg3bZ8DFk9cr WRTasuo0/r6cZjPgw2OF2Vn7dfd0CsuFrC/j43bxALhGehgSaWi/E/pxOLutUmX6lDaG DpDdt+aMt1LwAe0sB/X28Ej2XXSb9eyjjcV2EiooiBW694sGZf9devitIjnBX8z4Zfi5 d54AelMPdHSXhJeL1f1aLuTl6gAEsYmVfb+LTZcxEUAYyfR6PeY9Z2j85vTkxkZ5pUfT SRCA== X-Gm-Message-State: APjAAAXt9lBSTq+KA2bHpmfmszODDSOgnYhz0Lihz5Zk/cSLNMLQnGT6 7ProYxmqd61WYfHEGcjsJ5WiR1Wi2RCXlrrn4dLPK/6N X-Google-Smtp-Source: APXvYqxYUPuMliL8oYVuBb7aa3BkbDTVQ1io93T7FgeSDMFmIKXefSWZnitD0G/hjqEkd01ZjuVfZT12S1E6KMswsPo= X-Received: by 2002:a02:ba12:: with SMTP id z18mr3962449jan.16.1569333791977; Tue, 24 Sep 2019 07:03:11 -0700 (PDT) MIME-Version: 1.0 References: <20190923183424.ebnghzf67mx56aom@mutt-hbsd> In-Reply-To: From: Ed Maste Date: Tue, 24 Sep 2019 10:02:58 -0400 Message-ID: Subject: Re: Service disruption: git converter currently down To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: Shawn Webb , freebsd-git@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46d2w16N4vz4Vp5 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.46 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-4.28 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[46.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.28)[ip: (-5.88), ipnet: 209.85.128.0/17(-3.30), asn: 15169(-2.20), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[46.166.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2019 14:03:15 -0000 On Mon, 23 Sep 2019 at 14:42, Ulrich Sp=C3=B6rlein w= rote: > > Is there really that much value in having "git blame" work in that enviro= nment? > My mental model is of short-lived branches that get upstreamed, so I'm bi= ased > towards this not being all that big of a problem (for some at least). For FreeBSD developers working against the git mirror and committing to SVN I agree it's not much of a concern. But we have a significant number of downstream consumers using the git mirror with long-lived derived branches and churn here is going to be somewhat painful. I suspect your suggestion about merging from both broken and fixed to the derived branch is the most usable solution for folks using a long-lived merge workflow. > I'm 80% sure that you can just merge both branches and things will be fin= e > (though the exact incantation will surely be black magic). I'd love to > try this on > an actual repo though, I don't have the time to craft some test repo to v= erify > this assumption, and then find out that other repos are different). Yes, once we have some time to rerun the conversion (using new branch/merge configuration for the stable/10 issue) we should test this out. From owner-freebsd-git@freebsd.org Wed Sep 25 14:53:09 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2E4F31298BD for ; Wed, 25 Sep 2019 14:53:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46dgz80g3lz413L for ; Wed, 25 Sep 2019 14:53:07 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x743.google.com with SMTP id y144so5487593qkb.7 for ; Wed, 25 Sep 2019 07:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dj+T0AOydrV6JDHgQyNbcM8U49yc2hObkX1ABszTbW0=; b=YfGwxUNkv7uxAOZRxnYAGG7UURs72J2AA7d9KRxqS9zAUO9hoe4txuxcl3AtqA2rwg yI8MKhq59U94dXN/BHaP7SgfT+z3ggwhx0nQuZImLInH3VhT1R3HruimWXyPGB32/9aA 34qoG6suV7GIDWVHsGO5Dszz63qlf1Z71+fYvEmwLkSrEQiR8EJcb+TqeJ7tPaKWJOwq 1ByMf8DB1xdNV+hhUE3g+IqTa30GLRxGQo8uKcAhfrs3RmxLS9HRO2mOHj6gMK9HaYQH tDI5BDWe04PQW7CB9SBqesNQJBMomqkVH+4pVOgHMAaLbdqjgvUNzuGIsdu7u/3gNrlB NcrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dj+T0AOydrV6JDHgQyNbcM8U49yc2hObkX1ABszTbW0=; b=Tg5jsS8NPT5EJnTB4+jpncS+oZaldV9xyF39EqhwLFbzsgK7jcs//3acV1d2bzeNya PqSlaWoxlZ6LI8Ct1JqMKFWHZR7yx+c8YCRaeNGWsia+0iZzupKZfHIdJUmBWjsglZZf nKojUORJMakCWhcTjlQdAO3VySHssnaXRjuWbuBP63/OGbjU97X5LMxXavcKmM7G1jI3 aiiWVITdxxyRfvxgzF09ZJEAMZe30fbDCmOG9lP/uz42uRA1mFTs+KL+we9NZNLYpxnB RmETV4ECX7w6aU+LUjpaSjgt2iMWWz3ld82RENBjA4MLY+nzupFTY7DbEgydXHM1Vnky b42Q== X-Gm-Message-State: APjAAAUPQiKozFLkj8pF4V6BeeNkjOIh6DHRHf+cL5u7JdBUOHaEZ7vz CN0dvy/rZJLMboUsaPJYHbV1Obf8Vrvu6Y625/Oxhg== X-Google-Smtp-Source: APXvYqx2IfNs0Sir56GII6XL6JQHJKV5afi0AF6JLq8w/BjidAE4eVMtklPoB9aLf5u0bwMHpcE3nUCGrJtAt7WFTuA= X-Received: by 2002:a05:620a:12b6:: with SMTP id x22mr340503qki.495.1569423186552; Wed, 25 Sep 2019 07:53:06 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Wed, 25 Sep 2019 08:52:54 -0600 Message-ID: Subject: Re: Service disruption: git converter currently down To: Ed Maste Cc: Sean Chittenden , freebsd-git@freebsd.org, =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= X-Rspamd-Queue-Id: 46dgz80g3lz413L X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=YfGwxUNk; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::743) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.43 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[3.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.43)[ip: (2.69), ipnet: 2607:f8b0::/32(-2.61), asn: 15169(-2.18), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2019 14:53:09 -0000 On Tue, Sep 24, 2019 at 7:48 AM Ed Maste wrote: > On Mon, 23 Sep 2019 at 13:51, Sean Chittenden wrote: > > > > > > > > Please note however, that more "garbage" metadata escaped from SVN into > > > github, meaning 3rd parties have a hard time re-running the conversion > and > > > making sure that it matches SVN down to the metadata (i.e. timestamps). > > > > > Discussion of a force-push to github has occurred a few times and been > > explicitly ruled out because most of our corporate citizens use github to > > integrate changes from FreeBSD. > > Unfortunately it's not just a few instances of bogus metadata (commit > timestamps and authors), there are also a cases where the git > conversion misrepresents the operation performed and makes for > confusing history. > I'm not sure it's completely wrong... I will absolutely agree, however it's confusing. One example of this is r306097 / df422cbea30e. A file that had been > removed in head needed to be restored, and in SVN it was done by > copying from the stable/10 branch. The git converter interpreted that > as a merge, and then grafted the post-branch stable/10 history into > head as a result. That is why we have a bit over 9000 stable/10 > commits showing up in 'git log origin/master'. > git log --first-parent fixes that, however. Git log's notion of what's sensible to display by default is a poor match to what most people think is sensible. It is a common thing to have to do with other upstreams I track. qemu has this same issue because it merges to the mainline. It's the main reason that I think merge commits to master are evil in a project like ours. It was the best trick I learned when rebasing our sbruno's qemu-bsd-user branch to a 3.1 base. But this really was a merge from stable/10 into master, so we run afoul of a confusing git log default. We did copy the file from stable/10, though in a more ideal world this would be represented as a cherry-pick. git doesn't preserve meta-data for that operation, though (except it can add a note to the commit message that's free form). So representing it as a merge is likely the least worst thing it can do to preserve the metadata (although some would argue that the 'guess about copies' code obviates the need for keeping this data around). I'd offer the opinion that needing to know about things like git log --first-parent vs having to rebase every single downstream fork, the former is several orders of magnitude less pain. We can ensure that there are no back doors by doing a diff at a known good version, seeing know differences with the svn repo and signing the next commit (since it implicitly signs all previous ones), but that is tricky in a dual world and may need to be reserved for the svn read-only flag day that we all know is coming. Warner From owner-freebsd-git@freebsd.org Wed Sep 25 17:56:05 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9F93512E86B for ; Wed, 25 Sep 2019 17:56:05 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46dm2D5LCMz4FmQ for ; Wed, 25 Sep 2019 17:56:04 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f50.google.com with SMTP id b19so1104771iob.4 for ; Wed, 25 Sep 2019 10:56:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NQTxpy0uQggKkHW4u+siYOURpjRIRrFotdRm9E7f55c=; b=ENVSe9QiX0DkPnrdM+Piz2Jf0er5IHUwRRVqH5zNcrOnobPn9BKuDfPwvPK8QPpKCL KoJ1mLMMAQpcQKFpm9ZNOavg1EoXAM92nwBOPmzZYBgxxYb7mjHrFty9HSt2WOE9A8c4 9/7X0zJCXig+VShD2My5d2NdyJq7vKpeztq7oFc/Ei+uWjKlHNvCPzL25DsaCpEzjMg6 XQnixqBKW15M2jGk6mZt7snSsvbjZbLa83getAQSPBmLqd1VvwRDmEEsvf9lsW38CI+f KgFIMIi0ekCHXnWEvCUPOo/ScRtyf/cw68Q3Gl3YENU9yedthhOIyq788/1qoofSy2sz BwNQ== X-Gm-Message-State: APjAAAXMOhevXPVb+9uaDbTHjeSs2S6PbFYGbafiQsABGLlXILRk6Ax+ 99HKPABtE8iiaKjP66xztPGQ7dEJ6dxlQfqVbN4= X-Google-Smtp-Source: APXvYqz1K1gqWN/fyQ2HCiNEF2h71bg3mhloOnapgPL3O44mtd7Dtpr72c+doUVFxcNyxSQKQqY0pFRl113yjTzQzWg= X-Received: by 2002:a02:1c02:: with SMTP id c2mr6536913jac.118.1569434163227; Wed, 25 Sep 2019 10:56:03 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Wed, 25 Sep 2019 13:55:51 -0400 Message-ID: Subject: Re: Service disruption: git converter currently down To: Warner Losh Cc: Sean Chittenden , freebsd-git@freebsd.org, =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46dm2D5LCMz4FmQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.50 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-4.37 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[50.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.37)[ip: (-6.33), ipnet: 209.85.128.0/17(-3.29), asn: 15169(-2.18), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[50.166.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2019 17:56:05 -0000 On Wed, 25 Sep 2019 at 10:53, Warner Losh wrote: > > But this really was a merge from stable/10 into master, I think this is just a question of semantics, not VCS details - I would argue that it is not a merge in the ordinary sense of the word, and is certainly not a merge in the git sense. You're right that this should be represented in history as if it were a cherry pick. Bogusly adding over 9000 commits to the history is definitely not the "least wrong" thing the conversion could do. git log --first-parent isn't really a solution here either, because there are cases where one legitimately does want history from both parents, especially working in downstream projects. > I'd offer the opinion that needing to know about things like git log --first-parent vs having to rebase every single downstream fork, We won't need to rebase every fork - in no case should the path forward be worse than uqs's suggestion of a merge from both old/new conversions. From owner-freebsd-git@freebsd.org Wed Sep 25 19:50:11 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 93EF7130C7A for ; Wed, 25 Sep 2019 19:50:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46dpYt6571z4Nxq for ; Wed, 25 Sep 2019 19:50:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x844.google.com with SMTP id c3so789674qtv.10 for ; Wed, 25 Sep 2019 12:50:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CzblJQszxNRaHvys/vowcH8zRg35FAhUSV7Ui/wwIaE=; b=rmioObAtcCjv8zd97Ndf3xCuz8DhcMKgK+pZkqOkY5FpfhLqfGwHntrH2BNdJ0YLtA lTQzlPumpW+whbyzljLsyokB0A3If7Bu0U9OQ1bj+yxbeugCla9CnPOAYXkamVrY45rX vb7tA++8sG9ApzlQLcL+cdYnn69MK/egI6qsfZ8xL7wFhotS+KLbV68R9O3gVqyGqTEb zISIkEg+N78z1kv33yiFigbqlIX/I+D+jpUXeW2bk7BHEKPhXcsphZuvC8RxsnUknCni IwTyKFl56mk2cK/shUNWOQMe6JUP6WJsr7hTp4Wa0y+UTe8nCRgGh1CTl2FC9bNkHfxM N73w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CzblJQszxNRaHvys/vowcH8zRg35FAhUSV7Ui/wwIaE=; b=pvRCDVqnEk2JqkRDInGhn7AfAApTGjHCPIYIC70Txv3E9w2qKLPhWYsre5cGSfMFa9 9XKCU/AhbF+3CDTKsceoBszYteUhzQ3vPXXA0WbvDGX/NZ3SyvmvSrWIXKQHvw9QmGAb NwOA+pDUF/KtiDk18ijFjtPF8tpfGj8koBt7mU4rKO0e/wG83TQJK4r5rpjdl3Tejpsd FKHuwB1O1otNas4ezpExGAvx6AOTQsJyaJeW9FXR2BahVtRsPuLTVbcrd9DejPjnAGGJ MUXU3TSliJZZbXzXMBjkKu+WMCvQnFKnwylTwY1dMUTiLSYCDZnXGFawe70mS0MbfW4B aPUA== X-Gm-Message-State: APjAAAVZoKIy2JsKzVzeuCluYORRCH7vxCPi6MnIbSwh+7wqyDZel418 s/A+fNR5Mn5KyanwH87U/jJtv3oObZCxBtYmgOzuHg== X-Google-Smtp-Source: APXvYqwNoo1iHNpQqVm3GCq2BOjsXzdEpmSWg4Mj2PNazHPZl+6tSV6crKQHVBihgLzpeveax72+eiIKQpNtG9PnE1A= X-Received: by 2002:a05:6214:70c:: with SMTP id b12mr1138807qvz.87.1569441009431; Wed, 25 Sep 2019 12:50:09 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Wed, 25 Sep 2019 13:49:58 -0600 Message-ID: Subject: Re: Service disruption: git converter currently down To: Ed Maste Cc: Sean Chittenden , freebsd-git@freebsd.org, =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= X-Rspamd-Queue-Id: 46dpYt6571z4Nxq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=rmioObAt; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::844) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.45 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[4.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.46)[ip: (2.56), ipnet: 2607:f8b0::/32(-2.61), asn: 15169(-2.18), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2019 19:50:11 -0000 On Wed, Sep 25, 2019 at 11:56 AM Ed Maste wrote: > On Wed, 25 Sep 2019 at 10:53, Warner Losh wrote: > > > > But this really was a merge from stable/10 into master, > > I think this is just a question of semantics, not VCS details - I > would argue that it is not a merge in the ordinary sense of the word, > and is certainly not a merge in the git sense. You're right that this > should be represented in history as if it were a cherry pick. Bogusly > adding over 9000 commits to the history is definitely not the "least > wrong" thing the conversion could do. > git log always requires added care. There's not actually 9000 commits there. The tree looks fine topologically. Its purely an artifact of git log. > git log --first-parent isn't really a solution here either, because > there are cases where one legitimately does want history from both > parents, especially working in downstream projects. > I'm pretty sure it would be fine, even in that case. > > I'd offer the opinion that needing to know about things like git log > --first-parent vs having to rebase every single downstream fork, > > We won't need to rebase every fork - in no case should the path > forward be worse than uqs's suggestion of a merge from both old/new > conversions. > IMHO, uqs suggestion is a complete non-starter, at least the "git diff | git patch" one. It destroys all local history, commit messages, etc. Except for the most trivial cases, it's not really going to fly with our users. His other, followup ones might be workable into scripts. I'm not sure you can merge, as there's no common ancestor that's recent enough to give it a chance at succeeding (since the different exports would have different hashes starting fairly early in our history). My experience with qemu is that long-lived merge-updated branches become quite difficult to cope with after a while. It took me three weeks to sort out that relatively simple repo. A rebase has a chance of working for people following a 'rebase' work flow. I can write the scripts to do that because I basically already have them and they just need to be modified to do a 'onto' for the easy to write, but slightly risky case of rebasing them across the chasm to a new point on master (as opposed to rebasing them to the same point on master). The latter is possible, just needs a bit more scripting and wouldn't be too bad to write and the least risky way to update across the chasm for people that don't want to (or can't due to conflicts) jump in the easy but risky way. However, for people like CHERIBSD who follow a 'merge from upstream' model which never rebases (since that would be anti-social to their down streams), I'm having trouble understanding how that could work. At work, we basically do the merge from upstream with collapse model, which I'm having trouble seeing how to move from old hashes to new. I'd like to know what the plan for that would be and would happily test any solution there with a copy of our repo. I'd even be happy to run experiments in advance of there being something more public available to see what options do or don't work. Warner From owner-freebsd-git@freebsd.org Thu Sep 26 14:26:59 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3418E1284E3 for ; Thu, 26 Sep 2019 14:26:59 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f66.google.com (mail-io1-f66.google.com [209.85.166.66]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46fHLV1lgpz4JXK for ; Thu, 26 Sep 2019 14:26:57 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f66.google.com with SMTP id q1so7125951ion.1 for ; Thu, 26 Sep 2019 07:26:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=kfQVTR8HyHKAjB/xtjRXJH9iv9cT7fGk01688so4NOg=; b=lmYu5Bf9jaJPijaO1NWggIO2XY7BlW5F4hFjne8FI5H1mhMe8Ej/y69N/vffC+5a35 np8i0YJ4OS+5fPDO9tLs+HTBjOODjtyR93AfZNBELxEtwzbYcF7DfpRkIlD3KnUabmAa 4lX9HpE0O+2Bg5AS1VrNqLYNNbQTBb+6CzqbzAnCa0xmo/sES6WWeCFnm1oPRbl5wHW4 hBZGo/6CMJ0IGTby3Rnff9yNf5e5xyJVXzZfZnNkFX73Bp59FziDKWHflZNzozDxVvEb elv2LCj0hhDj3lam/KDWELHV8c7o6/4xwyuAQ79OptPjIKcFEe4f8uMI0LPqnnH+WFmN 7Img== X-Gm-Message-State: APjAAAWC0yGLDPgb3bGmG/MlNHtVTnOWL0nlijkARYze9iITEIVylbG8 zSDw2h8JmPEDxZnYaXbvqSArXlQXNy71HsmRxPA= X-Google-Smtp-Source: APXvYqwjfZreMMukPR2s7yUHymlc3HYfKb6BMQtBdb3PpBCTYIOD9d+a/NBH/K2Mjr6vKPZL25MG1EGsmGTtkzb9vMM= X-Received: by 2002:a02:6616:: with SMTP id k22mr3613546jac.129.1569508016219; Thu, 26 Sep 2019 07:26:56 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Thu, 26 Sep 2019 10:26:43 -0400 Message-ID: Subject: Re: Service disruption: git converter currently down To: Warner Losh Cc: Sean Chittenden , freebsd-git@freebsd.org, =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46fHLV1lgpz4JXK X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.66 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-3.20 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[66.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.20)[ip: (-0.46), ipnet: 209.85.128.0/17(-3.29), asn: 15169(-2.18), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[66.166.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2019 14:26:59 -0000 On Wed, 25 Sep 2019 at 15:50, Warner Losh wrote: > > git log always requires added care. There's not actually 9000 commits the= re. The tree looks fine topologically. Its purely an artifact of git log. This seems to be getting into a philosophical discussion of what it means for a commit to exist. But, given the constraints in the way git represents commits the history crafted by the svn-git exporter indeed shows thousands of "phantom" commits. The converter should (and with uqs tweaks, would) represent the offending commit here as if it were a cherry pick, not a merge. In order to really represent this correctly we need to add to git metadata tracking file operations. Recording that path d1/f1 was copied from d2/f2 at some hash would allow us to properly represent this case as well as renames/moves. >> git log --first-parent isn't really a solution here either, because >> there are cases where one legitimately does want history from both >> parents, especially working in downstream projects. > > I'm pretty sure it would be fine, even in that case. It's not fine, because it omits the commits I want to see. >> > I'd offer the opinion that needing to know about things like git log -= -first-parent vs having to rebase every single downstream fork, >> >> We won't need to rebase every fork - in no case should the path >> forward be worse than uqs's suggestion of a merge from both old/new >> conversions. > > IMHO, uqs suggestion is a complete non-starter, at least the "git diff | = git patch" one. It destroys all local history, commit messages, etc. Except= for the most trivial cases, it's not really going to fly with our users. H= is other, followup ones might be workable into scripts. diff | patch is not the suggestion; the suggestion is to perform a merge from the "new" conversion. Other options (e.g. some sort of scripted commit replay) are at least no worse than that base case. > I'm not sure you can merge, as there's no common ancestor that's recent e= nough to give it a chance at succeeding (since the different exports would = have different hashes starting fairly early in our history). My experience = with qemu is that long-lived merge-updated branches become quite difficult = to cope with after a while. It took me three weeks to sort out that relativ= ely simple repo. In fact, the merge works fine, even with completely unrelated histories. You can try this by merging 'svn_head' (from git svn) to 'master' (from svn2git), using `git merge --allow-unrelated-histories origin/svn_head`. The resulting history has two copies of every commit, but the file contents are unchanged over the merge. If you try this in a tree with changes (i.e., try applying it to a long-running merge-based branch) every modified file will result in a conflict, but they can be trivially resolved in favour of the first version. From that point on merging from the "new" conversion will work as expected. > A rebase has a chance of working for people following a 'rebase' work flo= w. Indeed, for rebase workflow it's fairly straightforward. > However, for people like CHERIBSD who follow a 'merge from upstream' mode= l which never rebases (since that would be anti-social to their down stream= s), I'm having trouble understanding how that could work. At work, we basic= ally do the merge from upstream with collapse model, which I'm having troub= le seeing how to move from old hashes to new. I'd like to know what the pla= n for that would be and would happily test any solution there with a copy o= f our repo. I'd even be happy to run experiments in advance of there being = something more public available to see what options do or don't work. Could you expand on the "merge from upstream with collapse" - specifically, can you provide an example command used when merging from FreeBSD? From owner-freebsd-git@freebsd.org Thu Sep 26 17:26:19 2019 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AEE7112E371 for ; Thu, 26 Sep 2019 17:26:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46fMKR2nY5z4ZjQ for ; Thu, 26 Sep 2019 17:26:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x743.google.com with SMTP id f16so2445386qkl.9 for ; Thu, 26 Sep 2019 10:26:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OLWLAdr6J9snFuvFCS3wz/bcM16s1Xry7KpGGywl1jI=; b=LO6sKgEnQypNPS9CfIKbO8QAu2UKQ9tF0EpT459RO0dhqf+4T1HzeYYNimF/CwA31w MbWc+97+MQ0vAWniz0Yu/CHaArqY4oDfW4rhogM5GTPEQm/0lyHQSUBUnH1ajKAhBumD qVEMHAQqwlkGoI/wyq0DYsFR5okyaxnbK3M+0gkAgGaQUN0Olmecma68DfryUC5n0sW6 2ArmhkWJQHCS88P4FEbI0AKZRPx/LMcTDSm0VFfjcLWQeBBchE18dTdJaiK0oV4VSQ8i 0tvfWB6X06Apgaw06Lc70Xh9KvcqMFFB5J6P9sKOuBE96u8d+xs6tn8UZQnvtM41Sm8B xrrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OLWLAdr6J9snFuvFCS3wz/bcM16s1Xry7KpGGywl1jI=; b=kW4Uy+foxKCgWBe9Zkj+YJxYEzAvFsXsGpSYBFz3OqHVCWVgwTB2bMu/EO8rGFJXk1 YWggS64w2S5gvNv2g6p33TZ24j64MzqjeDTGKxWDsYCnN+qh1enGIrhYdbTNYQHn2boM 7EbBXCnat52mciEJYh/hYFWrMPbdbzHtGx5A/VIbHPotdjzwMGRLTXZ2dY8K/vOTI5Fg zhn9AS4d3YMATNcchzUOBPuoYP0ASTh9HTAIyKM8K1jEVxgcuFE2rgltCLtN6fmBI8Hk Ef6wV8iL4DPaBU6IOG0jO16m0E99ek0fSsj6W9lUBB08wTg5z1W9ghPSYfVFNqdDp96J Pe1Q== X-Gm-Message-State: APjAAAX4yC2vp7S8XzjSMBUxBfLA3jZ8OyNIwJRx8+MU6ZmxMuo3lgNN qLfUD4RlKCoFaeVd68683mdR0KLWjWOUWVoKzLEEZg== X-Google-Smtp-Source: APXvYqwdcZx8QAZJPVq3ZsAk4BQd9CDGvcEhxtrV/lbe0Df4PK5eOMS9z9xAfLGJ3k+GxOhjnyLnltINRTnPGxoUyeU= X-Received: by 2002:a05:620a:6af:: with SMTP id i15mr4454654qkh.380.1569518778354; Thu, 26 Sep 2019 10:26:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Thu, 26 Sep 2019 11:26:07 -0600 Message-ID: Subject: Re: Service disruption: git converter currently down To: Ed Maste Cc: Sean Chittenden , freebsd-git@freebsd.org, =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= X-Rspamd-Queue-Id: 46fMKR2nY5z4ZjQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=LO6sKgEn; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::743) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.43 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-git@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[3.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.43)[ip: (2.67), ipnet: 2607:f8b0::/32(-2.60), asn: 15169(-2.18), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2019 17:26:19 -0000 On Thu, Sep 26, 2019 at 8:26 AM Ed Maste wrote: > On Wed, 25 Sep 2019 at 15:50, Warner Losh wrote: > > > > git log always requires added care. There's not actually 9000 commits > there. The tree looks fine topologically. Its purely an artifact of git log. > > This seems to be getting into a philosophical discussion of what it > means for a commit to exist. But, given the constraints in the way git > represents commits the history crafted by the svn-git exporter indeed > shows thousands of "phantom" commits. The converter should (and with > uqs tweaks, would) represent the offending commit here as if it were a > cherry pick, not a merge. > Yes. This is the artifact of 'git log' I was talking about. Unlike subversion, it will log both sides of the merge commit back to the common ancestor unless you tell it to do otherwise, which is why we're seeing both legs of the tree. But git also has no way to represent cherry-picks. It just does them as regular commits. > In order to really represent this correctly we need to add to git > metadata tracking file operations. Recording that path d1/f1 was > copied from d2/f2 at some hash would allow us to properly represent > this case as well as renames/moves. > Right. We're not going to get that since the 'git' way is to divine that the copies happened. Unlike subversion, git doesn't use paths for version control... > >> git log --first-parent isn't really a solution here either, because > >> there are cases where one legitimately does want history from both > >> parents, especially working in downstream projects. > > > > I'm pretty sure it would be fine, even in that case. > > It's not fine, because it omits the commits I want to see. > The --first-parent actually mirrors what svn log shows today. What commits do you think that it omits? > >> > I'd offer the opinion that needing to know about things like git log > --first-parent vs having to rebase every single downstream fork, > >> > >> We won't need to rebase every fork - in no case should the path > >> forward be worse than uqs's suggestion of a merge from both old/new > >> conversions. > > > > IMHO, uqs suggestion is a complete non-starter, at least the "git diff | > git patch" one. It destroys all local history, commit messages, etc. Except > for the most trivial cases, it's not really going to fly with our users. > His other, followup ones might be workable into scripts. > > diff | patch is not the suggestion; the suggestion is to perform a > merge from the "new" conversion. Other options (e.g. some sort of > scripted commit replay) are at least no worse than that base case. > I'm curious how the 'merge' operation would work with such a remote common ancestor, especially with the complicated glitches in the currently exported tree. I'll have to play around with this since I have a similar situation with my git-svn trees that have identical commits to the github commits, but with different hashes (due to the differing ways git-svn and the github converter export the rXXXXX number), though maybe not with such a deep common ancestor that the old and new conversion trees would have. For the rebase workflow, I agree, it's pretty simple to graft a series of commits to a new branch, even one without a common ancestor. I do it all the time to move things back and forth between my git from github tree and my git from git-svn trees depending on my needs. > I'm not sure you can merge, as there's no common ancestor that's recent > enough to give it a chance at succeeding (since the different exports would > have different hashes starting fairly early in our history). My experience > with qemu is that long-lived merge-updated branches become quite difficult > to cope with after a while. It took me three weeks to sort out that > relatively simple repo. > > In fact, the merge works fine, even with completely unrelated > histories. You can try this by merging 'svn_head' (from git svn) to > 'master' (from svn2git), using `git merge --allow-unrelated-histories > origin/svn_head`. The resulting history has two copies of every > commit, but the file contents are unchanged over the merge. > I'll have to try that to see how well it works. I'd not used allow-unrelated-histories and had frequently run into this issue. In the past, I'd been warned off using that flag, but I'll give it another try. > If you try this in a tree with changes (i.e., try applying it to a > long-running merge-based branch) every modified file will result in a > conflict, but they can be trivially resolved in favour of the first > version. From that point on merging from the "new" conversion will > work as expected. > OK. I'll have to give it a spin to see where it takes me. > > A rebase has a chance of working for people following a 'rebase' work > flow. > > Indeed, for rebase workflow it's fairly straightforward. > > > However, for people like CHERIBSD who follow a 'merge from upstream' > model which never rebases (since that would be anti-social to their down > streams), I'm having trouble understanding how that could work. At work, we > basically do the merge from upstream with collapse model, which I'm having > trouble seeing how to move from old hashes to new. I'd like to know what > the plan for that would be and would happily test any solution there with a > copy of our repo. I'd even be happy to run experiments in advance of there > being something more public available to see what options do or don't work. > > Could you expand on the "merge from upstream with collapse" - > specifically, can you provide an example command used when merging > from FreeBSD? > We basically have an upstream called 'FreeBSD' that we fetch into our git repo: % git fetch FreeBSD master and then we create a beanch % git checkout -b merge-branch-rXXXXXX then we do the merge: % git subtree -P FreeBSD merge FreeBSD/master $HASH # resolve conflicts % git commit % git push then use use stash's pull-request to manage the landing into our master, but it's effectively a % git checkout master % git merge merge-branch-rXXXXXX which results in a fairly ugly master for us, which is the other reason I know the difference between git log and svn log behaviors so well :(. Effectively, the merges from upstream show up as a single merge commit, plus a number of follow-on fix-up commits. git subtree is both awesome and evil. Warnre Warner