From owner-freebsd-git@freebsd.org Fri Dec 2 10:07:07 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1AD0C62447 for ; Fri, 2 Dec 2016 10:07:07 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x22c.google.com (mail-lf0-x22c.google.com [IPv6:2a00:1450:4010:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7ABE41EDA for ; Fri, 2 Dec 2016 10:07:07 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x22c.google.com with SMTP id c13so191361630lfg.0 for ; Fri, 02 Dec 2016 02:07:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to; bh=OUh1DDtGmre8v2lgYnzROeTSKfulWW6MSIxERtaf1DM=; b=Inde8RmdJqQrt9/HlGSVNOU2X1cLzFC3popRW321tHMyEmShHFN/6JeNN7TFPArLxS KNGT3Y5b8vLAbbSyH46X3D6D2UnuwChUfL46PUqnmhLlrddthah67k2suZ/0dXBZ6ax7 BReu4vaw70hFxgn30qHXz0dHT0FyNjhIxgHouveGP8um2pjg6GE4pi2EU8Ekdi9iYmP4 C8QdWLeyjxsulmCDkoN47qCgxVpyZxNFVcwc1K3wn1x2cgJ1VU/sMTy5oyxVrjK+R4ho rLmpKtLzvvb7miC3yNASWijWhjLGa04gm0y16APT7vRhjpNVejURelhA3upe3sHH+zFi LSIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=OUh1DDtGmre8v2lgYnzROeTSKfulWW6MSIxERtaf1DM=; b=BQUBkMWMXvDStbA8BQmUUEwZLMMxu7Zm/LgZtmPJI3xnC5OJbKDBuDfaUiKbqYI/1b Fb0j4uLUeNHONeyHI4jh9FhrNvy4YwGdqOMdc1VWPrKkzObZNFoKFZVCkg9wj8yXf0HP FQCZgrVd8m1ZPnc1DpenH8fInuAMGG0LWTtZQL0qZTwwtbdObNi+htJff0b8Q9SuTZg7 HZiOjangqYRoTswPu6cyFcHjnLTkfHDk0OHATbp4+L24doax78L/UQaQAnelBkzosLy2 BKy4f3lwF/957KgJmHtk88ZK/zPa0CfHqdW67J3+azk5+QGZhvt0DchFYd6SRUp8HaOt 0JlA== X-Gm-Message-State: AKaTC01pKvAQPyjXeWpnRjvfpiqmNZ6O4OxOD9tp733A/qEU5Y7RFEtB6pj1cOL7wBxKsJO7HaLY3gDTt5v1yw== X-Received: by 10.25.28.21 with SMTP id c21mr17087090lfc.91.1480673224899; Fri, 02 Dec 2016 02:07:04 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Fri, 2 Dec 2016 02:07:04 -0800 (PST) From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Fri, 2 Dec 2016 11:07:04 +0100 X-Google-Sender-Auth: HKU-Cm530m_9ak0gdFXhrr28K5I Message-ID: Subject: Git mirroring halted for freebsd-base To: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 10:07:08 -0000 The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed. The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket). I'll update this thread in about 24h. Uli From owner-freebsd-git@freebsd.org Sat Dec 3 11:42:59 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5DE9C621C2 for ; Sat, 3 Dec 2016 11:42:59 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x22f.google.com (mail-lf0-x22f.google.com [IPv6:2a00:1450:4010:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D1781F68 for ; Sat, 3 Dec 2016 11:42:59 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x22f.google.com with SMTP id t196so212773365lff.3 for ; Sat, 03 Dec 2016 03:42:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:content-transfer-encoding; bh=JmzoThjDRQUHpTJZy1l+Hn6Z33Te1GlRBbpugDKihuQ=; b=DIJc0eUQPt5Qu/pUmBMV4qHU7a5dRxzCZ0d16f98MkMuAssDC+QFAmZPLBjJFl1TbN FLodrse9J3hmd4U/5EtF9OJmki4u2AgGtre/d66HPh2Saz5vTlk5H44sIqnOAS81fXnI VZo+37SZ19XyMLLLDFJodKbyrVgsoKy929hgLsmdX7xr4Gx4wVW6J0+qss6YchB1f1H9 GreGpi1PkZ+X7kd5rKzfUJTFXAiBTwkPxKyp2HOweSSwRfGhilbFaQwFi3ChU93J4eNS mqc56z+TyAF336TydVO06Mu4060tWYtr9ZZZ3NTku66CTt8scoKUNm0TfKKO0DY8735r Corw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:content-transfer-encoding; bh=JmzoThjDRQUHpTJZy1l+Hn6Z33Te1GlRBbpugDKihuQ=; b=m8+QOrHXP7abK/5hu6DR+BxwvPiWT/Y5VZ87GKSTBka3mKKxXht9oEBAx96QrizOIa +GBMfrnVFVJR3V0Gi4Crnhs94mnD6WsxYXU7aD5Ntqnw+hpu+xg1MtRagsfdVjQFLWSb QlUIz6iVV81cqSuOdxxb44IfcDFqBn8KZcnlUk3mWkQXEju+loYOpT6z6JlzaSRDld4q EXUFDjtNp9zouxYSvsJkno8J6gVOquHWKW1goyBdlpdJQsb7+tV/13SOX/ViI43It33A zqpF2YXwjciKs6ustSpORg+qBNWzh/zrU6/KgrSMMhSUskt6yYVVdlTLTMYjhk918hIC GDGQ== X-Gm-Message-State: AKaTC02jX5IeyJULEU80GUhoVQeoxGVdZSHakPX2b1DEBkFttY938oMT21vlAFvTUZB+Fw814eKfJN/8Ixbswg== X-Received: by 10.25.17.207 with SMTP id 76mr18477347lfr.93.1480765376648; Sat, 03 Dec 2016 03:42:56 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Sat, 3 Dec 2016 03:42:56 -0800 (PST) In-Reply-To: References: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sat, 3 Dec 2016 12:42:56 +0100 X-Google-Sender-Auth: rpOK9uKnmw9px5KNueatoU8LSak Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 11:42:59 -0000 2016-12-02 11:07 GMT+01:00 Ulrich Sp=C3=B6rlein : > The conversion process started chewing up 100% cpu without making much pr= ogress, first attempts to rectify this have failed. > > The svn2git conversion and pushes to github have been halted. Pushes to b= itbucket also have been halted (we're reaching the 2GB limit imposed by bit= bucket). > > I'll update this thread in about 24h. > Uli Service is fully restored now for the github mirror, sorry for the downtime= . Bitbucket will stop working soon because of the size limitations. Speak up if you require this mirror to be kept up-to-date. Cheers, Uli From owner-freebsd-git@freebsd.org Sat Dec 3 12:01:37 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FB3FC6310C for ; Sat, 3 Dec 2016 12:01:37 +0000 (UTC) (envelope-from r@robakdesign.com) Received: from mail-ua0-f169.google.com (mail-ua0-f169.google.com [209.85.217.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30F32CF5 for ; Sat, 3 Dec 2016 12:01:36 +0000 (UTC) (envelope-from r@robakdesign.com) Received: by mail-ua0-f169.google.com with SMTP id 51so306879768uai.1 for ; Sat, 03 Dec 2016 04:01:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=4qamOlO0VMzQ0SjAiCmFpRH7wtYj5YHmwAyWo3pCCsY=; b=lY4EwwfQO5AXGVgMEhQ5Gr+s95Xz9lqco3J9UDwBWIvwEW453MhvNfTxwWRlEgQWJv 7iYHgiMWOqYmYGoA0nTpBEIoTxXBuFeIQxR4Cuu3xiMAp097uaYoma4jWkHPCU2EdOHH a2PIJSkZmhKJwfLROViMMoBsvHsNzpL9B9WRmnAHD+aAnvEwvT+3DcV7EmJHVqp0GeUM 0bZc65+h4EQtACJA0ZZP3IngOe9kZgt2dM6qmvHkSscIAc4tZUugrrRHefzSKxfECGV1 qOSOUhbQo9N0rw1rBE/IGnZrkUAWQKeaskqx0/dELMlu0WbbPG20s1a7YAGWEEM1OOFG VY3g== X-Gm-Message-State: AKaTC01iSb1wl44/1yGMSoGEyyPRBpjUOwLDTVtVYXzr/7cphB1NGFjYNl2hNoaWluxdlQ== X-Received: by 10.159.32.99 with SMTP id 90mr31188450uam.138.1480766003773; Sat, 03 Dec 2016 03:53:23 -0800 (PST) Received: from mail-ua0-f176.google.com (mail-ua0-f176.google.com. [209.85.217.176]) by smtp.gmail.com with ESMTPSA id f65sm2191115vkd.13.2016.12.03.03.53.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 03 Dec 2016 03:53:23 -0800 (PST) Received: by mail-ua0-f176.google.com with SMTP id 51so306773997uai.1 for ; Sat, 03 Dec 2016 03:53:23 -0800 (PST) X-Received: by 10.176.3.84 with SMTP id 78mr36194499uat.117.1480766002592; Sat, 03 Dec 2016 03:53:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.103.23.3 with HTTP; Sat, 3 Dec 2016 03:53:02 -0800 (PST) In-Reply-To: References: From: =?UTF-8?Q?Bart=C5=82omiej_Rutkowski?= Date: Sat, 3 Dec 2016 11:53:02 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 12:01:37 -0000 On Sat, Dec 3, 2016 at 11:42 AM, Ulrich Sp=C3=B6rlein wro= te: > 2016-12-02 11:07 GMT+01:00 Ulrich Sp=C3=B6rlein : > > The conversion process started chewing up 100% cpu without making much > progress, first attempts to rectify this have failed. > > > > The svn2git conversion and pushes to github have been halted. Pushes to > bitbucket also have been halted (we're reaching the 2GB limit imposed by > bitbucket). > > > > I'll update this thread in about 24h. > > Uli > > Service is fully restored now for the github mirror, sorry for the > downtime. > Bitbucket will stop working soon because of the size limitations. > Speak up if you require this mirror to be kept up-to-date. > > Cheers, > Uli > Thanks for the update and for your work! Perhaps we should tweet/update web page about such events and their progress? I've seen people asking about it on IRC. Kind regards, Bartek Rutkowski From owner-freebsd-git@freebsd.org Sat Dec 3 16:09:44 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA1C7C65478 for ; Sat, 3 Dec 2016 16:09:44 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x243.google.com (mail-lf0-x243.google.com [IPv6:2a00:1450:4010:c07::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F193138C; Sat, 3 Dec 2016 16:09:44 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x243.google.com with SMTP id o141so24782858lff.1; Sat, 03 Dec 2016 08:09:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=uujUFCpDz476v8KBHVpGNR4QDHdto5uoPDs+EIn8Qm0=; b=K5y8xiZbVeERlYOMyT/lk4Xgg7XXiGvBQcAzoyzsigLe/an5wWyzbwuIeUMcSMvcAW LVfwAtQALleAWZoyjOcai+3CrqREeuWGc+AV03egKAhsHMbXpownZEvteff0UbCVEfrA +pHerAARi7D0trsfDxyw2C3G1xuQQiCAdNza4L2/qlf+OSCuEfIJ9XYttZ7ixIXHaEPS sOoHOtd0c+pKaqyCSdJFgX6TVUf/VlnszEwcZ1ASNvkV+RJfOxY/v9ygRyvaq82d8XT1 dhURFYg5J19F51GFYbrV5MDAxwWzMp32u5URbsQFhWMPIUc1zAUtw9PB9Q/mE2U+9qOS deQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=uujUFCpDz476v8KBHVpGNR4QDHdto5uoPDs+EIn8Qm0=; b=Vj1js8D0lrUP0EBYW+Q0ry5kvyMHCa39vG8AK5PPhb1zjji04uc5OyiJPzVaDFDUdd nqQmBDOWQBgsk7eVCZw0gl0DXQ7//4TfmRflbX0m9yDYyyWRn1qIxXpcLwI3G1j98Id+ CaCJ8DdwhDH41BBkZxItIkJlLP8Ib+Lu409Cs6630UMg+jBVLwH17s/vUx5I8AbtkVMx NmUzXUcWaPQGKvRBep9AcARNopyc7c1Aoi77LfjhzIIG6lcZnWEE/XRoK1QAB4D7SwHm oKdfdPBhVdFm/9/HprL3bU/v6y+3dSDL+Ma5eB62Hmw6nHek8hRm+csyVt2RaUoiGVez pdTA== X-Gm-Message-State: AKaTC020aDoAD2cgfrVOYHDJcHpjuq87OLpVQUKtsxocQdomk4b0Hq3303TnOtk5L5JCi2U6btc5Ik8Z4JEf1w== X-Received: by 10.46.0.229 with SMTP id e98mr25453089lji.33.1480781382148; Sat, 03 Dec 2016 08:09:42 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Sat, 3 Dec 2016 08:09:41 -0800 (PST) In-Reply-To: References: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sat, 3 Dec 2016 17:09:41 +0100 X-Google-Sender-Auth: bATeoUUaphZdgG4tUYUgnzEvyfo Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: =?UTF-8?Q?Bart=C5=82omiej_Rutkowski?= Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 16:09:44 -0000 2016-12-03 12:53 GMT+01:00 Bart=C5=82omiej Rutkowski : > > > On Sat, Dec 3, 2016 at 11:42 AM, Ulrich Sp=C3=B6rlein w= rote: >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp=C3=B6rlein : >> > The conversion process started chewing up 100% cpu without making much >> > progress, first attempts to rectify this have failed. >> > >> > The svn2git conversion and pushes to github have been halted. Pushes t= o >> > bitbucket also have been halted (we're reaching the 2GB limit imposed by >> > bitbucket). >> > >> > I'll update this thread in about 24h. >> > Uli >> >> Service is fully restored now for the github mirror, sorry for the >> downtime. >> Bitbucket will stop working soon because of the size limitations. >> Speak up if you require this mirror to be kept up-to-date. >> >> Cheers, >> Uli > > > Thanks for the update and for your work! Perhaps we should tweet/update web > page about such events and their progress? I've seen people asking about it > on IRC. > > Kind regards, > Bartek Rutkowski > I left a note about this on IRC last night, too. But it's hard to know which would be the most appropriate channel. From owner-freebsd-git@freebsd.org Sat Dec 3 17:21:55 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8412C65DCD for ; Sat, 3 Dec 2016 17:21:55 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt0-x236.google.com (mail-qt0-x236.google.com [IPv6:2607:f8b0:400d:c0d::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D3ACD2 for ; Sat, 3 Dec 2016 17:21:55 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt0-x236.google.com with SMTP id w33so279576701qtc.3 for ; Sat, 03 Dec 2016 09:21:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=NhxryPUHFnN/E7aAC5XzspIOsbzpN9VSXoGoXj6QhlM=; b=Go47N0lzxxs1fKaoCbcdF0/J8pf0MLJsM8O1mK/fcXqPEPpV4bUDUksPq3oRoQTlQX MjQRH0d9n/G5KgL2ZNsmCqDebaFL8jwVMy4EGFhb/mgH/NR0aLHhLdzWAGxXZUkhJpL7 P1QxVTuS8Ymt4VM8AK5s4ZGGfioyEK3366YkY5SnHSRqgI3ucnkAdds6E7PE4XjZDrE1 ph91ERIrFBF8OWzPvVqkGw65OL9t0A4hs9i8rUoB3/19F4yaBkqbwu3uK7TMcuayk2hN R6pgJrfq0TfQhdqvsxu/fdPVhc288bYKkS0pk5p6o9XsVQ5IyjFXiEkNDhO0o8BSE4ug P2zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=NhxryPUHFnN/E7aAC5XzspIOsbzpN9VSXoGoXj6QhlM=; b=AcvHsTINIFicXjq/cj39xNY2UNcsTaossdhFrQd/ouRSZNHWH79umwYN2eH4jOh9WI XauZmI64SVzlcXZ6F1lnAq5nfIkN/Ppga/y5Hhu/wyDHA8XMPLE3g/2BfgHYg4VNDc4I z9E97c93R6GqURKIC44HAPR8KGYf1YQEhsxI48MsDjqi3XFOjgd3rbsqjzKvidbcpYd8 O4Ed98bb50M8heWdjvqzIlienvd7mSYwVLTDCBtVQaPNPlLGHfMK9rRkiVQNztyWfXm7 ktKkxeYNfaW8Wp5J4W7eryM7rp6TdlybCzgIdeszYa/49lrLaO88PdFts6AyA8ftoqr9 7HKw== X-Gm-Message-State: AKaTC00fz7YyoVK1YIheFUwujcqBVxQhcdLMcV9vwTufP9RFdidVyowm9DrusPT0/wzJc7t6 X-Received: by 10.237.58.138 with SMTP id o10mr43335606qte.76.1480785714731; Sat, 03 Dec 2016 09:21:54 -0800 (PST) Received: from mutt-hardenedbsd (pool-100-16-218-231.bltmmd.fios.verizon.net. [100.16.218.231]) by smtp.gmail.com with ESMTPSA id 5sm5303634qts.47.2016.12.03.09.21.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 03 Dec 2016 09:21:53 -0800 (PST) Date: Sat, 3 Dec 2016 12:21:51 -0500 From: Shawn Webb To: Ulrich Sp??rlein Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203172151.GA82677@mutt-hardenedbsd> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 17:21:56 -0000 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : > > The conversion process started chewing up 100% cpu without making much = progress, first attempts to rectify this have failed. > > > > The svn2git conversion and pushes to github have been halted. Pushes to= bitbucket also have been halted (we're reaching the 2GB limit imposed by b= itbucket). > > > > I'll update this thread in about 24h. > > Uli >=20 > Service is fully restored now for the github mirror, sorry for the downti= me. > Bitbucket will stop working soon because of the size limitations. > Speak up if you require this mirror to be kept up-to-date. >=20 > Cheers, > Uli Looks like it might be easier for some downstream projects to fully recreate their ports repositories from scratch than to try to merge from upstream. What caused the issue? What is going to be done to prevent it from happening again? Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQv8tAAoJEGqEZY9SRW7uBLIQAK8uZoZCVaXONcUcN+1zhQN5 4FH2GjlRBb5HEYAKWdLF5W3poLW5N5E+QJWGxLc98HYE8e8pY+ZKuo6t56saBRgn 1z0CmtHOxiy7Ei0oJ5vHiqg+jw/rxlbccM+5Dksz4bgs4LGBu1IlrN/3eaDCN8yt ycgAPpXY+sKC8aXLvziJx+Y958Or1GKcEuYyeVazExCaDMW/ngF9uTVFshvxUyzW UOR6DNgbxRRL6n7OYRREHMSgS4Aw1/H8gx9rMnsHri4dkJYPyWc3gCpS/JR5H665 GW3Jr5dcndfknaHEPIpK1pC0CWVTwYyc/hO9QRt9Wo5TtBBb3p068VlsT0ieUep8 IWhH/LftCtiJetmhGpaLdvW/9RGCtCPMLolhj3GmrBCgJK3D99UDdNnXfX2NILY9 +VqMUP6znon1uhxp+R0GNX6AZDw5z8jRnoDACA5EgnC5Q/XRentSeHvXy4oZn2gR DfDYDJbpSi5jyvruvFVU67NaSs4U1xvdBEMgIUg2wh5u9GvcrgAQDF7S8cPadb/k qOnxZnuyjpfiYDRtunr3NrywVVcDxLGTcxQSiS2QaqmtfnEcGP8hw4VhFVZM4Q5T jPup+enMzMqiXV0QEbG6heHwk01MghhoW+3o3sCiWVK01UNtz5HCug/dO9UDSh9Q fEYmWNjGPj+h9lz6kxMw =6RKr -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5-- From owner-freebsd-git@freebsd.org Sat Dec 3 17:25:33 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA50AC65E46 for ; Sat, 3 Dec 2016 17:25:33 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x242.google.com (mail-lf0-x242.google.com [IPv6:2a00:1450:4010:c07::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30B5225F for ; Sat, 3 Dec 2016 17:25:33 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x242.google.com with SMTP id o20so24837339lfg.3 for ; Sat, 03 Dec 2016 09:25:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=V0yRDGZUkXJzgsxsJTXkUhELCTOtmGIO5T5dvtGZRV4=; b=dzWT0MnVVz9Wk2ixvhCzuAbsqaIpk29K2k9aXqxR4dgQHUjJDFdcZHK8znXtKwnQy2 mdvdMYovkyh4P+bHHmsJaNIp28oh/46KKbJfYM1/rZnhatxRxqtdB5oFPRy/P86YGPIX Zf40XLM5ammZQZes5xYDp+IPwb1fYiZnFW8os4YSPbMShEKsNRw+9l2O4pPR1Qg3gxrc 9HWYKve6MVW58quUmQU/RRxku6oPPsEiP7x3pEebYKnjxDxAORVYLeHn45XE0iUdd3yA A4gk6/Mu+cCsD9eqIC73P6Vdo1G9Rzrab/1UMXWUnTarwycRsKoXJhV1y8YRDl992jkZ h1RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=V0yRDGZUkXJzgsxsJTXkUhELCTOtmGIO5T5dvtGZRV4=; b=ifs1JgyAyyewTqj3C0GQrRzCIc6Npv96b+9zTXzuH4TBxen1HOmdijqTOTUz7jVWt/ 4wKIrTIoPC1IiJD7lkV0YY9kYvOhkkjx2/MUmLOfLPL9oIG9p33SvbqeZfzZyemeEX/y uIEbhvUxRAMbl5lwXAe/OPElDuqJ2+eHcVwHybAzHAzaMtEAX2CSBpJ2QTaZ/pvKLY2M 8ByN62oW96bwm6KFHLUMVcN47+Xq6G3/rnEtXjhXlijzkibMg9GkoQZWiplk8QXZ1tw9 7APOQ4ax9oKBsJhOeNXhD40p6TmBx9Ak0gAdIl4JwLa6lEy43I3J20R04MkW9HpxCr6b zPoQ== X-Gm-Message-State: AKaTC01ptNib4fXZnVzG0xVgDk3bOJlG4bq3qGToG8Xnsq/0hk7paxA/WjKwgWkR2JKZPzDekc4vLCI1dgLpuA== X-Received: by 10.25.75.80 with SMTP id y77mr19240501lfa.173.1480785929828; Sat, 03 Dec 2016 09:25:29 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Sat, 3 Dec 2016 09:25:29 -0800 (PST) In-Reply-To: <20161203172151.GA82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sat, 3 Dec 2016 18:25:29 +0100 X-Google-Sender-Auth: 7kFevo395Klbzmd7SVWjaTxZW0E Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: Shawn Webb Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 17:25:33 -0000 2016-12-03 18:21 GMT+01:00 Shawn Webb : > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed. >> > >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket). >> > >> > I'll update this thread in about 24h. >> > Uli >> >> Service is fully restored now for the github mirror, sorry for the downtime. >> Bitbucket will stop working soon because of the size limitations. >> Speak up if you require this mirror to be kept up-to-date. >> >> Cheers, >> Uli > > Looks like it might be easier for some downstream projects to fully > recreate their ports repositories from scratch than to try to merge from > upstream. What are you referring to here? > What caused the issue? What is going to be done to prevent it from > happening again? I have no root cause, other than bitbucket changing permissions and somehow git ending up using 100% CPU for most of the operations. From owner-freebsd-git@freebsd.org Sat Dec 3 17:29:13 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3C8AC65FC6 for ; Sat, 3 Dec 2016 17:29:13 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt0-x233.google.com (mail-qt0-x233.google.com [IPv6:2607:f8b0:400d:c0d::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 98C3C91B for ; Sat, 3 Dec 2016 17:29:13 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt0-x233.google.com with SMTP id p16so279302243qta.0 for ; Sat, 03 Dec 2016 09:29:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=jzPnCP8wAVR29P9WbgaAxfeUsM0W6H4WckZlCVU/t9A=; b=vGLmmE7TQmpCMqU7gvBXfzZ/mdLLgUzedRkA9sT/fr4QN7Yvyf7ytgYENiDv3Upk/r vRWGdYSAo+oucd+5iZ4tJF+KONGteuIRoIEmncnb8odyrdQ0SfVrwJFkIL5sJQ0KpIxY oUD2c+PLFUXAkC517du+HX+6AEZGC14VT99eZyHqGrw7HSBHu29Dh6CNKKuimf9W7GEd h9thli3/NpfnsKsS+dSOsdqceMCee77SqnJ+Lam3S7NdCwOXm5Oz1ulEDwyLwzQsTNhC JEtUJEhBwbI22zcneqN4twjhvK9tkm/GAGTP3V+s0lD4/285XTXITsdvUFH3cVnFtJp6 pqQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=jzPnCP8wAVR29P9WbgaAxfeUsM0W6H4WckZlCVU/t9A=; b=FD1/KsmjwpY2fli/d9PIf41OhZUW3mfoWUhXiZ/h4GiTIagMHr7BmNl6JtIAOwIatd XylpjgeJfEuoirYmTCCx9wlORmT9oE7RqUORB7Cp+KFj/tYp7wK4cAa2NBIHrFQPLDCO VEzJbX4a+qFX3N7qsUbMbJPIYALSZT64xs1qNf+RxxET4PstKWfnouWPn5WqLr2kHIG/ Dzju7MJuoqPjRQX4BNSxNzVHgSUFWyG+IOTUkniiMF7nPKdA081Ab2cgA8PFuiui9QfI ah5SA4ygkheXb7LWyjfSzcIXVyMr1osRxYS0b2EKvpkC+945RJgFOoGGiDNS/tRpJrXn QXoQ== X-Gm-Message-State: AKaTC012iPLfeUquBgTGCdV7U7zuIf8SphXN4glKS1rmcyibnJM80sgBfV/6I4r0g7D5L+RT X-Received: by 10.200.49.247 with SMTP id i52mr42800885qte.108.1480786152729; Sat, 03 Dec 2016 09:29:12 -0800 (PST) Received: from mutt-hardenedbsd (pool-100-16-218-231.bltmmd.fios.verizon.net. [100.16.218.231]) by smtp.gmail.com with ESMTPSA id 124sm5348055qki.14.2016.12.03.09.29.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 03 Dec 2016 09:29:11 -0800 (PST) Date: Sat, 3 Dec 2016 12:29:10 -0500 From: Shawn Webb To: Ulrich Sp??rlein Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203172910.GB82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="RASg3xLB4tUQ4RcS" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 17:29:14 -0000 --RASg3xLB4tUQ4RcS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: > 2016-12-03 18:21 GMT+01:00 Shawn Webb : > > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : > >> > The conversion process started chewing up 100% cpu without making mu= ch progress, first attempts to rectify this have failed. > >> > > >> > The svn2git conversion and pushes to github have been halted. Pushes= to bitbucket also have been halted (we're reaching the 2GB limit imposed b= y bitbucket). > >> > > >> > I'll update this thread in about 24h. > >> > Uli > >> > >> Service is fully restored now for the github mirror, sorry for the dow= ntime. > >> Bitbucket will stop working soon because of the size limitations. > >> Speak up if you require this mirror to be kept up-to-date. > >> > >> Cheers, > >> Uli > > > > Looks like it might be easier for some downstream projects to fully > > recreate their ports repositories from scratch than to try to merge from > > upstream. >=20 > What are you referring to here? The ports repo at https://github.com/freebsd/freebsd-ports was force pushed. Now attempts at merging in upstream's ports tree into hardenedbsd's causes merge conflicts for hundreds of files, including files we didn't change. So I'm forced to either inspect hundreds of files, manually merging in the changes or recreate our ports tree from scratch, re-importing our changes in a single atomic commit. The second option sounds more appealing, though we'd lose the entire history of our changes. Additionally, anyone downstream from HardenedBSD might have to do the same. Domino affect. >=20 > > What caused the issue? What is going to be done to prevent it from > > happening again? >=20 > I have no root cause, other than bitbucket changing permissions and > somehow git ending up using 100% CPU for most of the operations. So no guarantees this massive screw-up won't happen again? --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --RASg3xLB4tUQ4RcS Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQwDlAAoJEGqEZY9SRW7u2IQQAKYkGsd+OtpOpaYDejfIMHmD HdW8l2OoWqwc/AQdCCwhjDNpLAGqFM1zWp32JVyg47xyL7SVXBosAjMn5zz5Iz13 L7Ysr6HSHFtN6R90GKOmUqNm/kMD+k8CIvD/I8/ziGNSKsxCZL7We6HHs3ecqEtB fa+ryfM9/I4T2/d80rYtaaF0LQQx/r4OIEIzVduUKbbdzRmBGihvdOgwhXTgrp+R s1yyLcQI3GKb44SpTFH+RYQQxhHt8XD3FasV1a0bxvZgmAzHwL/I81koID/D0D6D pQ2fKLyyMFIfoTXcC1MBIm9vmjW4Azg2NoiijapX0JXPovrEdE1nZZHPgCeu0wAG xvMJhaUK//tgJLBHsWbOSFwPxeoWuOLndG+VkN15DjAH0bGng2r1GgbOsVm6VoYS 1pXWktZjvaWMrhzTvADX8S9fG8PFPhkA6bcwBN3oW5GsAS4g/EEMW56NW5V5yB8o tvr3YNYTU3VOJVumkpgPqL2ofxzQh6MVMV06ZfJMjZk+36rojrkaGSwCoGpXEjcy qEJIk2N7Q1XsYjKUI9I+KU4i4d1yK4uhpwRkC9lVw3rNMAKRPNCp98Ycn1dgm2AV kb3u3bHFTaRG+I5ROmQPpC+nNJ0XH/VjsjqwuX6NvX9NlQgb6mE/mym9kjVYvEqM oL3GEviBcJfdzCisA1qf =mTN/ -----END PGP SIGNATURE----- --RASg3xLB4tUQ4RcS-- From owner-freebsd-git@freebsd.org Sat Dec 3 17:45:35 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3213FC65546 for ; Sat, 3 Dec 2016 17:45:35 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x243.google.com (mail-lf0-x243.google.com [IPv6:2a00:1450:4010:c07::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC28A14E8 for ; Sat, 3 Dec 2016 17:45:34 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x243.google.com with SMTP id 98so24762554lfs.0 for ; Sat, 03 Dec 2016 09:45:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=AfQX8QrK8ujMV+rghW2yLr6xBYatT4BNnslE2ZFFdBM=; b=obkIl0KujRFeTcunZ9w/zX5iKJbYEE0Gz9/Ksxrx3DKIKx+oTkACSDdUJBEW9KtorK DuC8e1Nlu8qs/jD73O3UUBXjX9/o8lIZIiap16e1BrCg4RHdV6PHdhV92dQ1A/43PSj3 Zbj7C4u8vEfx3cpROLBhBLjOUPA76J+zSpLUHUSJmADGi/kfO7ICsiAMsBEDt/rB62W3 NtPpuDwEyRnaZXWFWoIqsHgHJRC9yluX2ks3BTbg/Tb8RXsVgicdD2T5DpJFWTnt+G4X Fp7RWt/igo32g5xT+qbtL8XqUtgIYhsiJRnYAwqucxjlA7cMhz9BfN48wp74Jd/YYJDe cv3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=AfQX8QrK8ujMV+rghW2yLr6xBYatT4BNnslE2ZFFdBM=; b=GtWJLmOvyAFzx+cUkAVwmnwf30Bcl0p6ryqRi2P2bw1QdtwR0TjbqW6K3lB7mg+x8i q82SPtSkPacbN1iLrqVZcO5sY7kfSkpWx9eIBNCfKThelO/Ze7mNzouCLW2LWSRRBkaY N41hk2xHhyrPDZ/MzoH8Mes9IOfxZAWpEGZsG6fS7qmbjDqd+8ME9X1IvpxQFZCYMSdp Dn7rWAgDWfj8C3+6CzE4DsbWX3zP+1oaW0xo4T5wiBrdmyD4i9zF0/ppouBieHRA6t8E TgSdqSJZYTTDxWGsWE6t2gqlVGGYCmG6CZnC7zqnDBwRZpjGKcf04/8p2AQCumiVFzFD XKKw== X-Gm-Message-State: AKaTC015NDBBuut6EZGEhZV3eags6G8gVCsc2wffzCep0H6uoWb1osczvpfnGCfLMUudFZnhpMm/wWhPgs/TCQ== X-Received: by 10.25.28.21 with SMTP id c21mr19640376lfc.91.1480787132848; Sat, 03 Dec 2016 09:45:32 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Sat, 3 Dec 2016 09:45:32 -0800 (PST) In-Reply-To: <20161203172910.GB82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> <20161203172910.GB82677@mutt-hardenedbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sat, 3 Dec 2016 18:45:32 +0100 X-Google-Sender-Auth: 29hItCp0oH3MrhgJoY79uma7yb8 Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: Shawn Webb Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 17:45:35 -0000 2016-12-03 18:29 GMT+01:00 Shawn Webb : > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: >> 2016-12-03 18:21 GMT+01:00 Shawn Webb : >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : >> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed. >> >> > >> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket). >> >> > >> >> > I'll update this thread in about 24h. >> >> > Uli >> >> >> >> Service is fully restored now for the github mirror, sorry for the downtime. >> >> Bitbucket will stop working soon because of the size limitations. >> >> Speak up if you require this mirror to be kept up-to-date. >> >> >> >> Cheers, >> >> Uli >> > >> > Looks like it might be easier for some downstream projects to fully >> > recreate their ports repositories from scratch than to try to merge from >> > upstream. >> >> What are you referring to here? > > The ports repo at https://github.com/freebsd/freebsd-ports was force > pushed. Now attempts at merging in upstream's ports tree into > hardenedbsd's causes merge conflicts for hundreds of files, including > files we didn't change. > > So I'm forced to either inspect hundreds of files, manually merging in > the changes or recreate our ports tree from scratch, re-importing our > changes in a single atomic commit. The second option sounds more > appealing, though we'd lose the entire history of our changes. > > Additionally, anyone downstream from HardenedBSD might have to do the > same. Domino affect. I see. This shouldn't have happend, but as svnsync is non-transactional, we picked up some bad SVN metadata that made it into ports and base repos about a year ago. The SVN corruption was promptly fixed (I didn't ask for this), but that now leaves us with no way to actually re-do the conversion from scratch, as you'd need a corrupted SVN repo to produce the same results. You should be able to simply merge whatever "official" commit you last merged to with whatever the new "official" commit is now. This only affected metadata, so you'll get a clean merge (no conflicts) but you end up depending on 2x the history for about a year or so. Shouldn't be that much of a problem. Ask your local git wizard on how to do this best. >> > What caused the issue? What is going to be done to prevent it from >> > happening again? >> >> I have no root cause, other than bitbucket changing permissions and >> somehow git ending up using 100% CPU for most of the operations. > > So no guarantees this massive screw-up won't happen again? I said this before, and I'll say it again. This is a best-effort conversion and we're at the mercy of whatever SVN fucks up next. I provided clear instructions as to how to do the conversion in-house, and guess how many people actually wrote to me that they end up with different SHA hashes on github than they can produce in-house for both src and ports? What would be your guess? Exactly, 0 people have done the in-house conversion and have compared this to github. I could have put all kinds of backdoors in FreeBSD on github and not a single soul would've noticed. So if you depend on it, I would very much appreciate if you could do the same conversion in-house and report any drift as soon as possible, because it's a mess otherwise, as you can see. Any thoughts on how to fix this for src would also be appreciated, all I can think of is either pushing 2 heads and telling people to migrate, or doing the switchover on a flag day. Cheers, Uli From owner-freebsd-git@freebsd.org Sat Dec 3 18:12:37 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5DBAC65C5B for ; Sat, 3 Dec 2016 18:12:37 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 788B4633 for ; Sat, 3 Dec 2016 18:12:37 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x234.google.com with SMTP id x190so309983576qkb.0 for ; Sat, 03 Dec 2016 10:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=JCrsbZLY6OD/8cnnLEAgxMscnEtcJWC3nXDzJvBIJfw=; b=lY6UCjssXVbqvv1fMt4AGQg6iqLFYrEml06v1AaDiE38dN+tWPyvSFLuVBr2ZwPsGp 8TUobRAfW8v5Q3RPO1f/NgvjzoT9Fc6b+mk5F0xKzc4HLXt7z6tNObWwbYYO1Xl97p56 vaQPZH3lblu8RGIRho74hrbGlitHIiubBkGqyJW+pzuipmN282fAggla3LACkkj6yIOJ eaF+hXlywJIea2dvndN/b/5fLGj6H4YZLMcZsRhYHO7BY8t9ZCtcHhr+2nSN6etiPdN4 6XCxfoy5BXcuzTJlS5y28X7RVFcisiYad75QeMKC48sXD/qjwUuh3+jhfqWIQduuj8wy k+pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=JCrsbZLY6OD/8cnnLEAgxMscnEtcJWC3nXDzJvBIJfw=; b=WTzUWrQpeBQ0PfOEBIqbUxwR+zBu0jZw0GrwAcwQnX0lKk8pRi+YPyfCqDAMnoVXQ2 LXpY1TZ25F+cO1GhyN5zJhwYOX82LYL/Vkk77I1Y39Td19YVGlNISnM4FK2lVw26wNvo JXWjePAsGgAPpJUz0uC35m4+BQaEyM/kGfam+p0s7alCu92ZovIufpAsHvfrE2pXY2Cm NxdhPFdsUqzs0Cu4PbBGwpGjRBcB0u6JB0Tw/7fBw1YPX+VBoRBSn95Gltjfo5wasUUB 0+8Ul3tv0k59aRmFQnqSr/3HOWykOiiXFvYlsNUCbRF70SRLOV40wuzVxKHr4OBsp/Bg dQgA== X-Gm-Message-State: AKaTC00OOloaspRzVc5iRD7jrqDtgAh8pVUN32Y8DHzEC8YHU1pqLNYWt8ovzwNDnTyeal2T X-Received: by 10.55.78.147 with SMTP id c141mr49283118qkb.4.1480788756429; Sat, 03 Dec 2016 10:12:36 -0800 (PST) Received: from mutt-hardenedbsd (pool-100-16-218-231.bltmmd.fios.verizon.net. [100.16.218.231]) by smtp.gmail.com with ESMTPSA id y23sm5395417qta.40.2016.12.03.10.12.35 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 03 Dec 2016 10:12:35 -0800 (PST) Date: Sat, 3 Dec 2016 13:12:33 -0500 From: Shawn Webb To: Ulrich Sp??rlein Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203181233.GC82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> <20161203172910.GB82677@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="pAwQNkOnpTn9IO2O" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 18:12:37 -0000 --pAwQNkOnpTn9IO2O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: > 2016-12-03 18:29 GMT+01:00 Shawn Webb : > > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: > >> 2016-12-03 18:21 GMT+01:00 Shawn Webb : > >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : > >> >> > The conversion process started chewing up 100% cpu without making= much progress, first attempts to rectify this have failed. > >> >> > > >> >> > The svn2git conversion and pushes to github have been halted. Pus= hes to bitbucket also have been halted (we're reaching the 2GB limit impose= d by bitbucket). > >> >> > > >> >> > I'll update this thread in about 24h. > >> >> > Uli > >> >> > >> >> Service is fully restored now for the github mirror, sorry for the = downtime. > >> >> Bitbucket will stop working soon because of the size limitations. > >> >> Speak up if you require this mirror to be kept up-to-date. > >> >> > >> >> Cheers, > >> >> Uli > >> > > >> > Looks like it might be easier for some downstream projects to fully > >> > recreate their ports repositories from scratch than to try to merge = =66rom > >> > upstream. > >> > >> What are you referring to here? > > > > The ports repo at https://github.com/freebsd/freebsd-ports was force > > pushed. Now attempts at merging in upstream's ports tree into > > hardenedbsd's causes merge conflicts for hundreds of files, including > > files we didn't change. > > > > So I'm forced to either inspect hundreds of files, manually merging in > > the changes or recreate our ports tree from scratch, re-importing our > > changes in a single atomic commit. The second option sounds more > > appealing, though we'd lose the entire history of our changes. > > > > Additionally, anyone downstream from HardenedBSD might have to do the > > same. Domino affect. >=20 > I see. This shouldn't have happend, but as svnsync is > non-transactional, we picked up some bad SVN metadata that made it > into ports and base repos about a year ago. The SVN corruption was > promptly fixed (I didn't ask for this), but that now leaves us with no > way to actually re-do the conversion from scratch, as you'd need a > corrupted SVN repo to produce the same results. >=20 > You should be able to simply merge whatever "official" commit you last > merged to with whatever the new "official" commit is now. This only > affected metadata, so you'll get a clean merge (no conflicts) but you > end up depending on 2x the history for about a year or so. Shouldn't > be that much of a problem. Ask your local git wizard on how to do this > best. >=20 > >> > What caused the issue? What is going to be done to prevent it from > >> > happening again? > >> > >> I have no root cause, other than bitbucket changing permissions and > >> somehow git ending up using 100% CPU for most of the operations. > > > > So no guarantees this massive screw-up won't happen again? >=20 > I said this before, and I'll say it again. This is a best-effort > conversion and we're at the mercy of whatever SVN fucks up next. I > provided clear instructions as to how to do the conversion in-house, > and guess how many people actually wrote to me that they end up with > different SHA hashes on github than they can produce in-house for both > src and ports? >=20 > What would be your guess? >=20 > Exactly, 0 people have done the in-house conversion and have compared > this to github. I could have put all kinds of backdoors in FreeBSD on > github and not a single soul would've noticed. >=20 > So if you depend on it, I would very much appreciate if you could do > the same conversion in-house and report any drift as soon as possible, > because it's a mess otherwise, as you can see. >=20 > Any thoughts on how to fix this for src would also be appreciated, all > I can think of is either pushing 2 heads and telling people to > migrate, or doing the switchover on a flag day. Hey Uli, Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair of me to use that tone. I'm grateful for your efforts. I understand that supporting git isn't an official service provided by FreeBSD. One item on my Christmas wishlst would be to have official support for a read-only git mirror of the various FreeBSD projects (mainly src and ports). Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --pAwQNkOnpTn9IO2O Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQwsPAAoJEGqEZY9SRW7uIEYQAMXxWAhapwBqnIi2VCxt8nav cbcwIzIqZy9bbk9BQ5Va9NSFUZNsM89vZe2cq9Surdzb+ydxaYkQLuEXXhIkaNYc DDHfJZ//SyeEj7zFm0jDfLBv8OyrZzmNPSlWTcBJGr8mPK8nny+Yrj850HObnH7v w/cbGI5X5zYQr5JmtU+MNXcaWVmoSVLjmwxj84mnzcrzSX4mNH7vKRpcGB2I/rfa bPOhe9mm9ap2mYr6vckOxahrWRSykLkFHOeMiERQO6XOPeGnvZvdX8mENowVkX6U W3VGRyCyl9zlkd+y6YVWymYWJOcNakY1/eZgFmTOqVoMSA27xZvE7g7qwfXn2cSo lpgXUTnBbdYykpY5+cEuzTTnXk5RdO7fDUqhGmMpTscM/VkxpP9RL+/HNSm2KvYX bcuZF05CvqNIen85Bwhxi0pjrgCLqKnzsF4eIiXhaqhueqOUlJO4U7USLeM+YVw/ 5MWaWS/ZAiQlI3/Xw56h9ymRtiGA6A64NTkaF9w/d9M86vPwMX8KtXdQwCD/TYP5 koiUBWbHBRWPTZz8EI6/hwvWut7f1CUMmyTgNpHze5D/PIPuHdTGh79W57rUuDsg 4CY5BXKnOAqfhI79V9JDzLel39Kl/UqIt76TKtuw/wc1zooHBQMLbx0fITYlyaXP cRCLpEbBMewS2JRFjldr =dfpg -----END PGP SIGNATURE----- --pAwQNkOnpTn9IO2O-- From owner-freebsd-git@freebsd.org Sat Dec 3 18:16:11 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 763D7C65D33 for ; Sat, 3 Dec 2016 18:16:11 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-lf0-x244.google.com (mail-lf0-x244.google.com [IPv6:2a00:1450:4010:c07::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE97E842 for ; Sat, 3 Dec 2016 18:16:10 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-lf0-x244.google.com with SMTP id o141so24974674lff.1 for ; Sat, 03 Dec 2016 10:16:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=/JuyaC0pFM27BemYGlwg+1vWmGnRZ49VRHeXmK65pY8=; b=AOZ4ESiFotEIdlc4ik2ufbkSc/oXs+WuSL453qED/d2e/1/sABEU23+5PwaQLPGx5U 7KHTF9BEOFkhDmx3mL61KEQ0JScKc0O3wU4K2u0NES2eNcck2xeAvhaF/kZhI1/rKoIX YzQVgzhhvS4msnJqw6p8tBHGrFwxiBu8n2AsyPRkIJTjLeKvArMI9sK6kfgz/dhAiZWs MvgOQX7ea6fsQ8RBb4JjJ8MoWhtkZK83PCW/cYzc8b8bjXxr3na6zX6lJkyPh8EIQKOc TGAsniXuiKEcFLq1xmakMHFP3CiJlMuob/taZpORoYiofpSmBfYjVt7oZuK/zSEiUP1L qm+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=/JuyaC0pFM27BemYGlwg+1vWmGnRZ49VRHeXmK65pY8=; b=fX1nJdmIkWZPkrhkYdY4z0la+7MTE4g8i2eZH//XKi8M8Ioex36CsYV7EBjpWTwr+P 4lvHN189GcJGge0OZMp6x+Fzg44vNhB23OV0Ozoy/a6FPGsCIXLvn9rlhtYsOCENxULL QzVXpAEjkLLVxlHhNL964Ak9cJ4CWh42ufCBqfW7KH6CvaVq9fcG6X7nVKqd9Igo1ri6 v7uussFy8Y/CXmhRisZMdDojotbGE5ORhDf52Klao067ajjqDUinZJzGcuikXrnYp8jp Y/CZNqTV6vl6wEZVr8Z8VhBvWjz93a3mDo7W9hzhJSAeQN9DZbmKNqgS/0D4Ga0YJ9Cg t10g== X-Gm-Message-State: AKaTC00HY0DhJI9rHX6FqmraFTCPNWHYHqSQ4frgxSffcaXNNwonZW06/8ruz975USi1tWtU2DNqGjOXDXgEzA== X-Received: by 10.46.0.229 with SMTP id e98mr25610569lji.33.1480788968816; Sat, 03 Dec 2016 10:16:08 -0800 (PST) MIME-Version: 1.0 Sender: uspoerlein@gmail.com Received: by 10.25.196.140 with HTTP; Sat, 3 Dec 2016 10:16:08 -0800 (PST) In-Reply-To: <20161203181233.GC82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> <20161203172910.GB82677@mutt-hardenedbsd> <20161203181233.GC82677@mutt-hardenedbsd> From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Sat, 3 Dec 2016 19:16:08 +0100 X-Google-Sender-Auth: GYnRR1Tu-n__0yXjrsPs-G1Otrw Message-ID: Subject: Re: Git mirroring halted for freebsd-base To: Shawn Webb Cc: freebsd-git@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 18:16:11 -0000 2016-12-03 19:12 GMT+01:00 Shawn Webb : > On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: >> 2016-12-03 18:29 GMT+01:00 Shawn Webb : >> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: >> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb : >> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: >> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : >> >> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed. >> >> >> > >> >> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket). >> >> >> > >> >> >> > I'll update this thread in about 24h. >> >> >> > Uli >> >> >> >> >> >> Service is fully restored now for the github mirror, sorry for the downtime. >> >> >> Bitbucket will stop working soon because of the size limitations. >> >> >> Speak up if you require this mirror to be kept up-to-date. >> >> >> >> >> >> Cheers, >> >> >> Uli >> >> > >> >> > Looks like it might be easier for some downstream projects to fully >> >> > recreate their ports repositories from scratch than to try to merge from >> >> > upstream. >> >> >> >> What are you referring to here? >> > >> > The ports repo at https://github.com/freebsd/freebsd-ports was force >> > pushed. Now attempts at merging in upstream's ports tree into >> > hardenedbsd's causes merge conflicts for hundreds of files, including >> > files we didn't change. >> > >> > So I'm forced to either inspect hundreds of files, manually merging in >> > the changes or recreate our ports tree from scratch, re-importing our >> > changes in a single atomic commit. The second option sounds more >> > appealing, though we'd lose the entire history of our changes. >> > >> > Additionally, anyone downstream from HardenedBSD might have to do the >> > same. Domino affect. >> >> I see. This shouldn't have happend, but as svnsync is >> non-transactional, we picked up some bad SVN metadata that made it >> into ports and base repos about a year ago. The SVN corruption was >> promptly fixed (I didn't ask for this), but that now leaves us with no >> way to actually re-do the conversion from scratch, as you'd need a >> corrupted SVN repo to produce the same results. >> >> You should be able to simply merge whatever "official" commit you last >> merged to with whatever the new "official" commit is now. This only >> affected metadata, so you'll get a clean merge (no conflicts) but you >> end up depending on 2x the history for about a year or so. Shouldn't >> be that much of a problem. Ask your local git wizard on how to do this >> best. >> >> >> > What caused the issue? What is going to be done to prevent it from >> >> > happening again? >> >> >> >> I have no root cause, other than bitbucket changing permissions and >> >> somehow git ending up using 100% CPU for most of the operations. >> > >> > So no guarantees this massive screw-up won't happen again? >> >> I said this before, and I'll say it again. This is a best-effort >> conversion and we're at the mercy of whatever SVN fucks up next. I >> provided clear instructions as to how to do the conversion in-house, >> and guess how many people actually wrote to me that they end up with >> different SHA hashes on github than they can produce in-house for both >> src and ports? >> >> What would be your guess? >> >> Exactly, 0 people have done the in-house conversion and have compared >> this to github. I could have put all kinds of backdoors in FreeBSD on >> github and not a single soul would've noticed. >> >> So if you depend on it, I would very much appreciate if you could do >> the same conversion in-house and report any drift as soon as possible, >> because it's a mess otherwise, as you can see. >> >> Any thoughts on how to fix this for src would also be appreciated, all >> I can think of is either pushing 2 heads and telling people to >> migrate, or doing the switchover on a flag day. > > Hey Uli, > > Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair > of me to use that tone. > > I'm grateful for your efforts. I understand that supporting git isn't an > official service provided by FreeBSD. One item on my Christmas wishlst > would be to have official support for a read-only git mirror of the > various FreeBSD projects (mainly src and ports). > No worries, I feel your pain. My Friday evening was ruined too :( If you have the resources, *please* try a conversion run of all three repos too and report your findings. We need to make sure that the conversion is now more robust and we need to detect drift earlier :/ Cheers, Uli From owner-freebsd-git@freebsd.org Sat Dec 3 18:40:57 2016 Return-Path: Delivered-To: freebsd-git@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E9BBC649E0 for ; Sat, 3 Dec 2016 18:40:57 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x232.google.com (mail-qk0-x232.google.com [IPv6:2607:f8b0:400d:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F4561AA0 for ; Sat, 3 Dec 2016 18:40:57 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x232.google.com with SMTP id n21so310362611qka.3 for ; Sat, 03 Dec 2016 10:40:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=dcM4TMHySflsA8LkmqZkgrtj5ULbvufYr2fbEc2zYKs=; b=EZBlIWeNwpAkX3l6Jn0i68OVyfOjRmvwwib9+CEO6GrKioAkGgcvTd8DVbXEBSi4YL T/gdVt4wZmVdRSmmlyhKlcJpSyc3TbYJTPYAv7n/R3/2H7RByZgIry7aGxZXhNSoxofA 4IoFbrMl8ATgiUm0P2sQCtiCoEdBi6vBse6m8K3/pLN0zm+7Cw0lYQSJxzkv3IuoGrZl y7geQUZ6C3VHnQcPR7iASeT+dHGL4hS9fZ5Om9uzDs1HHPKTNv6e4gRawTRTHelzh4ok dkmTPfLXpAcCZZ9s/6484zESJeVpcP/JoNC+/uzHXAVOn5q14jn2fCnvkQeaGgUZc1YU txbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=dcM4TMHySflsA8LkmqZkgrtj5ULbvufYr2fbEc2zYKs=; b=Nz6yinfouMgg8HzMUBXq/vmZZoCWuEVKj/giJcKXPfjihOiR8ddYn6qdn2I6I1K+nb oxwOkl62x+Hi0ZgkR9dUdb2zy7OAvxrEWUJnl6gGrTkvuikpnOOoXWskxlAZhuX7aKEz LIat4g51Fwm6wJ3Snqa6MVRJ/ztUorMtZYx4UIUPQf59SfGecodJxghMFc5A+x5ZIvGx 9eT0bP/rFzym+zLP9K54b7k5Ccht7MRCzRpdqqF8coNaGZjKE9eE3jOkSvQuH6TQ8QU8 2lj79DSGEHXBDim50WMmT8XfCbN8+njdnHQMN3RfsLEsQmt7eDgodlZja+JDwGWPAwfm b+aQ== X-Gm-Message-State: AKaTC02SZ0UW07m9i2T+ITtuqBIxEVkBhVpFtFBTgxaNpiWGBnnlDmHBohQ/7OYD/shght14 X-Received: by 10.55.162.83 with SMTP id l80mr42876784qke.168.1480790456247; Sat, 03 Dec 2016 10:40:56 -0800 (PST) Received: from mutt-hardenedbsd (pool-100-16-218-231.bltmmd.fios.verizon.net. [100.16.218.231]) by smtp.gmail.com with ESMTPSA id y16sm5429987qka.32.2016.12.03.10.40.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 03 Dec 2016 10:40:54 -0800 (PST) Date: Sat, 3 Dec 2016 13:40:53 -0500 From: Shawn Webb To: Ulrich Sp??rlein Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203184053.GE82677@mutt-hardenedbsd> References: <20161203172151.GA82677@mutt-hardenedbsd> <20161203172910.GB82677@mutt-hardenedbsd> <20161203181233.GC82677@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="3O1VwFp74L81IIeR" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 18:40:57 -0000 --3O1VwFp74L81IIeR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 07:16:08PM +0100, Ulrich Sp??rlein wrote: > 2016-12-03 19:12 GMT+01:00 Shawn Webb : > > On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: > >> 2016-12-03 18:29 GMT+01:00 Shawn Webb : > >> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: > >> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb : > >> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > >> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein : > >> >> >> > The conversion process started chewing up 100% cpu without mak= ing much progress, first attempts to rectify this have failed. > >> >> >> > > >> >> >> > The svn2git conversion and pushes to github have been halted. = Pushes to bitbucket also have been halted (we're reaching the 2GB limit imp= osed by bitbucket). > >> >> >> > > >> >> >> > I'll update this thread in about 24h. > >> >> >> > Uli > >> >> >> > >> >> >> Service is fully restored now for the github mirror, sorry for t= he downtime. > >> >> >> Bitbucket will stop working soon because of the size limitations. > >> >> >> Speak up if you require this mirror to be kept up-to-date. > >> >> >> > >> >> >> Cheers, > >> >> >> Uli > >> >> > > >> >> > Looks like it might be easier for some downstream projects to ful= ly > >> >> > recreate their ports repositories from scratch than to try to mer= ge from > >> >> > upstream. > >> >> > >> >> What are you referring to here? > >> > > >> > The ports repo at https://github.com/freebsd/freebsd-ports was force > >> > pushed. Now attempts at merging in upstream's ports tree into > >> > hardenedbsd's causes merge conflicts for hundreds of files, including > >> > files we didn't change. > >> > > >> > So I'm forced to either inspect hundreds of files, manually merging = in > >> > the changes or recreate our ports tree from scratch, re-importing our > >> > changes in a single atomic commit. The second option sounds more > >> > appealing, though we'd lose the entire history of our changes. > >> > > >> > Additionally, anyone downstream from HardenedBSD might have to do the > >> > same. Domino affect. > >> > >> I see. This shouldn't have happend, but as svnsync is > >> non-transactional, we picked up some bad SVN metadata that made it > >> into ports and base repos about a year ago. The SVN corruption was > >> promptly fixed (I didn't ask for this), but that now leaves us with no > >> way to actually re-do the conversion from scratch, as you'd need a > >> corrupted SVN repo to produce the same results. > >> > >> You should be able to simply merge whatever "official" commit you last > >> merged to with whatever the new "official" commit is now. This only > >> affected metadata, so you'll get a clean merge (no conflicts) but you > >> end up depending on 2x the history for about a year or so. Shouldn't > >> be that much of a problem. Ask your local git wizard on how to do this > >> best. > >> > >> >> > What caused the issue? What is going to be done to prevent it from > >> >> > happening again? > >> >> > >> >> I have no root cause, other than bitbucket changing permissions and > >> >> somehow git ending up using 100% CPU for most of the operations. > >> > > >> > So no guarantees this massive screw-up won't happen again? > >> > >> I said this before, and I'll say it again. This is a best-effort > >> conversion and we're at the mercy of whatever SVN fucks up next. I > >> provided clear instructions as to how to do the conversion in-house, > >> and guess how many people actually wrote to me that they end up with > >> different SHA hashes on github than they can produce in-house for both > >> src and ports? > >> > >> What would be your guess? > >> > >> Exactly, 0 people have done the in-house conversion and have compared > >> this to github. I could have put all kinds of backdoors in FreeBSD on > >> github and not a single soul would've noticed. > >> > >> So if you depend on it, I would very much appreciate if you could do > >> the same conversion in-house and report any drift as soon as possible, > >> because it's a mess otherwise, as you can see. > >> > >> Any thoughts on how to fix this for src would also be appreciated, all > >> I can think of is either pushing 2 heads and telling people to > >> migrate, or doing the switchover on a flag day. > > > > Hey Uli, > > > > Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair > > of me to use that tone. > > > > I'm grateful for your efforts. I understand that supporting git isn't an > > official service provided by FreeBSD. One item on my Christmas wishlst > > would be to have official support for a read-only git mirror of the > > various FreeBSD projects (mainly src and ports). > > >=20 > No worries, I feel your pain. My Friday evening was ruined too :( >=20 > If you have the resources, *please* try a conversion run of all three > repos too and report your findings. We need to make sure that the > conversion is now more robust and we need to detect drift earlier :/ I wonder if there's a way for the script to determine whether history has been rewritten, causing a force push. If so, it could bail out early, sending an email to interested parties. The email would just say something to the effect of "uh oh, someone needs to pay extra special attention to me!" Then someone could inspect what's going on, determine if a force push is absolutely necessary, and if so, alert the community prior to the force push. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --3O1VwFp74L81IIeR Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQxGyAAoJEGqEZY9SRW7uDK0QAJrCZa9ZHoRNpZub1XhsKeFM YJfcGLazU7HPAIiUKqIwGfMataBOm7TpeEpUbXrJRORH2G2tZpEQC76kZ/UooF98 lqN5W/i4+Fsx+jtbRK0FsqNoNvimoOoqTzj9KgP1Ll+BElM7+zupqLjfFBEVoIAJ 1++IpTUeHxkrIBna3c8/MLl+dBAqBpRT4yFo0ZOVBjnfhgpt+YTIknD7T01BE7+m NPQeIXogcNiBX9qeZ8BGJFaHN3MI59yyiOOMaCWYBslEDCgVuyVIlc5JaUYl+gby Gfq/qBbEFcELSEa9qPmYSinXGD1cJuyx4b2UqufMZ/yjti6ZRBe4Y0izy2cAx5Sz 7vvmPbNonfBSYIffC9+YkLHdUwmU+UnwR63BMq+/kw+3+Xj7yMf7IOdWM2tpeXFa H9oh4Cd2gdXzFAVTHU6Sue6//nybUytrn4K3cruWeG76d8a4b7/QqSC1LcGqUJPa jOOXK6kEKekTxTHdShf4LIXE1BGZxmnDYIzU2zcZc4UkehKfeHFgKHb4M2gVttjd kDxT1eMiLZBx/mWkXK+z12viHuDdIOIZ+3S4j2uavdSlM27MfV1tn7j6YRRrC4ef rwSuiqfAsJmhvBfKbNtRijMEF6iVwtLYlBlZRIEqgFkTAFRXi8VjyBLft7rPdhoB x1DveGxzFdhrbLiWvVWl =zsC2 -----END PGP SIGNATURE----- --3O1VwFp74L81IIeR--