From owner-freebsd-hubs@FreeBSD.ORG Sun Jul 21 22:08:37 2013 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C8B6187F for ; Sun, 21 Jul 2013 22:08:37 +0000 (UTC) (envelope-from peter@wemm.org) Received: from mail-vc0-x233.google.com (mail-vc0-x233.google.com [IPv6:2607:f8b0:400c:c03::233]) by mx1.freebsd.org (Postfix) with ESMTP id 8C0156C5 for ; Sun, 21 Jul 2013 22:08:37 +0000 (UTC) Received: by mail-vc0-f179.google.com with SMTP id id13so893612vcb.38 for ; Sun, 21 Jul 2013 15:08:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wemm.org; s=google; h=mime-version:date:message-id:subject:from:to:content-type; bh=yXeHHsn1wfY8L5lg+egfBSWhOV889bRcW2QtsYgEjRc=; b=b5eKfOWX2kp5jFNeZgDCabtI2m/S2Se2UVGw6Fj+M/kd5ukD9DNCmgtjiM8N5KdvIo A/R9UMfShkfZWTpBoDbv0t3QJ8/mHHO2Y/MNp6+Co77ZEXAnMCcb4me7TpLexG5d5Uw6 6qdPF2eIAtAesIw7ge8D4iQj4WKFew938+hG0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :x-gm-message-state; bh=yXeHHsn1wfY8L5lg+egfBSWhOV889bRcW2QtsYgEjRc=; b=lTNudipOxQ4A3QFgG3RAjB1JPF1MM/42DjN7ud5wjN6TktMhW0SoPJr0LZ/le06dMC EzWAKPmQvv0tZbNtyDhPtK2Mvyy/azwVxjGsJ5nJGjtl9965cMANXHwrzWD8hZYh8ppj UvrKX1ToukHW1qq7szbXfwuH9HyZSTz5R4/P5IozdfaCcqNLzs9FQM5mgIExU52PbyjA s2eCMt3VnrYVU1MXqgvbGJSolSlzGjBMNd7EUj70Vw97y21eyf+Vx9RP6XlXx/y7vXTn 0CKbwpbVrF1lOOzhQ/DAvIUfi9gPPsj17B4LFR85xCT7Tfy7FpgxYjPWs8ABn5D+asXN Vv9A== MIME-Version: 1.0 X-Received: by 10.58.2.137 with SMTP id 9mr8665879veu.50.1374444516992; Sun, 21 Jul 2013 15:08:36 -0700 (PDT) Received: by 10.221.37.198 with HTTP; Sun, 21 Jul 2013 15:08:36 -0700 (PDT) Date: Sun, 21 Jul 2013 15:08:36 -0700 Message-ID: Subject: FYI: Upcoming cvsup changes From: Peter Wemm To: hubs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQljRuGpIJLyIZ85zlFisMAYHDZwd10BqCOpKEu/pto4+mvAWHro+YANm+hakT5dqCuSKwx5 X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 22:08:37 -0000 Right now, we have a rather fragile cvsup setup that's a bit painful to deal with. cvsup-master depends on a scan file for efficiency. So we have to double-cvsup internally to generate the scan files. It looks like this: www -> via cvsup(rsync mode) -> cvsup-in -> scan file -> cvsup-master -> mirrors www -> via rsync -> internal rsync mirror -> (not published) cvs -> via cvsup(cvs repo mode) -> cvsup-in -> scan file -> cvsup-master -> mirrors mail-> via cvsup(rsync mode) -> cvsup-in -> scan file -> cvsup-master -> mirrors mail -> via rsync -> internal rsync mirror -> (not published) gnats -> not in cvsup anymore, but it was like www/mail. gnats -> via rsync -> internal rsync -> semi-published on bit0 portsindex -> via rsync -> internal rsync -> semi-published on bit0 This means we have cvsupd on many more systems than we'd like. I'm planning to change it like this: www -> via rsync -> internal rsync mirror -> published mail -> via rsync -> internal rsync mirror -> published gnats -> via rsync -> internal rsync -> published portsindex -> via rsync -> internal rsync -> published and on a backend worker in the cluster: www -> rsync -> stage area mail -> rsync -> stage area gnats -> rsync -> stage area cvs(local file system) -> stage area then: stage area -> cvsupd (loopback) -> create scan file -> publish as cvsup-master. This should be mostly transparent for cvsup mirrors. As of today there is still some data that can only be obtained via cvsup and this change will finally fix it. After the change, cvsup will be 100% second-hand and/or converted data. cvsup has had a very good run, but this will likely be the final step in completing the deprecation of the system. Note that I did NOT say "turning off" - there is a fair bit of effort going into keeping it alive. This work is for sustaining it so it can wind down gracefully. But this is probably the last major chunk of work being spent on it at our end. (FYI, remember CTM? It's still going due to a madman^H^H^H^H^H^Hvolunteer. Said in the nicest possible way of course :) cvsup-master will likely be changing IP addresses but we'll reduce the dns TTL's to keep this as transparent as possible. Naming for the rsync infrastructure is internal right now but that'll be sorted out as part of this and documented. cvs tree status: 6.x (and earlier) hasn't been exported to cvs for quite a while. 7.x hasn't been exported to cvs for a few months. 8.x is still being exported once an hour. (including security support branches) 9.x is still being exported once an hour. (including security support branches) 10.x is not exported from svn to cvs. ports has not been exported since February. doc/www was never exported from svn to cvs. BTW; Some mirrors have been rapidly updating multiple times per hour. Please don't. Once an hour is more than enough - there will never be more than one cvsup-master update per hour. -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com; KI6FJV UTF-8: So you can \342\200\231 .. for when a ' just won't do ZFS must be the bacon of file systems. "everything's better with ZFS" From owner-freebsd-hubs@FreeBSD.ORG Sun Jul 21 23:00:52 2013 Return-Path: Delivered-To: hubs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6D31590F for ; Sun, 21 Jul 2013 23:00:52 +0000 (UTC) (envelope-from john.marshall@riverwillow.com.au) Received: from mail4.riverwillow.net.au (mail4.riverwillow.net.au [202.125.45.59]) by mx1.freebsd.org (Postfix) with ESMTP id 0331D89A for ; Sun, 21 Jul 2013 23:00:51 +0000 (UTC) Received: from [172.25.24.201] (riverw1.lnk.telstra.net [165.228.239.138]) (authenticated bits=0) by mail4.riverwillow.net.au (8.14.7/8.14.7) with ESMTP id r6LN0bPu052392 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT); Mon, 22 Jul 2013 09:00:38 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=riverwillow.com.au; s=m4001; t=1374447639; bh=ELYDBN4R4VNmIFfA+u0DOhrvI29jWFfKgZD+vGwfH9A=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=R26n+wH7LmJOfaMe3HZA+zw/rbHdSsqWQxVf3rWLgXJycM/j8F5TTAGKBhoIvHT+4 2CeYlIX758XXs1zCS/vHrkRCMktmzNetUJnA/+1IA3x/HhWDaJO9OJDmGhfXz6n4Fp BBKgQxBmwP9HlyNI1fM6bYmEtoN8cOPZZsp514HE= Message-ID: <51EC680C.10305@riverwillow.com.au> Date: Mon, 22 Jul 2013 09:00:28 +1000 From: John Marshall Organization: Riverwillow Pty Ltd User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: Peter Wemm Subject: Re: FYI: Upcoming cvsup changes References: In-Reply-To: X-Enigmail-Version: 1.5.1 OpenPGP: id=A29A84A2; url=http://pki.riverwillow.com.au/pgp/johnmarshall.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2JEKOGGUEUCSGHJWFSWGU" Cc: hubs@freebsd.org X-BeenThere: freebsd-hubs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "FreeBSD Distributions Hubs: mail sup ftp" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 23:00:52 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2JEKOGGUEUCSGHJWFSWGU Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 22/07/2013 08:08, Peter Wemm wrote: > Right now, we have a rather fragile cvsup setup that's a bit painful > to deal with. >=20 > cvsup-master depends on a scan file for efficiency. So we have to > double-cvsup internally to generate the scan files. >=20 > It looks like this: Thanks a lot for this Peter. It's really helpful to know what's going on and what to expect. I really appreciate you taking the time to sketch out all of this for CVSup mirror operators. I think that what you are doing sounds eminently sensible. Hope all goes well in implementing your proposed changes. --=20 John Marshall ------enig2JEKOGGUEUCSGHJWFSWGU Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iEYEARECAAYFAlHsaBUACgkQw/tAaKKahKK1QQCgi6AGAb9rqaq7e/TWuHTPcn/r w80AoI3lc3cTupyS7UP/MeFTDzhUfnW9 =3KDP -----END PGP SIGNATURE----- ------enig2JEKOGGUEUCSGHJWFSWGU--