Date: Fri, 13 Jun 2003 21:43:58 -0400 From: Ken Smith <kensmith@cse.Buffalo.EDU> To: freebsd-hubs@freebsd.org Subject: DRAFT - Release Guide Message-ID: <20030614014358.GA29321@electra.cse.Buffalo.EDU>
next in thread | raw e-mail | index | archive | help
Here is a draft of a Release Guide which would take most of the hassles out of Release Day I *think*. I want to say up front that if you read this it sounds like it's a list of demands being made by the Mirror Folks to the Release folks but that is NOT its intention. I wrote it in the context of it being exactly what I called it - a Release Guide to be read/followed while doing a new release. But the assumption is this will initially be presented to all parties for *discussion*, and the end result is concensus amongst everyone involved. I also want to say we all know there is way more stuff involved in a Release than just this stuff. I hope this is at least in general terms what people were expecting. If you have minor edits (my spelin wuz rong ur me grammer wuznt quit rite) could you just send it straight to me please? If you have any thoughts that you think should be discussed, or if there are major omissions, or if you think this whole thing is just one big mistake feel free to send to the list but again please don't just include this entire message in the reply - edit out anything that is irrelevant please. I'll wear asbestos undies for the next couple of days, flame away. If in the end we hash this thing out and decide that it's worth doing something with I'll take it to re@ and portmgr@ since it effects how both of them operate, stressing at first that it's for *discussion*. ---------------------------------------- Release Guide Rev. 0.0 ====================== Introduction ------------ In the early years FreeBSD Distributions were, compared to now, relatively simple to handle. The data got copied to ftp.freebsd.org and everyone got it from there. Due to the poplarity of FreeBSD that simple a distribution channel is no longer practical. The mirror system that now supports the distribution of FreeBSD has become a multi-tiered worldwide set of machines, and even in this state some of the more central distribution machines are under fairly heavy stress during Release Time. Because the Mirror System has needed to grow this large and because of certain aspects of the data in the FTP site there are limitations on what can be expected of the Mirror System. In particular there is now a significant amount of time between when data gets placed on the main distribution server (ftp-master.freebsd.org) and when it can be expected to be available from the majority of the mirror servers. Since it is highly desirable for the release files to be on the majority of the mirror servers at the point the Release is announced this effects the schedule for when the data should be placed on ftp-master. Terminology ----------- ftp-master: the master distribution server, data is initially loaded here Tier-1 server: official FreeBSD mirror site that pulls data from ftp-master Tier-2 server: official FreeBSD mirror site that pulls data from a Tier-1 server. Not all Tier-1 servers feed Tier-2 servers (we would like to set up more that do). sync: what happens when a mirror server pulls new data from its up-stream site. Different mechanisms can be used to do this (cvsup, rsync, mirror, omi, etc). Only newly updated files will be transferred through the network BUT typically a large number of files need to be checked to see if they should be transferred. Depending on how much of the FreeBSD FTP site is being checked (only checking sub-sections is possible) this could cause a significant load on both the client and server involved. staging the release: placing new files onto ftp-master but the permissions on the files are set such that anonymous users can not download the files. This would be done so that there is time for the files to be transferred to other mirror sites before a Release Date but users can't get the files until the permissions get changed. Under normal circumstances files are simply "posted" to ftp-master and downloadable by anyone once they get to the Mirror Servers. Mirror System Operations ------------------------ Most though not all of the mirror sites operate the same way. Some time in the wee hours of the morning a cron job starts up what syncs the FTP site. FreeBSD may be the only thing this mirror site carries, or it may carry other distributions (OpenBSD, Linux's, etc). The sync may be set up to check through the entire FreeBSD site but it may just check through some of it, doing other parts some other time (in particular the ports section is getting a bit large and may be handled as a separate thing - perhaps only a few days a week). Because of this it is up to a day before a Tier-1 system will begin transferring new data from ftp-master. If there is a lot of new data, and in particular a new set of packages, the sync may take as long as six to eight hours between systems that are relatively unloaded and have a decent network connection between them. So taking worst case scenarios into account it can be three days before large amounts of data can propagate from ftp-master to a Tier-2 site. This situation gets worse if there is something causing a larger than normal network load at the time. Handling Betas/Release Candidates --------------------------------- Users' expectations and general anxieties are smaller for the Betas and Release Candidates so timing of these is a little less critical. They seem to "naturally" appear a week or more between Beta's and RC's which is reasonable as far as propagation goes. The important thing for the Release Engineering Team to keep in mind is the propagation delay of two to three days for large amounts of data to make it through to the majority of Tier-2 sites. What is critical for the Mirror Sites is that there be a day or two worth of advanced notice (weekdays) that the Beta is coming AND a reasonable estimate of how much disk space will be needed. The Mirror sites that carry many distributions tend to keep the partition(s) dedicated to mirrored data fairly close to being full (maximizing how much potentially useful stuff is available to their users). If large amounts of new stuff is about to arrive some of the older stuff needs to be removed. Other sites can't afford the disk space to keep the entire FreeBSD FTP site and may need to adjust the sync scripts so they will not pull the Betas. The size estimates should be given for both the -RELEASE tree and the new package trees. It is most economical to the Mirror Sites if Beta-1 can be removed at the point Beta-2 is uploaded to ftp-master, and likewise for the RC(s). If that is possible size estimates for the follow-up Betas and RC's is not as critical because the change in disk space usage is close enough to zero - the size estimates are mostly needed to make sure the mirror sites' disk partitions don't get full. Similarly at the point the Release is being staged if possible remove the RC. If the Release has been finalized and is ready to be staged in the long run it is a dis-service for users to continue having access to the non-Release files (odds seem small someone would download an RC at this point and find/report a bug that would stop the release). If Beta-1 needs to co-exist with Beta-2 and so on then size estimates will again be needed a day or two before Beta-2 gets posted on ftp-master. Staging the Release ------------------- It would be best if the Packages could be loaded first, on the order of a week before the target Release Date. Sync-ing the ports directory is the worst part of the FTP site to deal with. It can at times wind up involving several sync attempts to get it all. The permissions on the files should not need any special attention - there seems to be no point to stopping users from being able to download them. Until Release Day they are basically useless so nobody is likely to try and get them. It is actually best if the -RELEASE stuff NOT appear until later because some end-users of Mirror sites will start to pester Mirror Operators as soon as something like an official Release directory structure shows up. The -RELEASE stuff should be posted to ftp-master three days before Release Day, with the permissions set so that access by "other" is turned off. Mirror sites are encouraged to run the sync as a different user than "ftp" so this will load the files onto the Mirror sites but they will not be downloadable until permissions are changed. Based on past history not all the packages for the different architectures are available at the same time, not to mention the -RELEASE trees. It is important that at least the i386 packages be available very early because of the relative popularity of that architecture. If packages for a given architecture are not ready by three days before Release Day it may be best to not post it to ftp-master until after Release Day. That way it will not compete for bandwidth with the -RELEASE trees being staged and it is probably most important to get as much of all the -RELEASE trees out as possible. Release Day ----------- Mail should be sent to hubs@freebsd.org a few days before Release Day giving the date and time that the Release Announcement will be sent out. At this time there is no "push" mechanism in the Mirror System so the opening up of the permissions needs to be either manual or it will happen at the time of the next automated sync. The tree on ftp-master should have its permissions changed and hopefully the Tier-1 sites will take care of the permissions at or close to the announcement time so down-stream Tier-2 sites can pick up the permission change fast. Consequences ------------ What happens if the guidelines are not followed? If the suggestion about not announcing disk space requirements when there is a large change some sites' disk partitions will fill, at which point the FreeBSD sync fails and any others that were supposed to happen afterwards also fail. This delays the propagation of the data. If the packages are not available on the mirror sites because of delays in getting them posted FTP-based installs users attempt before the packages become fully available may not complete correctly. Any data that gets posted to ftp-master after the Release Date will take a dramatically longer time to propagate to the Tier-2 servers. During Release times ftp-master becomes much more heavily loaded (slower). In addition to that the users are clogging the network pipes of the downstream sites so the syncs from these sites need to compete with all that network traffic. ---------------------------------------- -- Ken Smith - From there to here, from here to | kensmith@cse.buffalo.edu there, funny things are everywhere. | - Theodore Geisel |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030614014358.GA29321>