Date: Fri, 27 Dec 2002 22:30:09 +0100 From: "Simon L. Nielsen" <simon@nitro.dk> To: freebsd-binup@freebsd.org Subject: Proposal for binup design document Message-ID: <20021227213008.GC415@nitro.dk>
next in thread | raw e-mail | index | archive | help
--eAbsdosE1cNLO4uF Content-Type: multipart/mixed; boundary="J/dobhs11T7y2rNN" Content-Disposition: inline --J/dobhs11T7y2rNN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello I really think binup could be useful and fill a gap for FreeBSD. Therefor to try to get things moving and to get some kind of structure on binup I have begun make a kind of design document containing some thought / ideas about binup. I think it is important to discuss these issues (especially the HTTP proposal) so we know what to make binup do and how it should be done. I have a attached a text only version of the document, but it has been written in HTML and can be found at http://simon.nitro.dk/binup/binup-2003.html . I plan to start looking at the server database backend today and adding/changing the binup design document along the way. --=20 Simon L. Nielsen --J/dobhs11T7y2rNN Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="binup-2003.txt" Binup 2003 proposal Contents * [1]Introduction * [2]General * [3]HTTP * [4]Main use * [5]Package abstraction * [6]Package the base system * [7]Client/server communication * [8]Tasks Updated : $Date: 2002/12/27 21:17:46 $ binup 2003 introduction This is a proposal by Simon L. Nielsen <[9]simon@nitro.dk> for a design document for the FreeBSD Binary Updater (binup). The document contains some ideas for changes to the current binup code/design and the beginning to a general design document about binup. The document is very much work in progress... and a bit of a mess at the moment. I have called this document binup 2003 to differentiate it from the current "old" binup code. The [10]"old" binup document should be read before this document since the main design and goals haven't changed. Please note that this document is meant as an early draft and most likely contains flaws, bad ideas and things that have not been thought fully through. There is also most likely a lot of spelling and grammatical errors - please ignore them for now. It contains comments/ideas on the subjects I think needs to be addressed before real coding on binup 2003 can begin. I would very much like comments on the document and the ideas. I would prefer that comments are posted to the [11]binup mailing-list for discussion but private comments directly to me are also very welcome. I plan to keep an updated version of this document on [12]http://simon.nitro.dk/binup/binup-2003.html. General I think it would be a good idea to change the binup system to use HTTP instead of doing a new TCP level protocol. Mainly because I think this could save a lot of coding. See the [13]HTTP section for more. As suggested on the binup mailing-list some time ago I think binup should be made independent of the packages actually transfered by binup. This would make it possible to use distribution tar-balls, .tgz's, libh, binary patches or any other packaging format without any major change to binup. See the [14]package abstraction section for more. There have been talk about making the binup project do a package API (/var/db/pkg API) and do the breaking up the base system into packages. While I think these are good ideas that would certainly benefit binup I don't think we should wait for this to be finished since that may take a long time and it should be possible to do a good binary updated without these tasks completed. HTTP The current binup code uses a special protocol on top of TCP. This makes the protocol more flexible but also require a lot of code to get working. Instead binup could be implemented on top of HTTP. This would make some things a lot simpler since a lot of existing programs/code can be used both server and client side. The use of HTTP will simplify implementation since a lot of functionality will come "for free" e.g. file transfer, SSL and connection management. Standard server/clients also gives the advantage that less low level C code has to be written and therefore minimizing the risk for buffer overflows and other common C bugs. Using HTTP it would also be quite simple for the binup server to point a client at a normal FreeBSD mirror for the actual downloading of updated packages (or perhaps the client can pick it's preferred mirror site). A checksum/crypto signature could still be send from the binup server and therefore increasing security a bit for trojan-ed files. This might not be a big increase - but it would help a bit. The main drawback of using HTTP is the lack of state between requests but I don't see this as a major drawback since most the needed state code can be kept client side instead. Advantages of using HTTP : * Less reinventing the wheel * Will be using standard Internet protocol (big advantage if client is behind a firewall) * Simpler prototyping (a lot of the code can be initially implemented in a scripting language) * Less low level C code Disadvantages of using HTTP : * No state * Will mean scraping most of the current binup code Main binup uses This section contains some thoughts on where/how binup could be useful. I think this is important to keep in mind to determine exactly whan binup is supposed to do. The main uses for binup as I see it is (in more or less prioritized order) : 1. Security updates (e.g. 4.7p1 -> 4.7p2) 2. Port updates 3. FreeBSD upgrades (e.g. 4.6 -> 4.7) 4. Fresh FreeBSD installs with profiles I see two main types of operation for a binup server - private and public servers. A private server can for example be used for a company to have a simple way to deploy updates to a server park. In this scenario updated will most likely have to be tested before being deployed widespread in the administrators must therefor have ability to add a package to be updated on a per package basis (package in a general sense both OS upgrade and port upgrade). The public server could be used as a kind of replacement for buildworld/installworld for people who just want to do a network upgrade. This could also be very useful for people who want to keep up to date with security fixes without having to recompile everything for a "simple" small patch. When clients do an update the binup client should allow this to be done either automatically or in an interactive mode. Package abstraction I belive it would be useful for binup only to have a minimal "knowledge" of the packages being transfered. That way binup is not being bound to a specific packing format which might change once libh is fully implemented or the base system is packaged in some special format. This would also alow update packages to be available in multiple packaging formats which might be useful in some cases. It would of course make the tripwire functionality harder to implement, but it could still be done. This could mean that binup could be used to transfer all kind of odd package formats even .rpm, .deb packages if that should be needed. binup mainly need three pieces of information for each package as far as I can see : 1. Package name - must be unique 2. Package format - so the client knows how to install the package 3. Package version - to find new versions The real tricky one is package version since third party programs use all kinds of odd version schemes. Client/server communication I think that when a client wants to upgrade the client should get a list of the most recent packages for a profile the client specifies and then the client decides which packages to download and install. This way partial upgrades should be handles automatically and the privacy of the client is preserved since the server does not know which packages the client does actually have installed. The client -> server can be done by a simple HTTP GET or POST with the needed variables. For the server -> client data there must be some kind of formatting/encoding so the client can parse it in a simple way. The main formatting/encoding styles I see are comma separated or XML. XML is very general and easy to extend later but it has a few major drawbacks. The main drawback is the requirement of an XML parser client side and I think the binup client should depend on as few external components as possible to minimize problems when blowing things away under the client and when doing from scratch installs. Another drawback for XML is the extra overhead both from larger network transfers and from the parsing. Comma separated would require special parsing code to be written and would most likely break if new fields are added to the lines. The advantage is that it requires no external library to parse and it is rather fast to parse. I lean mostly to comma separated format since it does not require a XML library on the client... binup tasks The tasks to get a complete binup system as I see it (not all may be required to get first version of binup working) * Find out what binup 2003 should really do * Specify client/server protocol * binup server + Create database back-end + Implement server in some scripting language (perl/PHP/???) + Implement server in C (perhaps apache module ?) * binup client * package up base system + Package lib API (API for /var/db/pkg) + Find out how to do versioning (Problem with source installs) Implementation notes Database back-end should be implemented in server so it can be replaced later e.g. with dbm files if needed. $Id: binup-2003.html,v 1.6 2002/12/27 21:17:46 simon Exp $ References 1. #intro 2. #general 3. #http 4. #main-use 5. #package-abstraction 6. #package-base 7. #client-server-comm 8. #tasks 9. mailto:simon@nitro.dk 10. http://www.freebsd.org/projects/updater.html 11. mailto:freebsd-binup@FreeBSD.org 12. http://simon.nitro.dk/binup/binup-2003.html 13. #http 14. #package-abstraction --J/dobhs11T7y2rNN-- --eAbsdosE1cNLO4uF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQE+DMZg8kocFXgPTRwRAs2FAKCaIOn8QNEO1nJ0TlCjzqTQV2hVjACfTdAy 3QRJOpqSAKRV8Y7b5hyoMic= =blO5 -----END PGP SIGNATURE----- --eAbsdosE1cNLO4uF-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-binup" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021227213008.GC415>