From owner-freebsd-stable@FreeBSD.ORG Tue Jan 6 19:16:24 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B11CE106564A for ; Tue, 6 Jan 2009 19:16:24 +0000 (UTC) (envelope-from db@danielbond.org) Received: from mail.nsn.no (mailtwo.nsn.no [62.89.38.161]) by mx1.freebsd.org (Postfix) with SMTP id 22C228FC1B for ; Tue, 6 Jan 2009 19:16:23 +0000 (UTC) (envelope-from db@danielbond.org) Received: (qmail 5681 invoked by uid 0); 6 Jan 2009 19:16:22 -0000 Received: from unknown (HELO ?85.95.32.72?) (85.95.32.72) by mail.nsn.no with SMTP; 6 Jan 2009 19:16:22 -0000 Message-Id: <484FA671-3B56-4A7C-9551-37AD2D1AF7B3@danielbond.org> From: Daniel Bond To: Vincent Hoffman In-Reply-To: <49639546.5070608@unsane.co.uk> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Tue, 6 Jan 2009 20:16:21 +0100 References: <08B216B4-79AB-45AB-AB4D-C8CD62196B87@danielbond.org> <49639546.5070608@unsane.co.uk> X-Mailer: Apple Mail (2.930.3) Cc: freebsd-stable@freebsd.org Subject: Re: freebsd-update painfully slow - slower than source code build of world and kernel X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2009 19:16:25 -0000 Hi, thanks for useful and relevant information. However, this is just one part of the process. Generating the diffs, prepping the server, and the whole server-side setup/management of it is another - I am sure there are tools for this too. Cheers, Daniel. On Jan 6, 2009, at 6:30 PM, Vincent Hoffman wrote: > Daniel Bond wrote: >> The same could be said about CVSup, one could write a caching cvsup >> proxy-server, and then we could just get rid of all the other >> cvsup-servers, except two (like freebsd-update soon will have). The >> point is, for portsnap and freebsd-update to scale properly, it needs >> to be opened up to the public, like CVSup is. People running a single >> server at home, or maybee two, most like won't want to set up a PROXY >> server, and they would be required to update both servers at the same >> day for the Proxy server to actually cache something - which many may >> not want. And there are a lot of people running a few servers, here >> and there. >> >> >> >> Sure, a national squid-proxy could work - although, there is no >> individual proxy setting for portsnap/freebsd-update.. It honors >> HTTP_PROXY environment variable, which a lot of other tools also use. >> Some tools might not work via this proxy, especially for local >> addresses - the administrators of these servers probably don't want >> all the ports tarballs to go via these, and people could use them for >> nasty things. So, then we are back to manually setting/specifying the >> proxy-server, each time one wants to run the commands - which people >> might forget. (Is this getting complicated enough yet..?) We would >> basically be creating a whole lot of new potential problems for the >> users, to solve the problem in question.. >> >> >> I am also interested in learning how the portsnap protocol works, >> maybe there are potential issues with it, that a second eye might >> spot, or room for improvement? From what I gather, Colin is a very >> cleaver guy, so it is not very likely, but still, other people could >> learn from it. >> > well portsnap/freebsd-update are shell scripts so not too hard to > read. > The actual transfer protocol is piplined http and is done by > /usr/libexec/phttpget (in base so src code available > /usr/src/usr.sbin/portsnap/phttpget/phttpget.c ) > also see http://www.daemonology.net/phttpget/ > > >> I would like to see these tools as the default recommended tools to >> use in the future, and that is why I am so worried about this. >> The point I am trying to make is, or actually the question is: Why is >> freebsd-update (and portsnap) so secretive? Why can't the average Joe >> run his own portsnap-mirror at home? What are we afraid of? > I seem to remember once reading that Colin wanted to make it a more > polished system before he release it, but i cant find that email > anymore. > > Vince >> >> I don't see any problems with this, except maybe loosing some detail >> in Colin's nice graphs (which would be the case for proxies too). >> >> >> Cheers, >> >> >> Daniel. >> >> >> On Jan 6, 2009, at 5:42 PM, Christopher Arnold wrote: >> >>> >>> >>> On Tue, 6 Jan 2009, Daniel Bond wrote: >>> >>>> reading your answer, you are obviously confusing what I am saying >>>> about freebsd-update with the portsnap program. Also, I also wrote >>>> in my first post >>> No i'm not confusing them, just trying to follow two subjects at the >>> same time. Sorry if that is confusing. >>> >>>> that HTTP_PROXY / Caching proxy server does not help me much. This >>>> is because I download a lot of "initial tarball snapshots".. I >>>> would >>>> rarely see "Cache hits" in my proxy log. I guess I could set >>>> something up to fetch nightly via proxy, to keep the data in house, >>>> for when I need it. I don't want to use a PROXY server, I feel this >>>> is attacking the problem at the wrong end. >>>> >>> Ok, lets go again. Either you mirror (maybe by having a squid proxy >>> and walk the tree) and thats going to me even worse for you. Or you >>> use a squid proxy to keep stuff you need close to you and share >>> among >>> different installations. >>> >>> Or you setup one or more national squid proxies and configure your >>> machines manually just like you do with cvsup. >>> >>> >>> >>>> I agree, I am interested to hear the views of the wise ones. >>>> Personally I'm going back to CVSup until freebsd-update and >>>> portsnap >>>> mirrors are in a more distributed or usable state. >>>> >>> At least portsnap started to work for me earlier today. Havn't tried >>> update yet. >>> >>> But yes i agree, update and portsnap infrastructure could be done >>> better. >>> I have some ideas and will try to write them down in a while. >>> >>> /Chris >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to >>> "freebsd-stable-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org >> " > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > "