Date: Thu, 27 Sep 2001 10:19:27 +0300 From: Maxim Sobolev <sobomax@FreeBSD.org> To: Mario Sergio Fujikawa Ferreira <lioux@uol.com.br> Cc: Will Andrews <will@physics.purdue.edu>, FreeBSD-ports@FreeBSD.org, portmgr@FreeBSD.org, pat@databits.net, fenner@FreeBSD.org, kris@FreeBSD.org, green@FreeBSD.org, knu@FreeBSD.org, julian@FreeBSD.org, petef@FreeBSD.org, cwasser@v-wave.com, sjh-cl@horan.net.au, john_m_cooper@yahoo.com, matt@ipperformance.com, branson@windborne.net Subject: Re: review plz MASTER_SITES_n (ala OpenBSD) patch for bsd.port.mk Message-ID: <3BB2D2FF.B4742039@FreeBSD.org> References: <20010923055224.A93855@exxodus.fedaykin.here> <E15l7Nf-00072f-00@mx10.port.ru> <20010923133030.K69123@curie.physics.purdue.edu> <3BB05FD4.3E235748@FreeBSD.org> <20010927015548.A21765@exxodus.fedaykin.here>
next in thread | previous in thread | raw e-mail | index | archive | help
Mario Sergio Fujikawa Ferreira wrote: > On Sun, Sep 23, 2001 at 01:30:08PM -0500, Will Andrews wrote: > > On Sun, Sep 23, 2001 at 03:25:05PM +0400, Maxim Sobolev (sobomax@mail.ru) wrote: > > [snip] > > > like to test a final version of this patch plus patches to the > > various ports in my bento tree (mainly KDE and its dependneices) > > and see how it goes. Any objections to this method of testing? > > I am ALL for it. :) What better way to test if it works > than trying it against the whole ports tree. :PPP > Though, I stand by the patch, I am worried since I am fully > aware that no piece of software has not been fully tested untill > it has been, will this trial time disrupt anything? If not, no > problem. Just playing safe. :) > Also, I apologize for the delay on replying but I wanted > to both gather as much feedback as possible and be thurel on my > analysis. > > > > Why not in addition to postfixes in DISTFILES introduce the same > > > postfixes in MASTER_SITES and MASTER_SITE_SUBDIR? This would allow > > > to avoid introducing MASTER_SITES_foo, turning your example below > > > into: > > > > I like this: > > > > > MASTER_SITES= http://master7/:n \ > > > http://master1/:t http://master2/:t http://master3/%SUBDIR%/:t \ > > > http://master4/:y http://master5/:y http://master6/%SUBDIR%/:y \ > > > http://master8/:p > > > DISTFILES= file1:t file2:y file3:n file4:o:p > > > MASTER_SITE_SUBDIR= subdir:t subdir_test:y > > I find the idea appealing, it looks small and clean..... BUT > (there is always a but on long sentence:) > I will compare the previous proposal with our later > proposal exploring possible usage conditions: > > previous proposal (items prefixed with P) -> several MASTER_SITES_n > later proposal (items prefixed with L) -> single MASTER_SITES, all sites postfixed > > 1) how do we deal with the sites inside bsd.sites.mk? I will > use MASTER_SITE_SOURCEFORGE as an example. I have 3 files: > 1a) fileA can be obtained only in MASTER_SITE_SOURCEFORGE > 1b) fileB can be obtained in MASTER_SITE_SOURCEFORGE > and in another_place > 1c) fileC can only be obtained in another_site > > Since the idea of using bsd.sites.mk is that we > can do MASTER_SITES= ${MASTER_SITE_SOURCEFORGE} As I said in my original e-mail in this case you can do the following: MASTER_SITES= ${MASTER_SITE_SOURCEFORGE} ${MASTER_SITE_SOURCEFORGE:S/$/:a/} http://another_site/:b DISTFILES= fileA fileB:,a fileC:b MASTER_SITE_SIBDIR= normal_place another_place:a (please note that fileB actually has two postfixes - one empty, which should bring us [MASTER_SITE_SOURCEFORGE, normal_place], and second is a, which will bring [MASTER_SITE_SOURCEFORGE, another_place]) As you can see things are quite easy ;). -Maxim > > > * According to later proposal: > > L1.1) how do we deal with (1a) which seems to be both a common and > simple case? > > L1.1a) we could leave this as the default for the files > that no postfix :n; i.e., use the variable as it is > now, no postfixes > > L1.1b) or, we could postfix all elements of > MASTER_SITE_SOURCEFORGE inside bsd.sites.mk with > :SOURCEFORGE; then, we just postfix the files with it > > L1.2) how do we deal with (1b) which is a common variation > of (1a)? > L1.2a) we need to postfix fileB if we did (L1.1a), > but how we postfix if the original variable is not? > No idea.... Can be accomplished by some per-call > crazy regexp inside the calling Makefile but defeats > the simplification purpose of this proposal. > > L1.2b) if we did (L1.1b), we will need the , (comma) > operator suggested in the proposal which allows a > file to belong to more than one group at the same > time. Therefore, we could do > > DISTFILES= fileA:SOURCEFORGE fileB:SOURCEFORGE,OTHERSITE > > L1.3) (1c) seems a case pretty simple enough; as seen before, > solution to previous problems has led to (L1.2b) which leads > us to > L1.3a) we can use the default case for fileC > DISTFILES= fileA:SOURCEFORGE fileB:SOURCEFORGE,OTHERSITE \ > fileC > > L1.3b) or, we can add an additional group > DISTFILES= fileA:SOURCEFORGE fileB:SOURCEFORGE,OTHERSITE \ > fileC:SOMESITE > > *** Result: we need to postfix the sites inside bsd.sites.mk > *** Problem: we will need to fix ALL ports using these sites, > since they do not belong anymore to default which is the > group currently used by ALL ports > > * According to previous proposal: > > P1.1) how do we deal with (1a) which seems to be both a common and > simple case? > P1.1a) use the default group for this port (no > postfix) > > MASTER_SITES= ${MASTER_SITE_SOURCEFORGE} > DISTFILES= fileA > MASTER_SITE_SUBDIR= something something_else > > P1.1b) or, create a separate group > > MASTER_SITES_GROUPA= ${MASTER_SITE_SOURCEFORGE} > DISTFILES= fileA:GROUPA > MASTER_SITE_SUBDIR_GROUPA= something something_else > > P1.1c) or, have the patch automagically use > bsd.sites.mk > > DISTFILES= fileA:SOURCEFORGE > MASTER_SITE_SUBDIR_SOURCEFORGE= something something_else > > P1.2) how do we deal with (1b) which is a common variation > of (1a)? > P1.2a) if we did (P1.1a), we have 2 options: > P1.2a.1) create an additional group which adds > the default group > > MASTER_SITES= ${MASTER_SITE_SOURCEFORGE} > MASTER_SITES_GROUPB= ${MASTER_SITES} siteB > DISTFILES= fileA fileB:GROUPB > MASTER_SITE_SUBDIR= something something_else > MASTER_SITE_SUBDIR_GROUPB= other yetanother > > p1.2a.2) use the comma operator with an empty > argument and an additional group > > DISTFILES= fileA fileB:,GROUPB > > I don't think this is a good idea, > it is not semanticaly clean not to > mention visually confusing. We > could create a keyword _DEFAULT > which always refers to the default > group > > DISTFILES= fileA fileB:_DEFAULT,GROUPB > > P1.2b) if we did (P1.1b), we have 3 options: > P1.2b.1) add the additional group to the > default group > > MASTER_SITES= ${MASTER_SITES_GROUPA} siteB > DISTFILES= fileA:GROUPA fileB > > P1.2b.2) create an additional group which > adds the separate group > > MASTER_SITES_GROUPB= ${MASTER_SITES_GROUPA} siteB > DISTFILES= fileA:GROUPA fileB:GROUPB > > P1.2b.3) create an additional group but > use the comma separator to include the separate > group > > MASTER_SITES_GROUPB= siteB > DISTFILES= fileA:GROUPA fileB:GROUPA,GROUPB > > P1.2c) if we did (P1.1c), we have 3 options: > P1.2b.1) create an additional group and use the comma > operator > > MASTER_SITES_GROUPB= siteB > DISTFILES= fileA:SOURCEFORGE fileB:SOURCEFORGE,GROUPB > > P1.3) (1c) seems a case pretty simple enough > P1.3a) we can either use the default case > > MASTER_SITES= siteC > DISTIFILES= fileC > > P1.3b) or, create an additional group > > MASTER_SITES_GROUPC= siteC > DISTIFILES= fileC:GROUPC > > *** Result: all permutations are possible (no intrinsic design > "limitations") and we won't need to "fix" already working ports > > 2) how do we parse {MASTER,PATCH}_SITES{,_n}, > {DIST,PATCH}FILES{,_n}? > > P2.1) previous implementation already solved this issue > with a sample implementation > > *** Result: does not require "working" ports to be fixed > > L2.1) later implementation has this already solved by > *sobomax* and *will* > > > But, there's a small problem here. I believe ':' is a valid URI > > character, and I believe _some_ places have actually used this > > character. But I am not sure. Would it be possible to use a > > different character for the delimiter? > > As pointed out later in the thread, this shouldn't be a > problem as long as we do a ports sweep. > > > > It would not a big problem, because in MASTER_SITES presence of > > > postfix means absence of trailing `/' (hence it is easy to sort > > > it out), while possibility to find `:' in the MASTER_SITE_SUBDIR > > > is close to zero. > > > > Good point. How about we change the rules for MASTER_SITE_SUBDIR > > so that a / is required on the end? > > *** Result: does not require ports to be fixed > > 3) Pollution of port Makefiles > > P3.1) Let's examine an example > > MASTER_SITES= http://master7/ > MASTER_SITES_today= http://master1/ http://master2/ http://master3/%SUBDIR%/ > MASTER_SITES_yesterday= http://master4/ http://master5/ http://master6/%SUBDIR%/ > MASTER_SITES_p= http://master8/ > DISTFILES= file1:today file2:yesterday file3:n file4:o:p > MASTER_SITE_SUBDIR_today= subdir > MASTER_SITE_SUBDIR_yesterday= subdir_test > > This one looks polluted. TOO much noise inside the Makefile. However, > it is easy to spot the groups. Also, it scales without a problem > with several dozen group sites. > > L3.1) Let's examine the same example of (P3.1) > > MASTER_SITES= http://master7/ http://master1/:today http://master2/:today \ > http://master3/%SUBDIR%/:today http://master4/:yesterday \ > http://master5/:yesterday http://master6/%SUBDIR%/:yesterday \ > http://master8/:p > DISTFILES= file1:today file2:yesterday file3:n file4:o:p > MASTER_SITE_SUBDIR= subdir:today subdir_test:yesterday > > This one, obviously, looks cleaner in the eye. Less noise > inside the Makefile at first glance. However, as the number of sites > increase, too many postfixes need to be added. Scales with overhead (1 > postfix per site added per group) > > 4) How to share sites amongst groups > > P4.1) add one group to the other > > MASTER_SITES_SHARED= siteA > MASTER_SITES_ONE= ${MASTER_SITES_SHARED} > > P4.2) use the , (comma) operator > > MASTER_SITES_SHARED= siteA > MASTER_SITES_ONE= ${MASTER_SITES_SHARED} > DISTFILES= fileA:SHARED,ONE > > L4.1) replicate sites with different groups > > MASTER_SITES= siteA:GROUPA siteA:GROUPB siteB:GROUPB > DISTFILES= fileA:GROUPB > > L4.2) use the , (comma) operator > L4.2a) normal form > > MASTER_SITES= siteA:GROUPA siteB:GROUPB > DISTFILES= fileA:GROUPA,GROUPB > > L4.2b) advanced (an idea for development) > > This proposes that we could use the comma > operator in ${MASTER,PATCH}_SITES and > {MASTER,PATCH}_SITE_SUBDIR > > MASTER_SITES= siteA:GROUPA,GROUPB siteB:GROUPB > DISTFILES= fileA:GROUPB > > ----- > > This evaluation took quite some time. What is my conclusion? > > Well, the previous implementation is less intrusive in the > ports system, requiring no modification to existing ports but has a > pollution side which is undesirable but scales without a problem. (P3.1) > The later proposal is intrusive requiring a ports sweep > mostly due to the handling of bsd.sites.mk ((L1) conclusion). Also, it > "does not scale" well (at least to my eyes :) as seen in (L3.1), but this > shouldn't be a problem since most ports have only a few master sites. > We can accept some level of pollution in a few ports. > This analysis should get more ideas rolling and even more > feedback coming. Please, if you think I missed something or you find a > solution for something I said was a problem: LET US KNOW, contribute. > :-) > > ----- > > Ideas which came from this analysis: > > i1) , (comma) operator on all variables (L4.2b) > > i2) (based on an original idea of *green*) if we pursue the > previous proposal, we could: > i2.1) ditch the PATCH_SITES_n; therefore, we would > use {MASTER,PATCHES}_SITES which are the defaults > for respectively {MASTER,PATCH}FILES. Whenever we > want to use a group, we only use MASTER_SITES_n, > so postfixed :n files inside {PATCH,MASTER}FILES > all look for MASTER_SITES_n. This simplifies the > previous patch a bit. > > i2.2) also, following (i2.1) cue, we could collapse > {MASTER,PATCH}_SITE_SUBDIR_n using the ideas from > later proposal (L2.1) > > {MASTER,PATCH}_SITE_SUBDIR= a/:a b/:b > > i3) a keyword _DEFAULT (or, whatever) to identify the > default {MASTER,PATCHES}_SITES if we need to use them > with the comma operator (P1.2a) > > i4) if you get any, please let me know. We are in need > of good ideas. Keep them coming. > > ----- > > Regards, > > -- > Mario S F Ferreira - UnB - Brazil - "I guess this is a signature." > lioux at ( freebsd dot org | linf dot unb dot br ) > flames to beloved devnull@someotherworldbeloworabove.org > feature, n: a documented bug | bug, n: an undocumented feature To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3BB2D2FF.B4742039>