Date: Fri, 10 Oct 2014 08:26:16 +0200 From: Marcus von Appen <mva@FreeBSD.org> To: freebsd-ports@freebsd.org Subject: Re: [BRAINSTORMING] simplifying maintainer's life Message-ID: <20141010062616.GA1044@medusa.sysfault.org> In-Reply-To: <20140904091237.Horde.333o5ETImmGIX5gwkO04gA6@webmail.df.eu> References: <20140903145614.158f8e89@kalimero.tijl.coosemans.org> <20140903135029.GK63085@ivaldir.etoilebsd.net> <20140903165622.3bff54e0@kalimero.tijl.coosemans.org> <20140903150018.GL63085@ivaldir.etoilebsd.net> <5407649A.80500@FreeBSD.org> <20140903191256.GB1025@medusa.sysfault.org> <20140903212243.GK63931@graf.pompo.net> <20140904060011.GB1044@medusa.sysfault.org> <540804DB.5030600@FreeBSD.org> <20140904091237.Horde.333o5ETImmGIX5gwkO04gA6@webmail.df.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
--6TrnltStXW4iwmi0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On, Thu Sep 04, 2014, Marcus von Appen wrote: > > Matthew Seaman <matthew@freebsd.org>: > > > On 04/09/2014 07:00, Marcus von Appen wrote: > >>> - I often grep all plists to find which port could possibly provide such > >>> > header or such library (among non-installed ports, of course). > > > >> I do the same, but would argue that such a query service should belong to or > >> offered by a pkg search (as sort of counterpart to pkg which). > > > > We've toyed with that idea -- allowing 'pkg search' or similar to search > > on any file in any package known in the repositories. The biggest > > problem is that including all that data in the package catalogues would > > bloat their size by a very large amount. > > > > Rather than bloating the catalogues for any use, there was a separate > > index of files. Not sure whether that's being routinely built on the > > FreeBSD pkg cluster at the moment -- probably not, as it was only ever > > experimental, and didn't have any generally available consumers. > > I did not mean it to be offline available, since it becomes outdated too fast. > > > In many ways, I'd prefer to have this sort of functionality available as > > a web-app, thus saving users the necessity of downloading megabytes of > > data about ports / packages they would never use or care about. Needs > > someone to step up and write that application though. > > Not necessarily a web app, but a (web) service that's e.g. run somewhere on a > pkg builder or proxy and which can be queried by tools as well as web > services. > I gave the pkg repo output a quick shot with about 370 random packages, which creates a filesite.yaml of roughly 4.5 MB. Reading the file and transforming it into a tree to get access to the file entries resulted in about 80000 nodes (a compressed variant, where nodes with only a single child are merged, contains 78000 nodes). 370 packages are about 1 to 1.5 percent of the package amount, we currently have. Assuming linear growth, filesite.yaml would be around 450 MB in size, and a node tree for searching would contain around 8 million nodes. I doubt that a reasonably fast search service could be implemented on top of filesite.yaml alone. Storing everything in memory is not an option, since the index tree alone would consume far more than 300 MB (assuming an optimal word size of 20 bytes plus a bit of node and tree payload). Fragemented search over the file would cause heavy disk I/O and no matter, how many threads will perform the search, the disk I/O eventually will become the bottleneck. Searching the tree would be horribly slow, since the traversal would need either additional information in the structure (to avoid a complete BFS/DFS for file fragments) or keeping subtrees and helper structures in memory, which easily blows the minimum memory amount to use without a single query being executed. With this amount of information, Tries, DAWGs or generic DAGs would easily hit their limit and one would need to set up an incremental search based on seperate indices, which effectively leads to a document index and search implementation. My guess is that a webservice for searching the catalogue would be easier to be implemented based on a full text search engine, such as Solr or Lucene, since each particular entry within filesite.yaml is a specific and very small document. Cheers Marcus --6TrnltStXW4iwmi0 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlQ3fAgACgkQi68/ErJnpkcPVgCg1ofMvYriO/tDKtlA0FXziipP YmAAoM2CINk+XIIIY/3yUJeCV4FEjZLv =geJS -----END PGP SIGNATURE----- --6TrnltStXW4iwmi0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141010062616.GA1044>