From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 1 08:52:06 2013 Return-Path: Delivered-To: FreeBSD-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6052B264; Mon, 1 Apr 2013 08:52:06 +0000 (UTC) (envelope-from cnst++@FreeBSD.org) Received: from Cns.Cns.SU (unknown [IPv6:2001:470:7240::]) by mx1.freebsd.org (Postfix) with ESMTP id E81B3A35; Mon, 1 Apr 2013 08:52:05 +0000 (UTC) Received: from [127.0.0.1] (root@localhost [127.0.0.1]) by Cns.Cns.SU (8.14.5/8.14.5) with ESMTP id r318q4ca019732; Mon, 1 Apr 2013 01:52:04 -0700 (PDT) Message-ID: <51594AB3.4070002@FreeBSD.org> Date: Mon, 01 Apr 2013 01:52:03 -0700 From: "Constantine A. Murenin" Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 SeaMonkey/2.13.2 MIME-Version: 1.0 To: FreeBSD-current@FreeBSD.org, FreeBSD-hackers@FreeBSD.org Subject: BXR.SU, Super User's BSD Cross Reference w/ OpenGrok, publicly private beta Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Apr 2013 08:52:06 -0000 Dear FreeBSD-{current,hackers}@, It is my great pleasure to announce the immediate availability of a publicly private IPv6-only beta test of BXR.SU -- Super User's BSD Cross Reference. BXR.SU is based on an OpenGrok fork, but it's more than just OpenGrok. We've fixed a number of annoyances, eliminated features that just never worked right from the outright, and provided integration with tools like CVSweb (including awesome mirrors like allbsd.org), FreeBSD's ViewVC (SVN), as well as Gitweb from git.freebsd.your.org, plus a tad of other improvements, including a complete rewrite of an mdoc parser. Last, but definitely not least, is an extensive set of nginx rewrite rules that makes it a breeze to use BXR.SU as a deterministic URL compactor for referencing BSD source code. What's up with the publicly private beta test? We're launching today in a publicly private beta. Participation in the beta is invitation-only; everyone with IPv6 is invited. We're cooperating with ISPs around the world, and in order to be able to access BXR.SU during this beta phase, you must have a special token, also known as a publicly routable IPv6-address, with proper IPv6-connectivity and upstream peering. If you don't have IPv6 yet, but want to participate in this beta test ASAP, then ask your ISP for IPv6 ASAP! Else, if your ISP is not part of our beta rollout, you could try something like tunnelbroker.net from he.net. What's the release schedule? BXR.SU is available through IPv6 today, 2013-04-01. It is currently an IPv6-only site, with an IPv6-only glue, too. As an IPv6-only site, we hereby declare that 2013-04-04 is an IPv4 day. On April 4, we will temporarily enable IPv4 connectivity, for one day, to test the water. (We've heard that IPv4 has some connectivity issues related to NAT, double-NAT, carrier-grade NAT and NAT64, and some small percentage of users (but significant in absolute terms) might not be able to access the site if an A record is published, due to the plentiful of misconfigurations out there; so, we want to take things slow, and ensure our users don't suffer from any inferior connectivity.) If things do go well (we expect IPv6/IPv4-related improvements as time goes by), we will permanently publish an A record for BXR.SU on 2013-04-14. IPv4 glue records will be published shortly thereafter, on 2013-04-24 (we don't do this today, because we're afraid that the nameservers of some ISPs are not configured correctly, and our IPv6 users won't be able to access our site otherwise, so, we think it's a good idea to take things slow and in steps). But why another OpenGrok? Over the years, there have been a number of OpenGrok installations that have made it possible to study and grok BSD code, for which we are very thankful to their maintainers. However, as a general rule, none of them have been inclusive of all BSD flavours, all of them have had rather long and hard-to-remember URLs, which also didn't really look permanent at all, and, unfortunately, many of them no longer exist today, or some new uber-inclusive services like code.metager.de have recently flourished, with an astounding 8 second (yes, eight second) delay for satisfying any single search query (hot queries are returned in as little as just under 4 seconds by metager, yet everything is nonetheless buffered, so, you get no rendering at all for those whole 4 or 8 seconds). So, we thought this had to change. So, what's the deal? It's simple. Say, someone doesn't know who PHK is. You can point them to: http://bxr.su/s?q=phk Want to see if DragonFly keeps queue.h in sync? Take a look at: http://bxr.su/d/sys/sys/queue.h Want to look at FreeBSD's queue.h, to manually compare? Just change the "d" from "/d/" (or select and replace the whole world "DragonFly" from "/DragonFly/") to "f", and you're in FreeBSD: http://bxr.su/f/sys/sys/queue.h Too many /sys/sys/? We've still got you covered, thanks to nginx: http://bxr.su/o/queue.h Anyone uses TAILQ_SWAP? Is that a new thing? Check it out: http://bxr.su/search?q=TAILQ_SWAP Any mentions of "OpenBSD" or "NetBSD" in FreeBSD and DragonFly? http://bxr.su/f,d/s?q=OpenBSD+OR+NetBSD Who's this guy writing this email anyway? Is he BXR'able? http://bxr.su/s?q=%22Constantine%20A.%20Murenin%22%20OR%20cnst Etc. Just how fast is BXR.SU? We expect that most search requests should be fulfilled (search page results generated) in well under 100ms. In my tests, and according to OpenGrok metrics at the bottom of each search page, most search pages are generated in about 30 to 50ms, so, it does seem like there's some room to spare. In addition, of course we use nginx, so, once generated at 40ms, a page should be available immediately in no time should a subsequent identical request come along within a couple of seconds or so. How does it compare with fxr.watson.org? + we're based on OpenGrok, instead of LXR + we also index userland of all BSDs, not just the kernels like the fxr + we do daily updates and re-index of all 4 BSDs - we only track -CURRENT of each BSD + the -current that we track is actually current Is this a replacement to nxr.netbsd.org? Not necessarily. We've noted that the /gnu/ and /contrib/ directories of various BSDs have been constantly giving out lots of false positives on all kinds of search queries, and they have been excluded, for both the accuracy and disc space utilisation. This only affects userland, and mostly excludes stuff that noone really cares about anyways (some exceptions were made, and this is not really in stone anyways). Otherwise, BXR.SU is much faster than the nxr -- nxr search is unbuffered, but takes several seconds on cold runs, and about half a second on hot runs. How does it compare with code.metager.de? + we're over 20000% faster (0.04s vs. 8s). 'nuff said. I welcome comments, questions and suggestions. I hope that this site will be useful to the BSD community, and will join other great and invaluable BSD-related sites that we all use and depend on. In summary, what's unique about bxr.su? * Supports all 4 BSDs * Daily updates * Short URLs, deterministic URL shortener for BSD source code * Kernel + non-gnu userland * Very fast -- most search pages generated in under 50ms * IPv6-only for now, will be dual-stacked very soon Enjoy! Constantine.