From owner-freebsd-ipfw@freebsd.org Sun Jul 31 20:18:01 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D21ABA9F77 for ; Sun, 31 Jul 2016 20:18:01 +0000 (UTC) (envelope-from rj@obsigna.com) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0520E1457 for ; Sun, 31 Jul 2016 20:18:00 +0000 (UTC) (envelope-from rj@obsigna.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1469996277; l=7365; s=domk; d=obsigna.com; h=To:References:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From: Subject:Mime-Version:Content-Type; bh=eWS4rQkxY2qpgP/NkpCUz9VQcmdxruPgHdkjXu/q6zA=; b=w8YoYlzgCGlIIY0BoyKzjbxXsAOVVv6LMeSvxp+/uXPlJz2G/SCgicD2LUxs0W138pR 7MmlKxA1FTBXY0RcJ91fY3uj808YdDZKcbKFzHoiDoeFmzHlWxHwnx4sdZ2N7UQkHNjGn su9G6MLfqYy7mzw+5tyx2Vei0bWdpbFjDKg= X-RZG-AUTH: :O2kGeEG7b/pS1EK7WHa0hxqKZr4lnx6UhToX1IWHkW4X7v2ImaU2B+3KSGnPFnK/TBWBAh4A X-RZG-CLASS-ID: mo00 Received: from mail.obsigna.com (bfb6bfbf.virtua.com.br [191.182.191.191]) by smtp.strato.de (RZmta 38.13 DYNA|AUTH) with ESMTPSA id 6057c4s6VKHo917 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Sun, 31 Jul 2016 22:17:50 +0200 (CEST) Received: from rolf.projectworld.net (rolf.projectworld.net [192.168.222.25]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.obsigna.com (Postfix) with ESMTPSA id 20C05229861E; Sun, 31 Jul 2016 17:17:46 -0300 (BRT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ipfw divert filter for IPv4 geo-blocking From: "Dr. Rolf Jansen" In-Reply-To: <20160801030317.P29054@sola.nimnet.asn.au> Date: Sun, 31 Jul 2016 17:17:44 -0300 Cc: Ian Smith Content-Transfer-Encoding: quoted-printable Message-Id: <166B5A24-8E5A-444D-BBF5-2B5883DE76A7@obsigna.com> References: <61DFB3E2-6E34-4EEA-8AC6-70094CEACA72@cyclaero.com> <4D047727-F7D0-4BEE-BD42-2501F44C9550@obsigna.com> <9641D08A-0501-4AA2-9DF6-D5AFE6CB2975@obsigna.com> <4d76a492-17ae-cbff-f92f-5bbbb1339aad@freebsd.org> <677900fb-c717-743f-fcfe-86b603466e33@freebsd.org> <0D3C9016-7A4A-46BA-B35F-3844D07562A8@obsigna.com> <1E1DB7E0-D354-4D7A-B657-0ECF94C12CE0@obsigna.com> <50d405a4-3f8f-a706-9cac-d1162925e56a@freebsd.org> <9222BB10-C700-4DE7-83A3-BE7A38A11713@obsigna.com> <1B36CAD7-A139-436B-B7EC-0FFF232F9C6A@obsigna.com> <20160801030317.P29054@sola.nimnet.asn.au> To: freebsd-ipfw@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jul 2016 20:18:01 -0000 > Am 31.07.2016 um 15:38 schrieb Ian Smith : > On Sat, 30 Jul 2016 11:17:13 -0300, Dr. Rolf Jansen wrote: >> I finished the work on CIDR conformity of the IP ranges tables=20 >> generated by the tool geoip. The main constraint is that the start=20 >> and end address of an IP block given by the delegation files MUST BE=20= >> PRESERVED during the transformation to a set of CIDR records. This=20 >> target is achieved by: >>=20 >> 1. Finding the largest common netmask boundary of the start address = utilizing >> int(log2(addr_count)); then iteration like Euclid's algorithm in = computing >> a GCD. >>=20 >> 2. Output the CIDR with the given start address and the masklen = belonging >> to the found netmask. >>=20 >> 3. If the CIDR does not match the whole original IP range then set = the start >> address of the next CIDR block to the next boundary of the common = netmask, >> and loop over starting at 1. until the original range has been = satisfied. >>=20 >> I carefully tested the algorithm and a table that I pipe by the new=20= >> geoip tool into ipfw is 100 % identical to the output of the ipfw=20 >> command 'table N list'. >=20 > Great. I suppose that caters for some of the odd delegations one = sees,=20 > such as perhaps a /16 then a /15 (ie 3/4 of a /14) followed by maybe a=20= > /12, maybw with another /15 tacked on the end .. but I'm unsure if = that=20 > applies to country allocations as much as it does within countries. >=20 >> It is worth to note, that already the original RIR delegation files=20= >> contain 457 non CIDR conforming IPv4 ranges in a total of 165815=20 >> original records. I guess that this number will increase in the=20 >> future because the RIR's ran empty on new IPv4 ranges and are urged=20= >> to subdivide returned old ranges for new delegations. The above=20 >> algorithm is ready for this. >=20 > Yes, and just as well. I'm surprised it's as few as 457 .. I looked=20= > into it a bit back when 115.70/17 was first allocated to my ISP, after=20= > previously having been, as I recall, in China .. so of course we fell=20= > foul of a number of (probably perennially) out-of-date geoip blockers,=20= > for months in some cases .. malevolent beasts if not kept well fed :) >=20 >> Generally, CIDR conforming tables are more than twice as large as=20 >> optimized (joined adjacencies) IP range tables. All said changes have=20= >> been pushed to GitHup already. >=20 > So how many table entries does 'the world' come to, around 400,000? No, it is not that bad. The total number of original entries in the = delegation statistics files of all 5 RIR's is about 166000. The ipdb = tool which compiles these ranges into a consolidated sorted binary = table, that is suitable for loading it directly into a binary search = tree, reduces the number of entries to a bit more than one half, namely = ca. 83500. Consolidation primarily means, resolving of overlaps, because these = could not be handled in a meaningful way by a binary search tree. Only = as an additional benefit in the same go, that routine combines = adjacencies with the same country code, although, skipping the = combination is technically not a show stopper for the BST, this is only = to increase the performance. The geoip tool which generates the tables of CIDR ranges per country = code out of the consolidated tables would output a count of 167500 = entries for all countries. That is a little bit more than the original = count, however this table is still optimized, because original ranges = that when combined form a new valid CIDR are not broken down again, but = the combined CIDR is passed. >> I am still a little bit amazed how ipfw come to accept incorrect CIDR=20= >> ranges and arbitrarily moves the start/end addresses in order to=20 >> achieve CIDR conformity, and that without any further notice, and=20 >> that given that ipfw can be considered as being quite relevant to=20 >> system security. Or, may I assume that ipfw knows always better than=20= >> the user what should be allowed or denied. Otherwise, perhaps I am=20 >> the only one ever who input incorrect CIDR ranges for processing by=20= >> ipfw. >=20 > You've lost me here, Rolf. Do you mean that ipfw adds incorrect table=20= > entries for a given IPv4 address and mask length? Or that it c/should=20= > itself accept IP ranges and generate the needed CIDR entries to match? Perhaps an example may explain it better. Remember that the first = incarnation of geoip passed the incorrect range 201.222.20.0/20 to ipfw. = This is an incorrect CIDR because the start address does not match a = mask boundary defined by the given masklen. The point now is that this = error is caused by EITHER the masklen is incorrect OR the start address = is incorrect. ipfw can determine only that the CIDR is incorrect, and = does rectify it for further processing: # ipfw table 1 add 201.222.20.0/20 # ipfw table 1 list --> 201.222.16.0/20 0 So actually ipfw happily takes an incorrect CIDR and transforms this = into a correct one under the arbitrary assumption that the masklen is = the correct part and the start address is sort of variable. Technically = the addition by ipfw is a correct CIDR, but this is not necessarily the = range the user wanted to add. > If the former, how to reproduce for a bug report? If the latter, = might > you contemplate adding that functionality to ipfw Well, in the meantime, I saw that this kind of automatic rectification = of incorrect CIDR entries is within ipfw throughout. # ipfw add 50000 allow ip from 201.222.20.0/20 to any --> 50000 allow ip from 201.222.16.0/20 to any At least in this case the user is informed directly about the CIDR that = has been actually utilized, anyway, it is not exactly the range the user = has asked for, and the tiny difference can easily be overseen. Given that ipfw can't know what the user actually intended - did he = mistype the start address or obtained somehow a wrong masklen -, I = advocate that ipfw should accept the wrong CIDR and transform it to a = correct one exactly as above, but should output a warning, either to the = shell or to syslog in case no shell is connected. Ideally this warning = would give a useful explanation why the input CIDR is wrong and how it = could be made correct by either adjusting the masklen or the start = address. > - or is ipfw better=20 > off being driven to generate tables from the output of such as geoip? Of course, tools like geoip HAVE TO produce by 100 % valid CIDR, = otherwise the tool is buggy and the bug must be fixed as in the cse of = geoip. Of course, ipfw must not receive any so called "bug fix" that = immediately may break lots of firewall installations around the world, = given that it is so easy to inform invalid CIDR, and that ipfw happily = accepted these up to now. I don't want to look more catholic than the pope. My main concern is not = that ipfw does a choice for correcting invalid CIDR, my concern is that = the choice is sort of arbitrary. I advocate for logging a suitable = warning, but keep on processing invalid CIDR's by transforming it to = valid ones exactly as before. Best regards Rolf