From owner-freebsd-ipfw@FreeBSD.ORG Mon May 2 22:50:15 2011 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 065111065674 for ; Mon, 2 May 2011 22:50:15 +0000 (UTC) (envelope-from korodev@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C7C248FC1A for ; Mon, 2 May 2011 22:50:14 +0000 (UTC) Received: by iwn33 with SMTP id 33so7058565iwn.13 for ; Mon, 02 May 2011 15:50:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:from:date:message-id:subject:to :content-type; bh=cRRyfdFeiOCMukijyUVFdbmMxxdFRIRLh8q5oD1A5t0=; b=uqWd6CAZ/D05fKr36KEnweOdOb+k3b7Gct+IrRW/AYCdBOMPDRf0msrwhpJB22I0MU lNwvgK28JVWuNn7eGdyG8W5xjBaT119PMkKkEGCqdAppob/fZmPNXfCNHU4JFr9gfkRG yhzt7/OwWRvz+0Tf1Gp4zX6WdZVlWG1ba7pU8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; b=qs0Gd1eQJJx39pY2u3itgFjdNyZTbd5GzS6xs7MRg82SpumYMbeVDneFKGBpUoXPm2 mwgBSl5UKo1w1TK3AUXoFJyL/+towajkJb1RDcFtXIEMr5dUsPaVmGwMnom2940017RP SNk+EZgMLWseLXxlnJ4j6vGv6B+7BlMBXxTkY= Received: by 10.231.81.18 with SMTP id v18mr6759580ibk.42.1304374792102; Mon, 02 May 2011 15:19:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.128.140 with HTTP; Mon, 2 May 2011 15:19:32 -0700 (PDT) From: Korodev Date: Mon, 2 May 2011 17:19:32 -0500 Message-ID: To: freebsd-ipfw@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: IPFW Table Insertion in C, Dummynet, and an interesting problem. X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2011 22:50:15 -0000 Hey guys, I'm currently running some custom C code ,via an output plugin for Snort, which takes an IP and sticks it in an ipfw table. Once the packet enters the box, I'm using dummynet to delay the packet while snort analyzes it and inserts the IP into a table, after the piping delay is complete the rule is reinserted at the appropriate point and checked via a deny table lookup rule. I've done some fairly extensive testing which has led me here. I believe I'm either doing my IPFW insertion wrong, or there's a bug or tuning setting I'm unaware of. I'd be delighted if you guys could take the time to look through my explanation below and let me know if anything comes to mind :) First my physical setup is as follows: Pinger --> { eth 0 --> bridge0 --> eth1 } --> Host #My Freebsd/IPFW setup: FreeBSD 8.2, net.link.bridge.ipfw=1, net.inet.ip.fw.one_pass=0 #IPFW Ruleset 00100 count icmp from any to any 00300 pipe 1 icmp from any to any //pipe is configured with config delay 150ms 00400 deny log ip from any to any src-ip table(0) 00500 count icmp from any to any #IPFW C Insertion Code .... #include #include //data is a custom struct data->s = socket(AF_INET, SOCK_RAW, IPPROTO_RAW); data->ent.tbl = 0; data->ent.value = 0; data->ent.masklen = 32; data->ip.s_addr = p->iph->ip_src.s_addr; memcpy(&(data->ent.addr), &(data->ip), sizeof(struct in_addr)); setsockopt(data->s, IPPROTO_IP, IP_FW_TABLE_ADD, &(data->ent), sizeof(data->ent)); ... This is slightly simplified for this test case, but it's important to know the insertion works, just not in less than 150ms. (~200ms actually). I've been talking to the Snort guys a bit, and I won't post my snort conf here, but please take my word that it's quite stripped down. My test case as follows, consisingt of a sending a single ICMP ping packet from Pinger to Host. In theory, Snort, listening passively on eth0 (libpcap 1.1.1), will alert on the ICMP ping to Host insert. . I conducted the following timing tests using tcpdump, Snort's performance profiling, and my own C timing code. Here are my results: tcpdump shows that the packet hits eth0 at 51.647347 seconds, bridge0 at 51.647350 seconds, and the exit interface, eth1, at 51.797320. This (as expected) equates to 149.97300 milliseconds of time. Snort is passively listening on the eth0 interface using the pcap daq module. It's configured to spend a maximum time of 250 microseconds before triggering my output plugin. Using some C timing methods on the output plugin code above says that grabbing the IP and sending it to the IPFW socket (which is cached and already open at that point), takes about 7.8120 milliseconds (1 CPU tick). Since that's where I call setsockopt, the it just sits in the pipe for the remaining time, but upon falling to the next rule (deny src-ip table 0), it misses the check. The count rule following my deny rule verifies that the rule did reach the deny table rule. There's only one untested spot that I can see, and that's the time between I actually make the insertion via setsockopt and the time IPFW "actually" update the table in memory. All of my other operations are executing with testable and consistent speeds, which are FAR less than a 150 millisecond dummynet pipe. Am I inserting the IP into the table correctly? Is there another command I need to call to force IPFW to update the table faster? Or perhaps there's some kernel tuning I'm unaware of? If you've made it this far, then thanks for taking the time to read this and please let me know if you have thoughts on why the IP isn't making it in the table fast enough. If more info is needed, I'll be happy to provide it :) Thanks, \\korodev