From owner-freebsd-net@freebsd.org Wed Jul 1 14:31:52 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05F709917C8 for ; Wed, 1 Jul 2015 14:31:52 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 61BC41901 for ; Wed, 1 Jul 2015 14:31:51 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lbnk3 with SMTP id k3so14120426lbn.1 for ; Wed, 01 Jul 2015 07:31:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=VA6S0zvHWL75nFZSHisaRbUgzAgEmRPUshDJ7C4Wvg4=; b=fXATJbrl9eTPcGDUzoGGA36H4N9yYfEzYO84d3SvB17x3rw8FokysND988BHCPIlP0 mAQ4uqwTASU6bwQaq0Lcw2r8xqionarcDuLGX+xkxqsOrOWzJhWKf4LNJLhS0bOOdueN u+OGyV1qKOI5dGXa/V8tPKg6Q+sJehTT2/EVTe9LWVZFRdIWYMitkNHwtPv5Oj5ciKU8 2JbzzpxCXe+GNKnwzRoWcXa9bsS6ZQdrsFqUO25FudpxYlKt/gsIFTiw+r6/vLimWGla LlbY5Vw91mtAN9Qak1isa09t2Vm0WdtzTEEjeK5Gp2uS3orWjcfw4VnPl+bTalbvzbbL omTg== MIME-Version: 1.0 X-Received: by 10.112.220.7 with SMTP id ps7mr2145181lbc.72.1435761109201; Wed, 01 Jul 2015 07:31:49 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.230.103 with HTTP; Wed, 1 Jul 2015 07:31:49 -0700 (PDT) In-Reply-To: References: Date: Wed, 1 Jul 2015 16:31:49 +0200 X-Google-Sender-Auth: zBqlsdlAj6ToPSF9lW_HsHBK4LM Message-ID: Subject: Re: IPFW divert and suricata From: Luigi Rizzo To: Oliver Humpage Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jul 2015 14:31:52 -0000 On Wed, Jul 1, 2015 at 3:15 PM, Oliver Humpage wrote: > > Hello, > > I hope this is a good list to post this on, I have a feeling the solution > is somewhere obscure in the networking layer. > > I've set up an IPS system, using: > > * FreeBSD 10.1 (guest OS, plenty of RAM/CPU) > * ESXi 5.5 (host OS, using Intel X520 10Gb cards. Not overloaded, all > graphs show it's got plenty of RAM/CPU spare at all times) > * vmxnet3 drivers > * ipfw (very small ruleset, basically just a divert rule) > * suricata, in ipfw divert mode > > I'm having a couple of major issues. > > The first is that every so often, even with relatively little traffic, th= e > load on the box suddenly spikes and pings to a neighbouring router (via t= he > divert rule) go from <1ms to >300ms. Generally this resolves itself after= a > few minutes, although last night it went on for an hour until I restarted > ipfw and suricata. > =E2=80=8Btypical problems with divert sockets are 1. if the interface has offloads (checksums, tso, etc.) =E2=80=8B =E2=80=8B enabled, packets may be moved up and down with invalid checksums resulting in all sort of odd behaviours 2. likewise if there are huge segments (virtual nics often pass up and down up to 64k in one shot) the divert socket is unable to handle them and either truncates or drops disabling all accelerations and setting the mtu to 1500 or so usually fixes the above two=E2=80=8B. 3. divert probably loses important context on the packets (e.g. incoming or outgoing interface) so when traffic is reinjected bad things occur 4. performancewise, moving traffic back and forth to userspace is a bad idea. For the latter two, you might be better off using netmap on vmxnet3 (in emulated mode, also disabling offloads), and if i remember well a couple of years ago there were efforts to use =E2=80=8Bsuricata on top of netmap. Worst case, you can just use the netmap-enabled libpcap. =E2=80=8B cheers luigi The second is that if I do a large download, eg a FreeBSD ISO, the download > usually hangs somewhere between 5MB and 100MB through. I can see traffic > trying to get through on neighbouring routers, it's just the interface wi= th > the divert to suricata where they disappear into a black hole. The > connection speed is around 50Mb, btw. > > Now it's possible it's suricata being weird, but there's nothing untoward > in its events and stats logs, and if I replay the traffic from a pcap fil= e > then suricata processes everything fine (a pcap taken over a 90s period > during a slowdown is processed in under a second). So my guess is that if > suricata takes slightly longer than normal to process a packet, something > in the networking or ipfw divert system is tripping itself up. Maybe a > queue is filling up? > > I've set net.inet.ip.fw.dyn_buckets=3D16384, and done an ipfw flush, but > net.inet.ip.fw.curr_dyn_buckets is stubbornly sticking at 256: have I don= e > something wrong? Other tunables I've set are: > > kern.random.sys.harvest.ethernet=3D0 > kern.random.sys.harvest.point_to_point=3D0 > kern.random.sys.harvest.interrupt=3D0 > kern.ipc.soacceptqueue=3D1024 > > Can anyone suggest either tests to see what might be going wrong, or > tunables to help things run smoother? Both myself and a colleague have us= ed > FreeBSD for over 15 years, and never quite seen anything like it. > > Many thanks, > > Oliver. > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------