Date: Wed, 1 Jul 2015 16:31:49 +0200 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Oliver Humpage <oliver@watershed.co.uk> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: IPFW divert and suricata Message-ID: <CA%2BhQ2%2BjZGbBMT4pD8GD4_4nuX9jEE4NCOSykmydtYCgy=vK-sA@mail.gmail.com> In-Reply-To: <D632FEB9-4C62-451E-B2F6-333B7EDAE7C9@watershed.co.uk> References: <D632FEB9-4C62-451E-B2F6-333B7EDAE7C9@watershed.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jul 1, 2015 at 3:15 PM, Oliver Humpage <oliver@watershed.co.uk> wrote: > > Hello, > > I hope this is a good list to post this on, I have a feeling the solution > is somewhere obscure in the networking layer. > > I've set up an IPS system, using: > > * FreeBSD 10.1 (guest OS, plenty of RAM/CPU) > * ESXi 5.5 (host OS, using Intel X520 10Gb cards. Not overloaded, all > graphs show it's got plenty of RAM/CPU spare at all times) > * vmxnet3 drivers > * ipfw (very small ruleset, basically just a divert rule) > * suricata, in ipfw divert mode > > I'm having a couple of major issues. > > The first is that every so often, even with relatively little traffic, th= e > load on the box suddenly spikes and pings to a neighbouring router (via t= he > divert rule) go from <1ms to >300ms. Generally this resolves itself after= a > few minutes, although last night it went on for an hour until I restarted > ipfw and suricata. > =E2=80=8Btypical problems with divert sockets are 1. if the interface has offloads (checksums, tso, etc.) =E2=80=8B =E2=80=8B enabled, packets may be moved up and down with invalid checksums resulting in all sort of odd behaviours 2. likewise if there are huge segments (virtual nics often pass up and down up to 64k in one shot) the divert socket is unable to handle them and either truncates or drops disabling all accelerations and setting the mtu to 1500 or so usually fixes the above two=E2=80=8B. 3. divert probably loses important context on the packets (e.g. incoming or outgoing interface) so when traffic is reinjected bad things occur 4. performancewise, moving traffic back and forth to userspace is a bad idea. For the latter two, you might be better off using netmap on vmxnet3 (in emulated mode, also disabling offloads), and if i remember well a couple of years ago there were efforts to use =E2=80=8Bsuricata on top of netmap. Worst case, you can just use the netmap-enabled libpcap. =E2=80=8B cheers luigi The second is that if I do a large download, eg a FreeBSD ISO, the download > usually hangs somewhere between 5MB and 100MB through. I can see traffic > trying to get through on neighbouring routers, it's just the interface wi= th > the divert to suricata where they disappear into a black hole. The > connection speed is around 50Mb, btw. > > Now it's possible it's suricata being weird, but there's nothing untoward > in its events and stats logs, and if I replay the traffic from a pcap fil= e > then suricata processes everything fine (a pcap taken over a 90s period > during a slowdown is processed in under a second). So my guess is that if > suricata takes slightly longer than normal to process a packet, something > in the networking or ipfw divert system is tripping itself up. Maybe a > queue is filling up? > > I've set net.inet.ip.fw.dyn_buckets=3D16384, and done an ipfw flush, but > net.inet.ip.fw.curr_dyn_buckets is stubbornly sticking at 256: have I don= e > something wrong? Other tunables I've set are: > > kern.random.sys.harvest.ethernet=3D0 > kern.random.sys.harvest.point_to_point=3D0 > kern.random.sys.harvest.interrupt=3D0 > kern.ipc.soacceptqueue=3D1024 > > Can anyone suggest either tests to see what might be going wrong, or > tunables to help things run smoother? Both myself and a colleague have us= ed > FreeBSD for over 15 years, and never quite seen anything like it. > > Many thanks, > > Oliver. > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BhQ2%2BjZGbBMT4pD8GD4_4nuX9jEE4NCOSykmydtYCgy=vK-sA>