Date: Thu, 02 Jul 2015 12:13:21 +0800 From: Julian Elischer <julian@freebsd.org> To: Luigi Rizzo <rizzo@iet.unipi.it>, Oliver Humpage <oliver@watershed.co.uk> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: IPFW divert and suricata Message-ID: <5594BA61.2070501@freebsd.org> In-Reply-To: <CA%2BhQ2%2BjZGbBMT4pD8GD4_4nuX9jEE4NCOSykmydtYCgy=vK-sA@mail.gmail.com> References: <D632FEB9-4C62-451E-B2F6-333B7EDAE7C9@watershed.co.uk> <CA%2BhQ2%2BjZGbBMT4pD8GD4_4nuX9jEE4NCOSykmydtYCgy=vK-sA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 7/1/15 10:31 PM, Luigi Rizzo wrote: > On Wed, Jul 1, 2015 at 3:15 PM, Oliver Humpage <oliver@watershed.co.uk> > wrote: > >> Hello, >> >> I hope this is a good list to post this on, I have a feeling the solution >> is somewhere obscure in the networking layer. >> >> I've set up an IPS system, using: >> >> * FreeBSD 10.1 (guest OS, plenty of RAM/CPU) >> * ESXi 5.5 (host OS, using Intel X520 10Gb cards. Not overloaded, all >> graphs show it's got plenty of RAM/CPU spare at all times) >> * vmxnet3 drivers >> * ipfw (very small ruleset, basically just a divert rule) >> * suricata, in ipfw divert mode >> >> I'm having a couple of major issues. >> >> The first is that every so often, even with relatively little traffic, the >> load on the box suddenly spikes and pings to a neighbouring router (via the >> divert rule) go from <1ms to >300ms. Generally this resolves itself after a >> few minutes, although last night it went on for an hour until I restarted >> ipfw and suricata. >> > typical problems with divert sockets are > 1. if the interface has offloads (checksums, tso, etc.) > > enabled, packets may be > moved up and down with invalid checksums resulting in > all sort of odd behaviours > 2. likewise if there are huge segments (virtual nics often > pass up and down up to 64k in one shot) the divert socket > is unable to handle them and either truncates or drops > > disabling all accelerations and setting the mtu to 1500 or so > usually fixes the above two. > > 3. divert probably loses important context on the packets > (e.g. incoming or outgoing interface) so when traffic is > reinjected bad things occur actually this is one that it tries to keep. the interface name is in the sockaddr (ipv4) hidden at the end. If you use the same sockaddr that the packet arrived with, you will pass the interface back again.. (actually in the man page I think). > 4. performancewise, moving traffic back and forth to userspace > is a bad idea. > > For the latter two, you might be better off using netmap > on vmxnet3 (in emulated mode, also disabling offloads), > and if i remember well a > couple of years ago there were > efforts to use > suricata on top of netmap. > Worst case, you can just use the netmap-enabled libpcap. > > cheers > luigi > > The second is that if I do a large download, eg a FreeBSD ISO, the download >> usually hangs somewhere between 5MB and 100MB through. I can see traffic >> trying to get through on neighbouring routers, it's just the interface with >> the divert to suricata where they disappear into a black hole. The >> connection speed is around 50Mb, btw. >> >> Now it's possible it's suricata being weird, but there's nothing untoward >> in its events and stats logs, and if I replay the traffic from a pcap file >> then suricata processes everything fine (a pcap taken over a 90s period >> during a slowdown is processed in under a second). So my guess is that if >> suricata takes slightly longer than normal to process a packet, something >> in the networking or ipfw divert system is tripping itself up. Maybe a >> queue is filling up? >> >> I've set net.inet.ip.fw.dyn_buckets=16384, and done an ipfw flush, but >> net.inet.ip.fw.curr_dyn_buckets is stubbornly sticking at 256: have I done >> something wrong? Other tunables I've set are: >> >> kern.random.sys.harvest.ethernet=0 >> kern.random.sys.harvest.point_to_point=0 >> kern.random.sys.harvest.interrupt=0 >> kern.ipc.soacceptqueue=1024 >> >> Can anyone suggest either tests to see what might be going wrong, or >> tunables to help things run smoother? Both myself and a colleague have used >> FreeBSD for over 15 years, and never quite seen anything like it. >> >> Many thanks, >> >> Oliver. >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5594BA61.2070501>