From owner-freebsd-net@freebsd.org Wed Jul 1 13:20:50 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E10E59905FA for ; Wed, 1 Jul 2015 13:20:50 +0000 (UTC) (envelope-from oliver@watershed.co.uk) Received: from babbage.watershed.co.uk (babbage.watershed.co.uk [195.10.250.253]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 830C21EBF for ; Wed, 1 Jul 2015 13:20:50 +0000 (UTC) (envelope-from oliver@watershed.co.uk) Received: from oliver.local.watershed.co.uk (Oliver.local.watershed.co.uk [192.168.128.78]) (authenticated bits=0) by babbage.watershed.co.uk (8.15.1/8.15.1) with ESMTPSA id t61DF9H7083485 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Wed, 1 Jul 2015 14:15:10 +0100 (BST) (envelope-from oliver@watershed.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=watershed.co.uk; s=main; t=1435756510; bh=QbbMej0MdJNneFeH67YeIr+chEU/OEfmofINJ9/ImZ4=; h=From:Subject:Date:To; b=dbkOTqwB1LYgFjILLQQuC3N0acLBwSzVhGGuKEkgguOZntutny+Sy1xAsPvj/89hz 6bRZGJnHZ8WrN/wi2Jlt4sQzm5Hz2W4/jKSZivk20F3+eVRtEqE9K60lKhpiP9qK7L 8fkDTWJ0tIsd6GtYZ7t4i/mC95UmWhnWcve/IhFQf2SYePLKj5zlW2I3G3g/gcB6tI /kF9546ekODjaC6RYqBo33/gqE2jmqgHf15m7UfCYJz4g+eYeRHLAC4kqsrSNicNC2 7HI+CAsmA0GicryaDbpi6urZcnreHr7jMWDZS0+J+FSe6w+HtJ5ID6tKZyVSSVd++9 uw1PF7nBD2yTw== From: Oliver Humpage Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: IPFW divert and suricata Message-Id: Date: Wed, 1 Jul 2015 14:15:09 +0100 To: freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) X-Mailer: Apple Mail (2.1510) X-Virus-Scanned: by Sophos X-Scanned-By: MIMEDefang 2.74 on 195.10.250.253 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jul 2015 13:20:51 -0000 Hello, I hope this is a good list to post this on, I have a feeling the = solution is somewhere obscure in the networking layer. I've set up an IPS system, using: * FreeBSD 10.1 (guest OS, plenty of RAM/CPU) * ESXi 5.5 (host OS, using Intel X520 10Gb cards. Not overloaded, all = graphs show it's got plenty of RAM/CPU spare at all times) * vmxnet3 drivers * ipfw (very small ruleset, basically just a divert rule) * suricata, in ipfw divert mode I'm having a couple of major issues.=20 The first is that every so often, even with relatively little traffic, = the load on the box suddenly spikes and pings to a neighbouring router = (via the divert rule) go from <1ms to >300ms. Generally this resolves = itself after a few minutes, although last night it went on for an hour = until I restarted ipfw and suricata. The second is that if I do a large download, eg a FreeBSD ISO, the = download usually hangs somewhere between 5MB and 100MB through. I can = see traffic trying to get through on neighbouring routers, it's just the = interface with the divert to suricata where they disappear into a black = hole. The connection speed is around 50Mb, btw. Now it's possible it's suricata being weird, but there's nothing = untoward in its events and stats logs, and if I replay the traffic from = a pcap file then suricata processes everything fine (a pcap taken over a = 90s period during a slowdown is processed in under a second). So my = guess is that if suricata takes slightly longer than normal to process a = packet, something in the networking or ipfw divert system is tripping = itself up. Maybe a queue is filling up? I've set net.inet.ip.fw.dyn_buckets=3D16384, and done an ipfw flush, but = net.inet.ip.fw.curr_dyn_buckets is stubbornly sticking at 256: have I = done something wrong? Other tunables I've set are: kern.random.sys.harvest.ethernet=3D0 kern.random.sys.harvest.point_to_point=3D0 kern.random.sys.harvest.interrupt=3D0=20 kern.ipc.soacceptqueue=3D1024=20 Can anyone suggest either tests to see what might be going wrong, or = tunables to help things run smoother? Both myself and a colleague have = used FreeBSD for over 15 years, and never quite seen anything like it. Many thanks, Oliver.