Date: Wed, 1 Dec 2010 19:54:01 +0200 From: Eugene Perevyazko <john@dnepro.net> To: freebsd-net@freebsd.org Subject: Re: igb dual-port adapter 1200Mbps limit - what to tune? Message-ID: <20101201175401.GA69269@traktor.dnepro.net> In-Reply-To: <20101110110428.GA3505@traktor.dnepro.net> References: <20101110110428.GA3505@traktor.dnepro.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--k+w/mQv8wyuph6w0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Wed, Nov 10, 2010 at 01:04:28PM +0200, Eugene Perevyazko wrote:
>
> I have a router running RELENG_7 with two dual-port igbs - igb0 and igb1 are on
> 82575 on intel s5520ur mb and igb2 and igb3 are on 82576 on ET dual-port card.
> 82576 is in 8x slot.
> Main traffic flows from igb0+igb1 to igb2+igb3, less traffic goes back.
> There's no traffic flow in directions igb0 - igb1 and igb2 - igb3.
>
> There are vlans on all interfaces.
>
> igb0 and igb1 are outbound links.
> igb2 and igb3 are connected to switch.
> CPU is E5620@2.4GHz, 8 cores, irqs bound to different cores skipping HT ones.
> Tried 2 queues and 1 queue per iface, neither hitting cpu limit.
>
> The problem is that traffic through igb2+igb3 is limited at around 1200Mbps Tx
> while I was hoping for 1600-1800Mbps Tx.
>
I'd like to say that now this host is forwarding 1710 Mb/s @ 189 kpps in one direction + 411 Mb/s @ 140 kpps in reverse direction (30 minutes average).
The following changes were made:
- no lagg in use
- only one vlan left instead of 5 vlans
- added motherboard module 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)' with 2 em interfaces (only one used)
- 3 interfaces are connected to hosts with patchcords, not through switch
- hyperthreading turned off in bios
- igb driver patched to reduce irq rate and make it tunable (patch follows)
- hw.igb.num_queues=4, dev.igb.*.enable_aim=0, dev.igb.*.default_intrrate=4000 the last sysctl is added in patch)
- igb queues manually repinned to different cores
- ipfw rules minimized
igb driver used is 1.9.6/RELENG_7, igb.c version 1.3.2.12
Can't say for sure whether there is a key change or is it all changes summed up,
sorry.
--
Eugene Perevyazko
--k+w/mQv8wyuph6w0
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="igb.intrrate.patch"
--- if_igb.c.old 2010-09-27 21:34:04.000000000 +0300
+++ if_igb.c 2010-11-16 14:31:11.000000000 +0200
@@ -293,6 +293,17 @@ static int igb_enable_aim = TRUE;
TUNABLE_INT("hw.igb.enable_aim", &igb_enable_aim);
/*
+** default interrupt rate in ints/sec
+** applied to each queue separately
+** used if enable_aim=0
+*/
+static int igb_default_intrrate = IGB_INTS_PER_SEC;
+TUNABLE_INT("hw.igb.default_intrrate", &igb_default_intrrate);
+#undef IGB_DEFAULT_ITR
+/*#define IGB_DEFAULT_ITR 1000000000/(igb_default_intrrate * 256)*/
+#define IGB_DEFAULT_ITR ((1000000/igb_default_intrrate)<<2)
+
+/*
* MSIX should be the default for best performance,
* but this allows it to be forced off for testing.
*/
@@ -422,6 +433,11 @@ igb_attach(device_t dev)
OID_AUTO, "enable_aim", CTLTYPE_INT|CTLFLAG_RW,
&igb_enable_aim, 1, "Interrupt Moderation");
+ SYSCTL_ADD_INT(device_get_sysctl_ctx(dev),
+ SYSCTL_CHILDREN(device_get_sysctl_tree(dev)),
+ OID_AUTO, "default_intrrate", CTLTYPE_INT|CTLFLAG_RW,
+ &igb_default_intrrate, 1, "Default ints/sec (for each queue)");
+
callout_init_mtx(&adapter->timer, &adapter->core_mtx, 0);
/* Determine hardware and mac info */
@@ -2342,6 +2358,7 @@ igb_configure_queues(struct adapter *ada
}
/* Set the starting interrupt rate */
+ newitr &= 0x7FFC; /* Mask invalid bits */
if (hw->mac.type == e1000_82575)
newitr |= newitr << 16;
else
--k+w/mQv8wyuph6w0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20101201175401.GA69269>