From owner-freebsd-net@FreeBSD.ORG Mon Dec 20 16:39:40 2010 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D53D51065670 for ; Mon, 20 Dec 2010 16:39:40 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id 371078FC17 for ; Mon, 20 Dec 2010 16:39:39 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id oBKGdagG024961 for ; Mon, 20 Dec 2010 22:39:37 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D0F86C3.9020808@rdtc.ru> Date: Mon, 20 Dec 2010 22:39:31 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.1.10) Gecko/20100712 Thunderbird/3.0.5 MIME-Version: 1.0 To: "net@freebsd.org" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Subject: igb+lagg: poor input performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2010 16:39:40 -0000 Hi! I observe some kind of undetected bottle-neck with input traffic processing when traffic goes in through both ports of dual-port 82576-based card grouped to lagg1 (lacp mode). Ports are connected to Cisco 7606 and there are over 800 vlans created on top of lagg1. No output errors or buffer overflows at Cisco side. Cisco distributes traffic over ports just fine. Vlans carry PPPoE frames, I use mpd55 for PPPoE. As number of connected users grows, input traffic and pps grow up to 126Kpps and 560Mbps on lagg1 for 1500 active PPPoE links. Then mrtg draws nearly horizontal lines despite of growing users number. There is no congestion on uplink (lagg0). This system has 4-core Xeon E5507 at 2.27Ghz, hyper-threading disabled. All cores are nearly evenly loaded at level 60-65% only (almost interrupts). Horizontal lines are seen for distinct processor core's load, input traffic for lagg1 and pps for lagg1. I expect much more input traffic. I use 8.2-PRERELEASE/amd64 with latest igb(8) driver version 2.0.7. It has 4G RAM, over 3GB are free. sysctl net.inet.ip.intr_queue_drops shows zero value. /boot/loader.conf: vm.kmem_size=3G # for igb(4) hw.igb.rxd=4096 hw.igb.txd=4096 # for lagg(4) net.link.ifqmaxlen=10240 # for rtsock net.route.netisr_maxqlen=4096 # for ??? net.isr.defaultqlimit=4096 /etc/sysctl.conf: # netisr input queue size net.inet.ip.intr_queue_maxlen=10240 net.inet.ip.fastforwarding=1 net.inet.ip.dummynet.pipe_slot_limit=1000 net.inet.ip.dummynet.io_fast=1 net.isr.direct=0 net.isr.direct_force=0 dev.igb.0.rx_processing_limit=4096 dev.igb.1.rx_processing_limit=4096 kern.ipc.nmbclusters=100000 kern.ipc.nmbjumbop=100000 kern.ipc.maxsockbuf=83886080 net.graph.maxdgram=8388608 net.graph.recvspace=8388608 Where should I dig first? Eugene Grosbein