From owner-freebsd-net@FreeBSD.ORG Tue May 8 14:10:00 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 408CF106564A for ; Tue, 8 May 2012 14:10:00 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9C8FE8FC15 for ; Tue, 8 May 2012 14:09:59 +0000 (UTC) Received: by lagv3 with SMTP id v3so5846002lag.13 for ; Tue, 08 May 2012 07:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type; bh=MljTV6skMJ7NO4w1zC6wodokSU7dvOzVSQ3ZdF/X3KU=; b=VleRyY8aAHICl3w/TMYUAPFZ4pJjXakjR1bBr2udlDUHzRFiu3Kc/YvsIeYxdxeCgp guclLKrwNLpkMRXMybtjQqk6uRki345xT3GI6zhV04wTQVTDX/bR2ozJXJ61Rq1+H+p+ v3Je3z4OhoHBq9gbjPqq96H5I/ZzD7WsmRnQlEQAQrUkwNsnXnZHkRIVahjenhlgcobm H/EuHyS006D9cfv7IWum9AwU/87sSgb0sW9jZhwDdvuj3158GMIXOnZ7JowBqlK+M0ET dozBwUjB7Fec63yWQi6TSURRvIIGnaP7SVGp/cIBUIzWX+TgEv+nyXtbZ6dkUO7klV/T XGsQ== Received: by 10.112.87.170 with SMTP id az10mr3760269lbb.53.1336486198328; Tue, 08 May 2012 07:09:58 -0700 (PDT) Received: from [192.168.0.47] (buddy.uch.net. [193.108.248.29]) by mx.google.com with ESMTPS id oi3sm22531623lab.12.2012.05.08.07.09.55 (version=SSLv3 cipher=OTHER); Tue, 08 May 2012 07:09:56 -0700 (PDT) Message-ID: <4FA92931.7040700@gmail.com> Date: Tue, 08 May 2012 17:09:53 +0300 From: Sergey Saley User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> <1319527328469-4935272.post@n5.nabble.com> <1319530877390-4935427.post@n5.nabble.com> In-Reply-To: <1319530877390-4935427.post@n5.nabble.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 May 2012 14:10:00 -0000 25.10.2011 11:21, Sergey Saley ???????: > Jack Vogel wrote: >> On Tue, Oct 25, 2011 at 12:22 AM, Sergey Saley<sergeysaley@>wrote: >> >>> Ryan Stone-2 wrote: >>>> On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley<sergeysaley@> >>> wrote: >>>>> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >>>>> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >>>>> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. >>>> PPPoE is your problem. The Intel cards can't load-balance PPPoE >>>> traffic, so everything goes to one queue. It may be possible to write >>>> a netgraph module to load-balance the traffic across your CPUs. >>>> >>> OK, thank You for explanation. >>> And what about the large number of interrupts? >>> As for me, it's too much... >>> irq256: ix0:que 0 240536944 6132 >>> irq257: ix0:que 1 89090444 2271 >>> irq258: ix0:que 2 93222085 2376 >>> irq259: ix0:que 3 89435179 2280 >>> irq260: ix0:link 1 0 >>> irq261: ix1:que 0 269468769 6870 >>> irq262: ix1:que 1 110974 2 >>> irq263: ix1:que 2 434214 11 >>> irq264: ix1:que 3 112281 2 >>> irq265: ix1:link 1 0 >>> >>> >> How do you decide its 'too much' ? It may be that with your traffic you >> end >> up >> not being able to use offloads, just thinking. Its not like the hardware >> just "makes >> it up", it interrupts on the last descriptor of a packet which has the RS >> bit set. >> With TSO you will get larger chunks of data and thus less interrupts but >> your >> traffic probably doesn't qualify for it. >> > It's easy. I have several servers with a similar task and load. > About 30K pps, about 500-600M traffic, about 600-700 pppoe connections. > One difference - em > Here is a typical vmstat -i > > point06# vmstat -i > interrupt total rate > irq17: atapci0 6173367 0 > cpu0: timer 3904389748 465 > irq256: em0 3754877950 447 > irq257: em1 2962728160 352 > cpu2: timer 3904389720 465 > cpu1: timer 3904389720 465 > cpu3: timer 3904389721 465 > Total 22341338386 2661 > > point05# vmstat -i > interrupt total rate > irq14: ata0 35 0 > irq19: atapci1 8323568 0 > cpu0: timer 3905440143 465 > irq256: em0 3870403571 461 > irq257: em1 1541695487 183 > cpu1: timer 3905439895 465 > cpu3: timer 3905439895 465 > cpu2: timer 3905439895 465 > Total 21042182489 2506 > > point04# vmstat -i > interrupt total rate > irq19: atapci0 6047874 0 > cpu0: timer 3901683760 464 > irq256: em0 823774953 98 > irq257: em1 1340659093 159 > cpu1: timer 3901683730 464 > cpu2: timer 3901683730 464 > cpu3: timer 3901683730 464 > Total 17777216870 2117 > > BTW, maybe there is a possibility to make a traffic separation per several queues by vlan tag? That would be a partial solution...