From owner-freebsd-net@FreeBSD.ORG Tue Sep 23 16:36:04 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B606F58; Tue, 23 Sep 2014 16:36:04 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 083C8800; Tue, 23 Sep 2014 16:36:04 +0000 (UTC) Received: from [2a02:6b8:0:401:222:4dff:fe50:cd2f] (helo=ptichko.yndx.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1XWP5O-0001FO-DI; Tue, 23 Sep 2014 16:20:50 +0400 Message-ID: <5421A138.90803@FreeBSD.org> Date: Tue, 23 Sep 2014 20:35:04 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: How do I balance bandwidth over several virtual NICs? References: <5421310C.5010406@FreeBSD.org> <54218EF4.6090102@FreeBSD.org> <20140923154610.GD84074@onelab2.iet.unipi.it> In-Reply-To: <20140923154610.GD84074@onelab2.iet.unipi.it> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-net@freebsd.org" , Adrian Chadd , Elof Ofel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 16:36:04 -0000 On 23.09.2014 19:46, Luigi Rizzo wrote: > On Tue, Sep 23, 2014 at 07:17:08PM +0400, Alexander V. Chernikov wrote: >> On 23.09.2014 18:44, Luigi Rizzo wrote: > ... >> However, in addition to non-symmetric RSS (which is hopefully being >> addressed), there is another >> usual "producer - multuple consumers" problem: one snort process can >> start process packets very slowly, or hang, or crash. >> In that case host RX ring is getting full, NIC fails to push packets to >> given queue and start storing them inside >> its skid buffer (512k for Niantic afair). After that buffer becomes full >> traffic and all processing stops. > interesting. Actually, scary! > Do you have any reference to the data sheets documenting > that behaviour ? I have indeed received reports saying something Quoting latest 82599 datasheet: Flow control: 3.7.7.3.4 Packet Buffer Size .. Suggested buffer size (FC on): 512K Suggested buffer size (FC off, jumbo): 40Kb 7.1 Receive Functionality Packet reception consists of: • Recognizing the presence of a packet on the wire • Performing address filtering • DMA queue assignment • Storing the packet in the receive data FIFO ----------------------------------- ^^^^^^^^^^^ • Transferring the data to assigned receive queues in host memory • Updating the state of a receive descriptor. 7.1.9.1 Low Receive Descriptors Threshold The number of usable (free) descriptors for the hardware is the distance between the Tail and Head registers. When the tail reaches the head, there are no free descriptors and further packets might be either dropped or block the receive FIFO. In order to avoid this situation, the 82599 might generate a low latency interrupt (associated to the relevant > similar but always suspected user errors. > The fact that a starved queue can consume the entire internal > buffer seems a really bad bug. > > At least you can overcome this one by having the demux done > in software. Well, yes. Despite the fact that 82599 can handle huge (65k) RX queues, this does not help much unless there is some knob to prevent HOL. > > cheers > luigi >