From owner-freebsd-net@FreeBSD.ORG Fri Apr 18 17:56:15 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F9511065672 for ; Fri, 18 Apr 2008 17:56:15 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.28]) by mx1.freebsd.org (Postfix) with ESMTP id 008F98FC20 for ; Fri, 18 Apr 2008 17:56:14 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: by yw-out-2324.google.com with SMTP id 2so402268ywt.13 for ; Fri, 18 Apr 2008 10:56:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=2mHl8zAA6IZdpeqeGSmWJqU9QSNHMuX+tbmnnmg4WHE=; b=ghHgf+iZoRTrPLmqdlddfh2zyS2EcYYfiqowqaTIkWzH9bjC4y5LOsruMurvhwy65z4xFgovFBlV+/ShuC7r9ExeIQzoGWhHjdCaIAORcsImzmNADAhiyEjXJSYGc0etaSD0JOrZabgpXAkkY2oheWkQQRcqkzdYridtwB0OxLY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=KHaQpGwkWe3mBNKTNsYfJJ6W/ptPh4PRbb280Zhou8kqBBPALSe8Bhcjd6HrXbbW8dHoHWn58YWp1WtWy4WAfuN6+IDmwiP5jOnf6r8vDWZPWMOXiZ9IdnlE0MA7dTgLFbBwugkeRxKLinjLpRfC9KPcXNrDlT1Uv8wpgaeQmdA= Received: by 10.150.215.16 with SMTP id n16mr4029474ybg.104.1208541366270; Fri, 18 Apr 2008 10:56:06 -0700 (PDT) Received: by 10.151.15.21 with HTTP; Fri, 18 Apr 2008 10:56:06 -0700 (PDT) Message-ID: <3c0b01820804181056u389d108ejb8bf81d0941ee66@mail.gmail.com> Date: Fri, 18 Apr 2008 13:56:06 -0400 From: "Alexander Sack" To: "Bruce Evans" In-Reply-To: <20080418093328.B50187@delplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3c0b01820804160929i76cc04fdy975929e2a04c0368@mail.gmail.com> <200804161456.20823.jkim@FreeBSD.org> <3c0b01820804161328m77704ca0g43077a9718d446d4@mail.gmail.com> <200804161654.22452.jkim@FreeBSD.org> <3c0b01820804161402u3aac4425n41172294ad33a667@mail.gmail.com> <20080417112329.G47027@delplex.bde.org> <3c0b01820804170643w6b771ce9jdfc2dc5b240922b@mail.gmail.com> <20080418093328.B50187@delplex.bde.org> Cc: freebsd-net@freebsd.org, Dieter , Jung-uk Kim Subject: Re: bge dropping packets issue X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Apr 2008 17:56:15 -0000 On Thu, Apr 17, 2008 at 9:13 PM, Bruce Evans wrote: > > So the million dollar question: Do you believe that if I disable > > DEVICE_POLLING and use interrupt driven I/O, I could achieve zero > > packet loss over a 1Gbps link? This is the main issue I need to solve > > (solve means either no its not really achievable without a heavy > > rewrite of the driver OR yes it is with some tuning). If the answer > > is yes, then I have to understand the impact on the system in general. > > I just want to be sure I'm on a viable path through the BGE maze! > > > > I think you can get close enough if the bus and memory and CPU(s) > permit and you don't need to get too close to the theoretical limits. Thanks again Bruce for the detailed response. Here are my results: Good news: Well after fiddling around with it, it seems if I bump the number of rx_bds to 512, disable polling, and use net.isr.direct=1, bge does not drop packets anymore (as verified by assigning a counter within bge_ticks() when a packet is dropped as read by the hardware registers). What's interesting is that there is also an outOfRxBDs register you can read if you suspect chain starvation which I discovered after looking at the Linux driver's more complete stat structure. Bad news: Packets still get dropped but this time by BPF. It seems I pushed the problem upstream (in terms of the stack). The user land software listening in this instance is using BPF. I guess my next adventure is to understand how much can BPF take before dumping packets due to lack of buffer space - currently net.bpf.bufsize is 1048576 which is the maxbufsize. Is this common place for BPF to drop packets? (forgive me I have not searched the mailing list as I just confirmed these results by instrumenting BPF). Could I raise the maxbufsize and still operate safely? (I do have 8GB on a 64-bit system). Almost but no cigar.... -aps