From owner-freebsd-net@FreeBSD.ORG Tue Jun 21 19:36:21 2005 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1033D16A41C for ; Tue, 21 Jun 2005 19:36:21 +0000 (GMT) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id EA9BC43D4C for ; Tue, 21 Jun 2005 19:36:20 +0000 (GMT) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.12.11) with ESMTP id j5LJaI8n075530; Tue, 21 Jun 2005 12:36:18 -0700 (PDT) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id j5LJaIJL075529; Tue, 21 Jun 2005 12:36:18 -0700 (PDT) (envelope-from rizzo) Date: Tue, 21 Jun 2005 12:36:18 -0700 From: Luigi Rizzo To: Pieter de Boer Message-ID: <20050621123618.A75484@xorpc.icir.org> References: <42B722EF.2090203@thedarkside.nl> <20050620135044.B35720@xorpc.icir.org> <42B731CD.1040104@thedarkside.nl> <20050621075247.D63359@xorpc.icir.org> <42B84AC8.7050802@thedarkside.nl> <20050621102954.A66904@xorpc.icir.org> <42B8667D.9020504@thedarkside.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <42B8667D.9020504@thedarkside.nl>; from pieter@thedarkside.nl on Tue, Jun 21, 2005 at 09:11:57PM +0200 Cc: freebsd-net@freebsd.org Subject: Re: Issues with a Large Fat pipe Network simulation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jun 2005 19:36:21 -0000 On Tue, Jun 21, 2005 at 09:11:57PM +0200, Pieter de Boer wrote: > Luigi Rizzo wrote: > > > oh yes one thing... you are using 'via foo0' in your rule, > > which means the packet is intercepted both in the input and > > output path, which causes further contention on the queues. > Well, when using 'ip from client to server recv em0', packets get i said 'in recv em0' - you missed the 'in' keyword. > > I am pretty sure there is some issue there, also related to some > > timing issues and tcp window opening mode (slow start vs. linear) > I went to see if there were any sysctl's I could tune a bit. I found these: > net.inet.ip.intr_queue_maxlen: 50 > net.inet.ip.intr_queue_drops: 382136 > > I don't like drops. So I set intr_queue_maxlen to 400, and -poof-, the whoops... of course, i forgot that one too... which is not much of an issue if you use polling or bridging, that's why i forgot :) > speed went up to around 700mbit/s. Still not as fast as it was with 64KB > send/recv spaces, but it's a huge improvement nonetheless. > > I guess we probably should tune a bit more until we're confident that > the middle-box behaves correctly, before adding things like latency and > packet-loss :) > > Thanks for the advice! If you know other settings to tune on the > dummynetting host, I'd very much like to hear them. I'm pondering about > polling (which means we can't do SMP on the dummynet system, but it's > only pushing packets, so that shouldn't matter too much). At least we > have somewhat more info to work with now :) yes you should definitely enable polling if you can, and forget about smp - it's a router anyways, and multiple processors won't help. cheers luigi