From owner-freebsd-net@FreeBSD.ORG Tue Jun 21 19:12:00 2005 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5C71616A41C for ; Tue, 21 Jun 2005 19:12:00 +0000 (GMT) (envelope-from pieter@thedarkside.nl) Received: from mail.thelostparadise.com (129pc197.sshunet.nl [145.97.197.129]) by mx1.FreeBSD.org (Postfix) with ESMTP id F20F943D4C for ; Tue, 21 Jun 2005 19:11:59 +0000 (GMT) (envelope-from pieter@thedarkside.nl) Received: from [195.16.84.92] (92-unused.virt-ix.net [195.16.84.92]) by mail.thelostparadise.com (8.12.11/8.12.8) with ESMTP id j5LJBvxi045714; Tue, 21 Jun 2005 21:11:58 +0200 (CEST) (envelope-from pieter@thedarkside.nl) Message-ID: <42B8667D.9020504@thedarkside.nl> Date: Tue, 21 Jun 2005 21:11:57 +0200 From: Pieter de Boer User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050527) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Luigi Rizzo References: <42B722EF.2090203@thedarkside.nl> <20050620135044.B35720@xorpc.icir.org> <42B731CD.1040104@thedarkside.nl> <20050621075247.D63359@xorpc.icir.org> <42B84AC8.7050802@thedarkside.nl> <20050621102954.A66904@xorpc.icir.org> In-Reply-To: <20050621102954.A66904@xorpc.icir.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Issues with a Large Fat pipe Network simulation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jun 2005 19:12:00 -0000 Luigi Rizzo wrote: > oh yes one thing... you are using 'via foo0' in your rule, > which means the packet is intercepted both in the input and > output path, which causes further contention on the queues. Well, when using 'ip from client to server recv em0', packets get matched twice. When I set some latency on that pipe, the packets incur double the latency I set. With 'via em0' this isn't the case. I tried it out, but didn't seem to make much difference. > also you can set the queue size in kbytes, and can probably go as high > as 1000kb. maybe this helps too. Doesn't seem to do much either. > I am pretty sure there is some issue there, also related to some > timing issues and tcp window opening mode (slow start vs. linear) I went to see if there were any sysctl's I could tune a bit. I found these: net.inet.ip.intr_queue_maxlen: 50 net.inet.ip.intr_queue_drops: 382136 I don't like drops. So I set intr_queue_maxlen to 400, and -poof-, the speed went up to around 700mbit/s. Still not as fast as it was with 64KB send/recv spaces, but it's a huge improvement nonetheless. I guess we probably should tune a bit more until we're confident that the middle-box behaves correctly, before adding things like latency and packet-loss :) Thanks for the advice! If you know other settings to tune on the dummynetting host, I'd very much like to hear them. I'm pondering about polling (which means we can't do SMP on the dummynet system, but it's only pushing packets, so that shouldn't matter too much). At least we have somewhat more info to work with now :) -- Pieter