From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 08:53:21 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D9E9960; Wed, 21 Nov 2012 08:53:21 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9A21A8FC0C; Wed, 21 Nov 2012 08:53:20 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so6520250lah.13 for ; Wed, 21 Nov 2012 00:53:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=YupkuT8hrD7LDNuloj7sGQggq6PCNn24n9sijcpejt0=; b=Ja1HAVWx+arqZlD/OlkJzU5tiOUSNFSVivUX+VONl9WWwLYG5aE4AIJkwXE9l7kj+c 68UpFJS9k9FGl07mZ5dukmtS3Vxj3xxQRKsJV4rmH8Q7fwWlQM+bwnaLFoB9rA2PIbfy Ev33GHkle3Zwohwij5M+sqBHFLxKuDsmLPsoWzKJaLuQOuGL1XdMxkaITe++DwRVQd9/ 3vJrOh4A3AxZE7KWmfhWZ89SC10iCyofjtWC4R49LUIu/HkGWFFdwtof0TuzFOzpXLGg Y9ePi7YS7UogxzK0tdMImIkIy6LcEGy2GVI9Jwy6ylfog2m6K4zSjJb1QJ5QKFUmq3rw BScA== MIME-Version: 1.0 Received: by 10.152.144.164 with SMTP id sn4mr11590636lab.57.1353487999147; Wed, 21 Nov 2012 00:53:19 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.112.20.131 with HTTP; Wed, 21 Nov 2012 00:53:18 -0800 (PST) In-Reply-To: <50AC92A0.2000406@freebsd.org> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <50AC8393.3060001@freebsd.org> <50AC92A0.2000406@freebsd.org> Date: Wed, 21 Nov 2012 00:53:18 -0800 X-Google-Sender-Auth: 8SnuSXM9ORiJDucels07LsVt6JU Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Luigi Rizzo To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Barney Cordoba , Adrian Chadd , khatfield@socllc.net, "freebsd-net@freebsd.org" , Jim Thompson , Alfred Perlstein X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 08:53:21 -0000 On Wed, Nov 21, 2012 at 12:36 AM, Andre Oppermann wrote: > On 21.11.2012 09:04, Luigi Rizzo wrote: > >> On Tue, Nov 20, 2012 at 11:32 PM, Andre Oppermann >> wrote: >> >>> ... >> >> very cool. this seems similar to NAPI. >> > > I've heard about NAPI but haven't looked at it. So I'm not sure how it > have a look at some of the intel nic drivers in linux, they are very similar to the bsd ones so you can easily compare. NAPI is basically the same thing that you describe, there is no special logic and the only kernel support is (i believe) some device independent code that probably you'll end up writing too after the first 2-3 drivers, to avoid boring replications. works internally. In this case no special logic or kernel API support > is required. Every driver can be converted and immediately use it. > > > The only adjustment i'd suggest to your scheme, if possible, is to add >> some control (as it existed in the old polling architecture) to make sure >> that userspace is not starved by the ithreads and other kernel tasks >> (otherwise you can have livelock, as it happens with NAPI). >> I am afraid that simple priorities do not work, you either need some >> kind of fair scheduler, or put some hard limit on the cpu fraction used >> by the kernel tasks when there are userspace processes around. >> > > That's the quantum stuff I talked about. When the ithread has used up > its CPU quantum it gives up the CPU but schedules itself again at the > same time. Also contrary to its name the ithread does not run in > interrupt context but as a normal kernel thread with elevated priority. > it is the elevated priority that worries me, because it has potential to preempt userspace and cause livelock. Again, lacking a proper fair process scheduler, i think the only reliable way is to try and track the (approximate) cpu cycles consumed overall by the various ithreads over, say, the current tick, and once you go above the threshold drop the priority of those threads just above IDLEPRI. Then when the next tick arrives you raise the priorities again. Compared to when i did polling in 2000 (when i wanted to support the i486 soekris), now we can probably assume that a cheap timecounter is available on all architectures (even ARM and MIPS should have some tsc-like thing, right ?) and the cycles used vs cycles per tick can be accounted with a relatively fine grain. cheers luigi