From owner-freebsd-net@FreeBSD.ORG Wed Apr 29 06:37:23 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 867DF49C for ; Wed, 29 Apr 2015 06:37:23 +0000 (UTC) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F51D10B4 for ; Wed, 29 Apr 2015 06:37:23 +0000 (UTC) Received: by igblo3 with SMTP id lo3so39828227igb.0 for ; Tue, 28 Apr 2015 23:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=pfoRuQjHnb7EJ9LTM59QPJwrdcAe9Vq3zrH5v98z2bY=; b=VfV6Hp1zNIjZUFAWvBnHqcKGzLykYEMIS2K3n78bR2uSAgS00aK5p6WAmM5Swzrcxj lTErwcTmoRKKcCTLgaTwaPKJWfV/kmD603kB4AWk+27SN/4guE+HVvRCuktFHAg8BRR0 TVGoJGquEREt/amWggX6CN5b9VeuDQsIOwjN+BC7/ndXGaPsjXCeUPYrt//u4AtRnnqb esWm2peaHfP9gACoj0SF9fBz8asarZOpQDuiMzBaw6KF5ef7VWWYqO9NC9pw3FjhUrq8 yJ2ddNyVp5iyQyk7KF3kxrOoOxxPmBr23gV6Kkpg8Sne7cHPahYg7/ZAjpeW4sxXCYwX kEUw== MIME-Version: 1.0 X-Received: by 10.42.20.197 with SMTP id h5mr1833898icb.22.1430289442740; Tue, 28 Apr 2015 23:37:22 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.38.133 with HTTP; Tue, 28 Apr 2015 23:37:22 -0700 (PDT) In-Reply-To: <21824.26416.855441.21454@hergotha.csail.mit.edu> References: <4281350517-9417@kerio.tuxis.nl> <137094161.27589033.1430255162390.JavaMail.root@uoguelph.ca> <21824.26416.855441.21454@hergotha.csail.mit.edu> Date: Tue, 28 Apr 2015 23:37:22 -0700 X-Google-Sender-Auth: 9xv6cnHIUaOHWW4RFjcbckj3RXY Message-ID: Subject: Re: Frequent hickups on the networking layer From: Adrian Chadd To: Garrett Wollman Cc: Rick Macklem , FreeBSD Net , Mark Schouten Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Apr 2015 06:37:23 -0000 I've spoken to more than one company about this stuff and their answers are all the same: "we ignore the freebsd allocator, allocate a very large chunk of memory at boot, tell the VM it plainly just doesn't exist, and abuse it via the direct map." That gets around a lot of things, including the "oh how can we get 9k allocations if we can't find contiguous memory/KVA/either" problem - you just treat it as an array of 9k pages (or I'm guessing much larger - as you said, like ~ 64k), and allocate that way. That way there's no fragmentation to worry about - everything's just using a custom slab allocator for these large allocation sizes. It's kind of tempting to suggest freebsd support such a thing, as I can see increasing requirements for specialised applications that want this. One of the things that makes netmap so nice is it 100% avoids the allocators in the hot path - it grabs a big chunk of memory and allocates slots out of that via a bitmap and index values. -adrian