From owner-freebsd-arch@FreeBSD.ORG  Mon Aug 18 19:45:27 2014
Return-Path: <owner-freebsd-arch@FreeBSD.ORG>
Delivered-To: arch@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 4966461B;
 Mon, 18 Aug 2014 19:45:27 +0000 (UTC)
Received: from mail-qc0-x230.google.com (mail-qc0-x230.google.com
 [IPv6:2607:f8b0:400d:c01::230])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id CB8B73E3C;
 Mon, 18 Aug 2014 19:45:26 +0000 (UTC)
Received: by mail-qc0-f176.google.com with SMTP id m20so5351593qcx.35
 for <multiple recipients>; Mon, 18 Aug 2014 12:45:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:sender:in-reply-to:references:date:message-id:subject
 :from:to:cc:content-type;
 bh=JaebXXZUm9b4PmO3YqhfcQsFOSMkpoJPDSei3E/JaWo=;
 b=LV6sTVs2D9YIIjyXUmBIR1d/M3yrVo8Av+W1eRZVylLvNZrIWcndMvz5Cv9qr3+K+8
 bMprjgLqrcPLmGInynn/HO3qWd3uH8RLJAg8CM/DovlSup1kSJQ2Ac98ISmuY6ML+qR4
 BJW6pY+0zrQ3dANRZsJ6mWSC2MZnYMocApM0wgzNslgwwbUg42PG+mD5ELZeh/BQvcKE
 Yt1dk5ZxxLvneTI5zDqHPoL2SubrZD2wAujtcHUkKt5L8R/cAkh2hhbJdjaoSGm3w5+B
 AI7HSzS3+iV3DxXdMT9/vjjWc72ozHaPNcnsZjADgSP6RCqR1Y1eNBNxq62MCyctrRl2
 7W+Q==
MIME-Version: 1.0
X-Received: by 10.140.27.144 with SMTP id 16mr55305116qgx.18.1408391125698;
 Mon, 18 Aug 2014 12:45:25 -0700 (PDT)
Sender: adrian.chadd@gmail.com
Received: by 10.224.39.139 with HTTP; Mon, 18 Aug 2014 12:45:25 -0700 (PDT)
In-Reply-To: <20140818183925.GP2737@kib.kiev.ua>
References: <53F215A9.8010708@FreeBSD.org> <20140818183925.GP2737@kib.kiev.ua>
Date: Mon, 18 Aug 2014 12:45:25 -0700
X-Google-Sender-Auth: mTqPAID1-WA3Y8GAwUvn97dr36g
Message-ID: <CAJ-VmokXDtXSqCKo9wNVeqd9yQeZwgvjWqPA4tQnydx+0W_Gzg@mail.gmail.com>
Subject: Re: superpages for UMA
From: Adrian Chadd <adrian@freebsd.org>
To: Konstantin Belousov <kostikbel@gmail.com>
Content-Type: text/plain; charset=UTF-8
Cc: "freebsd-arch@freebsd.org" <arch@freebsd.org>,
 Gleb Smirnoff <glebius@freebsd.org>,
 "Alexander V. Chernikov" <melifaro@freebsd.org>,
 "Andrey V. Elsukov" <ae@freebsd.org>
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch/>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Aug 2014 19:45:27 -0000

Hi!

I dug into this a little bit last year. I saw a lot of time spent just
walking TLBs for VM pages when doing a lot of VM page -> network
pushing.

On the sandy bridge boxes with 1G page entries, the TLB only has 4 entries.

The high area of memory isn't 1G aligned, so we don't use 1G pages for
all the stuff that's allocated initially. That includes, among other
things, all the VM memory that you need.

The other thing that crept up was that we don't try to reserve memory
in any way - we'll just fragment stuff quickly from the pmap and
allocate where we can when we can. So there's currently no attempt to
allocate small kernel structures from the same underlying 1G page.

That'd be an interesting experiment - allocating VM entries and other
small things like rtentry and mbuf UMA entries from one or two 1GB
regions of memory. It may make better use of the 1G (or 2M) TLB
entries and keep things hot.



-a