From owner-freebsd-hackers@FreeBSD.ORG  Sat Jan  5 22:02:24 2013
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by hub.freebsd.org (Postfix) with ESMTP id 7FC2F827;
 Sat,  5 Jan 2013 22:02:24 +0000 (UTC)
 (envelope-from jasone@freebsd.org)
Received: from canonware.com (canonware.com [204.109.63.53])
 by mx1.freebsd.org (Postfix) with ESMTP id 548A082D;
 Sat,  5 Jan 2013 22:02:24 +0000 (UTC)
Received: from [192.168.168.12]
 (70-91-206-178-BusName-SFBA.hfc.comcastbusiness.net [70.91.206.178])
 (using TLSv1 with cipher AES128-SHA (128/128 bits))
 (No client certificate requested)
 by canonware.com (Postfix) with ESMTPSA id 1C7EE28431;
 Sat,  5 Jan 2013 13:54:03 -0800 (PST)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: L1 cache thrashing affects performance of HIMENO benchmark
From: Jason Evans <jasone@freebsd.org>
In-Reply-To: <CAJ-Vmo=O9v==Xpm8F2UUjkEAOaHbKydLMpn6Bz-5rRBt+2TEAg@mail.gmail.com>
Date: Sat, 5 Jan 2013 13:54:02 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <DFFCA030-3206-4EB2-88C8-262AB298FF9F@freebsd.org>
References: <CA+zcas1V0KzcBKjq1u3Rwu_Nm7hVkG+G+eHpHRDQG0_NcXoOWA@mail.gmail.com>
 <CAJ-Vmo=O9v==Xpm8F2UUjkEAOaHbKydLMpn6Bz-5rRBt+2TEAg@mail.gmail.com>
To: Adrian Chadd <adrian@freebsd.org>
X-Mailer: Apple Mail (2.1499)
Cc: freebsd-hackers@freebsd.org, Hakisho Nukama <nukama@gmail.com>
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 05 Jan 2013 22:02:24 -0000

On Jan 5, 2013, at 12:47 PM, Adrian Chadd <adrian@freebsd.org> wrote:
> On 5 January 2013 07:38, Hakisho Nukama <nukama@gmail.com> wrote:
>> FreeBSD (PCBSD) is slower compared to Linux and kFreeBSD in this
>> benchmark of HIMENO:
>> =
http://openbenchmarking.org/prospect/1202215-BY-FREEBSD9683/88ac7a01c6cb35=
5d7e7603224b2ee1e5a4cb881d
>> Also DragonFly BSD compares worse to kFreeBSD and Linux:
>> =
http://www.phoronix.com/scan.php?page=3Darticle&item=3Ddragonfly_linux_32&=
num=3D3
>> =
http://openbenchmarking.org/prospect/1206255-SU-DRAGONFLY55/88ac7a01c6cb35=
5d7e7603224b2ee1e5a4cb881d
>>=20
>> Matt, Venkatesh and Alex investigated this performance problem and
>> came to these results:
>> http://leaf.dragonflybsd.org/mailarchive/users/2013-01/msg00011.html
>=20
> I've CC'ed jasone on this as it's an interesting side-effect of memory
> allocation logic.
>=20
> Jason - any comments?

There are many variations on this class of performance problem, and the =
short of it is that only the application can have adequate understanding =
of data structure layout and access patterns to reliably make optimal =
use of the cache.  However, it is possible for the allocator to lay out =
memory in a more haphazard fashion than jemalloc, phkmalloc, etc. do, =
such that the application can be cache-oblivious and (usually) not =
suffer worst case consequences as happened in this case.  Extent-based =
allocators like dlmalloc often get this "for free" for a significant =
range of allocation sizes.  jemalloc could be modified to this end, but =
a full solution would necessarily increase internal fragmentation.  It =
might be worth experimenting with nonetheless.

Thanks,
Jason=