From owner-freebsd-hackers@FreeBSD.ORG Sun Jan 6 02:03:12 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8913963E; Sun, 6 Jan 2013 02:03:12 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-f169.google.com (mail-we0-f169.google.com [74.125.82.169]) by mx1.freebsd.org (Postfix) with ESMTP id D65E8E03; Sun, 6 Jan 2013 02:03:10 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id t49so8964863wey.14 for ; Sat, 05 Jan 2013 18:03:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=xC9bdqS1B/edfvfx0gF2OZDNWzLZhZJ4UvvnfBV0t1g=; b=qB3VmECGyPTuPZAvtWkEF7UtmZ7Qus9YCEUIDY2ecsb6xu/sb+1P+QsP71RfjgiizS 2+o4JiJ+CezAaSlej/S/mSQjytYUd3JvOk8gvfaPmVXjv6zxHWC42URlpMzhABL7a4vs RUyUji/O4I9lEFjThVDrCGDcPN6GdMkgTHDq0Ao6MlzsnMB+aI0jUftgS0YTFBZr8LS6 2EDQTpRNd0mtp/nC2Wh2AYwp2t5SimIN8LviuFEHqw2/S48fedfizOI3oWyLJKlqzL/1 RXkM2f1gpQlI1e84L7ftXqyhFa9oi8n3wuLU0b4T5OOfNEmhk9LkI2x7DFgdrHZV9Vj+ 7OMg== MIME-Version: 1.0 Received: by 10.194.93.40 with SMTP id cr8mr90440504wjb.16.1357437784136; Sat, 05 Jan 2013 18:03:04 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.217.57.9 with HTTP; Sat, 5 Jan 2013 18:03:04 -0800 (PST) In-Reply-To: References: Date: Sat, 5 Jan 2013 18:03:04 -0800 X-Google-Sender-Auth: GhmIR5NIG4HoT4nMbbaUVJYZKMQ Message-ID: Subject: Re: L1 cache thrashing affects performance of HIMENO benchmark From: Adrian Chadd To: Jason Evans Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Hakisho Nukama X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2013 02:03:12 -0000 On 5 January 2013 13:54, Jason Evans wrote: >> Jason - any comments? > > There are many variations on this class of performance problem, and the s= hort of it is that only the application can have adequate understanding of = data structure layout and access patterns to reliably make optimal use of t= he cache. However, it is possible for the allocator to lay out memory in a= more haphazard fashion than jemalloc, phkmalloc, etc. do, such that the ap= plication can be cache-oblivious and (usually) not suffer worst case conseq= uences as happened in this case. Extent-based allocators like dlmalloc oft= en get this "for free" for a significant range of allocation sizes. jemall= oc could be modified to this end, but a full solution would necessarily inc= rease internal fragmentation. It might be worth experimenting with nonethe= less. For at least this particular computational workload, the loss in throughput based on cache thrashing is significant enough to learn FreeBSD a negative mark in computational workloads. It'd be interesting to see which other workloads FreeBSD behaves poorly in. In fact, it'd be doubly interesting to get some people who _do_ computational workloads to do some profiling using oprofile/pmc and report back. Maybe if we wrote a wiki page on how to do this kind of profiling and how to interpret the results. In any case, yes - I think it's worth pursuing this further as it's very likely not the only workload that exhibits this kind of cache unhappiness. Adrian