From owner-freebsd-hackers@FreeBSD.ORG Tue Jan 8 17:23:28 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C60E952; Tue, 8 Jan 2013 17:23:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-f53.google.com (mail-wg0-f53.google.com [74.125.82.53]) by mx1.freebsd.org (Postfix) with ESMTP id 92D1AE75; Tue, 8 Jan 2013 17:23:27 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id fn15so552715wgb.32 for ; Tue, 08 Jan 2013 09:23:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=LOMGAqo8dwoiOZ6BsuZAYuC6LDIV5s5saff17MQrYvI=; b=mgarmcPlPXEbFupK8n+tKDU2OYpbD9XOlI55YdkeJUsOwHp8zbg98Rg8hNYefVMoel zm6hEt2LYcwPYO9OwfxtRenFA3I9n0OTk9tONtqYrOQwhKygIireH4IyBBz2BguwmGPB sEJmU3dN+X9mFwd/NNcyir9A0229mA356DwJcOju0qyjY7vNnOgtLRyaeDLHf6MZM9YA oF7DXYOv1ey+Le5oqa/VmcqrKXtTiOWhQod4W+Qst48+U+cy4YMs/4H1WaruRs3R1UJQ om6WmXEsSlQ28pJxyzWj/LNA8jX+0qZkGCLwMed52U/hKSH9wf+iP8PPLe3MKnzjvcC/ 6NSw== MIME-Version: 1.0 Received: by 10.195.13.67 with SMTP id ew3mr15409873wjd.59.1357665806648; Tue, 08 Jan 2013 09:23:26 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.217.57.9 with HTTP; Tue, 8 Jan 2013 09:23:26 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Jan 2013 09:23:26 -0800 X-Google-Sender-Auth: DJLf1YWEyqnguVaJh_tJ5cI9ADU Message-ID: Subject: Re: L1 cache thrashing affects performance of HIMENO benchmark From: Adrian Chadd To: Jason Evans Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Hakisho Nukama X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 17:23:28 -0000 ... can someone please file a FreeBSD PR with some example workloads and the dfbsd list summary? I'd like to make sure we don't lose this particular thread. It may be worth "teaching" jemalloc to offset the allocation sizes so they don't hit this degenerate cache case, then do a bunch of testing to ensure nothing has regressed. Thanks, adrian On 5 January 2013 18:03, Adrian Chadd wrote: > On 5 January 2013 13:54, Jason Evans wrote: > >>> Jason - any comments? >> >> There are many variations on this class of performance problem, and the = short of it is that only the application can have adequate understanding of= data structure layout and access patterns to reliably make optimal use of = the cache. However, it is possible for the allocator to lay out memory in = a more haphazard fashion than jemalloc, phkmalloc, etc. do, such that the a= pplication can be cache-oblivious and (usually) not suffer worst case conse= quences as happened in this case. Extent-based allocators like dlmalloc of= ten get this "for free" for a significant range of allocation sizes. jemal= loc could be modified to this end, but a full solution would necessarily in= crease internal fragmentation. It might be worth experimenting with noneth= eless. > > For at least this particular computational workload, the loss in > throughput based on cache thrashing is significant enough to learn > FreeBSD a negative mark in computational workloads. > > It'd be interesting to see which other workloads FreeBSD behaves poorly i= n. > > In fact, it'd be doubly interesting to get some people who _do_ > computational workloads to do some profiling using oprofile/pmc and > report back. Maybe if we wrote a wiki page on how to do this kind of > profiling and how to interpret the results. > > In any case, yes - I think it's worth pursuing this further as it's > very likely not the only workload that exhibits this kind of cache > unhappiness. > > > > Adrian