From owner-freebsd-hackers@FreeBSD.ORG Sat Jul 23 02:31:19 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1E96106564A for ; Sat, 23 Jul 2011 02:31:19 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id A34088FC08 for ; Sat, 23 Jul 2011 02:31:19 +0000 (UTC) Received: by iyb11 with SMTP id 11so3262839iyb.13 for ; Fri, 22 Jul 2011 19:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=WHypVBGubqP0b/jedWTg+FLOrQGIAWEYdtkBagGBbm4=; b=O1Od69rDIa/eNI4Yb/WCFM37go6Rxp1JqfPWx+GUR9W95hnIjE2+2TiGcW1y5L6CbO D79N9dfO7rw6NLiyvSKUAnBNWAbKaC5Lel6nweuYLVrIWg6nSmFC9kjeuh902bR4I6cS cQ4Ni5oBq/kKdVSl2lDBM+x/dD9th7QfkveJo= MIME-Version: 1.0 Received: by 10.142.223.3 with SMTP id v3mr1219466wfg.289.1311386839612; Fri, 22 Jul 2011 19:07:19 -0700 (PDT) Received: by 10.142.199.13 with HTTP; Fri, 22 Jul 2011 19:07:19 -0700 (PDT) Date: Sat, 23 Jul 2011 04:07:19 +0200 Message-ID: From: Davide Italiano To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: UMA large allocations issues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jul 2011 02:31:19 -0000 Hi. I'm a student and some time ago I started investigating a bit about the performance/fragmentation issue of large allocations within the UMA allocator. Benckmarks showed up that this problems of performances are mainly related to the fact that every call to uma_large_malloc() results in a call to kmem_malloc(), and this behaviour is really inefficient. I started doing some work. Here's somethin: First of all, I tried to define larger zones and let uma do it all as a first step. UMA can allocate slabs of more than one page. So I tried to define zones of 1,2,4,8 pages, moving ZMEM_KMAX up. I tested the solution w/ raidtest. Here there are some numbers. Here's the workload characterization: set mediasize=`diskinfo /dev/zvol/tank/vol | awk '{print $3}'` set sectorsize=`diskinfo /dev/zvol/tank/vol | awk '{print $2}'` raidtest genfile -s $mediasize -S $sectorsize -n 50000 # $mediasize = 10737418240 # $sectorsize = 512 Number of READ requests: 24924 Number of WRITE requests: 25076 Numbers of bytes to transmit: 3305292800 raidtest test -d /dev/zvol/tank/vol -n 4 ## tested using 4 cores, 1.5 GB Ram Results: Number of processes: 4 Bytes per second: 10146896 Requests per second: 153 Results: (4* PAGE_SIZE) Number of processes: 4 Bytes per second: 14793969 Requests per second: 223 Results: (8* PAGE_SIZE) Number of processes: 4 Bytes per second: 6855779 Requests per second: 103 The result of this tests is that defining larger zones is useful until the size of these zones is not too big. After some size, performances decreases significantly. As second step, alc@ proposed to create a new layer that sits between UMA and the VM subsystem. This layer can manage a pool of chunk that can be used to satisfy requests from uma_large_malloc so avoiding the overhead due to kmem_malloc() calls. I've recently started developing a patch (not yet full working) that implements this layer. First of all I'd like to concentrate my attention to the performance problem rather than the fragmentation one. So the patch that actually started to write doesn't care about fragmentation aspects. http://davit.altervista.org/uma_large_allocations.patch There are some questions to which I wasn't able to answer (for example, when it's safe to call kmem_malloc() to request memory). So, at the end of the day I'm asking for your opinion about this issue and I'm looking for a "mentor" (some kind of guidance) to continue this project. If someone is interested to help, it would be very appreciated. Best Davide Italiano