From owner-svn-src-head@freebsd.org Sat Jun 23 09:10:48 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4804F1018B6B for ; Sat, 23 Jun 2018 09:10:48 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pf0-x22a.google.com (mail-pf0-x22a.google.com [IPv6:2607:f8b0:400e:c00::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A0A6F7ADE8 for ; Sat, 23 Jun 2018 09:10:47 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mail-pf0-x22a.google.com with SMTP id r11-v6so4277999pfl.6 for ; Sat, 23 Jun 2018 02:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=UgZpLhkgSVowC84/7kIySMPKwKtYbcRXcA/Kn8aRiWM=; b=EVaj7WgvNA9oOPXewhWPJRIuXNuWkDewcDrctOn4eDjvpaJgCeeJjBr66YHkzDVpbC tZjS+xJ7eZhSXJ6afvTJhtA0TY6/C8QwYm8pvpxG/Q+rpPgJpxNg2dIuvrRGVl3/S0Qb t7R/jIFvmbl0TBuDI2Vw8NZ13zpRvB02VUPoLbH5c+A6iv4hnthHgCULIYft3ZD99HQM AgwMxbKbPXEOYwcSRGs2rTJN8EPUl04jftbFTzYz5cjFsKpibLSf4+z54I0p1FZn7wiE pfEvlUwt1Wu/qSj8FZm7ynn771U1vCTdh3THQkCHr7klOqAMiA1rob+vMGGRyJX2TT6G 8iNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=UgZpLhkgSVowC84/7kIySMPKwKtYbcRXcA/Kn8aRiWM=; b=ccHKnfGmPK22yiKzA83a8LrnZM8TGcEcLo7/JARoo7KPvNMMY47z4RGxclWIEk0x6N +kNCawcDkgIwfvoCf9HC5TdMP9yWJFSMaeSDz7I6axDz3L1UJsc7XTsq15GOKIYKVDX5 dAYUnTXJKFo6D0vHMa0IFSXM/Q/x+yq8f6PQkl2nwWMmZKW8BdCSyZ6ZONQc6bVzcqpo m8WU66qQjKmZYEZztLPfR2FUyjuLE5S1mzxbLE2t6IIvCCqjEnR3GDuoUL7vV5cbzFre kmIOPPRxCE2UkMHn4ZJYVdrxfSodnMRfWNoHP85FD3y+c+BhAIgzviegC+OVRRILukyw ASnQ== X-Gm-Message-State: APt69E1qPC1nMNAIv2U1/hOACW6JLsZJ0ZX4SUtSOBnVoCv3LTV8G3WH 6XrlhUSCcOvAvIyXe8y+cSSxzQ== X-Google-Smtp-Source: ADUXVKKIzVylAYhkfb24zpuPO9Fa6Fdg7C6x1YgjEtluq7xb+0En/Jwptpy2IdGx9hwKzLfyzocyfA== X-Received: by 2002:aa7:864d:: with SMTP id a13-v6mr5023727pfo.247.1529745046334; Sat, 23 Jun 2018 02:10:46 -0700 (PDT) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by smtp.gmail.com with ESMTPSA id b4-v6sm12996802pgq.51.2018.06.23.02.10.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 23 Jun 2018 02:10:45 -0700 (PDT) Date: Fri, 22 Jun 2018 23:08:28 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Jeff Roberson cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r335579 - head/sys/vm In-Reply-To: <201806230810.w5N8AAr2037440@repo.freebsd.org> Message-ID: References: <201806230810.w5N8AAr2037440@repo.freebsd.org> User-Agent: Alpine 2.21.999 (BSF 260 2018-02-26) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jun 2018 09:10:48 -0000 On Sat, 23 Jun 2018, Jeff Roberson wrote: > Author: jeff > Date: Sat Jun 23 08:10:09 2018 > New Revision: 335579 > URL: https://svnweb.freebsd.org/changeset/base/335579 > > Log: > Sort uma_zone fields according to 64 byte cache line with adjacent line > prefetch on 64bit architectures. Prior to this, two lines were needed > for the fast path and each line may fetch an unused adjacent neighbor. > - Move fields used by the fast path into a single line. > - Move constants into the adjacent line which is mostly used for > the spare bucket alloc 'medium path'. > - Unpad the mtx which is only used by the fast path and place it in I got a little overexcited about fast. The mtx is only used by the slow path. Jeff > a line with rarely used data. This aligns the cachelines better and > eliminates 128 bytes of wasted space. > > This gives a 45% improvement on a will-it-scale test on a 24 core machine. > > Reviewed by: mmacy > > Modified: > head/sys/vm/uma_int.h > > Modified: head/sys/vm/uma_int.h > ============================================================================== > --- head/sys/vm/uma_int.h Sat Jun 23 07:14:08 2018 (r335578) > +++ head/sys/vm/uma_int.h Sat Jun 23 08:10:09 2018 (r335579) > @@ -222,9 +222,8 @@ typedef struct uma_domain * uma_domain_t; > * > */ > struct uma_keg { > - struct mtx_padalign uk_lock; /* Lock for the keg */ > + struct mtx uk_lock; /* Lock for the keg */ > struct uma_hash uk_hash; > - > LIST_HEAD(,uma_zone) uk_zones; /* Keg's zones */ > > uint32_t uk_cursor; /* Domain alloc cursor. */ > @@ -315,40 +314,48 @@ typedef struct uma_zone_domain * uma_zone_domain_t; > * > */ > struct uma_zone { > - struct mtx_padalign uz_lock; /* Lock for the zone */ > - struct mtx_padalign *uz_lockptr; > - const char *uz_name; /* Text name of the zone */ > - > - LIST_ENTRY(uma_zone) uz_link; /* List of all zones in keg */ > + /* Offset 0, used in alloc/free fast/medium fast path and const. */ > + struct mtx *uz_lockptr; > + const char *uz_name; /* Text name of the zone */ > struct uma_zone_domain *uz_domain; /* per-domain buckets */ > - > - LIST_HEAD(,uma_klink) uz_kegs; /* List of kegs. */ > - struct uma_klink uz_klink; /* klink for first keg. */ > - > - uma_slaballoc uz_slab; /* Allocate a slab from the backend. */ > + uint32_t uz_flags; /* Flags inherited from kegs */ > + uint32_t uz_size; /* Size inherited from kegs */ > uma_ctor uz_ctor; /* Constructor for each allocation */ > uma_dtor uz_dtor; /* Destructor */ > uma_init uz_init; /* Initializer for each item */ > uma_fini uz_fini; /* Finalizer for each item. */ > + > + /* Offset 64, used in bucket replenish. */ > uma_import uz_import; /* Import new memory to cache. */ > uma_release uz_release; /* Release memory from cache. */ > void *uz_arg; /* Import/release argument. */ > - > - uint32_t uz_flags; /* Flags inherited from kegs */ > - uint32_t uz_size; /* Size inherited from kegs */ > - > - volatile u_long uz_allocs UMA_ALIGN; /* Total number of allocations */ > - volatile u_long uz_fails; /* Total number of alloc failures */ > - volatile u_long uz_frees; /* Total number of frees */ > - uint64_t uz_sleeps; /* Total number of alloc sleeps */ > + uma_slaballoc uz_slab; /* Allocate a slab from the backend. */ > uint16_t uz_count; /* Amount of items in full bucket */ > uint16_t uz_count_min; /* Minimal amount of items there */ > + /* 32bit pad on 64bit. */ > + LIST_ENTRY(uma_zone) uz_link; /* List of all zones in keg */ > + LIST_HEAD(,uma_klink) uz_kegs; /* List of kegs. */ > > + /* Offset 128 Rare. */ > + /* > + * The lock is placed here to avoid adjacent line prefetcher > + * in fast paths and to take up space near infrequently accessed > + * members to reduce alignment overhead. > + */ > + struct mtx uz_lock; /* Lock for the zone */ > + struct uma_klink uz_klink; /* klink for first keg. */ > /* The next two fields are used to print a rate-limited warnings. */ > const char *uz_warning; /* Warning to print on failure */ > struct timeval uz_ratecheck; /* Warnings rate-limiting */ > - > struct task uz_maxaction; /* Task to run when at limit */ > + > + /* 16 bytes of pad. */ > + > + /* Offset 256, atomic stats. */ > + volatile u_long uz_allocs UMA_ALIGN; /* Total number of allocations */ > + volatile u_long uz_fails; /* Total number of alloc failures */ > + volatile u_long uz_frees; /* Total number of frees */ > + uint64_t uz_sleeps; /* Total number of alloc sleeps */ > > /* > * This HAS to be the last item because we adjust the zone size >