From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 15 01:04:34 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 809DF16A400 for ; Fri, 15 Jun 2007 01:04:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 6901213C45A for ; Fri, 15 Jun 2007 01:04:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.13.8/8.13.7) with ESMTP id l5F14RET001011; Thu, 14 Jun 2007 18:04:27 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.13.8/8.13.4/Submit) id l5F14RjG001010; Thu, 14 Jun 2007 18:04:27 -0700 (PDT) Date: Thu, 14 Jun 2007 18:04:27 -0700 (PDT) From: Matthew Dillon Message-Id: <200706150104.l5F14RjG001010@apollo.backplane.com> To: Ivan Voras References: Cc: freebsd-hackers@freebsd.org Subject: Re: Reason for doing malloc / bzero over calloc (performance)? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2007 01:04:34 -0000 I'm going to throw a wrench in the works, because it all gets turned around the moment you find yourself in a SMP environment where several threads are running on different cpus at the same time, using the same shared VM space. The moment you have a situation like that where you are futzing with the page tables, i.e. using mmap() for demand-zero and munmap() to free, the operation becomes extremely expensive verses anything else because any update to the page table (specifically any removal of page table entries from the page table) requires a SMP synchronization to occur between all the cpu's actively sharing that VM space, and that's on top of the overhead of taking the page fault(s). This is true of any memory mapping the kernel has to do in kernel virtual memory (must be synchronized with ALL cpus) and any mapping the kernel does on behalf of userland for user memory (must be synchronized with any cpu's actively using that VM space, i.e. threaded user programs). The synchronization is required to properly invalidate stale mappings on other cpus and it must be done synchronously due to bugs in Intel/AMD related to changing page table entries on one cpu when instructions are executing using that memory on another cpu. There is no way to avoid it without tripping up on the Intel/AMD hardware bugs. From this point of view it is much, much better to bzero() memory that is already mapped then it is to map/unmap new memory. I recently audited DragonFly and found an insane number of IPIs flying about due to PAGE_SIZE'd kernel mallocs using the VM trick via kernel_map & kmem_alloc(). They all went away when I made the kernel malloc use the slab cache for allocations up to and including PAGE_SIZE*2 bytes. Fun, eh? -Matt Matthew Dillon