From owner-freebsd-arch@FreeBSD.ORG Sat Jul 19 03:07:08 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 334C71065674 for ; Sat, 19 Jul 2008 03:07:08 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.182]) by mx1.freebsd.org (Postfix) with ESMTP id 17BD88FC1B for ; Sat, 19 Jul 2008 03:07:08 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by wa-out-1112.google.com with SMTP id j4so256428wah.3 for ; Fri, 18 Jul 2008 20:07:07 -0700 (PDT) Received: by 10.115.60.1 with SMTP id n1mr670565wak.179.1216435336625; Fri, 18 Jul 2008 19:42:16 -0700 (PDT) Received: from ?10.0.1.199? ( [24.94.72.120]) by mx.google.com with ESMTPS id v38sm3540207wah.14.2008.07.18.19.42.14 (version=SSLv3 cipher=RC4-MD5); Fri, 18 Jul 2008 19:42:15 -0700 (PDT) Date: Fri, 18 Jul 2008 16:41:58 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: arch@freebsd.org Message-ID: <20080718163231.B954@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: attilio@freebsd.org, ivmaykov@gmail.com Subject: witness performance improvements X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jul 2008 03:07:08 -0000 Hello, I have a patch that improves witness performance available at: http://people.freebsd.org/~jeff/witness.diff This improvement comes at the cost of some significant space overhead. It changes the witness graph from a linked tree to a matrix based approach. Relationships can be quickly resolved with a table lookup. The table size is WITNESS_COUNT^2, or 1MB with the current count of 1024. This patch also makes struct witness objects persistent even after the last lock using this name has been removed. This is helpful for short lived objects which may be created frequently. To reduce lock contention on SMP witness_checkorder() now runs without the w_mtx when there are no lock violations. I also cache a lock_list_entry in each thread as allocating these requires the w_mtx. The entry is disposed of at thread_exit(). There is also a new sysctl that produces dot output which graphs lock order relationships with the graphviz program. Most of this work was done by Ilya Maykov while he was at Isilon systems. The locking work and some cleanup/porting/refinement was done by me on behalf of Nokia. The performance improvement can be significant. It is only on the order of 10-20% for buildkernel but on a packet forwarding test at nokia it sped things up by 5x putting a witness enabled kernel within about 50% of the performance of a kernel without. I believe buildworld isn't helped as much because forking and exiting a lot would then contend on the witness lock. I'm mostly interested in hearing what people have to say about the space bloat. I believe it is in a commit ready state. Thanks, Jeff