From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 26 11:29:46 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 940C3B64 for ; Sat, 26 Apr 2014 11:29:46 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 583AC1181 for ; Sat, 26 Apr 2014 11:29:46 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 65456359307; Sat, 26 Apr 2014 13:29:43 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 460F528497; Sat, 26 Apr 2014 13:29:43 +0200 (CEST) Date: Sat, 26 Apr 2014 13:29:43 +0200 From: Jilles Tjoelker To: Karl Pielorz Subject: Re: HAST performance / dtrace / _umtx_op Message-ID: <20140426112943.GA81447@stack.nl> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Apr 2014 11:29:46 -0000 On Sat, Apr 26, 2014 at 11:30:25AM +0100, Karl Pielorz wrote: > I've been looking at HAST recently - and noticed there's a huge > performance hit (on 10-STABLE anyway) even if you setup a disk with a > remote of 'none' - switching from the 'raw' disk to '/dev/hast/disk' > shows a 50% performance hit [in testing having an active secondary via > 10Gbit link doesn't seem to show any further performance drop]. > Thinking of digging deeper I setup a single disk HAST system (with no > remote node) and ran dtrace on the daemon handling that disk. I then > DD'ed 1Gbyte of data to that disk. > The output from dtrace shows: > " > Elapsed Times for PID 5219, > > SYSCALL TIME (ns) > pread 4439902 > pwrite 7617448542 > ioctl 11370282332 > sigtimedwait 20015976362 > _umtx_op 36514993910 > " > To my untrained eye that looks like it spent more time on locking than > anything else? - Followed by ioctl's - then, considerably less time on > pread/pwrite (which must be the bit that's copying the data) > Presumably the time spent in sigtimedwait would have actually been > 'waiting' time - i.e. not active CPU usage kind of time? > Does that seem to be the case? - At least then I'll hopefully have > some pointer where to look further for why there's such a performance > hit. _umtx_op implements blocking not only for mutexes but also for condition variables, and in the latter case waiting for a long time does not indicate a problem. -- Jilles Tjoelker