From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 31 09:10:11 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 44BAE70F; Thu, 31 Oct 2013 09:10:11 +0000 (UTC) (envelope-from satan@ukr.net) Received: from hell.ukr.net (hell.ukr.net [212.42.67.68]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F2EE82BA2; Thu, 31 Oct 2013 09:10:10 +0000 (UTC) Received: from satan by hell.ukr.net with local ID 1VboGX-0004CO-1f ; Thu, 31 Oct 2013 11:10:09 +0200 Date: Thu, 31 Oct 2013 11:10:09 +0200 From: Vitalij Satanivskij To: Andriy Gapon Subject: Re: FreeBSD 10.0-BETA1 #8 r256765M spend too much time in locks Message-ID: <20131031091008.GA15005@hell.ukr.net> References: <20131024074826.GA50853@hell.ukr.net> <20131024075023.GA52443@hell.ukr.net> <20131024115519.GA72359@hell.ukr.net> <20131024165218.GA82686@hell.ukr.net> <526A11B2.6090008@FreeBSD.org> <20131025072343.GA31310@hell.ukr.net> <526A4306.2060500@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <526A4306.2060500@FreeBSD.org> User-Agent: Mutt/1.5.22 (2013-10-16) Cc: Vitalij Satanivskij , freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Oct 2013 09:10:11 -0000 Andriy Gapon wrote: AG> on 25/10/2013 10:23 Vitalij Satanivskij said the following: AG> > AG> > AG> > http://quad.org.ua/profiling.tgz AG> > AG> > results of both methods AG> > AG> > but for pmcstat to few buffers configured by default so not all statistics in summary ^( AG> AG> From these profiling results alone I do not see pathologies. AG> It looks like you have a lot of I/O going on[*]. AG> My guess is that the I/O requests are sufficiently small and contiguous, so ZFS AG> performs a lot for I/O aggregation. For that it allocates and then frees a lot AG> of temporary buffers. AG> And it seems that that's where the locks are greatly contended and CPU is AG> burned. Specifically in KVA allocation in vmem_xalloc/vmem_xfree. AG> AG> You can try at least two approaches. AG> AG> 1. Disable I/O aggregation. AG> See the following knobs: AG> vfs.zfs.vdev.aggregation_limit: I/O requests are aggregated up to this size AG> vfs.zfs.vdev.read_gap_limit: Acceptable gap between two reads being aggregated AG> vfs.zfs.vdev.write_gap_limit: Acceptable gap between two writes being aggregated AG> AG> 2. Try to improve buffer allocation performance by using uma(9) for that. AG> vfs.zfs.zio.use_uma=1 AG> This is a boot time tunable. AG> AG> Footnotes: AG> [*] But perhaps there is some pathology that causes all that I/O to happen. I AG> can't tell that from the profiling data. So this could be another thing to try AG> to check. AG> Ok some new information Trying to Disable I/O aggregation by setting sysctl first to smaller values then to "0" cause to very high load Setting vfs.zfs.zio_use_uma=1 cause system panic and reboot at starup on freebsd 10 beta1 and beta2 (no debug in kernel) on freebsd 11 current r257395 on GENERIC kernel there is no panic but I heve no idea how much time will take setup and data merge on thouse system.