From owner-freebsd-hackers@freebsd.org Thu Jun 28 13:05:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3F6C1033267 for ; Thu, 28 Jun 2018 13:05:04 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A6217AF6A for ; Thu, 28 Jun 2018 13:05:04 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.200.3] (c-73-216-227-39.hsd1.va.comcast.net [73.216.227.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id A3E26270014A for ; Thu, 28 Jun 2018 09:04:56 -0400 (EDT) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu A3E26270014A Authentication-Results: duke.cs.duke.edu; dmarc=none header.from=cs.duke.edu DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1530191096; bh=g+h6YksUMRTPIJcsg2Kbv/44Wq1XXDz92UOVhpp4Nl4=; h=Subject:To:From:Date:From; b=IfdYCENuAsZSY5RF3Q9x+sMaWV/6qHU+WO9vOi0htg3GnbAVJqZE9exfDyG5Dpq4U ZnJF34iKRK3AsMkDU0urmab7SYeyixzNt2nIXyBvEwkA2OUuNNOKd9EABRNQbqJvv0 cpDKrGaL9AN3aC+4X/3FbKg9cdW1rvrZT+cGXaKwYK0Rh4WmVxLR8FgSN1X3GNs+PP 01B/9safrVGExPFkDdTV3MzsdouNYHA0QMnUcl4AFzwWvGk3L86+mj/VS+aNZatJnU +QT5Pknghai7F7129c9lPfsJ7w5bCZXdUrz60bZ1qALgKP2SF84vuQWbNyi6JHx3zQ cSd9dX+cY373Q== Subject: Re: high CPU usage in FreeBSD for a PCIe card driver To: freebsd-hackers@freebsd.org References: From: Andrew Gallatin Message-ID: Date: Thu, 28 Jun 2018 09:04:56 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 13:05:04 -0000 On 06/27/18 08:28, Steevan Rodrigues wrote: > Hi > > I come from Linux background and new to FreeBSD. > > I am investigating a high CPU usage issue on FreeBSD for PCIe driver of a <...> > > I wonder whether usage of mtx_lock is the cause of high CPU usage ? Any > pointers or suggestions welcome. It seems like you're experiencing lock contention. The next step is to determine which lock is contended and why. I would suggest using the lockstat program to identify the lock which is contended. Try this: lockstat -x aggsize=4m sleep 10 > out And this: lockstat -x aggsize=4m -s 10 sleep 10 > out-stacks The first will show you contended locks broken down by type. Eg: Adaptive mutex spin: 61262 events in 10.007 seconds (6122 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 15958 26% 26% 0.00 2960 tcp_hpts_lck __tcp_hpts_remove+0xb5 11723 19% 45% 0.00 2608 tcp_hpts_lck tcp_hpts_insert_diag+0xaf 6903 11% 56% 0.00 4605 tcp_hpts_lck tcp_hpts_thread+0x11e 6834 11% 68% 0.00 125868 sleep mtxpool _sleep+0x2e3 2756 4% 72% 0.00 1530 vm page sendfile_free_page+0x1c1 The second will show you the stacks associated with the lock contention, and may help you identify the source of contention if, for example, your driver is causing contention on a kernel lock. Drew