From owner-freebsd-hackers@freebsd.org Thu Jun 28 16:45:01 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C29A4101F951 for ; Thu, 28 Jun 2018 16:45:01 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: from mail-oi0-x22a.google.com (mail-oi0-x22a.google.com [IPv6:2607:f8b0:4003:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BEDB83C14 for ; Thu, 28 Jun 2018 16:45:01 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: by mail-oi0-x22a.google.com with SMTP id 21-v6so5802627oip.8 for ; Thu, 28 Jun 2018 09:45:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=7Y4hcIejctpVzkzejTxhZ6hxkftaS6m7RDl6WPW2/+g=; b=ddvVxzzwMkQo8Cr4HQdTUKH/G64FewWWDoKU9m0bWcuF9b8WZg/SYDE8hqz7Eq3m8D hV2ZIzSHgPof3IedEpQJ4btFy90Qkq61xIgQByQ9Y1YjzgJZB99tzHO2VZdTK6GOC4U+ nomVku3oqoz6JLJ/oo4O/moCHy+jaLTZq8hUoNdVj/iKqcN0pKvMmXrywUzL1JK3se13 GTPAD+FjEudM72nFL8mKzh9MQGvQrWbyadpnc98B3ESoqdnBqRCIcraHukpeMCzpc3yY 2hvTfFk/pjzKGtyVO9apGMcK4umIaSg04hjZzoPdrxkFckYkpABeWOuUsPS95f2uzA3R c6hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=7Y4hcIejctpVzkzejTxhZ6hxkftaS6m7RDl6WPW2/+g=; b=IGgT8h5s1/kXzSEJNunFc7dcnPYYr3MKphQMW0CKecFKlw0GR+TuDXXOyrvbTOSksI ZngJk2UTlAo4kzzPz6sKGiRKa1l1xhVSIJEYhaupG81QjE3TRGE1yFojbVj/MJDrGjta BmFDqqnxIyzmWaJioAyzzIlPcdBtQPz77lFjyD2RmLlNBoV+7ZZEuxT99xnffFUs8rpd I/0Ij3oE/RYcoU8cE9KYAjHALldjoEuWWOyiCMsRXGoP1LnJuKNCWwsm3Qq2bEB5RM3X uHvwy5Isx1QkbvIFZPv+9wbFjl1v9LMoU7+uraJf6CAf59oLTKZQlvplVeMjj0tVXwOZ ufFQ== X-Gm-Message-State: APt69E2nnTPBru/NmvJF2ABbaVQ3/ITy5R4MbbwHJfKeiAaMB0rOpyHa +JfXUY7/GZgulPQGFUb96Sz7H4drn3Aoqqs/9g4= X-Google-Smtp-Source: AAOMgpd9PiHJinE2kqYfa2y8HDf73c4JLzaVZNlrfe8HwjOGeZwrN/QsjGd9XRYsffGqxGir3RvKJ5hiCBXD9xLpCbk= X-Received: by 2002:aca:a806:: with SMTP id r6-v6mr6440382oie.213.1530204300302; Thu, 28 Jun 2018 09:45:00 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ac9:73cf:0:0:0:0:0 with HTTP; Thu, 28 Jun 2018 09:44:59 -0700 (PDT) In-Reply-To: References: From: Steevan Rodrigues Date: Thu, 28 Jun 2018 22:14:59 +0530 Message-ID: Subject: Re: high CPU usage in FreeBSD for a PCIe card driver To: Andrew Gallatin Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 16:45:02 -0000 Thank you so much for the suggestions . I will use these commands. Yes, I am already working on to identify lock contentions. Re-built FreeBSD kernel by enabling lock profiling and now I am able to see some issues with contention. Thanks Steevan On Thu, Jun 28, 2018 at 6:34 PM, Andrew Gallatin wrote: > On 06/27/18 08:28, Steevan Rodrigues wrote: > >> Hi >> >> I come from Linux background and new to FreeBSD. >> >> I am investigating a high CPU usage issue on FreeBSD for PCIe driver of a >> > > <...> > > >> I wonder whether usage of mtx_lock is the cause of high CPU usage ? Any >> pointers or suggestions welcome. >> > > > It seems like you're experiencing lock contention. The next step is to > determine which lock is contended and why. > > I would suggest using the lockstat program to identify the lock which > is contended. Try this: lockstat -x aggsize=4m sleep 10 > out > > And this: lockstat -x aggsize=4m -s 10 sleep 10 > out-stacks > > The first will show you contended locks broken down by type. Eg: > > > Adaptive mutex spin: 61262 events in 10.007 seconds (6122 events/sec) > > Count indv cuml rcnt nsec Lock Caller > ------------------------------------------------------------ > ------------------- > 15958 26% 26% 0.00 2960 tcp_hpts_lck __tcp_hpts_remove+0xb5 > 11723 19% 45% 0.00 2608 tcp_hpts_lck tcp_hpts_insert_diag+0xaf > 6903 11% 56% 0.00 4605 tcp_hpts_lck tcp_hpts_thread+0x11e > 6834 11% 68% 0.00 125868 sleep mtxpool _sleep+0x2e3 > 2756 4% 72% 0.00 1530 vm page sendfile_free_page+0x1c1 > > > The second will show you the stacks associated with the lock contention, > and may help you identify the source of contention if, for example, your > driver is causing contention on a kernel lock. > > Drew > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >