From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 5 12:40:52 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4A5114E1; Thu, 5 Sep 2013 12:40:52 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bk0-x230.google.com (mail-bk0-x230.google.com [IPv6:2a00:1450:4008:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B12292B1C; Thu, 5 Sep 2013 12:40:51 +0000 (UTC) Received: by mail-bk0-f48.google.com with SMTP id my13so720689bkb.35 for ; Thu, 05 Sep 2013 05:40:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=8Z3kcvelwxokhY3nBr3uMzq9VZsLTGH5NrWXxMd5fKY=; b=jElfaW8h/9NoTien/GZ6ue1dgyR6DFdPuI4tloJZGHxYulkd4ooNwB8cCm5ZpyS/zC 041akRXQNjWEuXZTF8JqYQB7KHBAuWoZpZH5hfiSu/gDAZAjn35XjIGF4/6v/AI25JTz agoCIbVK6jk7wJxAwmD2jn+/D1XBsGYyJ43ctqBDsanjzBnIHQUSdNzNC24tBGGX7rXk YCVyBA6L5Cex9IVzvL341l+FNt4m0Xv1flLB6giZOW89mk+pMyjIhzUIzBCggNCHQ//1 ZlRlZPU56qRYcCsMk6QvqbfLMpBgBfz4EZDqpYX3G37nY5zW11/f4CfGDX2AQrDzy5r8 IoBg== X-Received: by 10.204.121.201 with SMTP id i9mr7215543bkr.13.1378384849200; Thu, 05 Sep 2013 05:40:49 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id pn6sm7928380bkb.14.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Sep 2013 05:40:48 -0700 (PDT) Sender: Alexander Motin Message-ID: <52287BCD.4090507@FreeBSD.org> Date: Thu, 05 Sep 2013 15:40:45 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: hackers@freebsd.org Subject: Again about pbuf_mtx Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Sep 2013 12:40:52 -0000 Hi. Some may remember that not so long ago I complained about high lock congestion on pbuf_mtx. At that time switching the mutex to padalign reduced the problem. But now after improving scalability in CAM and GEOM and doing more then half million IOPS on 32-core system I again heavily hit that problem -- hwpmc shows about 30% of CPU time spent on that mutex spinning and another 30% of time spent on attempt of threads to go to sleep on that mutex and getting more collisions there. Trying to mitigate that I've made a patch (http://people.freebsd.org/~mav/pcpu_pbuf.patch) to split single queue of pbufs into several. That definitely cost some amount of KVA and memory, but on my tests it fixes problem redically, removing any measurable congestion there. The patch is not complete and don't even boot on i386 now, but I would like to hear opinions about the approach, or may be some better propositions. Another patch I've made (http://people.freebsd.org/~mav/si_threadcount.patch) removes lock acquisition from dev_relthread() by using atomics for reference counting. That fixes another congestion I see. This patch looks fine to me and the only congestion I see after that is on HBA driver locks, but may be I am missing something? Thank you. -- Alexander Motin