From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 21:26:50 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 177BCFDA for ; Thu, 27 Jun 2013 21:26:50 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x22e.google.com (mail-ee0-x22e.google.com [IPv6:2a00:1450:4013:c00::22e]) by mx1.freebsd.org (Postfix) with ESMTP id A5BFD1F9D for ; Thu, 27 Jun 2013 21:26:49 +0000 (UTC) Received: by mail-ee0-f46.google.com with SMTP id d41so655910eek.33 for ; Thu, 27 Jun 2013 14:26:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=UAiQgGwAe+Yj/ChRDXoxuuxVoZuwV+CGeSbOcCqL+IE=; b=zBBs6g620UKEI5RQ5/14MPNRmvogHNO45MLlUDKoHwuG9VvY8AFmCjr1v+oUvRIVWx kKiQHtmHk6ueDbNe7FmqdmbwaNkteDe/tIWqMM5pJLwUs2n57UrLn1vSPX6FyFVvbBHl +JBY90sa35yQOAnG+tW+pzwiVmD1UEwEpUIf0zOTJGIRgY/Dfz9IZprFjyScjMLnCGPr jwazps8Sgt4kVprniurLp5DeXnGC+ZqplzkaQ3jWDtpxmcKRTzzm0kjVYomKduqcMyqm G7mqBbWesandMxP9x7WTm1ZZTf2DHimJDXB38qj5i6swANwpDFnxSb45K3jIUie8EOg8 3uTg== X-Received: by 10.15.34.129 with SMTP id e1mr10845767eev.80.1372368408613; Thu, 27 Jun 2013 14:26:48 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id y1sm6335306eew.3.2013.06.27.14.26.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 27 Jun 2013 14:26:47 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CCAE14.6040504@FreeBSD.org> Date: Fri, 28 Jun 2013 00:26:44 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: hackers@freebsd.org Subject: b_freelist TAILQ/SLIST Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 21:26:50 -0000 Hi. While doing some profiles of GEOM/CAM IOPS scalability, on some test patterns I've noticed serious congestion with spinning on global pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is already very simple, I've tried to optimize probably the only thing possible there: switch bswlist from TAILQ to SLIST. As I can see, b_freelist field of struct buf is really used as TAILQ in some other places, so I've just added another SLIST_ENTRY field. And result appeared to be surprising -- I can no longer reproduce the issue at all. May be it was just unlucky synchronization of specific test, but I've seen in on two different systems and rechecked results with/without patch three times. The present patch is here: http://people.freebsd.org/~mav/buf_slist.patch The question is how to do it better? What is the KPI/KBI policy for struct buf? I could replace b_freelist by a union and keep KBI, but partially break KPI. Or I could add another field, probably breaking KBI, but keeping KPI. Or I could do something handmade with no breakage. Or this change is just a bad idea? -- Alexander Motin