From owner-freebsd-arch@FreeBSD.ORG Wed Nov 21 16:16:40 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A73816A4EE for ; Wed, 21 Nov 2007 16:16:40 +0000 (UTC) (envelope-from test@ns1.gakki.ne.jp) Received: from ns1.gakki.ne.jp (ns1.gakki.ne.jp [211.18.219.66]) by mx1.freebsd.org (Postfix) with ESMTP id E2FDA13C48E for ; Wed, 21 Nov 2007 16:16:39 +0000 (UTC) (envelope-from test@ns1.gakki.ne.jp) Received: from ns1.gakki.ne.jp (ns1.gakki.ne.jp [127.0.0.1]) by ns1.gakki.ne.jp (Postfix) with ESMTP id 35BE628678 for ; Thu, 22 Nov 2007 01:08:47 +0900 (JST) Received: (from test@localhost) by ns1.gakki.ne.jp (8.13.4/8.13.4/Submit) id lALG8l2I029076; Thu, 22 Nov 2007 01:08:47 +0900 Date: Thu, 22 Nov 2007 01:08:47 +0900 Message-Id: <200711211608.lALG8l2I029076@ns1.gakki.ne.jp> To: freebsd-arch@freebsd.org From: TotalMP3Converter.Offerts@ns1.gakki.ne.jp MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: New OFFERT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2007 16:16:40 -0000 TOTAL MP3 CONVERTER... THIS WEEK FOR FREE!!! The best just got better. Now you can appreciate our new product at its true value! MP3 Converter is a high quality product. Those who ever tried any other tool from Softplicity know that. Program features and advantages in comparison with similar programs: * Source formats are MP3, RA, APL, MPC, MP+, M4A, MP4, TTA, OFR, SPX, WAV, OGG, WMA, FLAC, CDA, AAC, APE, MPP, WV, XM, IT, S3M, MOD, MTM, UMX. This one could help me feel like normal user, not a fool!!! The only drawback is that its a trial version. But i think its definitely worth the money , This Program makes the best TOP QUALITY. Mp3's from wav's better than highly publicized. It just works :) Copyright © 1998-2007 [1]Total MP3 Converter. All rights reserved. [all-to-mp3.png] [2]FREE Download References 1. mailto:support@TotalConverter.com 2. http://h1.ripway.com/totaldownloads/TotalMP3Converter.exe From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 10:17:03 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAE0B16A46C; Thu, 22 Nov 2007 10:17:03 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id D86D313C465; Thu, 22 Nov 2007 10:17:03 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id CEDBB1A4D7E; Wed, 21 Nov 2007 14:23:19 -0800 (PST) Date: Wed, 21 Nov 2007 14:23:19 -0800 From: Alfred Perlstein To: arch@freebsd.org Message-ID: <20071121222319.GX44563@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: attilio@freebsd.org Subject: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 10:17:04 -0000 In summary, I am proposing (temporarily) making read-recursion on rwlocks not supported in order to avoid livelock due to writer starvation. More details: We currently have a problem with our implementation of rwlocks. I think this is a key issue for 7.x as what we decide to support will have rammifications for many years to come. We do not support writer priority or "fair share" usage of rwlocks between readers and writers. We have several choices to rectify this. 1. Disallow recursion on rwlocks (witness can be used to enforce this), this simplifies rwlocks such that we can avoid deadlock when a single reader is trying to recurse while a writer is pending. 2. Track ownership of rwlocks, this can be implemented with a "rwlock stack" in the per-thread control block (struct thread). Using this ownership information we can determine if someone is recursing and allow them to continue recursing despite a pending write request. I think the most simple solution currently is to drop support for recursive reads on rwlocks until we have the facility in place to properly support starvation avoidance. Why is this important? Simply put, developers that quickly "fix" some portion of code, whether that be a driver or part of the kernel proper who use read recursion will open the system to writer starvation and hence the system will destabilize, particulary for high load situations. I would like to get this in before 7.0-RELEASE becasue otherwise we're forced to implement something like the above mentioned solution #2, which will degrade performance for most use cases of rwlocks. Comments? -- - Alfred Perlstein From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 12:10:03 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E58B416A46B for ; Thu, 22 Nov 2007 12:10:03 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.184]) by mx1.freebsd.org (Postfix) with ESMTP id 6E1E713C4EB for ; Thu, 22 Nov 2007 12:10:03 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so2498993nfb for ; Thu, 22 Nov 2007 04:09:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=RaTKZQlpQ4iVassyC522YE6RPMJnqJ7t9LnS+WEqRFY=; b=PUHp8i+/zSt8f5dUEX6LH0hyZl/HgwKbNHLrhMenknHT00yIewwB48BVswZj+h74oPFhgylqawfOnkSC0RHx3fanrw3h62dh3at16XZL66ZgMZ0q/iSrYQcY3BcgGWviVQ6NUmGM4Vvt7IU8a6HQyAVMGGw2jPl4zEtN89id/NM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=OuFrxbmwGi3aseXxd5Mpw3Tt80E7G1mmm0vm/AkXZYU4RFWRFBEkCXT6ffiMTbQzFrhpY3I7vZvcNw6WKsuUqMvw+xa+/fZgAEg8bpfWNoWDgfct6pgJS03b4d6FqTTmdxI9UPL9N0WdfpP+YiX0kdE3vD3EuoVgeGLleh20xRM= Received: by 10.86.54.3 with SMTP id c3mr8388055fga.1195731723693; Thu, 22 Nov 2007 03:42:03 -0800 (PST) Received: by 10.86.28.19 with HTTP; Thu, 22 Nov 2007 03:42:03 -0800 (PST) Message-ID: <3bbf2fe10711220342i3360f077u2e093a676b1fdbe8@mail.gmail.com> Date: Thu, 22 Nov 2007 12:42:03 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Alfred Perlstein" In-Reply-To: <20071121222319.GX44563@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> X-Google-Sender-Auth: d9d43e22c4144d87 Cc: arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 12:10:04 -0000 2007/11/21, Alfred Perlstein : > In summary, I am proposing (temporarily) making read-recursion > on rwlocks not supported in order to avoid livelock due to writer > starvation. > > More details: > > We currently have a problem with our implementation of rwlocks. > > I think this is a key issue for 7.x as what we decide to support > will have rammifications for many years to come. > > We do not support writer priority or "fair share" usage of rwlocks > between readers and writers. > > We have several choices to rectify this. > > 1. Disallow recursion on rwlocks (witness can be used to enforce this), > this simplifies rwlocks such that we can avoid deadlock when a single > reader is trying to recurse while a writer is pending. > > 2. Track ownership of rwlocks, this can be implemented with a "rwlock > stack" in the per-thread control block (struct thread). Using this > ownership information we can determine if someone is recursing and > allow them to continue recursing despite a pending write request. > > I think the most simple solution currently is to drop support for > recursive reads on rwlocks until we have the facility in place > to properly support starvation avoidance. > > Why is this important? > > Simply put, developers that quickly "fix" some portion of code, > whether that be a driver or part of the kernel proper who use read > recursion will open the system to writer starvation and hence the > system will destabilize, particulary for high load situations. > > I would like to get this in before 7.0-RELEASE becasue otherwise > we're forced to implement something like the above mentioned solution > #2, which will degrade performance for most use cases of rwlocks. As we alredy discussed on IRC, I'm for assuming no recursion on readers (recursion on writers doesn't play any role here, so it is ok maintaining the current behaviour) and adding a mechanism to WITNESS in order to catch cases where such condition is violated. In this way we don't penalyze fast cases. Do you alredy have a patch for the wakeup / sleeping algorithm? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 15:46:17 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0965E16A417; Thu, 22 Nov 2007 15:46:17 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id 83A7413C45D; Thu, 22 Nov 2007 15:46:16 +0000 (UTC) (envelope-from max@love2party.net) Received: from amd64.laiers.local (dslb-088-066-042-157.pools.arcor-ip.net [88.66.42.157]) by mrelayeu.kundenserver.de (node=mrelayeu4) with ESMTP (Nemesis) id 0ML21M-1IvEAG21rY-0002TY; Thu, 22 Nov 2007 16:40:28 +0100 From: Max Laier Organization: FreeBSD To: freebsd-arch@freebsd.org Date: Thu, 22 Nov 2007 16:40:51 +0100 User-Agent: KMail/1.9.7 References: <20071121222319.GX44563@elvis.mu.org> In-Reply-To: <20071121222319.GX44563@elvis.mu.org> X-Face: ,,8R(x[kmU]tKN@>gtH1yQE4aslGdu+2]; R]*pL,U>^H?)gW@49@wdJ`H<=?utf-8?q?=25=7D*=5FBD=0A=09U=5For=3D=5CmOZf764=26nYj=3DJYbR1PW0ud?=>|!~,,CPC.1-D$FG@0h3#'5"k{V]a~.<=?utf-8?q?mZ=7D44=23Se=7Em=0A=09Fe=7E=5C=5DX5B=5D=5Fxj?=(ykz9QKMw_l0C2AQ]}Ym8)fU MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1509446.0MGOy2rOSm"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200711221641.02484.max@love2party.net> X-Provags-ID: V01U2FsdGVkX19M5TllSccag4YS7BrKGwBHGR4wXCe3wDR3geT pEbz5/6S+GU2H7X36d6z3Pk+jUQqXpT4B4lSZ7nGYe+ZhvzVo+ AAdPjALACHIOn/m4DUwSm5x0VqsMqxCW4XjA6984fk= Cc: attilio@freebsd.org, Stephan Uphoff , Alfred Perlstein Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 15:46:17 -0000 --nextPart1509446.0MGOy2rOSm Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Wednesday 21 November 2007, Alfred Perlstein wrote: > In summary, I am proposing (temporarily) making read-recursion > on rwlocks not supported in order to avoid livelock due to writer > starvation. > > More details: > > We currently have a problem with our implementation of rwlocks. > > I think this is a key issue for 7.x as what we decide to support > will have rammifications for many years to come. > > We do not support writer priority or "fair share" usage of rwlocks > between readers and writers. > > We have several choices to rectify this. > > 1. Disallow recursion on rwlocks (witness can be used to enforce this), > this simplifies rwlocks such that we can avoid deadlock when a single > reader is trying to recurse while a writer is pending. > > 2. Track ownership of rwlocks, this can be implemented with a "rwlock > stack" in the per-thread control block (struct thread). Using this > ownership information we can determine if someone is recursing and > allow them to continue recursing despite a pending write request. > > I think the most simple solution currently is to drop support for > recursive reads on rwlocks until we have the facility in place > to properly support starvation avoidance. > > Why is this important? > > Simply put, developers that quickly "fix" some portion of code, > whether that be a driver or part of the kernel proper who use read > recursion will open the system to writer starvation and hence the > system will destabilize, particulary for high load situations. > > I would like to get this in before 7.0-RELEASE becasue otherwise > we're forced to implement something like the above mentioned solution > #2, which will degrade performance for most use cases of rwlocks. > > Comments? rwlocks are already used in places that do recursive reads. The one place= =20 I'm certain about is pfil(9) and we need a proper sollution for this. =20 Before rwlocks were used, I had a handrolled locking that supported both=20 read/write semantics and starvation avoidance - at the cost of failing to=20 allow futher read access when a writer asked for access. This however,=20 was quite application specific and not the most efficient implementation=20 either. If we were to disallow read recursion, we should have some generic lock=20 type that does allow it. rmlock(9)s seem to support full priority=20 propagation even for recursed readers. Can they be MFCed so that we have=20 an alternative? Are they considered ready for production? Should we=20 switch pfil(9) to them? It seems like a perfect match. Obviously rmlocks are not a general replacement for rwlocks, but in the=20 case of pfil they are the even better fit. =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --nextPart1509446.0MGOy2rOSm Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHRaMOXyyEoT62BG0RAtQVAJ9smaoqoOsMT9pBENSunGrceFSNxgCfcOJu RtJGG6S6YmCYlXdxl7GyKcg= =68y9 -----END PGP SIGNATURE----- --nextPart1509446.0MGOy2rOSm-- From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 15:53:36 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2B9416A46B for ; Thu, 22 Nov 2007 15:53:36 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.188]) by mx1.freebsd.org (Postfix) with ESMTP id 48C5413C4CC for ; Thu, 22 Nov 2007 15:53:36 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so2558576nfb for ; Thu, 22 Nov 2007 07:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=rmtzPiFkYAjkG/NpD9Hib5+LqyWdyWkF91ARxdT+b2E=; b=Ycot668WxLFDLoktiVE1+Th8bYK9IUkDMyYg4AzqhO1xOo7J0RMWPmt5BDi4TlxSO7Kd/F7svna9jrl73KHhY8F+kUzvuoDikTt8f347klIxMQ/wIIqop5dYLkOVdNj6iezBdCBpoUJVZhbT6GN/DzyQCHDGUK8zpAD2IkM5Mys= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=rLGRkJ4yTLS1hrZ3oZHejKmKTb3OrUuc6zKccdyhI7uvYfWcNpZYMkeVRPN3CzEZQoUfJ9Sk7b45idIcFw56WCwchxQ92SJOCZ6DdUUlrvFKp5BpRIR0pftJDBPYNnqtpJRtpGGGkDc0SylC5Ad3gLe2/H1JYXgQ+U6wARZ0yo8= Received: by 10.86.36.11 with SMTP id j11mr8568111fgj.1195746808071; Thu, 22 Nov 2007 07:53:28 -0800 (PST) Received: by 10.86.28.19 with HTTP; Thu, 22 Nov 2007 07:53:28 -0800 (PST) Message-ID: <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> Date: Thu, 22 Nov 2007 16:53:28 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Max Laier" In-Reply-To: <200711221641.02484.max@love2party.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> X-Google-Sender-Auth: eba5d8fe509ea2f1 Cc: Stephan Uphoff , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 15:53:36 -0000 2007/11/22, Max Laier : > > rwlocks are already used in places that do recursive reads. The one place > I'm certain about is pfil(9) and we need a proper sollution for this. > Before rwlocks were used, I had a handrolled locking that supported both > read/write semantics and starvation avoidance - at the cost of failing to > allow futher read access when a writer asked for access. This however, > was quite application specific and not the most efficient implementation > either. I'm not a pfil(9) expert, but for what I've heard, rmlocks should be really good for it, shouldn't them? The concept is that if we want to maintain fast paths for rwlock we cannot do too much tricks there. And you can really deadlock if you allow recursion on readers... > If we were to disallow read recursion, we should have some generic lock > type that does allow it. rmlock(9)s seem to support full priority > propagation even for recursed readers. Can they be MFCed so that we have > an alternative? Are they considered ready for production? Should we > switch pfil(9) to them? It seems like a perfect match. I've not looked over rmlocks so much, but as they track readers they can do priority propagation (if it is not implemented, it could be done -- not sure about the cost of the operation though). Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 16:26:45 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 156A316A418; Thu, 22 Nov 2007 16:26:45 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.179]) by mx1.freebsd.org (Postfix) with ESMTP id 8145413C458; Thu, 22 Nov 2007 16:26:44 +0000 (UTC) (envelope-from max@love2party.net) Received: from amd64.laiers.local (dslb-088-066-042-157.pools.arcor-ip.net [88.66.42.157]) by mrelayeu.kundenserver.de (node=mrelayeu2) with ESMTP (Nemesis) id 0MKwtQ-1IvEsR0LTV-0001MF; Thu, 22 Nov 2007 17:26:37 +0100 From: Max Laier Organization: FreeBSD To: "Attilio Rao" Date: Thu, 22 Nov 2007 17:26:18 +0100 User-Agent: KMail/1.9.7 References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> In-Reply-To: <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> X-Face: ,,8R(x[kmU]tKN@>gtH1yQE4aslGdu+2]; R]*pL,U>^H?)gW@49@wdJ`H<=?utf-8?q?=25=7D*=5FBD=0A=09U=5For=3D=5CmOZf764=26nYj=3DJYbR1PW0ud?=>|!~,,CPC.1-D$FG@0h3#'5"k{V]a~.<=?utf-8?q?mZ=7D44=23Se=7Em=0A=09Fe=7E=5C=5DX5B=5D=5Fxj?=(ykz9QKMw_l0C2AQ]}Ym8)fU MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3738284.cOpIyjHv6B"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200711221726.27108.max@love2party.net> X-Provags-ID: V01U2FsdGVkX1/33zApj4Rn60gPfuH+f30VOP1Jwqtc1NNzDqY UcaHGyrvHnfNcmxjK+836DaOQynRJIfMA02kN9rz9ZbLlzWJKr rGsWUTN0Mv614ba1OF2tyWKCxTjbjlS/9wMUt5Bv7o= Cc: Stephan Uphoff , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 16:26:45 -0000 --nextPart3738284.cOpIyjHv6B Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Thursday 22 November 2007, Attilio Rao wrote: > 2007/11/22, Max Laier : > > rwlocks are already used in places that do recursive reads. The one > > place I'm certain about is pfil(9) and we need a proper sollution for > > this. Before rwlocks were used, I had a handrolled locking that > > supported both read/write semantics and starvation avoidance - at the > > cost of failing to allow futher read access when a writer asked for > > access. This however, was quite application specific and not the > > most efficient implementation either. > > I'm not a pfil(9) expert, but for what I've heard, rmlocks should be > really good for it, shouldn't them? > > The concept is that if we want to maintain fast paths for rwlock we > cannot do too much tricks there. And you can really deadlock if you > allow recursion on readers... How about adding rwlock_try_rlock() which would do the following: 1) Only variant to allow[1] read recursion and ... 2) ... only if no outstanding write requests 3) Let the caller deal with failure This can be implemented statically, so no overhead in the fast path. The=20 caller is in the best position to decide if it is recursing or not -=20 could keep that info on the stack - and can either fail[2] or do a normal=20 rwlock_rlock() which would wait for the writer to enter and exit. [2] In most situation where you use read locks you can fail or roll back=20 carefully as you didn't change anything - obviously. In pfil - for=20 instance - we just dropped the packet if there was a writer waiting. [1] "allow" in terms of WITNESS - if that can be done. > > If we were to disallow read recursion, we should have some generic > > lock type that does allow it. rmlock(9)s seem to support full > > priority propagation even for recursed readers. Can they be MFCed so > > that we have an alternative? Are they considered ready for > > production? Should we switch pfil(9) to them? It seems like a > > perfect match. > > I've not looked over rmlocks so much, but as they track readers they > can do priority propagation (if it is not implemented, it could be > done -- not sure about the cost of the operation though). > > Attilio =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --nextPart3738284.cOpIyjHv6B Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHRa2zXyyEoT62BG0RAkc+AJ98NfsoXSxOL8tDhTm9bCBuUi/QhACcDnd9 vPh6R30xaq+pPREBtb5BE1w= =IlV/ -----END PGP SIGNATURE----- --nextPart3738284.cOpIyjHv6B-- From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 19:04:31 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82EA516A46C for ; Thu, 22 Nov 2007 19:04:31 +0000 (UTC) (envelope-from ups@freebsd.org) Received: from smtpout08.prod.mesa1.secureserver.net (smtpout08-04.prod.mesa1.secureserver.net [64.202.165.12]) by mx1.freebsd.org (Postfix) with SMTP id 701F213C478 for ; Thu, 22 Nov 2007 19:04:31 +0000 (UTC) (envelope-from ups@freebsd.org) Received: (qmail 25194 invoked from network); 22 Nov 2007 18:37:47 -0000 Received: from unknown (66.23.216.53) by smtpout08-04.prod.mesa1.secureserver.net (64.202.165.12) with ESMTP; 22 Nov 2007 18:37:47 -0000 Message-ID: <4745CC84.3040600@freebsd.org> Date: Thu, 22 Nov 2007 13:37:56 -0500 From: Stephan Uphoff User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Attilio Rao References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> In-Reply-To: <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 19:04:31 -0000 Attilio Rao wrote: > 2007/11/22, Max Laier : > >> rwlocks are already used in places that do recursive reads. The one place >> I'm certain about is pfil(9) and we need a proper sollution for this. >> Before rwlocks were used, I had a handrolled locking that supported both >> read/write semantics and starvation avoidance - at the cost of failing to >> allow futher read access when a writer asked for access. This however, >> was quite application specific and not the most efficient implementation >> either. >> > > I'm not a pfil(9) expert, but for what I've heard, rmlocks should be > really good for it, shouldn't them? > > The concept is that if we want to maintain fast paths for rwlock we > cannot do too much tricks there. And you can really deadlock if you > allow recursion on readers... > > >> If we were to disallow read recursion, we should have some generic lock >> type that does allow it. rmlock(9)s seem to support full priority >> propagation even for recursed readers. Can they be MFCed so that we have >> an alternative? Are they considered ready for production? Should we >> switch pfil(9) to them? It seems like a perfect match. >> > > I've not looked over rmlocks so much, but as they track readers they > can do priority propagation (if it is not implemented, it could be > done -- not sure about the cost of the operation though). > Yes - rmlock does priority propagation to readers. (One reader at a time until no more readers are left) > Attilio > > > From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 19:12:42 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13BF516A41B for ; Thu, 22 Nov 2007 19:12:42 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from mx28.mail.ru (mx28.mail.ru [194.67.23.67]) by mx1.freebsd.org (Postfix) with ESMTP id 9BA0F13C461 for ; Thu, 22 Nov 2007 19:12:41 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from f106.mail.ru (f106.mail.ru [194.67.57.205]) by mx28.mail.ru (mPOP.Fallback_MX) with ESMTP id CADBE757C33 for ; Thu, 22 Nov 2007 19:44:48 +0300 (MSK) Received: from mail by f106.mail.ru with local id 1IvFAP-000BFB-00; Thu, 22 Nov 2007 19:44:41 +0300 Received: from [89.208.20.114] by koi.mail.ru with HTTP; Thu, 22 Nov 2007 19:44:41 +0300 From: dima <_pppp@mail.ru> To: Alfred Perlstein Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [89.208.20.114] Date: Thu, 22 Nov 2007 19:44:41 +0300 In-Reply-To: <20071121222319.GX44563@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 19:12:42 -0000 > In summary, I am proposing (temporarily) making read-recursion > on rwlocks not supported in order to avoid livelock due to writer > starvation. It's not the matter of recursion, but the implementation itself. Comments from the code: * Note that we don't make any attempt to try to block read * locks once a writer has blocked on the lock. The reason is * that we currently allow for read locks to recurse and we * don't keep track of all the holders of read locks. Thus, if * we were to block readers once a writer blocked and a reader * tried to recurse on their reader lock after a writer had * blocked we would end up in a deadlock since the reader would * be blocked on the writer, and the writer would be blocked * waiting for the reader to release its original read lock. Such a design would provide writer starvation regardless the presence of recursion. If a writer is waiting for the lock, no more readers should be allowed. It's a simple rule to avoid writer starvation in rw-locks. Recursion does need an accurate accounting of all the current readers. I've posted a reference implementation of rw-locks to this mailing list about a year ago. My proposal was to use an array to account the current readers (the array can be handled without any additional locking). It limits the maximum read contention, though. > More details: > > We currently have a problem with our implementation of rwlocks. > > I think this is a key issue for 7.x as what we decide to support > will have rammifications for many years to come. > > We do not support writer priority or "fair share" usage of rwlocks > between readers and writers. > > We have several choices to rectify this. > > 1. Disallow recursion on rwlocks (witness can be used to enforce this), > this simplifies rwlocks such that we can avoid deadlock when a single > reader is trying to recurse while a writer is pending. > > 2. Track ownership of rwlocks, this can be implemented with a "rwlock > stack" in the per-thread control block (struct thread). Using this > ownership information we can determine if someone is recursing and > allow them to continue recursing despite a pending write request. > > I think the most simple solution currently is to drop support for > recursive reads on rwlocks until we have the facility in place > to properly support starvation avoidance. > > Why is this important? > > Simply put, developers that quickly "fix" some portion of code, > whether that be a driver or part of the kernel proper who use read > recursion will open the system to writer starvation and hence the > system will destabilize, particulary for high load situations. > > I would like to get this in before 7.0-RELEASE becasue otherwise > we're forced to implement something like the above mentioned solution > #2, which will degrade performance for most use cases of rwlocks. > > Comments? > > -- > - Alfred Perlstein > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 19:58:28 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73E7916A41A for ; Thu, 22 Nov 2007 19:58:28 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 7F33713C4E7 for ; Thu, 22 Nov 2007 19:58:28 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id D79EC1A4D7C; Thu, 22 Nov 2007 11:58:23 -0800 (PST) Date: Thu, 22 Nov 2007 11:58:23 -0800 From: Alfred Perlstein To: dima <_pppp@mail.ru> Message-ID: <20071122195823.GH44563@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 19:58:28 -0000 * dima <_pppp@mail.ru> [071122 08:44] wrote: > > In summary, I am proposing (temporarily) making read-recursion > > on rwlocks not supported in order to avoid livelock due to writer > > starvation. > > It's not the matter of recursion, but the implementation itself. > Comments from the code: > > * Note that we don't make any attempt to try to block read > * locks once a writer has blocked on the lock. The reason is > * that we currently allow for read locks to recurse and we > * don't keep track of all the holders of read locks. Thus, if > * we were to block readers once a writer blocked and a reader > * tried to recurse on their reader lock after a writer had > * blocked we would end up in a deadlock since the reader would > * be blocked on the writer, and the writer would be blocked > * waiting for the reader to release its original read lock. > > Such a design would provide writer starvation regardless the presence of recursion. If a writer is waiting for the lock, no more readers should be allowed. It's a simple rule to avoid writer starvation in rw-locks. Recursion does need an accurate accounting of all the current readers. > I've posted a reference implementation of rw-locks to this mailing list about a year ago. My proposal was to use an array to account the current readers (the array can be handled without any additional locking). It limits the maximum read contention, though. Dima, if you put the list inside the thread (as mentioned in my email) then it scales to any number of readers. Although the number of simultanious discrete readlocks is limited by a constant. >From talking to Attilio and John, they do not want that as it is "slow". -Alfred > > > More details: > > > > We currently have a problem with our implementation of rwlocks. > > > > I think this is a key issue for 7.x as what we decide to support > > will have rammifications for many years to come. > > > > We do not support writer priority or "fair share" usage of rwlocks > > between readers and writers. > > > > We have several choices to rectify this. > > > > 1. Disallow recursion on rwlocks (witness can be used to enforce this), > > this simplifies rwlocks such that we can avoid deadlock when a single > > reader is trying to recurse while a writer is pending. > > > > 2. Track ownership of rwlocks, this can be implemented with a "rwlock > > stack" in the per-thread control block (struct thread). Using this > > ownership information we can determine if someone is recursing and > > allow them to continue recursing despite a pending write request. > > > > I think the most simple solution currently is to drop support for > > recursive reads on rwlocks until we have the facility in place > > to properly support starvation avoidance. > > > > Why is this important? > > > > Simply put, developers that quickly "fix" some portion of code, > > whether that be a driver or part of the kernel proper who use read > > recursion will open the system to writer starvation and hence the > > system will destabilize, particulary for high load situations. > > > > I would like to get this in before 7.0-RELEASE becasue otherwise > > we're forced to implement something like the above mentioned solution > > #2, which will degrade performance for most use cases of rwlocks. > > > > Comments? > > > > -- > > - Alfred Perlstein > > _______________________________________________ > > freebsd-arch@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > -- - Alfred Perlstein From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 20:00:04 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 452A816A417; Thu, 22 Nov 2007 20:00:02 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 43CA913C447; Thu, 22 Nov 2007 20:00:02 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 50A601A4D80; Thu, 22 Nov 2007 12:00:01 -0800 (PST) Date: Thu, 22 Nov 2007 12:00:01 -0800 From: Alfred Perlstein To: Max Laier Message-ID: <20071122200001.GI44563@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200711221641.02484.max@love2party.net> User-Agent: Mutt/1.4.2.3i Cc: attilio@freebsd.org, Stephan Uphoff , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 20:00:04 -0000 * Max Laier [071122 07:46] wrote: > On Wednesday 21 November 2007, Alfred Perlstein wrote: > > In summary, I am proposing (temporarily) making read-recursion > > on rwlocks not supported in order to avoid livelock due to writer > > starvation. > > > > rwlocks are already used in places that do recursive reads. The one place Max, I think what will happen is that we will mark further uses or read locks as recursive as "not supported", perhaps witness can temporarily grow a flag to ignore recursive read ops until the existing infrastructure is fixed. I will not get into alternatives for pfil, as it seems you've mostly worked it out. -Alfred From owner-freebsd-arch@FreeBSD.ORG Thu Nov 22 22:23:32 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24DD316A41A; Thu, 22 Nov 2007 22:23:32 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from mx28.mail.ru (mx28.mail.ru [194.67.23.67]) by mx1.freebsd.org (Postfix) with ESMTP id A0F5813C45B; Thu, 22 Nov 2007 22:23:31 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from f41.mail.ru (f41.mail.ru [194.67.57.79]) by mx28.mail.ru (mPOP.Fallback_MX) with ESMTP id 48D4C757FC9; Thu, 22 Nov 2007 23:19:59 +0300 (MSK) Received: from mail by f41.mail.ru with local id 1IvIWg-0007u7-00; Thu, 22 Nov 2007 23:19:54 +0300 Received: from [89.208.20.114] by koi.mail.ru with HTTP; Thu, 22 Nov 2007 23:19:54 +0300 From: dima <_pppp@mail.ru> To: Alfred Perlstein Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [89.208.20.114] Date: Thu, 22 Nov 2007 23:19:54 +0300 In-Reply-To: <20071122195823.GH44563@elvis.mu.org> References: <20071122195823.GH44563@elvis.mu.org> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: arch@freebsd.org Subject: Re[2]: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2007 22:23:32 -0000 > > > In summary, I am proposing (temporarily) making read-recursion > > > on rwlocks not supported in order to avoid livelock due to writer > > > starvation. > > > > It's not the matter of recursion, but the implementation itself. > > Comments from the code: > > > > * Note that we don't make any attempt to try to block read > > * locks once a writer has blocked on the lock. The reason is > > * that we currently allow for read locks to recurse and we > > * don't keep track of all the holders of read locks. Thus, if > > * we were to block readers once a writer blocked and a reader > > * tried to recurse on their reader lock after a writer had > > * blocked we would end up in a deadlock since the reader would > > * be blocked on the writer, and the writer would be blocked > > * waiting for the reader to release its original read lock. > > > > Such a design would provide writer starvation regardless the presence of recursion. If a writer is waiting for the lock, no more readers should be allowed. It's a simple rule to avoid writer starvation in rw-locks. Recursion does need an accurate accounting of all the current readers. > > I've posted a reference implementation of rw-locks to this mailing list about a year ago. My proposal was to use an array to account the current readers (the array can be handled without any additional locking). It limits the maximum read contention, though. > > Dima, if you put the list inside the thread (as mentioned in my email) > then it scales to any number of readers. Although the number of > simultanious discrete readlocks is limited by a constant. Well, I think struct thread is bloated enough to add more fields into it. > From talking to Attilio and John, they do not want that as it is "slow". The current design is probably fast, but incorrect. Writers will be starved even if we deny recursive read-locks. There are 2 ways to fix that: 1. Forbid recursive read-locks *and* add a flag which would mean "a writer is waiting" (and prohibit future read-lock acquisition). 2. Allow recursive read-locks, add the same flag, and add an accounting of all the current readers (so, only recursive read-lock acquisitions would be accepted if a writer is waiting). > -Alfred > > > > > > More details: > > > > > > We currently have a problem with our implementation of rwlocks. > > > > > > I think this is a key issue for 7.x as what we decide to support > > > will have rammifications for many years to come. > > > > > > We do not support writer priority or "fair share" usage of rwlocks > > > between readers and writers. > > > > > > We have several choices to rectify this. > > > > > > 1. Disallow recursion on rwlocks (witness can be used to enforce this), > > > this simplifies rwlocks such that we can avoid deadlock when a single > > > reader is trying to recurse while a writer is pending. > > > > > > 2. Track ownership of rwlocks, this can be implemented with a "rwlock > > > stack" in the per-thread control block (struct thread). Using this > > > ownership information we can determine if someone is recursing and > > > allow them to continue recursing despite a pending write request. > > > > > > I think the most simple solution currently is to drop support for > > > recursive reads on rwlocks until we have the facility in place > > > to properly support starvation avoidance. > > > > > > Why is this important? > > > > > > Simply put, developers that quickly "fix" some portion of code, > > > whether that be a driver or part of the kernel proper who use read > > > recursion will open the system to writer starvation and hence the > > > system will destabilize, particulary for high load situations. > > > > > > I would like to get this in before 7.0-RELEASE becasue otherwise > > > we're forced to implement something like the above mentioned solution > > > #2, which will degrade performance for most use cases of rwlocks. > > > > > > Comments? > > > > > > -- > > > - Alfred Perlstein > > > _______________________________________________ > > > freebsd-arch@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > > > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > > > > -- > - Alfred Perlstein > From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 00:55:53 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3ACBA16A419 for ; Fri, 23 Nov 2007 00:55:53 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.186]) by mx1.freebsd.org (Postfix) with ESMTP id CD5AE13C459 for ; Fri, 23 Nov 2007 00:55:52 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so2682852nfb for ; Thu, 22 Nov 2007 16:55:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=IDd3MX0XNR8zqvq71/8rwbjkmj2xPsqsJ8utixyw87U=; b=VieLECbMTsASdmotB5733QdrbWvyFYOM0brZcLJSPFdTGOfVda/n8vZxxEC4C7UwFYFrZOKvnjRf0FgqqWevcdnjRoYjqToQVn821VU+sN7E5LYY4kv5y8rl559XQmBLCBzOdMlZm3KFU0kpQOXYw/WlVhJcW/8eQIZVy+fh+Qk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=uTr+04QS/ZbYugNFH8gM+E5c9CsCzf15uNmplcIyTpLH4HIkhW1JNE/8gdU/CSHTx2Zkjf89e4qDARhIpfjiroNTHO06668hvTn5FO90UdWa05gh7iW0QRN35xLE6/rKZuO9ZQSM3/zMQMenvwjyFq+4zRqSa0gHJeWPQRrOHb4= Received: by 10.86.91.12 with SMTP id o12mr8969826fgb.1195779346322; Thu, 22 Nov 2007 16:55:46 -0800 (PST) Received: by 10.86.28.19 with HTTP; Thu, 22 Nov 2007 16:55:46 -0800 (PST) Message-ID: <3bbf2fe10711221655n33d4d0c7mbeece529a3d08642@mail.gmail.com> Date: Fri, 23 Nov 2007 01:55:46 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: dima <_pppp@mail.ru> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> X-Google-Sender-Auth: fb23b2f05dabc45f Cc: arch@freebsd.org, Alfred Perlstein Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 00:55:53 -0000 2007/11/22, dima <_pppp@mail.ru>: > > In summary, I am proposing (temporarily) making read-recursion > > on rwlocks not supported in order to avoid livelock due to writer > > starvation. > > It's not the matter of recursion, but the implementation itself. > Comments from the code: > > * Note that we don't make any attempt to try to block read > * locks once a writer has blocked on the lock. The reason is > * that we currently allow for read locks to recurse and we > * don't keep track of all the holders of read locks. Thus, if > * we were to block readers once a writer blocked and a reader > * tried to recurse on their reader lock after a writer had > * blocked we would end up in a deadlock since the reader would > * be blocked on the writer, and the writer would be blocked > * waiting for the reader to release its original read lock. > > Such a design would provide writer starvation regardless the presence of recursion. If a writer is waiting for the lock, no more readers should be allowed. It's a simple rule to avoid writer starvation in rw-locks. Recursion does need an accurate accounting of all the current readers. > I've posted a reference implementation of rw-locks to this mailing list about a year ago. My proposal was to use an array to account the current readers (the array can be handled without any additional locking). It limits the maximum read contention, though. The real problem with this is that currently we have a fast path which is only an atomic add. What you propose will make the fast path too slow (not mentioning the limitation), and this is somethign we should avoid if possible. Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 08:23:39 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7158816A418; Fri, 23 Nov 2007 08:23:39 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 53C3413C459; Fri, 23 Nov 2007 08:23:39 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 21FF41A4D7C; Fri, 23 Nov 2007 00:23:39 -0800 (PST) Date: Fri, 23 Nov 2007 00:23:39 -0800 From: Alfred Perlstein To: Max Laier Message-ID: <20071123082339.GN44563@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200711221726.27108.max@love2party.net> User-Agent: Mutt/1.4.2.3i Cc: Attilio Rao , Stephan Uphoff , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 08:23:39 -0000 * Max Laier [071122 14:40] wrote: > On Thursday 22 November 2007, Attilio Rao wrote: > > 2007/11/22, Max Laier : > > > rwlocks are already used in places that do recursive reads. The one > > > place I'm certain about is pfil(9) and we need a proper sollution for > > > this. Before rwlocks were used, I had a handrolled locking that > > > supported both read/write semantics and starvation avoidance - at the > > > cost of failing to allow futher read access when a writer asked for > > > access. This however, was quite application specific and not the > > > most efficient implementation either. > > > > I'm not a pfil(9) expert, but for what I've heard, rmlocks should be > > really good for it, shouldn't them? > > > > The concept is that if we want to maintain fast paths for rwlock we > > cannot do too much tricks there. And you can really deadlock if you > > allow recursion on readers... > > How about adding rwlock_try_rlock() which would do the following: > 1) Only variant to allow[1] read recursion and ... > 2) ... only if no outstanding write requests > 3) Let the caller deal with failure > > This can be implemented statically, so no overhead in the fast path. The > caller is in the best position to decide if it is recursing or not - > could keep that info on the stack - and can either fail[2] or do a normal > rwlock_rlock() which would wait for the writer to enter and exit. > > [2] In most situation where you use read locks you can fail or roll back > carefully as you didn't change anything - obviously. In pfil - for > instance - we just dropped the packet if there was a writer waiting. > > [1] "allow" in terms of WITNESS - if that can be done. The problem is that there is no tracking in the common case (without additional overhead), so you can't know if you're recursing, unless you track it yourself. -Alfred From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 08:24:59 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE8AB16A420 for ; Fri, 23 Nov 2007 08:24:59 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 96A5213C45B for ; Fri, 23 Nov 2007 08:24:59 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 7EEC51A4D7C; Fri, 23 Nov 2007 00:24:59 -0800 (PST) Date: Fri, 23 Nov 2007 00:24:59 -0800 From: Alfred Perlstein To: dima <_pppp@mail.ru> Message-ID: <20071123082459.GO44563@elvis.mu.org> References: <20071122195823.GH44563@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: arch@freebsd.org Subject: Re: Re[2]: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 08:24:59 -0000 * dima <_pppp@mail.ru> [071122 19:20] wrote: > > The current design is probably fast, but incorrect. Writers will be starved even if we deny recursive read-locks. There are 2 ways to fix that: > 1. Forbid recursive read-locks *and* add a flag which would mean "a writer is waiting" (and prohibit future read-lock acquisition). Yes, this would be done later. > 2. Allow recursive read-locks, add the same flag, and add an accounting of all the current readers (so, only recursive read-lock acquisitions would be accepted if a writer is waiting). Probably not going to happen as the extra bookkeeping is supposedly cost prohibitive. -Alfred From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 08:45:21 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 689A116A41A for ; Fri, 23 Nov 2007 08:45:21 +0000 (UTC) (envelope-from ups@freebsd.org) Received: from smtpout07.prod.mesa1.secureserver.net (smtpout07-04.prod.mesa1.secureserver.net [64.202.165.233]) by mx1.freebsd.org (Postfix) with SMTP id 3B7E013C46A for ; Fri, 23 Nov 2007 08:45:21 +0000 (UTC) (envelope-from ups@freebsd.org) Received: (qmail 1240 invoked from network); 23 Nov 2007 08:45:19 -0000 Received: from unknown (66.23.216.53) by smtpout07-04.prod.mesa1.secureserver.net (64.202.165.233) with ESMTP; 23 Nov 2007 08:45:17 -0000 Message-ID: <47469328.8020404@freebsd.org> Date: Fri, 23 Nov 2007 03:45:28 -0500 From: Stephan Uphoff User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alfred Perlstein References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> In-Reply-To: <20071123082339.GN44563@elvis.mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Attilio Rao , Max Laier , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 08:45:21 -0000 Alfred Perlstein wrote: > * Max Laier [071122 14:40] wrote: > >> On Thursday 22 November 2007, Attilio Rao wrote: >> >>> 2007/11/22, Max Laier : >>> >>>> rwlocks are already used in places that do recursive reads. The one >>>> place I'm certain about is pfil(9) and we need a proper sollution for >>>> this. Before rwlocks were used, I had a handrolled locking that >>>> supported both read/write semantics and starvation avoidance - at the >>>> cost of failing to allow futher read access when a writer asked for >>>> access. This however, was quite application specific and not the >>>> most efficient implementation either. >>>> >>> I'm not a pfil(9) expert, but for what I've heard, rmlocks should be >>> really good for it, shouldn't them? >>> >>> The concept is that if we want to maintain fast paths for rwlock we >>> cannot do too much tricks there. And you can really deadlock if you >>> allow recursion on readers... >>> >> How about adding rwlock_try_rlock() which would do the following: >> 1) Only variant to allow[1] read recursion and ... >> 2) ... only if no outstanding write requests >> 3) Let the caller deal with failure >> >> This can be implemented statically, so no overhead in the fast path. The >> caller is in the best position to decide if it is recursing or not - >> could keep that info on the stack - and can either fail[2] or do a normal >> rwlock_rlock() which would wait for the writer to enter and exit. >> >> [2] In most situation where you use read locks you can fail or roll back >> carefully as you didn't change anything - obviously. In pfil - for >> instance - we just dropped the packet if there was a writer waiting. >> >> [1] "allow" in terms of WITNESS - if that can be done. >> > > The problem is that there is no tracking in the common case (without > additional overhead), so you can't know if you're recursing, unless > you track it yourself. > > -Alfred > > > I talked with Attilio about that on IRC. Most common cases of writer starvation (but not all) could be solved by keeping a per thread count of shared acquired rwlocks. If a rwlock is currently locked as shared/read AND a thread is blocked on it to lock it exclusively/write - then new shared/read locks will only be granted to thread that already has a shared lock. (per thread shared counter is non zero) To be honest I am a bit twitchy about a lock without priority propagation - especially since in FreeBSD threads run with user priority in kernel space and can get preempted. Stephan From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 09:24:20 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 507F616A47B; Fri, 23 Nov 2007 09:24:20 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 2A01613C465; Fri, 23 Nov 2007 09:24:19 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 91B901A4D7E; Fri, 23 Nov 2007 01:24:15 -0800 (PST) Date: Fri, 23 Nov 2007 01:24:15 -0800 From: Alfred Perlstein To: Stephan Uphoff Message-ID: <20071123092415.GP44563@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47469328.8020404@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Attilio Rao , Max Laier , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 09:24:20 -0000 * Stephan Uphoff [071123 00:46] wrote: > Alfred Perlstein wrote: > >* Max Laier [071122 14:40] wrote: > > > >>On Thursday 22 November 2007, Attilio Rao wrote: > >> > >>>2007/11/22, Max Laier : > >>> > >>>>rwlocks are already used in places that do recursive reads. The one > >>>>place I'm certain about is pfil(9) and we need a proper sollution for > >>>>this. Before rwlocks were used, I had a handrolled locking that > >>>>supported both read/write semantics and starvation avoidance - at the > >>>>cost of failing to allow futher read access when a writer asked for > >>>>access. This however, was quite application specific and not the > >>>>most efficient implementation either. > >>>> > >>>I'm not a pfil(9) expert, but for what I've heard, rmlocks should be > >>>really good for it, shouldn't them? > >>> > >>>The concept is that if we want to maintain fast paths for rwlock we > >>>cannot do too much tricks there. And you can really deadlock if you > >>>allow recursion on readers... > >>> > >>How about adding rwlock_try_rlock() which would do the following: > >> 1) Only variant to allow[1] read recursion and ... > >> 2) ... only if no outstanding write requests > >> 3) Let the caller deal with failure > >> > >>This can be implemented statically, so no overhead in the fast path. The > >>caller is in the best position to decide if it is recursing or not - > >>could keep that info on the stack - and can either fail[2] or do a normal > >>rwlock_rlock() which would wait for the writer to enter and exit. > >> > >>[2] In most situation where you use read locks you can fail or roll back > >>carefully as you didn't change anything - obviously. In pfil - for > >>instance - we just dropped the packet if there was a writer waiting. > >> > >>[1] "allow" in terms of WITNESS - if that can be done. > >> > > > >The problem is that there is no tracking in the common case (without > >additional overhead), so you can't know if you're recursing, unless > >you track it yourself. > > > >-Alfred > > > > > > > I talked with Attilio about that on IRC. > Most common cases of writer starvation (but not all) could be solved by > keeping a per thread count of shared acquired rwlocks. > If a rwlock is currently locked as shared/read AND a thread is blocked > on it to lock it exclusively/write - then new shared/read locks > will only be granted to thread that already has a shared lock. (per > thread shared counter is non zero) > > To be honest I am a bit twitchy about a lock without priority > propagation - especially since in FreeBSD threads run with user priority > in kernel > space and can get preempted. > > Stephan That's an interesting hack, I guess it could be done. I would still like to disallow recursion. Can we come to a concensus on that? -- - Alfred Perlstein From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 11:27:44 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6EF916A417; Fri, 23 Nov 2007 11:27:44 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.179]) by mx1.freebsd.org (Postfix) with ESMTP id 2B9F413C4DB; Fri, 23 Nov 2007 11:27:43 +0000 (UTC) (envelope-from max@love2party.net) Received: from amd64.laiers.local (dslb-088-066-054-138.pools.arcor-ip.net [88.66.54.138]) by mrelayeu.kundenserver.de (node=mrelayeu0) with ESMTP (Nemesis) id 0MKwh2-1IvWh40ckc-0002py; Fri, 23 Nov 2007 12:27:35 +0100 From: Max Laier Organization: FreeBSD To: freebsd-arch@freebsd.org Date: Fri, 23 Nov 2007 12:28:33 +0100 User-Agent: KMail/1.9.7 References: <20071121222319.GX44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> In-Reply-To: <20071123092415.GP44563@elvis.mu.org> X-Face: ,,8R(x[kmU]tKN@>gtH1yQE4aslGdu+2]; R]*pL,U>^H?)gW@49@wdJ`H<=?utf-8?q?=25=7D*=5FBD=0A=09U=5For=3D=5CmOZf764=26nYj=3DJYbR1PW0ud?=>|!~,,CPC.1-D$FG@0h3#'5"k{V]a~.<=?utf-8?q?mZ=7D44=23Se=7Em=0A=09Fe=7E=5C=5DX5B=5D=5Fxj?=(ykz9QKMw_l0C2AQ]}Ym8)fU MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1382413.3MpHy4ZRLo"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200711231228.41382.max@love2party.net> X-Provags-ID: V01U2FsdGVkX1+rFkS8NKA7z/NP2NIxJJWf/SZhSlK0xVoJe3B TU8JDarPBPpBdSMzYVo1hquzC9q7LD4HKd2Kuo9CzIPIgPxApn REsibWrLt3HNyILmbAHrtFIRwp2LOfIcztXnObyJ/I= Cc: Stephan Uphoff , Attilio Rao , Alfred Perlstein Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 11:27:44 -0000 --nextPart1382413.3MpHy4ZRLo Content-Type: multipart/mixed; boundary="Boundary-01=_klrRHoFB8DwWuBG" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-01=_klrRHoFB8DwWuBG Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Friday 23 November 2007, Alfred Perlstein wrote: > * Stephan Uphoff [071123 00:46] wrote: > > Alfred Perlstein wrote: > > >* Max Laier [071122 14:40] wrote: > > >>On Thursday 22 November 2007, Attilio Rao wrote: > > >>>2007/11/22, Max Laier : > > >>>>rwlocks are already used in places that do recursive reads. The > > >>>> one place I'm certain about is pfil(9) and we need a proper > > >>>> sollution for this. Before rwlocks were used, I had a handrolled > > >>>> locking that supported both read/write semantics and starvation > > >>>> avoidance - at the cost of failing to allow futher read access > > >>>> when a writer asked for access. This however, was quite > > >>>> application specific and not the most efficient implementation > > >>>> either. > > >>> > > >>>I'm not a pfil(9) expert, but for what I've heard, rmlocks should > > >>> be really good for it, shouldn't them? > > >>> > > >>>The concept is that if we want to maintain fast paths for rwlock > > >>> we cannot do too much tricks there. And you can really deadlock > > >>> if you allow recursion on readers... > > >> > > >>How about adding rwlock_try_rlock() which would do the following: > > >> 1) Only variant to allow[1] read recursion and ... > > >> 2) ... only if no outstanding write requests > > >> 3) Let the caller deal with failure > > >> > > >>This can be implemented statically, so no overhead in the fast > > >> path. The caller is in the best position to decide if it is > > >> recursing or not - could keep that info on the stack - and can > > >> either fail[2] or do a normal rwlock_rlock() which would wait for > > >> the writer to enter and exit. > > >> > > >>[2] In most situation where you use read locks you can fail or roll > > >> back carefully as you didn't change anything - obviously. In pfil > > >> - for instance - we just dropped the packet if there was a writer > > >> waiting. > > >> > > >>[1] "allow" in terms of WITNESS - if that can be done. > > > > > >The problem is that there is no tracking in the common case (without > > >additional overhead), so you can't know if you're recursing, unless > > >you track it yourself. > > > > > >-Alfred > > > > I talked with Attilio about that on IRC. > > Most common cases of writer starvation (but not all) could be solved > > by keeping a per thread count of shared acquired rwlocks. > > If a rwlock is currently locked as shared/read AND a thread is > > blocked on it to lock it exclusively/write - then new shared/read > > locks will only be granted to thread that already has a shared lock. > > (per thread shared counter is non zero) > > > > To be honest I am a bit twitchy about a lock without priority > > propagation - especially since in FreeBSD threads run with user > > priority in kernel > > space and can get preempted. > > > > Stephan > > That's an interesting hack, I guess it could be done. > > I would still like to disallow recursion. > > Can we come to a concensus on that? As long as we properly take care of all the fallout before doing so, I'm=20 all for it. Attached is a diff to switch pfil over to rmlocks which works for me. =20 I'll post it to -net and do the switch in a few days unless I hear=20 otherwise. =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --Boundary-01=_klrRHoFB8DwWuBG Content-Type: text/x-diff; charset="iso-8859-1"; name="pfil.rmlock.diff" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="pfil.rmlock.diff" Index: pfil.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/sys/net/pfil.c,v retrieving revision 1.14 diff -u -r1.14 pfil.c =2D-- pfil.c 2 Feb 2006 03:13:15 -0000 1.14 +++ pfil.c 23 Nov 2007 09:06:12 -0000 @@ -34,7 +34,7 @@ #include #include #include =2D#include +#include #include #include #include @@ -66,11 +66,12 @@ pfil_run_hooks(struct pfil_head *ph, struct mbuf **mp, struct ifnet *ifp, int dir, struct inpcb *inp) { + struct rm_priotracker rmpt; struct packet_filter_hook *pfh; struct mbuf *m =3D *mp; int rv =3D 0; =20 =2D PFIL_RLOCK(ph); + PFIL_RLOCK(ph, &rmpt); KASSERT(ph->ph_nhooks >=3D 0, ("Pfil hook count dropped < 0")); for (pfh =3D pfil_hook_get(dir, ph); pfh !=3D NULL; pfh =3D TAILQ_NEXT(pfh, pfil_link)) { @@ -80,7 +81,7 @@ break; } } =2D PFIL_RUNLOCK(ph); + PFIL_RUNLOCK(ph, &rmpt); =09 *mp =3D m; return (rv); @@ -104,7 +105,7 @@ } PFIL_LIST_UNLOCK(); =20 =2D rw_init(&ph->ph_mtx, "PFil hook read/write mutex"); + PFIL_LOCK_INIT(ph); PFIL_WLOCK(ph); ph->ph_nhooks =3D 0; =20 @@ -143,7 +144,7 @@ free(pfh, M_IFADDR); TAILQ_FOREACH_SAFE(pfh, &ph->ph_out, pfil_link, pfnext) free(pfh, M_IFADDR); =2D rw_destroy(&ph->ph_mtx); + PFIL_LOCK_DESTROY(ph); =09 return (0); } Index: pfil.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/sys/net/pfil.h,v retrieving revision 1.16 diff -u -r1.16 pfil.h =2D-- pfil.h 8 Jun 2007 12:43:25 -0000 1.16 +++ pfil.h 23 Nov 2007 10:07:52 -0000 @@ -37,7 +37,7 @@ #include #include #include =2D#include +#include =20 struct mbuf; struct ifnet; @@ -69,7 +69,7 @@ pfil_list_t ph_out; int ph_type; int ph_nhooks; =2D struct rwlock ph_mtx; + struct rmlock ph_lock; union { u_long phu_val; void *phu_ptr; @@ -93,10 +93,13 @@ struct pfil_head *pfil_head_get(int, u_long); =20 #define PFIL_HOOKED(p) ((p)->ph_nhooks > 0) =2D#define PFIL_RLOCK(p) rw_rlock(&(p)->ph_mtx) =2D#define PFIL_WLOCK(p) rw_wlock(&(p)->ph_mtx) =2D#define PFIL_RUNLOCK(p) rw_runlock(&(p)->ph_mtx) =2D#define PFIL_WUNLOCK(p) rw_wunlock(&(p)->ph_mtx) +#define PFIL_LOCK_INIT(p) \ + rm_init(&(p)->ph_lock, "PFil hook read/write mutex", LO_RECURSABLE) +#define PFIL_LOCK_DESTROY(p) rm_destroy(&(p)->ph_lock) +#define PFIL_RLOCK(p, t) rm_rlock(&(p)->ph_lock, (t)) +#define PFIL_WLOCK(p) rm_wlock(&(p)->ph_lock) +#define PFIL_RUNLOCK(p, t) rm_runlock(&(p)->ph_lock, (t)) +#define PFIL_WUNLOCK(p) rm_wunlock(&(p)->ph_lock) #define PFIL_LIST_LOCK() mtx_lock(&pfil_global_lock) #define PFIL_LIST_UNLOCK() mtx_unlock(&pfil_global_lock) =20 --Boundary-01=_klrRHoFB8DwWuBG-- --nextPart1382413.3MpHy4ZRLo Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHRrlpXyyEoT62BG0RAqC0AKCBuMmC7YmsW331yCGrGzdoucy+QQCfZHi6 pVviPYRmNUns7i6x0jUpMzU= =9OYH -----END PGP SIGNATURE----- --nextPart1382413.3MpHy4ZRLo-- From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 15:57:02 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2356716A41A for ; Fri, 23 Nov 2007 15:57:02 +0000 (UTC) (envelope-from ups@freebsd.org) Received: from smtpauth13.prod.mesa1.secureserver.net (smtpauth13.prod.mesa1.secureserver.net [64.202.165.37]) by mx1.freebsd.org (Postfix) with SMTP id 8B85613C458 for ; Fri, 23 Nov 2007 15:57:01 +0000 (UTC) (envelope-from ups@freebsd.org) Received: (qmail 24905 invoked from network); 23 Nov 2007 15:57:00 -0000 Received: from unknown (66.23.216.53) by smtpauth13.prod.mesa1.secureserver.net (64.202.165.37) with ESMTP; 23 Nov 2007 15:57:00 -0000 Message-ID: <4746F858.4070301@freebsd.org> Date: Fri, 23 Nov 2007 10:57:12 -0500 From: Stephan Uphoff User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alfred Perlstein References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> In-Reply-To: <20071123092415.GP44563@elvis.mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Attilio Rao , Max Laier , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 15:57:02 -0000 Alfred Perlstein wrote: > * Stephan Uphoff [071123 00:46] wrote: > >> Alfred Perlstein wrote: >> >>> * Max Laier [071122 14:40] wrote: >>> >>> >>>> On Thursday 22 November 2007, Attilio Rao wrote: >>>> >>>> >>>>> 2007/11/22, Max Laier : >>>>> >>>>> >>>>>> rwlocks are already used in places that do recursive reads. The one >>>>>> place I'm certain about is pfil(9) and we need a proper sollution for >>>>>> this. Before rwlocks were used, I had a handrolled locking that >>>>>> supported both read/write semantics and starvation avoidance - at the >>>>>> cost of failing to allow futher read access when a writer asked for >>>>>> access. This however, was quite application specific and not the >>>>>> most efficient implementation either. >>>>>> >>>>>> >>>>> I'm not a pfil(9) expert, but for what I've heard, rmlocks should be >>>>> really good for it, shouldn't them? >>>>> >>>>> The concept is that if we want to maintain fast paths for rwlock we >>>>> cannot do too much tricks there. And you can really deadlock if you >>>>> allow recursion on readers... >>>>> >>>>> >>>> How about adding rwlock_try_rlock() which would do the following: >>>> 1) Only variant to allow[1] read recursion and ... >>>> 2) ... only if no outstanding write requests >>>> 3) Let the caller deal with failure >>>> >>>> This can be implemented statically, so no overhead in the fast path. The >>>> caller is in the best position to decide if it is recursing or not - >>>> could keep that info on the stack - and can either fail[2] or do a normal >>>> rwlock_rlock() which would wait for the writer to enter and exit. >>>> >>>> [2] In most situation where you use read locks you can fail or roll back >>>> carefully as you didn't change anything - obviously. In pfil - for >>>> instance - we just dropped the packet if there was a writer waiting. >>>> >>>> [1] "allow" in terms of WITNESS - if that can be done. >>>> >>>> >>> The problem is that there is no tracking in the common case (without >>> additional overhead), so you can't know if you're recursing, unless >>> you track it yourself. >>> >>> -Alfred >>> >>> >>> >>> >> I talked with Attilio about that on IRC. >> Most common cases of writer starvation (but not all) could be solved by >> keeping a per thread count of shared acquired rwlocks. >> If a rwlock is currently locked as shared/read AND a thread is blocked >> on it to lock it exclusively/write - then new shared/read locks >> will only be granted to thread that already has a shared lock. (per >> thread shared counter is non zero) >> >> To be honest I am a bit twitchy about a lock without priority >> propagation - especially since in FreeBSD threads run with user priority >> in kernel >> space and can get preempted. >> >> Stephan >> > > That's an interesting hack, I guess it could be done. > > I would still like to disallow recursion. > Oh - I am all for disallowing recursion. In my opinion the only valid place for a thread to acquire the same lock multiple times is inside a transaction system with full deadlock detection. The question is if we can do that this late in the game? Maybe we could make non recursive the default and add a call rw_allow_recurse or rw_init_recurse to allow recursion on a lock if we can't get away with the straight out removal of the option? (Or less desirable - keep the current default and add functions to disallow recursion) > Can we come to a concensus on that? > > +1 for disallowing recursion Stephan From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 16:14:22 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0799D16A477 for ; Fri, 23 Nov 2007 16:14:22 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outB.internet-mail-service.net (outB.internet-mail-service.net [216.240.47.225]) by mx1.freebsd.org (Postfix) with ESMTP id C041D13C469 for ; Fri, 23 Nov 2007 16:14:21 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 23 Nov 2007 08:14:21 -0800 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 46F49126AB4; Fri, 23 Nov 2007 08:14:20 -0800 (PST) Message-ID: <4746FC5B.3040809@elischer.org> Date: Fri, 23 Nov 2007 08:14:19 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alfred Perlstein References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> In-Reply-To: <20071123082339.GN44563@elvis.mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Attilio Rao , Max Laier , freebsd-arch@freebsd.org, Stephan Uphoff Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 16:14:22 -0000 Alfred Perlstein wrote: > * Max Laier [071122 14:40] wrote: >> On Thursday 22 November 2007, Attilio Rao wrote: >>> 2007/11/22, Max Laier : >>>> rwlocks are already used in places that do recursive reads. The one >>>> place I'm certain about is pfil(9) and we need a proper sollution for >>>> this. Before rwlocks were used, I had a handrolled locking that >>>> supported both read/write semantics and starvation avoidance - at the >>>> cost of failing to allow futher read access when a writer asked for >>>> access. This however, was quite application specific and not the >>>> most efficient implementation either. >>> I'm not a pfil(9) expert, but for what I've heard, rmlocks should be >>> really good for it, shouldn't them? >>> >>> The concept is that if we want to maintain fast paths for rwlock we >>> cannot do too much tricks there. And you can really deadlock if you >>> allow recursion on readers... >> How about adding rwlock_try_rlock() which would do the following: >> 1) Only variant to allow[1] read recursion and ... >> 2) ... only if no outstanding write requests >> 3) Let the caller deal with failure >> >> This can be implemented statically, so no overhead in the fast path. The >> caller is in the best position to decide if it is recursing or not - >> could keep that info on the stack - and can either fail[2] or do a normal >> rwlock_rlock() which would wait for the writer to enter and exit. >> >> [2] In most situation where you use read locks you can fail or roll back >> carefully as you didn't change anything - obviously. In pfil - for >> instance - we just dropped the packet if there was a writer waiting. >> >> [1] "allow" in terms of WITNESS - if that can be done. > > The problem is that there is no tracking in the common case (without > additional overhead), so you can't know if you're recursing, unless > you track it yourself. recursion is hard for the caller to know because unrelated code could have done the original calls. The only way it can know is if there is some generic recursion detection. > > -Alfred > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 16:34:35 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B901116A420 for ; Fri, 23 Nov 2007 16:34:35 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outM.internet-mail-service.net (outM.internet-mail-service.net [216.240.47.236]) by mx1.freebsd.org (Postfix) with ESMTP id 7C2F513C4DB for ; Fri, 23 Nov 2007 16:34:35 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 23 Nov 2007 08:34:29 -0800 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id C35A4126ABD; Fri, 23 Nov 2007 08:34:26 -0800 (PST) Message-ID: <47470112.8090005@elischer.org> Date: Fri, 23 Nov 2007 08:34:26 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alfred Perlstein References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <4746FC5B.3040809@elischer.org> In-Reply-To: <4746FC5B.3040809@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Attilio Rao , Max Laier , Stephan Uphoff , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 16:34:35 -0000 Julian Elischer wrote: > Alfred Perlstein wrote: >> * Max Laier [071122 14:40] wrote: >>> On Thursday 22 November 2007, Attilio Rao wrote: >>>> 2007/11/22, Max Laier : >>>>> rwlocks are already used in places that do recursive reads. The one >>>>> place I'm certain about is pfil(9) and we need a proper sollution for >>>>> this. Before rwlocks were used, I had a handrolled locking that >>>>> supported both read/write semantics and starvation avoidance - at the >>>>> cost of failing to allow futher read access when a writer asked for >>>>> access. This however, was quite application specific and not the >>>>> most efficient implementation either. >>>> I'm not a pfil(9) expert, but for what I've heard, rmlocks should be >>>> really good for it, shouldn't them? >>>> >>>> The concept is that if we want to maintain fast paths for rwlock we >>>> cannot do too much tricks there. And you can really deadlock if you >>>> allow recursion on readers... >>> How about adding rwlock_try_rlock() which would do the following: >>> 1) Only variant to allow[1] read recursion and ... >>> 2) ... only if no outstanding write requests >>> 3) Let the caller deal with failure >>> >>> This can be implemented statically, so no overhead in the fast path. >>> The caller is in the best position to decide if it is recursing or >>> not - could keep that info on the stack - and can either fail[2] or >>> do a normal rwlock_rlock() which would wait for the writer to enter >>> and exit. >>> >>> [2] In most situation where you use read locks you can fail or roll >>> back carefully as you didn't change anything - obviously. In pfil - >>> for instance - we just dropped the packet if there was a writer waiting. >>> >>> [1] "allow" in terms of WITNESS - if that can be done. >> >> The problem is that there is no tracking in the common case (without >> additional overhead), so you can't know if you're recursing, unless >> you track it yourself. > > recursion is hard for the caller to know because unrelated code could > have done the original calls. > The only way it can know is if there is some generic recursion detection. translation: make recursion as hard as possible, (or disallow) > >> >> -Alfred >> >> _______________________________________________ >> freebsd-arch@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-arch >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 16:45:52 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0A2C16A417; Fri, 23 Nov 2007 16:45:52 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id A09DC13C43E; Fri, 23 Nov 2007 16:45:52 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 3B72D1A4D7C; Fri, 23 Nov 2007 08:45:49 -0800 (PST) Date: Fri, 23 Nov 2007 08:45:49 -0800 From: Alfred Perlstein To: Stephan Uphoff Message-ID: <20071123164549.GR44563@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4746F858.4070301@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Attilio Rao , Max Laier , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 16:45:52 -0000 * Stephan Uphoff [071123 07:57] wrote: > Alfred Perlstein wrote: > > > >That's an interesting hack, I guess it could be done. > > > >I would still like to disallow recursion. > > > Oh - I am all for disallowing recursion. > In my opinion the only valid place for a thread to acquire the same lock > multiple times is inside a transaction system with full deadlock detection. > The question is if we can do that this late in the game? > Maybe we could make non recursive the default and add a call > rw_allow_recurse or rw_init_recurse to allow recursion on a lock if we > can't get away with > the straight out removal of the option? (Or less desirable - keep the > current default and add functions to disallow recursion) > >Can we come to a concensus on that? > > > > > > +1 for disallowing recursion > > Stephan The plan is to simply mention in the documentation that it's not allowed, we'll also remove any options associated with read recursion (if there are any). INVARIANTS to catch such things, and subsystems that still use it will be allowed to live for a while until they are fixed. -- - Alfred Perlstein From owner-freebsd-arch@FreeBSD.ORG Fri Nov 23 16:48:51 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75A8D16A420 for ; Fri, 23 Nov 2007 16:48:51 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.184]) by mx1.freebsd.org (Postfix) with ESMTP id CD4C113C461 for ; Fri, 23 Nov 2007 16:48:50 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so2889848nfb for ; Fri, 23 Nov 2007 08:48:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=qq0kTwLot5HtdCcRFk1yeg1a2y1wQ6nt5+4WLcoU7VU=; b=P3lgfdmD/RcxnXiG7wh42CZ6R8JBYt5PcYZ3zMyprDoxJxWZfOQh+4r1morTrgan9ZlgsZZe37rzyKZOSe0B5Llgc3hc1gl526cGY+1+HCW7dYfuKTMovBG2OrBcZ4qrODkHzAq7JUiSALAslBVz4dTepfQ9R1+AwnvqxY28+0M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=L06XmKsAEGoX5Yz5J41MNbaLzIIZcSsxqmv9gqQVcIp/GwhU8bjSbkDz08M1Ax8dATRDpmJwg/0Jg2snjSWP6WoWKsRrUY5l7cZ8zUkgcZuJbdRPugn9NS36FEDhGyB3Xgce/DTkxfild+ADTPQuV2BnKxCsswNRLgNXSWRzmqI= Received: by 10.86.79.19 with SMTP id c19mr9692265fgb.1195836525408; Fri, 23 Nov 2007 08:48:45 -0800 (PST) Received: by 10.86.28.19 with HTTP; Fri, 23 Nov 2007 08:48:45 -0800 (PST) Message-ID: <3bbf2fe10711230848t29a711ccg74f6cfc37f84a2e5@mail.gmail.com> Date: Fri, 23 Nov 2007 17:48:45 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Stephan Uphoff" In-Reply-To: <4746F858.4070301@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> X-Google-Sender-Auth: bee7c3e9aa6850ef Cc: Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2007 16:48:51 -0000 2007/11/23, Stephan Uphoff : > Alfred Perlstein wrote: > > > > That's an interesting hack, I guess it could be done. > > > > I would still like to disallow recursion. > > > Oh - I am all for disallowing recursion. > In my opinion the only valid place for a thread to acquire the same lock > multiple times is inside a transaction system with full deadlock detection. > The question is if we can do that this late in the game? > Maybe we could make non recursive the default and add a call > rw_allow_recurse or rw_init_recurse to allow recursion on a lock if we > can't get away with > the straight out removal of the option? (Or less desirable - keep the > current default and add functions to disallow recursion) This is still tricky as long as the other set of functions (downgrade, upgrade, etc.) rely on the bits settings and wakeup algorithms which will differ between the two version. My idea is to add recursion detection for readers in witness so that rwlock consumers which still recurse in read mode can be fixed and later detected. Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 00:35:35 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C072916A420 for ; Sat, 24 Nov 2007 00:35:35 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.187]) by mx1.freebsd.org (Postfix) with ESMTP id 5524913C45D for ; Sat, 24 Nov 2007 00:35:35 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so2986374nfb for ; Fri, 23 Nov 2007 16:35:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=dV2nJyuQali5mgGHszBZSbwYCbB1VJUZpdcQd0vOOl8=; b=XNWlzH83/gREDDoLE+5uCHRqNLTcknnoV60NdIR1W41rReXhwnbI2jva1D2IbbsoXBiF+QHrjnf2axRuI6v8TSBMkz8ge0FwbeBjvzLoHURw2BLugee6lpwVIMn2tNIxt0ctS+9SQ4FTCEtPXMAma7dn9SjJwPVbbuJsluSj2tA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=JQxmtAoklEofprp13RfWSscVmWAGv2qLLEHo1PmRLuZ4J9/BGZPKvosGHiZaoBxhdJUwkbgxLNZhlryGwGijNnyp9PR/otHprwHv1/K/A7QdbLJUdfz4cLNsLbtuPEyguXRYew3KH77Tq11Z43xr6pZaYJRWOo4/s7SQ6nJLg+A= Received: by 10.86.4.2 with SMTP id 2mr10085452fgd.1195864533800; Fri, 23 Nov 2007 16:35:33 -0800 (PST) Received: by 10.86.28.19 with HTTP; Fri, 23 Nov 2007 16:35:33 -0800 (PST) Message-ID: <3bbf2fe10711231635i72df8babucedd1e5bdef7175d@mail.gmail.com> Date: Sat, 24 Nov 2007 01:35:33 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Robert Watson" In-Reply-To: <20071123235346.E14018@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> X-Google-Sender-Auth: 28d0b9113e89c37d Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 00:35:35 -0000 2007/11/24, Robert Watson : > > On Fri, 23 Nov 2007, Stephan Uphoff wrote: > > >>> I talked with Attilio about that on IRC. Most common cases of writer > >>> starvation (but not all) could be solved by keeping a per thread count of > >>> shared acquired rwlocks. If a rwlock is currently locked as shared/read > >>> AND a thread is blocked on it to lock it exclusively/write - then new > >>> shared/read locks will only be granted to thread that already has a shared > >>> lock. (per thread shared counter is non zero) > >>> > >>> To be honest I am a bit twitchy about a lock without priority propagation > >>> - especially since in FreeBSD threads run with user priority in kernel > >>> space and can get preempted. > >> > >> That's an interesting hack, I guess it could be done. > >> > >> I would still like to disallow recursion. > >> > > Oh - I am all for disallowing recursion. In my opinion the only valid place > > for a thread to acquire the same lock multiple times is inside a transaction > > system with full deadlock detection. The question is if we can do that this > > late in the game? Maybe we could make non recursive the default and add a > > call rw_allow_recurse or rw_init_recurse to allow recursion on a lock if we > > can't get away with the straight out removal of the option? (Or less > > desirable - keep the current default and add functions to disallow > > recursion) > > While I'm no great fan of recursion, the reality is that many of our kernel > subsystems are not yet ready to disallow recursion on locks. Take a look at > the cases where we explicitly enable recursive acquisition for mutexes--in > practice, most network stack mutexes are recursive due to the recursive > calling in the network stack. While someday I'd like to think we'll be able > to eliminate some of that, but it won't be soon since it requires significant > reworking of very complicated code. The current model in which recursion is > explicitly enabled only where still required seems to work pretty well for the > existing code, although it's hard to say yet in the code I've looked at > whether read recursion would be required--the situations I have in mind would > require purely write recursion. There's one case in the UNIX domain socket > code where we do a locked test and conditional lock/unlock with an rwlock for > exclusive locking because recursion isn't currently supported, and that's not > a usage I'd like to encourage more of. I think however that Stephan is just referring to the readers recursion (as we are doing) that if present should be just in few points. If we want todo a radical switch of this genre this is the right moment, just before rwlock became too widespread. I have a patch against witness which would check for readers recursion and panic, I will post in the night or tomorrow. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 00:45:42 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7939416A41B for ; Sat, 24 Nov 2007 00:45:42 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 2746013C447 for ; Sat, 24 Nov 2007 00:45:42 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 7F46646B93; Fri, 23 Nov 2007 19:30:21 -0500 (EST) Date: Sat, 24 Nov 2007 00:27:04 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Stephan Uphoff In-Reply-To: <4746F858.4070301@freebsd.org> Message-ID: <20071123235346.E14018@fledge.watson.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Attilio Rao , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 00:45:42 -0000 On Fri, 23 Nov 2007, Stephan Uphoff wrote: >>> I talked with Attilio about that on IRC. Most common cases of writer >>> starvation (but not all) could be solved by keeping a per thread count of >>> shared acquired rwlocks. If a rwlock is currently locked as shared/read >>> AND a thread is blocked on it to lock it exclusively/write - then new >>> shared/read locks will only be granted to thread that already has a shared >>> lock. (per thread shared counter is non zero) >>> >>> To be honest I am a bit twitchy about a lock without priority propagation >>> - especially since in FreeBSD threads run with user priority in kernel >>> space and can get preempted. >> >> That's an interesting hack, I guess it could be done. >> >> I would still like to disallow recursion. >> > Oh - I am all for disallowing recursion. In my opinion the only valid place > for a thread to acquire the same lock multiple times is inside a transaction > system with full deadlock detection. The question is if we can do that this > late in the game? Maybe we could make non recursive the default and add a > call rw_allow_recurse or rw_init_recurse to allow recursion on a lock if we > can't get away with the straight out removal of the option? (Or less > desirable - keep the current default and add functions to disallow > recursion) While I'm no great fan of recursion, the reality is that many of our kernel subsystems are not yet ready to disallow recursion on locks. Take a look at the cases where we explicitly enable recursive acquisition for mutexes--in practice, most network stack mutexes are recursive due to the recursive calling in the network stack. While someday I'd like to think we'll be able to eliminate some of that, but it won't be soon since it requires significant reworking of very complicated code. The current model in which recursion is explicitly enabled only where still required seems to work pretty well for the existing code, although it's hard to say yet in the code I've looked at whether read recursion would be required--the situations I have in mind would require purely write recursion. There's one case in the UNIX domain socket code where we do a locked test and conditional lock/unlock with an rwlock for exclusive locking because recursion isn't currently supported, and that's not a usage I'd like to encourage more of. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 03:16:56 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06ABC16A419; Sat, 24 Nov 2007 03:16:56 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 0961513C468; Sat, 24 Nov 2007 03:16:55 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id C60901A4D7C; Fri, 23 Nov 2007 19:16:55 -0800 (PST) Date: Fri, 23 Nov 2007 19:16:55 -0800 From: Alfred Perlstein To: Robert Watson Message-ID: <20071124031655.GD71382@elvis.mu.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071123235346.E14018@fledge.watson.org> User-Agent: Mutt/1.4.2.3i Cc: Stephan Uphoff , Attilio Rao , freebsd-arch@freebsd.org, Max Laier Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 03:16:56 -0000 * Robert Watson [071123 16:27] wrote: > > On Fri, 23 Nov 2007, Stephan Uphoff wrote: > > >>>I talked with Attilio about that on IRC. Most common cases of writer > >>>starvation (but not all) could be solved by keeping a per thread count > >>>of shared acquired rwlocks. If a rwlock is currently locked as > >>>shared/read AND a thread is blocked on it to lock it exclusively/write - > >>>then new shared/read locks will only be granted to thread that already > >>>has a shared lock. (per thread shared counter is non zero) > >>> > >>>To be honest I am a bit twitchy about a lock without priority > >>>propagation - especially since in FreeBSD threads run with user priority > >>>in kernel space and can get preempted. > >> > >>That's an interesting hack, I guess it could be done. > >> > >>I would still like to disallow recursion. > >> > >Oh - I am all for disallowing recursion. In my opinion the only valid > >place for a thread to acquire the same lock multiple times is inside a > >transaction system with full deadlock detection. The question is if we can > >do that this late in the game? Maybe we could make non recursive the > >default and add a call rw_allow_recurse or rw_init_recurse to allow > >recursion on a lock if we can't get away with the straight out removal of > >the option? (Or less desirable - keep the current default and add > >functions to disallow recursion) > > While I'm no great fan of recursion, the reality is that many of our kernel > subsystems are not yet ready to disallow recursion on locks. Take a look > at the cases where we explicitly enable recursive acquisition for > mutexes--in practice, most network stack mutexes are recursive due to the > recursive calling in the network stack. While someday I'd like to think > we'll be able to eliminate some of that, but it won't be soon since it > requires significant reworking of very complicated code. The current model > in which recursion is explicitly enabled only where still required seems to > work pretty well for the existing code, although it's hard to say yet in > the code I've looked at whether read recursion would be required--the > situations I have in mind would require purely write recursion. There's > one case in the UNIX domain socket code where we do a locked test and > conditional lock/unlock with an rwlock for exclusive locking because > recursion isn't currently supported, and that's not a usage I'd like to > encourage more of. Yes that's all well and good, however we have to come to a consensus on whether to bite the bullet and implement recursion tracking on rwlocks or disallow it, because unless we do this, we open the system up to starvation live-lock. So let's make a decision about it instead of just talking about it. -- - Alfred Perlstein From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 03:30:31 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B992616A420 for ; Sat, 24 Nov 2007 03:30:31 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.187]) by mx1.freebsd.org (Postfix) with ESMTP id 6890E13C459 for ; Sat, 24 Nov 2007 03:30:31 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so9764nfb for ; Fri, 23 Nov 2007 19:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=xYoaVLkZaHT5BT5+fCdCnQF3LQiVdWEeUKtwjsFZCX4=; b=Qsoqh/tHYaQu+ENuyiGsWU4fsN6fmlHF8uVgleyvL5l04BOsmeUyWQqpH4I6qPdxcU8wcH43maCBW+2tpoKKsGFvfbaFh66ISyxXP2HA1AUnWc/zWuBQWV/GVZGJh2haK7s/nMMfJqFwv4rLQx0LVUqiUYKE1XaWicVUS4D7e2M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=q6P8yb5WzHpcFtb4n7pqybcgAe8oEJ12qSREyITTowxkei3nc5Xh1XdUiAGAbfFgSx+HWNW+fGP5em9613PiXHcR2c6SXfQfDItOSU4YNjaHSY6vkbySv9H2NCCDKcswmawQV/dqEljVYiLYTIKysJjO3r9enA5ERhK6iwRIBtI= Received: by 10.86.98.18 with SMTP id v18mr56018fgb.1195875029942; Fri, 23 Nov 2007 19:30:29 -0800 (PST) Received: by 10.86.28.19 with HTTP; Fri, 23 Nov 2007 19:30:29 -0800 (PST) Message-ID: <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> Date: Sat, 24 Nov 2007 04:30:29 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Robert Watson" In-Reply-To: <20071123235346.E14018@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> X-Google-Sender-Auth: 324b85c6f1f72a8c Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 03:30:31 -0000 2007/11/24, Robert Watson : > While I'm no great fan of recursion, the reality is that many of our kernel > subsystems are not yet ready to disallow recursion on locks. Take a look at > the cases where we explicitly enable recursive acquisition for mutexes--in > practice, most network stack mutexes are recursive due to the recursive > calling in the network stack. While someday I'd like to think we'll be able > to eliminate some of that, but it won't be soon since it requires significant > reworking of very complicated code. The current model in which recursion is > explicitly enabled only where still required seems to work pretty well for the > existing code, although it's hard to say yet in the code I've looked at > whether read recursion would be required--the situations I have in mind would > require purely write recursion. There's one case in the UNIX domain socket > code where we do a locked test and conditional lock/unlock with an rwlock for > exclusive locking because recursion isn't currently supported, and that's not > a usage I'd like to encourage more of. Oh, I just didn't notice this -- rwlock are only present in 7.0 and in 7.0 they support recursion in exclusive mode, so I'm not sure what do you mean with 'recursion isn't currently supported'. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 04:28:25 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72DBF16A41A for ; Sat, 24 Nov 2007 04:28:25 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.187]) by mx1.freebsd.org (Postfix) with ESMTP id 2673F13C461 for ; Sat, 24 Nov 2007 04:28:24 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so18004nfb for ; Fri, 23 Nov 2007 20:28:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; bh=H08l1x4lG/ZCGr2sWjFHyjhMk3eSvD0gdxjcicTOylY=; b=t9pLaRp5RPCwKKxuIlBfUVrPNKTHszqUzyh7YYuqdUcfh+7V/gr7psjG84Ue9vhoR1Fk+CyMnlzRdfaRt+RK+Xy/mp8pJ356i8N/4pQHoFExH+y32zCdwtVQvV6na/6kMlQRg4U0deP6ZXaFeGn6UQQrgyIb4KtJOkvlt3AhugI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=GpCBh1rsq0dMYImmnC0aCDwstAM6EbNeTSPu03ztRIXG95fMsCfdnwHgQCfxsKF6E+iAhfwjERLbvZlSuNEhCl/NeFCPswNmtjcUct8NtMCAOA5fm+rgZcZzA/oQ+uqDCN4tq/PsU101pSpCr9hI5TrXIlfJmcPkadmFsD7r6xA= Received: by 10.86.90.2 with SMTP id n2mr68360fgb.1195878503524; Fri, 23 Nov 2007 20:28:23 -0800 (PST) Received: by 10.86.28.19 with HTTP; Fri, 23 Nov 2007 20:28:23 -0800 (PST) Message-ID: <3bbf2fe10711232028p75ee280ax95dabeba406425cd@mail.gmail.com> Date: Sat, 24 Nov 2007 05:28:23 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: arch@freebsd.org, smp@freebsd.org, fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Google-Sender-Auth: 7f72cdd27691e8e5 Cc: Subject: [FYI] transferlockers() function removed -- lockmgr KPI changed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 04:28:25 -0000 Hello, as transferlockers() function is dangerous and no longer used in our FreeBSD stock tree, I decided to axe it from FreeBSD-CURRENT8.0 and a quick MFC is planned in 3 days. Any third-party module using it, should update their own source code with equivalent functions (for any further question or assistance, in particular with this, you can send them directly to me). Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 10:34:22 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65F9116A417; Sat, 24 Nov 2007 10:34:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id C3C3B13C458; Sat, 24 Nov 2007 10:34:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 3143F471DA; Sat, 24 Nov 2007 05:37:33 -0500 (EST) Date: Sat, 24 Nov 2007 10:34:12 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Attilio Rao In-Reply-To: <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> Message-ID: <20071124103231.A14018@fledge.watson.org> References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 10:34:22 -0000 On Sat, 24 Nov 2007, Attilio Rao wrote: > 2007/11/24, Robert Watson : > >> While I'm no great fan of recursion, the reality is that many of our kernel >> subsystems are not yet ready to disallow recursion on locks. Take a look >> at the cases where we explicitly enable recursive acquisition for >> mutexes--in practice, most network stack mutexes are recursive due to the >> recursive calling in the network stack. While someday I'd like to think >> we'll be able to eliminate some of that, but it won't be soon since it >> requires significant reworking of very complicated code. The current model >> in which recursion is explicitly enabled only where still required seems to >> work pretty well for the existing code, although it's hard to say yet in >> the code I've looked at whether read recursion would be required--the >> situations I have in mind would require purely write recursion. There's >> one case in the UNIX domain socket code where we do a locked test and >> conditional lock/unlock with an rwlock for exclusive locking because >> recursion isn't currently supported, and that's not a usage I'd like to >> encourage more of. > > Oh, I just didn't notice this -- rwlock are only present in 7.0 and in 7.0 > they support recursion in exclusive mode, so I'm not sure what do you mean > with 'recursion isn't currently supported'. I must have missed recursion arriving then -- I'll modify uipc_usrreq.c to set the recursion flag on the rwlock in UNIX domain sockets rather than doing the nasty hack that was previously required. At the time, the hack was added because it seemed recursion was not going to be added to rwlocks, but sonewconn() behavior for listen sockets really ended up requiring it. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 12:06:31 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55C6416A46C; Sat, 24 Nov 2007 12:06:31 +0000 (UTC) (envelope-from skip@menantico.com) Received: from vms173001pub.verizon.net (vms173001pub.verizon.net [206.46.173.1]) by mx1.freebsd.org (Postfix) with ESMTP id D9C8F13C4F0; Sat, 24 Nov 2007 12:06:30 +0000 (UTC) (envelope-from skip@menantico.com) Received: from mx.menantico.com ([71.188.11.206]) by vms173001.mailsrvcs.net (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPA id <0JS0006VDBRKJL95@vms173001.mailsrvcs.net>; Sat, 24 Nov 2007 04:57:20 -0600 (CST) Date: Sat, 24 Nov 2007 06:09:18 -0500 From: Skip Ford In-reply-to: <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> To: Attilio Rao Mail-followup-to: Attilio Rao , Robert Watson , Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Message-id: <20071124110918.GE16878@menantico.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <200711221641.02484.max@love2party.net> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , Robert Watson , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 12:06:31 -0000 Attilio Rao wrote: > 2007/11/24, Robert Watson : > > While I'm no great fan of recursion, the reality is that many of our kernel > > subsystems are not yet ready to disallow recursion on locks. Take a look at > > the cases where we explicitly enable recursive acquisition for mutexes--in > > practice, most network stack mutexes are recursive due to the recursive > > calling in the network stack. While someday I'd like to think we'll be able > > to eliminate some of that, but it won't be soon since it requires significant > > reworking of very complicated code. The current model in which recursion is > > explicitly enabled only where still required seems to work pretty well for the > > existing code, although it's hard to say yet in the code I've looked at > > whether read recursion would be required--the situations I have in mind would > > require purely write recursion. There's one case in the UNIX domain socket > > code where we do a locked test and conditional lock/unlock with an rwlock for > > exclusive locking because recursion isn't currently supported, and that's not > > a usage I'd like to encourage more of. > > Oh, I just didn't notice this -- rwlock are only present in 7.0 and in > 7.0 they support recursion in exclusive mode, so I'm not sure what do > you mean with 'recursion isn't currently supported'. locking(9) and rwlock(9) both say it isn't supported. -- Skip From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 13:53:54 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CAC816A419 for ; Sat, 24 Nov 2007 13:53:54 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.184]) by mx1.freebsd.org (Postfix) with ESMTP id A890513C4F6 for ; Sat, 24 Nov 2007 13:53:53 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so113442nfb for ; Sat, 24 Nov 2007 05:53:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=ZO7I8AWAyDFhPEqV2BwrLSTzjBtXz7hZ9OAbTU9wpTk=; b=Sjk+bMcmXKITy3qTyOE6YJXTSryH+ly1WyBFvlRmKzXKBZTEaWO77f+OS9Je+cFJcxg2UrQjsWnVEB4mXH3YeG0nZPryCLVX+taIWb94+g9RMFpgnqznWOA0O8ItFFs35IRtpC9DMwxTkSo9yP7uWp+7vjie+WVMllXDIABkmEI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=eTtzrqALmiUFmUshKcBCkijO8KiAC7Io8ZRqg5UnJbLUXPdJZuCXDO4AXkdKxoFI91TQ6rCeI6JdurhpP5Uj5CjKgOnctabGav5ZEiMy09F0In8wKCoNiMDvT4zMj+oWimjuuAcybFqNAhtXFV81W7e7p50mIst4Gri0yPZIcfY= Received: by 10.86.98.18 with SMTP id v18mr531002fgb.1195912432243; Sat, 24 Nov 2007 05:53:52 -0800 (PST) Received: by 10.86.28.19 with HTTP; Sat, 24 Nov 2007 05:53:52 -0800 (PST) Message-ID: <3bbf2fe10711240553k1eb35a5au23cae8af08f5864c@mail.gmail.com> Date: Sat, 24 Nov 2007 14:53:52 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Robert Watson" In-Reply-To: <20071124103231.A14018@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> <20071124103231.A14018@fledge.watson.org> X-Google-Sender-Auth: 828272a9965bc456 Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 13:53:54 -0000 2007/11/24, Robert Watson : > On Sat, 24 Nov 2007, Attilio Rao wrote: > > > 2007/11/24, Robert Watson : > > > >> While I'm no great fan of recursion, the reality is that many of our kernel > >> subsystems are not yet ready to disallow recursion on locks. Take a look > >> at the cases where we explicitly enable recursive acquisition for > >> mutexes--in practice, most network stack mutexes are recursive due to the > >> recursive calling in the network stack. While someday I'd like to think > >> we'll be able to eliminate some of that, but it won't be soon since it > >> requires significant reworking of very complicated code. The current model > >> in which recursion is explicitly enabled only where still required seems to > >> work pretty well for the existing code, although it's hard to say yet in > >> the code I've looked at whether read recursion would be required--the > >> situations I have in mind would require purely write recursion. There's > >> one case in the UNIX domain socket code where we do a locked test and > >> conditional lock/unlock with an rwlock for exclusive locking because > >> recursion isn't currently supported, and that's not a usage I'd like to > >> encourage more of. > > > > Oh, I just didn't notice this -- rwlock are only present in 7.0 and in 7.0 > > they support recursion in exclusive mode, so I'm not sure what do you mean > > with 'recursion isn't currently supported'. > > I must have missed recursion arriving then -- I'll modify uipc_usrreq.c to set > the recursion flag on the rwlock in UNIX domain sockets rather than doing the > nasty hack that was previously required. At the time, the hack was added > because it seemed recursion was not going to be added to rwlocks, but > sonewconn() behavior for listen sockets really ended up requiring it. attilio 2007-06-26 21:31:56 UTC FreeBSD src repository Modified files: sys/kern kern_rwlock.c sys/sys _rwlock.h rwlock.h Log: Introduce a new rwlocks initialization function: rw_init_flags. This is very similar to sx_init_flags: it initializes the rwlock using special flags passed as third argument (RW_DUPOK, RW_NOPROFILE, RW_NOWITNESS, RW_QUIET, RW_RECURSE). Among these, the most important new feature is probabilly that rwlocks can be acquired recursively now (for both shared and exclusive paths). Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 16:25:45 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19CA616A420; Sat, 24 Nov 2007 16:25:45 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id E51DC13C458; Sat, 24 Nov 2007 16:25:44 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id E027446CF9; Sat, 24 Nov 2007 11:28:45 -0500 (EST) Date: Sat, 24 Nov 2007 16:25:22 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Attilio Rao In-Reply-To: <3bbf2fe10711240553k1eb35a5au23cae8af08f5864c@mail.gmail.com> Message-ID: <20071124162322.V14018@fledge.watson.org> References: <20071121222319.GX44563@elvis.mu.org> <3bbf2fe10711220753u435ff4cbxa94d5b682292b970@mail.gmail.com> <200711221726.27108.max@love2party.net> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> <20071124103231.A14018@fledge.watson.org> <3bbf2fe10711240553k1eb35a5au23cae8af08f5864c@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 16:25:45 -0000 On Sat, 24 Nov 2007, Attilio Rao wrote: >> I must have missed recursion arriving then -- I'll modify uipc_usrreq.c to >> set the recursion flag on the rwlock in UNIX domain sockets rather than >> doing the nasty hack that was previously required. At the time, the hack >> was added because it seemed recursion was not going to be added to rwlocks, >> but sonewconn() behavior for listen sockets really ended up requiring it. > > attilio 2007-06-26 21:31:56 UTC > > FreeBSD src repository > > Modified files: > sys/kern kern_rwlock.c > sys/sys _rwlock.h rwlock.h > Log: > Introduce a new rwlocks initialization function: rw_init_flags. > This is very similar to sx_init_flags: it initializes the rwlock using > special flags passed as third argument (RW_DUPOK, RW_NOPROFILE, > RW_NOWITNESS, RW_QUIET, RW_RECURSE). > Among these, the most important new feature is probabilly that rwlocks > can be acquired recursively now (for both shared and exclusive paths). Yes, that was four months after I added rw_wowned(9) to work around the lack of recursion support. :-) However, it looks like the man page was never updated? It contains the following rather explicit language: Another important property is that shared holders of rwlock can recurse, but exclusive locks are not allowed to recurse. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Sat Nov 24 18:09:24 2007 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD5E416A478 for ; Sat, 24 Nov 2007 18:09:24 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.184]) by mx1.freebsd.org (Postfix) with ESMTP id 3CABB13C45A for ; Sat, 24 Nov 2007 18:09:23 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so165251nfb for ; Sat, 24 Nov 2007 10:09:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=c8Vu5uuJA3ylkRy37i4gLwjJxRNNgOhYI6VUr9Zmx0g=; b=t2r+QwhAS8pT/uQGGuf8DiymEAYWNxyYZK7s/rnyYEkcZpmh4qhUAc8np4RpXQFGsMLLx4lrF+ztB4xXlmiwQvRo3bEQ2Cs27jyGCHXfsDjLgm2zQS8ue8a8EaUdc3AfAzeA86nlUWWroZ8Ua3FvuBi4mfOy8986Ju4du4cphtc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=hiVN4ofTWVJwu1oYFieTUqcDmHogw2Ply3b209WE9FpofnUaGrrm8a+VPWNE2OnsUpVrtwhXogAHMORKRJpl0oGDdAcUUtu98FpgXWiOF6czIzLLcX9mI+OhRyBOe68HD0ypOAK0aRJx6VN2/HVtk331HRTwJ6GculT2cAiI3mA= Received: by 10.86.4.2 with SMTP id 2mr730162fgd.1195927762191; Sat, 24 Nov 2007 10:09:22 -0800 (PST) Received: by 10.86.28.19 with HTTP; Sat, 24 Nov 2007 10:09:22 -0800 (PST) Message-ID: <3bbf2fe10711241009g126e8bd1ldca2468e7956c902@mail.gmail.com> Date: Sat, 24 Nov 2007 19:09:22 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Robert Watson" In-Reply-To: <20071124162322.V14018@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071121222319.GX44563@elvis.mu.org> <20071123082339.GN44563@elvis.mu.org> <47469328.8020404@freebsd.org> <20071123092415.GP44563@elvis.mu.org> <4746F858.4070301@freebsd.org> <20071123235346.E14018@fledge.watson.org> <3bbf2fe10711231930m459dc800wbbb894b9fd50ca13@mail.gmail.com> <20071124103231.A14018@fledge.watson.org> <3bbf2fe10711240553k1eb35a5au23cae8af08f5864c@mail.gmail.com> <20071124162322.V14018@fledge.watson.org> X-Google-Sender-Auth: f94e150421d38068 Cc: Stephan Uphoff , Max Laier , Alfred Perlstein , freebsd-arch@freebsd.org Subject: Re: rwlocks, correctness over speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2007 18:09:24 -0000 2007/11/24, Robert Watson : > On Sat, 24 Nov 2007, Attilio Rao wrote: > > >> I must have missed recursion arriving then -- I'll modify uipc_usrreq.c to > >> set the recursion flag on the rwlock in UNIX domain sockets rather than > >> doing the nasty hack that was previously required. At the time, the hack > >> was added because it seemed recursion was not going to be added to rwlocks, > >> but sonewconn() behavior for listen sockets really ended up requiring it. > > > > attilio 2007-06-26 21:31:56 UTC > > > > FreeBSD src repository > > > > Modified files: > > sys/kern kern_rwlock.c > > sys/sys _rwlock.h rwlock.h > > Log: > > Introduce a new rwlocks initialization function: rw_init_flags. > > This is very similar to sx_init_flags: it initializes the rwlock using > > special flags passed as third argument (RW_DUPOK, RW_NOPROFILE, > > RW_NOWITNESS, RW_QUIET, RW_RECURSE). > > Among these, the most important new feature is probabilly that rwlocks > > can be acquired recursively now (for both shared and exclusive paths). > > Yes, that was four months after I added rw_wowned(9) to work around the lack > of recursion support. :-) However, it looks like the man page was never > updated? It contains the following rather explicit language: > > Another important property is that shared holders of rwlock can recurse, > but exclusive locks are not allowed to recurse. Yes, I'm going to fix the manpage ASAP. Thanks, for the report. Attilio -- Peace can only be achieved by understanding - A. Einstein