From owner-freebsd-arch@freebsd.org Sun Dec 27 16:44:52 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06939A52628; Sun, 27 Dec 2015 16:44:52 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A97C71F5B; Sun, 27 Dec 2015 16:44:51 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBRGiioL050482; Sun, 27 Dec 2015 11:44:44 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Sun, 27 Dec 2015 11:44:44 -0500 (EST) Date: Sun, 27 Dec 2015 11:44:44 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151226234424.GJ3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> <20151226234424.GJ3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Dec 2015 16:44:52 -0000 On Sun, 27 Dec 2015, Konstantin Belousov wrote: > On Sat, Dec 26, 2015 at 12:15:43PM -0500, Daniel Eischen wrote: [ snipped for brevity ] >> >> I agree, but the work that you are doing now would be basically >> thrown out later on. I will not stand in your way and appreciate >> any work you do. I would just rather that the struct change be >> made now for 11, even without any pshared or other changes. For >> once the struct change is made, pshared or other additions can >> be made afterward, even in the 11 branch because they would not >> break the ABI. > > Lock inlining was not done for ten years, now cost of doing it is > extremely high, as discussed above. Who would drive the change, and > with what time frame ? If me, I seriosly consider renaming libthr > to libthr2, but I had no time to think much about it. I could probably do the inlining of locks, pulling out that part of it from David's patch, and coordinating with you so we don't step on each other's toes. Perhaps just making the first element of the structs a self-reference at first, would help mitigate that... I'm not sure what renaming libthr to libthr2 solves that a version bump can't also. Can't we still tell whether both libthr.so.3 and libthr.so.4 have been loaded? Perhaps libthr2 is cleaner WRT keeping the old ABI (it could be dropped?). > Right now, I think that I want to commit my current patch and implement > robust mutexes as the next step, without ABI breakage. At least, this > seems to have fixed time-frame and can be made ready for 11.x. Lock > inlining might be not. Are there serious objections against the plan, > except that (lock inlining + pshared) is ideal situation, while the plan > is not (but more practical) ? What is the timeframe for 11.0-RELEASE? If not for 11.0, I would like to see it done soon in the 12-current branch afterward. In my mind, any pain will be the same in 11 or 12, nothing really is gained by waiting, only by not doing it (inlining locks) ever. -- DE From owner-freebsd-arch@freebsd.org Mon Dec 28 10:52:03 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E52DA53C68; Mon, 28 Dec 2015 10:52:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7FF21BB0; Mon, 28 Dec 2015 10:52:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBSApvrt058270 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 28 Dec 2015 12:51:57 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBSApvrt058270 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBSApv7X058269; Mon, 28 Dec 2015 12:51:57 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 28 Dec 2015 12:51:57 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151228105157.GQ3625@kib.kiev.ua> References: <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> <20151226234424.GJ3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 10:52:03 -0000 On Sun, Dec 27, 2015 at 11:44:44AM -0500, Daniel Eischen wrote: > On Sun, 27 Dec 2015, Konstantin Belousov wrote: > > > On Sat, Dec 26, 2015 at 12:15:43PM -0500, Daniel Eischen wrote: > [ snipped for brevity ] > >> > >> I agree, but the work that you are doing now would be basically > >> thrown out later on. I will not stand in your way and appreciate > >> any work you do. I would just rather that the struct change be > >> made now for 11, even without any pshared or other changes. For > >> once the struct change is made, pshared or other additions can > >> be made afterward, even in the 11 branch because they would not > >> break the ABI. > > > > Lock inlining was not done for ten years, now cost of doing it is > > extremely high, as discussed above. Who would drive the change, and > > with what time frame ? If me, I seriosly consider renaming libthr > > to libthr2, but I had no time to think much about it. > > I could probably do the inlining of locks, pulling out that part > of it from David's patch, and coordinating with you so we don't > step on each other's toes. Perhaps just making the first element > of the structs a self-reference at first, would help mitigate > that... Adding the self-pointer is good idea, but it would not work for the shared locks. If you are going to work on this, then I will scratch my patch, to not impede on your work. Taking out the inlining bits from the David patch, or (which would I do, if doing this) just reimplementing it from scratch is easy enough and just require some time. I estimated this job to take between one and two weeks. What is really thrilling is to manage the consequences of the ABI breakage. When I did the evaluation for the FF project, I put a six months extent for the whole work. This is for things like looking at the ports impact, trying to know in advance where mixed things start to break, monitoring the users complains about issues caused by the ABI break etc. We would see afterward if I overestimated the work, but I do the typical mistake of underestimating usually, whatever insane large the initial numbers are. > > I'm not sure what renaming libthr to libthr2 solves that a > version bump can't also. Can't we still tell whether both > libthr.so.3 and libthr.so.4 have been loaded? Perhaps libthr2 > is cleaner WRT keeping the old ABI (it could be dropped?). Yes, part of the change probably should be a prevention of simultaneous existence of libthr.so.3 and (libthr.so.4 or libthr2.so.1, whatever it is named) in one process. This should be done either with rtld facilities, I am not sure how. Or e.g. with dlopen("otherlib", RTLD_NOLOAD) in the constructor of each library and failing if dlopen() returns a valid handle. Another immediate point, not only libthr.so must be bumped, but also all base libraries depending on it must be. Even if some libraries do not directly record the dependency, they might require the bump as well, I am thinking about c++ runtime. This should allow the compat packages to provide useable libraries. IMO this is easier to see when the libraries names differ significantly in the name part, and not in the .so.n part. At least users would be less surprised, but also the work of the person who tracks the deps, would be easier too. > > > Right now, I think that I want to commit my current patch and implement > > robust mutexes as the next step, without ABI breakage. At least, this > > seems to have fixed time-frame and can be made ready for 11.x. Lock > > inlining might be not. Are there serious objections against the plan, > > except that (lock inlining + pshared) is ideal situation, while the plan > > is not (but more practical) ? > > What is the timeframe for 11.0-RELEASE? If not for 11.0, I would > like to see it done soon in the 12-current branch afterward. > In my mind, any pain will be the same in 11 or 12, nothing really > is gained by waiting, only by not doing it (inlining locks) ever. Difference is in causing more (ABI) breakage on the near-coming stable/11 users, or trying to fix or mitigate more of it in HEAD, using the local population as the canaries. You see my position, I would avoid ABI breakage at all costs. If I cannot talk you against this, please do not consider the work done after the inlining patch is committed to HEAD, it is actually only start there. From owner-freebsd-arch@freebsd.org Mon Dec 28 16:59:10 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FCF9A535BD; Mon, 28 Dec 2015 16:59:10 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 275051154; Mon, 28 Dec 2015 16:59:09 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBSGx2HY050700; Mon, 28 Dec 2015 11:59:02 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Mon, 28 Dec 2015 11:59:02 -0500 (EST) Date: Mon, 28 Dec 2015 11:59:02 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151228105157.GQ3625@kib.kiev.ua> Message-ID: References: <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> <20151226234424.GJ3625@kib.kiev.ua> <20151228105157.GQ3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 16:59:10 -0000 On Mon, 28 Dec 2015, Konstantin Belousov wrote: > On Sun, Dec 27, 2015 at 11:44:44AM -0500, Daniel Eischen wrote: >> On Sun, 27 Dec 2015, Konstantin Belousov wrote: >> >>> On Sat, Dec 26, 2015 at 12:15:43PM -0500, Daniel Eischen wrote: >> [ snipped for brevity ] >>>> >>>> I agree, but the work that you are doing now would be basically >>>> thrown out later on. I will not stand in your way and appreciate >>>> any work you do. I would just rather that the struct change be >>>> made now for 11, even without any pshared or other changes. For >>>> once the struct change is made, pshared or other additions can >>>> be made afterward, even in the 11 branch because they would not >>>> break the ABI. >>> >>> Lock inlining was not done for ten years, now cost of doing it is >>> extremely high, as discussed above. Who would drive the change, and >>> with what time frame ? If me, I seriosly consider renaming libthr >>> to libthr2, but I had no time to think much about it. >> >> I could probably do the inlining of locks, pulling out that part >> of it from David's patch, and coordinating with you so we don't >> step on each other's toes. Perhaps just making the first element >> of the structs a self-reference at first, would help mitigate >> that... > Adding the self-pointer is good idea, but it would not work for the > shared locks. If you are going to work on this, then I will scratch > my patch, to not impede on your work. Shared locks would work, but only for the new ABI (I guess that would be FBSD1.4 in HEAD, or FBSD1.5 if we bumped it again just for this). Another option is to just increase the size for the pthread synch types without really changing the first element (still a pointer to the allocated object - the implementation would stay the same). The only breakage would then be storage layout issues for new compiles linking against older ones. For FreeBSD-12, we could change the implementation to a real inlining (without changing the storage size), that gives a major release cycle for 3rd parties to make the change. I'm not sure if this would really be much of a benefit - the storage size change alone would probably be the greatest pain. > Taking out the inlining bits from the David patch, or (which would I do, > if doing this) just reimplementing it from scratch is easy enough and > just require some time. I estimated this job to take between one and > two weeks. I think a lot of David's patch is the renaming of all the elements of the public structs to prepend '__'. I was thinking it would be nice to have the public structs be something like this: struct pthread_mutex_t { uint32_t __x[IMPL_REQ + IMPL_SPARE + pad_to_CACHE_LINE_SIZE]; }; and then have libthr override the definition. That would make declaring PTHREAD_MUTEX_INITIALIZER, etc, a little magical, but avoid a lot of needless churn in libthr. But, I agree, just doing the inlining is basic grunt work and limited in duration. Your estimate seems reasonable. > What is really thrilling is to manage the consequences of the ABI > breakage. When I did the evaluation for the FF project, I put a six > months extent for the whole work. This is for things like looking at > the ports impact, trying to know in advance where mixed things start to > break, monitoring the users complains about issues caused by the ABI > break etc. There's not much we can really do to manage it, just make it known that all ports need to be recompiled on 11 in order to be on the safe side. The libmap.conf trick will not work either. If we could instrument libthr or rtld to emit a warning/error message with a URL to a FAQ or the the src/UPDATING entry that would be nice. > We would see afterward if I overestimated the work, but I do the typical > mistake of underestimating usually, whatever insane large the initial > numbers are. > >> >> I'm not sure what renaming libthr to libthr2 solves that a >> version bump can't also. Can't we still tell whether both >> libthr.so.3 and libthr.so.4 have been loaded? Perhaps libthr2 >> is cleaner WRT keeping the old ABI (it could be dropped?). > Yes, part of the change probably should be a prevention of simultaneous > existence of libthr.so.3 and (libthr.so.4 or libthr2.so.1, whatever > it is named) in one process. This should be done either with rtld > facilities, I am not sure how. Or e.g. with dlopen("otherlib", > RTLD_NOLOAD) in the constructor of each library and failing if dlopen() > returns a valid handle. > > Another immediate point, not only libthr.so must be bumped, but also all > base libraries depending on it must be. Even if some libraries do not > directly record the dependency, they might require the bump as well, I > am thinking about c++ runtime. This should allow the compat packages > to provide useable libraries. > > IMO this is easier to see when the libraries names differ significantly > in the name part, and not in the .so.n part. At least users would be > less surprised, but also the work of the person who tracks the deps, > would be easier too. >>> Right now, I think that I want to commit my current patch and implement >>> robust mutexes as the next step, without ABI breakage. At least, this >>> seems to have fixed time-frame and can be made ready for 11.x. Lock >>> inlining might be not. Are there serious objections against the plan, >>> except that (lock inlining + pshared) is ideal situation, while the plan >>> is not (but more practical) ? >> >> What is the timeframe for 11.0-RELEASE? If not for 11.0, I would >> like to see it done soon in the 12-current branch afterward. >> In my mind, any pain will be the same in 11 or 12, nothing really >> is gained by waiting, only by not doing it (inlining locks) ever. > > Difference is in causing more (ABI) breakage on the near-coming > stable/11 users, or trying to fix or mitigate more of it in HEAD, > using the local population as the canaries. > > You see my position, I would avoid ABI breakage at all costs. If I > cannot talk you against this, please do not consider the work done after > the inlining patch is committed to HEAD, it is actually only start > there. Ok, I do think we want to do it at some point. Perhaps 12-current would be better... -- DE From owner-freebsd-arch@freebsd.org Tue Dec 29 18:45:29 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 900F4A552E5; Tue, 29 Dec 2015 18:45:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [176.36.249.139]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0C2A91029; Tue, 29 Dec 2015 18:45:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBTIi6Mu014420 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 29 Dec 2015 20:44:06 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBTIi6Mu014420 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBTIi5bE014419; Tue, 29 Dec 2015 20:44:05 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 29 Dec 2015 20:44:05 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151229184405.GY3625@kib.kiev.ua> References: <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> <20151226234424.GJ3625@kib.kiev.ua> <20151228105157.GQ3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 18:45:29 -0000 On Mon, Dec 28, 2015 at 11:59:02AM -0500, Daniel Eischen wrote: > On Mon, 28 Dec 2015, Konstantin Belousov wrote: > > > On Sun, Dec 27, 2015 at 11:44:44AM -0500, Daniel Eischen wrote: > > Adding the self-pointer is good idea, but it would not work for the > > shared locks. If you are going to work on this, then I will scratch > > my patch, to not impede on your work. > > Shared locks would work, but only for the new ABI (I guess that > would be FBSD1.4 in HEAD, or FBSD1.5 if we bumped it again just > for this). I only mean that self-pointer sometimes would cause additional issues. > > Another option is to just increase the size for the pthread synch > types without really changing the first element (still a pointer > to the allocated object - the implementation would stay the same). > The only breakage would then be storage layout issues for new > compiles linking against older ones. For FreeBSD-12, we could > change the implementation to a real inlining (without changing > the storage size), that gives a major release cycle for 3rd > parties to make the change. I'm not sure if this would really > be much of a benefit - the storage size change alone would > probably be the greatest pain. It is a good intermediate point in the whole plan of changing the ABI. The work required to support pshared after the size is increased is not completely trivial, but is not a rocket science either. But your note about intermediate step might give additional freedom in executing the whole plan. > > > Taking out the inlining bits from the David patch, or (which would I do, > > if doing this) just reimplementing it from scratch is easy enough and > > just require some time. I estimated this job to take between one and > > two weeks. > > I think a lot of David's patch is the renaming of all the > elements of the public structs to prepend '__'. I was thinking > it would be nice to have the public structs be something like > this: > > struct pthread_mutex_t { > uint32_t __x[IMPL_REQ + IMPL_SPARE + pad_to_CACHE_LINE_SIZE]; > }; > > and then have libthr override the definition. That would > make declaring PTHREAD_MUTEX_INITIALIZER, etc, a little > magical, but avoid a lot of needless churn in libthr. This is very good suggestion, I fully agree. There are some more details, e.g. it would be better to use uint64_t or explicit align attribute, to get proper alignment, but overall idea is sound, of course. > > But, I agree, just doing the inlining is basic grunt work > and limited in duration. Your estimate seems reasonable. > > > What is really thrilling is to manage the consequences of the ABI > > breakage. When I did the evaluation for the FF project, I put a six > > months extent for the whole work. This is for things like looking at > > the ports impact, trying to know in advance where mixed things start to > > break, monitoring the users complains about issues caused by the ABI > > break etc. > > There's not much we can really do to manage it, just make it > known that all ports need to be recompiled on 11 in order to > be on the safe side. The libmap.conf trick will not work > either. No. This is probably the point where our positions diverge most. I do think that some mitigation measures can be applied, e.g. your self-pointer trick, check for presence of 'other' lib when loading given threading library, renaming etc. It should be thought out in advance. Another thing to do is to monitor mailing lists, react to the user issues, see if it is related to the ABI breakage, and possibly create another measures which would make some issues less painful. I consider both activities absolutely vital when doing this surgery. In principle, I may do both, but it requires more planning and is not immediately reachable comparing with the off-page approach to only get pshared right now. > > You see my position, I would avoid ABI breakage at all costs. If I > > cannot talk you against this, please do not consider the work done after > > the inlining patch is committed to HEAD, it is actually only start > > there. > > Ok, I do think we want to do it at some point. Perhaps 12-current > would be better... Some people in private also told me that they consider 11 too risky target for this change. We will see. From owner-freebsd-arch@freebsd.org Tue Dec 29 19:18:35 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA53DA55F3D; Tue, 29 Dec 2015 19:18:35 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 913C91822; Tue, 29 Dec 2015 19:18:34 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBTJIRG0015683; Tue, 29 Dec 2015 14:18:27 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Tue, 29 Dec 2015 14:18:27 -0500 (EST) Date: Tue, 29 Dec 2015 14:18:27 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151229184405.GY3625@kib.kiev.ua> Message-ID: References: <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> <20151226234424.GJ3625@kib.kiev.ua> <20151228105157.GQ3625@kib.kiev.ua> <20151229184405.GY3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Dec 2015 19:18:35 -0000 On Tue, 29 Dec 2015, Konstantin Belousov wrote: > On Mon, Dec 28, 2015 at 11:59:02AM -0500, Daniel Eischen wrote: >> On Mon, 28 Dec 2015, Konstantin Belousov wrote: >> >>> On Sun, Dec 27, 2015 at 11:44:44AM -0500, Daniel Eischen wrote: >>> Adding the self-pointer is good idea, but it would not work for the >>> shared locks. If you are going to work on this, then I will scratch >>> my patch, to not impede on your work. >> >> Shared locks would work, but only for the new ABI (I guess that >> would be FBSD1.4 in HEAD, or FBSD1.5 if we bumped it again just >> for this). > I only mean that self-pointer sometimes would cause additional issues. > >> >> Another option is to just increase the size for the pthread synch >> types without really changing the first element (still a pointer >> to the allocated object - the implementation would stay the same). >> The only breakage would then be storage layout issues for new >> compiles linking against older ones. For FreeBSD-12, we could >> change the implementation to a real inlining (without changing >> the storage size), that gives a major release cycle for 3rd >> parties to make the change. I'm not sure if this would really >> be much of a benefit - the storage size change alone would >> probably be the greatest pain. > It is a good intermediate point in the whole plan of changing the ABI. > The work required to support pshared after the size is increased is not > completely trivial, but is not a rocket science either. But your note > about intermediate step might give additional freedom in executing the > whole plan. > >> >>> Taking out the inlining bits from the David patch, or (which would I do, >>> if doing this) just reimplementing it from scratch is easy enough and >>> just require some time. I estimated this job to take between one and >>> two weeks. >> >> I think a lot of David's patch is the renaming of all the >> elements of the public structs to prepend '__'. I was thinking >> it would be nice to have the public structs be something like >> this: >> >> struct pthread_mutex_t { >> uint32_t __x[IMPL_REQ + IMPL_SPARE + pad_to_CACHE_LINE_SIZE]; >> }; >> >> and then have libthr override the definition. That would >> make declaring PTHREAD_MUTEX_INITIALIZER, etc, a little >> magical, but avoid a lot of needless churn in libthr. > This is very good suggestion, I fully agree. There are some more details, > e.g. it would be better to use uint64_t or explicit align attribute, to > get proper alignment, but overall idea is sound, of course. > >> >> But, I agree, just doing the inlining is basic grunt work >> and limited in duration. Your estimate seems reasonable. >> >>> What is really thrilling is to manage the consequences of the ABI >>> breakage. When I did the evaluation for the FF project, I put a six >>> months extent for the whole work. This is for things like looking at >>> the ports impact, trying to know in advance where mixed things start to >>> break, monitoring the users complains about issues caused by the ABI >>> break etc. >> >> There's not much we can really do to manage it, just make it >> known that all ports need to be recompiled on 11 in order to >> be on the safe side. The libmap.conf trick will not work >> either. > No. This is probably the point where our positions diverge most. > > I do think that some mitigation measures can be applied, e.g. your > self-pointer trick, check for presence of 'other' lib when loading > given threading library, renaming etc. It should be thought out in > advance. > > Another thing to do is to monitor mailing lists, react to the user > issues, see if it is related to the ABI breakage, and possibly create > another measures which would make some issues less painful. > > I consider both activities absolutely vital when doing this surgery. > In principle, I may do both, but it requires more planning and is > not immediately reachable comparing with the off-page approach to > only get pshared right now. > >>> You see my position, I would avoid ABI breakage at all costs. If I >>> cannot talk you against this, please do not consider the work done after >>> the inlining patch is committed to HEAD, it is actually only start >>> there. >> >> Ok, I do think we want to do it at some point. Perhaps 12-current >> would be better... > > Some people in private also told me that they consider 11 too risky target > for this change. We will see. Perhaps another option is doing all of what we said above and also use your suggestion of libthr2, but have both libthr and libthr2, where libthr2 is labeled experimental or something similar. Have a WITH_LIBTHR2 build flag that conditionally inlines the pthread locks and enables (links w/-pthread) libthr2 for FreeBSD base OS. A note/comment for the WITH_LIBTHR2 flag states that all ports must also be recompiled to be on the safe side. This allows one to switch back and forth, as long as any affected ports are also rebuilt. If we were to introduce this in a future 12-current, we might have the WITH_LIBTHR_FOO flag anyway, at least initially, so perhaps it wouldn't be extra work. -- DE From owner-freebsd-arch@freebsd.org Wed Dec 30 14:15:17 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EE1DA5665E; Wed, 30 Dec 2015 14:15:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D596114EA; Wed, 30 Dec 2015 14:15:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBUEFAxe047375 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 30 Dec 2015 16:15:11 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBUEFAxe047375 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBUEFAX0047373; Wed, 30 Dec 2015 16:15:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 30 Dec 2015 16:15:10 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151230141510.GC3625@kib.kiev.ua> References: <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> <20151226234424.GJ3625@kib.kiev.ua> <20151228105157.GQ3625@kib.kiev.ua> <20151229184405.GY3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Dec 2015 14:15:17 -0000 On Tue, Dec 29, 2015 at 02:18:27PM -0500, Daniel Eischen wrote: > Perhaps another option is doing all of what we said above > and also use your suggestion of libthr2, but have both libthr > and libthr2, where libthr2 is labeled experimental or something > similar. Have a WITH_LIBTHR2 build flag that conditionally > inlines the pthread locks and enables (links w/-pthread) libthr2 > for FreeBSD base OS. A note/comment for the WITH_LIBTHR2 flag > states that all ports must also be recompiled to be on the > safe side. This allows one to switch back and forth, as long > as any affected ports are also rebuilt. > > If we were to introduce this in a future 12-current, we might > have the WITH_LIBTHR_FOO flag anyway, at least initially, so > perhaps it wouldn't be extra work. This is cheating, but I agree that this is quite plausible approach. I think about also adding the __INLINE_PTHREAD_LOCKS preprocessor macro to select inlined locks at the compilation time. In other words, the user which wants to start experimenting with the stuff would do cc -D__INLINE_PTHREAD_LOCKS -c .... cc -o program ... -lpthread2 I may just do this after I finish with my patches. From owner-freebsd-arch@freebsd.org Sat Jan 2 18:34:31 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABBD2A5FB1B; Sat, 2 Jan 2016 18:34:31 +0000 (UTC) (envelope-from alexmcwhirter@triadic.us) Received: from SMTP.Tech.Triadic.US (smtp.tech.triadic.us [98.102.61.98]) by mx1.freebsd.org (Postfix) with ESMTP id 80CC51E07; Sat, 2 Jan 2016 18:34:31 +0000 (UTC) (envelope-from alexmcwhirter@triadic.us) Received: from localhost (unknown [10.128.0.32]) by SMTP.Tech.Triadic.US (Postfix) with ESMTP id 65B6A1040690; Sat, 2 Jan 2016 13:17:50 -0500 (EST) X-Virus-Scanned: amavisd-new at Tech.Triadic.US Received: from SMTP.Tech.Triadic.US ([IPv6:::ffff:10.128.0.24]) by localhost (Milter1.Tech.Triadic.US [IPv6:::ffff:10.128.0.32]) (amavisd-new, port 10024) with LMTP id 6hk195kKEENT; Sat, 2 Jan 2016 13:17:48 -0500 (EST) Received: from webmail.tech.triadic.us (unknown [10.128.0.56]) by SMTP.Tech.Triadic.US (Postfix) with ESMTPSA id 55F6B1040432; Sat, 2 Jan 2016 13:17:48 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 02 Jan 2016 13:17:48 -0500 From: alexmcwhirter@triadic.us To: Daniel Rudy Cc: Jukka Ukkonen , Adrian Chadd , Anna Wilcox , Jordan Hubbard , freebsd-arch , Sean Bruno , sparc64@freebsd.org, Marius Strobl , Warner Losh , owner-freebsd-sparc64@freebsd.org Subject: Re: Sparc64 doesn't care about you, and you shouldn't care about Sparc64 In-Reply-To: <907918196.5618077.1448540168305.JavaMail.yahoo@mail.yahoo.com> References: <907918196.5618077.1448540168305.JavaMail.yahoo@mail.yahoo.com> Message-ID: <122e82d505433d5b052b0f6e5ab28d1d@triadic.us> X-Sender: alexmcwhirter@triadic.us User-Agent: Roundcube Webmail/1.0.5 X-Mailman-Approved-At: Sat, 02 Jan 2016 18:48:52 +0000 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jan 2016 18:34:31 -0000 It turns out that i will have some time open up this year and i would love to do some work on SPARC64, i have plenty of machines laying around. Im also not opposed to hosting a machine or two in my office's DC for others who want to help. I guess i need to know what needs attention first? I'd like to do some work on the bootloader and installer (zfs from installer would be nice) and of course sun4v. It seems what needs the most attention is the build toolchain, am i correct? I can't recall what version of GCC SPARC64 is currently running, but 4.9 shouldn't be too hard to implement. I think clang is not really considerable at the moment and GCC 5.0+ will take more work, but 4.9 is still newer than what most GNU/Linux distros are providing excluding the bleeding edge ones. This would be my first real contribution to FreeBSD, so any pointers or docs are graciously accepted. From owner-freebsd-arch@freebsd.org Sat Jan 2 19:10:48 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 110D8A5E48A; Sat, 2 Jan 2016 19:10:48 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: from mail.soaustin.net (mail.soaustin.net [66.135.54.68]) by mx1.freebsd.org (Postfix) with ESMTP id E9FCD1141; Sat, 2 Jan 2016 19:10:47 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: by mail.soaustin.net (Postfix, from userid 502) id 158245607D; Sat, 2 Jan 2016 13:10:41 -0600 (CST) Date: Sat, 2 Jan 2016 13:10:40 -0600 From: Mark Linimon To: alexmcwhirter@triadic.us Cc: Daniel Rudy , Adrian Chadd , Anna Wilcox , sparc64@freebsd.org, Marius Strobl , Sean Bruno , Jukka Ukkonen , freebsd-arch@FreeBSD.org Subject: sparc64 TODO list (was: Re: Sparc64 doesn't care about you, and you shouldn't care about Sparc64) Message-ID: <20160102191040.GA1850@lonesome.com> References: <907918196.5618077.1448540168305.JavaMail.yahoo@mail.yahoo.com> <122e82d505433d5b052b0f6e5ab28d1d@triadic.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <122e82d505433d5b052b0f6e5ab28d1d@triadic.us> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jan 2016 19:10:48 -0000 [replies trimmed down a bit to just the most interested parties] On Sat, Jan 02, 2016 at 01:17:48PM -0500, alexmcwhirter@triadic.us wrote: > I guess I need to know what needs attention first? I think it depends on what you'd like to do. Here at the house I have the following status: - on hard drive: 10.2-PRERELEASE FreeBSD 10.2-PRERELEASE #0 r284970: Wed Jul 1 03:30:12 UTC 2015, up 72 days, 13:58. This is building packages. - netboot: sparc64-10 20151230 working. - netboot: sparc64-11 20151222 fails: "panic: pcib0: fatal DMC/PEC error" > I'd like to do some work on the bootloader and installer (zfs from > installer would be nice) I know there is some work being done on the bootloader for x86. I do not know if this carries over to non-x86. Hopefully someone on arch@ will comment. If not, please ping me offline. > and of course sun4v. A build of sun4v from 8-STABLE (last version it existed) on 20151120 just hangs. I believe the netboot setup is the same as for the above, with 90% confidence. IMHO it's going to take a great deal of work. I will probably only tinker with it from time to time. > It seems what needs the most attention is the build toolchain, am I > correct? Correct. There was some discussion earlier in the thread about "external toolchain support". This would allow us to use various gcc/clangs without having them in the base system. This would also help us out on various of the other tier-2 archs (in particular, those for arm/mips). I do not know the latest state. Perhaps a check of the freebsd-toolchain@ archives might help. If not, email me offline and I'll put you in touch with the right people. This is going to be a very involved task, however, so FYI. > I think clang is not really considerable at the moment We took a look at it several weeks ago and it needed help. I can put you in touch with the person who was interested. > This would be my first real contribution to FreeBSD, so any pointers > or docs are graciously accepted. First, thanks for the offers :-) Second, please be patient when waiting for answers. sparc64 support is on a "as time is available" status and not many developers prioritize it. I split my time between it and powerpc64 and ports work. I'll email you some of the basic documentation pointers offline from this discussion. mcl From owner-freebsd-arch@freebsd.org Sat Jan 2 19:09:57 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C2CFA5E3E1; Sat, 2 Jan 2016 19:09:57 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from d.mail.sonic.net (d.mail.sonic.net [64.142.111.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C73F1FFE; Sat, 2 Jan 2016 19:09:56 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from comporellon.tachypleus.net (75-101-50-44.static.sonic.net [75.101.50.44]) (authenticated bits=0) by d.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id u02J9hPV018405 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 2 Jan 2016 11:09:44 -0800 Subject: Re: Sparc64 doesn't care about you, and you shouldn't care about Sparc64 To: alexmcwhirter@triadic.us, Daniel Rudy References: <907918196.5618077.1448540168305.JavaMail.yahoo@mail.yahoo.com> <122e82d505433d5b052b0f6e5ab28d1d@triadic.us> Cc: Adrian Chadd , Anna Wilcox , sparc64@freebsd.org, Marius Strobl , Sean Bruno , freebsd-arch , Jukka Ukkonen , Jordan Hubbard , owner-freebsd-sparc64@freebsd.org From: Nathan Whitehorn Message-ID: <56882077.6080703@freebsd.org> Date: Sat, 2 Jan 2016 11:09:43 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <122e82d505433d5b052b0f6e5ab28d1d@triadic.us> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Sonic-CAuth: UmFuZG9tSVY3DE3kPEYvHNvD9Hl/WB23naBit8i2FqcZGXb+h2JCMyzXY2lHa310TQPkCiBiA5HmyxoJMNREVZt9qa8ouFt96wAYEIJxrcM= X-Sonic-ID: C;1DPOYYSx5RGbk+1jAoajKQ== M;6iMJYoSx5RGbk+1jAoajKQ== X-Spam-Flag: No X-Sonic-Spam-Details: 0.0/5.0 by cerberusd X-Mailman-Approved-At: Sat, 02 Jan 2016 19:45:25 +0000 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jan 2016 19:09:57 -0000 On 01/02/16 10:17, alexmcwhirter@triadic.us wrote: > It turns out that i will have some time open up this year and i would > love to do some work on SPARC64, i have plenty of machines laying > around. Im also not opposed to hosting a machine or two in my office's > DC for others who want to help. > > I guess i need to know what needs attention first? I'd like to do some > work on the bootloader and installer (zfs from installer would be > nice) and of course sun4v. It seems what needs the most attention is > the build toolchain, am i correct? I can't recall what version of GCC > SPARC64 is currently running, but 4.9 shouldn't be too hard to > implement. I think clang is not really considerable at the moment and > GCC 5.0+ will take more work, but 4.9 is still newer than what most > GNU/Linux distros are providing excluding the bleeding edge ones. ZFS should already work from the installer if you use the partition editor rather than the x86-only "ZFS" tool. I haven't tested it, but the support has been there for some time and should work. Toolchain work would be really helpful, I think (as would testing of those installer bits!). -Nathan > > This would be my first real contribution to FreeBSD, so any pointers > or docs are graciously accepted. > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >