From owner-freebsd-arch@FreeBSD.ORG Sun Feb 1 14:18:15 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 764D1EC for ; Sun, 1 Feb 2015 14:18:15 +0000 (UTC) Received: from song.uckmgae.com (song.uckmgae.com [107.179.37.84]) by mx1.freebsd.org (Postfix) with ESMTP id 0BE76A2F for ; Sun, 1 Feb 2015 14:18:15 +0000 (UTC) To: freebsd-arch@freebsd.org Subject: target email marketing Message-ID: Date: Sun, 01 Feb 2015 15:16:45 +0100 From: "John" Reply-To: alexliucontact@sina.com MIME-Version: 1.0 X-Mailer-LID: 5 X-Mailer-RecptId: 6480503 X-Mailer-SID: 92 X-Mailer-Sent-By: 1 Content-Type: text/plain; format=flowed; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 14:18:15 -0000 Hi, You are receiving this email because we wish you to use our target email marketing service. We specialize in providing target email marketing services to a number of businesses all over the world! Email marketing is one of the best marketing strategies of all time and has helped many businesses globally achieve their goals, double their profits and increase their client base. We have worked on a number of projects and campaigns, all our packages are tailor made and designed according to your requirements. Increase your client base and market your product to millions or let us bring the buying leads for you! We would love to be your marketing partners, would you be interested in email marketing services for your product or service? We can always help your business reach the next level! Our goal is to increase your business sales 2-5 times than now. If you would require more information please send us an email and we would be glad to discuss the project requirements with you! Looking forward to your positive response. Remember! It won't sell if nobody knows you have it. Kind Regards Email Marketing Specialist Email: alexliucontact@sina.com ------------------------------------------------- This e-mail message and its attachments (if any) are intended solely for the use of the addressee(s) hereof. In addition, this message and the attachments (if any) may contain information that is confidential, privileged and exempt from disclosure under applicable law. If you are not the intended recipient of this message, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. If you have received this message in error, please promptly notify the sender and immediately delete this message from your system. From owner-freebsd-arch@FreeBSD.ORG Sun Feb 1 15:35:01 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08252458; Sun, 1 Feb 2015 15:35:01 +0000 (UTC) Received: from smtp2.ore.mailhop.org (smtp2.ore.mailhop.org [54.186.57.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD69419F; Sun, 1 Feb 2015 15:35:00 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by smtp2.ore.mailhop.org with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.82) (envelope-from ) id 1YHvqp-0000fc-NU; Sun, 01 Feb 2015 14:50:15 +0000 Received: from revolution.hippie.lan (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id t11EoE7C073554; Sun, 1 Feb 2015 07:50:14 -0700 (MST) (envelope-from ian@freebsd.org) X-Mail-Handler: DuoCircle Outbound SMTP X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@duocircle.com (see https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information for abuse reporting information) X-MHO-User: U2FsdGVkX18uYwlF8eAv8lJAljjxMCGy Message-ID: <1422802214.15718.253.camel@freebsd.org> Subject: Re: Wrapper API for static bus_dma allocations From: Ian Lepore To: John Baldwin Date: Sun, 01 Feb 2015 07:50:14 -0700 In-Reply-To: <2800970.jY4xzTy9Hz@ralph.baldwin.cx> References: <2800970.jY4xzTy9Hz@ralph.baldwin.cx> Content-Type: text/plain; charset="us-ascii" X-Mailer: Evolution 3.12.8 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: 'freebsd-arch' X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 15:35:01 -0000 On Thu, 2015-01-29 at 15:37 -0500, John Baldwin wrote: > The bus_dma API to allocate a chunk of static DMA'able memory (e.g. for > descriptor rings) can be a bit obtuse to use in that it require a bit of > boilerplate to create a tag, allocate memory for the tag, then load it to get > the bus address. Similarly, freeing it also requires several steps. In > addition, some of the steps are a bit subtle (e.g. the need to check for an > error in the bus_dma callback instead of from bus_dmamap_load()) and not all > drivers get those correct. > > To try to make this simpler I've written a little wrapper API that tries to > provide a single call to allocate a buffer and a single call to free a buffer. > Each buffer is described by a structure defined by the API, and if the call to > allocate a buffer succeeds, the structure contains both a pointer to the > buffer in the kernel's address space as well as a bus address of the buffer. > > In the interests of simplicity, this API does not allow the buffer to be quite > as fully configured as the underlying bus_dma API, instead it aims to handle > the most common cases that are used in most drivers. As such, it assumes that > the buffer must be one contiguous range of DMA address space, and the only > parameters that can be specified are the parent tag, the alignment of the > buffer, the lowaddr parameter, the size of the buffer to allocate, and the > flags parameter that is normally passed to bus_dmamem_alloc(). I believe that > this should be sufficient to cover the vast majority of the drivers in our > tree. > > I've included below a patch that contains the wrapper API along with a sample > conversion of the ndis driver (chosen at random). If folks like this idea I > will update the patch to include manpage changes as well. > > --- //depot/vendor/freebsd/src/sys/compat/ndis/subr_ndis.c > +++ //depot/user/jhb/cleanup/sys/compat/ndis/subr_ndis.c [...] > +struct bus_dmamem { > + bus_dma_tag_t dma_tag; > + bus_dmamap_t dma_map; > + void *dma_vaddr; > + bus_addr_t dma_baddr; > +}; > + > +/* > + * Allocate a mapping. On success, zero is returned and the 'dma_vaddr' > + * and 'dma_baddr' fields are populated with the virtual and bus addresses, > + * respectively, of the mapping. > + */ > +int bus_dma_mem_create(struct bus_dmamem *mem, bus_dma_tag_t parent, > + bus_size_t alignment, bus_addr_t lowaddr, > + bus_size_t len, int flags); > + > +/* > + * Release a mapping created by bus_dma_mem_create(). > + */ > +void bus_dma_mem_free(struct bus_dmamem *mem); > + > #endif /* _BUS_DMA_H_ */ > > You introduce new functions named bus_dma_mem_xxxx(), and a new structure bus_dmamem. It's already hard enough to remember the oddball mix of underbars in busdma naming, can the new struct be bus_dma_mem? Also, the combo of create/free is most unsatisfying. Typically alloc/free come in pairs, as do create/destroy. These pairings are used pretty consistantly in busdma now, it'd be nice to not have to remember this pair is an exception. (I'm agnostic about whether the new functionality should be alloc/delete or create/destroy.) -- Ian From owner-freebsd-arch@FreeBSD.ORG Sun Feb 1 17:29:34 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7797256C; Sun, 1 Feb 2015 17:29:34 +0000 (UTC) Received: from smtp2.ore.mailhop.org (smtp2.ore.mailhop.org [54.186.57.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 58C0AD3B; Sun, 1 Feb 2015 17:29:33 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by smtp2.ore.mailhop.org with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.82) (envelope-from ) id 1YHyKy-0007US-UD; Sun, 01 Feb 2015 17:29:33 +0000 Received: from revolution.hippie.lan (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id t11HTVDD075015; Sun, 1 Feb 2015 10:29:31 -0700 (MST) (envelope-from ian@freebsd.org) X-Mail-Handler: DuoCircle Outbound SMTP X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@duocircle.com (see https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information for abuse reporting information) X-MHO-User: U2FsdGVkX19/wESK6aG5TCKmruWSESyP Message-ID: <1422811771.15718.269.camel@freebsd.org> Subject: Re: Wrapper API for static bus_dma allocations From: Ian Lepore To: John Baldwin Date: Sun, 01 Feb 2015 10:29:31 -0700 In-Reply-To: <2800970.jY4xzTy9Hz@ralph.baldwin.cx> References: <2800970.jY4xzTy9Hz@ralph.baldwin.cx> Content-Type: text/plain; charset="us-ascii" X-Mailer: Evolution 3.12.8 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: 'freebsd-arch' X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Feb 2015 17:29:34 -0000 On Thu, 2015-01-29 at 15:37 -0500, John Baldwin wrote: [...] > +int > +bus_dma_mem_create(struct bus_dmamem *mem, bus_dma_tag_t parent, > + bus_size_t alignment, bus_addr_t lowaddr, bus_size_t len, int flags) > +{ > + struct bus_dma_mem_cb_data d; > + int error; > + > + bzero(mem, sizeof(*mem)); > + error = bus_dma_tag_create(parent, alignment, 0, lowaddr, > + BUS_SPACE_MAXADDR, NULL, NULL, len, 1, len, 0, NULL, NULL, > + &mem->dma_tag); > + if (error) { > + bus_dma_mem_free(mem); > + return (error); > + } > + error = bus_dmamem_alloc(mem->dma_tag, &mem->dma_vaddr, flags, > + &mem->dma_map); > + if (error) { > + bus_dma_mem_free(mem); > + return (error); > + } > + d.mem = mem; > + error = bus_dmamap_load(mem->dma_tag, mem->dma_map, mem->dma_vaddr, len, > + bus_dma_mem_cb, &d, BUS_DMA_NOWAIT); > + if (error == 0) > + error = d.error; > + if (error) { > + bus_dma_mem_free(mem); > + return (error); > + } > + return (0); > +} > +[...] A couple more thoughts on this... It would be good to pass the flags to both bus_dma_tag_create() and bus_dmamem_alloc(). While the manpage seems to indicate that certain flags are specific to tag-create or mem-alloc, there is actually no overlap in values, so passing flags to tag-create that a given arch only uses in mem-alloc should be harmless. Implementations may be able optimize things if they can see flags at tag-create time that normally apply to allocation. Hmmm, it just popped into my head that a new flag that means "this is going to be a combined tag+map+buffer allocation" might allow for even more optimizations. For example you can avoid allocating any resources related to bouncing if you know that the only mapping that will ever be created with that tag+map is one belonging to a buffer that the implementation can carefully allocate in a way that avoids bouncing. Another way to achieve that would be to allow the whole implementation of bus_dma_mem_create() to be supplied by the arch, rather than having it be generic in kern. The more I think about it, the more I like this. Could we eliminate the alignment arg and instead document that the alignment of the allocated buffer will be at least the same as the size argument up to PAGE_SIZE, and will be PAGE_SIZE for anything 1 page or larger? There is a standard uma-based allocator available in kern/subr_busdma_buffalloc.c that implements this behavior. It's currently used only by arm, but should be trivial to use more widely. When I investigated alignment usage a couple years ago, I found nearly all existing uses of alignment specify a value <= the size in the tag. -- Ian From owner-freebsd-arch@FreeBSD.ORG Tue Feb 3 20:34:37 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 899D3815 for ; Tue, 3 Feb 2015 20:34:37 +0000 (UTC) Received: from mail166.atl21.rsgsv.net (mail166.atl21.rsgsv.net [205.201.133.166]) by mx1.freebsd.org (Postfix) with ESMTP id EC189A9D for ; Tue, 3 Feb 2015 20:34:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=k1; d=mail166.atl21.rsgsv.net; h=Subject:From:Reply-To:To:Date:Message-ID:List-ID:List-Unsubscribe:Sender:Content-Type:MIME-Version; i=Nonna=3Dsalvagedrive.com@mail166.atl21.rsgsv.net; bh=V0VnIA4T3VaNTKlqoWoxcr4OGsA=; b=tVIj03jzoui2Fww5p2W+EN1ujPyuCubYZ4fHXTMx/51c+5qp38qlxrZnc7g/fs7vcVyE0SFuwxL7 8UloRF5BPqGaXX0Q/ZHaLnBHDqPeOmyYVFq1JRJ9gD6gzSP9BRmbYiWbUE+CgY3+jRuXzjtRtyoy dBiwigrC1pcZ7eoqpwg= DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=k1; d=mail166.atl21.rsgsv.net; b=MkCS2hhmcOoIynEOhh2orDaFIvMqDAP7syvcULFhA7SeyQY5mwxMvLasPJ00jTHdqVFI0HX/iScv xPxAj0WYzYtFJVkfNvkRke3Dsp3iox8mkGLv3nYnoajLiYzoLdAIFtg+B3RlmFJDN/CHdHx5W383 ROTFZgp7weKCxQ6n8Pc=; Received: from (127.0.0.1) by mail166.atl21.rsgsv.net id hq4odq1lgi4i for ; Tue, 3 Feb 2015 20:34:27 +0000 (envelope-from ) Subject: =?utf-8?Q?Clean=20Title=20Cars=2C=20No=20Damage=2C=20No=20Auction=20Fees?= From: =?utf-8?Q?SalvageDrive.com?= Reply-To: =?utf-8?Q?SalvageDrive.com?= To: =?utf-8?Q??= Date: Tue, 3 Feb 2015 20:34:27 +0000 Message-ID: <38668d8a390eaafe346aac8d5c538f13530.20150203203415@mail166.atl21.rsgsv.net> X-Mailer: MailChimp Mailer - **CID347d6274fac538f13530** X-Campaign: mailchimp38668d8a390eaafe346aac8d5.347d6274fa X-campaignid: mailchimp38668d8a390eaafe346aac8d5.347d6274fa X-Report-Abuse: Please report abuse for this campaign here: http://www.mailchimp.com/abuse/abuse.phtml?u=38668d8a390eaafe346aac8d5&id=347d6274fa&e=c538f13530 X-MC-User: 38668d8a390eaafe346aac8d5 X-Feedback-ID: 30493583:30493583.927605:us8:mc X-Accounttype: pd Sender: "SalvageDrive.com" x-mcda: FALSE MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8"; format="fixed" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Feb 2015 20:34:37 -0000 https://www.salvagedrive.com ** Special Deals from Salvage Drive ------------------------------------------------------------ WILL MATCH OUR COMPETITORS PRICES! NO AUCTION FEE DEALS!!! (https://www.salvagedrive.com) ** 2008 TOYOTA CAMRY ------------------------------------------------------------ Clean Title 64=2C180 miles $6=2C800 https://www.salvagedrive.com/Cars/Category?SysId=3D901111647 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111647) ** 2006 LEXUS RX 330 ------------------------------------------------------------ Clean Title 178=2C684 miles $6=2C400 https://www.salvagedrive.com/Cars/Category?SysId=3D901071161 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1071161) ** 2010 HONDA ACCORD ------------------------------------------------------------ Clean Title 77=2C991 miles $8=2C900 https://www.salvagedrive.com/Cars/Category?SysId=3D901062007 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1062007) ** 2005 TOYOTA SIENNA ------------------------------------------------------------ Clear Title 101=2C651 miles $2=2C900 https://www.salvagedrive.com/Cars/Category?SysId=3D901111649 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111649) ** 2007 TOYOTA RAV4 ------------------------------------------------------------ Clean Title 87=2C500miles $9=2C100 https://www.salvagedrive.com/Cars/Category?SysId=3D901111646 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111646) ** 2007 LEXUS RX 350 ------------------------------------------------------------ Clean Title 140=2C301 miles $8=2C800 https://www.salvagedrive.com/Cars/Category?SysId=3D901111645 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111645) ** 2009 NISSAN MURANO ------------------------------------------------------------ Clean Title 166=2C730 miles $8=2C100 https://www.salvagedrive.com/Cars/Category?SysId=3D901085256 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1085256) ** 2008 HONDA CR-V ------------------------------------------------------------ Clean Title 124=2C601 miles $9=2C300 https://www.salvagedrive.com/Cars/Category?SysId=3D901111650 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111650) ** 2010 TOYOTA VENZA ------------------------------------------------------------ Clean Title 68=2C417 miles $12=2C900 https://www.salvagedrive.com/Cars/Category?SysId=3D901111644 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111644) ** 2008 LEXUS RX 350 ------------------------------------------------------------ Clean Title 93=2C212 miles $9=2C900 https://www.salvagedrive.com/Cars/Category?SysId=3D901111648 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111648) ** 2008 HONDA PILOT ------------------------------------------------------------ Clean Title 28=2C515 miles $19=2C000 https://www.salvagedrive.com/Cars/Category?SysId=3D901111651 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1111651) ** 2008 TOYOTA TACOMA ------------------------------------------------------------ Clean Title 189=2C341 miles $7=2C950 https://www.salvagedrive.com/Cars/Category?SysId=3D901024318 CLICK FOR MORE INFO (https://www.salvagedrive.com/Cars/Category?SysId=3D90= 1024318) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ** LET US SEARCH FOR YOU! TELL US WHAT YOU ARE LOOKING FOR AND OUR EXPERTS= WILL SEARCH FOR YOU BASED ON YOUR SPECIFIC BUDGET AND REQUIREMENTS. (http= s://www.salvagedrive.com) ** (https://www.salvagedrive.com) ** SalvageDrive.com (https://www.salvagedrive.com) 1-844-227-7411 Toll Free 1-347-492-1727 Tel. Skype: salvagedrive ** info@SalvageDrive.com (mailto:info@SalvageDrive.com) Don't forget to add info@salvagedrive.com to your Address Book to keep it= from skipping your inbox or getting caught in spam filters. We want your experience with the Salvage Drive to be as smooth and reassur= ing as possible. Accordingly=2C we diligently safeguard your privacy. If y= ou wish to review our Privacy Policy at any time=2C please click on the li= nk below=2C or copy and paste it into your web browser's location window ** Salvage Drive Privacy Policy (http://sdimages.salvagedrive.com/salvaged= rive/PrivacyPolicy.pdf) __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= _______________________ You can choose to unsubscribe from our Email Newsletters service by replyi= ng to this email with the word "STOP" and we will remove you from any futu= re mailings. __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= __________________________________________________________________________= _______________________ =C2=A9 2014 SalvageDrive=2C Inc. | All rights reserved Salvage Drive=2C Inc. | 217 Broadway | Suite 505 | New York | NY | 10007 This email was sent to arch@freebsd.org (mailto:arch@freebsd.org) why did I get this? (http://salvagedrive.us8.list-manage2.com/about?u=3D38= 668d8a390eaafe346aac8d5&id=3Db173b821c7&e=3Dc538f13530&c=3D347d6274fa) u= nsubscribe from this list (http://salvagedrive.us8.list-manage1.com/unsubs= cribe?u=3D38668d8a390eaafe346aac8d5&id=3Db173b821c7&e=3Dc538f13530&c=3D347d6= 274fa) update subscription preferences (http://salvagedrive.us8.list-m= anage.com/profile?u=3D38668d8a390eaafe346aac8d5&id=3Db173b821c7&e=3D= c538f13530) Salvage Drive=2C Inc. =C2=B7 217 Broadway =C2=B7 Suite 505 =C2=B7 New York= =2C NY 10007 =C2=B7 USA From owner-freebsd-arch@FreeBSD.ORG Thu Feb 5 17:57:08 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A5FC6D4A for ; Thu, 5 Feb 2015 17:57:08 +0000 (UTC) Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C14C62 for ; Thu, 5 Feb 2015 17:57:08 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id eu11so11495291pac.2 for ; Thu, 05 Feb 2015 09:57:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:content-type :content-transfer-encoding:subject:message-id:date:to:mime-version; bh=i1kH+8fEI760/xTq87e6c0qheBIE33L3KV8pn/CFQKw=; b=MNLlBbjiCeDoUUMeBkpEjFyEo6vosS5LBWI95dI+WRgbnQ5tAFPhNdaDEVLBcrUbjz TXy/BqInSaP9w9tzdOgZHNTP+EopO/YEQMCPgVOOpFrFAsoQ7f3DFeDDQ6zlfzNbj5x9 yoW28DfaJwRHFbysccGTML1hLN7NWxCTOodsQGvnqnxzPLA+23bEzrfG0P4YPQG7NHml 53do6qrfPrXKRqQmbtVDVxbUQyt/tADpH8Y8/m5CbG76LvxWLf6R8yFnf2I+ZHTaYMfk 6HgjzH024FpTmRZl9Pw+O/1kbJE6jmG0b79Owtz1o/V3p8O/eCa1056TRZj+5m8YnI5M /s1Q== X-Gm-Message-State: ALoCoQmML4Cgf3nUVjBF6jBLpPsw2/D7nosRf0KXSLCbpuaDmW8tipPFwlAp4CDEaO6+6tU7QiqK X-Received: by 10.68.227.6 with SMTP id rw6mr7689375pbc.2.1423159022650; Thu, 05 Feb 2015 09:57:02 -0800 (PST) Received: from [10.64.26.90] ([69.53.236.236]) by mx.google.com with ESMTPSA id ex1sm5659608pbc.80.2015.02.05.09.57.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Feb 2015 09:57:01 -0800 (PST) Sender: Warner Losh From: Warner Losh Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Better way to do conditional inclusion in make Message-Id: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Date: Thu, 5 Feb 2015 10:56:59 -0700 To: "freebsd-arch@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) X-Mailer: Apple Mail (2.2070.6) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 17:57:08 -0000 Greetings, I=E2=80=99ve started a pass through the tree to cleanup how we do = conditional inclusion in our build system. https://reviews.freebsd.org/D1781 It moves away from=20 .if ${MK_foo} !=3D =E2=80=9Cno=E2=80=9D FILES+=3D files .endif and SUBDIRS=3D =E2=80=A6 ${_foo} =E2=80=A6 ... .if ${MK_foo} !=3D =E2=80=9Cno=E2=80=9D _foo+=3D foo .endif and instead more directly assigns things. We know that MK_foo is always = going to be yes or no for build options. We can leverage that fact, and the fact that bmake is = so much better at variable expansion than fmake was (especially in the early days) to instead move = to something like: FILES=3Dlist of unconditional files here ${FILES.yes} FILES.${MK_foo}+=3Dfoo bar biz FILES.${MK_baz}+=3Dbaz bing boo which eliminates a whole lot of needless .if / .endif lines, lots of = extra blank lines, etc. Comments? Warner From owner-freebsd-arch@FreeBSD.ORG Thu Feb 5 18:09:48 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1ADFF51 for ; Thu, 5 Feb 2015 18:09:48 +0000 (UTC) Received: from mail-lb0-x22e.google.com (mail-lb0-x22e.google.com [IPv6:2a00:1450:4010:c04::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 845761FD for ; Thu, 5 Feb 2015 18:09:48 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id f15so10053008lbj.5 for ; Thu, 05 Feb 2015 10:09:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=fEc+ll7Qe7xmyeeeiGnngiPWAgNF2VDbGC8RIUwmt9o=; b=lAFBHGK/t9mF0PHrGMZAUa+ZwpH/kSRplBvBZqr8M1SoJRRQTQHHCDjVG1b37vyi/9 LMSErKLfGVuZoJ+bErRah/T1p5M98tsZMV60yQg/bWhDhFaB/nBCJutWAXjWGkw+H/bl LnpYbHDzEaKHLQxI0DRKuZVLIuApzCjc12xTjZeNTezoxnMAny7H0tRGxodhMFh8RKvO 6YzocJQjg/0amGr4QlQHyLud7cnMirOhzO8pwgqOZMUEauHXzv6fZ1EUa2Osn6zpzXu/ QGk2wUDrTRM8pH1fD33fG8/Vmny5UdnaHdPuS8pl+cVQwxaWqYafRhCrizStHdD1ctEm myzg== MIME-Version: 1.0 X-Received: by 10.112.32.34 with SMTP id f2mr5372968lbi.50.1423159786115; Thu, 05 Feb 2015 10:09:46 -0800 (PST) Sender: chmeeedalf@gmail.com Received: by 10.25.160.202 with HTTP; Thu, 5 Feb 2015 10:09:46 -0800 (PST) In-Reply-To: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Date: Thu, 5 Feb 2015 10:09:46 -0800 X-Google-Sender-Auth: FJfN2pbYR57M__aEAWaowRP6GmA Message-ID: Subject: Re: Better way to do conditional inclusion in make From: Justin Hibbits To: Warner Losh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 18:09:49 -0000 On Thu, Feb 5, 2015 at 9:56 AM, Warner Losh wrote: > Greetings, > > I=E2=80=99ve started a pass through the tree to cleanup how we do conditi= onal inclusion in our build system. > > https://reviews.freebsd.org/D1781 > > It moves away from > > .if ${MK_foo} !=3D =E2=80=9Cno=E2=80=9D > FILES+=3D files > .endif > > and > > SUBDIRS=3D =E2=80=A6 ${_foo} =E2=80=A6 > ... > .if ${MK_foo} !=3D =E2=80=9Cno=E2=80=9D > _foo+=3D foo > .endif > > and instead more directly assigns things. We know that MK_foo is always g= oing to be yes or no > for build options. We can leverage that fact, and the fact that bmake is = so much better at variable > expansion than fmake was (especially in the early days) to instead move t= o something like: > > FILES=3Dlist of unconditional files here ${FILES.yes} > FILES.${MK_foo}+=3Dfoo bar biz > FILES.${MK_baz}+=3Dbaz bing boo > > which eliminates a whole lot of needless .if / .endif lines, lots of extr= a blank lines, etc. > > Comments? > > Warner I like it. It makes it easier to see at a glance what options are available, and how the files are guarded. - Justin From owner-freebsd-arch@FreeBSD.ORG Thu Feb 5 18:35:18 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D66B910 for ; Thu, 5 Feb 2015 18:35:18 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0129.outbound.protection.outlook.com [65.55.169.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F035C757 for ; Thu, 5 Feb 2015 18:35:17 +0000 (UTC) Received: from CO2PR05CA043.namprd05.prod.outlook.com (10.141.241.171) by BLUPR05MB436.namprd05.prod.outlook.com (10.141.27.152) with Microsoft SMTP Server (TLS) id 15.1.75.20; Thu, 5 Feb 2015 18:35:09 +0000 Received: from BL2FFO11FD006.protection.gbl (2a01:111:f400:7c09::115) by CO2PR05CA043.outlook.office365.com (2a01:111:e400:1429::43) with Microsoft SMTP Server (TLS) id 15.1.81.19 via Frontend Transport; Thu, 5 Feb 2015 18:35:08 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD006.mail.protection.outlook.com (10.173.161.2) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Thu, 5 Feb 2015 18:35:07 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 5 Feb 2015 10:35:05 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t15IZ5W71796; Thu, 5 Feb 2015 10:35:05 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id EFCBF580A3; Thu, 5 Feb 2015 10:35:04 -0800 (PST) To: Warner Losh Subject: Re: Better way to do conditional inclusion in make In-Reply-To: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Comments: In-reply-to: Warner Losh message dated "Thu, 05 Feb 2015 10:56:59 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 5 Feb 2015 10:35:04 -0800 Message-ID: <10594.1423161304@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; bsdimp.com; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(24454002)(76506005)(77156002)(57986006)(62966003)(48376002)(50466002)(105596002)(106466001)(50226001)(2950100001)(558084003)(33716001)(77096005)(92566002)(110136001)(47776003)(117636001)(46102003)(50986999)(19580405001)(87936001)(19580395003)(76176999)(6806004)(86362001)(43043002)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:BLUPR05MB436; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:nov; PTR:InfoDomainNonexistent; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:BLUPR05MB436; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:BLUPR05MB436; X-Forefront-PRVS: 0478C23FE0 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:BLUPR05MB436; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2015 18:35:07.2574 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR05MB436 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 18:35:18 -0000 Warner Losh wrote: > FILES=list of unconditional files here ${FILES.yes} > FILES.${MK_foo}+=foo bar biz > FILES.${MK_baz}+=baz bing boo cute ;-) From owner-freebsd-arch@FreeBSD.ORG Thu Feb 5 21:32:15 2015 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A2DB95B9 for ; Thu, 5 Feb 2015 21:32:15 +0000 (UTC) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tensor.andric.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C47AF25 for ; Thu, 5 Feb 2015 21:32:14 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7::c54:7c69:b493:a847] (unknown [IPv6:2001:7b8:3a7:0:c54:7c69:b493:a847]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id A8D515C2E; Thu, 5 Feb 2015 22:32:11 +0100 (CET) Subject: Re: Better way to do conditional inclusion in make Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Content-Type: multipart/signed; boundary="Apple-Mail=_D69D7A71-52BE-496E-9387-44D267751ACD"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Pgp-Agent: GPGMail 2.5b4 (c621b2a+) From: Dimitry Andric In-Reply-To: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Date: Thu, 5 Feb 2015 22:32:06 +0100 Message-Id: References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> To: Warner Losh X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 21:32:15 -0000 --Apple-Mail=_D69D7A71-52BE-496E-9387-44D267751ACD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 05 Feb 2015, at 18:56, Warner Losh wrote: >=20 > We know that MK_foo is always going to be yes or no > for build options. We can leverage that fact, and the fact that bmake = is so much better at variable > expansion than fmake was (especially in the early days) to instead = move to something like: >=20 > FILES=3Dlist of unconditional files here ${FILES.yes} > FILES.${MK_foo}+=3Dfoo bar biz > FILES.${MK_baz}+=3Dbaz bing boo >=20 > which eliminates a whole lot of needless .if / .endif lines, lots of = extra blank lines, etc. >=20 > Comments? One disadvantage is that you then cannot intersperse MK_foo files or subdirs in between unconditional ones, and keep a certain order, e.g. like: FILES+=3D a b c .if ${MK_foo} !=3D "no" FILES+=3D d e f .endif FILES+=3D g h i Of course, this is only important for a few particular places, most of the tree should not care too much about the order in which subdirs or files are built. -Dimitry --Apple-Mail=_D69D7A71-52BE-496E-9387-44D267751ACD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.26 iEYEARECAAYFAlTT4VsACgkQsF6jCi4glqNuoACfRtkwBQTwesdlmOq9LNish03G jvIAnizuYR4uNVhg83D0kKUCQ7+pg0X0 =3wVX -----END PGP SIGNATURE----- --Apple-Mail=_D69D7A71-52BE-496E-9387-44D267751ACD-- From owner-freebsd-arch@FreeBSD.ORG Thu Feb 5 22:32:43 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 83EB1564 for ; Thu, 5 Feb 2015 22:32:43 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CE478DC for ; Thu, 5 Feb 2015 22:32:42 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id h15so2219307igd.4 for ; Thu, 05 Feb 2015 14:32:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=2UQMeoXxvvGzeUBlPQKyrDPqaucoSznttY/sbCBxIp8=; b=fc8fDV3O6X5mrzz8Ig/6XHHLyyFF2J4hy1rkh45MW6zONmBo+FyTcb8oa414i2Bvzc H614KI8WyoPyGLJNg30jtDeTT1hN/EQmNBvn8E3/rK4V+vdw5PzF373B/fblrto/Tu64 FA8KaUTctrKd2wwbjjQkLIiC94qZWdqSZu329qsvgj0GHO+VveahdKTzEvfFmy2wnLZa JbMNpCdr3wYawWYl/C3u0NInQGP7xB2oS8zRAg0tvGNKis+tFh0pKibuIvXOHISu12k7 RD6BCvRqn6aqS7EzmnYdU9JBXiyZ6MmnSs1PWyVcTvd+K5cL2Ae3VT2p6WIrUleT5xSP ZNdQ== X-Gm-Message-State: ALoCoQmXOxfG+arMKp7MvzaOu7Vd2S024OKowk/IDRd/lxNOE6OhT8Z52xYQ+bEp/iG2uO0kERhe X-Received: by 10.50.142.99 with SMTP id rv3mr966076igb.11.1423175562101; Thu, 05 Feb 2015 14:32:42 -0800 (PST) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id g11sm272703iod.28.2015.02.05.14.32.41 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Feb 2015 14:32:41 -0800 (PST) Sender: Warner Losh Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Better way to do conditional inclusion in make From: Warner Losh In-Reply-To: Date: Thu, 5 Feb 2015 15:32:40 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <2F98809C-14AA-4658-BCF4-15A6179515C9@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> To: Dimitry Andric X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 22:32:43 -0000 > On Feb 5, 2015, at 2:32 PM, Dimitry Andric wrote: >=20 > On 05 Feb 2015, at 18:56, Warner Losh wrote: >>=20 >> We know that MK_foo is always going to be yes or no >> for build options. We can leverage that fact, and the fact that bmake = is so much better at variable >> expansion than fmake was (especially in the early days) to instead = move to something like: >>=20 >> FILES=3Dlist of unconditional files here ${FILES.yes} >> FILES.${MK_foo}+=3Dfoo bar biz >> FILES.${MK_baz}+=3Dbaz bing boo >>=20 >> which eliminates a whole lot of needless .if / .endif lines, lots of = extra blank lines, etc. >>=20 >> Comments? >=20 > One disadvantage is that you then cannot intersperse MK_foo files or > subdirs in between unconditional ones, and keep a certain order, e.g. > like: >=20 > FILES+=3D a b c > .if ${MK_foo} !=3D "no" > FILES+=3D d e f > .endif > FILES+=3D g h i >=20 > Of course, this is only important for a few particular places, most of > the tree should not care too much about the order in which subdirs or > files are built. I can=E2=80=99t think of any places where this matters. The only place = where it used to matter was library ordering, but that=E2=80=99s been fixed. And if there = is any other places in the tree where this matters, that should be listed as a real = dependency and not rely on the order which isn=E2=80=99t -j safe. Warner= From owner-freebsd-arch@FreeBSD.ORG Thu Feb 5 23:33:53 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A0F1BA6D; Thu, 5 Feb 2015 23:33:53 +0000 (UTC) Received: from mail-ig0-x22a.google.com (mail-ig0-x22a.google.com [IPv6:2607:f8b0:4001:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67B69EF3; Thu, 5 Feb 2015 23:33:53 +0000 (UTC) Received: by mail-ig0-f170.google.com with SMTP id l13so8938868iga.1; Thu, 05 Feb 2015 15:33:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6NIfPStCglRvtxI6R1v29JV0c04XhLSXCJr6IatXt0s=; b=X8JNTH2AB3VuBGp7ynoFueThUC34XYCrraqA/arWt0P9F+VrrRoBF+61YGD65WfJIm KMGcPsgRnyCVbovcod5AzarTpbfUrUFbUErNLh0k6MMTAa9NYacg7EGo/yZxO2EP4H+f H5F0S+AY0e0mv814QyzjCxK7D9RKBb63fpa3ThucqgNEDyLJYrQ+YHHxADcaTuXE6H2A ZNibk9nPjiDoaN5XKrVhTxzBANqbkruoVmW5+i4iAEtzm1E8kaYHjofide3R8IkViZua 3KNCMSpoXrF6qqaXDc5fcAIk8wArP2jrLCm2FGVE+uqk/a/JTmDxMFniW2fx6u9i6VSH XmZw== MIME-Version: 1.0 X-Received: by 10.50.82.68 with SMTP id g4mr1266917igy.26.1423179232723; Thu, 05 Feb 2015 15:33:52 -0800 (PST) Received: by 10.50.182.233 with HTTP; Thu, 5 Feb 2015 15:33:52 -0800 (PST) In-Reply-To: References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Date: Thu, 5 Feb 2015 15:33:52 -0800 Message-ID: Subject: Re: Better way to do conditional inclusion in make From: NGie Cooper To: Dimitry Andric Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 23:33:53 -0000 On Thu, Feb 5, 2015 at 1:32 PM, Dimitry Andric wrote: > On 05 Feb 2015, at 18:56, Warner Losh wrote: >> >> We know that MK_foo is always going to be yes or no >> for build options. We can leverage that fact, and the fact that bmake is so much better at variable >> expansion than fmake was (especially in the early days) to instead move to something like: >> >> FILES=list of unconditional files here ${FILES.yes} >> FILES.${MK_foo}+=foo bar biz >> FILES.${MK_baz}+=baz bing boo >> >> which eliminates a whole lot of needless .if / .endif lines, lots of extra blank lines, etc. >> >> Comments? > > One disadvantage is that you then cannot intersperse MK_foo files or > subdirs in between unconditional ones, and keep a certain order, e.g. > like: > > FILES+= a b c > .if ${MK_foo} != "no" > FILES+= d e f > .endif > FILES+= g h i > > Of course, this is only important for a few particular places, most of > the tree should not care too much about the order in which subdirs or > files are built. Depending on implicit ordering like that is brittle and I'd rather not enable this kind of pattern... From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 00:41:51 2015 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 87E09580 for ; Fri, 6 Feb 2015 00:41:51 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 583F07F3 for ; Fri, 6 Feb 2015 00:41:50 +0000 (UTC) Received: from Julian-MBP3.local (ppp121-45-238-204.lns20.per1.internode.on.net [121.45.238.204]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t160fjfX051238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 5 Feb 2015 16:41:47 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <54D40DC4.9070907@freebsd.org> Date: Fri, 06 Feb 2015 08:41:40 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Warner Losh , "freebsd-arch@freebsd.org" Subject: Re: Better way to do conditional inclusion in make References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> In-Reply-To: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 00:41:51 -0000 On 2/6/15 1:56 AM, Warner Losh wrote: > Greetings, > > I’ve started a pass through the tree to cleanup how we do conditional inclusion in our build system. > > https://reviews.freebsd.org/D1781 > > It moves away from > > .if ${MK_foo} != “no” > FILES+= files > .endif > > and > > SUBDIRS= … ${_foo} … > ... > .if ${MK_foo} != “no” > _foo+= foo > .endif > > and instead more directly assigns things. We know that MK_foo is always going to be yes or no > for build options. We can leverage that fact, and the fact that bmake is so much better at variable > expansion than fmake was (especially in the early days) to instead move to something like: > > FILES=list of unconditional files here ${FILES.yes} > FILES.${MK_foo}+=foo bar biz > FILES.${MK_baz}+=baz bing boo > > which eliminates a whole lot of needless .if / .endif lines, lots of extra blank lines, etc. > > Comments? how does it cope with the case where a single file is dependent on either of two options. (we have this in our tree.. not sure if it occurs in the FreeBSD tree.) file could occur in both lists or twice in one list.. > Warner > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > > From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 01:06:57 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AE87729; Fri, 6 Feb 2015 01:06:57 +0000 (UTC) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 416769A0; Fri, 6 Feb 2015 01:06:57 +0000 (UTC) Received: by mail-ig0-f174.google.com with SMTP id b16so3728879igk.1; Thu, 05 Feb 2015 17:06:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EHQOw8PKuiFacPho93HxW8ikWPn3TceUJYBD+92x2a4=; b=PEnIHeAhwWqmvDoJIzSNj8/itMJoSSoC2PLQ8be69YL5OOR3LLq1R0AQGwQag+BhhB +JQmUUSMVS+CM2fxlb4ijuBed+UAKkyp6dEqAfc7vXwymo8i8JwudYn+oaH03fjya4y1 MPYohbKSeuFjh6gLLkz1AiYYaF0llvXMUSH+s22vR1T1ye2A940z8tL15R61sCMl1nin aUYmDWHC0Pz5KMwJSUYzb4zydmqTppHgfype6Mg/aex8dJlu5h/CTp4ufp/6Q5zHaL/e JtbcGcYtQN79gCLtvuzqfyU8LTdJ8Mttpss1fZTtrZcrqQzyVPJExCB4qiqIttBcnYPg ghYw== MIME-Version: 1.0 X-Received: by 10.43.79.129 with SMTP id zq1mr10311920icb.28.1423184815836; Thu, 05 Feb 2015 17:06:55 -0800 (PST) Received: by 10.50.182.233 with HTTP; Thu, 5 Feb 2015 17:06:55 -0800 (PST) In-Reply-To: <54D40DC4.9070907@freebsd.org> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> Date: Thu, 5 Feb 2015 17:06:55 -0800 Message-ID: Subject: Re: Better way to do conditional inclusion in make From: NGie Cooper To: Julian Elischer Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 01:06:57 -0000 On Thu, Feb 5, 2015 at 4:41 PM, Julian Elischer wrote: ... > how does it cope with the case where a single file is dependent on either of > two options. > (we have this in our tree.. not sure if it occurs in the FreeBSD tree.) > file could occur in both lists or twice in one list.. This is a good, valid point. I think that Warner's proposal will fix the simple case (using one knob), but not the more complex case. What concerns me about the short description of the implementation, (and something that I'm going to add to the phabricator review) is that this will: 1. Break using FILESGROUPS 2. Requires creating snippets for dealing with magic in bsd.*.mk (I wouldn't want this magic going into the general purpose snippets because it would probably break backwards compatibility). Thanks! From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 01:34:15 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9A44CCCE; Fri, 6 Feb 2015 01:34:15 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0127.outbound.protection.outlook.com [65.55.169.127]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3024FC36; Fri, 6 Feb 2015 01:34:14 +0000 (UTC) Received: from BL2PR05CA0023.namprd05.prod.outlook.com (10.255.226.23) by BN1PR05MB439.namprd05.prod.outlook.com (10.141.58.22) with Microsoft SMTP Server (TLS) id 15.1.81.19; Fri, 6 Feb 2015 01:19:19 +0000 Received: from BN1BFFO11FD037.protection.gbl (2a01:111:f400:7c10::1:165) by BL2PR05CA0023.outlook.office365.com (2a01:111:e400:c04::23) with Microsoft SMTP Server (TLS) id 15.1.75.20 via Frontend Transport; Fri, 6 Feb 2015 01:19:19 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD037.mail.protection.outlook.com (10.58.144.100) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 01:19:18 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 5 Feb 2015 17:19:17 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t161JHW65761; Thu, 5 Feb 2015 17:19:17 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 1FFC0580A3; Thu, 5 Feb 2015 17:19:17 -0800 (PST) To: Dimitry Andric Subject: Re: Better way to do conditional inclusion in make In-Reply-To: References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> Comments: In-reply-to: Dimitry Andric message dated "Thu, 05 Feb 2015 22:32:06 +0100." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 5 Feb 2015 17:19:17 -0800 Message-ID: <16583.1423185557@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; bsdimp.com; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(24454002)(46102003)(92566002)(117636001)(105596002)(106466001)(110136001)(6806004)(33716001)(50466002)(77096005)(76176999)(87936001)(86362001)(57986006)(62966003)(77156002)(76506005)(558084003)(47776003)(19580395003)(19580405001)(2950100001)(48376002)(50226001)(50986999)(43043002)(42262002)(62816006); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR05MB439; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:nov; PTR:InfoDomainNonexistent; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:BN1PR05MB439; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:BN1PR05MB439; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:BN1PR05MB439; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 01:19:18.7787 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1PR05MB439 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 01:34:15 -0000 Dimitry Andric wrote: > Of course, this is only important for a few particular places, most of > the tree should not care too much about the order in which subdirs or > files are built. If the order of such a list matters you have a problem. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 01:38:04 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C005E04; Fri, 6 Feb 2015 01:38:04 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0134.outbound.protection.outlook.com [157.56.111.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A638EC51; Fri, 6 Feb 2015 01:38:03 +0000 (UTC) Received: from CO2PR05CA050.namprd05.prod.outlook.com (10.141.241.178) by DM2PR05MB446.namprd05.prod.outlook.com (10.141.104.142) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 01:22:44 +0000 Received: from BN1BFFO11FD034.protection.gbl (2a01:111:f400:7c10::1:184) by CO2PR05CA050.outlook.office365.com (2a01:111:e400:1429::50) with Microsoft SMTP Server (TLS) id 15.1.81.19 via Frontend Transport; Fri, 6 Feb 2015 01:22:43 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD034.mail.protection.outlook.com (10.58.144.97) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 01:22:43 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 5 Feb 2015 17:22:23 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t161MNW67527; Thu, 5 Feb 2015 17:22:23 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 25732580A3; Thu, 5 Feb 2015 17:22:23 -0800 (PST) To: Julian Elischer Subject: Re: Better way to do conditional inclusion in make In-Reply-To: <54D40DC4.9070907@freebsd.org> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> Comments: In-reply-to: Julian Elischer message dated "Fri, 06 Feb 2015 08:41:40 +0800." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 5 Feb 2015 17:22:23 -0800 Message-ID: <6819.1423185743@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; bsdimp.com; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(24454002)(19580405001)(19580395003)(33716001)(92566002)(48376002)(47776003)(76176999)(50986999)(87936001)(86362001)(50466002)(50226001)(57986006)(76506005)(105596002)(46102003)(77096005)(117636001)(110136001)(106466001)(6806004)(2950100001)(77156002)(62966003)(43043002)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB446; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:nov; PTR:InfoDomainNonexistent; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB446; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:DM2PR05MB446; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB446; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 01:22:43.0417 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR05MB446 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 01:38:04 -0000 Julian Elischer wrote: > how does it cope with the case where a single file is dependent on > either of two options. > (we have this in our tree.. not sure if it occurs in the FreeBSD tree.) > file could occur in both lists or twice in one list.. If you accept the premise that order of the list shouldn't matter, one can simply apply :O:u to suppress dups. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 01:39:21 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 011E2EB3; Fri, 6 Feb 2015 01:39:20 +0000 (UTC) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC38EC5A; Fri, 6 Feb 2015 01:39:20 +0000 (UTC) Received: by mail-ig0-f169.google.com with SMTP id hl2so10076570igb.0; Thu, 05 Feb 2015 17:39:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=no64lzN8SjqkjbgfM0zD9B2fN4wZWXBRtNlFfxVbYvc=; b=03GPjg2j1QmGG6ur0DdqJo8ICnMhkSfIDPdkVPPguNvKCL3PfxfDP/Iq0BYBYysCbD xlXq24RQb+rgF6PtHOhEA9HOXc2eZMpUVdpnoIrLUp70jQN7KSDVv+pdszXzrGZci25E EWCJ5C1Fe0wOBrEE3eUrZ+dDh27+kWRjuJCvPuYlR2pC+P6ka3RkEZtDNeVX//GJiD57 rYc7kfFf0bqroa2CiaS1BcG2sASEdgx5ICL1DxI/3a7iiBxttclpiOcdwDEJB5hSvbkn qcUcJeSQb4udShJ+3+g7TaceN2PZyMsv2YsRQ5RFucdoY846LlfwJgMIyw3lYB/NGVdx LVSg== MIME-Version: 1.0 X-Received: by 10.50.66.243 with SMTP id i19mr1550799igt.7.1423186760089; Thu, 05 Feb 2015 17:39:20 -0800 (PST) Received: by 10.50.182.233 with HTTP; Thu, 5 Feb 2015 17:39:20 -0800 (PST) In-Reply-To: <5180.1423185968@chaos> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> <5180.1423185968@chaos> Date: Thu, 5 Feb 2015 17:39:20 -0800 Message-ID: Subject: Re: Better way to do conditional inclusion in make From: NGie Cooper To: "Simon J. Gerraty" Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 01:39:21 -0000 On Thu, Feb 5, 2015 at 5:26 PM, Simon J. Gerraty wrote: > NGie Cooper wrote: >> > how does it cope with the case where a single file is dependent on either of >> > two options. >> > (we have this in our tree.. not sure if it occurs in the FreeBSD tree.) >> > file could occur in both lists or twice in one list.. >> >> This is a good, valid point. I think that Warner's proposal will fix >> the simple case (using one knob), but not the more complex case. > > FILES:= ${FILES:O:u} > > should cover that case. Yes, but not this: .if ${MK_BAR} != "no" && ${MK_FOO} != "no" FILES+= a_lot_of_bar_in_my_foo .endif >> What concerns me about the short description of the implementation, >> (and something that I'm going to add to the phabricator review) is >> that this will: >> >> 1. Break using FILESGROUPS > > Why? The same reason why bsd.progs.mk didn't work with bsd.prog.mk on FreeBSD out of the box originally -- defaults: 10 FILESGROUPS?= FILES 11 12 .for group in ${FILESGROUPS} 13 buildfiles: ${${group}} 14 .endfor 15 Warner's change (based on what I understand, again I haven't looked at the review yet...) would require setting FILESGROUPS explicitly. So if you had a Makefile snippet that defines the non-default FILESGROUPS already, it will break that usecase. >> 2. Requires creating snippets for dealing with magic in bsd.*.mk (I >> wouldn't want this magic going into the general purpose snippets >> because it would probably break backwards compatibility). > > Not necessarily eg. if you clean/simplify the list after building it. I'll delay my reply on this because my other replies might change my question. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 01:40:45 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E797BF64; Fri, 6 Feb 2015 01:40:44 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0111.outbound.protection.outlook.com [65.55.169.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6B882C69; Fri, 6 Feb 2015 01:40:43 +0000 (UTC) Received: from BL2PR05CA0016.namprd05.prod.outlook.com (10.255.226.16) by BN1PR05MB439.namprd05.prod.outlook.com (10.141.58.22) with Microsoft SMTP Server (TLS) id 15.1.81.19; Fri, 6 Feb 2015 01:26:10 +0000 Received: from BL2FFO11FD022.protection.gbl (2a01:111:f400:7c09::181) by BL2PR05CA0016.outlook.office365.com (2a01:111:e400:c04::16) with Microsoft SMTP Server (TLS) id 15.1.75.20 via Frontend Transport; Fri, 6 Feb 2015 01:26:10 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD022.mail.protection.outlook.com (10.173.161.101) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 01:26:10 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 5 Feb 2015 17:26:09 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t161Q8W69684; Thu, 5 Feb 2015 17:26:08 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 8B714580A3; Thu, 5 Feb 2015 17:26:08 -0800 (PST) To: NGie Cooper Subject: Re: Better way to do conditional inclusion in make In-Reply-To: References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> Comments: In-reply-to: NGie Cooper message dated "Thu, 05 Feb 2015 17:06:55 -0800." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 5 Feb 2015 17:26:08 -0800 Message-ID: <5180.1423185968@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; freebsd.org; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(51444003)(51704005)(24454002)(46102003)(92566002)(117636001)(105596002)(106466001)(110136001)(6806004)(33716001)(50466002)(77096005)(76176999)(87936001)(1411001)(86362001)(57986006)(62966003)(77156002)(76506005)(47776003)(19580395003)(19580405001)(2950100001)(561944003)(48376002)(50226001)(50986999)(43043002)(42262002)(62816006); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR05MB439; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:nov; PTR:InfoDomainNonexistent; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:BN1PR05MB439; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:BN1PR05MB439; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:BN1PR05MB439; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 01:26:10.3332 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1PR05MB439 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 01:40:45 -0000 NGie Cooper wrote: > > how does it cope with the case where a single file is dependent on either of > > two options. > > (we have this in our tree.. not sure if it occurs in the FreeBSD tree.) > > file could occur in both lists or twice in one list.. > > This is a good, valid point. I think that Warner's proposal will fix > the simple case (using one knob), but not the more complex case. FILES:= ${FILES:O:u} should cover that case. > What concerns me about the short description of the implementation, > (and something that I'm going to add to the phabricator review) is > that this will: > > 1. Break using FILESGROUPS Why? > 2. Requires creating snippets for dealing with magic in bsd.*.mk (I > wouldn't want this magic going into the general purpose snippets > because it would probably break backwards compatibility). Not necessarily eg. if you clean/simplify the list after building it. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 03:18:31 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 88323F31 for ; Fri, 6 Feb 2015 03:18:31 +0000 (UTC) Received: from mail-ig0-f182.google.com (mail-ig0-f182.google.com [209.85.213.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 506CF949 for ; Fri, 6 Feb 2015 03:18:30 +0000 (UTC) Received: by mail-ig0-f182.google.com with SMTP id h15so4789714igd.3 for ; Thu, 05 Feb 2015 19:18:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=/vVY4Q9hWKjFem1Cn6IK3DnEb2yLapOtGTAL/katKwE=; b=ZHZLEXHrxcAh1wsGsnl4fqwnnTyoCaKu+sQbrbQgUy9XH3lIHQgcsD5ztTUpMi/r2S bciJ10bxiKisR2vV8uF0mCSawUNXFli61xCtiN9kfFcg+3UX1B7jpd+TtCOFT7LQ8yrE sH1uJdvxD6Sq8anBeG4tHojqFhmp0tnHthEFbqsgEGmypBJD6obulAA1J5diJw1l/DvJ GIYL2z2kM8spYMDMauvT6EKhI9ugxRLhqMVPM7IpZU/I7IZn2cXuY6X14NQBsv9eMZw/ 7waxNuO2aRpVLXTCrGcY2qATJ+4Uy8GzjCKNeCUa35Ds9pQLEyHKuRjPUsL0hVNMxPMX m3BA== X-Gm-Message-State: ALoCoQkeQrKpENaa31RgpSjjNm5f+eFXovR94NmF10hsRUhxFoyOJfvjIS1dmUxTgKnq+88TlCZ1 X-Received: by 10.42.77.9 with SMTP id g9mr10855970ick.78.1423192709850; Thu, 05 Feb 2015 19:18:29 -0800 (PST) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id 9sm625960iom.10.2015.02.05.19.18.29 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Feb 2015 19:18:29 -0800 (PST) Sender: Warner Losh Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Better way to do conditional inclusion in make From: Warner Losh In-Reply-To: <54D40DC4.9070907@freebsd.org> Date: Thu, 5 Feb 2015 20:18:28 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <07D8BEB5-8142-431A-B823-4900D664CEE2@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> To: Julian Elischer X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 03:18:31 -0000 > On Feb 5, 2015, at 5:41 PM, Julian Elischer = wrote: >=20 > On 2/6/15 1:56 AM, Warner Losh wrote: >> Greetings, >>=20 >> I=E2=80=99ve started a pass through the tree to cleanup how we do = conditional inclusion in our build system. >>=20 >> https://reviews.freebsd.org/D1781 >>=20 >> It moves away from >>=20 >> .if ${MK_foo} !=3D =E2=80=9Cno=E2=80=9D >> FILES+=3D files >> .endif >>=20 >> and >>=20 >> SUBDIRS=3D =E2=80=A6 ${_foo} =E2=80=A6 >> ... >> .if ${MK_foo} !=3D =E2=80=9Cno=E2=80=9D >> _foo+=3D foo >> .endif >>=20 >> and instead more directly assigns things. We know that MK_foo is = always going to be yes or no >> for build options. We can leverage that fact, and the fact that bmake = is so much better at variable >> expansion than fmake was (especially in the early days) to instead = move to something like: >>=20 >> FILES=3Dlist of unconditional files here ${FILES.yes} >> FILES.${MK_foo}+=3Dfoo bar biz >> FILES.${MK_baz}+=3Dbaz bing boo >>=20 >> which eliminates a whole lot of needless .if / .endif lines, lots of = extra blank lines, etc. >>=20 >> Comments? >=20 > how does it cope with the case where a single file is dependent on = either of two options. > (we have this in our tree.. not sure if it occurs in the FreeBSD = tree.) > file could occur in both lists or twice in one list.. We have this in our tree a fair amount. power, for example, is enabled = with either MK_ACPI or MK_APM. When it is either/or, just list it for both. FILES.${MK_APCI}+=3D acpiconf powerd FILES.${MK_APM}+=3D powerd The FILES:=3D${FILES:O:u} at the end sorts everything and removes the = duplicates. Warner= From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 03:20:11 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AA6663AA for ; Fri, 6 Feb 2015 03:20:11 +0000 (UTC) Received: from mail-ig0-f179.google.com (mail-ig0-f179.google.com [209.85.213.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7463A95A for ; Fri, 6 Feb 2015 03:20:11 +0000 (UTC) Received: by mail-ig0-f179.google.com with SMTP id l13so4789516iga.0 for ; Thu, 05 Feb 2015 19:20:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=nmIvXqfTEJCtLRXUneqiTEaEpBvRlKhg2Wxd56Mfd6o=; b=OGFf1ZCvk0N0skLli5DmXCHq0DlMJK08hP8QEhuhwaQAu9QYKUhN36Nm25muzwxhTT V55gJJlcUcZFFulwCV96oP2CTnH4BJbcPgBPDe8djPwu0RsIukNZvl5YTP9fUNAoEEl7 OinrWvJpf43ciWDbOUx8ofCSgK0+lM5Yl13tpf4MOnOOO2Dbfr26c59S0g1i/ZQevpdY U0MH3ph9BYTckqcTc5OdQ5w6D27Q5WVWlmjpln3Rdg3mPJhcF9RapSbmrTeqX7c3/e7J 9WzTNTdCrB7vuN0AuN+LrqpiODMSjCEvL91/iaEN4emWg/ljPvdiBc/apXI5cN7X8J4b MKNA== X-Gm-Message-State: ALoCoQnLrz/0xR8O6VjgyrHEOaZ3GdTRdyq7ELcMdGyi5ey6PEHyRhYISpcx/Y7pC53SNz4i+u8I X-Received: by 10.51.17.1 with SMTP id ga1mr128065igd.33.1423192810322; Thu, 05 Feb 2015 19:20:10 -0800 (PST) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id 3sm629314iof.8.2015.02.05.19.20.09 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Feb 2015 19:20:09 -0800 (PST) Sender: Warner Losh Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Better way to do conditional inclusion in make From: Warner Losh In-Reply-To: Date: Thu, 5 Feb 2015 20:20:09 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <105BE327-D1C3-492D-866C-F3F3FB48D991@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> To: NGie Cooper X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 03:20:11 -0000 > On Feb 5, 2015, at 6:06 PM, NGie Cooper wrote: >=20 > On Thu, Feb 5, 2015 at 4:41 PM, Julian Elischer = wrote: >=20 > ... >=20 >> how does it cope with the case where a single file is dependent on = either of >> two options. >> (we have this in our tree.. not sure if it occurs in the FreeBSD = tree.) >> file could occur in both lists or twice in one list.. >=20 > This is a good, valid point. I think that Warner's proposal will fix > the simple case (using one knob), but not the more complex case. This isn=E2=80=99t that valid a point. It=E2=80=99s a trivial annoyance. = When this construct can=E2=80=99t cope (like for things that deal with arch dependent = stuff), you fall back to old-school ifs. > What concerns me about the short description of the implementation, > (and something that I'm going to add to the phabricator review) is > that this will: >=20 > 1. Break using FILESGROUPS How? > 2. Requires creating snippets for dealing with magic in bsd.*.mk (I > wouldn't want this magic going into the general purpose snippets > because it would probably break backwards compatibility). Explain. Warner= From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 03:25:39 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F647F8C for ; Fri, 6 Feb 2015 03:25:39 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0133.outbound.protection.outlook.com [157.56.111.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA1B1A09 for ; Fri, 6 Feb 2015 03:25:38 +0000 (UTC) Received: from CY1PR0501MB1548.namprd05.prod.outlook.com (25.161.161.146) by CY1PR0501MB1385.namprd05.prod.outlook.com (25.160.148.139) with Microsoft SMTP Server (TLS) id 15.1.81.19; Fri, 6 Feb 2015 01:51:23 +0000 Received: from BLUPR0501MB1060.namprd05.prod.outlook.com (25.160.36.150) by CY1PR0501MB1548.namprd05.prod.outlook.com (25.161.161.146) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 01:51:21 +0000 Received: from BLUPR0501MB1060.namprd05.prod.outlook.com ([25.160.36.150]) by BLUPR0501MB1060.namprd05.prod.outlook.com ([25.160.36.150]) with mapi id 15.01.0075.002; Fri, 6 Feb 2015 01:51:21 +0000 From: Anuranjan Shukla To: "freebsd-arch@freebsd.org" Subject: Buggy sbspace() on 64bit builds? Thread-Topic: Buggy sbspace() on 64bit builds? Thread-Index: AQHQQa9oYD6vE33IJEufQkyrNnicWQ== Date: Fri, 6 Feb 2015 01:51:21 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.7.141117 x-originating-ip: [66.129.241.11] authentication-results: freebsd.org; dkim=none (message not signed) header.d=none; x-microsoft-antispam: BCL:0;PCL:0;RULEID:;SRVR:CY1PR0501MB1548;UriScan:; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:; SRVR:CY1PR0501MB1548; x-forefront-prvs: 047999FF16 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(92566002)(46102003)(99286002)(122556002)(36756003)(86362001)(40100003)(2351001)(50986999)(83506001)(229853001)(110136001)(106116001)(66066001)(102836002)(2900100001)(2656002)(87936001)(54356999)(450100001)(62966003)(77156002); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0501MB1548; H:BLUPR0501MB1060.namprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-ID: <1B2330DA3FE23348938BF9A6079DC9C8@namprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Feb 2015 01:51:21.5631 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0501MB1548 X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:CY1PR0501MB1385; X-OriginatorOrg: juniper.net Cc: Simon Gerraty X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 03:25:39 -0000 Hi, The way sbspace() is done today, it stores the result of subtraction of socket buffer variables (u_int) in longs, and returns a long. If one of the subtractions results in a -ve result (the other being positive), it's seen as a large +ve and sbspace() ends up returning the wrong value. I'm not sure if this is enough of a corner case for consumers at large to experience it, but at Juniper some of our implementation uses sbspace directly and trips up on this for amd64 builds. Any thoughts on what a fix should be for this? Thanks Anu --------------------------- long sbspace(struct sockbuf *sb) { long bleft; long mleft; if (sb->sb_flags & SB_STOP) return(0); bleft =3D sb->sb_hiwat - sb->sb_cc; mleft =3D sb->sb_mbmax - sb->sb_mbcnt; return((bleft < mleft) ? bleft : mleft); } From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 03:27:07 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C9630375 for ; Fri, 6 Feb 2015 03:27:07 +0000 (UTC) Received: from mail-ig0-f179.google.com (mail-ig0-f179.google.com [209.85.213.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91D1BA16 for ; Fri, 6 Feb 2015 03:27:07 +0000 (UTC) Received: by mail-ig0-f179.google.com with SMTP id l13so4840731iga.0 for ; Thu, 05 Feb 2015 19:27:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=fCwbwvlHXb4ahwpY9J38yD1EycYO0ixyRuOQmJSIYLc=; b=Y0Gj5R/bPUJnHYyYnvRNL9t58VB2/Kl7lgxbqwd2yg96HvRJjRDrg601MUUDNX0d5R 97SKcyb05Efnvpy/sdYKZs95tGOH0sGT6poxZ8FHI1voeSUs7WdrBoDE1VIcv9+AnPkd aVtiZ0C89G+WXQ3kpWLrgxylxXM38S0bo5YOzEGs/QkJTWTO888OW4/xY+764UjKHerV 16jLaJB+euF0iQB7Z2RFBqKGd0mSCqYD6oYEhYCj/u9U7L0nMmnnZSJpYSFZYOGY1k/+ QpVutFt1Vqcns1v/8tkHwHq8VC5xCr62czRgvVbr+zqPRuvCczT4HFCbQLIinZZHQbSO su8Q== X-Gm-Message-State: ALoCoQkfas5uoP4VkJf/Sm9+cZSo9mKkOt/BNvnoRuLSP8t+TBJ7RD7YclRqRqDO9RSFN9JTgY0F X-Received: by 10.42.64.193 with SMTP id h1mr11097453ici.32.1423193226560; Thu, 05 Feb 2015 19:27:06 -0800 (PST) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id m5sm603680ige.5.2015.02.05.19.27.05 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Feb 2015 19:27:06 -0800 (PST) Sender: Warner Losh Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Better way to do conditional inclusion in make From: Warner Losh In-Reply-To: <6819.1423185743@chaos> Date: Thu, 5 Feb 2015 20:27:05 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <6B52B39B-EC45-4723-99AE-71EFC917B9A0@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> <6819.1423185743@chaos> To: "Simon J. Gerraty" X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 03:27:08 -0000 > On Feb 5, 2015, at 6:22 PM, Simon J. Gerraty wrote: >=20 > Julian Elischer wrote: >> how does it cope with the case where a single file is dependent on >> either of two options. >> (we have this in our tree.. not sure if it occurs in the FreeBSD = tree.) >> file could occur in both lists or twice in one list.. >=20 > If you accept the premise that order of the list shouldn't matter, > one can simply apply :O:u to suppress dups. The order of the list doesn=E2=80=99t matter. If it does, that=E2=80=99s = a bug in the list and/or the dependencies and that should be fixed. I=E2=80=99m not aware of any = place in the tree where this matters that doesn=E2=80=99t have some secondary = ordering mechanism (see the libraries for one such thing). The phabricator review already has this construct to sort and suppress = dups. Where this conditional breaks down is when you have something predicated on MACHINE_ARCH and MK_foo. Makes it very hard to have FILES.${MACHINE} added as well. In user land this isn=E2=80=99t so common (it happens, = but not so much and most of it would disappear if we had some kind of always no flag for options that flat out can=E2=80=99t work on a given arch)[*]. In the = kernel though it happens a lot with a lot of annoying duplication. Warner [*] Consider having something like .if ${MACHINE} !=3D =E2=80=9Ci386=E2=80=9D && ${MACHINE} !=3D = =E2=80=9Camd64=E2=80=9D __ALWAYS_NO+=3D GERBILS PUPPIES WHALES .endif then MK_GERBILS would always be no (you aren=E2=80=99t allowed to set = another value), and you don=E2=80=99t have to check for architectures where it is used = to exclude it. I haven=E2=80=99t worked out all the details of this yet, which is why I = haven=E2=80=99t posted it for discussion.= From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 03:30:52 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04FD6BFF for ; Fri, 6 Feb 2015 03:30:52 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0621A3D for ; Fri, 6 Feb 2015 03:30:51 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id h15so4745922igd.4 for ; Thu, 05 Feb 2015 19:30:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=g1NUNFmZHu2qz+JDvn8EdUXE3slDrBH4J25US3QF984=; b=BD+3wtyuMizPd3nfpAxZqu4q2NGQZi5KbjAjKtBhX2t7C3/GCE6sgk1BZEsyVUuNzc x5yz1H4tyyaV1Ad50LHU9o3nMP6ggd5wD5xl07GBp4+hp5/ThzkpT/DXWPrS7ZLvv7Za 5yUbPF1Rb6mf+4P1PoqN+RU+fs94uSByqSXnslOvGiIiUXyMcawsHBpV1y1ULomewDFh ihiR31FW3f0yPMx5nOzdM3lJdADiAqzGlodB2h3NgPk51YC8KgIoTlcrMgT8dNjVhWb7 txy2KBBCO+NuO5bo63Fcmo5Dkb2d5N2riIuejDSI/xp8sGBXrTqm6zQNOBwaELArhNfK 78Ow== X-Gm-Message-State: ALoCoQl5AvK7gLP08TTOYdRv3UvsCV0gzmKc+Xvfsu7Mldwo9x4AUWc5JdO5SFmDtalYJGZYDWA0 X-Received: by 10.42.145.5 with SMTP id d5mr10858727icv.8.1423193445408; Thu, 05 Feb 2015 19:30:45 -0800 (PST) Received: from netflix-mac-wired.bsdimp.com ([50.253.99.174]) by mx.google.com with ESMTPSA id c6sm620195ioe.37.2015.02.05.19.30.44 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 05 Feb 2015 19:30:44 -0800 (PST) Sender: Warner Losh Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Better way to do conditional inclusion in make From: Warner Losh In-Reply-To: Date: Thu, 5 Feb 2015 20:30:44 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <4B573F3A-B1BA-4D6E-B2DB-5EA56ECF0321@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> <5180.1423185968@chaos> To: NGie Cooper X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-arch@freebsd.org" , "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 03:30:52 -0000 > On Feb 5, 2015, at 6:39 PM, NGie Cooper wrote: >=20 > On Thu, Feb 5, 2015 at 5:26 PM, Simon J. Gerraty = wrote: >> NGie Cooper wrote: >>>> how does it cope with the case where a single file is dependent on = either of >>>> two options. >>>> (we have this in our tree.. not sure if it occurs in the FreeBSD = tree.) >>>> file could occur in both lists or twice in one list.. >>>=20 >>> This is a good, valid point. I think that Warner's proposal will fix >>> the simple case (using one knob), but not the more complex case. >>=20 >> FILES:=3D ${FILES:O:u} >>=20 >> should cover that case. >=20 > Yes, but not this: >=20 > .if ${MK_BAR} !=3D "no" && ${MK_FOO} !=3D "no" > FILES+=3D a_lot_of_bar_in_my_foo > .endif There=E2=80=99s very few cases of this in the tree, so who cares :) You = just add the old school .if. >>> What concerns me about the short description of the implementation, >>> (and something that I'm going to add to the phabricator review) is >>> that this will: >>>=20 >>> 1. Break using FILESGROUPS >>=20 >> Why? >=20 > The same reason why bsd.progs.mk didn't work with bsd.prog.mk on > FreeBSD out of the box originally -- defaults: bsd.progs.mk is evil and should die :) I tried to leave this line out, = but couldn=E2=80=99t :) > 10 FILESGROUPS?=3D FILES > 11 > 12 .for group in ${FILESGROUPS} > 13 buildfiles: ${${group}} > 14 .endfor > 15 >=20 > Warner's change (based on what I understand, again I haven't looked at > the review yet=E2=80=A6) You should, it would answer why this isn=E2=80=99t an issue. > would require setting FILESGROUPS explicitly. So if > you had a Makefile snippet that defines the non-default FILESGROUPS > already, it will break that usecase. Why? I don=E2=80=99t set why it matters at all. This wouldn=E2=80=99t = affect the FILES variable expansion in line 13 at all. How would it break? >>> 2. Requires creating snippets for dealing with magic in bsd.*.mk (I >>> wouldn't want this magic going into the general purpose snippets >>> because it would probably break backwards compatibility). >>=20 >> Not necessarily eg. if you clean/simplify the list after building it. >=20 > I'll delay my reply on this because my other replies might change my = question. OK. Warner From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 06:10:22 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 384E3506; Fri, 6 Feb 2015 06:10:22 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0130.outbound.protection.outlook.com [157.56.110.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92C27C06; Fri, 6 Feb 2015 06:10:20 +0000 (UTC) Received: from CO2PR05CA047.namprd05.prod.outlook.com (10.141.241.175) by DM2PR05MB448.namprd05.prod.outlook.com (10.141.104.152) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 06:10:12 +0000 Received: from BN1BFFO11FD027.protection.gbl (2a01:111:f400:7c10::1:187) by CO2PR05CA047.outlook.office365.com (2a01:111:e400:1429::47) with Microsoft SMTP Server (TLS) id 15.1.81.19 via Frontend Transport; Fri, 6 Feb 2015 06:10:12 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD027.mail.protection.outlook.com (10.58.144.90) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 06:10:11 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 5 Feb 2015 22:10:10 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t166AAW98868; Thu, 5 Feb 2015 22:10:10 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id CB73C580A3; Thu, 5 Feb 2015 22:10:09 -0800 (PST) To: NGie Cooper Subject: Re: Better way to do conditional inclusion in make In-Reply-To: References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> <5180.1423185968@chaos> Comments: In-reply-to: NGie Cooper message dated "Thu, 05 Feb 2015 17:39:20 -0800." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 5 Feb 2015 22:10:09 -0800 Message-ID: <11865.1423203009@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; freebsd.org; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(24454002)(51704005)(50466002)(48376002)(62966003)(46102003)(93886004)(50986999)(19580395003)(117636001)(87936001)(19580405001)(86362001)(47776003)(33716001)(2950100001)(6806004)(1411001)(77156002)(106466001)(105596002)(110136001)(77096005)(50226001)(57986006)(76506005)(92566002)(76176999)(43043002)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB448; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:sfv; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB448; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:DM2PR05MB448; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB448; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 06:10:11.5582 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR05MB448 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 06:10:22 -0000 NGie Cooper wrote: > .if ${MK_BAR} != "no" && ${MK_FOO} != "no" > FILES+= a_lot_of_bar_in_my_foo > .endif > Actually the same idea can handle that: FILES+= ${FILES.yesyes} FILES.${MK_BAR}${MK_FOO}+= a_lot_of_bar_in_my_foo extend as far as you like > >> What concerns me about the short description of the implementation, > >> (and something that I'm going to add to the phabricator review) is > >> that this will: > >> > >> 1. Break using FILESGROUPS > > > > Why? > > The same reason why bsd.progs.mk didn't work with bsd.prog.mk on > FreeBSD out of the box originally -- defaults: > > 10 FILESGROUPS?= FILES > 11 > 12 .for group in ${FILESGROUPS} > 13 buildfiles: ${${group}} > 14 .endfor > 15 If you do FILES:= ${FILES} prior to the above and even if you don't I don't think the behavior would change - at least for the snippet quoted. ${${group}} would still expand to the same thing, and dependency rules don't care if something is listed more than once FILES:= ${FILES} would probably be a good idea anyway though. > Warner's change (based on what I understand, again I haven't looked at > the review yet...) would require setting FILESGROUPS explicitly. So if > you had a Makefile snippet that defines the non-default FILESGROUPS > already, it will break that usecase. I don't think Warners change - per the brief description makes no difference to FILESGROUPS?= FILES From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 06:37:10 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 57417C47; Fri, 6 Feb 2015 06:37:10 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0112.outbound.protection.outlook.com [157.56.111.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B1A45DF1; Fri, 6 Feb 2015 06:37:09 +0000 (UTC) Received: from BLUPR05CA0058.namprd05.prod.outlook.com (10.141.20.28) by DM2PR05MB448.namprd05.prod.outlook.com (10.141.104.152) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 06:37:01 +0000 Received: from BL2FFO11FD035.protection.gbl (2a01:111:f400:7c09::165) by BLUPR05CA0058.outlook.office365.com (2a01:111:e400:855::28) with Microsoft SMTP Server (TLS) id 15.1.75.20 via Frontend Transport; Fri, 6 Feb 2015 06:37:00 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD035.mail.protection.outlook.com (10.173.161.131) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 06:37:00 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 5 Feb 2015 22:36:59 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t166axW16027; Thu, 5 Feb 2015 22:36:59 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id CB33B580A3; Thu, 5 Feb 2015 22:36:58 -0800 (PST) To: Warner Losh Subject: Re: Better way to do conditional inclusion in make In-Reply-To: <6B52B39B-EC45-4723-99AE-71EFC917B9A0@bsdimp.com> References: <39C20BA1-E6B1-4DAE-95BB-8011A0A64D54@bsdimp.com> <54D40DC4.9070907@freebsd.org> <6819.1423185743@chaos> <6B52B39B-EC45-4723-99AE-71EFC917B9A0@bsdimp.com> Comments: In-reply-to: Warner Losh message dated "Thu, 05 Feb 2015 20:27:05 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Thu, 5 Feb 2015 22:36:58 -0800 Message-ID: <28404.1423204618@chaos> X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; bsdimp.com; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(24454002)(50466002)(62966003)(46102003)(93886004)(19580395003)(50986999)(117636001)(87936001)(19580405001)(86362001)(47776003)(33716001)(2950100001)(6806004)(23676002)(77156002)(106466001)(105596002)(110136001)(77096005)(50226001)(57986006)(76506005)(92566002)(76176999)(43043002)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB448; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:sfv; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB448; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:DM2PR05MB448; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB448; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 06:37:00.5416 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR05MB448 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 06:37:10 -0000 Warner Losh wrote: > [*] Consider having something like > .if ${MACHINE} !=3D =E2=80=9Ci386=E2=80=9D && ${MACHINE} !=3D =E2=80=9Cam= d64=E2=80=9D FWIW I started encouraging folk here to use the construct: .if ${MACHINE:Ni386:Namd64} =3D=3D "" # either .if ${MACHINE:Ni386:Namd64} !=3D "" # neither as it more easily extends to a long list while remaining readable From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 07:36:49 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 59F0B5F0 for ; Fri, 6 Feb 2015 07:36:49 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 1BA48402 for ; Fri, 6 Feb 2015 07:36:49 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.60.3]) by phk.freebsd.dk (Postfix) with ESMTP id 3F5623B8A2; Fri, 6 Feb 2015 07:36:42 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id t167afDF037283; Fri, 6 Feb 2015 07:36:41 GMT (envelope-from phk@phk.freebsd.dk) To: Anuranjan Shukla Subject: Re: Buggy sbspace() on 64bit builds? In-reply-to: From: "Poul-Henning Kamp" References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <37281.1423208201.1@critter.freebsd.dk> Date: Fri, 06 Feb 2015 07:36:41 +0000 Message-ID: <37282.1423208201@critter.freebsd.dk> Cc: Simon Gerraty , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 07:36:49 -0000 -------- In message , Anuranjan Shukla writes: >Hi, >The way sbspace() is done today, it stores the result of subtraction of >socket buffer variables (u_int) in longs, and returns a long. [...] >Any thoughts on what a fix should be for this? size_t or intptr_t ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 08:56:25 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 46A20A0E for ; Fri, 6 Feb 2015 08:56:25 +0000 (UTC) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id E426BDAC for ; Fri, 6 Feb 2015 08:56:24 +0000 (UTC) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 7EB47D66505; Fri, 6 Feb 2015 19:56:15 +1100 (AEDT) Date: Fri, 6 Feb 2015 19:56:15 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Anuranjan Shukla Subject: Re: Buggy sbspace() on 64bit builds? In-Reply-To: Message-ID: <20150206183036.S1246@besplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=Za4kaKlA c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=CIKehoALdAUTTtdk9aIA:9 a=CjuIK1q_8ugA:10 Cc: Simon Gerraty , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 08:56:25 -0000 On Fri, 6 Feb 2015, Anuranjan Shukla wrote: > The way sbspace() is done today, it stores the result of subtraction of > socket buffer variables (u_int) in longs, and returns a long. If one of > the subtractions results in a -ve result (the other being positive), it's > seen as a large +ve and sbspace() ends up returning the wrong value. Old versions used bogus casts to work in practice. > I'm not sure if this is enough of a corner case for consumers at large > to experience it, but at Juniper some of our implementation uses sbspace > directly and trips up on this for amd64 builds. Any thoughts on what a fix > should be for this? > > --------------------------- > long > sbspace(struct sockbuf *sb) > { > long bleft; > long mleft; > > if (sb->sb_flags & SB_STOP) > return(0); > bleft = sb->sb_hiwat - sb->sb_cc; This seems to be much more than a corner case. It is normal for high water marks to be exceeded (perhaps not so normal for sockets). Whenever the high water mark is exceeded, the subtraction overflows to a large u_int value. Then on 32-bit arches, assignment to long overflows again but gives the correct value, but on 64-bit arches it doesn't overflow and the result is a large positive long value. > mleft = sb->sb_mbmax - sb->sb_mbcnt; > return((bleft < mleft) ? bleft : mleft); > > } This isn't quite the same as in -current from 2 months ago. -current also has an ifdef and different style bugs. The working version in FreeBSD-5 is: X /* X * How much space is there in a socket buffer (so->so_snd or so->so_rcv)? X * This is problematical if the fields are unsigned, as the space might X * still be negative (cc > hiwat or mbcnt > mbmax). Should detect X * overflow and return 0. Should use "lmin" but it doesn't exist now. X */ X #define sbspace(sb) \ X ((long) imin((int)((sb)->sb_hiwat - (sb)->sb_cc), \ X (int)((sb)->sb_mbmax - (sb)->sb_mbcnt))) -current has the same comment with the last 2 sentences removed. These sentences were a poor description of the hackish fix. They were removed together with changing the hackish fix to a bug. The warning about the bug remains. All of the sb_ fields in the above are still u_int. This implements the intermediate C programmer mistake of using unsigned types for values that are known to be nonnegative. This gives overflow and sign extension bugs here (strictly, non-overflow that is harmful, and non-sign extension that is harmful). Plain int should be enough for holding nonnegative values for anyone. It wasn't with 16-bit ints, but FreeBSD and possibly BSD never supported any system with 16-bit ints. 32-bit ints are enough here. The inner bogus cast compensate for the type botch by converting to int. It has to know that the field types are at most unsigned with the same size, and not get sign extension bugs as in the current version by converting to a larger size. This depends on no overflow in the expression occuring when everything is done with plain ints. This is satisified since 32-bit ints are enough for anyone. Buffer bloat starts with buffers much smaller than 2G. Then imin() is used to compare the values. imin() is hard to use since it only works for args that are either int or representable as int, but here they are known to be _mis_representable as int if we didn't already cast them; imin() would do the same casts automatically, and this is what is wanted here. We do the conversions explicitly because working around the type errors is hard enough to understand even when it is explicit. Finally, we cast to long because that is what sbspace() always returned. This was only needed to support 16-bit ints. Old BSD code used longs excessively, and this has not been fixed in many internal APIs. Old code using corrected API: X #define sbspace(sb) \ X lmin((long)(sb)->sb_hiwat - (sb)->sb_cc, \ X (long)(sb)->sb_mbmax - (sb)->sb_mbcnt) Other bugs in the old code: - the comment about lmin() not existing has been false for 20+ years - using lmin() makes only a cosmetic difference. Delicate casts are still needed. Implicit ones would work, but I wrote one explicit one. On 64-bit arches, this just works, but on 32-bit arches we start with u_int and truncate to longs; this depends on the u_int values being much smaller than LONG_MAX, just like they are much smaller than INT_MAX. - overflow is technically impossible, so the comment shouldn't mention it - after detecting the technically impossible overflow, there is no need to return 0. sbspace() returns a count, and a negative count works just as well as 0, sincd the API is not broken by using unsigned types. In the current version, changing the local variables from long to int would restore the delicate conversions, slightly more obfuscated by using redundant explicit assignments instead of redundant explicit casts. Or more clearly, cast all the field values to int. There is no point in keeping the variables long and casting to long instead -- on 32-bit machines, this is equally delicate. But keep the long return for compatibility. Or using only unsigned types and overflow checking, write more complicated code, like people had to do for counting before negative numbers were invented: after also fixing some style bugs: long sbspace(struct sockbuf *sb) { u_int bleft, mleft; if (sb->sb_flags & SB_STOP) return (0); bleft = sb->sb_hiwat < sb->sb_cc ? 0 : sb->sb_hiwat - sb->sb_cc; mleft = sb->sb_mbmax < sb->sb_mbcnt ? 9 : sb->sb_mbmax - sb->sb_mbcnt; return (min(bleft, mleft)); } This still needs range analysis to ensure that the u_int returned by min() is representable as a long. This follows from everything being much smaller than INT32_MAX, so we should have used int all along. Recent bugs from overflow in 'int ticks' were not quite the opposite of this. 'ticks' really does need to be unsigned, but some differences of it probably need be signed. It is signed because that is what it was originally and that is probably still needed to avoid sign extension bugs for differences of it, and its overflow used to be benign. Now -fwrapv makes its overflow benign again and hides overflow bugs elswhere. I used to think that overflow in conversions gives undefined behaviour. Actually, it gives defined behaviour with an implementation-defined result. The bogus casts in the above depend on the result being reasonable for conversion from a large unsigned value to a signed integer (int or long) of the same size. A flag like -fwrapv might be needed to stop the compiler defining its result as weird. In the 1's complement case, the reasonable result for a difference of -1U is -0 = 0, but we would prefer a result of -1. This is not a problem since both -0 and -1 mean no space. Bruce From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 16:27:16 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BE77E0 for ; Fri, 6 Feb 2015 16:27:16 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0140.outbound.protection.outlook.com [157.56.111.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D01D994F for ; Fri, 6 Feb 2015 16:27:15 +0000 (UTC) Received: from CO2PR05CA019.namprd05.prod.outlook.com (10.141.241.147) by DM2PR05MB448.namprd05.prod.outlook.com (10.141.104.152) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 16:27:07 +0000 Received: from BN1AFFO11OLC001.protection.gbl (2a01:111:f400:7c10::135) by CO2PR05CA019.outlook.office365.com (2a01:111:e400:1429::19) with Microsoft SMTP Server (TLS) id 15.1.81.19 via Frontend Transport; Fri, 6 Feb 2015 16:27:07 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11OLC001.mail.protection.outlook.com (10.58.53.72) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 16:27:06 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 6 Feb 2015 08:27:05 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t16GR5W98624; Fri, 6 Feb 2015 08:27:05 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id D68F1580A3; Fri, 6 Feb 2015 08:27:04 -0800 (PST) To: Poul-Henning Kamp Subject: Re: Buggy sbspace() on 64bit builds? In-Reply-To: <37282.1423208201@critter.freebsd.dk> References: <37282.1423208201@critter.freebsd.dk> Comments: In-reply-to: Poul-Henning Kamp message dated "Fri, 06 Feb 2015 07:36:41 +0000." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Fri, 6 Feb 2015 08:27:04 -0800 Message-ID: <2705.1423240024@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(51704005)(24454002)(106466001)(77096005)(110136001)(77156002)(57986006)(50226001)(92566002)(76506005)(76176999)(50466002)(46102003)(48376002)(62966003)(33716001)(2950100001)(6806004)(47776003)(19580395003)(50986999)(86362001)(19580405001)(117636001)(87936001)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB448; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:None; MLV:sfv; LANG:en; Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB448; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:DM2PR05MB448; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB448; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 16:27:06.7850 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR05MB448 Cc: "freebsd-arch@freebsd.org" , Anuranjan Shukla , sjg@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 16:27:16 -0000 Poul-Henning Kamp wrote: > >Any thoughts on what a fix should be for this? > > size_t or intptr_t ? I don't think so, the bug lies in the fact that on 64bit long is much bigger than the uint's that are being manipulated. Using a result object the same size works. Here's a simple demo of the problem: #include #include #include int main(int argc, char *argv[]) { uint a, b; long r1; int r2; a = 1; b = 2; r1 = a - b; r2 = a - b; printf("r1=%ld\nr2=%d\n", r1, r2); exit(0); } 32bit version outputs: r1=-1 r2=-1 64bit version outputs: r1=4294967295 r2=-1 the r1 value is obviously not what is expected by the caller. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 16:35:12 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17D3435F for ; Fri, 6 Feb 2015 16:35:12 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0137.outbound.protection.outlook.com [207.46.100.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5F3EA17 for ; Fri, 6 Feb 2015 16:35:10 +0000 (UTC) Received: from BY2PR05CA040.namprd05.prod.outlook.com (10.141.250.30) by DM2PR05MB447.namprd05.prod.outlook.com (10.141.104.150) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 16:35:02 +0000 Received: from BY2FFO11FD017.protection.gbl (2a01:111:f400:7c0c::142) by BY2PR05CA040.outlook.office365.com (2a01:111:e400:2c5f::30) with Microsoft SMTP Server (TLS) id 15.1.81.19 via Frontend Transport; Fri, 6 Feb 2015 16:35:01 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD017.mail.protection.outlook.com (10.1.14.105) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Fri, 6 Feb 2015 16:35:01 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 6 Feb 2015 08:35:01 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t16GYxW01912; Fri, 6 Feb 2015 08:35:00 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 52B21580A3; Fri, 6 Feb 2015 08:34:59 -0800 (PST) To: Bruce Evans Subject: Re: Buggy sbspace() on 64bit builds? In-Reply-To: <20150206183036.S1246@besplex.bde.org> References: <20150206183036.S1246@besplex.bde.org> Comments: In-reply-to: Bruce Evans message dated "Fri, 06 Feb 2015 19:56:15 +1100." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Fri, 6 Feb 2015 08:34:59 -0800 Message-ID: <25403.1423240499@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; freebsd.org; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(51914003)(24454002)(76506005)(50466002)(57986006)(50986999)(117636001)(46102003)(76176999)(92566002)(6806004)(19580395003)(19580405001)(50226001)(86362001)(47776003)(2950100001)(110136001)(87936001)(105596002)(48376002)(77096005)(77156002)(33716001)(62966003)(106466001)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB447; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:sfv; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB447; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:DM2PR05MB447; X-Forefront-PRVS: 047999FF16 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB447; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2015 16:35:01.8349 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR05MB447 Cc: Anuranjan Shukla , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 16:35:12 -0000 Bruce Evans wrote: > In the current version, changing the local variables from long to int > would restore the delicate conversions, slightly more obfuscated by > using redundant explicit assignments instead of redundant explicit casts. And *this* is why I asked Anu to send the question - rather than just change it to int ;-) Using int addresses the immediate issue, but the bug is worth discussion and perhaps broader consideration. Thanks for the analysis. --sjg From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 18:51:13 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 055DB5B3 for ; Fri, 6 Feb 2015 18:51:13 +0000 (UTC) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id BBECCC88 for ; Fri, 6 Feb 2015 18:51:12 +0000 (UTC) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 166D6D6272D; Sat, 7 Feb 2015 05:51:06 +1100 (AEDT) Date: Sat, 7 Feb 2015 05:51:06 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: "Simon J. Gerraty" Subject: Re: Buggy sbspace() on 64bit builds? In-Reply-To: <2705.1423240024@chaos> Message-ID: <20150207053557.L3468@besplex.bde.org> References: <37282.1423208201@critter.freebsd.dk> <2705.1423240024@chaos> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=A5NVYcmG c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=oikiVm-hFv82Gm5nFPsA:9 a=CjuIK1q_8ugA:10 Cc: Poul-Henning Kamp , Anuranjan Shukla , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 18:51:13 -0000 On Fri, 6 Feb 2015, Simon J. Gerraty wrote: > Poul-Henning Kamp wrote: >>> Any thoughts on what a fix should be for this? >> >> size_t or intptr_t ? > > I don't think so, the bug lies in the fact that on 64bit long is much > bigger than the uint's that are being manipulated. > Using a result object the same size works. Yes, using typedefed types just complicates the analysis, and size_t is unsigned so using it tends to give the same (un)sign extension bugs as using u_int. Most variables should be plain int. It is then implicit that the values are not too large positive or negative for expressions involving them to overflow. The subtraction here is especially easy to understand. Its terms must be nonnegative, so overflow can't happen. Its terms should also not be very large, but that is not needed here. If variables have an unsigned type (except possibly u_char or u_short), then everywhere that they are used the reader has to worry about them actually needing to be unsigned for the purpose of holding very large values or overflow not really being possible (it just gives the result modulo 2**N). The bug could also be fixed by changing everything to size_t. On 64-bit arches, that would waste 32 bits for all the fields in the struct, and it still needs delicate conversions to put differences in signed longs. Using a consistently wide type does give a better chance of the delicate conversions occurring accidentally. Bruce From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 22:19:59 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 072F47BE for ; Fri, 6 Feb 2015 22:19:59 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0111.outbound.protection.outlook.com [157.56.110.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9BF98757 for ; Fri, 6 Feb 2015 22:19:58 +0000 (UTC) Received: from BLUPR0501MB1060.namprd05.prod.outlook.com (25.160.36.150) by CY1PR0501MB1547.namprd05.prod.outlook.com (25.161.161.145) with Microsoft SMTP Server (TLS) id 15.1.75.20; Fri, 6 Feb 2015 22:19:49 +0000 Received: from BLUPR0501MB1060.namprd05.prod.outlook.com ([25.160.36.150]) by BLUPR0501MB1060.namprd05.prod.outlook.com ([25.160.36.150]) with mapi id 15.01.0075.002; Fri, 6 Feb 2015 22:19:49 +0000 From: Anuranjan Shukla To: Bruce Evans Subject: Re: Buggy sbspace() on 64bit builds? Thread-Topic: Buggy sbspace() on 64bit builds? Thread-Index: AQHQQa9oYD6vE33IJEufQkyrNnicWZzjUieAgABadwA= Date: Fri, 6 Feb 2015 22:19:48 +0000 Message-ID: References: <20150206183036.S1246@besplex.bde.org> In-Reply-To: <20150206183036.S1246@besplex.bde.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.7.141117 x-originating-ip: [66.129.241.11] authentication-results: optusnet.com.au; dkim=none (message not signed) header.d=none; x-microsoft-antispam: BCL:0;PCL:0;RULEID:;SRVR:CY1PR0501MB1547; x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:; SRVR:CY1PR0501MB1547; x-forefront-prvs: 047999FF16 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(24454002)(479174004)(51704005)(377454003)(76176999)(54356999)(50986999)(62966003)(106116001)(36756003)(77156002)(92566002)(102836002)(2900100001)(19580395003)(2950100001)(87936001)(19580405001)(66066001)(86362001)(40100003)(99286002)(110136001)(122556002)(46102003)(2656002)(83506001); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0501MB1547; H:BLUPR0501MB1060.namprd05.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; Content-Type: text/plain; charset="us-ascii" Content-ID: <6B811B18F3770D4EABF4F3AC1F3DA6B2@namprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Feb 2015 22:19:48.7492 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0501MB1547 Cc: Simon Gerraty , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 22:19:59 -0000 On 2/6/15, 12:56 AM, "Bruce Evans" wrote: >long >sbspace(struct sockbuf *sb) >{ > u_int bleft, mleft; > > if (sb->sb_flags & SB_STOP) > return (0); > bleft =3D sb->sb_hiwat < sb->sb_cc ? 0 : sb->sb_hiwat - sb->sb_cc; > mleft =3D sb->sb_mbmax < sb->sb_mbcnt ? 9 : sb->sb_mbmax - sb->sb_mbcnt; > return (min(bleft, mleft)); >} This is sort of what we've been using past few years as a local change. Is this, along with return value being 'int' acceptable as a final determination? There are places like sockopt processing where the return is anyway being typecast to int like Bruce mentioned. Appreciate the inputs so far. Thanks Anu From owner-freebsd-arch@FreeBSD.ORG Fri Feb 6 23:18:16 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10496C0E for ; Fri, 6 Feb 2015 23:18:16 +0000 (UTC) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B45FBD7E for ; Fri, 6 Feb 2015 23:18:15 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 8EE2EB8419; Sat, 7 Feb 2015 00:18:12 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 7EAC728494; Sat, 7 Feb 2015 00:18:12 +0100 (CET) Date: Sat, 7 Feb 2015 00:18:12 +0100 From: Jilles Tjoelker To: Bruce Evans Subject: Re: Change default VFS timestamp precision? Message-ID: <20150206231812.GA13408@stack.nl> References: <201412161348.41219.jhb@freebsd.org> <20141216233844.GA1490@stack.nl> <20141217105256.J1490@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141217105256.J1490@besplex.bde.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 23:18:16 -0000 On Wed, Dec 17, 2014 at 11:39:37AM +1100, Bruce Evans wrote: > On Wed, 17 Dec 2014, Jilles Tjoelker wrote: > > Although some breakage may be caused, increasing precision sounds fine > > to me, but only to the level of microseconds, since there is no way to > > set a timestamp to the nanosecond (this would be futimens/utimensat). It > > is easy to be surprised when cp -p creates an file that appears older > > than the original. > As I said in too many more words in another reply. > > To avoid cross-arch surprises with applications that use > > second-resolution APIs, either all or no architectures should generate > > timestamps more accurate than seconds. > That might be a feature for finding bugs. And it doesn't work for > avoiding them, because foreign file systems may have uncontrollable > timestamp precision. E.g., a foreign nfs server. Architecture-dependent bugs are not nice. Bugs specific to foreign file systems are unavoidable. > > There is no benefit for the particular case of make(1), since it only > > uses timestamps in seconds. > Urk, I now rememeber seeing that 15-20 years ago and deciding that the > problem with timestamps for make couldn't be very large. :) > > A quick grep also shows various calls to utime(3) with a non-NULL timep > > argument. The ones in contrib/bmake and usr.bin/make definitely look > > incorrect. > Do you mean that they are incorrect because they use utime(3) and not > utimes(3), or just because they use the non-NULL timep? The latter > is just a style bug in at least usr.bin/make. Since it hasn't > caught up with the existence of utimes(3), it is reasonable for it > to also have not caught up with the existence of the non-NULL timep. > It just wants to set the time to the current time. It uses its own > copy of the current time for this. Calling any utime/utimes/utimens function with the current time subverts vfs.timestamp_precision. This may cause timestamps set by different applications to compare differently from provable ordering. For example, if "make -t" finishes within one second, a file created before it was started might have a timestamp 1423261861.1 while files touched by "make -t" will have timestamp 1423261861.0. Also, with NFS it is better to use the server's clock for timestamps, so timestamps in the same filesystem are consistent. Calling a utime/utimes function with a timestamp obtained from stat loses precision. If the timestamp comes from somewhere else, that other place (e.g. network protocol, archive file) may cause unavoidable precision loss. Calling utime() with a NULL timep is not inherently wrong, although futimes() or futimens() is preferable if the file is open. > Perhaps this is technically better too -- it makes "make -t" touch > everything with the same time. There is no need for "make -t" to touch everything with the same time if it processes files in dependency order. > This reminds me of related bugs in touch(1): > - touch also tries first with a non-NULL timep. So it uses its own idea > of the current time. This is normally the one returned a little in > the past by gettimeofday(). It has microseconds resolution. This > normally succeeds, giving a result that is inconsistent with the > one that would result from using a NULL timep, since the kernel now > honors vfs.timestamp_precision and that is not usually 2. Yes, this is wrong the same way as in "make -t". > - sometimes the first try fails. Then touch retries with a NULL timep, > if the semantics allow this. Sometimes this succeeds. Another bug: it retries with a NULL timep even if -a or -m is given, requesting only one of the timestamps be updated. The new functionality in futimens/utimensat allows doing "touch -a" and "touch -m" with the current time properly (setting one timestamp to vfs_timestamp and leaving the other timestamp unchanged). As with the old methods, only the owner of the file and root can do this. The below patch fixes these problems. > - sometimes both tries fail due to permissions problems. Then touch > used to retry using read() and write(). Someone removed this, with > the justification that all file systems support utimes(). But this > is broken, since touching for read requires only read permission, > while utimes() requires some sort of write permission. For touching > for write, I don't see any problem now except for the above ordering > problems: a NULL timep should be tried first since it requires fewer > permissions and gives the best semantics; if it fails then there is > no point in retrying with a non-NULL timep. "touch -a" with the current time could be done using read() on regular files (and readdir() on directories). This only accidentally worked using the old readwrite code. If the filesystem is mounted read-only or noatime, read() does not update the timestamp but this situation cannot be reliably distinguished from a coarse vfs.timestamp_precision on a file that had already been read recently. An fstat() call should follow anyway to force the timestamp to be updated like utime calls do. > The non-NULL timep in make gives the following bug since make doesn't > retry with a NULL timep: make -t fails if the files ar writeable but > not owned. This is best fixed by always using a NULL timep. Index: usr.bin/touch/touch.c =================================================================== --- usr.bin/touch/touch.c (revision 277645) +++ usr.bin/touch/touch.c (working copy) @@ -76,8 +76,8 @@ myname = basename(argv[0]); Aflag = aflag = cflag = mflag = timeset = 0; atflag = 0; - if (clock_gettime(CLOCK_REALTIME, &ts[0]) == -1) - err(1, "clock_gettime(CLOCK_REALTIME)"); + ts[0].tv_sec = ts[1].tv_sec = 0; + ts[0].tv_nsec = ts[1].tv_nsec = UTIME_NOW; while ((ch = getopt(argc, argv, "A:acd:fhmr:t:")) != -1) switch(ch) { @@ -152,6 +152,11 @@ ts[1] = ts[0]; } + if (!aflag) + ts[0].tv_nsec = UTIME_OMIT; + if (!mflag) + ts[1].tv_nsec = UTIME_OMIT; + if (*argv == NULL) usage(myname); @@ -183,11 +188,6 @@ continue; } - if (!aflag) - ts[0] = sb.st_atim; - if (!mflag) - ts[1] = sb.st_mtim; - /* * We're adjusting the times based on the file times, not a * specified time (that gets handled above). @@ -203,26 +203,9 @@ } } - /* Try utimensat(2). */ if (!utimensat(AT_FDCWD, *argv, ts, atflag)) continue; - /* If the user specified a time, nothing else we can do. */ - if (timeset || Aflag) { - rval = 1; - warn("%s", *argv); - continue; - } - - /* - * System V and POSIX 1003.1 require that a NULL argument - * set the access/modification times to the current time. - * The permission checks are different, too, in that the - * ability to write the file is sufficient. Take a shot. - */ - if (!utimensat(AT_FDCWD, *argv, NULL, atflag)) - continue; - rval = 1; warn("%s", *argv); } @@ -239,7 +222,7 @@ int yearset; char *p; /* Start with the current time. */ - now = tvp[0].tv_sec; + now = time(NULL); if ((t = localtime(&now)) == NULL) err(1, "localtime"); /* [[CC]YY]MMDDhhmm[.SS] */ @@ -301,7 +284,7 @@ time_t now; struct tm *t; /* Start with the current time. */ - now = tvp[0].tv_sec; + now = time(NULL); if ((t = localtime(&now)) == NULL) err(1, "localtime"); -- Jilles Tjoelker From owner-freebsd-arch@FreeBSD.ORG Sat Feb 7 01:10:46 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D017E964 for ; Sat, 7 Feb 2015 01:10:46 +0000 (UTC) Received: from mail-pd0-f172.google.com (mail-pd0-f172.google.com [209.85.192.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A2AB9B35 for ; Sat, 7 Feb 2015 01:10:46 +0000 (UTC) Received: by pdbft15 with SMTP id ft15so18122707pdb.11 for ; Fri, 06 Feb 2015 17:10:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version; bh=orTUFhTcvUecbINZtzqQIRqQDbTgaeBimMlrDNx7ozE=; b=xT5TpY/gfhQXlw9fZ5WOYQcEQ+D6vP+IjC5+UlfCjP9wHmtAHD+c+g+Hk+ECQ7YdNz TMjJ1UMiRCDgYVxETy083w8kFXcyDypmut2w7BcoBt+5fcujBJ+EEPw26lE9t99wO8JC rfTA5pA4sA1oWAokegRmJIk9vGfjAu2lV9E4hi6SJ7m4zM4nkXxzKEFn24Jn5Np+fFD6 FL487WyqHg8OijVaMKZxq5odFXyuiMDYLisuTZiGWkKIBuRQygRymZBGdG4klgBlaQgg MdznEUhK6n6csnvjt1GCgpxbpWupaA1w4UZ04LtnP7OeIjxLKtU0Mx5qUF5i6r99zt69 8Z3w== X-Received: by 10.66.122.69 with SMTP id lq5mr9993433pab.52.1423271438939; Fri, 06 Feb 2015 17:10:38 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:e47d:5c02:fe5f:40fa? ([2601:8:ab80:7d6:e47d:5c02:fe5f:40fa]) by mx.google.com with ESMTPSA id fx1sm9256697pdb.35.2015.02.06.17.10.38 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 06 Feb 2015 17:10:38 -0800 (PST) From: Garrett Cooper Content-Type: multipart/signed; boundary="Apple-Mail=_2D3E7938-9EBC-455C-ADD6-4B591D99FF4A"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: [RFC] Add log(9) support to witness(4) via the WITNESS_USE_SYSLOG kernel compile time option Message-Id: <965FE4D7-2C70-4FF5-B566-F8BD3C31DE58@gmail.com> Date: Fri, 6 Feb 2015 17:10:36 -0800 To: "freebsd-arch@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2015 01:10:46 -0000 --Apple-Mail=_2D3E7938-9EBC-455C-ADD6-4B591D99FF4A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi all, I=92ve opened a review request on phabricator for adding a = compile-time option for witness(4) to use log(9) instead of printf(9) = when printing out some output (mostly LORs and assorted warnings): = https://reviews.freebsd.org/D1794 . Any review input would be much appreciated. Thanks! -NGie --Apple-Mail=_2D3E7938-9EBC-455C-ADD6-4B591D99FF4A Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJU1WYMAAoJEMZr5QU6S73eZYsH/RcRloGO5PU/QApI1kWxWqix /0bvVZa6YB4KzWh/2gHbNhB+UpntbvX5JumrzhtwsVDiL/MnYqDAUygc9USmqEMI Tk3LW7ULjA4K/fl/W4wINNjsr70kHHlWKcXUtkfsiiFcftybRtVzwXTgXT10WZO8 M8lgAKbGJJfGpHjwljtVZRWx2ucv6x4CqQn0gfUtvEtOxaxGZK+1RqLc+oJQM4GN HAwKSQsnQycsyfEeGFI/+i+OPj7pCKpZFDf5F3Fo8R91xoRoZFGAf5RWqjZmQVam +8lsBlIOk2bsMXpGSmgQpzee3coZm5Z2ZhpgyuINvXk9VbuNLnmCMg8oACwIErU= =EZ3L -----END PGP SIGNATURE----- --Apple-Mail=_2D3E7938-9EBC-455C-ADD6-4B591D99FF4A-- From owner-freebsd-arch@FreeBSD.ORG Sat Feb 7 01:18:09 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 050B0ABB for ; Sat, 7 Feb 2015 01:18:09 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0145.outbound.protection.outlook.com [65.55.169.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F06EBA9 for ; Sat, 7 Feb 2015 01:18:08 +0000 (UTC) Received: from CO2PR05CA042.namprd05.prod.outlook.com (10.141.241.170) by DM2PR05MB447.namprd05.prod.outlook.com (10.141.104.150) with Microsoft SMTP Server (TLS) id 15.1.75.20; Sat, 7 Feb 2015 01:03:48 +0000 Received: from BY2FFO11FD016.protection.gbl (2a01:111:f400:7c0c::145) by CO2PR05CA042.outlook.office365.com (2a01:111:e400:1429::42) with Microsoft SMTP Server (TLS) id 15.1.81.19 via Frontend Transport; Sat, 7 Feb 2015 01:03:47 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD016.mail.protection.outlook.com (10.1.14.148) with Microsoft SMTP Server (TLS) id 15.1.87.10 via Frontend Transport; Sat, 7 Feb 2015 01:03:47 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 6 Feb 2015 17:03:45 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id t1713iW49292; Fri, 6 Feb 2015 17:03:44 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 3598F580A3; Fri, 6 Feb 2015 17:03:44 -0800 (PST) To: Anuranjan Shukla Subject: Re: Buggy sbspace() on 64bit builds? In-Reply-To: References: <20150206183036.S1246@besplex.bde.org> Comments: In-reply-to: Anuranjan Shukla message dated "Fri, 06 Feb 2015 22:19:48 +0000." From: "Simon J. Gerraty" X-Mailer: MH-E 8.0.3; nmh 1.3; GNU Emacs 22.3.1 Date: Fri, 6 Feb 2015 17:03:44 -0800 Message-ID: <5977.1423271024@chaos> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; freebsd.org; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(76506005)(50466002)(57986006)(92566002)(117636001)(46102003)(50986999)(76176999)(6806004)(19580405001)(19580395003)(86362001)(50226001)(47776003)(2950100001)(110136001)(87936001)(105596002)(48376002)(1941001)(62966003)(77156002)(77096005)(33716001)(106466001)(42262002)(62816006); DIR:OUT; SFP:1102; SCL:1; SRVR:DM2PR05MB447; H:P-EMF02-SAC.jnpr.net; FPR:; SPF:SoftFail; MLV:sfv; LANG:en; X-Microsoft-Antispam: UriScan:; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB447; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004); SRVR:DM2PR05MB447; X-Forefront-PRVS: 0480A51D4A X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:;SRVR:DM2PR05MB447; X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2015 01:03:47.1800 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.16] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR05MB447 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2015 01:18:09 -0000 Anuranjan Shukla wrote: > this, along with return value being 'int' acceptable as a final > determination? It is ok for the function to return long, so long as an int is used internally. Casting the int to long - implicit on return does no harm. #include #include #include int main(int argc, char *argv[]) { uint a, b; long r1; int r2; a = 1; b = 2; r1 = a - b; r2 = a - b; printf("r1=%ld\nr2=%d\nr3=%ld\n", r1, r2, (long)r2); exit(0); } r1=4294967295 r2=-1 r3=-1 so I think just using 'int' internally should work for now, perhaps with a comment saying the object size should match those being subtracted etc. From owner-freebsd-arch@FreeBSD.ORG Sat Feb 7 07:09:50 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1A134DB for ; Sat, 7 Feb 2015 07:09:50 +0000 (UTC) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id 7342DB3 for ; Sat, 7 Feb 2015 07:09:50 +0000 (UTC) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 280B5D61438; Sat, 7 Feb 2015 18:09:48 +1100 (AEDT) Date: Sat, 7 Feb 2015 18:09:47 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: "Simon J. Gerraty" Subject: Re: Buggy sbspace() on 64bit builds? In-Reply-To: <5977.1423271024@chaos> Message-ID: <20150207180005.F1334@besplex.bde.org> References: <20150206183036.S1246@besplex.bde.org> <5977.1423271024@chaos> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=baJSDo/B c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=OUXY8nFuAAAA:8 a=bdFpWTK2ga1vbW0cgXkA:9 a=CjuIK1q_8ugA:10 Cc: Anuranjan Shukla , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2015 07:09:50 -0000 On Fri, 6 Feb 2015, Simon J. Gerraty wrote: > Anuranjan Shukla wrote: >> this, along with return value being 'int' acceptable as a final >> determination? > > It is ok for the function to return long, > so long as an int is used internally. > Casting the int to long - implicit on return does no harm. > > #include > #include > #include > > int > main(int argc, char *argv[]) > { > uint a, b; > long r1; > int r2; > > a = 1; > b = 2; > > r1 = a - b; > r2 = a - b; > > printf("r1=%ld\nr2=%d\nr3=%ld\n", r1, r2, (long)r2); > exit(0); > } > > r1=4294967295 > r2=-1 > r3=-1 > > so I think just using 'int' internally should work for now, > perhaps with a comment saying the object size should match > those being subtracted etc. Yes, that is simplest. Bruce From owner-freebsd-arch@FreeBSD.ORG Sat Feb 7 10:15:41 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2CD1D1C for ; Sat, 7 Feb 2015 10:15:40 +0000 (UTC) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id B3A595EE for ; Sat, 7 Feb 2015 10:15:39 +0000 (UTC) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id C6C52423B06; Sat, 7 Feb 2015 21:15:31 +1100 (AEDT) Date: Sat, 7 Feb 2015 21:15:30 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Jilles Tjoelker Subject: Re: Change default VFS timestamp precision? In-Reply-To: <20150206231812.GA13408@stack.nl> Message-ID: <20150207190728.W1334@besplex.bde.org> References: <201412161348.41219.jhb@freebsd.org> <20141216233844.GA1490@stack.nl> <20141217105256.J1490@besplex.bde.org> <20150206231812.GA13408@stack.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=A5NVYcmG c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=Uy_ugoFUF1DTDgMAxxYA:9 a=CjuIK1q_8ugA:10 Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2015 10:15:41 -0000 On Sat, 7 Feb 2015, Jilles Tjoelker wrote: > On Wed, Dec 17, 2014 at 11:39:37AM +1100, Bruce Evans wrote: >> On Wed, 17 Dec 2014, Jilles Tjoelker wrote: >* ... >>> A quick grep also shows various calls to utime(3) with a non-NULL timep >>> argument. The ones in contrib/bmake and usr.bin/make definitely look >>> incorrect. > >> Do you mean that they are incorrect because they use utime(3) and not >> utimes(3), or just because they use the non-NULL timep? The latter >> is just a style bug in at least usr.bin/make. Since it hasn't >> caught up with the existence of utimes(3), it is reasonable for it >> to also have not caught up with the existence of the non-NULL timep. >> It just wants to set the time to the current time. It uses its own >> copy of the current time for this. > > Calling any utime/utimes/utimens function with the current time subverts > vfs.timestamp_precision. This may cause timestamps set by different > applications to compare differently from provable ordering. For example, > if "make -t" finishes within one second, a file created before it was > started might have a timestamp 1423261861.1 while files touched by "make > -t" will have timestamp 1423261861.0. This annoys me for backups too. Verifying backups includes checking their timestamps, and it is too hard to determine if a difference is small enough to not be an error. I mostly use old versions of FreeBSD which don't have *timens*(). So utilities cannot restores timestamps with nanoseconds resolution. Backup utilities are also further than most utilities from being able to back them up (some backup formats barely have seconds resolution). So times with nanoseconds (or even milliseconds) resolution shouldn't be created. vfs.timestampe_precision gives this. Except touch(1) subverts it. > Also, with NFS it is better to use the server's clock for timestamps, so > timestamps in the same filesystem are consistent. I think this happens in most cases. nfs has remarkably little code for setting timestamps. Hmm, now I wonder how its caching timeouts work when the server times differ significantly. It compares file mtimes and ctimes with timeouts, but the mtimes and ctimes are usually in server real time (or worse when an application steps the mtime backwards), but timeouts should be in client monotonic time. > ... > Calling utime() with a NULL timep is not inherently wrong, although > futimes() or futimens() is preferable if the file is open. I think it is as correct as possible for the current time. It doesn't subvert vfs.timestamp_precision. > ... >> This reminds me of related bugs in touch(1): >> - touch also tries first with a non-NULL timep. So it uses its own idea >> of the current time. This is normally the one returned a little in >> the past by gettimeofday(). It has microseconds resolution. This >> normally succeeds, giving a result that is inconsistent with the >> one that would result from using a NULL timep, since the kernel now >> honors vfs.timestamp_precision and that is not usually 2. > > Yes, this is wrong the same way as in "make -t". > >> - sometimes the first try fails. Then touch retries with a NULL timep, >> if the semantics allow this. Sometimes this succeeds. > > Another bug: it retries with a NULL timep even if -a or -m is given, > requesting only one of the timestamps be updated. Perhaps that is the best that it can do, or it can be treated as an error since the application/user shouldn't try to set only one of the timestamps on a file system that doesn't support both. However, it might support just the one asked for, and asking for both when only 1 is supported is not treated as an error. > The new functionality in futimens/utimensat allows doing "touch -a" and > "touch -m" with the current time properly (setting one timestamp to > vfs_timestamp and leaving the other timestamp unchanged). As with the > old methods, only the owner of the file and root can do this. When this gets down to individual file systems, the timestamp to be left unchanged becomes VNOVAL and is always (?) ignored. However, to make the usual case when changes to both timestamps are requested work, requests to change nonexistent attributes are usually ignored too. I don't see any better way to control this than a mount flag for every nonexistent attribute. msdosfs currently has hard-coded policies that differ for each nonexistent attribute. They are mostly too strict except for atimes. > The below patch fixes these problems. Looks good. > ... >> The non-NULL timep in make gives the following bug since make doesn't >> retry with a NULL timep: make -t fails if the files ar writeable but >> not owned. > > This is best fixed by always using a NULL timep. I think this is OK for msdosfs. The problems with setting timestamps in msdosfs are more in cp -p and mv. There the NULL timep cannot be used. msdosfs is very strict in this case. cp and mv are also strict, except mv was broken recently to hide the corresponding problems for unsettable flags (e.g., zfs sets the archive flag a lot, but nfs doesn't support preserving it and hard-codes strict failures for attempts to set it). The strictness is mostly just to fail to copy uncopyable attributes and just warn about this. Even the warnings are too much when they occur for every file. Even mv is mostly not required by POSIX to duplicate attributes. But it is required to warn. So nfs's strictness for chflags() is the correct default and msdosfs's non-strictness for atimes is an incorrect default, but more practical. > Index: usr.bin/touch/touch.c > =================================================================== > --- usr.bin/touch/touch.c (revision 277645) > +++ usr.bin/touch/touch.c (working copy) > ... > @@ -239,7 +222,7 @@ > int yearset; > char *p; > /* Start with the current time. */ > - now = tvp[0].tv_sec; > + now = time(NULL); > if ((t = localtime(&now)) == NULL) > err(1, "localtime"); > /* [[CC]YY]MMDDhhmm[.SS] */ This seems to be broken now, since time() is broken now. I think tvp was obtained using clock_gettime(CLOCK_REALTIME, ... ) (previously gettimeofday). These give the correct time. time() used to use gettimeofday() to get the same correct time. But someone broke time(). It now uses clock_gettime(TIME_SECOND, ... ). TIME_SECOND is getbintime() rounded down. getbintime() lags the current time by an average of tc_tick * 1/hz / 2 seconds. After rounding down, TIME_SECOND lags the current time by a full seconds for a duration of an average of tc_tick * 1/hz / 2 seconds every second. The bug is compatible with the one given by vfs.timestamp_precision = 0 and other low settings of the sysctl. The latter also uses the lagging clock. Previously, the incoherent clocks showed up in file times tests like: time(&t1); write to a file stat(file, &sb); assert(t1 <= sb.st_mtime); /* can fail due to incoherent clocks */ Add delays as necessary to compensate for the speed of light but not to compensate for buggy clocks. time() now uses the same buggy clock when vfs.timestamp_precision = 0, so the clocks are coherent because they are the same (unless the real time is stepped). OTOH, if CLOCK_GETTIME is used to read t1, then the bug is detected as before. When vfs.timestamp_precision is higher so that the time is up to date, then the incoherencies go in the other direction: write to a file stat(file, &sb); time(&t1); assert(t1 >= sb.st_mtime); /* can fail due to incoherent clocks */ This looks like a file times test, but it actually finds the buggy time(). > @@ -301,7 +284,7 @@ > time_t now; > struct tm *t; > /* Start with the current time. */ > - now = tvp[0].tv_sec; > + now = time(NULL); > if ((t = localtime(&now)) == NULL) > err(1, "localtime"); Similarly. THe comment should be attached to the or just omitted now that the code obiously fetches the (buggy) current time. For a quick fix, keep using clock_gettime(CLOCK_REALTIME, ...). I don't see a good way to be bug for bug compatible with the file system. A bad way is to write to a file on the same file system and stat it, as in the above tests. This gives a time slightly in the past according to the file system's clock. nfs gives interesting complications. The server's clock may be much more incoherent with CLOCK_REALTIME than the local clock. Using CLOCK_SECOND for time() is a bogus optimization since most of the overhead of clock_gettime() is in the syscall unless the timecounter hardware is slow. On freefall now, it is not even optimal or wrong, since on x86 clock_gettime() is not a syscall and the library is missing all the bogus optimizations; it uses binuptime() for everything. This results in CLOCK_REALTIME and CLOCK_SECOND both taking 34 nsec on freefall. It might be possible to reduce this to about 1 nanosecond for CLOCK_SECOND by duplicating more of the kernel. This is not clear, since the library uses excessive locking where the kernel is missing locking. Bruce From owner-freebsd-arch@FreeBSD.ORG Sat Feb 7 20:00:27 2015 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E2B99856 for ; Sat, 7 Feb 2015 20:00:27 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 746FFFE2 for ; Sat, 7 Feb 2015 20:00:27 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 726FE3592ED; Sat, 7 Feb 2015 21:00:23 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 4A67128494; Sat, 7 Feb 2015 21:00:23 +0100 (CET) Date: Sat, 7 Feb 2015 21:00:23 +0100 From: Jilles Tjoelker To: Bruce Evans Subject: Re: Change default VFS timestamp precision? Message-ID: <20150207200023.GA36549@stack.nl> References: <201412161348.41219.jhb@freebsd.org> <20141216233844.GA1490@stack.nl> <20141217105256.J1490@besplex.bde.org> <20150206231812.GA13408@stack.nl> <20150207190728.W1334@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150207190728.W1334@besplex.bde.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Feb 2015 20:00:28 -0000 On Sat, Feb 07, 2015 at 09:15:30PM +1100, Bruce Evans wrote: > On Sat, 7 Feb 2015, Jilles Tjoelker wrote: > > On Wed, Dec 17, 2014 at 11:39:37AM +1100, Bruce Evans wrote: > >> This reminds me of related bugs in touch(1): > >> - touch also tries first with a non-NULL timep. So it uses its own idea > >> of the current time. This is normally the one returned a little in > >> the past by gettimeofday(). It has microseconds resolution. This > >> normally succeeds, giving a result that is inconsistent with the > >> one that would result from using a NULL timep, since the kernel now > >> honors vfs.timestamp_precision and that is not usually 2. > > Yes, this is wrong the same way as in "make -t". > >> - sometimes the first try fails. Then touch retries with a NULL timep, > >> if the semantics allow this. Sometimes this succeeds. > > Another bug: it retries with a NULL timep even if -a or -m is given, > > requesting only one of the timestamps be updated. > Perhaps that is the best that it can do, or it can be treated as an error > since the application/user shouldn't try to set only one of the timestamps > on a file system that doesn't support both. However, it might support > just the one asked for, and asking for both when only 1 is supported is > not treated as an error. Perhaps I did not explain this clearly. My point was that, as a non-root user, "touch -a /dev/null" behaves as "touch /dev/null" instead of failing or using read(). > > The new functionality in futimens/utimensat allows doing "touch -a" and > > "touch -m" with the current time properly (setting one timestamp to > > vfs_timestamp and leaving the other timestamp unchanged). As with the > > old methods, only the owner of the file and root can do this. > When this gets down to individual file systems, the timestamp to be left > unchanged becomes VNOVAL and is always (?) ignored. However, to make > the usual case when changes to both timestamps are requested work, > requests to change nonexistent attributes are usually ignored too. I > don't see any better way to control this than a mount flag for every > nonexistent attribute. msdosfs currently has hard-coded policies that > differ for each nonexistent attribute. They are mostly too strict except > for atimes. The code in sys/kern/vfs_syscalls.c sets vattr.va_vaflags |= VA_UTIMES_NULL if the timestamp was null or two times UTIME_NOW. This is what file systems use for permission checks (via vn_utimes_perm()). Although msdosfs supports file systems without Windows 95 extensions that do not have atimes, this is rather rare and not worth much time for bug fixing. > > The below patch fixes these problems. > Looks good. Thanks for the review. > > ... > >> The non-NULL timep in make gives the following bug since make doesn't > >> retry with a NULL timep: make -t fails if the files ar writeable but > >> not owned. > > This is best fixed by always using a NULL timep. > I think this is OK for msdosfs. The problems with setting timestamps > in msdosfs are more in cp -p and mv. There the NULL timep cannot be > used. msdosfs is very strict in this case. cp and mv are also strict, > except mv was broken recently to hide the corresponding problems for > unsettable flags (e.g., zfs sets the archive flag a lot, but nfs doesn't > support preserving it and hard-codes strict failures for attempts to set > it). The strictness is mostly just to fail to copy uncopyable attributes > and just warn about this. Even the warnings are too much when they occur > for every file. Even mv is mostly not required by POSIX to duplicate > attributes. But it is required to warn. So nfs's strictness for > chflags() is the correct default and msdosfs's non-strictness for > atimes is an incorrect default, but more practical. > > Index: usr.bin/touch/touch.c > > =================================================================== > > --- usr.bin/touch/touch.c (revision 277645) > > +++ usr.bin/touch/touch.c (working copy) > > ... > > @@ -239,7 +222,7 @@ > > int yearset; > > char *p; > > /* Start with the current time. */ > > - now = tvp[0].tv_sec; > > + now = time(NULL); > > if ((t = localtime(&now)) == NULL) > > err(1, "localtime"); > > /* [[CC]YY]MMDDhhmm[.SS] */ > This seems to be broken now, since time() is broken now. I think tvp > was obtained using clock_gettime(CLOCK_REALTIME, ... ) (previously > gettimeofday). These give the correct time. time() used to use > gettimeofday() to get the same correct time. But someone broke time(). > It now uses clock_gettime(TIME_SECOND, ... ). TIME_SECOND is getbintime() > rounded down. getbintime() lags the current time by an average of > tc_tick * 1/hz / 2 seconds. After rounding down, TIME_SECOND lags the > current time by a full seconds for a duration of an average of > tc_tick * 1/hz / 2 seconds every second. > The bug is compatible with the one given by vfs.timestamp_precision = 0 > and other low settings of the sysctl. The latter also uses the lagging > clock. Previously, the incoherent clocks showed up in file times tests > like: > time(&t1); > write to a file > stat(file, &sb); > assert(t1 <= sb.st_mtime); /* can fail due to incoherent clocks */ > Add delays as necessary to compensate for the speed of light but not to > compensate for buggy clocks. > time() now uses the same buggy clock when vfs.timestamp_precision = 0, so > the clocks are coherent because they are the same (unless the real time is > stepped). OTOH, if CLOCK_GETTIME is used to read t1, then the bug is > detected as before. > When vfs.timestamp_precision is higher so that the time is up to date, > then the incoherencies go in the other direction: > write to a file > stat(file, &sb); > time(&t1); > assert(t1 >= sb.st_mtime); /* can fail due to incoherent clocks */ > This looks like a file times test, but it actually finds the buggy time(). > > @@ -301,7 +284,7 @@ > > time_t now; > > struct tm *t; > > /* Start with the current time. */ > > - now = tvp[0].tv_sec; > > + now = time(NULL); > > if ((t = localtime(&now)) == NULL) > > err(1, "localtime"); > Similarly. > THe comment should be attached to the or just omitted now that the code > obiously fetches the (buggy) current time. > For a quick fix, keep using clock_gettime(CLOCK_REALTIME, ...). I > don't see a good way to be bug for bug compatible with the file system. > A bad way is to write to a file on the same file system and stat it, > as in the above tests. This gives a time slightly in the past according > to the file system's clock. nfs gives interesting complications. The > server's clock may be much more incoherent with CLOCK_REALTIME than > the local clock. The code that now uses time() is for "touch -t" and the obsolete timestamp operand. It serves to supply the year if it is not given, and the seconds in the obsolete timestamp operand. It does not supply a full tv_sec. -- Jilles Tjoelker