From owner-freebsd-hackers@freebsd.org Tue Jun 12 17:04:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A02401007ED8 for ; Tue, 12 Jun 2018 17:04:26 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg0-x233.google.com (mail-pg0-x233.google.com [IPv6:2607:f8b0:400e:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2675D6EF5F for ; Tue, 12 Jun 2018 17:04:26 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg0-x233.google.com with SMTP id m5-v6so11749021pgd.3 for ; Tue, 12 Jun 2018 10:04:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=jTyOYCimflHFEnDSgQMYPt2hInVZ9zieZTlY4DYG9Hs=; b=dGcmkOSh8ZGeUfIYFCdbFJiPSKoXNG3doVdIcD0XeOZ5ub4jFgo0Dv5T4R8UUt75gH w8zYlOQjImcLmCt4zZtQcfFO5kiSipxIUf9Ew90kJFfMj/6tdlI5VZm+vf4WTq1cW7h1 qjuFpESTYrOTLsWU912VCJu2dpd7LFTVLMk+ECovfpI2FTm0DtxfJS83MWkC0KDILoVO EcBa0Kmr7grxfbR40N6+cyu0xHzhsWKeaPUE4k9R9xFrdhPwmIpup2WpDHv3cmXzTPSq zttaZCMX12e929u/ziyliaKhPaQsQXnfMlGxN+gUd7GJfEvTbWIkTHWngP9bOeotyUwa f0uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:subject:message-id :mime-version:content-disposition:user-agent; bh=jTyOYCimflHFEnDSgQMYPt2hInVZ9zieZTlY4DYG9Hs=; b=TxBSdMoYfnrBlYxDgzseGCpmD02PZ8HmN2sQzEVTiUrSQoSfgwWv6+wT5c36uNrrUn hGyvUxvsPOLAlH6gzQ/8S3ExEzjPqmqd0AgBdA6veoYMs8yuEllfkrtMbz4rgMfs4kvN 6V0ACa1B25wri+Y+71RxgkYiIX8GtB/xDEUeSHUBMp6B+jaoaJ65udozMq8BI/64Hmlk qq/k/uMNooH+/6fsnx7HpBXnK4G/tiA8WykDt7/UyKZdBtfp6AJBmoNVYAJ4EynSBVUz KSPmnSmZyLJpap8OUrTzjNssIu5gpm2NbMjnPGgIt6BHV6Ay+qn5+Lo43M3PQ/A4RFls ybWg== X-Gm-Message-State: APt69E2ZIxSBFMdNMTpj4I+P7TqF+3v0QEjKSabRu53WdjzT3WevOxCd cZrTYHEJgfVS0MXrsPsz9U7FS+WC X-Google-Smtp-Source: ADUXVKKT/W/RA7yAwmI2/+k+YDFx7VT/gmCiD42t0IsSefT+iV6ZSz0f5O5CJ5uKt7wGg53rofKnCw== X-Received: by 2002:a63:6741:: with SMTP id b62-v6mr1037976pgc.5.1528823064930; Tue, 12 Jun 2018 10:04:24 -0700 (PDT) Received: from raichu (toroon0560w-lp140-02-70-49-169-156.dsl.bell.ca. [70.49.169.156]) by smtp.gmail.com with ESMTPSA id g2-v6sm1098853pfc.4.2018.06.12.10.04.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jun 2018 10:04:23 -0700 (PDT) Sender: Mark Johnston Date: Tue, 12 Jun 2018 13:04:20 -0400 From: Mark Johnston To: freebsd-hackers@freebsd.org Subject: specifying alignment of loader files Message-ID: <20180612170420.GD56138@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 17:04:26 -0000 Hi, I'm writing some code which processes a file loaded by the loader. I want the file's contents to be available at a certain alignment in memory, and as far as I can see, the loader provides no alignment guarantees today. The access will happen early enough during boot that making an aligned copy of the data will be awkward, so I'd like the loader to provide the desired alignment. I'm considering adding a new "module_align" variable that would specify the alignment for a given file type, and plumb that through to command_load(). Does anyone have an alternate suggestion, or an objection to my proposal? From owner-freebsd-hackers@freebsd.org Tue Jun 12 17:11:27 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBCA410089F8 for ; Tue, 12 Jun 2018 17:11:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6BCCF6F5AF for ; Tue, 12 Jun 2018 17:11:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x235.google.com with SMTP id l19-v6so249209ioj.5 for ; Tue, 12 Jun 2018 10:11:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=x5REzk5jnfYN8HQkfIFFSUZC7R4vxhJRzMFl1C8KRK0=; b=UxVW9R06xUDEB3m0fM+FaxwHlmG2yjbgVLpjAN5zON0WbTuNtmya4bzOHnKvTWYg2l hXMadv1QgN60kU6+DQ6mlemNxlSbpb5ggyMk/4eOK634poRPyJlzSGdoR4KgzNdkTIaw OxSZFI9R54j7vBFJK/Vfx0aIcFDd+JuTS99hcN2rs0pQ4cX92U460u7nS9knRj19lYOP 29+FLWqg0ucDw15bzQK3Zu6Eimj5DK2EqxnzbswuJVqTmZ6LunO8s0SNQ43DgIvJdEUa 4Umbc52RY9u7Gc4tL8MRvURmMD9dSR/q78LzohSqMQ1mW/Gb4+AShe64QbgXgwNWdASF Hj2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=x5REzk5jnfYN8HQkfIFFSUZC7R4vxhJRzMFl1C8KRK0=; b=bfUvfFwJJLNXxDxN4SixGz3lUKtWLqTesxU5hYIxzUMC8K+ilfxhzGXKUGUvqRVulV 0V5jixgCSRvr6QQBrX9lHb3x0tUL5kzVNpwGJufyvkTOwuuQjqNXkhJDF7l56gh9O7bU AErxjtrugAZclE6zBF/x7UqFFpeBC34Zdi1XeKYNZ67RwlVX6Ev2F9ALwxj1eT7YSe5x 7o/+CCqcwOPaTW5Hagjc8jC9+QK/dhWwIJVLxYzWyOVdX4PtqT+GOC9HdQYJHiidUWMq PanyPH4cIybnzmbOSLQHKQ1Xbc7eNhbKqkwkW58ASLEZp4sELQd083Rjx0bngc6DDP2z eTxw== X-Gm-Message-State: APt69E0Use/nrXu7FqzqQn3JGMeEXikRsa/XUCtjpvtxnVuA3qmCwq1Q eXTi5UgW8t4uCt+kCX6a5/mTbBUxyTTPGHgB9V6qXw== X-Google-Smtp-Source: ADUXVKLTd/LUIh3yAREbCJveTdlrfH0QUlvI+LX8Q7ljLMYMF0ffRawSZVz/8N5Ig4YRq0SVpp/U0UxEKQvAR2IvpFw= X-Received: by 2002:a6b:29c4:: with SMTP id p187-v6mr1294731iop.299.1528823485829; Tue, 12 Jun 2018 10:11:25 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:d028:0:0:0:0:0 with HTTP; Tue, 12 Jun 2018 10:11:25 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <20180612170420.GD56138@raichu> References: <20180612170420.GD56138@raichu> From: Warner Losh Date: Tue, 12 Jun 2018 11:11:25 -0600 X-Google-Sender-Auth: OGJsI99B5ydZcbrr26KtNYGdVns Message-ID: Subject: Re: specifying alignment of loader files To: Mark Johnston Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 17:11:27 -0000 On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston wrote: > Hi, > > I'm writing some code which processes a file loaded by the loader. I > want the file's contents to be available at a certain alignment in > memory, and as far as I can see, the loader provides no alignment > guarantees today. The access will happen early enough during boot that > making an aligned copy of the data will be awkward, so I'd like the > loader to provide the desired alignment. > > I'm considering adding a new "module_align" variable that would specify > the alignment for a given file type, and plumb that through to > command_load(). Does anyone have an alternate suggestion, or an > objection to my proposal? > I thought the loader already did that for ELF sections... Why not wrap your file in such a segment? Warner From owner-freebsd-hackers@freebsd.org Tue Jun 12 17:16:54 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7523310093AD for ; Tue, 12 Jun 2018 17:16:54 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pl0-x233.google.com (mail-pl0-x233.google.com [IPv6:2607:f8b0:400e:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED0246F953 for ; Tue, 12 Jun 2018 17:16:53 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pl0-x233.google.com with SMTP id w17-v6so14740350pll.9 for ; Tue, 12 Jun 2018 10:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=P4z8BAnRlOMGDZdl6+/JYDTp5fSgWkcw39Gl7EylADI=; b=arRznr4i2R5ms6KBEcUUqmbRcJZluCQo+mb/adKLkvXCSgqDIkHchLeuwNYb6rqD8T v1Vlqr/jtRSQyR7ACd5mj+6Mjbaxx1L/FwCRCji8Ex11tZrbWipBRXPofQhSbyAe+O48 DB3DLwsQAx2In0jU4HNUyZWnmi32EPg5/yYt56L/1KZndmmTzmTlFXhwSK09e+kD34bf O2zhleFQbzRcvTjyQ3e5brtiqn8YI6+bZ6kTWK51hcrBysqKL/cvO99Kvx0QAoubpkLN md5Qpt6mUg3LxXtG42MhDpLMmBzdQoLvTuZP3eC+NgfKfSqVLBP29fmPvwmUzSYCeWZK O+HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=P4z8BAnRlOMGDZdl6+/JYDTp5fSgWkcw39Gl7EylADI=; b=YWwGpDsJwMllgcAv3wjGfspN7V2CKHFRqB9QYyaB+4ojXomE4nnhiVmEJe1U1PPkIK utXzbx7B0lJG2L4GUM1sXYQRDF1b2cBKljR8WDjqfcpjZB69lzXspeTN7yDtb+HXUy7u P+E4J6oYl0a45Z99koo4qRTKpjkDr8FKXMPOZiHPM//RzSpzKlpDn5O664JvLx6cB4+t 3Tfii+MSu8qovhi5OdEA3TLZHXTLzIEyujGN3uoFHwh5OsZfSSukco+zifBhQdkJo4Wi R6rQhT3M4OnOzCTrQonmqdIgIfTo8ndsr2K9amVD64gue7BBqPJq4gehrud+ouRUiVoZ oa5g== X-Gm-Message-State: APt69E1fTDLt/EoIcvSxBf1LVH78Q27kmf6kFsrrm7pR9umPyzZDCVpJ YEPAkz9UcKm6wh3fAOOGOlA= X-Google-Smtp-Source: ADUXVKIjVHTwbl3J3bPGoZK+b9OFer2prA0hia/5NY/D/40gF613TWbv9BIxmSStKM1yswhm1gYNAg== X-Received: by 2002:a17:902:2809:: with SMTP id e9-v6mr1327436plb.89.1528823813063; Tue, 12 Jun 2018 10:16:53 -0700 (PDT) Received: from raichu (toroon0560w-lp140-02-70-49-169-156.dsl.bell.ca. [70.49.169.156]) by smtp.gmail.com with ESMTPSA id o66-v6sm899612pfi.157.2018.06.12.10.16.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jun 2018 10:16:52 -0700 (PDT) Sender: Mark Johnston Date: Tue, 12 Jun 2018 13:16:49 -0400 From: Mark Johnston To: Warner Losh Cc: "freebsd-hackers@freebsd.org" Subject: Re: specifying alignment of loader files Message-ID: <20180612171649.GE56138@raichu> References: <20180612170420.GD56138@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 17:16:54 -0000 On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston wrote: > > > Hi, > > > > I'm writing some code which processes a file loaded by the loader. I > > want the file's contents to be available at a certain alignment in > > memory, and as far as I can see, the loader provides no alignment > > guarantees today. The access will happen early enough during boot that > > making an aligned copy of the data will be awkward, so I'd like the > > loader to provide the desired alignment. > > > > I'm considering adding a new "module_align" variable that would specify > > the alignment for a given file type, and plumb that through to > > command_load(). Does anyone have an alternate suggestion, or an > > objection to my proposal? > > > > I thought the loader already did that for ELF sections... Why not wrap your > file in such a segment? In this case it's a raw binary file (CPU microcode), and I want to be able to load it without any modifications or wrappers. From owner-freebsd-hackers@freebsd.org Tue Jun 12 17:45:03 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DB1F100BD83 for ; Tue, 12 Jun 2018 17:45:03 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0291070D0F for ; Tue, 12 Jun 2018 17:45:03 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x231.google.com with SMTP id r24-v6so352980ioh.9 for ; Tue, 12 Jun 2018 10:45:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=PqtB5GT4x1Q8vjRiz/oBrmxBtQmhZJqoIrhT4f7W4u4=; b=kV952aZP1tV5JCG62YGD0vb2z+ELJscj8yx81ogkMUtS3qRtByYVbk/i4Wag1AM22M isrvjW4AsGftCaF+zpu/E8So7Tld6VUH+BVCbObxT8kM0dE7x3Ss5M0IqiYvU9ijeXEE pKeD2wqKH7O+QZFmxkTKbqczUy0xat6vtC+tupXpeMlF8+E51Zi8ls6h+IZSxKfRAB4s xP2oEV8YRF8uxaMXparH9LSZ4+T6J+qGk8t+h6G/KSpOBJ0kZ3Jkw7l+zd9yKXqmUc+f x0HJCSCfNLBegS7O3osMy5DGX7srqXYIC16Ba9/VaPeLMg4uz+ZX6OPC5PwduPsgVjMb HvUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=PqtB5GT4x1Q8vjRiz/oBrmxBtQmhZJqoIrhT4f7W4u4=; b=AK2cfH5uBRee/0LLEYG7/71/1ptVAqUGtNFyKYnhoal2XbSX6MAuu3gPxE6riPY52F Oq5GHhgMk10Taue4pxZIK83zJVUZmXUznqjnSCW3iY6h//CorlK+MH74YqVOO9hOBi1y 8JNLWqDn5XAOy27Nx7i0SmOe0w5a5a1d8VNv564P5K+tHKI2XPJG1X+4faLxvsVz7AuV E9q7oirLQlf76YHaFunF8G5ZBSMhq/3wYhqO9VAIY0vtDjm0CVhjkzQFEMpbZuiXTtyb phqza/q3Vvw53arFh/NwQfBeekhZuR//yL+LAT7ky0ImHA1QBJGoUbO6pCfIgqJlAgEn WDKQ== X-Gm-Message-State: APt69E1He9mniGaVgDu8upVDuDLG+J2znms/2fCpGrz/XmdYQB1/fklK BfWBPB2hTsMSQ7RJhnBwByUJriJtfZJpWaSHngU1pw== X-Google-Smtp-Source: ADUXVKJnv08bJwwATTyZtmZdXMhHXeuZa0alFLLlGlb8lyZaZL9u3FXWaN9+OplAYy8vazJwszxblMErtfzzzZtel/4= X-Received: by 2002:a6b:d40c:: with SMTP id l12-v6mr1491446iog.37.1528825502271; Tue, 12 Jun 2018 10:45:02 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:d028:0:0:0:0:0 with HTTP; Tue, 12 Jun 2018 10:45:01 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <20180612171649.GE56138@raichu> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> From: Warner Losh Date: Tue, 12 Jun 2018 11:45:01 -0600 X-Google-Sender-Auth: t__rGbtHYKRLg0ZMnVEtwfRKsR4 Message-ID: Subject: Re: specifying alignment of loader files To: Mark Johnston Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 17:45:03 -0000 On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston wrote: > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > wrote: > > > > > Hi, > > > > > > I'm writing some code which processes a file loaded by the loader. I > > > want the file's contents to be available at a certain alignment in > > > memory, and as far as I can see, the loader provides no alignment > > > guarantees today. The access will happen early enough during boot that > > > making an aligned copy of the data will be awkward, so I'd like the > > > loader to provide the desired alignment. > > > > > > I'm considering adding a new "module_align" variable that would specify > > > the alignment for a given file type, and plumb that through to > > > command_load(). Does anyone have an alternate suggestion, or an > > > objection to my proposal? > > > > > > > I thought the loader already did that for ELF sections... Why not wrap > your > > file in such a segment? > > In this case it's a raw binary file (CPU microcode), and I want to be > able to load it without any modifications or wrappers. > How do you identify the type then? I'm ok in theory with this (though a variable is more flexible than needed), but that's my main concern. What's the alignment required btw? Warner From owner-freebsd-hackers@freebsd.org Tue Jun 12 17:51:35 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B36B100C855 for ; Tue, 12 Jun 2018 17:51:35 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg0-x236.google.com (mail-pg0-x236.google.com [IPv6:2607:f8b0:400e:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F2BE471389 for ; Tue, 12 Jun 2018 17:51:34 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg0-x236.google.com with SMTP id d2-v6so11805700pga.13 for ; Tue, 12 Jun 2018 10:51:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=bSPn5P++O/94TUXs+seoA2hU69qHc4tX2CW7pj4aVZ0=; b=N6s1RHGZTqIovvH0ZUpNZlC8kt370tT5X6+BlCmvadv7DERMcIdUgz4qXMC2Kg2GC5 FXsUNCZGm0URWC1sdKaoF7xxcLe/j55zsJMTE4d/UfTZfV31OQ/rrCK9KUikNqCXc1RB W1EaOPsngMukfLjdQlCKUVYOXrRXf/oa2oSb7YOOw00GzHTt82o3QKnrq1Pl8736x7kj mFD0aU30E+erx903l260LJ0WMB+RA+3QVbP0h4vbj7brojF+R3KeCkoeqti+7fySUk28 VV+MwJu8aMR+oOCvfmFIlm9y838WEj0A+r7S9oYVM1aM3w+I9DTpNh2cs8V9SacXuNkt cehw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=bSPn5P++O/94TUXs+seoA2hU69qHc4tX2CW7pj4aVZ0=; b=C0Ic83vZAn6mSoWv1HIZ4/qnEZO/6yovP6WWDB0eOh4HoQcjjJnHqN2xC36dzfsa/C Lp4K6BoDSaMo5FVvn+fkp1rYQRzi1B2/pDK219QX+yxX9ZDKXbpyyY81DZ+XHajX1Q2m 6FLGB0Vyia4YD2ENZ1zFORM60PPVf4+3UHPHAJDPT8Fwz7KiTth0uQdSCgjTSjKyZRFn A4t5/+1eO4nEB4vKAXcLJFFmbigN2hv6UPBukQESeQem1hWXhiFRu/0M6M1W1Uh8FZ/G ZziBaFBrjdzMAh8GveIESQUuotEwFOsBViMGzWkUysZL8qaSP/j3/4DG0hxIbqVGmSjg ezaw== X-Gm-Message-State: APt69E2KCBxMv4qsfR25M2iwks7FNgCpMBRU78LTESy2C7EBYcLMu7u4 +3edkiGLWRf6jbPUJEhE1Wg= X-Google-Smtp-Source: ADUXVKLh4mm+n8HsDPT856sPBbz8o52eLhVNM4SB54jmxlYFsZ+G2XoClWazuRsBSwbOuQfPLP1T3g== X-Received: by 2002:a63:8c10:: with SMTP id m16-v6mr1230569pgd.120.1528825894110; Tue, 12 Jun 2018 10:51:34 -0700 (PDT) Received: from raichu (toroon0560w-lp140-02-70-49-169-156.dsl.bell.ca. [70.49.169.156]) by smtp.gmail.com with ESMTPSA id w18-v6sm728921pgq.1.2018.06.12.10.51.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jun 2018 10:51:33 -0700 (PDT) Sender: Mark Johnston Date: Tue, 12 Jun 2018 13:51:30 -0400 From: Mark Johnston To: Warner Losh Cc: "freebsd-hackers@freebsd.org" Subject: Re: specifying alignment of loader files Message-ID: <20180612175130.GF56138@raichu> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 17:51:35 -0000 On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston wrote: > > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > > wrote: > > > > > > > Hi, > > > > > > > > I'm writing some code which processes a file loaded by the loader. I > > > > want the file's contents to be available at a certain alignment in > > > > memory, and as far as I can see, the loader provides no alignment > > > > guarantees today. The access will happen early enough during boot that > > > > making an aligned copy of the data will be awkward, so I'd like the > > > > loader to provide the desired alignment. > > > > > > > > I'm considering adding a new "module_align" variable that would specify > > > > the alignment for a given file type, and plumb that through to > > > > command_load(). Does anyone have an alternate suggestion, or an > > > > objection to my proposal? > > > > > > > > > > I thought the loader already did that for ELF sections... Why not wrap > > your > > > file in such a segment? > > > > In this case it's a raw binary file (CPU microcode), and I want to be > > able to load it without any modifications or wrappers. > > > > How do you identify the type then? I'm ok in theory with this (though a > variable is more flexible than needed), but that's my main concern. What's > the alignment required btw? It'd be a property of the type, e.g., cpu_ucode_name=/boot/firmware/... cpu_ucode_align=16 I do feel like having a separate variable is overkill, but I can't see a less invasive solution that isn't hacky. The required alignment for Intel is 16 bytes; I'm not yet sure whether AMD has a required alignment. From owner-freebsd-hackers@freebsd.org Tue Jun 12 17:55:29 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10B05100D161 for ; Tue, 12 Jun 2018 17:55:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x232.google.com (mail-it0-x232.google.com [IPv6:2607:f8b0:4001:c0b::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 94175718A4 for ; Tue, 12 Jun 2018 17:55:28 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x232.google.com with SMTP id a3-v6so475959itd.0 for ; Tue, 12 Jun 2018 10:55:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=3GAgAkO0lm7PsXM70MvNI1yjrR2bAgaa6pWfJaIen4U=; b=euLI2cx4/i2UKk8sC93MKYeAGqqWX0JxSbIHMGTgvSHKVVhugEyxKXDCbmfp3+8ViJ 2FePqZUpCgo4j6IcB9yVNXsn+cyx9hMs0H8/KlhibJQ9JdOz9qYBvOGdKNNl6ymltq97 35M6R1GAFVbKVpCYE7uaOUZAEenZRXgSDO/vJ3G4wfmf5VDp+PXgINXMGED6pUk1QY7y 2YlCYBiQqQEjRFpqPg2qzy82N+kjj46vByPqW0eW7KQ3JRlzlClCiZ6hlOYTlfwr1FDb j2F8NLdlUTn/6Cn+MY3uT3l4h3D4y9VKEQ8aPsH6prE2NrLkbAlyclu3jrg4fi8yqNS9 SIIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=3GAgAkO0lm7PsXM70MvNI1yjrR2bAgaa6pWfJaIen4U=; b=DyOm1dRFcicHjtLeGorC1mF9IPb2EC+T9ooPF3FLC5vs2oIRr3+bus6yzRBefj073H Y98jkaGyF24kx9MIvPaDb26F9LTm7/kMag8ES9I1f9Anci/PraDCZUXp2NqgQe5PdtqO 32uDZXde9b/J+hrstfiAwpOugD2ZMHa4Ftu8fjR0UpaLIRisk/9zEzv0nMygRzZaPKsv PyN6Uqijq1YV5CtmjDRgxuwhcyl+hJUWxXfe4UMTvIQS/HV1700RgUWY7GrXN20AUW7E dGXGXYEJP14G0G7JbuLOaEnLdJP02Z5Idy1qcXvOTxYZlOhxnYEbntSzZphZ3S9B4hZS c9nA== X-Gm-Message-State: APt69E08Lpr6xo2Fj0kjBgJg6npLD7pklWN8AZZ3kxL+b9Ogu4rWVKpd EbXvHGNLMGixODWWOIrPSRJ/LGdNMHJyKJaPOkChkQ== X-Google-Smtp-Source: ADUXVKJHLcuUH5yVioSeNeG8887k4wZJ0EaZ41gPVyIHr9lQ0U0Inw2u/8uZ5XgofQkcPGBshJk8WixU5gXTdOLKMgc= X-Received: by 2002:a24:5b81:: with SMTP id g123-v6mr1426991itb.1.1528826127927; Tue, 12 Jun 2018 10:55:27 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:d028:0:0:0:0:0 with HTTP; Tue, 12 Jun 2018 10:55:27 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <20180612175130.GF56138@raichu> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> From: Warner Losh Date: Tue, 12 Jun 2018 11:55:27 -0600 X-Google-Sender-Auth: ho6RJA7pB8eLGDwyKLuY2pomoaM Message-ID: Subject: Re: specifying alignment of loader files To: Mark Johnston Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 17:55:29 -0000 On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: > On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston > wrote: > > > > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm writing some code which processes a file loaded by the > loader. I > > > > > want the file's contents to be available at a certain alignment in > > > > > memory, and as far as I can see, the loader provides no alignment > > > > > guarantees today. The access will happen early enough during boot > that > > > > > making an aligned copy of the data will be awkward, so I'd like the > > > > > loader to provide the desired alignment. > > > > > > > > > > I'm considering adding a new "module_align" variable that would > specify > > > > > the alignment for a given file type, and plumb that through to > > > > > command_load(). Does anyone have an alternate suggestion, or an > > > > > objection to my proposal? > > > > > > > > > > > > > I thought the loader already did that for ELF sections... Why not > wrap > > > your > > > > file in such a segment? > > > > > > In this case it's a raw binary file (CPU microcode), and I want to be > > > able to load it without any modifications or wrappers. > > > > > > > How do you identify the type then? I'm ok in theory with this (though a > > variable is more flexible than needed), but that's my main concern. > What's > > the alignment required btw? > > It'd be a property of the type, e.g., > > cpu_ucode_name=/boot/firmware/... > cpu_ucode_align=16 > > I do feel like having a separate variable is overkill, but I can't see > a less invasive solution that isn't hacky. > > The required alignment for Intel is 16 bytes; I'm not yet sure whether > AMD has a required alignment. > OK. That seems sane. I was thinking it was more general than this, but I'm cool with what you've outlined. OTOH, wouldn't just loading all files on page boundaries suffice? Warner From owner-freebsd-hackers@freebsd.org Tue Jun 12 18:00:30 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1BAD100D903 for ; Tue, 12 Jun 2018 18:00:29 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7737271DAA; Tue, 12 Jun 2018 18:00:29 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 141491E64B; Tue, 12 Jun 2018 18:00:29 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-lf0-f48.google.com with SMTP id j13-v6so37386566lfb.13; Tue, 12 Jun 2018 11:00:29 -0700 (PDT) X-Gm-Message-State: APt69E0UJslw+7CUZJSXVyiUl8KvjGhSPNmocOUdU5NmlVzHRu3n7D2L uYG2V8F3PfZ7mOKimdSZegZMydsanWR6zOcus2Y= X-Google-Smtp-Source: ADUXVKI0aN4w6oEb02wU728NmNosKw7/R5DN1XEeSSNsQ+NsXBSRWM+B5u1hQpRSFIMy0FTtRT8XZk8clm3MBtjRyO0= X-Received: by 2002:a2e:1b0a:: with SMTP id b10-v6mr912432ljb.76.1528826427677; Tue, 12 Jun 2018 11:00:27 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:8582:0:0:0:0:0 with HTTP; Tue, 12 Jun 2018 11:00:06 -0700 (PDT) In-Reply-To: References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> From: Kyle Evans Date: Tue, 12 Jun 2018 13:00:06 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: specifying alignment of loader files To: Warner Losh , Mark Johnston Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 18:00:30 -0000 On Tue, Jun 12, 2018 at 12:55 PM, Warner Losh wrote: > On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: > >> On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: >> > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston >> wrote: >> > >> > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: >> > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston >> > > wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > I'm writing some code which processes a file loaded by the >> loader. I >> > > > > want the file's contents to be available at a certain alignment in >> > > > > memory, and as far as I can see, the loader provides no alignment >> > > > > guarantees today. The access will happen early enough during boot >> that >> > > > > making an aligned copy of the data will be awkward, so I'd like the >> > > > > loader to provide the desired alignment. >> > > > > >> > > > > I'm considering adding a new "module_align" variable that would >> specify >> > > > > the alignment for a given file type, and plumb that through to >> > > > > command_load(). Does anyone have an alternate suggestion, or an >> > > > > objection to my proposal? >> > > > > >> > > > >> > > > I thought the loader already did that for ELF sections... Why not >> wrap >> > > your >> > > > file in such a segment? >> > > >> > > In this case it's a raw binary file (CPU microcode), and I want to be >> > > able to load it without any modifications or wrappers. >> > > >> > >> > How do you identify the type then? I'm ok in theory with this (though a >> > variable is more flexible than needed), but that's my main concern. >> What's >> > the alignment required btw? >> >> It'd be a property of the type, e.g., >> >> cpu_ucode_name=/boot/firmware/... >> cpu_ucode_align=16 >> >> I do feel like having a separate variable is overkill, but I can't see >> a less invasive solution that isn't hacky. >> >> The required alignment for Intel is 16 bytes; I'm not yet sure whether >> AMD has a required alignment. >> > > OK. That seems sane. I was thinking it was more general than this, but I'm > cool with what you've outlined. > > OTOH, wouldn't just loading all files on page boundaries suffice? If the answer to the above isn't "yes," can we consider making this a flag to `load` parsed in command_load instead of its own ${module}_align variable? e.g. cpu_ucode_name=/boot/firmware/... cpu_ucode_flags="-a 16" The initial recommendation isn't a bad idea, but adding a new ${module}_flags requires changes in both interpreters that I'm not sure are strictly necessary when this will be relatively rarely used. Thanks, Kyle Evans From owner-freebsd-hackers@freebsd.org Tue Jun 12 18:01:19 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4760100DA64 for ; Tue, 12 Jun 2018 18:01:19 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5F46572024 for ; Tue, 12 Jun 2018 18:01:19 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-lf0-f45.google.com (mail-lf0-f45.google.com [209.85.215.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id F16091E64F for ; Tue, 12 Jun 2018 18:01:18 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-lf0-f45.google.com with SMTP id i83-v6so37395705lfh.5 for ; Tue, 12 Jun 2018 11:01:18 -0700 (PDT) X-Gm-Message-State: APt69E2KpV9isU27XqhALpW0BmIb+B3E86l+O/hCJz/SiVBEz3gMYNew XfAz+SIiDDJtpwbXX/JdHowpxYLYWL8TyefyjNo= X-Received: by 2002:a19:380a:: with SMTP id f10-v6mt929551lfa.47.1528826477477; Tue, 12 Jun 2018 11:01:17 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:8582:0:0:0:0:0 with HTTP; Tue, 12 Jun 2018 11:00:56 -0700 (PDT) In-Reply-To: References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> From: Kyle Evans Date: Tue, 12 Jun 2018 13:00:56 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: specifying alignment of loader files Cc: Warner Losh , Mark Johnston , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 18:01:19 -0000 On Tue, Jun 12, 2018 at 1:00 PM, Kyle Evans wrote: > On Tue, Jun 12, 2018 at 12:55 PM, Warner Losh wrote: >> On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: >> >>> On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: >>> > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston >>> wrote: >>> > >>> > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: >>> > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston >>> > > wrote: >>> > > > >>> > > > > Hi, >>> > > > > >>> > > > > I'm writing some code which processes a file loaded by the >>> loader. I >>> > > > > want the file's contents to be available at a certain alignment in >>> > > > > memory, and as far as I can see, the loader provides no alignment >>> > > > > guarantees today. The access will happen early enough during boot >>> that >>> > > > > making an aligned copy of the data will be awkward, so I'd like the >>> > > > > loader to provide the desired alignment. >>> > > > > >>> > > > > I'm considering adding a new "module_align" variable that would >>> specify >>> > > > > the alignment for a given file type, and plumb that through to >>> > > > > command_load(). Does anyone have an alternate suggestion, or an >>> > > > > objection to my proposal? >>> > > > > >>> > > > >>> > > > I thought the loader already did that for ELF sections... Why not >>> wrap >>> > > your >>> > > > file in such a segment? >>> > > >>> > > In this case it's a raw binary file (CPU microcode), and I want to be >>> > > able to load it without any modifications or wrappers. >>> > > >>> > >>> > How do you identify the type then? I'm ok in theory with this (though a >>> > variable is more flexible than needed), but that's my main concern. >>> What's >>> > the alignment required btw? >>> >>> It'd be a property of the type, e.g., >>> >>> cpu_ucode_name=/boot/firmware/... >>> cpu_ucode_align=16 >>> >>> I do feel like having a separate variable is overkill, but I can't see >>> a less invasive solution that isn't hacky. >>> >>> The required alignment for Intel is 16 bytes; I'm not yet sure whether >>> AMD has a required alignment. >>> >> >> OK. That seems sane. I was thinking it was more general than this, but I'm >> cool with what you've outlined. >> >> OTOH, wouldn't just loading all files on page boundaries suffice? > > If the answer to the above isn't "yes," can we consider making this a > flag to `load` parsed in command_load instead of its own > ${module}_align variable? e.g. > > cpu_ucode_name=/boot/firmware/... > cpu_ucode_flags="-a 16" > > The initial recommendation isn't a bad idea, > but adding a new ${module}_flags requires changes in both interpreters > that I'm not sure are strictly necessary when this will be relatively > rarely used. > > Thanks, > > Kyle Evans Sorry, that ${module}_flags should read ${module}_align. From owner-freebsd-hackers@freebsd.org Tue Jun 12 18:05:49 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADBF6100E25A for ; Tue, 12 Jun 2018 18:05:49 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg0-x232.google.com (mail-pg0-x232.google.com [IPv6:2607:f8b0:400e:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30A6472410 for ; Tue, 12 Jun 2018 18:05:49 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg0-x232.google.com with SMTP id 15-v6so11833694pge.2 for ; Tue, 12 Jun 2018 11:05:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=AQYw2MEWsven4xJ0HSK5q6Gl6xPdAJo69sS/svQkLOA=; b=Cawq43OGjtD2VCb+ZN5QL7YWU4TPD1VuB9KOHmVk16YJcgw0oZ9HFpyCfNk4C2aiKq rQx/F/1cS8HcCEONa3wco4Zd9lybdryvuheEf7M0M1toxLvCztIOX+s2CpngESRJtyVr BQl+GsyHBoA7CPuAbSrC5ov2L53sFNgsABJZ9uHA20OFUGb5+7DmpS7hJqzM3AMyEUb8 HkmVIugYbzO8xvZPmPU7U9oEWZgca39cHWszhZv2Sl7Nb+nHhzSN/JXj5dT6piiuMwSP a6aW8tL+98rcVaJnfoZ1OTPmihdbHMGHGXzrff6TPD/bGhgwQ3ThXlc2ROa1mHYBnEt6 lYfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=AQYw2MEWsven4xJ0HSK5q6Gl6xPdAJo69sS/svQkLOA=; b=Sl20s5LmS9E8maPCtYXmEP/GpfaOAdb5UyMHDVEq/gYiOk7rnvDKVYo+wN2+GDMT6+ /zGmQpsAta1TQrwUkHQt5KRiuDha+OnvVmD5fSYqIHzOWVmB8KMCN51Zvk3BJpPYnas+ DQdaznXm70ARRQOZTQwoTxI+HQdgNn+UNqlujgce92hxZxPH4/BsgMVhnnD5VhFH/yTr hJ0BSlE+THjhqXFjVEQJZElOAPGppukXjnhDVtq1XBMmfu2Sz8IHeVfTEIIAsZ7Nfcpv rjWIoUL7uGAgmeeeZZdnH9q2UN3Sexh331EIiA5776zi39RVWCZnLSJehgSz/iWwQpBf qJkA== X-Gm-Message-State: APt69E0snReMzrqjleIDTcwG27f0mAchnESNnpBd8rmFKAu89U4rZ/ZG wAE96WfC0obEViBRISlViCc= X-Google-Smtp-Source: ADUXVKK6/N8yRibpKlU1+nw5YRXE9IxETMCQdbQNDAsSfccT4GdyhXBAdzbVqQi09EptSihonhAv9A== X-Received: by 2002:a62:8202:: with SMTP id w2-v6mr1519917pfd.32.1528826748183; Tue, 12 Jun 2018 11:05:48 -0700 (PDT) Received: from raichu (toroon0560w-lp140-02-70-49-169-156.dsl.bell.ca. [70.49.169.156]) by smtp.gmail.com with ESMTPSA id 14-v6sm760840pgc.63.2018.06.12.11.05.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jun 2018 11:05:47 -0700 (PDT) Sender: Mark Johnston Date: Tue, 12 Jun 2018 14:05:44 -0400 From: Mark Johnston To: Warner Losh Cc: "freebsd-hackers@freebsd.org" Subject: Re: specifying alignment of loader files Message-ID: <20180612180544.GG56138@raichu> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 18:05:50 -0000 On Tue, Jun 12, 2018 at 11:55:27AM -0600, Warner Losh wrote: > On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: > > > On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > > > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston > > wrote: > > > > > > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm writing some code which processes a file loaded by the > > loader. I > > > > > > want the file's contents to be available at a certain alignment in > > > > > > memory, and as far as I can see, the loader provides no alignment > > > > > > guarantees today. The access will happen early enough during boot > > that > > > > > > making an aligned copy of the data will be awkward, so I'd like the > > > > > > loader to provide the desired alignment. > > > > > > > > > > > > I'm considering adding a new "module_align" variable that would > > specify > > > > > > the alignment for a given file type, and plumb that through to > > > > > > command_load(). Does anyone have an alternate suggestion, or an > > > > > > objection to my proposal? > > > > > > > > > > > > > > > > I thought the loader already did that for ELF sections... Why not > > wrap > > > > your > > > > > file in such a segment? > > > > > > > > In this case it's a raw binary file (CPU microcode), and I want to be > > > > able to load it without any modifications or wrappers. > > > > > > > > > > How do you identify the type then? I'm ok in theory with this (though a > > > variable is more flexible than needed), but that's my main concern. > > What's > > > the alignment required btw? > > > > It'd be a property of the type, e.g., > > > > cpu_ucode_name=/boot/firmware/... > > cpu_ucode_align=16 > > > > I do feel like having a separate variable is overkill, but I can't see > > a less invasive solution that isn't hacky. > > > > The required alignment for Intel is 16 bytes; I'm not yet sure whether > > AMD has a required alignment. > > > > OK. That seems sane. I was thinking it was more general than this, but I'm > cool with what you've outlined. > > OTOH, wouldn't just loading all files on page boundaries suffice? It would (and that's actually what I did in my test branch). We already have to do this for ELF, and the only other frequently loaded file I can think of is /boot/entropy. Can you think of any scenarios where forcing page alignment for all files would waste a substantial amount of memory? If not, I might just go ahead with that approach for now and wait for a good reason before introducing customizable alignments. From owner-freebsd-hackers@freebsd.org Tue Jun 12 18:08:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72A85100E72C for ; Tue, 12 Jun 2018 18:08:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pf0-x243.google.com (mail-pf0-x243.google.com [IPv6:2607:f8b0:400e:c00::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E7CA07260E; Tue, 12 Jun 2018 18:08:50 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pf0-x243.google.com with SMTP id c22-v6so12102982pfi.2; Tue, 12 Jun 2018 11:08:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=b6A/21zxPpl3kS6lAxH6lru4Wxh6O+rcaLrnKiedIGQ=; b=MM/1RnOU2is/9q0m0cC/rnNUTuBOwV7/lZtJyuYn40s0ikCuVEwoZe91qdTpnfKKVZ Fj1klanh1Ouv2lP1gJauIvnai/z37UAXPt1tvXIQuJOuYYlFEK4jjOwa6RVF5qANbZN2 97as0TPUOhO9UNtLNPPs/pAzP/XnwsTEpN45nFjXI2BX+RWN4bBCxILhWFnovUccmAq1 yk2LnpAqNNARjtFiCBCSoAMLg4rbUyn3xSwgLSkuy+kRjchZnfsnrNOif2OutrpaY5iq 0qZWU4Pz2g+AJa+zz93bmTLWqK1uc+zMPcKfTS8b3jlhUmLJYzX+HtvPu8f2L+pd+QQL 2R+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=b6A/21zxPpl3kS6lAxH6lru4Wxh6O+rcaLrnKiedIGQ=; b=On0TpsPOYxPPYvtHaxr71Pl3At6mXviuu/xCrx0MgSiKRj1W2o2fdH70TgjVOmSDwk NBWQhqY50Tddj5+eKfsSh5YIXwhkaBIAcpQ2R90CPoHD2DuroOhsAM0jqY5L9R65cSjK 8X3OV21sIBe1N0sJ6w/MpAj7aZa3IDTsBQ5uNURYVu5q5KIBpZeTdDpqBsvjbyClYOFH Sl8rhRNHES8MV6oi1Y//ETJGE8wfN7xdJc6a+RZOMUej6TaABbC1uUocVpcS0eqW6boY yc93U09uwcp3L/4t1yvj0/VEIQ1pNuCY7HLaWyPJuU/UyC3hw9FH3UiZU1q3rMkHUaIQ wtCQ== X-Gm-Message-State: APt69E2gX8YpmsZxaz3AueQLrBw8NGIkcARov0R6Onv+o01+6uQbDPVr UIKgleMU7QXuxZcxrZ6xzeHz3Q== X-Google-Smtp-Source: ADUXVKJjFAEm2JeqpAwD9Df0iC5oWwGuvuVdjVfNKpVah/eNIsodJn6wqGBFjH7m6DlPis8xq2QT0g== X-Received: by 2002:a65:60d2:: with SMTP id r18-v6mr1254701pgv.306.1528826929703; Tue, 12 Jun 2018 11:08:49 -0700 (PDT) Received: from raichu (toroon0560w-lp140-02-70-49-169-156.dsl.bell.ca. [70.49.169.156]) by smtp.gmail.com with ESMTPSA id x17-v6sm738923pgv.15.2018.06.12.11.08.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jun 2018 11:08:48 -0700 (PDT) Sender: Mark Johnston Date: Tue, 12 Jun 2018 14:08:46 -0400 From: Mark Johnston To: Kyle Evans Cc: Warner Losh , "freebsd-hackers@freebsd.org" Subject: Re: specifying alignment of loader files Message-ID: <20180612180846.GH56138@raichu> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 18:08:51 -0000 On Tue, Jun 12, 2018 at 01:00:06PM -0500, Kyle Evans wrote: > On Tue, Jun 12, 2018 at 12:55 PM, Warner Losh wrote: > > On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: > > > >> On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > >> > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston > >> wrote: > >> > > >> > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > >> > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > >> > > wrote: > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > I'm writing some code which processes a file loaded by the > >> loader. I > >> > > > > want the file's contents to be available at a certain alignment in > >> > > > > memory, and as far as I can see, the loader provides no alignment > >> > > > > guarantees today. The access will happen early enough during boot > >> that > >> > > > > making an aligned copy of the data will be awkward, so I'd like the > >> > > > > loader to provide the desired alignment. > >> > > > > > >> > > > > I'm considering adding a new "module_align" variable that would > >> specify > >> > > > > the alignment for a given file type, and plumb that through to > >> > > > > command_load(). Does anyone have an alternate suggestion, or an > >> > > > > objection to my proposal? > >> > > > > > >> > > > > >> > > > I thought the loader already did that for ELF sections... Why not > >> wrap > >> > > your > >> > > > file in such a segment? > >> > > > >> > > In this case it's a raw binary file (CPU microcode), and I want to be > >> > > able to load it without any modifications or wrappers. > >> > > > >> > > >> > How do you identify the type then? I'm ok in theory with this (though a > >> > variable is more flexible than needed), but that's my main concern. > >> What's > >> > the alignment required btw? > >> > >> It'd be a property of the type, e.g., > >> > >> cpu_ucode_name=/boot/firmware/... > >> cpu_ucode_align=16 > >> > >> I do feel like having a separate variable is overkill, but I can't see > >> a less invasive solution that isn't hacky. > >> > >> The required alignment for Intel is 16 bytes; I'm not yet sure whether > >> AMD has a required alignment. > >> > > > > OK. That seems sane. I was thinking it was more general than this, but I'm > > cool with what you've outlined. > > > > OTOH, wouldn't just loading all files on page boundaries suffice? > > If the answer to the above isn't "yes," can we consider making this a > flag to `load` parsed in command_load instead of its own > ${module}_align variable? e.g. > > cpu_ucode_name=/boot/firmware/... > cpu_ucode_flags="-a 16" > > The initial recommendation isn't a bad idea, > but adding a new ${module}_flags requires changes in both interpreters > that I'm not sure are strictly necessary when this will be relatively > rarely used. Oh, I didn't realize those flags get passed straight to command_load(). I think that's a better idea than what I suggested. I started this thread when I realized that I would need to write some Forth code to implement my idea. :) From owner-freebsd-hackers@freebsd.org Tue Jun 12 18:10:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04FD4100E9E2 for ; Tue, 12 Jun 2018 18:10:18 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 43C39727AF; Tue, 12 Jun 2018 18:10:16 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001) id E188F5A9F12; Tue, 12 Jun 2018 18:10:09 +0000 (UTC) Date: Tue, 12 Jun 2018 18:10:09 +0000 From: Brooks Davis To: Warner Losh Cc: Mark Johnston , "freebsd-hackers@freebsd.org" Subject: Re: specifying alignment of loader files Message-ID: <20180612181009.GA11238@spindle.one-eyed-alien.net> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OXfL5xGRrasGEqWY" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 18:10:18 -0000 --OXfL5xGRrasGEqWY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 12, 2018 at 11:55:27AM -0600, Warner Losh wrote: > On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: >=20 > > On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > > > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston > > wrote: > > > > > > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm writing some code which processes a file loaded by the > > loader. I > > > > > > want the file's contents to be available at a certain alignment= in > > > > > > memory, and as far as I can see, the loader provides no alignme= nt > > > > > > guarantees today. The access will happen early enough during b= oot > > that > > > > > > making an aligned copy of the data will be awkward, so I'd like= the > > > > > > loader to provide the desired alignment. > > > > > > > > > > > > I'm considering adding a new "module_align" variable that would > > specify > > > > > > the alignment for a given file type, and plumb that through to > > > > > > command_load(). Does anyone have an alternate suggestion, or an > > > > > > objection to my proposal? > > > > > > > > > > > > > > > > I thought the loader already did that for ELF sections... Why not > > wrap > > > > your > > > > > file in such a segment? > > > > > > > > In this case it's a raw binary file (CPU microcode), and I want to = be > > > > able to load it without any modifications or wrappers. > > > > > > > > > > How do you identify the type then? I'm ok in theory with this (though= a > > > variable is more flexible than needed), but that's my main concern. > > What's > > > the alignment required btw? > > > > It'd be a property of the type, e.g., > > > > cpu_ucode_name=3D/boot/firmware/... > > cpu_ucode_align=3D16 > > > > I do feel like having a separate variable is overkill, but I can't see > > a less invasive solution that isn't hacky. > > > > The required alignment for Intel is 16 bytes; I'm not yet sure whether > > AMD has a required alignment. > > >=20 > OK. That seems sane. I was thinking it was more general than this, but I'm > cool with what you've outlined. >=20 > OTOH, wouldn't just loading all files on page boundaries suffice? Page boundaries might be too much on very small systems, but there's a principled argument for 8-bytes just based on the aligment of primative types so 16 (or even 64) isn't much of a stretch. 16-bytes would future-proof for the CHERI variants that are likely to make it to silicon. -- Brooks --OXfL5xGRrasGEqWY Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJbIAyBAAoJEKzQXbSebgfApy4H/0i7/xXZV4ukxJ9QxcGZ07TP 1l25jQI5Kd0l96ERipNCZjHjnQx4/61PToVwQpx98NlIIKfYEAfLr1UxYPEUeFk6 90v/+Lvh8RV47MKXgPpv/qoRyg0R6NBUoOx//OW1/Qfz6qRwm48I5eilwQpikpAf SeiQBSW7W5dyQveh68HNox3keg0s+QeB9IXx34Neqfb4xHCcWrPgskmybkurZjXu V9gfmFcf9Uccs4XJ0qw694VH+tHiGz0x8j8JrtUmomglbVExxdfJzmhXaJw3H0G2 A57UcJIrCG5uyGuqjDrbOft/0q5zUCvrKeELHZmSyLccSP+xhcQ4tZo9vOVGZjg= =ZGER -----END PGP SIGNATURE----- --OXfL5xGRrasGEqWY-- From owner-freebsd-hackers@freebsd.org Tue Jun 12 18:13:27 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94164100F026 for ; Tue, 12 Jun 2018 18:13:27 +0000 (UTC) (envelope-from ben@bwidawsk.net) Received: from mail.bwidawsk.net (zangief.bwidawsk.net [107.170.211.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "bwidawsk.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 13B0D72CE6; Tue, 12 Jun 2018 18:13:27 +0000 (UTC) (envelope-from ben@bwidawsk.net) Received: by mail.bwidawsk.net (Postfix, from userid 5001) id 06947122809; Tue, 12 Jun 2018 11:13:19 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on zangief.bwidawsk.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-1.0 required=4.1 tests=ALL_TRUSTED=-1 shortcircuit=no autolearn=no autolearn_force=no version=3.4.1 Received: from mail.bwidawsk.net (unknown [134.134.139.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail.bwidawsk.net (Postfix) with ESMTPSA id 18E2F122547; Tue, 12 Jun 2018 11:13:17 -0700 (PDT) Date: Tue, 12 Jun 2018 11:13:16 -0700 From: Ben Widawsky To: Brooks Davis Cc: Warner Losh , "freebsd-hackers@freebsd.org" , Mark Johnston Subject: Re: specifying alignment of loader files Message-ID: <20180612181316.e67bv44exkvhmtyv@mail.bwidawsk.net> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> <20180612181009.GA11238@spindle.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180612181009.GA11238@spindle.one-eyed-alien.net> User-Agent: NeoMutt/20180323 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jun 2018 18:13:27 -0000 On 18-06-12 18:10:09, Brooks Davis wrote: > On Tue, Jun 12, 2018 at 11:55:27AM -0600, Warner Losh wrote: > > On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston wrote: > > > > > On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > > > > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston > > > wrote: > > > > > > > > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > > > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm writing some code which processes a file loaded by the > > > loader. I > > > > > > > want the file's contents to be available at a certain alignment in > > > > > > > memory, and as far as I can see, the loader provides no alignment > > > > > > > guarantees today. The access will happen early enough during boot > > > that > > > > > > > making an aligned copy of the data will be awkward, so I'd like the > > > > > > > loader to provide the desired alignment. > > > > > > > > > > > > > > I'm considering adding a new "module_align" variable that would > > > specify > > > > > > > the alignment for a given file type, and plumb that through to > > > > > > > command_load(). Does anyone have an alternate suggestion, or an > > > > > > > objection to my proposal? > > > > > > > > > > > > > > > > > > > I thought the loader already did that for ELF sections... Why not > > > wrap > > > > > your > > > > > > file in such a segment? > > > > > > > > > > In this case it's a raw binary file (CPU microcode), and I want to be > > > > > able to load it without any modifications or wrappers. > > > > > > > > > > > > > How do you identify the type then? I'm ok in theory with this (though a > > > > variable is more flexible than needed), but that's my main concern. > > > What's > > > > the alignment required btw? > > > > > > It'd be a property of the type, e.g., > > > > > > cpu_ucode_name=/boot/firmware/... > > > cpu_ucode_align=16 > > > > > > I do feel like having a separate variable is overkill, but I can't see > > > a less invasive solution that isn't hacky. > > > > > > The required alignment for Intel is 16 bytes; I'm not yet sure whether > > > AMD has a required alignment. > > > > > > > OK. That seems sane. I was thinking it was more general than this, but I'm > > cool with what you've outlined. > > > > OTOH, wouldn't just loading all files on page boundaries suffice? > > Page boundaries might be too much on very small systems, but there's > a principled argument for 8-bytes just based on the aligment of > primative types so 16 (or even 64) isn't much of a stretch. 16-bytes > would future-proof for the CHERI variants that are likely to make > it to silicon. > > -- Brooks I think native cacheline size is a pretty safe yet more conservative than page size bet ie. me too for 64. From owner-freebsd-hackers@freebsd.org Wed Jun 13 10:29:39 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 615AD101D820 for ; Wed, 13 Jun 2018 10:29:39 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-yb0-x229.google.com (mail-yb0-x229.google.com [IPv6:2607:f8b0:4002:c09::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 027F9799CB for ; Wed, 13 Jun 2018 10:29:38 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: by mail-yb0-x229.google.com with SMTP id x36-v6so727423ybi.0 for ; Wed, 13 Jun 2018 03:29:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=7qlUGokNvB82NoRe8/Oe8lGhDPG0ZnR5C9TXEsuZ5pI=; b=Ip1G0EluKIu4SUBhDBTxdcxsmiMuTgnILC5OT6iL7zMMkaZGwfDIg16D79ny7brUAO x7//hxA6HIJ3tH/k+BhRgtP8uleAgBX84/Uz9orWSSd7KwnqMlrEdxvFcgLFOzdn8v7o jfNgKSLWUh/CuOaCIeyYBTe1ZvWwUYMiCtaCo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=7qlUGokNvB82NoRe8/Oe8lGhDPG0ZnR5C9TXEsuZ5pI=; b=QYB+7r5Uf7bRoF+3ZiivT9pCRTdqS0LUkh4owEZ0YxCPqersMA5m7CyaQa+rvrffUv nijl5eaGOHmOf4MSr04hA/H3mZgFD4RK75xhtgYrC8sgWTB9uylE6UpCArvTFCSBt1qD 1eGcg6pVgRBCkbs458jAJHz2nPbz7epYaPY5VBb/++byedBcB1Q6PteGJ18Vv3yMVMOZ BIb24gFUFn4uqedyVIwSzx41+DfthietuXxXdAOEHcjKIen3jL5q1gxhBtULOIfqiE2K bKdWVj2/CdWIN3QlKravJnrAycBWHK0iReaWcEJn7XNVq8iokl6/kfil3k8vsfHuEgLC 45VQ== X-Gm-Message-State: APt69E3PUtKkG21c+rV1h0J2vHfN6sd4tB41yOkm/VoXgwx24GOB9yhy G+4pTrMAV6+UviOcKpaWRhgX5/UpQLPfC3VLY8BFBz8g X-Google-Smtp-Source: ADUXVKJf0FzCGsK8/zxY8F8O/Q81PKpkt01jHAnMVzES1IdasQuk0rT4rUROoS5GdqhwEvZ0fa8DzYP/HdoarkjZrdA= X-Received: by 2002:a25:bd41:: with SMTP id p1-v6mr1918510ybm.69.1528885778251; Wed, 13 Jun 2018 03:29:38 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a25:ef50:0:0:0:0:0 with HTTP; Wed, 13 Jun 2018 03:29:07 -0700 (PDT) In-Reply-To: References: <20180317173815.GM75576@funkthat.com> From: Eitan Adler Date: Wed, 13 Jun 2018 03:29:07 -0700 Message-ID: Subject: Re: gf128_add can be marked as __pure2 To: Eitan Adler , John Baldwin , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jun 2018 10:29:39 -0000 On 17 March 2018 at 11:38, Eitan Adler wrote: > On 17 March 2018 at 10:38, John-Mark Gurney wrote: >> Eitan Adler wrote this message on Mon, Feb 19, 2018 at 19:15 -0800: >>> Is there any reason not to apply this patch? >> >> I don't see why there wouldn't be.. >> >>> __pure2 means __attribute__((const)) which is correct in this case as >>> gf128_add read no global memory: For extra clarity, since this is crypto related, can you please accept here: https://reviews.freebsd.org/D15786 -- Eitan Adler From owner-freebsd-hackers@freebsd.org Wed Jun 13 13:46:15 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC5E3100CACA for ; Wed, 13 Jun 2018 13:46:14 +0000 (UTC) (envelope-from ed@nuxi.nl) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7471283F1B for ; Wed, 13 Jun 2018 13:46:14 +0000 (UTC) (envelope-from ed@nuxi.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 352A2100CAC0; Wed, 13 Jun 2018 13:46:14 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 228EC100CABE for ; Wed, 13 Jun 2018 13:46:14 +0000 (UTC) (envelope-from ed@nuxi.nl) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 86C9483F18 for ; Wed, 13 Jun 2018 13:46:12 +0000 (UTC) (envelope-from ed@nuxi.nl) Received: by mail-wm0-x22f.google.com with SMTP id p11-v6so5527414wmc.4 for ; Wed, 13 Jun 2018 06:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nuxi-nl.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=/lCr2brtWu3Oj0J5GWvM2AzXAwFvt7Kfl2N10f/bRqc=; b=KqrHQkwT6utX5P2Zy/vBZ9GmG3l48EpaTU88MB3qwG+ozfyNsU4K9AKRhu1GPpYw70 1ZdyQbVltur0akRX/awf5+arvPvYDf//zPg9Se/ZxB+avkJzBeS3NgWS6plzQth7qxE/ EULqc2hOv80fr42boBgGx/aJIh0+3+EQBuuHB8vDuu+98fxef52+q1mW5C1Ds/8LtENQ GAorwkd0Jt6YaXP6NKRnhe+JStwRZONqiKKtLvbU8VsoY15j9l8EjB+NrTkG+0LYdnKk GEiqVuzyPmW/KsWE/dZNTYy72XTamA7dKBHtDxymZttUY+v+eOc0IUKdMnOPESYAImJZ G7UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=/lCr2brtWu3Oj0J5GWvM2AzXAwFvt7Kfl2N10f/bRqc=; b=m9g8c21plaSCK+qbjQlSCDHvwD3Xy6qUMQOOdfWnpvMPQLmo7svs4iJQQih3EGwwDn iyXQ+WmhZcXJHqCfEmYkyJlxWYTjRtVPX4UecCbk4LXGg3epZN9VdbyHAj7Jcn8cM7La 0Q2JbrO7C56M+5labRW/ZxwMWMHu0s5shPnpfgsReo1qhTwH5sPQz7Fip1jVkajHOmeJ u9fdJTyrqvtNPUKZ7yv0TG8llPhTatDFhVcx7kNhZ3EoO0xwHeyez1YafsR1XoaDM272 XXWAVUiCqwXVuxWr0BhS43RP1RI5Lvfmz1HH4Txj3IKu9hRxI+3lRdXVllFiuLkBi1rn QiAQ== X-Gm-Message-State: APt69E1s2Nt0lJIQPAxSSptWFLemKoBVhzv9M+LsR1rh8RjVbqcnm45g 5bKNKl5NV2Di7SXDv55ozy/JrD0iNwW2H61dEZEkOQ== X-Google-Smtp-Source: ADUXVKJl+8KPbTmLxdgqiQvptjiYnXj0vh3fQ8UkLHhNjkkPLwq5waqBZ04BR9fjmUw7Z1ytvBef7uoG6fs4vfxSDmE= X-Received: by 2002:aa7:c542:: with SMTP id s2-v6mr4175686edr.224.1528897571204; Wed, 13 Jun 2018 06:46:11 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a50:8e0d:0:0:0:0:0 with HTTP; Wed, 13 Jun 2018 06:45:40 -0700 (PDT) In-Reply-To: References: From: Ed Schouten Date: Wed, 13 Jun 2018 15:45:40 +0200 Message-ID: Subject: Re: Cleanest way of merging back changes to syslogd(8)? To: Dimitry Andric Cc: hackers@freebsd.org, Dave Cottlehuber Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jun 2018 13:46:15 -0000 Hi Dimitry, 2018-06-06 11:21 GMT+02:00 Dimitry Andric : > The svn:mergeinfo property on the usr.sbin/syslogd directory will have > to be moved to the root of the stable/11 tree, I'm not sure off the top > of my head how that should be done. I did some searching around and it turns out that "svn mergeinfo --show-revs eligible" can be used to get a list of commits that "svn merge" would in fact try to merge back. By then running "svn merge" at the root of the repository with that list of commits, I was able to merge this back in such a way that the mergeinfo remains at the top of the tree. I've completed the merge in r335059. Thanks for your input Dimitry! Best regards, -- Ed Schouten Nuxi, 's-Hertogenbosch, the Netherlands From owner-freebsd-hackers@freebsd.org Wed Jun 13 16:26:18 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0341E101ACB8 for ; Wed, 13 Jun 2018 16:26:18 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C84C6C31C for ; Wed, 13 Jun 2018 16:26:17 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-RoutePath: aGlwcGll X-MHO-User: 38d538b5-6f24-11e8-afd2-4ddcc8249dd4 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound1.eu.mailhop.org (Halon) with ESMTPSA id 38d538b5-6f24-11e8-afd2-4ddcc8249dd4; Wed, 13 Jun 2018 16:10:01 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w5DG9xqI060862; Wed, 13 Jun 2018 10:09:59 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1528906199.12122.64.camel@freebsd.org> Subject: Re: specifying alignment of loader files From: Ian Lepore To: Mark Johnston , Warner Losh Cc: "freebsd-hackers@freebsd.org" Date: Wed, 13 Jun 2018 10:09:59 -0600 In-Reply-To: <20180612180544.GG56138@raichu> References: <20180612170420.GD56138@raichu> <20180612171649.GE56138@raichu> <20180612175130.GF56138@raichu> <20180612180544.GG56138@raichu> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jun 2018 16:26:18 -0000 On Tue, 2018-06-12 at 14:05 -0400, Mark Johnston wrote: > On Tue, Jun 12, 2018 at 11:55:27AM -0600, Warner Losh wrote: > > > > On Tue, Jun 12, 2018 at 11:51 AM, Mark Johnston > > wrote: > > > > > > > > On Tue, Jun 12, 2018 at 11:45:01AM -0600, Warner Losh wrote: > > > > > > > > On Tue, Jun 12, 2018 at 11:16 AM, Mark Johnston > > > org> > > > wrote: > > > > > > > > > > > > > > > > > > On Tue, Jun 12, 2018 at 11:11:25AM -0600, Warner Losh wrote: > > > > > > > > > > > > On Tue, Jun 12, 2018 at 11:04 AM, Mark Johnston > > > > > bsd.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm writing some code which processes a file loaded by > > > > > > > the > > > loader.  I > > > > > > > > > > > > > > > > > > > > > > > > > > > > > want the file's contents to be available at a certain > > > > > > > alignment in > > > > > > > memory, and as far as I can see, the loader provides no > > > > > > > alignment > > > > > > > guarantees today.  The access will happen early enough > > > > > > > during boot > > > that > > > > > > > > > > > > > > > > > > > > > > > > > > > > > making an aligned copy of the data will be awkward, so > > > > > > > I'd like the > > > > > > > loader to provide the desired alignment. > > > > > > > > > > > > > > I'm considering adding a new "module_align" variable that > > > > > > > would > > > specify > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the alignment for a given file type, and plumb that > > > > > > > through to > > > > > > > command_load().  Does anyone have an alternate > > > > > > > suggestion, or an > > > > > > > objection to my proposal? > > > > > > > > > > > > > I thought the loader already did that for ELF sections... > > > > > > Why not > > > wrap > > > > > > > > > > > > > > your > > > > > > > > > > > > file in such a segment? > > > > > In this case it's a raw binary file (CPU microcode), and I > > > > > want to be > > > > > able to load it without any modifications or wrappers. > > > > > > > > > How do you identify the type then? I'm ok in theory with this > > > > (though a > > > > variable is more flexible than needed), but that's my main > > > > concern. > > > What's > > > > > > > > the alignment required btw? > > > It'd be a property of the type, e.g., > > > > > > cpu_ucode_name=/boot/firmware/... > > > cpu_ucode_align=16 > > > > > > I do feel like having a separate variable is overkill, but I > > > can't see > > > a less invasive solution that isn't hacky. > > > > > > The required alignment for Intel is 16 bytes; I'm not yet sure > > > whether > > > AMD has a required alignment. > > > > > OK. That seems sane. I was thinking it was more general than this, > > but I'm > > cool with what you've outlined. > > > > OTOH, wouldn't just loading all files on page boundaries suffice? > It would (and that's actually what I did in my test branch).  We > already have to do this for ELF, and the only other frequently loaded > file I can think of is /boot/entropy.  Can you think of any scenarios > where forcing page alignment for all files would waste a substantial > amount of memory?  If not, I might just go ahead with that approach > for > now and wait for a good reason before introducing customizable > alignments. There is already an arch_loadaddr() method in the archsw which is used to align every loaded module/file/whatever. I think not all arches implement it, but they probably should, and each arch should probably use the biggest possible cacheline size for that arch as the minimum alignment. For some arches, page boundaries might make sense.  A nice example is the beri implementation which does this:   /* Align ELF objects at page boundaries; others at cache lines. */   align = (type == LOAD_ELF) ? PAGE_SIZE : CACHE_LINE_SIZE;   return (roundup2(addr, align)); For a nice description of how the function is intended to work, see the comment block above uboot_loadaddr() in stand/uboot/lib/copy.c. -- Ian From owner-freebsd-hackers@freebsd.org Thu Jun 14 08:59:40 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 947F81012D32 for ; Thu, 14 Jun 2018 08:59:40 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 03B2E7E84E; Thu, 14 Jun 2018 08:59:40 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: by mail-wm0-x243.google.com with SMTP id z6-v6so2308000wma.0; Thu, 14 Jun 2018 01:59:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=D9fEalTpjLx95ussHzDr0CCH8SzcMjcx7cqxc9ksAPE=; b=Lue9rpDvkNn/0Ixj+ecQOlkoyRtR4vvF1MHWecyb4brZrvJKZuIT+4ypkGXkOnRttE LCjoPTqRkcwn3HP1ZAxHKOep7IcF741TQ78HyPR8L2JSBRZye/fDrtXT+PXMHocS8uRx IcieJuYfMwqPbzGJNCiRm2sBM/Pw23kMqUD8Xxgap8XVumYWrjio+a3AOh7JCWXj//6+ 6cA5RiyCmEpTpWWHXkE7otELELHD7fB5Uh5AQ9VkJuqNXy+YBBDGeG0TskufV6w46JWa GObvpsv2f0LIoU5jOPucdioLW2kBVeiUPt87/TCI1ViyRT3NlJm1ZwopT0glVSqdej6D XhVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=D9fEalTpjLx95ussHzDr0CCH8SzcMjcx7cqxc9ksAPE=; b=i6Q9w7gIeIpDAIYY3ygdABiJxQMT1/YUqsuvyBb1fp5RVXLFj9wq6y7I7QP64+CBfI 9xdAKVX0XTlR7oWWgmKO2ZGWARyFcD8c1DIkyv3q6NQ51+FcpwSrXS/RFAtOsnRASik1 4ZjTxTafVWDvy1ZUAZPtTa3XHgRrrVSaGjkGf4TkuSMLKTsUtf9Y4XYNj8vTGYeLKlG0 HVMDKHtieply3Q+7D4R6H6l4cq0MXVSrTLP06YlbPBaFy/gcTMs/qnmr48RualCHDZc2 KlOxoN6pmFCo21wqUcjJGkvvbTv/uTdWESuwU1uaX0uUNjFinai6y3zIpjtE8I8oBCZO f65g== X-Gm-Message-State: APt69E0ukZjOl8Ck7SDny+D/LM+C5qlZdhV1EiyQid5RXr9LXD5z3I5f gkpB5mIarea/lNP32KyrkAVXqrU7HSZySKZWQJbzP3ll X-Google-Smtp-Source: ADUXVKJUk1NRh7yBehy36lNXGWCbDnhjnu0lFuJgDZMUpaR8MTEDBgWNX5P+sMUPvomSbFDA7MoUvyD/xVKDnTAhQWo= X-Received: by 2002:a50:ad78:: with SMTP id z53-v6mr1907754edc.306.1528966778764; Thu, 14 Jun 2018 01:59:38 -0700 (PDT) MIME-Version: 1.0 From: Pratyush Yadav Date: Thu, 14 Jun 2018 14:29:02 +0530 Message-ID: Subject: Bug in subr_bus_dma.c's bus_dmamap_load() when multiple dma maps are created from a single tag on x86 To: freebsd-hackers@freebsd.org Cc: Edward Napierala , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jun 2018 08:59:40 -0000 Hi everyone, I was looking at the busdma code for x86 and I notice that the claim in the bus_dma(9) man page might not be correct: "Multiple maps can be associated with one DMA tag." Looking at the code of sys/x86/x86/busdma_bounce.c (which I believe is the default busdma implementation for x86), I realize that a map does not necessarily use all the segments of the dma tag (specified by the parameter nsegments on tag creation). So how many segments does a map actually use? Looking at busdma_bounce.c:644,690 it seems we can calculate that from the pointer segp. It contains the index of starting segment on entrace to the load function, and the ending segment on exit. Sounds all fine so far. But then if we look at map_complete() (busdma_bounce.c:908), it returns the entire segments array of the dma *tag*. I emphasize tag because the tag's segments array holds the segments of all the maps created with the tag. Going a layer up, sys/kern/subr_bus_dma.c:328 simply passes nsegs(= -1) in the segp parameter to load, not bothering to check if some segments have been used already by other maps. It then calculates the number of segments used using the value of nsegs after return and calls map_complete() to get the segments array. It then passes the segments array to the driver callback with the number of segments, nsegs. Effectively, to the driver it seems that the map's segments start from index 0 and go till nsegs - 1. This is not true if another map has been loaded in the meantime with the same tag. We are possibly over-writing the previous map's segments. busdma_bounce.c:669 simply uses segs[0] if the value of segp is -1, which is what bus_dmamap_load() in subr_bus_dma.c:328 passes to _bus_dmamap_load_buffer(). segs[0] would also be used by any subsequent map's load. This means the previous map's values are over-written. This might cause problems when the driver is in its callback using the segments array and another map is loaded. Unfortunately, I do not posses the knowledge or experience with FreeBSD's drivers (drivers in general) to test for this bug. I have not looked at the code in other architectures. The same problem might present itself there as well. If I made a mistake somewhere in my reasoning, let me know. -- Regards, Pratyush Yadav From owner-freebsd-hackers@freebsd.org Thu Jun 14 10:53:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5D82101BF41 for ; Thu, 14 Jun 2018 10:53:24 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6779283AEA; Thu, 14 Jun 2018 10:53:24 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 67236260199; Thu, 14 Jun 2018 12:53:22 +0200 (CEST) Subject: Re: Bug in subr_bus_dma.c's bus_dmamap_load() when multiple dma maps are created from a single tag on x86 To: Pratyush Yadav , freebsd-hackers@freebsd.org Cc: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , Edward Napierala References: From: Hans Petter Selasky Message-ID: <8170cb61-8c85-3dc8-1e16-bdb6a2ada7db@selasky.org> Date: Thu, 14 Jun 2018 12:53:06 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jun 2018 10:53:25 -0000 On 06/14/18 10:59, Pratyush Yadav wrote: > Hi everyone, > > I was looking at the busdma code for x86 and I notice that the claim > in the bus_dma(9) man page might not be correct: "Multiple maps can be > associated with one DMA tag." > > Looking at the code of sys/x86/x86/busdma_bounce.c (which I believe is > the default busdma implementation for x86), I realize that a map does > not necessarily use all the segments of the dma tag (specified by the > parameter nsegments on tag creation). > > So how many segments does a map actually use? Looking at > busdma_bounce.c:644,690 it seems we can calculate that from the > pointer segp. It contains the index of starting segment on entrace to > the load function, and the ending segment on exit. > > Sounds all fine so far. But then if we look at map_complete() > (busdma_bounce.c:908), it returns the entire segments array of the dma > *tag*. I emphasize tag because the tag's segments array holds the > segments of all the maps created with the tag. > > Going a layer up, sys/kern/subr_bus_dma.c:328 simply passes nsegs(= > -1) in the segp parameter to load, not bothering to check if some > segments have been used already by other maps. It then calculates the > number of segments used using the value of nsegs after return and > calls map_complete() to get the segments array. It then passes the > segments array to the driver callback with the number of segments, > nsegs. > > Effectively, to the driver it seems that the map's segments start from > index 0 and go till nsegs - 1. This is not true if another map has > been loaded in the meantime with the same tag. > > We are possibly over-writing the previous map's segments. > busdma_bounce.c:669 simply uses segs[0] if the value of segp is -1, > which is what bus_dmamap_load() in subr_bus_dma.c:328 passes to > _bus_dmamap_load_buffer(). segs[0] would also be used by any > subsequent map's load. This means the previous map's values are > over-written. > > This might cause problems when the driver is in its callback using the > segments array and another map is loaded. > > Unfortunately, I do not posses the knowledge or experience with > FreeBSD's drivers (drivers in general) to test for this bug. > > I have not looked at the code in other architectures. The same problem > might present itself there as well. > > If I made a mistake somewhere in my reasoning, let me know. Hi, Use of bus_dmamap_load() on the same tag must be serialized using a mutex. Maybe it is not so clearly documented. Else like you see the temporary segs[] list may be overwritten. --HPS From owner-freebsd-hackers@freebsd.org Thu Jun 14 11:36:50 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3805B101F9A4 for ; Thu, 14 Jun 2018 11:36:50 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9419D85809; Thu, 14 Jun 2018 11:36:49 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: by mail-wm0-x242.google.com with SMTP id p126-v6so9997419wmb.2; Thu, 14 Jun 2018 04:36:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HFp8rPvNIHboXcg99GxHGX4sRkszBw1x94RGaVisS7Q=; b=Y3Oef10nrBmzIXb3XzNYMZ4LYK2ZPNmcF1l7D0jiCPelhwhCVxOATQu2R6duUwTsKV SFuTTFCw686t6A7qhJMM6iExF/G4P1YKy19C2TqZWsXIGPXHMQaS0yyr0MmBGMpgTDAH geiV/SK8QLynTpyZaJOVPoH835Nk0w//03nKGiujV8WoVWvNirnGlrS4VcKqmFTXbmEe 6bAxAFi/vB89s2rpbRkLgofXaW9lWWEsiiCN9HGnIBq60+VvEfsW45qQ4Fxz1/NsGMcB ps4sFVQmyKixTQ2ohyw0lQ/7ZcmONvLrxJLiMepXc2FmBWhA9aI6fUb16tKbecD6Uafz neOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HFp8rPvNIHboXcg99GxHGX4sRkszBw1x94RGaVisS7Q=; b=P++sJAIv6QFS+K37Y+i1iuMSEByvdbrrzXY1A1g/FbBiJdrOso2xePADw6S9vOfuux DHCWkisijAwc1XNOSfkenfH44oAp/JuZ3ve2H4IMt3nEuHxQZukyooL+tTPWAuBy2JXd xcCp8rQyv1Sw4dnfNDpuwi8jQ8I6OVdn9twpj91ePRZgITc8pwy53InP2mu8nK1uFMh6 GcBcw+vIXDZV1NnchFb3351VFtKcdujamTIj/WLiyw+6g43ceK6IGu0kqT3ZoOcHwG0i bl1irUF3uyA3WedeHKYdhIhNnC/QnLDcd3ckzARtIBgcVnDFJcxPJtY+gsXfFEJTaZcF 94pA== X-Gm-Message-State: APt69E0CD0ubww5wRKS8pVGAcUXTrLuLm8lRnHO3Ck8mN+RG94VkM/Pj Gky5/xnZSOgE+rEK/tz4x90oC9bR6O9TvE01wkQ= X-Google-Smtp-Source: ADUXVKLLo8zYKGzYJVg+JC4rRzPOSoK7yE66WAhkLZZnjXPit4EGyMmeQTwr1scvl8nVd8NML99SOBsE0oldZM3kqtw= X-Received: by 2002:a50:ad78:: with SMTP id z53-v6mr2368985edc.306.1528976208587; Thu, 14 Jun 2018 04:36:48 -0700 (PDT) MIME-Version: 1.0 References: <8170cb61-8c85-3dc8-1e16-bdb6a2ada7db@selasky.org> In-Reply-To: <8170cb61-8c85-3dc8-1e16-bdb6a2ada7db@selasky.org> From: Pratyush Yadav Date: Thu, 14 Jun 2018 17:06:12 +0530 Message-ID: Subject: Re: Bug in subr_bus_dma.c's bus_dmamap_load() when multiple dma maps are created from a single tag on x86 To: hps@selasky.org Cc: freebsd-hackers@freebsd.org, =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Edward Napierala Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jun 2018 11:36:50 -0000 On Thu, Jun 14, 2018 at 4:23 PM Hans Petter Selasky wrote: > > On 06/14/18 10:59, Pratyush Yadav wrote: > > Hi everyone, > > > > I was looking at the busdma code for x86 and I notice that the claim > > in the bus_dma(9) man page might not be correct: "Multiple maps can be > > associated with one DMA tag." > > > > Looking at the code of sys/x86/x86/busdma_bounce.c (which I believe is > > the default busdma implementation for x86), I realize that a map does > > not necessarily use all the segments of the dma tag (specified by the > > parameter nsegments on tag creation). > > > > So how many segments does a map actually use? Looking at > > busdma_bounce.c:644,690 it seems we can calculate that from the > > pointer segp. It contains the index of starting segment on entrace to > > the load function, and the ending segment on exit. > > > > Sounds all fine so far. But then if we look at map_complete() > > (busdma_bounce.c:908), it returns the entire segments array of the dma > > *tag*. I emphasize tag because the tag's segments array holds the > > segments of all the maps created with the tag. > > > > Going a layer up, sys/kern/subr_bus_dma.c:328 simply passes nsegs(= > > -1) in the segp parameter to load, not bothering to check if some > > segments have been used already by other maps. It then calculates the > > number of segments used using the value of nsegs after return and > > calls map_complete() to get the segments array. It then passes the > > segments array to the driver callback with the number of segments, > > nsegs. > > > > Effectively, to the driver it seems that the map's segments start from > > index 0 and go till nsegs - 1. This is not true if another map has > > been loaded in the meantime with the same tag. > > > > We are possibly over-writing the previous map's segments. > > busdma_bounce.c:669 simply uses segs[0] if the value of segp is -1, > > which is what bus_dmamap_load() in subr_bus_dma.c:328 passes to > > _bus_dmamap_load_buffer(). segs[0] would also be used by any > > subsequent map's load. This means the previous map's values are > > over-written. > > > > This might cause problems when the driver is in its callback using the > > segments array and another map is loaded. > > > > Unfortunately, I do not posses the knowledge or experience with > > FreeBSD's drivers (drivers in general) to test for this bug. > > > > I have not looked at the code in other architectures. The same problem > > might present itself there as well. > > > > If I made a mistake somewhere in my reasoning, let me know. > > > Hi, > > Use of bus_dmamap_load() on the same tag must be serialized using a > mutex. Maybe it is not so clearly documented. Else like you see the > temporary segs[] list may be overwritten. The man page does say "the most efficient way to protect the tag resources is through the lock that the driver uses." But I never thought it meant that the loads (among other things) must be serialized. Maybe it is better to explicitly mention it in the man page? -- Regards, Pratyush Yadav From owner-freebsd-hackers@freebsd.org Fri Jun 15 16:14:07 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A533101143C for ; Fri, 15 Jun 2018 16:14:07 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:300:2185:a:dead:bad:faff]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D3E6D75765 for ; Fri, 15 Jun 2018 16:14:06 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [104.207.135.49]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id w5FGE4qI057918; Fri, 15 Jun 2018 17:14:04 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id w5FGE3A6057917; Fri, 15 Jun 2018 17:14:03 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <201806151614.w5FGE3A6057917@donotpassgo.dyslexicfish.net> Date: Fri, 15 Jun 2018 17:14:03 +0100 Organization: Dyslexic Fish To: lankfordandrew@charter.net, freebsd-hackers@freebsd.org, eugen@grosbein.net, Cy.Schubert@komquats.com Subject: Re: Root partition and usrland on one slice, /usr/local ports and srcon another References: <20171112034352.186A1866@spqr.komquats.com> In-Reply-To: <20171112034352.186A1866@spqr.komquats.com> User-Agent: Heirloom mailx 12.4 7/29/08 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [104.207.135.49]); Fri, 15 Jun 2018 17:14:05 +0100 (BST) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 16:14:07 -0000 Eugene Grosbein wrote: > I still don't get what specific problems did you have > but I'm sure they were not due to symlinking as such links work just fine > for me with gcc, and with clang too. Cy Schubert replied: > Neither do I. I've been using a combination of symlinks, ufs, nullfs, zfs legacy and straight zfs in various configurations over the years on various systems with no problems whatsoever. I just came across one with multimedia/ffmpeg4: Debugging the install stage gives this, though the cpio error messages are sent to null by default: | + /usr/bin/find -Ed Changelog CREDITS INSTALL.md LICENSE.md MAINTAINERS README.md RELEASE_NOTES | + /usr/bin/cpio -dumpl /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/Changelog | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/CREDITS | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/INSTALL.md | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/LICENSE.md | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/MAINTAINERS | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/README.md | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/RELEASE_NOTES | 0 blocks | + /usr/bin/find -Ed Changelog CREDITS INSTALL.md LICENSE.md MAINTAINERS README.md RELEASE_NOTES '(' -type d -exec /bin/sh -xc 'cd /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg && chmod 755 "$@"' . {} + -o -type f -exec /bin/sh -xc 'cd /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg && chmod 0644 "$@"' . {} + ')' | + cd /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg | + chmod 0644 Changelog CREDITS INSTALL.md LICENSE.md MAINTAINERS README.md RELEASE_NOTES | chmod: Changelog: No such file or directory | chmod: CREDITS: No such file or directory | chmod: INSTALL.md: No such file or directory | chmod: LICENSE.md: No such file or directory | chmod: MAINTAINERS: No such file or directory | chmod: README.md: No such file or directory | chmod: RELEASE_NOTES: No such file or directory cheers, Jamie From owner-freebsd-hackers@freebsd.org Fri Jun 15 17:19:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DDCC1014EB0 for ; Fri, 15 Jun 2018 17:19:38 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (unknown [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E110D795BE for ; Fri, 15 Jun 2018 17:19:37 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w5FHJ3Kq068773 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 15 Jun 2018 19:19:04 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: jamie@catflap.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w5FHIjKR012884 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 16 Jun 2018 00:18:45 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: cpio regression (Re: Root partition and usrland on one slice, /usr/local ports and srcon another) To: Jamie Landeg-Jones , lankfordandrew@charter.net, freebsd-hackers@freebsd.org, Cy.Schubert@komquats.com References: <20171112034352.186A1866@spqr.komquats.com> <201806151614.w5FGE3A6057917@donotpassgo.dyslexicfish.net> From: Eugene Grosbein Message-ID: <5B23F4F0.7050502@grosbein.net> Date: Sat, 16 Jun 2018 00:18:40 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <201806151614.w5FGE3A6057917@donotpassgo.dyslexicfish.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -0.0 SPF_PASS SPF: sender matches SPF record * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 17:19:38 -0000 15.06.2018 23:14, Jamie Landeg-Jones wrote: >> I still don't get what specific problems did you have >> but I'm sure they were not due to symlinking as such links work just fine >> for me with gcc, and with clang too. > > Cy Schubert replied: > >> Neither do I. I've been using a combination of symlinks, ufs, nullfs, zfs legacy and straight zfs in various configurations over the years on various systems with no problems whatsoever. > > I just came across one with multimedia/ffmpeg4: > > Debugging the install stage gives this, though the cpio error messages are sent to null by default: > > | + /usr/bin/find -Ed Changelog CREDITS INSTALL.md LICENSE.md MAINTAINERS README.md RELEASE_NOTES > | + /usr/bin/cpio -dumpl /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg > | cpio: Cannot extract through symlink /scratch/root/ports_base/usr/ports/multimedia/ffmpeg/work/stage/usr/local/share/doc/ffmpeg/Changelog Well, I known this. Some time ago our cpio got "broken" in a way: it now demands "--insecure" option to work with symlinks same way it worked long before without this option. While I may understand motivation, this change still broke previously working scripts like /usr/src/tools/tools/nanobsd/nanobsd.sh (generates NanoBSD images) using cpio without "--insecure" when some parts of the building system are symlinked to other locations. From owner-freebsd-hackers@freebsd.org Sat Jun 16 17:32:03 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6DE8101CE3E; Sat, 16 Jun 2018 17:32:02 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8575068EDA; Sat, 16 Jun 2018 17:32:01 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.0.114] (unknown [130.149.50.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 4EE0761F99; Sat, 16 Jun 2018 17:31:52 +0000 (UTC) Subject: Re: sizeof jail parameter values To: James Gritton , freebsd-jail@freebsd.org Cc: freebsd-hackers@freebsd.org References: From: Fabian Freyer Message-ID: Date: Sat, 16 Jun 2018 19:31:44 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jun 2018 17:32:03 -0000 [reordered parts of the reply for better reading flow] On 05/18/2018 18:49, James Gritton wrote: > I would recommend skipping out on jail_getv(), which is really only > good for getting a few well-known parameters, and instead use the more > complete but more complex jailparam_init/get/export/free. Thanks! I ended up writing wrappers around the jail_get(2) and jail_set(2) interfaces and constructing the iovectors myself, which ended up quite a bit cleaner. The jailparam_{init,get,export,free} APIs are unnecessarily complex and don't seem to be a good fit (writing wrappers around wrappers around wrappers etc...). > There is a way to find the length of a string parameter, though there > isn't a good library interface for it.  The security.jail.param.* > sysctls describe the form of the parameters, giving the type. The > "contents" of these sysctls  are generally unused (and set to zero), but > for string parameters there's actually the max length of the string > (itself in string form). Thanks, this works great for strings! > For non-string parameters, the length in > string form depends on the type of the parameters, so for an int you'll > need as long as the string representation of an ant can be, etc.  I > don't know how much good C code will do for you for Rust work, but you > might want to take a look at jailparam_type() in the libjail source code. > It gets more complicated with array parameters, those that can hold an > arbitrary number of values.  The IP addresses are the best example of > that. I've now hit that snag. I see that the security.jail.param.ip4.addr and security.jail.param.ip6.addr sysctls contain the sizes of an in_addr_t and an in6_addr_t, respectively. How would I now determine the number of IPv4 and IPv6 addresses, or should I just allocate security.jail.jail_max_af_ips per family? I've tried to go through how libjail does it, but don't quite understand it, nor the implied race conditions (?) it attempts to mitigate by reading the vector multiple times: lib/libjail/jail.c: /* * Get the prison. If there are array elements, retry a few times * in case their sizes changed from under us. */ for (sanity = 0;; sanity++) { [...] Fabian From owner-freebsd-hackers@freebsd.org Sat Jun 16 23:32:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 459881002034; Sat, 16 Jun 2018 23:32:09 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gritton.org", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D315073A1B; Sat, 16 Jun 2018 23:32:08 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org ([127.0.0.131]) by gritton.org (8.15.2/8.15.2) with ESMTP id w5GNC185071868; Sat, 16 Jun 2018 17:12:02 -0600 (MDT) (envelope-from jamie@freebsd.org) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sat, 16 Jun 2018 17:12:01 -0600 From: James Gritton To: Fabian Freyer Cc: freebsd-jail@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: sizeof jail parameter values In-Reply-To: References: Message-ID: <92cb2f93a546e02a7fb5e11ea976e846@freebsd.org> X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.3.6 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jun 2018 23:32:09 -0000 On 2018-06-16 11:31, Fabian Freyer wrote: > On 05/18/2018 18:49, James Gritton wrote: > >> I would recommend skipping out on jail_getv(), which is really only >> good for getting a few well-known parameters, and instead use the more >> complete but more complex jailparam_init/get/export/free. > Thanks! I ended up writing wrappers around the jail_get(2) and > jail_set(2) interfaces and constructing the iovectors myself, which > ended up quite a bit cleaner. The jailparam_{init,get,export,free} > APIs are unnecessarily complex and don't seem to be a good fit > (writing wrappers around wrappers around wrappers etc...). They're an attempt to make generic handlers in C, which isn't exactly a language geared toward such things. If you're working only with a few specific known fields, your way is just as well. >> It gets more complicated with array parameters, those that can hold an >> arbitrary number of values.  The IP addresses are the best example of >> that. > I've now hit that snag. I see that the security.jail.param.ip4.addr > and security.jail.param.ip6.addr sysctls contain the sizes of an > in_addr_t and an in6_addr_t, respectively. How would I now determine > the number of IPv4 and IPv6 addresses, or should I just allocate > security.jail.jail_max_af_ips per family? I've tried to go through how > libjail does it, but don't quite understand it, nor the implied race > conditions (?) it attempts to mitigate by reading the vector multiple > times: > > lib/libjail/jail.c: > /* > * Get the prison. If there are array elements, retry a few times > * in case their sizes changed from under us. > */ > for (sanity = 0;; sanity++) { > [...] If you read a parameters with the value's iov_base set to NULL, it will return the parameter's length into your iov_len. So the way to read any variable-length parameter is to call jail_get(2) once with a NULL value, allocate a buffer according to the returned length, and then call it again with the allocated iov_base. The race condition I look for is the jail changing between the time I get the length and the time I read the value - like most races, very unlikely. Once again, this is for the generic case. If you have a value with a known (and reasonably sized) maximum, such as MAXHOSTNAMELEN or max_af_ips, it's easier to just use that. - James