From owner-freebsd-questions@freebsd.org Fri Mar 5 21:43:43 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F278568FD1 for ; Fri, 5 Mar 2021 21:43:43 +0000 (UTC) (envelope-from bob@proulx.com) Received: from havoc.proulx.com (havoc.proulx.com [96.88.95.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dsh7f3F4cz3n7f for ; Fri, 5 Mar 2021 21:43:42 +0000 (UTC) (envelope-from bob@proulx.com) Received: from joseki.proulx.com (localhost [127.0.0.1]) by havoc.proulx.com (Postfix) with ESMTP id D04D387 for ; Fri, 5 Mar 2021 14:43:34 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proulx.com; s=dkim2048; t=1614980614; bh=orGRWb/EMN+tR77F38D4NQd0vrpQ/Dbxev+R8Y18Fk8=; h=Date:From:To:Subject:References:In-Reply-To:From; b=n20qdKDVY8bCza7IiE/NLWdBsrdE/PbAPocT05h4h2se0hLCxWWM2J593Bmm1rXR+ wlv1bBA095utIqUeYEAUEgtidY/IUq/Nn+7LJBD2zvSk5A0eD+YwYgfHn0zXZXix5/ jOb7QYdasY3/YC5OUMZluQTUD+qaTvvixjPadfurvrwnSJoqqOdUv6cjkXWFyKqI/z 07gJU1TNmIGM0zpyOMoREQ+ZuYaS7mcoArtDUfLer2iQSE/hRdJ9+tpYmmwWzzfjtW uAMvmEOVh7jilRn6weG4KeArjsgSUXkaTBAzde9yAtxesNM1pNpuhiytHVfPmWgpZS AZt+t6U9jN3BQ== Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id 8FF5E21156 for ; Fri, 5 Mar 2021 14:43:34 -0700 (MST) Received: by hysteria.proulx.com (Postfix, from userid 1000) id 5084D2DC9C; Fri, 5 Mar 2021 14:43:34 -0700 (MST) Date: Fri, 5 Mar 2021 14:43:34 -0700 From: Bob Proulx To: freebsd-questions@freebsd.org Subject: Re: sed -i empty argument compatibility issue Message-ID: <20210305142352297368817@bob.proulx.com> Mail-Followup-To: freebsd-questions@freebsd.org References: <9178f6c5-631a-c2c2-c6b1-8def94a3397b@dreamchaser.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9178f6c5-631a-c2c2-c6b1-8def94a3397b@dreamchaser.org> X-Rspamd-Queue-Id: 4Dsh7f3F4cz3n7f X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=proulx.com header.s=dkim2048 header.b=n20qdKDV; dmarc=pass (policy=none) header.from=proulx.com; spf=pass (mx1.freebsd.org: domain of bob@proulx.com designates 96.88.95.61 as permitted sender) smtp.mailfrom=bob@proulx.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[96.88.95.61:from]; R_DKIM_ALLOW(-0.20)[proulx.com:s=dkim2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[96.88.95.61:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[proulx.com:+]; DMARC_POLICY_ALLOW(-0.50)[proulx.com,none]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:7922, ipnet:96.64.0.0/11, country:US]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2021 21:43:43 -0000 Gary Aitken wrote: > I'm trying to come up with a fix for a script in a port which invokes sed. > The port comes from a linux environment, and the offending line looks like this: > (This is in a cMake file.) > > COMMAND sed -i "/^# /d" "${outfile}" Unfortunately that is a GNU sed extension and use of it is intrinsically not portable usage. And the FreeBSD use of -i is a similar but slightly incompatible usage. The -i option is not specified by POSIX and so any use of it is non-standard by definition. > The issue is that linux sed expects the -I or -i extension modifier to > immediately follow the -i. In the above line, the extension is deliberately > missing to provide in-place editing. > > fbsd expects the extension to be separated from the -i by whitespace, or > doesn't work properly when it is empty or immediately follows the -i: The pedantic person that I am disagrees with this detail. FreeBSD sed is okay with an empty extension. "If a zero-length extension is given, no backup will be saved." A zero length extension would be this following. sed -i "" "/^# /d" "${outfile}" But that isn't what is happening here. Instead it is getting confused by seeing the "/^# /d" as the extension. And then it will be taking the "${outfile}" part and trying to use it as the sed edit command. What happens depends upon the file name. It's data dependent and depends upon the incoming data. Which can't end well. > So fbsd works with '-i ""' but linux requires '-i""' Right. It is a basic incompatibility. It's non-portable use of sed in the case you show. > Does anyone know a work-around for this problem? In order to port this then _something_ must be changed. As to what must be changed that is up to you and judegement and the larger view of everything in your environment. But off the top of my head these different possibilities come to mind. 1. Edit the script and change this to that. This makes it run on FreeBSD sed. Of course then this does not work the other direction. -COMMAND sed -i "/^# /d" "${outfile}" +COMMAND sed -i "" "/^# /d" "${outfile}" 2. Use perl. Change that edit command to be this. This should work the same everywhere that has perl. It does create a dependency upon perl that might not have been present before. perl -i -lpe 'next if /^# /' "${outfile}" 3. Avoid -i and use a temporary file, just like in the old days. However this introduces the need for temporary file handling. But maybe that isn't important here. sed "/^# /d" "${outfile}" > "$tmpfile" && mv "$tmpfile" "${outfile}" 4. Introduce a "sed" wrapper in PATH that intercepts the call to the real sed, detects this problematic usage case, and then DTRT does the right thing with it. This wrapper could have all of the correct temporary file handling needed to make this work. More involved to do this but if I were faced with a large unknown code base and expected to see a lot of this then I would consider it a good global workaround for portability. 5. Hack a local FreeBSD sed version to detect this case and DTRT with the option. 6. Compile GNU sed locally and install it in PATH earlier. This would probably be a generally good option globally. However it's possible you might run into portability problems in the reverse case. Good luck! I'll be interested in hearing how things work out for you. Bob