From owner-freebsd-questions@freebsd.org Tue Nov 7 17:13:07 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07F2FE5E3C6 for ; Tue, 7 Nov 2017 17:13:07 +0000 (UTC) (envelope-from byrnejb@harte-lyne.ca) Received: from inet08.hamilton.harte-lyne.ca (inet08.hamilton.harte-lyne.ca [216.185.71.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "inet08.hamilton.harte-lyne.ca", Issuer "CA_HLL_ISSUER_2016" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CCA4077A2A for ; Tue, 7 Nov 2017 17:13:06 +0000 (UTC) (envelope-from byrnejb@harte-lyne.ca) Received: from localhost (localhost [127.0.0.1]) by inet08.hamilton.harte-lyne.ca (Postfix) with ESMTP id 65A8A622C7 for ; Tue, 7 Nov 2017 12:12:58 -0500 (EST) X-Virus-Scanned: amavisd-new at harte-lyne.ca Received: from inet08.hamilton.harte-lyne.ca ([127.0.0.1]) by localhost (inet08.hamilton.harte-lyne.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O_k75VrVFZcW for ; Tue, 7 Nov 2017 12:12:56 -0500 (EST) Received: from webmail.harte-lyne.ca (inet04.hamilton.harte-lyne.ca [216.185.71.24]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by inet08.hamilton.harte-lyne.ca (Postfix) with ESMTPSA id A23716218B for ; Tue, 7 Nov 2017 12:12:55 -0500 (EST) Received: from 216.185.71.44 (SquirrelMail authenticated user byrnejb_hll) by webmail.harte-lyne.ca with HTTP; Tue, 7 Nov 2017 12:12:55 -0500 Message-ID: Date: Tue, 7 Nov 2017 12:12:55 -0500 Subject: sed - remove nul lines from file From: "James B. Byrne" To: freebsd-questions@freebsd.org Reply-To: byrnejb@harte-lyne.ca User-Agent: SquirrelMail/1.4.22-5.el6 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Nov 2017 17:13:07 -0000 I have a data file created by an ancient proprietary scripting language called QTP. There is a bug in this program which, on occasion, manifests itself by inserting output records consisting entirely of nul (^@) (\x00) bytes at regular intervals. In the present case every 47th. record consists entirely of nuls. The purpose of this data file is to feed a psql COPY statement for loading into a PostgreSQL database. The presence of the NUL characters prevents this. I have previously used the tr utility to remove the NUL characters but this requires me to manually remove the residual empty lines. I have tried various permutations of the sed invocation reproduced below to remove these lines directly but without success. The examples that I have found on StackExchange and various other self-help sites do not give the results claimed, at least not for me on FreeBSD. So, I would appreciate if anyone here can point out what I am doing wrong or how the sed on FreeBSD differs in behaviour for that used in the examples I have found. Given a file INFILE with records containing the following: . . . *93566000008166*,*CCTL*,*3072 49534494 * *93566000008166*,*CCTL*,*3072 49534493 * *93566000008166*,*CCTL*,*3072 49534497 * *93566000015962*,*CCTL*,*8156 4171000541 * ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ . . . *93566000198850*,*CCTL*,*417 1003874 * *93566000010320*,*CCTL*,*8084 2601553853102 * . . . I wish to remove (all) the line(s) with the nul (^@) characters. I have tried this: sed '/^\x00*$/d' INFILE > INFILE.sed and this: sed _E '/^\x00*$/d' INFILE > INFILE.sed but neither these nor the many other combinations that I have tried remove the lines. What is the method of accomplishing this in sed or is it not possible? -- *** e-Mail is NOT a SECURE channel *** Do NOT transmit sensitive data via e-Mail Do NOT open attachments nor follow links sent by e-Mail James B. Byrne mailto:ByrneJB@Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3