From owner-freebsd-questions@FreeBSD.ORG Fri Dec 18 15:31:49 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C49A106566B for ; Fri, 18 Dec 2009 15:31:49 +0000 (UTC) (envelope-from mkushnir@lohika.com) Received: from sbox.lohika.com (sbox.lohika.com [217.9.0.182]) by mx1.freebsd.org (Postfix) with ESMTP id C72AA8FC13 for ; Fri, 18 Dec 2009 15:31:48 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.47,418,1257112800"; d="scan'208";a="3510597" Received: from unknown (HELO dekker.lohika.com) ([172.20.100.30]) by sbox-local.lohika.com with ESMTP; 18 Dec 2009 17:31:45 +0200 Received: from emkushnir.lv.lohika.com ([172.22.60.77]) (authenticated bits=0) by dekker.lohika.com (8.14.2/8.14.2) with ESMTP id nBIFZXdr010679 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 18 Dec 2009 17:35:36 +0200 (EET) (envelope-from mkushnir@lohika.com) Message-ID: <4B2BA025.6050200@lohika.com> Date: Fri, 18 Dec 2009 17:30:45 +0200 From: Markiyan Kushnir User-Agent: Thunderbird 2.0.0.9 (X11/20080311) MIME-Version: 1.0 To: Nick Barnes References: <11404.1261135326@thrush.ravenbrook.com> In-Reply-To: <11404.1261135326@thrush.ravenbrook.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.94.2/10197/Fri Dec 18 03:59:38 2009 on dekker.lohika.com X-Virus-Status: Clean Cc: Giorgos Keramidas , Anton Shterenlikht , "freebsd-questions@freebsd.org" Subject: Re: editing a binary file X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Dec 2009 15:31:49 -0000 If you don't mind, I would add fixed buffer processing to your program. For some really huge files (or any other type of stream) which have really huge records, reading entire records into memory would get the box down. Markiyan /* convert.c: remove record length fields from Fortran output file. */ /* Nick Barnes, Ravenbrook Limited, 2009-12-18 */ #include #include #include #include #define BUFLEN 1024*1024 static char buf[BUFLEN]; int main(void) { long size, size_in, size_out; ssize_t bytes; assert(sizeof(size) == 4); int buflen; while((bytes = read(0, (void*)&size, sizeof(size)))) { if (bytes < 0) { fprintf(stderr, "read() returned %d\n", bytes); exit(1); } if (size <= 0) { fprintf(stderr, "Read bad record " "length %ld\n", size); exit(1); } size_in = size; buflen = size_in > BUFLEN ? BUFLEN : size_in; while (size_in > 0 && (bytes = read(0, buf, buflen)) > 0) { write(1, buf, bytes); size_in -= bytes; if (size_in < BUFLEN) { /* the last chunk is less than BUFLEN */ buflen = size_in; } } if (bytes == 0) { /* EOF */ if (size_in > 0) { fprintf(stderr, "premature end of " "file: %ld bytes unread.\n", size_in); exit(1); } break; } else if (bytes <= 0) { fprintf(stderr, "read() returned %d\n", bytes); exit(1); } if (read(0, (void *)&size_out, sizeof(size_out)) > 0) { if (size_out != (size)) { fprintf(stderr, "Mismatched record lengths: " "%ld, %ld\n", size, size_out); exit(1); } } } return 0; } Nick Barnes wrote: > If your Fortran file has the same word size and enddian-ness as your > C, this simple program convert.c will strip all the record length > fields. I just knocked it up now, no warranty, etc, but it works for > me. Use as a pipe: > > $ ls > convert.c test.f > $ gcc -Wall -Werror -ansi -pedantic convert.c -o convert > $ gfortran44 test.f -o test > $ ./test > $ ls -l test-output > -rw-r--r-- 1 nb nb 2460 Dec 18 11:17 test-output > $ ./convert < test-output > test-converted > $ ls -l test-converted > -rw-r--r-- 1 nb nb 2420 Dec 18 11:18 test-converted > $ > > The code does a fair amount of checking; if you get one of the error > messages, let us know. The most obvious unchecked problem is a short > read, which will complain about mismatched lengths. > > If your Fortran has different word sizes or enddian-ness (e.g. most of > the Fortran output files I use on the Clear Climate Code project > are generated on big-endian machines), > you will need to add code to tweak the 'size' value after reading it, > and when checking the record-end marker. > > Nick B > > /* convert.c: remove record length fields from Fortran output file. */ > /* Nick Barnes, Ravenbrook Limited, 2009-12-18 */ > > #include > #include > #include > #include > > int main(void) > { > long size; > char *buf; > ssize_t bytes; > assert(sizeof(size) == 4); > while(bytes = read(0, (void*)&size, sizeof(size))) { > if (bytes < 0) { > fprintf(stderr, "read() returned %ld\n", bytes); > exit(1); > } > if (size <= 0) { > fprintf(stderr, "Read bad record length %ld\n", size); > exit(1); > } > buf = (char*)malloc(size + sizeof(size)); > if (!buf) { > fprintf(stderr, "Couldn't allocate buffer of %ld bytes\n", > size + sizeof(size)); > exit(1); > } > bytes = read(0, buf, size + sizeof(size)); > if (bytes <= 0) { > fprintf(stderr, "read() returned %ld\n", bytes); > exit(1); > } > if ((*(long*)(buf+size)) != size) { > fprintf(stderr, "Mismatched record lengths: %ld, %ld\n", > size, *(long*)(buf+size)); > exit(1); > } > write(1, buf, size); > free(buf); > } > return 0; > } > > > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"