From owner-freebsd-bugs@FreeBSD.ORG Sat Jan 30 10:50:02 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21D301065672 for ; Sat, 30 Jan 2010 10:50:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F32908FC0A for ; Sat, 30 Jan 2010 10:50:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0UAo1RZ051419 for ; Sat, 30 Jan 2010 10:50:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0UAo1li051418; Sat, 30 Jan 2010 10:50:01 GMT (envelope-from gnats) Resent-Date: Sat, 30 Jan 2010 10:50:01 GMT Resent-Message-Id: <201001301050.o0UAo1li051418@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Mikolaj Golub Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2D8D106566B for ; Sat, 30 Jan 2010 10:45:23 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id C1E978FC12 for ; Sat, 30 Jan 2010 10:45:23 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o0UAjNTd094093 for ; Sat, 30 Jan 2010 10:45:23 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o0UAjNsN094092; Sat, 30 Jan 2010 10:45:23 GMT (envelope-from nobody) Message-Id: <201001301045.o0UAjNsN094092@www.freebsd.org> Date: Sat, 30 Jan 2010 10:45:23 GMT From: Mikolaj Golub To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: bin/143368: [patch] awk(1): number of open files is limited to small constant X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2010 10:50:02 -0000 >Number: 143368 >Category: bin >Synopsis: [patch] awk(1): number of open files is limited to small constant >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Jan 30 10:50:01 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Mikolaj Golub >Release: >Organization: >Environment: FreeBSD zhuzha.ua1 8.0-STABLE FreeBSD 8.0-STABLE #6: Sun Jan 24 21:36:17 EET 2010 root@zhuzha.ua1:/usr/obj/usr/src/sys/GENERIC i386 >Description: The maximal number of open files in awk(1) is limited by FOPEN_MAX constant defined for awk in usr.bin/awk/Makefile: CFLAGS+= -DHAS_ISBLANK -I. -I${AWKSRC} -DFOPEN_MAX=64 This might be low for today needs and today hardware allows much higher values. NetBSD has fixed their version of awk making the array of files dynamically allocated http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=37205 It would be nice to do the same in FreeBSD. >How-To-Repeat: kopusha:~% awk 'BEGIN {for (i=1; i <= 2000; ++i){print "test" > ("/tmp/file_" i)}}' awk: /tmp/file_62 makes too many open files source line number 1 >Fix: See the attached patch adopted from NetBSD (PR/37205: Aleksey Cheusov: nawk: a number of open files is limited to very small constant. Make the array of files dynamically allocated.). Patch attached with submission follows: diff -ru contrib/one-true-awk.orig/run.c contrib/one-true-awk/run.c --- contrib/one-true-awk.orig/run.c 2007-06-05 18:33:51.000000000 +0300 +++ contrib/one-true-awk/run.c 2010-01-30 12:27:38.000000000 +0200 @@ -1613,28 +1613,36 @@ FILE *fp; const char *fname; int mode; /* '|', 'a', 'w' => LE/LT, GT */ -} files[FOPEN_MAX] ={ - { NULL, "/dev/stdin", LT }, /* watch out: don't free this! */ - { NULL, "/dev/stdout", GT }, - { NULL, "/dev/stderr", GT } -}; +} *files; +size_t nfiles; void stdinit(void) /* in case stdin, etc., are not constants */ { + nfiles = FOPEN_MAX; + files = calloc(nfiles, sizeof(*files)); + if (files == NULL) + FATAL("can't allocate file memory for %zu files", nfiles); files[0].fp = stdin; + files[0].fname = "/dev/stdin"; + files[0].mode = LT; files[1].fp = stdout; + files[1].fname = "/dev/stdout"; + files[1].mode = GT; files[2].fp = stderr; + files[2].fname = "/dev/stderr"; + files[2].mode = GT; } FILE *openfile(int a, const char *us) { const char *s = us; - int i, m; + size_t i; + int m; FILE *fp = 0; if (*s == '\0') FATAL("null file name in print or getline"); - for (i=0; i < FOPEN_MAX; i++) + for (i = 0; i < nfiles; i++) if (files[i].fname && strcmp(s, files[i].fname) == 0) { if (a == files[i].mode || (a==APPEND && files[i].mode==GT)) return files[i].fp; @@ -1644,11 +1652,19 @@ if (a == FFLUSH) /* didn't find it, so don't create it! */ return NULL; - for (i=0; i < FOPEN_MAX; i++) - if (files[i].fp == 0) + for (i = 0; i < nfiles; i++) + if (files[i].fp == NULL) break; - if (i >= FOPEN_MAX) - FATAL("%s makes too many open files", s); + if (i >= nfiles) { + struct files *nf; + size_t nnf = nfiles + FOPEN_MAX; + nf = realloc(files, nnf * sizeof(*nf)); + if (nf == NULL) + FATAL("cannot grow files for %s and %zu files", s, nnf); + (void)memset(&nf[nfiles], 0, FOPEN_MAX * sizeof(*nf)); + nfiles = nnf; + files = nf; + } fflush(stdout); /* force a semblance of order */ m = a; if (a == GT) { @@ -1674,9 +1690,9 @@ const char *filename(FILE *fp) { - int i; + size_t i; - for (i = 0; i < FOPEN_MAX; i++) + for (i = 0; i < nfiles; i++) if (fp == files[i].fp) return files[i].fname; return "???"; @@ -1685,13 +1701,14 @@ Cell *closefile(Node **a, int n) { Cell *x; - int i, stat; + size_t i; + int stat; n = n; x = execute(a[0]); getsval(x); stat = -1; - for (i = 0; i < FOPEN_MAX; i++) { + for (i = 0; i < nfiles; i++) { if (files[i].fname && strcmp(x->sval, files[i].fname) == 0) { if (ferror(files[i].fp)) WARNING( "i/o error occurred on %s", files[i].fname ); @@ -1715,9 +1732,10 @@ void closeall(void) { - int i, stat; + size_t i; + int stat; - for (i = 0; i < FOPEN_MAX; i++) { + for (i = 0; i < nfiles; i++) { if (files[i].fp) { if (ferror(files[i].fp)) WARNING( "i/o error occurred on %s", files[i].fname ); @@ -1733,9 +1751,9 @@ void flush_all(void) { - int i; + size_t i; - for (i = 0; i < FOPEN_MAX; i++) + for (i = 0; i < nfiles; i++) if (files[i].fp) fflush(files[i].fp); } >Release-Note: >Audit-Trail: >Unformatted: