From owner-freebsd-stable@FreeBSD.ORG Thu Oct 30 00:09:08 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A2EECE86 for ; Thu, 30 Oct 2014 00:09:08 +0000 (UTC) Received: from mail.akips.com (mail.akips.com [65.19.130.19]) by mx1.freebsd.org (Postfix) with ESMTP id 83DB3DFC for ; Thu, 30 Oct 2014 00:09:08 +0000 (UTC) Received: from akips.com (CPE-120-146-191-2.static.qld.bigpond.net.au [120.146.191.2]) by mail.akips.com (Postfix) with ESMTPSA id 6D9EE17 for ; Thu, 30 Oct 2014 10:09:01 +1000 (EST) Date: Thu, 30 Oct 2014 10:08:53 +1000 From: Paul Koch To: freebsd-stable@FreeBSD.org Subject: Suspected kernel memory leak with mmap/sha1 ? Message-ID: <20141030100853.65a62326@akips.com> Organization: AKIPS X-Mailer: Claws Mail 3.10.1 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY, URIBL_BLOCKED autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on host1.akips.com X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Oct 2014 00:09:08 -0000 Hi, We have been trying to track down an odd problem with what appears to be a kernel memory leak, but are at a complete loss of what is going on. We have a cut down test program that reproduces the problem. FreeBSD shed62.akips.com 10.1-RC3 FreeBSD 10.1-RC3 #0 r273491: Thu Oct 23 05:08:22 UTC 2014 root@shed31.akips.com:/usr/obj/usr/src/sys/GENERIC amd64 Basically our app mmap's large data files (eg. 250M) which contain blocks of compressed data. Our uncompress runs the data through SHA1 before uncompressing it. What we see is active memory being consumed, but can't figure out what is consuming it as our process size stays constant. When our process exits, active memory stays high and is never freed. If enough memory leaks, then the machine starts swapping and game over for our application. We've tried it on both ufs and zfs with the same results. Interestingly if you umount the zfs file system the active memory immediately disappears. Without the SHA1_xxx it runs fine. It's almost like a denial of service problem. To show the problem, run top and watch active memory. Compile the test program below cc -o leaktest -O2 leaktest.c find /usr -type f | xargs ./leaktest For some reason, active memory skyrockets, especially when it encounters large files. Running the program does not reproduce the problem 100% of the time, but close to it. Any suggestions ?? ----------------------------------------------- #include #include #include #include #include #include #include #include #include #include #define BUFSIZE (1 * 1024 * 1024) #define TMIN(a,b) ({ \ typeof (a) _a = (a); \ typeof (b) _b = (b); \ _a < _b ? _a : _b; \ }) int main (int argc, char **argv) { int i, j, fd = -1, open_flags = O_RDONLY, prot_flags = PROT_READ, mmap_flags = 0; char *filename, *data = NULL, *buf = NULL, *p; size_t len, rlen; struct stat s; u_char md[20]; SHA_CTX SD; if ((buf = malloc ((size_t) BUFSIZE)) == NULL) { fprintf (stderr, "malloc: %s\n", strerror (errno)); goto END; } for (i = 1; i < argc; i++) { filename = argv[i]; if (stat (filename, &s) != 0) fprintf (stderr, "stat: %s %s\n", filename, strerror (errno)); else if ((fd = open (filename, open_flags)) == -1) fprintf (stderr, "open: %s %s\n", filename, strerror (errno)); else if ((data = mmap (NULL, (size_t) s.st_size, prot_flags, mmap_flags, fd, (off_t) 0)) == MAP_FAILED) { fprintf (stderr, "mmap: %s %s\n", filename, strerror (errno)); close (fd); fd = -1; } else { printf ("%s: %zd bytes\n", filename, s.st_size); p = data; len = s.st_size; while (len > 0) { rlen = TMIN (BUFSIZE, len); /* Copy BUFSIZE lumps into buf and modify it so the compiler * doesn't optimise it out */ memcpy (buf, p, rlen); for (j = 0; j < BUFSIZE; j += 1024) buf[j]++; SHA1_Init (&SD); SHA1_Update (&SD, p, rlen); SHA1_Final (md, &SD); p += rlen; len -= rlen; } if (munmap (data, (size_t) s.st_size) == -1) { fprintf (stderr, "munmap: %s %s\n", filename, strerror (errno)); goto END; } if (close (fd) == -1) { fprintf (stderr, "close: %s %s\n", filename, strerror (errno)); goto END; } } } END: free (buf); exit (0); } -- Paul Koch | Founder, CEO AKIPS Network Monitor http://www.akips.com Brisbane, Australia