From owner-freebsd-questions@FreeBSD.ORG Thu Jun 5 20:26:38 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1BD21065673 for ; Thu, 5 Jun 2008 20:26:38 +0000 (UTC) (envelope-from kirk@strauser.com) Received: from kanga.honeypot.net (kanga.honeypot.net [206.29.77.83]) by mx1.freebsd.org (Postfix) with ESMTP id 4DC378FC14 for ; Thu, 5 Jun 2008 20:26:38 +0000 (UTC) (envelope-from kirk@strauser.com) Received: from localhost (localhost [127.0.0.1]) by kanga.honeypot.net (Postfix) with ESMTP id 22F485DDE9E for ; Thu, 5 Jun 2008 15:08:35 -0500 (CDT) X-Virus-Scanned: amavisd-new at honeypot.net Received: from kanga.honeypot.net ([127.0.0.1]) by localhost (kanga.honeypot.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EdLVi3tnAoW6 for ; Thu, 5 Jun 2008 15:08:31 -0500 (CDT) Received: from janus.daycos.com (janus.daycos.com [10.45.12.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by kanga.honeypot.net (Postfix) with ESMTPSA id 674FA5DCD22 for ; Thu, 5 Jun 2008 15:08:31 -0500 (CDT) From: Kirk Strauser To: freebsd-questions@freebsd.org Date: Thu, 5 Jun 2008 15:08:28 -0500 User-Agent: KMail/1.9.9 X-Face: T+/_{qmjgbosI0J/e83I~w[&VF'w)!((xEpj///^bA/6?jHHS?nq+T8_+`nh"WnEWCWG, \}]Y2$)) =?utf-8?q?vLVz4ACChrEcb=7DCO=5EtYmMG=5C=0A=09ts=2Em=3F=5B7=5B6OwE*dAJ*9f+m?= =?utf-8?q?X=2E7R32qeN=5EDJ=5C?=(k@evW?IRQCy.^ MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200806051508.29424.kirk@strauser.com> Subject: Poor read() performance, and I can't profile it X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2008 20:26:38 -0000 I'm running a command (dumprecspg from my XBaseToPg project) on a FreeBSD 7 server. I've noticed that throughput on that program is a lot lower than I would have expected, and further investigation found it spending most of its time in the kernel, presumably in read() [1]. I was testing the same software on my desktop PC when I noticed that it ran *much* faster, and found that it was spending only about 1% as much time in the kernel on Linux as it was on FreeBSD. I ran a quick-and-dirty comparison of the same software on two different machines, the FreeBSD server being by far the more powerful of the two. I ran the same command on both machines from various filesystems (to rule out differences in drive performance), and posted the output of zsh's "time" command for the fastest run in each setting. The results are below. Any ideas what could be causing this horrible performance? I'm willing to try just about anything. FreeBSD on a Dell Poweredge 1600SC server: 7-STABLE from 2008-03-09 2x 2.4GHz P4 Xeon 3GB RAM Changes to /etc/make.conf: CPUTYPE?=pentium4 Kernel config: include GENERIC ident JAIL1 options PMAP_SHPGPERPROC=301 nooption SCHED_4BSD option SCHED_ULE root : Fujitsu 36GB, 10k RPM Best time: 6.37s user 9.68s system 99% cpu 16.068 total /tmp : tmpfs Best time: 6.29s user 10.88s system 99% cpu 17.194 total /fast : 4 Seagate Cheetah 36GB, 15k RPM SCSI320 drives in RAID-0 with gstripe, 128KB stripe size with kern.geom.stripe.fast enabled and stripe.fast_failed=0 Best time: 6.60s user 9.46s system 99% cpu 16.088 total Conclusion: Since gstat showed all drives as idle through most of all the tests, it looks like the rest is running entirely from buffers. Linux on a Dell Dimension 4600 desktop: Ubuntu 8.04 2.4GHz P4 1GB RAM root: WD 250GB SATA Best time: 7.60s user 0.92s system 97% cpu 8.722 total Conclusion: I don't know if there's an equivalent to gstat in Linux, but the system overhead is about one-hundredth as much as in FreeBSD. [1] I can't run gprof on FreeBSD because if I build the binary with -pg, then it segfaults on startup: $ gdb /tmp/xbase/bin/dumprecspg /tmp/dumprecspg.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Core was generated by `dumprecspg'. Program terminated with signal 11, Segmentation fault. Reading symbols from /tmp/xbase/lib/libxbase64.so.1.0...done. Loaded symbols for /tmp/xbase/lib/libxbase64.so.1.0 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0807110c in main (ac=Cannot access memory at address 0x18 ) at dumprecspg.cpp:63 63 int main(int ac,char** av) (gdb) -- Kirk Strauser