From owner-freebsd-net@FreeBSD.ORG Sat May 15 22:27:15 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4AA3416A4CE for ; Sat, 15 May 2004 22:27:15 -0700 (PDT) Received: from mail.dragondata.com (server2-b.dragondata.com [64.202.113.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 49A4243D46 for ; Sat, 15 May 2004 22:27:14 -0700 (PDT) (envelope-from toasty@dragondata.com) Received: (qmail 75027 invoked by uid 1092); 16 May 2004 05:29:49 -0000 Received: from toasty@dragondata.com by server2.dragondata.com by uid 82 with qmail-scanner-1.20rc3 (uvscan: v4.2.40/v4296. spamassassin: 2.60-cvs. Clear:RC:1:. Processed in 2.971648 secs); 16 May 2004 05:29:49 -0000 Received: from ppp045.dhcp.your.org (HELO ?199.165.179.45?) (199.165.179.45) by mail.dragondata.com with RC4-SHA encrypted SMTP; 16 May 2004 05:29:46 -0000 Mime-Version: 1.0 (Apple Message framework v613) Content-Transfer-Encoding: 7bit Message-Id: Content-Type: text/plain; charset=US-ASCII; format=flowed To: freebsd-net@freebsd.org From: Kevin Day Date: Sun, 16 May 2004 00:27:48 -0500 X-Mailer: Apple Mail (2.613) Subject: Sendfile performance regression from 4.x to 5.x X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 May 2004 05:27:15 -0000 We've got two nearly identically configured servers, serving the exact same load. Quick config: Dual P4 2.8Ghz with HTT enabled, 2GB RAM ahc SCSI adapter, with 4 drives in a vinum RAID5 config Dual bge gigabit network cards Server 1 is running 4.8 Server 2 is running 5.2.1 Other than OS version, everything is identical for all practical purposes. I'm running thttpd with USE_SENDFILE enabled. Both servers are receiving approximately equal numbers of requests to download some large files. (5MB to 200MB) Server 1 can easily push 300-500mbps. I'm pretty sure it could go higher, we just run into bandwidth limits at that point. With the caps removed, we've hit 800mbps for a short while. Server 2 seems to cap out around 80mbps, with thttpd spending most of it's time in "Giant" or "kqueue" according to top, the CPU around 90% idle. What's really strange is that right when thttpd starts up on 5.2, it will start sending 200mbps or higher for the first 10-15 minutes, and then start tapering off back down to 80mbps. It doesn't look like a thttpd bug, since it seems to work flawlessly on 4.x. I know a lot of work is being done on locking, sendfile and kqueue fun, so I'm not complaining. I just wanted someone to know this was happening, and hopefully have someone point out something obvious I'm missing that could be causing this... Anyone run into this before? -- Kevin