From owner-freebsd-fs@FreeBSD.ORG Mon Sep 12 12:33:37 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6C06106564A for ; Mon, 12 Sep 2011 12:33:37 +0000 (UTC) (envelope-from prvs=1236a5984e=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 3FB8B8FC0A for ; Mon, 12 Sep 2011 12:33:36 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 12 Sep 2011 13:23:00 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 12 Sep 2011 13:22:59 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014929202.msg for ; Mon, 12 Sep 2011 13:22:58 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1236a5984e=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Laszlo KAROLYI" , References: Date: Mon, 12 Sep 2011 13:23:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-2"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: Subject: Re: ZFS-lighttpd2-sendfile, too high IO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Sep 2011 12:33:37 -0000 sendfile doesn't work as you might expect on zfs, its not zero copy due to the use of none buffer pool memory. You do get a benefit but it requires double the amount of memory to get it so we've disabled sendfile under nginx for zfs based hosts for this very reason. Regards Steve ----- Original Message ----- From: "Laszlo KAROLYI" To: Sent: Monday, September 12, 2011 12:10 PM Subject: ZFS-lighttpd2-sendfile, too high IO > Hello, Recently I installed a FreeBSD with the newest 8.2-STABLE and zfs version. I use lighttpd2 and zfs on it, and munin to > monitor the outgoing bandwidth. Zpool version 28, zfs version 5, with the latest kernel. We have a big mp3 archive (half hour, > 256kbit/s mp3-s) which lighty serves. This means full random IO. When I see that the server serves 15mbit/sec, i see constantly > 8-10Mbyte/sec reads on the zfs raidz1 array, which is too much. As I could see from truss logs, lighty uses sendfile and > writev.My settings: vfs.zfs.l2c_only_size: 15106737664 ... ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.