From owner-freebsd-fs@FreeBSD.ORG Mon Sep 12 11:37:27 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A41CA106566C for ; Mon, 12 Sep 2011 11:37:27 +0000 (UTC) (envelope-from szoftos@freemail.hu) Received: from fmx05.freemail.hu (fmx05.freemail.hu [195.228.245.55]) by mx1.freebsd.org (Postfix) with SMTP id 1AB2F8FC16 for ; Mon, 12 Sep 2011 11:37:26 +0000 (UTC) Received: (qmail 52239 invoked from network); 12 Sep 2011 13:10:45 +0200 Received: from 195.228.245.211 (HELO localhost) (91.82.87.114) by fmx05.freemail.hu with SMTP; 12 Sep 2011 13:10:45 +0200 Date: Mon, 12 Sep 2011 13:10:45 +0200 (CEST) From: Laszlo KAROLYI To: freebsd-fs@freebsd.org Message-ID: X-Originating-IP: [91.82.87.114] X-HTTP-User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101 Firefox/5.0 MIME-Version: 1.0 Content-Type: TEXT/plain; CHARSET=ISO-8859-2 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS-lighttpd2-sendfile, too high IO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Sep 2011 11:37:27 -0000 Hello, Recently I installed a FreeBSD with the newest 8.2-STABLE and zfs version. I use lighttpd2 and zfs on it, and munin to monitor the outgoing bandwidth. Zpool version 28, zfs version 5, with the latest kernel. We have a big mp3 archive (half hour, 256kbit/s mp3-s) which lighty serves. This means full random IO. When I see that the server serves 15mbit/sec, i see constantly 8-10Mbyte/sec reads on the zfs raidz1 array, which is too much. As I could see from truss logs, lighty uses sendfile and writev.My settings: vfs.zfs.l2c_only_size: 15106737664 vfs.zfs.mfu_ghost_data_lsize: 661388288 vfs.zfs.mfu_ghost_metadata_lsize: 345885696 vfs.zfs.mfu_ghost_size: 1007273984 vfs.zfs.mfu_data_lsize: 1440963584 vfs.zfs.mfu_metadata_lsize: 24143872 vfs.zfs.mfu_size: 1523631104 vfs.zfs.mru_ghost_data_lsize: 5427200 vfs.zfs.mru_ghost_metadata_lsize: 522937344 vfs.zfs.mru_ghost_size: 528364544 vfs.zfs.mru_data_lsize: 1384169984 vfs.zfs.mru_metadata_lsize: 200904704 vfs.zfs.mru_size: 1728416256 vfs.zfs.anon_data_lsize: 0 vfs.zfs.anon_metadata_lsize: 0 vfs.zfs.anon_size: 1736192 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.arc_meta_limit: 1775121408 vfs.zfs.arc_meta_used: 895904760 vfs.zfs.arc_min: 887560704 vfs.zfs.arc_max: 7100485632 vfs.zfs.dedup.prefetch: 1 vfs.zfs.mdcomp_disable: 0 vfs.zfs.write_limit_override: 0 vfs.zfs.write_limit_inflated: 25327300608 vfs.zfs.write_limit_max: 1055304192 vfs.zfs.write_limit_min: 33554432 vfs.zfs.write_limit_shift: 3 vfs.zfs.no_write_throttle: 0 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 0 vfs.zfs.mg_alloc_failures: 8 vfs.zfs.check_hostid: 1 vfs.zfs.recover: 0 vfs.zfs.txg.synctime_ms: 1000 vfs.zfs.txg.timeout: 5 vfs.zfs.scrub_limit: 10 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 0 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.write_gap_limit: 4096 vfs.zfs.vdev.read_gap_limit: 32768 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 8 vfs.zfs.vdev.bio_flush_disable: 0 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_replay_disable: 0 vfs.zfs.zio.use_uma: 0 vfs.zfs.version.zpl: 5 vfs.zfs.version.spa: 28 vfs.zfs.version.acl: 1 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 Any suggestions? Thanks,Laszlo