From owner-freebsd-fs@FreeBSD.ORG Tue Aug 27 17:24:18 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2E0BF948 for ; Tue, 27 Aug 2013 17:24:18 +0000 (UTC) (envelope-from ericbrowning@skaggscatholiccenter.org) Received: from mail-pb0-f44.google.com (mail-pb0-f44.google.com [209.85.160.44]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 097C02647 for ; Tue, 27 Aug 2013 17:24:17 +0000 (UTC) Received: by mail-pb0-f44.google.com with SMTP id xa7so5125197pbc.3 for ; Tue, 27 Aug 2013 10:24:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=T0m3+R4QEt925Efw84ImpnrETjVgXV6dam0yEYBR4Mk=; b=nBxkFrHCnMVfSp0081mKBPp+5X1wvlDSv74uwYmR3oaWsDLw7pJT8SN/1rpHUNcWlK U/yCQxcKoNhv9dsnI2hnSL5bkc8b0Bor1G16v7f4OwOyTTB5BPzxDJxLn7vNQddh5LJv gX6XbVKYzooW3L1tgKMW4DMud4QuAjT+9G08jRAbaTffMvTsfB7WmJ4pFpeXyu2JmILY 842CTmrugFwrURJqPYxhX1kZbMHnwwEnErqZ5wh8X1/ux7jJxcCaXNi1U9SDPEMmQstK Eakpp8nsggkCj2SjF7GuNKTGHXPjKu4mZjEEZFkFMdEzNY61jm8/9si/fJrFfRaxEq7u vwJg== X-Gm-Message-State: ALoCoQlpsfhbAkF5Zj+8JmJdHuTyrcWnnq9SHMiBHnTqgQtzYYYYoybf0zqz+HkhR8Djzd7C+QlN MIME-Version: 1.0 X-Received: by 10.67.30.225 with SMTP id kh1mr9438436pad.148.1377624257216; Tue, 27 Aug 2013 10:24:17 -0700 (PDT) Received: by 10.70.26.4 with HTTP; Tue, 27 Aug 2013 10:24:17 -0700 (PDT) Date: Tue, 27 Aug 2013 11:24:17 -0600 Message-ID: Subject: NFS on ZFS pure SSD pool From: Eric Browning To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2013 17:24:18 -0000 Hello, first time posting to this list. I have a new server that is not living up to the promise of SSD speeds and NFS is maxing out the CPU. I'm new to FreeBSD but I've been reading up on it as much as I can. I have obscured my IP addresses and hostname with x's so just ignore that. Server has about 200 users on it each draing under 50Mb/s peak sustained around 1-2Mb/s I've followed some network tuning guides for our I350t4 nic and that has helped with network performance somewhat but the server is still experiencing heavy load with pegging the CPU at 1250% on average with only 50Mb/s of traffin in/out of the machine. All of the network tuning came from https://calomel.org/freebsd_network_tuning.html since it was relevant to the same nic that I have. Server Specs: FreeBSD 9.1 16 cores AMDx64 64GB of ram ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe Intel DC S3500 for ZIL and enabling/disabling has made no difference Used a spare DC S3700 for the ZIL and that made no difference either. NFS v3 & v4 for Mac home folders whose Cache fodler is redirected. I've tried: Compression on/of <-- no appreciable difference Deduplication on/off <-- no appreciable difference sync=disabled and sync=standard <-- no appreciable difference setting arc cache to 56GB and also to 32GB <-- no difference in performance in terms of kern. I've tried to follow the freebsd tuning guide: https://wiki.freebsd.org/ZFSTuningGuide to no avail either. I've read everything I can find on NFS on ZFS and nothing has helped. WHere am I going wrong? Here's /boot/loader: [quote] # ZFS tuning tweaks aio_load="YES" # Async IO system calls autoboot_delay="10" # reduce boot menu delay time from 10 to 3 seconds vfs.zfs.arc_max="56868864000" # Reserves 10GB or ram for system, leaves 56GB for ZFS vfs.zfs.cache_flush_disable="1" #vfs.zfs.prefetch_disble="1" vfs.zfs.write_limit_override="429496728" kern.ipc.nmbclusters="264144" # increase the number of network mbufs kern.maxfiles="65535" net.inet.tcp.syncache.hashsize="1024" # Size of the syncache hash table net.inet.tcp.syncache.bucketlimit="100" # Limit the number of entries permitted in each bucket of the hash table. net.inet.tcp.tcbhashsize="32768" # Link Aggregation loader tweaks. see: https://calomel.org/freebsd_network_tuning.html hw.igb.enable_msix="1" hw.igb.num_queues="0" hw.igb.enable_aim="1" hw.igb.max_interrupt_rate="32000" hw.igb.rxd="2048" hw.igb.txd="2048" hw.igb.rx_process_limit="4096" if_lagg_load="YES" [/quote] Here's etc/sysctl.conf: [quote] # $FreeBSD: release/9.1.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 kern.ipc.somaxconn=1024 kern.maxusers=272 #kern.maxvnodes=1096848 #increase this if necessary kern.ipc.maxsockbuf=8388608 net.inet.tcp.mssdflt=1460 net.inet.ip.forwarding=1 net.inet.ip.fastforwarding=1 dev.igb.2.fc=0 dev.igb.3.fc=0 dev.igb.4.fc=0 dev.igb.5.fc=0 dev.igb.2.rx_procesing_limit=10000 dev.igb.3.rx_procesing_limit=10000 dev.igb.4.rx_procesing_limit=10000 dev.igb.5.rx_procesing_limit=10000 net.inet.ip.redirect=0 net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent to IP .255 net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address Mask Request packets net.inet.icmp.maskrepl=0 # replies are not sent for ICMP address mask net.inet.icmp.log_redirect=0 # do not log redirected ICMP packet attempts net.inet.icmp.drop_redirect=1 # no redirected ICMP packets net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on initial connection net.inet.tcp.ecn.enable=1 # explicit congestion notification (ecn) warning: some ISP routers abuse it net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid spoofed icmp/udp floods net.inet.tcp.maxtcptw=15000 # max number of tcp time_wait states for closing connections net.inet.tcp.msl=5000 # 5 second maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most ICMP packets are dropped by others net.inet.tcp.rfc3042=0 # disable the limited transmit mechanism which can slow burst transmissions net.inet.ip.rtexpire=60 # 3600 secs net.inet.ip.rtminexpire=2 # 10 secs net.inet.ip.rtmaxcache=1024 # 128 entries [/quote] Here's /etc/rc.conf [quote] #ifconfig_igb2=" inet xxx.xx.x.xx netmask 255.255.248.0" hostname="xxxxxxxxxxxxxxxxxxx" # # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="NO" # ### LACP config ifconfig_igb2="up" ifconfig_igb3="up" ifconfig_igb4="up" ifconfig_igb5="up" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5 xxx.xx.x.xx netmask 255.255.248.0" ipvr_addrs_lagg0="xxx.xx.x.xx" defaultrouter="xxx.xx.x.xx" # ### Defaults for SSH, NTP, ZFS sshd_enable="YES" ntpd_enable="YES" zfs_enable="YES" # ## NFS Server rpcbind_enable="YES" nfs_server_enable="YES" mountd_flags="-r -l" nfsd_enable="YES" mountd_enable="YES" rpc_lockd_enable="NO" rpc_statd_enable="NO" nfs_server_flags="-u -t -n 128" nfsv4_server_enable="YES" nfsuserd_enable="YES" [/quote] Thanks in advance, -- Eric Browning