Date: Tue, 27 Aug 2013 11:24:17 -0600 From: Eric Browning <ericbrowning@skaggscatholiccenter.org> To: freebsd-fs@freebsd.org Subject: NFS on ZFS pure SSD pool Message-ID: <CAM=5oeDBJKo0qfcpaeUn6DJKB3WzxH0yhoK0U304P6S1tB3D-A@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, first time posting to this list. I have a new server that is not living up to the promise of SSD speeds and NFS is maxing out the CPU. I'm new to FreeBSD but I've been reading up on it as much as I can. I have obscured my IP addresses and hostname with x's so just ignore that. Server has about 200 users on it each draing under 50Mb/s peak sustained around 1-2Mb/s I've followed some network tuning guides for our I350t4 nic and that has helped with network performance somewhat but the server is still experiencing heavy load with pegging the CPU at 1250% on average with only 50Mb/s of traffin in/out of the machine. All of the network tuning came from https://calomel.org/freebsd_network_tuning.html since it was relevant to the same nic that I have. Server Specs: FreeBSD 9.1 16 cores AMDx64 64GB of ram ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe Intel DC S3500 for ZIL and enabling/disabling has made no difference Used a spare DC S3700 for the ZIL and that made no difference either. NFS v3 & v4 for Mac home folders whose Cache fodler is redirected. I've tried: Compression on/of <-- no appreciable difference Deduplication on/off <-- no appreciable difference sync=disabled and sync=standard <-- no appreciable difference setting arc cache to 56GB and also to 32GB <-- no difference in performance in terms of kern. I've tried to follow the freebsd tuning guide: https://wiki.freebsd.org/ZFSTuningGuide to no avail either. I've read everything I can find on NFS on ZFS and nothing has helped. WHere am I going wrong? Here's /boot/loader: [quote] # ZFS tuning tweaks aio_load="YES" # Async IO system calls autoboot_delay="10" # reduce boot menu delay time from 10 to 3 seconds vfs.zfs.arc_max="56868864000" # Reserves 10GB or ram for system, leaves 56GB for ZFS vfs.zfs.cache_flush_disable="1" #vfs.zfs.prefetch_disble="1" vfs.zfs.write_limit_override="429496728" kern.ipc.nmbclusters="264144" # increase the number of network mbufs kern.maxfiles="65535" net.inet.tcp.syncache.hashsize="1024" # Size of the syncache hash table net.inet.tcp.syncache.bucketlimit="100" # Limit the number of entries permitted in each bucket of the hash table. net.inet.tcp.tcbhashsize="32768" # Link Aggregation loader tweaks. see: https://calomel.org/freebsd_network_tuning.html hw.igb.enable_msix="1" hw.igb.num_queues="0" hw.igb.enable_aim="1" hw.igb.max_interrupt_rate="32000" hw.igb.rxd="2048" hw.igb.txd="2048" hw.igb.rx_process_limit="4096" if_lagg_load="YES" [/quote] Here's etc/sysctl.conf: [quote] # $FreeBSD: release/9.1.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 kern.ipc.somaxconn=1024 kern.maxusers=272 #kern.maxvnodes=1096848 #increase this if necessary kern.ipc.maxsockbuf=8388608 net.inet.tcp.mssdflt=1460 net.inet.ip.forwarding=1 net.inet.ip.fastforwarding=1 dev.igb.2.fc=0 dev.igb.3.fc=0 dev.igb.4.fc=0 dev.igb.5.fc=0 dev.igb.2.rx_procesing_limit=10000 dev.igb.3.rx_procesing_limit=10000 dev.igb.4.rx_procesing_limit=10000 dev.igb.5.rx_procesing_limit=10000 net.inet.ip.redirect=0 net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent to IP .255 net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address Mask Request packets net.inet.icmp.maskrepl=0 # replies are not sent for ICMP address mask net.inet.icmp.log_redirect=0 # do not log redirected ICMP packet attempts net.inet.icmp.drop_redirect=1 # no redirected ICMP packets net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on initial connection net.inet.tcp.ecn.enable=1 # explicit congestion notification (ecn) warning: some ISP routers abuse it net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid spoofed icmp/udp floods net.inet.tcp.maxtcptw=15000 # max number of tcp time_wait states for closing connections net.inet.tcp.msl=5000 # 5 second maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most ICMP packets are dropped by others net.inet.tcp.rfc3042=0 # disable the limited transmit mechanism which can slow burst transmissions net.inet.ip.rtexpire=60 # 3600 secs net.inet.ip.rtminexpire=2 # 10 secs net.inet.ip.rtmaxcache=1024 # 128 entries [/quote] Here's /etc/rc.conf [quote] #ifconfig_igb2=" inet xxx.xx.x.xx netmask 255.255.248.0" hostname="xxxxxxxxxxxxxxxxxxx" # # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="NO" # ### LACP config ifconfig_igb2="up" ifconfig_igb3="up" ifconfig_igb4="up" ifconfig_igb5="up" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5 xxx.xx.x.xx netmask 255.255.248.0" ipvr_addrs_lagg0="xxx.xx.x.xx" defaultrouter="xxx.xx.x.xx" # ### Defaults for SSH, NTP, ZFS sshd_enable="YES" ntpd_enable="YES" zfs_enable="YES" # ## NFS Server rpcbind_enable="YES" nfs_server_enable="YES" mountd_flags="-r -l" nfsd_enable="YES" mountd_enable="YES" rpc_lockd_enable="NO" rpc_statd_enable="NO" nfs_server_flags="-u -t -n 128" nfsv4_server_enable="YES" nfsuserd_enable="YES" [/quote] Thanks in advance, -- Eric Browning
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM=5oeDBJKo0qfcpaeUn6DJKB3WzxH0yhoK0U304P6S1tB3D-A>