Date: Tue, 27 Aug 2013 15:29:52 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Eric Browning <ericbrowning@skaggscatholiccenter.org> Cc: freebsd-fs@freebsd.org Subject: Re: NFS on ZFS pure SSD pool Message-ID: <2008996797.14358576.1377631792358.JavaMail.root@uoguelph.ca> In-Reply-To: <CAM=5oeDBJKo0qfcpaeUn6DJKB3WzxH0yhoK0U304P6S1tB3D-A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Eric Browning wrote: > Hello, first time posting to this list. I have a new server that is > not > living up to the promise of SSD speeds and NFS is maxing out the CPU. > I'm > new to FreeBSD but I've been reading up on it as much as I can. I > have > obscured my IP addresses and hostname with x's so just ignore that. > Server has about 200 users on it each draing under 50Mb/s peak > sustained > around 1-2Mb/s > > I've followed some network tuning guides for our I350t4 nic and that > has > helped with network performance somewhat but the server is still > experiencing heavy load with pegging the CPU at 1250% on average with > only > 50Mb/s of traffin in/out of the machine. All of the network tuning > came > from https://calomel.org/freebsd_network_tuning.html since it was > relevant > to the same nic that I have. > > Server Specs: > FreeBSD 9.1 > 16 cores AMDx64 > 64GB of ram > ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe > Intel DC S3500 for ZIL and enabling/disabling has made no > difference > Used a spare DC S3700 for the ZIL and that made no difference > either. > NFS v3 & v4 for Mac home folders whose Cache fodler is redirected. > > I've tried: > Compression on/of <-- no appreciable difference > Deduplication on/off <-- no appreciable difference > sync=disabled and sync=standard <-- no appreciable difference > setting arc cache to 56GB and also to 32GB <-- no difference in > performance > in terms of kern. > > I've tried to follow the freebsd tuning guide: > https://wiki.freebsd.org/ZFSTuningGuide to no avail either. I've > read > everything I can find on NFS on ZFS and nothing has helped. WHere am > I > going wrong? > You could try this patch: http://people.freebsd.org/~rmacklem/drc4-stable9.patch - After applying the patch and booting a kernel built from the patched sources, you need to increase the value of vfs.nfsd.tcphighwater. (Try something like 5000 for it as a starting point.) Although this patch is somewhat different code, it should be semantically the same as r254337 in head, that is scheduled to be MFC'd to stable/9 in a couple of weeks. rick > Here's /boot/loader: > [quote] > # ZFS tuning tweaks > aio_load="YES" # Async IO system calls > autoboot_delay="10" # reduce boot menu delay time from 10 to 3 > seconds > vfs.zfs.arc_max="56868864000" # Reserves 10GB or ram for system, > leaves > 56GB for ZFS > vfs.zfs.cache_flush_disable="1" > #vfs.zfs.prefetch_disble="1" > vfs.zfs.write_limit_override="429496728" > > kern.ipc.nmbclusters="264144" # increase the number of network > mbufs > kern.maxfiles="65535" > net.inet.tcp.syncache.hashsize="1024" # Size of the syncache hash > table > net.inet.tcp.syncache.bucketlimit="100" # Limit the number of entries > permitted in each bucket of the hash table. > net.inet.tcp.tcbhashsize="32768" > > # Link Aggregation loader tweaks. see: > https://calomel.org/freebsd_network_tuning.html > hw.igb.enable_msix="1" > hw.igb.num_queues="0" > hw.igb.enable_aim="1" > hw.igb.max_interrupt_rate="32000" > hw.igb.rxd="2048" > hw.igb.txd="2048" > hw.igb.rx_process_limit="4096" > if_lagg_load="YES" > [/quote] > > Here's etc/sysctl.conf: > [quote] > # $FreeBSD: release/9.1.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z > mux $ > # > # This file is read when going to multi-user and its contents piped > thru > # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for > details. > # > > # Uncomment this to prevent users from seeing information about > processes > that > # are being run under another UID. > #security.bsd.see_other_uids=0 > kern.ipc.somaxconn=1024 > kern.maxusers=272 > #kern.maxvnodes=1096848 #increase this if necessary > kern.ipc.maxsockbuf=8388608 > net.inet.tcp.mssdflt=1460 > net.inet.ip.forwarding=1 > net.inet.ip.fastforwarding=1 > dev.igb.2.fc=0 > dev.igb.3.fc=0 > dev.igb.4.fc=0 > dev.igb.5.fc=0 > dev.igb.2.rx_procesing_limit=10000 > dev.igb.3.rx_procesing_limit=10000 > dev.igb.4.rx_procesing_limit=10000 > dev.igb.5.rx_procesing_limit=10000 > net.inet.ip.redirect=0 > net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent > to IP > .255 > net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address > Mask > Request packets > net.inet.icmp.maskrepl=0 # replies are not sent for ICMP > address mask > net.inet.icmp.log_redirect=0 # do not log redirected ICMP packet > attempts > net.inet.icmp.drop_redirect=1 # no redirected ICMP packets > net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on > initial > connection > net.inet.tcp.ecn.enable=1 # explicit congestion notification > (ecn) > warning: some ISP routers abuse it > net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid > spoofed > icmp/udp floods > net.inet.tcp.maxtcptw=15000 # max number of tcp time_wait states > for > closing connections > net.inet.tcp.msl=5000 # 5 second maximum segment life > waiting for > an ACK in reply to a SYN-ACK or FIN-ACK > net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most > ICMP > packets are dropped by others > net.inet.tcp.rfc3042=0 # disable the limited transmit > mechanism > which can slow burst transmissions > net.inet.ip.rtexpire=60 # 3600 secs > net.inet.ip.rtminexpire=2 # 10 secs > net.inet.ip.rtmaxcache=1024 # 128 entries > [/quote] > > Here's /etc/rc.conf > [quote] > #ifconfig_igb2=" inet xxx.xx.x.xx netmask 255.255.248.0" > hostname="xxxxxxxxxxxxxxxxxxx" > # > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable > dumpdev="NO" > # > ### LACP config > ifconfig_igb2="up" > ifconfig_igb3="up" > ifconfig_igb4="up" > ifconfig_igb5="up" > cloned_interfaces="lagg0" > ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport > igb4 > laggport igb5 xxx.xx.x.xx netmask 255.255.248.0" > ipvr_addrs_lagg0="xxx.xx.x.xx" > defaultrouter="xxx.xx.x.xx" > # > ### Defaults for SSH, NTP, ZFS > sshd_enable="YES" > ntpd_enable="YES" > zfs_enable="YES" > # > ## NFS Server > rpcbind_enable="YES" > nfs_server_enable="YES" > mountd_flags="-r -l" > nfsd_enable="YES" > mountd_enable="YES" > rpc_lockd_enable="NO" > rpc_statd_enable="NO" > nfs_server_flags="-u -t -n 128" > nfsv4_server_enable="YES" > nfsuserd_enable="YES" > [/quote] > > Thanks in advance, > -- > Eric Browning > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2008996797.14358576.1377631792358.JavaMail.root>