Date: Wed, 01 Apr 2015 15:03:41 -0500 From: Adam Guimont <aguimont@tezzaron.com> To: freebsd-fs@freebsd.org Subject: NFSD high CPU usage Message-ID: <551C4F1D.1000206@tezzaron.com>
next in thread | raw e-mail | index | archive | help
I have an issue where NFSD will max out the CPU (1200% in this case) when a client workstation runs out of memory while trying to write via NFS. What also happens is the TCP Recv-Q fills up and causes connection timeouts for any other client trying to use the NFS server. I can reproduce the issue by running stress on a low-end client workstation. Change into the NFS mounted directory and then use stress to write via NFS and exhaust the memory, example: stress --cpu 2 --io 4 --vm 20 --hdd 4 The client workstation will eventually run out of memory trying to write into the NFS directory, fill the TCP Recv-Q on the NFS server, and then NFSD will max out the CPU. The actual client workstations (~50) are not running stress when this happens, it's a mixture of EDA tools (simulation and verification). For what it's worth, this is how I've been monitoring the TCP buffer queues where "xx.xxx.xx.xxx" is the IP address of the NFS server: cmdwatch -n1 'netstat -an | grep -e "Proto" -e "tcp4" | grep -e "Proto" -e "xx.xxx.xx.xxx.2049"' I have tried several tuning recommendations but it has not solved the problem. Has anyone else experienced this and is anyone else able to reproduce it? --- NFS server specs: OS = FreeBSD 10.0-RELEASE CPU = E5-1650 v3 Memory = 96GB Disks = 24x ST6000NM0034 in 4x raidz2 HBA = LSI SAS 9300-8i NIC = Intel 10Gb X540-T2 --- /boot/loader.conf autoboot_delay="3" geom_mirror_load="YES" mpslsi3_load="YES" cc_htcp_load="YES" --- /etc/rc.conf hostname="***" ifconfig_ix0="inet *** netmask 255.255.248.0 -tso -vlanhwtso" defaultrouter="***" sshd_enable="YES" ntpd_enable="YES" zfs_enable="YES" sendmail_enable="NO" nfs_server_enable="YES" nfs_server_flags="-h *** -t -n 128" nfs_client_enable="YES" rpcbind_enable="YES" rpc_lockd_enable="YES" rpc_statd_enable="YES" samba_enable="YES" atop_enable="YES" atop_interval="5" zabbix_agentd_enable="YES" --- /etc/sysctl.conf vfs.nfsd.server_min_nfsvers=3 vfs.nfsd.cachetcp=0 kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendspace=1048576 net.inet.tcp.recvspace=1048576 net.inet.tcp.sendbuf_inc=32768 net.inet.tcp.recvbuf_inc=65536 net.inet.tcp.keepidle=10000 net.inet.tcp.keepintvl=2500 net.inet.tcp.always_keepalive=1 net.inet.tcp.cc.algorithm=htcp net.inet.tcp.cc.htcp.adaptive_backoff=1 net.inet.tcp.cc.htcp.rtt_scaling=1 net.inet.tcp.sack.enable=0 kern.ipc.soacceptqueue=1024 net.inet.tcp.mssdflt=1460 net.inet.tcp.minmss=1300 net.inet.tcp.tso=0 --- Client workstations: OS = CentOS 6.6 x64 Mount options from `cat /proc/mounts` = rw,nosuid,noatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=***,mountvers=3,mountport=916,mountproto=udp,local_lock=none,addr=*** --- Regards, Adam Guimont
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?551C4F1D.1000206>