Date: Thu, 31 Jul 2014 13:50:17 -0400 From: John Jasen <jjasen@gmail.com> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: 4 million packets per second: Re: fastforward/routing: a 3 million packet-per-second system? Message-ID: <CAACLuR1TDdQnvVttBro4sw_2WOYf7S0C5pNPQ05xXKTxjtpJVw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Following up, toggling hw.cxgbe.cong_drop yielded over 4 million packets per second in testing. This is probably an artifact of the switch I'm using, and should not be used unless your switch or clients don't respect ethernet pause. Turning off ip/icmp redirects yielded a little more, perhaps another 100k or so. Increasing the hw.cxgbe.n*xq10g in loader.conf (below) pushed a bottleneck from t4_transmit into kernel locking, so should be considered for busy servers. I removed the kern.sched sysctls, without notable changes to performance. It looks like the system spends a lot of time in locks, which it looks like there are patches in HEAD to address this. There are also patches in HEAD to make the cxgbe drivers work with netmap, which can also be useful. more /boot/loader.conf carp_load="YES" netmap_load="YES" cxgbe_load="YES" net.isr.maxthreads=12 net.isr.bindthreads=1 net.isr.maxqlimit=60480 net.link.ifqmaxlen=90000 # additions in discussions with cxgbe maintainer, Navdeep Parhar ( np@freebsd.org) hw.cxgbe.toecaps_allowed=0 hw.cxgbe.ntxq10g=48 hw.cxgbe.nrxq10g=16 # end cxgbe recommended additions # experimenting with cluster size, no effect hw.cxgbe.largest_rx_cluster=65536 hw.cxgbe.safest_rx_cluster=16384 #cong_drop: increased throughput by ~20%. # not recommended for use with real switches that respond to tx_pause hw.cxgbe.cong_drop=1 more /etc/sysctl.conf net.inet.ip.fastforwarding=1 kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.point_to_point=0 kern.random.sys.harvest.interrupt=0 hw.intr_storm_threshold=25000000 # not even coming close to this. could be dropped kern.ipc.maxsockbuf=258291200 net.inet.tcp.recvbuf_max=258291200 net.inet.tcp.sendbuf_max=258291200 #kern.sched.slice=1 #kern.sched.interact=1 net.inet.icmp.log_redirect=0 net.inet.icmp.redirect=0 net.inet.ip.redirect=0 net.inet.icmp.drop_redirect=1 more /etc/rc.conf hostname="fb-test" #ifconfig_igb0="DHCP" sshd_enable="YES" #ntpd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" # OpenBSD PF options #pf_enable="YES" #pflog_enable="YES" # enable packet forwarding gateway_enable="YES" ipv6_gateway_enable="YES" # enable OpenBSD ftp-procy ftpproxy_enable="YES" #aliases used below to help shift clients during testing, and artifact of #testing other vendor card ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl0_alias0="inet 172.16.7.1 netmask 255.255.255.0" ifconfig_cxl1_alias0="inet 172.16.8.1 netmask 255.255.255.0" ifconfig_cxl2_alias0="inet 172.16.1.1 netmask 255.255.255.0" ifconfig_cxl3_alias0="inet 172.16.2.1 netmask 255.255.255.0" # ssh interface ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" On Tue, Jul 22, 2014 at 11:18 AM, John Jasen <jjasen@gmail.com> wrote: > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAACLuR1TDdQnvVttBro4sw_2WOYf7S0C5pNPQ05xXKTxjtpJVw>