From owner-freebsd-net@FreeBSD.ORG Thu Jul 31 17:50:20 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A241F989 for ; Thu, 31 Jul 2014 17:50:20 +0000 (UTC) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 165662104 for ; Thu, 31 Jul 2014 17:50:19 +0000 (UTC) Received: by mail-lb0-f172.google.com with SMTP id z11so2375567lbi.3 for ; Thu, 31 Jul 2014 10:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+1OWhNy4CnpwMB6YZXJFXisHpfaBWuiyDZJv2ZF9b9M=; b=PUHuPq+iqbPFqTyi1Na/neWELdaox2MLjxQri3Z3a3EiBAcSl+hGpTcMij+fnyH67m QD6OyJ8QVon+C/fU5+cK9XreaUNXz+OxXjT+TWotc9p6bFfR8AGDyEvPt3AZJJc7ASYs 5yb/KvWLuO65tUT4TLM9MEtNI65i4JLk4VHgrG9FJqelVQlNh73QRnK9gPVWOyP0R0v9 LmnMznvcd91K/rfjiY04BQubVrMsck8Dj3hpJqaJ/1UHrJYT+ZF7NbQintrggEqmumQ7 8s38MZ+K6+WuM9EEw4QQPer+/lVaayvEkSSxKavrZ+qwynx1N/khnn6T5QK37ErbhKBb nHqw== MIME-Version: 1.0 X-Received: by 10.112.171.134 with SMTP id au6mr13277684lbc.21.1406829017392; Thu, 31 Jul 2014 10:50:17 -0700 (PDT) Received: by 10.114.109.6 with HTTP; Thu, 31 Jul 2014 10:50:17 -0700 (PDT) Date: Thu, 31 Jul 2014 13:50:17 -0400 Message-ID: Subject: 4 million packets per second: Re: fastforward/routing: a 3 million packet-per-second system? From: John Jasen To: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 17:50:20 -0000 Following up, toggling hw.cxgbe.cong_drop yielded over 4 million packets per second in testing. This is probably an artifact of the switch I'm using, and should not be used unless your switch or clients don't respect ethernet pause. Turning off ip/icmp redirects yielded a little more, perhaps another 100k or so. Increasing the hw.cxgbe.n*xq10g in loader.conf (below) pushed a bottleneck from t4_transmit into kernel locking, so should be considered for busy servers. I removed the kern.sched sysctls, without notable changes to performance. It looks like the system spends a lot of time in locks, which it looks like there are patches in HEAD to address this. There are also patches in HEAD to make the cxgbe drivers work with netmap, which can also be useful. more /boot/loader.conf carp_load="YES" netmap_load="YES" cxgbe_load="YES" net.isr.maxthreads=12 net.isr.bindthreads=1 net.isr.maxqlimit=60480 net.link.ifqmaxlen=90000 # additions in discussions with cxgbe maintainer, Navdeep Parhar ( np@freebsd.org) hw.cxgbe.toecaps_allowed=0 hw.cxgbe.ntxq10g=48 hw.cxgbe.nrxq10g=16 # end cxgbe recommended additions # experimenting with cluster size, no effect hw.cxgbe.largest_rx_cluster=65536 hw.cxgbe.safest_rx_cluster=16384 #cong_drop: increased throughput by ~20%. # not recommended for use with real switches that respond to tx_pause hw.cxgbe.cong_drop=1 more /etc/sysctl.conf net.inet.ip.fastforwarding=1 kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.point_to_point=0 kern.random.sys.harvest.interrupt=0 hw.intr_storm_threshold=25000000 # not even coming close to this. could be dropped kern.ipc.maxsockbuf=258291200 net.inet.tcp.recvbuf_max=258291200 net.inet.tcp.sendbuf_max=258291200 #kern.sched.slice=1 #kern.sched.interact=1 net.inet.icmp.log_redirect=0 net.inet.icmp.redirect=0 net.inet.ip.redirect=0 net.inet.icmp.drop_redirect=1 more /etc/rc.conf hostname="fb-test" #ifconfig_igb0="DHCP" sshd_enable="YES" #ntpd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" # OpenBSD PF options #pf_enable="YES" #pflog_enable="YES" # enable packet forwarding gateway_enable="YES" ipv6_gateway_enable="YES" # enable OpenBSD ftp-procy ftpproxy_enable="YES" #aliases used below to help shift clients during testing, and artifact of #testing other vendor card ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up" ifconfig_cxl0_alias0="inet 172.16.7.1 netmask 255.255.255.0" ifconfig_cxl1_alias0="inet 172.16.8.1 netmask 255.255.255.0" ifconfig_cxl2_alias0="inet 172.16.1.1 netmask 255.255.255.0" ifconfig_cxl3_alias0="inet 172.16.2.1 netmask 255.255.255.0" # ssh interface ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0" On Tue, Jul 22, 2014 at 11:18 AM, John Jasen wrote: > >