Date: Mon, 30 Jun 2008 21:57:18 -0400 From: Mike Tancsa <mike@sentex.net> To: Steve Bertrand <steve@ibctech.ca> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Paul <paul@gtcomm.net> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <200807010157.m611vAOt083163@lava.sentex.ca> In-Reply-To: <48695BA6.7060207@ibctech.ca> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48695BA6.7060207@ibctech.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
At 06:18 PM 6/30/2008, Steve Bertrand wrote:
>Mike Tancsa wrote:
>>At 04:04 AM 6/29/2008, Paul wrote:
>>>This is just a question but who can get more than 400k pps
>>>forwarding performance ?
>>
>>OK, I setup 2 boxes on either end of a RELENG_7 box from about May
>>7th just now, to see with 2 boxes blasting across it how it would work.
>>*However*, this is with no firewall loaded and, I must enable ip
>>fast forwarding. Without that enabled, the box just falls over.
>>even at 20Kpps, I start seeing all sorts of messages spewing to
>>route -n monitor
>>
>>got message of size 96 on Mon Jun 30 15:39:10 2008
>>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0,
>>errno 0, flags:<DONE>
>>locks: inits:
>>sockaddrs: <DST>
>> default
>
>Mike,
>
>Is the monitor running on the 7.0 box in the middle you are testing?
In the middle
On the box that has
0[r7-router]# ifconfig -u
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1b:21:08:32:a8
inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1b:21:08:32:a9
inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:30:48:90:4c:ff
inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet 127.0.0.1 netmask 0xff000000
0[r7-router]#
If I dont have fast forwarding on, yes it seems one packet generates
one error message per packet forwarded. This sure does feel like a
driver bug, but I just tried on another machine with rl0 acting as a
forwarding interface and it too generates a message per packet forwarded!
From one end point that sends packets across r7-router
# ping -c3 -S 10.10.1.2 10.20.1.3
PING 10.20.1.3 (10.20.1.3) from 10.10.1.2: 56 data bytes
64 bytes from 10.20.1.3: icmp_seq=0 ttl=63 time=0.349 ms
64 bytes from 10.20.1.3: icmp_seq=1 ttl=63 time=0.319 ms
64 bytes from 10.20.1.3: icmp_seq=2 ttl=63 time=0.343 ms
--- 10.20.1.3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.319/0.337/0.349/0.013 ms
Here are all the messages generated on the intermediary router
1[r7-router]# route -n monitor
got message of size 96 on Mon Jun 30 21:36:41 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:41 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:42 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:42 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:42 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:43 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:43 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
got message of size 96 on Mon Jun 30 21:36:44 2008
RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno
0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
default
If I turn on fast forwarding on r7-router, route -n monitor doesnt
spew out messages
>I set up the same configuration, and even with almost no load (<
>1Kpps) can replicate these error messages by making the remote IP
>address (in your case 'default', disappear (ie: unplug the cable, DDoS etc).
>
>...to further, I can even replicate the problem at a single packet
>per second by trying to ping an IP address that I know for fact that
>the router can not get to.
I get it when I can ping a valid host on the other side, which responds
>Do you see these error messages if you set up a loopback address
>with an IP on the router, and effectively chop your test environment
>in half? In your case, can the router in the middle actually get to
>a default gateway for external addresses (when I perform the test,
>your 'default' is substituted with the IP I am trying to reach, so I
>am only assuming that 'default' is implying default gateway).
No, its only when the box is forwarding packets. If I do the
following on the router,
1[r7-router]# sysctl -w net.inet.ip.fastforwarding=0
net.inet.ip.fastforwarding: 1 -> 0
0[r7-router]#
0[r7-router]# ifconfig lo0 10.40.1.1/32 alias
0[r7-router]# route -n monitor
...all is quiet on the router if I do the following
# ping -c3 -S 10.10.1.2 10.40.1.1
PING 10.40.1.1 (10.40.1.1) from 10.10.1.2: 56 data bytes
64 bytes from 10.40.1.1: icmp_seq=0 ttl=64 time=0.221 ms
64 bytes from 10.40.1.1: icmp_seq=1 ttl=64 time=0.177 ms
64 bytes from 10.40.1.1: icmp_seq=2 ttl=64 time=0.213 ms
--- 10.40.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.177/0.204/0.221/0.019 ms
---Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200807010157.m611vAOt083163>