From owner-freebsd-net@FreeBSD.ORG Wed May 13 20:20:04 2009 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9DD4106564A for ; Wed, 13 May 2009 20:20:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D79AD8FC1D for ; Wed, 13 May 2009 20:20:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n4DKK4jF009870 for ; Wed, 13 May 2009 20:20:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n4DKK4h1009869; Wed, 13 May 2009 20:20:04 GMT (envelope-from gnats) Date: Wed, 13 May 2009 20:20:04 GMT Message-Id: <200905132020.n4DKK4h1009869@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: "Seth Mos" Cc: Subject: kern/127528: [icmp]: icmp socket receives icmp replies not owned by the process. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Seth Mos List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 May 2009 20:20:05 -0000 The following reply was made to PR kern/127528; it has been noted by GNATS. From: "Seth Mos" To: bug-followup@FreeBSD.org, seth.mos@xs4all.nl Cc: Subject: kern/127528: [icmp]: icmp socket receives icmp replies not owned by the process. Date: Wed, 13 May 2009 21:58:40 +0200 (CEST) Hi there again, it's been a while. We are currently on a pfSense 1.2.3-RC1 release and we are now getting a host of people on the forum and in the mailing lists that are complaining about the load balancer flapping. Some debugging turned out some very weird behaviour. Every now and then, depending on the concurrenty of icmp traffic originating from the FreeBSD host, ping will miss icmp replies. Here are the steps performed to debug this issue. For example the output tcpdump -ni vlan0 -t icmp Where vlan0 is the external interface. The IP address being pinged is a local gateway connected by ethernet so there is nothing but the switch in between. http://pastebin.com/f6608c875 The ping command issued is /sbin/ping -t 4 -oqc 5 -i 0.7 213.23.199.249 This command line will ping the target and exit with a return status of 0 when it receives a reply which can be any of the 5 attempts within the maximum of 4 seconds. The output clearly shows that all attempts have unique process IDs, which is good. It also shows that all attempt have a sequence number of 0 which is the 1st attempt to ping. There is however one attempt where this is different, on attempt with pid 46060 you will see that it attempted 4 times to ping the target, it got valid responses to all 5 requests!. However, ping exited with a return status of 1 which would imply a timeout. So now it is worse then the originally reported issue we had on 7.0. We can now make the FreeBSD distributed ping fail. Specifically, it fails to see the valid replies. For your reference I also pasted the tcpdump below. I can also reproduce this exact same behaviour by using fping. So I would not consider this a bug in the ping binary itself. Since the original PR had this issue with apinger, which is different from both ping and fping I am quite confident there is a serious regression in here. # uname -a FreeBSD noord.coltex.nl 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #0: Tue Mar 24 01:18:19 EDT 2009 sullrich@RELENG_1_2-snapshots.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7 i386 Kind regards, Seth Mos pfSense Developer -------------- 1. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 25068, seq 0, length 64 2. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 25068, seq 0, length 64 3. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 27372, seq 0, length 64 4. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 27372, seq 0, length 64 5. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 29676, seq 0, length 64 6. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 29676, seq 0, length 64 7. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 31468, seq 0, length 64 8. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 31468, seq 0, length 64 9. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 33772, seq 0, length 64 10. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 33772, seq 0, length 64 11. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 36076, seq 0, length 64 12. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 36076, seq 0, length 64 13. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq 0, length 64 14. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq 0, length 64 15. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq 1, length 64 16. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq 1, length 64 17. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 12525, seq 0, length 64 18. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 12525, seq 0, length 64 19. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 14829, seq 0, length 64 20. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 14829, seq 0, length 64 21. IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.120.254 unreachable - admin prohibited filter, length 60 22. IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.148.1 unreachable - admin prohibited filter, length 60 23. IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.112.1 unreachable - admin prohibited filter, length 60 24. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq 2, length 64 25. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq 2, length 64 26. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 17645, seq 0, length 64 27. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 17645, seq 0, length 64 28. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq 3, length 64 29. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq 3, length 64 30. IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq 4, length 64 31. IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq 4, length 64