From owner-freebsd-arm@freebsd.org Thu May 4 16:10:59 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66679D5D45C for ; Thu, 4 May 2017 16:10:59 +0000 (UTC) (envelope-from olavi.m.kumpulainen@gmail.com) Received: from mail-lf0-x236.google.com (mail-lf0-x236.google.com [IPv6:2a00:1450:4010:c07::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E5F9FAC9 for ; Thu, 4 May 2017 16:10:58 +0000 (UTC) (envelope-from olavi.m.kumpulainen@gmail.com) Received: by mail-lf0-x236.google.com with SMTP id h4so10905690lfj.3 for ; Thu, 04 May 2017 09:10:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:message-id:date:to; bh=P6+SnUXKgk/O4MsQTaBmqfT8Unef9piJAPcbLD9fLlg=; b=IotPxgNOuyPWKr/MnPMs5ujFh9eypNckCYlCk24STlRWuw/BZ2vuE5hrJ/HXOC+ZX2 TDOqpUA/uOdWtavoUvs7VQfQNxUlRvR3SPsGftZuhq9x8X0km+HJIR2NjhaoBEsngFsT YNTJbd7xrkWwcIiXYlyHFe7pW7PEL6GFxNPFByQFvH0HZQH0bC/MC3dKTJmgVkEZoqQ0 cwp9UZTNN+zR3gkEEtQcqlocWGmozAkYVEjHHjOHbserlJ32ylaTYO7U3Cx8Pjikpl6Q Ji3AK4sO6SLJ4RH8cqN68wcjXRPvBjccL2K/xkQI0Bp+z/WrB8v5ynks8ZXSUjZs8A2J jLKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:message-id:date:to; bh=P6+SnUXKgk/O4MsQTaBmqfT8Unef9piJAPcbLD9fLlg=; b=TVE2/AtM2UoyV03MCPh/rULarZcGelfsgAYD06kEnY91H617cs+xLSwly+eI44Nkus rVw4B79kMCJvCgKMWJCGt3Ig150zYgXK/w8ae1sfOh9FG0O7l2G8RO9qUOUH/rpBpT/D m/Ahtznma0Es/XB0igZ6Ax5d90lfKOUpNnz+NvMaU7IYlmFf+JbUBnop3FS0O7U+XJGh SkSYp1Wc2lzWg0S6dgkG0gSmrp1SD94vN8aPA8UozYJAcEOUauM13GuWlU2A7cd/Hqc2 ijDuRh3VV4MlsssPi9S5FxD2S4oxRNDMA/wNiE6H/04Fco8Y9qy0AWNIBQGofyTfNLo2 MJoQ== X-Gm-Message-State: AN3rC/7Y41VzXgdlvhem+HfhZQ9gzJ70WdopWS9KJOqM4giB3hGvNUjx boDFylQUrinVCGxEMdc= X-Received: by 10.25.40.129 with SMTP id o123mr12881553lfo.155.1493914256576; Thu, 04 May 2017 09:10:56 -0700 (PDT) Received: from [192.168.1.105] (c83-251-251-174.bredband.comhem.se. [83.251.251.174]) by smtp.gmail.com with ESMTPSA id r190sm518157lfr.8.2017.05.04.09.10.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 May 2017 09:10:54 -0700 (PDT) From: Olavi Kumpulainen Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: cpsw drops packets when stressed on BBB and 11.0-STABLE Message-Id: Date: Thu, 4 May 2017 18:10:55 +0200 To: freebsd-arm@freebsd.org X-Mailer: Apple Mail (2.3273) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 May 2017 16:10:59 -0000 Hi, I'm running a snapshot build of FreeBSD-11, FreeBSD beaglebone 11.0-STABLE FreeBSD 11.0-STABLE #0 r317153: Thu Apr = 20 09:21:26 UTC 2017 =20 on a BBB. I see that cpsw drops outgoing packets when stressed. =20 Out of some reason, dev.cpswss.0.stats.RxStartOfFrameOverruns increments = when packets are dropped which may be a hint on what=E2=80=99s going on.=20= The fact that RxStartOf... increases is confusing, because the packets = seem to be dropped in transmission.=20 Anyway - I=E2=80=99ve found a simple way to reproduce the problem, = namely by sending long pings. On the BBB: # tcpdump -ni cpsw0 icmp&=20 Initial state of RxStartOfFrameOverruns in BBB after playing around a = bit: # sysctl dev.cpswss.0.stats.RxStartOfFrameOverruns=20 dev.cpswss.0.stats.RxStartOfFrameOverruns: 86 =20 # ping -c 1 -s 14000 192.168.0.3 PING 192.168.0.3 (192.168.0.3): 14000 data bytes 11:36:57.965980 IP 192.168.0.158 > 192.168.0.3: ICMP echo request, id = 53762, seq 0, length 1480 11:36:57.966658 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.966826 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.966923 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.967009 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.967090 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.967173 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.967254 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.967336 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 11:36:57.967414 IP 192.168.0.158 > 192.168.0.3: ip-proto-1=20 (10 packets has supposedly been put into the tx ring in BBB)=20 Looking at RxStartOfFrameOverruns in the BBB, I see an increment by 5=E2=80= =A6 #sysctl dev.cpswss.0.stats.RxStartOfFrameOverruns=20 dev.cpswss.0.stats.RxStartOfFrameOverruns: 91 =20 I've set up a tcpdump on the target machine: $ sudo tcpdump -ni eth2 icmp 13:52:42.603199 IP 192.168.0.158 > 192.168.0.3: ICMP echo request, id = 53762, seq 0, length 1480 13:52:42.604697 IP 192.168.0.158 > 192.168.0.3: ip-proto-1=20 (Eight fragments lost!) Without tcpump in BBB, more packets seem to go through (showing tcpdump = on target); 13:56:08.396553 IP 192.168.0.158 > 192.168.0.3: ICMP echo request, id = 55554, seq 0, length 1480 13:56:08.397781 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 13:56:08.399029 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 13:56:08.400157 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 13:56:08.401409 IP 192.168.0.158 > 192.168.0.3: ip-proto-1=20 (Five packets lost)=20 Again, there's an increment in RxStartOfFrame...: # sysctl dev.cpswss.0.stats.RxStartOfFrameOverruns=20 dev.cpswss.0.stats.RxStartOfFrameOverruns: 96 =20 I added a printf in tx_enqueue() in an attempt to see what=E2=80=99s = going on, but doing so =E2=80=9Cfixed the bug=E2=80=9D =E2=80=93 = obviously by adding a delay in the forwarding code. Maybe we have a = timing/race between the driver and the cpsw hardware? Also, I tested sending 14k pings from the standard-installed Linux in = the BBB and that worked just fine. So the packets aren't lost between = the hosts (the machines are connected via the same switch). Any ideas? Cheers - /Olavi