Date: Tue, 1 Dec 2015 12:22:38 +0100 (CET) From: "elof2@sentor.se" <elof2@sentor.se> To: "Alexander V. Chernikov" <melifaro@ipfw.ru> Cc: freebsd-net <freebsd-net@freebsd.org> Subject: Re: ngrep/ixgbe bpf bug Message-ID: <alpine.BSF.2.00.1512011159060.54839@farmermaggot.shire.sentor.se> In-Reply-To: <944961448953115@web11h.yandex.ru> References: <alpine.BSF.2.00.1511051551350.49057@farmermaggot.shire.sentor.se> <alpine.BSF.2.00.1511301707450.54839@farmermaggot.shire.sentor.se> <944961448953115@web11h.yandex.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Yes, 100% of the traffic is vlan-tagged, but I get the same results with:
ngrep -d ix1 "q" vlan
no matches
If I invert the test to show all packets that do not contain "foobar", it
still matches 0 packets:
ngrep -d ix1 -v "foobar"
ix1: no IPv4 address assigned: Can't assign requested address
interface: ix1
don't match: foobar
^Cexit
263567 received, 0 dropped
So 263567 vlan-tagged packets are received, 0 dropped.
There are "q":s in there, and I promise that they don't all contain
"foobar". :)
If I add -R, to not drop privileges, I get the same result.
'tcpdump -i ix1 -lnes0' show packets just as it should:
12:14:02.113842 28:c0:da:db:c7:40 > 00:e0:20:11:0a:95, ethertype 802.1Q
(0x8100), length 139: vlan 123, p 0, ethertype IPv4, 10.123.123.123.993 >
10.321.321.321.50904: Flags [P.], seq 2632602368:2632602437, ack
2214392113, win 8316, options [nop,nop,TS val 3824950099 ecr 179534445],
length 69
...and so on... Everything looks just as expected.
#ifconfig ix1
ix1: flags=488c3<UP,BROADCAST,RUNNING,NOARP,SIMPLEX,MULTICAST,MONITOR> metric 0 mtu 1500
options=8403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 0c:c4:7a:58:e2:3d
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
Could ngrep fail because of monitor mode?
#ifconfig ix1 -monitor
#ifconfig ix1
ix1: flags=88c3<UP,BROADCAST,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 0c:c4:7a:58:e2:3d
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
#ngrep -d ix1 -v "foobar"
ix1: no IPv4 address assigned: Can't assign requested address
interface: ix1
don't match: foobar
^Cexit
157082 received, 0 dropped
Nope, same-same. Zero BPF matches.
#netstat -B
Pid Netif Flags Recv Drop Match Sblen Hblen Command
12293 ix1 p--s--- 157082 0 0 0 0 ngrep
^^^^^^
/Elof
On Tue, 1 Dec 2015, Alexander V. Chernikov wrote:
> Do you have vlans on top of ixgbe?
> And actually I wonder what does tcpdump show for the same expression.
> ( and tcpdump -i ixX -lnes0 might provide good traces on what is going on)
>
> 30.11.2015, 19:09, "elof2@sentor.se" <elof2@sentor.se>:
>> No one has a theory?
>>
>> /Elof
>>
>> On Thu, 5 Nov 2015, elof2@sentor.se wrote:
>>
>>> Hi all!
>>>
>>> Why don't ngrep work on ix interfaces?
>>>
>>> It shows nice values if I sniff on other interfaces, e.g. igb0 and
>>> bridge0:
>>>
>>> # ngrep -d igb0 "q" ip
>>> I see some matching packets. Everything looks good.
>>>
>>> # netstat -B
>>> Pid Netif Flags Recv Drop Match Sblen Hblen Command
>>> 1800 igb0 p--s--- 135 0 129 380 0 ngrep
>>> The BPF stats show Recv and Match values. Good.
>>>
>>> # ngrep -d bridge0 "q" ip
>>> I see some matching packets. Good.
>>>
>>> # netstat -B
>>> Pid Netif Flags Recv Drop Match Sblen Hblen Command
>>> 1901 bridge0 p--s--- 661897 0 659170 425606 0 ngrep
>>> Again, the BPF stats show Recv and Match values. Good.
>>>
>>> However, if I sniff on an ix interface:
>>>
>>> # ngrep -d ix1 "q" ip
>>> I get no matching packets!
>>>
>>> # netstat -B
>>> Pid Netif Flags Recv Drop Match Sblen Hblen Command
>>> 1816 ix1 p--s--- 45340 0 0 0 0 ngrep
>>> ^^^
>>> ...and the BPF stats always show zero Matches.
>>>
>>> Bug in the ixgbe driver or in ngrep?
>>>
>>> /Elof
>>>
>>> _______________________________________________
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>>
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
From owner-freebsd-net@freebsd.org Tue Dec 1 15:05:41 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id A93CEA2C34E
for <freebsd-net@mailman.ysv.freebsd.org>;
Tue, 1 Dec 2015 15:05:41 +0000 (UTC)
(envelope-from feld@FreeBSD.org)
Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com
[66.111.4.28])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 7DE371858
for <freebsd-net@freebsd.org>; Tue, 1 Dec 2015 15:05:41 +0000 (UTC)
(envelope-from feld@FreeBSD.org)
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
by mailout.nyi.internal (Postfix) with ESMTP id 01DDE20EE4
for <freebsd-net@freebsd.org>; Tue, 1 Dec 2015 10:05:33 -0500 (EST)
Received: from web3 ([10.202.2.213])
by compute3.internal (MEProxy); Tue, 01 Dec 2015 10:05:34 -0500
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=
messagingengine.com; h=content-transfer-encoding:content-type
:date:from:in-reply-to:message-id:mime-version:references
:subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=h6Km/b5eBPbP9pM
7oWEY80aCbf8=; b=erm4IwEGUa/MDL97e4qYHQfV4bnCl+VHV1LYNriQWJi/RHt
aRHsQN95VM3i98AEQsO/NmN6v9YRpPQeUxOdD+Rk98wuKF/crHDYPhbeq5TtsDZf
qACB0Tc6GcSRE406LLWQnbgh6vt6PVIHFkajAVlwQtUlbYOTZubgyxNX1Cn8=
Received: by web3.nyi.internal (Postfix, from userid 99)
id B156810BE42; Tue, 1 Dec 2015 10:05:33 -0500 (EST)
Message-Id: <1448982333.1269981.454734633.11BA4DB2@webmail.messagingengine.com>
X-Sasl-Enc: /aLTHO70yUjVu3j9jc57/mB3bySOz+ZeqX6SA9T1yJ8n 1448982333
From: Mark Felder <feld@FreeBSD.org>
To: wishmaster <artemrts@ukr.net>, freebsd-net@freebsd.org
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
X-Mailer: MessagingEngine.com Webmail Interface - ajax-b94e6169
In-Reply-To: <1448956697.854911427.15is5btc@frv34.fwdcdn.com>
References: <1448920706.962818.454005905.61CF9154@webmail.messagingengine.com>
<1448956697.854911427.15is5btc@frv34.fwdcdn.com>
Subject: Re: IPFW blocked my IPv6 NTP traffic
Date: Tue, 01 Dec 2015 09:05:33 -0600
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>;
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 01 Dec 2015 15:05:41 -0000
On Tue, Dec 1, 2015, at 02:02, wishmaster wrote:
>
> Hi, Mark.
>
>
> > I'm hoping someone can explain what happened here and this isn't a bug,
> > but if it is a bug I'll gladly open a PR.
> >
> > I noticed in my ipfw logs that I was getting a log of "DENY" entries for
> > an NTP server
> >
> > Nov 30 13:35:16 gw kernel: ipfw: 4540 Deny UDP
> > [2604:a880:800:10::bc:c004]:123 [2001:470:1f11:1e8::2]:58285 in via gif0
> >
> > Strange... I looked at ntpq output and sure enough I was trying to
> > communicate with that server. But why was it getting blocked? I don't
> > have a rule to allow IPv4 input from source port 123. I expected IPFW to
> > handle this for me. I know UDP is stateless, but firewalls are usually
> > able to "keep state" for UDP. I looked at my v4 rules which and I have
> > keep-state on there:
> >
> > # Allow all outgoing, skip to NAT
> > ######################################
> > $cmd 01300 skipto 5000 tcp from any to any out via $pif $ks
> > $cmd 01310 skipto 5000 udp from any to any out via $pif $ks
> > $cmd 01320 skipto 5000 icmp from any to any out via $pif
> > ######################################
> >
> > I noticed my outbound IPv6 didn't have $ks for udp, so I added it.
> > However, that had no effect. The solution was to add an incoming rule:
> >
> > $cmd 03755 allow udp from any to any src-port 123 in via $pif6 $ks
> >
> > This seems wrong. Thoughts?
> >
>
> What is your 5000 rule?
>
$cmd 05000 nat 1 ip4 from any to any out via $pif
> In general on public interface you should:
> $cmd 12345 allow log all from any to me 123 $ks
>
> And for outgoing traffic just:
> $cmd 1234 allow log all from me to any $ks
>
> This works for me.
>
Sure, I understand the risks and the workaround here but it just seems
odd that I have to do this for ipv6 when ipv4 has worked fine. With UDP
not really having "state" I understand why that traffic was blocked
because the NTP server was trying to send data to my port 58285. I did a
tcpdump and restarted ntpd to see what I could find other instances of
this in the ipv4 traffic:
root@gw:/usr/home/feld # tcpdump -i re0 -nttt dst port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:00:00.000000 IP 68.117.126.78.123 > 129.6.15.29.123: NTPv4, Client,
length 48
00:00:00.105172 IP 129.6.15.29.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:01.894636 IP 68.117.126.78.123 > 209.114.111.1.123: NTPv4, Client,
length 48
00:00:00.000068 IP 68.117.126.78.123 > 129.6.15.29.123: NTPv4, Client,
length 48
00:00:00.065549 IP 209.114.111.1.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:00.045821 IP 129.6.15.29.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:01.888500 IP 68.117.126.78.123 > 209.114.111.1.123: NTPv4, Client,
length 48
00:00:00.000066 IP 68.117.126.78.123 > 129.6.15.29.123: NTPv4, Client,
length 48
00:00:00.064383 IP 209.114.111.1.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:00.042029 IP 129.6.15.29.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:01.893509 IP 68.117.126.78.123 > 209.114.111.1.123: NTPv4, Client,
length 48
00:00:00.000061 IP 68.117.126.78.123 > 129.6.15.29.123: NTPv4, Client,
length 48
00:00:00.064418 IP 209.114.111.1.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:00.044275 IP 129.6.15.29.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:01.891510 IP 68.117.126.78.123 > 209.114.111.1.123: NTPv4, Client,
length 48
00:00:00.000068 IP 68.117.126.78.123 > 129.6.15.29.123: NTPv4, Client,
length 48
00:00:00.063342 IP 209.114.111.1.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:00.044807 IP 129.6.15.29.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:01.891549 IP 68.117.126.78.123 > 209.114.111.1.123: NTPv4, Client,
length 48
00:00:00.000067 IP 68.117.126.78.123 > 129.6.15.29.123: NTPv4, Client,
length 48
00:00:00.065415 IP 209.114.111.1.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:00.038239 IP 68.117.126.78.16205 > 69.164.201.165.123: NTPv4,
Client, length 48
00:00:01.896362 IP 68.117.126.78.123 > 209.114.111.1.123: NTPv4, Client,
length 48
00:00:00.064590 IP 209.114.111.1.123 > 68.117.126.78.123: NTPv4, Server,
length 48
00:00:06.057206 IP 68.117.126.78.16205 > 4.53.160.75.123: NTPv4, Client,
length 48
00:00:08.003668 IP 68.117.126.78.16205 > 72.20.40.62.123: NTPv4, Client,
length 48
00:00:08.051264 IP 68.117.126.78.16205 > 104.131.53.252.123: NTPv4,
Client, length 48
Notice how almost all of them are port 123 on both sides, but a few of
them are not. Why? The RFC says that NTP is supposed to be using port
123 as both the source and destination port, but I clearly have
something happening on port 16205. Is something screwy with ntpd in
CURRENT?
--
Mark Felder
ports-secteam member
feld@FreeBSD.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1512011159060.54839>