From owner-freebsd-questions@FreeBSD.ORG Tue Jan 3 06:40:55 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7454E106564A for ; Tue, 3 Jan 2012 06:40:55 +0000 (UTC) (envelope-from azanar@carrel.org) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id E21678FC1B for ; Tue, 3 Jan 2012 06:40:54 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so27142888wgb.31 for ; Mon, 02 Jan 2012 22:40:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.227.198.142 with SMTP id eo14mr49900157wbb.28.1325571461311; Mon, 02 Jan 2012 22:17:41 -0800 (PST) Received: by 10.216.12.148 with HTTP; Mon, 2 Jan 2012 22:17:41 -0800 (PST) Date: Mon, 2 Jan 2012 22:17:41 -0800 Message-ID: From: Ed Carrel To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: pf not seeing inbound packets on netgraph interface X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jan 2012 06:40:55 -0000 Hi freebsd-questions, I am running into a roadblock getting PF to filter traffic on a Netgraph interface representing an L2TP/IPSec connection. I have done some narrowing down of the problem, but was hoping to get some advice on figuring out where to go digging next, or things to try. Also, please let me know if I should ask this on another list. For context, here is what I have setup so far: I am running FreeBSD 8.2 with IPSec support compiled into the kernel. Here's the details from uname: # uname -a FreeBSD **** 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #2: Sun Nov 27 04:12:16 PST 2011 **** i386 I am following what seems like a typical setup of racoon(8) and setkey(8), and am having mpd5 handle construction of the L2TP nodes in netgraph. I can provide the details on the configuration of each of these, if desired. When I startup racoon in the foreground and ask mpd to construct an L2TP link, I can see both the IPSec tunnel and the L2TP link establish without a problem. I am able to ping the remote end, and, if I set up a routing rule, can contact and ssh to hosts at the other end of the tunnel. Here's how netgraph sees the world when thing are established: # ngctl list There are 13 total nodes: Name: Type: ksocket ID: 000001b3 Num hooks: 1 Name: Type: l2tp ID: 000001b1 Num hooks: 3 Name: Type: socket ID: 000001b0 Num hooks: 1 Name: ng0 Type: iface ID: 000001b6 Num hooks: 1 Name: ngctl26124 Type: socket ID: 000001bd Num hooks: 0 Name: ngctl19375 Type: socket ID: 000000ba Num hooks: 0 Name: mpd25875-stats Type: socket ID: 000001b8 Num hooks: 0 Name: mpd25875-WPLink-lt Type: tee ID: 000001af Num hooks: 2 Name: mpd25875-cso Type: socket ID: 000001ad Num hooks: 0 Name: mpd25875-eso Type: socket ID: 000001ae Num hooks: 0 Name: mpd25875-lso Type: socket ID: 000001ac Num hooks: 1 Name: mpd25875-WPBundle-1 Type: ppp ID: 000001b7 Num hooks: 3 Name: ng0-tee Type: tee ID: 000001b9 Num hooks: 2 # The problem I have is that PF only sees traffic on the outbound side of the netgraph interface. But, the rest of the network stack appears to see data going both ways: # ifconfig ng0 ng0: flags=88d1 metric 0 mtu 1322 inet 10.10.7.40 --> 10.10.0.2 netmask 0xffffffff # pfctl -vvs Interfaces -i ng0 ng0 Cleared: Sun Dec 25 21:14:44 2011 References: [ States: 0 Rules: 9 ] In4/Pass: [ Packets: 0 Bytes: 0 ] In4/Block: [ Packets: 0 Bytes: 0 ] Out4/Pass: [ Packets: 5555 Bytes: 446225 ] Out4/Block: [ Packets: 622 Bytes: 56336 ] In6/Pass: [ Packets: 0 Bytes: 0 ] In6/Block: [ Packets: 0 Bytes: 0 ] Out6/Pass: [ Packets: 0 Bytes: 0 ] Out6/Block: [ Packets: 0 Bytes: 0 ] # netstat -I ng0 -bn Name Mtu Network Address Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll ng0 1322 56 0 0 5069 98 0 6032 0 ng0 1322 10.10.7.40/32 10.10.7.40 56 - - 5069 54 - 3472 - I have stood up this interface several times, hence the differing numbers between the PF and netstat. The cause for concern is the lack of packets going through PF when inbound on ng0 -- no problem both seeing them and applying rules going outbound. There isn't a peep about the inbound traffic within pflog0, either. I can see traffic going both ways in tcpdump, and nothing looks peculiar about the packets. # tcpdump -c 10 -i ng0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ng0, link-type NULL (BSD loopback), capture size 96 bytes 22:06:37.201732 IP 10.10.7.40.43113 > 10.10.4.3.ssh: Flags [S], seq 3442571726, win 65535, options [mss 1282,nop,wscale 3,sackOK,TS val 694436002 ecr 0], length 0 22:06:37.263336 IP 10.10.4.3.ssh > 10.10.7.40.43113: Flags [S.], seq 1974232057, ack 3442571727, win 14480, options [mss 1282,sackOK,TS val 370681934 ecr 694436002,nop,wscale 7], length 0 22:06:37.263577 IP 10.10.7.40.43113 > 10.10.4.3.ssh: Flags [.], ack 1, win 8255, options [nop,nop,TS val 694436064 ecr 370681934], length 0 22:06:37.282835 IP 10.10.4.3.ssh > 10.10.7.40.43113: Flags [P.], ack 1, win 114, options [nop,nop,TS val 370681940 ecr 694436064], length 21 22:06:37.283931 IP 10.10.7.40.43113 > 10.10.4.3.ssh: Flags [P.], ack 22, win 8255, options [nop,nop,TS val 694436084 ecr 370681940], length 40 22:06:37.300729 IP 10.10.4.3.ssh > 10.10.7.40.43113: Flags [.], ack 41, win 114, options [nop,nop,TS val 370681945 ecr 694436084], length 0 22:06:37.300943 IP 10.10.7.40.43113 > 10.10.4.3.ssh: Flags [P.], ack 22, win 8255, options [nop,nop,TS val 694436101 ecr 370681945], length 848 22:06:37.304154 IP 10.10.4.3.ssh > 10.10.7.40.43113: Flags [P.], ack 41, win 114, options [nop,nop,TS val 370681945 ecr 694436084], length 984 22:06:37.372460 IP 10.10.4.3.ssh > 10.10.7.40.43113: Flags [.], ack 889, win 127, options [nop,nop,TS val 370681967 ecr 694436101], length 0 22:06:37.372663 IP 10.10.7.40.43113 > 10.10.4.3.ssh: Flags [P.], ack 1006, win 8255, options [nop,nop,TS val 694436173 ecr 370681945], length 24 10 packets captured 22 packets received by filter 0 packets dropped by kernel As I noted above, I can interact with hosts over the tunnel so long as PF blindly passes traffic. Attempting to do any sort of stateful connection tracking causes PF to litter /var/log/messages with notices of a "loose state match," which I think is to be expected since it is only seeing the outbound half the network conversation. Ideas on things to try or investigate next? I can provide a paste of any relevant config or log files, just let me know. Thanks, Ed Carrel