From owner-freebsd-net@freebsd.org Fri Dec 4 23:33:53 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E00BA40B41 for ; Fri, 4 Dec 2015 23:33:53 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D90011575 for ; Fri, 4 Dec 2015 23:33:52 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3pC9Mw3Z1vztN for ; Sat, 5 Dec 2015 00:33:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1449272025; x=1451864026; bh=lvb xa7QwFZCt1+7z2KZ9i4aEH/AahD+9AWd2ztUjYsc=; b=Ox5IQOnjMZfBmpqb+VT dLrfIE36fez1o4fz61TE84uHUX2vNyxV+EaSSFgGcrifKLQInBc6ZQEyJ3sAeLPh ECT6Z5z+J1zB7GVx2Ob3VxpiCbW35fEtqYznEElGlOLKf0qH1IkgbZHf9Yq2VhM2 uwn7916+S1B7dtxVdeZbvVdM= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id eQQsHFKTjKCQ for ; Sat, 5 Dec 2015 00:33:45 +0100 (CET) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 3pC9Ms0bB2ztM for ; Sat, 5 Dec 2015 00:33:45 +0100 (CET) Received: from nabiralnik.ijs.si (nabiralnik.ijs.si [IPv6:2001:1470:ff80::80:16]) by mildred.ijs.si (Postfix) with ESMTP id 3pC9Mr6Jwfz3r for ; Sat, 5 Dec 2015 00:33:44 +0100 (CET) Received: from neli.ijs.si ([193.2.4.95]) by nabiralnik.ijs.si with HTTP (HTTP/1.1 POST); Sat, 05 Dec 2015 00:33:44 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 05 Dec 2015 00:33:44 +0100 From: Mark Martinec To: freebsd-net@freebsd.org Subject: Re: Outgoing packets being sent via wrong interface Organization: Jozef Stefan Institute In-Reply-To: <20151125092145.e93151af70085c2b3393f149@neosystem.cz> References: <20151120155511.5fb0f3b07228a0c829fa223f@neosystem.org> <20151120163431.3449a473db9de23576d3a4b4@neosystem.org> <20151121212043.GC2307@vega.codepro.be> <20151122130240.165a50286cbaa9288ffc063b@neosystem.cz> <20151125092145.e93151af70085c2b3393f149@neosystem.cz> Message-ID: X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.1.3 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Dec 2015 23:33:53 -0000 On 2015-11-25 09:21, Daniel Bilik wrote: > It happened again, yesterday, and I can now definitely confirm > that it's related to default route. [...] > ... because again it was pushing outgoing packets wrong way, via public > interface, where it's dropped by pf... [...] > I've tried to just delete default route and enter it back to routing > table. > # route delete default ; sleep 1 ; route add default 82.x.y.29 > ... and voila, ping started to communicate with affected host... Seems like I have stumbled across the same problem as Daniel, as all that was said above applies to my case too. Running a fairly recent 10-STABLE, pf was disabled. 10.2-STABLE FreeBSD 10.2-STABLE #3 r291378: Fri Nov 27 12:45:53 CET 2015 ... amd64 In addition to a regular ethernet interface, I also have a gif tunnel. Routing directs a tunnel endpoint to the ethernet interface, and most of the rest goes through tunnel. Maybe worth mentioning that some processes (like ntpd) run in FIB 1 with their own routing able to force them to use a direct route. I was trying to set up an nginx proxy, but couldn't get it to respond to a remote client. After tcpdump sessions it turned out that a TCP SYN from a remote host came in through an ethernet interface, but a SYN ACK went out through gif, even through netstat -rn was still telling the default route is ethernet. Firewall was disabled. Funny thing is that a sshd process was still sending responses to the same remote host through the correct ethernet interface via a default route. Luckily I came across this thread, and sure enough, a: route delete default; route add default x.x.x.x solved the problem right away. The netstat -rn output of before and after the route reset were exactly the same. Unfortunately I can't reproduce the problem now, and I never experienced such problem in previous 10-STABLE revisions, or earlier. Anyway, just wanted to mention that possibly Daniel may not be alone with a default route becoming ineffective. Mark On 2015-11-25 09:21, Daniel Bilik wrote: > On Sun, 22 Nov 2015 13:02:40 +0100 > Daniel Bilik wrote: > >> Well, even though pf may play some role in the problem, I tend to >> suspect >> the routing table as the main trigger. There are several facts to >> support >> this... > > It happened again, yesterday, and I can now definitely confirm that > it's > related to default route. > > In this case, affected address was 192.168.2.33. This host was unable > to > connect to 192.168.2.15 (jail on the router), and router itself was > unable > to even ping the affected host... > > PING 192.168.2.33 (192.168.2.33): 56 data bytes > ping: sendto: Operation not permitted > ping: sendto: Operation not permitted > > ... because again it was pushing outgoing packets wrong way, via public > interface, where it's dropped by pf... > > 00:00:07.091814 rule 53..16777216/0(match): block out on re0: > 82.x.y.50 > 192.168.2.33: ICMP echo request, id 12037, seq 0, length > 64 > 00:00:01.011536 rule 53..16777216/0(match): block out on re0: > 82.x.y.50 > 192.168.2.33: ICMP echo request, id 12037, seq 1, length > 64 > > I've tried to just delete default route and enter it back to routing > table. > In one tmux session ping was running, in another session I've performed > this... > > # route delete default ; sleep 1 ; route add default 82.x.y.29 > > ... and voila, ping started to communicate with affected host... > > ping: sendto: Operation not permitted > ping: sendto: Operation not permitted > 64 bytes from 192.168.2.33: icmp_seq=12 ttl=128 time=0.535 ms > 64 bytes from 192.168.2.33: icmp_seq=13 ttl=128 time=0.264 ms > > Touching nothing else (pf etc.), not rebooting, just "refreshing" the > default route entry, and the problem disappeared.