From owner-freebsd-net@freebsd.org Thu Mar 3 20:00:30 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44783A93C74 for ; Thu, 3 Mar 2016 20:00:30 +0000 (UTC) (envelope-from pakhom706@gmail.com) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C4C7038A for ; Thu, 3 Mar 2016 20:00:29 +0000 (UTC) (envelope-from pakhom706@gmail.com) Received: by mail-lb0-x231.google.com with SMTP id cf7so20591796lbb.1 for ; Thu, 03 Mar 2016 12:00:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:subject:to:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=/vn6xBKeMfU0G8xz9a/UtrlTB939xcoCt5Qvjyno2u8=; b=q1g0NJYPEMfdgX+qRMPwfxoBMQO4WuJZkJ6baykOr7uP7o8NAtfhBanquThzsGYYht 5tQ/vDOwtk2liAPZwQQyrekABmx2T+0JAxuiU65rVQ1C+AtsxSJndTlmE6wYoSOnycaX llrWRYjMkr81lQpS1I0IvUSG1paSFUrZyfPa2j8IufpNb/OUxBch974NfkF7UU8r5wll 8PRxmb6Dmsps3r0SCQm7A09YDVh5RnJLyB0p5DBOo/Wn7jCZN4T5st+pidTQCmzIFjs3 hQBhWy1imO9pgl/0dK+h3/o7W6gUwj1nXmwi2NPk2+TA1doA5bKJHxKlcUNpZdZeKY79 /OBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:to:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=/vn6xBKeMfU0G8xz9a/UtrlTB939xcoCt5Qvjyno2u8=; b=DWdEqMhn5t/B/dh5K2tC516oNUrB8+5PcceCNlGIflHcFdgxG3DAC6+d8koU6Izbsh UILMLcZPxRm5MWtu20ai77cWpiT13A0AF8KytTUMp0M99Cv6KnVKSHE1fgYz8EIjTjgd deJ4ApAEEVNQvD/K0RjjTfk4h9YqY3gBr5VFi0qsZkIeUSPJXzV+Bp8T7fcqKPUGlPcm 0M9hkHXUyJZsJGMrJ9AVMqu0DCum68kSLt2eFNO4AGYRU2DQZXFTNBiYcgOShKj1qdkQ WTl/sBhwksh/R8/KD5bK8oq1MOVD8oe++v6Fz04I/UhgQ2HlITahAWG9qVGa04ChSUMs xeXw== X-Gm-Message-State: AD7BkJJtcB/S9St0JC6ZhUCCiPXKBaete+5Hdf7eOiXyw1Og2sP+Ebuw3dQiE72/3VRq5Q== X-Received: by 10.25.160.140 with SMTP id j134mr1840950lfe.1.1457035227819; Thu, 03 Mar 2016 12:00:27 -0800 (PST) Received: from [172.27.27.11] (mail.pakhom.spb.ru. [5.19.176.65]) by smtp.googlemail.com with ESMTPSA id ke9sm8868lbc.28.2016.03.03.12.00.26 for (version=TLSv1/SSLv3 cipher=OTHER); Thu, 03 Mar 2016 12:00:26 -0800 (PST) From: Pakhom Golynga Subject: routing issue To: freebsd-net@FreeBSD.org Message-ID: <56D897D9.2040502@gmail.com> Date: Thu, 3 Mar 2016 23:00:25 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 20:00:30 -0000 Hello all! Please help me to investigate this issue. I have problem on 10.2-RELEASE-p12 with multiple network interfaces and PF (rules, NAT) # ifconfig <--cut--> em0: flags=8843 metric 0 mtu 1500 options=4209b ether 00:25:90:64:14:50 inet 172.27.27.254 netmask 0xffffff00 broadcast 172.27.27.255 inet 172.27.27.240 netmask 0xffffffff broadcast 172.27.27.240 inet 172.27.27.252 netmask 0xffffffff broadcast 172.27.27.252 inet 172.27.27.251 netmask 0xffffffff broadcast 172.27.27.251 inet 172.27.27.250 netmask 0xffffffff broadcast 172.27.27.250 inet 172.27.27.249 netmask 0xffffffff broadcast 172.27.27.249 inet 172.27.27.248 netmask 0xffffffff broadcast 172.27.27.248 inet 172.27.27.247 netmask 0xffffffff broadcast 172.27.27.247 inet 172.27.27.246 netmask 0xffffffff broadcast 172.27.27.246 inet 172.27.27.245 netmask 0xffffffff broadcast 172.27.27.245 inet 172.27.27.244 netmask 0xffffffff broadcast 172.27.27.244 inet 172.27.27.243 netmask 0xffffffff broadcast 172.27.27.243 inet 172.27.27.242 netmask 0xffffffff broadcast 172.27.27.242 inet 172.27.27.241 netmask 0xffffffff broadcast 172.27.27.241 inet 172.27.27.239 netmask 0xffffffff broadcast 172.27.27.239 inet 172.27.27.238 netmask 0xffffffff broadcast 172.27.27.238 inet 172.27.27.237 netmask 0xffffffff broadcast 172.27.27.237 inet 172.27.27.236 netmask 0xffffffff broadcast 172.27.27.236 inet 172.27.27.235 netmask 0xffffffff broadcast 172.27.27.235 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8943 metric 0 mtu 1500 options=4209b ether 00:25:90:64:14:51 inet 10.24.44.50 netmask 0xfffffc00 broadcast 10.24.47.255 media: Ethernet autoselect (100baseTX ) status: active <--cut--> em0 - local network. Aliases on em0 for Jail's em1 - connected to ISP This configuration successfully operate a long time but recently (for no apparent reason) suddenly became manifest bug with routing. Regardless from uptime (sometimes 12 hours, and maybe a couple of days) after a certain time the traffic running between jails begins to route via em1 (instead lo0 (net.link.ether.inet.useloopback=1)): # tcpdump -n -e -ttt -i em1 host 172.27.27.252 or host 172.27.27.247 or host 172.27.27.248 00:00:00.054989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161568673 ecr 0], length 0 00:00:00.318036 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.60176: Flags [.], ack 772660872, win 1275, options [nop,nop,TS val 1737287913 ecr 161568991], length 0 00:00:00.036971 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.33080: Flags [.], ack 1, win 1275, options [nop,nop,TS val 1701623130 ecr 161568238], length 0 00:00:00.050001 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 66: 172.27.27.247.80 > 172.27.27.248.40200: Flags [.], ack 1, win 1275, options [nop,nop,TS val 1477098721 ecr 161568288], length 0 00:00:02.794992 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161571873 ecr 0], length 0 00:00:03.312026 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161575185 ecr 0], length 0 00:00:02.887982 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.24868 > 172.27.27.252.25: Flags [S], seq 2075254199, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161578073 ecr 0], length 0 00:00:00.111989 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 74: 172.27.27.247.29581 > 172.27.27.252.25: Flags [S], seq 1217253021, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 161578185 ecr 0], length 0 00:00:03.471007 00:25:90:64:14:51 > 68:bd:ab:5f:ea:d8, ethertype IPv4 (0x0800), length 108: 172.27.27.247.80 > 172.27.27.248.26056: Flags [FP.], seq 1448:1490, ack 1, win 1275, options [nop,nop,TS val 2142270801 ecr 161575507], length 42 In the same time: # route -vn get -host RTA_DST: inet 172.27.27.247; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags: locks: inits: sockaddrs: 172.27.27.247 link#0 route to: 172.27.27.247 destination: 172.27.27.247 fib: 0 interface: lo0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 16384 1 0 locks: inits: sockaddrs: 172.27.27.247 link#4 lo0 127.0.0.1 RTA_DST: inet 172.27.27.250; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags: locks: inits: sockaddrs: 172.27.27.250 link#0 route to: 172.27.27.250 destination: 172.27.27.250 fib: 0 interface: lo0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 16384 1 0 locks: inits: sockaddrs: 172.27.27.250 link#4 lo0 127.0.0.1 RTA_DST: inet 172.27.27.252; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags: locks: inits: sockaddrs: 172.27.27.252 link#0 route to: 172.27.27.252 destination: 172.27.27.252 fib: 0 interface: lo0 flags: recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 16384 1 0 locks: inits: sockaddrs: 172.27.27.252 link#4 lo0 127.0.0.1 # netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 10.24.44.1 UGS em1 10.24.44.0/22 link#5 U em1 10.24.44.50 link#5 UHS lo0 127.0.0.1 link#7 UH lo0 172.27.27.0/24 link#4 U em0 172.27.27.235 link#4 UHS lo0 172.27.27.235/32 link#4 U em0 172.27.27.236 link#4 UHS lo0 172.27.27.236/32 link#4 U em0 172.27.27.237 link#4 UHS lo0 172.27.27.237/32 link#4 U em0 172.27.27.238 link#4 UHS lo0 172.27.27.238/32 link#4 U em0 172.27.27.239 link#4 UHS lo0 172.27.27.239/32 link#4 U em0 172.27.27.240 link#4 UHS lo0 172.27.27.240/32 link#4 U em0 172.27.27.241 link#4 UHS lo0 172.27.27.241/32 link#4 U em0 172.27.27.242 link#4 UHS lo0 172.27.27.242/32 link#4 U em0 172.27.27.243 link#4 UHS lo0 172.27.27.243/32 link#4 U em0 172.27.27.244 link#4 UHS lo0 172.27.27.244/32 link#4 U em0 172.27.27.245 link#4 UHS lo0 172.27.27.245/32 link#4 U em0 172.27.27.246 link#4 UHS lo0 172.27.27.246/32 link#4 U em0 172.27.27.247 link#4 UHS lo0 172.27.27.247/32 link#4 U em0 172.27.27.248 link#4 UHS lo0 172.27.27.248/32 link#4 U em0 172.27.27.249 link#4 UHS lo0 172.27.27.249/32 link#4 U em0 172.27.27.250 link#4 UHS lo0 172.27.27.250/32 link#4 U em0 172.27.27.251 link#4 UHS lo0 172.27.27.251/32 link#4 U em0 172.27.27.252 link#4 UHS lo0 172.27.27.252/32 link#4 U em0 172.27.27.254 link#4 UHS lo0 172.27.28.0/24 link#8 U bridge0 172.27.28.254 link#8 UHS lo0 172.27.30.0/24 172.27.30.2 UGS tun0 172.27.30.1 link#12 UHS lo0 172.27.30.2 link#12 UH tun0 After the reboot everything works well for some time. Thanks!