From owner-freebsd-jail@freebsd.org Tue Oct 17 15:17:20 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51C4BE3CCED for ; Tue, 17 Oct 2017 15:17:20 +0000 (UTC) (envelope-from andrew.hotlab@hotmail.com) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-oln040092070078.outbound.protection.outlook.com [40.92.70.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C15317072C for ; Tue, 17 Oct 2017 15:17:19 +0000 (UTC) (envelope-from andrew.hotlab@hotmail.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=819DC/gV0AP7aeXEwfB34FhrSMkbdzXfy7mRDrvDb5A=; b=bTO53YLRAD0YHTuS527K42ifTXPVkEw6J51ct4OZ/ThCe/Mrg/rEnVxqdMTYhK/m614ggmzbk6EFn/HGmhePCs6gp/XuHK2gLoDBrv3CgngnS+8OG+EoIC998Tsw0kVbdBs8VkBuSC7u7D9zruky6uPyZfLKzjNIEn4jcZ1VKGDGSYXv5oWeKctgT0TEI7GEm/gDm6HpR+CW32MzXPgJtiMpRzEXkhGbbsACOznosD54Wzm1ooXz0xZV7K8ZjI512oXIhiy545s20UXhtnqZbW4sRKreXHcmopwEP25Pl4ZAnRSir3BGsgouU6y83wUnXHLxvO2MaHRX0X+PZVTkYQ== Received: from VE1EUR03FT051.eop-EUR03.prod.protection.outlook.com (10.152.18.52) by VE1EUR03HT036.eop-EUR03.prod.protection.outlook.com (10.152.19.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.77.10; Tue, 17 Oct 2017 15:17:16 +0000 Received: from AM5PR0201MB2467.eurprd02.prod.outlook.com (10.152.18.60) by VE1EUR03FT051.mail.protection.outlook.com (10.152.19.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.77.10 via Frontend Transport; Tue, 17 Oct 2017 15:17:16 +0000 Received: from AM5PR0201MB2467.eurprd02.prod.outlook.com ([fe80::f51c:8799:a5a:6830]) by AM5PR0201MB2467.eurprd02.prod.outlook.com ([fe80::f51c:8799:a5a:6830%17]) with mapi id 15.20.0077.022; Tue, 17 Oct 2017 15:17:16 +0000 From: Andrew Hotlab To: =?iso-8859-2?Q?Marko_Cupa=E6?= CC: "freebsd-jail@freebsd.org" Subject: Re: setfib (ez)jails and wierd routing Thread-Topic: setfib (ez)jails and wierd routing Thread-Index: AQHTOP2ZzRBrEXVI+0u9V9Fd3zH3XqLNPYkvgBlj1ACAAZ5FQg== Date: Tue, 17 Oct 2017 15:17:16 +0000 Message-ID: References: <20170929103258.2f912308@efreet-freebsd.kappastar.com> , <20171016161844.7ddb1fe7@efreet-freebsd.kappastar.com> In-Reply-To: <20171016161844.7ddb1fe7@efreet-freebsd.kappastar.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:F678B3CAA42174F5F1F3A1DAB80A5FA9CF26B8A7AE6779EB38F11F7947B05834; UpperCasedChecksum:BC01907E88A692457021FE3469F09D3A742CEBA1C2961C30707B8073819227E2; SizeAsReceived:7269; Count:47 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [WYESCzrNybo9srrv5Va+ioVdL35NhDGl] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VE1EUR03HT036; 6:YAfGQcnUDLuON9EOoeemWcFuEtowH+S2eXvjOYma5hxLq0dHD7I16BwG15pMbgP9yI1QH+voN9DO+C8CgA1pr3S/jAxU5RLbuMbG2M+Ud/4FE2H5nuEmot+A/EqWiChGSLivUkW7D5UYGwTo7TcxYVQ+3SnGWp1vTG3MoflaWa89gshp/04vTgNXjdr6a7mnErJfXmlbi7WsMXHvf7893couA98Lns/Ise3P28XTc8LJmOwFYkcEDrxrp3HNJS+AgrDzIm8Fv1Eq28HBpnifAQCbso476D5lGV84ZpemjS7QTnouKXJDSzXJrWtzRH8wpOouZbLL2ZkUfaJvn9CQ2g==; 5:SWJRJXm2tCFAnVezHAwBlYt+x2nZIWeXer3tg2LCXsIb+V08U6TEgKWub6w4cRlrX6QgbwhmVnUnXhCSNCsF5dcveKnck+OV/Buhm0qaLMqjEQOxO2PgElUr6NPVkc5MTMnoyprvTZtouWiCai76dg==; 24:Gxruetc/ge3QpMh4ic4RbtEWcNst2ZxSck+2OxtqjZRNRsTtmRxdiXlNtcQjTursWZVP7AF3QNOOMQiTam1771jCi2leNa7xGiBdgmMEegs=; 7:zmJaR7G84jzH0nnI8auavXFJ59CGTvl5v447WbF0RD2pyP7ENFoErxT9yAfFEy6TjQ7FEBqzqpLMMENx5g2aPd03Py7f/dNCHFxtqH0aWalIX47V+kDRoGTJMGl1EibrdpHZb3sXxCh1VsHX/4Xg320eXCH/9cw8H6eDVV87fejg4bdbvRa2mVMi7Z8C7FnnnOs1e6TQSI9fBDXagcmC7hqMbzg6cI/71EEnxn2SJ/Q= x-incomingheadercount: 47 x-eopattributedmessage: 0 x-ms-office365-filtering-correlation-id: 72be26fb-2478-4215-508e-08d51572271c x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1603101448)(1601125374)(1701031045); SRVR:VE1EUR03HT036; x-ms-traffictypediagnostic: VE1EUR03HT036: authentication-results: outbound.protection.outlook.com; spf=skipped (originating message); dkim=none (message not signed) header.d=none; dmarc=none action=none header.from=hotmail.com; x-exchange-antispam-report-test: UriScan:(150554046322364)(265634631926514)(131110393319338)(130873036417446)(194151415913766)(50823345283023); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(444000031); SRVR:VE1EUR03HT036; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:VE1EUR03HT036; x-forefront-prvs: 04631F8F77 x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:VE1EUR03HT036; H:AM5PR0201MB2467.eurprd02.prod.outlook.com; FPR:; SPF:None; LANG:; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Oct 2017 15:17:16.8477 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1EUR03HT036 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Oct 2017 15:17:20 -0000 ________________________________________ From: Marko Cupa=E6 Sent: Monday, October 16, 2017 4:18 PM To: Andrew Hotlab Cc: freebsd-jail@freebsd.org Subject: Re: setfib (ez)jails and wierd routing > On Sat, 30 Sep 2017 10:38:58 +0000 > Andrew Hotlab wrote: >=20 > > I'm running releng/10.3. Which release are you working on? >=20 > sorry for late reply. I'm running 11.1-RELEASE-p1. I am definitely > seeing packets with source addresses of my DMZ jails (fib2) exiting > through interface on local LAN. Those are mostly icmp echo replies that > should be coming from jails but are not due to the fact that jails > don't have raw sockets enables. So, echo replies are returned from > host (and not jails), whose default gateway is on internal network. >=20 I just setup a similar scenario on a FreeBSD 11.1 host. It seems that all is working fine (172.21.10.0/24 is the DMZ, while 192.168.1.0/24 is the LAN). Please see the following transcript: root@BSD11:~ # uname -msr FreeBSD 11.1-RELEASE amd64 root@BSD11:~ # ifconfig | egrep '^[a-z]|inet ' em0: flags=3D8843 metric 0 mtu 1500 inet 172.21.10.100 netmask 0xffffff00 broadcast 172.21.10.255=20 inet 172.21.10.101 netmask 0xffffffff broadcast 172.21.10.101=20 em1: flags=3D8843 metric 0 mtu 1500 inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255=20 lo0: flags=3D8049 metric 0 mtu 16384 inet 127.0.0.1 netmask 0xff000000 root@BSD11:~ # netstat -rnfinet Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.254 UGS em1 127.0.0.1 link#3 UH lo0 172.21.10.0/24 link#1 U em0 172.21.10.100 link#1 UHS lo0 172.21.10.101 link#1 UHS lo0 172.21.10.101/32 link#1 U em0 192.168.1.0/24 link#2 U em1 192.168.1.100 link#2 UHS lo0 root@BSD11:~ # setfib 1 netstat -rfinet Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 172.21.10.254 UGS em0 localhost link#3 UH lo0 172.21.10.0/24 link#1 U em0 172.21.10.101/32 link#1 U em0 192.168.1.0/24 link#2 U em1 root@BSD11:~ # cat /etc/jail.conf=20 exec.start =3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; jtest01 { host.hostname =3D "jtest01.test.lab"; path =3D /usr/jails/jtest01; ip4.addr =3D "em0|172.21.10.101/32"; persist; allow.raw_sockets; exec.fib =3D "1"; } root@BSD11:~ # jls JID IP Address Hostname Path 8 172.21.10.101 jtest01.test.lab /usr/jails/jtest01 root@BSD11:~ # ssh 172.21.10.101 'sysctl net.my_fibnum' Password for root@jtest01.test.lab: net.my_fibnum: 1 root@BSD11:~ # tcpdump -i em0 -n -p icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:07:19.524839 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 0, length 64 17:07:20.539686 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 1, length 64 17:07:21.551653 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 2, length 64 17:07:22.562764 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 3, length 64 ^C 4 packets captured 12 packets received by filter 0 packets dropped by kernel > Would freebsd-net be more appropriate list for this problem? Maybe, but I would double check your jail configuration before ask to that = list. My guess is that your jail might not be associated to the right fib.