From owner-freebsd-jail@freebsd.org Mon Oct 16 14:28:33 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E077E3BDED for ; Mon, 16 Oct 2017 14:28:33 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail.mimar.rs (tazar.mimar.rs [193.53.106.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 114F282A23 for ; Mon, 16 Oct 2017 14:28:32 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from tazar.mimar.rs (localhost [127.0.2.132]) by mail.mimar.rs (Postfix) with ESMTP id 85A4A620C08B; Mon, 16 Oct 2017 16:18:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mimar.rs; h= content-transfer-encoding:content-type:content-type:mime-version :x-mailer:organization:references:in-reply-to:message-id:subject :subject:from:from:date:date:received:received; s=mimar-0901; t= 1508163525; x=1509977926; bh=cnNLWnMVQbVsE0Tp1BkUL4ao50mIQp5O3pF PkgOXBYo=; b=KVvlXNvfMZidYR2LAidwxGx0HrjD8jXBYhA4QsU6TwuYhKcPQya BFvyd1KZWrHLIXOZIz3Ct946j8oVBxzqY1UQt63TBgh2FE9qSoYgCoPj43DT1cTy waMi/175sklEITjbqRt7omzIADV4XUo0odCTHvh+1dyAsrKhyH5UT2QU= X-Virus-Scanned: amavisd-new at mimar.rs Received: from mail.mimar.rs ([127.0.2.132]) by tazar.mimar.rs (amavis.mimar.rs [127.0.2.132]) (amavisd-new, port 10026) with LMTP id SlCtVZlX3tRe; Mon, 16 Oct 2017 16:18:45 +0200 (CEST) Received: from efreet-freebsd.kappastar.com (nat-nat.kappastar.com [193.53.106.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: marko.cupac) by mail.mimar.rs (Postfix) with ESMTPSA id 5DD23620BD58; Mon, 16 Oct 2017 16:18:45 +0200 (CEST) Date: Mon, 16 Oct 2017 16:18:44 +0200 From: Marko =?UTF-8?B?Q3VwYcSH?= To: Andrew Hotlab Cc: "freebsd-jail@freebsd.org" Subject: Re: setfib (ez)jails and wierd routing Message-ID: <20171016161844.7ddb1fe7@efreet-freebsd.kappastar.com> In-Reply-To: References: <20170929103258.2f912308@efreet-freebsd.kappastar.com> Organization: Mimar X-Mailer: Claws Mail 3.15.1 (GTK+ 2.24.31; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2017 14:28:33 -0000 On Sat, 30 Sep 2017 10:38:58 +0000 Andrew Hotlab wrote: > Hi Marko. I'm running an almost identical setup, but I do not have > this issue: ICMP echo reply packets are sent from the right > interface. The only difference is that I didn't defined additional > lo1 and lo2 interfaces, but I guess it shouldn't be the cause. >=20 > I'm running releng/10.3. Which release are you working on? Hi Andrew, sorry for late reply. I'm running 11.1-RELEASE-p1. I am definitely seeing packets with source addresses of my DMZ jails (fib2) exiting through interface on local LAN. Those are mostly icmp echo replies that should be coming from jails but are not due to the fact that jails don't have raw sockets enables. So, echo replies are returned from host (and not jails), whose default gateway is on internal network. Would freebsd-net be more appropriate list for this problem? Thank you in advance, --=20 Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupa=C4=87 https://www.mimar.rs/ From owner-freebsd-jail@freebsd.org Tue Oct 17 15:17:20 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51C4BE3CCED for ; Tue, 17 Oct 2017 15:17:20 +0000 (UTC) (envelope-from andrew.hotlab@hotmail.com) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-oln040092070078.outbound.protection.outlook.com [40.92.70.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C15317072C for ; Tue, 17 Oct 2017 15:17:19 +0000 (UTC) (envelope-from andrew.hotlab@hotmail.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=819DC/gV0AP7aeXEwfB34FhrSMkbdzXfy7mRDrvDb5A=; b=bTO53YLRAD0YHTuS527K42ifTXPVkEw6J51ct4OZ/ThCe/Mrg/rEnVxqdMTYhK/m614ggmzbk6EFn/HGmhePCs6gp/XuHK2gLoDBrv3CgngnS+8OG+EoIC998Tsw0kVbdBs8VkBuSC7u7D9zruky6uPyZfLKzjNIEn4jcZ1VKGDGSYXv5oWeKctgT0TEI7GEm/gDm6HpR+CW32MzXPgJtiMpRzEXkhGbbsACOznosD54Wzm1ooXz0xZV7K8ZjI512oXIhiy545s20UXhtnqZbW4sRKreXHcmopwEP25Pl4ZAnRSir3BGsgouU6y83wUnXHLxvO2MaHRX0X+PZVTkYQ== Received: from VE1EUR03FT051.eop-EUR03.prod.protection.outlook.com (10.152.18.52) by VE1EUR03HT036.eop-EUR03.prod.protection.outlook.com (10.152.19.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.77.10; Tue, 17 Oct 2017 15:17:16 +0000 Received: from AM5PR0201MB2467.eurprd02.prod.outlook.com (10.152.18.60) by VE1EUR03FT051.mail.protection.outlook.com (10.152.19.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.77.10 via Frontend Transport; Tue, 17 Oct 2017 15:17:16 +0000 Received: from AM5PR0201MB2467.eurprd02.prod.outlook.com ([fe80::f51c:8799:a5a:6830]) by AM5PR0201MB2467.eurprd02.prod.outlook.com ([fe80::f51c:8799:a5a:6830%17]) with mapi id 15.20.0077.022; Tue, 17 Oct 2017 15:17:16 +0000 From: Andrew Hotlab To: =?iso-8859-2?Q?Marko_Cupa=E6?= CC: "freebsd-jail@freebsd.org" Subject: Re: setfib (ez)jails and wierd routing Thread-Topic: setfib (ez)jails and wierd routing Thread-Index: AQHTOP2ZzRBrEXVI+0u9V9Fd3zH3XqLNPYkvgBlj1ACAAZ5FQg== Date: Tue, 17 Oct 2017 15:17:16 +0000 Message-ID: References: <20170929103258.2f912308@efreet-freebsd.kappastar.com> , <20171016161844.7ddb1fe7@efreet-freebsd.kappastar.com> In-Reply-To: <20171016161844.7ddb1fe7@efreet-freebsd.kappastar.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:F678B3CAA42174F5F1F3A1DAB80A5FA9CF26B8A7AE6779EB38F11F7947B05834; UpperCasedChecksum:BC01907E88A692457021FE3469F09D3A742CEBA1C2961C30707B8073819227E2; SizeAsReceived:7269; Count:47 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [WYESCzrNybo9srrv5Va+ioVdL35NhDGl] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VE1EUR03HT036; 6:YAfGQcnUDLuON9EOoeemWcFuEtowH+S2eXvjOYma5hxLq0dHD7I16BwG15pMbgP9yI1QH+voN9DO+C8CgA1pr3S/jAxU5RLbuMbG2M+Ud/4FE2H5nuEmot+A/EqWiChGSLivUkW7D5UYGwTo7TcxYVQ+3SnGWp1vTG3MoflaWa89gshp/04vTgNXjdr6a7mnErJfXmlbi7WsMXHvf7893couA98Lns/Ise3P28XTc8LJmOwFYkcEDrxrp3HNJS+AgrDzIm8Fv1Eq28HBpnifAQCbso476D5lGV84ZpemjS7QTnouKXJDSzXJrWtzRH8wpOouZbLL2ZkUfaJvn9CQ2g==; 5:SWJRJXm2tCFAnVezHAwBlYt+x2nZIWeXer3tg2LCXsIb+V08U6TEgKWub6w4cRlrX6QgbwhmVnUnXhCSNCsF5dcveKnck+OV/Buhm0qaLMqjEQOxO2PgElUr6NPVkc5MTMnoyprvTZtouWiCai76dg==; 24:Gxruetc/ge3QpMh4ic4RbtEWcNst2ZxSck+2OxtqjZRNRsTtmRxdiXlNtcQjTursWZVP7AF3QNOOMQiTam1771jCi2leNa7xGiBdgmMEegs=; 7:zmJaR7G84jzH0nnI8auavXFJ59CGTvl5v447WbF0RD2pyP7ENFoErxT9yAfFEy6TjQ7FEBqzqpLMMENx5g2aPd03Py7f/dNCHFxtqH0aWalIX47V+kDRoGTJMGl1EibrdpHZb3sXxCh1VsHX/4Xg320eXCH/9cw8H6eDVV87fejg4bdbvRa2mVMi7Z8C7FnnnOs1e6TQSI9fBDXagcmC7hqMbzg6cI/71EEnxn2SJ/Q= x-incomingheadercount: 47 x-eopattributedmessage: 0 x-ms-office365-filtering-correlation-id: 72be26fb-2478-4215-508e-08d51572271c x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1603101448)(1601125374)(1701031045); SRVR:VE1EUR03HT036; x-ms-traffictypediagnostic: VE1EUR03HT036: authentication-results: outbound.protection.outlook.com; spf=skipped (originating message); dkim=none (message not signed) header.d=none; dmarc=none action=none header.from=hotmail.com; x-exchange-antispam-report-test: UriScan:(150554046322364)(265634631926514)(131110393319338)(130873036417446)(194151415913766)(50823345283023); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(444000031); SRVR:VE1EUR03HT036; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:VE1EUR03HT036; x-forefront-prvs: 04631F8F77 x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:VE1EUR03HT036; H:AM5PR0201MB2467.eurprd02.prod.outlook.com; FPR:; SPF:None; LANG:; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Oct 2017 15:17:16.8477 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1EUR03HT036 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Oct 2017 15:17:20 -0000 ________________________________________ From: Marko Cupa=E6 Sent: Monday, October 16, 2017 4:18 PM To: Andrew Hotlab Cc: freebsd-jail@freebsd.org Subject: Re: setfib (ez)jails and wierd routing > On Sat, 30 Sep 2017 10:38:58 +0000 > Andrew Hotlab wrote: >=20 > > I'm running releng/10.3. Which release are you working on? >=20 > sorry for late reply. I'm running 11.1-RELEASE-p1. I am definitely > seeing packets with source addresses of my DMZ jails (fib2) exiting > through interface on local LAN. Those are mostly icmp echo replies that > should be coming from jails but are not due to the fact that jails > don't have raw sockets enables. So, echo replies are returned from > host (and not jails), whose default gateway is on internal network. >=20 I just setup a similar scenario on a FreeBSD 11.1 host. It seems that all is working fine (172.21.10.0/24 is the DMZ, while 192.168.1.0/24 is the LAN). Please see the following transcript: root@BSD11:~ # uname -msr FreeBSD 11.1-RELEASE amd64 root@BSD11:~ # ifconfig | egrep '^[a-z]|inet ' em0: flags=3D8843 metric 0 mtu 1500 inet 172.21.10.100 netmask 0xffffff00 broadcast 172.21.10.255=20 inet 172.21.10.101 netmask 0xffffffff broadcast 172.21.10.101=20 em1: flags=3D8843 metric 0 mtu 1500 inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255=20 lo0: flags=3D8049 metric 0 mtu 16384 inet 127.0.0.1 netmask 0xff000000 root@BSD11:~ # netstat -rnfinet Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.254 UGS em1 127.0.0.1 link#3 UH lo0 172.21.10.0/24 link#1 U em0 172.21.10.100 link#1 UHS lo0 172.21.10.101 link#1 UHS lo0 172.21.10.101/32 link#1 U em0 192.168.1.0/24 link#2 U em1 192.168.1.100 link#2 UHS lo0 root@BSD11:~ # setfib 1 netstat -rfinet Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 172.21.10.254 UGS em0 localhost link#3 UH lo0 172.21.10.0/24 link#1 U em0 172.21.10.101/32 link#1 U em0 192.168.1.0/24 link#2 U em1 root@BSD11:~ # cat /etc/jail.conf=20 exec.start =3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; jtest01 { host.hostname =3D "jtest01.test.lab"; path =3D /usr/jails/jtest01; ip4.addr =3D "em0|172.21.10.101/32"; persist; allow.raw_sockets; exec.fib =3D "1"; } root@BSD11:~ # jls JID IP Address Hostname Path 8 172.21.10.101 jtest01.test.lab /usr/jails/jtest01 root@BSD11:~ # ssh 172.21.10.101 'sysctl net.my_fibnum' Password for root@jtest01.test.lab: net.my_fibnum: 1 root@BSD11:~ # tcpdump -i em0 -n -p icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:07:19.524839 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 0, length 64 17:07:20.539686 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 1, length 64 17:07:21.551653 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 2, length 64 17:07:22.562764 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315= , seq 3, length 64 ^C 4 packets captured 12 packets received by filter 0 packets dropped by kernel > Would freebsd-net be more appropriate list for this problem? Maybe, but I would double check your jail configuration before ask to that = list. My guess is that your jail might not be associated to the right fib.