From owner-freebsd-net@freebsd.org Fri Jan 5 19:54:04 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9571EBD116 for ; Fri, 5 Jan 2018 19:54:04 +0000 (UTC) (envelope-from reshadpatuck1@gmail.com) Received: from mail-qk0-x232.google.com (mail-qk0-x232.google.com [IPv6:2607:f8b0:400d:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CDE17B233 for ; Fri, 5 Jan 2018 19:54:02 +0000 (UTC) (envelope-from reshadpatuck1@gmail.com) Received: by mail-qk0-x232.google.com with SMTP id j137so7314123qke.10 for ; Fri, 05 Jan 2018 11:54:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=H1dYT9mKFEyNb34T8HW1Jy7J945Cw1Q+hHEJDz5Y2E0=; b=Yf6s5Da1G6LU9PAYaQiAm8QXoaKF7QdDGN4EhQd0dYOA2+TCRAN1bhWIaKngJ7c7Hg X8PAbId41U6kv2YynF6zsB4fZiFvihfwo+Wn8PPoVCRTkk9dytzWXEF1oMBCnxFyBlvr 5td+SEg/u2IWWCuP+in8eAfH5haEEyFDXAZxS37qM+78BpM9trwN9g9dvYeoZc0G4sVy LLBosBKhvQt140gLSPFoq9f2r73xq5O5TMhVBlAMO9L9q7wrHjHOv6Cv/IvGtvJkxW5J igCgzN1HzHjCwbU1juRfn4TFI5H0fFG5jLRM8mEEKxenVke6N0ioSXLn7WPsqqFin38g yBkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=H1dYT9mKFEyNb34T8HW1Jy7J945Cw1Q+hHEJDz5Y2E0=; b=JXj/hbs1rkarSxYhJ7qggSdqR0VvJUfwMm+fBR+uR7nT7y12IazMrUhi6oeHyqH+6x sTQ7RXmn0hhgZ3tvb0UYLR+pFRJS2ouS3U9Jjw4xh/9MjUcvFmpbGCYGEgkWSGKALG52 hV0egOPAWxsby5lgWbCPaYS66rwqT7Ypd25jfh/waZvPB/DeYUy8ARjcq3azx5huIuRa GQQJV4sfJvqTd4I6Pt0u2Wjm+pSMF4BGmGEz2y9XMNOXTq4JDqMNHtzEl48x4B1LnU+4 ucTxe1j7QVWIMT3pnv80I6dSKJVVNNl1JHZZwdpm7p/qdYTDFgZWBAMfgBH4kVsVzXMc 1vlw== X-Gm-Message-State: AKwxytcfcAmcYiTvf3tZeqJE/cUi4+zM3Fdv/a+FaCGdYc7AZRV/bd2J NJJHEXErL5+5ZztDBsSh9Etbtehx9favGRi38MyfAEeJ X-Google-Smtp-Source: ACJfBov/xz9d16Oj8P4Zt7mjhOhjYaA0LDaKX6ODVPsV7Nu1PviIUil1JRReHTeSeB1MdS1+dIlc9deK9gTMt4caxdg= X-Received: by 10.55.15.2 with SMTP id z2mr5891466qkg.91.1515182040906; Fri, 05 Jan 2018 11:54:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.200.54.92 with HTTP; Fri, 5 Jan 2018 11:54:00 -0800 (PST) From: Reshad Patuck Date: Sat, 6 Jan 2018 01:24:00 +0530 Message-ID: Subject: [vnet][epair] epair interface stops working after some time To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jan 2018 19:54:04 -0000 Hey, I am having a strange issue with one of my servers. I have a couple of VNET jails FreeBSD 12 r321619 set up using if_bridge and epairs. Each VNET jail (and the host too) has a pf firewall limiting inbound traffic. Everything works as intended for some time (1-5 days), services inside the jail work and the jail can connect out to the rest of the network. After some time of working fine I suddenly find that the jails stop receiving traffic and can not send traffic out. Essentially the traffic on one end of the epair does not come out the other. I have linked to a diagram with my network setup for the jails. Essentially the same setup is running on another identical server at another location and has been running for atleast two weeks without any issues. The symptoms are as follows: - I can connect to the server via ssh (on igb0 at IP 192.168.1.50). - All connections from outside the jails work fine from (192.168.1.50 to external IPs) - I can not connect to any services running inside the jails from either outside or inside the server - I can not connect out from the jails (jexec in to the jails and then attempt to connect out) - When I attempt to connect out from one of the jails: - I see arp traffic (via tcpdump) on the epair inside the jail (epair0b) - I cant see the same arp traffic (via tcpdump) on the epair outside the jail (epair0a) - 'arp -a' insde the jails shows incomplete arps for any external IP I try to reach. - When I tcpdump on igb0, bridge0 or epair0a I see broadcast/multicast/general network traffic. - When I tcpdump on epair0b I see no traffic at all. I have done the following on both servers to test what happens: - Created a new epair interface epair3a and epair3b - upped both interfaces - given epair3a IP address 10.20.30.40/24 (I don't have this subnet anywhere in my network) - attempted to ping 10.20.30.50 - checked for any packets on epair3b On the server where epairs are working, I can see APR packets for 10.20.30.50, but on the server where epairs are not working I cant see any packets on epair3b. I can however see the arp packets on epair3a on both servers. This is the third time I have found this on the same server and the other server is still going strong. After rebooting the server this problem seems to go away temporarily, but seems to manifest itself again after some time. Any commands, ideas, thoughts on how to troubleshoot what is wrong here will be much appreciated. Please let me know if there is anything I can do the debug this issue or if you need any other information. Thanks and best regards, Reshad Link to network diagram: https://i.imgur.com/1XdRjt0.jpg