From owner-freebsd-cloud@freebsd.org Mon Jan 25 21:16:40 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADEFDA46B73 for ; Mon, 25 Jan 2016 21:16:40 +0000 (UTC) (envelope-from jlott@averesystems.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 59D75667 for ; Mon, 25 Jan 2016 21:16:40 +0000 (UTC) (envelope-from jlott@averesystems.com) Received: by mail-wm0-x235.google.com with SMTP id l65so80717868wmf.1 for ; Mon, 25 Jan 2016 13:16:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=averesystems.com; s=google; h=mime-version:date:message-id:subject:from:to:content-type; bh=G1YFhlEmTMy1/fc+kjcQB1bfRIEmRvRzYB1nxzSQQ48=; b=sDExKK+fYjjMKqVDm+ZsYI9A/3n9tFMKjD928699Rd+kojn0cdglaezpRF7DZqRaLN RqCgJn7vpJNpvYj6+GsHWkmHIvi16Eb3nx1ILcrsFaSVrVZ/OFzU5fKwgKi4jfDYjhXF x2TkRP+6TIaQYMGSPXN96Wu12P4Pgaqp0SiAU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=G1YFhlEmTMy1/fc+kjcQB1bfRIEmRvRzYB1nxzSQQ48=; b=lX2ygKg6xzwky4dsyMThdFvvxUHKfSE2yTuPp553ZkaRUwcrpDeSM5vhdcxvdMgGpd x9RZPR2wrKbWG66GrkcuUC6A08RlIMQuza2ZYUc3c8sc9DrA7b3Dl19mUey1q0eh5shw KHEBD8d8cNKFT+//D5LRdBC9Te5VsP9ZZen5h51I/fmezQqwgPVBBRjQCEPZ8qTYyxtz 9lfBbob6AdCrtBNZslE9Omrq9BT6yxo4tbGPPaaq5+ayMy7bq8nbueutqOHxkzzanEOE Dgxe95XSoDHv/f7eDdU7odnxQ/HlR/e+Kx9BHTecaAq3OQFLUBhMV/JODe5FD2KFJI9E OCAA== X-Gm-Message-State: AG10YOTHKZRSDvMDurvlS5DSMtk+3pyqvLgnsA+6+HYJFOAEZjpVVmf7aw+CdJdsmZDB9Mi9wNgk2EaTfVyEXY2d MIME-Version: 1.0 X-Received: by 10.28.134.147 with SMTP id i141mr21493707wmd.87.1453756598782; Mon, 25 Jan 2016 13:16:38 -0800 (PST) Received: by 10.27.176.95 with HTTP; Mon, 25 Jan 2016 13:16:38 -0800 (PST) Date: Mon, 25 Jan 2016 16:16:38 -0500 Message-ID: Subject: EC2 Advanced Networking in HEAD From: Jeremiah Lott To: freebsd-cloud@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 21:16:40 -0000 I'm having very inconsistent results using the latest 11.0-CURRENT snapshot AMI "FreeBSD 11.0-CURRENT-amd64-2016-01-22 (ami-0b95716b)". A pretty high percentage of the time (I'd say at least 50%), networking does not work when I create an instance. I am using various r3 instance types in the us-west-2b AZ. Sometimes when I create an instance, everything works fine and I get a DHCP lease for my ixv0 device. Other times, ixv0 is probed, but I never get a DHCP offer (which was the broken behavior from before). Changing the instance type or stopping/stopping the instance can switch an instance's NIC from "working" to "non-working" and vice-versa. However, I'm not convinced it has to do with instance type per-se (changing instance type changes a lot of things about the underlying hardware). Anyone else successfully using this latest AMI? Since advanced networking is on by default now, I'd expect others who are using the AMI to see problems as well. Jeremiah Lott Avere Systems From owner-freebsd-cloud@freebsd.org Mon Jan 25 21:30:06 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F1C1A46F79 for ; Mon, 25 Jan 2016 21:30:06 +0000 (UTC) (envelope-from bounces+73574-160b-freebsd-cloud=freebsd.org@sendgrid.net) Received: from o3.shared.sendgrid.net (o3.shared.sendgrid.net [208.117.48.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEDB7C1F for ; Mon, 25 Jan 2016 21:30:05 +0000 (UTC) (envelope-from bounces+73574-160b-freebsd-cloud=freebsd.org@sendgrid.net) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h=subject:to:references:from:mime-version:in-reply-to:content-type:content-transfer-encoding; s=smtpapi; bh=AnuTpE5y7EWRx7oUqMUycd/Nb2Q=; b=vv7roo+YO4SNeJhdxh KXDcxeLYsufouh712PyEX8S8IBX2CzwgKxvBYx8yrTlmqY5bYgij1j+SyZgvb1dw KtKVrtfPh5m8QgbfGJ63xo58xL7/R/hBmlNCz/XcMqZxa5+uQHxNXMi4y06AGQ7a 8KHffXLP0Lwb/XB3Agacboq64= Received: by filter0912p1mdw1.sendgrid.net with SMTP id filter0912p1mdw1.19039.56A693DB31 2016-01-25 21:30:03.431624651 +0000 UTC Received: from mail.tarsnap.com (ec2-54-86-246-204.compute-1.amazonaws.com [54.86.246.204]) by ismtpd0002p1iad1.sendgrid.net (SG) with ESMTP id pfipg7gqSk-U3j9ejsSL8g for ; Mon, 25 Jan 2016 21:30:02.861 +0000 (UTC) Received: (qmail 94553 invoked from network); 25 Jan 2016 21:27:23 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by ec2-107-20-205-189.compute-1.amazonaws.com with ESMTP; 25 Jan 2016 21:27:23 -0000 Received: (qmail 39580 invoked from network); 25 Jan 2016 21:30:01 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by clamshell.daemonology.net with SMTP; 25 Jan 2016 21:30:01 -0000 Subject: Re: EC2 Advanced Networking in HEAD To: Jeremiah Lott , freebsd-cloud@freebsd.org References: From: Colin Percival X-Enigmail-Draft-Status: N1110 Message-ID: <56A693D9.3010304@freebsd.org> Date: Mon, 25 Jan 2016 13:30:01 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SG-EID: XhyBwObMhraAR+zdwMupjd7LSrZ36RouVZZW3oDOpB13GJAkF4EyYoJmrFIPZkqfI1kr8qeMo/SixT jSY6cJpVzuTNUZ5UfbjrxNCiVLgf20ezeD229kHWgPV8B7mfR9ViKLX1Uqcva22AINXDgPPmMymw95 EGcrbshykhe6J35nmVhZcHcWB6yQX0cWFR0/YITxIY8vj7oIHq//vLq/t7pCSfk9njljsCnKN1hoDQ U= X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 21:30:06 -0000 On 01/25/16 13:16, Jeremiah Lott wrote: > I'm having very inconsistent results using the latest 11.0-CURRENT snapshot > AMI "FreeBSD 11.0-CURRENT-amd64-2016-01-22 (ami-0b95716b)". A pretty high > percentage of the time (I'd say at least 50%), networking does not work > when I create an instance. Just to rule out the obvious: Sometimes I accidentally launch instances into VPC subnets which default to not assigning public IP addresses. > I am using various r3 instance types in the > us-west-2b AZ. Sometimes when I create an instance, everything works fine > and I get a DHCP lease for my ixv0 device. Other times, ixv0 is probed, but > I never get a DHCP offer (which was the broken behavior from before). > Changing the instance type or stopping/stopping the instance can switch an > instance's NIC from "working" to "non-working" and vice-versa. However, I'm > not convinced it has to do with instance type per-se (changing instance > type changes a lot of things about the underlying hardware). Anyone else > successfully using this latest AMI? Since advanced networking is on by > default now, I'd expect others who are using the AMI to see problems as > well. It worked in my tests, but they were far from exhaustive. (This is why I turned it on in HEAD -- so that everybody else can find the bugs!) -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From owner-freebsd-cloud@freebsd.org Mon Jan 25 21:51:49 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6A5FA458FB for ; Mon, 25 Jan 2016 21:51:49 +0000 (UTC) (envelope-from jlott@averesystems.com) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 732D48FE for ; Mon, 25 Jan 2016 21:51:49 +0000 (UTC) (envelope-from jlott@averesystems.com) Received: by mail-wm0-x233.google.com with SMTP id 123so83307106wmz.0 for ; Mon, 25 Jan 2016 13:51:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=averesystems.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=0FiRQCghxiyBpGZVatT/dYKzoUcqPpEjlBQgt2S0G2g=; b=OeU7x8E3wZNoJf1H8ALbhMh/hd3RJ4LRtkf5JUpXz38DGTOuaZ9dEXz+dTwWqB+dcQ 6497pkK3QxSsrQpGWem2TQXUplLOLp8h43ChHuruXoBbZtsFv3XNd8LpeZYyjrJCU5wD u0eFuEIpWeNNIPKxnVgKaNn/m77zYid7VIpXs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=0FiRQCghxiyBpGZVatT/dYKzoUcqPpEjlBQgt2S0G2g=; b=Nk5+7FjzZvSPm4LXFOuzDgsiEgs5ddkaOTriyCqaH0YWrx+CF/PtZ2LxBtcSjTkTE+ CmLOBNzNZoWfL6WM3bwFPXGHr5/ASgOudolHTHmAM4hP9MghdtBsUesYtJJI2eR7P3T0 iIWKD+fs2OYdAT0DDF1uCq0mrEzY3fPZmhpH8YRLCpGJtIu8pKi4TRlz+3zGcUeRZyjy PqPyj50LUTB13KlTX/gEQGCmd87OvwphEwsIhZPQ10PF9blTx0s7iafKnM5rrNZW1n7r b1GyVh8s7ycm+VYkvaRhWb8cFeFxw7PbL46ta3gnhJprE6p4O3F8pgBEbHt7xbSMqCMQ s40g== X-Gm-Message-State: AG10YORru67O09U+asj+VnI3NYlGxO/3f0zSKQI6Q6m67W+HVwKODqEM47vO0BGii/L5wWHJWzlo9cffIdUdTlgy MIME-Version: 1.0 X-Received: by 10.194.192.198 with SMTP id hi6mr19782604wjc.141.1453758706593; Mon, 25 Jan 2016 13:51:46 -0800 (PST) Received: by 10.27.176.95 with HTTP; Mon, 25 Jan 2016 13:51:46 -0800 (PST) In-Reply-To: <56A693D9.3010304@freebsd.org> References: <56A693D9.3010304@freebsd.org> Date: Mon, 25 Jan 2016 16:51:46 -0500 Message-ID: Subject: Re: EC2 Advanced Networking in HEAD From: Jeremiah Lott To: Colin Percival Cc: freebsd-cloud@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 21:51:50 -0000 On Mon, Jan 25, 2016 at 4:30 PM, Colin Percival wrote: > On 01/25/16 13:16, Jeremiah Lott wrote: > > I'm having very inconsistent results using the latest 11.0-CURRENT > snapshot > > AMI "FreeBSD 11.0-CURRENT-amd64-2016-01-22 (ami-0b95716b)". A pretty high > > percentage of the time (I'd say at least 50%), networking does not work > > when I create an instance. > > Just to rule out the obvious: Sometimes I accidentally launch instances > into > VPC subnets which default to not assigning public IP addresses. > We use a VPN. I definitely think the instance itself is configured correctly, because I can have a working instance, and I just stop/start it (not a reboot, have to actually do "stop"), then it will sometimes stop working. Sometimes I have an instance that is not working, and a start/stop "fixes" it. You can also see in the "System Log" (eventually) that the DHCP is failing to get an offer for its private IP when it is "broken". The behavior appears to have some correlation to instance type, with r3.large and r3.xlarge working much more often than any of the larger r3 instance types. However, I've done this long enough that I know that this apparent pattern in instance types could easily just be coincidence. Short snippet from a "broken" instance: Starting dhclient. ixv0: link state changed to UP DHCPREQUEST on ixv0 to 255.255.255.255 port 67 DHCPREQUEST on ixv0 to 255.255.255.255 port 67 DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 8 DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 11 DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 14 DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 17 DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 11 No DHCPOFFERS received. When it works correctly, you will get an DHCPACK on the first request. Jeremiah