From owner-freebsd-hackers@freebsd.org  Tue May 28 12:52:17 2019
Return-Path: <owner-freebsd-hackers@freebsd.org>
Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A28015C122C
 for <freebsd-hackers@mailman.ysv.freebsd.org>;
 Tue, 28 May 2019 12:52:17 +0000 (UTC)
 (envelope-from agapon@gmail.com)
Received: from mail-lf1-f68.google.com (mail-lf1-f68.google.com
 [209.85.167.68])
 (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (2048 bits) client-digest SHA256)
 (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 46BA877B34;
 Tue, 28 May 2019 12:52:16 +0000 (UTC)
 (envelope-from agapon@gmail.com)
Received: by mail-lf1-f68.google.com with SMTP id m14so1288313lfp.12;
 Tue, 28 May 2019 05:52:16 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt
 :message-id:date:user-agent:mime-version:in-reply-to
 :content-language:content-transfer-encoding;
 bh=Vq8xG+BoH/WJLQc6Vg/2yKmwrYga6rfuQ5p0GC0Ko2g=;
 b=PjK3UGnvUXnfOtnX3/3+kAlUmvIAySQdsjN1U641kN/5ucEW1piAXiPd7Gb5FIV7kk
 zjQwmOxfRUI/X7E2OHwUyJhxEeOIRCihtZ3IlvIvSezcjQRhg1j8+3QB6L90LSG5j9rd
 djaDktozdlqdv7uswRzkhEGBIJOQ6wbCPo7ZuGchT+I4M8ATJSc/SCfS6FVE9dznoOSH
 U2+DdAULgWD7U68a2uOaQz+G2D6utKUz89gMqpGl4BDoWkjpoY39KpJavSd0F+l6oxZO
 4bZYwnsh+IE1uQ7vRB7EkMql9gVfjrnHSa9b9UWvBDsPje1SnyEnyDzt9V7CrfckImIb
 Bm9w==
X-Gm-Message-State: APjAAAU08gbGYekGb9nI0jhBfIvOR4ljfXyAARHj1iHpMRVV1JD8JJm1
 T9qpqGwGx0Hr+U/QgZtU+1YTrp5z
X-Google-Smtp-Source: APXvYqyh0okJ6g5DZEFidpZUBYHYxDvGvUOqLY3Qjm+XovpisFbvI8+qsmsUFtGzfp+GUsaIhO50hA==
X-Received: by 2002:a19:5044:: with SMTP id z4mr8700446lfj.80.1559047933862;
 Tue, 28 May 2019 05:52:13 -0700 (PDT)
Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96])
 by smtp.googlemail.com with ESMTPSA id w11sm2882599lfe.32.2019.05.28.05.52.11
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Tue, 28 May 2019 05:52:12 -0700 (PDT)
Subject: Re: Amazon AMIs
To: Warner Losh <imp@bsdimp.com>, Matthew Seaman <matthew@freebsd.org>
Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>,
 Colin Percival <cperciva@freebsd.org>, Alex Dupre <ale@freebsd.org>,
 Jonathan Anderson <jonathan@freebsd.org>
References: <f002c020-1e63-c12f-456e-e20f8546a701@FreeBSD.org>
 <8a139c9c-b98b-4a54-1d7c-0ea1e3dc7a72@freebsd.org>
 <53a0bd68-a6ba-e8ad-4af2-abeb22e92c03@FreeBSD.org>
 <E396A0C7-F939-48A1-9B57-D643806B303B@FreeBSD.org>
 <f51b836f-6a40-9be3-425a-e1afd301507b@FreeBSD.org>
 <CANCZdfp0ZWe11jXw_xnPi8G2QhyEeBTOcN_ks=sqJCqDjDYcQg@mail.gmail.com>
From: Andriy Gapon <avg@FreeBSD.org>
Openpgp: preference=signencrypt
Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata=
 xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7
 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l
 N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z
 AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i
 gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ
 /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4
 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8
 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS
 ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx
 rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw
 b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC
 WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV
 l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL
 rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO
 LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj
 GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/
 QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT
 eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/
 psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o
 JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R
 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H
 lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW
 be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+
 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY
 EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO
 jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye
 syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m
 WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI
 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+
 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN
 MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L
 HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM
 BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i
 ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs
 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR
 XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk
 LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4
 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym
 dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6
 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0
 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx
 NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp
Message-ID: <74ed9ad5-58d9-db02-2e20-b053ba5eeb87@FreeBSD.org>
Date: Tue, 28 May 2019 15:52:11 +0300
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <CANCZdfp0ZWe11jXw_xnPi8G2QhyEeBTOcN_ks=sqJCqDjDYcQg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Rspamd-Queue-Id: 46BA877B34
X-Spamd-Bar: ----
Authentication-Results: mx1.freebsd.org;
 spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.68
 as permitted sender) smtp.mailfrom=agapon@gmail.com
X-Spamd-Result: default: False [-4.09 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[];
 RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[];
 R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17];
 RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3];
 MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com];
 NEURAL_HAM_SHORT(-0.93)[-0.931,0];
 FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com];
 R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com];
 ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US];
 MID_RHS_MATCH_FROM(0.00)[];
 FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com];
 ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0];
 MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[];
 NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain];
 RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[FreeBSD.org];
 TO_MATCH_ENVRCPT_SOME(0.00)[];
 RCVD_IN_DNSWL_NONE(0.00)[68.167.85.209.list.dnswl.org : 127.0.5.0];
 IP_SCORE(-1.15)[ipnet: 209.85.128.0/17(-3.39), asn: 15169(-2.28), country:
 US(-0.06)]; 
 RWL_MAILSPIKE_POSSIBLE(0.00)[68.167.85.209.rep.mailspike.net : 127.0.0.17]
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 28 May 2019 12:52:17 -0000

On 21/02/2019 18:17, Warner Losh wrote:
> On Thu, Feb 21, 2019 at 1:02 AM Matthew Seaman <matthew@freebsd.org> wrote:
> 
>> On 20/02/2019 22:32, Jonathan Anderson wrote:
>>> On 20 Feb 2019, at 18:50, Alex Dupre wrote:
>>>
>>>> Colin Percival wrote:
>>>>> Last time I looked at this, we weren't handling hotplug/hotunplug of
>>>>> "NVMe"
>>>>> disks properly on the m5/c5/etc. instances.  I opted to recommend the
>>>>> instance
>>>>> which completely works rather than the one with slightly better
>>>>> performance...
>>>>
>>>> It does happen only on a few instances, but I get some freezes on new t3
>>>> machines: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235856
>>>>
>>>> They are indeed cheaper and more performant, but not 100% reliable in
>>>> every workload.
>>>
>>> https://www.xkcd.com/937 ?
>>>
>>
>> Yes, indeed.  That's a very good reason not to recommend the newest and
>> shiniest.  I haven't seen any problems so far, but then again it's only
>> been a day or so and we haven't got into the full testing regime quite
>> yet.  I'll let you know if we do run into problems.
>>
>> Is there work on hot-plug NVMe going on?  ISTR jmg@ mentioning hotplug
>> PCI at the dev summit at Stockholm EuroBSDCon, but not much since then.
>>
> 
> It's hot-unplug that doesn't work quite right. Hotplug works, I believe, if
> you have PCI_HP in your kernel, I believe.
> 
> What's needed is about a solid week of cleanup and testing in this area,
> however.

I am not sure about the context of the latest question... whether it was about
NVMe hot-plug on real modern hardware or whether it was still about AWS.
In the latter case, as Colin pointed out earlier[*], PCI_HP would not help at
all, because the emulated bridge is not a PCIe bridge.
It seems that they use a mechanism based on an older specification, PCI Standard
Hot-Plug Controller and Subsystem Specification.
I believe that FreeBSD never implemented it.

[*] http://www.daemonology.net/blog/2017-11-17-FreeBSD-EC2-C5-instances.html

-- 
Andriy Gapon