From nobody Tue Feb 22 19:51:41 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 64AD719DFC8A for ; Tue, 22 Feb 2022 19:52:00 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ua1-x929.google.com (mail-ua1-x929.google.com [IPv6:2607:f8b0:4864:20::929]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K38vM53JTz4c5C for ; Tue, 22 Feb 2022 19:51:59 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-ua1-x929.google.com with SMTP id c36so317972uae.13 for ; Tue, 22 Feb 2022 11:51:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=eEYNHCuQhE6K7A2mdayYGKz/lZugL1vw3OdJVr5axI8=; b=jUl/8rRRksg9xoFZ9FmYJwumq9LwWND3Y62opYKqFePx5REeSNjuIIed/bIBqjWk1s a3h3yj00EX+1c/osoUSTVnuPRzq9azYkQ6gASOy3Gt1m+rqNK+9YjZWAhlWn6EKcchUc q+cfzRJsV7qszX1HqrHDQXuw95mQiU65l27mptT3IJwbqAY+/p/jONESIlB81caQSemB dNUmQU9LV1fZXJE19CIesjSmIaZ3bJnPgQBcuztGJCDXs9vhMbm5eYYqRayItf3Oumi9 e9GeqigJSEyf3o+4PhbEQivFCe4hf8rpzFsYBvczRNkP67bAVdI1d9gLKXuDsSS/OZF/ UBRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eEYNHCuQhE6K7A2mdayYGKz/lZugL1vw3OdJVr5axI8=; b=55o0aNql2UHGw0+iq7xzR1357Z16ErIRTj4r1X9gIw0rYx92yvTusNl/Y4soZ45IML KCKoUYpmi2OzwBKMYV92w2+Pj466ASzOgllo22ucMchB+hi91LQPX7o+Zqi/ef4WR2u5 0Q6CZBOcvQBMYNPP/m8qIoO0lgiMqpSOhunqIzaO1xGI5X7y78qp/VxIWGrivNI0L5h/ kqdPTelZx62/iyOKGHB/SxONHSCd8G4TPLfzy5YThFSkkzyeZOhq3d09p4R4zzl8OhPQ fS2AjMEoeeL2ptUV9GW3suFj3WOoULCs4QGHJHOq9i0UVnEI7qT+3NBfMnuSnQuJAO1s 9wfw== X-Gm-Message-State: AOAM5335KNePjBtZGu9fFA9IAp+/l6EXDBtEzJS/Z8VaCftEJiqHxV4f qmnHefyPmypAB3xQ1tc8Uy5u7GpoiMVccGKMwLQICyGzW/c= X-Google-Smtp-Source: ABdhPJz59GD5p3c8QXAznk+2CozXFFuPgFxIBdhwxctlt0LYrz57atR0nu8Ytv1ijZKS9O785FmePWn7fHFJK2zKbrQ= X-Received: by 2002:ab0:7c5c:0:b0:345:735b:93ae with SMTP id d28-20020ab07c5c000000b00345735b93aemr1696083uaw.77.1645559512780; Tue, 22 Feb 2022 11:51:52 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Tue, 22 Feb 2022 12:51:41 -0700 Message-ID: Subject: Re: Instance drives in AWS comming up with the wrong size To: Pete French Cc: FreeBSD Stable Mailing List Content-Type: multipart/alternative; boundary="0000000000003764b805d8a0aed0" X-Rspamd-Queue-Id: 4K38vM53JTz4c5C X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b="jUl/8rRR"; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::929) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::929:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-stable]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --0000000000003764b805d8a0aed0 Content-Type: text/plain; charset="UTF-8" On Tue, Feb 22, 2022, 12:12 PM Pete French wrote: > So, I have a number of machines in AWS. They are all of type r5a.xlarge > which is supposed to have 140 gig of instance syorage on it. All these > machines started life as clones of the same dirve, ro are runnign the > same OS kernel, and they all have mysql on them. Some of them also have > Apache and some other software, but I thought the config of the base OS > was the same. Certainly /boot/loader.conf and /etc/sysctl.conf are > indentical and its the same kernel and userland running on all of them. > > but on the mysql-only of the machines the instance drives have the worng > size, around a gig, and are not recognised properly. Example, heres iw > what it is supposed to look like (from one of the Apache machines) > > root@sydney01:/usr/home/webadmin # diskinfo -v nda2 > nda2 > 512 # sectorsize > 150000000000 # mediasize in bytes (140G) > 292968750 # mediasize in sectors > 0 # stripesize > 0 # stripeoffset > Amazon EC2 NVMe Instance Storage # Disk descr. > AWSB7ABDF8FE8D0597AF # Disk ident. > nvme2 # Attachment > Yes # TRIM/UNMAP support > 0 # Rotation rate in RPM > > and here is one of the ones which is wrong... > > root@serpentine-sydy:~ # diskinfo -v nda2 > nda2 > 512 # sectorsize > 886571008 # mediasize in bytes (846M) > 1731584 # mediasize in sectors > 131072 # stripesize > 0 # stripeoffset > No # TRIM/UNMAP support > Unknown # Rotation rate in RPM > What does 'nvmecontrol identify nvme2' and 'nvmecontrol identify nda2' say for each? Warner But both machines, in dmesg, have lines which look like this: > > nda2 at nvme2 bus 0 scbus2 target 0 lun 1 > nda2: > nda2: Serial Number AWSB7ABDF8FE8D0597AF > nda2: nvme version 1.0 x0 (max x0) lanes PCIe Gen0 (max Gen0) link > nda2: 143051MB (292968750 512 byte sectors) > > So on both of then the detection says its the right size. > > The oddest this is that this is completely reproducible betwene data > centres - the above machines above are in Sydney, but I get precisely > the same result from the machines in North Virginia. So its something > about the config but what on earth could it be ? > > I am very puzzled - I have a set of database-nly machine sin Frankfurt, > and they behave fine! Poissibly I should just clone those to Australia > and the US, but I would like to find out what the magic difference is > between them. > > -pete. > > > --0000000000003764b805d8a0aed0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Feb 22, 2022, 12:12 PM Pete French <pete@twisted.org.uk> wrote:
So, I have a number of machines in AWS. The= y are all of type r5a.xlarge
which is supposed to have 140 gig of instance syorage on it. All these
machines started life as clones of the same dirve, ro are runnign the
same OS kernel, and they all have mysql on them. Some of them also have Apache and some other software, but I thought the config of the base OS was the same. Certainly /boot/loader.conf and /etc/sysctl.conf are
indentical and its the same kernel and userland running on all of them.

but on the mysql-only of the machines the instance drives have the worng size, around a gig, and are not recognised properly. Example, heres iw
what it is supposed to look like (from one of the Apache machines)

root@sydney01:/usr/home/webadmin # diskinfo -v nda2
nda2
=C2=A0 =C2=A0 =C2=A0 =C2=A0 512=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0# sectorsize
=C2=A0 =C2=A0 =C2=A0 =C2=A0 150000000000=C2=A0 =C2=A0 # mediasize in bytes = (140G)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 292968750=C2=A0 =C2=A0 =C2=A0 =C2=A0# mediasize= in sectors
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0# stripesize
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0# stripeoffset
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Amazon EC2 NVMe Instance Storage=C2=A0 =C2=A0 = =C2=A0 =C2=A0 # Disk descr.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 AWSB7ABDF8FE8D0597AF=C2=A0 =C2=A0 # Disk ident.=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nvme2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#= Attachment
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Yes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0# TRIM/UNMAP support
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0# Rotation rate in RPM

and here is one of the ones which is wrong...

root@serpentine-sydy:~ # diskinfo -v nda2
nda2
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0512=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0# sectorsize
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0886571008=C2=A0 =C2=A0 =C2=A0 =C2=A0# med= iasize in bytes (846M)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01731584=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= # mediasize in sectors
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0131072=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = # stripesize
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0# stripeoffset
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0No=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 # TRIM/UNMAP support
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Unknown=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= # Rotation rate in RPM

What does 'nvmecontrol identify nvme2' and &#= 39;nvmecontrol identify nda2' say for each?

=
Warner

But both machines, in dmesg, have lines which look like this:

nda2 at nvme2 bus 0 scbus2 target 0 lun 1
nda2: <Amazon EC2 NVMe Instance Storage 0 AWSB7ABDF8FE8D0597AF>
nda2: Serial Number AWSB7ABDF8FE8D0597AF
nda2: nvme version 1.0 x0 (max x0) lanes PCIe Gen0 (max Gen0) link
nda2: 143051MB (292968750 512 byte sectors)

So on both of then the detection says its the right size.

The oddest this is that this is completely reproducible betwene data
centres - the above machines above are in Sydney, but I get precisely
the same result from the machines in North Virginia. So its something
about the config but what on earth could it be ?

I am very puzzled - I have a set of database-nly machine sin Frankfurt, and they behave fine! Poissibly I should just clone those to Australia
and the US, but I would like to find out what the magic difference is
between them.

-pete.


--0000000000003764b805d8a0aed0--