From owner-freebsd-hackers@freebsd.org Sun Mar 10 22:06:36 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E96C1543AA6 for ; Sun, 10 Mar 2019 22:06:36 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: from out.migadu.com (out.migadu.com [91.121.223.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.migadu.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B24A69BA3 for ; Sun, 10 Mar 2019 22:06:33 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: (Migadu outbound); Sun, 10 Mar 2019 22:06:24 +0000 Received: from [192.168.1.141] ([62.122.208.146]) by out.migadu.com (Haraka/2.8.16) with ESMTPSA id 226BFB91-4C7C-49EE-8C50-5BD0BC0F0D4F.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 verify=FAIL); Sun, 10 Mar 2019 22:06:24 +0000 Date: Mon, 11 Mar 2019 01:06:20 +0300 From: Greg V Subject: Re: ARM Graviton AWS Processor (AMI Image) To: Martin Karrer Cc: freebsd-arm@freebsd.org, freebsd-hackers@freebsd.org, freebsd-cloud@freebsd.org Message-Id: <1552255580.21373.0@unrelenting.technology> In-Reply-To: <1548182399.2864.0@smtp.migadu.com> References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> X-Mailer: geary/master~gfcf07ad4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; bh=yFeukkt+RUN+oEqF1Ne+PZL5JrtX8FnxvXapLRyw2ig=; c=relaxed/simple; d=unrelenting.technology; h=from:subject:date:to; s=default; b=bO0c5gqCPwVq1Ip/7e9PLoCHqgYYn+WFSTytDddDRgn8sznjfCFWXKFXpa9DhrllgGrw463zaQfRMiT/1OlO7SQzvzRa2NHj6dm1G77lvRPWfWrgsf8hSF5S6rGRDWLq9jlwNuVmlxOttP38e/z9gyPeZOt4iqwvejLF6Rlf8FU= X-Rspamd-Queue-Id: 3B24A69BA3 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=unrelenting.technology header.s=default header.b=bO0c5gqC; dmarc=pass (policy=none) header.from=unrelenting.technology; spf=pass (mx1.freebsd.org: domain of greg@unrelenting.technology designates 91.121.223.63 as permitted sender) smtp.mailfrom=greg@unrelenting.technology X-Spamd-Result: default: False [-6.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[unrelenting.technology:s=default]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:91.121.223.63]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[unrelenting.technology:+]; DMARC_POLICY_ALLOW(-0.50)[unrelenting.technology,none]; MX_GOOD(-0.01)[aspmx1.migadu.com,aspmx2.migadu.com]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.63)[ip: (-9.89), ipnet: 91.121.0.0/16(-4.08), asn: 16276(0.84), country: FR(-0.01)]; ASN(0.00)[asn:16276, ipnet:91.121.0.0/16, country:FR]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Mailman-Approved-At: Mon, 11 Mar 2019 10:31:51 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2019 22:06:36 -0000 On Tue, Jan 22, 2019 at 9:39 PM, Greg V =20 wrote: > On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer =20 > wrote: >> My question is if there are any plans yet to support the Graviton=20 >> ARM =7Finstances of AWS? >>=20 >> We have a heavy load on FreeBSD and would also use the ARM=20 >> instances. =7FAre there any other interested parties? >=20 > I have tried this. It should work very well in theory, e.g. the=20 > network card driver (if_ena) compiles with no changes for aarch64,=20 > and in fact NetBSD has ported this driver and is up and running on=20 > these instances: https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4623 >=20 > But my result with FreeBSD was: nothing on the console after=20 > loader.efi hands control to the kernel. > [=E2=80=A6] Hello everyone, big update: FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4813 And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now: ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) ----- So, what I had to do / what should be done / how others can help get=20 this into a finished state: 1. Serial console: - I fixed it: https://reviews.freebsd.org/D19507 - (I learned some things about UARTs and their support in FreeBSD,=20 should write a blog post about that) 2. aarch64 build configuration: - if_ena network driver module should be enabled:=20 https://reviews.freebsd.org/D18372 - NVMe driver should be enabled in the GENERIC kernel config (device=20 nvme, device nvd) - BTW, why not also go with hw.nvme.use_nvd=3D"0" by default on=20 aarch64, IIRC that was done on powerpc64 3. VM image build system: - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was=20 MBR with EFI (???)): https://reviews.freebsd.org/D18371 - bsdec2-image-upload --arm64 flag should be supported: included=20 above ^^ - ec2.conf: amazon-ssm-agent shouldn't be installed when building=20 for aarch64 TARGET, since that's written in Go, and Go isn't ported to=20 FreeBSD/aarch64 yet:=20 https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aa= d01689bc=20 (untested patch, I actually used an interactive shell in between the=20 image build commands) - qemu-aarch64-static should be used for preinstalling pkgs when=20 chrooting into the image: rough version included above ^^ 4. ENA (Elastic Network Adapter) driver: - it works - except there's something funky with interrupt activation, and it=20 hits panic("Attempt to double activation of resource id: %u\n", res_id)=20 (for the management IRQ) on boot, so I applied the obvious silly=20 workaround of "don't panic":=20 https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba633922= 62938e62c - but still, it doesn't properly reactivate interrupts (and the box=20 becomes unreachable over the net) after going down and up again =E2=80=94=20 guess what does that on boot? dhclient applying the big jumbo MTU =E2=80=94= =20 so I set dhclient.conf to reject MTU changes:=20 https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6= d8c870f7f Would be great if someone familiar with interrupts and stuff could help=20 debug the ena driver and make it work without these hacks :) =