Date: Mon, 11 Mar 2019 01:06:20 +0300 From: Greg V <greg@unrelenting.technology> To: Martin Karrer <martin@bmalum.com> Cc: freebsd-arm@freebsd.org, freebsd-hackers@freebsd.org, freebsd-cloud@freebsd.org Subject: Re: ARM Graviton AWS Processor (AMI Image) Message-ID: <1552255580.21373.0@unrelenting.technology> In-Reply-To: <1548182399.2864.0@smtp.migadu.com> References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jan 22, 2019 at 9:39 PM, Greg V <greg@unrelenting.technology> wrote: > On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer <martin@bmalum.com> > wrote: >> My question is if there are any plans yet to support the Graviton >> ARM instances of AWS? >> >> We have a heavy load on FreeBSD and would also use the ARM >> instances. Are there any other interested parties? > > I have tried this. It should work very well in theory, e.g. the > network card driver (if_ena) compiles with no changes for aarch64, > and in fact NetBSD has ported this driver and is up and running on > these instances: https://dmesgd.nycbug.org/index.cgi?do=view&id=4623 > > But my result with FreeBSD was: nothing on the console after > loader.efi hands control to the kernel. > […] Hello everyone, big update: FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! https://dmesgd.nycbug.org/index.cgi?do=view&id=4813 And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now: ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) ----- So, what I had to do / what should be done / how others can help get this into a finished state: 1. Serial console: - I fixed it: https://reviews.freebsd.org/D19507 - (I learned some things about UARTs and their support in FreeBSD, should write a blog post about that) 2. aarch64 build configuration: - if_ena network driver module should be enabled: https://reviews.freebsd.org/D18372 - NVMe driver should be enabled in the GENERIC kernel config (device nvme, device nvd) - BTW, why not also go with hw.nvme.use_nvd="0" by default on aarch64, IIRC that was done on powerpc64 3. VM image build system: - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was MBR with EFI (???)): https://reviews.freebsd.org/D18371 - bsdec2-image-upload --arm64 flag should be supported: included above ^^ - ec2.conf: amazon-ssm-agent shouldn't be installed when building for aarch64 TARGET, since that's written in Go, and Go isn't ported to FreeBSD/aarch64 yet: https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aad01689bc (untested patch, I actually used an interactive shell in between the image build commands) - qemu-aarch64-static should be used for preinstalling pkgs when chrooting into the image: rough version included above ^^ 4. ENA (Elastic Network Adapter) driver: - it works - except there's something funky with interrupt activation, and it hits panic("Attempt to double activation of resource id: %u\n", res_id) (for the management IRQ) on boot, so I applied the obvious silly workaround of "don't panic": https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba63392262938e62c - but still, it doesn't properly reactivate interrupts (and the box becomes unreachable over the net) after going down and up again — guess what does that on boot? dhclient applying the big jumbo MTU — so I set dhclient.conf to reject MTU changes: https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6d8c870f7f Would be great if someone familiar with interrupts and stuff could help debug the ena driver and make it work without these hacks :)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1552255580.21373.0>