From owner-freebsd-arm@freebsd.org Thu Mar 21 04:14:18 2019 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B111F154D1EA for ; Thu, 21 Mar 2019 04:14:18 +0000 (UTC) (envelope-from mw@semihalf.com) Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 091BD6C8B5 for ; Thu, 21 Mar 2019 04:14:16 +0000 (UTC) (envelope-from mw@semihalf.com) Received: by mail-qk1-x741.google.com with SMTP id m65so7967857qkl.11 for ; Wed, 20 Mar 2019 21:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+9XiuRL1J5gTcDX493wvr3iNYbNkIhess8f9YUZMSAo=; b=uf/U3LcboQlFQPoN+b9QybPOUyweXDEwsKqV29CNVYBhbepeZVcDucWn7QbYuBoT4X C5m9dAKMjdbj9WLd3GVnltAwRz4lRl+tVXRCUG/Y10PYK7tDafEx7xGoH/P0pDi/QAHB /C2ZABknbLktg2e/PKhp91XT5fBHWWS6NOSgbcHA579MxJ2M6AvHnhJcOU2FmxghjHuS ++03ljHnw/TnJNFw0Fds8Gb1mS+RjCI1SGSTNHUY93SpI0INJLAUxGSEXbGoP2EnKyg6 uzAsYNe7OT1ix7FhpkyK1BwHBQ9bJwb67pQusPNFxlGcoZ0n/l4SPbn2VL5eLv20dNM7 J4zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+9XiuRL1J5gTcDX493wvr3iNYbNkIhess8f9YUZMSAo=; b=BtE6yhEwFURSLxtVAEGSETgj/F8uEoghvsVIFcOZoecgU2nA4uAuszxq74ytJatGsW IjqZMCG5PotWZzno/O4juT0kq2kLf1G4TbLwldJY0xoJV+sLF5WNQ36qySZ6lP93Urey AkA3GARvJYiJe6yrDWo4jAnQVhHqb/4bltrOLyO+RVozbvz0X32Zhmgkct3LP+Kte62/ +fXowAjtUP9/MElt60VfrqzzhkGM7mQBqcIR/Ua3RkPsckmlCWi0xGfz44tca19uPJwb 1bVEyjkW9U8QNGXdT1G1sjxmqKvFOYVD7AKrepznZI6g/01xIvbxU/qkwRDrrhOaFjBf Mv9A== X-Gm-Message-State: APjAAAU4WoBMuRdFik4RLCtpQgN4tlSoHxuG30ca3XF4Q/t/jlIfgPIA 5XlmRh0B8aR4gmrd9bDRGnO3JQhB7jdTe2oS9k/D2ovJ X-Google-Smtp-Source: APXvYqwlXoIEnb3iSjF9CbPhyu4kkBJDadFF8+lK9V9PCIRnOxgNGK+i1svq4mzC7LXg+25xt9+CWKN8sTjUhGEx+0s= X-Received: by 2002:a37:a4d1:: with SMTP id n200mr1056228qke.62.1553141656340; Wed, 20 Mar 2019 21:14:16 -0700 (PDT) MIME-Version: 1.0 References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> <010001699e6c9844-608d29c1-df68-4504-a611-430adf179bec-000000@email.amazonses.com> In-Reply-To: <010001699e6c9844-608d29c1-df68-4504-a611-430adf179bec-000000@email.amazonses.com> From: Marcin Wojtas Date: Thu, 21 Mar 2019 05:14:05 +0100 Message-ID: Subject: Re: ARM Graviton AWS Processor (AMI Image) To: Colin Percival Cc: Greg V , Martin Karrer , "Matushevsky, Alexander" , =?UTF-8?Q?Micha=C5=82_Krawczyk?= , =?UTF-8?B?UmFmYcWCIEtvemlr?= , "Tzalik, Guy" , freebsd-arm@freebsd.org X-Rspamd-Queue-Id: 091BD6C8B5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=semihalf-com.20150623.gappssmtp.com header.s=20150623 header.b=uf/U3Lcb X-Spamd-Result: default: False [-3.69 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.987,0]; R_DKIM_ALLOW(-0.20)[semihalf-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; DMARC_NA(0.00)[semihalf.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: ASPMX2.GOOGLEMAIL.com]; DKIM_TRACE(0.00)[semihalf-com.20150623.gappssmtp.com:+]; RCPT_COUNT_SEVEN(0.00)[8]; RCVD_IN_DNSWL_NONE(0.00)[1.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.85)[-0.851,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.54)[ip: (2.30), ipnet: 2607:f8b0::/32(-2.81), asn: 15169(-2.12), country: US(-0.07)] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 04:14:19 -0000 Hi Colin, I will handle those patches, thanks. Best regards, Marcin W dniu czw., 21.03.2019 o 05:05 Colin Percival napisa=C5=82(a): > Hi Micha=C5=82, > > Thanks, those two patches fix both problems. Can you get them into > the FreeBSD tree or should I commit them? > > Colin Percival > > On 3/20/19 3:24 AM, Micha=C5=82 Krawczyk wrote: > > Hi, > > > > we have solution for two of the issues with ENA, > > > > The double resource activation can be fixed by this patch: > > https://reviews.freebsd.org/D19655 > > > > And the issue with reloading module/reconfiguring it by using ifconfig > > can be fixed by using this patch: > > https://reviews.freebsd.org/D19641 > > > > Please check if that fixes it for you. > > > > Thanks, > > Micha=C5=82 > > > > > > =C5=9Br., 20 mar 2019 o 04:35 Colin Percival > napisa=C5=82(a): > >> > >> Hi ENA people, > >> > >> Any luck figuring out what was going on with the ENA driver here > >> with interrupt activation? > >> > >> Colin Percival > >> > >> On 3/11/19 1:35 AM, Marcin Wojtas wrote: > >>> +FreeBSD ENA maintainers > >>> > >>> W dniu pon., 11.03.2019 o 00:40 Greg V > >>> napisa=C5=82(a): > >>> > >>>> > >>>> On Tue, Jan 22, 2019 at 9:39 PM, Greg V > >>>> wrote: > >>>>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer > >>>>> wrote: > >>>>>> My question is if there are any plans yet to support the Graviton > >>>>>> ARM instances of AWS? > >>>>>> > >>>>>> We have a heavy load on FreeBSD and would also use the ARM > >>>>>> instances. Are there any other interested parties? > >>>>> > >>>>> I have tried this. It should work very well in theory, e.g. the > >>>>> network card driver (if_ena) compiles with no changes for aarch64, > >>>>> and in fact NetBSD has ported this driver and is up and running on > >>>>> these instances: https://dmesgd.nycbug.org/index.cgi?do=3Dview&id= =3D4623 > >>>>> > >>>>> But my result with FreeBSD was: nothing on the console after > >>>>> loader.efi hands control to the kernel. > >>>>> [=E2=80=A6] > >>>> > >>>> Hello everyone, big update: > >>>> > >>>> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! > >>>> > >>>> https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4813 > >>>> > >>>> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right > now: > >>>> > >>>> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) > >>>> > >>>> ----- > >>>> > >>>> So, what I had to do / what should be done / how others can help get > >>>> this into a finished state: > >>>> > >>>> 1. Serial console: > >>>> - I fixed it: https://reviews.freebsd.org/D19507 > >>>> - (I learned some things about UARTs and their support in FreeBSD= , > >>>> should write a blog post about that) > >>>> > >>>> 2. aarch64 build configuration: > >>>> - if_ena network driver module should be enabled: > >>>> https://reviews.freebsd.org/D18372 > >>>> - NVMe driver should be enabled in the GENERIC kernel config > (device > >>>> nvme, device nvd) > >>>> - BTW, why not also go with hw.nvme.use_nvd=3D"0" by default on > >>>> aarch64, IIRC that was done on powerpc64 > >>>> > >>>> 3. VM image build system: > >>>> - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 > was > >>>> MBR with EFI (???)): https://reviews.freebsd.org/D18371 > >>>> - bsdec2-image-upload --arm64 flag should be supported: included > >>>> above ^^ > >>>> - ec2.conf: amazon-ssm-agent shouldn't be installed when building > >>>> for aarch64 TARGET, since that's written in Go, and Go isn't ported = to > >>>> FreeBSD/aarch64 yet: > >>>> > >>>> > https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50= aad01689bc > >>>> (untested patch, I actually used an interactive shell in between the > >>>> image build commands) > >>>> - qemu-aarch64-static should be used for preinstalling pkgs when > >>>> chrooting into the image: rough version included above ^^ > >>>> > >>>> 4. ENA (Elastic Network Adapter) driver: > >>>> - it works > >>>> - except there's something funky with interrupt activation, and i= t > >>>> hits panic("Attempt to double activation of resource id: %u\n", > res_id) > >>>> (for the management IRQ) on boot, so I applied the obvious silly > >>>> workaround of "don't panic": > >>>> > >>>> > https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba6339= 2262938e62c > >>>> - but still, it doesn't properly reactivate interrupts (and the b= ox > >>>> becomes unreachable over the net) after going down and up again =E2= =80=94 > >>>> guess what does that on boot? dhclient applying the big jumbo MTU = =E2=80=94 > >>>> so I set dhclient.conf to reject MTU changes: > >>>> > >>>> > https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294c= c6d8c870f7f > >>>> > >>>> > >>>> Would be great if someone familiar with interrupts and stuff could > help > >>>> debug the ena driver and make it work without these hacks :) > >> -- > >> Colin Percival > >> Security Officer Emeritus, FreeBSD | The power to serve > >> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly > paranoid > > > > > > -- > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoi= d >