From owner-freebsd-arm@freebsd.org Mon Mar 11 08:35:25 2019 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A46A15375A4 for ; Mon, 11 Mar 2019 08:35:25 +0000 (UTC) (envelope-from mw@semihalf.com) Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1BDEB8A608 for ; Mon, 11 Mar 2019 08:35:23 +0000 (UTC) (envelope-from mw@semihalf.com) Received: by mail-qk1-x72f.google.com with SMTP id z13so2194350qki.2 for ; Mon, 11 Mar 2019 01:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=m+q2GQObguL+4V8sNF9RM4VFdlQStOagmq6ejfw/Wr4=; b=OjDNekRdXDXT/wpHYnZyj8a/hKPR5qch2In9TTHFDKKJtIu6d6xXMPmGyxQlYM81mF WXE2iUn+CsN2lXUdx2vCUy5inxqyaxWSleiXmt1cCb1WiPjpU2dsA09xLO5pAxZB/yzN JxDRibvS+r3fz41VywSF1rK0ujZmxXbuIimiLn/vi7wUJZKdqQbDIprfNxMpYauE6qlh a8c10CA0DDaY6okZ6amaVmak6y0MY01ni1m82r4r/axADNgvavJreqTNMH4OZPCvk9Ac oTwFCTVzP0bVL+8SmdP2qku+MhIkv+OB5kS8mCRWeedg1l9oMVboR8+Z3Hst35JJxjMA QA6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=m+q2GQObguL+4V8sNF9RM4VFdlQStOagmq6ejfw/Wr4=; b=YQWrQA2b3SlNIV58jrD65qNQWq28DnyBH9CDpD4C2kSlKpUOP1gLorN5a0u6OKKyOV ZLbOm+8zlrcggyDmm5oqHiAcJv4CrVHm6jWmtBDflQe2SDhnGrxjmYbgsjTTZP1uli+/ Lkiz0G//6H44e9Usw/7SIPfF8gjBjSK7L3D4Fw1rR0tT10JSgQ8rIefy+R1w+UNX35L0 pTHxEzpAyXCDIb6m0ibXNVS8VCpE2O0C9M2BtSenW1Z6RPjMsnemRlzmSrMD/6d1S0CF yFxGTyA1yUmZs0wVM1dVth3T25UfchGV1MUJyiMT8pm9zWgfkrZGYNM8kUC3BejEuyq0 rH/A== X-Gm-Message-State: APjAAAUJDwbZ988ECKHlrlM2/2LV3WZwsDU/6RWXQUw8bs/BaK4m7TO/ 4oExfLxOUNfpqSQXmbkMWkWhv4HK5RLy07ApHXiadX5n X-Google-Smtp-Source: APXvYqwFJ2utru3PKH+z0Sn7GS0FiYgWuWfiFnzAmZVvet7lHa88iAwbhk9taPaXcE0ftefKCG5HNDTtxYQmHg2b+ow= X-Received: by 2002:a37:b105:: with SMTP id a5mr23047025qkf.298.1552293322323; Mon, 11 Mar 2019 01:35:22 -0700 (PDT) MIME-Version: 1.0 References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> In-Reply-To: <1552255580.21373.0@unrelenting.technology> From: Marcin Wojtas Date: Mon, 11 Mar 2019 09:35:11 +0100 Message-ID: Subject: Re: ARM Graviton AWS Processor (AMI Image) To: Greg V Cc: Martin Karrer , "Matushevsky, Alexander" , =?UTF-8?Q?Micha=C5=82_Krawczyk?= , =?UTF-8?B?UmFmYcWCIEtvemlr?= , freebsd-arm@freebsd.org, freebsd-cloud@freebsd.org, freebsd-hackers@freebsd.org X-Rspamd-Queue-Id: 1BDEB8A608 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=semihalf-com.20150623.gappssmtp.com header.s=20150623 header.b=OjDNekRd X-Spamd-Result: default: False [-6.11 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[semihalf-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; DMARC_NA(0.00)[semihalf.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[semihalf-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[ASPMX2.GOOGLEMAIL.com,ALT2.ASPMX.L.GOOGLE.com,ASPMX.L.GOOGLE.com,ALT1.ASPMX.L.GOOGLE.com,ASPMX3.GOOGLEMAIL.com]; RCPT_COUNT_SEVEN(0.00)[8]; RCVD_IN_DNSWL_NONE(0.00)[f.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.94)[-0.935,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.87)[ip: (-9.45), ipnet: 2607:f8b0::/32(-2.74), asn: 15169(-2.07), country: US(-0.07)] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 08:35:25 -0000 +FreeBSD ENA maintainers W dniu pon., 11.03.2019 o 00:40 Greg V napisa=C5=82(a): > > On Tue, Jan 22, 2019 at 9:39 PM, Greg V > wrote: > > On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer > > wrote: > >> My question is if there are any plans yet to support the Graviton > >> ARM instances of AWS? > >> > >> We have a heavy load on FreeBSD and would also use the ARM > >> instances. Are there any other interested parties? > > > > I have tried this. It should work very well in theory, e.g. the > > network card driver (if_ena) compiles with no changes for aarch64, > > and in fact NetBSD has ported this driver and is up and running on > > these instances: https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D462= 3 > > > > But my result with FreeBSD was: nothing on the console after > > loader.efi hands control to the kernel. > > [=E2=80=A6] > > Hello everyone, big update: > > FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! > > https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4813 > > And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now: > > ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) > > ----- > > So, what I had to do / what should be done / how others can help get > this into a finished state: > > 1. Serial console: > - I fixed it: https://reviews.freebsd.org/D19507 > - (I learned some things about UARTs and their support in FreeBSD, > should write a blog post about that) > > 2. aarch64 build configuration: > - if_ena network driver module should be enabled: > https://reviews.freebsd.org/D18372 > - NVMe driver should be enabled in the GENERIC kernel config (device > nvme, device nvd) > - BTW, why not also go with hw.nvme.use_nvd=3D"0" by default on > aarch64, IIRC that was done on powerpc64 > > 3. VM image build system: > - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was > MBR with EFI (???)): https://reviews.freebsd.org/D18371 > - bsdec2-image-upload --arm64 flag should be supported: included > above ^^ > - ec2.conf: amazon-ssm-agent shouldn't be installed when building > for aarch64 TARGET, since that's written in Go, and Go isn't ported to > FreeBSD/aarch64 yet: > > https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50= aad01689bc > (untested patch, I actually used an interactive shell in between the > image build commands) > - qemu-aarch64-static should be used for preinstalling pkgs when > chrooting into the image: rough version included above ^^ > > 4. ENA (Elastic Network Adapter) driver: > - it works > - except there's something funky with interrupt activation, and it > hits panic("Attempt to double activation of resource id: %u\n", res_id) > (for the management IRQ) on boot, so I applied the obvious silly > workaround of "don't panic": > > https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba6339= 2262938e62c > - but still, it doesn't properly reactivate interrupts (and the box > becomes unreachable over the net) after going down and up again =E2=80=94 > guess what does that on boot? dhclient applying the big jumbo MTU =E2=80= =94 > so I set dhclient.conf to reject MTU changes: > > https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294c= c6d8c870f7f > > > Would be great if someone familiar with interrupts and stuff could help > debug the ena driver and make it work without these hacks :) > > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" >