From owner-freebsd-arm@freebsd.org Fri Nov 6 12:10:32 2015 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0821A275A2 for ; Fri, 6 Nov 2015 12:10:32 +0000 (UTC) (envelope-from zbb@semihalf.com) Received: from mail-lb0-x233.google.com (mail-lb0-x233.google.com [IPv6:2a00:1450:4010:c04::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91DAC131E for ; Fri, 6 Nov 2015 12:10:32 +0000 (UTC) (envelope-from zbb@semihalf.com) Received: by lbblt2 with SMTP id lt2so37273405lbb.3 for ; Fri, 06 Nov 2015 04:10:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf_com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=DHFmR+22dbUapBzRmTc8Mc3pVlfz0yL9eBu7lJCWTFc=; b=mMfRJ0MwUs2li7m+0fmWAXXwK2NRtzef8KcrM7W9KQ3h9Zo0/Qv1Dl48zeYqgRBbXD v8eOcLD6jBhhiViku9q/w+gOeHDzcW4GLXQgzqPkIE5M7LFYOl++tOsjmwKCqL35vFB/ GhWgOx/yZpsw5VZf5UtqjFu9t6sb11hsmP9FnUF0ivkQFf5vPrKqHpSl7B1cSFmZvMDR FxLYJT330vAA89OOPC28G/gMpRx4pZ5kAztg7PEkCPIze6huCCwziRvdRGUQAA1qQWJ/ S3n+kMopjGJXgz8wfXFdn2EpnBCnBYLfrlwVXTpa+TgXJjhdcsmtKecW8iEa2QNrYXqw rxpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=DHFmR+22dbUapBzRmTc8Mc3pVlfz0yL9eBu7lJCWTFc=; b=Qk4cJQYRE5OeruFzA0cbTJ41KgpkuP1JERoJSCafA4w+xDarUDdU67sllqKW81V/Rh DnYceE81VB36+sfAOIrCjlJesygxSs8KXHIoH1iC7ZphnPFB1IojpQIXWSAXhwIaWb6x qA5dJ+Mf2zy5nkyxafDPhQA0b581J9CA/2UT/JqeZQGuLqvbJPvJA1UAQc3BQUdvGtOE R5XzagxLXPZ9yjm/3jxtOwjbh1loPsTz7j21jEt6kdg5sRg4t/cRXppOVKpcX7CAsLol 3nj9QYOP76VhfzK1jyju1o+gLmr+gzdRHbpu6b+T+W3ZEhRRfMmFXTYlsX2ORWm2rhED ELbg== X-Gm-Message-State: ALoCoQnnYl6aWl7kxb4/MtbQGcSY13lkujZGxmeYv6br0ladwfnPH165kItF0PI5/7G77Fs7hlwP X-Received: by 10.112.202.233 with SMTP id kl9mr6618131lbc.87.1446811830029; Fri, 06 Nov 2015 04:10:30 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.91.20 with HTTP; Fri, 6 Nov 2015 04:10:10 -0800 (PST) In-Reply-To: <20151106113641.02e0f826@bender> References: <20151106094714.5e8632c6@bender.Home> <20151106113641.02e0f826@bender> From: Zbigniew Bodek Date: Fri, 6 Nov 2015 13:10:10 +0100 Message-ID: Subject: Re: HEADS UP: Cavium ThunderX support in the tree To: Andrew Turner Cc: "freebsd-arm@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2015 12:10:33 -0000 2015-11-06 12:36 GMT+01:00 Andrew Turner : > On Fri, 6 Nov 2015 12:23:11 +0100 > Zbigniew Bodek wrote: > >> 2015-11-06 10:47 GMT+01:00 Andrew Turner : >> > On Thu, 5 Nov 2015 17:14:03 +0100 >> > Zbigniew Bodek wrote: >> > >> >> Hello all, >> >> >> >> Semihalf is happy to officially announce that starting from SVN >> >> revision r289550 FreeBSD is ready to run on Cavium ThunderX >> >> system-on-chip! >> >> ThunderX is the first ARM64 (ARMv8) hardware platform to be >> >> supported by FreeBSD and the only one that introduces 96-CPU cores >> >> SMP. The integrated code includes support for: >> >> >> > ... >> >> - Network Interface Controller (VNIC) 1/10/40G >> > >> > We do? Then why don't I see it in ifconfig on the ThunderX in the >> > cluster? >> > >> > root@cavium:~ # ifconfig -a >> > em0: flags=8843 metric 0 mtu >> > 1500 >> > options=4019b >> > ether 00:15:17:0b:6b:08 >> > inet 192.168.5.167 netmask 0xffffff00 broadcast >> > 192.168.5.255 media: Ethernet autoselect (1000baseT ) >> > status: active >> > nd6 options=29 >> > lo0: flags=8049 metric 0 mtu 16384 >> > options=600003 >> > inet6 ::1 prefixlen 128 >> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >> > inet 127.0.0.1 netmask 0xff000000 >> > groups: lo >> > nd6 options=21 >> > >> > Andrew >> >> Strange. Works for me ;P >> >> First of all, networking cards on ThunderX are PCI Virtual Functions >> so you need to bring them up using iovctl. >> If you watched the youtube demo you should have seen that at first >> there are no interfaces in ifconfig and then I do iovctl and they >> appear. > > Ok, so it's similar to wlan where we need some extra configuration > before we can use it. What is the option I need to enable in rc.conf to > enable this? Yes. You need to invoke iovctl with proper parameters and configuration file. https://www.freebsd.org/cgi/man.cgi?query=iovctl.conf&sektion=5&apropos=0&manpath=FreeBSD+11-current > >> Secondly, EFI that runs on the board needs to pass the correct DTB to >> the kernel (it has to have PCIB, BGX, MDIO, PHYs nodes and they need >> to be in the correct order [MDIO needs to attach before BGX]). > If this is the case the code is broken. It should just work with > the vendor dtb and GENERIC. Thanks for quick evaluation but please see comment below. > > We already have to handle this in other parts of the code by having the > dependent drivers on an earlier pass than later drivers. MDIO is a platform device whereas BGX, NICPF (physical function) and NICVF (virtual function) are devices on PCI. Therefore if the PCIB is placed prior to MDIO in DTB all devices on this PCI bridge will be attached before MDIO is attached. As far as I know there is no straightforward way to create a logical relation between device on PCI and platform devices enumerated using DTB. That is why we asked Cavium to change the order in their DTBs. Of course if you know how to workaround that I would be happy to apply this change to "unbreak" the existing code. > > I also don't see any documentation. I can see the bgx driver, bit there > doesn't seem to be a manual page. > root@cavium:~ # man bgx > No manual entry for bgx > > Andrew Correct. There is no manual entry for: BGX, Cavium MDIO, NICPF and NICVF. Best regards zbb