Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 8 Nov 2015 16:55:01 +0100
From:      Marius Strobl <marius@alchemy.franken.de>
To:        Warner Losh <imp@bsdimp.com>
Cc:        sbruno@freebsd.org, freebsd-arch <freebsd-arch@freebsd.org>, sparc64@freebsd.org
Subject:   Re: Sparc64 doesn't care about you, and you shouldn't care about Sparc64
Message-ID:  <20151108155501.GA1901@alchemy.franken.de>
In-Reply-To: <2AAC0EF3-528B-476F-BA9C-CDC3004465D0@bsdimp.com>
References:  <563A5893.1030607@freebsd.org> <2AAC0EF3-528B-476F-BA9C-CDC3004465D0@bsdimp.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Nov 04, 2015 at 04:19:38PM -0700, Warner Losh wrote:
> 
> > On Nov 4, 2015, at 12:12 PM, Sean Bruno <sbruno@freebsd.org> wrote:
> > 
> > So here's the thing, Sparc64 is *just* barely alive in FreeBSD.
> 
> Has anybody actually booted it off a newish tree?

Yes, just works fine there as of r290419.

root@v215:~ # uname -a
FreeBSD v215.zeist.de 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r290419: Thu Nov  5 23:01:37 CET 2015     marius@v215.zeist.de:/tmp/obj/usr/src/sys/GENERIC  sparc64

> > There is exactly 1 Sparc64 machine as a ref box being hosted at Yahoo
> > for the project.  No new hardware is on the horizon.

Over time, linimon@ and me independently tried to get more sparc64
machines there but failed. As far as I was involved the goal was to
get hardware being offered for donation in the US there but it
turned out that's it's just impossible to coordinate that from over
here. However, I've been told that the primary reason for getting rid
of the collocation at Yahoo is to improve the situation with on-site
maintenance. So getting sparc64 hosts set up it the project cluster
may actually be feasible now. I probably should check with glen@.

Anyway, how many arm, mips, powerpc and powerpc64 reference machines
are there in the project cluster and available for general use by
developers? If there were, what would you do with them? According to
my experience, the very few !x86 machines that actually are in there
aren't really useful without root access.

> > None of the newer
> > Sparc64 processors have been tested to work on FreeBSD and nobody is
> > clamoring to get them working.

Unlike x86, sun4u and sun4v CPUs are only designed to be backwards
compatible as far as the userland is concerned, host-PCI{,e}-bridges
aren't compatible etc. So the kernel side always needs work and it
simply isn't a matter of "testing" newer CPUs and models to work.
Thus, the hardware is needed for developing support, see also below.

I'm not sure at which point I'd speak of people "clamoring" for
support of some hardware; f. e. there also isn't a request for
the graphics of Haswell and later Intel CPUs to be supported on
the mailing lists every other day but you'll certainly agree that
many are waiting for it. Now there likely are fewer people looking
forward for later sun4u and sun4v processors getting supported but
there definitely are people asking for it, just have a look at the
lists.

> > We're moving into a post-gcc base system now, and sparc64 is the obvious
> > "odd arch" here.  There's activity to get MIPS moved to clang and active
> > work to get powerpc moved fully to clang.  Leaving Sparc64 in base,
> > requires someone to either make clang DTRT or keep gcc 4.2.1-ish alive.

I don't see why sparc64 would be "the obvious odd arch" in that
regard. The real problem is switching an architecture for which
clang might have gotten en par with GCC after clang was changed
to require C++11 for bootstrap. Given that clang was only the
default on arm and x86 at that point in time, we are now stuck
without an in-tree upgrade path on all other architectures.
Granted, that might be lesser a problem on mips as these machines
typically don't have enough CPU and RAM that self-hosting would
be interesting in the first place. That still puts sparc64 into
the same boat as powerpc and powerpc64, though.

> There was some work to get clang to do the right thing for sparc64. Last
> I heard, the tree compiles with it. It didn?t boot, but at the time gcc-compiled
> kernels didn?t boot either. I?m not sure how this status has moved through time.
> It would be best to ask Marius Strobl, since he?s the only one committing
> to sparc64 sub-tree lately non-global-sweep cleanups.

Err, I'm sorry to say but pretty much all of these statements are
wrong or at least don't reflect reality properly.
As for clang, initial support for 64-bit SPARC v9 and FreeBSD/sparc64
was added in the late 3.5.x times (I don't remember the exact version
offhand). Yes, it does compile FreeBSD userland but the majority of
binaries do not work or put differently: For some strange reason a
clang compiled sort(1) doesn't crash _and_ apparently does the right
thing but I wasn't able to find any other program that worked correctly
when compiled with clang. By installing clang-compiled system libraries
one even totally hoses ones system, i. e. then also sort(1) instantly
crashes.
With clang 3.6.0 that got even worse; the front-end was changed to
select soft-float by default but the back-end didn't have support for
choosing between hard- and soft-float for 64-bit SPARC v9 so every
compiler invocation just bailed out. As a side note: I couldn't spot
any code that would suggest that hard-float support is implemented,
which should be the actual default, though. I fixed that (the soft-
float defaulting that is, not implementing hard-float support) and
some other astonishing bugs (or rather: hacked some things into
shape), which got some more things working and IIRC even a clang-
compiled kernel got considerably further. However, I didn't get
anywhere near the bottom of bugs and missing features. Generally,
clang on sparc64 gives more of an impression that it never has faced
a compiler test-suite. I haven't given clang 3.7.0 a try thus far but
the release notes and source changes don't suggest much has improved
in that regard since then.
Also, for me a toolchain is just what the name implies: A tool. If
it isn't sufficiently close to 100 % working I just use another one.

Regarding GCC-compiled kernles not working on sparc64 at that point
I'm not aware of such a problem in that time frame or generally of a
sparc64-specific breakage in the last couple of years (there was
one in userland a couple of months ago, though). I mean, sure, the
kernel build gets broken regularly as people don't even compile-test
their changes and sometimes that also results in run-time problems
but that certainly isn't what you are referring to. However, I can't
think of a sparc64 kernel breakage that got beyond that or even was
sparc64-specific.
What you _might_ be thinking of in that regard is the following:
When 64-bit SPARC v9 and FreeBSD/sparc64 support initially were
added, clang couldn't cope with code in the sparc64 pcpu.h. There
was a patch by jmg@ floating around which changed pcpu.h so that
clang could compile a kernel. However, that patch conceptually is
wrong and results in a non-working kernel even with GCC. Meanwhile,
clang has been taught to grok at least the code in pcpu.h (but not
generally to deal with the pattern in question), so at least that
situation has improved and said patch is obsolete.

As for getting forward, the FreeBSD Software License Policy
(https://www.freebsd.org/internal/software-license.html)
specifically allows for existing GPLv2 licensed software in
the FreeBSD source tree to be updated to GPLv3 licensed one.
The initial, longer draft of this policy posted by brooks@ to
developers@ even explicitly mentioned key technologies such
as toolchains of other licenses being allowed when no mature
BSD-licensed alternative exists. So I propose just that:
Let's upgrade binutils and GCC in base to recent versions.
Seriously. That way we 1) don't need to get external toolchain
support into shape, 2) don't need to solve the chicken-and-egg
problem of getting a toolchain onto a machine installed from
a distribution built with an external toolchain and 3) once
clang becomes mature on additional architectures, we have an
upgrade path. Don't get me wrong, I'm only proposing that
for !arm and !x86.
As a side note: A while back I talked to grehan@ and marcel@
regarding the immaturities of clang and - as expected -, a
GPL'ed toolchain just is no problem for either NetApp or
Juniper as the binaries they ship don't include the toolchain
itself. With the possible exception of the current incarnation
of SCO which apparently sells a FreeBSD-based OS likely having
a system compiler, for the same reason I can't think of why a
GPLv3 licensed toolchain would matter for any of the commercial
downstream consumers of FreeBSD. Thus, I really can't understand
all that aggression regarding making FreeBSD 11 clang-only.

> Here?s a breakdown of commits in different parts of sys. The ?Marius? column
> is for commits Marius has made in sparc64 only. The rest are the different
> architectures we currently support. I wrote this with mail.app, so formatting
> may be dicy.
> 
> Year		Marius	sparc64	mips	arm		powerpc	i386		amd64	x86	arm64
> 2015	5		32		164		445		144		168		247		109	168
> 2014	0		39		117		672		98		125		296		108	-
> 2013	14		65		235		455		217		142		235		67	-
> 2012	24		55		272		343		152		188		221		76	-
> 2011	78		131		205		105		172		189		182		56	-
> 2010	75		127		501		103		211		274		268		75	-
> 2009 	58		95		269		193		137		293		258		-	-
> 2008	65		109		65		167		161		304		222		-	-
> 
> sparc64 rate of change has fallen way off since 2011, both in terms of the
> number of commits, as well as the share of commits relative to other
> platforms. While I know that not all commits are treated equally, and that
> different commit styles in different parts of the tree may skew things,

Not only because of that incomplete last sentence it remains unclear
what you exactly would like to express with that numbers. There are
quite a few reasons why they look the way they do, though. First off,
FreeBSD/sparc64 is rock-solid, comparable to x86 in that regard, so it
just doesn't need constant changing. Moreover, for a fair comparison
you'd generally need to filter f. e. board and peripheral support from
sys/arm and sys/mips as sys/sparc64 doesn't have something like the
former and drivers for MACs, USB controllers etc. and even the uart(4)
bus front-ends live in sys/dev for sparc64. Similarly, you'd need to
exclude bhyve- and XEN-related commits from sys/{amd64,i386,x86}.
As a side note: Sure, sparc64 isn't exactly en par with x86 when it
comes to feature parity. However, IMO overall sparc64 competes rather
well with arm and mips in that regard or actually is somewhat better.
For example, neither arm nor mips have been converted to NEW_PCIB nor
do they provide GET_STACK_USAGE while sparc64 has both.

However, one reason certainly also is that some time ago I've decided
that I had spent enough money on hardware for working on FreeBSD that
I'd otherwise would not have bought (which mainly was sparc64 gear
but certainly not limited to that) and rely on donations instead.
That didn't turn out to work too well, though. A general problem in
that regard are shipping costs and taxes when stuff is located outside
the Schengen Area. Also, I've f. e. applied for the M5000 and T2000
machines that where up for donation last August in order to complete/
add support for newer CPUs (support for SPARC64-VI/VII and peripherals
in models based on them is mostly there but disabled as I couldn't
test it). Unfortunately, I haven't heard anything back regarding these
machines. However, that also means: Whoever got that equipment could
justify even better use than me, although I could point to the code I
had written for that gear.

> > I have asked around for help getting the Sparc64 qemu-bsd-user binary
> > working so I could at a minimum build packages, and I have gotten no
> > feedback from folks.  So the only option here is to resurrect sparc64
> > machines somewhere and start up builds on real hardware.

It would be more tempting to look at that if the 64-bit SPARC v9
support of QEMU generally would be up to speed but it isn't for
years. QEMU seemed to have made some progress in that regard in
current versions but it still just isn't there, yet; see the
recent attempts on freebsd-sparc64@ to get some of the bugs
fixed.
That isn't a SPARC-specific lack of interest or problem, though.
At work I do microkernel-based R&D on ARM and for that QEMU just
isn't usable either; it considerably lacks emulation of viable,
real ARMv7-based boards, specifically Cortex-A7-based ones, its
GIC emulation is foobar etc. Not that it would help me anything,
but QEMU doesn't even provide targets of something as popular as
the Raspberry Pi boards. Go figure.
As for sun4u system emulation, Simics worked okay for me. That
was before Wind River bought it, though.
Regarding real sun4u hardware, over time a considerable number
of machines has been donated for ports work AFAICT.

> > Let's just call it what it is, a dead end of the technology tree.
> > I move that we do NOT produce 11.0 versions for Sparc64 and it should be
> > dropped from the tree.
> 
> I concur.  I think sparc64 has had a nice run, but it?s time to recognize
> that the run is nearing its end.
> 

Do whatever you want, I'm tired of mere political arguments and
of fighting against them.

Marius




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151108155501.GA1901>