Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 25 Feb 2017 05:49:40 -0800
From:      Mark Millard <markmi@dsl-only.net>
To:        Mateusz Guzik <mjguzik@gmail.com>
Cc:        svn-src-head@freebsd.org, FreeBSD Current <freebsd-current@freebsd.org>, Justin Hibbits <chmeeedalf@gmail.com>, mjg@freebsd.org, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org>
Subject:   Re: svn commit: r313268 - head/sys/kern [through -r313271 for atomic_fcmpset use and later: fails on PowerMac G5 "Quad Core"; -r313266 works]
Message-ID:  <B38C6E4C-DC41-44A6-A99B-CD6FA5CB105C@dsl-only.net>
In-Reply-To: <477BA631-AB85-4E77-8BA3-CD2AFAD5E405@dsl-only.net>
References:  <2FD12B8F-2255-470A-98D4-2DCE9C7495F5@dsl-only.net> <20170220191044.GA8526@dft-labs.eu> <5D5235E1-6F84-4329-8ED5-35FCDB0A6A71@dsl-only.net> <20170225002300.GC19697@dft-labs.eu> <12339EDD-5663-40E0-8553-821EF9B6CDEB@dsl-only.net> <EB9DBDFA-BAE9-4BFF-8E8B-BF7698362A11@dsl-only.net> <477BA631-AB85-4E77-8BA3-CD2AFAD5E405@dsl-only.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2017-Feb-25, at 1:05 AM, Mark Millard <markmi@dsl-only.net> wrote:

> On 2017-Feb-24, at 11:46 PM, Mark Millard <markmi at dsl-only.net> =
wrote:
>=20
>> On 2017-Feb-24, at 8:25 PM, Mark Millard <markmi at dsl-only.net> =
wrote:
>>=20
>>> On 2017-Feb-24, at 4:23 PM, Mateusz Guzik <mjguzik at gmail.com> =
wrote:
>>>>=20
>>>> On Tue, Feb 21, 2017 at 01:37:25AM -0800, Mark Millard wrote:
>>>>> [Back to the powerpc64 context.]
>>>>>=20
>>>>> On 2017-Feb-20, at 11:10 AM, Mateusz Guzik <mjguzik at gmail.com> =
wrote:
>>>>>=20
>>>>>> On Sat, Feb 18, 2017 at 04:18:05AM -0800, Mark Millard wrote:
>>>>>>> [Note: I experiment with clang based powerpc64 builds,
>>>>>>> reporting problems that I find. Justin is familiar
>>>>>>> with this, as is Nathan.]
>>>>>>>=20
>>>>>>> I tried to update the PowerMac G5 (a so-called "Quad Core")
>>>>>>> that I have access to from head -r312761 to -r313864 and
>>>>>>> ended up with random panics and hang ups in fairly short
>>>>>>> order after booting.
>>>>>>>=20
>>>>>>> Some approximate bisecting for the kernel lead to:
>>>>>>> (sometimes getting part way into a buildkernel attempt
>>>>>>> for a different version before a failure happens)
>>>>>>>=20
>>>>>>> -r313266: works (just before use of atomic_fcmpset)
>>>>>>> vs.
>>>>>>> -r313271: fails (last of the "use atomic_fcmpset" check-ins)
>>>>>>>=20
>>>>>>> (I did not try -r313268 through -r313270 as the use was
>>>>>>> gradually added.)
>>>>>>>=20
>>>>>>> So I'm currently running a -r313864 world with a -r313266
>>>>>>> kernel.
>>>>>>>=20
>>>>>>> No kernel that I tried that was from before -r313266 had the
>>>>>>> problems.
>>>>>>>=20
>>>>>>> Any kernel that I tried that was from after -r313271 had the
>>>>>>> problems.
>>>>>>>=20
>>>>>>> Of course I did not try them all in other direction. :)
>>>>>>>=20
>>>>>>=20
>>>>>> I found that spin mutexes were not properly handling this, fixed =
in
>>>>>> r313996.
>>>>>>=20
>>>>>> Locally I added a if (cpu_tick() % 2) return (0); snipped to =
amd64
>>>>>> fcmpset to simulate failures. Everything works, while it would =
easily
>>>>>> fail without the patch.
>>>>>>=20
>>>>>> That said, I hope this concludes the 'missing check for =
not-reread value
>>>>>> of failed fcmpset' saga.
>>>>>>=20
>>>>>> --=20
>>>>>> Mateusz Guzik <mjguzik gmail.com>
>>>>>=20
>>>>> -r313999 is an improvement for powerpc64: it boots and I can
>>>>> log in on the old PowerMac G5 so-called "Quad Core".
>>>>>=20
>>>>> But, e.g., buildworld buildkernel eventually hangs and later
>>>>> the powerpc64 panics for "spin lock held too long".
>>>>>=20
>>>>=20
>>>> Allright, play time is over.
>>>>=20
>>>> Can you please:
>>>> 1. verify r313254 is stable for you
>>>> 2. apply =
https://people.freebsd.org/~mjg/patches/complete-locks.diff and
>>>> https://people.freebsd.org/~mjg/.junk/ppc.diff on top of it and =
retry
>>>> the test?
>>>>=20
>>>> This is a workaround which effectively disables the =
powerpc-specific
>>>> primitive and makes it use a cmpset wrapper instead. I don't have =
the
>>>> hardware to test right now and my attempts to boot in qemu also =
failed.
>>>>=20
>>>> That said, does not look like there are general fcmpset bugs left =
and
>>>> the remaining issue seems powerpc-specific.
>>>>=20
>>>> If this works, I'll commit the workaround for the time being as in =
few
>>>> weeks I'd like to start merging the work back to stable/11.
>>>>=20
>>>> --=20
>>>> Mateusz Guzik <mjguzik gmail.com>
>>>=20
>>> I've started a self-hosted powerpc64 -r313254 build
>>> based on running the -r313266 kernel. (The context=20
>>> sometimes do cross builds in is tied up with other
>>> things. -r313266 is what my prior bisection came up
>>> with as the last appearently-working kernel at the
>>> time.)
>>>=20
>>> So it will be a while before I have a -r313254 in
>>> place to try: the self-hosted build takes longer
>>> and so will not be installed for a while.
>>>=20
>>> To judge stability I'll probably have -e313254 build
>>> the patched update that you want me to test, initially
>>> doing a cleanworld. So that too will take a while.
>>>=20
>>> (The above wording presumes all goes well.)
>>>=20
>>> I'll let you know as I go along if I run into anything
>>> interesting.
>>>=20
>>>=20
>>> My builds are rebuilding both world and kernel since
>>> what turns into /usr/include/sys/* has changes in your
>>> patch.
>>>=20
>>> The builds are without MALLOC_PRODUCTION but are
>>> otherwise not debug builds.
>>>=20
>>>=20
>>> I've not seen anything indicating that anyone has
>>> been trying TARGET_ARCH=3Dpowerpc. I've been trying
>>> TARGET_ARCH=3Dpowerpc64 .
>>>=20
>>> While I do not have access to a true
>>> TARGET_ARCH=3Dpowerpc machine currently, such a build
>>> can be used on a PowerMac G5 so-called "Quad Core".
>>> So I could eventually build and try such on the one
>>> powerpc family machine that I currently have access
>>> to.
>>>=20
>>> clang 3.9.1 has a significant code generation problem
>>> for TARGET_ARCH=3Dpowerpc and so I'd have to use
>>> a gcc 4.2.1 based build for that sort of experiment.
>>> (There is no xtoolchain for 32-bit powerpc.)
>>>=20
>>> I use clang 3.9.1 or xtoolchain for
>>> TARGET_ARCH=3Dpowerpc64 and have been using clang 3.9.1
>>> in recent times. My primary powerpc family use has
>>> been to experiment with building based on the
>>> modern libc++ and reporting issues discovered in the
>>> attempts. This explains the clang/xtoolchain context.
>>>=20
>>> clang 3.9.1 has major problems for C++ exception
>>> handling for both powerpc64 and powerpc but a
>>> lot of FreeBSD is independent of throwing C++
>>> exceptions. By contrast xtoolchain-based works
>>> for C++ exception handling but lib32 fails
>>> to operate when built by a xtoolchain build.
>>=20
>> -r313254 had no trouble booting or building
>> the patched version or anything else involved
>> in getting there or installing.
>>=20
>> But the patched version failed quickly just
>> attempting cleanworld's recursive remove. (So
>> it did boot and let me log in.) The panic
>> description was:
>>=20
>> panic: vn_finished_secondary_write: neg cnt
>>=20
>>=20
>> The sources that are different from svn's -r313254
>> are (some tied to arm64 experiments, most everything
>> else tied to powerpc64 and/or powerpc, those not
>> from your patches are long standing from my
>> investigations or from Justin H.):
>>=20
>> # svnlite status /usr/src | sort
>> . . . (ignoring the ? lines) . . .
>> M       /usr/src/bin/sh/jobs.c
>> M       /usr/src/bin/sh/miscbltin.c
>> M       /usr/src/contrib/llvm/lib/Target/PowerPC/PPCInstrInfo.td
>> M       /usr/src/contrib/llvm/tools/lld/ELF/Target.cpp
>> M       /usr/src/lib/csu/powerpc64/Makefile
>> M       /usr/src/libexec/rtld-elf/Makefile
>> M       /usr/src/sys/arm/arm/gic.c
>> M       /usr/src/sys/boot/ofw/Makefile.inc
>> M       /usr/src/sys/boot/powerpc/Makefile.inc
>> M       /usr/src/sys/boot/powerpc/kboot/Makefile
>> M       /usr/src/sys/boot/uboot/Makefile.inc
>> M       /usr/src/sys/conf/kmod.mk
>> M       /usr/src/sys/ddb/db_main.c
>> M       /usr/src/sys/ddb/db_script.c
>> M       /usr/src/sys/kern/init_main.c
>> M       /usr/src/sys/kern/kern_condvar.c
>> M       /usr/src/sys/kern/kern_lock.c
>> M       /usr/src/sys/kern/kern_lockstat.c
>> M       /usr/src/sys/kern/kern_mutex.c
>> M       /usr/src/sys/kern/kern_rwlock.c
>> M       /usr/src/sys/kern/kern_sx.c
>> M       /usr/src/sys/kern/kern_synch.c
>> M       /usr/src/sys/kern/kern_thread.c
>> M       /usr/src/sys/kern/subr_lock.c
>> M       /usr/src/sys/kern/vfs_default.c
>> M       /usr/src/sys/kern/vfs_subr.c
>> M       /usr/src/sys/powerpc/include/atomic.h
>> M       /usr/src/sys/powerpc/ofw/ofw_machdep.c
>> M       /usr/src/sys/sys/lock.h
>> M       /usr/src/sys/sys/lockmgr.h
>> M       /usr/src/sys/sys/lockstat.h
>> M       /usr/src/sys/sys/mutex.h
>> M       /usr/src/sys/sys/rwlock.h
>> M       /usr/src/sys/sys/sdt.h
>> M       /usr/src/sys/sys/sx.h
>> M       /usr/src/sys/sys/systm.h
>=20
> To recover from the problem and again have a buildworld
> buildkernel present I've booted based on:
>=20
> A) The -r313254 kernel without your patches (kernel.old).
> B) The -r313254 world (which had your patches in its
>   build).
>=20
> I've reverted the /usr/src/ to not have your patches
> (but does have my prior ones from prior activity).
>=20
> I repeated the cleanworld to let it finish after its
> prior failure (that failed during a SSD trim activity).
>=20
> I've started buildworld buildkernel (with -j 4 as is
> normal for my context).
>=20
> So far this combination seems to be working fine. This
> suggests that the sys/sys/*.h files that ended up in
> /usr/include/sys/ and the sys/powerpc/include/atomic.h
> that ended up in /usr/include/machine/ were not problems
> as used in the world code --since those uses are still in
> place in the binaries being used. Only the kernel
> binaries seem to be a problem (not necessarily all of
> them).

Unfortunately it eventually got a panic for a Data Storage
Interrupt.

I may not be unable to do a self hosted build to get things
back to normal.=20

=3D=3D=3D
Mark Millard
markmi at dsl-only.net




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B38C6E4C-DC41-44A6-A99B-CD6FA5CB105C>