Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 6 May 2012 18:03:54 +0000
From:      "Bjoern A. Zeeb" <bz@FreeBSD.org>
To:        Brooks Davis <brooks@freebsd.org>, Ryan Stone <rysto32@gmail.com>
Cc:        svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org
Subject:   Re: svn commit: r234504 - in head/sys: amd64/conf i386/conf
Message-ID:  <3D17FEF6-8CC5-4BBB-B098-BEA7D28F377D@FreeBSD.org>
In-Reply-To: <20120421171128.GA6732@lor.one-eyed-alien.net>
References:  <201204202137.q3KLbhNj056524@svn.freebsd.org> <CAFMmRNy6Ew_A1%2BCAq5Off%2BNxYxEMBHs8ZgfyG7pvVbbR9sCk7Q@mail.gmail.com> <20120421171128.GA6732@lor.one-eyed-alien.net>

next in thread | previous in thread | raw e-mail | index | archive | help

On 21. Apr 2012, at 17:11 , Brooks Davis wrote:

> On Sat, Apr 21, 2012 at 09:45:57AM -0400, Ryan Stone wrote:
>> On Fri, Apr 20, 2012 at 5:37 PM, Brooks Davis <brooks@freebsd.org> =
wrote:
>>> Author: brooks
>>> Date: Fri Apr 20 21:37:42 2012
>>> New Revision: 234504
>>> URL: http://svn.freebsd.org/changeset/base/234504
>>>=20
>>> Log:
>>> ?Enable DTrace hooks in GENERIC.
>>>=20
>>> ?Reviewed by: ?gnn
>>> ?Approved by: ?core (jhb, imp)
>>> ?Requested by: a cast of thousands
>>> ?MFC after: ? ?3 days
>>=20
>> Excellent!  Thanks to everybody who helped make this happen, starting
>> with the participants at dtrace.conf who gave us the requisite whacks
>> with the clue-by-four.
>>=20
>> However, what is our policy for enabling features in -STABLE that are
>> known to be unstable?  If we MFC this I don't have the slightest =
worry
>> that somebody might see instability in their system just because the
>> hooks are all of a sudden there, but I would worry that somebody make
>> take DTrace hooks being enabled in GENERIC on -STABLE to imply that
>> DTrace is stable, start using it and being upset when they trip over =
a
>> DTrace bug.
>=20
> I think we should note that it remains experimental and somewhat buggy
> in the release notes and probably in the MFC message.  Otherwise, it's
> like any new feature.  If you start using a feature in production
> without extensive testing you may be surprised.

I am sorry but I am close to requesting a backout now but I hope someone
else might be able to debug it further, so I'll wait for another couple
of days.

I have heard this from multiple people and I am now at the point where
I wasted almost a day or work on this over the course of two weeks.

I am not able to finish HEAD snapshot release builds even after updating
the underlying kernel and world to a today's HEAD  (I had to disable the
WITH_CTF=3D to get the kernel now booted to build).

I have seen this on multiple setups for about two weeks.
The common factor is always that ctfmerge is hanging forever.  I have =
seen
this on both i386 and amd64.  I have seen it with modules and with
kernels built with NO_MODULES.  I have seen it on NFS and on local =
disks.

As son as I add the
	# Temprary as ctfmerge hangs.
	nomakeoptions   WITH_CTF
things are fine again.

At the moment it looks like this:

root 34611   0.0  0.1 35908  8736  -  I     4:19PM   0:00.07 ctfmerge -L =
VERSION -g -o if_ath.ko.debug if_ath.o if_ath_debug.o if_ath_keycache.o =
if_ath_sysctl.o if_ath_tx.o if_ath_tx_ht.o if_ath_led.o ah_osdep.o ah.o =
ah_regdom
root 39282   0.0  0.2 44104 14516  -  I     4:20PM   0:00.06 ctfmerge -L =
VERSION -g -o kernel.debug locore.o aic7xxx_reg_print.o =
aic79xx_reg_print.o cam.o cam_periph.o cam_queue.o cam_sim.o cam_xpt.o =
ata_all.o ata_xpt.o ata_pm


root@bz1:/home/bz # procstat -k 34611
  PID    TID COMM             TDNAME           KSTACK                    =
  =20
34611 100228 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
34611 100452 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
34611 100453 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
34611 100454 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
34611 100455 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
34611 100456 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
root@bz1:/home/bz # procstat -k 39282
  PID    TID COMM             TDNAME           KSTACK                    =
  =20
39282 100348 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep do_wait =
__umtx_op_wait_uint_private amd64_syscall Xfast_syscall=20
39282 100457 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
39282 100458 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
39282 100459 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
39282 100460 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20
39282 100461 ctfmerge         -                mi_switch =
sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex =
do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20


root@bz1:/home/bz # procstat -t 34611
  PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN   =20=

34611 100228 ctfmerge         -                  7  152 sleep   umtxn    =
=20
34611 100452 ctfmerge         -                  1  152 sleep   umtxn    =
=20
34611 100453 ctfmerge         -                  1  152 sleep   umtxn    =
=20
34611 100454 ctfmerge         -                  4  152 sleep   umtxn    =
=20
34611 100455 ctfmerge         -                  0  152 sleep   umtxn    =
=20
34611 100456 ctfmerge         -                  5  152 sleep   umtxn    =
=20
root@bz1:/home/bz # procstat -t 39282
  PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN   =20=

39282 100348 ctfmerge         -                  7  152 sleep   uwait    =
=20
39282 100457 ctfmerge         -                  2  152 sleep   umtxn    =
=20
39282 100458 ctfmerge         -                  4  152 sleep   umtxn    =
=20
39282 100459 ctfmerge         -                  1  152 sleep   umtxn    =
=20
39282 100460 ctfmerge         -                  5  152 sleep   umtxn    =
=20
39282 100461 ctfmerge         -                  0  152 sleep   umtxn    =
=20


root@bz1:/home/bz # procstat -i 34611 | grep -v -- '---$'
  PID COMM             SIG     FLAGS
34611 ctfmerge         INT      --C
34611 ctfmerge         QUIT     --C
34611 ctfmerge         TERM     --C
34611 ctfmerge         URG      -I-
34611 ctfmerge         TSTP     -I-
34611 ctfmerge         CHLD     -I-
34611 ctfmerge         TTIN     -I-
34611 ctfmerge         TTOU     -I-
34611 ctfmerge         IO       -I-
34611 ctfmerge         WINCH    -I-
34611 ctfmerge         INFO     -I-
34611 ctfmerge         32       --C
root@bz1:/home/bz # procstat -i 39282 | grep -v -- '---$'
  PID COMM             SIG     FLAGS
39282 ctfmerge         INT      --C
39282 ctfmerge         QUIT     --C
39282 ctfmerge         TERM     --C
39282 ctfmerge         URG      -I-
39282 ctfmerge         TSTP     -I-
39282 ctfmerge         CHLD     -I-
39282 ctfmerge         TTIN     -I-
39282 ctfmerge         TTOU     -I-
39282 ctfmerge         IO       -I-
39282 ctfmerge         WINCH    -I-
39282 ctfmerge         INFO     -I-
39282 ctfmerge         32       --C



root@bz1:/home/bz # procstat -j 34611 | grep -v -- '--$'
  PID    TID COMM             SIG     FLAGS
34611 100452 ctfmerge         INT      -B
34611 100452 ctfmerge         QUIT     -B
34611 100452 ctfmerge         TERM     -B
34611 100453 ctfmerge         INT      -B
34611 100453 ctfmerge         QUIT     -B
34611 100453 ctfmerge         TERM     -B
34611 100454 ctfmerge         INT      -B
34611 100454 ctfmerge         QUIT     -B
34611 100454 ctfmerge         TERM     -B
34611 100455 ctfmerge         INT      -B
34611 100455 ctfmerge         QUIT     -B
34611 100455 ctfmerge         TERM     -B
34611 100456 ctfmerge         INT      -B
34611 100456 ctfmerge         QUIT     -B
34611 100456 ctfmerge         TERM     -B
root@bz1:/home/bz # procstat -j 39282 | grep -v -- '--$'
  PID    TID COMM             SIG     FLAGS
39282 100457 ctfmerge         INT      -B
39282 100457 ctfmerge         QUIT     -B
39282 100457 ctfmerge         TERM     -B
39282 100458 ctfmerge         INT      -B
39282 100458 ctfmerge         QUIT     -B
39282 100458 ctfmerge         TERM     -B
39282 100459 ctfmerge         INT      -B
39282 100459 ctfmerge         QUIT     -B
39282 100459 ctfmerge         TERM     -B
39282 100460 ctfmerge         INT      -B
39282 100460 ctfmerge         QUIT     -B
39282 100460 ctfmerge         TERM     -B
39282 100461 ctfmerge         INT      -B
39282 100461 ctfmerge         QUIT     -B
39282 100461 ctfmerge         TERM     -B



So given I have been upset I started to investigate some more.

make -C /usr/src    buildworld  did not work either (no -j<n>).

So I started and limited the build to 1 core:

cpuset -l 1 make -C /usr/src buildkernel   did not work either.


So added the nomakeoptions WITH_CTF  and

make -C /usr/src -j8 buildkernel  completes.

So removed the makeoptions WITH_CTF from GENRIC and the nomakeoptions =
from my kernel including GENERIC and added it to the command line:

make -C /usr/src -j8 buildkernel WITH_CTF=3D=20

and that doesn't seem to work at all anymore?

"/usr/src/share/mk/bsd.own.mk", line 482: WITH_CTF and WITHOUT_CTF can't =
both be set.


So I did:
echo WITH_CTF=3D >> /etc/src.conf
make -C /usr/src -j8 buildkernel

and of course it did not work again.



Given all this and given it has reliably worked in the past, my =
conclusions is that it must be something related to ctfmerge or =
libraries?
/usr/bin/ctfmerge:
        libctf.so.2 =3D> /lib/libctf.so.2 (0x800829000)
        libdwarf.so.3 =3D> /usr/lib/libdwarf.so.3 (0x800a36000)
        libelf.so.1 =3D> /usr/lib/libelf.so.1 (0x800c3f000)
        libz.so.6 =3D> /lib/libz.so.6 (0x800e58000)
        libthr.so.3 =3D> /lib/libthr.so.3 (0x80106e000)
        libc.so.7 =3D> /lib/libc.so.7 (0x801293000)

What has changed the last 3ish months?

/bz

--=20
Bjoern A. Zeeb                                 You have to have visions!
   It does not matter how good you are. It matters what good you do!




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3D17FEF6-8CC5-4BBB-B098-BEA7D28F377D>