Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 8 Feb 2005 20:08:36 +0100
From:      Peter Holm <peter@holm.cc>
To:        John Baldwin <jhb@freebsd.org>
Cc:        freebsd-current@freebsd.org
Subject:   Re: Deadlock with option FULL_PREEMPTION
Message-ID:  <20050208190836.GA45874@peter.osted.lan>
In-Reply-To: <200502081317.40267.jhb@FreeBSD.org>
References:  <20050206134113.GA77071@peter.osted.lan> <200502081317.40267.jhb@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Feb 08, 2005 at 01:17:40PM -0500, John Baldwin wrote:
> On Sunday 06 February 2005 08:41 am, Peter Holm wrote:
> > With GENERIC HEAD from Feb 5 09:19 UTC + FULL_PREEMPTION +
> > mpsafe_vfs = 1 I ran into what appears to be the same deadlock
> > twice. This is the first one:
> > http://www.holm.cc/stress/log/cons114.html
> 
> What is the deadlock exactly?

top froze, console login froze after giving login name, but I
could ping the box.

> It looks like lots of threads banging on fork() 
> and that they are all waiting on an exclusive lock of allproc_lock while 
> holding a shared lock of proctree_lock (except for the 1 thread currently 
> doing a fork that is on a run queue because it was preempted by IRQ 0 which 
> kicked off softclock).  Can you get 'ps' output?
> 

Yes:

$ ps -alx -o flags -N kernel.debug -M /var/crash/vmcore.167
  UID   PID  PPID CPU PRI NI   VSZ  RSS MWCHAN STAT  TT       TIME COMMAND                F
    0     0     0   0 -16  0     0    0 allpro DLs   ??    0:00,11 [swapper]            200
    0     1     0 287   8  0   720    0 wait   DLs   ??    0:00,38 [init]              4200
    0     2     0 140  -8  0     0    0 -      DL    ??    0:02,95 [g_event]            204
    0     3     0 273  -8  0     0    0 -      DL    ??    4:00,96 [g_up]               204
    0     4     0  91  -8  0     0    0 -      DL    ??    1:02,45 [g_down]             204
    0     5     0   0   8  0     0    0 -      DL    ??    0:00,00 [kqueue taskq]       204
    0     6     0   0   8  0     0    0 -      RL    ??    0:00,00 [thread taskq]       204
    0     7     0   0 -84  0     0    0 actask DL    ??    0:00,00 [acpi_task0]         204
    0     8     0   0 -84  0     0    0 actask DL    ??    0:00,00 [acpi_task1]         204
    0     9     0   0 -84  0     0    0 actask DL    ??    0:00,00 [acpi_task2]         204
    0    10     0   0 -16  0     0    0 ktrace DL    ??    0:00,00 [ktrace]             204
    0    11     0 219 171  0     0    0 -      RL    ??   10:22,62 [idle: cpu0]         20c
    0    12     0   2 -60  0     0    0 -      WL    ??    0:00,04 [irq1: atkbd0]       204
    0    13     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq3:]              204
    0    14     0   0 -60  0     0    0 -      WL    ??    0:00,00 [irq4: sio0]         204
    0    15     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq5:]              204
    0    16     0   0 -64  0     0    0 -      WL    ??    0:00,00 [irq6: fdc0]         204
    0    17     0   0 -60  0     0    0 -      WL    ??    0:00,00 [irq7: ppc0]         204
    0    18     0   0 -84  0     0    0 -      WL    ??    0:00,00 [irq8: rtc]          204
    0    19     0   0 -52  0     0    0 -      WL    ??    0:00,00 [irq9: acpi0]        204
    0    20     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq10:]             204
    0    21     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq11:]             204
    0    22     0   0 -60  0     0    0 -      WL    ??    0:00,00 [irq12: psm0]        204
    0    23     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq13:]             204
    0    24     0   0 -64  0     0    0 -      WL    ??    0:25,54 [irq14: ata0]        204
    0    25     0   0 -64  0     0    0 -      WL    ??    0:00,00 [irq15: ata1]        204
    0    26     0   0 -64  0     0    0 -      WL    ??    0:00,00 [irq16: uhci0 uh     204
    0    27     0   0 -80  0     0    0 -      WL    ??    0:00,00 [irq17: pcm0]        204
    0    28     0   0 -64  0     0    0 -      WL    ??    0:00,00 [irq18: uhci2]       204
    0    29     0   0 -64  0     0    0 -      WL    ??    0:00,00 [irq19: uhci1]       204
    0    30     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq20:]             204
    0    31     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq21:]             204
    0    32     0  24 -68  0     0    0 -      WL    ??    0:56,13 [irq22: rl0]         204
    0    33     0   0 -21  0     0    0 -      WL    ??    0:00,00 [irq23:]             204
    0    34     0   0 -84  0     0    0 -      WL    ??    0:00,00 [irq0: clk]          204
    0    35     0 227 -44  0     0    0 -      WL    ??    2:15,59 [swi1: net]          204
    0    36     0 295 -32  0     0    0 -      WL    ??    1:20,97 [swi4: clock sio     20c
    0    37     0   0 -36  0     0    0 -      WL    ??    0:00,00 [swi3: vm]           204
    0    38     0  42 -16  0     0    0 -      DL    ??    0:03,48 [yarrow]             204
    0    39     0   0 -24  0     0    0 -      WL    ??    0:00,00 [swi6: task queu     204
    0    40     0   0 -24  0     0    0 -      WL    ??    0:00,00 [swi6:+]             204
    0    41     0   0 -24  0     0    0 -      WL    ??    0:00,00 [swi6: acpitaskq     204
    0    42     0   0 -28  0     0    0 -      WL    ??    0:00,00 [swi5:+]             204
    0    43     0   0 -40  0     0    0 -      WL    ??    0:00,00 [swi2: cambio]       204
    0    44     0   0   8  0     0    0 usbevt DL    ??    0:00,00 [usb0]               204
    0    45     0   0   8  0     0    0 usbtsk DL    ??    0:00,00 [usbtask]            204
    0    46     0   0   8  0     0    0 usbevt DL    ??    0:00,00 [usb1]               204
    0    47     0   0   8  0     0    0 usbevt DL    ??    0:00,00 [usb2]               204
    0    48     0   1   8  0     0    0 usbevt DL    ??    0:00,00 [usb3]               204
    0    49     0   0 -48  0     0    0 -      WL    ??    0:00,00 [swi0: sio]          204
    0    50     0   6  -8  0     0    0 -      DL    ??    0:00,14 [fdc0]               204
    0    51     0 239 -16  0     0    0 psleep DL    ??    3:59,81 [pagedaemon]         204
    0    52     0  36  20  0     0    0 psleep DL    ??    0:12,59 [vmdaemon]           204
    0    53     0  20 171  0     0    0 -      RL    ??    0:13,25 [pagezero]           20c
    0    54     0 139 -16  0     0    0 psleep DL    ??   20:59,10 [bufdaemon]          204
    0    55     0  13  -4  0     0    0 vlruwt DL    ??    0:00,28 [vnlru]              204
    0    56     0 283  -4  0     0    0 ufs    DL    ??    5:52,85 [syncer]             204
    0    57     0  27   8  0     0    0 -      DL    ??    0:00,00 [nfsiod 0]           204
    0    58     0  27   8  0     0    0 -      DL    ??    0:00,00 [nfsiod 1]           204
    0    59     0  28   8  0     0    0 -      DL    ??    0:00,00 [nfsiod 2]           204
    0    60     0  28   8  0     0    0 -      DL    ??    0:00,00 [nfsiod 3]           204
    0    61     0   0 -32  0     0    0 allpro DL    ??    0:01,64 [schedcpu]           204
    0   249     1 120 111  0   488    0 select Ds    ??    0:00,00 [devd]                 0
    0   274     1   6  96  0  1328  120 select Ds    ??    0:00,54 [syslogd]              0
    0   288     1   4  96  0  1472    0 select Ds    ??    0:00,18 [rpcbind]              0
    0   357     1 120 111  0  1352    0 select Ds    ??    0:00,02 [mountd]               0
    0   359     1  77 105  0  1296    0 select Ds    ??    0:00,09 [nfsd]                 0
    0   360   359 120   4  0  1236    0 -      D     ??    0:00,00 [nfsd]                 0
    0   361   359 120   4  0  1236    0 -      D     ??    0:00,00 [nfsd]                 0
    0   362   359 120   4  0  1236    0 -      D     ??    0:00,00 [nfsd]                 0
    0   363   359 120   4  0  1236    0 -      D     ??    0:00,00 [nfsd]                 0
    0   384     1   4  96  0  1244    0 select Ds    ??    0:00,14 [usbd]                 0
    0   410     1 110 109  0  2828    0 select Ds    ??    0:01,66 [ntpd]                 0
    0   426     1   0  96  0  3348    0 select Ds    ??    0:00,02 [sshd]               100
    0   432     1   0  96  0  3440  208 sysctl Ds    ??    0:00,96 [sendmail]           100
   25   436     1   0  96  0  3320    0 sysctl Ds    ??    0:00,05 [sendmail]           100
    0   448     1   1  96  0  1356    0 allpro Ds    ??    0:00,26 [cron]                 0
    0   499     1 112 110  0  1244    0 select Ds    ??    0:00,00 [moused]               0
    0   542   426  66   4  0  6092    0 sbwait Ds    ??    0:00,22 [sshd]              4100
 1001   544   542   0  96  0  6088  344 select D     ??    0:01,81 [sshd]               100
    0   552   426  13   4  0  6092    0 sbwait Ds    ??    0:00,22 [sshd]              4100
 1001   554   552   0  96  0  6248    0 select D     ??    0:02,62 [sshd]               100
 1001   545   544  38   8  0  2520    0 wait   Ds    p0    0:00,10 [bash]              4002
 1001   551   545  70 104  0  2452  624 allpro D+    p0    0:51,24 [top]               4002
 1001   555   554 162   8  0  2520    0 wait   Ds    p1    0:00,21 [bash]              4002
 1001  2120   555 247   8  0  1664    0 wait   D+    p1    0:00,03 [sh]                4002
 1001  2125  2120  22   8  0  1228    0 nanslp D+    p1    0:00,44 [run]               4002
 1001  2126  2125 247   8  0  1228    0 wait   D+    p1    0:00,00 [run]                  2
 1001  2127  2126 285   8  0  1232    0 wait   D+    p1    0:01,12 [run]                  2
 1001 10334  2127 281 131  0  1224    0 allpro D+    p1    0:00,02 [swap]              4002
 1001 10335  2127 280 131  0  1224    0 allpro D+    p1    0:00,02 [creat]             4002
 1001 10336  2127 280 131  0  1228    0 allpro D+    p1    0:00,02 [mkdir]             4002
 1001 10337  2127   0 -84  0     0    0 -      ZW    p1    0:00,00 <defunct>              0
 1001 10338  2127   0 -84  0     0    0 -      ZW    p1    0:00,00 <defunct>              0
 1001 10339  2127   0 -84  0     0    0 -      ZW    p1    0:00,00 <defunct>              0
 1001 10340  2127 280 131  0  1228    0 allpro D+    p1    0:00,02 [sysctl]            4002
 1001 10341  2127 280 131  0  1232    0 allpro D+    p1    0:00,02 [tcp]               4002
 1001 10342  2127 281 131  0  1228    0 allpro D+    p1    0:00,02 [udp]               4002
 1001 10343 10335 285   8  0  1224    0 wait   D+    p1    0:00,01 [creat]                2
 1001 10344 10340 285   8  0  1228    0 wait   D+    p1    0:00,00 [sysctl]               2
 1001 10345 10334 285   8  0  1224    0 wait   D+    p1    0:00,01 [swap]                 2
 1001 10346 10344 285 131  0  1228    0 allpro D+    p1    0:00,00 [sysctl]               2
 1001 10347 10345 285 131  0  1224    0 sysctl D+    p1    0:00,00 [swap]                 2
 1001 10348 10345 295 131  0 30684    0 -      R+    p1   88:53,23 [swap]                 2
 1001 10349 10345 285 131  0  1224    0 sysctl D+    p1    0:00,00 [swap]                 2
 1001 10350 10343 285  -4  0  1224    0 ufs    D+    p1    0:00,00 [creat]                2
 1001 10351 10343 290 132  0  1224    0 -      R+    p1    0:00,11 [creat]                2
 1001 10352 10343 285  -4  0  1224    0 ufs    D+    p1    0:00,00 [creat]                2
 1001 10353 10343 285  -4  0  1224    0 ufs    D+    p1    0:00,00 [creat]                2
 1001 10354 10343 289 132  0  1224    0 -      R+    p1    0:00,07 [creat]                2
 1001 10355 10343 285  -4  0  1224    0 ufs    D+    p1    0:00,00 [creat]                2
 1001 10356 10345 285 131  0  1224    0 sysctl D+    p1    0:00,00 [swap]                 2
 1001 10357 10345 285 131  0  1224    0 sysctl D+    p1    0:00,00 [swap]                 2
 1001 10358 10345 285 131  0  1224    0 sysctl D+    p1    0:00,00 [swap]                 2
 1001 10359 10345 285  -8  0  1232    0 piperd D+    p1    0:00,00 [swap]                 2
 1001 10360 10336 285   8  0  1228    0 wait   D+    p1    0:00,00 [mkdir]                2
 1001 10361 10341 285   8  0  1232    0 wait   D+    p1    0:00,02 [tcp]                  2
 1001 10362 10360 285 131  0  1228    0 sysctl D+    p1    0:00,00 [mkdir]                2
 1001 10363 10360 285  -4  0  1228    0 getblk D+    p1    0:00,00 [mkdir]                2
 1001 10364 10361 285   4  0  1236    0 sbwait D+    p1    0:00,04 [tcp]                  2
 1001 10365 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10366 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10367 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10368 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10369 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10370 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10371 10361 281   4  0  1232    0 sbwait D+    p1    0:00,03 [tcp]                  2
 1001 10372 10361 285 131  0  1232    0 allpro D+    p1    0:00,00 [tcp]                  2
 1001 10373 10361 284   4  0  1236    0 sbwait D+    p1    0:00,05 [tcp]                  2
 1001 10374 10342 285   8  0  1228    0 wait   D+    p1    0:00,01 [udp]                  2
 1001 10375 10374 290 132  0  1252    0 -      R+    p1    0:00,67 [udp]                  2
 1001 10376 10374 288 132  0  1252    0 -      R+    p1    0:00,03 [udp]                  2
 1001 10377 10374 288 132  0  1252    0 -      R+    p1    0:00,03 [udp]                  2
 1001 10378 10371 288 132  0  1320    0 -      R+    p1    0:00,05 [tcp]                  2
 1001 10382 10364 288 132  0  1320    0 -      R+    p1    0:00,07 [tcp]                  2
 1001 10383 10373 291 132  0  1320    0 -      R+    p1    0:00,08 [tcp]                  2
 1001 10385 10359 288 132  0  1664    0 -      R+    p1    0:00,02 [sh]                4002
 1001 10386 10385 288 132  0  1220    0 -      R+    p1    0:00,01 [swapinfo]          4002
 1001 10387 10385 288 132  0   288    0 -      R+    p1    0:00,00 [tail]              4002
    0   534     1   1  96  0   288    0 sysctl Ds+   v0    0:00,03 [login]             4002
    0   535     1 112   5  0  1300    0 ttyin  Ds+   v1    0:00,02 [getty]             4002
    0   536     1 112   5  0  1300    0 ttyin  Ds+   v2    0:00,02 [getty]             4002
    0   537     1 112   5  0  1300    0 ttyin  Ds+   v3    0:00,02 [getty]             4002
    0   538     1 112   5  0  1300    0 ttyin  Ds+   v4    0:00,02 [getty]             4002
    0   539     1 112   5  0  1300    0 ttyin  Ds+   v5    0:00,02 [getty]             4002
    0   540     1 112   5  0  1300    0 ttyin  Ds+   v6    0:00,02 [getty]             4002
    0   541     1 112   5  0  1300    0 ttyin  Ds+   v7    0:00,02 [getty]             4002

I have uploaded the other freeze, that has a "show witness", to
http://www.holm.cc/stress/log/cons114a.html.

Hope this helps.

- Peter

> -- 
> John Baldwin <jhb@FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
> "Power Users Use the Power to Serve"  =  http://www.FreeBSD.org

-- 
Peter Holm



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050208190836.GA45874>