Date: Sun, 27 Nov 2016 15:17:13 -0500 From: David Cross <dcrosstech@gmail.com> To: freebsd-hackers@freebsd.org Subject: Re: FreeBSD 11 i386 disk deadlock (I think) (now with reproduction steps!) Message-ID: <CAM9edeMYMhnkWid7Lig5D-FjhahniFm0VbFRm8ysyb85h29wXg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
So, narrowing this down, I think it has something to do with geli swap
(since I can easily reproduce it with geli swap, but have yet to reproduce
it without).. and I have a bit of a convoluted way almost anyone can
reproduce it with bhyve. (Note, I haven't been able to get a crashdump,
since apparently the VM system being locked up prevents that, but with
watchdogd, I have been able to get into DDB)
Anyway, my reproduction steps, I used the 11.0 Retail DVD, but I fully
suspect the 11.0-RELEASE image will be fine to install an i386 image into
bhyve; I install to vtbd disks (even though my 'real' case is to an ada
device, that this can be repro-ed across such wide "hardware" really
reduces the likelyhood of a device driver issue)
After its installed, I start my VM with the following (dropping memory to
the floor, well below my "real" machine, but the emulated machine is much
faster and I suspsect this is a race condition somewhere), note the options
to the virtio-blk device to pin it to "real" and not hit the host vmcache,
again speed seems to be key here, and slowing things down makes it more
likely to happen.
bhyveload -m 64M -d /usr/bhyve/11.0.1-i386.img fbsd11-i386
bhyve -u -A -c 1 -H -m 64M -C -s 0,hostbridge -s 1,lpc -s 2,virtio-net,tap0
-s 3,virtio-blk,/usr/bhyve/11.0.1-i386.img,nocache,direct -l
com1,/dev/nmdm0A fbsd11-i386
At this point:
Log into the VM
cd /usr/src
/usr/bin/make buildkernel
<wait>
For me this has hung 99% of the time at:
objcopy --strip-debug kernel
Once you've gotten here once, I have been able to just skip the rest of the
compile, cd /usr/obj/usr/src/sys/GENERIC run that command directly and
trigger the condition.
What I have at this point is the following DDB ps list:
db> ps
pid ppid pgrp uid state wmesg wchan cmd
645 633 633 0 DV pfault 0xc1c4f6d8 watchdogd
641 621 641 0 D+ pfault 0xc1c4f6d8 objcopy
633 1 633 0 Ds ppwait 0xc37f9994 watchdogd
621 620 621 0 SW+ pause 0xc3cb03a8 csh
620 1 620 0 SWs+ wait 0xc3cb0688 login
619 1 619 0 Ss+ ttyin 0xc35d0470 getty
618 1 618 0 Ss+ ttyin 0xc35d0670 getty
617 1 617 0 Ss+ ttyin 0xc35d0870 getty
616 1 616 0 Ss+ ttyin 0xc35d0a70 getty
615 1 615 0 Ss+ ttyin 0xc35d0c70 getty
614 1 614 0 Ss+ ttyin 0xc35d0e70 getty
613 1 613 0 Ss+ ttyin 0xc35d1c70 getty
612 1 612 0 Ss+ ttyin 0xc35d1a70 getty
567 1 567 0 Ss nanslp 0xc1bcd248 cron
563 1 563 25 SWs pause 0xc3a9ba30 sendmail
560 1 560 0 Ss select 0xc37e1124 sendmail
557 1 557 0 Ss select 0xc3728ca4 sshd
363 1 363 0 Ss select 0xc37e0f24 syslogd
290 1 290 0 Ds pfault 0xc1c4f6d8 devd
50 0 0 0 DL vmwait 0xc1c4f6d8 [g_eli[0] vtbd0p3]
21 0 0 0 DL vlruwt 0xc37f7688 [vnlru]
20 0 0 0 DL syncer 0xc1bee444 [syncer]
19 0 0 0 DL - 0xc1bede7c [bufspacedaemon]
18 0 0 0 DL (threaded) [bufdaemon]
100046 D psleep 0xc1bed284 [bufdaemon]
100056 D sdflush 0xc398f484 [/ worker]
100060 D sdflush 0xc398f084 [/usr worker]
100061 D sdflush 0xc398ec84 [/var worker]
17 0 0 0 DL pgzero 0xc1bf3a28 [pagezero]
16 0 0 0 DL psleep 0xc1bf38a4 [vmdaemon]
15 0 0 0 DL (threaded) [pagedaemon]
100043 D wswbuf0 0xc1bf30d4 [pagedaemon]
100050 D umarcl 0xc1bf34a0 [uma]
14 0 0 0 DL - 0xc1becd1c [soaiod4]
9 0 0 0 DL - 0xc1becd1c [soaiod3]
8 0 0 0 DL - 0xc1becd1c [soaiod2]
7 0 0 0 DL - 0xc1becd1c [soaiod1]
6 0 0 0 DL - 0xc1aa6eec [rand_harvestq]
5 0 0 0 DL waiting_ 0xc1c4c540 [sctp_iterator]
4 0 0 0 DL (threaded) [cam]
100019 D - 0xc19f2f80 [doneq0]
100038 D - 0xc19f2e2c [scanner]
3 0 0 0 DL crypto_r 0xc1bf27d8 [crypto returns]
2 0 0 0 DL crypto_w 0xc1bf2718 [crypto]
13 0 0 0 DL (threaded) [geom]
100013 D - 0xc1c47660 [g_event]
100014 D - 0xc1c47664 [g_up]
100015 D - 0xc1c47668 [g_down]
12 0 0 0 WL (threaded) [intr]
100004 I [swi4: clock (0)]
100005 I [swi3: vm]
100006 I [swi1: netisr 0]
100007 I [swi6: Giant taskq]
100010 I [swi5: fast taskq]
100012 I [swi6: task queue]
100023 I [irq264: virtio_pci0]
100024 I [irq265: virtio_pci0]
100025 I [irq266: virtio_pci0]
100028 I [irq267: virtio_pci1]
100029 I [irq268: virtio_pci1]
100030 I [irq1: atkbd0]
100031 I [irq12: psm0]
100032 I [swi0: uart uart]
100033 I [irq14: ata0]
100034 I [irq15: ata1]
11 0 0 0 RL CPU 0 [idle: cpu0]
1 0 1 0 SLs wait 0xc35ee9cc [init]
10 0 0 0 DL audit_wo 0xc1c4f430 [audit]
0 0 0 0 DLs (threaded) [kernel]
100000 D vmwait 0xc1c4f6d8 [swapper]
100008 D - 0xc35f4f00 [aiod_kick taskq]
100009 D - 0xc35f4e80 [thread taskq]
100011 D - 0xc35f4d00 [kqueue_ctx taskq]
100016 D - 0xc35f4380 [firmware taskq]
100020 D - 0xc35f3680 [acpi_task_0]
100021 D - 0xc35f3680 [acpi_task_1]
100022 D - 0xc35f3680 [acpi_task_2]
100026 D - 0xc35f6d00 [vtnet0 rxq 0]
100027 D - 0xc35f6c80 [vtnet0 txq 0]
100037 D - 0xc35f3a00 [CAM taskq]
I note that the swapper and that geli are both in vmwait, and a bunch of
other processes are in pfault, and the "crypto" drivers are in disk wait??
I am not sure what else to do here. I also started bhyve with -C, so I can
control-\ and get a process dump that includes guestvm.. but i am likewise
unsure where else to proceed. I am happy to provide core dumps/etc to
people, this is a dedicated VM with nothing in it.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM9edeMYMhnkWid7Lig5D-FjhahniFm0VbFRm8ysyb85h29wXg>