From owner-freebsd-hackers@freebsd.org Sun Nov 27 20:17:14 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8456C59263 for ; Sun, 27 Nov 2016 20:17:14 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x234.google.com (mail-yw0-x234.google.com [IPv6:2607:f8b0:4002:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8987C1222 for ; Sun, 27 Nov 2016 20:17:14 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x234.google.com with SMTP id a10so99895602ywa.3 for ; Sun, 27 Nov 2016 12:17:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=6xyN7d4cviGvY2tIDs4L0Ctgu3WJn0nawIKvSvLq5ho=; b=LZWoIZh6ASUTTQBTKD+Mhcij0kWHkFgt1s+AIEo58ZibeEquoF4haZQUFy5ggpkCcJ 5UUvJLOkxwS4PWMjS/VCGRMHrxL+Tkrk0iaR3LlKkcjcaSu7XXMxYcTAjSSQ8xdK9oQG x/p066mIlmiFeUagQX+zOJliQtqieLmPbraaKI8tGuE1RdbLuc5IEe8yuWKZUcra/i6j WhBq2oMaDHTuUpikbFGXEREKDi0MdG244CoWKibFl3vMbly4dDEzKSPKDh4QOH7/bYYo SBsaRkjlnXxRlRPVwzZrlBGiFGMCNVoX/SpcLATonvbkJV41VplR7y4V17z0N5XRzOpE +XrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=6xyN7d4cviGvY2tIDs4L0Ctgu3WJn0nawIKvSvLq5ho=; b=koowLUXriG1BcXAxOf7owWLmwpxl61rXBhs4inSV1egMiF8787fZCzLppJ9rp7Di8t YKntgdzy2Ujrsvc3BBukdXDCOwh1RCNvTO/fnyCJE8uXrH/KOBRFPAa+MjvtXE/QECQg 3UvtKLzUpu3FGVY1vxVp0HNCkpsNMU3ZmHmZvNS1H+y+9siXk8EMjopeLYiQuXBZoz7p fvtE9TyTaMUj9kkxr0dcGiDulfp0+HsDsdlnL/hXORdEsJ/sSW5Yp3SmXU8KRgiilNPf mzcY0Y84pubtgtbU0opbXodbt5OIarYtefAk8Q9tOaQqdAB4C6cY6QkyXn9xdG1tiugK CWxQ== X-Gm-Message-State: AKaTC01dEnF0WmR2MesRaP3VHgBazzYgkueFQdxMN6cLREQB9aL8T8kCuxLC8+pyKCt5JcW+L8nua4ClsbSAvA== X-Received: by 10.13.227.3 with SMTP id m3mr21490094ywe.272.1480277833372; Sun, 27 Nov 2016 12:17:13 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.201.197 with HTTP; Sun, 27 Nov 2016 12:17:13 -0800 (PST) From: David Cross Date: Sun, 27 Nov 2016 15:17:13 -0500 Message-ID: Subject: Re: FreeBSD 11 i386 disk deadlock (I think) (now with reproduction steps!) To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Nov 2016 20:17:14 -0000 So, narrowing this down, I think it has something to do with geli swap (since I can easily reproduce it with geli swap, but have yet to reproduce it without).. and I have a bit of a convoluted way almost anyone can reproduce it with bhyve. (Note, I haven't been able to get a crashdump, since apparently the VM system being locked up prevents that, but with watchdogd, I have been able to get into DDB) Anyway, my reproduction steps, I used the 11.0 Retail DVD, but I fully suspect the 11.0-RELEASE image will be fine to install an i386 image into bhyve; I install to vtbd disks (even though my 'real' case is to an ada device, that this can be repro-ed across such wide "hardware" really reduces the likelyhood of a device driver issue) After its installed, I start my VM with the following (dropping memory to the floor, well below my "real" machine, but the emulated machine is much faster and I suspsect this is a race condition somewhere), note the options to the virtio-blk device to pin it to "real" and not hit the host vmcache, again speed seems to be key here, and slowing things down makes it more likely to happen. bhyveload -m 64M -d /usr/bhyve/11.0.1-i386.img fbsd11-i386 bhyve -u -A -c 1 -H -m 64M -C -s 0,hostbridge -s 1,lpc -s 2,virtio-net,tap0 -s 3,virtio-blk,/usr/bhyve/11.0.1-i386.img,nocache,direct -l com1,/dev/nmdm0A fbsd11-i386 At this point: Log into the VM cd /usr/src /usr/bin/make buildkernel For me this has hung 99% of the time at: objcopy --strip-debug kernel Once you've gotten here once, I have been able to just skip the rest of the compile, cd /usr/obj/usr/src/sys/GENERIC run that command directly and trigger the condition. What I have at this point is the following DDB ps list: db> ps pid ppid pgrp uid state wmesg wchan cmd 645 633 633 0 DV pfault 0xc1c4f6d8 watchdogd 641 621 641 0 D+ pfault 0xc1c4f6d8 objcopy 633 1 633 0 Ds ppwait 0xc37f9994 watchdogd 621 620 621 0 SW+ pause 0xc3cb03a8 csh 620 1 620 0 SWs+ wait 0xc3cb0688 login 619 1 619 0 Ss+ ttyin 0xc35d0470 getty 618 1 618 0 Ss+ ttyin 0xc35d0670 getty 617 1 617 0 Ss+ ttyin 0xc35d0870 getty 616 1 616 0 Ss+ ttyin 0xc35d0a70 getty 615 1 615 0 Ss+ ttyin 0xc35d0c70 getty 614 1 614 0 Ss+ ttyin 0xc35d0e70 getty 613 1 613 0 Ss+ ttyin 0xc35d1c70 getty 612 1 612 0 Ss+ ttyin 0xc35d1a70 getty 567 1 567 0 Ss nanslp 0xc1bcd248 cron 563 1 563 25 SWs pause 0xc3a9ba30 sendmail 560 1 560 0 Ss select 0xc37e1124 sendmail 557 1 557 0 Ss select 0xc3728ca4 sshd 363 1 363 0 Ss select 0xc37e0f24 syslogd 290 1 290 0 Ds pfault 0xc1c4f6d8 devd 50 0 0 0 DL vmwait 0xc1c4f6d8 [g_eli[0] vtbd0p3] 21 0 0 0 DL vlruwt 0xc37f7688 [vnlru] 20 0 0 0 DL syncer 0xc1bee444 [syncer] 19 0 0 0 DL - 0xc1bede7c [bufspacedaemon] 18 0 0 0 DL (threaded) [bufdaemon] 100046 D psleep 0xc1bed284 [bufdaemon] 100056 D sdflush 0xc398f484 [/ worker] 100060 D sdflush 0xc398f084 [/usr worker] 100061 D sdflush 0xc398ec84 [/var worker] 17 0 0 0 DL pgzero 0xc1bf3a28 [pagezero] 16 0 0 0 DL psleep 0xc1bf38a4 [vmdaemon] 15 0 0 0 DL (threaded) [pagedaemon] 100043 D wswbuf0 0xc1bf30d4 [pagedaemon] 100050 D umarcl 0xc1bf34a0 [uma] 14 0 0 0 DL - 0xc1becd1c [soaiod4] 9 0 0 0 DL - 0xc1becd1c [soaiod3] 8 0 0 0 DL - 0xc1becd1c [soaiod2] 7 0 0 0 DL - 0xc1becd1c [soaiod1] 6 0 0 0 DL - 0xc1aa6eec [rand_harvestq] 5 0 0 0 DL waiting_ 0xc1c4c540 [sctp_iterator] 4 0 0 0 DL (threaded) [cam] 100019 D - 0xc19f2f80 [doneq0] 100038 D - 0xc19f2e2c [scanner] 3 0 0 0 DL crypto_r 0xc1bf27d8 [crypto returns] 2 0 0 0 DL crypto_w 0xc1bf2718 [crypto] 13 0 0 0 DL (threaded) [geom] 100013 D - 0xc1c47660 [g_event] 100014 D - 0xc1c47664 [g_up] 100015 D - 0xc1c47668 [g_down] 12 0 0 0 WL (threaded) [intr] 100004 I [swi4: clock (0)] 100005 I [swi3: vm] 100006 I [swi1: netisr 0] 100007 I [swi6: Giant taskq] 100010 I [swi5: fast taskq] 100012 I [swi6: task queue] 100023 I [irq264: virtio_pci0] 100024 I [irq265: virtio_pci0] 100025 I [irq266: virtio_pci0] 100028 I [irq267: virtio_pci1] 100029 I [irq268: virtio_pci1] 100030 I [irq1: atkbd0] 100031 I [irq12: psm0] 100032 I [swi0: uart uart] 100033 I [irq14: ata0] 100034 I [irq15: ata1] 11 0 0 0 RL CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc35ee9cc [init] 10 0 0 0 DL audit_wo 0xc1c4f430 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D vmwait 0xc1c4f6d8 [swapper] 100008 D - 0xc35f4f00 [aiod_kick taskq] 100009 D - 0xc35f4e80 [thread taskq] 100011 D - 0xc35f4d00 [kqueue_ctx taskq] 100016 D - 0xc35f4380 [firmware taskq] 100020 D - 0xc35f3680 [acpi_task_0] 100021 D - 0xc35f3680 [acpi_task_1] 100022 D - 0xc35f3680 [acpi_task_2] 100026 D - 0xc35f6d00 [vtnet0 rxq 0] 100027 D - 0xc35f6c80 [vtnet0 txq 0] 100037 D - 0xc35f3a00 [CAM taskq] I note that the swapper and that geli are both in vmwait, and a bunch of other processes are in pfault, and the "crypto" drivers are in disk wait?? I am not sure what else to do here. I also started bhyve with -C, so I can control-\ and get a process dump that includes guestvm.. but i am likewise unsure where else to proceed. I am happy to provide core dumps/etc to people, this is a dedicated VM with nothing in it.