From nobody Thu Nov 10 14:35:28 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N7PWv2Tyfz4XCwr for ; Thu, 10 Nov 2022 14:35:39 +0000 (UTC) (envelope-from dan@sunsaturn.com) Received: from sunsaturn.com (sunsaturn.com [IPv6:2604:4300:a:196::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "sunsaturn.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4N7PWs6sJJz4LSH; Thu, 10 Nov 2022 14:35:37 +0000 (UTC) (envelope-from dan@sunsaturn.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sunsaturn.com header.s=default header.b=eUSoGekx; spf=pass (mx1.freebsd.org: domain of dan@sunsaturn.com designates 2604:4300:a:196::3 as permitted sender) smtp.mailfrom=dan@sunsaturn.com; dmarc=none Received: by sunsaturn.com (Postfix, from userid 1001) id E015D69B44; Thu, 10 Nov 2022 08:35:28 -0600 (CST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=sunsaturn.com; s=default; t=1668090928; bh=zDJv4dWNqvOv9ki7Ol/7yLwcuy1BhFkMT3Ygk1+F+8k=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=eUSoGekxmtkDrhYi+eEYYotCU5srJUJC2Y8p+LRt5bbvStpjay0B3VpNwC8pCkUku bRxVQCxeLJCQrZI5ByVXULPoJBLXuUA57kR8euy08aHvJYp+m8klk8I3l45jMUSzL3 mOmmPWN7iY4JDjLU0Y8egR4aiM/AaCrSg3hEUlRc= Received: from localhost (localhost [127.0.0.1]) by sunsaturn.com (Postfix) with ESMTP id DF39D69BBF; Thu, 10 Nov 2022 08:35:28 -0600 (CST) Date: Thu, 10 Nov 2022 08:35:28 -0600 (CST) From: Dan The Man To: Alexander Motin cc: freebsd-current@FreeBSD.org Subject: Re: vfs.zfs.vol.recursive hang makes it impossible to mount zvol In-Reply-To: <62348dc1-7f9a-f8ae-db77-f2f5d8a709d8@FreeBSD.org> Message-ID: <5014f48c-f70d-abd5-32b2-796acccda928@sunsaturn.com> References: <62348dc1-7f9a-f8ae-db77-f2f5d8a709d8@FreeBSD.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="1969398039-1231111786-1668090928=:15763" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.07 / 15.00]; CTYPE_MIXED_BOGUS(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.997]; NEURAL_HAM_MEDIUM(-0.57)[-0.572]; R_SPF_ALLOW(-0.20)[+ip6:2604:4300:a:196::3]; R_DKIM_ALLOW(-0.20)[sunsaturn.com:s=default]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-current@FreeBSD.org]; MIME_TRACE(0.00)[0:+,1:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[sunsaturn.com:+]; ARC_NA(0.00)[]; ASN(0.00)[asn:33387, ipnet:2604:4300::/32, country:US]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[dan]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[sunsaturn.com: no valid DMARC record]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4N7PWs6sJJz4LSH X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --1969398039-1231111786-1668090928=:15763 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT Hi Alexander, Thank-you for getting back to me. I tried as you suggested going through scsi instead. While that does work being able to mount /dev/da* devices, an issue I have run into is that suspend/resume features do not seem to work with virtio-scsi. If I set it back to virtio-blk through original /dev/zvol/zroot/* devices or through the scsi /dev/da* devices then suspend/resume works. Let me elaborate, here is what I put together today based on your suggestion: router:/root # cat /etc/ctl.conf portal-group pg0 { discovery-auth-group no-authentication listen 127.0.0.1:3260 } target iqn.2005-02.com.sunsaturn:target0 { auth-group no-authentication portal-group pg0 #bhyve virti-iscsi disk - /dev/cam/ctl1.0 port ioctl/1 lun 0 { path /dev/zvol/zroot/asterisk serial 000c2937247001 device-id "iSCSI Disk 000c2937247001" option vendor "FreeBSD" option product "iSCSI Disk" option revision "0123" option insecure_tpc on } } target iqn.2005-02.com.sunsaturn:target1 { auth-group no-authentication portal-group pg0 #bhyve virti-iscsi disk - /dev/cam/ctl2.0 port ioctl/2 lun 0 { path /dev/zvol/zroot/asterisk2 serial 000c2937247002 device-id "iSCSI Disk 000c2937247002" option vendor "FreeBSD" option product "iSCSI Disk" option revision "0123" option insecure_tpc on } lun 1 { path /vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso #byhve seems to just hang when I set it to an actual CDROM so let it default to type 0 #device-type 5 serial 000c2937247003 device-id "iSCSI CDROM ISO 000c2937247003" option vendor "FreeBSD CDROM" option product "iSCSI CDROM" option revision "0123" option insecure_tpc on } } router:/root # cat /etc/iscsi.conf t0 { TargetAddress = 127.0.0.1:3260 TargetName = iqn.2005-02.com.sunsaturn:target0 } t1 { TargetAddress = 127.0.0.1:3260 TargetName = iqn.2005-02.com.sunsaturn:target1 } router:/root # tail -13 /etc/rc.conf #ISCSI - service ctld start && service iscsid start #server ctld_enable="YES" #load /etc/ctl.conf iscsid_enable="YES" #start iscsid process to connect to ctld #client - service iscsictl start iscsictl_enable="YES" #connect to all targets in /etc/iscsi.conf iscsictl_flags="-Aa" #to kill all sessions: #iscsictl -Ra #to restart everything: #iscsictl -Ra #service ctld stop && service iscsid stop #service ctld start && service iscsid start && service iscsictl start router:/root # Now anytime I use: -s 4:0,virtio-scsi,/dev/cam/ctl1.0 I get coredumps as follows when resuming the guest: ./asterisk.sh: line 101: 7621 Segmentation fault (core dumped) bhyve -c $CPU -m $RAM -w -H -A -s 0:0,hostbridge -s 4:0,virtio-scsi,$STORAGE -s 5:0,virtio-net,$TAP -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT -s 30,xhci,tablet -s 31,lpc -l com1,$SERIAL -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -r $DIR/default.ckp $VMNAME The exit code was : 139 However if I used: -s 4:0,virtio-blk,/dev/da1 or even: -s 4:0,virtio-blk,/dev/zvol/zroot/asterisk suspend/resume works properly. Is virtio-scsi supported through the ioctl devices? I'll include my test suspend/resume script if you want to have a look: #####asterisk.sh #!/bin/bash # # General script to test bhyve suspend/resume features # # Requirements: FreeBSD current # git clone https://git.FreeBSD.org/src.git /usr/src # cd /usr/src/sys/amd64/conf (edit MYKERNEL) cp GENERIC MYKERNEL; (add: options BHYVE_SNAPSHOT) # cd /usr/src # (find amount of CPUs and adjust -j below - "dmesg|grep SMP") # make -j12 buildworld -DWITH_BHYVE_SNAPSHOT -DWITH_MALLOC_PRODUCTION # make -j12 buildkernel KERNCONF=MYKERNEL # make installkernel KERNCONF=MYKERNEL # shutdown -r now # cd /usr/src; make installworld # shutdown -r now # etcupdate -B # pkg bootstrap -f #if new freebsd version # pkg upgrade -f #if new freebsd version # # Report anomolies to dan@sunsaturn.com ##############EDIT ME##################### HOST="127.0.0.1" # vncviewer 127.0.0.1:5900 - pkg install tightvnc PORT="5900" WIDTH="800" HEIGHT="600" VMNAME="asterisk" ISO="/vm/.iso/FreeBSD-14.0-CURRENT-amd64-20221103-5cc5c9254da-259005-disc1.iso" DIR="/vm/asterisk" # Used to hold files when guest suspended SERIAL="/dev/nmdm_asteriskA" # For "screen /dev/nmdm_asteriskB" - pkg install screen TAP="tap0" CPU="8" RAM="8G" #For testing virtio-scsi STORAGE="/dev/cam/ctl1.0" # port from /etc/ctl.conf(port ioctl/1) - core dumping on resume DEVICE="virtio-scsi" #for testing virtio-blk # Comment out above 2 lines if using these #DEVICE="virtio-blk" #STORAGE="/dev/zvol/zroot/asterisk" # Standard zvol #STORAGE="/dev/da1" # Block device created from iscsictl ######################################### usage() { echo "Usage: $1 start (Start the guest: $VMNAME)"; echo "Usage: $1 stop (Stop the guest: $VMNAME)"; echo "Usage: $1 resume (Resume the guest from last suspend: $VMNAME)"; echo "Usage: $1 suspend (Suspend the guest: $VMNAME)"; echo "Usage: $1 install (Install new guest: $VMNAME)"; exit } if [ ! -d "$DIR" ]; then mkdir -p $DIR fi #if [ -z "$2" ]; then # usage #else # VMNAME=$2 #fi if [ "$1" == "install" ]; then #Kill it before starting it echo "Execute: screen $SERIAL" bhyvectl --destroy --vm=$VMNAME bhyve -c $CPU -m $RAM -w -H -A \ -s 0:0,hostbridge \ -s 3:0,ahci-cd,$ISO \ -s 4:0,virtio-scsi,$STORAGE \ -s 5:0,virtio-net,$TAP \ -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \ -s 30,xhci,tablet \ -s 31,lpc -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ $VMNAME #kill it after bhyvectl --destroy --vm=$VMNAME elif [ "$1" == "start" ]; then while true do echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT" #Kill it before starting it bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1 if [ -f "$DIR/default.ckp" ]; then bhyve -c $CPU -m $RAM -w -H -A \ -s 0:0,hostbridge \ -s 4:0,virtio-scsi,$STORAGE \ -s 5:0,virtio-net,$TAP \ -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \ -s 30,xhci,tablet \ -s 31,lpc -l com1,$SERIAL \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ $VMNAME fi #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL #if [ "$?" != 0 ]; #then # echo "The exit code was not reboot code 0!: $?" # exit #fi echo "The exit code was : $?" exit done elif [ "$1" == "resume" ]; then while true do echo "Starting $VMNAME -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT" #Kill it before starting it bhyvectl --destroy --vm=$VMNAME > /dev/null 2>&1 if [ -f "$DIR/default.ckp" ]; then bhyve -c $CPU -m $RAM -w -H -A \ -s 0:0,hostbridge \ -s 4:0,virtio-scsi,$STORAGE \ -s 5:0,virtio-net,$TAP \ -s 29,fbuf,tcp=$HOST:$PORT,w=$WIDTH,h=$HEIGHT \ -s 30,xhci,tablet \ -s 31,lpc -l com1,$SERIAL \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -r $DIR/default.ckp \ $VMNAME fi #DISABLING REBOOT LOOP AS SUSPEND RETURNS ERROR CODE 0 AS WELL #if [ "$?" != 0 ]; #then # echo "The exit code was not reboot code 0!: $?" # exit #fi echo "The exit code was : $?" exit done elif [ "$1" == "suspend" ]; then bhyvectl --suspend $DIR/default.ckp --vm=$VMNAME elif [ "$1" == "stop" ]; then bhyvectl --destroy --vm=$VMNAME else usage fi ################ Dan. -- Dan The Man CEO & Founder Websites, Domains and Everything else http://www.SunSaturn.com/aboutus.php Email: Dan@SunSaturn.com PGP Key: https://SunSaturn.com/pgp.txt A1A7 6E84 FB0B 8994 C3B5 A1BA FF6F 4997 7311 C386 On Mon, 7 Nov 2022, Alexander Motin wrote: > On 07.11.2022 18:53, Dan The Man wrote: >> router:~ # sysctl vfs.zfs.vol.recursive=1 >> vfs.zfs.vol.recursive: 0 -> 1 >> router:~ # zpool import >>    pool: testing >>      id: 8013833172609421701 >>   state: ONLINE >>  action: The pool can be imported using its name or numeric identifier. >>  config: >> >>         testing                   ONLINE >>           zvol/zroot/asterisk2p3  ONLINE >> router:~ # zpool import -fR /mnt testing >> >> This hangs forever.... >> The only way to import that pool from the zvol that I know of..... > > Mounting ZFS from ZVOLs is blocked for a reason. It causes deadlocks due to > lock recursion. I don't know what you are trying to achieve, but as > alternatives, the ZVOL can be passed inside VM, it can be shared via iSCSI > (even inside the host itself), etc. > > -- > Alexander Motin > --1969398039-1231111786-1668090928=:15763--