From owner-svn-src-projects@freebsd.org Tue Dec 15 23:49:31 2015 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CA86A48B1A for ; Tue, 15 Dec 2015 23:49:31 +0000 (UTC) (envelope-from asomers@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6E95F15ED; Tue, 15 Dec 2015 23:49:31 +0000 (UTC) (envelope-from asomers@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id tBFNnUYa037346; Tue, 15 Dec 2015 23:49:30 GMT (envelope-from asomers@FreeBSD.org) Received: (from asomers@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id tBFNnTCr037332; Tue, 15 Dec 2015 23:49:29 GMT (envelope-from asomers@FreeBSD.org) Message-Id: <201512152349.tBFNnTCr037332@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: asomers set sender to asomers@FreeBSD.org using -f From: Alan Somers Date: Tue, 15 Dec 2015 23:49:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r292293 - in projects/zfsd/head/tests/sys/cddl/zfs: include tests/hotspare tests/zfsd X-SVN-Group: projects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2015 23:49:31 -0000 Author: asomers Date: Tue Dec 15 23:49:29 2015 New Revision: 292293 URL: https://svnweb.freebsd.org/changeset/base/292293 Log: Fix problems with disks disappearing during the ZFS tests, fix some other ZFS test problems while I'm at it. tests/sys/cddl/zfs/include/libsas.kshlib Make "i" a local instead of global variable in find_verify_sas_disk. The global definition of i was shadowing the local definition, causing find_verify_sas_disk to fail to find the disk. I think that the local variable would've shadowed the global instead of the other way around, but in this case another function explicitly typeset i an integer type, and find_verify_sas_disk was trying to set it to a string. The problem only happened if find_verify_sas_disk was called after wait_until_resilvered, which is why it's hard to notice. tests/sys/cddl/zfs/tests/hotspare/hotspare.kshlib Eliminate harmless error message on cleanup tests/sys/cddl/zfs/tests/zfsd/zfsd_replace_001_pos.ksh tests/sys/cddl/zfs/tests/zfsd/zfsd.cfg tests/sys/cddl/zfs/tests/zfsd/zfsd_test.sh Fix zfsd_replace_001_pos, which has been marked as an expected failure for a long time. It was failing because the parallel dd processes finished too quickly, so the test assumed that they had crashed. But there's really no reason why they need to be parallel, so I just replaced them with a single process. tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_001_neg.ksh tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_002_pos.ksh tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_003_pos.ksh tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_004_pos.ksh tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_007_pos.ksh tests/sys/cddl/zfs/tests/zfsd/zfsd_import_001_pos.ksh Merge common cleanup code into one place. Add a "camcontrol smpphylist" commmand for debugging purposes. Sponsored by: Spectra Logic Corp Modified: projects/zfsd/head/tests/sys/cddl/zfs/include/libsas.kshlib projects/zfsd/head/tests/sys/cddl/zfs/tests/hotspare/hotspare.kshlib projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.cfg projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_001_neg.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_002_pos.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_003_pos.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_004_pos.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_007_pos.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_import_001_pos.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_replace_001_pos.ksh projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_test.sh Modified: projects/zfsd/head/tests/sys/cddl/zfs/include/libsas.kshlib ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/include/libsas.kshlib Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/include/libsas.kshlib Tue Dec 15 23:49:29 2015 (r292293) @@ -43,6 +43,7 @@ function find_verify_sas_disk { typeset DISK=${1##*/} + typeset i if [ ! -c /dev/$DISK ]; then log_fail "Cannot find device \"/dev/$DISK\", arg is \"$1\"" Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/hotspare/hotspare.kshlib ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/hotspare/hotspare.kshlib Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/hotspare/hotspare.kshlib Tue Dec 15 23:49:29 2015 (r292293) @@ -33,7 +33,7 @@ function cleanup_devices_all { $RM -f ${devarray[*]} - rmdir $HOTSPARE_TMPDIR + [ -n "$HOTSPARE_TMPDIR" ] && rmdir $HOTSPARE_TMPDIR return 0 } Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.cfg ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.cfg Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.cfg Tue Dec 15 23:49:29 2015 (r292293) @@ -27,5 +27,3 @@ export TESTFILE=testfile.${TESTCASE_ID} export STF_TIMEOUT=300 -export SAS_IO_PROCESSES=2 -export SAS_IO_MB_PER_PROC=512 Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd.kshlib Tue Dec 15 23:49:29 2015 (r292293) @@ -131,3 +131,26 @@ function do_autoreplace fi } +function autoreplace_cleanup +{ + poolexists $TESTPOOL && \ + destroy_pool $TESTPOOL + + # See if the phy has been disabled, and try to re-enable it if possible. + echo DISK params are "$REMOVAL_DISK" $EXPANDER $PHY + if [ -n "$REMOVAL_DISK" ]; then + if [ -n "$EXPANDER" ] && [ -n "$PHY" ]; then + log_note "Renabling ${EXPANDER}:${PHY} for disk ${REMOVAL_DISK}" + enable_sas_disk $EXPANDER $PHY + # For debugging purposes, log the partial output of + # camcontrol to see if the disk actually came back. + log_note `camcontrol smpphylist ${EXPANDER} | ${GREP} "^ *${PHY}"` + fi + fi + + [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* + + partition_cleanup +} + + Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_001_neg.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_001_neg.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_001_neg.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -64,27 +64,10 @@ verify_runnable "global" verify_disk_count "$DISKS" 5 -function cleanup -{ - poolexists $TESTPOOL && \ - destroy_pool $TESTPOOL - - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$REMOVAL_DISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* - - partition_cleanup -} - - log_assert "A pool with the autoreplace property set will replace disks by physical path" -log_onexit cleanup + +log_onexit autoreplace_cleanup function verify_assertion { Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_002_pos.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_002_pos.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_002_pos.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -64,27 +64,10 @@ verify_runnable "global" verify_disk_count "$DISKS" 5 -function cleanup -{ - poolexists $TESTPOOL && \ - destroy_pool $TESTPOOL - - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$REMOVAL_DISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* - - partition_cleanup -} - log_assert "A pool with the autoreplace property will replace disks by physical path" -log_onexit cleanup +log_onexit autoreplace_cleanup function verify_assertion { Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_003_pos.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_003_pos.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_autoreplace_003_pos.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -66,27 +66,10 @@ verify_runnable "global" verify_disk_count "$DISKS" 5 -function cleanup -{ - poolexists $TESTPOOL && \ - destroy_pool $TESTPOOL - - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$REMOVAL_DISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* - - partition_cleanup -} - log_assert "A pool with the autoreplace property will replace disks by physical path" -log_onexit cleanup +log_onexit autoreplace_cleanup function verify_assertion { Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_004_pos.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_004_pos.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_004_pos.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -27,6 +27,7 @@ # ident "@(#)hotspare_replace_006_pos.ksh 1.0 12/08/10 SL" # . $STF_SUITE/tests/hotspare/hotspare.kshlib +. $STF_SUITE/tests/zfsd/zfsd.kshlib . $STF_SUITE/include/libsas.kshlib ################################################################################ @@ -61,27 +62,10 @@ verify_runnable "global" verify_disk_count "$DISKS" 5 -function cleanup -{ - poolexists $TESTPOOL && \ - destroy_pool $TESTPOOL - - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$REMOVAL_DISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* - - partition_cleanup -} - log_assert "Removing a disk from a pool results in the spare activating" -log_onexit cleanup +log_onexit autoreplace_cleanup function verify_assertion # spare_dev Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_007_pos.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_007_pos.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_hotspare_007_pos.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -27,6 +27,7 @@ # ident "@(#)hotspare_replace_006_pos.ksh 1.0 12/08/10 SL" # . $STF_SUITE/tests/hotspare/hotspare.kshlib +. $STF_SUITE/tests/zfsd/zfsd.kshlib . $STF_SUITE/include/libsas.kshlib ################################################################################ @@ -61,27 +62,10 @@ verify_runnable "global" verify_disk_count "$DISKS" 5 -function cleanup -{ - poolexists $TESTPOOL && \ - destroy_pool $TESTPOOL - - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$REMOVAL_DISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* - - partition_cleanup -} - log_assert "zfsd will spare missing drives on startup" -log_onexit cleanup +log_onexit autoreplace_cleanup function verify_assertion # spare_dev Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_import_001_pos.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_import_001_pos.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_import_001_pos.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -27,6 +27,7 @@ # ident "@(#)zfsd_zfsd_002_pos.ksh 1.0 12/08/10 SL" # . $STF_SUITE/tests/hotspare/hotspare.kshlib +. $STF_SUITE/tests/zfsd/zfsd.kshlib . $STF_SUITE/include/libsas.kshlib ################################################################################ @@ -65,24 +66,6 @@ verify_runnable "global" -function cleanup -{ - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$REMOVAL_DISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* - - poolexists "$TESTPOOL" && \ - destroy_pool "$TESTPOOL" - - - partition_cleanup -} - function verify_assertion # spare_dev { typeset sdev=$1 @@ -206,7 +189,7 @@ fi log_assert "If a removed drive gets reinserted while the pool is exported, it will replace its spare when reinserted." -log_onexit cleanup +log_onexit autoreplace_cleanup set_devs Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_replace_001_pos.ksh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_replace_001_pos.ksh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_replace_001_pos.ksh Tue Dec 15 23:49:29 2015 (r292293) @@ -34,65 +34,21 @@ # $FreeBSD$ . $STF_SUITE/tests/hotspare/hotspare.kshlib +. $STF_SUITE/tests/zfsd/zfsd.kshlib . $STF_SUITE/include/libtest.kshlib . $STF_SUITE/include/libsas.kshlib verify_runnable "global" -function cleanup -{ - if [[ -n "$child_pids" ]]; then - for wait_pid in $child_pids - do - $KILL $wait_pid - done - fi - - if poolexists $TESTPOOL; then - destroy_pool $TESTPOOL - fi - - # See if the phy has been disabled, and try to re-enable it if possible. - if [ ! -z "$TMPDISK" ]; then - if [ ! -z "$EXPANDER" ] && [ ! -z "$PHY" ]; then - enable_sas_disk $EXPANDER $PHY - fi - fi - - [[ -e $TESTDIR ]] && log_must $RM -rf $TESTDIR/* -} - log_assert "Failing a disk from a SAS expander is recognized by ZFS" -log_onexit cleanup +log_onexit autoreplace_cleanup child_pids="" -function run_io -{ - typeset -i processes=$1 - typeset -i mbcount=$2 - typeset i=0 - - while [[ $i -lt $processes ]]; do - log_note "Invoking dd if=/dev/zero of=$TESTDIR/$TESTFILE.$i &" - dd if=/dev/zero of=$TESTDIR/$TESTFILE.$i bs=1m count=$mbcount & - typeset pid=$! - - $SLEEP 1 - if ! $PS -p $pid > /dev/null 2>&1; then - log_fail "dd if=/dev/zero $TESTDIR/$TESTFILE.$i" - fi - - child_pids="$child_pids $pid" - ((i = i + 1)) - done - -} - set -A TMPDISKS $DISKS -typeset TMPDISK=${TMPDISKS[0]} -TMPDISK=${TMPDISK##*/} +typeset REMOVAL_DISK=${TMPDISKS[0]} +REMOVAL_DISK=${REMOVAL_DISK##*/} for type in "raidz" "mirror"; do # Create a pool on the supplied disks @@ -101,36 +57,32 @@ for type in "raidz" "mirror"; do log_must $ZFS set mountpoint=$TESTDIR $TESTPOOL/$TESTFS # Find the first disk, get the expander and phy - log_note "Looking for expander and phy information for $TMPDISK" - find_verify_sas_disk $TMPDISK + log_note "Looking for expander and phy information for $REMOVAL_DISK" + find_verify_sas_disk $REMOVAL_DISK - log_note "Disabling \"$TMPDISK\" on expander $EXPANDER phy $PHY" + log_note "Disabling \"$REMOVAL_DISK\" on expander $EXPANDER phy $PHY" # Disable the first disk. We have to do this first, because if # there is I/O active to the disable_sas_disk $EXPANDER $PHY # Check to make sure disk is gone. - camcontrol inquiry $TMPDISK > /dev/null 2>&1 + camcontrol inquiry $REMOVAL_DISK > /dev/null 2>&1 if [ $? = 0 ]; then - log_fail "Disk \"$TMPDISK\" was not removed" + log_fail "Disk \"$REMOVAL_DISK\" was not removed" fi # Write out data to make sure we can do I/O after the disk failure # XXX KDM should check the status returned from the dd instances - log_note "Running $SAS_IO_PROCESSES dd runs of $SAS_IO_MB_PER_PROC MB" - run_io $SAS_IO_PROCESSES $SAS_IO_MB_PER_PROC - - # Wait for the I/O to complete - wait + log_must dd if=/dev/zero of=$TESTDIR/$TESTFILE bs=1m count=512 # We waited for the child processes to complete, so they're done. child_pids="" # Check to make sure ZFS sees the disk as removed - $ZPOOL status $TESTPOOL |grep $TMPDISK |egrep -q 'REMOVED' + $ZPOOL status $TESTPOOL |grep $REMOVAL_DISK |egrep -q 'REMOVED' if [ $? != 0 ]; then $ZPOOL status $TESTPOOL - log_fail "disk $TMPDISK not listed as removed" + log_fail "disk $REMOVAL_DISK not listed as removed" fi # Make sure that the pool is degraded @@ -155,18 +107,18 @@ for type in "raidz" "mirror"; do if [ -z "$FOUNDDISK" ]; then camcontrol $EXPANDER - log_fail "Disk $TMPDISK has not appeared at phy $PHY on expander $EXPANDER after 50 seconds" + log_fail "Disk $REMOVAL_DISK has not appeared at phy $PHY on expander $EXPANDER after 50 seconds" else - log_note "Disk $TMPDISK is back as $FOUNDDISK" + log_note "Disk $REMOVAL_DISK is back as $FOUNDDISK" fi log_note "Raid type is $type" #Disk should have auto-joined the zpool. Verify it's status is online. - $ZPOOL status |grep $TMPDISK |grep ONLINE > /dev/null + $ZPOOL status |grep $REMOVAL_DISK |grep ONLINE > /dev/null if [ $? != 0 ]; then $ZPOOL status $TESTPOOL - log_fail "disk $TMPDISK did not automatically join the $TESTPOOL" + log_fail "disk $REMOVAL_DISK did not automatically join the $TESTPOOL" else log_note "After reinsertion, disk is back in pool and online" fi Modified: projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_test.sh ============================================================================== --- projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_test.sh Tue Dec 15 23:45:11 2015 (r292292) +++ projects/zfsd/head/tests/sys/cddl/zfs/tests/zfsd/zfsd_test.sh Tue Dec 15 23:49:29 2015 (r292293) @@ -458,7 +458,6 @@ zfsd_replace_001_pos_head() } zfsd_replace_001_pos_body() { - atf_expect_fail "TeamTrack P3_29531 zpool status prematurely indicates resilver completion" export TESTCASE_ID=$(echo $(atf_get ident) | cksum -o 2 | cut -f 1 -d " ") . $(atf_get_srcdir)/../../include/default.cfg . $(atf_get_srcdir)/zfsd.cfg