From owner-freebsd-fs@freebsd.org Mon Oct 2 18:41:59 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B930E244CB for ; Mon, 2 Oct 2017 18:41:59 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x232.google.com (mail-wr0-x232.google.com [IPv6:2a00:1450:400c:c0c::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A606F72EFA for ; Mon, 2 Oct 2017 18:41:58 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x232.google.com with SMTP id t76so4488651wrc.3 for ; Mon, 02 Oct 2017 11:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=O4GgAzQkqKJLVOcRiQ1Q0nO8AZLgq1TnDT9kiulgbNE=; b=VzS3DMUkxI2YWT2KtzLMH5PQuicJcfr2gDXnU+V8jDNVwJY5yacI0ETrRhPnKVoElk mpQxZFFROePDx/7klEmZjwqel5NbR7tdVUKw4ZCbwKyDsgXRtkdcfH3J8XM540IA/hNg v+VTKRUQs7holfXLmtyxfE0qIFuy3BzEDgeGzpJejeLQzVvTT83h1PAmG4FAgjTOvKhS pOXhlqgg81GEI0avvPMLHZQ/x1ya1fYEGCgBXzOGVNeg4aV0uIuWjI0rxfxJst2WFt+C PYsexULGzCnIHZy+q8d1cYaOf3VdccZvCKy/Bim7HRHZxg3ybRiVMmXFL62xVqWmqtV0 aR/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=O4GgAzQkqKJLVOcRiQ1Q0nO8AZLgq1TnDT9kiulgbNE=; b=iAsBD5B5YF+snMu4yoVV1QZcLVyHYYCp0yxcwHvBp/b4ZQLXOOaks23HEPB+INXj2b 8d5wT1FggykKjleM2yRoJUTPpEi0/WksaHUIlSXFqgfi4757r5VdxCtvSIPF7ciEKj79 q+EGwdWTZnJjLgLKWidzabqHPbqlCq1LYCWYGuIYLvv9pBRstbYiVys2eOYSdRdvKdNa XbP58dK85XIwG64SQoLtiheZBisGrExs3nPnojlZZoZs9VR2UZ7MmID3tfDoRMHwdmkh /rPIhFBOX27tJD6mC2olmj1xiVO/Y0E4iIFLhTKT68d7UT5Eh04YOu/ywGUmsVZepoJG pRbQ== X-Gm-Message-State: AHPjjUj1ieHkzjRG/H4DxyXSCDk7iu5c9obkBaCoQgxFlUP4Xc76lDgE 82JSN1FrUoZY7hbeUlxg3hERAxHCduM= X-Google-Smtp-Source: AOwi7QCNPWjHCK5FlybMVv4vevauYYDbqD+GbLDLFDPDQxoM9mxasB07RmvuHznlaYg9FDiJYSXQgg== X-Received: by 10.223.195.110 with SMTP id e43mr13548671wrg.219.1506969715984; Mon, 02 Oct 2017 11:41:55 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id n57sm16710146wrn.29.2017.10.02.11.41.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Oct 2017 11:41:54 -0700 (PDT) Subject: Re: ZFS stalled after some mirror disks were lost To: freebsd-fs@freebsd.org References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <71d4416a-3454-df36-adae-34c0b70cd84e@multiplay.co.uk> <8A189756-028A-465E-9962-D0181FAEBB79@gmail.com> <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> From: Steven Hartland Message-ID: <4f725113-bac3-64bb-9858-690811e73153@multiplay.co.uk> Date: Mon, 2 Oct 2017 19:41:56 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <953DD379-C03A-4737-BAD8-14BB2DB4AB05@gmail.com> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:41:59 -0000 I'm guessing that the devices haven't disconnected cleanly so are just stalling all requests to them and hence the pool. I'm not that familiar with iscsi, does it still show under under camcontrol or geom? Does iscsid have any options on how to treat failed devices? On 02/10/2017 19:28, Ben RUBSON wrote: > Before disconnecting the targets the pool was online without any issue. > >> On 02 Oct 2017, at 20:17, Ben RUBSON wrote: >> >> Unfortunately the command stalls / does not return :/ >> >>> On 02 Oct 2017, at 20:15, Steven Hartland wrote: >>> >>> What does zpool status report when you have disconnected the iscsi targets? >>> >>> On 02/10/2017 19:12, Ben RUBSON wrote: >>>> Hi, >>>> >>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>> >>>> home >>>> mirror-0 >>>> label/local1 >>>> label/local2 >>>> label/iscsi1 >>>> label/iscsi2 >>>> mirror-1 >>>> label/local3 >>>> label/local4 >>>> label/iscsi3 >>>> label/iscsi4 >>>> cache >>>> label/local5 >>>> label/local6 >>>> >>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>>> according to "zpool iostat", nothing on local disks (strange but I >>>> noticed that IOs always prefer iscsi disks to local disks). >>>> No write IOs. >>>> >>>> Let's disconnect all iSCSI disks : >>>> iscsictl -Ra >>>> >>>> Expected behavior : >>>> IO activity flawlessly continue on local disks. >>>> >>>> What happened : >>>> All IOs stalled, server only answers to IOs are made to its zroot pool. >>>> All commands related to the iSCSI disks (iscsictl), or to ZFS (zfs/zpool), >>>> don't return. >>>> >>>> Questions : >>>> Why this behavior ? >>>> How to know what happens ? (/var/log/messages says almost nothing) >>>> >>>> I already disconnected the iSCSI disks without any issue in the past, >>>> several times, but there were almost no IOs running. >>>> >>>> Thank you for your help ! >>>> >>>> Ben >>>> >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"