From owner-freebsd-scsi@freebsd.org Wed Jun 7 08:18:33 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 652A6BFE2F2 for ; Wed, 7 Jun 2017 08:18:33 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1080566719; Wed, 7 Jun 2017 08:18:32 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id v578ISF3076861; Wed, 7 Jun 2017 10:18:28 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id CB361A6E; Wed, 7 Jun 2017 10:18:27 +0200 (CEST) Message-ID: <5937B6C6.9020300@omnilan.de> Date: Wed, 07 Jun 2017 10:18:14 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Stephen Mcconnell CC: freebsd-scsi@freebsd.org, Scott Long , "Kenneth D. Merry" , Stephen Mcconnell Subject: Re: sporadic CAM (all devices) outage on 11-stable, mps(4), ahci(4) and bhyve(8) involved. [Was: Re: mps(4) blocks panic-reboot] References: <592FDE8C.1090609@omnilan.de> 12a36df9eff99c77ec621987efbe75fe@mail.gmail.com <59303484.1040609@omnilan.de> <593056E9.6000807@omnilan.de> <59305D4F.40707@omnilan.de> <59306693.6080304@omnilan.de> In-Reply-To: <59306693.6080304@omnilan.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Wed, 07 Jun 2017 10:18:28 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2017 08:18:33 -0000 Bezüglich Harry Schmalzbauer's Nachricht vom 01.06.2017 21:10 (localtime): > Bezüglich Stephen Mcconnell's Nachricht vom 01.06.2017 20:55 (localtime): >> Take a look at PR 212914. Could that be the issue? It was MFC'd to stable/11 >> with r309273 on Nov 28th, 2016. > Thanks a lot, but that's unrelated. Unfortunately, today a similar lockup occured :-( I was informed by mps(4): (da1:mps0:0:3:0): READ(10). CDB: 28 00 06 7e 4d 53 00 00 10 00 (da1:mps0:0:3:0): CAM status: Unrecoverable Host Bus Adapter Error (da1:mps0:0:3:0): Retrying command (da1:mps0:0:3:0): WRITE(10). CDB: 2a 00 06 f8 c5 1f 00 00 38 00 (da1:mps0:0:3:0): CAM status: Unrecoverable Host Bus Adapter Error (da1:mps0:0:3:0): Retrying command (da1:mps0:0:3:0): WRITE(10). CDB: 2a 00 06 f8 c5 1f 00 00 38 00 (da1:mps0:0:3:0): CAM status: SCSI Status Error (da1:mps0:0:3:0): SCSI status: Check Condition (da1:mps0:0:3:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da1:mps0:0:3:0): Error 6, Retries exhausted (da1:mps0:0:3:0): Invalidating pack But it seemed all drives got lost again (although the kernel message couldn't be printed anymore), since on another still responsive (memorydisk rootfs) session I could get the zpool status and zfs reported all members having outstanding requests: pool: cetusPsys state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://illumos.org/msg/ZFS-8000-JQ scan: none requested config: NAME STATE READ WRITE CKSUM cetusPsys ONLINE 370 13 0 mirror-0 ONLINE 40 12 0 gpt/cetusSYSzd1of4 ONLINE 3 26 0 da2 ONLINE 3 16 0 mirror-1 ONLINE 700 9 0 gpt/cetusSYSzd2of4 ONLINE 3 9 0 da3 ONLINE 3 54 0 I'll do anything I can do to help tracking this problem, since the one thing happened which I have taken massive precaution not to happen... a freezing hypervisor :-( Thanks, -harry (In case one is following any of my other recent PRs: This time, no passthru-enabled-VM was involved. The latter causes some very serious memory corruption IMHO... This machine is a XEON E3 with ECC, neither MBC nor MCE reports ECC errors... From owner-freebsd-scsi@freebsd.org Fri Jun 9 19:17:33 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D8AEBFB321 for ; Fri, 9 Jun 2017 19:17:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8C51D790EF for ; Fri, 9 Jun 2017 19:17:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v59JHWgw091023 for ; Fri, 9 Jun 2017 19:17:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 219866] [iscsi] ctld crashes inside ctl_datamove() Date: Fri, 09 Jun 2017 19:17:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 19:17:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219866 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-scsi@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-scsi@freebsd.org Fri Jun 9 19:18:12 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87295BFB415 for ; Fri, 9 Jun 2017 19:18:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 761DB791D1 for ; Fri, 9 Jun 2017 19:18:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v59JICgK092039 for ; Fri, 9 Jun 2017 19:18:12 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 219857] panic in scsi_cd code Date: Fri, 09 Jun 2017 19:18:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 19:18:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219857 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-scsi@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.=