From owner-freebsd-stable@FreeBSD.ORG Tue Dec 4 10:50:19 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 38719981 for ; Tue, 4 Dec 2012 10:50:19 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.102]) by mx1.freebsd.org (Postfix) with ESMTP id E0C4E8FC13 for ; Tue, 4 Dec 2012 10:50:18 +0000 (UTC) Received: from [78.35.171.127] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1TfpzV-0007mz-O1; Tue, 04 Dec 2012 11:44:41 +0100 Date: Tue, 4 Dec 2012 11:44:33 +0100 From: Fabian Keil To: Harald Schmalzbauer Subject: Re: geom using 100% cpu with failed da5. How to calm it down without cam passdev? Message-ID: <20121204114433.1de6b740@fabiankeil.de> In-Reply-To: <50BD9D46.5010905@omnilan.de> References: <50BD9D46.5010905@omnilan.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/kjEgJZw=P6FdnpjKCThOnKV"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: freebsd-stable@freebsd.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Dec 2012 10:50:19 -0000 --Sig_/kjEgJZw=P6FdnpjKCThOnKV Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Harald Schmalzbauer wrote: > I've a failed disk at a remote server, which shouldn't be a problem > actually. Welcome to geom ... > Just for info, here's the last shout: > kernel: (da5:mps0:0:5:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 > 0 0 0 length 0 SMID 256 command timeout cm 0xffffff8001c64800 ccb > 0xfffffe0007329000 > kernel: mps0: mpssas_alloc_tm freezing simq > kernel: mps0: timedout cm 0xffffff8001c64800 allocated tm > 0xffffff8001c50148 > kernel: (da5:mps0:0:5:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 > 0 0 0 length 0 SMID 256 completed timedout cm 0xffffff8001c64800 ccb > 0xfffffe0007329000 during recovery ioc 8048 scsi 0 state c > xf(noperiph:mps0:0:5:0): SMID 1 abort TaskMID 256 status 0x4a code 0x0 > count 1 > kernel: (noperiph:mps0:0:5:0): SMID 1 finished recovery after > aborting TaskMID 256 > kernel: mps0: mpssas_free_tm releasing simq > kernel: (da5:mps0:0:5:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 > 0 0 0 > kernel: (da5:mps0:0:5:0): CAM status: Command timeout > kernel: (da5:mps0:0:5:0): Retrying command > kernel: (da5:mps0:0:5:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 length 0 > SMID 981 terminated ioc 804b scsi 0 state 0 xfer 0 > kernel: mps0: mpssas_alloc_tm freezing simq > kernel: mps0: mpssas_remove_complete on handle 0x000e, IOCStatus=3D 0= x0 > kernel: mps0: mpssas_free_tm releasing simq > kernel: (da5:mps0:0:(pass7:5:mps0:0:0): lost device - 4 outstanding, > 2 refs > kernel: 5:0): passdevgonecb: devfs entry is gone > kernel: (da5:mps0:0:5:0): oustanding 3 > kernel: (da5:mps0:0:5:0): oustanding 2 > kernel: (da5:mps0:0:5:0): oustanding 1 > kernel: (da5:mps0:0:5:0): oustanding 0 >=20 > After reboot, 'camcontrol devlist' doesn't show any da5, > but 'geom disk list' _does_ show da5!!! >=20 > My problem is that geom is now consuming 100% of one core! > top -S: > 13 root 3 -8 - 0K 48K - 1 480:19 100.00% geom >=20 > Since there's no /dev/da5 I can't use camcontrol to stop anything, and > at the moment nobody can physically remove the failed drive. > How can I calm geom down? I reported a similar problem in: http://www.freebsd.org/cgi/query-pr.cgi?pr=3D171865 The PR contains a patch that I'm using as a workaround. > How can I find out what "geom" is doing/trying to do? > I guess it's related to the failed da5, but how can I know? DTrace might help. Fabian --Sig_/kjEgJZw=P6FdnpjKCThOnKV Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlC91B0ACgkQBYqIVf93VJ2lyACdGgVB0BJlpI1g2emsMm6XDF0q yx8An2GDy3/er5ECVl3e4FhmGXj1htNT =lWMq -----END PGP SIGNATURE----- --Sig_/kjEgJZw=P6FdnpjKCThOnKV--