From owner-freebsd-stable@FreeBSD.ORG Sun May 3 08:45:34 2015 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 59940CFD for ; Sun, 3 May 2015 08:45:34 +0000 (UTC) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D333141E for ; Sun, 3 May 2015 08:45:33 +0000 (UTC) Received: by qcbgy10 with SMTP id gy10so7255219qcb.3 for ; Sun, 03 May 2015 01:45:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=LG7wABuy5l4YmcBPdKYJeVIhrO0DbeAIzkMx3FOr05Y=; b=XMgdzLQP+KXogAp259+1aMP1Kj1yoA5tZleE3z+LsIw0XVAadZYUdzz4faTGWcfDaG I/snpBwNAkz5nYxRCWxV+tqSX/8Ktraff/70Yn5muLK2Y3VrFlwDTGtdcnCC2e9vAc8F 9O0eGD8JyXtiqtFdlddSCk0kGzoxHSclTyymPuk2d5H8G0kXPmnAtFHk9V20F1cP62sR +ZMmM1mdPbBSr/QFXR2MYMizm0dBx+R5kwWToHY4QH2hj2TMBDjqN73h2Mgsmt1/UqjU h2RfcV155ESKN9wIkzd4SQiKWG721c82Dw0kZiBqyCyieRYNzlxkLs9B68fDuCvOxgHH 8sSA== X-Gm-Message-State: ALoCoQmx8ZFjmcvE8VhxFf/O2pL5wf7eJybe3RDEpdedrtQITSbQdcfHFWbAr48BXYmBr1iJBzuT MIME-Version: 1.0 X-Received: by 10.140.194.65 with SMTP id p62mr21596398qha.76.1430632371121; Sat, 02 May 2015 22:52:51 -0700 (PDT) Received: by 10.140.85.84 with HTTP; Sat, 2 May 2015 22:52:51 -0700 (PDT) In-Reply-To: References: Date: Sat, 2 May 2015 23:52:51 -0600 Message-ID: Subject: Re: include/rpcsvc - mkdir: /usr/obj/lib32: Permission denied From: Will Andrews To: Oliver Pinter Cc: stable@freebsd.org, Glen Barber , Bryan Drewery Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 May 2015 08:45:34 -0000 Hi, That change has not yet been MFC'd. (It's on my list along with other commits.) --Will. On Sat, May 2, 2015 at 12:20 PM, Oliver Pinter wrote: > On 5/2/15, Oliver Pinter wrote: >> On 5/2/15, Oliver Pinter wrote: >>> Hi all! >>> >>> I try to build a modified 10-STABLE with jenkins, but I got the error >>> in the subject. >>> You can find the full build log under this link: >>> http://jenkins.hardenedbsd.org:8180/jenkins/job/HardenedBSD-10-experimental-amd64/1/console >>> >>> If you need more info, please ping me, or write to core@hardenedbsd.org . >>> >>> Thanks you, >>> Oliver >>> >> >> Seems like something LIB32_OBJTREE related changes are missing from >> 10-STABLE's Makefile.inc1... >> > > Especially this commit: > commit ad4df7fd320711af1c0e3bf59b9c7274c0cf95ae > Author: will > Date: Thu Sep 18 01:57:36 2014 +0000 > > Root the lib32 object tree under the overall object tree. > > This enables a common root directory for all object files for a given tree, > which eases sharing a common MAKEOBJDIRPREFIX, and cleaning up of > object trees. > > In particular, one can simply (from the source directory) rm -rf > /usr/obj$(pwd) > to destroy all object files for it. Or to copy/sync files, etc. > > Reviewed by: bdrewery > CR: https://reviews.freebsd.org/D796 > MFC after: 1 month > Sponsored by: Spectra Logic From owner-freebsd-stable@FreeBSD.ORG Sun May 3 11:47:25 2015 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 27C19524 for ; Sun, 3 May 2015 11:47:25 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E79F11474 for ; Sun, 3 May 2015 11:47:24 +0000 (UTC) Received: by igbyr2 with SMTP id yr2so66807168igb.0 for ; Sun, 03 May 2015 04:47:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=SQ22h3Aqk59PboQX9Y3t74d2Wrx6EuMxnXuF8K0/7zI=; b=IiCkT6WE4hqS3CnlZRdxb4hq8RtoPENGvR4L7cMh9YUf5G6ocNwDqNVSw1J0hEgk76 8xrwPVLU79A31X3pQjsa4SPYO4n8w+zl2xmkwBPumRGX5mfujufU4hPWkaF7oThaWv0J TydRn+jghIyHBMKCuOEUcHzVns2RuTy45wyd5TTjtoQ149IYuXkvZMrUUHXSqKGfFAgO dTOzLgJGdzfxvw2AUQCjQgGCts8nmuSe+ibfh93xiExUhQOur8PgbevZXV6lyXJgI2jl +rCAjApGZBcm+OnupLGutaIz1BOEz7vXOemFaubGb+PctwZBiPxbL6me0fBNw8rySR1+ X7Fw== X-Gm-Message-State: ALoCoQn0tOq2lTvVPEeX4lIlPuwKdt6uW4FTkGPk/59zstyOaFdxD0ewg+oQwg06jC9GfdPMdfar MIME-Version: 1.0 X-Received: by 10.43.60.76 with SMTP id wr12mr23247753icb.83.1430653638740; Sun, 03 May 2015 04:47:18 -0700 (PDT) Received: by 10.79.11.6 with HTTP; Sun, 3 May 2015 04:47:18 -0700 (PDT) In-Reply-To: References: Date: Sun, 3 May 2015 13:47:18 +0200 Message-ID: Subject: Re: include/rpcsvc - mkdir: /usr/obj/lib32: Permission denied From: Oliver Pinter To: Will Andrews Cc: stable@freebsd.org, Glen Barber , Bryan Drewery Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 May 2015 11:47:25 -0000 On 5/3/15, Will Andrews wrote: > Hi, > > That change has not yet been MFC'd. (It's on my list along with other > commits.) Yes, I known. But this would be a possible fix to my problem. > > --Will. > > On Sat, May 2, 2015 at 12:20 PM, Oliver Pinter > wrote: >> On 5/2/15, Oliver Pinter wrote: >>> On 5/2/15, Oliver Pinter wrote: >>>> Hi all! >>>> >>>> I try to build a modified 10-STABLE with jenkins, but I got the error >>>> in the subject. >>>> You can find the full build log under this link: >>>> http://jenkins.hardenedbsd.org:8180/jenkins/job/HardenedBSD-10-experimental-amd64/1/console >>>> >>>> If you need more info, please ping me, or write to core@hardenedbsd.org >>>> . >>>> >>>> Thanks you, >>>> Oliver >>>> >>> >>> Seems like something LIB32_OBJTREE related changes are missing from >>> 10-STABLE's Makefile.inc1... >>> >> >> Especially this commit: >> commit ad4df7fd320711af1c0e3bf59b9c7274c0cf95ae >> Author: will >> Date: Thu Sep 18 01:57:36 2014 +0000 >> >> Root the lib32 object tree under the overall object tree. >> >> This enables a common root directory for all object files for a given >> tree, >> which eases sharing a common MAKEOBJDIRPREFIX, and cleaning up of >> object trees. >> >> In particular, one can simply (from the source directory) rm -rf >> /usr/obj$(pwd) >> to destroy all object files for it. Or to copy/sync files, etc. >> >> Reviewed by: bdrewery >> CR: https://reviews.freebsd.org/D796 >> MFC after: 1 month >> Sponsored by: Spectra Logic > From owner-freebsd-stable@FreeBSD.ORG Sun May 3 17:30:46 2015 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 530DFA5F; Sun, 3 May 2015 17:30:46 +0000 (UTC) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 400131364; Sun, 3 May 2015 17:30:46 +0000 (UTC) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id 0D32B142; Sun, 3 May 2015 17:30:45 +0000 (UTC) Date: Sun, 3 May 2015 17:30:45 +0000 (GMT) From: jenkins-admin@freebsd.org To: jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org, freebsd-i386@FreeBSD.org, np@FreeBSD.org, kevlo@FreeBSD.org, mav@FreeBSD.org Message-ID: <419690107.0.1430674245851.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build is back to normal : FreeBSD_STABLE_10-i386 #39 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: FreeBSD_STABLE_10-i386 X-Jenkins-Result: SUCCESS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 May 2015 17:30:46 -0000 See From owner-freebsd-stable@FreeBSD.ORG Tue May 5 07:36:40 2015 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8671B231; Tue, 5 May 2015 07:36:40 +0000 (UTC) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 74DCA17C2; Tue, 5 May 2015 07:36:40 +0000 (UTC) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id CD4E350E; Tue, 5 May 2015 07:36:40 +0000 (UTC) Date: Tue, 5 May 2015 07:36:40 +0000 (GMT) From: jenkins-admin@freebsd.org To: jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org, freebsd-i386@FreeBSD.org, mav@FreeBSD.org Message-ID: <1896938573.9.1430811400285.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build is back to normal : FreeBSD_STABLE_9-i386 #21 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: FreeBSD_STABLE_9-i386 X-Jenkins-Result: SUCCESS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2015 07:36:40 -0000 See From owner-freebsd-stable@FreeBSD.ORG Tue May 5 11:39:39 2015 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE807183 for ; Tue, 5 May 2015 11:39:39 +0000 (UTC) Received: from mail-wg0-x241.google.com (mail-wg0-x241.google.com [IPv6:2a00:1450:400c:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B1C51518 for ; Tue, 5 May 2015 11:39:39 +0000 (UTC) Received: by wggy19 with SMTP id y19so16585262wgg.3 for ; Tue, 05 May 2015 04:39:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:from:subject:to:content-type:mime-version:reply-to :organization:date; bh=l5ShS+XWlf6df3ccKleyZNmpR4OtrebuuZ0a4Wvopmg=; b=kDVoqQ7gsypRBAW6lhC50zsdzWPWogkYDR9YfMBt+MhdcuVAxQPby6nsi6Pgb+4qw6 j8VoyrQjf3qFlMMdTHAIhke3uRQ20a+vgirfF048fVIQcq4w6WfZUqVtJGqJCOq5iY9W vMcrbJ/zqzWPp1uefSw4TK6oQrKRuOoLwkUxwuw/Ergy5fMheqUjp+lr+w0tft34CNGF 2CQdXnxp/oU86s9YdWUgWbxLw1UrzKo8X5DbJAeN6QBI37bmvJwgOZhd09p560KwK9/A 7+OoiZPCT7TuIN/incKnywHI36D00Vul+fRm4oVz+zW3AUKgQq764g5L0v1Un2x8PhiB ejEQ== X-Received: by 10.194.241.194 with SMTP id wk2mr15632924wjc.58.1430825978063; Tue, 05 May 2015 04:39:38 -0700 (PDT) Received: from IGLD-84-229-167-32.inter.net.il (IGLD-84-229-167-32.inter.net.il. [84.229.167.32]) by mx.google.com with ESMTPSA id k2sm15743870wix.4.2015.05.05.04.39.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 05 May 2015 04:39:35 -0700 (PDT) Message-ID: <5548abf7.a24fb40a.0c0d.4832@mx.google.com> From: "Rotem Hecht" Subject: Music composer for film, games and Tv commercials To: "stable" MIME-Version: 1.0 Reply-To: "info@rotemmusic.com" Organization: Rotem Hecht Date: Tue, 5 May 2015 14:39:36 +0300 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2015 11:39:40 -0000 Hi=20 =20 I hope you well =20 I am Rotem Hecht. I work as a professional music composer and sound d= esigner, I'm based in Israel and in Los Angeles, Ca and working with production companies all over the world. I compose original music for movies, television commercials, video gam= es, events, corporate videos and I am open to other media scenarios. My portfolio includes projects performed for Microsoft, Hershey's, Kre= -O, Hasbro, Mercedes, Audi, Nickelodeon, Hop! TV Channel and more =20 I=E2=80=99m capable of delivering high quality products at decent rate= s and in short period of time. I would like to offer you my services. Please check my portfolio on my= website www.rotemmusic.com =20 Thank You for your time. =20 All The Best , =20 Rotem Hecht Mob:+972-528-227726 rotemhecht@gmail.com Skype: rhecht1282=20 =20 Unsubscribe From owner-freebsd-stable@FreeBSD.ORG Wed May 6 09:43:39 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5AAB53F6 for ; Wed, 6 May 2015 09:43:39 +0000 (UTC) Received: from f242.virtualtarget.com.br (f242.virtualtarget.com.br [187.61.34.242]) by mx1.freebsd.org (Postfix) with ESMTP id E939E1CA8 for ; Wed, 6 May 2015 09:43:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=k1; d=dkim.vttrack.com.br; h=From:To:Subject:MIME-Version:Content-Type:Reply-To:List-Unsubscribe:Message-ID:Date; bh=zE6oNEqn1e8P+aMU8nFLR1fbVi0=; b=ehnxzLyjbcJuqkHBgh8lyXFiuuabI6YVje2Fbye9QoQWB8ExlQ0SjEF353R0fLd34lclz42K6KXJ n3Vcf30XUpZfNaWLcxUrRiDEfWkAZIiU4VvkY+pcQ8PL5PozANe+xFfh+AZIhkTY7PLzAOrLaZlS /FiQtIZZ/FPbSIostu8= Received: by f242.virtualtarget.com.br id h97goe0sh4sc for ; Wed, 6 May 2015 06:30:13 -0300 (envelope-from ) From: "Laura Shop" To: "" Subject: Ofertas em dobro , bolsas louis vuitton, chanel, prada, smartphones e muito mais. MIME-Version: 1.0 Reply-To: "Laura Shop" X-Hash: 22034-59-2376270-2e8cdcd0b7e0975904ef60655021654f X-DMA: 25014485 Message-ID: <0.3.9E.758.1D087DF4103DD72.61E7@f242.virtualtarget.com.br> Date: Wed, 6 May 2015 06:40:23 -0300 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2015 09:43:39 -0000 Caso n=E3o esteja visualizando corretamente esta mensagem, clique aqui [file:///C:/Users/Dina/Downloads/laurashop.htm#]. [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,984,2e8cdcd= 0b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLw=3D=3D,1,Zn= JlZWJzZC5vcmc=3D] =A0 =A0 =A0 =A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,985,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1Cb2xzYS1DaG= FuZWwtMi01NS1CYWJ5LS1IaWRyYXRhbnRlLVN0cmF3YmVycmllcy0tLUNoYW1wYWduZS1WaWN0b= 3JpYXMtU2VjcmV0LS1TcGxhc2gtU3RyYXdiZXJyaWVzLS0tQ2hhbXBhZ25lLVZpY3Rvcmlhcy1T= ZWNyZXQtL3Byb2QtMjkxODc4Ny8=3D,1,ZnJlZWJzZC5vcmc=3D] PROMO=C7AO EM DOBRO" BOLSA LOUIS VUITTON N=C9O MONOGRAN+BOLSA LOUIS VUITTON BERKELEY MONOGRAN " GRATIS LINDA CARTEIRA" Por Apenas: R$ 499,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,986,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,987,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0JvbHNhLUxvdW= lzLVZ1aXR0b24tQVJUU1ktRGFtaWVyLUF6dXIvcHJvZC0yOTAzNDA0Lw=3D=3D,1,ZnJlZWJzZC= 5vcmc=3D] =A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,988,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL09GRVJUQS1FTS= 1ET0JSTy0tQm9sc2EtUHJhZGEtQ2xhc3NpYy1TYWZmaWFuby1DYXJ0ZWlyYS1QcmFkYS1TYWZma= WFuby0tLVZBUklBUy1DT1JFUy9wcm9kLTI5MTg2NzIv,1,ZnJlZWJzZC5vcmc=3D] =A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,989,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1MSU5EQS1CT0= xTQS1MT1VJUy1WVUlUVE9OLUFSVFNZLUNBTlZBUy1EQU1JRVItRUJFTkUtQ0FSVEVJUkEtTE9VS= VMtVlVJVFRPTi1EQU1JRVItQVpVUi1ISURSQVRBTlRFLVNUUkFXQkVSUklFUy1CT0RZLVNQTEFT= SC1TVFJBV0JFUlJJRVMtLS0tR1JBVElTLUtJVC1NQVFVSUFHRU0tLS9wcm9kLTMwNTAwNTgvIw= =3D=3D,1,ZnJlZWJzZC5vcmc=3D]=A0 LINDA BOLSA LOUIS VUITTON ARTSY CANVAS DAMIER EBENE+CARTEIRA LOUIS VUITTON DAMIER AZUR+HIDRATANTE STRAWBERRIES+BODY SPLASH STRAWBERRIES. " GRATIS KIT MAQUIAGEM " Por Apenas: R$ 499,00 VER OFERTA [file:///C:/Users/Dina/Downloads/laurashop.htm#] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,990,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0JvbHNhLUNoYW= 5lbC1MZS1Cb3ktVmVybWVsaGEtUGVxdWVuYS9wcm9kLTI5MDQwNDcv,1,ZnJlZWJzZC5vcmc=3D] BOLSA LOUIS VUITTON ARTSY DAMIER AZUR + BOLSA CHANEL 2.55 BABY " GANHE GRATIS HIDRATANTE VICTORIA SECRETS"=A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,991,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1ET0JSQURJTk= hBLURJQS1EQVMtTUFFUy0tQk9MU0EtTE9VSVMtVlVJVFRPTi1ORU8tTU9OT0dSQU4tQk9MU0EtT= E9VSVMtVlVJVFRPTi1CRVJLRUxFWS1NT05PR1JBTi0tLUdSQVRJUy1MSU5EQS1DQVJURUlSQS0v= cHJvZC0zMDU3ODUwLyM=3D,1,ZnJlZWJzZC5vcmc=3D]=A0 Por Apenas: R$ 399,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,986,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,985,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1Cb2xzYS1DaG= FuZWwtMi01NS1CYWJ5LS1IaWRyYXRhbnRlLVN0cmF3YmVycmllcy0tLUNoYW1wYWduZS1WaWN0b= 3JpYXMtU2VjcmV0LS1TcGxhc2gtU3RyYXdiZXJyaWVzLS0tQ2hhbXBhZ25lLVZpY3Rvcmlhcy1T= ZWNyZXQtL3Byb2QtMjkxODc4Ny8=3D,1,ZnJlZWJzZC5vcmc=3D] BOLSA PRADA CLASSIC + CARTEIRA PRADA +HIDRATANTE E BODY SPLASH LOVE SPEEL VICTORIA SECRETS. Por Apenas: R$ 399,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,986,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,992,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0hpZHJhdGFudG= UtTG92ZS1TcGVsbC1WaWN0b3JpYXMtU2VjcmV0LS1TcGxhc2gtTG92ZS1TcGVsbC1WaWN0b3JpY= XMtU2VjcmV0L3Byb2QtMjkxODU1NS8=3D,1,ZnJlZWJzZC5vcmc=3D] LEVE DUAS BOLSAS CHANEL BABY 2.55 NAS CORES PRETA E VERMELHA E GANHE UM LINDO BRINDE DA VICTORIA SECRETS. Por Apenas: R$ 329,00 VER OFERTA [file:///C:/Users/Dina/Downloads/laurashop.htm#] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,993,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0JvbHNhLUNoYW= 5lbC0yLTU1LUJhYnktQnJhbmNhL3Byb2QtMjkwNDEyMS8=3D,1,ZnJlZWJzZC5vcmc=3D] BOLSA LOUIS VUITTON BERKELEY+HIDRATANTE LOVE SPELL+BODY SPLASH PURE DAYDREAM Por Apenas: R$ 399,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2376270,986,2e8cdcd0= b7e0975904ef60655021654f,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] From owner-freebsd-stable@FreeBSD.ORG Wed May 6 09:45:08 2015 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 32F2C51C for ; Wed, 6 May 2015 09:45:08 +0000 (UTC) Received: from f252.virtualtarget.com.br (f252.virtualtarget.com.br [187.61.34.252]) by mx1.freebsd.org (Postfix) with ESMTP id 314411CE0 for ; Wed, 6 May 2015 09:45:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=k1; d=dkim.vttrack.com.br; h=From:To:Subject:MIME-Version:Content-Type:Reply-To:List-Unsubscribe:Message-ID:Date; bh=DKJV9LWuOVwjpAhTsKVd951wPGc=; b=DKmgYzeltg5hDKI3OYq4iQoTkBppJmQN3We4ebgtaiGpt2AxttcwitccMvB8K4g1MbK0NLFP3Uja BOExpsOmo97K0xVmYRiANKdNScTS1+iHL4MviaGCprA3/GQTtH1Azv4kVWm1a0omcpU5Y5qtwVfM 6k0hBYOgspoRJCn7qak= Received: by f252.virtualtarget.com.br id h97ha40sh4sl for ; Wed, 6 May 2015 06:33:27 -0300 (envelope-from ) From: "Laura Shop" To: "" Subject: Ofertas em dobro , bolsas louis vuitton, chanel, prada, smartphones e muito mais. MIME-Version: 1.0 Reply-To: "Laura Shop" X-Hash: 22034-59-2403392-c024ea5d161de9b12c7b57cd467d9667 X-DMA: 25014485 Message-ID: <0.1.1.905.1D087DFB4D62B1A.61DF@f252.virtualtarget.com.br> Date: Wed, 6 May 2015 06:45:06 -0300 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2015 09:45:08 -0000 Caso n=E3o esteja visualizando corretamente esta mensagem, clique aqui [file:///C:/Users/Dina/Downloads/laurashop.htm#]. [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,984,c024ea5= d161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLw=3D=3D,1,Zn= JlZWJzZC5vcmc=3D] =A0 =A0 =A0 =A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,985,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1Cb2xzYS1DaG= FuZWwtMi01NS1CYWJ5LS1IaWRyYXRhbnRlLVN0cmF3YmVycmllcy0tLUNoYW1wYWduZS1WaWN0b= 3JpYXMtU2VjcmV0LS1TcGxhc2gtU3RyYXdiZXJyaWVzLS0tQ2hhbXBhZ25lLVZpY3Rvcmlhcy1T= ZWNyZXQtL3Byb2QtMjkxODc4Ny8=3D,1,ZnJlZWJzZC5vcmc=3D] PROMO=C7AO EM DOBRO" BOLSA LOUIS VUITTON N=C9O MONOGRAN+BOLSA LOUIS VUITTON BERKELEY MONOGRAN " GRATIS LINDA CARTEIRA" Por Apenas: R$ 499,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,986,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,987,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0JvbHNhLUxvdW= lzLVZ1aXR0b24tQVJUU1ktRGFtaWVyLUF6dXIvcHJvZC0yOTAzNDA0Lw=3D=3D,1,ZnJlZWJzZC= 5vcmc=3D] =A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,988,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL09GRVJUQS1FTS= 1ET0JSTy0tQm9sc2EtUHJhZGEtQ2xhc3NpYy1TYWZmaWFuby1DYXJ0ZWlyYS1QcmFkYS1TYWZma= WFuby0tLVZBUklBUy1DT1JFUy9wcm9kLTI5MTg2NzIv,1,ZnJlZWJzZC5vcmc=3D] =A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,989,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1MSU5EQS1CT0= xTQS1MT1VJUy1WVUlUVE9OLUFSVFNZLUNBTlZBUy1EQU1JRVItRUJFTkUtQ0FSVEVJUkEtTE9VS= VMtVlVJVFRPTi1EQU1JRVItQVpVUi1ISURSQVRBTlRFLVNUUkFXQkVSUklFUy1CT0RZLVNQTEFT= SC1TVFJBV0JFUlJJRVMtLS0tR1JBVElTLUtJVC1NQVFVSUFHRU0tLS9wcm9kLTMwNTAwNTgvIw= =3D=3D,1,ZnJlZWJzZC5vcmc=3D]=A0 LINDA BOLSA LOUIS VUITTON ARTSY CANVAS DAMIER EBENE+CARTEIRA LOUIS VUITTON DAMIER AZUR+HIDRATANTE STRAWBERRIES+BODY SPLASH STRAWBERRIES. " GRATIS KIT MAQUIAGEM " Por Apenas: R$ 499,00 VER OFERTA [file:///C:/Users/Dina/Downloads/laurashop.htm#] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,990,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0JvbHNhLUNoYW= 5lbC1MZS1Cb3ktVmVybWVsaGEtUGVxdWVuYS9wcm9kLTI5MDQwNDcv,1,ZnJlZWJzZC5vcmc=3D] BOLSA LOUIS VUITTON ARTSY DAMIER AZUR + BOLSA CHANEL 2.55 BABY " GANHE GRATIS HIDRATANTE VICTORIA SECRETS"=A0 [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,991,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1ET0JSQURJTk= hBLURJQS1EQVMtTUFFUy0tQk9MU0EtTE9VSVMtVlVJVFRPTi1ORU8tTU9OT0dSQU4tQk9MU0EtT= E9VSVMtVlVJVFRPTi1CRVJLRUxFWS1NT05PR1JBTi0tLUdSQVRJUy1MSU5EQS1DQVJURUlSQS0v= cHJvZC0zMDU3ODUwLyM=3D,1,ZnJlZWJzZC5vcmc=3D]=A0 Por Apenas: R$ 399,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,986,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,985,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyLy1Cb2xzYS1DaG= FuZWwtMi01NS1CYWJ5LS1IaWRyYXRhbnRlLVN0cmF3YmVycmllcy0tLUNoYW1wYWduZS1WaWN0b= 3JpYXMtU2VjcmV0LS1TcGxhc2gtU3RyYXdiZXJyaWVzLS0tQ2hhbXBhZ25lLVZpY3Rvcmlhcy1T= ZWNyZXQtL3Byb2QtMjkxODc4Ny8=3D,1,ZnJlZWJzZC5vcmc=3D] BOLSA PRADA CLASSIC + CARTEIRA PRADA +HIDRATANTE E BODY SPLASH LOVE SPEEL VICTORIA SECRETS. Por Apenas: R$ 399,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,986,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,992,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0hpZHJhdGFudG= UtTG92ZS1TcGVsbC1WaWN0b3JpYXMtU2VjcmV0LS1TcGxhc2gtTG92ZS1TcGVsbC1WaWN0b3JpY= XMtU2VjcmV0L3Byb2QtMjkxODU1NS8=3D,1,ZnJlZWJzZC5vcmc=3D] LEVE DUAS BOLSAS CHANEL BABY 2.55 NAS CORES PRETA E VERMELHA E GANHE UM LINDO BRINDE DA VICTORIA SECRETS. Por Apenas: R$ 329,00 VER OFERTA [file:///C:/Users/Dina/Downloads/laurashop.htm#] [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,993,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL0JvbHNhLUNoYW= 5lbC0yLTU1LUJhYnktQnJhbmNhL3Byb2QtMjkwNDEyMS8=3D,1,ZnJlZWJzZC5vcmc=3D] BOLSA LOUIS VUITTON BERKELEY+HIDRATANTE LOVE SPELL+BODY SPLASH PURE DAYDREAM Por Apenas: R$ 399,00 VER OFERTA [http://trk.mdkmail.com.br/index.dma/DmaClick?22034,59,2403392,986,c024ea5d= 161de9b12c7b57cd467d9667,aHR0cDovL3d3dy5sYXVyYXNob3AuY29tLmJyL2ltYWdlbS9pbm= RleC8xMTI4MDMxL00vMTAzLnBuZw=3D=3D,1,ZnJlZWJzZC5vcmc=3D] From owner-freebsd-stable@FreeBSD.ORG Thu May 7 08:07:54 2015 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CD94FFD for ; Thu, 7 May 2015 08:07:54 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CDC8F1B7D for ; Thu, 7 May 2015 08:07:53 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqGqT-000NpY-I8 for freebsd-stable@FreeBSD.org; Thu, 07 May 2015 11:07:49 +0300 Date: Thu, 7 May 2015 11:07:49 +0300 From: Slawa Olhovchenkov To: freebsd-stable@FreeBSD.org Subject: zfs, cam sticking on failed disk Message-ID: <20150507080749.GB1394@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 08:07:54 -0000 I have zpool of 12 vdev (zmirrors). One disk in one vdev out of service and stop serving reuquest: dT: 1.036s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0| ada0 0 0 0 0 0.0 0 0 0.0 0.0| ada1 1 0 0 0 0.0 0 0 0.0 0.0| ada2 0 0 0 0 0.0 0 0 0.0 0.0| ada3 0 0 0 0 0.0 0 0 0.0 0.0| da0 0 0 0 0 0.0 0 0 0.0 0.0| da1 0 0 0 0 0.0 0 0 0.0 0.0| da2 0 0 0 0 0.0 0 0 0.0 0.0| da3 0 0 0 0 0.0 0 0 0.0 0.0| da4 0 0 0 0 0.0 0 0 0.0 0.0| da5 0 0 0 0 0.0 0 0 0.0 0.0| da6 0 0 0 0 0.0 0 0 0.0 0.0| da7 0 0 0 0 0.0 0 0 0.0 0.0| da8 0 0 0 0 0.0 0 0 0.0 0.0| da9 0 0 0 0 0.0 0 0 0.0 0.0| da10 0 0 0 0 0.0 0 0 0.0 0.0| da11 0 0 0 0 0.0 0 0 0.0 0.0| da12 0 0 0 0 0.0 0 0 0.0 0.0| da13 0 0 0 0 0.0 0 0 0.0 0.0| da14 0 0 0 0 0.0 0 0 0.0 0.0| da15 0 0 0 0 0.0 0 0 0.0 0.0| da16 0 0 0 0 0.0 0 0 0.0 0.0| da17 0 0 0 0 0.0 0 0 0.0 0.0| da18 24 0 0 0 0.0 0 0 0.0 0.0| da19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 0 0 0 0 0.0 0 0 0.0 0.0| da20 0 0 0 0 0.0 0 0 0.0 0.0| da21 0 0 0 0 0.0 0 0 0.0 0.0| da22 0 0 0 0 0.0 0 0 0.0 0.0| da23 0 0 0 0 0.0 0 0 0.0 0.0| da24 0 0 0 0 0.0 0 0 0.0 0.0| da25 0 0 0 0 0.0 0 0 0.0 0.0| da26 0 0 0 0 0.0 0 0 0.0 0.0| da27 As result zfs operation on this pool stoped too. `zpool list -v` don't worked. `zpool detach tank da19` don't worked. Application worked with this pool sticking in `zfs` wchan and don't killed. # camcontrol tags da19 -v (pass19:isci0:0:3:0): dev_openings 7 (pass19:isci0:0:3:0): dev_active 25 (pass19:isci0:0:3:0): allocated 25 (pass19:isci0:0:3:0): queued 0 (pass19:isci0:0:3:0): held 0 (pass19:isci0:0:3:0): mintags 2 (pass19:isci0:0:3:0): maxtags 255 How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? From owner-freebsd-stable@FreeBSD.ORG Thu May 7 08:41:57 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F15AFC66 for ; Thu, 7 May 2015 08:41:57 +0000 (UTC) Received: from mail-wg0-f48.google.com (mail-wg0-f48.google.com [74.125.82.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9992F10B4 for ; Thu, 7 May 2015 08:41:57 +0000 (UTC) Received: by wgin8 with SMTP id n8so36160489wgi.0 for ; Thu, 07 May 2015 01:41:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=lGzQRWmMRKk1oLaJTaJKimL53kYiFCvA6br3pw4dvyI=; b=AuGKW2KdvwZI1jQexgZn3rgzrRH3ozVpz0vLZtcQjjkO/G3VLsNTyal50sZIgjGT9m hLklqtfURrLkWsEQ2yl2wS1+4ZXilaES+f3XppThVEPVK1hDhylGi5s49V7l+YkAyyoO APKau1j9TPNtQ3Rwn6PcfaJJmL8q4T4fCc/bjwOCvKae96YtM6PGmrhzv2+fjyuxS5ij CY55Oic6UvzCox9zUchIbIU6dwU2a2Dj5q7JM4rDWL6sQjWCZUFC9vcUZAL41UT01Usa 86C8VyTtDVu8aNBuCukmeOun8c9y6Tfk7Sv/sV7KYHZXUpYm3s/u0xeNg9cDWFwlIkfG b1BQ== X-Gm-Message-State: ALoCoQnr28eq0QXxlJs4XVd40HQcUKFY5sHJ7396xWeVUco9cPlD2UFXTeNp56WNxMBMWs5R3IAS X-Received: by 10.194.48.101 with SMTP id k5mr5128717wjn.79.1430988110491; Thu, 07 May 2015 01:41:50 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id bh7sm2256297wjb.8.2015.05.07.01.41.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 01:41:49 -0700 (PDT) Message-ID: <554B2547.1090307@multiplay.co.uk> Date: Thu, 07 May 2015 09:41:43 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507080749.GB1394@zxy.spb.ru> In-Reply-To: <20150507080749.GB1394@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 08:41:58 -0000 On 07/05/2015 09:07, Slawa Olhovchenkov wrote: > I have zpool of 12 vdev (zmirrors). > One disk in one vdev out of service and stop serving reuquest: > > dT: 1.036s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 0 0 0 0.0 0 0 0.0 0.0| ada0 > 0 0 0 0 0.0 0 0 0.0 0.0| ada1 > 1 0 0 0 0.0 0 0 0.0 0.0| ada2 > 0 0 0 0 0.0 0 0 0.0 0.0| ada3 > 0 0 0 0 0.0 0 0 0.0 0.0| da0 > 0 0 0 0 0.0 0 0 0.0 0.0| da1 > 0 0 0 0 0.0 0 0 0.0 0.0| da2 > 0 0 0 0 0.0 0 0 0.0 0.0| da3 > 0 0 0 0 0.0 0 0 0.0 0.0| da4 > 0 0 0 0 0.0 0 0 0.0 0.0| da5 > 0 0 0 0 0.0 0 0 0.0 0.0| da6 > 0 0 0 0 0.0 0 0 0.0 0.0| da7 > 0 0 0 0 0.0 0 0 0.0 0.0| da8 > 0 0 0 0 0.0 0 0 0.0 0.0| da9 > 0 0 0 0 0.0 0 0 0.0 0.0| da10 > 0 0 0 0 0.0 0 0 0.0 0.0| da11 > 0 0 0 0 0.0 0 0 0.0 0.0| da12 > 0 0 0 0 0.0 0 0 0.0 0.0| da13 > 0 0 0 0 0.0 0 0 0.0 0.0| da14 > 0 0 0 0 0.0 0 0 0.0 0.0| da15 > 0 0 0 0 0.0 0 0 0.0 0.0| da16 > 0 0 0 0 0.0 0 0 0.0 0.0| da17 > 0 0 0 0 0.0 0 0 0.0 0.0| da18 > 24 0 0 0 0.0 0 0 0.0 0.0| da19 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 0 0 0 0 0.0 0 0 0.0 0.0| da20 > 0 0 0 0 0.0 0 0 0.0 0.0| da21 > 0 0 0 0 0.0 0 0 0.0 0.0| da22 > 0 0 0 0 0.0 0 0 0.0 0.0| da23 > 0 0 0 0 0.0 0 0 0.0 0.0| da24 > 0 0 0 0 0.0 0 0 0.0 0.0| da25 > 0 0 0 0 0.0 0 0 0.0 0.0| da26 > 0 0 0 0 0.0 0 0 0.0 0.0| da27 > > As result zfs operation on this pool stoped too. > `zpool list -v` don't worked. > `zpool detach tank da19` don't worked. > Application worked with this pool sticking in `zfs` wchan and don't killed. > > # camcontrol tags da19 -v > (pass19:isci0:0:3:0): dev_openings 7 > (pass19:isci0:0:3:0): dev_active 25 > (pass19:isci0:0:3:0): allocated 25 > (pass19:isci0:0:3:0): queued 0 > (pass19:isci0:0:3:0): held 0 > (pass19:isci0:0:3:0): mintags 2 > (pass19:isci0:0:3:0): maxtags 255 > > How I can cancel this 24 requst? > Why this requests don't timeout (3 hours already)? > How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). > Why ZFS (or geom) don't timeout on request and don't rerouted to da18? > If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. With regards to not timing out this could be a default issue, but having a very quick look that's not obvious in the code as isci_io_request_construct etc do indeed set a timeout when CAM_TIME_INFINITY hasn't been requested. The sysctl hw.isci.debug_level may be able to provide more information, but be aware this can be spammy. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 09:50:53 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 73006255 for ; Thu, 7 May 2015 09:50:53 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2C40A192F for ; Thu, 7 May 2015 09:50:53 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqIS9-000Pc3-4S; Thu, 07 May 2015 12:50:49 +0300 Date: Thu, 7 May 2015 12:50:49 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507095048.GC1394@zxy.spb.ru> References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554B2547.1090307@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 09:50:53 -0000 On Thu, May 07, 2015 at 09:41:43AM +0100, Steven Hartland wrote: > On 07/05/2015 09:07, Slawa Olhovchenkov wrote: > > I have zpool of 12 vdev (zmirrors). > > One disk in one vdev out of service and stop serving reuquest: > > > > dT: 1.036s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > > 0 0 0 0 0.0 0 0 0.0 0.0| ada0 > > 0 0 0 0 0.0 0 0 0.0 0.0| ada1 > > 1 0 0 0 0.0 0 0 0.0 0.0| ada2 > > 0 0 0 0 0.0 0 0 0.0 0.0| ada3 > > 0 0 0 0 0.0 0 0 0.0 0.0| da0 > > 0 0 0 0 0.0 0 0 0.0 0.0| da1 > > 0 0 0 0 0.0 0 0 0.0 0.0| da2 > > 0 0 0 0 0.0 0 0 0.0 0.0| da3 > > 0 0 0 0 0.0 0 0 0.0 0.0| da4 > > 0 0 0 0 0.0 0 0 0.0 0.0| da5 > > 0 0 0 0 0.0 0 0 0.0 0.0| da6 > > 0 0 0 0 0.0 0 0 0.0 0.0| da7 > > 0 0 0 0 0.0 0 0 0.0 0.0| da8 > > 0 0 0 0 0.0 0 0 0.0 0.0| da9 > > 0 0 0 0 0.0 0 0 0.0 0.0| da10 > > 0 0 0 0 0.0 0 0 0.0 0.0| da11 > > 0 0 0 0 0.0 0 0 0.0 0.0| da12 > > 0 0 0 0 0.0 0 0 0.0 0.0| da13 > > 0 0 0 0 0.0 0 0 0.0 0.0| da14 > > 0 0 0 0 0.0 0 0 0.0 0.0| da15 > > 0 0 0 0 0.0 0 0 0.0 0.0| da16 > > 0 0 0 0 0.0 0 0 0.0 0.0| da17 > > 0 0 0 0 0.0 0 0 0.0 0.0| da18 > > 24 0 0 0 0.0 0 0 0.0 0.0| da19 > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > 0 0 0 0 0.0 0 0 0.0 0.0| da20 > > 0 0 0 0 0.0 0 0 0.0 0.0| da21 > > 0 0 0 0 0.0 0 0 0.0 0.0| da22 > > 0 0 0 0 0.0 0 0 0.0 0.0| da23 > > 0 0 0 0 0.0 0 0 0.0 0.0| da24 > > 0 0 0 0 0.0 0 0 0.0 0.0| da25 > > 0 0 0 0 0.0 0 0 0.0 0.0| da26 > > 0 0 0 0 0.0 0 0 0.0 0.0| da27 > > > > As result zfs operation on this pool stoped too. > > `zpool list -v` don't worked. > > `zpool detach tank da19` don't worked. > > Application worked with this pool sticking in `zfs` wchan and don't killed. > > > > # camcontrol tags da19 -v > > (pass19:isci0:0:3:0): dev_openings 7 > > (pass19:isci0:0:3:0): dev_active 25 > > (pass19:isci0:0:3:0): allocated 25 > > (pass19:isci0:0:3:0): queued 0 > > (pass19:isci0:0:3:0): held 0 > > (pass19:isci0:0:3:0): mintags 2 > > (pass19:isci0:0:3:0): maxtags 255 > > > > How I can cancel this 24 requst? > > Why this requests don't timeout (3 hours already)? > > How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). > > Why ZFS (or geom) don't timeout on request and don't rerouted to da18? > > > If they are in mirrors, in theory you can just pull the disk, isci will > report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. > With regards to not timing out this could be a default issue, but having I am understand, no universal acceptable timeout for all cases: good disk, good saturated disk, tape, tape library, failed disk, etc. In my case -- failed disk. This model already failed (other specimen) with same symptoms). May be exist some tricks for cancel/aborting all request in queue and removing disk from system? > a very quick look that's not obvious in the code as > isci_io_request_construct etc do indeed set a timeout when > CAM_TIME_INFINITY hasn't been requested. > > The sysctl hw.isci.debug_level may be able to provide more information, > but be aware this can be spammy. I am already have this situation, what command interesting after setting hw.isci.debug_level? From owner-freebsd-stable@FreeBSD.ORG Thu May 7 10:39:00 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 325837DE for ; Thu, 7 May 2015 10:39:00 +0000 (UTC) Received: from mail-wg0-f47.google.com (mail-wg0-f47.google.com [74.125.82.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBB451FE3 for ; Thu, 7 May 2015 10:38:59 +0000 (UTC) Received: by wgiu9 with SMTP id u9so39046558wgi.3 for ; Thu, 07 May 2015 03:38:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=CU4oywXV9SydiVNGIORVUd/Gdgr5ra2ts28i5BeugRM=; b=F4qyEzK/q/pDLGykXcSUGsgvPJcsjvbLEG1h5xve4opnmpVOy5iL8OFF6JHo5VfhrV fzPA7ooi//HrNBl48H2pxlKTkP/eCqAhBahSwCEXOw4QJ190Sh/1SzLT5/YvCcGRC3lo uM/PJU7zBMrofwchPwP9JRnJrMNAw9gZorDwy65jipDvwVkUoTLfTUdYI/dDnz2UL5Cq tJg4fAUSwinDBlnKlnOivlIZy5H7SYbDB2z4xmSMxZim5UmHzQLrZLE/FnQRiUppESBL UwuoyrKghFFIV0K4ifZW7PxXubJfbc2c3uRGSjvYBKOqAmnDkJoWjVoQzPjE8y/VaCeI P8sg== X-Gm-Message-State: ALoCoQmzuOgXuYWGYbSjqyZ7QSMXnEBHGLq6F7VGGy4QU5k2utYDF2szAamAS4rcDIcj53ZgAH+D X-Received: by 10.180.99.39 with SMTP id en7mr5392040wib.31.1430995132493; Thu, 07 May 2015 03:38:52 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id mc20sm3265848wic.15.2015.05.07.03.38.51 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 03:38:51 -0700 (PDT) Message-ID: <554B40B6.6060902@multiplay.co.uk> Date: Thu, 07 May 2015 11:38:46 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> In-Reply-To: <20150507095048.GC1394@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 10:39:00 -0000 On 07/05/2015 10:50, Slawa Olhovchenkov wrote: > On Thu, May 07, 2015 at 09:41:43AM +0100, Steven Hartland wrote: > >> On 07/05/2015 09:07, Slawa Olhovchenkov wrote: >>> I have zpool of 12 vdev (zmirrors). >>> One disk in one vdev out of service and stop serving reuquest: >>> >>> dT: 1.036s w: 1.000s >>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name >>> 0 0 0 0 0.0 0 0 0.0 0.0| ada0 >>> 0 0 0 0 0.0 0 0 0.0 0.0| ada1 >>> 1 0 0 0 0.0 0 0 0.0 0.0| ada2 >>> 0 0 0 0 0.0 0 0 0.0 0.0| ada3 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da0 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da1 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da2 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da3 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da4 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da5 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da6 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da7 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da8 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da9 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da10 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da11 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da12 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da13 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da14 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da15 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da16 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da17 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da18 >>> 24 0 0 0 0.0 0 0 0.0 0.0| da19 >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> 0 0 0 0 0.0 0 0 0.0 0.0| da20 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da21 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da22 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da23 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da24 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da25 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da26 >>> 0 0 0 0 0.0 0 0 0.0 0.0| da27 >>> >>> As result zfs operation on this pool stoped too. >>> `zpool list -v` don't worked. >>> `zpool detach tank da19` don't worked. >>> Application worked with this pool sticking in `zfs` wchan and don't killed. >>> >>> # camcontrol tags da19 -v >>> (pass19:isci0:0:3:0): dev_openings 7 >>> (pass19:isci0:0:3:0): dev_active 25 >>> (pass19:isci0:0:3:0): allocated 25 >>> (pass19:isci0:0:3:0): queued 0 >>> (pass19:isci0:0:3:0): held 0 >>> (pass19:isci0:0:3:0): mintags 2 >>> (pass19:isci0:0:3:0): maxtags 255 >>> >>> How I can cancel this 24 requst? >>> Why this requests don't timeout (3 hours already)? >>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). >>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? >>> >> If they are in mirrors, in theory you can just pull the disk, isci will >> report to cam and cam will report to ZFS which should all recover. > Yes, zmirror with da18. > I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. > >> With regards to not timing out this could be a default issue, but having > I am understand, no universal acceptable timeout for all cases: good > disk, good saturated disk, tape, tape library, failed disk, etc. > In my case -- failed disk. This model already failed (other specimen) > with same symptoms). > > May be exist some tricks for cancel/aborting all request in queue and > removing disk from system? Unlikely tbh, pulling the disk however should. > >> a very quick look that's not obvious in the code as >> isci_io_request_construct etc do indeed set a timeout when >> CAM_TIME_INFINITY hasn't been requested. >> >> The sysctl hw.isci.debug_level may be able to provide more information, >> but be aware this can be spammy. > I am already have this situation, what command interesting after > setting hw.isci.debug_level? I'm afraid I'm not familiar isci I'm afraid possibly someone else who is can chime in. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 10:46:57 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8AA1DB5A for ; Thu, 7 May 2015 10:46:57 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 45CF11162 for ; Thu, 7 May 2015 10:46:57 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqJKR-0000ZF-AI; Thu, 07 May 2015 13:46:55 +0300 Date: Thu, 7 May 2015 13:46:55 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507104655.GT62239@zxy.spb.ru> References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554B40B6.6060902@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 10:46:57 -0000 On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: > >>> How I can cancel this 24 requst? > >>> Why this requests don't timeout (3 hours already)? > >>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). > >>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? > >>> > >> If they are in mirrors, in theory you can just pull the disk, isci will > >> report to cam and cam will report to ZFS which should all recover. > > Yes, zmirror with da18. > > I am surprise that ZFS don't use da18. All zpool fully stuck. > A single low level request can only be handled by one device, if that > device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:00:55 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3A8595E2 for ; Thu, 7 May 2015 12:00:55 +0000 (UTC) Received: from mail-wg0-f46.google.com (mail-wg0-f46.google.com [74.125.82.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB6881ABB for ; Thu, 7 May 2015 12:00:54 +0000 (UTC) Received: by wgyo15 with SMTP id o15so41115230wgy.2 for ; Thu, 07 May 2015 05:00:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=3Uat6WaykJOkeVTViHmEpqK4BMnR60hQLI81lI9cq+8=; b=RjsZbEGmftSy3vsuSO/HeKpLFP8V/hvBGaGC9G1vnQBPJXvH3G3DQyEn7jT2DRPLmj gbJTPvGMpXTKg40cMCs2WLQ4ViZfbCaAcuhfXJ38wmohwJTxwTQ8zYdPWMwxNVbQfBVm 0Y3vaAP8itYGyrFVdYpgubQCW9EnWdR3I0UbpEjXritlr3wTY9Ey3rV7/gsWkY4tYd3/ 2WrPeTd34iM4mm8qpvWgm7hjHzL2Ddrp1HJ5auOTH565PEYSMq+1qXmgMGWJZ+X/GGFI rm7wWJCO/BCTFpSOFXpXJGgvUWbh3sYEtirw3HbNDTaxibIPAoRhPQuQRVoF5g4FIjsx +pXA== X-Gm-Message-State: ALoCoQnjBR2sDiRrqTqJCdlDVMJrJ9zenoYVT36xX11bS7uj6qjCRY8Ins8hb0hh5rdNkqOJ8oja X-Received: by 10.194.59.79 with SMTP id x15mr6608397wjq.81.1431000047011; Thu, 07 May 2015 05:00:47 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id l3sm3619817wik.16.2015.05.07.05.00.45 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 05:00:46 -0700 (PDT) Message-ID: <554B53E8.4000508@multiplay.co.uk> Date: Thu, 07 May 2015 13:00:40 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> In-Reply-To: <20150507104655.GT62239@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:00:55 -0000 On 07/05/2015 11:46, Slawa Olhovchenkov wrote: > On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: > >>>>> How I can cancel this 24 requst? >>>>> Why this requests don't timeout (3 hours already)? >>>>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). >>>>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? >>>>> >>>> If they are in mirrors, in theory you can just pull the disk, isci will >>>> report to cam and cam will report to ZFS which should all recover. >>> Yes, zmirror with da18. >>> I am surprise that ZFS don't use da18. All zpool fully stuck. >> A single low level request can only be handled by one device, if that >> device returns an error then ZFS will use the other device, but not until. > Why next requests don't routed to da18? > Current request stuck on da19 (unlikely, but understund), but why > stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. You didn't say which FreeBSD version you where running? Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:05:16 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49AA2746 for ; Thu, 7 May 2015 12:05:16 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 025581BBE for ; Thu, 7 May 2015 12:05:16 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqKY9-00020s-1m; Thu, 07 May 2015 15:05:09 +0300 Date: Thu, 7 May 2015 15:05:08 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507120508.GX62239@zxy.spb.ru> References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554B53E8.4000508@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:05:16 -0000 On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: > > > On 07/05/2015 11:46, Slawa Olhovchenkov wrote: > > On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: > > > >>>>> How I can cancel this 24 requst? > >>>>> Why this requests don't timeout (3 hours already)? > >>>>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). > >>>>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? > >>>>> > >>>> If they are in mirrors, in theory you can just pull the disk, isci will > >>>> report to cam and cam will report to ZFS which should all recover. > >>> Yes, zmirror with da18. > >>> I am surprise that ZFS don't use da18. All zpool fully stuck. > >> A single low level request can only be handled by one device, if that > >> device returns an error then ZFS will use the other device, but not until. > > Why next requests don't routed to da18? > > Current request stuck on da19 (unlikely, but understund), but why > > stuck all pool? > > Its still waiting for the request from the failed device to complete. As > far as ZFS currently knows there is nothing wrong with the device as its > had no failures. Can you explain some more? One requst waiting, understand. I am do next request. Some information need from vdev with failed disk. Failed disk more busy (queue long), why don't routed to mirror disk? Or, for metadata, to less busy vdev? > You didn't say which FreeBSD version you where running? 10-STABLE, r281264. From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:08:27 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 462439AA for ; Thu, 7 May 2015 12:08:27 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E32321C08 for ; Thu, 7 May 2015 12:08:26 +0000 (UTC) Received: by widdi4 with SMTP id di4so239491699wid.0 for ; Thu, 07 May 2015 05:08:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=300.nl; s=tip; h=from:content-type:subject:message-id:date:to:mime-version; bh=U9zD4vukbw9yo7ye3zuT+krlXTldcpzsIFL04QhM1NE=; b=hb/9H7caAv9dPr7Z7jhN+UVDpWa1Dc3W4PHFUU/gkMcMUdsp16NYbo52aZwsUuOuLN 0gYl+W5eGgjiz+iDaVdhY2Kqbe9v/cOVC3Qwg5WT5tVVpy3oaool5Ld5PXsvKt7E/Opi PzNu2i1kswNlp1i6ncuVM8IDkNASzW/+HxqiM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:subject:message-id:date:to :mime-version; bh=U9zD4vukbw9yo7ye3zuT+krlXTldcpzsIFL04QhM1NE=; b=JX0ekGh8eF/gyFCGKWUqliczlSWwjrNHVs0vnf2B5+Vz6rszWvBmSWxUgZoF6aXyju 2gUM7XzmEO0qkzwYreHwzNQD97dJ3GZMOqrYCUdCxYWCGrX0T8s7U23O+B97GV+eLVZx qsgwZDEVAUZxEy9Kua5cS5uhND8VVTojdlYNDXoJt8pfbwORv/o8itDkg43jG+Ka0LKU ZTnunCmAmnlabre2RTXVQQPz9Su3FvfsSEPm0O/1N1XIQzde12o2p5mQNZqXLMpZ7H+9 9GDee3mX7kldx4btS5np00dMNKK4loDkZL5DfNR7EDZ1NaPv1JSGCKgiFJVl8qJn+ST0 NeOg== X-Gm-Message-State: ALoCoQkppUMtGCpOEQlcpAkfHZzy19iFBqPSwihiU0sQBJ8yzDfp5dZSatpxVdXw/LZf+QfNuihR X-Received: by 10.180.84.97 with SMTP id x1mr6000705wiy.1.1431000505220; Thu, 07 May 2015 05:08:25 -0700 (PDT) Received: from [10.2.7.215] (kantoor.transip.nl. [80.69.69.100]) by mx.google.com with ESMTPSA id e2sm3680457wij.5.2015.05.07.05.08.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 07 May 2015 05:08:23 -0700 (PDT) From: Johan Schuijt-Li Subject: panic: pmap active 0xfffff8001b7154b8 Message-Id: Date: Thu, 7 May 2015 14:08:21 +0200 To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) X-Mailer: Apple Mail (2.2098) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:08:27 -0000 Hi, We=E2=80=99ve been seeing (seemingly) random reboots on 10.1-RELEASE = virtual machines (KVM virtualisation) on our production servers. In an = attempt to determine what was causing this we=E2=80=99ve switched to = running a kernel with INVARIANTS enabled. This resulted for us in the = following panic: Unread portion of the kernel message buffer: panic: pmap active 0xfffff8001b7154b8 cpuid =3D 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe03dd1493a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe03dd149450 vpanic() at vpanic+0x126/frame 0xfffffe03dd149490 kassert_panic() at kassert_panic+0x139/frame 0xfffffe03dd149500 pmap_remove_pages() at pmap_remove_pages+0x8c/frame 0xfffffe03dd1495f0 exec_new_vmspace() at exec_new_vmspace+0x16a/frame 0xfffffe03dd149650 exec_elf64_imgact() at exec_elf64_imgact+0x658/frame 0xfffffe03dd149720 kern_execve() at kern_execve+0x5e4/frame 0xfffffe03dd149a80 sys_execve() at sys_execve+0x37/frame 0xfffffe03dd149ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe03dd149bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03dd149bf0 --- syscall (59, FreeBSD ELF64, sys_execve), rip =3D 0x80158af1a, rsp =3D = 0x7fffffffac38, rbp =3D 0x7fffffffad40 --- I=E2=80=99ve only come across one other report here (without result = unfortunate): = https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html = Are other people aware of this issue or working on this? I can provide access to a VM with a kernel dump and the kernel build for = extra information if needed. - Johan= From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:35:21 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 893E990 for ; Thu, 7 May 2015 12:35:21 +0000 (UTC) Received: from mail-wg0-f51.google.com (mail-wg0-f51.google.com [74.125.82.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 24A841F3B for ; Thu, 7 May 2015 12:35:20 +0000 (UTC) Received: by wgyo15 with SMTP id o15so42045717wgy.2 for ; Thu, 07 May 2015 05:35:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=PracMYD1kIAZrzphs+rQZM5gh8XL/8mh/x8EDIY+9pc=; b=MPrznV6OpfdqV/2iob3Q5hlOX/Xv0L5ZQdrL7ALFQtdhMSrw7naQxmMsd7VGX7xS8K THQczgqJQimPoB3YX7EZjHhxNJXirXX6wot/zR1+1ub+NHH52Iz5hn/bBXgLdTf4yOok H5DdJ9iCquuzhftVq455z5OjFxkDFzlSxh1UNQVgFxMAwSdZTlLNt0RJbqGTvb1xK/w5 nAM7DG2qUH2nve+Q7rG4e8E2MaCalJhcW6Yr4zCjm7VdPwo3KGKmhsXPTMTWBrG3B26x nzIqd6Bvl0dbc7130wbORo1vlOMO4Qco3+PQVjE8YbzntdAxV4HV0uwoAdGtHp1xfFef 4Abg== X-Gm-Message-State: ALoCoQn1dUbhW4yFiMYtvb1G1LYdFOVft5CNusqaOVOGCqZ9aFkc6XU+pRJrB+AweSz78dModC6i X-Received: by 10.180.102.228 with SMTP id fr4mr6264603wib.4.1431002112970; Thu, 07 May 2015 05:35:12 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id dj7sm3260253wjb.3.2015.05.07.05.35.11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 05:35:11 -0700 (PDT) Message-ID: <554B5BF9.8020709@multiplay.co.uk> Date: Thu, 07 May 2015 13:35:05 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> In-Reply-To: <20150507120508.GX62239@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:35:21 -0000 On 07/05/2015 13:05, Slawa Olhovchenkov wrote: > On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: > >> >> On 07/05/2015 11:46, Slawa Olhovchenkov wrote: >>> On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: >>> >>>>>>> How I can cancel this 24 requst? >>>>>>> Why this requests don't timeout (3 hours already)? >>>>>>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). >>>>>>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? >>>>>>> >>>>>> If they are in mirrors, in theory you can just pull the disk, isci will >>>>>> report to cam and cam will report to ZFS which should all recover. >>>>> Yes, zmirror with da18. >>>>> I am surprise that ZFS don't use da18. All zpool fully stuck. >>>> A single low level request can only be handled by one device, if that >>>> device returns an error then ZFS will use the other device, but not until. >>> Why next requests don't routed to da18? >>> Current request stuck on da19 (unlikely, but understund), but why >>> stuck all pool? >> Its still waiting for the request from the failed device to complete. As >> far as ZFS currently knows there is nothing wrong with the device as its >> had no failures. > Can you explain some more? > One requst waiting, understand. > I am do next request. Some information need from vdev with failed > disk. Failed disk more busy (queue long), why don't routed to mirror > disk? Or, for metadata, to less busy vdev? As no error has been reported to ZFS, due to the stalled IO, there is no failed vdev. Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:44:18 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D8498649 for ; Thu, 7 May 2015 12:44:18 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B64410AC for ; Thu, 7 May 2015 12:44:18 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqLA0-0002fu-GA; Thu, 07 May 2015 15:44:16 +0300 Date: Thu, 7 May 2015 15:44:16 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507124416.GD1394@zxy.spb.ru> References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554B5BF9.8020709@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:44:18 -0000 On Thu, May 07, 2015 at 01:35:05PM +0100, Steven Hartland wrote: > > > On 07/05/2015 13:05, Slawa Olhovchenkov wrote: > > On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: > > > >> > >> On 07/05/2015 11:46, Slawa Olhovchenkov wrote: > >>> On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: > >>> > >>>>>>> How I can cancel this 24 requst? > >>>>>>> Why this requests don't timeout (3 hours already)? > >>>>>>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). > >>>>>>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? > >>>>>>> > >>>>>> If they are in mirrors, in theory you can just pull the disk, isci will > >>>>>> report to cam and cam will report to ZFS which should all recover. > >>>>> Yes, zmirror with da18. > >>>>> I am surprise that ZFS don't use da18. All zpool fully stuck. > >>>> A single low level request can only be handled by one device, if that > >>>> device returns an error then ZFS will use the other device, but not until. > >>> Why next requests don't routed to da18? > >>> Current request stuck on da19 (unlikely, but understund), but why > >>> stuck all pool? > >> Its still waiting for the request from the failed device to complete. As > >> far as ZFS currently knows there is nothing wrong with the device as its > >> had no failures. > > Can you explain some more? > > One requst waiting, understand. > > I am do next request. Some information need from vdev with failed > > disk. Failed disk more busy (queue long), why don't routed to mirror > > disk? Or, for metadata, to less busy vdev? > As no error has been reported to ZFS, due to the stalled IO, there is no > failed vdev. I see that device isn't failed (for both OS and ZFS). I am don't talk 'failed vdev'. I am talk 'busy vdev' or 'busy device'. > Yes in theory new requests should go to the other vdev, but there could > be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:46:55 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 883D7761 for ; Thu, 7 May 2015 12:46:55 +0000 (UTC) Received: from mail-wg0-f41.google.com (mail-wg0-f41.google.com [74.125.82.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 228CE10D4 for ; Thu, 7 May 2015 12:46:54 +0000 (UTC) Received: by wgiu9 with SMTP id u9so42311868wgi.3 for ; Thu, 07 May 2015 05:46:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=dIj1Aeo83O/tKItLoSWvA1xDdH2IxndCt8duwG78YdA=; b=VoTBSrXORdF2PuiiutUVTaSvw3d9FGCpCLLBpuk+/5DV55M7oVUrze2aLDSoPt/bGG ADucwwFW4Osq+BFgaq1hAhKDFto1tBaAhViKWrFcrjwjSVlBMtmQroqBtWOXHCxfv51d RgVWlsvwoW7wOAdexox7tld9M/OuGRHWVfmNt6nWwoy/w6S2+0zbZ+AacdMvg46//NHG f94aequXqzwRjL55BGxendlYA/AZBeHulPg3PhVWVp5zKA7wO5Rox0eHsPCbzKi0x4G2 r6vDAdkOBOnlcGYhxaoW0XY9Op85LLHsafwwkg1DmFf74yiAxDJrm9GoSYc3jTkh14Gj X7NA== X-Gm-Message-State: ALoCoQm/ArfrpVuGFXQ0gBoMQn2r413x46CE5NoruOIe02TZWwmW1xH3lWoRpou8JWIbNU3snEnn X-Received: by 10.180.84.65 with SMTP id w1mr6369668wiy.20.1431002807473; Thu, 07 May 2015 05:46:47 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id c5sm3262025wjf.40.2015.05.07.05.46.46 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 05:46:46 -0700 (PDT) Message-ID: <554B5EB0.1080208@multiplay.co.uk> Date: Thu, 07 May 2015 13:46:40 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> In-Reply-To: <20150507124416.GD1394@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:46:55 -0000 On 07/05/2015 13:44, Slawa Olhovchenkov wrote: > On Thu, May 07, 2015 at 01:35:05PM +0100, Steven Hartland wrote: > >> >> On 07/05/2015 13:05, Slawa Olhovchenkov wrote: >>> On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: >>> >>>> On 07/05/2015 11:46, Slawa Olhovchenkov wrote: >>>>> On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: >>>>> >>>>>>>>> How I can cancel this 24 requst? >>>>>>>>> Why this requests don't timeout (3 hours already)? >>>>>>>>> How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). >>>>>>>>> Why ZFS (or geom) don't timeout on request and don't rerouted to da18? >>>>>>>>> >>>>>>>> If they are in mirrors, in theory you can just pull the disk, isci will >>>>>>>> report to cam and cam will report to ZFS which should all recover. >>>>>>> Yes, zmirror with da18. >>>>>>> I am surprise that ZFS don't use da18. All zpool fully stuck. >>>>>> A single low level request can only be handled by one device, if that >>>>>> device returns an error then ZFS will use the other device, but not until. >>>>> Why next requests don't routed to da18? >>>>> Current request stuck on da19 (unlikely, but understund), but why >>>>> stuck all pool? >>>> Its still waiting for the request from the failed device to complete. As >>>> far as ZFS currently knows there is nothing wrong with the device as its >>>> had no failures. >>> Can you explain some more? >>> One requst waiting, understand. >>> I am do next request. Some information need from vdev with failed >>> disk. Failed disk more busy (queue long), why don't routed to mirror >>> disk? Or, for metadata, to less busy vdev? >> As no error has been reported to ZFS, due to the stalled IO, there is no >> failed vdev. > I see that device isn't failed (for both OS and ZFS). > I am don't talk 'failed vdev'. I am talk 'busy vdev' or 'busy device'. > >> Yes in theory new requests should go to the other vdev, but there could >> be some dependency issues preventing that such as a syncing TXG. > Currenly this pool must not have write activity (from application). > What about go to the other (mirror) device in the same vdev? > Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. From owner-freebsd-stable@FreeBSD.ORG Thu May 7 12:51:32 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 579958F8 for ; Thu, 7 May 2015 12:51:32 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0FBC511A9 for ; Thu, 7 May 2015 12:51:32 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqLH0-0002nZ-1P; Thu, 07 May 2015 15:51:30 +0300 Date: Thu, 7 May 2015 15:51:29 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507125129.GY62239@zxy.spb.ru> References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554B5EB0.1080208@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 12:51:32 -0000 On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: > >> Yes in theory new requests should go to the other vdev, but there could > >> be some dependency issues preventing that such as a syncing TXG. > > Currenly this pool must not have write activity (from application). > > What about go to the other (mirror) device in the same vdev? > > Same dependency? > Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? From owner-freebsd-stable@FreeBSD.ORG Thu May 7 13:05:27 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C3043C25 for ; Thu, 7 May 2015 13:05:27 +0000 (UTC) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5792612EF for ; Thu, 7 May 2015 13:05:27 +0000 (UTC) Received: by widdi4 with SMTP id di4so241484915wid.0 for ; Thu, 07 May 2015 06:05:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ELoJuW8XFOZ9RMvUwe3JvLoX78GIASpiDxVl51T8Src=; b=MBx5PQxp4CW/XcyYjK16DmTk2bFWH+kFXA8DkbtbmmMk2TYJocASt0q3Q8NL0RQOA6 iq8r4tVv5EEXNNKRSPThtJAUSPemucbWa2BkP12UOFosVtdMQvcz+mXc+RhMiauawMp+ dNrGrOtTw//9GdcphqHDp1Laioo70oBORCkYKTqHdDHWqSVT0S5EBQ8nei2QylD5Perk 84xxe6XZlLLpQZM16sd4oGAhhW8srOxoeDoLqmyD0opuZhItSBy5BQ/ADVQ4Y7aHQJ1J 7ARmsB3VGvJPTFGnIn651qcAgtLIHKciokLOLc/X3c2aZ21Up+5t7Mc/oGbOxxlJ/prF QWGA== X-Gm-Message-State: ALoCoQkEAj0Bj5zODUbdtQopjSoCHjmxgtxp6vDfwXQnA0Zdn3Hh0i+KVZeqkJhwROKxyFY2I64k X-Received: by 10.194.133.133 with SMTP id pc5mr7150936wjb.31.1431003919788; Thu, 07 May 2015 06:05:19 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id di9sm7350607wib.16.2015.05.07.06.05.17 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 06:05:17 -0700 (PDT) Message-ID: <554B6307.9020309@multiplay.co.uk> Date: Thu, 07 May 2015 14:05:11 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> In-Reply-To: <20150507125129.GY62239@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 13:05:27 -0000 On 07/05/2015 13:51, Slawa Olhovchenkov wrote: > On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: > >>>> Yes in theory new requests should go to the other vdev, but there could >>>> be some dependency issues preventing that such as a syncing TXG. >>> Currenly this pool must not have write activity (from application). >>> What about go to the other (mirror) device in the same vdev? >>> Same dependency? >> Yes, if there's an outstanding TXG, then I believe all IO will stall. > Where this TXG released? When all devices in all vdevs report > 'completed'? When at the least one device in all vdevs report > 'completed'? When at the least one device in at least one vdev report > 'completed'? When all devices have report completed or failed. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 13:10:11 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 279FFE6F for ; Thu, 7 May 2015 13:10:11 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D21141333 for ; Thu, 7 May 2015 13:10:10 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqLZ1-00036e-Vz; Thu, 07 May 2015 16:10:08 +0300 Date: Thu, 7 May 2015 16:10:07 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507131007.GZ62239@zxy.spb.ru> References: <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554B6307.9020309@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 13:10:11 -0000 On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: > > > On 07/05/2015 13:51, Slawa Olhovchenkov wrote: > > On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: > > > >>>> Yes in theory new requests should go to the other vdev, but there could > >>>> be some dependency issues preventing that such as a syncing TXG. > >>> Currenly this pool must not have write activity (from application). > >>> What about go to the other (mirror) device in the same vdev? > >>> Same dependency? > >> Yes, if there's an outstanding TXG, then I believe all IO will stall. > > Where this TXG released? When all devices in all vdevs report > > 'completed'? When at the least one device in all vdevs report > > 'completed'? When at the least one device in at least one vdev report > > 'completed'? > When all devices have report completed or failed. Thanks for explained. > Hence if you pull the disk things should continue as normal, with the > failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. From owner-freebsd-stable@FreeBSD.ORG Thu May 7 13:24:13 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE82B15D for ; Thu, 7 May 2015 13:24:13 +0000 (UTC) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 761331578 for ; Thu, 7 May 2015 13:24:13 +0000 (UTC) Received: by widdi4 with SMTP id di4so242192701wid.0 for ; Thu, 07 May 2015 06:24:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ju4qAZD1HRMK/5INikufCEUrhlgQfHbsVjU0ZELH+wg=; b=ITVKinpS4p7ycAFVdcoHpoTITY/8HgifyFd1kBdHApQ8Ok8y80MVFz9wIOz2K6uE3X IHmZMPTW4wB+ETgSdF/60yTtUYdT8G1TA2svz6Fsa/O2WHSrwWr+BKOxFwJwk1RG9fWr JYTPg//dMhor+CpRcNj3c6cQZixN1plx8npaXvyz0eVZj3IPviW4qsZut482Y3UR/+Cc +kGzEC5e+kWHKY1v+8A0H4+mkO49H2gipU3+DWLorG5uab/nTDB572zXG8Ix4uuImhbA nZTYqxVlHhzV1t9wwETi008s01nNe9GfJPPXzd8FikjINkA2CWzDwufPVkXzxQHdf8FU qtxA== X-Gm-Message-State: ALoCoQmsjQK1IH9qG30MlazXZc/C8FOBOBg7pJRmgCb+gq87hKxIPSIA5Dgytv3zhk6HFRpQ3dVd X-Received: by 10.194.223.66 with SMTP id qs2mr7777283wjc.6.1431005045176; Thu, 07 May 2015 06:24:05 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id l3sm7431517wiv.18.2015.05.07.06.24.04 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 06:24:04 -0700 (PDT) Message-ID: <554B676E.8020500@multiplay.co.uk> Date: Thu, 07 May 2015 14:23:58 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> <20150507131007.GZ62239@zxy.spb.ru> In-Reply-To: <20150507131007.GZ62239@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 13:24:14 -0000 On 07/05/2015 14:10, Slawa Olhovchenkov wrote: > On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: > >> >> On 07/05/2015 13:51, Slawa Olhovchenkov wrote: >>> On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: >>> >>>>>> Yes in theory new requests should go to the other vdev, but there could >>>>>> be some dependency issues preventing that such as a syncing TXG. >>>>> Currenly this pool must not have write activity (from application). >>>>> What about go to the other (mirror) device in the same vdev? >>>>> Same dependency? >>>> Yes, if there's an outstanding TXG, then I believe all IO will stall. >>> Where this TXG released? When all devices in all vdevs report >>> 'completed'? When at the least one device in all vdevs report >>> 'completed'? When at the least one device in at least one vdev report >>> 'completed'? >> When all devices have report completed or failed. > Thanks for explained. > >> Hence if you pull the disk things should continue as normal, with the >> failed device being marked as such. > I am have trouble to phisical access. > May be someone can be suggest software method to forced detach device > from system. In 11 that should be possible with devctl, but under 10 I'm not aware of anything that wouldn't involve some custom kernel level code I'm afraid. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 13:32:40 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D5A412C7 for ; Thu, 7 May 2015 13:32:40 +0000 (UTC) Received: from mail-wg0-f51.google.com (mail-wg0-f51.google.com [74.125.82.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E7CC166B for ; Thu, 7 May 2015 13:32:40 +0000 (UTC) Received: by wgyo15 with SMTP id o15so43719558wgy.2 for ; Thu, 07 May 2015 06:32:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ZHcqNoqge73HSIwXbfy8Zt8bpk5O3XbuFyEz2qtEdiQ=; b=J0xPMlUsumLnbtZQTCUozH/sdORjzGQbdHQ6ybn4yI9wEjfNkMJ6cjf+ZCx4FpvAMZ kQvOZ/9+uH0iIiBkJYPFFZhYAqbq9PhmMptmwjDAY6VNF4zQF0aOMysI65JNVkXoKx6S 3eVNCse7Ah7iyr7NEqYmKK3RuY+I6AsVyQ87JVNY08frmB7YewJjExANxpPfn833eE5h reyc3X7Rb2AGRh13+aZyWnNIu/tuHIq9gi3fEItwQ4ZwLgliLPDAkToh9vCs4T1pb1eY 7YF7ECZanubA/aKPkAmq/iNiyjBdwjyeY0fx4I5L0OYqBGl35ZK3FxNvlNcdyQt/1tjT k0Uw== X-Gm-Message-State: ALoCoQmesT+GU7sU4W4iUg9LMCo/7WjIVUuonsYYV4tN+Iorewx7TZX+7bvbb/iUgCw7r6QMdwPR X-Received: by 10.194.123.67 with SMTP id ly3mr7633396wjb.105.1431005558954; Thu, 07 May 2015 06:32:38 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id mv11sm7465099wic.23.2015.05.07.06.32.37 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 06:32:38 -0700 (PDT) Message-ID: <554B696F.6020902@multiplay.co.uk> Date: Thu, 07 May 2015 14:32:31 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Ronald Klop , Slawa Olhovchenkov CC: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> <20150507131007.GZ62239@zxy.spb.ru> <554B676E.8020500@multiplay.co.uk> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 13:32:41 -0000 On 07/05/2015 14:29, Ronald Klop wrote: > On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland > wrote: > >> >> >> On 07/05/2015 14:10, Slawa Olhovchenkov wrote: >>> On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: >>> >>>> >>>> On 07/05/2015 13:51, Slawa Olhovchenkov wrote: >>>>> On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: >>>>> >>>>>>>> Yes in theory new requests should go to the other vdev, but >>>>>>>> there could >>>>>>>> be some dependency issues preventing that such as a syncing TXG. >>>>>>> Currenly this pool must not have write activity (from application). >>>>>>> What about go to the other (mirror) device in the same vdev? >>>>>>> Same dependency? >>>>>> Yes, if there's an outstanding TXG, then I believe all IO will >>>>>> stall. >>>>> Where this TXG released? When all devices in all vdevs report >>>>> 'completed'? When at the least one device in all vdevs report >>>>> 'completed'? When at the least one device in at least one vdev report >>>>> 'completed'? >>>> When all devices have report completed or failed. >>> Thanks for explained. >>> >>>> Hence if you pull the disk things should continue as normal, with the >>>> failed device being marked as such. >>> I am have trouble to phisical access. >>> May be someone can be suggest software method to forced detach device >>> from system. >> In 11 that should be possible with devctl, but under 10 I'm not aware >> of anything that wouldn't involve some custom kernel level code I'm >> afraid. >> > > > Maybe I'm talking BS here, but does 'camcontrol eject' do something on > a disk? I wouldn't have thought so, I would expect that to only have an effect on removal media such as CDROM drives, but no harm in trying ;-) Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu May 7 13:35:49 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F06593D3 for ; Thu, 7 May 2015 13:35:49 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A8A331689 for ; Thu, 7 May 2015 13:35:49 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YqLxr-0003Ya-0W; Thu, 07 May 2015 16:35:47 +0300 Date: Thu, 7 May 2015 16:35:46 +0300 From: Slawa Olhovchenkov To: Ronald Klop Cc: Steven Hartland , freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk Message-ID: <20150507133546.GA62239@zxy.spb.ru> References: <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> <20150507131007.GZ62239@zxy.spb.ru> <554B676E.8020500@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 13:35:50 -0000 On Thu, May 07, 2015 at 03:29:20PM +0200, Ronald Klop wrote: > On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland > wrote: > > > > > > > On 07/05/2015 14:10, Slawa Olhovchenkov wrote: > >> On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: > >> > >>> > >>> On 07/05/2015 13:51, Slawa Olhovchenkov wrote: > >>>> On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: > >>>> > >>>>>>> Yes in theory new requests should go to the other vdev, but there > >>>>>>> could > >>>>>>> be some dependency issues preventing that such as a syncing TXG. > >>>>>> Currenly this pool must not have write activity (from application). > >>>>>> What about go to the other (mirror) device in the same vdev? > >>>>>> Same dependency? > >>>>> Yes, if there's an outstanding TXG, then I believe all IO will stall. > >>>> Where this TXG released? When all devices in all vdevs report > >>>> 'completed'? When at the least one device in all vdevs report > >>>> 'completed'? When at the least one device in at least one vdev report > >>>> 'completed'? > >>> When all devices have report completed or failed. > >> Thanks for explained. > >> > >>> Hence if you pull the disk things should continue as normal, with the > >>> failed device being marked as such. > >> I am have trouble to phisical access. > >> May be someone can be suggest software method to forced detach device > >> from system. > > In 11 that should be possible with devctl, but under 10 I'm not aware of > > anything that wouldn't involve some custom kernel level code I'm afraid. > > > > > Maybe I'm talking BS here, but does 'camcontrol eject' do something on a > disk? I am don't try 'camcontrol eject' but 'camcontrol identify' already stalled. Need control on adapter layer. From owner-freebsd-stable@FreeBSD.ORG Thu May 7 13:44:57 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6FC3B60A for ; Thu, 7 May 2015 13:44:57 +0000 (UTC) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EBFF179B for ; Thu, 7 May 2015 13:44:57 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1YqLrd-0004F8-DD; Thu, 07 May 2015 15:29:27 +0200 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "Slawa Olhovchenkov" , "Steven Hartland" Cc: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> <20150507131007.GZ62239@zxy.spb.ru> <554B676E.8020500@multiplay.co.uk> Date: Thu, 07 May 2015 15:29:20 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <554B676E.8020500@multiplay.co.uk> User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50, URIBL_BLOCKED autolearn=disabled version=3.3.2 X-Scan-Signature: e462de357cb394d64966911c06262bc8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 13:44:57 -0000 On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland wrote: > > > On 07/05/2015 14:10, Slawa Olhovchenkov wrote: >> On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: >> >>> >>> On 07/05/2015 13:51, Slawa Olhovchenkov wrote: >>>> On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: >>>> >>>>>>> Yes in theory new requests should go to the other vdev, but there >>>>>>> could >>>>>>> be some dependency issues preventing that such as a syncing TXG. >>>>>> Currenly this pool must not have write activity (from application). >>>>>> What about go to the other (mirror) device in the same vdev? >>>>>> Same dependency? >>>>> Yes, if there's an outstanding TXG, then I believe all IO will stall. >>>> Where this TXG released? When all devices in all vdevs report >>>> 'completed'? When at the least one device in all vdevs report >>>> 'completed'? When at the least one device in at least one vdev report >>>> 'completed'? >>> When all devices have report completed or failed. >> Thanks for explained. >> >>> Hence if you pull the disk things should continue as normal, with the >>> failed device being marked as such. >> I am have trouble to phisical access. >> May be someone can be suggest software method to forced detach device >> from system. > In 11 that should be possible with devctl, but under 10 I'm not aware of > anything that wouldn't involve some custom kernel level code I'm afraid. > Maybe I'm talking BS here, but does 'camcontrol eject' do something on a disk? Ronald. From owner-freebsd-stable@FreeBSD.ORG Thu May 7 14:05:50 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7075E4F for ; Thu, 7 May 2015 14:05:50 +0000 (UTC) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.126.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gromit.dlib.vt.edu", Issuer "Chumby Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9BE4F1A79 for ; Thu, 7 May 2015 14:05:50 +0000 (UTC) Received: from pmather.lib.vt.edu (pmather.lib.vt.edu [128.173.126.193]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id 48178D35; Thu, 7 May 2015 09:56:12 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: zfs, cam sticking on failed disk From: Paul Mather In-Reply-To: <554B53E8.4000508@multiplay.co.uk> Date: Thu, 7 May 2015 09:56:11 -0400 Cc: Slawa Olhovchenkov , freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <51E7F693-AA33-4BDD-8CEA-769D8EC20D36@gromit.dlib.vt.edu> References: <20150507080749.GB1394@zxy.spb.ru> <554B2547.1090307@multiplay.co.uk> <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.2098) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 14:05:50 -0000 On May 7, 2015, at 8:00 AM, Steven Hartland = wrote: > On 07/05/2015 11:46, Slawa Olhovchenkov wrote: >> On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: >>=20 >>>>>> How I can cancel this 24 requst? >>>>>> Why this requests don't timeout (3 hours already)? >>>>>> How I can forced detach this disk? (I am lready try `camcontrol = reset`, `camconrol rescan`). >>>>>> Why ZFS (or geom) don't timeout on request and don't rerouted to = da18? >>>>>>=20 >>>>> If they are in mirrors, in theory you can just pull the disk, isci = will >>>>> report to cam and cam will report to ZFS which should all recover. >>>> Yes, zmirror with da18. >>>> I am surprise that ZFS don't use da18. All zpool fully stuck. >>> A single low level request can only be handled by one device, if = that >>> device returns an error then ZFS will use the other device, but not = until. >> Why next requests don't routed to da18? >> Current request stuck on da19 (unlikely, but understund), but why >> stuck all pool? >=20 > Its still waiting for the request from the failed device to complete. = As far as ZFS currently knows there is nothing wrong with the device as = its had no failures. Maybe related to this, but if the drive stalls indefinitely, is it what = leads to the "panic: I/O to pool 'poolname' appears to be hung on vdev = guid GUID-ID at '/dev/somedevice'."? I have a 6-disk RAIDZ2 pool that is used for nightly rsync backups from = various systems. I believe one of the drives is a bit temperamental. = Very occasionally, I discover the backup has failed and the machine = actually paniced because of this drive, with a panic message like the = above. The panic backtrace includes references to vdev_deadman, which = sounds like some sort of dead man's switch/watchdog. It's a bit counter-intuitive that a single drive wandering off into = la-la land can not only cause an entire ZFS pool to wedge, but, worse = still, panic the whole machine. If I'm understanding this thread correctly, part of the problem is that = an I/O never completing is not the same as a failure to ZFS, and hence = ZFS can't call upon various resources in the pool and mechanisms at its = disposal to correct for that. Is that accurate? I would think that never-ending I/O requests would be a type of failure = that ZFS could sustain. It seems from the "hung on vdev" panic that it = does detect this situation, though the resolution (panic) is not ideal. = :-) Cheers, Paul.= From owner-freebsd-stable@FreeBSD.ORG Thu May 7 14:28:47 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EFA81794 for ; Thu, 7 May 2015 14:28:47 +0000 (UTC) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "ca.infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FEB91D73 for ; Thu, 7 May 2015 14:28:47 +0000 (UTC) Received: from ox-dell39.ox.adestra.com (no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged)) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.1/8.15.1) with ESMTPSA id t47ESUsa021976 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 7 May 2015 15:28:32 +0100 (BST) (envelope-from matthew@freebsd.org) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=freebsd.org DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk t47ESUsa021976 Authentication-Results: smtp.infracaninophile.co.uk/t47ESUsa021976; dkim=none reason="no signature"; dkim-adsp=none; dkim-atps=neutral X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged) claimed to be ox-dell39.ox.adestra.com Message-ID: <554B768E.8090705@freebsd.org> Date: Thu, 07 May 2015 15:28:30 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> <20150507131007.GZ62239@zxy.spb.ru> <554B676E.8020500@multiplay.co.uk> <554B696F.6020902@multiplay.co.uk> In-Reply-To: <554B696F.6020902@multiplay.co.uk> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Od1rfIBs561nhQrNclap7aGGflXABCcQd" X-Virus-Scanned: clamav-milter 0.98.6 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 14:28:48 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Od1rfIBs561nhQrNclap7aGGflXABCcQd Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 05/07/15 14:32, Steven Hartland wrote: >=20 >=20 > On 07/05/2015 14:29, Ronald Klop wrote: >> On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland >> wrote: >> >>> >>> >>> On 07/05/2015 14:10, Slawa Olhovchenkov wrote: >>>> On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: >>>> >>>>> >>>>> On 07/05/2015 13:51, Slawa Olhovchenkov wrote: >>>>>> On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: >>>>>> >>>>>>>>> Yes in theory new requests should go to the other vdev, but >>>>>>>>> there could >>>>>>>>> be some dependency issues preventing that such as a syncing TXG= =2E >>>>>>>> Currenly this pool must not have write activity (from applicatio= n). >>>>>>>> What about go to the other (mirror) device in the same vdev? >>>>>>>> Same dependency? >>>>>>> Yes, if there's an outstanding TXG, then I believe all IO will >>>>>>> stall. >>>>>> Where this TXG released? When all devices in all vdevs report >>>>>> 'completed'? When at the least one device in all vdevs report >>>>>> 'completed'? When at the least one device in at least one vdev rep= ort >>>>>> 'completed'? >>>>> When all devices have report completed or failed. >>>> Thanks for explained. >>>> >>>>> Hence if you pull the disk things should continue as normal, with t= he >>>>> failed device being marked as such. >>>> I am have trouble to phisical access. >>>> May be someone can be suggest software method to forced detach devic= e >>>> from system. >>> In 11 that should be possible with devctl, but under 10 I'm not aware= >>> of anything that wouldn't involve some custom kernel level code I'm >>> afraid. >>> >> >> >> Maybe I'm talking BS here, but does 'camcontrol eject' do something on= >> a disk? > I wouldn't have thought so, I would expect that to only have an effect > on removal media such as CDROM drives, but no harm in trying ;-) zpool offline -t zroot da19 Cheers, Matthew --Od1rfIBs561nhQrNclap7aGGflXABCcQd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJVS3aOXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQxOUYxNTRFQ0JGMTEyRTUwNTQ0RTNGMzAw MDUxM0YxMEUwQTlFNEU3AAoJEABRPxDgqeTnshYP/AioJFd200EuPkS4jv2tz7mg eL9aPcevnhGsjHdow9DWVNZTJyjRUVH9mKzyJcZHZsnD8xQptaK+pqRuaOEUeUtv l5PQ1BnpeFGuv8ogy19G5bSewiB+apanUZvcfvJJXObHxpDLflA6Xh0lx3xgHmSD ghFtzz6sdK3RJbHucKcF+ToVN7OeK5VGTUiNuXJriwOLME/wRzlzWXsGdB7xBBNz dTgx88sN/1YxUwvtG9PCEeYd6u4cQYSEKSw/9PR1jk/ZxKKmDYi82bi/n+kqGxZz HnYzJwQxeRl2aJuNlxIRGpO5vRPN9iZc89hKEVQmPnPz++sk/GwN09PYjAqSedwl wBufFlLEYQhtTwxO5Sd7BL2Vta3Dx1R7sP6rZS+55Dom/p0N+VuXF9hVWVkYlM0i 5wiVjw8MIzOVu5aM7nX7IoKgnrWhtw2NLbrxP756XVO7KMKPjpC8U4XIomFuShYv 1eKJaul+K/Jo0p0ISR/aV5qrzhWnQ+A4+X2+EeMLB0IdIXICxuxIdmoWdzKM16uj JLA+M3rdv5Nom7Kyyy3c575EyAvZFFHQDl9CnYWUx5Bi9msTM/tJOwmkE+7Ya9nM XuO0nd6QvForLuydD8lbiTpsbvLZQYvFIbDPhHL06+CC4X0mdxnADDW6yheRFAdN LRbiYILM//Gjtv7zW8o9 =JXkT -----END PGP SIGNATURE----- --Od1rfIBs561nhQrNclap7aGGflXABCcQd-- From owner-freebsd-stable@FreeBSD.ORG Thu May 7 15:05:52 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7232618D for ; Thu, 7 May 2015 15:05:52 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5771D11E8 for ; Thu, 7 May 2015 15:05:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id t47F5qnZ040094 for ; Thu, 7 May 2015 15:05:52 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id t47F5qSw040093 for freebsd-stable@freebsd.org; Thu, 7 May 2015 15:05:52 GMT (envelope-from bdrewery) Received: (qmail 55834 invoked from network); 7 May 2015 10:05:50 -0500 Received: from unknown (HELO ?10.10.1.139?) (freebsd@shatow.net@10.10.1.139) by sweb.xzibition.com with ESMTPA; 7 May 2015 10:05:50 -0500 Message-ID: <554B7F5D.3050805@FreeBSD.org> Date: Thu, 07 May 2015 10:06:05 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Johan Schuijt-Li , freebsd-stable@freebsd.org Subject: Re: panic: pmap active 0xfffff8001b7154b8 References: In-Reply-To: OpenPGP: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QQfDN5mvFGeedxAjOOKS5rVkpVbJB0QKb" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 15:05:52 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --QQfDN5mvFGeedxAjOOKS5rVkpVbJB0QKb Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 5/7/2015 7:08 AM, Johan Schuijt-Li wrote: > Hi, >=20 > We=E2=80=99ve been seeing (seemingly) random reboots on 10.1-RELEASE vi= rtual machines (KVM virtualisation) on our production servers. In an atte= mpt to determine what was causing this we=E2=80=99ve switched to running = a kernel with INVARIANTS enabled. This resulted for us in the following p= anic: >=20 > Unread portion of the kernel message buffer: > panic: pmap active 0xfffff8001b7154b8 > cpuid =3D 3 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe03d= d1493a0 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe03dd149450 > vpanic() at vpanic+0x126/frame 0xfffffe03dd149490 > kassert_panic() at kassert_panic+0x139/frame 0xfffffe03dd149500 > pmap_remove_pages() at pmap_remove_pages+0x8c/frame 0xfffffe03dd1495f0 > exec_new_vmspace() at exec_new_vmspace+0x16a/frame 0xfffffe03dd149650 > exec_elf64_imgact() at exec_elf64_imgact+0x658/frame 0xfffffe03dd149720= > kern_execve() at kern_execve+0x5e4/frame 0xfffffe03dd149a80 > sys_execve() at sys_execve+0x37/frame 0xfffffe03dd149ae0 > amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe03dd149bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03dd149bf0 > --- syscall (59, FreeBSD ELF64, sys_execve), rip =3D 0x80158af1a, rsp =3D= 0x7fffffffac38, rbp =3D 0x7fffffffad40 --- >=20 >=20 > I=E2=80=99ve only come across one other report here (without result unf= ortunate): > https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.ht= ml >=20 I looked around for the conclusion of that thread but could not find it. I was reproducing so often I'm sure this case was fixed. I may have privately contacted one of the VM maintainers to fix it. However lacking evidence I think it just stopped happening for me and I never reported anything useful. > Are other people aware of this issue or working on this? >=20 > I can provide access to a VM with a kernel dump and the kernel build fo= r extra information if needed. >=20 What we really need is a full core dump (minidump) and backtrace. This will let us inspect the pmap state. https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html= https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.= html --=20 Regards, Bryan Drewery --QQfDN5mvFGeedxAjOOKS5rVkpVbJB0QKb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJVS39dAAoJEDXXcbtuRpfPGvoH/24OKaEWx9LvP4M5YVxZ94Kb lMa6RkC7lZWXSzUptoMbQWdlbVC42oQIVgkgYlS8Hm1tn/lrAHstoxc5asgcxxSf SbkVbOnHc2MjXQFQ0ckZDwSv/cPwAV273/pRi7Wx9kNp1y5BFwZoc0EJCa2b6qeo NVJzYRCeEAHqsiMIeuMzDw0odseEleA5niBrc6T8pCLLCCvovYYpDQyv+H1BI9kT PxGeIs1LkvQF01hqp0lyT1MnewfVNmD9vR12zmrmFwm/U65Uxp+SFlDjh0STdHUt rBtLt52Ex8e0IrG1EL3rGbxxdqe6Ghe51jHvzn+MB/9zpWaQO5iEts6Sn16kg58= =hhmF -----END PGP SIGNATURE----- --QQfDN5mvFGeedxAjOOKS5rVkpVbJB0QKb-- From owner-freebsd-stable@FreeBSD.ORG Thu May 7 15:09:36 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 826E53D9 for ; Thu, 7 May 2015 15:09:36 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 67350121D for ; Thu, 7 May 2015 15:09:36 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id t47F9aWV040430 for ; Thu, 7 May 2015 15:09:36 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id t47F9aO8040425 for freebsd-stable@freebsd.org; Thu, 7 May 2015 15:09:36 GMT (envelope-from bdrewery) Received: (qmail 4731 invoked from network); 7 May 2015 10:09:34 -0500 Received: from unknown (HELO ?10.10.1.139?) (freebsd@shatow.net@10.10.1.139) by sweb.xzibition.com with ESMTPA; 7 May 2015 10:09:34 -0500 Message-ID: <554B803D.2020506@FreeBSD.org> Date: Thu, 07 May 2015 10:09:49 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Johan Schuijt-Li , freebsd-stable@freebsd.org Subject: Re: panic: pmap active 0xfffff8001b7154b8 References: <554B7F5D.3050805@FreeBSD.org> In-Reply-To: <554B7F5D.3050805@FreeBSD.org> OpenPGP: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dfBceo2OOr6t51vx9b6p0JucoIoFtbGlE" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 15:09:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --dfBceo2OOr6t51vx9b6p0JucoIoFtbGlE Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 5/7/2015 10:06 AM, Bryan Drewery wrote: > On 5/7/2015 7:08 AM, Johan Schuijt-Li wrote: >> Hi, >> >> We=E2=80=99ve been seeing (seemingly) random reboots on 10.1-RELEASE v= irtual machines (KVM virtualisation) on our production servers. In an att= empt to determine what was causing this we=E2=80=99ve switched to running= a kernel with INVARIANTS enabled. This resulted for us in the following = panic: >> >> Unread portion of the kernel message buffer: >> panic: pmap active 0xfffff8001b7154b8 >> cpuid =3D 3 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe03= dd1493a0 >> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe03dd149450 >> vpanic() at vpanic+0x126/frame 0xfffffe03dd149490 >> kassert_panic() at kassert_panic+0x139/frame 0xfffffe03dd149500 >> pmap_remove_pages() at pmap_remove_pages+0x8c/frame 0xfffffe03dd1495f0= >> exec_new_vmspace() at exec_new_vmspace+0x16a/frame 0xfffffe03dd149650 >> exec_elf64_imgact() at exec_elf64_imgact+0x658/frame 0xfffffe03dd14972= 0 >> kern_execve() at kern_execve+0x5e4/frame 0xfffffe03dd149a80 >> sys_execve() at sys_execve+0x37/frame 0xfffffe03dd149ae0 >> amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe03dd149bf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe03dd149bf0 >> --- syscall (59, FreeBSD ELF64, sys_execve), rip =3D 0x80158af1a, rsp = =3D 0x7fffffffac38, rbp =3D 0x7fffffffad40 --- >> >> >> I=E2=80=99ve only come across one other report here (without result un= fortunate): >> https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.h= tml >> >=20 > I looked around for the conclusion of that thread but could not find it= =2E Found it. https://svnweb.freebsd.org/base?view=3Drevision&revision=3D2710= 00 This fix is in 10.1-RELEASE, so yours must be a little different. --=20 Regards, Bryan Drewery --dfBceo2OOr6t51vx9b6p0JucoIoFtbGlE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJVS4A9AAoJEDXXcbtuRpfPTR0H/3IXL3w28XcJy/eCgERBSPSU x2KatZ7Gt2vk3atIWdF9sKFsN7KvmM3m442Z/+4WVvIrNzhxuO/UVLfx7FTA2Nwt JfPuPTtnDYlTWApPM525+vAH8Cim7MVuvuNGzzF+lu9yx8z0y3+VGn6LJvbFPvdn P3otCXOA46wDwGDgl2FPT7CU9rEa/SfiWzuu5+pnIb0YxDL+LWUSkf7SHTm0ABvv YF3PlxYBzEthtUpV/J+ZM/32m+UOxsjo7yCdAUIrrYPIdBqCQClAnScGurVAHREp vdwRPOwgUv445FuNm8JJP3CYk1tw86S015htRUV9lYT+GkS0SY+c3u6k6eLttvU= =L9t2 -----END PGP SIGNATURE----- --dfBceo2OOr6t51vx9b6p0JucoIoFtbGlE-- From owner-freebsd-stable@FreeBSD.ORG Thu May 7 15:13:31 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B3EDA6CA for ; Thu, 7 May 2015 15:13:31 +0000 (UTC) Received: from mail-wg0-x22c.google.com (mail-wg0-x22c.google.com [IPv6:2a00:1450:400c:c00::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66E8612EC for ; Thu, 7 May 2015 15:13:31 +0000 (UTC) Received: by wgiu9 with SMTP id u9so46788598wgi.3 for ; Thu, 07 May 2015 08:13:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=300.nl; s=tip; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=yt94pXnn0DAFNcPFbYXgRc6NgHlDpIoifXENybaKFUc=; b=sLyVyXn3V5LKW/X68UTulXAy8Rhq21jidvWQ+b7JRC/lJmCJLwsbQQxBPTQQZ06RnK QPrG1eUSVmbDRErSEJBkGMe9K7ZRcBe2bZYB+FisRM+/1lJ6p2MbIs29uYd2CHu5rOGg +DP3Pw2SuoNe4VxdgjurHDHuOEATXh98CZ5UM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=yt94pXnn0DAFNcPFbYXgRc6NgHlDpIoifXENybaKFUc=; b=AtuYr+UhW4VZjaV7kXdLPaPgmjPvEubsxVDQETfRKu+CXq43MKxOvfp/ul4yYbCU8N UaNh8sdXH8Peq5f6t9J0wroRsJEMo41azKy8i/XLeU5w7N00N0t82Xa7O+Y6xYuT9U+p /eU8ueU54TNQAUOPrt4HgPakM9m781VZFSAEQ1nz63EBvvfomFJYUuqDOHbO6ovvfMBi JqD1drgwtCNQb1/7ZsxAYN7+l4K0qdXW26K5afEy/D1tuZBgYemax+M5yvRYFNBu1uC7 GDAf29T4wmtj7rwFZfqKAJQlWJr6BoJgfrb3pgFvma4oZ2QQOPjg9WNQ/4cbOJjAR7fI O4QQ== X-Gm-Message-State: ALoCoQmvSQ8aUxZc/Pi+Grt/yP4JkM9+JxMj/CLnMy5OPyaYezMD6+lzsWk0Q34JxxAFLVz0s46P X-Received: by 10.194.81.169 with SMTP id b9mr8322617wjy.126.1431011609773; Thu, 07 May 2015 08:13:29 -0700 (PDT) Received: from [10.2.7.215] (kantoor.transip.nl. [80.69.69.100]) by mx.google.com with ESMTPSA id bh7sm3918967wjb.8.2015.05.07.08.13.28 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 07 May 2015 08:13:28 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: panic: pmap active 0xfffff8001b7154b8 From: Johan Schuijt-Li In-Reply-To: <554B7F5D.3050805@FreeBSD.org> Date: Thu, 7 May 2015 17:13:27 +0200 Cc: freebsd-stable@freebsd.org Message-Id: <13D07839-664F-4598-9036-E94D9AC7D0AE@300.nl> References: <554B7F5D.3050805@FreeBSD.org> To: Bryan Drewery X-Mailer: Apple Mail (2.2098) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 15:13:31 -0000 >=20 > What we really need is a full core dump (minidump) and backtrace. This > will let us inspect the pmap state. >=20 > = https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html = > = https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.h= tml = Sorry if my words were a bit unclear, but this is what I have. If you = could e-mail me a preferred username with public key I can give you = access to this. - Johan= From owner-freebsd-stable@FreeBSD.ORG Thu May 7 20:04:23 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C8F49459 for ; Thu, 7 May 2015 20:04:23 +0000 (UTC) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 631EC17A2 for ; Thu, 7 May 2015 20:04:23 +0000 (UTC) Received: by widdi4 with SMTP id di4so4407476wid.0 for ; Thu, 07 May 2015 13:04:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=PXOv+ENXpdfuYAhdYvU1/XIoMOU+CE6Ud2S9jQpyzxo=; b=KtQbR6jeYRpX8V0VDH5XRyhym2BmKvn4ZB9XkTiF/3XaPrR1axmCCvh7C4AhC4wjOv SYX3DoUPLzElTWqVC4icEYV+R4SoWSo5Jeq3DQyzsa2Ma+m1P5FXsUtnMLlwwe9U74bb 2hxXWeQFQCPc9PeSu4neQz4gcNi9CFaW4DYf+A/IeVQRklETE73aXF7eZIi4QUJX+Juf sPERR9eMvbDQgrKubPajhMVK3BbJ4elHST8MjbKhA8oZz0j2YEwPx6MtFs/Y7Nx2uHj1 0GQOiE1O1fZylw52Qf7D78BfAJ/n3ZyCL2pylCKYNE8zEEkW5tHAJ7MhG/TPys9ZiqPL 3hdw== X-Gm-Message-State: ALoCoQk5Jn0cdE6OQdmNXOVgq5ThOlGzmch8ASl1GJW02Cv2DoIYxinzUyMa9JxmQfmZ/aRQRwf3 X-Received: by 10.180.216.40 with SMTP id on8mr518148wic.55.1431029055283; Thu, 07 May 2015 13:04:15 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id l3sm9009676wiv.18.2015.05.07.13.04.14 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 May 2015 13:04:14 -0700 (PDT) Message-ID: <554BC538.5050406@multiplay.co.uk> Date: Thu, 07 May 2015 21:04:08 +0100 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: zfs, cam sticking on failed disk References: <20150507095048.GC1394@zxy.spb.ru> <554B40B6.6060902@multiplay.co.uk> <20150507104655.GT62239@zxy.spb.ru> <554B53E8.4000508@multiplay.co.uk> <20150507120508.GX62239@zxy.spb.ru> <554B5BF9.8020709@multiplay.co.uk> <20150507124416.GD1394@zxy.spb.ru> <554B5EB0.1080208@multiplay.co.uk> <20150507125129.GY62239@zxy.spb.ru> <554B6307.9020309@multiplay.co.uk> <20150507131007.GZ62239@zxy.spb.ru> <554B676E.8020500@multiplay.co.uk> <554B696F.6020902@multiplay.co.uk> <554B768E.8090705@freebsd.org> In-Reply-To: <554B768E.8090705@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2015 20:04:23 -0000 On 07/05/2015 15:28, Matthew Seaman wrote: > On 05/07/15 14:32, Steven Hartland wrote: >> >> I wouldn't have thought so, I would expect that to only have an effect >> on removal media such as CDROM drives, but no harm in trying ;-) > zpool offline -t zroot da19 > > That might work but it also might just wedge waiting for the outstanding IO to complete as I thought that was a "nice" off-line instead of a I don't care one. From owner-freebsd-stable@FreeBSD.ORG Fri May 8 22:37:31 2015 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02C486D6 for ; Fri, 8 May 2015 22:37:31 +0000 (UTC) Received: from fedex2.jetcafe.org (fedex2.jetcafe.org [205.147.26.23]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C9B621BA7 for ; Fri, 8 May 2015 22:37:30 +0000 (UTC) X-Envelope-To: Received: from [205.147.26.4] (hokkshideh.jetcafe.org [205.147.26.4]) by fedex2.jetcafe.org (8.14.9/8.14.9) with ESMTP id t48MFUMd041830 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Fri, 8 May 2015 15:15:31 -0700 (PDT) (envelope-from dave@jetcafe.org) Message-ID: <554D3582.1020707@jetcafe.org> Date: Fri, 08 May 2015 15:15:30 -0700 From: Dave Hayes User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org Subject: Pf, rtable, and rdr...bug? Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -1 ( out of 6) ALL_TRUSTED,SHORTCIRCUIT X-Spam-Checker-Version: SpamAssassin version 3.4.0 X-Scanned-By: MIMEDefang 2.75 on 205.147.26.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 May 2015 22:37:31 -0000 Hello everyone. I'm having a problem with using rdr in an existing pf that uses rtable. I'm running 10.1-STABLE #0 r282154 and I believe this is a bug, but it could also be something I haven't spotted. I have a firewall with three interfaces. The ip addresses have been changed to protect the innocent. :) - a slow net (1.2.3.0/24) interface: em0 @ 1.2.3.10 - a fast net (4.5.6.0/24) interface: em1 @ 4.5.6.10 - an internal net (192.168.4.0/24) interface: em2 @ 192.168.4.10 I route the internal net traffic over the fast cable net, and allow the internet net to access machines on the slower work net. Both default routes for the slow and fast net are .1 addresses (e.g. 1.2.3.1 and 4.5.6.1). I use an alias on both the slow and fast net (.42) to route the traffic from so I can see what's going on. I have net.fibs="2" in loader.conf and two different default routes set up for each fib. The default "default route" (fib 0) is 1.2.3.1. Here's my pf ruleset that works, paraphrased. $slow_net = "1.2.3.0/24" $slow_if = "em0" $slow_nat_ip = "1.2.3.42" $fast_net = "4.5.6.0/24" $fast_if = "em1" $fast_nat_ip = "4.5.6.42" $int_net = "192.168.4.0/24" $int_if = "em2" $int_ip = "192.168.4.10" # I don't alias this side table const { 10/8, 172.16/12, 192.168/16 } nat log in $fast_if inet from $int_if:network to ! $slow_net -> $fast_nat_ip nat log on $slow_if inet from $int_if:network to $slow_net -> $slow_nat_ip block in log all antispoof log quick for { $slow_if $fast_if $int_if } pass in log quick on $int_if inet from $int_net to !$slow_if:network modulate state rtable 1 pass in log quick on $int_if inet from $int_net to $slow_if:network modulate state rtable 0 pass log on $slow_if inet from ! to any modulate state pass out log inet from any to any modulate state So I tried to use rdr to forward some ports from the to a machine on the internal net: $webserver = "192.168.4.22" .... rdr on $fast_if inet proto tcp from any to port 80 -> $webserver This doesn't work. When I turn on tcpdump on all three interfaces, I see the packets coming in from the fast net to the internal net. The responses are appearing on the slow net, with the IP addresses of the fast net. So if I see this from em1: 14:34:11.887357 IP 10.11.12.13:18600 > 4.5.6.42:80 ... I then see the response...but on em0: 14:34:12.087283 IP 4.5.6.42:80 > 10.11.12.13:18600 ... Why doesn't this response packet go out the proper interface? Thanks in advance for any insight. If I don't hear from anyone, I'm going to assume this is a bug and file a bug report. -- Dave Hayes - Consultant - Altadena CA, USA - dave@jetcafe.org >>>> *The opinions expressed above are entirely my own* <<<< A path and a gateway have no meaning or use once the objective is in sight.