From owner-freebsd-stable@freebsd.org Tue Oct 20 16:44:58 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7EFC7A1A47B for ; Tue, 20 Oct 2015 16:44:58 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0EB431136 for ; Tue, 20 Oct 2015 16:44:58 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: by wicll6 with SMTP id ll6so54365666wic.0 for ; Tue, 20 Oct 2015 09:44:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=jl6Zuri1DQN+0/98Bx6C07vAAHBjEjzBruS/Z/lx8TI=; b=aW56OKHMliSfnezIgBwtQWp3iquufmxQi5el8ppTr+yLYx8QUjwZgjf+8Zyn5z7RO/ yBRaCk7hgIKj0Robws2x7rdiNOdFi+WCHbaAABG0p1IAfwbusYCkD3Jy0fkm47A4Srvc t9ep6Nn0MDym4jLZsAr703V7ypjZeoFX0/miGmdf8VZrf4qCLSvlFdsIB/EmZqtncpP1 Xbc+03VSBDNW2luVZcRW9+RFzw6FasaG6fkQRDwlQZzZ8OCXYbKn4bZULpKHzng9kQlc el8wJagwXGzc/kcdosKSLySDo4QKm5iXRJ2bNtYG9KndR4QUCCuWgYH2Tu+QlNB3kHxO 5SSw== X-Received: by 10.194.47.244 with SMTP id g20mr5923798wjn.124.1445359495913; Tue, 20 Oct 2015 09:44:55 -0700 (PDT) Received: from brick.home (adix164.neoplus.adsl.tpnet.pl. [79.184.205.164]) by smtp.gmail.com with ESMTPSA id hs5sm20075986wib.6.2015.10.20.09.44.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Oct 2015 09:44:54 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Tue, 20 Oct 2015 18:44:52 +0200 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: "Frank de Bot (lists)" Cc: FreeBSD Stable Subject: Re: Reloading iscsi target, iscsi initiator panics Message-ID: <20151020164452.GA3436@brick.home> Mail-Followup-To: "Frank de Bot (lists)" , FreeBSD Stable References: <561EC629.30106@searchy.net> <20151015113428.GA3741@brick.home> <561FEAF5.8080607@searchy.net> <20151015201212.GA4609@brick.home> <5620A389.5010901@searchy.net> <56240D5D.4060205@searchy.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <56240D5D.4060205@searchy.net> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 16:44:58 -0000 On 1018T2321, Frank de Bot (lists) wrote: > Frank de Bot wrote: > > Edward Tomasz Napierała wrote: > >> On 1015T2005, Frank de Bot (lists) wrote: > >>> Edward Tomasz Napierała wrote: > >>>> On 1014T2316, Frank de Bot (lists) wrote: > >>>>> Hello, > >>>>> > >>>>> I have a FreeBSD 10.2 server running as iSCSI target. Another FreeBSD > >>>>> 10.2 is an initiator and has several targets used. > >>>>> > >>>>> When I add a target and reload its config with 'service ctld reload', > >>>>> the FreeBSD initiator server panics. It's reproducable (I have a > >>>>> coredump, but I think it's clear what the problem is) > >>>> > >>>> How easy it is to reproduce, eg. does it happen about every time? > >>>> Could you paste the backtrace? > >>> > >>> The first time I didn't understand what happened, the second time I > >>> could relate it directly to the reloading of ctld on the iSCSI, within > >>> moments the server paniced. > >>> > >>> I got 2 backtraces: > >>> > >>> First one: > >>> > >>> Fatal trap 12: page fault while in kernel mode > >>> cpuid = 1; softdep_deallocate_dependencies: got error 6 while accessing > >> > >> This is the FFS on the initiator side panicing because the device it's > >> on went away. The softupdates code can't handle that very well. > >> > >> I have no idea why the devices went away and then reappeared, as visible > >> in the logs. What has the changed in the ctl.conf? Do you have any > >> iscsi-related sysctls set on the initiator side? > >> > > > > I've added a new target in the ctl.conf . On the linux server I also see > > a brief disconnect, but a reconnect is handled well. > > > > I haven't set any sysctl's related to iscsi > > > > My ctl.conf is (again anonymized): > > > > auth-group my-auth { > > chap "myiscsi" "verysecret" > > } > > portal-group pg0 { > > discovery-auth-group my-auth > > listen 10.13.37.2 > > } > > > > target iqn.2015-03.lan.my.nas:vmstorage-29 { > > auth-group my-auth > > portal-group pg0 > > lun 0 { > > path /tank/images/iscsi/vmstorage-29/vmstorage-29.img > > size 20484M > > blocksize 4096 > > option unmap on > > } > > } > > > > target iqn.2015-03.lan.my.nas:vmstorage-44 { > > auth-group my-auth > > portal-group pg0 > > lun 0 { > > path /tank/images/iscsi/vmstorage-44/vmstorage-44.img > > size 102404M > > blocksize 4096 > > option unmap on > > } > > } > > > > target iqn.2015-03.lan.my.nas:keyserver.my.nl { > > auth-group my-auth > > portal-group pg0 > > lun 0 { > > path /dev/zvol/tank/hosting_images/keyserver.my.nl > > blocksize 4096 > > option unmap on > > } > > } > > > > the vmstorage-44 is last added to the config > > > > I've started up my test environment, but I could not reproduce is there. > When reloading the target, a > SCSI sense: UNIT ATTENTION asc:3f,e (Reported LUNs data has changed) is > reported, but no device is destroyed. All servers are running the same > kernel and userland 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666 > > What can those device disconnects cause? Well, that's the thing: ctld was written to make really, really sure that reloading configuration does not affect LUNs and targets which hadn't changed. So, to be honest - no idea. Are you sure you didn't restart (as in, stop and then start again) ctld instead of reloading its configuration?