Date: Tue, 20 Oct 2015 18:44:52 +0200 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= <trasz@FreeBSD.org> To: "Frank de Bot (lists)" <lists@searchy.net> Cc: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: Reloading iscsi target, iscsi initiator panics Message-ID: <20151020164452.GA3436@brick.home> In-Reply-To: <56240D5D.4060205@searchy.net> References: <561EC629.30106@searchy.net> <20151015113428.GA3741@brick.home> <561FEAF5.8080607@searchy.net> <20151015201212.GA4609@brick.home> <5620A389.5010901@searchy.net> <56240D5D.4060205@searchy.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1018T2321, Frank de Bot (lists) wrote:
> Frank de Bot wrote:
> > Edward Tomasz Napierała wrote:
> >> On 1015T2005, Frank de Bot (lists) wrote:
> >>> Edward Tomasz Napierała wrote:
> >>>> On 1014T2316, Frank de Bot (lists) wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I have a FreeBSD 10.2 server running as iSCSI target. Another FreeBSD
> >>>>> 10.2 is an initiator and has several targets used.
> >>>>>
> >>>>> When I add a target and reload its config with 'service ctld reload',
> >>>>> the FreeBSD initiator server panics. It's reproducable (I have a
> >>>>> coredump, but I think it's clear what the problem is)
> >>>>
> >>>> How easy it is to reproduce, eg. does it happen about every time?
> >>>> Could you paste the backtrace?
> >>>
> >>> The first time I didn't understand what happened, the second time I
> >>> could relate it directly to the reloading of ctld on the iSCSI, within
> >>> moments the server paniced.
> >>>
> >>> I got 2 backtraces:
> >>>
> >>> First one:
> >>>
> >>> Fatal trap 12: page fault while in kernel mode
> >>> cpuid = 1; softdep_deallocate_dependencies: got error 6 while accessing
> >>
> >> This is the FFS on the initiator side panicing because the device it's
> >> on went away. The softupdates code can't handle that very well.
> >>
> >> I have no idea why the devices went away and then reappeared, as visible
> >> in the logs. What has the changed in the ctl.conf? Do you have any
> >> iscsi-related sysctls set on the initiator side?
> >>
> >
> > I've added a new target in the ctl.conf . On the linux server I also see
> > a brief disconnect, but a reconnect is handled well.
> >
> > I haven't set any sysctl's related to iscsi
> >
> > My ctl.conf is (again anonymized):
> >
> > auth-group my-auth {
> > chap "myiscsi" "verysecret"
> > }
> > portal-group pg0 {
> > discovery-auth-group my-auth
> > listen 10.13.37.2
> > }
> >
> > target iqn.2015-03.lan.my.nas:vmstorage-29 {
> > auth-group my-auth
> > portal-group pg0
> > lun 0 {
> > path /tank/images/iscsi/vmstorage-29/vmstorage-29.img
> > size 20484M
> > blocksize 4096
> > option unmap on
> > }
> > }
> >
> > target iqn.2015-03.lan.my.nas:vmstorage-44 {
> > auth-group my-auth
> > portal-group pg0
> > lun 0 {
> > path /tank/images/iscsi/vmstorage-44/vmstorage-44.img
> > size 102404M
> > blocksize 4096
> > option unmap on
> > }
> > }
> >
> > target iqn.2015-03.lan.my.nas:keyserver.my.nl {
> > auth-group my-auth
> > portal-group pg0
> > lun 0 {
> > path /dev/zvol/tank/hosting_images/keyserver.my.nl
> > blocksize 4096
> > option unmap on
> > }
> > }
> >
> > the vmstorage-44 is last added to the config
> >
>
> I've started up my test environment, but I could not reproduce is there.
> When reloading the target, a
> SCSI sense: UNIT ATTENTION asc:3f,e (Reported LUNs data has changed) is
> reported, but no device is destroyed. All servers are running the same
> kernel and userland 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666
>
> What can those device disconnects cause?
Well, that's the thing: ctld was written to make really, really sure
that reloading configuration does not affect LUNs and targets which
hadn't changed. So, to be honest - no idea. Are you sure you didn't
restart (as in, stop and then start again) ctld instead of reloading
its configuration?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151020164452.GA3436>