From owner-freebsd-bugs@freebsd.org Thu Feb 22 07:55:11 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97791F0FDD6 for ; Thu, 22 Feb 2018 07:55:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 13BD573848 for ; Thu, 22 Feb 2018 07:55:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 60D1F16693 for ; Thu, 22 Feb 2018 07:55:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w1M7tAPg083984 for ; Thu, 22 Feb 2018 07:55:10 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w1M7tA6M083983 for freebsd-bugs@FreeBSD.org; Thu, 22 Feb 2018 07:55:10 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 226110] Observing ASSERT at cfiscsi_session_delete+0x49 while running traffic on 100 iSCSI LUNs Date: Thu, 22 Feb 2018 07:55:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: manishk@chelsio.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 07:55:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D226110 Bug ID: 226110 Summary: Observing ASSERT at cfiscsi_session_delete+0x49 while running traffic on 100 iSCSI LUNs Product: Base System Version: CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: manishk@chelsio.com CC: trasz@FreeBSD.org Created attachment 190884 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D190884&action= =3Dedit core text file Observing an assert while running traffic on 100 iSCSI LUNs (used zvols as LUNs) on the target machine. The panic stack trace:=20=20 panic: destroying session with outstanding CTL pdus cpuid =3D 4 time =3D 1519157619 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe005f30f= 880 kdb_backtrace() at kdb_backtrace+0x53/frame 0xfffffe005f30f950 vpanic() at vpanic+0x268/frame 0xfffffe005f30fa20 kassert_panic() at kassert_panic+0xc7/frame 0xfffffe005f30fab0 cfiscsi_session_delete() at cfiscsi_session_delete+0x49/frame 0xfffffe005f30fae0 cfiscsi_maintenance_thread() at cfiscsi_maintenance_thread+0x110/frame 0xfffffe005f30fb30 fork_exit() at fork_exit+0x145/frame 0xfffffe005f30fbb0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe005f30fbb0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- KDB: enter: panic ------------------------------------- Steps to recreate the issue: ------------------------------------- The target machine exposes 100 zvols as iSCSI LUNs. The Initiator discovers these LUNS, creates file system on them, and runs traffic (iozone). The tar= get machine asserts after few hours. OS: FreeBSD 12.0-CURRENT (svn r329019) I'm using GENERIC config file, with "nooptions VIMAGE" =3D=3D>Start the Target (FreeBSD) 1. Create 100 zvols # zpool create iscsi # for ((i=3D0;i<100;i++)); do zfs create -V1G -o volmode=3Ddev iscsi/= d$i; done 2. configure the network interface to be used (Target_IP_Address) 3. /etc/ctl.conf file: portal-group pg0 { discovery-auth-group no-authentication listen } target iqn.2016-11.com.xyz.abc:0 { auth-group no-authentication portal-group pg0 lun 0 { path /dev/zvol/iscsi/d0 size 1G option vendor "foo" option product "bar" option revision "d0" } } . . 100 targets with 1 lun each 4. start the target daemon=20 # service ctld onestart =3D=3D>Initiator (Linux) 1. login/discover the target=20 # iscsiadm -m discovery -t st -p -l 2. List all Luns=20 # lsscsi=20=20=20=20=20=20=20=20 3. Format and create file system ONLY on all discovered luns (TAKE CARE= TO NOT TOUCH OTHER LUNS).=20 # mkfs.ext3 /dev/sdxx // for all discovered LUNs 4. mount all the devices formatted in the above step to different mount points # mount /dev/sdxx /mnt/iscsiyy // for all the luns formatted in = the above step 5. Run traffic on all luns using iozone # cd /mnt/iscsiyy # iozone -a -I -+d -g 512m & =3D=3D> Target machine asserts after 6-7 hours. --=20 You are receiving this mail because: You are the assignee for the bug.=