From owner-freebsd-bugs@freebsd.org Tue Feb 20 12:08:20 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F33AF2381D for ; Tue, 20 Feb 2018 12:08:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2140F73989 for ; Tue, 20 Feb 2018 12:08:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 345111F1AE for ; Tue, 20 Feb 2018 12:08:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w1KC8Jav015164 for ; Tue, 20 Feb 2018 12:08:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w1KC8JZ8015148 for freebsd-bugs@FreeBSD.org; Tue, 20 Feb 2018 12:08:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 226064] Observing panic at ctl_free_beio+0x80 while running traffic on 100 iSCSI LUNs Date: Tue, 20 Feb 2018 12:08:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: manishk@chelsio.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter cc attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Feb 2018 12:08:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D226064 Bug ID: 226064 Summary: Observing panic at ctl_free_beio+0x80 while running traffic on 100 iSCSI LUNs Product: Base System Version: CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: manishk@chelsio.com CC: mav@FreeBSD.org Created attachment 190824 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D190824&action= =3Dedit core text file Observing panic while running traffic on 100 iSCSI LUNs (used zvols as LUNs= ) on the target machine. The panic stack trace:=20=20 panic: Memory modified after free 0xfffff8016fadcb40(32) val=3D0 @ 0xfffff8016fadcb40 cpuid =3D 0 time =3D 1518462716 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe0065865= 370 kdb_backtrace() at kdb_backtrace+0x53/frame 0xfffffe0065865440 vpanic() at vpanic+0x268/frame 0xfffffe0065865510 kproc_shutdown() at kproc_shutdown/frame 0xfffffe0065865570 trash_ctor() at trash_ctor+0x6b/frame 0xfffffe00658655b0 uma_zalloc_arg() at uma_zalloc_arg+0x284/frame 0xfffffe00658656f0 bucket_alloc() at bucket_alloc+0x102/frame 0xfffffe0065865730 uma_zfree_arg() at uma_zfree_arg+0x647/frame 0xfffffe0065865820 uma_zfree() at uma_zfree+0x29/frame 0xfffffe0065865850 ctl_free_beio() at ctl_free_beio+0x80/frame 0xfffffe0065865880 ctl_complete_beio() at ctl_complete_beio+0x48/frame 0xfffffe00658658a0 ctl_be_block_dispatch_zvol() at ctl_be_block_dispatch_zvol+0x664/frame 0xfffffe00658659b0 ctl_be_block_worker() at ctl_be_block_worker+0x1fb/frame 0xfffffe0065865a40 taskqueue_run_locked() at taskqueue_run_locked+0x246/frame 0xfffffe0065865a= b0 taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe0065865= b30 fork_exit() at fork_exit+0x145/frame 0xfffffe0065865bb0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0065865bb0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- KDB: enter: panic ------------------------------------- Steps to recreate the issue: ------------------------------------- The target machine exposes 100 zvols as iSCSI LUNs. The Initiator discovers these LUNS, creates file system on them, and runs traffic (iozone). The tar= get machine panics after few hours. OS: FreeBSD 12.0-CURRENT (svn r329019) I'm using GENERIC config file, with "nooptions VIMAGE" =3D=3D>Start the Target (FreeBSD) 1. Create 100 zvols # zpool create iscsi # for ((i=3D0;i<100;i++)); do zfs create -V1G -o volmode=3Ddev iscsi/= d$i; done 2. configure the network interface to be used (Target_IP_Address) 2. /etc/ctl.conf file: portal-group pg0 { discovery-auth-group no-authentication listen } target iqn.2016-11.com.xyz.abc:0 { auth-group no-authentication portal-group pg0 lun 0 { path /dev/zvol/iscsi/d0 size 1G option vendor "foo" option product "bar" option revision "d0" } } . . 100 targets with 1 lun each 3. start the target daemon=20 # service ctld onestart =3D=3D>Initiator (Linux) 1. login/discover the target=20 # iscsiadm -m discovery -t st -p -l 2. List all Luns=20 # lsscsi=20=20=20=20=20=20=20=20 3. Format and create file system ONLY on all discovered luns (TAKE CARE= TO NOT TOUCH OTHER LUNS).=20 # mkfs.ext3 /dev/sdxx // for all discovered LUNs 4. mount all the devices formatted in the above step to different mount points # mount /dev/sdxx /mnt/iscsiyy // for all the luns formatted in = the above step 5. Run traffic on all luns using iozone # cd /mnt/iscsiyy # iozone -a -I -+d -g 512m & =3D=3D> Target machine panics after 3 to 10 hours. --=20 You are receiving this mail because: You are the assignee for the bug.=