From owner-freebsd-scsi@FreeBSD.ORG Mon Feb 24 07:12:54 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6AA5D6FD; Mon, 24 Feb 2014 07:12:54 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3E5D11C02; Mon, 24 Feb 2014 07:12:54 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s1O7Cs3U043169; Mon, 24 Feb 2014 07:12:54 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s1O7CsdJ043168; Mon, 24 Feb 2014 07:12:54 GMT (envelope-from linimon) Date: Mon, 24 Feb 2014 07:12:54 GMT Message-Id: <201402240712.s1O7CsdJ043168@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/186900: [iscsi] problem with ISCSICTL, TARGET MOVED, DELL EQUALLOGIC X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Feb 2014 07:12:54 -0000 Old Synopsis: ISCSICTL, TARGET MOVED, DELL EQUALLOGIC New Synopsis: [iscsi] problem with ISCSICTL, TARGET MOVED, DELL EQUALLOGIC Responsible-Changed-From-To: freebsd-amd64->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Mon Feb 24 07:12:14 UTC 2014 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=186900 From owner-freebsd-scsi@FreeBSD.ORG Mon Feb 24 11:06:57 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4AB22B26 for ; Mon, 24 Feb 2014 11:06:57 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 367D1162F for ; Mon, 24 Feb 2014 11:06:57 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s1OB6v9o027683 for ; Mon, 24 Feb 2014 11:06:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s1OB6umn027681 for freebsd-scsi@FreeBSD.org; Mon, 24 Feb 2014 11:06:56 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Feb 2014 11:06:56 GMT Message-Id: <201402241106.s1OB6umn027681@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Feb 2014 11:06:57 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/186900 scsi [iscsi] problem with ISCSICTL, TARGET MOVED, DELL EQUA o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 18 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Feb 25 23:00:34 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 29D10796; Tue, 25 Feb 2014 23:00:34 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EF83A15BD; Tue, 25 Feb 2014 23:00:33 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s1PN0Xd2041970; Tue, 25 Feb 2014 23:00:33 GMT (envelope-from trasz@freefall.freebsd.org) Received: (from trasz@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s1PN0XBT041969; Tue, 25 Feb 2014 23:00:33 GMT (envelope-from trasz) Date: Tue, 25 Feb 2014 23:00:33 GMT Message-Id: <201402252300.s1PN0XBT041969@freefall.freebsd.org> To: trasz@FreeBSD.org, freebsd-scsi@FreeBSD.org, trasz@FreeBSD.org From: trasz@FreeBSD.org Subject: Re: kern/186900: [iscsi] problem with ISCSICTL, TARGET MOVED, DELL EQUALLOGIC X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Feb 2014 23:00:34 -0000 Synopsis: [iscsi] problem with ISCSICTL, TARGET MOVED, DELL EQUALLOGIC Responsible-Changed-From-To: freebsd-scsi->trasz Responsible-Changed-By: trasz Responsible-Changed-When: Tue Feb 25 23:00:33 UTC 2014 Responsible-Changed-Why: I'll take it. http://www.freebsd.org/cgi/query-pr.cgi?pr=186900 From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 10:06:33 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC167975 for ; Fri, 28 Feb 2014 10:06:33 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id AE401182C for ; Fri, 28 Feb 2014 10:06:33 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 9C8769DD0ED for ; Fri, 28 Feb 2014 11:06:25 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: ServeRAID M5210e passthroughand syspd corruption Date: Fri, 28 Feb 2014 11:06:21 +0100 Message-Id: To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 10:06:34 -0000 Hello, I'm trying to make this card work. As I am going to use ZFS, I need it = in passthrough mode. However, both in 10-STABLE and 11-CURRENT it's corrupting data. The card is detected as follows: mfi0: port 0x4f00-0x4fff mem = 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on = pci22 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23=20 mfi0: FW MaxCmds =3D 240, limiting to 128 mfi0: MaxCmd =3D 240, Drv MaxCmd =3D 128, MaxSgl =3D 70, state =3D = 0xb73c00f0 mfi0: 10016 (446833719s/0x0020/info) - Shutdown command received from = host mfi0: 10017 (boot + 10s/0x0020/info) - Firmware initialization started = (PCI ID 005d/1000/045b/1014) mfi0: 10018 (boot + 10s/0x0020/info) - Firmware version 4.200.21-2840 mfi0: 10019 (boot + 12s/0x0020/info) - Package version 24.0.2-0013 mfi0: 10020 (boot + 12s/0x0020/info) - Board Revision 00AL055 mfi0: 10021 (boot + 30s/0x0004/info) - Enclosure (SES) discovered on PD = 08(c Port 0 - 3/p1) mfi0: 10022 (boot + 30s/0x0004/info) - Enclosure (SES) discovered on PD = 0c(c Port 0 - 3/p2) mfi0: 10023 (boot + 30s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) = communication restored mfi0: 10024 (boot + 30s/0x0004/info) - Enclosure PD 0c(c Port 0 - 3/p2) = communication restored mfi0: 10025 (boot + 30s/0x0002/info) - Inserted: Encl PD 08 mfisyspd0 on mfi0 And according to mfiutil, root@merde:/home/borjam # mfiutil show adapter mfi0 Adapter: Product Name: ServeRAID M5210e Serial Number: XXXXX Firmware: 24.0.2-0013 RAID Levels: JBOD, RAID0, RAID1, RAID10 Battery Backup: not present NVRAM: 32K Onboard Memory: 0M Minimum Stripe: 64K Maximum Stripe: 64K Curiously, other mfi cards which don't officially support syspd mode = work in passthrough without problems. And indeed mfisyspd is not adequate to use SSDs. It's much better to use = passthrough and the "da" driver. Is there any download I can try for the "mrsas" driver to test under = FreeBSD 10 and 11? I'd like to know if mrsas can make it work properly. Also, I am wondering. Is it possible that syspd support in the driver = may have caused trouble? I was considering purging it and try again. Thanks! Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 10:18:02 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8EAAE3B5 for ; Fri, 28 Feb 2014 10:18:02 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2B5511948 for ; Fri, 28 Feb 2014 10:18:01 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50008536365.msg for ; Fri, 28 Feb 2014 10:17:51 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 28 Feb 2014 10:17:51 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1136e49979=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> From: "Steven Hartland" To: "Borja Marcos" , References: Subject: Re: ServeRAID M5210e passthroughand syspd corruption Date: Fri, 28 Feb 2014 10:17:53 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 10:18:02 -0000 mfi is a raid card it doesn't really have a true passthrough mfisyspd is the best you'll get. I'd recommend you switch to an HBA card which uses mps driver instead. When you said you're seeing mfisyspd corruption what specifically are you seeing? Regards Steve ----- Original Message ----- From: "Borja Marcos" To: Sent: Friday, February 28, 2014 10:06 AM Subject: ServeRAID M5210e passthroughand syspd corruption > > Hello, > > I'm trying to make this card work. As I am going to use ZFS, I need it in passthrough mode. However, both in 10-STABLE and > 11-CURRENT > it's corrupting data. > > The card is detected as follows: > mfi0: port 0x4f00-0x4fff mem 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on pci22 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > mfi0: FW MaxCmds = 240, limiting to 128 > mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0 > mfi0: 10016 (446833719s/0x0020/info) - Shutdown command received from host > mfi0: 10017 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/045b/1014) > mfi0: 10018 (boot + 10s/0x0020/info) - Firmware version 4.200.21-2840 > mfi0: 10019 (boot + 12s/0x0020/info) - Package version 24.0.2-0013 > mfi0: 10020 (boot + 12s/0x0020/info) - Board Revision 00AL055 > mfi0: 10021 (boot + 30s/0x0004/info) - Enclosure (SES) discovered on PD 08(c Port 0 - 3/p1) > mfi0: 10022 (boot + 30s/0x0004/info) - Enclosure (SES) discovered on PD 0c(c Port 0 - 3/p2) > mfi0: 10023 (boot + 30s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) communication restored > mfi0: 10024 (boot + 30s/0x0004/info) - Enclosure PD 0c(c Port 0 - 3/p2) communication restored > mfi0: 10025 (boot + 30s/0x0002/info) - Inserted: Encl PD 08 > mfisyspd0 on mfi0 > > And according to mfiutil, > root@merde:/home/borjam # mfiutil show adapter > mfi0 Adapter: > Product Name: ServeRAID M5210e > Serial Number: XXXXX > Firmware: 24.0.2-0013 > RAID Levels: JBOD, RAID0, RAID1, RAID10 > Battery Backup: not present > NVRAM: 32K > Onboard Memory: 0M > Minimum Stripe: 64K > Maximum Stripe: 64K > > > Curiously, other mfi cards which don't officially support syspd mode work in passthrough without problems. And indeed > mfisyspd is not adequate to use SSDs. It's much better to use passthrough and the "da" driver. > > Is there any download I can try for the "mrsas" driver to test under FreeBSD 10 and 11? I'd like to know if mrsas can > make it work properly. > > Also, I am wondering. Is it possible that syspd support in the driver may have caused trouble? I was considering purging it and > try again. > > > Thanks! > > > > > Borja. > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 10:35:52 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90797ACD for ; Fri, 28 Feb 2014 10:35:52 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF181AE1 for ; Fri, 28 Feb 2014 10:35:51 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 986719DCEB2; Fri, 28 Feb 2014 11:35:50 +0100 (CET) Subject: Re: ServeRAID M5210e passthroughand syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos X-Priority: 3 In-Reply-To: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> Date: Fri, 28 Feb 2014 11:35:49 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> To: "Steven Hartland" X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 10:35:52 -0000 On Feb 28, 2014, at 11:17 AM, Steven Hartland wrote: > mfi is a raid card it doesn't really have a true passthrough mfisyspd = is > the best you'll get. I'd recommend you switch to an HBA card which = uses > mps driver instead. An example: History for 'zfspool': 2007-12-18.11:21:03 zpool create zfspool raidz2 /dev/da0s1d /dev/da1s1d = /dev/da2s1d /dev/da3s1d /dev/da4s1d /dev/da5s1d /dev/da6s1d /dev/da7s1d 2007-12-18.11:23:15 zfs create zfspool/root # mfiutil show adapter mfi0 Adapter: Product Name: PERC 5/i Integrated Serial Number: 12345 Firmware: 5.2.1-0067 RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID10, RAID50 Battery Backup: present NVRAM: 32K Onboard Memory: 256M Minimum Stripe: 8K Maximum Stripe: 128K # camcontrol devlist at scbus0 target 0 lun 0 (da0,pass0) at scbus0 target 1 lun 0 (da1,pass1) at scbus0 target 2 lun 0 (da2,pass2) at scbus0 target 3 lun 0 (da3,pass3) at scbus0 target 4 lun 0 (da4,pass4) at scbus0 target 5 lun 0 (da5,pass5) at scbus0 target 6 lun 0 (da6,pass6) at scbus0 target 7 lun 0 (da7,pass7) at scbus0 target 8 lun 0 (ses0,pass8) From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 10:42:05 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B63C0EB3 for ; Fri, 28 Feb 2014 10:42:05 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 762611BCF for ; Fri, 28 Feb 2014 10:42:05 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id BAD159DCFDC; Fri, 28 Feb 2014 11:32:04 +0100 (CET) Subject: Re: ServeRAID M5210e passthroughand syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos X-Priority: 3 In-Reply-To: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> Date: Fri, 28 Feb 2014 11:32:03 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> To: "Steven Hartland" X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 10:42:05 -0000 On Feb 28, 2014, at 11:17 AM, Steven Hartland wrote: > mfi is a raid card it doesn't really have a true passthrough mfisyspd = is > the best you'll get. I'd recommend you switch to an HBA card which = uses > mps driver instead. I know, unfortunately this brain dead (unless proven otherwise) IBM = machine doesn't seem to allow me to use a proper HBA. That said, since I asked for assistance in 2007 and Scott Long explained = how to turn an aac card into a passthrough device I have machines running like a charm. I am sure I can trust ZFS to = detect any problems due to that setup. > When you said you're seeing mfisyspd corruption what specifically are > you seeing? I described some of the problems several days ago, in short, either = using ZFS or UFS on one of those "mfisyspd" devices or a "da" created by = allowing passthrough to work and running a benchmark, I get data corruption. No = data corruption happens if I create a "raid 0" device on one disk.=20 I've tried with Samsung and OCZ SSDs and, just wondering if it might be = a SSD specific problem of some sort, a Seagate disk. No luck. With so many manufacturers (even Sun!) tending including these = controllers on their motherboards, and even refusing to sell alternative = options (who wants a "software RAID" when you can get "a hardware one", = they say) I think that it would be good to make sure that passthrough = mode is supported in the best possible way. Maybe we could lobby the LSI Overlords, the manufacturers, both, or just = try to make the best possible use of the most commonly found hardware. Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 10:45:51 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2CC8DF74 for ; Fri, 28 Feb 2014 10:45:51 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD63E1BFD for ; Fri, 28 Feb 2014 10:45:50 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50008536572.msg for ; Fri, 28 Feb 2014 10:45:49 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 28 Feb 2014 10:45:49 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1136e49979=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: From: "Steven Hartland" To: "Borja Marcos" References: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> Subject: Re: ServeRAID M5210e passthroughand syspd corruption Date: Fri, 28 Feb 2014 10:45:51 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 10:45:51 -0000 ----- Original Message ----- From: "Borja Marcos" To: "Steven Hartland" Cc: Sent: Friday, February 28, 2014 10:32 AM Subject: Re: ServeRAID M5210e passthroughand syspd corruption On Feb 28, 2014, at 11:17 AM, Steven Hartland wrote: > mfi is a raid card it doesn't really have a true passthrough mfisyspd is > the best you'll get. I'd recommend you switch to an HBA card which uses > mps driver instead. > I know, unfortunately this brain dead (unless proven otherwise) IBM > machine doesn't seem to allow me to use a proper HBA. Not PCI slots at all then? > That said, since I asked for assistance in 2007 and Scott Long explained > how to turn an aac card into a passthrough device I have machines running > like a charm. I am sure I can trust ZFS to detect any problems due to that > setup. Some older LSI controllers (both these are LSI underneath) had both a RAID and HBA firmware however new cards don't have this option. > When you said you're seeing mfisyspd corruption what specifically are > you seeing? > I described some of the problems several days ago, in short, either using > ZFS or UFS on one of those "mfisyspd" devices or a "da" created by allowing > passthrough to work and running a benchmark, I get data corruption. No data > corruption happens if I create a "raid 0" device on one disk. Specifics on this would be interesting, as there used be corruption issues with > 2TB disks on MFI but we fixed that quite some time ago. > I've tried with Samsung and OCZ SSDs and, just wondering if it might be a > SSD specific problem of some sort, a Seagate disk. No luck. We have a few machines in production here using MFI in passthrough via mfisyspd0 on ZFS and haven't seen any issues. In our case its a ThunderBolt not a Invader generation card so if could be there's a problem with Invader chipset support. > With so many manufacturers (even Sun!) tending including these controllers > on their motherboards, and even refusing to sell alternative options (who > wants a "software RAID" when you can get "a hardware one", they say) I > think that it would be good to make sure that passthrough mode is supported > in the best possible way. We just ensure we go with HBA card if required. > Maybe we could lobby the LSI Overlords, the manufacturers, both, or just > try to make the best possible use of the most commonly found hardware. Can't say we've had an issue, Dell, Supermicro all have options for HBA you just need to know the model numbers, which can be quite confusing. One chassis to avoid is the Dell C6220 series as it fails on 6Gbps speeds due to bad wiring. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 10:48:02 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 69312E4 for ; Fri, 28 Feb 2014 10:48:02 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 04FFD1C11 for ; Fri, 28 Feb 2014 10:48:00 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50008536580.msg for ; Fri, 28 Feb 2014 10:47:59 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 28 Feb 2014 10:47:59 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1136e49979=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-scsi@freebsd.org Message-ID: <1EC15BB093164EAF89B9FBA6D0D14EED@multiplay.co.uk> From: "Steven Hartland" To: "Borja Marcos" References: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> Subject: Re: ServeRAID M5210e passthroughand syspd corruption Date: Fri, 28 Feb 2014 10:48:00 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 10:48:02 -0000 Also make sure you have the latest FW as there's a known issue with older versions causing data corruption with WS commands on older versions. ----- Original Message ----- From: "Borja Marcos" To: "Steven Hartland" Cc: Sent: Friday, February 28, 2014 10:35 AM Subject: Re: ServeRAID M5210e passthroughand syspd corruption On Feb 28, 2014, at 11:17 AM, Steven Hartland wrote: > mfi is a raid card it doesn't really have a true passthrough mfisyspd is > the best you'll get. I'd recommend you switch to an HBA card which uses > mps driver instead. An example: History for 'zfspool': 2007-12-18.11:21:03 zpool create zfspool raidz2 /dev/da0s1d /dev/da1s1d /dev/da2s1d /dev/da3s1d /dev/da4s1d /dev/da5s1d /dev/da6s1d /dev/da7s1d 2007-12-18.11:23:15 zfs create zfspool/root # mfiutil show adapter mfi0 Adapter: Product Name: PERC 5/i Integrated Serial Number: 12345 Firmware: 5.2.1-0067 RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID10, RAID50 Battery Backup: present NVRAM: 32K Onboard Memory: 256M Minimum Stripe: 8K Maximum Stripe: 128K # camcontrol devlist at scbus0 target 0 lun 0 (da0,pass0) at scbus0 target 1 lun 0 (da1,pass1) at scbus0 target 2 lun 0 (da2,pass2) at scbus0 target 3 lun 0 (da3,pass3) at scbus0 target 4 lun 0 (da4,pass4) at scbus0 target 5 lun 0 (da5,pass5) at scbus0 target 6 lun 0 (da6,pass6) at scbus0 target 7 lun 0 (da7,pass7) at scbus0 target 8 lun 0 (ses0,pass8) ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 11:02:34 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CFD444E1 for ; Fri, 28 Feb 2014 11:02:34 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 55AF51D79 for ; Fri, 28 Feb 2014 11:02:33 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id AA7779DD006; Fri, 28 Feb 2014 12:02:32 +0100 (CET) Subject: Re: ServeRAID M5210e passthroughand syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos X-Priority: 3 In-Reply-To: Date: Fri, 28 Feb 2014 12:02:31 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <886C951B-AB52-44C6-A9CB-955222C601DA@sarenet.es> References: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> To: "Steven Hartland" X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 11:02:35 -0000 On Feb 28, 2014, at 11:45 AM, Steven Hartland wrote: >> I know, unfortunately this brain dead (unless proven otherwise) IBM >> machine doesn't seem to allow me to use a proper HBA. >=20 > Not PCI slots at all then? No, it was purchased in a funny configuration. Well, would be great if = it worked. The PCIe raiser card which would offer three slots is replaced by a = contraption with two disks and a LSI HBA (that's one of the "mps" ones but just with one = connector). The front of the machine has 24 slots for 2.5" disks. IBM makes them so = that you can use the two "back" disks for the OS, with a simple mirroring support, and = all of the front mounted disks for storage.=20 The concept is nice, *but* if only the "Invader" (that's indeed a proper = name!) could behave like a good HBA... >=20 >> That said, since I asked for assistance in 2007 and Scott Long = explained >> how to turn an aac card into a passthrough device I have machines = running >> like a charm. I am sure I can trust ZFS to detect any problems due to = that >> setup. >=20 > Some older LSI controllers (both these are LSI underneath) had both a = RAID > and HBA firmware however new cards don't have this option. I've done it with aac cards with the same results. Seems the bad apple = is this "Invader" thingy. >> When you said you're seeing mfisyspd corruption what specifically are >> you seeing? >=20 >> I described some of the problems several days ago, in short, either = using >> ZFS or UFS on one of those "mfisyspd" devices or a "da" created by = allowing >> passthrough to work and running a benchmark, I get data corruption. = No data >> corruption happens if I create a "raid 0" device on one disk.=20 >=20 > Specifics on this would be interesting, as there used be corruption = issues > with > 2TB disks on MFI but we fixed that quite some time ago. All of the disks are smaller than 2 TB. The exact models involved are: Samsung SSD 840 BB0Q (1 TB) OCZ-VERTEX4 1.5 (500 MB) and for the hard disk, a Dell branded Seagate (ST9146803SS) >> I've tried with Samsung and OCZ SSDs and, just wondering if it might = be a >> SSD specific problem of some sort, a Seagate disk. No luck. >=20 > We have a few machines in production here using MFI in passthrough via > mfisyspd0 on ZFS and haven't seen any issues. In our case its a = ThunderBolt > not a Invader generation card so if could be there's a problem with > Invader chipset support. The problem with mfisyspd (anyway, the Invader is corrupting using da = and mfisyspd) is that, as far as I know, it won't support TRIM nor quircks (such as 4K blocks) for = SSDs. Anyway, as I said, neither the syspd nor the "brute force" approach are = working) >> With so many manufacturers (even Sun!) tending including these = controllers >> on their motherboards, and even refusing to sell alternative options = (who >> wants a "software RAID" when you can get "a hardware one", they say) = I >> think that it would be good to make sure that passthrough mode is = supported >> in the best possible way. >=20 > We just ensure we go with HBA card if required. Yes, I think the solution will be to kick "turnkey, Windows oriented" = manufacturers such as Dell and IBM and specify the servers to the component level. Again, from my = experience, it can be quite a pain.=20 >> Maybe we could lobby the LSI Overlords, the manufacturers, both, or = just >> try to make the best possible use of the most commonly found = hardware. >=20 >=20 > Can't say we've had an issue, Dell, Supermicro all have options for = HBA > you just need to know the model numbers, which can be quite confusing. Of course. We've had some problems with Dell, though. I remember telling = a Dell sales rep to just shut up and serve the card I have ordered, period. = Sigh. > One chassis to avoid is the Dell C6220 series as it fails on 6Gbps = speeds > due to bad wiring. Thanks, useful information!! Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 11:04:28 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 41676566 for ; Fri, 28 Feb 2014 11:04:28 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id F3F861D9B for ; Fri, 28 Feb 2014 11:04:27 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id EBB1B9DC5FD; Fri, 28 Feb 2014 12:04:26 +0100 (CET) Subject: Re: ServeRAID M5210e passthroughand syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos X-Priority: 3 In-Reply-To: <1EC15BB093164EAF89B9FBA6D0D14EED@multiplay.co.uk> Date: Fri, 28 Feb 2014 12:04:25 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <0B1A8780A28942B78E8FA4E3CE5EAE3E@multiplay.co.uk> <1EC15BB093164EAF89B9FBA6D0D14EED@multiplay.co.uk> To: "Steven Hartland" X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 11:04:28 -0000 On Feb 28, 2014, at 11:48 AM, Steven Hartland wrote: > Also make sure you have the latest FW as there's a known issue with > older versions causing data corruption with WS commands on older > versions. Another problem is, this is a very turnkey thing sold by IBM. It's so = crazy I've been unable to boot the machine if I disable the motherboard "Invader"and install a Dell branded H200 (the "mps" HBA I've = got) instead. So I guess I can cause a disaster if I begin to fiddle with non IBM supplied firmware. Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 13:50:06 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3646FB2B for ; Fri, 28 Feb 2014 13:50:06 +0000 (UTC) Received: from mail-vc0-x235.google.com (mail-vc0-x235.google.com [IPv6:2607:f8b0:400c:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E7CCA1CCF for ; Fri, 28 Feb 2014 13:50:05 +0000 (UTC) Received: by mail-vc0-f181.google.com with SMTP id lg15so713422vcb.40 for ; Fri, 28 Feb 2014 05:50:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=sHsR2W45xZm0d+S0t/5J3kB9RYEfMXn3plJFVJvikeY=; b=l7d/oMp/5R4T+b0Y2aaBgWj/V+P1T4m+nGH/jo4KMAhXxG/yKtrAaQoHDfqNE9bQOU 5drCSVxKglxMT6YtSiAbARfwdtCT8ICWg7yKYRZANZjyVrduyb7sUy4geGQ/AWNGw2y+ Aza7iWv5WtBiK77qMSlQneROMF9YQcAfB8cGHeekRGuzwMdiTT+68tMhKyXqjxfoUyJ7 TURGK/7nDfPm77WDaVpLQ8pn/AF2Qsiqcp2d1uSJfSDcgDirQW1qhHW95se/lcGK7YED hHlvQNXKJHWs9NkL4nnhVQxZVauCUtSkkOwnyBGKnCsUCdZxBLPHvsLCtn056jn3UNiI 0wEA== MIME-Version: 1.0 X-Received: by 10.52.191.9 with SMTP id gu9mr5423177vdc.37.1393595404968; Fri, 28 Feb 2014 05:50:04 -0800 (PST) Sender: markjdb@gmail.com Received: by 10.220.162.68 with HTTP; Fri, 28 Feb 2014 05:50:04 -0800 (PST) In-Reply-To: References: Date: Fri, 28 Feb 2014 08:50:04 -0500 X-Google-Sender-Auth: 1OVSiFI529AOtBUSJ10qkG4_v0M Message-ID: Subject: Re: ServeRAID M5210e passthroughand syspd corruption From: Mark Johnston To: Borja Marcos Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 13:50:06 -0000 On Fri, Feb 28, 2014 at 5:06 AM, Borja Marcos wrote: > > Hello, > > I'm trying to make this card work. As I am going to use ZFS, I need it in passthrough mode. However, both in 10-STABLE and 11-CURRENT > it's corrupting data. Hello, Which revision of 11-CURRENT have you tried? In particular, do you have r261535? It contains some changes which affect Invader cards; I have no idea if they're connected to the corruption you're seeing, but they were needed in order to prevent firmware crashes on Fury cards. The mrsas driver is also available for download from the LSI product pages, e.g. under "software downloads" here: http://www.lsi.com/products/raid-controllers/pages/megaraid-sas-9341-4i.aspx -Mark > > The card is detected as follows: > mfi0: port 0x4f00-0x4fff mem 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on pci22 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > mfi0: FW MaxCmds = 240, limiting to 128 > mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0 > mfi0: 10016 (446833719s/0x0020/info) - Shutdown command received from host > mfi0: 10017 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/045b/1014) > mfi0: 10018 (boot + 10s/0x0020/info) - Firmware version 4.200.21-2840 > mfi0: 10019 (boot + 12s/0x0020/info) - Package version 24.0.2-0013 > mfi0: 10020 (boot + 12s/0x0020/info) - Board Revision 00AL055 > mfi0: 10021 (boot + 30s/0x0004/info) - Enclosure (SES) discovered on PD 08(c Port 0 - 3/p1) > mfi0: 10022 (boot + 30s/0x0004/info) - Enclosure (SES) discovered on PD 0c(c Port 0 - 3/p2) > mfi0: 10023 (boot + 30s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) communication restored > mfi0: 10024 (boot + 30s/0x0004/info) - Enclosure PD 0c(c Port 0 - 3/p2) communication restored > mfi0: 10025 (boot + 30s/0x0002/info) - Inserted: Encl PD 08 > mfisyspd0 on mfi0 > > And according to mfiutil, > root@merde:/home/borjam # mfiutil show adapter > mfi0 Adapter: > Product Name: ServeRAID M5210e > Serial Number: XXXXX > Firmware: 24.0.2-0013 > RAID Levels: JBOD, RAID0, RAID1, RAID10 > Battery Backup: not present > NVRAM: 32K > Onboard Memory: 0M > Minimum Stripe: 64K > Maximum Stripe: 64K > > > Curiously, other mfi cards which don't officially support syspd mode work in passthrough without problems. And indeed > mfisyspd is not adequate to use SSDs. It's much better to use passthrough and the "da" driver. > > Is there any download I can try for the "mrsas" driver to test under FreeBSD 10 and 11? I'd like to know if mrsas can > make it work properly. > > Also, I am wondering. Is it possible that syspd support in the driver may have caused trouble? I was considering purging it and > try again. > > > Thanks! > > > > > Borja. > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Fri Feb 28 15:02:02 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 226115F2; Fri, 28 Feb 2014 15:02:02 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id D5FCA151A; Fri, 28 Feb 2014 15:02:01 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id E383D9DCFD1; Fri, 28 Feb 2014 16:01:59 +0100 (CET) Subject: Re: ServeRAID M5210e passthroughand syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos In-Reply-To: Date: Fri, 28 Feb 2014 16:01:58 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <1371C416-5017-4F44-B629-170B06D01AD7@sarenet.es> References: To: Mark Johnston X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2014 15:02:02 -0000 On Feb 28, 2014, at 2:50 PM, Mark Johnston wrote: > Which revision of 11-CURRENT have you tried? In particular, do you > have r261535? It contains some changes which affect Invader cards; I > have no idea if they're connected to the corruption you're seeing, but > they were needed in order to prevent firmware crashes on Fury cards. Aha. I just tried with a snapshot downloaded from ftp.freebsd.org. I = just wanted to check if I saw the same corruption and I=20 didn't check out a more recent one. It-s r261642. I will check out a more recent one and try again. > The mrsas driver is also available for download from the LSI product > pages, e.g. under "software downloads" here: > = http://www.lsi.com/products/raid-controllers/pages/megaraid-sas-9341-4i.as= px Thank you, something else to try! Cheers, Borja. From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 3 11:06:52 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7162CEE1 for ; Mon, 3 Mar 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5E23D954 for ; Mon, 3 Mar 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s23B6qXP008651 for ; Mon, 3 Mar 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s23B6pHM008649 for freebsd-scsi@FreeBSD.org; Mon, 3 Mar 2014 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 3 Mar 2014 11:06:51 GMT Message-Id: <201403031106.s23B6pHM008649@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Mar 2014 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 17 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 4 14:27:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4E88D2C2 for ; Tue, 4 Mar 2014 14:27:00 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id 0C784C01 for ; Tue, 4 Mar 2014 14:26:59 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 015059DD12C for ; Tue, 4 Mar 2014 15:26:51 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Mystery IBM M1015 Date: Tue, 4 Mar 2014 15:26:50 +0100 Message-Id: <5C3B4F3E-2741-461C-ABFA-556AB004548A@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Mar 2014 14:27:00 -0000 Hi, After buying an IBM ServeRAID M1015 I am greeted by this wonderful = surprise: Shouldn't it be detected by mps instead of mfi?? Does anyone know if IBM = has changed the controller keeping the card name?=20 The system is FreeBSD-CURRENT. FreeBSD prueba 11.0-CURRENT FreeBSD 11.0-CURRENT #0: Fri Feb 28 18:43:03 = CET 2014 root@prueba:/usr/obj/usr/src/sys/GENERIC amd64 root@prueba:/tarari/bench # mfiutil -u 0 show adapter mfi0 Adapter: Product Name: ServeRAID M1015 SAS/SATA Controller Serial Number: SP40271422 Firmware: 20.10.1-0101 RAID Levels: JBOD, RAID0, RAID1, RAID10 Battery Backup: not present NVRAM: 32K Onboard Memory: 0M Minimum Stripe: 8K Maximum Stripe: 64K root@mierda:/tarari/bench #=20 From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 4 14:53:02 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B688AC46 for ; Tue, 4 Mar 2014 14:53:02 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 73781EE9 for ; Tue, 4 Mar 2014 14:53:02 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id ACDD69DD1D8 for ; Tue, 4 Mar 2014 15:52:54 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1283) Subject: Re: Mystery IBM M1015 From: Borja Marcos In-Reply-To: <5C3B4F3E-2741-461C-ABFA-556AB004548A@sarenet.es> Date: Tue, 4 Mar 2014 15:52:53 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <1B573147-A8A8-4FF9-9A65-6AB6667B4674@sarenet.es> References: <5C3B4F3E-2741-461C-ABFA-556AB004548A@sarenet.es> To: freebsd-scsi@freebsd.org X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Mar 2014 14:53:02 -0000 On Mar 4, 2014, at 3:26 PM, Borja Marcos wrote: >=20 > Hi, >=20 > After buying an IBM ServeRAID M1015 I am greeted by this wonderful = surprise: >=20 > Shouldn't it be detected by mps instead of mfi?? Does anyone know if = IBM has changed the > controller keeping the card name?=20 Nevermind, sorry, I should have flashed it :/ My apologies. Borja. From owner-freebsd-scsi@FreeBSD.ORG Wed Mar 5 12:37:24 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 803A566D; Wed, 5 Mar 2014 12:37:24 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id 3F03B296; Wed, 5 Mar 2014 12:37:23 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 3A9EF9DD099; Wed, 5 Mar 2014 13:37:22 +0100 (CET) Subject: Re: FreeBSD 10, ServeRAID M5210e, syspd corruption Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: Date: Wed, 5 Mar 2014 13:37:18 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <0D37534C-8AD5-46D5-8043-8D662370FF7C@sarenet.es> References: To: Tom Evans X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org, Stable Stable X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Mar 2014 12:37:24 -0000 On Feb 14, 2014, at 2:44 PM, Tom Evans wrote: > On Fri, Feb 14, 2014 at 12:44 PM, Borja Marcos = wrote: >> I am configuring an IBM server with FreeBSD 10-RELEASE, a ServeRAID = M5210e and 23 SSD disks. >>=20 >=20 > I'm afraid I have no solution to offer you for this issue, but with > this setup an mps(8) card (LSI SAS2008 and similar) in IT (passthru) > mode would work excellently. >=20 > Maybe easier to change the card than struggle to get it to do > something it doesn't want to? Finally I purchased an IBM M1015, reflashed it to the latest LSI "IT" = firmware, and I'll use the "Invader" card just to boot from the two back = mounted disks in mirror. Indeed, LSI2008 cards are the way to go. Unfortunately, cross-flashing = them feels a bit kludgy although admittedly I've done even dirtier stuff = :) Thanks! Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Mar 6 11:29:09 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B5FAAF01 for ; Thu, 6 Mar 2014 11:29:09 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 786273DD for ; Thu, 6 Mar 2014 11:29:09 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 6965A9DD452 for ; Thu, 6 Mar 2014 12:29:02 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: ServeRAID M5210e (LSI Invader) won't boot and mount root filesystem on 10-STABLE Date: Thu, 6 Mar 2014 12:28:58 +0100 Message-Id: To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Mar 2014 11:29:09 -0000 I just filled out a PR. http://www.freebsd.org/cgi/query-pr.cgi?pr=3D187312 11-CURRENT works, and a 10-STABLE kernel compiled with three files = copied from CURRENT (mfi_tbolt.c, mfi_pci.c, mfivar.h) works. Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Mar 6 19:01:13 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B1D294C for ; Thu, 6 Mar 2014 19:01:13 +0000 (UTC) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 09902CDB for ; Thu, 6 Mar 2014 19:01:12 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id oz11so3066182veb.6 for ; Thu, 06 Mar 2014 11:01:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/HCbg6wyrUabT2Qx4sfOqpIiB8VVKVn/hW3wDRvmjas=; b=yu5MM0cAXiPmGdTAFMxllAZn/rOUTJSbvmPbRr+x1LPMRzL8V+bp3CwcgWWcneOc9/ 5PSryG7xysQ8XNtN3S/AhxkTrnsbEjbvq16MgycTMyTEmqk1/5nejNDpESVEAVRwtQ1G Co29hVk6kKX7DntLmD4VTKW5nH+CqPhrh6KX3SPB9r3BTvjQFAE3zrkj+ZBNhOml1Vtx 9dUr0Msjj7COK+Ljk2kpVBnqk1B5EhA4q9ZHUOdKg9pyZp/dCsbGNyC2KZNegbseJTKc qpmPNealZLhQpwjiZX9QHu/sr7hq+Ro9vkX47zObSxTrP3I2ukUqC68SeARWzkI95gOJ k+rw== MIME-Version: 1.0 X-Received: by 10.58.100.100 with SMTP id ex4mr6546749veb.2.1394132472217; Thu, 06 Mar 2014 11:01:12 -0800 (PST) Sender: markjdb@gmail.com Received: by 10.220.162.68 with HTTP; Thu, 6 Mar 2014 11:01:12 -0800 (PST) In-Reply-To: References: Date: Thu, 6 Mar 2014 14:01:12 -0500 X-Google-Sender-Auth: O10yv1SwT6xGj-P5AiLvO2yCIPo Message-ID: Subject: Re: ServeRAID M5210e (LSI Invader) won't boot and mount root filesystem on 10-STABLE From: Mark Johnston To: Borja Marcos Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Mar 2014 19:01:13 -0000 On Thu, Mar 6, 2014 at 6:28 AM, Borja Marcos wrote: > > I just filled out a PR. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=187312 > > 11-CURRENT works, and a 10-STABLE kernel compiled with three files copied from CURRENT > (mfi_tbolt.c, mfi_pci.c, mfivar.h) works. Hi Borja, Thanks for the report. I'll merge the change to the stable branches within the next few days. -Mark From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 7 07:55:08 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1DECBD04; Fri, 7 Mar 2014 07:55:08 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id D2509A41; Fri, 7 Mar 2014 07:55:07 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id DC6789DD188; Fri, 7 Mar 2014 08:54:59 +0100 (CET) Subject: Re: ServeRAID M5210e (LSI Invader) won't boot and mount root filesystem on 10-STABLE Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos In-Reply-To: Date: Fri, 7 Mar 2014 08:52:52 +0100 Content-Transfer-Encoding: 7bit Message-Id: References: To: Mark Johnston X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2014 07:55:08 -0000 On Mar 6, 2014, at 8:01 PM, Mark Johnston wrote: > Hi Borja, > > Thanks for the report. I'll merge the change to the stable branches > within the next few days. Great, thank you. If you need me to test anything, just ping. ;) Seems that these Invaders are pesky troublemakers. Borja. From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 10 11:06:53 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 299271C2 for ; Mon, 10 Mar 2014 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 172A781D for ; Mon, 10 Mar 2014 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2AB6qsM043349 for ; Mon, 10 Mar 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2AB6q9d043347 for freebsd-scsi@FreeBSD.org; Mon, 10 Mar 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Mar 2014 11:06:52 GMT Message-Id: <201403101106.s2AB6q9d043347@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Mar 2014 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 17 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 11 16:20:23 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04E61ECA for ; Tue, 11 Mar 2014 16:20:23 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 7DBE6A6B for ; Tue, 11 Mar 2014 16:20:21 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 1FF559DCF1C for ; Tue, 11 Mar 2014 17:20:15 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: LSI SAS HBAs (mps) utility Date: Tue, 11 Mar 2014 17:20:11 +0100 Message-Id: <2336CAA2-7FC0-41EC-AE1D-0592A9E1E8E0@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 16:20:23 -0000 Hi I don't think it has been covered in this list. Searching for the Holy = Grail of disk identification I stumbled upon this: = https://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Ad= apters%20Common%20Files/SAS_SATA_6G_P12/SAS2IRCU_User_Guide.pdf There is a FreeBSD command for the following LSI Logic controllers: LSISAS2004 LSISAS2008 LSISAS2108 LSISAS2208 LSISAS2304 LSISAS2308 The program is called sas2ircu, and it can be downloaded doing a search = for HBAs, LSI SAS9211-8i, all asset types, and it can be found under "Miscellaneous", it's called SAS2IRCU_P18. There are commands to show a list of controllers, show the = configurations, etc. Interestingly, the utility seems to be oriented to "IR" cards, *but* it = can poke my "IT" card without issues. At least it can show me a list of drives, together with the slots where the drives are plugged. = *BUT* it doesn't seem to find the real slots. sigh. Any other interesting utility I should be aware of? Thanks! root@pruebassd:~ # sas2ircu list LSI Corporation SAS2 IR Configuration Utility. Version 18.00.00.00 (2013.11.18)=20 Copyright (c) 2009-2013 LSI Corporation. All rights reserved.=20 Adapter Vendor Device SubSys = SubSys=20 Index Type ID ID Pci Address Ven ID Dev = ID=20 ----- ------------ ------ ------ ----------------- ------ = ------=20 0 SAS2008 1000h 72h 00h:43h:00h:00h 1028h 1f1ch=20= SAS2IRCU: Utility Completed Successfully. root@pruebassd:~ # sas2ircu 0 display LSI Corporation SAS2 IR Configuration Utility. Version 18.00.00.00 (2013.11.18)=20 Copyright (c) 2009-2013 LSI Corporation. All rights reserved.=20 Read configuration has been initiated for controller 0 ------------------------------------------------------------------------ Controller information ------------------------------------------------------------------------ Controller type : SAS2008 BIOS version : 7.35.00.00 Firmware version : 18.00.00.00 Channel description : 1 Serial Attached SCSI Initiator ID : 0 Maximum physical devices : 255 Concurrent commands supported : 3432 Slot : 3 Segment : 0 Bus : 67 Device : 0 Function : 0 RAID Support : No ------------------------------------------------------------------------ IR Volume information ------------------------------------------------------------------------ ------------------------------------------------------------------------ Physical device information ------------------------------------------------------------------------ Initiator at ID #0 Device is a Hard disk Enclosure # : 1 Slot # : 0 SAS Address : 4433221-1-0300-0000 State : Ready (RDY) Size (in MB)/(in sectors) : 488386/1000215215 Manufacturer : ATA =20 Model Number : OCZ-VERTEX4 =20 Firmware Revision : 1.5=20 Serial No : OCZ1NFHW58H59644U7Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 1 Slot # : 1 SAS Address : 4433221-1-0200-0000 State : Ready (RDY) Size (in MB)/(in sectors) : 488386/1000215215 Manufacturer : ATA =20 Model Number : OCZ-VERTEX4 =20 Firmware Revision : 1.5=20 Serial No : OCZ5JAHC862Y945IB7L GUID : N/A Protocol : SATA Drive Type : SATA_SSD From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 11 16:35:03 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7461F1CB for ; Tue, 11 Mar 2014 16:35:03 +0000 (UTC) Received: from mail-we0-x22d.google.com (mail-we0-x22d.google.com [IPv6:2a00:1450:400c:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 10937BD9 for ; Tue, 11 Mar 2014 16:35:02 +0000 (UTC) Received: by mail-we0-f173.google.com with SMTP id w61so10300794wes.18 for ; Tue, 11 Mar 2014 09:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ZeNj2ug47lWuPA2dvld7WCSXbj32ZKzv37csQ+8ueeM=; b=hXf2UXgaSieqTEoBpxmRww7iXA9uBhhSvjXrmqN51LQyF8YekAMmefn+2Qst1xzdJe W2bChdqt4j3GBnTSZil7e6N7+w3sQ6Sy2r9Qxrf1wgjV9VzfCuPk1olbnu4x80JXWeUY w7z6Zk0idafYRthc5XxYNvEHZR31zugLAhYotx1h4UMIQmTC4tV1qZxf9xqAgjMUMBY5 X6kqIm9MtuzWTuloYeQZlKVhevP2xNKSf1wwZLFYkmJstS8xaoU0IL6qpBPYXk0hy99i pzoIkWdUKY4h6w4gpa9JKj3EMEpCOJ8oN4EigoE4/gZfMvLOX7Njdr+XsNUrC98X4KCZ W7cQ== MIME-Version: 1.0 X-Received: by 10.180.97.37 with SMTP id dx5mr3747286wib.53.1394555701234; Tue, 11 Mar 2014 09:35:01 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Tue, 11 Mar 2014 09:35:01 -0700 (PDT) In-Reply-To: <2336CAA2-7FC0-41EC-AE1D-0592A9E1E8E0@sarenet.es> References: <2336CAA2-7FC0-41EC-AE1D-0592A9E1E8E0@sarenet.es> Date: Tue, 11 Mar 2014 10:35:01 -0600 X-Google-Sender-Auth: TtJD9n7HKWw8n0u1H_1FFQ0qr3M Message-ID: Subject: Re: LSI SAS HBAs (mps) utility From: Alan Somers To: Borja Marcos Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 16:35:03 -0000 On Tue, Mar 11, 2014 at 10:20 AM, Borja Marcos wrote: > > Hi > > I don't think it has been covered in this list. Searching for the Holy Grail of disk identification I stumbled upon this: > https://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Adapters%20Common%20Files/SAS_SATA_6G_P12/SAS2IRCU_User_Guide.pdf > > There is a FreeBSD command for the following LSI Logic controllers: > > > LSISAS2004 > LSISAS2008 > LSISAS2108 > LSISAS2208 > LSISAS2304 > LSISAS2308 > > > The program is called sas2ircu, and it can be downloaded doing a search for HBAs, LSI SAS9211-8i, all asset types, > and it can be found under "Miscellaneous", it's called SAS2IRCU_P18. > > There are commands to show a list of controllers, show the configurations, etc. > > Interestingly, the utility seems to be oriented to "IR" cards, *but* it can poke my "IT" card without issues. At least it can show me > a list of drives, together with the slots where the drives are plugged. *BUT* it doesn't seem to find the real slots. sigh. What do you mean, it doesn't find the "real" slots? What are you expecting to see? Do you have a SAS expander in this system? If so, you have some hope of seeing useful information about the physical position of a drive. If not, then you won't get much better than knowing which SAS port is connected to a drive. > > Any other interesting utility I should be aware of? > > Thanks! > > > > > > > root@pruebassd:~ # sas2ircu list > LSI Corporation SAS2 IR Configuration Utility. > Version 18.00.00.00 (2013.11.18) > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > > Adapter Vendor Device SubSys SubSys > Index Type ID ID Pci Address Ven ID Dev ID > ----- ------------ ------ ------ ----------------- ------ ------ > 0 SAS2008 1000h 72h 00h:43h:00h:00h 1028h 1f1ch > SAS2IRCU: Utility Completed Successfully. > root@pruebassd:~ # sas2ircu 0 display > LSI Corporation SAS2 IR Configuration Utility. > Version 18.00.00.00 (2013.11.18) > Copyright (c) 2009-2013 LSI Corporation. All rights reserved. > > Read configuration has been initiated for controller 0 > ------------------------------------------------------------------------ > Controller information > ------------------------------------------------------------------------ > Controller type : SAS2008 > BIOS version : 7.35.00.00 > Firmware version : 18.00.00.00 > Channel description : 1 Serial Attached SCSI > Initiator ID : 0 > Maximum physical devices : 255 > Concurrent commands supported : 3432 > Slot : 3 > Segment : 0 > Bus : 67 > Device : 0 > Function : 0 > RAID Support : No > ------------------------------------------------------------------------ > IR Volume information > ------------------------------------------------------------------------ > ------------------------------------------------------------------------ > Physical device information > ------------------------------------------------------------------------ > Initiator at ID #0 > > Device is a Hard disk > Enclosure # : 1 > Slot # : 0 > SAS Address : 4433221-1-0300-0000 > State : Ready (RDY) > Size (in MB)/(in sectors) : 488386/1000215215 > Manufacturer : ATA > Model Number : OCZ-VERTEX4 > Firmware Revision : 1.5 > Serial No : OCZ1NFHW58H59644U7Y > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > Device is a Hard disk > Enclosure # : 1 > Slot # : 1 > SAS Address : 4433221-1-0200-0000 > State : Ready (RDY) > Size (in MB)/(in sectors) : 488386/1000215215 > Manufacturer : ATA > Model Number : OCZ-VERTEX4 > Firmware Revision : 1.5 > Serial No : OCZ5JAHC862Y945IB7L > GUID : N/A > Protocol : SATA > Drive Type : SATA_SSD > > > > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 11 16:45:56 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3142F656; Tue, 11 Mar 2014 16:45:56 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id E4F04CD6; Tue, 11 Mar 2014 16:45:55 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 8B1D99DCF4F; Tue, 11 Mar 2014 17:45:47 +0100 (CET) Subject: Re: LSI SAS HBAs (mps) utility Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos In-Reply-To: Date: Tue, 11 Mar 2014 17:45:46 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2336CAA2-7FC0-41EC-AE1D-0592A9E1E8E0@sarenet.es> To: Alan Somers X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 16:45:56 -0000 On Mar 11, 2014, at 5:35 PM, Alan Somers wrote: > What do you mean, it doesn't find the "real" slots? What are you > expecting to see? Do you have a SAS expander in this system? If so, > you have some hope of seeing useful information about the physical > position of a drive. If not, then you won't get much better than > knowing which SAS port is connected to a drive. I just notice that, since I flashed the card (A Dell H200) to so-called = "IT Mode" it doesn't show the expanders. This is what I saw before flashing the card: Feb 12 10:01:33 pruebassd kernel: ses0 at mps0 bus 0 scbus0 target 3 lun = 0 Feb 12 10:01:33 pruebassd kernel: ses0: Fixed Enclosure = Services SCSI-5 device=20 Feb 12 10:01:33 pruebassd kernel: ses0: 150.000MB/s transfers Feb 12 10:01:33 pruebassd kernel: ses0: Command Queueing enabled Feb 12 10:01:33 pruebassd kernel: ses0: SCSI-3 ENC Device And now, same hardware, different firmware, I don't see the "ses0" = expander, odd. Anyway, I pulled one of the disks (physical slot 4) and it wasn't the = disk identified by sas2ircu as connected to "1:4" but a different one. That I mean. Chaos... ;) Borja. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 11 16:59:15 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 79D2FCFC; Tue, 11 Mar 2014 16:59:15 +0000 (UTC) Received: from mail-wg0-x232.google.com (mail-wg0-x232.google.com [IPv6:2a00:1450:400c:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E4CEADF7; Tue, 11 Mar 2014 16:59:14 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id x13so10357787wgg.21 for ; Tue, 11 Mar 2014 09:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NjoE1o7y2gIU910340JLkPTdEDzT+HrSqfQi75aZtM4=; b=YHJWaMDyew11d2uPw5GFNrP0V8gLTeKgCLZYocTeYWjFDRIGrIAUpqjd+1SBVF428/ NpwdEIpAngexCT3ebz4INaPK5Nwi0igEoDMp8C8Zb6NNgAX+Ft/EkGtOAWMmjzihm30f UAkW3qr6DJ+qYv9+pv51LznB93RthXatSAsimFkKPCcIQ+Gu/92b4vKS9nTRZulQuBi8 tzGRugMjG7mUfuESncUaPWlnwo3vKNptKVd4KIcRbW92usOCXaXHBl6LuaI5u5Oy2Yb2 0Ao3uKv23u/5IY+IvXyiIzK4MybxlqBj9ytM73WWPg5dhF+73XxgfVuYFrJb//JdpyII Uo9A== MIME-Version: 1.0 X-Received: by 10.194.2.194 with SMTP id 2mr2134418wjw.73.1394557153317; Tue, 11 Mar 2014 09:59:13 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Tue, 11 Mar 2014 09:59:13 -0700 (PDT) In-Reply-To: References: <2336CAA2-7FC0-41EC-AE1D-0592A9E1E8E0@sarenet.es> Date: Tue, 11 Mar 2014 10:59:13 -0600 X-Google-Sender-Auth: Zkl8bDV1YC0Z7vH6YDpGlvhUN94 Message-ID: Subject: Re: LSI SAS HBAs (mps) utility From: Alan Somers To: Borja Marcos Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 16:59:15 -0000 On Tue, Mar 11, 2014 at 10:45 AM, Borja Marcos wrote: > > On Mar 11, 2014, at 5:35 PM, Alan Somers wrote: > >> What do you mean, it doesn't find the "real" slots? What are you >> expecting to see? Do you have a SAS expander in this system? If so, >> you have some hope of seeing useful information about the physical >> position of a drive. If not, then you won't get much better than >> knowing which SAS port is connected to a drive. > > I just notice that, since I flashed the card (A Dell H200) to so-called "IT Mode" it doesn't show the expanders. > > This is what I saw before flashing the card: > Feb 12 10:01:33 pruebassd kernel: ses0 at mps0 bus 0 scbus0 target 3 lun 0 > Feb 12 10:01:33 pruebassd kernel: ses0: Fixed Enclosure Services SCSI-5 device > Feb 12 10:01:33 pruebassd kernel: ses0: 150.000MB/s transfers > Feb 12 10:01:33 pruebassd kernel: ses0: Command Queueing enabled > Feb 12 10:01:33 pruebassd kernel: ses0: SCSI-3 ENC Device > > And now, same hardware, different firmware, I don't see the "ses0" expander, odd. Surprising. What does "camcontrol devlist" show you? > > Anyway, I pulled one of the disks (physical slot 4) and it wasn't the disk identified by sas2ircu as connected to "1:4" but > a different one. What do you mean by "physical slot 4"? The HBA doesn't know anything about physical positions. Even the SES expander probably isn't numbering the slots in the order that you expect. > > That I mean. Chaos... ;) I'm having trouble understanding your English. What does this sentence mean? -Alan > > > > > Borja. > From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 11 17:19:01 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD9237FA; Tue, 11 Mar 2014 17:19:00 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id 630726B; Tue, 11 Mar 2014 17:18:59 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 387929DCF53; Tue, 11 Mar 2014 18:18:59 +0100 (CET) Subject: Re: LSI SAS HBAs (mps) utility Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos In-Reply-To: Date: Tue, 11 Mar 2014 18:18:57 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <497D8998-02B5-434B-AFF2-1A7C6873C3E0@sarenet.es> References: <2336CAA2-7FC0-41EC-AE1D-0592A9E1E8E0@sarenet.es> To: Alan Somers X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2014 17:19:01 -0000 On Mar 11, 2014, at 5:59 PM, Alan Somers wrote: >> And now, same hardware, different firmware, I don't see the "ses0" = expander, odd. >=20 > Surprising. What does "camcontrol devlist" show you? It shows this: root@pruebassd:~ # camcontrol devlist at scbus0 target 0 lun 0 (pass0,da0) at scbus0 target 5 lun 0 (da1,pass1) at scbus0 target 7 lun 0 (pass2,da2) at scbus0 target 9 lun 0 (pass3,da3) at scbus0 target 12 lun 0 (pass4,da4) at scbus0 target 13 lun 0 (pass5,da5) at scbus0 target 15 lun 0 (pass6,da6) at scbus0 target 16 lun 0 (pass7,da7) at scbus2 target 0 lun 0 (da8,pass8) The ses0 device is gone. Full story follows below the "I can't = understand your English". ;) >> Anyway, I pulled one of the disks (physical slot 4) and it wasn't the = disk identified by sas2ircu as connected to "1:4" but >> a different one. >=20 > What do you mean by "physical slot 4"? The HBA doesn't know anything > about physical positions. Even the SES expander probably isn't > numbering the slots in the order that you expect. Yes, that=B4s what I mean. I was hoping to be able to determine the slot = for a given disk, but it's clear that I am wrong. >>=20 >> That I mean. Chaos... ;) >=20 > I'm having trouble understanding your English. What does this = sentence mean? Sorry, I wasn't explicit at all. Full story follows. I'm building a storage server and I'll probably build several ones. It's = based on ZFS of course, and I'm using SSDs for now. Due to the insistence of several manufacturers such as Dell or IBM, I = had several "RAID" cards, which, of course, we don't want in such a configuration. Turns out some of them (the ones sold by Dell = as H200 and the cards sold by IBM as M1015) can be turned into simple HBAs by replacing their firmware by what LSI Logic = calls "IT Mode" firmware.=20 I did it using the latest firmware version I found on the LSI Logic = website. The cards work, the performance is very good, and I don't need to battle "JBOD" pseudo disks and such. The SSDs receive the = TRIM commands perfectly, etc.=20 So far so good, but of course I need to make sure that an operator can = identify a failed disk without confusion. And that's where the fun is. Even being substantially cleaner with the "IT mode" firmware, these = cards are stilll a puzzle to use.=20 What I am looking for is a way to identify a physical disk. I guess I = will end up printing paper labels with the serial numbers, but it would be incredibly clumsy and error prone. Looking at the downloads section = on the LSI Logic website I found this tool, which seems to be a rough = equivalent of mfiutil(8). I didn't see this program mentioned on this list (or I don't remember) = and, as I have seen mentioned that there is no equivalent to mfiutil(8) = for the "mps" cards, well, I thought that it could be useful to someone, = even though it doesn't serve the purpose I hoped to achieve: identifying = the physical slot for a disk. That's why I pointed out that the = "enclosure:slot" information is useless. I hope it's more clear now.;)=20 Sorry if I was so vague. I've been playing with this for two weeks, = trying to find a solid configuration (and, for me, the definition of = "solid" includes "the operator can identify a disk without confusion") = and just now I notice that, after flashing the HBA, the enclosures have = disappeared in two machines :/ Cheers, Borja. From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 15 10:46:54 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 821) id DDABB868; Sat, 15 Mar 2014 10:46:54 +0000 (UTC) Date: Sat, 15 Mar 2014 10:46:54 +0000 From: John To: FreeBSD SCSI List Subject: START STOP UNIT cmd timeout ST3300655SS Message-ID: <20140315104654.GA79659@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Mar 2014 10:46:54 -0000 Hi Folks, I was recently given some slightly older shelves with ST3300655SS drives in them. These shelves came from a working system. I tried to attach them to a 10-stable systems with a 9207-8e card, latest firmware. mps0: port 0x5000-0x50ff mem 0xfadf0000-0xfadfffff,0xfad80000-0xfadbffff irq 32 at device 0.0 on pci6 mps0: Firmware: 18.00.00.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities: 5285c The system never comes up if it is booted with the shelf/drives attached. If I boot the system up and then plug the shelf in I can get: ses0 at mps0 bus 0 scbus2 target 23 lun 0 ses0: Fxied Enclosure Services SCSI-5 device ses0: Command Queueing enabled ses0: SCSI-3 ENC Device (da1:mps0:0:10:0): START STOP UNIT. DCB 1b 00 00 00 01 00 length 0 SMID 249 command timeout on 0xfffffe0000b826d0 ccb 0xfffff803fa1b4000 (noperiph:mps0:0:4294967295:0): SMID 1 Aborting command 0xfffffe0000b826d0 The above continues indefinitely never bringing the drives online. The doc for this drive states the following: If the drive receives a NOTIFY (ENABLE SPINUP) primitive through either port and has not received a START STOP UNIT command with the START bit equal to 0, the drive becomes ready for normal operations within 20 seconds (excluding the error recovery procedure). If the drive receives a START STOP UNIT command with the START bit equal to 0 before receiving a NOTIFY (ENABLE SPINUP) primitive, the drive waits for a START STOP UNIT command with the START bit equal to 1. After receiving a START STOP UNIT command with the START bit equal to 1, the drive waits for a NOTIFY (ENABLE SPINUP) primitive. After receiving a NOTIFY (ENABLE SPINUP) primitive through either port, the drive becomes ready for normal operations within 20 seconds (excluding the error recovery procedure). If the drive receives a START STOP UNIT command with the START bit and IMMED bit equal to 1 and does not receive a NOTIFY (ENABLE SPINUP) primitive within 5 seconds, the drive fails the START STOP UNIT command. The system is timing out, not failing the cmd. ? Thoughts? Thanks, John From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 17 11:06:53 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10AF9AC5 for ; Mon, 17 Mar 2014 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F173E2A6 for ; Mon, 17 Mar 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2HB6qLB011395 for ; Mon, 17 Mar 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2HB6qUw011393 for freebsd-scsi@FreeBSD.org; Mon, 17 Mar 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 17 Mar 2014 11:06:52 GMT Message-Id: <201403171106.s2HB6qUw011393@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2014 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 17 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 18 10:02:08 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21A27716 for ; Tue, 18 Mar 2014 10:02:08 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id D97737EE for ; Tue, 18 Mar 2014 10:02:07 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 689FD9DD515 for ; Tue, 18 Mar 2014 11:02:00 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: SATA on SAS and tags Date: Tue, 18 Mar 2014 11:01:59 +0100 Message-Id: <4C5437F0-E612-4907-8D52-6FCB15CC78E1@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 10:02:08 -0000 Hello, I guess this is a bug. I am using SATA SSDs connecting to a SAS = backplane and SAS controller (LSI2008, IT firmware, mps driver) and I've just noticed that CAM claims a maximum of 255 tags.=20 Shouldn't it be 16 using NCQ instead of TCQ? mps0: port 0xfc00-0xfcff mem = 0xdf2b0000-0xdf2bffff,0xdf2c0000-0xdf2fffff irq 32 at device 0.0 on pci2 mps0: Firmware: 18.00.00.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities: = 1285c= camcontrol devlist # camcontrol devlist at scbus0 target 8 lun 0 (pass0,da0) at scbus0 target 9 lun 0 (pass1,da1) at scbus0 target 10 lun 0 (da2,pass2) at scbus0 target 11 lun 0 (da3,pass3) at scbus0 target 12 lun 0 (da4,pass4) at scbus0 target 13 lun 0 (pass5,da5) at scbus0 target 14 lun 0 (da6,pass6) at scbus0 target 15 lun 0 (da7,pass7) at scbus0 target 17 lun 0 (da8,pass8) at scbus0 target 18 lun 0 (da9,pass9) at scbus0 target 19 lun 0 = (da10,pass10) at scbus0 target 20 lun 0 = (da11,pass11) at scbus0 target 21 lun 0 = (da12,pass12) at scbus0 target 25 lun 0 = (da13,pass13) # camcontrol tags da10 -v (pass10:mps0:0:19:0): dev_openings 255 (pass10:mps0:0:19:0): dev_active 0 (pass10:mps0:0:19:0): devq_openings 255 (pass10:mps0:0:19:0): devq_queued 0 (pass10:mps0:0:19:0): held -63 (pass10:mps0:0:19:0): mintags 2 (pass10:mps0:0:19:0): maxtags 255 Thanks! Borja. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 18 10:37:05 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3FC52154 for ; Tue, 18 Mar 2014 10:37:05 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 03CB2B52 for ; Tue, 18 Mar 2014 10:37:04 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id DE36620E7088A; Tue, 18 Mar 2014 10:36:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.1 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 9195320E70886; Tue, 18 Mar 2014 10:36:47 +0000 (UTC) Message-ID: <1B1689B9E3D94E45963FBE018E25CD64@multiplay.co.uk> From: "Steven Hartland" To: "Borja Marcos" , References: <4C5437F0-E612-4907-8D52-6FCB15CC78E1@sarenet.es> Subject: Re: SATA on SAS and tags Date: Tue, 18 Mar 2014 10:36:40 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 10:37:05 -0000 mps presents its devices as SCSI devices not ATA, using firmware based translation, so thats actually expected. Regards Steve ----- Original Message ----- From: "Borja Marcos" To: Sent: Tuesday, March 18, 2014 10:01 AM Subject: SATA on SAS and tags > > Hello, > > I guess this is a bug. I am using SATA SSDs connecting to a SAS backplane and SAS controller (LSI2008, IT firmware, mps driver) > and I've just noticed that CAM claims a maximum of 255 tags. > > Shouldn't it be 16 using NCQ instead of TCQ? > > > mps0: port 0xfc00-0xfcff mem 0xdf2b0000-0xdf2bffff,0xdf2c0000-0xdf2fffff irq 32 at device 0.0 on pci2 > mps0: Firmware: 18.00.00.00, Driver: 16.00.00.00-fbsd > mps0: IOCCapabilities: 1285c > > camcontrol devlist > # camcontrol devlist > at scbus0 target 8 lun 0 (pass0,da0) > at scbus0 target 9 lun 0 (pass1,da1) > at scbus0 target 10 lun 0 (da2,pass2) > at scbus0 target 11 lun 0 (da3,pass3) > at scbus0 target 12 lun 0 (da4,pass4) > at scbus0 target 13 lun 0 (pass5,da5) > at scbus0 target 14 lun 0 (da6,pass6) > at scbus0 target 15 lun 0 (da7,pass7) > at scbus0 target 17 lun 0 (da8,pass8) > at scbus0 target 18 lun 0 (da9,pass9) > at scbus0 target 19 lun 0 (da10,pass10) > at scbus0 target 20 lun 0 (da11,pass11) > at scbus0 target 21 lun 0 (da12,pass12) > at scbus0 target 25 lun 0 (da13,pass13) > > > # camcontrol tags da10 -v > (pass10:mps0:0:19:0): dev_openings 255 > (pass10:mps0:0:19:0): dev_active 0 > (pass10:mps0:0:19:0): devq_openings 255 > (pass10:mps0:0:19:0): devq_queued 0 > (pass10:mps0:0:19:0): held -63 > (pass10:mps0:0:19:0): mintags 2 > (pass10:mps0:0:19:0): maxtags 255 > > > Thanks! > > > > > > Borja. > > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 18 14:39:42 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C51325C2; Tue, 18 Mar 2014 14:39:42 +0000 (UTC) Received: from mtv-iport-1.cisco.com (mtv-iport-1.cisco.com [173.36.130.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 90BFDA26; Tue, 18 Mar 2014 14:39:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=6989; q=dns/txt; s=iport; t=1395153582; x=1396363182; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=iVxsnu9FXDILRykMhIyELIMizbVTOWMVZbxWYrciXDw=; b=EWwGz1n9ml4KRVaj407jM5ZsRXWP+2cX4jloo7emugdnOFZz7dtacr27 Rlmidts+V3R0Dls1kBZywnFvaEqexNf7AcE28HxbLTlieHV6xh7wHBO8z v53jiEE82XJ7IRsVgFz7uHJ4wyZvgHJ2layBaSovA106h0F6Fra+Av2Mt s=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkUFAAhaKFOrRDoI/2dsb2JhbABagwasSpZqgSQWdIIlAQEBBCcTPwwECxEEAQEKHgcPBTEBCQ4Th2UDENFkF4xKghgHBoMegRQBA4lSjQ2BZgGMaIVIg00 X-IronPort-AV: E=Sophos;i="4.97,678,1389744000"; d="scan'208";a="105109494" Received: from mtv-core-3.cisco.com ([171.68.58.8]) by mtv-iport-1.cisco.com with ESMTP; 18 Mar 2014 14:38:35 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-3.cisco.com (8.14.5/8.14.5) with ESMTP id s2IEcYPj017470 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 18 Mar 2014 14:38:35 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s2IEbdEp066016; Tue, 18 Mar 2014 07:37:39 -0700 (PDT) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s2IEbc0I066015; Tue, 18 Mar 2014 07:37:38 -0700 (PDT) (envelope-from ambrisko) Date: Tue, 18 Mar 2014 07:37:38 -0700 From: Doug Ambrisko To: "Desai, Kashyap" Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140318143738.GA65955@cisco.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Tue, 18 Mar 2014 15:24:18 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2014 14:39:42 -0000 Sounds good. I'll try it out and see how it works. I won't be able to make today's meeting. Thanks, Doug A. On Tue, Mar 18, 2014 at 09:23:23AM +0000, Desai, Kashyap wrote: | Hi, | | Please find the latest patch for upstream commit. | | - This patch include all latest and greatest feature supported by | driver released at LSI external site. | - mfi vs mrsas interopt is resolved using latest commit which uses | tunable " hw.mfi.mrsas_enable". | | man page is still under re-work. I will be posting man | page as a follow up action item. | | Thanks, Kashyap | | > -----Original Message----- | > From: Doug Ambrisko [mailto:ambrisko@ambrisko.com] | > Sent: Saturday, January 25, 2014 12:31 AM | > To: Doug Ambrisko | > Cc: Desai, Kashyap; freebsd-scsi@freebsd.org; scottl@netflix.com; | > sean_bruno@yahoo.com; dwhite@ixsystems.com; jpaetzel@freebsd.org; | > Maloy, Joe; Mankani, Krishnaraddi; McConnell, Stephen; Saxena, Sumit; | > Radford, Adam; Kenneth D. Merry | > Subject: Re: LSI - MR-Fusion controller driver patch and man page | > | > On Fri, Jan 24, 2014 at 10:53:56AM -0800, Doug Ambrisko wrote: | > | On Tue, Jan 07, 2014 at 10:11:39AM -0800, Doug Ambrisko wrote: | > | [snip] | > | | Yes, we can probably make the minimal change to mfi to allow mrsas | > | | to optionally take over. That can probably be done the quickest. | > | | > | Here is the patch I propose to mfi(4) to allow mrsas(4) to optionally | > | take newer cards. | > | > I noticed that this patch is partially incomplete since I didn't have | > FLAGS_MRSAS added to all of the TBOLT ID's. I'll fix that in the commit. | > | > | Index: mfi_pci.c | > | | > ========================================================== | > ========= | > | --- mfi_pci.c (revision 260231) | > | +++ mfi_pci.c (working copy) | > | @@ -112,6 +112,11 @@ | > | SYSCTL_INT(_hw_mfi, OID_AUTO, msi, CTLFLAG_RDTUN, &mfi_msi, 0, | > | "Enable use of MSI interrupts"); | > | | > | +static int mfi_mrsas_enable = 0; | > | +TUNABLE_INT("hw.mfi.mrsas_enable", &mfi_msi); SYSCTL_INT(_hw_mfi, | > | +OID_AUTO, mrsas_enable, CTLFLAG_RDTUN, &mfi_mrsas_enable, | > | + 0, "Allow mrasas to take newer cards"); | > | + | > | struct mfi_ident { | > | uint16_t vendor; | > | uint16_t device; | > | @@ -127,11 +132,11 @@ | > | {0x1000, 0x005b, 0x1028, 0x1f34, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Dell PERC H710P Mini (monolithics)"}, | > | {0x1000, 0x005b, 0x1028, 0x1f35, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Dell PERC H710 Adapter"}, | > | {0x1000, 0x005b, 0x1028, 0x1f37, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Dell PERC H710 Mini (blades)"}, | > | - {0x1000, 0x005b, 0x1028, 0x1f38, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Dell PERC H710 Mini (monolithics)"}, | > | - {0x1000, 0x005b, 0x8086, 0x9265, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Intel (R) RAID Controller RS25DB080"}, | > | - {0x1000, 0x005b, 0x8086, 0x9285, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Intel (R) RAID Controller RS25NB008"}, | > | - {0x1000, 0x005b, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "ThunderBolt"}, | > | - {0x1000, 0x005d, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT, "Invader"}, | > | + {0x1000, 0x005b, 0x1028, 0x1f38, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Dell PERC H710 Mini | > (monolithics)"}, | > | + {0x1000, 0x005b, 0x8086, 0x9265, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Intel (R) RAID Controller | > RS25DB080"}, | > | + {0x1000, 0x005b, 0x8086, 0x9285, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Intel (R) RAID Controller | > RS25NB008"}, | > | + {0x1000, 0x005b, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "ThunderBolt"}, | > | + {0x1000, 0x005d, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | > MFI_FLAGS_TBOLT| | > | +MFI_FLAGS_MRSAS, "Invader"}, | > | {0x1000, 0x0060, 0x1028, 0xffff, MFI_FLAGS_1078, "Dell PERC 6"}, | > | {0x1000, 0x0060, 0xffff, 0xffff, MFI_FLAGS_1078, "LSI MegaSAS | > 1078"}, | > | {0x1000, 0x0071, 0xffff, 0xffff, MFI_FLAGS_SKINNY, "Drake Skinny"}, | > | @@ -178,7 +183,13 @@ | > | | > | if ((id = mfi_find_ident(dev)) != NULL) { | > | device_set_desc(dev, id->desc); | > | - return (BUS_PROBE_DEFAULT); | > | + | > | + /* give priority to mrsas if tunable set */ | > | + TUNABLE_INT_FETCH("hw.mfi.mrsas_enable", | > &mfi_mrsas_enable); | > | + if ((id->flags & MFI_FLAGS_MRSAS) && mfi_mrsas_enable) | > | + return (BUS_PROBE_LOW_PRIORITY); | > | + else | > | + return (BUS_PROBE_DEFAULT); | > | } | > | return (ENXIO); | > | } | > | Index: mfivar.h | > | | > ========================================================== | > ========= | > | --- mfivar.h (revision 260231) | > | +++ mfivar.h (working copy) | > | @@ -199,6 +199,7 @@ | > | #define MFI_FLAGS_GEN2 (1<<6) | > | #define MFI_FLAGS_SKINNY (1<<7) | > | #define MFI_FLAGS_TBOLT (1<<8) | > | +#define MFI_FLAGS_MRSAS (1<<9) | > | // Start: LSIP200113393 | > | bus_dma_tag_t verbuf_h_dmat; | > | bus_dmamap_t verbuf_h_dmamap; | > | | > | This creates a hw.mfi.mrsas_enable tunable to control it. The method | > | via hints wasn't the best since for one the unit index was being | > | abused a non-unit specfic option. It was also a little strange to | > | have mrsas hint be in mfi(4). | > | | > | Then we need a minor change to mrsas.c | > | | > | | > | --- ../mrsas.orig/mrsas.c 2014-01-03 11:30:28.000000000 -0800 | > | +++ ./mrsas.c 2014-01-24 10:43:20.000000000 -0800 | > | @@ -328,25 +328,11 @@ static struct mrsas_ident * mrsas_find_i static | > | int mrsas_probe(device_t dev) { | > | struct mrsas_ident *id; | > | - unsigned int force = 0, ivar; | > | | > | if ((id = mrsas_find_ident(dev)) != NULL) { | > | - if (id->device == 0x005b || id->device == 0x005d) { | > | - resource_int_value("mrsas", 0, "fusion_force", &ivar); | > | - | > | - if (ivar == 0 || ivar == 1) | > | - force = ivar; | > | - | > | - device_set_desc(dev, id->desc); | > | - if (force) | > | - return (BUS_PROBE_DEFAULT); | > | - //return (BUS_PROBE_SPECIFIC); //give priority to MFI driver | > | - else | > | - return (BUS_PROBE_LOW_PRIORITY); | > | - } | > | - else | > | - device_set_desc(dev, id->desc); | > | - return (BUS_PROBE_DEFAULT); | > | + device_set_desc(dev, id->desc); | > | + /* between BUS_PROBE_DEFAULT and BUS_PROBE_LOW_PRIORITY | > */ | > | + return (-30); | > | } | > | return (ENXIO); | > | | > | So that its probe part way between mfi(4) results and then it doesn't | > | have to change. | > | | > | If no one has concerns then I'll check in the mfi(4) change. | > | > Thanks, | > | > Doug A. | From owner-freebsd-scsi@FreeBSD.ORG Thu Mar 20 23:56:35 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19448D48; Thu, 20 Mar 2014 23:56:35 +0000 (UTC) Received: from mtv-iport-1.cisco.com (mtv-iport-1.cisco.com [173.36.130.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D84308CF; Thu, 20 Mar 2014 23:56:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=8896; q=dns/txt; s=iport; t=1395359794; x=1396569394; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=CcqSs1f00Ts1hyz3XTWpKjYWPW/qh1wKX/ZeQDPOmZw=; b=fc4VoqJ4SPnPKTIjtEjCzoW45UboPofvwVQGuJI4SKtJgPuNKdHFBryj Ls3jhNL5/uIGUqKY+JfvXJTCJZn6dlKBaIQ6EF7ouVEQUYK0Fi6VyjANv XREKbFZCgwxyBqhAo2r2eclyV0sUbfoEyADKBkSBgBicYbh/Ju3uAJGtt k=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhoFAON/K1OrRDoG/2dsb2JhbABZgwatA5ZggRMWdIIlAQEBBCcTPwwECxEEAQEKHgcPBTEBCQ4Th2UDENACF4xNghgHBoMegRQBA4lSjQ6BZgGMaIVIg00 X-IronPort-AV: E=Sophos;i="4.97,699,1389744000"; d="scan'208";a="105355420" Received: from mtv-core-1.cisco.com ([171.68.58.6]) by mtv-iport-1.cisco.com with ESMTP; 20 Mar 2014 23:56:27 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-1.cisco.com (8.14.5/8.14.5) with ESMTP id s2KNuRPT008129 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 20 Mar 2014 23:56:27 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s2KNtYaK092908; Thu, 20 Mar 2014 16:55:34 -0700 (PDT) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s2KNtYxg092907; Thu, 20 Mar 2014 16:55:34 -0700 (PDT) (envelope-from ambrisko) Date: Thu, 20 Mar 2014 16:55:34 -0700 From: Doug Ambrisko To: "Desai, Kashyap" Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140320235534.GA92797@cisco.com> References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140318143738.GA65955@cisco.com> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Fri, 21 Mar 2014 00:36:56 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "Kenneth D. Merry" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 23:56:35 -0000 I kicked the tires. I did some tests with our home grown management tool that uses the standard LSI ioctl format and that has worked okay. I got a panic before when recreating a RAID. So things have improved. Doing some performance tests, I seem to have tracked down how it happens and why I didn't see a long time ago. Performance is okay with mfi versus mrasas on ThunderBolt cards (ie. 9266-8i) but when I used an Invader card (9361-8i) read performance is really bad. I'm using 4, 300G, 2.5" SAS disk (Toshiba AL13SEB300) in a RAID 10 configuration. Write performance is fine, read performance is really bad. With mfi performance is fine for read and write. The mrsas write performance is ~302974Kbytes and ~7738Kbytes for read. That is really bad. It can be seen with just a dd if=/dev/da0 of=/dev/zero bs=1m With mfi I see ~364128Kbytes writes and ~214817Kbytes reads. So something is strange. The firmware on card is: Name: LSI MegaRAID 9361-8i Serial Number: SV33505325 Firmware BIOS 6.13.00.1_4.14.05.00_0x06010600 Firmware CTLR 5.01-0005 Firmware APP 4.210.50-3015 Firmware NVDT 3.1310.00-0062 Firmware BTBL 3.00.00.00-0009 which was the lastest when I looks on the LSI web site a month ago or so. Any ideas of how to debug this? I recently started testing with the Invader card so that is why I started seeing performance issues. With ThunderBolt performance is okay and about the same. Thanks, Doug A. On Tue, Mar 18, 2014 at 07:37:38AM -0700, Doug Ambrisko wrote: | Sounds good. I'll try it out and see how it works. | | I won't be able to make today's meeting. | | Thanks, | | Doug A. | | On Tue, Mar 18, 2014 at 09:23:23AM +0000, Desai, Kashyap wrote: | | Hi, | | | | Please find the latest patch for upstream commit. | | | | - This patch include all latest and greatest feature supported by | | driver released at LSI external site. | | - mfi vs mrsas interopt is resolved using latest commit which uses | | tunable " hw.mfi.mrsas_enable". | | | | man page is still under re-work. I will be posting man | | page as a follow up action item. | | | | Thanks, Kashyap | | | | > -----Original Message----- | | > From: Doug Ambrisko [mailto:ambrisko@ambrisko.com] | | > Sent: Saturday, January 25, 2014 12:31 AM | | > To: Doug Ambrisko | | > Cc: Desai, Kashyap; freebsd-scsi@freebsd.org; scottl@netflix.com; | | > sean_bruno@yahoo.com; dwhite@ixsystems.com; jpaetzel@freebsd.org; | | > Maloy, Joe; Mankani, Krishnaraddi; McConnell, Stephen; Saxena, Sumit; | | > Radford, Adam; Kenneth D. Merry | | > Subject: Re: LSI - MR-Fusion controller driver patch and man page | | > | | > On Fri, Jan 24, 2014 at 10:53:56AM -0800, Doug Ambrisko wrote: | | > | On Tue, Jan 07, 2014 at 10:11:39AM -0800, Doug Ambrisko wrote: | | > | [snip] | | > | | Yes, we can probably make the minimal change to mfi to allow mrsas | | > | | to optionally take over. That can probably be done the quickest. | | > | | | > | Here is the patch I propose to mfi(4) to allow mrsas(4) to optionally | | > | take newer cards. | | > | | > I noticed that this patch is partially incomplete since I didn't have | | > FLAGS_MRSAS added to all of the TBOLT ID's. I'll fix that in the commit. | | > | | > | Index: mfi_pci.c | | > | | | > ========================================================== | | > ========= | | > | --- mfi_pci.c (revision 260231) | | > | +++ mfi_pci.c (working copy) | | > | @@ -112,6 +112,11 @@ | | > | SYSCTL_INT(_hw_mfi, OID_AUTO, msi, CTLFLAG_RDTUN, &mfi_msi, 0, | | > | "Enable use of MSI interrupts"); | | > | | | > | +static int mfi_mrsas_enable = 0; | | > | +TUNABLE_INT("hw.mfi.mrsas_enable", &mfi_msi); SYSCTL_INT(_hw_mfi, | | > | +OID_AUTO, mrsas_enable, CTLFLAG_RDTUN, &mfi_mrsas_enable, | | > | + 0, "Allow mrasas to take newer cards"); | | > | + | | > | struct mfi_ident { | | > | uint16_t vendor; | | > | uint16_t device; | | > | @@ -127,11 +132,11 @@ | | > | {0x1000, 0x005b, 0x1028, 0x1f34, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Dell PERC H710P Mini (monolithics)"}, | | > | {0x1000, 0x005b, 0x1028, 0x1f35, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Dell PERC H710 Adapter"}, | | > | {0x1000, 0x005b, 0x1028, 0x1f37, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Dell PERC H710 Mini (blades)"}, | | > | - {0x1000, 0x005b, 0x1028, 0x1f38, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Dell PERC H710 Mini (monolithics)"}, | | > | - {0x1000, 0x005b, 0x8086, 0x9265, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Intel (R) RAID Controller RS25DB080"}, | | > | - {0x1000, 0x005b, 0x8086, 0x9285, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Intel (R) RAID Controller RS25NB008"}, | | > | - {0x1000, 0x005b, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "ThunderBolt"}, | | > | - {0x1000, 0x005d, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT, "Invader"}, | | > | + {0x1000, 0x005b, 0x1028, 0x1f38, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Dell PERC H710 Mini | | > (monolithics)"}, | | > | + {0x1000, 0x005b, 0x8086, 0x9265, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Intel (R) RAID Controller | | > RS25DB080"}, | | > | + {0x1000, 0x005b, 0x8086, 0x9285, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "Intel (R) RAID Controller | | > RS25NB008"}, | | > | + {0x1000, 0x005b, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT| MFI_FLAGS_MRSAS, "ThunderBolt"}, | | > | + {0x1000, 0x005d, 0xffff, 0xffff, MFI_FLAGS_SKINNY| | | > MFI_FLAGS_TBOLT| | | > | +MFI_FLAGS_MRSAS, "Invader"}, | | > | {0x1000, 0x0060, 0x1028, 0xffff, MFI_FLAGS_1078, "Dell PERC 6"}, | | > | {0x1000, 0x0060, 0xffff, 0xffff, MFI_FLAGS_1078, "LSI MegaSAS | | > 1078"}, | | > | {0x1000, 0x0071, 0xffff, 0xffff, MFI_FLAGS_SKINNY, "Drake Skinny"}, | | > | @@ -178,7 +183,13 @@ | | > | | | > | if ((id = mfi_find_ident(dev)) != NULL) { | | > | device_set_desc(dev, id->desc); | | > | - return (BUS_PROBE_DEFAULT); | | > | + | | > | + /* give priority to mrsas if tunable set */ | | > | + TUNABLE_INT_FETCH("hw.mfi.mrsas_enable", | | > &mfi_mrsas_enable); | | > | + if ((id->flags & MFI_FLAGS_MRSAS) && mfi_mrsas_enable) | | > | + return (BUS_PROBE_LOW_PRIORITY); | | > | + else | | > | + return (BUS_PROBE_DEFAULT); | | > | } | | > | return (ENXIO); | | > | } | | > | Index: mfivar.h | | > | | | > ========================================================== | | > ========= | | > | --- mfivar.h (revision 260231) | | > | +++ mfivar.h (working copy) | | > | @@ -199,6 +199,7 @@ | | > | #define MFI_FLAGS_GEN2 (1<<6) | | > | #define MFI_FLAGS_SKINNY (1<<7) | | > | #define MFI_FLAGS_TBOLT (1<<8) | | > | +#define MFI_FLAGS_MRSAS (1<<9) | | > | // Start: LSIP200113393 | | > | bus_dma_tag_t verbuf_h_dmat; | | > | bus_dmamap_t verbuf_h_dmamap; | | > | | | > | This creates a hw.mfi.mrsas_enable tunable to control it. The method | | > | via hints wasn't the best since for one the unit index was being | | > | abused a non-unit specfic option. It was also a little strange to | | > | have mrsas hint be in mfi(4). | | > | | | > | Then we need a minor change to mrsas.c | | > | | | > | | | > | --- ../mrsas.orig/mrsas.c 2014-01-03 11:30:28.000000000 -0800 | | > | +++ ./mrsas.c 2014-01-24 10:43:20.000000000 -0800 | | > | @@ -328,25 +328,11 @@ static struct mrsas_ident * mrsas_find_i static | | > | int mrsas_probe(device_t dev) { | | > | struct mrsas_ident *id; | | > | - unsigned int force = 0, ivar; | | > | | | > | if ((id = mrsas_find_ident(dev)) != NULL) { | | > | - if (id->device == 0x005b || id->device == 0x005d) { | | > | - resource_int_value("mrsas", 0, "fusion_force", &ivar); | | > | - | | > | - if (ivar == 0 || ivar == 1) | | > | - force = ivar; | | > | - | | > | - device_set_desc(dev, id->desc); | | > | - if (force) | | > | - return (BUS_PROBE_DEFAULT); | | > | - //return (BUS_PROBE_SPECIFIC); //give priority to MFI driver | | > | - else | | > | - return (BUS_PROBE_LOW_PRIORITY); | | > | - } | | > | - else | | > | - device_set_desc(dev, id->desc); | | > | - return (BUS_PROBE_DEFAULT); | | > | + device_set_desc(dev, id->desc); | | > | + /* between BUS_PROBE_DEFAULT and BUS_PROBE_LOW_PRIORITY | | > */ | | > | + return (-30); | | > | } | | > | return (ENXIO); | | > | | | > | So that its probe part way between mfi(4) results and then it doesn't | | > | have to change. | | > | | | > | If no one has concerns then I'll check in the mfi(4) change. | | > | | > Thanks, | | > | | > Doug A. | | | | From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 21 08:10:53 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 48251E92; Fri, 21 Mar 2014 08:10:53 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id F041491C; Fri, 21 Mar 2014 08:10:52 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 8A7C29DC58C; Fri, 21 Mar 2014 09:10:43 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140320235534.GA92797@cisco.com> Date: Fri, 21 Mar 2014 09:10:40 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <08ba2a262fba45f687cdd3225f325110@BN1PR07MB247.namprd07.prod.outlook.com> <20140103211449.GA69721@cisco.com> <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> To: Doug Ambrisko X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Fri, 21 Mar 2014 11:32:44 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 08:10:53 -0000 On Mar 21, 2014, at 12:55 AM, Doug Ambrisko wrote: > With mfi I see ~364128Kbytes writes and ~214817Kbytes reads. So > something is strange. The firmware on card is: > Name: LSI MegaRAID 9361-8i > Serial Number: SV33505325 > Firmware BIOS 6.13.00.1_4.14.05.00_0x06010600 > Firmware CTLR 5.01-0005 > Firmware APP 4.210.50-3015 > Firmware NVDT 3.1310.00-0062 > Firmware BTBL 3.00.00.00-0009 > which was the lastest when I looks on the LSI web site a month ago or = so. I've got an Invader here, mfi0: port 0x4f00-0x4fff mem = 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on = pci22 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23=20 mfi0: FW MaxCmds =3D 240, limiting to 128 mfi0: MaxCmd =3D 240, Drv MaxCmd =3D 128, MaxSgl =3D 70, state =3D = 0xb73c00f0 mfi0: 11170 (448037009s/0x0020/info) - Shutdown command received from = host mfi0: 11171 (boot + 10s/0x0020/info) - Firmware initialization started = (PCI ID 005d/1000/045b/1014) mfi0: 11172 (boot + 10s/0x0020/info) - Firmware version 4.200.21-2840 mfi0: 11173 (boot + 12s/0x0020/info) - Package version 24.0.2-0013 mfi0: 11174 (boot + 12s/0x0020/info) - Board Revision 00AL055 let me know if I can help running any tests here. For now I am using it = just to boot and I have installed an LSI2008, as this invader corrupts = data on "syspd" or even plain passthrough disks. The server is not in production, so I have no problems to try drivers, = etc. Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 21 16:10:49 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7202D8BB; Fri, 21 Mar 2014 16:10:49 +0000 (UTC) Received: from mtv-iport-1.cisco.com (mtv-iport-1.cisco.com [173.36.130.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 173ECE82; Fri, 21 Mar 2014 16:10:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=2569; q=dns/txt; s=iport; t=1395418249; x=1396627849; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=SsPQ9JFF6AaSns0Up1BDNYrzSklO+tflYl17TS+cSFc=; b=j9WMXsGnTqrO81dt41Sf/WHd6zo6Yf8/oe6veXg5fILQmiaJXxf4ysoR Fi2KWQ+WgXGk7dPA1rD3BPyzBS1YtoxV9GPLzlOVFWOnmaDP6yJsh7sPc Xc15HjDUNM+rvO4ER4y37mM4dxx6cedX2zHVLMye4535Vl36AqFqDpm1e c=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AikFADpjLFOrRDoJ/2dsb2JhbABZgwbAY4MOgRUWdIImAQEEOj8QCyElDwVJiAvPdxeOageDJIEUAQOJUo52AZIxg00 X-IronPort-AV: E=Sophos;i="4.97,704,1389744000"; d="scan'208";a="105407949" Received: from mtv-core-4.cisco.com ([171.68.58.9]) by mtv-iport-1.cisco.com with ESMTP; 21 Mar 2014 16:10:47 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id s2LGAlWN013686 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 21 Mar 2014 16:10:47 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s2LG9twH099911; Fri, 21 Mar 2014 09:09:55 -0700 (PDT) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s2LG9sZM099910; Fri, 21 Mar 2014 09:09:54 -0700 (PDT) (envelope-from ambrisko) Date: Fri, 21 Mar 2014 09:09:54 -0700 From: Doug Ambrisko To: Borja Marcos Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140321160954.GB99545@cisco.com> References: <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Fri, 21 Mar 2014 16:38:02 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 16:10:49 -0000 On Fri, Mar 21, 2014 at 09:10:40AM +0100, Borja Marcos wrote: | | On Mar 21, 2014, at 12:55 AM, Doug Ambrisko wrote: | | > With mfi I see ~364128Kbytes writes and ~214817Kbytes reads. So | > something is strange. The firmware on card is: | > Name: LSI MegaRAID 9361-8i | > Serial Number: SV33505325 | > Firmware BIOS 6.13.00.1_4.14.05.00_0x06010600 | > Firmware CTLR 5.01-0005 | > Firmware APP 4.210.50-3015 | > Firmware NVDT 3.1310.00-0062 | > Firmware BTBL 3.00.00.00-0009 | > which was the lastest when I looks on the LSI web site a month ago or so. | | I've got an Invader here, | | mfi0: port 0x4f00-0x4fff mem 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on pci22 | mfi0: Using MSI | mfi0: Megaraid SAS driver Ver 4.23 | mfi0: FW MaxCmds = 240, limiting to 128 | mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0 | mfi0: 11170 (448037009s/0x0020/info) - Shutdown command received from host | mfi0: 11171 (boot + 10s/0x0020/info) - Firmware initialization started (PCI ID 005d/1000/045b/1014) | mfi0: 11172 (boot + 10s/0x0020/info) - Firmware version 4.200.21-2840 | mfi0: 11173 (boot + 12s/0x0020/info) - Package version 24.0.2-0013 | mfi0: 11174 (boot + 12s/0x0020/info) - Board Revision 00AL055 | | let me know if I can help running any tests here. For now I am using it | just to boot and I have installed an LSI2008, as this invader corrupts | data on "syspd" or even plain passthrough disks. Do you have a simple test case for that? There were issues in the SCSI command translation which should have been fixed via switching it to use the CAM translation code. Can you try it via a RAID volume of one disk? | The server is not in production, so I have no problems to try drivers, etc. You could try the LSI mrsas and see if the corruption goes away and then we could try to isolate the difference. mrsas does not support CAM passthrough. If both drivers are loaded in the kernel, you can prevent mfi from attaching and let mrsas attach via the tunable: hw.mfi.mrsas_enable=1 The mrsas device come up /dev/da* there are no aliases yet to the mfi one's. Also mfiutil won't work with mrsas but MegaCli/StorCli should. If you want to use mfiutil, you can switch to mfi to set that up and then switch back to mrsas. I need to finish the mrsas compat work so mfiutil can work on it. It's a little more complicated then I hoped. This is under -current. This should run under 9.X by copying mfi etc. to it. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 22 16:52:21 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04E16C8E for ; Sat, 22 Mar 2014 16:52:21 +0000 (UTC) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CAFDCFE3 for ; Sat, 22 Mar 2014 16:52:20 +0000 (UTC) Received: from nyi.unixathome.org (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id 64C6650830 for ; Sat, 22 Mar 2014 16:52:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by nyi.unixathome.org (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W4I_Cksd37yb for ; Sat, 22 Mar 2014 16:52:19 +0000 (UTC) Received: from smtp-auth.unixathome.org (smtp-auth.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) by nyi.unixathome.org (Postfix) with ESMTPSA id CE5D05082E for ; Sat, 22 Mar 2014 16:52:18 +0000 (UTC) From: Dan Langille Content-Type: multipart/signed; boundary="Apple-Mail=_23537D5E-D0D0-48B7-87DE-FEA5916C29B2"; protocol="application/pgp-signature"; micalg=pgp-sha1 Subject: Anyone have a SCSI terminator LVD/SE VHDCI? Message-Id: <3C49CB15-C457-41E4-AF0D-521EAFA34AAE@langille.org> Date: Sat, 22 Mar 2014 12:52:07 -0400 To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) X-Mailer: Apple Mail (2.1874) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Mar 2014 16:52:21 -0000 --Apple-Mail=_23537D5E-D0D0-48B7-87DE-FEA5916C29B2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 I=92m trying to locate a SCSI terminator LVD/SE VHDCI. Got one laying = around, unloved? --=20 Dan Langille - http://langille.org --Apple-Mail=_23537D5E-D0D0-48B7-87DE-FEA5916C29B2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlMtv70ACgkQCgsXFM/7nTx93QCcDXwUjBrQe0FXwmYboZ4+AJ6/ +wsAoIhuqVYVtV+tp7Kfabbpo44NJ1WG =b3Lq -----END PGP SIGNATURE----- --Apple-Mail=_23537D5E-D0D0-48B7-87DE-FEA5916C29B2-- From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 09:54:09 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6116B55 for ; Mon, 24 Mar 2014 09:54:09 +0000 (UTC) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0FC10AC2 for ; Mon, 24 Mar 2014 09:54:08 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 5123012A74B1; Mon, 24 Mar 2014 10:48:16 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 16.6770] X-CRM114-CacheID: sfid-20140324_10481_82B3A6B3 X-CRM114-Status: Good ( pR: 16.6770 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Mon Mar 24 10:48:16 2014 X-DSPAM-Confidence: 0.9963 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 532fff60511141094263958 X-DSPAM-Factors: 27, From*"Nagy, Attila" , 0.00051, FreeBSD, 0.00065, FreeBSD, 0.00065, kernel, 0.00085, cache, 0.00165, cache, 0.00165, I+use, 0.00229, timeout, 0.00277, timeout, 0.00277, disks, 0.00375, disks, 0.00375, fault, 0.00404, fault, 0.00404, 0+10, 0.00437, 0+10, 0.00437, stack, 0.00437, stack, 0.00437, ZFS, 0.00437, ZFS, 0.00437, zfs, 0.00437, initialization, 0.00477, or+just, 0.00525, Received*(japan.t+online.co.hu, 0.00582, Received*(japan.t, 0.00582, mem, 0.00582, Received*online.co.hu, 0.00582, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id 02C3E12A74A3 for ; Mon, 24 Mar 2014 10:48:14 +0100 (CET) Message-ID: <532FFF5E.2010900@fsn.hu> Date: Mon, 24 Mar 2014 10:48:14 +0100 From: "Nagy, Attila" MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Only 0.44 (always) days of uptime with ciss (w/HP SA P812) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 09:54:09 -0000 Hi, I have an HP DL360G7 with a HP SmartArray P812 in it, which crashes exactly (well, some minutes plus or minus, but on the graph it's nearly the same) at 0.44 days of uptime no matter what I do, load the machine until it's so hot, I can't touch it, or just leave it idle. The P812 has an HP MDS600 connected to it with 70 1TB disks, with a 6 disk RAID6 (ADG) setup. The volumes have 128k stripe size, because I use ZFS on top of them. The zpool is simply a stripe of the RAID6 volumes. What may be important: the controller's RAID6 initialization is still ongoing. In the first sentence idle means the zpool/zfs is just mounted and only some stat()s happening on them (crashes after 0.44 days) and fully loaded means gstat shows around 100% utilization on the disks nearly all the time (crashes after 0.44 days also). I've already tried with stable/9@r260621 and stable/10@r262152, it's the same. I've also tried with Linux (Ubuntu 13.10, hpsa driver, zfs on linux 0.6.2), it doesn't crash (neither idle or loaded). Already swapped the machine and the P812 to a different one, no effect. Everything (DL360, P812, MDS600, disks) has the latest firmware. The currently used ZFS is created under Linux to see whether this causes the problems, but of course there are many different things in the two OS (kernel, HP SA driver, block/SCSI layer and even ZFS is somewhat different). Linux works, FreeBSD crashes no matter what I do. The exact message I can see is (ciss0 is the built-in P411): ciss1: ADAPTER HEARTBEAT FAILED Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xfffffe0c59ff795d stack pointer = 0x28:0xfffffe0baf1ab9d0 frame pointer = 0x28:0xfffffe0baf1aba20 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi4: clock) trap number = 1 panic: privileged instruction fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0baf1ab560 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0baf1ab610 panic() at panic+0x155/frame 0xfffffe0baf1ab690 trap_fatal() at trap_fatal+0x3a2/frame 0xfffffe0baf1ab6f0 trap() at trap+0x794/frame 0xfffffe0baf1ab910 calltrap() at calltrap+0x8/frame 0xfffffe0baf1ab910 --- trap 0x1, rip = 0xfffffe0c59ff795d, rsp = 0xfffffe0baf1ab9d0, rbp = 0xfffffe0baf1aba20 --- (null)() at 0xfffffe0c59ff795d/frame 0xfffffe0baf1aba20 softclock_call_cc() at softclock_call_cc+0x16c/frame 0xfffffe0000e77120 kernphys() at 0xffffffff/frame 0xfffffe0000e778a0 kernphys() at 0xffffffff/frame 0xfffffe0000e78aa0 kernphys() at 0xffffffff/frame 0xfffffe0000e78c20 Uptime: 10h18m12s (da4:ciss1:0:0:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da4:ciss1:0:0:0): CAM status: Command timeout (da4:ciss1:0:0:0): Error 5, Retries exhausted (da4:ciss1:0:0:0): Synchronize cache failed (da5:ciss1:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da5:ciss1:0:1:0): CAM status: Command timeout (da5:ciss1:0:1:0): Error 5, Retries exhausted (da5:ciss1:0:1:0): Synchronize cache failed (da6:ciss1:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da6:ciss1:0:2:0): CAM status: Command timeout (da6:ciss1:0:2:0): Error 5, Retries exhausted (da6:ciss1:0:2:0): Synchronize cache failed (da7:ciss1:0:3:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da7:ciss1:0:3:0): CAM status: Command timeout (da7:ciss1:0:3:0): Error 5, Retries exhausted (da7:ciss1:0:3:0): Synchronize cache failed (da8:ciss1:0:4:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da8:ciss1:0:4:0): CAM status: Command timeout (da8:ciss1:0:4:0): Error 5, Retries exhausted (da8:ciss1:0:4:0): Synchronize cache failed (da9:ciss1:0:5:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da9:ciss1:0:5:0): CAM status: Command timeout (da9:ciss1:0:5:0): Error 5, Retries exhausted (da9:ciss1:0:5:0): Synchronize cache failed (da10:ciss1:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da10:ciss1:0:6:0): CAM status: Command timeout (da10:ciss1:0:6:0): Error 5, Retries exhausted (da10:ciss1:0:6:0): Synchronize cache failed (da11:ciss1:0:7:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da11:ciss1:0:7:0): CAM status: Command timeout (da11:ciss1:0:7:0): Error 5, Retries exhausted (da11:ciss1:0:7:0): Synchronize cache failed (da12:ciss1:0:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da12:ciss1:0:8:0): CAM status: Command timeout (da12:ciss1:0:8:0): Error 5, Retries exhausted (da12:ciss1:0:8:0): Synchronize cache failed (da13:ciss1:0:9:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da13:ciss1:0:9:0): CAM status: Command timeout (da13:ciss1:0:9:0): Error 5, Retries exhausted (da13:ciss1:0:9:0): Synchronize cache failed (da14:ciss1:0:10:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da14:ciss1:0:10:0): CAM status: Command timeout (da14:ciss1:0:10:0): Error 5, Retries exhausted (da14:ciss1:0:10:0): Synchronize cache failed Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... Dmesg says: ciss1: port 0x5000-0x50ff mem 0xfbe00000-0xfbffffff,0xfbdf0000-0xfbdf0fff irq 24 at device 0.0 on pci9 ciss1: PERFORMANT Transport da5 at ciss1 bus 0 scbus2 target 1 lun 0 da4 at ciss1 bus 0 scbus2 target 0 lun 0 da6 at ciss1 bus 0 scbus2 target 2 lun 0 da7 at ciss1 bus 0 scbus2 target 3 lun 0 da8 at ciss1 bus 0 scbus2 target 4 lun 0 da9 at ciss1 bus 0 scbus2 target 5 lun 0 da10 at ciss1 bus 0 scbus2 target 6 lun 0 da11 at ciss1 bus 0 scbus2 target 7 lun 0 da12 at ciss1 bus 0 scbus2 target 8 lun 0 da13 at ciss1 bus 0 scbus2 target 9 lun 0 da14 at ciss1 bus 0 scbus2 target 10 lun 0 I also find it interesting that the machine's IML (Integrated Management Log) contains this message after every crash: POST Error: 1719 - A controller failure event occurred prior to this power-up Which might show that the controller indeed locks up, but why does it do this under FreeBSD and doesn't under Linux? I've already tried hw.ciss.nop_message_heartbeat=1;ciss_force_transport=1;ciss_force_interrupt=1 without any effect (it freezes after the same time). Last time during the POST the controller said: Slot 2 HP Smart Array P812 Controller (1024MB, v6.40) 11 Logical Drives 1719-Slot 2 Drive Array - A controller failure event occurred prior to this power-up. (Previous lock up code = 0x13) Any ideas on what could cause this? Thanks, From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 11:06:52 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0AAB3186 for ; Mon, 24 Mar 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EB4CD17F for ; Mon, 24 Mar 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2OB6pYW013981 for ; Mon, 24 Mar 2014 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2OB6plS013979 for freebsd-scsi@FreeBSD.org; Mon, 24 Mar 2014 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Mar 2014 11:06:51 GMT Message-Id: <201403241106.s2OB6plS013979@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 17 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 08:03:54 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF1E8977; Mon, 24 Mar 2014 08:03:54 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 93A9AFA6; Mon, 24 Mar 2014 08:03:54 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id EB89A9DCC3A; Mon, 24 Mar 2014 09:03:46 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140321160954.GB99545@cisco.com> Date: Mon, 24 Mar 2014 09:03:43 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> References: <8c423414ecc2421fbace3eb9f386be91@BN1PR07MB247.namprd07.prod.outlook.com> <20140106182935.GC93278@cisco.com> <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> To: Doug Ambrisko X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Mon, 24 Mar 2014 11:37:08 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 08:03:54 -0000 On Mar 21, 2014, at 5:09 PM, Doug Ambrisko wrote: > Do you have a simple test case for that? There were issues in the = SCSI > command translation which should have been fixed via switching it to = use > the CAM translation code. Can you try it via a RAID volume of one = disk? Yes, I tried all those possibilities. "syspd" devices, passthrough (same = result, corrruption) and when creating single-disk RAID0 volumes it worked, no corruption. Just in case I tried with SSDs (Samsung EVO 840 and OCZ Vertex 4) and = Seagate SAS disks. > | The server is not in production, so I have no problems to try = drivers, etc. >=20 > You could try the LSI mrsas and see if the corruption goes away and = then > we could try to isolate the difference. mrsas does not support CAM > passthrough. If both drivers are loaded in the kernel, you can = prevent > mfi from attaching and let mrsas attach via the tunable: > hw.mfi.mrsas_enable=3D1 > The mrsas device come up /dev/da* there are no aliases yet to the mfi = one's. > Also mfiutil won't work with mrsas but MegaCli/StorCli should. If you = want > to use mfiutil, you can switch to mfi to set that up and then switch = back > to mrsas. I need to finish the mrsas compat work so mfiutil can work > on it. It's a little more complicated then I hoped. Actually I think that converting passthrough disks to "mfisyspd" or an = equivalent is a bad idea, unless, of course, there's a compelling reason I don't realize. For = instance, if you are using SSDs you need access to the TRIM command. So, if I can vote, I vote with arms and legs for "da" devices in case = it's a passthough. It's not a pressing need, I installed a LSI2008 card and I connected the = SAS backplane to it, so for now the Invader is a non-problem, but if it will help to run some tests with the mps = driver I can certainly do it. Borja. From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 12:32:59 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A265192B for ; Mon, 24 Mar 2014 12:32:59 +0000 (UTC) Received: from mail.ignoranthack.me (ujvl.x.rootbsd.net [199.102.79.106]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7EF46DC3 for ; Mon, 24 Mar 2014 12:32:59 +0000 (UTC) Received: from [192.168.1.228] (c-24-6-179-71.hsd1.ca.comcast.net [24.6.179.71]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 67BAA1929C8; Mon, 24 Mar 2014 12:32:56 +0000 (UTC) Subject: Re: Only 0.44 (always) days of uptime with ciss (w/HP SA P812) From: Sean Bruno To: "Nagy, Attila" In-Reply-To: <532FFF5E.2010900@fsn.hu> References: <532FFF5E.2010900@fsn.hu> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-iBA6hzzfyNGWCF1yk9c6" Date: Mon, 24 Mar 2014 05:32:55 -0700 Message-ID: <1395664375.25687.1.camel@powernoodle.corp.yahoo.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: sbruno@freebsd.org List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 12:32:59 -0000 --=-iBA6hzzfyNGWCF1yk9c6 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Mon, 2014-03-24 at 10:48 +0100, Nagy, Attila wrote: > Hi, >=20 > I have an HP DL360G7 with a HP SmartArray P812 in it, which crashes=20 > exactly (well, some minutes plus or minus, but on the graph it's nearly= =20 > the same) at 0.44 days of uptime no matter what I do, load the machine= =20 > until it's so hot, I can't touch it, or just leave it idle. > The P812 has an HP MDS600 connected to it with 70 1TB disks, with a 6=20 > disk RAID6 (ADG) setup. The volumes have 128k stripe size, because I use= =20 > ZFS on top of them. > The zpool is simply a stripe of the RAID6 volumes. > What may be important: the controller's RAID6 initialization is still=20 > ongoing. >=20 > In the first sentence idle means the zpool/zfs is just mounted and only= =20 > some stat()s happening on them (crashes after 0.44 days) and fully=20 > loaded means gstat shows around 100% utilization on the disks nearly all= =20 > the time (crashes after 0.44 days also). >=20 > I've already tried with stable/9@r260621 and stable/10@r262152, it's the= =20 > same. > I've also tried with Linux (Ubuntu 13.10, hpsa driver, zfs on linux=20 > 0.6.2), it doesn't crash (neither idle or loaded). > Already swapped the machine and the P812 to a different one, no effect.= =20 > Everything (DL360, P812, MDS600, disks) has the latest firmware. >=20 > The currently used ZFS is created under Linux to see whether this causes= =20 > the problems, but of course there are many different things in the two= =20 > OS (kernel, HP SA driver, block/SCSI layer and even ZFS is somewhat=20 > different). > Linux works, FreeBSD crashes no matter what I do. >=20 > The exact message I can see is (ciss0 is the built-in P411): > ciss1: ADAPTER HEARTBEAT FAILED >=20 >=20 > Fatal trap 1: privileged instruction fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > instruction pointer =3D 0x20:0xfffffe0c59ff795d > stack pointer =3D 0x28:0xfffffe0baf1ab9d0 > frame pointer =3D 0x28:0xfffffe0baf1aba20 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 12 (swi4: clock) > trap number =3D 1 > panic: privileged instruction fault > cpuid =3D 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame=20 > 0xfffffe0baf1ab560 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0baf1ab610 > panic() at panic+0x155/frame 0xfffffe0baf1ab690 > trap_fatal() at trap_fatal+0x3a2/frame 0xfffffe0baf1ab6f0 > trap() at trap+0x794/frame 0xfffffe0baf1ab910 > calltrap() at calltrap+0x8/frame 0xfffffe0baf1ab910 > --- trap 0x1, rip =3D 0xfffffe0c59ff795d, rsp =3D 0xfffffe0baf1ab9d0, rbp= =3D=20 > 0xfffffe0baf1aba20 --- > (null)() at 0xfffffe0c59ff795d/frame 0xfffffe0baf1aba20 > softclock_call_cc() at softclock_call_cc+0x16c/frame 0xfffffe0000e77120 > kernphys() at 0xffffffff/frame 0xfffffe0000e778a0 > kernphys() at 0xffffffff/frame 0xfffffe0000e78aa0 > kernphys() at 0xffffffff/frame 0xfffffe0000e78c20 > Uptime: 10h18m12s > (da4:ciss1:0:0:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da4:ciss1:0:0:0): CAM status: Command timeout > (da4:ciss1:0:0:0): Error 5, Retries exhausted > (da4:ciss1:0:0:0): Synchronize cache failed > (da5:ciss1:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da5:ciss1:0:1:0): CAM status: Command timeout > (da5:ciss1:0:1:0): Error 5, Retries exhausted > (da5:ciss1:0:1:0): Synchronize cache failed > (da6:ciss1:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da6:ciss1:0:2:0): CAM status: Command timeout > (da6:ciss1:0:2:0): Error 5, Retries exhausted > (da6:ciss1:0:2:0): Synchronize cache failed > (da7:ciss1:0:3:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da7:ciss1:0:3:0): CAM status: Command timeout > (da7:ciss1:0:3:0): Error 5, Retries exhausted > (da7:ciss1:0:3:0): Synchronize cache failed > (da8:ciss1:0:4:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da8:ciss1:0:4:0): CAM status: Command timeout > (da8:ciss1:0:4:0): Error 5, Retries exhausted > (da8:ciss1:0:4:0): Synchronize cache failed > (da9:ciss1:0:5:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da9:ciss1:0:5:0): CAM status: Command timeout > (da9:ciss1:0:5:0): Error 5, Retries exhausted > (da9:ciss1:0:5:0): Synchronize cache failed > (da10:ciss1:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da10:ciss1:0:6:0): CAM status: Command timeout > (da10:ciss1:0:6:0): Error 5, Retries exhausted > (da10:ciss1:0:6:0): Synchronize cache failed > (da11:ciss1:0:7:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da11:ciss1:0:7:0): CAM status: Command timeout > (da11:ciss1:0:7:0): Error 5, Retries exhausted > (da11:ciss1:0:7:0): Synchronize cache failed > (da12:ciss1:0:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da12:ciss1:0:8:0): CAM status: Command timeout > (da12:ciss1:0:8:0): Error 5, Retries exhausted > (da12:ciss1:0:8:0): Synchronize cache failed > (da13:ciss1:0:9:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da13:ciss1:0:9:0): CAM status: Command timeout > (da13:ciss1:0:9:0): Error 5, Retries exhausted > (da13:ciss1:0:9:0): Synchronize cache failed > (da14:ciss1:0:10:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00= =20 > 00 00 > (da14:ciss1:0:10:0): CAM status: Command timeout > (da14:ciss1:0:10:0): Error 5, Retries exhausted > (da14:ciss1:0:10:0): Synchronize cache failed > Automatic reboot in 15 seconds - press a key on the console to abort > Rebooting... >=20 > Dmesg says: > ciss1: port 0x5000-0x50ff mem=20 > 0xfbe00000-0xfbffffff,0xfbdf0000-0xfbdf0fff irq 24 at device 0.0 on pci9 > ciss1: PERFORMANT Transport > da5 at ciss1 bus 0 scbus2 target 1 lun 0 > da4 at ciss1 bus 0 scbus2 target 0 lun 0 > da6 at ciss1 bus 0 scbus2 target 2 lun 0 > da7 at ciss1 bus 0 scbus2 target 3 lun 0 > da8 at ciss1 bus 0 scbus2 target 4 lun 0 > da9 at ciss1 bus 0 scbus2 target 5 lun 0 > da10 at ciss1 bus 0 scbus2 target 6 lun 0 > da11 at ciss1 bus 0 scbus2 target 7 lun 0 > da12 at ciss1 bus 0 scbus2 target 8 lun 0 > da13 at ciss1 bus 0 scbus2 target 9 lun 0 > da14 at ciss1 bus 0 scbus2 target 10 lun 0 >=20 > I also find it interesting that the machine's IML (Integrated Management= =20 > Log) contains this message after every crash: > POST Error: 1719 - A controller failure event occurred prior to this=20 > power-up >=20 > Which might show that the controller indeed locks up, but why does it do= =20 > this under FreeBSD and doesn't under Linux? > I've already tried > hw.ciss.nop_message_heartbeat=3D1;ciss_force_transport=3D1;ciss_force_int= errupt=3D1 > without any effect (it freezes after the same time). >=20 > Last time during the POST the controller said: > Slot 2 HP Smart Array P812 Controller (1024MB, v6.40) 11 Logical= =20 > Drives > 1719-Slot 2 Drive Array - A controller failure event occurred prior to th= is > power-up. (Previous lock up code =3D 0x13) >=20 > Any ideas on what could cause this? >=20 > Thanks, > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" Can you open a p/r on this? I'd like to keep tracking ciss(4) issues. It seems like there is something odd with our driver when using multiple controllers. sean --=-iBA6hzzfyNGWCF1yk9c6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJTMCXzAAoJEBkJRdwI6BaHLwYH/RqsE/xjpjEFWo+Qnhe8Ewxx nCUCNHGfYE//HncIlv2KQI+MMa8LpfVgDfO53SHTJ/IIoRdogS4LyHZKdO2jk4Oe fnQ8MSjVzaguHAogsNUVoOVelatn5FrsL9DLmfn1LccJKd6ONlKwdlp6GQLpulZy s9ef3GTcKVwnE2BmNhzgO3gmeuBCUAZI4pgOOpW2vquD69sQmD1+qFX4O21vfHKZ /RCgucPa7nypgtGX2WKn9gd+/eDTTVRy9OeAq4JjiNyOpC9o+8QNb1Zbw6E8DJ1i nzHlJdZ49vH37peucigkhT0kXWVqJHVORJaEzvisXiZjdueQFLTwIDwNfNuzlXQ= =JnVI -----END PGP SIGNATURE----- --=-iBA6hzzfyNGWCF1yk9c6-- From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 17:10:18 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 78ADEB38; Mon, 24 Mar 2014 17:10:18 +0000 (UTC) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8EE0DDDD; Mon, 24 Mar 2014 17:10:17 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id A5490134674A; Mon, 24 Mar 2014 18:10:14 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 10.4788] X-CRM114-CacheID: sfid-20140324_18101_C9B30C9C X-CRM114-Status: Good ( pR: 10.4788 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Mon Mar 24 18:10:14 2014 X-DSPAM-Confidence: 0.6908 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 533066f6597614214611916 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00010, From*Attila, 0.00375, References*fsn.hu>, 0.00524, Received*(japan.t+online.co.hu, 0.00582, Received*(japan.t, 0.00582, Received*online.co.hu, 0.00582, Received*[195.228.243.99]), 0.00582, Received*online.co.hu+[195.228.243.99]), 0.00582, From*Attila+Nagy, 0.00655, From*Nagy+ Date: Mon, 24 Mar 2014 18:10:09 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 To: sbruno@freebsd.org Subject: Re: Only 0.44 (always) days of uptime with ciss (w/HP SA P812) References: <532FFF5E.2010900@fsn.hu> <1395664375.25687.1.camel@powernoodle.corp.yahoo.com> In-Reply-To: <1395664375.25687.1.camel@powernoodle.corp.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 17:10:18 -0000 On 03/24/2014 01:32 PM, Sean Bruno wrote: Can you open a p/r on this? I'd like to keep tracking ciss(4) issues. It seems like there is something odd with our driver when using multiple controllers. Why do you think it's caused by multiple controllers? Currently I can't remove the P411, but if you are sure about the issue I can try it later. [1]http://www.freebsd.org/cgi/query-pr.cgi?pr=187903 References 1. http://www.freebsd.org/cgi/query-pr.cgi?pr=187903 From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 18:17:33 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 37C75B23; Mon, 24 Mar 2014 18:17:33 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0C609892; Mon, 24 Mar 2014 18:17:33 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2OIHWXw052035; Mon, 24 Mar 2014 18:17:32 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2OIHWkD052034; Mon, 24 Mar 2014 18:17:32 GMT (envelope-from linimon) Date: Mon, 24 Mar 2014 18:17:32 GMT Message-Id: <201403241817.s2OIHWkD052034@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/187903: [ciss] Only 0.44 (always) days of uptime with ciss (w/HP SA P812) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 18:17:33 -0000 Old Synopsis: Only 0.44 (always) days of uptime with ciss (w/HP SA P812) New Synopsis: [ciss] Only 0.44 (always) days of uptime with ciss (w/HP SA P812) Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Mon Mar 24 18:16:58 UTC 2014 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=187903 From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 24 17:46:19 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1B1367F2; Mon, 24 Mar 2014 17:46:19 +0000 (UTC) Received: from mtv-iport-1.cisco.com (mtv-iport-1.cisco.com [173.36.130.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B99232D8; Mon, 24 Mar 2014 17:46:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=3070; q=dns/txt; s=iport; t=1395683178; x=1396892778; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=xtRQijtyFXUyEM3Id64rsaGA/nf5xctcCv0LNw7joSQ=; b=IVbqGkis03FHCDqEmEYxjw7PJL75gC8HzDRKWQwEqY2cdOEjrN8qd8oy Sd+mJJS2r6yxTr6C5w5uuTeqnb3TojqnLP381WCuN3+V7bcAlEoUMpMsy OkFB2F2/nu9s7osntFUOHRKWMbzSBpjvUkigQnOijvspXNZsZL25AXw6l 8=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AicFAOtuMFOrRDoG/2dsb2JhbABZgwbEAoEeFnSCJgEBBDo/EAshJQ8FSS6HXc9DF456B4MkgRQBA4lSgyGLVgGHSYpog00 X-IronPort-AV: E=Sophos;i="4.97,721,1389744000"; d="scan'208";a="105630299" Received: from mtv-core-1.cisco.com ([171.68.58.6]) by mtv-iport-1.cisco.com with ESMTP; 24 Mar 2014 17:46:11 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-1.cisco.com (8.14.5/8.14.5) with ESMTP id s2OHk8KQ031968 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Mar 2014 17:46:10 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s2OHjKth030459; Mon, 24 Mar 2014 10:45:20 -0700 (PDT) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s2OHjJ9A030458; Mon, 24 Mar 2014 10:45:19 -0700 (PDT) (envelope-from ambrisko) Date: Mon, 24 Mar 2014 10:45:19 -0700 From: Doug Ambrisko To: Borja Marcos Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140324174519.GA30345@cisco.com> References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Mon, 24 Mar 2014 18:20:37 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 17:46:19 -0000 On Mon, Mar 24, 2014 at 09:03:43AM +0100, Borja Marcos wrote: | | On Mar 21, 2014, at 5:09 PM, Doug Ambrisko wrote: | | > Do you have a simple test case for that? There were issues in the SCSI | > command translation which should have been fixed via switching it to use | > the CAM translation code. Can you try it via a RAID volume of one disk? | | Yes, I tried all those possibilities. "syspd" devices, passthrough | (same result, corrruption) and when creating single-disk RAID0 volumes | it worked, no corruption. Okay, so you don't have and simple test case to show this. What are you currently doing? I would think the syspd and the single disk RAID0 should be very similar. I'll take a look at that. | Just in case I tried with SSDs (Samsung EVO 840 and OCZ Vertex 4) and | Seagate SAS disks. So it seems generic. Size of disk might be useful. | > | The server is not in production, so I have no problems to try drivers, etc. | > | > You could try the LSI mrsas and see if the corruption goes away and then | > we could try to isolate the difference. mrsas does not support CAM | > passthrough. If both drivers are loaded in the kernel, you can prevent | > mfi from attaching and let mrsas attach via the tunable: | > hw.mfi.mrsas_enable=1 | > The mrsas device come up /dev/da* there are no aliases yet to the mfi one's. | > Also mfiutil won't work with mrsas but MegaCli/StorCli should. If you want | > to use mfiutil, you can switch to mfi to set that up and then switch back | > to mrsas. I need to finish the mrsas compat work so mfiutil can work | > on it. It's a little more complicated then I hoped. | | Actually I think that converting passthrough disks to "mfisyspd" or an | equivalent is a bad idea, unless, of course, there's a compelling reason | I don't realize. For instance, if you are using SSDs you need access to | the TRIM command. | | So, if I can vote, I vote with arms and legs for "da" devices in case it's | a passthough. Nothing is planned to be removed from mfi. However, LSI would like mrsas to be used on newer cards. We've let people LSI know that people use the pass through mode and having the logical volumes come up as /dev/daX and pass through would be confusing. Granted mrsas doesn't support pass through so that isn't a problem. | It's not a pressing need, I installed a LSI2008 card and I connected the | SAS backplane to it, so for now the Invader | is a non-problem, but if it will help to run some tests with the mps | driver I can certainly do it. The mps driver is a different thing, mrsas is LSI's replacement for mfi with newer cards. So if you could try syspd and single RAID 0 test with mrsas that would be great to see if that shows the same pattern of syspd failing with mfi and passing with single RAID 0. Again, the logical volumes will show up as /dev/da* with no pass through devices. If I can reproduce the problem then I can look into here or others might be able to find the solution. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 08:12:21 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DC9F048B; Tue, 25 Mar 2014 08:12:21 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id 7B25F91E; Tue, 25 Mar 2014 08:12:21 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 3FE289DCF24; Tue, 25 Mar 2014 09:12:13 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140324174519.GA30345@cisco.com> Date: Tue, 25 Mar 2014 09:12:09 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> To: Doug Ambrisko X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Tue, 25 Mar 2014 11:39:11 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 08:12:21 -0000 On Mar 24, 2014, at 6:45 PM, Doug Ambrisko wrote: > On Mon, Mar 24, 2014 at 09:03:43AM +0100, Borja Marcos wrote: > |=20 > | On Mar 21, 2014, at 5:09 PM, Doug Ambrisko wrote: > |=20 > | > Do you have a simple test case for that? There were issues in the = SCSI > | > command translation which should have been fixed via switching it = to use > | > the CAM translation code. Can you try it via a RAID volume of one = disk? > |=20 > | Yes, I tried all those possibilities. "syspd" devices, passthrough=20= > | (same result, corrruption) and when creating single-disk RAID0 = volumes=20 > | it worked, no corruption. >=20 > Okay, so you don't have and simple test case to show this. What are > you currently doing? I would think the syspd and the single disk = RAID0 > should be very similar. I'll take a look at that. Yes, sorry for not being so explicit. The server I tried has a backplane, with 23 disks fitted. At first I = assumed it would work, so I just created a raidz2 vdev and begun running benchmarks. That's where = I noticed that it was corrupting data badly. The disks were set as "JBOD" with the Invader's built-in configuration = tool and visible as mfisyspd drives. So I reduced it to using a single disk. Choosing one of them at random, = I did a newfs. Mounted, and executed bonnie++. I got a panic after some time. To avoid the inconvenience of = panics, I moved to ZFS (single disk vdev)=20 so that I could still detect corruption. The tests I did were: - Disk in JBOD/syspd mode, accessed through /dev/mfisyspd -> corruption - Disk in JBOD/syspd mode, passthrough enabled, accessed as /dev/daX -> = corruption - 1 disk RAID0 volume, accessed as /dev/mfid0 -> NO corruption And I understand that syspd and RAID0 should be really different. I = understand that it's a shallow layer doing little more than passing through commands? I should read the = driver thoroughly it seems... Pity there are no available documents. > | Just in case I tried with SSDs (Samsung EVO 840 and OCZ Vertex 4) = and=20 > | Seagate SAS disks. >=20 > So it seems generic. Size of disk might be useful. The Samsung are 1 TB, the OCZ are the 512 MB ones, and the Seagates are = 146 GB.=20 The SSDs (Samsung and OCZ) are SATA disks, the Seagate is SAS. The = Seagate was corrupted as well in the same test cases, so it doesn't seem to be linked to SATA-on-SAS or to = TRIM. TRIM, anyway, doesn't work on "mfisyspd" devices, which is a problem if you want to run ZFS on SSD = disks. > | Actually I think that converting passthrough disks to "mfisyspd" or = an > | equivalent is a bad idea, unless, of course, there's a compelling = reason > | I don't realize. For instance, if you are using SSDs you need access = to > | the TRIM command. > |=20 > | So, if I can vote, I vote with arms and legs for "da" devices in = case it's > | a passthough. >=20 > Nothing is planned to be removed from mfi. However, LSI would like = mrsas > to be used on newer cards. We've let people LSI know that people use > the pass through mode and having the logical volumes come up as = /dev/daX > and pass through would be confusing. Granted mrsas doesn't support = pass > through so that isn't a problem. Pass through is a critical feature when you are running ZFS. I think = it's a real problem if the new driver (the only one that will support newer cards, I guess) = won't support pass-through.=20 And of course I know that you can just buy making sure that you get HBAs = instead of RAID cards, but so many manufacturers bundle those RAID controllers in = fixed configurations that it wouldn't be practical. The ideal solution would be to have logical volumes come up as = /dev/mfid, /dev/mrsas, /dev/mfisyspd or whatever, and passthrough devices as real passthrough devices like = /dev/da. That would avoid confusion. So, please, LSI, consider it. It would really add to the versatility of = your cards and there's nothing to lose. > | It's not a pressing need, I installed a LSI2008 card and I connected = the=20 > | SAS backplane to it, so for now the Invader > | is a non-problem, but if it will help to run some tests with the mps=20= > | driver I can certainly do it. >=20 > The mps driver is a different thing, mrsas is LSI's replacement for = mfi > with newer cards. So if you could try syspd and single RAID 0 test > with mrsas that would be great to see if that shows the same pattern > of syspd failing with mfi and passing with single RAID 0. Again, > the logical volumes will show up as /dev/da* with no pass through > devices. If I can reproduce the problem then I can look into here > or others might be able to find the solution. I'll try, thanks :) Borja. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 11:42:50 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5252BFDD; Tue, 25 Mar 2014 11:42:50 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0239.outbound.protection.outlook.com [207.46.163.239]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C91CFEB9; Tue, 25 Mar 2014 11:42:49 +0000 (UTC) Received: from BN1PR07MB247.namprd07.prod.outlook.com (10.141.64.146) by BN1PR07MB438.namprd07.prod.outlook.com (10.141.63.13) with Microsoft SMTP Server (TLS) id 15.0.898.11; Tue, 25 Mar 2014 11:42:48 +0000 Received: from BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.243]) by BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.243]) with mapi id 15.00.0898.005; Tue, 25 Mar 2014 11:42:48 +0000 From: "Desai, Kashyap" To: Borja Marcos , Doug Ambrisko Subject: RE: LSI - MR-Fusion controller driver patch and man page Thread-Topic: LSI - MR-Fusion controller driver patch and man page Thread-Index: Ac7ykKZzGPABTpK8R3+D5UAv3TD3WQWOC9SAAHVVybAAG8S5gAAWATMAABupEoADWG5MAAAAPT6AClUK/9AACzY8AAB4EYUAABFKigAAELyuAACF5P6AABRP6YAAHkYYgAAGneGw Date: Tue, 25 Mar 2014 11:42:47 +0000 Message-ID: <8bd5b88321704b49baaf4538c6941292@BN1PR07MB247.namprd07.prod.outlook.com> References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.19.224.250] x-forefront-prvs: 01613DFDC8 x-forefront-antispam-report: SFV:NSPM; SFS:(10019001)(6009001)(428001)(52314003)(51444003)(199002)(189002)(377454003)(24454002)(51704005)(13464003)(95416001)(20776003)(92566001)(83322001)(87936001)(81686001)(63696002)(85306002)(93136001)(54316002)(81342001)(77096001)(74366001)(77982001)(59766001)(80976001)(56816005)(76796001)(19580405001)(19580395003)(76576001)(79102001)(81816001)(86362001)(56776001)(94316002)(76482001)(87266001)(80022001)(95666003)(76786001)(65816001)(90146001)(97186001)(46102001)(2656002)(47736001)(81542001)(31966008)(54356001)(4396001)(74662001)(53806001)(74502001)(47446002)(85852003)(66066001)(49866001)(93516002)(33646001)(97336001)(47976001)(50986001)(83072002)(74876001)(51856001)(74706001)(69226001)(98676001)(74316001)(24736002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR07MB438; H:BN1PR07MB247.namprd07.prod.outlook.com; FPR:EEFFFA35.A0F293AD.F2D9517F.4EF8BDC2.206B1; MLV:sfv; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (: lsi.com does not designate permitted sender hosts) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lsi.com X-Mailman-Approved-At: Tue, 25 Mar 2014 11:52:14 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 11:42:50 -0000 > -----Original Message----- > From: Borja Marcos [mailto:borjam@sarenet.es] > Sent: Tuesday, March 25, 2014 1:42 PM > To: Doug Ambrisko > Cc: Desai, Kashyap; scottl@netflix.com; Radford, Adam; Kenneth D. Merry; > sean_bruno@yahoo.com; Mankani, Krishnaraddi; dwhite@ixsystems.com; > Maloy, Joe; jpaetzel@freebsd.org; freebsd-scsi@freebsd.org; McConnell, > Stephen > Subject: Re: LSI - MR-Fusion controller driver patch and man page >=20 >=20 > On Mar 24, 2014, at 6:45 PM, Doug Ambrisko wrote: >=20 > > On Mon, Mar 24, 2014 at 09:03:43AM +0100, Borja Marcos wrote: > > | > > | On Mar 21, 2014, at 5:09 PM, Doug Ambrisko wrote: > > | > > | > Do you have a simple test case for that? There were issues in the > > | > SCSI command translation which should have been fixed via > > | > switching it to use the CAM translation code. Can you try it via a= RAID > volume of one disk? > > | > > | Yes, I tried all those possibilities. "syspd" devices, passthrough > > | (same result, corrruption) and when creating single-disk RAID0 > > | volumes it worked, no corruption. > > > > Okay, so you don't have and simple test case to show this. What are > > you currently doing? I would think the syspd and the single disk > > RAID0 should be very similar. I'll take a look at that. >=20 > Yes, sorry for not being so explicit. >=20 > The server I tried has a backplane, with 23 disks fitted. At first I as= sumed it > would work, so I just created a raidz2 vdev and begun running benchmarks= . > That's where I noticed that it was corrupting data badly. >=20 > The disks were set as "JBOD" with the Invader's built-in configuration to= ol > and visible as mfisyspd drives. >=20 > So I reduced it to using a single disk. Choosing one of them at random, I= did a > newfs. Mounted, and executed > bonnie++. I got a panic after some time. To avoid the inconvenience of > bonnie++panics, I moved to ZFS (single disk vdev) > so that I could still detect corruption. >=20 > The tests I did were: >=20 > - Disk in JBOD/syspd mode, accessed through /dev/mfisyspd -> corruption > - Disk in JBOD/syspd mode, passthrough enabled, accessed as /dev/daX -> > corruption > - 1 disk RAID0 volume, accessed as /dev/mfid0 -> NO corruption >=20 > And I understand that syspd and RAID0 should be really different. I > understand that it's a shallow layer doing little more than passing throu= gh > commands? I should read the driver thoroughly it seems... Pity there are = no > available documents. >=20 > > | Just in case I tried with SSDs (Samsung EVO 840 and OCZ Vertex 4) > > | and Seagate SAS disks. > > > > So it seems generic. Size of disk might be useful. >=20 >=20 > The Samsung are 1 TB, the OCZ are the 512 MB ones, and the Seagates are > 146 GB. > The SSDs (Samsung and OCZ) are SATA disks, the Seagate is SAS. The Seagat= e > was corrupted as well in the same test cases, so it doesn't seem to be li= nked > to SATA-on-SAS or to TRIM. TRIM, anyway, doesn't work on "mfisyspd" > devices, which is a problem if you want to run ZFS on SSD disks. >=20 > > | Actually I think that converting passthrough disks to "mfisyspd" or > > | an equivalent is a bad idea, unless, of course, there's a compelling > > | reason I don't realize. For instance, if you are using SSDs you need > > | access to the TRIM command. > > | > > | So, if I can vote, I vote with arms and legs for "da" devices in > > | case it's a passthough. > > > > Nothing is planned to be removed from mfi. However, LSI would like > > mrsas to be used on newer cards. We've let people LSI know that > > people use the pass through mode and having the logical volumes come > > up as /dev/daX and pass through would be confusing. Granted mrsas > > doesn't support pass through so that isn't a problem. >=20 > Pass through is a critical feature when you are running ZFS. I think it's= a real > problem if the new driver (the only one that will support newer cards, I > guess) won't support pass-through. >=20 > And of course I know that you can just buy making sure that you get HBAs > instead of RAID cards, but so many manufacturers bundle those RAID > controllers in fixed configurations that it wouldn't be practical. >=20 > The ideal solution would be to have logical volumes come up as /dev/mfid, > /dev/mrsas, /dev/mfisyspd or whatever, and passthrough devices as real > passthrough devices like /dev/da. That would avoid confusion. >=20 > So, please, LSI, consider it. It would really add to the versatility of y= our cards > and there's nothing to lose. Borja: driver will attach Raid volume and JBOD (SysPD) to the CAM layer. = It is not good to expose hidden raid volume or what we called as pass-throu= gh device here to the OS for many reason.. Other than management things li= ke SMART monitor, we cannot/should not do file system IO on pass-through de= vices. With it might be true that user always do file system IO on <= mfiX> deivce and consider /dev/daX as pass-through device... With a= ll device will be seen as . You cannot identify which will be a pass-t= hrough and which is configured device by LSI config utils. It is not a complex code change if pass-through device is required for , but it is just a matter of no use and more error prone to expose devic= es as pass-through.=20 None of the LSI driver does this including and in FreeBSD + <= megaraid_sas> and / in Linux. If you can express what functionality you think it is missing, if there is = not pass-through device ? Are you doing ZFS (File system IO) on Pass-throu= gh device. ?=20 If yes, then why can't you create JBOD/SysPD for that purpose? >=20 > > | It's not a pressing need, I installed a LSI2008 card and I connected > > | the SAS backplane to it, so for now the Invader is a non-problem, > > | but if it will help to run some tests with the mps driver I can > > | certainly do it. > > > > The mps driver is a different thing, mrsas is LSI's replacement for > > mfi with newer cards. So if you could try syspd and single RAID 0 > > test with mrsas that would be great to see if that shows the same > > pattern of syspd failing with mfi and passing with single RAID 0. > > Again, the logical volumes will show up as /dev/da* with no pass > > through devices. If I can reproduce the problem then I can look into > > here or others might be able to find the solution. >=20 > I'll try, thanks :) >=20 >=20 >=20 >=20 >=20 > Borja. >=20 >=20 >=20 From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 14:31:30 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F443961; Tue, 25 Mar 2014 14:31:30 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id BAD9823B; Tue, 25 Mar 2014 14:31:29 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 696D39DCEFC; Tue, 25 Mar 2014 15:31:27 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <8bd5b88321704b49baaf4538c6941292@BN1PR07MB247.namprd07.prod.outlook.com> Date: Tue, 25 Mar 2014 15:31:25 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> <8bd5b88321704b49baaf4538c6941292@BN1PR07MB247.namprd07.prod.outlook.com> To: "Desai, Kashyap" X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Tue, 25 Mar 2014 15:20:51 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , Doug Ambrisko , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 14:31:30 -0000 On Mar 25, 2014, at 12:42 PM, Desai, Kashyap wrote: > Borja: >=20 > driver will attach Raid volume and JBOD (SysPD) to the CAM = layer. It is not good to expose hidden raid volume or what we called as = pass-through device here to the OS for many reason.. Other than = management things like SMART monitor, we cannot/should not do file = system IO on pass-through devices. =20 Of course it's not a good idea to expose drives that are part of a = logical volume. But unconfigured drives should. Read on, please ;) > With it might be true that user always do file system IO on = deivce and consider /dev/daX as pass-through device... With = all device will be seen as . You cannot identify which will = be a pass-through and which is configured device by LSI config utils. Exposing devices as "da" should not be a mere "esthetic" decision. The = "da" driver has some stuff intended for direct access to disks, but not = for logical volumes created by other devices such as advanced RAID = cards. For example, the "da" device can issue TRIM commands, it reads = device serial numbers (which, now, can be used by GEOM to identify = disks), etc. Disks are more complicated now with that "advanced format" = thing and so I think it's very important for disks to be directly = accessible if you want/need it. Of course other features might be = introduced in the future. Features that may be added to the "da" driver = but which will probably be useless for a logical device, even outright = inappropiate. I would suggest you to offer choice, and, most critically, to offer a = _clear_ _choice_, as you have different kinds of customers. Some will = want/need logical volumes and advanced RAID stuff, others won't. In some = machines I have I am actually doing *both* things at the same time. I = may have a RAID card based mirror for certain tasks, maybe with a UFS = filesystem on it, relying on pass-through to the rest of the devices on = which I use ZFS. I think you should use a specific name for the logical devices, such as = the mfi driver does. If I see a "mfid" device name it's clear that it's = a logical device, not a "bare metal" hard disk, and that its behavior = and features depend mainly on the logical device magic in the card.=20 And you should offer a perfectly transparent pass-through option, maybe = restricted to disks not configured as "RAID" ones (to avoid accidents), = I mean, what you now call "syspd" mode. These disks, ideally, should not = be assigned to a special logic-volume like "mfisyspd" driver (or its = equivalent), but to the "da" driver so that all of the features I expect = from a bare metal hard disk would work. SMART, access to mode pages, = detecting sector sizes, serial numbers, whatever, would work without = hiccups. Doing it the current "syspd" way means that any new feature added to = disks must be added to the card firmware and to the "syspd" portion of = the driver, while keeping a clear access to the SAS (or SATA-on-SAS) = devices with no other manipulation would mean that the "da" driver would = have immediate access to those features with no need to add support to = the card firmware and driver. > It is not a complex code change if pass-through device is required for = , but it is just a matter of no use and more error prone to = expose devices as pass-through.=20 It is certainly error prone if you are using logical devices. But if you = are not using them (my case and there are many others in this situation) = the lack of a well supported pass through device can be error prone. =46rom a mere engineering point of view, it's a bad idea to add = unnecessary software layers. Advanced RAID card features are a lifesaver = for "classic" filesystems such as UFS/FFS, EXTwhateverFS, NTFS, etc, but = can get in the way of other filesystems such as ZFS. ZFS intends to = perform the functions of a RAID device itself.=20 > None of the LSI driver does this including and in = FreeBSD + and / in Linux. I've been using pass-through disks on Adaptec RAID cards (aac), and LSI = Logic (mps and mfi) with different levels of success for years. It can = be tricky, but ZFS works best with direct access to the disks. > If you can express what functionality you think it is missing, if = there is not pass-through device ? Of course. Some of the missing functionalities I would miss by not using = a pass through are: - Inability to support problematic disks with "quirks". The "da" driver = offers a flexible mechanism for that. If not using the da driver I lose = that ability, and you will agree with me that getting a manufacturer = (LSI) to update a cards firmware is much harder than doing it myself if = needed.=20 - Inability to support future/special features without a firmware update = for the card. An example is the diversity of block sizes in SSDs, or, = more recently, TRIM for SSDs. ZFS on FreeBSD now supports TRIM, and it's = important for performance and drive health. How does "syspd" handle it = currently?=20 =20 - Again I will insist on how additional software layers are a bad idea.=20= - Also, one of the "features" of LSI cards represents a serious = operational issue: the persistent assignment of target numbers to disk = serial numbers keeping a table of target-serial number mappings on = NVRAM. There were some recent messages in this list regarding that = problem. And it seems to happen even when using pass-through devices.=20 In the past I have had problems with ZFS and the "old" way of creating = "pseudo JBOD" devices on LSI cards by creating a RAID 0 logical volume = for each disk. For example, hot swapping a broken disk can be more error = prone if, apart from just extracting a disk and adding a new one, I = need to run certain tool to have it effectively recognised by the card = firmware. It adds unnecessary complexity. Moreover, in some cases (I = can't recall the exact details, as it happened several years ago) it = requires a reboot, which defeats the purpose of how swappable disks in = the first place. Please don't underestimate the operational impact of all this. An = operator swapping a disk at 3 am should not need to do any complex check = to determine the disk to extract. Nor he/she should require additional = actions such as "mfiutil online this", activate that or, of course, a = reboot, to have it recognised. ZFS (and, I presume, other advanced = filesystems) has its own commands for that, which include their own = sanity checks doing its best to avoid trouble.=20 > Are you doing ZFS (File system IO) on Pass-through device. ?=20 Indeed I am. And I know there are many successful setups doing the same. > If yes, then why can't you create JBOD/SysPD for that purpose? It's explained above but I will summarize. - Plain simple good engineering practice (avoiding unneeded software = layers), - Access to special/future features on disks - Better observability (monitoring, etc) - Simpler operational procedures which means safer systems operations = and better reliability. Let me be brutally honest here and, please, take no offense but take it = as feedback from a customer. Right now, advanced RAID cards can be more a liability than a desirable feature. = Look at all the places where people repurpose RAID cards to be simple HBAs doing all sorts of unsupported voodoo.=20 Ideally this shouldn't happen, but we are somewhat forced by server = manufacturers. At some point at least, for example, Dell refused to sell "IT mode" LSI2008 cards for = internal devices, selling them just with external SAS=20 connectors. So many people just repurpose the internal, "IR firmware" = cards to "IT mode" so that they can be simple HBAs even=20 though they still pose a problem with that target-serial number feature = in NVRAM. I have an IBM server here with an onboard Invader card which, obviously, has many more features. By defining some design guidelines for your hardware, firmware, and = drives, however, you can get to a win-win solution. If a card can fullfill both roles perfectly (advanced RAID features and plain = HBA) it will no longer be a liability. The same hardware will be appropiate for many purposes, and it will be even better for the = purchasing departments of us, your final customers. No need to be keeping track of several SKUs depending on the intended purpose. = Same card usable for, say, NTFS and ZFS depending only on configuration. And those design guidelines I am suggesting are simple: - Full functioning pass through mode with a minimal surprise component, = with the simplest, most transparent possible access from the CAM layer to the SAS/SATA commands so that those true pass-through devices = get assigned to the right drivers such as "ses", "da", "sa", etc. This = should be a core feature, not an add on to somewhat ease monitoring. - Making that transparent, pass through mode clearly distinguishable = from the logical volume magic, so that the device name reflects its nature and purpose. "mfid" (or "mrsasd", or whatever you like) would = the logical devices, avoiding attaching them to the standard CAM = drivers.=20 You could just repurpose the "syspd" configuration in the newer = cards/firmware versions so that drives marked as "syspd" become = perfectly transparent pass throughs. Please consider it, I am sure you will have many happy customers. =20 (And I hope you endured reading this message until the end!!) Thank you! Borja. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 17:47:10 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6473E39A; Tue, 25 Mar 2014 17:47:10 +0000 (UTC) Received: from mtv-iport-1.cisco.com (mtv-iport-1.cisco.com [173.36.130.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D9D25BF8; Tue, 25 Mar 2014 17:47:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=5200; q=dns/txt; s=iport; t=1395769629; x=1396979229; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=B1n7jqy2hgNkLDMOUcdetvYmEVerMC0GaIjU0uUN0lE=; b=Nv/Loo3ibh1qNzoOVfRW7llVT5YnO0vasEFk/4uHkTq+CfkEGhlAbYTk Lk0LGo75XWf6uA1C7gQBIoB2LjQiEwdiNNcHwmYxc+O6Zv4hlOUqw1wow tMXNjqXrPLqfFckCqQbuMQED5olERkNaP4Prio+SOoWc+nKRtYr7lONO0 E=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgIFAKjAMVOrRDoG/2dsb2JhbABZgwbDf4EcFnSCJgEBBCcTPxALISUPBUkuh13PTReObgeDJIEUAQOJUoMji1cBh0mKaYNO X-IronPort-AV: E=Sophos;i="4.97,730,1389744000"; d="scan'208";a="105722600" Received: from mtv-core-1.cisco.com ([171.68.58.6]) by mtv-iport-1.cisco.com with ESMTP; 25 Mar 2014 17:46:55 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-1.cisco.com (8.14.5/8.14.5) with ESMTP id s2PHksnu020258 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Mar 2014 17:46:55 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s2PHk7hu041585; Tue, 25 Mar 2014 10:46:07 -0700 (PDT) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s2PHk4gt041584; Tue, 25 Mar 2014 10:46:04 -0700 (PDT) (envelope-from ambrisko) Date: Tue, 25 Mar 2014 10:46:04 -0700 From: Doug Ambrisko To: Borja Marcos Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140325174604.GA41527@cisco.com> References: <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Tue, 25 Mar 2014 17:59:10 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 17:47:10 -0000 Quick question, what version of FreeBSD are you using? I netbooted current and then did: newfs /dev/da4 mount /dev/da4 /mnt cd /mnt ; bonnie++ -u 0 da4 is a Toshiba 300G SAS drive. This worked okay. Thanks, Doug A. On Tue, Mar 25, 2014 at 09:12:09AM +0100, Borja Marcos wrote: | | On Mar 24, 2014, at 6:45 PM, Doug Ambrisko wrote: | | > On Mon, Mar 24, 2014 at 09:03:43AM +0100, Borja Marcos wrote: | > | | > | On Mar 21, 2014, at 5:09 PM, Doug Ambrisko wrote: | > | | > | > Do you have a simple test case for that? There were issues in the SCSI | > | > command translation which should have been fixed via switching it to use | > | > the CAM translation code. Can you try it via a RAID volume of one disk? | > | | > | Yes, I tried all those possibilities. "syspd" devices, passthrough | > | (same result, corrruption) and when creating single-disk RAID0 volumes | > | it worked, no corruption. | > | > Okay, so you don't have and simple test case to show this. What are | > you currently doing? I would think the syspd and the single disk RAID0 | > should be very similar. I'll take a look at that. | | Yes, sorry for not being so explicit. | | The server I tried has a backplane, with 23 disks fitted. At first | I assumed it would work, so I just created a raidz2 vdev and begun running benchmarks. That's where I noticed that it was | corrupting data badly. | | The disks were set as "JBOD" with the Invader's built-in configuration tool and visible as mfisyspd drives. | | So I reduced it to using a single disk. Choosing one of them at random, I did a newfs. Mounted, and executed | bonnie++. I got a panic after some time. To avoid the inconvenience of panics, I moved to ZFS (single disk vdev) | so that I could still detect corruption. | | The tests I did were: | | - Disk in JBOD/syspd mode, accessed through /dev/mfisyspd -> corruption | - Disk in JBOD/syspd mode, passthrough enabled, accessed as /dev/daX -> corruption | - 1 disk RAID0 volume, accessed as /dev/mfid0 -> NO corruption | | And I understand that syspd and RAID0 should be really different. I understand that it's a shallow layer | doing little more than passing through commands? I should read the driver thoroughly it seems... Pity there | are no available documents. | | > | Just in case I tried with SSDs (Samsung EVO 840 and OCZ Vertex 4) and | > | Seagate SAS disks. | > | > So it seems generic. Size of disk might be useful. | | | The Samsung are 1 TB, the OCZ are the 512 MB ones, and the Seagates are 146 GB. | The SSDs (Samsung and OCZ) are SATA disks, the Seagate is SAS. The Seagate was corrupted as well in the | same test cases, so it doesn't seem to be linked to SATA-on-SAS or to TRIM. TRIM, anyway, doesn't work | on "mfisyspd" devices, which is a problem if you want to run ZFS on SSD disks. | | > | Actually I think that converting passthrough disks to "mfisyspd" or an | > | equivalent is a bad idea, unless, of course, there's a compelling reason | > | I don't realize. For instance, if you are using SSDs you need access to | > | the TRIM command. | > | | > | So, if I can vote, I vote with arms and legs for "da" devices in case it's | > | a passthough. | > | > Nothing is planned to be removed from mfi. However, LSI would like mrsas | > to be used on newer cards. We've let people LSI know that people use | > the pass through mode and having the logical volumes come up as /dev/daX | > and pass through would be confusing. Granted mrsas doesn't support pass | > through so that isn't a problem. | | Pass through is a critical feature when you are running ZFS. I think it's a real problem | if the new driver (the only one that will support newer cards, I guess) won't support | pass-through. | | And of course I know that you can just buy making sure that you get HBAs instead of | RAID cards, but so many manufacturers bundle those RAID controllers in fixed configurations | that it wouldn't be practical. | | The ideal solution would be to have logical volumes come up as /dev/mfid, /dev/mrsas, /dev/mfisyspd | or whatever, and passthrough devices as real passthrough devices like /dev/da. That would avoid | confusion. | | So, please, LSI, consider it. It would really add to the versatility of your cards and there's nothing | to lose. | | > | It's not a pressing need, I installed a LSI2008 card and I connected the | > | SAS backplane to it, so for now the Invader | > | is a non-problem, but if it will help to run some tests with the mps | > | driver I can certainly do it. | > | > The mps driver is a different thing, mrsas is LSI's replacement for mfi | > with newer cards. So if you could try syspd and single RAID 0 test | > with mrsas that would be great to see if that shows the same pattern | > of syspd failing with mfi and passing with single RAID 0. Again, | > the logical volumes will show up as /dev/da* with no pass through | > devices. If I can reproduce the problem then I can look into here | > or others might be able to find the solution. | | I'll try, thanks :) | | | | | | Borja. | | From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 18:19:27 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0F037CAC; Tue, 25 Mar 2014 18:19:27 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id BBA43C; Tue, 25 Mar 2014 18:19:26 +0000 (UTC) Received: from www.saremail.com (unknown [194.30.0.100]) by proxypop04.sare.net (Postfix) with ESMTPSA id 7F9EF9DD165; Tue, 25 Mar 2014 19:13:12 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Tue, 25 Mar 2014 19:13:12 +0100 From: borjam@sarenet.es To: Doug Ambrisko Subject: Re: LSI - MR-Fusion controller driver patch and man page In-Reply-To: <20140325174604.GA41527@cisco.com> References: <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> <20140325174604.GA41527@cisco.com> Message-ID: <4e943a2ed4b188b53af7cf71b66e4cda@sarenet.es> X-Sender: borjam@sarenet.es User-Agent: Saremail/0.8 X-Mailman-Approved-At: Tue, 25 Mar 2014 19:40:03 +0000 Cc: scottl@netflix.com, "Radford, Adam" , sean_bruno@yahoo.com, "Mankani, Krishnaraddi" , dwhite@ixsystems.com, "Maloy, Joe" , jpaetzel@freebsd.org, freebsd-scsi@freebsd.org, "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 18:19:27 -0000 El 25.03.2014 18:46, Doug Ambrisko escribió: > Quick question, what version of FreeBSD are you using? I netbooted > current and then did: > newfs /dev/da4 > mount /dev/da4 /mnt > cd /mnt ; bonnie++ -u 0 > da4 is a Toshiba 300G SAS drive. This worked okay. Now that I recall, I tried with 10 and 10-STABLE. Once I saw that it didn't work I abandoned the idea of using the Invader in that configuration, installing a LSI2008 based card (flashed IBM M1015) and using the Invader just to boot from a mirror. I also found out that the system didn't work when booting from the mirror with FreeBSD 10. but it did with -CURRENT. I copied the mfi driver from -CURRENT to 10-STABLE and I verified that it solved that particular problem (booting from an Invader logic volume) but, after discarding the Invader for ZFS, I didn't repeat the syspd/pass through tests. My fault. http://lists.freebsd.org/pipermail/freebsd-scsi/2014-March/006274.html So, first thing to do tomorrow, repeat the syspd/passthrough tests with STABLE and the backported mfi driver from -CURRENT. Sorry, my fault, I should have tried that :/ Borja. From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 19:13:59 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 60A06239; Tue, 25 Mar 2014 19:13:59 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2lp0203.outbound.protection.outlook.com [207.46.163.203]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A5DDA924; Tue, 25 Mar 2014 19:13:58 +0000 (UTC) Received: from BN1PR07MB247.namprd07.prod.outlook.com (10.141.64.146) by BN1PR07MB357.namprd07.prod.outlook.com (10.141.60.143) with Microsoft SMTP Server (TLS) id 15.0.908.10; Tue, 25 Mar 2014 19:13:54 +0000 Received: from BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.243]) by BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.243]) with mapi id 15.00.0898.005; Tue, 25 Mar 2014 19:13:54 +0000 From: "Desai, Kashyap" To: Borja Marcos Subject: RE: LSI - MR-Fusion controller driver patch and man page Thread-Topic: LSI - MR-Fusion controller driver patch and man page Thread-Index: Ac7ykKZzGPABTpK8R3+D5UAv3TD3WQWOC9SAAHVVybAAG8S5gAAWATMAABupEoADWG5MAAAAPT6AClUK/9AACzY8AAB4EYUAABFKigAAELyuAACF5P6AABRP6YAAHkYYgAAGneGwAAahB4AACQy0gA== Date: Tue, 25 Mar 2014 19:13:53 +0000 Message-ID: <45cbdd9366aa4a19997d4ca306d0cdcc@BN1PR07MB247.namprd07.prod.outlook.com> References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> <8bd5b88321704b49baaf4538c6941292@BN1PR07MB247.namprd07.prod.outlook.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.19.224.250] x-forefront-prvs: 01613DFDC8 x-forefront-antispam-report: SFV:NSPM; SFS:(10019001)(6009001)(428001)(24454002)(40224001)(377454003)(52604005)(51704005)(13464003)(189002)(199002)(54356001)(74662001)(80976001)(31966008)(51856001)(53806001)(81816001)(74876001)(87936001)(81686001)(19580405001)(74706001)(83322001)(74502001)(47446002)(19580395003)(85306002)(87266001)(56776001)(54316002)(2656002)(76482001)(33646001)(49866001)(47736001)(50986001)(47976001)(4396001)(77982001)(59766001)(98676001)(97336001)(65816001)(66066001)(80022001)(79102001)(20776003)(63696002)(46102001)(74366001)(94316002)(94946001)(93516002)(95416001)(77096001)(86362001)(69226001)(83072002)(85852003)(92566001)(95666003)(74316001)(90146001)(56816005)(93136001)(97186001)(81342001)(81542001)(76796001)(76786001)(76576001)(24736002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR07MB357; H:BN1PR07MB247.namprd07.prod.outlook.com; FPR:CA1FF2F5.AFF2A16C.BBD17173.12F9D941.20918; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (: lsi.com does not designate permitted sender hosts) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lsi.com X-Mailman-Approved-At: Tue, 25 Mar 2014 20:23:13 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , Doug Ambrisko , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 19:13:59 -0000 Borja: I have read your comment. First of all thanks for explaining with lots of t= echnical details. I definitely like to take this as feedback and will work= internally to find out how best we can handle this. As of now you cannot u= se mrsas driver as mfi pass-through. I have observed that most of the benefit you mentioned for pass-through is = mainly faced by some manufacturing divisions and we provide temporary drop = for some specific reason. That driver expose Un-configured drive to OS and = they can do FW upgrade of the Drives without doing lots of manual work. Let me explain you one fundamental problem with pass-through drive. Let's say you have 4 drives and all are exposed to OS as pass-through drive= . Now user can't recognize (using LSI provided configuration utils like sto= rcli/MegaCl), if those drive are used by end user. From LSI config utils, i= t is still Un-configured drive and valid for creating Raid volume. So this = is big issue managing physical disk if Un-configured drive are exposed and = used by user. You can run MR controller in JBOD mod where all drives will be default conv= erted as JBOD and visible to the OS. =20 Also, LSI controller support only T10 Thin provisioning standards. For all = JBOD drives command will go to the actual drive, but for Volumes it disable= s via setting values in vpd page 0xb0 for Volumes. controller map Volumes on bus-0 and syspd to bus-1..so you can easi= ly figure out Raid vs JBOD.=20 LSI developed CAM based HBA device driver and that was under guidanc= e of FreeBSD key folks. Our first goal is to meet driver with all = latest features (which Linux driver supports) and use CAM b= ase interface same as driver.=20 We will add new features as and when requested and prioritize.=20 Doug: I have to see your query regarding difference between Thunderbolt and Invad= er.=20 ~ Kashyap > -----Original Message----- > From: Borja Marcos [mailto:borjam@sarenet.es] > Sent: Tuesday, March 25, 2014 8:01 PM > To: Desai, Kashyap > Cc: Doug Ambrisko; scottl@netflix.com; Radford, Adam; Kenneth D. Merry; > sean_bruno@yahoo.com; Mankani, Krishnaraddi; dwhite@ixsystems.com; > Maloy, Joe; jpaetzel@freebsd.org; freebsd-scsi@freebsd.org; McConnell, > Stephen > Subject: Re: LSI - MR-Fusion controller driver patch and man page >=20 >=20 > On Mar 25, 2014, at 12:42 PM, Desai, Kashyap wrote: >=20 > > Borja: > > > > driver will attach Raid volume and JBOD (SysPD) to the CAM laye= r. > It is not good to expose hidden raid volume or what we called as pass- > through device here to the OS for many reason.. Other than management > things like SMART monitor, we cannot/should not do file system IO on pass= - > through devices. >=20 > Of course it's not a good idea to expose drives that are part of a logica= l > volume. But unconfigured drives should. Read on, please ;) >=20 > > With it might be true that user always do file system IO on > deivce and consider /dev/daX as pass-through device... With all > device will be seen as . You cannot identify which will be a pass- > through and which is configured device by LSI config utils. >=20 > Exposing devices as "da" should not be a mere "esthetic" decision. The "d= a" > driver has some stuff intended for direct access to disks, but not for lo= gical > volumes created by other devices such as advanced RAID cards. For example= , > the "da" device can issue TRIM commands, it reads device serial numbers > (which, now, can be used by GEOM to identify disks), etc. Disks are more > complicated now with that "advanced format" thing and so I think it's ver= y > important for disks to be directly accessible if you want/need it. Of co= urse > other features might be introduced in the future. Features that may be > added to the "da" driver but which will probably be useless for a logical > device, even outright inappropiate. >=20 > I would suggest you to offer choice, and, most critically, to offer a _cl= ear_ > _choice_, as you have different kinds of customers. Some will want/need > logical volumes and advanced RAID stuff, others won't. In some machines I > have I am actually doing *both* things at the same time. I may have a RAI= D > card based mirror for certain tasks, maybe with a UFS filesystem on it, r= elying > on pass-through to the rest of the devices on which I use ZFS. >=20 > I think you should use a specific name for the logical devices, such as t= he mfi > driver does. If I see a "mfid" device name it's clear that it's a logical= device, > not a "bare metal" hard disk, and that its behavior and features depend > mainly on the logical device magic in the card. >=20 > And you should offer a perfectly transparent pass-through option, maybe > restricted to disks not configured as "RAID" ones (to avoid accidents), I= mean, > what you now call "syspd" mode. These disks, ideally, should not be assig= ned > to a special logic-volume like "mfisyspd" driver (or its equivalent), but= to the > "da" driver so that all of the features I expect from a bare metal hard d= isk > would work. SMART, access to mode pages, detecting sector sizes, serial > numbers, whatever, would work without hiccups. >=20 > Doing it the current "syspd" way means that any new feature added to disk= s > must be added to the card firmware and to the "syspd" portion of the driv= er, > while keeping a clear access to the SAS (or SATA-on-SAS) devices with no > other manipulation would mean that the "da" driver would have immediate > access to those features with no need to add support to the card firmware > and driver. >=20 >=20 > > It is not a complex code change if pass-through device is required for > , but it is just a matter of no use and more error prone to expose > devices as pass-through. >=20 > It is certainly error prone if you are using logical devices. But if you = are not > using them (my case and there are many others in this situation) the lack= of a > well supported pass through device can be error prone. >=20 > From a mere engineering point of view, it's a bad idea to add unnecessary > software layers. Advanced RAID card features are a lifesaver for "classic= " > filesystems such as UFS/FFS, EXTwhateverFS, NTFS, etc, but can get in the > way of other filesystems such as ZFS. ZFS intends to perform the function= s of > a RAID device itself. >=20 > > None of the LSI driver does this including and in FreeBSD= + > and / in Linux. >=20 > I've been using pass-through disks on Adaptec RAID cards (aac), and LSI L= ogic > (mps and mfi) with different levels of success for years. It can be trick= y, but > ZFS works best with direct access to the disks. >=20 > > If you can express what functionality you think it is missing, if there= is not > pass-through device ? >=20 > Of course. Some of the missing functionalities I would miss by not using = a > pass through are: >=20 > - Inability to support problematic disks with "quirks". The "da" driver o= ffers a > flexible mechanism for that. If not using the da driver I lose that abili= ty, and > you will agree with me that getting a manufacturer (LSI) to update a card= s > firmware is much harder than doing it myself if needed. >=20 > - Inability to support future/special features without a firmware update = for > the card. An example is the diversity of block sizes in SSDs, or, more re= cently, > TRIM for SSDs. ZFS on FreeBSD now supports TRIM, and it's important for > performance and drive health. How does "syspd" handle it currently? >=20 > - Again I will insist on how additional software layers are a bad idea. >=20 > - Also, one of the "features" of LSI cards represents a serious operation= al > issue: the persistent assignment of target numbers to disk serial numbers > keeping a table of target-serial number mappings on NVRAM. There were > some recent messages in this list regarding that problem. And it seems to > happen even when using pass-through devices. >=20 > In the past I have had problems with ZFS and the "old" way of creating > "pseudo JBOD" devices on LSI cards by creating a RAID 0 logical volume fo= r > each disk. For example, hot swapping a broken disk can be more error pron= e > if, apart from just extracting a disk and adding a new one, I need to ru= n > certain tool to have it effectively recognised by the card firmware. It a= dds > unnecessary complexity. Moreover, in some cases (I can't recall the exact > details, as it happened several years ago) it requires a reboot, which de= feats > the purpose of how swappable disks in the first place. >=20 > Please don't underestimate the operational impact of all this. An operato= r > swapping a disk at 3 am should not need to do any complex check to > determine the disk to extract. Nor he/she should require additional actio= ns > such as "mfiutil online this", activate that or, of course, a reboot, to = have it > recognised. ZFS (and, I presume, other advanced filesystems) has its own > commands for that, which include their own sanity checks doing its best t= o > avoid trouble. >=20 > > Are you doing ZFS (File system IO) on Pass-through device. ? >=20 > Indeed I am. And I know there are many successful setups doing the same. >=20 > > If yes, then why can't you create JBOD/SysPD for that purpose? >=20 > It's explained above but I will summarize. >=20 > - Plain simple good engineering practice (avoiding unneeded software > layers), > - Access to special/future features on disks > - Better observability (monitoring, etc) > - Simpler operational procedures which means safer systems operations and > better reliability. >=20 > Let me be brutally honest here and, please, take no offense but take it a= s > feedback from a customer. Right now, advanced RAID cards can be more a > liability than a desirable feature. Look at all the places where people > repurpose RAID cards to be simple HBAs doing all sorts of unsupported > voodoo. >=20 > Ideally this shouldn't happen, but we are somewhat forced by server > manufacturers. At some point at least, for example, Dell refused to sell = "IT > mode" LSI2008 cards for internal devices, selling them just with external= SAS > connectors. So many people just repurpose the internal, "IR firmware" car= ds > to "IT mode" so that they can be simple HBAs even though they still pose = a > problem with that target-serial number feature in NVRAM. I have an IBM > server here with an onboard Invader card which, obviously, has many more > features. >=20 > By defining some design guidelines for your hardware, firmware, and drive= s, > however, you can get to a win-win solution. If a card can fullfill both r= oles > perfectly (advanced RAID features and plain HBA) it will no longer be a > liability. The same hardware will be appropiate for many purposes, and it= will > be even better for the purchasing departments of us, your final customers= . > No need to be keeping track of several SKUs depending on the intended > purpose. Same card usable for, say, NTFS and ZFS depending only on > configuration. >=20 > And those design guidelines I am suggesting are simple: >=20 > - Full functioning pass through mode with a minimal surprise component, > with the simplest, most transparent possible access from the CAM layer to > the SAS/SATA commands so that those true pass-through devices get > assigned to the right drivers such as "ses", "da", "sa", etc. This shoul= d be a > core feature, not an add on to somewhat ease monitoring. >=20 > - Making that transparent, pass through mode clearly distinguishable from > the logical volume magic, so that the device name reflects its nature and > purpose. "mfid" (or "mrsasd", or whatever you like) would the logical > devices, avoiding attaching them to the standard CAM drivers. >=20 >=20 > You could just repurpose the "syspd" configuration in the newer > cards/firmware versions so that drives marked as "syspd" become perfectly > transparent pass throughs. >=20 > Please consider it, I am sure you will have many happy customers. >=20 > (And I hope you endured reading this message until the end!!) >=20 >=20 > Thank you! >=20 >=20 >=20 >=20 >=20 >=20 >=20 > Borja. >=20 From owner-freebsd-scsi@FreeBSD.ORG Tue Mar 25 22:15:14 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 713258DD; Tue, 25 Mar 2014 22:15:14 +0000 (UTC) Received: from mtv-iport-1.cisco.com (mtv-iport-1.cisco.com [173.36.130.12]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2320CC33; Tue, 25 Mar 2014 22:15:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=374; q=dns/txt; s=iport; t=1395785714; x=1396995314; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=oT4dvMmvw1M7hAMLCiyh2DgbyyhN87ZduMy/W1mnPf0=; b=ABwDtukKwyfTf1ToLNJtIT7/G4HWXn7P7Ao0aMUWSUE2odnWTq/NEMLV Ztk3aF/MkLl8Pr4PGjGZZGoUnwxbdFHL4vGwVDYp5w8C/5446sPHJb931 5DDOCzX6wKnBb/97iZQBaGkQJdi+GRy+IXLrWcECqTBWzRmgDQaemOUb8 0=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AiMFAMn+MVOrRDoJ/2dsb2JhbABZgwbDfoEdFnSCJgEBBDo/EAshJQ8FSYgLz34Xjm4HgySBFAEDiVKOegGSMoNO X-IronPort-AV: E=Sophos;i="4.97,730,1389744000"; d="scan'208";a="105742519" Received: from mtv-core-4.cisco.com ([171.68.58.9]) by mtv-iport-1.cisco.com with ESMTP; 25 Mar 2014 22:15:10 +0000 Received: from dambriskodt.cisco.com (dambriskodt.cisco.com [171.71.230.115]) by mtv-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id s2PMFAaf023386 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Mar 2014 22:15:10 GMT Received: from dambriskodt.cisco.com (localhost [127.0.0.1]) by dambriskodt.cisco.com (8.14.7/8.14.3) with ESMTP id s2PMENAV043549; Tue, 25 Mar 2014 15:14:23 -0700 (PDT) (envelope-from ambrisko@dambriskodt.cisco.com) Received: (from ambrisko@localhost) by dambriskodt.cisco.com (8.14.7/8.14.3/Submit) id s2PMEMln043548; Tue, 25 Mar 2014 15:14:22 -0700 (PDT) (envelope-from ambrisko) Date: Tue, 25 Mar 2014 15:14:22 -0700 From: Doug Ambrisko To: "Desai, Kashyap" Subject: Re: LSI - MR-Fusion controller driver patch and man page Message-ID: <20140325221422.GA43490@cisco.com> References: <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> <8bd5b88321704b49baaf4538c6941292@BN1PR07MB247.namprd07.prod.outlook.com> <45cbdd9366aa4a19997d4ca306d0cdcc@BN1PR07MB247.namprd07.prod.outlook.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <45cbdd9366aa4a19997d4ca306d0cdcc@BN1PR07MB247.namprd07.prod.outlook.com> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Tue, 25 Mar 2014 22:56:58 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 22:15:14 -0000 On Tue, Mar 25, 2014 at 07:13:53PM +0000, Desai, Kashyap wrote: [snip] | Doug: | I have to see your query regarding difference between Thunderbolt and | Invader. Okay. That seems to be the missing piece that I didn't have before. Hopefully, it helps LSI recreate the issue. I should get a newer Fury card so I can test with that as well. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Wed Mar 26 06:14:53 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E40FBF93 for ; Wed, 26 Mar 2014 06:14:53 +0000 (UTC) Received: from mail-ve0-x231.google.com (mail-ve0-x231.google.com [IPv6:2607:f8b0:400c:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A57A9CB2 for ; Wed, 26 Mar 2014 06:14:53 +0000 (UTC) Received: by mail-ve0-f177.google.com with SMTP id sa20so1742388veb.22 for ; Tue, 25 Mar 2014 23:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=He5Ve7ng5yqdQ5r5CmSkuO0K1X6T8s7x6ALKwwD5zpQ=; b=duiD4pMGhYNMokpjTZaIx3uy5p7NoAhv6vcBDhpHoOBN5Nf3FGi1Dsc1ybrJZ9dVX5 babU8Y/Jvd+mSJhIbDTWtmLhEaGxk8QZ2pUKRS4vhUe3sDLrQF+lkTVgYw6pbDlXN2Sb 7cAKGl9dhCIcKNX6966sQCRNR5E+cr9VLblSM+Yl8GCSY+0yQw5P3ZXKBfn5VHW/HZ/o RqQYPXQk8TgB7kN59ME9Oh5vLBj5z/ORqXM/5GY9r6ol5xH5SRAnmUwd1nmhZGnZRQ+D +Dv7Id55soYZmM2OCc7rdEJkxDB7s1ffY5ZqavDtXvs3W/vpvdqNyplJEDCgu6eOmEAQ oWNQ== X-Received: by 10.52.251.199 with SMTP id zm7mr50759660vdc.21.1395814492849; Tue, 25 Mar 2014 23:14:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.108.10 with HTTP; Tue, 25 Mar 2014 23:14:32 -0700 (PDT) From: bharat singh Date: Wed, 26 Mar 2014 11:44:32 +0530 Message-ID: Subject: Adding write_same16 to CAM Target Layer To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 06:14:54 -0000 Hello, I am trying to add write_same16 (block zero) support to my FC stack. This i am trying to achieve by adding a hook in CTL layer, reading start_LBA and no of blocks to fill with 0. struct ctl_scsiio { >-------struct ctl_io_hdr io_hdr;>------/* common to all I/O types */ >-------uint32_t ext_sg_entries;>-----/* 0 = no S/G list, > 0 = num entries */ >-------uint8_t> *ext_data_ptr;>------/* data buffer or S/G list */ >-------uint32_t ext_data_len;>-------/* Data transfer length */ >-------uint32_t ext_data_filled;>----/* Amount of data filled so far */ >-------uint32_t kern_sg_entries;>----/* 0 = no S/G list, > 0 = num entries */ >-------uint32_t rem_sg_entries;>-----/* 0 = no S/G list, > 0 = num entries */ >-------uint8_t *kern_data_ptr;>-----/* data buffer or S/G list */ >-------uint32_t kern_data_len;>------/* Length of this S/G list/buffer */ >-------uint32_t kern_total_len;>-----/* Total length of this transaction */ >-------uint32_t kern_data_resid;>----/* Length left to transfer after this*/ >-------uint32_t kern_rel_offset;>----/* Byte Offset of this transfer */ >-------struct scsi_sense_data sense_data;>-/* sense data */ >-------uint8_t> sense_len;>-->-------/* Returned sense length */ >-------uint8_t> scsi_status;>>-------/* SCSI status byte */ >-------uint8_t> sense_residual;>-----/* sense residual length */ >-------uint32_t residual;>--->-------/* data residual length */ >-------uint32_t tag_num;>---->-------/* tag number */ >-------ctl_tag_type tag_type;>->-------/* simple, ordered, head of queue,etc.*/ >-------uint8_t cdb_len;>---->-------/* CDB length */ >-------uint8_t> cdb[CTL_MAX_CDBLEN];>/* CDB */ >-------int>---- (*be_move_done)(union ctl_io *io); /* called by fe */ >-------int (*io_cont)(union ctl_io *io); /* to continue processing */ >-------uint32_t ctl_lun_masking_disabled; /* Disable lun mask checks for ctladm calls */ }; For write_same16 initiators will send a block of 0s with start_LBA and no of blocks. Assuming that I am trying to: datalen = blocksize * num_blocks; uint8_t * databuf = malloc(datalen); bzero(databuf, 0, datalen); ctsio->kern_data_ptr = dataptr; ctsio->kern_data_len = datalen; lbalen.lba = lba; lbalen.len = num_blocks; memcpy(ctsio->io_hdr.ctl_private[CTL_PRIV_LBA_LEN].bytes, &lbalen, sizeof(lbalen)); ret = lun->backend->data_submit((union ctl_io *)ctsio); but it's failing in ISP layer with data overflow. My initiator is sending 2048 blocks of size 512B each and a payload of 512 bytes in data-out buffer. isp0: isp_target_start_ctio: [0x120154] data overflow by 1048064 bytes (1:3:0:1): WRITE SAME(16). CDB: 93 00 00 00 00 00 00 1f e8 00 00 00 08 00 00 00 Have any one done changes to the CTL read/write path, am I missing something regarding buffer population. Thanks for the help in advance. Thanks, Bharat From owner-freebsd-scsi@FreeBSD.ORG Wed Mar 26 08:23:48 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 592984D4; Wed, 26 Mar 2014 08:23:48 +0000 (UTC) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A5E568C0; Wed, 26 Mar 2014 08:23:46 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 418DC1334E43; Wed, 26 Mar 2014 09:23:39 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 9.7872] X-CRM114-CacheID: sfid-20140326_09233_41DB0C8F X-CRM114-Status: Good ( pR: 9.7872 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Wed Mar 26 09:23:39 2014 X-DSPAM-Confidence: 0.9952 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 53328e8b575603340586902 X-DSPAM-Factors: 27, From*"Nagy, Attila" , 0.00047, kernel, 0.00085, cache, 0.00164, wrote+>, 0.00181, >+It, 0.00229, timeout, 0.00277, >+>, 0.00311, fault, 0.00404, fault, 0.00404, stack, 0.00437, stack, 0.00437, wrote, 0.00517, wrote, 0.00517, a+separate, 0.00524, References*fsn.hu>, 0.00524, Received*(japan.t+online.co.hu, 0.00582, Received*(japan.t, 0.00582, Received*online.co.hu, 0.00582, Received*[195.228.243.99]), 0.00582, Received*online.co.hu+[195.228.243.99]), 0.00582, interrupt, 0.00655, =+0, 0.00689, =+0, 0.00689, Received*from+japan.t, 0.00747, Received*online.private+(japan.t, 0.00747, Received*japan.t+online.private, 0.00747, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id 0B3361334E2D; Wed, 26 Mar 2014 09:23:37 +0100 (CET) Message-ID: <53328E89.2050107@fsn.hu> Date: Wed, 26 Mar 2014 09:23:37 +0100 From: "Nagy, Attila" MIME-Version: 1.0 To: sbruno@freebsd.org Subject: Re: Only 0.44 (always) days of uptime with ciss (w/HP SA P812) References: <532FFF5E.2010900@fsn.hu> <1395664375.25687.1.camel@powernoodle.corp.yahoo.com> In-Reply-To: <1395664375.25687.1.camel@powernoodle.corp.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 08:23:48 -0000 On 03/24/14 13:32, Sean Bruno wrote: > > Can you open a p/r on this? I'd like to keep tracking ciss(4) issues. > It seems like there is something odd with our driver when using multiple > controllers. > Update on this: I've disabled the P410i (previously I wrote P411, but this is not a separate card in this machine) and it crashes the same way, so it's not related to multiple controllers: ciss0: ADAPTER HEARTBEAT FAILED Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xfffffe0c578b295d stack pointer = 0x28:0xfffffe0baf1ab9d0 frame pointer = 0x28:0xfffffe0baf1aba20 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi4: clock) trap number = 1 panic: privileged instruction fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0baf1ab560 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0baf1ab610 panic() at panic+0x155/frame 0xfffffe0baf1ab690 trap_fatal() at trap_fatal+0x3a2/frame 0xfffffe0baf1ab6f0 trap() at trap+0x794/frame 0xfffffe0baf1ab910 calltrap() at calltrap+0x8/frame 0xfffffe0baf1ab910 --- trap 0x1, rip = 0xfffffe0c578b295d, rsp = 0xfffffe0baf1ab9d0, rbp = 0xfffffe0baf1aba20 --- (null)() at 0xfffffe0c578b295d/frame 0xfffffe0baf1aba20 softclock_call_cc() at softclock_call_cc+0x16c/frame 0xfffffe0000e5c120 kernphys() at 0xffffffff/frame 0xfffffe0000e5cd20 kernphys() at 0xffffffff/frame 0xfffffe0000e5d200 kernphys() at 0xffffffff/frame 0xfffffe0000e5d920 Uptime: 10h34m30s (da0:ciss0:0:0:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 (da0:ciss0:0:0:0): CAM status: Command timeout (da0:ciss0:0:0:0): Error 5, Retries exhausted (da0:ciss0:0:0:0): Synchronize cache failed [...] From owner-freebsd-scsi@FreeBSD.ORG Thu Mar 27 09:42:50 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 30DF9F19; Thu, 27 Mar 2014 09:42:50 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id AB43D86A; Thu, 27 Mar 2014 09:42:49 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 1580B9DC8CC; Thu, 27 Mar 2014 10:42:41 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140325174604.GA41527@cisco.com> Date: Thu, 27 Mar 2014 10:42:37 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <779F5D6D-BCA2-4376-84BC-6B08A8F357D4@sarenet.es> References: <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> <20140325174604.GA41527@cisco.com> To: Doug Ambrisko X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Thu, 27 Mar 2014 11:40:58 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 09:42:50 -0000 On Mar 25, 2014, at 6:46 PM, Doug Ambrisko wrote: > Quick question, what version of FreeBSD are you using? I netbooted > current and then did: > newfs /dev/da4 > mount /dev/da4 /mnt > cd /mnt ; bonnie++ -u 0 > da4 is a Toshiba 300G SAS drive. This worked okay. Sorry for the delay, I had to rearrange some stuff to repeat the tests. For me, it corrupts data with -CURRENT, -STABLE (the one which has a = backport of the mfi driver from -CURRENT) and 10-RELEASE. I noticed (I didn't check it when I ran the first tests, as data = corruption was enough of a problem) that the controller is negotiating = just 150 MB/s. Not sure if the driver is lying, though.=20 # camcontrol devlist at scbus0 target 3 lun 0 (da0,pass0) at scbus0 target 4 lun 0 (da1,pass1) at scbus1 target 8 lun 0 (ses0,pass2) at scbus1 target 12 lun 0 = (ses1,pass3) at scbus1 target 15 lun 0 (da2,pass4) at scbus1 target 16 lun 0 (da3,pass5) at scbus1 target 17 lun 0 (da4,pass6) at scbus1 target 32 lun 0 (da5,pass7) at scbus1 target 33 lun 0 (da6,pass8) at scbus1 target 34 lun 0 (da7,pass9) at scbus1 target 35 lun 0 = (da8,pass10) at scbus1 target 36 lun 0 = (da9,pass11) at scbus1 target 37 lun 0 = (da10,pass12) at scbus1 target 39 lun 0 = (da11,pass13) at scbus1 target 40 lun 0 = (da12,pass14) at scbus1 target 41 lun 0 = (da13,pass15) at scbus1 target 42 lun 0 = (da14,pass16) # camcontrol inq ses0 pass2: Fixed Enclosure Services SCSI-4 device=20= pass2: 150.000MB/s transfers, Command Queueing Enabled # camcontrol inq da8 pass10: Fixed Direct Access SCSI-6 device=20 pass10: Serial Number S1D9NEADA08547X =20 pass10: 150.000MB/s transfers, Command Queueing Enabled But I've noticed that disk transfers to one of the SSDs peak around 200 = MB/s, while they reach 400 MB/s with mps and a LSI2008 card replacing the Invader. Looking for the mrsas driver to download I stumbled upon this: = http://www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%= 20Common%20Files/6.602.01.00_MR_Free-BSD_Driver.txt Bug Fixes/Enhancements: =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D SCGCQ00523785 (CSET) - Data Corruption observed on JBODs while running = IOs with 32k block size. SCGCQ00459075 (DFCT) - On converting from UG to JBOD and vice versa, fw = sees reply msQOverflow and hit montask. I'll try to install this driver on my 10-STABLE and see how it goes. Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Mar 27 14:29:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CD7D28F4; Thu, 27 Mar 2014 14:29:00 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) by mx1.freebsd.org (Postfix) with ESMTP id 6C441861; Thu, 27 Mar 2014 14:29:00 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 7C5759DD129; Thu, 27 Mar 2014 15:28:57 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140324174519.GA30345@cisco.com> Date: Thu, 27 Mar 2014 15:28:56 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> To: Doug Ambrisko X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Thu, 27 Mar 2014 15:29:34 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 14:29:00 -0000 On Mar 24, 2014, at 6:45 PM, Doug Ambrisko wrote: > Nothing is planned to be removed from mfi. However, LSI would like = mrsas > to be used on newer cards. We've let people LSI know that people use > the pass through mode and having the logical volumes come up as = /dev/daX > and pass through would be confusing. Granted mrsas doesn't support = pass > through so that isn't a problem. I've just tried a Joe job with the mrsas driver, installing it on = FreeBSD 10-STABLE. NO CORRUPTION (and the disks are indeed assigned to the "da" driver, = with trim and quirks working). I have just had a problem (reading a disk's attributes with smartctl = caused a disconnect and I had to do a camcontrol rescan to see the disk again) but it can be due to the = ugly kludge I've done to be able to compile mrsas in 10-STABLE. (Don't try this at home, kids, it was just a zero-effort shot which, = surprisingly, has yielded good results ;) ) I've downloaded the latest driver, 6.602.01.00_MR_Free-BSD_Driver.tgz = from the LSI website. I noticed the release notes mention a bug that causes data corruption with syspd disks = (bingo!).=20 So I installed it (copied the mrsas directories to /usr/src/sys/dev and = /usr/src/sys/modules), updated the files list, and compiled a kernel without the mfi driver. In order to compile I had to edit mrsas_cam.c, seems that the flags = have changed in recent versions. % diff mrsas_cam.c* 419,420c419,420 < if (!(ccb_h->flags & CAM_DATA_PADDR)) { //Virtual data address = BORJA < if (!(ccb_h->flags & CAM_DATA_SG)) { // Borja asumi = SCATTER_VALID es DATA_SG --- > if (!(ccb_h->flags & CAM_DATA_PHYS)) { //Virtual data address > if (!(ccb_h->flags & CAM_SCATTER_VALID)) { I am pretty sure my flag substitutions are wrong, but all I wanted to do = was a really quick test. The test has been successful and=20 at least in a couple of hours I haven't seen any data corruption at all. = Just that dropping issue when using smartctl to read the SMART data, but I wonder if it's caused by my obviously wrong kludge. I still notice that, for some reason, the devices are still negotiating = a low speed. It must be true, because the SSDs peak at around 200 MB/s = with the Invader, reaching 400 MB/s with a flashed LSI2008 card (latest = IT mode firmware). # camcontrol inq ses0 pass2: Fixed Enclosure Services SCSI-4 device=20= pass2: 150.000MB/s transfers, Command Queueing Enabled # camcontrol inq da14 pass16: Fixed Direct Access SCSI-6 device=20 pass16: Serial Number S1D9NEADA08911Y =20 pass16: 150.000MB/s transfers, Command Queueing Enabled I must also say, regarding my comments to LSI, that maybe I was wrong = with some of them. I see that with the disks configured as SYSPD on the Invader card they appear as "da" (alright, that I knew) *and* I can = see the additional features such as TRIM (which works) and of course disk quirks. Anyway of course I will insist that so-called syspd drives = should have a guarantee of maximum transparency. Mar 27 13:49:40 piruli kernel: da16 at mrsas0 bus 1 scbus2 target 23 lun = 0 Mar 27 13:49:40 piruli kernel: da16: Fixed Direct = Access SCSI-6 device=20 Mar 27 13:49:40 piruli kernel: da16: Serial Number OCZ-SKC094QRU20F444P Mar 27 13:49:40 piruli kernel: da16: 150.000MB/s transfers Mar 27 13:49:40 piruli kernel: da16: 488386MB (1000215216 512 byte = sectors: 255H 63S/T 62260C) Mar 27 13:49:40 piruli kernel: da16: quirks=3D0x8<4K> # camcontrol inq da16 pass18: Fixed Direct Access SCSI-6 device=20 pass18: Serial Number OCZ-SKC094QRU20F444P pass18: 150.000MB/s transfers, Command Queueing Enabled Still the negotiation seems to be failing, but I wonder if it's more of = a firmware problem with the controller, expander or both. Cheers, Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Mar 27 16:25:15 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EBD0F76A; Thu, 27 Mar 2014 16:25:15 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) by mx1.freebsd.org (Postfix) with ESMTP id A1355638; Thu, 27 Mar 2014 16:25:15 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id B8BA39DD15F; Thu, 27 Mar 2014 17:25:06 +0100 (CET) Subject: Re: LSI - MR-Fusion controller driver patch and man page Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: Date: Thu, 27 Mar 2014 17:25:05 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> To: Doug Ambrisko X-Mailer: Apple Mail (2.1283) X-Mailman-Approved-At: Thu, 27 Mar 2014 16:34:51 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 16:25:16 -0000 On Mar 27, 2014, at 3:28 PM, Borja Marcos wrote: > # camcontrol inq da16 > pass18: Fixed Direct Access SCSI-6 device=20 > pass18: Serial Number OCZ-SKC094QRU20F444P > pass18: 150.000MB/s transfers, Command Queueing Enabled >=20 > Still the negotiation seems to be failing, but I wonder if it's more = of a firmware problem with the controller, expander or both. Nevermind, seems that the driver is lying :) case XPT_PATH_INQ: { ccb->cpi.version_num =3D 1; ccb->cpi.hba_inquiry =3D 0; ccb->cpi.target_sprt =3D 0; ccb->cpi.hba_misc =3D 0; ccb->cpi.hba_eng_cnt =3D 0; ccb->cpi.max_lun =3D MRSAS_SCSI_MAX_LUNS; ccb->cpi.unit_number =3D cam_sim_unit(sim); ccb->cpi.bus_id =3D cam_sim_bus(sim); ccb->cpi.initiator_id =3D MRSAS_SCSI_INITIATOR_ID;=20 ccb->cpi.base_transfer_speed =3D 150000;=20 strncpy(ccb->cpi.sim_vid, "FreeBSD", SIM_IDLEN); strncpy(ccb->cpi.hba_vid, "LSI", HBA_IDLEN); strncpy(ccb->cpi.dev_name, cam_sim_name(sim), DEV_IDLEN); ccb->cpi.transport =3D XPORT_SPI; ccb->cpi.transport_version =3D 2; ccb->cpi.protocol =3D PROTO_SCSI; Borja. From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 07:38:54 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5161C135; Fri, 28 Mar 2014 07:38:54 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0240.outbound.protection.outlook.com [207.46.163.240]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D76728AF; Fri, 28 Mar 2014 07:38:53 +0000 (UTC) Received: from BN1PR07MB247.namprd07.prod.outlook.com (10.141.64.146) by BN1PR07MB359.namprd07.prod.outlook.com (10.141.60.145) with Microsoft SMTP Server (TLS) id 15.0.908.10; Fri, 28 Mar 2014 07:38:45 +0000 Received: from BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.243]) by BN1PR07MB247.namprd07.prod.outlook.com ([169.254.7.243]) with mapi id 15.00.0898.005; Fri, 28 Mar 2014 07:38:44 +0000 From: "Desai, Kashyap" To: Borja Marcos , Doug Ambrisko Subject: RE: LSI - MR-Fusion controller driver patch and man page Thread-Topic: LSI - MR-Fusion controller driver patch and man page Thread-Index: Ac7ykKZzGPABTpK8R3+D5UAv3TD3WQWOC9SAAHVVybAAG8S5gAAWATMAABupEoADWG5MAAAAPT6AClUK/9AACzY8AAB4EYUAABFKigAAELyuAACF5P6AABRP6YAAkAQBAAAEDnaAAB8AlIA= Date: Fri, 28 Mar 2014 07:38:43 +0000 Message-ID: References: <20140107181139.GC2080@cisco.com> <20140124185356.GA28724@ambrisko.com> <20140124190047.GA34975@ambrisko.com> <9c3fd2b15e9b4c2cb967519a3b7f98ad@BN1PR07MB247.namprd07.prod.outlook.com> <20140318143738.GA65955@cisco.com> <20140320235534.GA92797@cisco.com> <20140321160954.GB99545@cisco.com> <5C32A3C7-B28B-4E69-9DF0-EE53181085F7@sarenet.es> <20140324174519.GA30345@cisco.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.19.224.250] x-forefront-prvs: 01644DCF4A x-forefront-antispam-report: SFV:NSPM; SFS:(10019001)(6009001)(428001)(377454003)(51704005)(54094003)(43544003)(24454002)(13464003)(199002)(189002)(95416001)(90146001)(33646001)(93516002)(74366001)(86362001)(95666003)(80022001)(97336001)(20776003)(66066001)(69226001)(31966008)(98676001)(56816005)(74316001)(65816001)(85306002)(92566001)(94316002)(93136001)(49866001)(83072002)(85852003)(74706001)(4396001)(74876001)(81342001)(94946001)(47976001)(56776001)(81542001)(50986001)(77096001)(76576001)(76786001)(76796001)(51856001)(83322001)(19580405001)(19580395003)(15202345003)(87936001)(46102001)(97186001)(81816001)(54316002)(54356001)(47736001)(79102001)(63696002)(81686001)(15975445006)(2656002)(74502001)(53806001)(59766001)(77982001)(80976001)(74662001)(47446002)(76482001)(87266001)(24736002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN1PR07MB359; H:BN1PR07MB247.namprd07.prod.outlook.com; FPR:6C524205.ACD2B5C9.33C170B7.44D74948.20323; MLV:sfv; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (: lsi.com does not designate permitted sender hosts) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lsi.com X-Mailman-Approved-At: Fri, 28 Mar 2014 11:33:09 +0000 Cc: "scottl@netflix.com" , "Radford, Adam" , "sean_bruno@yahoo.com" , "Mankani, Krishnaraddi" , "dwhite@ixsystems.com" , "Maloy, Joe" , "jpaetzel@freebsd.org" , "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 07:38:54 -0000 > -----Original Message----- > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > scsi@freebsd.org] On Behalf Of Borja Marcos > Sent: Thursday, March 27, 2014 9:55 PM > To: Doug Ambrisko > Cc: scottl@netflix.com; Radford, Adam; sean_bruno@yahoo.com; Mankani, > Krishnaraddi; dwhite@ixsystems.com; Maloy, Joe; jpaetzel@freebsd.org; > freebsd-scsi@freebsd.org; Kenneth D. Merry; McConnell, Stephen > Subject: Re: LSI - MR-Fusion controller driver patch and man page >=20 >=20 > On Mar 27, 2014, at 3:28 PM, Borja Marcos wrote: >=20 > > # camcontrol inq da16 > > pass18: Fixed Direct Access SCSI-6 device > > pass18: Serial Number OCZ-SKC094QRU20F444P > > pass18: 150.000MB/s transfers, Command Queueing Enabled > > > > Still the negotiation seems to be failing, but I wonder if it's more of= a > firmware problem with the controller, expander or both. >=20 > Nevermind, seems that the driver is lying :) >=20 > case XPT_PATH_INQ: > { > ccb->cpi.version_num =3D 1; > ccb->cpi.hba_inquiry =3D 0; > ccb->cpi.target_sprt =3D 0; > ccb->cpi.hba_misc =3D 0; > ccb->cpi.hba_eng_cnt =3D 0; > ccb->cpi.max_lun =3D MRSAS_SCSI_MAX_LUNS; > ccb->cpi.unit_number =3D cam_sim_unit(sim); > ccb->cpi.bus_id =3D cam_sim_bus(sim); > ccb->cpi.initiator_id =3D MRSAS_SCSI_INITIATOR_ID; > ccb->cpi.base_transfer_speed =3D 150000; This we need to take care in driver based on Device id bases... Possible ch= anges for this .. if ((sc->device_id =3D=3D MRSAS_INVADER) || (sc->device_id =3D=3D MRSAS_F= URY)) { ccb->cpi.base_transfer_speed =3D 150000; } else ccb->cpi.base_transfer_speed =3D 300000; Is the " base_transfer_speed" used in // driver correct ? I = am bit confused - what is a correct value to be used here ? Any idea ? > strncpy(ccb->cpi.sim_vid, "FreeBSD", SIM_IDLEN); > strncpy(ccb->cpi.hba_vid, "LSI", HBA_IDLEN); > strncpy(ccb->cpi.dev_name, cam_sim_name(sim), DEV_IDLEN); > ccb->cpi.transport =3D XPORT_SPI; > ccb->cpi.transport_version =3D 2; > ccb->cpi.protocol =3D PROTO_SCSI; >=20 >=20 >=20 >=20 >=20 > Borja. >=20 > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 18:58:46 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A68E4679 for ; Fri, 28 Mar 2014 18:58:46 +0000 (UTC) Received: from gelt.unixathome.org (gelt.unixathome.org [64.90.182.98]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "gelt.unixathome.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6204E617 for ; Fri, 28 Mar 2014 18:58:46 +0000 (UTC) Received: from gelt.unixathome.org (localhost [127.0.0.1]) by gelt.unixathome.org (Postfix) with ESMTP id A0E6417036 for ; Fri, 28 Mar 2014 18:58:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from gelt.unixathome.org ([127.0.0.1]) by gelt.unixathome.org (gelt.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TBSESubgis4q for ; Fri, 28 Mar 2014 18:58:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by gelt.unixathome.org (Postfix) with ESMTP id 5AB5817029 for ; Fri, 28 Mar 2014 18:58:42 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 28 Mar 2014 14:58:41 -0400 From: Dan Langille To: freebsd-scsi@freebsd.org Subject: SYM8951U sees ch0 but not sa0 Organization: The FreeBSD Diary Message-ID: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> X-Sender: dan@langille.org User-Agent: Roundcube Webmail/0.9.5 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 18:58:46 -0000 I have installed a Symbios Ultra2 32-bit PCI SCSI Adapter SYM8951U in my FreeBSD 9.2 system. Attached to it is a Compaq StorageWorks MSL5026 tape library. I have never used this library before, but it was given to me by someone who did use it. It worked at one time. The system sees the changer, but not the drive. I've tried swapping the SCSI cable and the LVD/SE terminator on the tape library ports. Same result. After each cable retry, I run 'camcontrol rescan all'. [dan@knew:/dev] $ ls *sa* *ch* ls: *sa*: No such file or directory ch0 [dan@knew:/dev] $ Any ideas? Suggestions? Card settings? I conclude the cable is OK, given that I can get a list of the tapes in the library. The front panel of the library sees the tape drive. It has a SCSI ID assigned. $ sudo camcontrol amcontrol devlist at scbus7 target 3 lun 0 (pass10,ch0) (NOTE: I have removed the SATA drives from the camcontrol output) From dmesg, I see: sym0: <895> port 0xe000-0xe0ff mem 0xfebeec00-0xfebeecff,0xfebef000-0xfebeffff irq 22 at device 2.0 on pci4 sym0: Symbios NVRAM, ID 7, Fast-40, LVD, parity checking sym0: open drain IRQ line driver, using on-chip SRAM sym0: using LOAD/STORE-based firmware. ... uhub4: 3 ports with 3 removable, self powered (noperiph:sym0:0:-1:-1): SCSI BUS reset delivered. uhub2: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered ch0 at sym0 bus 0 scbus7 target 3 lun 0 ch0: Removable Changer SCSI-2 device ch0: 20.000MB/s transfers (10.000MHz, offset 15, 16bit) ch0: 25 slots, 1 drive, 1 picker, 1 portal ch0: quirks=0x2 da1 at mps0 bus 0 scbus0 target 10 lun 0 ... sym0 - the card I just installed ch0 - the tape changer I can get a list of the tapes: $ sudo mtx -f /dev/pass10 status Storage Changer /dev/pass10:1 Drives, 26 Slots ( 1 Import/Export ) Data Transfer Element 0:Empty Storage Element 1:Full :VolumeTag=TWZ040 Storage Element 2:Full :VolumeTag=TWZ039 Storage Element 3:Full :VolumeTag=TWZ044 Storage Element 4:Full :VolumeTag=TWZ043 Storage Element 5:Full :VolumeTag=TWZ042 Storage Element 6:Full :VolumeTag=TWZ041 Storage Element 7:Full :VolumeTag=TWZ046 Storage Element 8:Full :VolumeTag=TWZ045 Storage Element 9:Full :VolumeTag=TWZ047 Storage Element 10:Full :VolumeTag=TWZ049 Storage Element 11:Full :VolumeTag=TWZ050 Storage Element 12:Full :VolumeTag=TWZ051 Storage Element 13:Full :VolumeTag=TWZ061 Storage Element 14:Full :VolumeTag=TWZ060 Storage Element 15:Full :VolumeTag=TWZ065 Storage Element 16:Full :VolumeTag=TWZ064 Storage Element 17:Full :VolumeTag=TWZ062 Storage Element 18:Full :VolumeTag=TWZ048 Storage Element 19:Full :VolumeTag=TWZ059 Storage Element 20:Full :VolumeTag=TWZ058 Storage Element 21:Full :VolumeTag=TWZ057 Storage Element 22:Full :VolumeTag=TWZ055 Storage Element 23:Full :VolumeTag=TWZ054 Storage Element 24:Full :VolumeTag=TWZ053 Storage Element 25:Full :VolumeTag=TWZ067 Storage Element 26 IMPORT/EXPORT:Empty -- Dan Langille - http://langille.org/ From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 19:04:35 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BBADA76D for ; Fri, 28 Mar 2014 19:04:35 +0000 (UTC) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 851F16D2 for ; Fri, 28 Mar 2014 19:04:35 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id s2SIrWmu072867; Fri, 28 Mar 2014 12:53:32 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id s2SIrWtV072866; Fri, 28 Mar 2014 12:53:32 -0600 (MDT) (envelope-from ken) Date: Fri, 28 Mar 2014 12:53:32 -0600 From: "Kenneth D. Merry" To: bharat singh Subject: Re: Adding write_same16 to CAM Target Layer Message-ID: <20140328185331.GA72411@nargothrond.kdm.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 19:04:35 -0000 On Wed, Mar 26, 2014 at 11:44:32 +0530, bharat singh wrote: > Hello, > > I am trying to add write_same16 (block zero) support to my FC stack. > This i am trying to achieve by adding a hook in CTL layer, reading > start_LBA and no of blocks to fill with 0. > > struct ctl_scsiio { > >-------struct ctl_io_hdr io_hdr;>------/* common to all I/O types */ > >-------uint32_t ext_sg_entries;>-----/* 0 = no S/G list, > 0 = num > entries */ > >-------uint8_t> *ext_data_ptr;>------/* data buffer or S/G list */ > >-------uint32_t ext_data_len;>-------/* Data transfer length */ > >-------uint32_t ext_data_filled;>----/* Amount of data filled so far */ > >-------uint32_t kern_sg_entries;>----/* 0 = no S/G list, > 0 = num > entries */ > >-------uint32_t rem_sg_entries;>-----/* 0 = no S/G list, > 0 = num > entries */ > >-------uint8_t *kern_data_ptr;>-----/* data buffer or S/G list */ > >-------uint32_t kern_data_len;>------/* Length of this S/G list/buffer */ > >-------uint32_t kern_total_len;>-----/* Total length of this transaction > */ > >-------uint32_t kern_data_resid;>----/* Length left to transfer after > this*/ > >-------uint32_t kern_rel_offset;>----/* Byte Offset of this transfer */ > >-------struct scsi_sense_data sense_data;>-/* sense data */ > >-------uint8_t> sense_len;>-->-------/* Returned sense length */ > >-------uint8_t> scsi_status;>>-------/* SCSI status byte */ > >-------uint8_t> sense_residual;>-----/* sense residual length */ > >-------uint32_t residual;>--->-------/* data residual length */ > >-------uint32_t tag_num;>---->-------/* tag number */ > >-------ctl_tag_type tag_type;>->-------/* simple, ordered, head of > queue,etc.*/ > >-------uint8_t cdb_len;>---->-------/* CDB length */ > >-------uint8_t> cdb[CTL_MAX_CDBLEN];>/* CDB */ > >-------int>---- (*be_move_done)(union ctl_io *io); /* called by fe */ > >-------int (*io_cont)(union ctl_io *io); /* to continue processing > */ > >-------uint32_t ctl_lun_masking_disabled; /* Disable lun mask checks for > ctladm calls */ > }; > > For write_same16 initiators will send a block of 0s with start_LBA and no > of blocks. Assuming that I am trying to: > datalen = blocksize * num_blocks; > uint8_t * databuf = malloc(datalen); > bzero(databuf, 0, datalen); > ctsio->kern_data_ptr = dataptr; > ctsio->kern_data_len = datalen; > lbalen.lba = lba; > lbalen.len = num_blocks; > memcpy(ctsio->io_hdr.ctl_private[CTL_PRIV_LBA_LEN].bytes, &lbalen, > sizeof(lbalen)); > ret = lun->backend->data_submit((union ctl_io *)ctsio); > > but it's failing in ISP layer with data overflow. My initiator is sending > 2048 blocks of size 512B each and a payload of 512 bytes in data-out buffer. > isp0: isp_target_start_ctio: [0x120154] data overflow by 1048064 bytes > (1:3:0:1): WRITE SAME(16). CDB: 93 00 00 00 00 00 00 1f e8 00 00 00 08 00 > 00 00 > > Have any one done changes to the CTL read/write path, am I missing > something regarding buffer population. > Thanks for the help in advance. What back end are you using? The block backend, or are you writing your own? In the case of the block backend, it is expecting to do its own memory allocation and then call ctl_datamove() to trigger the DMA. In this case, you're doing a malloc, but if you're going to the block backend the data buffer will get overwritten. As for why you're getting an overrun, the amount of the overrun is 1048064 bytes. That is 512 less than 1048576. Is your blocksize and number of blocks set correctly? It looks like datalen is probably 512, but your initiator is sending 1MB. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 20:46:48 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DEF39341 for ; Fri, 28 Mar 2014 20:46:48 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5F80551 for ; Fri, 28 Mar 2014 20:46:48 +0000 (UTC) Received: from mandree.no-ip.org ([78.53.178.95]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LraSn-1XCH5u3nY5-013QE3 for ; Fri, 28 Mar 2014 21:46:40 +0100 Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id DB61223CEA9 for ; Fri, 28 Mar 2014 21:46:38 +0100 (CET) Message-ID: <5335DFAE.1050109@gmx.de> Date: Fri, 28 Mar 2014 21:46:38 +0100 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: SYM8951U sees ch0 but not sa0 References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> In-Reply-To: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:tvGkVqSqWVFTpq4Mic9pl3+p5A15g4Fd8eNErUGqnI0eZzgF5pQ fBa40TAOMtEi4AQns3THcq7NtBFZaEGzpHMLieoviFK+RRPQ7PHFCkHe9UP1/VZGW4H4i8X dtD1W0zF7BjaaQp/A7fLxL6m3F8zePGk/YpSzj7h1QTDArCKqBs7G4j0q4nVal9+KDyrNcp /K6gpRIz2CuO0EFOXcXHg== X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 20:46:48 -0000 Am 28.03.2014 19:58, schrieb Dan Langille: > I have installed a Symbios Ultra2 32-bit PCI SCSI Adapter SYM8951U in my > FreeBSD 9.2 system. Attached to it is a Compaq StorageWorks MSL5026 > tape library. I have never used this library before, but it was given > to me by someone who did use it. It worked at one time. > > The system sees the changer, but not the drive. I've tried swapping the > SCSI cable and the LVD/SE terminator on the tape library ports. Same > result. After each cable retry, I run 'camcontrol rescan all'. > > [dan@knew:/dev] $ ls *sa* *ch* > ls: *sa*: No such file or directory > ch0 > [dan@knew:/dev] $ > > Any ideas? Suggestions? Card settings? > > I conclude the cable is OK, given that I can get a list of the tapes in > the library. > > The front panel of the library sees the tape drive. It has a SCSI ID > assigned. I haven't much experience with multi-LUN devices, nor tape libraries, but OTOH I haven't had issues with Symbios (NCR, LSI) controllers + bare tape drives so far [*] - Is that drive's ID a separate target ID, or is it a LUN? + If it's a LUN, is it <= 7? + If it's a LUN, do the SYMBIOS settings permit scanning for LUNs at all? (Although it's not entirely clear to me from sym(4) if rescan would scan LUNs.) - Does camcontrol reportluns help any? - Can you have the host adaptor's BIOS scan the BUS and report devices and LUNs and see if that differs from FreeBSD's view of the world? __________ [*] One exception though: Tekram DC-390U were using the Ultra+Wide 53C875 chip but an 8-bit connector and used to cause hitches when connecting UW devies, at least with older drivers. I forgot if I ever checked whether this was ever fixed... From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 20:59:49 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BBBF929 for ; Fri, 28 Mar 2014 20:59:49 +0000 (UTC) Received: from gelt.unixathome.org (gelt.unixathome.org [64.90.182.98]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "gelt.unixathome.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F85E17C for ; Fri, 28 Mar 2014 20:59:48 +0000 (UTC) Received: from gelt.unixathome.org (localhost [127.0.0.1]) by gelt.unixathome.org (Postfix) with ESMTP id 2430417036 for ; Fri, 28 Mar 2014 20:59:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from gelt.unixathome.org ([127.0.0.1]) by gelt.unixathome.org (gelt.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yuQpNoJ80aN9 for ; Fri, 28 Mar 2014 20:59:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by gelt.unixathome.org (Postfix) with ESMTP id 9F67C17029 for ; Fri, 28 Mar 2014 20:59:45 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 28 Mar 2014 16:59:44 -0400 From: Dan Langille To: freebsd-scsi@freebsd.org Subject: Re: SYM8951U sees ch0 but not sa0 Organization: The FreeBSD Diary In-Reply-To: <5335DFAE.1050109@gmx.de> References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> <5335DFAE.1050109@gmx.de> Message-ID: <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> X-Sender: dan@langille.org User-Agent: Roundcube Webmail/0.9.5 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 20:59:49 -0000 On 2014-03-28 04:46 PM, Matthias Andree wrote: > Am 28.03.2014 19:58, schrieb Dan Langille: >> I have installed a Symbios Ultra2 32-bit PCI SCSI Adapter SYM8951U in >> my >> FreeBSD 9.2 system. Attached to it is a Compaq StorageWorks MSL5026 >> tape library. I have never used this library before, but it was given >> to me by someone who did use it. It worked at one time. >> >> The system sees the changer, but not the drive. I've tried swapping >> the >> SCSI cable and the LVD/SE terminator on the tape library ports. Same >> result. After each cable retry, I run 'camcontrol rescan all'. >> >> [dan@knew:/dev] $ ls *sa* *ch* >> ls: *sa*: No such file or directory >> ch0 >> [dan@knew:/dev] $ >> >> Any ideas? Suggestions? Card settings? >> >> I conclude the cable is OK, given that I can get a list of the tapes >> in >> the library. >> >> The front panel of the library sees the tape drive. It has a SCSI ID >> assigned. > > I haven't much experience with multi-LUN devices, nor tape libraries, > but OTOH I haven't had issues with Symbios (NCR, LSI) controllers + > bare > tape drives so far [*] > > - Is that drive's ID a separate target ID, or is it a LUN? > + If it's a LUN, is it <= 7? > + If it's a LUN, do the SYMBIOS settings permit scanning for LUNs at > all? (Although it's not entirely clear to me from sym(4) if rescan > would > scan LUNs.) I do not know, but I will check when I am next at the tape library. I wonder if that's a configuration item. FWIW, the previous owner of this tape library was not running FreeBSD, I'm sure. > - Does camcontrol reportluns help any? $ sudo camcontrol reportluns 7:3:0 1 LUN found 0 > - Can you have the host adaptor's BIOS scan the BUS and report devices > and LUNs and see if that differs from FreeBSD's view of the world? I did that the first time I booted the system. It may have been without the terminators attached. I photographed the console screen at the time The host adaptor's BIOS shows only the MSL5000, then, while booting, the Symbios displayed the MSL5000 as ID 3 and the card as ID 7. > __________ > [*] One exception though: Tekram DC-390U were using the Ultra+Wide > 53C875 chip but an 8-bit connector and used to cause hitches when > connecting UW devies, at least with older drivers. I forgot if I ever > checked whether this was ever fixed... It's knowledge like that which makes group lists so valuable. -- Dan Langille - http://langille.org/ From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 21:31:25 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 622CD50D for ; Fri, 28 Mar 2014 21:31:25 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE58D6DE for ; Fri, 28 Mar 2014 21:31:24 +0000 (UTC) Received: from mandree.no-ip.org ([78.53.178.95]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0M4CB5-1XKW0M0ULr-00rpfs for ; Fri, 28 Mar 2014 22:31:17 +0100 Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id 51D5623CEA9 for ; Fri, 28 Mar 2014 22:31:16 +0100 (CET) Message-ID: <5335EA24.4080406@gmx.de> Date: Fri, 28 Mar 2014 22:31:16 +0100 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: SYM8951U sees ch0 but not sa0 References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> <5335DFAE.1050109@gmx.de> <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> In-Reply-To: <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:H1iHzX6P9IR8rbDEqXBfjPfYMPjQzLmwLicdqFM8GqXQjPxLMsz tZb2xF7jJEf3Fy5lqGSJce2axQx0n+EPi2MAyBHiAjJB0wYgjrXHLsrYAidmh57AnXMKoqu mabJ77lfI4cyr4dT/FiYuHc+A9AEHFOyNpEp1/Fv2fz+SvZRbccqQfJ0YhI85NoN03wpLB3 uqSa/gP5/tQxBEy+KkZUw== X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 21:31:25 -0000 Am 28.03.2014 21:59, schrieb Dan Langille: >> - Does camcontrol reportluns help any? > > $ sudo camcontrol reportluns 7:3:0 > 1 LUN found > 0 Sorry for distracting you on the LUNs. The manual for HP StorageWorks MSL5000 and MSL6000 Series Tape Libraries clearly states the drives are individual SCSI targets. found as 3rd from the BOTTOM of Apparently (see page 199 in the PDF) there is no internal SCSI wiring, you need to add all SCSI wires by yourself - i. e. drives must be daisy-chained to the library controller. Can you see the drive if you move the SCSI cable and terminator from library to drive, or if you daisy-chain things? From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 21:55:17 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6BFE11E5; Fri, 28 Mar 2014 21:55:17 +0000 (UTC) Received: from gelt.unixathome.org (gelt.unixathome.org [64.90.182.98]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "gelt.unixathome.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3E4D6A50; Fri, 28 Mar 2014 21:55:17 +0000 (UTC) Received: from gelt.unixathome.org (localhost [127.0.0.1]) by gelt.unixathome.org (Postfix) with ESMTP id 7A85117036; Fri, 28 Mar 2014 21:55:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from gelt.unixathome.org ([127.0.0.1]) by gelt.unixathome.org (gelt.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nSdd31x2bxjl; Fri, 28 Mar 2014 21:55:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by gelt.unixathome.org (Postfix) with ESMTP id 6AEEE1701F; Fri, 28 Mar 2014 21:55:13 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 28 Mar 2014 17:55:12 -0400 From: Dan Langille To: Matthias Andree Subject: Re: SYM8951U sees ch0 but not sa0 Organization: The FreeBSD Diary In-Reply-To: <5335EA24.4080406@gmx.de> References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> <5335DFAE.1050109@gmx.de> <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> <5335EA24.4080406@gmx.de> Message-ID: X-Sender: dan@langille.org User-Agent: Roundcube Webmail/0.9.5 Cc: freebsd-scsi@freebsd.org, owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 21:55:17 -0000 On 2014-03-28 05:31 PM, Matthias Andree wrote: > Am 28.03.2014 21:59, schrieb Dan Langille: >>> - Does camcontrol reportluns help any? >> >> $ sudo camcontrol reportluns 7:3:0 >> 1 LUN found >> 0 > > Sorry for distracting you on the LUNs. > > The manual for HP StorageWorks MSL5000 and MSL6000 Series Tape > Libraries > clearly states the drives are individual SCSI targets. > > > found as > 3rd from the BOTTOM of > > > > Apparently (see page 199 in the PDF) there is no internal SCSI wiring, > you need to add all SCSI wires by yourself - i. e. drives must be > daisy-chained to the library controller. > > Can you see the drive if you move the SCSI cable and terminator from > library to drive, or if you daisy-chain things? I did try swapping the cable and the terminator between the two connectors on the rear of the library. I ran 'camcontrol rescan all' and 'camcontrol devlist' after each swap. Never saw the drive. By daisy-chain things, there are only two connectors on the rear, so I'm not sure what else I could try, other than the above. -- Dan Langille - http://langille.org/ From owner-freebsd-scsi@FreeBSD.ORG Fri Mar 28 23:33:31 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF98E672 for ; Fri, 28 Mar 2014 23:33:31 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66A9231B for ; Fri, 28 Mar 2014 23:33:31 +0000 (UTC) Received: from mandree.no-ip.org ([78.53.178.95]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0MaIw0-1WnYbX449g-00Jq9v; Sat, 29 Mar 2014 00:33:20 +0100 Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id 47A1323CEA9; Sat, 29 Mar 2014 00:33:18 +0100 (CET) Message-ID: <533606BE.3040502@gmx.de> Date: Sat, 29 Mar 2014 00:33:18 +0100 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Dan Langille Subject: Re: SYM8951U sees ch0 but not sa0 References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> <5335DFAE.1050109@gmx.de> <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> <5335EA24.4080406@gmx.de> In-Reply-To: X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:dDtXZiI0pVSHlFsD3JbTqmZD1+czD3Fnmdmi/JHpr7O+bWSjUw6 zsL9n1acxKysMJRG91VyIBBNtaToywKL2c+owDn4+bPmbSDNCAPeY3/3ObKbi5Cfox/AuE5 rk/DlU7l83fkjFhmi+rbWjOi67Y/pjmDBSsJ4QbY2hy5162VkjbW8LPH3vJIwS38p+u7ptT T1Mp2fXegKj9eaKT74pqQ== Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2014 23:33:31 -0000 Am 28.03.2014 22:55, schrieb Dan Langille: > I did try swapping the cable and the terminator between the two > connectors on the rear of the library. I ran 'camcontrol rescan all' > and 'camcontrol devlist' after each swap. Never saw the drive. > > By daisy-chain things, there are only two connectors on the rear, so I'm > not sure what else I could try, other than the above. Two are at least two too few. Sorry for the pun, I am not trying to make fun here. Did you get the library with at least one drive fitted? Check the drawings on pages 21, 199 and 201 in the PDF I mentioned: There should be 2 SCSI connector sockets for the library, and 2 more SCSI connector sockets for each tape drive. So an MSL5026 with two tape drives should have six SCSI connectors in total. They are at the very bottom end, possibly hidden under the protruding back part of the drives, so be sure to check with the eyes at the same level as the bottom of the library. HTH From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 29 00:27:40 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F81D57C; Sat, 29 Mar 2014 00:27:40 +0000 (UTC) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE0F5A08; Sat, 29 Mar 2014 00:27:39 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id oz11so6358536veb.6 for ; Fri, 28 Mar 2014 17:27:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gYrvypz/vAAdSmfviH6uMJ4upNwucnW82XaEbvN2B34=; b=VOLTJYTa6hGju1tN1wb68lOLQux9VSS2mR2JLnIcjfgMf3h+XrnF1cWnj7Uv2Qamn7 pIdA/anyIWStm6l2gsZgJD6R0602aFCG5pKbRhvBlIGwscZmIP5F+sIHdcB/rxFD4Hoz p6hkyI1YXEvzPWEzT/FJeMBPxTZ0mOyWwKcX3thTFYKfVFtMIa39oxOAIu1NEoNteWNC WcKoeEH7QpfCVoUZnMJK2Zfn7Y/eRG0TeVdz4jOZ9P/QtEy7QnS8GTx7Q4GVFD0alt7r OqSRXzS46Ugk9Uht0fCe6ZFmjRI68iE4gDZVSYgHhGFBN+4Iis1s8Nm5LtjmpcrdQh01 KwnQ== MIME-Version: 1.0 X-Received: by 10.220.104.210 with SMTP id q18mr9874637vco.9.1396052858833; Fri, 28 Mar 2014 17:27:38 -0700 (PDT) Received: by 10.58.108.10 with HTTP; Fri, 28 Mar 2014 17:27:38 -0700 (PDT) Received: by 10.58.108.10 with HTTP; Fri, 28 Mar 2014 17:27:38 -0700 (PDT) In-Reply-To: <20140328185331.GA72411@nargothrond.kdm.org> References: <20140328185331.GA72411@nargothrond.kdm.org> Date: Sat, 29 Mar 2014 05:57:38 +0530 Message-ID: Subject: Re: Adding write_same16 to CAM Target Layer From: bharat singh To: "Kenneth D. Merry" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2014 00:27:40 -0000 I am using block back end with block size 512bytes. Yes as per t10 doc initiator will send 512bytes of data to be filled over 1MB. Block size is 512 and no of blocks is 2048 in my case. >From CTL layer can I reallocate the data buffers. This command is needed for initiators hosted on ESXi. On 29-Mar-2014 12:23 AM, "Kenneth D. Merry" wrote: > On Wed, Mar 26, 2014 at 11:44:32 +0530, bharat singh wrote: > > Hello, > > > > I am trying to add write_same16 (block zero) support to my FC stack. > > This i am trying to achieve by adding a hook in CTL layer, reading > > start_LBA and no of blocks to fill with 0. > > > > struct ctl_scsiio { > > >-------struct ctl_io_hdr io_hdr;>------/* common to all I/O types */ > > >-------uint32_t ext_sg_entries;>-----/* 0 = no S/G list, > 0 = num > > entries */ > > >-------uint8_t> *ext_data_ptr;>------/* data buffer or S/G list */ > > >-------uint32_t ext_data_len;>-------/* Data transfer length */ > > >-------uint32_t ext_data_filled;>----/* Amount of data filled so far > */ > > >-------uint32_t kern_sg_entries;>----/* 0 = no S/G list, > 0 = num > > entries */ > > >-------uint32_t rem_sg_entries;>-----/* 0 = no S/G list, > 0 = num > > entries */ > > >-------uint8_t *kern_data_ptr;>-----/* data buffer or S/G list */ > > >-------uint32_t kern_data_len;>------/* Length of this S/G > list/buffer */ > > >-------uint32_t kern_total_len;>-----/* Total length of this > transaction > > */ > > >-------uint32_t kern_data_resid;>----/* Length left to transfer after > > this*/ > > >-------uint32_t kern_rel_offset;>----/* Byte Offset of this transfer > */ > > >-------struct scsi_sense_data sense_data;>-/* sense data */ > > >-------uint8_t> sense_len;>-->-------/* Returned sense length */ > > >-------uint8_t> scsi_status;>>-------/* SCSI status byte */ > > >-------uint8_t> sense_residual;>-----/* sense residual length */ > > >-------uint32_t residual;>--->-------/* data residual length */ > > >-------uint32_t tag_num;>---->-------/* tag number */ > > >-------ctl_tag_type tag_type;>->-------/* simple, ordered, head of > > queue,etc.*/ > > >-------uint8_t cdb_len;>---->-------/* CDB length */ > > >-------uint8_t> cdb[CTL_MAX_CDBLEN];>/* CDB */ > > >-------int>---- (*be_move_done)(union ctl_io *io); /* called by fe */ > > >-------int (*io_cont)(union ctl_io *io); /* to continue > processing > > */ > > >-------uint32_t ctl_lun_masking_disabled; /* Disable lun mask checks > for > > ctladm calls */ > > }; > > > > For write_same16 initiators will send a block of 0s with start_LBA and no > > of blocks. Assuming that I am trying to: > > datalen = blocksize * num_blocks; > > uint8_t * databuf = malloc(datalen); > > bzero(databuf, 0, datalen); > > ctsio->kern_data_ptr = dataptr; > > ctsio->kern_data_len = datalen; > > lbalen.lba = lba; > > lbalen.len = num_blocks; > > memcpy(ctsio->io_hdr.ctl_private[CTL_PRIV_LBA_LEN].bytes, &lbalen, > > sizeof(lbalen)); > > ret = lun->backend->data_submit((union ctl_io *)ctsio); > > > > but it's failing in ISP layer with data overflow. My initiator is sending > > 2048 blocks of size 512B each and a payload of 512 bytes in data-out > buffer. > > isp0: isp_target_start_ctio: [0x120154] data overflow by 1048064 bytes > > (1:3:0:1): WRITE SAME(16). CDB: 93 00 00 00 00 00 00 1f e8 00 00 00 08 00 > > 00 00 > > > > Have any one done changes to the CTL read/write path, am I missing > > something regarding buffer population. > > Thanks for the help in advance. > > What back end are you using? The block backend, or are you writing your > own? > > In the case of the block backend, it is expecting to do its own memory > allocation and then call ctl_datamove() to trigger the DMA. > > In this case, you're doing a malloc, but if you're going to the block > backend the data buffer will get overwritten. > > As for why you're getting an overrun, the amount of the overrun is 1048064 > bytes. That is 512 less than 1048576. Is your blocksize and number of > blocks set correctly? It looks like datalen is probably 512, but your > initiator is sending 1MB. > > Ken > -- > Kenneth Merry > ken@FreeBSD.ORG > From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 29 00:34:44 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F4816FA for ; Sat, 29 Mar 2014 00:34:44 +0000 (UTC) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "nyi.unixathome.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 50FFAAD9 for ; Sat, 29 Mar 2014 00:34:43 +0000 (UTC) Received: from nyi.unixathome.org (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id DFF6750830; Sat, 29 Mar 2014 00:34:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by nyi.unixathome.org (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PAW32q12hMi6; Sat, 29 Mar 2014 00:34:32 +0000 (UTC) Received: from smtp-auth.unixathome.org (smtp-auth.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) by nyi.unixathome.org (Postfix) with ESMTPSA id 756595082E ; Sat, 29 Mar 2014 00:34:32 +0000 (UTC) References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> <5335DFAE.1050109@gmx.de> <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> <5335EA24.4080406@gmx.de> <533606BE.3040502@gmx.de> Mime-Version: 1.0 (1.0) In-Reply-To: <533606BE.3040502@gmx.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <79F1210A-FFEB-4319-8516-05DDCBCB4398@langille.org> X-Mailer: iPhone Mail (11D167) From: Dan Langille Subject: Re: SYM8951U sees ch0 but not sa0 Date: Fri, 28 Mar 2014 20:34:28 -0400 To: Matthias Andree Cc: "freebsd-scsi@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2014 00:34:44 -0000 On Mar 28, 2014, at 7:33 PM, Matthias Andree wrote:= >=20 > Am 28.03.2014 22:55, schrieb Dan Langille: >=20 >> I did try swapping the cable and the terminator between the two >> connectors on the rear of the library. I ran 'camcontrol rescan all' >> and 'camcontrol devlist' after each swap. Never saw the drive. >>=20 >> By daisy-chain things, there are only two connectors on the rear, so I'm >> not sure what else I could try, other than the above. >=20 > Two are at least two too few. >=20 > Sorry for the pun, I am not trying to make fun here. >=20 > Did you get the library with at least one drive fitted? Check the > drawings on pages 21, 199 and 201 in the PDF I mentioned: > There should be 2 SCSI connector sockets for the library, and 2 more > SCSI connector sockets for each tape drive. So an MSL5026 with two tape > drives should have six SCSI connectors in total. They are at the very > bottom end, possibly hidden under the protruding back part of the > drives, so be sure to check with the eyes at the same level as the > bottom of the library. >=20 > HTH >=20 The front panel claims there is a drive installed. I will look closer. But i= nbet you're right. Thank you.=20 --=20 Dan Langille http://langille.org/= From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 29 14:27:35 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4C72657 for ; Sat, 29 Mar 2014 14:27:35 +0000 (UTC) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "nyi.unixathome.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A212F7FB for ; Sat, 29 Mar 2014 14:27:35 +0000 (UTC) Received: from nyi.unixathome.org (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id 511BA50830 for ; Sat, 29 Mar 2014 14:27:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by nyi.unixathome.org (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2cwTtd-H7Ms1 for ; Sat, 29 Mar 2014 14:27:33 +0000 (UTC) Received: from smtp-auth.unixathome.org (smtp-auth.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) by nyi.unixathome.org (Postfix) with ESMTPSA id D7DCA5082E for ; Sat, 29 Mar 2014 14:27:32 +0000 (UTC) From: Dan Langille Content-Type: multipart/signed; boundary="Apple-Mail=_2D1AE9A5-C008-4D45-AE6F-50AB2F35E867"; protocol="application/pgp-signature"; micalg=pgp-sha1 Message-Id: Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: Anyone have a SCSI terminator LVD/SE VHDCI? Date: Sat, 29 Mar 2014 10:27:27 -0400 References: <3C49CB15-C457-41E4-AF0D-521EAFA34AAE@langille.org> To: freebsd-scsi@freebsd.org In-Reply-To: <3C49CB15-C457-41E4-AF0D-521EAFA34AAE@langille.org> X-Mailer: Apple Mail (2.1874) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2014 14:27:35 -0000 --Apple-Mail=_2D1AE9A5-C008-4D45-AE6F-50AB2F35E867 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Mar 22, 2014, at 12:52 PM, Dan Langille wrote: > I=92m trying to locate a SCSI terminator LVD/SE VHDCI. Got one laying = around, unloved? I have one. 6, actually. Thanks. --=20 Dan Langille - http://langille.org --Apple-Mail=_2D1AE9A5-C008-4D45-AE6F-50AB2F35E867 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlM22F4ACgkQCgsXFM/7nTxv8ACgt4Gu1OQAVpFLUiHEiF9cJz9a fZUAn1CmhzXlJLILFj1xn1Tt4VQ1WGFj =LE+4 -----END PGP SIGNATURE----- --Apple-Mail=_2D1AE9A5-C008-4D45-AE6F-50AB2F35E867-- From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 29 14:33:05 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1E839984 for ; Sat, 29 Mar 2014 14:33:05 +0000 (UTC) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "nyi.unixathome.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E15BA8B0 for ; Sat, 29 Mar 2014 14:33:04 +0000 (UTC) Received: from nyi.unixathome.org (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id BF99550830; Sat, 29 Mar 2014 14:33:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by nyi.unixathome.org (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SbeEnZwHMDrS; Sat, 29 Mar 2014 14:33:03 +0000 (UTC) Received: from smtp-auth.unixathome.org (smtp-auth.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) by nyi.unixathome.org (Postfix) with ESMTPSA id 312825082E ; Sat, 29 Mar 2014 14:33:03 +0000 (UTC) Content-Type: multipart/signed; boundary="Apple-Mail=_83BF1962-B17D-4461-803C-B8476CA48652"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: SYM8951U sees ch0 but not sa0 From: Dan Langille In-Reply-To: <533606BE.3040502@gmx.de> Date: Sat, 29 Mar 2014 10:33:12 -0400 Message-Id: References: <831365dbd2a006fd36b7a1465d33fe00@mailjail.langille.org> <5335DFAE.1050109@gmx.de> <0406d95c101f43765e03272ac1bbd62b@mailjail.langille.org> <5335EA24.4080406@gmx.de> <533606BE.3040502@gmx.de> To: Matthias Andree X-Mailer: Apple Mail (2.1874) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2014 14:33:05 -0000 --Apple-Mail=_83BF1962-B17D-4461-803C-B8476CA48652 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Mar 28, 2014, at 7:33 PM, Matthias Andree wrote: > Am 28.03.2014 22:55, schrieb Dan Langille: > >> I did try swapping the cable and the terminator between the two >> connectors on the rear of the library. I ran 'camcontrol rescan all' >> and 'camcontrol devlist' after each swap. Never saw the drive. >> >> By daisy-chain things, there are only two connectors on the rear, so I'm >> not sure what else I could try, other than the above. > > Two are at least two too few. > > Sorry for the pun, I am not trying to make fun here. > > Did you get the library with at least one drive fitted? Check the > drawings on pages 21, 199 and 201 in the PDF I mentioned: > There should be 2 SCSI connector sockets for the library, and 2 more > SCSI connector sockets for each tape drive. So an MSL5026 with two tape > drives should have six SCSI connectors in total. They are at the very > bottom end, possibly hidden under the protruding back part of the > drives, so be sure to check with the eyes at the same level as the > bottom of the library. That was it. Thank you. at scbus7 target 1 lun 0 (pass12,sa1) at scbus7 target 3 lun 0 (pass10,ch0) The lessons: * remember what you did in the past * read the manual * change your perspective Much appreciated. :) -- Dan Langille - http://langille.org --Apple-Mail=_83BF1962-B17D-4461-803C-B8476CA48652 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlM22agACgkQCgsXFM/7nTxbqgCfTQzpwMY25RNfeLoOvCyo4FC7 stoAn01JAfRkF0GCdtjd96/9HgayiPWl =NuBq -----END PGP SIGNATURE----- --Apple-Mail=_83BF1962-B17D-4461-803C-B8476CA48652-- From owner-freebsd-scsi@FreeBSD.ORG Sat Mar 29 21:15:01 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5270E33B; Sat, 29 Mar 2014 21:15:01 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27D6CCBA; Sat, 29 Mar 2014 21:15:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2TLF15i005532; Sat, 29 Mar 2014 21:15:01 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2TLF0rg005531; Sat, 29 Mar 2014 21:15:00 GMT (envelope-from linimon) Date: Sat, 29 Mar 2014 21:15:00 GMT Message-Id: <201403292115.s2TLF0rg005531@freefall.freebsd.org> To: linimon@FreeBSD.org, mjacob@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/127927: [isp] isp(4) target driver crashes kernel when set up dma for CTIO2 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2014 21:15:01 -0000 Synopsis: [isp] isp(4) target driver crashes kernel when set up dma for CTIO2 Responsible-Changed-From-To: mjacob->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Sat Mar 29 21:14:27 UTC 2014 Responsible-Changed-Why: mjacob's bit has been returned for safekeeping. http://www.freebsd.org/cgi/query-pr.cgi?pr=127927 From owner-freebsd-scsi@FreeBSD.ORG Mon Mar 31 11:06:51 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 267BEB21 for ; Mon, 31 Mar 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 13E8CBAA for ; Mon, 31 Mar 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s2VB6o0v058819 for ; Mon, 31 Mar 2014 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s2VB6okE058817 for freebsd-scsi@FreeBSD.org; Mon, 31 Mar 2014 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 31 Mar 2014 11:06:50 GMT Message-Id: <201403311106.s2VB6okE058817@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2014 11:06:51 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 19 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 1 14:07:00 2014 Return-Path: Delivered-To: scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C9922CB6 for ; Tue, 1 Apr 2014 14:07:00 +0000 (UTC) Received: from mail-we0-f169.google.com (mail-we0-f169.google.com [74.125.82.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 185C8849 for ; Tue, 1 Apr 2014 14:06:59 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id w62so6403656wes.28 for ; Tue, 01 Apr 2014 07:06:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:from:to:cc:subject:date:mime-version :content-type; bh=zZOV4m+s9sOGAjjiz5Suoxdvj+7/l8Qn7iPZM9ap/h0=; b=Svyf2VSjUpIE66nqcMBBiUuhMliABdIAm0rVj8yB0rda0+SFvYjyw/At8zVnAXQfWI FpdzZz0uAWaYBRuGabpFg3sCxawgC562lFRuqlp3K/wJ812W2UxnPyPj1RLGQ2ZJSl6G q5TJMRXQDLgwAZcbDXamGfbYB5Q/StqsaU3BqA4oMYAsmUEVzenC8LuVsAdk/uwiEyMF NUwGMIgxEmckScxBn82lN0jl6GEq3+9y/rhutdFZgVnEvmeiRSvDr9+l63g17MLV53bi S8SekKvtjE48O15UaPrh7WtRarCRZoM03EWLSW9pOsR/g+n1As4KpLP214riyQzz3eki a9mA== X-Gm-Message-State: ALoCoQmf6HIWiWlH1KIaMTOv57/M7fftC3wPScK5M4gE8sAIssnFCA2/uq9EVyhtG7xzY7Bc4wgr X-Received: by 10.194.84.144 with SMTP id z16mr22815161wjy.23.1396359629356; Tue, 01 Apr 2014 06:40:29 -0700 (PDT) Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id t10sm33231383wiw.19.2014.04.01.06.40.27 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 01 Apr 2014 06:40:28 -0700 (PDT) Message-ID: <84D0D5ADFC374E3AB7EFB51DC941411A@multiplay.co.uk> From: "Steven Hartland" To: Subject: LSI mps phase16 -> phase 18 driver changes patch Date: Tue, 1 Apr 2014 14:40:24 +0100 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_094C_01CF4DB8.50E12F90" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Scott Long , Alexander Motin , "Kenneth D. Merry" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2014 14:07:00 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_094C_01CF4DB8.50E12F90 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit Hi guys attached is a patch which updates our mps with the changes LSI have implemented between phase 16 (our last sync up) and phase 18 (current latest LSI release) drivers. The main change seems to be the addition Start Stop Unit for SATA direct-attach devices in IR mode to avoid data corruption. Any objections to committing this to HEAD? Regards Steve ------=_NextPart_000_094C_01CF4DB8.50E12F90 Content-Type: application/octet-stream; name="mps-phase16-phase18.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="mps-phase16-phase18.patch" Import LSI phase16 - phase18 changes=0A= * Implements Start Stop Unit for SATA direct-attach devices in IR mode = to avoid=0A= data corruption.=0A= --- sys/dev/mps/mps_sas.c.orig 2014-04-01 11:05:42.538799904 +0000=0A= +++ sys/dev/mps/mps_sas.c 2014-04-01 11:36:50.202671306 +0000=0A= @@ -188,6 +188,16 @@=0A= }=0A= =0A= void=0A= +mpssas_release_simq_reinit(struct mpssas_softc *sassc)=0A= +{=0A= + if (sassc->flags & MPSSAS_QUEUE_FROZEN) {=0A= + sassc->flags &=3D ~MPSSAS_QUEUE_FROZEN;=0A= + xpt_release_simq(sassc->sim, 1);=0A= + mps_dprint(sassc->sc, MPS_INFO, "Unfreezing SIM queue\n");=0A= + }=0A= +}=0A= +=0A= +void=0A= mpssas_startup_decrement(struct mpssas_softc *sassc)=0A= {=0A= MPS_FUNCTRACE(sassc->sc);=0A= @@ -995,7 +1005,7 @@=0A= =0A= targ =3D &sassc->targets[cts->ccb_h.target_id];=0A= if (targ->handle =3D=3D 0x0) {=0A= - cts->ccb_h.status =3D CAM_SEL_TIMEOUT;=0A= + cts->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= break;=0A= }=0A= =0A= @@ -1112,6 +1122,14 @@=0A= wakeup(cm);=0A= completed =3D 1;=0A= }=0A= +=0A= + if (cm->cm_sc->io_cmds_active !=3D 0) {=0A= + cm->cm_sc->io_cmds_active--;=0A= + } else {=0A= + mps_dprint(cm->cm_sc, MPS_INFO, "Warning: "=0A= + "io_cmds_active is out of sync - resynching to "=0A= + "0\n");=0A= + }=0A= =0A= if ((completed =3D=3D 0) && (cm->cm_state !=3D MPS_CM_STATE_FREE)) {=0A= /* this should never happen, but if it does, log */=0A= @@ -1643,14 +1661,14 @@=0A= if (targ->handle =3D=3D 0x0) {=0A= mps_dprint(sc, MPS_ERROR, "%s NULL handle for target %u\n", =0A= __func__, csio->ccb_h.target_id);=0A= - csio->ccb_h.status =3D CAM_SEL_TIMEOUT;=0A= + csio->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= xpt_done(ccb);=0A= return;=0A= }=0A= if (targ->flags & MPS_TARGET_FLAGS_RAID_COMPONENT) {=0A= mps_dprint(sc, MPS_ERROR, "%s Raid component no SCSI IO "=0A= "supported %u\n", __func__, csio->ccb_h.target_id);=0A= - csio->ccb_h.status =3D CAM_TID_INVALID;=0A= + csio->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= xpt_done(ccb);=0A= return;=0A= }=0A= @@ -1681,13 +1699,16 @@=0A= =0A= if ((sc->mps_flags & MPS_FLAGS_SHUTDOWN) !=3D 0) {=0A= mps_dprint(sc, MPS_INFO, "%s shutting down\n", __func__);=0A= - csio->ccb_h.status =3D CAM_TID_INVALID;=0A= + csio->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= xpt_done(ccb);=0A= return;=0A= }=0A= =0A= cm =3D mps_alloc_command(sc);=0A= - if (cm =3D=3D NULL) {=0A= + if (cm =3D=3D NULL || (sc->mps_flags & MPS_FLAGS_DIAGRESET)) {=0A= + if (cm !=3D NULL) {=0A= + mps_free_command(sc, cm);=0A= + }=0A= if ((sassc->flags & MPSSAS_QUEUE_FROZEN) =3D=3D 0) {=0A= xpt_freeze_simq(sassc->sim, 1);=0A= sassc->flags |=3D MPSSAS_QUEUE_FROZEN;=0A= @@ -2166,6 +2187,18 @@=0A= }=0A= }=0A= =0A= + /*=0A= + * If this is a Start Stop Unit command and it was issued by the driver=0A= + * during shutdown, decrement the refcount to account for all of the=0A= + * commands that were sent. All SSU commands should be completed = before=0A= + * shutdown completes, meaning SSU_refcount will be 0 after SSU_started=0A= + * is TRUE.=0A= + */=0A= + if (sc->SSU_started && (csio->cdb_io.cdb_bytes[0] =3D=3D = START_STOP_UNIT)) {=0A= + mps_dprint(sc, MPS_INFO, "Decrementing SSU count.\n");=0A= + sc->SSU_refcount--;=0A= + }=0A= +=0A= /* Take the fast path to completion */=0A= if (cm->cm_reply =3D=3D NULL) {=0A= if ((ccb->ccb_h.status & CAM_STATUS_MASK) =3D=3D CAM_REQ_INPROG) {=0A= @@ -2990,7 +3023,7 @@=0A= mps_dprint(sc, MPS_ERROR,=0A= "%s: handle %d does not have a valid "=0A= "parent handle!\n", __func__, targ->handle);=0A= - ccb->ccb_h.status =3D CAM_REQ_INVALID;=0A= + ccb->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= goto bailout;=0A= }=0A= #ifdef OLD_MPS_PROBE=0A= @@ -3001,7 +3034,7 @@=0A= mps_dprint(sc, MPS_ERROR,=0A= "%s: handle %d does not have a valid "=0A= "parent target!\n", __func__, targ->handle);=0A= - ccb->ccb_h.status =3D CAM_REQ_INVALID;=0A= + ccb->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= goto bailout;=0A= }=0A= =0A= @@ -3011,7 +3044,7 @@=0A= "%s: handle %d parent %d does not "=0A= "have an SMP target!\n", __func__,=0A= targ->handle, parent_target->handle);=0A= - ccb->ccb_h.status =3D CAM_REQ_INVALID;=0A= + ccb->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= goto bailout;=0A= =0A= }=0A= @@ -3024,7 +3057,7 @@=0A= "%s: handle %d parent %d does not "=0A= "have an SMP target!\n", __func__,=0A= targ->handle, targ->parent_handle);=0A= - ccb->ccb_h.status =3D CAM_REQ_INVALID;=0A= + ccb->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= goto bailout;=0A= =0A= }=0A= @@ -3033,7 +3066,7 @@=0A= "%s: handle %d parent handle %d does "=0A= "not have a valid SAS address!\n",=0A= __func__, targ->handle, targ->parent_handle);=0A= - ccb->ccb_h.status =3D CAM_REQ_INVALID;=0A= + ccb->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= goto bailout;=0A= }=0A= =0A= @@ -3046,7 +3079,7 @@=0A= mps_dprint(sc, MPS_INFO,=0A= "%s: unable to find SAS address for handle %d\n",=0A= __func__, targ->handle);=0A= - ccb->ccb_h.status =3D CAM_REQ_INVALID;=0A= + ccb->ccb_h.status =3D CAM_DEV_NOT_THERE;=0A= goto bailout;=0A= }=0A= mpssas_send_smpcmd(sassc, ccb, sasaddr);=0A= @@ -3348,6 +3381,20 @@=0A= }=0A= =0A= xpt_path_string(local_path, path_str, sizeof(path_str));=0A= +=0A= + /*=0A= + * If this is a SATA direct-access end device,=0A= + * mark it so that a SCSI StartStopUnit command=0A= + * will be sent to it when the driver is being=0A= + * shutdown.=0A= + */=0A= + if ((cgd.inq_data.device =3D=3D T_DIRECT) && =0A= + (target->devinfo & MPI2_SAS_DEVICE_INFO_SATA_DEVICE) &&=0A= + ((target->devinfo & MPI2_SAS_DEVICE_INFO_MASK_DEVICE_TYPE) =3D=3D=0A= + MPI2_SAS_DEVICE_INFO_END_DEVICE)) {=0A= + lun->stop_at_shutdown =3D TRUE;=0A= + }=0A= +=0A= mps_dprint(sc, MPS_INFO, "Sending read cap: path %s handle %d\n",=0A= path_str, target->handle);=0A= =0A= --- sys/dev/mps/mps_sas_lsi.c.orig 2014-03-26 17:49:01.000000000 +0000=0A= +++ sys/dev/mps/mps_sas_lsi.c 2014-04-01 11:06:42.158795951 +0000=0A= @@ -120,6 +120,9 @@=0A= u64 *sas_address, u16 handle, u32 device_info);=0A= static int mpssas_volume_add(struct mps_softc *sc,=0A= u16 handle);=0A= +static void mpssas_SSU_to_SATA_devices(struct mps_softc *sc);=0A= +static void mpssas_stop_unit_done(struct cam_periph *periph,=0A= + union ccb *done_ccb);=0A= =0A= void=0A= mpssas_evt_handler(struct mps_softc *sc, uintptr_t data,=0A= @@ -910,6 +913,138 @@=0A= }=0A= =0A= /**=0A= + * mpssas_SSU_to_SATA_devices =0A= + * @sc: per adapter object=0A= + *=0A= + * Looks through the target list and issues a StartStopUnit SCSI = command to each=0A= + * SATA direct-access device. This helps to ensure that data = corruption is=0A= + * avoided when the system is being shut down. This must be called = after the IR=0A= + * System Shutdown RAID Action is sent if in IR mode.=0A= + *=0A= + * Return nothing.=0A= + */=0A= +static void=0A= +mpssas_SSU_to_SATA_devices(struct mps_softc *sc)=0A= +{=0A= + struct mpssas_softc *sassc =3D sc->sassc;=0A= + union ccb *ccb;=0A= + path_id_t pathid =3D cam_sim_path(sassc->sim);=0A= + target_id_t targetid;=0A= + struct mpssas_target *target;=0A= + struct mpssas_lun *lun;=0A= + char path_str[64];=0A= + struct timeval cur_time, start_time;=0A= +=0A= + /*=0A= + * For each LUN of each target, issue a StartStopUnit command to stop=0A= + * the device.=0A= + */=0A= + sc->SSU_started =3D TRUE;=0A= + sc->SSU_refcount =3D 0;=0A= + for (targetid =3D 0; targetid < sc->facts->MaxTargets; targetid++) {=0A= + target =3D &sassc->targets[targetid];=0A= + if (target->handle =3D=3D 0x0) {=0A= + continue;=0A= + }=0A= +=0A= + SLIST_FOREACH(lun, &target->luns, lun_link) {=0A= + ccb =3D xpt_alloc_ccb_nowait();=0A= + if (ccb =3D=3D NULL) {=0A= + mps_dprint(sc, MPS_FAULT, "Unable to alloc CCB "=0A= + "to stop unit.\n");=0A= + return;=0A= + }=0A= +=0A= + /*=0A= + * The stop_at_shutdown flag will be set if this LUN is=0A= + * a SATA direct-access end device.=0A= + */=0A= + if (lun->stop_at_shutdown) {=0A= + if (xpt_create_path(&ccb->ccb_h.path,=0A= + xpt_periph, pathid, targetid,=0A= + lun->lun_id) !=3D CAM_REQ_CMP) {=0A= + mps_dprint(sc, MPS_FAULT, "Unable to "=0A= + "create LUN path to stop unit.\n");=0A= + xpt_free_ccb(ccb);=0A= + return;=0A= + }=0A= + xpt_path_string(ccb->ccb_h.path, path_str,=0A= + sizeof(path_str));=0A= +=0A= + mps_dprint(sc, MPS_INFO, "Sending StopUnit: "=0A= + "path %s handle %d\n", path_str,=0A= + target->handle);=0A= + =0A= + /*=0A= + * Issue a START STOP UNIT command for the LUN.=0A= + * Increment the SSU counter to be used to=0A= + * count the number of required replies.=0A= + */=0A= + mps_dprint(sc, MPS_INFO, "Incrementing SSU "=0A= + "count\n");=0A= + sc->SSU_refcount++;=0A= + ccb->ccb_h.target_id =3D=0A= + xpt_path_target_id(ccb->ccb_h.path);=0A= + ccb->ccb_h.target_lun =3D lun->lun_id;=0A= + ccb->ccb_h.ppriv_ptr1 =3D sassc;=0A= + scsi_start_stop(&ccb->csio,=0A= + /*retries*/0,=0A= + mpssas_stop_unit_done,=0A= + MSG_SIMPLE_Q_TAG,=0A= + /*start*/FALSE,=0A= + /*load/eject*/0,=0A= + /*immediate*/FALSE,=0A= + MPS_SENSE_LEN,=0A= + /*timeout*/10000);=0A= + xpt_action(ccb);=0A= + }=0A= + }=0A= + }=0A= +=0A= + /*=0A= + * Wait until all of the SSU commands have completed or time has=0A= + * expired (60 seconds). pause for 100ms each time through. If any=0A= + * command times out, the target will be reset in the SCSI command=0A= + * timeout routine.=0A= + */=0A= + getmicrotime(&start_time);=0A= + while (sc->SSU_refcount) {=0A= + pause("mpswait", hz/10);=0A= + =0A= + getmicrotime(&cur_time);=0A= + if ((cur_time.tv_sec - start_time.tv_sec) > 60) {=0A= + mps_dprint(sc, MPS_FAULT, "Time has expired waiting "=0A= + "for SSU commands to complete.\n");=0A= + break;=0A= + }=0A= + }=0A= +}=0A= +=0A= +static void=0A= +mpssas_stop_unit_done(struct cam_periph *periph, union ccb *done_ccb)=0A= +{=0A= + struct mpssas_softc *sassc;=0A= + char path_str[64];=0A= +=0A= + sassc =3D (struct mpssas_softc *)done_ccb->ccb_h.ppriv_ptr1;=0A= +=0A= + xpt_path_string(done_ccb->ccb_h.path, path_str, sizeof(path_str));=0A= + mps_dprint(sassc->sc, MPS_INFO, "Completing stop unit for %s\n",=0A= + path_str);=0A= +=0A= + if (done_ccb =3D=3D NULL)=0A= + return;=0A= +=0A= + /*=0A= + * Nothing more to do except free the CCB and path. If the command=0A= + * timed out, an abort reset, then target reset will be issued during=0A= + * the SCSI Command process.=0A= + */=0A= + xpt_free_path(done_ccb->ccb_h.path);=0A= + xpt_free_ccb(done_ccb);=0A= +}=0A= +=0A= +/**=0A= * mpssas_ir_shutdown - IR shutdown notification=0A= * @sc: per adapter object=0A= *=0A= @@ -933,7 +1068,7 @@=0A= =0A= /* is IR firmware build loaded? */=0A= if (!sc->ir_firmware)=0A= - return;=0A= + goto out;=0A= =0A= /* are there any volumes? Look at IR target IDs. */=0A= // TODO-later, this should be looked up in the RAID config structure=0A= @@ -958,11 +1093,11 @@=0A= }=0A= =0A= if (!found_volume)=0A= - return;=0A= + goto out;=0A= =0A= if ((cm =3D mps_alloc_command(sc)) =3D=3D NULL) {=0A= printf("%s: command alloc failed\n", __func__);=0A= - return;=0A= + goto out;=0A= }=0A= =0A= action =3D (MPI2_RAID_ACTION_REQUEST *)cm->cm_req;=0A= @@ -978,4 +1113,7 @@=0A= */=0A= if (cm)=0A= mps_free_command(sc, cm);=0A= +=0A= +out:=0A= + mpssas_SSU_to_SATA_devices(sc);=0A= }=0A= --- sys/dev/mps/mps_sas.h.orig 2014-03-26 17:49:01.000000000 +0000=0A= +++ sys/dev/mps/mps_sas.h 2014-04-01 11:06:42.157795822 +0000=0A= @@ -35,6 +35,7 @@=0A= lun_id_t lun_id;=0A= uint8_t eedp_formatted;=0A= uint32_t eedp_block_size;=0A= + uint8_t stop_at_shutdown;=0A= };=0A= =0A= struct mpssas_target {=0A= @@ -154,6 +155,7 @@=0A= void mpssas_discovery_end(struct mpssas_softc *sassc);=0A= void mpssas_startup_increment(struct mpssas_softc *sassc);=0A= void mpssas_startup_decrement(struct mpssas_softc *sassc);=0A= +void mpssas_release_simq_reinit(struct mpssas_softc *sassc);=0A= =0A= struct mps_command * mpssas_alloc_tm(struct mps_softc *sc);=0A= void mpssas_free_tm(struct mps_softc *sc, struct mps_command *tm);=0A= --- sys/dev/mps/mps.c.orig 2014-04-01 11:05:42.598799869 +0000=0A= +++ sys/dev/mps/mps.c 2014-04-01 11:51:15.083611560 +0000=0A= @@ -141,6 +141,7 @@=0A= {=0A= uint32_t reg;=0A= int i, error, tries =3D 0;=0A= + uint8_t first_wait_done =3D FALSE;=0A= =0A= mps_dprint(sc, MPS_TRACE, "%s\n", __func__);=0A= =0A= @@ -183,15 +184,32 @@=0A= =0A= /* Wait up to 300 seconds in 50ms intervals */=0A= error =3D ETIMEDOUT;=0A= - for (i =3D 0; i < 60000; i++) {=0A= - /* wait 50 msec */=0A= - if (mtx_owned(&sc->mps_mtx) && sleep_flag =3D=3D CAN_SLEEP)=0A= - msleep(&sc->msleep_fake_chan, &sc->mps_mtx, 0,=0A= - "mpsdiag", hz/20);=0A= - else if (sleep_flag =3D=3D CAN_SLEEP)=0A= - pause("mpsdiag", hz/20);=0A= - else=0A= - DELAY(50 * 1000);=0A= + for (i =3D 0; i < 6000; i++) {=0A= + /*=0A= + * Wait 50 msec. If this is the first time through, wait 256=0A= + * msec to satisfy Diag Reset timing requirements.=0A= + */=0A= + if (first_wait_done) {=0A= + if (mtx_owned(&sc->mps_mtx) && sleep_flag =3D=3D CAN_SLEEP)=0A= + msleep(&sc->msleep_fake_chan, &sc->mps_mtx, 0,=0A= + "mpsdiag", hz/20);=0A= + else if (sleep_flag =3D=3D CAN_SLEEP)=0A= + pause("mpsdiag", hz/20);=0A= + else=0A= + DELAY(50 * 1000);=0A= + } else {=0A= + DELAY(256 * 1000);=0A= + first_wait_done =3D TRUE;=0A= + }=0A= + /*=0A= + * Check for the RESET_ADAPTER bit to be cleared first, then=0A= + * wait for the RESET state to be cleared, which takes a little=0A= + * longer.=0A= + */=0A= + reg =3D mps_regread(sc, MPI2_HOST_DIAGNOSTIC_OFFSET);=0A= + if (reg & MPI2_DIAG_RESET_ADAPTER) {=0A= + continue;=0A= + }=0A= reg =3D mps_regread(sc, MPI2_DOORBELL_OFFSET);=0A= if ((reg & MPI2_IOC_STATE_MASK) !=3D MPI2_IOC_STATE_RESET) {=0A= error =3D 0;=0A= @@ -237,7 +255,7 @@=0A= sleep_flags =3D (sc->mps_flags & MPS_FLAGS_ATTACH_DONE)=0A= ? CAN_SLEEP:NO_SLEEP;=0A= error =3D 0;=0A= - while (tries++ < 5) {=0A= + while (tries++ < 1200) {=0A= reg =3D mps_regread(sc, MPI2_DOORBELL_OFFSET);=0A= mps_dprint(sc, MPS_INIT, "Doorbell=3D 0x%x\n", reg);=0A= =0A= @@ -679,6 +697,9 @@=0A= mps_reinit(struct mps_softc *sc)=0A= {=0A= int error;=0A= + struct mpssas_softc *sassc;=0A= +=0A= + sassc =3D sc->sassc;=0A= =0A= MPS_FUNCTRACE(sc);=0A= =0A= @@ -759,6 +780,8 @@=0A= mps_dprint(sc, MPS_INFO, "%s finished sc %p post %u free %u\n", =0A= __func__, sc, sc->replypostindex, sc->replyfreeindex);=0A= =0A= + mpssas_release_simq_reinit(sassc);=0A= +=0A= return 0;=0A= }=0A= =0A= @@ -2533,6 +2556,7 @@=0A= mps_request_polled(struct mps_softc *sc, struct mps_command *cm)=0A= {=0A= int error, timeout =3D 0, rc;=0A= + struct timeval cur_time, start_time;=0A= =0A= error =3D 0;=0A= =0A= @@ -2540,22 +2564,33 @@=0A= cm->cm_complete =3D NULL;=0A= mps_map_command(sc, cm);=0A= =0A= + getmicrotime(&start_time);=0A= while ((cm->cm_flags & MPS_CM_FLAGS_COMPLETE) =3D=3D 0) {=0A= mps_intr_locked(sc);=0A= =0A= - DELAY(50 * 1000);=0A= - if (timeout++ > 1000) {=0A= + if (mtx_owned(&sc->mps_mtx))=0A= + msleep(&sc->msleep_fake_chan, &sc->mps_mtx, 0,=0A= + "mpspoll", hz/20);=0A= + else=0A= + pause("mpsdiag", hz/20);=0A= +=0A= + /*=0A= + * Check for real-time timeout and fail if more than 60 seconds.=0A= + */=0A= + getmicrotime(&cur_time);=0A= + timeout =3D cur_time.tv_sec - start_time.tv_sec;=0A= + if (timeout > 60) {=0A= mps_dprint(sc, MPS_FAULT, "polling failed\n");=0A= error =3D ETIMEDOUT;=0A= break;=0A= }=0A= }=0A= - =0A= +=0A= if (error) {=0A= mps_dprint(sc, MPS_FAULT, "Calling Reinit from %s\n", __func__);=0A= rc =3D mps_reinit(sc);=0A= - mps_dprint(sc, MPS_FAULT, "Reinit %s\n", =0A= - (rc =3D=3D 0) ? "success" : "failed");=0A= + mps_dprint(sc, MPS_FAULT, "Reinit %s\n", (rc =3D=3D 0) ? "success" :=0A= + "failed");=0A= }=0A= =0A= return (error);=0A= --- sys/dev/mps/mpsvar.h.orig 2014-03-26 17:49:01.000000000 +0000=0A= +++ sys/dev/mps/mpsvar.h 2014-04-01 11:30:54.245695971 +0000=0A= @@ -32,7 +32,7 @@=0A= #ifndef _MPSVAR_H=0A= #define _MPSVAR_H=0A= =0A= -#define MPS_DRIVER_VERSION "16.00.00.00-fbsd"=0A= +#define MPS_DRIVER_VERSION "18.00.00.00-fbsd"=0A= =0A= #define MPS_DB_MAX_WAIT 2500=0A= =0A= @@ -417,6 +417,10 @@=0A= =0A= char exclude_ids[80];=0A= struct timeval lastfail;=0A= +=0A= + /* StartStopUnit command handling at shutdown */=0A= + uint32_t SSU_refcount;=0A= + uint8_t SSU_started;=0A= };=0A= =0A= struct mps_config_params {=0A= ------=_NextPart_000_094C_01CF4DB8.50E12F90-- From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 1 19:55:51 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3F9CB817 for ; Tue, 1 Apr 2014 19:55:51 +0000 (UTC) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF21F2EB for ; Tue, 1 Apr 2014 19:55:50 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id s31Jtilf069060; Tue, 1 Apr 2014 13:55:44 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id s31JtiVx069059; Tue, 1 Apr 2014 13:55:44 -0600 (MDT) (envelope-from ken) Date: Tue, 1 Apr 2014 13:55:44 -0600 From: "Kenneth D. Merry" To: bharat singh Subject: Re: Adding write_same16 to CAM Target Layer Message-ID: <20140401195544.GA68366@nargothrond.kdm.org> References: <20140328185331.GA72411@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Apr 2014 19:55:51 -0000 On Sat, Mar 29, 2014 at 05:57:38 +0530, bharat singh wrote: > I am using block back end with block size 512bytes. Yes as per t10 doc > initiator will send 512bytes of data to be filled over 1MB. > > Block size is 512 and no of blocks is 2048 in my case. Hmm. If datalen is actually 1048576, and the initiator is sending the same amount of data, there shouldn't be a problem. But that doesn't agree with what you said above, or with the spec. The spec does say that it will transfer one block. If blocksize = 512 and num_blocks = 2048, you're asking for 1MB. It looks like the initiator may be sending 1MB. But I'm not sure why you're getting an overrun. If anything, I'd expect an underrun. > >From CTL layer can I reallocate the data buffers. Yes, but that will slow things down to free and allocate the buffers again. In order to scale WRITE SAME, you'd probably want to allocate say a 1MB buffer in the block backend. Transfer a 512 byte block from the initiator, and then copy it to the rest of the blocks in the 1MB and issue the write. Or, change the file dispatch path (ctl_be_block_dispatch_file()) to have a long S/G list of segments in a uio. For the device path (ctl_be_block_dispatch_dev()) that would be a more difficult thing to implement. There isn't currently a way that I know of to attach a scatter/gather list to a struct bio. (But there should be!) > This command is needed for initiators hosted on ESXi. It'll be nice to support ESXi. Are you able to contribute your changes back to FreeBSD once you've got them working? Ken > On 29-Mar-2014 12:23 AM, "Kenneth D. Merry" wrote: > > > On Wed, Mar 26, 2014 at 11:44:32 +0530, bharat singh wrote: > > > Hello, > > > > > > I am trying to add write_same16 (block zero) support to my FC stack. > > > This i am trying to achieve by adding a hook in CTL layer, reading > > > start_LBA and no of blocks to fill with 0. > > > > > > struct ctl_scsiio { > > > >-------struct ctl_io_hdr io_hdr;>------/* common to all I/O types */ > > > >-------uint32_t ext_sg_entries;>-----/* 0 = no S/G list, > 0 = num > > > entries */ > > > >-------uint8_t> *ext_data_ptr;>------/* data buffer or S/G list */ > > > >-------uint32_t ext_data_len;>-------/* Data transfer length */ > > > >-------uint32_t ext_data_filled;>----/* Amount of data filled so far > > */ > > > >-------uint32_t kern_sg_entries;>----/* 0 = no S/G list, > 0 = num > > > entries */ > > > >-------uint32_t rem_sg_entries;>-----/* 0 = no S/G list, > 0 = num > > > entries */ > > > >-------uint8_t *kern_data_ptr;>-----/* data buffer or S/G list */ > > > >-------uint32_t kern_data_len;>------/* Length of this S/G > > list/buffer */ > > > >-------uint32_t kern_total_len;>-----/* Total length of this > > transaction > > > */ > > > >-------uint32_t kern_data_resid;>----/* Length left to transfer after > > > this*/ > > > >-------uint32_t kern_rel_offset;>----/* Byte Offset of this transfer > > */ > > > >-------struct scsi_sense_data sense_data;>-/* sense data */ > > > >-------uint8_t> sense_len;>-->-------/* Returned sense length */ > > > >-------uint8_t> scsi_status;>>-------/* SCSI status byte */ > > > >-------uint8_t> sense_residual;>-----/* sense residual length */ > > > >-------uint32_t residual;>--->-------/* data residual length */ > > > >-------uint32_t tag_num;>---->-------/* tag number */ > > > >-------ctl_tag_type tag_type;>->-------/* simple, ordered, head of > > > queue,etc.*/ > > > >-------uint8_t cdb_len;>---->-------/* CDB length */ > > > >-------uint8_t> cdb[CTL_MAX_CDBLEN];>/* CDB */ > > > >-------int>---- (*be_move_done)(union ctl_io *io); /* called by fe */ > > > >-------int (*io_cont)(union ctl_io *io); /* to continue > > processing > > > */ > > > >-------uint32_t ctl_lun_masking_disabled; /* Disable lun mask checks > > for > > > ctladm calls */ > > > }; > > > > > > For write_same16 initiators will send a block of 0s with start_LBA and no > > > of blocks. Assuming that I am trying to: > > > datalen = blocksize * num_blocks; > > > uint8_t * databuf = malloc(datalen); > > > bzero(databuf, 0, datalen); > > > ctsio->kern_data_ptr = dataptr; > > > ctsio->kern_data_len = datalen; > > > lbalen.lba = lba; > > > lbalen.len = num_blocks; > > > memcpy(ctsio->io_hdr.ctl_private[CTL_PRIV_LBA_LEN].bytes, &lbalen, > > > sizeof(lbalen)); > > > ret = lun->backend->data_submit((union ctl_io *)ctsio); > > > > > > but it's failing in ISP layer with data overflow. My initiator is sending > > > 2048 blocks of size 512B each and a payload of 512 bytes in data-out > > buffer. > > > isp0: isp_target_start_ctio: [0x120154] data overflow by 1048064 bytes > > > (1:3:0:1): WRITE SAME(16). CDB: 93 00 00 00 00 00 00 1f e8 00 00 00 08 00 > > > 00 00 > > > > > > Have any one done changes to the CTL read/write path, am I missing > > > something regarding buffer population. > > > Thanks for the help in advance. > > > > What back end are you using? The block backend, or are you writing your > > own? > > > > In the case of the block backend, it is expecting to do its own memory > > allocation and then call ctl_datamove() to trigger the DMA. > > > > In this case, you're doing a malloc, but if you're going to the block > > backend the data buffer will get overwritten. > > > > As for why you're getting an overrun, the amount of the overrun is 1048064 > > bytes. That is 512 less than 1048576. Is your blocksize and number of > > blocks set correctly? It looks like datalen is probably 512, but your > > initiator is sending 1MB. > > > > Ken > > -- > > Kenneth Merry > > ken@FreeBSD.ORG > > -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Wed Apr 2 23:11:20 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 026CDAE6; Wed, 2 Apr 2014 23:11:20 +0000 (UTC) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67D36953; Wed, 2 Apr 2014 23:11:19 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id hm4so7389654wib.0 for ; Wed, 02 Apr 2014 16:11:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=eyh/LUScVezhzhrAGP6lBd7wxge5Put1OPWaKCPE/8U=; b=SAVGj9yBL0WtKcOsCTQ1PyzdKxtuXroVp7mfeWCK/jlH4MqNd7nVchr87o98tqzTou qjjDjjDMWY0DioCA3lkZL7B+CcFZpAAwbBeImh1qG8VquSa0EbdKu4pWLho0kFXdueXV cswAmi6frrNT4CjebX5yMbUtmjEd/wJilJtPxOP5u479U3fz86CuzLKmUZzPIwpSQTG3 zZZOWnoN+5SMnyHa9feQzGzk33ko78Js7PHmbfOwaHdEL4VhlSH8NxnDJR3KJHxn9r8R KEXGzNDmkcCnqfvHxJisU2tvcOIetvyHyeIGMzyVQJiaGSbeXxk3IpWgTRlfHh+8slAA vBiQ== X-Received: by 10.180.98.165 with SMTP id ej5mr5500417wib.33.1396480277695; Wed, 02 Apr 2014 16:11:17 -0700 (PDT) Received: from strashydlo.home (adbf227.neoplus.adsl.tpnet.pl. [79.184.5.227]) by mx.google.com with ESMTPSA id u1sm7722490eex.31.2014.04.02.16.11.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 02 Apr 2014 16:11:16 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Subject: Re: Adding write_same16 to CAM Target Layer Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-2 From: =?iso-8859-2?Q?Edward_Tomasz_Napiera=B3a?= In-Reply-To: <20140401195544.GA68366@nargothrond.kdm.org> Date: Thu, 3 Apr 2014 01:11:14 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140328185331.GA72411@nargothrond.kdm.org> <20140401195544.GA68366@nargothrond.kdm.org> To: Kenneth D. Merry X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org, bharat singh X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Apr 2014 23:11:20 -0000 Wiadomo=B6=E6 napisana przez Kenneth D. Merry w dniu 1 kwi 2014, o godz. = 21:55: > On Sat, Mar 29, 2014 at 05:57:38 +0530, bharat singh wrote: >=20 >> This command is needed for initiators hosted on ESXi. >=20 > It'll be nice to support ESXi. Are you able to contribute your = changes > back to FreeBSD once you've got them working? Hm, from my testing ESXi actually works fine as it is now, at least over iSCSI. What am I missing? From owner-freebsd-scsi@FreeBSD.ORG Fri Apr 4 11:50:45 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B0422A19; Fri, 4 Apr 2014 11:50:45 +0000 (UTC) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A92A41F0; Fri, 4 Apr 2014 11:50:44 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id z2so1078918wiv.0 for ; Fri, 04 Apr 2014 04:50:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=BVhoXrnUqZGR+87p9ZA/ViHaHUS/C8hS0hrWuMqrFdA=; b=dstro1iSsvIAx34SW4TRKDFGQYmYlqDYVg8TbcZEILrCgWhfArVe/0Fy5DjfR5TjPH 2hePW75JFpQfyXKKTHtfs0v60p+IdtKHqAq/Ro2jxZI+mTOroVUzypnN95rb7LnU+f0l ZOyEfD8qWNyonFc0sbQgasXA2MxXCDtPCaoWJS3vPTzNB8eVvklG5M92gQVizu9QVDkV HCreoriadPWSFakKpCi12w2jhDYEuwEN9ua6fOq+LYXBATCwxqneVX0qqASuavfeMuUI QO159o3IyuAecankQaTpYlynlYvg3MM1nAoCtk3L6lckYJMb+c4fwSKQOL1PP8CmR7Vj ufXw== X-Received: by 10.194.82.69 with SMTP id g5mr19148804wjy.85.1396612242946; Fri, 04 Apr 2014 04:50:42 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id y51sm19074307eeu.0.2014.04.04.04.50.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 04 Apr 2014 04:50:41 -0700 (PDT) Sender: Alexander Motin Message-ID: <533E9C8F.20005@FreeBSD.org> Date: Fri, 04 Apr 2014 14:50:39 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: FreeBSD SCSI , FreeBSD Hardware Subject: Are there still cd(4) changers? Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Apr 2014 11:50:45 -0000 Hi. Does anybody still use CD changer supported by cd(4) driver in FreeBSD, not ch(4)? Those changers have single drive, but report multiple LUNs with one LUN per CD slot. One device like I have in my table is 17 year old. All devices I can find with Google or eBay are CD, not even DVD, and are parallel ATA or parallel SCSI. Is there anything relevant still on market? I am asking this because code supporting that hardware in FreeBSD is heavily broken in head and stable/10 branches for several months now, and fix seems to be non-trivial. So my question is: does it worth bothering with rewrite, or we can just drop ~20KB of unused and quite complicated code? Dropping code does not mean those devices will be unusable, but only that _simultaneous_ access to different LUNs will become much slower due to inefficient scheduling of disk loads/unloads. -- Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Fri Apr 4 22:59:05 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21A56EC8; Fri, 4 Apr 2014 22:59:05 +0000 (UTC) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 99E6FD09; Fri, 4 Apr 2014 22:59:01 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id s34Mx0BW049683; Fri, 4 Apr 2014 16:59:00 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id s34Mx0oT049682; Fri, 4 Apr 2014 16:59:00 -0600 (MDT) (envelope-from ken) Date: Fri, 4 Apr 2014 16:59:00 -0600 From: "Kenneth D. Merry" To: Alexander Motin Subject: Re: Are there still cd(4) changers? Message-ID: <20140404225900.GA49468@nargothrond.kdm.org> References: <533E9C8F.20005@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <533E9C8F.20005@FreeBSD.org> User-Agent: Mutt/1.4.2i Cc: FreeBSD SCSI , FreeBSD Hardware X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Apr 2014 22:59:05 -0000 On Fri, Apr 04, 2014 at 14:50:39 +0300, Alexander Motin wrote: > Hi. > > Does anybody still use CD changer supported by cd(4) driver in FreeBSD, > not ch(4)? Those changers have single drive, but report multiple LUNs > with one LUN per CD slot. > > One device like I have in my table is 17 year old. All devices I can > find with Google or eBay are CD, not even DVD, and are parallel ATA or > parallel SCSI. Is there anything relevant still on market? I have one, but like yours, it's very old. (It's one of the ones I used when I wrote the changer code in the late 90's.) > I am asking this because code supporting that hardware in FreeBSD is > heavily broken in head and stable/10 branches for several months now, > and fix seems to be non-trivial. So my question is: does it worth > bothering with rewrite, or we can just drop ~20KB of unused and quite > complicated code? Dropping code does not mean those devices will be > unusable, but only that _simultaneous_ access to different LUNs will > become much slower due to inefficient scheduling of disk loads/unloads. I think it is fine to take it out. Let's see if anyone who is using one actually speaks up, but I would imagine that very few people are using them. Removing the code will simplify the driver, and it is becoming more difficult to find the hardware to even run one of those devices. I doubt we'll see anyone attempt to make one of those again. It'll probably remain like it is now -- either single disk devices, or large optical changers that would use ch(4) to do the changing. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Mon Apr 7 11:06:51 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D0D6BC3 for ; Mon, 7 Apr 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 22B07C0B for ; Mon, 7 Apr 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s37B6oDr071193 for ; Mon, 7 Apr 2014 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s37B6omr071191 for freebsd-scsi@FreeBSD.org; Mon, 7 Apr 2014 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 7 Apr 2014 11:06:50 GMT Message-Id: <201404071106.s37B6omr071191@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Apr 2014 11:06:51 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 19 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 8 21:44:34 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 027937F7 for ; Tue, 8 Apr 2014 21:44:34 +0000 (UTC) Received: from mail-ee0-x232.google.com (mail-ee0-x232.google.com [IPv6:2a00:1450:4013:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8C50A1912 for ; Tue, 8 Apr 2014 21:44:33 +0000 (UTC) Received: by mail-ee0-f50.google.com with SMTP id c13so1167153eek.23 for ; Tue, 08 Apr 2014 14:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=iydXTPhCVUrFz0CrjtOySldadYnqPXe5vWo9fPnFnKU=; b=uzys6y4fer7q+TWXkTR/IK63tCmXcAuZ1h4WedmPJyQtwqM5a/W2Mp3L5UkGv7d4DT IoyGmWwd/ey2E5c0gbNB1A68cQdXGAlvNYgqxDVzQ+XeYBR7UGq49rKP2oIJk1QHFO1Q DFGoQVFECSk2/TauYb8GOWyT/+YLczsLSv1gmh4QRGA91tTh/lBrdnGAJlW1pfjYxiDG iFajo26KuTFTMcpMqmpGvYcxIdreNs0WSIpyzlySDwBe6BzIFR7RstinF3dwUtTZ90Fq 9641J69G2gmdygaHoRQpa1A5qw7hcUbPEmVcCF37ltviRgc5+l6UWuOy6Do4tJK6kSRO 7gig== X-Received: by 10.15.33.136 with SMTP id c8mr98589eev.111.1396993471839; Tue, 08 Apr 2014 14:44:31 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id w12sm7150030eez.36.2014.04.08.14.44.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 08 Apr 2014 14:44:30 -0700 (PDT) Sender: Alexander Motin Message-ID: <53446DBC.705@FreeBSD.org> Date: Wed, 09 Apr 2014 00:44:28 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: FreeBSD SCSI , bharat singh Subject: Re: Adding write_same16 to CAM Target Layer Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2014 21:44:34 -0000 > I am trying to add write_same16 (block zero) support to my FC stack. I have just committed to head patch supporting UNMAP ans WRITE SAME commands: http://svnweb.freebsd.org/changeset/base/264274 Testers are welcome. :) -- Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Mon Apr 14 11:06:51 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C534B13A for ; Mon, 14 Apr 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2759166E for ; Mon, 14 Apr 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s3EB6pdc026004 for ; Mon, 14 Apr 2014 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s3EB6pPB026002 for freebsd-scsi@FreeBSD.org; Mon, 14 Apr 2014 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 14 Apr 2014 11:06:51 GMT Message-Id: <201404141106.s3EB6pPB026002@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Apr 2014 11:06:51 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 19 problems total. From owner-freebsd-scsi@FreeBSD.ORG Wed Apr 16 01:02:11 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F6DC3FC; Wed, 16 Apr 2014 01:02:11 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 469C4143A; Wed, 16 Apr 2014 01:02:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s3G12BQM000429; Wed, 16 Apr 2014 01:02:11 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s3G12B2r000428; Wed, 16 Apr 2014 01:02:11 GMT (envelope-from linimon) Date: Wed, 16 Apr 2014 01:02:11 GMT Message-Id: <201404160102.s3G12B2r000428@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/184505: [ahd] AHD controller/Bad Negotiation/3 out of 4 boots X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2014 01:02:11 -0000 Old Synopsis: AHD controller/Bad Negotiation/3 out of 4 boots New Synopsis: [ahd] AHD controller/Bad Negotiation/3 out of 4 boots Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Wed Apr 16 01:00:50 UTC 2014 Responsible-Changed-Why: assign. http://www.freebsd.org/cgi/query-pr.cgi?pr=184505 From owner-freebsd-scsi@FreeBSD.ORG Wed Apr 16 01:34:00 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D575211; Wed, 16 Apr 2014 01:34:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E4EB6173A; Wed, 16 Apr 2014 01:33:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s3G1XxtX011888; Wed, 16 Apr 2014 01:33:59 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s3G1XxGX011887; Wed, 16 Apr 2014 01:33:59 GMT (envelope-from linimon) Date: Wed, 16 Apr 2014 01:33:59 GMT Message-Id: <201404160133.s3G1XxGX011887@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/188270: [mpt] mpt unable to communicate with LSI7202XP Fibrechannel X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2014 01:34:00 -0000 Old Synopsis: mpt unable to communicate with LSI7202XP Fibrechannel New Synopsis: [mpt] mpt unable to communicate with LSI7202XP Fibrechannel Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Wed Apr 16 01:33:07 UTC 2014 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=188270 From owner-freebsd-scsi@FreeBSD.ORG Thu Apr 17 04:19:00 2014 Return-Path: Delivered-To: freebsd-scsi@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 972F9162; Thu, 17 Apr 2014 04:19:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B4501F9B; Thu, 17 Apr 2014 04:19:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s3H4J0e6057475; Thu, 17 Apr 2014 04:19:00 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s3H4J0hk057474; Thu, 17 Apr 2014 04:19:00 GMT (envelope-from linimon) Date: Thu, 17 Apr 2014 04:19:00 GMT Message-Id: <201404170419.s3H4J0hk057474@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-scsi@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/175670: [iscsi] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 04:19:00 -0000 Old Synopsis: smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) New Synopsis: [iscsi] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi Responsible-Changed-By: linimon Responsible-Changed-When: Thu Apr 17 04:18:15 UTC 2014 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=175670 From owner-freebsd-scsi@FreeBSD.ORG Thu Apr 17 16:57:58 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 85851125; Thu, 17 Apr 2014 16:57:58 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 08411170B; Thu, 17 Apr 2014 16:57:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 25C342041B7; Thu, 17 Apr 2014 18:48:34 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rv9z1mm1mHnb; Thu, 17 Apr 2014 18:48:33 +0200 (CEST) Received: from [10.7.0.6] (unknown [10.7.0.6]) by smtp.infotech.no (Postfix) with ESMTPA id 708A120416A; Thu, 17 Apr 2014 18:48:32 +0200 (CEST) Message-ID: <535005DC.7060609@interlog.com> Date: Thu, 17 Apr 2014 12:48:28 -0400 From: Douglas Gilbert User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-scsi@FreeBSD.org, Smartmontools Mailing List Subject: Re: kern/175670: [iscsi] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) References: <201404170419.s3H4J0hk057474@freefall.freebsd.org> In-Reply-To: <201404170419.s3H4J0hk057474@freefall.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: dgilbert@interlog.com List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 16:57:58 -0000 On 14-04-17 12:19 AM, linimon@FreeBSD.org wrote: > Old Synopsis: smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) > New Synopsis: [iscsi] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) > > Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi > Responsible-Changed-By: linimon > Responsible-Changed-When: Thu Apr 17 04:18:15 UTC 2014 > Responsible-Changed-Why: > reclassify. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=175670 I have one of those disks and it works for me in: FreeBSD 10.0-RELEASE #0 r260789 # smartctl -i /dev/pass2 smartctl 6.3 2014-04-10 r3888 [FreeBSD 10.0-RELEASE i386] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST33000650SS Revision: 0002 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Logical block size: 512 bytes Formatted with type 1 protection Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c50033111111 Serial number: 9XK0JN6Z0000xxxxxxxx Device type: disk Transport protocol: SAS Local Time is: Thu Apr 17 12:36:57 2014 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled This was built from the latest smartmontools developer repository because the FreeBSD ports version of smartmontool was broken, see below. The reporter had Revision 4 of the disk's firmware while mine is Revision 2. The reported error said the IEC mode page was broken. No surprise there, mine is too but in a different way: [root@sas ~]# sg_modes -p 0x1c /dev/pass2 SEAGATE ST33000650SS 0002 peripheral_type: disk [0x0] Mode parameter header from MODE SENSE(10): Mode data length=28, medium type=0x00, WP=0, DpoFua=1, longlba=0 Block descriptor length=8 > Direct access device block descriptors: Density code=0x0 00 ff ff ff ff 00 00 02 00 >> Informational exceptions control, page_control: current 00 9c 0a 10 00 00 00 00 00 00 00 00 01 [root@sas ~]# sg_modes -p 0x1c -L /dev/pass2 SEAGATE ST33000650SS 0002 peripheral_type: disk [0x0] Mode parameter header from MODE SENSE(10): Mode data length=36, medium type=0x00, WP=0, DpoFua=1, longlba=1 Block descriptor length=16 > longlba direct access device block descriptors: Density code=0x0 00 00 00 00 01 5d 50 a3 b0 00 00 00 00 00 >> Informational exceptions control, page_control: current 00 9c 0a 10 00 00 00 00 00 00 00 00 01 The long LBA version says the logical block size is 0 bytes. Well done Seagate! So IMO this is not a FreeBSD error and most likely not a smartmontools error. Also it has nothing to do with the HBA. But I did see this when I tried to build smartmontools from ports: [root@sas /usr/ports/sysutils/smartmontools]# make install make: "/usr/ports/Mk/bsd.port.mk" line 5121: warning: duplicate script for target "-depends" ignored make: "/usr/ports/Mk/bsd.port.mk" line 5118: warning: using previous script for "-depends" defined here make: "/usr/ports/Mk/bsd.port.mk" line 5121: warning: duplicate script for target "-depends" ignored make: "/usr/ports/Mk/bsd.port.mk" line 5118: warning: using previous script for "-depends" defined here make: "/usr/ports/Mk/bsd.port.mk" line 5121: warning: duplicate script for target "-depends" ignored make: "/usr/ports/Mk/bsd.port.mk" line 5118: warning: using previous script for "-depends" defined here make: "/usr/ports/Mk/bsd.port.mk" line 5118: warning: duplicate script for target "-depends" ignored make: "/usr/ports/Mk/bsd.port.mk" line 5118: warning: using previous script for "-depends" defined here make: "/usr/ports/Mk/bsd.port.mk" line 5121: warning: duplicate script for target "-depends" ignored make: "/usr/ports/Mk/bsd.port.mk" line 5118: warning: using previous script for "-depends" defined here ===> smartmontools-6.0 improper use of USE_PERL5. *** Error code 1 Stop. make: stopped in /usr/ports/sysutils/smartmontools Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Mon Apr 21 11:06:53 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70917114 for ; Mon, 21 Apr 2014 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 455531970 for ; Mon, 21 Apr 2014 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s3LB6rmd085838 for ; Mon, 21 Apr 2014 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s3LB6qbq085836 for freebsd-scsi@FreeBSD.org; Mon, 21 Apr 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 21 Apr 2014 11:06:52 GMT Message-Id: <201404211106.s3LB6qbq085836@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Apr 2014 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/188270 scsi [mpt] mpt unable to communicate with LSI7202XP Fibrech o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184505 scsi [ahd] AHD controller/Bad Negotiation/3 out of 4 boots o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/175670 scsi [iscsi] smartctl fails on SAS disk connected to an Int o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 22 problems total. From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 22 10:04:15 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 80041EF5 for ; Tue, 22 Apr 2014 10:04:15 +0000 (UTC) Received: from omr-d06.mx.aol.com (omr-d06.mx.aol.com [205.188.109.203]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33C8A1FD7 for ; Tue, 22 Apr 2014 10:04:15 +0000 (UTC) Received: from mtaout-mcc01.mx.aol.com (mtaout-mcc01.mx.aol.com [172.26.253.77]) by omr-d06.mx.aol.com (Outbound Mail Relay) with ESMTP id D1C5D70004138 for ; Tue, 22 Apr 2014 06:04:07 -0400 (EDT) Received: from MSG0000019 (unknown [122.177.189.10]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mtaout-mcc01.mx.aol.com (MUA/Third Party Client Interface) with ESMTPSA id 12E2638000082 for ; Tue, 22 Apr 2014 06:04:05 -0400 (EDT) From: "joel" To: Subject: New Method to Rank your website on 1st Page of Google Date: Tue, 22 Apr 2014 15:32:05 +0530 Message-ID: <016101cf5e12$2f848530$8e8d8f90$@web@aol.com> MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac9eEWVPrIQT79WySdGghL3hNIrOMA== Content-Language: en-us x-aol-global-disposition: G DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mx.aol.com; s=20121107; t=1398161047; bh=/K6RCqXLIiCWWJK3g1HXQEiomAIO/9IGn5VdTYYa7LY=; h=From:To:Subject:Message-ID:Date:MIME-Version:Content-Type; b=j/jD9rSAkhikXWjlki0dXpP+zk/Ud/B9aV0pONg0UCSwM5ia5P2kAYdFhP8rtNLsm HpOgaW1WVaJIeZFFWNL3vY10HZPjA0fT9qzkZ3n1h/AaHmycj6FIPejqIww1LVsbLX dm5XO7yF32we2IjR6ClMeoNuhqEOmY+5NcRJSTa4= x-aol-sid: 3039ac1afd4d53563e952574 X-AOL-IP: 122.177.189.10 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Apr 2014 10:04:15 -0000 Hello, We are a leading result oriented Web Solutions Company in India and we are doing this from last 3 years for many SEO agency based in USA, UK, Canada and Australia. We do all our work in an organic basis so that our clients get a permanent result. We will be optimizing your website in the major search engines like Google, Yahoo & Bing which results in improvements in keyword ranking, traffic & link popularity. Our SEO works helps your website in following ways: 1. Keyword Analysis 2. Competitor Analysis 3. SEO Analysis report 4. Finding & fixing all the technical errors from the website 5. On-page optimization (Designing stuff) 6. Adding landing pages if required. 7. Social Media Optimization etc 8. Off page optimization (Promotional stuff with other website to get quality Back links). 9. Content Writing 10. Reputation management. We assure you that after we start the SEO campaign for your website your site will be positioned in top page of Google, Yahoo, Msn search engines and your website will generate revenues as traffic will float your website. We do manual submissions which results in permanent quality back links, so that you can have your potential traffic on your prime pages and ultimately your ROI will increase. If you have any query, we will be more than happy to provide you our quick assistance. Best Regards! "Joel" Business Development Manager Other services offered by us: SMO, Web development (ASP .NET, VB, .NET, PHP, MYSQL, Os Commerce, shopping cart, joomla, Drupal, Magento... etc), Web Designing, Android development, I Phone apps ...etc Note :- If this is something you are interested, please respond to this email. If this is not your interest, don't worry, we will not email you again. From owner-freebsd-scsi@FreeBSD.ORG Mon Apr 28 11:06:54 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0502E56D for ; Mon, 28 Apr 2014 11:06:54 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CDB9E1AB1 for ; Mon, 28 Apr 2014 11:06:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s3SB6roe086262 for ; Mon, 28 Apr 2014 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s3SB6r7g086259 for freebsd-scsi@FreeBSD.org; Mon, 28 Apr 2014 11:06:53 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 28 Apr 2014 11:06:53 GMT Message-Id: <201404281106.s3SB6r7g086259@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Apr 2014 11:06:54 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/188270 scsi [mpt] mpt unable to communicate with LSI7202XP Fibrech o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184505 scsi [ahd] AHD controller/Bad Negotiation/3 out of 4 boots o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/175670 scsi [iscsi] smartctl fails on SAS disk connected to an Int o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 22 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon Apr 28 21:17:36 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 13D47AB5 for ; Mon, 28 Apr 2014 21:17:36 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id AFD8A1335 for ; Mon, 28 Apr 2014 21:17:35 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3SLHRvT041710 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 28 Apr 2014 22:17:28 +0100 (BST) Date: Mon, 28 Apr 2014 22:17:27 +0100 From: Karl Pielorz To: freebsd-scsi@freebsd.org Subject: New iSCSI Stack in 10.x - Prevent hangs on a dead target? Message-ID: X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Apr 2014 21:17:36 -0000 Hi, I've been setting up iSCSI with a couple of 10.x boxes recently (using the new iscsictl / ctld et'al). This seems to work well - but if I connect to a remote iSCSI target - and that host 'dies' I/O on the local /dev/daX device for that dead target just halts - for what seems to be 'indefinitely'. In /var/log/messages - I can see the system trying to reconnect (to the dead host) - but I can't see sign (or way of telling it) it to 'give up' and move on after some timeout. e.g. I have a bunch of iSCSI disks in use with ZFS - this works fine until the remote node dies (which takes half the disks with it). ZFS just halts all I/O then on the pool - until I do, e.g. 'iscsictl -R -p dead-host-ip'. Is there any way of setting this on either the initiator (or the target) - it looks like something like iSCSI Time2Retain might cover it - but I can't find anywhere to set that, or anything similar... Thanks, -Karl From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 29 08:43:10 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BFC73184 for ; Tue, 29 Apr 2014 08:43:10 +0000 (UTC) Received: from mail-ee0-x22c.google.com (mail-ee0-x22c.google.com [IPv6:2a00:1450:4013:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 534B063D for ; Tue, 29 Apr 2014 08:43:10 +0000 (UTC) Received: by mail-ee0-f44.google.com with SMTP id c41so10452eek.17 for ; Tue, 29 Apr 2014 01:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=sa2jT0paJJjFYoMbGygDAc5uO5eW8GN9hFZWTjQoKV0=; b=HzTEciFpRZ7Ta+ZYPUUbvL6VhQPmEtbbOkqRcbKRSFpjATeOumgCWBTISumfleePqV hotxItQ/dFh1sKkAAusY3uGFlnt03BpQuHCUg08JH/BDibi7uI0ZMPefaNABT+iLQOQW SiysapDTasfuv3Cq3w3lym5Zw9292KdPg/xIABa0v5iPFMzPyg/kkuXgMM5zI9/hW4kh roQ16f7/y1Eb0J7+URkarJS+e1bb85UDOEpVjLGVgyNgp1ONC++ke+6C8B8vQacjRD63 82D7BPpRTaM20adQra8jBuuID14y3wKGauc9EcQKXPR9+CGQsYN8bJsL23ytMJp7cB55 Nwdw== X-Received: by 10.14.214.68 with SMTP id b44mr39714598eep.0.1398760988597; Tue, 29 Apr 2014 01:43:08 -0700 (PDT) Received: from [192.168.1.14] (acc177.neoplus.adsl.tpnet.pl. [83.25.54.177]) by mx.google.com with ESMTPSA id q41sm57336399eez.7.2014.04.29.01.43.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 29 Apr 2014 01:43:07 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Subject: Re: New iSCSI Stack in 10.x - Prevent hangs on a dead target? Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-2 From: =?iso-8859-2?Q?Edward_Tomasz_Napiera=B3a?= In-Reply-To: Date: Tue, 29 Apr 2014 10:43:06 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <19D13ADF-EAE2-4CF7-BA67-562444CA0E2A@FreeBSD.org> References: To: Karl Pielorz X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Apr 2014 08:43:10 -0000 Wiadomo=B6=E6 napisana przez Karl Pielorz w dniu 28 kwi 2014, o godz. = 23:17: > Hi, >=20 > I've been setting up iSCSI with a couple of 10.x boxes recently (using = the new iscsictl / ctld et'al). >=20 > This seems to work well - but if I connect to a remote iSCSI target - = and that host 'dies' I/O on the local /dev/daX device for that dead = target just halts - for what seems to be 'indefinitely'. >=20 > In /var/log/messages - I can see the system trying to reconnect (to = the dead host) - but I can't see sign (or way of telling it) it to 'give = up' and move on after some timeout. >=20 > e.g. I have a bunch of iSCSI disks in use with ZFS - this works fine = until the remote node dies (which takes half the disks with it). >=20 > ZFS just halts all I/O then on the pool - until I do, e.g. 'iscsictl = -R -p dead-host-ip'. >=20 > Is there any way of setting this on either the initiator (or the = target) - it looks like something like iSCSI Time2Retain might cover it = - but I can't find anywhere to set that, or anything similar... In HEAD, there is "kern.iscsi.fail_on_disconnection" sysctl. It's = supposed to be used with gmultipath or ZFS; it basically makes connection error = result in IO error instead of "hang" until successful reconnection. The = initiator will still try to reconnect; when it succeeds, the disk devices will = reappear. Would that solve your problem? From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 29 09:14:44 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6822DACF; Tue, 29 Apr 2014 09:14:44 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 0DEBE99E; Tue, 29 Apr 2014 09:14:43 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3T9EfAh006133 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Apr 2014 10:14:41 +0100 (BST) Date: Tue, 29 Apr 2014 10:14:41 +0100 From: Karl Pielorz To: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Subject: Re: New iSCSI Stack in 10.x - Prevent hangs on a dead target? Message-ID: In-Reply-To: <19D13ADF-EAE2-4CF7-BA67-562444CA0E2A@FreeBSD.org> References: <19D13ADF-EAE2-4CF7-BA67-562444CA0E2A@FreeBSD.org> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Apr 2014 09:14:44 -0000 --On 29 April 2014 10:43:06 +0200 Edward Tomasz Napiera=C5=82a=20 wrote: > In HEAD, there is "kern.iscsi.fail_on_disconnection" sysctl. It's > supposed to be used with gmultipath or ZFS; it basically makes connection > error result in IO error instead of "hang" until successful reconnection. > The initiator will still try to reconnect; when it succeeds, the disk > devices will reappear. > > Would that solve your problem? Sounds like it would! - I did have a quick fgrep round the 10-stable source = on the system here - time2retain is mentioned in a .h file - but I couldn't = see any where else it was either used or referenced... Do you know if that sysctl/mechanism is likely to be ported to 10/stable? Thanks, -Karl From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 29 09:53:34 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A06721DE for ; Tue, 29 Apr 2014 09:53:34 +0000 (UTC) Received: from mail-ee0-x231.google.com (mail-ee0-x231.google.com [IPv6:2a00:1450:4013:c00::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3318CD9F for ; Tue, 29 Apr 2014 09:53:34 +0000 (UTC) Received: by mail-ee0-f49.google.com with SMTP id e53so112940eek.8 for ; Tue, 29 Apr 2014 02:53:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=dS/g89O8XIFaIB9yB/+rw6/XHKCtq4Mjg9AS5RWTZGE=; b=hffZIQgJFPIiKlnZOgWrz3ChfnXOxbEn+QYkjw6xojXDe+iLMyzHEIdYBXAHW6kpSo 9P7nGPqqungrmKiOQzDusrv+4wFJPUogsXV5JvLQ7fYQQJA3e0mIg87cA+qjSkOEMseP KvSQnKpyQm00e1WazL8CZQwchpRMl+KMTasHpb9Op8KS3cdgfouR9+ju+a1Vsmbdmy2E e21syIjo9TwuxMAwhaaZtgncJjC6zjwc6Bx5AFugkTdCBY62fzJg9ykYJMLe1YXVT7Nn stVXh5Q6eGhUSPb75TpZecDVY8O34i/q19FA+HRb9F5GRV1WYJtgj0MT84VtEnmuVCmu kqGg== X-Received: by 10.15.90.201 with SMTP id q49mr1918941eez.65.1398765211518; Tue, 29 Apr 2014 02:53:31 -0700 (PDT) Received: from [192.168.1.14] (acc177.neoplus.adsl.tpnet.pl. [83.25.54.177]) by mx.google.com with ESMTPSA id s46sm57805789ees.3.2014.04.29.02.53.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 29 Apr 2014 02:53:30 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Subject: Re: New iSCSI Stack in 10.x - Prevent hangs on a dead target? Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-2 From: =?iso-8859-2?Q?Edward_Tomasz_Napiera=B3a?= In-Reply-To: Date: Tue, 29 Apr 2014 11:53:28 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <358DDD5C-50F6-4997-9597-3FE40ED9A7B1@FreeBSD.org> References: <19D13ADF-EAE2-4CF7-BA67-562444CA0E2A@FreeBSD.org> To: Karl Pielorz X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Apr 2014 09:53:34 -0000 Wiadomo=B6=E6 napisana przez Karl Pielorz w dniu 29 kwi 2014, o godz. = 11:14: > --On 29 April 2014 10:43:06 +0200 Edward Tomasz Napiera=B3a = wrote: >=20 >> In HEAD, there is "kern.iscsi.fail_on_disconnection" sysctl. It's >> supposed to be used with gmultipath or ZFS; it basically makes = connection >> error result in IO error instead of "hang" until successful = reconnection. >> The initiator will still try to reconnect; when it succeeds, the disk >> devices will reappear. >>=20 >> Would that solve your problem? >=20 > Sounds like it would! - I did have a quick fgrep round the 10-stable = source on the system here - time2retain is mentioned in a .h file - but = I couldn't see any where else it was either used or referenced... >=20 > Do you know if that sysctl/mechanism is likely to be ported to = 10/stable? Yes, in a week or so. From owner-freebsd-scsi@FreeBSD.ORG Tue Apr 29 10:06:42 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9EDF2457; Tue, 29 Apr 2014 10:06:42 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 29BB9FA4; Tue, 29 Apr 2014 10:06:41 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s3TA6eP3010882 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Apr 2014 11:06:40 +0100 (BST) Date: Tue, 29 Apr 2014 11:06:40 +0100 From: Karl Pielorz To: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Subject: Re: New iSCSI Stack in 10.x - Prevent hangs on a dead target? Message-ID: <55BF991289CB8F36E3D84E9E@study64.tdx.co.uk> In-Reply-To: <358DDD5C-50F6-4997-9597-3FE40ED9A7B1@FreeBSD.org> References: <19D13ADF-EAE2-4CF7-BA67-562444CA0E2A@FreeBSD.org> <358DDD5C-50F6-4997-9597-3FE40ED9A7B1@FreeBSD.org> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Apr 2014 10:06:42 -0000 --On 29 April 2014 11:53:28 +0200 Edward Tomasz Napiera=C5=82a=20 wrote: >> Do you know if that sysctl/mechanism is likely to be ported to = 10/stable? > > Yes, in a week or so. Great! - I'll try to keep an eye out for it and try and leave everything=20 in place to easily test it here :-) Regards, -Karl From owner-freebsd-scsi@FreeBSD.ORG Mon May 5 11:06:50 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9965BEC6 for ; Mon, 5 May 2014 11:06:50 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6D4531CFD for ; Mon, 5 May 2014 11:06:50 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s45B6omi083242 for ; Mon, 5 May 2014 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s45B6nw3083240 for freebsd-scsi@FreeBSD.org; Mon, 5 May 2014 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 5 May 2014 11:06:49 GMT Message-Id: <201405051106.s45B6nw3083240@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 May 2014 11:06:50 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/188270 scsi [mpt] mpt unable to communicate with LSI7202XP Fibrech o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184505 scsi [ahd] AHD controller/Bad Negotiation/3 out of 4 boots o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/175670 scsi [iscsi] smartctl fails on SAS disk connected to an Int o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 22 problems total. From owner-freebsd-scsi@FreeBSD.ORG Wed May 7 09:05:06 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B9CB4EBE for ; Wed, 7 May 2014 09:05:06 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7BD91BB6 for ; Wed, 7 May 2014 09:05:05 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 0444A9DCD70 for ; Wed, 7 May 2014 10:59:15 +0200 (CEST) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Testing new mpr driver Date: Wed, 7 May 2014 10:59:14 +0200 Message-Id: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 09:05:06 -0000 Hi I just saw that there's a new driver for LSI3008 SAS3 cards. I have = updated a test system to -STABLE, installed one of those cards (in this = case, the OEM version sold by IBM as a HBA, with IT firmware). mpr0: port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 = at device 0.0 on pci17 Excellent so far I have noticed a difference with the mps based LSI2008 = cards: in the same hardware configuration as my previous tests (except, = of course, the HBA) it seems to be faster. And I am using just one cable = to connect the HBA to the backplane instead of two, I didn't have = another cable available. at scbus0 target 25 lun 0 (pass0,da0) at scbus0 target 26 lun 0 (pass1,da1) at scbus0 target 27 lun 0 (pass2,da2) at scbus0 target 28 lun 0 (pass3,da3) at scbus0 target 29 lun 0 (pass4,da4) at scbus0 target 30 lun 0 (pass5,da5) at scbus0 target 33 lun 0 (pass6,da6) at scbus0 target 34 lun 0 (pass7,da7) at scbus0 target 35 lun 0 (pass8,da8) at scbus0 target 263 lun 0 = (pass9,ses0) at scbus0 target 288 lun 0 = (pass10,ses1) I have created a ZFS pool with the SSDs. With the LSI2008 based card I = needed to run several bonnie++ instances in parallel in order to = saturate the I/O bandwidth, and with this card a single instance reaches = peak performance.=20 Does it make sense at all? I assume that the driver could be considered almost production ready. Is = it going to be included in 9.3 as well, as the manpage states, or is it = a typo? Thanks! Borja. From owner-freebsd-scsi@FreeBSD.ORG Wed May 7 13:34:16 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E185CC0 for ; Wed, 7 May 2014 13:34:16 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 1BE099B3 for ; Wed, 7 May 2014 13:34:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 552BF20416A; Wed, 7 May 2014 15:24:08 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B2LwvwxyczOp; Wed, 7 May 2014 15:24:08 +0200 (CEST) Received: from [10.7.0.6] (unknown [10.7.0.6]) by smtp.infotech.no (Postfix) with ESMTPA id 6AF63204163; Wed, 7 May 2014 15:24:07 +0200 (CEST) Message-ID: <536A33F6.7080401@interlog.com> Date: Wed, 07 May 2014 09:24:06 -0400 From: Douglas Gilbert Reply-To: dgilbert@interlog.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Borja Marcos , freebsd-scsi@freebsd.org Subject: Re: Testing new mpr driver References: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> In-Reply-To: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 13:34:16 -0000 On 14-05-07 04:59 AM, Borja Marcos wrote: > > Hi > > I just saw that there's a new driver for LSI3008 SAS3 cards. I have updated a test system to -STABLE, installed one of those cards (in this case, the OEM version sold by IBM as a HBA, with IT firmware). > > mpr0: port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > > > > > Excellent so far I have noticed a difference with the mps based LSI2008 cards: in the same hardware configuration as my previous tests (except, of course, the HBA) it seems to be faster. And I am using just one cable to connect the HBA to the backplane instead of two, I didn't have another cable available. > > at scbus0 target 25 lun 0 (pass0,da0) > at scbus0 target 26 lun 0 (pass1,da1) > at scbus0 target 27 lun 0 (pass2,da2) > at scbus0 target 28 lun 0 (pass3,da3) > at scbus0 target 29 lun 0 (pass4,da4) > at scbus0 target 30 lun 0 (pass5,da5) > at scbus0 target 33 lun 0 (pass6,da6) > at scbus0 target 34 lun 0 (pass7,da7) > at scbus0 target 35 lun 0 (pass8,da8) > at scbus0 target 263 lun 0 (pass9,ses0) > at scbus0 target 288 lun 0 (pass10,ses1) > > I have created a ZFS pool with the SSDs. With the LSI2008 based card I needed to run several bonnie++ instances in parallel in order to saturate the I/O bandwidth, and with this card a single instance reaches peak performance. > > Does it make sense at all? > > I assume that the driver could be considered almost production ready. Is it going to be included in 9.3 as well, as the manpage states, or is it a typo? I'm keen to try the mpr driver and have LSI 9300-4i/4i4e HBAs plus a 12 Gbps SSD and SAS-2 bits to test with it. [Free-standing SAS-3 expanders are not common (but Areca just released one).] My FreeBSD system is at 10.0-RELEASE-p2 and moving it to -STABLE looks like a fair amount of pain. Further I can see nothing on the FreeBSD site about the mpr driver. Could you supply links to any information about this driver? Is it likely to appear in 10.0-RELEASE-p? any time soon? FreeBSD is seriously lacking in the SAS-3 area compared to some other OSes. As SATA SSDs are strangled by their 6 Gbps (single port) interface, several new SAS 12 Gbps (SAS-3) products have appeared in the last month or so, with more to follow. Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Wed May 7 13:41:22 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08CA5EE2 for ; Wed, 7 May 2014 13:41:22 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BDADFA7C for ; Wed, 7 May 2014 13:41:21 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id E40EF9DD130; Wed, 7 May 2014 15:41:17 +0200 (CEST) Subject: Re: Testing new mpr driver Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos In-Reply-To: <536A33F6.7080401@interlog.com> Date: Wed, 7 May 2014 15:41:16 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> <536A33F6.7080401@interlog.com> To: dgilbert@interlog.com X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 13:41:22 -0000 On May 7, 2014, at 3:24 PM, Douglas Gilbert wrote: > I'm keen to try the mpr driver and have LSI 9300-4i/4i4e > HBAs plus a 12 Gbps SSD and SAS-2 bits to test with it. > [Free-standing SAS-3 expanders are not common (but Areca just > released one).] >=20 > My FreeBSD system is at 10.0-RELEASE-p2 and moving it to > -STABLE looks like a fair amount of pain. Further I can see > nothing on the FreeBSD site about the mpr driver. Could you > supply links to any information about this driver? Is it > likely to appear in 10.0-RELEASE-p? any time soon? I've developed a habit of checking the Subversion tree, and I noticed a = note in -STABLE,=20 the addition of the mpr(4) driver. I guess it will be on 10.1, as it has been committed to -STABLE. http://svnweb.freebsd.org/base?view=3Drevision&revision=3D265388 If you, for example, svnup your source tree to the latest and compile, = you should have a new driver in GENERIC, called mpr. Borja. From owner-freebsd-scsi@FreeBSD.ORG Wed May 7 18:46:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 88CA43F4 for ; Wed, 7 May 2014 18:46:00 +0000 (UTC) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 43366DD1 for ; Wed, 7 May 2014 18:45:59 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id s47IjwRh080549; Wed, 7 May 2014 12:45:58 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id s47IjvRG080548; Wed, 7 May 2014 12:45:57 -0600 (MDT) (envelope-from ken) Date: Wed, 7 May 2014 12:45:57 -0600 From: "Kenneth D. Merry" To: Borja Marcos Subject: Re: Testing new mpr driver Message-ID: <20140507184557.GA80243@nargothrond.kdm.org> References: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> User-Agent: Mutt/1.4.2i Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 18:46:00 -0000 On Wed, May 07, 2014 at 10:59:14 +0200, Borja Marcos wrote: > > Hi > > I just saw that there's a new driver for LSI3008 SAS3 cards. I have updated a test system to -STABLE, installed one of those cards (in this case, the OEM version sold by IBM as a HBA, with IT firmware). > > mpr0: port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 > > > > > Excellent so far I have noticed a difference with the mps based LSI2008 cards: in the same hardware configuration as my previous tests (except, of course, the HBA) it seems to be faster. And I am using just one cable to connect the HBA to the backplane instead of two, I didn't have another cable available. > > at scbus0 target 25 lun 0 (pass0,da0) > at scbus0 target 26 lun 0 (pass1,da1) > at scbus0 target 27 lun 0 (pass2,da2) > at scbus0 target 28 lun 0 (pass3,da3) > at scbus0 target 29 lun 0 (pass4,da4) > at scbus0 target 30 lun 0 (pass5,da5) > at scbus0 target 33 lun 0 (pass6,da6) > at scbus0 target 34 lun 0 (pass7,da7) > at scbus0 target 35 lun 0 (pass8,da8) > at scbus0 target 263 lun 0 (pass9,ses0) > at scbus0 target 288 lun 0 (pass10,ses1) > > I have created a ZFS pool with the SSDs. With the LSI2008 based card I needed to run several bonnie++ instances in parallel in order to saturate the I/O bandwidth, and with this card a single instance reaches peak performance. > > Does it make sense at all? That's hard to say. If you're using a 6Gb expander, you would have half of the available SAS bandwidth if you only connected four lanes from the controller to the expander instead of 8. If you somehow have a 12Gb expander (it isn't obvious from the model number above what the expander speed is), then you would have the same amount of bandwidth. One thing that could be happening is you may have lower latency through the new 12Gb controller. > I assume that the driver could be considered almost production ready. Is it going to be included in 9.3 as well, as the manpage states, or is it a typo? > It is the same driver that LSI has had on their web site, but brought up to date with all of the changes that have gone into mps(4) lately. We're planning to merge it to stable/9 so it will go into 9.3, but there were some bugs in stable/9 support that I fixed in the past couple of days. We have to wait for the 3 day waiting period to expire before merging the driver and all the fixes into stable/9. By the way, if you run with INVARIANTS enabled, you may run into some issues (i.e. a panic) on reboot until we merge r265485 to stable/10. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Wed May 7 19:04:37 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 24F9FBB8 for ; Wed, 7 May 2014 19:04:37 +0000 (UTC) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E9CCEF89 for ; Wed, 7 May 2014 19:04:36 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id s47J4Zhg081203; Wed, 7 May 2014 13:04:35 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id s47J4ZFN081202; Wed, 7 May 2014 13:04:35 -0600 (MDT) (envelope-from ken) Date: Wed, 7 May 2014 13:04:35 -0600 From: "Kenneth D. Merry" To: Douglas Gilbert Subject: Re: Testing new mpr driver Message-ID: <20140507190435.GB80243@nargothrond.kdm.org> References: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> <536A33F6.7080401@interlog.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <536A33F6.7080401@interlog.com> User-Agent: Mutt/1.4.2i Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2014 19:04:37 -0000 On Wed, May 07, 2014 at 09:24:06 -0400, Douglas Gilbert wrote: > On 14-05-07 04:59 AM, Borja Marcos wrote: > > > >Hi > > > >I just saw that there's a new driver for LSI3008 SAS3 cards. I have > >updated a test system to -STABLE, installed one of those cards (in this > >case, the OEM version sold by IBM as a HBA, with IT firmware). > > > >mpr0: port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 at > >device 0.0 on pci17 > > > > > > > > > >Excellent so far I have noticed a difference with the mps based LSI2008 > >cards: in the same hardware configuration as my previous tests (except, of > >course, the HBA) it seems to be faster. And I am using just one cable to > >connect the HBA to the backplane instead of two, I didn't have another > >cable available. > > > > at scbus0 target 25 lun 0 (pass0,da0) > > at scbus0 target 26 lun 0 (pass1,da1) > > at scbus0 target 27 lun 0 (pass2,da2) > > at scbus0 target 28 lun 0 (pass3,da3) > > at scbus0 target 29 lun 0 (pass4,da4) > > at scbus0 target 30 lun 0 (pass5,da5) > > at scbus0 target 33 lun 0 (pass6,da6) > > at scbus0 target 34 lun 0 (pass7,da7) > > at scbus0 target 35 lun 0 (pass8,da8) > > at scbus0 target 263 lun 0 (pass9,ses0) > > at scbus0 target 288 lun 0 (pass10,ses1) > > > >I have created a ZFS pool with the SSDs. With the LSI2008 based card I > >needed to run several bonnie++ instances in parallel in order to saturate > >the I/O bandwidth, and with this card a single instance reaches peak > >performance. > > > >Does it make sense at all? > > > >I assume that the driver could be considered almost production ready. Is > >it going to be included in 9.3 as well, as the manpage states, or is it a > >typo? > > I'm keen to try the mpr driver and have LSI 9300-4i/4i4e > HBAs plus a 12 Gbps SSD and SAS-2 bits to test with it. > [Free-standing SAS-3 expanders are not common (but Areca just > released one).] > > My FreeBSD system is at 10.0-RELEASE-p2 and moving it to > -STABLE looks like a fair amount of pain. Further I can see > nothing on the FreeBSD site about the mpr driver. Could you > supply links to any information about this driver? Is it > likely to appear in 10.0-RELEASE-p? any time soon? The -p releases are generally used for patching security vulnerabilities, I believe. If you'd rather not do a buildworld and installworld to upgrade from source, you can also install a FreeBSD snapshot of stable/10 once one is available with the driver. You'll want one that is r265388 or newer. There are snapshots here: ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/10.0/ The driver is the same driver as the one LSI has had on their web site, with the recent fixes from the mps(4) 6Gb driver applied. The mpr(4) driver is essentially the same driver as mps(4), but with 12Gb support added, WarpDrive support removed and the S/G list support changed for the new hardware. It is a separate driver to reduce the testing load on LSI for new driver releases. > FreeBSD is seriously lacking in the SAS-3 area compared to > some other OSes. As SATA SSDs are strangled by their 6 Gbps > (single port) interface, several new SAS 12 Gbps (SAS-3) > products have appeared in the last month or so, with more > to follow. Hopefully this will start to get better. We've just added a committer from LSI (Steve McConnell), so LSI will be able to maintain their drivers directly in the tree instead of going through other folks (like me) who have varying amounts of bandwidth to help. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Thu May 8 07:59:07 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EA4F1D58; Thu, 8 May 2014 07:59:07 +0000 (UTC) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA512BB2; Thu, 8 May 2014 07:59:06 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id D97479DE3BB; Thu, 8 May 2014 09:49:03 +0200 (CEST) Subject: Re: Testing new mpr driver Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140507184557.GA80243@nargothrond.kdm.org> Date: Thu, 8 May 2014 09:49:02 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <4DB83981-0D4D-4484-BC89-4ED8C02DCD0F@sarenet.es> References: <8A41AB90-AC2F-4200-91D6-3D3CF9E8A835@sarenet.es> <20140507184557.GA80243@nargothrond.kdm.org> To: Kenneth D. Merry X-Mailer: Apple Mail (2.1283) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 May 2014 07:59:08 -0000 On May 7, 2014, at 8:45 PM, Kenneth D. Merry wrote: > That's hard to say. If you're using a 6Gb expander, you would have = half of > the available SAS bandwidth if you only connected four lanes from the > controller to the expander instead of 8. If you somehow have a 12Gb > expander (it isn't obvious from the model number above what the = expander > speed is), then you would have the same amount of bandwidth. Anyway, as far as I understand (SAS expanders perform link switching, = right?) the actual speed will be limited by the end to end speed. As the disks I am using are SATA, not SAS, each = lane would be anyway working at=20 6 Gbps instead of 12.=20 > One thing that could be happening is you may have lower latency = through the > new 12Gb controller. As I saw it supports something called "fastpath" (I must read something, = I am a bit outdated in these matters) I imagined that it=20 might be a more efficient transfer method which, despite working at 6 = Gbps, might explain a gain in performance. > By the way, if you run with INVARIANTS enabled, you may run into some > issues (i.e. a panic) on reboot until we merge r265485 to stable/10. I am running stable/10, and I'm not running INVARIANTS. Anyway I will = be tracking stable/10 closely, thank you! Great to have LSI so seriously involved! Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu May 8 13:45:39 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EA4BBFE for ; Thu, 8 May 2014 13:45:39 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id EC6D5135 for ; Thu, 8 May 2014 13:45:38 +0000 (UTC) Received: from Mail-PC.tdx.co.uk (storm.tdx.co.uk [62.13.130.251]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s48DjZWK064677 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 8 May 2014 14:45:36 +0100 (BST) Date: Thu, 08 May 2014 14:45:35 +0100 From: Karl Pielorz To: freebsd-scsi@freebsd.org Subject: 'Wiring down' iSCSI devices / stop swapping dev nodes... Message-ID: <963D4F6D5D95769A2C8A07D1@Mail-PC.tdx.co.uk> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 May 2014 13:45:39 -0000 Hi, I'm using the new iSCSI stack on FreeBSD 10. Seems to work fine so far - but I've noticed an issue (which is probably not related to new/old stack). I have a number of iSCSI targets setup - when they initially connect, they appear as '/dev/da[6/7/8/9/10/11/12/13]' However, if the remote node dies - when the iSCSI targets reconnect some of the drives transpose positions. I'm not specifically reconnecting them - just bringing the remote back up (when ctld / iscsid load up) - causes the local node to 'see' the drives have returned after a while (i.e. I don't run any iscsictl commands or anything). e.g. This time round drive previously connected to '/dev/da11' has now transposed with the drive that was connected to '/dev/da12' Is there any way I can prevent this? - i.e. by pre-alloctating / specifying '/dev/da' entries for devices or something? It's easy to see it happening on this test system as the drives are a mix of 750Gb and 2Tb - so when a 2Tb switches places with a 750Gb it's pretty obvious :) -Karl From owner-freebsd-scsi@FreeBSD.ORG Thu May 8 14:12:30 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1AB23D76 for ; Thu, 8 May 2014 14:12:30 +0000 (UTC) Received: from post.cs.huji.ac.il (post.cs.huji.ac.il [132.65.116.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C44C45F9 for ; Thu, 8 May 2014 14:12:29 +0000 (UTC) Received: from th-04.cs.huji.ac.il ([132.65.80.125]) by post.cs.huji.ac.il with esmtp id 1WiP3i-000Jwh-1u; Thu, 08 May 2014 17:12:26 +0300 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: 'Wiring down' iSCSI devices / stop swapping dev nodes... From: Daniel Braniss In-Reply-To: <963D4F6D5D95769A2C8A07D1@Mail-PC.tdx.co.uk> Date: Thu, 8 May 2014 17:12:24 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: References: <963D4F6D5D95769A2C8A07D1@Mail-PC.tdx.co.uk> To: Karl Pielorz X-Mailer: Apple Mail (2.1874) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 May 2014 14:12:30 -0000 On May 8, 2014, at 4:45 PM, Karl Pielorz wrote: >=20 > Hi, >=20 > I'm using the new iSCSI stack on FreeBSD 10. Seems to work fine so far = - but I've noticed an issue (which is probably not related to new/old = stack). >=20 > I have a number of iSCSI targets setup - when they initially connect, = they appear as '/dev/da[6/7/8/9/10/11/12/13]' >=20 > However, if the remote node dies - when the iSCSI targets reconnect = some of the drives transpose positions. >=20 > I'm not specifically reconnecting them - just bringing the remote back = up (when ctld / iscsid load up) - causes the local node to 'see' the = drives have returned after a while (i.e. I don't run any iscsictl = commands or anything). >=20 > e.g. This time round drive previously connected to '/dev/da11' has now = transposed with the drive that was connected to '/dev/da12' >=20 > Is there any way I can prevent this? - i.e. by pre-alloctating / = specifying '/dev/da' entries for devices or something? >=20 > It's easy to see it happening on this test system as the drives are a = mix of 750Gb and 2Tb - so when a 2Tb switches places with a 750Gb it's = pretty obvious :) >=20 > -Karl one solution is to use gpart (8) gpart create -s GPT /dev/dan =85 gpart add -t freebsd-ufs -l d0/p1 /dev/dan and you can use now /dev/gpt/d0/p1 which [should] be true whatever iscsi decides to call device. danny From owner-freebsd-scsi@FreeBSD.ORG Thu May 8 18:32:55 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 74226DDE for ; Thu, 8 May 2014 18:32:55 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 1A5F0251 for ; Thu, 8 May 2014 18:32:54 +0000 (UTC) Received: from study64.tdx.co.uk (study64.tdx.co.uk [62.13.130.231]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s48IWegA090004 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 8 May 2014 19:32:41 +0100 (BST) Date: Thu, 08 May 2014 19:32:41 +0100 From: Karl Pielorz To: Daniel Braniss Subject: Re: 'Wiring down' iSCSI devices / stop swapping dev nodes... Message-ID: <88E7AF042B7C82964163CEAF@study64.tdx.co.uk> In-Reply-To: References: <963D4F6D5D95769A2C8A07D1@Mail-PC.tdx.co.uk> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 May 2014 18:32:55 -0000 --On 8 May 2014 17:12:24 +0300 Daniel Braniss wrote: > one solution is to use gpart (8) > gpart create -s GPT /dev/dan > =E2=80=A6 > gpart add -t freebsd-ufs -l d0/p1 /dev/dan > > and you can use now > /dev/gpt/d0/p1 > > which [should] be true whatever iscsi decides to call device. I hadn't thought of that - mostly because the devices underlying the iSCSI=20 are already gparted (i.e. the iSCSI target is a GPT partition on an=20 underlying disk). I'm just trying it now - looks like it will work (Ok, it's created - I need = to test failure etc.) Is there any performance hit from having 'GPT within a GPT' kind of thing=20 going on? - I suppose I could just share out the raw disk (i.e. /dev/daX)=20 via iSCSI and then there'd be only one layer of partitioning going on...=20 I'm not sure if that's a good idea or not :) -Karl From owner-freebsd-scsi@FreeBSD.ORG Fri May 9 06:17:38 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 01D23E80 for ; Fri, 9 May 2014 06:17:38 +0000 (UTC) Received: from post.cs.huji.ac.il (post.cs.huji.ac.il [132.65.116.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A9AEBB4A for ; Fri, 9 May 2014 06:17:37 +0000 (UTC) Received: from th-04.cs.huji.ac.il ([132.65.80.125]) by post.cs.huji.ac.il with esmtp id 1Wie7i-000IXt-J7; Fri, 09 May 2014 09:17:34 +0300 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: 'Wiring down' iSCSI devices / stop swapping dev nodes... From: Daniel Braniss In-Reply-To: <88E7AF042B7C82964163CEAF@study64.tdx.co.uk> Date: Fri, 9 May 2014 09:17:33 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <620EE312-8B43-4E65-A534-6CAA92982CAD@cs.huji.ac.il> References: <963D4F6D5D95769A2C8A07D1@Mail-PC.tdx.co.uk> <88E7AF042B7C82964163CEAF@study64.tdx.co.uk> To: Karl Pielorz X-Mailer: Apple Mail (2.1874) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 May 2014 06:17:38 -0000 On May 8, 2014, at 9:32 PM, Karl Pielorz wrote: >=20 >=20 > --On 8 May 2014 17:12:24 +0300 Daniel Braniss = wrote: >=20 >> one solution is to use gpart (8) >> gpart create -s GPT /dev/dan >> =85 >> gpart add -t freebsd-ufs -l d0/p1 /dev/dan >>=20 >> and you can use now >> /dev/gpt/d0/p1 >>=20 >> which [should] be true whatever iscsi decides to call device. >=20 > I hadn't thought of that - mostly because the devices underlying the = iSCSI are already gparted (i.e. the iSCSI target is a GPT partition on = an underlying disk). >=20 > I'm just trying it now - looks like it will work (Ok, it's created - I = need to test failure etc.) >=20 > Is there any performance hit from having 'GPT within a GPT' kind of = thing going on? - I suppose I could just share out the raw disk (i.e. = /dev/daX) via iSCSI and then there'd be only one layer of partitioning = going on... I'm not sure if that's a good idea or not :) just from my guts, I don=92t think there is any noticeable overhead, = considering that TCP already adds allot, one more indirection is probably negligible. danny From owner-freebsd-scsi@FreeBSD.ORG Mon May 12 11:06:51 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4FB96BFC for ; Mon, 12 May 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 245D326D9 for ; Mon, 12 May 2014 11:06:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s4CB6paV067958 for ; Mon, 12 May 2014 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s4CB6oX7067956 for freebsd-scsi@FreeBSD.org; Mon, 12 May 2014 11:06:50 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 12 May 2014 11:06:50 GMT Message-Id: <201405121106.s4CB6oX7067956@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2014 11:06:51 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/188270 scsi [mpt] mpt unable to communicate with LSI7202XP Fibrech o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184505 scsi [ahd] AHD controller/Bad Negotiation/3 out of 4 boots o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/175670 scsi [iscsi] smartctl fails on SAS disk connected to an Int o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 22 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon May 19 11:06:52 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B646B4C9 for ; Mon, 19 May 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8AC912DC7 for ; Mon, 19 May 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s4JB6qHt080157 for ; Mon, 19 May 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s4JB6quE080155 for freebsd-scsi@FreeBSD.org; Mon, 19 May 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 19 May 2014 11:06:52 GMT Message-Id: <201405191106.s4JB6quE080155@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 May 2014 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/188270 scsi [mpt] mpt unable to communicate with LSI7202XP Fibrech o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184505 scsi [ahd] AHD controller/Bad Negotiation/3 out of 4 boots o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/175670 scsi [iscsi] smartctl fails on SAS disk connected to an Int o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 22 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon May 26 07:55:18 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CACD89C6 for ; Mon, 26 May 2014 07:55:18 +0000 (UTC) Received: from mail-pa0-f66.google.com (mail-pa0-f66.google.com [209.85.220.66]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D7BD235F for ; Mon, 26 May 2014 07:55:17 +0000 (UTC) Received: by mail-pa0-f66.google.com with SMTP id ey11so4704921pad.9 for ; Mon, 26 May 2014 00:55:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:reply-to:from:to:subject:date :mime-version:content-type; bh=4cDB4G32QKQvj8Eo0hBiwirRF8DS51VJAS1l9dSUiIk=; b=BpKMYnKsAogMTamu+CisX2qceryk+mrTcBSGdAOdGKNrJXJsFHTTHlpS9Jl2CA9SB9 chXbuOg6SNBL3fVAlk6LszbxLgp3VIx66tiegPiUa70j7K7+VtBVd5fGmKF1UksIcZ/0 xSzxhg8Ltdg95Vr2IjP5LbUgdUOlWXxJDHTDn//ILheuBAdTgB4A2N0ktRSG+7hrD9Mi +1WfpjAQhnKDZO1Lz04MVYeZW98lNcBmqVQrxX4tfuNLOFCNhDnGvFU6WSj0mwqjmhpY JnZww+iG5tESuRYodzlFy9alkn9Uz1DMG1ZegmTnuf/3R11Qo9RnoRno4NqRBg0j0DEe d4Yg== X-Gm-Message-State: ALoCoQkR79PcQFi/MgZXAp82n4RmsBQYBT+9jhC2wTiajlgpUfAx1tqml04EYiDH7YCZ2JfGep7H X-Received: by 10.68.237.228 with SMTP id vf4mr25389459pbc.131.1401083478602; Sun, 25 May 2014 22:51:18 -0700 (PDT) Received: from MSG0000019 ([122.177.87.136]) by mx.google.com with ESMTPSA id be7sm53122741pad.9.2014.05.25.22.51.16 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Sun, 25 May 2014 22:51:17 -0700 (PDT) Message-ID: <1f3801cf78a6$7c99eb40$db01a8c0@MSG0000019> Reply-To: "Helen" From: "Helen" To: Subject: your website to appear on first page of Google, Yahoo and Bing ..!! Date: Mon, 26 May 2014 10:55:42 +0530 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2180 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2014 07:55:18 -0000 Hope you are well!! I am Helen, Business Development Manager We are a Delhi-NCR, India based, and leading Web Solutions company with = main competency in SEO (Search Engine Optimization), working as an = outsourced vendor for many reputed SEO agency based in USA, UK, Canada = and Australia. If you want your website to appear on first page of Google then please = let me know. We have a SEO Special Discount Offer going for the following package: - Monthly Task and responsibilities: - Complete SEO Packages =20 SEO and social media Link Building Campaigns Ranking Reports Website Anatomy Future Of Search Keyword Analysis =20 Siloing Concepts Article Submissions Blog Submissions Unique article writing Meta tags changes suggestions Keyword research Competitor Analysis Alt tag changes Interlinking wherever required. Keyword Density in site content. HTML Site Map XML site map and Submission in webmaster tool.etc. Please let us know in case you are interested. Regards! "Helen" Marketing Executive Other services offered by us: SMO, Web development (ASP .NET, VB, .NET, = PHP, MYSQL, Os Commerce, cart, joomla, Drupal, Magento... etc), Web = Designing, Android development, I Phone apps ...etc Note:- If this is something you are interested, please respond to this = email. If this is not your interest, don't worry, we will not email = you again From owner-freebsd-scsi@FreeBSD.ORG Mon May 26 11:06:52 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B28A0F20 for ; Mon, 26 May 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 85FC024EE for ; Mon, 26 May 2014 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s4QB6q4k032154 for ; Mon, 26 May 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s4QB6qq0032152 for freebsd-scsi@FreeBSD.org; Mon, 26 May 2014 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 May 2014 11:06:52 GMT Message-Id: <201405261106.s4QB6qq0032152@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2014 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/188270 scsi [mpt] mpt unable to communicate with LSI7202XP Fibrech o kern/187903 scsi [ciss] Only 0.44 (always) days of uptime with ciss (w/ o kern/186258 scsi [mps] Heap overrun in mps(4) o kern/184975 scsi [ses] SCSI Environmental Services (ses) driver report o kern/184505 scsi [ahd] AHD controller/Bad Negotiation/3 out of 4 boots o kern/184059 scsi [mps] mps SCSI driver causes FreeBSD to hang during bo o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/175670 scsi [iscsi] smartctl fails on SAS disk connected to an Int o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 22 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon Jun 2 07:31:11 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47D697CA for ; Mon, 2 Jun 2014 07:31:11 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F7FD23B5 for ; Mon, 2 Jun 2014 07:31:11 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s527VBfE089821 for ; Mon, 2 Jun 2014 08:31:11 +0100 (BST) (envelope-from no-reply-bugzilla-daemon@freebsd.org) From: no-reply-bugzilla-daemon@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 190489] [iscsi] Native ISCSI - UNIT_ATTENTION with KVM Date: Mon, 02 Jun 2014 07:31:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: http://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2014 07:31:11 -0000 http://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190489 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-scsi@FreeBSD.org Summary|Native ISCSI - |[iscsi] Native ISCSI - |UNIT_ATTENTION with KVM |UNIT_ATTENTION with KVM --- Comment #1 from Mark Linimon --- Over to maintainer(s). -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Mon Jun 2 16:46:12 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5FED5EAF for ; Mon, 2 Jun 2014 16:46:12 +0000 (UTC) Received: from mail-ig0-f177.google.com (mail-ig0-f177.google.com [209.85.213.177]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2ACBE2B0B for ; Mon, 2 Jun 2014 16:46:11 +0000 (UTC) Received: by mail-ig0-f177.google.com with SMTP id l13so3627403iga.10 for ; Mon, 02 Jun 2014 09:46:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:content-type:subject:message-id:date :to:mime-version; bh=tBocR1gwIgqwC5idtN1t9k+wqualinY1HeuP9ZtkHvI=; b=fe7YDKEFdIExVYBJYmBaVumssHoKqtrdOR6OHFYlOIr8/A6ccwTRUWYte9TBS/gf2n Ms4lFlCUwX3HTmBxCJMIy3g6qoVDrIqjheg3ddSUJjEwyiVlZ17g7aymbNFp/Aht9uD0 lflC91gMcxsb8z6Qsw7w1hX+BOt0kvpSGXV4esf/jfdzVBLdTJq58gGQ0JndVIepsSY8 BkYLbvzOpBb01KIbsbBAGKQ84OEdRteb1I1xz1mEHCLMQnqLjM6a0IQHOd1+kGUAp+uv pWx0dqG/4Y9/s75h9wrq0bLT1GVTblbHzoipB4741cxhsklLLdSgtk5TWFUwAFmh/otA cnOQ== X-Gm-Message-State: ALoCoQmQN8m6YRtZscAEbYYPDt/M2SKS3enJjWVxmB3hDRYtcujzIWIZ4JyTIUbkrLqadJE1Qa3B X-Received: by 10.43.75.2 with SMTP id yy2mr37096360icb.54.1401727565311; Mon, 02 Jun 2014 09:46:05 -0700 (PDT) Received: from [10.0.0.119] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id g3sm17022086igo.11.2014.06.02.09.46.04 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 02 Jun 2014 09:46:04 -0700 (PDT) Sender: Warner Losh From: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_73DD6227-4E02-4592-98BE-AF91A6D19F95"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: Review needed Message-Id: Date: Mon, 2 Jun 2014 10:46:11 -0600 To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.2\)) X-Mailer: Apple Mail (2.1878.2) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2014 16:46:12 -0000 --Apple-Mail=_73DD6227-4E02-4592-98BE-AF91A6D19F95 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Greetings, The code that combines adjacent ranges for BIO_DELETEs to optimize trims to the device assumes the list is sorted. Don't apply the optimization of not sorting the queue when we have SSDs to the delete_queue, since it causes more discard traffic to the drive. While one could argue that the higher levels should coalesce the trims, that's not done today, so some optimization at this level is needed. Diffs and a (hopefully) cool review tool: https://phabric.freebsd.org/D142 Comments? Warner --Apple-Mail=_73DD6227-4E02-4592-98BE-AF91A6D19F95 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTjKpTAAoJEGwc0Sh9sBEA8ggQAJSOn9MsyR+di1XKAIUnK62x jnaSvkBD3Fq1Zd3aBeHgtarVE06eSLeufun/hgmrzbMHOENT+DQPckoXlw7rte7K eYrKusn5HPrxVEJLPVgtHv88960cso/qnttgxCuKtj+vCxg9X6+Cieb8rRYz3Cos M6L6c0wyy+4/hoxODvGU7hEdXrzWJiGCbb2Xg0Pzm/C2ExxP98dliXOhbNHbKNel iZdSPHNgw83NSBtMxrp/tmNBf2SnS3JiqsigJ7A6XB1ajs/K/Wh5l0ATPx+JKsG1 Y0O2m7ylcNOPD22vZq4LC6RjIpGXsDzjeAkQTWXuZzfHPQTVsvpclRViCsUokCnJ cnS31dtv8dqPSLBPfd1crklBofIOs7wlk2HCLxZcVKWiQzWldP9JAyWL8BApzYiU /f8hxielJZmIQbLjleWmv1YtbirfJNfCtIzcZQiiMidACqoXdplWMJOKS7Rvjek/ r9zxd8Bi/e7ORD3uAthVJc5oiO4+N/p4SbARxLiFy756UF2yOswFzamOExKD+d7M 8QuXxWQVc1wI2i2E6Xoa9/eUUMW9IEETU0KAEJRD6m/eNJHIH0ANTPncRtJQ9vBb hCvjcTdXJblr8fxrKIzo5rr9t6+rDUA10SXMqBXqEh6tujphKm229RWXwgeMI9fL q17MTB2gsHj63bbHN/eh =odDi -----END PGP SIGNATURE----- --Apple-Mail=_73DD6227-4E02-4592-98BE-AF91A6D19F95-- From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 3 17:10:11 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DC92E3DF for ; Tue, 3 Jun 2014 17:10:11 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C31562092 for ; Tue, 3 Jun 2014 17:10:11 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s53HABIc091057 for ; Tue, 3 Jun 2014 18:10:11 +0100 (BST) (envelope-from no-reply-bugzilla-daemon@freebsd.org) From: no-reply-bugzilla-daemon@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 184505] [ahd] AHD controller/Bad Negotiation/3 out of 4 boots Date: Tue, 03 Jun 2014 17:10:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-BETA4 X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: In Discussion X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2014 17:10:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=184505 --- Comment #2 from Larry Rosenman --- I've wiped/reinstalled this box with 11.0, and the issue seems(!) to have abated. However, the box is no longer as busy as it was, as it was moved to my house vs. being my colo. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 3 18:22:55 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B4A91F4 for ; Tue, 3 Jun 2014 18:22:55 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2516727F0 for ; Tue, 3 Jun 2014 18:22:55 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s53IMqMf043503 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 3 Jun 2014 11:22:53 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s53IMqvQ043502; Tue, 3 Jun 2014 11:22:52 -0700 (PDT) (envelope-from jmg) Date: Tue, 3 Jun 2014 11:22:52 -0700 From: John-Mark Gurney To: Warner Losh Subject: Re: Review needed Message-ID: <20140603182252.GF31367@funkthat.com> Mail-Followup-To: John-Mark Gurney , Warner Losh , freebsd-scsi@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 03 Jun 2014 11:22:53 -0700 (PDT) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2014 18:22:55 -0000 Warner Losh wrote this message on Mon, Jun 02, 2014 at 10:46 -0600: > The code that combines adjacent ranges for BIO_DELETEs to optimize > trims to the device assumes the list is sorted. Don't apply the > optimization of not sorting the queue when we have SSDs to the > delete_queue, since it causes more discard traffic to the drive. While I'm puzzled by this statement, if I remove the double negative, I get: Do apply the optimization of sorting the queue when we have SSDs to the delete_queue, since it (what the new optimizations?) causes more discard traffic to the drive. Do you mean previously? > one could argue that the higher levels should coalesce the trims, > that's not done today, so some optimization at this level is needed. > > Diffs and a (hopefully) cool review tool: https://phabric.freebsd.org/D142 Please include a raw diff... A Phabric is only for FreeBSD committers, it takes the rest of the community out of the ability to review. Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 3 18:51:29 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3CDBEC28 for ; Tue, 3 Jun 2014 18:51:29 +0000 (UTC) Received: from mail-ig0-f170.google.com (mail-ig0-f170.google.com [209.85.213.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02B702A8D for ; Tue, 3 Jun 2014 18:51:28 +0000 (UTC) Received: by mail-ig0-f170.google.com with SMTP id h3so754088igd.3 for ; Tue, 03 Jun 2014 11:51:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=XaY4vfcTAul+m/Dr5I2LaeoIVtiPp3xnD5QjKrkYh1Q=; b=mj+u5spRMTOS8Y52IHDfbstwap7+ijkuLIzGsxiFAzN3y/jGrj/Pfc499wLnsbaalz g4r0ypsEBGM96eYnclwnKPFg7rbFn2xkz8A/7LYs29Gb3YekvQj+Q/SuYQOBHGL0aaGs KBD1SGe291xsA+C6kE/foCvihvrsrzcf85P1rZr6/NhbC+o6pSX4GGg+2Gx6rgInJV8h RioxqY1y5C17VOFvMg4yz8eru/bQxmFhyPRfr9jIcd8UB/Im/QanFItA7drXe51r83C2 zslbDSh4IgLU+bsmbAyY/UVJ+5X9RPoAHoz7FYNh3W9JK0yAUxxNXhspjbQA9257mJ1O Ks+w== X-Gm-Message-State: ALoCoQkS3eBZ86eclNEHJw/KsSULUi8D+00uH9fg4UVM4MvTFDgzWnQH2ftoFWDDJNE3PBJErOB4 X-Received: by 10.42.147.5 with SMTP id l5mr6032869icv.89.1401821024431; Tue, 03 Jun 2014 11:43:44 -0700 (PDT) Received: from [10.0.0.119] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id y7sm38769824igl.13.2014.06.03.11.43.43 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 03 Jun 2014 11:43:43 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_A8799E54-CDDB-4584-A0CB-D400B8AC3C11"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.2\)) Subject: Re: Review needed From: Warner Losh In-Reply-To: <20140603182252.GF31367@funkthat.com> Date: Tue, 3 Jun 2014 12:43:43 -0600 Message-Id: <8FE17C18-08AB-4324-A059-D277792CC630@bsdimp.com> References: <20140603182252.GF31367@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1878.2) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jun 2014 18:51:29 -0000 --Apple-Mail=_A8799E54-CDDB-4584-A0CB-D400B8AC3C11 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Jun 3, 2014, at 12:22 PM, John-Mark Gurney wrote: > Warner Losh wrote this message on Mon, Jun 02, 2014 at 10:46 -0600: >> The code that combines adjacent ranges for BIO_DELETEs to optimize >> trims to the device assumes the list is sorted. Don't apply the >> optimization of not sorting the queue when we have SSDs to the >> delete_queue, since it causes more discard traffic to the drive. = While >=20 > I'm puzzled by this statement, if I remove the double negative, I get: > Do apply the optimization of sorting the queue when we have SSDs to = the > delete_queue, since it (what the new optimizations?) causes more = discard > traffic to the drive. >=20 > Do you mean previously? Yes, it should reduce traffic because sorting allows ranges to be = collapsed. it =3D=3D the lack of sorting I=92ll tweak the commit message to be less opaque. >> one could argue that the higher levels should coalesce the trims, >> that's not done today, so some optimization at this level is needed. >>=20 >> Diffs and a (hopefully) cool review tool: = https://phabric.freebsd.org/D142 >=20 > Please include a raw diff... A Phabric is only for FreeBSD = committers, > it takes the rest of the community out of the ability to review. I thought people w/o an account could at least see the diff. Looks like = that=92s not the case (and somewhat lame). You can get the same thing from my hg patch queue: http://people.freebsd.org/~imp/patch-queue/bio_delete Warner --Apple-Mail=_A8799E54-CDDB-4584-A0CB-D400B8AC3C11 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTjhdfAAoJEGwc0Sh9sBEA94wQAKiVXPit6Sy85kaa7rlRqDyY 0W154CrfXXAAKhhROabcNTopEYwYYaIPlHlrYc3aqGa33rmfcu5uD8befWMA5FZg 9dtgWsKp5W3nAuGiCA3b+JskCkU2Ki1DU0PGY731jPMmfQ/oVeoMVB+Xbgd+cymX DfVSsjx167+9z+COR+efSag2kkR8c8tmb9AI8d/qhcGZGv1juD7Ci4gR1A7jxyb6 PQNlOBcyUNv5YKgSLFW5svpuwuqY6Yei4M7X5VzmDJH3xJ3/dKxYkX0jnk5+UPNX Or4vIYk3RnUudfqAkB/FDD/FeBj2qTNjvmn3L/hbVf4iADmDkGhHxJcrI9GEoMn+ +UEimmvs+GJ0JgZj7YsFdVAbNTJL4aXPp94pitz07dDkxIMrcoUWHAsLqPjcqg1V ucuO43tLdMPwXFAwW/g26dmPnjiVRNY2bdwcsXu/3aOD2lbw6vd22gdzY8OOvpA/ x3e8pYYcY8BUdYId1uH05jPdUqcKgX5YhElZZ5DZHYg9cZg1ywjHCyJVgL9uYeRg TtwuFjdQ/rJWy/Shw7QseP2v4849OTLDpMeVKc5GngoSm7BWpcwA5onBMv20YFDL kxc3sHLRmEtNJ5uA9tFEk9JkyDwHO4smhzKTZO5cfO85PNfb+C2t926zkZ75I7LH 6ws4/r4VBf8SfQiu8dXO =gTeQ -----END PGP SIGNATURE----- --Apple-Mail=_A8799E54-CDDB-4584-A0CB-D400B8AC3C11-- From owner-freebsd-scsi@FreeBSD.ORG Wed Jun 18 21:01:08 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A2513474 for ; Wed, 18 Jun 2014 21:01:08 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B54C2033 for ; Wed, 18 Jun 2014 21:01:08 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s5IL18bu013468 for ; Wed, 18 Jun 2014 22:01:08 +0100 (BST) (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 190489] [iscsi] Native ISCSI - UNIT_ATTENTION with KVM Date: Wed, 18 Jun 2014 21:01:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: trasz@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2014 21:01:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190489 Edward Tomasz Napierala changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |trasz@FreeBSD.org Assignee|freebsd-scsi@FreeBSD.org |trasz@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Wed Jun 18 21:05:21 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4C92758A for ; Wed, 18 Jun 2014 21:05:21 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34925206A for ; Wed, 18 Jun 2014 21:05:21 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s5IL5LT7093074 for ; Wed, 18 Jun 2014 22:05:21 +0100 (BST) (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 175670] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) Date: Wed, 18 Jun 2014 21:05:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: In Discussion X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2014 21:05:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=175670 Edward Tomasz Napierala changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |trasz@FreeBSD.org Summary|[iscsi] smartctl fails on |smartctl fails on SAS disk |SAS disk connected to an |connected to an Intel C600 |Intel C600 controller (isci |controller (isci driver) |driver) | --- Comment #2 from Edward Tomasz Napierala --- Remove "[iscsi]" prefix; it's in no way related to iSCSI - it's isci(4). -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Thu Jun 19 19:01:41 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C6C7AD25; Thu, 19 Jun 2014 19:01:41 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 4D3BA2FFF; Thu, 19 Jun 2014 19:01:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id E2DFF20407F; Thu, 19 Jun 2014 20:51:55 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8GH5KPauPC6I; Thu, 19 Jun 2014 20:51:55 +0200 (CEST) Received: from [10.7.0.6] (unknown [10.7.0.6]) by smtp.infotech.no (Postfix) with ESMTPA id EAA1E204236; Thu, 19 Jun 2014 20:51:54 +0200 (CEST) Message-ID: <53A33149.80801@interlog.com> Date: Thu, 19 Jun 2014 14:51:53 -0400 From: Douglas Gilbert Reply-To: dgilbert@interlog.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: [Bug 175670] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Christian Franke X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2014 19:01:41 -0000 On 14-06-18 05:05 PM, bugzilla-noreply@freebsd.org wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=175670 > > Edward Tomasz Napierala changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > CC| |trasz@FreeBSD.org > Summary|[iscsi] smartctl fails on |smartctl fails on SAS disk > |SAS disk connected to an |connected to an Intel C600 > |Intel C600 controller (isci |controller (isci driver) > |driver) | > > --- Comment #2 from Edward Tomasz Napierala --- > Remove "[iscsi]" prefix; it's in no way related to iSCSI - it's isci(4). This is an old report that I think should be retired. Below is the output from the same model of drive with 0005 firmware (current, available from Seagate) rather than 0004 shown in the report. This is with the latest development version of smartmontools tested on 10.0-RELEASE-p4 and a LSI 9212-4i4e SAS controller. I'm the smartmontools guy to fix that and I can't replicate it so I'll treat it as closed. Doug Gilbert # smartctl -a /dev/da1 smartctl 6.3 2014-06-19 r3915 [FreeBSD 10.0-RELEASE-p4 i386] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST33000650SS Revision: 0005 Compliance: SPC-4 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Logical block size: 512 bytes Formatted with type 1 protection Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c50033fe58db Serial number: xxxxxxxxxxxxxx Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Thu Jun 19 14:47:10 2014 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Current Drive Temperature: 32 C Drive Trip Temperature: 68 C Manufactured in week 18 of year 2011 Specified cycle count over device lifetime: 10000 Accumulated start-stop cycles: 239 Specified load-unload count over device lifetime: 300000 Accumulated load-unload cycles: 240 Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 16089640 Blocks received from initiator = 627264253 Blocks read from cache and sent to initiator = 385140 Number of read and write commands whose size <= segment size = 19988 Number of read and write commands whose size > segment size = 1483 Vendor (Seagate/Hitachi) factory information number of hours powered up = 236.02 number of minutes until next internal SMART test = 46 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 29236610 0 0 29236610 0 8.366 0 write: 0 0 0 0 0 326.215 0 verify: 10 0 0 10 0 0.000 0 Non-medium error count: 147 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Completed - 110 - [- - -] # 2 Background short Completed - 6 - [- - -] Long (extended) Self Test duration: 27600 seconds [460.0 minutes] From owner-freebsd-scsi@FreeBSD.ORG Fri Jun 20 15:14:16 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55DF0A7E for ; Fri, 20 Jun 2014 15:14:16 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C2722B65 for ; Fri, 20 Jun 2014 15:14:16 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s5KFEGSK074315 for ; Fri, 20 Jun 2014 16:14:16 +0100 (BST) (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 175670] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) Date: Fri, 20 Jun 2014 15:14:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: Issue Resolved X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2014 15:14:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=175670 Sean Bruno changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Issue Resolved CC| |sbruno@FreeBSD.org Resolution|--- |Unable to Reproduce --- Comment #3 from Sean Bruno --- Recieved on freebsd-scsi list from Douglas Gilbert: This is an old report that I think should be retired. Below is the output from the same model of drive with 0005 firmware (current, available from Seagate) rather than 0004 shown in the report. This is with the latest development version of smartmontools tested on 10.0-RELEASE-p4 and a LSI 9212-4i4e SAS controller. I'm the smartmontools guy to fix that and I can't replicate it so I'll treat it as closed. Doug Gilbert # smartctl -a /dev/da1 smartctl 6.3 2014-06-19 r3915 [FreeBSD 10.0-RELEASE-p4 i386] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST33000650SS Revision: 0005 Compliance: SPC-4 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Logical block size: 512 bytes Formatted with type 1 protection Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c50033fe58db Serial number: xxxxxxxxxxxxxx Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Thu Jun 19 14:47:10 2014 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Current Drive Temperature: 32 C Drive Trip Temperature: 68 C Manufactured in week 18 of year 2011 Specified cycle count over device lifetime: 10000 Accumulated start-stop cycles: 239 Specified load-unload count over device lifetime: 300000 Accumulated load-unload cycles: 240 Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 16089640 Blocks received from initiator = 627264253 Blocks read from cache and sent to initiator = 385140 Number of read and write commands whose size <= segment size = 19988 Number of read and write commands whose size > segment size = 1483 Vendor (Seagate/Hitachi) factory information number of hours powered up = 236.02 number of minutes until next internal SMART test = 46 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 29236610 0 0 29236610 0 8.366 0 write: 0 0 0 0 0 326.215 0 verify: 10 0 0 10 0 0.000 0 Non-medium error count: 147 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Completed - 110 - [- - -] # 2 Background short Completed - 6 - [- - -] Long (extended) Self Test duration: 27600 seconds [460.0 minutes] -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Fri Jun 20 15:14:45 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0F156ABA for ; Fri, 20 Jun 2014 15:14:45 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E17662B73 for ; Fri, 20 Jun 2014 15:14:44 +0000 (UTC) Received: from [192.168.200.204] (c-50-131-5-126.hsd1.ca.comcast.net [50.131.5.126]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id D32821936DE; Fri, 20 Jun 2014 15:14:42 +0000 (UTC) Subject: Re: [Bug 175670] smartctl fails on SAS disk connected to an Intel C600 controller (isci driver) From: Sean Bruno Reply-To: sbruno@freebsd.org To: dgilbert@interlog.com In-Reply-To: <53A33149.80801@interlog.com> References: <53A33149.80801@interlog.com> Content-Type: text/plain; charset="us-ascii" Date: Fri, 20 Jun 2014 08:14:40 -0700 Message-ID: <1403277280.43097.27.camel@bruno> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-scsi@freebsd.org, Christian Franke X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2014 15:14:45 -0000 On Thu, 2014-06-19 at 14:51 -0400, Douglas Gilbert wrote: > > === START OF INFORMATION SECTION === I've closed/updated the ticket this morning. Thank you for the report. sean From owner-freebsd-scsi@FreeBSD.ORG Fri Jun 20 15:15:17 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6E4FFAF1 for ; Fri, 20 Jun 2014 15:15:17 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 551332B7A for ; Fri, 20 Jun 2014 15:15:17 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s5KFFH1v085113 for ; Fri, 20 Jun 2014 16:15:17 +0100 (BST) (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 184505] [ahd] AHD controller/Bad Negotiation/3 out of 4 boots Date: Fri, 20 Jun 2014 15:15:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-BETA4 X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: Issue Resolved X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2014 15:15:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=184505 Sean Bruno changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Issue Resolved CC| |sbruno@FreeBSD.org Resolution|--- |Unable to Reproduce -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 24 12:57:33 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 475A0AFC for ; Tue, 24 Jun 2014 12:57:33 +0000 (UTC) Received: from exprod7og124.obsmtp.com (exprod7og124.obsmtp.com [64.18.2.26]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9FE6125CE for ; Tue, 24 Jun 2014 12:57:32 +0000 (UTC) Received: from mail-qg0-f42.google.com ([209.85.192.42]) (using TLSv1) by exprod7ob124.postini.com ([64.18.6.12]) with SMTP ID DSNKU6l1tsADOL5mtFzbRuDy0XMdOqBd7T+S@postini.com; Tue, 24 Jun 2014 05:57:32 PDT Received: by mail-qg0-f42.google.com with SMTP id e89so209204qgf.29 for ; Tue, 24 Jun 2014 05:57:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type; bh=JHigsGL/6haTcLZDxbiUhLZfgA/4ifVpzYHQthsXHhk=; b=L1io2gfn/GGRb8I1wjbq3ihltvNpScgfLWFsw8KOx/fngbcOXVdIXJLoHU1Ij73wYP Ly2iEBbqQrLBfngAvw+RAuZgUfz10Nn2e3wAE+bGXjF69GNjaQH817oIMICnuf1UIWUe Q4QAtSqPZVXt6rCkarbD+DgKix+XSnjbcyIpuAWlOdZ97IJKg1RqNWXVFIxFVMsCHdoN HOwWa+TAS88M3iJSoa0krs0QR6YaTc9xWzQ48gFrcwTsnMVERlBGwkx0zsQoRrmdsn3r hqhpF2Wg2CuS2DngejLeKMKuhXj6w9ebT5i3xl8euo3a4WMzeaL728F0YLwIZfiLYcop A7iQ== X-Gm-Message-State: ALoCoQnZkKAI/ZZDA7W2N0V+2D5ld5NFsulEZii6CvwfzVYfEq4XRveBpK/M1kGOEI1LF2fp4e2+9x36gyNGV1oexpDB3ylKOB2ud8IDqFBWDkX9EdagRISQRrqubmHpPqQZjVfmjzHEGYG4dRbiiDAPiFznPSo15Q== X-Received: by 10.140.32.197 with SMTP id h63mr1443471qgh.10.1403614646664; Tue, 24 Jun 2014 05:57:26 -0700 (PDT) X-Received: by 10.140.32.197 with SMTP id h63mr1443459qgh.10.1403614646566; Tue, 24 Jun 2014 05:57:26 -0700 (PDT) From: Sumit Saxena References: 3dc7e373251cfad0f571398c97957ca0@mail.gmail.com In-Reply-To: 3dc7e373251cfad0f571398c97957ca0@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac+Pql4EcXXHgdv0Rtmpb55Hi2B5LQAARg3g Date: Tue, 24 Jun 2014 18:27:24 +0530 Message-ID: <559aba5a124ee1e32ddffb1380399e28@mail.gmail.com> Subject: Kernel panic: message secondary GPT header is not in the last LBA To: freebsd-scsi@freebsd.org Content-Type: multipart/mixed; boundary=001a113a5bd06932ac04fc9480bb X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: Kashyap Desai , sumit.saxena@avagotech.com X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2014 12:57:33 -0000 --001a113a5bd06932ac04fc9480bb Content-Type: text/plain; charset=UTF-8 Hi All, While doing some testing on driver, I am facing kernel panic inside GEOM module. I am using FreeBSD10.0 64bit, installed on Virtual drive connected behind LSI MegaRAID SAS 9361 controller and two Enclosures- Dell MD1220 with total 39 drives are connected to the controller. As I convert unconfigured drives(connected to Enclosures) to JBOD(plain drive without any RAID configuration exposed to OS), kernel panic is observed inside GEOM module with below traces- =================================================== ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afebe51 ses1: pass30,da26: Element descriptor: 'SLOT 20 ' ses1: pass30,da26: SAS Device Slot Element: 1 Phys at Slot 20, Not All Phys ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 50080e5223c0f03f addr 5000c5004cf152f1 ses1: pass37,da33: Element descriptor: 'SLOT 21 ' ses1: pass37,da33: SAS Device Slot Element: 1 Phys at Slot 21, Not All Phys ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afd3659 ses1: pass21,da17: Element descriptor: 'SLOT 22 ' ses1: pass21,da17: SAS Device Slot Element: 1 Phys at Slot 22, Not All Phys ses1: phy 0: SAS device type 1 id 0 ses1: phy 0: protocols: Initiator( None ) Target( SSP ) ses1: phy 0: parent 50080e5223c0f03f addr 5000cca00baf22c1 GEOM: da12: the secondary GPT header is not in the last LBA. Fatal trap 18: integer divide fault while in kernel mode cpuid = 4; apic id = 10 instruction pointer = 0x20:0xffffffff80805045 stack pointer = 0x28:0xfffffe0c23ded9e0 frame pointer = 0x28:0xfffffe0c23deda30 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 13 (g_event) trap number = 18 panic: integer divide fault cpuid = 4 KDB: stack backtrace: #0 0xffffffff808cb220 at kdb_backtrace+0x60 #1 0xffffffff80892d05 at panic+0x155 #2 0xffffffff80c71ae2 at trap_fatal+0x3a2 #3 0xffffffff80c7171f at trap+0x7bf #4 0xffffffff80c587e2 at calltrap+0x8 #5 0xffffffff80803574 at g_label_taste+0x3a4 #6 0xffffffff80802106 at g_new_provider_event+0xb6 #7 0xffffffff807fe1d6 at g_run_events+0x166 #8 0xffffffff80864dda at fork_exit+0x9a #9 0xffffffff80c58d1e at fork_trampoline+0xe Uptime: 4m48s kernel trap 12 with interrupts disabled ===================================== I have attached complete dmesg logs. Has anyone faced similar issue with GEOM framework ? Thanks in advance, Sumit Saxena --001a113a5bd06932ac04fc9480bb Content-Type: application/octet-stream; name="kernel_panic_geom.rar" Content-Disposition: attachment; filename="kernel_panic_geom.rar" Content-Transfer-Encoding: base64 X-Attachment-Id: 55ab24e237505370_0.1 UmFyIRoHAM+QcwAADQAAAAAAAACeznSAkDoAiA0AANbYCgACMNnAk/iEzUQdMxUAIAAAAGtlcm5l bF9wYW5pY19nZW9tLmRhdADwiKJcEd2RERCRD8VBV7tTNvg9Jt6EY2m5k8NcIlms1lhhhoGgxZi3 VqZMNXEFNll2K+DNw8mXj68WXYm/RM0WVwxNXdYgwwMLWJawNVA1Vc1Du8vwy8PDxMTF34j8/LDw 7uzxFXMXHzzMxEfEDVU1f/+HLwWJZ/TTdazb61Fmxv8eT0/D5MnB1rHh7/f8Vju9/tdv+NjqcXDw 8nzWO/6jycno+61+v+7zcXHe45M4+/u+Gxx8l7i5MWGxh5cWbuGx2+/Y8WTzYuHl5P7//Ajt8GcP FwXt+xg4fN5r3BnZyRA+ZiC2L2ONlpAuSaCMWbcscfz57Yb13M9Uc45Yv8vHm/NuCPzzUeX/JizV LRb0zxPuTWH9EH+X1fZ/Dt+Le+yx3uv2Jo2nh3c1SyTfvsfVk++PKexk4tDmeu5gwYuPQZvbu92/ 3aq8OL7snq9joIW9pFyROH8eXzX4md2JNyTwXd3QH3d3udACWzJ88npNEH0aAeTi1CcePOfj2WdG iu3Nmqx1JrnN27d25RYsyzaevRqcrRjk4c6s5ZrNn67Fqfe+jxZ4W7lybd+asdufmpenlhdchTxd /e+vt9fchHcn0nS9rw+Hr9KSu5DlPN34pZp9+rw9q10AMvfGz1Q2o8sy1s3LN2e3nVJTkntUW4cS y/nNYuwuztilm92dztbni7OnPxXcsWtGOhLesRN971LbUErIvspda/n3tTN3SjEjc3NRoQ8X2Wdg V7pOddpRFd1HO2a1b000XLlOfOXoTqSGaeiF8n6dtFZE9lOf6RTTchHb2NprAvdpiT9I29NDPVFN 00OiSFfq3tCXwyZTVc97sUXOxoM2Wvu0sm/Ja7TlpSeGe5C+enB8O16enpk8Pa7P88tqcE8N2iGa iCUQ/5UQ693w/xptTgohu2YZ7ML7MOizNzwtbNOvZhz8XXl/pu/XLTX1HT93u12JdgVlwWYbtqGb RzTfa/+FtFqHVvdzes7LWnBahtZs44wBPPqp1O3er5+Lh0dRav1edYnZ38XmxcHJE6NX2lrzxIuY /8N7uxlHpLvqHvxve3N7Vpzo9vb/DybCXE71jweX0cda4a/vz41YR+Nzf38vrYVdCvqmiXnI/Qqf k9HnxRHMmGte/sg0ed6XMHDvxSTt8GTkyXone6kSxwZ4fNY8Wtmnq29veCx83tDWbK0ijHiuSYrM 00+CTHJPj0vYcPFX9smCPtPfmu0Zru+umXq1PE80K3PeqslkhL2xunW0E/HqrcF7mL1qe5PmGpJa IWo6vPey04ZoS9sbp15oed9Uu0W7mpi9RdtVL+DkUvV1zPnvFWTwn+MffDKvzw+gnxXJr+HNOKlb cLXOrz38vvFWUQn7Q38Mq2j5AhdS3sepXBg9nqPWocUVepoY7VM/aG6de18gKtontY7c0Jzwtb6u vF61YbcJ/jH1Ovbh57M0up27fw2sc1STSwtrkCLdOO5CXtjdOvc+QLxaW16a5du2aVuVW1wfKKcf 45XP2hurXu/IcLrFqcsy0rrcZl96qaaSE9gn8Kp/NJD6GjBj1MTY6VifPO4dm/q8+AV94qaWEvbG /hlWy/Il4NPas+zPfT1WlhW7DHNCeXnTrzVfRX72PNWDnxtFv86jw9egC24U+vt+Lc3eYsfX9m7L Jdkos7ukq1VGrOWaT4apeW4ctnrud7tSf52u52NgN7vGLxtKJrs1Fumty3pJu+ntfmUAWzGnkS08 fn1qHr08m5+NOp445FPH8J7MGParVHg3f5b3h7252dlr7tMrn6Sj4/OyHyzLTfdqnmyin93LPYgx d9y2vwd3/Se7/Kbs7AanB0ZEXJZV8Fu3Tntzwr2uz3+9mnnV9yeXF0ZMjVl6LHa8Gc2YL2GJqZNG XBHq5ycGsD373H6n3evufP+A5m/7cmH6Ox297ufu2Mx/6SZ1NZj9GUfS+lnm60nzRypXs7nY6kvq Pz2N3sdeutQjTV66Cvds6n57GKavij/6Nfv7ojbu53q8q35OWKia2Ecxr4LHZ4uLh4vdC2sK2C/f 7nSzkkw2LemC5pmtVeD/l4e34uz65y3oeoxaaYxxyK9T+S6/en5trYckkKaUaJ1r003xer/csX9L FuqPjl4a3Yx6XcGOnzmHTJgsW70POSwrfo+Om2X63jcMuzrNq8pcrN3FZ+JFk/kQdNZm1rDufGHW bXJrz/21sM5vxzhkkPySFvWnC65w93YkHe8N2OcO/uuZNMnzzWv354fREBJauUSxqZc96/fkmwy8 xJjmszamI/hZjHow6C72Dy4o0onyeP7sfH48MZ9vqaUum5crb88v+vO1rhx6Bc2/Towm9JGXy6uc eLg8nJ5fp0H6XPm/258mzxVqMsfTYjWivd5eV9EZbrLHk4eToDdNf9fLk0GaVdcu7k0KWMceLj9H HyYvNoH1pZ+Tl4kcEVwZMFMX+LFv/d5+b6/LpgjKMSffjr9Yq8EHo1taE7muUo1e2P9+XF6WMpDB 5+XJhics0I7lbiUWPYPt0DR9ozCWkLrZm/ZJsI8fVyS5gv6sq+whvtw3/H+AT+3OO1FUn7JfZD92 bDJrXHrN6ILLZjRsv2TewDVRoTooqrzn7WD46IUnhLVXgnmptx4oVtTQpRCdumlPrlx1tpiEW7lV LMOV+XnJmThp4xU1tQXQXju5Yqr5PHTHj463RuXfxePl1nNmmmlv8YiM0pT+ZyeOOJb1OaH/RWp7 DClyErVGHDehXHw8X2+PF9+TkiE3by7BcFm5hlxQ/Yrz83n4d/Jzntnb9nnj2oV1rE/l1JeaW7Gf F+3FxUxv1z6/nrygH9dDHljOO8kXZdKnJx1zr16/qGiaVf++rQve/ALrxa3k0VY72qbk6Hj8Z1jo J3m1FYvWc1/x0o+fJgsa+CSLMucV+7JxcnLEorxg0Xxf1Rs/vk2WmCJC/VH6uPSPoSzp44v6cVas YYmJ1efn6t6PzxGo419nro749LHLg1Pxe11Za0D1lE0nW9ZUJbeCzbuYP1/3c732wJ2YPrWIuQti kwTT4YwIX6zWis9iy6tqaFK0BHBDeSuhIrQm/e9UrSEYAmTzZOTYzfq89Byrclzz/VWE9jwd31bq 60asRo8+/qQ8lfzoNxxfWjYeSOGDpjOytzOKt4qZzTv3vJWr6Iny6L84u334kSJtxMbl4tdB/dAf rL0Z3TqVu6Rwwetp7g14fzXYxfY19QPXa36zR/kF0E/9gdCtXbsG5cNqDmeFyk/t9CDypRig7Zsw P0n4JzN2lymL9J9UlEsHaNvrsSo/BMWO/gg3ERipEbHF8PDyxcq2Ccc0RXz718QewyX7kW98XH61 jsViOPqqpfh3e6dJ6Dv+zzvb5/EN2zcvdLxHMRGGk/t9e6eu4dzHTHQTXoH8p+F9m3dxQbzwvSbf OhbfQg4X25bUHKo2+uxKj8FzvlpcyxPSfQvtSXYOWpcp+k/BbNyiDlUWqT6F9GPBByzJUfpPwWi3 bg5VF2k+hfPix9LdBPjyn6j8FntXoOZYwUn0L5sU0HKYqP1H4LNZwwcqjHSfQvl9NS5VGU/Ufgst mSDlUbe97mVH4JzNylymNvrsSo/BdOM9LmeG3ve5lR+F9GO/0rtlnLHM1H8p/2HET9K7HXiI2973 M2+hBwTRripcxX9vrsSk/BcU12DmeG3ukklSR8xKUt5Yyn1S4ZaXKqUiaj8NDBdog5VG1F2J8Jpt PBbkxw7aL+KaHZRgknh/c+MNJIvdMqfeeFJ9l3qT7Kt2/0zTaVR9l3qT6pblLlNKT6rdkpPqpUuU 0pPqtzPCo+yyog9t9dibfQg4L0/TacsD9J/KtuD2U/Ufy71ED9J/Kt6DtrKfqP5d63A/SfypRhg9 lP1H8u9t75ZoqPwTTh0vVqueG312Jt9CDgnT9Np39vfLNFR/2HDp5+mabS2+uxMp+C9P02nt75Zo qPwTFNag5nht9diVH4L0/Tae3vlmio/BMMvS9Wq4ZtvrsSo/Ben6bT2n8s0UtWoduD3G203RY0nw AAAAAAAAGMAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkAAAAAAAAAAAACHydvdtpgAAAAAAA AAAAAAAAAAABjAAAAAAAAAAAAAMYAxAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGAAAAAAAAAAAAAAA AAAAAMZA5t9CD2922mDGAAAAAAAAAAAAAYwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAYwAAAAAAMQAAQO7fOhbe7bTAAAAAAAAAAABiAAAAAAAAAAAAAAxgwAAAA AAAAAAAAAAAAAAAAAAABD/L/qev1+21a+r/j7v+dvltfEEd//+qRbfNG2+ya0AAAbfQg9vf0dVCB rb56bb3d0AAAAAAASAAAAAAAGU/t752SBSfh7Db3+k5wAAAAAAAAAAAkAEPCbfOg2922mAAAAAAA AAAAAAAAAAAAAAAAAAAAAABD523zoW30IOF+3u20wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQ+H71w 7b3baYAAAAAAAAABjAAAAAAAAAAAAAAAAAAABD2O30IOGDb3baYAAAAAAAAAAAAkAAAAAAAAAAAA AAAAAEPa/9rEPXsAQAcA --001a113a5bd06932ac04fc9480bb-- From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 24 13:08:47 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 951B0A4 for ; Tue, 24 Jun 2014 13:08:47 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7270E26FB for ; Tue, 24 Jun 2014 13:08:47 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s5OD8exv020019 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 24 Jun 2014 06:08:40 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s5OD8eCZ020018; Tue, 24 Jun 2014 06:08:40 -0700 (PDT) (envelope-from jmg) Date: Tue, 24 Jun 2014 06:08:40 -0700 From: John-Mark Gurney To: Sumit Saxena Subject: Re: Kernel panic: message secondary GPT header is not in the last LBA Message-ID: <20140624130840.GK1560@funkthat.com> Mail-Followup-To: John-Mark Gurney , Sumit Saxena , freebsd-scsi@freebsd.org, Kashyap Desai References: <559aba5a124ee1e32ddffb1380399e28@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <559aba5a124ee1e32ddffb1380399e28@mail.gmail.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 24 Jun 2014 06:08:40 -0700 (PDT) Cc: freebsd-scsi@freebsd.org, Kashyap Desai X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2014 13:08:47 -0000 Sumit Saxena wrote this message on Tue, Jun 24, 2014 at 18:27 +0530: > Hi All, > > > > While doing some testing on driver, I am facing kernel panic > inside GEOM module. I am using FreeBSD10.0 64bit, installed on Virtual > drive connected behind LSI MegaRAID SAS 9361 controller and two Enclosures- > Dell MD1220 with total 39 drives are connected to the controller. As I > convert unconfigured drives(connected to Enclosures) to JBOD(plain drive > without any RAID configuration exposed to OS), kernel panic is observed > inside GEOM module with below traces- > > > > =================================================== > > ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > > ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afebe51 > > ses1: pass30,da26: Element descriptor: 'SLOT 20 ' > > ses1: pass30,da26: SAS Device Slot Element: 1 Phys at Slot 20, Not All Phys > > ses1: phy 0: SAS device type 1 id 0 > > ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > > ses1: phy 0: parent 50080e5223c0f03f addr 5000c5004cf152f1 > > ses1: pass37,da33: Element descriptor: 'SLOT 21 ' > > ses1: pass37,da33: SAS Device Slot Element: 1 Phys at Slot 21, Not All Phys > > ses1: phy 0: SAS device type 1 id 0 > > ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > > ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afd3659 > > ses1: pass21,da17: Element descriptor: 'SLOT 22 ' > > ses1: pass21,da17: SAS Device Slot Element: 1 Phys at Slot 22, Not All Phys > > ses1: phy 0: SAS device type 1 id 0 > > ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > > ses1: phy 0: parent 50080e5223c0f03f addr 5000cca00baf22c1 > > GEOM: da12: the secondary GPT header is not in the last LBA. > > > > > > Fatal trap 18: integer divide fault while in kernel mode > > cpuid = 4; apic id = 10 > > instruction pointer = 0x20:0xffffffff80805045 > > stack pointer = 0x28:0xfffffe0c23ded9e0 > > frame pointer = 0x28:0xfffffe0c23deda30 > > code segment = base 0x0, limit 0xfffff, type 0x1b > > = DPL 0, pres 1, long 1, > def32 0, gran 1 > > processor eflags = interrupt enabled, resume, IOPL = 0 > > current process = 13 (g_event) > > trap number = 18 > > panic: integer divide fault > > cpuid = 4 > > KDB: stack backtrace: > > #0 0xffffffff808cb220 at kdb_backtrace+0x60 > > #1 0xffffffff80892d05 at panic+0x155 > > #2 0xffffffff80c71ae2 at trap_fatal+0x3a2 > > #3 0xffffffff80c7171f at trap+0x7bf > > #4 0xffffffff80c587e2 at calltrap+0x8 > > #5 0xffffffff80803574 at g_label_taste+0x3a4 > > #6 0xffffffff80802106 at g_new_provider_event+0xb6 > > #7 0xffffffff807fe1d6 at g_run_events+0x166 > > #8 0xffffffff80864dda at fork_exit+0x9a > > #9 0xffffffff80c58d1e at fork_trampoline+0xe > > Uptime: 4m48s > > kernel trap 12 with interrupts disabled Can you run w/ DDB enabled and get a dump? an integer divide makes me think of a divide by zero error... Are you sure things like sector size are set properly? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-scsi@FreeBSD.ORG Fri Jul 4 13:53:36 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 822C292D for ; Fri, 4 Jul 2014 13:53:36 +0000 (UTC) Received: from csmtp8.one.com (csmtp8.one.com [195.47.247.108]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0137F2CD1 for ; Fri, 4 Jul 2014 13:53:35 +0000 (UTC) Received: from webmail3 (webmail3.local.one.com [10.246.6.3]) by csmtp8.one.com (Postfix) with SMTP id 3B45B40000AB8 for ; Fri, 4 Jul 2014 13:47:09 +0000 (UTC) X-Originating-IP: 5.35.185.180 User-Agent: One.com webmail 6.3.0 MIME-Version: 1.0 Message-ID: <1404481628956.6459.61486@webmail3> Date: Fri, 04 Jul 2014 13:47:08 GMT To: From: "Christer Eriksson" Reply-To: Subject: Kernel panic: Page fault when loading kernel native iSCSI target (FreeBSD 10.0-STABLE #0 r268091) Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2014 13:53:36 -0000 Hello All, We are getting kernel panics while reading and writing to an iSCSI target. = It is the kernel implementation of iSCSI and we are running the initiators = in Windows 2012R2 with load on two 10 GE links. The problem is repeatable, = but occurs what appears to be within a random period from when the load is = initiated. No obvious useful info in dmesg or syslog. Backtrace from the kernel dump below. I will try to collect additional information upon request. Best Regards Christer Eriksson INFO ------------------------------------------------------------- Dump header from device /dev/ada1s1 Architecture: amd64 Architecture Version: 2 Dump Length: 2995580928B (2856 MB) Blocksize: 512 Dumptime: Fri Jul 4 14:50:34 2014 Hostname: TestArray1. Magic: FreeBSD Kernel Dump Version String: FreeBSD 10.0-STABLE #0 r268091: Tue Jul 1 15:40:42 CEST 201= 4 root@TestArray1.:/usr/obj/usr/src/sys/GENERIC Panic String: page fault Dump Parity: 455418556 Bounds: 3 Dump Status: good KGDB ------------------------------------------------------------- #kgdb /usr/obj/usr/src/sys/GENERIC/kernel.debug /var/crash/vmcore.3 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you ar= e welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid =3D 6; apic id =3D 06 fault virtual address=3D 0x0 fault code=3D supervisor write data, page not present instruction pointer=3D 0x20:0xffffffff80ce2766 stack pointer =3D 0x28:0xfffffe1049ba38f0 frame pointer =3D 0x28:0xfffffe1049ba3940 code segment=3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags=3D interrupt enabled, resume, IOPL =3D 0 current process=3D 0 (cfiscsirx) trap number=3D 12 panic: page fault cpuid =3D 6 KDB: stack backtrace: #0 0xffffffff8092a270 at kdb_backtrace+0x60 #1 0xffffffff808ef7c5 at panic+0x155 #2 0xffffffff80ce4a5f at trap_fatal+0x38f #3 0xffffffff80ce4d78 at trap_pfault+0x308 #4 0xffffffff80ce4430 at trap+0x4a0 #5 0xffffffff80ccae32 at calltrap+0x8 #6 0xffffffff81c304c0 at cfiscsi_handle_data_segment+0xf0 #7 0xffffffff81c30eda at cfiscsi_receive_callback+0x5ea #8 0xffffffff81c4f5bb at icl_receive_thread+0x11b #9 0xffffffff808c037a at fork_exit+0x9a #10 0xffffffff80ccb36e at fork_trampoline+0xe Uptime: 9m51s Dumping 2856 out of 65476 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/ums.ko.symbols...done. Loaded symbols for /boot/kernel/ums.ko.symbols Reading symbols from /boot/kernel/uhid.ko.symbols...done. Loaded symbols for /boot/kernel/uhid.ko.symbols Reading symbols from /boot/kernel/ctl.ko.symbols...done. Loaded symbols for /boot/kernel/ctl.ko.symbols Reading symbols from /boot/kernel/iscsi.ko.symbols...done. Loaded symbols for /boot/kernel/iscsi.ko.symbols Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. Loaded symbols for /boot/kernel/dtraceall.ko.symbols Reading symbols from /boot/kernel/cyclic.ko.symbols...done. Loaded symbols for /boot/kernel/cyclic.ko.symbols Reading symbols from /boot/kernel/dtrace.ko.symbols...done. Loaded symbols for /boot/kernel/dtrace.ko.symbols Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. Loaded symbols for /boot/kernel/dtmalloc.ko.symbols Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. Loaded symbols for /boot/kernel/dtnfscl.ko.symbols Reading symbols from /boot/kernel/fbt.ko.symbols...done. Loaded symbols for /boot/kernel/fbt.ko.symbols Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. Loaded symbols for /boot/kernel/fasttrap.ko.symbols Reading symbols from /boot/kernel/lockstat.ko.symbols...done. Loaded symbols for /boot/kernel/lockstat.ko.symbols Reading symbols from /boot/kernel/sdt.ko.symbols...done. Loaded symbols for /boot/kernel/sdt.ko.symbols Reading symbols from /boot/kernel/systrace.ko.symbols...done. Loaded symbols for /boot/kernel/systrace.ko.symbols Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols Reading symbols from /boot/kernel/profile.ko.symbols...done. Loaded symbols for /boot/kernel/profile.ko.symbols #0 doadump (textdump=3D) at pcpu.h:219 219pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump (textdump=3D) at pcpu.h:219 #1 0xffffffff808ef442 in kern_reboot (howto=3D260) at /usr/src/sys/kern/ker= n_shutdown.c:452 #2 0xffffffff808ef804 in panic (fmt=3D) at /usr/src/sy= s/kern/kern_shutdown.c:759 #3 0xffffffff80ce4a5f in trap_fatal (frame=3D, eva=3D<= value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:881 #4 0xffffffff80ce4d78 in trap_pfault (frame=3D0xfffffe1049ba3840, usermode= =3D) at /usr/src/sys/amd64/amd64/trap.c:692 #5 0xffffffff80ce4430 in trap (frame=3D0xfffffe1049ba3840) at /usr/src/sys/= amd64/amd64/trap.c:456 #6 0xffffffff80ccae32 in calltrap () at /usr/src/sys/amd64/amd64/exception.= S:232 #7 0xffffffff80ce2766 in bcopy () at /usr/src/sys/amd64/amd64/support.S:112= #8 0xffffffff8095be82 in m_copydata (m=3D, off=3D, len=3D, cp=3D) at /usr/src/sys/kern/uipc_mbuf.c:887 #9 0xffffffff81c304c0 in cfiscsi_handle_data_segment (request=3D0xfffff8024= 8460eb0, cdw=3D0xfffff80248484540) at /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:782 #10 0xffffffff81c30eda in cfiscsi_receive_callback (request=3D0xfffff802484= 60eb0) at /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:916 #11 0xffffffff81c4f5bb in icl_receive_thread (arg=3D0xfffff80248a16980) at = /usr/src/sys/modules/iscsi/../../dev/iscsi/icl.c:730 #12 0xffffffff808c037a in fork_exit (callout=3D0xffffffff81c4f4a0 , arg=3D0xfffff80248a16980, frame=3D0xfffffe1049ba3ac0) at /usr/src/sys/kern/kern_fork.c:995 #13 0xffffffff80ccb36e in fork_trampoline () at /usr/src/sys/amd64/amd64/ex= ception.S:606 #14 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) bt full #0 doadump (textdump=3D) at pcpu.h:219 No locals. #1 0xffffffff808ef442 in kern_reboot (howto=3D260) at /usr/src/sys/kern/ker= n_shutdown.c:452 No locals. #2 0xffffffff808ef804 in panic (fmt=3D) at /usr/src/sy= s/kern/kern_shutdown.c:759 ap =3D {{gp_offset =3D 16, fp_offset =3D 48, overflow_arg_area =3D 0xfffffe= 1049ba3530, reg_save_area =3D 0xfffffe1049ba34b0}} #3 0xffffffff80ce4a5f in trap_fatal (frame=3D, eva=3D<= value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:881 softseg =3D {ssd_base =3D 0, ssd_limit =3D 1048575, ssd_type =3D 27, ssd_dp= l =3D 0, ssd_p =3D 1, ssd_long =3D 1, ssd_def32 =3D 0, ssd_gran =3D 1} #4 0xffffffff80ce4d78 in trap_pfault (frame=3D0xfffffe1049ba3840, usermode= =3D) at /usr/src/sys/amd64/amd64/trap.c:692 rv =3D Cannot access memory at address 0x0 (kgdb) list *0xffffffff80ce2766 0xffffffff80ce2766 is at /usr/src/sys/amd64/amd64/support.S:113. 108cmpq%rcx,%rax/* overlapping && src < dst? */ 109jb1f 110 111shrq$3,%rcx/* copy by 64-bit words */ 112cld/* nope, copy forwards */ 113rep 114movsq 115movq%rdx,%rcx 116andq$7,%rcx/* any bytes left? */ 117rep (kgdb) up #1 0xffffffff808ef442 in kern_reboot (howto=3D260) at /usr/src/sys/kern/ker= n_shutdown.c:452 452doadump(TRUE); (kgdb) up #2 0xffffffff808ef804 in panic (fmt=3D) at /usr/src/sy= s/kern/kern_shutdown.c:759 759kern_reboot(bootopt); (kgdb) up #3 0xffffffff80ce4a5f in trap_fatal (frame=3D, eva=3D<= value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:881 881panic("%s", trap_msg[type]); (kgdb) up #4 0xffffffff80ce4d78 in trap_pfault (frame=3D0xfffffe1049ba3840, usermode= =3D) at /usr/src/sys/amd64/amd64/trap.c:692 692trap_fatal(frame, eva); (kgdb) up #5 0xffffffff80ce4430 in trap (frame=3D0xfffffe1049ba3840) at /usr/src/sys/= amd64/amd64/trap.c:456 456(void) trap_pfault(frame, FALSE); (kgdb) up #6 0xffffffff80ccae32 in calltrap () at /usr/src/sys/amd64/amd64/exception.= S:232 232calltrap Current language: auto; currently asm (kgdb) up #7 0xffffffff80ce2766 in bcopy () at /usr/src/sys/amd64/amd64/support.S:112= 112cld/* nope, copy forwards */ (kgdb) up #8 0xffffffff8095be82 in m_copydata (m=3D, off=3D, len=3D, cp=3D) at /usr/src/sys/kern/uipc_mbuf.c:887 887bcopy(mtod(m, caddr_t) + off, cp, count); Current language: auto; currently minimal (kgdb) up #9 0xffffffff81c304c0 in cfiscsi_handle_data_segment (request=3D0xfffff8024= 8460eb0, cdw=3D0xfffff80248484540) at /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:782 782icl_pdu_get_data(request, off, cdw->cdw_sg_addr, copy_len); (kgdb) up #10 0xffffffff81c30eda in cfiscsi_receive_callback (request=3D0xfffff802484= 60eb0) at /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:916 916done =3D cfiscsi_handle_data_segment(request, cdw); (kgdb) list 911 912io =3D cdw->cdw_ctl_io; 913KASSERT((io->io_hdr.flags & CTL_FLAG_DATA_MASK) !=3D CTL_FLAG_DATA_IN, 914 ("CTL_FLAG_DATA_IN")); 915 916done =3D cfiscsi_handle_data_segment(request, cdw); 917if (done) { 918CFISCSI_SESSION_LOCK(cs); 919TAILQ_REMOVE(&cs->cs_waiting_for_data_out, cdw, cdw_next); 920CFISCSI_SESSION_UNLOCK(cs); (kgdb) up #11 0xffffffff81c4f5bb in icl_receive_thread (arg=3D0xfffff80248a16980) at = /usr/src/sys/modules/iscsi/../../dev/iscsi/icl.c:730 730(ic->ic_receive)(response); (kgdb) list 725icl_pdu_free(response); 726icl_conn_fail(ic); 727return; 728} 729 730(ic->ic_receive)(response); 731} 732} 733 734static void (kgdb) up #12 0xffffffff808c037a in fork_exit (callout=3D0xffffffff81c4f4a0 , arg=3D0xfffff80248a16980, frame=3D0xfffffe1049ba3ac0) at /usr/src/sys/kern/kern_fork.c:995 995callout(arg, frame); (kgdb) list 990 * cpu_set_fork_handler intercepts this function call to 991 * have this call a non-return function to stay in kernel mode. 992 * initproc has its own fork handler, but it does return. 993 */ 994KASSERT(callout !=3D NULL, ("NULL callout in fork_exit")); 995callout(arg, frame); 996 997/* 998 * Check if a kernel thread misbehaved and returned from its main 999 * function. (kgdb) up #13 0xffffffff80ccb36e in fork_trampoline () at /usr/src/sys/amd64/amd64/ex= ception.S:606 606callfork_exit Current language: auto; currently asm (kgdb) list 601 602ENTRY(fork_trampoline) 603movq%r12,%rdi/* function */ 604movq%rbx,%rsi/* arg1 */ 605movq%rsp,%rdx/* trapframe pointer */ 606callfork_exit 607MEXITCOUNT 608jmpdoreti/* Handle any ASTs */ 609 610/* From owner-freebsd-scsi@FreeBSD.ORG Sun Jul 6 19:27:50 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A7405227 for ; Sun, 6 Jul 2014 19:27:50 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3D5FC2D86 for ; Sun, 6 Jul 2014 19:27:50 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id cc10so14831138wib.12 for ; Sun, 06 Jul 2014 12:27:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=+BgaJ5B9sMMPVJfVMGV8QCJ7Gvc1izU7dfOoZQU/Yv8=; b=KCmKSaUWpKdPMabbr5WLVm16AehMLhkL7Lbmpy9rTmgzhHoav4GgbcBLaZb/biyXIj K32d8zaayabdAWXew8c2qsdNyXl5FEo2+mSIMVCTYWeoOD4Jvrx5IfsPbRHMQkCnczSv EWaWae2px9ltB2pTukEEXcUZl/Tf6GEif3xJT6KGgb2LgBSB6B/M+Upajl9lr+fsQaDY sxvH4aF/Dat9W6BNsEmbOBFZDVfsc5L1vB+877WXJkoe5m/OJJ2qrWeEJx41wf5DVq21 OzFvvKjWQ6GJsw/kHMGKWbCLxkJqtQTuSBhEl/8YX3VUyXiZXfD8cgD8wiUVh5ojaErT hMow== X-Received: by 10.195.18.8 with SMTP id gi8mr26378685wjd.75.1404674868397; Sun, 06 Jul 2014 12:27:48 -0700 (PDT) Received: from brick.home (adbh103.neoplus.adsl.tpnet.pl. [79.184.7.103]) by mx.google.com with ESMTPSA id fb15sm106178217wid.23.2014.07.06.12.27.47 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 06 Jul 2014 12:27:47 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Sun, 6 Jul 2014 21:27:45 +0200 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= To: Christer Eriksson Subject: Re: Kernel panic: Page fault when loading kernel native iSCSI target (FreeBSD 10.0-STABLE #0 r268091) Message-ID: <20140706192745.GB3032@brick.home> Mail-Followup-To: Christer Eriksson , freebsd-scsi@freebsd.org References: <1404481628956.6459.61486@webmail3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1404481628956.6459.61486@webmail3> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jul 2014 19:27:50 -0000 On 0704T1347, Christer Eriksson wrote: > Hello All, > > We are getting kernel panics while reading and writing to an iSCSI target. It is the kernel implementation of iSCSI and we are running the initiators in Windows 2012R2 with load on two 10 GE links. The problem is repeatable, but occurs what appears to be within a random period from when the load is initiated. No obvious useful info in dmesg or syslog. Can you try to reproduce this with CURRENT, with all the default debugging flags? Otherwise, can you do it with 10.0 kernel built with debugging flags? The flags in question are: options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 8 01:45:58 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D0A146AC for ; Tue, 8 Jul 2014 01:45:58 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B7D5B29FA for ; Tue, 8 Jul 2014 01:45:58 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s681jwcl054958 for ; Tue, 8 Jul 2014 02:45:58 +0100 (BST) (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-scsi@FreeBSD.org Subject: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver Date: Tue, 08 Jul 2014 01:45:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-scsi@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 01:45:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-scsi@FreeBSD.org Summary|smartctl -H gives "ATA |[iscsi] smartctl -H gives |output registers missing" |"ATA output registers |for a disk using the isci |missing" for a disk using |driver |the isci driver --- Comment #1 from Mark Linimon --- Over to maintainers. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 8 03:21:48 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 98A3F65B for ; Tue, 8 Jul 2014 03:21:48 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 530D322B9 for ; Tue, 8 Jul 2014 03:21:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 867A82041CE; Tue, 8 Jul 2014 05:12:33 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T75nsy3zGvGr; Tue, 8 Jul 2014 05:12:31 +0200 (CEST) Received: from [192.168.48.86] (unknown [199.127.109.253]) by smtp.infotech.no (Postfix) with ESMTPA id E0792204171; Tue, 8 Jul 2014 05:12:30 +0200 (CEST) Message-ID: <53BB619B.4060908@interlog.com> Date: Mon, 07 Jul 2014 23:12:27 -0400 From: Douglas Gilbert Reply-To: dgilbert@interlog.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: samm2@users.sourceforge.net, Christian Franke X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 03:21:48 -0000 On 14-07-07 09:45 PM, bugzilla-noreply@freebsd.org wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 > > Mark Linimon changed: > > What |Removed |Added > ---------------------------------------------------------------------------- > Assignee|freebsd-bugs@FreeBSD.org |freebsd-scsi@FreeBSD.org > Summary|smartctl -H gives "ATA |[iscsi] smartctl -H gives > |output registers missing" |"ATA output registers > |for a disk using the isci |missing" for a disk using > |driver |the isci driver > > --- Comment #1 from Mark Linimon --- > Over to maintainers. > At the point of failure "whatever" produces this SCSI sense data: f0 00 01 00 50 40 00 00 00 c2 4f 00 00 1d 00 00 00 00 FreeBSD is wrong to print out 18 bytes because that is an 8 byte (deferred, fixed type) buffer because byte 7 (the additional length) is 0. Whatever produced that broken sense data is the probably culprit. It is trying to say there is "ATA pass-through information available" but fails to get its message across. Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 8 05:42:30 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A3BC8B24 for ; Tue, 8 Jul 2014 05:42:30 +0000 (UTC) Received: from mailout06.t-online.de (mailout06.t-online.de [194.25.134.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailout00.t-online.de", Issuer "TeleSec ServerPass DE-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 61D322D29 for ; Tue, 8 Jul 2014 05:42:30 +0000 (UTC) Received: from fwd29.aul.t-online.de (fwd29.aul.t-online.de [172.20.26.134]) by mailout06.t-online.de (Postfix) with SMTP id E4C2A169C69; Tue, 8 Jul 2014 07:37:07 +0200 (CEST) Received: from [192.168.2.108] (XKH930ZcQhlg6mHRKJoMgOOCIZBICKUIlaiux4V+DK759KJy7LRKZVPZJWlfJp1ZsC@[84.180.73.200]) by fwd29.t-online.de with esmtp id 1X4O5T-1cXZgG0; Tue, 8 Jul 2014 07:37:07 +0200 Message-ID: <53BB8381.2000109@t-online.de> Date: Tue, 08 Jul 2014 07:37:05 +0200 From: Christian Franke User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26.1 MIME-Version: 1.0 To: dgilbert@interlog.com Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver References: <53BB619B.4060908@interlog.com> In-Reply-To: <53BB619B.4060908@interlog.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-ID: XKH930ZcQhlg6mHRKJoMgOOCIZBICKUIlaiux4V+DK759KJy7LRKZVPZJWlfJp1ZsC X-TOI-MSGID: 105d288a-868d-4d68-a242-5385d655e1db Cc: freebsd-scsi@freebsd.org, samm2@users.sourceforge.net X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 05:42:30 -0000 Douglas Gilbert wrote: > On 14-07-07 09:45 PM, bugzilla-noreply@freebsd.org wrote: >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 >> >> Mark Linimon changed: >> >> What |Removed |Added >> ---------------------------------------------------------------------------- >> >> Assignee|freebsd-bugs@FreeBSD.org |freebsd-scsi@FreeBSD.org >> Summary|smartctl -H gives "ATA |[iscsi] smartctl -H >> gives >> |output registers missing" |"ATA output registers >> |for a disk using the isci |missing" for a disk >> using >> |driver |the isci driver >> >> --- Comment #1 from Mark Linimon --- >> Over to maintainers. >> > > At the point of failure "whatever" produces this SCSI sense data: > f0 00 01 00 50 40 00 00 00 c2 4f 00 00 1d 00 00 00 00 > Is this possibly fixed format sense data for SAT ATA PASS-THROUGH? If yes, this is a testcase for http://www.smartmontools.org/ticket/296 Christian From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 8 11:56:25 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4475546B for ; Tue, 8 Jul 2014 11:56:25 +0000 (UTC) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id BBF3A2A35 for ; Tue, 8 Jul 2014 11:56:21 +0000 (UTC) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.5/8.14.5) with ESMTP id s68BjxdP056750; Tue, 8 Jul 2014 12:45:59 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id s68BjxJM006457; Tue, 8 Jul 2014 12:45:59 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id s68Bjxn6006452; Tue, 8 Jul 2014 12:45:59 +0100 Date: Tue, 8 Jul 2014 12:45:59 +0100 Message-Id: <201407081145.s68Bjxn6006452@higson.cam.lispworks.com> From: Martin Simmons To: freebsd-scsi@FreeBSD.org In-reply-to: <53BB619B.4060908@interlog.com> ((Unparsable address -- Missing comma between addresses or badly-formatted address: "dgilbert at interlog.com_^_")) Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver References: <53BB619B.4060908@interlog.com> X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 11:56:25 -0000 >>>>> On Mon, 07 Jul 2014 23:12:27 -0400, Douglas Gilbert said: > > On 14-07-07 09:45 PM, bugzilla-noreply at freebsd.org wrote: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 > > > > Mark Linimon changed: > > > > What |Removed |Added > > ---------------------------------------------------------------------------- > > Assignee|freebsd-bugs at FreeBSD.org |freebsd-scsi at FreeBSD.org > > Summary|smartctl -H gives "ATA |[iscsi] smartctl -H gives > > |output registers missing" |"ATA output registers > > |for a disk using the isci |missing" for a disk using > > |driver |the isci driver > > > > --- Comment #1 from Mark Linimon --- > > Over to maintainers. > > > > At the point of failure "whatever" produces this SCSI sense data: > f0 00 01 00 50 40 00 00 00 c2 4f 00 00 1d 00 00 00 00 > > FreeBSD is wrong to print out 18 bytes because that is an 8 > byte (deferred, fixed type) buffer because byte 7 (the > additional length) is 0. Whatever produced that broken > sense data is the probably culprit. > > It is trying to say there is "ATA pass-through information > available" but fails to get its message across. Hi, I'm the original reporter. The sense data is smartctl's interpretation of the ccb union. In particular, the bytes are from ccb->csio.sense_data and it calculates 18 from ccb->csio.sense_len - ccb->csio.sense_resid (32 - 14). I don't know if that is correct or not. FWIW, here is the output from CentOS 6.3 on the same machine: REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK Input: FR=0xda, SC=...., LL=...., LM=0x4f, LH=0xc2, DEV=...., CMD=0xb0 [ata pass-through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00 ] scsi_status=0x2, host_status=0x0, driver_status=0x8 info=0x1 duration=17 milliseconds resid=0 >>> Sense buffer, len=22: 00 72 00 00 00 00 00 00 0e 09 0c 00 00 00 00 00 00 10 00 4f 00 c2 40 50 status=2: [desc] sense_key=0 asc=0 ascq=0 Values from ATA Return Descriptor are: 00 09 0c 00 00 00 00 00 00 00 4f 00 c2 40 50 [Duration: 0.016s] Output: ERR=0x00, SC=0x00, LL=0x00, LM=0x4f, LH=0xc2, DEV=0x40, STS=0x50 REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK returned 0 It appears to have the same ata pass-through command but completely different sense data. __Martin From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 8 12:08:13 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A9E2834 for ; Tue, 8 Jul 2014 12:08:13 +0000 (UTC) Received: from mail-ve0-x229.google.com (mail-ve0-x229.google.com [IPv6:2607:f8b0:400c:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E10BA2B44 for ; Tue, 8 Jul 2014 12:08:12 +0000 (UTC) Received: by mail-ve0-f169.google.com with SMTP id pa12so5636625veb.28 for ; Tue, 08 Jul 2014 05:08:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=tZh4Ex+kzMZkgYXDBbK37OFVOLQERuDw0xFvaC7YQJY=; b=EyGkznORUTJ+PxnfKvGfVAjBaxXi3/dEralQbtG1ciczE0rTp306UREYghHtGP6X5l E9qxWgwWMwvHd5T3HDX7NpUnzo1XR4uF6ysUr6/bPf8RGRcvoRK8ESaTdH5tH4QT50H0 uRu5qyYQ3ptVeNN/qGPjXVnpW6TFf36kpvC2vzGxpoD0+gSuyeYxOrPS/+yoANZlRNnc x2XXwaab/3VDM+tDI0u7sIzzzbRPl6+BKNbENIWlL9v1N9KCT4/EmK5YS8BUBD2BXXDH A3A7+YNlkse3NPEqW8v1ShdQqMXhOj3QqIEUJSXobF7UhR0Kojum/kt/ZUUdz2DVVmxa +drQ== X-Received: by 10.58.188.199 with SMTP id gc7mr33431313vec.4.1404821291877; Tue, 08 Jul 2014 05:08:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.156.71 with HTTP; Tue, 8 Jul 2014 05:07:51 -0700 (PDT) From: bharat singh Date: Tue, 8 Jul 2014 17:37:51 +0530 Message-ID: Subject: NPIV port login failures To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 12:08:13 -0000 Hello, I am trying to use virtual target ports for QLogic PCI to Fibre Channel Host Adapter for QLE2460 on freebsd9. Initiator hosted on a ESX 5.0 Increased no of chans to 4 and assigned unique WWPN/WWNN to the virtual ports. [root@fckern ~]# ctladm port -l Port Online Type Name pp vp WWNN WWPN Speed Vendor 0 YES IOCTL CTL ioctl 0 0 0 0 0 1 YES INTERNAL ctl2cam 0 0 0x500000009f287f00 0x500000009f287f02 0 2 YES INTERNAL CTL internal 0 0 0 0 0 3 YES FC isp0 0 0 0x2000001b328b413b 0x2100001b328b413b 4000000 Qlogic >>>> physical port 4 YES FC isp0 0 1 0x2000001b328b413c 0x2000001b328b413c 4000000 Qlogic >>>> virtual ports 5 YES FC isp0 0 2 0x2000001b328b413d 0x2000001b328b413d 4000000 Qlogic 6 YES FC isp0 0 3 0x2000001b328b413e 0x2000001b328b413e 4000000 Qlogic 7 YES FC isp0 0 4 0x2000001b328b413f 0x2000001b328b413f 4000000 Qlogic but initiator login to virtual ports is failing. [root@fckern ~]# ctladm wwpnlist No of logged in Initiators are 1: Initiator WWPN Target WWPN: 2100001b32060f71 2100001b328b413b >>>> only the physical port login is going through Jul 8 15:05:48 fckern kernel: isp0: Chan 1 PLOGX PortID 0xfffffc to N-Port handle 0x401: invalid parameter at offset 0xa Jul 8 15:05:48 fckern kernel: isp0: isp_fclink_test: Chan 1 cannot log into SNS Jul 8 15:05:48 fckern kernel: isp0: Chan 3 PLOGX PortID 0xfffffc to N-Port handle 0x403: invalid parameter at offset 0xa Jul 8 15:05:48 fckern kernel: isp0: isp_fclink_test: Chan 3 cannot log into SNS Jul 8 15:05:48 fckern kernel: isp0: Chan 4 PLOGX PortID 0xfffffc to N-Port handle 0x404: invalid parameter at offset 0xa Jul 8 15:05:48 fckern kernel: isp0: isp_fclink_test: Chan 4 cannot log into SNS Jul 8 15:05:48 fckern kernel: isp0: Chan 2 PLOGX PortID 0xfffffc to N-Port handle 0x402: invalid parameter at offset 0xa Jul 8 15:05:48 fckern kernel: isp0: isp_fclink_test: Chan 2 cannot log into SNS I have added the virtual ports to my switch's zone. ESX & switch have NPIV support enabled. Have anyone faced any similar issue ? -- Bharat Singh From owner-freebsd-scsi@FreeBSD.ORG Tue Jul 8 17:43:42 2014 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A4A2BFD2 for ; Tue, 8 Jul 2014 17:43:42 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id 2A6212D4B for ; Tue, 8 Jul 2014 17:43:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 5AE942041C0; Tue, 8 Jul 2014 19:43:39 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zKMuVr9ZXv51; Tue, 8 Jul 2014 19:43:37 +0200 (CEST) Received: from [192.168.48.86] (unknown [199.127.109.253]) by smtp.infotech.no (Postfix) with ESMTPA id 5BF1020415F; Tue, 8 Jul 2014 19:43:35 +0200 (CEST) Message-ID: <53BC2DC5.60005@interlog.com> Date: Tue, 08 Jul 2014 13:43:33 -0400 From: Douglas Gilbert Reply-To: dgilbert@interlog.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Martin Simmons , freebsd-scsi@FreeBSD.org Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver References: <53BB619B.4060908@interlog.com> <201407081145.s68Bjxn6006452@higson.cam.lispworks.com> In-Reply-To: <201407081145.s68Bjxn6006452@higson.cam.lispworks.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Alex Samorukov , Christian Franke X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jul 2014 17:43:42 -0000 On 14-07-08 07:45 AM, Martin Simmons wrote: >>>>>> On Mon, 07 Jul 2014 23:12:27 -0400, Douglas Gilbert said: >> >> On 14-07-07 09:45 PM, bugzilla-noreply at freebsd.org wrote: >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 >>> >>> Mark Linimon changed: >>> >>> What |Removed |Added >>> ---------------------------------------------------------------------------- >>> Assignee|freebsd-bugs at FreeBSD.org |freebsd-scsi at FreeBSD.org >>> Summary|smartctl -H gives "ATA |[iscsi] smartctl -H gives >>> |output registers missing" |"ATA output registers >>> |for a disk using the isci |missing" for a disk using >>> |driver |the isci driver >>> >>> --- Comment #1 from Mark Linimon --- >>> Over to maintainers. >>> >> >> At the point of failure "whatever" produces this SCSI sense data: >> f0 00 01 00 50 40 00 00 00 c2 4f 00 00 1d 00 00 00 00 >> >> FreeBSD is wrong to print out 18 bytes because that is an 8 >> byte (deferred, fixed type) buffer because byte 7 (the >> additional length) is 0. Whatever produced that broken >> sense data is the probably culprit. >> >> It is trying to say there is "ATA pass-through information >> available" but fails to get its message across. > > Hi, > > I'm the original reporter. > > The sense data is smartctl's interpretation of the ccb union. > > In particular, the bytes are from ccb->csio.sense_data and it calculates 18 > from ccb->csio.sense_len - ccb->csio.sense_resid (32 - 14). I don't know if > that is correct or not. > > FWIW, here is the output from CentOS 6.3 on the same machine: > > REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK > Input: FR=0xda, SC=...., LL=...., LM=0x4f, LH=0xc2, DEV=...., CMD=0xb0 > [ata pass-through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00 ] > scsi_status=0x2, host_status=0x0, driver_status=0x8 > info=0x1 duration=17 milliseconds resid=0 > >>> Sense buffer, len=22: > 00 72 00 00 00 00 00 00 0e 09 0c 00 00 00 00 00 00 > 10 00 4f 00 c2 40 50 > status=2: [desc] sense_key=0 asc=0 ascq=0 > Values from ATA Return Descriptor are: > 00 09 0c 00 00 00 00 00 00 00 4f 00 c2 40 50 > [Duration: 0.016s] > Output: ERR=0x00, SC=0x00, LL=0x00, LM=0x4f, LH=0xc2, DEV=0x40, STS=0x50 > REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK returned 0 > > It appears to have the same ata pass-through command but completely different > sense data. The SAT standard originally only defined "descriptor" sense data format for passing back ATA errors and warning. That is what is being done properly in the Centos 6.3 output that you have shown above. More recently someone at T10 objected that SAT ignored the D_SENSE flag which allows an application client to choose whether it wants "fixed" or "descriptor" sense data format. It looks like the FreeBSD case is trying to produce the equivalent "fixed" sense data, but it fails to format it properly. So in the FreeBSD case you are looking for the SAT Layer (SATL). It could be in the FreeBSD CAM mid-level, the driver code or firmware, or in a remote device like a USB bridge. However if it was in something like a USB bridge then you would expect FreeBSD and Linux to react the same way. So it looks like a FreeBSD bug. Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Wed Jul 9 10:47:11 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CA0A30F for ; Wed, 9 Jul 2014 10:47:11 +0000 (UTC) Received: from exprod7og122.obsmtp.com (exprod7og122.obsmtp.com [64.18.2.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B5F172CD5 for ; Wed, 9 Jul 2014 10:47:10 +0000 (UTC) Received: from mail-qg0-f54.google.com ([209.85.192.54]) (using TLSv1) by exprod7ob122.postini.com ([64.18.6.12]) with SMTP ID DSNKU70dqMSoE1WwCpE4kcmq+fsIt8vMUjiv@postini.com; Wed, 09 Jul 2014 03:47:10 PDT Received: by mail-qg0-f54.google.com with SMTP id q107so6139300qgd.27 for ; Wed, 09 Jul 2014 03:47:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type; bh=kriuWlWptEk3L9LZUu3HUhfhk3Gk8r2AFp0Ca1kh4JU=; b=Pi0t/KzL/DVwDjtKtVm0ZsggjkHjL4FDzKvwPV/LgpzUraNK3HVp1C4oxTYXBhGxjM RXO3bFkHa087WCAuc6WM7NCscULdjaQnLEzbz5MQjQvXG0nlLy4mICU37hEO3+E+eUVQ Qc2cy4H00FcaZ/ZsWrTKeIRsaqhLWvIzybUXS+yuEdnZZ8fNg73FP8QfQ0lpIIGeJsMn xDTfyHKAWfw9N6cuftGVymEI92u7Gk0j+DY1L6Ge2UUDc7Vymra0W3Z2Ngjk6gLiLdjq QbC5w1oqmd0nxismRDh42Vnx26h2FxJ20iaXED4Q1AZzLUYFvWxsVwCbN5mcfmqR8+mt 5fhw== X-Received: by 10.224.129.68 with SMTP id n4mr68321552qas.66.1404902383127; Wed, 09 Jul 2014 03:39:43 -0700 (PDT) X-Gm-Message-State: ALoCoQnf0FVb7JH5zFecooXeSj5/7HBSs80LIKv6ukfPYDjptHzGsUkGzWj0KpG2lUKUdbNxQXOZfTVaU2vqUDHbU8sItqHVjo5Dboi3tNvRrcO8Wmppfyf8kqY7dEZxND6qEEfcSx95qPOf1jNHTaAnO7BzXd2Z8A== X-Received: by 10.224.129.68 with SMTP id n4mr68321532qas.66.1404902382955; Wed, 09 Jul 2014 03:39:42 -0700 (PDT) From: Sumit Saxena References: <559aba5a124ee1e32ddffb1380399e28@mail.gmail.com> <20140624130840.GK1560@funkthat.com> In-Reply-To: <20140624130840.GK1560@funkthat.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHrqZ/meEJEiDrDuvx3ccVdJK+lWgGJHJxVm1MmakA= Date: Wed, 9 Jul 2014 16:09:41 +0530 Message-ID: <6f92e2e229eaea86429826ce5085e495@mail.gmail.com> Subject: RE: Kernel panic: message secondary GPT header is not in the last LBA To: John-Mark Gurney Content-Type: text/plain; charset=UTF-8 Cc: freebsd-scsi@freebsd.org, Kashyap Desai X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2014 10:47:11 -0000 >-----Original Message----- >From: John-Mark Gurney [mailto:jmg@funkthat.com] >Sent: Tuesday, June 24, 2014 6:39 PM >To: Sumit Saxena >Cc: freebsd-scsi@freebsd.org; Kashyap Desai >Subject: Re: Kernel panic: message secondary GPT header is not in the last >LBA > >Sumit Saxena wrote this message on Tue, Jun 24, 2014 at 18:27 +0530: >> Hi All, >> >> >> >> While doing some testing on driver, I am facing kernel panic >> inside GEOM module. I am using FreeBSD10.0 64bit, installed on Virtual >> drive connected behind LSI MegaRAID SAS 9361 controller and two >> Enclosures- Dell MD1220 with total 39 drives are connected to the >> controller. As I convert unconfigured drives(connected to Enclosures) >> to JBOD(plain drive without any RAID configuration exposed to OS), >> kernel panic is observed inside GEOM module with below traces- >> >> >> >> =================================================== >> >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) >> >> ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afebe51 >> >> ses1: pass30,da26: Element descriptor: 'SLOT 20 ' >> >> ses1: pass30,da26: SAS Device Slot Element: 1 Phys at Slot 20, Not All >> Phys >> >> ses1: phy 0: SAS device type 1 id 0 >> >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) >> >> ses1: phy 0: parent 50080e5223c0f03f addr 5000c5004cf152f1 >> >> ses1: pass37,da33: Element descriptor: 'SLOT 21 ' >> >> ses1: pass37,da33: SAS Device Slot Element: 1 Phys at Slot 21, Not All >> Phys >> >> ses1: phy 0: SAS device type 1 id 0 >> >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) >> >> ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afd3659 >> >> ses1: pass21,da17: Element descriptor: 'SLOT 22 ' >> >> ses1: pass21,da17: SAS Device Slot Element: 1 Phys at Slot 22, Not All >> Phys >> >> ses1: phy 0: SAS device type 1 id 0 >> >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) >> >> ses1: phy 0: parent 50080e5223c0f03f addr 5000cca00baf22c1 >> >> GEOM: da12: the secondary GPT header is not in the last LBA. >> >> >> >> >> >> Fatal trap 18: integer divide fault while in kernel mode >> >> cpuid = 4; apic id = 10 >> >> instruction pointer = 0x20:0xffffffff80805045 >> >> stack pointer = 0x28:0xfffffe0c23ded9e0 >> >> frame pointer = 0x28:0xfffffe0c23deda30 >> >> code segment = base 0x0, limit 0xfffff, type 0x1b >> >> = DPL 0, pres 1, long >> 1, >> def32 0, gran 1 >> >> processor eflags = interrupt enabled, resume, IOPL = 0 >> >> current process = 13 (g_event) >> >> trap number = 18 >> >> panic: integer divide fault >> >> cpuid = 4 >> >> KDB: stack backtrace: >> >> #0 0xffffffff808cb220 at kdb_backtrace+0x60 >> >> #1 0xffffffff80892d05 at panic+0x155 >> >> #2 0xffffffff80c71ae2 at trap_fatal+0x3a2 >> >> #3 0xffffffff80c7171f at trap+0x7bf >> >> #4 0xffffffff80c587e2 at calltrap+0x8 >> >> #5 0xffffffff80803574 at g_label_taste+0x3a4 >> >> #6 0xffffffff80802106 at g_new_provider_event+0xb6 >> >> #7 0xffffffff807fe1d6 at g_run_events+0x166 >> >> #8 0xffffffff80864dda at fork_exit+0x9a >> >> #9 0xffffffff80c58d1e at fork_trampoline+0xe >> >> Uptime: 4m48s >> >> kernel trap 12 with interrupts disabled > >Can you run w/ DDB enabled and get a dump? > >an integer divide makes me think of a divide by zero error... Are you sure >things like sector size are set properly? Sorry for delay in response, sector size and other disk parameters are set properly(verified by "diskinfo"), same set of drives works well on linux. We are not able to collect dump, enabled DDB but dump Is not getting collected. Partial dump is getting copied sometimes[seen message on console while dumping]. > >-- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." From owner-freebsd-scsi@FreeBSD.ORG Wed Jul 9 16:44:55 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A7358435 for ; Wed, 9 Jul 2014 16:44:55 +0000 (UTC) Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6768921FD for ; Wed, 9 Jul 2014 16:44:55 +0000 (UTC) Received: by mail-ob0-f181.google.com with SMTP id wp4so8154544obc.26 for ; Wed, 09 Jul 2014 09:44:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=SvnSU87XWrnBkLOnkBXWlF6VSn1hiSAPmvCLX425tKA=; b=xC/15+vn35OFSihyVJfnFYLoTkxlTnhkBbItuREQUir/INkQ0Rw8CAyfqRkxKSeAxP MH/gGikcnELvZIB3K04LshlXQLOcwE0dNLOiGnlGzFDeIUqpkbGjh7Beg3QYgJ27GO70 I9kKpOaAxiaDv+TpTZGlh58Nf6E9e8/MJqm7dOKwt8dP5y2yKQ6aE/bLZ8dbwDe2HVTX LKsAclb8LeyZ5rhr64zhWghJtoCmBm5F9A+LU+Hm20/qT/n0t+M+VhFYYJ6VTJP7brnR zEH/WqCvRU4Ko1LgJ3mlddIbFqP2TT5gJvVKymM4aogQByot5sTHNT6KaQnR6fu+RUsO oRJg== MIME-Version: 1.0 X-Received: by 10.182.133.69 with SMTP id pa5mr26585833obb.2.1404924294405; Wed, 09 Jul 2014 09:44:54 -0700 (PDT) Received: by 10.202.221.197 with HTTP; Wed, 9 Jul 2014 09:44:54 -0700 (PDT) In-Reply-To: <53BC2DC5.60005@interlog.com> References: <53BB619B.4060908@interlog.com> <201407081145.s68Bjxn6006452@higson.cam.lispworks.com> <53BC2DC5.60005@interlog.com> Date: Wed, 9 Jul 2014 09:44:54 -0700 Message-ID: Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver From: Jim Harris To: dgilbert@interlog.com Content-Type: multipart/mixed; boundary=e89a8ff1ce12834f2404fdc56d7f X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD-scsi , Alex Samorukov , Christian Franke X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2014 16:44:55 -0000 --e89a8ff1ce12834f2404fdc56d7f Content-Type: text/plain; charset=UTF-8 On Tue, Jul 8, 2014 at 10:43 AM, Douglas Gilbert wrote: > On 14-07-08 07:45 AM, Martin Simmons wrote: > >> On Mon, 07 Jul 2014 23:12:27 -0400, Douglas Gilbert said: >>>>>>> >>>>>> >>> On 14-07-07 09:45 PM, bugzilla-noreply at freebsd.org wrote: >>> >>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191717 >>>> >>>> Mark Linimon changed: >>>> >>>> What |Removed |Added >>>> ------------------------------------------------------------ >>>> ---------------- >>>> Assignee|freebsd-bugs at FreeBSD.org |freebsd-scsi at >>>> FreeBSD.org >>>> Summary|smartctl -H gives "ATA |[iscsi] smartctl -H >>>> gives >>>> |output registers missing" |"ATA output registers >>>> |for a disk using the isci |missing" for a disk >>>> using >>>> |driver |the isci driver >>>> >>>> --- Comment #1 from Mark Linimon --- >>>> Over to maintainers. >>>> >>>> >>> At the point of failure "whatever" produces this SCSI sense data: >>> f0 00 01 00 50 40 00 00 00 c2 4f 00 00 1d 00 00 00 00 >>> >>> FreeBSD is wrong to print out 18 bytes because that is an 8 >>> byte (deferred, fixed type) buffer because byte 7 (the >>> additional length) is 0. Whatever produced that broken >>> sense data is the probably culprit. >>> >>> It is trying to say there is "ATA pass-through information >>> available" but fails to get its message across. >>> >> >> Hi, >> >> I'm the original reporter. >> >> The sense data is smartctl's interpretation of the ccb union. >> >> In particular, the bytes are from ccb->csio.sense_data and it calculates >> 18 >> from ccb->csio.sense_len - ccb->csio.sense_resid (32 - 14). I don't know >> if >> that is correct or not. >> >> FWIW, here is the output from CentOS 6.3 on the same machine: >> >> REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK >> Input: FR=0xda, SC=...., LL=...., LM=0x4f, LH=0xc2, DEV=...., CMD=0xb0 >> [ata pass-through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00 ] >> scsi_status=0x2, host_status=0x0, driver_status=0x8 >> info=0x1 duration=17 milliseconds resid=0 >> >>> Sense buffer, len=22: >> 00 72 00 00 00 00 00 00 0e 09 0c 00 00 00 00 00 00 >> 10 00 4f 00 c2 40 50 >> status=2: [desc] sense_key=0 asc=0 ascq=0 >> Values from ATA Return Descriptor are: >> 00 09 0c 00 00 00 00 00 00 00 4f 00 c2 40 50 >> [Duration: 0.016s] >> Output: ERR=0x00, SC=0x00, LL=0x00, LM=0x4f, LH=0xc2, DEV=0x40, STS=0x50 >> REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK returned 0 >> >> It appears to have the same ata pass-through command but completely >> different >> sense data. >> > > The SAT standard originally only defined "descriptor" sense > data format for passing back ATA errors and warning. That is > what is being done properly in the Centos 6.3 output that > you have shown above. > > More recently someone at T10 objected that SAT ignored the D_SENSE > flag which allows an application client to choose whether it wants > "fixed" or "descriptor" sense data format. It looks like the FreeBSD > case is trying to produce the equivalent "fixed" sense data, but it > fails to format it properly. > > So in the FreeBSD case you are looking for the SAT Layer (SATL). It > could be in the FreeBSD CAM mid-level, the driver code or firmware, > or in a remote device like a USB bridge. However if it was in something > like a USB bridge then you would expect FreeBSD and Linux to react > the same way. So it looks like a FreeBSD bug. > > Doug Gilbert > > Apologies for missing this thread until now. The SATL that is generating this fixed format sense data is in the isci driver itself. I think isci is returning the fixed format sense data correctly. My reading of the SPC spec is that fixed format sense data is 18 bytes + additional sense length, so if sense_data[7] == 0, then the sense data should indeed be 18 bytes long. SAT-3 also indicates that for fixed format sense data for ATA PASSTHROUGH commands, there are no additional sense bytes - everything is returned in the INFORMATION and COMMAND-SPECIFIC INFORMATION bytes which are part of the first 18 bytes. I have looked a little at the smartctl code, and it does not look like it supports fixed format sense data - at least for ATA PASSTHROUGH. It expects additional sense length to be non-zero. smartctl could be fixed, but I think for compatibility, I should have isci return descriptor sense for ATA PASSTHROUGH commands instead. Martin - could you please test the attached patch? This passes your test case using smartctl 6.3 r3939. Thanks, -Jim --e89a8ff1ce12834f2404fdc56d7f Content-Type: application/octet-stream; name="sati_passthrough_sense.patch" Content-Disposition: attachment; filename="sati_passthrough_sense.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hxevf8v30 SW5kZXg6IHN5cy9kZXYvaXNjaS9zY2lsL2ludGVsX3Njc2kuaAo9PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMv ZGV2L2lzY2kvc2NpbC9pbnRlbF9zY3NpLmgJKHJldmlzaW9uIDI2NjQ3MykKKysrIHN5cy9kZXYv aXNjaS9zY2lsL2ludGVsX3Njc2kuaAkod29ya2luZyBjb3B5KQpAQCAtMTQ3LDYgKzE0Nyw4IEBA CiAjZGVmaW5lIFNDU0lfSU5GT1JNQVRJT05fREVTQ1JJUFRPUl9MRU5HVEggICAgICAgICAgICAg ICAgMHgwYwogI2RlZmluZSBTQ1NJX0JMT0NLX0RFU0NSSVBUT1JfQURESVRJT05BTF9MRU5HVEgg ICAgICAgICAgICAweDIKICNkZWZpbmUgU0NTSV9CTE9DS19ERVNDUklQVE9SX0xFTkdUSCAgICAg ICAgICAgICAgICAgICAgMHg0CisjZGVmaW5lIFNDU0lfQVRBX1NUQVRVU19SRVRVUk5fREVTQ1JJ UFRPUl9BRERJVElPTkFMX0xFTkdUSAkweGMKKyNkZWZpbmUgU0NTSV9BVEFfU1RBVFVTX1JFVFVS Tl9ERVNDUklQVE9SX0xFTkdUSAkJMHhlCiAKICNkZWZpbmUgU0NTSV9TRU5TRV9EQVRBX0RFU0Nf QklUICAgIDB4MDEKIApJbmRleDogc3lzL2Rldi9pc2NpL3NjaWwvc2F0aV9wYXNzdGhyb3VnaC5j Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT0KLS0tIHN5cy9kZXYvaXNjaS9zY2lsL3NhdGlfcGFzc3Rocm91Z2guYwkocmV2 aXNpb24gMjY2NDczKQorKysgc3lzL2Rldi9pc2NpL3NjaWwvc2F0aV9wYXNzdGhyb3VnaC5jCSh3 b3JraW5nIGNvcHkpCkBAIC0xNzYsOCArMTc2LDYgQEAKICAgIFU4ICAgICAgICAgICAgICAgICAg ICAqIHNlbnNlX2RhdGE7CiAgICBVMzIgICAgICAgICAgICAgICAgICAgICBzZW5zZV9sZW47CiAg ICBVOCAgICAgICAgICAgICAgICAgICAgKiBjZGI7Ci0gICB1bnNpZ25lZCBjaGFyICAgICAgICAg ICBzZWN0b3JfY291bnRfdXBwZXI7Ci0gICB1bnNpZ25lZCBjaGFyICAgICAgICAgICBsYmFfdXBw ZXI7CiAKICNpZmRlZiBTQVRJX1RSQU5TUE9SVF9TVVBQT1JUU19TQVMKICAgIFNDSV9TU1BfUkVT UE9OU0VfSVVfVCAqIHJzcF9pdSA9IChTQ0lfU1NQX1JFU1BPTlNFX0lVX1QqKQpAQCAtMjA4LDMy ICsyMDYsMjkgQEAKIAogICAgY2RiID0gc2F0aV9jYl9nZXRfY2RiX2FkZHJlc3Moc2NzaV9pbyk7 CiAKLSAgIGlmIChzYXRpX2dldF9hdGFfc2VjdG9yX2NvdW50X2V4dChyZWdpc3Rlcl9maXMpICE9 IDApIHsKLSAgICAgIHNlY3Rvcl9jb3VudF91cHBlciA9IDE7Ci0gICB9IGVsc2UgewotICAgICAg IHNlY3Rvcl9jb3VudF91cHBlciA9IDA7Ci0gICB9CisgICBzYXRpX3NldF9zZW5zZV9kYXRhX2J5 dGUoc2Vuc2VfZGF0YSwgc2Vuc2VfbGVuLCA4KzAsIFNDU0lfQVRBX1NUQVRVU19SRVRVUk5fREVT Q1JJUFRPUl9UWVBFKTsKKyAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBz ZW5zZV9sZW4sIDgrMSwgU0NTSV9BVEFfU1RBVFVTX1JFVFVSTl9ERVNDUklQVE9SX0FERElUSU9O QUxfTEVOR1RIKTsKKyAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5z ZV9sZW4sIDgrMiwgUEFTU1RIUk9VR0hfQ0RCX0VYVEVORChjZGIpKTsKKyAgIHNhdGlfc2V0X3Nl bnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5zZV9sZW4sIDgrMywgKFU4KXNhdGlfZ2V0X2F0 YV9lcnJvcihyZWdpc3Rlcl9maXMpKTsKKyAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5z ZV9kYXRhLCBzZW5zZV9sZW4sIDgrNSwgc2F0aV9nZXRfYXRhX3NlY3Rvcl9jb3VudChyZWdpc3Rl cl9maXMpKTsKKyAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5zZV9s ZW4sIDgrNywgc2F0aV9nZXRfYXRhX2xiYV9sb3cocmVnaXN0ZXJfZmlzKSk7CisgICBzYXRpX3Nl dF9zZW5zZV9kYXRhX2J5dGUoc2Vuc2VfZGF0YSwgc2Vuc2VfbGVuLCA4KzksIHNhdGlfZ2V0X2F0 YV9sYmFfbWlkKHJlZ2lzdGVyX2ZpcykpOworICAgc2F0aV9zZXRfc2Vuc2VfZGF0YV9ieXRlKHNl bnNlX2RhdGEsIHNlbnNlX2xlbiwgOCsxMSwgc2F0aV9nZXRfYXRhX2xiYV9oaWdoKHJlZ2lzdGVy X2ZpcykpOworICAgc2F0aV9zZXRfc2Vuc2VfZGF0YV9ieXRlKHNlbnNlX2RhdGEsIHNlbnNlX2xl biwgOCsxMiwgc2F0aV9nZXRfYXRhX2RldmljZShyZWdpc3Rlcl9maXMpKTsKKyAgIHNhdGlfc2V0 X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5zZV9sZW4sIDgrMTMsIChVOClzYXRpX2dl dF9hdGFfc3RhdHVzKHJlZ2lzdGVyX2ZpcykpOwogCi0gICBpZiAoc2F0aV9nZXRfYXRhX2xiYV9o aWdoX2V4dChyZWdpc3Rlcl9maXMpICE9IDAgfHwKLSAgICAgICBzYXRpX2dldF9hdGFfbGJhX21p ZF9leHQocmVnaXN0ZXJfZmlzKSAhPSAwIHx8Ci0gICAgICAgc2F0aV9nZXRfYXRhX2xiYV9sb3df ZXh0KHJlZ2lzdGVyX2ZpcykgIT0gMCkgewotICAgICAgbGJhX3VwcGVyID0gMTsKKyAgIGlmIChQ QVNTVEhST1VHSF9DREJfRVhURU5EKGNkYikpIHsKKyAgICAgIHNhdGlfc2V0X3NlbnNlX2RhdGFf Ynl0ZShzZW5zZV9kYXRhLCBzZW5zZV9sZW4sIDgrNCwgc2F0aV9nZXRfYXRhX3NlY3Rvcl9jb3Vu dF9leHQocmVnaXN0ZXJfZmlzKSk7CisgICAgICBzYXRpX3NldF9zZW5zZV9kYXRhX2J5dGUoc2Vu c2VfZGF0YSwgc2Vuc2VfbGVuLCA4KzYsIHNhdGlfZ2V0X2F0YV9sYmFfbG93X2V4dChyZWdpc3Rl cl9maXMpKTsKKyAgICAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5z ZV9sZW4sIDgrOCwgc2F0aV9nZXRfYXRhX2xiYV9taWRfZXh0KHJlZ2lzdGVyX2ZpcykpOworICAg ICAgc2F0aV9zZXRfc2Vuc2VfZGF0YV9ieXRlKHNlbnNlX2RhdGEsIHNlbnNlX2xlbiwgOCsxMCwg c2F0aV9nZXRfYXRhX2xiYV9oaWdoX2V4dChyZWdpc3Rlcl9maXMpKTsKICAgIH0gZWxzZSB7Ci0g ICAgICAgbGJhX3VwcGVyID0gMDsKKyAgICAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5z ZV9kYXRhLCBzZW5zZV9sZW4sIDgrNCwgMCk7CisgICAgICBzYXRpX3NldF9zZW5zZV9kYXRhX2J5 dGUoc2Vuc2VfZGF0YSwgc2Vuc2VfbGVuLCA4KzYsIDApOworICAgICAgc2F0aV9zZXRfc2Vuc2Vf ZGF0YV9ieXRlKHNlbnNlX2RhdGEsIHNlbnNlX2xlbiwgOCs4LCAwKTsKKyAgICAgIHNhdGlfc2V0 X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5zZV9sZW4sIDgrMTAsIDApOwogICAgfQog Ci0gICAvLyBJbmZvcm1hdGlvbiBzZWN0aW9uCi0gICBzYXRpX3NldF9zZW5zZV9kYXRhX2J5dGUo c2Vuc2VfZGF0YSwgc2Vuc2VfbGVuLCAzLCAgKFU4KXNhdGlfZ2V0X2F0YV9lcnJvcihyZWdpc3Rl cl9maXMpKTsKLSAgIHNhdGlfc2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5zZV9s ZW4sIDQsICAoVTgpc2F0aV9nZXRfYXRhX3N0YXR1cyhyZWdpc3Rlcl9maXMpKTsKLSAgIHNhdGlf c2V0X3NlbnNlX2RhdGFfYnl0ZShzZW5zZV9kYXRhLCBzZW5zZV9sZW4sIDUsICBzYXRpX2dldF9h dGFfZGV2aWNlKHJlZ2lzdGVyX2ZpcykpOwotICAgc2F0aV9zZXRfc2Vuc2VfZGF0YV9ieXRlKHNl bnNlX2RhdGEsIHNlbnNlX2xlbiwgNiwgIHNhdGlfZ2V0X2F0YV9zZWN0b3JfY291bnQocmVnaXN0 ZXJfZmlzKSk7Ci0KLSAgIC8vIENvbW1hbmQgc3BlY2lmaWMgc2VjdGlvbgotICAgc2F0aV9zZXRf c2Vuc2VfZGF0YV9ieXRlKHNlbnNlX2RhdGEsIHNlbnNlX2xlbiwgOCwgIChQQVNTVEhST1VHSF9D REJfRVhURU5EKGNkYikgPDwgNykgfCAoc2VjdG9yX2NvdW50X3VwcGVyIDw8IDYpIHwgKGxiYV91 cHBlciA8PCA1KSk7Ci0gICBzYXRpX3NldF9zZW5zZV9kYXRhX2J5dGUoc2Vuc2VfZGF0YSwgc2Vu c2VfbGVuLCA5LCAgc2F0aV9nZXRfYXRhX2xiYV9oaWdoKHJlZ2lzdGVyX2ZpcykpOwotICAgc2F0 aV9zZXRfc2Vuc2VfZGF0YV9ieXRlKHNlbnNlX2RhdGEsIHNlbnNlX2xlbiwgMTAsIHNhdGlfZ2V0 X2F0YV9sYmFfbWlkKHJlZ2lzdGVyX2ZpcykpOwotICAgc2F0aV9zZXRfc2Vuc2VfZGF0YV9ieXRl KHNlbnNlX2RhdGEsIHNlbnNlX2xlbiwgMTEsIHNhdGlfZ2V0X2F0YV9sYmFfbG93KHJlZ2lzdGVy X2ZpcykpOwotCiAgICBzZXF1ZW5jZS0+aXNfc2Vuc2VfcmVzcG9uc2Vfc2V0ID0gVFJVRTsKIH0K IApJbmRleDogc3lzL2Rldi9pc2NpL3NjaWwvc2F0aV91dGlsLmMKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lz L2Rldi9pc2NpL3NjaWwvc2F0aV91dGlsLmMJKHJldmlzaW9uIDI2NjQ3MykKKysrIHN5cy9kZXYv aXNjaS9zY2lsL3NhdGlfdXRpbC5jCSh3b3JraW5nIGNvcHkpCkBAIC00ODUsNyArNDg1LDkgQEAK IHN0YXRpYwogVTggc2F0aV9zY3NpX2dldF9zZW5zZV9kYXRhX3Jlc3BvbnNlX2NvZGUoU0FUSV9U UkFOU0xBVE9SX1NFUVVFTkNFX1QgKiBzZXF1ZW5jZSkKIHsKLSAgICBpZiAoc2VxdWVuY2UtPmRl dmljZS0+ZGVzY3JpcHRvcl9zZW5zZV9lbmFibGUpCisgICAgaWYgKHNlcXVlbmNlLT5kZXZpY2Ut PmRlc2NyaXB0b3Jfc2Vuc2VfZW5hYmxlIHx8CisgICAgICAgIHNlcXVlbmNlLT50eXBlID09IFNB VElfU0VRVUVOQ0VfQVRBX1BBU1NUSFJPVUdIXzEyIHx8CisgICAgICAgIHNlcXVlbmNlLT50eXBl ID09IFNBVElfU0VRVUVOQ0VfQVRBX1BBU1NUSFJPVUdIXzE2KQogICAgIHsKICAgICAgICByZXR1 cm4gU0NTSV9ERVNDUklQVE9SX0NVUlJFTlRfUkVTUE9OU0VfQ09ERTsKICAgICB9CkBAIC01NDgs NiArNTUwLDEyIEBACiAgICAgICAgLy8gJiYgIWRlZmluZWQoRElTQUJMRV9TQVRJX1dSSVRFKQog ICAgICAgICBsZW5ndGggKz0gU0NTSV9JTkZPUk1BVElPTl9ERVNDUklQVE9SX0xFTkdUSDsKICAg ICAgICAgYnJlYWs7CisjaWYgIWRlZmluZWQoRElTQUJMRV9TQVRJX0FUQV9QQVNTVEhST1VHSCkK KyAgICBjYXNlIFNDU0lfQVRBX1BBU1NUSFJVXzEyOgorICAgIGNhc2UgU0NTSV9BVEFfUEFTU1RI UlVfMTY6CisgICAgICAgIGxlbmd0aCArPSBTQ1NJX0FUQV9TVEFUVVNfUkVUVVJOX0RFU0NSSVBU T1JfTEVOR1RIOworICAgICAgICBicmVhazsKKyNlbmRpZiAvLyAhZGVmaW5lZChESVNBQkxFX1NB VElfQVRBX1BBU1NUSFJPVUdIKQogICAgIH0KIAogICAgIHJldHVybiBsZW5ndGg7Cg== --e89a8ff1ce12834f2404fdc56d7f-- From owner-freebsd-scsi@FreeBSD.ORG Wed Jul 9 18:23:20 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C0E2EC for ; Wed, 9 Jul 2014 18:23:20 +0000 (UTC) Received: from smtp.infotech.no (smtp.infotech.no [82.134.31.41]) by mx1.freebsd.org (Postfix) with ESMTP id A76E72B01 for ; Wed, 9 Jul 2014 18:23:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.infotech.no (Postfix) with ESMTP id 79C842041C0; Wed, 9 Jul 2014 20:23:11 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.6 (20110518) (Debian) at infotech.no Received: from smtp.infotech.no ([127.0.0.1]) by localhost (smtp.infotech.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iSa8cMPbl8lv; Wed, 9 Jul 2014 20:23:09 +0200 (CEST) Received: from [192.168.48.86] (unknown [199.127.109.253]) by smtp.infotech.no (Postfix) with ESMTPA id EA9B1204159; Wed, 9 Jul 2014 20:23:07 +0200 (CEST) Message-ID: <53BD888A.603@interlog.com> Date: Wed, 09 Jul 2014 14:23:06 -0400 From: Douglas Gilbert Reply-To: dgilbert@interlog.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Jim Harris Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver References: <53BB619B.4060908@interlog.com> <201407081145.s68Bjxn6006452@higson.cam.lispworks.com> <53BC2DC5.60005@interlog.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: FreeBSD-scsi , Alex Samorukov , Christian Franke X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jul 2014 18:23:20 -0000 On 14-07-09 12:44 PM, Jim Harris wrote: > > > > On Tue, Jul 8, 2014 at 10:43 AM, Douglas Gilbert > wrote: > > On 14-07-08 07:45 AM, Martin Simmons wrote: > > On Mon, 07 Jul 2014 23:12:27 -0400, Douglas Gilbert > said: > > > On 14-07-07 09:45 PM, bugzilla-noreply at freebsd.org > wrote: > > https://bugs.freebsd.org/__bugzilla/show_bug.cgi?id=__191717 > > > Mark Linimon changed: > > What |Removed |Added > ------------------------------__------------------------------__---------------- > Assignee|freebsd-bugs at FreeBSD.org > |freebsd-scsi at FreeBSD.org > Summary|smartctl -H gives "ATA |[iscsi] > smartctl -H gives > |output registers missing" |"ATA output > registers > |for a disk using the isci |missing" for > a disk using > |driver |the isci driver > > --- Comment #1 from Mark Linimon --- > Over to maintainers. > > > At the point of failure "whatever" produces this SCSI sense data: > f0 00 01 00 50 40 00 00 00 c2 4f 00 00 1d 00 00 00 00 > > FreeBSD is wrong to print out 18 bytes because that is an 8 > byte (deferred, fixed type) buffer because byte 7 (the > additional length) is 0. Whatever produced that broken > sense data is the probably culprit. > > It is trying to say there is "ATA pass-through information > available" but fails to get its message across. > > > Hi, > > I'm the original reporter. > > The sense data is smartctl's interpretation of the ccb union. > > In particular, the bytes are from ccb->csio.sense_data and it calculates 18 > from ccb->csio.sense_len - ccb->csio.sense_resid (32 - 14). I don't know if > that is correct or not. > > FWIW, here is the output from CentOS 6.3 on the same machine: > > REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK > Input: FR=0xda, SC=...., LL=...., LM=0x4f, LH=0xc2, DEV=...., CMD=0xb0 > [ata pass-through(16): 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00 ] > scsi_status=0x2, host_status=0x0, driver_status=0x8 > info=0x1 duration=17 milliseconds resid=0 > >>> Sense buffer, len=22: > 00 72 00 00 00 00 00 00 0e 09 0c 00 00 00 00 00 00 > 10 00 4f 00 c2 40 50 > status=2: [desc] sense_key=0 asc=0 ascq=0 > Values from ATA Return Descriptor are: > 00 09 0c 00 00 00 00 00 00 00 4f 00 c2 40 50 > [Duration: 0.016s] > Output: ERR=0x00, SC=0x00, LL=0x00, LM=0x4f, LH=0xc2, DEV=0x40, STS=0x50 > REPORT-IOCTL: Device=/dev/sdc Command=SMART STATUS CHECK returned 0 > > It appears to have the same ata pass-through command but completely > different > sense data. > > > The SAT standard originally only defined "descriptor" sense > data format for passing back ATA errors and warning. That is > what is being done properly in the Centos 6.3 output that > you have shown above. > > More recently someone at T10 objected that SAT ignored the D_SENSE > flag which allows an application client to choose whether it wants > "fixed" or "descriptor" sense data format. It looks like the FreeBSD > case is trying to produce the equivalent "fixed" sense data, but it > fails to format it properly. > > So in the FreeBSD case you are looking for the SAT Layer (SATL). It > could be in the FreeBSD CAM mid-level, the driver code or firmware, > or in a remote device like a USB bridge. However if it was in something > like a USB bridge then you would expect FreeBSD and Linux to react > the same way. So it looks like a FreeBSD bug. > > Doug Gilbert > > > Apologies for missing this thread until now. > > The SATL that is generating this fixed format sense data is in the isci driver > itself. I think isci is returning the fixed format sense data correctly. My > reading of the SPC spec is that fixed format sense data is 18 bytes + additional > sense length, so if sense_data[7] == 0, then the sense data should indeed be 18 > bytes long. Wrong. See spc4r37.pdf section 4.5.3 or any SCSI standard back to SCSI-2. In table 53 — "Fixed format sense data" in the byte 7 row is written: "ADDITIONAL SENSE LENGTH (n-7)". For the typical (fixed format) sense data length of 18, n will be 17 since the bytes count from 0. So byte 7 should contain 10 (0xa in hex) to indicate an 18 byte sense data buffer. Thanks for identifying the isci as the culprit. As a general rule in SCSI standards and drafts "additional length" fields count starting from the byte following that field. This is no exception, so the isci driver may be making the same mistake elsewhere. SAT-3 also indicates that for fixed format sense data for ATA > PASSTHROUGH commands, there are no additional sense bytes - everything is > returned in the INFORMATION and COMMAND-SPECIFIC INFORMATION bytes which are > part of the first 18 bytes. > > I have looked a little at the smartctl code, and it does not look like it > supports fixed format sense data - at least for ATA PASSTHROUGH. It expects > additional sense length to be non-zero. smartctl could be fixed, but I think > for compatibility, I should have isci return descriptor sense for ATA > PASSTHROUGH commands instead. > > Martin - could you please test the attached patch? This passes your test case > using smartctl 6.3 r3939. Doug Gilbert From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 09:21:14 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56D7B510 for ; Thu, 10 Jul 2014 09:21:14 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 29BB32508 for ; Thu, 10 Jul 2014 09:21:13 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6A9L88l025270 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 10 Jul 2014 02:21:09 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6A9L8Gn025269; Thu, 10 Jul 2014 02:21:08 -0700 (PDT) (envelope-from jmg) Date: Thu, 10 Jul 2014 02:21:08 -0700 From: John-Mark Gurney To: Sumit Saxena Subject: Re: Kernel panic: message secondary GPT header is not in the last LBA Message-ID: <20140710092108.GU45513@funkthat.com> Mail-Followup-To: John-Mark Gurney , Sumit Saxena , freebsd-scsi@freebsd.org, Kashyap Desai References: <559aba5a124ee1e32ddffb1380399e28@mail.gmail.com> <20140624130840.GK1560@funkthat.com> <6f92e2e229eaea86429826ce5085e495@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6f92e2e229eaea86429826ce5085e495@mail.gmail.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 10 Jul 2014 02:21:09 -0700 (PDT) Cc: freebsd-scsi@freebsd.org, Kashyap Desai X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 09:21:14 -0000 Sumit Saxena wrote this message on Wed, Jul 09, 2014 at 16:09 +0530: > >-----Original Message----- > >From: John-Mark Gurney [mailto:jmg@funkthat.com] > >Sent: Tuesday, June 24, 2014 6:39 PM > >To: Sumit Saxena > >Cc: freebsd-scsi@freebsd.org; Kashyap Desai > >Subject: Re: Kernel panic: message secondary GPT header is not in the > last > >LBA > > > >Sumit Saxena wrote this message on Tue, Jun 24, 2014 at 18:27 +0530: > >> Hi All, > >> > >> > >> > >> While doing some testing on driver, I am facing kernel panic > >> inside GEOM module. I am using FreeBSD10.0 64bit, installed on Virtual > >> drive connected behind LSI MegaRAID SAS 9361 controller and two > >> Enclosures- Dell MD1220 with total 39 drives are connected to the > >> controller. As I convert unconfigured drives(connected to Enclosures) > >> to JBOD(plain drive without any RAID configuration exposed to OS), > >> kernel panic is observed inside GEOM module with below traces- > >> > >> > >> > >> =================================================== > >> > >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > >> > >> ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afebe51 > >> > >> ses1: pass30,da26: Element descriptor: 'SLOT 20 ' > >> > >> ses1: pass30,da26: SAS Device Slot Element: 1 Phys at Slot 20, Not All > >> Phys > >> > >> ses1: phy 0: SAS device type 1 id 0 > >> > >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > >> > >> ses1: phy 0: parent 50080e5223c0f03f addr 5000c5004cf152f1 > >> > >> ses1: pass37,da33: Element descriptor: 'SLOT 21 ' > >> > >> ses1: pass37,da33: SAS Device Slot Element: 1 Phys at Slot 21, Not All > >> Phys > >> > >> ses1: phy 0: SAS device type 1 id 0 > >> > >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > >> > >> ses1: phy 0: parent 50080e5223c0f03f addr 5000c5001afd3659 > >> > >> ses1: pass21,da17: Element descriptor: 'SLOT 22 ' > >> > >> ses1: pass21,da17: SAS Device Slot Element: 1 Phys at Slot 22, Not All > >> Phys > >> > >> ses1: phy 0: SAS device type 1 id 0 > >> > >> ses1: phy 0: protocols: Initiator( None ) Target( SSP ) > >> > >> ses1: phy 0: parent 50080e5223c0f03f addr 5000cca00baf22c1 > >> > >> GEOM: da12: the secondary GPT header is not in the last LBA. > >> > >> > >> > >> > >> > >> Fatal trap 18: integer divide fault while in kernel mode > >> > >> cpuid = 4; apic id = 10 > >> > >> instruction pointer = 0x20:0xffffffff80805045 > >> > >> stack pointer = 0x28:0xfffffe0c23ded9e0 > >> > >> frame pointer = 0x28:0xfffffe0c23deda30 > >> > >> code segment = base 0x0, limit 0xfffff, type 0x1b > >> > >> = DPL 0, pres 1, long > >> 1, > >> def32 0, gran 1 > >> > >> processor eflags = interrupt enabled, resume, IOPL = 0 > >> > >> current process = 13 (g_event) > >> > >> trap number = 18 > >> > >> panic: integer divide fault > >> > >> cpuid = 4 > >> > >> KDB: stack backtrace: > >> > >> #0 0xffffffff808cb220 at kdb_backtrace+0x60 > >> > >> #1 0xffffffff80892d05 at panic+0x155 > >> > >> #2 0xffffffff80c71ae2 at trap_fatal+0x3a2 > >> > >> #3 0xffffffff80c7171f at trap+0x7bf > >> > >> #4 0xffffffff80c587e2 at calltrap+0x8 > >> > >> #5 0xffffffff80803574 at g_label_taste+0x3a4 > >> > >> #6 0xffffffff80802106 at g_new_provider_event+0xb6 > >> > >> #7 0xffffffff807fe1d6 at g_run_events+0x166 > >> > >> #8 0xffffffff80864dda at fork_exit+0x9a > >> > >> #9 0xffffffff80c58d1e at fork_trampoline+0xe > >> > >> Uptime: 4m48s > >> > >> kernel trap 12 with interrupts disabled > > > >Can you run w/ DDB enabled and get a dump? > > > >an integer divide makes me think of a divide by zero error... Are you > sure > >things like sector size are set properly? > > Sorry for delay in response, sector size and other disk parameters are set > properly(verified by "diskinfo"), same set of drives works well on linux. > We are not able to collect dump, enabled DDB but dump > Is not getting collected. Partial dump is getting copied sometimes[seen > message on console while dumping]. Can you run: addr2line -e /boot/kernel/kernel.symbols 0xffffffff80803574 This should give us the line number that the panic is happening on, and help debug it.. Also, is this FreeBSD 10.0-R? or is there a patch level associated w/ it? Easiest way to figure out is to include uname -a in the email so we know exactly what kernel source you are using.. Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 09:23:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 957F7570 for ; Thu, 10 Jul 2014 09:23:00 +0000 (UTC) Received: from gw.zefyris.com (sabik.zefyris.com [IPv6:2001:7a8:3c67:2::254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 273A7252F for ; Thu, 10 Jul 2014 09:22:59 +0000 (UTC) Received: from sekishi.zefyris.com (sekishi.zefyris.com [IPv6:2001:7a8:3c67:2::12]) by gw.zefyris.com (8.14.5/8.14.5) with ESMTP id s6A9Mprk008427; Thu, 10 Jul 2014 11:22:51 +0200 (CEST) Date: Thu, 10 Jul 2014 11:22:51 +0200 From: Francois Tigeot To: FreeBSD-scsi Subject: Data corruption with the mfi(4) driver Message-ID: <20140710092251.GA1206@sekishi.zefyris.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gw.zefyris.com [IPv6:2001:7a8:3c67:2::254]); Thu, 10 Jul 2014 11:22:52 +0200 (CEST) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 09:23:00 -0000 Hi, I have encountered catastrophic data corruption with LSI RAID adapters from recent Dell servers (R720xd). The adapters apparently worked fine at first but eventually destroyed the filesystems of volumes experiencing a high load. I found out the hard way a Postgres database import was a sure way to reproduce the problem. Switching to the mrsas(4) driver made my test machine stable again; I have yet to encounter a similar issue with it. Now, this particular server was running under DragonFly and not FreeBSD but the mfi(4) driver is mostly the same under both operating systems and I have found reports of data corruption issues with mfi(4) and recent LSI adapters in the archives of this list. DragonFly will most likely switch to use mrsas(4) by default for at least the LSI Thunderbolt serie of adapters to avoid data loss. This mail from the mfi(4) and mrsas(4) maintainer contains more details, including a partial list of impacted adapters: http://lists.dragonflybsd.org/pipermail/users/2014-July/128703.html -- Francois Tigeot From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:05:27 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A4736412 for ; Thu, 10 Jul 2014 10:05:27 +0000 (UTC) Received: from exprod7og127.obsmtp.com (exprod7og127.obsmtp.com [64.18.2.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15B07288F for ; Thu, 10 Jul 2014 10:05:26 +0000 (UTC) Received: from mail-lb0-f170.google.com ([209.85.217.170]) (using TLSv1) by exprod7ob127.postini.com ([64.18.6.12]) with SMTP ID DSNKU75lX9I4lZ5mmpSdLjzrai7yuTVll7wm@postini.com; Thu, 10 Jul 2014 03:05:27 PDT Received: by mail-lb0-f170.google.com with SMTP id 10so5930662lbg.15 for ; Thu, 10 Jul 2014 03:05:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:content-type; bh=eAzm6ZIqyIeSUqahkc/1fe701lbnBlsOH4OGE0CXRFE=; b=byZIEHL08FXcSHYJN2FrFCyXAUIJxFuh6imOJmgtCHzflfALJRgr+6gySLbLpYfxQv ivRzU4hIKUsfUmonAFkKzDRLEA7Er6PRs65nvm/bQDENJxq9qI5IgHQN01ExFiv6TWgB SDcPTFwqt2mHkOCur51cBWEUDdR1vDR/TbEnOod94Vzg8isFNy2Lu24o/RDBoV7JmCY3 9TbLKl8bqf62L8Cnk7OMBwPIXYzz+ZY5o4F8IToPsFDCONMT6JUsWyMmbLRsqj/LtfB8 WCLzCXp/v7y7SIXM62K83ZaaPzHWR1CcSYsV+QAagcIneOWs/9Dx5U6ElwgK4BesAV8Q ivdw== X-Received: by 10.112.241.74 with SMTP id wg10mr1988273lbc.3.1404985379175; Thu, 10 Jul 2014 02:42:59 -0700 (PDT) X-Gm-Message-State: ALoCoQn31UcyT0Sr2Quo9UN6UYIDbR33I+762+rxxU60HIpeSZYoPGDa9eayXMq4Sh1WjgBm/XP+ujY2E7UeWS9/TjfKrZV55X1p6zZviNezjbCFQD699J5osLjjwCm40ZmNRVCZ4hA1PF4rxOqeFyZh0M+ad4AYuw== X-Received: by 10.112.241.74 with SMTP id wg10mr1988260lbc.3.1404985379052; Thu, 10 Jul 2014 02:42:59 -0700 (PDT) From: Kashyap Desai References: <20140710092251.GA1206@sekishi.zefyris.com> In-Reply-To: <20140710092251.GA1206@sekishi.zefyris.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQIe8F7BDGMtuKKgyqUqY8F7PysmzJr6ZVzQ Date: Thu, 10 Jul 2014 15:12:58 +0530 Message-ID: <7a96b42e065ef52becc6430e8c951a55@mail.gmail.com> Subject: RE: Data corruption with the mfi(4) driver To: Francois Tigeot , FreeBSD-scsi Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:05:27 -0000 Francois Tigeot, Thanks for the post. Does it means DragFly will add below line in "loader.conf" to keep mrsas as default priority ? hw.mfi.mrsas_enable=1 OR Any plan to change the probe return value of mrsas and mfi ? ` Kashyap > -----Original Message----- > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > scsi@freebsd.org] On Behalf Of Francois Tigeot > Sent: Thursday, July 10, 2014 2:53 PM > To: FreeBSD-scsi > Subject: Data corruption with the mfi(4) driver > > Hi, > > I have encountered catastrophic data corruption with LSI RAID adapters from > recent Dell servers (R720xd). The adapters apparently worked fine at first but > eventually destroyed the filesystems of volumes experiencing a high load. > > I found out the hard way a Postgres database import was a sure way to > reproduce the problem. > Switching to the mrsas(4) driver made my test machine stable again; I have > yet to encounter a similar issue with it. > > Now, this particular server was running under DragonFly and not FreeBSD but > the mfi(4) driver is mostly the same under both operating systems and I have > found reports of data corruption issues with mfi(4) and recent LSI adapters in > the archives of this list. > > DragonFly will most likely switch to use mrsas(4) by default for at least the LSI > Thunderbolt serie of adapters to avoid data loss. > > This mail from the mfi(4) and mrsas(4) maintainer contains more details, > including a partial list of impacted adapters: > http://lists.dragonflybsd.org/pipermail/users/2014-July/128703.html > > -- > Francois Tigeot > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:19:29 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5F8BC89D for ; Thu, 10 Jul 2014 10:19:29 +0000 (UTC) Received: from gw.zefyris.com (sabik.zefyris.com [IPv6:2001:7a8:3c67:2::254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E5C1929A7 for ; Thu, 10 Jul 2014 10:19:28 +0000 (UTC) Received: from sekishi.zefyris.com (sekishi.zefyris.com [IPv6:2001:7a8:3c67:2::12]) by gw.zefyris.com (8.14.5/8.14.5) with ESMTP id s6AAJLBa009989; Thu, 10 Jul 2014 12:19:21 +0200 (CEST) Date: Thu, 10 Jul 2014 12:19:21 +0200 From: Francois Tigeot To: Kashyap Desai Subject: Re: RE: Data corruption with the mfi(4) driver Message-ID: <20140710101921.GB1206@sekishi.zefyris.com> References: <20140710092251.GA1206@sekishi.zefyris.com> <7a96b42e065ef52becc6430e8c951a55@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7a96b42e065ef52becc6430e8c951a55@mail.gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gw.zefyris.com [IPv6:2001:7a8:3c67:2::254]); Thu, 10 Jul 2014 12:19:22 +0200 (CEST) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:19:29 -0000 On Thu, Jul 10, 2014 at 03:12:58PM +0530, Kashyap Desai wrote: > > Does it means DragFly will add below line in "loader.conf" to keep mrsas > as default priority ? > hw.mfi.mrsas_enable=1 > > OR > > Any plan to change the probe return value of mrsas and mfi ? I guess the PCI ids for the problematic controllers will be removed from mfi(4), this is the simplest solution. By the way, I have confirmation mrsas(4) also fixes data corruption issues on FreeBSD. It was tested on a Dell R920 server with the default supplied LSI adapter. -- Francois Tigeot From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:20:55 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2DD8F8E3 for ; Thu, 10 Jul 2014 10:20:55 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id E65E729B5 for ; Thu, 10 Jul 2014 10:20:54 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 27CF320E7088C; Thu, 10 Jul 2014 10:20:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id F17D720E70886; Thu, 10 Jul 2014 10:20:42 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Francois Tigeot" , "FreeBSD-scsi" References: <20140710092251.GA1206@sekishi.zefyris.com> Subject: Re: Data corruption with the mfi(4) driver Date: Thu, 10 Jul 2014 11:20:38 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:20:55 -0000 I cant see any information on the actual corruption or cause in that linked thread do you have any actual details? There was known corruption issues but these where fixed long ago so would be good to confirm the details of what you where running and the HW when you had the issue. As a point of reference we have mfi backed DB machines here and have not had any issues with corruption and they have been in production for over 1 1/2 years. Regards Steve ----- Original Message ----- From: "Francois Tigeot" To: "FreeBSD-scsi" Sent: Thursday, July 10, 2014 10:22 AM Subject: Data corruption with the mfi(4) driver > Hi, > > I have encountered catastrophic data corruption with LSI RAID adapters from > recent Dell servers (R720xd). The adapters apparently worked fine at first > but eventually destroyed the filesystems of volumes experiencing a high load. > > I found out the hard way a Postgres database import was a sure way to > reproduce the problem. > Switching to the mrsas(4) driver made my test machine stable again; I have > yet to encounter a similar issue with it. > > Now, this particular server was running under DragonFly and not FreeBSD but > the mfi(4) driver is mostly the same under both operating systems and I have > found reports of data corruption issues with mfi(4) and recent LSI adapters > in the archives of this list. > > DragonFly will most likely switch to use mrsas(4) by default for at least the > LSI Thunderbolt serie of adapters to avoid data loss. > > This mail from the mfi(4) and mrsas(4) maintainer contains more details, > including a partial list of impacted adapters: > http://lists.dragonflybsd.org/pipermail/users/2014-July/128703.html > > -- > Francois Tigeot > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:25:07 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ED09A955 for ; Thu, 10 Jul 2014 10:25:07 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA4E62A41 for ; Thu, 10 Jul 2014 10:25:06 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id C22859DCDBE; Thu, 10 Jul 2014 12:19:22 +0200 (CEST) Subject: Re: Data corruption with the mfi(4) driver Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20140710092251.GA1206@sekishi.zefyris.com> Date: Thu, 10 Jul 2014 12:19:21 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <3F26B4E0-2840-48AE-807B-FFFA4502DB83@sarenet.es> References: <20140710092251.GA1206@sekishi.zefyris.com> To: Francois Tigeot X-Mailer: Apple Mail (2.1283) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:25:08 -0000 On Jul 10, 2014, at 11:22 AM, Francois Tigeot wrote: > Hi, >=20 > Now, this particular server was running under DragonFly and not = FreeBSD but > the mfi(4) driver is mostly the same under both operating systems and = I have > found reports of data corruption issues with mfi(4) and recent LSI = adapters > in the archives of this list. I reported data corruption affecting one of the new SAS3 controllers on = an IBM server. = http://lists.freebsd.org/pipermail/freebsd-scsi/2014-February/006260.html In my case, it happened when using the "passthrough" support (called = "syspd" or "jbod" by LSI). It did not corrupt data for me when using it as a RAID card. Cheers, Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:36:34 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3CAE0D55 for ; Thu, 10 Jul 2014 10:36:34 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 00E8C2B4A for ; Thu, 10 Jul 2014 10:36:33 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 356DC20E7088B; Thu, 10 Jul 2014 10:36:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 39D6120E70886; Thu, 10 Jul 2014 10:36:29 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Borja Marcos" , "Francois Tigeot" References: <20140710092251.GA1206@sekishi.zefyris.com> <3F26B4E0-2840-48AE-807B-FFFA4502DB83@sarenet.es> Subject: Re: Data corruption with the mfi(4) driver Date: Thu, 10 Jul 2014 11:36:24 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:36:34 -0000 Your card is is based on 3108 chipset where as those lised in the dragonflybsd report appear to be 2208 based, which is the same as what we're running here without any issue, so I suspect the problems are unrelated. I'm not sure if dragonflybsd imported our mfi fixes so they could well be seeing the original issues. Regards Steve ----- Original Message ----- From: "Borja Marcos" To: "Francois Tigeot" Cc: "FreeBSD-scsi" Sent: Thursday, July 10, 2014 11:19 AM Subject: Re: Data corruption with the mfi(4) driver > > On Jul 10, 2014, at 11:22 AM, Francois Tigeot wrote: > >> Hi, >> >> Now, this particular server was running under DragonFly and not FreeBSD but >> the mfi(4) driver is mostly the same under both operating systems and I have >> found reports of data corruption issues with mfi(4) and recent LSI adapters >> in the archives of this list. > > I reported data corruption affecting one of the new SAS3 controllers on an IBM server. > > http://lists.freebsd.org/pipermail/freebsd-scsi/2014-February/006260.html > > In my case, it happened when using the "passthrough" support (called "syspd" or "jbod" by LSI). It did not corrupt > data for me when using it as a RAID card. > > Cheers, > > > > > > Borja. > > > > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:38:58 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2FEADDBC for ; Thu, 10 Jul 2014 10:38:58 +0000 (UTC) Received: from gw.zefyris.com (sabik.zefyris.com [IPv6:2001:7a8:3c67:2::254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B558B2B6A for ; Thu, 10 Jul 2014 10:38:57 +0000 (UTC) Received: from sekishi.zefyris.com (sekishi.zefyris.com [IPv6:2001:7a8:3c67:2::12]) by gw.zefyris.com (8.14.5/8.14.5) with ESMTP id s6AAcreR010577; Thu, 10 Jul 2014 12:38:53 +0200 (CEST) Date: Thu, 10 Jul 2014 12:38:53 +0200 From: Francois Tigeot To: Steven Hartland Subject: Re: Data corruption with the mfi(4) driver Message-ID: <20140710103853.GC1206@sekishi.zefyris.com> References: <20140710092251.GA1206@sekishi.zefyris.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gw.zefyris.com [IPv6:2001:7a8:3c67:2::254]); Thu, 10 Jul 2014 12:38:53 +0200 (CEST) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:38:58 -0000 On Thu, Jul 10, 2014 at 11:20:38AM +0100, Steven Hartland wrote: > I cant see any information on the actual corruption or cause in that linked > thread do you have any actual details? > > There was known corruption issues but these where fixed long ago so would > be good to confirm the details of what you where running and the HW when you > had the issue. > > As a point of reference we have mfi backed DB machines here and have not > had any issues with corruption and they have been in production for over > 1 1/2 years. It is only visible with recent adapters like the Thunderbolt serie, and then under relatively high disk load. The whole Dell Rx20 generation of servers seem to be impacted; the previous Rx10 generation is safe. This bug report contains additional details as well as PCI ids from two different Dell machines having experienced filesystem destruction: http://bugs.dragonflybsd.org/issues/2683 HAMMER CRC32 errors were reported on the console and the kernel eventually crashed after some time; I didn't get crash dumps. -- Francois Tigeot From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:53:09 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9B18AFFD for ; Thu, 10 Jul 2014 10:53:09 +0000 (UTC) Received: from gw.zefyris.com (sabik.zefyris.com [IPv6:2001:7a8:3c67:2::254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 30B562CCC for ; Thu, 10 Jul 2014 10:53:09 +0000 (UTC) Received: from sekishi.zefyris.com (sekishi.zefyris.com [IPv6:2001:7a8:3c67:2::12]) by gw.zefyris.com (8.14.5/8.14.5) with ESMTP id s6AAr7RJ011017; Thu, 10 Jul 2014 12:53:07 +0200 (CEST) Date: Thu, 10 Jul 2014 12:53:07 +0200 From: Francois Tigeot To: Steven Hartland Subject: Re: Data corruption with the mfi(4) driver Message-ID: <20140710105307.GD1206@sekishi.zefyris.com> References: <20140710092251.GA1206@sekishi.zefyris.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gw.zefyris.com [IPv6:2001:7a8:3c67:2::254]); Thu, 10 Jul 2014 12:53:07 +0200 (CEST) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:53:09 -0000 On Thu, Jul 10, 2014 at 11:20:38AM +0100, Steven Hartland wrote: > > There was known corruption issues but these where fixed long ago so would > be good to confirm the details of what you where running and the HW when you > had the issue. What changes would that be ? FreeBSD 10 is impacted and the last changes in the mfi driver I see are from September 2013 ("Add PCI device ID for MegaRAID Invader cards.") -- Francois Tigeot From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 10:57:20 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB038144 for ; Thu, 10 Jul 2014 10:57:20 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 7E9062CF6 for ; Thu, 10 Jul 2014 10:57:20 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id AD8CF20E7088B; Thu, 10 Jul 2014 10:57:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 5482020E70886; Thu, 10 Jul 2014 10:57:15 +0000 (UTC) Message-ID: <67A5107C31744FF69240D129BBF58880@multiplay.co.uk> From: "Steven Hartland" To: "Francois Tigeot" References: <20140710092251.GA1206@sekishi.zefyris.com> <20140710103853.GC1206@sekishi.zefyris.com> Subject: Re: Data corruption with the mfi(4) driver Date: Thu, 10 Jul 2014 11:57:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 10:57:20 -0000 ----- Original Message ----- From: "Francois Tigeot" > On Thu, Jul 10, 2014 at 11:20:38AM +0100, Steven Hartland wrote: >> I cant see any information on the actual corruption or cause in that linked >> thread do you have any actual details? >> >> There was known corruption issues but these where fixed long ago so would >> be good to confirm the details of what you where running and the HW when you >> had the issue. >> >> As a point of reference we have mfi backed DB machines here and have not >> had any issues with corruption and they have been in production for over >> 1 1/2 years. > > It is only visible with recent adapters like the Thunderbolt serie, and then > under relatively high disk load. > > The whole Dell Rx20 generation of servers seem to be impacted; the previous > Rx10 generation is safe. > > This bug report contains additional details as well as PCI ids from two > different Dell machines having experienced filesystem destruction: > http://bugs.dragonflybsd.org/issues/2683 > > HAMMER CRC32 errors were reported on the console and the kernel eventually > crashed after some time; I didn't get crash dumps. That PCI ids is 2208 based = Thunderbolt for which we have quite a few fixes. As I mentioned, I'm not sure if dragonflybsd have our mfi fixes for those cards, so would be good to confirm this along with confirming that the report was after said fixes. Boris is using Invader not Thunderbolt so could well be totally different issue. Also fw can play a key roll in issues like this, so without knowing why and hence having a fix its impossible to tell if the two reports are related at all. Regards Steve From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:01:02 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8061330E for ; Thu, 10 Jul 2014 11:01:02 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3BA9F2D8E for ; Thu, 10 Jul 2014 11:01:01 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id BCF239DCC82; Thu, 10 Jul 2014 13:00:59 +0200 (CEST) Subject: Re: Data corruption with the mfi(4) driver Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Borja Marcos X-Priority: 3 In-Reply-To: Date: Thu, 10 Jul 2014 13:00:58 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <09E1044A-90E1-4FEF-9F36-A86C5D8BA970@sarenet.es> References: <20140710092251.GA1206@sekishi.zefyris.com> <3F26B4E0-2840-48AE-807B-FFFA4502DB83@sarenet.es> To: Steven Hartland X-Mailer: Apple Mail (2.1283) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:01:02 -0000 On Jul 10, 2014, at 12:36 PM, Steven Hartland wrote: > Your card is is based on 3108 chipset where as those lised in the = dragonflybsd > report appear to be 2208 based, which is the same as what we're = running here > without any issue, so I suspect the problems are unrelated. I see. And in my case corruption did *not* happen at all using it as a = raid card, offering "mfi" devices to the system, so it must be = unrelated. The Dragonfly bug report doesn't contain any information = about how the card was being used, anyway, which could be useful. Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:12:48 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3318A4FD for ; Thu, 10 Jul 2014 11:12:48 +0000 (UTC) Received: from gw.zefyris.com (sabik.zefyris.com [IPv6:2001:7a8:3c67:2::254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB0FA2E6B for ; Thu, 10 Jul 2014 11:12:47 +0000 (UTC) Received: from sekishi.zefyris.com (sekishi.zefyris.com [IPv6:2001:7a8:3c67:2::12]) by gw.zefyris.com (8.14.5/8.14.5) with ESMTP id s6ABCjtW011577; Thu, 10 Jul 2014 13:12:45 +0200 (CEST) Date: Thu, 10 Jul 2014 13:12:45 +0200 From: Francois Tigeot To: Borja Marcos Subject: Re: Data corruption with the mfi(4) driver Message-ID: <20140710111245.GE1206@sekishi.zefyris.com> References: <20140710092251.GA1206@sekishi.zefyris.com> <3F26B4E0-2840-48AE-807B-FFFA4502DB83@sarenet.es> <09E1044A-90E1-4FEF-9F36-A86C5D8BA970@sarenet.es> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <09E1044A-90E1-4FEF-9F36-A86C5D8BA970@sarenet.es> User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gw.zefyris.com [IPv6:2001:7a8:3c67:2::254]); Thu, 10 Jul 2014 13:12:45 +0200 (CEST) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:12:48 -0000 On Thu, Jul 10, 2014 at 01:00:58PM +0200, Borja Marcos wrote: > > The Dragonfly bug report doesn't contain any information about how the card was being used, anyway, which could be useful. One machine was used as a PostgreSQL database server and another one as a NFS server. The Postgres one died during database imports and the NFS server while receiving files from another box via rsync. -- Francois Tigeot From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:14:41 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 049F054D for ; Thu, 10 Jul 2014 11:14:41 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id BBB352E7C for ; Thu, 10 Jul 2014 11:14:40 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 1CF0420E7088B; Thu, 10 Jul 2014 11:14:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id C1AD620E70886; Thu, 10 Jul 2014 11:14:35 +0000 (UTC) Message-ID: <6136FBA7A28542FFA681F59BEE4DE5DC@multiplay.co.uk> From: "Steven Hartland" To: "Borja Marcos" References: <20140710092251.GA1206@sekishi.zefyris.com> <3F26B4E0-2840-48AE-807B-FFFA4502DB83@sarenet.es> <09E1044A-90E1-4FEF-9F36-A86C5D8BA970@sarenet.es> Subject: Re: Data corruption with the mfi(4) driver Date: Thu, 10 Jul 2014 12:14:31 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:14:41 -0000 Yer I looked at dragonflybsd driver and it doesn't include our fixes so I think we need to treat it as a seperate issue at this time. Regards Steve ----- Original Message ----- From: "Borja Marcos" On Jul 10, 2014, at 12:36 PM, Steven Hartland wrote: > Your card is is based on 3108 chipset where as those lised in the dragonflybsd > report appear to be 2208 based, which is the same as what we're running here > without any issue, so I suspect the problems are unrelated. I see. And in my case corruption did *not* happen at all using it as a raid card, offering "mfi" devices to the system, so it must be unrelated. The Dragonfly bug report doesn't contain any information about how the card was being used, anyway, which could be useful. Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:24:33 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 99C6E997 for ; Thu, 10 Jul 2014 11:24:33 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 5D03F2FA4 for ; Thu, 10 Jul 2014 11:24:32 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id DF8DE20E7088B; Thu, 10 Jul 2014 11:24:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 9079820E70886; Thu, 10 Jul 2014 11:24:26 +0000 (UTC) Message-ID: <765694DA8A58468484623092946127D1@multiplay.co.uk> From: "Steven Hartland" To: "Francois Tigeot" References: <20140710092251.GA1206@sekishi.zefyris.com> <20140710105307.GD1206@sekishi.zefyris.com> Subject: Re: Data corruption with the mfi(4) driver Date: Thu, 10 Jul 2014 12:24:21 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:24:33 -0000 ----- Original Message ----- From: "Francois Tigeot" > On Thu, Jul 10, 2014 at 11:20:38AM +0100, Steven Hartland wrote: >> >> There was known corruption issues but these where fixed long ago so would >> be good to confirm the details of what you where running and the HW when you >> had the issue. > > What changes would that be ? > > FreeBSD 10 is impacted and the last changes in the mfi driver I see are from > September 2013 ("Add PCI device ID for MegaRAID Invader cards.") To be explicit FreeBSD 10.0-RELEASE would be uneffected by the corruption bugs we fixed, however there has been at least one fix / enhancement that I'm aware of since 10's release for later cards. The key thing to determine here are precise factors of the corruption your seeing as it looks like there is some mixing up of issues Given this could you confirm:- 1. Hardware 1.1. LSI card / generation 1.2. Disks 1.3. Hotswap adapter / expander 2. Firmware 3. OS Version / driver version Regards Steve From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:33:22 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A73A5A54 for ; Thu, 10 Jul 2014 11:33:22 +0000 (UTC) Received: from gw.zefyris.com (sabik.zefyris.com [IPv6:2001:7a8:3c67:2::254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 37BAB2072 for ; Thu, 10 Jul 2014 11:33:22 +0000 (UTC) Received: from sekishi.zefyris.com (sekishi.zefyris.com [IPv6:2001:7a8:3c67:2::12]) by gw.zefyris.com (8.14.5/8.14.5) with ESMTP id s6ABXJ20012141; Thu, 10 Jul 2014 13:33:19 +0200 (CEST) Date: Thu, 10 Jul 2014 13:33:19 +0200 From: Francois Tigeot To: Steven Hartland Subject: Re: Data corruption with the mfi(4) driver Message-ID: <20140710113315.GA3449@sekishi.zefyris.com> References: <20140710092251.GA1206@sekishi.zefyris.com> <20140710105307.GD1206@sekishi.zefyris.com> <765694DA8A58468484623092946127D1@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <765694DA8A58468484623092946127D1@multiplay.co.uk> User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gw.zefyris.com [IPv6:2001:7a8:3c67:2::254]); Thu, 10 Jul 2014 13:33:20 +0200 (CEST) Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:33:22 -0000 On Thu, Jul 10, 2014 at 12:24:21PM +0100, Steven Hartland wrote: > > The key thing to determine here are precise factors of the corruption your > seeing as it looks like there is some mixing up of issues My original mail was intended as a head's up notice; I don't have enough information to fill in every detail and point to a particular root cause so far. > Given this could you confirm:- > 1. Hardware > 1.1. LSI card / generation Dell Rx20 server generation I have personally seen the issue on a Dell R720xd with a LSI Dell Perc H710 adapter. PCI Id: mfi0@pci0:3:0:0: class=0x010400 card=0x1f341028 chip=0x005b1000 rev=0x05 hdr=0x00 but have had private reports from at least two different machines having the same problem. All other parameters are variable. -- Francois Tigeot From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:50:46 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 719B6D5E for ; Thu, 10 Jul 2014 11:50:46 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 34A782181 for ; Thu, 10 Jul 2014 11:50:45 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 48E0320E7088B; Thu, 10 Jul 2014 11:50:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id D529420E70886; Thu, 10 Jul 2014 11:50:40 +0000 (UTC) Message-ID: <803159D4D0C44F879D21B6E0ECFE391F@multiplay.co.uk> From: "Steven Hartland" To: "Francois Tigeot" References: <20140710092251.GA1206@sekishi.zefyris.com> <20140710105307.GD1206@sekishi.zefyris.com> <765694DA8A58468484623092946127D1@multiplay.co.uk> <20140710113315.GA3449@sekishi.zefyris.com> Subject: Re: Data corruption with the mfi(4) driver Date: Thu, 10 Jul 2014 12:50:36 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:50:46 -0000 ----- Original Message ----- From: "Francois Tigeot" > On Thu, Jul 10, 2014 at 12:24:21PM +0100, Steven Hartland wrote: >> >> The key thing to determine here are precise factors of the corruption your >> seeing as it looks like there is some mixing up of issues > > My original mail was intended as a head's up notice; I don't have enough > information to fill in every detail and point to a particular root cause so > far. > >> Given this could you confirm:- >> 1. Hardware >> 1.1. LSI card / generation > > Dell Rx20 server generation > > I have personally seen the issue on a Dell R720xd with a LSI Dell Perc H710 > adapter. PCI Id: > mfi0@pci0:3:0:0: class=0x010400 card=0x1f341028 chip=0x005b1000 rev=0x05 > hdr=0x00 but have had private reports from at least two different machines > having the same problem. Perc H710 is 2208 based. > All other parameters are variable. They may be variable but need a point of reference, so detailing the examples is key. Regards Steve From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 11:58:34 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A6BFFB2 for ; Thu, 10 Jul 2014 11:58:34 +0000 (UTC) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BC9842263 for ; Thu, 10 Jul 2014 11:58:33 +0000 (UTC) Received: from th-04.cs.huji.ac.il ([132.65.80.125]) by kabab.cs.huji.ac.il with esmtp id 1X5CzX-00044Q-GH; Thu, 10 Jul 2014 14:58:23 +0300 Subject: Re: Data corruption with the mfi(4) driver Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Daniel Braniss X-Priority: 3 In-Reply-To: <803159D4D0C44F879D21B6E0ECFE391F@multiplay.co.uk> Date: Thu, 10 Jul 2014 14:58:22 +0300 Message-Id: References: <20140710092251.GA1206@sekishi.zefyris.com> <20140710105307.GD1206@sekishi.zefyris.com> <765694DA8A58468484623092946127D1@multiplay.co.uk> <20140710113315.GA3449@sekishi.zefyris.com> <803159D4D0C44F879D21B6E0ECFE391F@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 11:58:34 -0000 On Jul 10, 2014, at 2:50 PM, Steven Hartland = wrote: > ----- Original Message ----- From: "Francois Tigeot" = >=20 >=20 >> On Thu, Jul 10, 2014 at 12:24:21PM +0100, Steven Hartland wrote: >>> The key thing to determine here are precise factors of the = corruption your >>> seeing as it looks like there is some mixing up of issues >> My original mail was intended as a head's up notice; I don't have = enough >> information to fill in every detail and point to a particular root = cause so >> far. >>> Given this could you confirm:- >>> 1. Hardware >>> 1.1. LSI card / generation >> Dell Rx20 server generation >> I have personally seen the issue on a Dell R720xd with a LSI Dell = Perc H710 >> adapter. PCI Id: >> mfi0@pci0:3:0:0: class=3D0x010400 card=3D0x1f341028 chip=3D0x005b1000 = rev=3D0x05 >> hdr=3D0x00 but have had private reports from at least two different = machines >> having the same problem. >=20 > Perc H710 is 2208 based. >=20 >> All other parameters are variable. >=20 > They may be variable but need a point of reference, so detailing the = examples > is key. we have several machines with said PERCs, used as ZFS/NFS for postgres = to large dna sequences and have yet to receive complains of data corruption, this does not mean = the absence of them :-), but considering the amount of storage/servers some user should have = complained by now =85 so, is someone else in FreeBSD 9.2/3 seeing such corruptions? BTW, the only issue we had was when the server was connected at 10G we = did have a lot of data corruption which was solved by turning TSO off. danny >=20 > Regards > Steve > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to = "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 12:03:37 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2952759D for ; Thu, 10 Jul 2014 12:03:37 +0000 (UTC) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id B84DA233D for ; Thu, 10 Jul 2014 12:03:36 +0000 (UTC) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.5/8.14.5) with ESMTP id s6AC3Rw7054356; Thu, 10 Jul 2014 13:03:27 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id s6AC3QZY025968; Thu, 10 Jul 2014 13:03:26 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id s6AC3QYo025964; Thu, 10 Jul 2014 13:03:26 +0100 Date: Thu, 10 Jul 2014 13:03:26 +0100 Message-Id: <201407101203.s6AC3QYo025964@higson.cam.lispworks.com> From: Martin Simmons To: Jim Harris In-reply-to: (message from Jim Harris on Wed, 9 Jul 2014 09:44:54 -0700) Subject: Re: [Bug 191717] [iscsi] smartctl -H gives "ATA output registers missing" for a disk using the isci driver References: <53BB619B.4060908@interlog.com> <201407081145.s68Bjxn6006452@higson.cam.lispworks.com> <53BC2DC5.60005@interlog.com> Cc: freebsd-scsi@freebsd.org, samm@os2.kiev.ua, Christian.Franke@t-online.de X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 12:03:37 -0000 >>>>> On Wed, 9 Jul 2014 09:44:54 -0700, Jim Harris said: > > I have looked a little at the smartctl code, and it does not look like it > supports fixed format sense data - at least for ATA PASSTHROUGH. It > expects additional sense length to be non-zero. smartctl could be fixed, > but I think for compatibility, I should have isci return descriptor sense > for ATA PASSTHROUGH commands instead. > > Martin - could you please test the attached patch? This passes your test > case using smartctl 6.3 r3939. Thanks, that fixes my setup too with smartctl 6.2 r3874 from ports. __Martin From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 12:07:00 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A923E664 for ; Thu, 10 Jul 2014 12:07:00 +0000 (UTC) Received: from exprod7og111.obsmtp.com (exprod7og111.obsmtp.com [64.18.2.175]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 11B5E2363 for ; Thu, 10 Jul 2014 12:06:59 +0000 (UTC) Received: from mail-lb0-f179.google.com ([209.85.217.179]) (using TLSv1) by exprod7ob111.postini.com ([64.18.6.12]) with SMTP ID DSNKU76B3L7x+dqnn4gx0vOpc+4C0Obfpm9e@postini.com; Thu, 10 Jul 2014 05:07:00 PDT Received: by mail-lb0-f179.google.com with SMTP id z11so5899054lbi.24 for ; Thu, 10 Jul 2014 05:06:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:mime-version:thread-index:date:message-id :subject:to:cc:content-type; bh=DEQym0ubhC8xaE8zQ8BrO6xRWl7ztdizgMaxIyr40AQ=; b=blKnmYP8RP4CvX2BRzicHk2UdUqDPNyAQ0WEv1H9BCR0upA29ZgN2+5yLWyVuFvwjb 4fD/RkQEwd+HBpOCwttjwzlMBDvIe9FWY7p7EQyLi/fz2y9+AaZ+N0YOmY7VeYjmP2SK dqU0ocYKc742N6xggOPqf/nRFGvOi+IR8P0UFtJPM1FjJvNW2wZdCRFidBl6/OEVLPAZ vGw+W2G6J3T/ttKsXAtiSwMY8sLeXHetX0iIy9I7piBvXkMMpAO2nyQoEOXt74Gxjn/5 F84K4NUsuWIewNGnnfbolBixRVD9YFV2JBc2oywzvKajFz7qwtfCCATgltQ6630+HoCT NdBQ== X-Received: by 10.112.16.230 with SMTP id j6mr513036lbd.90.1404993656045; Thu, 10 Jul 2014 05:00:56 -0700 (PDT) X-Gm-Message-State: ALoCoQmVVIYr4qU0G3HOzms3gXBPy2oBBb21lF4ND4t8pccno321N7BzIDr6G2widl/cYfn01z6/ymP35pn5tjsx5TEr/Uhy6yqj+QnB2KNajdPdYG0OAheAmE9oIbaFnLWP2YQE5vFT9OCi40I8bQseYGRMfbokzw== X-Received: by 10.112.16.230 with SMTP id j6mr513009lbd.90.1404993655841; Thu, 10 Jul 2014 05:00:55 -0700 (PDT) From: Kashyap Desai MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac+cNplH5DNJUsQbTdigr3yA9M6iRw== Date: Thu, 10 Jul 2014 17:30:54 +0530 Message-ID: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> Subject: SSDs peformance on head/freebsd-10 stable using FIO To: mav@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 12:07:00 -0000 Hi Motin, I am trying to collect IOPs and throughput using FIO on FreeBSD-10-stable as below post mentioned that CAM can reach upto 1,000,000 IOPS using Fine-Grained CAM locking. http://www.freebsd.org/news/status/report-2013-07-2013-09.html#GEOM-Direct-= Dispatch-and-Fine-Grained-CAM-Locking I am using below FIO parameter. [global] ioengine=3Dposixaio buffered=3D0 rw=3Drandread bs=3D4K iodepth=3D32 numjobs=3D2 direct=3D1 runtime=3D60s thread group_reporting=3D1 [job1] filename=3D/dev/da0 [job2] filename=3D/dev/da1 [job3] filename=3D/dev/da2 [job4] filename=3D/dev/da3 [job4] filename=3D/dev/da4 .. I have 8 SSDs in my setup and all 8 SSDs are behind LSI=E2=80=99s 12Gp/s Me= gaRaid Controller as JBOD. I also found FIO can be used in Async mode after loading =E2=80=9Caio=E2=80=9D kernel module. Using single SSD, I am able to see 110K-130K IOPs. This IOPs counts are matching with what I see on Linux machine. Now, I am not able to scale IOPs on my machine after 200K. I see CPU is almost occupied and no idle time after IOPs reach to 200K. If you have any pointers to try with, I can do some experiment on my setup= . Thanks, Kashyap From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 12:31:05 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D7E35FF5 for ; Thu, 10 Jul 2014 12:31:05 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E65D255F for ; Thu, 10 Jul 2014 12:31:05 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id hi2so4439411wib.7 for ; Thu, 10 Jul 2014 05:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=KBUCDl8GqdzX7EAaa4Od9BmyV6fmtrqx9OLJK9X7H10=; b=wdn8XKITPyGjjcivp7zIFsMnN29hGIXduosziz2i8O60vDeQX4clj5yRj3VCMy28z9 rcJJR+zHp/4fuj80m/nmd//8qpLanfd1EB4Dd97QAd27AZsjJNjhSsFZicujSIHzHsMU 9r5xccxpNeFMN/3guZJfPIyRCetyB/T4HyHmIadhO5AJuIRcr1k78c2X7omFYlrou5ar zXsaZdPFRbMrTjdmveaEcCOGrmEcXhJdKhKrcZXGk8E1xbqO3XYpiq4k3J5Z5v9KOCUc NtMMyijPC/aPVQEJcSgZdNU9wjVzPZdfKfv8ifaINRE8mrpUDMnUTMYpqoFg9hOIhGLk KkQw== X-Received: by 10.194.89.168 with SMTP id bp8mr56412036wjb.73.1404995463593; Thu, 10 Jul 2014 05:31:03 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id ft17sm110111803wjc.14.2014.07.10.05.31.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 10 Jul 2014 05:31:02 -0700 (PDT) Sender: Alexander Motin Message-ID: <53BE8784.8060503@FreeBSD.org> Date: Thu, 10 Jul 2014 15:31:00 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Kashyap Desai Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO References: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> In-Reply-To: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 12:31:06 -0000 Hi, Kashyap. On 10.07.2014 15:00, Kashyap Desai wrote: > I am trying to collect IOPs and throughput using FIO on FreeBSD-10-stable > as below post mentioned that CAM can reach upto 1,000,000 IOPS using > Fine-Grained CAM locking. > > http://www.freebsd.org/news/status/report-2013-07-2013-09.html#GEOM-Direct-Dispatch-and-Fine-Grained-CAM-Locking > > I am using below FIO parameter. > > [global] > ioengine=posixaio > buffered=0 > rw=randread > bs=4K > iodepth=32 > numjobs=2 > direct=1 > runtime=60s > thread > group_reporting=1 > [job1] > filename=/dev/da0 > [job2] > filename=/dev/da1 > [job3] > filename=/dev/da2 > [job4] > filename=/dev/da3 > [job4] > filename=/dev/da4 > .. > > I have 8 SSDs in my setup and all 8 SSDs are behind LSI’s 12Gp/s MegaRaid > Controller as JBOD. I also found FIO can be used in Async mode after > loading “aio” kernel module. > > Using single SSD, I am able to see 110K-130K IOPs. This IOPs counts are > matching with what I see on Linux machine. > > Now, I am not able to scale IOPs on my machine after 200K. I see CPU is > almost occupied and no idle time after IOPs reach to 200K. > > If you have any pointers to try with, I can do some experiment on my setup. Getting such results I would immediately start doing profiling with pmcstat. Quite likely you are hitting some new lock congestion. Start with simple `pmcstat -n 100000000 -TS unhalted-cycles`. It it hard to say for sure what went wrong there without more data, so just couple thoughts: First of all, I've never tried aio in my benchmarks, only synchronous ones. Try to run 8 instances of `dd if=/dev/daX of=/dev/null bs=512` per each SSD same time, just as I did. You may vary number of dd's, but keep total below 256, or you mad to increase nswbuf limit in kern_vfs_bio_buffer_alloc(). For second, you are using single HBA, that should create significant congestion around its CAM SIM lock. Proper solution would be to add multiple queues support to the driver, and we discussed it with Scott Long for quite some time, but that requires more work (I hope you may be interested in it ;) ). Or you may just insert 3-4 HBAs. My million IOPS I was reaching with four 2008/2308 6Gbps HBAs and 16 SATA SSDs. Please tell me what you get, so we could continue investigation. -- Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 13:36:15 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BF758CFA for ; Thu, 10 Jul 2014 13:36:15 +0000 (UTC) Received: from exprod7og127.obsmtp.com (exprod7og127.obsmtp.com [64.18.2.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C1A442BB8 for ; Thu, 10 Jul 2014 13:36:13 +0000 (UTC) Received: from mail-lb0-f177.google.com ([209.85.217.177]) (using TLSv1) by exprod7ob127.postini.com ([64.18.6.12]) with SMTP ID DSNKU76WzBbhyFQrOPOP+J55ZF0H8XEkWey+@postini.com; Thu, 10 Jul 2014 06:36:13 PDT Received: by mail-lb0-f177.google.com with SMTP id u10so6017290lbd.22 for ; Thu, 10 Jul 2014 06:36:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type; bh=ijF6veRfhlWLgfxoy/zTG8Yd1bHfSMujRtwrop5QVlI=; b=Uw6Ul4STaP9FA/mM5BOLCKZKMGyw6THBW6eTHEoDdR0VykX0dbbck6rSQov1zkFJTo q392XjQ/THXrN5fLnfvOA+7s2pZVJfKU7/Hd/wyCAWorz95ikt9nGc9yc8Lv4PgQeSyM mWSJ6w+VOFMbpWUtSpexfWa3NKJOrHciFxw/ORvXu8EHriAtAf205KRYFhTJYxYCOzrf mF8UD+XQMdzEcro309BTlJsLK8zE3OQPA1/9R2VcIz+TqS2ucSKqMNjnjwlvP8grcw0S t6aVVUTXy7Mlx36zF7ohKYtLaoAfpHIsQoGY34oRIjajpWuO1NMimfTu1bVaXIb/SwLV hAYA== X-Received: by 10.112.19.133 with SMTP id f5mr4836006lbe.48.1404998931171; Thu, 10 Jul 2014 06:28:51 -0700 (PDT) X-Gm-Message-State: ALoCoQk3UVREL13Ip1B/a/xVp2WUsJwodVx7OQEAXzoSKC/gj1H4esI/28mlZTTWFdqiutmMj1tSqiNZ2pFwwnDcvqizKIBaL5lpODWsTRvpsfggEmEiVWF4tcZ4IBo8BeocpqBQ1cgTV30223hX1pnbkftr8Lsfrg== X-Received: by 10.112.19.133 with SMTP id f5mr4835972lbe.48.1404998930903; Thu, 10 Jul 2014 06:28:50 -0700 (PDT) From: Kashyap Desai References: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> <53BE8784.8060503@FreeBSD.org> In-Reply-To: <53BE8784.8060503@FreeBSD.org> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQG8AScvFWC5/cNVmZuoGGC8NZGkUAIYsfkbm6+53VA= Date: Thu, 10 Jul 2014 18:58:47 +0530 Message-ID: <9f138f242e278476e5c542d695e58bc8@mail.gmail.com> Subject: RE: SSDs peformance on head/freebsd-10 stable using FIO To: Alexander Motin Content-Type: multipart/mixed; boundary=14dae93d8c8631ddcf04fdd6ce99 X-Mailman-Approved-At: Thu, 10 Jul 2014 13:45:06 +0000 Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 13:36:15 -0000 --14dae93d8c8631ddcf04fdd6ce99 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > -----Original Message----- > From: Alexander Motin [mailto:mavbsd@gmail.com] On Behalf Of Alexander > Motin > Sent: Thursday, July 10, 2014 6:01 PM > To: Kashyap Desai > Cc: FreeBSD-scsi > Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO > > Hi, Kashyap. > > On 10.07.2014 15:00, Kashyap Desai wrote: > > I am trying to collect IOPs and throughput using FIO on > > FreeBSD-10-stable as below post mentioned that CAM can reach upto > > 1,000,000 IOPS using Fine-Grained CAM locking. > > > > http://www.freebsd.org/news/status/report-2013-07-2013- > 09.html#GEOM-Di > > rect-Dispatch-and-Fine-Grained-CAM-Locking > > > > I am using below FIO parameter. > > > > [global] > > ioengine=3Dposixaio > > buffered=3D0 > > rw=3Drandread > > bs=3D4K > > iodepth=3D32 > > numjobs=3D2 > > direct=3D1 > > runtime=3D60s > > thread > > group_reporting=3D1 > > [job1] > > filename=3D/dev/da0 > > [job2] > > filename=3D/dev/da1 > > [job3] > > filename=3D/dev/da2 > > [job4] > > filename=3D/dev/da3 > > [job4] > > filename=3D/dev/da4 > > .. > > > > I have 8 SSDs in my setup and all 8 SSDs are behind LSI=E2=80=99s 12Gp/= s > > MegaRaid Controller as JBOD. I also found FIO can be used in Async > > mode after loading =E2=80=9Caio=E2=80=9D kernel module. > > > > Using single SSD, I am able to see 110K-130K IOPs. This IOPs counts > > are matching with what I see on Linux machine. > > > > Now, I am not able to scale IOPs on my machine after 200K. I see CPU > > is almost occupied and no idle time after IOPs reach to 200K. > > > > If you have any pointers to try with, I can do some experiment on my > setup. > > Getting such results I would immediately start doing profiling with > pmcstat. > Quite likely you are hitting some new lock congestion. Start with simple > `pmcstat -n 100000000 -TS unhalted-cycles`. It it hard to say for sure > what > went wrong there without more data, so just couple I have attached profile output for the command mentioned above. I will dig further and see if this is what we have theoretical limit for CAM attached HBA. I am trying to isolate tools, tuning or Driver/FW issue at very first level= . > thoughts: > > First of all, I've never tried aio in my benchmarks, only synchronous > ones. Try > to run 8 instances of `dd if=3D/dev/daX of=3D/dev/null bs=3D512` per each= SSD > same time, just as I did. You may vary number of dd's, but keep total > below > 256, or you mad to increase nswbuf limit in kern_vfs_bio_buffer_alloc(). I ran multiple dd instance also and seeing IOPs throttle somewhere ~200K . Do we have any mechanism to check CAM layer's max IOPs support without involving actual Device ? Something like _null_ device driver which just send the command back to CAM layer ? > > For second, you are using single HBA, that should create significant > congestion around its CAM SIM lock. Proper solution would be to add > multiple queues support to the driver, and we discussed it with Scott Lon= g > for quite some time, but that requires more work (I hope you may be > interested in it ;) ). Or you may just insert 3-4 HBAs. My million IOPS I > was > reaching with four 2008/2308 6Gbps HBAs and 16 SATA SSDs. I remember this part and really good to contribute for this work. As part of this we have initiated multiple MSIx implementation in , which will have multiple reply queue per MSI-x. Do we really require to have multiple Submission queue at low level driver = ? I thought it will be a CAM interface for multi queue which _all_ low level drivers need to hook into . I just started gathering performance numbers on FreeBSD with more SSDs on LSI's 12Gbp/s card, so just getting familiar with tools and tunings. Earlier I used just one SSDs and some HDDs so no issue found w.r.t IOPs etc.. As part of this activity, I am trying to see if mrsas driver is able to meet expected performance without considering any other component as bottleneck. I mean, are we able to meet IOPs which CAM layer can handle. Kashyap > > Please tell me what you get, so we could continue investigation. > > -- > Alexander Motin --14dae93d8c8631ddcf04fdd6ce99 Content-Type: application/octet-stream; name="profile.graph" Content-Disposition: attachment; filename="profile.graph" Content-Transfer-Encoding: base64 X-Attachment-Id: 18d29643db74189c_0.1 QCBDUFVfQ0xLX1VOSEFMVEVEX0NPUkUgWzI5MSBzYW1wbGVzXQoKMTIuNzElICBbMzddICAgICAg IGNwdV9pZGxlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMzddICAgICAgICBzY2hl ZF9pZGxldGQKICAxMDAuMCUgIFszN10gICAgICAgICBmb3JrX2V4aXQKCjA5Ljk3JSAgWzI5XSAg ICAgICBYaW52bHJuZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA2Ljg3JSAgWzIwXSAgICAgICBp bnZscm5nX2hhbmRsZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowNS4xNSUgIFsxNV0gICAgICAg c21wX3RsYl9zaG9vdGRvd24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxNV0gICAg ICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzE1XSAgICAgICAgIGJpb2RvbmUK ICAgMTAwLjAlICBbMTVdICAgICAgICAgIGRhZG9uZQoKMDQuNDclICBbMTNdICAgICAgIGNwdV9z ZWFyY2hfaGlnaGVzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDg0LjYyJSAgWzExXSAgICAgICAg Y3B1X3NlYXJjaF9oaWdoZXN0CiAgNTQuNTUlICBbNl0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAg IDEwMC4wJSAgWzZdICAgICAgICAgICBmb3JrX2V4aXQKICA0NS40NSUgIFs1XSAgICAgICAgICBj cHVfc2VhcmNoX2hpZ2hlc3QKICAgMTAwLjAlICBbNV0gICAgICAgICAgIHNjaGVkX2lkbGV0ZAog MTUuMzglICBbMl0gICAgICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFsyXSAgICAgICAgICBm b3JrX2V4aXQKCjA0LjEyJSAgWzEyXSAgICAgICBfX210eF9sb2NrX3NsZWVwIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogNzUuMDAlICBbOV0gICAgICAgICBfX210eF9sb2NrX2ZsYWdzCiAgMTAwLjAl ICBbOV0gICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAg MTAwLjAlICBbOV0gICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogMDguMzMlICBbMV0gICAg ICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMV0gICAg ICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0YXJ0 CiAwOC4zMyUgIFsxXSAgICAgICAgIGRhc3RhcnQKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRf cnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdHJhdGVneQogMDguMzMlICBb MV0gICAgICAgICB2bWVtX3hhbGxvYwogIDEwMC4wJSAgWzFdICAgICAgICAgIHZtZW1fYWxsb2MK ICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfaW9fY2hlY2sKCjAyLjQxJSAgWzddICAgICAgICBj cHVfc2VhcmNoX2xvd2VzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzddICAgICAg ICAgY3B1X3NlYXJjaF9sb3dlc3QKICA3MS40MyUgIFs1XSAgICAgICAgICBjcHVfc2VhcmNoX2xv d2VzdAogICAxMDAuMCUgIFs1XSAgICAgICAgICAgc2NoZWRfcGlja2NwdQogIDI4LjU3JSAgWzJd ICAgICAgICAgIHNjaGVkX3BpY2tjcHUKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHNjaGVkX2Fk ZAoKMDIuNDElICBbN10gICAgICAgIGNwdV9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFs3XSAgICAgICAgIG1pX3N3aXRjaAogIDcxLjQzJSAgWzVdICAgICAgICAgIHNjaGVk X2lkbGV0ZAogICAxMDAuMCUgIFs1XSAgICAgICAgICAgZm9ya19leGl0CiAgMTQuMjklICBbMV0g ICAgICAgICAgY3JpdGljYWxfZXhpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVu dF9oYW5kbGUKICAxNC4yOSUgIFsxXSAgICAgICAgICBpdGhyZWFkX2xvb3AKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDEuNzIlICBbNV0gICAgICAgIHNjaGVkX2lkbGV0ZCBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzVdICAgICAgICAgZm9ya19leGl0CgowMS43 MiUgIFs1XSAgICAgICAgX210eF9sb2NrX3NwaW5fY29va2llIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogNDAuMDAlICBbMl0gICAgICAgICB0dXJuc3RpbGVfdHJ5d2FpdAogIDEwMC4wJSAgWzJdICAg ICAgICAgIF9fbXR4X2xvY2tfc2xlZXAKICAgMTAwLjAlICBbMl0gICAgICAgICAgIF9fbXR4X2xv Y2tfZmxhZ3MKIDIwLjAwJSAgWzFdICAgICAgICAgdGRxX2xvY2tfcGFpcgogIDEwMC4wJSAgWzFd ICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0 CiAyMC4wMCUgIFsxXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbMV0gICAg ICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiaW9k b25lCiAyMC4wMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFdICAgICAgICAg IHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0Cgow MS4zNyUgIFs0XSAgICAgICAgc2NoZWRfc3dpdGNoIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbNF0gICAgICAgICBtaV9zd2l0Y2gKICA3NS4wMCUgIFszXSAgICAgICAgICBpdGhyZWFk X2xvb3AKICAgMTAwLjAlICBbM10gICAgICAgICAgIGZvcmtfZXhpdAogIDI1LjAwJSAgWzFdICAg ICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0Cgow MS4zNyUgIFs0XSAgICAgICAgeHB0X2RvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFs0XSAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAw LjAlICBbNF0gICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzRdICAgICAg ICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCgow MS4zNyUgIFs0XSAgICAgICAgWGFwaWNfaXNyMSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjM3 JSAgWzRdICAgICAgICBnX2lvX2NoZWNrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb NF0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFs0XSAgICAgICAgICBkZXZfc3RyYXRl Z3lfY3N3CiAgIDEwMC4wJSAgWzRdICAgICAgICAgICBwaHlzaW8KCjAxLjM3JSAgWzRdICAgICAg ICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzRdICAgICAgICAg dm1lbV9hbGxvYwogIDEwMC4wJSAgWzRdICAgICAgICAgIGdfaW9fY2hlY2sKICAgMTAwLjAlICBb NF0gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDEuMzclICBbNF0gICAgICAgIGRhc3RhcnQgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAg MTAwLjAlICBbNF0gICAgICAgICAgZGFzdHJhdGVneQogICAxMDAuMCUgIFs0XSAgICAgICAgICAg Z19kaXNrX3N0YXJ0CgowMS4zNyUgIFs0XSAgICAgICAgc3BpbmxvY2tfZXhpdCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDUwLjAwJSAgWzJdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBb Ml0gICAgICAgICAgZm9ya19leGl0CiAyNS4wMCUgIFsxXSAgICAgICAgIHdha2V1cAogIDEwMC4w JSAgWzFdICAgICAgICAgIGJkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBidWZkb25lCiAy NS4wMCUgIFsxXSAgICAgICAgIHZtZW1feGZyZWUKICAxMDAuMCUgIFsxXSAgICAgICAgICBiaW9k b25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAxLjM3JSAgWzRdICAgICAgICBt cnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzRdICAg ICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog IDEwMC4wJSAgWzRdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFs0XSAgICAgICAg ICAgZm9ya19leGl0CgowMS4zNyUgIFs0XSAgICAgICAgYW1kNjRfc3lzY2FsbCBAIC9ib290L2tl cm5lbC9rZXJuZWwKCjAxLjAzJSAgWzNdICAgICAgICB2bV9mYXVsdF9xdWlja19ob2xkX3BhZ2Vz IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB2bWFwYnVmCiAgMTAw LjAlICBbM10gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBkZXZmc19y ZWFkX2YKCjAxLjAzJSAgWzNdICAgICAgICBpdGhyZWFkX2xvb3AgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFszXSAgICAgICAgIGZvcmtfZXhpdAoKMDEuMDMlICBbM10gICAgICAgIGNy aXRpY2FsX2VudGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMzMuMzMlICBbMV0gICAgICAgICB0 aHJlYWRfbG9ja19mbGFnc18KICAxMDAuMCUgIFsxXSAgICAgICAgICBjcml0aWNhbF9leGl0CiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogMzMuMzMlICBbMV0gICAg ICAgICB1bWFfemZyZWVfYXJnCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kZXZfZG9uZQogICAx MDAuMCUgIFsxXSAgICAgICAgICAgZ19pb19kZWxpdmVyCiAzMy4zMyUgIFsxXSAgICAgICAgIHNt cF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbMV0gICAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3Jh bmdlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiaW9kb25lCgowMS4wMyUgIFszXSAgICAgICAg c2xlZXBxX2Jyb2FkY2FzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAg ICAgd2FrZXVwCiAgMTAwLjAlICBbM10gICAgICAgICAgYmRvbmUKICAgMTAwLjAlICBbM10gICAg ICAgICAgIGJ1ZmRvbmUKCjAxLjAzJSAgWzNdICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2Ug QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIGJpb2RvbmUKICAxMDAu MCUgIFszXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbM10gICAgICAgICAgIHhwdF9kb25l X3Byb2Nlc3MKCjAxLjAzJSAgWzNdICAgICAgICB4cHRfZG9uZV9wcm9jZXNzIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB4cHRfZG9uZV90ZAogIDEwMC4wJSAgWzNd ICAgICAgICAgIGZvcmtfZXhpdAoKMDEuMDMlICBbM10gICAgICAgIHJlbHBidWYgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIHBoeXNpbwogIDEwMC4wJSAgWzNdICAg ICAgICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUgIFszXSAgICAgICAgICAgZG9maWxlcmVhZAoK MDEuMDMlICBbM10gICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzNdICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgMTAwLjAlICBbM10gICAgICAg ICAgZGFzdGFydAogICAxMDAuMCUgIFszXSAgICAgICAgICAgeHB0X3J1bl9hbGxvY3EKCjAxLjAz JSAgWzNdICAgICAgICBfX210eF91bmxvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA2 Ni42NyUgIFsyXSAgICAgICAgIHhwdF9ydW5fZGV2cQogIDEwMC4wJSAgWzJdICAgICAgICAgIHhw dF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZGFzdGFydAogMzMuMzMl ICBbMV0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEw MC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoK MDAuNjklICBbMl0gICAgICAgIHVtYV96ZnJlZV9hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1 MC4wMCUgIFsxXSAgICAgICAgIGdfZGV2X2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2lv X2RlbGl2ZXIKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQogNTAu MDAlICBbMV0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2Rp c2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDAuNjklICBb Ml0gICAgICAgIGdldHBidWYgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAg ICAgIHBoeXNpbwogIDEwMC4wJSAgWzJdICAgICAgICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUg IFsyXSAgICAgICAgICAgZG9maWxlcmVhZAoKMDAuNjklICBbMl0gICAgICAgIHhwdF9hY3Rpb25f ZGVmYXVsdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZGFzdGFy dAogIDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzJdICAg ICAgICAgICBkYXN0cmF0ZWd5CgowMC42OSUgIFsyXSAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jv b3Qva2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsyXSAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9i b290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsyXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1 bHQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRhc3RhcnQKCjAwLjY5JSAgWzJdICAgICAgICBz bGVlcHFfbG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgd2Fr ZXVwCiAgMTAwLjAlICBbMV0gICAgICAgICAgYmRvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAg IGJ1ZmRvbmUKIDUwLjAwJSAgWzFdICAgICAgICAgY3ZfYnJvYWRjYXN0cHJpCiAgMTAwLjAlICBb MV0gICAgICAgICAgdm1lbV94ZnJlZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYmlvZG9uZQoK MDAuNjklICBbMl0gICAgICAgIGRvcmV0aSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjY5JSAg WzJdICAgICAgICB4cHRfcnVuX2FsbG9jcSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgZGFzdHJhdGVneQogIDEwMC4wJSAgWzJdICAgICAgICAgIGdfZGlza19zdGFy dAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZ19pb19yZXF1ZXN0CgowMC42OSUgIFsyXSAgICAg ICAgYndhaXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIHBoeXNp bwogIDEwMC4wJSAgWzJdICAgICAgICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgZG9maWxlcmVhZAoKMDAuNjklICBbMl0gICAgICAgIGxvY2tfbXR4IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsyXSAgICAg ICAgICBid2FpdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgcGh5c2lvCgowMC42OSUgIFsyXSAg ICAgICAgX19zeXNfcmVhZCBAIC9saWIvbGliYy5zby43CgowMC42OSUgIFsyXSAgICAgICAgY3B1 X3NldF9zeXNjYWxsX3JldHZhbCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAg ICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuNjklICBbMl0gICAgICAgIHZtZW1feGZyZWUgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFsy XSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nl c3MKCjAwLjY5JSAgWzJdICAgICAgICB0aHJlYWRfbG9ja19mbGFnc18gQCAvYm9vdC9rZXJuZWwv a2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIHNsZWVwcV9hZGQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ3YWl0CiA1MC4wMCUgIFsxXSAg ICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAu NjklICBbMl0gICAgICAgIHNjaGVkX2Nob29zZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzJdICAgICAgICAgY2hvb3NldGhyZWFkCiAgMTAwLjAlICBbMl0gICAgICAgICAgc2NoZWRf c3dpdGNoCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBtaV9zd2l0Y2gKCjAwLjY5JSAgWzJdICAg ICAgICBwbWFwX2V4dHJhY3RfYW5kX2hvbGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsyXSAgICAgICAgIHZtX2ZhdWx0X3F1aWNrX2hvbGRfcGFnZXMKICAxMDAuMCUgIFsyXSAgICAg ICAgICB2bWFwYnVmCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBwaHlzaW8KCjAwLjY5JSAgWzJd ICAgICAgICBsYXBpY19pcGlfdmVjdG9yZWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsyXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbMl0gICAgICAgICAgcG1h cF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBiaW9kb25lCgowMC42 OSUgIFsyXSAgICAgICAgX19tdHhfbG9ja19mbGFncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUw LjAwJSAgWzFdICAgICAgICAgbXJzYXNfdW5tYXBfcmVxdWVzdCBAIC9ib290L2tlcm5lbC9tcnNh cy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2NtZF9kb25lCiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNf bWFwX3JlcXVlc3QKICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19idWlsZF9kY2RiCiAgIDEw MC4wJSAgWzFdICAgICAgICAgICBtcnNhc19hY3Rpb24KCjAwLjY5JSAgWzJdICAgICAgICB0ZHFf bW92ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgc2NoZWRfaWRs ZXRkCiAgMTAwLjAlICBbMl0gICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAg YmRvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJ1ZmRvbmUK ICAxMDAuMCUgIFsxXSAgICAgICAgICBidWZkb25lYmlvCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBnX2lvX2RlbGl2ZXIKCjAwLjM0JSAgWzFdICAgICAgICBkZXZ2bl9yZWZ0aHJlYWQgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRldmZzX3JlYWRfZgogIDEwMC4w JSAgWzFdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGtlcm5f cmVhZHYKCjAwLjM0JSAgWzFdICAgICAgICBYZmFzdF9zeXNjYWxsIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAoKMDAuMzQlICBbMV0gICAgICAgIGhhcmRjbG9ja19jbnQgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGhhbmRsZWV2ZW50cwogIDEwMC4wJSAgWzFdICAgICAg ICAgIHRpbWVyY2IKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGxhcGljX2hhbmRsZV90aW1lcgoK MDAuMzQlICBbMV0gICAgICAgIGRldnN0YXRfZW5kX3RyYW5zYWN0aW9uIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZzdGF0X2VuZF90cmFuc2FjdGlvbl9iaW9f YnQKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGRhZG9uZQoKMDAuMzQlICBbMV0gICAgICAgIHBtYXBfcWVudGVyIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnX2lvX2NoZWNrCiAgMTAwLjAl ICBbMV0gICAgICAgICAgZ19pb19yZXF1ZXN0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZf c3RyYXRlZ3lfY3N3CgowMC4zNCUgIFsxXSAgICAgICAgZGFzdHJhdGVneSBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19kaXNrX3N0YXJ0CiAgMTAwLjAlICBbMV0g ICAgICAgICAgZ19pb19yZXF1ZXN0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZfc3RyYXRl Z3lfY3N3CgowMC4zNCUgIFsxXSAgICAgICAga2Vybl9yZWFkdiBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3lzX3JlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBh bWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgdmZzX3RpbWVzdGFtcCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2ZnNfd3JpdGVfZgogIDEwMC4wJSAg WzFdICAgICAgICAgIGRvZmlsZXdyaXRlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBrZXJuX3dy aXRldgoKMDAuMzQlICBbMV0gICAgICAgIGdfaW9fZGVsaXZlciBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgMTAwLjAlICBbMV0g ICAgICAgICAgZGFkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNz CgowMC4zNCUgIFsxXSAgICAgICAgZ19kZXZfc3RyYXRlZ3kgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsxXSAgICAg ICAgICBwaHlzaW8KICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQl ICBbMV0gICAgICAgIG1hbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgeHB0X3J1bl9hbGxvY3EKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYXN0cmF0ZWd5CiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBnX2Rpc2tfc3RhcnQKCjAwLjM0JSAgWzFdICAgICAgICBz Y2hlZF9hZGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNldHJ1 bm5hYmxlCiAgMTAwLjAlICBbMV0gICAgICAgICAgc2xlZXBxX2Jyb2FkY2FzdAogICAxMDAuMCUg IFsxXSAgICAgICAgICAgd2FrZXVwCgowMC4zNCUgIFsxXSAgICAgICAgX19sb2NrbWdyX2FyZ3Mg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHJlbHBidWYKICAxMDAu MCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3Jl YWRfZgoKMDAuMzQlICBbMV0gICAgICAgIF9zbGVlcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgYndhaXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQlICBbMV0gICAgICAgIF9f Y2FwX3JpZ2h0c19pc19zZXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIGZnZXRfdW5sb2NrZWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBfZmdldAogICAxMDAuMCUg IFsxXSAgICAgICAgICAga2Vybl93cml0ZXYKCjAwLjM0JSAgWzFdICAgICAgICBib3VuY2VfYnVz X2RtYW1hcF9sb2FkX2J1ZmZlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgYnVzX2RtYW1hcF9sb2FkCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfbWFwX3Jl cXVlc3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbMV0gICAgICAgICAgIG1y c2FzX2J1aWxkX2RjZGIKCjAwLjM0JSAgWzFdICAgICAgICBzY2hlZF9waWNrY3B1IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAuMCUgIFsx XSAgICAgICAgICBzZXRydW5uYWJsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xlZXBxX2Jy b2FkY2FzdAoKMDAuMzQlICBbMV0gICAgICAgIGludHJfZXZlbnRfaGFuZGxlIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBpbnRyX2V4ZWN1dGVfaGFuZGxlcnMKICAx MDAuMCUgIFsxXSAgICAgICAgICBsYXBpY19oYW5kbGVfaW50cgoKMDAuMzQlICBbMV0gICAgICAg IGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHhwdF9yZWxl YXNlX2NjYgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRhZG9uZQogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuMzQlICBbMV0gICAgICAgIGNhbGxvdXRfbG9jayBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgX2NhbGxvdXRfc3RvcF9z YWZlCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwv bXJzYXMua28KICAgMTAwLjAlICBbMV0gICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAoKMDAu MzQlICBbMV0gICAgICAgIGRldl9yZWx0aHJlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIGRldmZzX3JlYWRfZgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRvZmls ZXJlYWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGtlcm5fcmVhZHYKCjAwLjM0JSAgWzFdICAg ICAgICBydW5xX2FkZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg dGRxX21vdmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAgIGNyaXRpY2FsX2V4aXQg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNwaW5sb2NrX2V4aXQK ICAxMDAuMCUgIFsxXSAgICAgICAgICBfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ3 YWl0CgowMC4zNCUgIFsxXSAgICAgICAgbXNpX3ZlY3RvciBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzFdICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJzCiAgMTAwLjAlICBbMV0g ICAgICAgICAgbGFwaWNfaGFuZGxlX2ludHIKCjAwLjM0JSAgWzFdICAgICAgICBmb2Zmc2V0X3Vu bG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2ZnNfd3Jp dGVfZgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRvZmlsZXdyaXRlCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBrZXJuX3dyaXRldgoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX3NsZWVwIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfc3dpdGNoCiAgMTAw LjAlICBbMV0gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9z bGVlcAoKMDAuMzQlICBbMV0gICAgICAgIHVubG9ja19tdHggQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0 CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBjYWxs b3V0X3Jlc2V0X3NidF9vbiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAg ICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAg ICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMC4zNCUgIFsxXSAgICAgICAgZGFkb25lIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfZG9uZV9wcm9jZXNz CiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfdGQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAgIHRzY19nZXRfdGltZWNvdW50X2xvd19s ZmVuY2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJpbnVwdGlt ZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRldnN0YXRfc3RhcnRfdHJhbnNhY3Rpb25fYmlvCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBnX2Rpc2tfc3RhcnQKCjAwLjM0JSAgWzFdICAgICAgICBz bGVlcHFfcmVzdW1lX3RocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgc2xlZXBxX2Jyb2FkY2FzdAogIDEwMC4wJSAgWzFdICAgICAgICAgIHdha2V1cAogICAx MDAuMCUgIFsxXSAgICAgICAgICAgYmRvbmUKCjAwLjM0JSAgWzFdICAgICAgICBwaHlzaW8gQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRldmZzX3JlYWRfZgogIDEw MC4wJSAgWzFdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGtl cm5fcmVhZHYKCjAwLjM0JSAgWzFdICAgICAgICBydW5xX2Nob29zZSBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc2NoZWRfY2hvb3NlCiAgMTAwLjAlICBbMV0gICAg ICAgICAgY2hvb3NldGhyZWFkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzY2hlZF9zd2l0Y2gK CjAwLjM0JSAgWzFdICAgICAgICB4cHRfZG9uZV90ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgZ19uZXdfYmlv IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZfc3RyYXRlZ3lf Y3N3CiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBkZXZmc19yZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3IEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFsx XSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRvZmlsZXJl YWQKCjAwLjM0JSAgWzFdICAgICAgICBtcnNhc19maXJlX2NtZCBAIC9ib290L2tlcm5lbC9tcnNh cy5rbwogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBkYXN0YXJ0CgowMC4zNCUgIFsxXSAgICAgICAgcnVucV9yZW1vdmUgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2Nob29zZQogIDEw MC4wJSAgWzFdICAgICAgICAgIGNob29zZXRocmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAg c2NoZWRfc3dpdGNoCgowMC4zNCUgIFsxXSAgICAgICAgdW1hX3phbGxvY19hcmcgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1hbGxvYwogIDEwMC4wJSAgWzFdICAg ICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5 CgowMC4zNCUgIFsxXSAgICAgICAgc3BpbmxvY2tfZW50ZXIgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbMV0gICAg ICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiaW9k b25lCgowMC4zNCUgIFsxXSAgICAgICAgZ19pb19yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMV0gICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgMTAwLjAlICBbMV0gICAg ICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19yZWFkX2YKCkAgQ1BV X0NMS19VTkhBTFRFRF9DT1JFIFsyOTUgc2FtcGxlc10KCjEzLjU2JSAgWzQwXSAgICAgICBjcHVf aWRsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzQwXSAgICAgICAgc2NoZWRfaWRs ZXRkCiAgMTAwLjAlICBbNDBdICAgICAgICAgZm9ya19leGl0CgowOS40OSUgIFsyOF0gICAgICAg WGludmxybmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowNi40NCUgIFsxOV0gICAgICAgaW52bHJu Z19oYW5kbGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDQuMDclICBbMTJdICAgICAgIHNtcF90 bGJfc2hvb3Rkb3duIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMTJdICAgICAgICBw bWFwX2ludmFsaWRhdGVfcmFuZ2UKICAxMDAuMCUgIFsxMl0gICAgICAgICBiaW9kb25lCiAgIDEw MC4wJSAgWzEyXSAgICAgICAgICBkYWRvbmUKCjAzLjczJSAgWzExXSAgICAgICBjcHVfc2VhcmNo X2hpZ2hlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxMV0gICAgICAgIGNwdV9z ZWFyY2hfaGlnaGVzdAogIDYzLjY0JSAgWzddICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAu MCUgIFs3XSAgICAgICAgICAgZm9ya19leGl0CiAgMzYuMzYlICBbNF0gICAgICAgICAgY3B1X3Nl YXJjaF9oaWdoZXN0CiAgIDEwMC4wJSAgWzRdICAgICAgICAgICBzY2hlZF9pZGxldGQKCjAzLjA1 JSAgWzldICAgICAgICBzY2hlZF9pZGxldGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFs5XSAgICAgICAgIGZvcmtfZXhpdAoKMDIuNzElICBbOF0gICAgICAgIHNwaW5sb2NrX2V4aXQg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAzNy41MCUgIFszXSAgICAgICAgIHNjaGVkX2lkbGV0ZAog IDEwMC4wJSAgWzNdICAgICAgICAgIGZvcmtfZXhpdAogMTIuNTAlICBbMV0gICAgICAgICBjYWxs b3V0X3Jlc2V0X3NidF9vbgogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9i b290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X3J1bl9kZXZx IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTIuNTAlICBbMV0gICAgICAgICB3YWtldXAKICAxMDAu MCUgIFsxXSAgICAgICAgICBiZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYnVmZG9uZQog MTIuNTAlICBbMV0gICAgICAgICB0aHJlYWRfbG9ja19mbGFnc18KICAxMDAuMCUgIFsxXSAgICAg ICAgICBpdGhyZWFkX2xvb3AKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAogMTIu NTAlICBbMV0gICAgICAgICBfX210eF9sb2NrX3NsZWVwCiAgMTAwLjAlICBbMV0gICAgICAgICAg X19tdHhfbG9ja19mbGFncwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfY21kX2RvbmUg QCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEyLjUwJSAgWzFdICAgICAgICAgaXRocmVhZF9sb29w IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoK MDIuNzElICBbOF0gICAgICAgIF9fbXR4X2xvY2tfc2xlZXAgQCAvYm9vdC9rZXJuZWwva2VybmVs CiA2Mi41MCUgIFs1XSAgICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MKICAxMDAuMCUgIFs1XSAgICAg ICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFs1 XSAgICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAxMi41MCUgIFsxXSAgICAgICAgIHhwdF9y dW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRf YWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RhcnQKIDEyLjUwJSAg WzFdICAgICAgICAgdm1lbV94ZnJlZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGJpb2RvbmUKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQogMTIuNTAlICBbMV0gICAgICAgICBkYWRvbmUK ICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICB4cHRfZG9uZV90ZAoKMDIuMzclICBbN10gICAgICAgIGNwdV9zd2l0Y2ggQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs3XSAgICAgICAgIG1pX3N3aXRjaAogIDQyLjg2JSAg WzNdICAgICAgICAgIHNsZWVwcV93YWl0CiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBfc2xlZXAK ICAyOC41NyUgIFsyXSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBbMl0gICAgICAg ICAgIGZvcmtfZXhpdAogIDE0LjI5JSAgWzFdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAu MCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CiAgMTQuMjklICBbMV0gICAgICAgICAgY3JpdGlj YWxfZXhpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKCjAyLjM3 JSAgWzddICAgICAgICB0aHJlYWRfbG9ja19mbGFnc18gQCAvYm9vdC9rZXJuZWwva2VybmVsCiA0 Mi44NiUgIFszXSAgICAgICAgIHNsZWVwcV9hZGQKICAxMDAuMCUgIFszXSAgICAgICAgICBfc2xl ZXAKICAgMTAwLjAlICBbM10gICAgICAgICAgIGJ3YWl0CiA0Mi44NiUgIFszXSAgICAgICAgIHNj aGVkX2lkbGV0ZAogIDEwMC4wJSAgWzNdICAgICAgICAgIGZvcmtfZXhpdAogMTQuMjklICBbMV0g ICAgICAgICB0dXJuc3RpbGVfd2FpdAogIDEwMC4wJSAgWzFdICAgICAgICAgIF9fbXR4X2xvY2tf c2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MKCjAyLjM3JSAg WzddICAgICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzdd ICAgICAgICAgdm1lbV9hbGxvYwogIDEwMC4wJSAgWzddICAgICAgICAgIGdfaW9fY2hlY2sKICAg MTAwLjAlICBbN10gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDIuMDMlICBbNl0gICAgICAgIGNw dV9zZWFyY2hfbG93ZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNl0gICAgICAg ICBjcHVfc2VhcmNoX2xvd2VzdAogIDY2LjY3JSAgWzRdICAgICAgICAgIGNwdV9zZWFyY2hfbG93 ZXN0CiAgIDEwMC4wJSAgWzRdICAgICAgICAgICBzY2hlZF9waWNrY3B1CiAgMzMuMzMlICBbMl0g ICAgICAgICAgc2NoZWRfcGlja2NwdQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgc2NoZWRfYWRk CgowMS42OSUgIFs1XSAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbNV0gICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAxMDAuMCUgIFs1XSAgICAg ICAgICBkYXN0YXJ0CiAgIDEwMC4wJSAgWzVdICAgICAgICAgICB4cHRfcnVuX2FsbG9jcQoKMDEu MzYlICBbNF0gICAgICAgIGNhbGxvdXRfbG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzRdICAgICAgICAgX2NhbGxvdXRfc3RvcF9zYWZlCiAgMTAwLjAlICBbNF0gICAgICAgICAg bXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbNF0gICAg ICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAoKMDEuMzYlICBbNF0gICAgICAgIGFtZDY0X3N5c2Nh bGwgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMS4zNiUgIFs0XSAgICAgICAgX19sb2NrbWdyX2Fy Z3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA3NS4wMCUgIFszXSAgICAgICAgIHJlbHBidWYKICAx MDAuMCUgIFszXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBbM10gICAgICAgICAgIGRldmZz X3JlYWRfZgogMjUuMDAlICBbMV0gICAgICAgICBnZXRwYnVmCiAgMTAwLjAlICBbMV0gICAgICAg ICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19yZWFkX2YKCjAxLjM2JSAg WzRdICAgICAgICBwaHlzaW8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAg ICAgIGRldmZzX3JlYWRfZgogIDEwMC4wJSAgWzRdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAw LjAlICBbNF0gICAgICAgICAgIGtlcm5fcmVhZHYKCjAxLjM2JSAgWzRdICAgICAgICBzY2hlZF9w aWNrY3B1IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNF0gICAgICAgICBzY2hlZF9h ZGQKICA1MC4wMCUgIFsyXSAgICAgICAgICBzZXRydW5uYWJsZQogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgc2xlZXBxX2Jyb2FkY2FzdAogIDUwLjAwJSAgWzJdICAgICAgICAgIGludHJfZXZlbnRf c2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRs ZQoKMDEuMzYlICBbNF0gICAgICAgIHNjaGVkX3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzRdICAgICAgICAgbWlfc3dpdGNoCiAgMTAwLjAlICBbNF0gICAgICAgICAgc2xl ZXBxX3dhaXQKICAgMTAwLjAlICBbNF0gICAgICAgICAgIF9zbGVlcAoKMDEuMzYlICBbNF0gICAg ICAgIF9fbXR4X3VubG9ja19mbGFncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzJd ICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAw LjAlICBbMl0gICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgaXRocmVhZF9sb29wCiA1MC4wMCUg IFsyXSAgICAgICAgIHhwdF9ydW5fZGV2cQogIDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9hY3Rp b25fZGVmYXVsdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZGFzdGFydAoKMDEuMzYlICBbNF0g ICAgICAgIHZtX2ZhdWx0X3F1aWNrX2hvbGRfcGFnZXMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFs0XSAgICAgICAgIHZtYXBidWYKICAxMDAuMCUgIFs0XSAgICAgICAgICBwaHlzaW8K ICAgMTAwLjAlICBbNF0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDEuMDIlICBbM10gICAgICAg IF9fbXR4X2xvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAzMy4zMyUgIFsxXSAgICAg ICAgIG1yc2FzX2ZpcmVfY21kIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0g ICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsx XSAgICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAzMy4zMyUgIFsxXSAgICAgICAgIG1yc2Fz X2dldF9tcHRfY21kIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAg ICAgbXJzYXNfYWN0aW9uCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVuX2RldnEgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAzMy4zMyUgIFsxXSAgICAgICAgIG1yc2FzX3VubWFwX3JlcXVl c3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19j bWRfZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCgowMS4w MiUgIFszXSAgICAgICAgc2NoZWRfcnVubmFibGUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFszXSAgICAgICAgIGNwdV9pZGxlCiAgMTAwLjAlICBbM10gICAgICAgICAgc2NoZWRfaWRs ZXRkCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBmb3JrX2V4aXQKCjAxLjAyJSAgWzNdICAgICAg ICBjcml0aWNhbF9leGl0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBbMl0gICAgICAg ICBzcGlubG9ja19leGl0CiAgNTAuMDAlICBbMV0gICAgICAgICAgdGhyZWFkX2xvY2tfYmxvY2sK ICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNjaGVkX2FkZAogIDUwLjAwJSAgWzFdICAgICAgICAg IGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRy X2V2ZW50X2hhbmRsZQogMzMuMzMlICBbMV0gICAgICAgICBtYWxsb2MKICAxMDAuMCUgIFsxXSAg ICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdHJhdGVn eQoKMDEuMDIlICBbM10gICAgICAgIHZtX3BhZ2VfdW5ob2xkX3BhZ2VzIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB2dW5tYXBidWYKICAxMDAuMCUgIFszXSAgICAg ICAgICBwaHlzaW8KICAgMTAwLjAlICBbM10gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDEuMDIl ICBbM10gICAgICAgIHZtZW1feGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsz XSAgICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFszXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAl ICBbM10gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAxLjAyJSAgWzNdICAgICAgICBkb3Jl dGkgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC42OCUgIFsyXSAgICAgICAgZGFzdHJhdGVneSBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZ19kaXNrX3N0YXJ0CiAg MTAwLjAlICBbMl0gICAgICAgICAgZ19pb19yZXF1ZXN0CiAgIDEwMC4wJSAgWzJdICAgICAgICAg ICBkZXZfc3RyYXRlZ3lfY3N3CgowMC42OCUgIFsyXSAgICAgICAgeHB0X2RvbmVfdGQgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGZvcmtfZXhpdAoKMDAuNjglICBb Ml0gICAgICAgIHBtYXBfZXh0cmFjdF9hbmRfaG9sZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzJdICAgICAgICAgdm1fZmF1bHRfcXVpY2tfaG9sZF9wYWdlcwogIDEwMC4wJSAgWzJd ICAgICAgICAgIHZtYXBidWYKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHBoeXNpbwoKMDAuNjgl ICBbMl0gICAgICAgIGRldnN0YXRfZW5kX3RyYW5zYWN0aW9uIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMl0gICAgICAgICBkZXZzdGF0X2VuZF90cmFuc2FjdGlvbl9iaW9fYnQKICAx MDAuMCUgIFsyXSAgICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMl0gICAg ICAgICAgIGRhZG9uZQoKMDAuNjglICBbMl0gICAgICAgIF9mZ2V0IEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMl0gICAgICAgICBrZXJuX3JlYWR2CiAgMTAwLjAlICBbMl0gICAgICAg ICAgc3lzX3JlYWQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjY4 JSAgWzJdICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4w JSAgWzJdICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4w JSAgWzJdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFsyXSAgICAgICAg ICAgZGFzdGFydAoKMDAuNjglICBbMl0gICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZCBAIC9ib290 L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbMl0gICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVf aGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMl0gICAgICAgICAgaXRo cmVhZF9sb29wCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjY4JSAgWzJd ICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJzCiAgMTAwLjAlICBbMl0gICAgICAgICAg bGFwaWNfaGFuZGxlX2ludHIKCjAwLjY4JSAgWzJdICAgICAgICBnZXRwYnVmIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFsyXSAgICAg ICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRvZmlsZXJlYWQKCjAw LjY4JSAgWzJdICAgICAgICB1bWFfemFsbG9jX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUw LjAwJSAgWzFdICAgICAgICAgbWFsbG9jCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9h bGxvY3EKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RyYXRlZ3kKIDUwLjAwJSAgWzFdICAg ICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAx MDAuMCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC42OCUgIFsyXSAgICAgICAgc3Bp bmxvY2tfZW50ZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIGNh bGxvdXRfbG9jawogIDEwMC4wJSAgWzFdICAgICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogICAx MDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMu a28KIDUwLjAwJSAgWzFdICAgICAgICAgc2xlZXBxX2xvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAgMTAwLjAlICBbMV0gICAgICAgICAgX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBi d2FpdAoKMDAuNjglICBbMl0gICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZSBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgYmlvZG9uZQogIDEwMC4wJSAgWzJdICAg ICAgICAgIGRhZG9uZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoK MDAuNjglICBbMl0gICAgICAgIGRldmZzX3JlYWRfZiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzJdICAgICAgICAgZG9maWxlcmVhZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGtlcm5f cmVhZHYKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHN5c19yZWFkCgowMC42OCUgIFsyXSAgICAg ICAgYmRvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGJ1ZmRv bmUKICAxMDAuMCUgIFsyXSAgICAgICAgICBidWZkb25lYmlvCiAgIDEwMC4wJSAgWzJdICAgICAg ICAgICBnX2lvX2RlbGl2ZXIKCjAwLjY4JSAgWzJdICAgICAgICBiY29weSBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBbMl0g ICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkYXN0 YXJ0CgowMC4zNCUgIFsxXSAgICAgICAgcmVscGJ1ZiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAgMTAwLjAlICBbMV0gICAgICAgICAgZGV2ZnNfcmVh ZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkb2ZpbGVyZWFkCgowMC4zNCUgIFsxXSAgICAg ICAgX210eF9sb2NrX3NwaW5fY29va2llIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICB0dXJuc3RpbGVfdHJ5d2FpdAogIDEwMC4wJSAgWzFdICAgICAgICAgIF9fbXR4 X2xvY2tfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHZtZW1feGFsbG9jCgowMC4zNCUg IFsxXSAgICAgICAgY2FtX2NjYnFfcmVtb3ZlX2NjYiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0 X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0YXJ0CgowMC4zNCUg IFsxXSAgICAgICAgX2NhbGxvdXRfc3RvcF9zYWZlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwog IDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAoKMDAuMzQlICBbMV0gICAgICAgIGZnZXRfdW5sb2NrZWQgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIF9mZ2V0CiAgMTAwLjAlICBbMV0gICAgICAgICAga2Vybl93 cml0ZXYKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHN5c193cml0ZQoKMDAuMzQlICBbMV0gICAg ICAgIGNwdV9mZXRjaF9zeXNjYWxsX2FyZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxXSAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFdICAgICAgICB0ZHFfbm90aWZ5 IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAx MDAuMCUgIFsxXSAgICAgICAgICBzZXRydW5uYWJsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAg c2xlZXBxX2Jyb2FkY2FzdAoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX2Nob29zZSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgY2hvb3NldGhyZWFkCiAgMTAwLjAl ICBbMV0gICAgICAgICAgc2NoZWRfc3dpdGNoCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtaV9z d2l0Y2gKCjAwLjM0JSAgWzFdICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RhcnQKICAxMDAuMCUgIFsxXSAgICAg ICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdHJhdGVneQoK MDAuMzQlICBbMV0gICAgICAgIGl0aHJlYWRfbG9vcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgdm1fcGFnZV9w YV90cnlyZWxvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHBt YXBfZXh0cmFjdF9hbmRfaG9sZAogIDEwMC4wJSAgWzFdICAgICAgICAgIHZtX2ZhdWx0X3F1aWNr X2hvbGRfcGFnZXMKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHZtYXBidWYKCjAwLjM0JSAgWzFd ICAgICAgICBzbGVlcHFfYnJvYWRjYXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICB3YWtldXAKICAxMDAuMCUgIFsxXSAgICAgICAgICBiZG9uZQogICAxMDAuMCUg IFsxXSAgICAgICAgICAgYnVmZG9uZQoKMDAuMzQlICBbMV0gICAgICAgIHN5c2NhbGxfdGhyZWFk X2VudGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBhbWQ2NF9z eXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgeHB0X3J1bl9hbGxvY3EgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RyYXRlZ3kKICAxMDAuMCUgIFsxXSAgICAg ICAgICBnX2Rpc2tfc3RhcnQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfaW9fcmVxdWVzdAoK MDAuMzQlICBbMV0gICAgICAgIFhhcGljX2lzcjEgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4z NCUgIFsxXSAgICAgICAga2Vybl9yZWFkdiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgc3lzX3JlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBhbWQ2NF9zeXNjYWxs CgowMC4zNCUgIFsxXSAgICAgICAgcmR0c2MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxXSAgICAgICAgIG1pX3N3aXRjaAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNsZWVwcV93YWl0 CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBfc2xlZXAKCjAwLjM0JSAgWzFdICAgICAgICBtcnNh c19kYXRhX2xvYWRfY2IgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAg ICAgYnVzX2RtYW1hcF9sb2FkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzFdICAg ICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBtcnNhc19idWlsZF9kY2RiCgowMC4zNCUgIFsxXSAgICAgICAgeHB0 X3JlbGVhc2VfY2NiIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBk YWRvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgIDEwMC4wJSAg WzFdICAgICAgICAgICB4cHRfZG9uZV90ZAoKMDAuMzQlICBbMV0gICAgICAgIG10eF9wb29sX2Zp bmQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJkb25lCiAgMTAw LjAlICBbMV0gICAgICAgICAgYnVmZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYnVmZG9u ZWJpbwoKMDAuMzQlICBbMV0gICAgICAgIGNyaXRpY2FsX2VudGVyIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMV0gICAgICAgICB0aHJlYWRfbG9ja19mbGFnc18KICAxMDAuMCUgIFsx XSAgICAgICAgICB0ZHFfbW92ZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2NoZWRfaWRsZXRk CgowMC4zNCUgIFsxXSAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBpdGhyZWFkX2xvb3AKICAxMDAuMCUg IFsxXSAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM0JSAgWzFdICAgICAgICBkYXN0YXJ0IEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEw MC4wJSAgWzFdICAgICAgICAgIGRhc3RyYXRlZ3kKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdf ZGlza19zdGFydAoKMDAuMzQlICBbMV0gICAgICAgIGF0b21pY19mZXRjaGFkZF9pbnQgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290 L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9i b290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9hY3Rpb25fZGVm YXVsdAoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX3JlbSBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzFdICAgICAgICAgdGRxX21vdmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBzY2hl ZF9pZGxldGQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0g ICAgICAgIHNjaGVkX2FkZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAg ICAgc2V0cnVubmFibGUKICAxMDAuMCUgIFsxXSAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAg IDEwMC4wJSAgWzFdICAgICAgICAgICB3YWtldXAKCjAwLjM0JSAgWzFdICAgICAgICBsYXBpY19o YW5kbGVfaW50ciBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBfc2xl ZXAgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJ3YWl0CiAgMTAw LjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19y ZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICBkb2ZpbGV3cml0ZSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAga2Vybl93cml0ZXYKICAxMDAuMCUgIFsxXSAgICAgICAg ICBzeXNfd3JpdGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0 JSAgWzFdICAgICAgICBmcmVlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICB4cHRfcmVsZWFzZV9jY2IKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYWRvbmUKICAgMTAw LjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjM0JSAgWzFdICAgICAgICBp bnRyX2V2ZW50X3NjaGVkdWxlX3RocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRy X2V4ZWN1dGVfaGFuZGxlcnMKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGxhcGljX2hhbmRsZV9p bnRyCgowMC4zNCUgIFsxXSAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJz YXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgMTAwLjAlICBb MV0gICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaXRocmVhZF9sb29wCgowMC4zNCUgIFsxXSAg ICAgICAgZm9mZnNldF9sb2NrX3VpbyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgZGV2ZnNfd3JpdGVfZgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRvZmlsZXdyaXRl CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBrZXJuX3dyaXRldgoKMDAuMzQlICBbMV0gICAgICAg IHBtYXBfa2V4dHJhY3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAg IGJvdW5jZV9idXNfZG1hbWFwX2xvYWRfYnVmZmVyCiAgMTAwLjAlICBbMV0gICAgICAgICAgYnVz X2RtYW1hcF9sb2FkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtcnNhc19tYXBfcmVxdWVzdCBA IC9ib290L2tlcm5lbC9tcnNhcy5rbwoKMDAuMzQlICBbMV0gICAgICAgIF9fc3lzX3JlYWQgQCAv bGliL2xpYmMuc28uNwoKMDAuMzQlICBbMV0gICAgICAgIGRvZmlsZXJlYWQgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGtlcm5fcmVhZHYKICAxMDAuMCUgIFsxXSAg ICAgICAgICBzeXNfcmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAoK MDAuMzQlICBbMV0gICAgICAgIHRkcV9sb2NrX3BhaXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZv cmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAgIGdfY2xvbmVfYmlvIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnX2Rldl9zdHJhdGVneQogIDEwMC4wJSAgWzFdICAg ICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBoeXNpbwoK MDAuMzQlICBbMV0gICAgICAgIGZvZmZzZXRfbG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgZm9mZnNldF9sb2NrX3VpbwogIDEwMC4wJSAgWzFdICAgICAgICAg IGRldmZzX3dyaXRlX2YKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRvZmlsZXdyaXRlCgowMC4z NCUgIFsxXSAgICAgICAgbXJzYXNfbWFwX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28K IDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfYnVpbGRfZGNkYgogIDEwMC4wJSAgWzFdICAgICAg ICAgIG1yc2FzX2FjdGlvbgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X3J1bl9kZXZxIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuMzQlICBbMV0gICAgICAgIGxvY2tfbXR4IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsxXSAg ICAgICAgICBid2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNCUgIFsx XSAgICAgICAgZGFkb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICB4cHRfZG9uZV9wcm9jZXNzCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfdGQKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAgIHRzY19n ZXRfdGltZWNvdW50X2xvd19sZmVuY2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIGJpbnVwdGltZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRldnN0YXRfc3RhcnRf dHJhbnNhY3Rpb25fYmlvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBnX2Rpc2tfc3RhcnQKCjAw LjM0JSAgWzFdICAgICAgICBjaG9vc2V0aHJlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIHNjaGVkX3N3aXRjaAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1pX3N3 aXRjaAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2NoZWRfaWRsZXRkCgowMC4zNCUgIFsxXSAg ICAgICAgdW1hX3pmcmVlX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgZ19pb19kZWxpdmVyCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNrX2RvbmVfc2lu Z2xlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjM0JSAgWzFdICAgICAgICBY dGltZXJpbnQgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNCUgIFsxXSAgICAgICAgZGV2c3Rh dF9zdGFydF90cmFuc2FjdGlvbl9iaW8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIGdfZGlza19zdGFydAogIDEwMC4wJSAgWzFdICAgICAgICAgIGdfaW9fcmVxdWVz dAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwoKMDAuMzQlICBbMV0g ICAgICAgIHJ1bnFfY2hvb3NlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICBzY2hlZF9jaG9vc2UKICAxMDAuMCUgIFsxXSAgICAgICAgICBjaG9vc2V0aHJlYWQKICAg MTAwLjAlICBbMV0gICAgICAgICAgIHNjaGVkX3N3aXRjaAoKQCBDUFVfQ0xLX1VOSEFMVEVEX0NP UkUgWzI4OCBzYW1wbGVzXQoKMTMuMTklICBbMzhdICAgICAgIGNwdV9pZGxlIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMzhdICAgICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFsz OF0gICAgICAgICBmb3JrX2V4aXQKCjEwLjA3JSAgWzI5XSAgICAgICBYaW52bHJuZyBAIC9ib290 L2tlcm5lbC9rZXJuZWwKCjA4LjY4JSAgWzI1XSAgICAgICBpbnZscm5nX2hhbmRsZXIgQCAvYm9v dC9rZXJuZWwva2VybmVsCgowNS4yMSUgIFsxNV0gICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBbMTBdICAgICAgICBjcHVfc2VhcmNoX2hpZ2hl c3QKICA4MC4wMCUgIFs4XSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBbOF0gICAg ICAgICAgIGZvcmtfZXhpdAogIDIwLjAwJSAgWzJdICAgICAgICAgIGNwdV9zZWFyY2hfaGlnaGVz dAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgc2NoZWRfaWRsZXRkCiAzMy4zMyUgIFs1XSAgICAg ICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzVdICAgICAgICAgIGZvcmtfZXhpdAoKMDMuODIl ICBbMTFdICAgICAgIGNwdV9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx MV0gICAgICAgIG1pX3N3aXRjaAogIDcyLjczJSAgWzhdICAgICAgICAgIHNjaGVkX2lkbGV0ZAog ICAxMDAuMCUgIFs4XSAgICAgICAgICAgZm9ya19leGl0CiAgMTguMTglICBbMl0gICAgICAgICAg aXRocmVhZF9sb29wCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3JrX2V4aXQKICAwOS4wOSUg IFsxXSAgICAgICAgICBjcml0aWNhbF9leGl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRy X2V2ZW50X2hhbmRsZQoKMDIuNDMlICBbN10gICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0IEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbN10gICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdAog IDg1LjcxJSAgWzZdICAgICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0CiAgIDEwMC4wJSAgWzZdICAg ICAgICAgICBzY2hlZF9waWNrY3B1CiAgMTQuMjklICBbMV0gICAgICAgICAgc2NoZWRfcGlja2Nw dQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2NoZWRfYWRkCgowMi4wOCUgIFs2XSAgICAgICAg c21wX3RsYl9zaG9vdGRvd24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs2XSAgICAg ICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzZdICAgICAgICAgIGJpb2RvbmUK ICAgMTAwLjAlICBbNl0gICAgICAgICAgIGRhZG9uZQoKMDIuMDglICBbNl0gICAgICAgIG1yc2Fz X2NvbXBsZXRlX2NtZCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbNl0gICAgICAg ICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAw LjAlICBbNl0gICAgICAgICAgaXRocmVhZF9sb29wCiAgIDEwMC4wJSAgWzZdICAgICAgICAgICBm b3JrX2V4aXQKCjAyLjA4JSAgWzZdICAgICAgICBzY2hlZF9pZGxldGQgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFs2XSAgICAgICAgIGZvcmtfZXhpdAoKMDIuMDglICBbNl0gICAgICAg IF9fbXR4X2xvY2tfc2xlZXAgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA2Ni42NyUgIFs0XSAgICAg ICAgIF9fbXR4X2xvY2tfZmxhZ3MKICAxMDAuMCUgIFs0XSAgICAgICAgICBtcnNhc19jbWRfZG9u ZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFs0XSAgICAgICAgICAgbXJzYXNf Y29tcGxldGVfY21kCiAxNi42NyUgIFsxXSAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tl cm5lbC9rZXJuZWwKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RhcnQKIDE2LjY3JSAgWzFdICAgICAgICAgdm1lbV94 ZnJlZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGJpb2RvbmUKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRhZG9uZQoKMDIuMDglICBbNl0gICAgICAgIF9fc3lzX3JlYWQgQCAvbGliL2xpYmMuc28u NwoKMDEuNzQlICBbNV0gICAgICAgIHNwaW5sb2NrX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVs CiA0MC4wMCUgIFsyXSAgICAgICAgIHdha2V1cAogIDEwMC4wJSAgWzJdICAgICAgICAgIGJkb25l CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBidWZkb25lCiAyMC4wMCUgIFsxXSAgICAgICAgIGl0 aHJlYWRfbG9vcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAogMjAuMDAlICBbMV0g ICAgICAgICB0dXJuc3RpbGVfdW5wZW5kCiAgMTAwLjAlICBbMV0gICAgICAgICAgX19tdHhfdW5s b2NrX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVuX2RldnEKIDIwLjAwJSAg WzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMV0gICAgICAgICAgZm9ya19leGl0 CgowMS43NCUgIFs1XSAgICAgICAgX19sb2NrbWdyX2FyZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVs CiA2MC4wMCUgIFszXSAgICAgICAgIHJlbHBidWYKICAxMDAuMCUgIFszXSAgICAgICAgICBwaHlz aW8KICAgMTAwLjAlICBbM10gICAgICAgICAgIGRldmZzX3JlYWRfZgogNDAuMDAlICBbMl0gICAg ICAgICBnZXRwYnVmCiAgMTAwLjAlICBbMl0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzJd ICAgICAgICAgICBkZXZmc19yZWFkX2YKCjAxLjc0JSAgWzVdICAgICAgICBkb3JldGkgQCAvYm9v dC9rZXJuZWwva2VybmVsCgowMS4zOSUgIFs0XSAgICAgICAgdm1lbV94YWxsb2MgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAgICAgIHZtZW1fYWxsb2MKICAxMDAuMCUgIFs0 XSAgICAgICAgICBnX2lvX2NoZWNrCiAgIDEwMC4wJSAgWzRdICAgICAgICAgICBnX2lvX3JlcXVl c3QKCjAxLjM5JSAgWzRdICAgICAgICBfX210eF91bmxvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiA1MC4wMCUgIFsyXSAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZCBAIC9ib290L2tl cm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzJdICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9o YW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGl0 aHJlYWRfbG9vcAogMjUuMDAlICBbMV0gICAgICAgICB4cHRfcnVuX2RldnEKICAxMDAuMCUgIFsx XSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRh c3RhcnQKIDI1LjAwJSAgWzFdICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21y c2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMS4z OSUgIFs0XSAgICAgICAgYnplcm8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsyXSAg ICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsyXSAgICAgICAgICBwaHlzaW8KICAg MTAwLjAlICBbMl0gICAgICAgICAgIGRldmZzX3JlYWRfZgogMjUuMDAlICBbMV0gICAgICAgICBr ZXJuX3dyaXRldgogIDEwMC4wJSAgWzFdICAgICAgICAgIHN5c193cml0ZQogICAxMDAuMCUgIFsx XSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAogMjUuMDAlICBbMV0gICAgICAgICBtcnNhc19hY3Rp b24gQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcnVu X2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRf YWN0aW9uX2RlZmF1bHQKCjAxLjM5JSAgWzRdICAgICAgICB0ZHFfbm90aWZ5IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbNF0gICAgICAgICBzY2hlZF9hZGQKICA1MC4wMCUgIFsyXSAg ICAgICAgICBzZXRydW5uYWJsZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgc2xlZXBxX2Jyb2Fk Y2FzdAogIDI1LjAwJSAgWzFdICAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDI1LjAwJSAgWzFdICAg ICAgICAgIHR1cm5zdGlsZV91bnBlbmQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9fbXR4X3Vu bG9ja19zbGVlcAoKMDEuMDQlICBbM10gICAgICAgIGdfZGlza19kb25lX3NpbmdsZSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZGFkb25lCiAgMTAwLjAlICBbM10g ICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogICAxMDAuMCUgIFszXSAgICAgICAgICAgeHB0X2Rv bmVfdGQKCjAxLjA0JSAgWzNdICAgICAgICBzY2hlZF9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFszXSAgICAgICAgIG1pX3N3aXRjaAogIDEwMC4wJSAgWzNdICAgICAgICAg IHNsZWVwcV93YWl0CiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBfc2xlZXAKCjAxLjA0JSAgWzNd ICAgICAgICBsYXBpY19pcGlfdmVjdG9yZWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFszXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbM10gICAgICAgICAgcG1h cF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBiaW9kb25lCgowMS4w NCUgIFszXSAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbM10gICAgICAgICBkYXN0YXJ0CiAgMTAwLjAlICBbM10gICAgICAgICAgeHB0X3J1 bl9hbGxvY3EKICAgMTAwLjAlICBbM10gICAgICAgICAgIGRhc3RyYXRlZ3kKCjAxLjA0JSAgWzNd ICAgICAgICBzbGVlcHFfbG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2LjY3JSAgWzJdICAg ICAgICAgd2FrZXVwCiAgMTAwLjAlICBbMl0gICAgICAgICAgYmRvbmUKICAgMTAwLjAlICBbMl0g ICAgICAgICAgIGJ1ZmRvbmUKIDMzLjMzJSAgWzFdICAgICAgICAgY3ZfYnJvYWRjYXN0cHJpCiAg MTAwLjAlICBbMV0gICAgICAgICAgdm1lbV94ZnJlZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAg YmlvZG9uZQoKMDEuMDQlICBbM10gICAgICAgIHNjaGVkX2Nob29zZSBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgY2hvb3NldGhyZWFkCiAgMTAwLjAlICBbM10gICAg ICAgICAgc2NoZWRfc3dpdGNoCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBtaV9zd2l0Y2gKCjAx LjA0JSAgWzNdICAgICAgICB1bWFfemFsbG9jX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDMz LjMzJSAgWzFdICAgICAgICAgZ19jbG9uZV9iaW8KICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2Rl dl9zdHJhdGVneQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogMzMu MzMlICBbMV0gICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgMTAwLjAlICBbMV0gICAgICAgICAg cGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19yZWFkX2YKIDMzLjMzJSAgWzFd ICAgICAgICAgbWFsbG9jCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RyYXRlZ3kKCjAxLjA0JSAgWzNdICAgICAgICBiZG9u ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgYnVmZG9uZQogIDEw MC4wJSAgWzNdICAgICAgICAgIGJ1ZmRvbmViaW8KICAgMTAwLjAlICBbM10gICAgICAgICAgIGdf aW9fZGVsaXZlcgoKMDAuNjklICBbMl0gICAgICAgIHRocmVhZF9sb2NrX2ZsYWdzXyBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgaW50cl9ldmVudF9zY2hlZHVsZV90 aHJlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogICAxMDAuMCUg IFsxXSAgICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJzCiA1MC4wMCUgIFsxXSAgICAgICAg IHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAuNjklICBb Ml0gICAgICAgIGtlcm5fd3JpdGV2IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBzeXNfd3JpdGUKICAxMDAuMCUgIFsyXSAgICAgICAgICBhbWQ2NF9zeXNjYWxsCgow MC42OSUgIFsyXSAgICAgICAgYnRfZnJlZXRyaW0gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFsyXSAgICAgICAgICBkYWRvbmUKICAg MTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjY5JSAgWzJdICAgICAg ICBkZXZzdGF0X2VuZF90cmFuc2FjdGlvbiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgZGV2c3RhdF9lbmRfdHJhbnNhY3Rpb25fYmlvX2J0CiAgMTAwLjAlICBbMl0g ICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkYWRv bmUKCjAwLjY5JSAgWzJdICAgICAgICBYYXBpY19pc3IxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoK MDAuNjklICBbMl0gICAgICAgIHBtYXBfcWVudGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMl0gICAgICAgICBnX2lvX2NoZWNrCiAgMTAwLjAlICBbMl0gICAgICAgICAgZ19pb19y ZXF1ZXN0CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CgowMC42OSUg IFsyXSAgICAgICAgX19zeXNfd3JpdGUgQCAvbGliL2xpYmMuc28uNwoKMDAuNjklICBbMl0gICAg ICAgIGdfZGV2X3N0cmF0ZWd5IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAg ICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgMTAwLjAlICBbMl0gICAgICAgICAgcGh5c2lvCiAgIDEw MC4wJSAgWzJdICAgICAgICAgICBkZXZmc19yZWFkX2YKCjAwLjY5JSAgWzJdICAgICAgICBjcml0 aWNhbF9lbnRlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgc2No ZWRfc3dpdGNoCiAgMTAwLjAlICBbMV0gICAgICAgICAgbWlfc3dpdGNoCiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBpdGhyZWFkX2xvb3AKIDUwLjAwJSAgWzFdICAgICAgICAgdW1hX3phbGxvY19h cmcKICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBwaHlzaW8KCjAwLjY5JSAgWzJdICAgICAgICBkYXN0YXJ0IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEwMC4wJSAg WzJdICAgICAgICAgIGRhc3RyYXRlZ3kKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGdfZGlza19z dGFydAoKMDAuNjklICBbMl0gICAgICAgIGRvZmlsZXdyaXRlIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMl0gICAgICAgICBrZXJuX3dyaXRldgogIDEwMC4wJSAgWzJdICAgICAgICAg IHN5c193cml0ZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuNjkl ICBbMl0gICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAg MTAwLjAlICBbMl0gICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzJdICAg ICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVs CgowMC42OSUgIFsyXSAgICAgICAgY2FsbG91dF9sb2NrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMl0gICAgICAgICBfY2FsbG91dF9zdG9wX3NhZmUKICAxMDAuMCUgIFsyXSAgICAg ICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFsy XSAgICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCgowMC42OSUgIFsyXSAgICAgICAgcnVucV9h ZGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIHNjaGVkX3N3aXRj aAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1pX3N3aXRjaAogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgY3JpdGljYWxfZXhpdAogNTAuMDAlICBbMV0gICAgICAgICB0ZHFfbW92ZQogIDEwMC4wJSAg WzFdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19l eGl0CgowMC42OSUgIFsyXSAgICAgICAgZGFkb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMl0gICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgMTAwLjAlICBbMl0gICAgICAgICAg eHB0X2RvbmVfdGQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuNjklICBb Ml0gICAgICAgIHJ1bnFfY2hvb3NlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBzY2hlZF9jaG9vc2UKICAxMDAuMCUgIFsyXSAgICAgICAgICBjaG9vc2V0aHJlYWQK ICAgMTAwLjAlICBbMl0gICAgICAgICAgIHNjaGVkX3N3aXRjaAoKMDAuNjklICBbMl0gICAgICAg IGFtZDY0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC42OSUgIFsyXSAgICAgICAg c3BpbmxvY2tfZW50ZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAg IHRocmVhZF9sb2NrX2ZsYWdzXwogIDEwMC4wJSAgWzFdICAgICAgICAgIHNjaGVkX2lkbGV0ZAog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CiA1MC4wMCUgIFsxXSAgICAgICAgIGNh bGxvdXRfbG9jawogIDEwMC4wJSAgWzFdICAgICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogICAx MDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMu a28KCjAwLjM1JSAgWzFdICAgICAgICBhdG9taWNfZmV0Y2hhZGRfaW50IEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwv bXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKCjAw LjM1JSAgWzFdICAgICAgICBwbWFwX2tleHRyYWN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBmcmVlCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3JlbGVhc2Vf Y2NiCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjM1JSAgWzFdICAgICAgICBn X2lvX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRl dl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBb MV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzUlICBbMV0gICAgICAgIGNhbXFfcmVtb3Zl IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBjYW1fY2NicV9yZW1v dmVfY2NiCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9kZXZxCiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKCjAwLjM1JSAgWzFdICAgICAgICBidXNfZG1h bWFwX2xvYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2Fz X21hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAg ICAgbXJzYXNfYnVpbGRfZGNkYgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfYWN0aW9u CgowMC4zNSUgIFsxXSAgICAgICAgZ19kZXZfZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgZ19pb19kZWxpdmVyCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19k aXNrX2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjM1JSAg WzFdICAgICAgICB0aHJlYWRfbG9ja19zZXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxXSAgICAgICAgIHNsZWVwcV9zd2l0Y2gKICAxMDAuMCUgIFsxXSAgICAgICAgICBzbGVlcHFf d2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgX3NsZWVwCgowMC4zNSUgIFsxXSAgICAgICAg ZGV2dm5fcmVmdGhyZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICBkZXZmc19yZWFkX2YKICAxMDAuMCUgIFsxXSAgICAgICAgICBkb2ZpbGVyZWFkCiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBrZXJuX3JlYWR2CgowMC4zNSUgIFsxXSAgICAgICAgYmlvcV90YWtl Zmlyc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RhcnQK ICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZGFzdHJhdGVneQoKMDAuMzUlICBbMV0gICAgICAgIHZmc190aW1lc3RhbXAgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRldmZzX3dyaXRlX2YKICAxMDAu MCUgIFsxXSAgICAgICAgICBkb2ZpbGV3cml0ZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAga2Vy bl93cml0ZXYKCjAwLjM1JSAgWzFdICAgICAgICB4cHRfZG9uZV9wcm9jZXNzIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfZG9uZV90ZAogIDEwMC4wJSAgWzFd ICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzUlICBbMV0gICAgICAgIHRkcV9sb2NrX3BhaXIgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2lkbGV0ZAogIDEw MC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzUlICBbMV0gICAgICAgIHNjc2lfcmVh ZF93cml0ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGFzdGFy dAogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkYXN0cmF0ZWd5CgowMC4zNSUgIFsxXSAgICAgICAgdW5sb2NrX210eCBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgX3NsZWVwCiAgMTAwLjAlICBbMV0g ICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBoeXNpbwoKMDAuMzUlICBb MV0gICAgICAgIGl0aHJlYWRfbG9vcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgZm9ya19leGl0CgowMC4zNSUgIFsxXSAgICAgICAgY2FsbG91dF9yZXNldF9zYnRf b24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2FzX2FjdGlv biBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5f ZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9h Y3Rpb25fZGVmYXVsdAoKMDAuMzUlICBbMV0gICAgICAgIG1yc2FzX2lzciBAIC9ib290L2tlcm5l bC9tcnNhcy5rbwogMTAwLjAlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxl cnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMV0gICAgICAgICAgaXRocmVhZF9s b29wCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM1JSAgWzFdICAgICAg ICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAl ICBbMV0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjM1JSAgWzFdICAgICAgICBkZXZm c19yZWFkX2YgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRvZmls ZXJlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBrZXJuX3JlYWR2CiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBzeXNfcmVhZAoKMDAuMzUlICBbMV0gICAgICAgIGdldHBidWYgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHBoeXNpbwogIDEwMC4wJSAgWzFdICAgICAg ICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZG9maWxlcmVhZAoKMDAu MzUlICBbMV0gICAgICAgIHhwdF9kb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4w JSAgWzFdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAu MzUlICBbMV0gICAgICAgIGRldmZzX3dyaXRlX2YgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIGRvZmlsZXdyaXRlCiAgMTAwLjAlICBbMV0gICAgICAgICAga2Vybl93 cml0ZXYKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHN5c193cml0ZQoKMDAuMzUlICBbMV0gICAg ICAgIHZtZW1feGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAg IGJpb2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjM1JSAgWzFdICAgICAgICBwY3B1X2ZpbmQgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGNwdV9pZGxlX3dha2V1cAog IDEwMC4wJSAgWzFdICAgICAgICAgIHRkcV9ub3RpZnkKICAgMTAwLjAlICBbMV0gICAgICAgICAg IHNjaGVkX2FkZAoKMDAuMzUlICBbMV0gICAgICAgIHJlbHBidWYgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHBoeXNpbwogIDEwMC4wJSAgWzFdICAgICAgICAgIGRl dmZzX3JlYWRfZgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZG9maWxlcmVhZAoKMDAuMzUlICBb MV0gICAgICAgIF9tdHhfbG9ja19zcGluX2Nvb2tpZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgdGRxX2xvY2tfcGFpcgogIDEwMC4wJSAgWzFdICAgICAgICAgIHNj aGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CgowMC4zNSUgIFsx XSAgICAgICAgdGRxX21vdmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzUl ICBbMV0gICAgICAgIG1yc2FzX3VubWFwX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28K IDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAg ICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRf ZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM1JSAgWzFdICAgICAg ICBfY2FsbG91dF9zdG9wX3NhZmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAg ICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBb MV0gICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBp bnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNSUg IFsxXSAgICAgICAgZmdldF91bmxvY2tlZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgX2ZnZXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBrZXJuX3dyaXRldgogICAx MDAuMCUgIFsxXSAgICAgICAgICAgc3lzX3dyaXRlCgowMC4zNSUgIFsxXSAgICAgICAgc2xlZXBx X3JlbGVhc2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHdha2V1 cAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBi dWZkb25lCgowMC4zNSUgIFsxXSAgICAgICAgeHB0X3J1bl9hbGxvY3EgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RyYXRlZ3kKICAxMDAuMCUgIFsxXSAgICAg ICAgICBnX2Rpc2tfc3RhcnQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfaW9fcmVxdWVzdAoK MDAuMzUlICBbMV0gICAgICAgIHR1cm5zdGlsZV90cnl3YWl0IEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMV0gICAgICAgICBfX210eF9sb2NrX3NsZWVwCiAgMTAwLjAlICBbMV0gICAg ICAgICAgcmVscGJ1ZgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNSUgIFsx XSAgICAgICAgZ19pc19nZW9tX3RocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgZ19pb19kZWxpdmVyCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNrX2Rv bmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjM1JSAgWzFdICAg ICAgICB1c2VycmV0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBh bWQ2NF9zeXNjYWxsCgowMC4zNSUgIFsxXSAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAxMDAu MCUgIFsxXSAgICAgICAgICBkYXN0YXJ0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVu X2FsbG9jcQoKMDAuMzUlICBbMV0gICAgICAgIHNjaGVkX3BpY2tjcHUgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFdICAgICAg ICAgIHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0 CgowMC4zNSUgIFsxXSAgICAgICAgaW50cl9ldmVudF9oYW5kbGUgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwogIDEwMC4wJSAg WzFdICAgICAgICAgIGxhcGljX2hhbmRsZV9pbnRyCgowMC4zNSUgIFsxXSAgICAgICAgYm91bmNl X2J1c19kbWFtYXBfbG9hZF9idWZmZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIGJ1c19kbWFtYXBfbG9hZAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX21h cF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBtcnNhc19idWlsZF9kY2RiCgowMC4zNSUgIFsxXSAgICAgICAgc2NoZWRfcHJpb3JpdHkgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4w JSAgWzFdICAgICAgICAgIHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzbGVl cHFfYnJvYWRjYXN0CgowMC4zNSUgIFsxXSAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9r ZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAg MTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaXRocmVhZF9sb29wCgowMC4z NSUgIFsxXSAgICAgICAgYmlvZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgZGFkb25lCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfcHJvY2Vzcwog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2RvbmVfdGQKCkAgQ1BVX0NMS19VTkhBTFRFRF9D T1JFIFsyOTkgc2FtcGxlc10KCjEyLjM3JSAgWzM3XSAgICAgICBjcHVfaWRsZSBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzM3XSAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBb MzddICAgICAgICAgZm9ya19leGl0CgoxMC4zNyUgIFszMV0gICAgICAgaW52bHJuZ19oYW5kbGVy IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMTAuMDMlICBbMzBdICAgICAgIFhpbnZscm5nIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAoKMDYuMzUlICBbMTldICAgICAgIHNtcF90bGJfc2hvb3Rkb3duIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMTldICAgICAgICBwbWFwX2ludmFsaWRhdGVf cmFuZ2UKICAxMDAuMCUgIFsxOV0gICAgICAgICBiaW9kb25lCiAgIDEwMC4wJSAgWzE5XSAgICAg ICAgICBkYWRvbmUKCjA0LjY4JSAgWzE0XSAgICAgICBjcHVfc2VhcmNoX2hpZ2hlc3QgQCAvYm9v dC9rZXJuZWwva2VybmVsCiA3OC41NyUgIFsxMV0gICAgICAgIGNwdV9zZWFyY2hfaGlnaGVzdAog IDYzLjY0JSAgWzddICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFs3XSAgICAgICAg ICAgZm9ya19leGl0CiAgMzYuMzYlICBbNF0gICAgICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0CiAg IDEwMC4wJSAgWzRdICAgICAgICAgICBzY2hlZF9pZGxldGQKIDIxLjQzJSAgWzNdICAgICAgICAg c2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbM10gICAgICAgICAgZm9ya19leGl0CgowMy4wMSUgIFs5 XSAgICAgICAgY3B1X3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzldICAg ICAgICAgbWlfc3dpdGNoCiAgNjYuNjclICBbNl0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEw MC4wJSAgWzZdICAgICAgICAgICBmb3JrX2V4aXQKICAyMi4yMiUgIFsyXSAgICAgICAgICBzbGVl cHFfd2FpdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgX3NsZWVwCiAgMTEuMTElICBbMV0gICAg ICAgICAgY3JpdGljYWxfZXhpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9o YW5kbGUKCjAzLjAxJSAgWzldICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzldICAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QKICA4OC44OSUg IFs4XSAgICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdAogICAxMDAuMCUgIFs4XSAgICAgICAgICAg c2NoZWRfcGlja2NwdQogIDExLjExJSAgWzFdICAgICAgICAgIHNjaGVkX3BpY2tjcHUKICAgMTAw LjAlICBbMV0gICAgICAgICAgIHNjaGVkX2FkZAoKMDIuMzQlICBbN10gICAgICAgIF9fbXR4X2xv Y2tfc2xlZXAgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAyOC41NyUgIFsyXSAgICAgICAgIF9fbXR4 X2xvY2tfZmxhZ3MKICAxMDAuMCUgIFsyXSAgICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290 L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFsyXSAgICAgICAgICAgbXJzYXNfY29tcGxldGVf Y21kCiAxNC4yOSUgIFsxXSAgICAgICAgIHZtZW1feGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAgMTAwLjAlICBbMV0gICAgICAgICAgYmlvZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAg ZGFkb25lCiAxNC4yOSUgIFsxXSAgICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAg IHhwdF9kb25lX3Byb2Nlc3MKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3RkCiAx NC4yOSUgIFsxXSAgICAgICAgIGdldHBidWYKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8K ICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgogMTQuMjklICBbMV0gICAgICAg ICBfc2xlZXAKICAxMDAuMCUgIFsxXSAgICAgICAgICBid2FpdAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgcGh5c2lvCiAxNC4yOSUgIFsxXSAgICAgICAgIHZtZW1feGFsbG9jCiAgMTAwLjAlICBb MV0gICAgICAgICAgdm1lbV9hbGxvYwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZ19pb19jaGVj awoKMDIuMDElICBbNl0gICAgICAgIHNjaGVkX3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzZdICAgICAgICAgbWlfc3dpdGNoCiAgMzMuMzMlICBbMl0gICAgICAgICAgc2xl ZXBxX3dhaXQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIF9zbGVlcAogIDMzLjMzJSAgWzJdICAg ICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZm9ya19leGl0CiAg MTYuNjclICBbMV0gICAgICAgICAgdHVybnN0aWxlX3dhaXQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIF9fbXR4X2xvY2tfc2xlZXAKICAxNi42NyUgIFsxXSAgICAgICAgICBpdGhyZWFkX2xvb3AK ICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDEuNjclICBbNV0gICAgICAgIHRo cmVhZF9sb2NrX2ZsYWdzXyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDQwLjAwJSAgWzJdICAgICAg ICAgaXRocmVhZF9sb29wCiAgMTAwLjAlICBbMl0gICAgICAgICAgZm9ya19leGl0CiA0MC4wMCUg IFsyXSAgICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGZvcmtfZXhp dAogMjAuMDAlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X3NjaGVkdWxlX3RocmVhZAogIDEwMC4w JSAgWzFdICAgICAgICAgIGludHJfZXZlbnRfaGFuZGxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBpbnRyX2V4ZWN1dGVfaGFuZGxlcnMKCjAxLjY3JSAgWzVdICAgICAgICB2bWVtX3hhbGxvYyBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzVdICAgICAgICAgdm1lbV9hbGxvYwogIDEw MC4wJSAgWzVdICAgICAgICAgIGdfaW9fY2hlY2sKICAgMTAwLjAlICBbNV0gICAgICAgICAgIGdf aW9fcmVxdWVzdAoKMDEuNjclICBbNV0gICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDYwLjAwJSAgWzNdICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgMTAwLjAl ICBbM10gICAgICAgICAgZGFzdGFydAogICAxMDAuMCUgIFszXSAgICAgICAgICAgeHB0X3J1bl9h bGxvY3EKIDQwLjAwJSAgWzJdICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogIDEwMC4wJSAgWzJd ICAgICAgICAgIHhwdF9kb25lX3RkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3JrX2V4aXQK CjAxLjY3JSAgWzVdICAgICAgICBfX210eF9sb2NrX2ZsYWdzIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogNDAuMDAlICBbMl0gICAgICAgICBtcnNhc19maXJlX2NtZCBAIC9ib290L2tlcm5lbC9tcnNh cy5rbwogIDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9r ZXJuZWwKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogNDAuMDAl ICBbMl0gICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAu MCUgIFsyXSAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEw MC4wJSAgWzJdICAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKIDIwLjAwJSAgWzFdICAgICAg ICAgbXJzYXNfZ2V0X21wdF9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsx XSAgICAgICAgICBtcnNhc19hY3Rpb24KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5f ZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjM0JSAgWzRdICAgICAgICB4cHRfZG9uZV9w cm9jZXNzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNF0gICAgICAgICB4cHRfZG9u ZV90ZAogIDEwMC4wJSAgWzRdICAgICAgICAgIGZvcmtfZXhpdAoKMDEuMzQlICBbNF0gICAgICAg IGl0aHJlYWRfbG9vcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzRdICAgICAgICAg Zm9ya19leGl0CgowMS4zNCUgIFs0XSAgICAgICAgc2NoZWRfaWRsZXRkIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbNF0gICAgICAgICBmb3JrX2V4aXQKCjAxLjM0JSAgWzRdICAgICAg ICBYYXBpY19pc3IxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDEuMzQlICBbNF0gICAgICAgIHNw aW5sb2NrX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA3NS4wMCUgIFszXSAgICAgICAgIHNj aGVkX2lkbGV0ZAogIDEwMC4wJSAgWzNdICAgICAgICAgIGZvcmtfZXhpdAogMjUuMDAlICBbMV0g ICAgICAgICBzbXBfdGxiX3Nob290ZG93bgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBtYXBfaW52 YWxpZGF0ZV9yYW5nZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYmlvZG9uZQoKMDEuMDAlICBb M10gICAgICAgIF9fbG9ja21ncl9hcmdzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBb Ml0gICAgICAgICByZWxwYnVmCiAgMTAwLjAlICBbMl0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4w JSAgWzJdICAgICAgICAgICBkZXZmc19yZWFkX2YKIDMzLjMzJSAgWzFdICAgICAgICAgZ2V0cGJ1 ZgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAg ZGV2ZnNfcmVhZF9mCgowMS4wMCUgIFszXSAgICAgICAgdm1lbV94ZnJlZSBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgYmlvZG9uZQogIDEwMC4wJSAgWzNdICAgICAg ICAgIGRhZG9uZQogICAxMDAuMCUgIFszXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDEu MDAlICBbM10gICAgICAgIGtlcm5fd3JpdGV2IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbM10gICAgICAgICBzeXNfd3JpdGUKICAxMDAuMCUgIFszXSAgICAgICAgICBhbWQ2NF9zeXNj YWxsCgowMS4wMCUgIFszXSAgICAgICAgZG9yZXRpIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDEu MDAlICBbM10gICAgICAgIGRhc3RyYXRlZ3kgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFszXSAgICAgICAgIGdfZGlza19zdGFydAogIDEwMC4wJSAgWzNdICAgICAgICAgIGdfaW9fcmVx dWVzdAogICAxMDAuMCUgIFszXSAgICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwoKMDEuMDAlICBb M10gICAgICAgIF9fc3lzX3JlYWQgQCAvbGliL2xpYmMuc28uNwoKMDEuMDAlICBbM10gICAgICAg IGdldHBidWYgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIHBoeXNp bwogIDEwMC4wJSAgWzNdICAgICAgICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUgIFszXSAgICAg ICAgICAgZG9maWxlcmVhZAoKMDEuMDAlICBbM10gICAgICAgIHBoeXNpbyBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgMTAwLjAlICBbM10g ICAgICAgICAgZG9maWxlcmVhZAogICAxMDAuMCUgIFszXSAgICAgICAgICAga2Vybl9yZWFkdgoK MDEuMDAlICBbM10gICAgICAgIF9tdHhfbG9ja19zcGluX2Nvb2tpZSBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDMzLjMzJSAgWzFdICAgICAgICAgc21wX3RsYl9zaG9vdGRvd24KICAxMDAuMCUgIFsx XSAgICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UKICAgMTAwLjAlICBbMV0gICAgICAgICAg IGJpb2RvbmUKIDMzLjMzJSAgWzFdICAgICAgICAgc2NoZWRfYWRkCiAgMTAwLjAlICBbMV0gICAg ICAgICAgc2V0cnVubmFibGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNsZWVwcV9icm9hZGNh c3QKIDMzLjMzJSAgWzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMV0gICAgICAg ICAgZm9ya19leGl0CgowMS4wMCUgIFszXSAgICAgICAgZ190cmFjZSBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDMzLjMzJSAgWzFdICAgICAgICAgZ19pb19yZXF1ZXN0CiAgMTAwLjAlICBbMV0gICAg ICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCiAz My4zMyUgIFsxXSAgICAgICAgIGdfaW9fZGVsaXZlcgogIDEwMC4wJSAgWzFdICAgICAgICAgIGdf ZGlza19kb25lX3NpbmdsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFkb25lCiAzMy4zMyUg IFsxXSAgICAgICAgIGdfZGV2X3N0cmF0ZWd5CiAgMTAwLjAlICBbMV0gICAgICAgICAgZGV2X3N0 cmF0ZWd5X2NzdwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMS4wMCUgIFszXSAg ICAgICAgX2NhbGxvdXRfc3RvcF9zYWZlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb M10gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4w JSAgWzNdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFszXSAgICAgICAg ICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAu NjclICBbMl0gICAgICAgIGF0b21pY19mZXRjaGFkZF9pbnQgQCAvYm9vdC9rZXJuZWwva2VybmVs CiA1MC4wMCUgIFsxXSAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5r bwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogNTAuMDAlICBb MV0gICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAx MDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpdGhyZWFkX2xvb3AKCjAwLjY3 JSAgWzJdICAgICAgICBmcmVlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAg ICAgICB4cHRfcmVsZWFzZV9jY2IKICAxMDAuMCUgIFsyXSAgICAgICAgICBkYWRvbmUKICAgMTAw LjAlICBbMl0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjY3JSAgWzJdICAgICAgICBr ZXJuX3JlYWR2IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBzeXNf cmVhZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjY3JSAgWzJdICAg ICAgICBsb2NrX210eCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAg X3NsZWVwCiAgMTAwLjAlICBbMl0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMl0gICAgICAg ICAgIHBoeXNpbwoKMDAuNjclICBbMl0gICAgICAgIF9fbXR4X3VubG9ja19mbGFncyBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9v dC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9j bWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAg MTAwLjAlICBbMV0gICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkYXN0YXJ0CgowMC42NyUgIFsyXSAgICAgICAgZGFkb25lIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgMTAwLjAlICBb Ml0gICAgICAgICAgeHB0X2RvbmVfdGQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGZvcmtfZXhp dAoKMDAuNjclICBbMl0gICAgICAgIFhmYXN0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVs CgowMC42NyUgIFsyXSAgICAgICAgYW1kNjRfc3lzY2FsbCBAIC9ib290L2tlcm5lbC9rZXJuZWwK CjAwLjY3JSAgWzJdICAgICAgICBzY2hlZF9hZGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIHNldHJ1bm5hYmxlCiAgMTAwLjAlICBbMl0gICAgICAgICAgc2xlZXBx X2Jyb2FkY2FzdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgd2FrZXVwCgowMC42NyUgIFsyXSAg ICAgICAgZ19pb19jaGVjayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAg ICAgZ19pb19yZXF1ZXN0CiAgMTAwLjAlICBbMl0gICAgICAgICAgZGV2X3N0cmF0ZWd5X2Nzdwog ICAxMDAuMCUgIFsyXSAgICAgICAgICAgcGh5c2lvCgowMC42NyUgIFsyXSAgICAgICAgcG1hcF9l eHRyYWN0X2FuZF9ob2xkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAg ICB2bV9mYXVsdF9xdWlja19ob2xkX3BhZ2VzCiAgMTAwLjAlICBbMl0gICAgICAgICAgdm1hcGJ1 ZgogICAxMDAuMCUgIFsyXSAgICAgICAgICAgcGh5c2lvCgowMC42NyUgIFsyXSAgICAgICAgY3Jp dGljYWxfZW50ZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIHRo cmVhZF9sb2NrX2ZsYWdzXwogIDEwMC4wJSAgWzFdICAgICAgICAgIHNsZWVwcV9hZGQKICAgMTAw LjAlICBbMV0gICAgICAgICAgIF9zbGVlcAogNTAuMDAlICBbMV0gICAgICAgICB1bWFfemFsbG9j X2FyZwogIDEwMC4wJSAgWzFdICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAgMTAwLjAlICBb MV0gICAgICAgICAgIHBoeXNpbwoKMDAuNjclICBbMl0gICAgICAgIGxhcGljX2lwaV92ZWN0b3Jl ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgc21wX3RsYl9zaG9v dGRvd24KICAxMDAuMCUgIFsyXSAgICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UKICAgMTAw LjAlICBbMl0gICAgICAgICAgIGJpb2RvbmUKCjAwLjY3JSAgWzJdICAgICAgICBzZXRydW5uYWJs ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgc2xlZXBxX2Jyb2Fk Y2FzdAogIDEwMC4wJSAgWzJdICAgICAgICAgIHdha2V1cAogICAxMDAuMCUgIFsyXSAgICAgICAg ICAgYmRvbmUKCjAwLjY3JSAgWzJdICAgICAgICBjYWxsb3V0X2xvY2sgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGNhbGxvdXRfcmVzZXRfc2J0X29uCiAgMTAwLjAl ICBbMl0gICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEw MC4wJSAgWzJdICAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCgow MC4zMyUgIFsxXSAgICAgICAgZGV2ZnNfcmVhZF9mIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBkb2ZpbGVyZWFkCiAgMTAwLjAlICBbMV0gICAgICAgICAga2Vybl9y ZWFkdgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc3lzX3JlYWQKCjAwLjMzJSAgWzFdICAgICAg ICBzbGVlcHFfd2FpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg X3NsZWVwCiAgMTAwLjAlICBbMV0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIHBoeXNpbwoKMDAuMzMlICBbMV0gICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0IEAgL2Jvb3Qv a2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2FzX2J1aWxkX2RjZGIKICAx MDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19hY3Rpb24KICAgMTAwLjAlICBbMV0gICAgICAgICAg IHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjMzJSAgWzFdICAgICAgICBm Z2V0X3JlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGtlcm5f cmVhZHYKICAxMDAuMCUgIFsxXSAgICAgICAgICBzeXNfcmVhZAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzMlICBbMV0gICAgICAgIHhwdF9ydW5fYWxsb2NxIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkYXN0cmF0ZWd5CiAgMTAw LjAlICBbMV0gICAgICAgICAgZ19kaXNrX3N0YXJ0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBn X2lvX3JlcXVlc3QKCjAwLjMzJSAgWzFdICAgICAgICBid2FpdCBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAgMTAwLjAlICBbMV0gICAgICAgICAgZGV2 ZnNfcmVhZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkb2ZpbGVyZWFkCgowMC4zMyUgIFsx XSAgICAgICAgZGV2c3RhdF9lbmRfdHJhbnNhY3Rpb24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIGRldnN0YXRfZW5kX3RyYW5zYWN0aW9uX2Jpb19idAogIDEwMC4w JSAgWzFdICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZGFkb25lCgowMC4zMyUgIFsxXSAgICAgICAgdm1fZmF1bHRfcXVpY2tfaG9sZF9wYWdlcyBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgdm1hcGJ1ZgogIDEwMC4w JSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVh ZF9mCgowMC4zMyUgIFsxXSAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdyBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAgMTAwLjAlICBbMV0gICAgICAgICAg ZGV2ZnNfcmVhZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkb2ZpbGVyZWFkCgowMC4zMyUg IFsxXSAgICAgICAgeHB0X2RvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAg ICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBb MV0gICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBp bnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zMyUg IFsxXSAgICAgICAgc2xlZXBxX3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgc2xlZXBxX3dhaXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBfc2xlZXAKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGJ3YWl0CgowMC4zMyUgIFsxXSAgICAgICAgUEhZU19UT19W TV9QQUdFIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBmcmVlCiAg MTAwLjAlICBbMV0gICAgICAgICAgeHB0X3JlbGVhc2VfY2NiCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBkYWRvbmUKCjAwLjMzJSAgWzFdICAgICAgICBfZmdldCBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAga2Vybl9yZWFkdgogIDEwMC4wJSAgWzFdICAgICAgICAg IHN5c19yZWFkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zMyUg IFsxXSAgICAgICAgZ19pc19nZW9tX3RocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgZ19pb19kZWxpdmVyCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNr X2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjMzJSAgWzFd ICAgICAgICBnX2lvX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAg ICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzMlICBbMV0gICAgICAgIHhw dF9yZWxlYXNlX2NjYiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg ZGFkb25lCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogICAxMDAuMCUg IFsxXSAgICAgICAgICAgeHB0X2RvbmVfdGQKCjAwLjMzJSAgWzFdICAgICAgICBfX2NhcF9yaWdo dHNfaW5pdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAga2Vybl93 cml0ZXYKICAxMDAuMCUgIFsxXSAgICAgICAgICBzeXNfd3JpdGUKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjMzJSAgWzFdICAgICAgICBkYXN0YXJ0IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEwMC4w JSAgWzFdICAgICAgICAgIGRhc3RyYXRlZ3kKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfZGlz a19zdGFydAoKMDAuMzMlICBbMV0gICAgICAgIGNhbGxvdXRfY2NfYWRkIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBjYWxsb3V0X3Jlc2V0X3NidF9vbgogIDEwMC4w JSAgWzFdICAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAx MDAuMCUgIFsxXSAgICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoK MDAuMzMlICBbMV0gICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwog MTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAg MTAwLjAlICBbMV0gICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkYXN0YXJ0CgowMC4zMyUgIFsxXSAgICAgICAgdXNlcnJldCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzMlICBbMV0g ICAgICAgIHZtX3BhZ2VfdW5ob2xkX3BhZ2VzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICB2dW5tYXBidWYKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzMlICBbMV0gICAgICAgIF9f c3lzX3dyaXRlIEAgL2xpYi9saWJjLnNvLjcKCjAwLjMzJSAgWzFdICAgICAgICBidWZkb25lIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBidWZkb25lYmlvCiAgMTAw LjAlICBbMV0gICAgICAgICAgZ19pb19kZWxpdmVyCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBn X2Rpc2tfZG9uZV9zaW5nbGUKCjAwLjMzJSAgWzFdICAgICAgICBkb2ZpbGV3cml0ZSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAga2Vybl93cml0ZXYKICAxMDAuMCUg IFsxXSAgICAgICAgICBzeXNfd3JpdGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGFtZDY0X3N5 c2NhbGwKCjAwLjMzJSAgWzFdICAgICAgICBzY2hlZF9waWNrY3B1IEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAg ICBpbnRyX2V2ZW50X3NjaGVkdWxlX3RocmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50 cl9ldmVudF9oYW5kbGUKCjAwLjMzJSAgWzFdICAgICAgICBtcnNhc19kYXRhX2xvYWRfY2IgQCAv Ym9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgYnVzX2RtYW1hcF9sb2Fk IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX21hcF9y ZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBt cnNhc19idWlsZF9kY2RiCgowMC4zMyUgIFsxXSAgICAgICAgZ19kaXNrX3N0YXJ0IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUg IFsxXSAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBw aHlzaW8KCjAwLjMzJSAgWzFdICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzFdICAgICAg ICAgIHhwdF9kb25lX3Byb2Nlc3MKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3Rk CgowMC4zMyUgIFsxXSAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMu a28KIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgMTAwLjAlICBbMV0g ICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaXRocmVhZF9sb29wCgowMC4zMyUgIFsxXSAgICAg ICAgZ19kZXZfZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg Z19pb19kZWxpdmVyCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjMzJSAgWzFdICAgICAgICBydW5xX2Fk ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc2NoZWRfYWRkCiAg MTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVudF9zY2hlZHVsZV90aHJlYWQKICAgMTAwLjAl ICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfaGFuZGxlCgowMC4zMyUgIFsxXSAgICAgICAgY3Jp dGljYWxfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3Bp bmxvY2tfZXhpdAogIDEwMC4wJSAgWzFdICAgICAgICAgIHRocmVhZF9sb2NrX2Jsb2NrCiAgIDEw MC4wJSAgWzFdICAgICAgICAgICBzY2hlZF9zd2l0Y2gKCjAwLjMzJSAgWzFdICAgICAgICBzbGVl cHFfbG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgd2FrZXVw CiAgMTAwLjAlICBbMV0gICAgICAgICAgYmRvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ1 ZmRvbmUKCkAgQ1BVX0NMS19VTkhBTFRFRF9DT1JFIFsyOTQgc2FtcGxlc10KCjEzLjk1JSAgWzQx XSAgICAgICBYaW52bHJuZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA5Ljg2JSAgWzI5XSAgICAg ICBjcHVfaWRsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzI5XSAgICAgICAgc2No ZWRfaWRsZXRkCiAgMTAwLjAlICBbMjldICAgICAgICAgZm9ya19leGl0CgowOC41MCUgIFsyNV0g ICAgICAgaW52bHJuZ19oYW5kbGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDUuNzglICBbMTdd ICAgICAgIHNtcF90bGJfc2hvb3Rkb3duIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MTddICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UKICAxMDAuMCUgIFsxN10gICAgICAgICBi aW9kb25lCiAgIDEwMC4wJSAgWzE3XSAgICAgICAgICBkYWRvbmUKCjAzLjc0JSAgWzExXSAgICAg ICBjcHVfc3dpdGNoIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMTFdICAgICAgICBt aV9zd2l0Y2gKICA4MS44MiUgIFs5XSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBb OV0gICAgICAgICAgIGZvcmtfZXhpdAogIDE4LjE4JSAgWzJdICAgICAgICAgIHNsZWVwcV93YWl0 CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBfc2xlZXAKCjAzLjQwJSAgWzEwXSAgICAgICBjcHVf c2VhcmNoX2hpZ2hlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA4MC4wMCUgIFs4XSAgICAgICAg IGNwdV9zZWFyY2hfaGlnaGVzdAogIDUwLjAwJSAgWzRdICAgICAgICAgIGNwdV9zZWFyY2hfaGln aGVzdAogICAxMDAuMCUgIFs0XSAgICAgICAgICAgc2NoZWRfaWRsZXRkCiAgNTAuMDAlICBbNF0g ICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzRdICAgICAgICAgICBmb3JrX2V4aXQK IDIwLjAwJSAgWzJdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMl0gICAgICAgICAg Zm9ya19leGl0CgowMi43MiUgIFs4XSAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFs4XSAgICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0CiAgODcu NTAlICBbN10gICAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QKICAgMTAwLjAlICBbN10gICAgICAg ICAgIHNjaGVkX3BpY2tjcHUKICAxMi41MCUgIFsxXSAgICAgICAgICBzY2hlZF9waWNrY3B1CiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBzY2hlZF9hZGQKCjAyLjcyJSAgWzhdICAgICAgICBfX210 eF9sb2NrX3NsZWVwIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNzUuMDAlICBbNl0gICAgICAgICBf X210eF9sb2NrX2ZsYWdzCiAgODMuMzMlICBbNV0gICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAv Ym9vdC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbNV0gICAgICAgICAgIG1yc2FzX2NvbXBs ZXRlX2NtZAogIDE2LjY3JSAgWzFdICAgICAgICAgIG1yc2FzX3VubWFwX3JlcXVlc3QKICAgMTAw LjAlICBbMV0gICAgICAgICAgIG1yc2FzX2NtZF9kb25lCiAxMi41MCUgIFsxXSAgICAgICAgIHhw dF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsxXSAgICAgICAgICB4 cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RhcnQKIDEyLjUw JSAgWzFdICAgICAgICAgdm1lbV94ZnJlZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGJpb2RvbmUK ICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDIuNzIlICBbOF0gICAgICAgIHNjaGVk X3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzhdICAgICAgICAgbWlfc3dp dGNoCiAgMzcuNTAlICBbM10gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbM10gICAg ICAgICAgIF9zbGVlcAogIDM3LjUwJSAgWzNdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAu MCUgIFszXSAgICAgICAgICAgZm9ya19leGl0CiAgMjUuMDAlICBbMl0gICAgICAgICAgc2NoZWRf aWRsZXRkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3JrX2V4aXQKCjAyLjcyJSAgWzhdICAg ICAgICBzcGlubG9ja19leGl0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjIuNTAlICBbNV0gICAg ICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFs1XSAgICAgICAgICBmb3JrX2V4aXQKIDEyLjUw JSAgWzFdICAgICAgICAgX210eF9sb2NrX3NwaW5fY29va2llCiAgMTAwLjAlICBbMV0gICAgICAg ICAgc21wX3RsYl9zaG9vdGRvd24KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBtYXBfaW52YWxp ZGF0ZV9yYW5nZQogMTIuNTAlICBbMV0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsxXSAgICAg ICAgICBid2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCiAxMi41MCUgIFsxXSAg ICAgICAgIGNhbGxvdXRfcmVzZXRfc2J0X29uCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNf YWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4 cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMi4wNCUgIFs2XSAgICAgICAgX19z eXNfcmVhZCBAIC9saWIvbGliYy5zby43CgowMS4zNiUgIFs0XSAgICAgICAgZ19kZXZfc3RyYXRl Z3kgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAgICAgIGRldl9zdHJhdGVn eV9jc3cKICAxMDAuMCUgIFs0XSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBbNF0gICAgICAg ICAgIGRldmZzX3JlYWRfZgoKMDEuMzYlICBbNF0gICAgICAgIHNjaGVkX2lkbGV0ZCBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzRdICAgICAgICAgZm9ya19leGl0CgowMS4zNiUgIFs0 XSAgICAgICAgeHB0X2RvbmVfcHJvY2VzcyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzRdICAgICAgICAgeHB0X2RvbmVfdGQKICAxMDAuMCUgIFs0XSAgICAgICAgICBmb3JrX2V4aXQK CjAxLjM2JSAgWzRdICAgICAgICBYYXBpY19pc3IxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDEu MzYlICBbNF0gICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1 MC4wMCUgIFsyXSAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwog IDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwK ICAgMTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogMjUuMDAlICBbMV0g ICAgICAgICBtcnNhc19nZXRfbXB0X2NtZCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4w JSAgWzFdICAgICAgICAgIG1yc2FzX2FjdGlvbgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0 X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMjUuMDAlICBbMV0gICAgICAgICBtcnNh c191bm1hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAg ICAgICAgbXJzYXNfY21kX2RvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIG1yc2FzX2NvbXBs ZXRlX2NtZAoKMDEuMDIlICBbM10gICAgICAgIHNjaGVkX3BpY2tjcHUgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIHNjaGVkX2FkZAogIDY2LjY3JSAgWzJdICAgICAg ICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBp bnRyX2V2ZW50X2hhbmRsZQogIDMzLjMzJSAgWzFdICAgICAgICAgIHR1cm5zdGlsZV91bnBlbmQK ICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9fbXR4X3VubG9ja19zbGVlcAoKMDEuMDIlICBbM10g ICAgICAgIF9tdHhfbG9ja19zcGluX2Nvb2tpZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2LjY3 JSAgWzJdICAgICAgICAgdHVybnN0aWxlX3RyeXdhaXQKICAxMDAuMCUgIFsyXSAgICAgICAgICBf X210eF9sb2NrX3NsZWVwCiAgIDUwLjAwJSAgWzFdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNz CiAgIDUwLjAwJSAgWzFdICAgICAgICAgICB2bWVtX3hhbGxvYwogMzMuMzMlICBbMV0gICAgICAg ICBzY2hlZF9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X3NjaGVkdWxlX3Ro cmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKCjAxLjAyJSAg WzNdICAgICAgICBsYXBpY19pcGlfdmVjdG9yZWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFszXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbM10gICAgICAgICAg cG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBiaW9kb25lCgow MS4wMiUgIFszXSAgICAgICAgdW1hX3pmcmVlX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzNdICAgICAgICAgZnJlZQogIDEwMC4wJSAgWzNdICAgICAgICAgIHhwdF9yZWxlYXNl X2NjYgogICAxMDAuMCUgIFszXSAgICAgICAgICAgZGFkb25lCgowMS4wMiUgIFszXSAgICAgICAg dGhyZWFkX2xvY2tfZmxhZ3NfIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBbMl0gICAg ICAgICBpbnRyX2V2ZW50X3NjaGVkdWxlX3RocmVhZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGlu dHJfZXZlbnRfaGFuZGxlCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBpbnRyX2V4ZWN1dGVfaGFu ZGxlcnMKIDMzLjMzJSAgWzFdICAgICAgICAgc2xlZXBxX3dhaXQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ3YWl0CgowMS4wMiUgIFszXSAg ICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFszXSAg ICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFszXSAg ICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbM10gICAgICAgICAgIGRhc3Rh cnQKCjAxLjAyJSAgWzNdICAgICAgICB4cHRfcnVuX2FsbG9jcSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZGFzdHJhdGVneQogIDEwMC4wJSAgWzNdICAgICAgICAg IGdfZGlza19zdGFydAogICAxMDAuMCUgIFszXSAgICAgICAgICAgZ19pb19yZXF1ZXN0CgowMS4w MiUgIFszXSAgICAgICAgX19tdHhfdW5sb2NrX2ZsYWdzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog NjYuNjclICBbMl0gICAgICAgICB4cHRfcnVuX2RldnEKICAxMDAuMCUgIFsyXSAgICAgICAgICB4 cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRhc3RhcnQKIDMzLjMz JSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAx MDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwK CjAxLjAyJSAgWzNdICAgICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzNdICAgICAgICAgdm1lbV9hbGxvYwogIDEwMC4wJSAgWzNdICAgICAgICAgIGdfaW9f Y2hlY2sKICAgMTAwLjAlICBbM10gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDEuMDIlICBbM10g ICAgICAgIGRhc3RhcnQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAg IHhwdF9ydW5fYWxsb2NxCiAgMTAwLjAlICBbM10gICAgICAgICAgZGFzdHJhdGVneQogICAxMDAu MCUgIFszXSAgICAgICAgICAgZ19kaXNrX3N0YXJ0CgowMS4wMiUgIFszXSAgICAgICAgYW1kNjRf c3lzY2FsbCBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjAyJSAgWzNdICAgICAgICB4cHRfcnVu X2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIHhwdF9hY3Rp b25fZGVmYXVsdAogIDEwMC4wJSAgWzNdICAgICAgICAgIGRhc3RhcnQKICAgMTAwLjAlICBbM10g ICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCgowMC42OCUgIFsyXSAgICAgICAgc2NoZWRfY2hvb3Nl IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBjaG9vc2V0aHJlYWQK ICAxMDAuMCUgIFsyXSAgICAgICAgICBzY2hlZF9zd2l0Y2gKICAgMTAwLjAlICBbMl0gICAgICAg ICAgIG1pX3N3aXRjaAoKMDAuNjglICBbMl0gICAgICAgIGRvcmV0aSBAIC9ib290L2tlcm5lbC9r ZXJuZWwKCjAwLjY4JSAgWzJdICAgICAgICBzY3NpX3JlYWRfd3JpdGUgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGRhc3RhcnQKICAxMDAuMCUgIFsyXSAgICAgICAg ICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZGFzdHJhdGVneQoKMDAu NjglICBbMl0gICAgICAgIF9tdHhfdHJ5bG9ja19mbGFnc18gQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsyXSAgICAgICAgIHZtX3BhZ2VfcGFfdHJ5cmVsb2NrCiAgMTAwLjAlICBbMl0g ICAgICAgICAgcG1hcF9leHRyYWN0X2FuZF9ob2xkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICB2 bV9mYXVsdF9xdWlja19ob2xkX3BhZ2VzCgowMC42OCUgIFsyXSAgICAgICAgX2ZnZXQgQCAvYm9v dC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIGtlcm5fd3JpdGV2CiAgMTAwLjAl ICBbMV0gICAgICAgICAgc3lzX3dyaXRlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBhbWQ2NF9z eXNjYWxsCiA1MC4wMCUgIFsxXSAgICAgICAgIGtlcm5fcmVhZHYKICAxMDAuMCUgIFsxXSAgICAg ICAgICBzeXNfcmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAu NjglICBbMl0gICAgICAgIHBoeXNpbyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJd ICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgMTAwLjAlICBbMl0gICAgICAgICAgZG9maWxlcmVhZAog ICAxMDAuMCUgIFsyXSAgICAgICAgICAga2Vybl9yZWFkdgoKMDAuNjglICBbMl0gICAgICAgIHJ1 bnFfY2hvb3NlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBzY2hl ZF9jaG9vc2UKICAxMDAuMCUgIFsyXSAgICAgICAgICBjaG9vc2V0aHJlYWQKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIHNjaGVkX3N3aXRjaAoKMDAuNjglICBbMl0gICAgICAgIGJjb3B5IEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICB4cHRfcnVuX2RldnEKICAxMDAu MCUgIFsxXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRhc3RhcnQKIDUwLjAwJSAgWzFdICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuNjglICBb Ml0gICAgICAgIHZtZW1feGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAg ICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFsyXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjY4JSAgWzJdICAgICAgICBidF9mcmVl dHJpbSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgYmlvZG9uZQog IDEwMC4wJSAgWzJdICAgICAgICAgIGRhZG9uZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgeHB0 X2RvbmVfcHJvY2VzcwoKMDAuNjglICBbMl0gICAgICAgIGRldnN0YXRfc3RhcnRfdHJhbnNhY3Rp b25fYmlvIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2Rpc2tf c3RhcnQKICAxMDAuMCUgIFsyXSAgICAgICAgICBnX2lvX3JlcXVlc3QKICAgMTAwLjAlICBbMl0g ICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKCjAwLjY4JSAgWzJdICAgICAgICBiemVybyBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jv b3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9kZXZxIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2FjdGlvbl9k ZWZhdWx0CiA1MC4wMCUgIFsxXSAgICAgICAgIF9fY2FwX3JpZ2h0c19pbml0CiAgMTAwLjAlICBb MV0gICAgICAgICAga2Vybl9yZWFkdgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc3lzX3JlYWQK CjAwLjY4JSAgWzJdICAgICAgICBzcGlubG9ja19lbnRlciBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDUwLjAwJSAgWzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMV0gICAgICAgICAg Zm9ya19leGl0CiA1MC4wMCUgIFsxXSAgICAgICAgIGNhbGxvdXRfbG9jawogIDEwMC4wJSAgWzFd ICAgICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJz YXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KCjAwLjY4JSAgWzJdICAgICAgICBz bGVlcHFfbG9jayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgX3Ns ZWVwCiAgMTAwLjAlICBbMl0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMl0gICAgICAgICAg IHBoeXNpbwoKMDAuNjglICBbMl0gICAgICAgIGNyaXRpY2FsX2V4aXQgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIHNwaW5sb2NrX2V4aXQKICA1MC4wMCUgIFsxXSAg ICAgICAgICB0dXJuc3RpbGVfd2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgX19tdHhfbG9j a19zbGVlcAogIDUwLjAwJSAgWzFdICAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFk CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQoKMDAuNjglICBbMl0g ICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzJdICAgICAgICAgYmlvZG9uZQogIDEwMC4wJSAgWzJdICAgICAgICAgIGRhZG9uZQogICAx MDAuMCUgIFsyXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuNjglICBbMl0gICAgICAg IGRhc3RyYXRlZ3kgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGdf ZGlza19zdGFydAogIDEwMC4wJSAgWzJdICAgICAgICAgIGdfaW9fcmVxdWVzdAogICAxMDAuMCUg IFsyXSAgICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwoKMDAuMzQlICBbMV0gICAgICAgIHRkcV9u b3RpZnkgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2Fk ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBzbGVlcHFfYnJvYWRjYXN0CgowMC4zNCUgIFsxXSAgICAgICAgaW50cl9leGVjdXRlX2hh bmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBsYXBpY19o YW5kbGVfaW50cgoKMDAuMzQlICBbMV0gICAgICAgIGl0aHJlYWRfbG9vcCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAg ICAgbXJzYXNfZ2V0X21wdF9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFd ICAgICAgICAgbXJzYXNfYWN0aW9uCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9kZXZx IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2FjdGlv bl9kZWZhdWx0CgowMC4zNCUgIFsxXSAgICAgICAgc2V0cnVubmFibGUgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNsZWVwcV9icm9hZGNhc3QKICAxMDAuMCUgIFsx XSAgICAgICAgICB3YWtldXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJkb25lCgowMC4zNCUg IFsxXSAgICAgICAgbWlfc3dpdGNoIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0g ICAgICAgICBzbGVlcHFfd2FpdAogIDEwMC4wJSAgWzFdICAgICAgICAgIF9zbGVlcAogICAxMDAu MCUgIFsxXSAgICAgICAgICAgYndhaXQKCjAwLjM0JSAgWzFdICAgICAgICBpcGlfYml0bWFwX2hh bmRsZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNCUgIFsxXSAgICAgICAgZGV2ZnNfcmVh ZF9mIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkb2ZpbGVyZWFk CiAgMTAwLjAlICBbMV0gICAgICAgICAga2Vybl9yZWFkdgogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgc3lzX3JlYWQKCjAwLjM0JSAgWzFdICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzFd ICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9k b25lX3RkCgowMC4zNCUgIFsxXSAgICAgICAga2Vybl9yZWFkdiBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3lzX3JlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBh bWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgc2NoZWRfcnVubmFibGUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGNwdV9pZGxlCiAgMTAwLjAlICBbMV0g ICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQK CjAwLjM0JSAgWzFdICAgICAgICBfX2xvY2ttZ3JfYXJncyBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzFdICAgICAgICAgcmVscGJ1ZgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNp bwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAg ICAgdm1fcGFnZV91bmhvbGRfcGFnZXMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIHZ1bm1hcGJ1ZgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAu MCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgX19jYXBf cmlnaHRzX2lzX3NldCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg ZmdldF91bmxvY2tlZAogIDEwMC4wJSAgWzFdICAgICAgICAgIF9mZ2V0CiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBrZXJuX3JlYWR2CgowMC4zNCUgIFsxXSAgICAgICAgYm91bmNlX2J1c19kbWFt YXBfbG9hZF9idWZmZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAg IGJ1c19kbWFtYXBfbG9hZAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0 IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtcnNhc19i dWlsZF9kY2RiCgowMC4zNCUgIFsxXSAgICAgICAgbXJzYXNfY29tcGxldGVfY21kIEAgL2Jvb3Qv a2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsxXSAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9o YW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsxXSAgICAgICAgICBpdGhy ZWFkX2xvb3AKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0g ICAgICAgIGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHhw dF9yZWxlYXNlX2NjYgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRhZG9uZQogICAxMDAuMCUgIFsx XSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuMzQlICBbMV0gICAgICAgIGludHJfZXZl bnRfc2NoZWR1bGVfdGhyZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGludHJfZXhlY3V0 ZV9oYW5kbGVycwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbGFwaWNfaGFuZGxlX2ludHIKCjAw LjM0JSAgWzFdICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwog MTAwLjAlICBbMV0gICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBpdGhyZWFkX2xvb3AKCjAwLjM0JSAgWzFdICAgICAgICBj YWxsb3V0X2xvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIF9j YWxsb3V0X3N0b3Bfc2FmZQogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAg L2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtcnNhc19jb21w bGV0ZV9jbWQKCjAwLjM0JSAgWzFdICAgICAgICBiaW9xX3Rha2VmaXJzdCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGFzdGFydAogIDEwMC4wJSAgWzFdICAgICAg ICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5Cgow MC4zNCUgIFsxXSAgICAgICAgZm9mZnNldF91bmxvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIGRldmZzX3dyaXRlX2YKICAxMDAuMCUgIFsxXSAgICAgICAgICBk b2ZpbGV3cml0ZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAga2Vybl93cml0ZXYKCjAwLjM0JSAg WzFdICAgICAgICBkYWRvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIHhwdF9kb25lX3Byb2Nlc3MKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfZG9uZV90ZAog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgZ2V0 cGJ1ZiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAg MTAwLjAlICBbMV0gICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBkb2ZpbGVyZWFkCgowMC4zNCUgIFsxXSAgICAgICAgWHRpbWVyaW50IEAgL2Jvb3Qva2VybmVs L2tlcm5lbAoKMDAuMzQlICBbMV0gICAgICAgIHNsZWVwcV9hZGQgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3 YWl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBy ZWxwYnVmIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBwaHlzaW8K ICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRvZmlsZXJlYWQKCjAwLjM0JSAgWzFdICAgICAgICBkZXZzdGF0X2VuZF90cmFuc2FjdGlv bl9iaW9fYnQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGdfaW9f ZGVsaXZlcgogIDEwMC4wJSAgWzFdICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQogICAxMDAu MCUgIFsxXSAgICAgICAgICAgZGFkb25lCgowMC4zNCUgIFsxXSAgICAgICAgZmdldF91bmxvY2tl ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgX2ZnZXQKICAxMDAu MCUgIFsxXSAgICAgICAgICBrZXJuX3dyaXRldgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc3lz X3dyaXRlCgowMC4zNCUgIFsxXSAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdyBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAgMTAwLjAlICBbMV0gICAgICAg ICAgZGV2ZnNfcmVhZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkb2ZpbGVyZWFkCgowMC4z NCUgIFsxXSAgICAgICAgdW1hX3phbGxvY19hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIG1hbGxvYwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fYWxs b2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5CgowMC4zNCUgIFsxXSAgICAg ICAgY2FtcV9yZW1vdmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAg IHhwdF9ydW5fZGV2cQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdGFydAoKMDAuMzQlICBbMV0gICAgICAgIGNwdV9m ZXRjaF9zeXNjYWxsX2FyZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIGFtZDY0X3N5c2NhbGwKCkAgQ1BVX0NMS19VTkhBTFRFRF9DT1JFIFsyOTMgc2FtcGxlc10K CjEzLjY1JSAgWzQwXSAgICAgICBjcHVfaWRsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzQwXSAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbNDBdICAgICAgICAgZm9ya19l eGl0CgoxMS42MCUgIFszNF0gICAgICAgWGludmxybmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCgow Ni40OCUgIFsxOV0gICAgICAgc21wX3RsYl9zaG9vdGRvd24gQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxOV0gICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzE5 XSAgICAgICAgIGJpb2RvbmUKICAgMTAwLjAlICBbMTldICAgICAgICAgIGRhZG9uZQoKMDYuMTQl ICBbMThdICAgICAgIGludmxybmdfaGFuZGxlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAzLjA3 JSAgWzldICAgICAgICBjcHVfc2VhcmNoX2hpZ2hlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA4 OC44OSUgIFs4XSAgICAgICAgIGNwdV9zZWFyY2hfaGlnaGVzdAogIDg3LjUwJSAgWzddICAgICAg ICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFs3XSAgICAgICAgICAgZm9ya19leGl0CiAgMTIu NTAlICBbMV0gICAgICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0CiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBzY2hlZF9pZGxldGQKIDExLjExJSAgWzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAw LjAlICBbMV0gICAgICAgICAgZm9ya19leGl0CgowMi43MyUgIFs4XSAgICAgICAgY3B1X3NlYXJj aF9sb3dlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs4XSAgICAgICAgIGNwdV9z ZWFyY2hfbG93ZXN0CiAgODcuNTAlICBbN10gICAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QKICAg MTAwLjAlICBbN10gICAgICAgICAgIHNjaGVkX3BpY2tjcHUKICAxMi41MCUgIFsxXSAgICAgICAg ICBzY2hlZF9waWNrY3B1CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzY2hlZF9hZGQKCjAyLjM5 JSAgWzddICAgICAgICBfX210eF9sb2NrX3NsZWVwIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogODUu NzElICBbNl0gICAgICAgICBfX210eF9sb2NrX2ZsYWdzCiAgODMuMzMlICBbNV0gICAgICAgICAg bXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbNV0gICAg ICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogIDE2LjY3JSAgWzFdICAgICAgICAgIG1yc2FzX3Vu bWFwX3JlcXVlc3QKICAgMTAwLjAlICBbMV0gICAgICAgICAgIG1yc2FzX2NtZF9kb25lCiAxNC4y OSUgIFsxXSAgICAgICAgIHZtZW1feGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAl ICBbMV0gICAgICAgICAgYmlvZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFkb25lCgow Mi4zOSUgIFs3XSAgICAgICAgY3B1X3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzddICAgICAgICAgbWlfc3dpdGNoCiAgMTAwLjAlICBbN10gICAgICAgICAgc2NoZWRfaWRs ZXRkCiAgIDEwMC4wJSAgWzddICAgICAgICAgICBmb3JrX2V4aXQKCjAyLjA1JSAgWzZdICAgICAg ICBzY2hlZF9pZGxldGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs2XSAgICAgICAg IGZvcmtfZXhpdAoKMDIuMDUlICBbNl0gICAgICAgIF9fc3lzX3JlYWQgQCAvbGliL2xpYmMuc28u NwoKMDEuNzElICBbNV0gICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiA2MC4wMCUgIFszXSAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21y c2FzLmtvCiAgMTAwLjAlICBbM10gICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4w JSAgWzNdICAgICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAyMC4wMCUgIFsxXSAgICAgICAgIG1yc2FzX2dldF9tcHRfY21kIEAgL2Jvb3Qv a2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfYWN0aW9uCiAgIDEw MC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAy MC4wMCUgIFsxXSAgICAgICAgIG1yc2FzX3VubWFwX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwvbXJz YXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jbWRfZG9uZQogICAxMDAuMCUgIFsx XSAgICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCgowMS43MSUgIFs1XSAgICAgICAgc3Bpbmxv Y2tfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDIwLjAwJSAgWzFdICAgICAgICAgX3NsZWVw CiAgMTAwLjAlICBbMV0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBo eXNpbwogMjAuMDAlICBbMV0gICAgICAgICBpdGhyZWFkX2xvb3AKICAxMDAuMCUgIFsxXSAgICAg ICAgICBmb3JrX2V4aXQKIDIwLjAwJSAgWzFdICAgICAgICAgdm1lbV94ZnJlZQogIDEwMC4wJSAg WzFdICAgICAgICAgIGJpb2RvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQogMjAu MDAlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X3NjaGVkdWxlX3RocmVhZAogIDEwMC4wJSAgWzFd ICAgICAgICAgIGludHJfZXZlbnRfaGFuZGxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRy X2V4ZWN1dGVfaGFuZGxlcnMKIDIwLjAwJSAgWzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAw LjAlICBbMV0gICAgICAgICAgZm9ya19leGl0CgowMS4zNyUgIFs0XSAgICAgICAgdGhyZWFkX2xv Y2tfZmxhZ3NfIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMjUuMDAlICBbMV0gICAgICAgICBzY2hl ZF9pZGxldGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBmb3JrX2V4aXQKIDI1LjAwJSAgWzFdICAg ICAgICAgaXRocmVhZF9sb29wCiAgMTAwLjAlICBbMV0gICAgICAgICAgZm9ya19leGl0CiAyNS4w MCUgIFsxXSAgICAgICAgIGNyaXRpY2FsX2V4aXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRy X2V2ZW50X2hhbmRsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRs ZXJzCiAyNS4wMCUgIFsxXSAgICAgICAgIHNsZWVwcV9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAg ICBfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ3YWl0CgowMS4zNyUgIFs0XSAgICAg ICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNF0gICAgICAg ICB4cHRfYWN0aW9uX2RlZmF1bHQKICAxMDAuMCUgIFs0XSAgICAgICAgICBkYXN0YXJ0CiAgIDEw MC4wJSAgWzRdICAgICAgICAgICB4cHRfcnVuX2FsbG9jcQoKMDEuMzclICBbNF0gICAgICAgIHBt YXBfa2V4dHJhY3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAgICAgIGJv dW5jZV9idXNfZG1hbWFwX2xvYWRfYnVmZmVyCiAgMTAwLjAlICBbNF0gICAgICAgICAgYnVzX2Rt YW1hcF9sb2FkCiAgIDEwMC4wJSAgWzRdICAgICAgICAgICBtcnNhc19tYXBfcmVxdWVzdCBAIC9i b290L2tlcm5lbC9tcnNhcy5rbwoKMDEuMzclICBbNF0gICAgICAgIHNsZWVwcV9sb2NrIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMl0gICAgICAgICB3YWtldXAKICAxMDAuMCUgIFsy XSAgICAgICAgICBiZG9uZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgYnVmZG9uZQogMjUuMDAl ICBbMV0gICAgICAgICBjdl9icm9hZGNhc3RwcmkKICAxMDAuMCUgIFsxXSAgICAgICAgICB2bWVt X3hmcmVlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiaW9kb25lCiAyNS4wMCUgIFsxXSAgICAg ICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0CiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBwaHlzaW8KCjAxLjM3JSAgWzRdICAgICAgICBfX210eF91bmxvY2tfZmxhZ3MgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAgICAgIHhwdF9ydW5fZGV2cQogIDEw MC4wJSAgWzRdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFs0XSAgICAg ICAgICAgZGFzdGFydAoKMDEuMzclICBbNF0gICAgICAgIHNjaGVkX3N3aXRjaCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzRdICAgICAgICAgbWlfc3dpdGNoCiAgNzUuMDAlICBbM10g ICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbM10gICAgICAgICAgIF9zbGVlcAogIDI1 LjAwJSAgWzFdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAg Zm9ya19leGl0CgowMS4wMiUgIFszXSAgICAgICAgc3BpbmxvY2tfZW50ZXIgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAzMy4zMyUgIFsxXSAgICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFd ICAgICAgICAgIGZvcmtfZXhpdAogMzMuMzMlICBbMV0gICAgICAgICBjYWxsb3V0X2xvY2sKICAx MDAuMCUgIFsxXSAgICAgICAgICBjYWxsb3V0X3Jlc2V0X3NidF9vbgogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAzMy4zMyUgIFsx XSAgICAgICAgIHNsZWVwcV9sb2NrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzFd ICAgICAgICAgIF9zbGVlcAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYndhaXQKCjAxLjAyJSAg WzNdICAgICAgICBjcml0aWNhbF9leGl0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBb Ml0gICAgICAgICBzcGlubG9ja19leGl0CiAgNTAuMDAlICBbMV0gICAgICAgICAgdGhyZWFkX2xv Y2tfYmxvY2sKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNjaGVkX2FkZAogIDUwLjAwJSAgWzFd ICAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBpbnRyX2V2ZW50X2hhbmRsZQogMzMuMzMlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X2hh bmRsZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwogICAxMDAu MCUgIFsxXSAgICAgICAgICAgbGFwaWNfaGFuZGxlX2ludHIKCjAxLjAyJSAgWzNdICAgICAgICB2 bWVtX3hmcmVlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICBiaW9k b25lCiAgMTAwLjAlICBbM10gICAgICAgICAgZGFkb25lCiAgIDEwMC4wJSAgWzNdICAgICAgICAg ICB4cHRfZG9uZV9wcm9jZXNzCgowMS4wMiUgIFszXSAgICAgICAgZGV2c3RhdF9lbmRfdHJhbnNh Y3Rpb24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIGRldnN0YXRf ZW5kX3RyYW5zYWN0aW9uX2Jpb19idAogIDEwMC4wJSAgWzNdICAgICAgICAgIGdfZGlza19kb25l X3NpbmdsZQogICAxMDAuMCUgIFszXSAgICAgICAgICAgZGFkb25lCgowMS4wMiUgIFszXSAgICAg ICAgbGFwaWNfaXBpX3ZlY3RvcmVkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10g ICAgICAgICBzbXBfdGxiX3Nob290ZG93bgogIDEwMC4wJSAgWzNdICAgICAgICAgIHBtYXBfaW52 YWxpZGF0ZV9yYW5nZQogICAxMDAuMCUgIFszXSAgICAgICAgICAgYmlvZG9uZQoKMDEuMDIlICBb M10gICAgICAgIFhhcGljX2lzcjEgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC42OCUgIFsyXSAg ICAgICAgeHB0X3J1bl9hbGxvY3EgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAg ICAgICAgIGRhc3RyYXRlZ3kKICAxMDAuMCUgIFsyXSAgICAgICAgICBnX2Rpc2tfc3RhcnQKICAg MTAwLjAlICBbMl0gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDAuNjglICBbMl0gICAgICAgIGdf ZGV2X3N0cmF0ZWd5IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBk ZXZfc3RyYXRlZ3lfY3N3CiAgMTAwLjAlICBbMl0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAg WzJdICAgICAgICAgICBkZXZmc19yZWFkX2YKCjAwLjY4JSAgWzJdICAgICAgICBwaHlzaW8gQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGRldmZzX3JlYWRfZgogIDEw MC4wJSAgWzJdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGtl cm5fcmVhZHYKCjAwLjY4JSAgWzJdICAgICAgICBnX2lvX2NoZWNrIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFsyXSAgICAg ICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBwaHlzaW8KCjAw LjY4JSAgWzJdICAgICAgICBjYWxsb3V0X2xvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogIDEwMC4wJSAgWzJdICAgICAgICAg IG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzJdICAg ICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKCjAwLjY4JSAgWzJdICAgICAgICByZWxwYnVmIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBwaHlzaW8KICAxMDAuMCUg IFsyXSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRvZmls ZXJlYWQKCjAwLjY4JSAgWzJdICAgICAgICBpdGhyZWFkX2xvb3AgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGZvcmtfZXhpdAoKMDAuNjglICBbMl0gICAgICAgIHRk cV9tb3ZlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBzY2hlZF9p ZGxldGQKICAxMDAuMCUgIFsyXSAgICAgICAgICBmb3JrX2V4aXQKCjAwLjY4JSAgWzJdICAgICAg ICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAg dm1lbV9hbGxvYwogIDEwMC4wJSAgWzJdICAgICAgICAgIGdfaW9fY2hlY2sKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDAuNjglICBbMl0gICAgICAgIG1yc2FzX2FjdGlv biBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbMl0gICAgICAgICB4cHRfcnVuX2Rl dnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMl0gICAgICAgICAgeHB0X2FjdGlv bl9kZWZhdWx0CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkYXN0YXJ0CgowMC42OCUgIFsyXSAg ICAgICAgbXJzYXNfaXNyIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsyXSAgICAg ICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAx MDAuMCUgIFsyXSAgICAgICAgICBpdGhyZWFkX2xvb3AKICAgMTAwLjAlICBbMl0gICAgICAgICAg IGZvcmtfZXhpdAoKMDAuNjglICBbMl0gICAgICAgIHByb3BhZ2F0ZV9wcmlvcml0eSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgdHVybnN0aWxlX3dhaXQKICAxMDAu MCUgIFsyXSAgICAgICAgICBfX210eF9sb2NrX3NsZWVwCiAgIDUwLjAwJSAgWzFdICAgICAgICAg ICBnX2Rpc2tfc3RhcnQKICAgNTAuMDAlICBbMV0gICAgICAgICAgIHJlbHBidWYKCjAwLjY4JSAg WzJdICAgICAgICBzY2hlZF9waWNrY3B1IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb Ml0gICAgICAgICBzY2hlZF9hZGQKICA1MC4wMCUgIFsxXSAgICAgICAgICBzZXRydW5uYWJsZQog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xlZXBxX2Jyb2FkY2FzdAogIDUwLjAwJSAgWzFdICAg ICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBpbnRyX2V2ZW50X2hhbmRsZQoKMDAuNjglICBbMl0gICAgICAgIHNjaGVkX2Nob29zZSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgY2hvb3NldGhyZWFkCiAgMTAw LjAlICBbMl0gICAgICAgICAgc2NoZWRfc3dpdGNoCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBt aV9zd2l0Y2gKCjAwLjY4JSAgWzJdICAgICAgICBnX2Rldl9kb25lIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAuMCUgIFsyXSAgICAg ICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRhZG9uZQoK MDAuNjglICBbMl0gICAgICAgIGRvcmV0aSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjY4JSAg WzJdICAgICAgICBtcnNhc19nZXRfbXB0X2NtZCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAw LjAlICBbMl0gICAgICAgICBtcnNhc19hY3Rpb24KICAxMDAuMCUgIFsyXSAgICAgICAgICB4cHRf cnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICB4 cHRfYWN0aW9uX2RlZmF1bHQKCjAwLjY4JSAgWzJdICAgICAgICBkYXN0cmF0ZWd5IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUg IFsyXSAgICAgICAgICBnX2lvX3JlcXVlc3QKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRldl9z dHJhdGVneV9jc3cKCjAwLjY4JSAgWzJdICAgICAgICBzbGVlcHFfYnJvYWRjYXN0IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB3YWtldXAKICAxMDAuMCUgIFsyXSAg ICAgICAgICBiZG9uZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgYnVmZG9uZQoKMDAuNjglICBb Ml0gICAgICAgIGJpb2RvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAg ICAgIGRhZG9uZQogIDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAgMTAw LjAlICBbMl0gICAgICAgICAgIHhwdF9kb25lX3RkCgowMC4zNCUgIFsxXSAgICAgICAgZ19pb19y ZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZfc3Ry YXRlZ3lfY3N3CiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkZXZmc19yZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICBjYW1xX3JlbW92ZSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgY2FtX2NjYnFfcmVtb3ZlX2Nj YgogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fZGV2cQogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMC4zNCUgIFsxXSAgICAgICAgY3B1X2ZldGNoX3N5 c2NhbGxfYXJncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1k NjRfc3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIG1yc2FzX3VubWFwX3JlcXVlc3QgQCAvYm9v dC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUKICAx MDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwK CjAwLjM0JSAgWzFdICAgICAgICB1c2VycmV0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgbXJzYXNfbWFw X21wdF9jbWRfc3RhdHVzIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsxXSAgICAg ICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGludHJfZXZlbnRf ZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGl0aHJlYWRfbG9vcAoKMDAuMzQlICBbMV0gICAgICAgIGRldnZuX3JlZnRocmVhZCBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2ZnNfd3JpdGVfZgog IDEwMC4wJSAgWzFdICAgICAgICAgIGRvZmlsZXdyaXRlCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBrZXJuX3dyaXRldgoKMDAuMzQlICBbMV0gICAgICAgIHNldHJ1bm5hYmxlIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAgMTAwLjAl ICBbMV0gICAgICAgICAgd2FrZXVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiZG9uZQoKMDAu MzQlICBbMV0gICAgICAgIFhmYXN0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4z NCUgIFsxXSAgICAgICAgcG1hcF9leHRyYWN0X2FuZF9ob2xkIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMV0gICAgICAgICB2bV9mYXVsdF9xdWlja19ob2xkX3BhZ2VzCiAgMTAwLjAl ICBbMV0gICAgICAgICAgdm1hcGJ1ZgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgow MC4zNCUgIFsxXSAgICAgICAgeHB0X3BhdGhfc2ltIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBtcnNhc19tYXBfcmVxdWVzdCBAIC9ib290L2tlcm5lbC9tcnNhcy5r bwogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2J1aWxkX2RjZGIKICAgMTAwLjAlICBbMV0g ICAgICAgICAgIG1yc2FzX2FjdGlvbgoKMDAuMzQlICBbMV0gICAgICAgIHNjc2lfcmVhZF93cml0 ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGFzdGFydAogIDEw MC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBkYXN0cmF0ZWd5CgowMC4zNCUgIFsxXSAgICAgICAga2Vybl9yZWFkdiBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3lzX3JlYWQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgX19zeXNfd3JpdGUgQCAvbGli L2xpYmMuc28uNwoKMDAuMzQlICBbMV0gICAgICAgIGdfaW9fZGVsaXZlciBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgMTAwLjAl ICBbMV0gICAgICAgICAgZGFkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfZG9uZV9w cm9jZXNzCgowMC4zNCUgIFsxXSAgICAgICAgY3JpdGljYWxfZW50ZXIgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGNhbGxvdXRfbG9jawogIDEwMC4wJSAgWzFdICAg ICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNf Y21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KCjAwLjM0JSAgWzFdICAgICAgICBkYXN0 YXJ0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2Fs bG9jcQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRhc3RyYXRlZ3kKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGdfZGlza19zdGFydAoKMDAuMzQlICBbMV0gICAgICAgIF9tdHhfdHJ5bG9ja19mbGFn c18gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHZtX3BhZ2VfcGFf dHJ5cmVsb2NrCiAgMTAwLjAlICBbMV0gICAgICAgICAgcG1hcF9leHRyYWN0X2FuZF9ob2xkCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICB2bV9mYXVsdF9xdWlja19ob2xkX3BhZ2VzCgowMC4zNCUg IFsxXSAgICAgICAgY2FsbG91dF9jY19hZGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxXSAgICAgICAgIGNhbGxvdXRfcmVzZXRfc2J0X29uCiAgMTAwLjAlICBbMV0gICAgICAgICAg bXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNCUgIFsxXSAgICAg ICAgc3lzX3JlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGFt ZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFdICAgICAgICBzY2hlZF9yZW0gQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHRkcV9tb3ZlCiAgMTAwLjAlICBbMV0gICAgICAg ICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM0 JSAgWzFdICAgICAgICBzY2hlZF9hZGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIHNldHJ1bm5hYmxlCiAgMTAwLjAlICBbMV0gICAgICAgICAgc2xlZXBxX2Jyb2Fk Y2FzdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgd2FrZXVwCgowMC4zNCUgIFsxXSAgICAgICAg X19sb2NrbWdyX2FyZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAg IGdldHBidWYKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX3ByaW9yaXR5IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAu MCUgIFsxXSAgICAgICAgICBzZXRydW5uYWJsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xl ZXBxX2Jyb2FkY2FzdAoKMDAuMzQlICBbMV0gICAgICAgIGJvdW5jZV9idXNfZG1hbWFwX2xvYWRf YnVmZmVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBidXNfZG1h bWFwX2xvYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19tYXBfcmVxdWVzdCBAIC9ib290 L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfYnVpbGRfZGNk YgoKMDAuMzQlICBbMV0gICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZCBAIC9ib290L2tlcm5lbC9t cnNhcy5rbwogMTAwLjAlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMV0gICAgICAgICAgaXRocmVhZF9sb29w CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM0JSAgWzFdICAgICAgICBm cmVlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcmVsZWFz ZV9jY2IKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjM0JSAgWzFdICAgICAgICBkZXZfcmVsdGhyZWFkIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZmc19yZWFkX2YKICAx MDAuMCUgIFsxXSAgICAgICAgICBkb2ZpbGVyZWFkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBr ZXJuX3JlYWR2CgowMC4zNCUgIFsxXSAgICAgICAgdGhyZWFkX2xvY2tfc2V0IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfc3dpdGNoCiAgMTAwLjAlICBb MV0gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9zbGVlcAoK MDAuMzQlICBbMV0gICAgICAgIHJ1bnFfYWRkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50 X3NjaGVkdWxlX3RocmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5k bGUKCjAwLjM0JSAgWzFdICAgICAgICB0ZHFfbG9ja19wYWlyIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFsxXSAgICAgICAg ICBmb3JrX2V4aXQKCjAwLjM0JSAgWzFdICAgICAgICBzbGVlcHFfd2FpdCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgX3NsZWVwCiAgMTAwLjAlICBbMV0gICAgICAg ICAgYndhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBoeXNpbwoKMDAuMzQlICBbMV0gICAg ICAgIHRocmVhZF9sb2NrX2Jsb2NrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0g ICAgICAgICBzY2hlZF9zd2l0Y2gKICAxMDAuMCUgIFsxXSAgICAgICAgICBtaV9zd2l0Y2gKICAg MTAwLjAlICBbMV0gICAgICAgICAgIHNsZWVwcV93YWl0CgowMC4zNCUgIFsxXSAgICAgICAgeHB0 X2RvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2FzX2Nt ZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJz YXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1 dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNCUgIFsxXSAgICAgICAgdW1h X3pmcmVlX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19k ZXZfZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGdfaW9fZGVsaXZlcgogICAxMDAuMCUgIFsx XSAgICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCgowMC4zNCUgIFsxXSAgICAgICAgWHRpbWVy aW50IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuMzQlICBbMV0gICAgICAgIGRldnN0YXRfc3Rh cnRfdHJhbnNhY3Rpb25fYmlvIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2lvX3JlcXVlc3QKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKCjAwLjM0JSAgWzFdICAgICAg ICBkZXZmc193cml0ZV9mIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICBkb2ZpbGV3cml0ZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGtlcm5fd3JpdGV2CiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBzeXNfd3JpdGUKCjAwLjM0JSAgWzFdICAgICAgICB2dW5tYXBidWYg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHBoeXNpbwogIDEwMC4w JSAgWzFdICAgICAgICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZG9m aWxlcmVhZAoKMDAuMzQlICBbMV0gICAgICAgIHNsZWVwcV9yZXN1bWVfdGhyZWFkIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAgMTAw LjAlICBbMV0gICAgICAgICAgd2FrZXVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiZG9uZQoK MDAuMzQlICBbMV0gICAgICAgIHNsZWVwcV9hZGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0CiAgIDEw MC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBfbXR4X2xvY2tf c3Bpbl9jb29raWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNj aGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0g ICAgICAgIGFtZDY0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNCUgIFsxXSAg ICAgICAgX2NhbGxvdXRfc3RvcF9zYWZlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4w JSAgWzFdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAu MzQlICBbMV0gICAgICAgIHJ1bnFfcmVtb3ZlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICBzY2hlZF9jaG9vc2UKICAxMDAuMCUgIFsxXSAgICAgICAgICBjaG9vc2V0 aHJlYWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNjaGVkX3N3aXRjaAoKMDAuMzQlICBbMV0g ICAgICAgIGJ6ZXJvIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBt cnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAg ICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKCjAwLjM0JSAgWzFdICAgICAgICB1bWFfemFsbG9jX2Fy ZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2X3N0cmF0ZWd5 X2NzdwogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZGV2ZnNfcmVhZF9mCgpAIENQVV9DTEtfVU5IQUxURURfQ09SRSBbMjk0IHNhbXBsZXNdCgox MS45MCUgIFszNV0gICAgICAgY3B1X2lkbGUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFszNV0gICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzM1XSAgICAgICAgIGZvcmtfZXhp dAoKMDkuODYlICBbMjldICAgICAgIGludmxybmdfaGFuZGxlciBAIC9ib290L2tlcm5lbC9rZXJu ZWwKCjA4Ljg0JSAgWzI2XSAgICAgICBYaW52bHJuZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA1 Ljc4JSAgWzE3XSAgICAgICBfX210eF9sb2NrX3NsZWVwIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog NTguODIlICBbMTBdICAgICAgICBfX210eF9sb2NrX2ZsYWdzCiAgMTAwLjAlICBbMTBdICAgICAg ICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbMTBd ICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogMTcuNjUlICBbM10gICAgICAgICB2bWVtX3hh bGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFszXSAgICAgICAgICB2bWVtX2Fs bG9jCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBnX2lvX2NoZWNrCiAwNS44OCUgIFsxXSAgICAg ICAgIHhwdF9ydW5fZGV2cQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVs dAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdGFydAogMDUuODglICBbMV0gICAgICAgICB4 cHRfZG9uZV9wcm9jZXNzCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfdGQKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAogMDUuODglICBbMV0gICAgICAgICB2bWVtX3hm cmVlCiAgMTAwLjAlICBbMV0gICAgICAgICAgYmlvZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZGFkb25lCiAwNS44OCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAg ICAgIGJ3YWl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjA0LjQyJSAgWzEzXSAg ICAgICBzbXBfdGxiX3Nob290ZG93biBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzEz XSAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgMTAwLjAlICBbMTNdICAgICAgICAgYmlv ZG9uZQogICAxMDAuMCUgIFsxM10gICAgICAgICAgZGFkb25lCgowMy40MCUgIFsxMF0gICAgICAg Y3B1X3NlYXJjaF9oaWdoZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogOTAuMDAlICBbOV0gICAg ICAgICBjcHVfc2VhcmNoX2hpZ2hlc3QKICA2Ni42NyUgIFs2XSAgICAgICAgICBzY2hlZF9pZGxl dGQKICAgMTAwLjAlICBbNl0gICAgICAgICAgIGZvcmtfZXhpdAogIDMzLjMzJSAgWzNdICAgICAg ICAgIGNwdV9zZWFyY2hfaGlnaGVzdAogICAxMDAuMCUgIFszXSAgICAgICAgICAgc2NoZWRfaWRs ZXRkCiAxMC4wMCUgIFsxXSAgICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAg ICAgIGZvcmtfZXhpdAoKMDMuMDYlICBbOV0gICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0IEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbOV0gICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdAog IDg4Ljg5JSAgWzhdICAgICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0CiAgIDEwMC4wJSAgWzhdICAg ICAgICAgICBzY2hlZF9waWNrY3B1CiAgMTEuMTElICBbMV0gICAgICAgICAgc2NoZWRfcGlja2Nw dQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2NoZWRfYWRkCgowMi4zOCUgIFs3XSAgICAgICAg Y3B1X3N3aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDcxLjQzJSAgWzVdICAgICAgICAgbWlf c3dpdGNoCiAgNDAuMDAlICBbMl0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzJd ICAgICAgICAgICBmb3JrX2V4aXQKICAyMC4wMCUgIFsxXSAgICAgICAgICBjcml0aWNhbF9leGl0 CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDIwLjAwJSAgWzFd ICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0 CiAgMjAuMDAlICBbMV0gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIF9zbGVlcAogMjguNTclICBbMl0gICAgICAgICBzY2hlZF9zd2l0Y2gKICAxMDAuMCUgIFsy XSAgICAgICAgICBtaV9zd2l0Y2gKICAgNTAuMDAlICBbMV0gICAgICAgICAgIHNsZWVwcV93YWl0 CiAgIDUwLjAwJSAgWzFdICAgICAgICAgICBpdGhyZWFkX2xvb3AKCjAxLjcwJSAgWzVdICAgICAg ICB0aHJlYWRfbG9ja19mbGFnc18gQCAvYm9vdC9rZXJuZWwva2VybmVsCiA4MC4wMCUgIFs0XSAg ICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzRdICAgICAgICAgIGZvcmtfZXhpdAogMjAu MDAlICBbMV0gICAgICAgICBzbGVlcHFfYWRkCiAgMTAwLjAlICBbMV0gICAgICAgICAgX3NsZWVw CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBid2FpdAoKMDEuNzAlICBbNV0gICAgICAgIGRvcmV0 aSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjcwJSAgWzVdICAgICAgICBzY2hlZF9zd2l0Y2gg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs1XSAgICAgICAgIG1pX3N3aXRjaAogIDYw LjAwJSAgWzNdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFszXSAgICAgICAgICAg Zm9ya19leGl0CiAgMjAuMDAlICBbMV0gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBb MV0gICAgICAgICAgIF9zbGVlcAogIDIwLjAwJSAgWzFdICAgICAgICAgIGl0aHJlYWRfbG9vcAog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CgowMS43MCUgIFs1XSAgICAgICAgc2No ZWRfaWRsZXRkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNV0gICAgICAgICBmb3Jr X2V4aXQKCjAxLjcwJSAgWzVdICAgICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzVdICAgICAgICAgdm1lbV9hbGxvYwogIDEwMC4wJSAgWzVdICAgICAgICAg IGdfaW9fY2hlY2sKICAgMTAwLjAlICBbNV0gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDEuNzAl ICBbNV0gICAgICAgIF9fbXR4X3VubG9ja19mbGFncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDQw LjAwJSAgWzJdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBbMl0gICAgICAgICAgeHB0 X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkYXN0YXJ0CiA0MC4wMCUg IFsyXSAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAw LjAlICBbMl0gICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzJdICAgICAg ICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAy MC4wMCUgIFsxXSAgICAgICAgIG1yc2FzX2dldF9tcHRfY21kIEAgL2Jvb3Qva2VybmVsL21yc2Fz LmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfYWN0aW9uCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMS43MCUgIFs1XSAg ICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNV0gICAg ICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAxMDAuMCUgIFs1XSAgICAgICAgICBkYXN0YXJ0CiAg IDEwMC4wJSAgWzVdICAgICAgICAgICB4cHRfcnVuX2FsbG9jcQoKMDEuMzYlICBbNF0gICAgICAg IGNyaXRpY2FsX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAyNS4wMCUgIFsxXSAgICAgICAg IHVtYV96YWxsb2NfYXJnCiAgMTAwLjAlICBbMV0gICAgICAgICAgZGV2X3N0cmF0ZWd5X2Nzdwog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCiAyNS4wMCUgIFsxXSAgICAgICAgIGludHJf ZXZlbnRfaGFuZGxlCiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJz CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBsYXBpY19oYW5kbGVfaW50cgogMjUuMDAlICBbMV0g ICAgICAgICBtYWxsb2MKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAx MDAuMCUgIFsxXSAgICAgICAgICAgZGFzdHJhdGVneQogMjUuMDAlICBbMV0gICAgICAgICBzcGlu bG9ja19leGl0CiAgMTAwLjAlICBbMV0gICAgICAgICAgdHVybnN0aWxlX3dhaXQKICAgMTAwLjAl ICBbMV0gICAgICAgICAgIF9fbXR4X2xvY2tfc2xlZXAKCjAxLjM2JSAgWzRdICAgICAgICBfX210 eF9sb2NrX2ZsYWdzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMl0gICAgICAgICBt cnNhc19maXJlX2NtZCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzJdICAgICAg ICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMl0gICAg ICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogNTAuMDAlICBbMl0gICAgICAgICBtcnNhc191bm1h cF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMl0gICAgICAgICAg bXJzYXNfY21kX2RvbmUKICAgMTAwLjAlICBbMl0gICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2Nt ZAoKMDEuMzYlICBbNF0gICAgICAgIGRhc3RhcnQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFs0XSAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgMTAwLjAlICBbNF0gICAgICAgICAgZGFz dHJhdGVneQogICAxMDAuMCUgIFs0XSAgICAgICAgICAgZ19kaXNrX3N0YXJ0CgowMS4zNiUgIFs0 XSAgICAgICAgc3BpbmxvY2tfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDI1LjAwJSAgWzFd ICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMV0gICAgICAgICAgZm9ya19leGl0CiAy NS4wMCUgIFsxXSAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgMTAwLjAlICBb MV0gICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGlu dHJfZXhlY3V0ZV9oYW5kbGVycwogMjUuMDAlICBbMV0gICAgICAgICBpdGhyZWFkX2xvb3AKICAx MDAuMCUgIFsxXSAgICAgICAgICBmb3JrX2V4aXQKIDI1LjAwJSAgWzFdICAgICAgICAgY2FsbG91 dF9yZXNldF9zYnRfb24KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9v dC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fZGV2cSBA IC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjAyJSAgWzNdICAgICAgICBnX2Rldl9zdHJhdGVneSBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZGV2X3N0cmF0ZWd5X2Nz dwogIDEwMC4wJSAgWzNdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFszXSAgICAgICAgICAg ZGV2ZnNfcmVhZF9mCgowMS4wMiUgIFszXSAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0IEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICBkYXN0YXJ0CiAgMTAwLjAlICBb M10gICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAgMTAwLjAlICBbM10gICAgICAgICAgIGRhc3Ry YXRlZ3kKCjAxLjAyJSAgWzNdICAgICAgICB4cHRfZG9uZV9wcm9jZXNzIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB4cHRfZG9uZV90ZAogIDEwMC4wJSAgWzNdICAg ICAgICAgIGZvcmtfZXhpdAoKMDEuMDIlICBbM10gICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZCBA IC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbM10gICAgICAgICBpbnRyX2V2ZW50X2V4 ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbM10gICAgICAg ICAgaXRocmVhZF9sb29wCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBmb3JrX2V4aXQKCjAxLjAy JSAgWzNdICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFszXSAgICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFszXSAgICAgICAgICBkYWRv bmUKICAgMTAwLjAlICBbM10gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAxLjAyJSAgWzNd ICAgICAgICB4cHRfZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAg ICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFszXSAg ICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbM10gICAgICAgICAgIGludHJf ZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjAyJSAgWzNd ICAgICAgICBwaHlzaW8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAg IGRldmZzX3JlYWRfZgogIDEwMC4wJSAgWzNdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAl ICBbM10gICAgICAgICAgIGtlcm5fcmVhZHYKCjAxLjAyJSAgWzNdICAgICAgICBfbXR4X2xvY2tf c3Bpbl9jb29raWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA2Ni42NyUgIFsyXSAgICAgICAgIHNj aGVkX2lkbGV0ZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGZvcmtfZXhpdAogMzMuMzMlICBbMV0g ICAgICAgICBzbXBfdGxiX3Nob290ZG93bgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBtYXBfaW52 YWxpZGF0ZV9yYW5nZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYmlvZG9uZQoKMDEuMDIlICBb M10gICAgICAgIGFtZDY0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC42OCUgIFsy XSAgICAgICAgZ19pb19kZWxpdmVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAxMDAuMCUgIFsyXSAgICAgICAgICBkYWRvbmUK ICAgMTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjY4JSAgWzJdICAg ICAgICBtaV9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAg IHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAogNTAuMDAlICBb MV0gICAgICAgICBjcml0aWNhbF9leGl0CiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVu dF9oYW5kbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwoK MDAuNjglICBbMl0gICAgICAgIGRldnN0YXRfZW5kX3RyYW5zYWN0aW9uIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBkZXZzdGF0X2VuZF90cmFuc2FjdGlvbl9iaW9f YnQKICAxMDAuMCUgIFsyXSAgICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIGRhZG9uZQoKMDAuNjglICBbMl0gICAgICAgIHNldHJ1bm5hYmxlIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAg MTAwLjAlICBbMl0gICAgICAgICAgd2FrZXVwCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBiZG9u ZQoKMDAuNjglICBbMl0gICAgICAgIHZtX3BhZ2VfdW5ob2xkX3BhZ2VzIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB2dW5tYXBidWYKICAxMDAuMCUgIFsyXSAgICAg ICAgICBwaHlzaW8KICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuNjgl ICBbMl0gICAgICAgIGl0aHJlYWRfbG9vcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgZm9ya19leGl0CgowMC42OCUgIFsyXSAgICAgICAgX19zeXNfcmVhZCBAIC9s aWIvbGliYy5zby43CgowMC42OCUgIFsyXSAgICAgICAgZ19kaXNrX3N0YXJ0IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFsy XSAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBwaHlz aW8KCjAwLjY4JSAgWzJdICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwvbXJzYXMu a28KIDEwMC4wJSAgWzJdICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogIDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFsy XSAgICAgICAgICAgZGFzdGFydAoKMDAuNjglICBbMl0gICAgICAgIHNwaW5sb2NrX2VudGVyIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICB0aHJlYWRfbG9ja19mbGFn c18KICAxMDAuMCUgIFsxXSAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICB3YWtldXAKIDUwLjAwJSAgWzFdICAgICAgICAgY2FsbG91dF9sb2NrCiAgMTAw LjAlICBbMV0gICAgICAgICAgY2FsbG91dF9yZXNldF9zYnRfb24KICAgMTAwLjAlICBbMV0gICAg ICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwoKMDAuNjglICBbMl0g ICAgICAgIGRldnZuX3JlZnRocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJd ICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgMTAwLjAlICBbMl0gICAgICAgICAgZG9maWxlcmVhZAog ICAxMDAuMCUgIFsyXSAgICAgICAgICAga2Vybl9yZWFkdgoKMDAuNjglICBbMl0gICAgICAgIHVt YV96YWxsb2NfYXJnIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICBk ZXZfc3RyYXRlZ3lfY3N3CiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBkZXZmc19yZWFkX2YKIDUwLjAwJSAgWzFdICAgICAgICAgbWFsbG9jCiAg MTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRhc3RyYXRlZ3kKCjAwLjY4JSAgWzJdICAgICAgICBkYXN0cmF0ZWd5IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsy XSAgICAgICAgICBnX2lvX3JlcXVlc3QKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRldl9zdHJh dGVneV9jc3cKCjAwLjY4JSAgWzJdICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzJdICAg ICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9kb25l X3RkCgowMC42OCUgIFsyXSAgICAgICAgYnplcm8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEw MC4wJSAgWzJdICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAg MTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAoKMDAuNjglICBbMl0gICAg ICAgIHhwdF9ydW5fYWxsb2NxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAg ICAgICBkYXN0cmF0ZWd5CiAgMTAwLjAlICBbMl0gICAgICAgICAgZ19kaXNrX3N0YXJ0CiAgIDEw MC4wJSAgWzJdICAgICAgICAgICBnX2lvX3JlcXVlc3QKCjAwLjY4JSAgWzJdICAgICAgICBkZXZz dGF0X3N0YXJ0X3RyYW5zYWN0aW9uX2JpbyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgZ19kaXNrX3N0YXJ0CiAgMTAwLjAlICBbMl0gICAgICAgICAgZ19pb19yZXF1 ZXN0CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CgowMC4zNCUgIFsx XSAgICAgICAgWGZhc3Rfc3lzY2FsbCBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFd ICAgICAgICBzbGVlcHFfd2FpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgX3NsZWVwCiAgMTAwLjAlICBbMV0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMV0g ICAgICAgICAgIHBoeXNpbwoKMDAuMzQlICBbMV0gICAgICAgIGxvY2tfbXR4IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsxXSAgICAg ICAgICBid2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNCUgIFsxXSAg ICAgICAgc3lzY2FsbF90aHJlYWRfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIGtlcm5fd3JpdGV2 IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzeXNfd3JpdGUKICAx MDAuMCUgIFsxXSAgICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgdm1f ZmF1bHRfcXVpY2tfaG9sZF9wYWdlcyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgdm1hcGJ1ZgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUg IFsxXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgZ19pb19yZXF1 ZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZfc3RyYXRl Z3lfY3N3CiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBkZXZmc19yZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICB1bWFfemZyZWVfYXJnIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAu MCUgIFsxXSAgICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRhZG9uZQoKMDAuMzQlICBbMV0gICAgICAgIHNsZWVwcV9hZGQgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAg IGJ3YWl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAg ICBfZmdldCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAga2Vybl93 cml0ZXYKICAxMDAuMCUgIFsxXSAgICAgICAgICBzeXNfd3JpdGUKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFdICAgICAgICBjYW1xX3JlbW92ZSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAw LjAlICBbMV0gICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBkYXN0YXJ0CgowMC4zNCUgIFsxXSAgICAgICAgYmlvcV9pbnNlcnRfdGFpbCBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGFzdHJhdGVneQogIDEwMC4wJSAg WzFdICAgICAgICAgIGdfZGlza19zdGFydAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZ19pb19y ZXF1ZXN0CgowMC4zNCUgIFsxXSAgICAgICAgdm1lbV94ZnJlZSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYmlvZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRh ZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuMzQlICBb MV0gICAgICAgIHNjaGVkX3VubGVuZF9wcmlvIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICB0dXJuc3RpbGVfdW5wZW5kCiAgMTAwLjAlICBbMV0gICAgICAgICAgX19t dHhfdW5sb2NrX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBidWZkb25lCgowMC4zNCUg IFsxXSAgICAgICAgZmdldF91bmxvY2tlZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgX2ZnZXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBrZXJuX3dyaXRldgogICAx MDAuMCUgIFsxXSAgICAgICAgICAgc3lzX3dyaXRlCgowMC4zNCUgIFsxXSAgICAgICAgbXJzYXNf aXNyIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsxXSAgICAgICAgIGludHJfZXZl bnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsxXSAg ICAgICAgICBpdGhyZWFkX2xvb3AKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoK MDAuMzQlICBbMV0gICAgICAgIHBtYXBfcWVudGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBnX2lvX2NoZWNrCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19pb19y ZXF1ZXN0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CgowMC4zNCUg IFsxXSAgICAgICAgbXJzYXNfZGF0YV9sb2FkX2NiIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAx MDAuMCUgIFsxXSAgICAgICAgIGJ1c19kbWFtYXBfbG9hZCBAIC9ib290L2tlcm5lbC9rZXJuZWwK ICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19tYXBfcmVxdWVzdCBAIC9ib290L2tlcm5lbC9t cnNhcy5rbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfYnVpbGRfZGNkYgoKMDAuMzQl ICBbMV0gICAgICAgIF9fY2FwX3JpZ2h0c19pbml0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBrZXJuX3JlYWR2CiAgMTAwLjAlICBbMV0gICAgICAgICAgc3lzX3Jl YWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFdICAg ICAgICB4cHRfcmVsZWFzZV9jY2IgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAg ICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAg MTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3RkCgowMC4zNCUgIFsxXSAgICAgICAgc2No ZWRfY2hvb3NlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBjaG9v c2V0aHJlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBzY2hlZF9zd2l0Y2gKICAgMTAwLjAlICBb MV0gICAgICAgICAgIG1pX3N3aXRjaAoKMDAuMzQlICBbMV0gICAgICAgIGNyaXRpY2FsX2VudGVy IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9zd2l0Y2gK ICAxMDAuMCUgIFsxXSAgICAgICAgICBtaV9zd2l0Y2gKICAgMTAwLjAlICBbMV0gICAgICAgICAg IGl0aHJlYWRfbG9vcAoKMDAuMzQlICBbMV0gICAgICAgIGxhcGljX2lwaV92ZWN0b3JlZCBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc21wX3RsYl9zaG9vdGRvd24K ICAxMDAuMCUgIFsxXSAgICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGJpb2RvbmUKCjAwLjM0JSAgWzFdICAgICAgICBjYWxsb3V0X2NjX2FkZCBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgY2FsbG91dF9yZXNldF9z YnRfb24KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwv bXJzYXMua28KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tl cm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBzY2hlZF9hZGQgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHR1cm5zdGlsZV91bnBlbmQKICAxMDAuMCUgIFsx XSAgICAgICAgICBfX210eF91bmxvY2tfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9f bXR4X3VubG9ja19mbGFncwoKMDAuMzQlICBbMV0gICAgICAgIF9fbG9ja21ncl9hcmdzIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnZXRwYnVmCiAgMTAwLjAlICBb MV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19yZWFkX2YK CjAwLjM0JSAgWzFdICAgICAgICBYYXBpY19pc3IxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAu MzQlICBbMV0gICAgICAgIF9zbGVlcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgYndhaXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBb MV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQlICBbMV0gICAgICAgIGdfaW9fY2hlY2sg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGdfaW9fcmVxdWVzdAog IDEwMC4wJSAgWzFdICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIHBoeXNpbwoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX3BpY2tjcHUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFd ICAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBpbnRyX2V2ZW50X2hhbmRsZQoKMDAuMzQlICBbMV0gICAgICAgIHN5c193cml0ZSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAu MzQlICBbMV0gICAgICAgIHNsZWVwcV9icm9hZGNhc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIHdha2V1cAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJkb25lCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBidWZkb25lCgowMC4zNCUgIFsxXSAgICAgICAgYnRfZnJl ZXRyaW0gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJpb2RvbmUK ICAxMDAuMCUgIFsxXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhw dF9kb25lX3Byb2Nlc3MKCjAwLjM0JSAgWzFdICAgICAgICBnX2Rldl9kb25lIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAuMCUgIFsx XSAgICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRh ZG9uZQoKMDAuMzQlICBbMV0gICAgICAgIHJ1bnFfYWRkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBzZXRy dW5uYWJsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xlZXBxX2Jyb2FkY2FzdAoKMDAuMzQl ICBbMV0gICAgICAgIHBtYXBfZXh0cmFjdF9hbmRfaG9sZCBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzFdICAgICAgICAgdm1fZmF1bHRfcXVpY2tfaG9sZF9wYWdlcwogIDEwMC4wJSAg WzFdICAgICAgICAgIHZtYXBidWYKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBoeXNpbwoKQCBD UFVfQ0xLX1VOSEFMVEVEX0NPUkUgWzI5NCBzYW1wbGVzXQoKMTAuNTQlICBbMzFdICAgICAgIGlu dmxybmdfaGFuZGxlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA5LjUyJSAgWzI4XSAgICAgICBY aW52bHJuZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA4LjE2JSAgWzI0XSAgICAgICBjcHVfaWRs ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzI0XSAgICAgICAgc2NoZWRfaWRsZXRk CiAgMTAwLjAlICBbMjRdICAgICAgICAgZm9ya19leGl0CgowNi4xMiUgIFsxOF0gICAgICAgc21w X3RsYl9zaG9vdGRvd24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxOF0gICAgICAg IHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzE4XSAgICAgICAgIGJpb2RvbmUKICAg MTAwLjAlICBbMThdICAgICAgICAgIGRhZG9uZQoKMDUuMTAlICBbMTVdICAgICAgIGNwdV9zZWFy Y2hfaGlnaGVzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDgwLjAwJSAgWzEyXSAgICAgICAgY3B1 X3NlYXJjaF9oaWdoZXN0CiAgNjYuNjclICBbOF0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEw MC4wJSAgWzhdICAgICAgICAgICBmb3JrX2V4aXQKICAzMy4zMyUgIFs0XSAgICAgICAgICBjcHVf c2VhcmNoX2hpZ2hlc3QKICAgMTAwLjAlICBbNF0gICAgICAgICAgIHNjaGVkX2lkbGV0ZAogMjAu MDAlICBbM10gICAgICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFszXSAgICAgICAgICBmb3Jr X2V4aXQKCjAzLjQwJSAgWzEwXSAgICAgICBjcHVfc2VhcmNoX2xvd2VzdCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzEwXSAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QKICA3MC4wMCUg IFs3XSAgICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdAogICAxMDAuMCUgIFs3XSAgICAgICAgICAg c2NoZWRfcGlja2NwdQogIDMwLjAwJSAgWzNdICAgICAgICAgIHNjaGVkX3BpY2tjcHUKICAgMTAw LjAlICBbM10gICAgICAgICAgIHNjaGVkX2FkZAoKMDIuNzIlICBbOF0gICAgICAgIHZtZW1feGZy ZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs4XSAgICAgICAgIGJpb2RvbmUKICAx MDAuMCUgIFs4XSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbOF0gICAgICAgICAgIHhwdF9k b25lX3Byb2Nlc3MKCjAyLjM4JSAgWzddICAgICAgICBjcHVfc3dpdGNoIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbN10gICAgICAgICBtaV9zd2l0Y2gKICA1Ny4xNCUgIFs0XSAgICAg ICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBbNF0gICAgICAgICAgIGZvcmtfZXhpdAogIDQy Ljg2JSAgWzNdICAgICAgICAgIHNsZWVwcV93YWl0CiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBf c2xlZXAKCjAyLjM4JSAgWzddICAgICAgICBzY2hlZF9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFs3XSAgICAgICAgIG1pX3N3aXRjaAogIDQyLjg2JSAgWzNdICAgICAgICAg IGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFszXSAgICAgICAgICAgZm9ya19leGl0CiAgMjguNTcl ICBbMl0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3Jr X2V4aXQKICAxNC4yOSUgIFsxXSAgICAgICAgICBjcml0aWNhbF9leGl0CiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDE0LjI5JSAgWzFdICAgICAgICAgIHNsZWVw cV93YWl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBfc2xlZXAKCjAyLjA0JSAgWzZdICAgICAg ICBfX210eF91bmxvY2tfZmxhZ3MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAzMy4zMyUgIFsyXSAg ICAgICAgIHhwdF9ydW5fZGV2cQogIDEwMC4wJSAgWzJdICAgICAgICAgIHhwdF9hY3Rpb25fZGVm YXVsdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZGFzdGFydAogMTYuNjclICBbMV0gICAgICAg ICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAg ICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKIDE2LjY3JSAgWzFdICAgICAgICAgbXJzYXNfY21k X2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNh c19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0 ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDE2LjY3JSAgWzFdICAgICAgICAgbXJz YXNfZ2V0X21wdF9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAg ICAgICBtcnNhc19hY3Rpb24KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fZGV2cSBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDE2LjY3JSAgWzFdICAgICAgICAgbXJzYXNfbWFwX3JlcXVl c3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19i dWlsZF9kY2RiCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtcnNhc19hY3Rpb24KCjAxLjcwJSAg WzVdICAgICAgICBfX210eF9sb2NrX3NsZWVwIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjAuMDAl ICBbM10gICAgICAgICBfX210eF9sb2NrX2ZsYWdzCiAgMTAwLjAlICBbM10gICAgICAgICAgbXJz YXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbM10gICAgICAg ICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogMjAuMDAlICBbMV0gICAgICAgICB4cHRfcnVuX2RldnEg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2FjdGlvbl9k ZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0YXJ0CiAyMC4wMCUgIFsxXSAgICAg ICAgIHZtZW1feGFsbG9jCiAgMTAwLjAlICBbMV0gICAgICAgICAgdm1lbV9hbGxvYwogICAxMDAu MCUgIFsxXSAgICAgICAgICAgZ19pb19jaGVjawoKMDEuNzAlICBbNV0gICAgICAgIGl0aHJlYWRf bG9vcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzVdICAgICAgICAgZm9ya19leGl0 CgowMS43MCUgIFs1XSAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbNV0gICAgICAgICBiaW9kb25lCiAgMTAwLjAlICBbNV0gICAgICAg ICAgZGFkb25lCiAgIDEwMC4wJSAgWzVdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMS43 MCUgIFs1XSAgICAgICAgZG9yZXRpIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDEuNzAlICBbNV0g ICAgICAgIHNwaW5sb2NrX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA0MC4wMCUgIFsyXSAg ICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGZvcmtfZXhpdAogMjAu MDAlICBbMV0gICAgICAgICBzbXBfdGxiX3Nob290ZG93bgogIDEwMC4wJSAgWzFdICAgICAgICAg IHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYmlvZG9uZQog MjAuMDAlICBbMV0gICAgICAgICBjYWxsb3V0X3Jlc2V0X3NidF9vbgogIDEwMC4wJSAgWzFdICAg ICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFsx XSAgICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMjAuMDAlICBb MV0gICAgICAgICB3YWtldXAKICAxMDAuMCUgIFsxXSAgICAgICAgICBiZG9uZQogICAxMDAuMCUg IFsxXSAgICAgICAgICAgYnVmZG9uZQoKMDEuNzAlICBbNV0gICAgICAgIHRocmVhZF9sb2NrX2Zs YWdzXyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDIwLjAwJSAgWzFdICAgICAgICAgc2xlZXBxX2Fk ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIF9zbGVlcAogICAxMDAuMCUgIFsxXSAgICAgICAgICAg YndhaXQKIDIwLjAwJSAgWzFdICAgICAgICAgc2xlZXBxX3dhaXQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ3YWl0CiAyMC4wMCUgIFsxXSAg ICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgMTAwLjAlICBbMV0gICAgICAgICAg aW50cl9ldmVudF9oYW5kbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXhlY3V0ZV9o YW5kbGVycwogMjAuMDAlICBbMV0gICAgICAgICBwcm9wYWdhdGVfcHJpb3JpdHkKICAxMDAuMCUg IFsxXSAgICAgICAgICB0dXJuc3RpbGVfd2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgX19t dHhfbG9ja19zbGVlcAogMjAuMDAlICBbMV0gICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAgMTAw LjAlICBbMV0gICAgICAgICAgd2FrZXVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiZG9uZQoK MDEuMzYlICBbNF0gICAgICAgIF9fc3lzX3JlYWQgQCAvbGliL2xpYmMuc28uNwoKMDEuMzYlICBb NF0gICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzRd ICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgMTAwLjAlICBbNF0gICAgICAgICAgZGFzdGFy dAogICAxMDAuMCUgIFs0XSAgICAgICAgICAgeHB0X3J1bl9hbGxvY3EKCjAxLjM2JSAgWzRdICAg ICAgICBzY2hlZF9pZGxldGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAg ICAgIGZvcmtfZXhpdAoKMDEuMzYlICBbNF0gICAgICAgIFhhcGljX2lzcjEgQCAvYm9vdC9rZXJu ZWwva2VybmVsCgowMS4zNiUgIFs0XSAgICAgICAgY3JpdGljYWxfZW50ZXIgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAyNS4wMCUgIFsxXSAgICAgICAgIHVtYV96YWxsb2NfYXJnCiAgMTAwLjAlICBb MV0gICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5 c2lvCiAyNS4wMCUgIFsxXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbMV0g ICAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBi aW9kb25lCiAyNS4wMCUgIFsxXSAgICAgICAgIHNsZWVwcV9sb2NrCiAgMTAwLjAlICBbMV0gICAg ICAgICAgY3ZfYnJvYWRjYXN0cHJpCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB2bWVtX3hmcmVl CiAyNS4wMCUgIFsxXSAgICAgICAgIHRocmVhZF9sb2NrX2ZsYWdzXwogIDEwMC4wJSAgWzFdICAg ICAgICAgIHNsZWVwcV9hZGQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9zbGVlcAoKMDEuMzYl ICBbNF0gICAgICAgIHVtYV96ZnJlZV9hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA3NS4wMCUg IFszXSAgICAgICAgIGdfZGV2X2RvbmUKICAxMDAuMCUgIFszXSAgICAgICAgICBnX2lvX2RlbGl2 ZXIKICAgMTAwLjAlICBbM10gICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQogMjUuMDAlICBb MV0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2Rpc2tfZG9u ZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDEuMDIlICBbM10gICAg ICAgIGdfZGV2X2RvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAg IGdfaW9fZGVsaXZlcgogIDEwMC4wJSAgWzNdICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQog ICAxMDAuMCUgIFszXSAgICAgICAgICAgZGFkb25lCgowMS4wMiUgIFszXSAgICAgICAgZGFkb25l IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB4cHRfZG9uZV9wcm9j ZXNzCiAgMTAwLjAlICBbM10gICAgICAgICAgeHB0X2RvbmVfdGQKICAgMTAwLjAlICBbM10gICAg ICAgICAgIGZvcmtfZXhpdAoKMDEuMDIlICBbM10gICAgICAgIF9tdHhfbG9ja19zcGluX2Nvb2tp ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2LjY3JSAgWzJdICAgICAgICAgc2NoZWRfaWRsZXRk CiAgMTAwLjAlICBbMl0gICAgICAgICAgZm9ya19leGl0CiAzMy4zMyUgIFsxXSAgICAgICAgIHNj aGVkX2FkZAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CgowMS4wMiUgIFszXSAgICAgICAgWGZhc3Rfc3lz Y2FsbCBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAxLjAyJSAgWzNdICAgICAgICBfX210eF9sb2Nr X2ZsYWdzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICBtcnNhc19j bWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzNdICAgICAgICAgIG1y c2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFszXSAgICAgICAgICAgaW50cl9ldmVudF9leGVj dXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuNjglICBbMl0gICAgICAgIGRv ZmlsZXdyaXRlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBrZXJu X3dyaXRldgogIDEwMC4wJSAgWzJdICAgICAgICAgIHN5c193cml0ZQogICAxMDAuMCUgIFsyXSAg ICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuNjglICBbMl0gICAgICAgIGJ6ZXJvIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgMTAw LjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19y ZWFkX2YKIDUwLjAwJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBbMV0gICAg ICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0YXJ0 CgowMC42OCUgIFsyXSAgICAgICAgYW1kNjRfc3lzY2FsbCBAIC9ib290L2tlcm5lbC9rZXJuZWwK CjAwLjY4JSAgWzJdICAgICAgICBnX2lzX2dlb21fdGhyZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogNTAuMDAlICBbMV0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAuMCUgIFsxXSAgICAgICAg ICBnX2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQogNTAu MDAlICBbMV0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZf c3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjY4JSAgWzJd ICAgICAgICBnX2lvX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAg ICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsyXSAgICAgICAgICBwaHlzaW8KICAg MTAwLjAlICBbMl0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuNjglICBbMl0gICAgICAgIF9f bG9ja21ncl9hcmdzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICBy ZWxwYnVmCiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBkZXZmc19yZWFkX2YKIDUwLjAwJSAgWzFdICAgICAgICAgZ2V0cGJ1ZgogIDEwMC4wJSAg WzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVhZF9m CgowMC42OCUgIFsyXSAgICAgICAgZnJlZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgeHB0X3JlbGVhc2VfY2NiCiAgMTAwLjAlICBbMl0gICAgICAgICAgZGFkb25l CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMC42OCUgIFsyXSAg ICAgICAgcGh5c2lvIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBk ZXZmc19yZWFkX2YKICAxMDAuMCUgIFsyXSAgICAgICAgICBkb2ZpbGVyZWFkCiAgIDEwMC4wJSAg WzJdICAgICAgICAgICBrZXJuX3JlYWR2CgowMC42OCUgIFsyXSAgICAgICAgX2ZnZXQgQCAvYm9v dC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIGtlcm5fcmVhZHYKICAxMDAuMCUg IFsxXSAgICAgICAgICBzeXNfcmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYW1kNjRfc3lz Y2FsbAogNTAuMDAlICBbMV0gICAgICAgICBrZXJuX3dyaXRldgogIDEwMC4wJSAgWzFdICAgICAg ICAgIHN5c193cml0ZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAu NjglICBbMl0gICAgICAgIGRldmZzX3dyaXRlX2YgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIGRvZmlsZXdyaXRlCiAgMTAwLjAlICBbMl0gICAgICAgICAga2Vybl93 cml0ZXYKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHN5c193cml0ZQoKMDAuNjglICBbMl0gICAg ICAgIGRldmZzX3JlYWRfZiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAg ICAgZG9maWxlcmVhZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGtlcm5fcmVhZHYKICAgMTAwLjAl ICBbMl0gICAgICAgICAgIHN5c19yZWFkCgowMC42OCUgIFsyXSAgICAgICAgYXRvbWljX2ZldGNo YWRkX2ludCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNf Y29tcGxldGVfY21kIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAg ICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAx MDAuMCUgIFsxXSAgICAgICAgICAgaXRocmVhZF9sb29wCiA1MC4wMCUgIFsxXSAgICAgICAgIG1y c2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAg IHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIHhwdF9hY3Rpb25fZGVmYXVsdAoKMDAuNjglICBbMl0gICAgICAgIHZtZW1feGFsbG9jIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB2bWVtX2FsbG9jCiAgMTAw LjAlICBbMl0gICAgICAgICAgZ19pb19jaGVjawogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZ19p b19yZXF1ZXN0CgowMC42OCUgIFsyXSAgICAgICAgeHB0X2RvbmVfcHJvY2VzcyBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgeHB0X2RvbmVfdGQKICAxMDAuMCUgIFsy XSAgICAgICAgICBmb3JrX2V4aXQKCjAwLjY4JSAgWzJdICAgICAgICBkYXN0YXJ0IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEwMC4w JSAgWzJdICAgICAgICAgIGRhc3RyYXRlZ3kKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGdfZGlz a19zdGFydAoKMDAuNjglICBbMl0gICAgICAgIGdfaW9fY2hlY2sgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGdfaW9fcmVxdWVzdAogIDEwMC4wJSAgWzJdICAgICAg ICAgIGRldl9zdHJhdGVneV9jc3cKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHBoeXNpbwoKMDAu NjglICBbMl0gICAgICAgIHNjaGVkX3BpY2tjcHUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGludHJfZXZl bnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBpbnRyX2V2ZW50X2hh bmRsZQoKMDAuMzQlICBbMV0gICAgICAgIG1yc2FzX3VubWFwX3JlcXVlc3QgQCAvYm9vdC9rZXJu ZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUKICAxMDAuMCUg IFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMV0gICAgICAgICAg IGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0 JSAgWzFdICAgICAgICB1c2VycmV0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0g ICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgeHB0X2FjdGlvbl9kZWZh dWx0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkYXN0YXJ0CiAg MTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRhc3RyYXRlZ3kKCjAwLjM0JSAgWzFdICAgICAgICBrZXJuX3dyaXRldiBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3lzX3dyaXRlCiAgMTAwLjAlICBbMV0g ICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIGRldnN0YXRfZW5kX3Ry YW5zYWN0aW9uIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZz dGF0X2VuZF90cmFuc2FjdGlvbl9iaW9fYnQKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2Rpc2tf ZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDAuMzQlICBbMV0g ICAgICAgIHNsZWVwcV9icm9hZGNhc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIHdha2V1cAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJkb25lCiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBidWZkb25lCgowMC4zNCUgIFsxXSAgICAgICAgc2xlZXBxX2xvY2sgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHdha2V1cAogIDEwMC4wJSAg WzFdICAgICAgICAgIGJkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBidWZkb25lCgowMC4z NCUgIFsxXSAgICAgICAgc3lzX3dyaXRlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgZGFzdHJhdGVneSBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19kaXNrX3N0YXJ0CiAg MTAwLjAlICBbMV0gICAgICAgICAgZ19pb19yZXF1ZXN0CiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBkZXZfc3RyYXRlZ3lfY3N3CgowMC4zNCUgIFsxXSAgICAgICAgcmR0c2MgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1pX3N3aXRjaAogIDEwMC4wJSAgWzFdICAg ICAgICAgIHNsZWVwcV93YWl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBfc2xlZXAKCjAwLjM0 JSAgWzFdICAgICAgICBYeGVuX2ludHJfdXBjYWxsIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAu MzQlICBbMV0gICAgICAgIF9fY2FwX3JpZ2h0c19pbml0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMV0gICAgICAgICBrZXJuX3JlYWR2CiAgMTAwLjAlICBbMV0gICAgICAgICAgc3lz X3JlYWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFd ICAgICAgICBiaW9kb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICBkYWRvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgIDEwMC4w JSAgWzFdICAgICAgICAgICB4cHRfZG9uZV90ZAoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX2Fk ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgaW50cl9ldmVudF9z Y2hlZHVsZV90aHJlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJzCgowMC4zNCUgIFsx XSAgICAgICAgc2NoZWRfcnVubmFibGUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIGNwdV9pZGxlCiAgMTAwLjAlICBbMV0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM0JSAgWzFdICAgICAgICBfc2xl ZXAgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJ3YWl0CiAgMTAw LjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19y ZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICBpbnRyX2V2ZW50X3NjaGVkdWxlX3RocmVhZCBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUK ICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V4ZWN1dGVfaGFuZGxlcnMKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGxhcGljX2hhbmRsZV9pbnRyCgowMC4zNCUgIFsxXSAgICAgICAgY2FsbG91 dF9sb2NrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfY2FsbG91 dF9zdG9wX3NhZmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290 L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbXJzYXNfY29tcGxldGVf Y21kCgowMC4zNCUgIFsxXSAgICAgICAgc2NzaV9hY3Rpb24gQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RhcnQKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRf cnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdHJhdGVneQoKMDAuMzQlICBb MV0gICAgICAgIGNyaXRpY2FsX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIHNwaW5sb2NrX2V4aXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBzY2hlZF9pZGxl dGQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAg IGRvZmlsZXJlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGtl cm5fcmVhZHYKICAxMDAuMCUgIFsxXSAgICAgICAgICBzeXNfcmVhZAogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIHRkcV9sb2NrX3BhaXIg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2lkbGV0ZAog IDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAgIHNsZWVw cV93YWl0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfc2xlZXAK ICAxMDAuMCUgIFsxXSAgICAgICAgICBid2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5 c2lvCgowMC4zNCUgIFsxXSAgICAgICAgc3lzY2FsbF90aHJlYWRfZXhpdCBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzQlICBbMV0g ICAgICAgIHZtX2ZhdWx0X3F1aWNrX2hvbGRfcGFnZXMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIHZtYXBidWYKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8K ICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQlICBbMV0gICAgICAg IHhwdF9kb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBtcnNh c19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAg IG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9l eGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuMzQlICBbMV0gICAgICAg IHJ1bnFfY2hvb3NlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBz Y2hlZF9jaG9vc2UKICAxMDAuMCUgIFsxXSAgICAgICAgICBjaG9vc2V0aHJlYWQKICAgMTAwLjAl ICBbMV0gICAgICAgICAgIHNjaGVkX3N3aXRjaAoKMDAuMzQlICBbMV0gICAgICAgIGJjb3B5IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2RldnEKICAx MDAuMCUgIFsxXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGRhc3RhcnQKCjAwLjM0JSAgWzFdICAgICAgICByZWxwYnVmIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFsxXSAgICAgICAg ICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRvZmlsZXJlYWQKCjAwLjM0 JSAgWzFdICAgICAgICBnX3RyYWNlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0g ICAgICAgICBnX2Rldl9zdHJhdGVneQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRldl9zdHJhdGVn eV9jc3cKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHBoeXNpbwoKMDAuMzQlICBbMV0gICAgICAg IGdfbmV3X2JpbyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2 X3N0cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsx XSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgbXJzYXNfYWN0aW9u IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAxMDAuMCUgIFsxXSAgICAgICAgIHhwdF9ydW5fZGV2 cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfYWN0aW9u X2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RhcnQKCjAwLjM0JSAgWzFdICAg ICAgICBkZXZfc3RyYXRlZ3lfY3N3IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0g ICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGRvZmlsZXJlYWQKCjAwLjM0JSAgWzFdICAgICAgICB1bWFfemFs bG9jX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2X3N0 cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgY2FtcV9yZW1vdmUgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHhwdF9ydW5fZGV2cQogIDEw MC4wJSAgWzFdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZGFzdGFydAoKMDAuMzQlICBbMV0gICAgICAgIGNwdV9zZXRfc3lzY2FsbF9yZXR2YWwg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGFtZDY0X3N5c2NhbGwK CjAwLjM0JSAgWzFdICAgICAgICBzY2hlZF9jaG9vc2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIGNob29zZXRocmVhZAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNj aGVkX3N3aXRjaAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbWlfc3dpdGNoCgpAIENQVV9DTEtf VU5IQUxURURfQ09SRSBbMjkxIHNhbXBsZXNdCgoxMS42OCUgIFszNF0gICAgICAgWGludmxybmcg QCAvYm9vdC9rZXJuZWwva2VybmVsCgowOS45NyUgIFsyOV0gICAgICAgaW52bHJuZ19oYW5kbGVy IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDguOTMlICBbMjZdICAgICAgIGNwdV9pZGxlIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMjZdICAgICAgICBzY2hlZF9pZGxldGQKICAxMDAu MCUgIFsyNl0gICAgICAgICBmb3JrX2V4aXQKCjA1LjE1JSAgWzE1XSAgICAgICBfX210eF9sb2Nr X3NsZWVwIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTMuMzMlICBbOF0gICAgICAgICBfX210eF9s b2NrX2ZsYWdzCiAgNzUuMDAlICBbNl0gICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9r ZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbNl0gICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2Nt ZAogIDEyLjUwJSAgWzFdICAgICAgICAgIG1yc2FzX2dldF9tcHRfY21kCiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBtcnNhc19hY3Rpb24KICAxMi41MCUgIFsxXSAgICAgICAgICBtcnNhc191bm1h cF9yZXF1ZXN0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtcnNhc19jbWRfZG9uZQogMjAuMDAl ICBbM10gICAgICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUg IFszXSAgICAgICAgICB2bWVtX2FsbG9jCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBnX2lvX2No ZWNrCiAwNi42NyUgIFsxXSAgICAgICAgIGdfZGlza19zdGFydAogIDEwMC4wJSAgWzFdICAgICAg ICAgIGdfaW9fcmVxdWVzdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2X3N0cmF0ZWd5X2Nz dwogMDYuNjclICBbMV0gICAgICAgICB2bWVtX3hmcmVlCiAgMTAwLjAlICBbMV0gICAgICAgICAg YmlvZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFkb25lCiAwNi42NyUgIFsxXSAgICAg ICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogIDEwMC4wJSAgWzFdICAgICAgICAgIGRhc3RhcnQKICAg MTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAwNi42NyUgIFsxXSAgICAgICAg IHhwdF9ydW5fZGV2cQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdGFydAoKMDQuMTIlICBbMTJdICAgICAgIGNwdV9z ZWFyY2hfbG93ZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMTJdICAgICAgICBj cHVfc2VhcmNoX2xvd2VzdAogIDc1LjAwJSAgWzldICAgICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0 CiAgIDEwMC4wJSAgWzldICAgICAgICAgICBzY2hlZF9waWNrY3B1CiAgMjUuMDAlICBbM10gICAg ICAgICAgc2NoZWRfcGlja2NwdQogICAxMDAuMCUgIFszXSAgICAgICAgICAgc2NoZWRfYWRkCgow My43OCUgIFsxMV0gICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogODEuODIlICBbOV0gICAgICAgICBjcHVfc2VhcmNoX2hpZ2hlc3QKICA3Ny43OCUgIFs3XSAg ICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBbN10gICAgICAgICAgIGZvcmtfZXhpdAog IDIyLjIyJSAgWzJdICAgICAgICAgIGNwdV9zZWFyY2hfaGlnaGVzdAogICAxMDAuMCUgIFsyXSAg ICAgICAgICAgc2NoZWRfaWRsZXRkCiAxOC4xOCUgIFsyXSAgICAgICAgIHNjaGVkX2lkbGV0ZAog IDEwMC4wJSAgWzJdICAgICAgICAgIGZvcmtfZXhpdAoKMDMuMDklICBbOV0gICAgICAgIGNwdV9z d2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiA4OC44OSUgIFs4XSAgICAgICAgIG1pX3N3aXRj aAogIDUwLjAwJSAgWzRdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFs0XSAgICAg ICAgICAgZm9ya19leGl0CiAgNTAuMDAlICBbNF0gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAw LjAlICBbNF0gICAgICAgICAgIF9zbGVlcAoKMDIuNzUlICBbOF0gICAgICAgIHNtcF90bGJfc2hv b3Rkb3duIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbOF0gICAgICAgICBwbWFwX2lu dmFsaWRhdGVfcmFuZ2UKICAxMDAuMCUgIFs4XSAgICAgICAgICBiaW9kb25lCiAgIDEwMC4wJSAg WzhdICAgICAgICAgICBkYWRvbmUKCjAyLjQxJSAgWzddICAgICAgICBzY2hlZF9zd2l0Y2ggQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs3XSAgICAgICAgIG1pX3N3aXRjaAogIDg1Ljcx JSAgWzZdICAgICAgICAgIHNsZWVwcV93YWl0CiAgIDEwMC4wJSAgWzZdICAgICAgICAgICBfc2xl ZXAKICAxNC4yOSUgIFsxXSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGZvcmtfZXhpdAoKMDIuNDElICBbN10gICAgICAgIGRvcmV0aSBAIC9ib290L2tlcm5l bC9rZXJuZWwKCjAyLjA2JSAgWzZdICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiA4My4zMyUgIFs1XSAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogIDEwMC4wJSAg WzVdICAgICAgICAgIGRhc3RhcnQKICAgMTAwLjAlICBbNV0gICAgICAgICAgIHhwdF9ydW5fYWxs b2NxCiAxNi42NyUgIFsxXSAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAxMDAuMCUgIFsxXSAg ICAgICAgICB4cHRfZG9uZV90ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0Cgow Mi4wNiUgIFs2XSAgICAgICAgdGhyZWFkX2xvY2tfZmxhZ3NfIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMzMuMzMlICBbMl0gICAgICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFsyXSAgICAgICAg ICBmb3JrX2V4aXQKIDMzLjMzJSAgWzJdICAgICAgICAgcHJvcGFnYXRlX3ByaW9yaXR5CiAgMTAw LjAlICBbMl0gICAgICAgICAgdHVybnN0aWxlX3dhaXQKICAgMTAwLjAlICBbMl0gICAgICAgICAg IF9fbXR4X2xvY2tfc2xlZXAKIDE2LjY3JSAgWzFdICAgICAgICAgc2xlZXBxX3dhaXQKICAxMDAu MCUgIFsxXSAgICAgICAgICBfc2xlZXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ3YWl0CiAx Ni42NyUgIFsxXSAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgMTAwLjAlICBb MV0gICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGlu dHJfZXhlY3V0ZV9oYW5kbGVycwoKMDEuNzIlICBbNV0gICAgICAgIF9fbG9ja21ncl9hcmdzIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNV0gICAgICAgICBnZXRwYnVmCiAgMTAwLjAl ICBbNV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzVdICAgICAgICAgICBkZXZmc19yZWFk X2YKCjAxLjcyJSAgWzVdICAgICAgICBpdGhyZWFkX2xvb3AgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFs1XSAgICAgICAgIGZvcmtfZXhpdAoKMDEuMzclICBbNF0gICAgICAgIHNwaW5s b2NrX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAyNS4wMCUgIFsxXSAgICAgICAgIHNjaGVk X2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAogMjUuMDAlICBbMV0gICAg ICAgICBfbXR4X2xvY2tfc3Bpbl9jb29raWUKICAxMDAuMCUgIFsxXSAgICAgICAgICBzbXBfdGxi X3Nob290ZG93bgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3Jhbmdl CiAyNS4wMCUgIFsxXSAgICAgICAgIHdha2V1cAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJkb25l CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBidWZkb25lCiAyNS4wMCUgIFsxXSAgICAgICAgIGl0 aHJlYWRfbG9vcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDEuMDMlICBbM10g ICAgICAgIGRldmZzX3dyaXRlX2YgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAg ICAgICAgIGRvZmlsZXdyaXRlCiAgMTAwLjAlICBbM10gICAgICAgICAga2Vybl93cml0ZXYKICAg MTAwLjAlICBbM10gICAgICAgICAgIHN5c193cml0ZQoKMDEuMDMlICBbM10gICAgICAgIG1pX3N3 aXRjaCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2LjY3JSAgWzJdICAgICAgICAgc2xlZXBxX3dh aXQKICAxMDAuMCUgIFsyXSAgICAgICAgICBfc2xlZXAKICAgMTAwLjAlICBbMl0gICAgICAgICAg IGJ3YWl0CiAzMy4zMyUgIFsxXSAgICAgICAgIGl0aHJlYWRfbG9vcAogIDEwMC4wJSAgWzFdICAg ICAgICAgIGZvcmtfZXhpdAoKMDEuMDMlICBbM10gICAgICAgIGdfaW9fcmVxdWVzdCBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogIDEw MC4wJSAgWzNdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFszXSAgICAgICAgICAgZGV2ZnNf cmVhZF9mCgowMS4wMyUgIFszXSAgICAgICAgdW1hX3phbGxvY19hcmcgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAzMy4zMyUgIFsxXSAgICAgICAgIG1hbGxvYwogIDEwMC4wJSAgWzFdICAgICAgICAg IHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5CiAzMy4z MyUgIFsxXSAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsxXSAgICAgICAgICBw aHlzaW8KICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgogMzMuMzMlICBbMV0g ICAgICAgICBnX2Nsb25lX2JpbwogIDEwMC4wJSAgWzFdICAgICAgICAgIGdfZGV2X3N0cmF0ZWd5 CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CgowMS4wMyUgIFszXSAg ICAgICAgYnplcm8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiA2Ni42NyUgIFsyXSAgICAgICAgIGdf Y2xvbmVfYmlvCiAgMTAwLjAlICBbMl0gICAgICAgICAgZ19kZXZfc3RyYXRlZ3kKICAgMTAwLjAl ICBbMl0gICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKIDMzLjMzJSAgWzFdICAgICAgICAgZGV2 X3N0cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsx XSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMS4wMyUgIFszXSAgICAgICAgX210eF9sb2NrX3Nw aW5fY29va2llIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBbMl0gICAgICAgICBzbXBf dGxiX3Nob290ZG93bgogIDEwMC4wJSAgWzJdICAgICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5n ZQogICAxMDAuMCUgIFsyXSAgICAgICAgICAgYmlvZG9uZQogMzMuMzMlICBbMV0gICAgICAgICBz Y2hlZF9pZGxldGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBmb3JrX2V4aXQKCjAxLjAzJSAgWzNd ICAgICAgICBnX2Rldl9zdHJhdGVneSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNd ICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzNdICAgICAgICAgIHBoeXNpbwog ICAxMDAuMCUgIFszXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMS4wMyUgIFszXSAgICAgICAg cG1hcF9pbnZhbGlkYXRlX3JhbmdlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10g ICAgICAgICBiaW9kb25lCiAgMTAwLjAlICBbM10gICAgICAgICAgZGFkb25lCiAgIDEwMC4wJSAg WzNdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMS4wMyUgIFszXSAgICAgICAgZGV2dm5f cmVmdGhyZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBbMl0gICAgICAgICBkZXZm c193cml0ZV9mCiAgMTAwLjAlICBbMl0gICAgICAgICAgZG9maWxld3JpdGUKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIGtlcm5fd3JpdGV2CiAzMy4zMyUgIFsxXSAgICAgICAgIGRldmZzX3JlYWRf ZgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGtlcm5fcmVhZHYKCjAxLjAzJSAgWzNdICAgICAgICB4cHRfZG9uZV9wcm9jZXNzIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB4cHRfZG9uZV90ZAogIDEwMC4w JSAgWzNdICAgICAgICAgIGZvcmtfZXhpdAoKMDEuMDMlICBbM10gICAgICAgIHNjaGVkX2lkbGV0 ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZm9ya19leGl0Cgow MS4wMyUgIFszXSAgICAgICAgY3JpdGljYWxfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2 LjY3JSAgWzJdICAgICAgICAgc3BpbmxvY2tfZXhpdAogIDUwLjAwJSAgWzFdICAgICAgICAgIHNj aGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CiAgNTAuMDAlICBb MV0gICAgICAgICAgX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBid2FpdAogMzMuMzMl ICBbMV0gICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGlu dHJfZXhlY3V0ZV9oYW5kbGVycwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgbGFwaWNfaGFuZGxl X2ludHIKCjAxLjAzJSAgWzNdICAgICAgICBjYWxsb3V0X2xvY2sgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiA2Ni42NyUgIFsyXSAgICAgICAgIGNhbGxvdXRfcmVzZXRfc2J0X29uCiAgMTAwLjAlICBb Ml0gICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4w JSAgWzJdICAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAzMy4z MyUgIFsxXSAgICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogIDEwMC4wJSAgWzFdICAgICAgICAg IG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKCjAwLjY5JSAgWzJdICAgICAgICBzY2hlZF9jaG9v c2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGNob29zZXRocmVh ZAogIDEwMC4wJSAgWzJdICAgICAgICAgIHNjaGVkX3N3aXRjaAogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgbWlfc3dpdGNoCgowMC42OSUgIFsyXSAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBkYXN0YXJ0CiAgMTAwLjAl ICBbMl0gICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRh c3RyYXRlZ3kKCjAwLjY5JSAgWzJdICAgICAgICBYZmFzdF9zeXNjYWxsIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAoKMDAuNjklICBbMl0gICAgICAgIGxhcGljX2lwaV92ZWN0b3JlZCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgc21wX3RsYl9zaG9vdGRvd24KICAxMDAu MCUgIFsyXSAgICAgICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UKICAgMTAwLjAlICBbMl0gICAg ICAgICAgIGJpb2RvbmUKCjAwLjY5JSAgWzJdICAgICAgICBkYXN0cmF0ZWd5IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsy XSAgICAgICAgICBnX2lvX3JlcXVlc3QKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRldl9zdHJh dGVneV9jc3cKCjAwLjY5JSAgWzJdICAgICAgICBjcml0aWNhbF9lbnRlciBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgdGhyZWFkX2xvY2tfZmxhZ3NfCiAgMTAwLjAl ICBbMV0gICAgICAgICAgc2xlZXBxX2FkZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgX3NsZWVw CiA1MC4wMCUgIFsxXSAgICAgICAgIHVtYV96ZnJlZV9hcmcKICAxMDAuMCUgIFsxXSAgICAgICAg ICBnX2Rldl9kb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBnX2lvX2RlbGl2ZXIKCjAwLjY5 JSAgWzJdICAgICAgICBhdG9taWNfZmV0Y2hhZGRfaW50IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog NTAuMDAlICBbMV0gICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwvbXJz YXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxlcnMg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpdGhyZWFkX2xv b3AKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2Fz LmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMC42OSUg IFsyXSAgICAgICAgc2NoZWRfYWRkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0g ICAgICAgICB0dXJuc3RpbGVfdW5wZW5kCiAgMTAwLjAlICBbMV0gICAgICAgICAgX19tdHhfdW5s b2NrX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVuX2RldnEKIDUwLjAwJSAg WzFdICAgICAgICAgc2V0cnVubmFibGUKICAxMDAuMCUgIFsxXSAgICAgICAgICBzbGVlcHFfYnJv YWRjYXN0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB3YWtldXAKCjAwLjY5JSAgWzJdICAgICAg ICBzY2hlZF9ydW5uYWJsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAg ICAgY3B1X2lkbGUKICAxMDAuMCUgIFsyXSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAl ICBbMl0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuNjklICBbMl0gICAgICAgIF9fbXR4X3VubG9j a19mbGFncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNf Z2V0X21wdF9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAg ICBtcnNhc19hY3Rpb24KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgbXJzYXNfY29tcGxldGVfY21k IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVu dF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgaXRocmVhZF9sb29wCgowMC42OSUgIFsyXSAgICAgICAgX19zeXNfcmVhZCBAIC9s aWIvbGliYy5zby43CgowMC42OSUgIFsyXSAgICAgICAgZGV2X3JlbHRocmVhZCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgMTAwLjAlICBb Ml0gICAgICAgICAgZG9maWxlcmVhZAogICAxMDAuMCUgIFsyXSAgICAgICAgICAga2Vybl9yZWFk dgoKMDAuNjklICBbMl0gICAgICAgIHhwdF9kb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMl0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwog IDEwMC4wJSAgWzJdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsyXSAg ICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAoKMDAuNjklICBbMl0gICAgICAgIHVtYV96ZnJlZV9hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsyXSAgICAgICAgIGdfZGV2X2RvbmUKICAxMDAuMCUgIFsyXSAgICAgICAgICBn X2lvX2RlbGl2ZXIKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQoK MDAuMzQlICBbMV0gICAgICAgIGludHJfZXZlbnRfaGFuZGxlIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMV0gICAgICAgICBpbnRyX2V4ZWN1dGVfaGFuZGxlcnMKICAxMDAuMCUgIFsx XSAgICAgICAgICBsYXBpY19oYW5kbGVfaW50cgoKMDAuMzQlICBbMV0gICAgICAgIHN5c2NhbGxf dGhyZWFkX2VudGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBh bWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgX19jYXBfcmlnaHRzX2luaXQgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGtlcm5fd3JpdGV2CiAgMTAwLjAl ICBbMV0gICAgICAgICAgc3lzX3dyaXRlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBhbWQ2NF9z eXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgbXJzYXNfZ2V0X21wdF9jbWQgQCAvYm9vdC9rZXJu ZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfYWN0aW9uCiAgMTAwLjAlICBb MV0gICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUg IFsxXSAgICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMC4zNCUgIFsxXSAgICAgICAgY3B1 X3NldF9zeXNjYWxsX3JldHZhbCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIGdfZGV2X2RvbmUgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGdfaW9fZGVsaXZlcgogIDEwMC4w JSAgWzFdICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZGFkb25lCgowMC4zNCUgIFsxXSAgICAgICAgdGhyZWFkX2xvY2tfc2V0IEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfc3dpdGNoCiAgMTAwLjAlICBb MV0gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9zbGVlcAoK MDAuMzQlICBbMV0gICAgICAgIF9fc3lzX3dyaXRlIEAgL2xpYi9saWJjLnNvLjcKCjAwLjM0JSAg WzFdICAgICAgICBtcnNhc19kYXRhX2xvYWRfY2IgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEw MC4wJSAgWzFdICAgICAgICAgYnVzX2RtYW1hcF9sb2FkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog IDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21y c2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBtcnNhc19idWlsZF9kY2RiCgowMC4zNCUg IFsxXSAgICAgICAga2Vybl9yZWFkdiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgc3lzX3JlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBhbWQ2NF9zeXNjYWxsCgow MC4zNCUgIFsxXSAgICAgICAgc2xlZXBxX3dhaXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0CiAgIDEw MC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBwbWFwX3FlbnRl ciBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19pb19jaGVjawog IDEwMC4wJSAgWzFdICAgICAgICAgIGdfaW9fcmVxdWVzdAogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZGV2X3N0cmF0ZWd5X2NzdwoKMDAuMzQlICBbMV0gICAgICAgIGxvY2tfbXR4IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsxXSAg ICAgICAgICBid2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNCUgIFsx XSAgICAgICAgZGFkb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICB4cHRfZG9uZV9wcm9jZXNzCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfdGQKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBbMV0gICAgICAgIGJ3YWl0 IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBwaHlzaW8KICAxMDAu MCUgIFsxXSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRv ZmlsZXJlYWQKCjAwLjM0JSAgWzFdICAgICAgICBtcnNhc19pc3IgQCAvYm9vdC9rZXJuZWwvbXJz YXMua28KIDEwMC4wJSAgWzFdICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzFdICAgICAgICAgIGl0aHJlYWRfbG9vcAog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgZ19p b19jaGVjayBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19pb19y ZXF1ZXN0CiAgMTAwLjAlICBbMV0gICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogICAxMDAuMCUg IFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNCUgIFsxXSAgICAgICAgZ19kaXNrX2RvbmVfc2lu Z2xlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkYWRvbmUKICAx MDAuMCUgIFsxXSAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICB4cHRfZG9uZV90ZAoKMDAuMzQlICBbMV0gICAgICAgIHR1cm5zdGlsZV9icm9hZGNhc3Qg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIF9fbXR4X3VubG9ja19z bGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fZGV2cQogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMC4zNCUgIFsxXSAgICAgICAgdm1fZmF1bHRf cXVpY2tfaG9sZF9wYWdlcyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAg ICAgdm1hcGJ1ZgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgZ2V0cGJ1ZiBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAgMTAwLjAlICBbMV0g ICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkb2ZpbGVyZWFk CgowMC4zNCUgIFsxXSAgICAgICAgdm1lbV94ZnJlZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgYmlvZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRhZG9uZQog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuMzQlICBbMV0gICAg ICAgIHNtcF9pbnZscGdfcmFuZ2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAg ICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGJpb2Rv bmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDAuMzQlICBbMV0gICAgICAgIGJk b25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBidWZkb25lCiAg MTAwLjAlICBbMV0gICAgICAgICAgYnVmZG9uZWJpbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAg Z19pb19kZWxpdmVyCgowMC4zNCUgIFsxXSAgICAgICAgWGFwaWNfaXNyMSBAIC9ib290L2tlcm5l bC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBwaHlzaW8gQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIGRldmZzX3JlYWRfZgogIDEwMC4wJSAgWzFdICAgICAgICAg IGRvZmlsZXJlYWQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGtlcm5fcmVhZHYKCjAwLjM0JSAg WzFdICAgICAgICBzbGVlcHFfcmVzdW1lX3RocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgc2xlZXBxX2Jyb2FkY2FzdAogIDEwMC4wJSAgWzFdICAgICAgICAg IHdha2V1cAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYmRvbmUKCjAwLjM0JSAgWzFdICAgICAg ICBiY29weSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1kNjRf c3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIGJpb3FfaW5zZXJ0X3RhaWwgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RyYXRlZ3kKICAxMDAuMCUgIFsxXSAg ICAgICAgICBnX2Rpc2tfc3RhcnQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfaW9fcmVxdWVz dAoKMDAuMzQlICBbMV0gICAgICAgIGJpbnVwdGltZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgMTAwLjAlICBbMV0gICAgICAg ICAgZGFkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMC4z NCUgIFsxXSAgICAgICAgY2FtX2NjYnFfcmVtb3ZlX2NjYiBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBbMV0gICAgICAgICAg eHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0YXJ0CgowMC4z NCUgIFsxXSAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAxMDAu MCUgIFsxXSAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAxMDAu MCUgIFsxXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIGRhc3RhcnQKCjAwLjM0JSAgWzFdICAgICAgICBmZ2V0X3VubG9ja2VkIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfZmdldAogIDEwMC4wJSAgWzFdICAgICAg ICAgIGtlcm5fcmVhZHYKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHN5c19yZWFkCgowMC4zNCUg IFsxXSAgICAgICAgc2xlZXBxX2xvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsx XSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0CiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBkZXZmc19yZWFkX2YgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRvZmlsZXJlYWQKICAxMDAu MCUgIFsxXSAgICAgICAgICBrZXJuX3JlYWR2CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzeXNf cmVhZAoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX3BpY2tjcHUgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFdICAgICAgICAg IHR1cm5zdGlsZV91bnBlbmQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9fbXR4X3VubG9ja19z bGVlcAoKMDAuMzQlICBbMV0gICAgICAgIGdfZGlza19zdGFydCBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19pb19yZXF1ZXN0CiAgMTAwLjAlICBbMV0gICAgICAg ICAgZGV2X3N0cmF0ZWd5X2NzdwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4z NCUgIFsxXSAgICAgICAgZGFzdGFydCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYXN0cmF0ZWd5 CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBnX2Rpc2tfc3RhcnQKCjAwLjM0JSAgWzFdICAgICAg ICBtcnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFd ICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogIDEwMC4wJSAgWzFdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgeHB0X3JlbGVhc2VfY2NiIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkYWRvbmUKICAxMDAuMCUgIFsx XSAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRf ZG9uZV90ZAoKQCBDUFVfQ0xLX1VOSEFMVEVEX0NPUkUgWzI5MCBzYW1wbGVzXQoKMTIuNzYlICBb MzddICAgICAgIGludmxybmdfaGFuZGxlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA5LjMxJSAg WzI3XSAgICAgICBjcHVfaWRsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzI3XSAg ICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMjddICAgICAgICAgZm9ya19leGl0CgowNy45 MyUgIFsyM10gICAgICAgWGludmxybmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowNi4yMSUgIFsx OF0gICAgICAgc21wX3RsYl9zaG9vdGRvd24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxOF0gICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzE4XSAgICAgICAg IGJpb2RvbmUKICAgMTAwLjAlICBbMThdICAgICAgICAgIGRhZG9uZQoKMDMuNDUlICBbMTBdICAg ICAgIF9fbXR4X2xvY2tfc2xlZXAgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA2MC4wMCUgIFs2XSAg ICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MKICAxMDAuMCUgIFs2XSAgICAgICAgICBtcnNhc19jbWRf ZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFs2XSAgICAgICAgICAgbXJz YXNfY29tcGxldGVfY21kCiAyMC4wMCUgIFsyXSAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsyXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQK ICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRhc3RhcnQKIDEwLjAwJSAgWzFdICAgICAgICAgdm1l bV94YWxsb2MKICAxMDAuMCUgIFsxXSAgICAgICAgICB2bWVtX2FsbG9jCiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBnX2lvX2NoZWNrCiAxMC4wMCUgIFsxXSAgICAgICAgIHhwdF9kb25lX3Byb2Nl c3MKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfZG9uZV90ZAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZm9ya19leGl0CgowMy4xMCUgIFs5XSAgICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogNjYuNjclICBbNl0gICAgICAgICBjcHVfc2VhcmNoX2hpZ2hl c3QKICA2Ni42NyUgIFs0XSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAwLjAlICBbNF0gICAg ICAgICAgIGZvcmtfZXhpdAogIDMzLjMzJSAgWzJdICAgICAgICAgIGNwdV9zZWFyY2hfaGlnaGVz dAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgc2NoZWRfaWRsZXRkCiAzMy4zMyUgIFszXSAgICAg ICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzNdICAgICAgICAgIGZvcmtfZXhpdAoKMDIuNzYl ICBbOF0gICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbOF0gICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdAogIDc1LjAwJSAgWzZdICAgICAgICAg IGNwdV9zZWFyY2hfbG93ZXN0CiAgIDEwMC4wJSAgWzZdICAgICAgICAgICBzY2hlZF9waWNrY3B1 CiAgMjUuMDAlICBbMl0gICAgICAgICAgc2NoZWRfcGlja2NwdQogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgc2NoZWRfYWRkCgowMi40MSUgIFs3XSAgICAgICAgY3B1X3N3aXRjaCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzddICAgICAgICAgbWlfc3dpdGNoCiAgNDIuODYlICBbM10g ICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbM10gICAgICAgICAgIF9zbGVlcAogIDQy Ljg2JSAgWzNdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFszXSAgICAgICAgICAg Zm9ya19leGl0CiAgMTQuMjklICBbMV0gICAgICAgICAgY3JpdGljYWxfZXhpdAogICAxMDAuMCUg IFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKCjAyLjQxJSAgWzddICAgICAgICBzY2hl ZF9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs3XSAgICAgICAgIG1pX3N3 aXRjaAogIDcxLjQzJSAgWzVdICAgICAgICAgIHNsZWVwcV93YWl0CiAgIDEwMC4wJSAgWzVdICAg ICAgICAgICBfc2xlZXAKICAxNC4yOSUgIFsxXSAgICAgICAgICBzY2hlZF9pZGxldGQKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAogIDE0LjI5JSAgWzFdICAgICAgICAgIGNyaXRp Y2FsX2V4aXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfaGFuZGxlCgowMi40 MSUgIFs3XSAgICAgICAgZG9yZXRpIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDIuMDclICBbNl0g ICAgICAgIHNwaW5sb2NrX2V4aXQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAzMy4zMyUgIFsyXSAg ICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzJdICAgICAgICAgIGZvcmtfZXhpdAogMzMu MzMlICBbMl0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsyXSAgICAgICAgICBid2FpdAogICAx MDAuMCUgIFsyXSAgICAgICAgICAgcGh5c2lvCiAxNi42NyUgIFsxXSAgICAgICAgIHdha2V1cAog IDEwMC4wJSAgWzFdICAgICAgICAgIGJkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBidWZk b25lCiAxNi42NyUgIFsxXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBbMV0g ICAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBi aW9kb25lCgowMi4wNyUgIFs2XSAgICAgICAgX19tdHhfdW5sb2NrX2ZsYWdzIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTYuNjclICBbMV0gICAgICAgICBtcnNhc19hY3Rpb24gQCAvYm9vdC9rZXJu ZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQK IDE2LjY3JSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBbMV0gICAgICAgICAg eHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0YXJ0CiAxNi42 NyUgIFsxXSAgICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtv CiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfYnVpbGRfZGNkYgogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgbXJzYXNfYWN0aW9uCiAxNi42NyUgIFsxXSAgICAgICAgIG1yc2FzX2dldF9tcHRf Y21kCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfYWN0aW9uCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxNi42NyUgIFsxXSAg ICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4w JSAgWzFdICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5l bC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGl0aHJlYWRfbG9vcAogMTYuNjclICBb MV0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4w JSAgWzFdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDEu NzIlICBbNV0gICAgICAgIHNjaGVkX2lkbGV0ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzVdICAgICAgICAgZm9ya19leGl0CgowMS4zOCUgIFs0XSAgICAgICAgY3JpdGljYWxfZXhp dCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDc1LjAwJSAgWzNdICAgICAgICAgc3BpbmxvY2tfZXhp dAogIDMzLjMzJSAgWzFdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZm9ya19leGl0CiAgMzMuMzMlICBbMV0gICAgICAgICAgX3NsZWVwCiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBid2FpdAogIDMzLjMzJSAgWzFdICAgICAgICAgIHRocmVhZF9sb2NrX2Zs YWdzXwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9zY2hlZHVsZV90aHJlYWQK IDI1LjAwJSAgWzFdICAgICAgICAgdW1hX3phbGxvY19hcmcKICAxMDAuMCUgIFsxXSAgICAgICAg ICBtYWxsb2MKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCgowMS4wMyUg IFszXSAgICAgICAgX210eF9sb2NrX3NwaW5fY29va2llIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog NjYuNjclICBbMl0gICAgICAgICB0ZHFfbG9ja19wYWlyCiAgMTAwLjAlICBbMl0gICAgICAgICAg c2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3JrX2V4aXQKIDMzLjMzJSAg WzFdICAgICAgICAgc21wX3RsYl9zaG9vdGRvd24KICAxMDAuMCUgIFsxXSAgICAgICAgICBwbWFw X2ludmFsaWRhdGVfcmFuZ2UKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJpb2RvbmUKCjAxLjAz JSAgWzNdICAgICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzNdICAgICAgICAgdm1lbV9hbGxvYwogIDEwMC4wJSAgWzNdICAgICAgICAgIGdfaW9fY2hlY2sK ICAgMTAwLjAlICBbM10gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDEuMDMlICBbM10gICAgICAg IF9fc3lzX3JlYWQgQCAvbGliL2xpYmMuc28uNwoKMDEuMDMlICBbM10gICAgICAgIHRocmVhZF9s b2NrX2ZsYWdzXyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2LjY3JSAgWzJdICAgICAgICAgc2xl ZXBxX2FkZAogIDEwMC4wJSAgWzJdICAgICAgICAgIF9zbGVlcAogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgYndhaXQKIDMzLjMzJSAgWzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBb MV0gICAgICAgICAgZm9ya19leGl0CgowMS4wMyUgIFszXSAgICAgICAgYmlvZG9uZSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDY2LjY3JSAgWzJdICAgICAgICAgZGFkb25lCiAgMTAwLjAlICBbMl0g ICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogICAxMDAuMCUgIFsyXSAgICAgICAgICAgeHB0X2Rv bmVfdGQKIDMzLjMzJSAgWzFdICAgICAgICAgZ19pb19kZWxpdmVyCiAgMTAwLjAlICBbMV0gICAg ICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUK CjAxLjAzJSAgWzNdICAgICAgICBjcml0aWNhbF9lbnRlciBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDY2LjY3JSAgWzJdICAgICAgICAgc21wX3RsYl9zaG9vdGRvd24KICAxMDAuMCUgIFsyXSAgICAg ICAgICBwbWFwX2ludmFsaWRhdGVfcmFuZ2UKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGJpb2Rv bmUKIDMzLjMzJSAgWzFdICAgICAgICAgdW1hX3phbGxvY19hcmcKICAxMDAuMCUgIFsxXSAgICAg ICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAx LjAzJSAgWzNdICAgICAgICBkYXN0YXJ0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb M10gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEwMC4wJSAgWzNdICAgICAgICAgIGRhc3RyYXRl Z3kKICAgMTAwLjAlICBbM10gICAgICAgICAgIGdfZGlza19zdGFydAoKMDEuMDMlICBbM10gICAg ICAgIGRldnZuX3JlZnRocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDY2LjY3JSAgWzJdICAg ICAgICAgZGV2ZnNfcmVhZF9mCiAgMTAwLjAlICBbMl0gICAgICAgICAgZG9maWxlcmVhZAogICAx MDAuMCUgIFsyXSAgICAgICAgICAga2Vybl9yZWFkdgogMzMuMzMlICBbMV0gICAgICAgICBkZXZm c193cml0ZV9mCiAgMTAwLjAlICBbMV0gICAgICAgICAgZG9maWxld3JpdGUKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGtlcm5fd3JpdGV2CgowMS4wMyUgIFszXSAgICAgICAgdm1lbV94ZnJlZSBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgYmlvZG9uZQogIDEwMC4w JSAgWzNdICAgICAgICAgIGRhZG9uZQogICAxMDAuMCUgIFszXSAgICAgICAgICAgeHB0X2RvbmVf cHJvY2VzcwoKMDEuMDMlICBbM10gICAgICAgIGFtZDY0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwv a2VybmVsCgowMS4wMyUgIFszXSAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbM10gICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKICAxMDAuMCUgIFsz XSAgICAgICAgICBkYXN0YXJ0CiAgIDEwMC4wJSAgWzNdICAgICAgICAgICB4cHRfcnVuX2FsbG9j cQoKMDAuNjklICBbMl0gICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdCBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZGFzdGFydAogIDEwMC4wJSAgWzJdICAgICAgICAg IHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkYXN0cmF0ZWd5CgowMC42 OSUgIFsyXSAgICAgICAgZ19kZXZfc3RyYXRlZ3kgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsyXSAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsyXSAgICAgICAgICBw aHlzaW8KICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuNjklICBbMl0g ICAgICAgIGZvZmZzZXRfdW5sb2NrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBkZXZmc193cml0ZV9mCiAgMTAwLjAlICBbMl0gICAgICAgICAgZG9maWxld3JpdGUK ICAgMTAwLjAlICBbMl0gICAgICAgICAgIGtlcm5fd3JpdGV2CgowMC42OSUgIFsyXSAgICAgICAg aXRocmVhZF9sb29wIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBm b3JrX2V4aXQKCjAwLjY5JSAgWzJdICAgICAgICBwY3B1X2ZpbmQgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIHRkcV9ub3RpZnkKICAxMDAuMCUgIFsyXSAgICAgICAg ICBzY2hlZF9hZGQKICAgNTAuMDAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVf dGhyZWFkCiAgIDUwLjAwJSAgWzFdICAgICAgICAgICB0dXJuc3RpbGVfdW5wZW5kCgowMC42OSUg IFsyXSAgICAgICAgZ19kaXNrX3N0YXJ0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb Ml0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFsyXSAgICAgICAgICBkZXZfc3RyYXRl Z3lfY3N3CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBwaHlzaW8KCjAwLjY5JSAgWzJdICAgICAg ICByZWxwYnVmIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBwaHlz aW8KICAxMDAuMCUgIFsyXSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMl0gICAg ICAgICAgIGRvZmlsZXJlYWQKCjAwLjY5JSAgWzJdICAgICAgICBtcnNhc19nZXRfbXB0X2NtZCBA IC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbMl0gICAgICAgICBtcnNhc19hY3Rpb24K ICAxMDAuMCUgIFsyXSAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQKCjAwLjY5JSAgWzJd ICAgICAgICBwbWFwX2tleHRyYWN0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBib3VuY2VfYnVzX2RtYW1hcF9sb2FkX2J1ZmZlcgogIDEwMC4wJSAgWzJdICAgICAg ICAgIGJ1c19kbWFtYXBfbG9hZAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgbXJzYXNfbWFwX3Jl cXVlc3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KCjAwLjY5JSAgWzJdICAgICAgICBfY2FsbG91 dF9zdG9wX3NhZmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIG1y c2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMl0gICAgICAg ICAgbXJzYXNfY29tcGxldGVfY21kCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBpbnRyX2V2ZW50 X2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC42OSUgIFsyXSAgICAg ICAgYnplcm8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIG1yc2Fz X2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhw dF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAg IHhwdF9hY3Rpb25fZGVmYXVsdAogNTAuMDAlICBbMV0gICAgICAgICBnX2Nsb25lX2JpbwogIDEw MC4wJSAgWzFdICAgICAgICAgIGdfZGV2X3N0cmF0ZWd5CiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBkZXZfc3RyYXRlZ3lfY3N3CgowMC42OSUgIFsyXSAgICAgICAgc2NoZWRfcGlja2NwdSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgc2NoZWRfYWRkCiAgNTAuMDAl ICBbMV0gICAgICAgICAgc2V0cnVubmFibGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNsZWVw cV9icm9hZGNhc3QKICA1MC4wMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X3NjaGVkdWxlX3Ro cmVhZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKCjAwLjY5JSAg WzJdICAgICAgICBidF9mcmVldHJpbSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJd ICAgICAgICAgYmlvZG9uZQogIDEwMC4wJSAgWzJdICAgICAgICAgIGRhZG9uZQogICAxMDAuMCUg IFsyXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuNjklICBbMl0gICAgICAgIGdfaW9f cmVxdWVzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZGV2X3N0 cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzJdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsyXSAg ICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC42OSUgIFsyXSAgICAgICAgZGFkb25lIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgMTAw LjAlICBbMl0gICAgICAgICAgeHB0X2RvbmVfdGQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGZv cmtfZXhpdAoKMDAuNjklICBbMl0gICAgICAgIHN5c19yZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMl0gICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC42OSUgIFsyXSAgICAgICAg c2V0cnVubmFibGUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIHNs ZWVwcV9icm9hZGNhc3QKICAxMDAuMCUgIFsyXSAgICAgICAgICB3YWtldXAKICAgNTAuMDAlICBb MV0gICAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDUw LjAwJSAgWzFdICAgICAgICAgICBiZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjY5JSAg WzJdICAgICAgICBfX2xvY2ttZ3JfYXJncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzJdICAgICAgICAgZ2V0cGJ1ZgogIDEwMC4wJSAgWzJdICAgICAgICAgIHBoeXNpbwogICAxMDAu MCUgIFsyXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgdW1hX3ph bGxvY19hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIG1hbGxv YwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkYXN0cmF0ZWd5CgowMC4zNCUgIFsxXSAgICAgICAgZ19pc19nZW9tX3RocmVhZCBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19pb19kZWxpdmVyCiAg MTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkYWRvbmUKCjAwLjM0JSAgWzFdICAgICAgICBjYW1xX3JlbW92ZSBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAwLjAlICBb MV0gICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBk YXN0YXJ0CgowMC4zNCUgIFsxXSAgICAgICAgc2NoZWRfY2hvb3NlIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMV0gICAgICAgICBjaG9vc2V0aHJlYWQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBzY2hlZF9zd2l0Y2gKICAgMTAwLjAlICBbMV0gICAgICAgICAgIG1pX3N3aXRjaAoKMDAu MzQlICBbMV0gICAgICAgIHRkcV9ub3RpZnkgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNldHJ1bm5hYmxl CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CgowMC4zNCUgIFsxXSAg ICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICBsYXBpY19oYW5kbGVfaW50cgoKMDAuMzQlICBbMV0gICAgICAgIHhwdF9h Y3Rpb24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhc3RhcnQK ICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZGFzdHJhdGVneQoKMDAuMzQlICBbMV0gICAgICAgIHR1cm5zdGlsZV90cnl3YWl0IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBfX210eF9sb2NrX3NsZWVw CiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgeHB0X2RvbmVfdGQKCjAwLjM0JSAgWzFdICAgICAgICBrZXJuX3dyaXRldiBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3lzX3dyaXRlCiAgMTAwLjAl ICBbMV0gICAgICAgICAgYW1kNjRfc3lzY2FsbAoKMDAuMzQlICBbMV0gICAgICAgIFhmYXN0X3N5 c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4zNCUgIFsxXSAgICAgICAgd2FrZXVwIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBiZG9uZQogIDEwMC4wJSAg WzFdICAgICAgICAgIGJ1ZmRvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJ1ZmRvbmViaW8K CjAwLjM0JSAgWzFdICAgICAgICBkZXZzdGF0X2VuZF90cmFuc2FjdGlvbiBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2c3RhdF9lbmRfdHJhbnNhY3Rpb25fYmlv X2J0CiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBkYWRvbmUKCjAwLjM0JSAgWzFdICAgICAgICBzbGVlcHFfYnJvYWRjYXN0 IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB3YWtldXAKICAxMDAu MCUgIFsxXSAgICAgICAgICBiZG9uZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYnVmZG9uZQoK MDAuMzQlICBbMV0gICAgICAgIHhwdF9ydW5fYWxsb2NxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMV0gICAgICAgICBkYXN0cmF0ZWd5CiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19k aXNrX3N0YXJ0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBnX2lvX3JlcXVlc3QKCjAwLjM0JSAg WzFdICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAu MCUgIFsxXSAgICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9kb25lX3By b2Nlc3MKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3RkCgowMC4zNCUgIFsxXSAg ICAgICAgbGFwaWNfaXBpX3ZlY3RvcmVkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICBzbXBfdGxiX3Nob290ZG93bgogIDEwMC4wJSAgWzFdICAgICAgICAgIHBtYXBf aW52YWxpZGF0ZV9yYW5nZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgYmlvZG9uZQoKMDAuMzQl ICBbMV0gICAgICAgIHNjc2lfcmVhZF93cml0ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgZGFzdGFydAogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fYWxs b2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5CgowMC4zNCUgIFsxXSAgICAg ICAgZGFzdHJhdGVneSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg Z19kaXNrX3N0YXJ0CiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19pb19yZXF1ZXN0CiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CgowMC4zNCUgIFsxXSAgICAgICAga2Vy bl9yZWFkdiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc3lzX3Jl YWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBhbWQ2NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAg ICAgeHB0X2RvbmVfcHJvY2VzcyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgeHB0X2RvbmVfdGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM0 JSAgWzFdICAgICAgICB0dXJuc3RpbGVfd2FpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgX19tdHhfbG9ja19zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIHhw dF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGFzdGFydAoKMDAuMzQl ICBbMV0gICAgICAgIF9fc3lzX3dyaXRlIEAgL2xpYi9saWJjLnNvLjcKCjAwLjM0JSAgWzFdICAg ICAgICBtdHhfcG9vbF9maW5kIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICBiZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ1ZmRvbmUKICAgMTAwLjAlICBbMV0g ICAgICAgICAgIGJ1ZmRvbmViaW8KCjAwLjM0JSAgWzFdICAgICAgICBoYW5kbGVldmVudHMgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHRpbWVyY2IKICAxMDAuMCUg IFsxXSAgICAgICAgICBsYXBpY19oYW5kbGVfdGltZXIKCjAwLjM0JSAgWzFdICAgICAgICBtYWxs b2MgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHhwdF9ydW5fYWxs b2NxCiAgMTAwLjAlICBbMV0gICAgICAgICAgZGFzdHJhdGVneQogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgZ19kaXNrX3N0YXJ0CgowMC4zNCUgIFsxXSAgICAgICAgYXRvbWljX2ZldGNoYWRkX2lu dCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfYWN0aW9u IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgeHB0X3J1bl9k ZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X2Fj dGlvbl9kZWZhdWx0CgowMC4zNCUgIFsxXSAgICAgICAgc2NoZWRfcnVubmFibGUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGNwdV9pZGxlCiAgMTAwLjAlICBbMV0g ICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQK CjAwLjM0JSAgWzFdICAgICAgICBfX2NhcF9yaWdodHNfaXNfc2V0IEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMV0gICAgICAgICBmZ2V0X3VubG9ja2VkCiAgMTAwLjAlICBbMV0gICAg ICAgICAgX2ZnZXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGtlcm5fcmVhZHYKCjAwLjM0JSAg WzFdICAgICAgICBkb2ZpbGV3cml0ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAga2Vybl93cml0ZXYKICAxMDAuMCUgIFsxXSAgICAgICAgICBzeXNfd3JpdGUKICAg MTAwLjAlICBbMV0gICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFdICAgICAgICBt cnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAg ICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog IDEwMC4wJSAgWzFdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgX19tdHhfbG9ja19mbGFncyBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfZ2V0X21wdF9jbWQgQCAv Ym9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19hY3Rpb24K ICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKCjAwLjM0JSAgWzFdICAgICAgICBnX2Rldl9kb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMV0gICAgICAgICBnX2lvX2RlbGl2ZXIKICAxMDAuMCUgIFsxXSAgICAgICAgICBn X2Rpc2tfZG9uZV9zaW5nbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDAuMzQl ICBbMV0gICAgICAgIHRocmVhZF9sb2NrX3NldCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgc2xlZXBxX3N3aXRjaAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNsZWVw cV93YWl0CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBfc2xlZXAKCjAwLjM0JSAgWzFdICAgICAg ICBzbGVlcHFfd2FpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAg X3NsZWVwCiAgMTAwLjAlICBbMV0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIHBoeXNpbwoKMDAuMzQlICBbMV0gICAgICAgIHNjaGVkX3NsZWVwIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfc3dpdGNoCiAgMTAwLjAlICBbMV0g ICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9zbGVlcAoKMDAu MzQlICBbMV0gICAgICAgIGxvY2tfbXR4IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICBfc2xlZXAKICAxMDAuMCUgIFsxXSAgICAgICAgICBid2FpdAogICAxMDAuMCUg IFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNCUgIFsxXSAgICAgICAgc3lzY2FsbF90aHJlYWRf ZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1kNjRfc3lz Y2FsbAoKMDAuMzQlICBbMV0gICAgICAgIGJ3YWl0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZmc19yZWFk X2YKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRvZmlsZXJlYWQKCjAwLjM0JSAgWzFdICAgICAg ICBjaG9vc2V0aHJlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAg IHNjaGVkX3N3aXRjaAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1pX3N3aXRjaAogICAxMDAuMCUg IFsxXSAgICAgICAgICAgc2xlZXBxX3dhaXQKCjAwLjM0JSAgWzFdICAgICAgICB2bV9mYXVsdF9x dWlja19ob2xkX3BhZ2VzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICB2bWFwYnVmCiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBkZXZmc19yZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICBnZXRwYnVmIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFsxXSAg ICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRvZmlsZXJlYWQK CjAwLjM0JSAgWzFdICAgICAgICB4cHRfZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAx MDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwK CjAwLjM0JSAgWzFdICAgICAgICBQSFlTX1RPX1ZNX1BBR0UgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIGZyZWUKICAxMDAuMCUgIFsxXSAgICAgICAgICB4cHRfcmVs ZWFzZV9jY2IKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9uZQoKMDAuMzQlICBbMV0gICAg ICAgIHVtYV96ZnJlZV9hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIGdfZGV2X2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2lvX2RlbGl2ZXIKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQoKMDAuMzQlICBbMV0gICAgICAg IGRldnN0YXRfc3RhcnRfdHJhbnNhY3Rpb25fYmlvIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2lv X3JlcXVlc3QKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKCjAwLjM0 JSAgWzFdICAgICAgICBkZXZmc193cml0ZV9mIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICBkb2ZpbGV3cml0ZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGtlcm5fd3Jp dGV2CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzeXNfd3JpdGUKCjAwLjM0JSAgWzFdICAgICAg ICBiY29weSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYW1kNjRf c3lzY2FsbAoKQCBDUFVfQ0xLX1VOSEFMVEVEX0NPUkUgWzI5MCBzYW1wbGVzXQoKMTIuMDclICBb MzVdICAgICAgIGNwdV9pZGxlIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMzVdICAg ICAgICBzY2hlZF9pZGxldGQKICAxMDAuMCUgIFszNV0gICAgICAgICBmb3JrX2V4aXQKCjExLjM4 JSAgWzMzXSAgICAgICBYaW52bHJuZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjA4Ljk3JSAgWzI2 XSAgICAgICBpbnZscm5nX2hhbmRsZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowNS44NiUgIFsx N10gICAgICAgc21wX3RsYl9zaG9vdGRvd24gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFsxN10gICAgICAgIHBtYXBfaW52YWxpZGF0ZV9yYW5nZQogIDEwMC4wJSAgWzE3XSAgICAgICAg IGJpb2RvbmUKICAgMTAwLjAlICBbMTddICAgICAgICAgIGRhZG9uZQoKMDMuNzklICBbMTFdICAg ICAgIF9fbXR4X2xvY2tfc2xlZXAgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA1NC41NSUgIFs2XSAg ICAgICAgIF9fbXR4X2xvY2tfZmxhZ3MKICAxMDAuMCUgIFs2XSAgICAgICAgICBtcnNhc19jbWRf ZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogICAxMDAuMCUgIFs2XSAgICAgICAgICAgbXJz YXNfY29tcGxldGVfY21kCiAxOC4xOCUgIFsyXSAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKICAxMDAuMCUgIFsyXSAgICAgICAgICB4cHRfYWN0aW9uX2RlZmF1bHQK ICAgMTAwLjAlICBbMl0gICAgICAgICAgIGRhc3RhcnQKIDE4LjE4JSAgWzJdICAgICAgICAgdm1l bV94YWxsb2MKICAxMDAuMCUgIFsyXSAgICAgICAgICB2bWVtX2FsbG9jCiAgIDEwMC4wJSAgWzJd ICAgICAgICAgICBnX2lvX2NoZWNrCiAwOS4wOSUgIFsxXSAgICAgICAgIGRhc3RhcnQKICAxMDAu MCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAgICAgICAg ZGFzdHJhdGVneQoKMDMuNDUlICBbMTBdICAgICAgIGNwdV9zZWFyY2hfaGlnaGVzdCBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDkwLjAwJSAgWzldICAgICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0CiAg NTUuNTYlICBbNV0gICAgICAgICAgc2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzVdICAgICAgICAg ICBmb3JrX2V4aXQKICA0NC40NCUgIFs0XSAgICAgICAgICBjcHVfc2VhcmNoX2hpZ2hlc3QKICAg MTAwLjAlICBbNF0gICAgICAgICAgIHNjaGVkX2lkbGV0ZAogMTAuMDAlICBbMV0gICAgICAgICBz Y2hlZF9pZGxldGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBmb3JrX2V4aXQKCjAzLjEwJSAgWzld ICAgICAgICBjcHVfc2VhcmNoX2xvd2VzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzldICAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QKICA2Ni42NyUgIFs2XSAgICAgICAgICBjcHVf c2VhcmNoX2xvd2VzdAogICAxMDAuMCUgIFs2XSAgICAgICAgICAgc2NoZWRfcGlja2NwdQogIDMz LjMzJSAgWzNdICAgICAgICAgIHNjaGVkX3BpY2tjcHUKICAgMTAwLjAlICBbM10gICAgICAgICAg IHNjaGVkX2FkZAoKMDIuNzYlICBbOF0gICAgICAgIGNwdV9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFs4XSAgICAgICAgIG1pX3N3aXRjaAogIDYyLjUwJSAgWzVdICAgICAg ICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFs1XSAgICAgICAgICAgZm9ya19leGl0CiAgMzcu NTAlICBbM10gICAgICAgICAgc2xlZXBxX3dhaXQKICAgMTAwLjAlICBbM10gICAgICAgICAgIF9z bGVlcAoKMDIuMDclICBbNl0gICAgICAgIF9fbXR4X3VubG9ja19mbGFncyBAIC9ib290L2tlcm5l bC9rZXJuZWwKIDMzLjMzJSAgWzJdICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kIEAgL2Jvb3Qv a2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMl0gICAgICAgICAgaW50cl9ldmVudF9leGVjdXRl X2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsyXSAgICAgICAgICAg aXRocmVhZF9sb29wCiAzMy4zMyUgIFsyXSAgICAgICAgIHhwdF9ydW5fZGV2cQogIDEwMC4wJSAg WzJdICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdAogICAxMDAuMCUgIFsyXSAgICAgICAgICAg ZGFzdGFydAogMTYuNjclICBbMV0gICAgICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5l bC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAx MDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTYuNjclICBbMV0gICAgICAgICBtcnNhc19nZXRfbXB0X2NtZCBAIC9i b290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIG1yc2FzX2FjdGlvbgog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAoKMDIuMDclICBbNl0gICAgICAgIGFtZDY0X3N5c2NhbGwgQCAvYm9vdC9rZXJuZWwva2VybmVs CgowMi4wNyUgIFs2XSAgICAgICAgc3BpbmxvY2tfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDMzLjMzJSAgWzJdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMl0gICAgICAgICAg Zm9ya19leGl0CiAxNi42NyUgIFsxXSAgICAgICAgIHRocmVhZF9sb2NrX2ZsYWdzXwogIDEwMC4w JSAgWzFdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9y a19leGl0CiAxNi42NyUgIFsxXSAgICAgICAgIHNsZWVwcV9icm9hZGNhc3QKICAxMDAuMCUgIFsx XSAgICAgICAgICB3YWtldXAKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGJkb25lCiAxNi42NyUg IFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0CiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBwaHlzaW8KIDE2LjY3JSAgWzFdICAgICAgICAgaW50cl9ldmVudF9o YW5kbGUKICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V4ZWN1dGVfaGFuZGxlcnMKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGxhcGljX2hhbmRsZV9pbnRyCgowMS43MiUgIFs1XSAgICAgICAg ZG9yZXRpIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDEuNzIlICBbNV0gICAgICAgIHRocmVhZF9s b2NrX2ZsYWdzXyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDQwLjAwJSAgWzJdICAgICAgICAgc2xl ZXBxX2FkZAogIDEwMC4wJSAgWzJdICAgICAgICAgIF9zbGVlcAogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgYndhaXQKIDIwLjAwJSAgWzFdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBb MV0gICAgICAgICAgZm9ya19leGl0CiAyMC4wMCUgIFsxXSAgICAgICAgIGNyaXRpY2FsX2V4aXQK ICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgaW50cl9leGVjdXRlX2hhbmRsZXJzCiAyMC4wMCUgIFsxXSAgICAgICAgIGludHJf ZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVudF9o YW5kbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwoKMDEu MzglICBbNF0gICAgICAgIHZtX3BhZ2VfdW5ob2xkX3BhZ2VzIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbNF0gICAgICAgICB2dW5tYXBidWYKICAxMDAuMCUgIFs0XSAgICAgICAgICBw aHlzaW8KICAgMTAwLjAlICBbNF0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDEuMzglICBbNF0g ICAgICAgIHNjaGVkX2lkbGV0ZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzRdICAg ICAgICAgZm9ya19leGl0CgowMS4zOCUgIFs0XSAgICAgICAgc3BpbmxvY2tfZW50ZXIgQCAvYm9v dC9rZXJuZWwva2VybmVsCiA3NS4wMCUgIFszXSAgICAgICAgIHRocmVhZF9sb2NrX2ZsYWdzXwog IDY2LjY3JSAgWzJdICAgICAgICAgIGludHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgIDEwMC4w JSAgWzJdICAgICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDMzLjMzJSAgWzFdICAgICAgICAg IGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZm9ya19leGl0CiAyNS4wMCUg IFsxXSAgICAgICAgIHNjaGVkX3N3aXRjaAogIDEwMC4wJSAgWzFdICAgICAgICAgIG1pX3N3aXRj aAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xlZXBxX3dhaXQKCjAxLjM4JSAgWzRdICAgICAg ICBzY2hlZF9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs0XSAgICAgICAg IG1pX3N3aXRjaAogIDc1LjAwJSAgWzNdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUg IFszXSAgICAgICAgICAgZm9ya19leGl0CiAgMjUuMDAlICBbMV0gICAgICAgICAgc2xlZXBxX3dh aXQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIF9zbGVlcAoKMDEuMzglICBbNF0gICAgICAgIF9f bG9ja21ncl9hcmdzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMl0gICAgICAgICBy ZWxwYnVmCiAgMTAwLjAlICBbMl0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzJdICAgICAg ICAgICBkZXZmc19yZWFkX2YKIDUwLjAwJSAgWzJdICAgICAgICAgZ2V0cGJ1ZgogIDEwMC4wJSAg WzJdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsyXSAgICAgICAgICAgZGV2ZnNfcmVhZF9m CgowMS4wMyUgIFszXSAgICAgICAgWGFwaWNfaXNyMSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAx LjAzJSAgWzNdICAgICAgICBiemVybyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDMzLjMzJSAgWzFd ICAgICAgICAgZ19jbG9uZV9iaW8KICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2Rldl9zdHJhdGVn eQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogMzMuMzMlICBbMV0g ICAgICAgICBrZXJuX3dyaXRldgogIDEwMC4wJSAgWzFdICAgICAgICAgIHN5c193cml0ZQogICAx MDAuMCUgIFsxXSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAogMzMuMzMlICBbMV0gICAgICAgICBk ZXZfc3RyYXRlZ3lfY3N3CiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAg WzFdICAgICAgICAgICBkZXZmc19yZWFkX2YKCjAxLjAzJSAgWzNdICAgICAgICBjYWxsb3V0X2xv Y2sgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA2Ni42NyUgIFsyXSAgICAgICAgIF9jYWxsb3V0X3N0 b3Bfc2FmZQogIDEwMC4wJSAgWzJdICAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2Vy bmVsL21yc2FzLmtvCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQK IDMzLjMzJSAgWzFdICAgICAgICAgY2FsbG91dF9yZXNldF9zYnRfb24gQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVs L21yc2FzLmtvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9r ZXJuZWwva2VybmVsCgowMS4wMyUgIFszXSAgICAgICAgbWlfc3dpdGNoIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogNjYuNjclICBbMl0gICAgICAgICBzbGVlcHFfd2FpdAogIDEwMC4wJSAgWzJdICAg ICAgICAgIF9zbGVlcAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgYndhaXQKIDMzLjMzJSAgWzFd ICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMV0gICAgICAgICAgZm9ya19leGl0Cgow MS4wMyUgIFszXSAgICAgICAgZGFzdGFydCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzNdICAgICAgICAgeHB0X3J1bl9hbGxvY3EKICAxMDAuMCUgIFszXSAgICAgICAgICBkYXN0cmF0 ZWd5CiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBnX2Rpc2tfc3RhcnQKCjAxLjAzJSAgWzNdICAg ICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzNdICAgICAg ICAgdm1lbV9hbGxvYwogIDEwMC4wJSAgWzNdICAgICAgICAgIGdfaW9fY2hlY2sKICAgMTAwLjAl ICBbM10gICAgICAgICAgIGdfaW9fcmVxdWVzdAoKMDEuMDMlICBbM10gICAgICAgIGdldHBidWYg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIHBoeXNpbwogIDEwMC4w JSAgWzNdICAgICAgICAgIGRldmZzX3JlYWRfZgogICAxMDAuMCUgIFszXSAgICAgICAgICAgZG9m aWxlcmVhZAoKMDEuMDMlICBbM10gICAgICAgIGl0aHJlYWRfbG9vcCBAIC9ib290L2tlcm5lbC9r ZXJuZWwKIDEwMC4wJSAgWzNdICAgICAgICAgZm9ya19leGl0CgowMS4wMyUgIFszXSAgICAgICAg eHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICB4 cHRfYWN0aW9uX2RlZmF1bHQKICAxMDAuMCUgIFszXSAgICAgICAgICBkYXN0YXJ0CiAgIDEwMC4w JSAgWzNdICAgICAgICAgICB4cHRfcnVuX2FsbG9jcQoKMDAuNjklICBbMl0gICAgICAgIGhhbmRs ZWV2ZW50cyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgdGltZXJj YgogIDEwMC4wJSAgWzJdICAgICAgICAgIGxhcGljX2hhbmRsZV90aW1lcgoKMDAuNjklICBbMl0g ICAgICAgIF9fc3lzX3JlYWQgQCAvbGliL2xpYmMuc28uNwoKMDAuNjklICBbMl0gICAgICAgIGRl dl9yZWx0aHJlYWQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGRl dmZzX3JlYWRfZgogIDEwMC4wJSAgWzJdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIGtlcm5fcmVhZHYKCjAwLjY5JSAgWzJdICAgICAgICBjcml0aWNhbF9leGl0 IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICBtYWxsb2MKICAxMDAu MCUgIFsxXSAgICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFsxXSAgICAgICAgICAg ZGFzdHJhdGVneQogNTAuMDAlICBbMV0gICAgICAgICBzcGlubG9ja19leGl0CiAgMTAwLjAlICBb MV0gICAgICAgICAgdGhyZWFkX2xvY2tfYmxvY2sKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNj aGVkX2FkZAoKMDAuNjklICBbMl0gICAgICAgIGF0b21pY19mZXRjaGFkZF9pbnQgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiA1MC4wMCUgIFsxXSAgICAgICAgIG1yc2FzX2FjdGlvbiBAIC9ib290L2tl cm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9hY3Rpb25fZGVmYXVs dAogNTAuMDAlICBbMV0gICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwv bXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxl cnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBpdGhyZWFk X2xvb3AKCjAwLjY5JSAgWzJdICAgICAgICB2bWVtX3hmcmVlIEAgL2Jvb3Qva2VybmVsL2tlcm5l bAogMTAwLjAlICBbMl0gICAgICAgICBiaW9kb25lCiAgMTAwLjAlICBbMl0gICAgICAgICAgZGFk b25lCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMC42OSUgIFsy XSAgICAgICAgZ19kaXNrX3N0YXJ0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFsyXSAgICAgICAgICBkZXZfc3RyYXRlZ3lf Y3N3CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBwaHlzaW8KCjAwLjY5JSAgWzJdICAgICAgICBw bWFwX2ludmFsaWRhdGVfcmFuZ2UgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAg ICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFsyXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBb Ml0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjY5JSAgWzJdICAgICAgICBwaHlzaW8g QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGRldmZzX3JlYWRfZgog IDEwMC4wJSAgWzJdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAlICBbMl0gICAgICAgICAg IGtlcm5fcmVhZHYKCjAwLjY5JSAgWzJdICAgICAgICBwY3B1X2ZpbmQgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIHRkcV9ub3RpZnkKICAxMDAuMCUgIFsyXSAgICAg ICAgICBzY2hlZF9hZGQKICAgNTAuMDAlICBbMV0gICAgICAgICAgIHR1cm5zdGlsZV91bnBlbmQK ICAgNTAuMDAlICBbMV0gICAgICAgICAgIHNldHJ1bm5hYmxlCgowMC42OSUgIFsyXSAgICAgICAg a2Vybl93cml0ZXYgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIHN5 c193cml0ZQogIDEwMC4wJSAgWzJdICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjY5JSAgWzJd ICAgICAgICBmZ2V0X3VubG9ja2VkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0g ICAgICAgICBfZmdldAogIDUwLjAwJSAgWzFdICAgICAgICAgIGtlcm5fd3JpdGV2CiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBzeXNfd3JpdGUKICA1MC4wMCUgIFsxXSAgICAgICAgICBrZXJuX3Jl YWR2CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzeXNfcmVhZAoKMDAuNjklICBbMl0gICAgICAg IGdfZGlza19kb25lX3NpbmdsZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAg ICAgICAgZGFkb25lCiAgMTAwLjAlICBbMl0gICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogICAx MDAuMCUgIFsyXSAgICAgICAgICAgeHB0X2RvbmVfdGQKCjAwLjM0JSAgWzFdICAgICAgICBfbXR4 X2xvY2tfc3Bpbl9jb29raWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQl ICBbMV0gICAgICAgIHhwdF9hY3Rpb25fZGVmYXVsdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEw MC4wJSAgWzFdICAgICAgICAgZGFzdGFydAogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9ydW5f YWxsb2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5CgowMC4zNCUgIFsxXSAg ICAgICAgZGV2dm5fcmVmdGhyZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0g ICAgICAgICBkZXZmc19yZWFkX2YKICAxMDAuMCUgIFsxXSAgICAgICAgICBkb2ZpbGVyZWFkCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBrZXJuX3JlYWR2CgowMC4zNCUgIFsxXSAgICAgICAgcG1h cF9leHRyYWN0X2FuZF9ob2xkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICB2bV9mYXVsdF9xdWlja19ob2xkX3BhZ2VzCiAgMTAwLjAlICBbMV0gICAgICAgICAgdm1h cGJ1ZgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgcGh5c2lvCgowMC4zNCUgIFsxXSAgICAgICAg YnRfZnJlZXRyaW0gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJp b2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbMV0gICAgICAg ICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjM0JSAgWzFdICAgICAgICBkZXZzdGF0X2VuZF90cmFu c2FjdGlvbiBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2c3Rh dF9lbmRfdHJhbnNhY3Rpb25fYmlvX2J0CiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kaXNrX2Rv bmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAwLjM0JSAgWzFdICAg ICAgICB4cHRfcnVuX2FsbG9jcSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgZGFzdHJhdGVneQogIDEwMC4wJSAgWzFdICAgICAgICAgIGdfZGlza19zdGFydAogICAx MDAuMCUgIFsxXSAgICAgICAgICAgZ19pb19yZXF1ZXN0CgowMC4zNCUgIFsxXSAgICAgICAgZGV2 ZnNfcmVhZF9mIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkb2Zp bGVyZWFkCiAgMTAwLjAlICBbMV0gICAgICAgICAga2Vybl9yZWFkdgogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgc3lzX3JlYWQKCjAwLjM0JSAgWzFdICAgICAgICBydW5xX2Nob29zZV9mcm9tIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9yZW0KICAxMDAu MCUgIFsxXSAgICAgICAgICB0ZHFfbW92ZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2NoZWRf aWRsZXRkCgowMC4zNCUgIFsxXSAgICAgICAgc2NzaV9yZWFkX3dyaXRlIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkYXN0YXJ0CiAgMTAwLjAlICBbMV0gICAgICAg ICAgeHB0X3J1bl9hbGxvY3EKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhc3RyYXRlZ3kKCjAw LjM0JSAgWzFdICAgICAgICBkYXN0cmF0ZWd5IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAl ICBbMV0gICAgICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2lvX3Jl cXVlc3QKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKCjAwLjM0JSAg WzFdICAgICAgICBsYXBpY19oYW5kbGVfdGltZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCgowMC4z NCUgIFsxXSAgICAgICAgdmZzX3RpbWVzdGFtcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgZGV2ZnNfd3JpdGVfZgogIDEwMC4wJSAgWzFdICAgICAgICAgIGRvZmls ZXdyaXRlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBrZXJuX3dyaXRldgoKMDAuMzQlICBbMV0g ICAgICAgIGdfaW9fZGVsaXZlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgMTAwLjAlICBbMV0gICAgICAgICAgZGFkb25lCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMC4zNCUgIFsxXSAgICAg ICAgbXR4X3Bvb2xfZmluZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAg ICAgYndhaXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBbMV0gICAg ICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQlICBbMV0gICAgICAgIGdfZGV2X3N0cmF0ZWd5IEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3 CiAgMTAwLjAlICBbMV0gICAgICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBk ZXZmc19yZWFkX2YKCjAwLjM0JSAgWzFdICAgICAgICBpbnRyX2Rpc2FibGVfc3JjIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X2hhbmRsZQogIDEw MC4wJSAgWzFdICAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgbGFwaWNfaGFuZGxlX2ludHIKCjAwLjM0JSAgWzFdICAgICAgICBjcml0aWNhbF9l bnRlciBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc2NoZWRfYWRk CiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVudF9zY2hlZHVsZV90aHJlYWQKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfaGFuZGxlCgowMC4zNCUgIFsxXSAgICAgICAg c2NoZWRfcmVtIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB0ZHFf bW92ZQogIDEwMC4wJSAgWzFdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFsxXSAg ICAgICAgICAgZm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgc2NoZWRfYWRkIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB0dXJuc3RpbGVfdW5wZW5kCiAgMTAw LjAlICBbMV0gICAgICAgICAgX19tdHhfdW5sb2NrX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICB4cHRfcnVuX2RldnEKCjAwLjM0JSAgWzFdICAgICAgICBsYXBpY19oYW5kbGVfaW50ciBA IC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBfc2xlZXAgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJ3YWl0CiAgMTAwLjAlICBbMV0gICAg ICAgICAgcGh5c2lvCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkZXZmc19yZWFkX2YKCjAwLjM0 JSAgWzFdICAgICAgICBzY2hlZF9wcmlvcml0eSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgc2NoZWRfYWRkCiAgMTAwLjAlICBbMV0gICAgICAgICAgc2V0cnVubmFi bGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNsZWVwcV9icm9hZGNhc3QKCjAwLjM0JSAgWzFd ICAgICAgICBkb2ZpbGV3cml0ZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAg ICAgICAga2Vybl93cml0ZXYKICAxMDAuMCUgIFsxXSAgICAgICAgICBzeXNfd3JpdGUKICAgMTAw LjAlICBbMV0gICAgICAgICAgIGFtZDY0X3N5c2NhbGwKCjAwLjM0JSAgWzFdICAgICAgICBtcnNh c19jb21wbGV0ZV9jbWQgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAg ICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEw MC4wJSAgWzFdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAuMCUgIFsxXSAgICAgICAgICAg Zm9ya19leGl0CgowMC4zNCUgIFsxXSAgICAgICAgaW50cl9ldmVudF9oYW5kbGUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwog IDEwMC4wJSAgWzFdICAgICAgICAgIGxhcGljX2hhbmRsZV9pbnRyCgowMC4zNCUgIFsxXSAgICAg ICAgcG1hcF9rZXh0cmFjdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAg ICAgZnJlZQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9yZWxlYXNlX2NjYgogICAxMDAuMCUg IFsxXSAgICAgICAgICAgZGFkb25lCgowMC4zNCUgIFsxXSAgICAgICAgZ19kZXZfZG9uZSBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19pb19kZWxpdmVyCiAgMTAw LjAlICBbMV0gICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBkYWRvbmUKCjAwLjM0JSAgWzFdICAgICAgICBiaW9xX3Rha2VmaXJzdCBAIC9ib290L2tl cm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGFzdGFydAogIDEwMC4wJSAgWzFdICAg ICAgICAgIHhwdF9ydW5fYWxsb2NxCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYXN0cmF0ZWd5 CgowMC4zNCUgIFsxXSAgICAgICAgYmdlX2FwZV9sb2NrIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMV0gICAgICAgICBiZ2VfbWlpYnVzX3JlYWRyZWcKICAxMDAuMCUgIFsxXSAgICAg ICAgICBicmdwaHlfc3RhdHVzCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBicmdwaHlfc2Vydmlj ZQoKMDAuMzQlICBbMV0gICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21y c2FzLmtvCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2FzX2J1aWxkX2RjZGIKICAxMDAuMCUgIFsx XSAgICAgICAgICBtcnNhc19hY3Rpb24KICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9ydW5f ZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBid2FpdCBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgcGh5c2lvCiAgMTAwLjAlICBb MV0gICAgICAgICAgZGV2ZnNfcmVhZF9mCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkb2ZpbGVy ZWFkCgowMC4zNCUgIFsxXSAgICAgICAgZGFkb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCiAgMTAwLjAlICBbMV0gICAgICAgICAg eHB0X2RvbmVfdGQKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGZvcmtfZXhpdAoKMDAuMzQlICBb MV0gICAgICAgIHhwdF9kb25lIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAg ICAgICBtcnNhc19jbWRfZG9uZSBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzFd ICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50 cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuMzQlICBb MV0gICAgICAgIGRldnN0YXRfc3RhcnRfdHJhbnNhY3Rpb25fYmlvIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMV0gICAgICAgICBnX2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsxXSAgICAg ICAgICBnX2lvX3JlcXVlc3QKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldl9zdHJhdGVneV9j c3cKCjAwLjM0JSAgWzFdICAgICAgICBpbnRyX2V4ZWN1dGVfaGFuZGxlcnMgQCAvYm9vdC9rZXJu ZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGxhcGljX2hhbmRsZV9pbnRyCgowMC4zNCUg IFsxXSAgICAgICAgdGRxX21vdmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAg ICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAgWzFdICAgICAgICAgIGZvcmtfZXhpdAoKMDAu MzQlICBbMV0gICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZSBAIC9ib290L2tlcm5lbC9rZXJuZWwK IDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9rZXJuZWwvbXJzYXMu a28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBb MV0gICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9r ZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBydW5xX3JlbW92ZSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgc2NoZWRfcmVtCiAgMTAwLjAlICBbMV0gICAgICAgICAg dGRxX21vdmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHNjaGVkX2lkbGV0ZAoKMDAuMzQlICBb MV0gICAgICAgIGdfaW9fcmVxdWVzdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgZGV2X3N0cmF0ZWd5X2NzdwogIDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVhZF9mCgpAIENQVV9DTEtfVU5IQUxURURf Q09SRSBbMjk0IHNhbXBsZXNdCgoxMC4yMCUgIFszMF0gICAgICAgY3B1X2lkbGUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFszMF0gICAgICAgIHNjaGVkX2lkbGV0ZAogIDEwMC4wJSAg WzMwXSAgICAgICAgIGZvcmtfZXhpdAoKMDkuMTglICBbMjddICAgICAgIHNtcF90bGJfc2hvb3Rk b3duIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMjddICAgICAgICBwbWFwX2ludmFs aWRhdGVfcmFuZ2UKICAxMDAuMCUgIFsyN10gICAgICAgICBiaW9kb25lCiAgIDEwMC4wJSAgWzI3 XSAgICAgICAgICBkYWRvbmUKCjA3LjQ4JSAgWzIyXSAgICAgICBpbnZscm5nX2hhbmRsZXIgQCAv Ym9vdC9rZXJuZWwva2VybmVsCgowNi40NiUgIFsxOV0gICAgICAgWGludmxybmcgQCAvYm9vdC9r ZXJuZWwva2VybmVsCgowNC43NiUgIFsxNF0gICAgICAgY3B1X3NlYXJjaF9sb3dlc3QgQCAvYm9v dC9rZXJuZWwva2VybmVsCiA4NS43MSUgIFsxMl0gICAgICAgIGNwdV9zZWFyY2hfbG93ZXN0CiAg ODMuMzMlICBbMTBdICAgICAgICAgY3B1X3NlYXJjaF9sb3dlc3QKICAgMTAwLjAlICBbMTBdICAg ICAgICAgIHNjaGVkX3BpY2tjcHUKICAxNi42NyUgIFsyXSAgICAgICAgICBzY2hlZF9waWNrY3B1 CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBzY2hlZF9hZGQKIDE0LjI5JSAgWzJdICAgICAgICAg c2NoZWRfcGlja2NwdQogIDEwMC4wJSAgWzJdICAgICAgICAgIHNjaGVkX2FkZAogICAxMDAuMCUg IFsyXSAgICAgICAgICAgc2V0cnVubmFibGUKCjA0LjQyJSAgWzEzXSAgICAgICBfX210eF9sb2Nr X3NsZWVwIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNzYuOTIlICBbMTBdICAgICAgICBfX210eF9s b2NrX2ZsYWdzCiAgMTAwLjAlICBbMTBdICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9r ZXJuZWwvbXJzYXMua28KICAgMTAwLjAlICBbMTBdICAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2Nt ZAogMTUuMzglICBbMl0gICAgICAgICB2bWVtX3hhbGxvYyBAIC9ib290L2tlcm5lbC9rZXJuZWwK ICAxMDAuMCUgIFsyXSAgICAgICAgICB2bWVtX2FsbG9jCiAgIDEwMC4wJSAgWzJdICAgICAgICAg ICBnX2lvX2NoZWNrCiAwNy42OSUgIFsxXSAgICAgICAgIHZtZW1feGZyZWUKICAxMDAuMCUgIFsx XSAgICAgICAgICBiaW9kb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBkYWRvbmUKCjAzLjc0 JSAgWzExXSAgICAgICBjcHVfc2VhcmNoX2hpZ2hlc3QgQCAvYm9vdC9rZXJuZWwva2VybmVsCiA3 Mi43MyUgIFs4XSAgICAgICAgIGNwdV9zZWFyY2hfaGlnaGVzdAogIDYyLjUwJSAgWzVdICAgICAg ICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUgIFs1XSAgICAgICAgICAgZm9ya19leGl0CiAgMzcu NTAlICBbM10gICAgICAgICAgY3B1X3NlYXJjaF9oaWdoZXN0CiAgIDEwMC4wJSAgWzNdICAgICAg ICAgICBzY2hlZF9pZGxldGQKIDI3LjI3JSAgWzNdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAw LjAlICBbM10gICAgICAgICAgZm9ya19leGl0CgowMy4wNiUgIFs5XSAgICAgICAgX19tdHhfbG9j a19mbGFncyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDMzLjMzJSAgWzNdICAgICAgICAgbXJzYXNf bWFwX3JlcXVlc3QgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFszXSAgICAgICAg ICBtcnNhc19idWlsZF9kY2RiCiAgIDEwMC4wJSAgWzNdICAgICAgICAgICBtcnNhc19hY3Rpb24K IDIyLjIyJSAgWzJdICAgICAgICAgbXJzYXNfY21kX2RvbmUKICAxMDAuMCUgIFsyXSAgICAgICAg ICBtcnNhc19jb21wbGV0ZV9jbWQKICAgMTAwLjAlICBbMl0gICAgICAgICAgIGludHJfZXZlbnRf ZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDIyLjIyJSAgWzJdICAgICAg ICAgbXJzYXNfdW5tYXBfcmVxdWVzdCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAg WzJdICAgICAgICAgIG1yc2FzX2NtZF9kb25lCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBtcnNh c19jb21wbGV0ZV9jbWQKIDExLjExJSAgWzFdICAgICAgICAgbXJzYXNfZ2V0X21wdF9jbWQKICAx MDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19hY3Rpb24KICAgMTAwLjAlICBbMV0gICAgICAgICAg IHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDExLjExJSAgWzFdICAgICAgICAg bXJzYXNfYWN0aW9uIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAg ICAgeHB0X3J1bl9kZXZxIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CgowMi43MiUgIFs4XSAgICAgICAgc2NoZWRfc3dpdGNo IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbOF0gICAgICAgICBtaV9zd2l0Y2gKICAz Ny41MCUgIFszXSAgICAgICAgICBpdGhyZWFkX2xvb3AKICAgMTAwLjAlICBbM10gICAgICAgICAg IGZvcmtfZXhpdAogIDI1LjAwJSAgWzJdICAgICAgICAgIHNjaGVkX2lkbGV0ZAogICAxMDAuMCUg IFsyXSAgICAgICAgICAgZm9ya19leGl0CiAgMTIuNTAlICBbMV0gICAgICAgICAgY3JpdGljYWxf ZXhpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5kbGUKICAxMi41MCUg IFsxXSAgICAgICAgICB0dXJuc3RpbGVfd2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgX19t dHhfbG9ja19zbGVlcAogIDEyLjUwJSAgWzFdICAgICAgICAgIHNsZWVwcV93YWl0CiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBfc2xlZXAKCjAyLjM4JSAgWzddICAgICAgICBzY2hlZF9pZGxldGQg QCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFs3XSAgICAgICAgIGZvcmtfZXhpdAoKMDIu MDQlICBbNl0gICAgICAgIGNwdV9zd2l0Y2ggQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUg IFs2XSAgICAgICAgIG1pX3N3aXRjaAogIDgzLjMzJSAgWzVdICAgICAgICAgIHNjaGVkX2lkbGV0 ZAogICAxMDAuMCUgIFs1XSAgICAgICAgICAgZm9ya19leGl0CiAgMTYuNjclICBbMV0gICAgICAg ICAgY3JpdGljYWxfZXhpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgaW50cl9ldmVudF9oYW5k bGUKCjAxLjcwJSAgWzVdICAgICAgICBfX3N5c19yZWFkIEAgL2xpYi9saWJjLnNvLjcKCjAxLjcw JSAgWzVdICAgICAgICBnZXRwYnVmIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbNV0g ICAgICAgICBwaHlzaW8KICAxMDAuMCUgIFs1XSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAw LjAlICBbNV0gICAgICAgICAgIGRvZmlsZXJlYWQKCjAxLjM2JSAgWzRdICAgICAgICB0aHJlYWRf bG9ja19mbGFnc18gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAyNS4wMCUgIFsxXSAgICAgICAgIGlu dHJfZXZlbnRfc2NoZWR1bGVfdGhyZWFkCiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVu dF9oYW5kbGUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXhlY3V0ZV9oYW5kbGVycwog MjUuMDAlICBbMV0gICAgICAgICBwcm9wYWdhdGVfcHJpb3JpdHkKICAxMDAuMCUgIFsxXSAgICAg ICAgICB0dXJuc3RpbGVfd2FpdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgX19tdHhfbG9ja19z bGVlcAogMjUuMDAlICBbMV0gICAgICAgICBzbGVlcHFfYnJvYWRjYXN0CiAgMTAwLjAlICBbMV0g ICAgICAgICAgd2FrZXVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBiZG9uZQogMjUuMDAlICBb MV0gICAgICAgICBpdGhyZWFkX2xvb3AKICAxMDAuMCUgIFsxXSAgICAgICAgICBmb3JrX2V4aXQK CjAxLjM2JSAgWzRdICAgICAgICBfbXR4X2xvY2tfc3Bpbl9jb29raWUgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiA3NS4wMCUgIFszXSAgICAgICAgIHNtcF90bGJfc2hvb3Rkb3duCiAgMTAwLjAlICBb M10gICAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAgIDEwMC4wJSAgWzNdICAgICAgICAg ICBiaW9kb25lCiAyNS4wMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFdICAg ICAgICAgIHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzbGVlcHFfYnJvYWRj YXN0CgowMS4zNiUgIFs0XSAgICAgICAgX19tdHhfdW5sb2NrX2ZsYWdzIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMjUuMDAlICBbMV0gICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQgQCAvYm9vdC9r ZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVf aGFuZGxlcnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBp dGhyZWFkX2xvb3AKIDI1LjAwJSAgWzFdICAgICAgICAgbXJzYXNfY21kX2RvbmUgQCAvYm9vdC9r ZXJuZWwvbXJzYXMua28KICAxMDAuMCUgIFsxXSAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQK ICAgMTAwLjAlICBbMV0gICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDI1LjAwJSAgWzFdICAgICAgICAgeHB0X3J1bl9kZXZxCiAgMTAw LjAlICBbMV0gICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBkYXN0YXJ0CiAyNS4wMCUgIFsxXSAgICAgICAgIG1yc2FzX2dldF9tcHRfY21kIEAgL2Jv b3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgbXJzYXNfYWN0aW9uCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICB4cHRfcnVuX2RldnEgQCAvYm9vdC9rZXJuZWwva2VybmVs CgowMS4zNiUgIFs0XSAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlIEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbNF0gICAgICAgICBiaW9kb25lCiAgMTAwLjAlICBbNF0gICAgICAg ICAgZGFkb25lCiAgIDEwMC4wJSAgWzRdICAgICAgICAgICB4cHRfZG9uZV9wcm9jZXNzCgowMS4z NiUgIFs0XSAgICAgICAgc3BpbmxvY2tfZXhpdCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAw JSAgWzJdICAgICAgICAgc2NoZWRfaWRsZXRkCiAgMTAwLjAlICBbMl0gICAgICAgICAgZm9ya19l eGl0CiAyNS4wMCUgIFsxXSAgICAgICAgIHRkcV9tb3ZlCiAgMTAwLjAlICBbMV0gICAgICAgICAg c2NoZWRfaWRsZXRkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQKIDI1LjAwJSAg WzFdICAgICAgICAgd2FrZXVwCiAgMTAwLjAlICBbMV0gICAgICAgICAgYmRvbmUKICAgMTAwLjAl ICBbMV0gICAgICAgICAgIGJ1ZmRvbmUKCjAxLjAyJSAgWzNdICAgICAgICB4cHRfYWN0aW9uX2Rl ZmF1bHQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIGRhc3RhcnQK ICAxMDAuMCUgIFszXSAgICAgICAgICB4cHRfcnVuX2FsbG9jcQogICAxMDAuMCUgIFszXSAgICAg ICAgICAgZGFzdHJhdGVneQoKMDEuMDIlICBbM10gICAgICAgIF9mZ2V0IEAgL2Jvb3Qva2VybmVs L2tlcm5lbAogMTAwLjAlICBbM10gICAgICAgICBrZXJuX3dyaXRldgogIDEwMC4wJSAgWzNdICAg ICAgICAgIHN5c193cml0ZQogICAxMDAuMCUgIFszXSAgICAgICAgICAgYW1kNjRfc3lzY2FsbAoK MDEuMDIlICBbM10gICAgICAgIHZtZW1feGFsbG9jIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbM10gICAgICAgICB2bWVtX2FsbG9jCiAgMTAwLjAlICBbM10gICAgICAgICAgZ19pb19j aGVjawogICAxMDAuMCUgIFszXSAgICAgICAgICAgZ19pb19yZXF1ZXN0CgowMS4wMiUgIFszXSAg ICAgICAgZ19kZXZfc3RyYXRlZ3kgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAg ICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFszXSAgICAgICAgICBwaHlzaW8KICAg MTAwLjAlICBbM10gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDEuMDIlICBbM10gICAgICAgIHZt ZW1feGZyZWUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFszXSAgICAgICAgIGJpb2Rv bmUKICAxMDAuMCUgIFszXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbM10gICAgICAgICAg IHhwdF9kb25lX3Byb2Nlc3MKCjAwLjY4JSAgWzJdICAgICAgICBzY2hlZF9jaG9vc2UgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIGNob29zZXRocmVhZAogIDEwMC4w JSAgWzJdICAgICAgICAgIHNjaGVkX3N3aXRjaAogICAxMDAuMCUgIFsyXSAgICAgICAgICAgbWlf c3dpdGNoCgowMC42OCUgIFsyXSAgICAgICAgaXRocmVhZF9sb29wIEAgL2Jvb3Qva2VybmVsL2tl cm5lbAogMTAwLjAlICBbMl0gICAgICAgICBmb3JrX2V4aXQKCjAwLjY4JSAgWzJdICAgICAgICBt cnNhc19nZXRfbXB0X2NtZCBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbMl0gICAg ICAgICBtcnNhc19hY3Rpb24KICAxMDAuMCUgIFsyXSAgICAgICAgICB4cHRfcnVuX2RldnEgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICB4cHRfYWN0aW9uX2Rl ZmF1bHQKCjAwLjY4JSAgWzJdICAgICAgICBkZXZzdGF0X2VuZF90cmFuc2FjdGlvbiBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZGV2c3RhdF9lbmRfdHJhbnNhY3Rp b25fYmlvX2J0CiAgMTAwLjAlICBbMl0gICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCiAgIDEw MC4wJSAgWzJdICAgICAgICAgICBkYWRvbmUKCjAwLjY4JSAgWzJdICAgICAgICB4cHRfcnVuX2Fs bG9jcSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgZGFzdHJhdGVn eQogIDEwMC4wJSAgWzJdICAgICAgICAgIGdfZGlza19zdGFydAogICAxMDAuMCUgIFsyXSAgICAg ICAgICAgZ19pb19yZXF1ZXN0CgowMC42OCUgIFsyXSAgICAgICAgWGFwaWNfaXNyMSBAIC9ib290 L2tlcm5lbC9rZXJuZWwKCjAwLjY4JSAgWzJdICAgICAgICBnX2lvX2NoZWNrIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2lvX3JlcXVlc3QKICAxMDAuMCUgIFsy XSAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBwaHlz aW8KCjAwLjY4JSAgWzJdICAgICAgICBtcnNhc19pc3IgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28K IDEwMC4wJSAgWzJdICAgICAgICAgaW50cl9ldmVudF9leGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzJdICAgICAgICAgIGl0aHJlYWRfbG9vcAogICAxMDAu MCUgIFsyXSAgICAgICAgICAgZm9ya19leGl0CgowMC42OCUgIFsyXSAgICAgICAgeHB0X2RvbmVf cHJvY2VzcyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzJdICAgICAgICAgeHB0X2Rv bmVfdGQKICAxMDAuMCUgIFsyXSAgICAgICAgICBmb3JrX2V4aXQKCjAwLjY4JSAgWzJdICAgICAg ICBnX2lvX2RlbGl2ZXIgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAg IGdfZGlza19kb25lX3NpbmdsZQogIDEwMC4wJSAgWzJdICAgICAgICAgIGRhZG9uZQogICAxMDAu MCUgIFsyXSAgICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwoKMDAuNjglICBbMl0gICAgICAgIGNy aXRpY2FsX2VudGVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogNTAuMDAlICBbMV0gICAgICAgICB1 bWFfemZyZWVfYXJnCiAgMTAwLjAlICBbMV0gICAgICAgICAgZ19kZXZfZG9uZQogICAxMDAuMCUg IFsxXSAgICAgICAgICAgZ19pb19kZWxpdmVyCiA1MC4wMCUgIFsxXSAgICAgICAgIHNtcF90bGJf c2hvb3Rkb3duCiAgMTAwLjAlICBbMV0gICAgICAgICAgcG1hcF9pbnZhbGlkYXRlX3JhbmdlCiAg IDEwMC4wJSAgWzFdICAgICAgICAgICBiaW9kb25lCgowMC42OCUgIFsyXSAgICAgICAgZ19kaXNr X3N0YXJ0IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBnX2lvX3Jl cXVlc3QKICAxMDAuMCUgIFsyXSAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4wJSAg WzJdICAgICAgICAgICBwaHlzaW8KCjAwLjY4JSAgWzJdICAgICAgICBmcmVlIEAgL2Jvb3Qva2Vy bmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICB4cHRfcmVsZWFzZV9jY2IKICAxMDAuMCUg IFsyXSAgICAgICAgICBkYWRvbmUKICAgMTAwLjAlICBbMl0gICAgICAgICAgIHhwdF9kb25lX3By b2Nlc3MKCjAwLjY4JSAgWzJdICAgICAgICBjYWxsb3V0X2xvY2sgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIF9jYWxsb3V0X3N0b3Bfc2FmZQogIDEwMC4wJSAgWzJd ICAgICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4w JSAgWzJdICAgICAgICAgICBtcnNhc19jb21wbGV0ZV9jbWQKCjAwLjY4JSAgWzJdICAgICAgICBk ZXZfcmVsdGhyZWFkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMl0gICAgICAgICBk ZXZmc19yZWFkX2YKICAxMDAuMCUgIFsyXSAgICAgICAgICBkb2ZpbGVyZWFkCiAgIDEwMC4wJSAg WzJdICAgICAgICAgICBrZXJuX3JlYWR2CgowMC42OCUgIFsyXSAgICAgICAgY3JpdGljYWxfZXhp dCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDUwLjAwJSAgWzFdICAgICAgICAgc3BpbmxvY2tfZXhp dAogIDEwMC4wJSAgWzFdICAgICAgICAgIHNsZWVwcV9zd2l0Y2gKICAgMTAwLjAlICBbMV0gICAg ICAgICAgIHNsZWVwcV93YWl0CiA1MC4wMCUgIFsxXSAgICAgICAgIGZyZWUKICAxMDAuMCUgIFsx XSAgICAgICAgICB4cHRfcmVsZWFzZV9jY2IKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRhZG9u ZQoKMDAuNjglICBbMl0gICAgICAgIGRhZG9uZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzJdICAgICAgICAgeHB0X2RvbmVfcHJvY2VzcwogIDEwMC4wJSAgWzJdICAgICAgICAgIHhw dF9kb25lX3RkCiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjY4JSAgWzJd ICAgICAgICBwaHlzaW8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAg IGRldmZzX3JlYWRfZgogIDEwMC4wJSAgWzJdICAgICAgICAgIGRvZmlsZXJlYWQKICAgMTAwLjAl ICBbMl0gICAgICAgICAgIGtlcm5fcmVhZHYKCjAwLjY4JSAgWzJdICAgICAgICBhbWQ2NF9zeXNj YWxsIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuNjglICBbMl0gICAgICAgIG1yc2FzX2FjdGlv biBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogMTAwLjAlICBbMl0gICAgICAgICB4cHRfcnVuX2Rl dnEgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMl0gICAgICAgICAgeHB0X2FjdGlv bl9kZWZhdWx0CiAgIDEwMC4wJSAgWzJdICAgICAgICAgICBkYXN0YXJ0CgowMC42OCUgIFsyXSAg ICAgICAgYnplcm8gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsyXSAgICAgICAgIG1y c2FzX2FjdGlvbiBAIC9ib290L2tlcm5lbC9tcnNhcy5rbwogIDEwMC4wJSAgWzJdICAgICAgICAg IHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJuZWwKICAgMTAwLjAlICBbMl0gICAgICAg ICAgIHhwdF9hY3Rpb25fZGVmYXVsdAoKMDAuMzQlICBbMV0gICAgICAgIHNldHJ1bm5hYmxlIEAg L2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBzbGVlcHFfYnJvYWRjYXN0 CiAgMTAwLjAlICBbMV0gICAgICAgICAgd2FrZXVwCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBi ZG9uZQoKMDAuMzQlICBbMV0gICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZCBAIC9ib290L2tlcm5l bC9tcnNhcy5rbwogMTAwLjAlICBbMV0gICAgICAgICBpbnRyX2V2ZW50X2V4ZWN1dGVfaGFuZGxl cnMgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAgMTAwLjAlICBbMV0gICAgICAgICAgaXRocmVhZF9s b29wCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBmb3JrX2V4aXQKCjAwLjM0JSAgWzFdICAgICAg ICBmZ2V0X3VubG9ja2VkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAg ICBfZmdldAogIDEwMC4wJSAgWzFdICAgICAgICAgIGtlcm5fd3JpdGV2CiAgIDEwMC4wJSAgWzFd ICAgICAgICAgICBzeXNfd3JpdGUKCjAwLjM0JSAgWzFdICAgICAgICBkYXN0YXJ0IEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEwMC4w JSAgWzFdICAgICAgICAgIGRhc3RyYXRlZ3kKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfZGlz a19zdGFydAoKMDAuMzQlICBbMV0gICAgICAgIG1yc2FzX2NtZF9kb25lIEAgL2Jvb3Qva2VybmVs L21yc2FzLmtvCiAxMDAuMCUgIFsxXSAgICAgICAgIG1yc2FzX2NvbXBsZXRlX2NtZAogIDEwMC4w JSAgWzFdICAgICAgICAgIGludHJfZXZlbnRfZXhlY3V0ZV9oYW5kbGVycyBAIC9ib290L2tlcm5l bC9rZXJuZWwKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGl0aHJlYWRfbG9vcAoKMDAuMzQlICBb MV0gICAgICAgIGJpb2RvbmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAg ICAgIGRhZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAgMTAw LjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3RkCgowMC4zNCUgIFsxXSAgICAgICAgWHhlbl9p bnRyX3VwY2FsbCBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAgICBtcnNh c19kYXRhX2xvYWRfY2IgQCAvYm9vdC9rZXJuZWwvbXJzYXMua28KIDEwMC4wJSAgWzFdICAgICAg ICAgYnVzX2RtYW1hcF9sb2FkIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogIDEwMC4wJSAgWzFdICAg ICAgICAgIG1yc2FzX21hcF9yZXF1ZXN0IEAgL2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBtcnNhc19idWlsZF9kY2RiCgowMC4zNCUgIFsxXSAgICAgICAgcnVu cV9yZW1vdmUgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVk X2Nob29zZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGNob29zZXRocmVhZAogICAxMDAuMCUgIFsx XSAgICAgICAgICAgc2NoZWRfc3dpdGNoCgowMC4zNCUgIFsxXSAgICAgICAgbGFwaWNfaGFuZGxl X3RpbWVyIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAoKMDAuMzQlICBbMV0gICAgICAgIGRldnZuX3Jl ZnRocmVhZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2ZnNf cmVhZF9mCiAgMTAwLjAlICBbMV0gICAgICAgICAgZG9maWxlcmVhZAogICAxMDAuMCUgIFsxXSAg ICAgICAgICAga2Vybl9yZWFkdgoKMDAuMzQlICBbMV0gICAgICAgIGdfZGV2X2RvbmUgQCAvYm9v dC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGdfaW9fZGVsaXZlcgogIDEwMC4w JSAgWzFdICAgICAgICAgIGdfZGlza19kb25lX3NpbmdsZQogICAxMDAuMCUgIFsxXSAgICAgICAg ICAgZGFkb25lCgowMC4zNCUgIFsxXSAgICAgICAgcnVucV9hZGQgQCAvYm9vdC9rZXJuZWwva2Vy bmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHNjaGVkX2FkZAogIDEwMC4wJSAgWzFdICAgICAgICAg IHNldHJ1bm5hYmxlCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzbGVlcHFfYnJvYWRjYXN0Cgow MC4zNCUgIFsxXSAgICAgICAgdW1hX3phbGxvY19hcmcgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKICAxMDAuMCUgIFsxXSAgICAgICAg ICBwaHlzaW8KICAgMTAwLjAlICBbMV0gICAgICAgICAgIGRldmZzX3JlYWRfZgoKMDAuMzQlICBb MV0gICAgICAgIGRvcmV0aSBAIC9ib290L2tlcm5lbC9rZXJuZWwKCjAwLjM0JSAgWzFdICAgICAg ICBkYXN0cmF0ZWd5IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBn X2Rpc2tfc3RhcnQKICAxMDAuMCUgIFsxXSAgICAgICAgICBnX2lvX3JlcXVlc3QKICAgMTAwLjAl ICBbMV0gICAgICAgICAgIGRldl9zdHJhdGVneV9jc3cKCjAwLjM0JSAgWzFdICAgICAgICBnX2Ns b25lX2JpbyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19kZXZf c3RyYXRlZ3kKICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZfc3RyYXRlZ3lfY3N3CiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBwbWFwX3FlbnRlciBA IC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZ19pb19jaGVjawogIDEw MC4wJSAgWzFdICAgICAgICAgIGdfaW9fcmVxdWVzdAogICAxMDAuMCUgIFsxXSAgICAgICAgICAg ZGV2X3N0cmF0ZWd5X2NzdwoKMDAuMzQlICBbMV0gICAgICAgIHNwaW5sb2NrX2VudGVyIEAgL2Jv b3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB0aHJlYWRfbG9ja19mbGFnc18K ICAxMDAuMCUgIFsxXSAgICAgICAgICBzbGVlcHFfYWRkCiAgIDEwMC4wJSAgWzFdICAgICAgICAg ICBfc2xlZXAKCjAwLjM0JSAgWzFdICAgICAgICBnX2Rpc2tfZG9uZV9zaW5nbGUgQCAvYm9vdC9r ZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRhZG9uZQogIDEwMC4wJSAgWzFdICAg ICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9kb25l X3RkCgowMC4zNCUgIFsxXSAgICAgICAgc2xlZXBxX2xvY2sgQCAvYm9vdC9rZXJuZWwva2VybmVs CiAxMDAuMCUgIFsxXSAgICAgICAgIF9zbGVlcAogIDEwMC4wJSAgWzFdICAgICAgICAgIGJ3YWl0 CiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBkZXZm c19yZWFkX2YgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGRvZmls ZXJlYWQKICAxMDAuMCUgIFsxXSAgICAgICAgICBrZXJuX3JlYWR2CiAgIDEwMC4wJSAgWzFdICAg ICAgICAgICBzeXNfcmVhZAoKMDAuMzQlICBbMV0gICAgICAgIHNsZWVwcV9icm9hZGNhc3QgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHdha2V1cAogIDEwMC4wJSAg WzFdICAgICAgICAgIGJkb25lCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBidWZkb25lCgowMC4z NCUgIFsxXSAgICAgICAgdW1hX3pmcmVlX2FyZyBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4w JSAgWzFdICAgICAgICAgZ19kZXZfZG9uZQogIDEwMC4wJSAgWzFdICAgICAgICAgIGdfaW9fZGVs aXZlcgogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZ19kaXNrX2RvbmVfc2luZ2xlCgowMC4zNCUg IFsxXSAgICAgICAgc2xlZXBxX2FkZCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFd ICAgICAgICAgX3NsZWVwCiAgMTAwLjAlICBbMV0gICAgICAgICAgYndhaXQKICAgMTAwLjAlICBb MV0gICAgICAgICAgIHBoeXNpbwoKMDAuMzQlICBbMV0gICAgICAgIGdfaW9fcmVxdWVzdCBAIC9i b290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgZGV2X3N0cmF0ZWd5X2Nzdwog IDEwMC4wJSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2 ZnNfcmVhZF9mCgowMC4zNCUgIFsxXSAgICAgICAgYnRfZnJlZXRyaW0gQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJpb2RvbmUKICAxMDAuMCUgIFsxXSAgICAgICAg ICBkYWRvbmUKICAgMTAwLjAlICBbMV0gICAgICAgICAgIHhwdF9kb25lX3Byb2Nlc3MKCjAwLjM0 JSAgWzFdICAgICAgICBydW5xX2Nob29zZSBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAg WzFdICAgICAgICAgc2NoZWRfY2hvb3NlCiAgMTAwLjAlICBbMV0gICAgICAgICAgY2hvb3NldGhy ZWFkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICBzY2hlZF9zd2l0Y2gKCjAwLjM0JSAgWzFdICAg ICAgICByZWxwYnVmIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBw aHlzaW8KICAxMDAuMCUgIFsxXSAgICAgICAgICBkZXZmc19yZWFkX2YKICAgMTAwLjAlICBbMV0g ICAgICAgICAgIGRvZmlsZXJlYWQKCjAwLjM0JSAgWzFdICAgICAgICBwbWFwX2V4dHJhY3RfYW5k X2hvbGQgQCAvYm9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHZtX2ZhdWx0 X3F1aWNrX2hvbGRfcGFnZXMKICAxMDAuMCUgIFsxXSAgICAgICAgICB2bWFwYnVmCiAgIDEwMC4w JSAgWzFdICAgICAgICAgICBwaHlzaW8KCjAwLjM0JSAgWzFdICAgICAgICBjcHVfc2V0X3N5c2Nh bGxfcmV0dmFsIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICBhbWQ2 NF9zeXNjYWxsCgowMC4zNCUgIFsxXSAgICAgICAgYXRvbWljX2ZldGNoYWRkX2ludCBAIC9ib290 L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgbXJzYXNfY29tcGxldGVfY21kIEAg L2Jvb3Qva2VybmVsL21yc2FzLmtvCiAgMTAwLjAlICBbMV0gICAgICAgICAgaW50cl9ldmVudF9l eGVjdXRlX2hhbmRsZXJzIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogICAxMDAuMCUgIFsxXSAgICAg ICAgICAgaXRocmVhZF9sb29wCgowMC4zNCUgIFsxXSAgICAgICAgc2NoZWRfYWRkIEAgL2Jvb3Qv a2VybmVsL2tlcm5lbAogMTAwLjAlICBbMV0gICAgICAgICB0dXJuc3RpbGVfdW5wZW5kCiAgMTAw LjAlICBbMV0gICAgICAgICAgX19tdHhfdW5sb2NrX3NsZWVwCiAgIDEwMC4wJSAgWzFdICAgICAg ICAgICBidWZkb25lCgowMC4zNCUgIFsxXSAgICAgICAgdm1fcGFnZV91bmhvbGRfcGFnZXMgQCAv Ym9vdC9rZXJuZWwva2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIHZ1bm1hcGJ1ZgogIDEwMC4w JSAgWzFdICAgICAgICAgIHBoeXNpbwogICAxMDAuMCUgIFsxXSAgICAgICAgICAgZGV2ZnNfcmVh ZF9mCgowMC4zNCUgIFsxXSAgICAgICAgbWFsbG9jIEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAw LjAlICBbMV0gICAgICAgICB4cHRfcnVuX2FsbG9jcQogIDEwMC4wJSAgWzFdICAgICAgICAgIGRh c3RyYXRlZ3kKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfZGlza19zdGFydAoKMDAuMzQlICBb MV0gICAgICAgIHNjaGVkX3ByaW9yaXR5IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAogMTAwLjAlICBb MV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBzZXRydW5uYWJsZQog ICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xlZXBxX2Jyb2FkY2FzdAoKMDAuMzQlICBbMV0gICAg ICAgIF9zbGVlcCBAIC9ib290L2tlcm5lbC9rZXJuZWwKIDEwMC4wJSAgWzFdICAgICAgICAgYndh aXQKICAxMDAuMCUgIFsxXSAgICAgICAgICBwaHlzaW8KICAgMTAwLjAlICBbMV0gICAgICAgICAg IGRldmZzX3JlYWRfZgoKMDAuMzQlICBbMV0gICAgICAgIGJ1ZmRvbmUgQCAvYm9vdC9rZXJuZWwv a2VybmVsCiAxMDAuMCUgIFsxXSAgICAgICAgIGJ1ZmRvbmViaW8KICAxMDAuMCUgIFsxXSAgICAg ICAgICBnX2lvX2RlbGl2ZXIKICAgMTAwLjAlICBbMV0gICAgICAgICAgIGdfZGlza19kb25lX3Np bmdsZQoKMDAuMzQlICBbMV0gICAgICAgIHhwdF9ydW5fZGV2cSBAIC9ib290L2tlcm5lbC9rZXJu ZWwKIDEwMC4wJSAgWzFdICAgICAgICAgeHB0X2FjdGlvbl9kZWZhdWx0CiAgMTAwLjAlICBbMV0g ICAgICAgICAgZGFzdGFydAogICAxMDAuMCUgIFsxXSAgICAgICAgICAgeHB0X3J1bl9hbGxvY3EK CjAwLjM0JSAgWzFdICAgICAgICBzY2hlZF9waWNrY3B1IEAgL2Jvb3Qva2VybmVsL2tlcm5lbAog MTAwLjAlICBbMV0gICAgICAgICBzY2hlZF9hZGQKICAxMDAuMCUgIFsxXSAgICAgICAgICBzZXRy dW5uYWJsZQogICAxMDAuMCUgIFsxXSAgICAgICAgICAgc2xlZXBxX2Jyb2FkY2FzdAoKMDAuMzQl ICBbMV0gICAgICAgIF9tdHhfdHJ5bG9ja19mbGFnc18gQCAvYm9vdC9rZXJuZWwva2VybmVsCiAx MDAuMCUgIFsxXSAgICAgICAgIHZtX3BhZ2VfcGFfdHJ5cmVsb2NrCiAgMTAwLjAlICBbMV0gICAg ICAgICAgcG1hcF9leHRyYWN0X2FuZF9ob2xkCiAgIDEwMC4wJSAgWzFdICAgICAgICAgICB2bV9m YXVsdF9xdWlja19ob2xkX3BhZ2VzCgo= --14dae93d8c8631ddcf04fdd6ce99-- From owner-freebsd-scsi@FreeBSD.ORG Thu Jul 10 23:15:01 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D60A7B0F for ; Thu, 10 Jul 2014 23:15:01 +0000 (UTC) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CDD320CF for ; Thu, 10 Jul 2014 23:15:01 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id l18so258909wgh.1 for ; Thu, 10 Jul 2014 16:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=Cx60mGFHeUlj/3bCGTKQPEplI3XGSuTRiYAJYt+N7ss=; b=TcpdX2/JjV6ikqrWrn4afOcQlZPX4R2hUtIey9g+IKbpTOemekJf29vsu+E1tBYcHJ BCtEtiEGHwecgOQXO4w3ho0I1nzB/FNnNpTBrzvkVTEadvbf5oNGqdiMxzu+/1Gqoule Z91G1tF+sTCzx6v6aykhKrZigRGK3jqLPf4y6FYBQ/K4zE4bahPF4XsXOfQ/tDXW7NcA W3Vf7gXKZKqI8YlfU3xFOk5omrFmPvV90xRLwtALzTdY+jdX3nFU6bYBCLXOoFMghG5w x4CSx8oigEHLzaV5kt+4kUSKkD/RufV+gzqXquM7yEPH6kIQgtFBWoKLHgMGeCqD6hs3 zSgQ== X-Received: by 10.180.83.225 with SMTP id t1mr85095wiy.28.1405034096597; Thu, 10 Jul 2014 16:14:56 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id 20sm1179534wjt.42.2014.07.10.16.14.55 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 10 Jul 2014 16:14:55 -0700 (PDT) Sender: Alexander Motin Message-ID: <53BF1E6C.5030806@FreeBSD.org> Date: Fri, 11 Jul 2014 02:14:52 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Kashyap Desai Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO References: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> <53BE8784.8060503@FreeBSD.org> <9f138f242e278476e5c542d695e58bc8@mail.gmail.com> In-Reply-To: <9f138f242e278476e5c542d695e58bc8@mail.gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 23:15:01 -0000 On 10.07.2014 16:28, Kashyap Desai wrote: > From: Alexander Motin [mailto:mavbsd@gmail.com] On Behalf Of Alexander >> On 10.07.2014 15:00, Kashyap Desai wrote: >>> I have 8 SSDs in my setup and all 8 SSDs are behind LSI’s 12Gp/s >>> MegaRaid Controller as JBOD. I also found FIO can be used in Async >>> mode after loading “aio” kernel module. >>> >>> Using single SSD, I am able to see 110K-130K IOPs. This IOPs counts >>> are matching with what I see on Linux machine. >>> >>> Now, I am not able to scale IOPs on my machine after 200K. I see CPU >>> is almost occupied and no idle time after IOPs reach to 200K. >>> >>> If you have any pointers to try with, I can do some experiment on my >> setup. >> >> Getting such results I would immediately start doing profiling with >> pmcstat. >> Quite likely you are hitting some new lock congestion. Start with simple >> `pmcstat -n 100000000 -TS unhalted-cycles`. It it hard to say for sure >> what >> went wrong there without more data, so just couple > I have attached profile output for the command mentioned above. I will dig > further and see if this is what we have theoretical limit for CAM attached > HBA. First thing I noticed in this profile output is bunch of TLB shutdowns. You can not reach reasonable performance from user-level without having HBA support unmapped I/O. Both mps and mpr drivers support it, but for some reason still not mrsas. Even at non-peak I/O rates on multi-core system TLB shutdowns in such case can eat additional 30% of CPU time. Another thing I see is mentioned congestion on driver's CAM SIM lock. You need either multiple cards or multiqueue. >> thoughts: >> >> First of all, I've never tried aio in my benchmarks, only synchronous >> ones. Try >> to run 8 instances of `dd if=/dev/daX of=/dev/null bs=512` per each SSD >> same time, just as I did. You may vary number of dd's, but keep total >> below >> 256, or you mad to increase nswbuf limit in kern_vfs_bio_buffer_alloc(). > > I ran multiple dd instance also and seeing IOPs throttle somewhere ~200K . > > Do we have any mechanism to check CAM layer's max IOPs support without > involving actual Device ? Something like _null_ device driver which just > send the command back to CAM layer ? There is not such one now. Such test would radically change timings of operation, and I am not sure how useful would results be. >> For second, you are using single HBA, that should create significant >> congestion around its CAM SIM lock. Proper solution would be to add >> multiple queues support to the driver, and we discussed it with Scott Long >> for quite some time, but that requires more work (I hope you may be >> interested in it ;) ). Or you may just insert 3-4 HBAs. My million IOPS I >> was >> reaching with four 2008/2308 6Gbps HBAs and 16 SATA SSDs. > > I remember this part and really good to contribute for this work. As part > of this we have initiated multiple MSIx implementation in , which > will have multiple reply queue per MSI-x. Cool! > Do we really require to have multiple Submission queue at low level driver ? > I thought it will be a CAM interface for multi queue which _all_ low level > drivers need to hook into . Now CAM is still oriented on single submission queue, but allows driver to have multiple completion queues. So I would start from implementing last ones, each bound to own MSI-X interrupt and calling completion without using the SIM lock or holding any other locks during the upcall. CAM provides way to avoid extra context switch in that case, that could be very useful. -- Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Mon Jul 14 08:36:26 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A50C9A3 for ; Mon, 14 Jul 2014 08:36:26 +0000 (UTC) Received: from exprod7og103.obsmtp.com (exprod7og103.obsmtp.com [64.18.2.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 651A72DB0 for ; Mon, 14 Jul 2014 08:36:24 +0000 (UTC) Received: from mail-la0-f44.google.com ([209.85.215.44]) (using TLSv1) by exprod7ob103.postini.com ([64.18.6.12]) with SMTP ID DSNKU8OWgpC1P3c6qAUkqahrz1R/W/t6fEJ6@postini.com; Mon, 14 Jul 2014 01:36:25 PDT Received: by mail-la0-f44.google.com with SMTP id e16so1096720lan.31 for ; Mon, 14 Jul 2014 01:36:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=9VJ6w0w1EJYHrISfERsrht5D+mdZslXolChnYESI7BE=; b=cXYkFaKfajzoi+1Mr1TQ3CGHwcQNhA9iP4qMwfpxbPsgV7aA/lP5AkM3vzitvRURYr rVTxKBtJVnca1JFX1Kh8/xGHm3aGz2ZgX36SZCzmEQdElQOkSIBkUWmgYMmMfdng5jFk ofHGZLJx4PaAGgxtEKdsYCaM0w7VA9SrVWNjmrdjZ1ZjEB6kMF99EHUeAMT8QpIk63yy O22B866ypec2AY0IeNuoCTPK2d3Ea/KV1NuLe7yYe7N/rTNCjaIj/VWhIbj568V7cTzm hqurdhqeS/67U5KRYVwfPs5X/hNQAxAXsWecTWgcB2VSMzZza9d6sK5CpdCH1yw9Ocrz Xo/g== X-Gm-Message-State: ALoCoQlcfrWNBCJGdjzA3SVB0wYhuXIlqJ9VdkIk87yyaF2qurUt3A32t1twNDlETJK0alykjZxtPSeDDiJ9tDJGOcU8ebFcN5DhI4mBlAhbFtfBEQl7wE+8Ev9H25DEy6B56OmIzU/ncgjaLMJ2StJWrhBwJ4/FJw== X-Received: by 10.152.36.169 with SMTP id r9mr13421735laj.14.1405326976753; Mon, 14 Jul 2014 01:36:16 -0700 (PDT) X-Received: by 10.152.36.169 with SMTP id r9mr13421727laj.14.1405326976654; Mon, 14 Jul 2014 01:36:16 -0700 (PDT) From: Kashyap Desai References: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> <53BE8784.8060503@FreeBSD.org> <9f138f242e278476e5c542d695e58bc8@mail.gmail.com> <53BF1E6C.5030806@FreeBSD.org> In-Reply-To: <53BF1E6C.5030806@FreeBSD.org> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQG8AScvFWC5/cNVmZuoGGC8NZGkUAIYsfkbAp/trHgBd5fqmJuU+KQA Date: Mon, 14 Jul 2014 14:06:15 +0530 Message-ID: Subject: RE: SSDs peformance on head/freebsd-10 stable using FIO To: Alexander Motin Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jul 2014 08:36:26 -0000 > -----Original Message----- > From: Alexander Motin [mailto:mavbsd@gmail.com] On Behalf Of Alexander > Motin > Sent: Friday, July 11, 2014 4:45 AM > To: Kashyap Desai > Cc: FreeBSD-scsi > Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO > > On 10.07.2014 16:28, Kashyap Desai wrote: > > From: Alexander Motin [mailto:mavbsd@gmail.com] On Behalf Of > Alexander > >> On 10.07.2014 15:00, Kashyap Desai wrote: > >>> I have 8 SSDs in my setup and all 8 SSDs are behind LSI=E2=80=99s 12G= p/s > >>> MegaRaid Controller as JBOD. I also found FIO can be used in Async > >>> mode after loading =E2=80=9Caio=E2=80=9D kernel module. > >>> > >>> Using single SSD, I am able to see 110K-130K IOPs. This IOPs > >>> counts are matching with what I see on Linux machine. > >>> > >>> Now, I am not able to scale IOPs on my machine after 200K. I see > >>> CPU is almost occupied and no idle time after IOPs reach to 200K. > >>> > >>> If you have any pointers to try with, I can do some experiment on > >>> my > >> setup. > >> > >> Getting such results I would immediately start doing profiling with > >> pmcstat. > >> Quite likely you are hitting some new lock congestion. Start with > >> simple `pmcstat -n 100000000 -TS unhalted-cycles`. It it hard to say > >> for sure what went wrong there without more data, so just couple > > I have attached profile output for the command mentioned above. I will > > dig further and see if this is what we have theoretical limit for CAM > > attached HBA. > > First thing I noticed in this profile output is bunch of TLB shutdowns. > You can not reach reasonable performance from user-level without having > HBA support unmapped I/O. Both mps and mpr drivers support it, but for > some reason still not mrsas. Even at non-peak I/O rates on multi-core > system > TLB shutdowns in such case can eat additional 30% of CPU time. Thanks.! For this part, I can try In mrsas. Can you help me to understand what you mean by unmapped I/O ? > > Another thing I see is mentioned congestion on driver's CAM SIM lock. > You need either multiple cards or multiqueue. > > >> thoughts: > >> > >> First of all, I've never tried aio in my benchmarks, only synchronous > >> ones. Try to run 8 instances of `dd if=3D/dev/daX of=3D/dev/null bs=3D= 512` > >> per each SSD same time, just as I did. You may vary number of dd's, > >> but keep total below 256, or you mad to increase nswbuf limit in > >> kern_vfs_bio_buffer_alloc(). > > > > I ran multiple dd instance also and seeing IOPs throttle somewhere ~200= K > > . > > > > Do we have any mechanism to check CAM layer's max IOPs support > without > > involving actual Device ? Something like _null_ device driver which > > just send the command back to CAM layer ? > > There is not such one now. Such test would radically change timings of > operation, and I am not sure how useful would results be. > > >> For second, you are using single HBA, that should create significant > >> congestion around its CAM SIM lock. Proper solution would be to add > >> multiple queues support to the driver, and we discussed it with Scott > >> Long for quite some time, but that requires more work (I hope you may > >> be interested in it ;) ). Or you may just insert 3-4 HBAs. My million > >> IOPS I was reaching with four 2008/2308 6Gbps HBAs and 16 SATA SSDs. > > > > I remember this part and really good to contribute for this work. As > > part of this we have initiated multiple MSIx implementation in > > , which will have multiple reply queue per MSI-x. > > Cool! > > > Do we really require to have multiple Submission queue at low level > > driver > ? > > I thought it will be a CAM interface for multi queue which _all_ low > > level drivers need to hook into . > > Now CAM is still oriented on single submission queue, but allows driver t= o > have multiple completion queues. So I would start from implementing last > ones, each bound to own MSI-X interrupt and calling completion without > using the SIM lock or holding any other locks during the upcall. > CAM provides way to avoid extra context switch in that case, that could b= e > very useful. > > -- > Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Mon Jul 14 09:23:40 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A892A19A for ; Mon, 14 Jul 2014 09:23:40 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3DBFB222E for ; Mon, 14 Jul 2014 09:23:40 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id ho1so2158362wib.10 for ; Mon, 14 Jul 2014 02:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=th7CLfNtDUwHZEWYrPY5GDhpDnZa8y4bH/6fHIkW/j0=; b=qv/Bx8zB4AAL3yNxtlqcBVWzO1ElYkVlI6KZ3U3puP3v1zqxKNwGqGwAYG00X9hWWF rjot6hnGFcM6swNUDulzIePMAL6/B8mAuv8EO1csEyZm9cAhYdGL8wSFiuE2J1kEr42Q QxnmgMO2wPqAn3YRg+ZwTKPA4XSw+Nn0Ysuas8kzjJKZ5T8Po28KsNrOVlTbwuxShQml w32JIqnb+Kt9TOi1Ccawq3H0M3nR1bkmgIidmQ5Knz2Gphbpma1PK1GU1nu7pf1U1snk gqd/4FWiOOlCvLQ5lRAhLFR0W5dc2wjJaxQtPL/2j5gNIWl6LYbuv2A7N3XXloxx3r1H tC3w== X-Received: by 10.180.91.225 with SMTP id ch1mr23518473wib.34.1405329816737; Mon, 14 Jul 2014 02:23:36 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id w10sm28931192wie.22.2014.07.14.02.23.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 14 Jul 2014 02:23:35 -0700 (PDT) Sender: Alexander Motin Message-ID: <53C3A195.4020400@FreeBSD.org> Date: Mon, 14 Jul 2014 12:23:33 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Kashyap Desai Subject: Re: SSDs peformance on head/freebsd-10 stable using FIO References: <8fbe38cdad1e66717a9de7fdf63812c2@mail.gmail.com> <53BE8784.8060503@FreeBSD.org> <9f138f242e278476e5c542d695e58bc8@mail.gmail.com> <53BF1E6C.5030806@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: FreeBSD-scsi X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jul 2014 09:23:40 -0000 On 14.07.2014 11:36, Kashyap Desai wrote: > From: Alexander Motin [mailto:mavbsd@gmail.com] On Behalf Of Alexander >> First thing I noticed in this profile output is bunch of TLB shutdowns. >> You can not reach reasonable performance from user-level without having >> HBA support unmapped I/O. Both mps and mpr drivers support it, but for >> some reason still not mrsas. Even at non-peak I/O rates on multi-core >> system >> TLB shutdowns in such case can eat additional 30% of CPU time. > > Thanks.! For this part, I can try In mrsas. Can you help me to understand > what you mean by unmapped I/O ? That is a capability to work with data not mapped into the kernel virtual address space, i.e. to work with physical addresses instead of virtual. Main prerequisite to support that is that driver should not try to access the transferred data (because it can't do it for addresses not mapped to KVA). If that is true, then usually only minor modification is needed to teach the driver to receive physical addresses from CAM. Looking on mps driver as example you may see PIM_UNMAPPED flag reporting unmapped I/O support to CAM, and bus_dmamap_load_ccb() helper function transparently doing all the physical address handling magic. -- Alexander Motin