From owner-freebsd-scsi@freebsd.org Mon Jul 27 10:02:42 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D74D29AC492 for ; Mon, 27 Jul 2015 10:02:42 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail1.yamagi.org (yugo.yamagi.org [212.48.122.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9BA2D9C5 for ; Mon, 27 Jul 2015 10:02:41 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from [192.168.100.101] (helo=aka) by mail1.yamagi.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.85 (FreeBSD)) (envelope-from ) id 1ZJeek-000IXV-41; Mon, 27 Jul 2015 11:25:11 +0200 Date: Mon, 27 Jul 2015 11:25:04 +0200 From: Yamagi Burmeister To: killing@multiplay.co.uk Cc: freebsd-scsi@freebsd.org Subject: Re: Device timeouts(?) with LSI SAS3008 on mpr(4) Message-Id: <20150727112504.9b2c4bef27953f1e3dd52123@yamagi.org> In-Reply-To: <55A38AE1.5010204@multiplay.co.uk> References: <20150707132416.71b44c90f7f4cd6014a304b2@yamagi.org> <20150713110148.1a27b973881b64ce2f9e98e0@yamagi.org> <55A3813C.7010002@multiplay.co.uk> <20150713112547.8f044beabe26672fd13fc528@yamagi.org> <55A38AE1.5010204@multiplay.co.uk> X-Mailer: Sylpheed 3.4.3 (GTK+ 2.24.27; amd64-portbld-freebsd10.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jul 2015 10:02:42 -0000 Hello, let me appologise for my late answer. My colleagues were in vacation and I had no time to pursue this problem. But now another round: - da0 and da1 are 80G Intel DC S3500 SSDs - All other devices are 800G Intel DC S3700 SSDs kern.cam.da.X.delete_max seem very high for all devices. Forcing the tuneable to a very conservative value of 65536 helps, no timeout in 72 hours and no measurable performance impact. So my guess is: - ZFS tries to TRIM too many blocks in one operation - The SSD blocks for some time while processing the TRIM command - The controller thinks that the SSD crashed and sends a reset I did some tests with the attached tool. I'm able to reproduce the timeouts when "enough" data was written to the device. The question is what's "enough" data. Sometimes 50G are enough and sometimes 75G can be trimmed without any problem. Nevertheless. A lower kern.cam.da.X.delete_max value helps to work around the problem. And everything else is just speculation. So: Problem solved. Thank you for your help and input. Regards, Yamagi On Mon, 13 Jul 2015 10:54:41 +0100 Steven Hartland wrote: > I assume da0 and da1 are a different disk then? > > With regards your disk setup are all of you disks SSD's if so why do you > have separate log and cache devices? > > One thing you could try is to limit the delete size. > > kern.geom.dev.delete_max_sectors limits the single request size allowed > by geom but then individual requests can be built back up in cam so I > don't think this will help you too much. > > Instead I would try limiting the individual device delete_max, so add > one line per disk into /boot/loader.conf of the form: > kern.cam.da.X.delete_max=1073741824 > > You can actually change these on the fly using sysctl, but in order to > catch an cleanup done on boot loader.conf is the best place to tune them > permanently. > > I've attached a little c util which you can use to do direct disk > deletes if you have a spare disk you can play with. > > Be aware that most controller optimise delete's out if they know the > cells are empty hence you do need to have written data to the sectors > each time you test a delete. > > As the requests go through geom anything over > kern.geom.dev.delete_max_sectors will be split but then may well be > recombined in CAM. > > Another relevant setting is vfs.zfs.vdev.trim_max_active which can be > used to limit the number of outstanding geom delete requests to the each > device. > > Oh one other thing, it would be interesting to see the output from > camcontrol identify e.g. > camcontrol identify da8 > camcontrol identify da0 > > Regards > Steve > > On 13/07/2015 10:25, Yamagi Burmeister wrote: > > On Mon, 13 Jul 2015 10:13:32 +0100 > > Steven Hartland wrote: > > > >> What do you see from: > >> sysctl -a | grep -E '(delete|trim)' > > % sysctl -a | grep -E '(delete|trim)' > > kern.geom.dev.delete_max_sectors: 262144 > > kern.cam.da.1.delete_max: 8589803520 > > kern.cam.da.1.delete_method: ATA_TRIM > > kern.cam.da.8.delete_max: 12884705280 > > kern.cam.da.8.delete_method: ATA_TRIM > > kern.cam.da.9.delete_max: 12884705280 > > kern.cam.da.9.delete_method: ATA_TRIM > > kern.cam.da.3.delete_max: 12884705280 > > kern.cam.da.3.delete_method: ATA_TRIM > > kern.cam.da.12.delete_max: 12884705280 > > kern.cam.da.12.delete_method: ATA_TRIM > > kern.cam.da.7.delete_max: 12884705280 > > kern.cam.da.7.delete_method: ATA_TRIM > > kern.cam.da.2.delete_max: 12884705280 > > kern.cam.da.2.delete_method: ATA_TRIM > > kern.cam.da.11.delete_max: 12884705280 > > kern.cam.da.11.delete_method: ATA_TRIM > > kern.cam.da.6.delete_max: 12884705280 > > kern.cam.da.6.delete_method: ATA_TRIM > > kern.cam.da.10.delete_max: 12884705280 > > kern.cam.da.10.delete_method: ATA_TRIM > > kern.cam.da.5.delete_max: 12884705280 > > kern.cam.da.5.delete_method: ATA_TRIM > > kern.cam.da.0.delete_max: 8589803520 > > kern.cam.da.0.delete_method: ATA_TRIM > > kern.cam.da.4.delete_max: 12884705280 > > kern.cam.da.4.delete_method: ATA_TRIM > > vfs.zfs.trim.max_interval: 1 > > vfs.zfs.trim.timeout: 30 > > vfs.zfs.trim.txg_delay: 32 > > vfs.zfs.trim.enabled: 1 > > vfs.zfs.vdev.trim_max_pending: 10000 > > vfs.zfs.vdev.bio_delete_disable: 0 > > vfs.zfs.vdev.trim_max_active: 64 > > vfs.zfs.vdev.trim_min_active: 1 > > vfs.zfs.vdev.trim_on_init: 1 > > kstat.zfs.misc.arcstats.deleted: 289783817 > > kstat.zfs.misc.zio_trim.failed: 431 > > kstat.zfs.misc.zio_trim.unsupported: 0 > > kstat.zfs.misc.zio_trim.success: 6457142235 > > kstat.zfs.misc.zio_trim.bytes: 88207753330688 > > > > > >> Also while your seeing time-outs what does the output from gstat -d -p > >> look like? > > I'll try to get that data but it may take a while. > > > > Thank you, > > Yamagi > > > -- Homepage: www.yamagi.org XMPP: yamagi@yamagi.org GnuPG/GPG: 0xEFBCCBCB From owner-freebsd-scsi@freebsd.org Mon Jul 27 23:47:26 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DD3D9ACB5E for ; Mon, 27 Jul 2015 23:47:26 +0000 (UTC) (envelope-from info1@dvl-software.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 6C0511ED5 for ; Mon, 27 Jul 2015 23:47:26 +0000 (UTC) (envelope-from info1@dvl-software.com) Received: by mailman.ysv.freebsd.org (Postfix) id 68C879ACB5D; Mon, 27 Jul 2015 23:47:26 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 684879ACB5C for ; Mon, 27 Jul 2015 23:47:26 +0000 (UTC) (envelope-from info1@dvl-software.com) Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B56D1ED4 for ; Mon, 27 Jul 2015 23:47:25 +0000 (UTC) (envelope-from info1@dvl-software.com) Received: by igk11 with SMTP id 11so84267339igk.1 for ; Mon, 27 Jul 2015 16:47:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=8TCureem/8aj+z/gGa8zR/7effOFIwepWY736MSyiFw=; b=WzQBdW3BfIRGuA13v/aeKdORVd3Ov8L7u2dXfMoToUEvh8s8oZ4cqKIPXW3WJoipXp DtOkDBxkdjGPGTJnCC9lV+0glOhOvsPANUmcrU0o68lbz1HelSnLaW6SDAnmKQRA4e1r Czgym1petxQfaPWyCs88IAwGZ+Px+bmwm8VZvQwgGZpl1Y+QcYdJNVGGIWgUiW+EZC/w NEwmhIc/lNYRf2i7ETuqWMr0dNysjwBJWBRRQwKNtLMZVE0D9Z3xzaAbnl1jWhTuSeMa AHKrJQv1OGYwv78BKtJM0hx2y5wDXiGhk+mSFkO5M58m/3Hv88F+EWm8F0w/BJ8yl7vi XkOw== X-Gm-Message-State: ALoCoQlfqkCv8vXBfFmbn8lNjOSa78Z07HbypfM+81msDI3aRzuQAvfAhGSkSMxZATOLF9sJRsOz MIME-Version: 1.0 X-Received: by 10.50.43.199 with SMTP id y7mr723810igl.69.1438040844640; Mon, 27 Jul 2015 16:47:24 -0700 (PDT) Received: by 10.64.133.111 with HTTP; Mon, 27 Jul 2015 16:47:24 -0700 (PDT) In-Reply-To: References: <25A20430-E91B-49C5-BB4D-58913A5BE20D@langille.org> Date: Mon, 27 Jul 2015 19:47:24 -0400 Message-ID: Subject: Re: selecting a SCSI PCI-E card From: Dan Langille To: David Gwynne Cc: scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jul 2015 23:47:26 -0000 Look here! mpt0: port 0x8000-0x80ff mem 0xfb220000-0xfb23ffff,0xfb200000-0xfb21ffff irq 19 at device 8.0 on pci15 mpt0: MPI Version=3D1.2.14.0 It just been installed in the server. Hopefully I can get the tape library. attached tomorrow. Photos here: https://twitter.com/DLangille/status/625814151223451648 At least they've both LDV/SE. David: Thank you. :) On Fri, Jul 3, 2015 at 10:57 PM, David Gwynne wrote: > I'd have a look for LSI20320IE cards on eBay. > > Cheers, > dlg > On 4 Jul 2015 06:33, "Dan Langille" wrote: > >> I have an external tape library and I'm considering moving it to a new >> system, which does not have a PCI slot. >> >> The existing card is a Symbios Ultra2 32-bit PCI SCSI Adapter SYM8951U >> and dmsg output can be found here: >> >> https://dan.langille.org/2013/08/18/knew/ >> >> I'm pretty sure the card is LVD/SE SCSI based on the terminator (Externa= l >> U320 VHDCI68M LVD/SE SCSI Terminator). >> >> Thew new system has not been built yet, but one motherboard I like does >> not have a PCI slot, only PCI-E 3.0 and PCI-E 2.0. >> >> Do you know of a card supported by FreeBSD which meets the above needs? = I >> was looking on eBay and was failing. >> >> I suppose I could solve the problem by moving away from DLT-8000 to >> LTO-4; if you have a system you want to get rid of... >> >> Thanks. >> >> =E2=80=94 >> Dan Langille >> http://langille.org/ >> >> >> >> >> >> From owner-freebsd-scsi@freebsd.org Tue Jul 28 08:45:47 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E801B9AAC29 for ; Tue, 28 Jul 2015 08:45:46 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84E8418FE for ; Tue, 28 Jul 2015 08:45:45 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wibxm9 with SMTP id xm9so149962868wib.1 for ; Tue, 28 Jul 2015 01:45:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=zTkFham4hTNW8PqIjteCHKUYO2bcQ72g64IIoXXedVc=; b=hLE6GSzYU7pt6r7DUOcPEsYlLp1kKzl0o1iTZMdXXk5QXIxq9QRPCvavs9XwAUsmZF 2i1kUzWn1jtitleQRTJoREHhuI9pSfWgM8G9kEZsQaUhBA2RGn/7/aPLTTT5Pfl8P0sX IQz1h6pchyoK4sLIfSpfffMyNU9m8WqdtxdzAGpEtFyrkNDgp5eYkrkLT9i+u1mlL8fN Qx9wPg/DGdx9LHY0Mpu74WuXrWnrnTJf27fdUYVfZ9YGlf0EgosXqq+nzFKC0sjlxj8H AGlyKlktZYij4oZC4zlcDpkpP7m4MvzviiAPHs6zRIvwUuyq6wQ+siXoo4m0yziEiWEK SFGA== X-Gm-Message-State: ALoCoQnEOckgt3BSf/BcWq6ZooCJdhTN5Lkio3pvQSPrQwRfdW+64hZg/qjLOEcoj3nL83uhtigR X-Received: by 10.194.249.100 with SMTP id yt4mr68010787wjc.0.1438073144327; Tue, 28 Jul 2015 01:45:44 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by smtp.gmail.com with ESMTPSA id fa8sm17859063wib.14.2015.07.28.01.45.43 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Jul 2015 01:45:43 -0700 (PDT) Subject: Re: Device timeouts(?) with LSI SAS3008 on mpr(4) To: Yamagi Burmeister References: <20150707132416.71b44c90f7f4cd6014a304b2@yamagi.org> <20150713110148.1a27b973881b64ce2f9e98e0@yamagi.org> <55A3813C.7010002@multiplay.co.uk> <20150713112547.8f044beabe26672fd13fc528@yamagi.org> <55A38AE1.5010204@multiplay.co.uk> <20150727112504.9b2c4bef27953f1e3dd52123@yamagi.org> Cc: freebsd-scsi@freebsd.org From: Steven Hartland Message-ID: <55B74132.6030909@multiplay.co.uk> Date: Tue, 28 Jul 2015 09:45:38 +0100 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150727112504.9b2c4bef27953f1e3dd52123@yamagi.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jul 2015 08:45:47 -0000 65536 is really small and will likely cause bottleneck when TRIMing large areas. I'd suggest a larger value 2 - 4 MB but at least matching kern.geom.dev.delete_max_sectors (262144) Regards Steve On 27/07/2015 10:25, Yamagi Burmeister wrote: > Hello, > let me appologise for my late answer. My colleagues were in vacation > and I had no time to pursue this problem. But now another round: > > - da0 and da1 are 80G Intel DC S3500 SSDs > - All other devices are 800G Intel DC S3700 SSDs > > kern.cam.da.X.delete_max seem very high for all devices. Forcing the > tuneable to a very conservative value of 65536 helps, no timeout in 72 > hours and no measurable performance impact. So my guess is: > > - ZFS tries to TRIM too many blocks in one operation > - The SSD blocks for some time while processing the TRIM command > - The controller thinks that the SSD crashed and sends a reset > > I did some tests with the attached tool. I'm able to reproduce the > timeouts when "enough" data was written to the device. The question is > what's "enough" data. Sometimes 50G are enough and sometimes 75G can be > trimmed without any problem. > > Nevertheless. A lower kern.cam.da.X.delete_max value helps to work > around the problem. And everything else is just speculation. So: > Problem solved. Thank you for your help and input. > > Regards, > Yamagi > > On Mon, 13 Jul 2015 10:54:41 +0100 > Steven Hartland wrote: > >> I assume da0 and da1 are a different disk then? >> >> With regards your disk setup are all of you disks SSD's if so why do you >> have separate log and cache devices? >> >> One thing you could try is to limit the delete size. >> >> kern.geom.dev.delete_max_sectors limits the single request size allowed >> by geom but then individual requests can be built back up in cam so I >> don't think this will help you too much. >> >> Instead I would try limiting the individual device delete_max, so add >> one line per disk into /boot/loader.conf of the form: >> kern.cam.da.X.delete_max=1073741824 >> >> You can actually change these on the fly using sysctl, but in order to >> catch an cleanup done on boot loader.conf is the best place to tune them >> permanently. >> >> I've attached a little c util which you can use to do direct disk >> deletes if you have a spare disk you can play with. >> >> Be aware that most controller optimise delete's out if they know the >> cells are empty hence you do need to have written data to the sectors >> each time you test a delete. >> >> As the requests go through geom anything over >> kern.geom.dev.delete_max_sectors will be split but then may well be >> recombined in CAM. >> >> Another relevant setting is vfs.zfs.vdev.trim_max_active which can be >> used to limit the number of outstanding geom delete requests to the each >> device. >> >> Oh one other thing, it would be interesting to see the output from >> camcontrol identify e.g. >> camcontrol identify da8 >> camcontrol identify da0 >> >> Regards >> Steve >> >> On 13/07/2015 10:25, Yamagi Burmeister wrote: >>> On Mon, 13 Jul 2015 10:13:32 +0100 >>> Steven Hartland wrote: >>> >>>> What do you see from: >>>> sysctl -a | grep -E '(delete|trim)' >>> % sysctl -a | grep -E '(delete|trim)' >>> kern.geom.dev.delete_max_sectors: 262144 >>> kern.cam.da.1.delete_max: 8589803520 >>> kern.cam.da.1.delete_method: ATA_TRIM >>> kern.cam.da.8.delete_max: 12884705280 >>> kern.cam.da.8.delete_method: ATA_TRIM >>> kern.cam.da.9.delete_max: 12884705280 >>> kern.cam.da.9.delete_method: ATA_TRIM >>> kern.cam.da.3.delete_max: 12884705280 >>> kern.cam.da.3.delete_method: ATA_TRIM >>> kern.cam.da.12.delete_max: 12884705280 >>> kern.cam.da.12.delete_method: ATA_TRIM >>> kern.cam.da.7.delete_max: 12884705280 >>> kern.cam.da.7.delete_method: ATA_TRIM >>> kern.cam.da.2.delete_max: 12884705280 >>> kern.cam.da.2.delete_method: ATA_TRIM >>> kern.cam.da.11.delete_max: 12884705280 >>> kern.cam.da.11.delete_method: ATA_TRIM >>> kern.cam.da.6.delete_max: 12884705280 >>> kern.cam.da.6.delete_method: ATA_TRIM >>> kern.cam.da.10.delete_max: 12884705280 >>> kern.cam.da.10.delete_method: ATA_TRIM >>> kern.cam.da.5.delete_max: 12884705280 >>> kern.cam.da.5.delete_method: ATA_TRIM >>> kern.cam.da.0.delete_max: 8589803520 >>> kern.cam.da.0.delete_method: ATA_TRIM >>> kern.cam.da.4.delete_max: 12884705280 >>> kern.cam.da.4.delete_method: ATA_TRIM >>> vfs.zfs.trim.max_interval: 1 >>> vfs.zfs.trim.timeout: 30 >>> vfs.zfs.trim.txg_delay: 32 >>> vfs.zfs.trim.enabled: 1 >>> vfs.zfs.vdev.trim_max_pending: 10000 >>> vfs.zfs.vdev.bio_delete_disable: 0 >>> vfs.zfs.vdev.trim_max_active: 64 >>> vfs.zfs.vdev.trim_min_active: 1 >>> vfs.zfs.vdev.trim_on_init: 1 >>> kstat.zfs.misc.arcstats.deleted: 289783817 >>> kstat.zfs.misc.zio_trim.failed: 431 >>> kstat.zfs.misc.zio_trim.unsupported: 0 >>> kstat.zfs.misc.zio_trim.success: 6457142235 >>> kstat.zfs.misc.zio_trim.bytes: 88207753330688 >>> >>> >>>> Also while your seeing time-outs what does the output from gstat -d -p >>>> look like? >>> I'll try to get that data but it may take a while. >>> >>> Thank you, >>> Yamagi >>> > From owner-freebsd-scsi@freebsd.org Tue Jul 28 17:01:47 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2AC8C9AD96E for ; Tue, 28 Jul 2015 17:01:47 +0000 (UTC) (envelope-from dan@langille.org) Received: from clavin1.langille.org (clavin.langille.org [162.208.116.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "clavin.langille.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0ACEBF54 for ; Tue, 28 Jul 2015 17:01:46 +0000 (UTC) (envelope-from dan@langille.org) Received: from (clavin1.int.langille.org (clavin1.int.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) with ESMTPSA id DFD5F62FE for ; Tue, 28 Jul 2015 17:01:38 +0000 (UTC) From: Dan Langille Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: dmesg output not as expected Message-Id: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> Date: Tue, 28 Jul 2015 13:01:37 -0400 To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jul 2015 17:01:47 -0000 I wish to move a tape library from one system to another. Today I moved = the cable from one system to another and ran 'camcontrol rescan all'. For sa0, why do I see 80MB/s on the old system vs 40MB/s on the new = system? Is this card related? Slot related? Similarly, ch0 goes from 20MB/s to 3.3MB/s....=20 I was hoping for a speed boost. :) Some of the camcontrol devlist output includes: at scbus14 target 1 lun 0 = (pass11,sa0) at scbus14 target 3 lun 0 = (pass12,ch0) This is on the existing system: I *think* this system holds a Symbios Ultra2 32-bit PCI SCSI Adapter = SYM8951U Jul 23 13:24:19 knew kernel: sym0: <895> port 0xe000-0xe0ff mem = 0xfebeec00-0xfebeecff,0xfebef000-0xfebeffff irq 22 at device 2.0 on pci4 Jul 23 13:24:19 knew kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity = checking Jul 23 13:24:19 knew kernel: sa0 at sym0 bus 0 scbus7 target 1 lun 0 Jul 23 13:24:19 knew kernel: sa0: Removable = Sequential Access SCSI-2 device=20 Jul 23 13:24:19 knew kernel: sa0: Serial Number CXB46H0716 =20 Jul 23 13:24:19 knew kernel: sa0: 80.000MB/s transfers (40.000MHz, = offset 31, 16bit) Jul 23 13:24:19 knew kernel: ch0 at sym0 bus 0 scbus7 target 3 lun 0 Jul 23 13:24:19 knew kernel: ch0: Removable = Changer SCSI-2 device=20 Jul 23 13:24:19 knew kernel: ch0: Serial Number 3G22JJP38S46 Jul 23 13:24:19 knew kernel: ch0: 20.000MB/s transfers (10.000MHz, = offset 15, 16bit) Jul 23 13:24:19 knew kernel: ch0: 25 slots, 1 drive, 1 picker, 1 portal Jul 23 13:24:19 knew kernel: ch0: quirks=3D0x2 This is the same tape drive on another system: LSI Logic Ultra320 (LSI20320IE) SCSI PCI-E mpt0: port 0x8000-0x80ff mem = 0xfb220000-0xfb23ffff,0xfb200000-0xfb21ffff irq 19 at device 8.0 on = pci15 mpt0: MPI Version=3D1.2.14.0 Jul 28 16:36:31 varm kernel: sa0 at mpt0 bus 0 scbus14 target 1 lun 0 Jul 28 16:36:31 varm kernel: sa0: Removable = Sequential Access SCSI-2 device Jul 28 16:36:31 varm kernel: sa0: Serial Number CXB46H0716 =20 Jul 28 16:36:31 varm kernel: sa0: 40.000MB/s transfers (40.000MHz DT, = offset 6) Jul 28 16:37:03 varm kernel: ch0 at mpt0 bus 0 scbus14 target 3 lun 0 Jul 28 16:37:03 varm kernel: ch0: Removable = Changer SCSI-2 device Jul 28 16:37:03 varm kernel: ch0: Serial Number 3G22JJP38S46 Jul 28 16:37:03 varm kernel: ch0: 3.300MB/s transfers Jul 28 16:37:03 varm kernel: ch0: 25 slots, 1 drive, 1 picker, 1 portal Jul 28 16:37:03 varm kernel: ch0: quirks=3D0x2 Thank you. =E2=80=94=20 Dan Langille http://langille.org/ From owner-freebsd-scsi@freebsd.org Wed Jul 29 02:18:47 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BF839AD44D for ; Wed, 29 Jul 2015 02:18:47 +0000 (UTC) (envelope-from dan@langille.org) Received: from clavin1.langille.org (clavin.langille.org [162.208.116.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "clavin.langille.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B8611F87 for ; Wed, 29 Jul 2015 02:18:46 +0000 (UTC) (envelope-from dan@langille.org) Received: from (clavin1.int.langille.org (clavin1.int.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) with ESMTPSA id B82F2665B for ; Wed, 29 Jul 2015 02:18:44 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: dmesg output not as expected From: Dan Langille In-Reply-To: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> Date: Tue, 28 Jul 2015 22:18:43 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> To: freebsd-scsi@freebsd.org X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jul 2015 02:18:47 -0000 > On Jul 28, 2015, at 1:01 PM, Dan Langille wrote: >=20 > I wish to move a tape library from one system to another. Today I = moved the cable from one system to another and ran 'camcontrol rescan = all'. >=20 > For sa0, why do I see 80MB/s on the old system vs 40MB/s on the new = system? Is this card related? Slot related? >=20 > Similarly, ch0 goes from 20MB/s to 3.3MB/s....=20 >=20 > I was hoping for a speed boost. :) >=20 > Some of the camcontrol devlist output includes: >=20 > at scbus14 target 1 lun 0 = (pass11,sa0) > at scbus14 target 3 lun 0 = (pass12,ch0) >=20 > This is on the existing system: >=20 > I *think* this system holds a Symbios Ultra2 32-bit PCI SCSI Adapter = SYM8951U >=20 > Jul 23 13:24:19 knew kernel: sym0: <895> port 0xe000-0xe0ff mem = 0xfebeec00-0xfebeecff,0xfebef000-0xfebeffff irq 22 at device 2.0 on pci4 > Jul 23 13:24:19 knew kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, = parity checking >=20 > Jul 23 13:24:19 knew kernel: sa0 at sym0 bus 0 scbus7 target 1 lun 0 > Jul 23 13:24:19 knew kernel: sa0: Removable = Sequential Access SCSI-2 device=20 > Jul 23 13:24:19 knew kernel: sa0: Serial Number CXB46H0716 =20 > Jul 23 13:24:19 knew kernel: sa0: 80.000MB/s transfers (40.000MHz, = offset 31, 16bit) >=20 > Jul 23 13:24:19 knew kernel: ch0 at sym0 bus 0 scbus7 target 3 lun 0 > Jul 23 13:24:19 knew kernel: ch0: = Removable Changer SCSI-2 device=20 > Jul 23 13:24:19 knew kernel: ch0: Serial Number 3G22JJP38S46 > Jul 23 13:24:19 knew kernel: ch0: 20.000MB/s transfers (10.000MHz, = offset 15, 16bit) > Jul 23 13:24:19 knew kernel: ch0: 25 slots, 1 drive, 1 picker, 1 = portal > Jul 23 13:24:19 knew kernel: ch0: quirks=3D0x2 >=20 > This is the same tape drive on another system: >=20 > LSI Logic Ultra320 (LSI20320IE) SCSI PCI-E >=20 > mpt0: port 0x8000-0x80ff mem = 0xfb220000-0xfb23ffff,0xfb200000-0xfb21ffff irq 19 at device 8.0 on = pci15 > mpt0: MPI Version=3D1.2.14.0 >=20 > Jul 28 16:36:31 varm kernel: sa0 at mpt0 bus 0 scbus14 target 1 lun 0 > Jul 28 16:36:31 varm kernel: sa0: Removable = Sequential Access SCSI-2 device > Jul 28 16:36:31 varm kernel: sa0: Serial Number CXB46H0716 =20 > Jul 28 16:36:31 varm kernel: sa0: 40.000MB/s transfers (40.000MHz DT, = offset 6) >=20 > Jul 28 16:37:03 varm kernel: ch0 at mpt0 bus 0 scbus14 target 3 lun 0 > Jul 28 16:37:03 varm kernel: ch0: = Removable Changer SCSI-2 device > Jul 28 16:37:03 varm kernel: ch0: Serial Number 3G22JJP38S46 > Jul 28 16:37:03 varm kernel: ch0: 3.300MB/s transfers > Jul 28 16:37:03 varm kernel: ch0: 25 slots, 1 drive, 1 picker, 1 = portal > Jul 28 16:37:03 varm kernel: ch0: quirks=3D0x2 After rebooting this server, I see this in /var/run/dmesg. Why does it = change? ch0 at mpt0 bus 0 scbus14 target 3 lun 0 ch0: Removable Changer SCSI-2 device ch0: Serial Number 3G22JJP38S46 ch0: 20.000MB/s transfers (10.000MHz DT, offset 15, 16bit) ch0: 25 slots, 1 drive, 1 picker, 1 portal ch0: quirks=3D0x2 I was hoping to get more through put on this newer system with a = different card. =46rom what I see, my speed is only half of what it = was. I'm using a PCI-E 3.0 X8 (in X16) slot. Have I missing something? =E2=80=94=20 Dan Langille http://langille.org/ From owner-freebsd-scsi@freebsd.org Wed Jul 29 02:30:36 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54BB79AD7FC for ; Wed, 29 Jul 2015 02:30:36 +0000 (UTC) (envelope-from david@gwynne.id.au) Received: from mail-pd0-f175.google.com (mail-pd0-f175.google.com [209.85.192.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E9B89FA for ; Wed, 29 Jul 2015 02:30:35 +0000 (UTC) (envelope-from david@gwynne.id.au) Received: by pdjr16 with SMTP id r16so81976035pdj.3 for ; Tue, 28 Jul 2015 19:30:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=2stkn7ZsAAibnbzMZZy3V5SZ3n9Do60n598cZBUqGHU=; b=CxJA+JRKi6mQT2hDZ5O5EVbM7MiO1vT2WqV8tDH9ds9xkhZSMhbGuzHzecUE/OnZsU 38R1VEP9YFJcTtLwT22kQcRTa0m3O5K5KS6Ipra6MO2D0sE7JMqRp/SzHZzxJWa5blGy qyvMrFMjAw1Y53WlfxoaRu2oOcJN2aBUtjkhJCexJtSdoPISaZ/RsrJCtwtK8l1yK1G8 9g2wUzBKVkbvc4Lmo9W1+z9THdyCiK+GGS+6drVKFbpeJrA+/GA85BuEsJ+dDTtZbfzQ bq9mEENKELbuaiL1kppCQXkmXiHyTQPKs+aFeqtLG/MkTBB/dsSTFgFwD6jrUUdSEiEc rHQg== X-Gm-Message-State: ALoCoQkgZmeeUhLLjWIl94S5hUXAA3a2NusHMGV1uV5kGSURrdjQA6+gA6kWsTTgphw8thIB9i/N X-Received: by 10.70.134.198 with SMTP id pm6mr88956347pdb.17.1438137034454; Tue, 28 Jul 2015 19:30:34 -0700 (PDT) Received: from opiate.eait.uq.edu.au (a82-47.nat.uq.edu.au. [130.102.82.47]) by smtp.gmail.com with ESMTPSA id xo14sm37715189pac.24.2015.07.28.19.30.31 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 28 Jul 2015 19:30:32 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: dmesg output not as expected From: David Gwynne In-Reply-To: Date: Wed, 29 Jul 2015 12:30:27 +1000 Cc: freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> To: Dan Langille X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jul 2015 02:30:36 -0000 > On 29 Jul 2015, at 12:18, Dan Langille wrote: >=20 >>=20 >> On Jul 28, 2015, at 1:01 PM, Dan Langille wrote: >>=20 >> I wish to move a tape library from one system to another. Today I = moved the cable from one system to another and ran 'camcontrol rescan = all'. >>=20 >> For sa0, why do I see 80MB/s on the old system vs 40MB/s on the new = system? Is this card related? Slot related? >>=20 >> Similarly, ch0 goes from 20MB/s to 3.3MB/s....=20 >>=20 >> I was hoping for a speed boost. :) >>=20 >> Some of the camcontrol devlist output includes: >>=20 >> at scbus14 target 1 lun 0 = (pass11,sa0) >> at scbus14 target 3 lun 0 = (pass12,ch0) >>=20 >> This is on the existing system: >>=20 >> I *think* this system holds a Symbios Ultra2 32-bit PCI SCSI Adapter = SYM8951U >>=20 >> Jul 23 13:24:19 knew kernel: sym0: <895> port 0xe000-0xe0ff mem = 0xfebeec00-0xfebeecff,0xfebef000-0xfebeffff irq 22 at device 2.0 on pci4 >> Jul 23 13:24:19 knew kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, = parity checking >>=20 >> Jul 23 13:24:19 knew kernel: sa0 at sym0 bus 0 scbus7 target 1 lun 0 >> Jul 23 13:24:19 knew kernel: sa0: Removable = Sequential Access SCSI-2 device=20 >> Jul 23 13:24:19 knew kernel: sa0: Serial Number CXB46H0716 =20 >> Jul 23 13:24:19 knew kernel: sa0: 80.000MB/s transfers (40.000MHz, = offset 31, 16bit) >>=20 >> Jul 23 13:24:19 knew kernel: ch0 at sym0 bus 0 scbus7 target 3 lun 0 >> Jul 23 13:24:19 knew kernel: ch0: = Removable Changer SCSI-2 device=20 >> Jul 23 13:24:19 knew kernel: ch0: Serial Number 3G22JJP38S46 >> Jul 23 13:24:19 knew kernel: ch0: 20.000MB/s transfers (10.000MHz, = offset 15, 16bit) >> Jul 23 13:24:19 knew kernel: ch0: 25 slots, 1 drive, 1 picker, 1 = portal >> Jul 23 13:24:19 knew kernel: ch0: quirks=3D0x2 >>=20 >> This is the same tape drive on another system: >>=20 >> LSI Logic Ultra320 (LSI20320IE) SCSI PCI-E >>=20 >> mpt0: port 0x8000-0x80ff mem = 0xfb220000-0xfb23ffff,0xfb200000-0xfb21ffff irq 19 at device 8.0 on = pci15 >> mpt0: MPI Version=3D1.2.14.0 >>=20 >> Jul 28 16:36:31 varm kernel: sa0 at mpt0 bus 0 scbus14 target 1 lun 0 >> Jul 28 16:36:31 varm kernel: sa0: Removable = Sequential Access SCSI-2 device >> Jul 28 16:36:31 varm kernel: sa0: Serial Number CXB46H0716 =20 >> Jul 28 16:36:31 varm kernel: sa0: 40.000MB/s transfers (40.000MHz DT, = offset 6) >>=20 >> Jul 28 16:37:03 varm kernel: ch0 at mpt0 bus 0 scbus14 target 3 lun 0 >> Jul 28 16:37:03 varm kernel: ch0: = Removable Changer SCSI-2 device >> Jul 28 16:37:03 varm kernel: ch0: Serial Number 3G22JJP38S46 >> Jul 28 16:37:03 varm kernel: ch0: 3.300MB/s transfers >> Jul 28 16:37:03 varm kernel: ch0: 25 slots, 1 drive, 1 picker, 1 = portal >> Jul 28 16:37:03 varm kernel: ch0: quirks=3D0x2 >=20 > After rebooting this server, I see this in /var/run/dmesg. Why does = it change? >=20 > ch0 at mpt0 bus 0 scbus14 target 3 lun 0 > ch0: Removable Changer SCSI-2 device > ch0: Serial Number 3G22JJP38S46 > ch0: 20.000MB/s transfers (10.000MHz DT, offset 15, 16bit) > ch0: 25 slots, 1 drive, 1 picker, 1 portal > ch0: quirks=3D0x2 >=20 > I was hoping to get more through put on this newer system with a = different card. =46rom what I see, my speed is only half of what it = was. >=20 > I'm using a PCI-E 3.0 X8 (in X16) slot. Have I missing something? technically parallel scsi isn't a hot pluggable medium. rebooting likely = lets it settle and negotiate better. you probably shouldnt care about the speed you talk to the changer at = btw. you're not sending data to the changer device, just commands to = move tapes around which are very low bandwidth. did your tape device negotiate faster after the reboot? dlg= From owner-freebsd-scsi@freebsd.org Wed Jul 29 02:33:08 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C0BC9AD93B for ; Wed, 29 Jul 2015 02:33:08 +0000 (UTC) (envelope-from dan@langille.org) Received: from clavin1.langille.org (clavin.langille.org [162.208.116.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "clavin.langille.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B439BEA for ; Wed, 29 Jul 2015 02:33:07 +0000 (UTC) (envelope-from dan@langille.org) Received: from (clavin1.int.langille.org (clavin1.int.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) with ESMTPSA id C25CF6660 ; Wed, 29 Jul 2015 02:33:06 +0000 (UTC) Subject: Re: dmesg output not as expected Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Content-Type: multipart/signed; boundary="Apple-Mail=_035793B9-0530-45D2-A2C1-7221D4D643CD"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5 From: Dan Langille In-Reply-To: Date: Tue, 28 Jul 2015 22:33:05 -0400 Cc: freebsd-scsi@freebsd.org Message-Id: References: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> To: David Gwynne X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jul 2015 02:33:08 -0000 --Apple-Mail=_035793B9-0530-45D2-A2C1-7221D4D643CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jul 28, 2015, at 10:30 PM, David Gwynne wrote: >=20 >=20 >> On 29 Jul 2015, at 12:18, Dan Langille wrote: >>=20 >>>=20 >>> On Jul 28, 2015, at 1:01 PM, Dan Langille wrote: >>>=20 >>> I wish to move a tape library from one system to another. Today I = moved the cable from one system to another and ran 'camcontrol rescan = all'. >>>=20 >>> For sa0, why do I see 80MB/s on the old system vs 40MB/s on the new = system? Is this card related? Slot related? >>>=20 >>> Similarly, ch0 goes from 20MB/s to 3.3MB/s.... >>>=20 >>> I was hoping for a speed boost. :) >>>=20 >>> Some of the camcontrol devlist output includes: >>>=20 >>> at scbus14 target 1 lun 0 = (pass11,sa0) >>> at scbus14 target 3 lun 0 = (pass12,ch0) >>>=20 >>> This is on the existing system: >>>=20 >>> I *think* this system holds a Symbios Ultra2 32-bit PCI SCSI Adapter = SYM8951U >>>=20 >>> Jul 23 13:24:19 knew kernel: sym0: <895> port 0xe000-0xe0ff mem = 0xfebeec00-0xfebeecff,0xfebef000-0xfebeffff irq 22 at device 2.0 on pci4 >>> Jul 23 13:24:19 knew kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, = parity checking >>>=20 >>> Jul 23 13:24:19 knew kernel: sa0 at sym0 bus 0 scbus7 target 1 lun 0 >>> Jul 23 13:24:19 knew kernel: sa0: Removable = Sequential Access SCSI-2 device >>> Jul 23 13:24:19 knew kernel: sa0: Serial Number CXB46H0716 >>> Jul 23 13:24:19 knew kernel: sa0: 80.000MB/s transfers (40.000MHz, = offset 31, 16bit) >>>=20 >>> Jul 23 13:24:19 knew kernel: ch0 at sym0 bus 0 scbus7 target 3 lun 0 >>> Jul 23 13:24:19 knew kernel: ch0: = Removable Changer SCSI-2 device >>> Jul 23 13:24:19 knew kernel: ch0: Serial Number 3G22JJP38S46 >>> Jul 23 13:24:19 knew kernel: ch0: 20.000MB/s transfers (10.000MHz, = offset 15, 16bit) >>> Jul 23 13:24:19 knew kernel: ch0: 25 slots, 1 drive, 1 picker, 1 = portal >>> Jul 23 13:24:19 knew kernel: ch0: quirks=3D0x2 >>>=20 >>> This is the same tape drive on another system: >>>=20 >>> LSI Logic Ultra320 (LSI20320IE) SCSI PCI-E >>>=20 >>> mpt0: port 0x8000-0x80ff mem = 0xfb220000-0xfb23ffff,0xfb200000-0xfb21ffff irq 19 at device 8.0 on = pci15 >>> mpt0: MPI Version=3D1.2.14.0 >>>=20 >>> Jul 28 16:36:31 varm kernel: sa0 at mpt0 bus 0 scbus14 target 1 lun = 0 >>> Jul 28 16:36:31 varm kernel: sa0: Removable = Sequential Access SCSI-2 device >>> Jul 28 16:36:31 varm kernel: sa0: Serial Number CXB46H0716 >>> Jul 28 16:36:31 varm kernel: sa0: 40.000MB/s transfers (40.000MHz = DT, offset 6) >>>=20 >>> Jul 28 16:37:03 varm kernel: ch0 at mpt0 bus 0 scbus14 target 3 lun = 0 >>> Jul 28 16:37:03 varm kernel: ch0: = Removable Changer SCSI-2 device >>> Jul 28 16:37:03 varm kernel: ch0: Serial Number 3G22JJP38S46 >>> Jul 28 16:37:03 varm kernel: ch0: 3.300MB/s transfers >>> Jul 28 16:37:03 varm kernel: ch0: 25 slots, 1 drive, 1 picker, 1 = portal >>> Jul 28 16:37:03 varm kernel: ch0: quirks=3D0x2 >>=20 >> After rebooting this server, I see this in /var/run/dmesg. Why does = it change? >>=20 >> ch0 at mpt0 bus 0 scbus14 target 3 lun 0 >> ch0: Removable Changer SCSI-2 device >> ch0: Serial Number 3G22JJP38S46 >> ch0: 20.000MB/s transfers (10.000MHz DT, offset 15, 16bit) >> ch0: 25 slots, 1 drive, 1 picker, 1 portal >> ch0: quirks=3D0x2 >>=20 >> I was hoping to get more through put on this newer system with a = different card. =46rom what I see, my speed is only half of what it = was. >>=20 >> I'm using a PCI-E 3.0 X8 (in X16) slot. Have I missing something? >=20 > technically parallel scsi isn't a hot pluggable medium. rebooting = likely lets it settle and negotiate better. >=20 > you probably shouldnt care about the speed you talk to the changer at = btw. you're not sending data to the changer device, just commands to = move tapes around which are very low bandwidth. Noted. However, sa0 went from 80 to 40MB/s... > did your tape device negotiate faster after the reboot? I did not notice. I'll compare soon. =E2=80=94 Dan Langille http://langille.org/ --Apple-Mail=_035793B9-0530-45D2-A2C1-7221D4D643CD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQJ8BAEBCgBmBQJVuDtiXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ1MTE2RjM0ODIzRDdERDM4OTY0OUJBNzdF QjIxNTlERUU5NzI3MzlGAAoJEOshWd7pcnOfW/IP/jBHrtKM2A8kNonrlC7Ktdv8 l6YGmFoOgf8i6J+/d0qpW6TlBWK0fS210q/djb+THKSQ8C354/oWajKaXyUukbx+ 1ISl+H+zRcQH1H0yV7phI/a5p4wJxxAmscFxnfjhJHgxQBmEkWRhhKM+mOcTXukS TAqeCG01k3Vjmlj/bo5aQia4hmr6qnXwfeIwoij986M6fGmoOt/zRrQ0UzbtxLT7 riBocKEso3pOcsHSSi/IQg9W5gcXsbX8Fp9a/CiNqk9rmrmHG0UeRg8KYjaCv9MY pNOLoL5OUWhxkvgdpcyt64bsgCs5k1B5O5oDomCE9lZ/T8tm99JL9pZ3Cq3Aotw2 nE0MS+zTuUcVkyOrM7a4ZtRthGL2ag9tjj9/XxkOst6VMxzinLWiMx5cI0juhaKK JKNxA21zL81I8PKRfw3DIHq5B+vdm4Gt4ZFd8rrFO7HibuGXYg18coRnPzAlPeqE /2j7qVJFVr2YO5ogrOObkLa8EyGjCYTJrJGPR7wDynWaVMDsEne9Y0E5gnAHhJv/ vNUTxkBXx3QU2HA/J5m2w2nVsBK4jZjQXl6ty2NcbohKQUb9M0OCftDac/n0nGtR yylJ3iOI5RIsri7Dfl0C2w9JdbRz1cvGxUZ9BUXEQh8f4MGn3eLRGiT65HzR153B 6SvCTbgYoCvfDw1k6KCL =C7VJ -----END PGP SIGNATURE----- --Apple-Mail=_035793B9-0530-45D2-A2C1-7221D4D643CD-- From owner-freebsd-scsi@freebsd.org Wed Jul 29 03:58:40 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45D1A9AB4B1 for ; Wed, 29 Jul 2015 03:58:40 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm21-vm7.bullet.mail.gq1.yahoo.com (nm21-vm7.bullet.mail.gq1.yahoo.com [98.136.217.54]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 159DF1197 for ; Wed, 29 Jul 2015 03:58:39 +0000 (UTC) (envelope-from scott4long@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1438142216; bh=Iy5GJLAsvHW9m1kN7RjZM2zHQJbDHwhCJ55MeqWSnC8=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=hrT4yonLdcxHv795E9GEt5VMx45oVnrkcWhrrW8wDMI7SLYWhFH1qKZyagDExWe94udDZUcWbz+orJdgvVjJW7A7A/X631t7BcMfHEh1VbyP5gnfqwWOPuzGZgSfbl7v+qbakDmFUYvTZwtx4axPgw/2by3ikQv88syonnn2UOMBgwmS8WBhoLTwk45w7acYeBHFSUH4DI4Utd7zIL5RE+i3w1Lfp1FVQyXUF+heTQCO/cg8+dLgK8iIqYwQpOHURzQzYQ+aG1spNC5dpMzebYRqiv1me3zrUSu+QEqPfDBVeKAoQ0spUZShKktXCTRZgruzMQyKd2mtmxD5li2+Yg== Received: from [98.137.12.189] by nm21.bullet.mail.gq1.yahoo.com with NNFMP; 29 Jul 2015 03:56:56 -0000 Received: from [208.71.42.214] by tm10.bullet.mail.gq1.yahoo.com with NNFMP; 29 Jul 2015 03:56:56 -0000 Received: from [127.0.0.1] by smtp225.mail.gq1.yahoo.com with NNFMP; 29 Jul 2015 03:56:56 -0000 X-Yahoo-Newman-Id: 254314.17331.bm@smtp225.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 1043a.4VM1nnnEgI_YQKah5.pDtxTOQ2Gk8rb0bEv9AjlxU C0rvwanwEoCP3mEL9WourMrjFz5TA6ac5KGnv5UBdV7byzAyXFZgL6CxFXwB DWw9Ik9No39QHxWhEUR.pm6dY3Vbzr.R_U01l3ifRknmspTMkqbnLs7ht_Se NxnwOMVs.qrAYyA4Tlp165R5ZLfSrdetQu8M4zFCJVU6OLJt35xdn9a0.nv0 .MGW6vSoyHo37ot1yRo6GkR9wWakO0stfhhujja02uoztpfEl3ZX6nhnxucM t6Mu0Caum9GzuO1wafclkfgUjja3bqTFk20wDELoswJ4lYTsjswtAHV5iF3G 9nNlSS5GV43OFnt4OCuDeQ5GXoBIoXjNp8tMgGsgZCx9sFHI9CPMAwQXQbWa TwdJgBZzSuRVGVxoZLsXA8wg.6pZaKZ7hm2Ks2S.HGNNEHNgfRlKndj6yBO4 SKHiO6IXv3AKzGYExp776o1cIIPv0JvAK19YBnqNLZHUhXQCN8spxBiTQPCR peJVgxox7.5chsAyVAgCivDSfP1U9Cg-- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: dmesg output not as expected From: Scott Long In-Reply-To: Date: Tue, 28 Jul 2015 21:56:54 -0600 Cc: freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <27741F67-95F8-4EEF-B42F-E80626799FCB@yahoo.com> References: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> To: Dan Langille X-Mailer: Apple Mail (2.2098) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jul 2015 03:58:40 -0000 > On Jul 28, 2015, at 8:18 PM, Dan Langille wrote: >=20 > After rebooting this server, I see this in /var/run/dmesg. Why does = it change? >=20 > ch0 at mpt0 bus 0 scbus14 target 3 lun 0 > ch0: Removable Changer SCSI-2 device > ch0: Serial Number 3G22JJP38S46 > ch0: 20.000MB/s transfers (10.000MHz DT, offset 15, 16bit) > ch0: 25 slots, 1 drive, 1 picker, 1 portal > ch0: quirks=3D0x2 >=20 > I was hoping to get more through put on this newer system with a = different card. =46rom what I see, my speed is only half of what it = was. >=20 > I'm using a PCI-E 3.0 X8 (in X16) slot. Have I missing something? >=20 The LSI controllers do Domain Validation. That means that they send a = series of test commands to each target to try to determine a safe speed = to negotiate to. The theory is that these test commands will determine = whether the target=E2=80=99s speed needs to be downshifted in order to = avoid problems with bad cables and bad connectors. It=E2=80=99s part of = the Ultra320 spec, and I think it was in the Ultra160 spec too, but was = rarely implemented there. It=E2=80=99s possible that the DV routine is = telling the card that these devices need to have their negotiated speed = reduced, and that the results it gets are a bit on-the-fence at times = and cause different observed speeds. It=E2=80=99s been a long time since I worked with DV, so my memory is a = bit fuzzy. The Adaptec hardware never did DV other than a really simple = algorithm in the BIOS for the U320 cards, and I=E2=80=99m pretty sure = that the Symbios cards pre-date DV all-together. I think that the MPT = controllers do DV in the firmware at boot, and the driver just reads the = results and uses that for the default negotiation settings. I don=E2=80=99= t see much in the MPT driver for debugging this, but I recall that the = DV algorithm that LSI used was pretty naive and overly-aggressive at = forcing down negotiation. You might want to look at what options the = LSI BIOS offers, or if you can use camcontrol to change the negotiation = up to where you think it should be. Scott From owner-freebsd-scsi@freebsd.org Wed Jul 29 13:22:51 2015 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FCF09AC022 for ; Wed, 29 Jul 2015 13:22:51 +0000 (UTC) (envelope-from dan@langille.org) Received: from clavin2.langille.org (clavin2.langille.org [199.233.228.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "clavin.langille.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DB96132D for ; Wed, 29 Jul 2015 13:22:51 +0000 (UTC) (envelope-from dan@langille.org) Received: from (clavin2.int.langille.org (clavin2.int.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) with ESMTPSA id 9D569AE52 ; Wed, 29 Jul 2015 13:22:43 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: dmesg output not as expected From: Dan Langille In-Reply-To: <27741F67-95F8-4EEF-B42F-E80626799FCB@yahoo.com> Date: Wed, 29 Jul 2015 09:22:42 -0400 Cc: freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0B4EA0B0-5A90-4D12-98F3-142DF829C086@langille.org> References: <5A302DC6-C3EF-4DB1-8AD1-A0314C048A28@langille.org> <27741F67-95F8-4EEF-B42F-E80626799FCB@yahoo.com> To: Scott Long X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jul 2015 13:22:51 -0000 > On Jul 28, 2015, at 11:56 PM, Scott Long wrote: >=20 >=20 >> On Jul 28, 2015, at 8:18 PM, Dan Langille wrote: >>=20 >> After rebooting this server, I see this in /var/run/dmesg. Why does = it change? >>=20 >> ch0 at mpt0 bus 0 scbus14 target 3 lun 0 >> ch0: Removable Changer SCSI-2 device >> ch0: Serial Number 3G22JJP38S46 >> ch0: 20.000MB/s transfers (10.000MHz DT, offset 15, 16bit) >> ch0: 25 slots, 1 drive, 1 picker, 1 portal >> ch0: quirks=3D0x2 >>=20 >> I was hoping to get more through put on this newer system with a = different card. =46rom what I see, my speed is only half of what it = was. >>=20 >> I'm using a PCI-E 3.0 X8 (in X16) slot. Have I missing something? >>=20 >=20 > The LSI controllers do Domain Validation. That means that they send a = series of test commands to each target to try to determine a safe speed = to negotiate to. The theory is that these test commands will determine = whether the target=E2=80=99s speed needs to be downshifted in order to = avoid problems with bad cables and bad connectors. It=E2=80=99s part of = the Ultra320 spec, and I think it was in the Ultra160 spec too, but was = rarely implemented there. It=E2=80=99s possible that the DV routine is = telling the card that these devices need to have their negotiated speed = reduced, and that the results it gets are a bit on-the-fence at times = and cause different observed speeds. >=20 > It=E2=80=99s been a long time since I worked with DV, so my memory is = a bit fuzzy. The Adaptec hardware never did DV other than a really = simple algorithm in the BIOS for the U320 cards, and I=E2=80=99m pretty = sure that the Symbios cards pre-date DV all-together. I think that the = MPT controllers do DV in the firmware at boot, and the driver just reads = the results and uses that for the default negotiation settings. I = don=E2=80=99t see much in the MPT driver for debugging this, but I = recall that the DV algorithm that LSI used was pretty naive and = overly-aggressive at forcing down negotiation. You might want to look = at what options the LSI BIOS offers, or if you can use camcontrol to = change the negotiation up to where you think it should be. I went into the BIOS and took photos of what I found. They seem to be = set to optimal values. Do you see anything I should investigate further? These photos are not necessarily in any particularly order. https://dl.dropboxusercontent.com/u/24218593/LSI-BIOS-1.jpeg https://dl.dropboxusercontent.com/u/24218593/LSI-BIOS-2.jpeg https://dl.dropboxusercontent.com/u/24218593/LSI-BIOS-3.jpeg https://dl.dropboxusercontent.com/u/24218593/LSI-BIOS-4.jpeg =E2=80=94=20 Dan Langille http://langille.org/