From owner-freebsd-geom@FreeBSD.ORG Mon Mar 2 08:25:48 2015 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A1089A4A for ; Mon, 2 Mar 2015 08:25:48 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 87D14E85 for ; Mon, 2 Mar 2015 08:25:48 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t228PmDT040770 for ; Mon, 2 Mar 2015 08:25:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-geom@FreeBSD.org Subject: [Bug 158197] [geom] geom_cache with size>1000 leads to panics Date: Mon, 02 Mar 2015 08:25:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: eugen@grosbein.net X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-geom@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Mar 2015 08:25:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=158197 --- Comment #8 from eugen@grosbein.net --- Today I had the same panic with 8.4-STABLE and single gcache instance serving disk slice containing the system. Unfortunally, crashdump was not generated but I have screenshot with KDB_TRACE: http://www.grosbein.net/img/panic-gcache.jpg After reboot: # gcache list Geom name: ca0 WroteBytes: 0 Writes: 0 CacheFull: 0 CacheMisses: 20 CacheHits: 88 CacheReadBytes: 324096 CacheReads: 108 ReadBytes: 328192 Reads: 116 InvalidEntries: 0 UsedEntries: 9 Entries: 9 TailOffset: 500107182080 BlockSize: 131072 Size: 1024 Providers: 1. Name: cache/ca0 Mediasize: 500107280896 (465G) Sectorsize: 512 Mode: r0w0e0 Consumers: 1. Name: mirror/gm0s1 Mediasize: 500107281408 (465G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 258048 Mode: r0w0e0 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 12:52:29 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49D8F8D1 for ; Wed, 4 Mar 2015 12:52:29 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 021CEEAA for ; Wed, 4 Mar 2015 12:52:28 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id DD2FE30FE34; Wed, 4 Mar 2015 13:44:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:content-type:content-transfer-encoding; s=20140709; bh=EU1VUyKwzGiqBDCuEaQiFmLQq8Y=; b=UR/hzZ7zjznzhEvAGRZvxBcQD3kw74u4NzdaS6B+tHu10CLCfVmOoPwFaMLJ3j0Lo4CaoChpUxSrPqvehIFMy2YEKsHg89Gtt/hWjSxiUauGlWWR0V8kGd+IWfOz9NFosem8RtfQiHKMPBP6gEABpDAIbP09RZCgq4BtzisCARo= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id ADFEB30FE3C for ; Wed, 4 Mar 2015 13:44:30 +0100 (CET) Received: (qmail 78151 invoked by uid 89); 4 Mar 2015 13:44:30 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 4 Mar 2015 13:44:30 +0100 Message-ID: <54F6FE2E.60303@bytecamp.net> Date: Wed, 04 Mar 2015 13:44:30 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Trim on gmirrored SSDs is slow and results in inresponsive system Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 12:52:29 -0000 Hi, I just want to point to the bugreport of a problem we and at least another one discovered. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197516 Maybe this behaviour could be confirmed by others and narrowed down. with kind regards, Robert Schulze From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 13:25:38 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E7D65D2 for ; Wed, 4 Mar 2015 13:25:38 +0000 (UTC) Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C74D23D for ; Wed, 4 Mar 2015 13:25:38 +0000 (UTC) Received: by widem10 with SMTP id em10so30538428wid.5 for ; Wed, 04 Mar 2015 05:25:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ybnjhxuFojMiW6S7gbUDcRSdmnD8Ymkt6SEKTOmlo8I=; b=HCixWGOB/RvqSsMhR0iXOPYsD03rNMZZxgYkrp42kkuWlJ2a5oQt19/YtTixXUz1kP bXngPBPt6MX7mGtkTg9KOX3OJuoLhS41E60sBu2YrKEsy1wcQ41CebRRegqrPj5VoiQl EeTq16UeiM07q1YEFlDI1Pglga/W+6Co7Uw15oAC37f7aB4U9n2rL1KUImJ0BAtd5AjP 8mb1U1iJBv35bpVvy/2TDbJRgqo6AyB7bXWsO6VOsgGyxsV+FjIB5ybAlUwzgXBo8fBv FWgsvNgoH6pHHPfUhszNZAW3iV3OIKTBhe8oJrh+GeuTIkN1y2MxBNc6fsoQsgAGTK/H IivQ== X-Gm-Message-State: ALoCoQlwORojUmsFJxoQC7jY8MzkoxLm4x6N70UcT+9EUOU2Cl/sVHluRRfMtQEdQLxtG/j2N8pS X-Received: by 10.194.185.68 with SMTP id fa4mr7565437wjc.111.1425475536237; Wed, 04 Mar 2015 05:25:36 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id ka1sm5999936wjc.2.2015.03.04.05.25.34 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Mar 2015 05:25:35 -0800 (PST) Message-ID: <54F707CC.6070105@multiplay.co.uk> Date: Wed, 04 Mar 2015 13:25:32 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> In-Reply-To: <54F6FE2E.60303@bytecamp.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 13:25:39 -0000 Do you have one disk which has really slow TRIM? What's the output for: camcontrol identify grep quirks /var/run/dmesg.boot sysctl kern.cam |grep sort_io_queue Also how does gstat -d -p compare between gmirror and none gmirror installs on the same machine? Do you see high kernel CPU when the deletes are happening? Regards Steve On 04/03/2015 12:44, Robert Schulze wrote: > Hi, > > I just want to point to the bugreport of a problem we and at least > another one discovered. > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197516 > > Maybe this behaviour could be confirmed by others and narrowed down. > > with kind regards, > Robert Schulze > _______________________________________________ > freebsd-geom@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-geom > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 14:19:46 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49BE6B3E for ; Wed, 4 Mar 2015 14:19:46 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C6927A17 for ; Wed, 4 Mar 2015 14:19:45 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id D4F8830FDA6; Wed, 4 Mar 2015 15:19:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=3nmuOf0xRjNuixb/Mq9OD9WXCMA=; b=Ufh4nRaOLGs6RfSwmOm6663pBQQ+oWoDAmLXnzChnXCemsrP7pm2AxlMYiN+TXMYXeYveel4KrtYQ87t9WRRwBBFQNhotZW675hkPNwIJ7eUz96zMGWt9aSNeHx587TU7JmsZTnsrts0lEur5dxSKmxlGlWpcySQ/p98pncI2TU= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id 8D28630FD99 for ; Wed, 4 Mar 2015 15:19:43 +0100 (CET) Received: (qmail 2513 invoked by uid 89); 4 Mar 2015 15:19:43 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 4 Mar 2015 15:19:43 +0100 Message-ID: <54F7147F.8070206@bytecamp.net> Date: Wed, 04 Mar 2015 15:19:43 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> In-Reply-To: <54F707CC.6070105@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 14:19:46 -0000 Hi, Am 04.03.2015 um 14:25 schrieb Steven Hartland: > Do you have one disk which has really slow TRIM? > please note, it does not depend on the disk, it is related to gmirror. > What's the output for: > camcontrol identify # camcontrol identify ada0 pass1: ATA-9 SATA 3.x device pass1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 512bytes) protocol ATA/ATAPI-9 SATA 3.x device model INTEL SSDSC2BB240G4 firmware revision D2010370 serial number BTWL404203MN240NGN WWN 55cd2e404b586c42 cylinders 16383 heads 16 sectors/track 63 sector size logical 512, physical 4096, offset 0 LBA supported 268435455 sectors LBA48 supported 468862128 sectors PIO supported PIO4 DMA supported WDMA2 UDMA6 media RPM non-rotating Feature Support Enabled Value Vendor read ahead yes yes write cache yes yes flush cache yes yes overlap no Tagged Command Queuing (TCQ) no no Native Command Queuing (NCQ) yes 32 tags NCQ Queue Management no NCQ Streaming no Receive & Send FPDMA Queued no SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management no no automatic acoustic management no no media status notification no no power-up in Standby no no write-read-verify no no unload yes yes general purpose logging yes yes free-fall no no Data Set Management (DSM/TRIM) yes DSM - max 512byte blocks yes 4 DSM - deterministic read yes zeroed Host Protected Area (HPA) yes no 468862128/468862128 HPA - Security no > grep quirks /var/run/dmesg.boot results in empty output, same as dmesg|grep quirks > sysctl kern.cam |grep sort_io_queue kern.cam.sort_io_queues: 1 kern.cam.ada.0.sort_io_queue: 0 kern.cam.ada.1.sort_io_queue: 0 kern.cam.da.0.sort_io_queue: -1 > > Also how does gstat -d -p compare between gmirror and none gmirror > installs on the same machine? Deleting on a non-mirrored UFS does not influence the system, because the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB file), with a high number of d/s (~ 100k). > Do you see high kernel CPU when the deletes are happening? no. Here are the top system processes after deleting a 5GB file: 0:40 144.87% cam 0:22 77.59% g_mirror var 0:14 62.79% bufdaemon 0:14 0.88% intr 0:13 0.59% geom 0:01 0.00% rand_harvestq 0:00 0.00% kernel with kind regards, Robert Schulze From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 14:38:29 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2068E11E for ; Wed, 4 Mar 2015 14:38:29 +0000 (UTC) Received: from mail-we0-f175.google.com (mail-we0-f175.google.com [74.125.82.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A97DCC6E for ; Wed, 4 Mar 2015 14:38:28 +0000 (UTC) Received: by wevl61 with SMTP id l61so46807850wev.2 for ; Wed, 04 Mar 2015 06:38:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=SPTnpfx7hJzgoe1hfH6WS9kNxmienLNU3eQ3WyoUGMM=; b=ZGaABhWmUbQLY2Sj4DpbZQnA6DnRTOtczLH2GAYxZcOId/LMqMfS6lguOKND13Dhtz KJPEcsef0dexyOMy1sw0ER8/vvRKidceyeKbXnzAwpZTBXtsT/JLu2T7kEzIG1GZfcI5 IHMXpigx/ZBrY/JifEPYD/Z1PoTfoOmos6KUIui3LfGsyPNoIYHT2GETfVQHf99JWywt IeuJwDlOHdRJMpCiLi8AzmkmSt8XvHYzle3A8cczIDWyNCnvbaDpdq+Fx9L3Z9+KCi4Y QebTaaTBybb1HSledE8sxQ9TBs/s9DGEWw5fVjiHOcfKSvkAc1xrRoqFxY+LwJMRjWVG 7vTw== X-Gm-Message-State: ALoCoQl8G2Y+H1D4vuIfpeR4cHWuGO5AE2i2ZVf4AAVN8vXz5kSLq8wGCABhCnWG6L/EG1GfV+2y X-Received: by 10.194.193.9 with SMTP id hk9mr9090474wjc.42.1425479901152; Wed, 04 Mar 2015 06:38:21 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id m9sm25451960wiz.24.2015.03.04.06.38.20 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Mar 2015 06:38:20 -0800 (PST) Message-ID: <54F718D9.1080201@multiplay.co.uk> Date: Wed, 04 Mar 2015 14:38:17 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> In-Reply-To: <54F7147F.8070206@bytecamp.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 14:38:29 -0000 On 04/03/2015 14:19, Robert Schulze wrote: > Hi, > > Am 04.03.2015 um 14:25 schrieb Steven Hartland: >> Do you have one disk which has really slow TRIM? >> > please note, it does not depend on the disk, it is related to gmirror. Yes but gmirror will wait for all disks to complete the delete before the top level delete is marked as complete, so if you have an issue with one of the underlying devices, be that cabling issue or disk on the way out, this could be your issue. >> Also how does gstat -d -p compare between gmirror and none gmirror >> installs on the same machine? > Deleting on a non-mirrored UFS does not influence the system, because > the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB > file), with a high number of d/s (~ 100k). Can you get snapshot of both in action please > >> Do you see high kernel CPU when the deletes are happening? > no. Here are the top system processes after deleting a 5GB file: > > 0:40 144.87% cam > 0:22 77.59% g_mirror var > 0:14 62.79% bufdaemon > 0:14 0.88% intr > 0:13 0.59% geom > 0:01 0.00% rand_harvestq > 0:00 0.00% kernel 144% in cam is very high and g_mirror is high too. What OS build is this? http://svnweb.freebsd.org/changeset/base/267118 could well be the cause of high CAM CPU, this was one of main reasons why the original bypass or sorting was added. Regards Steve From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 15:16:14 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9095ECD0 for ; Wed, 4 Mar 2015 15:16:14 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33E9D13C for ; Wed, 4 Mar 2015 15:16:13 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 4CB1D30FE3D; Wed, 4 Mar 2015 16:16:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=M42okwipkT9gu0ivbS6RMPDfP9s=; b=yOQy0Xat7YHMOuTkplXv7oZIuC8trhNaJtxZuj2+y2L46tUsRzjK6KCelmjxW8axqaqmjYZp/NAsVNuS5nP3gkJV5IzMTFceVooa92SuclAdxB6BeSInx3/M+zcpJQ1W/JkJYu23Wwoc10iO8BevCyf63as+hxPCSrkW/fXJo3I= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id 1FF4630FE2F for ; Wed, 4 Mar 2015 16:16:12 +0100 (CET) Received: (qmail 31551 invoked by uid 89); 4 Mar 2015 16:16:12 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 4 Mar 2015 16:16:12 +0100 Message-ID: <54F721BC.5070204@bytecamp.net> Date: Wed, 04 Mar 2015 16:16:12 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> In-Reply-To: <54F718D9.1080201@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 15:16:14 -0000 Hi, Am 04.03.2015 um 15:38 schrieb Steven Hartland: >>> Do you have one disk which has really slow TRIM? >>> >> please note, it does not depend on the disk, it is related to gmirror. > Yes but gmirror will wait for all disks to complete the delete before > the top level delete is marked as complete, so if you have an issue with > one of the underlying devices, be that cabling issue or disk on the way > out, this could be your issue. Both disks seem to be OK, see further down. >>> Also how does gstat -d -p compare between gmirror and none gmirror >>> installs on the same machine? >> Deleting on a non-mirrored UFS does not influence the system, because >> the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB >> file), with a high number of d/s (~ 100k). > Can you get snapshot of both in action please I've collected data via $ iostat -w0.5 ada0 ada1 directly after unlinking a 2 GB file for all possible cases: 1) non mirrored on ada0: tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 154 111.45 241 26.20 14.73 12 0.17 0 0 5 0 95 0 662 32.00 122437 3826.17 0.00 0 0.00 0 0 7 0 92 0 124 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 2) non mirrored on ada1: tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 3 2876 12.78 4 0.05 108.38 190 20.14 0 0 4 0 96 0 672 0.00 0 0.00 32.00 120446 3763.94 0 0 8 0 92 0 430 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 3) gmirrored with ada0+ada1: tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 7 174 110.65 927 100.13 126.38 728 89.82 0 0 3 0 97 0 684 32.00 5629 175.90 32.00 5629 175.90 0 0 26 0 74 0 459 32.00 3587 112.08 32.00 3615 112.96 0 0 22 0 78 0 393 32.00 2192 68.49 32.00 2169 67.79 0 0 21 0 79 0 346 32.00 2102 65.67 32.00 2100 65.61 0 0 20 0 80 0 338 32.00 5391 168.47 32.00 4793 149.79 0 0 20 0 80 0 391 32.00 4461 139.39 32.00 5735 179.22 0 0 23 0 77 0 438 32.00 3008 93.99 32.00 2374 74.19 0 0 25 0 75 0 406 32.00 2141 66.90 32.00 2100 65.61 0 0 24 0 76 0 367 32.00 1825 57.04 32.00 1825 57.04 0 0 23 0 77 0 340 32.00 1835 57.33 32.00 1806 56.45 0 0 25 0 75 0 321 32.00 1517 47.40 32.00 1485 46.41 0 0 23 0 77 0 289 32.00 1376 43.00 32.00 1385 43.29 0 0 27 0 73 0 273 32.00 1271 39.71 32.00 1291 40.35 0 0 23 0 77 0 267 32.00 1220 38.12 32.00 1197 37.42 0 0 23 0 77 0 314 32.00 5665 177.04 32.00 5196 162.36 0 0 20 0 80 0 412 31.98 5568 173.88 31.98 6147 191.98 0 0 22 0 78 0 492 32.00 5530 172.81 32.00 5426 169.57 0 0 23 0 77 0 459 32.00 2505 78.30 32.00 2549 79.65 0 0 27 0 73 0 393 32.00 1940 60.62 32.00 1942 60.68 0 0 22 0 78 0 367 32.00 1882 58.80 32.00 1883 58.86 0 0 19 0 81 0 379 32.00 5076 158.63 32.00 5477 171.15 0 0 20 0 80 0 611 32.00 4432 138.51 32.00 3951 123.48 0 0 20 0 80 0 380 32.00 3062 95.69 32.00 2254 70.43 0 0 29 0 71 0 370 32.00 2365 73.90 32.00 1912 59.74 0 0 24 0 76 0 345 32.00 1940 60.62 32.00 1825 57.05 0 0 24 0 76 0 329 32.00 1229 38.40 32.00 2380 74.38 0 0 19 0 81 0 372 32.00 1692 52.89 32.00 1587 49.59 0 0 26 0 74 0 415 32.00 1588 49.64 32.00 1496 46.74 0 0 25 0 75 0 346 32.00 1456 45.50 32.00 1429 44.66 0 0 26 0 74 0 346 32.00 1355 42.33 32.00 1283 40.10 0 0 24 0 76 0 356 32.00 1155 36.09 32.00 1501 46.92 0 0 25 0 75 0 385 32.00 1177 36.78 32.00 1091 34.08 0 0 23 0 77 0 393 32.00 1805 56.39 32.00 1823 56.98 0 0 24 0 76 0 365 32.00 2021 63.15 32.00 2064 64.50 0 0 26 0 74 0 344 32.00 1868 58.39 32.00 1938 60.56 0 0 25 0 75 0 380 32.00 1897 59.27 32.00 1936 60.50 0 0 27 0 73 0 344 32.00 1774 55.45 32.00 1778 55.57 0 0 23 0 77 0 334 32.00 1674 52.31 32.00 1693 52.91 0 0 25 0 75 0 329 32.00 2254 70.43 32.00 2428 75.89 0 0 19 0 81 0 406 32.00 6331 197.85 32.00 6337 198.02 0 0 24 0 76 0 408 32.00 2966 92.69 32.00 2925 91.40 0 0 25 0 75 0 353 32.00 2280 71.25 32.00 2167 67.73 0 0 25 0 75 0 575 32.00 5051 157.84 32.00 4844 151.38 0 0 18 0 82 0 421 32.00 3801 118.79 32.00 4171 130.33 0 0 23 0 77 0 397 32.00 466 14.56 32.00 470 14.68 0 0 3 0 97 0 344 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 4) gmirrored with only one component active (ada0) tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 132 124.10 176 21.34 15.00 10 0.15 0 0 4 0 96 0 643 32.00 50662 1583.18 0.00 0 0.00 0 0 19 0 81 0 458 32.00 11102 346.93 0.00 0 0.00 0 0 18 0 82 0 287 32.00 4051 126.58 0.00 0 0.00 0 0 17 0 83 0 244 32.00 3500 109.38 0.00 0 0.00 0 0 18 0 82 0 256 32.00 3358 104.93 0.00 0 0.00 0 0 19 0 81 0 244 32.00 3010 94.06 0.00 0 0.00 0 0 18 0 82 0 260 32.00 2710 84.68 0.00 0 0.00 0 0 18 0 82 0 257 32.00 2524 78.88 0.00 0 0.00 0 0 18 0 82 0 234 32.00 2345 73.29 0.00 0 0.00 0 0 17 0 82 0 245 32.00 2185 68.28 0.00 0 0.00 0 0 19 0 81 0 239 32.00 2072 64.73 0.00 0 0.00 0 0 17 0 82 0 244 32.00 1970 61.56 0.00 0 0.00 0 0 18 0 81 0 243 32.00 1878 58.70 0.00 0 0.00 0 0 18 0 82 0 241 32.00 1799 56.22 0.00 0 0.00 0 0 19 0 81 0 231 32.00 1928 60.25 0.00 0 0.00 0 0 18 0 82 0 254 32.00 2442 76.30 0.00 0 0.00 0 0 15 0 85 0 252 32.00 2457 76.77 0.00 0 0.00 0 0 12 0 88 0 246 32.00 2461 76.89 0.00 0 0.00 0 0 13 0 87 0 235 32.00 2472 77.24 0.00 0 0.00 0 0 12 0 88 0 194 32.00 2491 77.83 0.00 0 0.00 0 0 13 0 87 0 201 32.00 2517 78.65 0.00 0 0.00 0 0 13 0 87 0 429 32.00 2545 79.54 0.00 0 0.00 0 0 12 0 88 0 231 31.99 9419 294.23 0.00 0 0.00 0 0 12 0 88 0 303 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 5) gmirrored with only one component active (ada1) tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 153 11.75 5 0.06 100.91 298 29.40 0 0 5 0 95 0 635 0.00 0 0.00 32.00 31614 987.94 0 0 19 0 81 0 372 0.00 0 0.00 32.00 20236 632.38 0 0 17 0 83 0 331 0.50 2 0.00 32.00 16922 528.74 0 0 15 0 85 0 350 0.00 0 0.00 32.00 20795 649.84 0 0 17 0 83 0 278 0.00 0 0.00 32.00 17748 554.63 0 0 15 0 85 0 276 0.00 0 0.00 32.00 14885 465.16 0 0 8 0 92 0 272 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 >> 0:40 144.87% cam >> 0:22 77.59% g_mirror var >> 0:14 62.79% bufdaemon >> 0:14 0.88% intr >> 0:13 0.59% geom >> 0:01 0.00% rand_harvestq >> 0:00 0.00% kernel > 144% in cam is very high and g_mirror is high too. > > What OS build is this? $ uname -a FreeBSD XXX 10.1-RELEASE-p5 FreeBSD 10.1-RELEASE-p5 #23 r278313M: Fri Feb 13 11:26:23 CET 2015 root@XXX:/usr/obj/usr/src/sys/BC amd64 HTH, Robert Schulze From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 16:09:36 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F0DF9B5 for ; Wed, 4 Mar 2015 16:09:36 +0000 (UTC) Received: from mail-wg0-f44.google.com (mail-wg0-f44.google.com [74.125.82.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 13313939 for ; Wed, 4 Mar 2015 16:09:35 +0000 (UTC) Received: by wggz12 with SMTP id z12so10513451wgg.9 for ; Wed, 04 Mar 2015 08:09:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=31jgqmqjIttOadm/KpQURZl83NV5ByGmQTe/qKXFXB0=; b=dQv8YN4YIDgOM8YF8gI+w7RbkMTxZbXP9P7dX9YEuENdTqKl+jCr2RC34M7yN9YNr6 Wggb7dglLFJKZjWvbTgoO0/AltP8FahjRlNxZO42c51iisK1qtTCriZinQ+2oX9MfQF8 +ym1+MYo0dvMpwnVlRMxGyzLocn1CN315BUM0Dp8pSiUofeJqdZi6NJFe0VE7XmG13Ww frfY4eQ2aCkwC4D3a2XJEIGJ5mRXHWI+888CdKz2RtO1qs70QIrtpd/bcVSIoUMY+bUu 85IK6wRphXwmVaFjkMLKKztI052F0Pe7AN1oRx5xcIIVKa8WoIx3ZdiiXYjUFxWzFZ6d m5sQ== X-Gm-Message-State: ALoCoQkMTEjYktBRUjmCeNWfySQaqocJRfT8EaMf1BSf+2wNtcoTbP4Npuu14BbMZiOxJVY5XEVd X-Received: by 10.180.91.47 with SMTP id cb15mr14043710wib.39.1425485373775; Wed, 04 Mar 2015 08:09:33 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id cf12sm6584192wjb.10.2015.03.04.08.09.32 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Mar 2015 08:09:32 -0800 (PST) Message-ID: <54F72E39.2070105@multiplay.co.uk> Date: Wed, 04 Mar 2015 16:09:29 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net> In-Reply-To: <54F721BC.5070204@bytecamp.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 16:09:36 -0000 On 04/03/2015 15:16, Robert Schulze wrote: > Hi, > > Am 04.03.2015 um 15:38 schrieb Steven Hartland: >>>> Do you have one disk which has really slow TRIM? >>>> >>> please note, it does not depend on the disk, it is related to gmirror. >> Yes but gmirror will wait for all disks to complete the delete before >> the top level delete is marked as complete, so if you have an issue with >> one of the underlying devices, be that cabling issue or disk on the way >> out, this could be your issue. > Both disks seem to be OK, see further down. > >>>> Also how does gstat -d -p compare between gmirror and none gmirror >>>> installs on the same machine? >>> Deleting on a non-mirrored UFS does not influence the system, because >>> the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB >>> file), with a high number of d/s (~ 100k). >> Can you get snapshot of both in action please > I've collected data via > > $ iostat -w0.5 ada0 ada1 > > directly after unlinking a 2 GB file for all possible cases: > > 1) non mirrored on ada0: > > tty ada0 ada1 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 1 154 111.45 241 26.20 14.73 12 0.17 0 0 5 0 95 > 0 662 32.00 122437 3826.17 0.00 0 0.00 0 0 7 0 92 > 0 124 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 > > 2) non mirrored on ada1: > > tty ada0 ada1 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 3 2876 12.78 4 0.05 108.38 190 20.14 0 0 4 0 96 > 0 672 0.00 0 0.00 32.00 120446 3763.94 0 0 8 0 92 > 0 430 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 > > 3) gmirrored with ada0+ada1: > > tty ada0 ada1 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 7 174 110.65 927 100.13 126.38 728 89.82 0 0 3 0 97 > 0 684 32.00 5629 175.90 32.00 5629 175.90 0 0 26 0 74 > 0 459 32.00 3587 112.08 32.00 3615 112.96 0 0 22 0 78 > 0 393 32.00 2192 68.49 32.00 2169 67.79 0 0 21 0 79 > 0 346 32.00 2102 65.67 32.00 2100 65.61 0 0 20 0 80 > 0 338 32.00 5391 168.47 32.00 4793 149.79 0 0 20 0 80 > 0 391 32.00 4461 139.39 32.00 5735 179.22 0 0 23 0 77 > 0 438 32.00 3008 93.99 32.00 2374 74.19 0 0 25 0 75 > 0 406 32.00 2141 66.90 32.00 2100 65.61 0 0 24 0 76 > 0 367 32.00 1825 57.04 32.00 1825 57.04 0 0 23 0 77 > 0 340 32.00 1835 57.33 32.00 1806 56.45 0 0 25 0 75 > 0 321 32.00 1517 47.40 32.00 1485 46.41 0 0 23 0 77 > 0 289 32.00 1376 43.00 32.00 1385 43.29 0 0 27 0 73 > 0 273 32.00 1271 39.71 32.00 1291 40.35 0 0 23 0 77 > 0 267 32.00 1220 38.12 32.00 1197 37.42 0 0 23 0 77 > 0 314 32.00 5665 177.04 32.00 5196 162.36 0 0 20 0 80 > 0 412 31.98 5568 173.88 31.98 6147 191.98 0 0 22 0 78 > 0 492 32.00 5530 172.81 32.00 5426 169.57 0 0 23 0 77 > 0 459 32.00 2505 78.30 32.00 2549 79.65 0 0 27 0 73 > 0 393 32.00 1940 60.62 32.00 1942 60.68 0 0 22 0 78 > 0 367 32.00 1882 58.80 32.00 1883 58.86 0 0 19 0 81 > 0 379 32.00 5076 158.63 32.00 5477 171.15 0 0 20 0 80 > 0 611 32.00 4432 138.51 32.00 3951 123.48 0 0 20 0 80 > 0 380 32.00 3062 95.69 32.00 2254 70.43 0 0 29 0 71 > 0 370 32.00 2365 73.90 32.00 1912 59.74 0 0 24 0 76 > 0 345 32.00 1940 60.62 32.00 1825 57.05 0 0 24 0 76 > 0 329 32.00 1229 38.40 32.00 2380 74.38 0 0 19 0 81 > 0 372 32.00 1692 52.89 32.00 1587 49.59 0 0 26 0 74 > 0 415 32.00 1588 49.64 32.00 1496 46.74 0 0 25 0 75 > 0 346 32.00 1456 45.50 32.00 1429 44.66 0 0 26 0 74 > 0 346 32.00 1355 42.33 32.00 1283 40.10 0 0 24 0 76 > 0 356 32.00 1155 36.09 32.00 1501 46.92 0 0 25 0 75 > 0 385 32.00 1177 36.78 32.00 1091 34.08 0 0 23 0 77 > 0 393 32.00 1805 56.39 32.00 1823 56.98 0 0 24 0 76 > 0 365 32.00 2021 63.15 32.00 2064 64.50 0 0 26 0 74 > 0 344 32.00 1868 58.39 32.00 1938 60.56 0 0 25 0 75 > 0 380 32.00 1897 59.27 32.00 1936 60.50 0 0 27 0 73 > 0 344 32.00 1774 55.45 32.00 1778 55.57 0 0 23 0 77 > 0 334 32.00 1674 52.31 32.00 1693 52.91 0 0 25 0 75 > 0 329 32.00 2254 70.43 32.00 2428 75.89 0 0 19 0 81 > 0 406 32.00 6331 197.85 32.00 6337 198.02 0 0 24 0 76 > 0 408 32.00 2966 92.69 32.00 2925 91.40 0 0 25 0 75 > 0 353 32.00 2280 71.25 32.00 2167 67.73 0 0 25 0 75 > 0 575 32.00 5051 157.84 32.00 4844 151.38 0 0 18 0 82 > 0 421 32.00 3801 118.79 32.00 4171 130.33 0 0 23 0 77 > 0 397 32.00 466 14.56 32.00 470 14.68 0 0 3 0 97 > 0 344 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 > > 4) gmirrored with only one component active (ada0) > > tty ada0 ada1 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 1 132 124.10 176 21.34 15.00 10 0.15 0 0 4 0 96 > 0 643 32.00 50662 1583.18 0.00 0 0.00 0 0 19 0 81 > 0 458 32.00 11102 346.93 0.00 0 0.00 0 0 18 0 82 > 0 287 32.00 4051 126.58 0.00 0 0.00 0 0 17 0 83 > 0 244 32.00 3500 109.38 0.00 0 0.00 0 0 18 0 82 > 0 256 32.00 3358 104.93 0.00 0 0.00 0 0 19 0 81 > 0 244 32.00 3010 94.06 0.00 0 0.00 0 0 18 0 82 > 0 260 32.00 2710 84.68 0.00 0 0.00 0 0 18 0 82 > 0 257 32.00 2524 78.88 0.00 0 0.00 0 0 18 0 82 > 0 234 32.00 2345 73.29 0.00 0 0.00 0 0 17 0 82 > 0 245 32.00 2185 68.28 0.00 0 0.00 0 0 19 0 81 > 0 239 32.00 2072 64.73 0.00 0 0.00 0 0 17 0 82 > 0 244 32.00 1970 61.56 0.00 0 0.00 0 0 18 0 81 > 0 243 32.00 1878 58.70 0.00 0 0.00 0 0 18 0 82 > 0 241 32.00 1799 56.22 0.00 0 0.00 0 0 19 0 81 > 0 231 32.00 1928 60.25 0.00 0 0.00 0 0 18 0 82 > 0 254 32.00 2442 76.30 0.00 0 0.00 0 0 15 0 85 > 0 252 32.00 2457 76.77 0.00 0 0.00 0 0 12 0 88 > 0 246 32.00 2461 76.89 0.00 0 0.00 0 0 13 0 87 > 0 235 32.00 2472 77.24 0.00 0 0.00 0 0 12 0 88 > 0 194 32.00 2491 77.83 0.00 0 0.00 0 0 13 0 87 > 0 201 32.00 2517 78.65 0.00 0 0.00 0 0 13 0 87 > 0 429 32.00 2545 79.54 0.00 0 0.00 0 0 12 0 88 > 0 231 31.99 9419 294.23 0.00 0 0.00 0 0 12 0 88 > 0 303 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 > > 5) gmirrored with only one component active (ada1) > > tty ada0 ada1 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 1 153 11.75 5 0.06 100.91 298 29.40 0 0 5 0 95 > 0 635 0.00 0 0.00 32.00 31614 987.94 0 0 19 0 81 > 0 372 0.00 0 0.00 32.00 20236 632.38 0 0 17 0 83 > 0 331 0.50 2 0.00 32.00 16922 528.74 0 0 15 0 85 > 0 350 0.00 0 0.00 32.00 20795 649.84 0 0 17 0 83 > 0 278 0.00 0 0.00 32.00 17748 554.63 0 0 15 0 85 > 0 276 0.00 0 0.00 32.00 14885 465.16 0 0 8 0 92 > 0 272 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 Transaction sizes look like they are the same but the tps from gmirror is significantly lower. As iostat doesn't properly understand deletes it would still be nice to see that gstat -d -p output as that will also give us queue depth. Try backing out r268816 and see if that has any impact. Having a quick look through the code it looks like there is an accounting issue with sc_writes as it will register deletes on queue but not completion which will prevent g_mirror_idle working correctly. Regards Steve From owner-freebsd-geom@FreeBSD.ORG Thu Mar 5 14:10:27 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 86501128 for ; Thu, 5 Mar 2015 14:10:27 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3A0FED29 for ; Thu, 5 Mar 2015 14:10:26 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 3316E30FE52; Thu, 5 Mar 2015 15:10:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=52OJw8Z8IK8Ll0gBvzpTkIRzN4o=; b=7UBwfcTGbZKf4JUAngKJk04BnjzmP6gyqGE9QuBSTpOHS7rdKaVeXfGdYqUuNseJ5wa6sV3WqJCFUTj78dAsHrH5lsW/Av4fHRPFDwxPuUi6VzTPt14aU5ZNW5xY/L8U/eHiAioxuoTVDH2TRLTBzpKQA1fvOpuZ66hta2av+Ik= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id 05CBA30FE08 for ; Thu, 5 Mar 2015 15:10:17 +0100 (CET) Received: (qmail 40808 invoked by uid 89); 5 Mar 2015 15:10:17 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 5 Mar 2015 15:10:17 +0100 Message-ID: <54F863C9.3070002@bytecamp.net> Date: Thu, 05 Mar 2015 15:10:17 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net> <54F72E39.2070105@multiplay.co.uk> In-Reply-To: <54F72E39.2070105@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2015 14:10:27 -0000 Hi, Am 04.03.2015 um 17:09 schrieb Steven Hartland: > Transaction sizes look like they are the same but the tps from gmirror > is significantly lower. > > As iostat doesn't properly understand deletes it would still be nice to > see that gstat -d -p output as that will also give us queue depth. Since gstat doesnt just output row by row, I had to screenshot a running gstat and transcribe the values. Running gstat -b in a loop doesn't help here, since the system is inresponsive and won't start any program in time during the DELETE ops. So after deleting a 2GB file on a gmirror with two active SSDs I get: # gstat -dpfada -I3s dT: 3.189s w: 3.000s filter: ada L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 3342 0 0 0.0 0 0 0.0 3342 106939 10.2 80.5| ada0 0 3342 0 0 0.0 0 0 0.0 3342 106939 10.0 82.0| ada1 9 2606 0 0 0.0 0 0 0.0 2606 83400 60.8 98.3| ada0 9 2606 0 0 0.0 0 0 0.0 2606 83400 56.9 98.7| ada1 46 1705 0 0 0.0 0 0 0.0 1705 54565 62.5 98.4| ada0 51 1704 0 0 0.0 0 0 0.0 1704 54512 62.9 97.6| ada1 22 1540 0 0 0.0 0 0 0.0 1540 49285 20.9 96.6| ada0 24 1541 0 0 0.0 0 0 0.0 1541 49317 21.8 96.0| ada1 2 1496 0 0 0.0 0 0 0.0 1496 47887 13.3 98.6| ada0 7 1495 0 0 0.0 0 0 0.0 1495 47855 14.1 98.9| ada1 11 1032 0 0 0.0 0 0 0.0 1032 33010 18.2 98.8| ada0 10 1033 0 0 0.0 0 0 0.0 1033 33070 15.4 98.1| ada1 12 775 0 0 0.0 0 0 0.0 775 24798 26.2 97.5| ada0 16 773 0 0 0.0 0 0 0.0 773 24746 29.6 98.5| ada1 65 743 0 0 0.0 0 0 0.0 743 23784 25.4 98.8| ada0 59 746 0 0 0.0 0 0 0.0 746 23887 24.4 99.0| ada1 > > Try backing out r268816 and see if that has any impact. could you please explain what you mean by that? regards, Robert Schulze From owner-freebsd-geom@FreeBSD.ORG Thu Mar 5 14:37:06 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 43BFABD4 for ; Thu, 5 Mar 2015 14:37:06 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB1EBFA4 for ; Thu, 5 Mar 2015 14:37:04 +0000 (UTC) Received: by wggx12 with SMTP id x12so53841589wgg.6 for ; Thu, 05 Mar 2015 06:36:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=JSBMsSnVAved2HaaS2bDJpMi1x0U9xzs2hnfsqeVfRc=; b=FRszq2ND/21s65HdgOHRKqOf/01nOKH386gkYqVAh1eqQ8VQ3RXzMz5aTvUsxsjE0P afO8uh/eNOkBGYCU4XrxmyF+IckOGTseqzWXtd3N/JsQxogKVCPd8jwMWzwaK4dZopiB M7s0nkxAmOo18odHQD8TCx6poBnfSFabs2ucgJtNOp1Y7KdDRotfQokUXmtTZZzARRTs 2JaBvjcb3ZiTbSQhkv4ZW6mnA++S7DkrdJFcrYgVd9vnVk+0nLVnBL1vbck5klxtCs0a mRpXKV2+QHXceTkdXgeX2Jgp3RLopNnmFsPdr/pR4pruz7GI0tWbyMxVbxs3DNqHN1/G brvg== X-Gm-Message-State: ALoCoQkMSS/FjT9faWSCwiq7M7Fp79MhMW5vPWPt96QbvKnW/wxtq1qyk+PjuGtF24dh3IgylNuX X-Received: by 10.180.105.40 with SMTP id gj8mr67714342wib.67.1425566216970; Thu, 05 Mar 2015 06:36:56 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id hv5sm10792300wjb.16.2015.03.05.06.36.55 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Mar 2015 06:36:56 -0800 (PST) Message-ID: <54F86A06.4090301@multiplay.co.uk> Date: Thu, 05 Mar 2015 14:36:54 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net> <54F72E39.2070105@multiplay.co.uk> <54F863C9.3070002@bytecamp.net> In-Reply-To: <54F863C9.3070002@bytecamp.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2015 14:37:06 -0000 On 05/03/2015 14:10, Robert Schulze wrote: > Hi, > > Am 04.03.2015 um 17:09 schrieb Steven Hartland: >> Transaction sizes look like they are the same but the tps from gmirror >> is significantly lower. >> >> As iostat doesn't properly understand deletes it would still be nice to >> see that gstat -d -p output as that will also give us queue depth. > Since gstat doesnt just output row by row, I had to screenshot a running > gstat and transcribe the values. Running gstat -b in a loop doesn't help > here, since the system is inresponsive and won't start any program in > time during the DELETE ops. > > So after deleting a 2GB file on a gmirror with two active SSDs I get: > > # gstat -dpfada -I3s > > dT: 3.189s w: 3.000s filter: ada > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 0 3342 0 0 0.0 0 0 0.0 3342 106939 10.2 80.5| ada0 > 0 3342 0 0 0.0 0 0 0.0 3342 106939 10.0 82.0| ada1 > > 9 2606 0 0 0.0 0 0 0.0 2606 83400 60.8 98.3| ada0 > 9 2606 0 0 0.0 0 0 0.0 2606 83400 56.9 98.7| ada1 > > 46 1705 0 0 0.0 0 0 0.0 1705 54565 62.5 98.4| ada0 > 51 1704 0 0 0.0 0 0 0.0 1704 54512 62.9 97.6| ada1 > > 22 1540 0 0 0.0 0 0 0.0 1540 49285 20.9 96.6| ada0 > 24 1541 0 0 0.0 0 0 0.0 1541 49317 21.8 96.0| ada1 > > 2 1496 0 0 0.0 0 0 0.0 1496 47887 13.3 98.6| ada0 > 7 1495 0 0 0.0 0 0 0.0 1495 47855 14.1 98.9| ada1 > > 11 1032 0 0 0.0 0 0 0.0 1032 33010 18.2 98.8| ada0 > 10 1033 0 0 0.0 0 0 0.0 1033 33070 15.4 98.1| ada1 > > 12 775 0 0 0.0 0 0 0.0 775 24798 26.2 97.5| ada0 > 16 773 0 0 0.0 0 0 0.0 773 24746 29.6 98.5| ada1 > > 65 743 0 0 0.0 0 0 0.0 743 23784 25.4 98.8| ada0 > 59 746 0 0 0.0 0 0 0.0 746 23887 24.4 99.0| ada1 These values look totally reasonable in terms a d/s. How does this compare to the none gmirror case? >> Try backing out r268816 and see if that has any impact. > could you please explain what you mean by that? > Compile a version of the kernel with this change reverted. Details of the change can be seen here. https://svnweb.freebsd.org/base?view=revision&revision=268816 You can download the changes (only ata is relevant in your case) and apply the patch with reverse option. Once you've done that recompile and re-install your kernel. This feels like it might be an request ordering issue. Regards Steve From owner-freebsd-geom@FreeBSD.ORG Thu Mar 5 15:16:29 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D6A00B38 for ; Thu, 5 Mar 2015 15:16:29 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 89EAB6BB for ; Thu, 5 Mar 2015 15:16:29 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 4A1CB30FEA1; Thu, 5 Mar 2015 16:16:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=V7TcLDwBS1QtxLDp9NmI2zMExQs=; b=zOzWa9bmUlgXTf5pIrsiu2/a2AWK2UHSaWZH4r2ObRbfAv0a2Up2zo6c0n6vnkgSxzQdJus3eBdPO+4Lckx9b176YpWT1bCauEoR1W+CohFmTjN3llBcHVTHje2dLg0YDJT45LbTKBGGPXWK4W6hREalEvVwNQqbE1od9xnJXVc= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id 05CBA30FE8D for ; Thu, 5 Mar 2015 16:16:26 +0100 (CET) Received: (qmail 99174 invoked by uid 89); 5 Mar 2015 16:16:26 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 5 Mar 2015 16:16:26 +0100 Message-ID: <54F8734A.40409@bytecamp.net> Date: Thu, 05 Mar 2015 16:16:26 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net> <54F72E39.2070105@multiplay.co.uk> <54F863C9.3070002@bytecamp.net> <54F86A06.4090301@multiplay.co.uk> In-Reply-To: <54F86A06.4090301@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2015 15:16:29 -0000 Hi, > These values look totally reasonable in terms a d/s. > > How does this compare to the none gmirror case? Its barely measurable, but in this case I can start a loop directly after unlink with very small interval. The only measurement with non-null values is: dT: 0.205s w: 0.200s filter: ada L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 0 51537 5 156 2.4 0 0 0.0 51532 1649018 3.8 18.2 ada1 thats all. It takes not even half a second to push the DELETE ops to the SSD in case of non-gmirrored UFS. Again, with a 2 GB file. > Compile a version of the kernel with this change reverted. > > Details of the change can be seen here. > https://svnweb.freebsd.org/base?view=revision&revision=268816 done, but it did not change behaviour. regards, Robert Schulze -- /7\ bytecamp GmbH Geschwister-Scholl-Str. 10, 14776 Brandenburg a.d. Havel HRB15752, Amtsgericht Potsdam, Geschaeftsfuehrer: Bjoern Barnekow, Frank Rosenbaum, Sirko Zidlewitz tel +49 3381 79637-0 werktags 10-12,13-17 Uhr, fax +49 3381 79637-20 mail rs@bytecamp.net, web http://bytecamp.net/ From owner-freebsd-geom@FreeBSD.ORG Thu Mar 5 15:45:36 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7D134995 for ; Thu, 5 Mar 2015 15:45:36 +0000 (UTC) Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E518AAF for ; Thu, 5 Mar 2015 15:45:35 +0000 (UTC) Received: by wivr20 with SMTP id r20so7755607wiv.3 for ; Thu, 05 Mar 2015 07:45:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ixU5L/WnVjXtvhYlh27erMqrbScRiaSJh9IK/hCUD/Q=; b=d4W29K6W/kGP1l1jVRFASL1c3qIHVB4m8aybxE1CqA9f9mPGSWYzg4Gjt/ZmrT44Sm J3BrzCmEifJodxf1FwXrM0QD1PwNo3yEUSOi+IhMvGGvJ+lXLSdldaxFXiYxDu2Gtr3O qIZDi6H4YbV/6ye4KRgGMhBjnbBWz3sz6rUyYpXVO6Q0rfy0LyT1pg85vCuHgFvWGYHF 0jHDPFE57yWe4miBzwW7g/T+Ejjv50hiW5bZuhxszcAH2MCaXRKes1sJ8tidIqY3GQE/ /qgze30Ud7dOjYIJyNw+WE/xrv3d5eN2Jz3d4yWom6HQLPWDe9o67oK5rFfs6D/ZyyUW 1Pgw== X-Gm-Message-State: ALoCoQnjDBeJ/gNCV53GYensSYcBjhkH2Lt5cG64A+em4KUFVz8qL4SHgu5GpqZ7D797ml+STmGM X-Received: by 10.194.120.40 with SMTP id kz8mr20045693wjb.21.1425570328231; Thu, 05 Mar 2015 07:45:28 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id v5sm8979955wiw.24.2015.03.05.07.45.26 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Mar 2015 07:45:27 -0800 (PST) Message-ID: <54F87A14.8070504@multiplay.co.uk> Date: Thu, 05 Mar 2015 15:45:24 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net> <54F72E39.2070105@multiplay.co.uk> <54F863C9.3070002@bytecamp.net> <54F86A06.4090301@multiplay.co.uk> <54F8734A.40409@bytecamp.net> In-Reply-To: <54F8734A.40409@bytecamp.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2015 15:45:36 -0000 On 05/03/2015 15:16, Robert Schulze wrote: > Hi, > >> These values look totally reasonable in terms a d/s. >> >> How does this compare to the none gmirror case? > Its barely measurable, but in this case I can start a loop directly > after unlink with very small interval. The only measurement with > non-null values is: > > dT: 0.205s w: 0.200s filter: ada > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ada0 > 0 51537 5 156 2.4 0 0 0.0 51532 1649018 3.8 18.2 ada1 > > thats all. It takes not even half a second to push the DELETE ops to the > SSD in case of non-gmirrored UFS. Again, with a 2 GB file. That is a high rate, but not inconceivable. >> Compile a version of the kernel with this change reverted. >> >> Details of the change can be seen here. >> https://svnweb.freebsd.org/base?view=revision&revision=268816 > done, but it did not change behaviour. > Thanks, I think we need to do some dtrace instrumentation, to get a better picture of what's happening. If you pick a smaller file, say 10MB what does the following dtrace script report for the single disk no gmirror and for a single disk gmirror (if thats possible) #!/usr/sbin/dtrace -s #pragma D option quiet enum bio_type { BIO_DELETE = 0x04 }; fbt::adastrategy:entry /((struct bio *)arg0)->bio_cmd == BIO_DELETE/ { bp = (struct bio *)arg0; periph = (struct cam_periph *)bp->bio_disk->d_drv1; softc = (struct ada_softc *)periph->softc; printf("%Y ada%d bcount: %d, offset: %d\n", walltimestamp, periph->unit_number, bp->bio_bcount, bp->bio_pblkno); } From owner-freebsd-geom@FreeBSD.ORG Fri Mar 6 12:40:25 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C5CA7FFB for ; Fri, 6 Mar 2015 12:40:25 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 38F2A235 for ; Fri, 6 Mar 2015 12:40:24 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 18CA33150CF; Fri, 6 Mar 2015 13:40:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:references:in-reply-to:content-type; s=20140709; bh=bQLahbEuSzoqLfkTZq/6J+2g6DY=; b=7IMfjVim1GdflVA6XLoAG7m7fJXFRVAAqKLuJSBV+d+9UrIdcPDAykw4q+DG4OrB8SPpWTJZUNExbz28e6qhhjAbv3ENn19IRSHFY8gnA0D549WbEsd9AmQfbZ+bX/ZlNnt/Fd8h8GMEGk9ZE13wKlTYNVQL3VDmGcUPBpgR250= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id D66A43150DB for ; Fri, 6 Mar 2015 13:40:21 +0100 (CET) Received: (qmail 63585 invoked by uid 89); 6 Mar 2015 13:40:21 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 6 Mar 2015 13:40:21 +0100 Message-ID: <54F9A035.50506@bytecamp.net> Date: Fri, 06 Mar 2015 13:40:21 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> <54F721BC.5070204@bytecamp.net> <54F72E39.2070105@multiplay.co.uk> <54F863C9.3070002@bytecamp.net> <54F86A06.4090301@multiplay.co.uk> <54F8734A.40409@bytecamp.net> <54F87A14.8070504@multiplay.co.uk> In-Reply-To: <54F87A14.8070504@multiplay.co.uk> Content-Type: multipart/mixed; boundary="------------050905030609060803090808" X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Mar 2015 12:40:26 -0000 This is a multi-part message in MIME format. --------------050905030609060803090808 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit I accidently did not respond to the list yesterday, so.... Hi, Am 05.03.2015 um 16:45 schrieb Steven Hartland: > Thanks, I think we need to do some dtrace instrumentation, to get a > better picture of what's happening. > > If you pick a smaller file, say 10MB what does the following dtrace > script report for the single disk no gmirror and for a single disk > gmirror (if thats possible) > > #!/usr/sbin/dtrace -s > > #pragma D option quiet > > enum bio_type { > BIO_DELETE = 0x04 > }; > > fbt::adastrategy:entry > /((struct bio *)arg0)->bio_cmd == BIO_DELETE/ > { > bp = (struct bio *)arg0; > periph = (struct cam_periph *)bp->bio_disk->d_drv1; > softc = (struct ada_softc *)periph->softc; > printf("%Y ada%d bcount: %d, offset: %d\n", walltimestamp, > periph->unit_number, bp->bio_bcount, bp->bio_pblkno); > } see the attached files. regards, Robert Schulze --------------050905030609060803090808 Content-Type: text/plain; charset=UTF-8; name="gmirror-ada0" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="gmirror-ada0" 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502464 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502528 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502592 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502656 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502720 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502784 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502848 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501504 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501568 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501632 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501696 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501760 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501824 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501888 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501952 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502016 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502080 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502144 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502208 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502272 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502336 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8502400 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500096 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500160 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500224 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500288 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500352 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500416 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500480 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500544 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500608 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500672 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500736 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500800 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500864 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500928 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500992 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501056 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501120 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501184 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501248 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501312 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501376 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8501440 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497344 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497408 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497472 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497536 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497600 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497664 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497728 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497792 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497856 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497920 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497984 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498048 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498112 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498176 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498240 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498304 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498368 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498432 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498496 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498560 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498624 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498688 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498752 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498816 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498880 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8498944 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499008 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499072 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499136 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499200 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499264 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499328 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499392 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499456 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499520 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499584 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499648 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499712 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499776 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499840 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499904 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8499968 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8500032 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492544 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492608 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492672 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492736 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492800 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492864 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492928 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492992 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493056 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493120 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493184 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493248 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493312 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493376 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493440 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493504 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493568 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493632 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493696 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493760 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493824 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493888 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8493952 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494016 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494080 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494144 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494208 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494272 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494336 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494400 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494464 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494528 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494592 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494656 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494720 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494784 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494848 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494912 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8494976 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495040 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495104 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495168 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495232 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495296 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495360 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495424 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495488 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495552 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495616 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495680 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495744 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495808 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495872 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8495936 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496000 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496064 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496128 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496192 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496256 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496320 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496384 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496448 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496512 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496576 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496640 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496704 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496768 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496832 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496896 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8496960 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497024 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497088 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497152 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497216 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8497280 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484800 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484864 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484928 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484992 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485056 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485120 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485184 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485248 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485312 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485376 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485440 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485504 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485568 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485632 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485696 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485760 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485824 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485888 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8485952 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486016 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486080 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486144 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486208 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486272 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486336 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486400 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486464 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486528 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486592 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486656 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486720 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486784 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486848 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486912 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8486976 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487040 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487104 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487168 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487232 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487296 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487360 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487424 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487488 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487552 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487616 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487680 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487744 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487808 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487872 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8487936 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488000 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488064 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488128 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488192 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488256 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488320 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488384 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488448 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488512 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488576 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488640 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488704 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488768 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488832 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488896 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8488960 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489024 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489088 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489152 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489216 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489280 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489344 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489408 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489472 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489536 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489600 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489664 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489728 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489792 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489856 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489920 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8489984 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490048 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490112 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490176 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490240 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490304 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490368 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490432 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490496 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490560 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490624 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490688 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490752 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490816 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490880 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8490944 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491008 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491072 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491136 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491200 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491264 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491328 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491392 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491456 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491520 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491584 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491648 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491712 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491776 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491840 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491904 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8491968 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492032 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492096 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492160 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492224 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492288 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492352 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492416 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8492480 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482368 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482432 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482496 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482560 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482624 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482688 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482752 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482816 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482880 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8482944 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483008 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483072 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483136 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483200 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483264 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483328 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483392 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483456 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483520 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483584 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483648 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483712 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483776 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483840 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483904 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8483968 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484032 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484096 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484160 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484224 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484288 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484352 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484416 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484480 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484544 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484608 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484672 2015 Mar 5 15:52:40 ada0 bcount: 32768, offset: 8484736 --------------050905030609060803090808 Content-Type: text/plain; charset=UTF-8; name="gmirror-ada0+ada1" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="gmirror-ada0+ada1" 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502464 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502464 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502528 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502528 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502592 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502592 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502656 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502656 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502720 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502720 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502784 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502784 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502848 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502848 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500992 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500992 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501056 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501056 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501120 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501120 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501184 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501184 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501248 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501248 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501312 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501312 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501376 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501376 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501440 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501440 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501504 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501504 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501568 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501568 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501632 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501632 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501696 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501696 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501760 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501760 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501824 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501824 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501888 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501888 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8501952 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8501952 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502016 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502016 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502080 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502080 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502144 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502144 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502208 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502208 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502272 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502272 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502336 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502336 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8502400 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8502400 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497920 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497920 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497984 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497984 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498048 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498048 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498112 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498112 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498176 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498176 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498240 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498240 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498304 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498304 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498368 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498368 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498432 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498432 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498496 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498496 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498560 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498560 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498624 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498624 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498688 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498688 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498752 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498752 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498816 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498816 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498880 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498880 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8498944 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8498944 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499008 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499008 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499072 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499072 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499136 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499136 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499200 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499200 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499264 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499264 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499328 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499328 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499392 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499392 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499456 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499456 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499520 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499520 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499584 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499584 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499648 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499648 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499712 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499712 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499776 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499776 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499840 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499840 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499904 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499904 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8499968 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8499968 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500032 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500032 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500096 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500096 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500160 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500160 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500224 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500224 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500288 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500288 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500352 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500352 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500416 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500416 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500480 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500480 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500544 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500544 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500608 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500608 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500672 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500672 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500736 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500736 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500800 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500800 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500864 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500864 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8500928 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8500928 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488768 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488768 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488832 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488832 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488896 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488896 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488960 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488960 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489024 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489024 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489088 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489088 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489152 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489152 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489216 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489216 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489280 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489280 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489344 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489344 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489408 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489408 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489472 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489472 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489536 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489536 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489600 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489600 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489664 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489664 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489728 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489728 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489792 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489792 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489856 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489856 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489920 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489920 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8489984 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8489984 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490048 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490048 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490112 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490112 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490176 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490176 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490240 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490240 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490304 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490304 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490368 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490368 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490432 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490432 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490496 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490496 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490560 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490560 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490624 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490624 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490688 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490688 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490752 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490752 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490816 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490816 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490880 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490880 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8490944 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8490944 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491008 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491008 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491072 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491072 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491136 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491136 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491200 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491200 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491264 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491264 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491328 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491328 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491392 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491392 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491456 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491456 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491520 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491520 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491584 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491584 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491648 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491648 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491712 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491712 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491776 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491776 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491840 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491840 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491904 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491904 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8491968 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8491968 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492032 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492032 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492096 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492096 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492160 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492160 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492224 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492224 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492288 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492288 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492352 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492352 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492416 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492416 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492480 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492480 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492544 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492544 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492608 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492608 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492672 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492672 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492736 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492736 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492800 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492800 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492864 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492864 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492928 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492928 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8492992 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8492992 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493056 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493056 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493120 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493120 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493184 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493184 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493248 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493248 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493312 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493312 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493376 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493376 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493440 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493440 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493504 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493504 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493568 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493568 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493632 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493632 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493696 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493696 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493760 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493760 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493824 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493824 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493888 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493888 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8493952 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8493952 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494016 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494016 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494080 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494080 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494144 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494144 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494208 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494208 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494272 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494272 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494336 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494336 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494400 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494400 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494464 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494464 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494528 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494528 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494592 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494592 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494656 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494656 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494720 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494720 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494784 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494784 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494848 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494848 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494912 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494912 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8494976 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8494976 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495040 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495040 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495104 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495104 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495168 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495168 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495232 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495232 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495296 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495296 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495360 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495360 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495424 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495424 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495488 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495488 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495552 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495552 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495616 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495616 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495680 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495680 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495744 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495744 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495808 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495808 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495872 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495872 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8495936 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8495936 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496000 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496000 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496064 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496064 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496128 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496128 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496192 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496192 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496256 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496256 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496320 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496320 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496384 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496384 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496448 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496448 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496512 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496512 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496576 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496576 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496640 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496640 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496704 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496704 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496768 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496768 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496832 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496832 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496896 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496896 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8496960 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8496960 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497024 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497024 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497088 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497088 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497152 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497152 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497216 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497216 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497280 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497280 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497344 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497344 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497408 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497408 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497472 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497472 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497536 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497536 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497600 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497600 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497664 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497664 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497728 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497728 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497792 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497792 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8497856 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8497856 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482368 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482368 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482432 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482432 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482496 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482496 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482560 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482560 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482624 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482624 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482688 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482688 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482752 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482752 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482816 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482816 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482880 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482880 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8482944 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8482944 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483008 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483008 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483072 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483072 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483136 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483136 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483200 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483200 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483264 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483264 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483328 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483328 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483392 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483392 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483456 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483456 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483520 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483520 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483584 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483584 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483648 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483648 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483712 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483712 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483776 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483776 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483840 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483840 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483904 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483904 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8483968 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8483968 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484032 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484032 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484096 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484096 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484160 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484160 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484224 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484224 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484288 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484288 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484352 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484352 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484416 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484416 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484480 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484480 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484544 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484544 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484608 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484608 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484672 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484672 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484736 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484736 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484800 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484800 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484864 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484864 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484928 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484928 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8484992 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8484992 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485056 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485056 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485120 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485120 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485184 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485184 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485248 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485248 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485312 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485312 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485376 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485376 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485440 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485440 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485504 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485504 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485568 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485568 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485632 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485632 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485696 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485696 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485760 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485760 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485824 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485824 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485888 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485888 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8485952 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8485952 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486016 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486016 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486080 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486080 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486144 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486144 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486208 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486208 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486272 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486272 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486336 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486336 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486400 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486400 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486464 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486464 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486528 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486528 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486592 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486592 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486656 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486656 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486720 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486720 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486784 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486784 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486848 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486848 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486912 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486912 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8486976 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8486976 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487040 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487040 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487104 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487104 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487168 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487168 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487232 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487232 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487296 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487296 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487360 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487360 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487424 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487424 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487488 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487488 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487552 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487552 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487616 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487616 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487680 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487680 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487744 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487744 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487808 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487808 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487872 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487872 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8487936 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8487936 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488000 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488000 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488064 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488064 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488128 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488128 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488192 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488192 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488256 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488256 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488320 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488320 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488384 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488384 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488448 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488448 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488512 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488512 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488576 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488576 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488640 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488640 2015 Mar 5 15:49:26 ada1 bcount: 32768, offset: 8488704 2015 Mar 5 15:49:26 ada0 bcount: 32768, offset: 8488704 --------------050905030609060803090808 Content-Type: text/plain; charset=UTF-8; name="no-mirror" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="no-mirror" 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483200 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483264 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483328 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483392 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483456 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483520 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483584 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483648 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483712 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483776 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483840 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483904 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483968 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484032 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484096 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484160 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484224 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484288 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484352 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484416 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484480 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484544 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484608 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484672 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484736 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484800 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484864 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484928 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8484992 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485056 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485120 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485184 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485248 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485312 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485376 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485440 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485504 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485568 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485632 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485696 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485760 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485824 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485888 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8485952 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486016 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486080 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486144 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486208 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486272 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486336 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486400 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486464 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486528 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486592 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486656 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486720 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486784 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486848 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486912 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8486976 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487040 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487104 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487168 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487232 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487296 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487360 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487424 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487488 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487552 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487616 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487680 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487744 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487808 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487872 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8487936 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488000 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488064 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488128 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488192 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488256 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488320 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488384 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488448 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488512 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488576 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488640 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488704 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488768 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488832 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488896 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8488960 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489024 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489088 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489152 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489216 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489280 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489344 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489408 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489472 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489536 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489600 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489664 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489728 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489792 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489856 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489920 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8489984 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490048 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490112 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490176 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490240 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490304 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490368 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490432 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490496 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490560 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490624 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490688 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490752 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490816 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490880 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8490944 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491008 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491072 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491136 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491200 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491264 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491328 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491392 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491456 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491520 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491584 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491648 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491712 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491776 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491840 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491904 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8491968 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492032 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492096 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492160 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492224 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492288 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492352 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492416 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492480 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492544 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492608 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492672 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492736 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492800 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492864 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492928 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8492992 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493056 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493120 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493184 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493248 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493312 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493376 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493440 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493504 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493568 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493632 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493696 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493760 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493824 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493888 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8493952 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494016 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494080 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494144 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494208 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494272 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494336 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494400 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494464 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494528 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494592 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494656 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494720 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494784 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494848 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494912 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8494976 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495040 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495104 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495168 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495232 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495296 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495360 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495424 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495488 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495552 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495616 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495680 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495744 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495808 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495872 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8495936 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496000 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496064 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496128 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496192 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496256 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496320 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496384 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496448 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496512 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496576 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496640 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496704 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496768 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496832 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496896 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8496960 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497024 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497088 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497152 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497216 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497280 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497344 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497408 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497472 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497536 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497600 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497664 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497728 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497792 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497856 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497920 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8497984 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498048 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498112 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498176 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498240 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498304 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498368 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498432 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498496 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498560 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498624 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498688 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498752 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498816 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498880 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8498944 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499008 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499072 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499136 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499200 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499264 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499328 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499392 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499456 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499520 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499584 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499648 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499712 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499776 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499840 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499904 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8499968 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500032 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500096 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500160 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500224 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500288 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500352 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500416 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500480 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500544 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500608 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500672 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500736 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500800 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500864 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500928 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8500992 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501056 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501120 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501184 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501248 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501312 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501376 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501440 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501504 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501568 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501632 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501696 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501760 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501824 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501888 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8501952 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502016 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502080 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502144 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502208 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502272 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502336 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502400 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502464 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502528 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502592 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502656 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502720 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502784 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8502848 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483136 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483072 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8483008 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482944 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482880 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482816 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482752 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482688 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482624 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482560 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482496 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482432 2015 Mar 5 15:52:08 ada1 bcount: 32768, offset: 8482368 --------------050905030609060803090808-- From owner-freebsd-geom@FreeBSD.ORG Sat Mar 7 14:11:23 2015 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 11488897 for ; Sat, 7 Mar 2015 14:11:23 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EB3952C7 for ; Sat, 7 Mar 2015 14:11:22 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t27EBMj0012195 for ; Sat, 7 Mar 2015 14:11:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-geom@FreeBSD.org Subject: [Bug 197309] ggatec has been broken since rev 238119 Date: Sat, 07 Mar 2015 14:11:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-geom@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: component keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2015 14:11:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197309 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Component|kern |bin Keywords| |patch Assignee|freebsd-bugs@FreeBSD.org |freebsd-geom@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-geom@FreeBSD.ORG Sat Mar 7 14:39:31 2015 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BECE19C for ; Sat, 7 Mar 2015 14:39:31 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E64FE6E5 for ; Sat, 7 Mar 2015 14:39:30 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t27EdU4k037239 for ; Sat, 7 Mar 2015 14:39:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-geom@FreeBSD.org Subject: [Bug 197309] ggatec has been broken since rev 238119 Date: Sat, 07 Mar 2015 14:39:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: fk@fabiankeil.de X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-geom@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2015 14:39:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197309 fk@fabiankeil.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fk@fabiankeil.de --- Comment #1 from fk@fabiankeil.de --- What are the required steps to reproduce the problem? ggatec and ggated work as expected on all my systems and given that r238119 is from 2012, most of them already have it. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-geom@FreeBSD.ORG Sat Mar 7 14:58:30 2015 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D2EDE6 for ; Sat, 7 Mar 2015 14:58:30 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8352598E for ; Sat, 7 Mar 2015 14:58:30 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t27EwU8b055250 for ; Sat, 7 Mar 2015 14:58:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-geom@FreeBSD.org Subject: [Bug 194681] [geom] a race in gmirror cause kernel panic during shutdown Date: Sat, 07 Mar 2015 14:58:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-geom@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2015 14:58:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194681 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-geom@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-geom@FreeBSD.ORG Sat Mar 7 15:22:41 2015 Return-Path: Delivered-To: freebsd-geom@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 355CD782 for ; Sat, 7 Mar 2015 15:22:41 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BCAAD6E for ; Sat, 7 Mar 2015 15:22:41 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t27FMeV9045554 for ; Sat, 7 Mar 2015 15:22:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-geom@FreeBSD.org Subject: [Bug 196086] [geom] a race in gmirror cause kernel crash during startup Date: Sat, 07 Mar 2015 15:22:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-geom@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2015 15:22:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196086 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-geom@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug.