From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 13:53:38 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3FEF1A74 for ; Sun, 2 Jun 2013 13:53:38 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-we0-x22c.google.com (mail-we0-x22c.google.com [IPv6:2a00:1450:400c:c03::22c]) by mx1.freebsd.org (Postfix) with ESMTP id D085B12EB for ; Sun, 2 Jun 2013 13:53:37 +0000 (UTC) Received: by mail-we0-f172.google.com with SMTP id q56so954461wes.3 for ; Sun, 02 Jun 2013 06:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:message-id:date :to:mime-version:x-mailer; bh=sAGu79NssqFlvu1x0tY4Gcmzn48BixXYnQsY7sKQbik=; b=qlwDEsOUJwViBvp4XRvwhKqLuYBYFKO7wIAFllHWXusVQWaC8y+LCklMtXr0DIMxH+ BlgIAqvHG85X1PfIkAE2Tr+lfihVKQkeZQcEsGjxzJr6YFcaPvgRPZ2X/HJC7L2nrpkA EbUC0wBVJbz44+lgXGjxL+Yd36ob4CM7ZvKA4PhgnyuTeZaoem9SeHQnOeNPvLDxbNdr id4k5RV852SK8bh19AatuZBAQRhlAsGg0JoAmfOt7awXNZotpV4/pq+Y+xq8byhhOTwy 5o8aauWI+SVPfENsmCbrChXcj9LFhjqNes6EK8HzdDTlt2v9QWg6PKRLxcFH+0WYT5Ed ShjQ== X-Received: by 10.180.20.177 with SMTP id o17mr9248411wie.52.1370181216909; Sun, 02 Jun 2013 06:53:36 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id q13sm16714153wie.8.2013.06.02.06.53.35 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 06:53:35 -0700 (PDT) From: Alban Hertroys Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Corrupt GPT header on disk from twa array - fixable? Message-Id: Date: Sun, 2 Jun 2013 15:53:34 +0200 To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) X-Mailer: Apple Mail (2.1503) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 13:53:38 -0000 Hello list, I just replaced my home server and moved the disks from the old one over = to the new one. In the old server, 4 of the disks were connected to a = twa (3Ware 9550) controller, which of course has it's own way of marking = units/volumes on those disks. Before you start yelling at me, yes, of course I made backups ;) [*] The thing is, I have these disks in the new server and I found that I = (to my surprise) I can actually mount them! But, I'm missing a large = part and I am wondering if there's some method to access those last = partitions too. Here's what gpart show says about the problematic disk: # gpart show /dev/ada4 =3D> 34 41942972 ada4 GPT (931G) [CORRUPT] 34 128 1 freebsd-boot (64k) 162 1048448 2 freebsd-ufs (512M) 1048610 6291456 3 freebsd-swap (3.0G) 7340066 1048576 4 freebsd-ufs (512M) 8388642 2097152 5 freebsd-ufs (1.0G) 10485794 31457211 6 freebsd-ufs (15G) 41943005 1 - free - (512B) As you can see, most (about 910GB) of the disk is missing! This disk was = one half of a mirror on the twa controller, which had those disks split = in two again (I don't recall how, perhaps 2 different BSD slices?) I already looked if that part may perhaps have ended up as a different = device. On the old server, fstab was this: # cat /tmp/solfertje/etc/fstab # Device Mountpoint FStype Options Dump Pass# # These are the partitions listed above in gpart /dev/da0p2 / ufs rw 1 1 /dev/da0p3 none swap sw 0 0 /dev/da0p4 /var ufs rw 2 2 /dev/da0p5 /tmp ufs rw 2 2 /dev/da0p6 /usr ufs rw 2 2 # These are missing /dev/da1p1 /home ufs rw 2 2 /dev/da1p2 /media ufs rw 2 2 # These are on a different disk (ada2) /dev/da2p1 /media2 ufs rw 2 2 I don't _really_ need to get to those partitions, but it would be a = comfortable thought if it were possible somehow. [*] The reason I was trying to access those disks anyway is that I = thought I forgot to backup my database tables, but it turns out I had = just misplaced that backup and it has been restored now. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:03:10 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 96964DF4 for ; Sun, 2 Jun 2013 14:03:10 +0000 (UTC) (envelope-from prvs=186543ef20=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 25A261363 for ; Sun, 2 Jun 2013 14:03:09 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004126202.msg for ; Sun, 02 Jun 2013 15:03:03 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sun, 02 Jun 2013 15:03:03 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=186543ef20=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> From: "Steven Hartland" To: "Alban Hertroys" , References: Subject: Re: Corrupt GPT header on disk from twa array - fixable? Date: Sun, 2 Jun 2013 15:02:58 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:03:10 -0000 Does "gpart recover ada4" help at all? Be warned this could edit the partition on the disk and make it worse, but I've had success in the past with it. Regards Steve ----- Original Message ----- From: "Alban Hertroys" To: Sent: Sunday, June 02, 2013 2:53 PM Subject: Corrupt GPT header on disk from twa array - fixable? > Hello list, > > I just replaced my home server and moved the disks from the old one over to the new one. In the old server, 4 of the disks were > connected to a twa (3Ware 9550) controller, which of course has it's own way of marking units/volumes on those disks. > > Before you start yelling at me, yes, of course I made backups ;) [*] > > The thing is, I have these disks in the new server and I found that I (to my surprise) I can actually mount them! But, I'm > missing a large part and I am wondering if there's some method to access those last partitions too. > > Here's what gpart show says about the problematic disk: > > # gpart show /dev/ada4 > => 34 41942972 ada4 GPT (931G) [CORRUPT] > 34 128 1 freebsd-boot (64k) > 162 1048448 2 freebsd-ufs (512M) > 1048610 6291456 3 freebsd-swap (3.0G) > 7340066 1048576 4 freebsd-ufs (512M) > 8388642 2097152 5 freebsd-ufs (1.0G) > 10485794 31457211 6 freebsd-ufs (15G) > 41943005 1 - free - (512B) > > As you can see, most (about 910GB) of the disk is missing! This disk was one half of a mirror on the twa controller, which had > those disks split in two again (I don't recall how, perhaps 2 different BSD slices?) > I already looked if that part may perhaps have ended up as a different device. On the old server, fstab was this: > > # cat /tmp/solfertje/etc/fstab > # Device Mountpoint FStype Options Dump Pass# > > # These are the partitions listed above in gpart > /dev/da0p2 / ufs rw 1 1 > /dev/da0p3 none swap sw 0 0 > /dev/da0p4 /var ufs rw 2 2 > /dev/da0p5 /tmp ufs rw 2 2 > /dev/da0p6 /usr ufs rw 2 2 > > # These are missing > /dev/da1p1 /home ufs rw 2 2 > /dev/da1p2 /media ufs rw 2 2 > > # These are on a different disk (ada2) > /dev/da2p1 /media2 ufs rw 2 2 > > > I don't _really_ need to get to those partitions, but it would be a comfortable thought if it were possible somehow. > > > [*] The reason I was trying to access those disks anyway is that I thought I forgot to backup my database tables, but it turns > out I had just misplaced that backup and it has been restored now. > > Alban Hertroys > -- > If you can't see the forest for the trees, > cut the trees and you'll find there is no forest. > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:12:02 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A20F4FAD for ; Sun, 2 Jun 2013 14:12:02 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qa0-x235.google.com (mail-qa0-x235.google.com [IPv6:2607:f8b0:400d:c00::235]) by mx1.freebsd.org (Postfix) with ESMTP id 696CD13A8 for ; Sun, 2 Jun 2013 14:12:02 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id bs12so1330508qab.5 for ; Sun, 02 Jun 2013 07:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+1rN9gx5SklWjPQnF5NGq6dc5tnOBYjJxZz3H6mthYw=; b=DoG9Hy1aDqqrKiH949PIJR33uppcqYKxTy8xwbIcdNiwjfPlIoOpCweyPr6OQODbmw ZDd7FAzVXlNKiOUwoV3Gx6ioArmHOFKgrWpePnWuZHsiW2uviyGSQtlSHzVN/M8svndd ltPCMafpHOvGyp7hKifIy9kNXWXYuDmcM2CzyNShMAq4d1UJ2yWwX0n9O8ZxL4G7Dbiu XBMeEC2Y3mh1b6WGA9O+TTEbZVF6uXtj+uTOxx1MxhxXZ8BvOMt1G2tFAfEz9HCQi08+ XYEsJ+a+XyA1Ld2Lol3Kyyrbm0YpXTcnSsrLetAIXI4dULPHd6Zs4Sdqw1NcVWze4sVc NFvg== MIME-Version: 1.0 X-Received: by 10.49.120.198 with SMTP id le6mr17672677qeb.59.1370182321973; Sun, 02 Jun 2013 07:12:01 -0700 (PDT) Received: by 10.224.3.72 with HTTP; Sun, 2 Jun 2013 07:12:01 -0700 (PDT) In-Reply-To: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> Date: Sun, 2 Jun 2013 17:12:01 +0300 Message-ID: Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Kimmo Paasiala To: Alban Hertroys Content-Type: text/plain; charset=UTF-8 Cc: freebsd-stable@freebsd.org, Steven Hartland X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:12:02 -0000 On Sun, Jun 2, 2013 at 5:02 PM, Steven Hartland wrote: > Does "gpart recover ada4" help at all? > > Be warned this could edit the partition on the disk and make it worse, but > I've had success in the past with it. > > Regards > Steve > > ----- Original Message ----- From: "Alban Hertroys" > To: > Sent: Sunday, June 02, 2013 2:53 PM > Subject: Corrupt GPT header on disk from twa array - fixable? > > > >> Hello list, >> >> I just replaced my home server and moved the disks from the old one over >> to the new one. In the old server, 4 of the disks were connected to a twa >> (3Ware 9550) controller, which of course has it's own way of marking >> units/volumes on those disks. >> >> Before you start yelling at me, yes, of course I made backups ;) [*] >> >> The thing is, I have these disks in the new server and I found that I (to >> my surprise) I can actually mount them! But, I'm missing a large part and I >> am wondering if there's some method to access those last partitions too. >> >> Here's what gpart show says about the problematic disk: >> >> # gpart show /dev/ada4 >> => 34 41942972 ada4 GPT (931G) [CORRUPT] >> 34 128 1 freebsd-boot (64k) >> 162 1048448 2 freebsd-ufs (512M) >> 1048610 6291456 3 freebsd-swap (3.0G) >> 7340066 1048576 4 freebsd-ufs (512M) >> 8388642 2097152 5 freebsd-ufs (1.0G) >> 10485794 31457211 6 freebsd-ufs (15G) >> 41943005 1 - free - (512B) >> >> As you can see, most (about 910GB) of the disk is missing! This disk was >> one half of a mirror on the twa controller, which had those disks split in >> two again (I don't recall how, perhaps 2 different BSD slices?) >> I already looked if that part may perhaps have ended up as a different >> device. On the old server, fstab was this: >> >> # cat /tmp/solfertje/etc/fstab >> # Device Mountpoint FStype Options Dump Pass# >> >> # These are the partitions listed above in gpart >> /dev/da0p2 / ufs rw 1 1 >> /dev/da0p3 none swap sw 0 0 >> /dev/da0p4 /var ufs rw 2 2 >> /dev/da0p5 /tmp ufs rw 2 2 >> /dev/da0p6 /usr ufs rw 2 2 >> >> # These are missing >> /dev/da1p1 /home ufs rw 2 2 >> /dev/da1p2 /media ufs rw 2 2 >> >> # These are on a different disk (ada2) >> /dev/da2p1 /media2 ufs rw 2 2 >> >> >> I don't _really_ need to get to those partitions, but it would be a >> comfortable thought if it were possible somehow. >> >> >> [*] The reason I was trying to access those disks anyway is that I thought >> I forgot to backup my database tables, but it turns out I had just misplaced >> that backup and it has been restored now. >> >> Alban Hertroys >> -- >> If you can't see the forest for the trees, >> cut the trees and you'll find there is no forest. >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> > > Looking at the gpart(8) output it seems that only 20GBs of the disk is recognized by the disk driver but the GPT table still shows the full capacity 910GB. I'd say that the GPT table is in fact correct and if you can somehow get the disks to be recognized with full capacity they should be usable as they are. What does dmesg(8) say about the disks? -Kimmo From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:17:47 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 06D38315 for ; Sun, 2 Jun 2013 14:17:47 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) by mx1.freebsd.org (Postfix) with ESMTP id 91CB01476 for ; Sun, 2 Jun 2013 14:17:46 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id c10so1936598wiw.7 for ; Sun, 02 Jun 2013 07:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=/5k4cju+HIakM4IHvO5QSHFZUFCG2WWggwW5IpGMWHQ=; b=h9ef9KrWsTBKWSS6sZGCFfr/r4/BWCEzJBtUpaYrztOkAdNIb731aOGlEyHka0so4o Ces4v+aKvAWiDazwskYOGDqdQZvLJxUPF2zMH9o1TvdBRLALQ6yRSvdSxFNbqxophU8c bUNOJyAKWPK1qj3dfDFKoR1Be2WEx414r2oGBywGzU+918KhalCx5BfM+D1+JYzSQ0t4 yDWVlK7O+FQ0Q352gAOJHlTrR0zO2pj4dYFP3f9yVm0+qsjRGHZPslRR5jxtsSnztAAB B9myp4FsZAX2kCSnU5x7Vy4Z5FlhiR8TC/7n7YZCvcP0tFMlKwAM7uqkahLgqcPOwuVk Ra8A== X-Received: by 10.180.160.197 with SMTP id xm5mr9265292wib.63.1370182665815; Sun, 02 Jun 2013 07:17:45 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id fx7sm16811656wic.11.2013.06.02.07.17.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 07:17:44 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Alban Hertroys In-Reply-To: Date: Sun, 2 Jun 2013 16:17:43 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> To: Kimmo Paasiala X-Mailer: Apple Mail (2.1503) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:17:47 -0000 On Jun 2, 2013, at 16:12, Kimmo Paasiala wrote: >=20 > Looking at the gpart(8) output it seems that only 20GBs of the disk is > recognized by the disk driver but the GPT table still shows the full > capacity 910GB. I'd say that the GPT table is in fact correct and if > you can somehow get the disks to be recognized with full capacity they > should be usable as they are. What does dmesg(8) say about the disks? =46rom dmesg: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) ATA-8 SATA 2.x device usbd_setup_device_desc: getting device descriptor at addr 2 failed, = USB_ERR_IOERROR ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada2: Previously was known as ad8 ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada3: Previously was known as ad10 ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: ATA-8 SATA 2.x device usbd_req_re_enumerate: addr=3D2, set address failed! (USB_ERR_IOERROR, = ignored) ada4: 300.000MB/s transfers (SATA 2.x, usbd_setup_device_desc: getting = device descriptor at addr 2 failed, USB_ERR_IOERROR UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada4: Previously was known as ad12 ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 ada5: ATA-8 SATA 3.x device ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada5: Previously was known as ad14 SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 13371081 Hz quality 800 GEOM: ada2: the secondary GPT header is not in the last LBA. GEOM: ada3: the secondary GPT header is not in the last LBA. GEOM_MIRROR: Device mirror/boot launched (2/2). GEOM_MIRROR: Device mirror/swap launched (2/2). GEOM_MIRROR: Device mirror/root launched (2/2). GEOM: ada4: the secondary GPT header is not in the last LBA. GEOM: ada5: the secondary GPT header is not in the last LBA. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:19:04 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A58E0506 for ; Sun, 2 Jun 2013 14:19:04 +0000 (UTC) (envelope-from prvs=186543ef20=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 4981E15BC for ; Sun, 2 Jun 2013 14:19:03 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004126385.msg for ; Sun, 02 Jun 2013 15:19:04 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sun, 02 Jun 2013 15:19:04 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=186543ef20=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <2FAF6B543DC04149B2ADCD1CB7B77D08@multiplay.co.uk> From: "Steven Hartland" To: "Kimmo Paasiala" , "Alban Hertroys" References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> Subject: Re: Corrupt GPT header on disk from twa array - fixable? Date: Sun, 2 Jun 2013 15:19:00 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:19:04 -0000 ----- Original Message ----- From: "Kimmo Paasiala" > Looking at the gpart(8) output it seems that only 20GBs of the disk is > recognized by the disk driver but the GPT table still shows the full > capacity 910GB. I'd say that the GPT table is in fact correct and if > you can somehow get the disks to be recognized with full capacity they > should be usable as they are. What does dmesg(8) say about the disks? What does "camcontrol identify ada4" show? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:30:18 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1500A889 for ; Sun, 2 Jun 2013 14:30:18 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) by mx1.freebsd.org (Postfix) with ESMTP id A0DD81649 for ; Sun, 2 Jun 2013 14:30:17 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id hq7so1927583wib.12 for ; Sun, 02 Jun 2013 07:30:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:x-priority:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=suYlk2njgrIB+LCpNvn/fJg5O2aHNx11tiGRa3BIF3U=; b=tkxyhzd5lJdSGYs5ZRf0RTRjDWgR6lBt3VCN6/+ni/VspWZUcfxdlS+vEPvDY8v7jf VH4y0d1AHySusBgBV3Z4qeBFtdVQnj+jdKpjCVGZ9LEldf+AKxnq+H4Hb+Ivnn5NGYyL WJ1jthFybY6QvGmjShdHeKapf3bsVpA4UjP2sFkqaeJpL5CwdcsUb8v68U9QIK5oJOmU wDYSCc/pWpRNjMI5vuXtgQPqYF+84reoVmQ2NLjEYTLhFPES1k/Nw2sA68qdUNGyDmcD vE4tndn3CCiorzOca788xlYeyCp/YxXCubiWq9GaMz1Q+H9Syfl5y5YQ4YLiBuhPIcaz XOpg== X-Received: by 10.180.206.180 with SMTP id lp20mr9399028wic.41.1370183416797; Sun, 02 Jun 2013 07:30:16 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id ff10sm16899042wib.10.2013.06.02.07.30.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 07:30:15 -0700 (PDT) Subject: Re: Corrupt GPT header on disk from twa array - fixable? Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Content-Type: text/plain; charset=iso-8859-1 From: Alban Hertroys X-Priority: 3 In-Reply-To: <2FAF6B543DC04149B2ADCD1CB7B77D08@multiplay.co.uk> Date: Sun, 2 Jun 2013 16:30:14 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <791A4144-BCFB-4D49-9844-352F45BB6677@gmail.com> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2FAF6B543DC04149B2ADCD1CB7B77D08@multiplay.co.uk> To: "Steven Hartland" X-Mailer: Apple Mail (2.1503) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:30:18 -0000 On Jun 2, 2013, at 16:19, "Steven Hartland" = wrote: > ----- Original Message ----- From: "Kimmo Paasiala" = >=20 >> Looking at the gpart(8) output it seems that only 20GBs of the disk = is >> recognized by the disk driver but the GPT table still shows the full >> capacity 910GB. I'd say that the GPT table is in fact correct and if >> you can somehow get the disks to be recognized with full capacity = they >> should be usable as they are. What does dmesg(8) say about the disks? >=20 > What does "camcontrol identify ada4" show? You guys are asking good questions! I'm learning new stuff already :P # camcontrol identify ada4 pass4: ATA-8 SATA 2.x device pass4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) protocol ATA/ATAPI-8 SATA 2.x device model Hitachi HDS721010CLA332 firmware revision JP4OA39C serial number JP2930HQ0XPH3H WWN 5000cca35dcd0b0d cylinders 16383 heads 16 sectors/track 63 sector size logical 512, physical 512, offset 0 LBA supported 268435455 sectors LBA48 supported 1953525168 sectors PIO supported PIO4 DMA supported WDMA2 UDMA6=20 media RPM 7200 Feature Support Enabled Value Vendor read ahead yes yes write cache yes yes flush cache yes yes overlap no Tagged Command Queuing (TCQ) no no Native Command Queuing (NCQ) yes 32 tags SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management yes no 0/0x00 automatic acoustic management yes no 254/0xFE 128/0x80 media status notification no no power-up in Standby yes no write-read-verify no no unload no no free-fall no no data set management (TRIM) no For good measure: # uname -a FreeBSD solfertje 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 = 09:23:10 UTC 2012 = root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 (I'm building STABLE world as we speak) Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:40:32 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3D88CDF6 for ; Sun, 2 Jun 2013 14:40:32 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-we0-x231.google.com (mail-we0-x231.google.com [IPv6:2a00:1450:400c:c03::231]) by mx1.freebsd.org (Postfix) with ESMTP id CBE031886 for ; Sun, 2 Jun 2013 14:40:31 +0000 (UTC) Received: by mail-we0-f177.google.com with SMTP id m19so207651wev.36 for ; Sun, 02 Jun 2013 07:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:x-priority:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=/onkk5WAGWC5nJpvQ/PUMrpyM+SYhoylOXT0uaDNxrw=; b=u6XqNZfU61wvymDXkv4jt/ggMcpbxq+EkAohdriDOrrPy71IHkmhjq1phOVxy4Rj+V KfIyUY34kZj6Em3Cp6aKCoLhPV64HgKb78u7V6zDzcGDU2Xg5Zi/nJW68BK/AfAfULcD kQ6W+Infya+NI7UszXD7YoaWRK3zntuPG7ZwPMAbEI4kiynbQNgzZVA9jhEhMsO2f3KV z1kflVqBzxN66/rVz+LVXbUQjDkprvFqQGAAY5BaZNkukJafJeQwuG6oDRxQn9L5dl9A 6pqf1Ra4PK2Oc17ULcXcDoTuW65f6Du30mVtkcrJMjN88azIpHMXrBG1j3feqfsuw8FS l5sw== X-Received: by 10.180.187.170 with SMTP id ft10mr6449385wic.51.1370184030909; Sun, 02 Jun 2013 07:40:30 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id en3sm17033327wid.1.2013.06.02.07.40.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 07:40:30 -0700 (PDT) Subject: Re: Corrupt GPT header on disk from twa array - fixable? Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Content-Type: text/plain; charset=iso-8859-1 From: Alban Hertroys X-Priority: 3 In-Reply-To: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> Date: Sun, 2 Jun 2013 16:40:28 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.1503) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:40:32 -0000 On Jun 2, 2013, at 16:02, Steven Hartland = wrote: > Does "gpart recover ada4" help at all? >=20 > Be warned this could edit the partition on the disk and make it worse, = but I've had success in the past with it. I applied gpart recover to the other half of what was originally a = mirror and it marked the missing parts as empty - which they weren't, so = I'm glad I have two! (wouldn't have tried otherwise) I expected something like that would happen. The partition table only = looks corrupt when you look at it when it's not connected to the raid = controller. > ----- Original Message ----- From: "Alban Hertroys" = > To: > Sent: Sunday, June 02, 2013 2:53 PM > Subject: Corrupt GPT header on disk from twa array - fixable? >=20 >=20 >> Hello list, >>=20 >> I just replaced my home server and moved the disks from the old one = over to the new one. In the old server, 4 of the disks were connected to = a twa (3Ware 9550) controller, which of course has it's own way of = marking units/volumes on those disks. >>=20 >> Before you start yelling at me, yes, of course I made backups ;) [*] >>=20 >> The thing is, I have these disks in the new server and I found that I = (to my surprise) I can actually mount them! But, I'm missing a large = part and I am wondering if there's some method to access those last = partitions too. >>=20 >> Here's what gpart show says about the problematic disk: >>=20 >> # gpart show /dev/ada4 >> =3D> 34 41942972 ada4 GPT (931G) [CORRUPT] >> 34 128 1 freebsd-boot (64k) >> 162 1048448 2 freebsd-ufs (512M) >> 1048610 6291456 3 freebsd-swap (3.0G) >> 7340066 1048576 4 freebsd-ufs (512M) >> 8388642 2097152 5 freebsd-ufs (1.0G) >> 10485794 31457211 6 freebsd-ufs (15G) >> 41943005 1 - free - (512B) >>=20 >> As you can see, most (about 910GB) of the disk is missing! This disk = was one half of a mirror on the twa controller, which had those disks = split in two again (I don't recall how, perhaps 2 different BSD slices?) >> I already looked if that part may perhaps have ended up as a = different device. On the old server, fstab was this: >>=20 >> # cat /tmp/solfertje/etc/fstab >> # Device Mountpoint FStype Options Dump Pass# >>=20 >> # These are the partitions listed above in gpart >> /dev/da0p2 / ufs rw 1 1 >> /dev/da0p3 none swap sw 0 0 >> /dev/da0p4 /var ufs rw 2 2 >> /dev/da0p5 /tmp ufs rw 2 2 >> /dev/da0p6 /usr ufs rw 2 2 >>=20 >> # These are missing >> /dev/da1p1 /home ufs rw 2 2 >> /dev/da1p2 /media ufs rw 2 2 >>=20 >> # These are on a different disk (ada2) >> /dev/da2p1 /media2 ufs rw 2 2 >>=20 >>=20 >> I don't _really_ need to get to those partitions, but it would be a = comfortable thought if it were possible somehow. >>=20 >>=20 >> [*] The reason I was trying to access those disks anyway is that I = thought I forgot to backup my database tables, but it turns out I had = just misplaced that backup and it has been restored now. >>=20 >> Alban Hertroys >> -- >> If you can't see the forest for the trees, >> cut the trees and you'll find there is no forest. >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > This e.mail is private and confidential between Multiplay (UK) Ltd. = and the person or entity to whom it is addressed. In the event of = misdirection, the recipient is prohibited from using, copying, printing = or otherwise disseminating it or any information contained in it.=20 > In the event of misdirection, illegible or incomplete transmission = please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. >=20 Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:45:42 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 002F2AD5 for ; Sun, 2 Jun 2013 14:45:41 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) by mx1.freebsd.org (Postfix) with ESMTP id 8EB871C77 for ; Sun, 2 Jun 2013 14:45:41 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id j13so2444949wgh.33 for ; Sun, 02 Jun 2013 07:45:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; bh=lERlOpXxMPh/+s3h+pbak623Du1uRe6S2F8z7J1ViNw=; b=GknRov0oPTE9fg6+uIzRgDxPWLr1S6WqEYoiAn7eKo1XGRJUtxYxjOKQr1KhL+oiCT mcAShY52KWioPTebRUiNQ97YeqXp4OH5fnfC33aOXio0yWuwFzTAWrvT2MiF6KoBdVYQ BS3or5ZqfytlhN48QXDXeF8fOtdrMTIjmZgh98xI4w4oqVCQccdt4HruaVj5vFOPalo3 MCBKTX/7cGMV904yTwB39uKQ1ofngAU/4/wctVaekTlEB0xXZWsdTDyazsFYr4zw5YsK CVs0DuRf3eZ3gE05MHezNIvAYpD9I1uQOq/207U+YiqoFU4oMdxSRTJRpy1dV2m22GJ/ nTAA== X-Received: by 10.180.79.200 with SMTP id l8mr9429747wix.60.1370184340748; Sun, 02 Jun 2013 07:45:40 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id ff10sm16982323wib.10.2013.06.02.07.45.39 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 07:45:39 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Alban Hertroys In-Reply-To: Date: Sun, 2 Jun 2013 16:45:38 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.1503) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:45:42 -0000 I realise I only implied a fairly critical difference between the old = and new situations: On Jun 2, 2013, at 15:53, Alban Hertroys wrote: > Hello list, >=20 > I just replaced my home server and moved the disks from the old one = over to the new one. In the old server, 4 of the disks were connected to = a twa (3Ware 9550) controller, which of course has it's own way of = marking units/volumes on those disks. >=20 > The thing is, I have these disks in the new server and I found that = (to my surprise) I can actually mount them! But, I'm missing a large = part and I am wondering if there's some method to access those last = partitions too. The new server doesn't have the raid controller. The disks are directly = connected to the onboard SATA ports. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 14:46:09 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 66A7CBD5 for ; Sun, 2 Jun 2013 14:46:09 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id 1D59A1C89 for ; Sun, 2 Jun 2013 14:46:08 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r52Ek75n009106; Sun, 2 Jun 2013 08:46:07 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r52Ek7qe009103; Sun, 2 Jun 2013 08:46:07 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Sun, 2 Jun 2013 08:46:07 -0600 (MDT) From: Warren Block To: Alban Hertroys Subject: Re: Corrupt GPT header on disk from twa array - fixable? In-Reply-To: <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> Message-ID: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Sun, 02 Jun 2013 08:46:08 -0600 (MDT) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 14:46:09 -0000 On Sun, 2 Jun 2013, Alban Hertroys wrote: > On Jun 2, 2013, at 16:12, Kimmo Paasiala wrote: >> >> Looking at the gpart(8) output it seems that only 20GBs of the disk is >> recognized by the disk driver but the GPT table still shows the full >> capacity 910GB. I'd say that the GPT table is in fact correct and if >> you can somehow get the disks to be recognized with full capacity they >> should be usable as they are. What does dmesg(8) say about the disks? > > From dmesg: > > ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 > ada2: usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) > ATA-8 SATA 2.x device > usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR > ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada2: Command Queueing enabled > ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) > ada2: Previously was known as ad8 > ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 > ada3: ATA-8 SATA 2.x device > ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada3: Command Queueing enabled > ada3: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) > ada3: Previously was known as ad10 > ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 > ada4: ATA-8 SATA 2.x device > usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored) > ada4: 300.000MB/s transfers (SATA 2.x, usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR > UDMA6, PIO 8192bytes) > ada4: Command Queueing enabled > ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) > ada4: Previously was known as ad12 > ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 > ada5: ATA-8 SATA 3.x device > ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada5: Command Queueing enabled > ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) > ada5: Previously was known as ad14 > SMP: AP CPU #1 Launched! > Timecounter "TSC-low" frequency 13371081 Hz quality 800 > GEOM: ada2: the secondary GPT header is not in the last LBA. > GEOM: ada3: the secondary GPT header is not in the last LBA. > GEOM_MIRROR: Device mirror/boot launched (2/2). > GEOM_MIRROR: Device mirror/swap launched (2/2). > GEOM_MIRROR: Device mirror/root launched (2/2). > GEOM: ada4: the secondary GPT header is not in the last LBA. > GEOM: ada5: the secondary GPT header is not in the last LBA. There is a lot of stuff going on there. You switched from a hardware RAID card to something else in the new machine. Maybe a different card, or maybe just the motherboard. The old controller may have put metadata on the drives and hidden it. On a new controller, that metadata is not hidden. This would explain the "secondary GPT header is not in the last LBA" message. If the old controller "split" the combined drives into virtual volumes, there may be another GPT somewhere in the remainder of the drive. If you could find that, gnop(8) could be used with an offset to mount it. This could be another explanation for the GPT being "corrupt": the GPT at the start of the drive is for the first volume, the backup GPT at the end of the drive is for the second volume. Finally, GPT and gmirror are combined. That's a problematic combination because both want metadata in the last block of the drive. The new section in the Handbook about RAID1 (gmirror) describes that in the "Metadata Issues" section: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/GEOM-mirror.html From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 15:12:52 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 071B7800 for ; Sun, 2 Jun 2013 15:12:52 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-we0-x236.google.com (mail-we0-x236.google.com [IPv6:2a00:1450:400c:c03::236]) by mx1.freebsd.org (Postfix) with ESMTP id 91BF01D65 for ; Sun, 2 Jun 2013 15:12:51 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id w58so1019792wes.27 for ; Sun, 02 Jun 2013 08:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=yOZyN+jOEm+NNGePt8QHciqSgbayRLljv0ixLOhRYVM=; b=JhY2RhCA57sjMBWKPWJTTh7nEYbKvbBEnq6bJ/go2Pyow74YCBtyGg42N6s9+nWxtK 8c1RsLPY3AdgjA2U+I5nIiNvKKEoRhhmL9tSv1ejZ5GT8xxxtLIM3umq90+1GTwBqNiO UbKZnRx6plhpnmRs9fLaygS9e3+dilxWbRvucZi8S+JNsriWVj/53SyI6nTAvXH7J6qM K/fpydwqCc73SGFfRGSYLTq0J9Kcd5iVSsQO7s37++D0mK5v+5ZK6m2QlMhQIGQUl/Tx 8zxKUhtWwc1oiiHgb3mrEF9qx1diSLYgLXWYvbOBDcybMaBDYgBICIRO4cPZRZTtixeY Igbg== X-Received: by 10.180.108.3 with SMTP id hg3mr9515113wib.17.1370185970719; Sun, 02 Jun 2013 08:12:50 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id ff10sm17132786wib.10.2013.06.02.08.12.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 08:12:49 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Alban Hertroys In-Reply-To: Date: Sun, 2 Jun 2013 17:12:48 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> To: Warren Block X-Mailer: Apple Mail (2.1503) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 15:12:52 -0000 On Jun 2, 2013, at 16:46, Warren Block wrote: > On Sun, 2 Jun 2013, Alban Hertroys wrote: >=20 >> On Jun 2, 2013, at 16:12, Kimmo Paasiala wrote: >>>=20 >>> Looking at the gpart(8) output it seems that only 20GBs of the disk = is >>> recognized by the disk driver but the GPT table still shows the full >>> capacity 910GB. I'd say that the GPT table is in fact correct and if >>> you can somehow get the disks to be recognized with full capacity = they >>> should be usable as they are. What does dmesg(8) say about the = disks? >>=20 >> =46rom dmesg: >>=20 >> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 >> ada2: usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, = ignored) >> ATA-8 SATA 2.x device >> usbd_setup_device_desc: getting device descriptor at addr 2 failed, = USB_ERR_IOERROR >> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada2: Command Queueing enabled >> ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) >> ada2: Previously was known as ad8 >> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 >> ada3: ATA-8 SATA 2.x device >> ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >> ada3: Command Queueing enabled >> ada3: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) >> ada3: Previously was known as ad10 >> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 >> ada4: ATA-8 SATA 2.x device >> usbd_req_re_enumerate: addr=3D2, set address failed! = (USB_ERR_IOERROR, ignored) >> ada4: 300.000MB/s transfers (SATA 2.x, usbd_setup_device_desc: = getting device descriptor at addr 2 failed, USB_ERR_IOERROR >> UDMA6, PIO 8192bytes) >> ada4: Command Queueing enabled >> ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) >> ada4: Previously was known as ad12 >> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 >> ada5: ATA-8 SATA 3.x device >> ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) >> ada5: Command Queueing enabled >> ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) >> ada5: Previously was known as ad14 >> SMP: AP CPU #1 Launched! >> Timecounter "TSC-low" frequency 13371081 Hz quality 800 >> GEOM: ada2: the secondary GPT header is not in the last LBA. >> GEOM: ada3: the secondary GPT header is not in the last LBA. >> GEOM_MIRROR: Device mirror/boot launched (2/2). >> GEOM_MIRROR: Device mirror/swap launched (2/2). >> GEOM_MIRROR: Device mirror/root launched (2/2). >> GEOM: ada4: the secondary GPT header is not in the last LBA. >> GEOM: ada5: the secondary GPT header is not in the last LBA. >=20 > There is a lot of stuff going on there. >=20 > You switched from a hardware RAID card to something else in the new = machine. Maybe a different card, or maybe just the motherboard. The = old controller may have put metadata on the drives and hidden it. On a = new controller, that metadata is not hidden. This would explain the = "secondary GPT header is not in the last LBA" message. >=20 > If the old controller "split" the combined drives into virtual = volumes, there may be another GPT somewhere in the remainder of the = drive. If you could find that, gnop(8) could be used with an offset to = mount it. This could be another explanation for the GPT being "corrupt": = the GPT at the start of the drive is for the first volume, the backup = GPT at the end of the drive is for the second volume. It did indeed! I just sent a message about that, as I realised that = wasn't clear from my original description. I think gnop(8) is the answer = to my question. I've never worked with gnop before; is this a safe approach?: # kldload geom_nop # gnop create -v -o 41943006 -S 512 ada4 # mount /dev/ada4.nop /mnt I get the impression that gnop might be non-destructive, but that's not = entirely clear from the man page. I tried the above on ada5 (the other half of the mirror that I applied = gpart recover to earlier), but it spews: gnop: Invalid offset for provider ada5. What number does it expect for that offset? And what exactly is gpart = show showing? I was under the assumption that both would be sectors = (which judging from the numbers would be 512 bytes in size). > Finally, GPT and gmirror are combined. That's a problematic = combination because both want metadata in the last block of the drive. = The new section in the Handbook about RAID1 (gmirror) describes that in = the "Metadata Issues" section: > = http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/GEOM-mirror.html= I'm pretty sure the disks on the controller had nothing to do with = gmirror ever. Gmirror is only applied to a pair of new disks that I put in the (new) = server to be able to copy my data over. I hadn't expected to be able to = rely on those original disks to be readable at all without the = controller, so I needed some place to store the data. I like the = redundancy of a mirror, so I used gmirror for (only) the new disks. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 15:48:55 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 13F4325E for ; Sun, 2 Jun 2013 15:48:55 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by mx1.freebsd.org (Postfix) with ESMTP id AB7C01E62 for ; Sun, 2 Jun 2013 15:48:54 +0000 (UTC) Received: from mfilter18-d.gandi.net (mfilter18-d.gandi.net [217.70.178.146]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id 8E6EA41C05D; Sun, 2 Jun 2013 17:48:36 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter18-d.gandi.net Received: from relay5-d.mail.gandi.net ([217.70.183.197]) by mfilter18-d.gandi.net (mfilter18-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id fpLKrz2eezY6; Sun, 2 Jun 2013 17:48:34 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 2365C41C051; Sun, 2 Jun 2013 17:48:34 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 4C46A73A1C; Sun, 2 Jun 2013 08:48:32 -0700 (PDT) Date: Sun, 2 Jun 2013 08:48:32 -0700 From: Jeremy Chadwick To: Alban Hertroys Subject: Re: Corrupt GPT header on disk from twa array - fixable? Message-ID: <20130602154832.GA23072@icarus.home.lan> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Warren Block , Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 15:48:55 -0000 On Sun, Jun 02, 2013 at 05:12:48PM +0200, Alban Hertroys wrote: > > On Jun 2, 2013, at 16:46, Warren Block wrote: > > > On Sun, 2 Jun 2013, Alban Hertroys wrote: > > > >> On Jun 2, 2013, at 16:12, Kimmo Paasiala wrote: > >>> > >>> Looking at the gpart(8) output it seems that only 20GBs of the disk is > >>> recognized by the disk driver but the GPT table still shows the full > >>> capacity 910GB. I'd say that the GPT table is in fact correct and if > >>> you can somehow get the disks to be recognized with full capacity they > >>> should be usable as they are. What does dmesg(8) say about the disks? > >> > >> From dmesg: > >> > >> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 > >> ada2: usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) > >> ATA-8 SATA 2.x device > >> usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR > >> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > >> ada2: Command Queueing enabled > >> ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) > >> ada2: Previously was known as ad8 > >> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 > >> ada3: ATA-8 SATA 2.x device > >> ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > >> ada3: Command Queueing enabled > >> ada3: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) > >> ada3: Previously was known as ad10 > >> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 > >> ada4: ATA-8 SATA 2.x device > >> usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored) > >> ada4: 300.000MB/s transfers (SATA 2.x, usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR > >> UDMA6, PIO 8192bytes) > >> ada4: Command Queueing enabled > >> ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) > >> ada4: Previously was known as ad12 > >> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 > >> ada5: ATA-8 SATA 3.x device > >> ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > >> ada5: Command Queueing enabled > >> ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) > >> ada5: Previously was known as ad14 > >> SMP: AP CPU #1 Launched! > >> Timecounter "TSC-low" frequency 13371081 Hz quality 800 > >> GEOM: ada2: the secondary GPT header is not in the last LBA. > >> GEOM: ada3: the secondary GPT header is not in the last LBA. > >> GEOM_MIRROR: Device mirror/boot launched (2/2). > >> GEOM_MIRROR: Device mirror/swap launched (2/2). > >> GEOM_MIRROR: Device mirror/root launched (2/2). > >> GEOM: ada4: the secondary GPT header is not in the last LBA. > >> GEOM: ada5: the secondary GPT header is not in the last LBA. > > > > There is a lot of stuff going on there. > > > > You switched from a hardware RAID card to something else in the new machine. Maybe a different card, or maybe just the motherboard. The old controller may have put metadata on the drives and hidden it. On a new controller, that metadata is not hidden. This would explain the "secondary GPT header is not in the last LBA" message. > > > > If the old controller "split" the combined drives into virtual volumes, there may be another GPT somewhere in the remainder of the drive. If you could find that, gnop(8) could be used with an offset to mount it. This could be another explanation for the GPT being "corrupt": the GPT at the start of the drive is for the first volume, the backup GPT at the end of the drive is for the second volume. > > It did indeed! I just sent a message about that, as I realised that wasn't clear from my original description. I think gnop(8) is the answer to my question. > > I've never worked with gnop before; is this a safe approach?: > > # kldload geom_nop > # gnop create -v -o 41943006 -S 512 ada4 > # mount /dev/ada4.nop /mnt > > I get the impression that gnop might be non-destructive, but that's not entirely clear from the man page. > > I tried the above on ada5 (the other half of the mirror that I applied gpart recover to earlier), but it spews: > > gnop: Invalid offset for provider ada5. > > What number does it expect for that offset? And what exactly is gpart show showing? I was under the assumption that both would be sectors (which judging from the numbers would be 512 bytes in size). > > > Finally, GPT and gmirror are combined. That's a problematic combination because both want metadata in the last block of the drive. The new section in the Handbook about RAID1 (gmirror) describes that in the "Metadata Issues" section: > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/GEOM-mirror.html > > I'm pretty sure the disks on the controller had nothing to do with gmirror ever. > > Gmirror is only applied to a pair of new disks that I put in the (new) server to be able to copy my data over. I hadn't expected to be able to rely on those original disks to be readable at all without the controller, so I needed some place to store the data. I like the redundancy of a mirror, so I used gmirror for (only) the new disks. I think you're missing what Warren is telling you, because you have multiple things going on/complexities to deal with simultaneously. You haven't provided any details about your gmirror setup either. All we know at this point: > >> GEOM_MIRROR: Device mirror/boot launched (2/2). > >> GEOM_MIRROR: Device mirror/swap launched (2/2). > >> GEOM_MIRROR: Device mirror/root launched (2/2). My gut feeling is ada2 and ada3 make up the mirror, and the mirror is at the disk level (ada2 and ada3). I'm basing this on past evidence presented in the thread, and having to make assumptions. No "gmirror status" output = we have to make assumptions. Now, what Warren is telling you: gmirror + GPT do not play well together. This is a design flaw** on the part of gmirror. If you want to use gmirror with disks using GPT, your only solutions are to mirror the partitions (adaXpX) and not the disk (adaX), which has its own set of caveats, or to use the MBR scheme (and if these are 4K sectors disks, or you plan on using those, you're even more screwed). I will not bring ZFS into this discussion since that also opens up a can of worms -- I'm trying to stay focused. The errors you see on ada4 and ada5 about the backup GPT header can be dealt with in a different manner. But for (again, assuming) ada2 and ada3, you will see GPT "backup header corruption" messages indefinitely because of the above flaw. ** -- I will not get into a debate about terminology. I am aware of the history (which came first), and so on. It's a flaw. Linux md had the same problem when GPT was introduced, and it has since been fixed/addressed. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 18:39:37 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 42CAF18C for ; Sun, 2 Jun 2013 18:39:37 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id DDE9B138C for ; Sun, 2 Jun 2013 18:39:36 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r52IdZpp010131; Sun, 2 Jun 2013 12:39:35 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r52IdZpL010128; Sun, 2 Jun 2013 12:39:35 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Sun, 2 Jun 2013 12:39:35 -0600 (MDT) From: Warren Block To: Alban Hertroys Subject: Re: Corrupt GPT header on disk from twa array - fixable? In-Reply-To: <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> Message-ID: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Sun, 02 Jun 2013 12:39:36 -0600 (MDT) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 18:39:37 -0000 On Sun, 2 Jun 2013, Alban Hertroys wrote: > On Jun 2, 2013, at 16:46, Warren Block wrote: > I've never worked with gnop before; is this a safe approach?: > > # kldload geom_nop > # gnop create -v -o 41943006 -S 512 ada4 > # mount /dev/ada4.nop /mnt > > I get the impression that gnop might be non-destructive, but that's not entirely clear from the man page. Well, yes, but mount it read-only. gnop is (yet another) GEOM transform. It should be non-destructive as long as you don't write to it. > I tried the above on ada5 (the other half of the mirror that I applied gpart recover to earlier), but it spews: > > gnop: Invalid offset for provider ada5. > > What number does it expect for that offset? The trick would be figuring out what number was used by the RAID controller. > And what exactly is gpart show showing? I was under the assumption > that both would be sectors (which judging from the numbers would be > 512 bytes in size). Yes, it's sectors--the last column is human-readable. But the GEOM logical device might be constructed from the GPT parameters. It may not see the additional blocks on the physical device until the GPT tables are repaired. Which might corrupt the actual data. Really, the easiest way would be to temporarily install the old RAID controller and copy the data off the array. >> Finally, GPT and gmirror are combined. That's a problematic >> combination because both want metadata in the last block of the >> drive. The new section in the Handbook about RAID1 (gmirror) >> describes that in the "Metadata Issues" section: >> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/GEOM-mirror.html > > I'm pretty sure the disks on the controller had nothing to do with gmirror ever. > > Gmirror is only applied to a pair of new disks that I put in the (new) > server to be able to copy my data over. I hadn't expected to be able > to rely on those original disks to be readable at all without the > controller, so I needed some place to store the data. I like the > redundancy of a mirror, so I used gmirror for (only) the new disks. gmirror is good. GPT is also good. The combination is a problem. gmirror metadata overwrites the backup GPT, so those disks will show "corrupt" also. For now, the recommended workaround is to just use MBR, which doesn't have any metadata at the end of the disk. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 21:22:07 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9B5CC441 for ; Sun, 2 Jun 2013 21:22:07 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 28D9B1959 for ; Sun, 2 Jun 2013 21:22:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r52LM5P3042348; Mon, 3 Jun 2013 01:22:05 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Mon, 3 Jun 2013 01:22:05 +0400 (MSK) From: Dmitry Morozovsky To: Warren Block Subject: Re: Corrupt GPT header on disk from twa array - fixable? In-Reply-To: Message-ID: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Mon, 03 Jun 2013 01:22:05 +0400 (MSK) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 21:22:07 -0000 On Sun, 2 Jun 2013, Warren Block wrote: [snip] > gmirror is good. GPT is also good. The combination is a problem. gmirror > metadata overwrites the backup GPT, so those disks will show "corrupt" also. > For now, the recommended workaround is to just use MBR, which doesn't have any > metadata at the end of the disk. ... or gmirror not whole disks, but GPT partitions (as OP does, as far as I can tell from gmirror dmesg reports) -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 21:27:05 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 206296F4 for ; Sun, 2 Jun 2013 21:27:05 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id D55591992 for ; Sun, 2 Jun 2013 21:27:04 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r52LR3uM011360; Sun, 2 Jun 2013 15:27:03 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r52LR3p5011357; Sun, 2 Jun 2013 15:27:03 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Sun, 2 Jun 2013 15:27:03 -0600 (MDT) From: Warren Block To: Dmitry Morozovsky Subject: Re: Corrupt GPT header on disk from twa array - fixable? In-Reply-To: Message-ID: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Sun, 02 Jun 2013 15:27:03 -0600 (MDT) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 21:27:05 -0000 On Mon, 3 Jun 2013, Dmitry Morozovsky wrote: > On Sun, 2 Jun 2013, Warren Block wrote: > > [snip] > >> gmirror is good. GPT is also good. The combination is a problem. gmirror >> metadata overwrites the backup GPT, so those disks will show "corrupt" also. >> For now, the recommended workaround is to just use MBR, which doesn't have any >> metadata at the end of the disk. > > ... or gmirror not whole disks, but GPT partitions (as OP does, as far as I can > tell from gmirror dmesg reports) That works, but if there is more than one partition per disk, rebuilds fight with each other for the heads. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 21:46:57 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 212DA9E5 for ; Sun, 2 Jun 2013 21:46:57 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id A10F01A19 for ; Sun, 2 Jun 2013 21:46:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r52LktRe043028; Mon, 3 Jun 2013 01:46:55 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Mon, 3 Jun 2013 01:46:55 +0400 (MSK) From: Dmitry Morozovsky To: Warren Block Subject: Re: Corrupt GPT header on disk from twa array - fixable? In-Reply-To: Message-ID: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Mon, 03 Jun 2013 01:46:55 +0400 (MSK) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 21:46:57 -0000 On Sun, 2 Jun 2013, Warren Block wrote: > > > gmirror is good. GPT is also good. The combination is a problem. gmirror > > > metadata overwrites the backup GPT, so those disks will show "corrupt" > > > also. > > > For now, the recommended workaround is to just use MBR, which doesn't have > > > any > > > metadata at the end of the disk. > > > > ... or gmirror not whole disks, but GPT partitions (as OP does, as far as I > > can > > tell from gmirror dmesg reports) > > That works, but if there is more than one partition per disk, rebuilds fight > with each other for the heads. Right; OTOH, there is usually no more than one or two partitions which are under write pressure, so *usually* you'll find rebuilding, say, /dev/mirror/var and /dev/mirror/db (at least most of our gmirror-setup servers show that) On the third hand, if you have enough memory, ZFS is both simplier and more lazy regarding repairs ;) -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 22:07:48 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6C815E4E for ; Sun, 2 Jun 2013 22:07:48 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) by mx1.freebsd.org (Postfix) with ESMTP id 009671A99 for ; Sun, 2 Jun 2013 22:07:47 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id hr14so2116392wib.16 for ; Sun, 02 Jun 2013 15:07:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=HocGMVos9QL7+7BGdzQcGdhIJ1/fEVE4Gqe3Hriv/7Y=; b=rYmI1A065C6QV38SDTgBiV8pFyvOKro8mrNySNMnxYWW7gfz6XXeHbsbPy6QvD3eRm KpM1M661beTDsmX5ch77R/ghVZSGPfk+HJ7wZRAatOevLn38UGaj57wU681JMBsjgYcf HqPWTwLP6Ra39J/il0lgHn/piBo/mYn0eaBrQHPWgHkjuIn25xYAQ/ssgeoh1Y5Uvzzu TvhSR2Xk1dxhu4sFGfd/+YWpGdX1s20GF7d+wpShaaeyTzIcIA25ZePeJuGcQmh0SHOa 38AAwg3CJgxC+YKf7ddFMNYCM/IFbuZhUMjV9Xur6vvHu7rjxRURWmIeGOUqUFob7HY9 g8qA== X-Received: by 10.180.198.140 with SMTP id jc12mr10029651wic.53.1370210867194; Sun, 02 Jun 2013 15:07:47 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id q20sm19495120wiv.7.2013.06.02.15.07.45 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 15:07:46 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Alban Hertroys In-Reply-To: <20130602154832.GA23072@icarus.home.lan> Date: Mon, 3 Jun 2013 00:07:44 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <17D7EC66-768B-462F-97DC-2550FDE1AB59@gmail.com> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> <20130602154832.GA23072@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1503) Cc: Warren Block , Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 22:07:48 -0000 On Jun 2, 2013, at 17:48, Jeremy Chadwick wrote: > On Sun, Jun 02, 2013 at 05:12:48PM +0200, Alban Hertroys wrote: >>=20 >> On Jun 2, 2013, at 16:46, Warren Block wrote: >>=20 >>> On Sun, 2 Jun 2013, Alban Hertroys wrote: >>>=20 >>>> On Jun 2, 2013, at 16:12, Kimmo Paasiala = wrote: >>>>>=20 >>>>> Looking at the gpart(8) output it seems that only 20GBs of the = disk is >>>>> recognized by the disk driver but the GPT table still shows the = full >>>>> capacity 910GB. I'd say that the GPT table is in fact correct and = if >>>>> you can somehow get the disks to be recognized with full capacity = they >>>>> should be usable as they are. What does dmesg(8) say about the = disks? >>>>=20 >>>> =46rom dmesg: >>>>=20 >>>> ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 >>>> ada2: usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, = ignored) >>>> ATA-8 SATA 2.x device >>>> usbd_setup_device_desc: getting device descriptor at addr 2 failed, = USB_ERR_IOERROR >>>> ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >>>> ada2: Command Queueing enabled >>>> ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) >>>> ada2: Previously was known as ad8 >>>> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 >>>> ada3: ATA-8 SATA 2.x device >>>> ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) >>>> ada3: Command Queueing enabled >>>> ada3: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) >>>> ada3: Previously was known as ad10 >>>> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 >>>> ada4: ATA-8 SATA 2.x device >>>> usbd_req_re_enumerate: addr=3D2, set address failed! = (USB_ERR_IOERROR, ignored) >>>> ada4: 300.000MB/s transfers (SATA 2.x, usbd_setup_device_desc: = getting device descriptor at addr 2 failed, USB_ERR_IOERROR >>>> UDMA6, PIO 8192bytes) >>>> ada4: Command Queueing enabled >>>> ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) >>>> ada4: Previously was known as ad12 >>>> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 >>>> ada5: ATA-8 SATA 3.x device >>>> ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) >>>> ada5: Command Queueing enabled >>>> ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) >>>> ada5: Previously was known as ad14 >>>> SMP: AP CPU #1 Launched! >>>> Timecounter "TSC-low" frequency 13371081 Hz quality 800 >>>> GEOM: ada2: the secondary GPT header is not in the last LBA. >>>> GEOM: ada3: the secondary GPT header is not in the last LBA. >>>> GEOM_MIRROR: Device mirror/boot launched (2/2). >>>> GEOM_MIRROR: Device mirror/swap launched (2/2). >>>> GEOM_MIRROR: Device mirror/root launched (2/2). >>>> GEOM: ada4: the secondary GPT header is not in the last LBA. >>>> GEOM: ada5: the secondary GPT header is not in the last LBA. >>>=20 > I think you're missing what Warren is telling you, because you have > multiple things going on/complexities to deal with simultaneously. >=20 > You haven't provided any details about your gmirror setup either. All > we know at this point: >=20 >>>> GEOM_MIRROR: Device mirror/boot launched (2/2). >>>> GEOM_MIRROR: Device mirror/swap launched (2/2). >>>> GEOM_MIRROR: Device mirror/root launched (2/2). >=20 > My gut feeling is ada2 and ada3 make up the mirror, and the mirror is = at > the disk level (ada2 and ada3). I'm basing this on past evidence > presented in the thread, and having to make assumptions. No "gmirror > status" output =3D we have to make assumptions. The gmirror is actually on ada0+ada1, and not at all on the disks that I = copied the dmesg information of. Those disks weren't in the hardware = RAID array of the old server, and the gmirror isn't on the disks that = were in that RAID controller. I took care to keep those separate until I = can erase them and add them as "normal" disks. > Now, what Warren is telling you: gmirror + GPT do not play well > together. This is a design flaw** on the part of gmirror. If you = want > to use gmirror with disks using GPT, your only solutions are to mirror > the partitions (adaXpX) and not the disk (adaX), which has its own set > of caveats, or to use the MBR scheme (and if these are 4K sectors = disks, > or you plan on using those, you're even more screwed). I didn't know that, but just incidentally mirrored my partitions on ada0 = and ada1 instead of the entire disks. For the sake of removing the = confusion from this thread: # gmirror status Name Status Components mirror/boot COMPLETE ada0p1 (ACTIVE) ada1p1 (ACTIVE) mirror/swap COMPLETE ada0p2 (ACTIVE) ada1p2 (ACTIVE) mirror/root COMPLETE ada0p3 (ACTIVE) ada1p3 (ACTIVE) > The errors you see on ada4 and ada5 about the backup GPT header can be > dealt with in a different manner. >=20 > But for (again, assuming) ada2 and ada3, you will see GPT "backup = header > corruption" messages indefinitely because of the above flaw. I think that those messages actually stem from the same issue as I'm = having with the (hardware-)RAID volumes on ada4 and ada5: Apparently the = RAID controller reserved some space at the end of those volumes to store = information about the volume layout, very similar to how geom does that. The geom labels that I put on the partitions inside those volumes are = therefore not in the last sector of the disk, but they were in the last = sector of each volume while the disks were still attached to the = controller. Those messages on ada2 & 3 don't really bother me. I can read what's on = those disks the way they are (unlike the second volume on disks ada4 & = 5). Once I'm confident that I don't need anything that's on them = anymore, they'll be repartitioned and the problem will be gone. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 22:38:59 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A245B6B7 for ; Sun, 2 Jun 2013 22:38:59 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-we0-x22d.google.com (mail-we0-x22d.google.com [IPv6:2a00:1450:400c:c03::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 37AAD1B7F for ; Sun, 2 Jun 2013 22:38:59 +0000 (UTC) Received: by mail-we0-f173.google.com with SMTP id x55so1178302wes.4 for ; Sun, 02 Jun 2013 15:38:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=fLza5fY/4kJZL5MgM/sh9hiHvrSDUcci5XAPw/Q36h0=; b=r6MHeHzT/ueieJ1F9UZoWh11SLwdzhOr0mNifuzUtFEoMf7VNjlt8a8V1pm1VIZuoj WoDdNa8tLVMrcXZwsg9JrpW+Q2KdWolZBcGlkYp42WfapQ1iJm7McuNScakFNfHboC14 Is/+8ZMNQFRc9p8H3svO6kFBNGRylesipFJSriCurcpsgafejzt5VLMSS3XXTIywbyVa ZExI4KCWuvqvLid+X4/LmGThWsRVxHuIsNTJpzpYSADgoSF5lMNrmRONFTICTJ5ojXRw VlA4upADuBF0s6y30Gh8XY/HwZpyK+reJFiux3sp+OAkw4ZaoRHllG6bm0z4sSQRpYLF //Hw== X-Received: by 10.180.206.135 with SMTP id lo7mr10134983wic.64.1370212738346; Sun, 02 Jun 2013 15:38:58 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id fv11sm19581321wic.11.2013.06.02.15.38.56 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Jun 2013 15:38:57 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Alban Hertroys In-Reply-To: Date: Mon, 3 Jun 2013 00:38:56 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> To: Warren Block X-Mailer: Apple Mail (2.1503) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 22:38:59 -0000 On Jun 2, 2013, at 20:39, Warren Block wrote: > On Sun, 2 Jun 2013, Alban Hertroys wrote: >> On Jun 2, 2013, at 16:46, Warren Block wrote: >> I've never worked with gnop before; is this a safe approach?: >>=20 >> # kldload geom_nop >> # gnop create -v -o 41943006 -S 512 ada4 >> # mount /dev/ada4.nop /mnt >>=20 >> I get the impression that gnop might be non-destructive, but that's = not entirely clear from the man page. >=20 > Well, yes, but mount it read-only. gnop is (yet another) GEOM = transform. It should be non-destructive as long as you don't write to = it. Yup, writing to them obviously could be destructive. I was aware of = that, but I hadn't thought of mounting them read-only. Thanks. >> I tried the above on ada5 (the other half of the mirror that I = applied gpart recover to earlier), but it spews: >>=20 >> gnop: Invalid offset for provider ada5. >>=20 >> What number does it expect for that offset? >=20 > The trick would be figuring out what number was used by the RAID = controller. Ah right, I need the start of the next volume. I think my calculation = put me at the start of the RAID controllers volume header block for the = first volume - a few sectors too early probably. I suppose it shouldn't take too long finding the start of the second = volume with some trial and error now that I know to look past sector = 41943005 + 1 + volume header size. I'll have a look whether perhaps the = controllers' documentation mentions anything of a size... >=20 >> And what exactly is gpart show showing? I was under the assumption = that both would be sectors (which judging from the numbers would be 512 = bytes in size). >=20 > Yes, it's sectors--the last column is human-readable. But the GEOM = logical device might be constructed from the GPT parameters. It may not = see the additional blocks on the physical device until the GPT tables = are repaired. Which might corrupt the actual data. >=20 > Really, the easiest way would be to temporarily install the old RAID = controller and copy the data off the array. Well, that would mean I'd have to assemble the old server again, as the = controller is not compatible with the hardware in the new one. And that = would probably be unnecessary as well, since I already did copy the data = off those disks. I was just curious whether it would be possible to read that data off = the disks while I still have them (with their original contents) in the = new server in the eventuality that I _did_ forget to copy something over = or that something wasn't copied over correctly. I copied the data over a 100MBit ethernet link, which was the fastest = option I had with the old server; it had USB1 and no native SATA. Hence = the RAID controller, but that was on a now deprecated PCI-X channel = (those 64-bit parallel things) and all 4 ports were in use. Not to mention that the CPU was so old that it had a rather narrow = margin for operating temperatures and overheated several times during = the copying process, because rsync+sshd put a relatively high load on = the CPU (An old Athlon XP 2000+). If I manage to get those volumes mounted with gnop, that would give me = the opportunity to checksum the contents of the original disks with the = copied content on the new disks. I'm sure that rsync is quite reliable, but I had a few hangs of the old = server during the process and although rsync is capable of continueing = where it left off, it'd be nice to be able to verify that worked as = advertised. I'll experiment some more with gnop, but if that doesn't get me anywhere = I'll probably just wipe those old disks with a new GPT partition table = so that I can use them for storage again. There shouldn't be anything on = it that I don't have already. > gmirror is good. GPT is also good. The combination is a problem. = gmirror metadata overwrites the backup GPT, so those disks will show = "corrupt" also. For now, the recommended workaround is to just use MBR, = which doesn't have any metadata at the end of the disk. Or you mirror the partitions on the disk as Jeremy pointed out, in which = case the backup GPT goes to the end of the disk, while the gmirror = meta-data goes to the end of each partition within that space. I get = that now. Thanks for all the help, guys! If I figure out how to access those extra volumes, I'll report back. Who = knows who'll run into the same problem some day (possibly less prepared = for the situation). Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Sun Jun 2 23:09:56 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DC72D470 for ; Sun, 2 Jun 2013 23:09:56 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id 838AC1C71 for ; Sun, 2 Jun 2013 23:09:56 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r52N9tVg012005; Sun, 2 Jun 2013 17:09:55 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r52N9tcj012002; Sun, 2 Jun 2013 17:09:55 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Sun, 2 Jun 2013 17:09:55 -0600 (MDT) From: Warren Block To: Alban Hertroys Subject: Re: Corrupt GPT header on disk from twa array - fixable? In-Reply-To: Message-ID: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Sun, 02 Jun 2013 17:09:55 -0600 (MDT) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jun 2013 23:09:56 -0000 On Mon, 3 Jun 2013, Alban Hertroys wrote: >> >> Really, the easiest way would be to temporarily install the old RAID controller and copy the data off the array. > > Well, that would mean I'd have to assemble the old server again, as > the controller is not compatible with the hardware in the new one. And > that would probably be unnecessary as well, since I already did copy > the data off those disks. > > I was just curious whether it would be possible to read that data off > the disks while I still have them (with their original contents) in > the new server in the eventuality that I _did_ forget to copy > something over or that something wasn't copied over correctly. > > I copied the data over a 100MBit ethernet link, which was the fastest > option I had with the old server; it had USB1 and no native SATA. > Hence the RAID controller, but that was on a now deprecated PCI-X > channel (those 64-bit parallel things) and all 4 ports were in use. > Not to mention that the CPU was so old that it had a rather narrow > margin for operating temperatures and overheated several times during > the copying process, because rsync+sshd put a relatively high load on > the CPU (An old Athlon XP 2000+). PCI-X cards will operate in PCI slots. Or at least some will; I've done that with an Intel network card. The motherboard can't have components that block the unused part of the edge connector, or the offending card edge could be removed with extreme prejudice. From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 06:25:48 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A85237F4; Mon, 3 Jun 2013 06:25:48 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 589D6197F; Mon, 3 Jun 2013 06:25:47 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1UjOCz-0006oU-DS; Mon, 03 Jun 2013 09:25:33 +0300 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.3 To: pyunyh@gmail.com Subject: Re: SunFire X2200 ilo's bge1 DOWN/UP In-reply-to: <20130531054713.GB1478@michelle.cdnetworks.com> References: <20130529085544.GC3042@michelle.cdnetworks.com> <201305300859.20928.jhb@freebsd.org> <20130531054713.GB1478@michelle.cdnetworks.com> Comments: In-reply-to YongHyeon PYUN message dated "Fri, 31 May 2013 14:47:13 +0900." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 03 Jun 2013 09:25:33 +0300 From: Daniel Braniss Message-ID: Cc: freebsd-stable@freebsd.org, John Baldwin X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 06:25:48 -0000 > On Fri, May 31, 2013 at 08:24:47AM +0300, Daniel Braniss wrote: > > > On Thursday, May 30, 2013 2:44:35 am Daniel Braniss wrote: > > > > > --/04w6evG8XlLl3ft > > > > > Content-Type: text/x-diff; charset=us-ascii > > > > > Content-Disposition: attachment; filename="bge.media_sts.diff" > > > > > > > > > > Index: sys/dev/bge/if_bge.c > > > > > =================================================================== > > > > > --- sys/dev/bge/if_bge.c (revision 251021) > > > > > +++ sys/dev/bge/if_bge.c (working copy) > > > > > @@ -5583,6 +5583,10 @@ bge_ifmedia_sts(struct ifnet *ifp, struct ifmediar > > > > > > > > > > BGE_LOCK(sc); > > > > > > > > > > + if ((ifp->if_flags & IFF_UP) == 0) { > > > > > + BGE_UNLOCK(sc); > > > > > + return; > > > > > + } > > > > > if (sc->bge_flags & BGE_FLAG_TBI) { > > > > > ifmr->ifm_status = IFM_AVALID; > > > > > ifmr->ifm_active = IFM_ETHER; > > > > > > > > > > --/04w6evG8XlLl3ft-- > > > > after 18hs, the logs are empty! > > > > it seems the patch fixes the problem. > > > > > > > > now maybe it's time to hunt for who is randomly calling for bge_ifmedia_sts > > > > ... > > > > > > It could be any number of daemons that query interface state such as an > > > SNMP server, ladvd, etc. > > > > > > If you wanted help you could modify the patch so that it does something like > > > this: > > > > > #include > > > if (/* test for IFF_UP */) { > > > BGE_UNLOCK(sc); > > > if_printf(ifp, "state queried on down interface by pid %d (%s)", > > ------------------------------------------------------------------------------| > > add a \n > > > curthread->td_proc->p_pid, curthread->td_proc->p_comm); > > > return; > > > } > > > > > > -- > > > John Baldwin > > snmpd call this several times a second, (difficult to measeure since sysolog > > just says > > last message repeated 22 times > > in any case, the DOWN/UP appears once every few hours, oh well. > > I have now stopped the snmpd daemon, maybe there is someone else ... > > I have no idea why snmpd wants to know media status for interfaces > that are put into down state. The media status resolved after > bringing up the interface may be different one that was seen > before. > The patch also makes dhclient think driver got a valid link > regardless of link establishment. I guess that wouldn't be > issue though. I'll commit the patch after some more testing. > > Thanks for reporting and testing! > no problem! after more than 3 days, there were no more 'reports', so snmpd was the culprit. the snmpd we use is from ports, i'll try and see waht's going on ... thanks danny > > > > thanks, > > danny > > > > From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 07:14:45 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1613134A for ; Mon, 3 Jun 2013 07:14:45 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) by mx1.freebsd.org (Postfix) with ESMTP id 9E6921B44 for ; Mon, 3 Jun 2013 07:14:44 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id n12so2372101wgh.3 for ; Mon, 03 Jun 2013 00:14:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=JVyuJLDyRGt+PBnDwmaQgePppz3f6AhOf0dsZkX5ITQ=; b=pqctPjKY/E5R4aIPW64Cvzm9iIL4sJ6otaPfayCZRcsJXwg+LI5NOy4PcHIpdt/sAH YHJ+/NbiCMnINSZcX9aBwg0dURkK/pVaQtgTpA/MnEz6d3P/dNDnZhdqcOnXaOswl3OW ixUwg1Ru1bg3Hq632vK+tPUaZVkLPYteca1T19P+Raf9rRH2AABxIYCBHRfuM/c8FH5K 5gUnmosUBYER/41xjeJ6JrgKZv18ZJ70g08RLdVFYizU/dw87B2/fQEi0KTewqG3BagR WMrLRRJr24IJPP9lDZ4VrrIjWNbBoqgvGdmkh/QDkzGw6X++eTifHA5mJw1HKo6WZ3Rk Ju0w== X-Received: by 10.180.183.67 with SMTP id ek3mr11298883wic.44.1370243683874; Mon, 03 Jun 2013 00:14:43 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id dj7sm21308328wib.6.2013.06.03.00.14.42 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 00:14:42 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Corrupt GPT header on disk from twa array - fixable? From: Alban Hertroys In-Reply-To: Date: Mon, 3 Jun 2013 09:14:41 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> To: Warren Block X-Mailer: Apple Mail (2.1503) Cc: Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 07:14:45 -0000 On Jun 3, 2013, at 1:09, Warren Block wrote: > On Mon, 3 Jun 2013, Alban Hertroys wrote: >>>=20 >>> Really, the easiest way would be to temporarily install the old RAID = controller and copy the data off the array. >>=20 >> Well, that would mean I'd have to assemble the old server again, as = the controller is not compatible with the hardware in the new one. And = that would probably be unnecessary as well, since I already did copy the = data off those disks. >>=20 >> I was just curious whether it would be possible to read that data off = the disks while I still have them (with their original contents) in the = new server in the eventuality that I _did_ forget to copy something over = or that something wasn't copied over correctly. >>=20 >> I copied the data over a 100MBit ethernet link, which was the fastest = option I had with the old server; it had USB1 and no native SATA. Hence = the RAID controller, but that was on a now deprecated PCI-X channel = (those 64-bit parallel things) and all 4 ports were in use. Not to = mention that the CPU was so old that it had a rather narrow margin for = operating temperatures and overheated several times during the copying = process, because rsync+sshd put a relatively high load on the CPU (An = old Athlon XP 2000+). >=20 > PCI-X cards will operate in PCI slots. Or at least some will; I've = done that with an Intel network card. The motherboard can't have = components that block the unused part of the edge connector, or the = offending card edge could be removed with extreme prejudice. Not this 3Ware card. I remember buying that particular motherboard = because the card wouldn't fit in the PCI slots on the board I had. = There's a division in those PCI-X slots opposite of where there's one in = normal PCI slots and no groove in the card to match the division in the = PCI slot. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 07:56:38 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DB10AB5A for ; Mon, 3 Jun 2013 07:56:38 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 7F1731CC6 for ; Mon, 3 Jun 2013 07:56:38 +0000 (UTC) Received: from mfilter11-d.gandi.net (mfilter11-d.gandi.net [217.70.178.131]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id A92E5A8119; Mon, 3 Jun 2013 09:56:20 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter11-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter11-d.gandi.net (mfilter11-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id mlCtw29IXD4K; Mon, 3 Jun 2013 09:56:19 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id A0965A80B8; Mon, 3 Jun 2013 09:56:17 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id BE75B73A1C; Mon, 3 Jun 2013 00:56:15 -0700 (PDT) Date: Mon, 3 Jun 2013 00:56:15 -0700 From: Jeremy Chadwick To: Alban Hertroys Subject: Re: Corrupt GPT header on disk from twa array - fixable? Message-ID: <20130603075615.GA37302@icarus.home.lan> References: <7ABBEE71A96E411793E41BD97DA72BCE@multiplay.co.uk> <2943982C-719E-45D0-9B26-43B725738F83@gmail.com> <3659A498-F0EA-4AF3-80EA-40038DCA9CC7@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Warren Block , Kimmo Paasiala , freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 07:56:38 -0000 On Mon, Jun 03, 2013 at 09:14:41AM +0200, Alban Hertroys wrote: > > On Jun 3, 2013, at 1:09, Warren Block wrote: > > > On Mon, 3 Jun 2013, Alban Hertroys wrote: > >>> > >>> Really, the easiest way would be to temporarily install the old RAID controller and copy the data off the array. > >> > >> Well, that would mean I'd have to assemble the old server again, as the controller is not compatible with the hardware in the new one. And that would probably be unnecessary as well, since I already did copy the data off those disks. > >> > >> I was just curious whether it would be possible to read that data off the disks while I still have them (with their original contents) in the new server in the eventuality that I _did_ forget to copy something over or that something wasn't copied over correctly. > >> > >> I copied the data over a 100MBit ethernet link, which was the fastest option I had with the old server; it had USB1 and no native SATA. Hence the RAID controller, but that was on a now deprecated PCI-X channel (those 64-bit parallel things) and all 4 ports were in use. Not to mention that the CPU was so old that it had a rather narrow margin for operating temperatures and overheated several times during the copying process, because rsync+sshd put a relatively high load on the CPU (An old Athlon XP 2000+). > > > > PCI-X cards will operate in PCI slots. Or at least some will; I've done that with an Intel network card. The motherboard can't have components that block the unused part of the edge connector, or the offending card edge could be removed with extreme prejudice. > > Not this 3Ware card. I remember buying that particular motherboard because the card wouldn't fit in the PCI slots on the board I had. There's a division in those PCI-X slots opposite of where there's one in normal PCI slots and no groove in the card to match the division in the PCI slot. This is all besides-the-point, but to clarify: please see the following diagram: http://en.wikipedia.org/wiki/File:PCI_Keying.png I recommend seeing the caption under the diagram, in addition to reading the "Mixing of 32-bit and 64-bit PCI cards in different width slots" section: http://en.wikipedia.org/wiki/PCI-X It sounds like your 3Ware card is 5V PCI-X (32-bit or 64-bit is irrelevant), and your new motherboard only supports 3.3V PCI (which is pretty much the norm on all motherboards today when it comes to classic PCI). The 5V stuff is generally shunned (both with regards to PCI and PCI-X) and is uncommon at this point in time. You can find some server-class boards that offer this capability, such as Supermicro's UIO slots, where you purchase the proper type of "riser" (adapter) for the type of card you have, i.e. UIO->5.5V PCI-X 64-bit), but you will not find this on consumer/desktop or even "enthusiast" boards. Example: http://www.supermicro.com/support/resources/riser/riser.aspx If you want to know what kind of card it is, ask 3Ware or see the user manual. Note that many vendors do not disclose all the relevant data in the manual or on their site. That info: voltage (3.3V vs. 5V vs. universal), bus width (32-bit vs. 64-bit), and if 64-bit if the card will function in a 32-bit slot (some cards won't). Educational footnote: AGP is another one of those standards that went through the same nonsense (specifically 3.3V vs. 1.5V), except the situation was worse when some card manufacturers began selling 1.5V cards with incorrect notchings, resulting in smoke/fire when installed in a 3.3V slot. I have one such card, and keep it solely as a reminder of manufacturer/vendor idiocy. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 14:08:02 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B12DB373 for ; Mon, 3 Jun 2013 14:08:02 +0000 (UTC) (envelope-from michaelp@bsquare.com) Received: from owa.bsquare.com (vpn.bsquare.com [12.107.117.66]) by mx1.freebsd.org (Postfix) with ESMTP id 8C3041EF6 for ; Mon, 3 Jun 2013 14:08:01 +0000 (UTC) Received: from [10.150.16.15] (82.33.218.74) by BREAL.camelot.bsquare.com (192.168.100.67) with Microsoft SMTP Server (TLS) id 14.3.123.3; Mon, 3 Jun 2013 07:06:54 -0700 Message-ID: <51ACA2FD.2050609@bsquare.com> Date: Mon, 3 Jun 2013 15:06:53 +0100 From: Mike Pumford User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0 SeaMonkey/2.18 MIME-Version: 1.0 To: Subject: Re: 9.1-stable: ATI IXP600 AHCI: CAM timeout References: <201305291421.r4TELY8p042536@grabthar.secnetix.de> <1369840577.1258.45.camel@revolution.hippie.lan> In-Reply-To: <1369840577.1258.45.camel@revolution.hippie.lan> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [82.33.218.74] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 14:08:02 -0000 Ian Lepore wrote: > On Wed, 2013-05-29 at 16:21 +0200, Oliver Fromme wrote: >> Steven Hartland wrote: >> > Have you checked your sata cables and psu outputs? >> > >> > Both of these could be the underlying cause of poor signalling. >> >> I can't easily check that because it is a cheap rented >> server in a remote location. >> >> But I don't believe it is bad cabling or PSU anyway, or >> otherwise the problem would occur intermittently all the >> time if the load on the disks is sufficiently high. >> But it only occurs at tags=3 and above. At tags=2 it does >> not occur at all, no matter how hard I hammer on the disks. >> >> At the moment I'm inclined to believe that it is either >> a bug in the HDD firmware or in the controller. The disks >> aren't exactly new, they're 400 GB Samsung ones that are >> several years old. I think it's not uncommon to have bugs >> in the NCQ implementation in such disks. >> >> The only thing that puzzles me is the fact that the problem >> also disappears completely when I reduce the SATA rev from >> II to I, even at tags=32. >> > > It seems to me that you dismiss signaling problems too quickly. > Consider the possibilities... A bad cable leads to intermittant errors > at higher speeds. When NCQ is disabled or limited the software handles > these errors pretty much transparently. When NCQ is not limitted and > there are many outstanding requests, suddenly the error handling in the > software breaks down somehow and a minor recoverable problem becomes an > in-your-face error. > It could also be a software bug in the way CAM handles the failure of NCQ commands. When command queueing is used on a SCSI drive and a queued command fails only that command fails. A queued command failure on a SATA device fails ALL currently queued commands. I've not looked at the code but do the SATA CAM drivers do the right thing here? Less commands queued makes it less likely that multiple commands will be in progress when a failure occurs. A lower link rate also makes you more immune to signal failures. Mike From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 15:16:27 2013 Return-Path: Delivered-To: stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 61203D98; Mon, 3 Jun 2013 15:16:27 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id E9BB412B6; Mon, 3 Jun 2013 15:16:23 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1UjWVg-000ENh-GA; Mon, 03 Jun 2013 19:17:24 +0400 Date: Mon, 3 Jun 2013 19:17:24 +0400 From: Slawa Olhovchenkov To: Baptiste Daroussin Subject: Re: [HEADSUP] New pkg-devel 1.1.0 beta1 Message-ID: <20130603151724.GA54714@zxy.spb.ru> References: <20130530152053.GA19621@ithaqua.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130530152053.GA19621@ithaqua.etoilebsd.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: pkg@Freebsd.org, ports@FreeBSD.org, stable@FreeBSD.org, current@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 15:16:27 -0000 On Thu, May 30, 2013 at 05:20:54PM +0200, Baptiste Daroussin wrote: > The pkg developement team is proud to announce the new 1.1.0 beta1 release of > pkg. > - new experimental pkg convert (can convert from and to legacy pkg database) > pkg2ng now uses pkg convert (still recommanded to use pkg2ng) Converting packages from /var/db/pkg Converting pkg-1.1.0.b3_1... pkg: unknown keyword display, ignoring @display Installing pkg-1.1.0.b3_1...Segmentation fault (core dumped) From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 15:34:25 2013 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 16C67760; Mon, 3 Jun 2013 15:34:25 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-we0-x234.google.com (mail-we0-x234.google.com [IPv6:2a00:1450:400c:c03::234]) by mx1.freebsd.org (Postfix) with ESMTP id 2C145145E; Mon, 3 Jun 2013 15:34:24 +0000 (UTC) Received: by mail-we0-f180.google.com with SMTP id w56so1796301wes.39 for ; Mon, 03 Jun 2013 08:34:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=aLWwM1Ob8jzd/DyD697YoSGSW+XmULHlZjQNta2xVmM=; b=GigLVOGUh9TqQDSHX67Yx3CsM6jKWkx8oq07wZtzmOOfxhtXhfT1nMpTpIuVyl6Q+o 4/boMXeqJJpQBKRvA4qOFug0EZm7z50Og8lKNGkSGMqW38N4ZqrYTGQoKvzUK7hKPdHU 5eFVIBUmr+bhRrNu8pofpF+HvaV9qvLEWSWFMspIB5ZfUdEQXhoOWPmeUbhU/uv1Y7nh IuWJ/5iyhYQsv2BdnOjQGZwX6na+epKn1uCCrvHpHpibEbOUpya2fo+Y0m69P5zNTLl3 Zy/MLfbUQth4ESWUqQVE4/vZMcenG2rXiH26vaV0wNx0t0itmEja/8JTpQkLLspNGxii NQMw== X-Received: by 10.180.189.136 with SMTP id gi8mr13553695wic.11.1370273662764; Mon, 03 Jun 2013 08:34:22 -0700 (PDT) Received: from ithaqua.etoilebsd.net (ithaqua.etoilebsd.net. [37.59.37.188]) by mx.google.com with ESMTPSA id d5sm24226368wic.1.2013.06.03.08.34.21 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 03 Jun 2013 08:34:21 -0700 (PDT) Sender: Baptiste Daroussin Date: Mon, 3 Jun 2013 17:34:19 +0200 From: Baptiste Daroussin To: Slawa Olhovchenkov Subject: Re: [HEADSUP] New pkg-devel 1.1.0 beta1 Message-ID: <20130603153419.GL12427@ithaqua.etoilebsd.net> References: <20130530152053.GA19621@ithaqua.etoilebsd.net> <20130603151724.GA54714@zxy.spb.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4Y142/9l9nQlBiaj" Content-Disposition: inline In-Reply-To: <20130603151724.GA54714@zxy.spb.ru> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: pkg@Freebsd.org, ports@FreeBSD.org, stable@FreeBSD.org, current@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 15:34:25 -0000 --4Y142/9l9nQlBiaj Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 03, 2013 at 07:17:24PM +0400, Slawa Olhovchenkov wrote: > On Thu, May 30, 2013 at 05:20:54PM +0200, Baptiste Daroussin wrote: >=20 > > The pkg developement team is proud to announce the new 1.1.0 beta1 rele= ase of > > pkg. >=20 > > - new experimental pkg convert (can convert from and to legacy pkg data= base) > > pkg2ng now uses pkg convert (still recommanded to use pkg2ng) >=20 > Converting packages from /var/db/pkg > Converting pkg-1.1.0.b3_1... > pkg: unknown keyword display, ignoring @display > Installing pkg-1.1.0.b3_1...Segmentation fault (core dumped) >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" Have you run pkg2ng? regards, Bapt --4Y142/9l9nQlBiaj Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlGst3sACgkQ8kTtMUmk6ExXzgCgi+0tr450i9OveiYVM6GRa+i0 zZ4AoI6pJFuiGkDlfRcA2wJD8On2ebbC =tOSq -----END PGP SIGNATURE----- --4Y142/9l9nQlBiaj-- From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 15:38:03 2013 Return-Path: Delivered-To: stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D020FB2A; Mon, 3 Jun 2013 15:38:03 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 9FA6315E0; Mon, 3 Jun 2013 15:38:02 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1UjWqd-000EgW-NX; Mon, 03 Jun 2013 19:39:03 +0400 Date: Mon, 3 Jun 2013 19:39:03 +0400 From: Slawa Olhovchenkov To: Baptiste Daroussin Subject: Re: [HEADSUP] New pkg-devel 1.1.0 beta1 Message-ID: <20130603153903.GH34554@zxy.spb.ru> References: <20130530152053.GA19621@ithaqua.etoilebsd.net> <20130603151724.GA54714@zxy.spb.ru> <20130603153419.GL12427@ithaqua.etoilebsd.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130603153419.GL12427@ithaqua.etoilebsd.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: pkg@Freebsd.org, ports@FreeBSD.org, stable@FreeBSD.org, current@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 15:38:03 -0000 On Mon, Jun 03, 2013 at 05:34:19PM +0200, Baptiste Daroussin wrote: > On Mon, Jun 03, 2013 at 07:17:24PM +0400, Slawa Olhovchenkov wrote: > > On Thu, May 30, 2013 at 05:20:54PM +0200, Baptiste Daroussin wrote: > > > > > The pkg developement team is proud to announce the new 1.1.0 beta1 release of > > > pkg. > > > > > - new experimental pkg convert (can convert from and to legacy pkg database) > > > pkg2ng now uses pkg convert (still recommanded to use pkg2ng) > > > > Converting packages from /var/db/pkg > > Converting pkg-1.1.0.b3_1... > > pkg: unknown keyword display, ignoring @display > > Installing pkg-1.1.0.b3_1...Segmentation fault (core dumped) > > > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > Have you run pkg2ng? Yes, this is run pkg2ng. From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 15:47:53 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3ECFAF69 for ; Mon, 3 Jun 2013 15:47:53 +0000 (UTC) (envelope-from rwa@athabascau.ca) Received: from smtp.athabascau.ca (smtp.athabascau.ca [131.232.10.21]) by mx1.freebsd.org (Postfix) with ESMTP id 1A36916A8 for ; Mon, 3 Jun 2013 15:47:52 +0000 (UTC) Received: from CONVERSION-DAEMON.local.athabascau.ca by local.athabascau.ca (PMDF V6.2-1x12 #31425) id <0MNT0IY01QJRP8@local.athabascau.ca> for freebsd-stable@freebsd.org; Mon, 03 Jun 2013 09:47:51 -0600 (MDT) Received: from auwow.bogons ([192.168.1.2]) by local.athabascau.ca (PMDF V6.2-1x12 #31425) with ESMTPSA id <0MNT0JU96QJQFY@local.athabascau.ca> for freebsd-stable@freebsd.org; Mon, 03 Jun 2013 09:47:51 -0600 (MDT) Date: Mon, 03 Jun 2013 09:47:50 -0600 (MDT) From: Ross Alexander Subject: Re: freebsd-stable Digest, Vol 515, Issue 6 In-reply-to: X-X-Sender: rwa@auwow.bogons To: freebsd-stable@freebsd.org Message-id: Organization: Athabasca University X-Envelope-from: rwa@athabascau.ca MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) References: X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 15:47:53 -0000 Folks, Hate to follow myself up, but - on the one 9.1-STABLE machine where the disk i/o bogging down issue was a showstopper, I fixed it by reverting to 8.4-STABLE. Symptoms instantly went away, box became performant and responsive. regards, Ross -- Ross Alexander, (780) 675-6823 / (780) 689-0749, rwa@athabascau.ca "Always do right. This will gratify some people, and astound the rest." -- Samuel Clemens -- This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed. --- From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 16:08:02 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0E57671E for ; Mon, 3 Jun 2013 16:08:02 +0000 (UTC) (envelope-from rwa@athabascau.ca) Received: from smtp.athabascau.ca (smtp.athabascau.ca [131.232.10.21]) by mx1.freebsd.org (Postfix) with ESMTP id CD1261792 for ; Mon, 3 Jun 2013 16:08:01 +0000 (UTC) Received: from CONVERSION-DAEMON.local.athabascau.ca by local.athabascau.ca (PMDF V6.2-1x12 #31425) id <0MNT0NB01Q4THL@local.athabascau.ca> for freebsd-stable@freebsd.org; Mon, 03 Jun 2013 09:38:53 -0600 (MDT) Received: from auwow.bogons ([192.168.1.2]) by local.athabascau.ca (PMDF V6.2-1x12 #31425) with ESMTPSA id <0MNT0N9A7Q4L8M@local.athabascau.ca> for freebsd-stable@freebsd.org; Mon, 03 Jun 2013 09:38:52 -0600 (MDT) Date: Mon, 03 Jun 2013 09:38:45 -0600 (MDT) From: Ross Alexander Subject: 9.1-current disk throughput stalls ? X-X-Sender: rwa@auwow.bogons To: freebsd-stable@freebsd.org Message-id: Organization: Athabasca University X-Envelope-from: rwa@athabascau.ca MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) Cc: rwa@athabascau.ca X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 16:08:02 -0000 Folks, I wonder if anyone here has insight on a disk throughput problem that's come up over the last week or two. Now, I habitually run an 'svn up' and then rebuild world + kernel every Saturday morning on the home machines. It's all scripted and logged; I've been doing this for years and the process is very cut and dried. Saturday AM, I started it as usual - today it was still running, but only about 15% done. Normally it completes in 39 minutes, +/- 1 minute. What I've noticed is that disk performance on disk intensive stuff has gotten very flaky over the last two or three weeks. A buildworld, to pick an example, will run nicely for three to five minutes and then bog down. The disks stay busy, but forward progress slows to a crawl and then apparently stops. Individual cleandirs are taking five to ten seconds each on an otherwise unloaded machine. It feels like a vax-11/780 with 30 users and RA-80s, if anyone here remembers those days :). Here's a 'systat -vms': 5 users Load 0.30 0.30 0.27 Jun 3 09:07 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 84032 13908 1949112 40736 15071k count All 671192 16300 1076410k 61416 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 630 total 113 3573 29 113 630 83 26 26 zfod hdac1 16 ozfod xhci0 ehci 0.9%Sys 0.2%Intr 0.3%User 0.0%Nice 98.6%Idle %ozfod ohci0 ohci | | | | | | | | | | | daefr 93 emu10kx0 + prcfr 178 hpet0:t0 dtbuf 596 totfr hdac0 259 Namei Name-cache Dir-cache 329578 desvn react 359 ahci0 260 Calls hits % hits % 17505 numvn pdwak re0 261 475 294 62 14841 frevn pdpgs intrn Disks ada0 ada1 pass0 pass1 796676 wire KB/t 5.42 5.96 0.00 0.00 65484 act tps 197 192 0 0 45332 inact MB/s 1.04 1.12 0.00 0.00 cache %busy 74 82 0 0 15071692 free buf This is taken during the early stages of a builworld. The cleandir job steps are *crawling* along. Rattling the keyboard (USB or serial, although an SSH sessions seems to work sometimes as well) gets the buildworld doing some useful work again. Meanwhile, the apps load (which is two instances of WSPR, an instance of baudline, KDE, and a vncserver), which is soundcard I/O bound and does little to no disk I/O) runs along perfectly happily. The oldest kernel I have that shows the syndrome is - FreeBSD aukward.bogons 9.1-STABLE FreeBSD 9.1-STABLE #59 r250498: Sat May 11 00:03:15 MDT 2013 toor@aukward.bogons:/usr/obj/usr/src/sys/GENERIC amd64 H/W info - hw.machine: amd64 hw.model: AMD Phenom(tm) II X4 965 Processor hw.ncpu: 4 hw.physmem: 16883937280 hw.clockrate: 3411 kern.sched.name: ULE ahci0: port 0xa000-0xa007,0x9000-0x9003,\ 0x8000-0x8007,0x7000-0x7003,0x6000-0x600f mem 0xfe6ffc00-0xfe6fffff \ irq 19 at device 17.0 on pci0 ahci0: AHCI v1.20 with 6 6Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 [...] ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-6 SATA 1.x device ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 ada1 at ahcich2 bus 0 scbus2 target 0 lun 0 ada1: ATA-6 SATA 1.x device ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad8 I'm not paging, I don't have wild interrupt loads (checked with 'vmstat -i'), the ZFS pool is not in the middle of a scrub, but the machine has bad trivial response and buildworld doesn't get finished. I am seeing very similar behaviour on three other 9.1-current machines, all of which are AHCI/SATA setups, using both Seagate and WD disks (of random sizes and ages). All these boxes ran fine a month ago. BTW, when I do the rattle-keyboard-to-get-disks-going trick, the NFS daemon reports that the system clock slews badly - machine time drops behind wall clock time. Something is locking the clock update off. (Hmmm, I see I'm running a pre-5000/feature flags ZFS pool, FWTW. I'll run zpool upgrade, my bad.) regards, Ross -- Ross Alexander, (780) 675-6823 / (780) 689-0749, rwa@athabascau.ca "Always do right. This will gratify some people, and astound the rest." -- Samuel Clemens -- This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed. --- From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 18:40:36 2013 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D106F13E; Mon, 3 Jun 2013 18:40:36 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 006FE1F18; Mon, 3 Jun 2013 18:40:35 +0000 (UTC) Received: by mail-wi0-f172.google.com with SMTP id m6so3003463wiv.5 for ; Mon, 03 Jun 2013 11:40:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=AqERVold0oX+fyJ6/pElfeH8EuFokoR//spWAXdnExk=; b=Tu41oKJCSCP5NixABrUmL3sDai8fOejIoWi8jEMZlKJZ5INfqEPmpuMbQzUBG8r5V5 l9XmS9pl1CJHHfha4xtoA1AiyTGtXbaiJcF7kpGGQeXdM5nttnn7DkoZG2JRBVHeoMST pHG00VDpJYaq7wZi493ni4ujLQ7YZ+oMf81/OFK4SEqL5bvU+aWXtJvd6TE6R7TLWs0m jZk4+0ifukOqjw3vYW/G4//rHNIWLPW/8lFOr+hRKEkCCbaRhbi3uZilI1TiWbPYWQWl ww+JRyEH9uv+rdCz62E+dr+T/uhuSVyteQMh+cPgCcP9w7fNGlxKbBzjHPnmm4wLfUts Zx+g== X-Received: by 10.194.158.194 with SMTP id ww2mr20466668wjb.3.1370284834620; Mon, 03 Jun 2013 11:40:34 -0700 (PDT) Received: from ithaqua.etoilebsd.net (ithaqua.etoilebsd.net. [37.59.37.188]) by mx.google.com with ESMTPSA id dj7sm25302871wib.6.2013.06.03.11.40.33 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 03 Jun 2013 11:40:33 -0700 (PDT) Sender: Baptiste Daroussin Date: Mon, 3 Jun 2013 20:40:31 +0200 From: Baptiste Daroussin To: Slawa Olhovchenkov Subject: Re: [HEADSUP] New pkg-devel 1.1.0 beta1 Message-ID: <20130603184031.GM12427@ithaqua.etoilebsd.net> References: <20130530152053.GA19621@ithaqua.etoilebsd.net> <20130603151724.GA54714@zxy.spb.ru> <20130603153419.GL12427@ithaqua.etoilebsd.net> <20130603153903.GH34554@zxy.spb.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="gqEssfNGWsEa4HfM" Content-Disposition: inline In-Reply-To: <20130603153903.GH34554@zxy.spb.ru> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: pkg@Freebsd.org, ports@FreeBSD.org, stable@FreeBSD.org, current@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 18:40:36 -0000 --gqEssfNGWsEa4HfM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 03, 2013 at 07:39:03PM +0400, Slawa Olhovchenkov wrote: > On Mon, Jun 03, 2013 at 05:34:19PM +0200, Baptiste Daroussin wrote: >=20 > > On Mon, Jun 03, 2013 at 07:17:24PM +0400, Slawa Olhovchenkov wrote: > > > On Thu, May 30, 2013 at 05:20:54PM +0200, Baptiste Daroussin wrote: > > >=20 > > > > The pkg developement team is proud to announce the new 1.1.0 beta1 = release of > > > > pkg. > > >=20 > > > > - new experimental pkg convert (can convert from and to legacy pkg = database) > > > > pkg2ng now uses pkg convert (still recommanded to use pkg2ng) > > >=20 > > > Converting packages from /var/db/pkg > > > Converting pkg-1.1.0.b3_1... > > > pkg: unknown keyword display, ignoring @display > > > Installing pkg-1.1.0.b3_1...Segmentation fault (core dumped) > > >=20 > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd= =2Eorg" > >=20 > > Have you run pkg2ng? >=20 > Yes, this is run pkg2ng. Ok I'll have a look and fix asap. regards, Bapt --gqEssfNGWsEa4HfM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlGs4x8ACgkQ8kTtMUmk6EzFqQCff2/hQQ+wSQlXWiAp8IpaQZdm HGIAoJSN/MSJNRmoE8lELnOcWVhlY1Xd =Np7s -----END PGP SIGNATURE----- --gqEssfNGWsEa4HfM-- From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 18:53:53 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 18BBFC15 for ; Mon, 3 Jun 2013 18:53:53 +0000 (UTC) (envelope-from kentas@hush.com) Received: from smtp5.hushmail.com (smtp5a.hushmail.com [65.39.178.235]) by mx1.freebsd.org (Postfix) with ESMTP id 05EB11FF0 for ; Mon, 3 Jun 2013 18:53:52 +0000 (UTC) Received: from smtp5.hushmail.com (smtp5a.hushmail.com [65.39.178.235]) by smtp5.hushmail.com (Postfix) with SMTP id 0EF545808E for ; Mon, 3 Jun 2013 18:53:52 +0000 (UTC) Received: from smtp.hushmail.com (w7.hushmail.com [65.39.178.32]) by smtp5.hushmail.com (Postfix) with ESMTP; Mon, 3 Jun 2013 18:53:14 +0000 (UTC) Received: by smtp.hushmail.com (Postfix, from userid 99) id 5BB956F487; Mon, 3 Jun 2013 18:52:00 +0000 (UTC) MIME-Version: 1.0 Date: Mon, 03 Jun 2013 14:52:00 -0400 To: freebsd-stable@freebsd.org Subject: ZFS LZ4 Upgrade From: "Kenta Suzumoto" Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="UTF-8" Message-Id: <20130603185200.5BB956F487@smtp.hushmail.com> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 18:53:53 -0000 Hi. I'm planning on doing a ZFS root installation on a remote server very soon. The company only offers a 9.0 and 9.1 installation and "rescue" (nfs/pxe boot with ramdisk basically) system. I'd like to use LZ4 with the ZFS root pool, so I'm going to be upgrading to -STABLE once I have the initial system installed. Here's what I'll do: - install the 9.1 system - svn source, buildworld/kernel, install, reboot - upon booting the -STABLE system, begin enabling LZ4 compression on /usr/ports /usr/src etc. Will this work, or do I need to find some way to initially create the zpool with a -STABLE system? Is it just a matter of running "zfs upgrade" and "zpool upgrade" before enabling LZ4, or am I missing something? Thanks -kenta From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 19:24:58 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B1E92CF1; Mon, 3 Jun 2013 19:24:58 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from ns1.jnielsen.net (secure.freebsdsolutions.net [69.55.234.48]) by mx1.freebsd.org (Postfix) with ESMTP id 99BB91154; Mon, 3 Jun 2013 19:24:58 +0000 (UTC) Received: from [10.10.1.32] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by ns1.jnielsen.net (8.14.4/8.14.4) with ESMTP id r53J7oNX086079 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 3 Jun 2013 15:07:50 -0400 (EDT) (envelope-from lists@jnielsen.net) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: ZFS LZ4 Upgrade From: John Nielsen In-Reply-To: <20130603185200.5BB956F487@smtp.hushmail.com> Date: Mon, 3 Jun 2013 13:07:52 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20130603185200.5BB956F487@smtp.hushmail.com> To: Kenta Suzumoto X-Mailer: Apple Mail (2.1503) X-DCC-sonic.net-Metrics: ns1.jnielsen.net 1156; Body=3 Fuz1=3 Fuz2=3 X-Virus-Scanned: clamav-milter 0.97.5 at ns1.jnielsen.net X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 19:24:58 -0000 On Jun 3, 2013, at 12:52 PM, Kenta Suzumoto wrote: > Hi. I'm planning on doing a ZFS root installation on a remote server = very soon. The company only offers a 9.0 and 9.1 installation and = "rescue" (nfs/pxe boot with ramdisk basically) system. I'd like to use = LZ4 with the ZFS root pool, so I'm going to be upgrading to -STABLE once = I have the initial system installed. Here's what I'll do: >=20 > - install the 9.1 system > - svn source, buildworld/kernel, install, reboot > - upon booting the -STABLE system, begin enabling LZ4 compression on = /usr/ports /usr/src etc. >=20 > Will this work, or do I need to find some way to initially create the = zpool with a -STABLE system? Is it just a matter of running "zfs = upgrade" and "zpool upgrade" before enabling LZ4, or am I missing = something? Thanks That should work. Just keep in mind that blocks written before you = enable compression won't be compressed. So you may want to create a = _new_ ZFS for src (and ports if it already exists as well) after your = source upgrade, then copy the contents of /usr/src over to it. (Then = update the mountpoints as desired, etc.) JN From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 20:22:23 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6A1182E4; Mon, 3 Jun 2013 20:22:23 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id EC8981399; Mon, 3 Jun 2013 20:22:22 +0000 (UTC) Received: from mfilter5-d.gandi.net (mfilter5-d.gandi.net [217.70.178.132]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id 77C80A80B4; Mon, 3 Jun 2013 22:22:11 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter5-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter5-d.gandi.net (mfilter5-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id NqOKS5FkZWvs; Mon, 3 Jun 2013 22:22:09 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 34E9EA80C4; Mon, 3 Jun 2013 22:22:08 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id E818F73A1C; Mon, 3 Jun 2013 13:22:06 -0700 (PDT) Date: Mon, 3 Jun 2013 13:22:06 -0700 From: Jeremy Chadwick To: Mike Pumford Subject: Re: 9.1-stable: ATI IXP600 AHCI: CAM timeout Message-ID: <20130603202206.GA49602@icarus.home.lan> References: <201305291421.r4TELY8p042536@grabthar.secnetix.de> <1369840577.1258.45.camel@revolution.hippie.lan> <51ACA2FD.2050609@bsquare.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51ACA2FD.2050609@bsquare.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Alexander Motin , freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 20:22:23 -0000 On Mon, Jun 03, 2013 at 03:06:53PM +0100, Mike Pumford wrote: > Ian Lepore wrote: > >On Wed, 2013-05-29 at 16:21 +0200, Oliver Fromme wrote: > >>Steven Hartland wrote: > >> > Have you checked your sata cables and psu outputs? > >> > > >> > Both of these could be the underlying cause of poor signalling. > >> > >>I can't easily check that because it is a cheap rented > >>server in a remote location. > >> > >>But I don't believe it is bad cabling or PSU anyway, or > >>otherwise the problem would occur intermittently all the > >>time if the load on the disks is sufficiently high. > >>But it only occurs at tags=3 and above. At tags=2 it does > >>not occur at all, no matter how hard I hammer on the disks. > >> > >>At the moment I'm inclined to believe that it is either > >>a bug in the HDD firmware or in the controller. The disks > >>aren't exactly new, they're 400 GB Samsung ones that are > >>several years old. I think it's not uncommon to have bugs > >>in the NCQ implementation in such disks. > >> > >>The only thing that puzzles me is the fact that the problem > >>also disappears completely when I reduce the SATA rev from > >>II to I, even at tags=32. > >> > > > >It seems to me that you dismiss signaling problems too quickly. > >Consider the possibilities... A bad cable leads to intermittant errors > >at higher speeds. When NCQ is disabled or limited the software handles > >these errors pretty much transparently. When NCQ is not limitted and > >there are many outstanding requests, suddenly the error handling in the > >software breaks down somehow and a minor recoverable problem becomes an > >in-your-face error. > > > It could also be a software bug in the way CAM handles the failure > of NCQ commands. When command queueing is used on a SCSI drive and a > queued command fails only that command fails. A queued command > failure on a SATA device fails ALL currently queued commands. I've > not looked at the code but do the SATA CAM drivers do the right > thing here? Quoting T13/2015-D ATA8-ACS2 WD spec: "If an error occurs while the device is processing an NCQ command, then the device shall return command aborted for all NCQ commands that are in the queue and shall return command aborted for any new commands, except a READ LOG EXT command requesting log address 10h, until the device completes a READ LOG EXT command requesting log address 10h (i.e., reading the NCQ Command Error log) without error." While I can't easily provide an answer to your question, I can tell you that sys/dev/ahci/ahci.c does execute READ LOG EXT (command 0x2f) for certain scenarios (the code is in function ahci_issue_recovery()). The one person who can answer this question is mav@, who is now CC'd. > Less commands queued makes it less likely that multiple commands > will be in progress when a failure occurs. A lower link rate also > makes you more immune to signal failures. He isn't seeing SATA-level signal/link failure; the AHCI driver would complain about that, and those messages aren't there. Unless, of course, those messages are only visible when verbose booting is enabled (I hope not). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 20:32:09 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 09A204D8 for ; Mon, 3 Jun 2013 20:32:09 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by mx1.freebsd.org (Postfix) with ESMTP id 9E89D1478 for ; Mon, 3 Jun 2013 20:32:08 +0000 (UTC) Received: from mfilter1-d.gandi.net (mfilter1-d.gandi.net [217.70.178.130]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id 0DFED41C06C; Mon, 3 Jun 2013 22:31:51 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter1-d.gandi.net Received: from relay5-d.mail.gandi.net ([217.70.183.197]) by mfilter1-d.gandi.net (mfilter1-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id 7fKVDRDcTqED; Mon, 3 Jun 2013 22:31:49 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id C7BAF41C078; Mon, 3 Jun 2013 22:31:48 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id BDA5773A1C; Mon, 3 Jun 2013 13:31:46 -0700 (PDT) Date: Mon, 3 Jun 2013 13:31:46 -0700 From: Jeremy Chadwick To: Ross Alexander Subject: Re: 9.1-current disk throughput stalls ? Message-ID: <20130603203146.GB49602@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 20:32:09 -0000 On Mon, Jun 03, 2013 at 09:38:45AM -0600, Ross Alexander wrote: > I wonder if anyone here has insight on a disk throughput problem > that's come up over the last week or two. Now, I habitually run an > 'svn up' and then rebuild world + kernel every Saturday morning on the > home machines. It's all scripted and logged; I've been doing this for > years and the process is very cut and dried. Saturday AM, I started > it as usual - today it was still running, but only about 15% done. > Normally it completes in 39 minutes, +/- 1 minute. > > What I've noticed is that disk performance on disk intensive stuff has > gotten very flaky over the last two or three weeks. A buildworld, to > pick an example, will run nicely for three to five minutes and then > bog down. The disks stay busy, but forward progress slows to a crawl > and then apparently stops. Individual cleandirs are taking five to > ten seconds each on an otherwise unloaded machine. It feels like > a vax-11/780 with 30 users and RA-80s, if anyone here remembers those > days :). > > Here's a 'systat -vms': > > 5 users Load 0.30 0.30 0.27 Jun 3 09:07 > > Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER > Tot Share Tot Share Free in out in out > Act 84032 13908 1949112 40736 15071k count > All 671192 16300 1076410k 61416 pages > Proc: Interrupts > r p d s w Csw Trp Sys Int Sof Flt cow 630 total > 113 3573 29 113 630 83 26 26 zfod hdac1 16 > ozfod xhci0 ehci > 0.9%Sys 0.2%Intr 0.3%User 0.0%Nice 98.6%Idle %ozfod ohci0 ohci > | | | | | | | | | | | daefr 93 emu10kx0 > + prcfr 178 hpet0:t0 > dtbuf 596 totfr hdac0 259 > Namei Name-cache Dir-cache 329578 desvn react 359 ahci0 260 > Calls hits % hits % 17505 numvn pdwak re0 261 > 475 294 62 14841 frevn pdpgs > intrn > Disks ada0 ada1 pass0 pass1 796676 wire > KB/t 5.42 5.96 0.00 0.00 65484 act > tps 197 192 0 0 45332 inact > MB/s 1.04 1.12 0.00 0.00 cache > %busy 74 82 0 0 15071692 free > buf > > This is taken during the early stages of a builworld. The cleandir > job steps are *crawling* along. Rattling the keyboard (USB or serial, > although an SSH sessions seems to work sometimes as well) gets the > buildworld doing some useful work again. Meanwhile, the apps load > (which is two instances of WSPR, an instance of baudline, KDE, and a > vncserver), which is soundcard I/O bound and does little to no disk > I/O) runs along perfectly happily. > > The oldest kernel I have that shows the syndrome is - > > FreeBSD aukward.bogons 9.1-STABLE FreeBSD 9.1-STABLE #59 r250498: > Sat May 11 00:03:15 MDT 2013 > toor@aukward.bogons:/usr/obj/usr/src/sys/GENERIC amd64 > > H/W info - > > hw.machine: amd64 > hw.model: AMD Phenom(tm) II X4 965 Processor > hw.ncpu: 4 > hw.physmem: 16883937280 > hw.clockrate: 3411 > kern.sched.name: ULE > > ahci0: port 0xa000-0xa007,0x9000-0x9003,\ > 0x8000-0x8007,0x7000-0x7003,0x6000-0x600f mem 0xfe6ffc00-0xfe6fffff \ > irq 19 at device 17.0 on pci0 > ahci0: AHCI v1.20 with 6 6Gbps ports, Port Multiplier supported > ahcich0: at channel 0 on ahci0 > [...] > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ATA-6 SATA 1.x device > ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) > ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad4 > ada1 at ahcich2 bus 0 scbus2 target 0 lun 0 > ada1: ATA-6 SATA 1.x device > ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) > ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) > ada1: Previously was known as ad8 > > I'm not paging, I don't have wild interrupt loads (checked with > 'vmstat -i'), the ZFS pool is not in the middle of a scrub, but the > machine has bad trivial response and buildworld doesn't get finished. > I am seeing very similar behaviour on three other 9.1-current > machines, all of which are AHCI/SATA setups, using both Seagate and WD > disks (of random sizes and ages). All these boxes ran fine a month > ago. > > BTW, when I do the rattle-keyboard-to-get-disks-going trick, the NFS > daemon reports that the system clock slews badly - machine time drops > behind wall clock time. Something is locking the clock update off. > > (Hmmm, I see I'm running a pre-5000/feature flags ZFS pool, FWTW. > I'll run zpool upgrade, my bad.) 1. There is no such thing as 9.1-CURRENT. Either you meant 9.1-STABLE (what should be called stable/9) or -CURRENT (what should be called head). 2. Is there some reason you excluded details of your ZFS setup? "zpool status" would be a good start. 3. Do any of your filesystems/pools have ZFS compression enabled, or have in the past? 4. Do any of your filesystems/pools have ZFS dedup enabled, or have in the past? 5. Does the problem go away after a reboot? 6. Can you provide smartctl -x output for both ada0 and ada1? You will need to install ports/sysutils/smartmontools for this. The reason I'm asking for this is there may be one of your disks which is causing I/O transactions to stall for the entire pool (i.e. "single point of annoyance"). 7. Can you remove ZFS from the picture entirely (use UFS only) and re-test? My guess is that this is ZFS behaviour, particularly the ARC being flushed to disk, and your disks are old/slow. (Meaning: you have 16GB RAM + 4 core CPU but with very old disks). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 20:51:26 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 77235EB4 for ; Mon, 3 Jun 2013 20:51:26 +0000 (UTC) (envelope-from lukasz@gruner.lu) Received: from new2-smtp.messagingengine.com (new2-smtp.messagingengine.com [66.111.4.224]) by mx1.freebsd.org (Postfix) with ESMTP id 51C2C16CF for ; Mon, 3 Jun 2013 20:51:25 +0000 (UTC) Received: from compute5.internal (compute5.nyi.mail.srv.osa [10.202.2.45]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id D26A41EAA for ; Mon, 3 Jun 2013 16:42:47 -0400 (EDT) Received: from web3.nyi.mail.srv.osa ([10.202.2.213]) by compute5.internal (MEProxy); Mon, 03 Jun 2013 16:42:47 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:subject:date; s=smtpout; bh=DP1hLPeboNZWnlh7sLPGZcOqN7w=; b=uNrLtdkeaH2zgNbtOhoBzrTLu69z iDfpLJ8nhg+3Qg3TGEC1j6nUe1GEFgaU7NmBxBLUGe7J6MVx/q6NS7r/mMswfyJT GwAXMKsiAZtMOeBazqVADn5ePAS5pvuqnmewtQpYaqnzSjPfI3nXwXj3uK8Yqfe/ nmNDSIkkXM4h3uw= Received: by web3.nyi.mail.srv.osa (Postfix, from userid 99) id 86142C407A8; Mon, 3 Jun 2013 16:42:47 -0400 (EDT) Message-Id: <1370292167.23190.140661239372221.58EEC740@webmail.messagingengine.com> X-Sasl-Enc: ANLuLEy5QVmCp1TWM1WxTZo3YP1mQ04ApZNfxWycgAzH 1370292167 From: =?UTF-8?Q?=C5=81ukasz=20Gruner?= To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-8d2b2306 Subject: can not build xf86-video-intel Date: Mon, 03 Jun 2013 22:42:47 +0200 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 20:51:26 -0000 Hi, Recently I switched from using ports only to packages and now trying to enable kms. I've rebuild libdrm, but xf86-video-intel fails with the following error (any ideas how to fix that?): CC sna_display_fake.lo CC sna_driver.lo sna_driver.c:437:2: error: implicit declaration of function 'xf86getBoolValue' is invalid in C99 [-Werror,-Wimplicit-function-declaration] xf86getBoolValue(&val, xf86GetOptValString(sna->Options, id)); ^ 1 error generated. *** [sna_driver.lo] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-2.21.6/src/sna. *** [all-recursive] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-2.21.6/src/sna. *** [all-recursive] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-2.21.6/src. *** [all-recursive] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-2.21.6. *** [all] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-2.21.6. *** [do-build] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel. *** [reinstall] Error code 1 Stop in /usr/ports/x11-drivers/xf86-video-intel. From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 20:59:47 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 231C141A for ; Mon, 3 Jun 2013 20:59:47 +0000 (UTC) (envelope-from S.Kuzminsky@F5.com) Received: from mail.f5.com (mail.f5.com [208.85.209.139]) by mx1.freebsd.org (Postfix) with ESMTP id 028BB1745 for ; Mon, 3 Jun 2013 20:59:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=f5.com; i=S.Kuzminsky@f5.com; q=dns/txt; s=seattle; t=1370293187; x=1401829187; h=from:to:subject:date:message-id:content-id: content-transfer-encoding:mime-version; bh=tTaAFghoq9FrEIvLY3SEgVbb7AH/kIob7FxnvbypGss=; b=mZff4w9titn8XUqH2QboGGwuewCBWJmKEROsZPdTS49gAALhuDJpTL8S pH5i6n8o8AtAEeKvoC7MT7JI0tyWrmcebeJXqb7bk+ZXqFY/uGdlUFyqA +G9CFIF/EOI0388RAcYp3oJA2QCUG0vV+2nKIhSGXob8k4U345WDU0srz 0=; X-IronPort-AV: E=Sophos;i="4.87,795,1363132800"; d="scan'208";a="73633278" Received: from unknown (HELO exchmail.f5net.com) ([192.168.10.240]) by mail.f5.com with ESMTP/TLS/AES128-SHA; 03 Jun 2013 20:59:47 +0000 Received: from SEAEMBX01.olympus.F5Net.com ([fe80::3440:4256:38f6:d3a0]) by SEAECAS01.olympus.F5Net.com ([::1]) with mapi id 14.03.0123.003; Mon, 3 Jun 2013 13:59:46 -0700 From: Sebastian Kuzminsky To: "freebsd-stable@freebsd.org" Subject: bce kernel page faults and NMIs (was: Strange reboot since 9.1) Thread-Topic: bce kernel page faults and NMIs (was: Strange reboot since 9.1) Thread-Index: AQHOYJ1Gh9i9etiV2EChINQoNPiE0g== Date: Mon, 3 Jun 2013 20:59:45 +0000 Message-ID: <73664E92-986C-4582-9A43-EFA576931318@f5.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.16.250] Content-Type: text/plain; charset="us-ascii" Content-ID: <681DA93BF6892F47A394A98001995E11@F5.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 20:59:47 -0000 Howdy folks, this email is a follow-on to a 3-month-old thread about kernel= page faults from the bce driver[0]. 0: http://lists.freebsd.org/pipermail/freebsd-stable/2013-March/072713.htm= l Sorry to revive such an old thread, but a couple of bits of new information= has come to light here that may be useful for others. The header splitting suggestion that Marius Strobl made[1] did fix the ker= nel page fault rooted in bce_intr() that we were seeing (and that other fol= ks reported in the original thread). I'm no bce expert, but it looks to me= like the bce driver does not apply the same flow control to its page queue= as it does to its receive queue, maybe that's related to the problem? 1: http://lists.freebsd.org/pipermail/freebsd-stable/2013-March/072766.htm= l After disabling bce header splitting we stopped getting kernel page faults,= but we still had problems with this NIC (Broadcom NetXtreme II BCM5716 Gig= abit Ethernet) producing frequent PCI errors and occasional NMIs. I found this thread[2] that suggests that the NIC firmware version may be r= elevant to the NMI problem. The Red Hat people are reporting that firmware= version 6.0.1 is bad and 6.4.5 is good; 9.1 ships with 6.0.17, so who know= s what that means... We ended up reverting to the bce driver from FreeBSD = 7 and that fixed our NMI problems. (The bce driver from FreeBSD 7 also has= header splitting disabled by default: Bonus!) 2: https://bugzilla.redhat.com/show_bug.cgi?id=3D693542 --=20 Sebastian Kuzminsky= From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 21:48:33 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8486C83B for ; Mon, 3 Jun 2013 21:48:33 +0000 (UTC) (envelope-from rwa@athabascau.ca) Received: from smtp.athabascau.ca (smtp.athabascau.ca [131.232.10.21]) by mx1.freebsd.org (Postfix) with ESMTP id 4943519F2 for ; Mon, 3 Jun 2013 21:48:32 +0000 (UTC) Received: from CONVERSION-DAEMON.local.athabascau.ca by local.athabascau.ca (PMDF V6.2-1x12 #31425) id <0MNU0KJ0178UXA@local.athabascau.ca> for freebsd-stable@freebsd.org; Mon, 03 Jun 2013 15:48:31 -0600 (MDT) Received: from autopsy.pc.athabascau.ca ([131.232.4.80]) by local.athabascau.ca (PMDF V6.2-1x12 #31425) with ESMTPS id <0MNU0MC5Q78UGZ@local.athabascau.ca>; Mon, 03 Jun 2013 15:48:30 -0600 (MDT) Date: Mon, 03 Jun 2013 15:48:30 -0600 (MDT) From: Ross Alexander Subject: Re: 9.1-current disk throughput stalls ? In-reply-to: <20130603203146.GB49602@icarus.home.lan> X-X-Sender: rwa@autopsy.pc.athabascau.ca To: Jeremy Chadwick Message-id: Organization: Athabasca University X-Envelope-from: rwa@athabascau.ca MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) References: <20130603203146.GB49602@icarus.home.lan> Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 21:48:33 -0000 On Mon, 3 Jun 2013, Jeremy Chadwick wrote: > 1. There is no such thing as 9.1-CURRENT. Either you meant 9.1-STABLE > (what should be called stable/9) or -CURRENT (what should be called > head). > I wrote: >> The oldest kernel I have that shows the syndrome is - >> >> FreeBSD aukward.bogons 9.1-STABLE FreeBSD 9.1-STABLE #59 r250498: >> Sat May 11 00:03:15 MDT 2013 >> toor@aukward.bogons:/usr/obj/usr/src/sys/GENERIC amd64 See above. You're right, I shouldn't post after a 07:00 dentist's appt while my spouse is worrying me about the ins adjustor's report on the car damage :(. Hey, I'm very fallible. I'll try harder. > 2. Is there some reason you excluded details of your ZFS setup? > "zpool status" would be a good start. Thanks for the useful hint as to what info you need to diagnose. One of the machines ran a 5 drive zraid-1 pool (Mnemosyne). Another was a 2 drive gmirror, in the simplest possible gpart/gmirror setup. (Mnemosyne-sub-1.) The third is a 2 drive ZFS raid-1, again in the simplest possible gpart/gmirror manner (Aukward). The fourth is a conceptually identical 2 drive ZFS raid-1, swapping to a zvol (Griffon.) If you look on the FreeBSD wiki, the pages that say "bootable zfs gptzfsboot" and "bootable mirror" - https://wiki.freebsd.org/RootOnZFS http://www.freebsdwiki.net/index.php/RAID1,_Software,_How_to_setup Well, I just followed those in cookbook style (modulo device and pool names). Didn't see any reason to be creative; I build for reliability, not performance. Aukward is gpart/zfs raid-1 box #1: aukward:/u0/rwa > ls -l /dev/gpt total 0 crw-r----- 1 root operator 0x91 Jun 3 10:18 vol0 crw-r----- 1 root operator 0x8e Jun 3 10:18 vol1 aukward:/u0/rwa > zpool list -v NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT ult_root 111G 108G 2.53G 97% 1.00x ONLINE - mirror 111G 108G 2.53G - gpt/vol0 - - - - gpt/vol1 - - - - aukward:/u0/rwa > zpool status pool: ult_root state: ONLINE scan: scrub repaired 0 in 1h13m with 0 errors on Sun May 5 04:29:30 2013 config: NAME STATE READ WRITE CKSUM ult_root ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/vol0 ONLINE 0 0 0 gpt/vol1 ONLINE 0 0 0 errors: No known data errors (Yes, that machine has no swap. Has NEVER had swap, has 16 GB and uses maybe 10% at max load. Has been running 9.x since prerelease days, FWTW. The ARC is throttled to 2 GB; zfs-stats says I never get near using even that. It's just the box that drives the radios, a ham radio hobby machine.) Griffon is also gpart/zfs raid-1 - griffon:/u0/rwa > uname -a FreeBSD griffon.cs.athabascau.ca 9.1-STABLE FreeBSD 9.1-STABLE #25 r251062M: Tue May 28 10:39:13 MDT 2013 toor@griffon.cs.athabascau.ca:/usr/obj/usr/src/sys/GENERIC amd64 griffon:/u0/rwa > ls -l /dev/gpt total 0 crw-r----- 1 root operator 0x7b Jun 3 08:38 disk0 crw-r----- 1 root operator 0x80 Jun 3 08:38 disk1 crw-r----- 1 root operator 0x79 Jun 3 08:38 swap0 crw-r----- 1 root operator 0x7e Jun 3 08:38 swap1 and the pool is fat and happy - griffon:/u0/rwa > zpool status -v pool: pool0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool0 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors Note that swap is through ZFS zvol; griffon:/u0/rwa > cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# # # /dev/zvol/pool0/swap none swap sw 0 0 pool0 / zfs rw 0 0 pool0/tmp /tmp zfs rw 0 0 pool0/var /var zfs rw 0 0 pool0/usr /usr zfs rw 0 0 pool0/u0 /u0 zfs rw 0 0 /dev/cd0 /cdrom cd9660 ro,noauto 0 0 /dev/ada2s1d /mnt0 ufs rw,noauto 0 0 /dev/da0s1 /u0/rwa/camera msdosfs rw,noauto 0 0 The machine has 32 GB and never swaps. It runs virtualbox loads, anything from one to forty virtuals (little OpenBSD images.) Load is always light. As for the zraid-5 box (Mnemosyne), I first replaced the ZFS pool with a simple gpart/gmirror. The drives gmirrored are known to be good. That *also* ran like mud. Then I downgraded to 8.4-STABLE, GENERIC kernel, and it's just fine now thanks. I have the five zraid-1 disks that were pulled sitting in a second 4 core server chassis, on my desk, and they fail in that machine in the same way that the production box died. I'm 150 km away and the power went down over the weekend at the remote site so I'll have to wait until tomorrow to send you those details. For now, think cut-and-paste from freebsd wiki, nothing clever, everything as simple as possible. Film at 11. > 3. Do any of your filesystems/pools have ZFS compression enabled, or > have in the past? No; disk is too cheap to bother with that. > 4. Do any of your filesystems/pools have ZFS dedup enabled, or have in > the past? No; disk is too cheap to bother with that. > 5. Does the problem go away after a reboot? It goes away for a few minutes, and then comes back on little cat feet. Gradual slowdown. > 6. Can you provide smartctl -x output for both ada0 and ada1? You will > need to install ports/sysutils/smartmontools for this. The reason I'm > asking for this is there may be one of your disks which is causing I/O > transactions to stall for the entire pool (i.e. "single point of > annoyance"). Been down that path, good call, Mnemosyne (zraid-1) checked clean as a whistle. (Later) Griffon checks out clean, too. Both -x and -a. Aukward might have an iffy device, I will sched some self tests and post everything, all neatly tabulated. I've already fought a bad disk, and also just-slighly-iffy cables, in a ZFS context and that time was nothing like this one. > 7. Can you remove ZFS from the picture entirely (use UFS only) and > re-test? My guess is that this is ZFS behaviour, particularly the ARC > being flushed to disk, and your disks are old/slow. (Meaning: you have > 16GB RAM + 4 core CPU but with very old disks). Already did that. A gmirror 9.1 (Mnemosyne-sub-1) box slowly choked and died just like the ZFS instance did. An 8.4-STABLE back-rev without hardware changes was the fix. Also: I noticed that when I mounted the 9.1 zraid from an 8.4 flash fixit disk, everything ran quickly and stably. I did copies of about 635 GB worth of ~3 GB sized .pcap files out of the zraid onto a SCSI UFS and the ZFS disks were all about 75 to 80% busy for the ~8000 seconds the copy was running. No slowdowns, no stalls. BTW, I'd like to thank you for your kind interest, and please forgive my poor reporting skills - I'm at home, work is 150 k away, the phone keeps ringing, there are a lot of boxes, I'm sleep deprived, whine & snivel, grumble & moan ;) regards, Ross -- Ross Alexander, (780) 675-6823 desk / (780) 689-0749 cell, rwa@athabascau.ca "Always do right. This will gratify some people, and astound the rest." -- Samuel Clemens -- This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed. --- From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 22:34:42 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4094DD96 for ; Mon, 3 Jun 2013 22:34:42 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 9899E1C22 for ; Mon, 3 Jun 2013 22:34:41 +0000 (UTC) Received: from mfilter16-d.gandi.net (mfilter16-d.gandi.net [217.70.178.144]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id AB924A80B6; Tue, 4 Jun 2013 00:34:30 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter16-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter16-d.gandi.net (mfilter16-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id fKpDafzGWzRf; Tue, 4 Jun 2013 00:34:28 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 0F75BA80B5; Tue, 4 Jun 2013 00:34:27 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 0BC5D73A1C; Mon, 3 Jun 2013 15:34:26 -0700 (PDT) Date: Mon, 3 Jun 2013 15:34:26 -0700 From: Jeremy Chadwick To: Ross Alexander Subject: Re: 9.1-current disk throughput stalls ? Message-ID: <20130603223425.GA51402@icarus.home.lan> References: <20130603203146.GB49602@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 22:34:42 -0000 On Mon, Jun 03, 2013 at 03:48:30PM -0600, Ross Alexander wrote: > On Mon, 3 Jun 2013, Jeremy Chadwick wrote: > > >1. There is no such thing as 9.1-CURRENT. Either you meant 9.1-STABLE > >(what should be called stable/9) or -CURRENT (what should be called > >head). > > >I wrote: > >>The oldest kernel I have that shows the syndrome is - > >> > >> FreeBSD aukward.bogons 9.1-STABLE FreeBSD 9.1-STABLE #59 r250498: > >> Sat May 11 00:03:15 MDT 2013 > >> toor@aukward.bogons:/usr/obj/usr/src/sys/GENERIC amd64 > > See above. You're right, I shouldn't post after a 07:00 dentist's > appt while my spouse is worrying me about the ins adjustor's report > on the car damage :(. Hey, I'm very fallible. I'll try harder. > > >2. Is there some reason you excluded details of your ZFS setup? > >"zpool status" would be a good start. > > Thanks for the useful hint as to what info you need to diagnose. > > One of the machines ran a 5 drive zraid-1 pool (Mnemosyne). > > Another was a 2 drive gmirror, in the simplest possible gpart/gmirror setup. > (Mnemosyne-sub-1.) > > The third is a 2 drive ZFS raid-1, again in the simplest possible > gpart/gmirror manner (Aukward). > > The fourth is a conceptually identical 2 drive ZFS raid-1, swapping > to a zvol (Griffon.) > > If you look on the FreeBSD wiki, the pages that say "bootable zfs > gptzfsboot" and "bootable mirror" - > > https://wiki.freebsd.org/RootOnZFS > http://www.freebsdwiki.net/index.php/RAID1,_Software,_How_to_setup > > Well, I just followed those in cookbook style (modulo device and pool > names). Didn't see any reason to be creative; I build for > reliability, not performance. > > Aukward is gpart/zfs raid-1 box #1: > > aukward:/u0/rwa > ls -l /dev/gpt > total 0 > crw-r----- 1 root operator 0x91 Jun 3 10:18 vol0 > crw-r----- 1 root operator 0x8e Jun 3 10:18 vol1 > > aukward:/u0/rwa > zpool list -v > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > ult_root 111G 108G 2.53G 97% 1.00x ONLINE - > mirror 111G 108G 2.53G - > gpt/vol0 - - - - > gpt/vol1 - - - - > > aukward:/u0/rwa > zpool status > pool: ult_root > state: ONLINE > scan: scrub repaired 0 in 1h13m with 0 errors on Sun May 5 04:29:30 2013 > config: > > NAME STATE READ WRITE CKSUM > ult_root ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/vol0 ONLINE 0 0 0 > gpt/vol1 ONLINE 0 0 0 > > errors: No known data errors > > (Yes, that machine has no swap. Has NEVER had swap, has 16 GB and > uses maybe 10% at max load. Has been running 9.x since prerelease > days, FWTW. The ARC is throttled to 2 GB; zfs-stats says I never get > near using even that. It's just the box that drives the radios, > a ham radio hobby machine.) > > Griffon is also gpart/zfs raid-1 - > > griffon:/u0/rwa > uname -a > FreeBSD griffon.cs.athabascau.ca 9.1-STABLE FreeBSD 9.1-STABLE #25 r251062M: > Tue May 28 10:39:13 MDT 2013 > toor@griffon.cs.athabascau.ca:/usr/obj/usr/src/sys/GENERIC > amd64 > > griffon:/u0/rwa > ls -l /dev/gpt > total 0 > crw-r----- 1 root operator 0x7b Jun 3 08:38 disk0 > crw-r----- 1 root operator 0x80 Jun 3 08:38 disk1 > crw-r----- 1 root operator 0x79 Jun 3 08:38 swap0 > crw-r----- 1 root operator 0x7e Jun 3 08:38 swap1 > > and the pool is fat and happy - > > griffon:/u0/rwa > zpool status -v > pool: pool0 > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > pool0 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/disk0 ONLINE 0 0 0 > gpt/disk1 ONLINE 0 0 0 > > errors: No known data errors > > Note that swap is through ZFS zvol; > > griffon:/u0/rwa > cat /etc/fstab > # Device Mountpoint FStype Options Dump Pass# > # > # > /dev/zvol/pool0/swap none swap sw 0 0 > > pool0 / zfs rw 0 0 > pool0/tmp /tmp zfs rw 0 0 > pool0/var /var zfs rw 0 0 > pool0/usr /usr zfs rw 0 0 > pool0/u0 /u0 zfs rw 0 0 > > /dev/cd0 /cdrom cd9660 ro,noauto 0 0 > /dev/ada2s1d /mnt0 ufs rw,noauto 0 0 > /dev/da0s1 /u0/rwa/camera msdosfs rw,noauto 0 0 > > The machine has 32 GB and never swaps. It runs virtualbox loads, anything > from one to forty virtuals (little OpenBSD images.) Load is always light. > > As for the zraid-5 box (Mnemosyne), I first replaced the ZFS pool with > a simple gpart/gmirror. The drives gmirrored are known to be good. That > *also* ran like mud. Then I downgraded to 8.4-STABLE, GENERIC kernel, > and it's just fine now thanks. > > I have the five zraid-1 disks that were pulled sitting in a second 4 > core server chassis, on my desk, and they fail in that machine in the > same way that the production box died. I'm 150 km away and the power > went down over the weekend at the remote site so I'll have to wait > until tomorrow to send you those details. > > For now, think cut-and-paste from freebsd wiki, nothing clever, > everything as simple as possible. Film at 11. > > >3. Do any of your filesystems/pools have ZFS compression enabled, or > >have in the past? > > No; disk is too cheap to bother with that. > > >4. Do any of your filesystems/pools have ZFS dedup enabled, or have in > >the past? > > No; disk is too cheap to bother with that. > > >5. Does the problem go away after a reboot? > > It goes away for a few minutes, and then comes back on little cat feet. > Gradual slowdown. > > >6. Can you provide smartctl -x output for both ada0 and ada1? You will > >need to install ports/sysutils/smartmontools for this. The reason I'm > >asking for this is there may be one of your disks which is causing I/O > >transactions to stall for the entire pool (i.e. "single point of > >annoyance"). > > Been down that path, good call, Mnemosyne (zraid-1) checked clean as a > whistle. (Later) Griffon checks out clean, too. Both -x and -a. > Aukward might have an iffy device, I will sched some self tests and > post everything, all neatly tabulated. > > I've already fought a bad disk, and also just-slighly-iffy cables, > in a ZFS context and that time was nothing like this one. > > >7. Can you remove ZFS from the picture entirely (use UFS only) and > >re-test? My guess is that this is ZFS behaviour, particularly the ARC > >being flushed to disk, and your disks are old/slow. (Meaning: you have > >16GB RAM + 4 core CPU but with very old disks). > > Already did that. A gmirror 9.1 (Mnemosyne-sub-1) box slowly choked > and died just like the ZFS instance did. An 8.4-STABLE back-rev > without hardware changes was the fix. > > Also: I noticed that when I mounted the 9.1 zraid from an 8.4 flash > fixit disk, everything ran quickly and stably. I did copies of about > 635 GB worth of ~3 GB sized .pcap files out of the zraid onto a SCSI > UFS and the ZFS disks were all about 75 to 80% busy for the ~8000 > seconds the copy was running. No slowdowns, no stalls. > > BTW, I'd like to thank you for your kind interest, and please forgive > my poor reporting skills - I'm at home, work is 150 k away, the phone > keeps ringing, there are a lot of boxes, I'm sleep deprived, whine & > snivel, grumble & moan ;) All the above information is almost "too much". There are now multiple machines with multiple hardware devices (disks, controllers, etc.) and different setups to try and figure out. Each/every situation (system, etc.) needs to be analysed individually. So let's please focus on the one you called "aukward", because it's the one we have some details for. Please do not involve the other systems at this point in time. What we know at this point: 1. OS is amd64 [1] 2. System uses a 4-core CPU and has 16GB RAM [1] 3. Uses AHCI, driven by an ATI/AMD IXP700 [1] 4. Has two disks: ada0 and ada1, both of which are very old/slow WD1200JD (120GB, SATA150, 8MB cache, 512-byte sectors, 7200rpm); I have used these disks, so I speak from experience when I say old/slow [1] 5. Both disks use GPT partitioning [2], but partition layout we don't know the layout ("gpart show {ada0,ada1}" would be helpful), 6. ZFS is involved [1][2] 7. ZFS setup is a mirror (RAID-1-like), 8. Root filesystem uses ZFS, but we don't know what your filesystem layouts look like ("zfs get all" and "df -k" would be helpful) [2] 9. Compression nor dedup are used (good!!!) [2] 10. System does not use swap [2] 11. ARC is "throttled" to 2GB, but we don't know how you did this. I really need to see your sysctl.conf and loader.conf tunings [2] 12. Rolling back to 8.4-STABLE (date/build unknown) apparently fixes your issue (I would appreciate you running the system for 72 hours before making this statement, and doing the *exact same things* on it that cause the problem with 9.1-STABLE) [2] 13. Rebooting the system causes I/O to be fast again, for a little while, then gradually gets worse [2]. Pending things: i) Need data from #5 above ii) Need data from #8 above iii) Need data from #11 above iv) I still want to see smartctl -x output. I do not need you to "run self-tests" -- respectfully please just do what I ask. Most people do not know how to interpret/understand SMART results v) I really wish you would not have rolled this system back to 8.4-STABLE. For anyone to debug this, we need the system in a consistent state. Changing kernels/etc. vi) Would appreciate seeing "sysctl -a | grep zfs" when the I/O is fast (immediately after a reboot is fine) and again when the I/O is very slow. I do not care about "zfs-stats". vii) dmesg would also be useful (put it up on pastebin if you want). Please be aware the FreeBSD Wiki on ZFS is known to be outdated in many regards. I won't go into details. I have so many gut feelings at this point about your problem that is it almost unbearable. The possibilities are near endless at this point. Answers to the above could help narrow them down. Finally, wanted to briefly mention that your [2] repeatedly says "load" with no indication if you mean CPU load or disk load. Your phrasing indicate you're referring to CPU load, which is unrelated to disk load. For disk load, use "gstat -I500ms" (please ignore the busy% column). systat will not show you this in a coherent manner; you need to have two windows up (preferably one with "top -s 1" the other with gstat). You may be surprised at what's going on behind the scenes with disk load. [1]: http://lists.freebsd.org/pipermail/freebsd-stable/2013-June/073654.html [2]: http://lists.freebsd.org/pipermail/freebsd-stable/2013-June/073662.html -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 22:45:50 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 20811DE for ; Mon, 3 Jun 2013 22:45:50 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by mx1.freebsd.org (Postfix) with ESMTP id D0AC51C8E for ; Mon, 3 Jun 2013 22:45:49 +0000 (UTC) Received: from mfilter27-d.gandi.net (mfilter27-d.gandi.net [217.70.178.155]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id ED88441C064; Tue, 4 Jun 2013 00:45:37 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter27-d.gandi.net Received: from relay5-d.mail.gandi.net ([217.70.183.197]) by mfilter27-d.gandi.net (mfilter27-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id mn8QIyvBdXcR; Tue, 4 Jun 2013 00:45:36 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 0E24741C062; Tue, 4 Jun 2013 00:45:35 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 1813373A1C; Mon, 3 Jun 2013 15:45:34 -0700 (PDT) Date: Mon, 3 Jun 2013 15:45:34 -0700 From: Jeremy Chadwick To: Ross Alexander Subject: Re: 9.1-current disk throughput stalls ? Message-ID: <20130603224534.GA52022@icarus.home.lan> References: <20130603203146.GB49602@icarus.home.lan> <20130603223425.GA51402@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130603223425.GA51402@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 22:45:50 -0000 On Mon, Jun 03, 2013 at 03:34:26PM -0700, Jeremy Chadwick wrote: > 7. ZFS setup is a mirror (RAID-1-like), Should have referenced [2]. > 12. Rolling back to 8.4-STABLE (date/build unknown) apparently fixes > your issue (I would appreciate you running the system for 72 hours > before making this statement, and doing the *exact same things* on it > that cause the problem with 9.1-STABLE) [2] I should have used the word "exacerbate" instead of "cause". > v) I really wish you would not have rolled this system back to > 8.4-STABLE. For anyone to debug this, we need the system in a > consistent state. Changing kernels/etc. User error while using vim (I have an awful tendency to nuke entire lines when switching between input mode vs. navigation mode); last line should read "Changing kernels/etc. in the middle of troubleshooting a problem you ask for assistance with makes things very difficult". (And I say that knowing that rolling back as a form of testing is good, since it can help narrow things down to a specific version or release, i.e. a software problem). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Mon Jun 3 23:12:41 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EB921671 for ; Mon, 3 Jun 2013 23:12:41 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) by mx1.freebsd.org (Postfix) with ESMTP id D2E281D9B for ; Mon, 3 Jun 2013 23:12:41 +0000 (UTC) Received: from zeta.ixsystems.com (drawbridge.ixsystems.com [206.40.55.65]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by anubis.delphij.net (Postfix) with ESMTPSA id 2039AE70B; Mon, 3 Jun 2013 16:12:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=delphij.net; s=anubis; t=1370301161; bh=TkRAkp4UiOnZC9QrcveXMJHz/UY08DGT8gdGfL6vfxY=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=Ep/mGMbASUp3JODi9d1DuDghsh32yDK3KmI0nKthNzqkUwJ5GBmt4ubOyYVJSPXme lUbsPztz6LCvCjgJn6fR7NYCc69iogn1rVz1Z9JAWhO8tQElaQNev/Y96JdknFMrqA 5v+QIMVCbwoTuSjCi1+FanYaQq6mWw0kbfRSSgnc= Message-ID: <51AD22E8.8000600@delphij.net> Date: Mon, 03 Jun 2013 16:12:40 -0700 From: Xin Li Organization: The FreeBSD Project MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: ZFS LZ4 Upgrade References: <20130603185200.5BB956F487@smtp.hushmail.com> In-Reply-To: <20130603185200.5BB956F487@smtp.hushmail.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: d@delphij.net List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 23:12:42 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 06/03/13 11:52, Kenta Suzumoto wrote: > Hi. I'm planning on doing a ZFS root installation on a remote > server very soon. The company only offers a 9.0 and 9.1 > installation and "rescue" (nfs/pxe boot with ramdisk basically) > system. I'd like to use LZ4 with the ZFS root pool, so I'm going to > be upgrading to -STABLE once I have the initial system installed. > Here's what I'll do: > > - install the 9.1 system - svn source, buildworld/kernel, install, > reboot - upon booting the -STABLE system, begin enabling LZ4 > compression on /usr/ports /usr/src etc. > > Will this work, or do I need to find some way to initially create > the zpool with a -STABLE system? Is it just a matter of running > "zfs upgrade" and "zpool upgrade" before enabling LZ4, or am I > missing something? Thanks This should work, you don't have to have the pool created on a -STABLE system and can always do 'zpool upgrade' when necessary. Please note that if you want to upgrade the root pool, make sure that you have a fresh bootblock before rebooting ("zpool upgrade" should give you a reminder). The FreeBSD ZFS boot code supports booting from LZ4 compressed datasets. I would recommend using a dedicated root pool by the way, it have a lot of advantages, like you can use a separate ZIL device for the non-root pool, etc. and have the flexibility of using different RAID-Z layout for different purpose (e.g. mirror for boot pool and RAID-Z for the rest), so your remote hands would be able to just remove one disk and your system is bootable, in the case when the first disk's boot partition is physically damaged. Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJRrSLoAAoJEG80Jeu8UPuzLW8IAMRJJHloskYlrikKGw/Iao8A jlifL2sDpnztgqhGtfltXOIyfKJL7EDw8s8ccJq8Xy1vKy2pndOgCc7GRfuvYS64 07kGrEKQKAv0BcrD6uddtMFstDdrI7cnK3btyAavNJhXR/b2A8f8/jze13mPpdUJ DS2PBJ0rwmHqU7VXxkCq/M8atZGT7pq2ednPcHXX3QaqPmpopUtX89x6D99S39U7 a1UKN2Ic78CAc5R01I80y6II85yBylwIlT5lRPE/SVYbtBFsidhQQ8/Hx/Fcz3qI 7ceK6CiAoQy7ntBvE+uBbQ2630Z3m6kOmInFJD/DnQnf1n6twmBxqwsFTjITbHw= =pCl7 -----END PGP SIGNATURE----- From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 06:05:47 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 642022CA for ; Tue, 4 Jun 2013 06:05:47 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bk0-x22e.google.com (mail-bk0-x22e.google.com [IPv6:2a00:1450:4008:c01::22e]) by mx1.freebsd.org (Postfix) with ESMTP id E83091CEA for ; Tue, 4 Jun 2013 06:05:46 +0000 (UTC) Received: by mail-bk0-f46.google.com with SMTP id na10so2375953bkb.33 for ; Mon, 03 Jun 2013 23:05:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=JgIONbtErhP3gzONTiWpC1MPOs52ilys1sbIruZ0PS4=; b=QpFY6IVIWmxL43L7QqDju9YDM7ZHK7gfgsLqCYd0rdpiUewQsTG71fbnwMmZe2mNex LexReWJ8WfAVVgeC9QmwUT7XzK/XeoU2WNXM9wvJrdPqSIbiG7CG33VtI1VoK5ncxmfS UuhtCshHjgom15rVEW3D0UVuLVVmWeoyDgdLe7ly3KMHF+LCZHeNEnAAkmZ9qw6E6vCH jqZd9DGp0AFaPH+XC8GdUKW69GZsMCdDe++W1HoJG6zrAGtFgyhid73WbEbUw7OXisjz DfEYqBi6qvfXL3m5iMcJEV6LY3Ou12HiCr/TdnjGtEsCU2uywUKRHNih9PdJFT3qCTZI PObQ== X-Received: by 10.204.231.137 with SMTP id jq9mr5535148bkb.150.1370325946021; Mon, 03 Jun 2013 23:05:46 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id b12sm21457867bkz.0.2013.06.03.23.05.43 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 23:05:44 -0700 (PDT) Sender: Alexander Motin Message-ID: <51AD83B6.3090204@FreeBSD.org> Date: Tue, 04 Jun 2013 09:05:42 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130413 Thunderbird/17.0.5 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: 9.1-stable: ATI IXP600 AHCI: CAM timeout References: <201305291421.r4TELY8p042536@grabthar.secnetix.de> <1369840577.1258.45.camel@revolution.hippie.lan> <51ACA2FD.2050609@bsquare.com> <20130603202206.GA49602@icarus.home.lan> In-Reply-To: <20130603202206.GA49602@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org, Mike Pumford X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 06:05:47 -0000 On 03.06.2013 23:22, Jeremy Chadwick wrote: > On Mon, Jun 03, 2013 at 03:06:53PM +0100, Mike Pumford wrote: >> Ian Lepore wrote: >>> On Wed, 2013-05-29 at 16:21 +0200, Oliver Fromme wrote: >>>> Steven Hartland wrote: >>>> > Have you checked your sata cables and psu outputs? >>>> > >>>> > Both of these could be the underlying cause of poor signalling. >>>> >>>> I can't easily check that because it is a cheap rented >>>> server in a remote location. >>>> >>>> But I don't believe it is bad cabling or PSU anyway, or >>>> otherwise the problem would occur intermittently all the >>>> time if the load on the disks is sufficiently high. >>>> But it only occurs at tags=3 and above. At tags=2 it does >>>> not occur at all, no matter how hard I hammer on the disks. >>>> >>>> At the moment I'm inclined to believe that it is either >>>> a bug in the HDD firmware or in the controller. The disks >>>> aren't exactly new, they're 400 GB Samsung ones that are >>>> several years old. I think it's not uncommon to have bugs >>>> in the NCQ implementation in such disks. >>>> >>>> The only thing that puzzles me is the fact that the problem >>>> also disappears completely when I reduce the SATA rev from >>>> II to I, even at tags=32. >>>> >>> >>> It seems to me that you dismiss signaling problems too quickly. >>> Consider the possibilities... A bad cable leads to intermittant errors >>> at higher speeds. When NCQ is disabled or limited the software handles >>> these errors pretty much transparently. When NCQ is not limitted and >>> there are many outstanding requests, suddenly the error handling in the >>> software breaks down somehow and a minor recoverable problem becomes an >>> in-your-face error. >>> >> It could also be a software bug in the way CAM handles the failure >> of NCQ commands. When command queueing is used on a SCSI drive and a >> queued command fails only that command fails. A queued command >> failure on a SATA device fails ALL currently queued commands. I've >> not looked at the code but do the SATA CAM drivers do the right >> thing here? > > Quoting T13/2015-D ATA8-ACS2 WD spec: > > "If an error occurs while the device is processing an NCQ command, then > the device shall return command aborted for all NCQ commands that are in > the queue and shall return command aborted for any new commands, except > a READ LOG EXT command requesting log address 10h, until the device > completes a READ LOG EXT command requesting log address 10h (i.e., > reading the NCQ Command Error log) without error." > > While I can't easily provide an answer to your question, I can tell you > that sys/dev/ahci/ahci.c does execute READ LOG EXT (command 0x2f) for > certain scenarios (the code is in function ahci_issue_recovery()). I am not aware about any flows in present CAM ATA error recovery logic. READ LOG EXT sending indeed implemented on ahci(4) driver level (same as siis(4) and mvs(4)) since it was complicated/impossible to do in shared code because higher levels have no idea about tags allocation done by lower-level drivers. > The one person who can answer this question is mav@, who is now CC'd. > >> Less commands queued makes it less likely that multiple commands >> will be in progress when a failure occurs. A lower link rate also >> makes you more immune to signal failures. > > He isn't seeing SATA-level signal/link failure; the AHCI driver would > complain about that, and those messages aren't there. Unless, of > course, those messages are only visible when verbose booting is enabled > (I hope not). Just a curious history point: I had one old system on NVIDIA MCP55 chipset where Linux worked well before, but FreeBSD had problems with SATA -- all disk transfers were really slow, but without reporting any errors, and after some point system started to hang. That series of chipsets had long history of problems, so for some time I was looking for some way to handle it in software. But after many experiments I've accidentally found out that disabling 6 small but very powerful fans workarounded the problem. I've checked PSU voltages, and they were fine. Switching fans to separate PSU also helped. Finally I've just replaced system's main PSU with different one and problems have gone. My best guess was that capacitors in that PSU due to old age were unable to filter fan's electric noise that started to interfere with SATA and later other signals. Now the same PSU works perfectly fine in the same case with smaller Atom-based motherbard without any issues. I am not telling that ahci(4) driver is perfect, but hardware issues are always possible even if system worked fine before that. -- Alexander Motin From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 08:54:40 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E14D4650 for ; Tue, 4 Jun 2013 08:54:40 +0000 (UTC) (envelope-from pascal.braun@continum.net) Received: from mailsrv1.continum.net (mr1.continum.net [80.72.129.121]) by mx1.freebsd.org (Postfix) with ESMTP id D41B5126B for ; Tue, 4 Jun 2013 08:54:39 +0000 (UTC) Received: from zimbra.continum.net ([80.72.133.238]) by mr1.continum.net with esmtp (Exim 4.67) (envelope-from ) id 1Ujn0k-0006Rp-5A; Tue, 04 Jun 2013 10:54:34 +0200 Received: from localhost (localhost [127.0.0.1]) by zimbra.continum.net (Postfix) with ESMTP id 7334A19E02B; Tue, 4 Jun 2013 10:54:30 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.continum.net Received: from zimbra.continum.net ([127.0.0.1]) by localhost (zimbra.continum.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Ogbs4ybIgWW; Tue, 4 Jun 2013 10:54:30 +0200 (CEST) Received: from zimbra.continum.net (zimbra.continum.net [80.72.133.238]) by zimbra.continum.net (Postfix) with ESMTP id 1804A19E02A; Tue, 4 Jun 2013 10:54:30 +0200 (CEST) Date: Tue, 4 Jun 2013 10:54:30 +0200 (CEST) From: "Pascal Braun, Continum" To: Ronald Klop Message-ID: <1261869099.176558.1370336069971.JavaMail.root@continum.net> In-Reply-To: Subject: Re: ZFS crashing while zfs recv in progress MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_176556_478359006.1370336069968" X-Originating-IP: [80.72.130.250] X-Mailer: Zimbra 7.2.0_GA_2669 (ZimbraWebClient - GC23 (Linux)/7.2.0_GA_2669) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 08:54:40 -0000 ------=_Part_176556_478359006.1370336069968 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I've put swap on a seperate disk this time and re-run the zfs send / recv, = but i'm still getting the problem. Whats interesting is, i'm still getting = the same output about swap space on the console: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 36, size: 24576 The System is running from zroot and i'm trying to transfer to tank. zpool = Status output: --- pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/disk3 ONLINE 0 0 0 gpt/disk9 ONLINE 0 0 0 gpt/disk15 ONLINE 0 0 0 gpt/disk19 ONLINE 0 0 0 gpt/disk23 ONLINE 0 0 0 gpt/disk27 ONLINE 0 0 0 gpt/disk31 ONLINE 0 0 0 gpt/disk36 ONLINE 0 0 0 gpt/disk33 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk4 ONLINE 0 0 0 gpt/disk7 ONLINE 0 0 0 gpt/disk10 ONLINE 0 0 0 gpt/disk13 ONLINE 0 0 0 gpt/disk16 ONLINE 0 0 0 gpt/disk24 ONLINE 0 0 0 gpt/disk32 ONLINE 0 0 0 gpt/disk37 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 gpt/disk5 ONLINE 0 0 0 gpt/disk8 ONLINE 0 0 0 gpt/disk11 ONLINE 0 0 0 gpt/disk17 ONLINE 0 0 0 gpt/disk21 ONLINE 0 0 0 gpt/disk25 ONLINE 0 0 0 gpt/disk29 ONLINE 0 0 0 gpt/disk38 ONLINE 0 0 0 raidz2-3 ONLINE 0 0 0 gpt/disk12 ONLINE 0 0 0 gpt/disk14 ONLINE 0 0 0 gpt/disk18 ONLINE 0 0 0 gpt/disk22 ONLINE 0 0 0 gpt/disk26 ONLINE 0 0 0 gpt/disk30 ONLINE 0 0 0 gpt/disk34 ONLINE 0 0 0 gpt/disk35 ONLINE 0 0 0 gpt/disk39 ONLINE 0 0 0 spares gpt/disk20 AVAIL =20 errors: No known data errors pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk6 ONLINE 0 0 0 errors: No known data errors --- Attatched you'll also find the dmesg.boot. thanks, Pascal ----- Urspr=C3=BCngliche Mail ----- > On Wed, 29 May 2013 11:04:00 +0200, Pascal Braun, Continum >=20 > Please send more information about the new server. Sometimes there > are > bugs found in drivers with large disks, etc. Or firmware of hardware. > The contents of /var/run/dmesg.boot is interesting to a lot of > people. > As is the output of zpool status. >=20 > As you are having trouble with swap on zfs. Is it possible to put > that on > a separate disk for the test? >=20 > Ronald. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" >=20 ------=_Part_176556_478359006.1370336069968 Content-Type: application/octet-stream; name=dmesg.boot Content-Disposition: attachment; filename=dmesg.boot Content-Transfer-Encoding: base64 Q29weXJpZ2h0IChjKSAxOTkyLTIwMTIgVGhlIEZyZWVCU0QgUHJvamVjdC4KQ29weXJpZ2h0IChj KSAxOTc5LCAxOTgwLCAxOTgzLCAxOTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAx OTk0CglUaGUgUmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBDYWxpZm9ybmlhLiBBbGwgcmln aHRzIHJlc2VydmVkLgpGcmVlQlNEIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhlIEZy ZWVCU0QgRm91bmRhdGlvbi4KRnJlZUJTRCA5LjEtUkVMRUFTRSAjMCByMjQzODI1OiBUdWUgRGVj ICA0IDA5OjIzOjEwIFVUQyAyMDEyCiAgICByb290QGZhcnJlbGwuY3NlLmJ1ZmZhbG8uZWR1Oi91 c3Ivb2JqL3Vzci9zcmMvc3lzL0dFTkVSSUMgYW1kNjQKQ1BVOiBRdWFkLUNvcmUgQU1EIE9wdGVy b24odG0pIFByb2Nlc3NvciAyMzU2ICgyMzAwLjE0LU1IeiBLOC1jbGFzcyBDUFUpCiAgT3JpZ2lu ID0gIkF1dGhlbnRpY0FNRCIgIElkID0gMHgxMDBmMjMgIEZhbWlseSA9IDEwICBNb2RlbCA9IDIg IFN0ZXBwaW5nID0gMwogIEZlYXR1cmVzPTB4MTc4YmZiZmY8RlBVLFZNRSxERSxQU0UsVFNDLE1T UixQQUUsTUNFLENYOCxBUElDLFNFUCxNVFJSLFBHRSxNQ0EsQ01PVixQQVQsUFNFMzYsQ0xGTFVT SCxNTVgsRlhTUixTU0UsU1NFMixIVFQ+CiAgRmVhdHVyZXMyPTB4ODAyMDA5PFNTRTMsTU9OLENY MTYsUE9QQ05UPgogIEFNRCBGZWF0dXJlcz0weGVlNTAwODAwPFNZU0NBTEwsTlgsTU1YKyxGRlhT UixQYWdlMUdCLFJEVFNDUCxMTSwzRE5vdyErLDNETm93IT4KICBBTUQgRmVhdHVyZXMyPTB4N2Zm PExBSEYsQ01QLFNWTSxFeHRBUElDLENSOCxBQk0sU1NFNEEsTUFTLFByZWZldGNoLE9TVlcsSUJT PgogIFRTQzogUC1zdGF0ZSBpbnZhcmlhbnQKcmVhbCBtZW1vcnkgID0gMTcxNzk4NjkxODQgKDE2 Mzg0IE1CKQphdmFpbCBtZW1vcnkgPSAxNjUyMjYwMDQ0OCAoMTU3NTcgTUIpCkV2ZW50IHRpbWVy ICJMQVBJQyIgcXVhbGl0eSA0MDAKQUNQSSBBUElDIFRhYmxlOiA8U1VOICAgIFg0NTQwICAgPgpG cmVlQlNEL1NNUDogTXVsdGlwcm9jZXNzb3IgU3lzdGVtIERldGVjdGVkOiA4IENQVXMKRnJlZUJT RC9TTVA6IDIgcGFja2FnZShzKSB4IDQgY29yZShzKQogY3B1MCAoQlNQKTogQVBJQyBJRDogIDAK IGNwdTEgKEFQKTogQVBJQyBJRDogIDEKIGNwdTIgKEFQKTogQVBJQyBJRDogIDIKIGNwdTMgKEFQ KTogQVBJQyBJRDogIDMKIGNwdTQgKEFQKTogQVBJQyBJRDogIDQKIGNwdTUgKEFQKTogQVBJQyBJ RDogIDUKIGNwdTYgKEFQKTogQVBJQyBJRDogIDYKIGNwdTcgKEFQKTogQVBJQyBJRDogIDcKaW9h cGljMCA8VmVyc2lvbiAxLjE+IGlycXMgMC0yMyBvbiBtb3RoZXJib2FyZAppb2FwaWMxIDxWZXJz aW9uIDEuMT4gaXJxcyAyNC00NyBvbiBtb3RoZXJib2FyZAppb2FwaWMyIDxWZXJzaW9uIDEuMT4g aXJxcyA0OC03MSBvbiBtb3RoZXJib2FyZAprYmQxIGF0IGtiZG11eDAKbW9kdWxlX3JlZ2lzdGVy X2luaXQ6IE1PRF9MT0FEICh2ZXNhLCAweGZmZmZmZmZmODBiZjU1MTAsIDApIGVycm9yIDE5CmFj cGkwOiA8U1VOIFg0NTQwPiBvbiBtb3RoZXJib2FyZAphY3BpMDogUG93ZXIgQnV0dG9uIChmaXhl ZCkKdW5rbm93bjogSS9PIHJhbmdlIG5vdCBzdXBwb3J0ZWQKYWNwaTA6IHJlc2VydmF0aW9uIG9m IGZlYzAwMDAwLCAxMDAwICgzKSBmYWlsZWQKYWNwaTA6IHJlc2VydmF0aW9uIG9mIGZlZTAwMDAw LCAxMDAwICgzKSBmYWlsZWQKYWNwaTA6IHJlc2VydmF0aW9uIG9mIDAsIGEwMDAwICgzKSBmYWls ZWQKYWNwaTA6IHJlc2VydmF0aW9uIG9mIDEwMDAwMCwgZGRmMDAwMDAgKDMpIGZhaWxlZApjcHUw OiA8QUNQSSBDUFU+IG9uIGFjcGkwCmNwdTE6IDxBQ1BJIENQVT4gb24gYWNwaTAKY3B1MjogPEFD UEkgQ1BVPiBvbiBhY3BpMApjcHUzOiA8QUNQSSBDUFU+IG9uIGFjcGkwCmNwdTQ6IDxBQ1BJIENQ VT4gb24gYWNwaTAKY3B1NTogPEFDUEkgQ1BVPiBvbiBhY3BpMApjcHU2OiA8QUNQSSBDUFU+IG9u IGFjcGkwCmNwdTc6IDxBQ1BJIENQVT4gb24gYWNwaTAKYXR0aW1lcjA6IDxBVCB0aW1lcj4gcG9y dCAweDQwLTB4NDMgaXJxIDAgb24gYWNwaTAKVGltZWNvdW50ZXIgImk4MjU0IiBmcmVxdWVuY3kg MTE5MzE4MiBIeiBxdWFsaXR5IDAKRXZlbnQgdGltZXIgImk4MjU0IiBmcmVxdWVuY3kgMTE5MzE4 MiBIeiBxdWFsaXR5IDEwMAphdHJ0YzA6IDxBVCByZWFsdGltZSBjbG9jaz4gcG9ydCAweDcwLTB4 NzEgaXJxIDggb24gYWNwaTAKRXZlbnQgdGltZXIgIlJUQyIgZnJlcXVlbmN5IDMyNzY4IEh6IHF1 YWxpdHkgMApocGV0MDogPEhpZ2ggUHJlY2lzaW9uIEV2ZW50IFRpbWVyPiBpb21lbSAweGZlZDAw MDAwLTB4ZmVkMDBmZmYgb24gYWNwaTAKVGltZWNvdW50ZXIgIkhQRVQiIGZyZXF1ZW5jeSAyNTAw MDAwMCBIeiBxdWFsaXR5IDk1MApUaW1lY291bnRlciAiQUNQSS1mYXN0IiBmcmVxdWVuY3kgMzU3 OTU0NSBIeiBxdWFsaXR5IDkwMAphY3BpX3RpbWVyMDogPDI0LWJpdCB0aW1lciBhdCAzLjU3OTU0 NU1Iej4gcG9ydCAweGUwMDgtMHhlMDBiIG9uIGFjcGkwCnBjaWIwOiA8QUNQSSBIb3N0LVBDSSBi cmlkZ2U+IHBvcnQgMHhjZjgtMHhjZmYgb24gYWNwaTAKcGNpMDogPEFDUEkgUENJIGJ1cz4gb24g cGNpYjAKcGNpMDogPG1lbW9yeSwgUkFNPiBhdCBkZXZpY2UgMC4wIChubyBkcml2ZXIgYXR0YWNo ZWQpCmlzYWIwOiA8UENJLUlTQSBicmlkZ2U+IHBvcnQgMHhlZjAwLTB4ZWY3ZiBhdCBkZXZpY2Ug MS4wIG9uIHBjaTAKaXNhMDogPElTQSBidXM+IG9uIGlzYWIwCnBjaTA6IDxzZXJpYWwgYnVzLCBT TUJ1cz4gYXQgZGV2aWNlIDEuMSAobm8gZHJpdmVyIGF0dGFjaGVkKQpvaGNpMDogPG5WaWRpYSBu Rm9yY2UgTUNQNTUgVVNCIENvbnRyb2xsZXI+IG1lbSAweGRlN2ZmMDAwLTB4ZGU3ZmZmZmYgaXJx IDIxIGF0IGRldmljZSAyLjAgb24gcGNpMAp1c2J1czAgb24gb2hjaTAKZWhjaTA6IDxOVklESUEg bkZvcmNlIE1DUDU1IFVTQiAyLjAgY29udHJvbGxlcj4gbWVtIDB4ZGU3ZmVjMDAtMHhkZTdmZWNm ZiBpcnEgMjIgYXQgZGV2aWNlIDIuMSBvbiBwY2kwCnVzYnVzMTogRUhDSSB2ZXJzaW9uIDEuMAp1 c2J1czEgb24gZWhjaTAKcGNpYjE6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgNi4w IG9uIHBjaTAKcGNpMTogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjEKdmdhcGNpMDogPFZHQS1jb21w YXRpYmxlIGRpc3BsYXk+IHBvcnQgMHhhYzAwLTB4YWM3ZiBtZW0gMHhkZTgwMDAwMC0weGRlZmZm ZmZmLDB4ZGYzZTAwMDAtMHhkZjNmZmZmZiBpcnEgMTYgYXQgZGV2aWNlIDUuMCBvbiBwY2kxCm5m ZTA6IDxOVklESUEgbkZvcmNlIE1DUDU1IE5ldHdvcmtpbmcgQWRhcHRlcj4gcG9ydCAweDk0ODAt MHg5NDg3IG1lbSAweGRlN2ZkMDAwLTB4ZGU3ZmRmZmYsMHhkZTdmZTgwMC0weGRlN2ZlOGZmLDB4 ZGU3ZmU0MDAtMHhkZTdmZTQwZiBpcnEgMjMgYXQgZGV2aWNlIDguMCBvbiBwY2kwCm1paWJ1czA6 IDxNSUkgYnVzPiBvbiBuZmUwCmUxMDAwcGh5MDogPE1hcnZlbGwgODhFMTE0OSBHaWdhYml0IFBI WT4gUEhZIDIgb24gbWlpYnVzMAplMTAwMHBoeTA6ICBub25lLCAxMGJhc2VULCAxMGJhc2VULUZE WCwgMTAwYmFzZVRYLCAxMDBiYXNlVFgtRkRYLCAxMDAwYmFzZVQsIDEwMDBiYXNlVC1tYXN0ZXIs IDEwMDBiYXNlVC1GRFgsIDEwMDBiYXNlVC1GRFgtbWFzdGVyLCBhdXRvLCBhdXRvLWZsb3cKbmZl MDogRXRoZXJuZXQgYWRkcmVzczogMDA6MTQ6NGY6ZjI6YzY6ZTgKbmZlMTogPE5WSURJQSBuRm9y Y2UgTUNQNTUgTmV0d29ya2luZyBBZGFwdGVyPiBwb3J0IDB4OTQwMC0weDk0MDcgbWVtIDB4ZGU3 ZmMwMDAtMHhkZTdmY2ZmZiwweGRlN2ZlMDAwLTB4ZGU3ZmUwZmYsMHhkZTdmN2MwMC0weGRlN2Y3 YzBmIGlycSAyMCBhdCBkZXZpY2UgOS4wIG9uIHBjaTAKbWlpYnVzMTogPE1JSSBidXM+IG9uIG5m ZTEKZTEwMDBwaHkxOiA8TWFydmVsbCA4OEUxMTQ5IEdpZ2FiaXQgUEhZPiBQSFkgMyBvbiBtaWli dXMxCmUxMDAwcGh5MTogIG5vbmUsIDEwYmFzZVQsIDEwYmFzZVQtRkRYLCAxMDBiYXNlVFgsIDEw MGJhc2VUWC1GRFgsIDEwMDBiYXNlVCwgMTAwMGJhc2VULW1hc3RlciwgMTAwMGJhc2VULUZEWCwg MTAwMGJhc2VULUZEWC1tYXN0ZXIsIGF1dG8sIGF1dG8tZmxvdwpuZmUxOiBFdGhlcm5ldCBhZGRy ZXNzOiAwMDoxNDo0ZjpmMjpjNjplOQpwY2liMjogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRl dmljZSAxMC4wIG9uIHBjaTAKcGNpMjogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjIKbXB0MDogPExT SUxvZ2ljIFNBUy9TQVRBIEFkYXB0ZXI+IHBvcnQgMHhiODAwLTB4YjhmZiBtZW0gMHhkZjdmYzAw MC0weGRmN2ZmZmZmLDB4ZGY3ZTAwMDAtMHhkZjdlZmZmZiBpcnEgMTcgYXQgZGV2aWNlIDAuMCBv biBwY2kyCm1wdDA6IE1QSSBWZXJzaW9uPTEuNS4yMC4wCnBjaWIzOiA8QUNQSSBQQ0ktUENJIGJy aWRnZT4gYXQgZGV2aWNlIDExLjAgb24gcGNpMApwY2kzOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2li MwptcHQxOiA8TFNJTG9naWMgU0FTL1NBVEEgQWRhcHRlcj4gcG9ydCAweGM4MDAtMHhjOGZmIG1l bSAweGRmYmZjMDAwLTB4ZGZiZmZmZmYsMHhkZmJlMDAwMC0weGRmYmVmZmZmIGlycSAxOCBhdCBk ZXZpY2UgMC4wIG9uIHBjaTMKbXB0MTogTVBJIFZlcnNpb249MS41LjIwLjAKcGNpYjQ6IDxBQ1BJ IFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMTUuMCBvbiBwY2kwCnBjaTQ6IDxBQ1BJIFBDSSBi dXM+IG9uIHBjaWI0Cm1wdDI6IDxMU0lMb2dpYyBTQVMvU0FUQSBBZGFwdGVyPiBwb3J0IDB4ZDgw MC0weGQ4ZmYgbWVtIDB4ZGZmZmMwMDAtMHhkZmZmZmZmZiwweGRmZmUwMDAwLTB4ZGZmZWZmZmYg aXJxIDE4IGF0IGRldmljZSAwLjAgb24gcGNpNAptcHQyOiBNUEkgVmVyc2lvbj0xLjUuMjAuMApw Y2liNTogPEFDUEkgSG9zdC1QQ0kgYnJpZGdlPiBvbiBhY3BpMApwY2k2NDogPEFDUEkgUENJIGJ1 cz4gb24gcGNpYjUKcGNpNjQ6IDxtZW1vcnksIFJBTT4gYXQgZGV2aWNlIDAuMCAobm8gZHJpdmVy IGF0dGFjaGVkKQpwY2k2NDogPG1lbW9yeSwgUkFNPiBhdCBkZXZpY2UgMS4wIChubyBkcml2ZXIg YXR0YWNoZWQpCnBjaTY0OiA8c2VyaWFsIGJ1cywgU01CdXM+IGF0IGRldmljZSAxLjEgKG5vIGRy aXZlciBhdHRhY2hlZCkKbmZlMjogPE5WSURJQSBuRm9yY2UgTUNQNTUgTmV0d29ya2luZyBBZGFw dGVyPiBwb3J0IDB4NTQwMC0weDU0MDcgbWVtIDB4ZGQzZmUwMDAtMHhkZDNmZWZmZiwweGRkM2Zk YzAwLTB4ZGQzZmRjZmYsMHhkZDNmZDgwMC0weGRkM2ZkODBmIGlycSA0NSBhdCBkZXZpY2UgOC4w IG9uIHBjaTY0Cm1paWJ1czI6IDxNSUkgYnVzPiBvbiBuZmUyCmUxMDAwcGh5MjogPE1hcnZlbGwg ODhFMTE0OSBHaWdhYml0IFBIWT4gUEhZIDIgb24gbWlpYnVzMgplMTAwMHBoeTI6ICBub25lLCAx MGJhc2VULCAxMGJhc2VULUZEWCwgMTAwYmFzZVRYLCAxMDBiYXNlVFgtRkRYLCAxMDAwYmFzZVQs IDEwMDBiYXNlVC1tYXN0ZXIsIDEwMDBiYXNlVC1GRFgsIDEwMDBiYXNlVC1GRFgtbWFzdGVyLCBh dXRvLCBhdXRvLWZsb3cKbmZlMjogRXRoZXJuZXQgYWRkcmVzczogMDA6MTQ6NGY6ZjI6YzY6ZWEK bmZlMzogPE5WSURJQSBuRm9yY2UgTUNQNTUgTmV0d29ya2luZyBBZGFwdGVyPiBwb3J0IDB4NTA4 MC0weDUwODcgbWVtIDB4ZGQzZmMwMDAtMHhkZDNmY2ZmZiwweGRkM2ZkNDAwLTB4ZGQzZmQ0ZmYs MHhkZDNmZDAwMC0weGRkM2ZkMDBmIGlycSA0NiBhdCBkZXZpY2UgOS4wIG9uIHBjaTY0Cm1paWJ1 czM6IDxNSUkgYnVzPiBvbiBuZmUzCmUxMDAwcGh5MzogPE1hcnZlbGwgODhFMTE0OSBHaWdhYml0 IFBIWT4gUEhZIDMgb24gbWlpYnVzMwplMTAwMHBoeTM6ICBub25lLCAxMGJhc2VULCAxMGJhc2VU LUZEWCwgMTAwYmFzZVRYLCAxMDBiYXNlVFgtRkRYLCAxMDAwYmFzZVQsIDEwMDBiYXNlVC1tYXN0 ZXIsIDEwMDBiYXNlVC1GRFgsIDEwMDBiYXNlVC1GRFgtbWFzdGVyLCBhdXRvLCBhdXRvLWZsb3cK bmZlMzogRXRoZXJuZXQgYWRkcmVzczogMDA6MTQ6NGY6ZjI6YzY6ZWIKcGNpYjY6IDxBQ1BJIFBD SS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMTAuMCBvbiBwY2k2NApwY2k2NTogPEFDUEkgUENJIGJ1 cz4gb24gcGNpYjYKbXB0MzogPExTSUxvZ2ljIFNBUy9TQVRBIEFkYXB0ZXI+IHBvcnQgMHg2ODAw LTB4NjhmZiBtZW0gMHhkZDdmYzAwMC0weGRkN2ZmZmZmLDB4ZGQ3ZTAwMDAtMHhkZDdlZmZmZiBp cnEgNDAgYXQgZGV2aWNlIDAuMCBvbiBwY2k2NQptcHQzOiBNUEkgVmVyc2lvbj0xLjUuMjAuMApw Y2liNzogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSAxMS4wIG9uIHBjaTY0CnBjaTY2 OiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liNwptcHQ0OiA8TFNJTG9naWMgU0FTL1NBVEEgQWRhcHRl cj4gcG9ydCAweDc4MDAtMHg3OGZmIG1lbSAweGRkYmZjMDAwLTB4ZGRiZmZmZmYsMHhkZGJlMDAw MC0weGRkYmVmZmZmIGlycSA0MSBhdCBkZXZpY2UgMC4wIG9uIHBjaTY2Cm1wdDQ6IE1QSSBWZXJz aW9uPTEuNS4yMC4wCnBjaWI4OiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNlIDE1LjAg b24gcGNpNjQKcGNpNjc6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWI4Cm1wdDU6IDxMU0lMb2dpYyBT QVMvU0FUQSBBZGFwdGVyPiBwb3J0IDB4ODgwMC0weDg4ZmYgbWVtIDB4ZGRmZmMwMDAtMHhkZGZm ZmZmZiwweGRkZmUwMDAwLTB4ZGRmZWZmZmYgaXJxIDQxIGF0IGRldmljZSAwLjAgb24gcGNpNjcK bXB0NTogTVBJIFZlcnNpb249MS41LjIwLjAKcGNpYjk6IDxBQ1BJIEhvc3QtUENJIGJyaWRnZT4g b24gYWNwaTAKcGNpMTI4OiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liOQpwY2kxMjg6IDxtZW1vcnks IFJBTT4gYXQgZGV2aWNlIDAuMCAobm8gZHJpdmVyIGF0dGFjaGVkKQpwY2kxMjg6IDxtZW1vcnks IFJBTT4gYXQgZGV2aWNlIDEuMCAobm8gZHJpdmVyIGF0dGFjaGVkKQpwY2kxMjg6IDxzZXJpYWwg YnVzLCBTTUJ1cz4gYXQgZGV2aWNlIDEuMSAobm8gZHJpdmVyIGF0dGFjaGVkKQpwY2liMTA6IDxB Q1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMTAuMCBvbiBwY2kxMjgKcGNpMTI5OiA8QUNQ SSBQQ0kgYnVzPiBvbiBwY2liMTAKcGNpYjExOiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2 aWNlIDExLjAgb24gcGNpMTI4CnBjaTEzMDogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjExCnBjaWIx MjogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSAxNS4wIG9uIHBjaTEyOApwY2kxMzE6 IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWIxMgpiY2UwOiA8SFAgTkMzODJUIFBDSWUgRFAgTXVsdGlm dW5jdGlvbiBHaWdhYml0IFNlcnZlciBBZGFwdGVyIChDMCk+IG1lbSAweGRhMDAwMDAwLTB4ZGJm ZmZmZmYgaXJxIDY0IGF0IGRldmljZSAwLjAgb24gcGNpMTMxCm1paWJ1czQ6IDxNSUkgYnVzPiBv biBiY2UwCmJyZ3BoeTA6IDxCQ001NzA5IDEwLzEwMC8xMDAwYmFzZVQgUEhZPiBQSFkgMSBvbiBt aWlidXM0CmJyZ3BoeTA6ICAxMGJhc2VULCAxMGJhc2VULUZEWCwgMTAwYmFzZVRYLCAxMDBiYXNl VFgtRkRYLCAxMDAwYmFzZVQsIDEwMDBiYXNlVC1tYXN0ZXIsIDEwMDBiYXNlVC1GRFgsIDEwMDBi YXNlVC1GRFgtbWFzdGVyLCBhdXRvLCBhdXRvLWZsb3cKYmNlMDogRXRoZXJuZXQgYWRkcmVzczog NDQ6MWU6YTE6NTY6ODg6NzQKYmNlMDogQVNJQyAoMHg1NzA5MjAwMyk7IFJldiAoQzApOyBCdXMg KFBDSWUgeDQsIDIuNUdicHMpOyBCL0MgKDUuMi4yKTsgQnVmcyAoUlg6MjtUWDoyO1BHOjgpOyBG bGFncyAoU1BMVHxNU0kpCkNvYWwgKFJYOjYsNiwxOCwxODsgVFg6MjAsMjAsODAsODApCmJjZTE6 IDxIUCBOQzM4MlQgUENJZSBEUCBNdWx0aWZ1bmN0aW9uIEdpZ2FiaXQgU2VydmVyIEFkYXB0ZXIg KEMwKT4gbWVtIDB4ZDgwMDAwMDAtMHhkOWZmZmZmZiBpcnEgNjUgYXQgZGV2aWNlIDAuMSBvbiBw Y2kxMzEKbWlpYnVzNTogPE1JSSBidXM+IG9uIGJjZTEKYnJncGh5MTogPEJDTTU3MDkgMTAvMTAw LzEwMDBiYXNlVCBQSFk+IFBIWSAxIG9uIG1paWJ1czUKYnJncGh5MTogIDEwYmFzZVQsIDEwYmFz ZVQtRkRYLCAxMDBiYXNlVFgsIDEwMGJhc2VUWC1GRFgsIDEwMDBiYXNlVCwgMTAwMGJhc2VULW1h c3RlciwgMTAwMGJhc2VULUZEWCwgMTAwMGJhc2VULUZEWC1tYXN0ZXIsIGF1dG8sIGF1dG8tZmxv dwpiY2UxOiBFdGhlcm5ldCBhZGRyZXNzOiA0NDoxZTphMTo1Njo4ODo3NgpiY2UxOiBBU0lDICgw eDU3MDkyMDAzKTsgUmV2IChDMCk7IEJ1cyAoUENJZSB4NCwgMi41R2Jwcyk7IEIvQyAoNS4yLjIp OyBCdWZzIChSWDoyO1RYOjI7UEc6OCk7IEZsYWdzIChTUExUfE1TSSkKQ29hbCAoUlg6Niw2LDE4 LDE4OyBUWDoyMCwyMCw4MCw4MCkKYWNwaV9idXR0b24wOiA8UG93ZXIgQnV0dG9uPiBvbiBhY3Bp MAp1YXJ0MDogPDE2NTUwIG9yIGNvbXBhdGlibGU+IHBvcnQgMHgzZjgtMHgzZmYgaXJxIDQgZmxh Z3MgMHgxMCBvbiBhY3BpMAp1YXJ0MDogY29uc29sZSAoOTYwMCxuLDgsMSkKb3JtMDogPElTQSBP cHRpb24gUk9Ncz4gYXQgaW9tZW0gMHhjMDAwMC0weGM3ZmZmLDB4Y2UwMDAtMHhjZjdmZiBvbiBp c2EwCnNjMDogPFN5c3RlbSBjb25zb2xlPiBhdCBmbGFncyAweDEwMCBvbiBpc2EwCnNjMDogQ0dB IDwxNiB2aXJ0dWFsIGNvbnNvbGVzLCBmbGFncz0weDEwMD4KdmdhMDogPEdlbmVyaWMgSVNBIFZH QT4gYXQgcG9ydCAweDNkMC0weDNkYiBpb21lbSAweGI4MDAwLTB4YmZmZmYgb24gaXNhMApwcGMw OiBjYW5ub3QgcmVzZXJ2ZSBJL08gcG9ydCByYW5nZQpjdGw6IENBTSBUYXJnZXQgTGF5ZXIgbG9h ZGVkCmh3cHN0YXRlMDogPENvb2xgbidRdWlldCAyLjA+IG9uIGNwdTAKWkZTIGZpbGVzeXN0ZW0g dmVyc2lvbiA1ClpGUyBzdG9yYWdlIHBvb2wgdmVyc2lvbiAyOApUaW1lY291bnRlcnMgdGljayBl dmVyeSAxLjAwMCBtc2VjCnVzYnVzMDogMTJNYnBzIEZ1bGwgU3BlZWQgVVNCIHYxLjAKdXNidXMx OiA0ODBNYnBzIEhpZ2ggU3BlZWQgVVNCIHYyLjAKdWdlbjAuMTogPG5WaWRpYT4gYXQgdXNidXMw CnVodWIwOiA8blZpZGlhIE9IQ0kgcm9vdCBIVUIsIGNsYXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwg YWRkciAxPiBvbiB1c2J1czAKdWdlbjEuMTogPG5WaWRpYT4gYXQgdXNidXMxCnVodWIxOiA8blZp ZGlhIEVIQ0kgcm9vdCBIVUIsIGNsYXNzIDkvMCwgcmV2IDIuMDAvMS4wMCwgYWRkciAxPiBvbiB1 c2J1czEKdWh1YjA6IDEwIHBvcnRzIHdpdGggMTAgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQKZGEw IGF0IG1wdDAgYnVzIDAgc2NidXMwIHRhcmdldCAwIGx1biAwCmRhMDogPEFUQSBIaXRhY2hpIEhE UzcyMTAxIEEzTUE+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGEwOiAzMDAu MDAwTUIvcyB0cmFuc2ZlcnMKZGEwOiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGEwOiA5NTM4 NjlNQiAoMTk1MzUyNTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDEyMTYwMUMpCmRh NiBhdCBtcHQxIGJ1cyAwIHNjYnVzMSB0YXJnZXQgMCBsdW4gMApkYTY6IDxBVEEgSGl0YWNoaSBI RFM3MjEwMSBBM01BPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhNjogMzAw LjAwME1CL3MgdHJhbnNmZXJzCmRhNjogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhNjogOTUz ODY5TUIgKDE5NTM1MjUxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAxMjE2MDFDKQpk YTcgYXQgbXB0MSBidXMgMCBzY2J1czEgdGFyZ2V0IDEgbHVuIDAKZGE3OiA8QVRBIEhpdGFjaGkg SERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApkYTc6IDMw MC4wMDBNQi9zIHRyYW5zZmVycwpkYTc6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTc6IDE5 MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMp CmRhOCBhdCBtcHQxIGJ1cyAwIHNjYnVzMSB0YXJnZXQgMiBsdW4gMApkYTg6IDxBVEEgSGl0YWNo aSBIRFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhODog MzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhODogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhODog MTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAx QykKZGExIGF0IG1wdDAgYnVzIDAgc2NidXMwIHRhcmdldCAxIGx1biAwCmRhMTogPEFUQSBIaXRh Y2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGEx OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGExOiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGEx OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMy MDFDKQpkYTIgYXQgbXB0MCBidXMgMCBzY2J1czAgdGFyZ2V0IDIgbHVuIDAKZGEyOiA8QVRBIEhp dGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApk YTI6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTI6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApk YTI6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0 MzIwMUMpCmRhMyBhdCBtcHQwIGJ1cyAwIHNjYnVzMCB0YXJnZXQgNCBsdW4gMApkYTM6IDxBVEEg SGl0YWNoaSBIRFM1QzMwMiBBNTgwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2Ug CmRhMzogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMzogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVk CmRhMzogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1Qg MjQzMjAxQykKZGExMyBhdCBtcHQyIGJ1cyAwIHNjYnVzMiB0YXJnZXQgMSBsdW4gMApkYTEzOiA8 QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2 aWNlIApkYTEzOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGExMzogQ29tbWFuZCBRdWV1ZWluZyBl bmFibGVkCmRhMTM6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVI IDYzUy9UIDI0MzIwMUMpCmRhMTQgYXQgbXB0MiBidXMgMCBzY2J1czIgdGFyZ2V0IDMgbHVuIDAK ZGExNDogPEFUQSBIaXRhY2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NT SS01IGRldmljZSAKZGExNDogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMTQ6IENvbW1hbmQgUXVl dWVpbmcgZW5hYmxlZApkYTE0OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9y czogMjU1SCA2M1MvVCAyNDMyMDFDKQpkYTE1IGF0IG1wdDIgYnVzIDAgc2NidXMyIHRhcmdldCA0 IGx1biAwCmRhMTU6IDxBVEEgSGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNj ZXNzIFNDU0ktNSBkZXZpY2UgCmRhMTU6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTE1OiBDb21t YW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGExNTogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRl IHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykKZGExNiBhdCBtcHQyIGJ1cyAwIHNjYnVzMiB0 YXJnZXQgNSBsdW4gMApkYTE2OiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGly ZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApkYTE2OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGEx NjogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhMTY6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1 MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMpCmRhOSBhdCBtcHQxIGJ1cyAwIHNj YnVzMSB0YXJnZXQgNCBsdW4gMApkYTk6IDxBVEEgSGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhl ZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhOTogMzAwLjAwME1CL3MgdHJhbnNmZXJz CmRhOTogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhOTogMTkwNzcyOU1CICgzOTA3MDI5MTY4 IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykKZGExMCBhdCBtcHQxIGJ1cyAw IHNjYnVzMSB0YXJnZXQgNSBsdW4gMApkYTEwOiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4g Rml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApkYTEwOiAzMDAuMDAwTUIvcyB0cmFu c2ZlcnMKZGExMDogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhMTA6IDE5MDc3MjlNQiAoMzkw NzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMpCmRhMTEgYXQgbXB0 MSBidXMgMCBzY2J1czEgdGFyZ2V0IDYgbHVuIDAKZGExMTogPEFUQSBIaXRhY2hpIEhEUzVDMzAy IEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGExMTogMzAwLjAwME1C L3MgdHJhbnNmZXJzCmRhMTE6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTExOiAxOTA3NzI5 TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQpkYTEy IGF0IG1wdDEgYnVzIDAgc2NidXMxIHRhcmdldCA3IGx1biAwCmRhMTI6IDxBVEEgSGl0YWNoaSBI RFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhMTI6IDMw MC4wMDBNQi9zIHRyYW5zZmVycwpkYTEyOiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGExMjog MTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAx QykKZGE0IGF0IG1wdDAgYnVzIDAgc2NidXMwIHRhcmdldCA1IGx1biAwCmRhNDogPEFUQSBIaXRh Y2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGE0 OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGE0OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGE0 OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMy MDFDKQpkYTUgYXQgbXB0MCBidXMgMCBzY2J1czAgdGFyZ2V0IDYgbHVuIDAKZGE1OiA8QVRBIEhp dGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApk YTU6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTU6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApk YTU6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0 MzIwMUMpCnVodWIxOiAxMCBwb3J0cyB3aXRoIDEwIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkCmRh MTcgYXQgbXB0MiBidXMgMCBzY2J1czIgdGFyZ2V0IDYgbHVuIDAKZGExNzogPEFUQSBIaXRhY2hp IEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGExNzog MzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMTc6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTE3 OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMy MDFDKQpkYTE5IGF0IG1wdDMgYnVzIDAgc2NidXMzIHRhcmdldCAwIGx1biAwCmRhMTk6IDxBVEEg SGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2Ug CmRhMTk6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTE5OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJs ZWQKZGExOTogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNT L1QgMjQzMjAxQykKZGEyMCBhdCBtcHQzIGJ1cyAwIHNjYnVzMyB0YXJnZXQgMSBsdW4gMApkYTIw OiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUg ZGV2aWNlIApkYTIwOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGEyMDogQ29tbWFuZCBRdWV1ZWlu ZyBlbmFibGVkCmRhMjA6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAy NTVIIDYzUy9UIDI0MzIwMUMpCmRhMTggYXQgbXB0MiBidXMgMCBzY2J1czIgdGFyZ2V0IDcgbHVu IDAKZGExODogPEFUQSBIaXRhY2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3Mg U0NTSS01IGRldmljZSAKZGExODogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMTg6IENvbW1hbmQg UXVldWVpbmcgZW5hYmxlZApkYTE4OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2Vj dG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQpkYTIxIGF0IG1wdDMgYnVzIDAgc2NidXMzIHRhcmdl dCAyIGx1biAwCmRhMjE6IDxBVEEgSGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3Qg QWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhMjE6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTIxOiBD b21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGEyMTogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBi eXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykKZGEyMiBhdCBtcHQzIGJ1cyAwIHNjYnVz MyB0YXJnZXQgMyBsdW4gMApkYTIyOiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQg RGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApkYTIyOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMK ZGEyMjogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhMjI6IDE5MDc3MjlNQiAoMzkwNzAyOTE2 OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMpCmRhMjMgYXQgbXB0MyBidXMg MCBzY2J1czMgdGFyZ2V0IDQgbHVuIDAKZGEyMzogPEFUQSBIaXRhY2hpIEhEUzVDMzAyIEE4MDA+ IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGEyMzogMzAwLjAwME1CL3MgdHJh bnNmZXJzCmRhMjM6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTIzOiAxOTA3NzI5TUIgKDM5 MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQpkYTI0IGF0IG1w dDMgYnVzIDAgc2NidXMzIHRhcmdldCA1IGx1biAwCmRhMjQ6IDxBVEEgSGl0YWNoaSBIRFM1QzMw MiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhMjQ6IDMwMC4wMDBN Qi9zIHRyYW5zZmVycwpkYTI0OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGEyNDogMTkwNzcy OU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykKZGEy NSBhdCBtcHQzIGJ1cyAwIHNjYnVzMyB0YXJnZXQgNiBsdW4gMApkYTI1OiA8QVRBIEhpdGFjaGkg SERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApkYTI1OiAz MDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGEyNTogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhMjU6 IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIw MUMpCmRhMjYgYXQgbXB0MyBidXMgMCBzY2J1czMgdGFyZ2V0IDcgbHVuIDAKZGEyNjogPEFUQSBI aXRhY2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAK ZGEyNjogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMjY6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxl ZApkYTI2OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1Mv VCAyNDMyMDFDKQpkYTI3IGF0IG1wdDQgYnVzIDAgc2NidXM0IHRhcmdldCAwIGx1biAwCmRhMjc6 IDxBVEEgSGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBk ZXZpY2UgCmRhMjc6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTI3OiBDb21tYW5kIFF1ZXVlaW5n IGVuYWJsZWQKZGEyNzogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1 NUggNjNTL1QgMjQzMjAxQykKZGEzNCBhdCBtcHQ1IGJ1cyAwIHNjYnVzNSB0YXJnZXQgMyBsdW4g MApkYTM0OiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBT Q1NJLTUgZGV2aWNlIApkYTM0OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGEzNDogQ29tbWFuZCBR dWV1ZWluZyBlbmFibGVkCmRhMzQ6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0 b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMpCmRhMzUgYXQgbXB0NSBidXMgMCBzY2J1czUgdGFyZ2V0 IDQgbHVuIDAKZGEzNTogPEFUQSBIaXRhY2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBB Y2Nlc3MgU0NTSS01IGRldmljZSAKZGEzNTogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMzU6IENv bW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTM1OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5 dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQpkYTI4IGF0IG1wdDQgYnVzIDAgc2NidXM0 IHRhcmdldCAyIGx1biAwCmRhMjg6IDxBVEEgSGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhlZCBE aXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhMjg6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpk YTI4OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGEyODogMTkwNzcyOU1CICgzOTA3MDI5MTY4 IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAxQykKZGEyOSBhdCBtcHQ0IGJ1cyAw IHNjYnVzNCB0YXJnZXQgMyBsdW4gMApkYTI5OiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4g Rml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApkYTI5OiAzMDAuMDAwTUIvcyB0cmFu c2ZlcnMKZGEyOTogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhMjk6IDE5MDc3MjlNQiAoMzkw NzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMpCmRhMzAgYXQgbXB0 NCBidXMgMCBzY2J1czQgdGFyZ2V0IDQgbHVuIDAKZGEzMDogPEFUQSBIaXRhY2hpIEhEUzVDMzAy IEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGEzMDogMzAwLjAwME1C L3MgdHJhbnNmZXJzCmRhMzA6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTMwOiAxOTA3NzI5 TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQpkYTM2 IGF0IG1wdDUgYnVzIDAgc2NidXM1IHRhcmdldCA1IGx1biAwCmRhMzY6IDxBVEEgSGl0YWNoaSBI RFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktNSBkZXZpY2UgCmRhMzY6IDMw MC4wMDBNQi9zIHRyYW5zZmVycwpkYTM2OiBDb21tYW5kIFF1ZXVlaW5nIGVuYWJsZWQKZGEzNjog MTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgMjQzMjAx QykKZGEzNyBhdCBtcHQ1IGJ1cyAwIHNjYnVzNSB0YXJnZXQgNiBsdW4gMApkYTM3OiA8QVRBIEhp dGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFjY2VzcyBTQ1NJLTUgZGV2aWNlIApk YTM3OiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGEzNzogQ29tbWFuZCBRdWV1ZWluZyBlbmFibGVk CmRhMzc6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAyNTVIIDYzUy9U IDI0MzIwMUMpCmRhMzggYXQgbXB0NSBidXMgMCBzY2J1czUgdGFyZ2V0IDcgbHVuIDAKZGEzODog PEFUQSBIaXRhY2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERpcmVjdCBBY2Nlc3MgU0NTSS01IGRl dmljZSAKZGEzODogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRhMzg6IENvbW1hbmQgUXVldWVpbmcg ZW5hYmxlZApkYTM4OiAxOTA3NzI5TUIgKDM5MDcwMjkxNjggNTEyIGJ5dGUgc2VjdG9yczogMjU1 SCA2M1MvVCAyNDMyMDFDKQpkYTMxIGF0IG1wdDQgYnVzIDAgc2NidXM0IHRhcmdldCA1IGx1biAw CmRhMzE6IDxBVEEgSGl0YWNoaSBIRFM1QzMwMiBBODAwPiBGaXhlZCBEaXJlY3QgQWNjZXNzIFND U0ktNSBkZXZpY2UgCmRhMzE6IDMwMC4wMDBNQi9zIHRyYW5zZmVycwpkYTMxOiBDb21tYW5kIFF1 ZXVlaW5nIGVuYWJsZWQKZGEzMTogMTkwNzcyOU1CICgzOTA3MDI5MTY4IDUxMiBieXRlIHNlY3Rv cnM6IDI1NUggNjNTL1QgMjQzMjAxQykKZGEzMiBhdCBtcHQ0IGJ1cyAwIHNjYnVzNCB0YXJnZXQg NiBsdW4gMApkYTMyOiA8QVRBIEhpdGFjaGkgSERTNUMzMDIgQTgwMD4gRml4ZWQgRGlyZWN0IEFj Y2VzcyBTQ1NJLTUgZGV2aWNlIApkYTMyOiAzMDAuMDAwTUIvcyB0cmFuc2ZlcnMKZGEzMjogQ29t bWFuZCBRdWV1ZWluZyBlbmFibGVkCmRhMzI6IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0 ZSBzZWN0b3JzOiAyNTVIIDYzUy9UIDI0MzIwMUMpCmRhMzMgYXQgbXB0NCBidXMgMCBzY2J1czQg dGFyZ2V0IDcgbHVuIDAKZGEzMzogPEFUQSBIaXRhY2hpIEhEUzVDMzAyIEE4MDA+IEZpeGVkIERp cmVjdCBBY2Nlc3MgU0NTSS01IGRldmljZSAKZGEzMzogMzAwLjAwME1CL3MgdHJhbnNmZXJzCmRh MzM6IENvbW1hbmQgUXVldWVpbmcgZW5hYmxlZApkYTMzOiAxOTA3NzI5TUIgKDM5MDcwMjkxNjgg NTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAyNDMyMDFDKQpTTVA6IEFQIENQVSAjNSBMYXVu Y2hlZCEKU01QOiBBUCBDUFUgIzEgTGF1bmNoZWQhClNNUDogQVAgQ1BVICM3IExhdW5jaGVkIQpT TVA6IEFQIENQVSAjNCBMYXVuY2hlZCEKU01QOiBBUCBDUFUgIzYgTGF1bmNoZWQhClNNUDogQVAg Q1BVICMyIExhdW5jaGVkIQpTTVA6IEFQIENQVSAjMyBMYXVuY2hlZCEKVGltZWNvdW50ZXIgIlRT Qy1sb3ciIGZyZXF1ZW5jeSA4OTg0OTIzIEh6IHF1YWxpdHkgODAwCnVnZW4wLjI6IDxBbWVyaWNh biBNZWdhdHJlbmRzIEluYy4+IGF0IHVzYnVzMAp1a2JkMDogPEtleWJvYXJkIEludGVyZmFjZT4g b24gdXNidXMwCmtiZDAgYXQgdWtiZDAKdW1zMDogPE1vdXNlIEludGVyZmFjZT4gb24gdXNidXMw CnVnZW4xLjI6IDx2ZW5kb3IgMHgwNGI0PiBhdCB1c2J1czEKdWh1YjI6IDx2ZW5kb3IgMHgwNGI0 IHByb2R1Y3QgMHg2NTYwLCBjbGFzcyA5LzAsIHJldiAyLjAwLzAuMGIsIGFkZHIgMj4gb24gdXNi dXMxCnVodWIyOiBNVFQgZW5hYmxlZAp1aHViMjogNCBwb3J0cyB3aXRoIDQgcmVtb3ZhYmxlLCBz ZWxmIHBvd2VyZWQKVHJ5aW5nIHRvIG1vdW50IHJvb3QgZnJvbSB6ZnM6enJvb3QgW10uLi4KSVAg RmlsdGVyOiB2NC4xLjI4IGluaXRpYWxpemVkLiAgRGVmYXVsdCA9IHBhc3MgYWxsLCBMb2dnaW5n ID0gZW5hYmxlZApiY2UwOiBHaWdhYml0IGxpbmsgdXAhCmJjZTA6IEdpZ2FiaXQgbGluayB1cCEK ------=_Part_176556_478359006.1370336069968-- From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 09:26:03 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9213C1A8 for ; Tue, 4 Jun 2013 09:26:03 +0000 (UTC) (envelope-from prvs=1867102569=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 3770A15C6 for ; Tue, 4 Jun 2013 09:26:02 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004150401.msg for ; Tue, 04 Jun 2013 10:26:02 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 04 Jun 2013 10:26:02 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1867102569=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: From: "Steven Hartland" To: "Pascal Braun, Continum" , "Ronald Klop" References: <1261869099.176558.1370336069971.JavaMail.root@continum.net> Subject: Re: ZFS crashing while zfs recv in progress Date: Tue, 4 Jun 2013 10:26:00 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 09:26:03 -0000 If your seeing swap_pager warnings it indicates two things:- 1. Your seeing memory pressure 2. Something is causing significant delay on transations to disk. Additional information which maybe useful:- * mptutil show adapter * iostat 1 * Capture from top when the system is hung Regards Steve ----- Original Message ----- From: "Pascal Braun, Continum" To: "Ronald Klop" Cc: Sent: Tuesday, June 04, 2013 9:54 AM Subject: Re: ZFS crashing while zfs recv in progress I've put swap on a seperate disk this time and re-run the zfs send / recv, but i'm still getting the problem. Whats interesting is, i'm still getting the same output about swap space on the console: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 36, size: 24576 The System is running from zroot and i'm trying to transfer to tank. zpool Status output: --- pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/disk3 ONLINE 0 0 0 gpt/disk9 ONLINE 0 0 0 gpt/disk15 ONLINE 0 0 0 gpt/disk19 ONLINE 0 0 0 gpt/disk23 ONLINE 0 0 0 gpt/disk27 ONLINE 0 0 0 gpt/disk31 ONLINE 0 0 0 gpt/disk36 ONLINE 0 0 0 gpt/disk33 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk4 ONLINE 0 0 0 gpt/disk7 ONLINE 0 0 0 gpt/disk10 ONLINE 0 0 0 gpt/disk13 ONLINE 0 0 0 gpt/disk16 ONLINE 0 0 0 gpt/disk24 ONLINE 0 0 0 gpt/disk32 ONLINE 0 0 0 gpt/disk37 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 gpt/disk5 ONLINE 0 0 0 gpt/disk8 ONLINE 0 0 0 gpt/disk11 ONLINE 0 0 0 gpt/disk17 ONLINE 0 0 0 gpt/disk21 ONLINE 0 0 0 gpt/disk25 ONLINE 0 0 0 gpt/disk29 ONLINE 0 0 0 gpt/disk38 ONLINE 0 0 0 raidz2-3 ONLINE 0 0 0 gpt/disk12 ONLINE 0 0 0 gpt/disk14 ONLINE 0 0 0 gpt/disk18 ONLINE 0 0 0 gpt/disk22 ONLINE 0 0 0 gpt/disk26 ONLINE 0 0 0 gpt/disk30 ONLINE 0 0 0 gpt/disk34 ONLINE 0 0 0 gpt/disk35 ONLINE 0 0 0 gpt/disk39 ONLINE 0 0 0 spares gpt/disk20 AVAIL errors: No known data errors pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk6 ONLINE 0 0 0 errors: No known data errors --- Attatched you'll also find the dmesg.boot. thanks, Pascal ----- Ursprüngliche Mail ----- > On Wed, 29 May 2013 11:04:00 +0200, Pascal Braun, Continum > > Please send more information about the new server. Sometimes there > are > bugs found in drivers with large disks, etc. Or firmware of hardware. > The contents of /var/run/dmesg.boot is interesting to a lot of > people. > As is the output of zpool status. > > As you are having trouble with swap on zfs. Is it possible to put > that on > a separate disk for the test? > > Ronald. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" > -------------------------------------------------------------------------------- > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 09:53:23 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 63A35E9D for ; Tue, 4 Jun 2013 09:53:23 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 0566317AE for ; Tue, 4 Jun 2013 09:53:22 +0000 (UTC) Received: from mfilter24-d.gandi.net (mfilter24-d.gandi.net [217.70.178.152]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id 80989A80D9; Tue, 4 Jun 2013 11:53:05 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter24-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter24-d.gandi.net (mfilter24-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id 2Q98JrRbtRC5; Tue, 4 Jun 2013 11:53:03 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id A99E5A80F6; Tue, 4 Jun 2013 11:53:02 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id A58DF73A1C; Tue, 4 Jun 2013 02:53:00 -0700 (PDT) Date: Tue, 4 Jun 2013 02:53:00 -0700 From: Jeremy Chadwick To: "Pascal Braun, Continum" Subject: Re: ZFS crashing while zfs recv in progress Message-ID: <20130604095300.GA79993@icarus.home.lan> References: <1261869099.176558.1370336069971.JavaMail.root@continum.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1261869099.176558.1370336069971.JavaMail.root@continum.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Ronald Klop X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 09:53:23 -0000 On Tue, Jun 04, 2013 at 10:54:30AM +0200, Pascal Braun, Continum wrote: > I've put swap on a seperate disk this time and re-run the zfs send / recv, but i'm still getting the problem. Whats interesting is, i'm still getting the same output about swap space on the console: > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 36, size: 24576 What this means is covered here: http://www.freebsd.org/doc/en/books/faq/troubleshoot.html#idp75389104 Google "swap_pager: indefinite wait buffer" and you will see lots of people talking about this. The commonality in most situations is that the I/O subsystem is stalled or too busy to swap in/out a page of memory, and the kernel throws a nastygram about it. Basically, your I/O subsystem is taking too long to accomplish a task (swap in/out a page of memory to/from swap). It could be the disk taking too long, it could be the controller taking too long, it could be the disk being in bad shape, it could be the overall PCI/PCI-X/PCIe bus being overwhelmed, or could be something as simple as excessive CPU load (your pool/vdev setup I imagine is very CPU intensive). Your system is utterly massive. Warning: this is the first time I have taken a stab at such a huge system. The topology, for those wondering: * mpt0 (LSI SATA/SAS; rev 1.5.20.0), irq 17 |-- 6 disks attached |-> da0 = Hitachi HDS72101 A3MA, 1953525168 sectors |-> da1 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da2 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da3 = Hitachi HDS5C302 A580, 3907029168 sectors |-> da4 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da5 = Hitachi HDS5C302 A800, 3907029168 sectors * mpt1 (LSI SATA/SAS; rev 1.5.20.0), irq 18 |-- 7 disks attached |-> da6 = Hitachi HDS72101 A3MA, 1953525168 sectors |-> da7 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da8 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da9 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da10 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da11 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da12 = Hitachi HDS5C302 A800, 3907029168 sectors * mpt2 (LSI SATA/SAS; rev 1.5.20.0), irq 18 |-- 6 disks attached |-> da13 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da14 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da15 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da16 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da17 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da18 = Hitachi HDS5C302 A800, 3907029168 sectors * mpt3 (LSI SATA/SAS; rev 1.5.20.0), irq 40 |-- 8 disks attached |-> da19 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da20 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da21 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da22 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da23 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da24 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da25 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da26 = Hitachi HDS5C302 A800, 3907029168 sectors * mpt4 (LSI SATA/SAS; rev 1.5.20.0), irq 41 |-- 7 disks attached |-> da27 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da28 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da29 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da30 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da31 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da32 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da33 = Hitachi HDS5C302 A800, 3907029168 sectors * mpt5 (LSI SATA/SAS; rev 1.5.20.0), irq 41 (shares IRQ with mpt4) |-- 5 disks attached |-> da34 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da35 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da36 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da37 = Hitachi HDS5C302 A800, 3907029168 sectors |-> da38 = Hitachi HDS5C302 A800, 3907029168 sectors Things to note / things that caught my eye: - mpt2 shares an IRQ with mpt1 (irq 18) - mpt5 shares an IRQ with mpt4 (irq 41) - Disk da3 has a different drive firmware (A580) than the A800 drives. - (For readers only): Disks {da0,da6} are a completely different model and capacity than all the other drives in the array. These drives are used (in a mirror) for the ZFS root pool ("zroot"), not the big fat gigantic pool ("tank"). - I have not verified if any of these disks use 4KByte sectors (dmesg is not going to tell you the entire truth). I would appreciate seeing "smartctl -x" output from {da0,da1,da3} so I could get an idea. Your pools use gpt labelling so I am left with the hope that your labels refer to the partition with proper 4KB alignment regardless. - The system has only 16GB RAM. That's a bit shocking for something of this size. Moving on. Can you tell me what exact disk (e.g. daXX) in the above list you used for swap, and what kind of both system and disk load were going on at the time you saw the swap message? I'm looking for a capture of "gstat -I500ms" output (you will need a VERY long/big terminal window to capture this given how many disks you have) while I/O is happening, as well as "top -s 1" in another window. I would also like to see "zpool iostat -v 1" output while things are going on, to help possibly narrow down if there is a single disk causing the entire I/O subsystem for that controller to choke. Next: are you using compression or dedup on any of your filesystems? If not, have you ever in the past? Next: could we have your loader.conf and sysctl.conf please? My gut feeling is that if you're doing zfs {send,recv} for "tank" -- which you are -- multiple subsystems and busses are so incredibly overwhelmed by all the I/O and interrupts and *everything* that it's very hard for the swap I/O time slicer to get a decent share of time to swap something out to swap (even worse if that controller is overwhelmed with requests). Worse, you're using raidz2, which means even more CPU time + calculation overhead, which means less time for other tasks (threads). Everything on the system -- everything! -- is fighting for time at multiple levels. If you could put a swap disk on a dedicated controller (and no other disks on it), that would be ideal. Please do not use USB for this task (the USB stack may introduce its own set of complexities pertaining to interrupt usage). If all this turns out to be an "overall system overwhelmed" situation, my advice is to cut back on the usage. I would STRONGLY suggest in that case a 2nd system, and split the number of disks across both. I'm really surprised given how many disks/etc. you have you didn't choose to get an actual filer (Netapp). I sure as hell would have. I really do not know why people think ZFS is a full-blown replacement for a Netapp of this scale -- it isn't. Anyway take what I say with a grain of salt -- really. I'm just throwing out thoughts/ideas as I look over everything. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 16:32:30 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 375E8A70 for ; Tue, 4 Jun 2013 16:32:30 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: from mail-ie0-x230.google.com (mail-ie0-x230.google.com [IPv6:2607:f8b0:4001:c03::230]) by mx1.freebsd.org (Postfix) with ESMTP id 0F5781CD8 for ; Tue, 4 Jun 2013 16:32:30 +0000 (UTC) Received: by mail-ie0-f176.google.com with SMTP id at20so869262iec.35 for ; Tue, 04 Jun 2013 09:32:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=VAN45vDMTDGpXcTik4KRTZDluGO+MZgQf6FeSNy59t0=; b=fRr8unQ9arjk1P5gBci+LGds4yMnU5DFM6p2vnWCsZTfx5y83AwUszGjN8Jk6OJrFa 8RYX28seOTjypegI35NBQG6OriRZai/8wevuS5x9G/pcmkdZUpxu1iVLbp6U1x8mMVI+ rsTQfQVhyshqazokiongUSpjP3oULM03CXbQofpLMxFUCgIwdiRuDUuTkqKkMQ+17Pcc V296aZ5LreAa6hRb6619KdskazL8alUHfSODz37ueCNMAnqDGHC4E8Er7XgMibFaNBNi pTBbxCPdGlpuJ4si85GEJb4RwFJrDu/tycqMgfj0BT194+0+1dccM5OMwlFJKI9N71Bl +jLA== MIME-Version: 1.0 X-Received: by 10.50.109.231 with SMTP id hv7mr1288222igb.103.1370363549802; Tue, 04 Jun 2013 09:32:29 -0700 (PDT) Received: by 10.50.74.193 with HTTP; Tue, 4 Jun 2013 09:32:29 -0700 (PDT) In-Reply-To: <803931797.154623.1369818240686.JavaMail.root@continum.net> References: <631084870.154593.1369817992337.JavaMail.root@continum.net> <803931797.154623.1369818240686.JavaMail.root@continum.net> Date: Tue, 4 Jun 2013 11:32:29 -0500 Message-ID: Subject: Re: ZFS crashing while zfs recv in progress From: Scot Hetzel To: "Pascal Braun, Continum" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 16:32:30 -0000 On Wed, May 29, 2013 at 4:04 AM, Pascal Braun, Continum wrote: > > Hi, > > > I'm trying to send a zfs pool from an old freebsd 9.0 installation to a n= ew machine with freebsd 9.1. The pool is quite heavy (about 16TB, lots of s= napshots) and the receiving side keeps crashing on me. The command used to = transfer (run on the old 9.0 installation): > zfs send -R tank@snapshot | ssh 10.10.xx.xx zfs recv -F -d -v tank > > > After a few hours the system stops all writing and I can't start any new = processes. Processes still running like 'zpool iostat' are still working, o= r at least it is still reporting something. To me it looks like the filesys= tem just disappeared. Unfortunately I'm running root on zfs so I don't have= any logs about this. > The only message I sometimes find on the console are about not being able= to write to swap, which is also on zfs. > This could be where your problem is happening. While you can create a swap vol on ZFS, that swap vol also requires available memory from the system to perform the swap. As was suggested, try using a dedicated disk / partition as your swap volume. > > Do you have any ideas? I don't even know where to start. > > > regards, Pascal > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" --=20 DISCLAIMER: No electrons were maimed while sending this message. Only slightly bruised. From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 21:32:52 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 009369D0 for ; Tue, 4 Jun 2013 21:32:51 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) by mx1.freebsd.org (Postfix) with ESMTP id 901BF1D13 for ; Tue, 4 Jun 2013 21:32:51 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id hm9so669365wib.0 for ; Tue, 04 Jun 2013 14:32:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:message-id:date :to:mime-version:x-mailer; bh=7pSf+jPXFmUyQAv06L4iAT8Pv1vcmeQRlzZXQ9Ko8ks=; b=xMpAonIZnO1g2HthtQm9+m5FMO7m/8dRqqvEVrENp4eoJ8wmGkBecqDuAuqn8X7+aJ i5VfN6ayD0zRA8scHEMTwYmfTK1pr7OcEqwolakg2oVtvhmwDaZ/qLXEA/rOQLVNs5yF FofZzUW/mVB53nOqLdH58zVHEJBzrD7ifcJetzODNCmLGRgVR0i16jgHyNCLjZmaKWe+ fSR5ghk47l3wwx4CWJLSvCRNDI8ID//TohFMhqLxoNr+WKykYAsYDb9+uISyx4kDHjQA 73LT0qU+lZgTYl+JCoxfeZQ1h6bdxgsDwdE0RXHUdi0eBmRtEAQLGdYDw3/EacLDhGdG p1mg== X-Received: by 10.180.206.235 with SMTP id lr11mr3385981wic.59.1370381570789; Tue, 04 Jun 2013 14:32:50 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id h8sm5583928wiz.9.2013.06.04.14.32.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Jun 2013 14:32:49 -0700 (PDT) From: Alban Hertroys Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Serial terminal issues Message-Id: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> Date: Tue, 4 Jun 2013 23:32:48 +0200 To: "freebsd-stable@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) X-Mailer: Apple Mail (2.1503) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 21:32:52 -0000 I can't seem to get my serial terminal to work with my new system. I had a serial terminal connected to my old system that worked great and = copied /boot.config, /boot/loader.conf and /etc/ttys settings over from = it. The new system has a Gigabyte GA970A-UD3 board with just a serial header = on the board. I bought a serial connector backplate in an electronics = store and connected it to the board. Could the pinout be different or = something? > cat /boot.config=20 -D -S19200 > cat /boot/loader.conf=20 boot_multicons=3D"YES" boot_serial=3D"YES" comconsole_speed=3D"19200" console=3D"comconsole,vidconsole" > cat /etc/ttys ttyv0 "/usr/libexec/getty Pc" xterm on secure # Virtual terminals ttyv1 "/usr/libexec/getty Pc" xterm on secure ttyv2 "/usr/libexec/getty Pc" xterm on secure ttyv3 "/usr/libexec/getty Pc" xterm on secure ttyv4 "/usr/libexec/getty Pc" xterm on secure ttyv5 "/usr/libexec/getty Pc" xterm on secure ttyv6 "/usr/libexec/getty Pc" xterm on secure ttyv7 "/usr/libexec/getty Pc" xterm on secure ttyv8 "/usr/local/bin/xdm -nodaemon" xterm off secure # Serial terminals # The 'dialup' keyword identifies dialin lines to login, fingerd etc. ttyu0 "/usr/libexec/getty std.19200" vt320 on secure ttyu1 "/usr/libexec/getty std.9600" dialup off secure ttyu2 "/usr/libexec/getty std.9600" dialup off secure ttyu3 "/usr/libexec/getty std.9600" dialup off secure # Dumb console dcons "/usr/libexec/getty std.9600" vt100 off secure =46rom /var/log/messages: Jun 4 21:28:50 solfertje kernel: uart0: <16550 or compatible> port = 0x3f8-0x3ff irq=20 4 flags 0x10 on acpi0 Jun 4 21:28:50 solfertje kernel: uart0: console (19200,n,8,1) Jun 4 21:28:50 solfertje kernel: orm0: at iomem = 0xd5000-0xd67ff,0xd7000-0xd7fff on isa0 Jun 4 21:28:50 solfertje kernel: sc0: at flags 0x100 = on isa0 Jun 4 21:28:50 solfertje kernel: sc0: VGA <16 virtual consoles, = flags=3D0x300> Jun 4 21:28:50 solfertje kernel: vga0: at port = 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Jun 4 21:28:50 solfertje kernel: atkbdc0: = at port 0x60,0x64 on isa0 Jun 4 21:28:50 solfertje kernel: atkbd0: irq 1 on atkbdc0 Jun 4 21:28:50 solfertje kernel: kbd0 at atkbd0 Jun 4 21:28:50 solfertje kernel: atkbd0: [GIANT-LOCKED] I didn't see any options in the BIOS to set the console speed (just = address and IRQ, those are in the above). ISTR that my old mobo did = allow to set that information, but then again, that board (Tyan Tiger) = gave me access to the BIOS through the serial console. What am I missing? Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 22:17:10 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C375D1A9 for ; Tue, 4 Jun 2013 22:17:10 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) by mx1.freebsd.org (Postfix) with ESMTP id 88C6C1EED for ; Tue, 4 Jun 2013 22:17:10 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7::4521:e4f7:9b8a:a981] (unknown [IPv6:2001:7b8:3a7:0:4521:e4f7:9b8a:a981]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 5D9135C44; Wed, 5 Jun 2013 00:17:07 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Serial terminal issues From: Dimitry Andric In-Reply-To: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> Date: Wed, 5 Jun 2013 00:17:08 +0200 Content-Transfer-Encoding: 7bit Message-Id: References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> To: Alban Hertroys X-Mailer: Apple Mail (2.1503) Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 22:17:10 -0000 On Jun 4, 2013, at 23:32, Alban Hertroys wrote: > I can't seem to get my serial terminal to work with my new system. ... >> cat /boot.config > -D -S19200 > >> cat /boot/loader.conf > boot_multicons="YES" > boot_serial="YES" > comconsole_speed="19200" > console="comconsole,vidconsole" Does it work at 9600 baud? Otherwise, check RTS/CTS, etc. -Dimitry From owner-freebsd-stable@FreeBSD.ORG Tue Jun 4 22:57:31 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2471AE1A; Tue, 4 Jun 2013 22:57:31 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id DC4931117; Tue, 4 Jun 2013 22:57:30 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r54MvUeH048707; Tue, 4 Jun 2013 16:57:30 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r54MvUeI048704; Tue, 4 Jun 2013 16:57:30 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Tue, 4 Jun 2013 16:57:30 -0600 (MDT) From: Warren Block To: Dimitry Andric Subject: Re: Serial terminal issues In-Reply-To: Message-ID: References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Tue, 04 Jun 2013 16:57:30 -0600 (MDT) Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 22:57:31 -0000 On Wed, 5 Jun 2013, Dimitry Andric wrote: > On Jun 4, 2013, at 23:32, Alban Hertroys wrote: >> I can't seem to get my serial terminal to work with my new system. > ... >>> cat /boot.config >> -D -S19200 >> >>> cat /boot/loader.conf >> boot_multicons="YES" >> boot_serial="YES" >> comconsole_speed="19200" >> console="comconsole,vidconsole" > > Does it work at 9600 baud? Otherwise, check RTS/CTS, etc. Also, check that the ten-pin header is on in the right orientation, not 180 degrees around. From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 00:59:34 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F3BC9C89 for ; Wed, 5 Jun 2013 00:59:33 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx1.freebsd.org (Postfix) with ESMTP id 94D1C19A2 for ; Wed, 5 Jun 2013 00:59:33 +0000 (UTC) Received: from mfilter22-d.gandi.net (mfilter22-d.gandi.net [217.70.178.150]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id F06D4A80BC; Wed, 5 Jun 2013 02:59:21 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter22-d.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter22-d.gandi.net (mfilter22-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id ci3Tc94o1mmD; Wed, 5 Jun 2013 02:59:20 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id D6C2FA80B5; Wed, 5 Jun 2013 02:59:19 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 0F6AE73A33; Tue, 4 Jun 2013 17:59:18 -0700 (PDT) Date: Tue, 4 Jun 2013 17:59:18 -0700 From: Jeremy Chadwick To: Alban Hertroys Subject: Re: Serial terminal issues Message-ID: <20130605005918.GA5709@icarus.home.lan> References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 00:59:34 -0000 On Tue, Jun 04, 2013 at 11:32:48PM +0200, Alban Hertroys wrote: > I can't seem to get my serial terminal to work with my new system. > > I had a serial terminal connected to my old system that worked great > and copied /boot.config, /boot/loader.conf and /etc/ttys settings over > from it. > > The new system has a Gigabyte GA970A-UD3 board with just a serial > header on the board. I bought a serial connector backplate in an > electronics store and connected it to the board. Could the pinout be > different or something? It is very possible. You should have asked Gigabyte what exact product (specifically part number) to purchase that provided a header-to-backplane DB9 port, or if they could send you one (many will for free). Always use what the mainboard vendor tells you. Always. It's also very possible the cable you're using to connect from the Gigabyte board to something (you didn't disclose what) is wired wrong. > > cat /boot.config > -D -S19200 Please try using "-S19200 -Dh" (please read carefully). The order of the arguments may matter as well (there has been some speculation on the lists about this, and I do not care to do the analysis; just passing information on blindly). > > cat /boot/loader.conf > boot_multicons="YES" > boot_serial="YES" > comconsole_speed="19200" > console="comconsole,vidconsole" You do not need any of this given what /boot.config contains. Please remove it all. > > cat /etc/ttys > ... > ttyu0 "/usr/libexec/getty std.19200" vt320 on secure This is completely correct. > >From /var/log/messages: > Jun 4 21:28:50 solfertje kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq > 4 flags 0x10 on acpi0 > Jun 4 21:28:50 solfertje kernel: uart0: console (19200,n,8,1) > Jun 4 21:28:50 solfertje kernel: orm0: at iomem 0xd5000-0xd67ff,0xd7000-0xd7fff on isa0 > Jun 4 21:28:50 solfertje kernel: sc0: at flags 0x100 on isa0 > Jun 4 21:28:50 solfertje kernel: sc0: VGA <16 virtual consoles, flags=0x300> > Jun 4 21:28:50 solfertje kernel: vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Jun 4 21:28:50 solfertje kernel: atkbdc0: at port 0x60,0x64 on isa0 > Jun 4 21:28:50 solfertje kernel: atkbd0: irq 1 on atkbdc0 > Jun 4 21:28:50 solfertje kernel: kbd0 at atkbd0 > Jun 4 21:28:50 solfertje kernel: atkbd0: [GIANT-LOCKED] Okay, so what's the problem? You're not seeing any I/O on the serial port? How do you have your serial device connected to something on the other end? Meaning: solfertje has a serial port, and it's connected to what exactly? Are you sure the device its connected to is working? Is the cable a null modem cable (if it's attached to another PC), etc. etc. etc... Does your serial cable have **correct** hardware flow control (CTS/RTS) wiring, along with proper DCD tie-in? Many "adapters" or cables do not do this correct. The official handbook says this: http://www.freebsd.org/doc/en/articles/console-server/cabling.html Which appears to have been updated in some manner of speaking, making things "extra complicated". Below is a document I wrote when using FreeBSD x86 systems (those with DB9 serial ports), using a DB9-to-RJ45 adapter with *proper* wiring, to connect to MRV or Xyplex units -- the important part is what the RJ45 ends are signal-wise. Again, these use PROPER CTS/RTS flow control with proper DCD tie-in, and work with the "std.xxxxx" ttys(5) entries. ========= The LX-4016S-001 sports sixteen (16) RJ45 connectors. The cabling for the console port (Diag/Mgmt Port, also known as Port 0) comes with the unit. The cable to use for this port appears to be some sort of rollover cable (Cisco?); it's silver and flat, not round like CAT5. It's 8-pin though. A DB9 adapter also comes with this cable, which allows you to hook it up to a standard PC and access it. Ports labelled 1-16 require RJ45-to-DB9 adapters for hooking the MRV up to actual servers in the rack. The adapters that work best are the Xyplex XFDCE91 adapters, which support hardware flow control (RTS/CTS) and come pre-assembled. You can find the product here: Xyplex adapter XFDCE91 RJ45-to-DB9 APACN p/n 24490-15 -- http://www.apacn.com/ Pinout/wiring diagram: RJ45 DB9 Female Female =========== ======= (CTS) 1 <----> 7 (RTS) (DTR) 2 <----> 6 (DSR) (TxD) 3 <----> 2 (RxD) (TxD GND) 4 <----> 5 (GND) (RxD GND) 5 <----> 5 (GND) (RxD) 6 <----> 3 (TxD) (DSR/DCD) 7 <----> 4 (DTR) (RTS) 8 <----> 8 (CTS) Pins 1 (DCD) and 9 (RI) on the DB9 are unconnected/unused. With these adapters, use standard (not crossover!) CAT5/6 cables. ========= This diagram should allow you to build your own cable if need be, including a null-modem cable if you plan on doing a PC<-->PC (i.e. DB9 to DB9) connection. I just happened to use RJ45 stuff; for DB9-to-DB9; just follow the chart (signal/pin names) and go from there. If you already have a cable and you aren't sure of its wiring (which is very common -- sigh, stupid companies...), you will need to figure out the wiring using a multimetre (continuity test is all that's needed). > I didn't see any options in the BIOS to set the console speed (just > address and IRQ, those are in the above). ISTR that my old mobo did > allow to set that information, but then again, that board (Tyan Tiger) > gave me access to the BIOS through the serial console. This has absolutely no relevancy. Serial port speed settings in a BIOS pertain to BIOS-level console redirection -- that redirection is lost the instant anything (boot loader, kernel, etc.) touches SMI and/or interrupts and starts "fiddling" with the serial port. What you're adjusting in FreeBSD is 1) the FreeBSD boot loader touching the serial port, and 2) the FreeBSD kernel outputting to a serial port (it also initialises/sets the serial port), and 3) getty et al spawning a login prompt on the serial port. I would point you to my "FreeBSD via serial console and PXE" document, except there are one-offs specific to the PXE portions that are not relevant to your situation. The important part is that I've used FreeBSD serial console for almost 16 years and have a very good understanding of what works (including vs. what some developers say "should" work; i.e. reality vs. pragmatism). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 14:59:00 2013 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 56F12E57 for ; Wed, 5 Jun 2013 14:59:00 +0000 (UTC) (envelope-from office@bandagist-gattringer.at) Received: from vps004.i-gap.at (vps004.i-gap.at [83.170.68.62]) by mx1.freebsd.org (Postfix) with ESMTP id AFDFA1DAF for ; Wed, 5 Jun 2013 14:58:59 +0000 (UTC) Received: from eisrsv ([123.89.6.74]) (authenticated bits=0) by vps004.i-gap.at (8.13.8/8.13.8) with ESMTP id r55EwjwX004845 for ; Wed, 5 Jun 2013 16:58:53 +0200 Sender: office@bandagist-gattringer.at Date: Wed, 5 Jun 2013 22:58:54 +0800 From: "xycouture" To: Subject: Welcome your visit our factory in ChaoZhou China! Message-ID: <20130605225900376340@bandagist-gattringer.at> X-mailer: Foxmail 6, 13, 102, 15 [cn] Mime-Version: 1.0 X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-3.0 (vps004.i-gap.at [83.170.68.62]); Wed, 05 Jun 2013 16:58:58 +0200 (CEST) X-Spam-Status: No, score=1.4 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE, MIME_BASE64_TEXT autolearn=disabled version=3.2.5 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on vps004.i-gap.at Content-Type: text/plain; charset="utf-7" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: xyeveningdress@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 14:59:00 -0000 aG0NCkNIQU9aSE9VIFhVQU5ZSSBGQVNISU9OIENPLixMVEQgICAgIEV2ZW5pbmcgRHJlc3NlcyAg ICAgV2VkZGluZyBEcmVzc2VzICAgICBCcmlkZXNtYWlkIERyZXNzICAgICAgICBDb250YWN0IHRo ZSBFbWFpbDp4eWV2ZW5pbmdkcmVzcytBRUEtZ21haWwuY29tIA0KMmNyOTl6cHFicA0KDQoyMDEz LzYvNW9jdnpwcXZpbGENCkNoYW9aaG91IFh1YW5ZaSBGYXNoaW9uIENvLixMdGQgIGlzIHJlY2Vp dmUgdGhlIHJldGFpbCBvcmRlciBhbmQgd2hvbGVzYWxlIG9yZGVyIGluIG5vdytBQ0UtDQpXZWxj b21lIHlvdXIgdmlzaXQgb3VyIGZhY3RvcnkgaW4gQ2hhb1pob3UgQ2hpbmErQUNFLQ0KZ3o2bG01 ZTJvDQogICAgICAgICAgICAgICAgICANCiAgICAgICAgICAgICAgICAgIA0KICAgICAgICAgICAg ICAgICAgDQogICAgICAgICAgICAgICAgICAgDQpRUSsveG8tMTA4MzMwOTUNCk1vYmlsZToxMzcx NTgwMjg0MQ0KRW1haWw6eHlldmVuaW5nZHJlc3MrQUVBLWdtYWlsLmNvbQ0KV2ViOmh0dHA6Ly93 d3cueHlldmVuaW5nZHJlc3MuY29tDQpBZGQ6IE5PLjEgRkVOR1hJTiBSRC4rL3d3LUNIVU5HVUFO RyBWSUxMQUdFKy93dy1DSEFPWkhPVSsvd3ctR1VBTkdET05HKy93dy1DSElOQSANCnpwZjdtNQ0K DQpxbGNxeHVuenINCm9mZmljZStBRUEtYmFuZGFnaXN0LWdhdHRyaW5nZXIuYXQgIA0KZzdhbmpi bGYzaWhtbTFvcnh0MQ== From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 16:29:40 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DE3729DA; Wed, 5 Jun 2013 16:29:40 +0000 (UTC) (envelope-from dnaeon@gmail.com) Received: from mail-pd0-f181.google.com (mail-pd0-f181.google.com [209.85.192.181]) by mx1.freebsd.org (Postfix) with ESMTP id B3EBF11D5; Wed, 5 Jun 2013 16:29:40 +0000 (UTC) Received: by mail-pd0-f181.google.com with SMTP id bv13so2050234pdb.40 for ; Wed, 05 Jun 2013 09:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=lH2EaQ4z5v4QgmNj8jBek5iCIAKkfblwXkhyzt+prs0=; b=I1emCeoxeOJu3UehCaWGiCvSZ55UunXVxau1yYD0YqOyPtgNfCy8yc/CtbcqA1HmJq N6Sme+QiVJVdYq8WtFFVxXA0VXpqsXiFFBaQOIg966B8FCb+5P6cSRDzynVmrSFU3t3v vulIfes47ocWS37dBw0SkBkQb7m94suVIch5aP6K1JRmKdwFzVQucj4SonI+WLcOMDUG 72ObY+V5K02hS/hd3Kw6aXuLw6dQT/xlzVyyMbS90GvK4W+O7SYGVtxYzswodjILFQBh qgrrLOx91y7W2WsF8IzMvOVCfpsCt+zd1+LWWbVQ5mFWr1JLwO2PSVwFmVc2i9bXM2ot u5nQ== MIME-Version: 1.0 X-Received: by 10.68.219.70 with SMTP id pm6mr34185840pbc.154.1370449780203; Wed, 05 Jun 2013 09:29:40 -0700 (PDT) Received: by 10.70.54.234 with HTTP; Wed, 5 Jun 2013 09:29:40 -0700 (PDT) Date: Wed, 5 Jun 2013 19:29:40 +0300 Message-ID: Subject: Introducing py-pkg: Python wrappers for libpkg (pkgng) From: Marin Atanasov Nikolov To: pkg@freebsd.org, "freebsd-ports@freebsd.org" , ml-freebsd-stable Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 16:29:40 -0000 Hello, I've been working on getting Python wrappers for libpkg (pkgng) for the past few weeks and I think that I've reached the point to actually announce it :) Anyway, straight to the point. You can find on the below listed links information about the libpkg Python wrappers along with example usage, documentation and of course the code repository. * Introduction to py-pkg along with example usage [1] * Github repository of py-pkg [2] * Sphinx documentation of py-pkg [3] If there's interest in py-pkg I can also submit a PR for adding this to the Ports Tree. Regards, Marin [1]: http://unix-heaven.org/py-pkg [2]: https://github.com/dnaeon/py-pkg [3]: http://jenkins.unix-heaven.org/jenkins/job/py-pkg-docs/py-pkg_Sphinx_Documentation/? -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/ From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 17:59:47 2013 Return-Path: Delivered-To: freebsd-stable@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 143743F0; Wed, 5 Jun 2013 17:59:47 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id E08A4181D; Wed, 5 Jun 2013 17:59:46 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r55Hxk53093635; Wed, 5 Jun 2013 17:59:46 GMT (envelope-from adrian@freefall.freebsd.org) Received: (from adrian@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r55HxkOU093634; Wed, 5 Jun 2013 17:59:46 GMT (envelope-from adrian) Date: Wed, 5 Jun 2013 17:59:46 GMT Message-Id: <201306051759.r55HxkOU093634@freefall.freebsd.org> To: adrian@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-stable@FreeBSD.org From: adrian@FreeBSD.org Subject: Re: i386/179112: 9.1 installer panics with a kmem_malloc() failure on i386 embedded systems X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 17:59:47 -0000 Synopsis: 9.1 installer panics with a kmem_malloc() failure on i386 embedded systems Responsible-Changed-From-To: freebsd-i386->freebsd-stable Responsible-Changed-By: adrian Responsible-Changed-When: Wed Jun 5 17:59:21 UTC 2013 Responsible-Changed-Why: Punt to -stable, as this is not specifically an i386 issue. http://www.freebsd.org/cgi/query-pr.cgi?pr=179112 From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 19:30:02 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1CD58564 for ; Wed, 5 Jun 2013 19:30:02 +0000 (UTC) (envelope-from haramrae@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) by mx1.freebsd.org (Postfix) with ESMTP id AA7E71CB8 for ; Wed, 5 Jun 2013 19:30:01 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id hn14so5318521wib.4 for ; Wed, 05 Jun 2013 12:30:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=hmv2EBZE+9up8Dl4HWtxjsxvNtGdSAO5+v3Bw2//FTE=; b=VYODZm+mN7lX+FG6Ci3DsJMIKuXiwjOPZ9X5O49d+qBxaL7vQpRFJW4Ikf3cOUNtRY Kds/kLhBxlpDE/y9qkDQOTymGh1CsbzFsDM+w1K1+jEq2UdPR7fwC/6OHPXRRH7egb4c Q7y/X8WK71kBEmib4b5dBSButEoa+BrR2FKYIliB9tCWeKaudfOsScJlEd4ZZ04R9hy2 zwEKuSZHVt5C+HiDI0+Kg2DOSQbvqFTEF6Q3d178Cdyw5uE5N7/uIfUp29E0nFpPgOPx SgaFldRZ5thNbMQjKMwNOQpIXnBwGwwRqxY6Jyk/SgPs7nZTyDgHch4KkSeqXOJ0Krut tR9Q== X-Received: by 10.194.242.229 with SMTP id wt5mr23063418wjc.36.1370460599993; Wed, 05 Jun 2013 12:29:59 -0700 (PDT) Received: from hollewijn.internal (8d690a59.ftth.concepts.nl. [141.105.10.89]) by mx.google.com with ESMTPSA id fx7sm12539751wic.11.2013.06.05.12.29.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Jun 2013 12:29:58 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Serial terminal issues From: Alban Hertroys In-Reply-To: <20130605005918.GA5709@icarus.home.lan> Date: Wed, 5 Jun 2013 21:29:56 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <93196E06-1775-46A4-A3E9-1C8A3B8C4E94@gmail.com> References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> <20130605005918.GA5709@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1503) Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 19:30:02 -0000 On Jun 5, 2013, at 2:59, Jeremy Chadwick wrote: > On Tue, Jun 04, 2013 at 11:32:48PM +0200, Alban Hertroys wrote: >> I can't seem to get my serial terminal to work with my new system. >>=20 >> I had a serial terminal connected to my old system that worked great >> and copied /boot.config, /boot/loader.conf and /etc/ttys settings = over >> from it. >>=20 >> The new system has a Gigabyte GA970A-UD3 board with just a serial >> header on the board. I bought a serial connector backplate in an >> electronics store and connected it to the board. Could the pinout be >> different or something? >=20 > It is very possible. You should have asked Gigabyte what exact = product > (specifically part number) to purchase that provided a > header-to-backplane DB9 port, or if they could send you one (many will > for free). Always use what the mainboard vendor tells you. Always. I contacted Gigabyte, but haven't heard from them yet. For the Dutch readers: their website is at gigabyte.co.nl. You = absolutely don't want to go to gigabyte.nl - not safe for work, not at = all (guess where I was=85). > It's also very possible the cable you're using to connect from the > Gigabyte board to something (you didn't disclose what) is wired wrong. I doubt that. It has worked fine before in the same setup with the = previous PC and only stopped working once I connected it to the new PC. = The new PC is the only variable, I expect that's where the problem is. The other end of the serial cable is connected to a Bull serial terminal = (TWSH004/A). The cable has been that same serial cable for the last 10 = years or so. >>> cat /boot.config=20 >> -D -S19200 >=20 > Please try using "-S19200 -Dh" (please read carefully). The order of > the arguments may matter as well (there has been some speculation on = the > lists about this, and I do not care to do the analysis; just passing > information on blindly). That's interesting, I didn't expect that. I'll test that when I get = around to playing with the serial port again. That said, these settings worked in the old PC. I copied boot.config = straight over from my backups. >>> cat /boot/loader.conf=20 >> boot_multicons=3D"YES" >> boot_serial=3D"YES" >> comconsole_speed=3D"19200" >> console=3D"comconsole,vidconsole" >=20 > You do not need any of this given what /boot.config contains. Please > remove it all. Good to know, I never liked having this information in duplicate. >>> =46rom /var/log/messages: >> Jun 4 21:28:50 solfertje kernel: uart0: <16550 or compatible> port = 0x3f8-0x3ff irq=20 >> 4 flags 0x10 on acpi0 >> Jun 4 21:28:50 solfertje kernel: uart0: console (19200,n,8,1) >> Jun 4 21:28:50 solfertje kernel: orm0: at iomem = 0xd5000-0xd67ff,0xd7000-0xd7fff on isa0 >> Jun 4 21:28:50 solfertje kernel: sc0: at flags = 0x100 on isa0 >> Jun 4 21:28:50 solfertje kernel: sc0: VGA <16 virtual consoles, = flags=3D0x300> >> Jun 4 21:28:50 solfertje kernel: vga0: at port = 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 >> Jun 4 21:28:50 solfertje kernel: atkbdc0: at port 0x60,0x64 on isa0 >> Jun 4 21:28:50 solfertje kernel: atkbd0: irq 1 on = atkbdc0 >> Jun 4 21:28:50 solfertje kernel: kbd0 at atkbd0 >> Jun 4 21:28:50 solfertje kernel: atkbd0: [GIANT-LOCKED] >=20 > Okay, so what's the problem? You're not seeing any I/O on the serial > port? I get a blank screen with just a cursor. It's not blinking because = blinking cursors get on my nerves. Whether that means no I/O at all or the wrong kind of I/O I can't tell. = I haven't set up a terminal like this in 10 years as the previous setup = always just worked once set up. It's quite possible that I forgot some = important detail. > How do you have your serial device connected to something on the > other end? Meaning: solfertje has a serial port, and it's connected = to > what exactly? The Bull terminal mentioned above. > Are you sure the device its connected to is working? Well, it was still working great a few days ago before I replaced the PC = it was connected to, so I assume it still does. (snipped useful information that doesn't apply to my case as my cable = has proven to be correct) >=20 >> I didn't see any options in the BIOS to set the console speed (just >> address and IRQ, those are in the above). ISTR that my old mobo did >> allow to set that information, but then again, that board (Tyan = Tiger) >> gave me access to the BIOS through the serial console. >=20 > This has absolutely no relevancy. I thought as much, but it is one obvious difference between the old and = new situation... > Serial port speed settings in a BIOS pertain to BIOS-level console > redirection -- that redirection is lost the instant anything (boot > loader, kernel, etc.) touches SMI and/or interrupts and starts > "fiddling" with the serial port. That's the bit I wasn't entirely certain of - that there is no possible = interaction from having a BIOS console to the point where the OS takes = over. That's why I mentioned it. I assumed that if the BIOS had set up the serial port to 19200 baud and = the OS didn't specify it, that it would be possible that the speed set = up in the BIOS would still be in effect and that the serial terminal = just incidentally worked for the last 10 years because of that. = Far-fetched, I know. That's another theory shot down. > What you're adjusting in FreeBSD is 1) the FreeBSD boot loader = touching > the serial port, and 2) the FreeBSD kernel outputting to a serial port > (it also initialises/sets the serial port), and 3) getty et al = spawning > a login prompt on the serial port. Regarding 2). I found some references on the internet pertaining = settings in the kernel config file to adjust serial console settings. = Could that be what I'm missing? I'm fairly certain the last kernel I used on the old PC was the GENERIC = i386 kernel. I don't think there are any significant differences = regarding serial console between STABLE-i386 (of a few months ago) and = STABLE-amd64? > I would point you to my "FreeBSD via serial console and PXE" document, > except there are one-offs specific to the PXE portions that are not > relevant to your situation. The important part is that I've used > FreeBSD serial console for almost 16 years and have a very good > understanding of what works (including vs. what some developers say > "should" work; i.e. reality vs. pragmatism). True, but there's relatively much interaction between the different = parts that you need to get right before a serial console works at all. = Such as having everything on the same baud-rate, for an obvious example. My guess is that the problem is indeed the connection from the = motherboard header to the bracket on the case. I'll verify the cabling = of that part sometime this weekend. Thanks for the help so far. Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 20:28:29 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 902CCD23 for ; Wed, 5 Jun 2013 20:28:28 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from slow1-d.mail.gandi.net (slow1-d.mail.gandi.net [217.70.178.86]) by mx1.freebsd.org (Postfix) with ESMTP id EECA31F75 for ; Wed, 5 Jun 2013 20:28:27 +0000 (UTC) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by slow1-d.mail.gandi.net (Postfix) with ESMTP id 22B0A47A730 for ; Wed, 5 Jun 2013 22:09:36 +0200 (CEST) Received: from mfilter5-d.gandi.net (mfilter5-d.gandi.net [217.70.178.132]) by relay4-d.mail.gandi.net (Postfix) with ESMTP id 30A6A1720B4; Wed, 5 Jun 2013 22:09:19 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter5-d.gandi.net Received: from relay4-d.mail.gandi.net ([217.70.183.196]) by mfilter5-d.gandi.net (mfilter5-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id IxP4h7dBa80G; Wed, 5 Jun 2013 22:09:17 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 0DF6617209D; Wed, 5 Jun 2013 22:09:16 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id 48C6173A1C; Wed, 5 Jun 2013 13:09:14 -0700 (PDT) Date: Wed, 5 Jun 2013 13:09:14 -0700 From: Jeremy Chadwick To: Alban Hertroys Subject: Re: Serial terminal issues Message-ID: <20130605200914.GA19604@icarus.home.lan> References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> <20130605005918.GA5709@icarus.home.lan> <93196E06-1775-46A4-A3E9-1C8A3B8C4E94@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <93196E06-1775-46A4-A3E9-1C8A3B8C4E94@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 20:28:29 -0000 On Wed, Jun 05, 2013 at 09:29:56PM +0200, Alban Hertroys wrote: > {sniping stuff that is pending or has been acknowledged} > On Jun 5, 2013, at 2:59, Jeremy Chadwick wrote: > > Serial port speed settings in a BIOS pertain to BIOS-level console > > redirection -- that redirection is lost the instant anything (boot > > loader, kernel, etc.) touches SMI and/or interrupts and starts > > "fiddling" with the serial port. > > That's the bit I wasn't entirely certain of - that there is no possible interaction from having a BIOS console to the point where the OS takes over. That's why I mentioned it. > > I assumed that if the BIOS had set up the serial port to 19200 baud and the OS didn't specify it, that it would be possible that the speed set up in the BIOS would still be in effect and that the serial terminal just incidentally worked for the last 10 years because of that. Far-fetched, I know. Not far-fetched. Some system BIOSes with BIOS-level serial console redirection offer what you describe -- on Supermicro systems, you can toggle this capability in the BIOS, it's called "Continue CR After POST" (CR stands for Console Redirection). This is hard to explain without getting into the technicalities, so bear with me here. Get coffee, etc.. What this BIOS feature does is "retain" the SMI/interrupt mapping stuff, so that certain calls to interrupt 0x10 (the BIOS interrupt) for things like cursor movement, writing text/strings, etc. are done on the native console (ex. VGA) *as well* as sent to the serial port (and converted into escape sequences of your choice -- another BIOS option, "Console Type", lets you pick between things like vt100, ANSI, ASCII, etc.). This option is useful for things like option ROMs or HBAs (SCSI/SAS controllers, etc.) which print stuff *after* POST. I'm sure you've seen this. With "Continue CR After POST" disabled, those types of messages are only seen on the VGA console. However, regardless of the setting of "Continue CR After POST", the instant any x86 code starts tinkering with the SMI/interrupt stuff, that functionality is lost (and cannot be restored). In FreeBSD, this definitely happens when the kernel starts, but AFAIR not during the bootstraps. Instead, the bootstraps (that is: boot0, as well as boot2/loader) have the ability to speak to the serial port *directly*, rather than relying on interrupt 0x10. The -S19200 parameter in /boot.config causes the bootstraps **very** early on to set the serial port speed to 19200 baud. This could cause a problem if you have BIOS-level serial redirect set up and set to a different speed (ex. 57600), so naturally you need to make sure everything uses the same speed at all "stages". The -Dh parameter in /boot.config causes the bootstraps **very** early on to tell FreeBSD to write data to the VGA console/text console, in addition to the serial port (directly, not via interrupt 0x10). For how all that works (meaning how -D vs. -Dh behaves and at what ""stages"" of the FreeBSD boot process), please see the FreeBSD Handbook section 27.6.4.1: http://www.freebsd.org/doc/en/books/handbook/serialconsole-setup.html The handbook here is also outdated/wrong; it's talking about sio0 when it means to refer to uart0. flags for uart0 in this case will be 0x00010 (meaning uart0 is a potential serial console). Finally, the important/key part: the -Dh capability when used in /boot.config gets ""passed on"" to boot2/loader (so it knows to output data to the serial port as a console), and boot2/loader **ALSO** passes that information on to the kernel when it starts so that it knows to print data to the serial port too. Make sense? :-) This is why I advocate using /boot.config (or you can use /boot/config if you wish -- both in 9.1-RELEASE work (thanks des@ !)) rather than mucking about with /boot/loader.conf -- the added advantage is that you can actually get serial output at an earlier phase/stage, in case some of your boot blocks don't work. More specifically, with /boot.config you can actually get this on the serial port (if you bang on Escape or Enter repeatedly VERY early on in the boot process): >> FreeBSD/i386 BOOT Default: 0:ad(0,a)/boot/loader boot: But if you don't bang on keys, you won't ever see this. Anyway, sorry for the long ramble there, but the above is how it works. (I'm sure readers will go "My god, that is one of the best write-ups I've seen of how the serial console/boot process stuff works, why isn't this in the handbook!?" to which I will opt out/not respond to). > > What you're adjusting in FreeBSD is 1) the FreeBSD boot loader touching > > the serial port, and 2) the FreeBSD kernel outputting to a serial port > > (it also initialises/sets the serial port), and 3) getty et al spawning > > a login prompt on the serial port. > > Regarding 2). I found some references on the internet pertaining settings in the kernel config file to adjust serial console settings. Could that be what I'm missing? You're referring to either BOOT_COMCONSOLE_SPEED in /etc/make.conf or "options CONSPEED" in the kernel config. You do not need either of these **if** you use /boot.config with the -Sxxx flag. That has been the case since the late FreeBSD 7.x releases (some bugs were fixed in the bootloader code to help improve serial initialisation). If you want to try the "classic method" of using BOOT_COMCONSOLE_SPEED and "options CONSPEED" in your kernel config, you can read section 27.6.5.1 in the handbook: http://www.freebsd.org/doc/en/books/handbook/serialconsole-setup.html BOOT_COMCONSOLE_SPEED will require you to rewrite your boot blocks on your boot disk using gpart for the change to take effect. The CONSPEED parameter in your kernel config naturally just requires a kernel rebuild/reinstall. > I'm fairly certain the last kernel I used on the old PC was the GENERIC i386 kernel. I don't think there are any significant differences regarding serial console between STABLE-i386 (of a few months ago) and STABLE-amd64? No, there is no difference in this regard. You will experience the same problem on i386 as amd64. The only "known" problem with uart(4) has to do with ACPI -- SOMETIMES. Some people -- and they are in the minority -- have found that the uart(4) binding to acpi(4) causes problems (almost certainly with some ACPI tables, i.e. the motherboard/BIOS vendor is stupid/has bugs), and that by forcing uart to tie to the classic ISA (or is it PCI? I would need to find the thread where this was discussed) bus, suddenly uart0 begins working correctly for them. So, you can try booting with ACPI disabled as a test, but that might not be sufficient. Again, I can't remember and would need to dig up that thread. > > I would point you to my "FreeBSD via serial console and PXE" document, > > except there are one-offs specific to the PXE portions that are not > > relevant to your situation. The important part is that I've used > > FreeBSD serial console for almost 16 years and have a very good > > understanding of what works (including vs. what some developers say > > "should" work; i.e. reality vs. pragmatism). > > True, but there's relatively much interaction between the different parts that you need to get right before a serial console works at all. Such as having everything on the same baud-rate, for an obvious example. > > My guess is that the problem is indeed the connection from the motherboard header to the bracket on the case. I'll verify the cabling of that part sometime this weekend. There are a myriad of possibilities. This kind of thing I can diagnose within an hour or two in person, but will take me days or weeks to figure out via Email. Lots of PC things are like this, I'm sorry to say. In the meantime, one recommendation: please try to find a wiring diagram for the cable you're using (DB9 <--> Bull TWSH004/A terminal). I can't stress this enough. It may be that your cable doesn't actually tie DCD high (or is it low?) correctly and thus the uart(4) layer never thinks there's carrier detect. I do not generally recommend futzing about with stty against /dev/ttyu0 while getty is in operation. I have tried this on many occasions and either broke/wedged things or caused anomalous behaviour. If you think DCD might be a problem, try using "3wire.19200" instead of "std.19200" in /etc/ttys and run "init q" (to reload the config). If that works, then your cable is wired "undesirably", and will also lack flow control. See /etc/gettytab's comments about the 3wire entries for what they actually do/represent/how they differ from the std ones. Let us know what transpires. :-) -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 22:00:44 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 56E2494E for ; Wed, 5 Jun 2013 22:00:44 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id DC540168A for ; Wed, 5 Jun 2013 22:00:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r55M0app037539 for ; Thu, 6 Jun 2013 02:00:36 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Thu, 6 Jun 2013 02:00:36 +0400 (MSK) From: Dmitry Morozovsky To: freebsd-stable@FreeBSD.org Subject: TRIM support through ciss Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Thu, 06 Jun 2013 02:00:36 +0400 (MSK) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 22:00:44 -0000 Dear colleagues, I have a DB server with ciss and a bunch of disks (8 SAS + 2 Intel SATA SSD). However, this setup does not seem to support TRIM on SSDs: kstat.zfs.misc.zio_trim.bytes: 0 kstat.zfs.misc.zio_trim.success: 0 kstat.zfs.misc.zio_trim.unsupported: 418 kstat.zfs.misc.zio_trim.failed: 0 Excerpt from dmesg about SSD: da9 at ciss0 bus 0 scbus0 target 9 lun 0 da9: Fixed Direct Access SCSI-5 device da9: Serial Number PACCR9SZ7KJS da9: 135.168MB/s transfers da9: Command Queueing enabled da9: 114439MB (234371520 512 byte sectors: 255H 32S/T 28722C) da9: quirks=0x1 da9: Delete methods: the last line bothers me... Is there any tuning I missed? Thanks! -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 22:13:37 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 11999ECB for ; Wed, 5 Jun 2013 22:13:37 +0000 (UTC) (envelope-from prvs=1868436c14=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id AADC41727 for ; Wed, 5 Jun 2013 22:13:36 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004176069.msg for ; Wed, 05 Jun 2013 23:13:35 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 05 Jun 2013 23:13:35 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1868436c14=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@FreeBSD.org Message-ID: <22BEEC35F8964CACB8E625DA3068A12F@multiplay.co.uk> From: "Steven Hartland" To: "Dmitry Morozovsky" , References: Subject: Re: TRIM support through ciss Date: Wed, 5 Jun 2013 23:13:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 22:13:37 -0000 ----- Original Message ----- From: "Dmitry Morozovsky" To: Sent: Wednesday, June 05, 2013 11:00 PM Subject: TRIM support through ciss > Dear colleagues, > > I have a DB server with ciss and a bunch of disks (8 SAS + 2 Intel SATA SSD). > > However, this setup does not seem to support TRIM on SSDs: > > kstat.zfs.misc.zio_trim.bytes: 0 > kstat.zfs.misc.zio_trim.success: 0 > kstat.zfs.misc.zio_trim.unsupported: 418 > kstat.zfs.misc.zio_trim.failed: 0 > > > Excerpt from dmesg about SSD: > > da9 at ciss0 bus 0 scbus0 target 9 lun 0 > da9: Fixed Direct Access SCSI-5 device > da9: Serial Number PACCR9SZ7KJS > da9: 135.168MB/s transfers > da9: Command Queueing enabled > da9: 114439MB (234371520 512 byte sectors: 255H 32S/T 28722C) > da9: quirks=0x1 > da9: Delete methods: > > the last line bothers me... > > Is there any tuning I missed? What your seeing there is that the RAID array doesn't support any DELETE methods. For ZFS you really don't want to use a HW RAID, its much better to just give ZFS the raw disks as you gain nice features like self healing :) I've never used ciss, but if you can configure the disks as JBOD it may mean it can then support a delete method. In addition for TRIM to work all data disks in a pool need to support a delete method. L2ARC disks are treated seperately. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 22:16:52 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BD11B1AF for ; Wed, 5 Jun 2013 22:16:52 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by mx1.freebsd.org (Postfix) with ESMTP id 7F2281754 for ; Wed, 5 Jun 2013 22:16:52 +0000 (UTC) Received: from mfilter17-d.gandi.net (mfilter17-d.gandi.net [217.70.178.145]) by relay4-d.mail.gandi.net (Postfix) with ESMTP id CADFE172091; Thu, 6 Jun 2013 00:16:40 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter17-d.gandi.net Received: from relay4-d.mail.gandi.net ([217.70.183.196]) by mfilter17-d.gandi.net (mfilter17-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id 8Wwq-hg361xt; Thu, 6 Jun 2013 00:16:39 +0200 (CEST) X-Originating-IP: 67.180.84.87 Received: from jdc.koitsu.org (c-67-180-84-87.hsd1.ca.comcast.net [67.180.84.87]) (Authenticated sender: jdc@koitsu.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id CB796172089; Thu, 6 Jun 2013 00:16:38 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id ECF0973A1C; Wed, 5 Jun 2013 15:16:36 -0700 (PDT) Date: Wed, 5 Jun 2013 15:16:36 -0700 From: Jeremy Chadwick To: Dmitry Morozovsky Subject: Re: TRIM support through ciss Message-ID: <20130605221636.GA21946@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 22:16:52 -0000 On Thu, Jun 06, 2013 at 02:00:36AM +0400, Dmitry Morozovsky wrote: > Dear colleagues, > > I have a DB server with ciss and a bunch of disks (8 SAS + 2 Intel SATA SSD). > > However, this setup does not seem to support TRIM on SSDs: > > kstat.zfs.misc.zio_trim.bytes: 0 > kstat.zfs.misc.zio_trim.success: 0 > kstat.zfs.misc.zio_trim.unsupported: 418 > kstat.zfs.misc.zio_trim.failed: 0 > > > Excerpt from dmesg about SSD: > > da9 at ciss0 bus 0 scbus0 target 9 lun 0 > da9: Fixed Direct Access SCSI-5 device > da9: Serial Number PACCR9SZ7KJS > da9: 135.168MB/s transfers > da9: Command Queueing enabled > da9: 114439MB (234371520 512 byte sectors: 255H 32S/T 28722C) > da9: quirks=0x1 > da9: Delete methods: > > the last line bothers me... > > Is there any tuning I missed? I'm sure Steve will respond, but in the meantime... I assume this is you running stable/9 with r251419 or newer (which just got committed a few hours ago)? I haven't looked at the code, but it is very, VERY important to remember that you are *always* at the whim of 1) the controller driver (ciss(4) in this case), and 2) the controller firmware, as to whether or not certain pass-through commands are supported (in this case, since you have a SAS controller, this would be accomplished via a SCSI command that your controller does not support. Oh, it looks like Steve just replied and said more or less what I did. :-) Bottom line as "we" (the royal we, I guess) have been saying for many years now: any controller which operates in a RAID fashion and does not support "true JBOD" (meaning the controller acts a generic controller with no concept of RAID), will almost always get in the way. Instead, stick with true non-RAID controllers -- and yes I am aware choices are limited. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 22:57:07 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 54E84E7 for ; Wed, 5 Jun 2013 22:57:07 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id BF68118AF for ; Wed, 5 Jun 2013 22:57:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r55Mv5fe038845; Thu, 6 Jun 2013 02:57:05 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Thu, 6 Jun 2013 02:57:05 +0400 (MSK) From: Dmitry Morozovsky To: Steven Hartland Subject: Re: TRIM support through ciss In-Reply-To: <22BEEC35F8964CACB8E625DA3068A12F@multiplay.co.uk> Message-ID: References: <22BEEC35F8964CACB8E625DA3068A12F@multiplay.co.uk> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Thu, 06 Jun 2013 02:57:05 +0400 (MSK) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 22:57:07 -0000 On Wed, 5 Jun 2013, Steven Hartland wrote: > For ZFS you really don't want to use a HW RAID, its much better > to just give ZFS the raw disks as you gain nice features like > self healing :) I know this well, and use it for a dozen servers with regular SATA ("enterprrize-grade" though, like WD RE4s) extensively; however, ciss does not provide an easy way to expose disks -- only via RAID0 per disk > I've never used ciss, but if you can configure the disks as > JBOD it may mean it can then support a delete method. > > In addition for TRIM to work all data disks in a pool need to > support a delete method. L2ARC disks are treated seperately. well, to be a bit more precise: SSDs are split to have mirrored ZIL and splitted L2ARC, and SASes are mirrored in pairs: root@briareus:/usr/src# zpool status pool: br state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Fri Apr 26 15:29:32 2013 config: NAME STATE READ WRITE CKSUM br ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/br0 ONLINE 0 0 0 gpt/br4 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gpt/br1 ONLINE 0 0 0 gpt/br5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 gpt/br6 ONLINE 0 0 0 gpt/br2 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 gpt/br3 ONLINE 0 0 0 gpt/br7 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 gpt/br-zil0 ONLINE 0 0 0 gpt/br-zil1 ONLINE 0 0 0 cache gpt/br-cache0 ONLINE 0 0 0 gpt/br-cache1 ONLINE 0 0 0 errors: No known data errors All I need is TRIM support for ZIL and L2ARC... -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 23:01:26 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 730AF20F for ; Wed, 5 Jun 2013 23:01:26 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 0420118D9 for ; Wed, 5 Jun 2013 23:01:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r55N1OKF039128; Thu, 6 Jun 2013 03:01:24 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Thu, 6 Jun 2013 03:01:24 +0400 (MSK) From: Dmitry Morozovsky To: Jeremy Chadwick Subject: Re: TRIM support through ciss In-Reply-To: <20130605221636.GA21946@icarus.home.lan> Message-ID: References: <20130605221636.GA21946@icarus.home.lan> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Thu, 06 Jun 2013 03:01:24 +0400 (MSK) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 23:01:26 -0000 On Wed, 5 Jun 2013, Jeremy Chadwick wrote: [snip a bit] > I assume this is you running stable/9 with r251419 or newer (which just > got committed a few hours ago)? Yes, sure, I missed to mention it explicitely. > I haven't looked at the code, but it is very, VERY important to remember > that you are *always* at the whim of 1) the controller driver (ciss(4) > in this case), and 2) the controller firmware, as to whether or not > certain pass-through commands are supported (in this case, since you > have a SAS controller, this would be accomplished via a SCSI command > that your controller does not support. I tried to chase versions, and it seems I use for this Proliant the last set. > Oh, it looks like Steve just replied and said more or less what I did. > :-) > > Bottom line as "we" (the royal we, I guess) have been saying for many > years now: any controller which operates in a RAID fashion and does not > support "true JBOD" (meaning the controller acts a generic controller > with no concept of RAID), will almost always get in the way. Instead, > stick with true non-RAID controllers -- and yes I am aware choices are > limited. well, the server in question "has been fallen from the heaven", ya know ;-) if I were the builder, I suppose I'd use SuperMicro platform + non-RAID LSI HBA... on the other hand, battery-backed cache... not an easy choice, yes. -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Wed Jun 5 23:55:15 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 48DD6C68 for ; Wed, 5 Jun 2013 23:55:15 +0000 (UTC) (envelope-from prvs=1868436c14=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 0FE811A72 for ; Wed, 5 Jun 2013 23:55:13 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004178621.msg for ; Thu, 06 Jun 2013 00:55:08 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 06 Jun 2013 00:55:08 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1868436c14=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: From: "Steven Hartland" To: "Dmitry Morozovsky" References: <22BEEC35F8964CACB8E625DA3068A12F@multiplay.co.uk> Subject: Re: TRIM support through ciss Date: Thu, 6 Jun 2013 00:54:58 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 23:55:15 -0000 ----- Original Message ----- From: "Dmitry Morozovsky" > well, to be a bit more precise: SSDs are split to have mirrored ZIL and > splitted L2ARC, and SASes are mirrored in pairs: > > root@briareus:/usr/src# zpool status > pool: br > state: ONLINE > scan: scrub repaired 0 in 0h0m with 0 errors on Fri Apr 26 15:29:32 2013 > config: > > NAME STATE READ WRITE CKSUM > br ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/br0 ONLINE 0 0 0 ... > logs > mirror-4 ONLINE 0 0 0 > gpt/br-zil0 ONLINE 0 0 0 > gpt/br-zil1 ONLINE 0 0 0 > cache > gpt/br-cache0 ONLINE 0 0 0 > gpt/br-cache1 ONLINE 0 0 0 > > errors: No known data errors > > All I need is TRIM support for ZIL and L2ARC... The important question for this is do your see a supported delete method for your SSD's? If the answer is no then you may be out of luck unless there's a FW upgrade for your controller which fixes that or you can attach them to a different controller such as on board AHCI? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Thu Jun 6 00:05:37 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 357E0EDD for ; Thu, 6 Jun 2013 00:05:37 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id A8A251AC4 for ; Thu, 6 Jun 2013 00:05:36 +0000 (UTC) Received: from ur.dons.net.au (ppp118-210-215-193.lns20.adl6.internode.on.net [118.210.215.193]) (authenticated bits=0) by cain.gsoft.com.au (8.14.4/8.14.3) with ESMTP id r5604Ixa042886 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 6 Jun 2013 09:34:24 +0930 (CST) (envelope-from doconnor@gsoft.com.au) Subject: Re: Serial terminal issues Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Content-Type: text/plain; charset=windows-1252 From: "Daniel O'Connor" In-Reply-To: <93196E06-1775-46A4-A3E9-1C8A3B8C4E94@gmail.com> Date: Thu, 6 Jun 2013 09:34:17 +0930 Content-Transfer-Encoding: quoted-printable Message-Id: References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> <20130605005918.GA5709@icarus.home.lan> <93196E06-1775-46A4-A3E9-1C8A3B8C4E94@gmail.com> To: Alban Hertroys X-Mailer: Apple Mail (2.1503) X-Spam-Score: 0.163 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.67 on 203.31.81.10 Cc: Jeremy Chadwick , "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 00:05:37 -0000 On 06/06/2013, at 4:59, Alban Hertroys wrote: >> It is very possible. You should have asked Gigabyte what exact = product >> (specifically part number) to purchase that provided a >> header-to-backplane DB9 port, or if they could send you one (many = will >> for free). Always use what the mainboard vendor tells you. Always. >=20 > I contacted Gigabyte, but haven't heard from them yet. >=20 > For the Dutch readers: their website is at gigabyte.co.nl. You = absolutely don't want to go to gigabyte.nl - not safe for work, not at = all (guess where I was=85). We make our serial cables at work, it is fairly straightforward. You can = see in the manual which end of the motherboard connector is pin 1. We have ribbon cable and insulation displacement D9 & 2x5 headers on = hand though.. If you aren't certain the serial hardware works I suggest building or = buying a serial loopback connector and plugging it in. Then run.. cu -l /dev/cuXXX -s 9600 and typing and see if you get your typing echoed back. If you do then you know the serial hardware & driver are working. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Thu Jun 6 12:03:40 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DF360CD5 for ; Thu, 6 Jun 2013 12:03:40 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id 852071C8C for ; Thu, 6 Jun 2013 12:03:40 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1UkYgC-0003Pi-4E for freebsd-stable@freebsd.org; Thu, 06 Jun 2013 13:48:32 +0200 Received: from august.inf.tu-dresden.de ([141.76.48.124]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 06 Jun 2013 13:48:32 +0200 Received: from jsteckli by august.inf.tu-dresden.de with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 06 Jun 2013 13:48:32 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Julian Stecklina Subject: Reproducable Infiniband panic Date: Thu, 06 Jun 2013 13:48:21 +0200 Lines: 195 Message-ID: <51B07705.207@os.inf.tu-dresden.de> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: august.inf.tu-dresden.de User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 12:03:40 -0000 Hello, I see a reproducable panic when doing ibping and aborting it with ^C. My setup is two machines with Mellanox Infinihost III HCAs (one Linux one FreeBSD) connected back-to-back. Details below. I can upload 2 crash dumps, if this is useful. For some reason the port doesn't become ACTIVE, so no packets arrive, but that is probably unrelated. % uname -a FreeBSD cosel.inf.tu-dresden.de 9.1-STABLE FreeBSD 9.1-STABLE #0 r+b6547e3: Wed Jun 5 18:29:51 CEST 2013 julian@cosel.inf.tu-dresden.de:/usr/obj/usr/home/julian/src/freebsd/sys/COSEL amd64 % sudo ibping 2 ^C --- (Lid 2) ibping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5161 ms rtt min/avg/max = 0.000/0.000/0.000 ms Fatal trap 12: page fault while in kernel mode cpuid = 6; apic id = 06 fault virtual address = 0x18 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff807a3d83 stack pointer = 0x28:0xffffff8092c97890 frame pointer = 0x28:0xffffff8092c978b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1489 (ibping) trap number = 12 panic: page fault cpuid = 6 KDB: stack backtrace: #0 0xffffffff80632a96 at kdb_backtrace+0x66 #1 0xffffffff805f9fce at panic+0x1ce #2 0xffffffff808a7380 at trap_fatal+0x290 #3 0xffffffff808a76e1 at trap_pfault+0x211 #4 0xffffffff808a7c94 at trap+0x344 #5 0xffffffff80891043 at calltrap+0x8 #6 0xffffffff80513c39 at devfs_destroy_cdevpriv+0x69 #7 0xffffffff80513e47 at devfs_close_f+0x57 #8 0xffffffff805b4b23 at _fdrop+0x23 #9 0xffffffff805b65ec at closef+0x4c #10 0xffffffff805b76cc at fdfree+0x23c #11 0xffffffff805c4945 at exit1+0x305 #12 0xffffffff805c5d0e at sys_sys_exit+0xe #13 0xffffffff808a6b56 at amd64_syscall+0x5d6 #14 0xffffffff80891327 at Xfast_syscall+0xf7 Full backtrace from kgdb: #0 doadump (textdump=) at pcpu.h:234 No locals. #1 0xffffffff805f9aa4 in kern_reboot (howto=260) at /usr/home/julian/src/freebsd/sys/kern/kern_shutdown.c:449 _ep = (struct eventhandler_entry *) 0x0 _el = first_buf_printf = 1 #2 0xffffffff805f9fa7 in panic (fmt=0x1
) at /usr/home/julian/src/freebsd/sys/kern/kern_shutdown.c:637 td = (struct thread *) 0x1 bootopt = newpanic = ap = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0xffffff8092b934f0, reg_save_area = 0xffffff8092b93410}} panic_cpu = 7 buf = "page fault", '\0' #3 0xffffffff808a7380 in trap_fatal (frame=0xc, eva=) at /usr/home/julian/src/freebsd/sys/amd64/amd64/trap.c:878 code = ss = 40 type = 12 esp = softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27, ssd_dpl = 0, ssd_p = 1, ssd_long = 1, ssd_def32 = 0, ssd_gran = 1} msg = #4 0xffffffff808a76e1 in trap_pfault (frame=0xffffff8092b937e0, usermode=0) at /usr/home/julian/src/freebsd/sys/amd64/amd64/trap.c:794 id = va = 0 vm = map = 0xfffffe000b0a3498 rv = 0 ftype = 255 'ÿ' td = (struct thread *) 0xfffffe000b0af000 p = (struct proc *) 0xfffffe000b181950 eva = 24 #5 0xffffffff808a7c94 in trap (frame=0xffffff8092b937e0) at /usr/home/julian/src/freebsd/sys/amd64/amd64/trap.c:463 regs = {r_r15 = -2136840320, r_r14 = -547294202144, r_r13 = -547294202208, r_r12 = -2140660471, r_r11 = -2136840320, r_r10 = 594, r_r9 = -547294202160, r_r8 = -2198830683992, r_rdi = 0, r_rsi = -2136780531, r_rbp = 219043332096, r_rbx = -2198837989376, r_rdx = -547294202048, r_rcx = 2154444695, r_rax = -2133265824, r_trapno = 192571360, r_fs = 65024, r_gs = 65535, r_err = 525312, r_es = 65408, r_ds = 65535, r_rip = -2136840320, r_cs = -547294202064, r_rflags = -2133515200, r_rsp = -547294201968, r_ss = 0} td = (struct thread *) 0xfffffe000b0af000 p = i = ucode = code = 0 type = 12 addr = ksi = {ksi_link = {tqe_next = 0xfffffe000553ac00, tqe_prev = 0xfffffe000b0af000}, ksi_info = {si_signo = -1833355440, si_errno = -128, si_code = -2140661293, si_pid = -1, si_uid = 0, si_status = 0, si_addr = 0xfffffe0000000000, si_value = {sival_int = -1833355392, sival_ptr = 0xffffff8092b93780, sigval_int = -1833355392, sigval_ptr = 0xffffff8092b93780}, _reason = {_fault = {_trapno = -2138032854}, _timer = {_timerid = -2138032854, _overrun = -1}, _mesgq = {_mqd = -2138032854}, _poll = {_band = -2138032854}, __spare__ = { __spare1__ = -2138032854, __spare2__ = {192571360, -512, 192571360, -512, 2, 0, 0}}}}, ksi_flags = -1833355296, ksi_sigq = 0xffffffff806955f3} #6 0xffffffff80891043 in calltrap () at /usr/home/julian/src/freebsd/sys/amd64/amd64/exception.S:228 No locals. #7 0xffffffff807a3d83 in linux_file_dtor (cdp=0xfffffe000aeabb80) at /usr/home/julian/src/freebsd/sys/ofed/include/linux/linux_compat.c:214 filp = (struct linux_file *) 0xfffffe000aeabb80 #8 0xffffffff80513c39 in devfs_destroy_cdevpriv (p=0xfffffe0005772980) at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:159 No locals. #9 0xffffffff80513e47 in devfs_close_f (fp=0xfffffe000b0e9aa0, td=) at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:619 error = 0 fpop = (struct file *) 0x0 #10 0xffffffff805b4b23 in _fdrop (fp=0xfffffe000b0e9aa0, td=) at file.h:334 error = 0 #11 0xffffffff805b65ec in closef (fp=0xfffffe000b0e9aa0, td=0xfffffe000b0af000) at /usr/home/julian/src/freebsd/sys/kern/kern_descrip.c:2272 vp = lf = {l_start = 0, l_len = -2198837126832, l_pid = 4, l_type = 0, l_whence = 0, l_sysid = 185266176} fdtol = (struct filedesc_to_leader *) 0x0 fdp = fp_object = (struct file *) 0xfffffe000b0e9aa0 #12 0xffffffff805b76cc in fdfree (td=0xfffffe000b0af000) at /usr/home/julian/src/freebsd/sys/kern/kern_descrip.c:1976 fdp = (struct filedesc *) 0xfffffe000b1a8000 fpp = (struct file **) 0xfffffe000b1a8098 i = 3 fdtol = fp = (struct file *) 0xfffffe000b0e9aa0 cdir = jdir = rdir = vp = lf = {l_start = -547294201248, l_len = -2140995476, l_pid = 4, l_type = 0, l_whence = 0, l_sysid = 0} #13 0xffffffff805c4945 in exit1 (td=0xfffffe000b0af000, rv=0) at /usr/home/julian/src/freebsd/sys/kern/kern_exit.c:301 id = p = (struct proc *) 0xfffffe000b181950 nq = q = (struct proc *) 0x4 vtmp = ttyvp = plim = reason = #14 0xffffffff805c5d0e in sys_sys_exit (td=, uap=) at /usr/home/julian/src/freebsd/sys/kern/kern_exit.c:122 No locals. #15 0xffffffff808a6b56 in amd64_syscall (td=0xfffffe000b0af000, traced=0) at subr_syscall.c:135 sa = {code = 1, callp = 0xffffffff80d31330, args = {0, 0, 10, 0, 0, 0, 133124, -547294200768}, narg = 1} error = 0 ksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 2, si_errno = 0, si_code = 65542, si_pid = 0, si_uid = 0, si_status = 0, si_addr = 0x0, si_value = {sival_int = 0, sival_ptr = 0x0, sigval_int = 0, sigval_ptr = 0x0}, _reason = {_fault = { _trapno = 0}, _timer = {_timerid = 0, _overrun = 0}, _mesgq = {_mqd = 0}, _poll = {_band = 0}, __spare__ = {__spare1__ = 0, __spare2__ = {0, 0, 0, 0, 0, 0, 0}}}}, ksi_flags = 0, ksi_sigq = 0x0} #16 0xffffffff80891327 in Xfast_syscall () at /usr/home/julian/src/freebsd/sys/amd64/amd64/exception.S:387 No locals. #17 0x0000000800eda82c in ?? () Julian From owner-freebsd-stable@FreeBSD.ORG Thu Jun 6 13:21:43 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 63903405 for ; Thu, 6 Jun 2013 13:21:43 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-05.arcor-online.net (mail-in-05.arcor-online.net [151.189.21.45]) by mx1.freebsd.org (Postfix) with ESMTP id 2130B10FE for ; Thu, 6 Jun 2013 13:21:42 +0000 (UTC) Received: from mail-in-17-z2.arcor-online.net (mail-in-17-z2.arcor-online.net [151.189.8.34]) by mx.arcor.de (Postfix) with ESMTP id 9F0D195D8B for ; Thu, 6 Jun 2013 15:21:35 +0200 (CEST) Received: from mail-in-15.arcor-online.net (mail-in-15.arcor-online.net [151.189.21.55]) by mail-in-17-z2.arcor-online.net (Postfix) with ESMTP id 956BA367F46 for ; Thu, 6 Jun 2013 15:21:35 +0200 (CEST) X-Greylist: Passed host: 188.105.84.42 X-DKIM: Sendmail DKIM Filter v2.8.2 mail-in-15.arcor-online.net 723611AB53B Received: from lorvorc.mips.inka.de (dslb-188-105-084-042.pools.arcor-ip.net [188.105.84.42]) by mail-in-15.arcor-online.net (Postfix) with ESMTPS id 723611AB53B for ; Thu, 6 Jun 2013 15:21:35 +0200 (CEST) Received: from lorvorc.mips.inka.de (localhost [127.0.0.1]) by lorvorc.mips.inka.de (8.14.7/8.14.7) with ESMTP id r56DLYmW045065 for ; Thu, 6 Jun 2013 15:21:34 +0200 (CEST) (envelope-from mailnull@lorvorc.mips.inka.de) Received: (from mailnull@localhost) by lorvorc.mips.inka.de (8.14.7/8.14.7/Submit) id r56DLYba045061 for freebsd-stable@freebsd.org; Thu, 6 Jun 2013 15:21:34 +0200 (CEST) (envelope-from mailnull) From: naddy@mips.inka.de (Christian Weisgerber) Subject: Re: Serial terminal issues Date: Thu, 6 Jun 2013 13:21:34 +0000 (UTC) Message-ID: References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> Originator: naddy@mips.inka.de (Christian Weisgerber) To: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 13:21:43 -0000 Alban Hertroys wrote: > The new system has a Gigabyte GA970A-UD3 board with just a serial header > on the board. I bought a serial connector backplate in an electronics > store and connected it to the board. Could the pinout be different or > something? Yes. There are two different pinouts for motherboard serial headers. -- Christian "naddy" Weisgerber naddy@mips.inka.de From owner-freebsd-stable@FreeBSD.ORG Thu Jun 6 13:40:19 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3E7BBE36 for ; Thu, 6 Jun 2013 13:40:19 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id B0BA91226 for ; Thu, 6 Jun 2013 13:40:18 +0000 (UTC) Received: from ur.dons.net.au (ppp118-210-215-193.lns20.adl6.internode.on.net [118.210.215.193]) (authenticated bits=0) by cain.gsoft.com.au (8.14.4/8.14.3) with ESMTP id r56DdxuK095759 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 6 Jun 2013 23:10:05 +0930 (CST) (envelope-from doconnor@gsoft.com.au) Subject: Re: Serial terminal issues Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Content-Type: text/plain; charset=us-ascii From: "Daniel O'Connor" In-Reply-To: Date: Thu, 6 Jun 2013 23:09:59 +0930 Content-Transfer-Encoding: quoted-printable Message-Id: <471CA88F-8F56-44C9-8348-B009091BCB83@gsoft.com.au> References: <383C78B1-4F16-408F-8144-63B470D0C129@gmail.com> To: Christian Weisgerber X-Mailer: Apple Mail (2.1503) X-Spam-Score: 0.163 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.67 on 203.31.81.10 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 13:40:19 -0000 On 06/06/2013, at 22:51, Christian Weisgerber = wrote: > Alban Hertroys wrote: >=20 >> The new system has a Gigabyte GA970A-UD3 board with just a serial = header >> on the board. I bought a serial connector backplate in an electronics >> store and connected it to the board. Could the pinout be different or >> something? >=20 > Yes. There are two different pinouts for motherboard serial headers. You can probably reverse it yourself if you are careful though - open = one end and lift the cable out, flip it over and crimp it back in a vice = then trim the excess cable with a hobby knife. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Thu Jun 6 13:54:45 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D6C34470; Thu, 6 Jun 2013 13:54:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id D0BE012E2; Thu, 6 Jun 2013 13:54:44 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA04628; Thu, 06 Jun 2013 16:54:35 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <51B0949B.1050606@FreeBSD.org> Date: Thu, 06 Jun 2013 16:54:35 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130517 Thunderbird/17.0.6 MIME-Version: 1.0 To: Julian Stecklina Subject: Re: Reproducable Infiniband panic References: <51B07705.207@os.inf.tu-dresden.de> In-Reply-To: <51B07705.207@os.inf.tu-dresden.de> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 13:54:45 -0000 on 06/06/2013 14:48 Julian Stecklina said the following: > #7 0xffffffff807a3d83 in linux_file_dtor (cdp=0xfffffe000aeabb80) at > /usr/home/julian/src/freebsd/sys/ofed/include/linux/linux_compat.c:214 > filp = (struct linux_file *) 0xfffffe000aeabb80 > #8 0xffffffff80513c39 in devfs_destroy_cdevpriv (p=0xfffffe0005772980) > at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:159 > No locals. > #9 0xffffffff80513e47 in devfs_close_f (fp=0xfffffe000b0e9aa0, > td=) > at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:619 > error = 0 > fpop = (struct file *) 0x0 The problem seems to be in incorrect interaction between devfs_close_f and linux_file_dtor. The latter expects curthread->td_fpop to have a valid reasonable value. But the former sets curthread->td_fpop to fp only around vnops.fo_close() call and then restores it back to some (what?) previous value before calling devfs_fpdrop->devfs_destroy_cdevpriv. In this case the previous value is NULL. -- Andriy Gapon From owner-freebsd-stable@FreeBSD.ORG Thu Jun 6 18:59:25 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 61607E59; Thu, 6 Jun 2013 18:59:25 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 3EE5E1233; Thu, 6 Jun 2013 18:59:25 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 11379B918; Thu, 6 Jun 2013 14:59:23 -0400 (EDT) From: John Baldwin To: freebsd-stable@freebsd.org Subject: Re: Reproducable Infiniband panic Date: Thu, 6 Jun 2013 14:57:52 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <51B07705.207@os.inf.tu-dresden.de> <51B0949B.1050606@FreeBSD.org> In-Reply-To: <51B0949B.1050606@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201306061457.52278.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 06 Jun 2013 14:59:23 -0400 (EDT) Cc: Andriy Gapon , Julian Stecklina X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 18:59:25 -0000 On Thursday, June 06, 2013 9:54:35 am Andriy Gapon wrote: > on 06/06/2013 14:48 Julian Stecklina said the following: > > #7 0xffffffff807a3d83 in linux_file_dtor (cdp=0xfffffe000aeabb80) at > > /usr/home/julian/src/freebsd/sys/ofed/include/linux/linux_compat.c:214 > > filp = (struct linux_file *) 0xfffffe000aeabb80 > > #8 0xffffffff80513c39 in devfs_destroy_cdevpriv (p=0xfffffe0005772980) > > at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:159 > > No locals. > > #9 0xffffffff80513e47 in devfs_close_f (fp=0xfffffe000b0e9aa0, > > td=) > > at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:619 > > error = 0 > > fpop = (struct file *) 0x0 > > The problem seems to be in incorrect interaction between devfs_close_f and > linux_file_dtor. The latter expects curthread->td_fpop to have a valid reasonable > value. But the former sets curthread->td_fpop to fp only around vnops.fo_close() > call and then restores it back to some (what?) previous value before calling > devfs_fpdrop->devfs_destroy_cdevpriv. In this case the previous value is NULL. It is normally NULL in this case. Why does linux_file_dtor even look at td_fpop? Ah. I think it should not do that and make the data it uses in the dtor more self-contained: Index: sys/ofed/include/linux/linux_compat.c =================================================================== --- linux_compat.c (revision 251465) +++ linux_compat.c (working copy) @@ -212,7 +212,7 @@ linux_file_dtor(void *cdp) struct linux_file *filp; filp = cdp; - filp->f_op->release(curthread->td_fpop->f_vnode, filp); + filp->f_op->release(filp->f_vnode, filp); kfree(filp); } @@ -232,6 +232,7 @@ linux_dev_open(struct cdev *dev, int oflags, int d filp->f_dentry = &filp->f_dentry_store; filp->f_op = ldev->ops; filp->f_flags = file->f_flag; + filp->f_vnode = file->f_vnode; if (filp->f_op->open) { error = -filp->f_op->open(file->f_vnode, filp); if (error) { -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Fri Jun 7 01:03:25 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1CA8B4A1; Fri, 7 Jun 2013 01:03:25 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pb0-x22f.google.com (mail-pb0-x22f.google.com [IPv6:2607:f8b0:400e:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id E5BFF1F58; Fri, 7 Jun 2013 01:03:24 +0000 (UTC) Received: by mail-pb0-f47.google.com with SMTP id rr13so1331691pbb.34 for ; Thu, 06 Jun 2013 18:03:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=B8tWiao8pt+cA46yVWFnbNDbrO+o3MBQzDfmhBclyqA=; b=QWAIAhHOgk7lRTwEQwuAhnAU9jt9MsGSnXoQ+UIITSxC3tfcWn3BH5qqwHBpKE0Dwm s04wlTBPx0YxSIHATAc32IKqzQR0XF+bkLaDqOtWhgv3ZGiFAYQcI4UGDLkRt4GJTfCB YBjnLM4mxG9UL+JlIrzjP5padlUEII6q9Fc/qGQ96dfIZ3j/Ey4ZnW3mSd58hzNZBHce Y6k7udCYw4687G77wcleSQZfWFYk3ZpE3Bfp2rco8lHSrxUuqFDUlVDJVf5NqtMg/X+4 cW87J5eh/FsGK6v9/xGcy/sP07tJoxEyflWAoX6WVZ/FYbJs9BkNOdj2RiIg9vFQtBDP a1MQ== X-Received: by 10.66.228.34 with SMTP id sf2mr261836pac.134.1370567004437; Thu, 06 Jun 2013 18:03:24 -0700 (PDT) Received: from pyunyh@gmail.com (lpe4.p59-icn.cdngp.net. [114.111.62.249]) by mx.google.com with ESMTPSA id ag4sm74988717pbc.20.2013.06.06.18.03.20 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 06 Jun 2013 18:03:23 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Fri, 07 Jun 2013 10:03:13 +0900 From: YongHyeon PYUN Date: Fri, 7 Jun 2013 10:03:13 +0900 To: Daniel Braniss Subject: Re: SunFire X2200 ilo's bge1 DOWN/UP Message-ID: <20130607010313.GB3347@michelle.cdnetworks.com> References: <20130529085544.GC3042@michelle.cdnetworks.com> <201305300859.20928.jhb@freebsd.org> <20130531054713.GB1478@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org, John Baldwin X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2013 01:03:25 -0000 On Mon, Jun 03, 2013 at 09:25:33AM +0300, Daniel Braniss wrote: > > On Fri, May 31, 2013 at 08:24:47AM +0300, Daniel Braniss wrote: > > > > On Thursday, May 30, 2013 2:44:35 am Daniel Braniss wrote: > > > > > > --/04w6evG8XlLl3ft > > > > > > Content-Type: text/x-diff; charset=us-ascii > > > > > > Content-Disposition: attachment; filename="bge.media_sts.diff" > > > > > > > > > > > > Index: sys/dev/bge/if_bge.c > > > > > > =================================================================== > > > > > > --- sys/dev/bge/if_bge.c (revision 251021) > > > > > > +++ sys/dev/bge/if_bge.c (working copy) > > > > > > @@ -5583,6 +5583,10 @@ bge_ifmedia_sts(struct ifnet *ifp, struct ifmediar > > > > > > > > > > > > BGE_LOCK(sc); > > > > > > > > > > > > + if ((ifp->if_flags & IFF_UP) == 0) { > > > > > > + BGE_UNLOCK(sc); > > > > > > + return; > > > > > > + } > > > > > > if (sc->bge_flags & BGE_FLAG_TBI) { > > > > > > ifmr->ifm_status = IFM_AVALID; > > > > > > ifmr->ifm_active = IFM_ETHER; > > > > > > > > > > > > --/04w6evG8XlLl3ft-- > > > > > after 18hs, the logs are empty! > > > > > it seems the patch fixes the problem. > > > > > > > > > > now maybe it's time to hunt for who is randomly calling for bge_ifmedia_sts > > > > > ... > > > > > > > > It could be any number of daemons that query interface state such as an > > > > SNMP server, ladvd, etc. > > > > > > > > If you wanted help you could modify the patch so that it does something like > > > > this: > > > > > > > #include > > > > if (/* test for IFF_UP */) { > > > > BGE_UNLOCK(sc); > > > > if_printf(ifp, "state queried on down interface by pid %d (%s)", > > > ------------------------------------------------------------------------------| > > > add a \n > > > > curthread->td_proc->p_pid, curthread->td_proc->p_comm); > > > > return; > > > > } > > > > > > > > -- > > > > John Baldwin > > > snmpd call this several times a second, (difficult to measeure since sysolog > > > just says > > > last message repeated 22 times > > > in any case, the DOWN/UP appears once every few hours, oh well. > > > I have now stopped the snmpd daemon, maybe there is someone else ... > > > > I have no idea why snmpd wants to know media status for interfaces > > that are put into down state. The media status resolved after > > bringing up the interface may be different one that was seen > > before. > > The patch also makes dhclient think driver got a valid link > > regardless of link establishment. I guess that wouldn't be > > issue though. I'll commit the patch after some more testing. > > > > Thanks for reporting and testing! > > > no problem! > > after more than 3 days, there were no more 'reports', so snmpd was the culprit. > the snmpd we use is from ports, i'll try and see waht's going on ... > FYI: Committed in r251481. From owner-freebsd-stable@FreeBSD.ORG Fri Jun 7 09:07:35 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BAE7D3CF; Fri, 7 Jun 2013 09:07:35 +0000 (UTC) (envelope-from jsteckli@os.inf.tu-dresden.de) Received: from os.inf.tu-dresden.de (os.inf.tu-dresden.de [IPv6:2002:8d4c:3001:48::99]) by mx1.freebsd.org (Postfix) with ESMTP id 81B991242; Fri, 7 Jun 2013 09:07:35 +0000 (UTC) Received: from [2002:8d4c:3001:48:ea40:f2ff:fee2:6328] by os.inf.tu-dresden.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) id 1Uksdy-0001nT-IZ; Fri, 07 Jun 2013 11:07:34 +0200 Message-ID: <51B1A2D6.4030901@os.inf.tu-dresden.de> Date: Fri, 07 Jun 2013 11:07:34 +0200 From: Julian Stecklina User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6 MIME-Version: 1.0 To: John Baldwin , freebsd-stable@freebsd.org Subject: Re: Reproducable Infiniband panic References: <51B07705.207@os.inf.tu-dresden.de> <51B0949B.1050606@FreeBSD.org> <201306061457.52278.jhb@freebsd.org> In-Reply-To: <201306061457.52278.jhb@freebsd.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2013 09:07:35 -0000 On 06/06/2013 08:57 PM, John Baldwin wrote: > On Thursday, June 06, 2013 9:54:35 am Andriy Gapon wrote: [...] >> The problem seems to be in incorrect interaction between devfs_close_f and >> linux_file_dtor. The latter expects curthread->td_fpop to have a valid reasonable >> value. But the former sets curthread->td_fpop to fp only around vnops.fo_close() >> call and then restores it back to some (what?) previous value before calling >> devfs_fpdrop->devfs_destroy_cdevpriv. In this case the previous value is NULL. > > It is normally NULL in this case. Why does linux_file_dtor even look at > td_fpop? > > Ah. I think it should not do that and make the data it uses in the dtor more > self-contained: > > Index: sys/ofed/include/linux/linux_compat.c > =================================================================== > --- linux_compat.c (revision 251465) > +++ linux_compat.c (working copy) > @@ -212,7 +212,7 @@ linux_file_dtor(void *cdp) > struct linux_file *filp; > > filp = cdp; > - filp->f_op->release(curthread->td_fpop->f_vnode, filp); > + filp->f_op->release(filp->f_vnode, filp); > kfree(filp); > } > > @@ -232,6 +232,7 @@ linux_dev_open(struct cdev *dev, int oflags, int d > filp->f_dentry = &filp->f_dentry_store; > filp->f_op = ldev->ops; > filp->f_flags = file->f_flag; > + filp->f_vnode = file->f_vnode; > if (filp->f_op->open) { > error = -filp->f_op->open(file->f_vnode, filp); > if (error) { > Doesn't compile for me. Did you forget to add the f_vnode member to struct linux_file? sys/ofed/include/linux/linux_compat.c: In function 'linux_file_dtor': sys/ofed/include/linux/linux_compat.c:214: error: 'struct linux_file' has no member named 'f_vnode' sys/ofed/include/linux/linux_compat.c: In function 'linux_dev_open': sys/ofed/include/linux/linux_compat.c:234: error: 'struct linux_file' has no member named 'f_vnode' Julian From owner-freebsd-stable@FreeBSD.ORG Fri Jun 7 12:38:00 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A7C881B0 for ; Fri, 7 Jun 2013 12:38:00 +0000 (UTC) (envelope-from pascal.braun@continum.net) Received: from mailsrv1.continum.net (mr1.continum.net [80.72.129.121]) by mx1.freebsd.org (Postfix) with ESMTP id 6E9601CD0 for ; Fri, 7 Jun 2013 12:37:58 +0000 (UTC) Received: from zimbra.continum.net ([80.72.133.238]) by mr1.continum.net with esmtp (Exim 4.67) (envelope-from ) id 1UkvvX-0005TN-4i; Fri, 07 Jun 2013 14:37:55 +0200 Received: from localhost (localhost [127.0.0.1]) by zimbra.continum.net (Postfix) with ESMTP id 7A9DD1AE046; Fri, 7 Jun 2013 14:37:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at zimbra.continum.net Received: from zimbra.continum.net ([127.0.0.1]) by localhost (zimbra.continum.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hlEAxb01H68l; Fri, 7 Jun 2013 14:37:47 +0200 (CEST) Received: from zimbra.continum.net (zimbra.continum.net [80.72.133.238]) by zimbra.continum.net (Postfix) with ESMTP id 970FA19E02A; Fri, 7 Jun 2013 14:37:47 +0200 (CEST) Date: Fri, 7 Jun 2013 14:37:47 +0200 (CEST) From: "Pascal Braun, Continum" To: Jeremy Chadwick Message-ID: <1290657146.201020.1370608667469.JavaMail.root@continum.net> In-Reply-To: <20130604095300.GA79993@icarus.home.lan> Subject: Re: ZFS crashing while zfs recv in progress MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_201016_445496304.1370608667459" X-Originating-IP: [80.72.130.250] X-Mailer: Zimbra 7.2.0_GA_2669 (ZimbraWebClient - GC23 (Linux)/7.2.0_GA_2669) Cc: freebsd-stable@freebsd.org, Ronald Klop X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2013 12:38:00 -0000 ------=_Part_201016_445496304.1370608667459 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit first I'd like to thank you for your time and effort. > - Disk da3 has a different drive firmware (A580) than the A800 > drives. Somehow I did miss that. I can replace this disk with a A800 one, although I don't think this will change much. > - I have not verified if any of these disks use 4KByte sectors (dmesg > is > not going to tell you the entire truth). I would appreciate seeing > "smartctl -x" output from {da0,da1,da3} so I could get an idea. > Your > pools use gpt labelling so I am left with the hope that your labels > refer to the partition with proper 4KB alignment regardless. The 'tank' disks are real 512bytes disks. The zpool currently in use is ashift=9. I've also tried ashift=12 in the past, but it didn't help. You'll find the output of smartctl in the attachment. > Can you tell me what exact disk (e.g. daXX) in the above list you > used > for swap, and what kind of both system and disk load were going on at > the time you saw the swap message? > > I'm looking for a capture of "gstat -I500ms" output (you will need a > VERY long/big terminal window to capture this given how many disks > you > have) while I/O is happening, as well as "top -s 1" in another > window. > I would also like to see "zpool iostat -v 1" output while things are > going on, to help possibly narrow down if there is a single disk > causing > the entire I/O subsystem for that controller to choke. The swap disk in use is da28. The last output of top -s 1 that could be writen to disk was: --- last pid: 3653; load averages: 0.03, 0.19, 0.30 up 0+15:55:50 03:04:33 43 processes: 1 running, 41 sleeping, 1 zombie CPU: 0.3% user, 0.0% nice, 0.6% system, 0.1% interrupt, 99.0% idle Mem: 7456K Active, 27M Inact, 6767M Wired, 3404K Cache, 9053M Free Swap: 256G Total, 5784K Used, 256G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1917 root 1 22 0 33420K 2356K piperd 2 41:24 3.96% zfs 1913 root 1 21 0 71980K 5248K select 4 288:50 3.27% sshd 1853 root 1 20 0 29484K 2788K nanslp 0 3:13 0.00% gstat 1803 root 1 20 0 35476K 2128K nanslp 1 2:44 0.00% zpool 1798 root 1 20 0 16560K 2240K CPU0 7 1:07 0.00% top 1780 root 1 20 0 67884K 1792K select 2 0:23 0.00% sshd 1800 root 1 20 0 12052K 1484K select 6 0:17 0.00% script 1747 root 1 20 0 71980K 1868K select 1 0:13 0.00% sshd 3148 root 1 20 -20 21140K 8956K pause 7 0:11 0.00% atop 1850 root 1 20 0 12052K 1412K select 4 0:06 0.00% script 1784 root 1 20 0 67884K 1772K select 7 0:05 0.00% sshd 1652 nagios 1 20 0 12012K 1044K select 7 0:02 0.00% nrpe2 1795 root 1 20 0 12052K 1408K select 1 0:02 0.00% script 1538 root 1 20 0 11996K 960K nanslp 1 0:01 0.00% ipmon 1670 root 1 20 0 20272K 1876K select 1 0:01 0.00% sendmail 1677 root 1 20 0 14128K 1548K nanslp 2 0:00 0.00% cron 1547 root 1 20 0 12052K 1172K select 5 0:00 0.00% syslogd --- The last output of zpool iostat -v 1 capacity operations bandwidth pool alloc free read write read write -------------- ----- ----- ----- ----- ----- ----- tank 1.19T 63.8T 95 0 360K 0 raidz2 305G 16.0T 25 0 92.2K 0 gpt/disk3 - - 16 0 8.47K 0 gpt/disk9 - - 17 0 18.9K 0 gpt/disk15 - - 12 0 6.98K 0 gpt/disk19 - - 12 0 6.48K 0 gpt/disk23 - - 21 0 14.0K 0 gpt/disk27 - - 18 0 10.5K 0 gpt/disk31 - - 18 0 9.47K 0 gpt/disk36 - - 16 0 18.4K 0 gpt/disk33 - - 12 0 15.5K 0 raidz2 305G 16.0T 25 0 103K 0 gpt/disk1 - - 16 0 8.47K 0 gpt/disk4 - - 24 0 16.0K 0 gpt/disk7 - - 17 0 10.5K 0 gpt/disk10 - - 17 0 8.97K 0 gpt/disk13 - - 25 0 15.5K 0 gpt/disk16 - - 15 0 8.97K 0 gpt/disk24 - - 15 0 7.98K 0 gpt/disk32 - - 25 0 16.9K 0 gpt/disk37 - - 16 0 9.47K 0 raidz2 305G 16.0T 20 0 81.3K 0 gpt/disk2 - - 9 0 4.98K 0 gpt/disk5 - - 20 0 14.0K 0 gpt/disk8 - - 18 0 10.5K 0 gpt/disk11 - - 18 0 9.47K 0 gpt/disk17 - - 20 0 11.5K 0 gpt/disk21 - - 12 0 6.48K 0 gpt/disk25 - - 12 0 6.48K 0 gpt/disk29 - - 20 0 13.0K 0 gpt/disk38 - - 9 0 4.98K 0 raidz2 305G 16.0T 22 0 83.7K 0 gpt/disk12 - - 15 0 7.98K 0 gpt/disk14 - - 18 0 19.4K 0 gpt/disk18 - - 14 0 16.0K 0 gpt/disk22 - - 15 0 7.98K 0 gpt/disk26 - - 19 0 13.0K 0 gpt/disk30 - - 10 0 5.98K 0 gpt/disk34 - - 10 0 5.48K 0 gpt/disk35 - - 18 0 17.9K 0 gpt/disk39 - - 15 0 7.98K 0 -------------- ----- ----- ----- ----- ----- ----- zroot 2.67G 925G 0 0 0 0 mirror 2.67G 925G 0 0 0 0 gpt/disk0 - - 0 0 0 0 gpt/disk6 - - 0 0 0 0 -------------- ----- ----- ----- ----- ----- ----- and the last output of gstat -I500ms [gstat file] dT: 0.503s w: 0.500s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0| da0 0 10 10 5 4.6 0 0 0.0 4.6| da1 0 34 34 18 0.1 0 0 0.0 0.4| da2 0 8 8 6 0.2 0 0 0.0 0.1| da3 0 0 0 0 0.0 0 0 0.0 0.0| da0p1 0 0 0 0 0.0 0 0 0.0 0.0| da0p2 0 10 10 5 4.7 0 0 0.0 4.6| da1p1 0 34 34 18 0.2 0 0 0.0 0.5| da2p1 0 8 8 6 0.2 0 0 0.0 0.2| da3p1 0 18 18 12 0.2 0 0 0.0 0.2| da4 0 52 52 31 0.5 0 0 0.0 2.5| da5 0 0 0 0 0.0 0 0 0.0 0.0| da6 0 12 12 8 3.3 0 0 0.0 3.9| da7 0 32 32 17 0.1 0 0 0.0 0.4| da8 0 10 10 7 0.2 0 0 0.0 0.1| da9 0 12 12 7 0.1 0 0 0.0 0.2| da10 0 32 32 16 0.1 0 0 0.0 0.4| da11 0 42 42 22 0.1 0 0 0.0 0.5| da12 0 18 18 12 0.2 0 0 0.0 0.2| da13 0 62 62 38 0.1 0 0 0.0 0.8| da14 0 6 6 4 0.2 0 0 0.0 0.1| da15 0 14 14 8 0.2 0 0 0.0 0.2| da16 0 52 52 32 0.3 0 0 0.0 1.4| da17 0 40 40 21 0.1 0 0 0.0 0.5| da18 0 6 6 4 0.1 0 0 0.0 0.1| da19 0 0 0 0 0.0 0 0 0.0 0.0| da20 0 38 38 21 1.3 0 0 0.0 5.1| da21 0 40 40 20 0.1 0 0 0.0 0.5| da22 0 10 10 7 0.1 0 0 0.0 0.1| da23 0 14 14 8 3.4 0 0 0.0 4.7| da24 0 38 38 20 1.5 0 0 0.0 5.8| da25 0 62 62 39 0.1 0 0 0.0 0.8| da26 0 6 6 4 0.2 0 0 0.0 0.1| da27 0 0 0 0 0.0 0 0 0.0 0.0| da28 0 52 52 4 0.2 0 0 0.0 0.1| da29 0 70 70 36 0.1 0 0 0.0 0.9| da30 0 38 38 19 0.1 0 0 0.0 0.5| da31 0 0 0 0 0.0 0 0 0.0 0.0| da32 0 40 40 20 1.1 0 0 0.0 4.5| da33 0 70 70 35 0.1 0 0 0.0 0.9| da34 0 87 87 51 0.6 0 0 0.0 4.9| da35 0 54 54 32 0.1 0 0 0.0 0.7| da36 0 0 0 0 0.0 0 0 0.0 0.0| da37 0 8 8 4 18.8 0 0 0.0 3.8| da38 0 56 56 28 0.1 0 0 0.0 0.7| da39 [...] --- > Next: are you using compression or dedup on any of your filesystems? > If not, have you ever in the past? No, this pool was build from scratch without any compression or dedup. > Next: could we have your loader.conf and sysctl.conf please? loader.conf zfs_load="YES" vfs.root.mountfrom="zfs:zroot" console=comconsole sysctl.conf is empty > If you could put a swap disk on a dedicated controller (and no other > disks on it), that would be ideal. Please do not use USB for this > task > (the USB stack may introduce its own set of complexities pertaining > to > interrupt usage). I can't easily do this in the current setup. I would have to recreate the primary pool differently. Thanks again, Pascal ------=_Part_201016_445496304.1370608667459 Content-Type: text/plain; name=da0.txt Content-Disposition: attachment; filename=da0.txt Content-Transfer-Encoding: base64 c21hcnRjdGwgNi4xIDIwMTMtMDMtMTYgcjM4MDAgW0ZyZWVCU0QgOS4xLVJFTEVBU0UgYW1kNjRd IChsb2NhbCBidWlsZCkKQ29weXJpZ2h0IChDKSAyMDAyLTEzLCBCcnVjZSBBbGxlbiwgQ2hyaXN0 aWFuIEZyYW5rZSwgd3d3LnNtYXJ0bW9udG9vbHMub3JnCgo9PT0gU1RBUlQgT0YgSU5GT1JNQVRJ T04gU0VDVElPTiA9PT0KTW9kZWwgRmFtaWx5OiAgICAgSGl0YWNoaSBEZXNrc3RhciA3SzEwMDAu QwpEZXZpY2UgTW9kZWw6ICAgICBIaXRhY2hpIEhEUzcyMTAxMENMQTMzMApTZXJpYWwgTnVtYmVy OiAgICBKUDI5NDBOMTE4VlNNVgpMVSBXV04gRGV2aWNlIElkOiA1IDAwMGNjYSAzOWNkMjFlZWYK RmlybXdhcmUgVmVyc2lvbjogSlA0T0EzTUEKVXNlciBDYXBhY2l0eTogICAgMSwwMDAsMjA0LDg4 NiwwMTYgYnl0ZXMgWzEuMDAgVEJdClNlY3RvciBTaXplOiAgICAgIDUxMiBieXRlcyBsb2dpY2Fs L3BoeXNpY2FsClJvdGF0aW9uIFJhdGU6ICAgIDcyMDAgcnBtCkRldmljZSBpczogICAgICAgIElu IHNtYXJ0Y3RsIGRhdGFiYXNlIFtmb3IgZGV0YWlscyB1c2U6IC1QIHNob3ddCkFUQSBWZXJzaW9u IGlzOiAgIEFUQTgtQUNTIFQxMy8xNjk5LUQgcmV2aXNpb24gNApTQVRBIFZlcnNpb24gaXM6ICBT QVRBIDIuNiwgMy4wIEdiL3MKTG9jYWwgVGltZSBpczogICAgVHVlIEp1biAgNCAxMzoyOToyMiAy MDEzIENFU1QKU01BUlQgc3VwcG9ydCBpczogQXZhaWxhYmxlIC0gZGV2aWNlIGhhcyBTTUFSVCBj YXBhYmlsaXR5LgpTTUFSVCBzdXBwb3J0IGlzOiBFbmFibGVkCkFBTSBmZWF0dXJlIGlzOiAgIFVu YXZhaWxhYmxlCkFQTSBmZWF0dXJlIGlzOiAgIERpc2FibGVkClJkIGxvb2stYWhlYWQgaXM6IEVu YWJsZWQKV3JpdGUgY2FjaGUgaXM6ICAgRW5hYmxlZApBVEEgU2VjdXJpdHkgaXM6ICBEaXNhYmxl ZCwgTk9UIEZST1pFTiBbU0VDMV0KCj09PSBTVEFSVCBPRiBSRUFEIFNNQVJUIERBVEEgU0VDVElP TiA9PT0KU01BUlQgb3ZlcmFsbC1oZWFsdGggc2VsZi1hc3Nlc3NtZW50IHRlc3QgcmVzdWx0OiBQ QVNTRUQKCkdlbmVyYWwgU01BUlQgVmFsdWVzOgpPZmZsaW5lIGRhdGEgY29sbGVjdGlvbiBzdGF0 dXM6ICAoMHg4MikJT2ZmbGluZSBkYXRhIGNvbGxlY3Rpb24gYWN0aXZpdHkKCQkJCQl3YXMgY29t cGxldGVkIHdpdGhvdXQgZXJyb3IuCgkJCQkJQXV0byBPZmZsaW5lIERhdGEgQ29sbGVjdGlvbjog RW5hYmxlZC4KU2VsZi10ZXN0IGV4ZWN1dGlvbiBzdGF0dXM6ICAgICAgKCAgIDApCVRoZSBwcmV2 aW91cyBzZWxmLXRlc3Qgcm91dGluZSBjb21wbGV0ZWQKCQkJCQl3aXRob3V0IGVycm9yIG9yIG5v IHNlbGYtdGVzdCBoYXMgZXZlciAKCQkJCQliZWVuIHJ1bi4KVG90YWwgdGltZSB0byBjb21wbGV0 ZSBPZmZsaW5lIApkYXRhIGNvbGxlY3Rpb246IAkJKCA5ODEyKSBzZWNvbmRzLgpPZmZsaW5lIGRh dGEgY29sbGVjdGlvbgpjYXBhYmlsaXRpZXM6IAkJCSAoMHg1YikgU01BUlQgZXhlY3V0ZSBPZmZs aW5lIGltbWVkaWF0ZS4KCQkJCQlBdXRvIE9mZmxpbmUgZGF0YSBjb2xsZWN0aW9uIG9uL29mZiBz dXBwb3J0LgoJCQkJCVN1c3BlbmQgT2ZmbGluZSBjb2xsZWN0aW9uIHVwb24gbmV3CgkJCQkJY29t bWFuZC4KCQkJCQlPZmZsaW5lIHN1cmZhY2Ugc2NhbiBzdXBwb3J0ZWQuCgkJCQkJU2VsZi10ZXN0 IHN1cHBvcnRlZC4KCQkJCQlObyBDb252ZXlhbmNlIFNlbGYtdGVzdCBzdXBwb3J0ZWQuCgkJCQkJ U2VsZWN0aXZlIFNlbGYtdGVzdCBzdXBwb3J0ZWQuClNNQVJUIGNhcGFiaWxpdGllczogICAgICAg ICAgICAoMHgwMDAzKQlTYXZlcyBTTUFSVCBkYXRhIGJlZm9yZSBlbnRlcmluZwoJCQkJCXBvd2Vy LXNhdmluZyBtb2RlLgoJCQkJCVN1cHBvcnRzIFNNQVJUIGF1dG8gc2F2ZSB0aW1lci4KRXJyb3Ig bG9nZ2luZyBjYXBhYmlsaXR5OiAgICAgICAgKDB4MDEpCUVycm9yIGxvZ2dpbmcgc3VwcG9ydGVk LgoJCQkJCUdlbmVyYWwgUHVycG9zZSBMb2dnaW5nIHN1cHBvcnRlZC4KU2hvcnQgc2VsZi10ZXN0 IHJvdXRpbmUgCnJlY29tbWVuZGVkIHBvbGxpbmcgdGltZTogCSAoICAgMSkgbWludXRlcy4KRXh0 ZW5kZWQgc2VsZi10ZXN0IHJvdXRpbmUKcmVjb21tZW5kZWQgcG9sbGluZyB0aW1lOiAJICggMTY0 KSBtaW51dGVzLgpTQ1QgY2FwYWJpbGl0aWVzOiAJICAgICAgICgweDAwM2QpCVNDVCBTdGF0dXMg c3VwcG9ydGVkLgoJCQkJCVNDVCBFcnJvciBSZWNvdmVyeSBDb250cm9sIHN1cHBvcnRlZC4KCQkJ CQlTQ1QgRmVhdHVyZSBDb250cm9sIHN1cHBvcnRlZC4KCQkJCQlTQ1QgRGF0YSBUYWJsZSBzdXBw b3J0ZWQuCgpTTUFSVCBBdHRyaWJ1dGVzIERhdGEgU3RydWN0dXJlIHJldmlzaW9uIG51bWJlcjog MTYKVmVuZG9yIFNwZWNpZmljIFNNQVJUIEF0dHJpYnV0ZXMgd2l0aCBUaHJlc2hvbGRzOgpJRCMg QVRUUklCVVRFX05BTUUgICAgICAgICAgRkxBR1MgICAgVkFMVUUgV09SU1QgVEhSRVNIIEZBSUwg UkFXX1ZBTFVFCiAgMSBSYXdfUmVhZF9FcnJvcl9SYXRlICAgICBQTy1SLS0gICAwOTUgICAwOTUg ICAwMTYgICAgLSAgICA2NTU1MAogIDIgVGhyb3VnaHB1dF9QZXJmb3JtYW5jZSAgUC1TLS0tICAg MTM0ICAgMTM0ICAgMDU0ICAgIC0gICAgMTAwCiAgMyBTcGluX1VwX1RpbWUgICAgICAgICAgICBQ T1MtLS0gICAxMjQgICAxMjQgICAwMjQgICAgLSAgICAzMDUgKEF2ZXJhZ2UgMzA3KQogIDQgU3Rh cnRfU3RvcF9Db3VudCAgICAgICAgLU8tLUMtICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMTUK ICA1IFJlYWxsb2NhdGVkX1NlY3Rvcl9DdCAgIFBPLS1DSyAgIDEwMCAgIDEwMCAgIDAwNSAgICAt ICAgIDAKICA3IFNlZWtfRXJyb3JfUmF0ZSAgICAgICAgIFBPLVItLSAgIDEwMCAgIDEwMCAgIDA2 NyAgICAtICAgIDAKICA4IFNlZWtfVGltZV9QZXJmb3JtYW5jZSAgIFAtUy0tLSAgIDEzOCAgIDEz OCAgIDAyMCAgICAtICAgIDMxCiAgOSBQb3dlcl9Pbl9Ib3VycyAgICAgICAgICAtTy0tQy0gICAx MDAgICAxMDAgICAwMDAgICAgLSAgICA1MzcKIDEwIFNwaW5fUmV0cnlfQ291bnQgICAgICAgIFBP LS1DLSAgIDEwMCAgIDEwMCAgIDA2MCAgICAtICAgIDAKIDEyIFBvd2VyX0N5Y2xlX0NvdW50ICAg ICAgIC1PLS1DSyAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAgIDE1CjE5MiBQb3dlci1PZmZfUmV0 cmFjdF9Db3VudCAtTy0tQ0sgICAxMDAgICAxMDAgICAwMDAgICAgLSAgICAyNQoxOTMgTG9hZF9D eWNsZV9Db3VudCAgICAgICAgLU8tLUMtICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMjUKMTk0 IFRlbXBlcmF0dXJlX0NlbHNpdXMgICAgIC1PLS0tLSAgIDIwMCAgIDIwMCAgIDAwMCAgICAtICAg IDMwIChNaW4vTWF4IDI0LzM2KQoxOTYgUmVhbGxvY2F0ZWRfRXZlbnRfQ291bnQgLU8tLUNLICAg MTAwICAgMTAwICAgMDAwICAgIC0gICAgMAoxOTcgQ3VycmVudF9QZW5kaW5nX1NlY3RvciAgLU8t LS1LICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMAoxOTggT2ZmbGluZV9VbmNvcnJlY3RhYmxl ICAgLS0tUi0tICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMAoxOTkgVURNQV9DUkNfRXJyb3Jf Q291bnQgICAgLU8tUi0tICAgMjAwICAgMjAwICAgMDAwICAgIC0gICAgMAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgfHx8fHx8XyBLIGF1dG8ta2VlcAogICAgICAgICAgICAgICAgICAgICAg ICAgICAgfHx8fHxfXyBDIGV2ZW50IGNvdW50CiAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 fHx8X19fIFIgZXJyb3IgcmF0ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgfHx8X19fXyBT IHNwZWVkL3BlcmZvcm1hbmNlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICB8fF9fX19fIE8g dXBkYXRlZCBvbmxpbmUKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHxfX19fX18gUCBwcmVm YWlsdXJlIHdhcm5pbmcKCkdlbmVyYWwgUHVycG9zZSBMb2cgRGlyZWN0b3J5IFZlcnNpb24gMQpT TUFSVCAgICAgICAgICAgTG9nIERpcmVjdG9yeSBWZXJzaW9uIDEgW211bHRpLXNlY3RvciBsb2cg c3VwcG9ydF0KQWRkcmVzcyAgICBBY2Nlc3MgIFIvVyAgIFNpemUgIERlc2NyaXB0aW9uCjB4MDAg ICAgICAgR1BMLFNMICBSL08gICAgICAxICBMb2cgRGlyZWN0b3J5CjB4MDEgICAgICAgICAgIFNM ICBSL08gICAgICAxICBTdW1tYXJ5IFNNQVJUIGVycm9yIGxvZwoweDAzICAgICAgIEdQTCAgICAg Ui9PICAgICAgMSAgRXh0LiBDb21wcmVoZW5zaXZlIFNNQVJUIGVycm9yIGxvZwoweDA0ICAgICAg IEdQTCAgICAgUi9PICAgICAgNyAgRGV2aWNlIFN0YXRpc3RpY3MgbG9nCjB4MDYgICAgICAgICAg IFNMICBSL08gICAgICAxICBTTUFSVCBzZWxmLXRlc3QgbG9nCjB4MDcgICAgICAgR1BMICAgICBS L08gICAgICAxICBFeHRlbmRlZCBzZWxmLXRlc3QgbG9nCjB4MDkgICAgICAgICAgIFNMICBSL1cg ICAgICAxICBTZWxlY3RpdmUgc2VsZi10ZXN0IGxvZwoweDEwICAgICAgIEdQTCAgICAgUi9PICAg ICAgMSAgTkNRIENvbW1hbmQgRXJyb3IgbG9nCjB4MTEgICAgICAgR1BMICAgICBSL08gICAgICAx ICBTQVRBIFBoeSBFdmVudCBDb3VudGVycwoweDIwICAgICAgIEdQTCAgICAgUi9PICAgICAgMSAg U3RyZWFtaW5nIHBlcmZvcm1hbmNlIGxvZyBbT0JTLThdCjB4MjEgICAgICAgR1BMICAgICBSL08g ICAgICAxICBXcml0ZSBzdHJlYW0gZXJyb3IgbG9nCjB4MjIgICAgICAgR1BMICAgICBSL08gICAg ICAxICBSZWFkIHN0cmVhbSBlcnJvciBsb2cKMHg4MC0weDlmICBHUEwsU0wgIFIvVyAgICAgMTYg IEhvc3QgdmVuZG9yIHNwZWNpZmljIGxvZwoweGUwICAgICAgIEdQTCxTTCAgUi9XICAgICAgMSAg U0NUIENvbW1hbmQvU3RhdHVzCjB4ZTEgICAgICAgR1BMLFNMICBSL1cgICAgICAxICBTQ1QgRGF0 YSBUcmFuc2ZlcgoKU01BUlQgRXh0ZW5kZWQgQ29tcHJlaGVuc2l2ZSBFcnJvciBMb2cgVmVyc2lv bjogMCAoMSBzZWN0b3JzKQpObyBFcnJvcnMgTG9nZ2VkCgpTTUFSVCBFeHRlbmRlZCBTZWxmLXRl c3QgTG9nIFZlcnNpb246IDEgKDEgc2VjdG9ycykKTm8gc2VsZi10ZXN0cyBoYXZlIGJlZW4gbG9n Z2VkLiAgW1RvIHJ1biBzZWxmLXRlc3RzLCB1c2U6IHNtYXJ0Y3RsIC10XQoKU01BUlQgU2VsZWN0 aXZlIHNlbGYtdGVzdCBsb2cgZGF0YSBzdHJ1Y3R1cmUgcmV2aXNpb24gbnVtYmVyIDEKIFNQQU4g IE1JTl9MQkEgIE1BWF9MQkEgIENVUlJFTlRfVEVTVF9TVEFUVVMKICAgIDEgICAgICAgIDAgICAg ICAgIDAgIE5vdF90ZXN0aW5nCiAgICAyICAgICAgICAwICAgICAgICAwICBOb3RfdGVzdGluZwog ICAgMyAgICAgICAgMCAgICAgICAgMCAgTm90X3Rlc3RpbmcKICAgIDQgICAgICAgIDAgICAgICAg IDAgIE5vdF90ZXN0aW5nCiAgICA1ICAgICAgICAwICAgICAgICAwICBOb3RfdGVzdGluZwpTZWxl Y3RpdmUgc2VsZi10ZXN0IGZsYWdzICgweDApOgogIEFmdGVyIHNjYW5uaW5nIHNlbGVjdGVkIHNw YW5zLCBkbyBOT1QgcmVhZC1zY2FuIHJlbWFpbmRlciBvZiBkaXNrLgpJZiBTZWxlY3RpdmUgc2Vs Zi10ZXN0IGlzIHBlbmRpbmcgb24gcG93ZXItdXAsIHJlc3VtZSBhZnRlciAwIG1pbnV0ZSBkZWxh eS4KClNDVCBTdGF0dXMgVmVyc2lvbjogICAgICAgICAgICAgICAgICAzClNDVCBWZXJzaW9uICh2 ZW5kb3Igc3BlY2lmaWMpOiAgICAgICAyNTYgKDB4MDEwMCkKU0NUIFN1cHBvcnQgTGV2ZWw6ICAg ICAgICAgICAgICAgICAgIDEKRGV2aWNlIFN0YXRlOiAgICAgICAgICAgICAgICAgICAgICAgIEFj dGl2ZSAoMCkKQ3VycmVudCBUZW1wZXJhdHVyZTogICAgICAgICAgICAgICAgICAgIDI5IENlbHNp dXMKUG93ZXIgQ3ljbGUgTWluL01heCBUZW1wZXJhdHVyZTogICAgIDI5LzMxIENlbHNpdXMKTGlm ZXRpbWUgICAgTWluL01heCBUZW1wZXJhdHVyZTogICAgIDI0LzM2IENlbHNpdXMKVW5kZXIvT3Zl ciBUZW1wZXJhdHVyZSBMaW1pdCBDb3VudDogICAwLzAKU0NUIFRlbXBlcmF0dXJlIEhpc3Rvcnkg VmVyc2lvbjogICAgIDIKVGVtcGVyYXR1cmUgU2FtcGxpbmcgUGVyaW9kOiAgICAgICAgIDEgbWlu dXRlClRlbXBlcmF0dXJlIExvZ2dpbmcgSW50ZXJ2YWw6ICAgICAgICAxIG1pbnV0ZQpNaW4vTWF4 IHJlY29tbWVuZGVkIFRlbXBlcmF0dXJlOiAgICAgIDAvNjAgQ2Vsc2l1cwpNaW4vTWF4IFRlbXBl cmF0dXJlIExpbWl0OiAgICAgICAgICAgLTQwLzcwIENlbHNpdXMKVGVtcGVyYXR1cmUgSGlzdG9y eSBTaXplIChJbmRleCk6ICAgIDEyOCAoMTAxKQoKSW5kZXggICAgRXN0aW1hdGVkIFRpbWUgICBU ZW1wZXJhdHVyZSBDZWxzaXVzCiAxMDIgICAgMjAxMy0wNi0wNCAxMToyMiAgICAyOSAgKioqKioq KioqKgogLi4uICAgIC4uKCAyMiBza2lwcGVkKS4gICAgLi4gICoqKioqKioqKioKIDEyNSAgICAy MDEzLTA2LTA0IDExOjQ1ICAgIDI5ICAqKioqKioqKioqCiAxMjYgICAgMjAxMy0wNi0wNCAxMTo0 NiAgICAzMCAgKioqKioqKioqKioKIDEyNyAgICAyMDEzLTA2LTA0IDExOjQ3ICAgIDMwICAqKioq KioqKioqKgogICAwICAgIDIwMTMtMDYtMDQgMTE6NDggICAgMzAgICoqKioqKioqKioqCiAgIDEg ICAgMjAxMy0wNi0wNCAxMTo0OSAgICAyOSAgKioqKioqKioqKgogICAyICAgIDIwMTMtMDYtMDQg MTE6NTAgICAgMzAgICoqKioqKioqKioqCiAgIDMgICAgMjAxMy0wNi0wNCAxMTo1MSAgICAyOSAg KioqKioqKioqKgogLi4uICAgIC4uKCAgNiBza2lwcGVkKS4gICAgLi4gICoqKioqKioqKioKICAx MCAgICAyMDEzLTA2LTA0IDExOjU4ICAgIDI5ICAqKioqKioqKioqCiAgMTEgICAgMjAxMy0wNi0w NCAxMTo1OSAgICAzMCAgKioqKioqKioqKioKICAxMiAgICAyMDEzLTA2LTA0IDEyOjAwICAgIDI5 ICAqKioqKioqKioqCiAgMTMgICAgMjAxMy0wNi0wNCAxMjowMSAgICAyOSAgKioqKioqKioqKgog IDE0ICAgIDIwMTMtMDYtMDQgMTI6MDIgICAgMzAgICoqKioqKioqKioqCiAgMTUgICAgMjAxMy0w Ni0wNCAxMjowMyAgICAyOSAgKioqKioqKioqKgogLi4uICAgIC4uKCAgMiBza2lwcGVkKS4gICAg Li4gICoqKioqKioqKioKICAxOCAgICAyMDEzLTA2LTA0IDEyOjA2ICAgIDI5ICAqKioqKioqKioq CiAgMTkgICAgMjAxMy0wNi0wNCAxMjowNyAgICAzMCAgKioqKioqKioqKioKICAyMCAgICAyMDEz LTA2LTA0IDEyOjA4ICAgIDI5ICAqKioqKioqKioqCiAgMjEgICAgMjAxMy0wNi0wNCAxMjowOSAg ICAzMCAgKioqKioqKioqKioKICAyMiAgICAyMDEzLTA2LTA0IDEyOjEwICAgIDI5ICAqKioqKioq KioqCiAuLi4gICAgLi4oICAyIHNraXBwZWQpLiAgICAuLiAgKioqKioqKioqKgogIDI1ICAgIDIw MTMtMDYtMDQgMTI6MTMgICAgMjkgICoqKioqKioqKioKICAyNiAgICAyMDEzLTA2LTA0IDEyOjE0 ICAgIDMwICAqKioqKioqKioqKgogIDI3ICAgIDIwMTMtMDYtMDQgMTI6MTUgICAgMzAgICoqKioq KioqKioqCiAgMjggICAgMjAxMy0wNi0wNCAxMjoxNiAgICAyOSAgKioqKioqKioqKgogLi4uICAg IC4uKCAzNCBza2lwcGVkKS4gICAgLi4gICoqKioqKioqKioKICA2MyAgICAyMDEzLTA2LTA0IDEy OjUxICAgIDI5ICAqKioqKioqKioqCiAgNjQgICAgMjAxMy0wNi0wNCAxMjo1MiAgICAzMCAgKioq KioqKioqKioKICA2NSAgICAyMDEzLTA2LTA0IDEyOjUzICAgIDI5ICAqKioqKioqKioqCiAuLi4g ICAgLi4oICA4IHNraXBwZWQpLiAgICAuLiAgKioqKioqKioqKgogIDc0ICAgIDIwMTMtMDYtMDQg MTM6MDIgICAgMjkgICoqKioqKioqKioKICA3NSAgICAyMDEzLTA2LTA0IDEzOjAzICAgIDMwICAq KioqKioqKioqKgogIDc2ICAgIDIwMTMtMDYtMDQgMTM6MDQgICAgMjkgICoqKioqKioqKioKIC4u LiAgICAuLiggIDcgc2tpcHBlZCkuICAgIC4uICAqKioqKioqKioqCiAgODQgICAgMjAxMy0wNi0w NCAxMzoxMiAgICAyOSAgKioqKioqKioqKgogIDg1ICAgIDIwMTMtMDYtMDQgMTM6MTMgICAgMzAg ICoqKioqKioqKioqCiAgODYgICAgMjAxMy0wNi0wNCAxMzoxNCAgICAyOSAgKioqKioqKioqKgog Li4uICAgIC4uKCAgNiBza2lwcGVkKS4gICAgLi4gICoqKioqKioqKioKICA5MyAgICAyMDEzLTA2 LTA0IDEzOjIxICAgIDI5ICAqKioqKioqKioqCiAgOTQgICAgMjAxMy0wNi0wNCAxMzoyMiAgICAz MCAgKioqKioqKioqKioKICA5NSAgICAyMDEzLTA2LTA0IDEzOjIzICAgIDMwICAqKioqKioqKioq KgogIDk2ICAgIDIwMTMtMDYtMDQgMTM6MjQgICAgMzAgICoqKioqKioqKioqCiAgOTcgICAgMjAx My0wNi0wNCAxMzoyNSAgICAyOSAgKioqKioqKioqKgogIDk4ICAgIDIwMTMtMDYtMDQgMTM6MjYg ICAgMjkgICoqKioqKioqKioKICA5OSAgICAyMDEzLTA2LTA0IDEzOjI3ICAgIDI5ICAqKioqKioq KioqCiAxMDAgICAgMjAxMy0wNi0wNCAxMzoyOCAgICAzMCAgKioqKioqKioqKioKIDEwMSAgICAy MDEzLTA2LTA0IDEzOjI5ICAgIDMwICAqKioqKioqKioqKgoKU01BUlQgV1JJVEUgTE9HIGRvZXMg bm90IHJldHVybiBDT1VOVCBhbmQgTEJBX0xPVyByZWdpc3RlcgpTQ1QgKEdldCkgRXJyb3IgUmVj b3ZlcnkgQ29udHJvbCBjb21tYW5kIGZhaWxlZAoKRGV2aWNlIFN0YXRpc3RpY3MgKEdQIExvZyAw eDA0KQpQYWdlIE9mZnNldCBTaXplICAgICAgICAgVmFsdWUgIERlc2NyaXB0aW9uCiAgMSAgPT09 PT0gID0gICAgICAgICAgICAgICAgPSAgPT0gR2VuZXJhbCBTdGF0aXN0aWNzIChyZXYgMSkgPT0K ICAxICAweDAwOCAgNCAgICAgICAgICAgICAgIDE1ICBMaWZldGltZSBQb3dlci1PbiBSZXNldHMK ICAxICAweDAxMCAgNCAgICAgICAgICAgICAgNTM3ICBQb3dlci1vbiBIb3VycwogIDEgIDB4MDE4 ICA2ICAgICAgICAxNjM2OTM4MTMgIExvZ2ljYWwgU2VjdG9ycyBXcml0dGVuCiAgMSAgMHgwMjAg IDYgICAgICAgICAxMDE0MjcyNSAgTnVtYmVyIG9mIFdyaXRlIENvbW1hbmRzCiAgMSAgMHgwMjgg IDYgICAgICAyMDk2MzM4NTEyNyAgTG9naWNhbCBTZWN0b3JzIFJlYWQKICAxICAweDAzMCAgNiAg ICAgICAgICAgNzgxOTgyICBOdW1iZXIgb2YgUmVhZCBDb21tYW5kcwogIDMgID09PT09ICA9ICAg ICAgICAgICAgICAgID0gID09IFJvdGF0aW5nIE1lZGlhIFN0YXRpc3RpY3MgKHJldiAxKSA9PQog IDMgIDB4MDA4ICA0ICAgICAgICAgICAgICA1MzcgIFNwaW5kbGUgTW90b3IgUG93ZXItb24gSG91 cnMKICAzICAweDAxMCAgNCAgICAgICAgICAgICAgNTM3ICBIZWFkIEZseWluZyBIb3VycwogIDMg IDB4MDE4ICA0ICAgICAgICAgICAgICAgMjUgIEhlYWQgTG9hZCBFdmVudHMKICAzICAweDAyMCAg NCAgICAgICAgICAgICAgICAwICBOdW1iZXIgb2YgUmVhbGxvY2F0ZWQgTG9naWNhbCBTZWN0b3Jz CiAgMyAgMHgwMjggIDQgICAgICAgICAgICAgICAgMCAgUmVhZCBSZWNvdmVyeSBBdHRlbXB0cwog IDMgIDB4MDMwICA0ICAgICAgIDQyOTQ5NjcyOTUgIE51bWJlciBvZiBNZWNoYW5pY2FsIFN0YXJ0 IEZhaWx1cmVzCiAgNCAgPT09PT0gID0gICAgICAgICAgICAgICAgPSAgPT0gR2VuZXJhbCBFcnJv cnMgU3RhdGlzdGljcyAocmV2IDEpID09CiAgNCAgMHgwMDggIDQgICAgICAgICAgICAgICAgMCAg TnVtYmVyIG9mIFJlcG9ydGVkIFVuY29ycmVjdGFibGUgRXJyb3JzCiAgNCAgMHgwMTAgIDQgICAg ICAgICAgICAgICAgMSAgUmVzZXRzIEJldHdlZW4gQ21kIEFjY2VwdGFuY2UgYW5kIENvbXBsZXRp b24KICA1ICA9PT09PSAgPSAgICAgICAgICAgICAgICA9ICA9PSBUZW1wZXJhdHVyZSBTdGF0aXN0 aWNzIChyZXYgMSkgPT0KICA1ICAweDAwOCAgMSAgICAgICAgICAgICAgIDMwICBDdXJyZW50IFRl bXBlcmF0dXJlCiAgNSAgMHgwMTAgIDEgICAgICAgICAgICAgICAyOX4gQXZlcmFnZSBTaG9ydCBU ZXJtIFRlbXBlcmF0dXJlCiAgNSAgMHgwMTggIDEgICAgICAgICAgICAgICAgLX4gQXZlcmFnZSBM b25nIFRlcm0gVGVtcGVyYXR1cmUKICA1ICAweDAyMCAgMSAgICAgICAgICAgICAgIDM2ICBIaWdo ZXN0IFRlbXBlcmF0dXJlCiAgNSAgMHgwMjggIDEgICAgICAgICAgICAgICAyNCAgTG93ZXN0IFRl bXBlcmF0dXJlCiAgNSAgMHgwMzAgIDEgICAgICAgICAgICAgICAzNH4gSGlnaGVzdCBBdmVyYWdl IFNob3J0IFRlcm0gVGVtcGVyYXR1cmUKICA1ICAweDAzOCAgMSAgICAgICAgICAgICAgICAwfiBM b3dlc3QgQXZlcmFnZSBTaG9ydCBUZXJtIFRlbXBlcmF0dXJlCiAgNSAgMHgwNDAgIDEgICAgICAg ICAgICAgICAgLX4gSGlnaGVzdCBBdmVyYWdlIExvbmcgVGVybSBUZW1wZXJhdHVyZQogIDUgIDB4 MDQ4ICAxICAgICAgICAgICAgICAgIC1+IExvd2VzdCBBdmVyYWdlIExvbmcgVGVybSBUZW1wZXJh dHVyZQogIDUgIDB4MDUwICA0ICAgICAgICAgICAgICAgIDAgIFRpbWUgaW4gT3Zlci1UZW1wZXJh dHVyZQogIDUgIDB4MDU4ICAxICAgICAgICAgICAgICAgNjAgIFNwZWNpZmllZCBNYXhpbXVtIE9w ZXJhdGluZyBUZW1wZXJhdHVyZQogIDUgIDB4MDYwICA0ICAgICAgICAgICAgICAgIDAgIFRpbWUg aW4gVW5kZXItVGVtcGVyYXR1cmUKICA1ICAweDA2OCAgMSAgICAgICAgICAgICAgICAwICBTcGVj aWZpZWQgTWluaW11bSBPcGVyYXRpbmcgVGVtcGVyYXR1cmUKICA2ICA9PT09PSAgPSAgICAgICAg ICAgICAgICA9ICA9PSBUcmFuc3BvcnQgU3RhdGlzdGljcyAocmV2IDEpID09CiAgNiAgMHgwMDgg IDQgICAgICAgICAgICAgICA1MyAgTnVtYmVyIG9mIEhhcmR3YXJlIFJlc2V0cwogIDYgIDB4MDEw ICA0ICAgICAgICAgICAgICAgNTMgIE51bWJlciBvZiBBU1IgRXZlbnRzCiAgNiAgMHgwMTggIDQg ICAgICAgICAgICAgICAgMCAgTnVtYmVyIG9mIEludGVyZmFjZSBDUkMgRXJyb3JzCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHxfIH4gbm9ybWFsaXplZCB2YWx1ZQoKU0FUQSBQaHkgRXZl bnQgQ291bnRlcnMgKEdQIExvZyAweDExKQpJRCAgICAgIFNpemUgICAgIFZhbHVlICBEZXNjcmlw dGlvbgoweDAwMDEgIDIgICAgICAgICAgICAwICBDb21tYW5kIGZhaWxlZCBkdWUgdG8gSUNSQyBl cnJvcgoweDAwMDIgIDIgICAgICAgICAgICAwICBSX0VSUiByZXNwb25zZSBmb3IgZGF0YSBGSVMK MHgwMDA1ICAyICAgICAgICAgICAgMCAgUl9FUlIgcmVzcG9uc2UgZm9yIG5vbi1kYXRhIEZJUwow eDAwMDkgIDIgICAgICAgICAgICA1ICBUcmFuc2l0aW9uIGZyb20gZHJpdmUgUGh5UmR5IHRvIGRy aXZlIFBoeU5SZHkKMHgwMDBhICAyICAgICAgICAgICAgNSAgRGV2aWNlLXRvLWhvc3QgcmVnaXN0 ZXIgRklTZXMgc2VudCBkdWUgdG8gYSBDT01SRVNFVAoweDAwMGIgIDIgICAgICAgICAgICAwICBD UkMgZXJyb3JzIHdpdGhpbiBob3N0LXRvLWRldmljZSBGSVMKMHgwMDBkICAyICAgICAgICAgICAg MCAgTm9uLUNSQyBlcnJvcnMgd2l0aGluIGhvc3QtdG8tZGV2aWNlIEZJUwoK ------=_Part_201016_445496304.1370608667459 Content-Type: text/plain; name=da1.txt Content-Disposition: attachment; filename=da1.txt Content-Transfer-Encoding: base64 c21hcnRjdGwgNi4xIDIwMTMtMDMtMTYgcjM4MDAgW0ZyZWVCU0QgOS4xLVJFTEVBU0UgYW1kNjRd IChsb2NhbCBidWlsZCkKQ29weXJpZ2h0IChDKSAyMDAyLTEzLCBCcnVjZSBBbGxlbiwgQ2hyaXN0 aWFuIEZyYW5rZSwgd3d3LnNtYXJ0bW9udG9vbHMub3JnCgo9PT0gU1RBUlQgT0YgSU5GT1JNQVRJ T04gU0VDVElPTiA9PT0KTW9kZWwgRmFtaWx5OiAgICAgSGl0YWNoaSBEZXNrc3RhciA1SzMwMDAK RGV2aWNlIE1vZGVsOiAgICAgSGl0YWNoaSBIRFM1QzMwMjBBTEE2MzIKU2VyaWFsIE51bWJlcjog ICAgTUwyMjIwRjMzNDlTVEUKTFUgV1dOIERldmljZSBJZDogNSAwMDBjY2EgMzY5ZWMzY2E5CkZp cm13YXJlIFZlcnNpb246IE1MNk9BODAwClVzZXIgQ2FwYWNpdHk6ICAgIDIsMDAwLDM5OCw5MzQs MDE2IGJ5dGVzIFsyLjAwIFRCXQpTZWN0b3IgU2l6ZTogICAgICA1MTIgYnl0ZXMgbG9naWNhbC9w aHlzaWNhbApSb3RhdGlvbiBSYXRlOiAgICA1OTQwIHJwbQpEZXZpY2UgaXM6ICAgICAgICBJbiBz bWFydGN0bCBkYXRhYmFzZSBbZm9yIGRldGFpbHMgdXNlOiAtUCBzaG93XQpBVEEgVmVyc2lvbiBp czogICBBVEE4LUFDUyBUMTMvMTY5OS1EIHJldmlzaW9uIDQKU0FUQSBWZXJzaW9uIGlzOiAgU0FU QSAyLjYsIDYuMCBHYi9zIChjdXJyZW50OiAxLjUgR2IvcykKTG9jYWwgVGltZSBpczogICAgVHVl IEp1biAgNCAxMzoyODoyNCAyMDEzIENFU1QKU01BUlQgc3VwcG9ydCBpczogQXZhaWxhYmxlIC0g ZGV2aWNlIGhhcyBTTUFSVCBjYXBhYmlsaXR5LgpTTUFSVCBzdXBwb3J0IGlzOiBFbmFibGVkCkFB TSBmZWF0dXJlIGlzOiAgIFVuYXZhaWxhYmxlCkFQTSBmZWF0dXJlIGlzOiAgIERpc2FibGVkClJk IGxvb2stYWhlYWQgaXM6IEVuYWJsZWQKV3JpdGUgY2FjaGUgaXM6ICAgRW5hYmxlZApBVEEgU2Vj dXJpdHkgaXM6ICBEaXNhYmxlZCwgTk9UIEZST1pFTiBbU0VDMV0KCj09PSBTVEFSVCBPRiBSRUFE IFNNQVJUIERBVEEgU0VDVElPTiA9PT0KU01BUlQgb3ZlcmFsbC1oZWFsdGggc2VsZi1hc3Nlc3Nt ZW50IHRlc3QgcmVzdWx0OiBQQVNTRUQKCkdlbmVyYWwgU01BUlQgVmFsdWVzOgpPZmZsaW5lIGRh dGEgY29sbGVjdGlvbiBzdGF0dXM6ICAoMHg4MikJT2ZmbGluZSBkYXRhIGNvbGxlY3Rpb24gYWN0 aXZpdHkKCQkJCQl3YXMgY29tcGxldGVkIHdpdGhvdXQgZXJyb3IuCgkJCQkJQXV0byBPZmZsaW5l IERhdGEgQ29sbGVjdGlvbjogRW5hYmxlZC4KU2VsZi10ZXN0IGV4ZWN1dGlvbiBzdGF0dXM6ICAg ICAgKCAgIDApCVRoZSBwcmV2aW91cyBzZWxmLXRlc3Qgcm91dGluZSBjb21wbGV0ZWQKCQkJCQl3 aXRob3V0IGVycm9yIG9yIG5vIHNlbGYtdGVzdCBoYXMgZXZlciAKCQkJCQliZWVuIHJ1bi4KVG90 YWwgdGltZSB0byBjb21wbGV0ZSBPZmZsaW5lIApkYXRhIGNvbGxlY3Rpb246IAkJKDIyMTE3KSBz ZWNvbmRzLgpPZmZsaW5lIGRhdGEgY29sbGVjdGlvbgpjYXBhYmlsaXRpZXM6IAkJCSAoMHg1Yikg U01BUlQgZXhlY3V0ZSBPZmZsaW5lIGltbWVkaWF0ZS4KCQkJCQlBdXRvIE9mZmxpbmUgZGF0YSBj b2xsZWN0aW9uIG9uL29mZiBzdXBwb3J0LgoJCQkJCVN1c3BlbmQgT2ZmbGluZSBjb2xsZWN0aW9u IHVwb24gbmV3CgkJCQkJY29tbWFuZC4KCQkJCQlPZmZsaW5lIHN1cmZhY2Ugc2NhbiBzdXBwb3J0 ZWQuCgkJCQkJU2VsZi10ZXN0IHN1cHBvcnRlZC4KCQkJCQlObyBDb252ZXlhbmNlIFNlbGYtdGVz dCBzdXBwb3J0ZWQuCgkJCQkJU2VsZWN0aXZlIFNlbGYtdGVzdCBzdXBwb3J0ZWQuClNNQVJUIGNh cGFiaWxpdGllczogICAgICAgICAgICAoMHgwMDAzKQlTYXZlcyBTTUFSVCBkYXRhIGJlZm9yZSBl bnRlcmluZwoJCQkJCXBvd2VyLXNhdmluZyBtb2RlLgoJCQkJCVN1cHBvcnRzIFNNQVJUIGF1dG8g c2F2ZSB0aW1lci4KRXJyb3IgbG9nZ2luZyBjYXBhYmlsaXR5OiAgICAgICAgKDB4MDEpCUVycm9y IGxvZ2dpbmcgc3VwcG9ydGVkLgoJCQkJCUdlbmVyYWwgUHVycG9zZSBMb2dnaW5nIHN1cHBvcnRl ZC4KU2hvcnQgc2VsZi10ZXN0IHJvdXRpbmUgCnJlY29tbWVuZGVkIHBvbGxpbmcgdGltZTogCSAo ICAgMSkgbWludXRlcy4KRXh0ZW5kZWQgc2VsZi10ZXN0IHJvdXRpbmUKcmVjb21tZW5kZWQgcG9s bGluZyB0aW1lOiAJICggMzY5KSBtaW51dGVzLgpTQ1QgY2FwYWJpbGl0aWVzOiAJICAgICAgICgw eDAwM2QpCVNDVCBTdGF0dXMgc3VwcG9ydGVkLgoJCQkJCVNDVCBFcnJvciBSZWNvdmVyeSBDb250 cm9sIHN1cHBvcnRlZC4KCQkJCQlTQ1QgRmVhdHVyZSBDb250cm9sIHN1cHBvcnRlZC4KCQkJCQlT Q1QgRGF0YSBUYWJsZSBzdXBwb3J0ZWQuCgpTTUFSVCBBdHRyaWJ1dGVzIERhdGEgU3RydWN0dXJl IHJldmlzaW9uIG51bWJlcjogMTYKVmVuZG9yIFNwZWNpZmljIFNNQVJUIEF0dHJpYnV0ZXMgd2l0 aCBUaHJlc2hvbGRzOgpJRCMgQVRUUklCVVRFX05BTUUgICAgICAgICAgRkxBR1MgICAgVkFMVUUg V09SU1QgVEhSRVNIIEZBSUwgUkFXX1ZBTFVFCiAgMSBSYXdfUmVhZF9FcnJvcl9SYXRlICAgICBQ Ty1SLS0gICAxMDAgICAxMDAgICAwMTYgICAgLSAgICAwCiAgMiBUaHJvdWdocHV0X1BlcmZvcm1h bmNlICBQLVMtLS0gICAxMzUgICAxMzUgICAwNTQgICAgLSAgICA5NwogIDMgU3Bpbl9VcF9UaW1l ICAgICAgICAgICAgUE9TLS0tICAgMTM2ICAgMTM2ICAgMDI0ICAgIC0gICAgNDAzIChBdmVyYWdl IDQwMikKICA0IFN0YXJ0X1N0b3BfQ291bnQgICAgICAgIC1PLS1DLSAgIDEwMCAgIDEwMCAgIDAw MCAgICAtICAgIDYxCiAgNSBSZWFsbG9jYXRlZF9TZWN0b3JfQ3QgICBQTy0tQ0sgICAxMDAgICAx MDAgICAwMDUgICAgLSAgICAwCiAgNyBTZWVrX0Vycm9yX1JhdGUgICAgICAgICBQTy1SLS0gICAx MDAgICAxMDAgICAwNjcgICAgLSAgICAwCiAgOCBTZWVrX1RpbWVfUGVyZm9ybWFuY2UgICBQLVMt LS0gICAxNDYgICAxNDYgICAwMjAgICAgLSAgICAyOQogIDkgUG93ZXJfT25fSG91cnMgICAgICAg ICAgLU8tLUMtICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMTY4NgogMTAgU3Bpbl9SZXRyeV9D b3VudCAgICAgICAgUE8tLUMtICAgMTAwICAgMTAwICAgMDYwICAgIC0gICAgMAogMTIgUG93ZXJf Q3ljbGVfQ291bnQgICAgICAgLU8tLUNLICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgNjEKMTky IFBvd2VyLU9mZl9SZXRyYWN0X0NvdW50IC1PLS1DSyAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAg IDEwOQoxOTMgTG9hZF9DeWNsZV9Db3VudCAgICAgICAgLU8tLUMtICAgMTAwICAgMTAwICAgMDAw ICAgIC0gICAgMTA5CjE5NCBUZW1wZXJhdHVyZV9DZWxzaXVzICAgICAtTy0tLS0gICAxOTMgICAx OTMgICAwMDAgICAgLSAgICAzMSAoTWluL01heCAyMS8zOSkKMTk2IFJlYWxsb2NhdGVkX0V2ZW50 X0NvdW50IC1PLS1DSyAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAgIDAKMTk3IEN1cnJlbnRfUGVu ZGluZ19TZWN0b3IgIC1PLS0tSyAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAgIDAKMTk4IE9mZmxp bmVfVW5jb3JyZWN0YWJsZSAgIC0tLVItLSAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAgIDAKMTk5 IFVETUFfQ1JDX0Vycm9yX0NvdW50ICAgIC1PLVItLSAgIDIwMCAgIDIwMCAgIDAwMCAgICAtICAg IDAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHx8fHx8fF8gSyBhdXRvLWtlZXAKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHx8fHx8X18gQyBldmVudCBjb3VudAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgfHx8fF9fXyBSIGVycm9yIHJhdGUKICAgICAgICAgICAgICAgICAgICAg ICAgICAgIHx8fF9fX18gUyBzcGVlZC9wZXJmb3JtYW5jZQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgfHxfX19fXyBPIHVwZGF0ZWQgb25saW5lCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICB8X19fX19fIFAgcHJlZmFpbHVyZSB3YXJuaW5nCgpHZW5lcmFsIFB1cnBvc2UgTG9nIERpcmVj dG9yeSBWZXJzaW9uIDEKU01BUlQgICAgICAgICAgIExvZyBEaXJlY3RvcnkgVmVyc2lvbiAxIFtt dWx0aS1zZWN0b3IgbG9nIHN1cHBvcnRdCkFkZHJlc3MgICAgQWNjZXNzICBSL1cgICBTaXplICBE ZXNjcmlwdGlvbgoweDAwICAgICAgIEdQTCxTTCAgUi9PICAgICAgMSAgTG9nIERpcmVjdG9yeQow eDAxICAgICAgICAgICBTTCAgUi9PICAgICAgMSAgU3VtbWFyeSBTTUFSVCBlcnJvciBsb2cKMHgw MyAgICAgICBHUEwgICAgIFIvTyAgICAgIDEgIEV4dC4gQ29tcHJlaGVuc2l2ZSBTTUFSVCBlcnJv ciBsb2cKMHgwNCAgICAgICBHUEwgICAgIFIvTyAgICAgIDcgIERldmljZSBTdGF0aXN0aWNzIGxv ZwoweDA2ICAgICAgICAgICBTTCAgUi9PICAgICAgMSAgU01BUlQgc2VsZi10ZXN0IGxvZwoweDA3 ICAgICAgIEdQTCAgICAgUi9PICAgICAgMSAgRXh0ZW5kZWQgc2VsZi10ZXN0IGxvZwoweDA4ICAg ICAgIEdQTCAgICAgUi9PICAgICAgMiAgUG93ZXIgQ29uZGl0aW9ucyBsb2cKMHgwOSAgICAgICAg ICAgU0wgIFIvVyAgICAgIDEgIFNlbGVjdGl2ZSBzZWxmLXRlc3QgbG9nCjB4MTAgICAgICAgR1BM ICAgICBSL08gICAgICAxICBOQ1EgQ29tbWFuZCBFcnJvciBsb2cKMHgxMSAgICAgICBHUEwgICAg IFIvTyAgICAgIDEgIFNBVEEgUGh5IEV2ZW50IENvdW50ZXJzCjB4MjAgICAgICAgR1BMICAgICBS L08gICAgICAxICBTdHJlYW1pbmcgcGVyZm9ybWFuY2UgbG9nIFtPQlMtOF0KMHgyMSAgICAgICBH UEwgICAgIFIvTyAgICAgIDEgIFdyaXRlIHN0cmVhbSBlcnJvciBsb2cKMHgyMiAgICAgICBHUEwg ICAgIFIvTyAgICAgIDEgIFJlYWQgc3RyZWFtIGVycm9yIGxvZwoweDgwLTB4OWYgIEdQTCxTTCAg Ui9XICAgICAxNiAgSG9zdCB2ZW5kb3Igc3BlY2lmaWMgbG9nCjB4ZTAgICAgICAgR1BMLFNMICBS L1cgICAgICAxICBTQ1QgQ29tbWFuZC9TdGF0dXMKMHhlMSAgICAgICBHUEwsU0wgIFIvVyAgICAg IDEgIFNDVCBEYXRhIFRyYW5zZmVyCgpTTUFSVCBFeHRlbmRlZCBDb21wcmVoZW5zaXZlIEVycm9y IExvZyBWZXJzaW9uOiAxICgxIHNlY3RvcnMpCk5vIEVycm9ycyBMb2dnZWQKClNNQVJUIEV4dGVu ZGVkIFNlbGYtdGVzdCBMb2cgVmVyc2lvbjogMSAoMSBzZWN0b3JzKQpObyBzZWxmLXRlc3RzIGhh dmUgYmVlbiBsb2dnZWQuICBbVG8gcnVuIHNlbGYtdGVzdHMsIHVzZTogc21hcnRjdGwgLXRdCgpT TUFSVCBTZWxlY3RpdmUgc2VsZi10ZXN0IGxvZyBkYXRhIHN0cnVjdHVyZSByZXZpc2lvbiBudW1i ZXIgMQogU1BBTiAgTUlOX0xCQSAgTUFYX0xCQSAgQ1VSUkVOVF9URVNUX1NUQVRVUwogICAgMSAg ICAgICAgMCAgICAgICAgMCAgTm90X3Rlc3RpbmcKICAgIDIgICAgICAgIDAgICAgICAgIDAgIE5v dF90ZXN0aW5nCiAgICAzICAgICAgICAwICAgICAgICAwICBOb3RfdGVzdGluZwogICAgNCAgICAg ICAgMCAgICAgICAgMCAgTm90X3Rlc3RpbmcKICAgIDUgICAgICAgIDAgICAgICAgIDAgIE5vdF90 ZXN0aW5nClNlbGVjdGl2ZSBzZWxmLXRlc3QgZmxhZ3MgKDB4MCk6CiAgQWZ0ZXIgc2Nhbm5pbmcg c2VsZWN0ZWQgc3BhbnMsIGRvIE5PVCByZWFkLXNjYW4gcmVtYWluZGVyIG9mIGRpc2suCklmIFNl bGVjdGl2ZSBzZWxmLXRlc3QgaXMgcGVuZGluZyBvbiBwb3dlci11cCwgcmVzdW1lIGFmdGVyIDAg bWludXRlIGRlbGF5LgoKU0NUIFN0YXR1cyBWZXJzaW9uOiAgICAgICAgICAgICAgICAgIDMKU0NU IFZlcnNpb24gKHZlbmRvciBzcGVjaWZpYyk6ICAgICAgIDI1NiAoMHgwMTAwKQpTQ1QgU3VwcG9y dCBMZXZlbDogICAgICAgICAgICAgICAgICAgMQpEZXZpY2UgU3RhdGU6ICAgICAgICAgICAgICAg ICAgICAgICAgQWN0aXZlICgwKQpDdXJyZW50IFRlbXBlcmF0dXJlOiAgICAgICAgICAgICAgICAg ICAgMzEgQ2Vsc2l1cwpQb3dlciBDeWNsZSBNaW4vTWF4IFRlbXBlcmF0dXJlOiAgICAgMzEvMzMg Q2Vsc2l1cwpMaWZldGltZSAgICBNaW4vTWF4IFRlbXBlcmF0dXJlOiAgICAgMjEvMzkgQ2Vsc2l1 cwpVbmRlci9PdmVyIFRlbXBlcmF0dXJlIExpbWl0IENvdW50OiAgIDAvMApTQ1QgVGVtcGVyYXR1 cmUgSGlzdG9yeSBWZXJzaW9uOiAgICAgMgpUZW1wZXJhdHVyZSBTYW1wbGluZyBQZXJpb2Q6ICAg ICAgICAgMSBtaW51dGUKVGVtcGVyYXR1cmUgTG9nZ2luZyBJbnRlcnZhbDogICAgICAgIDEgbWlu dXRlCk1pbi9NYXggcmVjb21tZW5kZWQgVGVtcGVyYXR1cmU6ICAgICAgMC82MCBDZWxzaXVzCk1p bi9NYXggVGVtcGVyYXR1cmUgTGltaXQ6ICAgICAgICAgICAtNDAvNzAgQ2Vsc2l1cwpUZW1wZXJh dHVyZSBIaXN0b3J5IFNpemUgKEluZGV4KTogICAgMTI4ICg5MykKCkluZGV4ICAgIEVzdGltYXRl ZCBUaW1lICAgVGVtcGVyYXR1cmUgQ2Vsc2l1cwogIDk0ICAgIDIwMTMtMDYtMDQgMTE6MjEgICAg MzIgICoqKioqKioqKioqKioKIC4uLiAgICAuLiggMzAgc2tpcHBlZCkuICAgIC4uICAqKioqKioq KioqKioqCiAxMjUgICAgMjAxMy0wNi0wNCAxMTo1MiAgICAzMiAgKioqKioqKioqKioqKgogMTI2 ICAgIDIwMTMtMDYtMDQgMTE6NTMgICAgMzEgICoqKioqKioqKioqKgogLi4uICAgIC4uKCA5NCBz a2lwcGVkKS4gICAgLi4gICoqKioqKioqKioqKgogIDkzICAgIDIwMTMtMDYtMDQgMTM6MjggICAg MzEgICoqKioqKioqKioqKgoKU01BUlQgV1JJVEUgTE9HIGRvZXMgbm90IHJldHVybiBDT1VOVCBh bmQgTEJBX0xPVyByZWdpc3RlcgpTQ1QgKEdldCkgRXJyb3IgUmVjb3ZlcnkgQ29udHJvbCBjb21t YW5kIGZhaWxlZAoKRGV2aWNlIFN0YXRpc3RpY3MgKEdQIExvZyAweDA0KQpQYWdlIE9mZnNldCBT aXplICAgICAgICAgVmFsdWUgIERlc2NyaXB0aW9uCiAgMSAgPT09PT0gID0gICAgICAgICAgICAg ICAgPSAgPT0gR2VuZXJhbCBTdGF0aXN0aWNzIChyZXYgMSkgPT0KICAxICAweDAwOCAgNCAgICAg ICAgICAgICAgIDYxICBMaWZldGltZSBQb3dlci1PbiBSZXNldHMKICAxICAweDAxMCAgNCAgICAg ICAgICAgICAxNjg2ICBQb3dlci1vbiBIb3VycwogIDEgIDB4MDE4ICA2ICAgICAgIDMwNzk5NTgz NzAgIExvZ2ljYWwgU2VjdG9ycyBXcml0dGVuCiAgMSAgMHgwMjAgIDYgICAgICAgICA0NTMwMDgx NCAgTnVtYmVyIG9mIFdyaXRlIENvbW1hbmRzCiAgMSAgMHgwMjggIDYgICAgICAgIDI3NDk3NTA3 NyAgTG9naWNhbCBTZWN0b3JzIFJlYWQKICAxICAweDAzMCAgNiAgICAgICAgICA2NTk5NjI0ICBO dW1iZXIgb2YgUmVhZCBDb21tYW5kcwogIDMgID09PT09ICA9ICAgICAgICAgICAgICAgID0gID09 IFJvdGF0aW5nIE1lZGlhIFN0YXRpc3RpY3MgKHJldiAxKSA9PQogIDMgIDB4MDA4ICA0ICAgICAg ICAgICAgIDE2ODQgIFNwaW5kbGUgTW90b3IgUG93ZXItb24gSG91cnMKICAzICAweDAxMCAgNCAg ICAgICAgICAgICAxNjg0ICBIZWFkIEZseWluZyBIb3VycwogIDMgIDB4MDE4ICA0ICAgICAgICAg ICAgICAxMDkgIEhlYWQgTG9hZCBFdmVudHMKICAzICAweDAyMCAgNCAgICAgICAgICAgICAgICAw ICBOdW1iZXIgb2YgUmVhbGxvY2F0ZWQgTG9naWNhbCBTZWN0b3JzCiAgMyAgMHgwMjggIDQgICAg ICAgICAgICAgICAxNSAgUmVhZCBSZWNvdmVyeSBBdHRlbXB0cwogIDMgIDB4MDMwICA0ICAgICAg ICAgICAgICAgIDcgIE51bWJlciBvZiBNZWNoYW5pY2FsIFN0YXJ0IEZhaWx1cmVzCiAgNCAgPT09 PT0gID0gICAgICAgICAgICAgICAgPSAgPT0gR2VuZXJhbCBFcnJvcnMgU3RhdGlzdGljcyAocmV2 IDEpID09CiAgNCAgMHgwMDggIDQgICAgICAgICAgICAgICAgMCAgTnVtYmVyIG9mIFJlcG9ydGVk IFVuY29ycmVjdGFibGUgRXJyb3JzCiAgNCAgMHgwMTAgIDQgICAgICAgICAgICAgICAgMCAgUmVz ZXRzIEJldHdlZW4gQ21kIEFjY2VwdGFuY2UgYW5kIENvbXBsZXRpb24KICA1ICA9PT09PSAgPSAg ICAgICAgICAgICAgICA9ICA9PSBUZW1wZXJhdHVyZSBTdGF0aXN0aWNzIChyZXYgMSkgPT0KICA1 ICAweDAwOCAgMSAgICAgICAgICAgICAgIDMxICBDdXJyZW50IFRlbXBlcmF0dXJlCiAgNSAgMHgw MTAgIDEgICAgICAgICAgICAgICAzMX4gQXZlcmFnZSBTaG9ydCBUZXJtIFRlbXBlcmF0dXJlCiAg NSAgMHgwMTggIDEgICAgICAgICAgICAgICAzMH4gQXZlcmFnZSBMb25nIFRlcm0gVGVtcGVyYXR1 cmUKICA1ICAweDAyMCAgMSAgICAgICAgICAgICAgIDM5ICBIaWdoZXN0IFRlbXBlcmF0dXJlCiAg NSAgMHgwMjggIDEgICAgICAgICAgICAgICAyMSAgTG93ZXN0IFRlbXBlcmF0dXJlCiAgNSAgMHgw MzAgIDEgICAgICAgICAgICAgICAzNX4gSGlnaGVzdCBBdmVyYWdlIFNob3J0IFRlcm0gVGVtcGVy YXR1cmUKICA1ICAweDAzOCAgMSAgICAgICAgICAgICAgIDI1fiBMb3dlc3QgQXZlcmFnZSBTaG9y dCBUZXJtIFRlbXBlcmF0dXJlCiAgNSAgMHgwNDAgIDEgICAgICAgICAgICAgICAzM34gSGlnaGVz dCBBdmVyYWdlIExvbmcgVGVybSBUZW1wZXJhdHVyZQogIDUgIDB4MDQ4ICAxICAgICAgICAgICAg ICAgMjV+IExvd2VzdCBBdmVyYWdlIExvbmcgVGVybSBUZW1wZXJhdHVyZQogIDUgIDB4MDUwICA0 ICAgICAgICAgICAgICAgIDAgIFRpbWUgaW4gT3Zlci1UZW1wZXJhdHVyZQogIDUgIDB4MDU4ICAx ICAgICAgICAgICAgICAgNjAgIFNwZWNpZmllZCBNYXhpbXVtIE9wZXJhdGluZyBUZW1wZXJhdHVy ZQogIDUgIDB4MDYwICA0ICAgICAgICAgICAgICAgIDAgIFRpbWUgaW4gVW5kZXItVGVtcGVyYXR1 cmUKICA1ICAweDA2OCAgMSAgICAgICAgICAgICAgICAwICBTcGVjaWZpZWQgTWluaW11bSBPcGVy YXRpbmcgVGVtcGVyYXR1cmUKICA2ICA9PT09PSAgPSAgICAgICAgICAgICAgICA9ICA9PSBUcmFu c3BvcnQgU3RhdGlzdGljcyAocmV2IDEpID09CiAgNiAgMHgwMDggIDQgICAgICAgICAgICAgNTEy NyAgTnVtYmVyIG9mIEhhcmR3YXJlIFJlc2V0cwogIDYgIDB4MDEwICA0ICAgICAgICAgICAgICAz NzUgIE51bWJlciBvZiBBU1IgRXZlbnRzCiAgNiAgMHgwMTggIDQgICAgICAgICAgICAgICAgMCAg TnVtYmVyIG9mIEludGVyZmFjZSBDUkMgRXJyb3JzCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHxfIH4gbm9ybWFsaXplZCB2YWx1ZQoKU0FUQSBQaHkgRXZlbnQgQ291bnRlcnMgKEdQIExv ZyAweDExKQpJRCAgICAgIFNpemUgICAgIFZhbHVlICBEZXNjcmlwdGlvbgoweDAwMDEgIDIgICAg ICAgICAgICAwICBDb21tYW5kIGZhaWxlZCBkdWUgdG8gSUNSQyBlcnJvcgoweDAwMDIgIDIgICAg ICAgICAgICAwICBSX0VSUiByZXNwb25zZSBmb3IgZGF0YSBGSVMKMHgwMDAzICAyICAgICAgICAg ICAgMCAgUl9FUlIgcmVzcG9uc2UgZm9yIGRldmljZS10by1ob3N0IGRhdGEgRklTCjB4MDAwNCAg MiAgICAgICAgICAgIDAgIFJfRVJSIHJlc3BvbnNlIGZvciBob3N0LXRvLWRldmljZSBkYXRhIEZJ UwoweDAwMDUgIDIgICAgICAgICAgICAwICBSX0VSUiByZXNwb25zZSBmb3Igbm9uLWRhdGEgRklT CjB4MDAwNiAgMiAgICAgICAgICAgIDAgIFJfRVJSIHJlc3BvbnNlIGZvciBkZXZpY2UtdG8taG9z dCBub24tZGF0YSBGSVMKMHgwMDA3ICAyICAgICAgICAgICAgMCAgUl9FUlIgcmVzcG9uc2UgZm9y IGhvc3QtdG8tZGV2aWNlIG5vbi1kYXRhIEZJUwoweDAwMDkgIDIgICAgICAgICAgIDExICBUcmFu c2l0aW9uIGZyb20gZHJpdmUgUGh5UmR5IHRvIGRyaXZlIFBoeU5SZHkKMHgwMDBhICAyICAgICAg ICAgICAxMSAgRGV2aWNlLXRvLWhvc3QgcmVnaXN0ZXIgRklTZXMgc2VudCBkdWUgdG8gYSBDT01S RVNFVAoweDAwMGIgIDIgICAgICAgICAgICAwICBDUkMgZXJyb3JzIHdpdGhpbiBob3N0LXRvLWRl dmljZSBGSVMKMHgwMDBkICAyICAgICAgICAgICAgMCAgTm9uLUNSQyBlcnJvcnMgd2l0aGluIGhv c3QtdG8tZGV2aWNlIEZJUwoK ------=_Part_201016_445496304.1370608667459 Content-Type: text/plain; name=da3.txt Content-Disposition: attachment; filename=da3.txt Content-Transfer-Encoding: base64 c21hcnRjdGwgNi4xIDIwMTMtMDMtMTYgcjM4MDAgW0ZyZWVCU0QgOS4xLVJFTEVBU0UgYW1kNjRd IChsb2NhbCBidWlsZCkKQ29weXJpZ2h0IChDKSAyMDAyLTEzLCBCcnVjZSBBbGxlbiwgQ2hyaXN0 aWFuIEZyYW5rZSwgd3d3LnNtYXJ0bW9udG9vbHMub3JnCgo9PT0gU1RBUlQgT0YgSU5GT1JNQVRJ T04gU0VDVElPTiA9PT0KTW9kZWwgRmFtaWx5OiAgICAgSGl0YWNoaSBEZXNrc3RhciA1SzMwMDAK RGV2aWNlIE1vZGVsOiAgICAgSGl0YWNoaSBIRFM1QzMwMjBBTEE2MzIKU2VyaWFsIE51bWJlcjog ICAgTUw0MjIwRjMxODdTTUsKTFUgV1dOIERldmljZSBJZDogNSAwMDBjY2EgMzY5ZDFkNzljCkZp cm13YXJlIFZlcnNpb246IE1MNk9BNTgwClVzZXIgQ2FwYWNpdHk6ICAgIDIsMDAwLDM5OCw5MzQs MDE2IGJ5dGVzIFsyLjAwIFRCXQpTZWN0b3IgU2l6ZTogICAgICA1MTIgYnl0ZXMgbG9naWNhbC9w aHlzaWNhbApSb3RhdGlvbiBSYXRlOiAgICA1OTQwIHJwbQpEZXZpY2UgaXM6ICAgICAgICBJbiBz bWFydGN0bCBkYXRhYmFzZSBbZm9yIGRldGFpbHMgdXNlOiAtUCBzaG93XQpBVEEgVmVyc2lvbiBp czogICBBVEE4LUFDUyBUMTMvMTY5OS1EIHJldmlzaW9uIDQKU0FUQSBWZXJzaW9uIGlzOiAgU0FU QSAyLjYsIDYuMCBHYi9zIChjdXJyZW50OiAxLjUgR2IvcykKTG9jYWwgVGltZSBpczogICAgVHVl IEp1biAgNCAxMzoyOTozMCAyMDEzIENFU1QKU01BUlQgc3VwcG9ydCBpczogQXZhaWxhYmxlIC0g ZGV2aWNlIGhhcyBTTUFSVCBjYXBhYmlsaXR5LgpTTUFSVCBzdXBwb3J0IGlzOiBFbmFibGVkCkFB TSBmZWF0dXJlIGlzOiAgIFVuYXZhaWxhYmxlCkFQTSBmZWF0dXJlIGlzOiAgIERpc2FibGVkClJk IGxvb2stYWhlYWQgaXM6IEVuYWJsZWQKV3JpdGUgY2FjaGUgaXM6ICAgRW5hYmxlZApBVEEgU2Vj dXJpdHkgaXM6ICBEaXNhYmxlZCwgTk9UIEZST1pFTiBbU0VDMV0KCj09PSBTVEFSVCBPRiBSRUFE IFNNQVJUIERBVEEgU0VDVElPTiA9PT0KU01BUlQgb3ZlcmFsbC1oZWFsdGggc2VsZi1hc3Nlc3Nt ZW50IHRlc3QgcmVzdWx0OiBQQVNTRUQKCkdlbmVyYWwgU01BUlQgVmFsdWVzOgpPZmZsaW5lIGRh dGEgY29sbGVjdGlvbiBzdGF0dXM6ICAoMHg4MikJT2ZmbGluZSBkYXRhIGNvbGxlY3Rpb24gYWN0 aXZpdHkKCQkJCQl3YXMgY29tcGxldGVkIHdpdGhvdXQgZXJyb3IuCgkJCQkJQXV0byBPZmZsaW5l IERhdGEgQ29sbGVjdGlvbjogRW5hYmxlZC4KU2VsZi10ZXN0IGV4ZWN1dGlvbiBzdGF0dXM6ICAg ICAgKCAgIDApCVRoZSBwcmV2aW91cyBzZWxmLXRlc3Qgcm91dGluZSBjb21wbGV0ZWQKCQkJCQl3 aXRob3V0IGVycm9yIG9yIG5vIHNlbGYtdGVzdCBoYXMgZXZlciAKCQkJCQliZWVuIHJ1bi4KVG90 YWwgdGltZSB0byBjb21wbGV0ZSBPZmZsaW5lIApkYXRhIGNvbGxlY3Rpb246IAkJKDIzOTg1KSBz ZWNvbmRzLgpPZmZsaW5lIGRhdGEgY29sbGVjdGlvbgpjYXBhYmlsaXRpZXM6IAkJCSAoMHg1Yikg U01BUlQgZXhlY3V0ZSBPZmZsaW5lIGltbWVkaWF0ZS4KCQkJCQlBdXRvIE9mZmxpbmUgZGF0YSBj b2xsZWN0aW9uIG9uL29mZiBzdXBwb3J0LgoJCQkJCVN1c3BlbmQgT2ZmbGluZSBjb2xsZWN0aW9u IHVwb24gbmV3CgkJCQkJY29tbWFuZC4KCQkJCQlPZmZsaW5lIHN1cmZhY2Ugc2NhbiBzdXBwb3J0 ZWQuCgkJCQkJU2VsZi10ZXN0IHN1cHBvcnRlZC4KCQkJCQlObyBDb252ZXlhbmNlIFNlbGYtdGVz dCBzdXBwb3J0ZWQuCgkJCQkJU2VsZWN0aXZlIFNlbGYtdGVzdCBzdXBwb3J0ZWQuClNNQVJUIGNh cGFiaWxpdGllczogICAgICAgICAgICAoMHgwMDAzKQlTYXZlcyBTTUFSVCBkYXRhIGJlZm9yZSBl bnRlcmluZwoJCQkJCXBvd2VyLXNhdmluZyBtb2RlLgoJCQkJCVN1cHBvcnRzIFNNQVJUIGF1dG8g c2F2ZSB0aW1lci4KRXJyb3IgbG9nZ2luZyBjYXBhYmlsaXR5OiAgICAgICAgKDB4MDEpCUVycm9y IGxvZ2dpbmcgc3VwcG9ydGVkLgoJCQkJCUdlbmVyYWwgUHVycG9zZSBMb2dnaW5nIHN1cHBvcnRl ZC4KU2hvcnQgc2VsZi10ZXN0IHJvdXRpbmUgCnJlY29tbWVuZGVkIHBvbGxpbmcgdGltZTogCSAo ICAgMSkgbWludXRlcy4KRXh0ZW5kZWQgc2VsZi10ZXN0IHJvdXRpbmUKcmVjb21tZW5kZWQgcG9s bGluZyB0aW1lOiAJICggNDAwKSBtaW51dGVzLgpTQ1QgY2FwYWJpbGl0aWVzOiAJICAgICAgICgw eDAwM2QpCVNDVCBTdGF0dXMgc3VwcG9ydGVkLgoJCQkJCVNDVCBFcnJvciBSZWNvdmVyeSBDb250 cm9sIHN1cHBvcnRlZC4KCQkJCQlTQ1QgRmVhdHVyZSBDb250cm9sIHN1cHBvcnRlZC4KCQkJCQlT Q1QgRGF0YSBUYWJsZSBzdXBwb3J0ZWQuCgpTTUFSVCBBdHRyaWJ1dGVzIERhdGEgU3RydWN0dXJl IHJldmlzaW9uIG51bWJlcjogMTYKVmVuZG9yIFNwZWNpZmljIFNNQVJUIEF0dHJpYnV0ZXMgd2l0 aCBUaHJlc2hvbGRzOgpJRCMgQVRUUklCVVRFX05BTUUgICAgICAgICAgRkxBR1MgICAgVkFMVUUg V09SU1QgVEhSRVNIIEZBSUwgUkFXX1ZBTFVFCiAgMSBSYXdfUmVhZF9FcnJvcl9SYXRlICAgICBQ Ty1SLS0gICAxMDAgICAxMDAgICAwMTYgICAgLSAgICAwCiAgMiBUaHJvdWdocHV0X1BlcmZvcm1h bmNlICBQLVMtLS0gICAxMzEgICAxMzEgICAwNTQgICAgLSAgICAxMTIKICAzIFNwaW5fVXBfVGlt ZSAgICAgICAgICAgIFBPUy0tLSAgIDEzNSAgIDEzNSAgIDAyNCAgICAtICAgIDQwNyAoQXZlcmFn ZSA0MDcpCiAgNCBTdGFydF9TdG9wX0NvdW50ICAgICAgICAtTy0tQy0gICAxMDAgICAxMDAgICAw MDAgICAgLSAgICA4NAogIDUgUmVhbGxvY2F0ZWRfU2VjdG9yX0N0ICAgUE8tLUNLICAgMTAwICAg MTAwICAgMDA1ICAgIC0gICAgMAogIDcgU2Vla19FcnJvcl9SYXRlICAgICAgICAgUE8tUi0tICAg MTAwICAgMTAwICAgMDY3ICAgIC0gICAgMAogIDggU2Vla19UaW1lX1BlcmZvcm1hbmNlICAgUC1T LS0tICAgMTQ4ICAgMTQ4ICAgMDIwICAgIC0gICAgMjgKICA5IFBvd2VyX09uX0hvdXJzICAgICAg ICAgIC1PLS1DLSAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAgIDIzMTQKIDEwIFNwaW5fUmV0cnlf Q291bnQgICAgICAgIFBPLS1DLSAgIDEwMCAgIDEwMCAgIDA2MCAgICAtICAgIDAKIDEyIFBvd2Vy X0N5Y2xlX0NvdW50ICAgICAgIC1PLS1DSyAgIDEwMCAgIDEwMCAgIDAwMCAgICAtICAgIDg0CjE5 MiBQb3dlci1PZmZfUmV0cmFjdF9Db3VudCAtTy0tQ0sgICAxMDAgICAxMDAgICAwMDAgICAgLSAg ICA4OQoxOTMgTG9hZF9DeWNsZV9Db3VudCAgICAgICAgLU8tLUMtICAgMTAwICAgMTAwICAgMDAw ICAgIC0gICAgODkKMTk0IFRlbXBlcmF0dXJlX0NlbHNpdXMgICAgIC1PLS0tLSAgIDIwMCAgIDIw MCAgIDAwMCAgICAtICAgIDMwIChNaW4vTWF4IDE3LzM2KQoxOTYgUmVhbGxvY2F0ZWRfRXZlbnRf Q291bnQgLU8tLUNLICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMAoxOTcgQ3VycmVudF9QZW5k aW5nX1NlY3RvciAgLU8tLS1LICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMAoxOTggT2ZmbGlu ZV9VbmNvcnJlY3RhYmxlICAgLS0tUi0tICAgMTAwICAgMTAwICAgMDAwICAgIC0gICAgMAoxOTkg VURNQV9DUkNfRXJyb3JfQ291bnQgICAgLU8tUi0tICAgMjAwICAgMjAwICAgMDAwICAgIC0gICAg MAogICAgICAgICAgICAgICAgICAgICAgICAgICAgfHx8fHx8XyBLIGF1dG8ta2VlcAogICAgICAg ICAgICAgICAgICAgICAgICAgICAgfHx8fHxfXyBDIGV2ZW50IGNvdW50CiAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8fHx8X19fIFIgZXJyb3IgcmF0ZQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgfHx8X19fXyBTIHNwZWVkL3BlcmZvcm1hbmNlCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICB8fF9fX19fIE8gdXBkYXRlZCBvbmxpbmUKICAgICAgICAgICAgICAgICAgICAgICAgICAg IHxfX19fX18gUCBwcmVmYWlsdXJlIHdhcm5pbmcKCkdlbmVyYWwgUHVycG9zZSBMb2cgRGlyZWN0 b3J5IFZlcnNpb24gMQpTTUFSVCAgICAgICAgICAgTG9nIERpcmVjdG9yeSBWZXJzaW9uIDEgW211 bHRpLXNlY3RvciBsb2cgc3VwcG9ydF0KQWRkcmVzcyAgICBBY2Nlc3MgIFIvVyAgIFNpemUgIERl c2NyaXB0aW9uCjB4MDAgICAgICAgR1BMLFNMICBSL08gICAgICAxICBMb2cgRGlyZWN0b3J5CjB4 MDEgICAgICAgICAgIFNMICBSL08gICAgICAxICBTdW1tYXJ5IFNNQVJUIGVycm9yIGxvZwoweDAz ICAgICAgIEdQTCAgICAgUi9PICAgICAgMSAgRXh0LiBDb21wcmVoZW5zaXZlIFNNQVJUIGVycm9y IGxvZwoweDA0ICAgICAgIEdQTCAgICAgUi9PICAgICAgNyAgRGV2aWNlIFN0YXRpc3RpY3MgbG9n CjB4MDYgICAgICAgICAgIFNMICBSL08gICAgICAxICBTTUFSVCBzZWxmLXRlc3QgbG9nCjB4MDcg ICAgICAgR1BMICAgICBSL08gICAgICAxICBFeHRlbmRlZCBzZWxmLXRlc3QgbG9nCjB4MDggICAg ICAgR1BMICAgICBSL08gICAgICAxICBQb3dlciBDb25kaXRpb25zIGxvZwoweDA5ICAgICAgICAg ICBTTCAgUi9XICAgICAgMSAgU2VsZWN0aXZlIHNlbGYtdGVzdCBsb2cKMHgxMCAgICAgICBHUEwg ICAgIFIvTyAgICAgIDEgIE5DUSBDb21tYW5kIEVycm9yIGxvZwoweDExICAgICAgIEdQTCAgICAg Ui9PICAgICAgMSAgU0FUQSBQaHkgRXZlbnQgQ291bnRlcnMKMHgyMCAgICAgICBHUEwgICAgIFIv TyAgICAgIDEgIFN0cmVhbWluZyBwZXJmb3JtYW5jZSBsb2cgW09CUy04XQoweDIxICAgICAgIEdQ TCAgICAgUi9PICAgICAgMSAgV3JpdGUgc3RyZWFtIGVycm9yIGxvZwoweDIyICAgICAgIEdQTCAg ICAgUi9PICAgICAgMSAgUmVhZCBzdHJlYW0gZXJyb3IgbG9nCjB4ODAtMHg5ZiAgR1BMLFNMICBS L1cgICAgIDE2ICBIb3N0IHZlbmRvciBzcGVjaWZpYyBsb2cKMHhlMCAgICAgICBHUEwsU0wgIFIv VyAgICAgIDEgIFNDVCBDb21tYW5kL1N0YXR1cwoweGUxICAgICAgIEdQTCxTTCAgUi9XICAgICAg MSAgU0NUIERhdGEgVHJhbnNmZXIKClNNQVJUIEV4dGVuZGVkIENvbXByZWhlbnNpdmUgRXJyb3Ig TG9nIFZlcnNpb246IDEgKDEgc2VjdG9ycykKTm8gRXJyb3JzIExvZ2dlZAoKU01BUlQgRXh0ZW5k ZWQgU2VsZi10ZXN0IExvZyBWZXJzaW9uOiAxICgxIHNlY3RvcnMpCk5vIHNlbGYtdGVzdHMgaGF2 ZSBiZWVuIGxvZ2dlZC4gIFtUbyBydW4gc2VsZi10ZXN0cywgdXNlOiBzbWFydGN0bCAtdF0KClNN QVJUIFNlbGVjdGl2ZSBzZWxmLXRlc3QgbG9nIGRhdGEgc3RydWN0dXJlIHJldmlzaW9uIG51bWJl ciAxCiBTUEFOICBNSU5fTEJBICBNQVhfTEJBICBDVVJSRU5UX1RFU1RfU1RBVFVTCiAgICAxICAg ICAgICAwICAgICAgICAwICBOb3RfdGVzdGluZwogICAgMiAgICAgICAgMCAgICAgICAgMCAgTm90 X3Rlc3RpbmcKICAgIDMgICAgICAgIDAgICAgICAgIDAgIE5vdF90ZXN0aW5nCiAgICA0ICAgICAg ICAwICAgICAgICAwICBOb3RfdGVzdGluZwogICAgNSAgICAgICAgMCAgICAgICAgMCAgTm90X3Rl c3RpbmcKU2VsZWN0aXZlIHNlbGYtdGVzdCBmbGFncyAoMHgwKToKICBBZnRlciBzY2FubmluZyBz ZWxlY3RlZCBzcGFucywgZG8gTk9UIHJlYWQtc2NhbiByZW1haW5kZXIgb2YgZGlzay4KSWYgU2Vs ZWN0aXZlIHNlbGYtdGVzdCBpcyBwZW5kaW5nIG9uIHBvd2VyLXVwLCByZXN1bWUgYWZ0ZXIgMCBt aW51dGUgZGVsYXkuCgpTQ1QgU3RhdHVzIFZlcnNpb246ICAgICAgICAgICAgICAgICAgMwpTQ1Qg VmVyc2lvbiAodmVuZG9yIHNwZWNpZmljKTogICAgICAgMjU2ICgweDAxMDApClNDVCBTdXBwb3J0 IExldmVsOiAgICAgICAgICAgICAgICAgICAxCkRldmljZSBTdGF0ZTogICAgICAgICAgICAgICAg ICAgICAgICBBY3RpdmUgKDApCkN1cnJlbnQgVGVtcGVyYXR1cmU6ICAgICAgICAgICAgICAgICAg ICAzMCBDZWxzaXVzClBvd2VyIEN5Y2xlIE1pbi9NYXggVGVtcGVyYXR1cmU6ICAgICAyOS8zMSBD ZWxzaXVzCkxpZmV0aW1lICAgIE1pbi9NYXggVGVtcGVyYXR1cmU6ICAgICAxNy8zNiBDZWxzaXVz ClVuZGVyL092ZXIgVGVtcGVyYXR1cmUgTGltaXQgQ291bnQ6ICAgMC8wClNDVCBUZW1wZXJhdHVy ZSBIaXN0b3J5IFZlcnNpb246ICAgICAyClRlbXBlcmF0dXJlIFNhbXBsaW5nIFBlcmlvZDogICAg ICAgICAxIG1pbnV0ZQpUZW1wZXJhdHVyZSBMb2dnaW5nIEludGVydmFsOiAgICAgICAgMSBtaW51 dGUKTWluL01heCByZWNvbW1lbmRlZCBUZW1wZXJhdHVyZTogICAgICAwLzYwIENlbHNpdXMKTWlu L01heCBUZW1wZXJhdHVyZSBMaW1pdDogICAgICAgICAgIC00MC83MCBDZWxzaXVzClRlbXBlcmF0 dXJlIEhpc3RvcnkgU2l6ZSAoSW5kZXgpOiAgICAxMjggKDEyMCkKCkluZGV4ICAgIEVzdGltYXRl ZCBUaW1lICAgVGVtcGVyYXR1cmUgQ2Vsc2l1cwogMTIxICAgIDIwMTMtMDYtMDQgMTE6MjIgICAg MzEgICoqKioqKioqKioqKgogLi4uICAgIC4uKCA2NCBza2lwcGVkKS4gICAgLi4gICoqKioqKioq KioqKgogIDU4ICAgIDIwMTMtMDYtMDQgMTI6MjcgICAgMzEgICoqKioqKioqKioqKgogIDU5ICAg IDIwMTMtMDYtMDQgMTI6MjggICAgMzAgICoqKioqKioqKioqCiAuLi4gICAgLi4oIDYwIHNraXBw ZWQpLiAgICAuLiAgKioqKioqKioqKioKIDEyMCAgICAyMDEzLTA2LTA0IDEzOjI5ICAgIDMwICAq KioqKioqKioqKgoKU01BUlQgV1JJVEUgTE9HIGRvZXMgbm90IHJldHVybiBDT1VOVCBhbmQgTEJB X0xPVyByZWdpc3RlcgpTQ1QgKEdldCkgRXJyb3IgUmVjb3ZlcnkgQ29udHJvbCBjb21tYW5kIGZh aWxlZAoKRGV2aWNlIFN0YXRpc3RpY3MgKEdQIExvZyAweDA0KQpQYWdlIE9mZnNldCBTaXplICAg ICAgICAgVmFsdWUgIERlc2NyaXB0aW9uCiAgMSAgPT09PT0gID0gICAgICAgICAgICAgICAgPSAg PT0gR2VuZXJhbCBTdGF0aXN0aWNzIChyZXYgMSkgPT0KICAxICAweDAwOCAgNCAgICAgICAgICAg ICAgIDg0ICBMaWZldGltZSBQb3dlci1PbiBSZXNldHMKICAxICAweDAxMCAgNCAgICAgICAgICAg ICAyMzE0ICBQb3dlci1vbiBIb3VycwogIDEgIDB4MDE4ICA2ICAgICAgIDQzMDE3NDI5NDMgIExv Z2ljYWwgU2VjdG9ycyBXcml0dGVuCiAgMSAgMHgwMjAgIDYgICAgICAgICA1MTQyOTYyMiAgTnVt YmVyIG9mIFdyaXRlIENvbW1hbmRzCiAgMSAgMHgwMjggIDYgICAgIDI0OTY5NjY5MjA0OCAgTG9n aWNhbCBTZWN0b3JzIFJlYWQKICAxICAweDAzMCAgNiAgICAgICAgICA4MzM3MDk5ICBOdW1iZXIg b2YgUmVhZCBDb21tYW5kcwogIDMgID09PT09ICA9ICAgICAgICAgICAgICAgID0gID09IFJvdGF0 aW5nIE1lZGlhIFN0YXRpc3RpY3MgKHJldiAxKSA9PQogIDMgIDB4MDA4ICA0ICAgICAgICAgICAg IDIzMTEgIFNwaW5kbGUgTW90b3IgUG93ZXItb24gSG91cnMKICAzICAweDAxMCAgNCAgICAgICAg ICAgICAyMzExICBIZWFkIEZseWluZyBIb3VycwogIDMgIDB4MDE4ICA0ICAgICAgICAgICAgICAg ODkgIEhlYWQgTG9hZCBFdmVudHMKICAzICAweDAyMCAgNCAgICAgICAgICAgICAgICAwICBOdW1i ZXIgb2YgUmVhbGxvY2F0ZWQgTG9naWNhbCBTZWN0b3JzCiAgMyAgMHgwMjggIDQgICAgICAgICAg ICAgICAxMSAgUmVhZCBSZWNvdmVyeSBBdHRlbXB0cwogIDMgIDB4MDMwICA0ICAgICAgICAgICAg ICAgIDcgIE51bWJlciBvZiBNZWNoYW5pY2FsIFN0YXJ0IEZhaWx1cmVzCiAgNCAgPT09PT0gID0g ICAgICAgICAgICAgICAgPSAgPT0gR2VuZXJhbCBFcnJvcnMgU3RhdGlzdGljcyAocmV2IDEpID09 CiAgNCAgMHgwMDggIDQgICAgICAgICAgICAgICAgMCAgTnVtYmVyIG9mIFJlcG9ydGVkIFVuY29y cmVjdGFibGUgRXJyb3JzCiAgNCAgMHgwMTAgIDQgICAgICAgICAgICAgICAgMCAgUmVzZXRzIEJl dHdlZW4gQ21kIEFjY2VwdGFuY2UgYW5kIENvbXBsZXRpb24KICA1ICA9PT09PSAgPSAgICAgICAg ICAgICAgICA9ICA9PSBUZW1wZXJhdHVyZSBTdGF0aXN0aWNzIChyZXYgMSkgPT0KICA1ICAweDAw OCAgMSAgICAgICAgICAgICAgIDMwICBDdXJyZW50IFRlbXBlcmF0dXJlCiAgNSAgMHgwMTAgIDEg ICAgICAgICAgICAgICAzMH4gQXZlcmFnZSBTaG9ydCBUZXJtIFRlbXBlcmF0dXJlCiAgNSAgMHgw MTggIDEgICAgICAgICAgICAgICAyOX4gQXZlcmFnZSBMb25nIFRlcm0gVGVtcGVyYXR1cmUKICA1 ICAweDAyMCAgMSAgICAgICAgICAgICAgIDM2ICBIaWdoZXN0IFRlbXBlcmF0dXJlCiAgNSAgMHgw MjggIDEgICAgICAgICAgICAgICAxNyAgTG93ZXN0IFRlbXBlcmF0dXJlCiAgNSAgMHgwMzAgIDEg ICAgICAgICAgICAgICAzNH4gSGlnaGVzdCBBdmVyYWdlIFNob3J0IFRlcm0gVGVtcGVyYXR1cmUK ICA1ICAweDAzOCAgMSAgICAgICAgICAgICAgIDIwfiBMb3dlc3QgQXZlcmFnZSBTaG9ydCBUZXJt IFRlbXBlcmF0dXJlCiAgNSAgMHgwNDAgIDEgICAgICAgICAgICAgICAzMX4gSGlnaGVzdCBBdmVy YWdlIExvbmcgVGVybSBUZW1wZXJhdHVyZQogIDUgIDB4MDQ4ICAxICAgICAgICAgICAgICAgMjN+ IExvd2VzdCBBdmVyYWdlIExvbmcgVGVybSBUZW1wZXJhdHVyZQogIDUgIDB4MDUwICA0ICAgICAg ICAgICAgICAgIDAgIFRpbWUgaW4gT3Zlci1UZW1wZXJhdHVyZQogIDUgIDB4MDU4ICAxICAgICAg ICAgICAgICAgNjAgIFNwZWNpZmllZCBNYXhpbXVtIE9wZXJhdGluZyBUZW1wZXJhdHVyZQogIDUg IDB4MDYwICA0ICAgICAgICAgICAgICAgIDAgIFRpbWUgaW4gVW5kZXItVGVtcGVyYXR1cmUKICA1 ICAweDA2OCAgMSAgICAgICAgICAgICAgICAwICBTcGVjaWZpZWQgTWluaW11bSBPcGVyYXRpbmcg VGVtcGVyYXR1cmUKICA2ICA9PT09PSAgPSAgICAgICAgICAgICAgICA9ICA9PSBUcmFuc3BvcnQg U3RhdGlzdGljcyAocmV2IDEpID09CiAgNiAgMHgwMDggIDQgICAgICAgICAgICAgMTM1NCAgTnVt YmVyIG9mIEhhcmR3YXJlIFJlc2V0cwogIDYgIDB4MDEwICA0ICAgICAgICAgICAgICAyNDQgIE51 bWJlciBvZiBBU1IgRXZlbnRzCiAgNiAgMHgwMTggIDQgICAgICAgICAgICAgICAgMCAgTnVtYmVy IG9mIEludGVyZmFjZSBDUkMgRXJyb3JzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHxf IH4gbm9ybWFsaXplZCB2YWx1ZQoKU0FUQSBQaHkgRXZlbnQgQ291bnRlcnMgKEdQIExvZyAweDEx KQpJRCAgICAgIFNpemUgICAgIFZhbHVlICBEZXNjcmlwdGlvbgoweDAwMDEgIDIgICAgICAgICAg ICAwICBDb21tYW5kIGZhaWxlZCBkdWUgdG8gSUNSQyBlcnJvcgoweDAwMDIgIDIgICAgICAgICAg ICAwICBSX0VSUiByZXNwb25zZSBmb3IgZGF0YSBGSVMKMHgwMDAzICAyICAgICAgICAgICAgMCAg Ul9FUlIgcmVzcG9uc2UgZm9yIGRldmljZS10by1ob3N0IGRhdGEgRklTCjB4MDAwNCAgMiAgICAg ICAgICAgIDAgIFJfRVJSIHJlc3BvbnNlIGZvciBob3N0LXRvLWRldmljZSBkYXRhIEZJUwoweDAw MDUgIDIgICAgICAgICAgICAwICBSX0VSUiByZXNwb25zZSBmb3Igbm9uLWRhdGEgRklTCjB4MDAw NiAgMiAgICAgICAgICAgIDAgIFJfRVJSIHJlc3BvbnNlIGZvciBkZXZpY2UtdG8taG9zdCBub24t ZGF0YSBGSVMKMHgwMDA3ICAyICAgICAgICAgICAgMCAgUl9FUlIgcmVzcG9uc2UgZm9yIGhvc3Qt dG8tZGV2aWNlIG5vbi1kYXRhIEZJUwoweDAwMDkgIDIgICAgICAgICAgICA1ICBUcmFuc2l0aW9u IGZyb20gZHJpdmUgUGh5UmR5IHRvIGRyaXZlIFBoeU5SZHkKMHgwMDBhICAyICAgICAgICAgICAg NSAgRGV2aWNlLXRvLWhvc3QgcmVnaXN0ZXIgRklTZXMgc2VudCBkdWUgdG8gYSBDT01SRVNFVAow eDAwMGIgIDIgICAgICAgICAgICAwICBDUkMgZXJyb3JzIHdpdGhpbiBob3N0LXRvLWRldmljZSBG SVMKMHgwMDBkICAyICAgICAgICAgICAgMCAgTm9uLUNSQyBlcnJvcnMgd2l0aGluIGhvc3QtdG8t ZGV2aWNlIEZJUwoK ------=_Part_201016_445496304.1370608667459-- From owner-freebsd-stable@FreeBSD.ORG Fri Jun 7 16:51:17 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CB623C34 for ; Fri, 7 Jun 2013 16:51:17 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 5B5F31A24 for ; Fri, 7 Jun 2013 16:51:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id r57GoIA9080159; Fri, 7 Jun 2013 20:50:18 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Fri, 7 Jun 2013 20:50:18 +0400 (MSK) From: Dmitry Morozovsky To: "Pascal Braun, Continum" Subject: Re: ZFS crashing while zfs recv in progress In-Reply-To: <1290657146.201020.1370608667469.JavaMail.root@continum.net> Message-ID: References: <1290657146.201020.1370608667469.JavaMail.root@continum.net> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Fri, 07 Jun 2013 20:50:19 +0400 (MSK) Cc: Jeremy Chadwick , freebsd-stable@freebsd.org, Ronald Klop X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2013 16:51:17 -0000 On Fri, 7 Jun 2013, Pascal Braun, Continum wrote: [snip] > > If you could put a swap disk on a dedicated controller (and no other > > disks on it), that would be ideal. Please do not use USB for this > > task > > (the USB stack may introduce its own set of complexities pertaining > > to > > interrupt usage). > > I can't easily do this in the current setup. I would have to recreate the primary pool differently. Don't you have a place for (possibly, 2.5") disk inside a case, so you can connect it directly to mobo AHCI? -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------ From owner-freebsd-stable@FreeBSD.ORG Fri Jun 7 17:27:14 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B45D174F for ; Fri, 7 Jun 2013 17:27:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 932521BD4 for ; Fri, 7 Jun 2013 17:27:14 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id CAC6EB946; Fri, 7 Jun 2013 13:27:13 -0400 (EDT) From: John Baldwin To: Julian Stecklina Subject: Re: Reproducable Infiniband panic Date: Fri, 7 Jun 2013 12:06:52 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <51B07705.207@os.inf.tu-dresden.de> <201306061457.52278.jhb@freebsd.org> <51B1A2D6.4030901@os.inf.tu-dresden.de> In-Reply-To: <51B1A2D6.4030901@os.inf.tu-dresden.de> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201306071206.52994.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 07 Jun 2013 13:27:13 -0400 (EDT) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2013 17:27:14 -0000 On Friday, June 07, 2013 5:07:34 am Julian Stecklina wrote: > On 06/06/2013 08:57 PM, John Baldwin wrote: > > On Thursday, June 06, 2013 9:54:35 am Andriy Gapon wrote: > [...] > >> The problem seems to be in incorrect interaction between devfs_close_f and > >> linux_file_dtor. The latter expects curthread->td_fpop to have a valid reasonable > >> value. But the former sets curthread->td_fpop to fp only around vnops.fo_close() > >> call and then restores it back to some (what?) previous value before calling > >> devfs_fpdrop->devfs_destroy_cdevpriv. In this case the previous value is NULL. > > > > It is normally NULL in this case. Why does linux_file_dtor even look at > > td_fpop? > > > > Ah. I think it should not do that and make the data it uses in the dtor more > > self-contained: > > > > Index: sys/ofed/include/linux/linux_compat.c > > =================================================================== > > --- linux_compat.c (revision 251465) > > +++ linux_compat.c (working copy) > > @@ -212,7 +212,7 @@ linux_file_dtor(void *cdp) > > struct linux_file *filp; > > > > filp = cdp; > > - filp->f_op->release(curthread->td_fpop->f_vnode, filp); > > + filp->f_op->release(filp->f_vnode, filp); > > kfree(filp); > > } > > > > @@ -232,6 +232,7 @@ linux_dev_open(struct cdev *dev, int oflags, int d > > filp->f_dentry = &filp->f_dentry_store; > > filp->f_op = ldev->ops; > > filp->f_flags = file->f_flag; > > + filp->f_vnode = file->f_vnode; > > if (filp->f_op->open) { > > error = -filp->f_op->open(file->f_vnode, filp); > > if (error) { > > > > Doesn't compile for me. Did you forget to add the f_vnode member to > struct linux_file? > > sys/ofed/include/linux/linux_compat.c: In function 'linux_file_dtor': > sys/ofed/include/linux/linux_compat.c:214: error: 'struct linux_file' > has no member named 'f_vnode' > sys/ofed/include/linux/linux_compat.c: In function 'linux_dev_open': > sys/ofed/include/linux/linux_compat.c:234: error: 'struct linux_file' > has no member named 'f_vnode' Oof it's in another header: Index: sys/ofed/include/linux/fs.h =================================================================== --- fs.h (revision 251494) +++ fs.h (working copy) @@ -73,6 +73,7 @@ struct linux_file { struct dentry f_dentry_store; struct selinfo f_selinfo; struct sigio *f_sigio; + struct vnode *f_vnode; }; #define file linux_file -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Sat Jun 8 10:13:48 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 853ED367 for ; Sat, 8 Jun 2013 10:13:48 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id 14CD21D74 for ; Sat, 8 Jun 2013 10:13:47 +0000 (UTC) Received: from ur.dons.net.au ([118.210.215.193]) (authenticated bits=0) by cain.gsoft.com.au (8.14.4/8.14.3) with ESMTP id r58ADODA064252 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 8 Jun 2013 19:43:30 +0930 (CST) (envelope-from doconnor@gsoft.com.au) From: "Daniel O'Connor" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Sat, 8 Jun 2013 19:43:24 +0930 Subject: Multicast panic caused by elasticsearch To: "freebsd-stable@freebsd.org stable" Message-Id: <248F626A-4B8B-48EA-85F9-F411784DF8E6@gsoft.com.au> Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) X-Mailer: Apple Mail (2.1503) X-Spam-Score: -0.272 () BAYES_00,RDNS_NONE X-Scanned-By: MIMEDefang 2.67 on 203.31.81.10 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 10:13:48 -0000 Hi, I was experimenting with Logstash + elasticsearch on FreeBSD 9 - = initially I downloaded it by hand (I forgot to check for a port) and it = worked fine. I then tried the port and this forced me to use a different java version = (was jdk-16.0.3p4_25 now openjdk6-b27) and it seems that the new one = causes a panic. Unfortunately crashdumps aren't working properly, however I did get the = panic message.. in6p_lookup_mcast_ifp: not multicast This is from /usr/src/sys/netinet6/in6_mcast.c around line 1775. Has anyone else seen such a problem? Some quick googling didn't show = anything. uname is.. FreeBSD maarsy-rdb.maarsy.rocketrange.no 9.0-CURRENT FreeBSD 9.0-CURRENT = #0 r224195: Tue Jul 19 17:45:03 CST 2011 = radar@maarsy-acq3.gsoft.com.au:/usr/obj/usr/src/sys/GENERIC amd64 -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Sat Jun 8 17:23:10 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5C1108A9 for ; Sat, 8 Jun 2013 17:23:10 +0000 (UTC) (envelope-from alp@sfedu.ru) Received: from mail.r61.net (mail.r61.net [195.208.245.238]) by mx1.freebsd.org (Postfix) with ESMTP id 124F817AE for ; Sat, 8 Jun 2013 17:23:08 +0000 (UTC) Received: from www.webmail.sfedu.ru (roundcube.r61.net [195.208.245.250]) (Authenticated sender: alp@sfedu.ru) by mail.r61.net (MTA) with ESMTPA id F1E6BA61352 for ; Sat, 8 Jun 2013 21:08:50 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=rsu.ru; s=email; t=1370711331; bh=7WVVha3TQ6Mkw7BwOtSOiIbVZIY=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:Date:From:To: Subject:Message-ID; b=U5BHKuFrFv2FkUIOpK7FJ7zFV2aWZ1LhuLQi1U78HO8qlqkYCXTEmhRExRVFzzXB/ CRK0wtEC23i5TQtckUDAiWrJdH2PGkcVeOw5a6KKC0kzDtxcVOp5rCe11yvP4eR76/ yVzeudMDguBzkkgCQLT3eTR5g/pIAeYWhket8HsQ= Received: from [2.92.110.227] by webmail.sfedu.ru with HTTP (HTTP/1.1 POST); Sat, 08 Jun 2013 21:08:50 +0400 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 08 Jun 2013 21:08:50 +0400 From: Alexander Pyhalov To: freebsd-stable@freebsd.org Subject: Possible 8.4 regression Message-ID: <4fffaaf8a6667175fca94ce32f25a06a@sfedu.ru> X-Sender: alp@rsu.ru X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.3.7 (mail.r61.net [0.0.0.0]); Sat, 08 Jun 2013 21:08:51 +0400 (MSK) X-Spam-Status: No, score=-100.0 required=5.0 tests=UNPARSEABLE_RELAY, USER_IN_WHITELIST autolearn=disabled version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.r61.net X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 17:23:10 -0000 Hello. Just wanted to share a notice. I had a 8.3 system with PostgreSQL running in a jail. rc.conf has the following lines: jail_enable="YES" jail_sysvipc_allow="YES" jail_mount_enable="YES" jail_devfs_enable="YES" jail_pgsql_rootdir="/jails/run/pgsql" jail_pgsql_hostname="pgsql.freebsd" jail_pgsql_ip="my.ip" jail_pgsql_interface="em0" It was running normally. However, after update to 8.4 I had to add the following parameter jail_pgsql_parameters="allow.sysvipc" Without it shmget in jail didn't work. -- System Administrator of Southern Federal University Computer Center From owner-freebsd-stable@FreeBSD.ORG Sat Jun 8 17:30:30 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 06CB2AC5 for ; Sat, 8 Jun 2013 17:30:30 +0000 (UTC) (envelope-from guy.helmer@gmail.com) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) by mx1.freebsd.org (Postfix) with ESMTP id C8E3B180A for ; Sat, 8 Jun 2013 17:30:29 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id va7so8068283obc.27 for ; Sat, 08 Jun 2013 10:30:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=N90eo+De/EjaPUp0qPh94eJTniVGwPhWbUuYP0gmBRg=; b=CZX3ujmb8hUimsUrBioYBPi86ovMo70Hd1vG0jZuJfhdhpreT/YEKJLFi9bZMWqxCF QYYkr4qh1NFjHgFsC8FYdKnELuKfMLPmjpc/qLhN6cRE5kGygQsqIdhAHUNtqjcp4LEi AIq9D47wkCD43NaIlxUAomIeVwSyIoKpFA6/KN3rUE+K3MBMOMe55PEOR5vItdbUtwOh 90LSlqSi91OSvWMetIHfeQ2O0+rg1h4+80x31EwIkPNHoT2N6++ld6/57nwf4wbh0vRi wkflO4MiX2v7JW4S4MD00gh2eY4qR5RMqACq1yl31YkVLQ4l3HgVtpYCUwWfFP5s4dhK BROQ== X-Received: by 10.182.233.227 with SMTP id tz3mr2837021obc.23.1370712629454; Sat, 08 Jun 2013 10:30:29 -0700 (PDT) Received: from [192.168.1.135] (173-25-205-212.client.mchsi.com. [173.25.205.212]) by mx.google.com with ESMTPSA id ei16sm7220879oeb.7.2013.06.08.10.30.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 08 Jun 2013 10:30:28 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Multicast panic caused by elasticsearch From: Guy Helmer In-Reply-To: <248F626A-4B8B-48EA-85F9-F411784DF8E6@gsoft.com.au> Date: Sat, 8 Jun 2013 12:30:31 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <248F626A-4B8B-48EA-85F9-F411784DF8E6@gsoft.com.au> To: Daniel O'Connor X-Mailer: Apple Mail (2.1508) Cc: "freebsd-stable@freebsd.org stable" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 17:30:30 -0000 On Jun 8, 2013, at 5:13 AM, Daniel O'Connor = wrote: > Hi, > I was experimenting with Logstash + elasticsearch on FreeBSD 9 - = initially I downloaded it by hand (I forgot to check for a port) and it = worked fine. >=20 > I then tried the port and this forced me to use a different java = version (was jdk-16.0.3p4_25 now openjdk6-b27) and it seems that the new = one causes a panic. >=20 > Unfortunately crashdumps aren't working properly, however I did get = the panic message.. > in6p_lookup_mcast_ifp: not multicast >=20 > This is from /usr/src/sys/netinet6/in6_mcast.c around line 1775. >=20 > Has anyone else seen such a problem? Some quick googling didn't show = anything. >=20 > uname is.. > FreeBSD maarsy-rdb.maarsy.rocketrange.no 9.0-CURRENT FreeBSD = 9.0-CURRENT #0 r224195: Tue Jul 19 17:45:03 CST 2011 = radar@maarsy-acq3.gsoft.com.au:/usr/obj/usr/src/sys/GENERIC amd64 >=20 >=20 FWIW, I have not had any problem with elasticsearch on 9.1-stable from = about mid-May. Guy From owner-freebsd-stable@FreeBSD.ORG Sat Jun 8 20:56:14 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 752686C for ; Sat, 8 Jun 2013 20:56:14 +0000 (UTC) (envelope-from marie@surveysecond.com) Received: from smtp055.yesmail1.com (smtp055.yesmail1.com [184.175.181.55]) by mx1.freebsd.org (Postfix) with ESMTP id 52C1812F9 for ; Sat, 8 Jun 2013 20:56:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=dk; d=surveysecond.com; h=To:From:Reply-To:Subject:List-Unsubscribe:Content-Type:Content-Transfer-Encoding:Message-ID:Date; i=marie@surveysecond.com; bh=i0PZKne95ge0pyTo8VeZfCG/kXM=; b=nfIvGCy95Q+PRP9148Gs2V/mWBVchYsQHmxvOEMzGRL0bkVa6NjwRU7ekfC4UZbQ117g7/97vw1l IjYeZwcdTF8VxGxkMFgrarQhIDtr/KyuuIHQuryhmGNLcT6u+fhiONKzzYdnJ6kwk96YD9PRtfmk OixTwS5TwqYphCWPUJs= DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=dk; d=surveysecond.com; b=lnKascbCQKRo/uZCfI4Km4MWa5b0MUeAAZ8c4R3EjDTRFzTBq0TcJR3HB4Vf9ZqgfkBPdIjBaXJm 8Saj0LrSgXr4+Z/jXhFeqjhEqXsPaIdgK17pLCzs+XLmtcO9utfvdrvgBXSk8MK6RZgzUIecuP9j kk2DCPMHYmTRQ8irxpU=; Received: by smtp055.yesmail1.com id hmed6o2iaa82 for ; Sat, 8 Jun 2013 15:46:05 -0500 (envelope-from ) To: freebsd-stable@freebsd.org, From: "Hookup.com" Subject: Hook Up Tonight!!!! X-MailSystem: 7mfs2t5i-476fki0OYU46XM Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Message-ID: <0.0.1.BE0.1CE64893222082C.0@smtp055.yesmail1.com> Date: Sat, 8 Jun 2013 15:46:05 -0500 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: marie@surveysecond.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 20:56:14 -0000 Find Your Partner! If you are seeking someone to have a good time with, this is the place to b= e! No charge to browse thousands of single near you. No charge to make or r= eceive an initial contact with other members. No charge to start having fun= with online dating. Sign up at no cost today! Join below Tonight!! http://trkcm.surveysecond.com/c/1370724363sDyMEfkcd7/7mfs2t5i-476fki0OYU46X= M/1 =0A=0AAlthough this is an advertisement_we hope this message has helped you= .If you would like to quit receiving messages then you can go the link belo= w.=0A=0Ahttp://trkcm.surveysecond.com/u/7mfs2t5i-476fki0OYU46XM=0A=0ACarter= &Conway ENT. 706 dakota avenue St Cloud FL 34769 United States=0A From owner-freebsd-stable@FreeBSD.ORG Sat Jun 8 21:45:41 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A0E8AA30 for ; Sat, 8 Jun 2013 21:45:41 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper.allbsd.org [IPv6:2001:2f0:104:e001::32]) by mx1.freebsd.org (Postfix) with ESMTP id 1EC7A1620 for ; Sat, 8 Jun 2013 21:45:40 +0000 (UTC) Received: from alph.d.allbsd.org (p2175-ipbf701funabasi.chiba.ocn.ne.jp [122.25.209.175]) (authenticated bits=128) by mail.allbsd.org (8.14.5/8.14.5) with ESMTP id r58LjNuL060288 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 9 Jun 2013 06:45:34 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) (authenticated bits=0) by alph.d.allbsd.org (8.14.5/8.14.5) with ESMTP id r58LjM0k062320; Sun, 9 Jun 2013 06:45:23 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Sun, 09 Jun 2013 06:45:12 +0900 (JST) Message-Id: <20130609.064512.733259070666191163.hrs@allbsd.org> To: alp@rsu.ru Subject: Re: Possible 8.4 regression From: Hiroki Sato In-Reply-To: <4fffaaf8a6667175fca94ce32f25a06a@sfedu.ru> References: <4fffaaf8a6667175fca94ce32f25a06a@sfedu.ru> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.5 on Emacs 24.3 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Multipart/Signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="--Security_Multipart(Sun_Jun__9_06_45_12_2013_933)--" Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.4 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (mail.allbsd.org [133.31.130.32]); Sun, 09 Jun 2013 06:45:35 +0900 (JST) X-Spam-Status: No, score=-94.4 required=13.0 tests=CONTENT_TYPE_PRESENT, ONLY1HOPDIRECT,RCVD_IN_PBL,SAMEHELOBY2HOP,USER_IN_WHITELIST,X_CHINESE_RELAY autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on gatekeeper.allbsd.org Cc: freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 21:45:41 -0000 ----Security_Multipart(Sun_Jun__9_06_45_12_2013_933)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Alexander Pyhalov wrote in <4fffaaf8a6667175fca94ce32f25a06a@sfedu.ru>: al> Hello. al> al> Just wanted to share a notice. al> I had a 8.3 system with PostgreSQL running in a jail. al> rc.conf has the following lines: al> al> jail_enable="YES" al> jail_sysvipc_allow="YES" al> jail_mount_enable="YES" al> jail_devfs_enable="YES" al> al> jail_pgsql_rootdir="/jails/run/pgsql" al> jail_pgsql_hostname="pgsql.freebsd" al> jail_pgsql_ip="my.ip" al> jail_pgsql_interface="em0" al> al> It was running normally. However, after update to 8.4 I had to add the al> following parameter al> jail_pgsql_parameters="allow.sysvipc" al> al> Without it shmget in jail didn't work. Thank you for the report. This affects jail_set_hostname_allow and jail_socket_unixiproute_only as well. I will add it to Errata. -- Hiroki ----Security_Multipart(Sun_Jun__9_06_45_12_2013_933)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iEYEABECAAYFAlGzpegACgkQTyzT2CeTzy2/8gCgt4IF4CJ/iSVkU6aHgTDd23oF zT0AoJnR2jAhYD+2EegopGk0c5TWZfhf =luRB -----END PGP SIGNATURE----- ----Security_Multipart(Sun_Jun__9_06_45_12_2013_933)---- From owner-freebsd-stable@FreeBSD.ORG Sat Jun 8 22:43:46 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6E576BE1; Sat, 8 Jun 2013 22:43:46 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp1.bway.net (smtp1.v6.bway.net [IPv6:2607:d300:1::27]) by mx1.freebsd.org (Postfix) with ESMTP id 4C50817CD; Sat, 8 Jun 2013 22:43:46 +0000 (UTC) Received: from frankentosh.sporklab.com (foon.sporktines.com [96.57.144.66]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp1.bway.net (Postfix) with ESMTPSA id 6FF8B9586E; Sat, 8 Jun 2013 18:43:38 -0400 (EDT) Subject: Re: Possible 8.4 regression Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=utf-8 From: Charles Sprickman In-Reply-To: <20130609.064512.733259070666191163.hrs@allbsd.org> Date: Sat, 8 Jun 2013 18:43:37 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4fffaaf8a6667175fca94ce32f25a06a@sfedu.ru> <20130609.064512.733259070666191163.hrs@allbsd.org> To: Hiroki Sato X-Pgp-Agent: GPGMail 1.3.3 X-Mailer: Apple Mail (2.1085) Cc: freebsd-stable@FreeBSD.org, alp@rsu.ru X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 22:43:46 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jun 8, 2013, at 5:45 PM, Hiroki Sato wrote: > Alexander Pyhalov wrote > in <4fffaaf8a6667175fca94ce32f25a06a@sfedu.ru>: >=20 > al> Hello. > al> > al> Just wanted to share a notice. > al> I had a 8.3 system with PostgreSQL running in a jail. > al> rc.conf has the following lines: > al> > al> jail_enable=3D"YES" > al> jail_sysvipc_allow=3D"YES" > al> jail_mount_enable=3D"YES" > al> jail_devfs_enable=3D"YES" > al> > al> jail_pgsql_rootdir=3D"/jails/run/pgsql" > al> jail_pgsql_hostname=3D"pgsql.freebsd" > al> jail_pgsql_ip=3D"my.ip" > al> jail_pgsql_interface=3D"em0" > al> > al> It was running normally. However, after update to 8.4 I had to add = the > al> following parameter > al> jail_pgsql_parameters=3D"allow.sysvipc" > al> > al> Without it shmget in jail didn't work. >=20 > Thank you for the report. This affects jail_set_hostname_allow and > jail_socket_unixiproute_only as well. I will add it to Errata. And allow.raw_sockets=E2=80=A6 Charles >=20 > -- Hiroki -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJRs7OaAAoJEMfwH0dqLIp2w+kIAKn1LW9nUZHlv9Hpm4+M/+j/ OhITQDkTqUcvQZcXyUN3wMmWLSJTL0eYpu57pMp3tsfr61rv/GS3t51Lw0aUPwG3 LDLNZ5rslURffbad+vTlUA2RYfcu0aNk1bvEQxRobdA4ql2BoC59K2vkOzE9WB59 YXvG0Pl1Y7MiSrGZMGiKGDDUJqllBzHCl7PwSvSBK9JBPh0A2LS9mEzTRc3clUIh bm3WvPhfuAvgypxncOkVk4OARpbYunHlC70AkOqohvdqFQzw4BaVXFUru5yvTJ+Y h8t4LtYu5px015brZEox61ECrIavvA1ujv7mQNyWDrgFzNI1IS3eLGx4mr/01io=3D =3DbbMA -----END PGP SIGNATURE-----