From owner-freebsd-fs@freebsd.org Sun Jul 30 06:34:09 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B519DB03D8 for ; Sun, 30 Jul 2017 06:34:09 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8DAD867700 for ; Sun, 30 Jul 2017 06:34:08 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:26:9b0e:1926:57f5:2ee4:923c] (dynamic-2a02-2698-26-0-0.perm.ertelecom.ru [IPv6:2a02:2698:26:9b0e:1926:57f5:2ee4:923c] (may be forged)) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v6U6Y25R047677 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Sun, 30 Jul 2017 11:34:02 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1501396442; bh=pqPAlkjtWgmZVZpNHufK9MFTKTV3PERWALgRNWMCDH0=; h=Subject:To:References:From:Date:In-Reply-To; b=rUvhhKl3iMybr13cEuCo1282nqtwgfbW0BRD1wZdUOx3Y4+QbCj0C+4x3ydDp9t5G IK55ceQTzUQcsKoroY8FYKl3Ri/5WgBhbQcOEMkdLOUGGzkiiGcKUOiFmUwVJ5W+2m i0/pHGnOk3epbrWJkSqLr/6eF0qeHAPmA4bMa4mo= Subject: Re: ZFS and ISCSI question To: freebsd-fs@freebsd.org References: From: "Eugene M. Zheganin" Message-ID: Date: Sun, 30 Jul 2017 11:34:02 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 06:34:09 -0000 On 29.07.2017 23:59, Kaya Saman wrote: > Hi, > > > I'm trying to find out how ZFS and ISCSI work together. So far I have > followed the guide in the handbook: > > > https://www.freebsd.org/doc/handbook/network-iscsi.html > > > Based on the guide what I have done is created 3 demo files using dd: > > > disk1, disk2, disk3 > > > then created a zpool around them: zpool create iscsi-tst /data/disk1 > /data/disk2 /data/disk3 > > > and finally a ZVOL: zfs create -V iscsi-tst/tank > > > From the guide a basic (for testing only) ctl.conf: > > > portal-group pg0 { > discovery-auth-group no-authentication > listen 0.0.0.0 > listen [::] > } > > target iqn.2012-06.com.example:target0 { > auth-group no-authentication > portal-group pg0 > > lun 0 { > path iscsi-tst/tank > size 500M > } > } > > > and then on the Initiator machine: > > > iscsictl -A -p -t iqn.2012-06.com.example:target0 > > > I can confirm a connection: > > > # iscsictl -L > Target name Target portal State > iqn.2012-06.com.example:target0 Connected: da24 > > > But if we look at dmesg output it is saying the size cannot be queried? > > > da24 at iscsi4 bus 0 scbus9 target 0 lun 0 > da24: Fixed Direct Access SPC-4 SCSI device > da24: Serial Number MYSERIAL 0 > da24: 150.000MB/s transfers > da24: Command Queueing enabled > da24: Attempt to query device size failed: NOT READY, Logical unit not > ready, manual > > > I am wondering what I have done wrong and additionally is it possible > to get the Initiator to see the zpool/ZVOL so that I can create a zfs > dataset on it? > You forgot to enter the zvol size - f -V argument. I'm surprised it was even created. Eugene. From owner-freebsd-fs@freebsd.org Sun Jul 30 10:36:27 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 603F2DB3ED4 for ; Sun, 30 Jul 2017 10:36:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D9476C5DB for ; Sun, 30 Jul 2017 10:36:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6UAaRpf093192 for ; Sun, 30 Jul 2017 10:36:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221075] regression: 11.1 is unable to mount ZFS / on boot Date: Sun, 30 Jul 2017 10:36:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: seschwar@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 10:36:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221075 --- Comment #10 from seschwar@gmail.com --- To gather a few more data points I tested several other releases and snapsh= ots. Their boot environments were created as described in comment #0. The results were as follows: - 10.3-RELEASE: works - 10.3-STABLE: works - 11.0-RELEASE: works - 11.1-RELEASE: broken as described in comment #0 - 11.1-STABLE: broken as described in comment #0 - 12.0-CURRENT: works In order to pinpoint the breakage I checked the revisions between 11.0 and = 11.1 in a way similar to git-bisect (binary search). So after compiling a few kernels and rebooting several times I found base r318197 to introduce the breakage. Revisions before that are able to mount my root filesystem just fine. But revisions after (and including) base r318197 result in "Mounting from zfs:ssd/ROOT/(...) failed with error 6." during boot. I don't know anything about hardware or kernel development, so I'm just speculating here: base r318197 only touches MMC and SD stuff, so I'm not su= re how that could break assembling my ZFS pool, which is located on SATA disks= .=20 But the notebook does have a SD card reader... Also base r318197 mentions = that a few commit were only partially applied. Maybe something went wrong when backporting them, which would explain why 12.0-CURRENT works, but 11.1 does= n't. I hope this is helpful in pinpointing the cause. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jul 30 10:39:39 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F4CBDB40AF for ; Sun, 30 Jul 2017 10:39:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6DDF56C80B for ; Sun, 30 Jul 2017 10:39:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6UAdd7x097697 for ; Sun, 30 Jul 2017 10:39:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221075] regression: 11.1 is unable to mount ZFS / on boot Date: Sun, 30 Jul 2017 10:39:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: seschwar@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 10:39:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221075 --- Comment #11 from seschwar@gmail.com --- I'm trying to verify that base r318197 is the cause of my troubles by apply= its diff to 11.0-RELASE and its reverse diff to 11.1-RELEASE to see if that cha= nges anything. Unfortunately the diffs don't apply cleanly. I'll see what I ca= n do about that. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jul 30 10:54:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3E82DB4567 for ; Sun, 30 Jul 2017 10:54:13 +0000 (UTC) (envelope-from zhao6014@gmail.com) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BFFDF6CEEC for ; Sun, 30 Jul 2017 10:54:13 +0000 (UTC) (envelope-from zhao6014@gmail.com) Received: by mail-oi0-x230.google.com with SMTP id a9so163614303oih.0 for ; Sun, 30 Jul 2017 03:54:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oPPKphoANpv5YE98v3+8GJuC0TyKB8zJ2wsDVudf7pk=; b=jUcmoZqpGdcxJDki3O1qT/8BoU8B0FsgENIT7swdLysT5Q/CdYby9tAMvHHmGPMQig r59VqW3LkGtgj9/SJV5E+NGhhC+fKg6GQlW08qTHegWUC0iZ9KCUOpaiefcDtdrjmeMf uKzxSdrsMHcY2jrwrmKUXn6YpzF/gIkPheJQD/gZnN0YDhOM4BVxbgv13iajqj63+vlZ k2YPy5adfVfynMVxwYkcXGQ3UPOETMN/WhpiZK33qkd9/QtgJPtP+ve/931CuzQ5jLIY La+/En9Q6au3Lr20IPRevyKAV2fw1BQ5ZU4IQr+dn5bU9oYULl6EebMwL4I/i3FBmFnh vhzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oPPKphoANpv5YE98v3+8GJuC0TyKB8zJ2wsDVudf7pk=; b=syL/+1MJnXJSpKPDn9alvQua20ZjyeimtW+3HkxiqnjADMjEB61pnNKS9XEx69fx/k WjvODr96Dkyk//dpS6MuPrjA7420y/3BDcL4RA3GaxkbdfW8fXBHMEQMnZVvmw4RCJ8w VO2KWClh1ltZ0PgAVEx/e+spErKo9U6R3u8QbqXK9AudAfBxtdWVlmZKj7flehXQurV8 KB7KiwiVKUOo6sArnaIwfyJcRdqy+9RZth0yTmcXVXYbTR6WCTcRgWxW10ENiF8Vya2X dfaQXYF0a0uIxamY1qStZ6jvVCEVtT54ziUiKC2TBvipuvMYQepP5gQLybqQ3hSc4jtF 8qOg== X-Gm-Message-State: AIVw113eRDuMM+upTcVVQIBOpyV1w/Sphxsn6C46+6J/ImTErIPLmjPo 3vt7ufrJGb79sdMDCP8xYBph2/iXxg== X-Received: by 10.202.71.205 with SMTP id u196mr11212905oia.49.1501412053095; Sun, 30 Jul 2017 03:54:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.128.197 with HTTP; Sun, 30 Jul 2017 03:54:12 -0700 (PDT) Received: by 10.74.128.197 with HTTP; Sun, 30 Jul 2017 03:54:12 -0700 (PDT) In-Reply-To: References: From: Jov Date: Sun, 30 Jul 2017 18:54:12 +0800 Message-ID: Subject: Re: ZFS and ISCSI question To: Kaya Saman Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 10:54:14 -0000 See the inline comment about path. Jov 2017=E5=B9=B47=E6=9C=8830=E6=97=A5 2:59 AM=EF=BC=8C"Kaya Saman" =E5=86=99=E9=81=93=EF=BC=9A Hi, I'm trying to find out how ZFS and ISCSI work together. So far I have followed the guide in the handbook: https://www.freebsd.org/doc/handbook/network-iscsi.html Based on the guide what I have done is created 3 demo files using dd: disk1, disk2, disk3 then created a zpool around them: zpool create iscsi-tst /data/disk1 /data/disk2 /data/disk3 and finally a ZVOL: zfs create -V iscsi-tst/tank >From the guide a basic (for testing only) ctl.conf: portal-group pg0 { discovery-auth-group no-authentication listen 0.0.0.0 listen [::] } target iqn.2012-06.com.example:target0 { auth-group no-authentication portal-group pg0 lun 0 { path iscsi-tst/tank Does this path really exist? Mybe should be prefixed with /dev/zvol ? size 500M } } and then on the Initiator machine: iscsictl -A -p -t iqn.2012-06.com.example:target0 I can confirm a connection: # iscsictl -L Target name Target portal State iqn.2012-06.com.example:target0 Connected: da24 But if we look at dmesg output it is saying the size cannot be queried? da24 at iscsi4 bus 0 scbus9 target 0 lun 0 da24: Fixed Direct Access SPC-4 SCSI device da24: Serial Number MYSERIAL 0 da24: 150.000MB/s transfers da24: Command Queueing enabled da24: Attempt to query device size failed: NOT READY, Logical unit not ready, manual I am wondering what I have done wrong and additionally is it possible to get the Initiator to see the zpool/ZVOL so that I can create a zfs dataset on it? Thanks. Kaya _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sun Jul 30 12:19:32 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 094BFDB653D for ; Sun, 30 Jul 2017 12:19:32 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8ECA16F55E for ; Sun, 30 Jul 2017 12:19:31 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id m85so54527010wma.0 for ; Sun, 30 Jul 2017 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=MDV+PatwAs575hv5sIPm8EI5ZLyDgWLZBOEL3fpjbgo=; b=ufELKjWXzY5pZ6PZqlGQRf3XYoIX+DbS1p0Ta3dC68CM1wxZ2JLdJ7sV202rvpHTDS HhBj/5LZB5eQdgIw/Hbf+Q/HN2NQVrbpxJMW6GxgkVRDOJ98tH3P0bmsPmuvDGk7qAhb 1gu3UaB4c/wYcpnuuSAmk5XDQtCqp96j6rANyRKJ9doy4k9a7VokULwjErkZsWBdSVKq 0WzymjMnfZG7VAw7nhYPMhDo2SGAkKIyTLq7ZgERdMUvvFF6UOb48t2FcKThuRmXiWB3 P1fasJ9y2qeEi2KLbMJrj+s4HqZwkeoT/GBDBoelnR1+sLaDNy8qfJcsdYEiDiE58II0 hL3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=MDV+PatwAs575hv5sIPm8EI5ZLyDgWLZBOEL3fpjbgo=; b=UvEruQu8FVe/Jq3U5ta0I+msoDr+jZkTUecnPKpdM+SCBgyLTPGsl3I7walz33au4s zk3OLLQebLmkI/H9K/4VGqfmfGjbNQTHRSO+UmsaD3tnXFfDwAUW8NTCmwF8m95Za2us o9Lx7jQPXCKOrdF3XsN2p5G0tYAdbLtV4WUIV/3Y6IsHXpSFpsUgCidnyMEhvxwa6BJ+ VU1SBpzeEQnboZdoDMXLYitRuaDJ1f9W5183LgGC6HIax92YO6IBpQXJctVcI43uo4MS xpZnvMkq7QLUEvsrYn8Ymfp8dlRsj7AFa5ErxoxveRb7dDK3HzOFbKDx2apn7GEORa9t EuFg== X-Gm-Message-State: AIVw110f5gY8jIr/QoL3wgnmQ3ZmBUN70M2DBTFi42Eh1lQnMd86HAWZ gz4pQrOmwcY/lU6fs5o= X-Received: by 10.28.189.133 with SMTP id n127mr9097668wmf.23.1501417169358; Sun, 30 Jul 2017 05:19:29 -0700 (PDT) Received: from x220.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.gmail.com with ESMTPSA id i185sm8767230wmf.34.2017.07.30.05.19.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Jul 2017 05:19:28 -0700 (PDT) Subject: Re: ZFS and ISCSI question To: Jov Cc: freebsd-fs References: From: Kaya Saman Message-ID: <0f4905d2-6567-a83c-45f8-435c7c987d5b@gmail.com> Date: Sun, 30 Jul 2017 13:19:27 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 12:19:32 -0000 You forgot to enter the zvol size - f -V argument. I'm surprised it was even created. Eugene. _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Correct! :-) # zfs create -V iscsi-tst/tank bad volume size 'iscsi-tst/tank': bad numeric value 'iscsi-tst/tank So that makes it: zfs create -V 900M iscsi-tst/tank ------- target iqn.2012-06.com.example:target 0 { auth-group no-authentication portal-group pg0 lun 0 { path iscsi-tst/tank Does this path really exist? Mybe should be prefixed with /dev/zvol ? I was thinking ZFS terms but the path has got to be /dev/zvol: # ls /dev/zvol iscsi-tst so the /etc/ctl.conf file reads: target iqn.2012-06.com.example:target0 { auth-group no-authentication portal-group pg0 lun 0 { path /dev/zvol/iscsi-tst/tank size 900M } } Many Thanks for the responses! I understand that iscsi works at the "block device" level but how would one go about using ZFS on the initiator? The standard ZFS commands can be run: zpool followed by zfs FS-set on the Initiator machine however, it doesn't seem right to first create a ZFS pool on the Target system then create another one on the same pool on the Initiator. Would zpool import/export work or does something else need to be done to get the Initiator to create a ZFS data set? From owner-freebsd-fs@freebsd.org Sun Jul 30 19:47:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8743DC0F92 for ; Sun, 30 Jul 2017 19:47:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CE8F47EA6F for ; Sun, 30 Jul 2017 19:47:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6UJlEsi050135 for ; Sun, 30 Jul 2017 19:47:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221064] zfs should not be able to shadow mount on root directory from userspace Date: Sun, 30 Jul 2017 19:47:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: wheelcomplex@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 19:47:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221064 --- Comment #9 from David NewHamlet --- (In reply to Poul-Henning Kamp from comment #8) I agree with you. Before disable rootfs shadow mounting by default, notice should be announced in a period of time (at least 1-2 release cycle). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jul 30 21:00:58 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEEB7DC214E for ; Sun, 30 Jul 2017 21:00:58 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4E12809A8 for ; Sun, 30 Jul 2017 21:00:58 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6UL01pt038712 for ; Sun, 30 Jul 2017 21:00:58 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201707302100.v6UL01pt038712@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 30 Jul 2017 21:00:58 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Jul 2017 21:00:58 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Jul 31 10:05:18 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1795DCF23D for ; Mon, 31 Jul 2017 10:05:18 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6185570D3B for ; Mon, 31 Jul 2017 10:05:17 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (net206-94.perm.ertelecom.ru [46.146.206.94] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id v6VA5Bkc018370 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Mon, 31 Jul 2017 15:05:12 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Subject: Re: ZFS and ISCSI question To: freebsd-fs@freebsd.org References: <0f4905d2-6567-a83c-45f8-435c7c987d5b@gmail.com> From: "Eugene M. Zheganin" Message-ID: <27221f18-07d7-64c6-289a-81e839b10d67@norma.perm.ru> Date: Mon, 31 Jul 2017 15:05:11 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <0f4905d2-6567-a83c-45f8-435c7c987d5b@gmail.com> Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spamd-Result: default: False [1.50 / 25.00] RBL_SPAMHAUS_PBL(2.00)[94.206.146.46.zen.spamhaus.org : 127.0.0.10] HFILTER_HOSTNAME_UNKNOWN(2.50)[] BAYES_HAM(-3.00)[99.99%] DMARC_NA(0.00)[norma.perm.ru] MIME_GOOD(-0.10)[text/plain] R_DKIM_NA(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] RCPT_COUNT_1(0.00)[] MID_RHS_MATCH_FROM(0.00)[] RECEIVED_SPAMHAUS(0.00)[94.206.146.46.zen.spamhaus.org] TO_MATCH_ENVRCPT_ALL(0.00)[] FROM_HAS_DN(0.00)[] TO_DN_NONE(0.00)[] FROM_EQ_ENVFROM(0.00)[] RCVD_COUNT_1(0.00)[] ONCE_RECEIVED(0.10)[] X-Rspamd-Server: localhost X-Rspamd-Scan-Time: 0.62 X-Rspamd-Queue-ID: v6VA5Bkc018370 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jul 2017 10:05:19 -0000 Hi. On 30.07.2017 17:19, Kaya Saman wrote: > > > I understand that iscsi works at the "block device" level but how > would one go about using ZFS on the initiator? > > The standard ZFS commands can be run: > > zpool followed by zfs FS-set on the Initiator machine > > however, it doesn't seem right to first create a ZFS pool on the > Target system then create another one on the same pool on the Initiator. > > > > Would zpool import/export work or does something else need to be done > to get the Initiator to create a ZFS data set? Zvol is a block device indeed, but, even it is an entity from a parent zfs pool, it doesn't contain any filesystem, including zfs. Thus the kernel won't see anything. So you have to create a zpool first with 'zpool create'. Eugene. From owner-freebsd-fs@freebsd.org Mon Jul 31 12:10:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94008DAB211 for ; Mon, 31 Jul 2017 12:10:07 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wr0-x234.google.com (mail-wr0-x234.google.com [IPv6:2a00:1450:400c:c0c::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F2F774AD9 for ; Mon, 31 Jul 2017 12:10:07 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wr0-x234.google.com with SMTP id k71so124766551wrc.2 for ; Mon, 31 Jul 2017 05:10:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=KaJ/vISiXoeOT6RA00SwLuDtfDq2FA4ALAw0nJivaq8=; b=gm/gKbLly9vgS8r9Txw29o0r8A/GwUpxOUu5/XOua6U22+CZkFCI34JPlbPCd/Z5p+ yQrZprfiMkFkLWUT5EmgKhplXGSj8ofbgsp/ZOb7H4iVegUWrCfMyjlg4OdwK3pC0ZzS xmGjWaSegZfcmoNpGzmvCsdoC9qpkKuZHrdgLDkpRcW9l0EDDPNYv/3j8gCcd4qKVp7h TkSSnTZMrn6riOPLF9W8xhWi1OaW8/nnDpD+Vk4w0baM9t3ErylpK/uosmLm1HYmRr25 SpaO9CF8zF6qq983ehCXUGoBv85PKk++4P/gENxn2CCL+9f8lHPkD1MLTyT5G1eapEIC nzqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=KaJ/vISiXoeOT6RA00SwLuDtfDq2FA4ALAw0nJivaq8=; b=f4vYMC/rMssLOuqte1udq+LLM4MLc0GLm6nF6cuwzBNjctYZgWR89gm0GSK3P2qmdY jHl0aDcoU+5wo3cvN7dYvIjLvS7FLp3/1K5dndlWsGZ6vhIqjEIps0Rlf6diw0N/Jmjq YWYngz+RmNxX7iWGhVQIh4it5HSqS+QvpV/txe2VW6sfQSVsZyAKZxZzCYLEMNW5BNQp fNlVPQnLCo5hWZDqind42hSKzoWncqidf1tM/K+jKTHMvjm/IwSYphHldGjnVt5p9ddL L9Q3B5ByFPC/9eFWVdkGYxSBKgGUgClenUgKCJtcsP3KaitAEvQY/CxYto3yw2X1tiQD YIsg== X-Gm-Message-State: AIVw113L/BW3vPAL/AAWjgOkWnzuIzYsHNcIf4lfIk5ashNbj2sMcZLK MmppINlDvhrn64RheSE= X-Received: by 10.223.165.129 with SMTP id g1mr11348218wrc.167.1501503004037; Mon, 31 Jul 2017 05:10:04 -0700 (PDT) Received: from x220.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.gmail.com with ESMTPSA id r128sm582328wmb.27.2017.07.31.05.10.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 31 Jul 2017 05:10:02 -0700 (PDT) Subject: Re: ZFS and ISCSI question To: freebsd-fs@freebsd.org References: <0f4905d2-6567-a83c-45f8-435c7c987d5b@gmail.com> <27221f18-07d7-64c6-289a-81e839b10d67@norma.perm.ru> From: Kaya Saman Message-ID: Date: Mon, 31 Jul 2017 13:10:02 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <27221f18-07d7-64c6-289a-81e839b10d67@norma.perm.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jul 2017 12:10:07 -0000 On 07/31/2017 11:05 AM, Eugene M. Zheganin wrote: > Hi. > > On 30.07.2017 17:19, Kaya Saman wrote: >> >> >> I understand that iscsi works at the "block device" level but how >> would one go about using ZFS on the initiator? >> >> The standard ZFS commands can be run: >> >> zpool followed by zfs FS-set on the Initiator machine >> >> however, it doesn't seem right to first create a ZFS pool on the >> Target system then create another one on the same pool on the Initiator. >> >> >> >> Would zpool import/export work or does something else need to be done >> to get the Initiator to create a ZFS data set? > Zvol is a block device indeed, but, even it is an entity from a parent > zfs pool, it doesn't contain any filesystem, including zfs. Thus the > kernel won't see anything. So you have to create a zpool first with > 'zpool create'. > > Eugene. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Hmm.... basically what I am trying to acheive is to be able to create a zpool and zfs file system on the Initiator system. Of course on the Target one could run: zpool create pool_1 zfs -V pool_1/zvol and on the Initiator: zpool create pool zfs create pool/fs-set but would that be recommended as one would be using 2x zpools on the same "device list"? As an alternative I have tried something like this: /etc/ctl.conf portal-group pg0 { discovery-auth-group no-authentication listen 0.0.0.0 listen [::] } target iqn.2012-06.com.example:target0 { auth-group no-authentication portal-group pg0 lun 0 { # path /dev/zvol/iscsi-tst/tank # size 900M path /data/disk1 size 200M } lun 1 { path /data/disk2 size 200M } # lun 2 { # path /data/disk3 # size 500M # } # lun 3 { # path /data/disk4 # size 500M # } } target iqn.2012-06.com.example:target1 { auth-group no-authentication portal-group pg0 lun 2 { path /data/disk3 size 500M } lun 3 { path /data/disk4 size 500M } } Then on Initiator: # iscsictl -L Target name Target portal State iqn.2012-06.com.example:target0 Connected: da24 da25 iqn.2012-06.com.example:target1 Connected: da26 da27 so then the zpool becomes: iscsi-tst 672M 936K 671M - 0% 0% 1.00x ONLINE - # zpool status iscsi-tst pool: iscsi-tst state: ONLINE status: One or more devices are configured to use a non-native block size. Expect reduced performance. action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool. scan: none requested config: NAME STATE READ WRITE CKSUM iscsi-tst ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da24 ONLINE 0 0 0 block size: 8192B configured, 16384B native da25 ONLINE 0 0 0 block size: 8192B configured, 16384B native mirror-1 ONLINE 0 0 0 da26 ONLINE 0 0 0 block size: 8192B configured, 16384B native da27 ONLINE 0 0 0 block size: 8192B configured, 16384B native errors: No known data errors Then zfs dataset: # zfs list iscsi-tst NAME USED AVAIL REFER MOUNTPOINT iscsi-tst 816K 639M 192K /iscsi-tst # zfs list iscsi-tst/tank NAME USED AVAIL REFER MOUNTPOINT iscsi-tst/tank 192K 639M 192K /iscsi-tst/tank So for best redundancy like "hot swap" etc... what would be the best solution or is there an ISCSI "Best Practice" to not get totally burned if something goes wrong with an iscsi attached drive? <-- taking backups of data excluded of course :-) Regards, Kaya From owner-freebsd-fs@freebsd.org Mon Jul 31 21:03:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA85FDBD9D1 for ; Mon, 31 Jul 2017 21:03:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9E65268BCB for ; Mon, 31 Jul 2017 21:03:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6VL3ZVP011958 for ; Mon, 31 Jul 2017 21:03:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221075] regression: 11.1 is unable to mount ZFS / on boot Date: Mon, 31 Jul 2017 21:03:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: marius@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jul 2017 21:03:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221075 --- Comment #12 from Marius Strobl --- (In reply to seschwar from comment #10) Unfortunately, I don't see how r318197 could have an impact on ZFS either; that revision only touches bits related to mmc(4), mmcsd(4) and correspondi= ng bridge drivers, but no KPI, no global structure or anything like that. Moreover, according to attachments 184813 and 184814, the drivers in questi= on also don't attach on that machine (is the card reader you mention a USB one= ?). The partial merges only refer to e. g. ARM-specific bridge drivers that sim= ply don't exist in stable/11, so these omissions should impact you even less. The one thing that comes to my mind is that since r318197, mmcsd(4) depends on geom_flasmap(4) and that the latter might cause some interaction with ZF= S. Given that I'm not familiar with how ZFS interfaces with GEOM, I have no id= ea what actually could be going wrong, though. As far as geom_flasmap(4) is concerned, I can't spot an obvious bug in that regard and its code doesn't differ between head and stable/11. One - not necessarily conclusive - way to test the theory of interaction between geom_flasmap(4) and ZFS would be to = use a kernel built from a configuration with mmcsd(4) removed, so this dependen= cy isn't dragged in. In general, I'd suggest to look for unmerged fixes to GEOM and to the parts of ZFS that interface with GEOM. When interfacing mmcsd(4) with geom_flashmap(4), I've had a hard time with races in GEOM; if g_retaste(9) is called (so that an additional provider - now additionally offered by a GEOM class due to a change in configuration - should get picked up) while re-tasting is already in progress, the already ongoing re-tasting won't take note of the new provider and the new re-taste event won't actually result in an additional re-taste cycle being scheduled. If ZFS is suffering from the same or a similar bug, this doesn't necessarily mean that the problem you are hitting is present in stable/11 only; with he= ad you just might be lucky that timing differences resulting from geom_flashma= p(4) also being in the mix don't lead to a race. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 1 00:26:27 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54E42DC1609 for ; Tue, 1 Aug 2017 00:26:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 36E976F462 for ; Tue, 1 Aug 2017 00:26:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v710QRIB025813 for ; Tue, 1 Aug 2017 00:26:27 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 209508] zfs import assertion failed in avl_add() Date: Tue, 01 Aug 2017 00:26:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jonc@chen.org.nz X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 00:26:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209508 Jonathan Chen changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonc@chen.org.nz --- Comment #1 from Jonathan Chen --- I hit this error trying to expand a zfs partition, with 11.1-RELEASE # gpart delete -i 4 ada0 # gpart resize -i 3 ada0 ... some errors appeared which I didn't write down ... # mkdir /tmp/z # zpool import -f -R /tmp/z irontree Assertion failed: (avl_find() succeeded inside avl_add()), file /usr/src/cddl/lib/libavl/../../../sys/cddl/contrib/opensolaris/common/avl/a= vl.c, line 649. I had to use 10.3-RELEASE to fix this. Using the same commands, I managed to recover and expand my zpool successfully. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 1 15:34:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66236DB0976 for ; Tue, 1 Aug 2017 15:34:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 54B6B7FA28 for ; Tue, 1 Aug 2017 15:34:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v71FY6RB067043 for ; Tue, 1 Aug 2017 15:34:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221133] ZFS failure on partition resize Date: Tue, 01 Aug 2017 15:34:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 15:34:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221133 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |regression Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 1 15:35:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2E5DDB0CA1 for ; Tue, 1 Aug 2017 15:35:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 920617FD96 for ; Tue, 1 Aug 2017 15:35:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v71FZb5u070267 for ; Tue, 1 Aug 2017 15:35:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Tue, 01 Aug 2017 15:35:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 15:35:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Aug 1 16:22:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47B61DB2422 for ; Tue, 1 Aug 2017 16:22:37 +0000 (UTC) (envelope-from demis@yandex.ru) Received: from forward102p.mail.yandex.net (forward102p.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0737E82049 for ; Tue, 1 Aug 2017 16:22:37 +0000 (UTC) (envelope-from demis@yandex.ru) Received: from mxback6g.mail.yandex.net (mxback6g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:167]) by forward102p.mail.yandex.net (Yandex) with ESMTP id B30354301C35 for ; Tue, 1 Aug 2017 19:22:18 +0300 (MSK) Received: from web39g.yandex.ru (web39g.yandex.ru [95.108.252.209]) by mxback6g.mail.yandex.net (nwsmtp/Yandex) with ESMTP id wmdPzslmfk-MILq5On0; Tue, 01 Aug 2017 19:22:18 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1501604538; bh=qXs5lcu8cJPz81KXjLtPuwrPO7sZDK/A5mczRrfGciY=; h=From:To:Subject:Message-Id:Date; b=gdzAD1vQRcMXowNdnyBnaYZINPd8M96vdU9Ss58LkLWw3A4FKjkCJbjlOvsWMP9HO sgoGI1Epi3kDDoEBx4pU5+YI/vu4ZoCmpPlgAC37z6K+3O6EVKWj5rUzdGcz4GwuKl kRHqHzmQV1gkhAzMmHtR9ExTszZ0gu8JjawqiiLI= Authentication-Results: mxback6g.mail.yandex.net; dkim=pass header.i=@yandex.ru Received: by web39g.yandex.ru with HTTP; Tue, 01 Aug 2017 19:22:18 +0300 From: DemIS To: freebsd-fs Subject: Re-installing clean system with resizing partitions causes a kernel panic MIME-Version: 1.0 Message-Id: <3534511501604538@web39g.yandex.ru> X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Tue, 01 Aug 2017 19:22:18 +0300 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Aug 2017 16:22:37 -0000 Can someone verify and explain why a simple re-installing clean system with resizing partitions causes a kernel panic? Tested on FreeBSD 9.3 , 10.3, 11.0. The effect is easily repeated in VirtualBox with a fixed size of VM. Demis. From owner-freebsd-fs@freebsd.org Wed Aug 2 08:34:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6ADCDC9E81 for ; Wed, 2 Aug 2017 08:34:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C58A27FB06 for ; Wed, 2 Aug 2017 08:34:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v728YGPk090668 for ; Wed, 2 Aug 2017 08:34:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Wed, 02 Aug 2017 08:34:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: delphij@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 08:34:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 --- Comment #2 from Xin LI --- (In reply to Alexander Motin from comment #1) Doing 'zpool import' with 11.1-RELEASE binary set on 11.0-RELEASE kernel wo= uld cause zpool to crash (also zfs mount -a) with invalid argument. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 2 10:34:25 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77276DCC7E9 for ; Wed, 2 Aug 2017 10:34:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 661B684452 for ; Wed, 2 Aug 2017 10:34:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v72AYO60046708 for ; Wed, 2 Aug 2017 10:34:25 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Wed, 02 Aug 2017 10:34:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 10:34:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 --- Comment #3 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Wed Aug 2 10:33:47 UTC 2017 New revision: 321921 URL: https://svnweb.freebsd.org/changeset/base/321921 Log: Add compat shim part missed at r305197. This fixes compatibility between old kernel and new ZFS tools. It seems to be tradition to forget it. :( PR: 221112 MFC after: 3 days Changes: head/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_compat.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 2 10:35:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F000DCC85B for ; Wed, 2 Aug 2017 10:35:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8DA51844F9 for ; Wed, 2 Aug 2017 10:35:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v72AZ4FP047726 for ; Wed, 2 Aug 2017 10:35:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Wed, 02 Aug 2017 10:35:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 10:35:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 Alexander Motin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |In Progress --- Comment #4 from Alexander Motin --- Oh, I found missed part of compat shim. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 2 12:59:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11699DD06A3; Wed, 2 Aug 2017 12:59:45 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 470C56408F; Wed, 2 Aug 2017 12:59:43 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1dcszy-0007ZC-55; Wed, 02 Aug 2017 14:43:38 +0200 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, "Eugene M. Zheganin" Cc: freebsd-stable Subject: Re: some general zfs tuning (for iSCSI) References: <8b41e7d6-7a2c-d456-2eee-93efd81aa86a@norma.perm.ru> Date: Wed, 02 Aug 2017 14:43:31 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <8b41e7d6-7a2c-d456-2eee-93efd81aa86a@norma.perm.ru> User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-As-Hash: bdb49c4ff80bd276e321aade33e76e02752072e2 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: ++ X-Spam-Score: 2.0 X-Spam-Status: No, score=2.0 required=5.0 tests=ALL_TRUSTED, BAYES_95 autolearn=disabled version=3.4.0 X-Scan-Signature: 788438cbfdc4dc137ce560360a3a99c7 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 12:59:45 -0000 On Fri, 28 Jul 2017 12:56:11 +0200, Eugene M. Zheganin wrote: > Hi, > > > I'm using several FreeBSD zfs installations as the iSCSI production > systems, they basically consist of an LSI HBA, and a JBOD with a bunch > of SSD disks (12-24, Intel, Toshiba or Sandisk (avoid Sandisks btw)). > And I observe a problem very often: gstat shows 20-30% of disk load, but > the system reacts very slowly: cloning a dataset takes 10 seconds, > similar operations aren't lightspeeding too. To my knowledge, until the > disks are 90-100% busy, this shouldn't happen. My systems are equipped > with 32-64 gigs of RAM, and the only tuning I use is limiting the ARC > size (in a very tender manner - at least to 16 gigs) and playing with > TRIM. The number of datasets is high enough - hundreds of clones, dozens > of snapshots, most of teh data ovjects are zvols. Pools aren't > overfilled, most are filled up to 60-70% (no questions about low space > pools, but even in this case the situation is clearer - %busy goes up in > the sky). > > So, my question is - is there some obvious zfs tuning not mentioned in > the Handbook ? On the other side - handbook isn't much clear on how to > tune zfs, it's written mostly in the manner of "these are sysctl iods > you can play with". Of course I have seen several ZFS tuning guides. > Like Opensolaris one, but they are mostly file- and > application-specific. Is there some special approach to tune ZFS in the > environment with loads of disks ? I don't know.... like tuning the vdev > cache or something simllar. ? > > > Thanks. > > Eugene. What version of FreeBSD are you running? What is the system doing during all this? How are your pools setup (raidz1/2/3, mirror, 3mirror)? How is your iSCSI configured and what are the clients doing with it? Is the data distributed evenly on all disks? Do the clients write a lot of sync data? I think this kind of information helps people helping you. Regards, Ronald. From owner-freebsd-fs@freebsd.org Wed Aug 2 18:27:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2D70DB027E for ; Wed, 2 Aug 2017 18:27:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4B2F70B55 for ; Wed, 2 Aug 2017 18:27:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v72IRUmm025490 for ; Wed, 2 Aug 2017 18:27:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221077] Boot from ZFS fails following freebsd-update Date: Wed, 02 Aug 2017 18:27:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bacon4000@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 18:27:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221077 --- Comment #5 from Jason Bacon --- Couldn't hold off on restoring this system, but I saved a copy of /boot and dd'd a copy of the boot block just in case someone wants to examine it at s= ome point. I have other root-on-zfs installations that are less critical, so if this happens again, we can use them for diagnosis. For my mission-critical systems, until this issue is understood and resolve= d, I'm installing to UFS2 and leaving most of the disk free for a post-install= ZFS pool setup, e.g. Filesystem Size Used Avail Capacity Mounted on /dev/mfid0p2 242G 3.7G 219G 2% / devfs 1.0K 1.0K 0B 100% /dev procfs 4.0K 4.0K 0B 100% /proc fdescfs 1.0K 1.0K 0B 100% /dev/fd zroot/share1 5.0T 726G 4.3T 14% /share1 zroot/sharedapps 4.3T 5.3G 4.3T 0% /sharedapps zroot/tmp 4.3T 68K 4.3T 0% /tmp zroot/home 4.3T 7.1G 4.3T 0% /usr/home zroot/var 4.3T 502M 4.3T 0% /var zroot/var/cache 4.3T 25K 4.3T 0% /var/cache zroot/var/cache/pkg 4.3T 211M 4.3T 0% /var/cache/pkg --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 2 20:54:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 454BFDB4A38 for ; Wed, 2 Aug 2017 20:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33C4E779B2 for ; Wed, 2 Aug 2017 20:54:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v72KsjE2029003 for ; Wed, 2 Aug 2017 20:54:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221188] zfs: bin/chmod/chmod_test:v_flag fails on ZFS, not UFS (ZFS updates mode for files unnecessarily) Date: Wed, 02 Aug 2017 20:54:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 20:54:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221188 Ngie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 2 21:19:24 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C74BDB510A for ; Wed, 2 Aug 2017 21:19:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8AB2F7C64A for ; Wed, 2 Aug 2017 21:19:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v72LJORs028099 for ; Wed, 2 Aug 2017 21:19:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221188] zfs: bin/chmod/chmod_test:v_flag fails on ZFS, not UFS (ZFS updates mode for files unnecessarily) Date: Wed, 02 Aug 2017 21:19:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 21:19:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221188 --- Comment #1 from commit-hook@freebsd.org --- A commit references this bug: Author: ngie Date: Wed Aug 2 21:18:55 UTC 2017 New revision: 321949 URL: https://svnweb.freebsd.org/changeset/base/321949 Log: Add expected failures for ZFS - :f_flag fails on ZFS because UF_IMMUTABLE isn't supported [1]. - :v_flag fails on ZFS because the mode for foo is [always] updated unnecessarily. get_filesystem(..) (supporting function that was added to the test script) is based on equivalent logic in usr.bin/extattr/tests/extattr_test.sh . MFC after: 1 week PR: 221189 [1], 221188 [2] Changes: head/bin/chmod/tests/chmod_test.sh --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Aug 2 21:21:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 880D1DB539A for ; Wed, 2 Aug 2017 21:21:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 769B87CA45 for ; Wed, 2 Aug 2017 21:21:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v72LLTDX034845 for ; Wed, 2 Aug 2017 21:21:29 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221188] zfs: bin/chmod/chmod_test:v_flag fails on ZFS, not UFS (ZFS updates mode for files unnecessarily) Date: Wed, 02 Aug 2017 21:21:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Aug 2017 21:21:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221188 --- Comment #2 from commit-hook@freebsd.org --- A commit references this bug: Author: ngie Date: Wed Aug 2 21:20:50 UTC 2017 New revision: 321950 URL: https://svnweb.freebsd.org/changeset/base/321950 Log: Always use first parameter passed to get_filesystem(..) instead of discar= ding it and using `.` instead. MFC after: 1 week MFC with: r321949 PR: 221189 [1], 221188 [2] Changes: head/bin/chmod/tests/chmod_test.sh --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 3 05:12:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65479DCA7A7 for ; Thu, 3 Aug 2017 05:12:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 536486704A for ; Thu, 3 Aug 2017 05:12:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v735CMlF072918 for ; Thu, 3 Aug 2017 05:12:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221188] zfs: bin/chmod/chmod_test:v_flag fails on ZFS, not UFS (ZFS updates mode for files unnecessarily) Date: Thu, 03 Aug 2017 05:12:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ngie@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 05:12:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221188 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |ngie@FreeBSD.org --- Comment #3 from Mark Linimon --- over to committer. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 3 05:22:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 405CCDCAABC for ; Thu, 3 Aug 2017 05:22:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EBDE67454 for ; Thu, 3 Aug 2017 05:22:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v735M8ah099071 for ; Thu, 3 Aug 2017 05:22:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221189] zfs: bin/chmod/chmod_test:f_flag fails on ZFS, not UFS; UF_IMMUTABLE not supported on ZFS Date: Thu, 03 Aug 2017 05:22:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 05:22:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221189 Ngie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|ngie@FreeBSD.org |freebsd-fs@FreeBSD.org --- Comment #4 from Ngie Cooper --- Still a ZFS bug. I just added a hack to bypass the issue in the tests. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 3 05:22:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1744DCAC1D for ; Thu, 3 Aug 2017 05:22:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DFA8867516 for ; Thu, 3 Aug 2017 05:22:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v735MXXf000567 for ; Thu, 3 Aug 2017 05:22:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221188] zfs: bin/chmod/chmod_test:v_flag fails on ZFS, not UFS (ZFS updates mode for files unnecessarily) Date: Thu, 03 Aug 2017 05:22:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_severity assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 05:22:34 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221188 Ngie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Affects Only Me |Affects Some People Assignee|ngie@FreeBSD.org |freebsd-fs@FreeBSD.org --- Comment #4 from Ngie Cooper --- Still a ZFS bug. I just added a hack to bypass the issue in the tests. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Aug 3 08:45:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 930E3DD13C3; Thu, 3 Aug 2017 08:45:54 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 155876D9B3; Thu, 3 Aug 2017 08:45:53 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (net206-94.perm.ertelecom.ru [46.146.206.94] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id v738jgWe083065 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 3 Aug 2017 13:45:43 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Subject: Re: some general zfs tuning (for iSCSI) To: freebsd-fs@freebsd.org Cc: freebsd-stable References: <8b41e7d6-7a2c-d456-2eee-93efd81aa86a@norma.perm.ru> From: "Eugene M. Zheganin" Message-ID: Date: Thu, 3 Aug 2017 13:45:42 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=iso-8859-15; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spamd-Result: default: False [1.50 / 25.00] BAYES_HAM(-3.00)[100.00%] RBL_SPAMHAUS_PBL(2.00)[94.206.146.46.zen.spamhaus.org : 127.0.0.10] HFILTER_HOSTNAME_UNKNOWN(2.50)[] DMARC_NA(0.00)[norma.perm.ru] MIME_GOOD(-0.10)[text/plain] R_DKIM_NA(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] MID_RHS_MATCH_FROM(0.00)[] RECEIVED_SPAMHAUS(0.00)[94.206.146.46.zen.spamhaus.org] TO_DN_SOME(0.00)[] RCPT_COUNT_2(0.00)[] TO_MATCH_ENVRCPT_ALL(0.00)[] FROM_HAS_DN(0.00)[] FROM_EQ_ENVFROM(0.00)[] RCVD_COUNT_1(0.00)[] ONCE_RECEIVED(0.10)[] X-Rspamd-Server: localhost X-Rspamd-Scan-Time: 1.40 X-Rspamd-Queue-ID: v738jgWe083065 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 08:45:54 -0000 Hi. On 02.08.2017 17:43, Ronald Klop wrote: > On Fri, 28 Jul 2017 12:56:11 +0200, Eugene M. Zheganin > wrote: > >> Hi, >> >> >> I'm using several FreeBSD zfs installations as the iSCSI production >> systems, they basically consist of an LSI HBA, and a JBOD with a >> bunch of SSD disks (12-24, Intel, Toshiba or Sandisk (avoid Sandisks >> btw)). And I observe a problem very often: gstat shows 20-30% of disk >> load, but the system reacts very slowly: cloning a dataset takes 10 >> seconds, similar operations aren't lightspeeding too. To my >> knowledge, until the disks are 90-100% busy, this shouldn't happen. >> My systems are equipped with 32-64 gigs of RAM, and the only tuning I >> use is limiting the ARC size (in a very tender manner - at least to >> 16 gigs) and playing with TRIM. The number of datasets is high enough >> - hundreds of clones, dozens of snapshots, most of teh data ovjects >> are zvols. Pools aren't overfilled, most are filled up to 60-70% (no >> questions about low space pools, but even in this case the situation >> is clearer - %busy goes up in the sky). >> >> So, my question is - is there some obvious zfs tuning not mentioned >> in the Handbook ? On the other side - handbook isn't much clear on >> how to tune zfs, it's written mostly in the manner of "these are >> sysctl iods you can play with". Of course I have seen several ZFS >> tuning guides. Like Opensolaris one, but they are mostly file- and >> application-specific. Is there some special approach to tune ZFS in >> the environment with loads of disks ? I don't know.... like tuning >> the vdev cache or something simllar. ? >> >> > > What version of FreeBSD are you running? Well, different ones. Mostly some versions of 11.0-RELEASE-pX and 11-STABLE. > What is the system doing during all this? What do you meant by "what" ? Nothing else except serving iSCSI - it's the main purpose of every one of these servers. > How are your pools setup (raidz1/2/3, mirror, 3mirror)? zroot is a mirrored two-disk pool, others are raidz, mostly spans of multiple 5-disk radizs. > How is your iSCSI configured and what are the clients doing with it? Using the kernel ctld of course. As you may know ctl.conf dosn't suppose any performance tweaks, it's just a way of organizing the authorization layer. Clients are the VMWare ESX hypervisors, using iSCSI as disk devices, as for ESX SRs, and as direct iSCSI disks in Windows VMs. > Is the data distributed evenly on all disks? It's not. Does it ever ditrubute evenly anywhere ? > Do the clients write a lot of sync data? What do you exactly mean by "sync data" ? Eugene. From owner-freebsd-fs@freebsd.org Thu Aug 3 11:38:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDBD0DB25FB; Thu, 3 Aug 2017 11:38:38 +0000 (UTC) (envelope-from Nikolaus@rath.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A198C7388A; Thu, 3 Aug 2017 11:38:37 +0000 (UTC) (envelope-from Nikolaus@rath.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 406A820692; Thu, 3 Aug 2017 07:38:36 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute1.internal (MEProxy); Thu, 03 Aug 2017 07:38:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=9+vCIbP7JKHp55JDgz z6D/bqUpUDalDk2IVK7wDHjrw=; b=RJGihGkNVrhk9lqu6hehGnr8+9qG3HpzUb 8242ZCwCcA5dVDqz7B7JaQvMjMyh42cMYBp3A5gxNV0zxd54wdlBsxc+vhPYDctn xGKO0Zi1kMEjBoJcJcMp8Ja4REu73r3/oEkIY+XI3tApLut7MhE+PkPp3+/FKxU6 l0Yf7AfLq0WsNy/RcSen4b+mxay0miKeKjrUrmVgi9JLfKdMrXm4OO+1yJRNKfHj i5hPCahBhsc7FWr7qdGK0/CaPjppwHJlJmN7HgQt59Wpr/fyO7s2VT+AMsls7QTS 7DlOIWrU3F8TVN6AOXN6Uw+oxAe5LcCYfghxmdPvbSAraYYLcSuw== X-ME-Sender: X-Sasl-enc: RHaOgjIHnozV/PSBRvjjL9garvGXE8OLOYO8avUVuxLx 1501760315 Received: from ebox.rath.org (ebox.rath.org [45.79.69.51]) by mail.messagingengine.com (Postfix) with ESMTPA id EDF167E683; Thu, 3 Aug 2017 07:38:35 -0400 (EDT) Received: from vostro.rath.org (vostro [192.168.12.4]) by ebox.rath.org (Postfix) with ESMTPS id 15EE33748; Thu, 3 Aug 2017 11:38:33 +0000 (UTC) Received: by vostro.rath.org (Postfix, from userid 1000) id 727071029B3; Thu, 3 Aug 2017 13:38:28 +0200 (CEST) From: Nikolaus Rath To: freebsd-ports@freebsd.org, freebsd-fs@freebsd.org Subject: State of FUSE on FreeBSD Mail-Copies-To: never Mail-Followup-To: freebsd-ports@freebsd.org, freebsd-fs@freebsd.org Date: Thu, 03 Aug 2017 13:38:28 +0200 Message-ID: <87y3r0ankb.fsf@vostro.rath.org> User-Agent: Gnus/5.130014 (Ma Gnus v0.14) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 11:38:38 -0000 Hello, I am the upstream maintainer of libfuse. I'd like to refresh / improve the FreeBSD support in libfuse. My goal is for libfuse not to require any FreeBSD specific patches. After taking a look at https://github.com/freebsd/freebsd/tree/master/sbin/mount_fusefs, https://svn.freebsd.org/ports/head/sysutils/fusefs-libs/, and https://github.com/libfuse/libfuse/issues/173, it seems to me that: - A lot of upstream code that was actually intended to support FreeBSD is actually patched out when libfuse is installed via ports. - The mount.fusefs and fusermount binaries are not installed from libfuse at all, and are instead provided by a "sysutils/fusefs-libs" package(?) - Some additional patches are necessary to get libfuse to work. Is that correct so far, or am I looking at the wrong place? If so, my tentative plan would be to: =20 - Not build fusermount and mount.fusefs on FreeBSD at all. This would allow getting rid of mount_bsd.c (and the corresponding patch) completely. =20=20 - Integrate the helper.c patch upstream using #ifdefs - As far as I can tell, the mount_util.[ch] patch is a no-op that should be dropped anyway. Personally, I don't use FreeBSD and I don't have an easy way to test on FreeBSD either. So I would appreciate any input. Best, -Nikolaus --=20 GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F =C2=BBTime flies like an arrow, fruit flies like a Banana.=C2= =AB From owner-freebsd-fs@freebsd.org Thu Aug 3 11:57:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36D5CDB3670; Thu, 3 Aug 2017 11:57:07 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15133743AA; Thu, 3 Aug 2017 11:57:07 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: by freefall.freebsd.org (Postfix, from userid 1235) id 36B7035C1; Thu, 3 Aug 2017 11:57:06 +0000 (UTC) Date: Thu, 3 Aug 2017 13:57:05 +0200 From: Baptiste Daroussin To: freebsd-ports@freebsd.org, freebsd-fs@freebsd.org Subject: Re: State of FUSE on FreeBSD Message-ID: <20170803115705.agsg6dm745dsriab@ivaldir.net> References: <87y3r0ankb.fsf@vostro.rath.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="tyyvnr6ejsjlsjcb" Content-Disposition: inline In-Reply-To: <87y3r0ankb.fsf@vostro.rath.org> User-Agent: NeoMutt/20170609 (1.8.3) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 11:57:07 -0000 --tyyvnr6ejsjlsjcb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 03, 2017 at 01:38:28PM +0200, Nikolaus Rath wrote: > Hello, >=20 > I am the upstream maintainer of libfuse. I'd like to refresh / improve > the FreeBSD support in libfuse. My goal is for libfuse not to require > any FreeBSD specific patches. >=20 > After taking a look at > https://github.com/freebsd/freebsd/tree/master/sbin/mount_fusefs, > https://svn.freebsd.org/ports/head/sysutils/fusefs-libs/, and > https://github.com/libfuse/libfuse/issues/173, it seems to me that: >=20 > - A lot of upstream code that was actually intended to support FreeBSD > is actually patched out when libfuse is installed via ports. >=20 > - The mount.fusefs and fusermount binaries are not installed from > libfuse at all, and are instead provided by a "sysutils/fusefs-libs" > package(?) >=20 > - Some additional patches are necessary to get libfuse to work. >=20 >=20 > Is that correct so far, or am I looking at the wrong place? Yes it it :) >=20 >=20 > If so, my tentative plan would be to: > =20 > - Not build fusermount and mount.fusefs on FreeBSD at all. This would > allow getting rid of mount_bsd.c (and the corresponding patch) > completely. It is correct, we don't need those (not that right now the package fusefs-l= ibs is a bit wrong because it still installs the fusermount manpage beside not installing the utility itself.) > =20 > - Integrate the helper.c patch upstream using #ifdefs Correct >=20 > - As far as I can tell, the mount_util.[ch] patch is a no-op that should > be dropped anyway. Correct the patch are leftovers from the time we didn't have mount_fusefs in base, so they are now "useless" for us. >=20 >=20 > Personally, I don't use FreeBSD and I don't have an easy way to test on > FreeBSD either. So I would appreciate any input. I can help you testing if you need, do not hesitate to bother me :) Best regards, Bapt --tyyvnr6ejsjlsjcb Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgOTj3suS2urGXVU3Y4mL3PG3PloFAlmDD4sACgkQY4mL3PG3 PlpXHBAAqh4BI6yg1K77c7ZX4gxbSk0Lbfn2uDD0j4IWc0dT5Azi+vRB8UKgRXEM vVKSSjHfVaY7O8wwLSMDZrKVAlV8pCGaOrz0kOVxCpEeXu2q2CVaRlksGoGBsDSU VXFBp0YyeifEbR8r+oZUcQ4Ruzd9Hd0YXDVQqq4U+eSQli0bxQ4lz3SxScbfmJbw 9qX9vzueVS/9XfVuxZFJeKuruHlQtzREncOSv/PE5lrYbHOpnn8YZUffMc7J/qNv ofZ2+TMQqoU/A0g99Pqju0vykUyWhByPZ5+q4TJiW79K/rA10y5+yJvXWf0Uo0CZ CLMO6cx+CwknTTvV34pxMwoQjO/it3sUz/3LEpPy2yTq7tzoi3cdnRjmr7K+sDdR vgOXab/lJ8zdfF0bTl/WGh/AgUA2spo50uguRNTu9PLIe2mOMSzE4hp7molGExm1 VEVzh66CZpNHVVGOwyydsmjjq/LXuRxTd+RdeJtVcEGDWhTZE9rEj3h0wQy4Vugv e/M7YHPrlx0CPw4bfTwPX3VHRmsIjwP9CNMqs+UgsFa3dzb46xkWd2f17AYGT98A eiYjW5vb04ZWplLEkY90JzQQUIHdOVAMO89uk6wANgLlslKn7rjvJxKuNpccuysW qSKCz/tM4+W42IxEJg/RNubCAstXsHtbjQiskhnNUOloVC+/jcI= =GLVU -----END PGP SIGNATURE----- --tyyvnr6ejsjlsjcb-- From owner-freebsd-fs@freebsd.org Thu Aug 3 15:23:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCF9CDC321F for ; Thu, 3 Aug 2017 15:23:10 +0000 (UTC) (envelope-from Nikolaus@rath.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8DCF78266B for ; Thu, 3 Aug 2017 15:23:10 +0000 (UTC) (envelope-from Nikolaus@rath.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 0829E20AD3 for ; Thu, 3 Aug 2017 11:23:09 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute1.internal (MEProxy); Thu, 03 Aug 2017 11:23:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s= fm1; bh=KCu9VI/yrpfad3g0HgWUDX8vV4tW9QZvYE7QibYDNSc=; b=TZBWasyh R1uqoN5tDwjb5YFNCyH1oljWJKwTXgAqrDPaJe3JPmoenDP/d+zWBLQ+CprXjyry 30/BmxSO8DnkYpqC+6KPskG+Xhqd09QKxh1+tM9NdpQJhyOBw4Gr655qyuZ1E4He 0niSl4uVvYLgPH0k4X6LdoJ4bx5UcWZSySByMZZorRW+ht71eMJGeNBsDhHXg7Aq WOfXaS40vfIKaLov8pyvhunN51NSFljVkriG9fv3xleRrkIMUeS7lhuwrEyhFaVP skLZFXBGnjkMmpXVZ/5U6Z/aoGwAZdu5YmSxYQZXqFrnZNjUlXrUFlTlCpFeu+aU vzWIsKTLshvjsA== X-ME-Sender: X-Sasl-enc: o0IZxZWH2wqU9JzFUlwyNgq81EnQfy7bHxI9NbtUwCyZ 1501773788 Received: from ebox.rath.org (ebox.rath.org [45.79.69.51]) by mail.messagingengine.com (Postfix) with ESMTPA id B877C7F356 for ; Thu, 3 Aug 2017 11:23:08 -0400 (EDT) Received: from vostro.rath.org (vostro [192.168.12.4]) by ebox.rath.org (Postfix) with ESMTPS id A78CB119 for ; Thu, 3 Aug 2017 15:23:07 +0000 (UTC) Received: by vostro.rath.org (Postfix, from userid 1000) id 648DE1029B3; Thu, 3 Aug 2017 17:23:06 +0200 (CEST) From: Nikolaus Rath To: freebsd-fs@freebsd.org Subject: Re: State of FUSE on FreeBSD References: <87y3r0ankb.fsf@vostro.rath.org> <20170803115705.agsg6dm745dsriab@ivaldir.net> Mail-Copies-To: never Mail-Followup-To: freebsd-fs@freebsd.org Date: Thu, 03 Aug 2017 17:23:06 +0200 In-Reply-To: <20170803115705.agsg6dm745dsriab@ivaldir.net> (Baptiste Daroussin's message of "Thu, 3 Aug 2017 13:57:05 +0200") Message-ID: <87shh8ad5x.fsf@vostro.rath.org> User-Agent: Gnus/5.130014 (Ma Gnus v0.14) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 15:23:10 -0000 On Aug 03 2017, Baptiste Daroussin wrote: > On Thu, Aug 03, 2017 at 01:38:28PM +0200, Nikolaus Rath wrote: >> Hello, >>=20 >> I am the upstream maintainer of libfuse. I'd like to refresh / improve >> the FreeBSD support in libfuse. My goal is for libfuse not to require >> any FreeBSD specific patches. >>=20 [...] >> Personally, I don't use FreeBSD and I don't have an easy way to test on >> FreeBSD either. So I would appreciate any input. > > I can help you testing if you need, do not hesitate to bother me :) Thanks for the pull request! I've merged it and applied some extra fixups on top that seemed right to me. It would be great if you could give it a shot. Code is at https://github.com/libfuse/libfuse, master branch. Best, -Nikolaus --=20 GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F =C2=BBTime flies like an arrow, fruit flies like a Banana.=C2= =AB From owner-freebsd-fs@freebsd.org Thu Aug 3 20:22:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7715DDD170D; Thu, 3 Aug 2017 20:22:14 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 51D716989E; Thu, 3 Aug 2017 20:22:14 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (220-253-171-15.dyn.iinet.net.au [220.253.171.15]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id v73KM8v9094749 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 3 Aug 2017 13:22:11 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: State of FUSE on FreeBSD To: Nikolaus@rath.org References: <87y3r0ankb.fsf@vostro.rath.org> From: Julian Elischer Cc: freebsd-ports@freebsd.org, freebsd-fs@freebsd.org Message-ID: <39c45819-2754-ed73-a0e7-681969f58de7@freebsd.org> Date: Fri, 4 Aug 2017 04:22:03 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <87y3r0ankb.fsf@vostro.rath.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Aug 2017 20:22:14 -0000 On 3/8/17 7:38 pm, Nikolaus Rath wrote: > Hello, > > I am the upstream maintainer of libfuse. I'd like to refresh / improve > the FreeBSD support in libfuse. My goal is for libfuse not to require > any FreeBSD specific patches. Hi, great to hear from you! at $JOB we use libfuse and We have always worried that we don't really have anyone in FreeBSD that really has fuse as a priority. Rick Maclem has looked at it a little, as have a few people (see the commit logs) but generally as long as it more or less works, people tend to steer clear of it. For us libfuse "just worked" so we haven't touched it. We had to touch the kernel part of fuse a little so remove some overly restrictive permission checks, and fix some issues to do with page boundaries (from memory) but nothing in the library. > After taking a look at > https://github.com/freebsd/freebsd/tree/master/sbin/mount_fusefs, > https://svn.freebsd.org/ports/head/sysutils/fusefs-libs/, and > https://github.com/libfuse/libfuse/issues/173, it seems to me that: > > - A lot of upstream code that was actually intended to support FreeBSD > is actually patched out when libfuse is installed via ports. > > - The mount.fusefs and fusermount binaries are not installed from > libfuse at all, and are instead provided by a "sysutils/fusefs-libs" > package(?) that may be possible, We just mount by doing calls to the library in our code so I haven't looked at it. > - Some additional patches are necessary to get libfuse to work. > > > Is that correct so far, or am I looking at the wrong place? > > > If so, my tentative plan would be to: > > - Not build fusermount and mount.fusefs on FreeBSD at all. This would > allow getting rid of mount_bsd.c (and the corresponding patch) > completely. I don't think I've ever seen them used. > > - Integrate the helper.c patch upstream using #ifdefs > > - As far as I can tell, the mount_util.[ch] patch is a no-op that should > be dropped anyway. > > > Personally, I don't use FreeBSD and I don't have an easy way to test on > FreeBSD either. So I would appreciate any input. got vmware or Xen or virtualBox or HyperV? we provide a bunch of virtual images. > > > Best, > -Nikolaus > From owner-freebsd-fs@freebsd.org Sat Aug 5 08:35:42 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 337E4DC6514 for ; Sat, 5 Aug 2017 08:35:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 223FC6B1AB for ; Sat, 5 Aug 2017 08:35:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v758ZewS039850 for ; Sat, 5 Aug 2017 08:35:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Sat, 05 Aug 2017 08:35:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 08:35:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Sat Aug 5 08:35:13 UTC 2017 New revision: 322078 URL: https://svnweb.freebsd.org/changeset/base/322078 Log: MFC r321921: Add compat shim part missed at r305197. This fixes compatibility between old kernel and new ZFS tools. It seems to be tradition to forget it. PR: 221112 Changes: _U stable/11/ stable/11/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_compat.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 5 08:36:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5518DDC6674 for ; Sat, 5 Aug 2017 08:36:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 438C26B392 for ; Sat, 5 Aug 2017 08:36:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v758aipr041344 for ; Sat, 5 Aug 2017 08:36:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Sat, 05 Aug 2017 08:36:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 08:36:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 --- Comment #6 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Sat Aug 5 08:36:04 UTC 2017 New revision: 322079 URL: https://svnweb.freebsd.org/changeset/base/322079 Log: MFC r321921: Add compat shim part missed at r305197. This fixes compatibility between old kernel and new ZFS tools. It seems to be tradition to forget it. PR: 221112 Changes: _U stable/10/ stable/10/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_compat.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 5 08:45:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 889EADC6F52 for ; Sat, 5 Aug 2017 08:45:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 760976BBD3 for ; Sat, 5 Aug 2017 08:45:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v758jb8r067154 for ; Sat, 5 Aug 2017 08:45:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221112] zfs(8)/zpool(8) backward compatibility broken Date: Sat, 05 Aug 2017 08:45:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 08:45:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221112 Alexander Motin changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|In Progress |Closed --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 5 16:58:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F0B5DBD9E9 for ; Sat, 5 Aug 2017 16:58:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D8437F67A for ; Sat, 5 Aug 2017 16:58:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v75GwWTA057650 for ; Sat, 5 Aug 2017 16:58:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 221189] zfs: bin/chmod/chmod_test:f_flag fails on ZFS, not UFS; UF_IMMUTABLE not supported on ZFS Date: Sat, 05 Aug 2017 16:58:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 16:58:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221189 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: ngie Date: Sat Aug 5 16:58:02 UTC 2017 New revision: 322101 URL: https://svnweb.freebsd.org/changeset/base/322101 Log: Don't check result of chflags in f_flag_cleanup() This will prevent false positives from occurring if the test is run on ZFS since ZFS doesn't support fflags throbbing like UFS. PR: 221189 MFC after: 4 days MFC with: r321949 Changes: head/bin/chmod/tests/chmod_test.sh --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Aug 5 17:09:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9B6EDBEA15; Sat, 5 Aug 2017 17:09:14 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A974803B1; Sat, 5 Aug 2017 17:09:13 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (dynamic-2a02-2698-26-0-0.perm.ertelecom.ru [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (may be forged)) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v75H98jY007140 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 5 Aug 2017 22:09:09 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1501952949; bh=9rFsB6tx5omdomCj49fVy8H9H3xUovof3EKN1OS6VfE=; h=To:Cc:From:Subject:Date; b=Ehb/WDtHVBwMjIZN6CeA32kP/pn804AMyz1iowH0oyuqiuLu4Jv6imnQQEPQXEJWN beNvNmM/pSdDNoDTYt7H3+390FqVWRXWvuIA7QaqFYJJl599rlfNaZQ/76TxLQH9mD aAeyCEAcPLzD71CWaAOnjmv3QPHjSwYQmkTWOtk4= To: freebsd-stable@FreeBSD.org Cc: freebsd-fs@freebsd.org From: "Eugene M. Zheganin" Subject: a strange and terrible saga of the cursed iSCSI ZFS SAN Message-ID: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> Date: Sat, 5 Aug 2017 22:08:54 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:09:15 -0000 Hi, I got a problem that I cannot solve just by myself. I have a iSCSI zfs SAN system that crashes, corrupting it's data. I'll be short, and try to describe it's genesis shortly: 1) autumn 2016, SAN is set up, supermicro server, external JBOD, sandisk ssds, several redundant pools, FreeBSD 11.x (probably release, don't really remember - see below). 2) this is working just fine until early spring 2017 3) system starts to crash (various panics): panic: general protection fault panic: page fault panic: Solaris(panic): zfs: allocating allocated segment(offset=6599069589504 size=81920) panic: page fault panic: page fault panic: Solaris(panic): zfs: allocating allocated segment(offset=8245779054592 size=8192) panic: page fault panic: page fault panic: page fault panic: Solaris(panic): zfs: allocating allocated segment(offset=1792100934656 size=46080) 4) we memtested it immidiately, no problems found. 5) we switch sandisks to toshibas, we switch also the server to an identical one, JBOD to an identical one, leaving same cables. 6) crashes don't stop. 7) we found that field engineers physically damaged (sic!) the SATA cables (main one and spare ones), and that 90% of the disks show ICRC SMART errors. 8) we replaced the cable (brand new HP one). 9) ATA SMART errors stopped increasing. 10) crashes continue. 11) we decided that probably when ZFS was moved over damaged cables between JBODs it was somehow damaged too, so now it's panicking because of that. so we wiped the data completely, reinitialized the SAN system and put it back into the production. we even dd'ed each disk with zeroes (!) - just in case. Important note: the data was imported using zfs send from another, stable system that is runing in production in another DC. 12) today we got another panic. btw the pools look now like this: # zpool status -v pool: data state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 62 raidz1-0 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 raidz1-2 ONLINE 0 0 62 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 da14 ONLINE 0 0 0 da15 ONLINE 0 0 0 da16 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: data/userdata/worker208:<0x1> pool: userdata state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM userdata ONLINE 0 0 216K mirror-0 ONLINE 0 0 432K gpt/userdata0 ONLINE 0 0 432K gpt/userdata1 ONLINE 0 0 432K errors: Permanent errors have been detected in the following files: userdata/worker36:<0x1> userdata/worker30:<0x1> userdata/worker31:<0x1> userdata/worker35:<0x1> 12) somewhere between p.5 and p.10 the pool became deduplicated (not directly connected to the problem, just for production reasons). So, concluding: we had bad hardware, we replaced EACH piece (server, adapter, JBOD, cable, disks), and crashes just don't stop. We have 5 another iSCSI SAN systems, almost fully identical that don't crash. Crashes on this particular system began when it was running same set of versions that stable systems. So, besides calling an exorcist, I would like to hear what other options do I have, I really would. And I want to also ask - what happens when the system's memory isn't enough for deduplication - does it crash, or does the problem of mounting the pool appear, like some articles mention ? This message could been encumbered by junky data like the exact FreeBSD releases we ran (asuuming it's normal for some 11.x revisions to crash and damage the data, and some - not, which I believe it's a nonsense), by the pool configurations and disk lists (assuming the same - that you can provoque data loss by some redundant pool configurations - not considering raidz with more than 5 disks - which I believe is not true), and so on. but I decided not to include this until requested. And as I also said, we have 5 another SAN systems running similar/identical configurations without major problems. Thanks. Eugene. From owner-freebsd-fs@freebsd.org Sat Aug 5 17:17:09 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 769C0DBF304; Sat, 5 Aug 2017 17:17:09 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EA57880A1E; Sat, 5 Aug 2017 17:17:08 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (dynamic-2a02-2698-26-0-0.perm.ertelecom.ru [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (may be forged)) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v75HH5Od007451 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 5 Aug 2017 22:17:05 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1501953426; bh=5Rj0MOK5Zp+fXqvmVRPQ3oiWuRY57fHpAHlIRc8XapQ=; h=Subject:From:To:Cc:References:Date:In-Reply-To; b=eAiV6zEnd35O/25aq3n+amRLkDFFOrjd7dP6+YqVz4GnUuTChRkd5V6x5aADeGqe9 S9B064BPGb0gFNwPFuTwjJD82jHrHPHJtxYIr17TYtTc4tOtgPifQuuQxPN7aK0ScU H1UykqzMMUpcXAbRCBeXrlaiszUfF5hZqXcYHOeI= Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN From: "Eugene M. Zheganin" To: freebsd-stable@FreeBSD.org Cc: freebsd-fs@freebsd.org References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> Message-ID: <1d53f489-5135-7633-fef4-35d26e4969dc@norma.perm.ru> Date: Sat, 5 Aug 2017 22:16:51 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:17:09 -0000 Hi, On 05.08.2017 22:08, Eugene M. Zheganin wrote: > > pool: userdata > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://illumos.org/msg/ZFS-8000-8A > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > userdata ONLINE 0 0 216K > mirror-0 ONLINE 0 0 432K > gpt/userdata0 ONLINE 0 0 432K > gpt/userdata1 ONLINE 0 0 432K That would be funny, if not that sad, but while writing this message, the pool started to look like below (I just asked zpool status twice in a row, comparing to what it was): [root@san1:~]# zpool status userdata pool: userdata state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM userdata ONLINE 0 0 728K mirror-0 ONLINE 0 0 1,42M gpt/userdata0 ONLINE 0 0 1,42M gpt/userdata1 ONLINE 0 0 1,42M errors: 4 data errors, use '-v' for a list [root@san1:~]# zpool status userdata pool: userdata state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM userdata ONLINE 0 0 730K mirror-0 ONLINE 0 0 1,43M gpt/userdata0 ONLINE 0 0 1,43M gpt/userdata1 ONLINE 0 0 1,43M errors: 4 data errors, use '-v' for a list So, you see, the error rate is like speed of light. And I'm not sure if the data access rate is that enormous, looks like they are increasing on their own. So may be someone have an idea on what this really means. Thanks. Eugene. From owner-freebsd-fs@freebsd.org Sat Aug 5 17:22:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D98F1DBF8B8 for ; Sat, 5 Aug 2017 17:22:50 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22a.google.com (mail-wr0-x22a.google.com [IPv6:2a00:1450:400c:c0c::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6B48380E9D for ; Sat, 5 Aug 2017 17:22:50 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22a.google.com with SMTP id k71so26263148wrc.2 for ; Sat, 05 Aug 2017 10:22:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=pjKajy8Sd1TejAbrnMSJaTlsRknzgRNARDjKDpG0R/E=; b=pQlDpvR8Pt0noNjAlVxAED8QVgvnl62bl/MUiGRokkEkX5DOWXDwPz6btiTWy4jJgJ NnCLYC6ElhLSfmUZAiiZ5PiuADPdtzqef3cp/QU+4RU5NUWnS0jPQwTHGp1F5mc4oNGz R5zMr9WK+7jeAkLHLPUQsekKb19LFM44lhbN0I0DhU/zfKjiPkqLTq11lFvrF4lUwJ0E iiPQ1UYt5rJ5eTfVIgJpD3kmmace2GWPPurwG3B872xs++yxLIheC/fUl3S+111W58s7 d3xUQLGb801BW886Hx03FYmcyvLS18GRoDb+OwfHxVnkksXEltUPuIi1BPIEig/cwF5l MPpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=pjKajy8Sd1TejAbrnMSJaTlsRknzgRNARDjKDpG0R/E=; b=cD8VrnHFfagrrB0c39dLL5cWCGWGUWC1sxrNIShhJI7QwyD1KtS67c8pJ79gMh+JyR yjH8rtehEUSOWpFdKxwbZJ5TkjMSrEg4tjnRjHTA2MD103+N3ZKb/2Eyc9yFqgBll9Ug l7JRbl6TEdIHV0VxPeaqsvKAIs1nuO6+MuDOE5WYndUrQsr4aA1niTHxnmCYKMnPc4eT 3P/F/pXvHy+cJYRZaUtHHN8wHZwkMjx3PZ78HH570wOw5393mP8RPbIr6yCPvLBe/ohM TkS++OHBPih9Pnbc3eaO7wERKkWxhU8Nz6mSyGlkMBFAyCoL6xlOxIJPahbc9fvScUB2 Z07w== X-Gm-Message-State: AIVw111HRQClmlrlwLWj2ONgtUQaZ47qL0Qho6XMcmPbneT6l3iYiTOm 8DqfzJXGgWvIluYub1E= X-Received: by 10.223.174.90 with SMTP id u26mr4132686wrd.237.1501953768426; Sat, 05 Aug 2017 10:22:48 -0700 (PDT) Received: from ben.home (LFbn-1-6951-179.w90-116.abo.wanadoo.fr. [90.116.132.179]) by smtp.gmail.com with ESMTPSA id 77sm6943823wmk.8.2017.08.05.10.22.47 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 05 Aug 2017 10:22:47 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN From: Ben RUBSON In-Reply-To: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> Date: Sat, 5 Aug 2017 19:22:46 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <8D11D326-6611-4798-AE3E-9411200A9556@gmail.com> References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:22:50 -0000 > On 05 Aug 2017, at 19:08, Eugene M. Zheganin = wrote: >=20 > (...) >=20 > 1) autumn 2016, SAN is set up, supermicro server, external JBOD, = sandisk ssds, several redundant pools, FreeBSD 11.x (probably release, = don't really remember - see below). Hi Eugene, Which SAS adapter ? SAS 2008 ? Ben= From owner-freebsd-fs@freebsd.org Sat Aug 5 17:49:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E4BADC0B7E; Sat, 5 Aug 2017 17:49:41 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qt0-x22c.google.com (mail-qt0-x22c.google.com [IPv6:2607:f8b0:400d:c0d::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBA65818F2; Sat, 5 Aug 2017 17:49:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qt0-x22c.google.com with SMTP id t37so23413542qtg.5; Sat, 05 Aug 2017 10:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=0R6LJ9fGoWE4JTkpSXDZaeltyd+X4nKwBC0ysqsqVK0=; b=Bi8oS1icEWVGaNit04OMI3AbGS2BPw3dWAjqAciodvdMn7E85C6+0w0cylbzM8aCMh dvyNFjhlUDHcjdJXvXnonI8H6N9SYS8F8/qJDDhz+nseXlitfjdK75uePGulQjgqJbJn e6JSlgX0XALN+CtHKEX3MlJy8sXhxgNWsPSH5diOsA+DppcNoBCeHJnecg8Y/AOZzPJu OGV3eiPvBVx4ERy6A86YgOwY4lxHPGdn/W2Lrn+/mdoEwQQhzfJuJa0eZUSD7BKJwR6y Lm+ChhR96o6UxmClIKY17Bik8oS+XFE0aC95QC84HQ3wz4RJoQz3dbW0zGrYtZ1TbKDG TPPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=0R6LJ9fGoWE4JTkpSXDZaeltyd+X4nKwBC0ysqsqVK0=; b=mkIxkBg4TKzREANqFmOkGSc19gNYxdSUnDfmz30CuUKmxALRaNRBlWB13ThqtC1GhZ RmsQFS6kp9ArWrKd38ZugTKO2eMcAhGx/nDWMI/Ymsc0H4H8J/HSJ+pTdlO/QPxVUaFC /QbnoCjoahMcYhF3H0ehZz9DJLhr/ssikwjChCGK4Dm+H172NT3yJgYf4+qe8+z592mj 7SwmV8vm+7ZqP0iecEXX3h5gWLPKdsJscy3UlgikFs5mQMAzGS1fLjFmiixQgvekJZYr voyQtiY/HIF9SRGqq26yMfJqa4fNuW5ckMsPRgG9RDjBby7yWRuQivGvWNSRMl9waxud 1TiA== X-Gm-Message-State: AHYfb5ikPTW5AbjkGWntSB/2iBrDTJS/WM/lZuJajy+qy6HZYYbk3+9o V4xj5bmSBfIL3YGMVPTqIa8Nn77WXA== X-Received: by 10.237.42.88 with SMTP id k24mr8782754qtf.58.1501955379916; Sat, 05 Aug 2017 10:49:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.32.136 with HTTP; Sat, 5 Aug 2017 10:49:39 -0700 (PDT) Received: by 10.140.32.136 with HTTP; Sat, 5 Aug 2017 10:49:39 -0700 (PDT) In-Reply-To: References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> From: Freddie Cash Date: Sat, 5 Aug 2017 10:49:39 -0700 Message-ID: Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN To: "Eugene M. Zheganin" Cc: FreeBSD Stable , FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:49:41 -0000 On Aug 5, 2017 10:09 AM, "Eugene M. Zheganin" wrote: And I want to also ask - what happens when the system's memory isn't enough for deduplication - does it crash, or does the problem of mounting the pool appear, like some articles mention ? Can't really help with the other issues, but can speak to this one. "Insufficient" RAM for dedupe won't be an issue during normal operation, reading and writing to the pool. Performance will slow down over time as the DDT increases in size taking up ARC/L2ARC space, possibly even spilling over into raw disk reads. Where the issues crop up is during snapshot deletion and filesystem destruction. Those operations can run the system out of usable kmem and lock things up. (Deletions cause a lot of churn in the DDT.) On reboot, the pool import process will continue the deletion process, which will run the system out of kmen locking it up. Rinse and repeat for a week until the deletion process completes. L2ARC helps, but not nearly as much as adding more RAM! Device replacement will also be an issue as the resilver process will be extremely slow (at least, it is for us with a very full, very fragmented pool; equally full pools without d we dedupe resilver very quickly). And don't try to replace two drives in separate vdevs unless you want to increase the resilver time by a huge factor. This is from experience with a 24-drive system with only 48 GB of RAM, and a 90-disk system with 128 GB of RAM. Both with dedupe enabled. Both running at about 90% full, very fragmented as they create and delete snapshots of all filesystems on a daily basis. They've been running file about 5 years now, possibly longer. Cheers, Freddie From owner-freebsd-fs@freebsd.org Sat Aug 5 17:50:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 501C0DC0D31 for ; Sat, 5 Aug 2017 17:50:45 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B734581ACA for ; Sat, 5 Aug 2017 17:50:44 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (dynamic-2a02-2698-26-0-0.perm.ertelecom.ru [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (may be forged)) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v75HofZc008602 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Sat, 5 Aug 2017 22:50:41 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1501955441; bh=7ngH7kknh+7fROeZMyEoocvkbibi8uwICWJUsLPOOzg=; h=Subject:To:References:From:Date:In-Reply-To; b=UcJeWrmwdjuez5oZ43FxKfwimN7efYTG3EAr5qH4jJYGwbFztiBMr96IV62fz3r7J 7ww9f39rF+d+mbgYt4Dnc+kZg3uCWe8mtwl1GChRLATJVtdVECnoiSgvAjE8zUzoO1 rObhe2c+89NBkpKCdRRDawqiEEMfzjdHAZlceZGs= Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN To: freebsd-fs@freebsd.org References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> <8D11D326-6611-4798-AE3E-9411200A9556@gmail.com> From: "Eugene M. Zheganin" Message-ID: <25a4d73c-698e-38bd-7957-f2895a01e351@norma.perm.ru> Date: Sat, 5 Aug 2017 22:50:27 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <8D11D326-6611-4798-AE3E-9411200A9556@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:50:45 -0000 Hi, On 05.08.2017 22:22, Ben RUBSON wrote: >> On 05 Aug 2017, at 19:08, Eugene M. Zheganin wrote: >> >> (...) >> >> 1) autumn 2016, SAN is set up, supermicro server, external JBOD, sandisk ssds, several redundant pools, FreeBSD 11.x (probably release, don't really remember - see below). > Hi Eugene, > > Which SAS adapter ? > SAS 2008 ? Exactly. What's wrong with it ? I'm running latest (or at least not that old) firmware on em. Eugene. From owner-freebsd-fs@freebsd.org Sat Aug 5 17:52:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E991BDC1001; Sat, 5 Aug 2017 17:52:35 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay07.ispgateway.de (smtprelay07.ispgateway.de [134.119.228.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE52281DA8; Sat, 5 Aug 2017 17:52:35 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.133.192] (helo=fabiankeil.de) by smtprelay07.ispgateway.de with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1de3FR-0006iJ-Oq; Sat, 05 Aug 2017 19:52:25 +0200 Date: Sat, 5 Aug 2017 19:51:44 +0200 From: Fabian Keil To: "Eugene M. Zheganin" Cc: freebsd-stable@FreeBSD.org, freebsd-fs@freebsd.org Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN Message-ID: <20170805195144.1caf98dc@fabiankeil.de> In-Reply-To: <1d53f489-5135-7633-fef4-35d26e4969dc@norma.perm.ru> References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> <1d53f489-5135-7633-fef4-35d26e4969dc@norma.perm.ru> Reply-To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/iPaWgOy.Hpp7k6QXYIQOmL8"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:52:36 -0000 --Sig_/iPaWgOy.Hpp7k6QXYIQOmL8 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable "Eugene M. Zheganin" wrote: > On 05.08.2017 22:08, Eugene M. Zheganin wrote: > > > > pool: userdata > > state: ONLINE > > status: One or more devices has experienced an error resulting in data > > corruption. Applications may be affected. > > action: Restore the file in question if possible. Otherwise restore the > > entire pool from backup. > > see: http://illumos.org/msg/ZFS-8000-8A > > scan: none requested > > config: > > > > NAME STATE READ WRITE CKSUM > > userdata ONLINE 0 0 216K > > mirror-0 ONLINE 0 0 432K > > gpt/userdata0 ONLINE 0 0 432K > > gpt/userdata1 ONLINE 0 0 432K =20 > That would be funny, if not that sad, but while writing this message,=20 > the pool started to look like below (I just asked zpool status twice in=20 > a row, comparing to what it was): >=20 > [root@san1:~]# zpool status userdata > pool: userdata > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://illumos.org/msg/ZFS-8000-8A > scan: none requested > config: >=20 > NAME STATE READ WRITE CKSUM > userdata ONLINE 0 0 728K > mirror-0 ONLINE 0 0 1,42M > gpt/userdata0 ONLINE 0 0 1,42M > gpt/userdata1 ONLINE 0 0 1,42M >=20 > errors: 4 data errors, use '-v' for a list > [root@san1:~]# zpool status userdata > pool: userdata > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://illumos.org/msg/ZFS-8000-8A > scan: none requested > config: >=20 > NAME STATE READ WRITE CKSUM > userdata ONLINE 0 0 730K > mirror-0 ONLINE 0 0 1,43M > gpt/userdata0 ONLINE 0 0 1,43M > gpt/userdata1 ONLINE 0 0 1,43M >=20 > errors: 4 data errors, use '-v' for a list >=20 > So, you see, the error rate is like speed of light. And I'm not sure if=20 > the data access rate is that enormous, looks like they are increasing on= =20 > their own. > So may be someone have an idea on what this really means. Quoting a comment from sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_m= isc.c: /* * If destroy encounters an EIO while reading metadata (e.g. indirect * blocks), space referenced by the missing metadata can not be freed. * Normally this causes the background destroy to become "stalled", as * it is unable to make forward progress. While in this stalled state, * all remaining space to free from the error-encountering filesystem is * "temporarily leaked". Set this flag to cause it to ignore the EIO, * permanently leak the space from indirect blocks that can not be read, * and continue to free everything else that it can. * * The default, "stalling" behavior is useful if the storage partially * fails (i.e. some but not all i/os fail), and then later recovers. In * this case, we will be able to continue pool operations while it is * partially failed, and when it recovers, we can continue to free the * space, with no leaks. However, note that this case is actually * fairly rare. * * Typically pools either (a) fail completely (but perhaps temporarily, * e.g. a top-level vdev going offline), or (b) have localized, * permanent errors (e.g. disk returns the wrong data due to bit flip or * firmware bug). In case (a), this setting does not matter because the * pool will be suspended and the sync thread will not be able to make * forward progress regardless. In case (b), because the error is * permanent, the best we can do is leak the minimum amount of space, * which is what setting this flag will do. Therefore, it is reasonable * for this flag to normally be set, but we chose the more conservative * approach of not setting it, so that there is no possibility of * leaking space in the "partial temporary" failure case. */ In FreeBSD the "flag" currently isn't easily reachable due to the lack of a powerful kernel debugger (like mdb in Solaris offsprings) but it can be made reachable with a sysctl using the patch from: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218954 Fabian --Sig_/iPaWgOy.Hpp7k6QXYIQOmL8 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCWYYFsQAKCRAFiohV/3dU nd3AAJ94LgHj630WLpNwyH3SKQj2l6hF9ACgqm2KgnEqE0xGYO0wswxBFpktykA= =hqGa -----END PGP SIGNATURE----- --Sig_/iPaWgOy.Hpp7k6QXYIQOmL8-- From owner-freebsd-fs@freebsd.org Sat Aug 5 18:00:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89E23DC15D8 for ; Sat, 5 Aug 2017 18:00:12 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F0FF882032 for ; Sat, 5 Aug 2017 18:00:11 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (dynamic-2a02-2698-26-0-0.perm.ertelecom.ru [IPv6:2a02:2698:26:9b0e:d7a:6d28:b07e:5ba6] (may be forged)) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v75I08gP008963 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Sat, 5 Aug 2017 23:00:08 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1501956009; bh=yNvzxBRAIxy7GG0sKWX+52QLL8t7/o4kOH8u7xprkAc=; h=Subject:From:To:References:Date:In-Reply-To; b=WiA7mZz41o9nTJyUZqsoLCHJQdcI3lOI0mjNeyy54ZWpw6NPInSVEB00XyeTUEiKP 6yxSUKSQrZCcDrnUs32OxZEID8igG4I2yAST7sfrnTaZJXFDdS+2w6enarNHUO7NDh +hJq6JsYyll2qi+CxJTD2zIkzZfMjX1Mb1JzJ5Bs= Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN From: "Eugene M. Zheganin" To: freebsd-fs@freebsd.org References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> <8D11D326-6611-4798-AE3E-9411200A9556@gmail.com> <25a4d73c-698e-38bd-7957-f2895a01e351@norma.perm.ru> Message-ID: <8ff8d88c-ec9a-5586-b131-c9bc1fe17d95@norma.perm.ru> Date: Sat, 5 Aug 2017 22:59:54 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <25a4d73c-698e-38bd-7957-f2895a01e351@norma.perm.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 18:00:12 -0000 Hi, On 05.08.2017 22:50, Eugene M. Zheganin wrote: > Hi, > > On 05.08.2017 22:22, Ben RUBSON wrote: >>> On 05 Aug 2017, at 19:08, Eugene M. Zheganin wrote: >>> >>> (...) >>> >>> 1) autumn 2016, SAN is set up, supermicro server, external JBOD, >>> sandisk ssds, several redundant pools, FreeBSD 11.x (probably >>> release, don't really remember - see below). >> Hi Eugene, >> >> Which SAS adapter ? >> SAS 2008 ? > Exactly. What's wrong with it ? I'm running latest (or at least not > that old) firmware on em. > > No, wait, my bad, it's LSISAS3008. mpr(4) one. Eugene. From owner-freebsd-fs@freebsd.org Sat Aug 5 18:20:37 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D9E3DC2686 for ; Sat, 5 Aug 2017 18:20:37 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E215B82A02 for ; Sat, 5 Aug 2017 18:20:36 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id t201so41079422wmt.1 for ; Sat, 05 Aug 2017 11:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=TL19KXzkSIWLXTJNP07eHGl8EWGYo7oU0xKTKaAjlRM=; b=lnC6g8qwyNiD0l+jm48QCTdG7Bi925E9TVoplwN6aesnycGgpSEDwkdKtSZlhJJR6l Cqig2KjR81jvjSt0iMbVxwhf0so5O1H/dBYjYmGhvCeH7mhLsoT1OZv4QHImAg6KrzRb CmtcTDxQP2AIlS9Z45ve3wxar6WWHwpK8X9aMYvNFIyiEEUijrSzOKt0Tbc+vuobnEEb kYXsZFNUNGFn29ueC+A52Rr0WThYubU66WQXx4PmYws6JsLiXP4LU0hbuaNUUUtuaz30 HIUKa/CRFAv2XBGCoPjzUPSvaxGFcW/mXVhMHKTO5uzk7UR6UIrimKTmvtb0S/6aFA+Y owDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=TL19KXzkSIWLXTJNP07eHGl8EWGYo7oU0xKTKaAjlRM=; b=PfJ3NWG2PuekZ2cou/BBMVd/8hcWfQpxhaI2KaA+l7b9muVBwFvffpk0xNLK0qEnJE JU9aES78l5tC52pgPEob3Ox7M9G4zFiyUzhvS4KDmatCL4MacgP2bNZFzpLyEoDb8z0F q1BeZ2YhslFk/UXttWWeKjEVASlz0/95Uh7s0QruImRRBmyALKMrOP4sMN59Ct0DXwE6 rQ1gu3b+VHyt1EbWgsj47aqN925roVNv6vCzRDUHBe2I1eoRCmaD4UTZKl37KKMMNCsB lfJH5MJ30R1iJwO9gOXbvtQH250EQK/8hupeyheLhWjNFkK+sCZOeHFd6oimGpek7UMg 3sag== X-Gm-Message-State: AHYfb5hF3m4pE0tV+ONP3MpXavp0g7S51G4SRiEVBidrhFVQ2d4abH3C lNMRVap5z6Bqe2/Le90= X-Received: by 10.28.187.86 with SMTP id l83mr3274206wmf.165.1501957234448; Sat, 05 Aug 2017 11:20:34 -0700 (PDT) Received: from ben.home (LFbn-1-6951-179.w90-116.abo.wanadoo.fr. [90.116.132.179]) by smtp.gmail.com with ESMTPSA id g17sm6214534wrc.28.2017.08.05.11.20.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 05 Aug 2017 11:20:33 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN From: Ben RUBSON In-Reply-To: <8ff8d88c-ec9a-5586-b131-c9bc1fe17d95@norma.perm.ru> Date: Sat, 5 Aug 2017 20:20:32 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <7F068B61-6167-4C58-A925-7956B1D023D6@gmail.com> References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> <8D11D326-6611-4798-AE3E-9411200A9556@gmail.com> <25a4d73c-698e-38bd-7957-f2895a01e351@norma.perm.ru> <8ff8d88c-ec9a-5586-b131-c9bc1fe17d95@norma.perm.ru> To: Freebsd fs X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 18:20:37 -0000 > On 05 Aug 2017, at 19:59, Eugene M. Zheganin = wrote: >=20 > Hi, >=20 > On 05.08.2017 22:50, Eugene M. Zheganin wrote: >> Hi, >>=20 >> On 05.08.2017 22:22, Ben RUBSON wrote: >>>> On 05 Aug 2017, at 19:08, Eugene M. Zheganin = wrote: >>>>=20 >>>> (...) >>>>=20 >>>> 1) autumn 2016, SAN is set up, supermicro server, external JBOD, = sandisk ssds, several redundant pools, FreeBSD 11.x (probably release, = don't really remember - see below). >>> Hi Eugene, >>>=20 >>> Which SAS adapter ? >>> SAS 2008 ? >> Exactly. What's wrong with it ? I'm running latest (or at least not = that old) firmware on em. >>=20 > No, wait, my bad, it's LSISAS3008. mpr(4) one. Nothing wrong but things such as verifying the firmware version. Certainly you use the same (IT ?) version on all your 6 HBA. Not sure about the 3008, but we had a discussion on this list some = months ago about 2008 firmwares, some were said to be bad. You could also enable some HBA debugging, we never know : sysctl dev.mpr.0.debug_level=3D0x1B Ben