From owner-freebsd-fs@freebsd.org Sun Jun 11 21:00:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 407D5B94C11 for ; Sun, 11 Jun 2017 21:00:47 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 36AD5768C0 for ; Sun, 11 Jun 2017 21:00:47 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5BL012l064872 for ; Sun, 11 Jun 2017 21:00:47 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201706112100.v5BL012l064872@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 11 Jun 2017 21:00:47 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jun 2017 21:00:47 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Jun 12 06:13:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53C82BF4D9A for ; Mon, 12 Jun 2017 06:13:53 +0000 (UTC) (envelope-from amutu@amutu.com) Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0867883B6B for ; Mon, 12 Jun 2017 06:13:52 +0000 (UTC) (envelope-from amutu@amutu.com) Received: by mail-oi0-x233.google.com with SMTP id e11so9333554oia.2 for ; Sun, 11 Jun 2017 23:13:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amutu-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=EMY7hwj8VduAIfatVYPv/s+Tmyp3dT5M+0FOgsPaaV4=; b=d+XOO0Lab2LqlhmRaq6YJDQUpp1WCckXj92gbILuMp7zVbq7/72ima+FEuTHpMX9Ww YP3b5sDvLPs0hNfABAY2l77gEtWPkCAhWNea26HxOHthIbzEuEYHQIMxdr/xMPwRe1i3 IyU1/X1APiUsIfZPw0iQjHU9RZSqXcr74RvgeY3cdLQkCYTemPVTIQf5wrTFD4GlLmKn MovwjRb8YF+P9nByBVJ2G05+DfAC1tDIrcyLQnfYMw4L4fCTz9O8v0lT0/zTPzCB4k2D tSLVWgGunaIUFwXX3pv/+8XUCzEfNyuxJSZ7s0je5IAvMVF0OzEytxlFZlbjvRQ2z+P0 UOwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=EMY7hwj8VduAIfatVYPv/s+Tmyp3dT5M+0FOgsPaaV4=; b=kuMWnqFYJ2T76PfLwq0uJ5OMB9qBukM//obiGXQvTJT3sKWAe6nZKuO2LPN3n4KoH6 F9bRs7yZVOvCz/Eg5At46/gMm4DxT44izd+xPl+MCxyaj0aPr4mjTxWiULj0YLWHw5au r115gvI02fptQ660A5gjNQDX4+c4bITYsp9y1K4gbTzWbFfaTuCnHWkX/x0DOs3kw1f6 X15fFDEfCiQv2tdniSu0bR7vBKaM+BxV9MrWHOkTiBZwiAHYdXrwxqZHhn1rYlkCP2Bz pgqarFxBr8hBtpymFvn8lq7yaLRXPhWLkiCPDcsTCCXeyyEQv/WFGJxA82dj9FI+ls0d e9nQ== X-Gm-Message-State: AODbwcA5wWe4JFCb90raOQUlZrR3znMPs3uVCzseExK+2wieuIHburjY JESH8PIuDIVkG6if X-Received: by 10.202.69.6 with SMTP id s6mr27176694oia.193.1497248031982; Sun, 11 Jun 2017 23:13:51 -0700 (PDT) Received: from mail-oi0-f43.google.com (mail-oi0-f43.google.com. [209.85.218.43]) by smtp.gmail.com with ESMTPSA id c42sm4496772otc.58.2017.06.11.23.13.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 11 Jun 2017 23:13:51 -0700 (PDT) Received: by mail-oi0-f43.google.com with SMTP id s64so6241453oif.1; Sun, 11 Jun 2017 23:13:51 -0700 (PDT) X-Received: by 10.202.227.3 with SMTP id a3mr1852330oih.52.1497248031209; Sun, 11 Jun 2017 23:13:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.133.136 with HTTP; Sun, 11 Jun 2017 23:13:30 -0700 (PDT) In-Reply-To: <4410c303314a4d11832a8d248e0b53e1@DM2PR58MB013.032d.mgd.msft.net> References: <4410c303314a4d11832a8d248e0b53e1@DM2PR58MB013.032d.mgd.msft.net> From: Jov Date: Mon, 12 Jun 2017 14:13:30 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD performance drop < 24 hours To: "Caza, Aaron" Cc: "freebsd-hackers@freebsd.org" , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jun 2017 06:13:53 -0000 >From the output of explain analyze of PG, the problem can be excluded from the database.I am not a fs expert, I CCed freebsd-fs@freebsd.org.It may be helpful if you provide more info such as sysctl -a | grep zfs after degradation. 2017-06-12 12:50 GMT+08:00 Caza, Aaron : > Thanks, Jov, for you suggestions. Per your e-mail I added =E2=80=9Cexpla= in > analyze=E2=80=9D to the script: > > > > #!/bin/sh > psql --username=3Dtest --password=3Dsupersecret -h /db -d test << EOL > \timing on > explain analyze select count(*) from test; > \q > EOL > > Sample run of above script before degradation: > > Timing is on. > > QUERY > PLAN > > ------------------------------------------------------------ > ------------------------------------------------------------------------ > > Aggregate (cost=3D3350822.35..3350822.36 rows=3D1 width=3D0) (actual > time=3D60234.556..60234.556 rows=3D1 loops=3D1) > > -> Seq Scan on test (cost=3D0.00..3296901.08 rows=3D21568508 width= =3D0) > (actual time=3D1.126..57021.470 rows=3D21568508 loops=3D1) > > Planning time: 4.968 ms > > Execution time: 60234.649 ms > > (4 rows) > > > > Time: 60248.503 ms > > test$ uptime > > 10:33PM up 7 mins, 3 users, load averages: 1.68, 1.79, 0.94 > > > > > > Sample run of above script after degradation (~11.33 hours uptime): > > Timing is on. > > QUERY > PLAN > > ------------------------------------------------------------ > ------------------------------------------------------------------------- > > Aggregate (cost=3D3350822.35..3350822.36 rows=3D1 width=3D0) (actual > time=3D485669.361..485669.361 rows=3D1 loops=3D1) > > -> Seq Scan on test (cost=3D0.00..3296901.08 rows=3D21568508 width= =3D0) > (actual time=3D0.008..483241.253 rows=3D21568508 loops=3D1) > > Planning time: 0.529 ms > > Execution time: 485669.411 ms > > (4 rows) > > > > Time: 485670.432 ms > > test$ uptime > > 9:59PM up 11:21, 2 users, load averages: 1.11, 2.13, 2.14 > > > > > > Regarding dd=E2=80=99ing the pgdata directory, that didn=E2=80=99t work f= or me as Postgres > splits the database up into multiple 2GB files =E2=80=93 dd=E2=80=99ing o= f a 2GB file on a > system with 8GB ram doesn=E2=80=99t seem representative. I opted to crea= te a 16GB > file (dd if=3D/dev/random of=3D/testdb/test bs=3D1m count=3D16000) on th= e > pertinent ZFS file system then performed dd operation on that: > > > > Sample of run after degradation (~11.66 hours uptime): > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 274.841792 secs (61043176 bytes/sec) > > test$ uptime > > 10:25PM up 11:46, 2 users, load averages: 1.00, 1.28, 1.59 > > > > > > After rebooting, we can see **MUCH** before performance: > > test$ dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 19.456043 secs (862313883 bytes/sec) > > test$ dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 19.375321 secs (865906473 bytes/sec) > > test$ dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 19.173458 secs (875022968 bytes/sec) > > test$ uptime > > 10:30PM up 4 mins, 3 users, load averages: 3.52, 1.62, 0.69 > > > > These tests were conducted with the previously mentioned Samsung 850 Pro > 256GB SSDs (Intel Xeon E31240 with 8GB ram). There=E2=80=99s essentially= nothing > else running on this system (99.5-100% idle) and no other disk activity. > > > > Regards, > > A > > > > *From:* Jov [mailto:amutu@amutu.com] > *Sent:* Sunday, June 11, 2017 5:50 PM > *To:* Caza, Aaron > *Cc:* freebsd-hackers@freebsd.org; Allan Jude > > *Subject:* [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD > performance drop < 24 hours > > > > To exclude the fs problem=EF=BC=8CI will do a dd test on the pgdata data = set > after the performance drop,if the read and/or write utility can reach 100= % > or performance expected then I will say the problem is not fs or os. > > > > For pg,what's your output of explain analyze before and after performance > drop? > > > > 2017=E5=B9=B46=E6=9C=8812=E6=97=A5 12:51 AM=EF=BC=8C"Caza, Aaron" =E5=86=99=E9=81=93=EF=BC=9A > > Thanks Allan for the suggestions. I tried gstat -d but deletes (d/s) > doesn't seem to be it as it stays at 0 despite vfs.zfs.trim.enabled=3D1. > > This is most likely due to the "layering" I use as, for historical > reasons, I have GEOM ELI set up to essentially emulate 4k sectors > regardless of the underlying media. I do my own alignment and partition > sizing as well as have the ZFS record size set to 8k for Postgres. > > In gstat, the SSDs %busy is 90-100% on startup after reboot. Once the > performance degradation hits (<24 hours later), I'm seeing %busy at ~10%. > > #!/bin/sh > psql --username=3Dtest --password=3Dsupersecret -h /db -d test << EOL > \timing on > select count(*) from test; > \q > EOL > > Sample run of above script after reboot (before degradation hits) (Samsun= g > 850 Pros in ZFS mirror): > Timing is on. > count > ---------- > 21568508 > (1 row) > > Time: 57029.262 ms > > Sample run of above script after degradation (Samsung 850 Pros in ZFS > mirror): > Timing is on. > count > ---------- > 21568508 > (1 row) > > Time: 583595.239 ms > (Uptime ~1 day in this particular case.) > > > Any other suggestions? > > Regards, > A > > -----Original Message----- > From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd-hackers@ > freebsd.org] On Behalf Of Allan Jude > Sent: Saturday, June 10, 2017 9:40 PM > To: freebsd-hackers@freebsd.org > Subject: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD > performance drop < 24 hours > > On 06/10/2017 12:36, Slawa Olhovchenkov wrote: > > On Sat, Jun 10, 2017 at 04:25:59PM +0000, Caza, Aaron wrote: > > > >> Gents, > >> > >> I'm experiencing an issue where iterating over a PostgreSQL table of > ~21.5 million rows (select count(*)) goes from ~35 seconds to ~635 second= s > on Intel 540 SSDs. This is using a FreeBSD 10 amd64 stable kernel back > from Jan 2017. SSDs are basically 2 drives in a ZFS mirrored zpool. I'm > using PostgreSQL 9.5.7. > >> > >> I've tried: > >> > >> * Using the FreeBSD10 amd64 stable kernel snapshot of May 25, > 2017. > >> > >> * Tested on half a dozen machines with different models of SSDs: > >> > >> o Intel 510s (120GB) in ZFS mirrored pair > >> > >> o Intel 520s (120GB) in ZFS mirrored pair > >> > >> o Intel 540s (120GB) in ZFS mirrored pair > >> > >> o Samsung 850 Pros (256GB) in ZFS mirrored pair > >> > >> * Using bonnie++ to remove Postgres from the equation and > performance does indeed drop. > >> > >> * Rebooting server and immediately re-running test and > performance is back to original. > >> > >> * Tried using Karl Denninger's patch from PR187594 (which took > some work to find a kernel that the FreeBSD10 patch would both apply and > compile cleanly against). > >> > >> * Tried disabling ZFS lz4 compression. > >> > >> * Ran the same test on a FreeBSD9.0 amd64 system using PostgreSQ= L > 9.1.3 with 2 Intel 520s in ZFS mirrored pair. System had 165 days uptime > and test took ~80 seconds after which I rebooted and re-ran test and was > still at ~80 seconds (older processor and memory in this system). > >> > >> I realize that there's a whole lot of info I'm not including (dmesg, > zfs-stats -a, gstat, et cetera): I'm hoping some enlightened individual > will be able to point me to a solution with only the above to go on. > > > > Just a random guess: can you try r307264 (I am mean regression in > > r307266)? > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@ > freebsd.org" > > > > This sounds a bit like an issue I investigated for a customer a few month= s > ago. > > Look at gstat -d (includes DELETE operations like TRIM) > > If you see a lot of that happening, but try: vfs.zfs.trim.enabled=3D0 in > /boot/loader.conf and see if your issues go away. > > the FreeBSD TRIM code for ZFS basicallys waits until the sector has been > free for a while (to avoid doing a TRIM on a block we'll immediately > reuse), so your benchmark will run file for a little while, then suddenly > the TRIM will kick in. > > For postgres, fio, bonnie++ etc, make sure the ZFS dataset you are storin= g > the data on / benchmarking has a recordsize that matches the workload. > > If you are doing a write-only benchmark, and you see lots of reads in > gstat, you know you are having to do read/modify/write's, and that is why > your performance is so bad. > > > -- > Allan Jude > _______________________________________________ > freebsd-hackers@freebsd.org mailing list https://lists.freebsd.org/ > mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > > This message may contain confidential and privileged information. If it > has been sent to you in error, please reply to advise the sender of the > error and then immediately delete it. If you are not the intended > recipient, do not read, copy, disclose or otherwise use this message. The > sender disclaims any liability for such unauthorized use. PLEASE NOTE tha= t > all incoming e-mails sent to Weatherford e-mail accounts will be archived > and may be scanned by us and/or by external service providers to detect a= nd > prevent threats to our systems, investigate illegal or inappropriate > behavior, and/or eliminate unsolicited promotional e-mails (spam). This > process could result in deletion of a legitimate e-mail before it is read > by its intended recipient at our organization. Moreover, based on the > scanning results, the full text of e-mails and attachments may be made > available to Weatherford security and other personnel for review and > appropriate action. If you have any concerns about this process, > please contact us at dataprivacy@weatherford.com. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > From owner-freebsd-fs@freebsd.org Mon Jun 12 22:16:26 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 157ADC79672 for ; Mon, 12 Jun 2017 22:16:26 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-oln040092067017.outbound.protection.outlook.com [40.92.67.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C20081DDB for ; Mon, 12 Jun 2017 22:16:24 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) Received: from AM5EUR02FT032.eop-EUR02.prod.protection.outlook.com (10.152.8.51) by AM5EUR02HT119.eop-EUR02.prod.protection.outlook.com (10.152.9.130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.1143.11; Mon, 12 Jun 2017 22:16:03 +0000 Received: from AM4PR05MB1714.eurprd05.prod.outlook.com (10.152.8.55) by AM5EUR02FT032.mail.protection.outlook.com (10.152.8.109) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Mon, 12 Jun 2017 22:16:03 +0000 Received: from AM4PR05MB1714.eurprd05.prod.outlook.com ([fe80::1926:1bbf:ff94:fe53]) by AM4PR05MB1714.eurprd05.prod.outlook.com ([fe80::1926:1bbf:ff94:fe53%15]) with mapi id 15.01.1157.017; Mon, 12 Jun 2017 22:16:03 +0000 From: kc atgb To: "freebsd-fs@freebsd.org" Subject: Re: Problem with zpool remove of log device Thread-Topic: Problem with zpool remove of log device Thread-Index: AQHS48l6WE5shmvvSUeFr52IoccjVw== Date: Mon, 12 Jun 2017 22:16:02 +0000 Message-ID: References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net> In-Reply-To: <0410af$1dldvp4@ipmail04.adl6.internode.on.net> Accept-Language: fr-FR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=hotmail.fr; x-incomingtopheadermarker: OriginalChecksum:4B77FE149CEF2F6C4D0FB187D81E5C18A278EFC4DFB1474CDA285DADBD9AB647; UpperCasedChecksum:8F91F19F5E29BBA6A1D19B685F44BDC9613B07353C5EAD97BC2F44DF06A979B4; SizeAsReceived:7300; Count:45 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [o/nisRpYUo2krGeeNnjgwF9C7JW7t1R4] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM5EUR02HT119; 24:jkQYIXnSqYh7b5Iue+v0olhgvYVRjxP8P0GFWqMgEcS6hkueiAlu++Am6F39bz6ytZLrCA9B4MK4b//TgpNRw0XIDgnh+YWOCuQNht4GFNE=; 7:UfTVDuqw765Laek7Q1NSA/rnvCqo3tJ9Rn0FqG/1OFLspYBQtX9RYTzAsRgFTQ/eyTkqxVfFsyHYXbfeAcFgrLXBSXwSBYVk1XwDiJuMNOXFHJS/GnxW/s+EneArS0b5LDqf78e4K8ojPcIxB977IKhvUs2n7Ujgf7uSl4ZRyrLK3lBdznCTUvoBZemxz92ZMiacfr3JtosD2wN3JDEddJWLhRkrdWpDScR14d6pB2fO59Tw6vOYd7zZJ6wqLoIswePASJ+IoodPX306hGaLp3YpGkVLNBAp/sas/+FLA7otbquMKBDKud034jrqR9eI x-incomingheadercount: 45 x-eopattributedmessage: 0 x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:AM5EUR02HT119; H:AM4PR05MB1714.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; x-ms-traffictypediagnostic: AM5EUR02HT119: x-ms-office365-filtering-correlation-id: 7c5d87e4-9dfc-4fbc-064f-08d4b1e09c71 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322274)(1601125374)(1603101448)(1701031045); SRVR:AM5EUR02HT119; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(444000031); SRVR:AM5EUR02HT119; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM5EUR02HT119; x-forefront-prvs: 03361FCC43 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-ID: <32AC67395AC55E4ABFA56D3DC1D2FAE5@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jun 2017 22:16:02.8231 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5EUR02HT119 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jun 2017 22:16:26 -0000 Le mer. 07 juin 2017 08:21:09 CEST Stephen McKay a =E9crit: > On Friday, 26th May 2017, lukasz@wasikowski.net wrote: >=20 > >I cant remove log device from pool - operation ends ok, but log device > >is still in the pool (bug?). > > > ># uname -a > >FreeBSD xxx.yyy.com 11.0-STABLE FreeBSD 11.0-STABLE #0 r316543: Thu Apr > >6 08:22:43 CEST 2017 root@xxx.yyy.com:/usr/obj/usr/src/sys/YYY amd6= 4 > > > ># zpool status tank > >[..snip..] > > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > mirror-0 ONLINE 0 0 0 > > ada2p3 ONLINE 0 0 0 > > ada3p3 ONLINE 0 0 0 > > logs > > mirror-1 ONLINE 0 0 0 > > gpt/tankssdzil0 ONLINE 0 0 0 block size: 512= B configured, 4096B native > > gpt/tankssdzil1 ONLINE 0 0 0 block size: 512= B configured, 4096B native >=20 > >When I try to remove log device operation ends without errors: > > > ># zpool remove tank mirror-1; echo $? > >0 > > > >But the log device is still there: > >[..snip..] > >I'd like to remove it - how should I proceed? >=20 > Does your system still write to the log? Use "zfs iostat -v 1" to > check. I think it is probably no longer be in use and only the final > disconnection failed. >=20 > What does "zpool list -v" tell you? If you have a non-zero ALLOC > column for your log mirror and the log is no longer being used then > you may have hit an accounting bug in zfs that the zfsonlinux people > ran into a while ago. >=20 > I had this problem when I tried to remove a log mirror from a pool > I have been using for years. I solved it by tweaking the zfsonlinux > hack a bit and slotting it into 9.3. >=20 > If you apply this hack be sure to have a full backup first! When I > used it, I did my backup and a scrub then booted the hacked kernel, > issued the zfs remove command (which succeeded), reverted the kernel, > then scrubbed again. All went well. >=20 > Good luck! >=20 > Here's the patch against 9.3 (should be close even for 11.0): >=20 > Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (revision 309860= ) > +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (working copy) > @@ -5446,6 +5446,18 @@ > ASSERT(vd =3D=3D vd->vdev_top); > =20 > /* > + * slog stuck hack - barnes333@gmail.com > + * https://github.com/zfsonlinux/zfs/issues/1422 > + */ > + if (vd->vdev_islog && vd->vdev_removing > + && vd->vdev_state =3D=3D VDEV_STATE_OFFLINE > + && vd->vdev_stat.vs_alloc > 0) { > + printf("ZFS: slog stuck hack - clearing vs_alloc: %llu\n", > + (unsigned long long)vd->vdev_stat.vs_alloc); > + vd->vdev_stat.vs_alloc =3D 0; > + } > + > + /* > * Only remove any devices which are empty. > */ > if (vd->vdev_stat.vs_alloc !=3D 0) >=20 > Cheers, >=20 > Stephen. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 I have this case once again. The first time it was one month ago.=20 I had to backup ma datas and destroy and recreate the pool to remove the "f= aulted" log device.=20 I'll try your patch. I hope I'll be more lucky than OP. I have to backup fi= rst again.=20 In my opinion, maybe this problem is related to a certain type of data or a= ctivity. I have my pool for few years now and added a log only some months = ago.=20 It is a little bit strange that it happened to me twice in so little laps o= f time and others are not affected.=20 K. From owner-freebsd-fs@freebsd.org Tue Jun 13 16:04:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE57BBF85C9 for ; Tue, 13 Jun 2017 16:04:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC6F681688 for ; Tue, 13 Jun 2017 16:04:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5DG4qEL097781 for ; Tue, 13 Jun 2017 16:04:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 210721] Rerooting to an imported ZFS pool triggers panic Date: Tue, 13 Jun 2017 16:04:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to bug_status attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 16:04:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210721 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |avg@FreeBSD.org Status|New |Open --- Comment #4 from Andriy Gapon --- Created attachment 183449 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D183449&action= =3Dedit proposed patch Could you please try this patch? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 13 21:58:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC2BDBFF400 for ; Tue, 13 Jun 2017 21:58:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4FF96A91B for ; Tue, 13 Jun 2017 21:58:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5DLwXRO083202 for ; Tue, 13 Jun 2017 21:58:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219961] [regression][nullfs] Nullfs ko won't load _and_ options NULLFS doesn't work, either. Date: Tue, 13 Jun 2017 21:58:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 21:58:34 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219961 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |re@FreeBSD.org Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 13 22:13:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C7BBBFF985 for ; Tue, 13 Jun 2017 22:13:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3A1E46E1C9 for ; Tue, 13 Jun 2017 22:13:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5DMDT2Y039841 for ; Tue, 13 Jun 2017 22:13:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219961] [regression][nullfs] Nullfs ko won't load _and_ options NULLFS doesn't work, either. Date: Tue, 13 Jun 2017 22:13:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: rgrimes@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 22:13:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219961 Rodney W. Grimes changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rgrimes@FreeBSD.org --- Comment #1 from Rodney W. Grimes --- Your expecting to be able to load an 11.1 kernel module on 11.0 kernel? This is expected behavior as far as I can see, you need to build, install and reboot to a new kernel in order to be able to run new modules. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 13 22:16:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24B31BFFC0E for ; Tue, 13 Jun 2017 22:16:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12E946E357 for ; Tue, 13 Jun 2017 22:16:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5DMGpsE044465 for ; Tue, 13 Jun 2017 22:16:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219961] [regression][nullfs] Nullfs ko won't load _and_ options NULLFS doesn't work, either. Date: Tue, 13 Jun 2017 22:16:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dgilbert@eicat.ca X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 22:16:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219961 --- Comment #2 from dgilbert@eicat.ca --- (In reply to Rodney W. Grimes from comment #1) No. To be specific. Without changing sources, I tried rebuilding a module= in case it was bad/corrupt. But the kernel won't load nullfs. I've remade the whole kernel with make buildkernel/installkernel. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 13 22:57:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CB47C0888C for ; Tue, 13 Jun 2017 22:57:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A3FD6F739 for ; Tue, 13 Jun 2017 22:57:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5DMvv2w039823 for ; Tue, 13 Jun 2017 22:57:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219961] [regression][nullfs] Nullfs ko won't load _and_ options NULLFS doesn't work, either. Date: Tue, 13 Jun 2017 22:57:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dgilbert@eicat.ca X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 22:57:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219961 dgilbert@eicat.ca changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Not A Bug --- Comment #3 from dgilbert@eicat.ca --- Urgh. Bitten by boot environments. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 13 23:33:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34DBAC093F1 for ; Tue, 13 Jun 2017 23:33:04 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: from asp.reflexion.net (outbound-mail-210-12.reflexion.net [208.70.210.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF768709D1 for ; Tue, 13 Jun 2017 23:33:03 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: (qmail 15199 invoked from network); 13 Jun 2017 23:36:54 -0000 Received: from unknown (HELO mail-cs-02.app.dca.reflexion.local) (10.81.19.2) by 0 (rfx-qmail) with SMTP; 13 Jun 2017 23:36:54 -0000 Received: by mail-cs-02.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Tue, 13 Jun 2017 19:32:56 -0400 (EDT) Received: (qmail 13482 invoked from network); 13 Jun 2017 23:32:55 -0000 Received: from unknown (HELO mail.quorumlabs.com) (64.74.133.216) by 0 (rfx-qmail) with (AES128-SHA encrypted) SMTP; 13 Jun 2017 23:32:55 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.02.0318.001; Tue, 13 Jun 2017 16:32:54 -0700 From: Shiva Bhanujan To: "freebsd-fs@FreeBSD.org" Subject: Bug 214164 - camcontrol devlist fails on ioctl Thread-Topic: Bug 214164 - camcontrol devlist fails on ioctl Thread-Index: AdLknWE9Yc3cLfRhSI6lWvv1XW+P9w== Date: Tue, 13 Jun 2017 23:32:54 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35DA45C@QLEXC01.Quorum.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.65] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 23:33:04 -0000 Hello, I've hit this issue in FreeBSD 11.0-RELEASE (p10). Can somebody please let= me know if this is being addressed? Or there is a way to get camcontrol t= o work? [FreeBSD ~]# camcontrol devlist camcontrol: error sending CAMIOCOMMAND ioctl: Inappropriate ioctl for devic= e [FreeBSD ~]# Regards, Shiva From owner-freebsd-fs@freebsd.org Tue Jun 13 23:34:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A293AC09546 for ; Tue, 13 Jun 2017 23:34:17 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: from asp.reflexion.net (outbound-mail-210-12.reflexion.net [208.70.210.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5F3F370B32 for ; Tue, 13 Jun 2017 23:34:16 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: (qmail 23155 invoked from network); 13 Jun 2017 23:34:15 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 13 Jun 2017 23:34:15 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Tue, 13 Jun 2017 19:34:15 -0400 (EDT) Received: (qmail 11043 invoked from network); 13 Jun 2017 23:34:15 -0000 Received: from unknown (HELO mail.quorumlabs.com) (64.74.133.216) by 0 (rfx-qmail) with (AES128-SHA encrypted) SMTP; 13 Jun 2017 23:34:15 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.02.0318.001; Tue, 13 Jun 2017 16:34:13 -0700 From: Shiva Bhanujan To: Ronald Klop , =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: "freebsd-fs@freebsd.org" , Jeremy Faulkner , Shiva Bhanujan Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcfgADVzwCAAu69hYAEWqgAgJC1a0CAAIDAAIAMPuzC Date: Tue, 13 Jun 2017 23:34:12 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35DA475@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.65] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 23:34:17 -0000 Hi Ronald, thanks for the info. I'm going to work w/ FreeBSD 11.0, as it has = restartable send/receive. the compressed send is desirable though. Regards, Shiva ________________________________ From: Ronald Klop =5Bronald-lists=40klop.ws=5D Sent: Monday, June 05, 2017 2:33 PM To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan Cc: freebsd-fs=40freebsd.org; Jeremy Faulkner Subject: Re: FreeBSD restartable send/receive over WAN Hi, I'm not authoritative about this, but I see the feature in FreeBSD 12: https://www.freebsd.org/cgi/man.cgi?query=3Dzfs&apropos=3D0&sektion=3D8&man= path=3DFreeBSD+12-current&arch=3Ddefault&format=3Dhtml (option -c to 'zfs send') But not in FreeBSD 11. https://www.freebsd.org/cgi/man.cgi?query=3Dzfs&apropos=3D0&sektion=3D8&man= path=3DFreeBSD+11-current&arch=3Ddefault&format=3Dhtml Regards, Ronald. On Mon, 05 Jun 2017 23:06:41 +0200, Shiva Bhanujan > wrote: > Hi Ronald, > > Ideally, I'd like to send/receive the compressed blocks, and don't wish > to compress/uncompress on the fly. Is compressed send/receive > available on FreeBSD and if so, in what version? > > Regards, > Shiva > > > ________________________________________ > From: Ronald Klop = =5Bronald-lists=40klop.ws=5D > Sent: Sunday, March 05, 2017 3:01 AM > To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan > Cc: freebsd-fs=40freebsd.org; Jeremy = Faulkner > Subject: Re: FreeBSD restartable send/receive over WAN > > On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan > > wrote: > >> I ran the same set of tests between 2 FreeBSD instances, connected on a >> 1G LAN. The the comms was setup using mbuffer. Here's the basic >> command. >> >> time zfs send -v =7C =7C mbuffer -O :8099 -b = 1024 >> -m 128M -P 10 -q -l /tmp/mbuffer.log >> mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log =7C = >> =7C zfs recv >> >> Here are the results that I got. >> >> no compression: >> real 3m18.591s >> user 0m0.390s >> sys 0m8.177s >> >> xz -0: >> real 7m26.349s >> user 7m6.436s >> sys 0m8.471s >> >> pxz -0: >> real 2m28.470s >> user 6m44.168s >> sys 0m12.002s >> >> gzip: >> real 3m12.482s >> user 3m8.260s >> sys 0m4.974s >> >> lz4: >> real 1m58.363s >> user 0m10.000s >> sys 0m8.708s >> >> >> >> lz4 still comes out best. Is there anything else that I might be >> missing in my tests? don't have a real setup at this time to test WAN >> connections, but I'd like to think that these results can be >> extrapolated to a WAN link also. > > > Uhm, in your previous test no-compression came out best. I thought you > wanted to test if sending on-disk compressed blocks over a network was a > gain. Not a plain test of compression algoritms. > > As your current numbers show: the overhead of mbuffer.log is much higher > than the overhead of compression+decompression for the lz4 case. real: > 0m29 to 1m58. (All other compressors are actually faster with mbuffer in > between? Probably because of more efficient buffering by mbuffer.) > As the overhead of mbuffer is large it is a question how much gain you > get > from removing the extra compression step by sending the on-disk > compressed > blocks. > I guess the time would go from 1m58 to about 1m30. > > NB: By splitting your zfs send/receive command in two, the numbers of > time > can be affected. Time prints the output as soon as zfs send ends. > > Have fun, > Ronald. > > > >> ________________________________________ >> From: Ronald Klop = =5Bronald-lists=40klop.ws=5D >> Sent: Tuesday, February 28, 2017 11:44 AM >> To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan >> Cc: freebsd-fs=40freebsd.org; Jeremy = Faulkner >> Subject: Re: FreeBSD restartable send/receive over WAN >> >> On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan >> > wrote: >> >>> thanks for all the pointers for the compression algorithms. I ran a >>> few >>> tests to compare compression overhead. These are local zfs >>> send/receive, and no network is involved. >>> >>> time zfs send -v =7C =7C =7C zfs >>> receive -s >>> >>> Here are the performance results that I got. >>> >>> no compression: >>> real 0m20.892s >>> user 0m0.000s >>> sys 0m5.587s >>> >>> xz -0: >>> real 8m38.569s >>> user 10m28.551s >>> sys 0m6.866s >>> >>> pxz -0: >>> real 4m38.448s >>> user 10m55.863s >>> sys 0m13.324s >>> >>> gzip: >>> real 3m51.297s >>> user 4m12.035s >>> sys 0m4.696s >>> >>> lz4: >>> real 0m29.912s >>> user 0m16.543s >>> sys 0m10.810s >>> >>> >>> lz4 has the least overhead in terms of time. pxz/xz seem to be >>> prohibitive give the above results. Unless, there is something basic >>> I'm missing? >>> >>> I was really hoping that compressed sends would be available, as that >>> would actively eliminate this overhead, given that we use lz4 as the >>> compression algorithm when writing to disks. >> >> >> Why don't you test this with a network in between? That would give much >> more valuable numbers to compare for your use case. >> The number above say nothing about the gain vs the bottleneck you are >> trying to remove. >> >> Ronald. >> >> >> >> >>> >>> >>> ________________________________ >>> From: Karli Sj=F6berg = =5Bkarli.sjoberg=40slu.se=5D >>> Sent: Sunday, February 26, 2017 8:41 AM >>> To: Gary Palmer >>> Cc: Shiva Bhanujan; Jeremy Faulkner; = freebsd-fs=40freebsd.org >>> Subject: Re: FreeBSD restartable send/receive over WAN >>> >>> >>> Den 26 feb. 2017 4:16 em skrev Gary Palmer = ::> >>>> >>>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: >>>> > The compression that we use on our ZFS filesystems is lz4. So, if I >>>> have to pipe it through a compression algorithm, that'd be >>>> uncompressing and compressing it 4 times. >>>> > >>>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) >>>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) >>>> > >>>> > isn't this quite expensive? We have to transfer multi terabyte >>>> files >>>> on a WAN link. I'm also of the understanding that gzip by itself is >>>> single-threaded, so that'd peg one of the CPUs to 100%. there might >>>> be >>>> other compression algorithms that can be used, but sending the ZFS as >>>> it is compressed on the filesystem is something that would be optimal, >>>> and would reduce the overhead of the additional =5Bde=5Dcompressions = that >>>> are taking place? >>>> >>>> Without going into the efficiency part of your message: >>>> >>>> archivers/pigz: Parallel GZIP >>>> archivers/pbzip2: Parallel BZIP2 >>>> archivers/pixz: Parallel, indexing version of XZ >>>> archivers/pxz: Parallel LZMA compressor using liblzma >>> >>> Also worth mentioning is, obviously: >>> archivers/lz4 >>> >>> :) >>> >>> /K >>> >>>> >>>> Regards, >>>> >>>> Gary >>>> >>>> > >>>> > ________________________________________ >>>> > From: = owner-freebsd-fs=40freebsd.org = =5Bowner-freebsd-fs=40freebsd.org=5D= on >>>> behalf of Jeremy Faulkner = =5Bgldisater=40gmail.com=5D >>>> > Sent: Saturday, February 25, 2017 4:03 PM >>>> > To: freebsd-fs=40freebsd.org >>>> > Subject: Re: FreeBSD restartable send/receive over WAN >>>> > >>>> > Pipe it through a compressor >>>> > >>>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >>>> > > Hi, >>>> > > >>>> > > I just tried restartable send/receive in 10.3 and it works like a >>>> charm. I was wondering if compressed send has made its way into >>>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the >>>> -c/--compressed option. Any pointers? >>>> > > >>>> > > Regards, >>>> > > Shiva >>>> > > >>>> > > >>>> > > ________________________________________ >>>> > > From: = owner-freebsd-fs=40freebsd.org = =5Bowner-freebsd-fs=40freebsd.org=5D >>>> on behalf of Adam Nowacki = =5Bnowakpl=40platinum.linux.pl=5D >>>> > > Sent: Thursday, February 16, 2017 10:41 AM >>>> > > To: freebsd-fs=40freebsd.org >>>> > > Subject: Re: FreeBSD restartable send/receive over WAN >>>> > > >>>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >>>> > >> Hello, >>>> > >> >>>> > >> I was wondering if restartable send/receive is available in >>>> FreeBSD? We're running 10.2 and have a requirement of sending and >>>> receiving ZFS snapshots over a WAN link. The snapshots could be more >>>> than a few terabytes. >>>> > >> >>>> > >> Can somebody please give me pointers, and if this feature is or >>>> isn't available in FreeBSD? >>>> > > >>>> > > FreeBSD 10.3 and later. >>>> > > >>>> > > _______________________________________________ >>>> > > freebsd-fs=40freebsd.org mailing = list >>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > > _______________________________________________ >>>> > > freebsd-fs=40freebsd.org mailing = list >>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > > >>>> > _______________________________________________ >>>> > freebsd-fs=40freebsd.org mailing = list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > _______________________________________________ >>>> > freebsd-fs=40freebsd.org mailing = list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > >>>> _______________________________________________ >>>> freebsd-fs=40freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>> _______________________________________________ >>> freebsd-fs=40freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = =22freebsd-fs-unsubscribe=40freebsd.org=22 ________________________________ From owner-freebsd-fs@freebsd.org Wed Jun 14 00:42:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BA6AC0AA8C for ; Wed, 14 Jun 2017 00:42:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE59D729CF for ; Wed, 14 Jun 2017 00:42:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5E0g0in047298 for ; Wed, 14 Jun 2017 00:42:00 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Wed, 14 Jun 2017 00:42:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 00:42:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jun 14 05:34:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F4FCD86678 for ; Wed, 14 Jun 2017 05:34:51 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15DFC7A8F1 for ; Wed, 14 Jun 2017 05:34:49 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id A7E9A20AA5 for ; Wed, 14 Jun 2017 01:34:48 -0400 (EDT) Received: from web6 ([10.202.2.216]) by compute7.internal (MEProxy); Wed, 14 Jun 2017 01:34:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=mesmtp; bh=L7CJhsrfk74rn7abxg/cDE/ooC L1tckb9okP7XWLCpU=; b=duaq1l5MVXnqymiQDBZNbUsL6N9r01nrKTplokUUTc 8nj2KHxVGyVxTczHrt1fgBWJT54RCRkeQIo+HPYrLUxmUC4sd1I6eRRhMxsvGpSb btDVzyrLTkeuF9VhN51rqcfUURWDt6SlJxDfwlTuhPLaoY2MOX5kNsQK9Hm4EBBx o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=L7CJhs rfk74rn7abxg/cDE/ooCL1tckb9okP7XWLCpU=; b=n5AbYU3Ne+Eu7qxlHSNfTC 6LsY5JdT9PCr1GP6CjpthtVfgWU3t57/SxCVNBTL1vry7zDEERmQIkHc5kqfF2bF 3m3yOCcBJJrH5SrKJV9awAxUtvQosGS/7gkABHi5ThpcTmEldWGbrLOm05Ur2wY0 QT3SBh6asNiIK9hlm22O6Vo6h54fE+eVb/eJ0mGAztoaix9n58OtI8FXtIP2FVkx UlLbKVDAPsYbZMi3p7CCdjrbID43lkTpH1AV9wYlM/nZIDu9DlLgSPrw9prwYVP6 p8QXoC7nO0+feJtOlY5vfsnOG43mXcZ0V/inLsaqxNjGyZ0fcEtvbsjq190KSLSA == X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 7E7CA48002; Wed, 14 Jun 2017 01:34:48 -0400 (EDT) Message-Id: <1497418488.877884.1008774696.75A50952@webmail.messagingengine.com> From: Dave Cottlehuber To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-1fe24561 References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35DA475@QLEXC01.Quorum.local> Subject: Re: FreeBSD restartable send/receive over WAN Date: Wed, 14 Jun 2017 07:34:48 +0200 In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35DA475@QLEXC01.Quorum.local> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 05:34:51 -0000 On Wed, 14 Jun 2017, at 01:34, Shiva Bhanujan wrote: > Hi Ronald, > > thanks for the info. I'm going to work w/ FreeBSD 11.0, as it has > restartable send/receive. the compressed send is desirable though. > > Regards, > Shiva From owner-freebsd-fs@freebsd.org Wed Jun 14 05:45:55 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3AC2D8690E for ; Wed, 14 Jun 2017 05:45:55 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B5B17AE17 for ; Wed, 14 Jun 2017 05:45:55 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 7D11920B53 for ; Wed, 14 Jun 2017 01:45:54 -0400 (EDT) Received: from web6 ([10.202.2.216]) by compute7.internal (MEProxy); Wed, 14 Jun 2017 01:45:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=mesmtp; bh=Es41S5uC/V0EaWQglx8S1l4qB7 THBmIBOY6XRC0z8u8=; b=DfFuRtyKOj9nroFtbPNPb42aVYI3qhkSPI1hKTRFcY iL34Kyt5KBPGSaaCBOBz+NKN6+NqSSljIbdIwL9rpf78DN08rEvhiLkrEx7uZDZq 5qo720vPmMo1WVYJhaRove4843adPX0a5qVz5ij7NvFGltjO7qVPi47mdGxyb320 w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=Es41S5 uC/V0EaWQglx8S1l4qB7THBmIBOY6XRC0z8u8=; b=Ba5cHVKsmPRBGZzeJC74NG ZH0tT4mn9Tv280T8nwudlaEA/RWE7htgZXofCgmlPp5XICMOBrM0BilDgq5fflD7 wlZckFiJSmbZI7w5Tw/2+UEKm6vxUwqWICun7mDZpb/KviSEqR+8Nyx7TuSnpRaI 4gWeMdq7+xBrAoDDmjEhU8XBg94QZuHEixaaAYY1n5Rtc7XBBILTR2awIKpsW4sL 4nwyjEeypaZOAkiowlXtGJ9TYyNAEcDkS96pxy/L0o4q3ykG1m12Q7L1WjXXL4ar NL6Hkh8sDEaeBppjnlOjj2eeunds0bZOZ9AfWhezjD99fTSCtwGv/y8hL8BQSFnA == X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 528AB48002; Wed, 14 Jun 2017 01:45:54 -0400 (EDT) Message-Id: <1497419154.879644.1008780888.162DA368@webmail.messagingengine.com> From: Dave Cottlehuber To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-1fe24561 Subject: Re: FreeBSD restartable send/receive over WAN Date: Wed, 14 Jun 2017 07:45:54 +0200 In-Reply-To: <1497418488.877884.1008774696.75A50952@webmail.messagingengine.com> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35DA475@QLEXC01.Quorum.local> <1497418488.877884.1008774696.75A50952@webmail.messagingengine.com> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 05:45:55 -0000 > On Wed, 14 Jun 2017, at 01:34, Shiva Bhanujan wrote: > Hi Ronald, > > thanks for the info. I'm going to work w/ FreeBSD 11.0, as it has > restartable send/receive. the compressed send is desirable though. > > Regards, > Shivas A while back I hacked up a self-restarting zfs send, using mbuffer, which in my setup was faster than ssh. I can't recall if it can terminate itself or not, but it did the job, transferring 350GB over the Pacific Ocean migrating off servers. - use shells/fish - on source side while true sudo zfs send -Levt \ (ssh -p 2200 root@10.144.157.90 zfs get -Hpo value receive_resume_token zroot/shared/got) \ | mbuffer -4 -s 128k -m 1G -O 10.144.157.90:9000 end - on sink side while true mbuffer -4 -s 128k -m 1G -I 9000 \ | zfs recv -Fuvs zroot/shared/got end NB there's no encryption (security/hpenc is an excellent choice), or replace mbuffer with spiped. NNB https://gist.github.com/dch/26f2c660fee145c24ffc79126e9cbb21 is the pretty version. I'd love to hear back wrt perfomance/throughput. A+ Dave From owner-freebsd-fs@freebsd.org Wed Jun 14 14:13:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C99E4D8EBC1 for ; Wed, 14 Jun 2017 14:13:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9212D64EFB for ; Wed, 14 Jun 2017 14:13:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 4AC09435DD; Wed, 14 Jun 2017 16:13:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fxHlCW32W3xW; Wed, 14 Jun 2017 16:13:36 +0200 (CEST) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id A8A0F435DB for ; Wed, 14 Jun 2017 16:13:36 +0200 (CEST) To: FreeBSD Filesystems From: Willem Jan Withagen Subject: Strangely truncated names in linprocfs in jail Message-ID: <39fc567b-a9d6-ea7d-31a9-51d87f0cb803@digiware.nl> Date: Wed, 14 Jun 2017 16:13:36 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 14:13:45 -0000 Hi, I have mount linprocfs in a jail on /compat/linux/proc And that seems to work. Other than that I have strangely truncated names there: root@ceph-0:/compat/linux/proc # ls 81117 81218 devices meminfo net self sys 81185 cmdline filesyst? mounts partitio? stat uptime 81216 cpuinfo loadavg mtab scsi swaps version Both filesystems and partitions seem to malformed. And even stranger, in the linprocfs mount on the actual host (not jailed) those 2 files do not show. Anybody else tried this? --WjW From owner-freebsd-fs@freebsd.org Wed Jun 14 14:23:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B684D8F081 for ; Wed, 14 Jun 2017 14:23:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 597736573D for ; Wed, 14 Jun 2017 14:23:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5EENqFE073665 for ; Wed, 14 Jun 2017 14:23:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Wed, 14 Jun 2017 14:23:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pfribeiro@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 14:23:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 --- Comment #1 from pfribeiro@gmail.com --- I've done some further testing on this by isolating the issue to a 'zfs sen= d' and 'zfs recv' on the same machine between the external USB drive (on /dev/= da0) and a USB stick (on /dev/da1). 1. I made sure that there were no existing pools on both devices, by destro= ying the pools and doing 'zpool labelclear -f /dev/da0' and 'zpool labelclear -f /dev/da1'. root@sunflower:~ # zpool labelclear -f /dev/da0 root@sunflower:~ # zpool labelclear -f /dev/da1 2. Created two pools: root@sunflower:~ # zpool create -O atime=3Doff -O compression=3Dlz4 master = /dev/da0 root@sunflower:~ # zpool create -O atime=3Doff -O compression=3Dlz4 slave /= dev/da1 root@sunflower:~ # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT master 7.25T 360K 7.25T - 0% 0% 1.00x ONLINE - slave 7.25G 78.5K 7.25G - 0% 0% 1.00x ONLINE - zroot 51.5G 1.11G 50.4G - 0% 2% 1.00x ONLINE - 3. Created some test file on the master pool: root@sunflower:~ # touch /master/test root@sunflower:~ # echo "Hello" > /master/test 4. Created a snapshot on the master pool: root@sunflower:~ # zfs snapshot -r "master@1" 5. Exported the 'slave' pool and imported unmounted (not sure this matters really): root@sunflower:~ # zpool export slave root@sunflower:~ # zpool import -N slave 6. Sent the snapshot across to the slave pool: root@sunflower:~ # zfs send -Rev "master@1" | zfs recv -Fu slave full send of master@1 estimated size is 10K total estimated size is 10K TIME SENT SNAPSHOT root@sunflower:~ # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT master 7.25T 588K 7.25T - 0% 0% 1.00x ONLINE - slave 7.25G 376K 7.25G - 0% 0% 1.00x ONLINE - zroot 51.5G 1.11G 50.4G - 0% 2% 1.00x ONLINE - 7. Tried to export and import the slave a few times: root@sunflower:~ # zpool export slave root@sunflower:~ # zpool import slave root@sunflower:~ # zpool export slave cannot export 'slave': pool is busy ...and bang the problem is back. This seems a little crazy. I would appreciate if someone could try and help me reproduce this problem = on their system, and perhaps point me to some further debugging steps. I'm out= of ideas as to what else I could/should test at this point. Thanks --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Jun 15 11:59:56 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D20EED88A36 for ; Thu, 15 Jun 2017 11:59:56 +0000 (UTC) (envelope-from mckay@freebsd.org) Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by mx1.freebsd.org (Postfix) with ESMTP id 2FD9572CF7; Thu, 15 Jun 2017 11:59:55 +0000 (UTC) (envelope-from mckay@freebsd.org) Message-Id: X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2A+/wDZdEJZAEbA0HZcgxqBQIg+h2CUKJR1QwGFWgQCAoJbWAMBAQEBAQIPAQEBMk+FGQZWFQ4QCANGOR4GE4orrHWXZIQ7EgGDQIJPBYlHiEyMNpVciTSGUJR/VoEAC4EBCIdqLodYgjABAgM X-IronPort-SPAM: SPAM Received: from ppp118-208-192-70.bras1.hba1.internode.on.net (HELO localhost) ([118.208.192.70]) by ipmail07.adl2.internode.on.net with ESMTP; 15 Jun 2017 21:24:20 +0930 From: Stephen McKay To: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= cc: Stephen McKay , freebsd-fs@freebsd.org Subject: Re: Problem with zpool remove of log device References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net> <4df1ea6d-148e-f3ab-d204-9277c513a468@wasikowski.net> <0fc687$sij78n@ipmail05.adl6.internode.on.net> In-Reply-To: from =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= at "Fri, 09 Jun 2017 12:15:32 +0200" Date: Thu, 15 Jun 2017 21:54:18 +1000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jun 2017 11:59:56 -0000 Sorry for the slow response. I've been away (without email) for a few days. On Friday, 9th June 2017, lukasz@wasikowski.net wrote: >W dniu 2017-06-09 o 05:18, Stephen McKay pisze: > >> while : >> do >> date > foo >> fsync foo >> done >> >> With this running, my system does 600 writes per second to the log >> according to zpool iostat. That drops to zero once I kill the script. > >Zero, so no writes to log are performed during execution of this script. OK. I believe this means your log is in a "pending removal" state and has not been finally removed because ZFS thinks there's still data stored on it. I'm happy for true ZFS experts to confirm or deny this theory. Anybody? >I applied this patch to 11.1-PRERELASE, nothing changed. Still zpool >remove exits with errcode 0, but log device is still attached to pool. Thanks for trying this out, but I managed to leave out essential information. In my rush to do things before going away (see above), I didn't read my notes on this event. The patch has the safety feature of requiring the log to be offline. This means you will have to break the mirror (by detaching one disk from it), then offline the remaining disk, and finally, trigger the hack by attempting to remove the remaining disk. When I was stuck in this situation, I had already reduced my log to a single disk before discovering the accounting error problem, so I don't know if you can activate the patch code without first breaking the mirror. I don't think you can offline a mirror. I've not tried. I've now had time to review my notes (sorry I didn't do that first up). My pool had a data mirror-0 (gpt/data0 and gpt/data1) and a log mirror-1 (gpt/log0 and gpt/log1). The sequence I did, minus most of the failed attempts at random stuff, status checks, and so forth, was: # zpool remove pool mirror-1 #Did nothing but should have worked. # zpool detach pool gpt/log1 #Broke the log mirror. # zpool remove pool gpt/log0 #Did nothing but should have worked. # zpool offline pool gpt/log0 #Just fiddling to change the state. # zpool remove pool gpt/log0 #Still nothing. ... Discovered plausible hack. Built and booted hacked kernel. ... # zpool remove pool gpt/log0 #Glorious success! So, my log was already down to one offline disk before I got hacky. That's why I forgot to mention it. You could break your mirror or you could modify the hack to remove the VDEV_STATE_OFFLINE check. If you have already saved all your important data then either would be fine as an experiment. Cheers, Stephen. From owner-freebsd-fs@freebsd.org Sat Jun 17 20:52:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 415B0C094DC for ; Sat, 17 Jun 2017 20:52:50 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [5.196.167.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E3CA477B6C; Sat, 17 Jun 2017 20:52:49 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [IPv6:2001:41d0:2:1276::1]) by mail.freebsd.systems (Postfix) with ESMTP id CD07072C; Sat, 17 Jun 2017 22:52:40 +0200 (CEST) X-Virus-Scanned: amavisd-new at freebsd.systems Received: from mail.freebsd.systems ([5.196.167.1]) by mail.freebsd.systems (scan.freebsd.systems [5.196.167.1]) (amavisd-new, port 10026) with ESMTP id hGJQ5eCRLeZ9; Sat, 17 Jun 2017 22:52:40 +0200 (CEST) Received: from [192.168.168.1] (89-70-62-144.dynamic.chello.pl [89.70.62.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.freebsd.systems (Postfix) with ESMTPSA id 2BD29729; Sat, 17 Jun 2017 22:52:40 +0200 (CEST) Authentication-Results: mail.freebsd.systems; dmarc=none header.from=wasikowski.net Authentication-Results: mail.freebsd.systems; spf=pass smtp.mailfrom=lukasz@wasikowski.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wasikowski.net; s=default; t=1497732760; bh=b+t+9t0XQRu5neJcgTxBjsOh0meko6vTagCIOrT96KE=; h=To:Cc:References:From:Date:In-Reply-To; b=bT2T9l53uv2EzMhFTOhTLjAONPFCihnovppVJkC3pUBzEGwEF/qr0Dx2ZqadhpfHS zUXiB9dw2bxI0BBLmsizG24LkwkiJKAftXwcauq+GBQGgwOUzjgePnCM94+U+lvluA f7pYMxkVtwrjfTrIR/yQnCN72S/X7X0LE3gPFR8vHsKfH5fKhun1p6fJH+O2TTYahM +Jr6sscMGb6hakwkgfhYaCcRGAX0IJdTYA2zGA2zzE1a809zkcQ7XTedlgtv7u4CHd JBU04PVE+GcgE7fU9b5Eqcack3Pfr2znFV3WVV+HlbN+i+OksUEYbn8IHH6JUF8ax1 19r2Ex6eAAcbQ== Subject: Re: Problem with zpool remove of log device To: Stephen McKay Cc: freebsd-fs@freebsd.org References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net> <4df1ea6d-148e-f3ab-d204-9277c513a468@wasikowski.net> <0fc687$sij78n@ipmail05.adl6.internode.on.net> From: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= Message-ID: <31ae8d40-9c28-a14c-2b7c-b62a6125df04@wasikowski.net> Date: Sat, 17 Jun 2017 22:52:41 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: pl Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jun 2017 20:52:50 -0000 W dniu 2017-06-15 o 13:54, Stephen McKay pisze: > On Friday, 9th June 2017, lukasz@wasikowski.net wrote: > >> W dniu 2017-06-09 o 05:18, Stephen McKay pisze: >> >>> while : >>> do >>> date > foo >>> fsync foo >>> done >>> >>> With this running, my system does 600 writes per second to the log >>> according to zpool iostat. That drops to zero once I kill the script. >> >> Zero, so no writes to log are performed during execution of this script. > > OK. I believe this means your log is in a "pending removal" state and > has not been finally removed because ZFS thinks there's still data > stored on it. I'm happy for true ZFS experts to confirm or deny this > theory. Anybody? > >> I applied this patch to 11.1-PRERELASE, nothing changed. Still zpool >> remove exits with errcode 0, but log device is still attached to pool. > > Thanks for trying this out, but I managed to leave out essential > information. > > In my rush to do things before going away (see above), I didn't read my > notes on this event. The patch has the safety feature of requiring the > log to be offline. This means you will have to break the mirror (by > detaching one disk from it), then offline the remaining disk, and finally, > trigger the hack by attempting to remove the remaining disk. > > When I was stuck in this situation, I had already reduced my log to > a single disk before discovering the accounting error problem, so I > don't know if you can activate the patch code without first breaking > the mirror. I don't think you can offline a mirror. I've not tried. > > I've now had time to review my notes (sorry I didn't do that first up). > My pool had a data mirror-0 (gpt/data0 and gpt/data1) and a log mirror-1 > (gpt/log0 and gpt/log1). The sequence I did, minus most of the failed > attempts at random stuff, status checks, and so forth, was: > > # zpool remove pool mirror-1 #Did nothing but should have worked. > # zpool detach pool gpt/log1 #Broke the log mirror. > # zpool remove pool gpt/log0 #Did nothing but should have worked. > # zpool offline pool gpt/log0 #Just fiddling to change the state. > # zpool remove pool gpt/log0 #Still nothing. > ... Discovered plausible hack. Built and booted hacked kernel. ... > # zpool remove pool gpt/log0 #Glorious success! > > So, my log was already down to one offline disk before I got hacky. > That's why I forgot to mention it. > > You could break your mirror or you could modify the hack to remove the > VDEV_STATE_OFFLINE check. If you have already saved all your important > data then either would be fine as an experiment. My tests was done after zpool remove pool mirror-1, zpool detach pool gpt/log1. I don't remember if I offlined gpt/log0. It's possible that I haven't done it. Bulit and booted hacked kernel, zpool remove pool gpt/log0 didn't work. Unfortunately, lease of this box ended 14.06, I zeroed drives on 13.06 :( Again, thank you for your help! -- best regards, Lukasz Wasikowski From owner-freebsd-fs@freebsd.org Sat Jun 17 22:32:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B55BEC0AC6D for ; Sat, 17 Jun 2017 22:32:29 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-oln040092071057.outbound.protection.outlook.com [40.92.71.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41B2A79F98 for ; Sat, 17 Jun 2017 22:32:28 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) Received: from DB5EUR03FT014.eop-EUR03.prod.protection.outlook.com (10.152.20.56) by DB5EUR03HT028.eop-EUR03.prod.protection.outlook.com (10.152.20.242) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.1157.12; Sat, 17 Jun 2017 22:32:26 +0000 Received: from AM4PR05MB1714.eurprd05.prod.outlook.com (10.152.20.57) by DB5EUR03FT014.mail.protection.outlook.com (10.152.20.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Sat, 17 Jun 2017 22:32:26 +0000 Received: from AM4PR05MB1714.eurprd05.prod.outlook.com ([fe80::1926:1bbf:ff94:fe53]) by AM4PR05MB1714.eurprd05.prod.outlook.com ([fe80::1926:1bbf:ff94:fe53%15]) with mapi id 15.01.1157.022; Sat, 17 Jun 2017 22:32:26 +0000 From: kc atgb To: "freebsd-fs@freebsd.org" Subject: Re: Problem with zpool remove of log device Thread-Topic: Problem with zpool remove of log device Thread-Index: AQHS48l6WE5shmvvSUeFr52IoccjV6Ipq40A Date: Sat, 17 Jun 2017 22:32:26 +0000 Message-ID: References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net> In-Reply-To: Accept-Language: fr-FR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=hotmail.fr; x-incomingtopheadermarker: OriginalChecksum:E1081F7F99241A16B64A7CFB1402AA9DA7E761A49B0A3CAC138AD389EA4F0E30; UpperCasedChecksum:220293D3CD204853E32C3430EC831864A08EA088C53A8C834F504C414746A798; SizeAsReceived:7415; Count:45 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [HT0buQjcUtkQrCTyLHgpqY4Q/3+777+I] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB5EUR03HT028; 7:cjdWjA7YAKf5f3wP7TZwxORy0Sn8MKlucaxm1XDfmoKibhDpTXjknIwc8I0DZGlwVID32Ensl171C7XCcPqC2LVvWgmok1MYDxYLX7GC0IGn+wgRTQVytNWPVbSratvdvYktd01PCx45QNJatNytyN+U1webBKufku8SDxBWY6k35K2xn8MnhTZ89UQtUH3+XXFRSvpadHyfcw0LBJhpxwCfvuCVUlnzy5j6rzriq+WdDl3xHiU+JQ2VcZHGswxjUYKQ5Fu47tVMYmRqciA2G5c8rQX5+UbGJOYQhQ28xTEcRcoJMn6ESW36fc6HFGFsvShBynIH33HmO1P94fXb71QA2otQMgabxsfMP3f4vjWfWlDbW92WGcREA4wKj8TMeLtx+4DRxP+CREtpYfPBriNbGa0Ws0HlAyRJVq7IwKfF4fgZAUMR2fjI6vLpxwgVK3C2cs1Hn4z2BfkQy2BXRvQKCjCB1DXMbzb29bLkqXtTdH6iUpC4JvgvWE2gRBSM095NmIej/fWb23BumprXmbZ0a6ucMeiUWFE7qVX+2qMlG0c0/v6l5jnzpZRft+83o5itKBPTrIZ1nVo3TW1ZwEcoOm/HYh2OdzVGtpuTvdkYy3b+T2VTBm6rN+krdQRhAEuZbbR2xremnz2b3a44de5oc7NribVbiajFV4EG63QuTfUiw+r48XbeRUMlJbK6ciBHdhfCaaf8YIZhYJr3D6mheQiIXhy8W73G3wH2U28vrO5Qfkcl3pbjIhCZHpVmwVll0xzfLDuh2elEaiaa1A== x-incomingheadercount: 45 x-eopattributedmessage: 0 x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:DB5EUR03HT028; H:AM4PR05MB1714.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; x-ms-traffictypediagnostic: DB5EUR03HT028: x-ms-office365-filtering-correlation-id: be82fb74-2291-4dd3-529a-08d4b5d0ba28 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322274)(1603101448)(1601125374)(1701031045); SRVR:DB5EUR03HT028; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(444000031); SRVR:DB5EUR03HT028; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DB5EUR03HT028; x-forefront-prvs: 034119E4F6 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-ID: <765B861DD714984F885760EE371ED6E4@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Jun 2017 22:32:26.0550 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5EUR03HT028 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jun 2017 22:32:29 -0000 Le lun. 12 juin 2017 22:16:02 CEST kc atgb a =E9crit: >=20 >=20 > Le mer. 07 juin 2017 08:21:09 CEST > Stephen McKay a =E9crit: >=20 > > On Friday, 26th May 2017, lukasz@wasikowski.net wrote: > >=20 > > >I cant remove log device from pool - operation ends ok, but log device > > >is still in the pool (bug?). > > > > > ># uname -a > > >FreeBSD xxx.yyy.com 11.0-STABLE FreeBSD 11.0-STABLE #0 r316543: Thu Ap= r > > >6 08:22:43 CEST 2017 root@xxx.yyy.com:/usr/obj/usr/src/sys/YYY am= d64 > > > > > ># zpool status tank > > >[..snip..] > > > > > > NAME STATE READ WRITE CKSUM > > > tank ONLINE 0 0 0 > > > mirror-0 ONLINE 0 0 0 > > > ada2p3 ONLINE 0 0 0 > > > ada3p3 ONLINE 0 0 0 > > > logs > > > mirror-1 ONLINE 0 0 0 > > > gpt/tankssdzil0 ONLINE 0 0 0 block size: 5= 12B configured, 4096B native > > > gpt/tankssdzil1 ONLINE 0 0 0 block size: 5= 12B configured, 4096B native > >=20 > > >When I try to remove log device operation ends without errors: > > > > > ># zpool remove tank mirror-1; echo $? > > >0 > > > > > >But the log device is still there: > > >[..snip..] > > >I'd like to remove it - how should I proceed? > >=20 > > Does your system still write to the log? Use "zfs iostat -v 1" to > > check. I think it is probably no longer be in use and only the final > > disconnection failed. > >=20 > > What does "zpool list -v" tell you? If you have a non-zero ALLOC > > column for your log mirror and the log is no longer being used then > > you may have hit an accounting bug in zfs that the zfsonlinux people > > ran into a while ago. > >=20 > > I had this problem when I tried to remove a log mirror from a pool > > I have been using for years. I solved it by tweaking the zfsonlinux > > hack a bit and slotting it into 9.3. > >=20 > > If you apply this hack be sure to have a full backup first! When I > > used it, I did my backup and a scrub then booted the hacked kernel, > > issued the zfs remove command (which succeeded), reverted the kernel, > > then scrubbed again. All went well. > >=20 > > Good luck! > >=20 > > Here's the patch against 9.3 (should be close even for 11.0): > >=20 > > Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (revision 3098= 60) > > +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (working copy) > > @@ -5446,6 +5446,18 @@ > > ASSERT(vd =3D=3D vd->vdev_top); > > =20 > > /* > > + * slog stuck hack - barnes333@gmail.com > > + * https://github.com/zfsonlinux/zfs/issues/1422 > > + */ > > + if (vd->vdev_islog && vd->vdev_removing > > + && vd->vdev_state =3D=3D VDEV_STATE_OFFLINE > > + && vd->vdev_stat.vs_alloc > 0) { > > + printf("ZFS: slog stuck hack - clearing vs_alloc: %llu\n", > > + (unsigned long long)vd->vdev_stat.vs_alloc); > > + vd->vdev_stat.vs_alloc =3D 0; > > + } > > + > > + /* > > * Only remove any devices which are empty. > > */ > > if (vd->vdev_stat.vs_alloc !=3D 0) > >=20 > > Cheers, > >=20 > > Stephen. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >=20 >=20 > I have this case once again. The first time it was one month ago.=20 > I had to backup ma datas and destroy and recreate the pool to remove the = "faulted" log device.=20 >=20 > I'll try your patch. I hope I'll be more lucky than OP. I have to backup = first again.=20 >=20 > In my opinion, maybe this problem is related to a certain type of data or= activity. I have my pool for few years now and added a log only some month= s ago.=20 > It is a little bit strange that it happened to me twice in so little laps= of time and others are not affected.=20 >=20 > K. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 I have succesfully applied the patch and build the kernel. The removal of l= og=20 device has worked too. It was in offline state, then I had to remove the dr= ive so it was marked as unavailable before removal. =20 My FreeBSD version : FreeBSD my.host.name 9.3-STABLE FreeBSD 9.3-STABLE #0 r315141: Sun Mar 12=20 16:00:24 CET 2017 root@my.host.name:/usr/obj/usr/src/sys/GENERIC amd64 I'm still curious about why is it happening. Any idea ?=20 K.