From owner-freebsd-fs@freebsd.org Sun Sep 6 05:20:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 177149CB986 for ; Sun, 6 Sep 2015 05:20:02 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from pv33p03im-asmtp001.me.com (pv33p03im-asmtp001.me.com [17.143.180.10]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E4F24634 for ; Sun, 6 Sep 2015 05:20:01 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from [10.20.30.60] (75-101-82-48.static.sonic.net [75.101.82.48]) by pv33p03im-asmtp001.me.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Mar 31 2015)) with ESMTPSA id <0NU800IXHPH3RD40@pv33p03im-asmtp001.me.com> for freebsd-fs@freebsd.org; Sun, 06 Sep 2015 05:19:55 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2015-09-06_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=3.23174820238137e-12 compositescore=0.994460221061599 phishscore=0 kscore.is_spamscore=0 rbsscore=0.994460221061599 recipient_to_sender_totalscore=0 spamscore=0 urlsuspectscore=0.994460221061599 adultscore=0 kscore.compositescore=0 circleOfTrustscore=0 suspectscore=0 recipient_domain_to_sender_totalscore=0 bulkscore=0 recipient_domain_to_sender_domain_totalscore=0 recipient_to_sender_domain_totalscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1412110000 definitions=main-1509060099 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 9.0 \(3093\)) Subject: Re: CEPH + FreeBSD From: Jordan Hubbard In-reply-to: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> Date: Sat, 05 Sep 2015 22:19:50 -0700 Cc: freebsd-fs@freebsd.org, Rakshith Venkatesh Content-transfer-encoding: quoted-printable Message-id: <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.3093) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 05:20:02 -0000 > On Sep 3, 2015, at 4:17 AM, Rick Macklem wrote: >=20 > Slightly off topic but, btw, there is a port of GLusterFS and those = folks do seem > interested in seeing it brought "up to speed". I am not sure how = mature it is at > this point, but it has been known to build on amd64. (I don't have an = amd64 machine, > so I haven't gotten around to building/testing it, but I do plan to = try and use > it as a basis for a pNFS server, if I can figure out how to get the FH = info out of it. > I'm working on that;-) There are at least two distributed (multi-node) object stores for = FreeBSD that I know of. One is glusterfs, for which I=E2=80=99m not even really clear on the = status of the ports for. I don=E2=80=99t see any glusterfs port in the = master branch of https://github.com/freebsd/freebsd-ports (or = https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that = matter). Our FreeNAS ports tree (https://github.com/freenas/ports), in which we = have a bit more latitude to add and curate our own ports, has both a = net/glusterfs and sysutils/glusterfs, from separate sources (looks like = we need to clean things up) - net/glusterfs lists = craig001@lerwick.hopto.org as the MAINTAINER and is at version 3.6.2. = The sysutils/glusterfs port lists bapt@FreeBSD.org as the MAINTAINER and = is at version 20140811. I=E2=80=99m not really sure about the provenance since we were simply = evaluating glusterfs for awhile and may have pulled in interim versions = from those sources, but obviously it would be best to have an official = maintainer and someone in the FreeBSD project actually curating a = glusterfs port so that all users of FreeBSD can use it. It would also = be fairly key to your own efforts, assuming you decide to pursue = glusterfs as a foundation technology for pNFS. The other object store, which is pretty mature and is currently leading = the pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. = There is a port currently in databases/riak but it=E2=80=99s pretty out = of date at version 1.4.12 (the current version is 2.0.1, with 2.0 being = a major upgrade of RiakCS). We are very interested in investigating various ways of shimming RiakCS = to NFS, using RiakCS a back-end store. Is that something you=E2=80=99d = be amenable to discussing? I=E2=80=99d be happy to send you an amd64 = architecture machine to develop on. :) - Jordan From owner-freebsd-fs@freebsd.org Sun Sep 6 14:36:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92BF99C5C8B for ; Sun, 6 Sep 2015 14:36:20 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2965B174 for ; Sun, 6 Sep 2015 14:36:19 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id E65B615340A; Sun, 6 Sep 2015 16:36:14 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s8yoypDs8uZy; Sun, 6 Sep 2015 16:35:39 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id BC8A3153408; Sun, 6 Sep 2015 16:35:39 +0200 (CEST) Subject: Re: can zfs snapshot be used to back LUN in ctl.conf ? To: Zeus Panchenko , freebsd-fs@freebsd.org References: <20150827150027.10825@smtp.new-ukraine.org> From: Willem Jan Withagen Message-ID: <55EC4F3B.7090804@digiware.nl> Date: Sun, 6 Sep 2015 16:35:39 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20150827150027.10825@smtp.new-ukraine.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 14:36:20 -0000 On 27-8-2015 14:00, Zeus Panchenko wrote: > greetings, > > please help me to understand where to look at ... > > recently I switched from istgt to ctld and now wonder, whether can zfs > snapshot be used to back the LUN for ctld? > > istgt do allows that, while ctld fails to start and complains > > but if I copy the file from zfs snapshot to some place, then ctld starts > as expected ... As far as I know those snapshots are read-only. So other that mount and use for reading backup stuff it does not really serve a purpose? And than still you'd next a "consistent" fs on the volume to do any useful work. So that would only work for filesystems that need a recover that does not involve writing to the volume. Otherwise if you force mount something like UFS, you run the risk of crashes? So unless you promote a the volume to writable things will not work. For files as backing volume that happens if you copy it to a writeable location, or clone the volume the file is on. If you are using ZVOL, than you have to clone snapshot. So I'm not sure if ctld aborts because it tests writing the backing storage and fails if it cannot. --WjW > > > bellow the details are: > > ---[ ctld debug quotation start ]------------------------------------------- > ... > ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file011 > ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file012 > ctld: error returned from LUN creation request: ctl_be_block_open: error opening /storage/win/.zfs/snapshot/daily-2015-08-22/file013 > ctld: failed to add lun 0, target iqn.2007-09.jp.ne.peach.istgt:file013 > ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file014 > ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file015 > ... > ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file052 > ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file053 > ctld: not listening on portal-group "default", not assigned to any target > ctld: listening on 10.100.21.47, portal-group "alfa" > ctld: listening on 10.100.21.47, portal-group "beta" > ctld: failed to apply configuration; exiting > /etc/rc.d/ctld: WARNING: failed to start ctld > ---[ ctld debug quotation end ]------------------------------------------- > > ---[ ctl.conf quotation start ]------------------------------------------- > target iqn.2007-09.jp.ne.peach.istgt:file013 { > alias "file013-users" > portal-group alfa > auth-group ag-file013 > lun 0 { > path /storage/win/.zfs/snapshot/daily-2015-08-22/file013 > size 300G > } > } > ---[ ctl.conf quotation end ]------------------------------------------- > > the very file exists: >> stat /storagez/win/.zfs/snapshot/daily-2015-08-22/traders.ts.ibs > 3500296891 22 -rw-r--r-- 1 root wheel 4294967295 214748364800 "Oct 23 07:41:38 2013" "Aug 21 04:00:28 2015" "Aug 21 04:00:28 2015" "Oct 23 07:41:38 2013" 131072 419838466 0x800 /storage/win/.zfs/snapshot/daily-2015-08-22/file013 > > > another question: can ctld be configured to ignore unavailable config > parts? like unaccessible/missconfigured LUNs From owner-freebsd-fs@freebsd.org Sun Sep 6 15:55:00 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AFCB9CB181 for ; Sun, 6 Sep 2015 15:55:00 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D5D1876 for ; Sun, 6 Sep 2015 15:54:59 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by wicfx3 with SMTP id fx3so65247776wic.1 for ; Sun, 06 Sep 2015 08:54:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type; bh=KVWmtOwlKKVdb/DSWD76KTnTAm0LnjmiK2JoOWDpkhY=; b=Fg1Bf785qWH6HVvKtMMGYSAlfDZ//e9RHzgvmxn9/mjR8ISnYW2EiRUeWbnQvGPS9I HVNMcLfS4BQlGIDqZZ3vFeYaZuCdFoqTXfQ1QZ3qvBx92nulNDzMXBr5RUljl3slaH7h mR/1Mp8xTszyFpln0p8dR2ih3hVUBTIgAUVRUHPfCL92jbmZlc01TGUaf2FmCN4pnuTn e9DzBCL1WjHij/8Zat1KtfMmSyiiHZEA7ImL/iP+OAyWsCf6pt27OnbNfhmEpN60nqcz Vq+aYPoBwYlkWZCd9tAzumQqLYB/Ag/WLa5bAXd0qzrW+O8Q9C1mFg4CaXSNlTUqU6g/ /oKQ== X-Received: by 10.194.78.164 with SMTP id c4mr26198544wjx.65.1441554898031; Sun, 06 Sep 2015 08:54:58 -0700 (PDT) Received: from Johans-MacBook-Air.local (92-70-102-130.glasvezel.netexpo.nl. [92.70.102.130]) by smtp.googlemail.com with ESMTPSA id p5sm15717678wiy.17.2015.09.06.08.54.57 for (version=TLSv1/SSLv3 cipher=OTHER); Sun, 06 Sep 2015 08:54:57 -0700 (PDT) Subject: Re: can zfs snapshot be used to back LUN in ctl.conf ? To: freebsd-fs@freebsd.org References: <20150827150027.10825@smtp.new-ukraine.org> <55EC4F3B.7090804@digiware.nl> From: Johan Hendriks Message-ID: <55EC61D1.6010206@gmail.com> Date: Sun, 6 Sep 2015 17:54:57 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55EC4F3B.7090804@digiware.nl> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 15:55:00 -0000 Op 06/09/15 om 16:35 schreef Willem Jan Withagen: > On 27-8-2015 14:00, Zeus Panchenko wrote: >> greetings, >> >> please help me to understand where to look at ... >> >> recently I switched from istgt to ctld and now wonder, whether can zfs >> snapshot be used to back the LUN for ctld? >> >> istgt do allows that, while ctld fails to start and complains >> >> but if I copy the file from zfs snapshot to some place, then ctld starts >> as expected ... > As far as I know those snapshots are read-only. > > So other that mount and use for reading backup stuff it does not really > serve a purpose? And than still you'd next a "consistent" fs on the > volume to do any useful work. So that would only work for filesystems > that need a recover that does not involve writing to the volume. > Otherwise if you force mount something like UFS, you run the risk of > crashes? > > So unless you promote a the volume to writable things will not work. > For files as backing volume that happens if you copy it to a writeable > location, or clone the volume the file is on. If you are using ZVOL, > than you have to clone snapshot. > > So I'm not sure if ctld aborts because it tests writing the backing > storage and fails if it cannot. > > --WjW I just noticed something beeing mentioned on freshbsd.org. mav @ HEAD - r287499 - 2015 -09 -06 09 :54 :56 Move setting of media parameters inside open routines. This is preparation for possibility to open/close media several times per LUN life cycle. While there, rename variables to reduce confusion. As additional bonus this allows to open read-only media, such as ZFS snapshots. So I think in HEAD it should be possible from now on after revision 287499 gr Johan >> >> bellow the details are: >> >> ---[ ctld debug quotation start ]------------------------------------------- >> ... >> ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file011 >> ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file012 >> ctld: error returned from LUN creation request: ctl_be_block_open: error opening /storage/win/.zfs/snapshot/daily-2015-08-22/file013 >> ctld: failed to add lun 0, target iqn.2007-09.jp.ne.peach.istgt:file013 >> ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file014 >> ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file015 >> ... >> ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file052 >> ctld: adding lun 0, target iqn.2007-09.jp.ne.peach.istgt:file053 >> ctld: not listening on portal-group "default", not assigned to any target >> ctld: listening on 10.100.21.47, portal-group "alfa" >> ctld: listening on 10.100.21.47, portal-group "beta" >> ctld: failed to apply configuration; exiting >> /etc/rc.d/ctld: WARNING: failed to start ctld >> ---[ ctld debug quotation end ]------------------------------------------- >> >> ---[ ctl.conf quotation start ]------------------------------------------- >> target iqn.2007-09.jp.ne.peach.istgt:file013 { >> alias "file013-users" >> portal-group alfa >> auth-group ag-file013 >> lun 0 { >> path /storage/win/.zfs/snapshot/daily-2015-08-22/file013 >> size 300G >> } >> } >> ---[ ctl.conf quotation end ]------------------------------------------- >> >> the very file exists: >>> stat /storagez/win/.zfs/snapshot/daily-2015-08-22/traders.ts.ibs >> 3500296891 22 -rw-r--r-- 1 root wheel 4294967295 214748364800 "Oct 23 07:41:38 2013" "Aug 21 04:00:28 2015" "Aug 21 04:00:28 2015" "Oct 23 07:41:38 2013" 131072 419838466 0x800 /storage/win/.zfs/snapshot/daily-2015-08-22/file013 >> >> >> another question: can ctld be configured to ignore unavailable config >> parts? like unaccessible/missconfigured LUNs > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sun Sep 6 17:15:55 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA4349C58B3 for ; Sun, 6 Sep 2015 17:15:55 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: from smtp.new-ukraine.org (smtp.new-ukraine.org [148.251.53.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.new-ukraine.org", Issuer "smtp.new-ukraine.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 10B26DAF for ; Sun, 6 Sep 2015 17:15:54 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: on behalf of honored client by smtp.new-ukraine.org with ESMTP id t86HB9Bm070268 on Sun, 6 Sep 2015 20:11:15 +0300 (EEST) Message-ID: <20150906201059.70248@smtp.new-ukraine.org> Date: Sun, 06 Sep 2015 20:10:59 +0300 From: "Zeus Panchenko" To: "Willem Jan Withagen" Cc: Subject: Re: can zfs snapshot be used to back LUN in ctl.conf ? In-reply-to: Your message of Sun, 6 Sep 2015 16:35:39 +0200 <55EC4F3B.7090804@digiware.nl> References: <20150827150027.10825@smtp.new-ukraine.org> <55EC4F3B.7090804@digiware.nl> Organization: I.B.S. LLC Reply-To: "Zeus Panchenko" X-Attribution: zeus Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAFVBMVEWxsbGdnZ3U1NQTExN cXFzx8fG/v7+f8hyWAAACXUlEQVQ4jUWSwXYiIRBFi4yyhtjtWpmRdTL0ZC3TJOukDa6Rc+T/P2F eFepwtFvr8upVFVDua8mLWw6La4VIKTuMdAPOebdU55sQs3n/D1xFFPFGVGh4AHKttr5K0bS6g7N ZCge7qpVLB+f1Z2WAj2OKXwIWt/bXpdXSiu8KXbviWkHxF5td9+lg2e3xlI2SCvatK8YLfHyh9lw 15yrad8Va5eXg4Llr7QmAaC+dL9sDt9iad/DX3OKvLMBf+dm0A0QuMrTvYIevSik1IaSVvgjIHt5 lSCG2ynNRpEcBZ8cgDWk+Ns99qzsYYV3MZoppWzGtYlTO9+meG6m/g92iNO9LfQB2JZsMpoJs7QG ku2KtabRK0bZRwDLyBDvwlxTm6ZlP7qyOqLcfqtLexpDSB4M0H3I/PQy1emvjjzgK+A0LmMKl6Lq zlqzh0VGAw440F6MJd8cY0nI7wiF/fVIBGY7UNCAXy6DmfYGCLLI0wtDbVcDUMqtJLmAhLqODQAe riERAxXJ1/QYGpa0ymqyytpKC19MNXHjvFmEsfcHIrncFR4xdbYWgmfEGLCcZokpGbGj1egMR+6M 1BkNX1pDdhPcOXpAnAeLQUwQLYepgQoZVNGS61yaE8CYA7gYAcWKzwGstACY2HTFvvOwk4FXAG/a mKHni/EcA/GkOk7I0IK7UMIf3+SahU8/FJdiE7KcuWdM3MFocUDEEIX9LfJoo4xV5tnNKc3jJuSs SZWgnnhepgU1zN4Hii18yW4RwDX52CXUtk0Hqz6cHOIUkWaX8fDcB+J7y1y2xDHwjv/8Buu8Ekz6 7tXQAAAAASUVORK5CYII= X-Mailer: MH-E 8.3.1; GNU Mailutils 2.99.98; GNU Emacs 24.3.1 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" X-NewUkraine-Agent: mailfromd (7.99.92) X-NewUkraine-URL: https://mail.prozora-kraina.org/smtp.html X-NewUkraine-VirStat: NO X-NewUkraine-VirScan: ScanPE, ScanELF, ScanOLE2, ScanMail, PhishingSignatures, ScanHTML, ScanPDF X-NewUkraine-SpamStat: NO X-NewUkraine-SpamScore: -1.600 of 3.500 X-NewUkraine-SpamKeys: AWL,BAYES_00,NO_RECEIVED,NO_RELAYS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 17:15:55 -0000 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Willem Jan Withagen wrote: > > istgt do allows that, while ctld fails to start and complains > >=20 > > but if I copy the file from zfs snapshot to some place, then ctld starts > > as expected ... >=20 > As far as I know those snapshots are read-only. yes, and it is just what I want > So other that mount and use for reading backup stuff it does not really > serve a purpose? file is NTFS volume, I need to provide it to win host ... ctld refuses to do that even in ro mode :( > And than still you'd next a "consistent" fs on the > volume to do any useful work. So that would only work for filesystems > that need a recover that does not involve writing to the volume. yes, no writing ... just reading ... and I wonder why istgt was doing that fine but ctld refuses :( =2D-=20 Zeus V. Panchenko jid:zeus@im.ibs.dn.ua IT Dpt., I.B.S. LLC GMT+2 (EET) --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlXsc6MACgkQr3jpPg/3oyrS5gCg7rFgIcAUPZiPXgAnXB9n0uL3 x8sAn1CyMuZxy480gh7K6K9upJ4NUNuT =h32M -----END PGP SIGNATURE----- --=-=-=-- From owner-freebsd-fs@freebsd.org Sun Sep 6 18:09:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39D0B9CC3F3 for ; Sun, 6 Sep 2015 18:09:22 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F24D2858 for ; Sun, 6 Sep 2015 18:09:21 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 78FD1153431; Sun, 6 Sep 2015 20:09:19 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J1s-1LtyGSG9; Sun, 6 Sep 2015 20:09:09 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id 14A7A153416; Sun, 6 Sep 2015 19:58:37 +0200 (CEST) Subject: Re: can zfs snapshot be used to back LUN in ctl.conf ? To: Zeus Panchenko References: <20150827150027.10825@smtp.new-ukraine.org> <55EC4F3B.7090804@digiware.nl> <20150906201059.70248@smtp.new-ukraine.org> Cc: freebsd-fs@freebsd.org From: Willem Jan Withagen Message-ID: <55EC7ECC.5050201@digiware.nl> Date: Sun, 6 Sep 2015 19:58:36 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20150906201059.70248@smtp.new-ukraine.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 18:09:22 -0000 On 6-9-2015 19:10, Zeus Panchenko wrote: > Willem Jan Withagen wrote: >>> istgt do allows that, while ctld fails to start and complains >>> >>> but if I copy the file from zfs snapshot to some place, then ctld starts >>> as expected ... >> >> As far as I know those snapshots are read-only. > > yes, and it is just what I want > >> So other that mount and use for reading backup stuff it does not really >> serve a purpose? > > file is NTFS volume, I need to provide it to win host ... ctld refuses > to do that even in ro mode :( > > >> And than still you'd next a "consistent" fs on the >> volume to do any useful work. So that would only work for filesystems >> that need a recover that does not involve writing to the volume. > > yes, no writing ... just reading ... and I wonder why istgt was doing > that fine but ctld refuses :( Okeeee, I'm in a sort of similar setup, but haven't tried snapshotting. Difference here is that I use a zvol as background storage. Now is there one thing which has bitten me before which making rsync backups... It is impossible to do getcwd when that is place within a snapshot, unless the snapdir flag is set to visible on the volume. Not doing so results in an error. So perhaps something like this is the difference between the two providers Example: sent 151 bytes received 368 bytes 1,038.00 bytes/sec total size is 0 speedup is 0.00 Now it works: # zfs get snapdir zfsraid/backups NAME PROPERTY VALUE SOURCE zfsraid/backups snapdir visible received # rsync -rav . sending incremental file list drwxr-xr-x 7 2015/06/17 11:35:11 . drwxr-xr-x 2 2010/01/29 13:13:49 Digiware Now it doesn't: # zfs set snapdir=hidden zfsraid/backups # rsync -rav . rsync: getcwd(): No such file or directory (2) rsync error: errors selecting input/output files, dirs (code 3) at util.c(1101) [Receiver=3.1.1] So you might need to set the property of the volume your backing file is on. --WjW From owner-freebsd-fs@freebsd.org Sun Sep 6 21:00:06 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD5BF9CB5C2 for ; Sun, 6 Sep 2015 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 86810CB8 for ; Sun, 6 Sep 2015 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t86L06Se090537 for ; Sun, 6 Sep 2015 21:00:06 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201509062100.t86L06Se090537@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 06 Sep 2015 21:00:06 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 21:00:06 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 3 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Sun Sep 6 22:18:41 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5894D9CC07E for ; Sun, 6 Sep 2015 22:18:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id E9C0012E1 for ; Sun, 6 Sep 2015 22:18:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:O0U6JxVv4D2UCaggKfsecM4/fOHV8LGtZVwlr6E/grcLSJyIuqrYZhyDt8tkgFKBZ4jH8fUM07OQ6PC8HzRRqs/e7DgrS99laVwssY0uhQsuAcqIWwXQDcXBSGgEJvlET0Jv5HqhMEJYS47UblzWpWCuv3ZJQk2sfTR8Kum9IIPOlcP/j7n0oM2PJV0Zz2PiPftbF1afk0b4joEum4xsK6I8mFPig0BjXKBo/15uPk+ZhB3m5829r9ZJ+iVUvO89pYYbCf2pN4xxd7FTDSwnPmYp/4Wr8ECbFUrcrkcbB0cRiBZBBUDl8RvwV439+n/4sfBx0S+aIMf8RKo4cTWp66B2RFnjjyJRZBAj92SCsM17j+p+qRmioxF6i9rOZYieN/5ze4vAetwHSG5ZXoBaXnoSUcuHc4ITAr9Zbq5jpI7nqg5L9EPmCA== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BYAgAyu+xV/61jaINeg3dpBoMeuk0BCYF3hXkCgVcUAQEBAQEBAQGBCYIdggcBAQQjVhACAQgSBgICDRkCAkkOAgQTiC4NtC6TeAEBAQEGAQEBAQEZBIEihVGDdoEFhEEXATMHgmmBQwWVFz6FCok8hDORDINqAiaEHCIzh0SBBQEBAQ X-IronPort-AV: E=Sophos;i="5.17,481,1437451200"; d="scan'208";a="237002272" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 06 Sep 2015 18:18:33 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 71E2915F563; Sun, 6 Sep 2015 18:18:33 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id zNzkbqJlzNOa; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 9402615F565; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id y4UTipf5FowJ; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7652C15F563; Sun, 6 Sep 2015 18:18:32 -0400 (EDT) Date: Sun, 6 Sep 2015 18:18:32 -0400 (EDT) From: Rick Macklem To: Jordan Hubbard Cc: freebsd-fs@freebsd.org, Rakshith Venkatesh Message-ID: <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> Subject: Re: CEPH + FreeBSD MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: CEPH + FreeBSD Thread-Index: KowScjIbpQ2TT+IvpGwjo+f1Q1ORrg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 22:18:41 -0000 Jordan Hubbard wrote: >=20 > > On Sep 3, 2015, at 4:17 AM, Rick Macklem wrote: > >=20 > > Slightly off topic but, btw, there is a port of GLusterFS and those fol= ks > > do seem > > interested in seeing it brought "up to speed". I am not sure how mature= it > > is at > > this point, but it has been known to build on amd64. (I don't have an a= md64 > > machine, > > so I haven't gotten around to building/testing it, but I do plan to try= and > > use > > it as a basis for a pNFS server, if I can figure out how to get the FH = info > > out of it. > > I'm working on that;-) >=20 > There are at least two distributed (multi-node) object stores for FreeBSD > that I know of. >=20 > One is glusterfs, for which I=E2=80=99m not even really clear on the stat= us of the > ports for. I don=E2=80=99t see any glusterfs port in the master branch o= f > https://github.com/freebsd/freebsd-ports (or > https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that > matter). >=20 > Our FreeNAS ports tree (https://github.com/freenas/ports), in which we ha= ve a > bit more latitude to add and curate our own ports, has both a net/gluster= fs > and sysutils/glusterfs, from separate sources (looks like we need to clea= n > things up) - net/glusterfs lists craig001@lerwick.hopto.org as the > MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port lists > bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. >=20 > I=E2=80=99m not really sure about the provenance since we were simply eva= luating > glusterfs for awhile and may have pulled in interim versions from those > sources, but obviously it would be best to have an official maintainer an= d > someone in the FreeBSD project actually curating a glusterfs port so that > all users of FreeBSD can use it. It would also be fairly key to your own > efforts, assuming you decide to pursue glusterfs as a foundation technolo= gy > for pNFS. >=20 > The other object store, which is pretty mature and is currently leading t= he > pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. There= is > a port currently in databases/riak but it=E2=80=99s pretty out of date at= version > 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade of > RiakCS). >=20 > We are very interested in investigating various ways of shimming RiakCS t= o > NFS, using RiakCS a back-end store. Is that something you=E2=80=99d be = amenable to > discussing? I=E2=80=99d be happy to send you an amd64 architecture mach= ine to > develop on. :) >=20 Hmm. From a quick look at their web page (I looked once before as well), I = don't think RiakCS has what I need to do pNFS in a reasonable (for me) amount of = effort. Two things that glusterFS has that I am hoping to use (and I don't think Ri= akCS has either of these) are: - A Fuse file system interface which allows the kernel nfsd to access the s= tore as a file system, so that it can provide the metadata services (NFS without = the reads/writes). - A userland NFSv3 server in each node which will allow the node to act as = a data server. If I am wrong and RiakCS does support a VFS file system interface (via Fuse= or ???), then please correct me. With that, it might be a reasonable alternative. I'll admit I've spent a little time looking at the glusterFS sources and ha= ven't yet solved the problem of how to generate the file handles I need, but that sou= nds trivial compared with an entire Fuse and/or VFS file system interface, I think? In general, using a cloud object store to implement a pNFS server is a *mis= *use of the technology, imho. I think it may be possible with glusterFS, since that= technology seems to be based on a cluster file system, which is what a pNFS server can= also use. I think there would be a lot of work involved in mapping a POSIX file syste= m onto the Riak database and then exporting that via NFS, etc. It might also be more p= ractical to do this via a userland NFS service than the kernel based one currently in F= reeBSD. (glusterFS is starting to use the NFS-ganesha server, but I believe it is p= retty Linux specific, so I doubt it would be useful for Riak running on FreeBSD?) rick > - Jordan >=20 >=20 From owner-freebsd-fs@freebsd.org Mon Sep 7 06:13:16 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D1FF9CCA0F for ; Mon, 7 Sep 2015 06:13:16 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from pv33p03im-asmtp001.me.com (pv33p03im-asmtp001.me.com [17.143.180.10]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E8B20640 for ; Mon, 7 Sep 2015 06:13:15 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from [10.20.30.11] (75-101-82-48.static.sonic.net [75.101.82.48]) by pv33p03im-asmtp001.me.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Mar 31 2015)) with ESMTPSA id <0NUA00PSOMLT2810@pv33p03im-asmtp001.me.com> for freebsd-fs@freebsd.org; Mon, 07 Sep 2015 06:13:09 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2015-09-07_01:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 compositescore=0.994460221061599 phishscore=0 kscore.is_spamscore=0 rbsscore=0.994460221061599 recipient_to_sender_totalscore=0 spamscore=0 urlsuspectscore=0.994460221061599 adultscore=0 kscore.compositescore=0 circleOfTrustscore=0 suspectscore=0 recipient_domain_to_sender_totalscore=0 bulkscore=0 recipient_domain_to_sender_domain_totalscore=0 recipient_to_sender_domain_totalscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1412110000 definitions=main-1509070115 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 9.0 \(3093\)) Subject: glusterfs + FreeBSD [was Re: CEPH + FreeBSD] From: Jordan Hubbard In-reply-to: <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> Date: Sun, 06 Sep 2015 23:13:05 -0700 Cc: freebsd-fs@freebsd.org, Rakshith Venkatesh , Justin Clift Content-transfer-encoding: quoted-printable Message-id: <382BA1FD-032E-4F21-9460-D2119340E21B@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.3093) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 06:13:16 -0000 [ Adding Justin Clift to CC line ] > On Sep 6, 2015, at 3:18 PM, Rick Macklem wrote: >=20 > Hmm. =46rom a quick look at their web page (I looked once before as = well), I don't > think RiakCS has what I need to do pNFS in a reasonable (for me) = amount of effort. > Two things that glusterFS has that I am hoping to use (and I don't = think RiakCS has > either of these) are: > - A Fuse file system interface which allows the kernel nfsd to access = the store as > a file system, so that it can provide the metadata services (NFS = without the reads/writes). > - A userland NFSv3 server in each node which will allow the node to = act as a data server. >=20 > If I am wrong and RiakCS does support a VFS file system interface (via = Fuse or ???), then > please correct me. You are not wrong. RiakCS is basically just a distributed object store = with an Amazon S3 compatible API. It doesn=E2=80=99t attempt to expose = the object store as a filesystem or provide any other modes of access to = it other than via S3 or its own =E2=80=9Cdatabase API=E2=80=9D since = it=E2=80=99s also (at its heart) a distributed database. In all fairness to Basho, I also think that was probably the right call. = It=E2=80=99s not a file system and doesn=E2=80=99t pretend to be one, = it=E2=80=99s just a distributed database that can store arbitrary = numbers (and sizes) of objects in a cluster with fault-tolerance, = multi-data center tenancy and so on (I also have no connection with = Basho so if it sounds like I=E2=80=99m trying to sales pitch it or = something, I=E2=80=99m not, I=E2=80=99m simply trying to be accurate in = describing it given that we=E2=80=99re also evaluating it as a possible = technology to incorporate into FreeNAS, for which the topic of = S3-compatible object storage comes up fairly frequently. Now glusterfs is another kettle of fish, and like I said, I think the = bigger question there is just what FreeBSD=E2=80=99s relationship with = it is. The https://wiki.freebsd.org/GlusterFS wiki was created over a = year ago and seems to be suffering from an overall lack of specific = goals. Does the FreeBSD project want glusterfs and/or is there anyone = in the FreeBSD community even interested in deploying it in a production = scenario? Is anyone willing to actually maintain a port, just as the = minimum price of admission? The bug tracking this, namely = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194409, has been = open for almost a year now and while it looks like =E2=80=98craig001=E2=80= =9D has been trying fairly hard to get 3.7.2 up and working, he=E2=80=99s = hitting some walls and it=E2=80=99s not even clear he=E2=80=99s in = contact with the right folks. As the previous glusterfs mail thread indicated, I was also happy to add = an earlier version of his port to the FreeNAS repo and play with it = there, though I was forced to take it back out of FreeNAS when we ran = into a number of issues just trying to make it work with the Linux = glusterfs reference VMs we set up. Should the underlying port get some = love again, it would be the simplest thing in the world for me to = refresh our copy of it and uncomment the line which adds glusterfs to = the FreeNAS manifest (and being a storage appliance, it=E2=80=99s rather = of an obvious test bed for this since people running FreeNAS are also = the target audience for technologies like this). If getting glusterfs up as a stable, working substrate is also a = prerequisite for getting pNFS, then I think a lot of us in the storage = industry would be the first to say =E2=80=9CWhat do we have to do? How = can we help?=E2=80=9D I already volunteered to send Rick an AMD64 = architecture machine for development, but I suspect that=E2=80=99s = actually the least of our obstacles right now. :-) - Jordan From owner-freebsd-fs@freebsd.org Mon Sep 7 06:39:58 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F5C79CBAC4 for ; Mon, 7 Sep 2015 06:39:58 +0000 (UTC) (envelope-from www-data@meldrar.postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher DHE-RSA-AES256-SHA256 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A181262 for ; Mon, 7 Sep 2015 06:39:57 +0000 (UTC) (envelope-from www-data@meldrar.postgresql.org) Received: from www-data by meldrar.postgresql.org with local (Exim 4.80) (envelope-from ) id 1ZYq5j-0003Zk-UO; Mon, 07 Sep 2015 06:39:48 +0000 To: Jordan Hubbard , Kaleb Keithley , Niels de Vos Subject: Re: glusterfs + FreeBSD [was Re: CEPH + FreeBSD] X-PHP-Originating-Script: 0:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 07 Sep 2015 06:39:47 +0000 From: justin@postgresql.org Cc: Rick Macklem , , Rakshith Venkatesh , Justin Clift , In-Reply-To: <382BA1FD-032E-4F21-9460-D2119340E21B@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <382BA1FD-032E-4F21-9460-D2119340E21B@icloud.com> Message-ID: <051286539482bdb811694897aacd1b2f@postgresql.org> X-Sender: justin@postgresql.org User-Agent: Roundcube Webmail/0.7.2 Sender: www-data X-Pg-Spam-Score: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 06:39:58 -0000 [ adding Jeff and Kaleb (Gluster folk) ] (remainder of the reply is at the bottom) On 2015-09-07 06:13, Jordan Hubbard wrote: > [ Adding Justin Clift to CC line ] > >> On Sep 6, 2015, at 3:18 PM, Rick Macklem >> wrote: >> >> Hmm. From a quick look at their web page (I looked once before as >> well), I don't >> think RiakCS has what I need to do pNFS in a reasonable (for me) >> amount of effort. >> Two things that glusterFS has that I am hoping to use (and I don't >> think RiakCS has >> either of these) are: >> - A Fuse file system interface which allows the kernel nfsd to >> access the store as >> a file system, so that it can provide the metadata services (NFS >> without the reads/writes). >> - A userland NFSv3 server in each node which will allow the node to >> act as a data server. >> >> If I am wrong and RiakCS does support a VFS file system interface >> (via Fuse or ???), then >> please correct me. > > You are not wrong. RiakCS is basically just a distributed object > store with an Amazon S3 compatible API. It doesn’t attempt to expose > the object store as a filesystem or provide any other modes of access > to it other than via S3 or its own “database API” since it’s also (at > its heart) a distributed database. > > In all fairness to Basho, I also think that was probably the right > call. It’s not a file system and doesn’t pretend to be one, it’s > just > a distributed database that can store arbitrary numbers (and sizes) > of > objects in a cluster with fault-tolerance, multi-data center tenancy > and so on (I also have no connection with Basho so if it sounds like > I’m trying to sales pitch it or something, I’m not, I’m simply trying > to be accurate in describing it given that we’re also evaluating it > as > a possible technology to incorporate into FreeNAS, for which the > topic > of S3-compatible object storage comes up fairly frequently. > > Now glusterfs is another kettle of fish, and like I said, I think the > bigger question there is just what FreeBSD’s relationship with it is. > The https://wiki.freebsd.org/GlusterFS wiki was created over a year > ago and seems to be suffering from an overall lack of specific goals. > Does the FreeBSD project want glusterfs and/or is there anyone in the > FreeBSD community even interested in deploying it in a production > scenario? Is anyone willing to actually maintain a port, just as the > minimum price of admission? > > The bug tracking this, namely > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194409, has been > open for almost a year now and while it looks like ‘craig001” has > been > trying fairly hard to get 3.7.2 up and working, he’s hitting some > walls and it’s not even clear he’s in contact with the right folks. > > As the previous glusterfs mail thread indicated, I was also happy to > add an earlier version of his port to the FreeNAS repo and play with > it there, though I was forced to take it back out of FreeNAS when we > ran into a number of issues just trying to make it work with the > Linux > glusterfs reference VMs we set up. Should the underlying port get > some love again, it would be the simplest thing in the world for me > to > refresh our copy of it and uncomment the line which adds glusterfs to > the FreeNAS manifest (and being a storage appliance, it’s rather of > an > obvious test bed for this since people running FreeNAS are also the > target audience for technologies like this). > > If getting glusterfs up as a stable, working substrate is also a > prerequisite for getting pNFS, then I think a lot of us in the > storage > industry would be the first to say “What do we have to do? How can > we > help?” I already volunteered to send Rick an AMD64 architecture > machine for development, but I suspect that’s actually the least of > our obstacles right now. :-) A couple of points... ;) * We submitted a patch for FreeBSD's fuse... that seems to have just sat, and sat, and sat... waiting for review and gone nowhere. That would be good to get fixed. (just saying) * Our guy who was putting in most of the effort for cross platform porting (Harsha), left Red Hat. Just added him to CC anyway, in case he has insight. :) * I also left Red Hat. I'm still doing a bit of stuff with Gluster, but that's likely to taper off after I finish getting the new "Gluster Forge" project done. Hopefully Niels (he knows Gluster NFS) and Kaleb (Gluster greybeard ;>) can move this forward. Regards and best wishes, Justin Clift From owner-freebsd-fs@freebsd.org Mon Sep 7 07:11:03 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2143F9CCC96 for ; Mon, 7 Sep 2015 07:11:03 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from pv33p03im-asmtp001.me.com (pv33p03im-asmtp001.me.com [17.143.180.10]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EB2F717D6 for ; Mon, 7 Sep 2015 07:11:02 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from [10.20.30.11] (75-101-82-48.static.sonic.net [75.101.82.48]) by pv33p03im-asmtp001.me.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Mar 31 2015)) with ESMTPSA id <0NUA004CTP9PIM20@pv33p03im-asmtp001.me.com> for freebsd-fs@freebsd.org; Mon, 07 Sep 2015 07:10:43 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2015-09-07_01:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 compositescore=0.994460221061599 phishscore=0 kscore.is_spamscore=0 rbsscore=0.994460221061599 recipient_to_sender_totalscore=0 spamscore=0 urlsuspectscore=0.994460221061599 adultscore=0 kscore.compositescore=0 circleOfTrustscore=0 suspectscore=1 recipient_domain_to_sender_totalscore=0 bulkscore=0 recipient_domain_to_sender_domain_totalscore=0 recipient_to_sender_domain_totalscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1412110000 definitions=main-1509070124 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 9.0 \(3093\)) Subject: Re: glusterfs + FreeBSD [was Re: CEPH + FreeBSD] From: Jordan Hubbard In-reply-to: <051286539482bdb811694897aacd1b2f@postgresql.org> Date: Mon, 07 Sep 2015 00:10:37 -0700 Cc: Kaleb Keithley , Niels de Vos , Rick Macklem , freebsd-fs@freebsd.org, Rakshith Venkatesh , Justin Clift , harsha@harshavardhana.net Content-transfer-encoding: quoted-printable Message-id: <531C82BC-5DE8-4C1C-BD46-6788D3F2CD6E@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <382BA1FD-032E-4F21-9460-D2119340E21B@icloud.com> <051286539482bdb811694897aacd1b2f@postgresql.org> To: justin@postgresql.org X-Mailer: Apple Mail (2.3093) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 07:11:03 -0000 > On Sep 6, 2015, at 11:39 PM, justin@postgresql.org wrote: >=20 > A couple of points... ;) Thanks for the rapid reply! > * We submitted a patch for FreeBSD's fuse... that seems to have just = sat, > and sat, and sat... waiting for review and gone nowhere. That would = be > good to get fixed. (just saying) Can you give a citation to this? E.g. a bug # or a github pull request = or whatever you used to submit it? Perhaps I can help shepherd it in. = A working FUSE is good for a lot of various things. > * Our guy who was putting in most of the effort for cross platform = porting > (Harsha), left Red Hat. Just added him to CC anyway, in case he has > insight. :) >=20 > * I also left Red Hat. I'm still doing a bit of stuff with Gluster, = but > that's likely to taper off after I finish getting the new "Gluster = Forge" > project done. Huh. Are there any of you left, over at Red Hat? ;-) Nonetheless, thanks for adding more appropriate folks to the CC line and = hey, if any of you filesystems folks are looking for a job, we=E2=80=99re = hiring. :-) (Sorry, sorry, I know this is the wrong list for that but hey, I = couldn=E2=80=99t resist!). - Jordan From owner-freebsd-fs@freebsd.org Mon Sep 7 08:05:21 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9D9E9CCD5D for ; Mon, 7 Sep 2015 08:05:21 +0000 (UTC) (envelope-from www-data@meldrar.postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher DHE-RSA-AES256-SHA256 (256/256 bits)) (Client CN "*.postgresql.org", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 837F736E for ; Mon, 7 Sep 2015 08:05:21 +0000 (UTC) (envelope-from www-data@meldrar.postgresql.org) Received: from www-data by meldrar.postgresql.org with local (Exim 4.80) (envelope-from ) id 1ZYrQM-0006Qs-T7; Mon, 07 Sep 2015 08:05:10 +0000 To: Jordan Hubbard Subject: Re: glusterfs + FreeBSD [was Re: CEPH + FreeBSD] X-PHP-Originating-Script: 0:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 07 Sep 2015 08:05:10 +0000 From: justin@postgresql.org Cc: Kaleb Keithley , Niels de Vos , Rick Macklem , , Rakshith Venkatesh , Justin Clift , In-Reply-To: <531C82BC-5DE8-4C1C-BD46-6788D3F2CD6E@icloud.com> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <382BA1FD-032E-4F21-9460-D2119340E21B@icloud.com> <051286539482bdb811694897aacd1b2f@postgresql.org> <531C82BC-5DE8-4C1C-BD46-6788D3F2CD6E@icloud.com> Message-ID: X-Sender: justin@postgresql.org User-Agent: Roundcube Webmail/0.7.2 Sender: www-data X-Pg-Spam-Score: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 08:05:21 -0000 On 2015-09-07 07:10, Jordan Hubbard wrote: >> On Sep 6, 2015, at 11:39 PM, justin@postgresql.org wrote: >> >> A couple of points... ;) > > Thanks for the rapid reply! > >> * We submitted a patch for FreeBSD's fuse... that seems to have just >> sat, >> and sat, and sat... waiting for review and gone nowhere. That >> would be >> good to get fixed. (just saying) > > Can you give a citation to this? E.g. a bug # or a github pull > request or whatever you used to submit it? Perhaps I can help > shepherd it in. A working FUSE is good for a lot of various things. Went hunting, and it might have been this one, which was merged after all: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192701 (I really haven't been keeping an eye on things ;>) >> * Our guy who was putting in most of the effort for cross platform >> porting >> (Harsha), left Red Hat. Just added him to CC anyway, in case he >> has >> insight. :) >> >> * I also left Red Hat. I'm still doing a bit of stuff with Gluster, >> but >> that's likely to taper off after I finish getting the new "Gluster >> Forge" >> project done. > > Huh. Are there any of you left, over at Red Hat? ;-) I think the Red Hat headcount is about 7-8k now, so there's probably a few left. ;) (I think it was about 3k when I first started). > Nonetheless, thanks for adding more appropriate folks to the CC line > and hey, if any of you filesystems folks are looking for a job, we’re > hiring. :-) > > (Sorry, sorry, I know this is the wrong list for that but hey, I > couldn’t resist!). Meh. I'm over IT. Getting into CNC stuff atm, and hopefully biomedical stuff soonish. ;) + Justin From owner-freebsd-fs@freebsd.org Tue Sep 8 15:07:43 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81875A00C2B for ; Tue, 8 Sep 2015 15:07:43 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5F67F1DBE for ; Tue, 8 Sep 2015 15:07:42 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from c-66-41-25-68.hsd1.mn.comcast.net ([66.41.25.68] helo=[192.168.0.107]) by mail.physics.umn.edu with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1ZZJqE-000Cnm-R9 for freebsd-fs@freebsd.org; Tue, 08 Sep 2015 09:25:46 -0500 Subject: Re: CEPH + FreeBSD To: freebsd-fs@freebsd.org References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> From: Graham Allan Message-ID: <55EEEFE7.9080809@physics.umn.edu> Date: Tue, 8 Sep 2015 09:25:43 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 15:07:43 -0000 On 9/6/2015 12:19 AM, Jordan Hubbard wrote: > > There are at least two distributed (multi-node) object stores for > FreeBSD that I know of. > > One is glusterfs, for which I’m not even really clear on the status > of the ports for. I don’t see any glusterfs port in the master > branch of https://github.com/freebsd/freebsd-ports (or > https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for > that matter). > > Our FreeNAS ports tree (https://github.com/freenas/ports), in which > we have a bit more latitude to add and curate our own ports, has both > a net/glusterfs and sysutils/glusterfs, from separate sources (looks > like we need to clean things up) - net/glusterfs lists > craig001@lerwick.hopto.org as the MAINTAINER and is at version 3.6.2. > The sysutils/glusterfs port lists bapt@FreeBSD.org as the MAINTAINER > and is at version 20140811. For the sake of completeness MooseFS is also available on FreeBSD and probably also satisfies your crtieria (never used it, just seen it mentioned before on this list). I'm still hoping to get a chance to play with glusterfs sometime in the near future mostly on Sci Linux, but with some opportunity to check it out on FreeBSD as well. Though my interest is more in a distributed *posix* filesystem, and we're already abusing HDFS via its fuse interface as such; perhaps I shouldn't be going down the same path again! Graham From owner-freebsd-fs@freebsd.org Tue Sep 8 23:15:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 975C89CCCB6 for ; Tue, 8 Sep 2015 23:15:51 +0000 (UTC) (envelope-from fusionfoto@gmail.com) Received: from mail-la0-x22d.google.com (mail-la0-x22d.google.com [IPv6:2a00:1450:4010:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23B441355 for ; Tue, 8 Sep 2015 23:15:51 +0000 (UTC) (envelope-from fusionfoto@gmail.com) Received: by lanb10 with SMTP id b10so78803848lan.3 for ; Tue, 08 Sep 2015 16:15:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=smjRWiC/Opiyoq4DSjDGW03BFCiCAvtrBe/tKbjQ0Do=; b=xtZaRqYzlyANLP6r9cBkzUnq5V5JWN8MiR6/GbAeUrVnHqCy0qfafdw16nfwTZxW+G 6JmLXwMPv5QzjwSNiaJbVMDqu9ItnNlsie1s5GK4O0yY+FRbbwNSPpOiIrj+DoJRTbiL VXyRr9FB/h2682TvElpzHWkg84u0NZlbuqjAtObZ03CkCIHPbWxfDgTZI1XRPPuepz8Q QIqml4UXl4DFllWpvaLQEem7Pk6jfUzqTnJ4LLU1tmX5NXoimT6x0qHDI2TYvmzSVc8w 9j+E552Ma7N3lTTefxrBoXxjOHjO2nFFgoZM7fbfglIvMiXkcoXtSqYUcPuVY9OSdstg AFSA== MIME-Version: 1.0 X-Received: by 10.152.7.68 with SMTP id h4mr25509088laa.94.1441754149295; Tue, 08 Sep 2015 16:15:49 -0700 (PDT) Received: by 10.114.245.197 with HTTP; Tue, 8 Sep 2015 16:15:49 -0700 (PDT) Date: Tue, 8 Sep 2015 19:15:49 -0400 Message-ID: Subject: TRIM support (same bug as linux?) From: FF To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 23:15:51 -0000 I'm asking a pretty vague question and I apologize in advance, not trying to troll. The question has to do with whether FreeBSD is using TRIM the same way as recent Linux kernels. https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ seems to imply that there are instabilities that can occur. Trying to avoid duplicating effort if this has already been addressed or if its a complete dead alley because there isn't a commonality. Thanks in advance! -- FF From owner-freebsd-fs@freebsd.org Tue Sep 8 23:42:49 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ABFDA00CDE for ; Tue, 8 Sep 2015 23:42:49 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C5671907 for ; Tue, 8 Sep 2015 23:42:48 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wiclk2 with SMTP id lk2so137720485wic.0 for ; Tue, 08 Sep 2015 16:42:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=DykaCeFZ2MieToxZzrBqJJ9ceXm7rEvwI5aggLvU09M=; b=PBF4sfDJYLlqD5ZulsjHv4weKhZWb5AO6R9mNDIYmxvwSoQ0JnnwLW7by2ce6u0r9T zAght1YOC29JhJZdU+p4H5N5CYXTZIS8ArjUVqVrTXsTUcyzLCJ5E6N2CszvKo2oEBIW gEG3j97hXkokZmKxwNy39+/dxF/WCQJwHe4MjEjMbpeWKfYH2BphS/peYvG0ByGDWRUs i7x2DIWJUxLNo50mU6I+FDkarKBamCJ2S69Ne5QdLSn6CNdGjjGRPAZ8Y7FyMpSl8ycE rwlf1CJTOsnYhYqSBLJB4saYtWv/DeRNlGIGF0/GJqBhzpcY9XMCMiO04X39Wec+U2pR BpZg== X-Gm-Message-State: ALoCoQnOYBMbPgIfVhIZjiJ3muC7RYaEWXRa4xluV4L6AYHccaDu5lgDKt0t7ODFZAyeK+7t6PjG X-Received: by 10.194.58.40 with SMTP id n8mr55481179wjq.134.1441755761358; Tue, 08 Sep 2015 16:42:41 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by smtp.gmail.com with ESMTPSA id ev6sm855556wib.12.2015.09.08.16.42.39 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 08 Sep 2015 16:42:40 -0700 (PDT) Subject: Re: TRIM support (same bug as linux?) To: freebsd-fs@freebsd.org References: From: Steven Hartland Message-ID: <55EF7265.2050309@multiplay.co.uk> Date: Wed, 9 Sep 2015 00:42:29 +0100 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2015 23:42:49 -0000 Nope FreeBSD is not affected by this. On 09/09/2015 00:15, FF wrote: > I'm asking a pretty vague question and I apologize in advance, not trying > to troll. > > The question has to do with whether FreeBSD is using TRIM the same way as > recent Linux kernels. > > https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ seems > to imply that there are instabilities that can occur. Trying to avoid > duplicating effort if this has already been addressed or if its a complete > dead alley because there isn't a commonality. > > Thanks in advance! > > From owner-freebsd-fs@freebsd.org Wed Sep 9 01:11:34 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06C5DA004D4 for ; Wed, 9 Sep 2015 01:11:34 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com [209.85.192.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE8DC11F3 for ; Wed, 9 Sep 2015 01:11:33 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by qgx61 with SMTP id 61so98530037qgx.3 for ; Tue, 08 Sep 2015 18:11:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=1DmmEtoUJ53gFYHYmmF8Adg9a6Ywc+pGYPP4CvJWJx8=; b=gA58QIc7oGHq6Xw86P6JrbaIVTmgdLBsaBYiLliUVHg7FqevA/Vs6j8VcMCaVmul7E aZ6WYva73f4ChTMLPH3yzHvhPhXNoL1OTQOQ2l0gcJjSqukRWSI0cSwJa8nXrSV8PAeV bWNm2Wt4pMzbJsbJpNcu4mE2w1zHFbzUXcWolXMkaC4y0Vxin/NGQxh7u+EAHCEtm1fc +AzcMJ1YbMyYZMnEHW6TKilam+i5w2n+UFDOexeRQaTBJAl3xUBDtoukt+o3zdUJV1U1 yEJIb8cl+BwnM3QZ/qPz3jgsdK3r4A0CEMtBxTkudKaplW/OhZbZu/Of6XfHV5Amy/Tp wOWg== X-Gm-Message-State: ALoCoQm2US04+gNNMHRWfSmj3scBaMUo2CG1MrWOwraItPIcfI45cXEfgHDYUrfhZjn1OUa6xBKT X-Received: by 10.140.94.111 with SMTP id f102mr39610448qge.102.1441761085932; Tue, 08 Sep 2015 18:11:25 -0700 (PDT) Received: from ?IPv6:2600:1017:b417:a4b4:113a:e36f:3d61:2861? ([2600:1017:b417:a4b4:113a:e36f:3d61:2861]) by smtp.gmail.com with ESMTPSA id 4sm2945529qhr.33.2015.09.08.18.11.24 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 08 Sep 2015 18:11:25 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: TRIM support (same bug as linux?) From: Mark Saad X-Mailer: iPhone Mail (12H321) In-Reply-To: <55EF7265.2050309@multiplay.co.uk> Date: Tue, 8 Sep 2015 21:11:24 -0400 Cc: "freebsd-fs@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <1BCC8B97-6C79-4864-8405-BA687577808A@longcount.org> References: <55EF7265.2050309@multiplay.co.uk> To: Steven Hartland X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 01:11:34 -0000 > On Sep 8, 2015, at 7:42 PM, Steven Hartland wrot= e: >=20 > Nope FreeBSD is not affected by this. >=20 >> On 09/09/2015 00:15, FF wrote: >> I'm asking a pretty vague question and I apologize in advance, not trying= >> to troll. >>=20 >> The question has to do with whether FreeBSD is using TRIM the same way as= >> recent Linux kernels. >>=20 >> https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ seem= s >> to imply that there are instabilities that can occur. Trying to avoid >> duplicating effort if this has already been addressed or if its a complet= e >> dead alley because there isn't a commonality. >>=20 >> Thanks in advance! >=20 It would be interesting if anyone could explain the reasons why ufs/ffs and z= fs and FreeBSD are not effected by this issue . My rudimentary understandi= ng of the issue was that it's wasn't just a software glitch in the ext files= ystem and bad firmware but a combination of that and Linux poorly supporting= a ata modes that are not supported at all on FreeBSD . Is that correct ? --- Mark Saad | nonesuch@longcount.org > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 9 01:28:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE7C4A00BBB for ; Wed, 9 Sep 2015 01:28:02 +0000 (UTC) (envelope-from thomasrcurry@gmail.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 790EE17AB for ; Wed, 9 Sep 2015 01:28:02 +0000 (UTC) (envelope-from thomasrcurry@gmail.com) Received: by iofh134 with SMTP id h134so5512875iof.0 for ; Tue, 08 Sep 2015 18:28:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9Bpm+Lv9ZANPcU2YjS/plCzyR4jOeTsTOgRGjry5QP8=; b=x7e8z/wKO152pWIZAXChYZa/ZQW0nIvxy6ctrGpmTWyXayet2SnVi4iEDCvyYRb/8s bX09JgE0Hor/clOG+2VXE7MfC9yolGye8OcCbslQvbhFsQa+ggWvIt9dc9SrNhLiHVTF IhAIFxBUY6hDEpyfHmuY1pGtpmbwnt6amYPUHPn2LIarA4MD+3S9F0+M0mLhgLfYASR7 5q6eOpTOFEL7MlXJKs73vLR1O5z8MzPULaDR2qHv+oCBpNKJ/BGm1WAeccbsoIau9F4U QeuWJhD1wvCg5Q0ib6MXMRgrjSzS99UmIAQBaYLoCiduxWRImaiCUhMC7ckQmsc9QaS0 qQaQ== MIME-Version: 1.0 X-Received: by 10.107.10.165 with SMTP id 37mr45278657iok.120.1441762081815; Tue, 08 Sep 2015 18:28:01 -0700 (PDT) Received: by 10.107.4.72 with HTTP; Tue, 8 Sep 2015 18:28:01 -0700 (PDT) In-Reply-To: <1BCC8B97-6C79-4864-8405-BA687577808A@longcount.org> References: <55EF7265.2050309@multiplay.co.uk> <1BCC8B97-6C79-4864-8405-BA687577808A@longcount.org> Date: Tue, 8 Sep 2015 21:28:01 -0400 Message-ID: Subject: Re: TRIM support (same bug as linux?) From: Tom Curry To: Mark Saad Cc: Steven Hartland , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 01:28:02 -0000 I'm no expert but if memory serves the issue had to do with concurrent and/or queued trim which is not implemented in FreeBSD. On Tue, Sep 8, 2015 at 9:11 PM, Mark Saad wrote: > > > > On Sep 8, 2015, at 7:42 PM, Steven Hartland > wrote: > > > > Nope FreeBSD is not affected by this. > > > >> On 09/09/2015 00:15, FF wrote: > >> I'm asking a pretty vague question and I apologize in advance, not > trying > >> to troll. > >> > >> The question has to do with whether FreeBSD is using TRIM the same way > as > >> recent Linux kernels. > >> > >> https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ > seems > >> to imply that there are instabilities that can occur. Trying to avoid > >> duplicating effort if this has already been addressed or if its a > complete > >> dead alley because there isn't a commonality. > >> > >> Thanks in advance! > > > > It would be interesting if anyone could explain the reasons why ufs/ffs > and zfs and FreeBSD are not effected by this issue . My rudimentary > understanding of the issue was that it's wasn't just a software glitch in > the ext filesystem and bad firmware but a combination of that and Linux > poorly supporting a ata modes that are not supported at all on FreeBSD . > Is that correct ? > > > --- > Mark Saad | nonesuch@longcount.org > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Sep 9 01:31:43 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7709AA00E51 for ; Wed, 9 Sep 2015 01:31:43 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qk0-f182.google.com (mail-qk0-f182.google.com [209.85.220.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3606D1BAA for ; Wed, 9 Sep 2015 01:31:42 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by qkfq186 with SMTP id q186so52340633qkf.1 for ; Tue, 08 Sep 2015 18:31:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=GhMwpj3mi441jp+SsOrf9fVYsFj2wCVSaH58JcfXfQw=; b=FedXDZhV641SIGCgojGu4pyDxh9DMnvaqJjcRfiaTPG7eiq4NXQtixkvi6cKoGkOWe tOQnA20bqmPCST0II+ipCAsD0yC4ZptqnLZSQxAY/C9DkCKAcLVQw3Jcw9RYFKgawCGN jiMNJtk4+2OOSmQO69wW/zCCCCSAR/JQUWPvaoJpgyMd/Q5fZbb5TibFfmaE1gx4knXG oM5fY8aztVWYUMPpUkzdjV4IP+cxL8MYPEEO3F/o/MoDyNd0Gkd2Y9r3hGFy/GD+ujED d/LLJ9CWyr/vXN56bvQz1lCAETEHUKHRqrtWUtrqiHbDcWqz2I4ASos2x9+/OWOTjs2q 69Ww== X-Gm-Message-State: ALoCoQnD26/mEm4qnrUlwUFxCDJDURa/l6fQYy87a5WdFdWwQ8/ps915jzVaHFTp/Bk8NVsYjHHa X-Received: by 10.55.33.74 with SMTP id h71mr39970593qkh.71.1441762295740; Tue, 08 Sep 2015 18:31:35 -0700 (PDT) Received: from ?IPv6:2600:1017:b417:a4b4:113a:e36f:3d61:2861? ([2600:1017:b417:a4b4:113a:e36f:3d61:2861]) by smtp.gmail.com with ESMTPSA id c109sm2968671qga.16.2015.09.08.18.31.35 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 08 Sep 2015 18:31:35 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: CEPH + FreeBSD From: Mark Saad X-Mailer: iPhone Mail (12H321) In-Reply-To: <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> Date: Tue, 8 Sep 2015 21:31:34 -0400 Cc: Jordan Hubbard , "freebsd-fs@freebsd.org" , Rakshith Venkatesh Content-Transfer-Encoding: quoted-printable Message-Id: References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 01:31:43 -0000 All What about leofs. It's in ports has and s3 obj store and NFS support out o= f the box http://www.freshports.org/databases/leofs/ http://leo-project.net --- Mark Saad | nonesuch@longcount.org > On Sep 6, 2015, at 6:18 PM, Rick Macklem wrote: >=20 > Jordan Hubbard wrote: >>=20 >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem wrote: >>>=20 >>> Slightly off topic but, btw, there is a port of GLusterFS and those folk= s >>> do seem >>> interested in seeing it brought "up to speed". I am not sure how mature i= t >>> is at >>> this point, but it has been known to build on amd64. (I don't have an am= d64 >>> machine, >>> so I haven't gotten around to building/testing it, but I do plan to try a= nd >>> use >>> it as a basis for a pNFS server, if I can figure out how to get the FH i= nfo >>> out of it. >>> I'm working on that;-) >>=20 >> There are at least two distributed (multi-node) object stores for FreeBSD= >> that I know of. >>=20 >> One is glusterfs, for which I=E2=80=99m not even really clear on the stat= us of the >> ports for. I don=E2=80=99t see any glusterfs port in the master branch o= f >> https://github.com/freebsd/freebsd-ports (or >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that >> matter). >>=20 >> Our FreeNAS ports tree (https://github.com/freenas/ports), in which we ha= ve a >> bit more latitude to add and curate our own ports, has both a net/gluster= fs >> and sysutils/glusterfs, from separate sources (looks like we need to clea= n >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port lists >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. >>=20 >> I=E2=80=99m not really sure about the provenance since we were simply eva= luating >> glusterfs for awhile and may have pulled in interim versions from those >> sources, but obviously it would be best to have an official maintainer an= d >> someone in the FreeBSD project actually curating a glusterfs port so that= >> all users of FreeBSD can use it. It would also be fairly key to your own= >> efforts, assuming you decide to pursue glusterfs as a foundation technolo= gy >> for pNFS. >>=20 >> The other object store, which is pretty mature and is currently leading t= he >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. There= is >> a port currently in databases/riak but it=E2=80=99s pretty out of date at= version >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade of >> RiakCS). >>=20 >> We are very interested in investigating various ways of shimming RiakCS t= o >> NFS, using RiakCS a back-end store. Is that something you=E2=80=99d be a= menable to >> discussing? I=E2=80=99d be happy to send you an amd64 architecture mach= ine to >> develop on. :) > Hmm. =46rom a quick look at their web page (I looked once before as well),= I don't > think RiakCS has what I need to do pNFS in a reasonable (for me) amount of= effort. > Two things that glusterFS has that I am hoping to use (and I don't think R= iakCS has > either of these) are: > - A Fuse file system interface which allows the kernel nfsd to access the s= tore as > a file system, so that it can provide the metadata services (NFS without t= he reads/writes). > - A userland NFSv3 server in each node which will allow the node to act as= a data server. >=20 > If I am wrong and RiakCS does support a VFS file system interface (via Fus= e or ???), then > please correct me. With that, it might be a reasonable alternative. > I'll admit I've spent a little time looking at the glusterFS sources and h= aven't yet > solved the problem of how to generate the file handles I need, but that so= unds trivial > compared with an entire Fuse and/or VFS file system interface, I think? >=20 > In general, using a cloud object store to implement a pNFS server is a *mi= s*use of > the technology, imho. I think it may be possible with glusterFS, since tha= t technology > seems to be based on a cluster file system, which is what a pNFS server ca= n also use. >=20 > I think there would be a lot of work involved in mapping a POSIX file syst= em onto the > Riak database and then exporting that via NFS, etc. It might also be more p= ractical to > do this via a userland NFS service than the kernel based one currently in = FreeBSD. > (glusterFS is starting to use the NFS-ganesha server, but I believe it is p= retty Linux specific, > so I doubt it would be useful for Riak running on FreeBSD?) >=20 > rick >=20 >> - Jordan > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 9 01:42:29 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 598129CD318 for ; Wed, 9 Sep 2015 01:42:29 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: from mail-qg0-x236.google.com (mail-qg0-x236.google.com [IPv6:2607:f8b0:400d:c04::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11C281F2E for ; Wed, 9 Sep 2015 01:42:29 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: by qgev79 with SMTP id v79so98830885qge.0 for ; Tue, 08 Sep 2015 18:42:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type; bh=ooIYsWVFf8UEm3qIq6pkx2WmiBJrvtHRCggxRIznGd8=; b=bSUg6HLqeB6yvIgbzFR4nL0FBQ8ta1EJ4w9EmbIWtcRl6i8KEk6Y3Zi0chEn6CitIH 19+frkVpimzHb3Tp64JaHKqh5Hi+ch5qVpvKXzjpWd8lkNtijTg5sQmVWydcnCik3zYC +b/c1PkvrTT90iFjXR7/36I3N1/1oZnABVwkCdGGf/Ri/4Rqid/UjYX0MY92qy95u6jV Xr7SKsLrG+maWU0/dwLxe0Kz1fydWeMkA4PFtGG1iKUH+kVX/HkBsc5upErb1BObkvKA 5eUTyT7fkU+ZZ9V2NV34C2JNkOLmDAIwXuvHNXcAxgBaLko5ByM45fnXfbZXUY2Zs0nr 8QwA== X-Received: by 10.140.150.139 with SMTP id 133mr43151487qhw.58.1441762948059; Tue, 08 Sep 2015 18:42:28 -0700 (PDT) Received: from kan ([2601:18f:0:1570:226:18ff:fe00:232e]) by smtp.gmail.com with ESMTPSA id s52sm2983604qge.19.2015.09.08.18.42.26 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Sep 2015 18:42:26 -0700 (PDT) Date: Tue, 8 Sep 2015 21:42:21 -0400 From: Alexander Kabaev To: Tom Curry Cc: Mark Saad , "freebsd-fs@freebsd.org" Subject: Re: TRIM support (same bug as linux?) Message-ID: <20150908214221.7a1702e0@kan> In-Reply-To: References: <55EF7265.2050309@multiplay.co.uk> <1BCC8B97-6C79-4864-8405-BA687577808A@longcount.org> X-Mailer: Claws Mail 3.12.0 (GTK+ 2.24.28; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/Z_P.WkuqF3EOgH1ykpD7vRg"; protocol="application/pgp-signature" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 01:42:29 -0000 --Sig_/Z_P.WkuqF3EOgH1ykpD7vRg Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Tue, 8 Sep 2015 21:28:01 -0400 Tom Curry wrote: > I'm no expert but if memory serves the issue had to do with concurrent > and/or queued trim which is not implemented in FreeBSD. >=20 > On Tue, Sep 8, 2015 at 9:11 PM, Mark Saad > wrote: >=20 > > > > > > > On Sep 8, 2015, at 7:42 PM, Steven Hartland > > > > > wrote: > > > > > > Nope FreeBSD is not affected by this. > > > > > >> On 09/09/2015 00:15, FF wrote: > > >> I'm asking a pretty vague question and I apologize in advance, > > >> not > > trying > > >> to troll. > > >> > > >> The question has to do with whether FreeBSD is using TRIM the > > >> same way > > as > > >> recent Linux kernels. > > >> > > >> https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ > > seems > > >> to imply that there are instabilities that can occur. Trying to > > >> avoid duplicating effort if this has already been addressed or > > >> if its a > > complete > > >> dead alley because there isn't a commonality. > > >> > > >> Thanks in advance! > > > > > > > It would be interesting if anyone could explain the reasons why > > ufs/ffs and zfs and FreeBSD are not effected by this issue . My > > rudimentary understanding of the issue was that it's wasn't just a > > software glitch in the ext filesystem and bad firmware but a > > combination of that and Linux poorly supporting a ata modes that > > are not supported at all on FreeBSD . Is that correct ? > > > > > > --- > > Mark Saad | nonesuch@longcount.org Wasn't the root cause the improper bio splitting code in md/raid0 code and nothing special in firmware? FreeBSD shares no such code and as such is not affected. --=20 Alexander Kabaev --Sig_/Z_P.WkuqF3EOgH1ykpD7vRg Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJV7459XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRDNUY3RDk5NTk5QjY0MUUxM0M1MTU2OTEw NzEzMjI5OTkyNzkyRTdFAAoJEAcTIpmSeS5+NacQAKygqqyAOdTvbQUQ3EaVTGC2 GK8OzzgBqvG1oXQmNZj95Dd5tEBwWbLfq4A48eFDKcFlmdjC/ldVTYlG4GrYiiml c8h9UarrLA9nhjVFBCYep4iRT8/jS13Xq0ejmZETltrjoEssLLbIhO4oQaXebGwB qWlKUunWkMtO5akJ2ryoNTuBCKHPnHMhRF6U4rTtAZvooMizz3DuqerD0CD7PIov tRRxwN1KTGyZfBRxfApAIfyQoSWIOTP3QxyZFZ0M+wqz7YDqPQSZUsivnY4YExUp se/iFLR/ltYD7UKl3wE5l93pc7Ly/xWLb8/w56gwZApIuTzmPTM6AzvH2D3WJ34/ PWfq3O2v8oFN+0MTULjU6/m9RbDn3QLGmiMob6/IFp17ANRP70+glhK/W8EDmsNj jnMt8QwDTKqkCcruWLUr8/zsyQdeQJB8O2TdhCzs99foeyqHa2wvfjPYWqgNJLvm V1KX2g1bGv23h3UGOIR4wRR6sFTXkPPkDOOqP25WrJSkkjAkDgekSUHcIRCbJnsZ /KFND6xT6kvPRMx6MJBP0/pvZQOvaLCO9xPWzwk+CDBZQoFS39EaXOzXrw5O6CR1 DU4voYLwRHEVE4RI28VILzr65v8ZgDSjJMBcSnhT6Vw21i5dvrVmZwCZF4Ehtqcl ezzxVDJnv0LHt6pVZH4r =nkda -----END PGP SIGNATURE----- --Sig_/Z_P.WkuqF3EOgH1ykpD7vRg-- From owner-freebsd-fs@freebsd.org Wed Sep 9 01:59:56 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AACE79CDB95 for ; Wed, 9 Sep 2015 01:59:56 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-yk0-x22c.google.com (mail-yk0-x22c.google.com [IPv6:2607:f8b0:4002:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6556014EF for ; Wed, 9 Sep 2015 01:59:56 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by ykei199 with SMTP id i199so148393612yke.0 for ; Tue, 08 Sep 2015 18:59:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=13zxjw/fbaoOT+y7tICfPqIfGYzeACGtkB8PQnfhNrU=; b=krrhGgrJzgwx0imzV5YpwJLfummNs8YChxx7WgjuVzdaUfgl6srOlyMZtMEe8pU3Qz o09Im3A9aBO6SOCJoq3501HmlxDBDg9z8BM5i1Z/zaTwqkp7zI0Wxsjo+jCvO0IQv86s EHY8B2cpq1xClzKUzcqbVCOpVKOabO1/3FYV7N/dewFni9RSCliv8W2JYp1vUYqD41VG SguI/Kisur55ozsbsIK/4o8Z0MBwwq0uglEwprWHS4OodbG9US6BA/iRWmdPow/nGZkA aLB/i7Jfcz/A42+6m6XKi44cKAzNOT6OZ7NhPc5Vr7IWc5NhRxiZ20jlvwvZRwqZON4P F+6g== MIME-Version: 1.0 X-Received: by 10.170.111.67 with SMTP id d64mr33885498ykb.70.1441763994811; Tue, 08 Sep 2015 18:59:54 -0700 (PDT) Received: by 10.103.82.149 with HTTP; Tue, 8 Sep 2015 18:59:54 -0700 (PDT) In-Reply-To: References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> Date: Wed, 9 Sep 2015 11:59:54 +1000 Message-ID: Subject: Re: CEPH + FreeBSD From: Outback Dingo To: Mark Saad Cc: Rick Macklem , "freebsd-fs@freebsd.org" , Rakshith Venkatesh , Jordan Hubbard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 01:59:56 -0000 On Wed, Sep 9, 2015 at 11:31 AM, Mark Saad wrote: > All > What about leofs. It's in ports has and s3 obj store and NFS support ou= t > of the box > > > http://www.freshports.org/databases/leofs/ > http://leo-project.net LeoFS supperts NFSv3 and does not have a lock manager.... > > > > --- > Mark Saad | nonesuch@longcount.org > > > On Sep 6, 2015, at 6:18 PM, Rick Macklem wrote: > > > > Jordan Hubbard wrote: > >> > >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem wrote= : > >>> > >>> Slightly off topic but, btw, there is a port of GLusterFS and those > folks > >>> do seem > >>> interested in seeing it brought "up to speed". I am not sure how > mature it > >>> is at > >>> this point, but it has been known to build on amd64. (I don't have an > amd64 > >>> machine, > >>> so I haven't gotten around to building/testing it, but I do plan to > try and > >>> use > >>> it as a basis for a pNFS server, if I can figure out how to get the F= H > info > >>> out of it. > >>> I'm working on that;-) > >> > >> There are at least two distributed (multi-node) object stores for > FreeBSD > >> that I know of. > >> > >> One is glusterfs, for which I=E2=80=99m not even really clear on the s= tatus of > the > >> ports for. I don=E2=80=99t see any glusterfs port in the master branc= h of > >> https://github.com/freebsd/freebsd-ports (or > >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that > >> matter). > >> > >> Our FreeNAS ports tree (https://github.com/freenas/ports), in which we > have a > >> bit more latitude to add and curate our own ports, has both a > net/glusterfs > >> and sysutils/glusterfs, from separate sources (looks like we need to > clean > >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the > >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port lists > >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. > >> > >> I=E2=80=99m not really sure about the provenance since we were simply = evaluating > >> glusterfs for awhile and may have pulled in interim versions from thos= e > >> sources, but obviously it would be best to have an official maintainer > and > >> someone in the FreeBSD project actually curating a glusterfs port so > that > >> all users of FreeBSD can use it. It would also be fairly key to your > own > >> efforts, assuming you decide to pursue glusterfs as a foundation > technology > >> for pNFS. > >> > >> The other object store, which is pretty mature and is currently leadin= g > the > >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. > There is > >> a port currently in databases/riak but it=E2=80=99s pretty out of date= at > version > >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade o= f > >> RiakCS). > >> > >> We are very interested in investigating various ways of shimming RiakC= S > to > >> NFS, using RiakCS a back-end store. Is that something you=E2=80=99d = be > amenable to > >> discussing? I=E2=80=99d be happy to send you an amd64 architecture m= achine to > >> develop on. :) > > Hmm. From a quick look at their web page (I looked once before as well)= , > I don't > > think RiakCS has what I need to do pNFS in a reasonable (for me) amount > of effort. > > Two things that glusterFS has that I am hoping to use (and I don't thin= k > RiakCS has > > either of these) are: > > - A Fuse file system interface which allows the kernel nfsd to access > the store as > > a file system, so that it can provide the metadata services (NFS > without the reads/writes). > > - A userland NFSv3 server in each node which will allow the node to act > as a data server. > > > > If I am wrong and RiakCS does support a VFS file system interface (via > Fuse or ???), then > > please correct me. With that, it might be a reasonable alternative. > > I'll admit I've spent a little time looking at the glusterFS sources an= d > haven't yet > > solved the problem of how to generate the file handles I need, but that > sounds trivial > > compared with an entire Fuse and/or VFS file system interface, I think? > > > > In general, using a cloud object store to implement a pNFS server is a > *mis*use of > > the technology, imho. I think it may be possible with glusterFS, since > that technology > > seems to be based on a cluster file system, which is what a pNFS server > can also use. > > > > I think there would be a lot of work involved in mapping a POSIX file > system onto the > > Riak database and then exporting that via NFS, etc. It might also be > more practical to > > do this via a userland NFS service than the kernel based one currently > in FreeBSD. > > (glusterFS is starting to use the NFS-ganesha server, but I believe it > is pretty Linux specific, > > so I doubt it would be useful for Riak running on FreeBSD?) > > > > rick > > > >> - Jordan > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Sep 9 02:26:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 000CAA008BF for ; Wed, 9 Sep 2015 02:26:00 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qg0-f45.google.com (mail-qg0-f45.google.com [209.85.192.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ABA281FF8 for ; Wed, 9 Sep 2015 02:26:00 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by qgez77 with SMTP id z77so100011695qge.1 for ; Tue, 08 Sep 2015 19:25:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=sZxAh77LQTkt0+YM1iDdcMu5yRzThQK0vc2W9Pzy9YU=; b=BuE+l4P6Gw4dd099INcrlP6N7YgDdkL4L/J53Vtl/TFwOolR0uD9o/3l8/RHqJwUTA AJYcRLi03LER07BP6PdtE+3M1p3PxVv2CCF7Oj08U+D7pHsYIq8y4XcNjVUlhz9HFQtV NB6KZcmOH/7YPbXgfRBbd9DzgrQku4j7qQ7AST2N8e1aeCzUwAo4795MomENH6GWRON6 fUARBouxsTQnTApbSia0lxpORU5ZwipEFWY9QceU/N3whyQtNYRXcvuu+wbOmmEXhMiy q9gLzOIfJD3gnmVLiJc63Q2M7jJqAOOVLE1JDYj5rjImc4RaMM7h8UnkjKqVXZ4nXJ5o 0y8A== X-Gm-Message-State: ALoCoQkoqtjmbMQSOqt3ziaN2y/sodBNv2rSGkmFlPwqDygjzEIiBYr7u1B/wQ8LNzI6+etokn/C X-Received: by 10.140.151.212 with SMTP id 203mr43018119qhx.1.1441765553511; Tue, 08 Sep 2015 19:25:53 -0700 (PDT) Received: from ?IPv6:2600:1017:b417:a4b4:113a:e36f:3d61:2861? ([2600:1017:b417:a4b4:113a:e36f:3d61:2861]) by smtp.gmail.com with ESMTPSA id p29sm3069566qkp.48.2015.09.08.19.25.52 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 08 Sep 2015 19:25:52 -0700 (PDT) Mime-Version: 1.0 (1.0) Subject: Re: CEPH + FreeBSD From: Mark Saad X-Mailer: iPhone Mail (12H321) In-Reply-To: Date: Tue, 8 Sep 2015 22:25:51 -0400 Cc: Rick Macklem , "freebsd-fs@freebsd.org" , Rakshith Venkatesh , Jordan Hubbard Message-Id: <8551C101-0E4B-4619-BF52-1B87BC6565D8@longcount.org> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> To: Outback Dingo Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 02:26:01 -0000 > On Sep 8, 2015, at 9:59 PM, Outback Dingo wrote: >=20 >=20 >=20 >> On Wed, Sep 9, 2015 at 11:31 AM, Mark Saad wrote= : >> All >> What about leofs. It's in ports has and s3 obj store and NFS support ou= t of the box >>=20 >>=20 >> http://www.freshports.org/databases/leofs/ >> http://leo-project.net >=20 > LeoFS supperts NFSv3 and does not have a lock manager....=20 >=20 Well, nlm it's not a hard requirement for most NFS things . NFS is there for= direct access of the data . So in its current form it's not a pnfs setup , b= ut there is no reason it could not be extended . =20 Well it could be worse , manta, joyent's compute and storage platform has a= NFS server written in node.js . Yes nfs in JavaScript , what else would you= expect from them . :) --- Mark Saad | nonesuch@longcount.org > =20 >>=20 >>=20 >>=20 >> --- >> Mark Saad | nonesuch@longcount.org >>=20 >> > On Sep 6, 2015, at 6:18 PM, Rick Macklem wrote: >> > >> > Jordan Hubbard wrote: >> >> >> >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem wrote= : >> >>> >> >>> Slightly off topic but, btw, there is a port of GLusterFS and those f= olks >> >>> do seem >> >>> interested in seeing it brought "up to speed". I am not sure how matu= re it >> >>> is at >> >>> this point, but it has been known to build on amd64. (I don't have an= amd64 >> >>> machine, >> >>> so I haven't gotten around to building/testing it, but I do plan to t= ry and >> >>> use >> >>> it as a basis for a pNFS server, if I can figure out how to get the FH= info >> >>> out of it. >> >>> I'm working on that;-) >> >> >> >> There are at least two distributed (multi-node) object stores for Free= BSD >> >> that I know of. >> >> >> >> One is glusterfs, for which I=E2=80=99m not even really clear on the s= tatus of the >> >> ports for. I don=E2=80=99t see any glusterfs port in the master branc= h of >> >> https://github.com/freebsd/freebsd-ports (or >> >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that= >> >> matter). >> >> >> >> Our FreeNAS ports tree (https://github.com/freenas/ports), in which we= have a >> >> bit more latitude to add and curate our own ports, has both a net/glus= terfs >> >> and sysutils/glusterfs, from separate sources (looks like we need to c= lean >> >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the >> >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port lists= >> >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. >> >> >> >> I=E2=80=99m not really sure about the provenance since we were simply e= valuating >> >> glusterfs for awhile and may have pulled in interim versions from thos= e >> >> sources, but obviously it would be best to have an official maintainer= and >> >> someone in the FreeBSD project actually curating a glusterfs port so t= hat >> >> all users of FreeBSD can use it. It would also be fairly key to your o= wn >> >> efforts, assuming you decide to pursue glusterfs as a foundation techn= ology >> >> for pNFS. >> >> >> >> The other object store, which is pretty mature and is currently leadin= g the >> >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. Th= ere is >> >> a port currently in databases/riak but it=E2=80=99s pretty out of date= at version >> >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade o= f >> >> RiakCS). >> >> >> >> We are very interested in investigating various ways of shimming RiakC= S to >> >> NFS, using RiakCS a back-end store. Is that something you=E2=80=99d b= e amenable to >> >> discussing? I=E2=80=99d be happy to send you an amd64 architecture m= achine to >> >> develop on. :) >> > Hmm. =46rom a quick look at their web page (I looked once before as wel= l), I don't >> > think RiakCS has what I need to do pNFS in a reasonable (for me) amount= of effort. >> > Two things that glusterFS has that I am hoping to use (and I don't thin= k RiakCS has >> > either of these) are: >> > - A Fuse file system interface which allows the kernel nfsd to access t= he store as >> > a file system, so that it can provide the metadata services (NFS witho= ut the reads/writes). >> > - A userland NFSv3 server in each node which will allow the node to act= as a data server. >> > >> > If I am wrong and RiakCS does support a VFS file system interface (via = Fuse or ???), then >> > please correct me. With that, it might be a reasonable alternative. >> > I'll admit I've spent a little time looking at the glusterFS sources an= d haven't yet >> > solved the problem of how to generate the file handles I need, but that= sounds trivial >> > compared with an entire Fuse and/or VFS file system interface, I think?= >> > >> > In general, using a cloud object store to implement a pNFS server is a *= mis*use of >> > the technology, imho. I think it may be possible with glusterFS, since t= hat technology >> > seems to be based on a cluster file system, which is what a pNFS server= can also use. >> > >> > I think there would be a lot of work involved in mapping a POSIX file s= ystem onto the >> > Riak database and then exporting that via NFS, etc. It might also be mo= re practical to >> > do this via a userland NFS service than the kernel based one currently i= n FreeBSD. >> > (glusterFS is starting to use the NFS-ganesha server, but I believe it i= s pretty Linux specific, >> > so I doubt it would be useful for Riak running on FreeBSD?) >> > >> > rick >> > >> >> - Jordan >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 From owner-freebsd-fs@freebsd.org Wed Sep 9 02:29:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7587A00AC6 for ; Wed, 9 Sep 2015 02:29:54 +0000 (UTC) (envelope-from vrock28@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 655DB10F4 for ; Wed, 9 Sep 2015 02:29:54 +0000 (UTC) (envelope-from vrock28@gmail.com) Received: by oixx17 with SMTP id x17so70491728oix.0 for ; Tue, 08 Sep 2015 19:29:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=3F16OzMLbkf3G4Mk8oSdCLIXJAS2+YeIQ10GIEh1rSs=; b=e8gOSll3lNkdYKpVwvO4mepU95LvicxgXvZSbcJk73pnS6lTvDKkYF86J7f2x8egrg e6QqQY6gtEFHJ7I0z1TbJXs0yWvpprtgFx4Y+GiIkZX+xaY5f1F1jL+SMvQgheu4+M53 +FL2/FtSoVto1vD5OqvWhD5Gh9nXmBn6S6yVXN/dvYU5gIedJ6AWZ9TbbXJOsBJxZQqt 2Og+Ezdd3aM24REVAZ9Q+LjmNPFvR6D1L4zoN13LUIH42fQfm0QTM3aGHNNkdDkhZ7P+ ldaNgqdBGJHd41Cf/gZ1pWBJEOJi/wfim5qmxJZDBDQyTqjVo+gQQnO2EoOOS827JsNz rxCg== MIME-Version: 1.0 X-Received: by 10.202.214.133 with SMTP id n127mr10991111oig.52.1441765793570; Tue, 08 Sep 2015 19:29:53 -0700 (PDT) Received: by 10.202.197.147 with HTTP; Tue, 8 Sep 2015 19:29:53 -0700 (PDT) Received: by 10.202.197.147 with HTTP; Tue, 8 Sep 2015 19:29:53 -0700 (PDT) In-Reply-To: <8551C101-0E4B-4619-BF52-1B87BC6565D8@longcount.org> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <8551C101-0E4B-4619-BF52-1B87BC6565D8@longcount.org> Date: Wed, 9 Sep 2015 07:59:53 +0530 Message-ID: Subject: Re: CEPH + FreeBSD From: Rakshith Venkatesh To: Mark Saad Cc: Rick Macklem , freebsd-fs@freebsd.org, Jordan Hubbard , Outback Dingo Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 02:29:54 -0000 If there is a need to support cifs along with NFS then locking surely is needed to maintain data sanity and avoid problems due to concurrent access to files. On 09-Sep-2015 7:55 am, "Mark Saad" wrote: > > On Sep 8, 2015, at 9:59 PM, Outback Dingo wrote: > > > > On Wed, Sep 9, 2015 at 11:31 AM, Mark Saad wrote= : > >> All >> What about leofs. It's in ports has and s3 obj store and NFS support >> out of the box >> >> >> http://www.freshports.org/databases/leofs/ >> http://leo-project.net > > > LeoFS supperts NFSv3 and does not have a lock manager.... > > > Well, nlm it's not a hard requirement for most NFS things . NFS is there > for direct access of the data . So in its current form it's not a pnfs > setup , but there is no reason it could not be extended . > > Well it could be worse , manta, joyent's compute and storage platform ha= s > a NFS server written in node.js . Yes nfs in JavaScript , what else would > you expect from them . :) > > > --- > Mark Saad | nonesuch@longcount.org > > > >> >> >> >> --- >> Mark Saad | nonesuch@longcount.org >> >> > On Sep 6, 2015, at 6:18 PM, Rick Macklem wrote: >> > >> > Jordan Hubbard wrote: >> >> >> >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem >> wrote: >> >>> >> >>> Slightly off topic but, btw, there is a port of GLusterFS and those >> folks >> >>> do seem >> >>> interested in seeing it brought "up to speed". I am not sure how >> mature it >> >>> is at >> >>> this point, but it has been known to build on amd64. (I don't have a= n >> amd64 >> >>> machine, >> >>> so I haven't gotten around to building/testing it, but I do plan to >> try and >> >>> use >> >>> it as a basis for a pNFS server, if I can figure out how to get the >> FH info >> >>> out of it. >> >>> I'm working on that;-) >> >> >> >> There are at least two distributed (multi-node) object stores for >> FreeBSD >> >> that I know of. >> >> >> >> One is glusterfs, for which I=E2=80=99m not even really clear on the = status of >> the >> >> ports for. I don=E2=80=99t see any glusterfs port in the master bran= ch of >> >> https://github.com/freebsd/freebsd-ports (or >> >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for tha= t >> >> matter). >> >> >> >> Our FreeNAS ports tree (https://github.com/freenas/ports), in which >> we have a >> >> bit more latitude to add and curate our own ports, has both a >> net/glusterfs >> >> and sysutils/glusterfs, from separate sources (looks like we need to >> clean >> >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the >> >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port list= s >> >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. >> >> >> >> I=E2=80=99m not really sure about the provenance since we were simply >> evaluating >> >> glusterfs for awhile and may have pulled in interim versions from tho= se >> >> sources, but obviously it would be best to have an official maintaine= r >> and >> >> someone in the FreeBSD project actually curating a glusterfs port so >> that >> >> all users of FreeBSD can use it. It would also be fairly key to your >> own >> >> efforts, assuming you decide to pursue glusterfs as a foundation >> technology >> >> for pNFS. >> >> >> >> The other object store, which is pretty mature and is currently >> leading the >> >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. >> There is >> >> a port currently in databases/riak but it=E2=80=99s pretty out of dat= e at >> version >> >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade = of >> >> RiakCS). >> >> >> >> We are very interested in investigating various ways of shimming >> RiakCS to >> >> NFS, using RiakCS a back-end store. Is that something you=E2=80=99d= be >> amenable to >> >> discussing? I=E2=80=99d be happy to send you an amd64 architecture = machine to >> >> develop on. :) >> > Hmm. From a quick look at their web page (I looked once before as >> well), I don't >> > think RiakCS has what I need to do pNFS in a reasonable (for me) amoun= t >> of effort. >> > Two things that glusterFS has that I am hoping to use (and I don't >> think RiakCS has >> > either of these) are: >> > - A Fuse file system interface which allows the kernel nfsd to access >> the store as >> > a file system, so that it can provide the metadata services (NFS >> without the reads/writes). >> > - A userland NFSv3 server in each node which will allow the node to ac= t >> as a data server. >> > >> > If I am wrong and RiakCS does support a VFS file system interface (via >> Fuse or ???), then >> > please correct me. With that, it might be a reasonable alternative. >> > I'll admit I've spent a little time looking at the glusterFS sources >> and haven't yet >> > solved the problem of how to generate the file handles I need, but tha= t >> sounds trivial >> > compared with an entire Fuse and/or VFS file system interface, I think= ? >> > >> > In general, using a cloud object store to implement a pNFS server is a >> *mis*use of >> > the technology, imho. I think it may be possible with glusterFS, since >> that technology >> > seems to be based on a cluster file system, which is what a pNFS serve= r >> can also use. >> > >> > I think there would be a lot of work involved in mapping a POSIX file >> system onto the >> > Riak database and then exporting that via NFS, etc. It might also be >> more practical to >> > do this via a userland NFS service than the kernel based one currently >> in FreeBSD. >> > (glusterFS is starting to use the NFS-ganesha server, but I believe it >> is pretty Linux specific, >> > so I doubt it would be useful for Riak running on FreeBSD?) >> > >> > rick >> > >> >> - Jordan >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > > From owner-freebsd-fs@freebsd.org Wed Sep 9 09:09:52 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9211C9CC284 for ; Wed, 9 Sep 2015 09:09:52 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5483F1483 for ; Wed, 9 Sep 2015 09:09:52 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 1E36215340A; Wed, 9 Sep 2015 11:09:38 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hNQkb5923xMJ; Wed, 9 Sep 2015 11:09:18 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:2502:d7ab:7a1b:8457] (unknown [IPv6:2001:4cb8:3:1:2502:d7ab:7a1b:8457]) by smtp.digiware.nl (Postfix) with ESMTP id 94A54153430; Wed, 9 Sep 2015 11:09:18 +0200 (CEST) Subject: Re: TRIM support (same bug as linux?) To: Alexander Kabaev , Tom Curry References: <55EF7265.2050309@multiplay.co.uk> <1BCC8B97-6C79-4864-8405-BA687577808A@longcount.org> <20150908214221.7a1702e0@kan> Cc: "freebsd-fs@freebsd.org" From: Willem Jan Withagen Organization: Digiware Management b.v. Message-ID: <55EFF739.5020608@digiware.nl> Date: Wed, 9 Sep 2015 11:09:13 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20150908214221.7a1702e0@kan> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 09:09:52 -0000 On 9-9-2015 03:42, Alexander Kabaev wrote: > On Tue, 8 Sep 2015 21:28:01 -0400 > Tom Curry wrote: > >> I'm no expert but if memory serves the issue had to do with concurrent >> and/or queued trim which is not implemented in FreeBSD. >> >> On Tue, Sep 8, 2015 at 9:11 PM, Mark Saad >> wrote: >> >>> >>> >>>> On Sep 8, 2015, at 7:42 PM, Steven Hartland >>>> >>> wrote: >>>> >>>> Nope FreeBSD is not affected by this. >>>> >>>>> On 09/09/2015 00:15, FF wrote: >>>>> I'm asking a pretty vague question and I apologize in advance, >>>>> not >>> trying >>>>> to troll. >>>>> >>>>> The question has to do with whether FreeBSD is using TRIM the >>>>> same way >>> as >>>>> recent Linux kernels. >>>>> >>>>> https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ >>> seems >>>>> to imply that there are instabilities that can occur. Trying to >>>>> avoid duplicating effort if this has already been addressed or >>>>> if its a >>> complete >>>>> dead alley because there isn't a commonality. >>>>> >>>>> Thanks in advance! >>>> >>> >>> It would be interesting if anyone could explain the reasons why >>> ufs/ffs and zfs and FreeBSD are not effected by this issue . My >>> rudimentary understanding of the issue was that it's wasn't just a >>> software glitch in the ext filesystem and bad firmware but a >>> combination of that and Linux poorly supporting a ata modes that >>> are not supported at all on FreeBSD . Is that correct ? >>> >>> >>> --- >>> Mark Saad | nonesuch@longcount.org > > Wasn't the root cause the improper bio splitting code in md/raid0 code > and nothing special in firmware? FreeBSD shares no such code and as > such is not affected. More or less, What happened as a consequence of the split in the code is that there was a possibility that 2 threads would run parallel: one executing a trim and a second one that already executed under the assumption that the trim was completed. And what I believe to be the case is that due to not correct ordering the second thread sometimes could write in space that was going to be trimmed. And thus that data would be lost, causing corruption. Perhaps this typical code is not present in FreeBSD. But it is just a matter of code enginering/refactoring/.... in complex code to make these kind of mistakes happen. Being careful, use may eyes on critial code, tools are all ways to prevent this. But then still: code is written by humans. --WjW From owner-freebsd-fs@freebsd.org Wed Sep 9 09:20:44 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5755F9CCA8B for ; Wed, 9 Sep 2015 09:20:44 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA9371E1F for ; Wed, 9 Sep 2015 09:20:43 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wicgb1 with SMTP id gb1so108279452wic.1 for ; Wed, 09 Sep 2015 02:20:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=3xpcomy5Y2uk9lxX9uDnzH67zgwUyaqmM030NFkKUJY=; b=Gmjo9X9kmIAEMaRCw4BVA9S+0fq+rbeZrl4oElNoogQDiaXujmkK7V+mbj3ErDxGLm 0GmEVQq2pyrHYW93TQolLrfDbypwCn1EGqAjb5tYVE/Tb1Hl8x4AKrRHkIWCicMh/n2M 9TsnJc0IZrOqFJlvteH8XlppuJydYeFAAaVOUpdtrKIjJTyNtRt/JCugkvbTrqdWu1Ua YeSP0sm3F3GSQPwIKVIbO94Jz0geA56ASYOuVrblUUh5Ap0ILXjgZKh/UYryvH3H5Qfe iI+DlXeODLyIaVQ8NoYdGubx8bowQ4yCwB7V7h9z1lGl6/065lz/0YMIXBk8MJHQWB14 7iqA== X-Gm-Message-State: ALoCoQkYnBcW3PK9sULXmffIYzqFfhqtBXSMSBAqBKSfldni0t+EDkrohQZI1PnMrE5u4bRVQH+n X-Received: by 10.194.239.167 with SMTP id vt7mr58889480wjc.5.1441790441790; Wed, 09 Sep 2015 02:20:41 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by smtp.gmail.com with ESMTPSA id bi6sm9185020wjc.25.2015.09.09.02.20.40 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Sep 2015 02:20:40 -0700 (PDT) Subject: Re: TRIM support (same bug as linux?) To: freebsd-fs@freebsd.org References: <55EF7265.2050309@multiplay.co.uk> <1BCC8B97-6C79-4864-8405-BA687577808A@longcount.org> <20150908214221.7a1702e0@kan> <55EFF739.5020608@digiware.nl> From: Steven Hartland Message-ID: <55EFF9DF.2020201@multiplay.co.uk> Date: Wed, 9 Sep 2015 10:20:31 +0100 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55EFF739.5020608@digiware.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 09:20:44 -0000 That was the original theory wasn't that in the end On 09/09/2015 10:09, Willem Jan Withagen wrote: > On 9-9-2015 03:42, Alexander Kabaev wrote: >> On Tue, 8 Sep 2015 21:28:01 -0400 >> Tom Curry wrote: >> >>> I'm no expert but if memory serves the issue had to do with concurrent >>> and/or queued trim which is not implemented in FreeBSD. >>> >>> On Tue, Sep 8, 2015 at 9:11 PM, Mark Saad >>> wrote: >>> >>>> >>>> >>>>> On Sep 8, 2015, at 7:42 PM, Steven Hartland >>>>> >>>> wrote: >>>>> >>>>> Nope FreeBSD is not affected by this. >>>>> >>>>>> On 09/09/2015 00:15, FF wrote: >>>>>> I'm asking a pretty vague question and I apologize in advance, >>>>>> not >>>> trying >>>>>> to troll. >>>>>> >>>>>> The question has to do with whether FreeBSD is using TRIM the >>>>>> same way >>>> as >>>>>> recent Linux kernels. >>>>>> >>>>>> https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ >>>> seems >>>>>> to imply that there are instabilities that can occur. Trying to >>>>>> avoid duplicating effort if this has already been addressed or >>>>>> if its a >>>> complete >>>>>> dead alley because there isn't a commonality. >>>>>> >>>>>> Thanks in advance! >>>>> >>>> >>>> It would be interesting if anyone could explain the reasons why >>>> ufs/ffs and zfs and FreeBSD are not effected by this issue . My >>>> rudimentary understanding of the issue was that it's wasn't just a >>>> software glitch in the ext filesystem and bad firmware but a >>>> combination of that and Linux poorly supporting a ata modes that >>>> are not supported at all on FreeBSD . Is that correct ? >>>> >>>> >>>> --- >>>> Mark Saad | nonesuch@longcount.org >> >> Wasn't the root cause the improper bio splitting code in md/raid0 code >> and nothing special in firmware? FreeBSD shares no such code and as >> such is not affected. > > More or less, > > What happened as a consequence of the split in the code is that there > was a possibility that 2 threads would run parallel: > one executing a trim > and a second one that already executed under the assumption that the > trim was completed. > > And what I believe to be the case is that due to not correct ordering > the second thread sometimes could write in space that was going to be > trimmed. And thus that data would be lost, causing corruption. > > Perhaps this typical code is not present in FreeBSD. But it is just a > matter of code enginering/refactoring/.... in complex code to make > these kind of mistakes happen. > Being careful, use may eyes on critial code, tools are all ways to > prevent this. But then still: code is written by humans. > > --WjW > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 9 10:15:37 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF9889CDB89 for ; Wed, 9 Sep 2015 10:15:37 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from host202-129-static.10-188-b.business.telecomitalia.it (host202-129-static.10-188-b.business.telecomitalia.it [188.10.129.202]) by mx1.freebsd.org (Postfix) with ESMTP id 780571989 for ; Wed, 9 Sep 2015 10:15:37 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from [192.168.0.60] (MAURIZIO-PC [192.168.0.60]) by host202-129-static.10-188-b.business.telecomitalia.it (Postfix) with ESMTP id 9451132373 for ; Wed, 9 Sep 2015 12:10:16 +0200 (CEST) From: Maurizio Vairani Subject: Re: TRIM support (same bug as linux?) To: freebsd-fs@freebsd.org References: Message-ID: <55F00567.90102@cloverinformatica.it> Date: Wed, 9 Sep 2015 12:09:43 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 10:15:37 -0000 Il 09/09/2015 01:15, FF ha scritto: > I'm asking a pretty vague question and I apologize in advance, not trying > to troll. > > The question has to do with whether FreeBSD is using TRIM the same way as > recent Linux kernels. > > https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ seems > to imply that there are instabilities that can occur. Trying to avoid > duplicating effort if this has already been addressed or if its a complete > dead alley because there isn't a commonality. > > Thanks in advance! > > I am using TRIM without problem on: - 2 x OCZ-VERTEX4 128GB in mirror on FreeBSD, updated now to 10.2-RELEASE, from Feb. '13 - Samsung SSD 840 EVO 120GB FreeBSD 10.1-RELEASE from Oct '14 - Trascend SSD370 64GB on FreeBSD 10.2-STABLE from Apr. '15 Maurizio From owner-freebsd-fs@freebsd.org Wed Sep 9 10:41:12 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A685BA00C65 for ; Wed, 9 Sep 2015 10:41:12 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6F25B1914 for ; Wed, 9 Sep 2015 10:41:12 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from crest.local (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id 506966D7A for ; Wed, 9 Sep 2015 12:41:09 +0200 (CEST) Subject: Re: TRIM support (same bug as linux?) To: freebsd-fs@freebsd.org References: From: Jan Bramkamp Message-ID: <55F00CC4.7060600@rlwinm.de> Date: Wed, 9 Sep 2015 12:41:08 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 10:41:12 -0000 On 09/09/15 01:15, FF wrote: > I'm asking a pretty vague question and I apologize in advance, not trying > to troll. > > The question has to do with whether FreeBSD is using TRIM the same way as > recent Linux kernels. > > https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ seems > to imply that there are instabilities that can occur. Trying to avoid > duplicating effort if this has already been addressed or if its a complete > dead alley because there isn't a commonality. At first the leading theory was that Linux triggered a firmware bug in the SSDs or that a bug in the handling of queued TRIM operations is responsible. The real culprit was a bug in the code for linear mdraid volumes causing the wrong blocks to be unmapped with TRIM. From owner-freebsd-fs@freebsd.org Wed Sep 9 12:20:33 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6215A00A3F for ; Wed, 9 Sep 2015 12:20:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5B0B914A0 for ; Wed, 9 Sep 2015 12:20:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:g0TAFhSIOgb5dJsvAj5nA5vMr9psv+yvbD5Q0YIujvd0So/mwa65Zx2N2/xhgRfzUJnB7Loc0qyN4/ymADBLvcbJmUtBWaIPfidNsd8RkQ0kDZzNImzAB9muURYHGt9fXkRu5XCxPBsdMs//Y1rPvi/6tmZKSV3BPAZ4bt74BpTVx5zukbvip9uKP04U1HKUWvBbElaflU3prM4YgI9veO4a6yDihT92QdlQ3n5iPlmJnhzxtY+a9Z9n9DlM6bp6r5YTGY2zRakzTKRZATI6KCh1oZSz7ViQBTaJ/WYWB2UKjgJTUU+C6BDhQoy3vDH3u+Bm1G+dJ8KxSLk1XTGr6eBvSQT0iSEJMHk36mzagNd8yaxA8y6m8jti34Tda4LdGPt4caSVKdQHWWBIVcVdVipOBauzaoIOC6wKOuMO/KfnoF5blxq1BkGJDejszjJNzivs2KQx0OAsFCnb2wM9EtYWsDLfpYOmZ+8pTempwfyQnn34ZPRM1GK4sdCQfw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DABADMIvBV/61jaINdg0YxaQaDHroKgW0KhS9KAoFwEgEBAQEBAQEBgQmCHYIHAQEEAQEBICsgCxACAQgSBgICDRkCAicBCRgOAgQIBwQBCBQEiA0NtVOUPQEBAQEGAQEBAQEdgSKFUYN2gQWEOgEBBRcBMweCLgwvEoExBZUYPoUKhQ6ELoQzgx+NboNqAiaEHCIzB4cDOoEFAQEB X-IronPort-AV: E=Sophos;i="5.17,496,1437451200"; d="scan'208";a="237418319" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 09 Sep 2015 08:20:25 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C40D915F565; Wed, 9 Sep 2015 08:20:25 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 7zhqnZDvfr3j; Wed, 9 Sep 2015 08:20:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7840715F56D; Wed, 9 Sep 2015 08:20:24 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Re828nJ6cNry; Wed, 9 Sep 2015 08:20:24 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 55D6D15F565; Wed, 9 Sep 2015 08:20:24 -0400 (EDT) Date: Wed, 9 Sep 2015 08:20:24 -0400 (EDT) From: Rick Macklem To: Outback Dingo Cc: Mark Saad , freebsd-fs@freebsd.org, Rakshith Venkatesh , Jordan Hubbard Message-ID: <488408636.4345946.1441801224232.JavaMail.zimbra@uoguelph.ca> In-Reply-To: References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> Subject: Re: CEPH + FreeBSD MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: CEPH + FreeBSD Thread-Index: n1CLYu3Luc+d+Rgy3/q9eSnSA2e9Dg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 12:20:33 -0000 Outback Dingo wrote: > On Wed, Sep 9, 2015 at 11:31 AM, Mark Saad wrote= : >=20 > > All > > What about leofs. It's in ports has and s3 obj store and NFS support = out > > of the box > > > > > > http://www.freshports.org/databases/leofs/ > > http://leo-project.net >=20 >=20 > LeoFS supperts NFSv3 and does not have a lock manager.... >=20 I doubt lack of a lock manager is an issue for what I want to do, since the= NFSv4.1 metadata server (just a regular NFSv4.1 server that can give out layouts fo= r reading/writing the data directly on the data servers) handles the locking. It is actually = much easier to keep track of the locking in the NFSv4.1 server and not have to worry about lock= ing on the underlying cluster FS. All I intend to do with a NFSv3 server on the data s= erver(s) is do Read/Write RPCs. Everything else is handled via the NFSv4.1 metadata server= . (The original RFC required use of NFSv4.1 read/write ops on the data server= s, but a new layout type called flex files supports NFSv3 Read/Write for the = data servers.) The key issue for me is whether or not it has a VFS interface to a POSIX li= ke file system (via FUSE or ???). At a quick glance at the web page, I don't s= ee any mention of this? Why? Well, simply the fact that I am looking at extending the current kerne= l based NFSv4.1 server to support pNFS. Obviously, there are other ways a NFSv= 4.1/pNFS server can be built (userland NFS-Ganehsa that is on Linux, for exampl= e), but that isn't what I'm interested in doing. Btw, I took a quick look at MooseFS and it does seem to have this and could= be an alternative to glusterFS. It isn't an object store and only appears to= have a single metadata server, which might be a limitation for the long term? It sounds like MooseFS uses custom prototcol for the chunk/data servers and I don't feel like trying to define yet another layout type= , so I think I would need to add a partial NFSv3 server to the chunk/data ser= vers. I will be looking more closely at both glusterFS and MooseFS soon. If there are yet more of these cluster object stores that you think might b= e worth considering, feel free to mention them. (I thought I had looked at most of = them, but hadn't noticed MooseFS, so...) Thanks for all the comments, rick >=20 >=20 > > > > > > > > --- > > Mark Saad | nonesuch@longcount.org > > > > > On Sep 6, 2015, at 6:18 PM, Rick Macklem wrote= : > > > > > > Jordan Hubbard wrote: > > >> > > >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem wro= te: > > >>> > > >>> Slightly off topic but, btw, there is a port of GLusterFS and those > > folks > > >>> do seem > > >>> interested in seeing it brought "up to speed". I am not sure how > > mature it > > >>> is at > > >>> this point, but it has been known to build on amd64. (I don't have = an > > amd64 > > >>> machine, > > >>> so I haven't gotten around to building/testing it, but I do plan to > > try and > > >>> use > > >>> it as a basis for a pNFS server, if I can figure out how to get the= FH > > info > > >>> out of it. > > >>> I'm working on that;-) > > >> > > >> There are at least two distributed (multi-node) object stores for > > FreeBSD > > >> that I know of. > > >> > > >> One is glusterfs, for which I=E2=80=99m not even really clear on the= status of > > the > > >> ports for. I don=E2=80=99t see any glusterfs port in the master bra= nch of > > >> https://github.com/freebsd/freebsd-ports (or > > >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for th= at > > >> matter). > > >> > > >> Our FreeNAS ports tree (https://github.com/freenas/ports), in which = we > > have a > > >> bit more latitude to add and curate our own ports, has both a > > net/glusterfs > > >> and sysutils/glusterfs, from separate sources (looks like we need to > > clean > > >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the > > >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port lis= ts > > >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. > > >> > > >> I=E2=80=99m not really sure about the provenance since we were simpl= y evaluating > > >> glusterfs for awhile and may have pulled in interim versions from th= ose > > >> sources, but obviously it would be best to have an official maintain= er > > and > > >> someone in the FreeBSD project actually curating a glusterfs port so > > that > > >> all users of FreeBSD can use it. It would also be fairly key to you= r > > own > > >> efforts, assuming you decide to pursue glusterfs as a foundation > > technology > > >> for pNFS. > > >> > > >> The other object store, which is pretty mature and is currently lead= ing > > the > > >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. > > There is > > >> a port currently in databases/riak but it=E2=80=99s pretty out of da= te at > > version > > >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade= of > > >> RiakCS). > > >> > > >> We are very interested in investigating various ways of shimming Ria= kCS > > to > > >> NFS, using RiakCS a back-end store. Is that something you=E2=80=99= d be > > amenable to > > >> discussing? I=E2=80=99d be happy to send you an amd64 architecture= machine to > > >> develop on. :) > > > Hmm. From a quick look at their web page (I looked once before as wel= l), > > I don't > > > think RiakCS has what I need to do pNFS in a reasonable (for me) amou= nt > > of effort. > > > Two things that glusterFS has that I am hoping to use (and I don't th= ink > > RiakCS has > > > either of these) are: > > > - A Fuse file system interface which allows the kernel nfsd to access > > the store as > > > a file system, so that it can provide the metadata services (NFS > > without the reads/writes). > > > - A userland NFSv3 server in each node which will allow the node to a= ct > > as a data server. > > > > > > If I am wrong and RiakCS does support a VFS file system interface (vi= a > > Fuse or ???), then > > > please correct me. With that, it might be a reasonable alternative. > > > I'll admit I've spent a little time looking at the glusterFS sources = and > > haven't yet > > > solved the problem of how to generate the file handles I need, but th= at > > sounds trivial > > > compared with an entire Fuse and/or VFS file system interface, I thin= k? > > > > > > In general, using a cloud object store to implement a pNFS server is = a > > *mis*use of > > > the technology, imho. I think it may be possible with glusterFS, sinc= e > > that technology > > > seems to be based on a cluster file system, which is what a pNFS serv= er > > can also use. > > > > > > I think there would be a lot of work involved in mapping a POSIX file > > system onto the > > > Riak database and then exporting that via NFS, etc. It might also be > > more practical to > > > do this via a userland NFS service than the kernel based one currentl= y > > in FreeBSD. > > > (glusterFS is starting to use the NFS-ganesha server, but I believe i= t > > is pretty Linux specific, > > > so I doubt it would be useful for Riak running on FreeBSD?) > > > > > > rick > > > > > >> - Jordan > > > _______________________________________________ > > > freebsd-fs@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > >=20 From owner-freebsd-fs@freebsd.org Wed Sep 9 12:28:44 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54E87A00E9E for ; Wed, 9 Sep 2015 12:28:44 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id D1DA11996 for ; Wed, 9 Sep 2015 12:28:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:j0JndBICXHC9zZlXUdmcpTZWNBhigK39O0sv0rFitYgVKfXxwZ3uMQTl6Ol3ixeRBMOAu64C0rad7/CocFdDyKjCmUhKSIZLWR4BhJdetC0bK+nBN3fGKuX3ZTcxBsVIWQwt1Xi6NU9IBJS2PAWK8TWM5DIfUi/yKRBybrysXNWC04Lui6vuq9X6WEZhunmUWftKNhK4rAHc5IE9oLBJDeIP8CbPuWZCYO9MxGlldhq5lhf44dqsrtY4q3wD89pozcNLUL37cqIkVvQYSW1+ayFmrPDtrgTJGAuT+mMHACJRlhtTHxOD4gv3U53qvm39rOU63SCbOcj/S/cwWC++7qFlT1jmkioKPSU1tW/M2fB32ZhSowmhpgB/i7DZZoKcKPdlfuuJY8kdTmkbDu5eUiVABsW3aI5ZXMQbOuMNlYj2pBMrpBC9AQSpTLf1zzZDhXv72IUn1Os8HAXe3EorFoRd4zzvsNzpOfJKAqiOx67SwGCGNqsO1A== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DBBAD3JPBV/61jaINdg3dpBoMeu3cKhS9KAoFyEQEBAQEBAQEBgQmCHYIGAQEBAwEBAQEgKyALBQsCAQgOBAYCAg0ZAgIhBgEJGAENAgQIBwQBCBQEh3gDCggNtVOPQg2EcAEBAQEGAQEBAQEdgSKFUYN2gQWCT4FrAQEFAhUBMweCLjsSgTEFlRg+hQqFDnWDOYQzjT2DUINqAiaEHCIzB4Z7CBcjgQUBAQE X-IronPort-AV: E=Sophos;i="5.17,496,1437451200"; d="scan'208";a="237419475" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 09 Sep 2015 08:28:42 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 2D2E815F565; Wed, 9 Sep 2015 08:28:42 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id KZ39HIe-bpKM; Wed, 9 Sep 2015 08:28:40 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C726C15F56D; Wed, 9 Sep 2015 08:28:40 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id wIaoUjDU2DRl; Wed, 9 Sep 2015 08:28:40 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A243A15F565; Wed, 9 Sep 2015 08:28:40 -0400 (EDT) Date: Wed, 9 Sep 2015 08:28:40 -0400 (EDT) From: Rick Macklem To: Rakshith Venkatesh Cc: Mark Saad , freebsd-fs@freebsd.org, Jordan Hubbard , Outback Dingo Message-ID: <1276471646.4354662.1441801720640.JavaMail.zimbra@uoguelph.ca> In-Reply-To: References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <8551C101-0E4B-4619-BF52-1B87BC6565D8@longcount.org> Subject: Re: CEPH + FreeBSD MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: CEPH + FreeBSD Thread-Index: QzgDbtFzb3x+SYm6lo9C4wIMZtp4Kg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 12:28:44 -0000 Rakshith Venkatesh wrote: > If there is a need to support cifs along with NFS then locking surely is > needed to maintain data sanity and avoid problems due to concurrent acces= s > to files. For my case, this doesn't affect choice of cluster file system, since the l= ocking is taken care of by the NFSv4.1 server. You are, however, correct that it w= ould be really nice to have the NFSv4.1 server working more closely with Samba (or other o= pen source CIFS implementations) for the Windows op locks, etc. Right now, the NFSv4.1 server can optionally grab POSIX style advisory lock= s on the underlying file system so that Samba (and local syscalls) can see them. It does nothing w.r.t. the Opens (which are basically Windows op locks, or = whatever Samba calls them? I'll admit I've avoided this, since I know nothing about Samba. (I'm only o= ne retired guy doing this in his spare time.) Soneday, it would be nice if someone who knows the Samb= a side could work on this? rick > On 09-Sep-2015 7:55 am, "Mark Saad" wrote: >=20 > > > > On Sep 8, 2015, at 9:59 PM, Outback Dingo wrot= e: > > > > > > > > On Wed, Sep 9, 2015 at 11:31 AM, Mark Saad wro= te: > > > >> All > >> What about leofs. It's in ports has and s3 obj store and NFS support > >> out of the box > >> > >> > >> http://www.freshports.org/databases/leofs/ > >> http://leo-project.net > > > > > > LeoFS supperts NFSv3 and does not have a lock manager.... > > > > > > Well, nlm it's not a hard requirement for most NFS things . NFS is ther= e > > for direct access of the data . So in its current form it's not a pnfs > > setup , but there is no reason it could not be extended . > > > > Well it could be worse , manta, joyent's compute and storage platform = has > > a NFS server written in node.js . Yes nfs in JavaScript , what else wou= ld > > you expect from them . :) > > > > > > --- > > Mark Saad | nonesuch@longcount.org > > > > > > > >> > >> > >> > >> --- > >> Mark Saad | nonesuch@longcount.org > >> > >> > On Sep 6, 2015, at 6:18 PM, Rick Macklem wrot= e: > >> > > >> > Jordan Hubbard wrote: > >> >> > >> >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem > >> wrote: > >> >>> > >> >>> Slightly off topic but, btw, there is a port of GLusterFS and thos= e > >> folks > >> >>> do seem > >> >>> interested in seeing it brought "up to speed". I am not sure how > >> mature it > >> >>> is at > >> >>> this point, but it has been known to build on amd64. (I don't have= an > >> amd64 > >> >>> machine, > >> >>> so I haven't gotten around to building/testing it, but I do plan t= o > >> try and > >> >>> use > >> >>> it as a basis for a pNFS server, if I can figure out how to get th= e > >> FH info > >> >>> out of it. > >> >>> I'm working on that;-) > >> >> > >> >> There are at least two distributed (multi-node) object stores for > >> FreeBSD > >> >> that I know of. > >> >> > >> >> One is glusterfs, for which I=E2=80=99m not even really clear on th= e status of > >> the > >> >> ports for. I don=E2=80=99t see any glusterfs port in the master br= anch of > >> >> https://github.com/freebsd/freebsd-ports (or > >> >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for t= hat > >> >> matter). > >> >> > >> >> Our FreeNAS ports tree (https://github.com/freenas/ports), in which > >> we have a > >> >> bit more latitude to add and curate our own ports, has both a > >> net/glusterfs > >> >> and sysutils/glusterfs, from separate sources (looks like we need t= o > >> clean > >> >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the > >> >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port li= sts > >> >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. > >> >> > >> >> I=E2=80=99m not really sure about the provenance since we were simp= ly > >> evaluating > >> >> glusterfs for awhile and may have pulled in interim versions from t= hose > >> >> sources, but obviously it would be best to have an official maintai= ner > >> and > >> >> someone in the FreeBSD project actually curating a glusterfs port s= o > >> that > >> >> all users of FreeBSD can use it. It would also be fairly key to yo= ur > >> own > >> >> efforts, assuming you decide to pursue glusterfs as a foundation > >> technology > >> >> for pNFS. > >> >> > >> >> The other object store, which is pretty mature and is currently > >> leading the > >> >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. > >> There is > >> >> a port currently in databases/riak but it=E2=80=99s pretty out of d= ate at > >> version > >> >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrad= e of > >> >> RiakCS). > >> >> > >> >> We are very interested in investigating various ways of shimming > >> RiakCS to > >> >> NFS, using RiakCS a back-end store. Is that something you=E2=80= =99d be > >> amenable to > >> >> discussing? I=E2=80=99d be happy to send you an amd64 architectur= e machine to > >> >> develop on. :) > >> > Hmm. From a quick look at their web page (I looked once before as > >> well), I don't > >> > think RiakCS has what I need to do pNFS in a reasonable (for me) amo= unt > >> of effort. > >> > Two things that glusterFS has that I am hoping to use (and I don't > >> think RiakCS has > >> > either of these) are: > >> > - A Fuse file system interface which allows the kernel nfsd to acces= s > >> the store as > >> > a file system, so that it can provide the metadata services (NFS > >> without the reads/writes). > >> > - A userland NFSv3 server in each node which will allow the node to = act > >> as a data server. > >> > > >> > If I am wrong and RiakCS does support a VFS file system interface (v= ia > >> Fuse or ???), then > >> > please correct me. With that, it might be a reasonable alternative. > >> > I'll admit I've spent a little time looking at the glusterFS sources > >> and haven't yet > >> > solved the problem of how to generate the file handles I need, but t= hat > >> sounds trivial > >> > compared with an entire Fuse and/or VFS file system interface, I thi= nk? > >> > > >> > In general, using a cloud object store to implement a pNFS server is= a > >> *mis*use of > >> > the technology, imho. I think it may be possible with glusterFS, sin= ce > >> that technology > >> > seems to be based on a cluster file system, which is what a pNFS ser= ver > >> can also use. > >> > > >> > I think there would be a lot of work involved in mapping a POSIX fil= e > >> system onto the > >> > Riak database and then exporting that via NFS, etc. It might also be > >> more practical to > >> > do this via a userland NFS service than the kernel based one current= ly > >> in FreeBSD. > >> > (glusterFS is starting to use the NFS-ganesha server, but I believe = it > >> is pretty Linux specific, > >> > so I doubt it would be useful for Riak running on FreeBSD?) > >> > > >> > rick > >> > > >> >> - Jordan > >> > _______________________________________________ > >> > freebsd-fs@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org= " > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > >> > > > > >=20 From owner-freebsd-fs@freebsd.org Wed Sep 9 12:48:56 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D49FA017DC for ; Wed, 9 Sep 2015 12:48:56 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 40BD21074 for ; Wed, 9 Sep 2015 12:48:56 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by igbkq10 with SMTP id kq10so98391577igb.0 for ; Wed, 09 Sep 2015 05:48:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EMNbN7EKp2bU/tTZ8MXNszVFAU6CAFqmlbfZhAQ9Wvo=; b=i3um/rzuR+hEAn7wlT9XPjU8iajD3gQONmwoJRMVmnp8hTaW/cKQ8yhekTR2WNOfSe 2Qd2b2X0BOzPxC2fjHSeCdNaN8KEY9EFtwTViW+uqsKdz4x9yc47tVMj1u0utaaq8YdA Q4+ADMjvVDjR6S7K9mfuqaePqxFgy1mUv/T+UUV4iPJyWpqBD/CtPtIlBx8gffxdGCg4 dc2Y3c8qBUzl5qwWhZPa16pgKDz3e+OxUrX7WPDdkvlUE7w+fh1WOrZ8e86ILrkmmErA 7zAG9KmILElyvMba5GPNW+wciPzLpUbaPJOWFplZkMruiFG9HK1jznEaprsjKxVD0lbc v0KA== MIME-Version: 1.0 X-Received: by 10.50.43.170 with SMTP id x10mr50732342igl.12.1441802935350; Wed, 09 Sep 2015 05:48:55 -0700 (PDT) Received: by 10.65.15.33 with HTTP; Wed, 9 Sep 2015 05:48:55 -0700 (PDT) In-Reply-To: <488408636.4345946.1441801224232.JavaMail.zimbra@uoguelph.ca> References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <488408636.4345946.1441801224232.JavaMail.zimbra@uoguelph.ca> Date: Wed, 9 Sep 2015 05:48:55 -0700 Message-ID: Subject: Re: CEPH + FreeBSD From: Mehmet Erol Sanliturk To: Rick Macklem Cc: Outback Dingo , freebsd-fs@freebsd.org, Rakshith Venkatesh , Jordan Hubbard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 12:48:56 -0000 On Wed, Sep 9, 2015 at 5:20 AM, Rick Macklem wrote: > Outback Dingo wrote: > > On Wed, Sep 9, 2015 at 11:31 AM, Mark Saad > wrote: > > > > > All > > > What about leofs. It's in ports has and s3 obj store and NFS suppor= t > out > > > of the box > > > > > > > > > http://www.freshports.org/databases/leofs/ > > > http://leo-project.net > > > > > > LeoFS supperts NFSv3 and does not have a lock manager.... > > > I doubt lack of a lock manager is an issue for what I want to do, since > the NFSv4.1 > metadata server (just a regular NFSv4.1 server that can give out layouts > for reading/writing > the data directly on the data servers) handles the locking. It is actuall= y > much easier to keep > track of the locking in the NFSv4.1 server and not have to worry about > locking on the > underlying cluster FS. All I intend to do with a NFSv3 server on the data > server(s) is do > Read/Write RPCs. Everything else is handled via the NFSv4.1 metadata > server. > (The original RFC required use of NFSv4.1 read/write ops on the data > servers, > but a new layout type called flex files supports NFSv3 Read/Write for th= e > data servers.) > > The key issue for me is whether or not it has a VFS interface to a POSIX > like > file system (via FUSE or ???). At a quick glance at the web page, I don't > see > any mention of this? > Why? Well, simply the fact that I am looking at extending the current > kernel based > NFSv4.1 server to support pNFS. Obviously, there are other ways a > NFSv4.1/pNFS > server can be built (userland NFS-Ganehsa that is on Linux, for > example), but > that isn't what I'm interested in doing. > > Btw, I took a quick look at MooseFS and it does seem to have this and > could be an > alternative to glusterFS. It isn't an object store and only appears > to have a > single metadata server, which might be a limitation for the long ter= m? > It sounds like MooseFS uses custom prototcol for the chunk/data > servers and I don't feel like trying to define yet another layout > type, so I > think I would need to add a partial NFSv3 server to the chunk/data > servers. > > I will be looking more closely at both glusterFS and MooseFS soon. > > If there are yet more of these cluster object stores that you think might > be worth > considering, feel free to mention them. (I thought I had looked at most o= f > them, but > hadn't noticed MooseFS, so...) > > Thanks for all the comments, rick > > http://www.xtreemfs.org/ https://github.com/xtreemfs/xtreemfs ( BSD ) Mehmet Erol Sanliturk > > > > > > > > > > > > > > > --- > > > Mark Saad | nonesuch@longcount.org > > > > > > > On Sep 6, 2015, at 6:18 PM, Rick Macklem > wrote: > > > > > > > > Jordan Hubbard wrote: > > > >> > > > >>> On Sep 3, 2015, at 4:17 AM, Rick Macklem > wrote: > > > >>> > > > >>> Slightly off topic but, btw, there is a port of GLusterFS and tho= se > > > folks > > > >>> do seem > > > >>> interested in seeing it brought "up to speed". I am not sure how > > > mature it > > > >>> is at > > > >>> this point, but it has been known to build on amd64. (I don't hav= e > an > > > amd64 > > > >>> machine, > > > >>> so I haven't gotten around to building/testing it, but I do plan = to > > > try and > > > >>> use > > > >>> it as a basis for a pNFS server, if I can figure out how to get > the FH > > > info > > > >>> out of it. > > > >>> I'm working on that;-) > > > >> > > > >> There are at least two distributed (multi-node) object stores for > > > FreeBSD > > > >> that I know of. > > > >> > > > >> One is glusterfs, for which I=E2=80=99m not even really clear on t= he status > of > > > the > > > >> ports for. I don=E2=80=99t see any glusterfs port in the master b= ranch of > > > >> https://github.com/freebsd/freebsd-ports (or > > > >> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for > that > > > >> matter). > > > >> > > > >> Our FreeNAS ports tree (https://github.com/freenas/ports), in > which we > > > have a > > > >> bit more latitude to add and curate our own ports, has both a > > > net/glusterfs > > > >> and sysutils/glusterfs, from separate sources (looks like we need = to > > > clean > > > >> things up) - net/glusterfs lists craig001@lerwick.hopto.org as the > > > >> MAINTAINER and is at version 3.6.2. The sysutils/glusterfs port > lists > > > >> bapt@FreeBSD.org as the MAINTAINER and is at version 20140811. > > > >> > > > >> I=E2=80=99m not really sure about the provenance since we were sim= ply > evaluating > > > >> glusterfs for awhile and may have pulled in interim versions from > those > > > >> sources, but obviously it would be best to have an official > maintainer > > > and > > > >> someone in the FreeBSD project actually curating a glusterfs port = so > > > that > > > >> all users of FreeBSD can use it. It would also be fairly key to > your > > > own > > > >> efforts, assuming you decide to pursue glusterfs as a foundation > > > technology > > > >> for pNFS. > > > >> > > > >> The other object store, which is pretty mature and is currently > leading > > > the > > > >> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho. > > > There is > > > >> a port currently in databases/riak but it=E2=80=99s pretty out of = date at > > > version > > > >> 1.4.12 (the current version is 2.0.1, with 2.0 being a major > upgrade of > > > >> RiakCS). > > > >> > > > >> We are very interested in investigating various ways of shimming > RiakCS > > > to > > > >> NFS, using RiakCS a back-end store. Is that something you=E2=80= =99d be > > > amenable to > > > >> discussing? I=E2=80=99d be happy to send you an amd64 architectu= re > machine to > > > >> develop on. :) > > > > Hmm. From a quick look at their web page (I looked once before as > well), > > > I don't > > > > think RiakCS has what I need to do pNFS in a reasonable (for me) > amount > > > of effort. > > > > Two things that glusterFS has that I am hoping to use (and I don't > think > > > RiakCS has > > > > either of these) are: > > > > - A Fuse file system interface which allows the kernel nfsd to acce= ss > > > the store as > > > > a file system, so that it can provide the metadata services (NFS > > > without the reads/writes). > > > > - A userland NFSv3 server in each node which will allow the node to > act > > > as a data server. > > > > > > > > If I am wrong and RiakCS does support a VFS file system interface > (via > > > Fuse or ???), then > > > > please correct me. With that, it might be a reasonable alternative. > > > > I'll admit I've spent a little time looking at the glusterFS source= s > and > > > haven't yet > > > > solved the problem of how to generate the file handles I need, but > that > > > sounds trivial > > > > compared with an entire Fuse and/or VFS file system interface, I > think? > > > > > > > > In general, using a cloud object store to implement a pNFS server i= s > a > > > *mis*use of > > > > the technology, imho. I think it may be possible with glusterFS, > since > > > that technology > > > > seems to be based on a cluster file system, which is what a pNFS > server > > > can also use. > > > > > > > > I think there would be a lot of work involved in mapping a POSIX fi= le > > > system onto the > > > > Riak database and then exporting that via NFS, etc. It might also b= e > > > more practical to > > > > do this via a userland NFS service than the kernel based one > currently > > > in FreeBSD. > > > > (glusterFS is starting to use the NFS-ganesha server, but I believe > it > > > is pretty Linux specific, > > > > so I doubt it would be useful for Riak running on FreeBSD?) > > > > > > > > rick > > > > > > > >> - Jordan > > > > _______________________________________________ > > > > freebsd-fs@freebsd.org mailing list > > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.or= g > " > > > _______________________________________________ > > > freebsd-fs@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 9 13:39:21 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 246BAA005BC for ; Wed, 9 Sep 2015 13:39:21 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 023981309 for ; Wed, 9 Sep 2015 13:39:20 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from c-66-41-25-68.hsd1.mn.comcast.net ([66.41.25.68] helo=[192.168.0.107]) by mail.physics.umn.edu with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1ZZfao-000BT0-U7 for freebsd-fs@freebsd.org; Wed, 09 Sep 2015 08:39:19 -0500 Subject: Re: CEPH + FreeBSD To: freebsd-fs@freebsd.org References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <488408636.4345946.1441801224232.JavaMail.zimbra@uoguelph.ca> From: Graham Allan Message-ID: <55F03688.6060407@physics.umn.edu> Date: Wed, 9 Sep 2015 08:39:20 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <488408636.4345946.1441801224232.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 13:39:21 -0000 On 9/9/2015 7:20 AM, Rick Macklem wrote: > > If there are yet more of these cluster object stores that you think might be worth > considering, feel free to mention them. (I thought I had looked at most of them, but > hadn't noticed MooseFS, so...) Not adding anything new, but I found this an interesting paper comparing some of the contenders. https://hal.archives-ouvertes.fr/hal-00789086/document I felt glusterfs seems like it would be the fastest and easiest to get up and running (at least on linux). Graham From owner-freebsd-fs@freebsd.org Wed Sep 9 15:43:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FCA2A01B5D for ; Wed, 9 Sep 2015 15:43:01 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [IPv6:2001:5a8:4:7e72:d250:99ff:fe57:4030]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02AAB1DE8 for ; Wed, 9 Sep 2015 15:43:00 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.14.9) with ESMTP id t89FgrPc012080; Wed, 9 Sep 2015 08:42:54 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201509091542.t89FgrPc012080@chez.mckusick.com> From: Kirk McKusick To: Willem Jan Withagen Subject: Re: TRIM support (same bug as linux?) cc: Alexander Kabaev , Tom Curry , "freebsd-fs@freebsd.org" In-reply-to: <55EFF739.5020608@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <12078.1441813373.1@chez.mckusick.com> Date: Wed, 09 Sep 2015 08:42:53 -0700 X-Spam-Status: No, score=0.1 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 15:43:01 -0000 > To: Alexander Kabaev , Tom Curry > From: Willem Jan Withagen > Subject: Re: TRIM support (same bug as linux?) > Date: Wed, 9 Sep 2015 11:09:13 +0200 > > On 9-9-2015 03:42, Alexander Kabaev wrote: >> Wasn't the root cause the improper bio splitting code in md/raid0 code >> and nothing special in firmware? FreeBSD shares no such code and as >> such is not affected. > > More or less, > > What happened as a consequence of the split in the code is that there > was a possibility that 2 threads would run parallel: > one executing a trim > and a second one that already executed under the assumption that the > trim was completed. > > And what I believe to be the case is that due to not correct ordering > the second thread sometimes could write in space that was going to be > trimmed. And thus that data would be lost, causing corruption. > > Perhaps this typical code is not present in FreeBSD. But it is just > a matter of code enginering/refactoring/.... in complex code to > make these kind of mistakes happen. Being careful, use may eyes > on critial code, tools are all ways to prevent this. But then still: > code is written by humans. > > --WjW I cannot speak to ZFS in FreeBSD, but the TRIM support if UFS/FFS implements TRIM as follows: filesystem frees block TRIM command is dispatched to disk via GEOM callback from GEOM that disk has reported TRIM command complete block is released to filesystem freelist The block cannot be reallocated until it has been placed on the freelist. So, unless the disk reports that it has acted on the TRIM before it has actually done so, it should not be possible to allocate prematurely. Kirk McKusick From owner-freebsd-fs@freebsd.org Wed Sep 9 16:52:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E86BEA0132C for ; Wed, 9 Sep 2015 16:52:19 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from pv33p03im-asmtp001.me.com (pv33p03im-asmtp001.me.com [17.143.180.10]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C0A9713DE for ; Wed, 9 Sep 2015 16:52:19 +0000 (UTC) (envelope-from jordanhubbard@icloud.com) Received: from [10.20.30.60] (75-101-82-48.static.sonic.net [75.101.82.48]) by pv33p03im-asmtp001.me.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Mar 31 2015)) with ESMTPSA id <0NUF00FIX5IMW040@pv33p03im-asmtp001.me.com> for freebsd-fs@freebsd.org; Wed, 09 Sep 2015 16:52:05 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2015-09-09_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=3.13944348295792e-09 compositescore=0.994460221061599 phishscore=0 kscore.is_spamscore=0 rbsscore=0.994460221061599 recipient_to_sender_totalscore=0 spamscore=0 urlsuspectscore=0.994460221061599 adultscore=0 kscore.compositescore=0 circleOfTrustscore=0 suspectscore=0 recipient_domain_to_sender_totalscore=0 bulkscore=0 recipient_domain_to_sender_domain_totalscore=0 recipient_to_sender_domain_totalscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1412110000 definitions=main-1509090240 Content-type: text/plain; charset=utf-8 MIME-version: 1.0 (Mac OS X Mail 9.0 \(3093\)) Subject: Re: CEPH + FreeBSD From: Jordan Hubbard In-reply-to: Date: Wed, 09 Sep 2015 09:51:57 -0700 Cc: Mark Saad , freebsd-fs@freebsd.org Content-transfer-encoding: quoted-printable Message-id: References: <100306673.40344407.1441279047901.JavaMail.zimbra@uoguelph.ca> <1564D4FA-9BE1-4E37-8E91-F14A009D6B62@icloud.com> <838814506.1858817.1441577912291.JavaMail.zimbra@uoguelph.ca> <8551C101-0E4B-4619-BF52-1B87BC6565D8@longcount.org> To: Rakshith Venkatesh X-Mailer: Apple Mail (2.3093) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 16:52:20 -0000 > On Sep 8, 2015, at 7:29 PM, Rakshith Venkatesh = wrote: >=20 > If there is a need to support cifs along with NFS then locking surely = is > needed to maintain data sanity and avoid problems due to concurrent = access > to files. Well, sort of. The file sharing semantics between NFS and CIFS are so = different in not just in how they lock files, but in the Windows vs Unix = permission models, that it=E2=80=99s essentially lunatic to attempt to = share the same ZFS dataset to both NFS and Windows clients. FreeNAS does not even allow you to do that - you pick a dataset for CIFS = sharing and it applies a completely different set of default ACLs and = even goes so far as to disable Unix permission changes (chown/chmod) = completely so that clueless system admins don=E2=80=99t do local Unix = permission operations on the files and end up blowing certainly critical = (to Windows) ACLs off, rendering the share unusable to Windows clients. = It happened so often that we ultimately had to lay down the law and = explicitly disallow that scenario. In the past, Samba3 also used to essentially lie about a lot of the Unix = permissions to Windows clients in order to provide some semblance of = interoperability, but that ended up causing more problems than it solved = (it=E2=80=99s better to be predictable than magical) and in Samba4, = it=E2=80=99s far more inclined to simply tell the truth, which also = tripped up a lot of sysadmins who were used to it lying before. Once = they managed to unscramble their permissions and actually leverage ACLs = to the full extent they were meant to be used, they learned what the = Samba team already knew: NFS ACLs !=3D Windows ACLs. Or, as they said it best in Ghostbusters: Egon Spengler: There's something very important I forgot to tell you. Peter Venkman: What? Spengler: Don't cross the streams. Venkman: Why? Spengler: It would be bad. Venkman: I'm fuzzy on the whole good/bad thing. What do you mean, "bad"? Spengler: Try to imagine all life as you know it stopping = instantaneously and every molecule in your body exploding at the speed = of light. Ray Stantz: Total protonic reversal! Venkman: Right. That's bad. Okay. All right. Important safety tip. = Thanks, Egon. So yeah, the dream of sharing CIFS and NFS from the same dataset is = probably never going to be a reality. I mean, it=E2=80=99s *technically = possible* just like all things are technically possible, but you=E2=80=99d= have to come up with some sort of underlying =E2=80=9CUniversal = permissions and locking model=E2=80=9D that could then be mapped and = translated upwards to both Unix and Windows clients such that each saw = exactly what they wanted and needed to see, and then you=E2=80=99d need = to commit to maintaining that model across multiple versions of Windows = (in which the semantics have changed in subtle ways over time) and Unix. = The Samba folks really tried to fake that up in userland and as near as = I can figure out, they eventually realized it was just a terrible place = to be and stopped trying to force incompatible species to live together, = which is what we also decided. Yeah, it kind of sucks not being able to = share exactly the same files for R/W access to both Windows and Unix = clients simultaneously. Deal with it. Life is cruel. :-) JFYI, - Jordan From owner-freebsd-fs@freebsd.org Wed Sep 9 17:17:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3F6CA01DF3 for ; Wed, 9 Sep 2015 17:17:20 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "wonkity.com", Issuer "wonkity.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A3EF12F7 for ; Wed, 9 Sep 2015 17:17:20 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.15.2/8.15.2) with ESMTPS id t89HHIV5088367 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 9 Sep 2015 11:17:18 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.15.2/8.15.2/Submit) with ESMTP id t89HHIl5088362; Wed, 9 Sep 2015 11:17:18 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Wed, 9 Sep 2015 11:17:18 -0600 (MDT) From: Warren Block To: Jan Bramkamp cc: freebsd-fs@freebsd.org Subject: Re: TRIM support (same bug as linux?) In-Reply-To: <55F00CC4.7060600@rlwinm.de> Message-ID: References: <55F00CC4.7060600@rlwinm.de> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Wed, 09 Sep 2015 11:17:19 -0600 (MDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2015 17:17:20 -0000 On Wed, 9 Sep 2015, Jan Bramkamp wrote: > On 09/09/15 01:15, FF wrote: >> I'm asking a pretty vague question and I apologize in advance, not trying >> to troll. >> >> The question has to do with whether FreeBSD is using TRIM the same way as >> recent Linux kernels. >> >> https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/ seems >> to imply that there are instabilities that can occur. Trying to avoid >> duplicating effort if this has already been addressed or if its a complete >> dead alley because there isn't a commonality. > > At first the leading theory was that Linux triggered a firmware bug in the > SSDs or that a bug in the handling of queued TRIM operations is responsible. > The real culprit was a bug in the code for linear mdraid volumes causing the > wrong blocks to be unmapped with TRIM. They have test code to duplicate the bug here: https://github.com/algolia/trimtester The trim_periodic.sh script uses fstrim, a Linux command that "is used on a mounted filesystem to discard (or "trim") blocks which are not in use by the filesystem." In a batch. No idea how often that is used in real life. From owner-freebsd-fs@freebsd.org Fri Sep 11 02:56:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6395FA02292 for ; Fri, 11 Sep 2015 02:56:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 522521298 for ; Fri, 11 Sep 2015 02:56:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 4B4D31B62 for ; Fri, 11 Sep 2015 02:56:22 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 06EE010B26 for ; Fri, 11 Sep 2015 02:56:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id eVam36w0sDF8 for ; Fri, 11 Sep 2015 02:56:20 +0000 (UTC) To: freebsd-fs@freebsd.org DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com B84A410B1B From: Bryan Drewery Subject: ZFS LOR Organization: FreeBSD Message-ID: <55F242D1.1030408@FreeBSD.org> Date: Thu, 10 Sep 2015 19:56:17 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2015 02:56:22 -0000 On r287335. > lock order reversal: > 1st 0xfffff80866d28068 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1224 > 2nd 0xfffff806a3abca28 zfs_gfs (zfs_gfs) @ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/gfs.c:494 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe35557173c0 > witness_checkorder() at witness_checkorder+0xedc/frame 0xfffffe3555717440 > __lockmgr_args() at __lockmgr_args+0xad0/frame 0xfffffe3555717580 > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe35557175a0 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x101/frame 0xfffffe35557175d0 > _vn_lock() at _vn_lock+0xc2/frame 0xfffffe3555717640 > gfs_file_create() at gfs_file_create+0x73/frame 0xfffffe3555717680 > gfs_dir_create() at gfs_dir_create+0x1d/frame 0xfffffe35557176c0 > zfsctl_mknode_snapdir() at zfsctl_mknode_snapdir+0x47/frame 0xfffffe3555717720 > gfs_dir_lookup() at gfs_dir_lookup+0x191/frame 0xfffffe35557177c0 > gfs_vop_lookup() at gfs_vop_lookup+0x1d/frame 0xfffffe35557177e0 > zfsctl_root_lookup() at zfsctl_root_lookup+0xf5/frame 0xfffffe3555717870 > zfsctl_umount_snapshots() at zfsctl_umount_snapshots+0x80/frame 0xfffffe35557178f0 > zfs_umount() at zfs_umount+0x7b/frame 0xfffffe3555717930 > dounmount() at dounmount+0x533/frame 0xfffffe35557179b0 > sys_unmount() at sys_unmount+0x3c0/frame 0xfffffe3555717ae0 > amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe3555717bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe3555717bf0 > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x80193182a, rsp = 0x7fffffffb518, rbp = 0x7fffffffb570 --- -- Regards, Bryan Drewery From owner-freebsd-fs@freebsd.org Fri Sep 11 09:27:11 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C312A00BD7 for ; Fri, 11 Sep 2015 09:27:11 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (vps.rulingia.com [103.243.244.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps.rulingia.com", Issuer "CAcert Class 3 Root" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 91D1E14E4 for ; Fri, 11 Sep 2015 09:27:10 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (c220-239-242-83.belrs5.nsw.optusnet.com.au [220.239.242.83]) by vps.rulingia.com (8.15.2/8.15.2) with ESMTPS id t8B9HBUK090867 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 11 Sep 2015 19:17:17 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.9/8.14.9) with ESMTP id t8B9H5wH003175 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 11 Sep 2015 19:17:05 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.9/8.14.9/Submit) id t8B9H5Ii003174 for freebsd-fs@freebsd.org; Fri, 11 Sep 2015 19:17:05 +1000 (AEST) (envelope-from peter) Date: Fri, 11 Sep 2015 19:17:05 +1000 From: Peter Jeremy To: freebsd-fs@freebsd.org Subject: Re: Panic in zfs_blkptr_verify() Message-ID: <20150911091705.GA3087@server.rulingia.com> References: <20150829062743.GA2996@server.rulingia.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="UlVJffcvxoiEqYs2" Content-Disposition: inline In-Reply-To: <20150829062743.GA2996@server.rulingia.com> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender succeeded STARTTLS authentication, not delayed by milter-greylist-4.4.3 (vps.rulingia.com [103.243.244.15]); Fri, 11 Sep 2015 19:17:17 +1000 (AEST) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2015 09:27:11 -0000 --UlVJffcvxoiEqYs2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2015-Aug-29 16:27:43 +1000, Peter Jeremy wro= te: >I'm trying to upgrade my main (amd64) server from 10-stable r276177 to >r287251 but the new kernel consistently panics: > >panic: Solaris(panic): blkptr at 0xfffff80015961848 DVA 0 has invalid OFFS= ET 15724224479232 After wasting a lot of time working on the assumption that I had a corrupt blkptr_t, I worked out that problem was actually a corrupt vdev asize. It seems that the vdev and details had been read from /boot/zfs/zpool.cache without any sanity checks and (for unknown reasons) my zpool.cache hadn't been updated for about 18 months - during which time, I'd expanded the pool from 12TB to 16TB. Manually updating my cache with zpool set cachefile=3D/boot/zfs/zpool.cache zroot zpool set cachefile=3D/boot/zfs/zpool.cache tank seems to have solved the problem. (And my zpool.cache is now being automatically updated when I reboot). --=20 Peter Jeremy --UlVJffcvxoiEqYs2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJV8pwRXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0x60P/2Ca3qDyzxe7gVdpPp72CBVT 5LOmI9XS8yNg8qea2FjYTbtVzxS1hSfyUUIVivAC48oA/PGyk5xIf/r9kH3O26lX uWtHC0xjg17ZuyMaWzJxDFqv6lI0cIRVK17Zo+xlbIujrkvmmq2D76PrIA60ZIFR 734uy5vjaBAvovvReZmXezhHNbGMImbENX2gaHeD0Hl51xddbq258WC/ML3OHKn3 zDMzKMOLJcwHe8kNd6rennUDbT7/IBAr/r9UsnDyWNqYig4qBm3mI6pk1Sqdsgcv qdUtsizQd8VsxslhaAJQWtBl/QugCAv01ENPNFiKa/tYH37zgwamnj01GcaHvSHr WrklKKa41CVyX3DB7Fqln+WdlFBV66dStazeFG+hfHHZpIlHqYdn4TwsQl3TZy2y FbVROyDkPqmn91DMhxSOINThsXo2AGoJCM+cKHUixMfb1Kl13fDyqzLG0V0WdoMr vLZxSuCG9aPrIIwwnOXa61lbr4me9W3HV3Hz8NnDjncviGtaFR/JIxgFcYU+CsCd 7P3/92rRtlJoXY+OPm0hC/39rDBhxbny2zgNOZLCOiQe0uaZOl0Zw/+pL66SLhWA lQOxv2v4ME4yXk7j0YxZRdBlGB5oHNINVG80c8lFaY38RHYv4VT+7V6KdWdY3W1d C/0l/FNKxyWwVZlyCa+z =NqYX -----END PGP SIGNATURE----- --UlVJffcvxoiEqYs2-- From owner-freebsd-fs@freebsd.org Fri Sep 11 16:07:49 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F5D1A0205D for ; Fri, 11 Sep 2015 16:07:49 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 00E07108E for ; Fri, 11 Sep 2015 16:07:48 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: by ioii196 with SMTP id i196so103266506ioi.3 for ; Fri, 11 Sep 2015 09:07:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:date:message-id:subject:from:to:content-type; bh=IBkdDRvovcOa4E2PUEKWfsuWAe9WD4HYT2QnLBDtDIM=; b=GpOi1AYDtW41lzwfq/x7/DtjtLfsRn5l4wXQqDLxR6ZI4xlfXiRo2Wwe0QCe/ekJSn ldi0deYfKw640eCsvqsRFwwa7S1TMY/7xfPHvS98XQY9bByDuHdjPzCn3+A5G7LL/GXY U1nJDUvgXElfDvK8o2rk9jKiFqJ6pXg0bLJd4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=IBkdDRvovcOa4E2PUEKWfsuWAe9WD4HYT2QnLBDtDIM=; b=OvTg+zPuuT4a1fZ2himC7hnS/ECKIVovtNfVKkegy8l1LJ0/BwJ8+26b9WcWfSkysg 1REyuUmbbD3Gp08Xct7OZkdIlU+PTr1cPS+lLFxw0JmJPHIYE3985M3Wrk01+BMV1+za 6BlCg7OXE/qzwPL1StvUOh+vH4vr1Dbcz4eK+9Q8XxcFlPxuR11kTPxcK/GYV/0VdA1i ErCh2DZWlKzJ//bOK7W7iX+QUjQDcGTeGSL1zvnfsmjstQa+UFao/V5bRj5MMUKcBE6T IJDz5rsHQQrl5WkjJXpQb954Xe78/icj5lha0SZeLgmIsRAoDNpFWLbc3goW4iNG8pZf t7qg== X-Gm-Message-State: ALoCoQlQGS5y+IsTzH/AjQsYFRzbGDouE96ec5bu2NuS+aVSSsRhf4K3/Pty4TPHmMLO9+UJmah6 MIME-Version: 1.0 X-Received: by 10.107.3.94 with SMTP id 91mr4969645iod.178.1441987667922; Fri, 11 Sep 2015 09:07:47 -0700 (PDT) Received: by 10.36.85.197 with HTTP; Fri, 11 Sep 2015 09:07:47 -0700 (PDT) Date: Fri, 11 Sep 2015 09:07:47 -0700 Message-ID: Subject: zfs_trim_enabled destroys zio_free() performance From: Matthew Ahrens To: freebsd-fs , Alexander Motin Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2015 16:07:49 -0000 I discovered that when destroying a ZFS snapshot, we can end up using several seconds of CPU via this stack trace: kernel`spinlock_exit+0x2d kernel`taskqueue_enqueue+0x12c zfs.ko`zio_issue_async+0x7c zfs.ko`zio_execute+0x162 zfs.ko`dsl_scan_free_block_cb+0x15f zfs.ko`bpobj_iterate_impl+0x25d zfs.ko`bpobj_iterate_impl+0x46e zfs.ko`dsl_scan_sync+0x152 zfs.ko`spa_sync+0x5c1 zfs.ko`txg_sync_thread+0x3a6 kernel`fork_exit+0x9a kernel`0xffffffff80d0acbe 6558 ms This is not good for performance since, in addition to the CPU cost, it doesn't allow the sync thread to do anything else, and this is observable as periods where we don't do any write i/o to disk for several seconds. The problem is that when zfs_trim_enabled is set (which it is by default), zio_free_sync() always sets ZIO_STAGE_ISSUE_ASYNC, causing the free to be dispatched to a taskq. Since each task completes very quickly, there is a large locking and context switching overhead -- we would be better off just processing the free in the caller's context. I'm not sure exactly why we need to go async when trim is enabled, but it seems like at least we should not bother going async if trim is not actually being used (e.g. with an all-spinning-disk pool). It would also be worth investigating not going async even when trim is useful (e.g. on SSD-based pools). Here is the relevant code: zio_free_sync(): if (zfs_trim_enabled) stage |= ZIO_STAGE_ISSUE_ASYNC | ZIO_STAGE_VDEV_IO_START | ZIO_STAGE_VDEV_IO_ASSESS; /* * GANG and DEDUP blocks can induce a read (for the gang block header, * or the DDT), so issue them asynchronously so that this thread is * not tied up. */ else if (BP_IS_GANG(bp) || BP_GET_DEDUP(bp)) stage |= ZIO_STAGE_ISSUE_ASYNC; --matt From owner-freebsd-fs@freebsd.org Fri Sep 11 17:00:44 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D0AFA01AE9 for ; Fri, 11 Sep 2015 17:00:44 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06059147C for ; Fri, 11 Sep 2015 17:00:44 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by wiclk2 with SMTP id lk2so71649796wic.0 for ; Fri, 11 Sep 2015 10:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=9sIP0LUYDboDSbFjXC1NgiRTUBThCd8ktsxVGDlGkxU=; b=YqprS1F/HNedJCI2Y/tDlfXJ64HCeAU83IWCDM9bqK+hsp7dkBz6JeBuAfpZC/it1/ mrxFniqZlDVbAMF9qLQjgLmfxmSEs5iUkUi10oeBkB0LKJbtzbnX1y2z/EUydNb5o8pe hXnSYCPekNw1w4jF1PGsoUcEgY1MIKOTBWpK2iaBvaFeSbG4+XPVIb1pCG45txmYoLCt UwOHUVK1HcTaMvAOYHdbAYrT/UZLQj6eW0ubMSwU6h3PqYfyvi7yZr6/dZjQ3w5vDqsR plm4oSxcIG5KcHEYdxAmz6nC1SzbUojMRkMy4fMs+dcTx74cnIZ9utFBJuANOmyfD+08 HJVQ== X-Received: by 10.194.179.103 with SMTP id df7mr87967167wjc.69.1441990841464; Fri, 11 Sep 2015 10:00:41 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([2a01:d0:c0a9:3:c685:8ff:fe11:1aa2]) by smtp.googlemail.com with ESMTPSA id z2sm72098wij.1.2015.09.11.10.00.40 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 11 Sep 2015 10:00:40 -0700 (PDT) Sender: Alexander Motin Message-ID: <55F308B7.3020302@FreeBSD.org> Date: Fri, 11 Sep 2015 20:00:39 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Matthew Ahrens , freebsd-fs Subject: Re: zfs_trim_enabled destroys zio_free() performance References: In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2015 17:00:44 -0000 Hi. The code in question was added by me at r253992. Commit message tells it was made to decouple locks. I don't remember much more details, but may be it can be redone somehow else. On 11.09.2015 19:07, Matthew Ahrens wrote: > I discovered that when destroying a ZFS snapshot, we can end up using > several seconds of CPU via this stack trace: > > kernel`spinlock_exit+0x2d > kernel`taskqueue_enqueue+0x12c > zfs.ko`zio_issue_async+0x7c > zfs.ko`zio_execute+0x162 > zfs.ko`dsl_scan_free_block_cb+0x15f > zfs.ko`bpobj_iterate_impl+0x25d > zfs.ko`bpobj_iterate_impl+0x46e > zfs.ko`dsl_scan_sync+0x152 > zfs.ko`spa_sync+0x5c1 > zfs.ko`txg_sync_thread+0x3a6 > kernel`fork_exit+0x9a > kernel`0xffffffff80d0acbe > 6558 ms > > This is not good for performance since, in addition to the CPU cost, it > doesn't allow the sync thread to do anything else, and this is > observable as periods where we don't do any write i/o to disk for > several seconds. > > The problem is that when zfs_trim_enabled is set (which it is by > default), zio_free_sync() always sets ZIO_STAGE_ISSUE_ASYNC, causing the > free to be dispatched to a taskq. Since each task completes very > quickly, there is a large locking and context switching overhead -- we > would be better off just processing the free in the caller's context. > > I'm not sure exactly why we need to go async when trim is enabled, but > it seems like at least we should not bother going async if trim is not > actually being used (e.g. with an all-spinning-disk pool). It would > also be worth investigating not going async even when trim is useful > (e.g. on SSD-based pools). > > Here is the relevant code: > > zio_free_sync(): > if (zfs_trim_enabled) > stage |= ZIO_STAGE_ISSUE_ASYNC | ZIO_STAGE_VDEV_IO_START | > ZIO_STAGE_VDEV_IO_ASSESS; > /* > * GANG and DEDUP blocks can induce a read (for the gang block > header, > * or the DDT), so issue them asynchronously so that this thread is > * not tied up. > */ > else if (BP_IS_GANG(bp) || BP_GET_DEDUP(bp)) > stage |= ZIO_STAGE_ISSUE_ASYNC; > > --matt -- Alexander Motin From owner-freebsd-fs@freebsd.org Fri Sep 11 19:04:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAD389CD755 for ; Fri, 11 Sep 2015 19:04:02 +0000 (UTC) (envelope-from sef@kithrup.com) Received: from kithrup.com (Kithrup.COM [64.142.31.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 25D1B1BA0 for ; Fri, 11 Sep 2015 19:04:01 +0000 (UTC) (envelope-from sef@kithrup.com) Received: from kithrup.com (localhost [127.0.0.1]) by kithrup.com (8.14.4/8.14.4) with ESMTP id t8BJ3oOb088069 for ; Fri, 11 Sep 2015 12:03:50 -0700 (PDT) (envelope-from sef@kithrup.com) Received: (from sef@localhost) by kithrup.com (8.14.4/8.14.4/Submit) id t8BJ3nSi088068 for freebsd-fs@freebsd.org; Fri, 11 Sep 2015 12:03:49 -0700 (PDT) (envelope-from sef) Date: Fri, 11 Sep 2015 12:03:49 -0700 (PDT) From: Sean Eric Fagan Message-Id: <201509111903.t8BJ3nSi088068@kithrup.com> To: freebsd-fs@freebsd.org Subject: chmod, freebsd10, zfs, and ACLs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2015 19:04:02 -0000 On FreeBSD 10, with ZFS, doing: touch fred ; setfacl -m user:www:full_set::deny fred ; getfacl fred ; chmod g=u fred ; getfacl fred does not work the way I expect: to be precise, the chmod causes the user:www ACE to go away. This does not happen with posix1e ACLs. I'm not sure if this is a ZFS issue, or a nfs4 acl issue. wibblewobble? Sean. From owner-freebsd-fs@freebsd.org Fri Sep 11 19:12:25 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D30A89CDB44 for ; Fri, 11 Sep 2015 19:12:25 +0000 (UTC) (envelope-from sef@kithrup.com) Received: from kithrup.com (Kithrup.COM [64.142.31.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 963B61FBC for ; Fri, 11 Sep 2015 19:12:25 +0000 (UTC) (envelope-from sef@kithrup.com) Received: from kithrup.com (localhost [127.0.0.1]) by kithrup.com (8.14.4/8.14.4) with ESMTP id t8BJCOWl089026 for ; Fri, 11 Sep 2015 12:12:24 -0700 (PDT) (envelope-from sef@kithrup.com) Received: (from sef@localhost) by kithrup.com (8.14.4/8.14.4/Submit) id t8BJCO6O089025 for freebsd-fs@freebsd.org; Fri, 11 Sep 2015 12:12:24 -0700 (PDT) (envelope-from sef) Date: Fri, 11 Sep 2015 12:12:24 -0700 (PDT) From: Sean Eric Fagan Message-Id: <201509111912.t8BJCO6O089025@kithrup.com> To: freebsd-fs@freebsd.org Subject: Re: chmod, freebsd10, zfs, and ACLs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2015 19:12:25 -0000 Okay, it's the property aclmode. When set to discard, it does weirdness. When set to passthrough, it behaves as I would expect. Sean. From owner-freebsd-fs@freebsd.org Sat Sep 12 11:15:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A322DA0263E for ; Sat, 12 Sep 2015 11:15:54 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qg0-f41.google.com (mail-qg0-f41.google.com [209.85.192.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 659FC1036 for ; Sat, 12 Sep 2015 11:15:53 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by qgt47 with SMTP id 47so82134274qgt.2 for ; Sat, 12 Sep 2015 04:15:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:content-transfer-encoding :mime-version:subject:message-id:date:references:to; bh=xSp3/DkxT5l54HdODgOYABj7VaPmq0kI3dt25DWd3og=; b=mwI3Rz1Ly4hFG0M//PHx+3U8i9lXF2zlkKNow3K2e8FJkr2meMuLKIOAPlP3VPv0GV 6FBOS+mAIe1Bqv6SYDF0ByNz3L/FNfOFAvk/4lPS6ZuNi2nYFDZAJCQgDX6exIm+S/mE bTQCF1EZrtG3qHiGgYeBZtxTov5zEIlcD3LHOh7kSTRS7h8+qKqXhZ/UZgvqdVS/M9jN oQ6ayvBnpdqidqn8rmIhjLfbW2Sl08MjDzkLDnRavyi4x0fCUfhwvlQlpOTA7UTtFZW4 22J4E0J+QSdX5qcjLEgVHiZqVFYTQ2l7ZqHO7U61pQOVpE6woaYLJAoEabSRRvJcCN62 sRgg== X-Gm-Message-State: ALoCoQkku7+RHnFF/iZ7w5vgcdrFOHVCVrMYLFxgYam0lr/E9bHFvqrNlRssalREWnnmPbXndO7u X-Received: by 10.140.92.133 with SMTP id b5mr5493273qge.55.1442056547303; Sat, 12 Sep 2015 04:15:47 -0700 (PDT) Received: from [192.168.1.113] (ool-182c6ffa.dyn.optonline.net. [24.44.111.250]) by smtp.gmail.com with ESMTPSA id r5sm1870338qkr.26.2015.09.12.04.15.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 12 Sep 2015 04:15:46 -0700 (PDT) From: Mark Saad Mime-Version: 1.0 (1.0) Subject: Fwd: L2ARC and pool corruption Message-Id: <76AD038E-4D72-48BB-8A3B-BD9BD76F9350@longcount.org> Date: Sat, 12 Sep 2015 07:15:45 -0400 References: To: freebsd-fs@freebsd.org X-Mailer: iPhone Mail (12H321) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Sep 2015 11:15:54 -0000 --- Mark Saad | nonesuch@longcount.org Begin forwarded message: > From: Ryan Zezeski > Subject: L2ARC and pool corruption >=20 > A bug was discovered in L2ARC that can cause pool metadata corruption and u= ltimately prevents the pool from being imported. This bug was introduced bac= k in Dec 2014. A fix is currently under review. >=20 > I recommend that we disable L2ARC for any SmartOS platform images that inc= lude this commit: >=20 > commit 89c86e32293a30cdd7af530c38b2073fee01411c > Author: Chris Williamson > Date: Mon Dec 29 19:12:23 2014 -0800 >=20 > 5408 managing ZFS cache devices requires lots of RAM > Reviewed by: Christopher Siden > Reviewed by: George Wilson > Reviewed by: Matthew Ahrens > Reviewed by: Don Brady > Reviewed by: Josef 'Jeff' Sipek > Approved by: Garrett D'Amore >=20 > If you are unsure if a given platform includes this commit then send me th= e version string and I'll find out. >=20 > Here is the announcement on the OmniOS discussion list with more informati= on: >=20 > http://permalink.gmane.org/gmane.os.omnios.general/5113 >=20 > And the official bug report with root cause analysis: >=20 > https://www.illumos.org/issues/6214 >=20 > -Z From owner-freebsd-fs@freebsd.org Sat Sep 12 11:19:32 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F474A027F1 for ; Sat, 12 Sep 2015 11:19:32 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qg0-f52.google.com (mail-qg0-f52.google.com [209.85.192.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F35431102 for ; Sat, 12 Sep 2015 11:19:31 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by qgev79 with SMTP id v79so82127098qge.0 for ; Sat, 12 Sep 2015 04:19:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:content-transfer-encoding :mime-version:subject:message-id:date:references:in-reply-to:to; bh=+z4MhNLJeDmrqDdY9VKzyoUfN0VC1x7uvMtiR3rY+gQ=; b=QNTtX+rqSfA/BCODAroJ0m0N1iOyEGNWPdcJUrBfLchDSbQ0mW6L6udJjWx9DCIhK8 x9xavhTEnom3GckG48SsKENXk5w7SI/JfBP8gGnYJKeaighleSFMrFG6sIqh5sUYb21a KkbkVwmhFJFqXC+Bi3rAk+ateb1Vm8lr4fEgR2siNq6CLg/onEaUnH2Kn18VcMIDTxOC 7YS5Vo9uCvWGHEziHyMCFlE3DXFwnn/NqnnB/BTUodKTcBcwye2S9hxFP9Q383TicC/a +kWlribNLWvYqd0uVKPm+or0wHlZFMMyLAMsq1IQhWgiKjl2KMLtaBl4XvTPOHxFrhw0 zkjA== X-Gm-Message-State: ALoCoQlqQCHyyeu2CIJT1L++6L6qzTZOxCGDuZG+6MlgAx5rScmteAUNzczb+RAjRvWKslh81o8/ X-Received: by 10.140.236.194 with SMTP id h185mr5863418qhc.45.1442056765398; Sat, 12 Sep 2015 04:19:25 -0700 (PDT) Received: from [192.168.1.113] (ool-182c6ffa.dyn.optonline.net. [24.44.111.250]) by smtp.gmail.com with ESMTPSA id 103sm1884323qkx.7.2015.09.12.04.19.24 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 12 Sep 2015 04:19:25 -0700 (PDT) From: Mark Saad Mime-Version: 1.0 (1.0) Subject: Re: L2ARC and pool corruption Message-Id: Date: Sat, 12 Sep 2015 07:19:24 -0400 References: <76AD038E-4D72-48BB-8A3B-BD9BD76F9350@longcount.org> In-Reply-To: <76AD038E-4D72-48BB-8A3B-BD9BD76F9350@longcount.org> To: "freebsd-fs@freebsd.org" X-Mailer: iPhone Mail (12H321) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Sep 2015 11:19:32 -0000 All I forgot to include my question . In any case Ryan and I were talking abou= t this issue at work and I was wondering if and how this issue would effect t= he FreeBSD implementation of zfs . I didn't see any commits that appear to a= ddress this . --- Mark Saad | nonesuch@longcount.org > On Sep 12, 2015, at 7:15 AM, Mark Saad wrote: >=20 >=20 >=20 > --- > Mark Saad | nonesuch@longcount.org >=20 > Begin forwarded message: >=20 >> From: Ryan Zezeski >> Subject: L2ARC and pool corruption >>=20 >> A bug was discovered in L2ARC that can cause pool metadata corruption and= ultimately prevents the pool from being imported. This bug was introduced b= ack in Dec 2014. A fix is currently under review. >>=20 >> I recommend that we disable L2ARC for any SmartOS platform images that in= clude this commit: >>=20 >> commit 89c86e32293a30cdd7af530c38b2073fee01411c >> Author: Chris Williamson >> Date: Mon Dec 29 19:12:23 2014 -0800 >>=20 >> 5408 managing ZFS cache devices requires lots of RAM >> Reviewed by: Christopher Siden >> Reviewed by: George Wilson >> Reviewed by: Matthew Ahrens >> Reviewed by: Don Brady >> Reviewed by: Josef 'Jeff' Sipek >> Approved by: Garrett D'Amore >>=20 >> If you are unsure if a given platform includes this commit then send me t= he version string and I'll find out. >>=20 >> Here is the announcement on the OmniOS discussion list with more informat= ion: >>=20 >> http://permalink.gmane.org/gmane.os.omnios.general/5113 >>=20 >> And the official bug report with root cause analysis: >>=20 >> https://www.illumos.org/issues/6214 >>=20 >> -Z From owner-freebsd-fs@freebsd.org Sat Sep 12 15:21:40 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86D91A018F0 for ; Sat, 12 Sep 2015 15:21:40 +0000 (UTC) (envelope-from thomasrcurry@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6347F1FC2 for ; Sat, 12 Sep 2015 15:21:40 +0000 (UTC) (envelope-from thomasrcurry@gmail.com) Received: by igbni9 with SMTP id ni9so57606930igb.0 for ; Sat, 12 Sep 2015 08:21:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=FsZ/jTIAvzA+30XVTvRADAHpDsD2Krfn3EQ73E7DiLM=; b=rveYuADrh153mrXrtx909iCkTxfc7/gV7S9W9yXATNIWLqm7Gh+2RJg58wFRK3l1hF RAMGQ4XKOSidM0zjH+EYJv865EmiobUhwRxa6mSO4BXTAqBT0A9qkhkNnVXYiuJmjfA5 XU+1f2HRwvQ5WKbNLXWPr5VLrFipBh///iWnGXoi7BfEGLlKY+Toml2xp/Qkg56U1Vaw hgJr1P9LBSySKzTDUtZS7ow0ye1nbRF4SCKtAdLcD/UtarMs1pWgMrih5B7XGsVdFwGq EH3dQG9m9YhG1/WfD5uwVrFQaaZ2UyrREdA3uhUHhSnMiYwkiBq216wEDTwreQcY59yw S1og== MIME-Version: 1.0 X-Received: by 10.50.62.46 with SMTP id v14mr5088718igr.79.1442071299707; Sat, 12 Sep 2015 08:21:39 -0700 (PDT) Received: by 10.107.4.72 with HTTP; Sat, 12 Sep 2015 08:21:39 -0700 (PDT) In-Reply-To: References: <76AD038E-4D72-48BB-8A3B-BD9BD76F9350@longcount.org> Date: Sat, 12 Sep 2015 11:21:39 -0400 Message-ID: Subject: Re: L2ARC and pool corruption From: Tom Curry To: Mark Saad Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Sep 2015 15:21:40 -0000 On Sat, Sep 12, 2015 at 7:19 AM, Mark Saad wrote: > All > I forgot to include my question . In any case Ryan and I were talking > about this issue at work and I was wondering if and how this issue would > effect the FreeBSD implementation of zfs . I didn't see any commits that > appear to address this . > > --- > Mark Saad | nonesuch@longcount.org > > https://github.com/freebsd/freebsd/commit/0ab19d08a4167c4e486420d8ea4ec5968cbc4f42 Looks like it was merged into head 5 hours ago.