From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 04:53:36 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C13471065670; Mon, 6 Feb 2012 04:53:36 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from mail.kirov.so-ups.ru (mail.kirov.so-ups.ru [178.74.170.1]) by mx1.freebsd.org (Postfix) with ESMTP id 5FA118FC0C; Mon, 6 Feb 2012 04:53:36 +0000 (UTC) Received: from kas30pipe.localhost (localhost.kirov.so-ups.ru [127.0.0.1]) by mail.kirov.so-ups.ru (Postfix) with SMTP id 7A44FB8024; Mon, 6 Feb 2012 08:38:18 +0400 (MSK) Received: from kirov.so-ups.ru (unknown [172.21.81.1]) by mail.kirov.so-ups.ru (Postfix) with ESMTP id 74AC7B801F; Mon, 6 Feb 2012 08:38:18 +0400 (MSK) Received: by ns.kirov.so-ups.ru (Postfix, from userid 1010) id 70A68B8F56; Mon, 6 Feb 2012 08:38:18 +0400 (MSK) Received: from [127.0.0.1] (elsukov.kirov.oduur.so [10.118.3.52]) by ns.kirov.so-ups.ru (Postfix) with ESMTP id 37A1FB8F3E; Mon, 6 Feb 2012 08:38:18 +0400 (MSK) Message-ID: <4F2F5934.1020207@FreeBSD.org> Date: Mon, 06 Feb 2012 08:38:12 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Yar Tikhiy References: <201202040350.q143oAw4086985@freefall.freebsd.org> In-Reply-To: <201202040350.q143oAw4086985@freefall.freebsd.org> X-Enigmail-Version: 1.3.4 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig4D8074A4DD087803BEA61118" X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0284], KAS30/Release X-SpamTest-Info: Not protected Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: bin/145309: bsdlabel: Editing disk label invalidates the whole device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 04:53:36 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig4D8074A4DD087803BEA61118 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable On 04.02.2012 7:50, Yar Tikhiy wrote: > Hi there, > =20 > Sorry but FreeBSD 9.0-RELEASE still appears to have this issue. When > installed using BSD label partitioning scheme, a modification to > ada0's label seems to nuke the kernel's view of the disk -- I can't > think of a better way to explain it. The disk itself is OK and the > change makes it OK to the disk but the kernel can no more use the root= > partition until rebooted, returning weird errnos such as EIO or EXIO. > No idea here if the bug is limited to BSD label scheme. Hi, Yar When you are in single user mode your root filesystem is mounted read-onl= y. When you run bsdlabel it opens geom provider for writing and this trigger= s spoiling for it. When bsdlabel closes provider GEOM_PART destroys it and creates again. But VFS code seems loses it. --=20 WBR, Andrey V. Elsukov --------------enig4D8074A4DD087803BEA61118 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) iQEcBAEBAgAGBQJPL1k6AAoJEAHF6gQQyKF66lkH/iiXEWBI9bxoWoIJRH0WVD+Y 6ZE1ASIYv0EaEqZEDpOTede5em1JdwpUg/XLYGZcG3qK8PeIA5FG2eFTL5AyDtqK LuinN19Bo6qPxqIUB7LX4EQyQ0Z7crfoWiUPxg2dHIjHgmXQw6GeDeoAIwrEevhV j1wL11YOHpszeWctriF3r1JE0tKLwzmm9nA6kMu/hbbQDQkR7RGpas4VaenrSGIU J2GvY7DqVdnToviSxjppGdTFhHm5LPpB47eHBl/QWCF4W+23naDodYP46ENCcXBC 9E+fGxsQsGw4jbbPWGG0yFcju5POon+VcKanls+Lzp4yF613fv4Sx0tyDwmns+Q= =bOTP -----END PGP SIGNATURE----- --------------enig4D8074A4DD087803BEA61118-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 09:26:25 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60D19106566B for ; Mon, 6 Feb 2012 09:26:25 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id 97FE28FC14 for ; Mon, 6 Feb 2012 09:26:24 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id CA4FE13D69 for ; Mon, 6 Feb 2012 10:26:23 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id n0pasSUl_4yW for ; Mon, 6 Feb 2012 10:26:18 +0100 (CET) Received: from [10.0.3.3] (188-167-66-148.dynamic.chello.sk [188.167.66.148]) by mail.vx.sk (Postfix) with ESMTPSA id 60E0113D5D for ; Mon, 6 Feb 2012 10:26:17 +0100 (CET) Message-ID: <4F2F9CB9.3040000@FreeBSD.org> Date: Mon, 06 Feb 2012 10:26:17 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org X-Enigmail-Version: 1.3.5 Content-Type: multipart/mixed; boundary="------------060709020802060008040208" Cc: Subject: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 09:26:25 -0000 This is a multi-part message in MIME format. --------------060709020802060008040208 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit The devfs(8) command supports configuring specific rulesets for devfs(5) mounts. However, it operates on already mounted devfs filesystems only and it is impossible to configure a specific ruleset on mount-time. The attached patch adds a "ruleset" mount option to devfs mounts. The ruleset is automatically applied upon mount time. If the ruleset doesn't exist, an empty ruleset with the given numer is created and can be modified with devfs(8) later. The patch is also available at: http://people.freebsd.org/~mm/patches/devfs/devfs_mount_ruleset.patch Please review and/or comment my attached patch. -- Martin Matuska FreeBSD committer http://blog.vx.sk --------------060709020802060008040208 Content-Type: text/plain; name="devfs_mount_ruleset.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="devfs_mount_ruleset.patch" SW5kZXg6IHNoYXJlL21hbi9tYW41L2RldmZzLjUKPT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc2hhcmUv bWFuL21hbjUvZGV2ZnMuNQkocmV2aXNpb24gMjMxMDY1KQorKysgc2hhcmUvbWFuL21hbjUv ZGV2ZnMuNQkod29ya2luZyBjb3B5KQpAQCAtOTAsNiArOTAsMjkgQEAgYW5kCiAuUGEgMiAu CiAuWHIgZmRlc2NmcyA1CiBjcmVhdGVzIGZpbGVzIGZvciBhbGwgb3BlbiBkZXNjcmlwdG9y cy4KKy5QcAorVGhlIG9wdGlvbnMgYXJlIGFzIGZvbGxvd3M6CisuQmwgLXRhZyAtd2lkdGgg aW5kZW50CisuSXQgRmwgbyBBciBvcHRpb25zCitVc2UgdGhlIHNwZWNpZmllZCBtb3VudAor LkFyIG9wdGlvbnMgLAorYXMgZGVzY3JpYmVkIGluCisuWHIgbW91bnQgOCAuCitUaGUgZm9s bG93aW5nIGRldmZzIGZpbGUgc3lzdGVtLXNwZWNpZmljIG9wdGlvbnMgYXJlIGF2YWlsYWJs ZToKKy5CbCAtdGFnIC13aWR0aCBpbmRlbnQKKy5JdCBDbSBydWxlc2V0IE5zIE5vID0gTnMg QXIgcnVsZXNldAorU2V0IHJ1bGVzZXQgbnVtYmVyCisuQXIgcnVsZXNldAorYXMgdGhlIGN1 cnJlbnQgcnVsZXNldCBmb3IgdGhlIG1vdW50LXBvaW50IGFuZCBhcHBseSBhbGwgaXRzIHJ1 bGVzLiBJZiB0aGUKK3J1bGVzZXQgbnVtYmVyCisuQXIgcnVsZXNldAorZG9lcyBub3QgZXhp c3QsIGFuIGVtcHR5IHJ1bGVzZXQgd2l0aCB0aGUgbnVtYmVyCisuQXIgcnVsZXNldAoraXMg Y3JlYXRlZC4gU2VlCisuWHIgZGV2ZnMgOAorZm9yIG1vcmUgaW5mb3JtYXRpb24gb24gd29y a2luZyB3aXRoIGRldmZzIHJ1bGVzZXRzLgorLkVsCisuRWwKIC5TaCBGSUxFUwogLkJsIC10 YWcgLXdpZHRoIC9kZXYvWFhYWCAtY29tcGFjdAogLkl0IFBhIC9kZXYKSW5kZXg6IHN5cy9m cy9kZXZmcy9kZXZmc19ydWxlLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lzL2ZzL2RldmZzL2Rl dmZzX3J1bGUuYwkocmV2aXNpb24gMjMxMDY1KQorKysgc3lzL2ZzL2RldmZzL2RldmZzX3J1 bGUuYwkod29ya2luZyBjb3B5KQpAQCAtNzcxLDMgKzc3MSwxNyBAQCBkZXZmc19ydWxlc19j bGVhbnVwKHN0cnVjdCBkZXZmc19tb3VudCAqZG0pCiAJCWRldmZzX3J1bGVzZXRfcmVhcChk cyk7CiAJfQogfQorCisvKgorICogTWFrZSByc251bSB0aGUgYWN0aXZlIHJ1bGVzZXQgZm9y IGRtIChsb2NrZWQpCisgKi8KK3ZvaWQKK2RldmZzX3J1bGVzZXRfc2V0KGRldmZzX3JzbnVt IHJzbnVtLCBzdHJ1Y3QgZGV2ZnNfbW91bnQgKmRtKQoreworCisJc3hfYXNzZXJ0KCZkbS0+ ZG1fbG9jaywgU1hfWExPQ0tFRCk7CisKKwlzeF94bG9jaygmc3hfcnVsZXMpOworCWRldmZz X3J1bGVzZXRfdXNlKHJzbnVtLCBkbSk7CisJc3hfeHVubG9jaygmc3hfcnVsZXMpOworfQpJ bmRleDogc3lzL2ZzL2RldmZzL2RldmZzX3Zmc29wcy5jCj09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHN5 cy9mcy9kZXZmcy9kZXZmc192ZnNvcHMuYwkocmV2aXNpb24gMjMxMDY1KQorKysgc3lzL2Zz L2RldmZzL2RldmZzX3Zmc29wcy5jCSh3b3JraW5nIGNvcHkpCkBAIC02NSw2ICs2NSw3IEBA IGRldmZzX21vdW50KHN0cnVjdCBtb3VudCAqbXApCiAJaW50IGVycm9yOwogCXN0cnVjdCBk ZXZmc19tb3VudCAqZm1wOwogCXN0cnVjdCB2bm9kZSAqcnZwOworCWRldmZzX3JzbnVtIHJz bnVtOwogCiAJaWYgKGRldmZzX3VuciA9PSBOVUxMKQogCQlkZXZmc191bnIgPSBuZXdfdW5y aGRyKDAsIElOVF9NQVgsIE5VTEwpOwpAQCAtNzQsNiArNzUsMTggQEAgZGV2ZnNfbW91bnQo c3RydWN0IG1vdW50ICptcCkKIAlpZiAobXAtPm1udF9mbGFnICYgKE1OVF9VUERBVEUgfCBN TlRfUk9PVEZTKSkKIAkJcmV0dXJuIChFT1BOT1RTVVBQKTsKIAorCXJzbnVtID0gMDsKKwor CWlmIChtcC0+bW50X29wdG5ldyAhPSBOVUxMICYmCisJICAgIHZmc19nZXRvcHQobXAtPm1u dF9vcHRuZXcsICJydWxlc2V0IiwgTlVMTCwgTlVMTCkgPT0gMCkgeworCQlpZiAodmZzX3Nj YW5vcHQobXAtPm1udF9vcHRuZXcsICJydWxlc2V0IiwgIiVkIiwKKwkJICAgICZyc251bSkg IT0gMSkgeworCQkJdmZzX21vdW50X2Vycm9yKG1wLCAiJXMiLAorCQkJICAgICJpbnZhbGlk IHJ1bGVzZXQgc3BlY2lmaWNhdGlvbiIpOworCQkJcmV0dXJuIChFSU5WQUwpOworCQl9CisJ fQorCiAJZm1wID0gbWFsbG9jKHNpemVvZiAqZm1wLCBNX0RFVkZTLCBNX1dBSVRPSyB8IE1f WkVSTyk7CiAJZm1wLT5kbV9pZHggPSBhbGxvY191bnIoZGV2ZnNfdW5yKTsKIAlzeF9pbml0 KCZmbXAtPmRtX2xvY2ssICJkZXZmc21vdW50Iik7CkBAIC0xMDEsNiArMTE0LDEyIEBAIGRl dmZzX21vdW50KHN0cnVjdCBtb3VudCAqbXApCiAJCXJldHVybiAoZXJyb3IpOwogCX0KIAor CWlmIChyc251bSAhPSAwKSB7CisJCXN4X3hsb2NrKCZmbXAtPmRtX2xvY2spOworCQlkZXZm c19ydWxlc2V0X3NldChyc251bSwgZm1wKTsKKwkJc3hfeHVubG9jaygmZm1wLT5kbV9sb2Nr KTsKKwl9CisKIAlWT1BfVU5MT0NLKHJ2cCwgMCk7CiAKIAl2ZnNfbW91bnRlZGZyb20obXAs ICJkZXZmcyIpOwpJbmRleDogc3lzL2ZzL2RldmZzL2RldmZzLmgKPT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQot LS0gc3lzL2ZzL2RldmZzL2RldmZzLmgJKHJldmlzaW9uIDIzMTA2NSkKKysrIHN5cy9mcy9k ZXZmcy9kZXZmcy5oCSh3b3JraW5nIGNvcHkpCkBAIC0xODIsNiArMTgyLDcgQEAgdm9pZAlk ZXZmc19ydWxlc19hcHBseShzdHJ1Y3QgZGV2ZnNfbW91bnQgKiwgc3RydWMKIHZvaWQJZGV2 ZnNfcnVsZXNfY2xlYW51cChzdHJ1Y3QgZGV2ZnNfbW91bnQgKik7CiBpbnQJZGV2ZnNfcnVs ZXNfaW9jdGwoc3RydWN0IGRldmZzX21vdW50ICosIHVfbG9uZywgY2FkZHJfdCwKIAkgICAg c3RydWN0IHRocmVhZCAqKTsKK3ZvaWQJZGV2ZnNfcnVsZXNldF9zZXQoZGV2ZnNfcnNudW0g cnNudW0sIHN0cnVjdCBkZXZmc19tb3VudCAqZG0pOwogaW50CWRldmZzX2FsbG9jdihzdHJ1 Y3QgZGV2ZnNfZGlyZW50ICosIHN0cnVjdCBtb3VudCAqLCBpbnQsCiAJICAgIHN0cnVjdCB2 bm9kZSAqKik7CiBjaGFyCSpkZXZmc19mcXBuKGNoYXIgKiwgc3RydWN0IGRldmZzX21vdW50 ICosIHN0cnVjdCBkZXZmc19kaXJlbnQgKiwK --------------060709020802060008040208-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 09:39:32 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E845106564A for ; Mon, 6 Feb 2012 09:39:32 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id 0CFA98FC0A for ; Mon, 6 Feb 2012 09:39:32 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 65C92202E8 for ; Mon, 6 Feb 2012 10:39:31 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id UqOHz13F2R2M for ; Mon, 6 Feb 2012 10:39:26 +0100 (CET) Received: from [10.0.3.3] (188-167-66-148.dynamic.chello.sk [188.167.66.148]) by mail.vx.sk (Postfix) with ESMTPSA id 3090F202E1 for ; Mon, 6 Feb 2012 10:39:26 +0100 (CET) Message-ID: <4F2F9FCD.40700@FreeBSD.org> Date: Mon, 06 Feb 2012 10:39:25 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org X-Enigmail-Version: 1.3.5 Content-Type: multipart/mixed; boundary="------------070004050401040308090607" Cc: Subject: [CFR][DEVFS] rc.conf option devfs_load_rulesets X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 09:39:32 -0000 This is a multi-part message in MIME format. --------------070004050401040308090607 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit FreeBSD includes four system rulesets in /etc/defaults/devfs.rules and allows users to configure their custom rulesets in /etc/devfs.rules. However, if not using jails or not specifying at least one of the "devfs_system_ruleset" or "devfs_set_rulesets" variables, there is no way to automatically load the rules from these configuration files. The attached patch introduces a "devfs_load_rulesets" yes/no variable, that allows the user to have the devfs rules always loaded on startup or if manually running /etc/rc.d/devfs start. The patch is also available at: http://people.freebsd.org/~mm/patches/devfs/devfs_load_rulesets.patch Please review and/or comment my attached patch. -- Martin Matuska FreeBSD committer http://blog.vx.sk --------------070004050401040308090607 Content-Type: text/plain; name="devfs_load_rulesets.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="devfs_load_rulesets.patch" SW5kZXg6IGV0Yy9kZWZhdWx0cy9yYy5jb25mCj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIGV0Yy9kZWZh dWx0cy9yYy5jb25mCShyZXZpc2lvbiAyMzEwNjUpCisrKyBldGMvZGVmYXVsdHMvcmMuY29u Zgkod29ya2luZyBjb3B5KQpAQCAtNjQ4LDYgKzY0OCw3IEBAIGRldmZzX3J1bGVzZXRzPSIv ZXRjL2RlZmF1bHRzL2RldmZzLnJ1bGVzIC9ldGMvZGV2CiBkZXZmc19zeXN0ZW1fcnVsZXNl dD0iIgkjIFRoZSBuYW1lIChOT1QgbnVtYmVyKSBvZiBhIHJ1bGVzZXQgdG8gYXBwbHkgdG8g L2RldgogZGV2ZnNfc2V0X3J1bGVzZXRzPSIiCSMgQSBsaXN0IG9mIC9tb3VudC9kZXY9cnVs ZXNldF9uYW1lIHNldHRpbmdzIHRvCiAJCQkjIGFwcGx5IChtdXN0IGJlIG1vdW50ZWQgYWxy ZWFkeSwgaS5lLiBmc3RhYig1KSkKK2RldmZzX2xvYWRfcnVsZXNldHM9Ik5PIgkjIExvYWQg ZGV2ZnMgcnVsZXNldHMgb24gc3RhcnR1cAogcGVyZm9ybWFuY2VfY3hfbG93ZXN0PSJISUdI IgkjIE9ubGluZSBDUFUgaWRsZSBzdGF0ZQogcGVyZm9ybWFuY2VfY3B1X2ZyZXE9Ik5PTkUi CSMgT25saW5lIENQVSBmcmVxdWVuY3kKIGVjb25vbXlfY3hfbG93ZXN0PSJISUdIIgkjIE9m ZmxpbmUgQ1BVIGlkbGUgc3RhdGUKSW5kZXg6IGV0Yy9yYy5kL2RldmZzCj09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT0KLS0tIGV0Yy9yYy5kL2RldmZzCShyZXZpc2lvbiAyMzEwNjUpCisrKyBldGMvcmMuZC9k ZXZmcwkod29ya2luZyBjb3B5KQpAQCAtMTYsNyArMTYsOCBAQCBzdG9wX2NtZD0nOicKIAog ZGV2ZnNfc3RhcnQoKQogewotCWlmIFsgLW4gIiRkZXZmc19zeXN0ZW1fcnVsZXNldCIgLW8g LW4gIiRkZXZmc19zZXRfcnVsZXNldHMiIF07IHRoZW4KKwlpZiBbIC1uICIkZGV2ZnNfc3lz dGVtX3J1bGVzZXQiIC1vIC1uICIkZGV2ZnNfc2V0X3J1bGVzZXRzIiBdIHx8IFwKKwkgICAg Y2hlY2t5ZXNubyBkZXZmc19sb2FkX3J1bGVzZXRzOyB0aGVuCiAJCWRldmZzX2luaXRfcnVs ZXNldHMKIAkJaWYgWyAtbiAiJGRldmZzX3N5c3RlbV9ydWxlc2V0IiBdOyB0aGVuCiAJCQlk ZXZmc19zZXRfcnVsZXNldCAkZGV2ZnNfc3lzdGVtX3J1bGVzZXQgL2Rldgo= --------------070004050401040308090607-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 09:51:04 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 118CC106566B for ; Mon, 6 Feb 2012 09:51:04 +0000 (UTC) (envelope-from kevlo@kevlo.org) Received: from ns.kevlo.org (kevlo.org [220.128.136.52]) by mx1.freebsd.org (Postfix) with ESMTP id 823BE8FC0A for ; Mon, 6 Feb 2012 09:51:03 +0000 (UTC) Received: from rtsl.kevlo.org (ns.kevlo.org [220.128.136.52] (may be forged)) by ns.kevlo.org (8.14.5/8.14.5) with ESMTP id q169aSv8010061; Mon, 6 Feb 2012 17:36:29 +0800 (CST) (envelope-from kevlo@kevlo.org) Message-ID: <4F2F9F1E.3070904@kevlo.org> Date: Mon, 06 Feb 2012 17:36:30 +0800 From: Kevin Lo User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120130 Thunderbird/9.0 MIME-Version: 1.0 To: Gleb Kurtsou References: <20111013181602.GA35354@regency.nsu.ru> <20111014085405.GA3711@tops> In-Reply-To: <20111014085405.GA3711@tops> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Alexey Dokuchaev , fs@freebsd.org Subject: Re: Re: Call for msdosfs/ntfs experts (or better, maintainers) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 09:51:04 -0000 On -28163/01/-9 03:59, Gleb Kurtsou wrote: > On (14/10/2011 01:16), Alexey Dokuchaev wrote: >> Hello there, >> >> For quite a while already, our FAT and NTFS support need some love to >> shine on them, I believe. AFAICT, they currently have no maintainers >> and thus are not receiving proper care. >> >> Case 1. FreeBSD still has problems with UTF-8 locale and correct >> handling of e.g. Chinese characters in both filesystems. Patches were >> worked out to address this problem; they are available here: >> >> http://deadshot.googlecode.com/svn/trunk/freebsd-patch/filesystem/ >> >> PR kern/133174 was filed on 29 Mar 2009 with the original patch for >> msdosfs, which I've cleaned up a bit per style(9). No action was taken >> since then. >> >> I've contacted the original author of these patches. He's very >> collaborative and is eager to provide all the guidance required to >> review and include these changes in our code base. Any takers? It's >> a shame for us not to be on par with Apple and even OpenBSD/NetBSD (as >> I've been told, they support UTF-8 out of the box). >> >> Case 2. Apparently, Apple actually released their NTFS implementation >> under BSD license which seems quite worthy to take a look at: >> >> http://opensource.apple.com/source/ntfs/ntfs-78/kext/ > Sounds very interesting. I'd peek it up and do a FreeBSD port, but I > won't be able to start at least in two months. I've just started > reviewing and testing this year FUSE GSoC project. It works quite ok for > me (ntfs, encfs), no more random panics etc. But I was told it somewhat > reduces number of supported file systems. I'm going to publish it on > github soon. > > Question remains if it's worth doing the port from darwin if we have > working fuse ntfs. Well, I think it's worth it. There is a known problem with exporting a fuse mount using NFS, so you can't use fusefs-ntfs to mount ntfs on one machine and then export that using NFS. > How mature darwin implementation is? Is there public > source repository with history? > > attilio@ is working on MPSAFE NTFS: > http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS Kevin From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 11:07:00 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28AE6106567C for ; Mon, 6 Feb 2012 11:07:00 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 15CE08FC27 for ; Mon, 6 Feb 2012 11:07:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q16B70Ou007796 for ; Mon, 6 Feb 2012 11:07:00 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q16B6xom007794 for freebsd-fs@FreeBSD.org; Mon, 6 Feb 2012 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 6 Feb 2012 11:06:59 GMT Message-Id: <201202061106.q16B6xom007794@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 11:07:00 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164462 fs [nfs] NFSv4 mounting fails to mount; asks for stronger o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/162362 fs [snapshots] [panic] ufs with snapshot(s) panics when g o kern/162083 fs [zfs] [panic] zfs unmount -f pool o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161897 fs [zfs] [patch] zfs partition probing causing long delay o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161674 fs [ufs] snapshot on journaled ufs doesn't work o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161511 fs [unionfs] Filesystem deadlocks when using multiple uni o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159971 fs [ffs] [panic] panic with soft updates journaling durin o kern/159930 fs [ufs] [panic] kernel core o kern/159663 fs [socket] [nullfs] sockets don't work though nullfs mou o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158711 fs [ffs] [panic] panic in ffs_blkfree and ffs_valloc o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157722 fs [geli] unable to newfs a geli encrypted partition o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs f kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files f sparc/123566 fs [zfs] zpool import issue: EOVERFLOW o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount f kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 265 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 11:16:46 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E82161065680 for ; Mon, 6 Feb 2012 11:16:46 +0000 (UTC) (envelope-from gkontos.mail@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id B8E4C8FC1C for ; Mon, 6 Feb 2012 11:16:46 +0000 (UTC) Received: by iaeo4 with SMTP id o4so12639792iae.13 for ; Mon, 06 Feb 2012 03:16:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=qQcihKU/odTuSlPUrugHgKsqbNjRaZIEk3diAaF6KsM=; b=YrBRJGl37EIdvMIMmOI7Bx9V2xPN3OVii/R/YV23hHUJ63VUIh0MRi2Uzd7nD+qaYW OeESrJm1+unYchrGVFHQhL66PWpC8Un2h4/OBDfygoX35RhZEjQp6W/Hu1EIbJeXCEjy 6f/Qou/ZqGVItpHK4iP7ATEZlYIhdhE/eLq2w= MIME-Version: 1.0 Received: by 10.50.153.133 with SMTP id vg5mr9609554igb.8.1328525689382; Mon, 06 Feb 2012 02:54:49 -0800 (PST) Received: by 10.231.231.17 with HTTP; Mon, 6 Feb 2012 02:54:49 -0800 (PST) Date: Mon, 6 Feb 2012 12:54:49 +0200 Message-ID: From: George Kontostanos To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: HAST considarations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 11:16:47 -0000 Greetings list, I have been experimenting with HAST on a raidz ZFS pool and I have stumbled upon some issues. So, any suggestions are highly appreciated :) My testing environment is consisted of a Linux KVM machine with an Intel Core2Duo and 4G RAM with 2 SATA drives striped (for performance) since I don't have the metal to experiment. When someone is considering to use a HA solution for storage then I am quite sure that their set up would include redundant switches and NICs. For best performance the HAST synchronization should be independent from the CARP interfaces, possibly consisting with more than 1 NIC with lag. At least that's what I think. The first problem appeared when trying to setup the resources. It seems that the resource name is tight up with the machines hostname. My hast.conf looks like this: resource disk1 { on hast1 { local /dev/da0 remote hast2 } on hast2 { local /dev/da0 remote hast1 } } resource disk2 { on hast1 { local /dev/da1 remote hast2 } on hast2 { local /dev/da1 remote hast1 } } However, the hostnames are storage1 and storage2 with storage being the shared CARP IP. Both hast1 & hast2 resolve in /etc/hosts and are connected to a different private vlan but when trying to create the first resource the system complained that only hast1 is an acceptable resource name. After changing both machines hostnames I was able to create the resources and a zpool mirror consisted upon /dev/hast/disk1 & /dev/hast/disk2 Manual failover also worked by exporting the pool from the master, switching roles and then importing the pool on the slave. Yet, looking at the example scripts I realized that there is a lot of work to be done since automatic failover is based upon the CARP device and is limited to one resource. I replicated the resources at the script level in order to switch roles for all the devices. Still it looks very basic and crapy. Also, I added the NIC witch is being used for HAST replication to the /etc/devd.conf besides CARP I see this being at an early stage but I was wondering if anyone has to share some success (or not) stories. Also, does anyone else share my point regarding separating CARP and HAST replication? If yes I would be very interesting to know how this was done. Regards -- George Kontostanos Aicom telecoms ltd http://www.aisecure.net From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 11:38:08 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 633B41065670 for ; Mon, 6 Feb 2012 11:38:08 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 0189E8FC08 for ; Mon, 6 Feb 2012 11:38:07 +0000 (UTC) Received: from [192.92.129.180] ([192.92.129.180]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q16Bbd0a034756 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 6 Feb 2012 13:37:45 +0200 (EET) (envelope-from daniel@digsys.bg) Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=iso-8859-1 From: Daniel Kalchev In-Reply-To: Date: Mon, 6 Feb 2012 13:37:38 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: George Kontostanos X-Mailer: Apple Mail (2.1257) Cc: freebsd-fs@freebsd.org Subject: Re: HAST considarations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 11:38:08 -0000 My only issue with this setup is that on reboot, CARP becomes mater for = a moment, then hears the other (real) master and switched to backup = mode. This will however trigger the devd scripts. It can be fixed by = more checks in the scripts, but really, CARP should not start as master. Otherwise, I have beaten it a lot and it seems to work ok. Daniel On Feb 6, 2012, at 12:54 PM, George Kontostanos wrote: > Greetings list, >=20 > I have been experimenting with HAST on a raidz ZFS pool and I have > stumbled upon some issues. So, any suggestions are highly appreciated > :) >=20 > My testing environment is consisted of a Linux KVM machine with an > Intel Core2Duo and 4G RAM with 2 SATA drives striped (for performance) > since I don't have the metal to experiment. >=20 > When someone is considering to use a HA solution for storage then I am > quite sure that their set up would include redundant switches and > NICs. For best performance the HAST synchronization should be > independent from the CARP interfaces, possibly consisting with more > than 1 NIC with lag. At least that's what I think. >=20 > The first problem appeared when trying to setup the resources. It > seems that the resource name is tight up with the machines hostname. > My hast.conf looks like this: >=20 > resource disk1 { > on hast1 { > local /dev/da0 > remote hast2 > } > on hast2 { > local /dev/da0 > remote hast1 > } > } >=20 > resource disk2 { > on hast1 { > local /dev/da1 > remote hast2 > } > on hast2 { > local /dev/da1 > remote hast1 > } > } >=20 > However, the hostnames are storage1 and storage2 with storage being > the shared CARP IP. Both hast1 & hast2 resolve in /etc/hosts and are > connected to a different private vlan but when trying to create the > first resource the system complained that only hast1 is an acceptable > resource name. >=20 > After changing both machines hostnames I was able to create the > resources and a zpool mirror consisted upon /dev/hast/disk1 & > /dev/hast/disk2 >=20 > Manual failover also worked by exporting the pool from the master, > switching roles and then importing the pool on the slave. Yet, looking > at the example scripts I realized that there is a lot of work to be > done since automatic failover is based upon the CARP device and is > limited to one resource. I replicated the resources at the script > level in order to switch roles for all the devices. Still it looks > very basic and crapy. > Also, I added the NIC witch is being used for HAST replication to the > /etc/devd.conf besides CARP >=20 > I see this being at an early stage but I was wondering if anyone has > to share some success (or not) stories. Also, does anyone else share > my point regarding separating CARP and HAST replication? If yes I > would be very interesting to know how this was done. >=20 > Regards >=20 > --=20 > George Kontostanos > Aicom telecoms ltd > http://www.aisecure.net > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 11:43:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DF2C106564A for ; Mon, 6 Feb 2012 11:43:18 +0000 (UTC) (envelope-from gkontos.mail@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id DF44B8FC0A for ; Mon, 6 Feb 2012 11:43:17 +0000 (UTC) Received: by iaeo4 with SMTP id o4so12682870iae.13 for ; Mon, 06 Feb 2012 03:43:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=5ULUCJmVm7AyMo8Zm8crB2thlqFCFDNy0Bxd2ZpeIao=; b=w7+IQeFjsLRTyi9266JlozizDqlyIIlXXS2fMFH8LCRnkjDxu6CbETBDMVx6t2Y98Y 9rOMD0/znWMs5s1DPRyyKqX01d7Uv/SIheCux1Anp1bKQjUxiinVo9VVJ4zz3gy5gAr6 qUJTDiYlJjkQnGqsrR4CyOnbB4u3UMsscejjo= MIME-Version: 1.0 Received: by 10.50.153.133 with SMTP id vg5mr9823587igb.8.1328528597189; Mon, 06 Feb 2012 03:43:17 -0800 (PST) Received: by 10.231.231.17 with HTTP; Mon, 6 Feb 2012 03:43:17 -0800 (PST) In-Reply-To: References: Date: Mon, 6 Feb 2012 13:43:17 +0200 Message-ID: From: George Kontostanos To: Daniel Kalchev Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: HAST considarations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 11:43:18 -0000 On Mon, Feb 6, 2012 at 1:37 PM, Daniel Kalchev wrote: > My only issue with this setup is that on reboot, CARP becomes mater for a= moment, then hears the other (real) master and switched to backup mode. Th= is will however trigger the devd scripts. It can be fixed by more checks in= the scripts, but really, CARP should not start as master. > > Otherwise, I have beaten it a lot and it seems to work ok. > > Daniel Do you use the CARP device also for hast synchronization? George Kontostanos Aicom telecoms ltd http://www.aisecure.net From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 11:43:20 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBF3F106566B for ; Mon, 6 Feb 2012 11:43:20 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 49C768FC17 for ; Mon, 6 Feb 2012 11:43:19 +0000 (UTC) Received: by bkbzx1 with SMTP id zx1so6345786bkb.13 for ; Mon, 06 Feb 2012 03:43:18 -0800 (PST) Received: by 10.204.133.212 with SMTP id g20mr8486209bkt.82.1328528597718; Mon, 06 Feb 2012 03:43:17 -0800 (PST) Received: from dfleuriot-at-hi-media.com ([83.167.62.196]) by mx.google.com with ESMTPS id bw9sm44631395bkb.8.2012.02.06.03.43.16 (version=SSLv3 cipher=OTHER); Mon, 06 Feb 2012 03:43:16 -0800 (PST) Message-ID: <4F2FBCD2.6000603@my.gd> Date: Mon, 06 Feb 2012 12:43:14 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: HAST considarations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 11:43:20 -0000 This issue is due to a bug in OpenBSD 3.8's implementation of CARP. It triggers if you have net.inet.carp.preempt=1 on the node. If the sysctl is set, the interface assumes MASTERship immediately upon being brought up, then yields in the presence of a better master. The patch for this issue was commited by Gleb to 8-STABLE a long while ago, as well as 9.0-RELEASE and 8.3-RELEASE when it gets out. In the meantime, if you're not runing 8-STABLE or 9.0-RELEASE, you can still patch manually. Details + patch in my original PR here: http://www.freebsd.org/cgi/query-pr.cgi?pr=161123 Commit 226367 to HEAD: http://freshbsd.org/commit/freebsd/r226367 We've been running with the patched version for over 6 months now and have yet to encounter any problem. On 2/6/12 12:37 PM, Daniel Kalchev wrote: > My only issue with this setup is that on reboot, CARP becomes mater for a moment, then hears the other (real) master and switched to backup mode. This will however trigger the devd scripts. It can be fixed by more checks in the scripts, but really, CARP should not start as master. > > Otherwise, I have beaten it a lot and it seems to work ok. > > Daniel > > On Feb 6, 2012, at 12:54 PM, George Kontostanos wrote: > >> Greetings list, >> >> I have been experimenting with HAST on a raidz ZFS pool and I have >> stumbled upon some issues. So, any suggestions are highly appreciated >> :) >> >> My testing environment is consisted of a Linux KVM machine with an >> Intel Core2Duo and 4G RAM with 2 SATA drives striped (for performance) >> since I don't have the metal to experiment. >> >> When someone is considering to use a HA solution for storage then I am >> quite sure that their set up would include redundant switches and >> NICs. For best performance the HAST synchronization should be >> independent from the CARP interfaces, possibly consisting with more >> than 1 NIC with lag. At least that's what I think. >> >> The first problem appeared when trying to setup the resources. It >> seems that the resource name is tight up with the machines hostname. >> My hast.conf looks like this: >> >> resource disk1 { >> on hast1 { >> local /dev/da0 >> remote hast2 >> } >> on hast2 { >> local /dev/da0 >> remote hast1 >> } >> } >> >> resource disk2 { >> on hast1 { >> local /dev/da1 >> remote hast2 >> } >> on hast2 { >> local /dev/da1 >> remote hast1 >> } >> } >> >> However, the hostnames are storage1 and storage2 with storage being >> the shared CARP IP. Both hast1 & hast2 resolve in /etc/hosts and are >> connected to a different private vlan but when trying to create the >> first resource the system complained that only hast1 is an acceptable >> resource name. >> >> After changing both machines hostnames I was able to create the >> resources and a zpool mirror consisted upon /dev/hast/disk1 & >> /dev/hast/disk2 >> >> Manual failover also worked by exporting the pool from the master, >> switching roles and then importing the pool on the slave. Yet, looking >> at the example scripts I realized that there is a lot of work to be >> done since automatic failover is based upon the CARP device and is >> limited to one resource. I replicated the resources at the script >> level in order to switch roles for all the devices. Still it looks >> very basic and crapy. >> Also, I added the NIC witch is being used for HAST replication to the >> /etc/devd.conf besides CARP >> >> I see this being at an early stage but I was wondering if anyone has >> to share some success (or not) stories. Also, does anyone else share >> my point regarding separating CARP and HAST replication? If yes I >> would be very interesting to know how this was done. >> >> Regards >> >> -- >> George Kontostanos >> Aicom telecoms ltd >> http://www.aisecure.net >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 11:52:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67BA9106564A for ; Mon, 6 Feb 2012 11:52:09 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id E8DBE8FC1A for ; Mon, 6 Feb 2012 11:52:08 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q16Bpw48034817 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 6 Feb 2012 13:52:04 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <4F2FBEDE.5020403@digsys.bg> Date: Mon, 06 Feb 2012 13:51:58 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120201 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F2FBCD2.6000603@my.gd> In-Reply-To: <4F2FBCD2.6000603@my.gd> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: HAST considarations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 11:52:09 -0000 On 06.02.12 13:43, Damien Fleuriot wrote: > This issue is due to a bug in OpenBSD 3.8's implementation of CARP. > > It triggers if you have net.inet.carp.preempt=1 on the node. > > If the sysctl is set, the interface assumes MASTERship immediately upon > being brought up, then yields in the presence of a better master. I know about this patch, but on my systems net.inet.carp.preempt=0 I was running 8-stable, now 9-stable on these servers and observe the same behavior. George Kontostanos: My setup has 1G interfaces for the CARP/Internet and 10G interfaces for the backend/HAST. I am doing hast over the 10G interfaces. For a system with part of 10k SAS drives, and ZFS mirror (each element of the mirror is an HAST provider), running bonnie++ I see about 100MB/sec flow to the secondary HAST and that about saturates the disks as well (50-60MB/sec.. should have been better) I had earlier experiment with 4 drives in each system and that replicated at up to 230 MB/sec. Daniel From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 12:36:20 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0645106566B; Mon, 6 Feb 2012 12:36:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 302448FC0C; Mon, 6 Feb 2012 12:36:19 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q16CaDMm047184 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 6 Feb 2012 14:36:13 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q16CaDDK087578; Mon, 6 Feb 2012 14:36:13 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q16CaD2o087577; Mon, 6 Feb 2012 14:36:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 6 Feb 2012 14:36:13 +0200 From: Konstantin Belousov To: Martin Matuska Message-ID: <20120206123613.GR3283@deviant.kiev.zoral.com.ua> References: <4F2F9CB9.3040000@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2fzWg4CSUQ6kpzr7" Content-Disposition: inline In-Reply-To: <4F2F9CB9.3040000@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 12:36:20 -0000 --2fzWg4CSUQ6kpzr7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Feb 06, 2012 at 10:26:17AM +0100, Martin Matuska wrote: > The devfs(8) command supports configuring specific rulesets for devfs(5) > mounts. > However, it operates on already mounted devfs filesystems only and it is > impossible to configure a specific ruleset on mount-time. >=20 > The attached patch adds a "ruleset" mount option to devfs mounts. > The ruleset is automatically applied upon mount time. If the ruleset > doesn't exist, an empty ruleset with the given numer is created and > can be modified with devfs(8) later. >=20 > The patch is also available at: > http://people.freebsd.org/~mm/patches/devfs/devfs_mount_ruleset.patch >=20 > Please review and/or comment my attached patch. >=20 Did you tested this with witness turned on ? It seems not. --2fzWg4CSUQ6kpzr7 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk8vyT0ACgkQC3+MBN1Mb4gAEQCg7/7qp3JIVjKXYp+iW6s3sznU E/4AoL4WhSc/wIvqgUrgyJtDJeSKmthl =hngp -----END PGP SIGNATURE----- --2fzWg4CSUQ6kpzr7-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 12:37:30 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F993106564A for ; Mon, 6 Feb 2012 12:37:30 +0000 (UTC) (envelope-from gkontos.mail@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4CAAD8FC12 for ; Mon, 6 Feb 2012 12:37:30 +0000 (UTC) Received: by iaeo4 with SMTP id o4so12771215iae.13 for ; Mon, 06 Feb 2012 04:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9Jy8aAr99lnsT+MxIFkN/yw23Bj1g1p3siVH27RMi6A=; b=s0DN4r3KgppUseeSEcvBdr0xLNJOx7oItaC6gOFsdReIlUaPhmxkHRW/aBrPkxZxXg 5nzxLYVXZjP11qbwcutxEwE0QqftORX9z36ICLA6pKBGiDHFWbGNXPsWwI7Frf8Cq2cY xnHhTVYBfE73JxEQv+VR9vQI+KZXNhHgpirE8= MIME-Version: 1.0 Received: by 10.42.180.9 with SMTP id bs9mr16619648icb.0.1328531849444; Mon, 06 Feb 2012 04:37:29 -0800 (PST) Received: by 10.231.231.17 with HTTP; Mon, 6 Feb 2012 04:37:29 -0800 (PST) In-Reply-To: <4F2FBEDE.5020403@digsys.bg> References: <4F2FBCD2.6000603@my.gd> <4F2FBEDE.5020403@digsys.bg> Date: Mon, 6 Feb 2012 14:37:29 +0200 Message-ID: From: George Kontostanos To: Daniel Kalchev Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: HAST considarations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 12:37:30 -0000 On Mon, Feb 6, 2012 at 1:51 PM, Daniel Kalchev wrote: > > > On 06.02.12 13:43, Damien Fleuriot wrote: >> >> This issue is due to a bug in OpenBSD 3.8's implementation of CARP. >> >> It triggers if you have net.inet.carp.preempt=1 on the node. >> >> If the sysctl is set, the interface assumes MASTERship immediately upon >> being brought up, then yields in the presence of a better master. > > > I know about this patch, but on my systems > > net.inet.carp.preempt=0 > > I was running 8-stable, now 9-stable on these servers and observe the same > behavior. > > George Kontostanos: > > My setup has 1G interfaces for the CARP/Internet and 10G interfaces for the > backend/HAST. I am doing hast over the 10G interfaces. For a system with > part of 10k SAS drives, and ZFS mirror (each element of the mirror is an > HAST provider), running bonnie++ I see about 100MB/sec flow to the secondary > HAST and that about saturates the disks as well (50-60MB/sec.. should have > been better) > I had earlier experiment with 4 drives in each system and that replicated at > up to 230 MB/sec. > > Daniel I wonder how that numbers would look on striped raidz1 with 10 disks with separate mirrored cache devices. Another question that comes to my mind right now is if the failover scripts should bring up/down the required services for data sharing. That is given the fact that the pool is not mounted until their role is primary, samba for example would complain about it. Also, how about backup strategies? My feeling is that for true HA the only option would be to have a network backup solution that would take over from the currently active node. Of course this requires the failover scripts to start /stop this service as well. Regards -- George Kontostanos Aicom telecoms ltd http://www.aisecure.net From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 13:00:29 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91FA91065670 for ; Mon, 6 Feb 2012 13:00:29 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id 2A1D08FC13 for ; Mon, 6 Feb 2012 13:00:29 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 56E0F21221 for ; Mon, 6 Feb 2012 14:00:28 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id aLNq6STSg24J for ; Mon, 6 Feb 2012 14:00:23 +0100 (CET) Received: from [10.0.3.3] (188-167-66-148.dynamic.chello.sk [188.167.66.148]) by mail.vx.sk (Postfix) with ESMTPSA id DEC622120B for ; Mon, 6 Feb 2012 14:00:22 +0100 (CET) Message-ID: <4F2FCEE6.1080400@FreeBSD.org> Date: Mon, 06 Feb 2012 14:00:22 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org References: <4F2F9CB9.3040000@FreeBSD.org> <20120206123613.GR3283@deviant.kiev.zoral.com.ua> In-Reply-To: <20120206123613.GR3283@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 13:00:29 -0000 On 6. 2. 2012 13:36, Konstantin Belousov wrote: > On Mon, Feb 06, 2012 at 10:26:17AM +0100, Martin Matuska wrote: >> The devfs(8) command supports configuring specific rulesets for devfs(5) >> mounts. >> However, it operates on already mounted devfs filesystems only and it is >> impossible to configure a specific ruleset on mount-time. >> >> The attached patch adds a "ruleset" mount option to devfs mounts. >> The ruleset is automatically applied upon mount time. If the ruleset >> doesn't exist, an empty ruleset with the given numer is created and >> can be modified with devfs(8) later. >> >> The patch is also available at: >> http://people.freebsd.org/~mm/patches/devfs/devfs_mount_ruleset.patch >> >> Please review and/or comment my attached patch. >> > Did you tested this with witness turned on ? It seems not. Yes, I did, no warnings or error messages on my console. -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 13:04:22 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49797106566C; Mon, 6 Feb 2012 13:04:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id AF55E8FC13; Mon, 6 Feb 2012 13:04:21 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q16D4Eu6049595 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 6 Feb 2012 15:04:14 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q16D4Ee9087784; Mon, 6 Feb 2012 15:04:14 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q16D4ErW087783; Mon, 6 Feb 2012 15:04:14 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 6 Feb 2012 15:04:14 +0200 From: Konstantin Belousov To: Martin Matuska Message-ID: <20120206130414.GS3283@deviant.kiev.zoral.com.ua> References: <4F2F9CB9.3040000@FreeBSD.org> <20120206123613.GR3283@deviant.kiev.zoral.com.ua> <4F2FCEE6.1080400@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8H2h1WtNxqGVRR8A" Content-Disposition: inline In-Reply-To: <4F2FCEE6.1080400@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 13:04:22 -0000 --8H2h1WtNxqGVRR8A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Feb 06, 2012 at 02:00:22PM +0100, Martin Matuska wrote: > On 6. 2. 2012 13:36, Konstantin Belousov wrote: > > On Mon, Feb 06, 2012 at 10:26:17AM +0100, Martin Matuska wrote: > >> The devfs(8) command supports configuring specific rulesets for devfs(= 5) > >> mounts. > >> However, it operates on already mounted devfs filesystems only and it = is > >> impossible to configure a specific ruleset on mount-time. > >> > >> The attached patch adds a "ruleset" mount option to devfs mounts. > >> The ruleset is automatically applied upon mount time. If the ruleset > >> doesn't exist, an empty ruleset with the given numer is created and > >> can be modified with devfs(8) later. > >> > >> The patch is also available at: > >> http://people.freebsd.org/~mm/patches/devfs/devfs_mount_ruleset.patch > >> > >> Please review and/or comment my attached patch. > >> > > Did you tested this with witness turned on ? It seems not. > Yes, I did, no warnings or error messages on my console. Are you sure ? Did you tried to change rulesets after mounting some devfs with ruleset option ? =46rom what I see, sx_rules is after dm_lock, e.g. in devfs_rules_ioctl(), while your patch insists on reversed order. --8H2h1WtNxqGVRR8A Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk8vz80ACgkQC3+MBN1Mb4i+KgCfcgdZL8oph+GjHsjy5qZt2mP6 JUUAoKaS7y2sQZxZIJl5vWI6HSCMohab =bloM -----END PGP SIGNATURE----- --8H2h1WtNxqGVRR8A-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 13:13:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CC40106566C for ; Mon, 6 Feb 2012 13:13:39 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id D06A28FC0A for ; Mon, 6 Feb 2012 13:13:38 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 3D44D2184D; Mon, 6 Feb 2012 14:13:38 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id Whx8CXN260W1; Mon, 6 Feb 2012 14:13:27 +0100 (CET) Received: from [10.0.3.3] (188-167-66-148.dynamic.chello.sk [188.167.66.148]) by mail.vx.sk (Postfix) with ESMTPSA id 9760621836; Mon, 6 Feb 2012 14:13:27 +0100 (CET) Message-ID: <4F2FD1F6.7020903@FreeBSD.org> Date: Mon, 06 Feb 2012 14:13:26 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: Konstantin Belousov References: <4F2F9CB9.3040000@FreeBSD.org> <20120206123613.GR3283@deviant.kiev.zoral.com.ua> <4F2FCEE6.1080400@FreeBSD.org> <20120206130414.GS3283@deviant.kiev.zoral.com.ua> In-Reply-To: <20120206130414.GS3283@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 13:13:39 -0000 On 6. 2. 2012 14:04, Konstantin Belousov wrote: > On Mon, Feb 06, 2012 at 02:00:22PM +0100, Martin Matuska wrote: >> On 6. 2. 2012 13:36, Konstantin Belousov wrote: >>> On Mon, Feb 06, 2012 at 10:26:17AM +0100, Martin Matuska wrote: >>>> The devfs(8) command supports configuring specific rulesets for devfs(5) >>>> mounts. >>>> However, it operates on already mounted devfs filesystems only and it is >>>> impossible to configure a specific ruleset on mount-time. >>>> >>>> The attached patch adds a "ruleset" mount option to devfs mounts. >>>> The ruleset is automatically applied upon mount time. If the ruleset >>>> doesn't exist, an empty ruleset with the given numer is created and >>>> can be modified with devfs(8) later. >>>> >>>> The patch is also available at: >>>> http://people.freebsd.org/~mm/patches/devfs/devfs_mount_ruleset.patch >>>> >>>> Please review and/or comment my attached patch. >>>> >>> Did you tested this with witness turned on ? It seems not. >> Yes, I did, no warnings or error messages on my console. > Are you sure ? Did you tried to change rulesets after mounting > some devfs with ruleset option ? > > From what I see, sx_rules is after dm_lock, e.g. in devfs_rules_ioctl(), > while your patch insists on reversed order. In my patch sx_rules is after dm_lock, too: devfs_vfsops.c: if (rsnum != 0) { sx_xlock(&fmp->dm_lock); devfs_ruleset_set(rsnum, fmp); sx_xunlock(&fmp->dm_lock); } devfs_rule.c: void devfs_ruleset_set(devfs_rsnum rsnum, struct devfs_mount *dm) { sx_assert(&dm->dm_lock, SX_XLOCKED); sx_xlock(&sx_rules); devfs_ruleset_use(rsnum, dm); sx_xunlock(&sx_rules); } -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 16:11:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA5F1106566B for ; Mon, 6 Feb 2012 16:11:31 +0000 (UTC) (envelope-from peter@pean.org) Received: from velox.its.uu.se (velox.its.uu.se [130.238.7.74]) by mx1.freebsd.org (Postfix) with ESMTP id 8AE8C8FC21 for ; Mon, 6 Feb 2012 16:11:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at uu.se Received: from nyx.uppmax.uu.se (nyx.uppmax.uu.se [130.238.137.40]) by velox.its.uu.se (Postfix) with ESMTP id 2A17735434 for ; Mon, 6 Feb 2012 16:52:11 +0100 (CET) Message-ID: <4F2FF72B.6000509@pean.org> Date: Mon, 06 Feb 2012 16:52:11 +0100 From: =?ISO-8859-1?Q?Peter_Ankerst=E5l?= User-Agent: Thunderbird 2.0.0.24 (X11/20111108) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:11:31 -0000 Hi, I want to investigate if it is possible to create your own usable HPC storage using zfs and some network filesystem like nfs. Just a thought experiment.. A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD deives for cache. Preferrably in mirror where applicable. Connected to this machine we will have about 410 3TB drives to give approx 1PB of usable storage in a 8+2 raidz configuration. Connected to this will be a ~800 nodes big HPC cluster that will access the storage in parallell is this even possible or do we need to distribute the meta data load over many servers? If that is the case, does it exist any software for FreeBSD that could accomplish this distribution (pNFS dosent seem to be anywhere close to usable in FreeBSD) or do I need to call NetApp or Panasas right away? It would be really nice if I could build my own storage solution. Other possible solutions to this problem is extremley welcome. Best Regards Peter Ankerstål From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 16:22:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1623106568A for ; Mon, 6 Feb 2012 16:22:07 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.emeryville.ca.mail.comcast.net (qmta09.emeryville.ca.mail.comcast.net [76.96.30.96]) by mx1.freebsd.org (Postfix) with ESMTP id CA0618FC08 for ; Mon, 6 Feb 2012 16:22:07 +0000 (UTC) Received: from omta15.emeryville.ca.mail.comcast.net ([76.96.30.71]) by qmta09.emeryville.ca.mail.comcast.net with comcast id Wf9d1i0061Y3wxoA9gN7Ym; Mon, 06 Feb 2012 16:22:07 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta15.emeryville.ca.mail.comcast.net with comcast id WgN61i00E1t3BNj8bgN6mL; Mon, 06 Feb 2012 16:22:07 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 415CD102C19; Mon, 6 Feb 2012 08:22:06 -0800 (PST) Date: Mon, 6 Feb 2012 08:22:06 -0800 From: Jeremy Chadwick To: Peter Ankerst?l Message-ID: <20120206162206.GA541@icarus.home.lan> References: <4F2FF72B.6000509@pean.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F2FF72B.6000509@pean.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:22:08 -0000 On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: > I want to investigate if it is possible to create your own usable > HPC storage using zfs and some > network filesystem like nfs. > > Just a thought experiment.. > A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) > I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD > deives for cache. > Preferrably in mirror where applicable. > > Connected to this machine we will have about 410 3TB drives to give approx > 1PB of usable storage in a 8+2 raidz configuration. > > Connected to this will be a ~800 nodes big HPC cluster that will > access the storage in parallell > is this even possible or do we need to distribute the meta data load > over many servers? If that is the case, > does it exist any software for FreeBSD that could accomplish this > distribution (pNFS dosent seem to be > anywhere close to usable in FreeBSD) or do I need to call NetApp or > Panasas right away? It would be > really nice if I could build my own storage solution. > > Other possible solutions to this problem is extremley welcome. For starters I'd love to know: - What single motherboard supports up to 192GB of RAM - How you plan on getting roughly 410 hard disks (or 422 assuming an additional 12 SSDs) hooked up to a single machine If you are considering investing the time and especially money (the cost here is almost unfathomable, IMO) into this, I strongly recommend you consider an actual hardware filer (e.g. NetApp). Your performance and reliability will be much greater, plus you will get overall better support from NetApp in the case something goes wrong. In the case you run into problems with FreeBSD (and I can assure you in this kind of setup you will) with this kind of extensive setup, you will be at the mercy of developers' time/schedules with absolutely no guarantee that your problem will be solved. You definitely want a support contract. Thus, go NetApp. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 16:39:05 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D814B106566C for ; Mon, 6 Feb 2012 16:39:05 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from mail.ultra-secure.de (mail.ultra-secure.de [78.47.114.122]) by mx1.freebsd.org (Postfix) with ESMTP id 253198FC13 for ; Mon, 6 Feb 2012 16:39:04 +0000 (UTC) Received: (qmail 90599 invoked by uid 89); 6 Feb 2012 15:39:01 -0000 Received: by simscan 1.4.0 ppid: 90594, pid: 90596, t: 0.1342s scanners: attach: 1.4.0 clamav: 0.97.3/m:54/d:14404 Received: from unknown (HELO suse2.iptech.internal) (rainer@ultra-secure.de@212.71.117.70) by mail.ultra-secure.de with ESMTPA; 6 Feb 2012 15:39:01 -0000 Date: Mon, 6 Feb 2012 17:39:02 +0100 From: Rainer Duffner To: freebsd-fs@freebsd.org Message-ID: <20120206173902.6ad0514c@suse2.iptech.internal> In-Reply-To: <20120206162206.GA541@icarus.home.lan> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.22.1; x86_64-unknown-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:39:05 -0000 Am Mon, 6 Feb 2012 08:22:06 -0800 schrieb Jeremy Chadwick : > On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: > > Other possible solutions to this problem is extremley welcome. >=20 > For starters I'd love to know: Recently, someone had a similar proposal on the CentoS list. Except, he really wants to build it, it's 2 PB and it's supposed to be housed in his garage. Someone explained to him that with the 28k BTU, his garage would get rather warm.... There's been a presentation on the last LISA about "Your First Peta-Byte". It's a interesting talk, and entertaining too. It's on youtube. =46rom building our own filers with Solaris and COTS shelves, I can say that the fun quickly wears of... From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 16:41:54 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88B06106566C for ; Mon, 6 Feb 2012 16:41:54 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 461BC8FC17 for ; Mon, 6 Feb 2012 16:41:53 +0000 (UTC) Received: by vbbfa15 with SMTP id fa15so6169867vbb.13 for ; Mon, 06 Feb 2012 08:41:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=hyjuTtSwtSSy97skF52fZfsrrWmSo+ogLR17qdxn1/o=; b=qhXPXWbwaW3Jwn0h5LzYXPmBD5JWNjr2g/Ud8e3YIEIxJ52C4SAf0nJ/uY3lo7+bVq gUQy8Q3ebD/Ji5/9c5/Tf0lpEFCDOC+lgMmYYh6y1ySKOHiDujlvw60DSr65uXRyflK4 E1Xqu0LewpHDkFY95Qy40V7MInOqArO7D6f0s= MIME-Version: 1.0 Received: by 10.52.23.169 with SMTP id n9mr6029795vdf.15.1328546513497; Mon, 06 Feb 2012 08:41:53 -0800 (PST) Received: by 10.220.181.4 with HTTP; Mon, 6 Feb 2012 08:41:53 -0800 (PST) In-Reply-To: <20120206162206.GA541@icarus.home.lan> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> Date: Mon, 6 Feb 2012 08:41:53 -0800 Message-ID: From: Freddie Cash To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:41:54 -0000 On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick wrote: > On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >> I want to investigate if it is possible to create your own usable >> HPC storage using zfs and some >> network filesystem like nfs. >> >> Just a thought experiment.. >> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) >> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >> deives for =C2=A0cache. >> Preferrably in =C2=A0mirror where applicable. >> >> Connected to this machine we will have about 410 3TB drives to give appr= ox >> 1PB of usable storage in a 8+2 raidz configuration. >> >> Connected to this will be a ~800 nodes big HPC cluster that will >> access the storage in parallell >> is this even possible or do we need to distribute the meta data load >> over many servers? If that is the case, >> does it exist any software for FreeBSD that could =C2=A0accomplish this >> distribution (pNFS =C2=A0dosent seem to be >> anywhere close to usable in FreeBSD) or do I need to call NetApp or >> Panasas right away? It would be >> really nice if I could build my own storage solution. >> >> Other possible solutions to this problem is extremley welcome. > > For starters I'd love to know: > > - What single motherboard supports up to 192GB of RAM SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM slots). It's an AMD board, but there should be variants that support Intel CPUs. It's not uncommon to support 256 GB of RAM these days, although 128 GB boards are much more common. > - How you plan on getting roughly 410 hard disks (or 422 assuming > =C2=A0an additional 12 SSDs) hooked up to a single machine In a "head node" + "JBOD" setup? Where the head node has a mobo that supports multiple PCIe x8 and PCIe x16 slots, and is stuffed full of 16-24 port multi-lane SAS/SATA controllers with external ports that are cabled up to external JBOD boxes. The SSDs would be connected to the mobo SAS/SATA ports. Each JBOD box contains nothing but power, SAS/SATA backplane, and harddrives. Possibly using SAS expanders. We're considering doing the same for our SAN/NAS setup for centralising storage for our VM hosts, although not quite to the same scale as the OP. :) > If you are considering investing the time and especially money (the cost > here is almost unfathomable, IMO) into this, I strongly recommend you > consider an actual hardware filer (e.g. NetApp). =C2=A0Your performance a= nd > reliability will be much greater, plus you will get overall better > support from NetApp in the case something goes wrong. =C2=A0In the case y= ou > run into problems with FreeBSD (and I can assure you in this kind of > setup you will) with this kind of extensive setup, you will be at the > mercy of developers' time/schedules with absolutely no guarantee that > your problem will be solved. =C2=A0You definitely want a support contract= . > Thus, go NetApp. For an HPC setup like the OP wants, where performance and uptime are critical, I agree. You don't want to be skimping on the hardware and software. However, if you have the money for a NetApp setup like this ($ 500,000+ US I'm guessing), then you also have the money to hire a FreeBSD developer(s) to work on the parts of the system that are critical to this (NFS, ZFS, CAM, drivers, scheduler, GEOM, etc). Then, you could go with a white-box, custom build and have the support in-house. --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:14:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE4F11065673 for ; Mon, 6 Feb 2012 17:14:40 +0000 (UTC) (envelope-from michael.aronsen@gmail.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 39AAD8FC14 for ; Mon, 6 Feb 2012 17:14:39 +0000 (UTC) Received: by wibhn14 with SMTP id hn14so6965291wib.13 for ; Mon, 06 Feb 2012 09:14:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; bh=M1ueNhsWgI5btNp2XuVsGRTYwYj/oo30cNkaujBopjk=; b=soJ1aXDaOl4l8sPLyg1VuA1t6coULA+dCWZQ9L6+W86YF6pEXQ9Rhe1dKTg+YzyWOZ +1P/7YAyXXv/Y1SUAmq7xGO4wkPg5USjpTb2GlL7K33ZnBXPtdJfC4HWqjEQD1XjOEEq EKzFGf4+MW0X34nldY5oVVxpbVxoKdHP/BZ9g= Received: by 10.180.86.198 with SMTP id r6mr14367227wiz.2.1328546971480; Mon, 06 Feb 2012 08:49:31 -0800 (PST) Received: from mbp.lan ([2.107.247.158]) by mx.google.com with ESMTPS id ex2sm48042252wib.1.2012.02.06.08.49.30 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 06 Feb 2012 08:49:31 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1251.1) From: Michael Aronsen In-Reply-To: <20120206162206.GA541@icarus.home.lan> Date: Mon, 6 Feb 2012 17:49:39 +0100 Message-Id: References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1251.1) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:14:40 -0000 Hi, On Feb 6, 2012, at 17:22 , Jeremy Chadwick wrote: > - What single motherboard supports up to 192GB of RAM Get an HP DL580/585 - they support 2TB/1TB RAM. > - How you plan on getting roughly 410 hard disks (or 422 assuming > an additional 12 SSDs) hooked up to a single machine Use LSI SAS92XX 4 (x4) port external controllers, and SuperMicro = SC847E26-RJBOD1 disk shelves. Each disk shelf needs 2 ports on the LSI controller, which means you get = 90 disks per LSI card. The DL580/585's have 11 PCIe slots, so you'd end up with 990 disks per = server using this setup. >=20 > If you are considering investing the time and especially money (the = cost > here is almost unfathomable, IMO) into this, I strongly recommend you > consider an actual hardware filer (e.g. NetApp). Your performance and > reliability will be much greater, plus you will get overall better > support from NetApp in the case something goes wrong. In the case you > run into problems with FreeBSD (and I can assure you in this kind of > setup you will) with this kind of extensive setup, you will be at the > mercy of developers' time/schedules with absolutely no guarantee that > your problem will be solved. You definitely want a support contract. > Thus, go NetApp. We have NetApp's at our University for home storage, but I would = struggle to recommend them for HPC storage. A dedicated HPC filesystem such as Lustre or FhGFS = (http://www.fhgfs.com/cms/) will almost certainly give you better = performance as they're purpose made. We use FhGFS in a rather small setup (44 TB usable space and ~200 HPC = nodes), but they do have installations with 700TB+. The setup consists of 2 metadata nodes and 4 storage nodes, all = supermicro servers with 24 WD Velociraptor 600 GB 10K RPM disks. This setup gives us 4.8GB/sec write and 4.3GB/sec read speeds, all for a = lot less than a comparable NetApp solution (we paid around =8030.000). It now has support for mirroring on a per folder level for resilience. Currently it only runs on Linux but i'm considering a FreeBSD port to = get ZFS for volume management and now that OFED is in FreeBSD 9, = Infinifband is possible. I'd highly recommend a parallel filesystem, unfortunately not many, if = any, are available on FreeBSD at this time. Regards, Michael From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:25:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45B441065673 for ; Mon, 6 Feb 2012 17:25:07 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from dedihh.fuckner.net (dedihh.fuckner.net [81.209.183.161]) by mx1.freebsd.org (Postfix) with ESMTP id BF0F78FC13 for ; Mon, 6 Feb 2012 17:25:06 +0000 (UTC) Received: from dedihh.fuckner.net (localhost [127.0.0.1]) by dedihh.fuckner.net (Postfix) with ESMTP id 8C9C8275C for ; Mon, 6 Feb 2012 18:25:05 +0100 (CET) X-Virus-Scanned: amavisd-new at fuckner.net Received: from dedihh.fuckner.net ([127.0.0.1]) by dedihh.fuckner.net (dedihh.fuckner.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id 7zU7cBaYGcFQ for ; Mon, 6 Feb 2012 18:24:58 +0100 (CET) Received: from c64.rebootking.de (e176135028.adsl.alicedsl.de [85.176.135.28]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by dedihh.fuckner.net (Postfix) with ESMTPSA id B0E51274A for ; Mon, 6 Feb 2012 18:24:58 +0100 (CET) Message-ID: <4F300CEA.5000901@fuckner.net> Date: Mon, 06 Feb 2012 18:24:58 +0100 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20120131 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:25:07 -0000 On 02/06/2012 05:41 PM, Freddie Cash wrote: Hi all, > On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick > wrote: >> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>> I want to investigate if it is possible to create your own usable >>> HPC storage using zfs and some network filesystem like nfs. especially HPS sounds interesting to me- but for HPC you typicially need fast r/w-access for all nodes in the cluster. That's why Lustre uses several storages for concurring access over a fast link (typicially Infiniband) Another thing to think about is CPU: you probably need weeks for a rebuild of a single disk in a Petabyte Filesystem- I haven't tried this with ZFS yet, but I'm really interested if anyone already did this. The whole setup sounds a little bit like the system shown by aberdeen: http://www.aberdeeninc.com/abcatg/petabyte-storage.htm schematics at tomshardware: http://www.tomshardware.de/fotoreportage/137-Aberdeen-petarack-petabyte-sas.html The Problem with Aberdeen is they don't use Zil/ L2Arc. >>> Just a thought experiment.. >>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) >>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >>> deives for cache. >>> Preferrably in mirror where applicable. >>> >>> Connected to this machine we will have about 410 3TB drives to give approx >>> 1PB of usable storage in a 8+2 raidz configuration. I don't know what the situation is for the rest of the world, but 3TB currently is still hard to buy in Europe/ Germany. >>> Connected to this will be a ~800 nodes big HPC cluster that will >>> access the storage in parallell what is your typical load pattern? >>> is this even possible or do we need to distribute the meta data load >>> over many servers? It is a good idea to have >>> If that is the case, >>> does it exist any software for FreeBSD that could accomplish this >>> distribution (pNFS dosent seem to be >>> anywhere close to usable in FreeBSD) or do I need to call NetApp or >>> Panasas right away? not that I know of > SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM > slots). It's an AMD board, but there should be variants that support > Intel CPUs. It's not uncommon to support 256 GB of RAM these days, > although 128 GB boards are much more common. Currently Intel CPUs have 3 Memory Channels. If you have 2 Sockets, 2 Dimms per Channel, 3 Channels- 12 Dimms with cheap 16GB Modules is 192GB. 32GB are also available today ;-) >> - How you plan on getting roughly 410 hard disks (or 422 assuming >> an additional 12 SSDs) hooked up to a single machine > > In a "head node" + "JBOD" setup? Where the head node has a mobo that > supports multiple PCIe x8 and PCIe x16 slots, and is stuffed full of > 16-24 port multi-lane SAS/SATA controllers with external ports that > are cabled up to external JBOD boxes. The SSDs would be connected to > the mobo SAS/SATA ports. > > Each JBOD box contains nothing but power, SAS/SATA backplane, and > harddrives. Possibly using SAS expanders. If you use Supermicro I would use X8DTH-iF, some LSI HBA (9200-8e, 2x Multilane external) and some JBOD-Chassis (like SUpermicro 847E16-RJBOD1) Regards, Michael! From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:35:01 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 163151065672 for ; Mon, 6 Feb 2012 17:35:01 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 91CFA8FC17 for ; Mon, 6 Feb 2012 17:35:00 +0000 (UTC) Received: from digsys226-136.pip.digsys.bg (digsys226-136.pip.digsys.bg [193.68.136.226]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q16HYn5J038404 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 6 Feb 2012 19:34:55 +0200 (EET) (envelope-from daniel@digsys.bg) Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Daniel Kalchev In-Reply-To: <4F300CEA.5000901@fuckner.net> Date: Mon, 6 Feb 2012 19:34:49 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <413B1A6F-B076-4F50-90EA-7E17CF4B6E36@digsys.bg> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> To: Michael Fuckner X-Mailer: Apple Mail (2.1257) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:35:01 -0000 On Feb 6, 2012, at 7:24 PM, Michael Fuckner wrote: > Another thing to think about is CPU: you probably need weeks for a = rebuild of a single disk in a Petabyte Filesystem- I haven't tried this = with ZFS yet, but I'm really interested if anyone already did this. This is where ZFS will shine. Depending on how you stripe disks, you can = either get super fast resilver (if you go for stripe of mirrors), to = fast (if you go for small number of disks raidz) to reasonable (if you = of for large number of disks raidz). If you need high TPS you will want = to go with mirrors anyway. The thing is doable with commodity hardware, but I wonder how one ever = backups such setup? Daniel= From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:38:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF573106564A for ; Mon, 6 Feb 2012 17:38:31 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 834C98FC08 for ; Mon, 6 Feb 2012 17:38:31 +0000 (UTC) Received: by vbbfa15 with SMTP id fa15so6235164vbb.13 for ; Mon, 06 Feb 2012 09:38:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=mIKF0AB4gGQhsMIQ8ibZah4k1K6hcFfG3DtV/Pvo4Wo=; b=jUEa3uUUObVIHWbGj/Z8P/SA2KsyaqnKbEv8Y8+oeLSqTmlj6e/hXOenFfA37T9uNY BdjQ5VcVMCaBjW240Fw2pNMZolc2BSQBH0HLwoxDRUxp3HOpQqFPOfPbrWgtYjBOKVin B6kO0lr4zRJj6sd4scA5erc52/5zxkx4o8MAY= MIME-Version: 1.0 Received: by 10.52.29.11 with SMTP id f11mr2203526vdh.66.1328549910781; Mon, 06 Feb 2012 09:38:30 -0800 (PST) Received: by 10.220.181.4 with HTTP; Mon, 6 Feb 2012 09:38:30 -0800 (PST) In-Reply-To: <413B1A6F-B076-4F50-90EA-7E17CF4B6E36@digsys.bg> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> <413B1A6F-B076-4F50-90EA-7E17CF4B6E36@digsys.bg> Date: Mon, 6 Feb 2012 09:38:30 -0800 Message-ID: From: Freddie Cash To: Daniel Kalchev Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:38:31 -0000 On Mon, Feb 6, 2012 at 9:34 AM, Daniel Kalchev wrote: > On Feb 6, 2012, at 7:24 PM, Michael Fuckner wrote: >> Another thing to think about is CPU: you probably need weeks for a rebui= ld of a single disk in a Petabyte Filesystem- I haven't tried this with ZFS= yet, but I'm really interested if anyone already did this. > > This is where ZFS will shine. Depending on how you stripe disks, you can = either get super fast resilver (if you go for stripe of mirrors), to fast (= if you go for small number of disks raidz) to reasonable (if you of for lar= ge number of disks raidz). If you need high TPS you will want to go with mi= rrors anyway. > > The thing is doable with commodity hardware, but I wonder how one ever ba= ckups such setup? With a second box configured similarily. :) Although, trying to find "downtime" to do the backups ... --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:39:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 631EC106566B for ; Mon, 6 Feb 2012 17:39:18 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 298418FC16 for ; Mon, 6 Feb 2012 17:39:17 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q16HdG0P004494; Mon, 6 Feb 2012 11:39:16 -0600 (CST) Date: Mon, 6 Feb 2012 11:39:16 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Michael Fuckner In-Reply-To: <4F300CEA.5000901@fuckner.net> Message-ID: References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 06 Feb 2012 11:39:16 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:39:18 -0000 On Mon, 6 Feb 2012, Michael Fuckner wrote: > > Another thing to think about is CPU: you probably need weeks for a rebuild of > a single disk in a Petabyte Filesystem- I haven't tried this with ZFS yet, > but I'm really interested if anyone already did this. Why would a disk rebuild take longer for a petabyte filesystem rather than a tens of gigabytes filesystem? The time to rebuild the disk primarily depends on the RAID type used for the zfs vdev (mirrors, raidz1, raidz2, raidz3), how many disks there are in the vdev, the degree of fragmentation, the amount of data stored on that disk, and the disk seek times. In a huge system, it makes sense to be more conservative about the zfs vdev design, and use more vdevs with fewer disks per vdev. Using anything less than raidz2 would be an error. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:40:22 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2695B106566C for ; Mon, 6 Feb 2012 17:40:22 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from gw02.mail.saunalahti.fi (gw02.mail.saunalahti.fi [195.197.172.116]) by mx1.freebsd.org (Postfix) with ESMTP id CEF428FC16 for ; Mon, 6 Feb 2012 17:40:21 +0000 (UTC) Received: from a91-153-116-96.elisa-laajakaista.fi (a91-153-116-96.elisa-laajakaista.fi [91.153.116.96]) by gw02.mail.saunalahti.fi (Postfix) with SMTP id 0496D139814; Mon, 6 Feb 2012 19:40:17 +0200 (EET) Date: Mon, 6 Feb 2012 19:40:16 +0200 From: Jaakko Heinonen To: Martin Matuska Message-ID: <20120206174016.GA2270@a91-153-116-96.elisa-laajakaista.fi> References: <4F2F9CB9.3040000@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F2F9CB9.3040000@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:40:22 -0000 On 2012-02-06, Martin Matuska wrote: > The attached patch adds a "ruleset" mount option to devfs mounts. > + if (mp->mnt_optnew != NULL && > + vfs_getopt(mp->mnt_optnew, "ruleset", NULL, NULL) == 0) { > + if (vfs_scanopt(mp->mnt_optnew, "ruleset", "%d", > + &rsnum) != 1) { > + vfs_mount_error(mp, "%s", > + "invalid ruleset specification"); > + return (EINVAL); > + } > + } The "ruleset" mount option will persist after the devfs_mount() call. It will get out of sync if the ruleset is later changed with the devfs command. devfs_mount() seems to miss a vfs_filteropt(9) call. -- Jaakko From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:58:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53A57106564A for ; Mon, 6 Feb 2012 17:58:39 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 25CA38FC13 for ; Mon, 6 Feb 2012 17:58:38 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id q16HUDuR040954 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 6 Feb 2012 09:30:14 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4F300E76.9010605@freebsd.org> Date: Mon, 06 Feb 2012 09:31:34 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.26) Gecko/20120129 Thunderbird/3.1.18 MIME-Version: 1.0 To: Michael Fuckner References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> In-Reply-To: <4F300CEA.5000901@fuckner.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:58:39 -0000 On 2/6/12 9:24 AM, Michael Fuckner wrote: > On 02/06/2012 05:41 PM, Freddie Cash wrote: > Hi all, > >> On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick >> wrote: >>> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>>> I want to investigate if it is possible to create your own usable >>>> HPC storage using zfs and some network filesystem like nfs.If you >>>> use Supermicro I would use X8DTH-iF, some LSI HBA (9200-8e, 2x >>>> Multilane external) and some JBOD-Chassis (like SUpermicro >>>> 847E16-RJBOD1) > no-one seems to have mentioned the obvious route.. a cluster of machines, using the new iSCSI code to make some of them subservient to the others. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:58:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA3BA106566C for ; Mon, 6 Feb 2012 17:58:39 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 77FAD8FC14 for ; Mon, 6 Feb 2012 17:58:39 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id q16HPLnD040930 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 6 Feb 2012 09:25:24 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4F300D52.1050502@freebsd.org> Date: Mon, 06 Feb 2012 09:26:42 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.26) Gecko/20120129 Thunderbird/3.1.18 MIME-Version: 1.0 To: Freddie Cash References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:58:39 -0000 On 2/6/12 8:41 AM, Freddie Cash wrote: > On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick > wrote: >> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>> I want to investigate if it is possible to create your own usable >>> HPC storage using zfs and some >>> network filesystem like nfs. >>> >>> Just a thought experiment.. >>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) >>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >>> deives for cache. >>> Preferrably in mirror where applicable. >>> >>> Connected to this machine we will have about 410 3TB drives to give approx >>> 1PB of usable storage in a 8+2 raidz configuration. >>> >>> Connected to this will be a ~800 nodes big HPC cluster that will >>> access the storage in parallell >>> is this even possible or do we need to distribute the meta data load >>> over many servers? If that is the case, >>> does it exist any software for FreeBSD that could accomplish this >>> distribution (pNFS dosent seem to be >>> anywhere close to usable in FreeBSD) or do I need to call NetApp or >>> Panasas right away? It would be >>> really nice if I could build my own storage solution. >>> >>> Other possible solutions to this problem is extremley welcome. >> For starters I'd love to know: >> >> - What single motherboard supports up to 192GB of RAM > SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM > slots). It's an AMD board, but there should be variants that support > Intel CPUs. It's not uncommon to support 256 GB of RAM these days, > although 128 GB boards are much more common. > common wisdom for ZFS is 1GB of RAM per TB of storage.. so 256GB might not be enough. people who have actually tried ZFS more than me may want to comment more on this.. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 18:04:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37DA21065670 for ; Mon, 6 Feb 2012 18:04:09 +0000 (UTC) (envelope-from peter@pean.org) Received: from smtprelay-b11.telenor.se (smtprelay-b11.telenor.se [62.127.194.20]) by mx1.freebsd.org (Postfix) with ESMTP id E40538FC17 for ; Mon, 6 Feb 2012 18:04:08 +0000 (UTC) Received: from ipb4.telenor.se (ipb4.telenor.se [195.54.127.167]) by smtprelay-b11.telenor.se (Postfix) with ESMTP id 183BFCE57; Mon, 6 Feb 2012 19:04:08 +0100 (CET) X-SENDER-IP: [85.225.5.217] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjRIAHgVME9V4QXZPGdsb2JhbABCr0cZAQEBATcygXIBAQQBViMFCwsYHBItDAoUBhMJh3MDsGCLKQQGAwINAgcHBAYBNAYKA4J9BRgCCwIFgReCVmMElSiSXA X-IronPort-AV: E=Sophos;i="4.73,372,1325458800"; d="scan'208";a="1804421054" Received: from c-d905e155.166-7-64736c14.cust.bredbandsbolaget.se (HELO pi.lan) ([85.225.5.217]) by ipb4.telenor.se with ESMTP; 06 Feb 2012 19:04:07 +0100 Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=iso-8859-1 From: =?iso-8859-1?Q?Peter_Ankerst=E5l?= In-Reply-To: <4F300E76.9010605@freebsd.org> Date: Mon, 6 Feb 2012 19:04:07 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <709F7F6B-3B29-4A89-85FA-22E4CDD0A042@pean.org> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> <4F300E76.9010605@freebsd.org> To: Julian Elischer X-Mailer: Apple Mail (2.1257) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 18:04:09 -0000 -- Peter Ankerst=E5l peter@pean.org http://www.pean.org/ On 6 feb 2012, at 18:31, Julian Elischer wrote: > On 2/6/12 9:24 AM, Michael Fuckner wrote: >> On 02/06/2012 05:41 PM, Freddie Cash wrote: >> Hi all, >>=20 >>> On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick >>> wrote: >>>> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>>>> I want to investigate if it is possible to create your own usable >>>>> HPC storage using zfs and some network filesystem like nfs.If you = use Supermicro I would use X8DTH-iF, some LSI HBA (9200-8e, 2x Multilane = external) and some JBOD-Chassis (like SUpermicro 847E16-RJBOD1) >>=20 >=20 > no-one seems to have mentioned the obvious route.. >=20 > a cluster of machines, using the new iSCSI code to make some of them = subservient to the others. >=20 I have thought about iSCSI but will this actually scale meta data = performance? From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 18:04:41 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8A06106564A for ; Mon, 6 Feb 2012 18:04:41 +0000 (UTC) (envelope-from peter@pean.org) Received: from smtprelay-b11.telenor.se (smtprelay-b11.telenor.se [62.127.194.20]) by mx1.freebsd.org (Postfix) with ESMTP id 664978FC1B for ; Mon, 6 Feb 2012 18:04:41 +0000 (UTC) Received: from ipb5.telenor.se (ipb5.telenor.se [195.54.127.168]) by smtprelay-b11.telenor.se (Postfix) with ESMTP id 7A86DCE43 for ; Mon, 6 Feb 2012 18:47:54 +0100 (CET) X-SENDER-IP: [85.225.5.217] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjJIALcRME9V4QXZPGdsb2JhbABDr0MZAQEBATcygXIBAQQBViMFCwsYHBItDAoUBi6HYQO4costBwICIQYBNAYNgn0FGAILAgWDbWMElSiLDYdP X-IronPort-AV: E=Sophos;i="4.73,372,1325458800"; d="scan'208";a="40314967" Received: from c-d905e155.166-7-64736c14.cust.bredbandsbolaget.se (HELO pi.lan) ([85.225.5.217]) by ipb5.telenor.se with ESMTP; 06 Feb 2012 18:47:53 +0100 Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=windows-1252 From: =?iso-8859-1?Q?Peter_Ankerst=E5l?= In-Reply-To: Date: Mon, 6 Feb 2012 18:47:53 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> To: Michael Aronsen X-Mailer: Apple Mail (2.1257) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 18:04:41 -0000 On 6 feb 2012, at 17:49, Michael Aronsen wrote: > Hi, >=20 > On Feb 6, 2012, at 17:22 , Jeremy Chadwick wrote: >> - What single motherboard supports up to 192GB of RAM >=20 > Get an HP DL580/585 - they support 2TB/1TB RAM. >=20 >> - How you plan on getting roughly 410 hard disks (or 422 assuming >> an additional 12 SSDs) hooked up to a single machine >=20 > Use LSI SAS92XX 4 (x4) port external controllers, and SuperMicro = SC847E26-RJBOD1 disk shelves. > Each disk shelf needs 2 ports on the LSI controller, which means you = get 90 disks per LSI card. > The DL580/585's have 11 PCIe slots, so you'd end up with 990 disks per = server using this setup. >=20 >>=20 >=20 > We have NetApp's at our University for home storage, but I would = struggle to recommend them for HPC storage. >=20 > A dedicated HPC filesystem such as Lustre or FhGFS = (http://www.fhgfs.com/cms/) will almost certainly give you better = performance as they're purpose made. >=20 > We use FhGFS in a rather small setup (44 TB usable space and ~200 HPC = nodes), but they do have installations with 700TB+. > The setup consists of 2 metadata nodes and 4 storage nodes, all = supermicro servers with 24 WD Velociraptor 600 GB 10K RPM disks. > This setup gives us 4.8GB/sec write and 4.3GB/sec read speeds, all for = a lot less than a comparable NetApp solution (we paid around =8030.000). > It now has support for mirroring on a per folder level for resilience. >=20 > Currently it only runs on Linux but i'm considering a FreeBSD port to = get ZFS for volume management and now that OFED is in FreeBSD 9, = Infinifband is possible. >=20 > I'd highly recommend a parallel filesystem, unfortunately not many, if = any, are available on FreeBSD at this time. >=20 Thanks for the input. We recently had a visit by NetApp and Whamcloud = actually and they where pitching for a NetApp+Whamcloud(lustre) = installation. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 18:04:42 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16276106566C for ; Mon, 6 Feb 2012 18:04:42 +0000 (UTC) (envelope-from peter@pean.org) Received: from smtprelay-b11.telenor.se (smtprelay-b11.telenor.se [62.127.194.20]) by mx1.freebsd.org (Postfix) with ESMTP id 974D98FC1D for ; Mon, 6 Feb 2012 18:04:41 +0000 (UTC) Received: from ipb4.telenor.se (ipb4.telenor.se [195.54.127.167]) by smtprelay-b11.telenor.se (Postfix) with ESMTP id 299CBCD8A for ; Mon, 6 Feb 2012 18:44:01 +0100 (CET) X-SENDER-IP: [85.225.5.217] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjdIAMIQME9V4QXZPGdsb2JhbABCr0cZAQEBATcygXIBAQQBViMFCwsOChwSIQwMChQGEwkSh2EDsFmIFYMUBAYDAg0CBwcEBgE0BoMKBRgCCwIFgReCVmMElSiLCodS X-IronPort-AV: E=Sophos;i="4.73,372,1325458800"; d="scan'208";a="1804415825" Received: from c-d905e155.166-7-64736c14.cust.bredbandsbolaget.se (HELO pi.lan) ([85.225.5.217]) by ipb4.telenor.se with ESMTP; 06 Feb 2012 18:44:00 +0100 Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=iso-8859-1 From: =?iso-8859-1?Q?Peter_Ankerst=E5l?= In-Reply-To: Date: Mon, 6 Feb 2012 18:43:59 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <7321E3F2-35E2-43F4-9932-EC55F5AA9D0B@pean.org> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> To: Freddie Cash X-Mailer: Apple Mail (2.1257) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 18:04:42 -0000 -- Peter Ankerst=E5l peter@pean.org http://www.pean.org/ On 6 feb 2012, at 17:41, Freddie Cash wrote: > On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick > wrote: >> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>> I want to investigate if it is possible to create your own usable >>> HPC storage using zfs and some >>> network filesystem like nfs. >>>=20 >>> Just a thought experiment.. >>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or = more) >>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >>> deives for cache. >>> Preferrably in mirror where applicable. >>>=20 >>> Connected to this machine we will have about 410 3TB drives to give = approx >>> 1PB of usable storage in a 8+2 raidz configuration. >>>=20 >>> Connected to this will be a ~800 nodes big HPC cluster that will >>> access the storage in parallell >>> is this even possible or do we need to distribute the meta data load >>> over many servers? If that is the case, >>> does it exist any software for FreeBSD that could accomplish this >>> distribution (pNFS dosent seem to be >>> anywhere close to usable in FreeBSD) or do I need to call NetApp or >>> Panasas right away? It would be >>> really nice if I could build my own storage solution. >>>=20 >>> Other possible solutions to this problem is extremley welcome. >>=20 >> For starters I'd love to know: >>=20 >> - What single motherboard supports up to 192GB of RAM >=20 > SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM > slots). It's an AMD board, but there should be variants that support > Intel CPUs. It's not uncommon to support 256 GB of RAM these days, > although 128 GB boards are much more common. Yeah, the one I was looking at was SuperMicro X8DTU-F, but yeah, the = more money RAM the better. >=20 >> - How you plan on getting roughly 410 hard disks (or 422 assuming >> an additional 12 SSDs) hooked up to a single machine >=20 > In a "head node" + "JBOD" setup? Where the head node has a mobo that > supports multiple PCIe x8 and PCIe x16 slots, and is stuffed full of > 16-24 port multi-lane SAS/SATA controllers with external ports that > are cabled up to external JBOD boxes. The SSDs would be connected to > the mobo SAS/SATA ports. >=20 > Each JBOD box contains nothing but power, SAS/SATA backplane, and > harddrives. Possibly using SAS expanders. >=20 > We're considering doing the same for our SAN/NAS setup for > centralising storage for our VM hosts, although not quite to the same > scale as the OP. :) Yep, NetApp has disk-shelves that can be configured JBOD that fits 60 = drives into 4U. :D >=20 >> If you are considering investing the time and especially money (the = cost >> here is almost unfathomable, IMO) into this, I strongly recommend you >> consider an actual hardware filer (e.g. NetApp). Your performance = and >> reliability will be much greater, plus you will get overall better >> support from NetApp in the case something goes wrong. In the case = you >> run into problems with FreeBSD (and I can assure you in this kind of >> setup you will) with this kind of extensive setup, you will be at the >> mercy of developers' time/schedules with absolutely no guarantee that >> your problem will be solved. You definitely want a support contract. >> Thus, go NetApp. >=20 > For an HPC setup like the OP wants, where performance and uptime are > critical, I agree. You don't want to be skimping on the hardware and > software. >=20 A big consideration for us is also the installation. If we go with = something like NetApp they can install the system and we don't need to put in the extra = hours (probably a lot) the get the thing running. But being a huge fan of BSD = I wanted to at least look up the possibility to build our own system. > However, if you have the money for a NetApp setup like this ($ > 500,000+ US I'm guessing), then you also have the money to hire a > FreeBSD developer(s) to work on the parts of the system that are > critical to this (NFS, ZFS, CAM, drivers, scheduler, GEOM, etc). > Then, you could go with a white-box, custom build and have the support > in-house. >=20 > --=20 > Freddie Cash > fjwcash@gmail.com >=20 From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 18:05:58 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 802D3106566B for ; Mon, 6 Feb 2012 18:05:58 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from mo-p05-ob6.rzone.de (mo-p05-ob6.rzone.de [IPv6:2a01:238:20a:202:53f5::1]) by mx1.freebsd.org (Postfix) with ESMTP id DF1AC8FC14 for ; Mon, 6 Feb 2012 18:05:57 +0000 (UTC) X-RZG-AUTH: :LWIKdA2leu0bPbLmhzXgqn0MTG6qiKEwQRWfNxSw4HzYIwjsnvdDt2oX8drk23mufl0bSeTV X-RZG-CLASS-ID: mo05 Received: from [192.168.179.39] (hmbg-5f760af7.pool.mediaWays.net [95.118.10.247]) by post.strato.de (mrclete mo6) (RZmta 27.6 DYNA|AUTH) with (DHE-RSA-AES128-SHA encrypted) ESMTPA id z052f4o16HI0MC for ; Mon, 6 Feb 2012 19:05:38 +0100 (MET) Message-ID: <4F301671.3060507@brockmann-consult.de> Date: Mon, 06 Feb 2012 19:05:37 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F2FF72B.6000509@pean.org> In-Reply-To: <4F2FF72B.6000509@pean.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 18:05:58 -0000 Am 06.02.2012 16:52, schrieb Peter Ankerstål: > Hi, > > I want to investigate if it is possible to create your own usable HPC > storage using zfs and some > network filesystem like nfs. > > Just a thought experiment.. > A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) > I addition the machine will use 3-6 SSD drives for ZIL FYI: With ESXi's NFS client (**the most evil of anything doing synchronous writes), striping my ZIL made no difference. Either way, it used 100% load on the SSDs and went the same pathetic speed. (gstripe worked to some extent) Using "-o sync" in the Linux NFS client does not use the ZIL at max load*. If you can suggest another way to test (without ESXi), I will try it on 3 SSDs (4 if the one that I RMAed comes back in time) and let you know the results. * With a pool that was created in 8.2-STABLE and upgraded to v28, it did use max load. Now it doesn't. But is recreating that situation the right way to test? It is not what you will be using. ** For more info about ESXi's terrible NFS performance, see this graph http://doub.home.xs4all.nl/bench/sync.png and find the mail I got it from in this mailing list fromjoh.hendriks@gmail.com with the title "Re: ZFS sync / ZIL clarification". > and 3-6 SSD deives for cache. > Preferrably in mirror where applicable. > > Connected to this machine we will have about 410 3TB drives to give > approx > 1PB of usable storage in a 8+2 raidz configuration. > > Connected to this will be a ~800 nodes big HPC cluster that will > access the storage in parallell > is this even possible or do we need to distribute the meta data load > over many servers? If that is the case, > does it exist any software for FreeBSD that could accomplish this > distribution (pNFS dosent seem to be > anywhere close to usable in FreeBSD) or do I need to call NetApp or > Panasas right away? It would be > really nice if I could build my own storage solution. > > Other possible solutions to this problem is extremley welcome. > > Best Regards > Peter Ankerstål > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 18:18:44 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D04361065670 for ; Mon, 6 Feb 2012 18:18:44 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from dedihh.fuckner.net (dedihh.fuckner.net [81.209.183.161]) by mx1.freebsd.org (Postfix) with ESMTP id 8A97B8FC1A for ; Mon, 6 Feb 2012 18:18:44 +0000 (UTC) Received: from dedihh.fuckner.net (localhost [127.0.0.1]) by dedihh.fuckner.net (Postfix) with ESMTP id 01AE830E5; Mon, 6 Feb 2012 19:18:43 +0100 (CET) X-Virus-Scanned: amavisd-new at fuckner.net Received: from dedihh.fuckner.net ([127.0.0.1]) by dedihh.fuckner.net (dedihh.fuckner.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id sBuKRpN77Bw9; Mon, 6 Feb 2012 19:18:33 +0100 (CET) Received: from c64.rebootking.de (e176135028.adsl.alicedsl.de [85.176.135.28]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by dedihh.fuckner.net (Postfix) with ESMTPSA id B60ED30DA; Mon, 6 Feb 2012 19:18:33 +0100 (CET) Message-ID: <4F301979.6090302@fuckner.net> Date: Mon, 06 Feb 2012 19:18:33 +0100 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20120131 Thunderbird/10.0 MIME-Version: 1.0 To: Daniel Kalchev References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> <413B1A6F-B076-4F50-90EA-7E17CF4B6E36@digsys.bg> In-Reply-To: <413B1A6F-B076-4F50-90EA-7E17CF4B6E36@digsys.bg> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 18:18:44 -0000 On 02/06/2012 06:34 PM, Daniel Kalchev wrote: > The thing is doable with commodity hardware, but I wonder how one ever backups such setup? zfs send to a second machine? From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 19:28:44 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0B20106564A; Mon, 6 Feb 2012 19:28:44 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id A13A68FC12; Mon, 6 Feb 2012 19:28:44 +0000 (UTC) Received: from core2.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 0B978214C7; Mon, 6 Feb 2012 20:28:44 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core2.vx.sk (amavisd-new, unix socket) with LMTP id rsnRjvVRkekf; Mon, 6 Feb 2012 20:28:38 +0100 (CET) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id C1B05214B7; Mon, 6 Feb 2012 20:28:38 +0100 (CET) Message-ID: <4F3029E6.6000900@FreeBSD.org> Date: Mon, 06 Feb 2012 20:28:38 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120130 Thunderbird/10.0 MIME-Version: 1.0 To: Jaakko Heinonen References: <4F2F9CB9.3040000@FreeBSD.org> <20120206174016.GA2270@a91-153-116-96.elisa-laajakaista.fi> In-Reply-To: <20120206174016.GA2270@a91-153-116-96.elisa-laajakaista.fi> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@FreeBSD.org Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 19:28:45 -0000 Dňa 6.2.2012 18:40, Jaakko Heinonen wrote / napísal(a): > On 2012-02-06, Martin Matuska wrote: >> The attached patch adds a "ruleset" mount option to devfs mounts. >> + if (mp->mnt_optnew != NULL && >> + vfs_getopt(mp->mnt_optnew, "ruleset", NULL, NULL) == 0) { >> + if (vfs_scanopt(mp->mnt_optnew, "ruleset", "%d", >> + &rsnum) != 1) { >> + vfs_mount_error(mp, "%s", >> + "invalid ruleset specification"); >> + return (EINVAL); >> + } >> + } > The "ruleset" mount option will persist after the devfs_mount() call. It > will get out of sync if the ruleset is later changed with the devfs > command. It is not displayed in mount -v / -p and is not used anywhere on existing mounts, devfs_mount() doesn't support MNT_UPDATE. If we would add update support than we still wouldn't rely on this option as it would be read from devfs. > devfs_mount() seems to miss a vfs_filteropt(9) call. There wasn't a vfs_filteropt(9) before, too. I can not tell whether this is desired, but what I can tell if you introduce a vfs_filteropt you change the behavior of mounting devfs. Maybe it is desired, but this is something open for discussion, I do not feel privileged to decide alone what options are to be allowed (readonly, noatime, nosymfollow, etc.). As a minimum, you need the "from" option allowed. This may very well break backwards compatibility. -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 19:47:52 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 134DD106566B; Mon, 6 Feb 2012 19:47:52 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from gw02.mail.saunalahti.fi (gw02.mail.saunalahti.fi [195.197.172.116]) by mx1.freebsd.org (Postfix) with ESMTP id C11B48FC1F; Mon, 6 Feb 2012 19:47:51 +0000 (UTC) Received: from jh (a91-153-115-208.elisa-laajakaista.fi [91.153.115.208]) by gw02.mail.saunalahti.fi (Postfix) with SMTP id A2C25139C1F; Mon, 6 Feb 2012 21:47:48 +0200 (EET) Date: Mon, 6 Feb 2012 21:47:48 +0200 From: Jaakko Heinonen To: Martin Matuska Message-ID: <20120206194747.GA21280@jh> References: <4F2F9CB9.3040000@FreeBSD.org> <20120206174016.GA2270@a91-153-116-96.elisa-laajakaista.fi> <4F3029E6.6000900@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F3029E6.6000900@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.org Subject: Re: [CFR][DEVFS] Add "ruleset" mount option X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 19:47:52 -0000 On 2012-02-06, Martin Matuska wrote: > It is not displayed in mount -v / -p and is not used anywhere on > existing mounts, devfs_mount() doesn't support MNT_UPDATE. True. There has been some suggestions to add an interface to query string mount options. > If we would add update support than we still wouldn't rely on this > option as it would be read from devfs. If there is no use for the value after initial mount, IMO it shouldn't be stored. This again demonstrates the fragility of nmount. -- Jaakko From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 20:08:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2B891065670 for ; Mon, 6 Feb 2012 20:08:07 +0000 (UTC) (envelope-from erik@tefre.com) Received: from mta1-filtered.netlife.no (mail.netlife.no [62.92.26.226]) by mx1.freebsd.org (Postfix) with ESMTP id 681E78FC21 for ; Mon, 6 Feb 2012 20:08:07 +0000 (UTC) Received: from amavis2.netlife.no (amavishost [10.115.1.12]) by mta1-filtered.netlife.no (Postfix) with ESMTP id 11F3C3307C0 for ; Mon, 6 Feb 2012 20:51:42 +0100 (CET) X-Virus-Scanned: amavisd-new at netlife.no Received: from mta1-submission.netlife.no ([62.92.26.226]) by amavis2.netlife.no (amavis2.netlife.no [10.115.1.12]) (amavisd-new, port 10026) with ESMTP id luzycQzy_FcJ for ; Mon, 6 Feb 2012 20:51:41 +0100 (CET) Received: from [10.0.0.78] (203.84-49-43.nextgentel.com [84.49.43.203]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: erik@tefre.com) by mta1-submission.netlife.no (Postfix) with ESMTPSA id CC2BC3307BA for ; Mon, 6 Feb 2012 20:51:41 +0100 (CET) Message-ID: <4F302F60.3010302@tefre.com> Date: Mon, 06 Feb 2012 20:52:00 +0100 From: Erik Stian Tefre User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:9.0) Gecko/20111229 Thunderbird/9.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 20:08:07 -0000 On 02/06/2012 05:49 PM, Michael Aronsen wrote: >> - How you plan on getting roughly 410 hard disks (or 422 assuming >> an additional 12 SSDs) hooked up to a single machine > > Use LSI SAS92XX 4 (x4) port external controllers, and SuperMicro SC847E26-RJBOD1 disk shelves. > Each disk shelf needs 2 ports on the LSI controller, which means you get 90 disks per LSI card. > The DL580/585's have 11 PCIe slots, so you'd end up with 990 disks per server using this setup. The backplanes/expanders in the SC847E16-RJBOD1 (E16 = SATA/single expander version, E26 = SAS/dual expander version) can be daisy chained if you want to lower the number of controllers/ports. I have no E26 to test on but I guess it may support daisy chaining too. The LSI 9205-8e controller claims to support 1024 devices being connected to its two 4x6gb ports. Does anyone happen to know many daisy chain jumps are supported by SAS? Would it be possible to daisy chain 10 shelves with 450 drives onto 1 of those controllers? That would require 10 jumps on two chains or 20 jumps on one chain... (Yes, 6000*8/450 = 107 Mbit per drive, but ignore that part. ;) -- Erik From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 20:36:18 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id EB871106564A for ; Mon, 6 Feb 2012 20:36:18 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-150-251.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 1EB2A14EBF5; Mon, 6 Feb 2012 20:36:17 +0000 (UTC) Message-ID: <4F3039C1.5060200@FreeBSD.org> Date: Mon, 06 Feb 2012 12:36:17 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:10.0) Gecko/20120201 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org X-Enigmail-Version: 1.3.5 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Practical limit to number of mount points on a system? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 20:36:19 -0000 Howdy, I have some general questions regarding how many file system mounts one could reasonably make on a system (assume lots of CPU and RAM for the sake of argument). So in a general sense: 1. Is there a hard limit? 2. Do the type(s) of the mount(s) matter? 3. Does mounting in jails matter vs. mounting directly on the host? 4. And the $64 question, if I had a setup with lots of jails, and each had a nullfs, devfs, and 2 or 3 NFS mounts, how many jails could I have without causing an implosion? :) Thanks, Doug -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 20:41:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37F191065672 for ; Mon, 6 Feb 2012 20:41:07 +0000 (UTC) (envelope-from Kamil.Choudhury@anserinae.net) Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.120]) by mx1.freebsd.org (Postfix) with ESMTP id E89238FC1A for ; Mon, 6 Feb 2012 20:41:06 +0000 (UTC) Received: from hrndva-omtalb.mail.rr.com ([10.128.143.53]) by hrndva-qmta01.mail.rr.com with ESMTP id <20120206183612704.MOWC11472@hrndva-qmta01.mail.rr.com>; Mon, 6 Feb 2012 18:36:12 +0000 X-Authority-Analysis: v=2.0 cv=fNy7LOme c=1 sm=0 a=qe0RvMpo0P4Rp0DQO452oA==:17 a=WWGGoYozHbgA:10 a=kj9zAlcOel0A:10 a=xqWC_Br6kY4A:10 a=w-boLisARSGk8YcPp9QA:9 a=CjuIK1q_8ugA:10 a=qe0RvMpo0P4Rp0DQO452oA==:117 X-Cloudmark-Score: 0 X-Originating-IP: 68.173.236.44 Received: from [68.173.236.44] ([68.173.236.44:62294] helo=janus.anserinae.net) by hrndva-oedge03.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id 73/14-11045-55D103F4; Mon, 06 Feb 2012 18:35:06 +0000 Received: from JANUS.anserinae.net ([fe80::192c:4b89:9fe9:dc6d]) by janus.anserinae.net ([fe80::192c:4b89:9fe9:dc6d%11]) with mapi; Mon, 6 Feb 2012 13:34:55 -0500 From: Kamil Choudhury To: Michael Fuckner , Daniel Kalchev Thread-Topic: HPC and zfs. Thread-Index: AQHM5OoWn6a1UqnBhUSVAefkkYKv3pYwYOwAgAAFh4CAAAwKAIAAAsCAgAAMOID//6z9hA== Date: Mon, 6 Feb 2012 18:34:51 +0000 Message-ID: <3CEE2DA4348D944399A67E308B78D38A057C6E3E@janus.anserinae.net> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <4F300CEA.5000901@fuckner.net> <413B1A6F-B076-4F50-90EA-7E17CF4B6E36@digsys.bg>, <4F301979.6090302@fuckner.net> In-Reply-To: <4F301979.6090302@fuckner.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-fs@freebsd.org" Subject: RE: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 20:41:07 -0000 > zfs send to a second machine?=0A= =0A= Is there not likely to be a degradation in performance while doing the back= up? =0A= =0A= ...which ties into something that I think is missing from this thread: a st= atement of what this massive =0A= pool of storage is going to be used for. "HPC" is a pretty broad term, and = isn't quite a specification in =0A= my book. =0A= =0A= I have faith that the relevant BSD technologies can be cobbled together to = get OP to the performance=0A= level he needs, but -- and perhaps I have failed basic reading comprehensio= n here -- I don't think we =0A= have a clear enough picture of what that level actually /is/ to offer advic= e on what is almost certainly=0A= going to be a multi-million dollar installation! =0A= =0A= KC= From owner-freebsd-fs@FreeBSD.ORG Tue Feb 7 04:31:21 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8DE591065743; Tue, 7 Feb 2012 04:31:21 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 728088FC0C; Tue, 7 Feb 2012 04:31:21 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q174VLlH081240; Tue, 7 Feb 2012 04:31:21 GMT (envelope-from mckusick@freefall.freebsd.org) Received: (from mckusick@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q174VLhE081222; Tue, 7 Feb 2012 04:31:21 GMT (envelope-from mckusick) Date: Tue, 7 Feb 2012 04:31:21 GMT Message-Id: <201202070431.q174VLhE081222@freefall.freebsd.org> To: sh_tanaka@yahoo.co.jp, mckusick@FreeBSD.org, freebsd-fs@FreeBSD.org, mckusick@FreeBSD.org From: mckusick@FreeBSD.org Cc: Subject: Re: kern/161674: [ufs] snapshot on journaled ufs doesn't work X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 04:31:21 -0000 Synopsis: [ufs] snapshot on journaled ufs doesn't work State-Changed-From-To: open->closed State-Changed-By: mckusick State-Changed-When: Tue Feb 7 04:27:05 UTC 2012 State-Changed-Why: This bug was fixed in head in -r230249: Make sure all intermediate variables holding mount flags (mnt_flag) and that all internal kernel calls passing mount flags are declared as uint64_t so that flags in the top 32-bits are not lost. The soft update journalling flag is located in the top 32-bits of mnt_flag and was being lost due to this bug. This fix was MFC'ed to 9 in -r230725 on Sun Jan 29 08:03:45 2012. Responsible-Changed-From-To: freebsd-fs->mckusick Responsible-Changed-By: mckusick Responsible-Changed-When: Tue Feb 7 04:27:05 UTC 2012 Responsible-Changed-Why: I have taken responsibility for this bug. http://www.freebsd.org/cgi/query-pr.cgi?pr=161674 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 7 10:02:30 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE8561065676; Tue, 7 Feb 2012 10:02:30 +0000 (UTC) (envelope-from kevlo@FreeBSD.org) Received: from ns.kevlo.org (kevlo.org [220.128.136.52]) by mx1.freebsd.org (Postfix) with ESMTP id 36FD68FC0C; Tue, 7 Feb 2012 10:02:29 +0000 (UTC) Received: from [127.0.0.1] (kevlo.org [220.128.136.52]) by ns.kevlo.org (8.14.5/8.14.5) with ESMTP id q179psUp017228; Tue, 7 Feb 2012 17:51:54 +0800 (CST) (envelope-from kevlo@FreeBSD.org) Message-ID: <1328608318.2117.9.camel@nsl> From: Kevin Lo To: eadler@FreeBSD.org Date: Tue, 07 Feb 2012 17:51:58 +0800 In-Reply-To: <201201021517.q02FHAOO041155@freefall.freebsd.org> References: <201201021517.q02FHAOO041155@freefall.freebsd.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.2- Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Cc: freebsd-fs@FreeBSD.org, gcooper@FreeBSD.org Subject: Re: kern/117314: [ntfs] Long-filename only NTFS fs'es cause kernel panics on read X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 10:02:30 -0000 eadler@freebsd.org wrote: > Synopsis: [ntfs] Long-filename only NTFS fs'es cause kernel panics on read > > State-Changed-From-To: open->feedback > State-Changed-By: eadler > State-Changed-When: Mon Jan 2 15:17:09 UTC 2012 > State-Changed-Why: > Is this still an issue? > > http://www.freebsd.org/cgi/query-pr.cgi?pr=117314 For the record, I can't reproduce this problem on 9.0 and -current. Kevin From owner-freebsd-fs@FreeBSD.ORG Tue Feb 7 16:22:57 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8999D106564A for ; Tue, 7 Feb 2012 16:22:57 +0000 (UTC) (envelope-from dorionpatrick@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6622E8FC14 for ; Tue, 7 Feb 2012 16:22:57 +0000 (UTC) Received: by daec6 with SMTP id c6so7292861dae.13 for ; Tue, 07 Feb 2012 08:22:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=m4wSurQBuGQTIA6sBoFMA/NzfxXRveh/hJTk4M5LpsY=; b=ihPlf/Ek5DY9Wg6dmCRFIF2RjVdXKbIDYDd2JEjFBLAT3u6VWX7CYCrX7Jvl7XRIuT QxZb7/R1bPJhR1RrQMXwtp7ufgTsm21UXX0HA4IKp6J7xZsMHJgbOY6bvRkuLcrKhylO ii5XRY3IYKH8IK+9MI1gry0Z/xDGD5YSermJw= MIME-Version: 1.0 Received: by 10.68.73.8 with SMTP id h8mr59057765pbv.109.1328630163737; Tue, 07 Feb 2012 07:56:03 -0800 (PST) Received: by 10.68.221.231 with HTTP; Tue, 7 Feb 2012 07:56:03 -0800 (PST) Date: Tue, 7 Feb 2012 10:56:03 -0500 Message-ID: From: Patrick Dorion To: freebsd-fs@freebsd.org X-Mailman-Approved-At: Tue, 07 Feb 2012 16:52:18 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS boots RO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 16:22:57 -0000 Hi, I've been having a problem for some time now. When I reboot, the root ZFS mounts readonly=on source=temporary. As a result, it's unable to launch the services and the buffer is rather uninformative about the condition. I thought it was a problem in the configuration but I destroyed the zpool before installing 9.0-R and I tried to reboot with and without loader.conf, rc.conf and fstab configuration parameters for the filesystem. The only non-default setting for the filesystem is utf8only=on and I never had an issue with that in the past. Any ideas? Patrick Dorion From owner-freebsd-fs@FreeBSD.ORG Tue Feb 7 20:30:31 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 293B4106564A for ; Tue, 7 Feb 2012 20:30:31 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F10CF8FC1B for ; Tue, 7 Feb 2012 20:30:30 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q17KUUdW003881 for ; Tue, 7 Feb 2012 20:30:30 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q17KUUTE003876; Tue, 7 Feb 2012 20:30:30 GMT (envelope-from gnats) Date: Tue, 7 Feb 2012 20:30:30 GMT Message-Id: <201202072030.q17KUUTE003876@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: "Steven Hartland" Cc: Subject: Re: kern/161897: [zfs] [patch] zfs partition probing causing long delay at BTX loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Steven Hartland List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 20:30:31 -0000 The following reply was made to PR kern/161897; it has been noted by GNATS. From: "Steven Hartland" To: Cc: Subject: Re: kern/161897: [zfs] [patch] zfs partition probing causing long delay at BTX loader Date: Tue, 7 Feb 2012 09:26:07 -0000 Any update on this? From owner-freebsd-fs@FreeBSD.ORG Tue Feb 7 21:21:47 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1E1B1065670 for ; Tue, 7 Feb 2012 21:21:47 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 60CCD8FC12 for ; Tue, 7 Feb 2012 21:21:47 +0000 (UTC) Received: by yenl12 with SMTP id l12so4059987yen.13 for ; Tue, 07 Feb 2012 13:21:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=iy63AhwdSSVJvSVLkMTH79zc45v+X9NfzOwNdnH5rSM=; b=TOna4UKJ6Y1CZV+88X+a23LI4lDaSoEhU6XgZUnV28FtmTAd4yO0VwacFg3CkiVPZE OhRjEpE+aODzCrLtP9Hsev1EM4alLwXIXnbcw5kvt3zepA47YBPhq1QD2rak2NliN4I4 ujoO05pnIux0AXD8WFK0w1bfIE6kuoRiEIl1w= MIME-Version: 1.0 Received: by 10.236.184.131 with SMTP id s3mr34824885yhm.32.1328649706651; Tue, 07 Feb 2012 13:21:46 -0800 (PST) Received: by 10.236.50.82 with HTTP; Tue, 7 Feb 2012 13:21:46 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Feb 2012 21:21:46 +0000 Message-ID: From: krad To: Patrick Dorion Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS boots RO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 21:21:47 -0000 On 7 February 2012 15:56, Patrick Dorion wrote: > Hi, > > I've been having a problem for some time now. When I reboot, the root ZFS > mounts readonly=on source=temporary. As a result, it's unable to launch > the services and the buffer is rather uninformative about the condition. > > I thought it was a problem in the configuration but I destroyed the zpool > before installing 9.0-R and I tried to reboot with and without loader.conf, > rc.conf and fstab configuration parameters for the filesystem. The only > non-default setting for the filesystem is utf8only=on and I never had an > issue with that in the past. > > Any ideas? > > > youe > Patrick Dorion > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > have you got similar in your loader.conf? vfs.root.mountfrom="zfs:system-4k/be/root20120104" vfs.root.mountfrom.options=rw From owner-freebsd-fs@FreeBSD.ORG Tue Feb 7 23:10:13 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87F691065696 for ; Tue, 7 Feb 2012 23:10:13 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 773DC8FC0C for ; Tue, 7 Feb 2012 23:10:13 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q17NAD31052231 for ; Tue, 7 Feb 2012 23:10:13 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q17NAD0h052230; Tue, 7 Feb 2012 23:10:13 GMT (envelope-from gnats) Date: Tue, 7 Feb 2012 23:10:13 GMT Message-Id: <201202072310.q17NAD0h052230@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Yar Tikhiy Cc: Subject: Re: bin/145309: bsdlabel: Editing disk label invalidates the whole device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Yar Tikhiy List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Feb 2012 23:10:13 -0000 The following reply was made to PR bin/145309; it has been noted by GNATS. From: Yar Tikhiy To: "Andrey V. Elsukov" Cc: Pawel Jakub Dawidek , bug-followup@freebsd.org Subject: Re: bin/145309: bsdlabel: Editing disk label invalidates the whole device Date: Wed, 8 Feb 2012 10:08:54 +1100 Hi Andrey, 2012/2/6 Andrey V. Elsukov : > On 04.02.2012 7:50, Yar Tikhiy wrote: >> >> =A0Sorry but FreeBSD 9.0-RELEASE still appears to have this issue. =A0Wh= en >> =A0installed using BSD label partitioning scheme, a modification to >> =A0ada0's label seems to nuke the kernel's view of the disk -- I can't >> =A0think of a better way to explain it. =A0The disk itself is OK and the >> =A0change makes it OK to the disk but the kernel can no more use the roo= t >> =A0partition until rebooted, returning weird errnos such as EIO or EXIO. >> =A0No idea here if the bug is limited to BSD label scheme. > > When you are in single user mode your root filesystem is mounted read-onl= y. > When you run bsdlabel it opens geom provider for writing and this trigger= s spoiling for it. > When bsdlabel closes provider GEOM_PART destroys it and creates again. > But VFS code seems loses it. Sorry but do you think it's intended behavior or not? It doesn't look so to me and, IMMSMR, it wasn't there before. Please correct me if I'm wrong. Thanks, Yar From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 00:28:25 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 074BD1065679 for ; Wed, 8 Feb 2012 00:28:25 +0000 (UTC) (envelope-from dorionpatrick@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id D34AA8FC0A for ; Wed, 8 Feb 2012 00:28:24 +0000 (UTC) Received: by daec6 with SMTP id c6so9390dae.13 for ; Tue, 07 Feb 2012 16:28:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BS3ZRDyKMrNQr41tmxmfFfGmIlZIzMigS7oMaAdE1Pk=; b=gzIguguHXPFBz7Di9BGegH64R/5YSy0zhIkAyMKcMYC/xQbkfEV5T+nOTj64fCSTMU mMPbNUe+5HXrbJRX5M+ptEtrCu1SYzhWs59b5iG0em42jiUTeY18epd7VmxnxnfZevXd Fy+kgu3B9f/rMfsR6Brt1PdNETlsmBHTlmWVw= MIME-Version: 1.0 Received: by 10.68.74.41 with SMTP id q9mr43266382pbv.92.1328660904372; Tue, 07 Feb 2012 16:28:24 -0800 (PST) Received: by 10.68.221.231 with HTTP; Tue, 7 Feb 2012 16:28:24 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Feb 2012 19:28:24 -0500 Message-ID: From: Patrick Dorion To: krad Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS boots RO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 00:28:25 -0000 There were no options but now the kernel's output shows: Root mounting from zfs:zroot [rw]... Though it remains read only. On Tue, Feb 7, 2012 at 4:21 PM, krad wrote: > > > On 7 February 2012 15:56, Patrick Dorion wrote: > >> Hi, >> >> I've been having a problem for some time now. When I reboot, the root ZFS >> mounts readonly=on source=temporary. As a result, it's unable to launch >> the services and the buffer is rather uninformative about the condition. >> >> I thought it was a problem in the configuration but I destroyed the zpool >> before installing 9.0-R and I tried to reboot with and without >> loader.conf, >> rc.conf and fstab configuration parameters for the filesystem. The only >> non-default setting for the filesystem is utf8only=on and I never had an >> issue with that in the past. >> >> Any ideas? >> >> >> youe >> Patrick Dorion >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > > have you got similar in your loader.conf? > > vfs.root.mountfrom="zfs:system-4k/be/root20120104" > vfs.root.mountfrom.options=rw > From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 01:52:30 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A1AE1065670; Wed, 8 Feb 2012 01:52:30 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 009328FC08; Wed, 8 Feb 2012 01:52:30 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q181qTHu011573; Wed, 8 Feb 2012 01:52:29 GMT (envelope-from eadler@freefall.freebsd.org) Received: (from eadler@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q181qTPr011569; Wed, 8 Feb 2012 01:52:29 GMT (envelope-from eadler) Date: Wed, 8 Feb 2012 01:52:29 GMT Message-Id: <201202080152.q181qTPr011569@freefall.freebsd.org> To: gcooper@FreeBSD.org, eadler@FreeBSD.org, freebsd-fs@FreeBSD.org From: eadler@FreeBSD.org Cc: Subject: Re: kern/117314: [ntfs] Long-filename only NTFS fs'es cause kernel panics on read X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 01:52:30 -0000 Synopsis: [ntfs] Long-filename only NTFS fs'es cause kernel panics on read State-Changed-From-To: feedback->closed State-Changed-By: eadler State-Changed-When: Wed Feb 8 01:52:29 UTC 2012 State-Changed-Why: this appears to be fixed. If this is still a problem mail me and I'll reopen it. http://www.freebsd.org/cgi/query-pr.cgi?pr=117314 From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 07:28:20 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B970F106566B for ; Wed, 8 Feb 2012 07:28:20 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.10]) by mx1.freebsd.org (Postfix) with ESMTP id 48DFE8FC0A for ; Wed, 8 Feb 2012 07:28:20 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mrbap2) with ESMTP (Nemesis) id 0MZ776-1SG0DO3Uzc-00LdWG; Wed, 08 Feb 2012 08:28:18 +0100 Message-ID: <4F322411.9020003@brockmann-consult.de> Date: Wed, 08 Feb 2012 08:28:17 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:qi4U3UyLftjGEqMOkrUzs/VsT2nakuvLU9AezOqfqc1 nwXMJTogA61kWXaVdlcw22mqDTYgZ7wNrGNQD7dUZGPFHAahDB 2cEKPDYowweJ3tMAJ0MV9hZDROVJBMNGd2MNcOad8k59fVx49m 4jzX/TYAR93Ivt5meCi7nIBnc7xmkrOPNxewyCkIpjL6WJjolG rIFU328O7NXqn0E0N/rAfUK/KxKwDswGlp3NUhNrkTRNHHBQ27 jqZZZ1GIQu24NQY16TM5c3mlzciJRWDwcdHH/COPyKk1Ee8ZBs I4TDNYDGoCSsspWngJU6q0IwtdIepUXtA8Bmt8cZ93oXKw/ooB Gz/Lb3wGzp+0k0LQVfyu4UgdE/5yrgS2xd3i9Kqyf Subject: Re: ZFS boots RO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 07:28:20 -0000 On 02/08/2012 01:28 AM, Patrick Dorion wrote: > There were no options but now the kernel's output shows: > > Root mounting from zfs:zroot [rw]... > > Though it remains read only. Are you sure the pool is read only, or only the datasets? # zpool get readonly examplepool NAME PROPERTY VALUE SOURCE tank readonly off - # zfs get readonly examplepool NAME PROPERTY VALUE SOURCE tank readonly off default Try this: zpool set readonly=off rootpool zfs set readonly=off rootpool mount -a zfs mount -a mount -o rw,remount rootpool/ > > On Tue, Feb 7, 2012 at 4:21 PM, krad wrote: > >> >> On 7 February 2012 15:56, Patrick Dorion wrote: >> >>> Hi, >>> >>> I've been having a problem for some time now. When I reboot, the root ZFS >>> mounts readonly=on source=temporary. As a result, it's unable to launch >>> the services and the buffer is rather uninformative about the condition. >>> >>> I thought it was a problem in the configuration but I destroyed the zpool >>> before installing 9.0-R and I tried to reboot with and without >>> loader.conf, >>> rc.conf and fstab configuration parameters for the filesystem. The only >>> non-default setting for the filesystem is utf8only=on and I never had an >>> issue with that in the past. >>> >>> Any ideas? >>> >>> >>> youe >>> Patrick Dorion >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> have you got similar in your loader.conf? >> >> vfs.root.mountfrom="zfs:system-4k/be/root20120104" >> vfs.root.mountfrom.options=rw >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de -------------------------------------------- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 07:31:52 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E2E1106564A; Wed, 8 Feb 2012 07:31:52 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from mail.kirov.so-ups.ru (mail.kirov.so-ups.ru [178.74.170.1]) by mx1.freebsd.org (Postfix) with ESMTP id 1BA108FC17; Wed, 8 Feb 2012 07:31:51 +0000 (UTC) Received: from kas30pipe.localhost (localhost.kirov.so-ups.ru [127.0.0.1]) by mail.kirov.so-ups.ru (Postfix) with SMTP id 2EE42B8071; Wed, 8 Feb 2012 11:31:50 +0400 (MSK) Received: from kirov.so-ups.ru (unknown [172.21.81.1]) by mail.kirov.so-ups.ru (Postfix) with ESMTP id 29329B8070; Wed, 8 Feb 2012 11:31:50 +0400 (MSK) Received: by ns.kirov.so-ups.ru (Postfix, from userid 1010) id 244BCB8F6C; Wed, 8 Feb 2012 11:31:50 +0400 (MSK) Received: from [127.0.0.1] (elsukov.kirov.oduur.so [10.118.3.52]) by ns.kirov.so-ups.ru (Postfix) with ESMTP id CD54FB8F45; Wed, 8 Feb 2012 11:31:49 +0400 (MSK) Message-ID: <4F3224E5.10406@FreeBSD.org> Date: Wed, 08 Feb 2012 11:31:49 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Yar Tikhiy References: <201202072310.q17NAD0h052230@freefall.freebsd.org> In-Reply-To: <201202072310.q17NAD0h052230@freefall.freebsd.org> X-Enigmail-Version: 1.3.4 Content-Type: multipart/mixed; boundary="------------030908090907000300080105" X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0284], KAS30/Release X-SpamTest-Info: Not protected Cc: freebsd-fs@FreeBSD.org, Marcel Moolenaar , Pawel Jakub Dawidek Subject: Re: bin/145309: bsdlabel: Editing disk label invalidates the whole device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 07:31:52 -0000 This is a multi-part message in MIME format. --------------030908090907000300080105 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit On 08.02.2012 3:10, Yar Tikhiy wrote: > > When you are in single user mode your root filesystem is mounted read-onl= > y. > > When you run bsdlabel it opens geom provider for writing and this trigger= > s spoiling for it. > > When bsdlabel closes provider GEOM_PART destroys it and creates again. > > But VFS code seems loses it. > > Sorry but do you think it's intended behavior or not? It doesn't look > so to me and, IMMSMR, it wasn't there before. Please correct me if > I'm wrong. GEOM_BSD class uses g_slice interface to implement partitions. It grabs an extra exclusive access bit when provider is opened first time. This grants him protection from spoiling. GEOM_PART class does this only when provider is opened for writing. We can try to change this behavior or just don't use bsdlabel :) Changes may lead to unexpected problems. If you want to test you can try attached patch (untested). -- WBR, Andrey V. Elsukov --------------030908090907000300080105 Content-Type: text/plain; name="g_part.c.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="g_part.c.diff" Index: head/sys/geom/part/g_part.c =================================================================== --- head/sys/geom/part/g_part.c (revision 231127) +++ head/sys/geom/part/g_part.c (working copy) @@ -1948,8 +1948,13 @@ g_part_access(struct g_provider *pp, int dr, int d cp = LIST_FIRST(&pp->geom->consumer); - /* We always gain write-exclusive access. */ - return (g_access(cp, dr, dw, dw + de)); + /* On first open, grab an extra "exclusive" bit */ + if (cp->acr == 0 && cp->acw == 0 && cp->ace == 0) + de++; + /* ... and let go of it on last close */ + if ((cp->acr + dr) == 0 && (cp->acw + dw) == 0 && (cp->ace + de) == 1) + de--; + return (g_access(cp, dr, dw, de)); } static void --------------030908090907000300080105-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 09:40:29 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id D6D011065679; Wed, 8 Feb 2012 09:40:29 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-150-251.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 20DE114F0E0; Wed, 8 Feb 2012 09:40:24 +0000 (UTC) Message-ID: <4F324308.6020008@FreeBSD.org> Date: Wed, 08 Feb 2012 01:40:24 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:10.0) Gecko/20120201 Thunderbird/10.0 MIME-Version: 1.0 To: Martin Matuska References: <4F2F9FCD.40700@FreeBSD.org> In-Reply-To: <4F2F9FCD.40700@FreeBSD.org> X-Enigmail-Version: 1.3.5 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: [CFR][DEVFS] rc.conf option devfs_load_rulesets X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 09:40:29 -0000 On 02/06/2012 01:39, Martin Matuska wrote: > FreeBSD includes four system rulesets in /etc/defaults/devfs.rules and > allows users to configure their custom rulesets in /etc/devfs.rules. > However, if not using jails or not specifying at least one of the > "devfs_system_ruleset" or "devfs_set_rulesets" variables, there is no > way to automatically load the rules from these configuration files. > > The attached patch introduces a "devfs_load_rulesets" yes/no variable, > that allows the user to have the devfs rules always loaded on startup or > if manually running /etc/rc.d/devfs start. > > The patch is also available at: > http://people.freebsd.org/~mm/patches/devfs/devfs_load_rulesets.patch > > Please review and/or comment my attached patch. I actually ran into this same problem myself today, and I suppose GMTA because I did essentially the same thing ... even gave the knob the same name. In no way did I intend to steal your work, sorry. For future reference, if you end a line with an operator (like || or &&) there is no reason to use the \ for line continuation, sh sorts it out for you. Doug -- It's always a long day; 86400 doesn't fit into a short. Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 10:06:36 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB0DF106564A for ; Wed, 8 Feb 2012 10:06:36 +0000 (UTC) (envelope-from prvs=1385245108=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 704B08FC08 for ; Wed, 8 Feb 2012 10:06:36 +0000 (UTC) X-Spam-Processed: mail1.multiplay.co.uk, Wed, 08 Feb 2012 09:55:58 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017940175.msg for ; Wed, 08 Feb 2012 09:55:57 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1385245108=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@FreeBSD.ORG Message-ID: From: "Steven Hartland" To: Date: Wed, 8 Feb 2012 09:56:57 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Subject: zfs fs dependencies causing boot failure X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 10:06:37 -0000 Is there any current way to ensure that zfs mounts are processed before entries in fstab? In our case here we have some legacy ufs fs hosted on zvol's which are mounted to points within the standard zfs fs e.g. /dev/zvol/tank/data/legacy /data/jails/legacy ufs rw 2 2 To our horror this currently prevents the machine from booting so would be good if there was some way to prevent this? Looking around there seems to have been something similar developed /etc/rc.d/zvol but this only deals with swap it seems. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 18:06:03 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22F9A1065676 for ; Wed, 8 Feb 2012 18:06:03 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id CDCFA8FC19 for ; Wed, 8 Feb 2012 18:06:02 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id 264F0418; Wed, 8 Feb 2012 19:06:01 +0100 (CET) Date: Wed, 8 Feb 2012 19:04:47 +0100 From: Pawel Jakub Dawidek To: Steven Hartland Message-ID: <20120208180446.GA1336@garage.freebsd.pl> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WIyZ46R2i8wDzkSu" Content-Disposition: inline In-Reply-To: X-OS: FreeBSD 10.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@FreeBSD.ORG Subject: Re: zfs fs dependencies causing boot failure X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 18:06:03 -0000 --WIyZ46R2i8wDzkSu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 08, 2012 at 09:56:57AM -0000, Steven Hartland wrote: > Is there any current way to ensure that zfs mounts are processed > before entries in fstab? >=20 > In our case here we have some legacy ufs fs hosted on zvol's which > are mounted to points within the standard zfs fs e.g. > /dev/zvol/tank/data/legacy /data/jails/legacy ufs rw 2 2 >=20 > To our horror this currently prevents the machine from booting so > would be good if there was some way to prevent this? >=20 > Looking around there seems to have been something similar developed > /etc/rc.d/zvol but this only deals with swap it seems. Try adding 'late' keyword to the options field for you UFS file system in /etc/fstab. Such file system should be mounted by rc.d/mountlate scripts which comes after rc.d/zfs. For example: /dev/zvol/tank/data/legacy /data/jails/legacy ufs rw,late 2 2 --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://tupytaj.pl --WIyZ46R2i8wDzkSu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk8yuT4ACgkQForvXbEpPzTmDQCgvtNDMqf5/Yer/0tbk9X9Lw3v +PwAmwYLWhse8fIezbsq4+gojv5Op/7p =DZ3e -----END PGP SIGNATURE----- --WIyZ46R2i8wDzkSu-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 19:18:43 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B1B71065672 for ; Wed, 8 Feb 2012 19:18:43 +0000 (UTC) (envelope-from nester@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3B4A48FC0C for ; Wed, 8 Feb 2012 19:18:42 +0000 (UTC) Received: by qaea17 with SMTP id a17so701736qae.13 for ; Wed, 08 Feb 2012 11:18:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=date:from:to:subject:message-id:reply-to:mime-version:content-type :content-disposition:user-agent; bh=/D46MCZaW+1VicYoM8qsDkQo207vyVUU92xJAUfqbQw=; b=gDuUny04BEk7WV169AGfI39+47JNG1zh6M4V4Eo/ZDoxpmx+Fsw7Rz6BzFgMt6P9B0 Qo5q6GX+IQYzHzOxQFC0U42ViUPH+8Xu2yuNj6eTO0UB5/yNambzQoRf7GKqWyM7SDOa C0VFzj9bFioW5tgWpovvQbjTnQLVvyFNF0kW0= Received: by 10.224.195.2 with SMTP id ea2mr28102425qab.84.1328726860901; Wed, 08 Feb 2012 10:47:40 -0800 (PST) Received: from localhost (pool-108-18-105-239.washdc.fios.verizon.net. [108.18.105.239]) by mx.google.com with ESMTPS id el3sm296999qab.8.2012.02.08.10.47.39 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 08 Feb 2012 10:47:40 -0800 (PST) Date: Wed, 8 Feb 2012 13:47:28 -0500 From: Tom Vier To: freebsd-fs@FreeBSD.ORG Message-ID: <20120208184728.GH6814@zero> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Subject: zfs checksums on non-raidz X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Tom Vier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 19:18:43 -0000 In mirror and single dev zpools, are data and metadata still checksummed, or only when using raidz? I've only been able to determine so far that the uberblocks at the zpool level are always checksummed (according to Sun's on-disk format pdf). -- Tom Vier DSA Key ID 0x15741ECE From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 19:26:34 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4550106564A for ; Wed, 8 Feb 2012 19:26:34 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 84DE78FC0A for ; Wed, 8 Feb 2012 19:26:34 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q18JQXmY023154; Wed, 8 Feb 2012 13:26:33 -0600 (CST) Date: Wed, 8 Feb 2012 13:26:33 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Tom Vier In-Reply-To: <20120208184728.GH6814@zero> Message-ID: References: <20120208184728.GH6814@zero> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Wed, 08 Feb 2012 13:26:33 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: zfs checksums on non-raidz X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 19:26:34 -0000 On Wed, 8 Feb 2012, Tom Vier wrote: > In mirror and single dev zpools, are data and metadata still checksummed, or > only when using raidz? Checksums are done at the decoded/reassembled filesystem block level (e.g. 128K) so they are performed regardless of the underlying storage topology or compression used. Checksums are still in effect even if there is no data redundancy. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 19:28:20 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12E11106564A for ; Wed, 8 Feb 2012 19:28:20 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smtp-out0.tiscali.nl (smtp-out0.tiscali.nl [195.241.79.175]) by mx1.freebsd.org (Postfix) with ESMTP id C779E8FC14 for ; Wed, 8 Feb 2012 19:28:19 +0000 (UTC) Received: from [212.182.167.131] (helo=sjakie.klop.ws) by smtp-out0.tiscali.nl with esmtp (Exim) (envelope-from ) id 1RvCqJ-000805-44; Wed, 08 Feb 2012 20:06:11 +0100 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id AF6BA1006C; Wed, 8 Feb 2012 20:05:53 +0100 (CET) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, =?utf-8?Q?Peter_Ankerst=C3=A5l?= References: <4F2FF72B.6000509@pean.org> Date: Wed, 08 Feb 2012 20:05:53 +0100 MIME-Version: 1.0 From: "Ronald Klop" Message-ID: In-Reply-To: <4F2FF72B.6000509@pean.org> User-Agent: Opera Mail/11.61 (FreeBSD) Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 19:28:20 -0000 On Mon, 06 Feb 2012 16:52:11 +0100, Peter Ankerst=C3=A5l = wrote: > Hi, > > I want to investigate if it is possible to create your own usable HPC =20 > storage using zfs and some > network filesystem like nfs. > > Just a thought experiment.. > A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) > I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD =20 > deives for cache. > Preferrably in mirror where applicable. > > Connected to this machine we will have about 410 3TB drives to give =20 > approx > 1PB of usable storage in a 8+2 raidz configuration. > > Connected to this will be a ~800 nodes big HPC cluster that will access= =20 > the storage in parallell > is this even possible or do we need to distribute the meta data load =20 > over many servers? If that is the case, > does it exist any software for FreeBSD that could accomplish this =20 > distribution (pNFS dosent seem to be > anywhere close to usable in FreeBSD) or do I need to call NetApp or =20 > Panasas right away? It would be > really nice if I could build my own storage solution. > > Other possible solutions to this problem is extremley welcome. > > Best Regards > Peter Ankerst=C3=A5l You might make a call to Backblaze. http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-c= heap-cloud-storage/ Ronald. From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 19:29:47 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9828106564A for ; Wed, 8 Feb 2012 19:29:47 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from mo-p05-ob6.rzone.de (mo-p05-ob6.rzone.de [IPv6:2a01:238:20a:202:53f5::1]) by mx1.freebsd.org (Postfix) with ESMTP id 250B88FC0A for ; Wed, 8 Feb 2012 19:29:46 +0000 (UTC) X-RZG-AUTH: :LWIKdA2leu0bPbLmhzXgqn0MTG6qiKEwQRWfNxSw4HzYIwjsnvdDt2QV8d370mKouLGgFPM= X-RZG-CLASS-ID: mo05 Received: from [192.168.179.39] (hmbg-4d069da4.pool.mediaWays.net [77.6.157.164]) by smtp.strato.de (klopstock mo7) (RZmta 27.6 DYNA|AUTH) with (DHE-RSA-AES128-SHA encrypted) ESMTPA id 60207eo18I4Fa7 for ; Wed, 8 Feb 2012 20:29:26 +0100 (MET) Message-ID: <4F32CD14.1030508@brockmann-consult.de> Date: Wed, 08 Feb 2012 20:29:24 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <20120208184728.GH6814@zero> In-Reply-To: <20120208184728.GH6814@zero> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: zfs checksums on non-raidz X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 19:29:47 -0000 Am 08.02.2012 19:47, schrieb Tom Vier: > In mirror and single dev zpools, are data and metadata still checksummed, or > only when using raidz? > > I've only been able to determine so far that the uberblocks at the zpool level > are always checksummed (according to Sun's on-disk format pdf). > Everything, even a single disk, is checksummed. Test it with: zpool create testpool gpt/testslice cp /some/big/test/file /testpool/ # next command is dangerous because it means you can now directly write to disks and damage the data sysctl kern.geom.debugflags=0x10 # next command is more dangerous, because it intentionally destroys data on the slice (be careful about what you put in of=... ) dd if=/dev/zero of=/dev/gpt/testslice bs=1M count=5000 # after this command, you should have some checksum errors md5 /testpool/file # after this command and waiting, you should have all the errors there are (all files, including not read by the md5 above) zpool scrub testpool ...wait... zpool status testpool Now you probably (depending on whether or not the part we wrote with dd actually went on top of where data was placed) have checksum errors, and a message about an unrecoverable file. If it was raidz or mirror, it would have automatically fixed it for you, but with no redundancy, all the checksum can do is tell you the file is broken. From owner-freebsd-fs@FreeBSD.ORG Wed Feb 8 19:31:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C95C106578E for ; Wed, 8 Feb 2012 19:31:40 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from mo-p05-ob6.rzone.de (mo-p05-ob6.rzone.de [IPv6:2a01:238:20a:202:53f5::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8906D8FC17 for ; Wed, 8 Feb 2012 19:31:39 +0000 (UTC) X-RZG-AUTH: :LWIKdA2leu0bPbLmhzXgqn0MTG6qiKEwQRWfNxSw4HzYIwjsnvdDt2QV8d370mKouLGgFPM= X-RZG-CLASS-ID: mo05 Received: from [192.168.179.39] (hmbg-4d069da4.pool.mediaWays.net [77.6.157.164]) by smtp.strato.de (fruni mo41) (RZmta 27.6 DYNA|AUTH) with (DHE-RSA-AES128-SHA encrypted) ESMTPA id i04a38o18IkjaI for ; Wed, 8 Feb 2012 20:31:16 +0100 (MET) Message-ID: <4F32CD83.6000101@brockmann-consult.de> Date: Wed, 08 Feb 2012 20:31:15 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <20120208184728.GH6814@zero> <4F32CD14.1030508@brockmann-consult.de> In-Reply-To: <4F32CD14.1030508@brockmann-consult.de> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: zfs checksums on non-raidz X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 19:31:40 -0000 ps. Try what I showed you in a vm rather than a real system, to keep your data safe. Am 08.02.2012 20:29, schrieb Peter Maloney: > Am 08.02.2012 19:47, schrieb Tom Vier: >> In mirror and single dev zpools, are data and metadata still checksummed, or >> only when using raidz? >> >> I've only been able to determine so far that the uberblocks at the zpool level >> are always checksummed (according to Sun's on-disk format pdf). >> > Everything, even a single disk, is checksummed. > > Test it with: > > > zpool create testpool gpt/testslice > cp /some/big/test/file /testpool/ > > # next command is dangerous because it means you can now directly write > to disks and damage the data > sysctl kern.geom.debugflags=0x10 > > # next command is more dangerous, because it intentionally destroys data > on the slice (be careful about what you put in of=... ) > dd if=/dev/zero of=/dev/gpt/testslice bs=1M count=5000 > > # after this command, you should have some checksum errors > md5 /testpool/file > > # after this command and waiting, you should have all the errors there > are (all files, including not read by the md5 above) > zpool scrub testpool > > ...wait... > > zpool status testpool > > Now you probably (depending on whether or not the part we wrote with dd > actually went on top of where data was placed) have checksum errors, and > a message about an unrecoverable file. If it was raidz or mirror, it > would have automatically fixed it for you, but with no redundancy, all > the checksum can do is tell you the file is broken. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Feb 9 13:29:52 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62CE31065670 for ; Thu, 9 Feb 2012 13:29:52 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.186]) by mx1.freebsd.org (Postfix) with ESMTP id E4FFE8FC13 for ; Thu, 9 Feb 2012 13:29:51 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mreu3) with ESMTP (Nemesis) id 0Mfeu7-1S9zqn1ED5-00Omyy; Thu, 09 Feb 2012 14:29:50 +0100 Message-ID: <4F33CA4D.6060400@brockmann-consult.de> Date: Thu, 09 Feb 2012 14:29:49 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: freebsd-fs@freebsd.org X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:jiWBvVJchfwvi5iPwvfloJuItVFPOJgaFVCcDoxs8t1 aAnDgCxkA7ost1etPScOeIN6uVcFO3x1mC7DABE+RYgo57ilra 9RqYFQ1AWXgN+ooFpHEKe1/m4VwYwWgpUofL1fR2FQboOEj3ep Xd+DbdPxOksSFhIPiejTgzwHHMTs5jzfA6SGDSDVgikawYrh3B NZGmnEITPC8iRNH++0AKXwp0myUgsWrth50GLeJthsHoyalANg /8HFksJrcmjU3AJuowTcndJMkXS6rl2QqD9uBvLRn7uLxlsoUm EIK/2qEUnAjWXknauZdCeIn9dz4TofzsUMtpHp1uxxkw5hfB58 tBRTjt0FuGWXByA9GLH+QJUwiVZY7dJJn6ekdOGHX Subject: zfs snapdir NFS hang X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 13:29:52 -0000 So, I have an issue where after some point (arbitrary number of snapshots? a specific snapshot? gremlins?), exporting a directory that contains a .zfs directory, whether or not snapdir=hidden is set, then listing the directory in the NFS client causes a total hang of the dataset and some commands like "zdb -d poolname". I don't know the root cause, so I don't know how to reproduce it, or create a PR. eg. |# echo /tank/dataset -maproot=root 10.10.10.10 >> /etc/exports| |# kill -HUP `cat /var/run/mountd.pid`| |# tail /var/log/messages| Code: Feb 8 15:47:54 bcnas1 mountd[46760]: can't delete exports for /tank/dataset/.zfs/snapshot/replication-20120204134001: Invalid argument Feb 8 15:47:54 bcnas1 mountd[46760]: can't delete exports for /tank/dataset/.zfs/snapshot/replication-20120208140000: Invalid argument ... (I would think the above shows that the NFS server is not really compatible with this situation / buggy) |# ssh 10.10.10.10 "mount bcnas1:/tank/dataset /mountpoint ; ls /mountpoint/.zfs/snapshot"| (hang on this command, and anything else after this poing using the same dataset) There was a point when this would not hang, but instead just show many directories, and then for many other snapshots (all of the ones listed in the /var/log/messages errors and more), there would be strange binary files, or directories with the wrong files in them (it would show me a subdirectory of the correct root of the snapshot). All of these problems happen whether or not I set snapdir=hidden or snapdir=visible. I am currently running 8-STABLE from Sept. 28th. Today an identical problem happened, and I rebooted to fix it. I don't know if it was the same cause, but I would like to find out. Can someone give me ideas of how to track the problem, or tell me which source files I should open up in /usr/src, hack apart or add debugging and either: * find the root cause of the problem * prevent NFS from exporting any .zfs directories Or does someone know if this has been fixed in the latest 8-STABLE or 9? The best workaround I can think of is reorganizing all my datasets so the root directory is empty except one directory, and then share only that subdirectory which does not contain a .zfs directory (or any other child datasets). But ideally, nfs clients should be able to view snapshots to recover files. From owner-freebsd-fs@FreeBSD.ORG Thu Feb 9 15:42:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB4C31065675 for ; Thu, 9 Feb 2012 15:42:12 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id 4D5358FC19 for ; Thu, 9 Feb 2012 15:42:12 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mreu3) with ESMTP (Nemesis) id 0Ld8Qn-1SLT6J39dZ-00iEe1; Thu, 09 Feb 2012 16:42:11 +0100 Message-ID: <4F33E952.80102@brockmann-consult.de> Date: Thu, 09 Feb 2012 16:42:10 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F33CA4D.6060400@brockmann-consult.de> In-Reply-To: <4F33CA4D.6060400@brockmann-consult.de> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:n++W0t6aoGuOC50Yyuz4CeT50SLViH1PIL59B98FS+F 0R1UkWftG0hj4VZCI5L86sfOOIel+3uhyN3uPsh3EWgQiX4PCw zz+zX7GEnXA6eksKurxZuNPbq5qXDR6b/SPXqwwqZJlclnEACl f9ZGXc/GnmjSVXuWSJ9EYYUehWOcWMHnm8ewbc+nF3eXhK7AOg +JhsQdPQoaY6K0ezkky2SgwQqJMQSplruahlm1dKJXIkMffVsP jEg5Q5IdP5G4HuqisJFYRN1aFjme8un7aTlbiDMy7RwHtmkF28 n33V6oy4YQniprzY6AgKmlQt6s7YkHGoDfXTej613GtY2qO7AC hukx9ZPHnnfzt45nB5rXfV+g7z81Hbn5O5V+0iMmt Subject: Re: zfs snapdir NFS hang X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 15:42:13 -0000 I just tested it with a FreeBSD NFS client, and it all looks correct. So I guess only the Linux client triggers this strange behavior (wrong directories, files instead of directories, etc.). But the /var/log/messages error messages clearly show the server is messed up, not (only?) the client. And obviously the hang can't be blamed on the client. linuxclient # uname -a Linux peter 2.6.38-12-generic #51-Ubuntu SMP Wed Sep 28 14:27:32 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux freebsdclient # uname -a FreeBSD bczfsvm1.bc.local 8.2-STABLE-20120104 FreeBSD 8.2-STABLE-20120104 #0: Mon Feb 6 12:10:32 UTC 2012 root@bczfsvm1.bc.local:/usr/obj/usr/src/sys/GENERIC amd64 And for the record, I have tested full scans of the .zfs/snapshot directory using "find" to see if it "brings the server to its knees" as is often said, but even that does not cause any problems at all (with 48 GB of memory on this machine). I am currently running another run of that test, and will tell you the result in 20 or so hours when it is done. # time find /tank/bcnasvm1/.zfs/snapshot -type f > /dev/null 2>&1 On 02/09/2012 02:29 PM, Peter Maloney wrote: > So, I have an issue where after some point (arbitrary number of > snapshots? a specific snapshot? gremlins?), exporting a directory that > contains a .zfs directory, whether or not snapdir=hidden is set, then > listing the directory in the NFS client causes a total hang of the > dataset and some commands like "zdb -d poolname". I don't know the root > cause, so I don't know how to reproduce it, or create a PR. > > eg. > |# echo /tank/dataset -maproot=root 10.10.10.10 >> /etc/exports| > |# kill -HUP `cat /var/run/mountd.pid`| > |# tail /var/log/messages| > Code: > > Feb 8 15:47:54 bcnas1 mountd[46760]: can't delete exports for /tank/dataset/.zfs/snapshot/replication-20120204134001: Invalid argument > Feb 8 15:47:54 bcnas1 mountd[46760]: can't delete exports for /tank/dataset/.zfs/snapshot/replication-20120208140000: Invalid argument > ... > > (I would think the above shows that the NFS server is not really > compatible with this situation / buggy) > |# ssh 10.10.10.10 "mount bcnas1:/tank/dataset /mountpoint ; ls > /mountpoint/.zfs/snapshot"| > (hang on this command, and anything else after this poing using the same > dataset) > > There was a point when this would not hang, but instead just show many > directories, and then for many other snapshots (all of the ones listed > in the /var/log/messages errors and more), there would be strange binary > files, or directories with the wrong files in them (it would show me a > subdirectory of the correct root of the snapshot). > > All of these problems happen whether or not I set snapdir=hidden or > snapdir=visible. > > I am currently running 8-STABLE from Sept. 28th. > > Today an identical problem happened, and I rebooted to fix it. I don't > know if it was the same cause, but I would like to find out. > > Can someone give me ideas of how to track the problem, or tell me which > source files I should open up in /usr/src, hack apart or add debugging > and either: > > * find the root cause of the problem > * prevent NFS from exporting any .zfs directories > > Or does someone know if this has been fixed in the latest 8-STABLE or 9? > > The best workaround I can think of is reorganizing all my datasets so > the root directory is empty except one directory, and then share only > that subdirectory which does not contain a .zfs directory (or any other > child datasets). But ideally, nfs clients should be able to view > snapshots to recover files. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de -------------------------------------------- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 9 23:16:38 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37712106566C; Thu, 9 Feb 2012 23:16:38 +0000 (UTC) (envelope-from feld@feld.me) Received: from mwi1.coffeenet.org (unknown [IPv6:2607:f4e0:100:300::2]) by mx1.freebsd.org (Postfix) with ESMTP id 040108FC13; Thu, 9 Feb 2012 23:16:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me; s=blargle; h=Message-Id:From:Content-Transfer-Encoding:Mime-Version:Subject:Date:To:Content-Type; bh=V6j1Fq0QQIeVT/1s/AP6rysAf+asGdjvnAcWRKxtysQ=; b=pQ7NURNzGwc6shhctRKruVAlxGLSSigRlz7Ilb7RaoS+kzimds8qDGtXsF4cPFWTUgiNTeLZO9uzJqvpTNhsTMPCanN8DdWm9I23y0Lw6aY85mfN5Z7fko//opSmP0CJ; Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org) by mwi1.coffeenet.org with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1RvdEB-0001cV-H2; Thu, 09 Feb 2012 17:16:37 -0600 Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4) with esmtpsa id 1328829389-3359-3358/5/14; Thu, 9 Feb 2012 23:16:29 +0000 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Date: Thu, 9 Feb 2012 17:16:29 -0600 Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable From: Mark Felder Message-Id: User-Agent: Opera Mail/12.00 (FreeBSD) X-SA-Score: -1.0 Cc: Subject: zfs commit breaks zvol, istgt? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 23:16:38 -0000 Hi all, Previous kernel I was running on a test SAN was 9-STABLE from Jan 24th. =20 Sorry, no commit # -- didn't have svn on the machine back then. Today I built r231282 because it had an interesting fix in it: r231141 | mm | 2012-02-07 11:57:33 -0600 (Tue, 07 Feb 2012) | 25 lines MFC r230514: Merge illumos revisions 13572, 13573, 13574: Rev. 13572: disk sync write perf regression when slog is used post oi_148 [1] Rev. 13573: crash during reguid causes stale config [2] allow and unallow missing from zpool history since removal of pyzfs [5] Rev. 13574: leaking a vdev when removing an l2cache device [3] memory leak when adding a file-based l2arc device [4] leak in ZFS from metaslab_group_create and zfs_ereport_checksum [6] References: https://www.illumos.org/issues/1909 [1] https://www.illumos.org/issues/1949 [2] https://www.illumos.org/issues/1951 [3] https://www.illumos.org/issues/1952 [4] https://www.illumos.org/issues/1953 [5] https://www.illumos.org/issues/1954 [6] Obtained from: illumos (issues #1909, #1949, #1951, #1952, #1953, #1954) After booting into this kernel iSCSI was hosed. None of the ESXi servers = =20 looking at it could do any I/O at all. Weird errors in the istgt log, = too: Feb 9 16:26:23 zfs-san2 istgt[8177]: Login from =20 iqn.1998-01.com.vmware:esx1-21ecbe81 (172.16.17.41) on =20 iqn.2011-12.net.supranet.san2.istgt:lun7 LU7 (172.16.17.182:3260,1), =20 ISID=3D23d000002, TSIH=3D4, CID=3D0, HeaderDigest=3Doff, DataDigest=3Doff Feb 9 16:26:23 zfs-san2 istgt[8177]: istgt_iscsi.c: =20 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed =20 (errno=3D32) Feb 9 16:26:23 zfs-san2 istgt[8177]: istgt_iscsi.c:4984:sender: =20 ***ERROR*** iscsi_write_pdu() failed on =20 iqn.2011-12.net.supranet.san2.istgt:lun7,t,0x0001(iqn.1998-01.com.vmware:= esx1-21ecbe81,i,0x00023d000002) I didn't see any other commits between that could cause this, but can =20 anyone else confirm? After rebooting into the Jan 24th kernel everything = =20 went back to normal... Thanks, Mark zfs-san2# zfs list NAME USED AVAIL REFER MOUNTPOINT tank 8.25T 7.76T 1.05M /tank tank/LUN1 1.03T 8.77T 24.7G - tank/LUN2 1.03T 8.77T 19.7G - tank/LUN3 1.03T 8.70T 93.6G - tank/LUN4 1.03T 8.77T 19.3G - tank/LUN5 1.03T 8.79T 44.1K - tank/LUN6 1.03T 8.79T 44.1K - tank/LUN7 1.03T 8.79T 44.1K - tank/LUN8 1.03T 8.79T 44.1K - tank/nfs 2.30G 7.76T 2.30G /tank/nfs zfs-san2# zpool status pool: tank state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Thu Jan 26 17:05:40 = 2012 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 multipath/disk01 ONLINE 0 0 0 multipath/disk02 ONLINE 0 0 0 multipath/disk03 ONLINE 0 0 0 multipath/disk04 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 multipath/disk05 ONLINE 0 0 0 multipath/disk06 ONLINE 0 0 0 multipath/disk07 ONLINE 0 0 0 multipath/disk08 ONLINE 0 0 0 raidz1-2 ONLINE 0 0 0 multipath/disk09 ONLINE 0 0 0 multipath/disk10 ONLINE 0 0 0 multipath/disk11 ONLINE 0 0 0 multipath/disk12 ONLINE 0 0 0 logs da1 ONLINE 0 0 0 cache da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 errors: No known data errors From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 10:19:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A066106564A for ; Fri, 10 Feb 2012 10:19:31 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id B3A688FC14 for ; Fri, 10 Feb 2012 10:19:30 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mrbap4) with ESMTP (Nemesis) id 0LnShi-1SS6zH34RS-00gxgJ; Fri, 10 Feb 2012 11:19:29 +0100 Message-ID: <4F34EF30.1030706@brockmann-consult.de> Date: Fri, 10 Feb 2012 11:19:28 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F33CA4D.6060400@brockmann-consult.de> <4F33E952.80102@brockmann-consult.de> In-Reply-To: <4F33E952.80102@brockmann-consult.de> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:S1CoNA657bsX0vJiE80q9BUsNlMv2i7qwDQMsPZgujT RqZeIIhtG8LMy3cVnwZCrTcochMmtw/oRt/fIW7KekGOu4CRLW Eqgs/xrAxwgpjjwETIwDyajct6TL1fli/JMcaVyYLWOQ8mKyOz bIjVzW6Pot5Xa/YkZFM7AktWmmEdIjA6REG3+nDeKCuI93xwr4 Rd6VucfQfnNHxH3IiFEnKmnRmuJna5OgBsUxUSZIjUR7X7ga7a UCcmghqIKTYDSCyZkLUXHOP3Mu1GMGkynK7hkIkk0bI+P18ruE 6+JxAJylTlN408FsqfwxBZtwJ2ReARwcquiDHG5CLRPF5cnzUZ 1VaujUksVqO99sFBUg0DyWeC80XbGS3Z46iOx8+Qz Subject: Re: zfs snapdir NFS hang X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 10:19:31 -0000 On 02/09/2012 04:42 PM, Peter Maloney wrote: > ... > And for the record, I have tested full scans of the .zfs/snapshot > directory using "find" to see if it "brings the server to its knees" as > is often said, but even that does not cause any problems at all (with 48 > GB of memory on this machine). I am currently running another run of > that test, and will tell you the result in 20 or so hours when it is done. > > # time find /tank/bcnasvm1/.zfs/snapshot -type f > /dev/null 2>&1 The result was that the system seemed fine, for a while, but after fiddling with NFS and snapshots, eventually every new process (eg. a local login on the machine terminals) would be killed with a message about being out of swap space. I guess it can only eat all your memory when you have more data or snapshots, because fun stuff like this didn't happen before. Or perhaps I didn't catch it before, because I didn't try fiddling with other things the last time I tested it. From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 11:30:21 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BFAB106564A for ; Fri, 10 Feb 2012 11:30:21 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 05F578FC0C for ; Fri, 10 Feb 2012 11:30:21 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q1ABUK1Y044753 for ; Fri, 10 Feb 2012 11:30:20 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q1ABUK7S044750; Fri, 10 Feb 2012 11:30:20 GMT (envelope-from gnats) Date: Fri, 10 Feb 2012 11:30:20 GMT Message-Id: <201202101130.q1ABUK7S044750@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Peter Maloney Cc: Subject: Re: kern/161968: [zfs] [hang] renaming snapshot with -r including a zvol snapshot causes total ZFS freeze/lockup X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Peter Maloney List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 11:30:21 -0000 The following reply was made to PR kern/161968; it has been noted by GNATS. From: Peter Maloney To: bug-followup@FreeBSD.org, peter.maloney@brockmann-consult.de Cc: Subject: Re: kern/161968: [zfs] [hang] renaming snapshot with -r including a zvol snapshot causes total ZFS freeze/lockup Date: Fri, 10 Feb 2012 12:11:41 +0100 I tested this again using 8-STABLE (csup'd on 2012-01-04): FreeBSD bczfsvm1.bc.local 8.2-STABLE-20120104 FreeBSD 8.2-STABLE-20120104 #0: Mon Feb 6 12:10:32 UTC 2012 root@bczfsvm1.bc.local:/usr/obj/usr/src/sys/GENERIC amd64 on hardware: DELL PowerEdge 2850 - tested with a zfs stripe, raidz1, and raidz2 and a SuperMicro dual xeon system - tested with a zfs mirror And it didn't hang. Now there are just brief pauses every 3-5 loops (instead of hangs?). So if someone tests this in 9.0-STABLE and finds that it doesn't hang, this PR should be closed. -- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de -------------------------------------------- From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 11:54:45 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3ABFD106564A; Fri, 10 Feb 2012 11:54:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 62D0C8FC0C; Fri, 10 Feb 2012 11:54:43 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA16156; Fri, 10 Feb 2012 13:54:33 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <4F350579.7020109@FreeBSD.org> Date: Fri, 10 Feb 2012 13:54:33 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120206 Thunderbird/10.0 MIME-Version: 1.0 To: Mark Felder References: In-Reply-To: X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: zfs commit breaks zvol, istgt? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 11:54:45 -0000 on 10/02/2012 01:16 Mark Felder said the following: > Hi all, > > Previous kernel I was running on a test SAN was 9-STABLE from Jan 24th. Sorry, no > commit # -- didn't have svn on the machine back then. > > Today I built r231282 because it had an interesting fix in it: > > r231141 | mm | 2012-02-07 11:57:33 -0600 (Tue, 07 Feb 2012) | 25 lines > > MFC r230514: > Merge illumos revisions 13572, 13573, 13574: > > Rev. 13572: > disk sync write perf regression when slog is used post oi_148 [1] > > Rev. 13573: > crash during reguid causes stale config [2] > allow and unallow missing from zpool history since removal of pyzfs [5] > > Rev. 13574: > leaking a vdev when removing an l2cache device [3] > memory leak when adding a file-based l2arc device [4] > leak in ZFS from metaslab_group_create and zfs_ereport_checksum [6] > > References: > https://www.illumos.org/issues/1909 [1] > https://www.illumos.org/issues/1949 [2] > https://www.illumos.org/issues/1951 [3] > https://www.illumos.org/issues/1952 [4] > https://www.illumos.org/issues/1953 [5] > https://www.illumos.org/issues/1954 [6] > > Obtained from: illumos (issues #1909, #1949, #1951, #1952, #1953, #1954) > > > After booting into this kernel iSCSI was hosed. None of the ESXi servers looking > at it could do any I/O at all. Weird errors in the istgt log, too: > > Feb 9 16:26:23 zfs-san2 istgt[8177]: Login from > iqn.1998-01.com.vmware:esx1-21ecbe81 (172.16.17.41) on > iqn.2011-12.net.supranet.san2.istgt:lun7 LU7 (172.16.17.182:3260,1), > ISID=23d000002, TSIH=4, CID=0, HeaderDigest=off, DataDigest=off > Feb 9 16:26:23 zfs-san2 istgt[8177]: istgt_iscsi.c: > 777:istgt_iscsi_write_pdu_internal: ***ERROR*** iscsi_write() failed (errno=32) Are you positive that this breakage is ZFS related? BTW, errno 32 is EPIPE. > Feb 9 16:26:23 zfs-san2 istgt[8177]: istgt_iscsi.c:4984:sender: ***ERROR*** > iscsi_write_pdu() failed on > iqn.2011-12.net.supranet.san2.istgt:lun7,t,0x0001(iqn.1998-01.com.vmware:esx1-21ecbe81,i,0x00023d000002) > > > > > I didn't see any other commits between that could cause this, but can anyone else > confirm? After rebooting into the Jan 24th kernel everything went back to normal... > > > > Thanks, > > > > Mark > > > zfs-san2# zfs list > NAME USED AVAIL REFER MOUNTPOINT > tank 8.25T 7.76T 1.05M /tank > tank/LUN1 1.03T 8.77T 24.7G - > tank/LUN2 1.03T 8.77T 19.7G - > tank/LUN3 1.03T 8.70T 93.6G - > tank/LUN4 1.03T 8.77T 19.3G - > tank/LUN5 1.03T 8.79T 44.1K - > tank/LUN6 1.03T 8.79T 44.1K - > tank/LUN7 1.03T 8.79T 44.1K - > tank/LUN8 1.03T 8.79T 44.1K - > tank/nfs 2.30G 7.76T 2.30G /tank/nfs > > > > zfs-san2# zpool status > pool: tank > state: ONLINE > scan: scrub repaired 0 in 0h0m with 0 errors on Thu Jan 26 17:05:40 2012 > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > multipath/disk01 ONLINE 0 0 0 > multipath/disk02 ONLINE 0 0 0 > multipath/disk03 ONLINE 0 0 0 > multipath/disk04 ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > multipath/disk05 ONLINE 0 0 0 > multipath/disk06 ONLINE 0 0 0 > multipath/disk07 ONLINE 0 0 0 > multipath/disk08 ONLINE 0 0 0 > raidz1-2 ONLINE 0 0 0 > multipath/disk09 ONLINE 0 0 0 > multipath/disk10 ONLINE 0 0 0 > multipath/disk11 ONLINE 0 0 0 > multipath/disk12 ONLINE 0 0 0 > logs > da1 ONLINE 0 0 0 > cache > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > > errors: No known data errors > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 13:55:28 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F9C0106566B for ; Fri, 10 Feb 2012 13:55:28 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from mx0.hoeg.nl (mx0.hoeg.nl [IPv6:2a01:4f8:101:5343::aa]) by mx1.freebsd.org (Postfix) with ESMTP id E11108FC08 for ; Fri, 10 Feb 2012 13:55:27 +0000 (UTC) Received: by mx0.hoeg.nl (Postfix, from userid 1000) id 25B9E2A28CCE; Fri, 10 Feb 2012 14:55:27 +0100 (CET) Date: Fri, 10 Feb 2012 14:55:27 +0100 From: Ed Schouten To: fs@freebsd.org Message-ID: <20120210135527.GR1860@hoeg.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JViz224v3YRbOSfM" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Subject: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 13:55:28 -0000 --JViz224v3YRbOSfM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi all, It seems the default timestamp precision sysctl (vfs.timestamp_precision) is currently set to 0 by default, meaning we don't do any sub-second timestamps on files. Looking at the code, it seems that vfs.timestamp_precision=3D1 will let it use a cached value with 1 / HZ precision and it looks like it should have little overhead. Would anyone object if I were to change the default from 0 to 1? --=20 Ed Schouten WWW: http://80386.nl/ --JViz224v3YRbOSfM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iQIcBAEBAgAGBQJPNSHPAAoJEG5e2P40kaK7jmQP+wXjcKunhXA/kuBJH53bUzXt rAr3mojSf2DxLd+ldJxmbuylszwG6b06IZVK8I7nbQQVZMoZa1x1YhRmnN86Fv9j Sy0a1O+7LF1ddFClIKVasO1ULW+BbaRNTIEjhGqTiXJ4W96HluGSCx2jnaZHCvCE 86vb0QPzBq7Nbfb+n+uMT0JbIy9+z21zoH7TbWTFB9kQoTlX5l0VB7uFOSknvHKQ 3WTwZij90DaF7PfSX/YaksI/FTsG92Pqgm4VRKtzHgzJmRn8+kSvoCu7pEtXM08R +m1jbMQSrPNSueiqMIVNdxridfUiKyjae+gAg+7r3s7RjaiJ/8vSxfJLul3wk5WK FzZhp0CoG+Tetp0310qHummN2MyFtqEXaF5hUSTeK1YWp6m/8CQz3RGHkS7nQD1Q 3VUlcTU66MMRt63MPvswKoIutmzPSufW4zB93F1z9pv5jVdjXfNnieU3rRAGOx7v FfOARp6Vzs5eIUcqFKvWO5x4ob2Jellw/QOSWT2bcF9kKoX3WJdJMo9eCTPP26Py YF77pSCX1WaPTZuecaXFeRpk35QKnqr4VZgBhJ59JqSGooOlSwyUkuID3xsAju+u J/Tgi23KybqHdpZrGkmkmDNtrR96ygBjZKRR93P5UceIIsJ8dPy5Qwk8W9X4KJnJ fLmJz7yK8RTWGrfwqceJ =RWu+ -----END PGP SIGNATURE----- --JViz224v3YRbOSfM-- From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 13:57:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 975B9106566B; Fri, 10 Feb 2012 13:57:04 +0000 (UTC) (envelope-from feld@feld.me) Received: from mwi1.coffeenet.org (unknown [IPv6:2607:f4e0:100:300::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6C9298FC12; Fri, 10 Feb 2012 13:57:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me; s=blargle; h=In-Reply-To:Message-Id:From:Mime-Version:Date:References:Subject:To:Content-Type; bh=16GEyaZ0EdthBBs9AnfegiEPACkaNqMkXoyOWq3I7RE=; b=k+hUKEGRK/+KjEW/PttlAUWcJlNfrZocv+R5u1Qz2n7wZfe+LxDSyxrnOGmK2wTnSBpX5ZG+48NBhFg+ipU+W0gWwONikGuft6Zbpu3+90rgqddAXeyTxcdm+yRV0fgO; Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org) by mwi1.coffeenet.org with esmtp (Exim 4.77 (FreeBSD)) (envelope-from ) id 1RvqyF-000Ow6-A3; Fri, 10 Feb 2012 07:57:03 -0600 Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4) with esmtpsa id 1328882216-3359-3358/5/15; Fri, 10 Feb 2012 13:56:56 +0000 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, freebsd-current@freebsd.org References: <4F350579.7020109@FreeBSD.org> Date: Fri, 10 Feb 2012 07:56:56 -0600 Mime-Version: 1.0 From: Mark Felder Message-Id: In-Reply-To: <4F350579.7020109@FreeBSD.org> User-Agent: Opera Mail/12.00 (FreeBSD) X-SA-Score: -1.5 Cc: Subject: Re: zfs commit breaks zvol, istgt? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 13:57:04 -0000 I think I may have jumped the gun on this guys -- I'm seeing this behavior again. I've got a few more tests in mind now that I need to perform before I can really nail down what's going on. From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 16:46:30 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 333951065689 for ; Fri, 10 Feb 2012 16:46:30 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id AE57A8FC13 for ; Fri, 10 Feb 2012 16:46:29 +0000 (UTC) Received: by eaan10 with SMTP id n10so1144724eaa.13 for ; Fri, 10 Feb 2012 08:46:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=QnR+9zbYQ/3sWBtvlEfM+ucUHrf4QqQ93o9h9TZu+3I=; b=A5WUcHRyGEGmZCoMZnCvGKSSrGnbjeMIo1TFhBWvdFb+GLAm1Ge2HNB+fBTfSIxFcc Uj3WRdjh3hXFG6sJT2UOaidqk8LJG5MfkVDq8yd0oeNT47QWNGUJbBaBHwsUK5HJwbUK /aFwtGVufjSWvWrvFgQKgaZgItEQ4A3jkVTjk= Received: by 10.213.13.209 with SMTP id d17mr1226839eba.20.1328891026334; Fri, 10 Feb 2012 08:23:46 -0800 (PST) Received: from ernst.jennejohn.org (p578E2245.dip.t-dialin.net. [87.142.34.69]) by mx.google.com with ESMTPS id a58sm23685285eeb.8.2012.02.10.08.23.43 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 10 Feb 2012 08:23:45 -0800 (PST) Date: Fri, 10 Feb 2012 17:23:42 +0100 From: Gary Jennejohn To: Ed Schouten Message-ID: <20120210172342.7974cc32@ernst.jennejohn.org> In-Reply-To: <20120210135527.GR1860@hoeg.nl> References: <20120210135527.GR1860@hoeg.nl> X-Mailer: Claws Mail 3.8.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: fs@freebsd.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 16:46:30 -0000 On Fri, 10 Feb 2012 14:55:27 +0100 Ed Schouten wrote: > It seems the default timestamp precision sysctl > (vfs.timestamp_precision) is currently set to 0 by default, meaning we > don't do any sub-second timestamps on files. Looking at the code, it > seems that vfs.timestamp_precision=1 will let it use a cached value with > 1 / HZ precision and it looks like it should have little overhead. > > Would anyone object if I were to change the default from 0 to 1? > Is this only visible in the kernel? I don't see any difference in the output of ls -lT whether the sysctl is set to 0 or 1. -- Gary Jennejohn From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 16:51:23 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60F5C106566C for ; Fri, 10 Feb 2012 16:51:23 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from mx0.hoeg.nl (mx0.hoeg.nl [IPv6:2a01:4f8:101:5343::aa]) by mx1.freebsd.org (Postfix) with ESMTP id EED5D8FC12 for ; Fri, 10 Feb 2012 16:51:22 +0000 (UTC) Received: by mx0.hoeg.nl (Postfix, from userid 1000) id 07BA52A28CCE; Fri, 10 Feb 2012 17:51:22 +0100 (CET) Date: Fri, 10 Feb 2012 17:51:22 +0100 From: Ed Schouten To: Gary Jennejohn Message-ID: <20120210165122.GT1860@hoeg.nl> References: <20120210135527.GR1860@hoeg.nl> <20120210172342.7974cc32@ernst.jennejohn.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jk5HhkA9Kov5Oxsi" Content-Disposition: inline In-Reply-To: <20120210172342.7974cc32@ernst.jennejohn.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: fs@freebsd.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 16:51:23 -0000 --jk5HhkA9Kov5Oxsi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Gary, * Gary Jennejohn , 20120210 17:23: > Is this only visible in the kernel? I don't see any difference in the > output of ls -lT whether the sysctl is set to 0 or 1. It is visible for userspace, but I guess not for ls(1). You can test it by doing: echo foo > foo echo bar > bar diff -u foo bar And it prints the header at the top with the raw timestamps. It's not really useful for a regular user to see the nanoseconds, but for tools like make(1) and others that depend on timestamps to do decision making, it would be nice to have. --=20 Ed Schouten WWW: http://80386.nl/ --jk5HhkA9Kov5Oxsi Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iQIcBAEBAgAGBQJPNUsJAAoJEG5e2P40kaK7j9YQAId2XvJhCprq8eIbxa1pVuqo JnJOEiwdyci6ZAKhwvCCyAM4ZkARhnirp/v9kWy7rQWwMfojgxglC64L2SbpLaBO EDPgRQ1ORrvi8lcR83vtbXldOSDTCBG+i/R8RoEfEw6WXCx5RITdPRhLspG8TqfU fYOcXkPjq96iNyoXfmaKXSim4MgGHzrFp1Ulc43reS2Ly7hGYO8AHNWKKbLgtedl DKvbD0i3fN1Rn+LLDvBhmM8N9iScarKJgsCgSGKmwJevduxgr1uojhmXfPBwiTO/ 5RKnyOSUdRxmMOnssgaVkSKwu4I7m5FvA1NF4dq5c5TZdSSB8aESCcoLiFAp4LaR 2SlCLp5+p50DGv/pqNcQBuDdBPSH9SklXczZdwtbRLkwJhumFZvUONJ4zWotO/rl F1l2XD8pIM6vIJ/tBCpymemnnBl9h0KZcWHyKTgd2G09B/qi+f7yOG3kHNwAshrm TJNbrsPG7CvmQlyeWZY1vykaq6SpX/ZcSliEoODe60ix0hPU30/pMISVSIkwqVNe CXOmHuOMgJSzvDT6621bDQsCMv1Qid3p2IyuBuJDPM7Qs76j0/OqnDkYjJncIyvp Ep9XJZlCeRA79EKBfutIXZq21Jp+H4rXbz+N6/JJRHbWUyzyCL+fMyxBoHDLaIUj QN3W1X8ZkcQ1R+Cf3kSG =iM8N -----END PGP SIGNATURE----- --jk5HhkA9Kov5Oxsi-- From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 17:00:59 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10BAA1065672 for ; Fri, 10 Feb 2012 17:00:59 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-lpp01m020-f182.google.com (mail-lpp01m020-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 88A0B8FC28 for ; Fri, 10 Feb 2012 17:00:58 +0000 (UTC) Received: by lbbgj3 with SMTP id gj3so2180875lbb.13 for ; Fri, 10 Feb 2012 09:00:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=sNESFrb0qhyo4b1j8ulKCXBdbklCk1857SJ0Wo62P7M=; b=IQBk5Z4yfsnSi+cC3oSgyDo7IvKWGcY2Oc+CujozNvZOFNh3+irLKpzHGi3St+a42m CZRWdbp/iBN8qDut0P39bycg1kV2SZDHn45NyBPg17konsY8RlGYhkf7eTkrO9XgpN/G NnWJ9Uz0xP5AlAmSIrU5WOkIVEsiCA7MJhKHI= MIME-Version: 1.0 Received: by 10.152.110.102 with SMTP id hz6mr5050482lab.21.1328891851599; Fri, 10 Feb 2012 08:37:31 -0800 (PST) Received: by 10.152.18.4 with HTTP; Fri, 10 Feb 2012 08:37:31 -0800 (PST) In-Reply-To: <20120210135527.GR1860@hoeg.nl> References: <20120210135527.GR1860@hoeg.nl> Date: Fri, 10 Feb 2012 19:37:31 +0300 Message-ID: From: Sergey Kandaurov To: Ed Schouten Content-Type: text/plain; charset=ISO-8859-1 Cc: fs@freebsd.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 17:00:59 -0000 On 10 February 2012 17:55, Ed Schouten wrote: > Hi all, > > It seems the default timestamp precision sysctl > (vfs.timestamp_precision) is currently set to 0 by default, meaning we > don't do any sub-second timestamps on files. Looking at the code, it > seems that vfs.timestamp_precision=1 will let it use a cached value with > 1 / HZ precision and it looks like it should have little overhead. > > Would anyone object if I were to change the default from 0 to 1? > [Yep, sorry I didn't read this mail before replying to your another mail.] I am for this idea. Increasing vfs.timestamp_precision will allow to use nanosecond precision for all those *stat() and *times() syscalls which operate on struct timespec. FWIW, NetBSD uses only nanotime() inside vfs_timestamp() since its initial appearance in 2006. -- wbr, pluknet From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 17:58:52 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C9DA1065673; Fri, 10 Feb 2012 17:58:52 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0E8DA8FC0C; Fri, 10 Feb 2012 17:58:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q1AHwpsx008144; Fri, 10 Feb 2012 17:58:51 GMT (envelope-from mckusick@freefall.freebsd.org) Received: (from mckusick@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q1AHwpec008140; Fri, 10 Feb 2012 17:58:51 GMT (envelope-from mckusick) Date: Fri, 10 Feb 2012 17:58:51 GMT Message-Id: <201202101758.q1AHwpec008140@freefall.freebsd.org> To: mckusick@FreeBSD.org, freebsd-fs@FreeBSD.org, mckusick@FreeBSD.org From: mckusick@FreeBSD.org Cc: Subject: Re: kern/159971: [ffs] [panic] panic with soft updates journaling during load testing X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 17:58:52 -0000 Synopsis: [ffs] [panic] panic with soft updates journaling during load testing Responsible-Changed-From-To: freebsd-fs->mckusick Responsible-Changed-By: mckusick Responsible-Changed-When: Fri Feb 10 17:58:19 UTC 2012 Responsible-Changed-Why: I will take responsibility for this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=159971 From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 20:07:42 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 235401065672; Fri, 10 Feb 2012 20:07:42 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EA47E8FC08; Fri, 10 Feb 2012 20:07:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q1AK7fOa026427; Fri, 10 Feb 2012 20:07:41 GMT (envelope-from mckusick@freefall.freebsd.org) Received: (from mckusick@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q1AK7faL026423; Fri, 10 Feb 2012 20:07:41 GMT (envelope-from mckusick) Date: Fri, 10 Feb 2012 20:07:41 GMT Message-Id: <201202102007.q1AK7faL026423@freefall.freebsd.org> To: mckusick@FreeBSD.org, freebsd-fs@FreeBSD.org, freebsd-bugs@FreeBSD.org From: mckusick@FreeBSD.org Cc: Subject: Re: kern/158711: [usb] [panic] panic in ffs_blkfree and ffs_valloc X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 20:07:42 -0000 Synopsis: [usb] [panic] panic in ffs_blkfree and ffs_valloc Responsible-Changed-From-To: freebsd-fs->freebsd-bugs Responsible-Changed-By: mckusick Responsible-Changed-When: Fri Feb 10 20:06:52 UTC 2012 Responsible-Changed-Why: While looking through old bugs classified to the filesystem maintainers I came across this one. The panic's experienced here came about because of failed updates to the filesystem metadata that apparently were not reported back as errors to the filesystem. In any event, the bug lies in the USB subsystem and not in the filesystem, so this bug should be redirected to the USB maintainers. It may well have been fixed by now, but if not should be investigated by them. http://www.freebsd.org/cgi/query-pr.cgi?pr=158711 From owner-freebsd-fs@FreeBSD.ORG Fri Feb 10 21:14:56 2012 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB3521065672 for ; Fri, 10 Feb 2012 21:14:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id C47448FC16 for ; Fri, 10 Feb 2012 21:14:55 +0000 (UTC) Received: from mail02.syd.optusnet.com.au (mail02.syd.optusnet.com.au [211.29.132.183]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q1AIPcjK005127 for ; Sat, 11 Feb 2012 05:25:38 +1100 Received: from c211-30-171-136.carlnfd1.nsw.optusnet.com.au (c211-30-171-136.carlnfd1.nsw.optusnet.com.au [211.30.171.136]) by mail02.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q1AIPSPZ007395 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 11 Feb 2012 05:25:29 +1100 Date: Sat, 11 Feb 2012 05:25:28 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Sergey Kandaurov In-Reply-To: Message-ID: <20120211042121.B3653@besplex.bde.org> References: <20120210135527.GR1860@hoeg.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Ed Schouten , fs@FreeBSD.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 21:14:56 -0000 On Fri, 10 Feb 2012, Sergey Kandaurov wrote: > On 10 February 2012 17:55, Ed Schouten wrote: >> Hi all, >> >> It seems the default timestamp precision sysctl >> (vfs.timestamp_precision) is currently set to 0 by default, meaning we >> don't do any sub-second timestamps on files. Looking at the code, it >> seems that vfs.timestamp_precision=1 will let it use a cached value with >> 1 / HZ precision and it looks like it should have little overhead. >> >> Would anyone object if I were to change the default from 0 to 1? Sure. The setting of 1 is too buggy to use in its current implementation. I don't know of any fixed version, so there is little experience with a usable version. I also wouldn't use getnanotime() for anything. But I like nanotime(). % void % vfs_timestamp(struct timespec *tsp) % { % struct timeval tv; % % switch (timestamp_precision) { % case TSP_SEC: % tsp->tv_sec = time_second; % tsp->tv_nsec = 0; This gives seconds precision. It is correct. % break; % case TSP_HZ: % getnanotime(tsp); % break; I must have been asleep when I reviewed this for jdp in 1999. This doesn't give 1/HZ precision. It gives nanoseconds precision with garbage in the low bits, and about 1/HZ accuracy. To fix it, round down to 1/HZ precision, or at least to microseconds precision. The garbage in the low bits matters mainly because there is no way to preserve it. utimes(2) only supports microseconds precision. % case TSP_USEC: % microtime(&tv); % TIMEVAL_TO_TIMESPEC(&tv, tsp); % break; This gives microseconds precision, but in a silly way. It should call nanotime() and then round down to microseconds precision. % case TSP_NSEC: % default: The default should be an error. % nanotime(tsp); % break; % } % } I mostly use TSP_SEC, but there are some buggy file systems and/or utilities (cvsup?, or perhaps just scp from a system using a different timestamp precision) that produce sub-seconds precision. I notice this when I veryify timestamps in backups. The backup formats support microseconds precision at best, so any extra precision in the files in active file systems gives a verification failure. > [Yep, sorry I didn't read this mail before replying to your another mail.] > > I am for this idea. Increasing vfs.timestamp_precision will allow > to use nanosecond precision for all those *stat() and *times() > syscalls which operate on struct timespec. > > FWIW, NetBSD uses only nanotime() inside vfs_timestamp() since its > initial appearance in 2006. Does NetBSD's nanotime() have full nsec precision and hardware slowness? There is no hardware yet that can deliver anywhere near nsec accuracy, so the precision might as well be limited to usec. i8254 timecounters take/took 5-50 usec just to read. ACPI-fast is relatively worse on today's faster CPUs (1-2 usec). The non-serializing TSC used to take only ~10 instructions on Athlons, but it is non-serializing and was always non-P-state-invariant. P-state-invariant versions take much longer (seems to be about 50 cycles in hardware and another 50 in software for core2), and TSC-low intentionally wastes about 7 low bits, so its precision is about 64 nsec which is about the same time as nanotime() takes to read it. I use TSP_NSEC only for POSIX conformance tests, to break the tests finding of the bug that even TSP_SEC is broken. It is broken because time_second is incoherent with the time(3). time_second and the time reported by all the get*time() functions lags the time reported by the (non-get)*time(), and the lag is random relative to seconds (and other) boundaries, so rounding to a seconds (or other) boundary gives different results. These differences are visible to applications doing tests like: touch(file); stat(file); sleep(1); touch(file); stat(file); assert(file_mtime_increased_y_at_least_1_second); This should also show the leap seconds bug in POSIX times (the file time shouldn't change across a leap second). Some of the tests do lots of file timestamp changing (I also have to turn off my usual optimization of mounting with noatime to get them to pass). They run fast enough even with TSP_NSEC, at least if the timecounter is a fast TSC. File time updates just don't happen enough for their speed to matter much, provided they are cached enough. ffs uses the mark-for-update caching strategy which works well. It avoids not only writing to disk, but even reading the timer a lot. Some other filesystems like devfs are not careful about this, so the slowness of silly operations like dd with a block size of 1 on /dev/zero to /dev/null becmes even more extreme if TSP_NSEC or TSP_USEC is used and the timecounter is slow. Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Feb 11 02:10:35 2012 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76C1C106564A for ; Sat, 11 Feb 2012 02:10:35 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail08.syd.optusnet.com.au (mail08.syd.optusnet.com.au [211.29.132.189]) by mx1.freebsd.org (Postfix) with ESMTP id 148078FC0C for ; Sat, 11 Feb 2012 02:10:34 +0000 (UTC) Received: from c211-30-171-136.carlnfd1.nsw.optusnet.com.au (c211-30-171-136.carlnfd1.nsw.optusnet.com.au [211.30.171.136]) by mail08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q1B2AQpk031974 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 11 Feb 2012 13:10:27 +1100 Date: Sat, 11 Feb 2012 13:10:26 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans In-Reply-To: <20120211042121.B3653@besplex.bde.org> Message-ID: <20120211121947.M903@besplex.bde.org> References: <20120210135527.GR1860@hoeg.nl> <20120211042121.B3653@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Ed Schouten , fs@FreeBSD.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 02:10:35 -0000 PS: On Sat, 11 Feb 2012, Bruce Evans wrote: > On Fri, 10 Feb 2012, Sergey Kandaurov wrote: > >> On 10 February 2012 17:55, Ed Schouten wrote: >>> Hi all, >>> >>> It seems the default timestamp precision sysctl >>> (vfs.timestamp_precision) is currently set to 0 by default, meaning we >>> don't do any sub-second timestamps on files. Looking at the code, it >>> seems that vfs.timestamp_precision=1 will let it use a cached value with >>> 1 / HZ precision and it looks like it should have little overhead. >>> >>> Would anyone object if I were to change the default from 0 to 1? > > Sure. The setting of 1 is too buggy to use in its current implementation. > I don't know of any fixed version, so there is little experience with > a usable version. I also wouldn't use getnanotime() for anything. > But I like nanotime(). [ ... the bugs are that it gives nanoseconds precision with garbage in the low bits, and shares an old bug with a setting of 0 (TSP_SEC) ] Other bugs near here: - vfs.timestamp_precision is still global, so you can't change the default for individual mount points or even for individual vfs's. The global is only good enough for a quick hack. - touch(1) normally uses gettimeofday() and utimes(). Thus it normally gives microseconds precision and hopefully close to microseconds accuracy (if you are using ntpd). But when utimes() fails due to insufficient security and the user didn't specify a time, touch(1) falls back to using utimes() with a null timeptr, to ask the kernel to use the current time. The kernel uses vfs_timestamp() for this, so it sets times incompatibitly with touch(1) unless vfs.timestamp_precision=2 (TSP_USEC). The kernel used to use microtime() for this, but it was changed in 2009 to use vfs_timestamp(), with no reason given in its log message except a cryptic pointer to a Debian PR. The bug is mostly in touch(1). - the recently-removed fallback to setting times by writing in touch(1) gave another incompatible way of setting times. Timstamping bugs not so near here: - POSIX.1-1988 avoided the design error and unportability of timevals by inventing timespecs. It failed to invent an interfaces to set these. It only has utime(), which only sets times in seconds. - POSIX.1-2001 restored the design error of timevals, and utimes() to set them (but utimes() was marked as legacy). It still failed to invent an interface to set its own timespecs. - POSIX is apparently inventing futimesat(). This only sets timevals. POSIX is apparently still failing to invent an interface to set its own timespecs. Any utime()-like interface invented after timespecs should support timespecs, with setting timevals only supported by compatibility cruft in this interface. - FreeBSD also has birthtimes, but no way to set them. Any utime()-like interface invented after birthtimes should support them. FreeBSD's man page says essentially this. futimesat() is unfortunately being standardized, so it can't be this syscall. - in the man page, the times arg is declared as "*times" for everything except futimesat(). For the latter, it is declared as times[2]. This is a further regression in the spelling of the arg. utime(2) has a better API (struct instead of array) and a better spelling of the arg (timep). POSIX.1-2001 uses "times" for the utime() and times[2] for utimes(). - I also wish for a way to set ctimes. Without this, metadata cannot be restored from backups. I use a hack to restore ctimes by abusing utime*(). Bruce From owner-freebsd-fs@FreeBSD.ORG Sat Feb 11 07:20:30 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22EEA1065670 for ; Sat, 11 Feb 2012 07:20:30 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9A2788FC18 for ; Sat, 11 Feb 2012 07:20:29 +0000 (UTC) Received: by lagz14 with SMTP id z14so4374177lag.13 for ; Fri, 10 Feb 2012 23:20:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=uv0EutbvBHWoXJevs2st0BRq6RP2IHdo/W00IVEhagE=; b=eVlXxDIjNWaulqRNgQzQ2yOrGw+6Fcn+MBxJUSJghERgfB3PTnef91e2tO3giVqTJL lv1OMTkQCIF7tL2sIHNxrEKxBNprGaQFUYWUyB2ucvXwmCzyO3Nuswq7GlfJAJzbQV/n y3A8XglMzIzvamS9NZSswBFw0zOUQmxsMlKOQ= MIME-Version: 1.0 Received: by 10.152.122.74 with SMTP id lq10mr6830632lab.7.1328944828191; Fri, 10 Feb 2012 23:20:28 -0800 (PST) Received: by 10.152.18.4 with HTTP; Fri, 10 Feb 2012 23:20:28 -0800 (PST) In-Reply-To: <20120211121947.M903@besplex.bde.org> References: <20120210135527.GR1860@hoeg.nl> <20120211042121.B3653@besplex.bde.org> <20120211121947.M903@besplex.bde.org> Date: Sat, 11 Feb 2012 10:20:28 +0300 Message-ID: From: Sergey Kandaurov To: Bruce Evans Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Ed Schouten , fs@freebsd.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 07:20:30 -0000 On 11 February 2012 06:10, Bruce Evans wrote: > PS: > > > On Sat, 11 Feb 2012, Bruce Evans wrote: > >> On Fri, 10 Feb 2012, Sergey Kandaurov wrote: >> >>> On 10 February 2012 17:55, Ed Schouten wrote: >>>> >>>> Hi all, >>>> >>>> It seems the default timestamp precision sysctl >>>> (vfs.timestamp_precision) is currently set to 0 by default, meaning we >>>> don't do any sub-second timestamps on files. Looking at the code, it >>>> seems that vfs.timestamp_precision=3D1 will let it use a cached value = with >>>> 1 / HZ precision and it looks like it should have little overhead. >>>> >>>> Would anyone object if I were to change the default from 0 to 1? >> >> >> Sure. =A0The setting of 1 is too buggy to use in its current implementat= ion. >> I don't know of any fixed version, so there is little experience with >> a usable version. =A0I also wouldn't use getnanotime() for anything. >> But I like nanotime(). > > > [ ... the bugs are that it gives nanoseconds precision with garbage in th= e > low bits, and shares an old bug with a setting of 0 (TSP_SEC) ] > > Other bugs near here: > - vfs.timestamp_precision is still global, so you can't change the defaul= t > =A0for individual mount points or even for individual vfs's. =A0The globa= l > =A0is only good enough for a quick hack. Agreed. :-/ That is the main obstacle in my pov why it was not changed yet. > - touch(1) normally uses gettimeofday() and utimes(). =A0Thus it normally > =A0gives microseconds precision and hopefully close to microseconds > =A0accuracy (if you are using ntpd). =A0But when utimes() fails due to > =A0insufficient security and the user didn't specify a time, touch(1) > =A0falls back to using utimes() with a null timeptr, to ask the kernel > =A0to use the current time. =A0The kernel uses vfs_timestamp() for this, > =A0so it sets times incompatibitly with touch(1) unless > =A0vfs.timestamp_precision=3D2 (TSP_USEC). =A0The kernel used to use > =A0microtime() for this, but it was changed in 2009 to use > =A0vfs_timestamp(), with no reason given in its log message except a > =A0cryptic pointer to a Debian PR. =A0The bug is mostly in touch(1). > - the recently-removed fallback to setting times by writing in touch(1) > =A0gave another incompatible way of setting times. In POSIX-2008 touch description mentions now that touch shall perform actions equivalent to futimens() or utimensat() (see below) depending on whether the specified file exists. It also introduces the concept of finegrained timestamps - a fractional part of second like: touch -d "2007-11-12 10:15:30.002Z" -d is a new way to specify your own time but in ISO 8601:2000 format > Timstamping bugs not so near here: > - POSIX.1-1988 avoided the design error and unportability of timevals > =A0by inventing timespecs. =A0It failed to invent an interfaces to set > =A0these. =A0It only has utime(), which only sets times in seconds. > - POSIX.1-2001 restored the design error of timevals, and utimes() > =A0to set them (but utimes() was marked as legacy). =A0It still failed to > =A0invent an interface to set its own timespecs. > - POSIX is apparently inventing futimesat(). =A0This only sets timevals. > =A0POSIX is apparently still failing to invent an interface to set its > =A0own timespecs. =A0Any utime()-like interface invented after timespecs > =A0should support timespecs, with setting timevals only supported by > =A0compatibility cruft in this interface.[...] =A0futimesat() is unfortun= ately being > =A0standardized[..] POSIX.1-2008 invented two *utimes() syscalls since then which both operate on timespecs. I implemented them a year ago, but failed to commit yet for some reason, particularity because of overcomplicated utimes(2) man page. - futimens() is like our futimes() but works with timespec(); - utimensat() is like our futimesat(). It is "ours" because it was never standardized. Our manpage for futimesat() only mentions that it "follows The Open Group Extended API Set 2 specification." Linux manpage provides somewhat better history: This system call is nonstandard. It was implemented from a specification = that was proposed for POSIX.1, but that specification was replaced by the one f= or utimensat(2). utimensat() diverges from futimesat() by enabling usage of timespec and addition of the optional flag with only one existing bit that follows other *at() family convention: AT_SYMLINK_NOFOLLOW If path names a symbolic link, the symbolic link's dates are changed. --=20 wbr, pluknet From owner-freebsd-fs@FreeBSD.ORG Sat Feb 11 10:48:01 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C072106564A for ; Sat, 11 Feb 2012 10:48:01 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail01.syd.optusnet.com.au (mail01.syd.optusnet.com.au [211.29.132.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8ACBD8FC0A for ; Sat, 11 Feb 2012 10:48:00 +0000 (UTC) Received: from c211-30-171-136.carlnfd1.nsw.optusnet.com.au (c211-30-171-136.carlnfd1.nsw.optusnet.com.au [211.30.171.136]) by mail01.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q1BAlo3e022410 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 11 Feb 2012 21:47:51 +1100 Date: Sat, 11 Feb 2012 21:47:50 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Sergey Kandaurov In-Reply-To: Message-ID: <20120211204120.N2344@besplex.bde.org> References: <20120210135527.GR1860@hoeg.nl> <20120211042121.B3653@besplex.bde.org> <20120211121947.M903@besplex.bde.org> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-688338940-1328957270=:2344" Cc: Ed Schouten , fs@freebsd.org Subject: Re: Increase timestamp precision? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 10:48:01 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-688338940-1328957270=:2344 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Sat, 11 Feb 2012, Sergey Kandaurov wrote: > On 11 February 2012 06:10, Bruce Evans wrote: >> - touch(1) normally uses gettimeofday() and utimes(). =A0Thus it normall= y >> ... > > In POSIX-2008 touch description mentions now that touch shall perform > actions equivalent to futimens() or utimensat() (see below) depending > on whether the specified file exists. It also introduces the concept of > finegrained timestamps - a fractional part of second like: > touch -d "2007-11-12 10:15:30.002Z" > -d is a new way to specify your own time but in ISO 8601:2000 format I saw a little of this on the POSIX mailing list. POSIX has to specify what happens when unrepresentable times are requested to be set or used, not just for file times but also for timeouts. I'm not sure how far it has got. > >> Timstamping bugs not so near here: >> ... >> - POSIX is apparently inventing futimesat(). =A0This only sets timevals. >> =A0POSIX is apparently still failing to invent an interface to set its >> =A0own timespecs. =A0Any utime()-like interface invented after timespecs >> =A0should support timespecs, with setting timevals only supported by >> =A0compatibility cruft in this interface.[...] =A0futimesat() is unfortu= nately being >> =A0standardized[..] > > POSIX.1-2008 invented two *utimes() syscalls since then which both operat= e > on timespecs. > I implemented them a year ago, but failed to commit yet for some reason, > particularity because of overcomplicated utimes(2) man page. Do you mean that the FreeBSD man page is already overcomplicated, and that adding to it work make it more so? It certainly has long descrptions of the possible errors. The following seem to be wrong: % ERRORS % The utimes() and lutimes() system calls will fail if: % ... % [EPERM] The times argument is not NULL and the calling % process's effective user ID does not match the ow= ner % of the file and is not the super-user. %=20 % [EPERM] The named file has its immutable or append-only f= lag % set, see the chflags(2) manual page for more info= rma- % tion. Bad grammar (comma splice). This error can happen for almost all vfs syscalls, and man pages shouldn't be made even more verbose with pointers to chflags(2) for it. They aren't similarly verbose for other attributes, and the flags attribute is now 20 years old, so it should be as familiar as the others. %=20 % [EROFS] The file system containing the file is mounted re= ad- % only. These 3 should apply to all the utimes syscalls, since they are related to writing the results to the underlying file system's inode and not related to the pathname used to access the file. %=20 % In addition to the errors returned by the utimes(), the futimesat() = may % fail if: "may fail"? This weasel for "this standard is broken, so that broken but influential systems can conform to it". > - futimens() is like our futimes() but works with timespec(); > - utimensat() is like our futimesat(). It is "ours" because it was never > standardized. Our manpage for futimesat() only mentions that it "follows > The Open Group Extended API Set 2 specification." > Linux manpage provides somewhat better history: > This system call is nonstandard. It was implemented from a specification= that > was proposed for POSIX.1, but that specification was replaced by the one = for > utimensat(2). Hmm, I just noticed that at least old-POSIX doesn't have futimes() either. futimesns() wouldn't be missed so much in portable code that had to avoid futimes(). I wonder if FreeBSD doesn't need futimes(2) either, since it can try using utimes() on "/dev/stdin" after swapping the fd with 0 in case /dev/fd/ is not present. > utimensat() diverges from futimesat() by enabling usage of timespec and > addition of the optional flag with only one existing bit that follows oth= er > *at() family convention: > AT_SYMLINK_NOFOLLOW > If path names a symbolic link, the symbolic link's dates are > changed. Sounds like POSIX did the right thing, except for not implementing futimens(), but futimens() can easily be implemented as part of utimesnsat() by putting another flag somewhere to indicate that the fd is the thing to be changed and that the string is not used. Bruce --0-688338940-1328957270=:2344-- From owner-freebsd-fs@FreeBSD.ORG Sat Feb 11 11:56:21 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D05761065670 for ; Sat, 11 Feb 2012 11:56:21 +0000 (UTC) (envelope-from guyyur@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6D0F78FC1C for ; Sat, 11 Feb 2012 11:56:21 +0000 (UTC) Received: by werm13 with SMTP id m13so3847397wer.13 for ; Sat, 11 Feb 2012 03:56:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=xnZ6g3XpBXf+aQTFYFAZbUZqCw5KnpBZxBP5TrAG8Q4=; b=VbpmmzHX/HeApboEmYi3BmnzDHe+NqiygvmCebXi/Y6+pStTuuf+imHnJMz0wN97+P cCkvzhTDPcJT1Wewn/VDD9eqCuoXAQP+0eORchwrpjRB7GxC2V/Try6u5+ze55qaODRk aTl7doPx7f5UAM8E/q+b3GSvrbJ0oDmEs20fw= MIME-Version: 1.0 Received: by 10.216.131.91 with SMTP id l69mr2092250wei.28.1328959765929; Sat, 11 Feb 2012 03:29:25 -0800 (PST) Received: by 10.180.85.74 with HTTP; Sat, 11 Feb 2012 03:29:25 -0800 (PST) Date: Sat, 11 Feb 2012 13:29:25 +0200 Message-ID: From: Guy Yur To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: "mount -u -o ro" stuck on ufs mount with journaled soft-updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 11:56:21 -0000 Hi, When updating a mount to read-only for ufs with journaled soft-updates it can get stuck in ffs_sync when there are open unlinked files. The issue doesn't occur when using only soft-updates. >From prints added to softdep_check_suspend I see softdep_deps and ump->softdep_deps are 1 each time the function is called for the specified mount point after doing "mount -u -o ro" terminal 1 prepare /dev/md0 with journaled soft-updates, newfs -U -j mount /dev/md0 /mnt touch /mnt/a.txt tail -f /mnt/a.txt terminal 2 rm /mnt/a.txt mount -u -o ro /mnt mount -u -o ro /mnt Happens with physical hdd as well, for example /dev/ada0p2 Tested on 9.0-RELEASE and head r231393 VirtualBox 1 CPU and 2 CPU guests on i7-860 host Atom 330 host with journaled soft-updates # mount -u -o ro /mnt fsync: giving up on dirty 0xc4697770: tag devfs, type VCHR usecount 1, writecount 0, refcount 6 mountedhere 0xc4654500 flags () v_object 0xc469daa0 ref 0 pages 11 lock type devfs: EXCL by thread 0xc466f2e0 (pid 1518) dev md0 mount: /dev/md0 : Resource temporarily unavailable # mount -u -o ro /mnt mount is stuck taking 100% CPU with soft-updates only # mount -u -o ro /mnt mount: /dev/md0 : Device busy # mount -u -o ro /mnt mount: /dev/md0 : Device busy Thanks, Guy Yur From owner-freebsd-fs@FreeBSD.ORG Sat Feb 11 16:12:32 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9D64106566B; Sat, 11 Feb 2012 16:12:32 +0000 (UTC) (envelope-from arno@heho.snv.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id ACADC8FC0A; Sat, 11 Feb 2012 16:12:30 +0000 (UTC) Received: from heho.snv.jussieu.fr (heho.snv.jussieu.fr [134.157.184.22]) by shiva.jussieu.fr (8.14.4/jtpda-5.4) with ESMTP id q1BFtaqe031450 ; Sat, 11 Feb 2012 16:55:49 +0100 (CET) X-Ids: 165 Received: from heho.snv.jussieu.fr (localhost [127.0.0.1]) by heho.snv.jussieu.fr (8.14.3/8.14.3) with ESMTP id q1BFrBHV047752; Sat, 11 Feb 2012 16:53:11 +0100 (CET) (envelope-from arno@heho.snv.jussieu.fr) Received: (from arno@localhost) by heho.snv.jussieu.fr (8.14.3/8.14.3/Submit) id q1BFrBiw047749; Sat, 11 Feb 2012 16:53:11 +0100 (CET) (envelope-from arno) To: freebsd-stable@freebsd.org From: "Arno J. Klaassen" Date: Sat, 11 Feb 2012 16:53:10 +0100 Message-ID: User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Miltered: at jchkmail.jussieu.fr with ID 4F368F78.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 4F368F78.000/134.157.184.22/heho.snv.jussieu.fr/heho.snv.jussieu.fr/ Cc: freebsd-fs@freebsd.org Subject: 9-stable : geli + one-disk ZFS fails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 16:12:32 -0000 Hello, I finally decided to 'play' a bit with ZFS on a notebook, some years old, but I installed a brand new disk and memtest passes OK. I installed base+ports on partition 2, using 'classical' UFS. I crypted partition 3 and created a single zpool on it containing 4 Z-"file-systems" : [root@cc ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT zfiles 10.7G 377G 152K /zfiles zfiles/home 10.6G 377G 119M /zfiles/home zfiles/home/arno 10.5G 377G 2.35G /zfiles/home/arno zfiles/home/arno/.priv 192K 377G 192K /zfiles/home/arno/.priv zfiles/home/arno/.scito 8.18G 377G 8.18G /zfiles/home/arno/.scito I export the ZFS's via nfs and rsynced on the other machine some backup of my current note-book (geli + UFS, (almost) same 9-stable version, no problem) to the ZFS's. Quite fast, I see on the notebook : [root@cc /usr/temp]# zpool status -v pool: zfiles state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub repaired 0 in 0h1m with 11 errors on Sat Feb 11 14:55:34 2012 config: NAME STATE READ WRITE CKSUM zfiles ONLINE 0 0 11 ada0s3.eli ONLINE 0 0 23 errors: Permanent errors have been detected in the following files: /zfiles/home/arno/.scito/contrib/XNAT.tar [root@cc /usr/temp]# md5 /zfiles/home/arno/.scito/contrib/XNAT.tar md5: /zfiles/home/arno/.scito/contrib/XNAT.tar: Input/output error [root@cc /usr/temp]# As said, memtest is OK, nothing is logged to the console, UFS on the same disk works OK (I did some tests copying and comparing random data) and smartctl as well seems to trust the disk : SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) # 1 Extended offline Completed without error 00% 388 # 2 Short offline Completed without error 00% 387 Am I doing something wrong and/or let me know what I could provide as extra info to try to solve this (dmesg.boot at the end of this mail). Thanx a lot in advance, best, Arno ################### demsg.boot ####### Table 'FACP' at 0xbdd90200 Table 'APIC' at 0xbdd90390 APIC: Found table at 0xbdd90390 APIC: Using the MADT enumerator. MADT: Found CPU APIC ID 0 ACPI ID 1: enabled SMP: Added CPU 0 (AP) MADT: Found CPU APIC ID 1 ACPI ID 2: enabled SMP: Added CPU 1 (AP) MADT: Found CPU APIC ID 130 ACPI ID 3: disabled MADT: Found CPU APIC ID 131 ACPI ID 4: disabled Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-STABLE #0: Fri Feb 3 22:48:57 CET 2012 toor@cc:/usr/obj/raid1/bsd/9/src/sys/VR603 amd64 Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff80bba000. Preloaded /boot/zfs/zpool.cache "/boot/zfs/zpool.cache" at 0xffffffff80bba200. Calibrating TSC clock ... TSC clock: 2161296371 Hz CPU: Intel(R) Pentium(R) Dual CPU T3400 @ 2.16GHz (2161.30-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fd Family = 6 Model = f Stepping = 13 Features=0xbfebfbff Features2=0xe39d AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 3221225472 (3072 MB) Physical memory chunk(s): 0x0000000000001000 - 0x0000000000095fff, 610304 bytes (149 pages) 0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages) 0x0000000000be9000 - 0x00000000b8402fff, 3078725632 bytes (751642 pages) avail memory = 3057152000 (2915 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: INTR: Adding local APIC 1 as a target FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 x86bios: IVT 0x000000-0x0004ff at 0xfffffe0000000000 x86bios: SSEG 0x001000-0x001fff at 0xffffff8000210000 x86bios: EBDA 0x099000-0x09ffff at 0xfffffe0000099000 x86bios: ROM 0x0a0000-0x0fefff at 0xfffffe00000a0000 APIC: CPU 0 has ACPI ID 1 APIC: CPU 1 has ACPI ID 2 ULE: setup cpu 0 ULE: setup cpu 1 ACPI: RSDP 0xf9420 00014 (v00 ACPIAM) ACPI: RSDT 0xbdd90000 00048 (v01 MSI_NB MEGABOOK 20091013 MSFT 00000097) ACPI: FACP 0xbdd90200 00084 (v01 MSI_NB MEGABOOK 20091013 MSFT 00000097) ACPI: DSDT 0xbdd905c0 072D3 (v01 1ADTS 1ADTS012 00000012 INTL 20051117) ACPI: FACS 0xbdd9e000 00040 ACPI: APIC 0xbdd90390 0006C (v01 MSI_NB MEGABOOK 20091013 MSFT 00000097) ACPI: MCFG 0xbdd90400 0003C (v01 MSI_NB MEGABOOK 20091013 MSFT 00000097) ACPI: SLIC 0xbdd90440 00176 (v01 MSI_NB MEGABOOK 20091013 MSFT 00000097) ACPI: OEMB 0xbdd9e040 00072 (v01 MSI_NB MEGABOOK 20091013 MSFT 00000097) ACPI: HPET 0xbdd9a5c0 00038 (v01 MSI_NB OEMHPET 20091013 MSFT 00000097) ACPI: ASF! 0xbdd9a600 00099 (v32 LEGEND I865PASF 00000001 INTL 20051117) ACPI: GSCI 0xbdd9e0c0 02024 (v01 MSI_NB GMCHSCI 20091013 MSFT 00000097) ACPI: SSDT 0xbdda0a50 004F0 (v01 PmRef CpuPm 00003000 INTL 20051117) MADT: Found IO APIC ID 2, Interrupt 0 at 0xfec00000 ioapic0: Routing external 8259A's -> intpin 0 MADT: Interrupt override: source 0, irq 2 ioapic0: Routing IRQ 0 -> intpin 2 MADT: Interrupt override: source 9, irq 9 ioapic0: intpin 9 trigger: level ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x00000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 wlan: <802.11 Link Layer> random: kbd: new array size 4 kbd1 at kbdmux0 mem: nfslock: pseudo-device io: null: acpi0: on motherboard PCIe: Memory Mapped configuration base @ 0xe0000000 ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 ACPI: Executed 1 blocks of module-level executable AML code acpi0: Power Button (fixed) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, bdd00000 (3) failed ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI Error: No handler for Region [EC__] (0xfffffe000178c700) [EmbeddedControl] (20110527/evregion-421) ACPI Error: Region EmbeddedControl (ID=3) has no handler (20110527/exfldio-310) ACPI Error: Method parse/execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/psparse-560) ACPI Error: Method execution failed [\\_SB_.PCI0.SBRG.EC__.BAT1._STA] (Node 0xfffffe0001792980), AE_NOT_EXIST (20110527/uteval-113) ACPI timer: 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 -> 10 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 ACPI: SSDT 0xbdda04b0 00594 (v01 PmRef Cpu0Cst 00003001 INTL 20051117) ACPI: Dynamic OEM Table Load: ACPI: SSDT 0 00594 (v01 PmRef Cpu0Cst 00003001 INTL 20051117) cpu1: on acpi0 ACPI: SSDT 0xbdda0420 00085 (v01 PmRef Cpu1Cst 00003000 INTL 20051117) ACPI: Dynamic OEM Table Load: ACPI: SSDT 0 00085 (v01 PmRef Cpu1Cst 00003000 INTL 20051117) acpi_ec0: port 0x62,0x66 on acpi0 pci_link0: Index IRQ Rtd Ref IRQs Initial Probe 0 10 N 0 3 4 6 7 10 11 12 14 15 Validation 0 10 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pci_link1: Index IRQ Rtd Ref IRQs Initial Probe 0 5 N 0 5 Validation 0 5 N 0 5 After Disable 0 255 N 0 5 pci_link2: Index IRQ Rtd Ref IRQs Initial Probe 0 15 N 0 3 4 6 7 10 11 12 14 15 Validation 0 15 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pci_link3: Index IRQ Rtd Ref IRQs Initial Probe 0 11 N 0 3 4 6 7 10 11 12 14 15 Validation 0 11 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pci_link4: Index IRQ Rtd Ref IRQs Initial Probe 0 255 N 0 3 4 6 7 10 11 12 14 15 Validation 0 255 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pci_link5: Index IRQ Rtd Ref IRQs Initial Probe 0 7 N 0 3 4 6 7 10 11 12 14 15 Validation 0 7 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pci_link6: Index IRQ Rtd Ref IRQs Initial Probe 0 3 N 0 3 4 6 7 10 11 12 14 15 Validation 0 3 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pci_link7: Index IRQ Rtd Ref IRQs Initial Probe 0 14 N 0 3 4 6 7 10 11 12 14 15 Validation 0 14 N 0 3 4 6 7 10 11 12 14 15 After Disable 0 255 N 0 3 4 6 7 10 11 12 14 15 pcib0: port 0xcf8-0xcff on acpi0 pcib0: decoding 4 range 0-0xcf7 pcib0: decoding 4 range 0xd00-0xffff pcib0: decoding 3 range 0xa0000-0xbffff pcib0: decoding 3 range 0xd0000-0xdffff pcib0: decoding 3 range 0xbde00000-0xdfffffff pcib0: decoding 3 range 0xf0000000-0xfed8ffff pci0: on pcib0 pci0: domain=0, physical bus=0 found-> vendor=0x8086, dev=0x2a40, revid=0x07 domain=0, bus=0, slot=0, func=0 class=06-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x2090, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x2a42, revid=0x07 domain=0, bus=0, slot=2, func=0 class=03-00-00, hdrtype=0x00, mfdev=1 cmdreg=0x0007, statreg=0x0090, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=10 powerspec 3 supports D0 D3 current D0 MSI supports 1 message map[10]: type Memory, range 64, base 0xfd800000, size 22, enabled pcib0: allocated type 3 (0xfd800000-0xfdbfffff) for rid 10 of pci0:0:2:0 map[18]: type Prefetchable Memory, range 64, base 0xd0000000, size 28, enabled pcib0: allocated type 3 (0xd0000000-0xdfffffff) for rid 18 of pci0:0:2:0 map[20]: type I/O Port, range 32, base 0xb400, size 3, enabled pcib0: allocated type 4 (0xb400-0xb407) for rid 20 of pci0:0:2:0 pcib0: matched entry for 0.2.INTA pcib0: slot 2 INTA hardwired to IRQ 16 found-> vendor=0x8086, dev=0x2a43, revid=0x07 domain=0, bus=0, slot=2, func=1 class=03-80-00, hdrtype=0x00, mfdev=1 cmdreg=0x0007, statreg=0x0090, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) powerspec 3 supports D0 D3 current D0 map[10]: type Memory, range 64, base 0xfdd00000, size 20, enabled pcib0: allocated type 3 (0xfdd00000-0xfddfffff) for rid 10 of pci0:0:2:1 found-> vendor=0x8086, dev=0x2937, revid=0x03 domain=0, bus=0, slot=26, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=10 map[20]: type I/O Port, range 32, base 0xb800, size 5, enabled pcib0: allocated type 4 (0xb800-0xb81f) for rid 20 of pci0:0:26:0 pcib0: matched entry for 0.26.INTA pcib0: slot 26 INTA hardwired to IRQ 16 found-> vendor=0x8086, dev=0x2938, revid=0x03 domain=0, bus=0, slot=26, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=7 map[20]: type I/O Port, range 32, base 0xb480, size 5, enabled pcib0: allocated type 4 (0xb480-0xb49f) for rid 20 of pci0:0:26:1 pcib0: matched entry for 0.26.INTB pcib0: slot 26 INTB hardwired to IRQ 21 found-> vendor=0x8086, dev=0x293c, revid=0x03 domain=0, bus=0, slot=26, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=15 powerspec 2 supports D0 D3 current D0 map[10]: type Memory, range 32, base 0xfdefec00, size 10, enabled pcib0: allocated type 3 (0xfdefec00-0xfdefefff) for rid 10 of pci0:0:26:7 pcib0: matched entry for 0.26.INTC pcib0: slot 26 INTC hardwired to IRQ 18 found-> vendor=0x8086, dev=0x293e, revid=0x03 domain=0, bus=0, slot=27, func=0 class=04-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=3 powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit map[10]: type Memory, range 64, base 0xfdef8000, size 14, enabled pcib0: allocated type 3 (0xfdef8000-0xfdefbfff) for rid 10 of pci0:0:27:0 pcib0: matched entry for 0.27.INTA pcib0: slot 27 INTA hardwired to IRQ 22 found-> vendor=0x8086, dev=0x2940, revid=0x03 domain=0, bus=0, slot=28, func=0 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0107, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x02 (500 ns), maxlat=0x00 (0 ns) intpin=a, irq=5 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.28.INTA pcib0: slot 28 INTA hardwired to IRQ 17 found-> vendor=0x8086, dev=0x2942, revid=0x03 domain=0, bus=0, slot=28, func=1 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0107, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x02 (500 ns), maxlat=0x00 (0 ns) intpin=b, irq=10 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.28.INTB pcib0: slot 28 INTB hardwired to IRQ 16 found-> vendor=0x8086, dev=0x2946, revid=0x03 domain=0, bus=0, slot=28, func=3 class=06-04-00, hdrtype=0x01, mfdev=1 cmdreg=0x0106, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x02 (500 ns), maxlat=0x00 (0 ns) intpin=d, irq=11 powerspec 2 supports D0 D3 current D0 MSI supports 1 message pcib0: matched entry for 0.28.INTD pcib0: slot 28 INTD hardwired to IRQ 19 found-> vendor=0x8086, dev=0x2934, revid=0x03 domain=0, bus=0, slot=29, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=14 map[20]: type I/O Port, range 32, base 0xc000, size 5, enabled pcib0: allocated type 4 (0xc000-0xc01f) for rid 20 of pci0:0:29:0 pcib0: matched entry for 0.29.INTA pcib0: slot 29 INTA hardwired to IRQ 23 found-> vendor=0x8086, dev=0x2935, revid=0x03 domain=0, bus=0, slot=29, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=11 map[20]: type I/O Port, range 32, base 0xbc00, size 5, enabled pcib0: allocated type 4 (0xbc00-0xbc1f) for rid 20 of pci0:0:29:1 pcib0: matched entry for 0.29.INTB pcib0: slot 29 INTB hardwired to IRQ 19 found-> vendor=0x8086, dev=0x2936, revid=0x03 domain=0, bus=0, slot=29, func=2 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=15 map[20]: type I/O Port, range 32, base 0xb880, size 5, enabled pcib0: allocated type 4 (0xb880-0xb89f) for rid 20 of pci0:0:29:2 pcib0: matched entry for 0.29.INTC pcib0: slot 29 INTC hardwired to IRQ 18 found-> vendor=0x8086, dev=0x293a, revid=0x03 domain=0, bus=0, slot=29, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=14 powerspec 2 supports D0 D3 current D0 map[10]: type Memory, range 32, base 0xfdeff000, size 10, enabled pcib0: allocated type 3 (0xfdeff000-0xfdeff3ff) for rid 10 of pci0:0:29:7 pcib0: matched entry for 0.29.INTA pcib0: slot 29 INTA hardwired to IRQ 23 found-> vendor=0x8086, dev=0x2448, revid=0x93 domain=0, bus=0, slot=30, func=0 class=06-04-01, hdrtype=0x01, mfdev=0 cmdreg=0x0104, statreg=0x0010, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x02 (500 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x2919, revid=0x03 domain=0, bus=0, slot=31, func=0 class=06-01-00, hdrtype=0x00, mfdev=1 cmdreg=0x0007, statreg=0x0210, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) found-> vendor=0x8086, dev=0x2929, revid=0x03 domain=0, bus=0, slot=31, func=2 class=01-06-01, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x02b0, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=11 powerspec 3 supports D0 D3 current D0 MSI supports 16 messages map[10]: type I/O Port, range 32, base 0xcc00, size 3, enabled pcib0: allocated type 4 (0xcc00-0xcc07) for rid 10 of pci0:0:31:2 map[14]: type I/O Port, range 32, base 0xc880, size 2, enabled pcib0: allocated type 4 (0xc880-0xc883) for rid 14 of pci0:0:31:2 map[18]: type I/O Port, range 32, base 0xc800, size 3, enabled pcib0: allocated type 4 (0xc800-0xc807) for rid 18 of pci0:0:31:2 map[1c]: type I/O Port, range 32, base 0xc480, size 2, enabled pcib0: allocated type 4 (0xc480-0xc483) for rid 1c of pci0:0:31:2 map[20]: type I/O Port, range 32, base 0xc400, size 5, enabled pcib0: allocated type 4 (0xc400-0xc41f) for rid 20 of pci0:0:31:2 map[24]: type Memory, range 32, base 0xfdeff800, size 11, enabled pcib0: allocated type 3 (0xfdeff800-0xfdefffff) for rid 24 of pci0:0:31:2 pcib0: matched entry for 0.31.INTB pcib0: slot 31 INTB hardwired to IRQ 19 found-> vendor=0x8086, dev=0x2930, revid=0x03 domain=0, bus=0, slot=31, func=3 class=0c-05-00, hdrtype=0x00, mfdev=0 cmdreg=0x0003, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=15 map[10]: type Memory, range 64, base 0xfdeff400, size 8, enabled pcib0: allocated type 3 (0xfdeff400-0xfdeff4ff) for rid 10 of pci0:0:31:3 map[20]: type I/O Port, range 32, base 0x400, size 5, enabled pcib0: allocated type 4 (0x400-0x41f) for rid 20 of pci0:0:31:3 pcib0: matched entry for 0.31.INTC pcib0: slot 31 INTC hardwired to IRQ 18 vgapci0: port 0xb400-0xb407 mem 0xfd800000-0xfdbfffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: aperture size is 256M, detected 32764k stolen memory vgapci1: mem 0xfdd00000-0xfddfffff at device 2.1 on pci0 pci0: at device 26.0 (no driver attached) pci0: at device 26.1 (no driver attached) pci0: at device 26.7 (no driver attached) pci0: at device 27.0 (no driver attached) pcib1: irq 17 at device 28.0 on pci0 pcib0: allocated type 4 (0xd000-0xdfff) for rid 1c of pcib1 pcib0: allocated type 3 (0xfdf00000-0xfdffffff) for rid 20 of pcib1 pcib0: allocated type 3 (0xf9f00000-0xf9ffffff) for rid 24 of pcib1 pcib1: domain 0 pcib1: secondary bus 2 pcib1: subordinate bus 2 pcib1: I/O decode 0xd000-0xdfff pcib1: memory decode 0xfdf00000-0xfdffffff pcib1: prefetched decode 0xf9f00000-0xf9ffffff pci2: on pcib1 pci2: domain=0, physical bus=2 found-> vendor=0x10ec, dev=0x8168, revid=0x02 domain=0, bus=2, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=10 powerspec 3 supports D0 D1 D2 D3 current D0 MSI supports 1 message, 64 bit MSI-X supports 2 messages in map 0x20 map[10]: type I/O Port, range 32, base 0xd800, size 8, enabled pcib1: allocated I/O port range (0xd800-0xd8ff) for rid 10 of pci0:2:0:0 map[18]: type Memory, range 64, base 0xfdfff000, size 12, enabled pcib1: allocated memory range (0xfdfff000-0xfdffffff) for rid 18 of pci0:2:0:0 map[20]: type Prefetchable Memory, range 64, base 0xf9ff0000, size 16, enabled pcib1: allocated prefetch range (0xf9ff0000-0xf9ffffff) for rid 20 of pci0:2:0:0 pcib1: matched entry for 2.0.INTA pcib1: slot 0 INTA hardwired to IRQ 16 pci2: at device 0.0 (no driver attached) pcib2: irq 16 at device 28.1 on pci0 pcib0: allocated type 4 (0xe000-0xefff) for rid 1c of pcib2 pcib0: allocated type 3 (0xfe000000-0xfeafffff) for rid 20 of pcib2 pcib0: allocated type 3 (0xfa000000-0xfcffffff) for rid 24 of pcib2 pcib2: domain 0 pcib2: secondary bus 3 pcib2: subordinate bus 4 pcib2: I/O decode 0xe000-0xefff pcib2: memory decode 0xfe000000-0xfeafffff pcib2: prefetched decode 0xfa000000-0xfcffffff pci3: on pcib2 pci3: domain=0, physical bus=3 pcib3: irq 19 at device 28.3 on pci0 pcib0: allocated type 3 (0xfeb00000-0xfebfffff) for rid 20 of pcib3 pcib3: domain 0 pcib3: secondary bus 5 pcib3: subordinate bus 5 pcib3: memory decode 0xfeb00000-0xfebfffff pcib3: no prefetched decode pci5: on pcib3 pci5: domain=0, physical bus=5 found-> vendor=0x168c, dev=0x001c, revid=0x01 domain=0, bus=5, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=11 powerspec 2 supports D0 D3 current D0 MSI supports 1 message MSI-X supports 1 message in map 0x10 map[10]: type Memory, range 64, base 0xfebf0000, size 16, enabled pcib3: allocated memory range (0xfebf0000-0xfebfffff) for rid 10 of pci0:5:0:0 pcib3: matched entry for 5.0.INTA pcib3: slot 0 INTA hardwired to IRQ 19 pci5: at device 0.0 (no driver attached) pci0: at device 29.0 (no driver attached) pci0: at device 29.1 (no driver attached) pci0: at device 29.2 (no driver attached) pci0: at device 29.7 (no driver attached) pcib4: at device 30.0 on pci0 pcib4: domain 0 pcib4: secondary bus 1 pcib4: subordinate bus 1 pcib4: no prefetched decode pcib4: Subtractively decoded bridge. pcib4: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P1 - AE_NOT_FOUND pci1: on pcib4 pci1: domain=0, physical bus=1 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc41f mem 0xfdeff800-0xfdefffff irq 19 at device 31.2 on pci0 ahci0: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 256 to local APIC 0 vector 49 ahci0: using IRQ 256 for MSI ahci0: AHCI v1.20 with 4 3Gbps ports, Port Multiplier not supported ahci0: Caps: 64bit NCQ SNTF SS ALP AL CLO 3Gbps PMD SSC PSC 32cmd CCC EM eSATA 4ports ahci0: Caps2: ahci0: EM Caps: ALHD XMT SMB LED ahcich0: at channel 0 on ahci0 ahcich0: Caps: ahcich1: at channel 1 on ahci0 ahcich1: Caps: ahcich2: not probed (disabled) ahcich3: not probed (disabled) ahcich4: at channel 4 on ahci0 ahcich4: Caps: ahcich5: at channel 5 on ahci0 ahcich5: Caps: pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 acpi_tz0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 ioapic0: routing intpin 2 (ISA IRQ 0) to lapic 0 vector 50 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock (resolution 1000000us, adjustment 0.500000000s) ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 51 Event timer "RTC" frequency 32768 Hz quality 0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 atkbd: the current kbd controller command byte 0065 atkbd: keyboard ID 0x41ab (2) kbd0 at atkbd0 kbd0: atkbd0, AT 101/102 (2), config:0x0, flags:0x3d0000 ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 52 atkbd0: [GIANT-LOCKED] psm0: unable to allocate IRQ psmcpnp0: irq 12 on acpi0 psm0: current command byte:0065 psm0: irq 12 on atkbdc0 ioapic0: routing intpin 12 (ISA IRQ 12) to lapic 0 vector 53 psm0: [GIANT-LOCKED] psm0: model IntelliMouse, device ID 3-00, 3 buttons psm0: config:00000000, flags:00000008, packet size:4 psm0: syncmask:08, syncbits:00 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 hpet0: vendor 0x8086, rev 0x1, 14318180Hz 64bit, 4 timers, legacy route hpet0: t0: irqs 0x00f00000 (0), 64bit, periodic hpet0: t1: irqs 0x00f00000 (0) hpet0: t2: irqs 0x00f00800 (0) hpet0: t3: irqs 0x00f01000 (0) Timecounter "HPET" frequency 14318180 Hz quality 950 ioapic0: routing intpin 20 (PCI IRQ 20) to lapic 0 vector 54 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 acpi_acad0: on acpi0 battery0: on acpi0 acpi_lid0: on acpi0 acpi_button1: on acpi0 acpi0: wakeup code va 0xffffff80d4544000 pa 0x4000 pcib0: allocated type 3 (0xa0000-0xa07ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa0800-0xa0fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa1000-0xa17ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa1800-0xa1fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa2000-0xa27ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa2800-0xa2fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa3000-0xa37ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa3800-0xa3fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa4000-0xa47ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa4800-0xa4fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa5000-0xa57ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa5800-0xa5fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa6000-0xa67ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa6800-0xa6fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa7000-0xa77ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa7800-0xa7fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa8000-0xa87ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa8800-0xa8fff) for rid 0 of orm0 pcib0: allocated type 3 (0xa9000-0xa97ff) for rid 0 of orm0 pcib0: allocated type 3 (0xa9800-0xa9fff) for rid 0 of orm0 pcib0: allocated type 3 (0xaa000-0xaa7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xaa800-0xaafff) for rid 0 of orm0 pcib0: allocated type 3 (0xab000-0xab7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xab800-0xabfff) for rid 0 of orm0 pcib0: allocated type 3 (0xac000-0xac7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xac800-0xacfff) for rid 0 of orm0 pcib0: allocated type 3 (0xad000-0xad7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xad800-0xadfff) for rid 0 of orm0 pcib0: allocated type 3 (0xae000-0xae7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xae800-0xaefff) for rid 0 of orm0 pcib0: allocated type 3 (0xaf000-0xaf7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xaf800-0xaffff) for rid 0 of orm0 pcib0: allocated type 3 (0xb0000-0xb07ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb0800-0xb0fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb1000-0xb17ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb1800-0xb1fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb2000-0xb27ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb2800-0xb2fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb3000-0xb37ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb3800-0xb3fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb4000-0xb47ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb4800-0xb4fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb5000-0xb57ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb5800-0xb5fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb6000-0xb67ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb6800-0xb6fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb7000-0xb77ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb7800-0xb7fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb8000-0xb87ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb8800-0xb8fff) for rid 0 of orm0 pcib0: allocated type 3 (0xb9000-0xb97ff) for rid 0 of orm0 pcib0: allocated type 3 (0xb9800-0xb9fff) for rid 0 of orm0 pcib0: allocated type 3 (0xba000-0xba7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xba800-0xbafff) for rid 0 of orm0 pcib0: allocated type 3 (0xbb000-0xbb7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbb800-0xbbfff) for rid 0 of orm0 pcib0: allocated type 3 (0xbc000-0xbc7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbc800-0xbcfff) for rid 0 of orm0 pcib0: allocated type 3 (0xbd000-0xbd7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbd800-0xbdfff) for rid 0 of orm0 pcib0: allocated type 3 (0xbe000-0xbe7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbe800-0xbefff) for rid 0 of orm0 pcib0: allocated type 3 (0xbf000-0xbf7ff) for rid 0 of orm0 pcib0: allocated type 3 (0xbf800-0xbffff) for rid 0 of orm0 pcib0: allocated type 3 (0xd0000-0xd07ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd0800-0xd0fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd1000-0xd17ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd1800-0xd1fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd2000-0xd27ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd2800-0xd2fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd3000-0xd37ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd3800-0xd3fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd4000-0xd47ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd4800-0xd4fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd5000-0xd57ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd5800-0xd5fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd6000-0xd67ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd6800-0xd6fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd7000-0xd77ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd7800-0xd7fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd8000-0xd87ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd8800-0xd8fff) for rid 1 of orm0 pcib0: allocated type 3 (0xd9000-0xd97ff) for rid 1 of orm0 pcib0: allocated type 3 (0xd9800-0xd9fff) for rid 1 of orm0 pcib0: allocated type 3 (0xda000-0xda7ff) for rid 1 of orm0 pcib0: allocated type 3 (0xda800-0xdafff) for rid 1 of orm0 pcib0: allocated type 3 (0xdb000-0xdb7ff) for rid 1 of orm0 pcib0: allocated type 3 (0xdb800-0xdbfff) for rid 1 of orm0 pcib0: allocated type 3 (0xdc000-0xdc7ff) for rid 1 of orm0 pcib0: allocated type 3 (0xdc800-0xdcfff) for rid 1 of orm0 pcib0: allocated type 3 (0xdd000-0xdd7ff) for rid 1 of orm0 pcib0: allocated type 3 (0xdd800-0xddfff) for rid 1 of orm0 pcib0: allocated type 3 (0xde000-0xde7ff) for rid 1 of orm0 pcib0: allocated type 3 (0xde800-0xdefff) for rid 1 of orm0 pcib0: allocated type 3 (0xdf000-0xdf7ff) for rid 1 of orm0 pcib0: allocated type 3 (0xdf800-0xdffff) for rid 1 of orm0 isa_probe_children: disabling PnP devices atkbdc: atkbdc0 already exists; skipping it atrtc: atrtc0 already exists; skipping it attimer: attimer0 already exists; skipping it sc: sc0 already exists; skipping it isa_probe_children: probing non-PnP devices orm0: at iomem 0xc0000-0xcf7ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sc0: fb0, kbd1, terminal emulator: scteken (teken terminal) vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 pcib0: allocated type 4 (0x3c0-0x3df) for rid 0 of vga0 pcib0: allocated type 3 (0xa0000-0xbffff) for rid 0 of vga0 fdc0 failed to probe at port 0x3f0 irq 6 drq 2 on isa0 ppc0 failed to probe at irq 7 on isa0 pcib0: allocated type 4 (0x3f8-0x3ff) for rid 0 of uart0 uart0: failed to probe at port 0x3f8-0x3ff irq 4 on isa0 pcib0: allocated type 4 (0x2f8-0x2ff) for rid 0 of uart1 uart1: failed to probe at port 0x2f8-0x2ff irq 3 on isa0 isa_probe_children: probing PnP devices Device configuration finished. procfs registered Timecounters tick every 1.000 msec vlan: initialized, using hash tables with chaining lo0: bpf attached ahcich0: AHCI reset... ahcich0: SATA connect time=100us status=00000123 ahcich0: AHCI reset: device found ahcich1: AHCI reset... ahcich1: SATA offline status=00000004 ahcich1: AHCI reset: device not found ahcich4: AHCI reset... ahcich4: SATA offline status=00000004 ahcich4: AHCI reset: device not found ahcich5: AHCI reset... ahcich5: SATA connect time=900us status=00000113 ahcich5: AHCI reset: device found ahcich5: AHCI reset: device ready after 0ms (aprobe1:ahcich5:0:0:0): SIGNATURE: eb14 acpi_acad0: acline initialization start battery0: battery initialization start acpi_acad0: On Line acpi_acad0: acline initialization done, tried 1 times ahcich5: SNTF 0x0001 ahcich0: AHCI reset: device ready after 100ms (aprobe0:ahcich0:0:0:0): SIGNATURE: 0000 pass0 at ahcich0 bus 0 scbus0 target 0 lun 0 pass0: ATA-8 SATA 2.x device GEOM: new disk cd0 pass0: Serial Number 5YX0J5YD (cd0:ahcich5:0:0:0): SCSI status error (cd0:ahcich5:0:0:0): READ CAPACITY. CDB: 25 0 0 0 0 0 0 0 0 0 (cd0:ahcich5:0:0:0): CAM status: SCSI Status Error (cd0:ahcich5:0:0:0): SCSI status: Check Condition (cd0:ahcich5:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present - tray open) (cd0:ahcich5:0:0:0): Error 6, Unretryable error cd0 at ahcich5 bus 0 scbus3 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: Serial Number 30651780 1165921Q111 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray open pass0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) pass0: Command Queueing enabled pass1 at ahcich5 bus 0 scbus3 target 0 lun 0 pass1: Removable CD-ROM SCSI-0 device pass1: Serial Number 30651780 1165921Q111 pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 (cd0:ahcich5:0:0:0): SCSI status error (cd0:ahcich5:0:0:0): READ CAPACITY. CDB: 25 0 0 0 0 0 0 0 0 0 (cd0:ahcich5:0:0:0): CAM status: SCSI Status Error (cd0:ahcich5:0:0:0): SCSI status: Check Condition (cd0:ahcich5:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present - tray open) (cd0:ahcich5:0:0:0): Error 6, Unretryable error ada0: ATA-8 SATA 2.x device ada0: Serial Number 5YX0J5YD ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! cpu1 AP: ID: 0x01000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 TSC timecounter disabled: C3 enabled. Timecounter "TSC" frequency 2161296371 Hz quality -1000 (cd0:ahcich5:0:0:0): SCSI status error (cd0:ahcich5:0:0:0): READ CAPACITY. CDB: 25 0 0 0 0 0 0 0 0 0 (cd0:ahcich5:0:0:0): CAM status: SCSI Status Error (cd0:ahcich5:0:0:0): SCSI status: Check Condition (cd0:ahcich5:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present - tray open) (cd0:ahcich5:0:battery0: battery initialization done, tried 1 times 0:0): Error 6, Unretryable error (cd0:ahcich5:0:0:0): SCSI status error (cd0:ahcich5:0:0:0): READ CAPACITY. CDB: 25 0 0 0 0 0 0 0 0 0 (cd0:ahcich5:0:0:0): CAM status: SCSI Status Error (cd0:ahcich5:0:0:0): SCSI status: Check Condition (cd0:ahcich5:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present - tray open) (cd0:ahcich5:0:0:0): Error 6, Unretryable error GEOM: new disk ada0 GEOM: ada0s3: media size does not match label. Trying to mount root from ufs:/dev/ada0s2a [rw,noatime]... start_init: trying /sbin/init ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version 5 ZFS storage pool version 28 (cd0:ahcich5:0:0:0): SCSI status error (cd0:ahcich5:0:0:0): READ CAPACITY. CDB: 25 0 0 0 0 0 0 0 0 0 (cd0:ahcich5:0:0:0): CAM status: SCSI Status Error (cd0:ahcich5:0:0:0): SCSI status: Check Condition (cd0:ahcich5:0:0:0): SCSI sense: NOT READY asc:3a,2 (Medium not present - tray open) (cd0:ahcich5:0:0:0): Error 6, Unretryable error tap0: bpf attached tap0: Ethernet address: 00:bd:0b:07:00:00 pci0: driver added found-> vendor=0x8086, dev=0x2937, revid=0x03 domain=0, bus=0, slot=26, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=16 pci0:0:26:0: reprobing on driver added found-> vendor=0x8086, dev=0x2938, revid=0x03 domain=0, bus=0, slot=26, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=21 pci0:0:26:1: reprobing on driver added found-> vendor=0x8086, dev=0x293c, revid=0x03 domain=0, bus=0, slot=26, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 powerspec 2 supports D0 D3 current D0 pci0:0:26:7: reprobing on driver added found-> vendor=0x8086, dev=0x293e, revid=0x03 domain=0, bus=0, slot=27, func=0 class=04-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=22 powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit pci0:0:27:0: reprobing on driver added found-> vendor=0x8086, dev=0x2934, revid=0x03 domain=0, bus=0, slot=29, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=23 pci0:0:29:0: reprobing on driver added found-> vendor=0x8086, dev=0x2935, revid=0x03 domain=0, bus=0, slot=29, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=19 pci0:0:29:1: reprobing on driver added found-> vendor=0x8086, dev=0x2936, revid=0x03 domain=0, bus=0, slot=29, func=2 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 pci0:0:29:2: reprobing on driver added found-> vendor=0x8086, dev=0x293a, revid=0x03 domain=0, bus=0, slot=29, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=23 powerspec 2 supports D0 D3 current D0 pci0:0:29:7: reprobing on driver added found-> vendor=0x8086, dev=0x2930, revid=0x03 domain=0, bus=0, slot=31, func=3 class=0c-05-00, hdrtype=0x00, mfdev=0 cmdreg=0x0003, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 pci0:0:31:3: reprobing on driver added pci1: driver added pci2: driver added found-> vendor=0x10ec, dev=0x8168, revid=0x02 domain=0, bus=2, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=16 powerspec 3 supports D0 D1 D2 D3 current D0 MSI supports 1 message, 64 bit MSI-X supports 2 messages in map 0x20 pci0:2:0:0: reprobing on driver added re0: port 0xd800-0xd8ff mem 0xfdfff000-0xfdffffff,0xf9ff0000-0xf9ffffff irq 16 at device 0.0 on pci2 re0: MSI count : 1 re0: MSI-X count : 2 re0: attempting to allocate 1 MSI-X vectors (2 supported) msi: routing MSI-X IRQ 257 to local APIC 0 vector 55 re0: using IRQ 257 for MSI-X re0: Using 1 MSI-X message re0: ASPM disabled re0: Chip rev. 0x3c000000 re0: MAC rev. 0x00400000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: OUI 0x00e04c, model 0x0011, rev. 2 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: bpf attached re0: Ethernet address: 00:24:21:61:e0:20 pci3: driver added pci5: driver added found-> vendor=0x168c, dev=0x001c, revid=0x01 domain=0, bus=5, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=19 powerspec 2 supports D0 D3 current D0 MSI supports 1 message MSI-X supports 1 message in map 0x10 pci0:5:0:0: reprobing on driver added pci0: driver added found-> vendor=0x8086, dev=0x2937, revid=0x03 domain=0, bus=0, slot=26, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=16 pci0:0:26:0: reprobing on driver added found-> vendor=0x8086, dev=0x2938, revid=0x03 domain=0, bus=0, slot=26, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=21 pci0:0:26:1: reprobing on driver added found-> vendor=0x8086, dev=0x293c, revid=0x03 domain=0, bus=0, slot=26, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 powerspec 2 supports D0 D3 current D0 pci0:0:26:7: reprobing on driver added found-> vendor=0x8086, dev=0x293e, revid=0x03 domain=0, bus=0, slot=27, func=0 class=04-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=22 powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit pci0:0:27:0: reprobing on driver added found-> vendor=0x8086, dev=0x2934, revid=0x03 domain=0, bus=0, slot=29, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=23 pci0:0:29:0: reprobing on driver added found-> vendor=0x8086, dev=0x2935, revid=0x03 domain=0, bus=0, slot=29, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=19 pci0:0:29:1: reprobing on driver added found-> vendor=0x8086, dev=0x2936, revid=0x03 domain=0, bus=0, slot=29, func=2 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 pci0:0:29:2: reprobing on driver added found-> vendor=0x8086, dev=0x293a, revid=0x03 domain=0, bus=0, slot=29, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=23 powerspec 2 supports D0 D3 current D0 pci0:0:29:7: reprobing on driver added found-> vendor=0x8086, dev=0x2930, revid=0x03 domain=0, bus=0, slot=31, func=3 class=0c-05-00, hdrtype=0x00, mfdev=0 cmdreg=0x0003, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 pci0:0:31:3: reprobing on driver added pci1: driver added pci2: driver added pci3: driver added pci5: driver added found-> vendor=0x168c, dev=0x001c, revid=0x01 domain=0, bus=5, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0007, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=19 powerspec 2 supports D0 D3 current D0 MSI supports 1 message MSI-X supports 1 message in map 0x10 pci0:5:0:0: reprobing on driver added ath0: mem 0xfebf0000-0xfebfffff irq 19 at device 0.0 on pci5 ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 56 ath0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps ath0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps ath0: AR2425 mac 14.2 RF5424 phy 7.0 ath0: Use hw queue 1 for WME_AC_BE traffic ath0: Use hw queue 0 for WME_AC_BK traffic ath0: Use hw queue 2 for WME_AC_VI traffic ath0: Use hw queue 3 for WME_AC_VO traffic ath0: Use hw queue 8 for CAB traffic ath0: Use hw queue 9 for beacons ath0: using multicast key search crypto: cryptosoft0: on motherboard crypto: assign cryptosoft0 driver id 0, flags 100663296 crypto: cryptosoft0 registers alg 1 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 2 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 3 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 4 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 5 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 16 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 6 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 7 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 18 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 19 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 20 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 8 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 15 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 9 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 10 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 13 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 14 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 11 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 22 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 21 flags 0 maxoplen 0 crypto: cryptosoft0 registers alg 17 flags 0 maxoplen 0 GEOM_ELI: Device ada0s3.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software pci0: driver added found-> vendor=0x8086, dev=0x2937, revid=0x03 domain=0, bus=0, slot=26, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=16 pci0:0:26:0: reprobing on driver added found-> vendor=0x8086, dev=0x2938, revid=0x03 domain=0, bus=0, slot=26, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=21 pci0:0:26:1: reprobing on driver added found-> vendor=0x8086, dev=0x293c, revid=0x03 domain=0, bus=0, slot=26, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 powerspec 2 supports D0 D3 current D0 pci0:0:26:7: reprobing on driver added found-> vendor=0x8086, dev=0x293e, revid=0x03 domain=0, bus=0, slot=27, func=0 class=04-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=22 powerspec 2 supports D0 D3 current D0 MSI supports 1 message, 64 bit pci0:0:27:0: reprobing on driver added found-> vendor=0x8086, dev=0x2934, revid=0x03 domain=0, bus=0, slot=29, func=0 class=0c-03-00, hdrtype=0x00, mfdev=1 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=23 pci0:0:29:0: reprobing on driver added found-> vendor=0x8086, dev=0x2935, revid=0x03 domain=0, bus=0, slot=29, func=1 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=b, irq=19 pci0:0:29:1: reprobing on driver added found-> vendor=0x8086, dev=0x2936, revid=0x03 domain=0, bus=0, slot=29, func=2 class=0c-03-00, hdrtype=0x00, mfdev=0 cmdreg=0x0005, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 pci0:0:29:2: reprobing on driver added found-> vendor=0x8086, dev=0x293a, revid=0x03 domain=0, bus=0, slot=29, func=7 class=0c-03-20, hdrtype=0x00, mfdev=0 cmdreg=0x0006, statreg=0x0290, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=23 powerspec 2 supports D0 D3 current D0 pci0:0:29:7: reprobing on driver added found-> vendor=0x8086, dev=0x2930, revid=0x03 domain=0, bus=0, slot=31, func=3 class=0c-05-00, hdrtype=0x00, mfdev=0 cmdreg=0x0003, statreg=0x0280, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=c, irq=18 pci0:0:31:3: reprobing on driver added pci1: driver added pci2: driver added pci3: driver added pci5: driver added est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est0 attach returned 6 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est1 attach returned 6 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est0 attach returned 6 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est1 attach returned 6 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est0 attach returned 6 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est1 attach returned 6 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est1 attach returned 6 p4tcc1: on cpu1 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est0 attach returned 6 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est1 attach returned 6 coretemp0: on cpu0 coretemp0: Setting TjMax=100 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est0 attach returned 6 coretemp1: on cpu1 coretemp1: Setting TjMax=100 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 6130d2b06000d2b device_attach: est1 attach returned 6 From owner-freebsd-fs@FreeBSD.ORG Sat Feb 11 20:42:50 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6574F106564A for ; Sat, 11 Feb 2012 20:42:50 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm30-vm0.bullet.mail.sp2.yahoo.com (nm30-vm0.bullet.mail.sp2.yahoo.com [98.139.91.238]) by mx1.freebsd.org (Postfix) with SMTP id 411CF8FC0C for ; Sat, 11 Feb 2012 20:42:50 +0000 (UTC) Received: from [98.139.91.65] by nm30.bullet.mail.sp2.yahoo.com with NNFMP; 11 Feb 2012 20:29:02 -0000 Received: from [98.136.185.44] by tm5.bullet.mail.sp2.yahoo.com with NNFMP; 11 Feb 2012 20:29:02 -0000 Received: from [127.0.0.1] by smtp105.mail.gq1.yahoo.com with NNFMP; 11 Feb 2012 20:29:02 -0000 X-Yahoo-Newman-Id: 212652.97915.bm@smtp105.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: Lxc6gagVM1lelhJkoyuzuqUt_WKMbFguycMPqzbPN2HqOHt gOYUblTHQbKQMod0n3IvSaR3rLuLRfVQ5JIkg8Hr0xdvaf2uFNLbeplsx5rH ndgBAP0ZMSg_HIvRy6YFETbrGxKyH.W1543IDdKlBlltt0JG4PrqoGo.PlDA 5zXXaeDd45Q8qZZFUYweMz7X5JHI4jpoQ_QKaRfkn18Wy4AvvaSAgd0a.bT7 .RH3k13C0bZrC_OmEXikosreCgwt3AGUqvVbB25HJf0hEnW0tZg_w6K207A1 GIs16snN3jS4pEKUoO3Y7HckXT40rhfnSNLldxAX_xY73huKXgWGS8Hz8RwR qJ55zinptOD2Ml.QmiIIHlt51GSVzxrDFN3B8xQg08f_gr80IPZbbJuvlSrZ Ym7icFfcGd5rzS74SJcbrWYWIoyDI1zDqHcnbsyga8Vbkr8Tethot2f2Sq4s QjZCnGsCODWDns5zTA2FxGaM47zps9HsFGhtWshOXbYrN.x0Q5W7x7hnfE7. JRuP330MkcPbFb7dF8aKcW4JluggKnwPob7eBk7EkWYOP3SFIKH4Er1wSJhG 1v2Sfr7Ze.qAg8DUu5gjgFs7d5IZbg1LOgSAYx8DfXTme9WbG1f4JmPfIay9 _h0IWbrM- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf Received: from [192.168.10.101] (pfg@200.118.157.7 with plain) by smtp105.mail.gq1.yahoo.com with SMTP; 11 Feb 2012 12:29:01 -0800 PST Message-ID: <4F36CF8B.3000509@FreeBSD.org> Date: Sat, 11 Feb 2012 15:28:59 -0500 From: Pedro Giffuni User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@FreeBSD.org Subject: [CFT] ext2/3 nanosecond timestamps patch. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 20:42:50 -0000 Hello; I have been working on implementing some ext4 features that can work in ext3 mode in preparation towards bringing some more of the ext4 work Zheng Liu did for his GSoC 2010 project. The first of this features is nanosecond/birthtime timestamping, which basically means that if the filesystem was created with big inodes we carry the nanosecond and birthtime stamps that UFS2 has always supported. Apparently the appropriate fields in the on-disk inode have been approved for a long time but support for this in ext3 has not been widely distributed. In preparation for ext4 most linux distributions did enable by default such bigger inodes and some people do use nanosecond timestamps in ext3. I would really appreciate if ext2/ext3 users test this patch: http://people.freebsd.org/~pfg/patches/patch-ext2fs-ns_timestamps It should apply cleanly to 10-current or to a recent 9.0-stable. IMPORTANT: do avoid testing this patch in old systems that may be using Extended Attributes. You will probably not notice any change but do let me know how it goes. best regards, Pedro. ps. FWIW, while here I also implemented the basic inode versioning stuff from Lustre but it's doesn't seem to any use: http://wiki.lustre.org/images/1/16/Inodever-HLD.pdf