Date: Mon, 24 Mar 2014 19:00:59 -0400 From: mikej <mikej@mikej.com> To: Karl Denninger <karl@denninger.net>, <freebsd-fs@freebsd.org> Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix [SB QUAR: Fri Mar 21 16:01:12 2014] Message-ID: <c42f6df8791cac4479d9e49a20c150be@mail.mikej.com> In-Reply-To: <53304D8D.7080107@denninger.net> References: <201403201710.s2KHA0e9043051@freefall.freebsd.org> <31e2f5092048c128add6f4cd3d136f4d@mail.mikej.com> <532CD3BC.5020305@denninger.net> <b389a3b124e16432b446d73e794e9c86@mail.mikej.com> <53304D8D.7080107@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Karl, I don't know why the PR system doesn't allow multiple attachments for patches to be sent so they can easily be downloaded. I continue to be unsuccessful at getting patches out of email that don't have line wrap are tab issues. So, again if you could send a link or post the latest patch I would be glad to let it churn. If I need an education about the PR system, please someone help me. That last patch you sent me privately applied like this against r263618M. Hmm... Looks like a new-style context diff to me... The text leading up to this was: -------------------------- |*** arc.c.original Thu Mar 13 09:18:48 2014 |--- arc.c Fri Mar 21 09:04:23 2014 -------------------------- Patching file arc.c using Plan A... Hunk #1 succeeded at 18. Hunk #2 succeeded at 210. Hunk #3 succeeded at 281. Hunk #4 succeeded at 2539. Hunk #5 succeeded at 2582. Hunk #6 succeeded at 2599. done [root@charon /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs]# but failed to build. I hope others chime in but so far for the updates from these patches have been positive for me understanding though that everyone's workloads vary. Unless I knowingly drive the system into resource starvation everything is much better in not holding wired and not driving swap. My main background resource hogs are poudriere a random load. Regards, --mikej On 2014-03-24 11:21, Karl Denninger wrote: > Mike; > > Did the patch apply cleanly? > > That declaration is in <sys/vmmeter.h> which should be included up > near the top of the file if NEWRECLAIM is defined and the patch > applied ok. > > See here, last entry (that's the most-recent rev), revert to the > stock arc.c (or put arc.c.orig back, which should be the original > file) and re-apply it. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=187594 > > Lastly, what OS rev are you running? The patch is against 10-STABLE; > it is ok against both the current checked-in rev of arc.c and the > previous (prior to the new feature flags being added a week or so > ago) > rev back. > > It sounds like the include didn't get applied. > > On 3/24/2014 10:00 AM, mikej wrote: >> Karl, >> >> Not being a C coder it appears a declaration is missing. >> >> --- arc.o --- >> >> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2614:15: >> error: use of undeclared identifier 'cnt' >> freepages = cnt.v_free_target - (cnt.v_free_target / >> 33); >> >> Thanks again, >> Michael Jung >> >> >> On 2014-03-21 20:05, Karl Denninger wrote: >>> Here 'ya go... >>> >>> Please keep me posted (the list is best as the more public >>> commentary >>> the better, and if this needs more tuning that's the way to find >>> out!) >>> on how it works for you. >>> >>> I have it in production at this point and am happy with it -- the >>> current default is at the pager "wakeup" level less 3%, but it of >>> course can be tuned manually. >>> >>> On 3/21/2014 3:59 PM, mikej wrote: >>> >>>> Karl, >>>> >>>> I've looked at my raw mailbox and something is trashing tabs and >>>> line >>>> length for your more recent patches in email. >>>> >>>> I did not see any attachments, nor updates to the PR for download >>>> - >>>> would >>>> you mind sending me the latest patch as an attachment? >>>> >>>> Thanks for your work, I believe this is going to add real >>>> stability >>>> >>>> without having to set vfs.zfs.arc_max and other tunables. >>>> >>>> Kind regards, >>>> >>>> Michael Jung >>>> >>>> On 2014-03-20 13:10, Karl Denninger wrote: >>>> >>>>> The following reply was made to PR kern/187594; it has been noted >>>>> by GNATS. >>>>> >>>>> From: Karl Denninger [1] >>>>> To: bug-followup@FreeBSD.org [2], karl@fs.denninger.net [3] >>>>> Cc: >>>>> Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem >>>>> and fix >>>>> Date: Thu, 20 Mar 2014 12:00:54 -0500 >>>>> >>>>> This is a cryptographically signed message in MIME format. >>>>> >>>>> --------------ms010508000607000909070805 >>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>>>> Content-Transfer-Encoding: quoted-printable >>>>> >>>>> Responsive to avg's comment and with another overnight and >>>>> daytime load=20 >>>>> of testing on multiple machines with varying memory configs >>>>> from 4-24GB=20 >>>>> of RAM here is another version of the patch. >>>>> >>>>> The differences are: >>>>> >>>>> 1. No longer use kernel_sysctlbyname, include the VM header >>>>> file and get = >>>>> >>>>> the values directly (less overhead.) Remove the variables no >>>>> longer need= >>>>> ed. >>>>> >>>>> 2. Set the default free RAM level for ARC shrinkage to >>>>> v_free_target=20 >>>>> less 3% as I was able to provoke a stall once with it set to a >>>>> 5%=20 >>>>> reservation, was able to provoke it with the parameter set to >>>>> 10% with a = >>>>> >>>>> lot of work and was able to do so "on demand" with it set to >>>>> 20%. With=20 >>>>> a 5% invasion initiating a scrub with very heavy I/O and image >>>>> load=20 >>>>> (hundreds of web and database processes) provoked a ~10 second >>>>> system=20 >>>>> stall. With it set to 3% I have not been able to reproduce >>>>> the stall=20 >>>>> yet the inactive page count remains stable even under extremely >>>>> heavy=20 >>>>> load, indicating that page-stealing remains effective when >>>>> required. =20 >>>>> Note that for my workload even with this level set above >>>>> v_free_target,=20 >>>>> which would imply no page stealing by the VM system before ARC >>>>> expansion = >>>>> >>>>> is halted, I do not get unbridled inactive page growth. >>>>> >>>>> As before vfs.zfs.zrc_freepages and >>>>> vfs.zfs.arc_freepage_percent remain=20 >>>>> as accessible knobs if you wish to twist them for some reason >>>>> to=20 >>>>> compensate for an unusual load profile or machine >>>>> configuration. >>>>> >>>>> *** arc.c.original Thu Mar 13 09:18:48 2014 >>>>> --- arc.c Thu Mar 20 11:51:48 2014 >>>>> *************** >>>>> *** 18,23 **** >>>>> --- 18,94 ---- >>>>> * >>>>> * CDDL HEADER END >>>>> */ >>>>> + >>>>> + /* Karl Denninger (karl@denninger.net [4]), 3/20/2014, >>>>> FreeBSD-specific >>>>> + * >>>>> + * If "NEWRECLAIM" is defined, change the "low memory" >>>>> warning that cau= >>>>> ses >>>>> + * the ARC cache to be pared down. The reason for the >>>>> change is that t= >>>>> he >>>>> + * apparent attempted algorithm is to start evicting ARC >>>>> cache when fre= >>>>> e >>>>> + * pages fall below 25% of installed RAM. This maps >>>>> reasonably well to= >>>>> how >>>>> + * Solaris is documented to behave; when "lotsfree" is >>>>> invaded ZFS is t= >>>>> old >>>>> + * to pare down. >>>>> + * >>>>> + * The problem is that on FreeBSD machines the system >>>>> doesn't appear to= >>>>> be >>>>> + * getting what the authors of the original code thought >>>>> they were look= >>>>> ing at >>>>> + * with its test -- or at least not what Solaris did -- and >>>>> as a result= >>>>> that >>>>> + * test never triggers. That leaves the only reclaim >>>>> trigger as the "p= >>>>> aging >>>>> + * needed" status flag, and by the time * that trips the >>>>> system is alre= >>>>> ady >>>>> + * in low-memory trouble. This can lead to severe >>>>> pathological behavio= >>>>> r >>>>> + * under the following scenario: >>>>> + * - The system starts to page and ARC is evicted. >>>>> + * - The system stops paging as ARC's eviction drops wired >>>>> RAM a bit. >>>>> + * - ARC starts increasing its allocation again, and wired >>>>> memory grows= >>>>> =2E >>>>> + * - A new image is activated, and the system once again >>>>> attempts to pa= >>>>> ge. >>>>> + * - ARC starts to be evicted again. >>>>> + * - Back to #2 >>>>> + * >>>>> + * Note that ZFS's ARC default (unless you override it in >>>>> /boot/loader.= >>>>> conf) >>>>> + * is to allow the ARC cache to grab nearly all of free RAM, >>>>> provided n= >>>>> obody >>>>> + * else needs it. That would be ok if we evicted cache >>>>> when required. >>>>> + * >>>>> + * Unfortunately the system can get into a state where it >>>>> never >>>>> + * manages to page anything of materiality back in, as if >>>>> there is acti= >>>>> ve >>>>> + * I/O the ARC will start grabbing space once again as soon >>>>> as the memo= >>>>> ry >>>>> + * contention state drops. For this reason the "paging is >>>>> occurring" f= >>>>> lag >>>>> + * should be the **last resort** condition for ARC eviction; >>>>> you want t= >>>>> o >>>>> + * (as Solaris does) start when there is material free RAM >>>>> left BUT the= >>>>> >>>>> + * vm system thinks it needs to be active to steal pages >>>>> back in the at= >>>>> tempt >>>>> + * to never get into the condition where you're potentially >>>>> paging off >>>>> + * executables in favor of leaving disk cache allocated. >>>>> + * >>>>> + * To fix this we change how we look at low memory, >>>>> declaring two new >>>>> + * runtime tunables. >>>>> + * >>>>> + * The new sysctls are: >>>>> + * vfs.zfs.arc_freepages (free pages required to call RAM >>>>> "sufficient")= >>>>> >>>>> + * vfs.zfs.arc_freepage_percent (additional reservation >>>>> percentage, def= >>>>> ault 0) >>>>> + * >>>>> + * vfs.zfs.arc_freepages is initialized from >>>>> vm.v_free_target, less 3%.= >>>>> >>>>> + * This should insure that we allow the VM system to steal >>>>> pages first,= >>>>> >>>>> + * but pare the cache before we suspend processes attempting >>>>> to get mor= >>>>> e >>>>> + * memory, thereby avoiding "stalls." You can set this >>>>> higher if you w= >>>>> ish, >>>>> + * or force a specific percentage reservation as well, but >>>>> doing so may= >>>>> >>>>> + * cause the cache to pare back while the VM system remains >>>>> willing to >>>>> + * allow "inactive" pages to accumulate. The challenge is >>>>> that image >>>>> + * activation can force things into the page space on a >>>>> repeated basis >>>>> + * if you allow this level to be too small (the above >>>>> pathological >>>>> + * behavior); the defaults should avoid that behavior but >>>>> the sysctls >>>>> + * are exposed should your workload require adjustment. >>>>> + * >>>>> + * If we're using this check for low memory we are replacing >>>>> the previo= >>>>> us >>>>> + * ones, including the oddball "random" reclaim that appears >>>>> to fire fa= >>>>> r >>>>> + * more often than it should. We still trigger if the >>>>> system pages. >>>>> + * >>>>> + * If you turn on NEWRECLAIM_DEBUG then the kernel will >>>>> print on the co= >>>>> nsole >>>>> + * status messages when the reclaim status trips on and off, >>>>> along with= >>>>> the >>>>> + * page count aggregate that triggered it (and the free >>>>> space) for each= >>>>> >>>>> + * event. >>>>> + */ >>>>> + >>>>> + #define NEWRECLAIM >>>>> + #undef NEWRECLAIM_DEBUG >>>>> + >>>>> + >>>>> /* >>>>> * Copyright (c) 2005, 2010, Oracle and/or its >>>>> affiliates. All rights = >>>>> reserved. >>>>> * Copyright (c) 2013 by Delphix. All rights reserved. >>>>> *************** >>>>> *** 139,144 **** >>>>> --- 210,222 ---- >>>>> =20 >>>>> #include >>>>> =20 >>>>> + #ifdef NEWRECLAIM >>>>> + #ifdef __FreeBSD__ >>>>> + #include >>>>> + #include >>>>> + #endif >>>>> + #endif /* NEWRECLAIM */ >>>>> + >>>>> #ifdef illumos >>>>> #ifndef _KERNEL >>>>> /* set with ZFS_DEBUG=3Dwatch, to enable watchpoints on >>>>> frozen buffers= >>>>> */ >>>>> *************** >>>>> *** 203,218 **** >>>>> --- 281,316 ---- >>>>> int zfs_arc_shrink_shift =3D 0; >>>>> int zfs_arc_p_min_shift =3D 0; >>>>> int zfs_disable_dup_eviction =3D 0; >>>>> + #ifdef NEWRECLAIM >>>>> + #ifdef __FreeBSD__ >>>>> + static int freepages =3D 0; /* This much memory >>>>> is considered critical = >>>>> */ >>>>> + static int percent_target =3D 0; /* Additionally >>>>> reserve "X" percent fr= >>>>> ee RAM */ >>>>> + #endif /* __FreeBSD__ */ >>>>> + #endif /* NEWRECLAIM */ >>>>> =20 >>>>> TUNABLE_QUAD("vfs.zfs.arc_max", &zfs_arc_max); >>>>> TUNABLE_QUAD("vfs.zfs.arc_min", &zfs_arc_min); >>>>> TUNABLE_QUAD("vfs.zfs.arc_meta_limit", >>>>> &zfs_arc_meta_limit); >>>>> + #ifdef NEWRECLAIM >>>>> + #ifdef __FreeBSD__ >>>>> + TUNABLE_INT("vfs.zfs.arc_freepages", &freepages); >>>>> + TUNABLE_INT("vfs.zfs.arc_freepage_percent", &percent_target); >>>>> >>>>> + #endif /* __FreeBSD__ */ >>>>> + #endif /* NEWRECLAIM */ >>>>> + >>>>> SYSCTL_DECL(_vfs_zfs); >>>>> SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_max, CTLFLAG_RDTUN, >>>>> &zfs_arc_max,= >>>>> 0, >>>>> "Maximum ARC size"); >>>>> SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_min, CTLFLAG_RDTUN, >>>>> &zfs_arc_min,= >>>>> 0, >>>>> "Minimum ARC size"); >>>>> =20 >>>>> + #ifdef NEWRECLAIM >>>>> + #ifdef __FreeBSD__ >>>>> + SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_freepages, CTLFLAG_RWTUN, >>>>> &freepages= >>>>> , 0, "ARC Free RAM Pages Required"); >>>>> + SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_freepage_percent, >>>>> CTLFLAG_RWTUN, &pe= >>>>> rcent_target, 0, "ARC Free RAM Target percentage"); >>>>> + #endif /* __FreeBSD__ */ >>>>> + #endif /* NEWRECLAIM */ >>>>> + >>>>> /* >>>>> * Note that buffers can be in one of 6 states: >>>>> * ARC_anon - anonymous (discussed below) >>>>> *************** >>>>> *** 2438,2443 **** >>>>> --- 2536,2546 ---- >>>>> { >>>>> =20 >>>>> #ifdef _KERNEL >>>>> + #ifdef NEWRECLAIM_DEBUG >>>>> + static int xval =3D -1; >>>>> + static int oldpercent =3D 0; >>>>> + static int oldfreepages =3D 0; >>>>> + #endif /* NEWRECLAIM_DEBUG */ >>>>> =20 >>>>> if (needfree) >>>>> return (1); >>>>> *************** >>>>> *** 2476,2481 **** >>>>> --- 2579,2585 ---- >>>>> return (1); >>>>> =20 >>>>> #if defined(__i386) >>>>> + >>>>> /* >>>>> * If we're on an i386 platform, it's possible >>>>> that we'll exhaust the= >>>>> >>>>> * kernel heap space before we ever run out of >>>>> available physical >>>>> *************** >>>>> *** 2492,2502 **** >>>>> return (1); >>>>> #endif >>>>> #else /* !sun */ >>>>> if (kmem_used() > (kmem_size() * 3) / 4) >>>>> return (1); >>>>> #endif /* sun */ >>>>> =20 >>>>> - #else >>>>> if (spa_get_random(100) =3D=3D 0) >>>>> return (1); >>>>> #endif >>>>> --- 2596,2658 ---- >>>>> return (1); >>>>> #endif >>>>> #else /* !sun */ >>>>> + >>>>> + #ifdef NEWRECLAIM >>>>> + #ifdef __FreeBSD__ >>>>> + /* >>>>> + * Implement the new tunable free RAM algorithm. We check >>>>> the free pag= >>>>> es >>>>> + * against the minimum specified target and the percentage >>>>> that should = >>>>> be >>>>> + * free. If we're low we ask for ARC cache shrinkage. If >>>>> this is defi= >>>>> ned >>>>> + * on a FreeBSD system the older checks are not performed. >>>>> + * >>>>> + * Check first to see if we need to init freepages, then >>>>> test. >>>>> + */ >>>>> + if (!freepages) { /* If zero then >>>>> (re)init */ >>>>> + freepages =3D cnt.v_free_target - >>>>> (cnt.v_free_target / 33); >>>>> + #ifdef NEWRECLAIM_DEBUG >>>>> + printf("ZFS ARC: Default >>>>> vfs.zfs.arc_freepages to [%u] [%u less 3%%]= >>>>> n", freepages, cnt.v_free_target); >>>>> + #endif /* NEWRECLAIM_DEBUG */ >>>>> + } >>>>> + #ifdef NEWRECLAIM_DEBUG >>>>> + if (percent_target !=3D oldpercent) { >>>>> + printf("ZFS ARC: Reservation percent change >>>>> to [%d], [%d] pages, [%d]= >>>>> freen", percent_target, cnt.v_page_count, cnt.v_free_count); >>>>> + oldpercent =3D percent_target; >>>>> + } >>>>> + if (freepages !=3D oldfreepages) { >>>>> + printf("ZFS ARC: Low RAM page change to [%d], >>>>> [%d] pages, [%d] freen= >>>>> ", freepages, cnt.v_page_count, cnt.v_free_count); >>>>> + oldfreepages =3D freepages; >>>>> + } >>>>> + #endif /* NEWRECLAIM_DEBUG */ >>>>> + /* >>>>> + * Now figure out how much free RAM we require to call the >>>>> ARC cache st= >>>>> atus >>>>> + * "ok". Add the percentage specified of the total to the >>>>> base require= >>>>> ment. >>>>> + */ >>>>> + >>>>> + if (cnt.v_free_count < freepages + ((cnt.v_page_count >>>>> / 100) * percent= >>>>> _target)) { >>>>> + #ifdef NEWRECLAIM_DEBUG >>>>> + if (xval !=3D 1) { >>>>> + printf("ZFS ARC: RECLAIM total %u, >>>>> free %u, free pct (%u), reserved = >>>>> (%u), target pct (%u)n", cnt.v_page_count, cnt.v_free_count, >>>>> ((cnt.v_fre= >>>>> e_count * 100) / cnt.v_page_count), freepages, percent_target); >>>>> >>>>> + xval =3D 1; >>>>> + } >>>>> + #endif /* NEWRECLAIM_DEBUG */ >>>>> + return(1); >>>>> + } else { >>>>> + #ifdef NEWRECLAIM_DEBUG >>>>> + if (xval !=3D 0) { >>>>> + printf("ZFS ARC: NORMAL total %u, >>>>> free %u, free pct (%u), reserved (= >>>>> %u), target pct (%u)n", cnt.v_page_count, cnt.v_free_count, >>>>> ((cnt.v_free= >>>>> _count * 100) / cnt.v_page_count), freepages, percent_target); >>>>> + xval =3D 0; >>>>> + } >>>>> + #endif /* NEWRECLAIM_DEBUG */ >>>>> + return(0); >>>>> + } >>>>> + >>>>> + #endif /* __FreeBSD__ */ >>>>> + #endif /* NEWRECLAIM */ >>>>> + >>>>> if (kmem_used() > (kmem_size() * 3) / 4) >>>>> return (1); >>>>> #endif /* sun */ >>>>> =20 >>>>> if (spa_get_random(100) =3D=3D 0) >>>>> return (1); >>>>> #endif >>>>> >>>>> --=20 >>>>> -- Karl >>>>> karl@denninger.net [5] >>>>> >>>>> --------------ms010508000607000909070805 >>>>> Content-Type: application/pkcs7-signature; name="smime.p7s" >>>>> Content-Transfer-Encoding: base64 >>>>> Content-Disposition: attachment; filename="smime.p7s" >>>>> Content-Description: S/MIME Cryptographic Signature >>>>> >>>>> >>>>> >>>> >>> >>> >>> MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC >>>>> >>>>> >>>>> >>>> >>> >>> >>> BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI >>>>> >>>>> >>>>> >>>> >>> >>> >>> EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM >>>>> >>>>> >>>>> >>>> >>> >>> >>> TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv >>>>> >>>>> >>>>> >>>> >>> >>> >>> bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 >>>>> >>>>> >>>>> >>>> >>> >>> >>> MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg >>>>> >>>>> >>>>> >>>> >>> >>> >>> RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG >>>>> >>>>> >>>>> >>>> >>> >>> >>> SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th >>>>> >>>>> >>>>> >>>> >>> >>> >>> d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W >>>>> >>>>> >>>>> >>>> >>> >>> >>> 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV >>>>> >>>>> >>>>> >>>> >>> >>> >>> jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 >>>>> >>>>> >>>>> >>>> >>> >>> >>> SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY >>>>> >>>>> >>>>> >>>> >>> >>> >>> 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 >>>>> >>>>> >>>>> >>>> >>> >>> >>> Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 >>>>> >>>>> >>>>> >>>> >>> >>> >>> GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci >>>>> >>>>> >>>>> >>>> >>> >>> >>> WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN >>>>> >>>>> >>>>> >>>> >>> >>> >>> nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB >>>>> >>>>> >>>>> >>>> >>> >>> >>> o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg >>>>> >>>>> >>>>> >>>> >>> >>> >>> hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 >>>>> >>>>> >>>>> >>>> >>> >>> >>> +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG >>>>> >>>>> >>>>> >>>> >>> >>> >>> CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy >>>>> >>>>> >>>>> >>>> >>> >>> >>> bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO >>>>> >>>>> >>>>> >>>> >>> >>> >>> 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c >>>>> >>>>> >>>>> >>>> >>> >>> >>> L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 >>>>> >>>>> >>>>> >>>> >>> >>> >>> YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD >>>>> >>>>> >>>>> >>>> >>> >>> >>> pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE >>>>> >>>>> >>>>> >>>> >>> >>> >>> f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk >>>>> >>>>> >>>>> >>>> >>> >>> >>> YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD >>>>> >>>>> >>>>> >>>> >>> >>> >>> VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 >>>>> >>>>> >>>>> >>>> >>> >>> >>> aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ >>>>> >>>>> >>>>> >>>> >>> >>> >>> KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMjAxNzAwNTRaMCMGCSqGSIb3DQEJBDEW >>>>> >>>>> >>>>> >>>> >>> >>> >>> BBQrRNHcjN1dwAlZu7blrh+3Vu7++TBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL >>>>> >>>>> >>>>> >>>> >>> >>> >>> BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA >>>>> >>>>> >>>>> >>>> >>> >>> >>> MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV >>>>> >>>>> >>>>> >>>> >>> >>> >>> BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT >>>>> >>>>> >>>>> >>>> >>> >>> >>> EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq >>>>> >>>>> >>>>> >>>> >>> >>> >>> hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG >>>>> >>>>> >>>>> >>>> >>> >>> >>> 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV >>>>> >>>>> >>>>> >>>> >>> >>> >>> BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk >>>>> >>>>> >>>>> >>>> >>> >>> >>> YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh >>>>> >>>>> >>>>> >>>> >>> >>> >>> c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAsIbg6OIk2XnrPw25FA9+s4FCnKlo >>>>> >>>>> >>>>> >>>> >>> >>> >>> Wz3KtfA59Gf2jX8yKEQW26k1QK6w62Zc8KaB5LypaOK1rZ5bipeu6rGHhgaG1oPXUMmcc2p4 >>>>> >>>>> >>>>> >>>> >>> >>> >>> E18URDskAfn5apoJ/n54nR94OqHfQ/EPBx711pxYtAGBLFOOzwrU2MEZCl2KBydI+Bw/E75R >>>>> >>>>> >>>>> >>>> >>> >>> >>> WRIk6y0NqSWjgVWU2tJwnOEZj/2UGQCSvJ7h5t1n7idbDIfT88/hvAW3b3knRwPxwpZretXq >>>>> >>>>> >>>>> >>>> >>> >>> >>> 2BGgmv8lojr7Zui5sR/YdDjSK2yGHqo0mWkSAHp0Wts8okcoJNZSEispFRh56MWCIoJ51cki >>>>> >>>>> >>>>> >>>> >>> >>> >>> pCZH/vX1EEsfka3CrlE7LWABAYf1biy+Xq/Bfxgq9oAaknGF2yM0jgR7xnjLYLvbv5pjt7ar >>>>> >>>>> >>>>> >>>> >>> >>> >>> TH2JslJMYkJPKiYFJNEgVJ9wTVQtrCPJQPTk3R1qD3YFraly5Mgjwy5Ax5n8SW858WWOxHeP >>>>> >>>>> >>>>> >>>> >>> >>> >>> vmL0j1boO0Re9qeAb9v/z8z3NPkFPZhBrEz3g6INCWil+2Vx1yruJvxm1oN9OMQSt2qY38rj >>>>> >>>>> >>>>> >>>> >>> >>> >>> XWhWVxoQtW39LZc/xSNR41DQXvPJ8VOvyrmvLm7uTm4+lQYVUwNuLNbDFlj8slkAeXF/eR1S >>>>> >>>>> >>>>> >>>> >>> >>> >>> 4VuWtwexxCco+xGjbPTcZgap976XsvlRWOmjmwqZyGNuW7ZmcODQPFjQvpnBkx9Rm5cLndK6 >>>>> >>>>> OVorTQkAAAAAAAA= >>>>> --------------ms010508000607000909070805-- >>>>> >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org [6] mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs [7] >>>>> To unsubscribe, send any mail to >>>>> "freebsd-fs-unsubscribe@freebsd.org" [8] >>> >>> -- Karl Denninger >>> karl@denninger.net [9] >>> _The Market Ticker_ >>> >>> >>> >>> Links: >>> ------ >>> [1] mailto:karl@denninger.net >>> [2] mailto:bug-followup@FreeBSD.org >>> [3] mailto:karl@fs.denninger.net >>> [4] mailto:karl@denninger.net >>> [5] mailto:karl@denninger.net >>> [6] mailto:freebsd-fs@freebsd.org >>> [7] http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> [8] mailto:freebsd-fs-unsubscribe@freebsd.org >>> [9] mailto:karl@denninger.net >> >> >> >> %SPAMBLOCK-SYS: Matched [+mikej <mikej@mikej.com>], message ok >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c42f6df8791cac4479d9e49a20c150be>