Date: Mon, 24 Mar 2014 10:21:49 -0500 From: Karl Denninger <karl@denninger.net> To: mikej <mikej@mikej.com>, freebsd-fs@freebsd.org Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix [SB QUAR: Fri Mar 21 16:01:12 2014] Message-ID: <53304D8D.7080107@denninger.net> In-Reply-To: <b389a3b124e16432b446d73e794e9c86@mail.mikej.com> References: <201403201710.s2KHA0e9043051@freefall.freebsd.org> <31e2f5092048c128add6f4cd3d136f4d@mail.mikej.com> <532CD3BC.5020305@denninger.net> <b389a3b124e16432b446d73e794e9c86@mail.mikej.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms010404080205080205080700 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Mike; Did the patch apply cleanly? That declaration is in <sys/vmmeter.h> which should be included up near=20 the top of the file if NEWRECLAIM is defined and the patch applied ok. See here, last entry (that's the most-recent rev), revert to the stock=20 arc.c (or put arc.c.orig back, which should be the original file) and=20 re-apply it. http://www.freebsd.org/cgi/query-pr.cgi?pr=3D187594 Lastly, what OS rev are you running? The patch is against 10-STABLE; it = is ok against both the current checked-in rev of arc.c and the previous=20 (prior to the new feature flags being added a week or so ago) rev back. It sounds like the include didn't get applied. On 3/24/2014 10:00 AM, mikej wrote: > Karl, > > Not being a C coder it appears a declaration is missing. > > --- arc.o --- > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/z= fs/arc.c:2614:15:=20 > error: use of undeclared identifier 'cnt' > freepages =3D cnt.v_free_target - (cnt.v_free_target / = 33); > > Thanks again, > Michael Jung > > > On 2014-03-21 20:05, Karl Denninger wrote: >> Here 'ya go... >> >> Please keep me posted (the list is best as the more public commentary= >> the better, and if this needs more tuning that's the way to find out!)= >> on how it works for you. >> >> I have it in production at this point and am happy with it -- the >> current default is at the pager "wakeup" level less 3%, but it of >> course can be tuned manually. >> >> On 3/21/2014 3:59 PM, mikej wrote: >> >>> Karl, >>> >>> I've looked at my raw mailbox and something is trashing tabs and >>> line >>> length for your more recent patches in email. >>> >>> I did not see any attachments, nor updates to the PR for download - >>> would >>> you mind sending me the latest patch as an attachment? >>> >>> Thanks for your work, I believe this is going to add real stability >>> >>> without having to set vfs.zfs.arc_max and other tunables. >>> >>> Kind regards, >>> >>> Michael Jung >>> >>> On 2014-03-20 13:10, Karl Denninger wrote: >>> >>>> The following reply was made to PR kern/187594; it has been noted >>>> by GNATS. >>>> >>>> From: Karl Denninger [1] >>>> To: bug-followup@FreeBSD.org [2], karl@fs.denninger.net [3] >>>> Cc: >>>> Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem >>>> and fix >>>> Date: Thu, 20 Mar 2014 12:00:54 -0500 >>>> >>>> This is a cryptographically signed message in MIME format. >>>> >>>> --------------ms010508000607000909070805 >>>> Content-Type: text/plain; charset=3DISO-8859-1; format=3Dflowed >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> Responsive to avg's comment and with another overnight and >>>> daytime load=3D20 >>>> of testing on multiple machines with varying memory configs >>>> from 4-24GB=3D20 >>>> of RAM here is another version of the patch. >>>> >>>> The differences are: >>>> >>>> 1. No longer use kernel_sysctlbyname, include the VM header >>>> file and get =3D >>>> >>>> the values directly (less overhead.) Remove the variables no >>>> longer need=3D >>>> ed. >>>> >>>> 2. Set the default free RAM level for ARC shrinkage to >>>> v_free_target=3D20 >>>> less 3% as I was able to provoke a stall once with it set to a >>>> 5%=3D20 >>>> reservation, was able to provoke it with the parameter set to >>>> 10% with a =3D >>>> >>>> lot of work and was able to do so "on demand" with it set to >>>> 20%. With=3D20 >>>> a 5% invasion initiating a scrub with very heavy I/O and image >>>> load=3D20 >>>> (hundreds of web and database processes) provoked a ~10 second >>>> system=3D20 >>>> stall. With it set to 3% I have not been able to reproduce >>>> the stall=3D20 >>>> yet the inactive page count remains stable even under extremely >>>> heavy=3D20 >>>> load, indicating that page-stealing remains effective when >>>> required. =3D20 >>>> Note that for my workload even with this level set above >>>> v_free_target,=3D20 >>>> which would imply no page stealing by the VM system before ARC >>>> expansion =3D >>>> >>>> is halted, I do not get unbridled inactive page growth. >>>> >>>> As before vfs.zfs.zrc_freepages and >>>> vfs.zfs.arc_freepage_percent remain=3D20 >>>> as accessible knobs if you wish to twist them for some reason >>>> to=3D20 >>>> compensate for an unusual load profile or machine >>>> configuration. >>>> >>>> *** arc.c.original Thu Mar 13 09:18:48 2014 >>>> --- arc.c Thu Mar 20 11:51:48 2014 >>>> *************** >>>> *** 18,23 **** >>>> --- 18,94 ---- >>>> * >>>> * CDDL HEADER END >>>> */ >>>> + >>>> + /* Karl Denninger (karl@denninger.net [4]), 3/20/2014, >>>> FreeBSD-specific >>>> + * >>>> + * If "NEWRECLAIM" is defined, change the "low memory" >>>> warning that cau=3D >>>> ses >>>> + * the ARC cache to be pared down. The reason for the >>>> change is that t=3D >>>> he >>>> + * apparent attempted algorithm is to start evicting ARC >>>> cache when fre=3D >>>> e >>>> + * pages fall below 25% of installed RAM. This maps >>>> reasonably well to=3D >>>> how >>>> + * Solaris is documented to behave; when "lotsfree" is >>>> invaded ZFS is t=3D >>>> old >>>> + * to pare down. >>>> + * >>>> + * The problem is that on FreeBSD machines the system >>>> doesn't appear to=3D >>>> be >>>> + * getting what the authors of the original code thought >>>> they were look=3D >>>> ing at >>>> + * with its test -- or at least not what Solaris did -- and >>>> as a result=3D >>>> that >>>> + * test never triggers. That leaves the only reclaim >>>> trigger as the "p=3D >>>> aging >>>> + * needed" status flag, and by the time * that trips the >>>> system is alre=3D >>>> ady >>>> + * in low-memory trouble. This can lead to severe >>>> pathological behavio=3D >>>> r >>>> + * under the following scenario: >>>> + * - The system starts to page and ARC is evicted. >>>> + * - The system stops paging as ARC's eviction drops wired >>>> RAM a bit. >>>> + * - ARC starts increasing its allocation again, and wired >>>> memory grows=3D >>>> =3D2E >>>> + * - A new image is activated, and the system once again >>>> attempts to pa=3D >>>> ge. >>>> + * - ARC starts to be evicted again. >>>> + * - Back to #2 >>>> + * >>>> + * Note that ZFS's ARC default (unless you override it in >>>> /boot/loader.=3D >>>> conf) >>>> + * is to allow the ARC cache to grab nearly all of free RAM, >>>> provided n=3D >>>> obody >>>> + * else needs it. That would be ok if we evicted cache >>>> when required. >>>> + * >>>> + * Unfortunately the system can get into a state where it >>>> never >>>> + * manages to page anything of materiality back in, as if >>>> there is acti=3D >>>> ve >>>> + * I/O the ARC will start grabbing space once again as soon >>>> as the memo=3D >>>> ry >>>> + * contention state drops. For this reason the "paging is >>>> occurring" f=3D >>>> lag >>>> + * should be the **last resort** condition for ARC eviction; >>>> you want t=3D >>>> o >>>> + * (as Solaris does) start when there is material free RAM >>>> left BUT the=3D >>>> >>>> + * vm system thinks it needs to be active to steal pages >>>> back in the at=3D >>>> tempt >>>> + * to never get into the condition where you're potentially >>>> paging off >>>> + * executables in favor of leaving disk cache allocated. >>>> + * >>>> + * To fix this we change how we look at low memory, >>>> declaring two new >>>> + * runtime tunables. >>>> + * >>>> + * The new sysctls are: >>>> + * vfs.zfs.arc_freepages (free pages required to call RAM >>>> "sufficient")=3D >>>> >>>> + * vfs.zfs.arc_freepage_percent (additional reservation >>>> percentage, def=3D >>>> ault 0) >>>> + * >>>> + * vfs.zfs.arc_freepages is initialized from >>>> vm.v_free_target, less 3%.=3D >>>> >>>> + * This should insure that we allow the VM system to steal >>>> pages first,=3D >>>> >>>> + * but pare the cache before we suspend processes attempting >>>> to get mor=3D >>>> e >>>> + * memory, thereby avoiding "stalls." You can set this >>>> higher if you w=3D >>>> ish, >>>> + * or force a specific percentage reservation as well, but >>>> doing so may=3D >>>> >>>> + * cause the cache to pare back while the VM system remains >>>> willing to >>>> + * allow "inactive" pages to accumulate. The challenge is >>>> that image >>>> + * activation can force things into the page space on a >>>> repeated basis >>>> + * if you allow this level to be too small (the above >>>> pathological >>>> + * behavior); the defaults should avoid that behavior but >>>> the sysctls >>>> + * are exposed should your workload require adjustment. >>>> + * >>>> + * If we're using this check for low memory we are replacing >>>> the previo=3D >>>> us >>>> + * ones, including the oddball "random" reclaim that appears >>>> to fire fa=3D >>>> r >>>> + * more often than it should. We still trigger if the >>>> system pages. >>>> + * >>>> + * If you turn on NEWRECLAIM_DEBUG then the kernel will >>>> print on the co=3D >>>> nsole >>>> + * status messages when the reclaim status trips on and off, >>>> along with=3D >>>> the >>>> + * page count aggregate that triggered it (and the free >>>> space) for each=3D >>>> >>>> + * event. >>>> + */ >>>> + >>>> + #define NEWRECLAIM >>>> + #undef NEWRECLAIM_DEBUG >>>> + >>>> + >>>> /* >>>> * Copyright (c) 2005, 2010, Oracle and/or its >>>> affiliates. All rights =3D >>>> reserved. >>>> * Copyright (c) 2013 by Delphix. All rights reserved. >>>> *************** >>>> *** 139,144 **** >>>> --- 210,222 ---- >>>> =3D20 >>>> #include >>>> =3D20 >>>> + #ifdef NEWRECLAIM >>>> + #ifdef __FreeBSD__ >>>> + #include >>>> + #include >>>> + #endif >>>> + #endif /* NEWRECLAIM */ >>>> + >>>> #ifdef illumos >>>> #ifndef _KERNEL >>>> /* set with ZFS_DEBUG=3D3Dwatch, to enable watchpoints on >>>> frozen buffers=3D >>>> */ >>>> *************** >>>> *** 203,218 **** >>>> --- 281,316 ---- >>>> int zfs_arc_shrink_shift =3D3D 0; >>>> int zfs_arc_p_min_shift =3D3D 0; >>>> int zfs_disable_dup_eviction =3D3D 0; >>>> + #ifdef NEWRECLAIM >>>> + #ifdef __FreeBSD__ >>>> + static int freepages =3D3D 0; /* This much memory >>>> is considered critical =3D >>>> */ >>>> + static int percent_target =3D3D 0; /* Additionally >>>> reserve "X" percent fr=3D >>>> ee RAM */ >>>> + #endif /* __FreeBSD__ */ >>>> + #endif /* NEWRECLAIM */ >>>> =3D20 >>>> TUNABLE_QUAD("vfs.zfs.arc_max", &zfs_arc_max); >>>> TUNABLE_QUAD("vfs.zfs.arc_min", &zfs_arc_min); >>>> TUNABLE_QUAD("vfs.zfs.arc_meta_limit", >>>> &zfs_arc_meta_limit); >>>> + #ifdef NEWRECLAIM >>>> + #ifdef __FreeBSD__ >>>> + TUNABLE_INT("vfs.zfs.arc_freepages", &freepages); >>>> + TUNABLE_INT("vfs.zfs.arc_freepage_percent", &percent_target); >>>> >>>> + #endif /* __FreeBSD__ */ >>>> + #endif /* NEWRECLAIM */ >>>> + >>>> SYSCTL_DECL(_vfs_zfs); >>>> SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_max, CTLFLAG_RDTUN, >>>> &zfs_arc_max,=3D >>>> 0, >>>> "Maximum ARC size"); >>>> SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_min, CTLFLAG_RDTUN, >>>> &zfs_arc_min,=3D >>>> 0, >>>> "Minimum ARC size"); >>>> =3D20 >>>> + #ifdef NEWRECLAIM >>>> + #ifdef __FreeBSD__ >>>> + SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_freepages, CTLFLAG_RWTUN, >>>> &freepages=3D >>>> , 0, "ARC Free RAM Pages Required"); >>>> + SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_freepage_percent, >>>> CTLFLAG_RWTUN, &pe=3D >>>> rcent_target, 0, "ARC Free RAM Target percentage"); >>>> + #endif /* __FreeBSD__ */ >>>> + #endif /* NEWRECLAIM */ >>>> + >>>> /* >>>> * Note that buffers can be in one of 6 states: >>>> * ARC_anon - anonymous (discussed below) >>>> *************** >>>> *** 2438,2443 **** >>>> --- 2536,2546 ---- >>>> { >>>> =3D20 >>>> #ifdef _KERNEL >>>> + #ifdef NEWRECLAIM_DEBUG >>>> + static int xval =3D3D -1; >>>> + static int oldpercent =3D3D 0; >>>> + static int oldfreepages =3D3D 0; >>>> + #endif /* NEWRECLAIM_DEBUG */ >>>> =3D20 >>>> if (needfree) >>>> return (1); >>>> *************** >>>> *** 2476,2481 **** >>>> --- 2579,2585 ---- >>>> return (1); >>>> =3D20 >>>> #if defined(__i386) >>>> + >>>> /* >>>> * If we're on an i386 platform, it's possible >>>> that we'll exhaust the=3D >>>> >>>> * kernel heap space before we ever run out of >>>> available physical >>>> *************** >>>> *** 2492,2502 **** >>>> return (1); >>>> #endif >>>> #else /* !sun */ >>>> if (kmem_used() > (kmem_size() * 3) / 4) >>>> return (1); >>>> #endif /* sun */ >>>> =3D20 >>>> - #else >>>> if (spa_get_random(100) =3D3D=3D3D 0) >>>> return (1); >>>> #endif >>>> --- 2596,2658 ---- >>>> return (1); >>>> #endif >>>> #else /* !sun */ >>>> + >>>> + #ifdef NEWRECLAIM >>>> + #ifdef __FreeBSD__ >>>> + /* >>>> + * Implement the new tunable free RAM algorithm. We check >>>> the free pag=3D >>>> es >>>> + * against the minimum specified target and the percentage >>>> that should =3D >>>> be >>>> + * free. If we're low we ask for ARC cache shrinkage. If >>>> this is defi=3D >>>> ned >>>> + * on a FreeBSD system the older checks are not performed. >>>> + * >>>> + * Check first to see if we need to init freepages, then >>>> test. >>>> + */ >>>> + if (!freepages) { /* If zero then >>>> (re)init */ >>>> + freepages =3D3D cnt.v_free_target - >>>> (cnt.v_free_target / 33); >>>> + #ifdef NEWRECLAIM_DEBUG >>>> + printf("ZFS ARC: Default >>>> vfs.zfs.arc_freepages to [%u] [%u less 3%%]=3D >>>> n", freepages, cnt.v_free_target); >>>> + #endif /* NEWRECLAIM_DEBUG */ >>>> + } >>>> + #ifdef NEWRECLAIM_DEBUG >>>> + if (percent_target !=3D3D oldpercent) { >>>> + printf("ZFS ARC: Reservation percent change >>>> to [%d], [%d] pages, [%d]=3D >>>> freen", percent_target, cnt.v_page_count, cnt.v_free_count); >>>> + oldpercent =3D3D percent_target; >>>> + } >>>> + if (freepages !=3D3D oldfreepages) { >>>> + printf("ZFS ARC: Low RAM page change to [%d], >>>> [%d] pages, [%d] freen=3D >>>> ", freepages, cnt.v_page_count, cnt.v_free_count); >>>> + oldfreepages =3D3D freepages; >>>> + } >>>> + #endif /* NEWRECLAIM_DEBUG */ >>>> + /* >>>> + * Now figure out how much free RAM we require to call the >>>> ARC cache st=3D >>>> atus >>>> + * "ok". Add the percentage specified of the total to the >>>> base require=3D >>>> ment. >>>> + */ >>>> + >>>> + if (cnt.v_free_count < freepages + ((cnt.v_page_count >>>> / 100) * percent=3D >>>> _target)) { >>>> + #ifdef NEWRECLAIM_DEBUG >>>> + if (xval !=3D3D 1) { >>>> + printf("ZFS ARC: RECLAIM total %u, >>>> free %u, free pct (%u), reserved =3D >>>> (%u), target pct (%u)n", cnt.v_page_count, cnt.v_free_count, >>>> ((cnt.v_fre=3D >>>> e_count * 100) / cnt.v_page_count), freepages, percent_target); >>>> >>>> + xval =3D3D 1; >>>> + } >>>> + #endif /* NEWRECLAIM_DEBUG */ >>>> + return(1); >>>> + } else { >>>> + #ifdef NEWRECLAIM_DEBUG >>>> + if (xval !=3D3D 0) { >>>> + printf("ZFS ARC: NORMAL total %u, >>>> free %u, free pct (%u), reserved (=3D >>>> %u), target pct (%u)n", cnt.v_page_count, cnt.v_free_count, >>>> ((cnt.v_free=3D >>>> _count * 100) / cnt.v_page_count), freepages, percent_target); >>>> + xval =3D3D 0; >>>> + } >>>> + #endif /* NEWRECLAIM_DEBUG */ >>>> + return(0); >>>> + } >>>> + >>>> + #endif /* __FreeBSD__ */ >>>> + #endif /* NEWRECLAIM */ >>>> + >>>> if (kmem_used() > (kmem_size() * 3) / 4) >>>> return (1); >>>> #endif /* sun */ >>>> =3D20 >>>> if (spa_get_random(100) =3D3D=3D3D 0) >>>> return (1); >>>> #endif >>>> >>>> --=3D20 >>>> -- Karl >>>> karl@denninger.net [5] >>>> >>>> --------------ms010508000607000909070805 >>>> Content-Type: application/pkcs7-signature; name=3D"smime.p7s" >>>> Content-Transfer-Encoding: base64 >>>> Content-Disposition: attachment; filename=3D"smime.p7s" >>>> Content-Description: S/MIME Cryptographic Signature >>>> >>>> >>>> >>> >> >> MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTz= CC >>>> >>>> >>>> >>> >> >> BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQ= QI >>>> >>>> >>>> >>> >> >> EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcy= BM >>>> >>>> >>>> >>> >> >> TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3= Rv >>>> >>>> >>>> >>> >> >> bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMz= E5 >>>> >>>> >>>> >>> >> >> MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcm= wg >>>> >>>> >>>> >>> >> >> RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCS= qG >>>> >>>> >>>> >>> >> >> SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6= th >>>> >>>> >>>> >>> >> >> d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf= 6W >>>> >>>> >>>> >>> >> >> 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLn= iV >>>> >>>> >>>> >>> >> >> jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIam= O5 >>>> >>>> >>>> >>> >> >> SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFN= oY >>>> >>>> >>>> >>> >> >> 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2= Z8 >>>> >>>> >>>> >>> >> >> Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvx= P4 >>>> >>>> >>>> >>> >> >> GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4R= ci >>>> >>>> >>>> >>> >> >> WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcb= JN >>>> >>>> >>>> >>> >> >> nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQ= AB >>>> >>>> >>>> >>> >> >> o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBg= lg >>>> >>>> >>>> >>> >> >> hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFH= w4 >>>> >>>> >>>> >>> >> >> +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MD= gG >>>> >>>> >>>> >>> >> >> CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLm= Ny >>>> >>>> >>>> >>> >> >> bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnox= LO >>>> >>>> >>>> >>> >> >> 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN= /c >>>> >>>> >>>> >>> >> >> L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIc= J1 >>>> >>>> >>>> >>> >> >> YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMc= WD >>>> >>>> >>>> >>> >> >> pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmN= wE >>>> >>>> >>>> >>> >> >> f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcm= lk >>>> >>>> >>>> >>> >> >> YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGg= YD >>>> >>>> >>>> >>> >> >> VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZX= J2 >>>> >>>> >>>> >>> >> >> aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCw= YJ >>>> >>>> >>>> >>> >> >> KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMjAxNzAwNTRaMCMGCSqGSIb3DQEJBD= EW >>>> >>>> >>>> >>> >> >> BBQrRNHcjN1dwAlZu7blrh+3Vu7++TBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKj= AL >>>> >>>> >>>> >>> >> >> BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAg= FA >>>> >>>> >>>> >>> >> >> MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBg= NV >>>> >>>> >>>> >>> >> >> BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBA= oT >>>> >>>> >>>> >>> >> >> EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBg= kq >>>> >>>> >>>> >>> >> >> hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2Bgsqhk= iG >>>> >>>> >>>> >>> >> >> 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBg= NV >>>> >>>> >>>> >>> >> >> BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3= Vk >>>> >>>> >>>> >>> >> >> YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdW= Rh >>>> >>>> >>>> >>> >> >> c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAsIbg6OIk2XnrPw25FA9+s4FCnK= lo >>>> >>>> >>>> >>> >> >> Wz3KtfA59Gf2jX8yKEQW26k1QK6w62Zc8KaB5LypaOK1rZ5bipeu6rGHhgaG1oPXUMmcc2= p4 >>>> >>>> >>>> >>> >> >> E18URDskAfn5apoJ/n54nR94OqHfQ/EPBx711pxYtAGBLFOOzwrU2MEZCl2KBydI+Bw/E7= 5R >>>> >>>> >>>> >>> >> >> WRIk6y0NqSWjgVWU2tJwnOEZj/2UGQCSvJ7h5t1n7idbDIfT88/hvAW3b3knRwPxwpZret= Xq >>>> >>>> >>>> >>> >> >> 2BGgmv8lojr7Zui5sR/YdDjSK2yGHqo0mWkSAHp0Wts8okcoJNZSEispFRh56MWCIoJ51c= ki >>>> >>>> >>>> >>> >> >> pCZH/vX1EEsfka3CrlE7LWABAYf1biy+Xq/Bfxgq9oAaknGF2yM0jgR7xnjLYLvbv5pjt7= ar >>>> >>>> >>>> >>> >> >> TH2JslJMYkJPKiYFJNEgVJ9wTVQtrCPJQPTk3R1qD3YFraly5Mgjwy5Ax5n8SW858WWOxH= eP >>>> >>>> >>>> >>> >> >> vmL0j1boO0Re9qeAb9v/z8z3NPkFPZhBrEz3g6INCWil+2Vx1yruJvxm1oN9OMQSt2qY38= rj >>>> >>>> >>>> >>> >> >> XWhWVxoQtW39LZc/xSNR41DQXvPJ8VOvyrmvLm7uTm4+lQYVUwNuLNbDFlj8slkAeXF/eR= 1S >>>> >>>> >>>> >>> >> >> 4VuWtwexxCco+xGjbPTcZgap976XsvlRWOmjmwqZyGNuW7ZmcODQPFjQvpnBkx9Rm5cLnd= K6 >>>> >>>> OVorTQkAAAAAAAA=3D >>>> --------------ms010508000607000909070805-- >>>> >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org [6] mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs [7] >>>> To unsubscribe, send any mail to >>>> "freebsd-fs-unsubscribe@freebsd.org" [8] >> >> --=20 >> Karl Denninger >> karl@denninger.net [9] >> _The Market Ticker_ >> >> >> >> Links: >> ------ >> [1] mailto:karl@denninger.net >> [2] mailto:bug-followup@FreeBSD.org >> [3] mailto:karl@fs.denninger.net >> [4] mailto:karl@denninger.net >> [5] mailto:karl@denninger.net >> [6] mailto:freebsd-fs@freebsd.org >> [7] http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> [8] mailto:freebsd-fs-unsubscribe@freebsd.org >> [9] mailto:karl@denninger.net > > > > %SPAMBLOCK-SYS: Matched [+mikej <mikej@mikej.com>], message ok > --=20 -- Karl karl@denninger.net --------------ms010404080205080205080700 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFTzCC BUswggQzoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwgZ0xCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkqhkiG9w0BCQEWIGN1c3Rv bWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0MB4XDTEzMDgyNDE5MDM0NFoXDTE4MDgyMzE5 MDM0NFowWzELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExFzAVBgNVBAMTDkthcmwg RGVubmluZ2VyMSEwHwYJKoZIhvcNAQkBFhJrYXJsQGRlbm5pbmdlci5uZXQwggIiMA0GCSqG SIb3DQEBAQUAA4ICDwAwggIKAoICAQC5n2KBrBmG22nVntVdvgKCB9UcnapNThrW1L+dq6th d9l4mj+qYMUpJ+8I0rTbY1dn21IXQBoBQmy8t1doKwmTdQ59F0FwZEPt/fGbRgBKVt3Quf6W 6n7kRk9MG6gdD7V9vPpFV41e+5MWYtqGWY3ScDP8SyYLjL/Xgr+5KFKkDfuubK8DeNqdLniV jHo/vqmIgO+6NgzPGPgmbutzFQXlxUqjiNAAKzF2+Tkddi+WKABrcc/EqnBb0X8GdqcIamO5 SyVmuM+7Zdns7D9pcV16zMMQ8LfNFQCDvbCuuQKMDg2F22x5ekYXpwjqTyfjcHBkWC8vFNoY 5aFMdyiN/Kkz0/kduP2ekYOgkRqcShfLEcG9SQ4LQZgqjMpTjSOGzBr3tOvVn5LkSJSHW2Z8 Q0dxSkvFG2/lsOWFbwQeeZSaBi5vRZCYCOf5tRd1+E93FyQfpt4vsrXshIAk7IK7f0qXvxP4 GDli5PKIEubD2Bn+gp3vB/DkfKySh5NBHVB+OPCoXRUWBkQxme65wBO02OZZt0k8Iq0i4Rci WV6z+lQHqDKtaVGgMsHn6PoeYhjf5Al5SP+U3imTjF2aCca1iDB5JOccX04MNljvifXgcbJN nkMgrzmm1ZgJ1PLur/ADWPlnz45quOhHg1TfUCLfI/DzgG7Z6u+oy4siQuFr9QT0MQIDAQAB o4HWMIHTMAkGA1UdEwQCMAAwEQYJYIZIAYb4QgEBBAQDAgWgMAsGA1UdDwQEAwIF4DAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFHw4 +LnuALyLA5Cgy7T5ZAX1WzKPMB8GA1UdIwQYMBaAFF3U3hpBZq40HB5VM7B44/gmXiI0MDgG CWCGSAGG+EIBAwQrFilodHRwczovL2N1ZGFzeXN0ZW1zLm5ldDoxMTQ0My9yZXZva2VkLmNy bDANBgkqhkiG9w0BAQUFAAOCAQEAZ0L4tQbBd0hd4wuw/YVqEBDDXJ54q2AoqQAmsOlnoxLO 31ehM/LvrTIP4yK2u1VmXtUumQ4Ao15JFM+xmwqtEGsh70RRrfVBAGd7KOZ3GB39FP2TgN/c L5fJKVxOqvEnW6cL9QtvUlcM3hXg8kDv60OB+LIcSE/P3/s+0tEpWPjxm3LHVE7JmPbZIcJ1 YMoZvHh0NSjY5D0HZlwtbDO7pDz9sZf1QEOgjH828fhtborkaHaUI46pmrMjiBnY6ujXMcWD pxtikki0zY22nrxfTs5xDWGxyrc/cmucjxClJF6+OYVUSaZhiiHfa9Pr+41okLgsRB0AmNwE f6ItY3TI8DGCBQowggUGAgEBMIGjMIGdMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxvcmlk YTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRwwGgYD VQQDExNDdWRhIFN5c3RlbXMgTExDIENBMS8wLQYJKoZIhvcNAQkBFiBjdXN0b21lci1zZXJ2 aWNlQGN1ZGFzeXN0ZW1zLm5ldAIBCDAJBgUrDgMCGgUAoIICOzAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNDAzMjQxNTIxNDlaMCMGCSqGSIb3DQEJBDEW BBQ8R6eIDrkwnhANaqPEPIhxb7J+fDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjAL BglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFA MAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIG0BgkrBgEEAYI3EAQxgaYwgaMwgZ0xCzAJBgNV BAYTAlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoT EEN1ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExLzAtBgkq hkiG9w0BCQEWIGN1c3RvbWVyLXNlcnZpY2VAY3VkYXN5c3RlbXMubmV0AgEIMIG2BgsqhkiG 9w0BCRACCzGBpqCBozCBnTELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNV BAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3Vk YSBTeXN0ZW1zIExMQyBDQTEvMC0GCSqGSIb3DQEJARYgY3VzdG9tZXItc2VydmljZUBjdWRh c3lzdGVtcy5uZXQCAQgwDQYJKoZIhvcNAQEBBQAEggIAovOqhFRrVfYvrfAs/zuZnA8yvH0e wkqQa2c/sAqOSZQzJRzlMfSisD3N8WBROnZSnScnoBhi4sGIymEp27I+xxncXLQWv1VtP4D6 QeBWw5/fFcKvKk/4dbhsdHeznXikRn7pxpEApS8qKe7+pPGps7dGBeVZwkWudaAQD0G9lpsf HEmvXkIEEJgbkpywj8zfuZKnHv5DdDtQ5t2UKmM959fnUhWe7d2f4utnOdMO0LBzu+4r8vKw w5r2qkUuaJwijDGJ/1T5kwqTVFEfs2GKgabZYGRVUiawtnlWLhyLpaFYZshAGZNfFEAIdy7R CWLhSpgXpWXdM6VFdJtxzuPQQjNqEe85Y7xNwbRut8zrmP6cTX/XaX7sI8meWR+nvuill6BQ KSoL2r2Wc+89FIQ+aekKAgw8A7JN2/K4jU9lw8UrHW9ar9pKoMdngdOU02Hqc9YfaS/47BZP VmOGPl7yAODda6AGn5w4B1To95xgPS7kXxuSTvnGRQ20Tzfn1oZ5B8pe0JOTwY0zQg6NjqvW XgxISVRKX9IJqQqaREDfaYF6LzvNsmVrCWfz14/viycwM3vmCtlPywvAj+inNoWWWXzAEUJL A6xPW855FwfaXB+GZ9f9qCx/muwpEWyJkSt/w5CcL/OvgWxBzpT2qH4FC1u44O9RWHR1wvHS lMqWetkAAAAAAAA= --------------ms010404080205080205080700--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53304D8D.7080107>