Date: Mon, 20 Jun 2005 16:30:48 -0400 From: Eric Millbrandt <emillbrandt@coldhaus.com> To: John Baldwin <jhb@FreeBSD.org> Cc: freebsd-alpha@FreeBSD.org Subject: Re: Kernel trap on linux compat load Message-ID: <42B72778.5090009@coldhaus.com> In-Reply-To: <200506201433.46335.jhb@FreeBSD.org> References: <42AA1C23.4080901@coldhaus.com> <200506170924.32120.jhb@FreeBSD.org> <42B3471D.7050608@coldhaus.com> <200506201433.46335.jhb@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms090201090606080204050108 Content-Type: text/plain; charset=ISO-8859-6; format=flowed Content-Transfer-Encoding: 7bit John Baldwin wrote: >On Friday 17 June 2005 05:56 pm, Eric Millbrandt wrote: > > >>John Baldwin wrote: >> >> >>>On Thursday 16 June 2005 08:21 pm, Eric Millbrandt wrote: >>> >>> >>>>John Baldwin wrote: >>>> >>>> >>>>>On Friday 10 June 2005 07:02 pm, Eric Millbrandt wrote: >>>>> >>>>> >>>>>>I've noticed that linux compatibility caused my system to crash since I >>>>>>upgraded from RELENG_4_9 (RELENG_4_10, RELENG_4_11, RELENG_4). I >>>>>>upgraded using buildworld, I haven't had a change to test using a clean >>>>>>binary install. Has anyone else seen this behavior? It's easy to >>>>>>reproduce, the system traps right on linux compatibility load. The >>>>>>only similar behavior I've found on google is an unanswered post >>>>>>http://lists.freebsd.org/pipermail/freebsd-alpha/2004-June/001555.html >>>>>> >>>>>>Jun 8 07:18:17 mongoloid /kernel: fatal kernel trap: >>>>>>Jun 8 07:18:17 mongoloid /kernel: >>>>>>Jun 8 07:18:17 mongoloid /kernel: trap entry = 0x4 (unaligned access >>>>>>fault) Jun 8 07:18:17 mongoloid /kernel: a0 = >>>>>>0xfffffe0015285e34 Jun 8 07:18:17 mongoloid /kernel: a1 = >>>>>>0x2d >>>>>>Jun 8 07:18:17 mongoloid /kernel: a2 = 0x1 >>>>>>Jun 8 07:18:17 mongoloid /kernel: pc = 0xfffffc0000386e68 >>>>>>Jun 8 07:18:17 mongoloid /kernel: ra = 0xfffffc0000386dd8 >>>>>>Jun 8 07:18:17 mongoloid /kernel: curproc = 0xfffffe001177e780 >>>>>>Jun 8 07:18:17 mongoloid /kernel: pid = 173, comm = ldconfig >>>>>>Jun 8 07:18:17 mongoloid /kernel: >>>>>>Jun 8 07:18:17 mongoloid /kernel: panic: trap >>>>>> >>>>>>EEM >>>>>> >>>>>> >>>>>Did you rebuild your kernel with buildkernel / installkernel? I believe >>>>>that buildworld on 4.x still builds and installs new modules into >>>>>/modules including the linux and osf1 compat modules. New modules >>>>>aren't guaranteed to work on older kernels, so if your kernel is still >>>>>4.9, that is probably your problem. >>>>> >>>>> >>>>I used buildworld, buildkernel, installkernel, drop to single user, >>>>installworld, and mergemaster. This is with linux_compat loading from >>>>rc.conf. I misspoke earlier loading linux.ko works fine but running a >>>>linux binary, /compat/linux/sbin/ldconfig or sophie, causes the trap. >>>>On a side not osf1_compat works fine. >>>> >>>>Here are my stack traces after running /compat/linux/sbin/ldconfig... >>>>Debugger() at Debugger=0x2c >>>>panic() at panic+0x100 >>>>trap() at trap+0x600 >>>>XentUna() at XentUna+0x2c >>>>--- unaligned access fault (from ipl0) --- >>>>kernel_sysctl at kernel_sysctl+ox1a8 >>>>347() at -0x1fffe4a63d0 >>>>Here ddb traps on itself (memory management fault) >>>> >>>>This one is from running sophie >>>>Debugger() at Debugger=0x2c >>>>panic() at panic+0x100 >>>>trap() at trap+0x600 >>>>XentUna() at XentUna+0x2c >>>>--- unaligned access fault (from ipl0) --- >>>>kernel_sysctl at kernel_sysctl+ox1a8 >>>>linux_newuname() at linux_newuname+0xd0 >>>>syscall() at syscal+0x224 >>>>XentSys at XentSys+0x5c >>>>--- syscall (339, Linux ELF, linux_newuname) --- >>>>--- user mode --- >>>> >>>> >>>Ok, this is helpful. Can you do 'gdb kernel.debug' and then type 'list >>>*kernel_sysctl+0x1a8'? This will give us the source file/line that it >>>faulted at. Thanks! >>> >>> >>Ok here is what I found. >> >>(gdb) list *kernel_sysctl+0x1a8 >>0xfffffc000038e9a8 is in kernel_sysctl >>(/usr/src/sys/kern/kern_sysctl.c:938). >>933 if (req.oldptr && req.oldidx > req.oldlen) >>934 *retval = req.oldlen; >>935 else >>936 *retval = req.oldidx; >>937 } >>938 return (error); >>939 } >>940 >>941 int >>942 kernel_sysctlbyname(struct proc *p, char *name, void *old, >>size_t *oldlenp, >> >> > >Ok, try this patch please: > >Index: compat/linux/linux_misc.c >=================================================================== >RCS file: /usr/cvs/src/sys/compat/linux/linux_misc.c,v >retrieving revision 1.85.2.11 >diff -u -r1.85.2.11 linux_misc.c >--- compat/linux/linux_misc.c 23 Mar 2004 12:16:48 -0000 1.85.2.11 >+++ compat/linux/linux_misc.c 20 Jun 2005 18:33:07 -0000 >@@ -687,7 +687,8 @@ > struct l_new_utsname utsname; > char *osrelease, *osname; > int name[2]; >- int error, plen, olen; >+ int error; >+ size_t plen, olen; > > #ifdef DEBUG > if (ldebug(newuname)) > > > Ok, I rebuilt and installed linux.ko and linux binaries appear to be working! At least sophie, compiled for linux, catches viruses again. Let me know if you need any further testing and where you will apply the patch in cvs. Thanks for the help. --------------ms090201090606080204050108 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIITvzCC BYkwggTyoAMCAQICAQgwDQYJKoZIhvcNAQEEBQAwgbAxCzAJBgNVBAYTAklMMQ8wDQYDVQQI EwZJc3JhZWwxDjAMBgNVBAcTBUVpbGF0MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMRowGAYD VQQLExFDQSBBdXRob3JpdHkgRGVwLjEpMCcGA1UEAxMgRnJlZSBTU0wgQ2VydGlmaWNhdGlv biBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEmFkbWluQHN0YXJ0Y29tLm9yZzAeFw0wNTA0 MDUxNDQ5MzNaFw0xMDA0MDQxNDQ5MzNaMIGuMQswCQYDVQQGEwJJTDEPMA0GA1UECBMGSXNy YWVsMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSMwIQYDVQQLExpTZWN1cmUgQ2VydGlmaWNh dGUgU2lnbmluZzEuMCwGA1UEAxMlU3RhcnRDb20gQ2xhc3MxIFByaW1hcnkgRW1haWwgRnJl ZSBDQTEhMB8GCSqGSIb3DQEJARYSYWRtaW5Ac3RhcnRjb20ub3JnMIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAtTnJ9dYuxbgFN4JqsMrQjrKVuxb62kRaveztSKudMSwd5I48 sSb2vS/u1bZuR0zQAsMLrgpFEzN2as0YhIdeQlO1W5hggX85t5zJgB3ofp/lRshh5wqhQ0EW OnuLA12u/rfRGbUP66VJBu70RuKiWydUdWzWEvn6WlykP318bZs6roE60LQqbOTSOB49G+AG MMYc7CncDlSp0A7hHyH1hVNoiD1yWe3fEfzJKPc9xVmjtzFYET+42TI08uhurOTTWswDoDde PkZvbQAUv/0Z4wgt52bUH+dhARjpKOPGYBNePsgipEGsNthdehymLEzlMd5Uv0jdBVcmCje2 8N1Y4wIDAQABo4ICLTCCAikwDwYDVR0TAQH/BAUwAwEB/zALBgNVHQ8EBAMCAuQwHQYDVR0O BBYEFNyALXXvwKqzpGTAvkbI13FNk3rGMIHdBgNVHSMEgdUwgdKAFByJw5bMvf4y1Q2MgTG2 mJ2NKGSNoYG2pIGzMIGwMQswCQYDVQQGEwJJTDEPMA0GA1UECBMGSXNyYWVsMQ4wDAYDVQQH EwVFaWxhdDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjEaMBgGA1UECxMRQ0EgQXV0aG9yaXR5 IERlcC4xKTAnBgNVBAMTIEZyZWUgU1NMIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSEwHwYJ KoZIhvcNAQkBFhJhZG1pbkBzdGFydGNvbS5vcmeCAQAwHQYDVR0RBBYwFIESYWRtaW5Ac3Rh cnRjb20ub3JnMB0GA1UdEgQWMBSBEmFkbWluQHN0YXJ0Y29tLm9yZzARBglghkgBhvhCAQEE BAMCAAcwUAYJYIZIAYb4QgENBEMWQVN0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1l ZGlhdGUgRnJlZSBTU0wgRW1haWwgQ2VydGlmaWNhdGVzMDIGCWCGSAGG+EIBBAQlFiNodHRw Oi8vY2VydC5zdGFydGNvbS5vcmcvY2EtY3JsLmNybDAzBglghkgBhvhCAQMEJhYkaHR0cDov L2NlcnQuc3RhcnRjb20ub3JnL2NydC1jcmwuY3JsMA0GCSqGSIb3DQEBBAUAA4GBADbqXbMU lLBeaS2whXoQ5OhRDMcoFjNYUc5pvITYQ2WavmEla6789oemn2bSUTBBySvH4+eOMnFzMRtz RtSWFOwQrAQur7B08Ws0D/sgbkorD2n0akPs/Fx0wcA5Qzy/GsqkiMmm3SMJ1pF41hSETUHH dUZOum+UoiBqjZLbv57rMIIHFTCCBf2gAwIBAgICViIwDQYJKoZIhvcNAQEFBQAwga4xCzAJ BgNVBAYTAklMMQ8wDQYDVQQIEwZJc3JhZWwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xIzAh BgNVBAsTGlNlY3VyZSBDZXJ0aWZpY2F0ZSBTaWduaW5nMS4wLAYDVQQDEyVTdGFydENvbSBD bGFzczEgUHJpbWFyeSBFbWFpbCBGcmVlIENBMSEwHwYJKoZIhvcNAQkBFhJhZG1pbkBzdGFy dGNvbS5vcmcwHhcNMDUwNTE1MjI1NjUyWhcNMDYwNTE1MjI1NjUyWjCBjzELMAkGA1UEBhMC VVMxEjAQBgNVBAcTCVdvcmNlc3RlcjEYMBYGA1UEChMPRXJpYyBNaWxsYnJhbmR0MSkwJwYD VQQDEyBTdGFydENvbSBGcmVlIENlcnRpZmljYXRlIE1lbWJlcjEnMCUGCSqGSIb3DQEJARYY ZW1pbGxicmFuZHRAY29sZGhhdXMuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAwa056eA8DrRgjRVO6M4N+lu5tiuf8OsUx09MeYKvuy73VW0/bYFMMWGTa+MwCHxXTo42 Y8hW2oHyDIr7krr5MynRccBzyjsdMhIkllXdrevNmAdMB5GTbsxp7zuZd7I/mvF+fF6XDvfN fs0iqCR6MqEl9aANpOoR/7AFkZ/IODs5CpMOrSr7J0XlAfAg7p7db8PnJMvPC9HtwuUogSmN bMtJ6MM67uV9DJwAqpeJEG+aOIN+/EwPWrOsXgT4Z+I+suXbXI/1emmXaGoxURI0qSIwyJZK O7SP5S4DavX28BYSU5J08G6n+tAu7ajpr3rCnmAMu0Jw4k2WGd4DLT5BcQIDAQABo4IDWDCC A1QwDAYDVR0TAQH/BAIwADALBgNVHQ8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsG AQUFBwMEMB0GA1UdDgQWBBTQvt4hbQFBwEsmbknEYO8seXE8wzCB3QYDVR0jBIHVMIHSgBTc gC1178Cqs6RkwL5GyNdxTZN6xqGBtqSBszCBsDELMAkGA1UEBhMCSUwxDzANBgNVBAgTBklz cmFlbDEOMAwGA1UEBxMFRWlsYXQxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xGjAYBgNVBAsT EUNBIEF1dGhvcml0eSBEZXAuMSkwJwYDVQQDEyBGcmVlIFNTTCBDZXJ0aWZpY2F0aW9uIEF1 dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSYWRtaW5Ac3RhcnRjb20ub3JnggEIMIHcBgNVHSAE gdQwgdEwgc4GAQAwgcgwLwYIKwYBBQUHAgEWI2h0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9w b2xpY3kucGRmMDUGCCsGAQUFBwIBFilodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvaW50ZXJt ZWRpYXRlLnBkZjBeBggrBgEFBQcCAjBSMBQaDVN0YXJ0Q29tIEx0ZC4wAwIBARo6TGltaXRl ZCBMaWFiaWxpdHksIHNlZSBodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvcG9saWN5LnBkZjA1 BgNVHR8ELjAsMCqgKKAmhiRodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvY3J0LWNybC5jcmww EQYJYIZIAYb4QgEBBAQDAgWgMC4GCWCGSAGG+EIBDQQhFh9TdGFydENvbSBGcmVlIEVtYWls IENlcnRpZmljYXRlMDIGCWCGSAGG+EIBBAQlFiNodHRwOi8vY2VydC5zdGFydGNvbS5vcmcv Y2EtY3JsLmNybDAzBglghkgBhvhCAQMEJhYkaHR0cDovL2NlcnQuc3RhcnRjb20ub3JnL2Ny dC1jcmwuY3JsMDIGCWCGSAGG+EIBCAQlFiNodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvcG9s aWN5LnBkZjAjBgNVHRIEHDAahhhodHRwOi8vY2VydC5zdGFydGNvbS5vcmcwDQYJKoZIhvcN AQEFBQADggEBAF7PtshubKCW8Q/eRQCOayKDL/yTJyg1WY/Ab+/cf0dKPjpK3nfprb3ZIIAC cz00Ve+NtyxXi/4bSccD68a2pZ6Z6iIViw9yLxGy2jyTDh41vXjPsnZpMbNUI2CjqRYM6jJh I5tl+XL+u+pNb3Me/eXBBkNkbRgGv2jljz7MTQy47LxmUhYHpLZQtNks9tm4MeDvRsII1A3l jMKA4y8SF4xEIaoMgXuX7Ad/n/C9xzMX/gZmdj0ZScc2yMjBZ9PSrVMOFEViH0rwg93YhUtP uDFZ4rqCmeRUC8khvslG1xJaknOnQIVqEitGKNYk5hnklFPjPUsC8rysxMy9vible+swggcV MIIF/aADAgECAgJWIjANBgkqhkiG9w0BAQUFADCBrjELMAkGA1UEBhMCSUwxDzANBgNVBAgT BklzcmFlbDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjEjMCEGA1UECxMaU2VjdXJlIENlcnRp ZmljYXRlIFNpZ25pbmcxLjAsBgNVBAMTJVN0YXJ0Q29tIENsYXNzMSBQcmltYXJ5IEVtYWls IEZyZWUgQ0ExITAfBgkqhkiG9w0BCQEWEmFkbWluQHN0YXJ0Y29tLm9yZzAeFw0wNTA1MTUy MjU2NTJaFw0wNjA1MTUyMjU2NTJaMIGPMQswCQYDVQQGEwJVUzESMBAGA1UEBxMJV29yY2Vz dGVyMRgwFgYDVQQKEw9FcmljIE1pbGxicmFuZHQxKTAnBgNVBAMTIFN0YXJ0Q29tIEZyZWUg Q2VydGlmaWNhdGUgTWVtYmVyMScwJQYJKoZIhvcNAQkBFhhlbWlsbGJyYW5kdEBjb2xkaGF1 cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBrTnp4DwOtGCNFU7ozg36 W7m2K5/w6xTHT0x5gq+7LvdVbT9tgUwxYZNr4zAIfFdOjjZjyFbagfIMivuSuvkzKdFxwHPK Ox0yEiSWVd2t682YB0wHkZNuzGnvO5l3sj+a8X58XpcO981+zSKoJHoyoSX1oA2k6hH/sAWR n8g4OzkKkw6tKvsnReUB8CDunt1vw+cky88L0e3C5SiBKY1sy0nowzru5X0MnACql4kQb5o4 g378TA9as6xeBPhn4j6y5dtcj/V6aZdoajFREjSpIjDIlko7tI/lLgNq9fbwFhJTknTwbqf6 0C7tqOmvesKeYAy7QnDiTZYZ3gMtPkFxAgMBAAGjggNYMIIDVDAMBgNVHRMBAf8EAjAAMAsG A1UdDwQEAwIE8DAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwHQYDVR0OBBYEFNC+ 3iFtAUHASyZuScRg7yx5cTzDMIHdBgNVHSMEgdUwgdKAFNyALXXvwKqzpGTAvkbI13FNk3rG oYG2pIGzMIGwMQswCQYDVQQGEwJJTDEPMA0GA1UECBMGSXNyYWVsMQ4wDAYDVQQHEwVFaWxh dDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjEaMBgGA1UECxMRQ0EgQXV0aG9yaXR5IERlcC4x KTAnBgNVBAMTIEZyZWUgU1NMIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSEwHwYJKoZIhvcN AQkBFhJhZG1pbkBzdGFydGNvbS5vcmeCAQgwgdwGA1UdIASB1DCB0TCBzgYBADCByDAvBggr BgEFBQcCARYjaHR0cDovL2NlcnQuc3RhcnRjb20ub3JnL3BvbGljeS5wZGYwNQYIKwYBBQUH AgEWKWh0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9pbnRlcm1lZGlhdGUucGRmMF4GCCsGAQUF BwICMFIwFBoNU3RhcnRDb20gTHRkLjADAgEBGjpMaW1pdGVkIExpYWJpbGl0eSwgc2VlIGh0 dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9wb2xpY3kucGRmMDUGA1UdHwQuMCwwKqAooCaGJGh0 dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9jcnQtY3JsLmNybDARBglghkgBhvhCAQEEBAMCBaAw LgYJYIZIAYb4QgENBCEWH1N0YXJ0Q29tIEZyZWUgRW1haWwgQ2VydGlmaWNhdGUwMgYJYIZI AYb4QgEEBCUWI2h0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9jYS1jcmwuY3JsMDMGCWCGSAGG +EIBAwQmFiRodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvY3J0LWNybC5jcmwwMgYJYIZIAYb4 QgEIBCUWI2h0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9wb2xpY3kucGRmMCMGA1UdEgQcMBqG GGh0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZzANBgkqhkiG9w0BAQUFAAOCAQEAXs+2yG5soJbx D95FAI5rIoMv/JMnKDVZj8Bv79x/R0o+Okred+mtvdkggAJzPTRV7423LFeL/htJxwPrxral npnqIhWLD3IvEbLaPJMOHjW9eM+ydmkxs1QjYKOpFgzqMmEjm2X5cv676k1vcx795cEGQ2Rt GAa/aOWPPsxNDLjsvGZSFgektlC02Sz22bgx4O9GwgjUDeWMwoDjLxIXjEQhqgyBe5fsB3+f 8L3HMxf+BmZ2PRlJxzbIyMFn09KtUw4URWIfSvCD3diFS0+4MVniuoKZ5FQLySG+yUbXElqS c6dAhWoSK0Yo1iTmGeSUU+M9SwLyvKzEzL2+JuV76zGCBCYwggQiAgEBMIG1MIGuMQswCQYD VQQGEwJJTDEPMA0GA1UECBMGSXNyYWVsMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSMwIQYD VQQLExpTZWN1cmUgQ2VydGlmaWNhdGUgU2lnbmluZzEuMCwGA1UEAxMlU3RhcnRDb20gQ2xh c3MxIFByaW1hcnkgRW1haWwgRnJlZSBDQTEhMB8GCSqGSIb3DQEJARYSYWRtaW5Ac3RhcnRj b20ub3JnAgJWIjAJBgUrDgMCGgUAoIICRTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwG CSqGSIb3DQEJBTEPFw0wNTA2MjAyMDMwNDhaMCMGCSqGSIb3DQEJBDEWBBQKTWYrjdQfo+dq fULQGz7m956PUzBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIA gDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBxgYJKwYBBAGCNxAE MYG4MIG1MIGuMQswCQYDVQQGEwJJTDEPMA0GA1UECBMGSXNyYWVsMRYwFAYDVQQKEw1TdGFy dENvbSBMdGQuMSMwIQYDVQQLExpTZWN1cmUgQ2VydGlmaWNhdGUgU2lnbmluZzEuMCwGA1UE AxMlU3RhcnRDb20gQ2xhc3MxIFByaW1hcnkgRW1haWwgRnJlZSBDQTEhMB8GCSqGSIb3DQEJ ARYSYWRtaW5Ac3RhcnRjb20ub3JnAgJWIjCByAYLKoZIhvcNAQkQAgsxgbiggbUwga4xCzAJ BgNVBAYTAklMMQ8wDQYDVQQIEwZJc3JhZWwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xIzAh BgNVBAsTGlNlY3VyZSBDZXJ0aWZpY2F0ZSBTaWduaW5nMS4wLAYDVQQDEyVTdGFydENvbSBD bGFzczEgUHJpbWFyeSBFbWFpbCBGcmVlIENBMSEwHwYJKoZIhvcNAQkBFhJhZG1pbkBzdGFy dGNvbS5vcmcCAlYiMA0GCSqGSIb3DQEBAQUABIIBALdVKJ7LzdCkZhx180KR6SlAjlB1YCKj RwyskzNW8ChxY54PjGup6NtaIGu0K/ovQltme3CnIHctUvGq5gPQ2UxizJ9aSPMhS9VAzU3m kCrXD7P4i5TxGG6kO7c2YYHb9WkgFatpf4oMNhfK4uO4H5+PbiPsoI5f0fXaYpLfybj4HQXr XVSoWSddqEVBIRx6d4WzTW6ial8vcilXJa3zWdXjqnKfYgujBSJFPYt5MR0co3b3I9IoJWOG FuQ5J/18B7XJATB8hg4YOtBgsiJTMF/wGMbow2A6XPzezbzlsD8W5/MWAIxNj4JFc9/SGYMj a02+YhndLap2m7h/j3vQVc0AAAAAAAA= --------------ms090201090606080204050108--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42B72778.5090009>