Date: Wed, 20 Nov 2024 08:29:42 -0800 From: Mark Millard <marklmi@yahoo.com> To: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, FreeBSD ARM List <freebsd-arm@freebsd.org> Cc: Jessica Clarke <jrtc27@freebsd.org> Subject: Re: Should stable/1[34] and supported 1[34].*-RELEASE have -ftls-model=initial-exec fixes MFC'd and/or EC'd for armv7? Message-ID: <7E60FC8D-1770-444F-A4FA-953A2A2FEDB8@yahoo.com> In-Reply-To: <CFD64ABC-321C-4B6F-BBA3-1823480192A9@yahoo.com> References: <CFD64ABC-321C-4B6F-BBA3-1823480192A9@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_294CD89F-DEEB-4FB5-89D5-BE573B9F8557 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii [Just CC'ing Jessica C. instead of kib. This is based on = https://reviews.freebsd.org/D42415 having Jessica's note: If IE is fixed then lib/libc/Makefile probably should enable it on arm = as a follow-up, which I *think* is the only architecture not covered by = that if statement, unless I'm missing something ] On Nov 17, 2024, at 10:50, Mark Millard <marklmi@yahoo.com> wrote: [Not that they could be timed for 14.2-RELEASE at this point.] Given an update to the bootstrap lang/rust compiler that has already been fixed, the below fixes why lang/rust has not built on the official package build server s for 1[34].* since at least lang/rust 1.77.0 . (The traceable records do not go back beyond that. But it may be that rust became dependent at 1.77.0 for all I know.) It also blocks private armv7 lang/rust builds for stable/1[34] . Other packages may have build failures that are related but I'll use the lang/rust activity here: used as the test case. I have locally tested building lang/rust 1.82.0_1 on stable/14 both with and without the changes in question. With the 2 commits it built just fine, unlike without. (I did not try just 1 of the 2, just the pair as I unit.) The primary, recent bugzilla activity is at: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282663 But the below avoids going through the discovery process and history that is recorded there. (There are also other related bugzilla's around but this one is where the potential MFC's were identified.) See also the fallout records for lang/rust for armv7: = https://portsfallout.com/fallout?port=3Dlang%2Frust&maintainer=3D&env=3Dar= mv7&category=3D&flavor=3D (but ignore the lang/rustpython lines that also match the pattern used). Only main got the fixes back in 2023-Nov and only main has worked for lang/rust for the official armv7 package build servers since then for any records that I've found -- lang/rust 1.77.0 and later. The 2 commits are: = https://cgit.freebsd.org/src/commit/?id=3D98fd69f0090da73d9d0451bd769d7752= 468284c6 = https://cgit.freebsd.org/src/commit/?id=3D6e5b1ff71e01bd48172483cb6df921f8= 4300ea3a I show the details below. Whitespace might not be accurately preserved = below. https://cgit.freebsd.org/src/commit/?id=3D98fd69f0090d is: author R. Christian McDonald <rcm@rcm.sh> 2023-11-03 12:56:58 +0000 committer Kristof Provost <kp@FreeBSD.org> 2023-11-03 21:43:40 +0000 . . . rtld/arm: fix initial-exec (IE) thread-local storage relocation net/frr[89] revealed an interesting edge-case on arm when dynamically linking a shared library that declares more than one static TLS variable with at least one using the "initial-exec" TLS model. In the case of frr[89], this library was libfrr.so which essentially does the following: #include <stdio.h> #include "lib.h" static __thread int *a __attribute__((tls_model("initial-exec"))); void lib_test() { static __thread int b =3D -1; printf("&a =3D %p\n", &a); printf(" a =3D %p\n", a); printf("\n"); printf("&b =3D %p\n", &b); printf(" b =3D %d\n", b); } Allocates a file scoped `static __thread` pointer with tls_model("initial-exec") and later a block scoped TLS int. Notice in the above minimal reproducer, `b =3D=3D -1`. The relocation process does the wrong thing and ends up pointing both `a` and `b` at the same place in memory. The output of the above in the broken state is: &a =3D 0x4009c018 a =3D 0xffffffff &b =3D 0x4009c018 b =3D -1 With the patch applied, the output becomes: &a =3D 0x4009c01c a =3D 0x0 &b =3D 0x4009c018 b =3D -1 Reviewed by: kib Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D42415/ Diffstat -rw-r--r-- libexec/rtld-elf/arm/reloc.c 7=09 1 files changed, 5 insertions, 2 deletions diff --git a/libexec/rtld-elf/arm/reloc.c b/libexec/rtld-elf/arm/reloc.c index c3e95940be74..6efc9f499761 100644 --- a/libexec/rtld-elf/arm/reloc.c +++ b/libexec/rtld-elf/arm/reloc.c @@ -280,10 +280,13 @@ reloc_nonplt_object(Obj_Entry *obj, const Elf_Rel = *rel, SymCache *cache, return -1; tmp =3D (Elf_Addr)def->st_value + = defobj->tlsoffset; - if (__predict_true(RELOC_ALIGNED_P(where))) + if (__predict_true(RELOC_ALIGNED_P(where))) { + tmp +=3D *where; *where =3D tmp; - else + } else { + tmp +=3D load_ptr(where); store_ptr(where, tmp); + } dbg("TLS_TPOFF32 %s in %s --> %p", obj->strtab + obj->symtab[symnum].st_name, obj->path, (void *)tmp); https://cgit.freebsd.org/src/commit/?id=3D6e5b1ff71e01 is: author R. Christian McDonald <rcm@rcm.sh> 2023-11-09 20:22:21 = +0000 committer Kristof Provost <kp@FreeBSD.org> 2023-11-09 = 20:24:23 +0000 . . . Reviewed by: kib Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D42415/ libc: enable initial-exec (IE) as default thread-local storage model on = arm As suggested by jrtc27@ in https://reviews.freebsd.org/D42415, this patch enables IE as default thread-local storage model in libc on arm. Reviewed by: kib Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D42445 Diffstat -rw-r--r-- lib/libc/Makefile 4=09 1 files changed, 0 insertions, 4 deletions diff --git a/lib/libc/Makefile b/lib/libc/Makefile index 7540eb8c21ad..e817104642b8 100644 --- a/lib/libc/Makefile +++ b/lib/libc/Makefile @@ -54,11 +54,7 @@ CFLAGS+=3D${CANCELPOINTS_CFLAGS} # Use a more efficient TLS model for libc since we can reasonably assume = that # it will be loaded during program startup. -.if ${LIBC_ARCH} =3D=3D "aarch64" || ${LIBC_ARCH} =3D=3D "amd64" || \ - ${LIBC_ARCH} =3D=3D "i386" || ${LIBC_ARCH} =3D=3D "riscv" || \ - ${LIBC_ARCH:Mpowerpc*} !=3D "" CFLAGS+=3D -ftls-model=3Dinitial-exec -.endif # # Link with static libcompiler_rt.a. =3D=3D=3D Mark Millard marklmi at yahoo.com =3D=3D=3D Mark Millard marklmi at yahoo.com --Apple-Mail=_294CD89F-DEEB-4FB5-89D5-BE573B9F8557 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;">[Just CC'ing = Jessica C. instead of kib. This is based on <a = href=3D"https://reviews.freebsd.org/D42415">https://reviews.freebsd.org/D4= 2415</a><div>having Jessica's note:</div><div><br></div><div><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = "Segoe UI", "Segoe UI Emoji", "Segoe UI = Symbol", Lato, "Helvetica Neue", Helvetica, Arial, = sans-serif; background-color: rgb(255, 255, 255);">If IE is fixed then = lib/libc/Makefile probably should enable it on arm as a follow-up, which = I *think* is the only architecture not covered by that if statement, = unless I'm missing something</span></div><div>]<br = id=3D"lineBreakAtBeginningOfMessage"><div><br><div>On Nov 17, 2024, at = 10:50, Mark Millard <marklmi@yahoo.com> wrote:</div><br = class=3D"Apple-interchange-newline"><div><div>[Not that they could be = timed for 14.2-RELEASE at this point.]<br><br>Given an update to the = bootstrap lang/rust compiler that has<br>already been fixed, the below = fixes why lang/rust has not<br>built on the official package build = server s for 1[34].* since<br>at least lang/rust 1.77.0 . (The = traceable<br>records do not go back beyond that. But it may be = that<br>rust became dependent at 1.77.0 for all I know.) It = also<br>blocks private armv7 lang/rust builds for stable/1[34] = .<br><br>Other packages may have build failures that are related<br>but = I'll use the lang/rust activity here: used as the<br>test case.<br><br>I = have locally tested building lang/rust 1.82.0_1 on<br>stable/14 both = with and without the changes in question.<br>With the 2 commits it built = just fine, unlike without.<br>(I did not try just 1 of the 2, just the = pair as I unit.)<br><br>The primary, recent bugzilla activity is = at:<br><br>https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D282663<br><= br>But the below avoids going through the discovery process = and<br>history that is recorded there. (There are also other = related<br>bugzilla's around but this one is where the potential = MFC's<br>were identified.)<br><br>See also the fallout records for = lang/rust for = armv7:<br><br>https://portsfallout.com/fallout?port=3Dlang%2Frust&main= tainer=3D&env=3Darmv7&category=3D&flavor=3D<br><br>(but = ignore the lang/rustpython lines that also match the pattern = used).<br><br><br>Only main got the fixes back in 2023-Nov and only main = has worked for<br>lang/rust for the official armv7 package build servers = since then<br>for any records that I've found -- lang/rust 1.77.0 and = later.<br><br><br>The 2 commits = are:<br><br>https://cgit.freebsd.org/src/commit/?id=3D98fd69f0090da73d9d04= 51bd769d7752468284c6<br><br>https://cgit.freebsd.org/src/commit/?id=3D6e5b= 1ff71e01bd48172483cb6df921f84300ea3a<br><br>I show the details below. = Whitespace might not be accurately preserved = below.<br><br><br>https://cgit.freebsd.org/src/commit/?id=3D98fd69f0090d = is:<br><br>author R. Christian McDonald <rcm@rcm.sh> 2023-11-03 = 12:56:58 +0000<br>committer Kristof Provost <kp@FreeBSD.org> = 2023-11-03 21:43:40 +0000<br>. . .<br><br>rtld/arm: fix initial-exec = (IE) thread-local storage relocation<br><br>net/frr[89] revealed an = interesting edge-case on arm when dynamically<br>linking a shared = library that declares more than one static TLS variable<br>with at least = one using the "initial-exec" TLS model. In the case<br>of frr[89], = this library was libfrr.so which essentially does = the<br>following:<br><br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>#include = <stdio.h><br><br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>#include "lib.h"<br><br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>static = __thread int *a<br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> = </span>__attribute__((tls_model("initial-exec")));<br><br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>void = lib_test()<br><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>{<br><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>static __thread int b =3D -1;<br><br><span class=3D"Apple-tab-span"= style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>printf("&a =3D %p\n", = &a);<br><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>printf(" a =3D %p\n", a);<br><br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>printf("\n");<br><br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>printf("&b =3D %p\n", &b);<br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>printf(" = b =3D %d\n", b);<br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>}<br><br>Allocates a file scoped = `static __thread` pointer with<br>tls_model("initial-exec") and later a = block scoped TLS int. Notice in<br>the above minimal reproducer, `b =3D=3D= -1`. The relocation process does<br>the wrong thing and ends up = pointing both `a` and `b` at the same place<br>in memory.<br><br>The = output of the above in the broken state is:<br><br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>&a =3D = 0x4009c018<br><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span> a =3D 0xffffffff<br><br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>&b =3D 0x4009c018<br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span> b =3D = -1<br><br>With the patch applied, the output becomes:<br><br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>&a =3D = 0x4009c01c<br><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span> a =3D 0x0<br><br><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>&b =3D 0x4009c018<br><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span> b =3D = -1<br><br>Reviewed by: kib<br>Sponsored by: Rubicon Communications, LLC = ("Netgate")<br>Differential Revision: = https://reviews.freebsd.org/D42415/<br><br>Diffstat<br>-rw-r--r--<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>libexec/rtld-elf/arm/reloc.c<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>7<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><br>1 files changed, 5 = insertions, 2 deletions<br>diff --git a/libexec/rtld-elf/arm/reloc.c = b/libexec/rtld-elf/arm/reloc.c<br>index c3e95940be74..6efc9f499761 = 100644<br>--- a/libexec/rtld-elf/arm/reloc.c<br>+++ = b/libexec/rtld-elf/arm/reloc.c<br>@@ -280,10 +280,13 @@ = reloc_nonplt_object(Obj_Entry *obj, const Elf_Rel *rel, SymCache = *cache,<br> <span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>return -1;<br><br> <span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>tmp =3D = (Elf_Addr)def->st_value + defobj->tlsoffset;<br>-<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>if = (__predict_true(RELOC_ALIGNED_P(where)))<br>+<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>if = (__predict_true(RELOC_ALIGNED_P(where))) {<br>+<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>tmp +=3D = *where;<br> <span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>*where =3D tmp;<br>-<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>else<br>+<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>} else = {<br>+<span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>tmp +=3D load_ptr(where);<br> <span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>store_ptr(where, tmp);<br>+<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>}<br> = <span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>dbg("TLS_TPOFF32 %s in %s --> %p",<br> <span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span> = obj->strtab + obj->symtab[symnum].st_name,<br> = <span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span> obj->path, (void = *)tmp);<br><br><br>https://cgit.freebsd.org/src/commit/?id=3D6e5b1ff71e01 = is:<br><br>author<span class=3D"Apple-tab-span" style=3D"white-space:pre">= </span>R. Christian McDonald <rcm@rcm.sh><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>2023-11-09 20:22:21 +0000<br>committer<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>Kristof = Provost <kp@FreeBSD.org><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>2023-11-09 20:24:23 +0000<br>. . = .<br><br>Reviewed by:<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>kib<br>Sponsored by:<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>Rubicon = Communications, LLC ("Netgate")<br>Differential Revision:<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>https://reviews.freebsd.org/D42415/<br><br>libc: enable = initial-exec (IE) as default thread-local storage model on arm<br><br>As = suggested by jrtc27@ in https://reviews.freebsd.org/D42415, = this<br>patch enables IE as default thread-local storage model in libc = on arm.<br><br>Reviewed by:<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>kib<br>Sponsored by:<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>Rubicon = Communications, LLC ("Netgate")<br>Differential Revision:<span = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>https://reviews.freebsd.org/D42445<br><br>Diffstat<br>-rw-r--r--<sp= an class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span>lib/libc/Makefile<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>4<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><br>1 files changed, 0 = insertions, 4 deletions<br>diff --git a/lib/libc/Makefile = b/lib/libc/Makefile<br>index 7540eb8c21ad..e817104642b8 100644<br>--- = a/lib/libc/Makefile<br>+++ b/lib/libc/Makefile<br>@@ -54,11 +54,7 @@ = CFLAGS+=3D${CANCELPOINTS_CFLAGS}<br><br> # Use a more efficient TLS = model for libc since we can reasonably assume that<br> # it will be = loaded during program startup.<br>-.if ${LIBC_ARCH} =3D=3D "aarch64" || = ${LIBC_ARCH} =3D=3D "amd64" || \<br>- ${LIBC_ARCH} =3D=3D= "i386" || ${LIBC_ARCH} =3D=3D "riscv" || \<br>- = ${LIBC_ARCH:Mpowerpc*} !=3D ""<br> CFLAGS+=3D = -ftls-model=3Dinitial-exec<br>-.endif<br><br> #<br> # Link with static = libcompiler_rt.a.<br><br>=3D=3D=3D<br>Mark Millard<br>marklmi at = yahoo.com<br><br></div></div></div><br><div> <div dir=3D"auto" style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0); letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: = break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;"><div dir=3D"auto" style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;"><div dir=3D"auto" style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;"><div = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;"><div style=3D"color: rgb(0, 0, 0); = font-variant-caps: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style=3D"orphans:= 2; widows: 2; margin: 0px; line-height: normal;"><font face=3D"Courier = New">=3D=3D=3D</font></div><div style=3D"orphans: 2; widows: 2; margin: = 0px; line-height: normal;"><font face=3D"Courier New">Mark = Millard</font></div><div style=3D"orphans: 2; widows: 2; margin: 0px; = line-height: normal;"><font face=3D"Courier New"><span style=3D"font-size:= 12px;">marklmi at = yahoo.com<br></span></font></div></div></div></div></div></div> </div> <br></div></body></html>= --Apple-Mail=_294CD89F-DEEB-4FB5-89D5-BE573B9F8557--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7E60FC8D-1770-444F-A4FA-953A2A2FEDB8>