Date: Mon, 20 Feb 2023 12:32:42 +0000 From: Andrew Turner <andrew@fubar.geek.nz> To: Mark Millard <marklmi@yahoo.com> Cc: bob prohaska <fbsd@www.zefox.net>, "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org> Subject: Re: Armv7 panic on -current, rpi2 buildworld Message-ID: <CCE79CB7-BA79-4682-AC7C-4D5E8EC0A21A@fubar.geek.nz> In-Reply-To: <A137339C-683A-42B6-95CA-0EE5B4156562@yahoo.com> References: <20230215025741.GA32086@www.zefox.net> <CANCZdfou0s5Xz4_0pPdNSQnFH9qk9NAY=GyB7pBwnVNPvGS4Qw@mail.gmail.com> <A81B272C-8AE0-4DB8-A399-6862A13E4394@yahoo.com> <A137339C-683A-42B6-95CA-0EE5B4156562@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_8B4575DD-EC51-4B1A-8535-DCF534833113 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Can you try with 24abb6b82102eec577eff9bd8dd7726e8cab89f4? There were = conditional branch instructions that may mean the function to save the = VFP state was not being run. Andrew > On 16 Feb 2023, at 19:35, Mark Millard <marklmi@yahoo.com> wrote: >=20 > On Feb 14, 2023, at 23:16, Mark Millard <marklmi@yahoo.com = <mailto:marklmi@yahoo.com>> wrote: >=20 >> On Feb 14, 2023, at 20:16, Warner Losh <imp@bsdimp.com> wrote: >>=20 >>> Sorry to top post... what program was dumping core? Looks like a too = strict assert >>=20 >> Just a possible point, given recent kernel floating >> point work: >>=20 >> Because of Bob's note, I tried to do a typical build >> and test of some benchmark programs that I sometimes >> use that involve floating point in some of the >> programs, some use with multithreading involved. (As >> FreeBSD and g++ progress I tend to do this once and >> a while, not as often on armv7 as on aarch64.) >>=20 >> On armv7, I now get a message about a failure of an >> internal cross-check, which also leads to the program >> being stopped early. The messaging from run to run >> varies what the failure is, but the runs should not >> vary and should not fail the cross-checks --and >> previously did not, including when I last tried armv7. >> (Not recently.) >>=20 >> For the specific example failure, the initial serial >> (single thread) test with float involved works but the >> following multi-thread test in the same program fails >> and causes the program to stop when it notices there >> is a problem. >>=20 >> The programs that do not test floating point do not >> fail. These can involve floating point outside the >> algorithm benchmarked, but with no multi-threading >> involved for such and no floating point based cross- >> checks involved. >>=20 >> At this point it is far from obvious to me how I >> would trackdown the specifics of what leads to the >> failed cross-checks. But the above is suggestive of >> there being problems for armv7 handling of saving >> and restoring floating point context for >> multi-threading. I've no clue if such are limited >> to the floating point values or not. >>=20 >>> Warner >>>=20 >>> On Tue, Feb 14, 2023, 7:57 PM bob prohaska <fbsd@www.zefox.net> = wrote: >>> Building world on an RPi2 armv7, buildworld stopped with >>> bob@www:/usr/src % panic: Called fill_fpregs while the kernel is = using the VFP >>> cpuid =3D 0 >>> time =3D 1676427410 >>> KDB: stack backtrace: >>> db_trace_self() at db_trace_self >>> pc =3D 0xc05e8160 lr =3D 0xc007aa04 = (db_trace_self_wrapper+0x30) >>> sp =3D 0xde2c5790 fp =3D 0xde2c58a8 >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >>> pc =3D 0xc007aa04 lr =3D 0xc02e9c54 (vpanic+0x140) >>> sp =3D 0xde2c58b0 fp =3D 0xde2c58d0 >>> r4 =3D 0x00000100 r5 =3D 0x00000000 >>> r6 =3D 0xc07372ef r7 =3D 0xc0b13968 >>> vpanic() at vpanic+0x140 >>> pc =3D 0xc02e9c54 lr =3D 0xc02e9a34 (dump_savectx) >>> sp =3D 0xde2c58d8 fp =3D 0xde2c58dc >>> r4 =3D 0xd70c8600 r5 =3D 0xde2c5e90 >>> r6 =3D 0xc3398090 r7 =3D 0xe0cfc440 >>> r8 =3D 0xc3398080 r9 =3D 0xd70c8600 >>> r10 =3D 0xde2c5960 >>> dump_savectx() at dump_savectx >>> pc =3D 0xc02e9a34 lr =3D 0xc05f51dc (set_regs) >>> sp =3D 0xde2c58e4 fp =3D 0xde2c58f8 >>> set_regs() at set_regs >>> pc =3D 0xc05f51dc lr =3D 0xc026f8f0 = (elf32_get_fpregset+0x2c) >>> sp =3D 0xde2c5900 fp =3D 0xde2c5908 >>> r4 =3D 0xc3398090 r5 =3D 0xc026f8c4 >>> elf32_get_fpregset() at elf32_get_fpregset+0x2c >>> pc =3D 0xc026f8f0 lr =3D 0xc026d848 (elf32_coredump+0x308) >>> sp =3D 0xde2c5910 fp =3D 0xde2c5988 >>> r4 =3D 0xc0902a7c r10 =3D 0xde2c5960 >>> elf32_coredump() at elf32_coredump+0x308 >>> pc =3D 0xc026d848 lr =3D 0xc02eea74 (sigexit+0xce0) >>> sp =3D 0xde2c5990 fp =3D 0xde2c5cf8 >>> r4 =3D 0x0000004e r5 =3D 0xdf580b60 >>> r6 =3D 0xdf580a78 r7 =3D 0xc026d540 >>> r8 =3D 0xdddcb2bc r9 =3D 0xdf580ad4 >>> r10 =3D 0x00000000 >>> sigexit() at sigexit+0xce0 >>> pc =3D 0xc02eea74 lr =3D 0xc02ef36c (postsig+0x128) >>> sp =3D 0xde2c5d00 fp =3D 0xde2c5d88 >>> r4 =3D 0x00000006 r5 =3D 0xdd43fba0 >>> r6 =3D 0xde2c5d20 r7 =3D 0xde2c5d18 >>> r8 =3D 0xdddcb1f8 r9 =3D 0xdf3d9ab8 >>> r10 =3D 0x00000005 >>> postsig() at postsig+0x128 >>> pc =3D 0xc02ef36c lr =3D 0xc02f316c (ast_sig+0x11c) >>> sp =3D 0xde2c5d90 fp =3D 0xde2c5e08 >>> r4 =3D 0xdd43fba0 r5 =3D 0xdddcb2bc >>> r6 =3D 0xc0734d22 r7 =3D 0x00000000 >>> r8 =3D 0xdddcb1f8 r9 =3D 0x00000ab8 >>> r10 =3D 0x22530384 >>> ast_sig() at ast_sig+0x11c >>> pc =3D 0xc02f316c lr =3D 0xc035444c (ast_handler+0xe0) >>> sp =3D 0xde2c5e10 fp =3D 0xde2c5e28 >>> r4 =3D 0xde2c5e40 r5 =3D 0x0000000e >>> r6 =3D 0x00004000 r7 =3D 0xc096b59c >>> r8 =3D 0xdd43fba0 r9 =3D 0x00000001 >>> ast_handler() at ast_handler+0xe0 >>> pc =3D 0xc035444c lr =3D 0xc035435c (ast+0x20) >>> sp =3D 0xde2c5e30 fp =3D 0xde2c5e38 >>> r4 =3D 0xde2c5e40 r5 =3D 0xdd43fba0 >>> r6 =3D 0x00000000 r7 =3D 0x000001b1 >>> r8 =3D 0x22c4b500 r9 =3D 0x00000000 >>> ast() at ast+0x20 >>> pc =3D 0xc035435c lr =3D 0xc05eaa88 (swi_exit+0x3c) >>> sp =3D 0xde2c5e40 fp =3D 0xbb9fbe38 >>> r4 =3D 0x60000013 r5 =3D 0xdd43fba0 >>> swi_exit() at swi_exit+0x3c >>> pc =3D 0xc05eaa88 lr =3D 0xc05eaa88 (swi_exit+0x3c) >>> sp =3D 0xde2c5e40 fp =3D 0xbb9fbe38 >>> KDB: enter: panic >>> [ thread pid 81621 tid 101111 ] >>> Stopped at kdb_enter+0x54: ldrb r15, [r15, r15, ror r15]! >>> db> bt >>> Tracing pid 81621 tid 101111 td 0xdd43fba0 >>> db_trace_self() at db_trace_self >>> pc =3D 0xc05e8160 lr =3D 0xc00774a0 (db_stack_trace+0x140) >>> sp =3D 0xde2c55d8 fp =3D 0xde2c55f0 >>> db_stack_trace() at db_stack_trace+0x140 >>> pc =3D 0xc00774a0 lr =3D 0xc00770f0 (db_command+0x310) >>> sp =3D 0xde2c55f8 fp =3D 0xde2c56a0 >>> r4 =3D 0xc0745722 r5 =3D 0x00000062 >>> r6 =3D 0x00000000 r10 =3D 0x00000000 >>> db_command() at db_command+0x310 >>> pc =3D 0xc00770f0 lr =3D 0xc0076db8 (db_command_loop+0x64) >>> sp =3D 0xde2c56a8 fp =3D 0xde2c56b8 >>> r4 =3D 0xc07ac186 r5 =3D 0xc07ab7fe >>> r6 =3D 0xc0986f5c r7 =3D 0xc0b13968 >>> r8 =3D 0xc0b23738 r9 =3D 0x00000000 >>> r10 =3D 0x00000001 >>> db_command_loop() at db_command_loop+0x64 >>> pc =3D 0xc0076db8 lr =3D 0xc007ab88 (db_trap+0x128) >>> sp =3D 0xde2c56c0 fp =3D 0xde2c57d8 >>> r4 =3D 0x00000000 r5 =3D 0xc0986f50 >>> r6 =3D 0xc0b23758 r10 =3D 0x00000001 >>> db_trap() at db_trap+0x128 >>> pc =3D 0xc007ab88 lr =3D 0xc033bb84 (kdb_trap+0x258) >>> sp =3D 0xde2c57e0 fp =3D 0xde2c5808 >>> r4 =3D 0xc078390c r5 =3D 0xc08d5270 >>> r6 =3D 0xc0b23758 r7 =3D 0xc0b13968 >>> kdb_trap() at kdb_trap+0x258 >>> pc =3D 0xc033bb84 lr =3D 0xc05eaab8 (exception_exit) >>> sp =3D 0xde2c5810 fp =3D 0xde2c58a8 >>> r4 =3D 0x200000d3 r5 =3D 0x00000000 >>> r6 =3D 0xc07372ef r7 =3D 0xc0b13968 >>> r8 =3D 0xc093fa0c r9 =3D 0xde2c58e4 >>> r10 =3D 0xc0b13a68 >>> exception_exit() at exception_exit >>> pc =3D 0xc05eaab8 lr =3D 0xc033b044 (kdb_enter+0x50) >>> sp =3D 0xde2c58a0 fp =3D 0xde2c58a8 >>> r0 =3D 0x00000000 r1 =3D 0x00000001 >>> r2 =3D 0x00000012 r3 =3D 0x00000000 >>> r4 =3D 0xc0b23748 r5 =3D 0x00000000 >>> r6 =3D 0xc07372ef r7 =3D 0xc0b13968 >>> r8 =3D 0xc093fa0c r9 =3D 0xde2c58e4 >>> r10 =3D 0xc0b13a68 r12 =3D 0x00000000 >>> kdb_enter() at kdb_enter+0x58 >>> pc =3D 0xc033b04c lr =3D 0xc02e9ca0 (vpanic+0x18c) >>> sp =3D 0xde2c58b0 fp =3D 0xde2c58d0 >>> r4 =3D 0x00000100 r10 =3D 0xc0b13a68 >>> vpanic() at vpanic+0x18c >>> pc =3D 0xc02e9ca0 lr =3D 0xc02e9a34 (dump_savectx) >>> sp =3D 0xde2c58d8 fp =3D 0xde2c58dc >>> r4 =3D 0xd70c8600 r5 =3D 0xde2c5e90 >>> r6 =3D 0xc3398090 r7 =3D 0xe0cfc440 >>> r8 =3D 0xc3398080 r9 =3D 0xd70c8600 >>> r10 =3D 0xde2c5960 >>> dump_savectx() at dump_savectx >>> pc =3D 0xc02e9a34 lr =3D 0xc05f51dc (set_regs) >>> sp =3D 0xde2c58e4 fp =3D 0xde2c58f8 >>> set_regs() at set_regs >>> pc =3D 0xc05f51dc lr =3D 0xc026f8f0 = (elf32_get_fpregset+0x2c) >>> sp =3D 0xde2c5900 fp =3D 0xde2c5908 >>> r4 =3D 0xc3398090 r5 =3D 0xc026f8c4 >>> elf32_get_fpregset() at elf32_get_fpregset+0x2c >>> pc =3D 0xc026f8f0 lr =3D 0xc026d848 (elf32_coredump+0x308) >>> sp =3D 0xde2c5910 fp =3D 0xde2c5988 >>> r4 =3D 0xc0902a7c r10 =3D 0xde2c5960 >>> elf32_coredump() at elf32_coredump+0x308 >>> pc =3D 0xc026d848 lr =3D 0xc02eea74 (sigexit+0xce0) >>> sp =3D 0xde2c5990 fp =3D 0xde2c5cf8 >>> r4 =3D 0x0000004e r5 =3D 0xdf580b60 >>> r6 =3D 0xdf580a78 r7 =3D 0xc026d540 >>> r8 =3D 0xdddcb2bc r9 =3D 0xdf580ad4 >>> r10 =3D 0x00000000 >>> sigexit() at sigexit+0xce0 >>> pc =3D 0xc02eea74 lr =3D 0xc02ef36c (postsig+0x128) >>> sp =3D 0xde2c5d00 fp =3D 0xde2c5d88 >>> r4 =3D 0x00000006 r5 =3D 0xdd43fba0 >>> r6 =3D 0xde2c5d20 r7 =3D 0xde2c5d18 >>> r8 =3D 0xdddcb1f8 r9 =3D 0xdf3d9ab8 >>> r10 =3D 0x00000005 >>> postsig() at postsig+0x128 >>> pc =3D 0xc02ef36c lr =3D 0xc02f316c (ast_sig+0x11c) >>> sp =3D 0xde2c5d90 fp =3D 0xde2c5e08 >>> r4 =3D 0xdd43fba0 r5 =3D 0xdddcb2bc >>> r6 =3D 0xc0734d22 r7 =3D 0x00000000 >>> r8 =3D 0xdddcb1f8 r9 =3D 0x00000ab8 >>> r10 =3D 0x22530384 >>> ast_sig() at ast_sig+0x11c >>> pc =3D 0xc02f316c lr =3D 0xc035444c (ast_handler+0xe0) >>> sp =3D 0xde2c5e10 fp =3D 0xde2c5e28 >>> r4 =3D 0xde2c5e40 r5 =3D 0x0000000e >>> r6 =3D 0x00004000 r7 =3D 0xc096b59c >>> r8 =3D 0xdd43fba0 r9 =3D 0x00000001 >>> ast_handler() at ast_handler+0xe0 >>> pc =3D 0xc035444c lr =3D 0xc035435c (ast+0x20) >>> sp =3D 0xde2c5e30 fp =3D 0xde2c5e38 >>> r4 =3D 0xde2c5e40 r5 =3D 0xdd43fba0 >>> r6 =3D 0x00000000 r7 =3D 0x000001b1 >>> r8 =3D 0x22c4b500 r9 =3D 0x00000000 >>> ast() at ast+0x20 >>> pc =3D 0xc035435c lr =3D 0xc05eaa88 (swi_exit+0x3c) >>> sp =3D 0xde2c5e40 fp =3D 0xbb9fbe38 >>> r4 =3D 0x60000013 r5 =3D 0xdd43fba0 >>> swi_exit() at swi_exit+0x3c >>> pc =3D 0xc05eaa88 lr =3D 0xc05eaa88 (swi_exit+0x3c) >>> sp =3D 0xde2c5e40 fp =3D 0xbb9fbe38 >>> db>=20 >>>=20 >>> The machine was last updated about a week ago, the >>> sources were updated earlier today. This panic is >>> new to me. >>=20 >=20 > I now have a small C++ program that, when aborted > by SIGABRT on armv7 (say via control-\), gets the > above type of FreeBSD crash while trying to produce > the *.core file (debug style armv7 kernel in use). >=20 > I've sent the authors of the recent > VFP-use-in-armv7-kernel changes the details, also: > Warner L. . >=20 > I previously sent them a small C program that gets a > KASSERT based panic for a debug armv7 kernel when > run under gdb or lldb with a breakpoint at a > specific routine. >=20 > In general, looks like armv7 floating point use is now > problematical on main's [so: 14's] armv7 kernel until > more work is done. >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com <http://yahoo.com/>; --Apple-Mail=_8B4575DD-EC51-4B1A-8535-DCF534833113 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;">Can you try = with 24abb6b82102eec577eff9bd8dd7726e8cab89f4? There were = conditional branch instructions that may mean the function to save the = VFP state was not being = run.<div><br></div><div>Andrew<br><div><br><blockquote = type=3D"cite"><div>On 16 Feb 2023, at 19:35, Mark Millard = <marklmi@yahoo.com> wrote:</div><br = class=3D"Apple-interchange-newline"><div><meta charset=3D"UTF-8"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">On Feb 14, 2023, at 23:16, Mark Millard = <</span><a href=3D"mailto:marklmi@yahoo.com" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px;">marklmi@yahoo.com</a><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">> wrote:</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;">On Feb 14, 2023, = at 20:16, Warner Losh <imp@bsdimp.com> wrote:<br><br><blockquote = type=3D"cite">Sorry to top post... what program was dumping core? Looks = like a too strict assert<br></blockquote><br>Just a possible point, = given recent kernel floating<br>point work:<br><br>Because of Bob's = note, I tried to do a typical build<br>and test of some benchmark = programs that I sometimes<br>use that involve floating point in some of = the<br>programs, some use with multithreading involved. (As<br>FreeBSD = and g++ progress I tend to do this once and<br>a while, not as often on = armv7 as on aarch64.)<br><br>On armv7, I now get a message about a = failure of an<br>internal cross-check, which also leads to the = program<br>being stopped early. The messaging from run to run<br>varies = what the failure is, but the runs should not<br>vary and should not fail = the cross-checks --and<br>previously did not, including when I last = tried armv7.<br>(Not recently.)<br><br>For the specific example failure, = the initial serial<br>(single thread) test with float involved works but = the<br>following multi-thread test in the same program fails<br>and = causes the program to stop when it notices there<br>is a = problem.<br><br>The programs that do not test floating point do = not<br>fail. These can involve floating point outside the<br>algorithm = benchmarked, but with no multi-threading<br>involved for such and no = floating point based cross-<br>checks involved.<br><br>At this point it = is far from obvious to me how I<br>would trackdown the specifics of what = leads to the<br>failed cross-checks. But the above is suggestive = of<br>there being problems for armv7 handling of saving<br>and restoring = floating point context for<br>multi-threading. I've no clue if such are = limited<br>to the floating point values or not.<br><br><blockquote = type=3D"cite">Warner<br><br>On Tue, Feb 14, 2023, 7:57 PM bob prohaska = <fbsd@www.zefox.net> wrote:<br>Building world on an RPi2 armv7, = buildworld stopped with<br>bob@www:/usr/src % panic: Called fill_fpregs = while the kernel is using the VFP<br>cpuid =3D 0<br>time =3D = 1676427410<br>KDB: stack backtrace:<br>db_trace_self() at = db_trace_self<br> pc =3D = 0xc05e8160 lr =3D 0xc007aa04 = (db_trace_self_wrapper+0x30)<br> = sp =3D 0xde2c5790 fp =3D 0xde2c58a8<br>db_trace_self_wrapper() at = db_trace_self_wrapper+0x30<br> pc= =3D 0xc007aa04 lr =3D 0xc02e9c54 = (vpanic+0x140)<br> sp =3D = 0xde2c58b0 fp =3D = 0xde2c58d0<br> r4 =3D = 0x00000100 r5 =3D = 0x00000000<br> r6 =3D = 0xc07372ef r7 =3D 0xc0b13968<br>vpanic() at = vpanic+0x140<br> pc =3D = 0xc02e9c54 lr =3D 0xc02e9a34 = (dump_savectx)<br> sp =3D = 0xde2c58d8 fp =3D = 0xde2c58dc<br> r4 =3D = 0xd70c8600 r5 =3D = 0xde2c5e90<br> r6 =3D = 0xc3398090 r7 =3D = 0xe0cfc440<br> r8 =3D = 0xc3398080 r9 =3D = 0xd70c8600<br> r10 =3D = 0xde2c5960<br>dump_savectx() at = dump_savectx<br> pc =3D = 0xc02e9a34 lr =3D 0xc05f51dc = (set_regs)<br> sp =3D = 0xde2c58e4 fp =3D 0xde2c58f8<br>set_regs() at = set_regs<br> pc =3D 0xc05f51dc = lr =3D 0xc026f8f0 = (elf32_get_fpregset+0x2c)<br> sp = =3D 0xde2c5900 fp =3D = 0xde2c5908<br> r4 =3D = 0xc3398090 r5 =3D 0xc026f8c4<br>elf32_get_fpregset() at = elf32_get_fpregset+0x2c<br> pc = =3D 0xc026f8f0 lr =3D 0xc026d848 = (elf32_coredump+0x308)<br> sp =3D= 0xde2c5910 fp =3D = 0xde2c5988<br> r4 =3D = 0xc0902a7c r10 =3D 0xde2c5960<br>elf32_coredump() at = elf32_coredump+0x308<br> pc =3D = 0xc026d848 lr =3D 0xc02eea74 = (sigexit+0xce0)<br> sp =3D = 0xde2c5990 fp =3D = 0xde2c5cf8<br> r4 =3D = 0x0000004e r5 =3D = 0xdf580b60<br> r6 =3D = 0xdf580a78 r7 =3D = 0xc026d540<br> r8 =3D = 0xdddcb2bc r9 =3D = 0xdf580ad4<br> r10 =3D = 0x00000000<br>sigexit() at = sigexit+0xce0<br> pc =3D = 0xc02eea74 lr =3D 0xc02ef36c = (postsig+0x128)<br> sp =3D = 0xde2c5d00 fp =3D = 0xde2c5d88<br> r4 =3D = 0x00000006 r5 =3D = 0xdd43fba0<br> r6 =3D = 0xde2c5d20 r7 =3D = 0xde2c5d18<br> r8 =3D = 0xdddcb1f8 r9 =3D = 0xdf3d9ab8<br> r10 =3D = 0x00000005<br>postsig() at = postsig+0x128<br> pc =3D = 0xc02ef36c lr =3D 0xc02f316c = (ast_sig+0x11c)<br> sp =3D = 0xde2c5d90 fp =3D = 0xde2c5e08<br> r4 =3D = 0xdd43fba0 r5 =3D = 0xdddcb2bc<br> r6 =3D = 0xc0734d22 r7 =3D = 0x00000000<br> r8 =3D = 0xdddcb1f8 r9 =3D = 0x00000ab8<br> r10 =3D = 0x22530384<br>ast_sig() at = ast_sig+0x11c<br> pc =3D = 0xc02f316c lr =3D 0xc035444c = (ast_handler+0xe0)<br> sp =3D = 0xde2c5e10 fp =3D = 0xde2c5e28<br> r4 =3D = 0xde2c5e40 r5 =3D = 0x0000000e<br> r6 =3D = 0x00004000 r7 =3D = 0xc096b59c<br> r8 =3D = 0xdd43fba0 r9 =3D 0x00000001<br>ast_handler() at = ast_handler+0xe0<br> pc =3D = 0xc035444c lr =3D 0xc035435c = (ast+0x20)<br> sp =3D = 0xde2c5e30 fp =3D = 0xde2c5e38<br> r4 =3D = 0xde2c5e40 r5 =3D = 0xdd43fba0<br> r6 =3D = 0x00000000 r7 =3D = 0x000001b1<br> r8 =3D = 0x22c4b500 r9 =3D 0x00000000<br>ast() at = ast+0x20<br> pc =3D 0xc035435c = lr =3D 0xc05eaa88 = (swi_exit+0x3c)<br> sp =3D = 0xde2c5e40 fp =3D = 0xbb9fbe38<br> r4 =3D = 0x60000013 r5 =3D 0xdd43fba0<br>swi_exit() at = swi_exit+0x3c<br> pc =3D = 0xc05eaa88 lr =3D 0xc05eaa88 = (swi_exit+0x3c)<br> sp =3D = 0xde2c5e40 fp =3D 0xbb9fbe38<br>KDB: enter: panic<br>[ thread pid = 81621 tid 101111 ]<br>Stopped at = kdb_enter+0x54: ldrb = r15, [r15, r15, ror r15]!<br>db> bt<br>Tracing pid = 81621 tid 101111 td 0xdd43fba0<br>db_trace_self() at = db_trace_self<br> pc =3D = 0xc05e8160 lr =3D 0xc00774a0 = (db_stack_trace+0x140)<br> sp =3D= 0xde2c55d8 fp =3D 0xde2c55f0<br>db_stack_trace() at = db_stack_trace+0x140<br> pc =3D = 0xc00774a0 lr =3D 0xc00770f0 = (db_command+0x310)<br> sp =3D = 0xde2c55f8 fp =3D = 0xde2c56a0<br> r4 =3D = 0xc0745722 r5 =3D = 0x00000062<br> r6 =3D = 0x00000000 r10 =3D 0x00000000<br>db_command() at = db_command+0x310<br> pc =3D = 0xc00770f0 lr =3D 0xc0076db8 = (db_command_loop+0x64)<br> sp =3D= 0xde2c56a8 fp =3D = 0xde2c56b8<br> r4 =3D = 0xc07ac186 r5 =3D = 0xc07ab7fe<br> r6 =3D = 0xc0986f5c r7 =3D = 0xc0b13968<br> r8 =3D = 0xc0b23738 r9 =3D = 0x00000000<br> r10 =3D = 0x00000001<br>db_command_loop() at = db_command_loop+0x64<br> pc =3D = 0xc0076db8 lr =3D 0xc007ab88 = (db_trap+0x128)<br> sp =3D = 0xde2c56c0 fp =3D = 0xde2c57d8<br> r4 =3D = 0x00000000 r5 =3D = 0xc0986f50<br> r6 =3D = 0xc0b23758 r10 =3D 0x00000001<br>db_trap() at = db_trap+0x128<br> pc =3D = 0xc007ab88 lr =3D 0xc033bb84 = (kdb_trap+0x258)<br> sp =3D = 0xde2c57e0 fp =3D = 0xde2c5808<br> r4 =3D = 0xc078390c r5 =3D = 0xc08d5270<br> r6 =3D = 0xc0b23758 r7 =3D 0xc0b13968<br>kdb_trap() at = kdb_trap+0x258<br> pc =3D = 0xc033bb84 lr =3D 0xc05eaab8 = (exception_exit)<br> sp =3D = 0xde2c5810 fp =3D = 0xde2c58a8<br> r4 =3D = 0x200000d3 r5 =3D = 0x00000000<br> r6 =3D = 0xc07372ef r7 =3D = 0xc0b13968<br> r8 =3D = 0xc093fa0c r9 =3D = 0xde2c58e4<br> r10 =3D = 0xc0b13a68<br>exception_exit() at = exception_exit<br> pc =3D = 0xc05eaab8 lr =3D 0xc033b044 = (kdb_enter+0x50)<br> sp =3D = 0xde2c58a0 fp =3D = 0xde2c58a8<br> r0 =3D = 0x00000000 r1 =3D = 0x00000001<br> r2 =3D = 0x00000012 r3 =3D = 0x00000000<br> r4 =3D = 0xc0b23748 r5 =3D = 0x00000000<br> r6 =3D = 0xc07372ef r7 =3D = 0xc0b13968<br> r8 =3D = 0xc093fa0c r9 =3D = 0xde2c58e4<br> r10 =3D 0xc0b13a68 r12 = =3D 0x00000000<br>kdb_enter() at = kdb_enter+0x58<br> pc =3D = 0xc033b04c lr =3D 0xc02e9ca0 = (vpanic+0x18c)<br> sp =3D = 0xde2c58b0 fp =3D = 0xde2c58d0<br> r4 =3D = 0x00000100 r10 =3D 0xc0b13a68<br>vpanic() at = vpanic+0x18c<br> pc =3D = 0xc02e9ca0 lr =3D 0xc02e9a34 = (dump_savectx)<br> sp =3D = 0xde2c58d8 fp =3D = 0xde2c58dc<br> r4 =3D = 0xd70c8600 r5 =3D = 0xde2c5e90<br> r6 =3D = 0xc3398090 r7 =3D = 0xe0cfc440<br> r8 =3D = 0xc3398080 r9 =3D = 0xd70c8600<br> r10 =3D = 0xde2c5960<br>dump_savectx() at = dump_savectx<br> pc =3D = 0xc02e9a34 lr =3D 0xc05f51dc = (set_regs)<br> sp =3D = 0xde2c58e4 fp =3D 0xde2c58f8<br>set_regs() at = set_regs<br> pc =3D 0xc05f51dc = lr =3D 0xc026f8f0 = (elf32_get_fpregset+0x2c)<br> sp = =3D 0xde2c5900 fp =3D = 0xde2c5908<br> r4 =3D = 0xc3398090 r5 =3D 0xc026f8c4<br>elf32_get_fpregset() at = elf32_get_fpregset+0x2c<br> pc = =3D 0xc026f8f0 lr =3D 0xc026d848 = (elf32_coredump+0x308)<br> sp =3D= 0xde2c5910 fp =3D = 0xde2c5988<br> r4 =3D = 0xc0902a7c r10 =3D 0xde2c5960<br>elf32_coredump() at = elf32_coredump+0x308<br> pc =3D = 0xc026d848 lr =3D 0xc02eea74 = (sigexit+0xce0)<br> sp =3D = 0xde2c5990 fp =3D = 0xde2c5cf8<br> r4 =3D = 0x0000004e r5 =3D = 0xdf580b60<br> r6 =3D = 0xdf580a78 r7 =3D = 0xc026d540<br> r8 =3D = 0xdddcb2bc r9 =3D = 0xdf580ad4<br> r10 =3D = 0x00000000<br>sigexit() at = sigexit+0xce0<br> pc =3D = 0xc02eea74 lr =3D 0xc02ef36c = (postsig+0x128)<br> sp =3D = 0xde2c5d00 fp =3D = 0xde2c5d88<br> r4 =3D = 0x00000006 r5 =3D = 0xdd43fba0<br> r6 =3D = 0xde2c5d20 r7 =3D = 0xde2c5d18<br> r8 =3D = 0xdddcb1f8 r9 =3D = 0xdf3d9ab8<br> r10 =3D = 0x00000005<br>postsig() at = postsig+0x128<br> pc =3D = 0xc02ef36c lr =3D 0xc02f316c = (ast_sig+0x11c)<br> sp =3D = 0xde2c5d90 fp =3D = 0xde2c5e08<br> r4 =3D = 0xdd43fba0 r5 =3D = 0xdddcb2bc<br> r6 =3D = 0xc0734d22 r7 =3D = 0x00000000<br> r8 =3D = 0xdddcb1f8 r9 =3D = 0x00000ab8<br> r10 =3D = 0x22530384<br>ast_sig() at = ast_sig+0x11c<br> pc =3D = 0xc02f316c lr =3D 0xc035444c = (ast_handler+0xe0)<br> sp =3D = 0xde2c5e10 fp =3D = 0xde2c5e28<br> r4 =3D = 0xde2c5e40 r5 =3D = 0x0000000e<br> r6 =3D = 0x00004000 r7 =3D = 0xc096b59c<br> r8 =3D = 0xdd43fba0 r9 =3D 0x00000001<br>ast_handler() at = ast_handler+0xe0<br> pc =3D = 0xc035444c lr =3D 0xc035435c = (ast+0x20)<br> sp =3D = 0xde2c5e30 fp =3D = 0xde2c5e38<br> r4 =3D = 0xde2c5e40 r5 =3D = 0xdd43fba0<br> r6 =3D = 0x00000000 r7 =3D = 0x000001b1<br> r8 =3D = 0x22c4b500 r9 =3D 0x00000000<br>ast() at = ast+0x20<br> pc =3D 0xc035435c = lr =3D 0xc05eaa88 = (swi_exit+0x3c)<br> sp =3D = 0xde2c5e40 fp =3D = 0xbb9fbe38<br> r4 =3D = 0x60000013 r5 =3D 0xdd43fba0<br>swi_exit() at = swi_exit+0x3c<br> pc =3D = 0xc05eaa88 lr =3D 0xc05eaa88 = (swi_exit+0x3c)<br> sp =3D = 0xde2c5e40 fp =3D 0xbb9fbe38<br>db><span = class=3D"Apple-converted-space"> </span><br><br>The machine was = last updated about a week ago, the<br>sources were updated earlier = today. This panic is<br>new to me.<br></blockquote><br></blockquote><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">I now have a small C++ program that, when = aborted</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">by SIGABRT on armv7 (say = via control-\), gets the</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">above = type of FreeBSD crash while trying to produce</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">the *.core file (debug style armv7 kernel = in use).</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">I've sent the authors of = the recent</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">VFP-use-in-armv7-kernel = changes the details, also:</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">Warner = L. .</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">I previously sent them a = small C program that gets a</span><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">KASSERT = based panic for a debug armv7 kernel when</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">run = under gdb or lldb with a breakpoint at a</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">specific routine.</span><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">In = general, looks like armv7 floating point use is now</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">problematical on main's [so: 14's] armv7 = kernel until</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">more work is = done.</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">=3D=3D=3D</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">Mark Millard</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">marklmi = at<span class=3D"Apple-converted-space"> </span></span><a = href=3D"http://yahoo.com/" style=3D"font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: = 0px;">yahoo.com</a></div></blockquote></div><br></div></body></html>= --Apple-Mail=_8B4575DD-EC51-4B1A-8535-DCF534833113--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CCE79CB7-BA79-4682-AC7C-4D5E8EC0A21A>