Date: Mon, 15 Jan 2024 11:53:31 +0100 From: Peter Blok <pblok@bsd4all.org> To: FreeBSD User <freebsd@walstatt-de.de> Cc: Cy Schubert <Cy.Schubert@cschubert.com>, Rick Macklem <rick.macklem@gmail.com>, Ronald Klop <ronald-lists@klop.ws>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: NFSv4 crash of CURRENT Message-ID: <683EF50F-6665-4664-A7CE-1EFE50076FB0@bsd4all.org> In-Reply-To: <D6D597BF-A98B-4864-B754-CB0B9A7A5A66@bsd4all.org> References: <20240113193324.3fd54295@thor.intern.walstatt.dynvpn.de> <1369645989.13766.1705178331205@localhost> <CAM5tNy5aat8vUn2fsX9jV=D9yGZdnO20Q0Ea7qtszx%2BzSES2bw@mail.gmail.com> <20240115043412.B6998C8@slippy.cwsent.com> <20240115064704.611fe0c4@thor.intern.walstatt.dynvpn.de> <D6D597BF-A98B-4864-B754-CB0B9A7A5A66@bsd4all.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_727E0BFF-35CE-446A-A17B-7C5202616E94
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Hi,
Forgot to mention I=E2=80=99m on 13-stable. The fix that is causing the =
crash with automounted NFS is:
commit cc5cda1dbaa907ce52074f47264cc45b5a7d6c8b
Author: Konstantin Belousov <kib@FreeBSD.org>
Date: Tue Jan 2 00:22:44 2024 +0200
nfsclient: limit situations when we do unlocked read-ahead by nfsiod
=20
(cherry picked from commit 70dc6b2ce314a0f32755005ad02802fca7ed186e)
When I remove the fix, the problem is gone. Add it back and the crash =
happens.
Peter
> On 15 Jan 2024, at 09:31, Peter Blok <pblok@bsd4all.org> wrote:
>=20
> Hi,
>=20
> I do have a crash on a NFS client with stable of today =
(4c4633fdffbe8e4b6d328c2bc9bb3edacc9ab50a). It is also autofs related. =
Maybe it is the same problem.
>=20
> I have ports automounted on /am/ports. When I do cd /am/ports/sys and =
type tab to autocomplete it crashes with the below stack trace. If I =
plainly mount ports on /usr/ports and do the same everything works. I am =
using NFSv3
>=20
> Peter
>=20
>=20
>=20
>=20
> Fatal trap 12: page fault while in kernel mode
> cpuid =3D 2; apic id =3D 04
> fault virtual address =3D 0x89
> fault code =3D supervisor read data, page not present
> instruction pointer =3D 0x20:0xffffffff809645d4
> stack pointer =3D 0x28:0xfffffe00acadb830
> frame pointer =3D 0x28:0xfffffe00acadb830
> code segment =3D base 0x0, limit 0xfffff, type 0x1b
> =3D DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags =3D interrupt enabled, resume, IOPL =3D 0
> current process =3D 6869 (csh)
> trap number =3D 12
> panic: page fault
> cpuid =3D 2
> time =3D 1705306940
> KDB: stack backtrace:
> #0 0xffffffff806232f5 at kdb_backtrace+0x65
> #1 0xffffffff805d7a02 at vpanic+0x152
> #2 0xffffffff805d78a3 at panic+0x43
> #3 0xffffffff809d58ad at trap_fatal+0x38d
> #4 0xffffffff809d58ff at trap_pfault+0x4f
> #5 0xffffffff809af048 at calltrap+0x8
> #6 0xffffffff804c7a7e at ncl_bioread+0xb7e
> #7 0xffffffff804b9d90 at nfs_readdir+0x1f0
> #8 0xffffffff8069c61a at vop_sigdefer+0x2a
> #9 0xffffffff809f8ae0 at VOP_READDIR_APV+0x20
> #10 0xffffffff81ce75de at autofs_readdir+0x2ce
> #11 0xffffffff809f8ae0 at VOP_READDIR_APV+0x20
> #12 0xffffffff806c3002 at kern_getdirentries+0x222
> #13 0xffffffff806c33a9 at sys_getdirentries+0x29
> #14 0xffffffff809d6180 at amd64_syscall+0x110
> #15 0xffffffff809af95b at fast_syscall_common+0xf8
>=20
>=20
>=20
>> On 15 Jan 2024, at 06:46, FreeBSD User <freebsd@walstatt-de.de =
<mailto:freebsd@walstatt-de.de>> wrote:
>>=20
>> Am Sun, 14 Jan 2024 20:34:12 -0800
>> Cy Schubert <Cy.Schubert@cschubert.com =
<mailto:Cy.Schubert@cschubert.com>> schrieb:
>>=20
>>> In message =
<CAM5tNy5aat8vUn2fsX9jV=3DD9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmail.c =
<mailto:CAM5tNy5aat8vUn2fsX9jV=3DD9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmail.c=
>
>>> om> =20
>>> , Rick Macklem writes:
>>>> On Sat, Jan 13, 2024 at 12:39=3DE2=3D80=3DAFPM Ronald Klop =
<ronald-lists@klop.ws <mailto:ronald-lists@klop.ws>>=3D
>>>> wrote: =20
>>>>>=20
>>>>>=20
>>>>> Van: FreeBSD User <freebsd@walstatt-de.de =
<mailto:freebsd@walstatt-de.de>>
>>>>> Datum: 13 januari 2024 19:34
>>>>> Aan: FreeBSD CURRENT <freebsd-current@freebsd.org =
<mailto:freebsd-current@freebsd.org>>
>>>>> Onderwerp: NFSv4 crash of CURRENT
>>>>>=20
>>>>> Hello,
>>>>>=20
>>>>> running CURRENT client (FreeBSD 15.0-CURRENT #4 =
main-n267556-69748e62e82a=3D =20
>>>> : Sat Jan 13 18:08:32 =20
>>>>> CET 2024 amd64). One NFSv4 server is same OS revision as the =
mentioned cl=3D =20
>>>> ient, other is FreeBSD =20
>>>>> 13.2-RELEASE-p8. Both offer NFSv4 filesystems, non-kerberized.
>>>>>=20
>>>>> I can crash the client reproducable by accessing the one or other =
NFSv4 F=3D =20
>>>> S (a simple ls -la). =20
>>>>> The NFSv4 FS is backed by ZFS (if this matters). I do not have =
physicla a=3D =20
>>>> ccess to the client =20
>>>>> host, luckily the box recovers. =20
>>>> Did you rebuild both the nfscommon and nfscl modules from the same =
sources?
>>>> I did a commit to main that changes the interface between these two
>>>> modules and did bump the
>>>> __FreeBSD_version to 1500010, which should cause both to be =
rebuilt.
>>>> (If you have "options NFSCL" in your kernel config, both should =
have
>>>> been rebuilt as a part of
>>>> the kernel build.)
>>>>=20
>>>=20
>>> Is anyone by chance seeing autofs in the backtrace too?
>>>=20
>>>=20
>>=20
>> Hello Cy Shubert,
>>=20
>> I forgot to mention that those crashes occur with autofs mounted =
filesystems. Good question,
>> by the way, I will check whether crashes also happen when mounting =
the tradidional way.
>>=20
>> Kind regards,
>>=20
>> oh
>>=20
>> --=20
>> O. Hartmann
>=20
--Apple-Mail=_727E0BFF-35CE-446A-A17B-7C5202616E94
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;" =
class=3D"">Hi,<div class=3D""><br class=3D""></div><div class=3D"">Forgot =
to mention I=E2=80=99m on 13-stable. The fix that is causing the crash =
with automounted NFS is:</div><div class=3D""><br class=3D""></div><div =
class=3D""><div class=3D"">commit =
cc5cda1dbaa907ce52074f47264cc45b5a7d6c8b</div><div class=3D"">Author: =
Konstantin Belousov <<a href=3D"mailto:kib@FreeBSD.org" =
class=3D"">kib@FreeBSD.org</a>></div><div class=3D"">Date: Tue =
Jan 2 00:22:44 2024 +0200</div><div class=3D""><br class=3D""></div><div =
class=3D""> nfsclient: limit situations when we do unlocked =
read-ahead by nfsiod</div><div class=3D""> </div><div =
class=3D""> (cherry picked from commit =
70dc6b2ce314a0f32755005ad02802fca7ed186e)</div><div class=3D""><br =
class=3D""></div><div class=3D"">When I remove the fix, the problem is =
gone. Add it back and the crash happens.</div><div class=3D""><br =
class=3D""></div><div class=3D"">Peter</div><div><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 15 =
Jan 2024, at 09:31, Peter Blok <<a href=3D"mailto:pblok@bsd4all.org" =
class=3D"">pblok@bsd4all.org</a>> wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><meta =
http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii" =
class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: =
space; line-break: after-white-space;" class=3D"">Hi,<div class=3D""><br =
class=3D""></div><div class=3D"">I do have a crash on a NFS client with =
stable of today (4c4633fdffbe8e4b6d328c2bc9bb3edacc9ab50a). It is also =
autofs related. Maybe it is the same problem.</div><div class=3D""><br =
class=3D""></div><div class=3D"">I have ports automounted on /am/ports. =
When I do cd /am/ports/sys and type tab to autocomplete it crashes with =
the below stack trace. If I plainly mount ports on /usr/ports and do the =
same everything works. I am using NFSv3</div><div class=3D""><br =
class=3D""></div><div class=3D"">Peter</div><div class=3D""><br =
class=3D""></div><div class=3D""><br class=3D""></div><div class=3D""><br =
class=3D""></div><div class=3D""><br class=3D""></div><div class=3D""><div=
class=3D"">Fatal trap 12: page fault while in kernel mode</div><div =
class=3D"">cpuid =3D 2; apic id =3D 04</div><div class=3D"">fault =
virtual address<span class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>=3D 0x89</div><div class=3D"">fault code<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>=3D=
supervisor read data, page not present</div><div class=3D"">instruction =
pointer<span class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>=3D 0x20:0xffffffff809645d4</div><div class=3D"">stack =
pointer<span class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span> =3D 0x28:0xfffffe00acadb830</div><div =
class=3D"">frame pointer<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> </span> =3D =
0x28:0xfffffe00acadb830</div><div class=3D"">code segment<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>=3D=
base 0x0, limit 0xfffff, type 0x1b</div><div class=3D""><span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>=3D DPL 0, pres 1, long 1, def32 0, gran 1</div><div =
class=3D"">processor eflags<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> </span>=3D interrupt enabled, resume, =
IOPL =3D 0</div><div class=3D"">current process<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>=3D=
6869 (csh)</div><div class=3D"">trap number<span class=3D"Apple-tab-span"=
style=3D"white-space:pre"> </span>=3D 12</div><div =
class=3D"">panic: page fault</div><div class=3D"">cpuid =3D 2</div><div =
class=3D"">time =3D 1705306940</div><div class=3D"">KDB: stack =
backtrace:</div><div class=3D"">#0 0xffffffff806232f5 at =
kdb_backtrace+0x65</div><div class=3D"">#1 0xffffffff805d7a02 at =
vpanic+0x152</div><div class=3D"">#2 0xffffffff805d78a3 at =
panic+0x43</div><div class=3D"">#3 0xffffffff809d58ad at =
trap_fatal+0x38d</div><div class=3D"">#4 0xffffffff809d58ff at =
trap_pfault+0x4f</div><div class=3D"">#5 0xffffffff809af048 at =
calltrap+0x8</div><div class=3D"">#6 0xffffffff804c7a7e at =
ncl_bioread+0xb7e</div><div class=3D"">#7 0xffffffff804b9d90 at =
nfs_readdir+0x1f0</div><div class=3D"">#8 0xffffffff8069c61a at =
vop_sigdefer+0x2a</div><div class=3D"">#9 0xffffffff809f8ae0 at =
VOP_READDIR_APV+0x20</div><div class=3D"">#10 0xffffffff81ce75de at =
autofs_readdir+0x2ce</div><div class=3D"">#11 0xffffffff809f8ae0 at =
VOP_READDIR_APV+0x20</div><div class=3D"">#12 0xffffffff806c3002 at =
kern_getdirentries+0x222</div><div class=3D"">#13 0xffffffff806c33a9 at =
sys_getdirentries+0x29</div><div class=3D"">#14 0xffffffff809d6180 at =
amd64_syscall+0x110</div><div class=3D"">#15 0xffffffff809af95b at =
fast_syscall_common+0xf8</div></div><div class=3D""><div class=3D""><br =
class=3D""></div><div class=3D""><br class=3D""></div><div class=3D""><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 15 =
Jan 2024, at 06:46, FreeBSD User <<a =
href=3D"mailto:freebsd@walstatt-de.de" =
class=3D"">freebsd@walstatt-de.de</a>> wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><meta =
charset=3D"UTF-8" class=3D""><span style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">Am Sun, 14 Jan 2024 20:34:12 -0800</span><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
13px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
13px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;" class=3D"">Cy Schubert <</span><a =
href=3D"mailto:Cy.Schubert@cschubert.com" style=3D"font-family: =
Helvetica; font-size: 13px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; orphans: auto; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; =
-webkit-text-stroke-width: 0px;" =
class=3D"">Cy.Schubert@cschubert.com</a><span style=3D"caret-color: =
rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: =
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: =
normal; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">> schrieb:</span><br style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><blockquote type=3D"cite" =
style=3D"font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D"">In message <<a =
href=3D"mailto:CAM5tNy5aat8vUn2fsX9jV=3DD9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.=
gmail.c" =
class=3D"">CAM5tNy5aat8vUn2fsX9jV=3DD9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmai=
l.c</a><br class=3D"">om> <br class=3D"">, Rick Macklem =
writes:<br class=3D""><blockquote type=3D"cite" class=3D"">On Sat, Jan =
13, 2024 at 12:39=3DE2=3D80=3DAFPM Ronald Klop <<a =
href=3D"mailto:ronald-lists@klop.ws" =
class=3D"">ronald-lists@klop.ws</a>>=3D<br class=3D"">wrote: =
<br class=3D""><blockquote type=3D"cite" class=3D""><br =
class=3D""><br class=3D"">Van: FreeBSD User <<a =
href=3D"mailto:freebsd@walstatt-de.de" =
class=3D"">freebsd@walstatt-de.de</a>><br class=3D"">Datum: 13 =
januari 2024 19:34<br class=3D"">Aan: FreeBSD CURRENT <<a =
href=3D"mailto:freebsd-current@freebsd.org" =
class=3D"">freebsd-current@freebsd.org</a>><br class=3D"">Onderwerp: =
NFSv4 crash of CURRENT<br class=3D""><br class=3D"">Hello,<br =
class=3D""><br class=3D"">running CURRENT client (FreeBSD 15.0-CURRENT =
#4 main-n267556-69748e62e82a=3D <br class=3D""></blockquote>: Sat =
Jan 13 18:08:32 <br class=3D""><blockquote type=3D"cite" =
class=3D"">CET 2024 amd64). One NFSv4 server is same OS revision as the =
mentioned cl=3D <br class=3D""></blockquote>ient, other is FreeBSD =
<br class=3D""><blockquote type=3D"cite" class=3D"">13.2-RELEASE-p8.=
Both offer NFSv4 filesystems, non-kerberized.<br class=3D""><br =
class=3D"">I can crash the client reproducable by accessing the one or =
other NFSv4 F=3D <br class=3D""></blockquote>S (a simple ls -la). =
<br class=3D""><blockquote type=3D"cite" class=3D"">The NFSv4 FS =
is backed by ZFS (if this matters). I do not have physicla a=3D =
<br class=3D""></blockquote>ccess to the client <br =
class=3D""><blockquote type=3D"cite" class=3D"">host, luckily the box =
recovers. <br class=3D""></blockquote>Did you rebuild both the =
nfscommon and nfscl modules from the same sources?<br class=3D"">I did a =
commit to main that changes the interface between these two<br =
class=3D"">modules and did bump the<br class=3D"">__FreeBSD_version to =
1500010, which should cause both to be rebuilt.<br class=3D"">(If you =
have "options NFSCL" in your kernel config, both should have<br =
class=3D"">been rebuilt as a part of<br class=3D"">the kernel build.)<br =
class=3D""><br class=3D""></blockquote><br class=3D"">Is anyone by =
chance seeing autofs in the backtrace too?<br class=3D""><br =
class=3D""><br class=3D""></blockquote><br style=3D"caret-color: rgb(0, =
0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">Hello Cy Shubert,</span><br style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">I forgot to mention that those crashes occur with autofs =
mounted filesystems. Good question,</span><br style=3D"caret-color: =
rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: =
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: =
normal; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">by the way, I will check whether crashes also happen when =
mounting the tradidional way.</span><br style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">Kind regards,</span><br style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, =
0); font-family: Helvetica; font-size: 13px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; float: none; display: inline !important;" =
class=3D"">oh</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 13px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none;" class=3D""><br style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 13px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none;" class=3D""><span style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 13px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none; float: none; display: inline !important;" class=3D"">--<span =
class=3D"Apple-converted-space"> </span></span><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
13px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
13px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;" class=3D"">O. =
Hartmann</span></div></blockquote></div><br =
class=3D""></div></div></div></blockquote></div><br =
class=3D""></div></body></html>=
--Apple-Mail=_727E0BFF-35CE-446A-A17B-7C5202616E94--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?683EF50F-6665-4664-A7CE-1EFE50076FB0>