Date: Mon, 15 Jan 2024 11:53:31 +0100 From: Peter Blok <pblok@bsd4all.org> To: FreeBSD User <freebsd@walstatt-de.de> Cc: Cy Schubert <Cy.Schubert@cschubert.com>, Rick Macklem <rick.macklem@gmail.com>, Ronald Klop <ronald-lists@klop.ws>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: NFSv4 crash of CURRENT Message-ID: <683EF50F-6665-4664-A7CE-1EFE50076FB0@bsd4all.org> In-Reply-To: <D6D597BF-A98B-4864-B754-CB0B9A7A5A66@bsd4all.org> References: <20240113193324.3fd54295@thor.intern.walstatt.dynvpn.de> <1369645989.13766.1705178331205@localhost> <CAM5tNy5aat8vUn2fsX9jV=D9yGZdnO20Q0Ea7qtszx%2BzSES2bw@mail.gmail.com> <20240115043412.B6998C8@slippy.cwsent.com> <20240115064704.611fe0c4@thor.intern.walstatt.dynvpn.de> <D6D597BF-A98B-4864-B754-CB0B9A7A5A66@bsd4all.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
Hi,
Forgot to mention I’m on 13-stable. The fix that is causing the crash with automounted NFS is:
commit cc5cda1dbaa907ce52074f47264cc45b5a7d6c8b
Author: Konstantin Belousov <kib@FreeBSD.org>
Date: Tue Jan 2 00:22:44 2024 +0200
nfsclient: limit situations when we do unlocked read-ahead by nfsiod
(cherry picked from commit 70dc6b2ce314a0f32755005ad02802fca7ed186e)
When I remove the fix, the problem is gone. Add it back and the crash happens.
Peter
> On 15 Jan 2024, at 09:31, Peter Blok <pblok@bsd4all.org> wrote:
>
> Hi,
>
> I do have a crash on a NFS client with stable of today (4c4633fdffbe8e4b6d328c2bc9bb3edacc9ab50a). It is also autofs related. Maybe it is the same problem.
>
> I have ports automounted on /am/ports. When I do cd /am/ports/sys and type tab to autocomplete it crashes with the below stack trace. If I plainly mount ports on /usr/ports and do the same everything works. I am using NFSv3
>
> Peter
>
>
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 04
> fault virtual address = 0x89
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff809645d4
> stack pointer = 0x28:0xfffffe00acadb830
> frame pointer = 0x28:0xfffffe00acadb830
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 6869 (csh)
> trap number = 12
> panic: page fault
> cpuid = 2
> time = 1705306940
> KDB: stack backtrace:
> #0 0xffffffff806232f5 at kdb_backtrace+0x65
> #1 0xffffffff805d7a02 at vpanic+0x152
> #2 0xffffffff805d78a3 at panic+0x43
> #3 0xffffffff809d58ad at trap_fatal+0x38d
> #4 0xffffffff809d58ff at trap_pfault+0x4f
> #5 0xffffffff809af048 at calltrap+0x8
> #6 0xffffffff804c7a7e at ncl_bioread+0xb7e
> #7 0xffffffff804b9d90 at nfs_readdir+0x1f0
> #8 0xffffffff8069c61a at vop_sigdefer+0x2a
> #9 0xffffffff809f8ae0 at VOP_READDIR_APV+0x20
> #10 0xffffffff81ce75de at autofs_readdir+0x2ce
> #11 0xffffffff809f8ae0 at VOP_READDIR_APV+0x20
> #12 0xffffffff806c3002 at kern_getdirentries+0x222
> #13 0xffffffff806c33a9 at sys_getdirentries+0x29
> #14 0xffffffff809d6180 at amd64_syscall+0x110
> #15 0xffffffff809af95b at fast_syscall_common+0xf8
>
>
>
>> On 15 Jan 2024, at 06:46, FreeBSD User <freebsd@walstatt-de.de <mailto:freebsd@walstatt-de.de>> wrote:
>>
>> Am Sun, 14 Jan 2024 20:34:12 -0800
>> Cy Schubert <Cy.Schubert@cschubert.com <mailto:Cy.Schubert@cschubert.com>> schrieb:
>>
>>> In message <CAM5tNy5aat8vUn2fsX9jV=D9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmail.c <mailto:CAM5tNy5aat8vUn2fsX9jV=D9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmail.c>
>>> om>
>>> , Rick Macklem writes:
>>>> On Sat, Jan 13, 2024 at 12:39=E2=80=AFPM Ronald Klop <ronald-lists@klop.ws <mailto:ronald-lists@klop.ws>>=
>>>> wrote:
>>>>>
>>>>>
>>>>> Van: FreeBSD User <freebsd@walstatt-de.de <mailto:freebsd@walstatt-de.de>>
>>>>> Datum: 13 januari 2024 19:34
>>>>> Aan: FreeBSD CURRENT <freebsd-current@freebsd.org <mailto:freebsd-current@freebsd.org>>
>>>>> Onderwerp: NFSv4 crash of CURRENT
>>>>>
>>>>> Hello,
>>>>>
>>>>> running CURRENT client (FreeBSD 15.0-CURRENT #4 main-n267556-69748e62e82a=
>>>> : Sat Jan 13 18:08:32
>>>>> CET 2024 amd64). One NFSv4 server is same OS revision as the mentioned cl=
>>>> ient, other is FreeBSD
>>>>> 13.2-RELEASE-p8. Both offer NFSv4 filesystems, non-kerberized.
>>>>>
>>>>> I can crash the client reproducable by accessing the one or other NFSv4 F=
>>>> S (a simple ls -la).
>>>>> The NFSv4 FS is backed by ZFS (if this matters). I do not have physicla a=
>>>> ccess to the client
>>>>> host, luckily the box recovers.
>>>> Did you rebuild both the nfscommon and nfscl modules from the same sources?
>>>> I did a commit to main that changes the interface between these two
>>>> modules and did bump the
>>>> __FreeBSD_version to 1500010, which should cause both to be rebuilt.
>>>> (If you have "options NFSCL" in your kernel config, both should have
>>>> been rebuilt as a part of
>>>> the kernel build.)
>>>>
>>>
>>> Is anyone by chance seeing autofs in the backtrace too?
>>>
>>>
>>
>> Hello Cy Shubert,
>>
>> I forgot to mention that those crashes occur with autofs mounted filesystems. Good question,
>> by the way, I will check whether crashes also happen when mounting the tradidional way.
>>
>> Kind regards,
>>
>> oh
>>
>> --
>> O. Hartmann
>
[-- Attachment #2 --]
<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">Forgot to mention I’m on 13-stable. The fix that is causing the crash with automounted NFS is:</div><div class=""><br class=""></div><div class=""><div class="">commit cc5cda1dbaa907ce52074f47264cc45b5a7d6c8b</div><div class="">Author: Konstantin Belousov <<a href="mailto:kib@FreeBSD.org" class="">kib@FreeBSD.org</a>></div><div class="">Date: Tue Jan 2 00:22:44 2024 +0200</div><div class=""><br class=""></div><div class=""> nfsclient: limit situations when we do unlocked read-ahead by nfsiod</div><div class=""> </div><div class=""> (cherry picked from commit 70dc6b2ce314a0f32755005ad02802fca7ed186e)</div><div class=""><br class=""></div><div class="">When I remove the fix, the problem is gone. Add it back and the crash happens.</div><div class=""><br class=""></div><div class="">Peter</div><div><br class=""><blockquote type="cite" class=""><div class="">On 15 Jan 2024, at 09:31, Peter Blok <<a href="mailto:pblok@bsd4all.org" class="">pblok@bsd4all.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">I do have a crash on a NFS client with stable of today (4c4633fdffbe8e4b6d328c2bc9bb3edacc9ab50a). It is also autofs related. Maybe it is the same problem.</div><div class=""><br class=""></div><div class="">I have ports automounted on /am/ports. When I do cd /am/ports/sys and type tab to autocomplete it crashes with the below stack trace. If I plainly mount ports on /usr/ports and do the same everything works. I am using NFSv3</div><div class=""><br class=""></div><div class="">Peter</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">Fatal trap 12: page fault while in kernel mode</div><div class="">cpuid = 2; apic id = 04</div><div class="">fault virtual address<span class="Apple-tab-span" style="white-space:pre"> </span>= 0x89</div><div class="">fault code<span class="Apple-tab-span" style="white-space:pre"> </span>= supervisor read data, page not present</div><div class="">instruction pointer<span class="Apple-tab-span" style="white-space:pre"> </span>= 0x20:0xffffffff809645d4</div><div class="">stack pointer<span class="Apple-tab-span" style="white-space:pre"> </span> = 0x28:0xfffffe00acadb830</div><div class="">frame pointer<span class="Apple-tab-span" style="white-space:pre"> </span> = 0x28:0xfffffe00acadb830</div><div class="">code segment<span class="Apple-tab-span" style="white-space:pre"> </span>= base 0x0, limit 0xfffff, type 0x1b</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>= DPL 0, pres 1, long 1, def32 0, gran 1</div><div class="">processor eflags<span class="Apple-tab-span" style="white-space:pre"> </span>= interrupt enabled, resume, IOPL = 0</div><div class="">current process<span class="Apple-tab-span" style="white-space:pre"> </span>= 6869 (csh)</div><div class="">trap number<span class="Apple-tab-span" style="white-space:pre"> </span>= 12</div><div class="">panic: page fault</div><div class="">cpuid = 2</div><div class="">time = 1705306940</div><div class="">KDB: stack backtrace:</div><div class="">#0 0xffffffff806232f5 at kdb_backtrace+0x65</div><div class="">#1 0xffffffff805d7a02 at vpanic+0x152</div><div class="">#2 0xffffffff805d78a3 at panic+0x43</div><div class="">#3 0xffffffff809d58ad at trap_fatal+0x38d</div><div class="">#4 0xffffffff809d58ff at trap_pfault+0x4f</div><div class="">#5 0xffffffff809af048 at calltrap+0x8</div><div class="">#6 0xffffffff804c7a7e at ncl_bioread+0xb7e</div><div class="">#7 0xffffffff804b9d90 at nfs_readdir+0x1f0</div><div class="">#8 0xffffffff8069c61a at vop_sigdefer+0x2a</div><div class="">#9 0xffffffff809f8ae0 at VOP_READDIR_APV+0x20</div><div class="">#10 0xffffffff81ce75de at autofs_readdir+0x2ce</div><div class="">#11 0xffffffff809f8ae0 at VOP_READDIR_APV+0x20</div><div class="">#12 0xffffffff806c3002 at kern_getdirentries+0x222</div><div class="">#13 0xffffffff806c33a9 at sys_getdirentries+0x29</div><div class="">#14 0xffffffff809d6180 at amd64_syscall+0x110</div><div class="">#15 0xffffffff809af95b at fast_syscall_common+0xf8</div></div><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 15 Jan 2024, at 06:46, FreeBSD User <<a href="mailto:freebsd@walstatt-de.de" class="">freebsd@walstatt-de.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Am Sun, 14 Jan 2024 20:34:12 -0800</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Cy Schubert <</span><a href="mailto:Cy.Schubert@cschubert.com" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Cy.Schubert@cschubert.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">> schrieb:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">In message <<a href="mailto:CAM5tNy5aat8vUn2fsX9jV=D9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmail.c" class="">CAM5tNy5aat8vUn2fsX9jV=D9yGZdnO20Q0Ea7qtszx+zSES2bw@mail.gmail.c</a><br class="">om> <br class="">, Rick Macklem writes:<br class=""><blockquote type="cite" class="">On Sat, Jan 13, 2024 at 12:39=E2=80=AFPM Ronald Klop <<a href="mailto:ronald-lists@klop.ws" class="">ronald-lists@klop.ws</a>>=<br class="">wrote: <br class=""><blockquote type="cite" class=""><br class=""><br class="">Van: FreeBSD User <<a href="mailto:freebsd@walstatt-de.de" class="">freebsd@walstatt-de.de</a>><br class="">Datum: 13 januari 2024 19:34<br class="">Aan: FreeBSD CURRENT <<a href="mailto:freebsd-current@freebsd.org" class="">freebsd-current@freebsd.org</a>><br class="">Onderwerp: NFSv4 crash of CURRENT<br class=""><br class="">Hello,<br class=""><br class="">running CURRENT client (FreeBSD 15.0-CURRENT #4 main-n267556-69748e62e82a= <br class=""></blockquote>: Sat Jan 13 18:08:32 <br class=""><blockquote type="cite" class="">CET 2024 amd64). One NFSv4 server is same OS revision as the mentioned cl= <br class=""></blockquote>ient, other is FreeBSD <br class=""><blockquote type="cite" class="">13.2-RELEASE-p8. Both offer NFSv4 filesystems, non-kerberized.<br class=""><br class="">I can crash the client reproducable by accessing the one or other NFSv4 F= <br class=""></blockquote>S (a simple ls -la). <br class=""><blockquote type="cite" class="">The NFSv4 FS is backed by ZFS (if this matters). I do not have physicla a= <br class=""></blockquote>ccess to the client <br class=""><blockquote type="cite" class="">host, luckily the box recovers. <br class=""></blockquote>Did you rebuild both the nfscommon and nfscl modules from the same sources?<br class="">I did a commit to main that changes the interface between these two<br class="">modules and did bump the<br class="">__FreeBSD_version to 1500010, which should cause both to be rebuilt.<br class="">(If you have "options NFSCL" in your kernel config, both should have<br class="">been rebuilt as a part of<br class="">the kernel build.)<br class=""><br class=""></blockquote><br class="">Is anyone by chance seeing autofs in the backtrace too?<br class=""><br class=""><br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Hello Cy Shubert,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I forgot to mention that those crashes occur with autofs mounted filesystems. Good question,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">by the way, I will check whether crashes also happen when mounting the tradidional way.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Kind regards,</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">oh</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">O. Hartmann</span></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></body></html>
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?683EF50F-6665-4664-A7CE-1EFE50076FB0>