Date: Tue, 7 Nov 2023 17:51:47 +0200 From: Oleg Nauman <oleg.nauman@gmail.com> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: FreeBSD Current <current@freebsd.org>, manu@freebsd.org Subject: Re: Kernel with INVARIANTS panicing if drm is loaded Message-ID: <CAC5YPTsz9zhG6=Rmobt6knamKJXG2DdNmb0MbyymBB=xyzeAzQ@mail.gmail.com> In-Reply-To: <20231107043221.A4E1CFA@slippy.cwsent.com> References: <2806934.Tv9pZ1cZgq@moonset.home> <20231107043221.A4E1CFA@slippy.cwsent.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000005ce56f060991f3ed
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Tue, Nov 7, 2023 at 6:33=E2=80=AFAM Cy Schubert <Cy.Schubert@cschubert.c=
om>
wrote:
> In message <2806934.Tv9pZ1cZgq@moonset.home>, "Oleg V. Nauman" writes:
> > I am observing kernel panic when entering multiuser mode after
> sucessful
> > system boot. It happens when I load CURRENT kernel with INVARIANTS and
> drm
> > module loaded ( drm-515-kmod-5.15.118_1 in particular ) . drm module an=
d
> kene
> > l
> > are in sync
> >
> >
> > FreeBSD moonset.home 15.0-CURRENT FreeBSD 15.0-CURRENT #3 main-n266267-
> > e116e040f309: Sun Nov 5 10:00:51 EET 2023 root@moonset.home
> :/usr/obj/usr
> > /
> > src/amd64.amd64/sys/moonset amd64
> >
> > panic: sleepq_add: td 0xfffff8000203c000 to sleep on wchan
> 0xfffff800021d8648
> >
> > with sleeping prohibited
> > ...............
> > Reading symbols from /boot/kernel/kernel...
> > Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
> >
> > Unread portion of the kernel message buffer:
> > panic: malloc(M_WAITOK) with sleeping prohibited
> > cpuid =3D 0
> > time =3D 1699171621
> > .............
> > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > 57 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n"
> (offsetof(struct
> > pcpu,
> > (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > #1 doadump (textdump=3D<optimized out>)
> > at /usr/src/sys/kern/kern_shutdown.c:405
> > #2 0xffffffff81e2ec53 in vt_kms_postswitch () from /boot/modules/drm.k=
o
> > #3 0xffffffff8043ad6e in vt_window_switch (vw=3D0xfffff800021d8640)
> > at /usr/src/sys/dev/vt/vt_core.c:595
> > #4 0xffffffff804ec583 in kern_reboot (howto=3D4)
> > at /usr/src/sys/kern/kern_shutdown.c:501
> > #5 0xffffffff804eccfa in vpanic (
> > fmt=3D0xffffffff808263fb "malloc(M_WAITOK) with sleeping prohibited=
",
> > ap=3Dap@entry=3D0xfffffe00ce259850) at
> /usr/src/sys/kern/kern_shutdown.c:970
> > #6 0xffffffff804ecb03 in panic (fmt=3D<unavailable>)
> > at /usr/src/sys/kern/kern_shutdown.c:894
> > #7 0xffffffff804c8a04 in malloc_dbg (vap=3D<optimized out>,
> > sizep=3D<optimized out>, mtp=3D<optimized out>, flags=3D<optimized =
out>)
> > at /usr/src/sys/kern/kern_malloc.c:540
> > #8 0xffffffff804c885c in malloc (size=3D<optimized out>,
> > mtp=3D0xffffffff81e931c0 <M_KMALLOC>, flags=3D<unavailable>)
> > at /usr/src/sys/kern/kern_malloc.c:641
> > #9 0xffffffff81c5a750 in intel_atomic_state_alloc ()
> > from /boot/modules/i915kms.ko
> > #10 0xffffffff81dfe404 in drm_client_modeset_commit_atomic ()
> > from /boot/modules/drm.ko
> > #11 0xffffffff81dfe614 in drm_client_modeset_commit_locked ()
> > from /boot/modules/drm.ko
> > #12 0xffffffff81dfe7a1 in drm_client_modeset_commit ()
> > from /boot/modules/drm.ko
> > #13 0xffffffff81e41ab3 in drm_fb_helper_restore_fbdev_mode_unlocked ()
> > from /boot/modules/drm.ko
> > #14 0xffffffff81e2ed91 in vt_kms_postswitch () from /boot/modules/drm.k=
o
> > #15 0xffffffff8043ac31 in vt_window_switch (vw=3D0xfffff80001d3a600,
> > vw@entry=3D0xffffffff80a47178 <vt_conswindow>)
> > at /usr/src/sys/dev/vt/vt_core.c:612
> > #16 0xffffffff8043be0f in vtterm_cngrab (tm=3D<unavailable>,
> > tm@entry=3D<error reading variable: value is not available>)
> > at /usr/src/sys/dev/vt/vt_core.c:1863
> > #17 0xffffffff804893f6 in cngrab () at /usr/src/sys/kern/kern_cons.c:38=
5
> > #18 0xffffffff804ecc79 in vpanic (
> > fmt=3D0xffffffff808799a9 "%s: td %p to sleep on wchan %p with sleep=
ing
> > prohibited", ap=3Dap@entry=3D0xfffffe00ce259c20)
> > at /usr/src/sys/kern/kern_shutdown.c:942
> > #19 0xffffffff804ecb03 in panic (fmt=3D<unavailable>)
> > at /usr/src/sys/kern/kern_shutdown.c:894
> > #20 0xffffffff805465b3 in sleepq_add (wchan=3Dwchan@entry
> =3D0xfffff800021d8648,
> > lock=3Dlock@entry=3D0xfffff80001823340,
> > wmesg=3Dwmesg@entry=3D0xffffffff808b194e "tq_drain", flags=3D0,
> > flags@entry=3D<unavailable>, queue=3Dqueue@entry=3D0)
> > at /usr/src/sys/kern/subr_sleepqueue.c:326
> > #21 0xffffffff804f8efe in _sleep (ident=3Dident@entry=3D0xfffff800021d8=
648,
> > lock=3Dlock@entry=3D0xfffff80001823340, priority=3Dpriority@entry=
=3D0,
> > wmesg=3D0xffffffff808b194e "tq_drain", sbt=3Dsbt@entry=3D0, pr=3Dpr=
@entry=3D0,
> > flags=3D256) at /usr/src/sys/kern/kern_synch.c:207
> > #22 0xffffffff8054cffb in TQ_SLEEP (tq=3D0xfffff80001823300,
> > p=3D0xfffff800021d8648, wm=3D<optimized out>)
> > at /usr/src/sys/kern/subr_taskqueue.c:124
> > #23 taskqueue_drain (queue=3D0xfffff80001823300, task=3D0xfffff800021d8=
648)
> > at /usr/src/sys/kern/subr_taskqueue.c:614
> > #24 0xffffffff81e2ed35 in vt_kms_postswitch () from /boot/modules/drm.k=
o
> > #25 0xffffffff8043ac31 in vt_window_switch (vw=3D0xfffff800021d8648,
> > vw@entry=3D0xfffff800038fb180) at /usr/src/sys/dev/vt/vt_core.c:612
> > #26 0xffffffff8043b3b2 in vt_late_window_switch (vw=3D0xfffff800038fb18=
0)
> > at /usr/src/sys/dev/vt/vt_core.c:468
> > #27 vt_proc_window_switch (vw=3D0xfffff800038fb180)
> > at /usr/src/sys/dev/vt/vt_core.c:553
> > #28 0xffffffff8043e318 in vt_processkey (
> > kbd=3D0xffffffff80cef898 <default_kbd>, vd=3D0xffffffff80a472c8
> <vt_consdev>,
> > c=3D<optimized out>) at /usr/src/sys/dev/vt/vt_core.c:903
> > #29 vt_kbdevent (kbd=3D0xffffffff80cef898 <default_kbd>, event=3D<optim=
ized
> out>,
> > arg=3D0xffffffff80a472c8 <vt_consdev>) at
> /usr/src/sys/dev/vt/vt_core.c:101
> > 8
> > #30 0xffffffff8078ffcf in atkbd_intr (kbd=3D0xffffffff80cef898
> <default_kbd>,
> > arg=3D<optimized out>) at /usr/src/sys/dev/atkbdc/atkbd.c:565
> > #31 0xffffffff804b1376 in intr_event_execute_handlers
> (ie=3D0xfffff800010ece00,
> > p=3D<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
> > #32 ithread_execute_handlers (ie=3D0xfffff800010ece00, p=3D<optimized o=
ut>)
> > at /usr/src/sys/kern/kern_intr.c:1218
> > #33 ithread_loop (arg=3Darg@entry=3D0xfffff80001c5aea0)
> > at /usr/src/sys/kern/kern_intr.c:1306
> > #34 0xffffffff804adae2 in fork_exit (
> > callout=3D0xffffffff804b1120 <ithread_loop>, arg=3D0xfffff80001c5ae=
a0,
> > frame=3D0xfffffe00ce259f40) at /usr/src/sys/kern/kern_fork.c:1160
> > #35 <signal handler called>
> > #36 0x0000000000000b88 in ?? ()
> > Backtrace stopped: Cannot access memory at address 0xbc7
> > (kgdb)
> >
> >
> >
> >
> >
> >
>
> Can you submit a PR for this? GFP_KERNEL is an alias for M_WAITOK, which
> is
> verboten when intel_atomic_state_alloc() makes its call to kzalloc(), an
> alias for kmalloc().
>
>
>
Done, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274954
Thank you
> --
> Cheers,
> Cy Schubert <Cy.Schubert@cschubert.com>
> FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org
> NTP: <cy@nwtime.org> Web: https://nwtime.org
>
> e^(i*pi)+1=3D0
>
>
>
>
>
--0000000000005ce56f060991f3ed
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Tue, Nov 7, 2023 at 6:33=E2=80=AFA=
M Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com">Cy.Schubert@=
cschubert.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddi=
ng-left:1ex">In message <2806934.Tv9pZ1cZgq@moonset.home>, "Oleg=
V. Nauman" writes:<br>
>=C2=A0 I am observing kernel panic when entering multiuser mode after s=
ucessful <br>
> system boot. It happens when I load CURRENT kernel with INVARIANTS and=
drm <br>
> module loaded ( drm-515-kmod-5.15.118_1 in particular ) . drm module a=
nd kene<br>
> l <br>
> are in sync<br>
><br>
><br>
> FreeBSD moonset.home 15.0-CURRENT FreeBSD 15.0-CURRENT #3 main-n266267=
-<br>
> e116e040f309: Sun Nov=C2=A0 5 10:00:51 EET 2023=C2=A0 =C2=A0 =C2=A0roo=
t@moonset.home:/usr/obj/usr<br>
> /<br>
> src/amd64.amd64/sys/moonset=C2=A0 amd64<br>
><br>
> panic: sleepq_add: td 0xfffff8000203c000 to sleep on wchan 0xfffff8000=
21d8648<br>
>=C2=A0 <br>
> with sleeping prohibited<br>
> ...............<br>
> Reading symbols from /boot/kernel/kernel...<br>
> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...<br>
><br>
> Unread portion of the kernel message buffer:<br>
> panic: malloc(M_WAITOK) with sleeping prohibited<br>
> cpuid =3D 0<br>
> time =3D 1699171621<br>
> .............<br>
> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57<br>
> 57=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 __asm("movq %%=
gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(struct <b=
r>
> pcpu,<br>
> (kgdb) #0=C2=A0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.=
h:57<br>
> #1=C2=A0 doadump (textdump=3D<optimized out>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_shutdown.c:405<br>
> #2=C2=A0 0xffffffff81e2ec53 in vt_kms_postswitch () from /boot/modules=
/drm.ko<br>
> #3=C2=A0 0xffffffff8043ad6e in vt_window_switch (vw=3D0xfffff800021d86=
40)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/dev/vt/vt_core.c:595<br>
> #4=C2=A0 0xffffffff804ec583 in kern_reboot (howto=3D4)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_shutdown.c:501<br>
> #5=C2=A0 0xffffffff804eccfa in vpanic (<br>
>=C2=A0 =C2=A0 =C2=A0fmt=3D0xffffffff808263fb "malloc(M_WAITOK) wit=
h sleeping prohibited",<br>
>=C2=A0 =C2=A0 =C2=A0ap=3Dap@entry=3D0xfffffe00ce259850) at /usr/src/sys=
/kern/kern_shutdown.c:970<br>
> #6=C2=A0 0xffffffff804ecb03 in panic (fmt=3D<unavailable>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_shutdown.c:894<br>
> #7=C2=A0 0xffffffff804c8a04 in malloc_dbg (vap=3D<optimized out>=
,<br>
>=C2=A0 =C2=A0 =C2=A0sizep=3D<optimized out>, mtp=3D<optimized =
out>, flags=3D<optimized out>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_malloc.c:540<br>
> #8=C2=A0 0xffffffff804c885c in malloc (size=3D<optimized out>,<b=
r>
>=C2=A0 =C2=A0 =C2=A0mtp=3D0xffffffff81e931c0 <M_KMALLOC>, flags=
=3D<unavailable>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_malloc.c:641<br>
> #9=C2=A0 0xffffffff81c5a750 in intel_atomic_state_alloc ()<br>
>=C2=A0 =C2=A0 from /boot/modules/i915kms.ko<br>
> #10 0xffffffff81dfe404 in drm_client_modeset_commit_atomic ()<br>
>=C2=A0 =C2=A0 from /boot/modules/drm.ko<br>
> #11 0xffffffff81dfe614 in drm_client_modeset_commit_locked ()<br>
>=C2=A0 =C2=A0 from /boot/modules/drm.ko<br>
> #12 0xffffffff81dfe7a1 in drm_client_modeset_commit ()<br>
>=C2=A0 =C2=A0 from /boot/modules/drm.ko<br>
> #13 0xffffffff81e41ab3 in drm_fb_helper_restore_fbdev_mode_unlocked ()=
<br>
>=C2=A0 =C2=A0 from /boot/modules/drm.ko<br>
> #14 0xffffffff81e2ed91 in vt_kms_postswitch () from /boot/modules/drm.=
ko<br>
> #15 0xffffffff8043ac31 in vt_window_switch (vw=3D0xfffff80001d3a600,<b=
r>
>=C2=A0 =C2=A0 =C2=A0vw@entry=3D0xffffffff80a47178 <vt_conswindow>=
)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/dev/vt/vt_core.c:612<br>
> #16 0xffffffff8043be0f in vtterm_cngrab (tm=3D<unavailable>,<br>
>=C2=A0 =C2=A0 =C2=A0tm@entry=3D<error reading variable: value is not=
available>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/dev/vt/vt_core.c:1863<br>
> #17 0xffffffff804893f6 in cngrab () at /usr/src/sys/kern/kern_cons.c:3=
85<br>
> #18 0xffffffff804ecc79 in vpanic (<br>
>=C2=A0 =C2=A0 =C2=A0fmt=3D0xffffffff808799a9 "%s: td %p to sleep o=
n wchan %p with sleeping <br>
> prohibited", ap=3Dap@entry=3D0xfffffe00ce259c20)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_shutdown.c:942<br>
> #19 0xffffffff804ecb03 in panic (fmt=3D<unavailable>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_shutdown.c:894<br>
> #20 0xffffffff805465b3 in sleepq_add (wchan=3Dwchan@entry=3D0xfffff800=
021d8648,<br>
>=C2=A0 =C2=A0 =C2=A0lock=3Dlock@entry=3D0xfffff80001823340,<br>
>=C2=A0 =C2=A0 =C2=A0wmesg=3Dwmesg@entry=3D0xffffffff808b194e "tq_d=
rain", flags=3D0,<br>
>=C2=A0 =C2=A0 =C2=A0flags@entry=3D<unavailable>, queue=3Dqueue@en=
try=3D0)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/subr_sleepqueue.c:326<br>
> #21 0xffffffff804f8efe in _sleep (ident=3Dident@entry=3D0xfffff800021d=
8648,<br>
>=C2=A0 =C2=A0 =C2=A0lock=3Dlock@entry=3D0xfffff80001823340, priority=3D=
priority@entry=3D0,<br>
>=C2=A0 =C2=A0 =C2=A0wmesg=3D0xffffffff808b194e "tq_drain", sb=
t=3Dsbt@entry=3D0, pr=3Dpr@entry=3D0,<br>
>=C2=A0 =C2=A0 =C2=A0flags=3D256) at /usr/src/sys/kern/kern_synch.c:207<=
br>
> #22 0xffffffff8054cffb in TQ_SLEEP (tq=3D0xfffff80001823300,<br>
>=C2=A0 =C2=A0 =C2=A0p=3D0xfffff800021d8648, wm=3D<optimized out>)=
<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/subr_taskqueue.c:124<br>
> #23 taskqueue_drain (queue=3D0xfffff80001823300, task=3D0xfffff800021d=
8648)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/subr_taskqueue.c:614<br>
> #24 0xffffffff81e2ed35 in vt_kms_postswitch () from /boot/modules/drm.=
ko<br>
> #25 0xffffffff8043ac31 in vt_window_switch (vw=3D0xfffff800021d8648,<b=
r>
>=C2=A0 =C2=A0 =C2=A0vw@entry=3D0xfffff800038fb180) at /usr/src/sys/dev/=
vt/vt_core.c:612<br>
> #26 0xffffffff8043b3b2 in vt_late_window_switch (vw=3D0xfffff800038fb1=
80)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/dev/vt/vt_core.c:468<br>
> #27 vt_proc_window_switch (vw=3D0xfffff800038fb180)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/dev/vt/vt_core.c:553<br>
> #28 0xffffffff8043e318 in vt_processkey (<br>
>=C2=A0 =C2=A0 =C2=A0kbd=3D0xffffffff80cef898 <default_kbd>, vd=3D=
0xffffffff80a472c8 <vt_consdev>,<br>
>=C2=A0 =C2=A0 =C2=A0c=3D<optimized out>) at /usr/src/sys/dev/vt/v=
t_core.c:903<br>
> #29 vt_kbdevent (kbd=3D0xffffffff80cef898 <default_kbd>, event=
=3D<optimized out>,<br>
>=C2=A0 =C2=A0 =C2=A0arg=3D0xffffffff80a472c8 <vt_consdev>) at /us=
r/src/sys/dev/vt/vt_core.c:101<br>
> 8<br>
> #30 0xffffffff8078ffcf in atkbd_intr (kbd=3D0xffffffff80cef898 <def=
ault_kbd>,<br>
>=C2=A0 =C2=A0 =C2=A0arg=3D<optimized out>) at /usr/src/sys/dev/at=
kbdc/atkbd.c:565<br>
> #31 0xffffffff804b1376 in intr_event_execute_handlers (ie=3D0xfffff800=
010ece00,<br>
>=C2=A0 =C2=A0 =C2=A0p=3D<optimized out>) at /usr/src/sys/kern/ker=
n_intr.c:1205<br>
> #32 ithread_execute_handlers (ie=3D0xfffff800010ece00, p=3D<optimiz=
ed out>)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_intr.c:1218<br>
> #33 ithread_loop (arg=3Darg@entry=3D0xfffff80001c5aea0)<br>
>=C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/kern_intr.c:1306<br>
> #34 0xffffffff804adae2 in fork_exit (<br>
>=C2=A0 =C2=A0 =C2=A0callout=3D0xffffffff804b1120 <ithread_loop>, =
arg=3D0xfffff80001c5aea0,<br>
>=C2=A0 =C2=A0 =C2=A0frame=3D0xfffffe00ce259f40) at /usr/src/sys/kern/ke=
rn_fork.c:1160<br>
> #35 <signal handler called><br>
> #36 0x0000000000000b88 in ?? ()<br>
> Backtrace stopped: Cannot access memory at address 0xbc7<br>
> (kgdb)<br>
><br>
><br>
><br>
><br>
><br>
><br>
<br>
Can you submit a PR for this? GFP_KERNEL is an alias for M_WAITOK, which is=
<br>
verboten when intel_atomic_state_alloc() makes its call to kzalloc(), an <b=
r>
alias for kmalloc().<br>
<br><br></blockquote><div><br></div><div>Done,=C2=A0 <a href=3D"https://bug=
s.freebsd.org/bugzilla/show_bug.cgi?id=3D274954">https://bugs.freebsd.org/b=
ugzilla/show_bug.cgi?id=3D274954</a></div><div>=C2=A0</div><div>Thank you</=
div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px =
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
-- <br>
Cheers,<br>
Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com" target=3D"_bla=
nk">Cy.Schubert@cschubert.com</a>><br>
FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 <a href=
=3D"https://FreeBSD.org" rel=3D"noreferrer" target=3D"_blank">https://FreeB=
SD.org</a><br>
NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:cy@nwtim=
e.org" target=3D"_blank">cy@nwtime.org</a>>=C2=A0 =C2=A0 Web:=C2=A0 <a h=
ref=3D"https://nwtime.org" rel=3D"noreferrer" target=3D"_blank">https://nwt=
ime.org</a><br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 e^(i*pi)+1=3D0<br>
<br>
<br>
<br>
<br>
</blockquote></div></div>
--0000000000005ce56f060991f3ed--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAC5YPTsz9zhG6=Rmobt6knamKJXG2DdNmb0MbyymBB=xyzeAzQ>