Date: Wed, 2 Mar 2022 19:26:18 +0200 From: Ze Dupsys <zedupsys@gmail.com> To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <roger.pau@citrix.com> Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: <CAOEWpzfsajhbvXfAw5-F1p83jjmSggobANBEyeYFAfiumAWRCA@mail.gmail.com> In-Reply-To: <Yh93uLIBqk5NC2xf@Air-de-Roger> References: <CAOEWpzc2WVViMJHrrtuU-G_7yck4eehm6b=JQPSZU1MH-bzmiw@mail.gmail.com> <202203011540.221FeR4f028103@nfbcal.org> <CAOEWpzdC41ithfd7R_qa66%2Bsh_UXeku7OcVC_b%2BXUaLr_9SSTA@mail.gmail.com> <Yh93uLIBqk5NC2xf@Air-de-Roger>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000086a47505d93f92ea Content-Type: multipart/alternative; boundary="00000000000086a47305d93f92e8" --00000000000086a47305d93f92e8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Today managed to crash lab Dom0 with: xen_cmdline=3D"dom0_mem=3D6144M dom0_max_vcpus=3D2 dom0=3Dpvh,verbose=3D1 console=3Dvga,com1 com1=3D9600,8n1 guest_loglvl=3Dall loglvl=3Dall sync_con= sole=3D1 reboot=3Dno" I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as in attachment, panic was with the same fingerprint as the one with "rman_is_region_manager" line already reported. The scripts i ran in parallel generally were the same as attached in bug report, just a bit modified. 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and instead of 2GB, it writes 4BG in each ZVOL created dd if=3D/dev/zero) 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write 3GB in it's /tmp, restart DomU, removes /tmp, repeat) 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 disks attached, turn off DomU, repeat) 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. Around dom id 108, system started to behave suspiciously, xl list showed DomUs created, but they did not really start up, script timeout-ed for ssh connection, no vnc. When i did xl destroy manually, and xl create, system panic happened. I have log files for all serial output, if there is anything useful, i can provide. On disk log files seems to loose latest messages due to crash. On Wed, Mar 2, 2022 at 3:57 PM Roger Pau Monn=C3=A9 <roger.pau@citrix.com> = wrote: > On Wed, Mar 02, 2022 at 10:57:37AM +0200, Ze Dupsys wrote: > > Hello, > > > > I started using XEN on one pre-production machine (with aim to use late= r > in > > production) with 12.2, but since it experienced random crashes i update= d > to > > 13.0 in hope that errors might disappear. > > > > I do not know how detailed should i write, so that this email is not to= o > > long, but gives enough info. > > > > FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rule= s > > for NATting are used. Zpool is composed of 2 mirrored disks. There is a > > ZVOL volmode=3Ddev for each VM and VM's jail that are attached as raw > devices > > to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, > with > > VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have > > bridge interfaces, where DomU's are connected depending on their > > "zone/network", those that have allowed outgoing connections are NATted > by > > IPFW on specific physical NIC and IP. > > So from the traces on the ticket: > > panic: pmap_growkernel: no memory to grow kernel > cpuid =3D 0 > time =3D 1646123072 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff81073eed at pmap_growkernel+0x27d > #4 0xffffffff80f2dae8 at vm_map_insert+0x248 > #5 0xffffffff80f30249 at vm_map_find+0x549 > #6 0xffffffff80f2bf76 at kmem_init+0x226 > #7 0xffffffff80c73341 at vmem_xalloc+0xcb1 > #8 0xffffffff80c72c3b at vmem_xalloc+0x5ab > #9 0xffffffff80f2bfce at kmem_init+0x27e > #10 0xffffffff80c73341 at vmem_xalloc+0xcb1 > #11 0xffffffff80c72c3b at vmem_xalloc+0x5ab > #12 0xffffffff80c72646 at vmem_alloc+0x46 > #13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96 > #14 0xffffffff80f21a2a at uma_prealloc+0x23a > #15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e > #16 0xffffffff80f1f6af at uma_set_align+0x8f > #17 0xffffffff82463362 at abd_borrow_buf_copy+0x22 > Uptime: 4m9s > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > fault virtual address =3D 0x22710028 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80c45892 > stack pointer =3D 0x28:0xfffffe0096600930 > frame pointer =3D 0x28:0xfffffe0096600930 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 1496 (devmatch) > trap number =3D 12 > panic: page fault > cpuid =3D 0 > time =3D 1646123791 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff80c62011 at rman_is_region_manager+0x241 > #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #9 0xffffffff80c1949c at kernel_sysctl+0x43c > #10 0xffffffff80c19b13 at userland_sysctl+0x173 > #11 0xffffffff80c1995f at sys___sysctl+0x5f > #12 0xffffffff8108baac at amd64_syscall+0x10c > #13 0xffffffff8106243e at Xfast_syscall+0xfe > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 1; apic id =3D 02 > fault virtual address =3D 0x68 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff824a599d > stack pointer =3D 0x28:0xfffffe00b1e27910 > frame pointer =3D 0x28:0xfffffe00b1e279b0 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 0 (xbbd7 taskq) > trap number =3D 12 > panic: page fault > cpuid =3D 1 > time =3D 1646122723 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff8248935a at dmu_read+0x2a > #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa > #9 0xffffffff80a7f214 at xbd_instance_create+0xa394 > #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a > #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 > #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac > #13 0xffffffff80bc7e3e at fork_exit+0x7e > #14 0xffffffff81062b9e at fork_trampoline+0xe > Uptime: 1h44m10s > > This all look to me like Out of Memory conditions, can you check with > `top` whats going on with your memory? > > Might also be helpful to record periodic calls to `vmstat -m | sort -k > 2 -r` to try to figure out what's using so much memory. > > Regards, Roger. > --00000000000086a47305d93f92e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Today managed to crash lab Dom0 with:</div><div>xen_c= mdline=3D"dom0_mem=3D6144M dom0_max_vcpus=3D2 dom0=3Dpvh,verbose=3D1 c= onsole=3Dvga,com1 com1=3D9600,8n1 guest_loglvl=3Dall loglvl=3Dall sync_cons= ole=3D1 reboot=3Dno"</div><div><br></div><div>I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as in at= tachment, panic was with the same fingerprint as the one with "rman_is= _region_manager" line already reported.<br></div><div><br></div><div>T= he scripts i ran in parallel generally were the same as attached in bug rep= ort, just a bit modified.</div><div>1) ./libexec.sh zfs_volstress_fast_4g (= this just creates new ZVOLs and instead of 2GB, it writes 4BG in each ZVOL = created dd if=3D/dev/zero) <br></div><div>2)=C2=A0 ./test_vm1_zvol_3gb.sh (= this loops commands: start first DomU, write 3GB in it's /tmp, restart = DomU, removes /tmp, repeat)</div><div>3) ./test_vm2_zvol_5_on_off.sh (this = loops: start second DomU, which has 5 disks attached, turn off DomU, repeat= )<br></div><div>4) monitoring, sleep 120 seconds, print vmstat | sort in se= rial output.<br></div><div><br></div><div>Around dom id 108, system started= to behave suspiciously, xl list showed DomUs created, but they did not rea= lly start up, script timeout-ed for ssh connection, no vnc. When i did xl d= estroy manually, and xl create, system panic happened.</div><div><br></div>= <div>I have log files for all serial output, if there is anything useful, i= can provide. On disk log files seems to loose latest messages due to crash= .<br></div><br></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class= =3D"gmail_attr">On Wed, Mar 2, 2022 at 3:57 PM Roger Pau Monn=C3=A9 <<a = href=3D"mailto:roger.pau@citrix.com">roger.pau@citrix.com</a>> wrote:<br= ></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;= border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, Mar 02, 20= 22 at 10:57:37AM +0200, Ze Dupsys wrote:<br> > Hello,<br> > <br> > I started using XEN on one pre-production machine (with aim to use lat= er in<br> > production) with 12.2, but since it experienced random crashes i updat= ed to<br> > 13.0 in hope that errors might disappear.<br> > <br> > I do not know how detailed should i write, so that this email is not t= oo<br> > long, but gives enough info.<br> > <br> > FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rul= es<br> > for NATting are used. Zpool is composed of 2 mirrored disks. There is = a<br> > ZVOL volmode=3Ddev for each VM and VM's jail that are attached as = raw devices<br> > to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, = with<br> > VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i = have<br> > bridge interfaces, where DomU's are connected depending on their<b= r> > "zone/network", those that have allowed outgoing connections= are NATted by<br> > IPFW on specific physical NIC and IP.<br> <br> So from the traces on the ticket:<br> <br> panic: pmap_growkernel: no memory to grow kernel<br> cpuid =3D 0<br> time =3D 1646123072<br> KDB: stack backtrace:<br> #0 0xffffffff80c57525 at kdb_backtrace+0x65<br> #1 0xffffffff80c09f01 at vpanic+0x181<br> #2 0xffffffff80c09d73 at panic+0x43<br> #3 0xffffffff81073eed at pmap_growkernel+0x27d<br> #4 0xffffffff80f2dae8 at vm_map_insert+0x248<br> #5 0xffffffff80f30249 at vm_map_find+0x549<br> #6 0xffffffff80f2bf76 at kmem_init+0x226<br> #7 0xffffffff80c73341 at vmem_xalloc+0xcb1<br> #8 0xffffffff80c72c3b at vmem_xalloc+0x5ab<br> #9 0xffffffff80f2bfce at kmem_init+0x27e<br> #10 0xffffffff80c73341 at vmem_xalloc+0xcb1<br> #11 0xffffffff80c72c3b at vmem_xalloc+0x5ab<br> #12 0xffffffff80c72646 at vmem_alloc+0x46<br> #13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96<br> #14 0xffffffff80f21a2a at uma_prealloc+0x23a<br> #15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e<br> #16 0xffffffff80f1f6af at uma_set_align+0x8f<br> #17 0xffffffff82463362 at abd_borrow_buf_copy+0x22<br> Uptime: 4m9s<br> <br> <br> Fatal trap 12: page fault while in kernel mode<br> cpuid =3D 0; apic id =3D 00<br> fault virtual address=C2=A0 =C2=A0=3D 0x22710028<br> fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor r= ead data, page not present<br> instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff80c45892<br> stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe0096= 600930<br> frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe0096= 600930<br> code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base rx0, limit 0= xfffff, type 0x1b<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1<br> processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0<br> current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 1496 (devmatch)<br> trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12<br> panic: page fault<br> cpuid =3D 0<br> time =3D 1646123791<br> KDB: stack backtrace:<br> #0 0xffffffff80c57525 at kdb_backtrace+0x65<br> #1 0xffffffff80c09f01 at vpanic+0x181<br> #2 0xffffffff80c09d73 at panic+0x43<br> #3 0xffffffff8108b1a7 at trap+0xbc7<br> #4 0xffffffff8108b1ff at trap+0xc1f<br> #5 0xffffffff8108a85d at trap+0x27d<br> #6 0xffffffff81061b18 at calltrap+0x8<br> #7 0xffffffff80c62011 at rman_is_region_manager+0x241<br> #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101<br> #9 0xffffffff80c1949c at kernel_sysctl+0x43c<br> #10 0xffffffff80c19b13 at userland_sysctl+0x173<br> #11 0xffffffff80c1995f at sys___sysctl+0x5f<br> #12 0xffffffff8108baac at amd64_syscall+0x10c<br> #13 0xffffffff8106243e at Xfast_syscall+0xfe<br> <br> <br> Fatal trap 12: page fault while in kernel mode<br> cpuid =3D 1; apic id =3D 02<br> fault virtual address=C2=A0 =C2=A0=3D 0x68<br> fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor r= ead data, page not present<br> instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff824a599d<br> stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00b1= e27910<br> frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00b1= e279b0<br> code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base rx0, limit 0= xfffff, type 0x1b<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1<br> processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0<br> current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0 (xbbd7 taskq)<br> trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12<br> panic: page fault<br> cpuid =3D 1<br> time =3D 1646122723<br> KDB: stack backtrace:<br> #0 0xffffffff80c57525 at kdb_backtrace+0x65<br> #1 0xffffffff80c09f01 at vpanic+0x181<br> #2 0xffffffff80c09d73 at panic+0x43<br> #3 0xffffffff8108b1a7 at trap+0xbc7<br> #4 0xffffffff8108b1ff at trap+0xc1f<br> #5 0xffffffff8108a85d at trap+0x27d<br> #6 0xffffffff81061b18 at calltrap+0x8<br> #7 0xffffffff8248935a at dmu_read+0x2a<br> #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa<br> #9 0xffffffff80a7f214 at xbd_instance_create+0xa394<br> #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a<br> #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1<br> #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac<br> #13 0xffffffff80bc7e3e at fork_exit+0x7e<br> #14 0xffffffff81062b9e at fork_trampoline+0xe<br> Uptime: 1h44m10s<br> <br> This all look to me like Out of Memory conditions, can you check with<br> `top` whats going on with your memory?<br> <br> Might also be helpful to record periodic calls to `vmstat -m | sort -k<br> 2 -r` to try to figure out what's using so much memory.<br> <br> Regards, Roger.<br> </blockquote></div> --00000000000086a47305d93f92e8-- --00000000000086a47505d93f92ea Content-Type: application/octet-stream; name="vmstat_before_crash.log" Content-Disposition: attachment; filename="vmstat_before_crash.log" Content-Transfer-Encoding: base64 Content-ID: <f_l09tdsh80> X-Attachment-Id: f_l09tdsh80 Li9iaW4vbGliZXhlYy5zaDogPT09IFZNU1RBVCA9PT0KIE5GU0Qgc2Vzc2lvbiAgICAgMSAgICAg MUsgICAgICAgIDEgIDEwMjQKICAgIENBTSBxdWV1ZSAgICAxNiAgICAgNUsgICAgICAgNTcgIDE2 LDMyLDUxMgogICBDQU0gcGVyaXBoICAgICA2ICAgICAySyAgICAgICAyNCAgMTYsMzIsNjQsMTI4 LDI1NgogICAgIENBTSBwYXRoICAgIDExICAgICAxSyAgICAgICA4MiAgMzIKIE5GU0QgbGNrZmls ZSAgICAgMSAgICAgMUsgICAgICAgIDEgIDI1NgpDQU0gZGV2IHF1ZXVlICAgICA3ICAgICAxSyAg ICAgICAgNyAgNjQKICAgICAgQ0FNIFhQVCAgICAzMyAgICAgM0sgICAgICAgNzQgIDE2LDMyLDEy OCw1MTIsMTAyNCwyMDQ4LDY1NTM2Ck5GU0QgVjRjbGllbnQgICAgIDEgICAgIDFLICAgICAgICAx ICAyNTYKICAgICAgQ0FNIFNJTSAgICAgNyAgICAgMksgICAgICAgIDcgIDI1NgogICAgICAgICBU eXBlIEluVXNlIE1lbVVzZSBSZXF1ZXN0cyAgU2l6ZShzKQpDQU0gSS9PIFNjaGVkdWxlciAgICAg MiAgICAgMUsgICAgICAgIDIgIDEyOAogICAgICBDQU0gREVWICAgICA5ICAgIDE4SyAgICAgICAy MiAgMjA0OAogICAgICBDQU0gQ0NCICAgIDEwICAgIDIwSyAgODM3MTk1NiAgMjA0OAogICAgICBz b2xhcmlzIDU2OTMxNCAxMTQwNjVLIDQ5ODUyMzQwMCAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEy LDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYzODQsMzI3NjgsNjU1MzYKICAgIHN5c2N0bG9pZCAzMjQz MiAgMTgzM0sgICAgNTM3NTUgIDE2LDMyLDY0LDEyOAogICAgICAgZGV2YnVmIDE3OTYxIDM1MTI3 SyAgICAxODMwMCAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIs NjU1MzYKICAgICAgICAgeGJiZCAxMTk2OCAzODQ3MDZLICAgMjExMDc0ICA2NCwyMDQ4LDY1NTM2 CiAgICAgICBhY3BpY2EgIDk3NjggIDEwMjVLICAgMjU1MTA0ICAxNiwzMiw2NCwxMjgsMjU2LDM4 NCw1MTIsMTAyNCwyMDQ4CiAgICAgICAgICBidXMgIDEyOTYgICAxMTZLICAgIDI4NTMxICAxNiwz Miw2NCwxMjgsMjU2LDM4NCw0MDk2CiAgICAgICAgIHVtdHggIDExNTIgICAxNDRLICAgICAxMTUy ICAxMjgKICAgICAga2R0cmFjZSAgIDYyNSAgIDE0M0sgICAgMTUxMDQgIDY0LDI1NgogICAgICAg bW9kdWxlICAgNTA4ICAgIDY0SyAgICAgIDUwOCAgMTI4CiAgICAgICB4ZW5idXMgICAzOTcgICAg NjFLICAgIDI0ODU0ICAxNiwzMiw2NCwyNTYsNDA5NgogICAgICAgICAgb3NkICAgMzYzICAgICA3 SyAgICAgMTAwMyAgMTYsMzIsNjQsMTI4LDI1NgogICAgICAgbGlua2VyICAgMzU2ICAyMTE3SyAg ICAgMTQxMiAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYz ODQKICAgICAgICAga29iaiAgIDMzNiAgMTM0NEsgICAgICA2NzMgIDQwOTYKICAgICAgICAgcm1h biAgIDI0NCAgICAyOEsgICAgIDMwMTkgIDE2LDMyLDEyOAphdWRpdF9ldmNsYXNzICAgMjM2ICAg ICA4SyAgICAgIDI5NCAgMzIKICAgIHRhc2txdWV1ZSAgIDIxNiAgICAyNUsgICAgICA2NjkgIDE2 LDMyLDY0LDEyOCwyNTYKICAgICAgICAgIFVNQSAgIDIxNSAgICAgNUsgICAgICAyMTUgIDE2LDMy CiAgICAgICBERVZGUzMgICAyMDQgICAgNTFLICAgICAgNTI4ICAyNTYKICAgICAgc3VicHJvYyAg IDE2NSAgIDMyMEsgICAgMTM0MzUgIDM4NCw0MDk2CiAgICAgICBERVZGUzEgICAxNjEgICAxNjFL ICAgICAgNDkwICAxMDI0CiAgICAgIGl0aHJlYWQgICAxNTQgICAgMjZLICAgICAgNTg5ICAzMiwx MjgsMjU2CiAgICAgICBpZmFkZHIgICAxMzAgICAgMThLICAgICAgNTM2ICAxNiwzMiw2NCwxMjgs MjU2LDM4NCwyMDQ4LDQwOTYKIGV2ZW50aGFuZGxlciAgIDEzMCAgICAxMUsgICAgICAxMzAgIDY0 LDEyOAogICAgICAgICBHRU9NICAgMTI1ICAgIDIwSyAgICAgMTI5MCAgMTYsMzIsNjQsMTI4LDI1 Niw1MTIsMTAyNCwyMDQ4LDQwOTYsODE5MiwxNjM4NAogICAgICAgICBrZW52ICAgMTE2ICAgIDEz SyAgICAgIDExNiAgMTYsMzIsNjQsMTI4LDI1Niw4MTkyCiAgICAgICBidXMtc2MgICAxMTAgICAz NjVLICAgICA2MzA3ICAxNiwzMiw2NCwxMjgsMjU2LDM4NCw1MTIsMTAyNCwyMDQ4LDQwOTYsODE5 MiwxNjM4NCwzMjc2OCw2NTUzNgogICAgICAgS1RSQUNFICAgMTAwICAgIDEzSyAgICAgIDEwMCAg MTI4CiAgICAgICAgc2VsZmQgICAgOTUgICAgIDZLIDMyNTIxOTcwOCAgNjQKICAgICAgcHdkZGVz YyAgICA3MSAgICAgNUsgICAgMTMzNDEgIDY0CiAgICAgICBrcXVldWUgICAgNzEgICAgIDVLICAg IDEzMzQxICA2NAogICBERVZGU19SVUxFICAgIDU5ICAgIDIxSyAgICAgICA1OSAgNjQsMzg0CiAg ICBwcm9jLWFyZ3MgICAgNTcgICAgIDNLICAgICA5NjU2ICAxNiwzMiw2NCwxMjgsMjU2CiAgcHJp dmNtZF9kZXYgICAgNTcgICAgIDJLICA4NTk4NTQ4ICAxNiwzMiw2NCwyNTYsNTEyLDEwMjQsMjA0 OCw4MTkyLDE2Mzg0CiAgIGtzdGF0X2RhdGEgICAgNTYgICAgNTZLICAgICAgMjM5ICAxMDI0CnZt X2ZpY3RpdGlvdXMgICAgNTIgIDc0NTlLICAgICA1MjYwICA2NCwxMjgsMjU2LDEwMjQsODE5Miwz Mjc2OAogICAgICAgICAgVVNCICAgIDUxICAgIDgySyAgICAgICA1NyAgMTYsMzIsNjQsMTI4LDI1 NiwzODQsMTAyNCwyMDQ4LDQwOTYsODE5MgogICAgICAgVVNCZGV2ICAgIDQ5ICAgIDE1SyAgICAg ICA1MSAgMzIsNjQsMTI4LDI1NiwzODQsNDA5NgogICAgICAgIG1vdW50ICAgIDQ2ICAgICAySyAg ICAgIDEzNyAgMTYsMzIsNjQsMTI4LDI1NiwxMDI0CiAgICAgICAgaWZsaWIgICAgNDUgICAzMTZL ICAgICAgIDUzICAxNiw2NCwxMjgsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYzODQsMzI3NjgK ICAgICAgIERFVkZTUCAgICA0MSAgICAgM0sgICAgIDMyMDIgIDY0CiAgICAgeGVuX2ludHIgICAg MzcgICAgIDlLICAgICAgIDM3ICAxMjgsNDA5NgogICAgICBsbHRhYmxlICAgIDMxICAgICA5SyAg ICAgIDI3OCAgMjU2LDM4NAogICAgICBzZXNzaW9uICAgIDMxICAgICA0SyAgICAgIDU3NyAgMTI4 CiAgICAgIGFjcGlzZW0gICAgMzEgICAgIDRLICAgICAgIDMxICAxMjgKICAgICAgICAgdGVtcCAg ICAzMCAgICAxN0sgICAgMzI4OTkgIDE2LDMyLDY0LDEyOCwyNTYsMzg0LDUxMiwxMDI0LDIwNDgs ODE5Miw2NTUzNgogICAgICAgIGxvY2tmICAgIDMwICAgICA0SyAgMzUxNTkyMyAgNjQsMTI4CiAg ICAgICBVbml0bm8gICAgMzAgICAgIDJLICAgICAgODkwICAzMiw2NAogICAgICBhY3BpZGV2ICAg IDMwICAgICAySyAgICAgICAzMCAgNjQKICB1ZnNfZGlyaGFzaCAgICAyNSAgICAxM0sgICAgICAz NTkgIDE2LDMyLDY0LDEyOCw1MTIKICAgICAgICB2dGJ1ZiAgICAyNCAgMTk2OEsgICAgICAgNDYg IDQwOTYKICAgICAgICAgIHR0eSAgICAyNCAgICAyNEsgICAgICAxNDEgIDEwMjQKICAgICAgICAg Y3JlZCAgICAyMyAgICAgNksgICAgIDIxOTggIDI1NgogICAgIHhlbnN0b3JlICAgIDIyICAgICA1 SyAgIDI1NDk3NyAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDQwOTYsMTYzODQKICBldGhlcl9t dWx0aSAgICAyMiAgICAgMksgICAgICAgMjIgIDE2LDMyLDY0LDEyOAogICBldnRjaG5fZGV2ICAg IDIwICAgIDI2SyAyODEwNjA1MSAgNjQsMjU2LDQwOTYKICAgIHBmc19ub2RlcyAgICAyMCAgICAg OEsgICAgICAgMjAgIDM4NAogICAgICAgc2VsZWN0ICAgIDIwICAgICAzSyAgICAgMTkyMiAgMTI4 CiAgIGd0YXNrcXVldWUgICAgMTggICAgMjZLICAgICAgIDE4ICAxNiwzMiwyNTYsODE5MgogICAg ICAgc29uYW1lICAgIDE4ICAgICAySyAgICAyNTA5OCAgMTYsMzIsMTI4CiAgICAgICAgICBwY2Ig ICAgMTcgIDE1NDVLICAgICAgNzEwICAxNiwzMiw2NCwxMDI0LDIwNDgKICAgICAgIHBsaW1pdCAg ICAxNyAgICAgNUsgICAgIDExMjkgIDI1NgogICAgIHJvdXRldGJsICAgIDE2ICAgICA0SyAgICAg ICAxNiAgMzIsNjQsMTI4LDM4NCwxMDI0CiAgICAgcGNpX2xpbmsgICAgMTYgICAgIDJLICAgICAg IDE2ICA2NCwxMjgKICAgICAgIGdudGRldiAgICAxNiAgICAgMUsgICAgICA5NTEgIDMyLDY0CiAg ICAgICAgREVWRlMgICAgMTYgICAgIDFLICAgICAgIDE3ICAxNiwxMjgKICAgIGluNl9tdWx0aSAg ICAxNSAgICAgMksgICAgICAgMTUgIDMyLDI1NgogICAgICAgICAgIHZ0ICAgIDEzICAgIDQxSyAg ICAgICAxMyAgMzg0LDQwOTYsMzI3NjgKICAgICAgICBoaG9vayAgICAxMyAgICAgNEsgICAgICAg MTMgIDI1NgogICAgICAgICAgcHRzICAgIDExICAgICAzSyAgICAgIDEyOCAgMjU2CiAgICAgdGVy bWluYWwgICAgMTEgICAgIDNLICAgICAgIDExICAyNTYKICAgICAgZGV2c3RhdCAgICAxMCAgICAy MUsgICAgICAgMTAgIDMyLDQwOTYKICAgIHRvcG9ub2RlcyAgICAxMCAgICAgMksgICAgICAgMTAg IDEyOAogICAgICAgIGNsb25lICAgICA5ICAgICAySyAgICAgICAgOSAgMTI4CiAgICAgICBrYmRt dXggICAgIDggICAgMjJLICAgICAgICA4ICAxNiwzODQsMTAyNCwyMDQ4LDE2Mzg0CiAgaW5wY2Jw b2xpY3kgICAgIDggICAgIDFLICAgICAxODAwICAzMgplbmNhcF9leHBvcnRfaG9zdCAgICAgOCAg ICAgMUsgICAgICAgIDggIDMyLDY0CiAgICAgICBwcmlzb24gICAgIDggICAgIDFLICAgICAgICA4 ICAxNiwzMgogICAgICAgaXNhZGV2ICAgICA4ICAgICAxSyAgICAgICAgOCAgMTI4CiAgICAgICAg aWZuZXQgICAgIDcgICAgMTNLICAgICAgIDY1ICAxMjgsMjA0OAogICAgICAgIGV2ZGV2ICAgICA3 ICAgICA3SyAgICAgICAgNyAgMTAyNAogICAgIGZpbGVjYXBzICAgICA3ICAgICAxSyAgICAgNTI5 NCAgMTYsMzIsNjQKICAgICAgIGlwNm5kcCAgICAgNyAgICAgMUsgICAgICAgNjUgIDY0LDI1Ngog ICAgICAgZmVlZGVyICAgICA3ICAgICAxSyAgICAgICAgNyAgMzIKICAgICBuZXh1c2RldiAgICAg NyAgICAgMUsgICAgICAgIDcgIDE2CiAgICAgICAgICBMUk8gICAgIDYgICAxMjBLICAgICAgICA2 ICA4MTkyLDMyNzY4CiAgICAgICAgIGlnbXAgICAgIDYgICAgIDFLICAgICAgIDY0ICAxMjgKICAg ICAgICAgIG1sZCAgICAgNiAgICAgMUsgICAgICAgNjQgIDEyOAogICAgICAgICAgQlBGICAgICA2 ICAgICAxSyAgICAgICA2NCAgMTI4CiAgICAgICAgbmhvcHMgICAgIDYgICAgIDFLICAgICAgICA2 ICAxMjgKICAgICBmaWxlZGVzYyAgICAgNSAgICAyMEsgICAgICA1NzkgIDQwOTYKICAgIHVmc19t b3VudCAgICAgNSAgICAxN0sgICAgICAgIDYgIDMyLDY0LDUxMiw0MDk2LDgxOTIKICAgICAgZXZl bnRmZCAgICAgNSAgICAgMUsgICAgICAzMDAgIDEyOAogICAgICAgIGxpbnV4ICAgICA1ICAgICAx SyAgICAgICAgNiAgMTYsNjQKICAgICAgICAgIE1DQSAgICAgNSAgICAgMUsgICAgICAgIDUgIDE2 LDMyLDEyOAogICAgICAgICBwZmlsICAgICA1ICAgICAxSyAgICAgICAgNSAgMTI4CiAgICAgICAg ICBtc2kgICAgIDUgICAgIDFLICAgICAgICA1ICAxMjgKICAgICAgICAgaW50ciAgICAgNCAgIDQ3 MksgICAgICAgIDQgIDY1NTM2CiAgICAgICAgICBzZW0gICAgIDQgICAxMDZLICAgICAgICA0ICAy MDQ4LDQwOTYKICAgICAgICAgIG1zZyAgICAgNCAgICAzMEsgICAgICAgIDQgIDIwNDgsNDA5Niw4 MTkyLDE2Mzg0CiAgICAgdmZzY2FjaGUgICAgIDMgIDIwNDlLICAgICAgICAzICAyNTYsNTEyCiAg ICAgICAgIHZtZW0gICAgIDMgIDEwNjBLICAgICAgIDE1ICA1MTIsMTAyNCwyMDQ4LDQwOTYsMTYz ODQsMzI3NjgKICAgICAgdGlkaGFzaCAgICAgMyAgIDE1OEsgICAgICAgIDMgIDIwNDgsMzI3NjgK ICAgICAgICAgcHJvYyAgICAgMyAgICAzM0sgICAgICAgIDMgIDEwMjQsMTYzODQKICAgICAgICAg c2J1ZiAgICAgMyAgICAgMksgICAyNTkzNjEgIDE2LDMyLDY0LDEyOCwyNTYsNTEyLDEwMjQsNDA5 NiwxNjM4NAogIHZub2RlbWFya2VyICAgICAzICAgICAySyAgICAgIDY3NiAgNTEyCiBsaW51eGN1 cnJlbnQgICAgIDMgICAgIDFLICAgICAgICA0ICA2NCwyNTYKICAgICAgIGNyeXB0byAgICAgMyAg ICAgMUsgICAgICAgIDMgIDY0LDI1NgogICBsb2dpbmNsYXNzICAgICAzICAgICAxSyAgICAgICAg MyAgNjQKICAgICAgICAgIHR1biAgICAgMyAgICAgMUsgICAgICAgIDMgIDMyCiAgICAgaW5fbXVs dGkgICAgIDMgICAgIDFLICAgICAgICAzICAyNTYKICAgICAgY2FsbG91dCAgICAgMiAgIDUxMksg ICAgICAgIDIgIAogICAgIG10eF9wb29sICAgICAyICAgIDcySyAgICAgICAgMiAgODE5Miw2NTUz NgogICAgICAgICAgcnBjICAgICAyICAgICA4SyAgICAgICAgMiAgNDA5NgogICAgICB1aWRpbmZv ICAgICAyICAgICA1SyAgICAgICA1NSAgMTI4LDQwOTYKICAgIGlwc2VjLXNhcSAgICAgMiAgICAg MksgICAgICAgIDIgIDEwMjQKICAgICAgICB0YXNrcSAgICAgMiAgICAgMUsgICAgICAgIDIgIDY0 LDEyOAogICAgc2ZzX25vZGVzICAgICAyICAgICAxSyAgICAgICAgMiAgMzg0CiAgICBhY3BpX3Bl cmYgICAgIDIgICAgIDFLICAgICAgICAyICAzODQKICAgICAgICAgY3B1cyAgICAgMiAgICAgMUsg ICAgICAgIDIgIDE2CiAgICAgIGpibG9ja3MgICAgIDIgICAgIDFLICAgICAgICAyICAxMjgsMjU2 CiAgICAgICBuZXdibGsgICAgIDEgIDIwNDhLICAgICAgMzg4ICAyNTYKICAgICBpbm9kZWRlcCAg ICAgMSAgMTAyNEsgICAgIDI3MjYgIDM4NAogICAgdWZzX3F1b3RhICAgICAxICAxMDI0SyAgICAg ICAgMSAgCiAgICAgdmZzX2hhc2ggICAgIDEgIDEwMjRLICAgICAgICAxICAKICAgICAgcGFnZWRl cCAgICAgMSAgIDI1NksgICAgICAxNjAgIDI1NgogICAgdm5ldF9kYXRhICAgICAxICAgMTY4SyAg ICAgICAgMSAgCiAgIHRmb19jY2FjaGUgICAgIDEgICAxMjhLICAgICAgICAxICAKICAgICBzeW5j YWNoZSAgICAgMSAgICA2OEsgICAgICAgIDEgIAogICAgIGFjcGl0YXNrICAgICAxICAgIDY0SyAg ICAgICAgMSAgNjU1MzYKICAgIGhvc3RjYWNoZSAgICAgMSAgICAzMksgICAgICAgIDEgIDMyNzY4 CiAgICAgICAgICBzaG0gICAgIDEgICAgMzJLICAgICAgICAxICAzMjc2OAogIHBmc192bmNhY2hl ICAgICAxICAgIDE2SyAgICAgICAgMSAgMTYzODQKICAgIGJtc2FmZW1hcCAgICAgMSAgICAgOEsg ICAgIDEwMjcgIDI1Niw4MTkyCiAgICAgICAgc2htZmQgICAgIDEgICAgIDhLICAgICAgICAxICA4 MTkyCiAgIGxvY2FsX2FwaWMgICAgIDEgICAgIDRLICAgICAgICAxICA0MDk2CiAgICAgIG1lbWRl c2MgICAgIDEgICAgIDRLICAgICAgICAxICA0MDk2CiAgICAgIGlvX2FwaWMgICAgIDEgICAgIDJL ICAgICAgICAxICAyMDQ4CiAgICAgICBzeXNjdGwgICAgIDEgICAgIDFLICAgICAxNjkyICAzMiw2 NAogICAgIGZyZWV3b3JrICAgICAxICAgICAxSyAgICAgIDMxOSAgMzIsMTI4CiAgICAgYWNwaWlu dHIgICAgIDEgICAgIDFLICAgICAgICAxICA2NAogICAgICB0Y3BmdW5jICAgICAxICAgICAxSyAg ICAgICAgMSAgNjQKICAgICAgICAgdm5ldCAgICAgMSAgICAgMUsgICAgICAgIDEgIDY0CiAgICAg ICAgICBwbWMgICAgIDEgICAgIDFLICAgICAgICAxICA2NAogICAgICBzb2Z0ZGVwICAgICAxICAg ICAxSyAgICAgICAgMSAgNTEyCmNoYWNoYTIwcmFuZG9tICAgICAxICAgICAxSyAgICAgICAgMSAg Mzg0CnZuZXRfZGF0YV9mcmVlICAgICAxICAgICAxSyAgICAgICAgMSAgMzIKICAgICAgUGVyLWNw dSAgICAgMSAgICAgMUsgICAgICAgIDEgIDMyCiAgICAgICB2bm9kZXMgICAgIDEgICAgIDFLICAg ICAgICAxICAyNTYKICAgICAgICAga3RscyAgICAgMSAgICAgMUsgICAgICAgIDEgIDI1NgogICAg ICAgICBjZGV2ICAgICAxICAgICAxSyAgICAgICAgMSAgMjU2CiAgICAgcDEwMDMuMWIgICAgIDEg ICAgIDFLICAgICAgICAxICAxNgogICAgIHBtY2hvb2tzICAgICAxICAgICAxSyAgICAgICAgMSAg MTI4CiAgICAgICBhcG1kZXYgICAgIDEgICAgIDFLICAgICAgICAxICAxMjgKICBpcHNlY3BvbGlj eSAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQKICBmcHVrZXJuX2N0eCAgICAgMSAgICAgMUsg ICAgICAgIDEgIDEwMjQKICAgICBzZWNhc3ZhciAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQK ICAgICAgIHNhaGVhZCAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQKICAgICAgICAgIGlvdiAg ICAgMCAgICAgMEsgIDg5MTE3ODcgIDE2LDY0LDEyOCwyNTYsMzg0LDUxMgogICAgc3lzY3RsdG1w ICAgICAwICAgICAwSyAgICAxMzQ2NyAgMTYsMzIsNjQsMjU2LDEwMjQKICAgICAganNlZ2RlcCAg ICAgMCAgICAgMEsgICAgIDQ4NjYgIDY0CiAgICAgIGphZGRyZWYgICAgIDAgICAgIDBLICAgICAy MjQwICAxMjgKICAgICAgIGRpcmFkZCAgICAgMCAgICAgMEsgICAgIDIyMjggIDEyOAogICAgICBq cmVtcmVmICAgICAwICAgICAwSyAgICAgMjIyMCAgMTI4CiAgICAgICBkaXJyZW0gICAgIDAgICAg IDBLICAgICAyMjA4ICAxMjgKICAgICAgZW50cm9weSAgICAgMCAgICAgMEsgICAgIDE2OTQgIDE2 LDMyLDY0LDQwOTYKICAgICBmcmVlZmlsZSAgICAgMCAgICAgMEsgICAgICA4MzcgIDY0CiAgICAg cHJvY2Rlc2MgICAgIDAgICAgIDBLICAgICAgNzQ0ICAxMjgKICAgICBpb2N0bG9wcyAgICAgMCAg ICAgMEsgICAgICA0NzggIDI1NiwzODQsMTAyNAogICAgIHNhdmVkaW5vICAgICAwICAgICAwSyAg ICAgIDQwOSAgMjU2CiAgICAgICAgIGpzZWcgICAgIDAgICAgIDBLICAgICAgNDA2ICAxMjgKICAg ICAgam5ld2JsayAgICAgMCAgICAgMEsgICAgICAzODcgIDEyOAogICAgIGZyZWVibGtzICAgICAw ICAgICAwSyAgICAgIDMxOCAgMjU2CiAgICAgICAgc2JkZXAgICAgIDAgICAgIDBLICAgICAgMjI4 ICA2NAogICAgcmFpZF9kYXRhICAgICAwICAgICAwSyAgICAgIDE5MiAgMzIsMTI4LDI1NgogICAg ICAgc3RhdGZzICAgICAwICAgICAwSyAgICAgICA2NSAgNDA5NgogICAgamZyZWVmcmFnICAgICAw ICAgICAwSyAgICAgICAxOSAgMTI4CiAgICAgZnJlZWZyYWcgICAgIDAgICAgIDBLICAgICAgIDE5 ICAxMjgKICAgICAgICBta2RpciAgICAgMCAgICAgMEsgICAgICAgMTIgIDEyOAogICAgIG1idWZf dGFnICAgICAwICAgICAwSyAgICAgICAgNyAgMzIKICAgIG5ld2RpcmJsayAgICAgMCAgICAgMEsg ICAgICAgIDYgIDY0Cm5hbWVpX3RyYWNrZXIgICAgIDAgICAgIDBLICAgICAgICA1ICAzMgogICBj bF9zYXZlYnVmICAgICAwICAgICAwSyAgICAgICAgNCAgNjQKICAgbWFkdF90YWJsZSAgICAgMCAg ICAgMEsgICAgICAgIDIgIDMyLDQwOTYKICAgICAgICBzaWdpbyAgICAgMCAgICAgMEsgICAgICAg IDEgIDY0Ci4vYmluL2xpYmV4ZWMuc2g6ID09PSBWTVNUQVQgPT09Cg== --00000000000086a47505d93f92ea--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOEWpzfsajhbvXfAw5-F1p83jjmSggobANBEyeYFAfiumAWRCA>