Date: Sun, 20 Feb 2022 20:53:09 +0000 From: Michael Jung <mikej@paymentallianceintl.com> To: freebsd-current <freebsd-current@freebsd.org> Subject: New panic in main-n253273-a52d8d4a6c6: Message-ID: <b7ca5b5e7a9a4df3951a2cb2f1d63552@MAIL-HUB.pai.local>
index | next in thread | raw e-mail
[-- Attachment #1 --]
Hi!
The box was quite busy at the time. The only odd thing I am aware of and which I do
not think is related is I have not been able to expand one of my zpool's. ZFS sees my
added draid2:2d:10c:0s vdev but I can't seem to force zpool's expansion - my bet this
is somehow related to the mirrored special device in the pool even though autoexpand
is set to 'on' for the pool.
Not asking anyone to solve this, I plan on putting together a "how to reproduce" and
post to freebsd-fs@ but wanted to note this oddity.
--mikej
This is a unmodified GENERIC kernel.
Unread portion of the kernel message buffer:
panic: refcount 0xfffff801cc119284 wraparound
cpuid = 7
time = 1645382575
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0119a83c80
vpanic() at vpanic+0x17f/frame 0xfffffe0119a83cd0
panic() at panic+0x43/frame 0xfffffe0119a83d30
fdclose() at fdclose/frame 0xfffffe0119a83dc0
closefp_impl() at closefp_impl+0x77/frame 0xfffffe0119a83e00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0119a83f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0119a83f30
--- syscall (6, FreeBSD ELF64, sys_close), rip = 0x3309a531b62a, rsp = 0x3309a2802bc8, rbp = 0x3309a2803000 ---
KDB: enter: panic
__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1 doadump (textdump=textdump@entry=0)
at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xffffffff804c86ba in db_dump (dummy=<optimized out>,
dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)
at /usr/src/sys/ddb/db_command.c:575
#3 0xffffffff804c8572 in db_command (last_cmdp=<optimized out>,
cmd_table=<optimized out>, dopager=dopager@entry=1)
at /usr/src/sys/ddb/db_command.c:482
#4 0xffffffff804c81cd in db_command_loop ()
at /usr/src/sys/ddb/db_command.c:535
#5 0xffffffff804cb806 in db_trap (type=<optimized out>, code=<optimized out>)
at /usr/src/sys/ddb/db_main.c:270
#6 0xffffffff80c566fb in kdb_trap (type=type@entry=3, code=code@entry=0,
tf=tf@entry=0xfffffe0119a83bb0) at /usr/src/sys/kern/subr_kdb.c:733
#7 0xffffffff810e75ca in trap (frame=0xfffffe0119a83bb0)
at /usr/src/sys/amd64/amd64/trap.c:609
#8 <signal handler called>
#9 kdb_enter (why=0xffffffff812e5eaf "panic", msg=<optimized out>)
at /usr/src/sys/kern/subr_kdb.c:506
#10 0xffffffff80c08c50 in vpanic (
fmt=0xffffffff8122606d "refcount %p wraparound",
ap=ap@entry=0xfffffe0119a83d10) at /usr/src/sys/kern/kern_shutdown.c:908
#11 0xffffffff80c089e3 in panic (
fmt=0xffffffff81e8cf40 <cnputs_mtx> "\332#*\201\377\377\377\377")
at /usr/src/sys/kern/kern_shutdown.c:844
#12 0xffffffff80ba9180 in _refcount_update_saturated (count=0xfffff801cc119284)
at /usr/src/sys/sys/refcount.h:55
#13 refcount_releasen (count=0xfffff801cc119284, n=1)
at /usr/src/sys/sys/refcount.h:154
#14 refcount_release (count=0xfffff801cc119284)
at /usr/src/sys/sys/refcount.h:174
#15 closef (fp=<optimized out>, td=0xfffffe00df8d0560)
at /usr/src/sys/kern/kern_descrip.c:2776
#16 0xffffffff80bad587 in closefp_impl (fdp=0xfffffe012b7c0430, fd=4,
fp=0xfffff801cc119280, td=0xfffffe00df8d0560, audit=true)
at /usr/src/sys/kern/kern_descrip.c:1300
#17 0xffffffff810e838e in syscallenter (td=<optimized out>)
at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#18 amd64_syscall (td=0xfffffe00df8d0560, traced=0)
at /usr/src/sys/amd64/amd64/trap.c:1191
#19 <signal handler called>
#20 0x00003309a531b62a in ?? ()
Backtrace stopped: Cannot access memory at address 0x3309a2802bc8
(kgdb)
http://mail.mikej.com/core.txt.0
http://mail.mikej.com/info.0
http://mail.mikej.com/kernel.full.0
http://mail.mikej.com/vmcore.0
CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
[-- Attachment #2 --]
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<style type="text/css">.style1 {font-family: "Times New Roman";}</style></head><body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">Hi!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">The box was quite busy at the time. The only odd thing I am aware of and which I do
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">not think is related is I have not been able to expand one of my zpool’s. ZFS sees my<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">added draid2:2d:10c:0s vdev but I can’t seem to force zpool’s expansion – my bet this<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">is somehow related to the mirrored special device in the pool even though autoexpand<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">is set to ‘on’ for the pool.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">Not asking anyone to solve this, I plan on putting together a “how to reproduce” and
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">post to freebsd-fs@ but wanted to note this oddity.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">--mikej<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">This is a unmodified GENERIC kernel.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">Unread portion of the kernel message buffer:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">panic: refcount 0xfffff801cc119284 wraparound<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">cpuid = 7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">time = 1645382575<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">KDB: stack backtrace:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0119a83c80<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">vpanic() at vpanic+0x17f/frame 0xfffffe0119a83cd0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">panic() at panic+0x43/frame 0xfffffe0119a83d30<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">fdclose() at fdclose/frame 0xfffffe0119a83dc0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">closefp_impl() at closefp_impl+0x77/frame 0xfffffe0119a83e00<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0119a83f30<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0119a83f30<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">--- syscall (6, FreeBSD ELF64, sys_close), rip = 0x3309a531b62a, rsp = 0x3309a2802bc8, rbp = 0x3309a2803000 ---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">KDB: enter: panic<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">(kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#1 doadump (textdump=textdump@entry=0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/kern/kern_shutdown.c:399<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#2 0xffffffff804c86ba in db_dump (dummy=<optimized out>,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/ddb/db_command.c:575<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#3 0xffffffff804c8572 in db_command (last_cmdp=<optimized out>,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> cmd_table=<optimized out>, dopager=dopager@entry=1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/ddb/db_command.c:482<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#4 0xffffffff804c81cd in db_command_loop ()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/ddb/db_command.c:535<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#5 0xffffffff804cb806 in db_trap (type=<optimized out>, code=<optimized out>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/ddb/db_main.c:270<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#6 0xffffffff80c566fb in kdb_trap (type=type@entry=3, code=code@entry=0,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> tf=tf@entry=0xfffffe0119a83bb0) at /usr/src/sys/kern/subr_kdb.c:733<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#7 0xffffffff810e75ca in trap (frame=0xfffffe0119a83bb0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/amd64/amd64/trap.c:609<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#8 <signal handler called><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#9 kdb_enter (why=0xffffffff812e5eaf "panic", msg=<optimized out>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/kern/subr_kdb.c:506<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#10 0xffffffff80c08c50 in vpanic (<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> fmt=0xffffffff8122606d "refcount %p wraparound",
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> ap=ap@entry=0xfffffe0119a83d10) at /usr/src/sys/kern/kern_shutdown.c:908<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#11 0xffffffff80c089e3 in panic (<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> fmt=0xffffffff81e8cf40 <cnputs_mtx> "\332#*\201\377\377\377\377")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/kern/kern_shutdown.c:844<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#12 0xffffffff80ba9180 in _refcount_update_saturated (count=0xfffff801cc119284)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/sys/refcount.h:55<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#13 refcount_releasen (count=0xfffff801cc119284, n=1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/sys/refcount.h:154<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#14 refcount_release (count=0xfffff801cc119284)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/sys/refcount.h:174<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#15 closef (fp=<optimized out>, td=0xfffffe00df8d0560)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/kern/kern_descrip.c:2776<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#16 0xffffffff80bad587 in closefp_impl (fdp=0xfffffe012b7c0430, fd=4,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> fp=0xfffff801cc119280, td=0xfffffe00df8d0560, audit=true)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/kern/kern_descrip.c:1300<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#17 0xffffffff810e838e in syscallenter (td=<optimized out>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#18 amd64_syscall (td=0xfffffe00df8d0560, traced=0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""> at /usr/src/sys/amd64/amd64/trap.c:1191<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#19 <signal handler called><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">#20 0x00003309a531b62a in ?? ()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">Backtrace stopped: Cannot access memory at address 0x3309a2802bc8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New"">(kgdb)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://mail.mikej.com/core.txt.0">http://mail.mikej.com/core.txt.0</a><o:p></o:p></span></p>;
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://mail.mikej.com/info.0">http://mail.mikej.com/info.0</a><o:p></o:p></span></p>;
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://mail.mikej.com/kernel.full.0">http://mail.mikej.com/kernel.full.0</a><o:p></o:p></span></p>;
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><a href="http://mail.mikej.com/vmcore.0">http://mail.mikej.com/vmcore.0</a><o:p></o:p></span></p>;
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> </p>
<div style="FONT-SIZE: 10px; BORDER-TOP: #666666 1px solid; FONT-FAMILY: Verdana; WIDTH: 410px; BORDER-BOTTOM: #666666 1px solid; PADDING-BOTTOM: 5px; PADDING-TOP: 5px; PADDING-LEFT: 5px; PADDING-RIGHT: 5px">
CONFIDENTIALITY NOTE: This message is intended only for the use<br>
of the individual or entity to whom it is addressed and may <br>
contain information that is privileged, confidential, and <br>
exempt from disclosure under applicable law. If the reader <br>
of this message is not the intended recipient, you are hereby <br>
notified that any dissemination, distribution or copying <br>
of this communication is strictly prohibited. If you have <br>
received this transmission in error, please notify us by <br>
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99, <br>
2101 High Wickham Place, Suite 101, Louisville, KY 40245<br>
<br>
<br>
<br>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
<p style="FONT-SIZE: 10px; FONT-FAMILY: Verdana"></p>
</div>
</div>
<br><br><p style="font-family: Verdana; font-size:10pt; color:#666666;"><b>Disclaimer</b></p><p style="font-family: Verdana; font-size:8pt; color:#666666;">The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.<br><br>This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.</p></body></html>
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b7ca5b5e7a9a4df3951a2cb2f1d63552>