From owner-freebsd-current@FreeBSD.ORG Sat Mar 1 02:15:14 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 020FF1065673 for ; Sat, 1 Mar 2008 02:15:14 +0000 (UTC) (envelope-from mikej@paymentallianceintl.com) Received: from mx2.confluenttech.com (mx2.confluentasp.com [216.26.153.14]) by mx1.freebsd.org (Postfix) with ESMTP id 9B6A18FC19 for ; Sat, 1 Mar 2008 02:15:13 +0000 (UTC) (envelope-from mikej@paymentallianceintl.com) Received: from calvin.pai.local ([10.0.6.33]) by mx2.confluenttech.com (8.14.1/8.12.9) with ESMTP id m1TMlirQ099069 for ; Fri, 29 Feb 2008 17:47:54 -0500 (EST) (envelope-from mikej@paymentallianceintl.com) x-mimeole: Produced By Microsoft MimeOLE V6.00.3790.4133 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Date: Fri, 29 Feb 2008 17:46:33 -0500 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Importance: normal Priority: normal Thread-Topic: Two (2) Lock Order Reversion - tracking RELENG_7 Thread-Index: Ach7JO7dw9ulb29vQi+QyBdD7pUg2g== From: "Michael Jung" To: Subject: Two (2) Lock Order Reversion - tracking RELENG_7 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Mar 2008 02:15:14 -0000 Hello everyone: Please see complete dmesg output for two LOR's I'm tracking: *default release=cvs tag=RELENG_7 For now, this is a spare box that can be subjected to any type of testing - you only need to ask. I'm an avid advocate of FreeBSD but make no claims in understanding internals from anything but a 7956 foot level - and that varies ;-) FreeBSD zega.mikej.com 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #0: Sun Feb 24 12:44:54 UTC 2008 mikej@zega.mikej.com:/usr/obj/usr/src/sys/ZEGA i386 GENERIC kernel other than: options KVA_PAGES=512 options KDB # Enable kernel debugger support. options DDB # Support DDB. options GDB # Support remote GDB. options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed. AND: /boot/loader.conf kern.sync_on_panic=0 console="comconsole vidconsole" vfs.zfs.prefetch_disable="1" zfs_load="YES" vfs.zfs.arc_max="320M" vm.kmem_size_max="1500M" vm.kmem_size="1500M" Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-PRERELEASE #0: Sun Feb 24 12:44:54 UTC 2008 mikej@zega.mikej.com:/usr/obj/usr/src/sys/ZEGA WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.66GHz (2660.02-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Features2=0x4400 real memory = 2146697216 (2047 MB) avail memory = 2087800832 (1991 MB) ACPI APIC Table: WITNESS: spin lock intrcnt not in order list ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) hptrr: HPT RocketRAID controller driver v1.1 (Feb 24 2008 12:41:19) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 7ff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 p4tcc0: on cpu0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: on hostb0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xac00-0xac7f mem 0xe0000000-0xefffffff,0xfd9c0000-0xfd9fffff irq 16 at device 0.0 on pci1 uhci0: port 0xe000-0xe01f irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xe400-0xe41f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xe800-0xe81f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xec00-0xec1f irq 16 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xfebffc00-0xfebfffff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered pcib2: at device 30.0 on pci0 pci2: on pcib2 em0: port 0xbc00-0xbc3f mem 0xfeae0000-0xfeafffff,0xfeac0000-0xfeadffff irq 18 at device 0.0 on pci2 em0: Ethernet address: 00:1b:21:10:1d:55 em0: [FILTER] twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0xb800-0xb80f mem 0xfea9fc00-0xfea9fc0f,0xfe000000-0xfe7fffff irq 19 at device 1.0 on pci2 twe0: [GIANT-LOCKED] twe0: [ITHREAD] twe0: 4 ports, Firmware FE7X 1.05.00.068, BIOS BE7X 1.08.00.048 rl0: port 0xb400-0xb4ff mem 0xfea9f800-0xfea9f8ff irq 22 at device 5.0 on pci2 miibus0: on rl0 rlphy0: PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl0: Ethernet address: 00:50:2c:a4:f7:b3 rl0: [ITHREAD] isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 31.5 (no driver attached) acpi_button0: on acpi0 acpi_button1: on acpi0 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio0: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd1fff,0xd2800-0xd37ff pnpid ORM0000 on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: on ppc0 ppbus0: [ITHREAD] plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounter "TSC" frequency 2660018636 Hz quality 800 Timecounters tick every 1.000 msec hptrr: no controller detected. ZFS filesystem version 6 ZFS storage pool version 6 ad0: 194479MB at ata0-master UDMA100 ad1: 194481MB at ata0-slave UDMA100 ad2: 194481MB at ata1-master UDMA100 ad3: 194481MB at ata1-slave UDMA100 twed0: on twe0 twed0: 715422MB (1465185024 sectors) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/twed0s1a em0: link state changed to UP lock order reversal: 1st 0x80ba3c20 sched lock (sched lock) @ /usr/src/sys/kern/subr_trap.c:86 2nd 0x80c081c8 sio (sio) @ /usr/src/sys/dev/sio/sio.c:2556 KDB: stack backtrace: db_trace_self_wrapper(80a964f8,fae2aae0,8076f525,80a978e6,80c081c8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(80a978e6,80c081c8,80b649c0,80b649c0,80ab5543,...) at kdb_backtrace+0x29 witness_checkorder(80c081c8,9,80ab5543,9fc,fae2ab10,...) at witness_checkorder+0x5e5 _mtx_lock_spin_flags(80c081c8,0,80ab5543,9fc,fae2ab2c,...) at _mtx_lock_spin_flags+0x32 sio_cnputc(80b649e0,66,fae2acc0,5,66,...) at sio_cnputc+0x86 cnputc(66,fae2acc0,fae2ab80,80765a61,3c6,...) at cnputc+0x5f putcons(3c6,80a94f2d,15c3804,885c8000,80a971d1,...) at putcons+0x17 putchar(66,fae2acc0,80ba3cc0,893fdc34,0,...) at putchar+0x61 kvprintf(80a971d0,80765a00,fae2acc0,a,fae2acec,...) at kvprintf+0x75 printf(80a971d0,0,80a971b2,56,885c3894,...) at printf+0x4e userret(885c8000,fae2ad38,80a971b2,fc,1020804) at userret+0xab ast(fae2ad38) at ast+0x34b doreti_ast() at doreti_ast+0x17 failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() failed to set signal flags properly for ast() lock order reversal: 1st 0x85e07ac8 process slock (process slock) @ /usr/src/sys/kern/kern_resource.c:1066 2nd 0x80c081c8 sio (sio) @ /usr/src/sys/dev/sio/sio.c:2556 KDB: stack backtrace: db_trace_self_wrapper(80a964f8,fad3a93c,8076f525,80a978e6,80c081c8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(80a978e6,80c081c8,80b649c0,80b649c0,80ab5543,...) at kdb_backtrace+0x29 witness_checkorder(80c081c8,9,80ab5543,9fc,85e12630,...) at witness_checkorder+0x5e5 _mtx_lock_spin_flags(80c081c8,0,80ab5543,9fc,80ba8738,...) at _mtx_lock_spin_flags+0x32 sio_cnputc(80b649e0,63,fad3ab1c,5,63,...) at sio_cnputc+0x86 cnputc(63,fad3ab1c,fad3a9dc,80765a61,0,...) at cnputc+0x5f putcons(0,80beeba8,1e126c8,c36,80a94179,...) at putcons+0x17 putchar(63,fad3ab1c,200246,0,fad3aa10,...) at putchar+0x61 kvprintf(80a94178,80765a00,fad3ab1c,a,fad3ab48,...) at kvprintf+0x75 printf(80a94178,4291db58,1f,6cf6ce72,1d,...) at printf+0x4e calcru1(fad3ac38,8073b690,fad3ac30,85e126ec,8a15032a,15aa0) at calcru1+0x25e calcru(85e07ab0,fad3ac30,fad3ac38,42a,85e07ab0,...) at calcru+0xdd rufetchcalc(85e07ab0,fad3ac30,fad3ac30,fad3ac38,fad3acfc,...) at rufetchcalc+0x53 kern_getrusage(85e12630,0,fad3ac30,0,0,...) at kern_getrusage+0x61 getrusage(85e12630,fad3acfc,8,80a97500,80b39598,...) at getrusage+0x27 syscall(fad3ad38) at syscall+0x2d3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (117, FreeBSD ELF32, getrusage), eip = 0x809ed83, esp = 0x7fbfe88c, ebp = 0x8148000 --- calcru: runtime went backwards from 134260841304 usec to 126382165618 usec for pid 827 (dnetc) calcru: runtime went backwards from 4534 usec to 4267 usec for pid 919 (getty) calcru: runtime went backwards from 4652 usec to 4378 usec for pid 904 (getty) calcru: runtime went backwards from 4572 usec to 4303 usec for pid 903 (getty) calcru: runtime went backwards from 4610 usec to 4339 usec for pid 902 (getty) calcru: runtime went backwards from 4495 usec to 4230 usec for pid 901 (getty) calcru: runtime went backwards from 4648 usec to 4375 usec for pid 900 (getty) calcru: runtime went backwards from 4420 usec to 4160 usec for pid 899 (getty) calcru: runtime went backwards from 4481 usec to 4218 usec for pid 898 (getty) calcru: runtime went backwards from 53281990 usec to 51509780 usec for pid 855 (cron) calcru: runtime went backwards from 7674352130 usec to 7224054531 usec for pid 839 (sshd) calcru: runtime went backwards from 43225 usec to 40688 usec for pid 745 (nfsd) calcru: runtime went backwards from 7988 usec to 7519 usec for pid 743 (mountd) calcru: runtime went backwards from 3613 usec to 3401 usec for pid 592 (devd) calcru: runtime went backwards from 28022 usec to 26242 usec for pid 592 (devd) calcru: runtime went backwards from 83141 usec to 79411 usec for pid 128 (spa_zio_intr_1) calcru: runtime went backwards from 18948528 usec to 18427925 usec for pid 50 (softdepflush) calcru: runtime went backwards from 1551354 usec to 1528787 usec for pid 47 (bufdaemon) calcru: runtime went backwards from 8955 usec to 8430 usec for pid 40 (irq1: atkbd0) calcru: runtime went backwards from 8022203 usec to 7770714 usec for pid 27 (irq19: uhci1 twe0) calcru: runtime went backwards from 22096 usec to 20692 usec for pid 20 (swi5: +) calcru: runtime went backwards from 277929 usec to 260275 usec for pid 17 (swi6: task queue) calcru: runtime went backwards from 2438 usec to 2295 usec for pid 12 (swi1: net) calcru: runtime went backwards from 89469281 usec to 84551522 usec for pid 11 (idle: cpu0) calcru: runtime went backwards from 1088731 usec to 1024840 usec for pid 1 (init) calcru: runtime went backwards from 49829588 usec to 46905404 usec for pid 1 (init) calcru: runtime went backwards from 146817 usec to 137491 usec for pid 0 (swapper) #ps -ax PID TT STAT TIME COMMAND 0 ?? WLs 0:00.14 [swapper] 1 ?? ILs 0:01.04 /sbin/init -- 2 ?? DL 0:10.63 [g_event] 3 ?? DL 0:21.83 [g_up] 4 ?? DL 0:16.37 [g_down] 5 ?? DL 0:00.00 [system_taskq] 6 ?? DL 0:00.00 [kqueue taskq] 7 ?? DL 0:00.00 [acpi_task_0] 8 ?? DL 0:00.00 [acpi_task_1] 9 ?? DL 0:00.00 [acpi_task_2] 10 ?? DL 0:00.00 [audit] 11 ?? RL 1:43.84 [idle: cpu0] 12 ?? WL 0:00.00 [swi1: net] 13 ?? WL 6:41.69 [swi4: clock sio] 14 ?? WL 0:00.00 [swi3: vm] 15 ?? DL 0:07.92 [yarrow] 16 ?? WL 0:00.30 [swi6: Giant taskq] 17 ?? WL 0:00.26 [swi6: task queue] 18 ?? DL 0:00.00 [xpt_thrd] 19 ?? WL 0:00.00 [swi2: cambio] 20 ?? WL 0:00.02 [swi5: +] 21 ?? DL 0:00.00 [thread taskq] 22 ?? WL 0:00.00 [irq9: acpi0] 23 ?? WL 0:00.00 [irq16: uhci0 uhci3] 24 ?? DL 0:00.01 [usb0] 25 ?? DL 0:00.00 [usbtask-hc] 26 ?? DL 0:00.00 [usbtask-dr] 27 ?? WL 0:09.38 [irq19: uhci1 twe0] 28 ?? DL 0:00.01 [usb1] 29 ?? WL 0:02.81 [irq18: em0 uhci2] 30 ?? DL 0:00.01 [usb2] 31 ?? DL 0:00.01 [usb3] 32 ?? WL 0:00.00 [irq23: ehci0] 33 ?? DL 0:00.02 [usb4] 34 ?? DL 0:10.49 [em0 taskq] 35 ?? WL 0:00.00 [irq22: rl0] 36 ?? WL 0:00.22 [irq14: ata0] 37 ?? WL 0:00.23 [irq15: ata1] 38 ?? WL 0:00.00 [swi0: sio] 39 ?? DL 0:00.79 [fdc0] 40 ?? WL 0:00.01 [irq1: atkbd0] 41 ?? WL 0:00.00 [irq7: ppbus0 ppc0] 42 ?? DL 0:00.00 [sctp_iterator] 43 ?? DL 0:00.99 [arc_reclaim_thread] 44 ?? DL 0:00.22 [pagedaemon] 45 ?? DL 0:00.00 [vmdaemon] 46 ?? DL 0:00.00 [pagezero] 47 ?? DL 0:01.74 [bufdaemon] 48 ?? DL 9:47.21 [syncer] 49 ?? DL 0:01.94 [vnlru] 50 ?? DL 0:21.50 [softdepflush] 51 ?? DL 0:22.08 [schedcpu] 125 ?? DL 0:00.00 [spa_zio_issue_0] 126 ?? DL 0:00.00 [spa_zio_intr_0] 127 ?? DL 0:00.00 [spa_zio_issue_1] 128 ?? DL 0:00.08 [spa_zio_intr_1] 129 ?? DL 0:00.03 [spa_zio_issue_2] 130 ?? DL 0:02.80 [spa_zio_intr_2] 131 ?? DL 0:00.00 [spa_zio_issue_3] 132 ?? DL 0:00.00 [spa_zio_intr_3] 133 ?? DL 0:00.00 [spa_zio_issue_4] 134 ?? DL 0:00.00 [spa_zio_intr_4] 135 ?? DL 0:00.00 [spa_zio_issue_5] 136 ?? DL 0:00.00 [spa_zio_intr_5] 137 ?? DL 0:00.03 [vdev:worker ad0] 138 ?? DL 0:00.03 [vdev:worker ad1] 139 ?? DL 0:00.03 [vdev:worker ad2] 140 ?? DL 0:00.03 [vdev:worker ad3] 141 ?? DL 0:00.40 [txg_thread_enter] 142 ?? DL 0:03.00 [txg_thread_enter] 143 ?? DL 0:00.40 [txg_thread_enter] 144 ?? DL 0:00.00 [zil_clean] 145 ?? DL 0:00.01 [zil_clean] 592 ?? Is 0:00.00 /sbin/devd 646 ?? Is 0:00.45 /usr/sbin/syslogd -s 722 ?? Ss 0:00.25 /usr/sbin/rpcbind 743 ?? Is 0:00.01 /usr/sbin/mountd -r 745 ?? Is 0:00.04 nfsd: master (nfsd) 747 ?? I 0:19.35 nfsd: server (nfsd) 748 ?? I 0:00.24 nfsd: server (nfsd) 749 ?? I 0:00.02 nfsd: server (nfsd) 750 ?? I 0:00.01 nfsd: server (nfsd) 801 ?? Ss 0:07.91 /usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift 827 ?? RNs 2585:41.13 /usr/local/distributed.net/dnetc -quiet 839 ?? Is 0:00.01 /usr/sbin/sshd 845 ?? Ss 0:03.38 sendmail: accepting connections (sendmail) 849 ?? Is 0:00.08 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) 855 ?? Is 0:00.60 /usr/sbin/cron -s 45427 ?? Is 0:00.08 sshd: mikej [priv] (sshd) 45431 ?? S 0:04.18 sshd: mikej@ttyp0 (sshd) 919 v0 Is+ 0:00.00 /usr/libexec/getty Pc ttyv0 898 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1 899 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2 900 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3 901 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4 902 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5 903 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6 904 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7 10793 p0 R+ 0:00.00 ps -ax 10794 p0 R+ 0:00.00 su (bash) 45433 p0 Is 0:00.03 -bash (bash) 45453 p0 I 0:00.01 su 45454 p0 S 0:00.20 su (bash) DUMB? QUESTION: Why did the LOR not trigger a kernel TRAP? - At 46 I'm still trying to learn. These errors did not occur at boot, but sometime I'm estimating 1+days after the system was up and running CONFIDENTIALITY NOTE: This message is intended only for the use of the individual or entity to whom it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, please notify us by telephone at (502) 212-4001 or notify us at PAI , Dept. 99, 11857 Commonwealth Drive, Louisville, KY 40299. Thank you.