Date: Mon, 23 Nov 2020 11:52:51 +0100 From: Peter Blok <pblok@bsd4all.org> To: Kristof Provost <kp@FreeBSD.org> Cc: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: Commit 367705+367706 causes a pabic Message-ID: <AAFB77E4-1D1B-4C4B-BAF5-66C5455521AF@bsd4all.org> In-Reply-To: <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> References: <CD3B0F62-3790-4C63-A92C-9694256823CD@bsd4all.org> <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <BD8D114E-9F12-4580-A0D5-5A5BAC75DF27@FreeBSD.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Kristof, With commit 367705+367706 and if_bridge statically linked. It crashes = while adding an epair of a jail. With commit 367705+367706 and if_bridge dynamically loaded there is a = crash at reboot #0 0xffffffff8069ddc5 at kdb_backtrace+0x65 #1 0xffffffff80652c8b at vpanic+0x17b #2 0xffffffff80652b03 at panic+0x43 #3 0xffffffff809c8951 at trap_fatal+0x391 #4 0xffffffff809c89af at trap_pfault+0x4f #5 0xffffffff809c7ff6 at trap+0x286 #6 0xffffffff809a1ec8 at calltrap+0x8 #7 0xffffffff8079f7ed at ip_input+0x63d #8 0xffffffff8077a07a at netisr_dispatch_src+0xca #9 0xffffffff8075a6f8 at ether_demux+0x138 #10 0xffffffff8075b9bb at ether_nh_input+0x33b #11 0xffffffff8077a07a at netisr_dispatch_src+0xca #12 0xffffffff8075ab1b at ether_input+0x4b #13 0xffffffff8077a80b at swi_net+0x12b #14 0xffffffff8061e10c at ithread_loop+0x23c #15 0xffffffff8061afbe at fork_exit+0x7e #16 0xffffffff809a2efe at fork_trampoline+0xe Peter > On 21 Nov 2020, at 17:22, Peter Blok <pblok@bsd4all.org> wrote: >=20 > Kristof, >=20 > With a GENERIC kernel it does NOT happen. I do have a different iflib = related panic at reboot, but I=E2=80=99ll report that separately. >=20 > I brought the two config files closer together and found out that if I = remove if_bridge from the config file and have it loaded dynamically = when the bridge is created, the problem no longer happens and everything = works ok. >=20 > Peter >=20 >> On 20 Nov 2020, at 15:53, Kristof Provost <kp@FreeBSD.org> wrote: >>=20 >> I still can=E2=80=99t reproduce that panic. >>=20 >> Does it happen immediately after you start a vnet jail? >>=20 >> Does it also happen with a GENERIC kernel? >>=20 >> Regards, >> Kristof >>=20 >> On 20 Nov 2020, at 14:53, Peter Blok wrote: >>=20 >>> The panic with ipsec code in the backtrace was already very strange. = I was using IPsec, but only on one interface totally separate from the = members of the bridge as well as the bridge itself. The jails were not = doing any ipsec as well. Note that panic was a while ago and it was = after the 1st bridge epochification was done on stable-12 which was = later backed out >>>=20 >>> Today the system is no longer using ipsec, but it is still compiled = in. I can remove it if need be for a test >>>=20 >>>=20 >>> src.conf >>> WITHOUT_KERBEROS=3Dyes >>> WITHOUT_GSSAPI=3Dyes >>> WITHOUT_SENDMAIL=3Dtrue >>> WITHOUT_MAILWRAPPER=3Dtrue >>> WITHOUT_DMAGENT=3Dtrue >>> WITHOUT_GAMES=3Dtrue >>> WITHOUT_IPFILTER=3Dtrue >>> WITHOUT_UNBOUND=3Dtrue >>> WITHOUT_PROFILE=3Dtrue >>> WITHOUT_ATM=3Dtrue >>> WITHOUT_BSNMP=3Dtrue >>> #WITHOUT_CROSS_COMPILER=3Dtrue >>> WITHOUT_DEBUG_FILES=3Dtrue >>> WITHOUT_DICT=3Dtrue >>> WITHOUT_FLOPPY=3Dtrue >>> WITHOUT_HTML=3Dtrue >>> WITHOUT_HYPERV=3Dtrue >>> WITHOUT_NDIS=3Dtrue >>> WITHOUT_NIS=3Dtrue >>> WITHOUT_PPP=3Dtrue >>> WITHOUT_TALK=3Dtrue >>> WITHOUT_TESTS=3Dtrue >>> WITHOUT_WIRELESS=3Dtrue >>> #WITHOUT_LIB32=3Dtrue >>> WITHOUT_LPR=3Dtrue >>>=20 >>> make.conf >>> KERNCONF=3DBHYVE >>> MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge = bridgestp if_vxlan pflog libmchain libiconv smbfs linux linux64 = linux_common linuxkpi linprocfs linsysfs ext2fs >>> DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 = python3=3D3.8 >>> OPTIONS_UNSET=3DDOCS NLS MANPAGES >>>=20 >>> BHYVE >>> cpu HAMMER >>> ident BHYVE >>>=20 >>> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug = symbols >>> makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support >>>=20 >>> options CAMDEBUG >>>=20 >>> options SCHED_ULE # ULE scheduler >>> options PREEMPTION # Enable kernel thread = preemption >>> options INET # InterNETworking >>> options INET6 # IPv6 communications protocols >>> options IPSEC >>> options TCP_OFFLOAD # TCP offload >>> options TCP_RFC7413 # TCP FASTOPEN >>> options SCTP # Stream Control Transmission = Protocol >>> options FFS # Berkeley Fast Filesystem >>> options SOFTUPDATES # Enable FFS soft updates = support >>> options UFS_ACL # Support for access control = lists >>> options UFS_DIRHASH # Improve performance on big = directories >>> options UFS_GJOURNAL # Enable gjournal-based UFS = journaling >>> options QUOTA # Enable disk quotas for UFS >>> options SUIDDIR >>> options NFSCL # Network Filesystem Client >>> options NFSD # Network Filesystem Server >>> options NFSLOCKD # Network Lock Manager >>> options MSDOSFS # MSDOS Filesystem >>> options CD9660 # ISO 9660 Filesystem >>> options FUSEFS >>> options NULLFS # NULL filesystem >>> options UNIONFS >>> options FDESCFS # File descriptor = filesystem >>> options PROCFS # Process filesystem (requires = PSEUDOFS) >>> options PSEUDOFS # Pseudo-filesystem framework >>> options GEOM_PART_GPT # GUID Partition Tables. >>> options GEOM_RAID # Soft RAID functionality. >>> options GEOM_LABEL # Provides labelization >>> options GEOM_ELI # Disk encryption. >>> options COMPAT_FREEBSD32 # Compatible with i386 binaries >>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >>> options COMPAT_FREEBSD9 # Compatible with FreeBSD9 >>> options COMPAT_FREEBSD10 # Compatible with FreeBSD10 >>> options COMPAT_FREEBSD11 # Compatible with FreeBSD11 >>> options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI >>> options KTRACE # ktrace(1) support >>> options STACK # stack(9) support >>> options SYSVSHM # SYSV-style shared memory >>> options SYSVMSG # SYSV-style message queues >>> options SYSVSEM # SYSV-style semaphores >>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions >>> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. >>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >>> options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) >>> options AUDIT # Security event auditing >>> options CAPABILITY_MODE # Capsicum capability mode >>> options CAPABILITIES # Capsicum capabilities >>> options MAC # TrustedBSD MAC Framework >>> options MAC_PORTACL >>> options MAC_NTPD >>> options KDTRACE_FRAME # Ensure frames are compiled in >>> options KDTRACE_HOOKS # Kernel DTrace hooks >>> options DDB_CTF # Kernel ELF linker loads CTF = data >>> options INCLUDE_CONFIG_FILE # Include this file in kernel >>>=20 >>> # Debugging support. Always need this: >>> options KDB # Enable kernel debugger = support. >>> options KDB_TRACE # Print a stack trace for a = panic. >>> options KDB_UNATTENDED >>>=20 >>> # Make an SMP-capable kernel by default >>> options SMP # Symmetric MultiProcessor = Kernel >>> options EARLY_AP_STARTUP >>>=20 >>> # CPU frequency control >>> device cpufreq >>> device cpuctl >>> device coretemp >>>=20 >>> # Bus support. >>> device acpi >>> options ACPI_DMAR >>> device pci >>> options PCI_IOV # PCI SR-IOV support >>>=20 >>> device iicbus >>> device iicbb >>>=20 >>> device iic >>> device ic >>> device iicsmb >>>=20 >>> device ichsmb >>> device smbus >>> device smb >>>=20 >>> #device jedec_dimm >>>=20 >>> # ATA controllers >>> device ahci # AHCI-compatible SATA = controllers >>> device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA >>>=20 >>> # SCSI Controllers >>> device mps # LSI-Logic MPT-Fusion 2 >>>=20 >>> # ATA/SCSI peripherals >>> device scbus # SCSI bus (required for = ATA/SCSI) >>> device da # Direct Access (disks) >>> device cd # CD >>> device pass # Passthrough device = (direct ATA/SCSI access) >>> device ses # Enclosure Services = (SES and SAF-TE) >>> device sg >>>=20 >>> device cfiscsi >>> device ctl # CAM Target Layer >>> device iscsi >>>=20 >>> # atkbdc0 controls both the keyboard and the PS/2 mouse >>> device atkbdc # AT keyboard controller >>> device atkbd # AT keyboard >>> device psm # PS/2 mouse >>>=20 >>> device kbdmux # keyboard multiplexer >>>=20 >>> # vt is the new video console driver >>> device vt >>> device vt_vga >>> device vt_efifb >>>=20 >>> # Serial (COM) ports >>> device uart # Generic UART driver >>>=20 >>> # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure >>> device iflib >>> device em # Intel PRO/1000 Gigabit = Ethernet Family >>> device ix # Intel PRO/10GbE PCIE = PF Ethernet >>>=20 >>> # Network stack virtualization. >>> options VIMAGE >>>=20 >>> # Pseudo devices. >>> device crypto >>> device cryptodev >>> device loop # Network loopback >>> device random # Entropy device >>> device padlock_rng # VIA Padlock RNG >>> device rdrand_rng # Intel Bull Mountain = RNG >>> device ipmi >>> device smbios >>> device vpd >>> device aesni # AES-NI OpenCrypto = module >>> device ether # Ethernet support >>> device lagg >>> device vlan # 802.1Q VLAN support >>> device tuntap # Packet tunnel. >>> device md # Memory "disks" >>> device gif # IPv6 and IPv4 = tunneling >>> device firmware # firmware assist module >>>=20 >>> device pf >>> #device pflog >>> #device pfsync >>>=20 >>> # The `bpf' device enables the Berkeley Packet Filter. >>> # Be aware of the administrative consequences of enabling this! >>> # Note that 'bpf' is required for DHCP. >>> device bpf # Berkeley packet filter >>>=20 >>> # The `epair' device implements a virtual back-to-back connected = Ethernet >>> # like interface pair. >>> device epair >>>=20 >>> # USB support >>> options USB_DEBUG # enable debug msgs >>> device uhci # UHCI PCI->USB = interface >>> device ohci # OHCI PCI->USB = interface >>> device ehci # EHCI PCI->USB = interface (USB 2.0) >>> device xhci # XHCI PCI->USB = interface (USB 3.0) >>> device usb # USB Bus (required) >>> device uhid >>> device ukbd # Keyboard >>> device umass # Disks/Mass storage - = Requires scbus and da >>> device ums >>>=20 >>> device filemon >>>=20 >>> device if_bridge >>>=20 >>>> On 20 Nov 2020, at 12:53, Kristof Provost <kp@FreeBSD.org> wrote: >>>>=20 >>>> Can you share your kernel config file (and src.conf / make.conf if = they exist)? >>>>=20 >>>> This second panic is in the IPSec code. My current thinking is that = your kernel config is triggering a bug that=E2=80=99s manifesting in = multiple places, but not actually caused by those places. >>>>=20 >>>> I=E2=80=99d like to be able to reproduce it so we can debug it. >>>>=20 >>>> Best regards, >>>> Kristof >>>>=20 >>>> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>>>> Hi Kristof, >>>>>=20 >>>>> This is 12-stable. With the previous bridge epochification that = was backed out my config had a panic too. >>>>>=20 >>>>> I don=E2=80=99t have any local modifications. I did a clean = rebuild after removing /usr/obj/usr >>>>>=20 >>>>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko = and nmdm.ko as modules. Everything else is statically linked. I have = removed all drivers not needed for the hardware at hand. >>>>>=20 >>>>> My bridge is between two vlans from the same trunk and the jail = epair devices as well as the bhyve tap devices. >>>>>=20 >>>>> The panic happens when the jails are starting. >>>>>=20 >>>>> I can try to narrow it down over the weekend and make the crash = dump available for analysis. >>>>>=20 >>>>> Previously I had the following crash with 363492 >>>>>=20 >>>>> kernel trap 12 with interrupts disabled >>>>>=20 >>>>>=20 >>>>> Fatal trap 12: page fault while in kernel mode >>>>> cpuid =3D 2; apic id =3D 02 >>>>> fault virtual address =3D 0xffffffff00000410 >>>>> fault code =3D supervisor read data, page not = present >>>>> instruction pointer =3D 0x20:0xffffffff80692326 >>>>> stack pointer =3D 0x28:0xfffffe00c06097b0 >>>>> frame pointer =3D 0x28:0xfffffe00c06097f0 >>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>>> processor eflags =3D resume, IOPL =3D 0 >>>>> current process =3D 2030 (ifconfig) >>>>> trap number =3D 12 >>>>> panic: page fault >>>>> cpuid =3D 2 >>>>> time =3D 1595683412 >>>>> KDB: stack backtrace: >>>>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>>>> #1 0xffffffff8064d67b at vpanic+0x17b >>>>> #2 0xffffffff8064d4f3 at panic+0x43 >>>>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>>>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>>>> #5 0xffffffff809cb9b6 at trap+0x286 >>>>> #6 0xffffffff809a5b28 at calltrap+0x8 >>>>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>>>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>>>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>>>> #10 0xffffffff8075274f at ifioctl+0x47f >>>>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>>>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>>>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>>>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On 20 Nov 2020, at 11:30, Kristof Provost <kp@FreeBSD.org> wrote: >>>>>>=20 >>>>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = <mailto:peter.blok@bsd4all.org> wrote: >>>>>>> I=E2=80=99m afraid the last Epoch fix for bridge is not solving = the problem ( or perhaps creates a new ). >>>>>>>=20 >>>>>> We=E2=80=99re talking about the stable/12 branch, right? >>>>>>=20 >>>>>>> This seems to happen when the jail epair is added to the bridge. >>>>>>>=20 >>>>>> There must be something more to it than that. I=E2=80=99ve run = the bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>>>>=20 >>>>>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>>>>=20 >>>>>>> kernel trap 12 with interrupts disabled >>>>>>>=20 >>>>>>>=20 >>>>>>> Fatal trap 12: page fault while in kernel mode >>>>>>> cpuid =3D 6; apic id =3D 06 >>>>>>> fault virtual address =3D 0xc10 >>>>>>> fault code =3D supervisor read data, page not = present >>>>>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>>>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>>>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>>>> =3D DPL 0, pres 1, long 1, def32 0, gran = 1 >>>>>>> processor eflags =3D resume, IOPL =3D 0 >>>>>>> current process =3D 1686 (jail) >>>>>>> trap number =3D 12 >>>>>>> panic: page fault >>>>>>> cpuid =3D 6 >>>>>>> time =3D 1605811310 >>>>>>> KDB: stack backtrace: >>>>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>>>>=20 >>>>>> This panic is rather odd. This isn=E2=80=99t even the bridge = code. This is during initial creation of the vnet. I don=E2=80=99t = really see how this could even trigger panics. >>>>>> That panic looks as if something corrupted the net_epoch_preempt, = by overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >>>>>>=20 >>>>>> Best regards, >>>>>> Kristof >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 --Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIzMTA1MjUxWjAvBgkqhkiG9w0BCQQxIgQgylwhoXHE RB0XAzFzJW/cnRjL+y0Qj5NdIuGXjLpIaaYwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBABf1fGOn9oO4fqG7NMUn8YzqEduGArgd RAIGoPj+QDUXPX0e+DSMWHA7k1ZDjfKX5rhN92qIZVAMG/Egea010SjNoERAmx6oM5qcTy70l0eQ BKMyNfBYSIxWqRBEhIZW2z2pvDLy46bEh5vaKC0cj7WjkUUt1qs6P/PeR3oAEz9MZNHfFo7u9Et9 ABW3xSEMPmFtTIG6EyBlr6jPXAbZ/ymLy6shxlh4yTnjDHYcJlm1b/Z7orJV4ud67cjj00b14h2c WYGV2pIYc+t40D45mmGPCV2yslKnLLbm/Db1VQP5SslTQE4KlTxvJHh38u3jC2jhnXdVoW/DEoIV H+ZJKQoAAAAAAAA= --Apple-Mail=_EFD48E1D-FDDB-48C6-9795-32CADED890C4--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AAFB77E4-1D1B-4C4B-BAF5-66C5455521AF>