From owner-freebsd-stable@freebsd.org Sat Nov 21 16:22:59 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 61C8346A8CE for ; Sat, 21 Nov 2020 16:22:59 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq6.tb.mail.iss.as9143.net (smtpq6.tb.mail.iss.as9143.net [212.54.42.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CddxY56rJz4sKg; Sat, 21 Nov 2020 16:22:57 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.136] (helo=smtp12.tb.mail.iss.as9143.net) by smtpq6.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kgVf4-0006eL-UT; Sat, 21 Nov 2020 17:22:54 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp12.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1kgVf4-0006CV-IL; Sat, 21 Nov 2020 17:22:54 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id A09E2311; Sat, 21 Nov 2020 17:23:35 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Uky2o2zIv_l4; Sat, 21 Nov 2020 17:23:34 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 43E7A289; Sat, 21 Nov 2020 17:23:34 +0100 (CET) From: Peter Blok Message-Id: <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> Content-Type: multipart/signed; boundary="Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Commit 367705+367706 causes a pabic Date: Sat, 21 Nov 2020 17:22:55 +0100 In-Reply-To: <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> Cc: FreeBSD Stable To: Kristof Provost References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=fshi2H0f c=1 sm=1 tr=0 ts=5fb93ede a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=6I5d2MoRAAAA:8 a=6Q3WNqvRAAAA:8 a=-4xoYQfftjp4DLVD92UA:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=IjZwj45LgO3ly-622nXo:22 a=I8PBwKCn76L9oNdl0isp:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4CddxY56rJz4sKg X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.169 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RECEIVED_SPAMHAUS_PBL(0.00)[82.101.198.11:received]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.54.42.169:from]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RCVD_IN_DNSWL_LOW(-0.10)[212.54.42.169:from]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[bsd4all.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[212.54.42.169:from:127.0.2.255]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.54.42.169:from]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 16:22:59 -0000 --Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Kristof, With a GENERIC kernel it does NOT happen. I do have a different iflib = related panic at reboot, but I=E2=80=99ll report that separately. I brought the two config files closer together and found out that if I = remove if_bridge from the config file and have it loaded dynamically = when the bridge is created, the problem no longer happens and everything = works ok. Peter > On 20 Nov 2020, at 15:53, Kristof Provost wrote: >=20 > I still can=E2=80=99t reproduce that panic. >=20 > Does it happen immediately after you start a vnet jail? >=20 > Does it also happen with a GENERIC kernel? >=20 > Regards, > Kristof >=20 > On 20 Nov 2020, at 14:53, Peter Blok wrote: >=20 >> The panic with ipsec code in the backtrace was already very strange. = I was using IPsec, but only on one interface totally separate from the = members of the bridge as well as the bridge itself. The jails were not = doing any ipsec as well. Note that panic was a while ago and it was = after the 1st bridge epochification was done on stable-12 which was = later backed out >>=20 >> Today the system is no longer using ipsec, but it is still compiled = in. I can remove it if need be for a test >>=20 >>=20 >> src.conf >> WITHOUT_KERBEROS=3Dyes >> WITHOUT_GSSAPI=3Dyes >> WITHOUT_SENDMAIL=3Dtrue >> WITHOUT_MAILWRAPPER=3Dtrue >> WITHOUT_DMAGENT=3Dtrue >> WITHOUT_GAMES=3Dtrue >> WITHOUT_IPFILTER=3Dtrue >> WITHOUT_UNBOUND=3Dtrue >> WITHOUT_PROFILE=3Dtrue >> WITHOUT_ATM=3Dtrue >> WITHOUT_BSNMP=3Dtrue >> #WITHOUT_CROSS_COMPILER=3Dtrue >> WITHOUT_DEBUG_FILES=3Dtrue >> WITHOUT_DICT=3Dtrue >> WITHOUT_FLOPPY=3Dtrue >> WITHOUT_HTML=3Dtrue >> WITHOUT_HYPERV=3Dtrue >> WITHOUT_NDIS=3Dtrue >> WITHOUT_NIS=3Dtrue >> WITHOUT_PPP=3Dtrue >> WITHOUT_TALK=3Dtrue >> WITHOUT_TESTS=3Dtrue >> WITHOUT_WIRELESS=3Dtrue >> #WITHOUT_LIB32=3Dtrue >> WITHOUT_LPR=3Dtrue >>=20 >> make.conf >> KERNCONF=3DBHYVE >> MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge = bridgestp if_vxlan pflog libmchain libiconv smbfs linux linux64 = linux_common linuxkpi linprocfs linsysfs ext2fs >> DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 python3=3D3.8= >> OPTIONS_UNSET=3DDOCS NLS MANPAGES >>=20 >> BHYVE >> cpu HAMMER >> ident BHYVE >>=20 >> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug = symbols >> makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support >>=20 >> options CAMDEBUG >>=20 >> options SCHED_ULE # ULE scheduler >> options PREEMPTION # Enable kernel thread = preemption >> options INET # InterNETworking >> options INET6 # IPv6 communications protocols >> options IPSEC >> options TCP_OFFLOAD # TCP offload >> options TCP_RFC7413 # TCP FASTOPEN >> options SCTP # Stream Control Transmission = Protocol >> options FFS # Berkeley Fast Filesystem >> options SOFTUPDATES # Enable FFS soft updates = support >> options UFS_ACL # Support for access control = lists >> options UFS_DIRHASH # Improve performance on big = directories >> options UFS_GJOURNAL # Enable gjournal-based UFS = journaling >> options QUOTA # Enable disk quotas for UFS >> options SUIDDIR >> options NFSCL # Network Filesystem Client >> options NFSD # Network Filesystem Server >> options NFSLOCKD # Network Lock Manager >> options MSDOSFS # MSDOS Filesystem >> options CD9660 # ISO 9660 Filesystem >> options FUSEFS >> options NULLFS # NULL filesystem >> options UNIONFS >> options FDESCFS # File descriptor = filesystem >> options PROCFS # Process filesystem (requires = PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_PART_GPT # GUID Partition Tables. >> options GEOM_RAID # Soft RAID functionality. >> options GEOM_LABEL # Provides labelization >> options GEOM_ELI # Disk encryption. >> options COMPAT_FREEBSD32 # Compatible with i386 binaries >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >> options COMPAT_FREEBSD9 # Compatible with FreeBSD9 >> options COMPAT_FREEBSD10 # Compatible with FreeBSD10 >> options COMPAT_FREEBSD11 # Compatible with FreeBSD11 >> options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI >> options KTRACE # ktrace(1) support >> options STACK # stack(9) support >> options SYSVSHM # SYSV-style shared memory >> options SYSVMSG # SYSV-style message queues >> options SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions >> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) >> options AUDIT # Security event auditing >> options CAPABILITY_MODE # Capsicum capability mode >> options CAPABILITIES # Capsicum capabilities >> options MAC # TrustedBSD MAC Framework >> options MAC_PORTACL >> options MAC_NTPD >> options KDTRACE_FRAME # Ensure frames are compiled in >> options KDTRACE_HOOKS # Kernel DTrace hooks >> options DDB_CTF # Kernel ELF linker loads CTF = data >> options INCLUDE_CONFIG_FILE # Include this file in kernel >>=20 >> # Debugging support. Always need this: >> options KDB # Enable kernel debugger = support. >> options KDB_TRACE # Print a stack trace for a = panic. >> options KDB_UNATTENDED >>=20 >> # Make an SMP-capable kernel by default >> options SMP # Symmetric MultiProcessor = Kernel >> options EARLY_AP_STARTUP >>=20 >> # CPU frequency control >> device cpufreq >> device cpuctl >> device coretemp >>=20 >> # Bus support. >> device acpi >> options ACPI_DMAR >> device pci >> options PCI_IOV # PCI SR-IOV support >>=20 >> device iicbus >> device iicbb >>=20 >> device iic >> device ic >> device iicsmb >>=20 >> device ichsmb >> device smbus >> device smb >>=20 >> #device jedec_dimm >>=20 >> # ATA controllers >> device ahci # AHCI-compatible SATA = controllers >> device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA >>=20 >> # SCSI Controllers >> device mps # LSI-Logic MPT-Fusion 2 >>=20 >> # ATA/SCSI peripherals >> device scbus # SCSI bus (required for = ATA/SCSI) >> device da # Direct Access (disks) >> device cd # CD >> device pass # Passthrough device = (direct ATA/SCSI access) >> device ses # Enclosure Services = (SES and SAF-TE) >> device sg >>=20 >> device cfiscsi >> device ctl # CAM Target Layer >> device iscsi >>=20 >> # atkbdc0 controls both the keyboard and the PS/2 mouse >> device atkbdc # AT keyboard controller >> device atkbd # AT keyboard >> device psm # PS/2 mouse >>=20 >> device kbdmux # keyboard multiplexer >>=20 >> # vt is the new video console driver >> device vt >> device vt_vga >> device vt_efifb >>=20 >> # Serial (COM) ports >> device uart # Generic UART driver >>=20 >> # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure >> device iflib >> device em # Intel PRO/1000 Gigabit = Ethernet Family >> device ix # Intel PRO/10GbE PCIE = PF Ethernet >>=20 >> # Network stack virtualization. >> options VIMAGE >>=20 >> # Pseudo devices. >> device crypto >> device cryptodev >> device loop # Network loopback >> device random # Entropy device >> device padlock_rng # VIA Padlock RNG >> device rdrand_rng # Intel Bull Mountain = RNG >> device ipmi >> device smbios >> device vpd >> device aesni # AES-NI OpenCrypto = module >> device ether # Ethernet support >> device lagg >> device vlan # 802.1Q VLAN support >> device tuntap # Packet tunnel. >> device md # Memory "disks" >> device gif # IPv6 and IPv4 = tunneling >> device firmware # firmware assist module >>=20 >> device pf >> #device pflog >> #device pfsync >>=20 >> # The `bpf' device enables the Berkeley Packet Filter. >> # Be aware of the administrative consequences of enabling this! >> # Note that 'bpf' is required for DHCP. >> device bpf # Berkeley packet filter >>=20 >> # The `epair' device implements a virtual back-to-back connected = Ethernet >> # like interface pair. >> device epair >>=20 >> # USB support >> options USB_DEBUG # enable debug msgs >> device uhci # UHCI PCI->USB = interface >> device ohci # OHCI PCI->USB = interface >> device ehci # EHCI PCI->USB = interface (USB 2.0) >> device xhci # XHCI PCI->USB = interface (USB 3.0) >> device usb # USB Bus (required) >> device uhid >> device ukbd # Keyboard >> device umass # Disks/Mass storage - = Requires scbus and da >> device ums >>=20 >> device filemon >>=20 >> device if_bridge >>=20 >>> On 20 Nov 2020, at 12:53, Kristof Provost wrote: >>>=20 >>> Can you share your kernel config file (and src.conf / make.conf if = they exist)? >>>=20 >>> This second panic is in the IPSec code. My current thinking is that = your kernel config is triggering a bug that=E2=80=99s manifesting in = multiple places, but not actually caused by those places. >>>=20 >>> I=E2=80=99d like to be able to reproduce it so we can debug it. >>>=20 >>> Best regards, >>> Kristof >>>=20 >>> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>>> Hi Kristof, >>>>=20 >>>> This is 12-stable. With the previous bridge epochification that was = backed out my config had a panic too. >>>>=20 >>>> I don=E2=80=99t have any local modifications. I did a clean rebuild = after removing /usr/obj/usr >>>>=20 >>>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko = and nmdm.ko as modules. Everything else is statically linked. I have = removed all drivers not needed for the hardware at hand. >>>>=20 >>>> My bridge is between two vlans from the same trunk and the jail = epair devices as well as the bhyve tap devices. >>>>=20 >>>> The panic happens when the jails are starting. >>>>=20 >>>> I can try to narrow it down over the weekend and make the crash = dump available for analysis. >>>>=20 >>>> Previously I had the following crash with 363492 >>>>=20 >>>> kernel trap 12 with interrupts disabled >>>>=20 >>>>=20 >>>> Fatal trap 12: page fault while in kernel mode >>>> cpuid =3D 2; apic id =3D 02 >>>> fault virtual address =3D 0xffffffff00000410 >>>> fault code =3D supervisor read data, page not present >>>> instruction pointer =3D 0x20:0xffffffff80692326 >>>> stack pointer =3D 0x28:0xfffffe00c06097b0 >>>> frame pointer =3D 0x28:0xfffffe00c06097f0 >>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>> processor eflags =3D resume, IOPL =3D 0 >>>> current process =3D 2030 (ifconfig) >>>> trap number =3D 12 >>>> panic: page fault >>>> cpuid =3D 2 >>>> time =3D 1595683412 >>>> KDB: stack backtrace: >>>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>>> #1 0xffffffff8064d67b at vpanic+0x17b >>>> #2 0xffffffff8064d4f3 at panic+0x43 >>>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>>> #5 0xffffffff809cb9b6 at trap+0x286 >>>> #6 0xffffffff809a5b28 at calltrap+0x8 >>>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>>> #10 0xffffffff8075274f at ifioctl+0x47f >>>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>> On 20 Nov 2020, at 11:30, Kristof Provost wrote: >>>>>=20 >>>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = wrote: >>>>>> I=E2=80=99m afraid the last Epoch fix for bridge is not solving = the problem ( or perhaps creates a new ). >>>>>>=20 >>>>> We=E2=80=99re talking about the stable/12 branch, right? >>>>>=20 >>>>>> This seems to happen when the jail epair is added to the bridge. >>>>>>=20 >>>>> There must be something more to it than that. I=E2=80=99ve run the = bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>>>=20 >>>>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>>>=20 >>>>>> kernel trap 12 with interrupts disabled >>>>>>=20 >>>>>>=20 >>>>>> Fatal trap 12: page fault while in kernel mode >>>>>> cpuid =3D 6; apic id =3D 06 >>>>>> fault virtual address =3D 0xc10 >>>>>> fault code =3D supervisor read data, page not = present >>>>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>>>> processor eflags =3D resume, IOPL =3D 0 >>>>>> current process =3D 1686 (jail) >>>>>> trap number =3D 12 >>>>>> panic: page fault >>>>>> cpuid =3D 6 >>>>>> time =3D 1605811310 >>>>>> KDB: stack backtrace: >>>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>>>=20 >>>>> This panic is rather odd. This isn=E2=80=99t even the bridge code. = This is during initial creation of the vnet. I don=E2=80=99t really see = how this could even trigger panics. >>>>> That panic looks as if something corrupted the net_epoch_preempt, = by overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >>>>>=20 >>>>> Best regards, >>>>> Kristof >>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" --Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIxMTYyMjU1WjAvBgkqhkiG9w0BCQQxIgQg2lnqsKLW ohIuOQjw1vzmZKK+V+gbeoeyk/LVUFsig4owgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAJ2bRojW3XsYwMHChArVS/Nv/jvCy7e5 XtMUTyFl0CLVziXnZoAh9wD/vSpCkXiWuZ8adnDNGEPA7o+9HMEbdTSprDBNjaIsnCsHSh1dZhh8 cwfCNjF/vCHJMa/K5wAMgcFNWpr+ZBUUAgj9CV2AIpOI2L/BCditXoZktPin5wYrthc8XuV1F73+ VmYkTenfWha4DOt+XnclxE2LabgO40g8cQjUCXPd5bfk5TBdRsN6jpiAg9qMGPwFNS3Wuf5Q3B0/ KkiYEYsANkvHFxyR4NYPVypgBhxvCBLWK9/c98fnzcBX3Upcl06era8rBV4uVjDhWsv/d20Bkddn KdL93z0AAAAAAAA= --Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA--