From nobody Wed Mar 2 17:26:18 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D213519F89CC for ; Wed, 2 Mar 2022 17:25:49 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K81H10Vwkz4XWX for ; Wed, 2 Mar 2022 17:25:49 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-pl1-x630.google.com with SMTP id z2so2139845plg.8 for ; Wed, 02 Mar 2022 09:25:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0B1TttZCtrKyN6S0dLLr5yh5o8OGkx4VXXA2Ej13s/E=; b=Szeh/nA4oCf9087cLdLI9IGOpss3/QpUsfrqS4Uq81eKHUVv7YWCzfZKoQUyI1a20Y dOL7sS5DvlI77tex5GjnObTZ48nTgAGgimOtzZZRu2EjxJCEUeI6EDQJ2sL1vXmwooYS fYXCPF2iE6HljopKkYbudWNetZxlfNax47iitv9d3Wb6ckXfr0OSiZlD0i2KFV2pzklT dm+6sKP16fSWfnqLcHSKo1LYPM4aGo7kjN7rJTPB/RMY/AbBmGv6xtD3suDo3uZ/LqBT gfodUyRn+56JzzyprruvN4ZCIhoWUNVYyBe+GNZH1iwkpBw95b9evQ5rNleRU8/yfEgy GQXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0B1TttZCtrKyN6S0dLLr5yh5o8OGkx4VXXA2Ej13s/E=; b=5ITUbV/j5F5SuE1dDAFBlKp6jO6OqYBBecfje0jz+1Iy2Y/Fq5Xduz3EXdaBzPq9Fa 5TvWt6g14R34TrXOlrHEoX7/RMQb5H3CxWgYQ9gkj6GtvN1xkv8y29a5rQICSYTfNfKT ZIXo8Cylg4NIEXckZ6pTdM48inKxTGIY8CqJcBsM+5NYYdOUagRWOeyHPogcKSUWANsT i5+f7TbNkGmDl5DUtFvKMG8d2U/rRdnox386YZlDg3crMLquxBlk6azMmkVKYiIWm7yP Z6K3rUVagK8V/TN1XBYOcDTuKIqRT5rYRyU9D//Y9dDiVPfbeIwqi8IGLbwSi+ZX074f RCNw== X-Gm-Message-State: AOAM531awtbOmKcNrQlX3sPiJ5eWA3exz8SpBySxxRMzy4lcARqRwnVa epRJpjSTWmU3ueWiqf0h6/IvZTCnSiz8w9uJ5Q8= X-Google-Smtp-Source: ABdhPJwaMUHTPQxs7AV/ZKO9oDBuh3X+aIoOUztrwEuwkGRuH9veKu2E7O1dhlFtIga3pRzYw5RRQxcHgOsl7rgCxPs= X-Received: by 2002:a17:902:bc83:b0:149:b26a:b9c8 with SMTP id bb3-20020a170902bc8300b00149b26ab9c8mr31746402plb.143.1646241948008; Wed, 02 Mar 2022 09:25:48 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 References: <202203011540.221FeR4f028103@nfbcal.org> In-Reply-To: From: Ze Dupsys Date: Wed, 2 Mar 2022 19:26:18 +0200 Message-ID: Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org Content-Type: multipart/mixed; boundary="00000000000086a47505d93f92ea" X-Rspamd-Queue-Id: 4K81H10Vwkz4XWX X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="Szeh/nA4"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2607:f8b0:4864:20::630 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_ATTACHMENT(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.995]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::630:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --00000000000086a47505d93f92ea Content-Type: multipart/alternative; boundary="00000000000086a47305d93f92e8" --00000000000086a47305d93f92e8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Today managed to crash lab Dom0 with: xen_cmdline=3D"dom0_mem=3D6144M dom0_max_vcpus=3D2 dom0=3Dpvh,verbose=3D1 console=3Dvga,com1 com1=3D9600,8n1 guest_loglvl=3Dall loglvl=3Dall sync_con= sole=3D1 reboot=3Dno" I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as in attachment, panic was with the same fingerprint as the one with "rman_is_region_manager" line already reported. The scripts i ran in parallel generally were the same as attached in bug report, just a bit modified. 1) ./libexec.sh zfs_volstress_fast_4g (this just creates new ZVOLs and instead of 2GB, it writes 4BG in each ZVOL created dd if=3D/dev/zero) 2) ./test_vm1_zvol_3gb.sh (this loops commands: start first DomU, write 3GB in it's /tmp, restart DomU, removes /tmp, repeat) 3) ./test_vm2_zvol_5_on_off.sh (this loops: start second DomU, which has 5 disks attached, turn off DomU, repeat) 4) monitoring, sleep 120 seconds, print vmstat | sort in serial output. Around dom id 108, system started to behave suspiciously, xl list showed DomUs created, but they did not really start up, script timeout-ed for ssh connection, no vnc. When i did xl destroy manually, and xl create, system panic happened. I have log files for all serial output, if there is anything useful, i can provide. On disk log files seems to loose latest messages due to crash. On Wed, Mar 2, 2022 at 3:57 PM Roger Pau Monn=C3=A9 = wrote: > On Wed, Mar 02, 2022 at 10:57:37AM +0200, Ze Dupsys wrote: > > Hello, > > > > I started using XEN on one pre-production machine (with aim to use late= r > in > > production) with 12.2, but since it experienced random crashes i update= d > to > > 13.0 in hope that errors might disappear. > > > > I do not know how detailed should i write, so that this email is not to= o > > long, but gives enough info. > > > > FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rule= s > > for NATting are used. Zpool is composed of 2 mirrored disks. There is a > > ZVOL volmode=3Ddev for each VM and VM's jail that are attached as raw > devices > > to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, > with > > VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i have > > bridge interfaces, where DomU's are connected depending on their > > "zone/network", those that have allowed outgoing connections are NATted > by > > IPFW on specific physical NIC and IP. > > So from the traces on the ticket: > > panic: pmap_growkernel: no memory to grow kernel > cpuid =3D 0 > time =3D 1646123072 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff81073eed at pmap_growkernel+0x27d > #4 0xffffffff80f2dae8 at vm_map_insert+0x248 > #5 0xffffffff80f30249 at vm_map_find+0x549 > #6 0xffffffff80f2bf76 at kmem_init+0x226 > #7 0xffffffff80c73341 at vmem_xalloc+0xcb1 > #8 0xffffffff80c72c3b at vmem_xalloc+0x5ab > #9 0xffffffff80f2bfce at kmem_init+0x27e > #10 0xffffffff80c73341 at vmem_xalloc+0xcb1 > #11 0xffffffff80c72c3b at vmem_xalloc+0x5ab > #12 0xffffffff80c72646 at vmem_alloc+0x46 > #13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96 > #14 0xffffffff80f21a2a at uma_prealloc+0x23a > #15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e > #16 0xffffffff80f1f6af at uma_set_align+0x8f > #17 0xffffffff82463362 at abd_borrow_buf_copy+0x22 > Uptime: 4m9s > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 0; apic id =3D 00 > fault virtual address =3D 0x22710028 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80c45892 > stack pointer =3D 0x28:0xfffffe0096600930 > frame pointer =3D 0x28:0xfffffe0096600930 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 1496 (devmatch) > trap number =3D 12 > panic: page fault > cpuid =3D 0 > time =3D 1646123791 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff80c62011 at rman_is_region_manager+0x241 > #8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101 > #9 0xffffffff80c1949c at kernel_sysctl+0x43c > #10 0xffffffff80c19b13 at userland_sysctl+0x173 > #11 0xffffffff80c1995f at sys___sysctl+0x5f > #12 0xffffffff8108baac at amd64_syscall+0x10c > #13 0xffffffff8106243e at Xfast_syscall+0xfe > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 1; apic id =3D 02 > fault virtual address =3D 0x68 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff824a599d > stack pointer =3D 0x28:0xfffffe00b1e27910 > frame pointer =3D 0x28:0xfffffe00b1e279b0 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 0 (xbbd7 taskq) > trap number =3D 12 > panic: page fault > cpuid =3D 1 > time =3D 1646122723 > KDB: stack backtrace: > #0 0xffffffff80c57525 at kdb_backtrace+0x65 > #1 0xffffffff80c09f01 at vpanic+0x181 > #2 0xffffffff80c09d73 at panic+0x43 > #3 0xffffffff8108b1a7 at trap+0xbc7 > #4 0xffffffff8108b1ff at trap+0xc1f > #5 0xffffffff8108a85d at trap+0x27d > #6 0xffffffff81061b18 at calltrap+0x8 > #7 0xffffffff8248935a at dmu_read+0x2a > #8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa > #9 0xffffffff80a7f214 at xbd_instance_create+0xa394 > #10 0xffffffff80a7b1ea at xbd_instance_create+0x636a > #11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1 > #12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac > #13 0xffffffff80bc7e3e at fork_exit+0x7e > #14 0xffffffff81062b9e at fork_trampoline+0xe > Uptime: 1h44m10s > > This all look to me like Out of Memory conditions, can you check with > `top` whats going on with your memory? > > Might also be helpful to record periodic calls to `vmstat -m | sort -k > 2 -r` to try to figure out what's using so much memory. > > Regards, Roger. > --00000000000086a47305d93f92e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Today managed to crash lab Dom0 with:
xen_c= mdline=3D"dom0_mem=3D6144M dom0_max_vcpus=3D2 dom0=3Dpvh,verbose=3D1 c= onsole=3Dvga,com1 com1=3D9600,8n1 guest_loglvl=3Dall loglvl=3Dall sync_cons= ole=3D1 reboot=3Dno"

I wrote ' vmstat -m | sort -k 2 -r' each 120 seconds, the latest one was as in at= tachment, panic was with the same fingerprint as the one with "rman_is= _region_manager" line already reported.

T= he scripts i ran in parallel generally were the same as attached in bug rep= ort, just a bit modified.
1) ./libexec.sh zfs_volstress_fast_4g (= this just creates new ZVOLs and instead of 2GB, it writes 4BG in each ZVOL = created dd if=3D/dev/zero)
2)=C2=A0 ./test_vm1_zvol_3gb.sh (= this loops commands: start first DomU, write 3GB in it's /tmp, restart = DomU, removes /tmp, repeat)
3) ./test_vm2_zvol_5_on_off.sh (this = loops: start second DomU, which has 5 disks attached, turn off DomU, repeat= )
4) monitoring, sleep 120 seconds, print vmstat | sort in se= rial output.

Around dom id 108, system started= to behave suspiciously, xl list showed DomUs created, but they did not rea= lly start up, script timeout-ed for ssh connection, no vnc. When i did xl d= estroy manually, and xl create, system panic happened.

=
I have log files for all serial output, if there is anything useful, i= can provide. On disk log files seems to loose latest messages due to crash= .


On Wed, Mar 2, 2022 at 3:57 PM Roger Pau Monn=C3=A9 <roger.pau@citrix.com> wrote:
On Wed, Mar 02, 20= 22 at 10:57:37AM +0200, Ze Dupsys wrote:
> Hello,
>
> I started using XEN on one pre-production machine (with aim to use lat= er in
> production) with 12.2, but since it experienced random crashes i updat= ed to
> 13.0 in hope that errors might disappear.
>
> I do not know how detailed should i write, so that this email is not t= oo
> long, but gives enough info.
>
> FreeBSD Dom0 is installed on ZFS, somewhat basic install, IPFW and rul= es
> for NATting are used. Zpool is composed of 2 mirrored disks. There is = a
> ZVOL volmode=3Ddev for each VM and VM's jail that are attached as = raw devices
> to DomU. At the moment DomUs contain FreeBSD, some 12.0 to 13.0, UFS, = with
> VNET jails, epairs all bridged to DomU's xn0 interface. On Dom0 i = have
> bridge interfaces, where DomU's are connected depending on their > "zone/network", those that have allowed outgoing connections= are NATted by
> IPFW on specific physical NIC and IP.

So from the traces on the ticket:

panic: pmap_growkernel: no memory to grow kernel
cpuid =3D 0
time =3D 1646123072
KDB: stack backtrace:
#0 0xffffffff80c57525 at kdb_backtrace+0x65
#1 0xffffffff80c09f01 at vpanic+0x181
#2 0xffffffff80c09d73 at panic+0x43
#3 0xffffffff81073eed at pmap_growkernel+0x27d
#4 0xffffffff80f2dae8 at vm_map_insert+0x248
#5 0xffffffff80f30249 at vm_map_find+0x549
#6 0xffffffff80f2bf76 at kmem_init+0x226
#7 0xffffffff80c73341 at vmem_xalloc+0xcb1
#8 0xffffffff80c72c3b at vmem_xalloc+0x5ab
#9 0xffffffff80f2bfce at kmem_init+0x27e
#10 0xffffffff80c73341 at vmem_xalloc+0xcb1
#11 0xffffffff80c72c3b at vmem_xalloc+0x5ab
#12 0xffffffff80c72646 at vmem_alloc+0x46
#13 0xffffffff80f2b616 at kmem_malloc_domainset+0x96
#14 0xffffffff80f21a2a at uma_prealloc+0x23a
#15 0xffffffff80f235de at sysctl_handle_uma_zone_cur+0xe2e
#16 0xffffffff80f1f6af at uma_set_align+0x8f
#17 0xffffffff82463362 at abd_borrow_buf_copy+0x22
Uptime: 4m9s


Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address=C2=A0 =C2=A0=3D 0x22710028
fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor r= ead data, page not present
instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff80c45892
stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe0096= 600930
frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe0096= 600930
code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base rx0, limit 0= xfffff, type 0x1b
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0
current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 1496 (devmatch)
trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12
panic: page fault
cpuid =3D 0
time =3D 1646123791
KDB: stack backtrace:
#0 0xffffffff80c57525 at kdb_backtrace+0x65
#1 0xffffffff80c09f01 at vpanic+0x181
#2 0xffffffff80c09d73 at panic+0x43
#3 0xffffffff8108b1a7 at trap+0xbc7
#4 0xffffffff8108b1ff at trap+0xc1f
#5 0xffffffff8108a85d at trap+0x27d
#6 0xffffffff81061b18 at calltrap+0x8
#7 0xffffffff80c62011 at rman_is_region_manager+0x241
#8 0xffffffff80c1a051 at sbuf_new_for_sysctl+0x101
#9 0xffffffff80c1949c at kernel_sysctl+0x43c
#10 0xffffffff80c19b13 at userland_sysctl+0x173
#11 0xffffffff80c1995f at sys___sysctl+0x5f
#12 0xffffffff8108baac at amd64_syscall+0x10c
#13 0xffffffff8106243e at Xfast_syscall+0xfe


Fatal trap 12: page fault while in kernel mode
cpuid =3D 1; apic id =3D 02
fault virtual address=C2=A0 =C2=A0=3D 0x68
fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor r= ead data, page not present
instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff824a599d
stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00b1= e27910
frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00b1= e279b0
code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base rx0, limit 0= xfffff, type 0x1b
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0
current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0 (xbbd7 taskq)
trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12
panic: page fault
cpuid =3D 1
time =3D 1646122723
KDB: stack backtrace:
#0 0xffffffff80c57525 at kdb_backtrace+0x65
#1 0xffffffff80c09f01 at vpanic+0x181
#2 0xffffffff80c09d73 at panic+0x43
#3 0xffffffff8108b1a7 at trap+0xbc7
#4 0xffffffff8108b1ff at trap+0xc1f
#5 0xffffffff8108a85d at trap+0x27d
#6 0xffffffff81061b18 at calltrap+0x8
#7 0xffffffff8248935a at dmu_read+0x2a
#8 0xffffffff82456a3a at zvol_geom_bio_strategy+0x2aa
#9 0xffffffff80a7f214 at xbd_instance_create+0xa394
#10 0xffffffff80a7b1ea at xbd_instance_create+0x636a
#11 0xffffffff80c6b1c1 at taskqueue_run+0x2a1
#12 0xffffffff80c6c4dc at taskqueue_thread_loop+0xac
#13 0xffffffff80bc7e3e at fork_exit+0x7e
#14 0xffffffff81062b9e at fork_trampoline+0xe
Uptime: 1h44m10s

This all look to me like Out of Memory conditions, can you check with
`top` whats going on with your memory?

Might also be helpful to record periodic calls to `vmstat -m | sort -k
2 -r` to try to figure out what's using so much memory.

Regards, Roger.
--00000000000086a47305d93f92e8-- --00000000000086a47505d93f92ea Content-Type: application/octet-stream; name="vmstat_before_crash.log" Content-Disposition: attachment; filename="vmstat_before_crash.log" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_l09tdsh80 Li9iaW4vbGliZXhlYy5zaDogPT09IFZNU1RBVCA9PT0KIE5GU0Qgc2Vzc2lvbiAgICAgMSAgICAg MUsgICAgICAgIDEgIDEwMjQKICAgIENBTSBxdWV1ZSAgICAxNiAgICAgNUsgICAgICAgNTcgIDE2 LDMyLDUxMgogICBDQU0gcGVyaXBoICAgICA2ICAgICAySyAgICAgICAyNCAgMTYsMzIsNjQsMTI4 LDI1NgogICAgIENBTSBwYXRoICAgIDExICAgICAxSyAgICAgICA4MiAgMzIKIE5GU0QgbGNrZmls ZSAgICAgMSAgICAgMUsgICAgICAgIDEgIDI1NgpDQU0gZGV2IHF1ZXVlICAgICA3ICAgICAxSyAg ICAgICAgNyAgNjQKICAgICAgQ0FNIFhQVCAgICAzMyAgICAgM0sgICAgICAgNzQgIDE2LDMyLDEy OCw1MTIsMTAyNCwyMDQ4LDY1NTM2Ck5GU0QgVjRjbGllbnQgICAgIDEgICAgIDFLICAgICAgICAx ICAyNTYKICAgICAgQ0FNIFNJTSAgICAgNyAgICAgMksgICAgICAgIDcgIDI1NgogICAgICAgICBU eXBlIEluVXNlIE1lbVVzZSBSZXF1ZXN0cyAgU2l6ZShzKQpDQU0gSS9PIFNjaGVkdWxlciAgICAg MiAgICAgMUsgICAgICAgIDIgIDEyOAogICAgICBDQU0gREVWICAgICA5ICAgIDE4SyAgICAgICAy MiAgMjA0OAogICAgICBDQU0gQ0NCICAgIDEwICAgIDIwSyAgODM3MTk1NiAgMjA0OAogICAgICBz b2xhcmlzIDU2OTMxNCAxMTQwNjVLIDQ5ODUyMzQwMCAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEy LDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYzODQsMzI3NjgsNjU1MzYKICAgIHN5c2N0bG9pZCAzMjQz MiAgMTgzM0sgICAgNTM3NTUgIDE2LDMyLDY0LDEyOAogICAgICAgZGV2YnVmIDE3OTYxIDM1MTI3 SyAgICAxODMwMCAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIs NjU1MzYKICAgICAgICAgeGJiZCAxMTk2OCAzODQ3MDZLICAgMjExMDc0ICA2NCwyMDQ4LDY1NTM2 CiAgICAgICBhY3BpY2EgIDk3NjggIDEwMjVLICAgMjU1MTA0ICAxNiwzMiw2NCwxMjgsMjU2LDM4 NCw1MTIsMTAyNCwyMDQ4CiAgICAgICAgICBidXMgIDEyOTYgICAxMTZLICAgIDI4NTMxICAxNiwz Miw2NCwxMjgsMjU2LDM4NCw0MDk2CiAgICAgICAgIHVtdHggIDExNTIgICAxNDRLICAgICAxMTUy ICAxMjgKICAgICAga2R0cmFjZSAgIDYyNSAgIDE0M0sgICAgMTUxMDQgIDY0LDI1NgogICAgICAg bW9kdWxlICAgNTA4ICAgIDY0SyAgICAgIDUwOCAgMTI4CiAgICAgICB4ZW5idXMgICAzOTcgICAg NjFLICAgIDI0ODU0ICAxNiwzMiw2NCwyNTYsNDA5NgogICAgICAgICAgb3NkICAgMzYzICAgICA3 SyAgICAgMTAwMyAgMTYsMzIsNjQsMTI4LDI1NgogICAgICAgbGlua2VyICAgMzU2ICAyMTE3SyAg ICAgMTQxMiAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYz ODQKICAgICAgICAga29iaiAgIDMzNiAgMTM0NEsgICAgICA2NzMgIDQwOTYKICAgICAgICAgcm1h biAgIDI0NCAgICAyOEsgICAgIDMwMTkgIDE2LDMyLDEyOAphdWRpdF9ldmNsYXNzICAgMjM2ICAg ICA4SyAgICAgIDI5NCAgMzIKICAgIHRhc2txdWV1ZSAgIDIxNiAgICAyNUsgICAgICA2NjkgIDE2 LDMyLDY0LDEyOCwyNTYKICAgICAgICAgIFVNQSAgIDIxNSAgICAgNUsgICAgICAyMTUgIDE2LDMy CiAgICAgICBERVZGUzMgICAyMDQgICAgNTFLICAgICAgNTI4ICAyNTYKICAgICAgc3VicHJvYyAg IDE2NSAgIDMyMEsgICAgMTM0MzUgIDM4NCw0MDk2CiAgICAgICBERVZGUzEgICAxNjEgICAxNjFL ICAgICAgNDkwICAxMDI0CiAgICAgIGl0aHJlYWQgICAxNTQgICAgMjZLICAgICAgNTg5ICAzMiwx MjgsMjU2CiAgICAgICBpZmFkZHIgICAxMzAgICAgMThLICAgICAgNTM2ICAxNiwzMiw2NCwxMjgs MjU2LDM4NCwyMDQ4LDQwOTYKIGV2ZW50aGFuZGxlciAgIDEzMCAgICAxMUsgICAgICAxMzAgIDY0 LDEyOAogICAgICAgICBHRU9NICAgMTI1ICAgIDIwSyAgICAgMTI5MCAgMTYsMzIsNjQsMTI4LDI1 Niw1MTIsMTAyNCwyMDQ4LDQwOTYsODE5MiwxNjM4NAogICAgICAgICBrZW52ICAgMTE2ICAgIDEz SyAgICAgIDExNiAgMTYsMzIsNjQsMTI4LDI1Niw4MTkyCiAgICAgICBidXMtc2MgICAxMTAgICAz NjVLICAgICA2MzA3ICAxNiwzMiw2NCwxMjgsMjU2LDM4NCw1MTIsMTAyNCwyMDQ4LDQwOTYsODE5 MiwxNjM4NCwzMjc2OCw2NTUzNgogICAgICAgS1RSQUNFICAgMTAwICAgIDEzSyAgICAgIDEwMCAg MTI4CiAgICAgICAgc2VsZmQgICAgOTUgICAgIDZLIDMyNTIxOTcwOCAgNjQKICAgICAgcHdkZGVz YyAgICA3MSAgICAgNUsgICAgMTMzNDEgIDY0CiAgICAgICBrcXVldWUgICAgNzEgICAgIDVLICAg IDEzMzQxICA2NAogICBERVZGU19SVUxFICAgIDU5ICAgIDIxSyAgICAgICA1OSAgNjQsMzg0CiAg ICBwcm9jLWFyZ3MgICAgNTcgICAgIDNLICAgICA5NjU2ICAxNiwzMiw2NCwxMjgsMjU2CiAgcHJp dmNtZF9kZXYgICAgNTcgICAgIDJLICA4NTk4NTQ4ICAxNiwzMiw2NCwyNTYsNTEyLDEwMjQsMjA0 OCw4MTkyLDE2Mzg0CiAgIGtzdGF0X2RhdGEgICAgNTYgICAgNTZLICAgICAgMjM5ICAxMDI0CnZt X2ZpY3RpdGlvdXMgICAgNTIgIDc0NTlLICAgICA1MjYwICA2NCwxMjgsMjU2LDEwMjQsODE5Miwz Mjc2OAogICAgICAgICAgVVNCICAgIDUxICAgIDgySyAgICAgICA1NyAgMTYsMzIsNjQsMTI4LDI1 NiwzODQsMTAyNCwyMDQ4LDQwOTYsODE5MgogICAgICAgVVNCZGV2ICAgIDQ5ICAgIDE1SyAgICAg ICA1MSAgMzIsNjQsMTI4LDI1NiwzODQsNDA5NgogICAgICAgIG1vdW50ICAgIDQ2ICAgICAySyAg ICAgIDEzNyAgMTYsMzIsNjQsMTI4LDI1NiwxMDI0CiAgICAgICAgaWZsaWIgICAgNDUgICAzMTZL ICAgICAgIDUzICAxNiw2NCwxMjgsNTEyLDEwMjQsMjA0OCw0MDk2LDgxOTIsMTYzODQsMzI3NjgK ICAgICAgIERFVkZTUCAgICA0MSAgICAgM0sgICAgIDMyMDIgIDY0CiAgICAgeGVuX2ludHIgICAg MzcgICAgIDlLICAgICAgIDM3ICAxMjgsNDA5NgogICAgICBsbHRhYmxlICAgIDMxICAgICA5SyAg ICAgIDI3OCAgMjU2LDM4NAogICAgICBzZXNzaW9uICAgIDMxICAgICA0SyAgICAgIDU3NyAgMTI4 CiAgICAgIGFjcGlzZW0gICAgMzEgICAgIDRLICAgICAgIDMxICAxMjgKICAgICAgICAgdGVtcCAg ICAzMCAgICAxN0sgICAgMzI4OTkgIDE2LDMyLDY0LDEyOCwyNTYsMzg0LDUxMiwxMDI0LDIwNDgs ODE5Miw2NTUzNgogICAgICAgIGxvY2tmICAgIDMwICAgICA0SyAgMzUxNTkyMyAgNjQsMTI4CiAg ICAgICBVbml0bm8gICAgMzAgICAgIDJLICAgICAgODkwICAzMiw2NAogICAgICBhY3BpZGV2ICAg IDMwICAgICAySyAgICAgICAzMCAgNjQKICB1ZnNfZGlyaGFzaCAgICAyNSAgICAxM0sgICAgICAz NTkgIDE2LDMyLDY0LDEyOCw1MTIKICAgICAgICB2dGJ1ZiAgICAyNCAgMTk2OEsgICAgICAgNDYg IDQwOTYKICAgICAgICAgIHR0eSAgICAyNCAgICAyNEsgICAgICAxNDEgIDEwMjQKICAgICAgICAg Y3JlZCAgICAyMyAgICAgNksgICAgIDIxOTggIDI1NgogICAgIHhlbnN0b3JlICAgIDIyICAgICA1 SyAgIDI1NDk3NyAgMTYsMzIsNjQsMTI4LDI1NiwzODQsNTEyLDQwOTYsMTYzODQKICBldGhlcl9t dWx0aSAgICAyMiAgICAgMksgICAgICAgMjIgIDE2LDMyLDY0LDEyOAogICBldnRjaG5fZGV2ICAg IDIwICAgIDI2SyAyODEwNjA1MSAgNjQsMjU2LDQwOTYKICAgIHBmc19ub2RlcyAgICAyMCAgICAg OEsgICAgICAgMjAgIDM4NAogICAgICAgc2VsZWN0ICAgIDIwICAgICAzSyAgICAgMTkyMiAgMTI4 CiAgIGd0YXNrcXVldWUgICAgMTggICAgMjZLICAgICAgIDE4ICAxNiwzMiwyNTYsODE5MgogICAg ICAgc29uYW1lICAgIDE4ICAgICAySyAgICAyNTA5OCAgMTYsMzIsMTI4CiAgICAgICAgICBwY2Ig ICAgMTcgIDE1NDVLICAgICAgNzEwICAxNiwzMiw2NCwxMDI0LDIwNDgKICAgICAgIHBsaW1pdCAg ICAxNyAgICAgNUsgICAgIDExMjkgIDI1NgogICAgIHJvdXRldGJsICAgIDE2ICAgICA0SyAgICAg ICAxNiAgMzIsNjQsMTI4LDM4NCwxMDI0CiAgICAgcGNpX2xpbmsgICAgMTYgICAgIDJLICAgICAg IDE2ICA2NCwxMjgKICAgICAgIGdudGRldiAgICAxNiAgICAgMUsgICAgICA5NTEgIDMyLDY0CiAg ICAgICAgREVWRlMgICAgMTYgICAgIDFLICAgICAgIDE3ICAxNiwxMjgKICAgIGluNl9tdWx0aSAg ICAxNSAgICAgMksgICAgICAgMTUgIDMyLDI1NgogICAgICAgICAgIHZ0ICAgIDEzICAgIDQxSyAg ICAgICAxMyAgMzg0LDQwOTYsMzI3NjgKICAgICAgICBoaG9vayAgICAxMyAgICAgNEsgICAgICAg MTMgIDI1NgogICAgICAgICAgcHRzICAgIDExICAgICAzSyAgICAgIDEyOCAgMjU2CiAgICAgdGVy bWluYWwgICAgMTEgICAgIDNLICAgICAgIDExICAyNTYKICAgICAgZGV2c3RhdCAgICAxMCAgICAy MUsgICAgICAgMTAgIDMyLDQwOTYKICAgIHRvcG9ub2RlcyAgICAxMCAgICAgMksgICAgICAgMTAg IDEyOAogICAgICAgIGNsb25lICAgICA5ICAgICAySyAgICAgICAgOSAgMTI4CiAgICAgICBrYmRt dXggICAgIDggICAgMjJLICAgICAgICA4ICAxNiwzODQsMTAyNCwyMDQ4LDE2Mzg0CiAgaW5wY2Jw b2xpY3kgICAgIDggICAgIDFLICAgICAxODAwICAzMgplbmNhcF9leHBvcnRfaG9zdCAgICAgOCAg ICAgMUsgICAgICAgIDggIDMyLDY0CiAgICAgICBwcmlzb24gICAgIDggICAgIDFLICAgICAgICA4 ICAxNiwzMgogICAgICAgaXNhZGV2ICAgICA4ICAgICAxSyAgICAgICAgOCAgMTI4CiAgICAgICAg aWZuZXQgICAgIDcgICAgMTNLICAgICAgIDY1ICAxMjgsMjA0OAogICAgICAgIGV2ZGV2ICAgICA3 ICAgICA3SyAgICAgICAgNyAgMTAyNAogICAgIGZpbGVjYXBzICAgICA3ICAgICAxSyAgICAgNTI5 NCAgMTYsMzIsNjQKICAgICAgIGlwNm5kcCAgICAgNyAgICAgMUsgICAgICAgNjUgIDY0LDI1Ngog ICAgICAgZmVlZGVyICAgICA3ICAgICAxSyAgICAgICAgNyAgMzIKICAgICBuZXh1c2RldiAgICAg NyAgICAgMUsgICAgICAgIDcgIDE2CiAgICAgICAgICBMUk8gICAgIDYgICAxMjBLICAgICAgICA2 ICA4MTkyLDMyNzY4CiAgICAgICAgIGlnbXAgICAgIDYgICAgIDFLICAgICAgIDY0ICAxMjgKICAg ICAgICAgIG1sZCAgICAgNiAgICAgMUsgICAgICAgNjQgIDEyOAogICAgICAgICAgQlBGICAgICA2 ICAgICAxSyAgICAgICA2NCAgMTI4CiAgICAgICAgbmhvcHMgICAgIDYgICAgIDFLICAgICAgICA2 ICAxMjgKICAgICBmaWxlZGVzYyAgICAgNSAgICAyMEsgICAgICA1NzkgIDQwOTYKICAgIHVmc19t b3VudCAgICAgNSAgICAxN0sgICAgICAgIDYgIDMyLDY0LDUxMiw0MDk2LDgxOTIKICAgICAgZXZl bnRmZCAgICAgNSAgICAgMUsgICAgICAzMDAgIDEyOAogICAgICAgIGxpbnV4ICAgICA1ICAgICAx SyAgICAgICAgNiAgMTYsNjQKICAgICAgICAgIE1DQSAgICAgNSAgICAgMUsgICAgICAgIDUgIDE2 LDMyLDEyOAogICAgICAgICBwZmlsICAgICA1ICAgICAxSyAgICAgICAgNSAgMTI4CiAgICAgICAg ICBtc2kgICAgIDUgICAgIDFLICAgICAgICA1ICAxMjgKICAgICAgICAgaW50ciAgICAgNCAgIDQ3 MksgICAgICAgIDQgIDY1NTM2CiAgICAgICAgICBzZW0gICAgIDQgICAxMDZLICAgICAgICA0ICAy MDQ4LDQwOTYKICAgICAgICAgIG1zZyAgICAgNCAgICAzMEsgICAgICAgIDQgIDIwNDgsNDA5Niw4 MTkyLDE2Mzg0CiAgICAgdmZzY2FjaGUgICAgIDMgIDIwNDlLICAgICAgICAzICAyNTYsNTEyCiAg ICAgICAgIHZtZW0gICAgIDMgIDEwNjBLICAgICAgIDE1ICA1MTIsMTAyNCwyMDQ4LDQwOTYsMTYz ODQsMzI3NjgKICAgICAgdGlkaGFzaCAgICAgMyAgIDE1OEsgICAgICAgIDMgIDIwNDgsMzI3NjgK ICAgICAgICAgcHJvYyAgICAgMyAgICAzM0sgICAgICAgIDMgIDEwMjQsMTYzODQKICAgICAgICAg c2J1ZiAgICAgMyAgICAgMksgICAyNTkzNjEgIDE2LDMyLDY0LDEyOCwyNTYsNTEyLDEwMjQsNDA5 NiwxNjM4NAogIHZub2RlbWFya2VyICAgICAzICAgICAySyAgICAgIDY3NiAgNTEyCiBsaW51eGN1 cnJlbnQgICAgIDMgICAgIDFLICAgICAgICA0ICA2NCwyNTYKICAgICAgIGNyeXB0byAgICAgMyAg ICAgMUsgICAgICAgIDMgIDY0LDI1NgogICBsb2dpbmNsYXNzICAgICAzICAgICAxSyAgICAgICAg MyAgNjQKICAgICAgICAgIHR1biAgICAgMyAgICAgMUsgICAgICAgIDMgIDMyCiAgICAgaW5fbXVs dGkgICAgIDMgICAgIDFLICAgICAgICAzICAyNTYKICAgICAgY2FsbG91dCAgICAgMiAgIDUxMksg ICAgICAgIDIgIAogICAgIG10eF9wb29sICAgICAyICAgIDcySyAgICAgICAgMiAgODE5Miw2NTUz NgogICAgICAgICAgcnBjICAgICAyICAgICA4SyAgICAgICAgMiAgNDA5NgogICAgICB1aWRpbmZv ICAgICAyICAgICA1SyAgICAgICA1NSAgMTI4LDQwOTYKICAgIGlwc2VjLXNhcSAgICAgMiAgICAg MksgICAgICAgIDIgIDEwMjQKICAgICAgICB0YXNrcSAgICAgMiAgICAgMUsgICAgICAgIDIgIDY0 LDEyOAogICAgc2ZzX25vZGVzICAgICAyICAgICAxSyAgICAgICAgMiAgMzg0CiAgICBhY3BpX3Bl cmYgICAgIDIgICAgIDFLICAgICAgICAyICAzODQKICAgICAgICAgY3B1cyAgICAgMiAgICAgMUsg ICAgICAgIDIgIDE2CiAgICAgIGpibG9ja3MgICAgIDIgICAgIDFLICAgICAgICAyICAxMjgsMjU2 CiAgICAgICBuZXdibGsgICAgIDEgIDIwNDhLICAgICAgMzg4ICAyNTYKICAgICBpbm9kZWRlcCAg ICAgMSAgMTAyNEsgICAgIDI3MjYgIDM4NAogICAgdWZzX3F1b3RhICAgICAxICAxMDI0SyAgICAg ICAgMSAgCiAgICAgdmZzX2hhc2ggICAgIDEgIDEwMjRLICAgICAgICAxICAKICAgICAgcGFnZWRl cCAgICAgMSAgIDI1NksgICAgICAxNjAgIDI1NgogICAgdm5ldF9kYXRhICAgICAxICAgMTY4SyAg ICAgICAgMSAgCiAgIHRmb19jY2FjaGUgICAgIDEgICAxMjhLICAgICAgICAxICAKICAgICBzeW5j YWNoZSAgICAgMSAgICA2OEsgICAgICAgIDEgIAogICAgIGFjcGl0YXNrICAgICAxICAgIDY0SyAg ICAgICAgMSAgNjU1MzYKICAgIGhvc3RjYWNoZSAgICAgMSAgICAzMksgICAgICAgIDEgIDMyNzY4 CiAgICAgICAgICBzaG0gICAgIDEgICAgMzJLICAgICAgICAxICAzMjc2OAogIHBmc192bmNhY2hl ICAgICAxICAgIDE2SyAgICAgICAgMSAgMTYzODQKICAgIGJtc2FmZW1hcCAgICAgMSAgICAgOEsg ICAgIDEwMjcgIDI1Niw4MTkyCiAgICAgICAgc2htZmQgICAgIDEgICAgIDhLICAgICAgICAxICA4 MTkyCiAgIGxvY2FsX2FwaWMgICAgIDEgICAgIDRLICAgICAgICAxICA0MDk2CiAgICAgIG1lbWRl c2MgICAgIDEgICAgIDRLICAgICAgICAxICA0MDk2CiAgICAgIGlvX2FwaWMgICAgIDEgICAgIDJL ICAgICAgICAxICAyMDQ4CiAgICAgICBzeXNjdGwgICAgIDEgICAgIDFLICAgICAxNjkyICAzMiw2 NAogICAgIGZyZWV3b3JrICAgICAxICAgICAxSyAgICAgIDMxOSAgMzIsMTI4CiAgICAgYWNwaWlu dHIgICAgIDEgICAgIDFLICAgICAgICAxICA2NAogICAgICB0Y3BmdW5jICAgICAxICAgICAxSyAg ICAgICAgMSAgNjQKICAgICAgICAgdm5ldCAgICAgMSAgICAgMUsgICAgICAgIDEgIDY0CiAgICAg ICAgICBwbWMgICAgIDEgICAgIDFLICAgICAgICAxICA2NAogICAgICBzb2Z0ZGVwICAgICAxICAg ICAxSyAgICAgICAgMSAgNTEyCmNoYWNoYTIwcmFuZG9tICAgICAxICAgICAxSyAgICAgICAgMSAg Mzg0CnZuZXRfZGF0YV9mcmVlICAgICAxICAgICAxSyAgICAgICAgMSAgMzIKICAgICAgUGVyLWNw dSAgICAgMSAgICAgMUsgICAgICAgIDEgIDMyCiAgICAgICB2bm9kZXMgICAgIDEgICAgIDFLICAg ICAgICAxICAyNTYKICAgICAgICAga3RscyAgICAgMSAgICAgMUsgICAgICAgIDEgIDI1NgogICAg ICAgICBjZGV2ICAgICAxICAgICAxSyAgICAgICAgMSAgMjU2CiAgICAgcDEwMDMuMWIgICAgIDEg ICAgIDFLICAgICAgICAxICAxNgogICAgIHBtY2hvb2tzICAgICAxICAgICAxSyAgICAgICAgMSAg MTI4CiAgICAgICBhcG1kZXYgICAgIDEgICAgIDFLICAgICAgICAxICAxMjgKICBpcHNlY3BvbGlj eSAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQKICBmcHVrZXJuX2N0eCAgICAgMSAgICAgMUsg ICAgICAgIDEgIDEwMjQKICAgICBzZWNhc3ZhciAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQK ICAgICAgIHNhaGVhZCAgICAgMSAgICAgMUsgICAgICAgIDEgIDEwMjQKICAgICAgICAgIGlvdiAg ICAgMCAgICAgMEsgIDg5MTE3ODcgIDE2LDY0LDEyOCwyNTYsMzg0LDUxMgogICAgc3lzY3RsdG1w ICAgICAwICAgICAwSyAgICAxMzQ2NyAgMTYsMzIsNjQsMjU2LDEwMjQKICAgICAganNlZ2RlcCAg ICAgMCAgICAgMEsgICAgIDQ4NjYgIDY0CiAgICAgIGphZGRyZWYgICAgIDAgICAgIDBLICAgICAy MjQwICAxMjgKICAgICAgIGRpcmFkZCAgICAgMCAgICAgMEsgICAgIDIyMjggIDEyOAogICAgICBq cmVtcmVmICAgICAwICAgICAwSyAgICAgMjIyMCAgMTI4CiAgICAgICBkaXJyZW0gICAgIDAgICAg IDBLICAgICAyMjA4ICAxMjgKICAgICAgZW50cm9weSAgICAgMCAgICAgMEsgICAgIDE2OTQgIDE2 LDMyLDY0LDQwOTYKICAgICBmcmVlZmlsZSAgICAgMCAgICAgMEsgICAgICA4MzcgIDY0CiAgICAg cHJvY2Rlc2MgICAgIDAgICAgIDBLICAgICAgNzQ0ICAxMjgKICAgICBpb2N0bG9wcyAgICAgMCAg ICAgMEsgICAgICA0NzggIDI1NiwzODQsMTAyNAogICAgIHNhdmVkaW5vICAgICAwICAgICAwSyAg ICAgIDQwOSAgMjU2CiAgICAgICAgIGpzZWcgICAgIDAgICAgIDBLICAgICAgNDA2ICAxMjgKICAg ICAgam5ld2JsayAgICAgMCAgICAgMEsgICAgICAzODcgIDEyOAogICAgIGZyZWVibGtzICAgICAw ICAgICAwSyAgICAgIDMxOCAgMjU2CiAgICAgICAgc2JkZXAgICAgIDAgICAgIDBLICAgICAgMjI4 ICA2NAogICAgcmFpZF9kYXRhICAgICAwICAgICAwSyAgICAgIDE5MiAgMzIsMTI4LDI1NgogICAg ICAgc3RhdGZzICAgICAwICAgICAwSyAgICAgICA2NSAgNDA5NgogICAgamZyZWVmcmFnICAgICAw ICAgICAwSyAgICAgICAxOSAgMTI4CiAgICAgZnJlZWZyYWcgICAgIDAgICAgIDBLICAgICAgIDE5 ICAxMjgKICAgICAgICBta2RpciAgICAgMCAgICAgMEsgICAgICAgMTIgIDEyOAogICAgIG1idWZf dGFnICAgICAwICAgICAwSyAgICAgICAgNyAgMzIKICAgIG5ld2RpcmJsayAgICAgMCAgICAgMEsg ICAgICAgIDYgIDY0Cm5hbWVpX3RyYWNrZXIgICAgIDAgICAgIDBLICAgICAgICA1ICAzMgogICBj bF9zYXZlYnVmICAgICAwICAgICAwSyAgICAgICAgNCAgNjQKICAgbWFkdF90YWJsZSAgICAgMCAg ICAgMEsgICAgICAgIDIgIDMyLDQwOTYKICAgICAgICBzaWdpbyAgICAgMCAgICAgMEsgICAgICAg IDEgIDY0Ci4vYmluL2xpYmV4ZWMuc2g6ID09PSBWTVNUQVQgPT09Cg== --00000000000086a47505d93f92ea--