From owner-freebsd-hackers@freebsd.org Sun Nov 1 18:48:48 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6BC84562AC; Sun, 1 Nov 2020 18:48:48 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CPQ745KXpz4S1n; Sun, 1 Nov 2020 18:48:48 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 81081263AE; Sun, 1 Nov 2020 18:48:48 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 59CCB3F557; Sun, 1 Nov 2020 19:48:46 +0100 (CET) From: "Kristof Provost" To: "Carsten =?utf-8?q?B=C3=A4cker?=" Cc: "John-Mark Gurney" , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Date: Sun, 01 Nov 2020 19:48:45 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 18:48:48 -0000 On 31 Oct 2020, at 16:06, Carsten Bäcker wrote: > Am 29.10.2020 um 22:39 schrieb Kristof Provost: >> >> On 29 Oct 2020, at 22:36, John-Mark Gurney wrote: >> >> Kristof Provost wrote this message on Thu, Oct 29, 2020 at 21:30 >> +0100: >> >> On 29 Oct 2020, at 16:30, Carsten Bäcker wrote: >> >> Sure, i am willing to help. >> >> Device is a Raspberry Pi 3B (not +), using the >> onboard-ethernet. >> I attached a bunch of information. >> >> Configuration is stripped down to the minimum required to >> reproduce >> the >> problem. >> >> Okay, so that???s an SMC2 LAN9514_ETH device. >> That???s the dev/usb/net/if_smsc.c driver. >> >> However, before we dig into that driver we should make sure >> that we???re >> really looking at a checksum problem. >> It???s entirely normal for TX checksums to be incorrect when >> logged on >> the sending host itself (if the hardware does checksum >> offloading the >> checksum in the packet sent to the MAC is incorrect, and left >> to the >> hardware to fix). >> >> So, can you confirm that the `"[bad udp cksum 0xe58a -> >> 0x482d!]` you >> reported was on an inbound packet? And let???s be safe: try >> to >> capture >> packets on a different machine. That???ll give us the true >> packet, after >> the hardware has done checksum calculations. >> >> One interesting point is that the smsc driver claims to not do TX >> offload, >> and a brief check shows that it doesn't allow a user to set the >> TXCSUM. >> >> It does seem to do RX offload, and the comments in the driver suggest >> some .. ahem, creative hardware behaviour: >> >> |/* The checksum appears to be simplistically calculated * over the >> udp/tcp header and data up to the end of the * eth frame. Which means >> if the eth frame is padded * the csum calculation is incorrectly >> performed over * the padding bytes as well. Therefore to be safe we * >> ignore the H/W csum on frames less than or equal to * 64 bytes. * * >> Ignore H/W csum for non-IPv4 packets. */ | >> >> It’s not impossible that there’s some more issues like that in >> the >> hardware, or even that there are different chip revisions out there. >> >> That also matches up with |ifconfig ue0 -rxcsum| fixing things. >> >> Best regards, >> Kristof >> > Hello again, > > just gave it another try, using "ping heise.de" (twice) from within > the > jail. > > This is the output of tcpdump: > # tcpdump -i pflog0 -vv > tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), > capture size 262144 bytes > 14:06:04.293996 IP (tos 0x0, ttl 64, id 19576, offset 0, flags [none], > proto UDP (17), length 54) >     PC-192-168-178-3.fritz.box.52421 > fritz.box.domain: [bad udp > cksum > 0xe589 -> 0x0a2c!] 64654+ A? heise.de. (26) > > Netstat: > # netstat -s -p udp | grep checksum >         4 with bad checksum >         0 with no checksum > > Packet-capture from the router is attached. > Okay… but that doesn’t actually demonstrate anything at all. It shows an outbound packet with an incorrect checksum. See what I’ve mentioned before about checksums potentially being calculated by hardware. Given that there’s been a reply to that DNS query the checksum clearly got set correctly after the packet was captured. Best, Kristof From owner-freebsd-hackers@freebsd.org Mon Nov 2 12:58:47 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F752451795 for ; Mon, 2 Nov 2020 12:58:47 +0000 (UTC) (envelope-from mr@freebsd.org) Received: from app.eeeit.de (app.eeeit.de [188.68.43.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CPtJk4jZZz4VpZ for ; Mon, 2 Nov 2020 12:58:46 +0000 (UTC) (envelope-from mr@freebsd.org) Received: from localhost (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: mike@reifenberger.com) by app.eeeit.de (Postfix) with ESMTPSA id B00D36EDA2 for ; Mon, 2 Nov 2020 13:58:38 +0100 (CET) Received: from ipbcc2b3a3.dynamic.kabel-deutschland.de (ipbcc2b3a3.dynamic.kabel-deutschland.de [188.194.179.163]) by app.eeeit.de (Horde Framework) with HTTPS; Mon, 02 Nov 2020 12:58:38 +0000 Date: Mon, 02 Nov 2020 12:58:38 +0000 Message-ID: <20201102125838.Horde.Bw5P8k6zMzKNtK0qEQkS1VD@app.eeeit.de> From: MR To: freebsd-hackers@freebsd.org Subject: Upgrade of PHP dependent ports User-Agent: Horde Application Framework 5 Content-Type: multipart/mixed; boundary="=_jNvvFAPy6N6qMg4sc91yHPk" MIME-Version: 1.0 X-Rspamd-Queue-Id: 4CPtJk4jZZz4VpZ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:197540, ipnet:188.68.32.0/20, country:DE] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2020 12:58:47 -0000 This message is in MIME format. --=_jNvvFAPy6N6qMg4sc91yHPk Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Hi, once I installed nextcloud when PHP 7.2 was the default. Now as PHP 7.4 is the default pkg upgrade would pull in php74 packages but not for nextcloud* packages. Similar situation is for python or postgres. I had postgres10-[client|server] installed and pkg upgrade tried to install postgres12-cliend which of course failed. Is there a official way to avoid this? How do others update if the default versions of the underlaying ports changes? Thanks in advance. greetings --- mike mr@freebsd.org -- greetings --- mike mr@freebsd.org --=_jNvvFAPy6N6qMg4sc91yHPk Content-Type: message/rfc822; name="Weitergeleitete Nachricht" Date: Sat, 31 Oct 2020 10:41:22 +0000 Message-ID: <20201031104122.Horde.lvxp0CjTzgHuo_4gHtXKmGJ@app.eeeit.de> From: MR To: ports@freebsd.org Subject: Upgrade of PHP dependent ports Received: from ipbcc2b3a3.dynamic.kabel-deutschland.de (ipbcc2b3a3.dynamic.kabel-deutschland.de [188.194.179.163]) by app.eeeit.de (Horde Framework) with HTTPS; Sat, 31 Oct 2020 10:41:22 +0000 User-Agent: Horde Application Framework 5 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes MIME-Version: 1.0 Content-Disposition: inline Hi, once I installed nextcloud when PHP 7.2 was the default. Now as PHP 7.4 is the default pkg upgrade would pull in php74 packages but not for nextcloud* packages. Similar situation is for python or postgres. I had postgres10-[client|server] installed and pkg upgrade tried to install postgres12-cliend which of course failed. Is there a official way to avoid this? How do others update if the default versions of the underlaying ports changes? Thanks in advance. CU --- Michael (mr@) -- greetings --- mike mr@freebsd.org --=_jNvvFAPy6N6qMg4sc91yHPk-- From owner-freebsd-hackers@freebsd.org Tue Nov 3 04:52:21 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 320B144A593; Tue, 3 Nov 2020 04:52:21 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQHT00j6jz4ZWN; Tue, 3 Nov 2020 04:52:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by mail-pf1-x42b.google.com with SMTP id c20so13127293pfr.8; Mon, 02 Nov 2020 20:52:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=hpsq7Qk3SOFKwSXnJKfMoDvg1nxqDivHRyFif22sQh0=; b=Gp3Jm41gDGv3cp4vVqRJHgvE0STgYUV04FThQuFgLZqex5OhaZJI2c2hn5v8qgrhPo msjCkoGMU37osk08ozyeFUIhbjIHR8fUHukE8kamTeu8iTy3vsKBnR1zm/a+ErkDtaQt BA78BSk9TPFBCGdlY6LRG19yfDyo1bZQcAtfVVdqk05KoncqyvyQpzSM5YKtZ3y7rnxl J8cEcyLkWRZ1CXTlYYUIMSV136OHJxJBgQN8/cpwzMfRR+Wk/bXsGcro+abs88FRNzrY RkRaUP4sIe1fi6//VesLAtcaMrugYqazC+SkSLDiYI9I8xYXRuMDKWb3P013EdydgCsy dA9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=hpsq7Qk3SOFKwSXnJKfMoDvg1nxqDivHRyFif22sQh0=; b=izI4gK5VNb1dsU5PY34xyMuGNsCyB+g7CFxGnmBb+b48/+7RbYrSC5PNevsfV7PWym 7b7MAj3HX1ZXyUJ4+NvjB2FbByFhfyfQMXMNbGxeeHeAuSkDON2IspRMMRgFSBA74+zK 6yn+Dh+iMcJR0jMfwHoIHL5T+M20obaknPQcRYHfro4Ethm4OcAslVS4Zj9iRsSsTi4j vsDYRwtl2td3gPs2lJP8dONmq6IKwvNpIVQPfQ8dwnyhLNtZLSJnDzvf5HKc8QkQGtr2 UyQ/8zObVLRTaJMaFXVbab9wH+e9jvcOp7Vp29a5i9wXS0ScbxvtIMj/8KGVz0rtlzV8 /L7w== X-Gm-Message-State: AOAM531HQrvapXYvMpUIuWUHMyMZX3BFLWszb+LzB9v6HzcrZ4tGeJli zcu0qfvKM0jWU5eM3qAuJnOi0Ocq9Ec= X-Google-Smtp-Source: ABdhPJwbgeVuz7qsym5Q/YazL/gA5mS5dvFeH0QecRxacxwTgxMXPlFtRyRNYC8mehTn4cm4Hq6KEQ== X-Received: by 2002:a65:6649:: with SMTP id z9mr11103680pgv.18.1604379138276; Mon, 02 Nov 2020 20:52:18 -0800 (PST) Received: from localhost ([210.108.244.139]) by smtp.gmail.com with ESMTPSA id g7sm513903pfi.16.2020.11.02.20.52.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Nov 2020 20:52:17 -0800 (PST) Date: Tue, 3 Nov 2020 13:52:15 +0900 From: YongHyeon PYUN To: Kristof Provost Cc: John-Mark Gurney , Carsten =?iso-8859-1?Q?B=E4cker?= , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Message-ID: <20201103045215.GA2524@michelle> References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> X-Rspamd-Queue-Id: 4CQHT00j6jz4ZWN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Gp3Jm41g; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of pyunyh@gmail.com designates 2607:f8b0:4864:20::42b as permitted sender) smtp.mailfrom=pyunyh@gmail.com X-Spamd-Result: default: False [-3.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.83)[-0.827]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.05)[-1.055]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.019]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::42b:from]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_CC(0.00)[funkthat.com,gmx.de,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers,freebsd-arm] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 04:52:21 -0000 On Thu, Oct 29, 2020 at 10:39:39PM +0100, Kristof Provost wrote: [...] > It does seem to do RX offload, and the comments in the driver suggest some > .. ahem, creative hardware behaviour: > > /* The checksum appears to be simplistically calculated > * over the udp/tcp header and data up to the end of the > * eth frame. Which means if the eth frame is padded > * the csum calculation is incorrectly performed over > * the padding bytes as well. Therefore to be safe we > * ignore the H/W csum on frames less than or equal to > * 64 bytes. > * > * Ignore H/W csum for non-IPv4 packets. > */ > > It’s not impossible that there’s some more issues like that in the hardware, > or even that there are different chip revisions out there. > > That also matches up with `ifconfig ue0 -rxcsum` fixing things. > I'm not sure where the root cause is but it seems smsc(4) needs improvement in RX checksum handling. Quick reading RX handler indicates RX checksum offloading does not work when: o IP datagram has IP options field o TCP/UDP packet was fragmented o UDP datagrams with 0 checksum value See fxp(4), gem(4) and hme(4) for implementation. It looks like smsc(4) uses the following RX format but I don't know actual RX format of H/W(no access to datasheet). <---------------------------- actlen --------------------------------------------------> <------------- pktlen ------------------------> rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial checksum(2 bytes) If the format above is correct smsc(4) needs more strict RX length validation(USB transfer size vs actual packet length). This indicates smsc(4) does not have to copy FCS bytes. Also given that H/W always appends FCS, it's also suspicious H/W does not correctly compute RX checksum for frames less than or equal to 64 bytes. I don't have H/W and some spare time to fix this though. :-( From owner-freebsd-hackers@freebsd.org Tue Nov 3 08:15:58 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 894BD45018F; Tue, 3 Nov 2020 08:15:58 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQMzy39W1z4lFR; Tue, 3 Nov 2020 08:15:58 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 2437336CF2; Tue, 3 Nov 2020 08:15:58 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id C46484D176; Tue, 3 Nov 2020 09:15:52 +0100 (CET) From: "Kristof Provost" To: "YongHyeon PYUN" Cc: "John-Mark Gurney" , "Carsten =?utf-8?q?B=C3=A4cker?=" , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Date: Tue, 03 Nov 2020 09:15:51 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <20201103045215.GA2524@michelle> References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 08:15:58 -0000 On 3 Nov 2020, at 5:52, YongHyeon PYUN wrote: > > I'm not sure where the root cause is but it seems smsc(4) needs > improvement in RX checksum handling. That=E2=80=99s my thinking as well, yes. I=E2=80=99m not quite sure how t= hough, = because the received packet should contain full and correct checksums = regardless of the hardware=E2=80=99s checksum handling, so pf shouldn=E2=80= =99t have = any issues updating checksums. Ah, except that the driver also copies the checksum into the mbuf=E2=80=99= s = csum_data field, and if that checksum is wrong we would indeed see pf = update an incorrect checksum, and we=E2=80=99d get forwarded packets with= bad = checksums. That also matches with the observation that the problem goes = away when rx checksum offload is disabled. > Quick reading RX handler > indicates RX checksum offloading does not work when: > o IP datagram has IP options field > o TCP/UDP packet was fragmented > o UDP datagrams with 0 checksum value > See fxp(4), gem(4) and hme(4) for implementation. > > It looks like smsc(4) uses the following RX format but I don't > know actual RX format of H/W(no access to datasheet). > Happily Microchip do publish the datasheet: = http://ww1.microchip.com/downloads/en//softwarelibrary/man-lan95xx-dat/la= n9512_lan9512i%20databook%20rev.%201.2%20(03-01-12).pdf It may indeed be the case that this happens when the IP (or TCP) header = has options, but without a clear capture (or ideally, access to a setup = to reproduce the problem) it=E2=80=99s hard to tell. The datasheet has the great benefit of existing and being published, but = it does lack a bit of detail when it comes to the RX checksum offload = engine. > <---------------------------- actlen = > --------------------------------------------------> > <------------- pktlen ------------------------> > rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial = > checksum(2 bytes) > > If the format above is correct smsc(4) needs more strict RX length > validation(USB transfer size vs actual packet length). This > indicates smsc(4) does not have to copy FCS bytes. > Also given that H/W always appends FCS, it's also suspicious H/W > does not correctly compute RX checksum for frames less than or > equal to 64 bytes. > > I don't have H/W and some spare time to fix this though. :-( > Sadly I don=E2=80=99t have this hardware either. Best regards, Kristof From owner-freebsd-hackers@freebsd.org Tue Nov 3 08:19:18 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5EFCB45038F; Tue, 3 Nov 2020 08:19:18 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQN3n10WZz4l5D; Tue, 3 Nov 2020 08:19:16 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 82E582600E0; Tue, 3 Nov 2020 09:19:07 +0100 (CET) Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) To: YongHyeon PYUN , Kristof Provost Cc: John-Mark Gurney , =?UTF-8?Q?Carsten_B=c3=a4cker?= , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> From: Hans Petter Selasky Message-ID: <46d08198-530c-cb4b-efa8-4edaf89471c1@selasky.org> Date: Tue, 3 Nov 2020 09:19:05 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20201103045215.GA2524@michelle> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CQN3n10WZz4l5D X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-2.10 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.02)[-1.023]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; ARC_NA(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.19)[0.185]; NEURAL_HAM_MEDIUM(-0.96)[-0.964]; FREEMAIL_TO(0.00)[gmail.com,FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; FREEMAIL_CC(0.00)[funkthat.com,gmx.de,freebsd.org]; MAILMAN_DEST(0.00)[freebsd-arm,freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 08:19:18 -0000 On 2020-11-03 05:52, YongHyeon PYUN wrote: > On Thu, Oct 29, 2020 at 10:39:39PM +0100, Kristof Provost wrote: > > [...] > >> It does seem to do RX offload, and the comments in the driver suggest some >> .. ahem, creative hardware behaviour: >> >> /* The checksum appears to be simplistically calculated >> * over the udp/tcp header and data up to the end of the >> * eth frame. Which means if the eth frame is padded >> * the csum calculation is incorrectly performed over >> * the padding bytes as well. Therefore to be safe we >> * ignore the H/W csum on frames less than or equal to >> * 64 bytes. >> * >> * Ignore H/W csum for non-IPv4 packets. >> */ >> >> It’s not impossible that there’s some more issues like that in the hardware, >> or even that there are different chip revisions out there. >> >> That also matches up with `ifconfig ue0 -rxcsum` fixing things. >> > > I'm not sure where the root cause is but it seems smsc(4) needs > improvement in RX checksum handling. Quick reading RX handler > indicates RX checksum offloading does not work when: > o IP datagram has IP options field > o TCP/UDP packet was fragmented > o UDP datagrams with 0 checksum value > See fxp(4), gem(4) and hme(4) for implementation. > > It looks like smsc(4) uses the following RX format but I don't > know actual RX format of H/W(no access to datasheet). > > <---------------------------- actlen --------------------------------------------------> > <------------- pktlen ------------------------> > rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial checksum(2 bytes) > > If the format above is correct smsc(4) needs more strict RX length > validation(USB transfer size vs actual packet length). This > indicates smsc(4) does not have to copy FCS bytes. > Also given that H/W always appends FCS, it's also suspicious H/W > does not correctly compute RX checksum for frames less than or > equal to 64 bytes. > > I don't have H/W and some spare time to fix this though. :-( I can review and test patches. Seems like there is need for a fix. --HPS From owner-freebsd-hackers@freebsd.org Tue Nov 3 08:46:12 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8F2C6450DF0; Tue, 3 Nov 2020 08:46:12 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQNfq6JYSz4nXC; Tue, 3 Nov 2020 08:46:11 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id A99212600E0; Tue, 3 Nov 2020 09:46:09 +0100 (CET) Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) From: Hans Petter Selasky To: YongHyeon PYUN , Kristof Provost Cc: freebsd-arm@freebsd.org, freebsd-hackers@freebsd.org References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> <46d08198-530c-cb4b-efa8-4edaf89471c1@selasky.org> Message-ID: <4dfaa9a3-c085-8466-a6e4-19f988b5ed3d@selasky.org> Date: Tue, 3 Nov 2020 09:46:08 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <46d08198-530c-cb4b-efa8-4edaf89471c1@selasky.org> Content-Type: multipart/mixed; boundary="------------812ED0F32473780BBAF6ADAC" Content-Language: en-US X-Rspamd-Queue-Id: 4CQNfq6JYSz4nXC X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-2.23 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-patch]; HAS_ATTACHMENT(0.00)[]; DMARC_NA(0.00)[selasky.org]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.17)[0.168]; NEURAL_HAM_LONG(-1.03)[-1.031]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.07)[-1.067]; FREEMAIL_TO(0.00)[gmail.com,FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm,freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 08:46:12 -0000 This is a multi-part message in MIME format. --------------812ED0F32473780BBAF6ADAC Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 2020-11-03 09:19, Hans Petter Selasky wrote: > It looks like smsc(4) uses the following RX format but I don't > know actual RX format of H/W(no access to datasheet). > > <---------------------------- actlen > --------------------------------------------------> >                 <------------- pktlen ------------------------> > rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial > checksum(2 bytes) Hi, I wonder if the checksum is zero, when not valid, and that we should check for this in the driver! Can you try this patch? Also enabling debugging in the SMSC driver would be useful. --HPS --------------812ED0F32473780BBAF6ADAC Content-Type: text/x-patch; charset=UTF-8; name="smsc.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="smsc.diff" Index: sys/dev/usb/net/if_smsc.c =================================================================== --- sys/dev/usb/net/if_smsc.c (revision 367268) +++ sys/dev/usb/net/if_smsc.c (working copy) @@ -1020,7 +1020,7 @@ pktlen -= 2; /* The checksum appears to be simplistically calculated - * over the udp/tcp header and data up to the end of the + * over the UDP/TCP header and data up to the end of the * eth frame. Which means if the eth frame is padded * the csum calculation is incorrectly performed over * the padding bytes as well. Therefore to be safe we @@ -1031,27 +1031,21 @@ */ if ((be16toh(eh->ether_type) == ETHERTYPE_IP) && (pktlen > ETHER_MIN_LEN)) { - struct ip *ip; + /* Copy the TCP/UDP checksum from the last 2 bytes + * of the transfer and put in the csum_data field. + */ + usbd_copy_out(pc, (off + pktlen), &m->m_pkthdr.csum_data, 2); - ip = (struct ip *)(eh + 1); - if ((ip->ip_v == IPVERSION) && - ((ip->ip_p == IPPROTO_TCP) || - (ip->ip_p == IPPROTO_UDP))) { + /* The data is copied in network order, but the + * csum algorithm in the kernel expects it to be + * in host network order. + */ + m->m_pkthdr.csum_data = ntohs(m->m_pkthdr.csum_data); + + if (m->m_pkthdr.csum_data != 0) { /* Indicate the UDP/TCP csum has been calculated */ m->m_pkthdr.csum_flags |= CSUM_DATA_VALID; - /* Copy the TCP/UDP checksum from the last 2 bytes - * of the transfer and put in the csum_data field. - */ - usbd_copy_out(pc, (off + pktlen), - &m->m_pkthdr.csum_data, 2); - - /* The data is copied in network order, but the - * csum algorithm in the kernel expects it to be - * in host network order. - */ - m->m_pkthdr.csum_data = ntohs(m->m_pkthdr.csum_data); - smsc_dbg_printf(sc, "RX checksum offloaded (0x%04x)\n", m->m_pkthdr.csum_data); } --------------812ED0F32473780BBAF6ADAC-- From owner-freebsd-hackers@freebsd.org Tue Nov 3 09:46:37 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 728144530AD for ; Tue, 3 Nov 2020 09:46:37 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQQ0Y0h9Kz4sTn for ; Tue, 3 Nov 2020 09:46:37 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 15832452EAE; Tue, 3 Nov 2020 09:46:37 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1423F452F0F for ; Tue, 3 Nov 2020 09:46:37 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-oln040092070101.outbound.protection.outlook.com [40.92.70.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQQ0T719Sz4s9n for ; Tue, 3 Nov 2020 09:46:33 +0000 (UTC) (envelope-from xtouqh@hotmail.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DX1X/YOFhubLHZ0ULV1oG6agPLGCz3u9W+VqaeCbyH5cVYeJpL/WOYTNgkXA2bttx0Dwpta9RTR9/gR/5GxdxWOcG0ukABIZR3A6DPWODym87LUXsipEC4scCWkkyjqQVu0OHWdMj4HK2j3v5UZHF9KTV/SynDSvU1VizI4lSlgth2vyJDaNQ5l/jWPiuAzW/YXsqXk9jrVbEe+LB5jR9zkIybqp/x02epyAKlfdqYAyrmxumo8bZgdudbqEyoou1JisI4ZL6dejMNgDPugtFliYt4NL/z3e+6WbT1tI4iQ4A7we8LOmkCTP8KDo6pXTsm0f19ZtJJfoiPGWIJqx4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ryACjASzv93anvtVxHD/1Mdh8KDquUR/efBPi7vJ1sc=; b=O3BwEsxr3Dvy19UpX/lhJvI1AA/szIw+B5UubuU7XWF0sSHkXBIdr6o+P3LnwrHdpB5cwE4eWAe/gxCzPQ/DU3uGjKM2bM+0MsbPGtWeAkvrnawFKrbCvnQHtTPxlKNxYc1YpwbI2tE/6LpxPzGmythwUvO3gMPU3OneypKGmYWuKrImt1hbA2JxMPg6ArzpWZJHJsmKIizWRvEpCt2G5U3GsgDSgsB2zG07ZaOvrGxaEkE02rYCiFMOHriTBiLSwV0VqbxczcX5CtqZllaN41MwUOU2ksDjbgyHZfN8QMIOMWJdFJyVkSItdArtg/hu76Uu/MfP9LO8XGaMGkwBlA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ryACjASzv93anvtVxHD/1Mdh8KDquUR/efBPi7vJ1sc=; b=NPmxd5YYL67DGEkzYsZlFonrQiROFz6wcG/k2cg6AlJEc9fy0uFXHYuO3Eia1oRRbvOkEeH2xOcMTvRG92Vkb+qyM7Gv+kPl6vlInxpfdqkI+WKwU+yvEVeJNbn3K4rRhCMa6mfXxx9S2jGXTRFJY1tZsw3MAv5PBtZPwf21qXz+wG9Vh7pcKudGlzWemIp9wQMK0Q00PLssg0m+mn5a47vzTc4Yl5CoNLovQYKuPZ1DIDVYG+RsoVj6j2iLux3Nts2VBvtlON9b3qz22RJA9y/HDDPgsPNh4i3tmFlfWQaKP6bxQuOY7OURCZOuQHjzdZVbevODsFYrxAyUM4eQ6g== Received: from VE1EUR03FT030.eop-EUR03.prod.protection.outlook.com (2a01:111:e400:7e09::51) by VE1EUR03HT197.eop-EUR03.prod.protection.outlook.com (2a01:111:e400:7e09::283) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15; Tue, 3 Nov 2020 09:46:30 +0000 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com (2a01:111:e400:7e09::4a) by VE1EUR03FT030.mail.protection.outlook.com (2a01:111:e400:7e09::66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15 via Frontend Transport; Tue, 3 Nov 2020 09:46:30 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:06742524CFD7474CDEF1BF96E8FBC056FEC68059409EDF7DC26F2F897CE878B8; UpperCasedChecksum:D1FC500943D7670BDEF3208FBB7852D35952339167E8AF47B0C867260940F1D1; SizeAsReceived:8474; Count:45 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8]) by AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8%7]) with mapi id 15.20.3499.032; Tue, 3 Nov 2020 09:46:30 +0000 To: hackers@freebsd.org From: xtouqh@hotmail.com Subject: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array Message-ID: Date: Tue, 3 Nov 2020 12:46:29 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TMN: [mhTGEAgC1a8fc9VBo9bVHJIhtDMC4Crc] X-ClientProxiedBy: FR2P281CA0007.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::17) To AM0PR06MB3986.eurprd06.prod.outlook.com (2603:10a6:208:b6::28) X-Microsoft-Original-Message-ID: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.6] (91.240.124.157) by FR2P281CA0007.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.10 via Frontend Transport; Tue, 3 Nov 2020 09:46:29 +0000 X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 45 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: 7cde9655-6740-4230-e121-08d87fdd5725 X-MS-TrafficTypeDiagnostic: VE1EUR03HT197: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZTSypa5SitzF2ICiBcEEb99Mc2MBNeRm3dF4fcRiO722PCsK6D845lZ6YV9X4Cdcuik+j93DEEakpphw3QwFFRQtGPl64BSJTkbWuBjtZAj7O2bnCxCFMNz8f3b7vT/VvuzUTQHR02E+wbsX2JHUMPcdsf+KggvFVKZ7xFZh7QmpMShfMbY8ZJUlwJJhGV5D3/YFfziiOOI4vVnYWWaLfd2Uf2g7v9KBX49mcsUJZf1MDvdCWrzv0hQ95aRyJwHQ X-MS-Exchange-AntiSpam-MessageData: Gh9LfbqxVc0P6Rx0BMbvlfkQyNpApN0z5Q7KHWTIDcAEFZYr4VlfdYz7NRG1Q/ytXGB0V+J+oleh4JnC83+eaVv9nIGs9KcLbeBv7MZlY3niF4cAUPb7NdQtngG9RhddR4VDhtmTKZ1+aqC6Q4qGDA== X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7cde9655-6740-4230-e121-08d87fdd5725 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2020 09:46:30.3013 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT030.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1EUR03HT197 X-Rspamd-Queue-Id: 4CQQ0T719Sz4s9n X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=NPmxd5YY; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of xtouqh@hotmail.com designates 40.92.70.101 as permitted sender) smtp.mailfrom=xtouqh@hotmail.com X-Spamd-Result: default: False [-2.88 / 15.00]; FREEMAIL_FROM(0.00)[hotmail.com]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/15]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[hotmail.com:+]; SUBJECT_HAS_EXCLAIM(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; NEURAL_HAM_SHORT(-1.21)[-1.211]; RCVD_IN_DNSWL_LOW(-0.10)[40.92.70.101:from]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.06)[-1.057]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; RCVD_COUNT_FIVE(0.00)[5]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.016]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FROM_NO_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.70.101:from]; MAILMAN_DEST(0.00)[hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 09:46:37 -0000 I'm looking at the current sys/compat/linux/linux_errno.c source, specifically this function: https://svnweb.freebsd.org/base/head/sys/compat/linux/linux_errno.c?revision=367132&view=markup#l24, and noticed that sizeof() usage there seems to be bogus as I mentioned in https://reviews.freebsd.org/D26974#inline-168811. What I'm wondering about is why KASSERT() is not triggering there -- I have added the following printf() right below KASSERT() showing that we indeed read outside of the array, and some of the linux_errtbl[i] values are 0: printf("%s:linux_errtbl[%d]=%d\n", __func__, i, linux_errtbl[i]); But, if I add the following check before printf(), it seems to be never true: if (linux_errtbl[i] == 0) printf("%s:linux_errtbl[%d]=%d\n", __func__, i, linux_errtbl[i]); So how come printed values are 0, but KASSERT(value != 0) and if (value == 0) are never true? I tried to reproduce this in simple userland test case, but the check seems to be working correctly there (though still reading outside of array if using sizeof() for final index). What am I missing here? From owner-freebsd-hackers@freebsd.org Tue Nov 3 09:55:36 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5C436453143 for ; Tue, 3 Nov 2020 09:55:36 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQQBw0W11z4sWJ for ; Tue, 3 Nov 2020 09:55:36 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.nyi.freebsd.org (Postfix) id 0FB74453387; Tue, 3 Nov 2020 09:55:36 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0F8314531C0 for ; Tue, 3 Nov 2020 09:55:36 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQQBt6VwTz4sT6 for ; Tue, 3 Nov 2020 09:55:34 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 4B1AB2600E0; Tue, 3 Nov 2020 10:55:27 +0100 (CET) Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array To: xtouqh@hotmail.com, hackers@freebsd.org References: From: Hans Petter Selasky Message-ID: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> Date: Tue, 3 Nov 2020 10:55:25 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CQQBt6VwTz4sT6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-2.08 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[selasky.org]; NEURAL_SPAM_SHORT(0.25)[0.246]; NEURAL_HAM_LONG(-1.01)[-1.009]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.02)[-1.019]; FREEMAIL_TO(0.00)[hotmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 09:55:36 -0000 On 2020-11-03 10:46, xtouqh@hotmail.com wrote: > I'm looking at the current sys/compat/linux/linux_errno.c source, > specifically this function: > https://svnweb.freebsd.org/base/head/sys/compat/linux/linux_errno.c?revision=367132&view=markup#l24, > and noticed that sizeof() usage there seems to be bogus as I mentioned > in https://reviews.freebsd.org/D26974#inline-168811. > > What I'm wondering about is why KASSERT() is not triggering there -- I > have added the following printf() right below KASSERT() showing that we > indeed read outside of the array, and some of the linux_errtbl[i] values > are 0: > > printf("%s:linux_errtbl[%d]=%d\n", __func__, i, linux_errtbl[i]); > > But, if I add the following check before printf(), it seems to be never > true: > > if (linux_errtbl[i] == 0) >     printf("%s:linux_errtbl[%d]=%d\n", __func__, i, linux_errtbl[i]); > > So how come printed values are 0, but KASSERT(value != 0) and if (value > == 0) are never true?  I tried to reproduce this in simple userland test > case, but the check seems to be working correctly there (though still > reading outside of array if using sizeof() for final index).  What am I > missing here? Did you enable INVARIANTS when compiling the kernel? --HPS From owner-freebsd-hackers@freebsd.org Tue Nov 3 10:00:08 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E6283453707 for ; Tue, 3 Nov 2020 10:00:08 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQQJ83y22z4t3T for ; Tue, 3 Nov 2020 10:00:08 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 85AE84533D6; Tue, 3 Nov 2020 10:00:08 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 856BE4534CB for ; Tue, 3 Nov 2020 10:00:08 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from EUR06-VI1-obe.outbound.protection.outlook.com (mail-vi1eur06olkn2045.outbound.protection.outlook.com [40.92.17.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQQJ714Krz4tDt for ; Tue, 3 Nov 2020 10:00:06 +0000 (UTC) (envelope-from xtouqh@hotmail.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DLBqHa5UFZMsQGmp5OR+SOoMPXopvgKF2q8Sk90c2KDi6SHss3hzvXgY6+ggozi7e+X7HuxmqxR2ICNNSUK6M0fzUc/amTlibqxyF6yCiWYRWP/H+qrhsG4evjBtcJE07UzrS6qwJC4QSrEUYCVa/P9nsDALROm2n+BLVMEIxfrAxcfmcdVbXZjSEtWid1EdREnOa9mo0XG+NvE2j166nayusRP6yVP/570jmCluk135IcPzqJ2bBYONKIOTTdPaZ1hAmBEmbGb+En1cwIFFFOEhLQip8LtW3hVkM130u8LjNC35hpnwpT2sTKuE3GQUfRA9whOMXExd9pIBWTvbsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5roKBYwYwa2BuxdLt3GhXKkiNHMB5vbWJsRslitLSeY=; b=dD9tI5Va20l/g73oJ5L64ZiaT2SGfy5gDbJmZnxbtNMDqbtEwnWm93jYDdz4aJ5ZLnD9As39gLzU2WeumHAvRF85cl/XqiDTsSAPQbycSAuc8uaHy0+21FaWH3pguDTkn+XzMZ+jzPUpR1oAxeNB+ssko6tfFmhXlQQOdaQ7UtitLlaLTvsGq+yFGkeBHlSPdV7FkK+3MDK8sQVgluz0PRQqZHMartc9rhozMeC61Tjiq5pN7oo1jr/RdIwtDQBg+NXQRj8PQfr9JvqXSxB6ugCEKvJCOpYTvhzak3cqLMf+kJZFC0AdtKBAtSIs6W/kVXaCGx0T2g58LCjonN1HNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5roKBYwYwa2BuxdLt3GhXKkiNHMB5vbWJsRslitLSeY=; b=qj07wZ+uZTD8FNYcqux0dtSwjBwH6aJh0VWf6BagVEK991JE0N709VIO/V5/NcqTXdzJv24iS/EPkVjKqbkr21khztY3J6F1vmot9JlbZvQov2UPhFRUzTnkwu7Rhhfl6cPGBma9hfP1GUwGYeuDwSCVXVuXrQZeb3/hj8Nx/Eeb0kTrPxEFH11bYM3L8KVwqwOKvHnFc/Ps1JTNerljjoUj4W2Gq/kWSV4tOBcqfX3EJ+DFLI22w6b9roFjjNWY36Q9K9cVph3T+tZeaMZRgO0jxcF6iPR+kjl/janAmermyNVlsB9HysT83fMZn7FdSEO2SwRJJO5caa9EEVgOxw== Received: from VI1EUR06FT022.eop-eur06.prod.protection.outlook.com (2a01:111:e400:fc37::4d) by VI1EUR06HT106.eop-eur06.prod.protection.outlook.com (2a01:111:e400:fc37::285) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15; Tue, 3 Nov 2020 10:00:03 +0000 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com (2a01:111:e400:fc37::43) by VI1EUR06FT022.mail.protection.outlook.com (2a01:111:e400:fc37::220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15 via Frontend Transport; Tue, 3 Nov 2020 10:00:03 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:848AB1F2FF93C57B631742E395C7A3072F9E45EA6CDF27C77C503CF95BB2DC05; UpperCasedChecksum:D726C02C7A6661E5D0D80727392AC9D95E58DDF859B881BB3DDE00C144470158; SizeAsReceived:8731; Count:47 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8]) by AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8%7]) with mapi id 15.20.3499.032; Tue, 3 Nov 2020 10:00:03 +0000 Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array To: Hans Petter Selasky , hackers@freebsd.org References: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> From: xtouqh@hotmail.com Message-ID: Date: Tue, 3 Nov 2020 13:00:02 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 In-Reply-To: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TMN: [U7R9O91sBpedanvsLEXPfCVv3Nvy+gT2] X-ClientProxiedBy: FR2P281CA0014.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::24) To AM0PR06MB3986.eurprd06.prod.outlook.com (2603:10a6:208:b6::28) X-Microsoft-Original-Message-ID: <8d0572ce-cbbe-e777-0ee3-50c80fe54d29@hotmail.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.6] (91.240.124.157) by FR2P281CA0014.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.10 via Frontend Transport; Tue, 3 Nov 2020 10:00:03 +0000 X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 47 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: 48bea9d5-77d0-4734-540e-08d87fdf3c0c X-MS-TrafficTypeDiagnostic: VI1EUR06HT106: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: E+bD45JtYPvjQ7yRV55n/O3vzIDYRZm2Il46dP86n8bnmQnZ6d3YIt95OjQz6A83hjT4WzWtyy8S3S1ZI1c/PS+LoSQxRkwzmkuRdYoQBlCpit9WRhvF8o9qBL+RhTuOzWISioJfUARWUHKa4GsKeItKUewwXAn+CQ+n5DAuUbLx9W/LeINzfiXhS6xAAD+3/5orJDHOoSukLvQqddaeyU8lsS+HeHEk+Ff03XuFqyE4wQQ7qzcA5gWIOlGql6Is X-MS-Exchange-AntiSpam-MessageData: Pn8GFBjhoSibztwiHkcaPg9PcCJOAUAX9RImigc7iRRd62dMtWoayHR2p0zeCN8FEEs/SW4NGMP+jfYJNs1gHaZu078mJ6gbyLd+6EDSJtGxI5ZOMZBwkE85Pb2ubt0itvptRcGDMA6Abw/i8O2fMA== X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-Network-Message-Id: 48bea9d5-77d0-4734-540e-08d87fdf3c0c X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2020 10:00:03.6949 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-AuthSource: VI1EUR06FT022.eop-eur06.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1EUR06HT106 X-Rspamd-Queue-Id: 4CQQJ714Krz4tDt X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=qj07wZ+u; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of xtouqh@hotmail.com designates 40.92.17.45 as permitted sender) smtp.mailfrom=xtouqh@hotmail.com X-Spamd-Result: default: False [-2.56 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[hotmail.com]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/15]; DKIM_TRACE(0.00)[hotmail.com:+]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.002]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.03)[-1.031]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; RCVD_COUNT_FIVE(0.00)[5]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; NEURAL_HAM_LONG(-1.03)[-1.029]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NO_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.92.17.45:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.17.45:from]; MAILMAN_DEST(0.00)[hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 10:00:09 -0000 Hans Petter Selasky wrote: > On 2020-11-03 10:46, xtouqh@hotmail.com wrote: >> I'm looking at the current sys/compat/linux/linux_errno.c source, >> specifically this function: >> https://svnweb.freebsd.org/base/head/sys/compat/linux/linux_errno.c?revision=367132&view=markup#l24, >> and noticed that sizeof() usage there seems to be bogus as I mentioned >> in https://reviews.freebsd.org/D26974#inline-168811. >> >> What I'm wondering about is why KASSERT() is not triggering there -- I >> have added the following printf() right below KASSERT() showing that >> we indeed read outside of the array, and some of the linux_errtbl[i] >> values are 0: >> >> printf("%s:linux_errtbl[%d]=%d\n", __func__, i, linux_errtbl[i]); >> >> But, if I add the following check before printf(), it seems to be >> never true: >> >> if (linux_errtbl[i] == 0) >>      printf("%s:linux_errtbl[%d]=%d\n", __func__, i, linux_errtbl[i]); >> >> So how come printed values are 0, but KASSERT(value != 0) and if >> (value == 0) are never true?  I tried to reproduce this in simple >> userland test case, but the check seems to be working correctly there >> (though still reading outside of array if using sizeof() for final >> index).  What am I missing here? > > Did you enable INVARIANTS when compiling the kernel? Yes, using amd64 GENERIC on -CURRENT, and that function itself is ifdef'ed INVARIANTS, so if it's executed, INVARIANTS are there when building the module and KASSERT() is not no-op. Though even without KASSERT(), simple if() is still not doing what I expect, so there must be something I'm missing. From owner-freebsd-hackers@freebsd.org Tue Nov 3 10:13:15 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A6944453F0D for ; Tue, 3 Nov 2020 10:13:15 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQQbH3zxpz4v7v for ; Tue, 3 Nov 2020 10:13:15 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.nyi.freebsd.org (Postfix) id 8699D453F0C; Tue, 3 Nov 2020 10:13:15 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 865F1453D34 for ; Tue, 3 Nov 2020 10:13:15 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQQbH09m5z4v3q for ; Tue, 3 Nov 2020 10:13:14 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id B425D2600E0; Tue, 3 Nov 2020 11:13:13 +0100 (CET) Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array To: xtouqh@hotmail.com, hackers@freebsd.org References: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> From: Hans Petter Selasky Message-ID: Date: Tue, 3 Nov 2020 11:13:12 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4CQQbH09m5z4v3q X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-2.08 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[selasky.org]; NEURAL_SPAM_SHORT(0.25)[0.249]; NEURAL_HAM_LONG(-1.01)[-1.006]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.02)[-1.023]; FREEMAIL_TO(0.00)[hotmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 10:13:15 -0000 Hi, Should: for (i = 1; i < sizeof(linux_errtbl); i++) { Be: for (i = 1; i < sizeof(linux_errtbl)/sizeof(linux_errtbl[0]); i++) { Or: for (i = 1; i < (int)nitems(linux_errtbl); i++) { --HPS From owner-freebsd-hackers@freebsd.org Tue Nov 3 10:17:58 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BF261453FCC for ; Tue, 3 Nov 2020 10:17:58 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQQhk2vZnz4vfd for ; Tue, 3 Nov 2020 10:17:58 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 63A03453EED; Tue, 3 Nov 2020 10:17:58 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63635453FCB for ; Tue, 3 Nov 2020 10:17:58 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04olkn0817.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe0c::817]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQQhh2WrZz4vPQ for ; Tue, 3 Nov 2020 10:17:55 +0000 (UTC) (envelope-from xtouqh@hotmail.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F65JvpUFnDVca8kZx0eORTj4jt60YVPTceGU6ScPtLpjSki6I2RMDhR6fH4JzqJ5LyTGlc3tp5xWxFMu5DZeuKnYvtb/ekzHXRJh7vc0Kdy68zgm5/XjhtUNwcRJoVi8guih+M6u98BSOS3weqMmEm0ZlRrU7h3YC9MWy1SAYzTMIDrnCw1jy1ta3GQBwBHobJycabmcZIelFwV671Ii8lQeTN4yQnt+/gy+axECB9EznnV31W5PKx1/Rbvya+vevb5ygm8NAYhuqdXM20FmaDFIQYndU2L0PwhYxpnYm/jTP5fBkEKhNeXoLcrGwDSTjiC/1icC/Pxhnqp86YMz7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=czBAACCZXurUSrERRW2xXAzQOyVeGqyCLDypITlh2nA=; b=kuTo29m6r3wz9ovj2vs7JZDpQmxeLTUUSAGFo+cLwtqUB8D5X5V38s5ErNlCl1ftec7aOrUbAlNCSdiQP//D9Ic0+9zirWGQF1L6a9j2dXzCL9TlhGK5hSG4l1ppyJNkeauLSY2ofWSnIHM0mSQFaljbjVRbm/ETUCzzhj8lpNy6N1JNM9Y8JSZvQLost540dSLSdWTvzApEZn74YEuCAOzrE5wrmkUMBopir8k1gn4OPaR0/qLFX2P88Vy+5bEuaw8Adsrt6FwW/VQKqPdBE2jAV+5zRwJh61KtM7JBi2TNrJ+CjGSgKR+L+fKXuzm47u3/UrX+RvAqvnzJCp0NOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=czBAACCZXurUSrERRW2xXAzQOyVeGqyCLDypITlh2nA=; b=GyER9gyZ93QGd/e88Zrq2UrPdlFuDcMbjdrJaUDof24mk9zWR2CCRZ9/qi+ID2olvVJqNYcvCSidVTECqFIcLsyjCbMahERj0csfGkfFdVKZGgtbnsV0f0+pQEJXntk9p8QDZ0iXtaE5pzhmgobk0KRZ7ntLoaLrqIuyo1hpm0UC9NfCrzmaGHTnsp1lPt9XYp0O8K0toqsk+Nh286/kkrSbMRDyI2X1jNR1GTWOpOqh+xgaHi3cspzVw+T0/7kIoavbR6MYfAdkQl0/TBJ8QOVwe2M8m1TvUUJU6QhGqvrxm/WGF4H4rhThOGHj3f0DJ0T2xKDuB/qNcyB+HjBFYg== Received: from DB3EUR04FT010.eop-eur04.prod.protection.outlook.com (2a01:111:e400:7e0c::49) by DB3EUR04HT048.eop-eur04.prod.protection.outlook.com (2a01:111:e400:7e0c::322) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15; Tue, 3 Nov 2020 10:17:52 +0000 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com (2a01:111:e400:7e0c::4e) by DB3EUR04FT010.mail.protection.outlook.com (2a01:111:e400:7e0c::146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15 via Frontend Transport; Tue, 3 Nov 2020 10:17:52 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:7F07B32EDA43AEC9E269B8923E0C6392087986DF1014BCA24434E99990E66DEE; UpperCasedChecksum:5523C737B84A02E2EF5DD94BA62C83EBD94E02AF8117186E3CDB8FBB3C7AF369; SizeAsReceived:8857; Count:47 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8]) by AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8%7]) with mapi id 15.20.3499.032; Tue, 3 Nov 2020 10:17:52 +0000 Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array To: Hans Petter Selasky , hackers@freebsd.org References: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> From: xtouqh@hotmail.com Message-ID: Date: Tue, 3 Nov 2020 13:17:51 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TMN: [YFBkGaOo6Wgap/Sti3zNHp5lgu07j0m5] X-ClientProxiedBy: FR2P281CA0009.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::19) To AM0PR06MB3986.eurprd06.prod.outlook.com (2603:10a6:208:b6::28) X-Microsoft-Original-Message-ID: <07e18962-5e69-b3cc-f3d0-dd388ff3f500@hotmail.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.6] (91.240.124.157) by FR2P281CA0009.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.10 via Frontend Transport; Tue, 3 Nov 2020 10:17:52 +0000 X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 47 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: b70c7a47-e82f-45f4-0130-08d87fe1b92f X-MS-TrafficTypeDiagnostic: DB3EUR04HT048: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ULSifWbJuK/KgybtFrQRxG/jvCrsu2WaXgOoYaEmGnu6zP1CMvbunXLy55qWP6ku+mMkUo5eID1GI6Kek9wqK9Bl1cSQdoa5UPAQ4vXkFkT1yYFGtnHLZNGKMu9/hstRIdeooOcpPKx+NqZ7w1TxCUPyVIVuUT25Z0lAb8zt+1I1famp8yLGWjqFLM9MXlsa2yeyVs8j3W2q68/yp9ZrbA== X-MS-Exchange-AntiSpam-MessageData: 3SqQtivVURjWz59BfgkAgYLdZLzt3xl6XnRa+rF0kk0r/vGs+qU6o0NAcX+0EwnG+TlH3AWUF7EGC4N8FbXsNhXrdfQXhg3mDUgAQZ1Zt7dH/K5MFi8n3AhNYfPdkpHtCMt0qrFnCUH/qVJo/or3pQ== X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-Network-Message-Id: b70c7a47-e82f-45f4-0130-08d87fe1b92f X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2020 10:17:52.6388 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-AuthSource: DB3EUR04FT010.eop-eur04.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3EUR04HT048 X-Rspamd-Queue-Id: 4CQQhh2WrZz4vPQ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=GyER9gyZ; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of xtouqh@hotmail.com designates 2a01:111:f400:fe0c::817 as permitted sender) smtp.mailfrom=xtouqh@hotmail.com X-Spamd-Result: default: False [-2.88 / 15.00]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.035]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; RCVD_COUNT_FIVE(0.00)[5]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[hotmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.03)[-1.031]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[hotmail.com:+]; SUBJECT_HAS_EXCLAIM(0.00)[]; FROM_NO_DN(0.00)[]; NEURAL_HAM_SHORT(-1.32)[-1.317]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; MAILMAN_DEST(0.00)[hackers]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 10:17:58 -0000 Hans Petter Selasky wrote: > Hi, > > Should: > >         for (i = 1; i < sizeof(linux_errtbl); i++) { > > Be: > >         for (i = 1; i < sizeof(linux_errtbl)/sizeof(linux_errtbl[0]); > i++) { > > > Or: > >         for (i = 1; i < (int)nitems(linux_errtbl); i++) { That's right, and I added the same comment in the review. My question is different though -- with the issue present, KASSERT() should have triggered (there are 0 values with incorrect indexes, added printf() confirms that) exposing the bug, but it does not -- WHY? -- I just want to understand what's going on. From owner-freebsd-hackers@freebsd.org Tue Nov 3 10:27:25 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BA6394541D3 for ; Tue, 3 Nov 2020 10:27:25 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQQvd3JnKz3RPl for ; Tue, 3 Nov 2020 10:27:25 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.nyi.freebsd.org (Postfix) id 6FE224541D2; Tue, 3 Nov 2020 10:27:25 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6FA5C453E21 for ; Tue, 3 Nov 2020 10:27:25 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQQvc2TJRz3RXF for ; Tue, 3 Nov 2020 10:27:24 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 237FD2602B8; Tue, 3 Nov 2020 11:27:23 +0100 (CET) Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array To: xtouqh@hotmail.com, hackers@freebsd.org References: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> From: Hans Petter Selasky Message-ID: <563d4b8d-bbf6-662f-6899-192f55342a86@selasky.org> Date: Tue, 3 Nov 2020 11:27:18 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CQQvc2TJRz3RXF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-3.41 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; TO_DN_NONE(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.07)[-1.067]; FREEMAIL_TO(0.00)[hotmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.034]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.01)[-1.009]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 10:27:25 -0000 On 2020-11-03 11:17, xtouqh@hotmail.com wrote: > Hans Petter Selasky wrote: >> Hi, >> >> Should: >> >>          for (i = 1; i < sizeof(linux_errtbl); i++) { >> >> Be: >> >>          for (i = 1; i < sizeof(linux_errtbl)/sizeof(linux_errtbl[0]); >> i++) { >> >> >> Or: >> >>          for (i = 1; i < (int)nitems(linux_errtbl); i++) { > > That's right, and I added the same comment in the review.  My question > is different though -- with the issue present, KASSERT() should have > triggered (there are 0 values with incorrect indexes, added printf() > confirms that) exposing the bug, but it does not -- WHY? -- I just want > to understand what's going on. Hi, You would need to run kgdb to dump the content of linux_errtbl and beyond to see what data is there. If the linux_errtbl is in the .text section then likely some other table follows after it, likely with non-zero data, so the KASSERT() doesn't trigger :-( --HPS From owner-freebsd-hackers@freebsd.org Tue Nov 3 11:16:23 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F075455056 for ; Tue, 3 Nov 2020 11:16:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQS071GC5z3VHS for ; Tue, 3 Nov 2020 11:16:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 295274551AB; Tue, 3 Nov 2020 11:16:23 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 27D71455398 for ; Tue, 3 Nov 2020 11:16:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQS054r1Xz3VKY for ; Tue, 3 Nov 2020 11:16:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 0A3BGEtt072261 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 3 Nov 2020 13:16:17 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 0A3BGEtt072261 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 0A3BGDdP072260; Tue, 3 Nov 2020 13:16:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 3 Nov 2020 13:16:13 +0200 From: Konstantin Belousov To: xtouqh@hotmail.com Cc: Hans Petter Selasky , hackers@freebsd.org Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array Message-ID: <20201103111613.GP2654@kib.kiev.ua> References: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4CQS054r1Xz3VKY X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-0.75 / 15.00]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-0.09)[-0.087]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-0.85)[-0.849]; R_SPF_SOFTFAIL(0.00)[~all:c]; NEURAL_SPAM_SHORT(0.19)[0.191]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; FREEMAIL_TO(0.00)[hotmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[hackers]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 11:16:23 -0000 On Tue, Nov 03, 2020 at 01:17:51PM +0300, xtouqh@hotmail.com wrote: > Hans Petter Selasky wrote: > > Hi, > > > > Should: > > > >         for (i = 1; i < sizeof(linux_errtbl); i++) { > > > > Be: > > > >         for (i = 1; i < sizeof(linux_errtbl)/sizeof(linux_errtbl[0]); > > i++) { > > > > > > Or: > > > >         for (i = 1; i < (int)nitems(linux_errtbl); i++) { > > That's right, and I added the same comment in the review. My question is > different though -- with the issue present, KASSERT() should have triggered > (there are 0 values with incorrect indexes, added printf() confirms that) > exposing the bug, but it does not -- WHY? -- I just want to understand > what's going on. I think this is a poster child for the current undefined behaviour treatment by compilers. You are accessing beyond array last element, and compiler can prove it, so it allowed to do anything. From owner-freebsd-hackers@freebsd.org Tue Nov 3 11:45:41 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BA99245584F for ; Tue, 3 Nov 2020 11:45:41 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQSdx3Pc5z3Wh8 for ; Tue, 3 Nov 2020 11:45:41 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 72E7E45584E; Tue, 3 Nov 2020 11:45:41 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 716D54557CE for ; Tue, 3 Nov 2020 11:45:41 +0000 (UTC) (envelope-from xtouqh@hotmail.com) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05olkn2073.outbound.protection.outlook.com [40.92.89.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQSdw1RSvz3WMm for ; Tue, 3 Nov 2020 11:45:39 +0000 (UTC) (envelope-from xtouqh@hotmail.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A/ywJ2BTbOsUHKJcqjphEDH+zdFYvAFRlsQSt7LaaFmQbB8DFe2tr1WGRdbzrlONy0R3M98JYkoUwwkyBT3tKf9696FteGJoazhYfmY9uVD3seZ1zTleMA1aC4tFjXGYPHkFL8W/H2FzeLXiX3Wd94WNwcwbugnabP3P7J12v1blhvNkUWOfTXTN4NP6khEIrUyQp1bpqbQZNcL9PW4rQi1IawlAZvfLHtOcIYrkvgmv6ar+fP3QNBzcZiz2Y6Y54IssSY8mEL98z+bD0TtgGLHIu7v9upX9jpMSGKAiPYtENbUidvYKuMrXJa8gf6v6T1yPA1Eam2h4Fxf5kG2gMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oa4OjwCU7ikz4HK+gkgaWaHCDkCmDzJqaFIjloN09jQ=; b=H15b+edCoR+ofir5WZJfnRcrQbtg7TQK3+Qlkm67PybPBEwVZm2EJkV+8SzceCerDTJry1CXGmB54mLgsMXMFOJQo9sMyOndey0dqaVkEHYmfSQ5SQB2IMy6qhpxT0KqJjYgDIA3zTVVBNBd6NfMt5XG1V61/emmaep2eXC6kV197ZQdWE30wDvzPljQHC0VorL5c0kcRx1R+BPZ7mvHp29j6hd82XiLJE131dNO2y5xPVQ2PJYp3gmPEYwFQPYqF/fveut1NQvpfbR7N5i5SJ6+NyEAVFvUn+brYpZZJtzHoRHpTrWJibMbU/6T+9HimB294bEYUlQA9Te8QCm6Ng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oa4OjwCU7ikz4HK+gkgaWaHCDkCmDzJqaFIjloN09jQ=; b=UbHQiwB5h7O7V1vTcYS+akzZ6pWs+BmIwMD1KjCUhakrGWTtKM6iVewmKUfr4wK2K1nHxu2xOWQKtR2QMQDdYIuSid6kSyNEPIzGbMFEipwNrAmbBS+LWIFFLwI0f5FC2pqIil29dPKPybjsVgDMh+dXe8Qiln6oZu4fPPOb8DZVAEjo5k9FB4qk1lRTcHsL5Iv4UUtN2fB3kcr7orMELh6LjBZ92MWdTw0HeRMX5gaG6RiTSCd59+J+9pNocZEN1M5x4i1Nkv7Rk9NBH/8GDKnueUdr7RlTV+ZcQlNkIIu4ocNQKyuNxIHDnuO7z4lqxyE1e59u/ifzmrOefmSA3w== Received: from DB8EUR05FT049.eop-eur05.prod.protection.outlook.com (2a01:111:e400:fc0f::43) by DB8EUR05HT126.eop-eur05.prod.protection.outlook.com (2a01:111:e400:fc0f::414) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15; Tue, 3 Nov 2020 11:45:37 +0000 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com (2a01:111:e400:fc0f::50) by DB8EUR05FT049.mail.protection.outlook.com (2a01:111:e400:fc0f::453) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3520.15 via Frontend Transport; Tue, 3 Nov 2020 11:45:37 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:E5D0965E4E5FABBA8806623986F90972824A3B5446350A2765BBEC0F1490B74F; UpperCasedChecksum:850E82D881C56A1393B31F7A38F5353DCA5132F6F2B2A1779FFEDF9660C2D193; SizeAsReceived:9038; Count:48 Received: from AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8]) by AM0PR06MB3986.eurprd06.prod.outlook.com ([fe80::759a:af46:6f2:8fb8%7]) with mapi id 15.20.3499.032; Tue, 3 Nov 2020 11:45:37 +0000 Subject: Re: KASSERT(val != 0) not triggering in linux_errno.c reading outside of array To: Konstantin Belousov Cc: Hans Petter Selasky , hackers@freebsd.org References: <77d2eef0-9cc8-aa39-6d28-a7fb41e233ac@selasky.org> <20201103111613.GP2654@kib.kiev.ua> From: xtouqh@hotmail.com Message-ID: Date: Tue, 3 Nov 2020 14:45:36 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 In-Reply-To: <20201103111613.GP2654@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TMN: [qwCrWIv4h9bMqP9PMZXUyf0aWPL7x9Ei] X-ClientProxiedBy: AM0PR04CA0069.eurprd04.prod.outlook.com (2603:10a6:208:1::46) To AM0PR06MB3986.eurprd06.prod.outlook.com (2603:10a6:208:b6::28) X-Microsoft-Original-Message-ID: <3e5a904c-5cd1-1657-1e2f-d1fd66a36f6b@hotmail.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.1.6] (91.240.124.157) by AM0PR04CA0069.eurprd04.prod.outlook.com (2603:10a6:208:1::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.19 via Frontend Transport; Tue, 3 Nov 2020 11:45:36 +0000 X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 48 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: 7f6db867-2ede-496c-a017-08d87fedfb53 X-MS-TrafficTypeDiagnostic: DB8EUR05HT126: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AXFleqQQRxYZ6KEkAC5dz1QSqinTKGfIsnogf58kcehTnzfeRh30BVBb6TdT45uNXpDQT/35xlleTcYRf85WzDxnMQj1rYfKgEqrJaPFsPjjHVBKdjK4L/XaevnFqTH0HriwFLe9iagcvw6Ni+HxcVc36FHxfpAFQnldg8C5/dHxaXNo9DbXQkdJyRPkqJNG36pL6mMTc1pGsmHfHCQk8Q== X-MS-Exchange-AntiSpam-MessageData: UMKcD1bbM3RQFsGvNYLRcjOkgC7qAxreyzJ5dRC9BW83FI69JZcOul1xpLuCjXgjx7zu/0Pu63SC+Ns52DyrOhcIoYZb2UBy7Br0/dMHy70AaXoNSAh9MjZRjIWJll4+RgycaSs1rP6bLQ3ZTZR2rg== X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7f6db867-2ede-496c-a017-08d87fedfb53 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2020 11:45:37.5922 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-AuthSource: DB8EUR05FT049.eop-eur05.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8EUR05HT126 X-Rspamd-Queue-Id: 4CQSdw1RSvz3WMm X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=UbHQiwB5; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of xtouqh@hotmail.com designates 40.92.89.73 as permitted sender) smtp.mailfrom=xtouqh@hotmail.com X-Spamd-Result: default: False [-2.36 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[hotmail.com]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/15]; DKIM_TRACE(0.00)[hotmail.com:+]; SUBJECT_HAS_EXCLAIM(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; NEURAL_HAM_SHORT(-0.80)[-0.804]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; NEURAL_HAM_MEDIUM(-1.04)[-1.041]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; RCVD_COUNT_FIVE(0.00)[5]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.01)[-1.013]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NO_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[40.92.89.73:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.89.73:from]; MAILMAN_DEST(0.00)[hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 11:45:41 -0000 Konstantin Belousov wrote: > On Tue, Nov 03, 2020 at 01:17:51PM +0300, xtouqh@hotmail.com wrote: >> Hans Petter Selasky wrote: >>> Hi, >>> >>> Should: >>> >>>         for (i = 1; i < sizeof(linux_errtbl); i++) { >>> >>> Be: >>> >>>         for (i = 1; i < sizeof(linux_errtbl)/sizeof(linux_errtbl[0]); >>> i++) { >>> >>> >>> Or: >>> >>>         for (i = 1; i < (int)nitems(linux_errtbl); i++) { >> >> That's right, and I added the same comment in the review. My question is >> different though -- with the issue present, KASSERT() should have triggered >> (there are 0 values with incorrect indexes, added printf() confirms that) >> exposing the bug, but it does not -- WHY? -- I just want to understand >> what's going on. > > I think this is a poster child for the current undefined behaviour treatment > by compilers. You are accessing beyond array last element, and compiler can > prove it, so it allowed to do anything. Makes sense, thank you. Just for the record, if I compile userland test case with -O2, it behaves the same; previously I was building without explicitly specifying optimization level, and it did "work". From owner-freebsd-hackers@freebsd.org Tue Nov 3 16:42:38 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DC60645D883 for ; Tue, 3 Nov 2020 16:42:38 +0000 (UTC) (envelope-from google@oprs.eu) Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQbDY6K4Qz476R for ; Tue, 3 Nov 2020 16:42:37 +0000 (UTC) (envelope-from google@oprs.eu) Received: by mail-ot1-f45.google.com with SMTP id i18so11825202ots.0 for ; Tue, 03 Nov 2020 08:42:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ovejuvWU1Xyj0QM9Z2xJjaVsmSqGDJEpxlY2K0/Broc=; b=Zm7VIWhFtZgua2JsdYDnk99+3hnk7/wgESRFq7PSsAK9BeY7FrsiMQ0v1p0DM/4j/P QstzJfaXkmv2paT5nqZeBI1elvErUbqtWV1yrXJ6Uk9Lvg/RmSDM+y2RsoxflS40LgZ3 pI1kvcbYdHAozPJJ5bNmQqbFYYV30dNvM8usRfAygcK2aDnSdq41QUE8s6QYozfoFOOS 9MZbh6R4DXRacK96VFsYK8EbSqtW+eB7vc1hTtRzWUhpx1XTCikZnli2RlKBps+EZEp4 Zk2PcZsV5Ifl1OLYU1QxcSHdQUCGVvUGRqWRFWN6tNb4PpuYEC0ml45X1T0uRs/SAzV2 ftEg== X-Gm-Message-State: AOAM531/4JWwQvF+OoLbvVSKKok/hYEQiuxN+v5sCLxOiaPJDVlxej4b //SzIcvJ7zV4h1KuJkw1pfr/aOuWy2boDj2O/RoXsDccPwbPwHo3 X-Google-Smtp-Source: ABdhPJw/f8gx6PUBOVQulaHBIlhIzti1cjBHZj0fPuOLmTtHF3uFtA5LsQ/mv/ZcdSHs5Xgg4BADMZti9zeN0AuPz2g= X-Received: by 2002:a9d:61:: with SMTP id 88mr15152272ota.109.1604421755946; Tue, 03 Nov 2020 08:42:35 -0800 (PST) MIME-Version: 1.0 From: Olivier Piras Date: Tue, 3 Nov 2020 17:42:24 +0100 Message-ID: Subject: page fault in ifunit() To: freebsd-hackers@freebsd.org Content-Type: multipart/mixed; boundary="000000000000d5a6f205b3368c41" X-Rspamd-Queue-Id: 4CQbDY6K4Qz476R X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of google@oprs.eu has no SPF policy when checking 209.85.210.45) smtp.mailfrom=google@oprs.eu X-Spamd-Result: default: False [5.28 / 15.00]; RWL_MAILSPIKE_GOOD(0.00)[209.85.210.45:from]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_SHORT(-0.28)[-0.276]; MIME_BASE64_TEXT(0.10)[]; FORGED_SENDER(0.30)[freebsd@oprs.eu,google@oprs.eu]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:+,5:~,6:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[freebsd@oprs.eu,google@oprs.eu]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain,text/x-csrc]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; AUTH_NA(1.00)[]; MIME_BAD_ATTACHMENT(1.60)[c]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.83)[0.831]; DMARC_NA(0.00)[oprs.eu]; NEURAL_SPAM_LONG(0.83)[0.829]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.45:from]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 16:42:38 -0000 --000000000000d5a6f205b3368c41 Content-Type: text/plain; charset="UTF-8" Hi, I'm getting page fault errors on CURRENT while trying to get a pointer to an ifnet instance with ifunit(). My code is running as a KLD, I've attached a stripped down version that reproduces the issue (just load the module, brace yourself, and open /dev/test). The weird thing is that the page fault only seems to occur on DEVFS operations (test_dev_open() in that instance). The very same call to ifunit() works as expected in the init part of the module. Apparently someone ran into the same problem a couple of years ago [1]. Unfortunately they didn't follow up on the issue. [2] leads me to believe that it has something to do with VNET, so I tried to play with CURVNET_SET() / CURVNET_RESTORE() in an effort to set a context for the current VNET instance, to no avail (I have to admit it was mostly a shot in the dark though, as VNET isn't something I'm particularly familiar with). I'm having a hard time figuring out why the problem only occurs in the context of DEVFS operations; I'm currently reading up on VNET. Any pointers would be greatly appreciated. Regards, -Olivier. [1] http://freebsd.1045724.x6.nabble.com/Page-fault-inside-ifunit-ref-FreeBSD12-0-CURRENT-td6254170.html [2] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176992 --000000000000d5a6f205b3368c41 Content-Type: text/plain; charset="US-ASCII"; name="trap12.txt" Content-Disposition: attachment; filename="trap12.txt" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_kh26v6202 RmF0YWwgdHJhcCAxMjogcGFnZSBmYXVsdCB3aGlsZSBpbiBrZXJuZWwgbW9kZQpjcHVpZCA9IDI7 IGFwaWMgaWQgPSAwMgpmYXVsdCB2aXJ0dWFsIGFkZHJlc3MgICA9IDB4MjgKZmF1bHQgY29kZSAg ICAgICAgICAgICAgPSBzdXBlcnZpc29yIHJlYWQgZGF0YSwgcGFnZSBub3QgcHJlc2VudAppbnN0 cnVjdGlvbiBwb2ludGVyICAgICA9IDB4MjA6MHhmZmZmZmZmZjgwY2JiZWVlCnN0YWNrIHBvaW50 ZXIgICAgICAgICAgID0gMHgyODoweGZmZmZmZTAwMjQ4YjY2MzAKZnJhbWUgcG9pbnRlciAgICAg ICAgICAgPSAweDI4OjB4ZmZmZmZlMDAyNDhiNjY2MApjb2RlIHNlZ21lbnQgICAgICAgICAgICA9 IGJhc2UgMHgwLCBsaW1pdCAweGZmZmZmLCB0eXBlIDB4MWIKICAgICAgICAgICAgICAgICAgICAg ICAgPSBEUEwgMCwgcHJlcyAxLCBsb25nIDEsIGRlZjMyIDAsIGdyYW4gMQpwcm9jZXNzb3IgZWZs YWdzICAgICAgICA9IGludGVycnVwdCBlbmFibGVkLCByZXN1bWUsIElPUEwgPSAwCmN1cnJlbnQg cHJvY2VzcyAgICAgICAgID0gMjEyMyAoY2F0KQp0cmFwIG51bWJlciAgICAgICAgICAgICA9IDEy CnBhbmljOiBwYWdlIGZhdWx0CmNwdWlkID0gMgp0aW1lID0gMTYwNDQxNTA0MgpLREI6IHN0YWNr IGJhY2t0cmFjZToKIzAgMHhmZmZmZmZmZjgwYzBiMGM1IGF0IGtkYl9iYWNrdHJhY2UrMHg2NQoj MSAweGZmZmZmZmZmODBiYmYxYmIgYXQgdnBhbmljKzB4MTdiCiMyIDB4ZmZmZmZmZmY4MGJiZjAz MyBhdCBwYW5pYysweDQzCiMzIDB4ZmZmZmZmZmY4MTA5MDkxMSBhdCB0cmFwX2ZhdGFsKzB4Mzkx CiM0IDB4ZmZmZmZmZmY4MTA5MDk2ZiBhdCB0cmFwX3BmYXVsdCsweDRmCiM1IDB4ZmZmZmZmZmY4 MTA4ZmZiNiBhdCB0cmFwKzB4Mjg2CiM2IDB4ZmZmZmZmZmY4MTA2N2I0OCBhdCBjYWxsdHJhcCsw eDgKIzcgMHhmZmZmZmZmZjgyODNlMmMyIGF0IHRlc3RfZGV2X29wZW4rMHgyMgojOCAweGZmZmZm ZmZmODBhNzhiNTAgYXQgZGV2ZnNfb3BlbisweDEyMAojOSAweGZmZmZmZmZmODEyNDc2OTUgYXQg Vk9QX09QRU5fQVBWKzB4NzUKIzEwIDB4ZmZmZmZmZmY4MGM5ZGUxNyBhdCB2bl9vcGVuX3Zub2Rl KzB4MWI3CiMxMSAweGZmZmZmZmZmODBjOWRhNjMgYXQgdm5fb3Blbl9jcmVkKzB4M2EzCiMxMiAw eGZmZmZmZmZmODBjOTVlMTMgYXQga2Vybl9vcGVuYXQrMHgyMTMKIzEzIDB4ZmZmZmZmZmY4MTA5 MTRjNyBhdCBhbWQ2NF9zeXNjYWxsKzB4Mzg3CiMxNCAweGZmZmZmZmZmODEwNjg0NmUgYXQgZmFz dF9zeXNjYWxsX2NvbW1vbisweGY4ClVwdGltZTogMjBtMjZzCkR1bXBpbmcgMjg0IG91dCBvZiAz ODc4IE1COi4uNiUuLjEyJS4uMjMlLi4zNCUuLjQ1JS4uNTElLi42MiUuLjc0JS4uODUlLi45NiUK Cl9fY3VydGhyZWFkICgpIGF0IC91c3IvaG9tZS9vcHJzL2dpdC9mcmVlYnNkL3N0YWJsZS8xMi9z eXMvYW1kNjQvaW5jbHVkZS9wY3B1X2F1eC5oOjU1CjU1ICAgICAgL3Vzci9ob21lL29wcnMvZ2l0 L2ZyZWVic2Qvc3RhYmxlLzEyL3N5cy9hbWQ2NC9pbmNsdWRlL3BjcHVfYXV4Lmg6IE5vIHN1Y2gg ZmlsZSBvciBkaXJlY3RvcnkuCihrZ2RiKSBidAojMCAgX19jdXJ0aHJlYWQgKCkgYXQgL3Vzci9o b21lL29wcnMvZ2l0L2ZyZWVic2Qvc3RhYmxlLzEyL3N5cy9hbWQ2NC9pbmNsdWRlL3BjcHVfYXV4 Lmg6NTUKIzEgIGRvYWR1bXAgKHRleHRkdW1wPTxvcHRpbWl6ZWQgb3V0PikgYXQgL3Vzci9ob21l L29wcnMvZ2l0L2ZyZWVic2Qvc3RhYmxlLzEyL3N5cy9rZXJuL2tlcm5fc2h1dGRvd24uYzozNzEK IzIgIDB4ZmZmZmZmZmY4MGJiZWRkNSBpbiBrZXJuX3JlYm9vdCAoaG93dG89MjYwKSBhdCAvdXNy L2hvbWUvb3Bycy9naXQvZnJlZWJzZC9zdGFibGUvMTIvc3lzL2tlcm4va2Vybl9zaHV0ZG93bi5j OjQ1MQojMyAgMHhmZmZmZmZmZjgwYmJmMjEzIGluIHZwYW5pYyAoZm10PTxvcHRpbWl6ZWQgb3V0 PiwgYXA9PG9wdGltaXplZCBvdXQ+KSBhdCAvdXNyL2hvbWUvb3Bycy9naXQvZnJlZWJzZC9zdGFi bGUvMTIvc3lzL2tlcm4va2Vybl9zaHV0ZG93bi5jOjg4MAojNCAgMHhmZmZmZmZmZjgwYmJmMDMz IGluIHBhbmljIChmbXQ9PHVuYXZhaWxhYmxlPikgYXQgL3Vzci9ob21lL29wcnMvZ2l0L2ZyZWVi c2Qvc3RhYmxlLzEyL3N5cy9rZXJuL2tlcm5fc2h1dGRvd24uYzo4MDcKIzUgIDB4ZmZmZmZmZmY4 MTA5MDkxMSBpbiB0cmFwX2ZhdGFsIChmcmFtZT0weGZmZmZmZTAwMjQ4YjY1NzAsIGV2YT00MCkg YXQgL3Vzci9ob21lL29wcnMvZ2l0L2ZyZWVic2Qvc3RhYmxlLzEyL3N5cy9hbWQ2NC9hbWQ2NC90 cmFwLmM6OTIxCiM2ICAweGZmZmZmZmZmODEwOTA5NmYgaW4gdHJhcF9wZmF1bHQgKGZyYW1lPTB4 ZmZmZmZlMDAyNDhiNjU3MCwgdXNlcm1vZGU9PG9wdGltaXplZCBvdXQ+LCBzaWdubz08b3B0aW1p emVkIG91dD4sIHVjb2RlPTxvcHRpbWl6ZWQgb3V0PikKICAgIGF0IC91c3IvaG9tZS9vcHJzL2dp dC9mcmVlYnNkL3N0YWJsZS8xMi9zeXMvYW1kNjQvYW1kNjQvdHJhcC5jOjczOQojNyAgMHhmZmZm ZmZmZjgxMDhmZmI2IGluIHRyYXAgKGZyYW1lPTB4ZmZmZmZlMDAyNDhiNjU3MCkgYXQgL3Vzci9o b21lL29wcnMvZ2l0L2ZyZWVic2Qvc3RhYmxlLzEyL3N5cy9hbWQ2NC9hbWQ2NC90cmFwLmM6NDA1 CiM4ICA8c2lnbmFsIGhhbmRsZXIgY2FsbGVkPgojOSAgMHhmZmZmZmZmZjgwY2JiZWVlIGluIGlm dW5pdCAobmFtZT0weGZmZmZmZmZmODI4M2U0YmYgImVtMCIpIGF0IC91c3IvaG9tZS9vcHJzL2dp dC9mcmVlYnNkL3N0YWJsZS8xMi9zeXMvbmV0L2lmLmM6MjQ4MgojMTAgMHhmZmZmZmZmZjgyODNl MmMyIGluIHRlc3RfZGV2X29wZW4gKCkgZnJvbSAuL3Rlc3RfZGV2LmtvCiMxMSAweGZmZmZmZmZm ODBhNzhiNTAgaW4gZGV2ZnNfb3BlbiAoYXA9MHhmZmZmZmUwMDI0OGI2NzgwKSBhdCAvdXNyL2hv bWUvb3Bycy9naXQvZnJlZWJzZC9zdGFibGUvMTIvc3lzL2ZzL2RldmZzL2RldmZzX3Zub3BzLmM6 MTE0MQojMTIgMHhmZmZmZmZmZjgxMjQ3Njk1IGluIFZPUF9PUEVOX0FQViAodm9wPTB4ZmZmZmZm ZmY4MWFmNWYxMCA8ZGV2ZnNfc3BlY29wcz4sIGE9MHhmZmZmZmUwMDI0OGI2NzgwKSBhdCB2bm9k ZV9pZi5jOjQ2NwojMTMgMHhmZmZmZmZmZjgwYzlkZTE3IGluIFZPUF9PUEVOICh2cD0weGZmZmZm ODAwMmVhYjk1YTAsIG1vZGU9MSwgY3JlZD0weGZmZmZmODAwMDQ0Mjc3MDAsIHRkPTxvcHRpbWl6 ZWQgb3V0PiwgZnA9MHhmZmZmZjgwMDA0NDdlZGMwKSBhdCAuL3Zub2RlX2lmLmg6MTk2CiMxNCB2 bl9vcGVuX3Zub2RlICh2cD0weGZmZmZmODAwMmVhYjk1YTAsIGZtb2RlPTEsIGNyZWQ9MHhmZmZm ZjgwMDA0NDI3NzAwLCB0ZD08b3B0aW1pemVkIG91dD4sIGZwPTB4ZmZmZmY4MDAwNDQ3ZWRjMCkK ICAgIGF0IC91c3IvaG9tZS9vcHJzL2dpdC9mcmVlYnNkL3N0YWJsZS8xMi9zeXMva2Vybi92ZnNf dm5vcHMuYzozOTQKIzE1IDB4ZmZmZmZmZmY4MGM5ZGE2MyBpbiB2bl9vcGVuX2NyZWQgKG5kcD0w eGZmZmZmZTAwMjQ4YjY5NTgsIGZsYWdwPTB4ZmZmZmZlMDAyNDhiNmE5NCwgY21vZGU9MCwgdm5f b3Blbl9mbGFncz08b3B0aW1pemVkIG91dD4sIGNyZWQ9MHhmZmZmZjgwMDA0NDI3NzAwLAogICAg ZnA9MHhmZmZmZjgwMDA0NDdlZGMwKSBhdCAvdXNyL2hvbWUvb3Bycy9naXQvZnJlZWJzZC9zdGFi bGUvMTIvc3lzL2tlcm4vdmZzX3Zub3BzLmM6MzAxCiMxNiAweGZmZmZmZmZmODBjOTVlMTMgaW4g a2Vybl9vcGVuYXQgKHRkPTB4ZmZmZmY4MDAwNDU4NjAwMCwgZmQ9LTEwMCwgcGF0aD0weDdmZmZm ZmZmZWU2YyA8ZXJyb3I6IENhbm5vdCBhY2Nlc3MgbWVtb3J5IGF0IGFkZHJlc3MgMHg3ZmZmZmZm ZmVlNmM+LAogICAgcGF0aHNlZz1VSU9fVVNFUlNQQUNFLCBmbGFncz0xLCBtb2RlPTxvcHRpbWl6 ZWQgb3V0PikgYXQgL3Vzci9ob21lL29wcnMvZ2l0L2ZyZWVic2Qvc3RhYmxlLzEyL3N5cy9rZXJu L3Zmc19zeXNjYWxscy5jOjExMTQKIzE3IDB4ZmZmZmZmZmY4MTA5MTRjNyBpbiBzeXNjYWxsZW50 ZXIgKHRkPTB4ZmZmZmY4MDAwNDU4NjAwMCkKICAgIGF0IC91c3IvaG9tZS9vcHJzL2dpdC9mcmVl YnNkL3N0YWJsZS8xMi9zeXMvYW1kNjQvYW1kNjQvLi4vLi4va2Vybi9zdWJyX3N5c2NhbGwuYzox NDQKIzE4IGFtZDY0X3N5c2NhbGwgKHRkPTB4ZmZmZmY4MDAwNDU4NjAwMCwgdHJhY2VkPTApIGF0 IC91c3IvaG9tZS9vcHJzL2dpdC9mcmVlYnNkL3N0YWJsZS8xMi9zeXMvYW1kNjQvYW1kNjQvdHJh cC5jOjExNjMKIzE5IDxzaWduYWwgaGFuZGxlciBjYWxsZWQ+CiMyMCAweDAwMDAwMDA4MDAzOTc5 MWEgaW4gPz8gKCkKQmFja3RyYWNlIHN0b3BwZWQ6IENhbm5vdCBhY2Nlc3MgbWVtb3J5IGF0IGFk ZHJlc3MgMHg3ZmZmZmZmZmU2MDgKKGtnZGIpCg== --000000000000d5a6f205b3368c41 Content-Type: application/octet-stream; name=Makefile Content-Disposition: attachment; filename=Makefile Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_kh23qcqc0 U1JDUz10ZXN0X2Rldi5jCktNT0Q9dGVzdF9kZXYKQ0ZMQUdTPS1nCgouaW5jbHVkZSA8YnNkLmtt b2QubWs+Cg== --000000000000d5a6f205b3368c41-- From owner-freebsd-hackers@freebsd.org Tue Nov 3 16:54:02 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5BBF145D659; Tue, 3 Nov 2020 16:54:02 +0000 (UTC) (envelope-from Lowell@Be-Well.Ilk.Org) Received: from be-well.ilk.org (be-well.ilk.org [23.30.133.173]) by mx1.freebsd.org (Postfix) with ESMTP id 4CQbTj4g62z47WW; Tue, 3 Nov 2020 16:54:01 +0000 (UTC) (envelope-from Lowell@Be-Well.Ilk.Org) Received: from lowell-desk.be-well.ilk.org (router.lan [172.30.250.2]) by be-well.ilk.org (Postfix) with ESMTP id BB06B33C25; Tue, 3 Nov 2020 11:53:55 -0500 (EST) Received: by lowell-desk.be-well.ilk.org (Postfix, from userid 1147) id 02189163251E; Tue, 3 Nov 2020 11:53:54 -0500 (EST) From: Lowell Gilbert To: YongHyeon PYUN Cc: Kristof Provost , John-Mark Gurney , Carsten =?iso-8859-1?Q?B=E4cker?= , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> Date: Tue, 03 Nov 2020 11:53:54 -0500 In-Reply-To: <20201103045215.GA2524@michelle> (YongHyeon PYUN's message of "Tue, 3 Nov 2020 13:52:15 +0900") Message-ID: <44v9embcv1.fsf@Be-Well.Ilk.Org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CQbTj4g62z47WW X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of Lowell@Be-Well.Ilk.Org has no SPF policy when checking 23.30.133.173) smtp.mailfrom=Lowell@Be-Well.Ilk.Org X-Spamd-Result: default: False [0.28 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; FORGED_SENDER(0.30)[freebsd-lists@Be-Well.Ilk.Org,Lowell@Be-Well.Ilk.Org]; FREEMAIL_CC(0.00)[FreeBSD.org,funkthat.com,gmx.de,freebsd.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.64)[-0.635]; NEURAL_HAM_LONG(-0.46)[-0.460]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[Ilk.Org]; AUTH_NA(1.00)[]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.07)[0.072]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:7922, ipnet:23.30.0.0/15, country:US]; FROM_NEQ_ENVFROM(0.00)[freebsd-lists@Be-Well.Ilk.Org,Lowell@Be-Well.Ilk.Org]; MAILMAN_DEST(0.00)[freebsd-arm,freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 16:54:02 -0000 YongHyeon PYUN writes: > On Thu, Oct 29, 2020 at 10:39:39PM +0100, Kristof Provost wrote: > > [...] > >> It does seem to do RX offload, and the comments in the driver suggest so= me >> .. ahem, creative hardware behaviour: >>=20 >> /* The checksum appears to be simplistically calculated >> * over the udp/tcp header and data up to the end of the >> * eth frame. Which means if the eth frame is padded >> * the csum calculation is incorrectly performed over >> * the padding bytes as well. Therefore to be safe we >> * ignore the H/W csum on frames less than or equal to >> * 64 bytes. >> * >> * Ignore H/W csum for non-IPv4 packets. >> */ >>=20 >> It=92s not impossible that there=92s some more issues like that in the h= ardware, >> or even that there are different chip revisions out there. >>=20 >> That also matches up with `ifconfig ue0 -rxcsum` fixing things. >>=20 > > I'm not sure where the root cause is but it seems smsc(4) needs > improvement in RX checksum handling. Quick reading RX handler > indicates RX checksum offloading does not work when: > o IP datagram has IP options field > o TCP/UDP packet was fragmented > o UDP datagrams with 0 checksum value > See fxp(4), gem(4) and hme(4) for implementation. > > It looks like smsc(4) uses the following RX format but I don't > know actual RX format of H/W(no access to datasheet).=20=20=20=20=20=20= =20=20=20=20=20=20=20 > > <---------------------------- actlen ------------------------------------= --------------> > <------------- pktlen ------------------------> > rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial ch= ecksum(2 bytes) > > If the format above is correct smsc(4) needs more strict RX length > validation(USB transfer size vs actual packet length). This > indicates smsc(4) does not have to copy FCS bytes. > Also given that H/W always appends FCS, it's also suspicious H/W > does not correctly compute RX checksum for frames less than or > equal to 64 bytes. For the 89530 at least, the data sheet states that checksum offload and padding stripping are mutually exclusive. I would assume that enabling both of them will produce incorrect results. I would *not* assume that all chips covered by this driver behave the same way, although it's likely.=20 ---------- From owner-freebsd-hackers@freebsd.org Tue Nov 3 19:14:44 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 89D63460E82; Tue, 3 Nov 2020 19:14:44 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQfc32nmxz4JT7; Tue, 3 Nov 2020 19:14:42 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 0A3JEe3i035970 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 3 Nov 2020 11:14:40 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 0A3JEdop035969; Tue, 3 Nov 2020 11:14:39 -0800 (PST) (envelope-from jmg) Date: Tue, 3 Nov 2020 11:14:39 -0800 From: John-Mark Gurney To: Kristof Provost Cc: Carsten =?iso-8859-1?Q?B=E4cker?= , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Message-ID: <20201103191439.GV31099@funkthat.com> Mail-Followup-To: Kristof Provost , Carsten =?iso-8859-1?Q?B=E4cker?= , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Tue, 03 Nov 2020 11:14:40 -0800 (PST) X-Rspamd-Queue-Id: 4CQfc32nmxz4JT7 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.24 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.02)[-1.024]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.47)[-0.473]; NEURAL_HAM_MEDIUM(-0.94)[-0.943]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-hackers,freebsd-arm]; FREEMAIL_CC(0.00)[gmx.de,freebsd.org] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 19:14:44 -0000 Kristof Provost wrote this message on Sun, Nov 01, 2020 at 19:48 +0100: > On 31 Oct 2020, at 16:06, Carsten Bäcker wrote: > > Packet-capture from the router is attached. > > > Okay??? but that doesn???t actually demonstrate anything at all. > It shows an outbound packet with an incorrect checksum. See what I???ve > mentioned before about checksums potentially being calculated by > hardware. > Given that there???s been a reply to that DNS query the checksum clearly > got set correctly after the packet was captured. Actually, isn't that odd that the outbound checksum is wrong? Since the chip doesn't advertise (or it shouldn't) tx checksum handling, FreeBSD's stack should have computed the checksum. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@freebsd.org Tue Nov 3 19:20:06 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 17E13460CFC; Tue, 3 Nov 2020 19:20:06 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQfkF6x4bz4Jmp; Tue, 3 Nov 2020 19:20:05 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id B72D714554; Tue, 3 Nov 2020 19:20:05 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id AE4A64E6A8; Tue, 3 Nov 2020 20:20:03 +0100 (CET) From: "Kristof Provost" To: "John-Mark Gurney" Cc: "Carsten =?utf-8?q?B=C3=A4cker?=" , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Date: Tue, 03 Nov 2020 20:20:02 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <20201103191439.GV31099@funkthat.com> References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103191439.GV31099@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 19:20:06 -0000 On 3 Nov 2020, at 20:14, John-Mark Gurney wrote: > Kristof Provost wrote this message on Sun, Nov 01, 2020 at 19:48 > +0100: >> On 31 Oct 2020, at 16:06, Carsten Bäcker wrote: >>> Packet-capture from the router is attached. >>> >> Okay??? but that doesn???t actually demonstrate anything at all. >> It shows an outbound packet with an incorrect checksum. See what >> I???ve >> mentioned before about checksums potentially being calculated by >> hardware. >> Given that there???s been a reply to that DNS query the checksum >> clearly >> got set correctly after the packet was captured. > > Actually, isn't that odd that the outbound checksum is wrong? Since > the chip doesn't advertise (or it shouldn't) tx checksum handling, > FreeBSD's stack should have computed the checksum. > Bear in mind that this capture was made with pflog, not the usual BTF_MTAP() in the driver. That means it was captured before the stack does checksum calculations. Just before it, in fact. See ip_output(). The checksum calculation is pretty much immediately after the pfil hook. Best regards, Kristof From owner-freebsd-hackers@freebsd.org Tue Nov 3 23:36:13 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0266D467904 for ; Tue, 3 Nov 2020 23:36:13 +0000 (UTC) (envelope-from jeremie.galarneau@efficios.com) Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQmPm2FFSz4cKD for ; Tue, 3 Nov 2020 23:36:12 +0000 (UTC) (envelope-from jeremie.galarneau@efficios.com) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 970E62D0E5E for ; Tue, 3 Nov 2020 18:36:11 -0500 (EST) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 7419oMpVzrNf for ; Tue, 3 Nov 2020 18:36:11 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 5F94F2D1031 for ; Tue, 3 Nov 2020 18:36:11 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 5F94F2D1031 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1604446571; bh=rv4MYGoTAe2dw3/rUVvL82ndCY8AuSSEeh8DjT4+VlU=; h=MIME-Version:From:Date:Message-ID:To; b=hoenW3EtI/y9m0YDjtTJlRR9s6BtxYNjHgkUrl7UlfYxfHMFkk5WDqes/vNhnlAj5 eyOPbWD7l4cZssgEAsbm3wUaNKvZb3qElm92zu8YsWEsGhVGxMB2pbTe527fAn1BoA 9QtfLuFnmTFozraqOLCLAznKpEy1Z7+m5TPwg1cKr3ghXNoyDL4fOOTH3dGnFrvaVd R7LAuKUFzuESdDmee21QmuqqHxWdchzMQUr3Ansn/lKS2sB9CDaSiDvg9NdjGnPCU1 9HdPos33mVMsiLcoXALJGngV5Y2A8sgRWE+SNTRD/lqJIU5Vk2S2PQpkVZfMWgjdQc fqb243q9CNPkA== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id yPMsrHo14moC for ; Tue, 3 Nov 2020 18:36:11 -0500 (EST) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mail.efficios.com (Postfix) with ESMTPSA id 2340C2D1030 for ; Tue, 3 Nov 2020 18:36:11 -0500 (EST) Received: by mail-wr1-f52.google.com with SMTP id s9so20177219wro.8 for ; Tue, 03 Nov 2020 15:36:11 -0800 (PST) X-Gm-Message-State: AOAM533lbHDBDXAOMXDeoV9sDLAMVZliNohEHMb+0VYGQfa4w3a3Q39L 1/ww1NlaYiNEX9uAy0BpGvN5GN23VQa6sLtYzqk= X-Google-Smtp-Source: ABdhPJzhIye0Ji/qFru0p7U6r5PCvuRqI1Lt/ziR9ltFr06ONmz8qJCHkEAY0eoHLcVnh6l3VWEKVCqG+wkxvngR2q8= X-Received: by 2002:adf:de89:: with SMTP id w9mr28492013wrl.212.1604446570120; Tue, 03 Nov 2020 15:36:10 -0800 (PST) MIME-Version: 1.0 From: =?UTF-8?Q?J=C3=A9r=C3=A9mie_Galarneau?= Date: Tue, 3 Nov 2020 18:35:34 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: poll() POLLHUP behaviour inconsistency To: freebsd-hackers@freebsd.org Cc: Michael Jeanson Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CQmPm2FFSz4cKD X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=efficios.com header.s=default header.b=hoenW3Et; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (mx1.freebsd.org: domain of jeremie.galarneau@efficios.com designates 167.114.26.124 as permitted sender) smtp.mailfrom=jeremie.galarneau@efficios.com X-Spamd-Result: default: False [-2.35 / 15.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[efficios.com:s=default]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.05)[-1.045]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_MEDIUM(-0.97)[-0.972]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[efficios.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[efficios.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.003]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(1.67)[subject]; ASN(0.00)[asn:16276, ipnet:167.114.0.0/17, country:FR]; RCVD_COUNT_SEVEN(0.00)[7]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 23:36:13 -0000 Hi, Michael and myself are porting code from Linux to FreeBSD and we have notic= ed a peculiar difference in the way poll() events are handled. In short, we have a process that monitors the lifetime of other processes. = It does so by sharing a pipe between the parent and the child on every fork: read-end in the parent, write-end in the child. The pipe is not used to communicate; it's only used to poll() on the death of the child process. On Linux, poll() is called with a POLLHUP event and nothing else. When the child process dies, the poll() call returns with 'revents =3D=3D POLLHU= P'. After some head scratching, we figured that on FreeBSD (12.1 and 12.2) if t= he child process died while the parent was not waiting in poll(), we would get 'revents =3D=3D POLLHUP' on the next invocation of poll(), like we do on Li= nux. However, if the parent is in poll() when the child dies, the call to poll() never unblocks. This resulted in occasional hangs in the application. I am joining a reproducer [1]. As indicated, changing the 'events' to 'POLLIN | POLLHUP' causes both event= s to be delivered in both cases (child dies before/during calling poll()). The following excerpts of the FreeBSD, Linux, and Open Specification seem in agreement that passing POLLHUP is unnecessary as it is checked implicitl= y. FreeBSD (POLL(2)) This flag is always checked, even if not present in the events bitmask [.= ..] Open Group: This flag is only valid in the revents bitmask; it shall be ignored in th= e events member. Linux (poll(2)): Hang up (only returned in revents; ignored in events). I am surprised by the behaviour being different depending on the moment the child process' death occurs. This is not a big deal for us to work-around, but I would like to know if I should open a bug and try to fix it or if this is intentional (and perhaps documented?) behaviour. Thanks! J=C3=A9r=C3=A9mie Galarneau [1] https://gist.github.com/jgalar/5c3c2673b69fa0df652bda80a88f860c --=20 J=C3=A9r=C3=A9mie Galarneau EfficiOS Inc. http://www.efficios.com From owner-freebsd-hackers@freebsd.org Tue Nov 3 23:42:45 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5FABC467BB5 for ; Tue, 3 Nov 2020 23:42:45 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-yb1-xb2c.google.com (mail-yb1-xb2c.google.com [IPv6:2607:f8b0:4864:20::b2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQmYJ2sSQz4ckJ; Tue, 3 Nov 2020 23:42:44 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: by mail-yb1-xb2c.google.com with SMTP id a12so16379502ybg.9; Tue, 03 Nov 2020 15:42:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=+Zx5I1TOnfzRp9Qah4GrGLA/AVPm8nDPnchyg9nYrFg=; b=mm5/7DZ+3BhnDrT/LcoRmZDTOnzJlpYbf6gs1sRfqzYml0s0zRr4yIGWgujihBlb8B O2Jeh8AjfYL2RQEcILFEraslzZ/6Dh3L8C57Y9YfBzkl2JkWdDWq9rizOk1U5XRtYz4B RKJRVQwLMMjz9X13002ILXw4a74pVKmqwUWoGP6Wevjci7/t3aNUX1rTOnsZYyQFWeqt U3vsiZ5MCaGFX1mgdU9hY6xQq02agLooznamhDQ3vFTXKKb63FZQ4Jczev0ojcgwL0tj vLU2ATqvs/1I7Y4LVuwVSypBrQESn3aD4bAZqibp2OCOfeDzIzQaK/ZGb4a5/fldmXf9 bQEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=+Zx5I1TOnfzRp9Qah4GrGLA/AVPm8nDPnchyg9nYrFg=; b=eJraDsgroLwiO8NAJZYB3/dXDRFggEth77tQU5piXBxJfxUP0ht4XIQuN1NjQaNM8y XTtJFqqtAT9hW5c3pY+nZ4jTCJry6Rw4Af5D6Pu46lZANQK2QfBr8y+rtm9U7/JBachD Pr4q2P4yv/YSj8S6ooT22d380mtXQ9A16NcNrgmBnAZfmHwo1wJ1iRAeBHz4qR6/Ycww 0NxC/GIxvjhincAq1/JODd00KmMk7cgRgBOUBYUSNtfDMMjobPd3HnSG14z/0ye6vXgv l9Xi94tcPMIn/ic0M20jB5rNwDTfeZ7YWPntW6lwamhaiE5e8MaNEqA25mapMXhryBVV aKfQ== X-Gm-Message-State: AOAM531srVrMcblfzHJcPLcRr429Wz/uejKZSwGto8rhZlTCmiMBVDZR qOQaJY4FAs7I5/2hFqhj9F00/1ArWIVbwc7dhP+4c14v X-Google-Smtp-Source: ABdhPJxsTDqsF8VcHBsiyE4XhX0Ug5ZaqTHVAa8BmzDbo5EB6I8yFJXgfu2YQhfQeglh/Nh0aOdJ5+kD5XjY3lgrTdw= X-Received: by 2002:a25:cc14:: with SMTP id l20mr33382949ybf.478.1604446963139; Tue, 03 Nov 2020 15:42:43 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a25:d8d2:0:0:0:0:0 with HTTP; Tue, 3 Nov 2020 15:42:42 -0800 (PST) In-Reply-To: References: From: Oliver Pinter Date: Wed, 4 Nov 2020 00:42:42 +0100 Message-ID: Subject: Re: poll() POLLHUP behaviour inconsistency To: =?UTF-8?Q?J=C3=A9r=C3=A9mie_Galarneau?= , mjg@freebsd.org Cc: "freebsd-hackers@freebsd.org" , Michael Jeanson X-Rspamd-Queue-Id: 4CQmYJ2sSQz4ckJ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=mm5/7DZ+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of oliverpntr@gmail.com designates 2607:f8b0:4864:20::b2c as permitted sender) smtp.mailfrom=oliverpntr@gmail.com X-Spamd-Result: default: False [-3.33 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; NEURAL_HAM_LONG(-1.04)[-1.037]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2c:from]; NEURAL_HAM_SHORT(-0.29)[-0.287]; NEURAL_HAM_MEDIUM(-1.01)[-1.006]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 23:42:45 -0000 CC: mjg@ On Wednesday, November 4, 2020, J=C3=A9r=C3=A9mie Galarneau < jeremie.galarneau@efficios.com> wrote: > Hi, > > Michael and myself are porting code from Linux to FreeBSD and we have > noticed a > peculiar difference in the way poll() events are handled. > > In short, we have a process that monitors the lifetime of other processes= . > It > does so by sharing a pipe between the parent and the child on every fork: > read-end in the parent, write-end in the child. The pipe is not used to > communicate; it's only used to poll() on the death of the child process. > > On Linux, poll() is called with a POLLHUP event and nothing else. When > the child process dies, the poll() call returns with 'revents =3D=3D POLL= HUP'. > > After some head scratching, we figured that on FreeBSD (12.1 and 12.2) if > the > child process died while the parent was not waiting in poll(), we would g= et > 'revents =3D=3D POLLHUP' on the next invocation of poll(), like we do on = Linux. > However, if the parent is in poll() when the child dies, the call to poll= () > never unblocks. This resulted in occasional hangs in the application. > > I am joining a reproducer [1]. > > > As indicated, changing the 'events' to 'POLLIN | POLLHUP' causes both > events to > be delivered in both cases (child dies before/during calling poll()). > > The following excerpts of the FreeBSD, Linux, and Open Specification seem > in agreement that passing POLLHUP is unnecessary as it is checked > implicitly. > > FreeBSD (POLL(2)) > This flag is always checked, even if not present in the events bitmask > [...] > > Open Group: > This flag is only valid in the revents bitmask; it shall be ignored in > the > events member. > > Linux (poll(2)): > Hang up (only returned in revents; ignored in events). > > > I am surprised by the behaviour being different depending on the moment t= he > child process' death occurs. > > This is not a big deal for us to work-around, but I would like to know if= I > should open a bug and try to fix it or if this is intentional (and perhap= s > documented?) behaviour. > > Thanks! > J=C3=A9r=C3=A9mie Galarneau > > [1] https://gist.github.com/jgalar/5c3c2673b69fa0df652bda80a88f860c > > -- > J=C3=A9r=C3=A9mie Galarneau > EfficiOS Inc. > http://www.efficios.com > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > From owner-freebsd-hackers@freebsd.org Wed Nov 4 00:12:13 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9D303448E0F for ; Wed, 4 Nov 2020 00:12:13 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQnCJ64Zwz4fHR for ; Wed, 4 Nov 2020 00:12:12 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-wm1-x32e.google.com with SMTP id p22so872347wmg.3 for ; Tue, 03 Nov 2020 16:12:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=qdFE7B6HxXwRfV9VKWtC+niIHZHqjPPlR5eZFEdpNfQ=; b=XGR0lSx4Q+g0CmSaIV1xmnun+TaRhkedacu5mJy/ABxxjh1vLzxso6oDyUG7kuTfiH g6XPPXhv+0w0Wf8Ko2cJ8N7JtbXuRsH804PxSo1AD/A7UuAU97W0ftSuFNkJ2ka4OOfB vhX5aRjXpKGlxqtedpQif2pDjRnCgmla944zj6hTO6sL6b5F78Oxph7uIiNEHgB22wdy LRiC/QQs4olCAV1ou+JibbczeQbOAX249K22kMwDQ2I3PG/6uyO4aFkjY+KSTpzwhqKC VWzz3i4OEPyFwpipY6+w8Z8mLMuejTpsnv42oa1GGdDiM2JUZECCZF2tI9jifsnii7OK CMeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=qdFE7B6HxXwRfV9VKWtC+niIHZHqjPPlR5eZFEdpNfQ=; b=dsar3ruXjmf80+fK+u58U974lMXuBacYd7D8iuxIlghFaHLuz+8Azq7/08yxV46Gcp 6o5BxqNVzbk14kiCxNg4wv1pyKUvA0K5stW1yPIkNf3y/u0KwYEO+DFLjqusUn/MtGur lzm/s+RabdboBqZAbxKCq/27tBuAdT1qEt2M8Q4c3tFJd/hTdSOqEI65oRvm3oFp2sDe xjU3eAroV+bTq2r4Tyvna8g3f0FRpE1hbh3Y7oq+nEZocP3ajoyY9GeKmlCgmtO5IvN4 TQuBXZMiqa0eMk3/w1q6UOA558Tv96di0rhHRDasyS3Ovd38sP5WzB3ntbbiYoziinj2 LttQ== X-Gm-Message-State: AOAM531vQiCvdnUdSzpv7hc2zNtxo3TIJOGKGXeW1b7uDaqShjDyAg2o t9kUUjCqc9y+lzX8DIUSrneif0N/UUWOx5iPDZzSnyz97IA= X-Google-Smtp-Source: ABdhPJxrkNztjycDpTLt/1F7C9E04eNMdgfonY+dukc++Wix625HOsu8TXGdknGK9w4jrP3+IviYI6/DyQfcMGtM25M= X-Received: by 2002:a1c:2441:: with SMTP id k62mr1781400wmk.10.1604448730643; Tue, 03 Nov 2020 16:12:10 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a5d:4c4f:0:0:0:0:0 with HTTP; Tue, 3 Nov 2020 16:12:09 -0800 (PST) In-Reply-To: References: From: Mateusz Guzik Date: Wed, 4 Nov 2020 01:12:09 +0100 Message-ID: Subject: Re: poll() POLLHUP behaviour inconsistency To: =?UTF-8?Q?J=C3=A9r=C3=A9mie_Galarneau?= Cc: freebsd-hackers@freebsd.org, Michael Jeanson Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CQnCJ64Zwz4fHR X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=XGR0lSx4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::32e as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-3.18 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.03)[-1.031]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32e:from]; NEURAL_HAM_SHORT(-0.15)[-0.148]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 00:12:13 -0000 On 11/4/20, J=C3=A9r=C3=A9mie Galarneau wr= ote: > Hi, > > Michael and myself are porting code from Linux to FreeBSD and we have > noticed a > peculiar difference in the way poll() events are handled. > > In short, we have a process that monitors the lifetime of other processes= . > It > does so by sharing a pipe between the parent and the child on every fork: > read-end in the parent, write-end in the child. The pipe is not used to > communicate; it's only used to poll() on the death of the child process. > > On Linux, poll() is called with a POLLHUP event and nothing else. When > the child process dies, the poll() call returns with 'revents =3D=3D POLL= HUP'. > > After some head scratching, we figured that on FreeBSD (12.1 and 12.2) if > the > child process died while the parent was not waiting in poll(), we would g= et > 'revents =3D=3D POLLHUP' on the next invocation of poll(), like we do on = Linux. > However, if the parent is in poll() when the child dies, the call to poll= () > never unblocks. This resulted in occasional hangs in the application. > > I am joining a reproducer [1]. > > > As indicated, changing the 'events' to 'POLLIN | POLLHUP' causes both eve= nts > to > be delivered in both cases (child dies before/during calling poll()). > > The following excerpts of the FreeBSD, Linux, and Open Specification seem > in agreement that passing POLLHUP is unnecessary as it is checked > implicitly. > > FreeBSD (POLL(2)) > This flag is always checked, even if not present in the events bitmask > [...] > > Open Group: > This flag is only valid in the revents bitmask; it shall be ignored in = the > events member. > > Linux (poll(2)): > Hang up (only returned in revents; ignored in events). > > > I am surprised by the behaviour being different depending on the moment t= he > child process' death occurs. > > This is not a big deal for us to work-around, but I would like to know if= I > should open a bug and try to fix it or if this is intentional (and perhap= s > documented?) behaviour. > > Thanks! > J=C3=A9r=C3=A9mie Galarneau > > [1] https://gist.github.com/jgalar/5c3c2673b69fa0df652bda80a88f860c > Thanks for the detailed report with a reproducer. pipe_poll checks for POLLIN | POLLRDNORM and POLLOUT | POLLWRNORM in order to decide whether to add itself to the list of waiters. Since you don't specify any of it and POLLHUP condition is not met, it neglects to do anything but at the same time does not return any events to poll itself. Then poll blocks waiting for wakeups which never come since pipe_poll did not add us anywhere. A trivial hack looks like this: diff --git a/sys/kern/sys_pipe.c b/sys/kern/sys_pipe.c index 239cf3bbdfb..59bc03e032a 100644 --- a/sys/kern/sys_pipe.c +++ b/sys/kern/sys_pipe.c @@ -1458,13 +1458,13 @@ pipe_poll(struct file *fp, int events, struct ucred *active_cred, } if (revents =3D=3D 0) { - if (fp->f_flag & FREAD && events & (POLLIN | POLLRDNORM)) { + if (fp->f_flag & FREAD) { selrecord(td, &rpipe->pipe_sel); if (SEL_WAITING(&rpipe->pipe_sel)) rpipe->pipe_state |=3D PIPE_SEL; } - if (fp->f_flag & FWRITE && events & (POLLOUT | POLLWRNORM))= { + if (fp->f_flag & FWRITE) { selrecord(td, &wpipe->pipe_sel); if (SEL_WAITING(&wpipe->pipe_sel)) wpipe->pipe_state |=3D PIPE_SEL; With this in place the reproducer passes. I don't know yet if this is just a pipe or general poll problem. I don't know what the right fix is right now, may take few days. This may or may not be a candidate for errata for the 12.2 release, depending on how the extensive the real fix will turn out to be. That said you may need to implement a workaround regardless of the issue getting fixed. Thanks, --=20 Mateusz Guzik From owner-freebsd-hackers@freebsd.org Wed Nov 4 01:30:24 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05BA044A91D; Wed, 4 Nov 2020 01:30:24 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQpxV6Tthz3V4f; Wed, 4 Nov 2020 01:30:22 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by mail-pg1-x52b.google.com with SMTP id f38so15170733pgm.2; Tue, 03 Nov 2020 17:30:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=cpQ3S2rbVhd8LdJqsmdfB8QRGRwRUT8yneoStHUogpo=; b=kthhDJEvtdCiDWfG1OOsE/P6jPHxmYpph+skhbomoELHaH2JwMgZ/Bpx2/ynzEsnoT GkvL/8Bn/go7zvQ/7i/TG9StU6TwFc4Rn8vxrxSZANjEhmUhhsjqc737yoaYxLUnN4j+ 5YDJnJJzsY7qaZxxmfBFXHPGIAGFAdJKC1oP1mc6CZ2c02U8mUnZ+U59jYmrPI+wCadh eO8ClK10z6OAcmkYMzCGhpOu7SMIbR6UYIjMHz5mLyDaSbi1TcVRKDA63zUq+PJiVclH B6HWAQGNYdRjTrXlA18FhQPJ6elvpS2Z9N7id/mAb79XbMk77+NiS/ruAXxnc7sleMfT nmcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=cpQ3S2rbVhd8LdJqsmdfB8QRGRwRUT8yneoStHUogpo=; b=SE9a32+FYIqfKpM1r9Y14nQzqJwsePaOtOkY6GWl0Tm5UvrsWyP2lIityX/PdRtHnd I0lEXxSUYsqrPqggXpJQXB9W4kz852Zfa9OxR9d2gpfXKPhUc1euB6gLqCvvfBtHZDYl bm9Y+ETlC9+tFMvJTFRpYYs7sZUYKZt6AUZfETr0BQJR3Z6NLhBRDpVcy0h57vgM7luv S3w/XWgz5akUiu0LGAwXkdT57VyQ8RGHRvMN7lOmE1kfRc7a40vNxtpcrrNdbUP9RySK PPomws92E8tH3hs8Hn7GhkjgZ4i3U8ZWGDkoGNZrIVmKUjYk5HV8JJzzS7Es8emhYeoz uYjQ== X-Gm-Message-State: AOAM533S0Mq0g1aJNjp/z8uoeWvWIFfw2LXZNjRj4wDIaSo52WIJrUcB bYC+AtqzOEENe4GX/eD7/i4= X-Google-Smtp-Source: ABdhPJw7y+SSoS0X5AvSvzzwpaD+Vf/J+QVMB0j/4mdYAzuJ1Q2QtrGEPOhQWpUo4iqdloVoPCYnjQ== X-Received: by 2002:a05:6a00:88d:b029:164:5a00:29b8 with SMTP id q13-20020a056a00088db02901645a0029b8mr29271628pfj.10.1604453420990; Tue, 03 Nov 2020 17:30:20 -0800 (PST) Received: from localhost ([210.108.244.139]) by smtp.gmail.com with ESMTPSA id d22sm191473pgv.87.2020.11.03.17.30.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Nov 2020 17:30:20 -0800 (PST) Date: Wed, 4 Nov 2020 10:30:17 +0900 From: YongHyeon PYUN To: Lowell Gilbert Cc: freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Message-ID: <20201104013017.GA48398@michelle> References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> <44zh3ybd0s.fsf@Be-Well.Ilk.Org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <44zh3ybd0s.fsf@Be-Well.Ilk.Org> X-Rspamd-Queue-Id: 4CQpxV6Tthz3V4f X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kthhDJEv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of pyunyh@gmail.com designates 2607:f8b0:4864:20::52b as permitted sender) smtp.mailfrom=pyunyh@gmail.com X-Spamd-Result: default: False [-3.52 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.97)[-0.975]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; NEURAL_HAM_LONG(-1.06)[-1.056]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52b:from]; NEURAL_HAM_SHORT(-0.99)[-0.992]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers,freebsd-arm]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 01:30:24 -0000 On Tue, Nov 03, 2020 at 11:50:27AM -0500, Lowell Gilbert wrote: > YongHyeon PYUN writes: > > > On Thu, Oct 29, 2020 at 10:39:39PM +0100, Kristof Provost wrote: > > > > [...] > > > >> It does seem to do RX offload, and the comments in the driver suggest some > >> .. ahem, creative hardware behaviour: > >> > >> /* The checksum appears to be simplistically calculated > >> * over the udp/tcp header and data up to the end of the > >> * eth frame. Which means if the eth frame is padded > >> * the csum calculation is incorrectly performed over > >> * the padding bytes as well. Therefore to be safe we > >> * ignore the H/W csum on frames less than or equal to > >> * 64 bytes. > >> * > >> * Ignore H/W csum for non-IPv4 packets. > >> */ > >> > >> It’s not impossible that there’s some more issues like that in the hardware, > >> or even that there are different chip revisions out there. > >> > >> That also matches up with `ifconfig ue0 -rxcsum` fixing things. > >> > > > > I'm not sure where the root cause is but it seems smsc(4) needs > > improvement in RX checksum handling. Quick reading RX handler > > indicates RX checksum offloading does not work when: > > o IP datagram has IP options field > > o TCP/UDP packet was fragmented > > o UDP datagrams with 0 checksum value > > See fxp(4), gem(4) and hme(4) for implementation. > > > > It looks like smsc(4) uses the following RX format but I don't > > know actual RX format of H/W(no access to datasheet). > > > > <---------------------------- actlen --------------------------------------------------> > > <------------- pktlen ------------------------> > > rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial checksum(2 bytes) > > > > If the format above is correct smsc(4) needs more strict RX length > > validation(USB transfer size vs actual packet length). This > > indicates smsc(4) does not have to copy FCS bytes. > > Also given that H/W always appends FCS, it's also suspicious H/W > > does not correctly compute RX checksum for frames less than or > > equal to 64 bytes. > > For the 89530 at least, the data sheet states that checksum offload > and padding stripping are mutually exclusive. I would assume that > enabling both of them will produce incorrect results. I would *not* > assume that all chips covered by this driver behave the same way, > although it's likely. I see. Publicly available data sheet was too much sanitized such that it's useless to implement checksum offloading. Given that you have access to data sheet, could you describe exact RX format of H/W? FreeBSD's USB stack requires copy operation for received frames so driver does not need any padding in RX path and ethernet frame alignment in mbuf can be adjusted in the copy operation. Since FCS bytes have no use in upper stack at this moment, it would be even better to strip FCS if there is way to do that in H/W. For TX checksum offloading support, we need more information on H/W's TX handling(TX_CTRL_0 and TX_CTRL_1) too. I guess it needs special magic sequence of programming(i.e. existence of SMSC_TX_CTRL_0_FIRST_SEG and SMSC_TX_CTRL_0_LAST_SEG) but it could be greatly simplified to program the H/W for TX checksum offloading since driver also have to copy entire mbuf chain to USB stack(i.e single big TX segment for the H/W). From owner-freebsd-hackers@freebsd.org Wed Nov 4 06:36:50 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 44D3945283B; Wed, 4 Nov 2020 06:36:50 +0000 (UTC) (envelope-from carbaecker@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQxl52Mjmz4Js4; Wed, 4 Nov 2020 06:36:48 +0000 (UTC) (envelope-from carbaecker@gmx.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1604471806; bh=uFkcaMWwvRbtq0yolrsH/akSHbmxOHCVinz9rM6fO6Y=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=ITeMeXMmFD1JMk1uJ5udVmMkf8p1TSGjIA0Fi7ldrTVfApF80Zx6jCihN3S9KYMwy CA5m/IKWR6FfbjnmxBAkm52rIfUqSVVgYLjx1YGfrEYj8eO3wWCiEQcrhh90sH/zlN 5Eh0OuKg6D4g8CfemgE3d8Jll3BT5YvoBp1ncZn0= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.178.30] ([94.31.96.148]) by mail.gmx.com (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N7QxL-1kHtC708K5-017kbB; Wed, 04 Nov 2020 07:36:46 +0100 Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) To: Hans Petter Selasky , YongHyeon PYUN , Kristof Provost Cc: freebsd-arm@freebsd.org, freebsd-hackers@freebsd.org References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> <46d08198-530c-cb4b-efa8-4edaf89471c1@selasky.org> <4dfaa9a3-c085-8466-a6e4-19f988b5ed3d@selasky.org> From: =?UTF-8?Q?Carsten_B=c3=a4cker?= Message-ID: Date: Wed, 4 Nov 2020 07:36:44 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: <4dfaa9a3-c085-8466-a6e4-19f988b5ed3d@selasky.org> Content-Language: de-DE X-Provags-ID: V03:K1:YW7fqmbw0thFe1Tz4NTlGvCYR+MPZ6uJtHuaa/6S4FVSYCON2ed +FeN91vYYPO5hYAP470ue3TJHEsI6vmxGqpR22FPmKDHvCOJqjjtLppeSpllEHKD9Hczb9I 9+S3yX0y1nGhZzdjsN8BwJVjAjADKhlMbsw4VZNlIZj3AyNNCx8ZjX3WhJTpO7iv8Isi9dp q2nTkU4x5G38rBdHbt9Hw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:kHIykTro/iw=:e5f1MSIoG+QfkF46rED31a GHeK9oqISFXpybOBjyLREq8ss5pgVKWEdywelskSKv5d0qBEOqltIurgQyI6oJ1kH/hiK4KxV Wwp4MLiLvI8BrdQkPlqbrrFuigC5W/esScBkIqUD8/DS4cAu3LRaX8WM+sxiSUAotsX4WrDGG g7dZMXM73TGMnYhpxbslQUZZYK+qb+iKIBKjzoamYjJdBPo4jdvHbXzowXK3VL/M3Xtki888E FSFM1J/VSdbw7JwzYzsl7IYpInNMCqpnN92t8UdUjKeXR/1U9R0oxEo7P2sK/wcaq9Zqkgywg FN+h0kETGbGtLlRsgB/0AOHlVu0nw8vQlOTw15/v/TUjNKdFANZVuYAwkVNzs1oGq5iyE5PyP 1T0JYn0S0IfxLOt5aAWLAm/yZ8eBehiNRyH2LWjXZNQnfHgBRRA4w6khjnqRc+X2rM+uqIVzJ AnMjpi2hDjRP8icSLQoOUNwojJKRcVsGo77EZtJxKSoqHWLIulZmtRMarkJGCve8SSfSRiIPP E3GYAJhuX6BGABalmf0koAHzVvgdVfnTxKB2MZtZy+Ty481CySi4vlY88KCeAogPAPtRts+NU 4ZnSpX7yC42FfbUHYWwaJkLMsXvP+iU4VgnHs7sZXQ5/ObxQO/yhbE0vaMRbt7suevXV1V0uy 7Gsls0K6ipwbnrJ6MfKQ8yOsg6z3VMqkjyLvhEOiu7ImoEugwAno48LkKJdHFS8SDiP3Cs8ml SGO5Tv3oriK3dGeL88rNoJbZlXEnMCIOvn+K0RxdALxFyEd1Y+eMW5dV0rr7epriCIj7vFj2U lboMF5E1ELrV4v9m0Vz+uaCyaxrAZ6OIl5IiNskSGjcyGVIYRry8ALIcR+mZ3urr2S0UftLOo B/xBhlLQn6zP8812in8Q== X-Rspamd-Queue-Id: 4CQxl52Mjmz4Js4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.b=ITeMeXMm; dmarc=none; spf=pass (mx1.freebsd.org: domain of carbaecker@gmx.de designates 212.227.15.18 as permitted sender) smtp.mailfrom=carbaecker@gmx.de X-Spamd-Result: default: False [-3.25 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmx.de]; R_SPF_ALLOW(-0.20)[+ip4:212.227.15.0/25]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[gmx.net:+]; NEURAL_HAM_SHORT(-0.60)[-0.599]; FREEMAIL_TO(0.00)[selasky.org,gmail.com,FreeBSD.org]; RECEIVED_SPAMHAUS_PBL(0.00)[94.31.96.148:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmx.de]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; R_DKIM_ALLOW(-0.20)[gmx.net:s=badeba3b8450]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.15.18:from]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.06)[-1.064]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[gmx.de]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.15.18:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-arm,freebsd-hackers] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 06:36:50 -0000 Am 03.11.2020 um 09:46 schrieb Hans Petter Selasky: > On 2020-11-03 09:19, Hans Petter Selasky wrote: >> It looks like smsc(4) uses the following RX format but I don't >> know actual RX format of H/W(no access to datasheet). >> >> <---------------------------- actlen >> --------------------------------------------------> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 <------------- pktlen ------------------------= > >> rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | >> partial checksum(2 bytes) > > Hi, > > I wonder if the checksum is zero, when not valid, and that we should > check for this in the driver! > > Can you try this patch? > > Also enabling debugging in the SMSC driver would be useful. > > --HPS > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g" Hi, i applied the patch on -CURRENT but got a panic right after loading the kernel. Most likely an unrelated problem. But i was able to apply the patch on releng/12.2 (with an offset). Unfortunately it doesn't change the previously described behavior with rxcsum and i didn't manage to get any reasonable debug-output. Since i can easily reproduce the problem. How else can i help? Regards, Carsten From owner-freebsd-hackers@freebsd.org Wed Nov 4 08:52:30 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 42B674560C3; Wed, 4 Nov 2020 08:52:30 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CR0ld1hq3z4SVT; Wed, 4 Nov 2020 08:52:28 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id DB066260198; Wed, 4 Nov 2020 09:52:25 +0100 (CET) Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) To: =?UTF-8?Q?Carsten_B=c3=a4cker?= , YongHyeon PYUN , Kristof Provost Cc: freebsd-arm@freebsd.org, freebsd-hackers@freebsd.org References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> <46d08198-530c-cb4b-efa8-4edaf89471c1@selasky.org> <4dfaa9a3-c085-8466-a6e4-19f988b5ed3d@selasky.org> From: Hans Petter Selasky Message-ID: <9e1f9407-e2ff-6486-256e-012626b534fb@selasky.org> Date: Wed, 4 Nov 2020 09:52:24 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CR0ld1hq3z4SVT X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-2.19 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.05)[-1.054]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; ARC_NA(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.15)[0.153]; NEURAL_HAM_MEDIUM(-0.99)[-0.992]; FREEMAIL_TO(0.00)[gmx.de,gmail.com,FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm,freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 08:52:30 -0000 On 2020-11-04 07:36, Carsten Bäcker wrote: > Am 03.11.2020 um 09:46 schrieb Hans Petter Selasky: >> On 2020-11-03 09:19, Hans Petter Selasky wrote: >>> It looks like smsc(4) uses the following RX format but I don't >>> know actual RX format of H/W(no access to datasheet). >>> >>> <---------------------------- actlen >>> --------------------------------------------------> >>>                  <------------- pktlen ------------------------> >>> rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | >>> partial checksum(2 bytes) >> >> Hi, >> >> I wonder if the checksum is zero, when not valid, and that we should >> check for this in the driver! >> >> Can you try this patch? >> >> Also enabling debugging in the SMSC driver would be useful. >> >> --HPS >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe@freebsd.org" > Hi, > > i applied the patch on -CURRENT but got a panic right after loading the > kernel. Most likely an unrelated problem. > > But i was able to apply the patch on releng/12.2 (with an offset). > Unfortunately it doesn't change the previously described behavior with > rxcsum and i didn't manage to get any reasonable debug-output. > > Since i can easily reproduce the problem. How else can i help? > Can you enable the debug knob for SMSC: sysctl hw.usb.smsc.debug=16 It will print the RX checksum per frame received. Just do this with the unpatched driver while doing some traffic. Capture log messages from dmesg or /var/log/messages. --HPS From owner-freebsd-hackers@freebsd.org Wed Nov 4 22:01:58 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 187A54677C9 for ; Wed, 4 Nov 2020 22:01:58 +0000 (UTC) (envelope-from jeremie.galarneau@efficios.com) Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRLGY2M6Gz4Ljn for ; Wed, 4 Nov 2020 22:01:57 +0000 (UTC) (envelope-from jeremie.galarneau@efficios.com) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 39B6F2E1AFD for ; Wed, 4 Nov 2020 17:01:56 -0500 (EST) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id j9S3g0Kl2axz for ; Wed, 4 Nov 2020 17:01:55 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 7304F2E1E0F for ; Wed, 4 Nov 2020 17:01:55 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 7304F2E1E0F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1604527315; bh=MkuRwMqIwQuw8xoUG/B3gi5dYQpLqRzdd6qPZA1dD2o=; h=MIME-Version:From:Date:Message-ID:To; b=qPHDa5tWVx1H3h6T5tCEVoPMrQ9KiiPj5/Dpsc5FSWDKmmKZutWWqd+yZfnr0trXQ vF7SH4ivIVSv8irRriOxkTPt8HB5yrVUiOIKkD+Aji0O0KWDp/Ph4RMjCFMog0g4jn 2iI9e9sQKkAkHWPUtTRO4dRxmkzjziUxL3mWF5FdULTXRfLzdkeFgQ90SNOeEPsFbA H9OTopNbxUQaTT7hjsritCzfxX114qkxHdwbgQuGJlv33OJ0jErvmHR3aF9kcWJ7EK SQXtEgYf/hUg+NNFfQD+ju2Ee1ETY9VBPzLp8sDSj5jJgq3rHvFPWZUqw88TJjttmN wf7Z4L69GGcaQ== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id XfnuoL9GdeYo for ; Wed, 4 Nov 2020 17:01:55 -0500 (EST) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mail.efficios.com (Postfix) with ESMTPSA id 161A52E1BF5 for ; Wed, 4 Nov 2020 17:01:55 -0500 (EST) Received: by mail-wr1-f42.google.com with SMTP id n15so127527wrq.2 for ; Wed, 04 Nov 2020 14:01:55 -0800 (PST) X-Gm-Message-State: AOAM530l7+AXaUqbEFKGkP/cPktSRdC6SKTsv5monB3RnTNy13lMqQrf fDOpvhzIWbus1eS+63ced8IJQnuiUmFFptR7hmU= X-Google-Smtp-Source: ABdhPJz6dAG9XVcKD5JPcDOGcn7GWh4+kMl01gTbEhaS9Tt32RPOsIlhucNzexbyzrLuSwvlULSLB44gEG5ent8T1uY= X-Received: by 2002:adf:80c8:: with SMTP id 66mr46325wrl.415.1604527313859; Wed, 04 Nov 2020 14:01:53 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?J=C3=A9r=C3=A9mie_Galarneau?= Date: Wed, 4 Nov 2020 17:01:18 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: poll() POLLHUP behaviour inconsistency To: Mateusz Guzik Cc: freebsd-hackers@freebsd.org, Michael Jeanson Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CRLGY2M6Gz4Ljn X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=efficios.com header.s=default header.b=qPHDa5tW; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (mx1.freebsd.org: domain of jeremie.galarneau@efficios.com designates 167.114.26.124 as permitted sender) smtp.mailfrom=jeremie.galarneau@efficios.com X-Spamd-Result: default: False [-3.07 / 15.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[efficios.com:s=default]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.03)[-1.031]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[efficios.com:+]; DMARC_POLICY_ALLOW(-0.50)[efficios.com,none]; NEURAL_HAM_SHORT(-1.45)[-1.449]; NEURAL_HAM_MEDIUM(-1.02)[-1.016]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(1.43)[subject]; ASN(0.00)[asn:16276, ipnet:167.114.0.0/17, country:FR]; RCVD_COUNT_SEVEN(0.00)[7]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 22:01:58 -0000 On Tue, 3 Nov 2020 at 19:12, Mateusz Guzik wrote: > > On 11/4/20, J=C3=A9r=C3=A9mie Galarneau = wrote: > > Hi, > > > > Michael and myself are porting code from Linux to FreeBSD and we have > > noticed a > > peculiar difference in the way poll() events are handled. > > > > In short, we have a process that monitors the lifetime of other process= es. > > It > > does so by sharing a pipe between the parent and the child on every for= k: > > read-end in the parent, write-end in the child. The pipe is not used to > > communicate; it's only used to poll() on the death of the child process= . > > > > On Linux, poll() is called with a POLLHUP event and nothing else. When > > the child process dies, the poll() call returns with 'revents =3D=3D PO= LLHUP'. > > > > After some head scratching, we figured that on FreeBSD (12.1 and 12.2) = if > > the > > child process died while the parent was not waiting in poll(), we would= get > > 'revents =3D=3D POLLHUP' on the next invocation of poll(), like we do o= n Linux. > > However, if the parent is in poll() when the child dies, the call to po= ll() > > never unblocks. This resulted in occasional hangs in the application. > > > > I am joining a reproducer [1]. > > > > > > As indicated, changing the 'events' to 'POLLIN | POLLHUP' causes both e= vents > > to > > be delivered in both cases (child dies before/during calling poll()). > > > > The following excerpts of the FreeBSD, Linux, and Open Specification se= em > > in agreement that passing POLLHUP is unnecessary as it is checked > > implicitly. > > > > FreeBSD (POLL(2)) > > This flag is always checked, even if not present in the events bitmas= k > > [...] > > > > Open Group: > > This flag is only valid in the revents bitmask; it shall be ignored i= n the > > events member. > > > > Linux (poll(2)): > > Hang up (only returned in revents; ignored in events). > > > > > > I am surprised by the behaviour being different depending on the moment= the > > child process' death occurs. > > > > This is not a big deal for us to work-around, but I would like to know = if I > > should open a bug and try to fix it or if this is intentional (and perh= aps > > documented?) behaviour. > > > > Thanks! > > J=C3=A9r=C3=A9mie Galarneau > > > > [1] https://gist.github.com/jgalar/5c3c2673b69fa0df652bda80a88f860c > > > > Thanks for the detailed report with a reproducer. > > pipe_poll checks for POLLIN | POLLRDNORM and POLLOUT | POLLWRNORM in > order to decide whether to add itself to the list of waiters. Since > you don't specify any of it and POLLHUP condition is not met, it > neglects to do anything but at the same time does not return any > events to poll itself. Then poll blocks waiting for wakeups which > never come since pipe_poll did not add us anywhere. > > A trivial hack looks like this: > diff --git a/sys/kern/sys_pipe.c b/sys/kern/sys_pipe.c > index 239cf3bbdfb..59bc03e032a 100644 > --- a/sys/kern/sys_pipe.c > +++ b/sys/kern/sys_pipe.c > @@ -1458,13 +1458,13 @@ pipe_poll(struct file *fp, int events, struct > ucred *active_cred, > } > > if (revents =3D=3D 0) { > - if (fp->f_flag & FREAD && events & (POLLIN | POLLRDNORM))= { > + if (fp->f_flag & FREAD) { > selrecord(td, &rpipe->pipe_sel); > if (SEL_WAITING(&rpipe->pipe_sel)) > rpipe->pipe_state |=3D PIPE_SEL; > } > > - if (fp->f_flag & FWRITE && events & (POLLOUT | POLLWRNORM= )) { > + if (fp->f_flag & FWRITE) { > selrecord(td, &wpipe->pipe_sel); > if (SEL_WAITING(&wpipe->pipe_sel)) > wpipe->pipe_state |=3D PIPE_SEL; > > With this in place the reproducer passes. I don't know yet if this is > just a pipe or general poll problem. > > I don't know what the right fix is right now, may take few days. This > may or may not be a candidate for errata for the 12.2 release, > depending on how the extensive the real fix will turn out to be. > > That said you may need to implement a workaround regardless of the > issue getting fixed. Hi, Thanks for the great (and quick!) reply. Let me know if I can help with the fix in any way. J=C3=A9r=C3=A9mie > > Thanks, > -- > Mateusz Guzik --=20 J=C3=A9r=C3=A9mie Galarneau EfficiOS Inc. http://www.efficios.com From owner-freebsd-hackers@freebsd.org Wed Nov 4 23:23:03 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1634A468EBB for ; Wed, 4 Nov 2020 23:23:03 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRN460f3tz4QHw for ; Wed, 4 Nov 2020 23:23:01 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-wm1-x32d.google.com with SMTP id e2so3897786wme.1 for ; Wed, 04 Nov 2020 15:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=FucmCRi1W9UO345cFotboJLMamdMPRExa8nv0nbjmcI=; b=DzdWFRznlus0QlOzhlkEta4fCRio9hpt3jDSG8OYsfGCj8pYZk8FnmDRKNI+Z8g4EO cpJH0HPWqsKnL5rr44VzVwvSuTWZoLG4OZXpQP8BUn8FSQi9S3gfThHMh66+5/Kd9xhd zlgJNTxDN7yto9aneNkyx1wbdC/5g0KmqS2c5pqiprZl/EcvGgacNBACOr82qcmtYY7c 5qOsg74XLl3n3Wzw/HU8MNvDQLLMbsz+76wWjrV2oPngJ0No0POJBxivEQ0h9oyyXyNE PiSkmuUGAspbE798Q+ns0pMTIYEaEXjFefhFRgnPzllw0MJf1xWubg2vJCv83V73yzwG tdIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=FucmCRi1W9UO345cFotboJLMamdMPRExa8nv0nbjmcI=; b=piYEMzfz4LbwD2TObSiW4USKg9XKLlZKr+9j9vSmxLv2x7hGYOpu2LZqD/QqgLhCgq oD0peBtl7npUmAkHO+1K4ajIyGPfwrHsLrzGmGyipA5gLvrt7q9HnqGSVdH8/KmEMi6a XTeBIHSWqpesr8aZJpNYk0+WsYBo6Q4A/2WgawXCyF2BiEFUp1pgCUPJYViKdXvGXOoj GdKvZ1+GrdGVtBQaVyMy8/Lpfj47YY4dOVvx0bZIbXsnPwWI/RFg1quAgRpGfzcp42iM 1wnM926eN9qLDtVW+YX5rI4DuKjIdwkkeEo/W4ySCY4N00Qvq2k0sx8EDX0ARlbPR004 dZGQ== X-Gm-Message-State: AOAM530Y89GBoKqULoiW7kQYd+n0+X9w+AUCIPV8UpIHXONFf1LWfPAt XiJBPhgc1Ytp7tmFh3B7AGKQPj6pZihfrdlvpA8d+9XEfag= X-Google-Smtp-Source: ABdhPJyHtnOQppMv8mGrfukJLdxOcRALkuwz20/PajwnnVQP+WYfWOxkvOqT0j+br4HMN3UlKiZP+xn1snk7kib4A6k= X-Received: by 2002:a1c:e442:: with SMTP id b63mr118198wmh.10.1604532178537; Wed, 04 Nov 2020 15:22:58 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a5d:4c4f:0:0:0:0:0 with HTTP; Wed, 4 Nov 2020 15:22:57 -0800 (PST) In-Reply-To: References: From: Mateusz Guzik Date: Thu, 5 Nov 2020 00:22:57 +0100 Message-ID: Subject: Re: poll() POLLHUP behaviour inconsistency To: =?UTF-8?Q?J=C3=A9r=C3=A9mie_Galarneau?= Cc: freebsd-hackers@freebsd.org, Michael Jeanson Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CRN460f3tz4QHw X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=DzdWFRzn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::32d as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-4.10 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.01)[-1.011]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.03)[-1.031]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32d:from]; NEURAL_HAM_SHORT(-1.06)[-1.059]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 23:23:03 -0000 On 11/4/20, J=C3=A9r=C3=A9mie Galarneau wr= ote: > On Tue, 3 Nov 2020 at 19:12, Mateusz Guzik wrote: >> >> On 11/4/20, J=C3=A9r=C3=A9mie Galarneau = wrote: >> > Hi, >> > >> > Michael and myself are porting code from Linux to FreeBSD and we have >> > noticed a >> > peculiar difference in the way poll() events are handled. >> > >> > In short, we have a process that monitors the lifetime of other >> > processes. >> > It >> > does so by sharing a pipe between the parent and the child on every >> > fork: >> > read-end in the parent, write-end in the child. The pipe is not used t= o >> > communicate; it's only used to poll() on the death of the child >> > process. >> > >> > On Linux, poll() is called with a POLLHUP event and nothing else. When >> > the child process dies, the poll() call returns with 'revents =3D=3D >> > POLLHUP'. >> > >> > After some head scratching, we figured that on FreeBSD (12.1 and 12.2) >> > if >> > the >> > child process died while the parent was not waiting in poll(), we woul= d >> > get >> > 'revents =3D=3D POLLHUP' on the next invocation of poll(), like we do = on >> > Linux. >> > However, if the parent is in poll() when the child dies, the call to >> > poll() >> > never unblocks. This resulted in occasional hangs in the application. >> > >> > I am joining a reproducer [1]. >> > >> > >> > As indicated, changing the 'events' to 'POLLIN | POLLHUP' causes both >> > events >> > to >> > be delivered in both cases (child dies before/during calling poll()). >> > >> > The following excerpts of the FreeBSD, Linux, and Open Specification >> > seem >> > in agreement that passing POLLHUP is unnecessary as it is checked >> > implicitly. >> > >> > FreeBSD (POLL(2)) >> > This flag is always checked, even if not present in the events >> > bitmask >> > [...] >> > >> > Open Group: >> > This flag is only valid in the revents bitmask; it shall be ignored = in >> > the >> > events member. >> > >> > Linux (poll(2)): >> > Hang up (only returned in revents; ignored in events). >> > >> > >> > I am surprised by the behaviour being different depending on the momen= t >> > the >> > child process' death occurs. >> > >> > This is not a big deal for us to work-around, but I would like to know >> > if I >> > should open a bug and try to fix it or if this is intentional (and >> > perhaps >> > documented?) behaviour. >> > >> > Thanks! >> > J=C3=A9r=C3=A9mie Galarneau >> > >> > [1] https://gist.github.com/jgalar/5c3c2673b69fa0df652bda80a88f860c >> > >> >> Thanks for the detailed report with a reproducer. >> >> pipe_poll checks for POLLIN | POLLRDNORM and POLLOUT | POLLWRNORM in >> order to decide whether to add itself to the list of waiters. Since >> you don't specify any of it and POLLHUP condition is not met, it >> neglects to do anything but at the same time does not return any >> events to poll itself. Then poll blocks waiting for wakeups which >> never come since pipe_poll did not add us anywhere. >> >> A trivial hack looks like this: >> diff --git a/sys/kern/sys_pipe.c b/sys/kern/sys_pipe.c >> index 239cf3bbdfb..59bc03e032a 100644 >> --- a/sys/kern/sys_pipe.c >> +++ b/sys/kern/sys_pipe.c >> @@ -1458,13 +1458,13 @@ pipe_poll(struct file *fp, int events, struct >> ucred *active_cred, >> } >> >> if (revents =3D=3D 0) { >> - if (fp->f_flag & FREAD && events & (POLLIN | POLLRDNORM)= ) >> { >> + if (fp->f_flag & FREAD) { >> selrecord(td, &rpipe->pipe_sel); >> if (SEL_WAITING(&rpipe->pipe_sel)) >> rpipe->pipe_state |=3D PIPE_SEL; >> } >> >> - if (fp->f_flag & FWRITE && events & (POLLOUT | >> POLLWRNORM)) { >> + if (fp->f_flag & FWRITE) { >> selrecord(td, &wpipe->pipe_sel); >> if (SEL_WAITING(&wpipe->pipe_sel)) >> wpipe->pipe_state |=3D PIPE_SEL; >> >> With this in place the reproducer passes. I don't know yet if this is >> just a pipe or general poll problem. >> >> I don't know what the right fix is right now, may take few days. This >> may or may not be a candidate for errata for the 12.2 release, >> depending on how the extensive the real fix will turn out to be. >> >> That said you may need to implement a workaround regardless of the >> issue getting fixed. > > Hi, > > Thanks for the great (and quick!) reply. > Let me know if I can help with the fix in any way. > The fix landed in https://svnweb.freebsd.org/changeset/base/367352 in the development branch and I'm going to merge it in few days to the stable/12 -- that is it will be a part of future releases. I don't intend to work on issuing an errata for 12.2 as the problem seems to have an easy workaround. However, if fixing this is of importance I can look into it. --=20 Mateusz Guzik From owner-freebsd-hackers@freebsd.org Thu Nov 5 00:30:40 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F0EA7469FC1 for ; Thu, 5 Nov 2020 00:30:40 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRPZ81s9Gz4Stc for ; Thu, 5 Nov 2020 00:30:40 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.16.1/8.16.1) with ESMTPS id 0A50Ucs0056117 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 4 Nov 2020 16:30:39 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.16.1/8.16.1/Submit) id 0A50UcLj056116 for freebsd-hackers@freebsd.org; Wed, 4 Nov 2020 16:30:38 -0800 (PST) (envelope-from sgk) Date: Wed, 4 Nov 2020 16:30:38 -0800 From: Steve Kargl To: freebsd-hackers@freebsd.org Subject: Resetting freebsd bugzilla password? Message-ID: <20201105003038.GA56109@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspamd-Queue-Id: 4CRPZ81s9Gz4Stc X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=washington.edu (policy=none); spf=none (mx1.freebsd.org: domain of sgk@troutmask.apl.washington.edu has no SPF policy when checking 128.95.76.21) smtp.mailfrom=sgk@troutmask.apl.washington.edu X-Spamd-Result: default: False [0.16 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.78)[-0.780]; NEURAL_HAM_MEDIUM(-0.71)[-0.707]; NEURAL_SPAM_SHORT(0.64)[0.643]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US]; MAILMAN_DEST(0.00)[freebsd-hackers]; DMARC_POLICY_SOFTFAIL(0.10)[washington.edu : No valid SPF, No valid DKIM, none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 00:30:41 -0000 Anyone have a HowTo method for changing a FreeBSD bugzilla account password, where the current password has been forgotten? Clicking on 'Forgot Password' in the login banner results in a web page asking for the email address. The system is not configured to allow password change requests. Go to https://bugs.freebsd.org/bugzilla/enter_bug.cgi one finds again a place to reset the password, but one gets the above message when any attempt is made. -- Steve From owner-freebsd-hackers@freebsd.org Thu Nov 5 00:47:04 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2522446A6EB for ; Thu, 5 Nov 2020 00:47:04 +0000 (UTC) (envelope-from gonzo@bluezbox.com) Received: from id.bluezbox.com (id.bluezbox.com [45.55.20.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRPx338Pdz4VMn for ; Thu, 5 Nov 2020 00:47:03 +0000 (UTC) (envelope-from gonzo@bluezbox.com) Received: from localhost ([127.0.0.1] helo=id.bluezbox.com) by id.bluezbox.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94 (FreeBSD)) (envelope-from ) id 1kaTQW-0009CD-Ee; Wed, 04 Nov 2020 16:46:56 -0800 Received: (from gonzo@localhost) by id.bluezbox.com (8.15.2/8.15.2/Submit) id 0A50ku5f035352; Wed, 4 Nov 2020 16:46:56 -0800 (PST) (envelope-from gonzo@bluezbox.com) X-Authentication-Warning: id.bluezbox.com: gonzo set sender to gonzo@bluezbox.com using -f Date: Wed, 4 Nov 2020 16:46:56 -0800 From: Oleksandr Tymoshenko To: Steve Kargl Cc: freebsd-hackers@freebsd.org Subject: Re: Resetting freebsd bugzilla password? Message-ID: <20201105004656.GA35323@bluezbox.com> References: <20201105003038.GA56109@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201105003038.GA56109@troutmask.apl.washington.edu> X-Operating-System: FreeBSD/11.2-RELEASE-p10 (amd64) X-Spam-Level: -- X-Spam-Report: Spam detection software, running on the system "id.bluezbox.com", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see The administrator of that system for details. Content preview: Steve Kargl (sgk@troutmask.apl.washington.edu) wrote: > Anyone have a HowTo method for changing a FreeBSD bugzilla > account password, where the current password has been forgotten? > Clicking on 'For [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Rspamd-Queue-Id: 4CRPx338Pdz4VMn X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of gonzo@bluezbox.com designates 45.55.20.155 as permitted sender) smtp.mailfrom=gonzo@bluezbox.com X-Spamd-Result: default: False [-1.28 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[gonzo]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; DMARC_NA(0.00)[bluezbox.com]; MID_RHS_MATCH_FROM(0.00)[]; NEURAL_HAM_LONG(-1.07)[-1.073]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.07)[0.075]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.98)[-0.985]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:14061, ipnet:45.55.0.0/19, country:US]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 00:47:04 -0000 Steve Kargl (sgk@troutmask.apl.washington.edu) wrote: > Anyone have a HowTo method for changing a FreeBSD bugzilla > account password, where the current password has been forgotten? > Clicking on 'Forgot Password' in the login banner results in > a web page asking for the email address. > > The system is not configured to allow password change requests. > > Go to https://bugs.freebsd.org/bugzilla/enter_bug.cgi > one finds again a place to reset the password, but one gets > the above message when any attempt is made. Hi Steve, Did you try reset password for @FreeBSD.org account? Bugzilla uses Kerberos/LDAP passwords for such users. Instruction how to perform reset is here: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/kerberos-ldap.html -- gonzo From owner-freebsd-hackers@freebsd.org Thu Nov 5 02:04:20 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A750A46EE22 for ; Thu, 5 Nov 2020 02:04:20 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRRfC417Bz4cJX for ; Thu, 5 Nov 2020 02:04:19 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.16.1/8.16.1) with ESMTPS id 0A523iV4056404 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 4 Nov 2020 18:03:44 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.16.1/8.16.1/Submit) id 0A523ijZ056403; Wed, 4 Nov 2020 18:03:44 -0800 (PST) (envelope-from sgk) Date: Wed, 4 Nov 2020 18:03:44 -0800 From: Steve Kargl To: Oleksandr Tymoshenko Cc: freebsd-hackers@freebsd.org Subject: Re: Resetting freebsd bugzilla password? Message-ID: <20201105020344.GB56376@troutmask.apl.washington.edu> References: <20201105003038.GA56109@troutmask.apl.washington.edu> <20201105004656.GA35323@bluezbox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201105004656.GA35323@bluezbox.com> X-Rspamd-Queue-Id: 4CRRfC417Bz4cJX X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=washington.edu (policy=none); spf=none (mx1.freebsd.org: domain of sgk@troutmask.apl.washington.edu has no SPF policy when checking 128.95.76.21) smtp.mailfrom=sgk@troutmask.apl.washington.edu X-Spamd-Result: default: False [-0.66 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.980]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.36)[0.358]; NEURAL_HAM_LONG(-1.04)[-1.041]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US]; MAILMAN_DEST(0.00)[freebsd-hackers]; DMARC_POLICY_SOFTFAIL(0.10)[washington.edu : No valid SPF, No valid DKIM, none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 02:04:20 -0000 On Wed, Nov 04, 2020 at 04:46:56PM -0800, Oleksandr Tymoshenko wrote: > Steve Kargl (sgk@troutmask.apl.washington.edu) wrote: > > Anyone have a HowTo method for changing a FreeBSD bugzilla > > account password, where the current password has been forgotten? > > Clicking on 'Forgot Password' in the login banner results in > > a web page asking for the email address. > > > > The system is not configured to allow password change requests. > > > > Go to https://bugs.freebsd.org/bugzilla/enter_bug.cgi > > one finds again a place to reset the password, but one gets > > the above message when any attempt is made. > > Hi Steve, > > Did you try reset password for @FreeBSD.org account? > Bugzilla uses Kerberos/LDAP passwords for such users. > Instruction how to perform reset is here: > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/kerberos-ldap.html > Thanks for the pointer. Sure would have been nice to get a pointer to the URL instead of a generic error messages. -- Steve From owner-freebsd-hackers@freebsd.org Thu Nov 5 06:47:23 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADDEB44C6AB; Thu, 5 Nov 2020 06:47:23 +0000 (UTC) (envelope-from carbaecker@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRYwp3GK5z4r4j; Thu, 5 Nov 2020 06:47:22 +0000 (UTC) (envelope-from carbaecker@gmx.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1604558839; bh=Z6qBi7u8kRLlV+NAl9BsbXYZfr9cQYzsekFoSDwoNHc=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=C4NiT1T9sJSZ3zCnH9mjY9OkoLhrPGOqcvZv9WEHJN2I9QoAKfpJubUAHTKzCmPsh 2p/gWcnvztizVmvoIOSZ7SG85rpfwOBrXcFBYYjOVu1FBwx5LJhd6uQqvVn0jHGegH W8yyAgH++aLY6+nvQK2TYepOFDpa4bHYtf2l+zJE= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.178.30] ([94.31.96.148]) by mail.gmx.com (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MAwbp-1kTiM336Xh-00BJYO; Thu, 05 Nov 2020 07:47:18 +0100 Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) To: Hans Petter Selasky , YongHyeon PYUN , Kristof Provost Cc: freebsd-arm@freebsd.org, freebsd-hackers@freebsd.org References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> <46d08198-530c-cb4b-efa8-4edaf89471c1@selasky.org> <4dfaa9a3-c085-8466-a6e4-19f988b5ed3d@selasky.org> <9e1f9407-e2ff-6486-256e-012626b534fb@selasky.org> From: =?UTF-8?Q?Carsten_B=c3=a4cker?= Message-ID: <5b9e2422-e43c-c1d6-a2ba-5e7544618881@gmx.de> Date: Thu, 5 Nov 2020 07:47:17 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: <9e1f9407-e2ff-6486-256e-012626b534fb@selasky.org> Content-Type: multipart/mixed; boundary="------------EE2C18579A1D26EAA6F28607" Content-Language: de-DE X-Provags-ID: V03:K1:10w26K1BDwRJgz41NysnxpbYBHptBNd3lf6XVDRz9HrQ8agrLCu G8/P8P6n9cKIp3tHA2PwtBuM6k0IlJ1pGSft+5A/mcsMtAx4PNCyrCVQEzhcKgvRh+4NXf8 otSXFS97ybt0QaxXkFYJpt5/Q4HtSs4Yrkk/PgodT7Sta22odIKA5JL7EzObbJ9S0r5jMAp 6iEWSBVqWugdqd3ifgfNg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:HMyUbYUbkos=:rOFQvgzSRV7e0G2hIAPsDu nmS/lwIjRkbSJPMutrjVSWB8BqXGQ8zKe46JJZgkQLObeRkNPlrNZ5/puRnEblWmgwTp5GkdD NA3AaoVJGXRb8duRCjRkbpvaQEUVwEZft0rbstny+t6DYhOyAtMEhT6RhyJpTNh5GfZCM7NkX PaUIPd+r1llvW0jlHFBiKUVMqtw7Ul099FIKjqvC+04gZ5eEKf6yRZTPu86No1B0oCpqyqFZM ABVsLTKwbpB8TJjcMhHifwqDIdElNopQuM0Tn40ZkDEx3Tlk4Kcd0x1uskkX+3g8oJWcNCHyP fWNhkyzPzV14M2WzPZnhEh78h22n7QtoyO9w8ecLgVmVSclJEZe6WeFvOlWKSo5oXCU1nfyc9 agzLjgn6/lIUcpuJ98TjHj1GUZZCU804RzbyXa70ylbzNJrhYWSP1Z5Y9QJTgpjVYX6CmNib1 Eyu3/v/Y0SRxjpUBFjilU3r0g0bA6EfurtOmklp3EKsnxJEUUB54F8aL0oA8f5J9uEyklHt/b D653lfGs7Rm9Qqg4wY0o+ATpMmZZcmk/4bSjDtg3nJH+nXc42vDSK6ml1MxXOb6/16579nsgm +ndzOj/pTOWHSZFXu5RNDT+2eL9eB3qcfqnS2T797XitQsuagvREttWDZjCsKE4A4y2OdpuUg RweLf183X7kFoza4s6ZL6YMxCLLCNtifuVOQJq84SR3fTAbRqluR1GoIfDQZtXd+GPmynqRpb WXX0++AhjIs50c25RGEE9+hi6vuTE8+OoK1eEQ92u4cl4jefRjs7AGFfEXV3l2/PUDMt8qQdL +cKa9Kd9HHlBYJXU/xIOpxRY9re2exNoQyFcVgMTeCRrZB0CT62E38El8Ph3o09T+3HPtPp8I 08eUAfwHmQNpJfwkV0Kw== X-Rspamd-Queue-Id: 4CRYwp3GK5z4r4j X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.b=C4NiT1T9; dmarc=none; spf=pass (mx1.freebsd.org: domain of carbaecker@gmx.de designates 212.227.17.21 as permitted sender) smtp.mailfrom=carbaecker@gmx.de X-Spamd-Result: default: False [-0.90 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmx.de]; R_SPF_ALLOW(-0.20)[+ip4:212.227.17.0/27]; HAS_ATTACHMENT(0.00)[]; MIME_BASE64_TEXT_BOGUS(1.00)[]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[gmx.net:+]; MIME_BASE64_TEXT(0.10)[]; CTYPE_MIXED_BOGUS(1.00)[]; NEURAL_HAM_SHORT(-0.41)[-0.411]; FREEMAIL_TO(0.00)[selasky.org,gmail.com,FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+]; FREEMAIL_ENVFROM(0.00)[gmx.de]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.17.21:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.971]; R_DKIM_ALLOW(-0.20)[gmx.net:s=badeba3b8450]; RECEIVED_SPAMHAUS_PBL(0.00)[94.31.96.148:received]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.020]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; DMARC_NA(0.00)[gmx.de]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.17.21:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers,freebsd-arm] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 06:47:23 -0000 This is a multi-part message in MIME format. --------------EE2C18579A1D26EAA6F28607 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Am 04.11.2020 um 09:52 schrieb Hans Petter Selasky: > On 2020-11-04 07:36, Carsten B=C3=A4cker wrote: >> Am 03.11.2020 um 09:46 schrieb Hans Petter Selasky: >>> On 2020-11-03 09:19, Hans Petter Selasky wrote: >>>> It looks like smsc(4) uses the following RX format but I don't >>>> know actual RX format of H/W(no access to datasheet). >>>> >>>> <---------------------------- actlen >>>> --------------------------------------------------> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 <------------- pktlen ------------------------= > >>>> rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | >>>> partial checksum(2 bytes) >>> >>> Hi, >>> >>> I wonder if the checksum is zero, when not valid, and that we should >>> check for this in the driver! >>> >>> Can you try this patch? >>> >>> Also enabling debugging in the SMSC driver would be useful. >>> >>> --HPS >>> >>> _______________________________________________ >>> freebsd-hackers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> To unsubscribe, send any mail to >>> "freebsd-hackers-unsubscribe@freebsd.org" >> Hi, >> >> i applied the patch on -CURRENT but got a panic right after loading the >> kernel. Most likely an unrelated problem. >> >> But i was able to apply the patch on releng/12.2 (with an offset). >> Unfortunately it doesn't change the previously described behavior with >> rxcsum and i didn't manage to get any reasonable debug-output. >> >> Since i can easily reproduce the problem. How else can i help? >> > > Can you enable the debug knob for SMSC: > > sysctl hw.usb.smsc.debug=3D16 > > It will print the RX checksum per frame received. Just do this with > the unpatched driver while doing some traffic. Capture log messages > from dmesg or /var/log/messages. > > --HPS > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" Hi, please see the attached the logs. Following command was issued from host + jail: "sysctl hw.usb.smsc.debug=3D16 && sleep 1 && [jexec www-public] ping -c 1 heise.de && sleep 1 && sysctl hw.usb.smsc.debug=3D0" The host-log doesn't show any "RX checksum offloaded", probably the packet length... Changes from the patch have been reverted. Kernel config has been modified to include USB_DEBUG. While there i removed "smsc", so we can load it as a module for easier testing. Meanwhile i upgraded to 12.2: FreeBSD 12.2-RELEASE #5 r367195M: Thu Nov=C2=A0 5 05:52:39 UTC 2020 root@sysbuild:/usr/obj/usr/src_12.2/arm64.aarch64/sys/GENERIC_DBG arm64 aarch64 Regards, Carsten --------------EE2C18579A1D26EAA6F28607 Content-Type: text/plain; charset=UTF-8; name="host_rxcsum" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="host_rxcsum" Ck5vdiAgNSAwNjoxODozNiBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFj dGxlbiAxMTYKTm92ICA1IDA2OjE4OjM2IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogcnhoZHIgMHgwMDZlMDAyMCA6IHBrdGxlbiAxMTAgOiBhY3RsZW4gMTE2IDogb2Zm IDYKTm92ICA1IDA2OjE4OjM2IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog YWN0bGVuIDExMApOb3YgIDUgMDY6MTg6MzYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiByeGhkciAweDAwNjgwMDIwIDogcGt0bGVuIDEwNCA6IGFjdGxlbiAxMTAgOiBv ZmYgNgpOb3YgIDUgMDY6MTg6MzYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzogcngg OiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjM2IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2NiA6IGFjdGxlbiA3MiA6IG9m ZiA2Ck5vdiAgNSAwNjoxODozNiBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTg6MzYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2IDogYWN0bGVuIDcyIDogb2Zm IDYK --------------EE2C18579A1D26EAA6F28607 Content-Type: text/plain; charset=UTF-8; name="jail_norxcsum" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="jail_norxcsum" Ck5vdiAgNSAwNjoxOToyNiBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFj dGxlbiAyMDAKTm92ICA1IDA2OjE5OjI2IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogcnhoZHIgMHgwMGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2Zm IDYKTm92ICA1IDA2OjE5OjI2IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog YWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBv ZmYgNgpOb3YgIDUgMDY6MTk6MjYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzogcngg OiBhY3RsZW4gMzI0Ck5vdiAgNSAwNjoxOToyNiBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiByeCA6IHJ4aGRyIDB4MDEzZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6 IG9mZiA2Ck5vdiAgNSAwNjoxOToyNiBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IGFjdGxlbiAxMDAKTm92ICA1IDA2OjE5OjI2IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogcnhoZHIgMHgwMDVlMDAyMCA6IHBrdGxlbiA5NCA6IGFjdGxlbiAxMDAg OiBvZmYgNgpOb3YgIDUgMDY6MTk6MjYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiBhY3RsZW4gMjAwCk5vdiAgNSAwNjoxOToyNiBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIw MCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNiBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IGFjdGxlbiAxOTYKTm92ICA1IDA2OjE5OjI2IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGJlMDAyMCA6IHBrdGxlbiAxOTAgOiBhY3RsZW4g MTk2IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI2IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogYWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjYgZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxl biAzMjQgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjYgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiBhY3RsZW4gNzAKTm92ICA1IDA2OjE5OjI2IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQwMjQyMCA6IHBrdGxlbiA2NCA6IGFjdGxl biA3MCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiByeCA6IGFjdGxlbiAxOTYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGJlMDAyMCA6IHBrdGxlbiAxOTAgOiBhY3Rs ZW4gMTk2IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogYWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFj dGxlbiAzMjQgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAwCk5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDog YWN0bGVuIDIwMCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTgg OiBhY3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMx OCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI0Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDEzZTAwMjAgOiBwa3RsZW4g MzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDAKTm92ICA1IDA2OjE5OjI3IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGMyMDAyMCA6IHBrdGxl biAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjcgZ2Vu ZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0 bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAwCk5vdiAgNSAwNjoxOToyNyBn ZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBw a3RsZW4gMTk0IDogYWN0bGVuIDIwMCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3 IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6 IHBrdGxlbiAzMTggOiBhY3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6 MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzIwMDIw IDogcGt0bGVuIDE5NCA6IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2Vu ZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI0Ck5vdiAgNSAwNjox OToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDEzZTAw MjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBn ZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2 OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNl MDAyMCA6IHBrdGxlbiAzMTggOiBhY3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3 IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUg MDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAw YzIwMDIwIDogcGt0bGVuIDE5NCA6IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6 MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI0Ck5vdiAg NSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4 MDEzZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAwNjox OToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDAKTm92 ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIg MHgwMGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYKTm92ICA1IDA2 OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNApO b3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhk ciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpOb3YgIDUg MDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAw Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4 aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIwMCA6IG9mZiA2Ck5vdiAg NSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAx OTYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog cnhoZHIgMHgwMGJlMDAyMCA6IHBrdGxlbiAxOTAgOiBhY3RsZW4gMTk2IDogb2ZmIDYKTm92 ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVu IDMyNApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzogcngg OiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpO b3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3Rs ZW4gMjAwCk5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIwMCA6IG9mZiA2 Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFj dGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBhY3RsZW4gMzI0IDogb2Zm IDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog YWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiByeGhkciAweDAwYzIwMDIwIDogcGt0bGVuIDE5NCA6IGFjdGxlbiAyMDAgOiBv ZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzogcngg OiBhY3RsZW4gMzI0Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IApO b3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDEz ZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToy NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDAKTm92ICA1 IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgw MGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYKTm92ICA1IDA2OjE5 OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNApOb3Yg IDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAw eDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpOb3YgIDUgMDY6 MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI0Ck5v diAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRy IDB4MDEzZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAw NjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDAK Tm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnho ZHIgMHgwMGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYKTm92ICA1 IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMy NApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBy eGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpOb3Yg IDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4g MjAwCk5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIwMCA6IG9mZiA2Ck5v diAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxl biAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBhY3RsZW4gMzI0IDogb2ZmIDYK Tm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0 bGVuIDIwMApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiByeGhkciAweDAwYzIwMDIwIDogcGt0bGVuIDE5NCA6IGFjdGxlbiAyMDAgOiBvZmYg NgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBh Y3RsZW4gMTk2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IHJ4aGRyIDB4MDBiZTAwMjAgOiBwa3RsZW4gMTkwIDogYWN0bGVuIDE5NiA6IG9m ZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBhY3RsZW4gMzI0IDog b2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogYWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQg OiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiBhY3RsZW4gMjAwCk5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIw MCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBhY3RsZW4g MzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzIwMDIwIDogcGt0bGVuIDE5NCA6IGFjdGxl biAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiBhY3RsZW4gMTk2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBiZTAwMjAgOiBwa3RsZW4gMTkwIDogYWN0 bGVuIDE5NiA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBh Y3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzIwMDIwIDogcGt0bGVuIDE5NCA6 IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBiZTAwMjAgOiBwa3RsZW4gMTkw IDogYWN0bGVuIDE5NiA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAz MTggOiBhY3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzIwMDIwIDogcGt0bGVu IDE5NCA6IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk2Ck5vdiAgNSAwNjoxOToyNyBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBiZTAwMjAgOiBwa3Rs ZW4gMTkwIDogYWN0bGVuIDE5NiA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI3IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBr dGxlbiAzMTggOiBhY3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6Mjcg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzIwMDIwIDog cGt0bGVuIDE5NCA6IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI0Ck5vdiAgNSAwNjoxOToy NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDEzZTAwMjAg OiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDAKTm92ICA1IDA2OjE5 OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGMyMDAy MCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDE5NgpOb3YgIDUgMDY6 MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYmUw MDIwIDogcGt0bGVuIDE5MCA6IGFjdGxlbiAxOTYgOiBvZmYgNgpOb3YgIDUgMDY6MTk6Mjcg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI0Ck5vdiAgNSAw NjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDEz ZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToy NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDAKTm92ICA1 IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgw MGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYKTm92ICA1IDA2OjE5 OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNApOb3Yg IDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAw eDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYgNgpOb3YgIDUgMDY6 MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAwCk5v diAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRy IDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIwMCA6IG9mZiA2Ck5vdiAgNSAw NjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQK Tm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnho ZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBhY3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1 IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIw MApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBy eGhkciAweDAwYzIwMDIwIDogcGt0bGVuIDE5NCA6IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3Yg IDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4g MzI0Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IHJ4aGRyIDB4MDEzZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMyNCA6IG9mZiA2Ck5v diAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxl biAyMDAKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogcnhoZHIgMHgwMGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4gMjAwIDogb2ZmIDYK Tm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0 bGVuIDMyNApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQgOiBvZmYg NgpOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBh Y3RsZW4gMjAwCk5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0bGVuIDIwMCA6IG9m ZiA2Ck5vdiAgNSAwNjoxOToyNyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IGFjdGxlbiAxOTYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogcnhoZHIgMHgwMGJlMDAyMCA6IHBrdGxlbiAxOTAgOiBhY3RsZW4gMTk2IDog b2ZmIDYKTm92ICA1IDA2OjE5OjI3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogYWN0bGVuIDMyNApOb3YgIDUgMDY6MTk6MjcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiByeGhkciAweDAxM2UwMDIwIDogcGt0bGVuIDMxOCA6IGFjdGxlbiAzMjQg OiBvZmYgNgpOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiBhY3RsZW4gMzI0Ck5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDEzZTAwMjAgOiBwa3RsZW4gMzE4IDogYWN0bGVuIDMy NCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IGFjdGxlbiAyMDAKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGMyMDAyMCA6IHBrdGxlbiAxOTQgOiBhY3RsZW4g MjAwIDogb2ZmIDYKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogYWN0bGVuIDE5MgpOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYmEwMDIwIDogcGt0bGVuIDE4NiA6IGFjdGxl biAxOTIgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiBhY3RsZW4gMjAwCk5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMjAwMjAgOiBwa3RsZW4gMTk0IDogYWN0 bGVuIDIwMCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IGFjdGxlbiAzMjQKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTNlMDAyMCA6IHBrdGxlbiAzMTggOiBh Y3RsZW4gMzI0IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMApOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzIwMDIwIDogcGt0bGVuIDE5NCA6 IGFjdGxlbiAyMDAgOiBvZmYgNgpOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gOTQKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDU4MDAyMCA6IHBrdGxlbiA4OCA6 IGFjdGxlbiA5NCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IGFjdGxlbiAxMDgKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDY2MDAyMCA6IHBrdGxlbiAxMDIg OiBhY3RsZW4gMTA4IDogb2ZmIDYKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDEwMApOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNWUwMDIwIDogcGt0bGVuIDk0 IDogYWN0bGVuIDEwMCA6IG9mZiA2Ck5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiA3MApOb3YgIDUgMDY6MTk6MjggZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDAyNDIwIDogcGt0bGVuIDY0 IDogYWN0bGVuIDcwIDogb2ZmIDYKTm92ICA1IDA2OjE5OjI4IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDcwCk5vdiAgNSAwNjoxOToyOCBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA0MDI0MjAgOiBwa3RsZW4gNjQg OiBhY3RsZW4gNzAgOiBvZmYgNgo= --------------EE2C18579A1D26EAA6F28607 Content-Type: text/plain; charset=UTF-8; name="jail_rxcsum" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="jail_rxcsum" Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFj dGxlbiA3MgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2IDogYWN0bGVuIDcyIDogb2ZmIDYK Tm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0 bGVuIDcyCk5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IHJ4aGRyIDB4MDA0MjI0MjAgOiBwa3RsZW4gNjYgOiBhY3RsZW4gNzIgOiBvZmYgNgpO b3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3Rs ZW4gNzIKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2NiA6IGFjdGxlbiA3MiA6IG9mZiA2Ck5v diAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxl biAyMDIKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYK Tm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNr c3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4g MzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1 IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMy NgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBy eGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3Yg IDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0g b2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYg OiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6 MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk4Ck5v diAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRy IDB4MDBjMDAwMjAgOiBwa3RsZW4gMTkyIDogYWN0bGVuIDE5OCA6IG9mZiA2Ck5vdiAgNSAw NjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZs b2FkZWQgKDB4MWEwMikKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFj dGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1 NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUg MDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAw NDIwNDIwIDogcGt0bGVuIDY2IDogYWN0bGVuIDcyIDogb2ZmIDYKTm92ICA1IDA2OjE3OjU3 IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUg MDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAx NDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6 NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVk ICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IGFjdGxlbiAxOTgKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGMwMDAyMCA6IHBrdGxlbiAxOTIgOiBhY3RsZW4g MTk4IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxYTAyKQpOb3YgIDUgMDY6MTc6NTcgZ2Vu ZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjox Nzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAw MjAgOiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1NyBn ZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4 MTk4MikKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogYWN0bGVuIDIwMgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiByeGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIg OiBvZmYgNgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog UlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1NyBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU3 IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6 IHBrdGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU3IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgy KQpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBh Y3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9m ZiA2Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBj aGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTcgZ2Vu ZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0 bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5v diAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxl biAyMDIKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4 IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYK Tm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNr c3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4g MzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1 IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIw MgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBy eGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3Yg IDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0g b2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAzMjAg OiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUgMDY6 MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAyCk5v diAgNSAwNjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRy IDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVuIDIwMiA6IG9mZiA2Ck5vdiAgNSAw NjoxNzo1NyBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZs b2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE3OjU3IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFj dGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTcgZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1 OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1 IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgw MTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3 OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRl ZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiBhY3RsZW4gMjAyCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVu IDIwMiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE3OjU4IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6 MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAw MDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTgg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgw eDE5ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAy IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1 OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAg OiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4 MikKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog YWN0bGVuIDIwMgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiByeGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBv ZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlgg Y2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU4IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBr dGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpO b3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3Rs ZW4gMzI2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2 Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVj a3N1bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVu IDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAg NSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAx OTgKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog cnhoZHIgMHgwMGMwMDAyMCA6IHBrdGxlbiAxOTIgOiBhY3RsZW4gMTk4IDogb2ZmIDYKTm92 ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3Vt IG9mZmxvYWRlZCAoMHgxYTAyKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIw IDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2 OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMgpO b3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhk ciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUg MDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2Zm bG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBh Y3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTc6 NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1 IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgw MDQyMjQyMCA6IHBrdGxlbiA2NiA6IGFjdGxlbiA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1 OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIKTm92ICA1 IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgw MGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1IDA2OjE3 OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRl ZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIwIDogYWN0bGVu IDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2OjE3OjU4IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMgpOb3YgIDUgMDY6 MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzQw MDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTgg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgw eDE5ZmUpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBhY3RsZW4gMzI2 IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1 OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAg OiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4 MikKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog YWN0bGVuIDIwMgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiAKTm92 ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzQw MDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTgg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgw eDE5ZmUpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDog Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogZGVidWc6IHJ4IDogcnhoZHIgMHgw MTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3 OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRl ZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiBhY3RsZW4gMjAyCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVu IDIwMiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE3OjU4IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6 MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAw MDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTgg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgw eDE5ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAy IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk4Ck5vdiAgNSAwNjoxNzo1 OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMDAwMjAg OiBwa3RsZW4gMTkyIDogYWN0bGVuIDE5OCA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MWEw MikKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog YWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBv ZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlgg Y2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU4IGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBr dGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpO b3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3Rs ZW4gMjAyCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBy eCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVuIDIwMiA6IG9mZiA2 Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVj a3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVu IDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAg NSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAy MDIKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDog cnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92 ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3Vt IG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk4Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMDAwMjAgOiBwa3RsZW4gMTky IDogYWN0bGVuIDE5OCA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MWEwMikKTm92ICA1IDA2 OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpO b3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhk ciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUg MDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2Zm bG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBh Y3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6 NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk4Ck5vdiAg NSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4 MDBjMDAwMjAgOiBwa3RsZW4gMTkyIDogYWN0bGVuIDE5OCA6IG9mZiA2Ck5vdiAgNSAwNjox Nzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2Fk ZWQgKDB4MWEwMikKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxl biAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1OCBn ZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIKTm92ICA1IDA2 OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGM0 MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4 IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAo MHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiBhY3RsZW4gMTk4Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjMDAwMjAgOiBwa3RsZW4gMTkyIDogYWN0bGVuIDE5 OCA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MWEwMikKTm92ICA1IDA2OjE3OjU4IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6 NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIw IDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2Vu ZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5 ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDog b2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJY IGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1OCBn ZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBw a3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikK Tm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0 bGVuIDIwMgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiByeGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYg NgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hl Y2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAxOTgKTm92ICA1IDA2OjE3OjU4IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGMwMDAyMCA6IHBrdGxl biAxOTIgOiBhY3RsZW4gMTk4IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxYTAyKQpOb3Yg IDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4g MzI2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5v diAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1 bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5 NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAw NjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYK Tm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnho ZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1 IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9m ZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAyCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDog YWN0bGVuIDIwMiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE3 OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3Yg IDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAw eDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6 MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9h ZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiByeCA6IGFjdGxlbiAxNDQKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2NiA6IGFjdGxl biAxNDQgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2IDogYWN0bGVuIDE0NCA6 IG9mZiA3OApOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiBhY3RsZW4gMjAyCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVuIDIw MiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVn OiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE3OjU4IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6 NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIw IDogcGt0bGVuIDMyMCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTggZ2Vu ZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5 ODIpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVi dWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDog b2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJY IGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2Ck5vdiAgNSAwNjoxNzo1OCBn ZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDE0MDAwMjAgOiBw a3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OCBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTk4MikK Tm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0 bGVuIDIwMgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog cnggOiByeGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYg NgpOb3YgIDUgMDY6MTc6NTggZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hl Y2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1OCBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAxOTgKTm92ICA1IDA2OjE3OjU4IGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGMwMDAyMCA6IHBrdGxl biAxOTIgOiBhY3RsZW4gMTk4IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU4IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxYTAyKQpOb3Yg IDUgMDY6MTc6NTkgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4g MzI2Ck5vdiAgNSAwNjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6 IHJ4aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5v diAgNSAwNjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1 bSBvZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2OjE3OjU5IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTc6NTkgZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMy MCA6IGFjdGxlbiAzMjYgOiBvZmYgNgpOb3YgIDUgMDY6MTc6NTkgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAw NjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIK Tm92ICA1IDA2OjE3OjU5IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnho ZHIgMHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1 IDA2OjE3OjU5IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9m ZmxvYWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTc6NTkgZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk0Ck5vdiAgNSAwNjoxNzo1OSBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBiYzAwMjAgOiBwa3RsZW4gMTg4IDog YWN0bGVuIDE5NCA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MWEwNikKTm92ICA1IDA2OjE3 OjU5IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMgpOb3Yg IDUgMDY6MTc6NTkgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAw eDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUgMDY6 MTc6NTkgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9h ZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE3OjU5IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBhY3Rs ZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE3OjU5IGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTc6NTkg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAyCk5vdiAgNSAw NjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBj NDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVuIDIwMiA6IG9mZiA2Ck5vdiAgNSAwNjoxNzo1 OSBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQg KDB4MTlmZSkKTm92ICA1IDA2OjE3OjU5IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogYWN0bGVuIDk2Ck5vdiAgNSAwNjoxNzo1OSBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA1YTAwMjAgOiBwa3RsZW4gOTAgOiBhY3RsZW4gOTYg OiBvZmYgNgpOb3YgIDUgMDY6MTc6NTkgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzog UlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDFhNjYpCk5vdiAgNSAwNjoxNzo1OSBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTc6NTkg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDog cGt0bGVuIDY2IDogYWN0bGVuIDcyIDogb2ZmIDYKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMgpOb3YgIDUgMDY6MTg6MDAg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzQwMDIwIDog cGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUp Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFj dGxlbiAzMjYKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAzMjAgOiBhY3RsZW4gMzI2IDogb2Zm IDYKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNo ZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMjAyCk5vdiAgNSAwNjoxODowMCBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3Rs ZW4gMTk2IDogYWN0bGVuIDIwMiA6IG9mZiA2Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92 ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVu IDIwMgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzogcngg OiByeGhkciAweDAwYzQwMDIwIDogcGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpO b3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tz dW0gb2ZmbG9hZGVkICgweDE5ZmUpCk5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAzMjYKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMTQwMDAyMCA6IHBrdGxlbiAz MjAgOiBhY3RsZW4gMzI2IDogb2ZmIDYKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgxOTgyKQpOb3YgIDUg MDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMzI2 Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4 aGRyIDB4MDE0MDAwMjAgOiBwa3RsZW4gMzIwIDogYWN0bGVuIDMyNiA6IG9mZiA2Ck5vdiAg NSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBv ZmZsb2FkZWQgKDB4MTk4MikKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogYWN0bGVuIDE5NApOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYmMwMDIwIDogcGt0bGVuIDE4OCA6 IGFjdGxlbiAxOTQgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDFhMDYpCk5vdiAgNSAwNjox ODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIKTm92 ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIg MHgwMGM0MDAyMCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1IDA2 OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxv YWRlZCAoMHgxOWZlKQpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiBhY3RsZW4gMTY2Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBhMDAwMjAgOiBwa3RsZW4gMTYwIDogYWN0 bGVuIDE2NiA6IG9mZiA2Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MWEyMikKTm92ICA1IDA2OjE4OjAw IGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDcyCk5vdiAgNSAw NjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA0 MjI0MjAgOiBwa3RsZW4gNjYgOiBhY3RsZW4gNzIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDAg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gMTk0Ck5vdiAgNSAw NjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDBi YzAwMjAgOiBwa3RsZW4gMTg4IDogYWN0bGVuIDE5NCA6IG9mZiA2Ck5vdiAgNSAwNjoxODow MCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBSWCBjaGVja3N1bSBvZmZsb2FkZWQg KDB4MWEwNikKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogYWN0bGVuIDMyNgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogcnggOiByeGhkciAweDAxNDAwMDIwIDogcGt0bGVuIDMyMCA6IGFjdGxlbiAz MjYgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1 ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ODIpCk5vdiAgNSAwNjoxODowMCBnZW5l cmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiAyMDIKTm92ICA1IDA2OjE4 OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMGM0MDAy MCA6IHBrdGxlbiAxOTYgOiBhY3RsZW4gMjAyIDogb2ZmIDYKTm92ICA1IDA2OjE4OjAwIGdl bmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNoZWNrc3VtIG9mZmxvYWRlZCAoMHgx OWZlKQpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1Zzogcngg OiBhY3RsZW4gMjAyCk5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiByeCA6IHJ4aGRyIDB4MDBjNDAwMjAgOiBwa3RsZW4gMTk2IDogYWN0bGVuIDIwMiA6 IG9mZiA2Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiBS WCBjaGVja3N1bSBvZmZsb2FkZWQgKDB4MTlmZSkKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDIwMgpOb3YgIDUgMDY6MTg6MDAg Z2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwYzQwMDIwIDog cGt0bGVuIDE5NiA6IGFjdGxlbiAyMDIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogUlggY2hlY2tzdW0gb2ZmbG9hZGVkICgweDE5ZmUp Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IGFj dGxlbiAxOTgKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6 IHJ4IDogcnhoZHIgMHgwMGMwMDAyMCA6IHBrdGxlbiAxOTIgOiBhY3RsZW4gMTk4IDogb2Zm IDYKTm92ICA1IDA2OjE4OjAwIGdlbmVyaWMga2VybmVsOiBzbXNjMDogZGVidWc6IFJYIGNo ZWNrc3VtIG9mZmxvYWRlZCAoMHgxYTAyKQpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjAwIGdlbmVy aWMga2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxl biA2NiA6IGFjdGxlbiA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxODowMCBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTg6MDAgZ2VuZXJp YyBrZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVu IDY2IDogYWN0bGVuIDcyIDogb2ZmIDYKTm92ICA1IDA2OjE4OjAyIGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDcyCk5vdiAgNSAwNjoxODowMiBnZW5lcmlj IGtlcm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA0MjI0MjAgOiBwa3RsZW4g NjYgOiBhY3RsZW4gNzIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDIgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjAyIGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2 NiA6IGFjdGxlbiA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxODowNCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiA5NgpOb3YgIDUgMDY6MTg6MDQgZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNWEwMDIwIDogcGt0bGVuIDkw IDogYWN0bGVuIDk2IDogb2ZmIDYKTm92ICA1IDA2OjE4OjA0IGdlbmVyaWMga2VybmVsOiBz bXNjMDogCk5vdiAgNSAwNjoxODowNCBnZW5lcmljIGtlcm5lbDogZGVidWc6IFJYIGNoZWNr c3VtIG9mZmxvYWRlZCAoMHgxYTY2KQpOb3YgIDUgMDY6MTg6MDQgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjA0IGdlbmVyaWMg a2VybmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2 NiA6IGFjdGxlbiA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxODowNCBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTg6MDQgZ2VuZXJpYyBr ZXJuZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2 IDogYWN0bGVuIDcyIDogb2ZmIDYKTm92ICA1IDA2OjE4OjA2IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogYWN0bGVuIDcyCk5vdiAgNSAwNjoxODowNiBnZW5lcmljIGtl cm5lbDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA0MjI0MjAgOiBwa3RsZW4gNjYg OiBhY3RsZW4gNzIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MDYgZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjA2IGdlbmVyaWMga2Vy bmVsOiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2NiA6 IGFjdGxlbiA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxODowOCBnZW5lcmljIGtlcm5lbDogc21z YzA6IGRlYnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTg6MDggZ2VuZXJpYyBrZXJu ZWw6IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2IDog YWN0bGVuIDcyIDogb2ZmIDYKTm92ICA1IDA2OjE4OjEwIGdlbmVyaWMga2VybmVsOiBzbXNj MDogZGVidWc6IHJ4IDogYWN0bGVuIDcyCk5vdiAgNSAwNjoxODoxMCBnZW5lcmljIGtlcm5l bDogc21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA0MjI0MjAgOiBwa3RsZW4gNjYgOiBh Y3RsZW4gNzIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MTAgZ2VuZXJpYyBrZXJuZWw6IHNtc2Mw OiBkZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjEwIGdlbmVyaWMga2VybmVs OiBzbXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2NiA6IGFj dGxlbiA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxODoxMiBnZW5lcmljIGtlcm5lbDogc21zYzA6 IGRlYnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTg6MTIgZ2VuZXJpYyBrZXJuZWw6 IHNtc2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2IDogYWN0 bGVuIDcyIDogb2ZmIDYKTm92ICA1IDA2OjE4OjEyIGdlbmVyaWMga2VybmVsOiBzbXNjMDog ZGVidWc6IHJ4IDogYWN0bGVuIDcyCk5vdiAgNSAwNjoxODoxMiBnZW5lcmljIGtlcm5lbDog c21zYzA6IGRlYnVnOiByeCA6IHJ4aGRyIDB4MDA0MjI0MjAgOiBwa3RsZW4gNjYgOiBhY3Rs ZW4gNzIgOiBvZmYgNgpOb3YgIDUgMDY6MTg6MTQgZ2VuZXJpYyBrZXJuZWw6IHNtc2MwOiBk ZWJ1ZzogcnggOiBhY3RsZW4gNzIKTm92ICA1IDA2OjE4OjE0IGdlbmVyaWMga2VybmVsOiBz bXNjMDogZGVidWc6IHJ4IDogcnhoZHIgMHgwMDQyMjQyMCA6IHBrdGxlbiA2NiA6IGFjdGxl biA3MiA6IG9mZiA2Ck5vdiAgNSAwNjoxODoxNCBnZW5lcmljIGtlcm5lbDogc21zYzA6IGRl YnVnOiByeCA6IGFjdGxlbiA3MgpOb3YgIDUgMDY6MTg6MTQgZ2VuZXJpYyBrZXJuZWw6IHNt c2MwOiBkZWJ1ZzogcnggOiByeGhkciAweDAwNDIyNDIwIDogcGt0bGVuIDY2IDogYWN0bGVu IDcyIDogb2ZmIDYK --------------EE2C18579A1D26EAA6F28607-- From owner-freebsd-hackers@freebsd.org Thu Nov 5 12:20:40 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CF852467005 for ; Thu, 5 Nov 2020 12:20:40 +0000 (UTC) (envelope-from pjfloyd@wanadoo.fr) Received: from smtp.smtpout.orange.fr (smtp07.smtpout.orange.fr [80.12.242.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "Bizanga Labs SMTP Client Certificate", Issuer "Bizanga Labs CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CRjKN0d6nz4cNm for ; Thu, 5 Nov 2020 12:20:39 +0000 (UTC) (envelope-from pjfloyd@wanadoo.fr) Received: from [192.168.1.17] ([2.7.192.66]) by mwinf5d42 with ME id ocLe230061SQe2i03cLezE; Thu, 05 Nov 2020 13:20:38 +0100 X-ME-Helo: [192.168.1.17] X-ME-Auth: cGpmbG95ZEB3YW5hZG9vLmZy X-ME-Date: Thu, 05 Nov 2020 13:20:38 +0100 X-ME-IP: 2.7.192.66 Subject: Re: Extra memory mapping seen on freebsd-12 which was not seen in freebsd-11 To: freebsd-hackers@freebsd.org References: <9CCF59F6-06F2-4352-94E5-C508E165D0C2@wanadoo.fr> From: Paul Floyd Message-ID: <88f9ce22-2c31-9e17-9aa8-60347816800f@wanadoo.fr> Date: Thu, 5 Nov 2020 13:20:37 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4CRjKN0d6nz4cNm X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of pjfloyd@wanadoo.fr has no SPF policy when checking 80.12.242.129) smtp.mailfrom=pjfloyd@wanadoo.fr X-Spamd-Result: default: False [1.90 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[80.12.242.129:from]; FREEMAIL_FROM(0.00)[wanadoo.fr]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[wanadoo.fr]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[80.12.242.129:from:127.0.2.255]; RWL_MAILSPIKE_POSSIBLE(0.00)[80.12.242.129:from]; NEURAL_SPAM_SHORT(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[80.12.242.129:from]; RECEIVED_SPAMHAUS_PBL(0.00)[2.7.192.66:received]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[wanadoo.fr]; ASN(0.00)[asn:3215, ipnet:80.12.240.0/20, country:FR]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 12:20:40 -0000 On 10/17/20 6:29 AM, karnajit wangkhem wrote: > Hi Paul, > > The mappings of these applications existed prior to the guard change, which > was fine as no mapping existed on the memory range. With migration to > stable 12, I was doubting that these mappings are no longer correct. But at > the same time, does valgrind have to own this segment, which only came post > certain freebsd releases? Hi I'm not sure if Valgrind needs to own the guest stack in order to be able to grow it. Could you open a bugzilla item for this (on valgrind-devel)? That way I'll be sure not to forget about it. A+ Paul From owner-freebsd-hackers@freebsd.org Fri Nov 6 10:23:57 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 25D0E2E903C for ; Fri, 6 Nov 2020 10:23:57 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CSGhD5zFhz4v7x for ; Fri, 6 Nov 2020 10:23:56 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: by mailman.nyi.freebsd.org (Postfix) id CD1152E8FAE; Fri, 6 Nov 2020 10:23:56 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CCDB92E9303 for ; Fri, 6 Nov 2020 10:23:56 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from s2gw.ddhf.dk (s2gw.ddhf.dk [130.226.214.244]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CSGhC55Qkz4vKZ for ; Fri, 6 Nov 2020 10:23:55 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by s2gw.ddhf.dk (Postfix) with ESMTPS id B91B8C3F3A for ; Fri, 6 Nov 2020 10:23:36 +0000 (UTC) Received: (from phk@localhost) by critter.freebsd.dk (8.16.1/8.16.1/Submit) id 0A6ANorU057255; Fri, 6 Nov 2020 10:23:50 GMT (envelope-from phk) To: hackers@freebsd.org Subject: Have trick, not sure where to document... From: Poul-Henning Kamp MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <57253.1604658230.1@critter.freebsd.dk> Date: Fri, 06 Nov 2020 10:23:50 +0000 Message-ID: <57254.1604658230@critter.freebsd.dk> X-Rspamd-Queue-Id: 4CSGhC55Qkz4vKZ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of phk@critter.freebsd.dk designates 130.226.214.244 as permitted sender) smtp.mailfrom=phk@critter.freebsd.dk X-Spamd-Result: default: False [0.14 / 15.00]; RCVD_TLS_ALL(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[130.226.214.244:from]; FREEFALL_USER(0.00)[phk]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[hackers@freebsd.org]; NEURAL_SPAM_MEDIUM(0.47)[0.469]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[130.226.214.244:from:127.0.2.255]; ARC_NA(0.00)[]; NEURAL_HAM_SHORT(-0.33)[-0.329]; DMARC_NA(0.00)[freebsd.dk]; FORGED_SENDER(0.30)[phk@phk.freebsd.dk,phk@critter.freebsd.dk]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1835, ipnet:130.226.0.0/16, country:EU]; FROM_NEQ_ENVFROM(0.00)[phk@phk.freebsd.dk,phk@critter.freebsd.dk]; MAILMAN_DEST(0.00)[hackers]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 10:23:57 -0000 I had some trouble with a particular remote site, and wanted to have the hardware watchdog protect the boot-sequence, but the BIOS had no support for this. The solution I found was to use the UEFI shell to arm the watchdog before the actuall boot. After digging around in the abysmal UEFI ecosystem some time, I ended up with a bunch of "MM" commands in startup.nsh, before it launches BOOTx64.efi: MM 4e ;IO :87 -n MM 4e ;IO :87 -n MM 4e ;IO :20 -n MM 4f ;IO -n MM 4e ;IO :21 -n MM 4f ;IO -n [...] MM 4e ;IO :aa -n BOOTx64.efi This seems like a genuinely useful trick to have in the tool-box and now I'm trying to find out where in our documentation something like this belongs ? It seems too specialized and with too many sharp edges for the handbook ? Wiki/UEFI seems rather historical... Suggestions ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-hackers@freebsd.org Fri Nov 6 11:02:58 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 333962EA171 for ; Fri, 6 Nov 2020 11:02:58 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CSHYF6NqDz3Cd3 for ; Fri, 6 Nov 2020 11:02:57 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id DB4742E9FF1; Fri, 6 Nov 2020 11:02:57 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DB0E72EA170 for ; Fri, 6 Nov 2020 11:02:57 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CSHYF5pb7z3CgH for ; Fri, 6 Nov 2020 11:02:57 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-ed1-f52.google.com with SMTP id k9so831984edo.5 for ; Fri, 06 Nov 2020 03:02:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=xAwLqVncggMgP5jot2o04EQBiwEv6tLAxJ4jCOFPnFQ=; b=WBPKxpcnVps3VPAx8FwxPK2uS231KzmdGkEW1WEDJFb+amdhynB5aRruFazrBH4+1E I5yvvQZGXSTjDcSawaDyJ5wbwaGrkR6k/Oy5PfYYHuHs8D959GNrhpDgxY7hZknH0YO7 J4mIdD9vq1tHSspzjZcNuZhZgq0zQPjXOCMpJ64jeHUK+UIK1gpzleQcd+bl0tSAvJa7 wBqJFvInG4xJOvqOs25EzRWiX4v6eL2IDSKeoUCzGZiE6TZojBnArNs1NKOHq+n+Nb2d qjfTrIJ6TBxnL16qOO6psa8a1RSnbuWx0iOjmWmbx6CeXwQHVCePA5t+RtuLzufnhmqQ K+Lw== X-Gm-Message-State: AOAM530BtDl8AmrfaO7bwkQAWEUo0rVLIPVmeC4ncDyAkLV/q9pL48pv H1V3xm5kS+v/Cc98mcEA+spnVZ4AtO0= X-Google-Smtp-Source: ABdhPJzZ9x6Lswp6DQ7aesWJ8RFQo4JP3mxKimh7s1xweXtRCBrJfjG6oRF3ATHnLaFuOfwrTNopNg== X-Received: by 2002:a05:6402:17ac:: with SMTP id j12mr1364082edy.31.1604660576249; Fri, 06 Nov 2020 03:02:56 -0800 (PST) Received: from ?IPv6:2a02:8109:98c0:1bc0:5e5f:67ff:fef4:ffd8? ([2a02:8109:98c0:1bc0:5e5f:67ff:fef4:ffd8]) by smtp.gmail.com with ESMTPSA id j1sm622277ejd.47.2020.11.06.03.02.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 Nov 2020 03:02:55 -0800 (PST) Subject: Re: Have trick, not sure where to document... To: Poul-Henning Kamp , hackers@freebsd.org References: <57254.1604658230@critter.freebsd.dk> From: Mateusz Piotrowski <0mp@FreeBSD.org> Message-ID: <6225bae3-0bb4-bd14-51f3-de72eaa4dc98@FreeBSD.org> Date: Fri, 6 Nov 2020 12:02:56 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: <57254.1604658230@critter.freebsd.dk> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4CSHYF5pb7z3CgH X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 11:02:58 -0000 On 11/6/20 11:23 AM, Poul-Henning Kamp wrote: > This seems like a genuinely useful trick to have in the tool-box > and now I'm trying to find out where in our documentation something > like this belongs ? > > It seems too specialized and with too many sharp edges for the handbook ? > > Wiki/UEFI seems rather historical... > > Suggestions ? > If you can think of a manual page that would be appropriate then it might be nice to but it there. Otherwise, perhaphs the FAQ book (https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/book.html)? From owner-freebsd-hackers@freebsd.org Fri Nov 6 15:13:51 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1A82F440F2B for ; Fri, 6 Nov 2020 15:13:51 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4CSP6k3Rg9z3sjd for ; Fri, 6 Nov 2020 15:13:50 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: by mailman.nyi.freebsd.org (Postfix) id 760C0440F29; Fri, 6 Nov 2020 15:13:50 +0000 (UTC) Delivered-To: hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 75D28440B71 for ; Fri, 6 Nov 2020 15:13:50 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by mx1.freebsd.org (Postfix) with ESMTP id 4CSP6k276hz3t5m for ; Fri, 6 Nov 2020 15:13:49 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from [194.32.164.27] ([194.32.164.27]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id 0A6FDhJE046923; Fri, 6 Nov 2020 15:13:43 GMT (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Have trick, not sure where to document... From: Bob Bishop In-Reply-To: <57254.1604658230@critter.freebsd.dk> Date: Fri, 6 Nov 2020 15:13:43 +0000 Cc: hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0BAC0602-9CA6-4C5B-8AA7-46986714C8CF@gid.co.uk> References: <57254.1604658230@critter.freebsd.dk> To: Poul-Henning Kamp X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CSP6k276hz3t5m X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 15:13:51 -0000 Hi, > On 6 Nov 2020, at 10:23, Poul-Henning Kamp wrote: >=20 > I had some trouble with a particular remote site, and wanted to have > the hardware watchdog protect the boot-sequence, but the BIOS had no > support for this. >=20 > The solution I found was to use the UEFI shell to arm the watchdog > before the actuall boot. >=20 > After digging around in the abysmal UEFI ecosystem some time, I ended > up with a bunch of "MM" commands in startup.nsh, before it launches > BOOTx64.efi: >=20 > MM 4e ;IO :87 -n > MM 4e ;IO :87 -n > MM 4e ;IO :20 -n > MM 4f ;IO -n > MM 4e ;IO :21 -n > MM 4f ;IO -n > [...] > MM 4e ;IO :aa -n > BOOTx64.efi >=20 > This seems like a genuinely useful trick to have in the tool-box > and now I'm trying to find out where in our documentation something > like this belongs ? >=20 > It seems too specialized and with too many sharp edges for the = handbook ? >=20 > Wiki/UEFI seems rather historical... >=20 > Suggestions ? boot(8). If that is deemed inappropriate, then boot(8) should definitely = have a reference to the actual place. > --=20 > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by = incompetence. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" >=20 -- Bob Bishop rb@gid.co.uk From owner-freebsd-hackers@freebsd.org Fri Nov 6 19:47:21 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E1E41447DE4 for ; Fri, 6 Nov 2020 19:47:21 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CSWBK655Bz4gHL for ; Fri, 6 Nov 2020 19:47:21 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from [192.168.1.10] (c-98-207-126-143.hsd1.ca.comcast.net [98.207.126.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: rpokala) by smtp.freebsd.org (Postfix) with ESMTPSA id 8198D2EC73 for ; Fri, 6 Nov 2020 19:47:21 +0000 (UTC) (envelope-from rpokala@freebsd.org) User-Agent: Microsoft-MacOutlook/16.42.20101102 Date: Fri, 06 Nov 2020 11:47:16 -0800 Subject: Re: Have trick, not sure where to document... From: Ravi Pokala To: "freebsd-hackers@freebsd.org" Message-ID: <9498D30F-4FDE-40BE-9941-B7C2DBD6E9DA@panasas.com> Thread-Topic: Have trick, not sure where to document... Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 19:47:21 -0000 -----Original Message----- Date: Fri, 6 Nov 2020 12:02:56 +0100 From: Mateusz Piotrowski <0mp@FreeBSD.org> To: Poul-Henning Kamp , hackers@freebsd.org Subject: Re: Have trick, not sure where to document... Message-ID: <6225bae3-0bb4-bd14-51f3-de72eaa4dc98@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed On 11/6/20 11:23 AM, Poul-Henning Kamp wrote: > This seems like a genuinely useful trick to have in the tool-box > and now I'm trying to find out where in our documentation something > like this belongs ? > > It seems too specialized and with too many sharp edges for the handbook ? > > Wiki/UEFI seems rather historical... > > Suggestions ? > If you can think of a manual page that would be appropriate then it might be nice to but it there. Otherwise, perhaphs the FAQ book (https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/book.html)? If the system has IPMI, there are ways to configure it such that it will enable the watchdog during POST, then disable it when it hands off to the OS. At which time, of course, the OS can re-enable it. Interestingly, it looks like `ipmitool' does not offer a way to set the BMC watchdog configuration, just report, reset, and disable it. But ipmi(4) can configure things, in concert with `watchdogd'. More importantly, the IPMI specs describe how to set different configurations for different stages of operation: - POST - POST, after timing out the previous POST - OS loading (meaning after POST, before watchdog driver takes over) - OS normal operation (after the watchdog driver is running) It should be possible to teach ipmi(4) how to configure the timer for POST and OS-load, then have `watchdogd' issue those commands as part of shutdown. https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html ; section 27. -Ravi (rpokala@) From owner-freebsd-hackers@freebsd.org Fri Nov 6 19:49:31 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E839D2D0248 for ; Fri, 6 Nov 2020 19:49:31 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from s2gw.ddhf.dk (s2gw.ddhf.dk [130.226.214.244]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CSWDq5ZCWz4gkn; Fri, 6 Nov 2020 19:49:31 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by s2gw.ddhf.dk (Postfix) with ESMTPS id CC4D6C3F3A; Fri, 6 Nov 2020 19:49:15 +0000 (UTC) Received: (from phk@localhost) by critter.freebsd.dk (8.16.1/8.16.1/Submit) id 0A6JnUxR020509; Fri, 6 Nov 2020 19:49:30 GMT (envelope-from phk) To: Ravi Pokala cc: "freebsd-hackers@freebsd.org" Subject: Re: Have trick, not sure where to document... In-reply-to: <9498D30F-4FDE-40BE-9941-B7C2DBD6E9DA@panasas.com> From: "Poul-Henning Kamp" References: <9498D30F-4FDE-40BE-9941-B7C2DBD6E9DA@panasas.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <20507.1604692170.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Fri, 06 Nov 2020 19:49:30 +0000 Message-ID: <20508.1604692170@critter.freebsd.dk> X-Rspamd-Queue-Id: 4CSWDq5ZCWz4gkn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 19:49:32 -0000 -------- Ravi Pokala writes: > If the system has IPMI, there are ways to configure it such that it will= enable the watchdog during POST, then disable it when it = > hands off to the OS. At which time, of course, the OS can re-enable it. It does not. I specifically do *not* want it to turn the watchdog off when handing to the OS, I want to catch problems that cause hanging during boot, so that I, with suitable patience (or automation), can catch it next time it resets. -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-hackers@freebsd.org Fri Nov 6 23:53:55 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A456D2D7399 for ; Fri, 6 Nov 2020 23:53:55 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CScfp3hJyz3FH1 for ; Fri, 6 Nov 2020 23:53:54 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 0A6NrqIa030546 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 6 Nov 2020 15:53:52 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 0A6Nrqq4030545; Fri, 6 Nov 2020 15:53:52 -0800 (PST) (envelope-from jmg) Date: Fri, 6 Nov 2020 15:53:52 -0800 From: John-Mark Gurney To: Eric McCorkle Cc: FreeBSD Hackers Subject: Re: Mounting encrypted ZFS datasets/GELI for users? Message-ID: <20201106235352.GE31099@funkthat.com> Mail-Followup-To: Eric McCorkle , FreeBSD Hackers References: <8d467e98-237f-c6a2-72de-94c0195ec964@metricspace.net> <20201026221215.GB31099@funkthat.com> <794d789d-4056-4152-e7f6-bf9d10d42518@metricspace.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <794d789d-4056-4152-e7f6-bf9d10d42518@metricspace.net> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Fri, 06 Nov 2020 15:53:52 -0800 (PST) X-Rspamd-Queue-Id: 4CScfp3hJyz3FH1 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [0.20 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 23:53:55 -0000 Eric McCorkle wrote this message on Sat, Oct 31, 2020 at 15:48 -0400: > On 10/26/20 6:12 PM, John-Mark Gurney wrote: > > Eric McCorkle wrote this message on Mon, Oct 05, 2020 at 09:45 -0400: > >> I'm presently looking into options presented by ZFS encryption. One > >> idea I had was something like this (I'm going to go with ZFS for now, > >> but you could presumably do something like this with GELI, with more > >> effort). > > > > I'd still recommend using GELI. Even w/ ZFS's native encryption, the > > metadata for ZFS remains unencrypted, and able to be munged. If you > > geli w/ ZFS and a strong checksum, like sha512/256, I believe that this > > is the equiavlent to authenticated encryption, ala geli's authenticated > > mode, but with significantly less overhead... > > Something to note is that GELI's authenticated mode changes the block > size, because it uses the last bytes in each block to hold the MAC. > This is likely to have consequences for performance. Which is why I don't use it, and use GELI's unauthenticated mode (default) w/ ZFS strong hashes... > However, this also does suggest a ZFS feature that would create a MAC > code for the root block of the filesystem (I am less familiar with the > ZFS on-disk format, but as it's a write-once format with MAC information > stored at each block pointer, this would have the effect of protecting > the entire filesystem from offline tampering. Yes, that is one of the reasons I'm not interested in ZFS native encryption is that it only protects datasets, but the rest of the metadata is not protected... Adding MACs all of the ZFS metadata blocks would be good for native ZFS encryption, but short of encrypting ALL the ZFS metadata, it doesn't satisfy my needs... The reason I'm fine w/ GELI + sha512/256 is the way AES blocks work.. GELI w/ AES-XTS, each 16byte chunk is encrypted w/ a stream cipher, so there is zero overlap between chunks. W/ sha512/256, the hash is 32bytes. So, an attacker would have to corrupt TWO 16 byte blocks of data wich exactly the correct data, to get ZFS to trust the checksum. If an attacker can do that, they very likely have the encryption key, so it's game over anyways. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@freebsd.org Sat Nov 7 22:58:25 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D323E2D440B for ; Sat, 7 Nov 2020 22:58:25 +0000 (UTC) (envelope-from samspeed@mail.ru) Received: from f745.i.mail.ru (f745.i.mail.ru [128.140.172.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CTCNJ3M7bz3tq8 for ; Sat, 7 Nov 2020 22:58:24 +0000 (UTC) (envelope-from samspeed@mail.ru) Received: by f745.i.mail.ru with local (envelope-from ) id 1kbX9z-0002n6-Aq for freebsd-hackers@freebsd.org; Sun, 08 Nov 2020 01:58:15 +0300 Received: by e.mail.ru with HTTP; Sun, 08 Nov 2020 01:58:15 +0300 From: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= To: freebsd-hackers@freebsd.org Subject: =?UTF-8?B?SGFyZCBwaW4gdGhyZWFkIGZvciAxIGNwdSB3aXRoIGRpc2FibGVkIGludGVy?= =?UTF-8?B?cnVwdHMsIHNoZWR1bGVyIGV0Yy4=?= MIME-Version: 1.0 X-Mailer: Mail.Ru Mailer 1.0 Date: Sun, 08 Nov 2020 01:58:15 +0300 Reply-To: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= X-Priority: 3 (Normal) Message-ID: <1604789895.225748578@f745.i.mail.ru> X-7564579A: B8F34718100C35BD X-77F55803: 119C1F4DF6A9251C9F281DEBF96F126DE0A0160C88204197CF1DA7E7CD6312698FD872164937FA4C7E0098069C90F3CADB2AF678930E23283DE59938244A42EE43006C731734714C X-7FA49CB5: 70AAF3C13DB7016878DA827A17800CE73C45176E3332E48FD82A6BABE6F325AC08BE7437D75B48FABCF491FFA38154B613377AFFFEAFD269EE3A9D0FB4FE0F99B10A181A22533F51C2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE7F1A1961AA7DC6F57EA1F7E6F0F101C674E70A05D1297E1BBC6CDE5D1141D2B1C2DC215EEE240C743826A3A09676BA8884A88DCE2E666313F9FA2833FD35BB23D9E625A9149C048EE33AC447995A7AD18C26CFBAC0749D213D2E47CDBA5A96583BD4B6F7A4D31EC0BB23A54CFFDBC96A8389733CBF5DBD5E9D5E8D9A59859A8B6D082881546D93491CC7F00164DA146DA6F5DAA56C3B73B23B0D06ED40D7E105A75ECD9A6C639B01BC09775C1D3CA48CF7C6C241D9975906435872C767BF85DA22EF20D2F80756B5F40A5AABA2AD3711975ECD9A6C639B01B78DA827A17800CE7EE061C4D93700B7A731C566533BA786A40A5AABA2AD371193C9F3DD0FB1AF5EB82E77451A5C57BD33C9F3DD0FB1AF5EB4E70A05D1297E1BBCB5012B2E24CD356 X-C8649E89: 20E39C52C4DE07B7DF40DD2FBC34032840F2282B14A0645F782CF5AECD173CB3FD334A436A1363969F595CEF8576C34B X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5+wYjsrrSY/u8Y3PrTqANeitKFiSd6Yd7yPpbiiZ/d5BsxIjK0jGQgCHUM3Ry2Lt2G3MDkMauH3h0dBdQGj+BB/iPzQYh7XS329fgu+/vnDh1bi9z07B+wokn684aU9QXw== X-Mailru-Internal-Actual: A:0.8980933338445 X-Mailru-MI: 900 X-Mailru-Sender: 015BA6FC0EFA223A2C3604936842D16A1ED19B18173DAB0698798FBD0607C754BEB94853A9F08A38DBB6F488A40E03C2BC76079D514E5C4AC62600DFC3A8D7422F84BEB09BBDFF42A72022385E78452A9FC4A2B1A9835AE10D4ABDE8C577C2ED X-Mras: Ok X-Rspamd-Queue-Id: 4CTCNJ3M7bz3tq8 X-Spamd-Bar: - X-Spamd-Result: default: False [-1.39 / 15.00]; HAS_REPLYTO(0.00)[samspeed@mail.ru]; FREEMAIL_FROM(0.00)[mail.ru]; R_SPF_ALLOW(-0.20)[+ip4:128.140.168.0/21]; TO_DN_NONE(0.00)[]; MIME_BASE64_TEXT_BOGUS(1.00)[]; DKIM_TRACE(0.00)[mail.ru:+]; HAS_X_PRIO_THREE(0.00)[3]; DMARC_POLICY_ALLOW(-0.50)[mail.ru,reject]; MAIL_RU_MAILER_BASE64(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[128.140.172.165:from]; FREEMAIL_ENVFROM(0.00)[mail.ru]; ASN(0.00)[asn:47764, ipnet:128.140.168.0/21, country:RU]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail3]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[mail.ru:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.91)[-0.909]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_REPLYTO(0.00)[mail.ru]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[128.140.172.165:from:127.0.2.255]; NEURAL_SPAM_SHORT(0.52)[0.518]; MAILMAN_DEST(0.00)[freebsd-hackers] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2020 22:58:25 -0000 CkhpLCBJIHdyb3RlIHNvbWUga2VybmVsIG1vZHVsZSB3aXRoIHJlYWx0aW1lIGtlcm5lbCB0aHJl YWQgZm9yIGxhdGVuY3kgdGVzdC4KV2l0aCBlbmFibGVkIGludGVycnVwdHMgbGF0ZW5zeSB2ZXJ5 IGJhZCBmb3IgZ3BpbyByZWFsdGltZSBkcml2aW5nLgpBZGRpbmcgYmVnaW5cZW5kIGNyaXRpY2Fs IHNlY3Rpb24gbWFrZSBqaXR0ZXIgMjAwbnMoaXRzIHBlcmZlY3QpIGJ1dCBoYW5nIGtlcm5lbAph ZnRlciAxNSBzZWNvbmRzIG9mIHdvcmsgb24gbXkgb3JhbmdlIHBpLXplcm8uCkhvdyB0byBkaXNh YmxlIHdhdGNoZG9nIG9yIGFueSBlbHNlID8KwqAKwqAKwqAKI2luY2x1ZGUgPHN5cy90eXBlcy5o PgojaW5jbHVkZSA8c3lzL21vZHVsZS5oPgojaW5jbHVkZSA8c3lzL3N5c3RtLmg+wqAgLyogdXBy aW50ZiAqLwojaW5jbHVkZSA8c3lzL2Vycm5vLmg+CiNpbmNsdWRlIDxzeXMvcGFyYW0uaD7CoCAv KiBkZWZpbmVzIHVzZWQgaW4ga2VybmVsLmggKi8KI2luY2x1ZGUgPHN5cy9rZXJuZWwuaD4gLyog dHlwZXMgdXNlZCBpbiBtb2R1bGUgaW5pdGlhbGl6YXRpb24gKi8KI2luY2x1ZGUgPHN5cy9rdGhy ZWFkLmg+CiNpbmNsdWRlIDxzeXMvcHJvYy5oPgojaW5jbHVkZSA8c3lzL3NjaGVkLmg+CiNpbmNs dWRlIDxzeXMvY3B1c2V0Lmg+CiNpbmNsdWRlIDxzeXMvdGltZXRjLmg+CiNpbmNsdWRlIDxzeXMv dGltZS5oPgojaW5jbHVkZSA8c3lzL211dGV4Lmg+CnN0YXRpYyBpbnQgwqDCoCDCoMKgwqAgwqDC oMKgIMKgwqDCoCDCoMKgwqAgwqBkVGltZU1heCA9IDA7CnN0YXRpYyBpbnQgwqDCoCDCoMKgwqAg wqDCoMKgIMKgwqDCoCDCoMKgwqAgwqBkVGltZU1pbiA9IDB4N0ZGRkZGRkY7CnN0YXRpYyB1bnNp Z25lZCBpbnTCoMKgIMKgwqDCoCDCoMKgwqAgwqBkVGltZUF2ZyA9IDA7CnN0YXRpYyB1bnNpZ25l ZCBpbnTCoMKgIMKgwqDCoCDCoMKgwqAgwqBJbnRDb3VudCA9IDA7CnN0YXRpYyBpbnQgwqDCoCDC oMKgwqAgwqDCoMKgIMKgwqDCoCDCoMKgwqAgwqBUaWNrOwpzdGF0aWMgdm9sYXRpbGUgaW50IMKg wqAgwqDCoMKgIMKgVGhyZWFkRXhpdDsKc3RhdGljIHN0cnVjdCB0aHJlYWTCoMKgIMKgwqDCoCDC oCpUaHJlYWREYXRhOwpzdGF0aWMgY3B1c2V0X3TCoMKgIMKgwqDCoCDCoMKgwqAgwqDCoMKgIMKg Q3B1U2V0OwpzdGF0aWMgdV9pbnQ2NF90wqDCoCDCoMKgwqAgwqDCoMKgIMKgTWF4VGltZXIgPSAw LCBNaW5UaW1lciA9IDAsIFRpbWVyID0gMDsKCnN0YXRpYyB2b2lkIFRocmVhZEZ1bmMoKQp7CsKg wqAgwqBjcHVzZXRfc2V0dGhyZWFkKCBUaHJlYWREYXRhLT50ZF90aWQsICZDcHVTZXQgKTsKwqDC oCDCoHRocmVhZF9sb2NrKCBUaHJlYWREYXRhICk7CsKgwqAgwqBzY2hlZF9jbGFzcyggVGhyZWFk RGF0YSwgUFJJX1JFQUxUSU1FICk7CsKgwqAgwqB0aHJlYWRfdW5sb2NrKCBUaHJlYWREYXRhICk7 CsKgwqAgwqBzY2hlZF9waW4oKTsKwqDCoCDCoHdoaWxlKCBUaHJlYWRFeGl0ID09IDAgKQrCoMKg IMKgewrCoMKgIMKgwqDCoCDCoHVfaW50IFQxLFQyOwrCoMKgIMKgwqDCoCDCoGZvciggOyAhVGhy ZWFkRXhpdCA7IFRpbWVyKyspCsKgwqAgwqDCoMKgIMKgewrCoMKgIMKgwqDCoCDCoMKgwqAgwqBU MT10aW1lY291bnRlci0+dGNfZ2V0X3RpbWVjb3VudCh0aW1lY291bnRlcik7CsKgwqAgwqDCoMKg IMKgwqDCoCDCoFQyPXRpbWVjb3VudGVyLT50Y19nZXRfdGltZWNvdW50KHRpbWVjb3VudGVyKTsK wqDCoCDCoMKgwqAgwqDCoMKgIMKgVGljayA9IFQyIC0gVDE7CsKgwqAgwqDCoMKgIMKgwqDCoCDC oGlmICggVGljayA+IGRUaW1lTWF4ICkgeyBNYXhUaW1lciA9IFRpbWVyOyBkVGltZU1heCA9IFRp Y2s7IH0KwqDCoCDCoMKgwqAgwqDCoMKgIMKgaWYgKCBUaWNrIDwgZFRpbWVNaW4gKSB7IE1pblRp bWVyID0gVGltZXI7IGRUaW1lTWluID0gVGljazsgfQrCoMKgIMKgwqDCoCDCoMKgwqAgwqBpZiAo IFRpY2sgPiAxICkKwqDCoCDCoMKgwqAgwqDCoMKgIMKgewrCoMKgIMKgwqDCoCDCoMKgwqAgwqDC oMKgIMKgSW50Q291bnQrKzsKwqDCoCDCoMKgwqAgwqDCoMKgIMKgwqDCoCDCoGlmICggZFRpbWVB dmcgPiAwICkKwqDCoCDCoMKgwqAgwqDCoMKgIMKgwqDCoCDCoMKgwqAgwqBkVGltZUF2ZyA9ICgg ZFRpbWVBdmcgKiAxNSArIFRpY2sgKiAxNiApIC8gMTY7CsKgwqAgwqDCoMKgIMKgwqDCoCDCoMKg wqAgwqBlbHNlCsKgwqAgwqDCoMKgIMKgwqDCoCDCoMKgwqAgwqDCoMKgIMKgZFRpbWVBdmcgPSBU aWNrKjE2OwrCoMKgIMKgwqDCoCDCoMKgwqAgwqB9CsKgwqAgwqDCoMKgIMKgfQrCoMKgIMKgfQrC oMKgIMKgVGhyZWFkRXhpdCA9IDI7CsKgwqAgwqBzY2hlZF91bnBpbigpOwrCoMKgIMKga3RocmVh ZF9leGl0KCk7Cn0Kc3RhdGljIHN0cnVjdCBrdGhyZWFkX2Rlc2MgVGhyZWFkRGVzYyA9IHsgImdw aW9fcnRfdGhyZWFkIiwgVGhyZWFkRnVuYywgJlRocmVhZERhdGEgfTsKc3RhdGljIGludCBncGlv X3J0X2xvYWRlcihzdHJ1Y3QgbW9kdWxlICptLCBpbnQgd2hhdCwgdm9pZCAqYXJnKQp7CsKgwqAg wqBpbnQgZXJyID0gMDsKwqDCoCDCoHN3aXRjaCAod2hhdCkgewrCoMKgIMKgY2FzZSBNT0RfTE9B RDrCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgLyoga2xkbG9hZCAqLwrCoMKgIMKgwqDC oCDCoFRocmVhZEV4aXQgPSAwOwrCoMKgIMKgwqDCoCDCoG1lbXNldCggJkNwdVNldCwgMCwgc2l6 ZW9mKENwdVNldCkgKTsKwqDCoCDCoMKgwqAgwqBDUFVfU0VUKCAzLCAmQ3B1U2V0ICk7CsKgwqAg wqDCoMKgIMKga3RocmVhZF9zdGFydCggJlRocmVhZERlc2MgKTsKwqDCoCDCoMKgwqAgwqB1cHJp bnRmKCJHUElPIHJlYWx0aW1lIGV4dGVuaW9ucyBLTEQgbG9hZGVkLlxuIik7CsKgwqAgwqDCoMKg IMKgYnJlYWs7CsKgwqAgwqBjYXNlIE1PRF9VTkxPQUQ6CsKgwqAgwqDCoMKgIMKgVGhyZWFkRXhp dCA9IDE7CsKgwqAgwqDCoMKgIMKgd2hpbGUoIFRocmVhZEV4aXQgPT0gMSApOwrCoMKgIMKgwqDC oCDCoHVwcmludGYoIkdQSU8gaml0dGVyIGRUaW1lTWF4ID0gJWQgKCVsbGQpIFx0ZFRpbWVBdmcg PSAlZCBcdGRUaW1lTWluID0gJWQoJWxsZCkgXHQlbGxkXHQlZFx0XG4iLCBkVGltZU1heCwgTWF4 VGltZXIsIGRUaW1lQXZnLzE2LCBkVGltZU1pbiwgTWluVGltZXIsIFRpbWVyLCBJbnRDb3VudCAp OwrCoMKgIMKgwqDCoCDCoHVwcmludGYoIkdQSU8gcmVhbHRpbWUgZXh0ZW5pb25zIEtMRCB1bmxv YWRlZC5cbiIpOwrCoMKgIMKgwqDCoCDCoGJyZWFrOwrCoMKgIMKgZGVmYXVsdDoKwqDCoCDCoMKg wqAgwqBlcnIgPSBFT1BOT1RTVVBQOwrCoMKgIMKgwqDCoCDCoGJyZWFrOwrCoMKgIMKgfQrCoMKg IMKgcmV0dXJuKGVycik7Cn0Kc3RhdGljIG1vZHVsZWRhdGFfdCBncGlvX3J0X21vZCA9IHvCoMKg IMKgImdwaW9fcnQiLMKgwqAgwqBncGlvX3J0X2xvYWRlcizCoMKgIMKgTlVMTCB9OwpERUNMQVJF X01PRFVMRShncGlvX3J0LCBncGlvX3J0X21vZCwgU0lfU1VCX0tMRCwgU0lfT1JERVJfQU5ZKTsK wqAKwqAKwqAKLS0KQW5kcmV5IFNtYWdpbg==