From owner-freebsd-hackers@freebsd.org Tue Nov 3 04:52:21 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 320B144A593; Tue, 3 Nov 2020 04:52:21 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQHT00j6jz4ZWN; Tue, 3 Nov 2020 04:52:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by mail-pf1-x42b.google.com with SMTP id c20so13127293pfr.8; Mon, 02 Nov 2020 20:52:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=hpsq7Qk3SOFKwSXnJKfMoDvg1nxqDivHRyFif22sQh0=; b=Gp3Jm41gDGv3cp4vVqRJHgvE0STgYUV04FThQuFgLZqex5OhaZJI2c2hn5v8qgrhPo msjCkoGMU37osk08ozyeFUIhbjIHR8fUHukE8kamTeu8iTy3vsKBnR1zm/a+ErkDtaQt BA78BSk9TPFBCGdlY6LRG19yfDyo1bZQcAtfVVdqk05KoncqyvyQpzSM5YKtZ3y7rnxl J8cEcyLkWRZ1CXTlYYUIMSV136OHJxJBgQN8/cpwzMfRR+Wk/bXsGcro+abs88FRNzrY RkRaUP4sIe1fi6//VesLAtcaMrugYqazC+SkSLDiYI9I8xYXRuMDKWb3P013EdydgCsy dA9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=hpsq7Qk3SOFKwSXnJKfMoDvg1nxqDivHRyFif22sQh0=; b=izI4gK5VNb1dsU5PY34xyMuGNsCyB+g7CFxGnmBb+b48/+7RbYrSC5PNevsfV7PWym 7b7MAj3HX1ZXyUJ4+NvjB2FbByFhfyfQMXMNbGxeeHeAuSkDON2IspRMMRgFSBA74+zK 6yn+Dh+iMcJR0jMfwHoIHL5T+M20obaknPQcRYHfro4Ethm4OcAslVS4Zj9iRsSsTi4j vsDYRwtl2td3gPs2lJP8dONmq6IKwvNpIVQPfQ8dwnyhLNtZLSJnDzvf5HKc8QkQGtr2 UyQ/8zObVLRTaJMaFXVbab9wH+e9jvcOp7Vp29a5i9wXS0ScbxvtIMj/8KGVz0rtlzV8 /L7w== X-Gm-Message-State: AOAM531HQrvapXYvMpUIuWUHMyMZX3BFLWszb+LzB9v6HzcrZ4tGeJli zcu0qfvKM0jWU5eM3qAuJnOi0Ocq9Ec= X-Google-Smtp-Source: ABdhPJwbgeVuz7qsym5Q/YazL/gA5mS5dvFeH0QecRxacxwTgxMXPlFtRyRNYC8mehTn4cm4Hq6KEQ== X-Received: by 2002:a65:6649:: with SMTP id z9mr11103680pgv.18.1604379138276; Mon, 02 Nov 2020 20:52:18 -0800 (PST) Received: from localhost ([210.108.244.139]) by smtp.gmail.com with ESMTPSA id g7sm513903pfi.16.2020.11.02.20.52.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Nov 2020 20:52:17 -0800 (PST) Date: Tue, 3 Nov 2020 13:52:15 +0900 From: YongHyeon PYUN To: Kristof Provost Cc: John-Mark Gurney , Carsten =?iso-8859-1?Q?B=E4cker?= , freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Message-ID: <20201103045215.GA2524@michelle> References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> X-Rspamd-Queue-Id: 4CQHT00j6jz4ZWN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Gp3Jm41g; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of pyunyh@gmail.com designates 2607:f8b0:4864:20::42b as permitted sender) smtp.mailfrom=pyunyh@gmail.com X-Spamd-Result: default: False [-3.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.83)[-0.827]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.05)[-1.055]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.019]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::42b:from]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_CC(0.00)[funkthat.com,gmx.de,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers,freebsd-arm] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 04:52:21 -0000 On Thu, Oct 29, 2020 at 10:39:39PM +0100, Kristof Provost wrote: [...] > It does seem to do RX offload, and the comments in the driver suggest some > .. ahem, creative hardware behaviour: > > /* The checksum appears to be simplistically calculated > * over the udp/tcp header and data up to the end of the > * eth frame. Which means if the eth frame is padded > * the csum calculation is incorrectly performed over > * the padding bytes as well. Therefore to be safe we > * ignore the H/W csum on frames less than or equal to > * 64 bytes. > * > * Ignore H/W csum for non-IPv4 packets. > */ > > It’s not impossible that there’s some more issues like that in the hardware, > or even that there are different chip revisions out there. > > That also matches up with `ifconfig ue0 -rxcsum` fixing things. > I'm not sure where the root cause is but it seems smsc(4) needs improvement in RX checksum handling. Quick reading RX handler indicates RX checksum offloading does not work when: o IP datagram has IP options field o TCP/UDP packet was fragmented o UDP datagrams with 0 checksum value See fxp(4), gem(4) and hme(4) for implementation. It looks like smsc(4) uses the following RX format but I don't know actual RX format of H/W(no access to datasheet). <---------------------------- actlen --------------------------------------------------> <------------- pktlen ------------------------> rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial checksum(2 bytes) If the format above is correct smsc(4) needs more strict RX length validation(USB transfer size vs actual packet length). This indicates smsc(4) does not have to copy FCS bytes. Also given that H/W always appends FCS, it's also suspicious H/W does not correctly compute RX checksum for frames less than or equal to 64 bytes. I don't have H/W and some spare time to fix this though. :-(