From owner-freebsd-arm@freebsd.org Wed Nov 4 01:30:24 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05BA044A91D; Wed, 4 Nov 2020 01:30:24 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CQpxV6Tthz3V4f; Wed, 4 Nov 2020 01:30:22 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by mail-pg1-x52b.google.com with SMTP id f38so15170733pgm.2; Tue, 03 Nov 2020 17:30:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=cpQ3S2rbVhd8LdJqsmdfB8QRGRwRUT8yneoStHUogpo=; b=kthhDJEvtdCiDWfG1OOsE/P6jPHxmYpph+skhbomoELHaH2JwMgZ/Bpx2/ynzEsnoT GkvL/8Bn/go7zvQ/7i/TG9StU6TwFc4Rn8vxrxSZANjEhmUhhsjqc737yoaYxLUnN4j+ 5YDJnJJzsY7qaZxxmfBFXHPGIAGFAdJKC1oP1mc6CZ2c02U8mUnZ+U59jYmrPI+wCadh eO8ClK10z6OAcmkYMzCGhpOu7SMIbR6UYIjMHz5mLyDaSbi1TcVRKDA63zUq+PJiVclH B6HWAQGNYdRjTrXlA18FhQPJ6elvpS2Z9N7id/mAb79XbMk77+NiS/ruAXxnc7sleMfT nmcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=cpQ3S2rbVhd8LdJqsmdfB8QRGRwRUT8yneoStHUogpo=; b=SE9a32+FYIqfKpM1r9Y14nQzqJwsePaOtOkY6GWl0Tm5UvrsWyP2lIityX/PdRtHnd I0lEXxSUYsqrPqggXpJQXB9W4kz852Zfa9OxR9d2gpfXKPhUc1euB6gLqCvvfBtHZDYl bm9Y+ETlC9+tFMvJTFRpYYs7sZUYKZt6AUZfETr0BQJR3Z6NLhBRDpVcy0h57vgM7luv S3w/XWgz5akUiu0LGAwXkdT57VyQ8RGHRvMN7lOmE1kfRc7a40vNxtpcrrNdbUP9RySK PPomws92E8tH3hs8Hn7GhkjgZ4i3U8ZWGDkoGNZrIVmKUjYk5HV8JJzzS7Es8emhYeoz uYjQ== X-Gm-Message-State: AOAM533S0Mq0g1aJNjp/z8uoeWvWIFfw2LXZNjRj4wDIaSo52WIJrUcB bYC+AtqzOEENe4GX/eD7/i4= X-Google-Smtp-Source: ABdhPJw7y+SSoS0X5AvSvzzwpaD+Vf/J+QVMB0j/4mdYAzuJ1Q2QtrGEPOhQWpUo4iqdloVoPCYnjQ== X-Received: by 2002:a05:6a00:88d:b029:164:5a00:29b8 with SMTP id q13-20020a056a00088db02901645a0029b8mr29271628pfj.10.1604453420990; Tue, 03 Nov 2020 17:30:20 -0800 (PST) Received: from localhost ([210.108.244.139]) by smtp.gmail.com with ESMTPSA id d22sm191473pgv.87.2020.11.03.17.30.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Nov 2020 17:30:20 -0800 (PST) Date: Wed, 4 Nov 2020 10:30:17 +0900 From: YongHyeon PYUN To: Lowell Gilbert Cc: freebsd-hackers@freebsd.org, freebsd-arm@freebsd.org Subject: Re: Problem with checksum offloading on RPi3 (PF + Jails involved) Message-ID: <20201104013017.GA48398@michelle> References: <748edc3d-4ef7-c4de-291f-7c0b460a6052@gmx.de> <5130ee46-5832-d4df-d774-c6bd32e10b30@gmx.de> <20201029213622.GM31099@funkthat.com> <55713894-A896-4F12-ABB9-93DFEB2F16B9@FreeBSD.org> <20201103045215.GA2524@michelle> <44zh3ybd0s.fsf@Be-Well.Ilk.Org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <44zh3ybd0s.fsf@Be-Well.Ilk.Org> X-Rspamd-Queue-Id: 4CQpxV6Tthz3V4f X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kthhDJEv; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of pyunyh@gmail.com designates 2607:f8b0:4864:20::52b as permitted sender) smtp.mailfrom=pyunyh@gmail.com X-Spamd-Result: default: False [-3.52 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.97)[-0.975]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; NEURAL_HAM_LONG(-1.06)[-1.056]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52b:from]; NEURAL_HAM_SHORT(-0.99)[-0.992]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers,freebsd-arm]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 01:30:24 -0000 On Tue, Nov 03, 2020 at 11:50:27AM -0500, Lowell Gilbert wrote: > YongHyeon PYUN writes: > > > On Thu, Oct 29, 2020 at 10:39:39PM +0100, Kristof Provost wrote: > > > > [...] > > > >> It does seem to do RX offload, and the comments in the driver suggest some > >> .. ahem, creative hardware behaviour: > >> > >> /* The checksum appears to be simplistically calculated > >> * over the udp/tcp header and data up to the end of the > >> * eth frame. Which means if the eth frame is padded > >> * the csum calculation is incorrectly performed over > >> * the padding bytes as well. Therefore to be safe we > >> * ignore the H/W csum on frames less than or equal to > >> * 64 bytes. > >> * > >> * Ignore H/W csum for non-IPv4 packets. > >> */ > >> > >> It’s not impossible that there’s some more issues like that in the hardware, > >> or even that there are different chip revisions out there. > >> > >> That also matches up with `ifconfig ue0 -rxcsum` fixing things. > >> > > > > I'm not sure where the root cause is but it seems smsc(4) needs > > improvement in RX checksum handling. Quick reading RX handler > > indicates RX checksum offloading does not work when: > > o IP datagram has IP options field > > o TCP/UDP packet was fragmented > > o UDP datagrams with 0 checksum value > > See fxp(4), gem(4) and hme(4) for implementation. > > > > It looks like smsc(4) uses the following RX format but I don't > > know actual RX format of H/W(no access to datasheet). > > > > <---------------------------- actlen --------------------------------------------------> > > <------------- pktlen ------------------------> > > rxhdr(4 bytes) | padding (2 bytes) | RX frame | FCS(4 bytes) | partial checksum(2 bytes) > > > > If the format above is correct smsc(4) needs more strict RX length > > validation(USB transfer size vs actual packet length). This > > indicates smsc(4) does not have to copy FCS bytes. > > Also given that H/W always appends FCS, it's also suspicious H/W > > does not correctly compute RX checksum for frames less than or > > equal to 64 bytes. > > For the 89530 at least, the data sheet states that checksum offload > and padding stripping are mutually exclusive. I would assume that > enabling both of them will produce incorrect results. I would *not* > assume that all chips covered by this driver behave the same way, > although it's likely. I see. Publicly available data sheet was too much sanitized such that it's useless to implement checksum offloading. Given that you have access to data sheet, could you describe exact RX format of H/W? FreeBSD's USB stack requires copy operation for received frames so driver does not need any padding in RX path and ethernet frame alignment in mbuf can be adjusted in the copy operation. Since FCS bytes have no use in upper stack at this moment, it would be even better to strip FCS if there is way to do that in H/W. For TX checksum offloading support, we need more information on H/W's TX handling(TX_CTRL_0 and TX_CTRL_1) too. I guess it needs special magic sequence of programming(i.e. existence of SMSC_TX_CTRL_0_FIRST_SEG and SMSC_TX_CTRL_0_LAST_SEG) but it could be greatly simplified to program the H/W for TX checksum offloading since driver also have to copy entire mbuf chain to USB stack(i.e single big TX segment for the H/W).