From owner-svn-src-head@freebsd.org Sun Jun 14 22:05:45 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5B0C83468EE for ; Sun, 14 Jun 2020 22:05:45 +0000 (UTC) (envelope-from jrtc27@jrtc27.com) Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49lT6w5khSz3Sxp for ; Sun, 14 Jun 2020 22:05:44 +0000 (UTC) (envelope-from jrtc27@jrtc27.com) Received: by mail-wm1-f68.google.com with SMTP id g10so12740979wmh.4 for ; Sun, 14 Jun 2020 15:05:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=mEHnXo+1wUMFhVpeBENtiOfcOAkEMSvrOFnqttI1WQ8=; b=acsnbM5x+2RGN/xx0X64PWgvkQX1K9nMo2M3wbrHOJiX8kfL450ST9eZA5Dv/OHA4R gHTXPaH3oCbKO5CjvrEr1sqC+h87e6hjJnRKQlc6ENNfkRH8slbL0XcEHNUoRzQvYckw PvcP9BGPBZ+m3472Vw1ByCdeHwrWawD9IRPmrEz/G4MlCa6IfDy+jNiWuQfzjHOOaIsc QJLzTHTlGd2aq3waqOug1mW4yZdqO+LK+QxVWtv6TQ7WSsc49jrMtNS/y0WEfESGYrrd 3JHJZXFw088oJe4/W33QiW8GT0vrS6pzYVOdT1RZwHd9PupvPr9lqWxeJ7j9nXTWFAbD U8cQ== X-Gm-Message-State: AOAM530pdI8cR8iIBnECcYLajU098REFx4brg0lyrrWQla8AWh1OKuYO XtnNGWIszSTT+rZEFp0QNy5mJA== X-Google-Smtp-Source: ABdhPJzIJPosos1yXC/0BMBVs7cLrbE7OJ6Dg0OU3CvtIwQOc89E2fwage8f7vc9Y9xqLtONqNMGSA== X-Received: by 2002:a1c:6a13:: with SMTP id f19mr10257620wmc.142.1592172343350; Sun, 14 Jun 2020 15:05:43 -0700 (PDT) Received: from [192.168.149.251] (trinity-students-nat.trin.cam.ac.uk. [131.111.193.104]) by smtp.gmail.com with ESMTPSA id c6sm20913472wma.15.2020.06.14.15.05.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Jun 2020 15:05:42 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: svn commit: r361944 - in head/sys/dev/virtio: . network From: Jessica Clarke In-Reply-To: <20200614212230.GC68578@tom-desk.erg.abdn.ac.uk> Date: Sun, 14 Jun 2020 23:05:42 +0100 Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0FD9E443-1B40-4495-B2C0-4803121EF911@freebsd.org> References: <202006082151.058LpabU003001@repo.freebsd.org> <20200614195126.GB68578@tom-desk.erg.abdn.ac.uk> <97EEF019-16A4-4626-A484-A00979B52A74@freebsd.org> <20200614212230.GC68578@tom-desk.erg.abdn.ac.uk> To: Tom Jones , Vincenzo Maffione X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49lT6w5khSz3Sxp X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jrtc27@jrtc27.com designates 209.85.128.68 as permitted sender) smtp.mailfrom=jrtc27@jrtc27.com X-Spamd-Result: default: False [-1.56 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-head@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_FIVE(0.00)[5]; NEURAL_HAM_LONG(-1.03)[-1.035]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.04)[-0.038]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.68:from]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; FORGED_SENDER(0.30)[jrtc27@freebsd.org,jrtc27@jrtc27.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.68:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[jrtc27@freebsd.org,jrtc27@jrtc27.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jun 2020 22:05:45 -0000 On 14 Jun 2020, at 22:22, Tom Jones wrote: > On Sun, Jun 14, 2020 at 09:56:03PM +0100, Jessica Clarke wrote: >> On 14 Jun 2020, at 20:51, Tom Jones wrote: >>> On Mon, Jun 08, 2020 at 09:51:36PM +0000, Jessica Clarke wrote: >>>> Author: jrtc27 >>>> Date: Mon Jun 8 21:51:36 2020 >>>> New Revision: 361944 >>>> URL: https://svnweb.freebsd.org/changeset/base/361944 >>>>=20 >>>> Log: >>>> virtio: Support non-legacy network device and queue >>>>=20 >>>> The non-legacy interface always defines num_buffers in the header, >>>> regardless of whether VIRTIO_NET_F_MRG_RXBUF, just leaving it = unused. We >>>> also need to ensure our virtqueue doesn't filter out = VIRTIO_F_VERSION_1 >>>> during negotiation, as it supports non-legacy transports just fine. = This >>>> fixes network packet transmission on TinyEMU. >>>>=20 >>>> Reviewed by: br, brooks (mentor), jhb (mentor) >>>> Approved by: br, brooks (mentor), jhb (mentor) >>>> Differential Revision: https://reviews.freebsd.org/D25132 >>>>=20 >>>> Modified: >>>> head/sys/dev/virtio/network/if_vtnet.c >>>> head/sys/dev/virtio/network/if_vtnetvar.h >>>> head/sys/dev/virtio/virtio.c >>>> head/sys/dev/virtio/virtqueue.c >>>>=20 >>>=20 >>> Hi Jessica, >>>=20 >>> After updating my current bhyve vm today (on a 12.1 host), = networking no longer >>> works. Reverting this commit seems to resolve the issue. I think = vtnet is not >>> passing enough data up to the ip layer. >>>=20 >>> If I capture on the tap interface for the vm I see arp requests and = arp >>> replies, however kern.msgbuf is full of:=20 >>>=20 >>> <5>arp: short packet received on vtnet0 >>>=20 >>> and netstat does not see any replies to arp requests: >>>=20 >>> root@freebsd-current:~ # netstat -s -p arp >>> arp: >>> 11 ARP requests sent >>> 0 ARP requests failed to sent >>> 0 ARP replies sent >>> 0 ARP requests received >>> 0 ARP replies received >>> 0 ARP packets received >>> 24 total packets dropped due to no ARP entry >>> 2 ARP entrys timed out >>> 0 Duplicate IPs seen >>>=20 >>> If I set up an arp entry manually I can see ICMP echo requests and = responses on >>> the tap interface, but the vm does not see the responses.=20 >>>=20 >>> root@freebsd-current:~ # netstat -s -p ip >>> ip: >>> 7 total packets received >>> 0 bad header checksums >>> 0 with size smaller than minimum >>> 7 with data size < data length >>> 0 with ip length > max ip packet size >>> 0 with header length < data size >>> 0 with data length < header length >>>=20 >>> The line >>>=20 >>> 7 with data size < data length >>>=20 >>> makes me think that vtnet is truncating packets.=20 >>>=20 >>> markj pointed me at this bug in irc which might also be related: >>>=20 >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D247242 >>=20 >> Hi Tom, >> Sorry about that; it seems bhyve hits the "legacy and no MrgRxBuf" >> case. Could you please try the patch below? >>=20 >> Jess >>=20 >=20 > This changed fixed the issue for me. Please feel free to add=20 >=20 > Tested By: thj=20 >=20 > when you commit. Great, thanks for the report. > In testing I this lor went by, I wonder if this is something you care = about: >=20 > acquiring duplicate lock of same type: "vtnet0-rx0" > 1st vtnet0-rx0 @ = /usr/home/tj/code/freebsd/projects/review-D25220/sys/dev/virtio/network/if= _vtnet.c:1780 > 2nd vtnet0-rx0 @ = /usr/home/tj/code/freebsd/projects/review-D25220/sys/kern/subr_taskqueue.c= :281 > stack backtrace: > #0 0xffffffff80c32881 at witness_debugger+0x71 > #1 0xffffffff80ba3e54 at __mtx_lock_flags+0x94 > #2 0xffffffff80c24bd2 at taskqueue_enqueue+0x42 > #3 0xffffffff80a1af99 at vtnet_rxq_tq_intr+0xb9 > #4 0xffffffff80c2520a at taskqueue_run_locked+0xaa > #5 0xffffffff80c26284 at taskqueue_thread_loop+0x94 > #6 0xffffffff80b830e0 at fork_exit+0x80 > #7 0xffffffff81040eae at fork_trampoline+0xe Hm, I think that's just a false-positive, because if_vtnet constructs the taskqueue using the same name as its own internal mutexes. Though the locking around vtnet_rx_vq_intr and vtnet_rxq_tq_intr is a bit fishy given they're rather similar yet inconsistent. I would imagine rxq->vtnrx_stats.vrxs_rescheduled is supposed to be protected by that mutex, but wouldn't like to say whether taskqueue_enqueue needs to be. Vincenzo, you recently touched code around there, perhaps you could be persuaded to have a quick look?.. Jess