From owner-freebsd-net@freebsd.org Wed Feb 19 22:48:56 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E71202478A9 for ; Wed, 19 Feb 2020 22:48:55 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from cell.glebi.us (glebi.us [162.251.186.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "cell.glebi.us", Issuer "cell.glebi.us" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48NCZG4v9jz4M2H; Wed, 19 Feb 2020 22:48:54 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from cell.glebi.us (localhost [127.0.0.1]) by cell.glebi.us (8.15.2/8.15.2) with ESMTPS id 01JMmjBR046828 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 19 Feb 2020 14:48:46 -0800 (PST) (envelope-from glebius@freebsd.org) Received: (from glebius@localhost) by cell.glebi.us (8.15.2/8.15.2/Submit) id 01JMmjop046827; Wed, 19 Feb 2020 14:48:45 -0800 (PST) (envelope-from glebius@freebsd.org) X-Authentication-Warning: cell.glebi.us: glebius set sender to glebius@freebsd.org using -f Date: Wed, 19 Feb 2020 14:48:45 -0800 From: Gleb Smirnoff To: Kristof Provost Cc: freebsd-net Subject: Re: vtnet IFF_NEEDSEPOCH? Message-ID: <20200219224845.GI5741@FreeBSD.org> References: <20200218193708.GH5741@FreeBSD.org> <5520BD42-7D17-4561-A2CD-C690B159D15E@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5520BD42-7D17-4561-A2CD-C690B159D15E@FreeBSD.org> X-Rspamd-Queue-Id: 48NCZG4v9jz4M2H X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.70 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.70)[-0.699,0]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Feb 2020 22:48:56 -0000 On Tue, Feb 18, 2020 at 10:08:50PM +0100, Kristof Provost wrote: K> > K> with vtnet: K> > K> K> > K> DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 5 K> > K> panic: Assertion in_epoch(net_epoch_preempt) failed at K> > K> /usr/src/sys/net/netisr.c:1093 K> > K> cpuid = 0 K> > K> time = 1581981733 K> > K> KDB: stack backtrace: K> > K> db_trace_self() at db_trace_self K> > K> db_fetch_ksymtab() at db_fetch_ksymtab+0x12a K> > K> kdb_backtrace() at kdb_backtrace+0x2c K> > K> vpanic() at vpanic+0x144 K> > K> panic() at panic+0x26 K> > K> netisr_dispatch_src() at netisr_dispatch_src+0x3c0 K> > K> netisr_dispatch() at netisr_dispatch+0x10 K> > K> ether_ifattach() at ether_ifattach+0x2fa K> > K> vtmmio_attach() at vtmmio_attach+0x490c K> > K> vtmmio_attach() at vtmmio_attach+0x4624 K> > K> vtmmio_attach() at vtmmio_attach+0x544a K> > K> virtqueue_intr() at virtqueue_intr+0xc K> > K> vtmmio_attach() at vtmmio_attach+0x2008 K> > K> db_dump_intr_event() at db_dump_intr_event+0x730 K> > K> fork_exit() at fork_exit+0x68 K> > K> fork_trampoline() at fork_trampoline+0xa K> > K> KDB: enter: panic K> > K> [ thread pid 12 tid 100023 ] K> > K> Stopped at kdb_enter+0x44: sd zero,0(a0) K> > K> db> K> > K> K> > K> It seems pretty clear that the vtmmio path doesn’t enter epoch K> > before it K> > K> runs the vtnet_attach() code. K> > K> On the other hand, I run vtnet CURRENT guests in bhyve, and don’t K> > see this K> > K> panic. In that case it lives on top of PCI rather than mmio, but I K> > don’t K> > K> see why/where that’d enter epoch. K> > K> > The transition from ether_ifattach to netisr_dispatch looks strange. K> > Is that something run trough EVENTHANDLER_INVOKE? K> > K> > Can you please print in kgdb? K> > K> > > list *ether_ifattach+0x2fa K> > K> Not immediately, no. This is risc-v, and there’s no kgdb for it right now. K> K> I can give you objdump output however: K> K> ffffffc00036fd6c : K> /* K> * Perform common duties while attaching to interface list K> */ K> void K> ether_ifattach(struct ifnet *ifp, const u_int8_t *lla) K> { K> ffffffc00036fd6c: 7139 addi sp,sp,-64 K> ffffffc00036fd6e: fc06 sd ra,56(sp) K> ffffffc00036fd70: f822 sd s0,48(sp) K> ffffffc00036fd72: f426 sd s1,40(sp) K> ffffffc00036fd74: f04a sd s2,32(sp) K> ffffffc00036fd76: ec4e sd s3,24(sp) K> ffffffc00036fd78: e852 sd s4,16(sp) K> ffffffc00036fd7a: e456 sd s5,8(sp) K> ffffffc00036fd7c: e05a sd s6,0(sp) K> ffffffc00036fd7e: 0080 addi s0,sp,64 K> ffffffc00036fd80: 892e mv s2,a1 K> ffffffc00036fd82: 89aa mv s3,a0 K> ffffffc00036fd84: 4539 li a0,14 K> K> … K> if (__predict_false(ifp->if_flags & IFF_NEEDSEPOCH)) K> ffffffc000370036: 0709c503 lbu a0,112(s3) K> ffffffc00037003a: 02057513 andi a0,a0,32 K> ffffffc00037003e: e92d bnez K> a0,ffffffc0003700b0 K> while (m) { K> ffffffc000370040: 020a8763 beqz K> s5,ffffffc00037006e K> ffffffc000370044: 5a7d li s4,-1 K> mn = m->m_nextpkt; K> ffffffc000370046: 008ab483 ld s1,8(s5) K> m->m_nextpkt = NULL; K> ffffffc00037004a: 000ab423 sd zero,8(s5) K> MPASS((m->m_pkthdr.csum_flags & CSUM_SND_TAG) == 0); K> ffffffc00037004e: 038aa503 lw a0,56(s5) K> ffffffc000370052: 08aa5963 bge K> s4,a0,ffffffc0003700e4 K> KASSERT(m->m_pkthdr.rcvif == ifp, ("%s: ifnet mismatch m K> %p " K> ffffffc000370056: 020ab683 ld a3,32(s5) K> ffffffc00037005a: 0b369763 bne K> a3,s3,ffffffc000370108 K> netisr_dispatch(NETISR_ETHER, m); K> ffffffc00037005e: 4515 li a0,5 K> ffffffc000370060: 85d6 mv a1,s5 K> ffffffc000370062: 0000c097 auipc ra,0xc K> ffffffc000370066: 12c080e7 jalr 300(ra) # K> ffffffc00037c18e K> ffffffc00037006a: 8aa6 mv s5,s1 K> while (m) { K> ffffffc00037006c: fce9 bnez K> s1,ffffffc000370046 K> if (__predict_false(ifp->if_flags & IFF_NEEDSEPOCH)) K> ffffffc00037006e: 0709c503 lbu a0,112(s3) K> ffffffc000370072: 02057513 andi a0,a0,32 K> ffffffc000370076: e939 bnez K> a0,ffffffc0003700cc K> ffffffc000370078: 00023503 ld a0,0(tp) # 0 K> K> CURVNET_RESTORE(); K> ffffffc00037007c: 4e053503 ld a0,1248(a0) K> ffffffc000370080: c971 beqz K> a0,ffffffc000370154 K> ffffffc000370082: 00090a63 beqz K> s2,ffffffc000370096 K> ffffffc000370086: 5e4a7537 lui a0,0x5e4a7 K> ffffffc00037008a: f285051b addiw a0,a0,-216 K> ffffffc00037008e: 01092583 lw a1,16(s2) K> ffffffc000370092: 0ca59163 bne K> a1,a0,ffffffc000370154 K> ffffffc000370096: 00023503 ld a0,0(tp) # 0 K> K> ffffffc00037009a: 4f253023 sd s2,1248(a0) # K> 5e4a74e0 K> } K> K> K> In fact, that looks like it’s inside ether_input(), not ether_ifattach(). K> K> ffffffc00036fff6 : K> { K> ffffffc00036fff6: 711d addi sp,sp,-96 Right, it looks much more like the trace should go via ether_input(), not via ether_ifattach(). K> So I suspect our call stack is something like virtqueue_intr() -> K> vtnet_rx_vq_intr() -> vtnet_rxq_eof() -> vtnet_rxq_input() -> K> ether_input(). K> I don’t see anything entering epoch in that path, which presumably explains K> the panic, but I still don’t understand why my bhyve current vm doesn’t K> panic in the same way. On bhyve we enter it through interrupt handler, and this is where we enter the epoch. Does RISC-V has interrupt handling by the MI code in sys/kern/kern_intr.c as other platforms? -- Gleb Smirnoff