Date: Wed, 19 Feb 2020 14:48:45 -0800 From: Gleb Smirnoff <glebius@freebsd.org> To: Kristof Provost <kp@freebsd.org> Cc: freebsd-net <freebsd-net@freebsd.org> Subject: Re: vtnet IFF_NEEDSEPOCH? Message-ID: <20200219224845.GI5741@FreeBSD.org> In-Reply-To: <5520BD42-7D17-4561-A2CD-C690B159D15E@FreeBSD.org> References: <B45B8E77-CDB2-450E-951C-1E6E6BCC2527@FreeBSD.org> <20200218193708.GH5741@FreeBSD.org> <5520BD42-7D17-4561-A2CD-C690B159D15E@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Feb 18, 2020 at 10:08:50PM +0100, Kristof Provost wrote:
K> > K> with vtnet:
K> > K>
K> > K> DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 5
K> > K> panic: Assertion in_epoch(net_epoch_preempt) failed at
K> > K> /usr/src/sys/net/netisr.c:1093
K> > K> cpuid = 0
K> > K> time = 1581981733
K> > K> KDB: stack backtrace:
K> > K> db_trace_self() at db_trace_self
K> > K> db_fetch_ksymtab() at db_fetch_ksymtab+0x12a
K> > K> kdb_backtrace() at kdb_backtrace+0x2c
K> > K> vpanic() at vpanic+0x144
K> > K> panic() at panic+0x26
K> > K> netisr_dispatch_src() at netisr_dispatch_src+0x3c0
K> > K> netisr_dispatch() at netisr_dispatch+0x10
K> > K> ether_ifattach() at ether_ifattach+0x2fa
K> > K> vtmmio_attach() at vtmmio_attach+0x490c
K> > K> vtmmio_attach() at vtmmio_attach+0x4624
K> > K> vtmmio_attach() at vtmmio_attach+0x544a
K> > K> virtqueue_intr() at virtqueue_intr+0xc
K> > K> vtmmio_attach() at vtmmio_attach+0x2008
K> > K> db_dump_intr_event() at db_dump_intr_event+0x730
K> > K> fork_exit() at fork_exit+0x68
K> > K> fork_trampoline() at fork_trampoline+0xa
K> > K> KDB: enter: panic
K> > K> [ thread pid 12 tid 100023 ]
K> > K> Stopped at kdb_enter+0x44: sd zero,0(a0)
K> > K> db>
K> > K>
K> > K> It seems pretty clear that the vtmmio path doesn’t enter epoch
K> > before it
K> > K> runs the vtnet_attach() code.
K> > K> On the other hand, I run vtnet CURRENT guests in bhyve, and don’t
K> > see this
K> > K> panic. In that case it lives on top of PCI rather than mmio, but I
K> > don’t
K> > K> see why/where that’d enter epoch.
K> >
K> > The transition from ether_ifattach to netisr_dispatch looks strange.
K> > Is that something run trough EVENTHANDLER_INVOKE?
K> >
K> > Can you please print in kgdb?
K> >
K> > > list *ether_ifattach+0x2fa
K> >
K> Not immediately, no. This is risc-v, and there’s no kgdb for it right now.
K>
K> I can give you objdump output however:
K>
K> ffffffc00036fd6c <ether_ifattach>:
K> /*
K> * Perform common duties while attaching to interface list
K> */
K> void
K> ether_ifattach(struct ifnet *ifp, const u_int8_t *lla)
K> {
K> ffffffc00036fd6c: 7139 addi sp,sp,-64
K> ffffffc00036fd6e: fc06 sd ra,56(sp)
K> ffffffc00036fd70: f822 sd s0,48(sp)
K> ffffffc00036fd72: f426 sd s1,40(sp)
K> ffffffc00036fd74: f04a sd s2,32(sp)
K> ffffffc00036fd76: ec4e sd s3,24(sp)
K> ffffffc00036fd78: e852 sd s4,16(sp)
K> ffffffc00036fd7a: e456 sd s5,8(sp)
K> ffffffc00036fd7c: e05a sd s6,0(sp)
K> ffffffc00036fd7e: 0080 addi s0,sp,64
K> ffffffc00036fd80: 892e mv s2,a1
K> ffffffc00036fd82: 89aa mv s3,a0
K> ffffffc00036fd84: 4539 li a0,14
K>
K> …
K> if (__predict_false(ifp->if_flags & IFF_NEEDSEPOCH))
K> ffffffc000370036: 0709c503 lbu a0,112(s3)
K> ffffffc00037003a: 02057513 andi a0,a0,32
K> ffffffc00037003e: e92d bnez
K> a0,ffffffc0003700b0 <ether_input+0xba>
K> while (m) {
K> ffffffc000370040: 020a8763 beqz
K> s5,ffffffc00037006e <ether_input+0x78>
K> ffffffc000370044: 5a7d li s4,-1
K> mn = m->m_nextpkt;
K> ffffffc000370046: 008ab483 ld s1,8(s5)
K> m->m_nextpkt = NULL;
K> ffffffc00037004a: 000ab423 sd zero,8(s5)
K> MPASS((m->m_pkthdr.csum_flags & CSUM_SND_TAG) == 0);
K> ffffffc00037004e: 038aa503 lw a0,56(s5)
K> ffffffc000370052: 08aa5963 bge
K> s4,a0,ffffffc0003700e4 <ether_input+0xee>
K> KASSERT(m->m_pkthdr.rcvif == ifp, ("%s: ifnet mismatch m
K> %p "
K> ffffffc000370056: 020ab683 ld a3,32(s5)
K> ffffffc00037005a: 0b369763 bne
K> a3,s3,ffffffc000370108 <ether_input+0x112>
K> netisr_dispatch(NETISR_ETHER, m);
K> ffffffc00037005e: 4515 li a0,5
K> ffffffc000370060: 85d6 mv a1,s5
K> ffffffc000370062: 0000c097 auipc ra,0xc
K> ffffffc000370066: 12c080e7 jalr 300(ra) #
K> ffffffc00037c18e <netisr_dispatch>
K> ffffffc00037006a: 8aa6 mv s5,s1
K> while (m) {
K> ffffffc00037006c: fce9 bnez
K> s1,ffffffc000370046 <ether_input+0x50>
K> if (__predict_false(ifp->if_flags & IFF_NEEDSEPOCH))
K> ffffffc00037006e: 0709c503 lbu a0,112(s3)
K> ffffffc000370072: 02057513 andi a0,a0,32
K> ffffffc000370076: e939 bnez
K> a0,ffffffc0003700cc <ether_input+0xd6>
K> ffffffc000370078: 00023503 ld a0,0(tp) # 0
K> <kernel_lma-0x80200000>
K> CURVNET_RESTORE();
K> ffffffc00037007c: 4e053503 ld a0,1248(a0)
K> ffffffc000370080: c971 beqz
K> a0,ffffffc000370154 <ether_input+0x15e>
K> ffffffc000370082: 00090a63 beqz
K> s2,ffffffc000370096 <ether_input+0xa0>
K> ffffffc000370086: 5e4a7537 lui a0,0x5e4a7
K> ffffffc00037008a: f285051b addiw a0,a0,-216
K> ffffffc00037008e: 01092583 lw a1,16(s2)
K> ffffffc000370092: 0ca59163 bne
K> a1,a0,ffffffc000370154 <ether_input+0x15e>
K> ffffffc000370096: 00023503 ld a0,0(tp) # 0
K> <kernel_lma-0x80200000>
K> ffffffc00037009a: 4f253023 sd s2,1248(a0) #
K> 5e4a74e0 <kernel_lma-0x21d58b20>
K> }
K>
K>
K> In fact, that looks like it’s inside ether_input(), not ether_ifattach().
K>
K> ffffffc00036fff6 <ether_input>:
K> {
K> ffffffc00036fff6: 711d addi sp,sp,-96
Right, it looks much more like the trace should go via ether_input(),
not via ether_ifattach().
K> So I suspect our call stack is something like virtqueue_intr() ->
K> vtnet_rx_vq_intr() -> vtnet_rxq_eof() -> vtnet_rxq_input() ->
K> ether_input().
K> I don’t see anything entering epoch in that path, which presumably explains
K> the panic, but I still don’t understand why my bhyve current vm doesn’t
K> panic in the same way.
On bhyve we enter it through interrupt handler, and this is where we
enter the epoch. Does RISC-V has interrupt handling by the MI code
in sys/kern/kern_intr.c as other platforms?
--
Gleb Smirnoff
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200219224845.GI5741>