Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 6 Aug 2018 09:21:26 +0200
From:      Hans Petter Selasky <hps@selasky.org>
To:        Matthew Macy <mmacy@freebsd.org>, Roman Bogorodskiy <novel@freebsd.org>
Cc:        freebsd-current@freebsd.org
Subject:   Re: panic after ifioctl/if_clone_destroy
Message-ID:  <a03803e6-5f1e-1960-c6a1-c7477f0ac9d4@selasky.org>
In-Reply-To: <CAPrugNqVUoP0V8%2ByKTbCZgMoDu22xvCfUuga2LbKabjyi_=__A@mail.gmail.com>
References:  <20180805153556.GA1957@kloomba> <CAPrugNqVUoP0V8%2ByKTbCZgMoDu22xvCfUuga2LbKabjyi_=__A@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi,

I think the problem is the thread pointed to by tdwait exited. I would 
say it is not allowed to peek into the other records threads, because 
they may change under the hood and are not protected by the current context.


>         if (record->er_cpuid != curcpu) {

This optimisation is invalid or needs to be revisited:
>                 /*
>                  * If the head of the list is running, we can wait for it
>                  * to remove itself from the list and thus save us the
>                  * overhead of a migration
>                  */
>                 if ((tdwait = TAILQ_FIRST(&record->er_tdlist)) != NULL &&
>                     TD_IS_RUNNING(tdwait->et_td)) {
>                         gen = record->er_gen;
>                         thread_unlock(td);
>                         do {
>                                 cpu_spinwait();
>                         } while (tdwait == TAILQ_FIRST(&record->er_tdlist) &&
>                             gen == record->er_gen && TD_IS_RUNNING(tdwait->et_td) &&
>                             spincount++ < MAX_ADAPTIVE_SPIN);
>                         thread_lock(td);
>                         return;
>                 }


--HPS

On 08/05/18 22:01, Matthew Macy wrote:
> If you could give me a self-contained reproducer that would expedite a fix.
> 
> Thanks.
> -M
> 
> On Sun, Aug 5, 2018 at 08:36 Roman Bogorodskiy <novel@freebsd.org> wrote:
> 
>> Running -CURRENT r336863 on amd64. Get the following panic right after
>> (or during) boot:
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 2; apic id = 04
>> fault virtual address   = 0xdeadc2ff
>> fault code              = supervisor read data, page not present
>> instruction pointer     = 0x20:0xffffffff80bd7858
>> stack pointer           = 0x28:0xfffffe008b445580
>> frame pointer           = 0x28:0xfffffe008b4455c0
>> code segment            = base 0x0, limit 0xfffff, type 0x1b
>>                          = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags        = interrupt enabled, resume, IOPL = 0
>> current process         = 903 (libvirtd)
>>
>> Traceback is:
>>
>> (kgdb) #0  doadump (textdump=0) at pcpu.h:230
>> #1  0xffffffff8043dc7b in db_dump (dummy=<value optimized out>,
>>      dummy2=<value optimized out>, dummy3=<value optimized out>,
>>      dummy4=<value optimized out>) at /usr/src/sys/ddb/db_command.c:574
>> #2  0xffffffff8043da49 in db_command (cmd_table=<value optimized out>)
>>      at /usr/src/sys/ddb/db_command.c:481
>> #3  0xffffffff8043d7c4 in db_command_loop ()
>>      at /usr/src/sys/ddb/db_command.c:534
>> #4  0xffffffff804409ef in db_trap (type=<value optimized out>,
>>      code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:252
>> #5  0xffffffff80bdd513 in kdb_trap (type=12, code=0, tf=<value optimized
>> out>)
>>      at /usr/src/sys/kern/subr_kdb.c:693
>> #6  0xffffffff810769f1 in trap_fatal (frame=0xfffffe008b4454c0,
>> eva=3735929599)
>>      at /usr/src/sys/amd64/amd64/trap.c:884
>> #7  0xffffffff81076b12 in trap_pfault (frame=0xfffffe008b4454c0,
>>      usermode=<value optimized out>) at pcpu.h:230
>> #8  0xffffffff8107611a in trap (frame=0xfffffe008b4454c0)
>>      at /usr/src/sys/amd64/amd64/trap.c:427
>> #9  0xffffffff810518ac in calltrap ()
>>      at /usr/src/sys/amd64/amd64/exception.S:230
>> #10 0xffffffff80bd7858 in epoch_block_handler_preempt (
>>      global=<value optimized out>, cr=0xfffffe00760c3a00,
>>      arg=<value optimized out>) at /usr/src/sys/kern/subr_epoch.c:256
>> #11 0xffffffff803994fd in ck_epoch_synchronize_wait (
>>      global=0xfffff800030c5680,
>>      cb=0xffffffff80bd77a0 <epoch_block_handler_preempt>, ct=0x0)
>>      at /usr/src/sys/contrib/ck/src/ck_epoch.c:407
>> #12 0xffffffff80bd7630 in epoch_wait_preempt (epoch=0xfffff800030c5680)
>>      at /usr/src/sys/kern/subr_epoch.c:389
>> #13 0xffffffff80c983bf in if_delgroup (ifp=0xfffff80003aab800,
>>      groupname=0xfffff80005ff5e00 "bridge") at /usr/src/sys/net/if.c:1514
>> #14 0xffffffff80c9f2b2 in if_clone_destroyif (ifc=0xfffff80005ff5e00,
>>      ifp=0xfffff80003aab800) at /usr/src/sys/net/if_clone.c:325
>> #15 0xffffffff80c9f0d5 in if_clone_destroy (name=0xfffffe008b4458d0
>> "virbr0")
>>      at /usr/src/sys/net/if_clone.c:288
>> #16 0xffffffff80c9a2c3 in ifioctl (so=0xfffff80007edca38, cmd=2149607801,
>>      data=<value optimized out>, td=<value optimized out>)
>>      at /usr/src/sys/net/if.c:3053
>> #17 0xffffffff80c04259 in kern_ioctl (td=0xfffff80007c1a580,
>>      fd=<value optimized out>, com=<value optimized out>,
>>      data=<value optimized out>) at file.h:330
>> #18 0xffffffff80c03f2e in sys_ioctl (td=0xfffff80007c1a580,
>>      uap=0xfffff80007c1a940) at /usr/src/sys/kern/sys_generic.c:712
>> #19 0xffffffff81077401 in amd64_syscall (td=0xfffff80007c1a580, traced=0)
>>      at subr_syscall.c:135
>> #20 0xffffffff8105218d in fast_syscall_common ()
>>      at /usr/src/sys/amd64/amd64/exception.S:500
>> #21 0x00000008028f4c0a in ?? ()
>>
>>
>> Previous frame inner to this frame (corrupt stack?)
>>
>>
>> Current language:  auto; currently minimal
>>
>>
>> (kgdb)
>>
>> It looks like panic happens during network interfaces related
>> operations. Couple of dmesg lines before panic:
>>
>> Aug  5 19:02:42 romashka rtsold[585]: <rtsock_input_ifannounce> interface
>> bridge0 removed
>> Aug  5 19:02:42 romashka kernel: bridge0: Ethernet address:
>> 02:af:41:48:c7:00
>> Aug  5 19:02:42 romashka kernel: bridge0: changing name to 'virbr-ab'
>> Aug  5 19:02:42 romashka kernel: tap0: Ethernet address: 00:bd:8d:11:f7:00
>> Aug  5 19:02:42 romashka kernel: tap0: link state changed to UP
>> Aug  5 19:02:42 romashka kernel: tap0: changing name to 'virbr-ab-nic'
>> Aug  5 19:02:42 romashka kernel: virbr-ab-nic: promiscuous mode enabled
>> Aug  5 19:02:42 romashka kernel: virbr-ab: link state changed to UP
>> Aug  5 19:02:42 romashka rtsold[585]: <rtsock_input_ifannounce> interface
>> tap0 removed
>> Aug  5 19:02:43 romashka dnsmasq[1047]: setting --bind-interfaces option
>> because of OS limitations
>> Aug  5 19:02:43 romashka dnsmasq[1047]: warning: no upstream servers
>> configured
>> Aug  5 19:02:43 romashka kernel: virbr-ab-nic: link state changed to DOWN
>> Aug  5 19:02:43 romashka kernel: virbr-ab: link state changed to DOWN
>> Aug  5 19:02:43 romashka kernel: bridge1: Ethernet address:
>> 02:af:41:48:c7:01
>> Aug  5 19:02:43 romashka kernel: bridge1: changing name to 'virbr0'
>> Aug  5 19:02:43 romashka rtsold[585]: <rtsock_input_ifannounce> interface
>> bridge1 removed
>> Aug  5 19:02:43 romashka kernel: tap1: Ethernet address: 00:bd:53:14:f7:01
>> Aug  5 19:02:43 romashka kernel: tap1: link state changed to UP
>> Aug  5 19:02:43 romashka kernel: tap1: changing name to 'virbr0-nic'
>> Aug  5 19:02:43 romashka kernel: virbr0: link state changed to UP
>> Aug  5 19:02:43 romashka kernel: virbr0-nic: promiscuous mode enabled
>> Aug  5 19:02:43 romashka rtsold[585]: <rtsock_input_ifannounce> interface
>> tap1 removed
>> Aug  5 19:05:03 romashka syslogd: kernel boot file is /boot/kernel/kernel
>> Aug  5 19:05:03 romashka kernel:
>> Aug  5 19:05:03 romashka syslogd: last message repeated 1 times
>> Aug  5 19:05:03 romashka kernel: Fatal trap 12: page fault while in kernel
>> mode
>>
>> If I disable libvirt service, system completes booting fine. What it
>> tries to do on start, it creates a couple of bridge(4) and tap(4)
>> devices, adds tap devices to bridges it created, and possibly destroy
>> these interfaces in case of errors. It also starts dnsmasq on some of
>> these interfaces.
>>
>> This problem started to appear about 2-4 weeks ago.
>>
>> Roman Bogorodskiy
>>
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
> 




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a03803e6-5f1e-1960-c6a1-c7477f0ac9d4>