Date: Sat, 31 Oct 2015 18:01:15 +0300 From: Alexander V. Chernikov <melifaro@freebsd.org> To: Adrian Chadd <adrian@freebsd.org> Cc: freebsd-current <freebsd-current@freebsd.org>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: panic in arptimer in r289937 Message-ID: <1733241446303675@web19h.yandex.ru> In-Reply-To: <CAJ-Vmo=JjHonDqOYK%2BJDaf9581dRU5_KoaSTnY27JnzQm0v56w@mail.gmail.com> References: <CAJ-VmonvVyTNuYv_as41yPCFdfR5T3FE45DP9MKAc-eyzXzPUg@mail.gmail.com> <2739461446298483@web2h.yandex.ru> <CAJ-Vmo=JjHonDqOYK%2BJDaf9581dRU5_KoaSTnY27JnzQm0v56w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
31.10.2015, 16:46, "Adrian Chadd" <adrian@freebsd.org>:
> On 31 October 2015 at 09:34, Alexander V. Chernikov
> <melifaro@freebsd.org> wrote:
>> 31.10.2015, 05:32, "Adrian Chadd" <adrian@freebsd.org>:
>>> Hiya,
>>>
>>> Here's a panic from arptimer:
>> Hi Adrian,
>>
>> As far as I see, line 205 in if_ether.c is IF_AFDATA_LOCK(ifp) which happens after LLE_WUNLOCK().
>> So, it looks like (pre-cached) ifp had been freed before locking ifdata.
>> Do you have any more details on that? (e.g. was some interface detached at that moment, is it reproducible, etc..)
>>
>> From a quick glance, potential use-after-free has been possible for quite a long time, but I wonder why it hasn't been observed before.
>> Probably lltable_free() changes might have triggered that.
>>
>> I'll take a deeper look on that and reply.
>
> Hiya!
>
> Thanks for your quick response.
>
> I mean, I use wifi, and ARPs can get lost / transmit can get delayed /
> etc. I'm also testing through a MIPS CPU based bridge, so I'm also not
I remember that :)
> bridging at line rate. (The above is from one of the x86 laptops doing
> the traffic test.) These are both reasons why I may be poking at a
> path that you don't normally see. :)
Yup. So, once again, could you provide a bit more details? :)
Was it related with any interface being destroyed?
What was the test scenario (just bridging between interfaces?)
Is this reproducible?
>
> I appreciate you taking a very quick look at this!
>
> Thanks,
>
> -adrian
>
>>> (kgdb) bt
>>> #0 doadump (textdump=0) at pcpu.h:221
>>> #1 0xffffffff803666b6 in db_fncall (dummy1=<value optimized out>,
>>> dummy2=<value optimized out>, dummy3=<value optimized out>,
>>> dummy4=<value optimized out>) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/ddb/db_command.c:568
>>> #2 0xffffffff8036614e in db_command (cmd_table=0x0) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/ddb/db_command.c:440
>>> #3 0xffffffff80365ee4 in db_command_loop () at
>>> /usr/home/adrian/work/freebsd/head/src/sys/ddb/db_command.c:493
>>> #4 0xffffffff8036897b in db_trap (type=<value optimized out>, code=0)
>>> at /usr/home/adrian/work/freebsd/head/src/sys/ddb/db_main.c:251
>>> #5 0xffffffff8096c0f3 in kdb_trap (type=9, code=0, tf=<value
>>> optimized out>) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/kern/subr_kdb.c:654
>>> #6 0xffffffff80d34c81 in trap_fatal (frame=0xfffffe022815d7a0,
>>> eva=<value optimized out>) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/amd64/amd64/trap.c:829
>>> #7 0xffffffff80d34951 in trap (frame=<value optimized out>) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/amd64/amd64/trap.c:203
>>> #8 0xffffffff80d149f7 in calltrap () at
>>> /usr/home/adrian/work/freebsd/head/src/sys/amd64/amd64/exception.S:234
>>> #9 0xffffffff8092c3fb in _rw_wlock_cookie (c=0xdeadc0dedeadc2de,
>>> file=0xffffffff81211b1f
>>> "/usr/home/adrian/work/freebsd/head/src/sys/netinet/if_ether.c",
>>> line=205)
>>> at /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_rwlock.c:261
>>> #10 0xffffffff80a2487f in arptimer (arg=0xfffff8005ecc4000) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/netinet/if_ether.c:205
>>> #11 0xffffffff80944c24 in softclock_call_cc (c=0xfffff8005ecc40a8,
>>> cc=0xffffffff81b2d480, direct=0) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_timeout.c:722
>>> #12 0xffffffff80944f87 in softclock (arg=<value optimized out>) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_timeout.c:851
>>> #13 0xffffffff808f7eb6 in intr_event_execute_handlers (p=<value
>>> optimized out>, ie=0xfffff800035a6600) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_intr.c:1262
>>> #14 0xffffffff808f8546 in ithread_loop (arg=0xfffff800032c47c0) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_intr.c:1275
>>> #15 0xffffffff808f57a4 in fork_exit (callout=0xffffffff808f84a0
>>> <ithread_loop>, arg=0xfffff800032c47c0, frame=0xfffffe022815dac0) at
>>> /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_fork.c:1011
>>> #16 0xffffffff80d14f2e in fork_trampoline () at
>>> /usr/home/adrian/work/freebsd/head/src/sys/amd64/amd64/exception.S:609
>>> #17 0x0000000000000000 in ?? ()
>>> Current language: auto; currently minimal
>>>
>>> (kgdb) print *(struct llentry *)c_arg
>>> $2 = {lle_next = {le_next = 0x0, le_prev = 0xfffff8005e867dc8},
>>> r_l3addr = {addr4 = {s_addr = 16782508}, addr6 = {__u6_addr =
>>> {__u6_addr8 = 0xfffff8005ecc4010 "�\024", __u6_addr16 =
>>> 0xfffff8005ecc4010,
>>> __u6_addr32 = 0xfffff8005ecc4010}}}, ll_addr = {mac_aligned =
>>> 110869256150596, mac16 = 0xfffff8005ecc4020, mac8 = 0xfffff8005ecc4020
>>> "D\036���d"}, spare0 = 0, spare1 = 0, lle_tbl = 0xfffff8005e867e00,
>>> lle_head = 0xfffff8005e867dc8, lle_free = 0xffffffff80a2c5d0
>>> <in_lltable_destroy_lle>, la_hold = 0x0, la_numheld = 0, la_expire =
>>> 2110, la_flags = 1, la_asked = 0, la_preempt = 5, ln_state = 0,
>>> ln_router = 0, ln_ntick = 0,
>>> lle_refcnt = 1, lle_chain = {le_next = 0x0, le_prev = 0x0},
>>> lle_timer = {c_links = {le = {le_next = 0x0, le_prev =
>>> 0xffffffff81b2d588}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0,
>>> tqe_prev = 0xffffffff81b2d588}},
>>> c_time = 9066299815445, c_precision = 322122525000, c_arg =
>>> 0xfffff8005ecc4000, c_func = 0xffffffff80a246e0 <arptimer>, c_lock =
>>> 0x0, c_flags = 0, c_iflags = 144, c_cpu = 0}, lle_lock = {lock_object
>>> = {
>>> lo_name = 0xffffffff8120fbce "lle", lo_flags = 90374144, lo_data
>>> = 0, lo_witness = 0xfffffe0000b53c80}, rw_lock = 1}}
>>>
>>> ..
>>>
>>> (kgdb) print *((struct llentry *)c_arg)->lle_tbl
>>> $4 = {llt_link = {sle_next = 0xdeadc0dedeadc0de}, llt_af = -559038242,
>>> llt_hsize = -559038242, lle_head = 0xdeadc0dedeadc0de, llt_ifp =
>>> 0xdeadc0dedeadc0de, llt_lookup = 0xdeadc0dedeadc0de,
>>> llt_alloc_entry = 0xdeadc0dedeadc0de, llt_delete_entry =
>>> 0xdeadc0dedeadc0de, llt_prefix_free = 0xdeadc0dedeadc0de,
>>> llt_dump_entry = 0xdeadc0dedeadc0de, llt_hash = 0xdeadc0dedeadc0de,
>>> llt_match_prefix = 0xdeadc0dedeadc0de,
>>> llt_free_entry = 0xdeadc0dedeadc0de, llt_foreach_entry =
>>> 0xdeadc0dedeadc0de, llt_link_entry = 0xdeadc0dedeadc0de,
>>> llt_unlink_entry = 0xdeadc0dedeadc0de, llt_fill_sa_entry =
>>> 0xdeadc0dedeadc0de,
>>> llt_free_tbl = 0xdeadc0dedeadc0de}
>>>
>>> :(
>>>
>>> Any ideas on where next to look?
>>>
>>> -adrian
>>> _______________________________________________
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1733241446303675>