Date: Mon, 30 Oct 2017 19:57:47 -0400 From: Viktor Dukhovni <freebsd@dukhovni.org> To: freebsd-net@freebsd.org Subject: FreeBSD 11.1-RELEASE: Kernel panic in ipv6_output() via tcp6_usr_connect() Message-ID: <FCC0833F-AA88-4F27-9DA3-4FA1218C49DB@dukhovni.org>
next in thread | raw e-mail | index | archive | help
I am using FreeBSD 11.1 as the O/S for my DANE/SMTP adoption scanner.
The system has an IPv4 static IPv4 and also a corresponding 6to4
address on stf0.
The system is stable when I run IPv4-only scans, but crashes quickly
as soon as I start a bulk scan that also connects to the IPv6 addresses
of remote SMTP servers. Indeed after getting the destination address
of the connection that caused the panic (see below) I can now reproduce
the problem at will with just:
$ nc 2a01:5b40:0:2201::1 25
The system has ZFS and two igb network interfaces. The inside network
is my RFC1918 home network, so I use ipfw NAT rules for that, and set:
ifconfig_igb0=3D"inet ... -tso -txcsum"
since hardware checksums don't seem to play along with ipfw and NAT.
The scans run on the machine itself, not an internal node.
After figuring out how build a debug kernel, and switch off encrypted
swap, I got the following stack trace:
#0 doadump (textdump=3D<value optimized out>) at pcpu.h:222
#1 0xffffffff80a6b6f1 in kern_reboot (howto=3D260) at =
/usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80a6bbb0 in vpanic (fmt=3D<value optimized out>, ap=3D<value=
optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80a6b9e3 in panic (fmt=3D<value optimized out>) at =
/usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff80edf832 in trap_fatal (frame=3D0xfffffe1041cc7260, =
eva=3D16) at /usr/src/sys/amd64/amd64/trap.c:801
#5 0xffffffff80edf889 in trap_pfault (frame=3D0xfffffe1041cc7260, =
usermode=3D0) at pcpu.h:222
#6 0xffffffff80edf0c6 in trap (frame=3D0xfffffe1041cc7260) at =
/usr/src/sys/amd64/amd64/trap.c:421
#7 0xffffffff80ec3641 in calltrap () at =
/usr/src/sys/amd64/amd64/exception.S:236
#8 0xffffffff80a49b50 in m_freem (mb=3D0x10) at =
/usr/src/sys/kern/kern_mbuf.c:952
#9 0xffffffff80c82b84 in ip6_output (m0=3D0xfffff80049f8e000, =
opt=3D<value optimized out>, ro=3D0xfffff80329e90530, flags=3D<value =
optimized out>,=20
im6o=3D<value optimized out>, ifpp=3D0x0, inp=3D<value optimized =
out>) at /usr/src/sys/netinet6/ip6_output.c:1040
#10 0xffffffff80c53269 in tcp_output (tp=3D0xfffff8027335c820) at =
/usr/src/sys/netinet/tcp_output.c:1403
#11 0xffffffff80c60f1b in tcp6_usr_connect (so=3D0xfffff80171dcc6c0, =
nam=3D<value optimized out>, td=3D0xfffff8017154c000) at =
/usr/src/sys/netinet/tcp_usrreq.c:642
#12 0xffffffff80af9bef in kern_connectat (td=3D<value optimized out>, =
dirfd=3D-100, fd=3D<value optimized out>, sa=3D0xfffff803296f3200)
at /usr/src/sys/kern/uipc_syscalls.c:584
#13 0xffffffff80af9aa7 in sys_connect (td=3D0xfffff8017154c000, =
uap=3D0xfffffe1041cc7930) at /usr/src/sys/kern/uipc_syscalls.c:549
#14 0xffffffff80ee0394 in amd64_syscall (td=3D0xfffff8017154c000, =
traced=3D0) at subr_syscall.c:135
#15 0xffffffff80ec392b in Xfast_syscall () at =
/usr/src/sys/amd64/amd64/exception.S:396
#16 0x0000000803beee5a in ?? ()
Peeking at frame 8, it seems that "mb" is not a plausible mbuf pointer:
(kgdb) fr 8
#8 0xffffffff80a49b50 in m_freem (mb=3D0x10) at =
/usr/src/sys/kern/kern_mbuf.c:952
952 MBUF_PROBE1(m__freem, mb);
(kgdb) l
947 */
948 void
949 m_freem(struct mbuf *mb)
950 {
951
952 MBUF_PROBE1(m__freem, mb);
953 while (mb !=3D NULL)
954 mb =3D m_free(mb);
955 }
(kgdb) p mb
$1 =3D (struct mbuf *) 0x10
Up a frame up I see:
(kgdb) up
#9 0xffffffff80c82b84 in ip6_output (m0=3D0xfffff80049f8e000, =
opt=3D<value optimized out>, ro=3D0xfffff80329e90530, flags=3D<value =
optimized out>,=20
im6o=3D<value optimized out>, ifpp=3D0x0, inp=3D<value optimized =
out>) at /usr/src/sys/netinet6/ip6_output.c:1040
1040 m_freem(m0);
(kgdb) l
1035 * Remove leading garbages.
1036 */
1037 sendorfree:
1038 m =3D m0->m_nextpkt;
1039 m0->m_nextpkt =3D 0;
1040 m_freem(m0);
1041 for (m0 =3D m; m; m =3D m0) {
1042 m0 =3D m->m_nextpkt;
1043 m->m_nextpkt =3D 0;
1044 if (error =3D=3D 0) {
(kgdb) p *m0
$4 =3D {{m_next =3D 0x10, m_slist =3D {sle_next =3D 0x10}, m_stailq =3D =
{stqe_next =3D 0x10}}, {m_nextpkt =3D 0x0, m_slistpkt =3D {sle_next =3D =
0x0}, m_stailqpkt =3D ...
Walking further up:
(kgdb) up
#10 0xffffffff80c53269 in tcp_output (tp=3D0xfffff8027335c820) at =
/usr/src/sys/netinet/tcp_output.c:1403
1403 error =3D ip6_output(m, =
tp->t_inpcb->in6p_outputopts,
(kgdb) l
1398 /* Save packet, if requested. */
1399 tcp_pcap_add(th, m, &(tp->t_outpkts));
1400 #endif
1401
1402 /* TODO: IPv6 IP6TOS_ECT bit on */
1403 error =3D ip6_output(m, =
tp->t_inpcb->in6p_outputopts,
1404 &tp->t_inpcb->inp_route6,
1405 ((so->so_options & SO_DONTROUTE) ? =
IP_ROUTETOIF : 0),
1406 NULL, NULL, tp->t_inpcb);
1407
(kgdb) up
#11 0xffffffff80c60f1b in tcp6_usr_connect (so=3D0xfffff80171dcc6c0, =
nam=3D<value optimized out>, td=3D0xfffff8017154c000) at =
/usr/src/sys/netinet/tcp_usrreq.c:642
642 error =3D tp->t_fb->tfb_tcp_output(tp);
(kgdb) l
637 (so->so_options & SO_NO_OFFLOAD) =3D=3D 0 &&
638 (error =3D tcp_offload_connect(so, nam)) =3D=3D 0)
639 goto out;
640 #endif
641 tcp_timer_activate(tp, TT_KEEP, TP_KEEPINIT(tp));
642 error =3D tp->t_fb->tfb_tcp_output(tp);
643
644 out:
645 TCPDEBUG2(PRU_CONNECT);
646 TCP_PROBE2(debug__user, tp, PRU_CONNECT);
(kgdb) up
#12 0xffffffff80af9bef in kern_connectat (td=3D<value optimized out>, =
dirfd=3D-100, fd=3D<value optimized out>, sa=3D0xfffff803296f3200)
at /usr/src/sys/kern/uipc_syscalls.c:584
584 error =3D soconnect(so, sa, td);
(kgdb) p *so
$2 =3D {so_count =3D 1, so_type =3D 1, so_options =3D 0, so_linger =3D =
0, so_state =3D 260, so_qstate =3D 0, so_pcb =3D 0xfffff80329e903a0, =
so_vnet =3D 0x0,=20
so_proto =3D 0xffffffff8198a2c0, so_head =3D 0x0, so_incomp =3D =
{tqh_first =3D 0x0, tqh_last =3D 0xfffff80171dcc6f0}, so_comp =3D =
{tqh_first =3D 0x0,=20
tqh_last =3D 0xfffff80171dcc700}, so_list =3D {tqe_next =3D 0x0, =
tqe_prev =3D 0x0}, so_qlen =3D 0, so_incqlen =3D 0, so_qlimit =3D 0, =
so_timeo =3D 0, so_error =3D 0,=20
so_sigio =3D 0x0, so_oobmark =3D 0, so_rcv =3D {sb_sel =3D {si_tdlist =
=3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, si_note =3D {kl_list =3D =
{slh_first =3D 0x0},=20
kl_lock =3D 0xffffffff80a23fb0 <knlist_mtx_lock>, kl_unlock =3D =
0xffffffff80a23ff0 <knlist_mtx_unlock>,=20
kl_assert_locked =3D 0xffffffff80a24030 =
<knlist_mtx_assert_locked>, kl_assert_unlocked =3D 0xffffffff80a24040 =
<knlist_mtx_assert_unlocked>,=20
kl_lockarg =3D 0xfffff80171dcc790, kl_autodestroy =3D 0}, si_mtx =
=3D 0x0}, sb_mtx =3D {lock_object =3D {lo_name =3D 0xffffffff8143469a =
"so_rcv",
lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0x0}, =
mtx_lock =3D 4}, sb_sx =3D {lock_object =3D {lo_name =3D =
0xffffffff814346ab "so_rcv_sx",
lo_flags =3D 36896768, lo_data =3D 0, lo_witness =3D 0x0}, =
sx_lock =3D 1}, sb_state =3D 0, sb_mb =3D 0x0, sb_mbtail =3D 0x0, =
sb_lastrecord =3D 0x0, sb_sndptr =3D 0x0,
sb_fnrdy =3D 0x0, sb_sndptroff =3D 0, sb_acc =3D 0, sb_ccc =3D 0, =
sb_hiwat =3D 65536, sb_mbcnt =3D 0, sb_mcnt =3D 0, sb_ccnt =3D 0, =
sb_mbmax =3D 524288, sb_ctl =3D 0,
sb_lowat =3D 1, sb_timeo =3D 0, sb_flags =3D 2048, sb_upcall =3D 0, =
sb_upcallarg =3D 0x0, sb_aiojobq =3D {tqh_first =3D 0x0, tqh_last =3D =
0xfffff80171dcc848},
sb_aiotask =3D {ta_link =3D {stqe_next =3D 0x0}, ta_pending =3D 0, =
ta_priority =3D 0, ta_func =3D 0xffffffff80ad2450 <soaio_rcv>, =
ta_context =3D 0xfffff80171dcc6c0}},
so_snd =3D {sb_sel =3D {si_tdlist =3D {tqh_first =3D 0x0, tqh_last =3D =
0x0}, si_note =3D {kl_list =3D {slh_first =3D 0x0}, kl_lock =3D =
0xffffffff80a23fb0 <knlist_mtx_lock>,
kl_unlock =3D 0xffffffff80a23ff0 <knlist_mtx_unlock>, =
kl_assert_locked =3D 0xffffffff80a24030 <knlist_mtx_assert_locked>,
kl_assert_unlocked =3D 0xffffffff80a24040 =
<knlist_mtx_assert_unlocked>, kl_lockarg =3D 0xfffff80171dcc8c8, =
kl_autodestroy =3D 0}, si_mtx =3D 0x0}, sb_mtx =3D {
lock_object =3D {lo_name =3D 0xffffffff81434693 "so_snd", lo_flags =
=3D 16973824, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 4}, sb_sx =
=3D {lock_object =3D {
lo_name =3D 0xffffffff814346a1 "so_snd_sx", lo_flags =3D =
36896768, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, sb_state =3D=
0, sb_mb =3D 0x0,
sb_mbtail =3D 0x0, sb_lastrecord =3D 0x0, sb_sndptr =3D 0x0, =
sb_fnrdy =3D 0x0, sb_sndptroff =3D 0, sb_acc =3D 0, sb_ccc =3D 0, =
sb_hiwat =3D 32768, sb_mbcnt =3D 0,
sb_mcnt =3D 0, sb_ccnt =3D 0, sb_mbmax =3D 262144, sb_ctl =3D 0, =
sb_lowat =3D 2048, sb_timeo =3D 0, sb_flags =3D 2048, sb_upcall =3D 0, =
sb_upcallarg =3D 0x0,
sb_aiojobq =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff80171dcc980}, =
sb_aiotask =3D {ta_link =3D {stqe_next =3D 0x0}, ta_pending =3D 0, =
ta_priority =3D 0,
ta_func =3D 0xffffffff80ad2d00 <soaio_snd>, ta_context =3D =
0xfffff80171dcc6c0}}, so_cred =3D 0xfffff80049ca4900, so_label =3D 0x0, =
so_peerlabel =3D 0x0,
so_gencnt =3D 21655, so_emuldata =3D 0x0, so_accf =3D 0x0, osd =3D =
{osd_nslots =3D 0, osd_slots =3D 0x0, osd_next =3D {le_next =3D 0x0, =
le_prev =3D 0x0}}, so_fibnum =3D 0,
so_user_cookie =3D 0, so_pspare =3D 0xfffff80171dcca08, so_ispare =3D =
0xfffff80171dcca18}
(kgdb) p/x (*sa->sa_data)@22
$7 =3D {0x0, 0x19, 0x0, 0x0, 0x0, 0x0, 0x2a, 0x1, 0x5b, 0x40, 0x0, 0x0, =
0x22, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1}
So the connection seems to be port 25 (expected), flow info 0, at
2a01:5b40:0:2201::1 (which is mx01.domeneshop.no, also not surprising).
Anything further I can report? It seems I'll have to disable IPv6 for
now...
--=20
Viktor.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FCC0833F-AA88-4F27-9DA3-4FA1218C49DB>