Date: Mon, 02 Feb 2015 19:57:43 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 148807] [panic] 8.1-RELEASE "panic: sbdrop" and "panic: sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load Message-ID: <bug-148807-2472-IkTXTBw9li@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-148807-2472@https.bugs.freebsd.org/bugzilla/> References: <bug-148807-2472@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D148807 --- Comment #14 from Andrey V. Elsukov <ae@FreeBSD.org> --- Second panic: panic: sbsndptr: sockbuf 0xfffffe03e62b5c20 and mbuf 0xfffffe01d8fd3900 clashing cpuid =3D 31 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff90d4fca= 430 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff90d4fca4f0 panic() at panic+0x1ce/frame 0xffffff90d4fca5f0 sbsndptr() at sbsndptr+0xe4/frame 0xffffff90d4fca610 tcp_output() at tcp_output+0x16cd/frame 0xffffff90d4fca7c0 tcp_usr_send() at tcp_usr_send+0x325/frame 0xffffff90d4fca820 sosend_generic() at sosend_generic+0x3f6/frame 0xffffff90d4fca8c0 soo_write() at soo_write+0x5e/frame 0xffffff90d4fca8f0 dofilewrite() at dofilewrite+0x85/frame 0xffffff90d4fca940 kern_writev() at kern_writev+0x6c/frame 0xffffff90d4fca980 sys_write() at sys_write+0x64/frame 0xffffff90d4fca9d0 amd64_syscall() at amd64_syscall+0x5ea/frame 0xffffff90d4fcaaf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff90d4fcaaf0 --- syscall (4, FreeBSD ELF64, sys_write), rip =3D 0x802da3bec, rsp =3D 0x7fffffffdae8, rbp =3D 0x7fffffffdbf0 --- Uptime: 1m48s Dumping 3468 out of 65475 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/if_igb.ko...Reading symbols from /boot/kernel/if_igb.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_igb.ko Reading symbols from /boot/kernel/aac.ko...Reading symbols from /boot/kernel/aac.ko.symbols...done. done. Loaded symbols for /boot/kernel/aac.ko Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from /boot/kernel/ipdivert.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipdivert.ko Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipfw.ko Reading symbols from /boot/kernel/t5fw_cfg.ko...Reading symbols from /boot/kernel/t5fw_cfg.ko.symbols...done. done. Loaded symbols for /boot/kernel/t5fw_cfg.ko Reading symbols from /boot/kernel/if_cxgbe.ko...Reading symbols from /boot/kernel/if_cxgbe.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_cxgbe.ko Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipmi.ko Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /boot/kernel/smbus.ko.symbols...done. done. Loaded symbols for /boot/kernel/smbus.ko #0 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:271 271 if (textdump && textdump_pending) { (kgdb) bt #0 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:271 #1 0xffffffff80907eb4 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:454 #2 0xffffffff809083a7 in panic (fmt=3D0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:642 #3 0xffffffff809766e4 in sbsndptr (sb=3D<value optimized out>, off=3D<value optimized out>, len=3D<value optimized out>, moff=3D<value optimized out>) at /usr/src/sys/kern/uipc_sockbuf.c:985 #4 0xffffffff80aaedbd in tcp_output (tp=3D0xfffffe03e675a3d0) at /usr/src/sys/netinet/tcp_output.c:954 #5 0xffffffff80abc555 in tcp_usr_send (so=3D0xfffffe03e62b5aa0, flags=3D0, m=3D0xfffffe01d8fd2200, nam=3D0x0, control=3D<value optimized out>, td=3D0xfffffe0021e90000) at /usr/src/sys/netinet/tcp_usrreq.c:874 #6 0xffffffff8097c1f6 in sosend_generic (so=3D0xfffffe03e62b5aa0, addr=3D0= x0, uio=3D0xffffff90d4fca990, top=3D0xfffffe01d8fd2200, control=3D0x0, flags=3D= <value optimized out>,=20 td=3D0xfffffe0021e90000) at /usr/src/sys/kern/uipc_socket.c:1376 #7 0xffffffff8095ea6e in soo_write (fp=3D<value optimized out>, uio=3D0xffffff90d4fca990, active_cred=3D<value optimized out>, flags=3D<val= ue optimized out>,=20 td=3D<value optimized out>) at /usr/src/sys/kern/sys_socket.c:102 #8 0xffffffff80957195 in dofilewrite (td=3D0xfffffe0021e90000, fd=3D3, fp=3D0xfffffe0021cf3820, auio=3D0xffffff90d4fca990, offset=3D<value optimiz= ed out>, flags=3D0) at file.h:295 #9 0xffffffff809574cc in kern_writev (td=3D0xfffffe0021e90000, fd=3D3, auio=3D0xffffff90d4fca990) at /usr/src/sys/kern/sys_generic.c:477 #10 0xffffffff80957554 in sys_write (td=3D<value optimized out>, uap=3D<val= ue optimized out>) at /usr/src/sys/kern/sys_generic.c:393 #11 0xffffffff80cfea4a in amd64_syscall (td=3D0xfffffe0021e90000, traced=3D= 0) at subr_syscall.c:135 #12 0xffffffff80ce8ac7 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #13 0x0000000802da3bec in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) p *(struct sockbuf *)0xfffffe03e62b5c20 $1 =3D {sb_sel =3D {si_tdlist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, si= _note =3D {kl_list =3D {slh_first =3D 0x0}, kl_lock =3D 0xffffffff808cd0c0 <knlist_mt= x_lock>,=20 kl_unlock =3D 0xffffffff808cd090 <knlist_mtx_unlock>, kl_assert_locke= d =3D 0xffffffff808c9a10 <knlist_mtx_assert_locked>,=20 kl_assert_unlocked =3D 0xffffffff808c9a20 <knlist_mtx_assert_unlocked= >, kl_lockarg =3D 0xfffffe03e62b5c68}, si_mtx =3D 0x0}, sb_mtx =3D {lock_objec= t =3D { lo_name =3D 0xffffffff80f3e7fd "so_snd", lo_flags =3D 16973824, lo_da= ta =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 18446741875255214080}, sb_sx =3D {lock_ob= ject =3D { lo_name =3D 0xffffffff80f3ed6b "so_snd_sx", lo_flags =3D 36896768, lo= _data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 18446741875255214080}, sb_state =3D 0,= =20 sb_mb =3D 0xfffffe01f4069900, sb_mbtail =3D 0xfffffe01d8fd3900, sb_lastre= cord =3D 0xfffffe01f4069900, sb_sndptr =3D 0xfffffe01d8fd3900, sb_sndptroff =3D 1632= , sb_cc =3D 1716,=20 sb_hiwat =3D 131376, sb_mbcnt =3D 4864, sb_mcnt =3D 11, sb_ccnt =3D 1, sb= _mbmax =3D 1051008, sb_ctl =3D 0, sb_lowat =3D 2048, sb_timeo =3D 0, sb_flags =3D 2048= , sb_upcall =3D 0,=20 sb_upcallarg =3D 0x0} (kgdb) p *(struct mbuf *)0xfffffe01d8fd3900 $2 =3D {m_hdr =3D {mh_next =3D 0x0, mh_nextpkt =3D 0x0, mh_data =3D 0xfffff= e01d8fd3928 "", mh_len =3D 68, mh_flags =3D 0, mh_type =3D 1, pad =3D "\000\000\000\000= \000"}, M_dat =3D {MH =3D { MH_pkthdr =3D {rcvif =3D 0xb1dee9e530000000, header =3D 0xf10fc01307a= ab916, len =3D -337628730, flowid =3D 2682375970, csum_flags =3D -966380398, csum_data= =3D -1624117065,=20 tso_segsz =3D 11596, PH_vt =3D {vt_vtag =3D 31606, vt_nrecs =3D 316= 06}, tags =3D {slh_first =3D 0xa2b0a659a4311f25}}, MH_dat =3D {MH_ext =3D { ext_buf =3D 0x43772562c99aa431 <Address 0x43772562c99aa431 out of bounds>, ext_free =3D 0x7e1cffd9b6b13fc6, ext_arg1 =3D 0x731c9ab425536605,= =20 ext_arg2 =3D 0xebc6cac44b21a941, ext_size =3D 520953289, ref_cnt = =3D 0x5165381046dcad94, ext_type =3D 1308134978},=20 MH_databuf =3D "1=EF=BF=BD\232=EF=BF=BDb%wC=EF=BF=BD?=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD\= 034~\005fS%=EF=BF=BD\232\034sA=EF=BF=BD!K=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD\035\r\037I=DC=A1q\224=EF=BF=BD=EF=BF=BDF\0208eQB\216=EF=BF=BDM= =EF=BF=BDP=EF=BF=BD/\000\026OS^Lq%=EF=BF=BDMY\212\200\030\b\004\021\000\000= \000\001\001\b\n2=EF=BF=BD=EF=BF=BD \v=EF=BF=BD=EF=BF=BDO\000\000\000 =EF=BF=BD=EF=BF=BDn=EF=BF=BD=D9=BB=EF=BF=BDEr\032S\201\220\220=EF=BF=BD=EF= =BF=BDI=EF=BF=BD\"\210\233\v\0223?=3D=EF=BF=BD*a|\231\001\022=EF=BF=BD6}=EF= =BF=BDG=EF=BF=BD\026=EF=BF=BD\036z\n\023=EF=BF=BD<=EF=BF=BD=EF=BF=BD=EF=BF= =BDB8=EF=BF=BD\200\000\000\000\000\000\000\002%\220=EF=BF=BD=EF=BF=BD=EF=BF= =BDB8\001\003Ip\000\000\000"}},=20 M_databuf =3D "\000\000\0000=EF=BF=BD=EF=BF=BD=DE=B1\026=EF=BF=BD=EF=BF=BD\a\023=EF=BF=BD= \017=EF=BF=BD=EF=BF=BD1=EF=BF=BD=EF=BF=BD\"=EF=BF=BD=EF=BF=BD\237\2224f=C6= =B7=EF=BF=BD1\237L-v{X=EF=BF=BD\235\214%\0371=EF=BF=BDY=EF=BF=BD=EF=BF=BD= =EF=BF=BD1=EF=BF=BD\232=EF=BF=BDb%wC=EF=BF=BD?=EF=BF=BD=EF=BF=BD=EF=BF=BD= =EF=BF=BD\034~\005fS%=EF=BF=BD\232\034sA=EF=BF=BD!K=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD\035\r\037I=DC=A1q\224=EF=BF=BD=EF=BF=BDF\0208eQB\216= =EF=BF=BDM=EF=BF=BDP=EF=BF=BD/\000\026OS^Lq%=EF=BF=BDMY\212\200\030\b\004\0= 21\000\000\000\001\001\b\n2=EF=BF=BD=EF=BF=BD \v=EF=BF=BD=EF=BF=BDO\000\000\000 =EF=BF=BD=EF=BF=BDn=EF=BF=BD=D9=BB=EF=BF=BDEr\032S\201\220\220=EF=BF=BD=EF= =BF=BDI=EF=BF=BD\"\210\233\v\0223?=3D=EF=BF=BD*a|\231\001\022=EF=BF=BD6}=EF= =BF=BDG=EF=BF=BD\026=EF=BF=BD\036z\n\023=EF=BF=BD<=EF=BF=BD=EF=BF=BD=EF=BF= =BDB8=EF=BF=BD\200\000\000\000\000\000\000"...}} (kgdb) f 6 #6 0xffffffff8097c1f6 in sosend_generic (so=3D0xfffffe03e62b5aa0, addr=3D0= x0, uio=3D0xffffff90d4fca990, top=3D0xfffffe01d8fd2200, control=3D0x0, flags=3D= <value optimized out>,=20 td=3D0xfffffe0021e90000) at /usr/src/sys/kern/uipc_socket.c:1376 1376 error =3D (*so->so_proto->pr_usrreqs->pru_send)(so, (kgdb) p *so $3 =3D {so_count =3D 1, so_type =3D 1, so_options =3D 12, so_linger =3D 0, = so_state =3D 258, so_qstate =3D 0, so_pcb =3D 0xfffffe03e678a640, so_vnet =3D 0x0,=20 so_proto =3D 0xffffffff8143c3f0, so_head =3D 0x0, so_incomp =3D {tqh_firs= t =3D 0x0, tqh_last =3D 0x0}, so_comp =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, so_li= st =3D {tqe_next =3D 0x0,=20 tqe_prev =3D 0xfffffe01d8f96040}, so_qlen =3D 0, so_incqlen =3D 0, so_q= limit =3D 0, so_timeo =3D 0, so_error =3D 0, so_sigio =3D 0x0, so_oobmark =3D 0, so_aioj= obq =3D { tqh_first =3D 0x0, tqh_last =3D 0xfffffe03e62b5b20}, so_rcv =3D {sb_sel= =3D {si_tdlist =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffffe03e62b5b30}, si_not= e =3D {kl_list =3D { slh_first =3D 0x0}, kl_lock =3D 0xffffffff808cd0c0 <knlist_mtx_lo= ck>, kl_unlock =3D 0xffffffff808cd090 <knlist_mtx_unlock>,=20 kl_assert_locked =3D 0xffffffff808c9a10 <knlist_mtx_assert_locked>, kl_assert_unlocked =3D 0xffffffff808c9a20 <knlist_mtx_assert_unlocked>,=20 kl_lockarg =3D 0xfffffe03e62b5b78}, si_mtx =3D 0xffffff800e02f670},= sb_mtx =3D {lock_object =3D {lo_name =3D 0xffffffff80f3e7f6 "so_rcv", lo_flags =3D= 16973824,=20 lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 4}, sb_sx =3D {loc= k_object =3D {lo_name =3D 0xffffffff80f3ed75 "so_rcv_sx", lo_flags =3D 36896768, lo_data= =3D 0,=20 lo_witness =3D 0x0}, sx_lock =3D 1}, sb_state =3D 0, sb_mb =3D 0x0,= sb_mbtail =3D 0x0, sb_lastrecord =3D 0x0, sb_sndptr =3D 0x0, sb_sndptroff =3D 0, sb_cc = =3D 0,=20 sb_hiwat =3D 131376, sb_mbcnt =3D 0, sb_mcnt =3D 0, sb_ccnt =3D 0, sb_m= bmax =3D 1051008, sb_ctl =3D 0, sb_lowat =3D 1, sb_timeo =3D 0, sb_flags =3D 2056, s= b_upcall =3D 0,=20 sb_upcallarg =3D 0x0}, so_snd =3D {sb_sel =3D {si_tdlist =3D {tqh_first= =3D 0x0, tqh_last =3D 0x0}, si_note =3D {kl_list =3D {slh_first =3D 0x0},=20 kl_lock =3D 0xffffffff808cd0c0 <knlist_mtx_lock>, kl_unlock =3D 0xffffffff808cd090 <knlist_mtx_unlock>,=20 kl_assert_locked =3D 0xffffffff808c9a10 <knlist_mtx_assert_locked>, kl_assert_unlocked =3D 0xffffffff808c9a20 <knlist_mtx_assert_unlocked>,=20 kl_lockarg =3D 0xfffffe03e62b5c68}, si_mtx =3D 0x0}, sb_mtx =3D {lo= ck_object =3D {lo_name =3D 0xffffffff80f3e7fd "so_snd", lo_flags =3D 16973824, lo_dat= a =3D 0,=20 lo_witness =3D 0x0}, mtx_lock =3D 18446741875255214080}, sb_sx =3D {lock_object =3D {lo_name =3D 0xffffffff80f3ed6b "so_snd_sx", lo_flags =3D = 36896768, lo_data =3D 0,=20 lo_witness =3D 0x0}, sx_lock =3D 18446741875255214080}, sb_state = =3D 0, sb_mb =3D 0xfffffe01f4069900, sb_mbtail =3D 0xfffffe01d8fd3900,=20 sb_lastrecord =3D 0xfffffe01f4069900, sb_sndptr =3D 0xfffffe01d8fd3900, sb_sndptroff =3D 1632, sb_cc =3D 1716, sb_hiwat =3D 131376, sb_mbcnt =3D 48= 64, sb_mcnt =3D 11,=20 sb_ccnt =3D 1, sb_mbmax =3D 1051008, sb_ctl =3D 0, sb_lowat =3D 2048, s= b_timeo =3D 0, sb_flags =3D 2048, sb_upcall =3D 0, sb_upcallarg =3D 0x0}, so_cred =3D 0xfffffe01f48ce900,=20 so_label =3D 0x0, so_peerlabel =3D 0x0, so_gencnt =3D 13244, so_emuldata = =3D 0x0, so_accf =3D 0x0, so_fibnum =3D 0, so_user_cookie =3D 0} (kgdb) set $inp=3D(struct inpcb *)so->so_pcb (kgdb) p *$inp $4 =3D {inp_hash =3D {le_next =3D 0x0, le_prev =3D 0xfffffe0012f573b0}, inp_pcbgrouphash =3D {le_next =3D 0x0, le_prev =3D 0x0}, inp_list =3D {le_n= ext =3D 0xfffffe03e679bc80,=20 le_prev =3D 0xfffffe03e6743020}, inp_ppcb =3D 0xfffffe03e675a3d0, inp_p= cbinfo =3D 0xffffffff81531060, inp_pcbgroup =3D 0x0, inp_pcbgroup_wild =3D {le_next = =3D 0x0,=20 le_prev =3D 0x0}, inp_socket =3D 0xfffffe03e62b5aa0, inp_cred =3D 0xfffffe01f48ce900, inp_flow =3D 3457486592, inp_flags =3D 545300480, inp_f= lags2 =3D 0, inp_vflag =3D 6 '\006',=20 inp_ip_ttl =3D 64 '@', inp_ip_p =3D 0 '\0', inp_ip_minttl =3D 0 '\0', inp= _flowid =3D 1779132015, inp_refcount =3D 1, inp_pspare =3D {0x0, 0x0, 0x0, 0x0, 0x0}, inp_ispare =3D {0, 0,=20 0, 0, 0, 0}, inp_inc =3D {inc_flags =3D 1 '\001', inc_len =3D 0 '\0', i= nc_fibnum =3D 0, inc_ie =3D {ie_fport =3D 21327, ie_lport =3D 5632, ie_dependfaddr =3D {ie46_foreign =3D { ia46_pad32 =3D {3087401514, 17039360, 4283245058}, ia46_addr4 =3D= {s_addr =3D 801984766}}, ie6_foreign =3D {__u6_addr =3D { __u6_addr8 =3D "*\002\006=EF=BF=BD\000\000\004\001\002\"M=EF=BF= =BD=EF=BF=BDP=EF=BF=BD/", __u6_addr16 =3D {554, 47110, 0, 260, 8706, 65357, 20734, 12237}, __u6_addr32 =3D {30874= 01514, 17039360,=20 4283245058, 801984766}}}}, ie_dependladdr =3D {ie46_local =3D {ia46_pad32 =3D {3087401514, 917504, 0}, ia46_addr4 =3D {s_addr =3D 1375797= 248}}, ie6_local =3D { __u6_addr =3D {__u6_addr8 =3D "*\002\006=EF=BF=BD\000\000\016\000\000\000\000\000\000\000\001R", __u6_add= r16 =3D {554, 47110, 0, 14, 0, 0, 0, 20993}, __u6_addr32 =3D { 3087401514, 917504, 0, 1375797248}}}}, ie6_zoneid =3D 0}}, inp_label =3D 0x0, inp_sp =3D 0x0, inp_depend4 =3D {inp4_ip_tos =3D 0 '\0', inp4_options =3D 0x0,=20 inp4_moptions =3D 0x0}, inp_depend6 =3D {inp6_options =3D 0x0, inp6_out= putopts =3D 0xfffffe0013424500, inp6_moptions =3D 0x0, inp6_icmp6filt =3D 0x0, inp6_cks= um =3D 0,=20 inp6_hops =3D -1}, inp_portlist =3D {le_next =3D 0xfffffe03e6d8f640, le= _prev =3D 0xfffffe03e6743140}, inp_phd =3D 0xfffffe03e6dfa540, inp_gencnt =3D 1509, i= np_lle =3D 0x0,=20 inp_rt =3D 0x0, inp_lock =3D {lock_object =3D {lo_name =3D 0xffffffff80f5= 9235 "tcpinp", lo_flags =3D 90898432, lo_data =3D 0, lo_witness =3D 0x0}, rw_loc= k =3D 18446741875255214080}} --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-148807-2472-IkTXTBw9li>