Skip site navigation (1)Skip section navigation (2) 
Date:      Wed, 24 Oct 2012 12:20:01 GMT
From:      "Miller, Vincent (Rick)" <vmiller@verisign.com>
To:        freebsd-bugs@FreeBSD.org
Subject:   kern/172963: Kernel panic in udp_input()
Message-ID:  <201210241220.q9OCK17C027826@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help 
The following reply was made to PR kern/172963; it has been noted by GNATS.

From: "Miller, Vincent (Rick)" <vmiller@verisign.com>
To: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>,
        "Miller, Vincent
 (Rick)" <vmiller@verisign.com>
Cc:  
Subject: kern/172963: Kernel panic in udp_input()
Date: Wed, 24 Oct 2012 12:10:16 +0000

 --_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_
 Content-Type: text/plain; charset="us-ascii"
 Content-Transfer-Encoding: quoted-printable
 
 Some of our developers spent a significant amount of time troubleshooting t=
 his.  The race condition has been identified.  The relevant information is =
 below:
 
 It is a race condition between:
 
  - udp_input()
  - udp_detach()
  - udp_pcblist()
 
  How to reproduce:
 
  - Call  sysctl(" net.inet.udp.pcblist") continuously with:
 
 $ while true; do sysctl -x net.inet.udp.pcblist > /dev/null; done;
 
  - Open a bunch of UDP sockets (to slow down udp_pcblist() call):
 
 $ for port in $(jot - 20000 24000 1); do socat -u -T 1 UDP4-LISTEN:$port,re=
 useaddr GOPEN:/dev/null & done
 
  - Launch a UDP server that close() after each request it received:
 
 $ while true; do socat -u -T 0.0001 UDP4-LISTEN:12345,reuseaddr GOPEN:/dev/=
 null; done
 
  - Bombard this server with UDP request from another machine (could be also=
  on the same machine):
 
 $ while true; do socat -u EXEC:'/bin/echo' UDP4:10.51.33.40:12345 & sleep 0=
 .0001; done
 
  - Check these messages in /var/log/messages after 5/10 minutes:
 
 Oct 24 09:05:21 flumpe4-qa2 kernel: udp_input(): Using freed inp 0xffffff0d=
 f43d7930 inp->inp_refcount 1 inp->inp_ppcb 0
 Oct 24 09:05:21 flumpe4-qa2 kernel: udp_pcblist(): Using freed inp 0xffffff=
 0df43d7930 inp->inp_refcount 1 inp->inp_ppcb 0
 
  (inp pointer being the same for udp_input() _and_ udp_pcblist(), _and_ inp=
 ->inp_ppcb being NULL)
 
  Which is the proof that: udp_input() retrieves an inp pointer 0xffffff0df4=
 3d7930 _and_ this inp has been released in_pcbrele() but not deleted becaus=
 e someone had still a reference on it (flag INP_FREED introduced by debug p=
 atch) _and_ udp_detach() has been called on this inp (inp->inp_ppcb being N=
 ULL) _and_ this is udp_pcblist() that holds this reference on this inp.
 
  Without the patch the crash will occur in udp_input():
 
         /* Check inp state */
         if ((inp->inp_flags2 & INP_FREED) && (inp->inp_socket =3D=3D NULL))=
  {
                 log(LOG_INFO, "udp_input(): Using freed inp %p inp->inp_ref=
 count %d inp->inp_ppcb %p\n",
                         inp, inp->inp_refcount, inp->inp_ppcb);
                 INP_RUNLOCK(inp);
                 goto badunlocked;
         }
         up =3D intoudpcb(inp); // intoudpcb(ip) being ((struct udpcb *)(ip)=
 ->inp_ppcb) and inp_ppcb being NULL...
         if (up->u_tun_func =3D=3D NULL) { // Panic here in default kernel
 
 --_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_
 Content-Type: text/html; charset="us-ascii"
 Content-ID: <6AB7A0C7A250C648A166B4B1E0BEA58E@verisign.com>
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
 >
 </head>
 <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
 e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami=
 ly: Calibri, sans-serif; ">
 <div>
 <div>
 <div>Some of our developers spent a significant amount of time troubleshoot=
 ing this. &nbsp;The race condition has been identified. &nbsp;The relevant =
 information is below:</div>
 <div><br>
 </div>
 <div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 It is a race condition between:</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;- udp_input()</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;-&nbsp;udp_detach()</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;-&nbsp;udp_pcblist()</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;How to reproduce:</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;- Call &nbsp;sysctl(&quot;&nbsp;net.inet.udp.pcblist&quot;) continuou=
 sly with:</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <font face=3D"Courier" style=3D"font-size: 12px; ">$ while true; do sysctl =
 -x net.inet.udp.pcblist &gt; /dev/null; done;</font></div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;- Open a bunch of UDP sockets (to slow down udp_pcblist() call):</div=
 >
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <font face=3D"Courier" style=3D"font-size: 12px; ">$ for port in $(jot - 20=
 000 24000 1); do socat -u -T 1 UDP4-LISTEN:$port,reuseaddr GOPEN:/dev/null =
 &amp; done</font></div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;- Launch a UDP server that close() after each request it received:</d=
 iv>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <font face=3D"Courier" style=3D"font-size: 12px; ">$ while true; do socat -=
 u -T 0.0001 UDP4-LISTEN:12345,reuseaddr GOPEN:/dev/null; done</font></div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;- Bombard this server with UDP request from another machine (could be=
  also on the same machine):</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <font face=3D"Courier" style=3D"font-size: 12px; ">$ while true; do socat -=
 u EXEC:'/bin/echo' UDP4:10.51.33.40:12345 &amp; sleep 0.0001; done</font></=
 div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;- Check these messages in /var/log/messages after 5/10 minutes:</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <div><font face=3D"Courier" style=3D"font-size: 12px; ">Oct 24 09:05:21 flu=
 mpe4-qa2 kernel: udp_input(): Using freed inp 0xffffff0df43d7930 inp-&gt;in=
 p_refcount 1 inp-&gt;inp_ppcb 0</font></div>
 <div><font face=3D"Courier" style=3D"font-size: 12px; ">Oct 24 09:05:21 flu=
 mpe4-qa2 kernel: udp_pcblist(): Using freed inp 0xffffff0df43d7930 inp-&gt;=
 inp_refcount 1 inp-&gt;inp_ppcb 0</font></div>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <div><br>
 </div>
 </div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;(inp pointer being the same for udp_input() _and_ udp_pcblist(), _and=
 _ inp-&gt;inp_ppcb being NULL)</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"font-family: Calibri; font-size: medium; "><font face=3D"Cali=
 bri,sans-serif">&nbsp;Which is the proof that:&nbsp;</font><font face=3D"Ca=
 libri,sans-serif">udp_input() retrieves an&nbsp;inp pointer&nbsp;</font><fo=
 nt face=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0, 0); font-family: C=
 alibri, sans-serif; font-size: 14px; ">0xffffff0df43d7930</font><font face=
 =3D"Calibri,sans-serif" style=3D"color: rgb(0, 0, 0); font-family: Calibri,=
  sans-serif; font-size: 14px; ">&nbsp;_and_&nbsp;</font><font face=3D"Calib=
 ri,sans-serif" style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-ser=
 if; font-size: 14px; ">this
  inp has been released in_pcbrele() but not deleted because someone had sti=
 ll a&nbsp;reference on it&nbsp;(flag INP_FREED introduced by debug patch</f=
 ont><font face=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0, 0); font-fa=
 mily: Calibri, sans-serif; font-size: 14px; ">)
  _and_ udp</font><font face=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0=
 , 0); font-family: Calibri, sans-serif; font-size: 14px; ">_detach() has be=
 en called on this inp (inp-&gt;inp_ppcb being NULL) _and_ this is udp_pcbli=
 st() that holds this reference on this&nbsp;</font><font face=3D"Calibri,sa=
 ns-serif" style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; f=
 ont-size: 14px; ">inp</font><font face=3D"Calibri,sans-serif">.</font></div=
 >
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;&nbsp;</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 &nbsp;Without the patch the crash will occur in udp_input():</div>
 <div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
 ize: 14px; ">
 <br>
 </div>
 <div style=3D"font-family: Calibri; font-size: medium; ">
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; /* Check inp state */</font></div>
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; if ((inp-&gt;inp_flags2 &amp; INP_=
 FREED) &amp;&amp; (inp-&gt;inp_socket =3D=3D NULL)) {</font></div>
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; log(LO=
 G_INFO, &quot;udp_input(): Using freed inp %p inp-&gt;inp_refcount %d inp-&=
 gt;inp_ppcb %p\n&quot;,</font></div>
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;=
  &nbsp; &nbsp; &nbsp; inp, inp-&gt;inp_refcount, inp-&gt;inp_ppcb);</font><=
 /div>
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; INP_RU=
 NLOCK(inp);</font></div>
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; goto b=
 adunlocked;</font></div>
 <div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
 ize: 12px; ">&nbsp; &nbsp; &nbsp; &nbsp; }</font></div>
 <div><font face=3D"Courier" style=3D"color: rgb(0, 0, 0); font-size: 12px; =
 ">&nbsp; &nbsp; &nbsp; &nbsp; up =3D intoudpcb(inp); //&nbsp;</font><font f=
 ace=3D"Courier"><span style=3D"font-size: 12px; ">intoudpcb(ip) being ((str=
 uct udpcb *)(ip)-&gt;inp_ppcb) and&nbsp;</span></font><span style=3D"font-f=
 amily: Courier; font-size: 12px; ">inp_ppcb
  being NULL...</span></div>
 <div>
 <div><font face=3D"Courier"><span style=3D"font-size: 12px; ">&nbsp; &nbsp;=
  &nbsp; &nbsp; if (up-&gt;u_tun_func =3D=3D NULL) { // Panic here in&nbsp;d=
 efault kernel</span></font></div>
 </div>
 </div>
 </div>
 </div>
 </div>
 </body>
 </html>
 
 --_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_--

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201210241220.q9OCK17C027826>