From owner-freebsd-bugs@FreeBSD.ORG Wed Oct 24 12:20:02 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4BE9E17D for ; Wed, 24 Oct 2012 12:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.FreeBSD.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id 32BE98FC08 for ; Wed, 24 Oct 2012 12:20:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9OCK11m027827 for ; Wed, 24 Oct 2012 12:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9OCK17C027826; Wed, 24 Oct 2012 12:20:01 GMT (envelope-from gnats) Date: Wed, 24 Oct 2012 12:20:01 GMT Message-Id: <201210241220.q9OCK17C027826@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org Cc: From: "Miller, Vincent (Rick)" Subject: kern/172963: Kernel panic in udp_input() X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: "Miller, Vincent \(Rick\)" List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 12:20:02 -0000 The following reply was made to PR kern/172963; it has been noted by GNATS. From: "Miller, Vincent (Rick)" To: "bug-followup@FreeBSD.org" , "Miller, Vincent (Rick)" Cc: Subject: kern/172963: Kernel panic in udp_input() Date: Wed, 24 Oct 2012 12:10:16 +0000 --_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Some of our developers spent a significant amount of time troubleshooting t= his. The race condition has been identified. The relevant information is = below: It is a race condition between: - udp_input() - udp_detach() - udp_pcblist() How to reproduce: - Call sysctl(" net.inet.udp.pcblist") continuously with: $ while true; do sysctl -x net.inet.udp.pcblist > /dev/null; done; - Open a bunch of UDP sockets (to slow down udp_pcblist() call): $ for port in $(jot - 20000 24000 1); do socat -u -T 1 UDP4-LISTEN:$port,re= useaddr GOPEN:/dev/null & done - Launch a UDP server that close() after each request it received: $ while true; do socat -u -T 0.0001 UDP4-LISTEN:12345,reuseaddr GOPEN:/dev/= null; done - Bombard this server with UDP request from another machine (could be also= on the same machine): $ while true; do socat -u EXEC:'/bin/echo' UDP4:10.51.33.40:12345 & sleep 0= .0001; done - Check these messages in /var/log/messages after 5/10 minutes: Oct 24 09:05:21 flumpe4-qa2 kernel: udp_input(): Using freed inp 0xffffff0d= f43d7930 inp->inp_refcount 1 inp->inp_ppcb 0 Oct 24 09:05:21 flumpe4-qa2 kernel: udp_pcblist(): Using freed inp 0xffffff= 0df43d7930 inp->inp_refcount 1 inp->inp_ppcb 0 (inp pointer being the same for udp_input() _and_ udp_pcblist(), _and_ inp= ->inp_ppcb being NULL) Which is the proof that: udp_input() retrieves an inp pointer 0xffffff0df4= 3d7930 _and_ this inp has been released in_pcbrele() but not deleted becaus= e someone had still a reference on it (flag INP_FREED introduced by debug p= atch) _and_ udp_detach() has been called on this inp (inp->inp_ppcb being N= ULL) _and_ this is udp_pcblist() that holds this reference on this inp. Without the patch the crash will occur in udp_input(): /* Check inp state */ if ((inp->inp_flags2 & INP_FREED) && (inp->inp_socket =3D=3D NULL))= { log(LOG_INFO, "udp_input(): Using freed inp %p inp->inp_ref= count %d inp->inp_ppcb %p\n", inp, inp->inp_refcount, inp->inp_ppcb); INP_RUNLOCK(inp); goto badunlocked; } up =3D intoudpcb(inp); // intoudpcb(ip) being ((struct udpcb *)(ip)= ->inp_ppcb) and inp_ppcb being NULL... if (up->u_tun_func =3D=3D NULL) { // Panic here in default kernel --_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_ Content-Type: text/html; charset="us-ascii" Content-ID: <6AB7A0C7A250C648A166B4B1E0BEA58E@verisign.com> Content-Transfer-Encoding: quoted-printable
Some of our developers spent a significant amount of time troubleshoot= ing this.  The race condition has been identified.  The relevant = information is below:

It is a race condition between:

 - udp_input()
 - udp_detach()
 - udp_pcblist()

 How to reproduce:

 - Call  sysctl(" net.inet.udp.pcblist") continuou= sly with:

$ while true; do sysctl = -x net.inet.udp.pcblist > /dev/null; done;

 - Open a bunch of UDP sockets (to slow down udp_pcblist() call):

$ for port in $(jot - 20= 000 24000 1); do socat -u -T 1 UDP4-LISTEN:$port,reuseaddr GOPEN:/dev/null = & done

 - Launch a UDP server that close() after each request it received:

$ while true; do socat -= u -T 0.0001 UDP4-LISTEN:12345,reuseaddr GOPEN:/dev/null; done

 - Bombard this server with UDP request from another machine (could be= also on the same machine):

$ while true; do socat -= u EXEC:'/bin/echo' UDP4:10.51.33.40:12345 & sleep 0.0001; done

 - Check these messages in /var/log/messages after 5/10 minutes:

Oct 24 09:05:21 flu= mpe4-qa2 kernel: udp_input(): Using freed inp 0xffffff0df43d7930 inp->in= p_refcount 1 inp->inp_ppcb 0
Oct 24 09:05:21 flu= mpe4-qa2 kernel: udp_pcblist(): Using freed inp 0xffffff0df43d7930 inp->= inp_refcount 1 inp->inp_ppcb 0

 (inp pointer being the same for udp_input() _and_ udp_pcblist(), _and= _ inp->inp_ppcb being NULL)

 Which is the proof that: udp_input() retrieves an inp pointer 0xffffff0df43d7930 _and_ this inp has been released in_pcbrele() but not deleted because someone had sti= ll a reference on it (flag INP_FREED introduced by debug patch) _and_ udp_detach() has be= en called on this inp (inp->inp_ppcb being NULL) _and_ this is udp_pcbli= st() that holds this reference on this inp.
  
 Without the patch the crash will occur in udp_input():

        /* Check inp state */
        if ((inp->inp_flags2 & INP_= FREED) && (inp->inp_socket =3D=3D NULL)) {
                log(LO= G_INFO, "udp_input(): Using freed inp %p inp->inp_refcount %d inp-&= gt;inp_ppcb %p\n",
                 =       inp, inp->inp_refcount, inp->inp_ppcb);<= /div>
                INP_RU= NLOCK(inp);
                goto b= adunlocked;
        }
intoudpcb(ip) being ((str= uct udpcb *)(ip)->inp_ppcb) and inp_ppcb being NULL...
   =     if (up->u_tun_func =3D=3D NULL) { // Panic here in d= efault kernel
--_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_--