Date: Fri, 1 Aug 2003 21:51:01 -0500 (CDT) From: Mike Silbersack <silby@silby.com> To: Scot Loach <sloach@sandvine.com> Cc: "'freebsd-net@freebsd.org'" <freebsd-net@freebsd.org> Subject: Re: TCP socket shutdown race condition Message-ID: <20030801214411.A2165@odysseus.silby.com> In-Reply-To: <FE045D4D9F7AED4CBFF1B3B813C8533701AE86B5@mail.sandvine.com> References: <FE045D4D9F7AED4CBFF1B3B813C8533701AE86B5@mail.sandvine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 1 Aug 2003, Scot Loach wrote:
> Earlier this week one of our FreeBSD 4.7 boxes panic'd. I've posted the
> stack trace at the end of this message. Using google, I've found several
> references to this panic over the past three years, but it seems its never
> been taken to root cause.
>
> The box crashes because the cr_uidinfo pointer in the so_cred structure is
> null. However, on closer inspection the so_cred structure is corrupted
> (cr_ref=3279453304 for example), so I'm guessing it has already been freed.
> Looking closer at the socket, I see that the SS_NOFDREF flag is set, which
> supports my theory. The tcpcb is in the CLOSED state, and has the SENTFIN
> flag set.
About how many concurrent connections are you pushing this machine to?
There's an unfortunate problem with uidinfo in 4.x:
struct uidinfo {
LIST_ENTRY(uidinfo) ui_hash;
rlim_t ui_sbsize; /* socket buffer space consumed */
long ui_proccnt; /* number of processes */
uid_t ui_uid; /* uid */
u_short ui_ref; /* reference count */
};
It doesn't look like we have any seatbelts preventing ui_ref from
overflowing, thus causing an early free on the way back down, thereby
making all the other references to the structure junk. Can you try going
into kern_resource.c, finding the function uifind, and changing:
if (uip == NULL)
uip = uicreate(uid);
uip->ui_ref++;
return (uip);
to
if (uip == NULL)
uip = uicreate(uid);
uip->ui_ref++;
if (uip->ui_ref == 0)
panic("ui_ref overflowed");
return (uip);
That would confirm that it is the problem you're running into. If that is
the case, please tell us so that we can transition to the political side
of the problem. :)
Mike "Silby" Silbersack
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030801214411.A2165>