From owner-freebsd-stable@FreeBSD.ORG Thu Jul 15 16:22:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3539A1065676 for ; Thu, 15 Jul 2010 16:22:54 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.westchester.pa.mail.comcast.net (qmta15.westchester.pa.mail.comcast.net [76.96.59.228]) by mx1.freebsd.org (Postfix) with ESMTP id D37E18FC18 for ; Thu, 15 Jul 2010 16:22:53 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta15.westchester.pa.mail.comcast.net with comcast id iBR81e0031ei1Bg5FGNt3c; Thu, 15 Jul 2010 16:22:53 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta24.westchester.pa.mail.comcast.net with comcast id iGNs1e00C3LrwQ23kGNsKQ; Thu, 15 Jul 2010 16:22:53 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 1EC0E9B425; Thu, 15 Jul 2010 09:22:51 -0700 (PDT) Date: Thu, 15 Jul 2010 09:22:51 -0700 From: Jeremy Chadwick To: Reko Turja Message-ID: <20100715162251.GA73929@icarus.home.lan> References: <4C3CC831.7040005@kaarposoft.dk> <20100713210729.GA11943@icarus.home.lan> <0228E401B70A4023A6F86A2ADAE59EF9@rivendell> <008D0251AE4F4A2DBAA1369410565B61@rivendell> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <008D0251AE4F4A2DBAA1369410565B61@rivendell> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org, Henrik /KaarPoSoft Subject: Re: openldap client GSSAPI authentication segfaults in fbsd8stablei386 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jul 2010 16:22:54 -0000 On Thu, Jul 15, 2010 at 12:14:58AM +0300, Reko Turja wrote: > >I had similar issue with 8-RELEASE and cyrus-sasl2 with cyrus-saslauthd linked against system kerberos. > > > >(uname -a xxx.xxx.xxx 8.0-RELEASE-p3 FreeBSD 8.0-RELEASE-p3 #1: > >Sat Jun 12 00:39:22 EEST 2010 > >root@xxx.xxx.xxx:/usr/obj/usr/src/sys/WWW i386) > > > >The problem manifested itself with pretty much the same backtrace > >when using cyradm tool for administering cyrus mailboxes and due > >time constraints I solved my issue by removing all the gssapi > >plugin libs from /usr/local/lib/sasl2, so my solution isn't really > >applicable in your case. > > After building perl, cyrus-sasl2 and userland/kernel with debug symbols > I was able to get the following backtrace. > > #0 free (ptr=0x280871c0) at /usr/src/lib/libc/stdlib/malloc.c:3889 > 3889 arena_dalloc(chunk->arena, chunk, ptr); > [New Thread 286ae140 (LWP 100273)] > (gdb) bt > #0 free (ptr=0x280871c0) at /usr/src/lib/libc/stdlib/malloc.c:3889 > #1 0x2899ed32 in gss_release_buffer (minor_status=0xbfbfe4b8, > buffer=0x280871cc) at /usr/src/lib/libgssapi/gss_release_buffer.c:41 > #2 0x2899e6e2 in _gss_mg_error (m=0x28a86480, maj=851968, min=2) at /usr/src/lib/libgssapi/gss_display_status.c:240 > #3 0x2899afd9 in gss_init_sec_context (minor_status=0xbfbfe5a4, initiator_cred_handle=0x0, context_handle=0x28630164, > target_name=0x2861b380, input_mech_type=0x0, req_flags=58, time_req=0, input_chan_bindings=0x0, input_token=0x0, > actual_mech_type=0x0, output_token=0xbfbfe5a8, ret_flags=0xbfbfe594, time_rec=0x0) > at /usr/src/lib/libgssapi/gss_init_sec_context.c:156 > #4 0x289936c9 in gssapi_client_mech_step (conn_context=0x28630160, params=0x28a97080, serverin=0x0, serverinlen=0, > prompt_need=0xbfbfe8b0, clientout=0xbfbfe8ac, clientoutlen=0xbfbfe8a8, oparams=0x28a95860) at gssapi.c:1418 > #5 0x285810f6 in sasl_client_step (conn=0x28a95000, serverin=0x0, > serverinlen=0, prompt_need=0xbfbfe8b0, clientout=0xbfbfe8ac, > clientoutlen=0xbfbfe8a8) at client.c:655 > #6 0x28580ef7 in sasl_client_start (conn=0x28a95000, > mechlist=0x2861b360 "GSSAPI DIGEST-MD5 CRAM-MD5 ", > prompt_need=0xbfbfe8b0, > clientout=0xbfbfe8ac, clientoutlen=0xbfbfe8a8, mech=0xbfbfe8b8) at client.c:603 > #7 0x2856a94c in imclient_authenticate () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Cyrus/IMAP/IMAP.so > #8 0x28566f5e in XS_Cyrus__IMAP__authenticate () from > /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Cyrus/IMAP/IMAP.so > #9 0x281d8f30 in Perl_pp_entersub () at pp_hot.c:2888 > #10 0x281878bc in Perl_runops_debug () at dump.c:1968 > #11 0x280d80a9 in S_run_body (oldscope=1) at perl.c:2431 > #12 0x280d7535 in perl_run (my_perl=0x28601100) at perl.c:2349 > #13 0x08048930 in main (argc=6, argv=0xbfbfec44, env=0xbfbfec60) at perlmain.c:117 > > I'm complete GDB-idjit though so any help in getting usable information > from the following trace would be appreciated - I have the dump etc. > stored away for further digging of course. The dump is only useful if it's used on that exact system it came from, so don't rebuild any of the software/etc. until we get this figured out. The code in question: src/lib/libgssapi/gss_display_status.c 186 struct mg_thread_ctx { 187 gss_OID mech; 188 OM_uint32 maj_stat; 189 OM_uint32 min_stat; 190 gss_buffer_desc maj_error; 191 gss_buffer_desc min_error; 192 }; ... 231 void 232 _gss_mg_error(struct _gss_mech_switch *m, OM_uint32 maj, OM_uint32 min) 233 { 234 OM_uint32 major_status, minor_status; 235 OM_uint32 message_content; 236 struct mg_thread_ctx *mg; 237 238 mg = &last_error_context; 239 240 gss_release_buffer(&minor_status, &mg->maj_error); 241 gss_release_buffer(&minor_status, &mg->min_error); src/lib/libgssapi/gss_release_buffer.c 34 OM_uint32 35 gss_release_buffer(OM_uint32 *minor_status, 36 gss_buffer_t buffer) 37 { 38 39 *minor_status = 0; 40 if (buffer->value) 41 free(buffer->value); src/include/gssapi/gssapi.h 106 typedef struct gss_buffer_desc_struct { 107 size_t length; 108 void *value; 109 } gss_buffer_desc, *gss_buffer_t; So I'm going to take a shot in the dark at this one, with some supporting evidence. It looks like gss_release_buffer.c:40 is improper/incomplete -- it's only checking to see if the pointer is non-NULL, and then blindly calls free(). It doesn't bother to check to see if buffer->length is non-zero. It seems gss_release_buffer() is basically mindless about what it frees anyway, which is bad design given the complexity of this thing. Well, some MIT Kerberos dudes complained about it too, and solved it by adding "magic numbers" to the structure so that you can tell what "type" of buffer it is: http://krbdev.mit.edu/rt/Ticket/Display.html?id=2145 The patch/code there is in no way shape or form compatible with what we have in our repo... but I did notice something interesting in their gss_release_buffer() routine/diff which correlates with my idea: - if ((buffer->length) && - (buffer->value)) { They're checking both buffer->length *and* buffer->value. We're not. That said, can you please execute the following in gdb and provide the output? (gdb) p/x gss_release_buffer.c::buffer->value (gdb) p/x gss_release_buffer.c::buffer->length Furthermore, relevant bug (PR 144754) indicates there's an easier way to induce this problem, so I'm going to see if I can reproduce it here locally. It's almost certainly the same problem but induced via a slightly different context. http://lists.freebsd.org/pipermail/freebsd-bugs/2010-March/038956.html I'll report back once I poke around with that. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |