From owner-freebsd-questions@freebsd.org Fri Nov 6 05:45:09 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D249F2DF7F3 for ; Fri, 6 Nov 2020 05:45:09 +0000 (UTC) (envelope-from plmahan@gmail.com) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CS8VX6Wxnz4dCX for ; Fri, 6 Nov 2020 05:45:08 +0000 (UTC) (envelope-from plmahan@gmail.com) Received: by mail-lf1-x130.google.com with SMTP id y184so216594lfa.12 for ; Thu, 05 Nov 2020 21:45:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=X09Y5KZ7em5CnOjQYaPGC28feNPFi0Y+Xu1DQ3Gdhlw=; b=igf4fJzDNS6QIJHY4Xu6Sp91K1Wfm7UX+jQrrfbM3o7jHxaH8zt/WdRdYW6LIlZsN1 M/l43LHoPktNAkpfw0etJ3xBSdHJIftfRk5mCyz7NRjQ7CltKemOkjr6/7YOchvGapty MRbN3HSaM/TTrxyatgC36fUW989VHRk/MX/Kbk/B2Eg+vx8QTvqemJ+OeqBHZcboZM9y nS6PhHlnlWwAFVqRTMcJ4eHrs9WKmV6HQpfmAy7PMkCo3lP8meX77bfV+pZtYuotWO2W Ha88xxqnfmX2rSOYFsb6hD/AYjEUQACPI8ve9VGrqlNfHCQkdbS9XAsbkAnMgSaJvF/i cP8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=X09Y5KZ7em5CnOjQYaPGC28feNPFi0Y+Xu1DQ3Gdhlw=; b=OVyiDu6wZvV0UvTpSKPg2ZGg6fk4sslisyhmeVwW6OOQupWhafvgVZgaLwagv7MjTb kD4iExjIYU6p+aZHAcLlk+WfnpowH64zCWouWdyW/lzntFsczsbMaLvr55UkQl/215Ed NH0VCmEoaiaAXtdg3ytPGZdasxItKm/rtt9n9ClVG2Nk9Jfo6h1ja8ZkBy/Msmu9bPDj +/xlDMjOj9bIL98R61+99tpCll3sbjcn/4cjM4rKt+sABKAPsEPaDtYaBEdPIkhKtLRS Xx84PFpzOyJJAScNcF5s7gppf9Bazn3QJtqSkX9n4FrX7crPmKaJbL5VJHcJ9H+A2aCJ 6d7Q== X-Gm-Message-State: AOAM532H+ZwwvArQNCaRCxAQxOQiXyE0KKqY7DC38UNpe4oDsdQo8ASC 339ts9N979efWxwZPKv0EFzXUH7WBHNLKmaFSXTspgMAhBtmmw== X-Google-Smtp-Source: ABdhPJyZD56ChmkquU92Ar7n6MXgPnNka7QSnSvheW7b5K9BJz3sZiHsieWNmpcxhKX4Ftd28v2QS2g/Y86kaoqVpPE= X-Received: by 2002:a05:6512:368a:: with SMTP id d10mr192611lfs.579.1604641506768; Thu, 05 Nov 2020 21:45:06 -0800 (PST) MIME-Version: 1.0 References: <0764e7ef-bd81-a6c5-47c4-7cd539a428f5@nomadlogic.org> In-Reply-To: <0764e7ef-bd81-a6c5-47c4-7cd539a428f5@nomadlogic.org> From: Patrick Mahan Date: Thu, 5 Nov 2020 21:44:55 -0800 Message-ID: Subject: Re: Helping understand cause of SIGSEGV To: Pete Wright Cc: questions list X-Rspamd-Queue-Id: 4CS8VX6Wxnz4dCX X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=igf4fJzD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of plmahan@gmail.com designates 2a00:1450:4864:20::130 as permitted sender) smtp.mailfrom=plmahan@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::130:from]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::130:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::130:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 05:45:09 -0000 On Thu, Nov 5, 2020 at 5:01 PM Pete Wright wrote: > > > On 11/5/20 4:01 PM, Patrick Mahan wrote: > > > >> | thread #1, name = 'fluent-bit', stop reason = signal SIGABRT >> * frame #0: 0x000000004087100a libc.so.7`__sys_thr_kill at thr_kill.S:4 >> frame #1: 0x00000000407e6c84 libc.so.7`__raise(s=6) at raise.c:52:10 >> frame #2: 0x000000004089a5d9 libc.so.7`abort at abort.c:67:8 >> frame #3: 0x000000000034a7a8 >> fluent-bit`flb_signal_handler(signal=11) at fluent-bit.c:418:9 >> frame #4: 0x00000000406d1c20 >> libthr.so.3`handle_signal(actp=0x00007fffdfffc600, sig=11, >> info=0x00007fffdfffc9f0, ucp=0x00007fffdfffc680) at thr_sig.c:303:3 >> frame #5: 0x00000000406d11ef libthr.so.3`thr_sighandler(sig=11, >> info=0x00007fffdfffc9f0, _ucp=0x00007fffdfffc680) at thr_sig.c:246:2 >> frame #6: 0x00007fffffffe193 >> frame #7: 0x000000000036fe0c fluent-bit`tasks_start [inlined] >> output_params_set(th=0x00000000416091c0, data=0x000000004165d980, >> bytes=128, tag="random.0", tag_len=8, i_ins=0x0000000040e58000, >> out_plugin=0x0000000040e2dfc0, out_context=0x00000000416051e0, >> config=0x0000000040e19180) at flb_output.h:429:5 >> > > I would look at what is happening here in output_params_set(). Something > is accessing out of bounds memory. > > > > thanks for your response Patrick i really appreciate it. > > So here is where output_params_set() is defined - with an interesting > comment that i haven't chased down yet: > > 521 /* Workaround for makecontext() */ > 522 output_params_set(th, > 523 buf, > 524 size, > 525 tag, > 526 tag_len, > 527 i_ins, > 528 o_ins->p, > 529 o_ins->context, > 530 config); > 531 return th; > 532 } > 533 > > and the frame from the backtrace is this for reference: > frame #8: 0x000000000036fd14 fluent-bit`tasks_start [inlined] > flb_output_thread(task=0x00000000416410a0, i_ins=0x0000000040e58000, > o_ins=0x0000000040e5b000, config=0x0000000040e19180, > buf=0x000000004165d980, size=128, tag="random.0", tag_len=8) at > flb_output.h:522 > > and then later on line 429 of flb_output.h it does this: > 428 FLB_TLS_SET(flb_libco_params, params); > 429 co_switch(th->callee); > > like i said i'm not really sure how to grok this, but it sounds like one > of the params in output_params_set isn't being set correctly. hopefully > the code snippet makes the error more obvious :) > > Okay, I don't know lldb very well. But according to the GDB to LLDB command map it uses the same commands to move between frames. So at startup you want to ensure you are in thread 1 (thread select 1). That should place you in the last frame on the stack (frame #0). You just move up the stack using the command 'up' until you are in frame #7. Once there you need to dump the contents of 'th' using the command 'p *th' or 'frame variable -T *th'. I suspect the value of th->callee is incorrect. The next frame on the stack is - frame #6: 0x00007fffffffe193 This is different from the rest of the stack addresses. So I suspect it is out of bounds. Patrick