From owner-freebsd-questions@freebsd.org Fri Nov 6 00:01:14 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 606392D4442 for ; Fri, 6 Nov 2020 00:01:14 +0000 (UTC) (envelope-from plmahan@gmail.com) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CS0sj3cvKz3py7 for ; Fri, 6 Nov 2020 00:01:13 +0000 (UTC) (envelope-from plmahan@gmail.com) Received: by mail-lf1-x131.google.com with SMTP id l28so4846441lfp.10 for ; Thu, 05 Nov 2020 16:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IM4Gqglf51KMwv/adT20SV85FG/hn/2Z5s29y1UTFf4=; b=n8uokKvIllP4KyE8ik2wy1m/JI/dR6mJtMeG27bKGYIPZZ/9cy0VsmLkT0jn9XUL2y GLj2PkWvBoEEle/MHhmSJ2opTS9cLwVqJ8pDJGVebBN+bwhnAup5fWGk8MVuPu9C8Guf 3U/j/l/wfEmTigWnqPOpm0zt/ilFRx+vCoMd+RScOFC0VOdGu8wI8gUu12h7s9nxx+OT fmL3fHlTgO2DIC/aAlddG/mVcMXjyhJ39Gn7kwhJVfMy0jbrLJ0cPHSEUXXtskrPOUCj vIPH2yeyiNXbU3XzjymH1WyP11Nbd9J/zu3FCtQ6PZBVU0yK5DR9fV8kHYT8mk/a1bhn C2jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IM4Gqglf51KMwv/adT20SV85FG/hn/2Z5s29y1UTFf4=; b=VAUIy+uOwQCH8BuxUoEvnIIYxQ5dDUN/YRDgtGniQPaMoGrEocyu92BXwMKsTpidp3 iikirwbuMjjJBREfun0a16/YaoWXG+SS5WIGxUq1v239izhOyNUSV0p6HSNwcUjw5w/Q LYSVWRfsVdjA6DCR1gSelUlDCIlxU3Wa2+UNmyqnnQFe+fnHZ4pGC/yS28Rl0kaRHXrq FBncoOaEuBi8KXt9nelHzsMAZ2DsHDntnOfhWKh0FkhMOsmNQcr/onoijiVZLn2wRbzJ EmsstwAcokBwcqsrW4cU6XNcSy7bYFinc/mPZL26xntMNLQotWplRsQPgBaOZk0FW9d2 whCQ== X-Gm-Message-State: AOAM531ZBPl72gaXqSi+juvFqSSDb2vS80568usMMRO8DVK5nvfYvZFY Q0m4zwrHTJysgnxTuDj3cW6K2oa7odneL5RecKDVp3wt X-Google-Smtp-Source: ABdhPJw4B0Ixh4UdqUDF6+/9H/0S1B4be0siUCNb7EEbnf59lULqnOMnRNvChaJ7Ffzx5lHu3gsu3IbBZJKlrFcxp6M= X-Received: by 2002:a19:107:: with SMTP id 7mr1835986lfb.141.1604620871213; Thu, 05 Nov 2020 16:01:11 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Patrick Mahan Date: Thu, 5 Nov 2020 16:01:00 -0800 Message-ID: Subject: Re: Helping understand cause of SIGSEGV To: Pete Wright Cc: questions list X-Rspamd-Queue-Id: 4CS0sj3cvKz3py7 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=n8uokKvI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of plmahan@gmail.com designates 2a00:1450:4864:20::131 as permitted sender) smtp.mailfrom=plmahan@gmail.com X-Spamd-Result: default: False [-1.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::131:from]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; NEURAL_SPAM_SHORT(1.00)[0.999]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::131:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::131:from]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 00:01:14 -0000 On Thu, Nov 5, 2020 at 10:20 AM Pete Wright wrote: > i was hoping someone with more experience in C and threading can help me > better understand this SIGSEGV fault I'm hitting. the background is I'm > trying to get sysutils/fluent-bit working on 12.2-RELEASE as well as on > CURRENT. > > I've created the following PR and upstream github issues: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250825 > https://github.com/fluent/fluent-bit/issues/2747 > > so where i am at now is i am able to generate a useful core file when it > hits the fault and can step through it in lldb. > > here is the output from fluent-bit: > |[2020/11/04 22:18:12] [debug] [task] created task=0x416410a0 id=0 OK > [2020/11/04 22:18:12] [trace] [thread 0x416091c0] created (custom data > at 0x416091e0, size=80 > [2020/11/04 22:18:12] [engine] caught signal (SIGSEGV) > #0 0x406d11ee in thr_sighandler() at > /lib/libthr/thread/thr_sig.c:246 > Abort trap (core dumped)| > > > here's an example from one of the threads (the github link above has the > full backtrace) from lldb: > > | thread #1, name = 'fluent-bit', stop reason = signal SIGABRT > * frame #0: 0x000000004087100a libc.so.7`__sys_thr_kill at thr_kill.S:4 > frame #1: 0x00000000407e6c84 libc.so.7`__raise(s=6) at raise.c:52:10 > frame #2: 0x000000004089a5d9 libc.so.7`abort at abort.c:67:8 > frame #3: 0x000000000034a7a8 > fluent-bit`flb_signal_handler(signal=11) at fluent-bit.c:418:9 > frame #4: 0x00000000406d1c20 > libthr.so.3`handle_signal(actp=0x00007fffdfffc600, sig=11, > info=0x00007fffdfffc9f0, ucp=0x00007fffdfffc680) at thr_sig.c:303:3 > frame #5: 0x00000000406d11ef libthr.so.3`thr_sighandler(sig=11, > info=0x00007fffdfffc9f0, _ucp=0x00007fffdfffc680) at thr_sig.c:246:2 > frame #6: 0x00007fffffffe193 > frame #7: 0x000000000036fe0c fluent-bit`tasks_start [inlined] > output_params_set(th=0x00000000416091c0, data=0x000000004165d980, > bytes=128, tag="random.0", tag_len=8, i_ins=0x0000000040e58000, > out_plugin=0x0000000040e2dfc0, out_context=0x00000000416051e0, > config=0x0000000040e19180) at flb_output.h:429:5 > I would look at what is happening here in output_params_set(). Something is accessing out of bounds memory. > frame #8: 0x000000000036fd14 fluent-bit`tasks_start [inlined] > flb_output_thread(task=0x00000000416410a0, i_ins=0x0000000040e58000, > o_ins=0x0000000040e5b000, config=0x0000000040e19180, > buf=0x000000004165d980, size=128, tag="random.0", tag_len=8) at > flb_output.h:522 > frame #9: 0x000000000036fab5 > fluent-bit`tasks_start(in=0x0000000040e58000, config=0x0000000040e19180) > at flb_engine_dispatch.c:190 > frame #10: 0x000000000036f6fd fluent-bit`flb_engine_dispatch(id=0, > in=0x0000000040e58000, config=0x0000000040e19180) at > flb_engine_dispatch.c:293:5 > frame #11: 0x000000000036ce10 > fluent-bit`flb_engine_flush(config=0x0000000040e19180, > in_force=0x0000000000000000) at flb_engine.c:85:9 > frame #12: 0x000000000036d43a fluent-bit`flb_engine_start [inlined] > flb_engine_handle_event(fd=20, mask=1, config=0x0000000040e19180) at > flb_engine.c:292:13 > frame #13: 0x000000000036d3e6 > fluent-bit`flb_engine_start(config=0x0000000040e19180) at flb_engine.c:559 > frame #14: 0x00000000003564ec > fluent-bit`flb_lib_worker(data=0x0000000040e19180) at flb_lib.c:488:11 > frame #15: 0x00000000406c87db > libthr.so.3`thread_start(curthread=0x0000000040e12500) at > thr_create.c:292:16 > > Patrick