From owner-freebsd-ports@freebsd.org Thu Oct 4 10:36:06 2018 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE8A210C7A69 for ; Thu, 4 Oct 2018 10:36:05 +0000 (UTC) (envelope-from kpnemesis@gmail.com) Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 488608EAAC for ; Thu, 4 Oct 2018 10:36:05 +0000 (UTC) (envelope-from kpnemesis@gmail.com) Received: by mail-ed1-x52f.google.com with SMTP id j62-v6so8082202edd.7 for ; Thu, 04 Oct 2018 03:36:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YH+979SzMKINRiRhH4RqaYzKBLW41RhS+VkwlQTo8wE=; b=WKODJOvWMtMqV4vUW+6wnWD3PrDMhVwmrBO+eR8YMiDEVQY/3RyMOEN4m4WJJ6rfUH dksYp9or2NvWjDDWR+0rbgNKErJLS2D+zh6pB6vHrcYCugY2mHR6tamT5e5GL73hTGk7 jsGblW2LC18Huo1++QVRcoTyfV+Nc5sGNyKJ2L8zCuxBK6C4pYlemts13xvG2/V0mwr1 Ko1iEiVZf+sgeAAZ0CW+TfO8IWvovsGY3eXUhB/DfwEFIfGMuhp6JWkWy0Lq83F3vtV5 G5QIS6MZJM8v6iCQ++rnHj1Rf8rDm9VEAH4dFpQsAmXsEEMPK6IrFZVXqDDHiAwBi7L2 /LhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YH+979SzMKINRiRhH4RqaYzKBLW41RhS+VkwlQTo8wE=; b=QBhMU1fQDxsr0YJzB5BIG0avqxzY+qSCdD/CeCm9YM9hxPAmoIFJXUAEkpiBTiy+Zx DcZRF2hjc9UTvrM2iqr6/91H8jbJHv7n2gV/kxCONyc8BH163DMRqdI5b8RPWu5mzlF2 6loVhDVV+0aBPEpmhvgcLxBjrP5xVDCZGnHn/InMzgXM8kOD8fy2sqvW2JKEgH99Xhqa so/HWhXHPQHFD9BCUITZ886X0x2DTUsxgtP8Kh8dfetO9OMWjX8rOlW2lYOcf9EWYfrz b1vdNQS52hdsIaVguTBR47Tb/vZZ4hZwf1dpKJZkeIjBOxjsu9gX3bm46D4AXtKwWZhZ t8kw== X-Gm-Message-State: ABuFfogJ8MhC216DaYmo49zELxCwGvsjubgFLTA/Y/YPSwZY8eCGvrqm KP3JZ0p65Q5mDnqTH7EexKe+pBZI6QMFcwLzTyrd7A== X-Google-Smtp-Source: ACcGV61tsf0VonA1gjFPbnUTeyi0yrVZIOHmEq06ctWYlIPazV3dq5fh4OFfj4eig33SXVlDfd/uHtEH0+9BwgrUQCM= X-Received: by 2002:a50:8fa4:: with SMTP id y33-v6mr7713398edy.131.1538649364090; Thu, 04 Oct 2018 03:36:04 -0700 (PDT) MIME-Version: 1.0 References: <4e0c6da9-1942-8a64-cd26-89c7f3cfe6c0@belgacom.net> In-Reply-To: From: Kernel Panic Date: Thu, 4 Oct 2018 11:35:53 +0100 Message-ID: Subject: Re: Logstash failing to process messages To: Benny Goemans Cc: freebsd-ports@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 10:36:06 -0000 Just as an update, we upgraded the ElasticStack to 6.3.2 a couple of months ago and Logstash has not crashed since, so whatever the problem was it appears to have been fixed in the later release. On Thu, 24 May 2018 at 13:47, Kernel Panic wrote: > Thanks for getting back to me, yes I suspect it has something to do with > my filters though I've no idea which one it could be as I'm filtering on > beats and syslog inputs. As a work around I've just added a cron command to > restart Logstash every morning at 01:00, though obviously that means I'm > losing non-beat events whilst it restarts. Please let me know if upgrading > to the latest versions helps you, if it doesn't then perhaps a PR needs to > be filed. > > On 24 May 2018 at 11:25, Benny Goemans wrote: > >> I have seen the same issue. In my case however, I had about OOM caused by >> parsing long grok patterns. I didn't have these in 5.3 either so I suspect >> it's a memory leak somewhere. >> I have since upgraded everything to 6.x and am waiting to see if the same >> issue persists. >> >> Regards, >> Benny Goemans >> >> On 23-05-2018 17:23, Kernel Panic wrote: >> >>> Hello, I'll just list the versions before I start: >>> >>> FreeBSD 11.1 >>> >>> Logstash 6.23 >>> Elasticsearch 5.6.8 >>> Kibana 5.6.8 >>> >>> The issue I'm having is that after a few days Logstash will stop >>> processing >>> any messages; I'm using the same config file that I used with Logstash >>> 5.3.0 which worked without issue and was rock-solid. There's nothing in >>> the >>> Logstash log file apart from messages about a field in my Cisco logs >>> being >>> the wrong type and therefore failing to index, however this has always >>> been >>> the case. I have tried enabling the 'dead letter' feature in Logstash to >>> process these Cisco logs but that just makes Logstash even more unstable. >>> >>> The Logstash service doesn't actually crash, it just stops processing >>> messages and fails to respond to the restart command so I end up having >>> to >>> reboot the server. I should say though that Logstash continues to respond >>> the the monitor API commands. >>> >>> I have tried updating all Logstash plugins however that has not fixed the >>> issue. >>> >>> As I said, I never had any problems with Logstash 5.3.0 but the latest >>> version (and version 5.6.8) just seem to become unstable after a few >>> days. >>> >>> Any help is greatly appreciated. >>> _______________________________________________ >>> freebsd-ports@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-ports >>> To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" >>> >> >> >> >