From owner-freebsd-questions@freebsd.org Sun Jul 11 19:31:34 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CF9EE66D9AE for ; Sun, 11 Jul 2021 19:31:34 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [IPv6:2001:470:0:19b::b869:801b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "holgerdanske.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GNH856zWhz4sDp for ; Sun, 11 Jul 2021 19:31:33 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Sun, 11 Jul 2021 12:31:23 -0700 Subject: Re: Analyzing Log files of very large size To: freebsd-questions@freebsd.org References: From: David Christensen Message-ID: Date: Sun, 11 Jul 2021 12:31:22 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4GNH856zWhz4sDp X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of dpchrist@holgerdanske.com has no SPF policy when checking 2001:470:0:19b::b869:801b) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-0.13 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:0:19b::b869:801b:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; HAS_WP_URI(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2001:470:0:19b::b869:801b:from:127.0.2.255]; NEURAL_SPAM_MEDIUM(0.97)[0.972]; ARC_NA(0.00)[]; AUTH_NA(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_NA(0.00)[holgerdanske.com]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2021 19:31:34 -0000 On 7/11/21 5:13 AM, KK CHN wrote: > List, > > I am in a requirement to analyze large log files of sonic wall firewall > around 50 GB. for a suspect attack. > > What tools and solutions need to be deployed for handling this much large > files and pls enlighten me with your expertise and reference materials if > any. > > All are tcp / ip communications, DNS UDP transports .. On 7/11/21 5:31 AM, Korolev Sergey wrote: > Is it a plain text file? On 7/11/21 7:13 AM, KK CHN wrote: > Yes, it is. On 7/11/21 7:38 AM, Vlad Markov wrote: > I used to use split to break up large log files into manageable pieces. From there it depends on how you work. At first we used grep then we moved on to using perl regex to analyze logs. If this is personal project, I could see doing it in Perl. But, this sounds like the kind of problem that would benefit from concurrent and/or distributed programming; and Perl was not designed for such. So, you will have to work harder if you want those features. But if this project is for an employer or client, I would recommend starting with the commercial-off-the-shelf (COTS) log analysis tool made by the hardware vendor. Train up on it. Buy a support contract: https://www.sonicwall.com/wp-content/uploads/2019/01/sonicwall-analyzer.pdf David