From owner-freebsd-questions@freebsd.org Sun Jul 11 14:38:42 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1025668276 for ; Sun, 11 Jul 2021 14:38:42 +0000 (UTC) (envelope-from dvoich@optonline.net) Received: from mta3.srv.hcvlny.cv.net (mta3.srv.hcvlny.cv.net [167.206.4.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mail.optonline.net", Issuer "DigiCert Global CA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GN8fB1LcKz3hWs for ; Sun, 11 Jul 2021 14:38:41 +0000 (UTC) (envelope-from dvoich@optonline.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=optonline.net; s=dkim-001; t=1626014321; bh=SAsiVYb/g+K43JRc/nGHqqE8PGuXKqkh8JdrjnMElG4=; h=Date:From:To:Subject:Message-Id; b=gwS6fn/9UXbLQZjovV8KJfY1gdasPamP+oUSEc9InJsjG064JWqOOsmRe6hdUuw/o IJ26ckz5UppOJ9hW16Jn6f8Ihqpph9PA/4VUttE9YgN55oKCaQumsfRcP7V5QU+J6C gTOtdN6bsnybwVONOTS9hpV/YpDYfHp5o2knv0MI3qaRJRKwCJ6gIeFKiB/xuHNJX1 sYveDd9iEhFEkOGPM1th80AZnhNdqgFt/9aeAnJqzOQyvoOOiAVN+FOzCY9WcEC9rj APdEzdo7NwBLSIDJ+UoJPyZMF+63Po2hrIiTmdCjFE5c/Q/yZ303IPPLufveEUjHES vhhMzI0/HG75Q== X-Content-Analysis: v=2.3 cv=c+swvS1l c=1 sm=1 tr=0 a=HwdfnXRxwAscIlw0plT8Bw==:117 a=HwdfnXRxwAscIlw0plT8Bw==:17 a=kj9zAlcOel0A:10 a=pGLkceISAAAA:8 a=1-5IeoEqAAAA:8 a=uaUzlq7pTKPnv3L8bGgA:9 a=CjuIK1q_8ugA:10 a=v_kyThoKIqzX6gtT0hLe:22 Received: from [24.185.145.223] ([24.185.145.223:38017] helo=happy.dwarf7.net) by mta3.srv.hcvlny.cv.net (envelope-from ) (ecelerity 3.6.9.48312 r(Core:3.6.9.0)) with ESMTP id 81/B7-14511-0720BE06; Sun, 11 Jul 2021 10:38:41 -0400 Date: Sun, 11 Jul 2021 10:38:39 -0400 From: Vlad Markov To: freebsd-questions@freebsd.org Subject: Re: Analyzing Log files of very large size Message-Id: <20210711103839.61dfd4baafa38984f208b707@optonline.net> In-Reply-To: References: X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4GN8fB1LcKz3hWs X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=optonline.net header.s=dkim-001 header.b=gwS6fn/9; dmarc=none; spf=pass (mx1.freebsd.org: domain of dvoich@optonline.net designates 167.206.4.198 as permitted sender) smtp.mailfrom=dvoich@optonline.net X-Spamd-Result: default: False [-4.05 / 15.00]; RWL_MAILSPIKE_GOOD(0.00)[167.206.4.198:from]; FREEMAIL_FROM(0.00)[optonline.net]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:167.206.4.192/27]; DKIM_TRACE(0.00)[optonline.net:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[24.185.145.223:received]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[optonline.net]; ASN(0.00)[asn:6128, ipnet:167.206.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[167.206.4.198:from]; R_DKIM_ALLOW(-0.20)[optonline.net:s=dkim-001]; RCVD_IN_DNSWL_LOW(-0.10)[167.206.4.198:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[optonline.net]; NEURAL_HAM_MEDIUM(-0.95)[-0.953]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[optonline.net:dkim]; SPAMHAUS_ZRD(0.00)[167.206.4.198:from:127.0.2.255]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2021 14:38:42 -0000 On Sun, 11 Jul 2021 19:43:41 +0530 KK CHN wrote: > Yes, it is. > > On Sun, Jul 11, 2021 at 6:02 PM Korolev Sergey wrote: > > > Is it a plain text file? > > > > On 11 Jul 2021, at 22:13, KK CHN wrote: > > > > List, > > > > I am in a requirement to analyze large log files of sonic wall firewall > > around 50 GB. for a suspect attack. > > > > What tools and solutions need to be deployed for handling this much large > > files and pls enlighten me with your expertise and reference materials if > > any. > > > > All are tcp / ip communications, DNS UDP transports .. > > > > Regards, > > Kris I used to use split to break up large log files into manageable pieces. From there it depends on how you work. At first we used grep then we moved on to using perl regex to analyze logs. Vlad --