Date: Sun, 11 Jul 2021 12:31:22 -0700 From: David Christensen <dpchrist@holgerdanske.com> To: freebsd-questions@freebsd.org Subject: Re: Analyzing Log files of very large size Message-ID: <e797b547-4084-351d-08a9-31784b10fecd@holgerdanske.com> In-Reply-To: <CAKgGyB_TJrLWSjcnc9491Gg0Q5CLqLdmWx2yga_Ez7-gE6YcKQ@mail.gmail.com> References: <CAKgGyB_TJrLWSjcnc9491Gg0Q5CLqLdmWx2yga_Ez7-gE6YcKQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 7/11/21 5:13 AM, KK CHN wrote: > List, > > I am in a requirement to analyze large log files of sonic wall firewall > around 50 GB. for a suspect attack. > > What tools and solutions need to be deployed for handling this much large > files and pls enlighten me with your expertise and reference materials if > any. > > All are tcp / ip communications, DNS UDP transports .. On 7/11/21 5:31 AM, Korolev Sergey wrote: > Is it a plain text file? On 7/11/21 7:13 AM, KK CHN wrote: > Yes, it is. On 7/11/21 7:38 AM, Vlad Markov wrote: > I used to use split to break up large log files into manageable pieces. From there it depends on how you work. At first we used grep then we moved on to using perl regex to analyze logs. If this is personal project, I could see doing it in Perl. But, this sounds like the kind of problem that would benefit from concurrent and/or distributed programming; and Perl was not designed for such. So, you will have to work harder if you want those features. But if this project is for an employer or client, I would recommend starting with the commercial-off-the-shelf (COTS) log analysis tool made by the hardware vendor. Train up on it. Buy a support contract: https://www.sonicwall.com/wp-content/uploads/2019/01/sonicwall-analyzer.pdf David
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e797b547-4084-351d-08a9-31784b10fecd>