From owner-freebsd-questions@freebsd.org Tue Apr 18 00:08:28 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E9A3D4290F for ; Tue, 18 Apr 2017 00:08:28 +0000 (UTC) (envelope-from mj001@rogers.com) Received: from nm28-vm9.bullet.mail.gq1.yahoo.com (nm28-vm9.bullet.mail.gq1.yahoo.com [98.136.216.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D539326 for ; Tue, 18 Apr 2017 00:08:27 +0000 (UTC) (envelope-from mj001@rogers.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rogers.com; s=s2048; t=1492473977; bh=km7hfEkzREUtO5L+KCjBNqAZmTiJimaI6jj/c6vlBz4=; h=Subject:To:References:From:Cc:Date:In-Reply-To:From:Subject; b=WyMXE+tSUGs8YsjyzHaWAaRmZjiI/roBi18ApVI4H2GzgbEbViNRq9YFWZBx7lzEyAqkuBAxwLiz72u3dpQ70QvIIusBRzXiTRFdTLs8kNYZCg/XN5GT/gtK1Zjdf/DwhWKM6g95oaVqdJzORXUw1ot9xlUcnKEdGS1J9w54lcrXWDyacVB0wWnYAv1Hww4/19fypPLK82fTPs/5sNHNywhxcTnyzD0M6r9FpeteseU2McWx+Wxiji35nbufTbsqVo57MIkfKwLwvrYTc/KYQArEZt/rqmNWiudhljm+sw3DmzyTbQfPXnwqOqiilE7qCTfUtupbn6mkhwGfF0wEJQ== Received: from [98.137.12.62] by nm28.bullet.mail.gq1.yahoo.com with NNFMP; 18 Apr 2017 00:06:17 -0000 Received: from [208.71.42.197] by tm7.bullet.mail.gq1.yahoo.com with NNFMP; 18 Apr 2017 00:06:17 -0000 Received: from [127.0.0.1] by smtp208.mail.gq1.yahoo.com with NNFMP; 18 Apr 2017 00:06:17 -0000 X-Yahoo-Newman-Id: 819310.77095.bm@smtp208.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: gBZ6GNYVM1kk9rN.m5N2ROcx1mxhCOCqJ51qI3K.Q_J8XJH gtgVQkZUhcKppmAAscXnOAshSkzkOH056sBvZvWCWx69R5NCjoVNgQxtmfYd 544HTM34Ch5rmGGaKIHO_X_27ui5nffz0IMlEJwxxKao0NTZR.zbsQsdQogK KcYm2gwejBBo1ffC6Vq0QbyUCyKW3PRdA4vmgEBgQ3vAt1HJjQeAKghZjakw JE0ZbI721ujYJzB47R0oR3eNmaOMFmjo_g9lAUMRAo.hKbG3KOJltXX9FdP1 w_ktQwBNegQymgJEFGK3oXER3qKW75d.o1pN0clE4iJxzhcP_XFrk4aWPs6j UmvRVR3fd4LgKfGdvcyfcIjsbY2o1rFy3YItVY.Tf7d_Hhe5VuqmEUQ9vOSH tf5SZ7s.5wRb5mRSexIlXzs4PN6X3T.nuYnSpMTbL1zmNuWBqBONbodu83.w qcljwUycsJBXvDi1nJw1BgpLwCxokaED6VcyokEQvsIWAMZscMePtnIUDWx5 XCG9JNMY0nvtlTC0Uq1.JjHiUGNV8.2vOxdRqozVPl6ZyZv.DgPceH8ZOLH0 - X-Yahoo-SMTP: TLQliKmswBBvVNxNe7d23d.rcRexF9Jldbuuisg- Subject: Re: awk help To: Ernie Luzar References: <58F25A01.1060208@gmail.com> <7951DF71-5CD3-4B53-9CB4-13CAA8945983@huiekin.org> <58F4CD14.7090008@gmail.com> <58F53EEA.2030206@gmail.com> From: Mike Jeays Cc: "freebsd-questions@FreeBSD. ORG" Message-ID: <7b381f8f-e2a5-26ea-075e-96ae35efb25d@rogers.com> Date: Mon, 17 Apr 2017 20:06:16 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <58F53EEA.2030206@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Apr 2017 00:08:28 -0000 On 17-04-17 06:17 PM, Ernie Luzar wrote: > Andreas Perstinger wrote: >> On 2017-04-17 16:11, Ernie Luzar wrote: >>> When I first tested /^Address/ and /^ Hits/ produced no output. I >>> changed them to /Address/ and /Hits/ and this produced output. I >>> could not find any reference to the ^ sign, so I would like to know >>> what is it suppose to do? >> >> "^" inside a regular expression is an anchor and matches the beginning >> of the line. (See "man re_format" or e.g. >> http://www.regular-expressions.info/anchors.html ). In the example >> you've posted, the lines containing "Address" and "Hits" are indented >> which means there are spaces/tabs between the beginning of the line and >> these words. Thus the patterns don't match. >> >>> I am not having success using the system commands rm & touch as shown >>> in the following example. >>> >>> awk 'BEGIN { "date +%Y%m%d" | getline date hits_yes = >>> "/etc/ipf_pool_awk_hits_yes" hits_no = "/etc/ipf_pool_awk_hits_no" rm >>> hits_yes rm hits_no "touch hits_yes" "touch hits_no" }' $hits_rpt >> >> You need to use the built-in function "system" in order to use system >> commands, e.g. >> >> system("rm " hits_yes) >> >> This concatenates the literal string "rm " with the content of the awk >> variable "hits_yes" which results in the string "rm >> /etc/ipf_pool_awk_hits_yes" and this command is then executed. >> >>> I know the date system command is working, but can't figure out how >>> to code rm & touch to get them to work. Is this even possible? >> >> The "date" command works without using the "system" function because it >> is part of the special syntax for the "getline" function. >> >> But I wonder whether you really need to use commands like "rm" and >> "touch" inside an awk script. What are you trying to accomplish? >> >> Bye, Andreas > > > This is what I am trying to accomplish. > > In general I am experimenting with ipfilter ippools {IE; in-core > tables). I used a ippool command that generates the 2 line record pair > report that I posted about in my first post. > > I have written a csh "process hits" script that takes 5+ minutes to > process that report. I have seen awk used in some public scripts but I > have never used it before. I wanted to learn awk and though it would > be a good idea to rewrite my csh "process hits" script in awk. To have > a fair comparison I needed the awk version to do the rm & touch on the > files that the csh version does. > > Well to say the least, I was shocked at the run time results. Using > the same hits.rpt file as input, the csh script took 5 minutes to > complete and the awk script took less than 1 second. They both output > the same file of ip address that have a hit count > than zero. These > two files have the same size and contain the same number of lines and > diff shows no differences between the files. > > Its obvious that awk is far superior in performance over native csh > programming. > > I have another csh script to expire records from the master file that > runs a long time. The csh script follows; > > # The following logic removes expired records > for line in `cat $temp_master_db`; do > ip=`echo -n $line | cut -w -f 2` > date=`echo -n $line | cut -w -f 1` > > if [ "$on_one" = "YES" ]; then > on_one="NO" > previous_ip="$ip" > previous_date="$date" > continue > fi > > if [ "$ip" != "$previous_ip" ]; then > > if [ $previous_date -le $expire_date ]; then > # Drop the record from the master db file as expired. > previous_ip="$ip" > previous_date="$date" > continue > else > db_rec="$previous_date $previous_ip" > echo "${db_rec}" >> $master_db_new > previous_ip="$ip" > previous_date="$date" > fi > else > # Here current ip and previous_ip are the same. > # Check if expired. > if [ $previous_date -le $expire_date ]; then > # Drop the record from the master db file as expired. > previous_ip="$ip" > previous_date="$date" > continue > fi > if [ $previous_date -le $date ]; then > # Drop the record from the master db file as expired. > previous_ip="$ip" > previous_date="$date" > continue > fi > > db_rec="$previous_date $previous_ip" > echo "${db_rec}" >> $master_db_new > previous_ip="$ip" > previous_date="$date" > > fi > done > > # At EOF, must still process previous record. > if [ $previous_date -le $expire_date ]; then > db_rec="$previous_date $previous_ip" > echo "${db_rec}" >> $master_db_new > fi > > > Is there some standard awk model to achieve this previous-save logic? > > Also can a csh $variable be used inside of an awk program? > > Thanks That is an amazing difference in performance - I might have expected a five to ten times improvement, but not 300+ times. I don't see anything very time-consuming in the script above. Is it possible for you to post the equivalent csh and awk scripts? Either I or someone with more experience with csh might be able to spot the problem. > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org"