From owner-freebsd-questions@freebsd.org Wed Apr 19 15:02:16 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A72DDD45863 for ; Wed, 19 Apr 2017 15:02:16 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-qt0-x235.google.com (mail-qt0-x235.google.com [IPv6:2607:f8b0:400d:c0d::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5EE3F191F for ; Wed, 19 Apr 2017 15:02:16 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-qt0-x235.google.com with SMTP id m36so21290799qtb.0 for ; Wed, 19 Apr 2017 08:02:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=hfs/BTHfog7EdKI7AoWnbrFyeeCOreyuAwpPhVBiYn0=; b=X8F2i1izndWFZTzEjLe7yXQr+M9MqSnLu0geZDmh8M4OrwRrGheCB6+WdezUN2UT2U VaKilLw9rc0PmB9Celdbg+ErXqztkloDf1TRS6wsJ9B3WhsvufeXFWZv7ZsT2WHLR09P KDbDVS4lq2GlRaNdnM3GrHAtaAUUqVD6YD88HLI0inacOT+83RwY/VSu04j39PNGGvQx fMXQT+ytd9OR70NmythP4EJK4/hCHrlZXkUQfcHTPdUuUv5sn6EbgP/VDkTqA3muh+jr u/51kJEcMTmHj/IzEUbohwE2SldwZCV7lFMxCEwQvHfR5jU8e9Xvj272kUTQKHysbPkX vuJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=hfs/BTHfog7EdKI7AoWnbrFyeeCOreyuAwpPhVBiYn0=; b=tleYr0bCKSJJ/gnm6y/PzRjxDwNNkohCBmx++udd3CoVDBi5wsk+UeG3vYbRymGGeQ 0BCdx8GvS1KXquHefnWhB6jd7KNx62qKMxTfvMPhcPOw1WYTxkdupuOfYsQVAsPBD8ze sHqXQqQjBnBiyOvR2I3e00alKeIUos2ZUVjK5N1IJA3B70LWtgWL5uSegPQgJ5KidkGz maAWJiWxAAbuBePtP43K81MIiXPU2od7aDj7tajhdvYbIhpTB56SrhK0l4BV6Xgp4NI0 auh0PZwN8aTowupHo7SkOwvh+rYTUxa4f5oqrZFaNMKN8V04Cfp74cO3lr6ZekSZzPO6 /zJg== X-Gm-Message-State: AN3rC/62gtJxOzI0pP6y7djYGxmmqgbWZ/kvSwiY+RYqNxVYzvCDJqfH qQLHRz9YpOWbNhen X-Received: by 10.200.40.246 with SMTP id j51mr3327846qtj.58.1492614135176; Wed, 19 Apr 2017 08:02:15 -0700 (PDT) Received: from [10.0.10.3] (cpe-74-141-88-57.neo.res.rr.com. [74.141.88.57]) by smtp.googlemail.com with ESMTPSA id g66sm2017329qkb.55.2017.04.19.08.02.14 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 19 Apr 2017 08:02:14 -0700 (PDT) Message-ID: <58F77BFE.50108@gmail.com> Date: Wed, 19 Apr 2017 11:02:22 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Andreas Perstinger CC: freebsd-questions@freebsd.org Subject: Re: awk help References: <58F25A01.1060208@gmail.com> <7951DF71-5CD3-4B53-9CB4-13CAA8945983@huiekin.org> <58F4CD14.7090008@gmail.com> <58F53EEA.2030206@gmail.com> <7b381f8f-e2a5-26ea-075e-96ae35efb25d@rogers.com> <58F61027.3090100@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2017 15:02:16 -0000 Andreas Perstinger wrote: > (Sorry for messing up parts of the quoting in my former mail.) > > On 2017-04-18 19:40, Andreas Perstinger wrote: >> I think awk is the better tool for your task but you could still >> simplify your shell script a little bit: > > After hitting the send button I realized that there is a simpler > solution using a classical Unix pipe: > > #!/bin/sh > > added_date="`date +%Y%m%d`" > > hits_rpt="hits_rpt" > hits_new="hits.yes" > hits_no="hits.no" > truncate -s 0 $hits_rpt $hits_new $hits_no > > ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null > > tail -n +4 $hits_rpt | # start at 4th line > paste - - | # join two consecutive lines > sed -e 's:^ *::' -e 's:/32::' | # remove spaces at the beginning > # and "/32" suffix from IP address > cut -w -f 2,4 | # extract IP and Hits from combined line > # (at 2nd and 4th field) > while read ip hits # read IP and Hits from each line > do # and do your work > if [ "$hits" -gt 0 ]; then > echo "$added_date ${ip};" >> $hits_new > fi > echo "$hits ${ip};" >> $hits_no > done > exit 0 > > If the "ippool" uses tabs in the output just add "tr '\t' ' '" between > the "paste" and "sed" step. > > Bye, Andreas I really like this coding method. Had to change the tail command to bypass 3 lines and the cut commend was selecting the incorrect fields. The following is the script I tested against live data. #! /bin/sh date added_date="`date +%Y%m%d`" # hits_rpt="/root/hits.rpt" hits_rpt="/etc/ipf_pool_hits" hits_yes="/etc/ipf_pool_hits3_yes" hits_no="/etc/ipf_pool_hits3_no" truncate -s 0 $hits_yes $hits_no $hits_rpt ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null tail -n +3 $hits_rpt | # start at 4th line paste - - | # join two consecutive lines #sed -e 's:^ *::' -e 's:/32::' | # remove spaces at the beginning # # and "/32" suffix from IP address cut -w -f 3,5 | # extract IP and Hits from combined line # (at 3rd and 5th fields) while read ip hits # read IP and Hits from each line do # and do your work if [ "$hits" -gt 0 ]; then ip=${ip%%/*} # strip off /32 echo "$added_date ${ip};" >> $hits_yes else echo "$hits ${ip};" >> $hits_no fi done date exit 0 I ran this script using sed command and without it and in both cases it ran in one second. I think that the paste command is where all the run elapse time saving is occurring. In my original script that took 5 minutes to run I had manually coded logic to combine the 2 lines together. From this experience I learned a creative use of the truncate & tail commands. But the real eye opener was the usage of the paste command. I didn't even know that command existed. I used the date command to time the runs, problem is it only goes to a second. What is really needed here is a timer in hundredths of a second. Is there such a command that does that? Thank you Andreas for taking the time to teach me these creative usages of these standard commands.