From owner-freebsd-questions@freebsd.org Tue Apr 18 13:09:58 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B0C5D42F05 for ; Tue, 18 Apr 2017 13:09:58 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3171516E2 for ; Tue, 18 Apr 2017 13:09:58 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-io0-x236.google.com with SMTP id o22so68391309iod.3 for ; Tue, 18 Apr 2017 06:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=PlXao+8Y6o6DKuuUkpijeHWoV/trcfXUjY8/ErP7+8s=; b=coG/OvoR/Gllnx9KTppPlIk6JkWjMJ2uF+FYm/tVo5FfQYQatHozkHnCf1rkqMCUYw JrJZ9iMxvh2S/WV2WG3vw8bGfoCaMnd0ClZH/uKfCXOH7PVpNxoBm6Fm8a6zMFNQv9sM Xnlxoir22nXFaEs2dg/IA6FOhwSA/7QxGaqA29zg7X6PhER9gLHbTidjyWEk6G3pjdg4 Pqc9AVGRP8ktSBs91OFH7IdbqeLaDT9nhLs8lv4oklDfhO1qgLXuhd4PQZ+kY7L6NUxP 4fHByMH/9JuqMayUln1GZysRdRNm+1Pz9/Wg4vyGfa2GEdaDARXw83l9Fg2DMy5qTkws Khrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=PlXao+8Y6o6DKuuUkpijeHWoV/trcfXUjY8/ErP7+8s=; b=X9EoRxhLNm2Wp7nFG1j41e3gkWZYRYEgPiQe7wnVUsyfPZFJLqjrFpZZWTFiKKlH+r 6ZoEx35U4vvqVFsxXXAysIbrxTklRsLJ2dpT8xpaHkxczM0DmDggS4z8m3ZmqX9ffeES EWKljLaMvTmXwbPcKVjverV25sAXLY+9rLCaz00Wfo/JhDXM4z03P8UoyrmWYzQtoqad kjRESPvbpg0JO8kLtMZu5Sk6qCJkxC+N5X1n8uzylrSIHVZ0ibynPQwk/Yeb7HTxR32g BqUttbFQb0nyH8CDKE5cU9x+fiKxgUE7c65abWuwSwZQB27OTGmBz89SL8rm+rlBKjqV Janw== X-Gm-Message-State: AN3rC/4Bg7llN+qdPh0GrVWUXnpVjmDPyHJzeoyUxe9YFzV555y6Rq9h ZO3A+IKTQOWp4w== X-Received: by 10.107.8.226 with SMTP id h95mr13261137ioi.163.1492520997456; Tue, 18 Apr 2017 06:09:57 -0700 (PDT) Received: from [10.0.10.3] (cpe-74-141-88-57.neo.res.rr.com. [74.141.88.57]) by smtp.googlemail.com with ESMTPSA id v23sm5563530ioi.61.2017.04.18.06.09.56 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 18 Apr 2017 06:09:57 -0700 (PDT) Message-ID: <58F61027.3090100@gmail.com> Date: Tue, 18 Apr 2017 09:09:59 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Mike Jeays CC: "freebsd-questions@FreeBSD. ORG" Subject: Re: awk help References: <58F25A01.1060208@gmail.com> <7951DF71-5CD3-4B53-9CB4-13CAA8945983@huiekin.org> <58F4CD14.7090008@gmail.com> <58F53EEA.2030206@gmail.com> <7b381f8f-e2a5-26ea-075e-96ae35efb25d@rogers.com> In-Reply-To: <7b381f8f-e2a5-26ea-075e-96ae35efb25d@rogers.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Apr 2017 13:09:58 -0000 Mike Jeays wrote: > snip > That is an amazing difference in performance - I might have expected a > five to ten times improvement, but not 300+ times. > I don't see anything very time-consuming in the script above. Is it > possible for you to post the equivalent csh and awk scripts? Either I or > someone with more experience with csh might be able to spot the problem. Here are the 2 scripts #! /bin/sh # This script is designed to process the o/p from the 2 line hit count # Make the input file read a line at a time, not a field at a time. IFS=$'\n' set -f added_date="`date +%Y%m%d`" hits_rpt="/etc/ipf_pool_hits_rpt" rm $hits_rpt touch $hits_rpt hits_new="/etc/ipf_pool.hits.yes" rm $hits_new touch $hits_new hits_no="/etc/ipf_pool.hits.no" rm $hits_no touch $hits_no ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null for line in `cat $hits_rpt`; do # drop the first 3 rpt lines poollist_line="" poollist_line=`echo -n $line | grep poollist` [ -n "${poollist_line}" ] && continue role_line="" role_line=`echo -n $line | grep Role` [ -n "${role_line}" ] && continue nodes_line="" nodes_line=`echo -n $line | grep Nodes` [ -n "${nodes_line}" ] && continue in_line1=`echo -n $line | grep Address:` [ -n "${in_line1}" ] && save_in_line1="${in_line1}" in_line2=`echo -n $line | grep Hits` [ -n "${in_line2}" ] && save_in_line2="${in_line2}" if [ "${save_in_line1}" -a "${save_in_line2}" ]; then build_line1=${save_in_line1##*:} build_line1=${build_line1%%/*} build_line1="${build_line1};" build_line2=${save_in_line2##*Hits } # So remove everything to the right of the word Bytes. build_line2=${build_line2%%Bytes*} if [ ${build_line2} -gt 0 ]; then db_rec="$added_date ${build_line1}" echo "${db_rec}" >> $hits_new fi build_line="${build_line2} ${build_line1}" echo "${build_line}" >> $hits_no in_line1="" in_line2="" save_in_line1="" save_in_line2="" else continue fi done exit 0 #! /bin/sh hits_rpt="/etc/ipf_pool_hits" # ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null awk 'BEGIN { "date +%Y%m%d" | getline date hits_yes = "/etc/ipf_pool_awk_hits_yes" hits_no = "/etc/ipf_pool_awk_hits_no" system("rm " hits_yes) system("touch " hits_yes) system("rm " hits_no) system("touch " hits_no) } /Address/ { address = $2; sub("/32", ";", address); got_address = 1; } /Hits/ { if (got_address) { hits = $2; if (hits > 0) { print date, " ", address > hits_yes; got_address = 0; } else { print "hit no", hits, address > hits_no; got_address = 0; } } }' $hits_rpt Note; the ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null command is run in the first script and the file it creates in used as input in the second script so both scripts process the same data. This ippool command is not the source of the big time difference between the 2 scripts, where the first one takes 5 minutes and the second takes 1 second.