From owner-freebsd-questions@freebsd.org Sat Apr 8 17:35:12 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3698DD332E2 for ; Sat, 8 Apr 2017 17:35:12 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x242.google.com (mail-it0-x242.google.com [IPv6:2607:f8b0:4001:c0b::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E302A336 for ; Sat, 8 Apr 2017 17:35:11 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x242.google.com with SMTP id w11so1760023itb.0 for ; Sat, 08 Apr 2017 10:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=/sKxXKaDD83hKjygH8nANXd4FpxVYoIQpt/5EYAj59Q=; b=i7zxV4JFeVDka70m8fXgAYjE28T6bs3WpEBZ7hN5Vb1EOI+nTcjqalgf8tz9Qq89d2 x90v/qNGqA023kJo1XVTyCHZkZciCuYq70xMflJj8RuNQoO3X5DIFE2gabaV1l9t5Gpz tVBgT537/TZKqlllWzzUy71TIeE+fDhTqVltQphC+sA7x4DQmI7IeoEAUvAgmzJ6ZWbq q60eSd0luF7g8Psras7aNUy7Yr8xEVF52/ReRtVaspzD+VaNmfs60PLYjxcPJlNYPats kJHppv0xiSopXG0tkULo/4NJjyeN5j0gH6dSC6qV1IMohowxAm41EZJ6Lql14X/2ikEa LB/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=/sKxXKaDD83hKjygH8nANXd4FpxVYoIQpt/5EYAj59Q=; b=K/xid9oG4Kre835C62nzggvvEiu2Le/h1agfDKfzQD6+IF9VYtUHldGZUfVFRzWGSl 1m3YSwanSSAQz48/s4f7l1wwfuJAoSsU2rTEJtT+z2IxuXz9R6hwSE3+HreGJRqzlREo NzZyVg4cwwpQL5ql3plWxPdgzfO5sRCJC5/uE8rVEgE06LusUrMkxznpb8hziNCtFz6j LTG3aDEIBx3y0Zz6WhmIDKXSyAQZUU2FoxMkRiKphulDJr9XHLOQ6AtlB/coin4nC9fM +5MkWYhzhGiop+dgXaSbBlBXaxOgrHbjvT/8mrRJMwZYywJABgvzNCVG+tSMSvV2/8Tu Mq1g== X-Gm-Message-State: AN3rC/5ZhT4yKmdCpqtXWmTccVBSl2uJsPZdjSvrH98QlEFDE+F7YTP6 6lp3GU6P4tgWNQg6 X-Received: by 10.36.207.193 with SMTP id y184mr4789619itf.1.1491672911278; Sat, 08 Apr 2017 10:35:11 -0700 (PDT) Received: from [10.0.10.3] (cpe-74-141-88-57.neo.res.rr.com. [74.141.88.57]) by smtp.googlemail.com with ESMTPSA id g19sm4038405ioj.37.2017.04.08.10.35.10 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 08 Apr 2017 10:35:10 -0700 (PDT) Message-ID: <58E91F4D.90005@gmail.com> Date: Sat, 08 Apr 2017 13:35:09 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Polytropon CC: RW , freebsd-questions@freebsd.org Subject: Re: Is there a database built into the base system References: <58E696BD.6050503@gmail.com> <69607026-F68C-4D9D-A826-3EFE9ECE12AB@mac.com> <58E69E59.6020108@gmail.com> <20170406210516.c63644064eb99f7b60dbd8f4@sohara.org> <58E6AFC0.2080404@gmail.com> <20170407001101.GA5885@tau1.ceti.pl> <20170407210629.GR2787@mailboy.kipshouse.net> <58E83E19.8010709@gmail.com> <20170408145503.69ddf649@gumby.homeunix.com> <58E9171F.3060405@gmail.com> <20170408191633.70d1f303.freebsd@edvax.de> In-Reply-To: <20170408191633.70d1f303.freebsd@edvax.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Apr 2017 17:35:12 -0000 Polytropon wrote: > On Sat, 08 Apr 2017 13:00:15 -0400, Ernie Luzar wrote: >> Here is my first try at using awk to Read every record in the input >> file and drop duplicates records from output file. >> >> >> This what the data looks like. >> /etc >cat /ip.org.sorted >> 1.121.136.228; >> 1.186.172.200; >> 1.186.172.210; >> 1.186.172.218; >> 1.186.172.218; >> 1.186.172.218; >> 1.34.169.204; >> 101.109.155.81; >> 101.109.155.81; >> 101.109.155.81; >> 101.109.155.81; >> 104.121.89.129; > > Why not simply use "sort | uniq" to eliminate duplicates? > > > >> /etc >cat /root/bin/ipf.table.awk.dup >> #! /bin/sh >> >> file_in="/ip.org.sorted" >> file_out="/ip.no-dups" >> >> awk '{ in_ip = $1 }' >> END { (if in_ip = prev_ip) >> next >> else >> prev_ip > $file_out >> prev_ip = in_ip >> } $file_in >> >> When I run this script it just hangs there. I have to ctrl/c to break >> out of it. What is wrong with my awk command? > > For each line, you store the 1st field (in this case, the entire > line) in in_ip, and you overwrite (!) that variable with each new > line. At the end of the file (!!!) you make a comparison and even > request the next data line. Additionally, keep an eye on the quotes > you use: '...' will keep the $ in $file_out, that's now a variable > inside awk which is empty. The '...' close before END, so outside > of awk. Remember that awk reads from standard input, so your > redirection for the input file would need to be "< $file_in", > or useless use of cat, "cat $file_in | awk > $file_out". > > In your specific case, I'd say not that awk is the wrong tool. > If you simply want to eliminate duplicates, use the classic > UNIX approach "sort | uniq". Both tools are part of the OS. > The awk script I posted is a learning tool. I know about "sort | uniq" I though "end" was end of line not end of file. So how should that awk command look to drop dups from the out put file?