From owner-freebsd-questions@freebsd.org Thu Apr 6 21:33:21 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5AD9D32CF6 for ; Thu, 6 Apr 2017 21:33:21 +0000 (UTC) (envelope-from karly@kipshouse.net) Received: from mgmt.ironboy.kipshouse.net (ironboy.kipshouse.net [IPv6:2001:470:835a:4242::42]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mx.kipshouse.net", Issuer "Starfield Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 82CA72FB for ; Thu, 6 Apr 2017 21:33:21 +0000 (UTC) (envelope-from karly@kipshouse.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=kipshouse.org; i=@kipshouse.org; q=dns/txt; s=kh-ss; t=1491514401; x=1523050401; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; z=Date:=20Thu,=206=20Apr=202017=2014:05:41=20-0700|From: =20Karl=20Young=20|To:=20Ernie=20Luz ar=20|Cc:=20Steve=20O'Hara-Smith=20,=20freebsd-questions@freebsd.org |Subject:=20Re:=20Is=20there=20a=20database=20built=20int o=20the=20base=20system|Message-ID:=20<20170406210540.GO2 787@mailboy.kipshouse.net>|References:=20<58E696BD.605050 3@gmail.com>=0D=0A=20<69607026-F68C-4D9D-A826-3EFE9ECE12A B@mac.com>=0D=0A=20<58E69E59.6020108@gmail.com>=0D=0A=20< 20170406210516.c63644064eb99f7b60dbd8f4@sohara.org>=0D=0A =20<58E6AFC0.2080404@gmail.com>|MIME-Version:=201.0 |In-Reply-To:=20<58E6AFC0.2080404@gmail.com>; bh=BoxnVfnLjy1aOgZBvD1vyNKp85xOMIeUkKq25JYnM2E=; b=WKLZ0TRV+6+1rL3rYgVYAZt4Sk3z3enCqN5zwwtjC/AUVYZvF902nGtC Q8Ui9id+fpxRyQYS7T5eWa8ectjfE9bXHrKjyFdWxD8v+/T+6kDjYMuRU X/skK6njaVJhkCPXNEDwL4WoEV87fBHSwrmvibJfwO+vmN7nIu9bHVIxl qJmLUY0dUhh7GSYQFqkNL3FV0uU5yIlV9ZdfAVf5E5N6vfxLQ+mefIg/F iKSVBF+Z5OFIti3AtY+O5peSCYh5p5soaplP/rNMAf6o4YKYpoWxSgWQf pyOpM0qhZdQTjiO7sjqUEvdhxRXlKiGADIwTkN2oWzyq8BmALRKXMAK3D g==; Authentication-Results: d2.ironport.kipshouse.net; dkim=none (message not signed) header.i=none; spf=None smtp.pra=karly@kipshouse.org; spf=None smtp.mailfrom=karly@kipshouse.net; spf=None smtp.helo=postmaster@mailboy.kipshouse.net Received-SPF: None (d2.ironport.kipshouse.net: no sender authenticity information available from domain of karly@kipshouse.org) identity=pra; client-ip=2001:470:835a:1010::26; receiver=d2.ironport.kipshouse.net; envelope-from="karly@kipshouse.net"; x-sender="karly@kipshouse.org"; x-conformance=sidf_compatible Received-SPF: None (d2.ironport.kipshouse.net: no sender authenticity information available from domain of karly@kipshouse.net) identity=mailfrom; client-ip=2001:470:835a:1010::26; receiver=d2.ironport.kipshouse.net; envelope-from="karly@kipshouse.net"; x-sender="karly@kipshouse.net"; x-conformance=sidf_compatible Received-SPF: None (d2.ironport.kipshouse.net: no sender authenticity information available from domain of postmaster@mailboy.kipshouse.net) identity=helo; client-ip=2001:470:835a:1010::26; receiver=d2.ironport.kipshouse.net; envelope-from="karly@kipshouse.net"; x-sender="postmaster@mailboy.kipshouse.net"; x-conformance=sidf_compatible X-SBRS: None X-MID: 242750 X-RemoteIP: 2001:470:835a:1010::26 X-RemoteHost: 2001:470:835a:1010::26, mailboypriv.kipshouse.net X-IronPort-AV: E=McAfee;i="5800,7501,8490"; a="242750" Received: from mailboypriv.kipshouse.net (HELO mailboy.kipshouse.net) ([IPv6:2001:470:835a:1010::26]) by d2.ironport.kipshouse.net with ESMTP; 06 Apr 2017 14:33:20 -0700 Received: by mailboy.kipshouse.net (Postfix, from userid 500) id 3B8CD43F00; Thu, 6 Apr 2017 14:05:41 -0700 (PDT) Date: Thu, 6 Apr 2017 14:05:41 -0700 From: Karl Young To: Ernie Luzar Cc: Steve O'Hara-Smith , freebsd-questions@freebsd.org Subject: Re: Is there a database built into the base system Message-ID: <20170406210540.GO2787@mailboy.kipshouse.net> References: <58E696BD.6050503@gmail.com> <69607026-F68C-4D9D-A826-3EFE9ECE12AB@mac.com> <58E69E59.6020108@gmail.com> <20170406210516.c63644064eb99f7b60dbd8f4@sohara.org> <58E6AFC0.2080404@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <58E6AFC0.2080404@gmail.com> X-Arbitrary-Number-Of-The-Day: 42 X-URL: http://www.kipshouse.org/karly X-Work-URL: http://www.cisco.com/ X-Disclaimer: My opinions do not necessarily represent those of my employer. User-Agent: Mutt/1.5.20 (2009-12-10) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Apr 2017 21:33:21 -0000 Ernie Luzar(luzar722@gmail.com)@2017.04.06 17:14:40 -0400: > Steve O'Hara-Smith wrote: > >On Thu, 06 Apr 2017 16:00:25 -0400 > >Ernie Luzar wrote: > > > >>Chuck Swiger wrote: > >>>On Apr 6, 2017, at 12:27 PM, Ernie Luzar wrote: > >>>>Looking for a simple database thats comes with the base system, is > >>>>there one? > >>>There's likely an old flavor of BerkeleyDB around if you just need a > >>>key-value store. > >>> > >>>If you're looking for a SQL database, start with SQLite (cf > >>>databases/sqlite3 port), and then look towards MySQL or Postgres. > >>> > >>>Regards, > >>BerkeleyDB sounds like something to look at. Did a "locate > >>Berkeley" cmd and nothing popped out. Is it called something > >>else? > > > > man db > > > > Will get you all the details. > > > >>Writing csh scripts manipulating text files containing lists of > >>IP address. Don't want any port as a required dependent. I > >>though I > > > > Hmm Berkeley DB is a C API so perhaps not so convenient in csh, > >what do you want a database to do for you ? > > > > Here is the project description. analyze the unsolicited traffic > hitting my front door to determine if the same source ip address is > cycling through a range of ip address on a hourly, daily, weekly > basis. My test box ipfilter firewall uses "keep state" on everything > that is allowed out. Nothing is allowed in. Using ipfilter ippool > in-core ip address table to capture count of times said unsolicited > inbound source ip address hits my front door. Have 3 flat text files > containing about 2000 ip address having a record size of 30 bytes. > I am afraid I may be approaching the max file size that csh can handle. > Thinking of simple db where the 3 files are indexed and can be > read/written with out sequentially process all the records. At the > proof of concept stage. > > I have programmed in pear script before where I can open a file and > process a record sequentially where only the next record is > buffered. csh does not have that kind of file handling that I know > of. > > You have any suggestions? I'd suggest, Python, Perl, TCL, Ruby, ... anything but csh (or bash) for a job like this. I know Python has BDB support: https://docs.python.org/2/library/bsddb.html I'm sure all the other languages you might want to use will have support for BDB, or sqlite. Although reading and writing a 60K csv file probably isn't that big a deal, unless you're doing it quite often. In Python you could import a CSV file into a dictionary, which makes lookup very fast. -karl