Date: Sat, 31 Mar 2018 19:16:01 +0530 From: Reshad Patuck <reshadpatuck1@gmail.com> To: freebsd-net@freebsd.org Subject: [netgraph] ng_bpf filter large list of IP addresses Message-ID: <D4E00F05-F64D-4446-A572-E1A0E06B2C47@gmail.com>
next in thread | raw e-mail | index | archive | help
Hey, =E2=80=8B I am trying to load a bpf filter into netgraph's ng_bpf for filtering out = thousands of separate individual IP addresses=2E =E2=80=8B I am using a simple c program to generate output that I can load into ng_b= pf using a shell=2E =E2=80=8B This works fine for upto a list of about 250 IP addresses, but as I get up= to larger IP lists I hit kern=2Eargmax (262144 bytes)=2E =E2=80=8B Whenever I try to load a larger filter into ng_bpf using a file I run into= an error saying: ``` ngctl: send msg: Invalid argument ngctl: line 1: error in file ``` I have attached debug output for the same=2E =E2=80=8B My ng_bpf node 'em1-bpf' has two hooks, 'in' and 'out'=2E =E2=80=8B I have linked to a paste with the following files: - ngtl-command -> the ngctl command which runs correctly from a command li= ne - ngctl-config -> the ngctl config file with the same filter - bpf=2Ec -> a c file that takes netgraph node details a pcap-filter and c= onverts it to a ngctl command - ngctl -> debug 5 in a ngctl shell for running the config file =E2=80=8B Please let me know what I am doing wrong with the ngctl config file and if= there is another way, maybe something more direct to load a binary bpf fil= ter directly into ng_bpf=2E =E2=80=8B As a hack around this I plan to have two ng_bpfs with multiple nodes betwe= en themselves filtering parts of the IP list=2E This works but I am not sure of the performance implications of this=2E =E2=80=8B Any suggestions/improvements general tips would be really helpful=2E =E2=80=8B Link to files: https://paste=2Eee/p/BHOoG =E2=80=8B Thanks and best regards, =E2=80=8B Reshad From owner-freebsd-net@freebsd.org Sat Mar 31 14:12:38 2018 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F9DCF63D47 for <freebsd-net@mailman.ysv.freebsd.org>; Sat, 31 Mar 2018 14:12:38 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DF3DF85767 for <freebsd-net@freebsd.org>; Sat, 31 Mar 2018 14:12:37 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w2VECMDf089249 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 31 Mar 2018 16:12:23 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: reshadpatuck1@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w2VECHh4047808 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 31 Mar 2018 21:12:18 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: [netgraph] ng_bpf filter large list of IP addresses To: Reshad Patuck <reshadpatuck1@gmail.com>, freebsd-net@freebsd.org References: <D4E00F05-F64D-4446-A572-E1A0E06B2C47@gmail.com> From: Eugene Grosbein <eugen@grosbein.net> Message-ID: <5ABF973D.5070009@grosbein.net> Date: Sat, 31 Mar 2018 21:12:13 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <D4E00F05-F64D-4446-A572-E1A0E06B2C47@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 31 Mar 2018 14:12:38 -0000 31.03.2018 20:46, Reshad Patuck wrote: [skip] > Please let me know what I am doing wrong with the ngctl config file and if there is another way, > maybe something more direct to load a binary bpf filter directly into ng_bpf. [skip] Please read ngctl(8) manual page carefully. There are other ways. First, you may move all arguments to ngctl from command line to a file and run ngctl -f filename. Second, as for many other utilities, you can use dash (-) instead of filename to make ngctl read its arguments from standard input, e.g. this is the same as "ngctl ls": # echo ls | ngctl -f - There are 9 total nodes: Name: em0 Type: ether ID: 00000001 Num hooks: 0 Then, for shell script, you can use << such as: #!/bin/sh ngctl -f - << EOF msg em1-bpf: setprogram $program EOF All these methods impose no limits on size of such control messages. However, there is loader tunnable net.graph.maxdgram that imposes another limit on size of binary representation of control message that ngctl passes to a kernel and you may need to increase it at some point. I increase it upto 8 megabytes for my purposes.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D4E00F05-F64D-4446-A572-E1A0E06B2C47>