From owner-freebsd-net@freebsd.org Sat Mar 31 13:46:07 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5B7D0F5AB1E for ; Sat, 31 Mar 2018 13:46:07 +0000 (UTC) (envelope-from reshadpatuck1@gmail.com) Received: from mail-pf0-x229.google.com (mail-pf0-x229.google.com [IPv6:2607:f8b0:400e:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D41E18464D for ; Sat, 31 Mar 2018 13:46:06 +0000 (UTC) (envelope-from reshadpatuck1@gmail.com) Received: by mail-pf0-x229.google.com with SMTP id y69so7119625pfb.5 for ; Sat, 31 Mar 2018 06:46:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:user-agent:mime-version:content-transfer-encoding:subject:to :from:message-id; bh=pM+dvDc0BD6QpcHcFbsHMg+5FE+N/O9FtzIp7GNUqy0=; b=bX9shs5hPuM9SgCDAhudoUhf8UrYfY6h/QkcsvzN02EIXu6Xa1x0rdT+lr++iPNHWK 2N9wQG5vJl/uTSWbvutRGdPRs3pERhIhCayvQwBzrwH2uur+B96mycLuXxJUpT2tobE1 rFh+il6KnkYp8+rAwxLmFipk9IsRf0O0TZdvwaok1l+aXaeybtaPxjI2pGyVy4WdDUaX zwD08P0YntstxGVeW4Ek3YcOk1+/11mKQwlvdM7586kFGnfbe/mKfglCHBBr/EwB8Uai 5MaKu8o4KeN/h3f4tUGW/AD5Mp3YdxbtaHn389s89umJqu26RsjVevHGaC7xC3op2UMs zuMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:user-agent:mime-version :content-transfer-encoding:subject:to:from:message-id; bh=pM+dvDc0BD6QpcHcFbsHMg+5FE+N/O9FtzIp7GNUqy0=; b=rhIzyLX88SHFo4Z89oO4Wi9D9GULtUDiADLDNBK2uIyzx75Jlmv9f353BOkMHeG+/V FuGzXVwbWs5zRJ/cx2sljvUMtiHvSDDK3KJKwgRthYidqrqfzU/kOEF/3KPksLZd4bHV 1ul37SHDzi2X9KyaxfINxXft4B/ioXPIEkqVjDmrtrhU/Z5TCXwb7YUxpr7Efo/bzOwJ rETCJR/g/yH5LZmFXtD2NqGKa2lKHIdMpHdd8jeV4CM9AfwHeOtuzafaeazqOuP30PlO ctmWoRR2dwBQRTjQYPesAXtfcrwLV2BWy33B1rEdsPEHpiHD3pUn+0qyuAofDFQc6znp wgLw== X-Gm-Message-State: AElRT7Eg7L1xgcRyuDjEn5VP3jO/VEbPG7ZD3WLyHkhpBMnm2CJIVZpj 8R4GNqFhjT/8BeiDh+nOYFaC8qjn3WU= X-Google-Smtp-Source: AIpwx4//D2oeLRpOnOvImQ+RtgmCw7ZwNoGl7wHH30g9DNUyhSL/tRx9jVQ6nWg+Q29uN6hj5ZWpuQ== X-Received: by 10.99.149.86 with SMTP id t22mr1951230pgn.144.1522503965565; Sat, 31 Mar 2018 06:46:05 -0700 (PDT) Received: from ?IPv6:2402:3a80:690:709c:6704:6ca5:9bd0:6a05? ([2402:3a80:690:709c:6704:6ca5:9bd0:6a05]) by smtp.gmail.com with ESMTPSA id z28sm18900879pgc.15.2018.03.31.06.46.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 31 Mar 2018 06:46:05 -0700 (PDT) Date: Sat, 31 Mar 2018 19:16:01 +0530 User-Agent: K-9 Mail for Android MIME-Version: 1.0 Subject: [netgraph] ng_bpf filter large list of IP addresses To: freebsd-net@freebsd.org From: Reshad Patuck Message-ID: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2018 13:46:07 -0000 Hey, =E2=80=8B I am trying to load a bpf filter into netgraph's ng_bpf for filtering out = thousands of separate individual IP addresses=2E =E2=80=8B I am using a simple c program to generate output that I can load into ng_b= pf using a shell=2E =E2=80=8B This works fine for upto a list of about 250 IP addresses, but as I get up= to larger IP lists I hit kern=2Eargmax (262144 bytes)=2E =E2=80=8B Whenever I try to load a larger filter into ng_bpf using a file I run into= an error saying: ``` ngctl: send msg: Invalid argument ngctl: line 1: error in file ``` I have attached debug output for the same=2E =E2=80=8B My ng_bpf node 'em1-bpf' has two hooks, 'in' and 'out'=2E =E2=80=8B I have linked to a paste with the following files: - ngtl-command -> the ngctl command which runs correctly from a command li= ne - ngctl-config -> the ngctl config file with the same filter - bpf=2Ec -> a c file that takes netgraph node details a pcap-filter and c= onverts it to a ngctl command - ngctl -> debug 5 in a ngctl shell for running the config file =E2=80=8B Please let me know what I am doing wrong with the ngctl config file and if= there is another way, maybe something more direct to load a binary bpf fil= ter directly into ng_bpf=2E =E2=80=8B As a hack around this I plan to have two ng_bpfs with multiple nodes betwe= en themselves filtering parts of the IP list=2E This works but I am not sure of the performance implications of this=2E =E2=80=8B Any suggestions/improvements general tips would be really helpful=2E =E2=80=8B Link to files: https://paste=2Eee/p/BHOoG =E2=80=8B Thanks and best regards, =E2=80=8B Reshad From owner-freebsd-net@freebsd.org Sat Mar 31 14:12:38 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F9DCF63D47 for ; Sat, 31 Mar 2018 14:12:38 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DF3DF85767 for ; Sat, 31 Mar 2018 14:12:37 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w2VECMDf089249 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 31 Mar 2018 16:12:23 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: reshadpatuck1@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w2VECHh4047808 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 31 Mar 2018 21:12:18 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: [netgraph] ng_bpf filter large list of IP addresses To: Reshad Patuck , freebsd-net@freebsd.org References: From: Eugene Grosbein Message-ID: <5ABF973D.5070009@grosbein.net> Date: Sat, 31 Mar 2018 21:12:13 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2018 14:12:38 -0000 31.03.2018 20:46, Reshad Patuck wrote: [skip] > Please let me know what I am doing wrong with the ngctl config file and if there is another way, > maybe something more direct to load a binary bpf filter directly into ng_bpf. [skip] Please read ngctl(8) manual page carefully. There are other ways. First, you may move all arguments to ngctl from command line to a file and run ngctl -f filename. Second, as for many other utilities, you can use dash (-) instead of filename to make ngctl read its arguments from standard input, e.g. this is the same as "ngctl ls": # echo ls | ngctl -f - There are 9 total nodes: Name: em0 Type: ether ID: 00000001 Num hooks: 0 Then, for shell script, you can use << such as: #!/bin/sh ngctl -f - << EOF msg em1-bpf: setprogram $program EOF All these methods impose no limits on size of such control messages. However, there is loader tunnable net.graph.maxdgram that imposes another limit on size of binary representation of control message that ngctl passes to a kernel and you may need to increase it at some point. I increase it upto 8 megabytes for my purposes.