From nobody Mon Nov 1 20:30:28 2021 X-Original-To: pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 44D42182CAD9 for ; Mon, 1 Nov 2021 20:44:39 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-oln040092072078.outbound.protection.outlook.com [40.92.72.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HjlQG1FVsz4hwS for ; Mon, 1 Nov 2021 20:44:38 +0000 (UTC) (envelope-from kisscoolandthegangbang@hotmail.fr) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GOjYp7cMlzFCVlgKUVmG365HvdqDsPy8V/KEoOussm+tBzK/POSHBloyzs8QZFDS8FE1Tt0R9hEl7+PWwrkXLvjMX/mPEyGvIthZuKF4oLsCE8FFll/v4t4CuPlQZFkRaAGlOfeOXpk6cviQjREZZn8siv0+nMl0wntUjDLOnJgqukoSHR9ICEAkgZrkyc/OcdGqvlqQ2KqYaQ8y51CqwA/wCcimNhQvJ+wgdWeaSRXS9LZ6jTkKCdChkroHze2ZCCPrMNWyLlY9JWect0vAzTNaVwcu1kTTX07kySIKHhQh476ioVwJ2I4JuwzM2w0/aW8HkdyP0lg8zROocEVDxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cTjCRmcU1S4Ew+NGNdXTQwtapMMdQAueprtXutBs7sU=; b=Ac1cQlbuJekwa5rx+hao78mnC713MvRBUDCThDI68hv1GdbHTpdSjJRZ78kkdaPWBNZN/cWKVZNZFbXBVentdEn3IyHZFrSTu1RIrLJ47Wj2KzIJ+h+EVqDikiN3qCeXVf3giuoye+talDzZiu6aj4UF2gCH/nP65Sjbk+0mE6x5OIXJ6DabAvKfOb0ST/tng8O5x89l1xSJgkZx2WL7jUHHQ+oLAn7mtsKQY11U+g+8cE3GLq8EBqmzHP/Y41Q89ia0b39Qk2vtRaC1rrRdd8tJVlgC4qk+xd/euBBQRJpNIkPHvesWUqMEPhw1fxavN+YqHlT+3oO7qtwJSq4+IQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Received: from AM9PR07MB7956.eurprd07.prod.outlook.com (2603:10a6:20b:30d::20) by AM0PR07MB3955.eurprd07.prod.outlook.com (2603:10a6:208:46::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.4; Mon, 1 Nov 2021 20:28:39 +0000 Received: from AM9PR07MB7956.eurprd07.prod.outlook.com ([fe80::cde2:f4ca:8325:6a10]) by AM9PR07MB7956.eurprd07.prod.outlook.com ([fe80::cde2:f4ca:8325:6a10%7]) with mapi id 15.20.4669.008; Mon, 1 Nov 2021 20:28:39 +0000 Date: Mon, 1 Nov 2021 21:30:28 +0100 From: kaycee gb To: pf@freebsd.org Subject: Re: pf on a bhyve host Message-ID: In-Reply-To: References: X-Mailer: Claws Mail 3.17.6 (GTK+ 2.24.31; x86_64-slackware-linux-gnu) Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TMN: [9rg296iOlALr4aUTtzLXuIYZW889JTtH] X-ClientProxiedBy: AM3PR05CA0146.eurprd05.prod.outlook.com (2603:10a6:207:3::24) To AM9PR07MB7956.eurprd07.prod.outlook.com (2603:10a6:20b:30d::20) X-Microsoft-Original-Message-ID: <20211101213028.2629be9a@slackstro.home.lan> List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mail.lacabanedeladmin.trickip.net (93.1.37.139) by AM3PR05CA0146.eurprd05.prod.outlook.com (2603:10a6:207:3::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14 via Frontend Transport; Mon, 1 Nov 2021 20:28:38 +0000 Received: from slackstro.home.lan ([172.16.93.19]) (authenticated bits=0) by mail.lacabanedeladmin.trickip.net (8.15.2/8.15.2) with ESMTPSA id 1A1KSZQt003020 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 1 Nov 2021 21:28:36 +0100 (CET) (envelope-from kisscoolandthegangbang@hotmail.fr) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1d2a0dad-03a0-4857-4974-08d99d763035 X-MS-TrafficTypeDiagnostic: AM0PR07MB3955: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +MSvwnOULqWcxNmW783Nai2Tt0dK2Ny5x2B7uqWr/8ysu5WsLdMpLRMAnx45oyhzaR6Gum3NOgCLGRT0V/LreRJd5CJtfzb7WRDBhoMPwH9a0i0r53UZ9/6kAGYWW1vfQP0iky+Sa4s+av7OuGJLDWDqtZ/wnsX5W3OQedBhSpXlOQItYv6hSFF4d1+0pvCtVU5zFYh3Fm9cFh3QMVGIKaKoMulyVi1TGSHJCEzM5auQBj6nxrn8yvnLdHVD8VcyFGXUBTU+rjrZfBD5eUAxL/5ii5GnpoN1Q+qGxHzEAqotSZNKB4Nq7IOtt5snZ/SA/k396WEDKoprJpsukXSDwa+RgA+uDWPB4DBdGsG6ilZdX78Y+2NC88PWq10ynR4Eil9O3eOzrRvk3UpfKduUia0jv/5YH8oEmawrZquKI51DzE5sayHhprjnfqOkVgwO6FpmgckvrJ72Z2avBZdkHI754R1wqrVAy5wDfsxprSv5HgJP/YwqT4uKU8tubxnf3U0QW+eURNuUTZQpglxZHTWc1b3Trs0wJWD72wkmH5E= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 6i1uCJDmi4nAWbVVpj9mjR/Lgsb725DARr50ZH7M5XkFhrl7tJUKb/Rkjs+CFCphWceiRlhMyqmvSQR95Ou5IctgSSiTpZyWOJlkwTlmD/lfInN/GhYUP2ICST2kTXz0vJVn822qon8PSIZm+3mjhA/QkEtoaWGdHIHqWklVSC5ehXGTNRHHe7ta9cA1UW+7EQOMoC5jvNAnmPLZBmyuKAvHay1RRtEjh4MK1JN2DsTbNpD2Rj16r3n1u3AoaAhij45T1AZfBHj5maLotQvTAwdAWaL0sjWGPh78Epo1sugS0xdJypv8dZeDPugPMadsZri14TcoXXQi1kzCPKWuekh27cdbT0yXO0YqPrOD5GqwetiCxqmwi7jX0lflrgC+7tq+yElQw/qArOfz721vEXFAE2nEnDG0adv4vX1KyLK1bQUmPkVxCx4hZoxEROzIUTWRrnFs/8f6NReal5Og2lsW+syW95fGjpy4DFIZWhyMJGz1mgOdigDoA42ln/xBGasA1J4T+M1Or4f87p06CMHvYsQN/Ws8G3AUkDGyU3G2sS19hincShY8j7ys3MlGd9DTg0keCM7xQ1k7+3VMEfBVaCfaSu5qG373wjDijTMFrWT7lTIWrZK+jBv1glCFA6/c0p1SOX7Fjcx7Q3ijh62mNJo6sN1g2++yVHFQvaZ49FtasRvwLjGEJQ+b28Y/ X-OriginatorOrg: sct-15-20-3174-8-msonline-outlook-466f4.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: 1d2a0dad-03a0-4857-4974-08d99d763035 X-MS-Exchange-CrossTenant-AuthSource: AM9PR07MB7956.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2021 20:28:39.4188 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR07MB3955 X-Rspamd-Queue-Id: 4HjlQG1FVsz4hwS X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=hotmail.fr; spf=pass (mx1.freebsd.org: domain of kisscoolandthegangbang@hotmail.fr designates 40.92.72.78 as permitted sender) smtp.mailfrom=kisscoolandthegangbang@hotmail.fr X-Spamd-Result: default: False [-2.86 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_COUNT_FIVE(0.00)[5]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.72.78:from]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[hotmail.fr]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[pf@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[93.1.37.139:received]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/15]; DMARC_POLICY_ALLOW(-0.50)[hotmail.fr,none]; RCVD_IN_DNSWL_NONE(0.00)[40.92.72.78:from]; NEURAL_SPAM_SHORT(0.94)[0.942]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[hotmail.fr]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N Hi,=20 Le Sun, 31 Oct 2021 12:48:48 +0000, tech-lists a =C3=A9crit : > Hello pf@ >=20 > (the context is a 12.2-p10 host and various bhyve guests) >=20 > What's the best way to have pf protect the host (on igb0) but=20 > leave the traffic for the tap devices unexamined? It seems, for example >=20 > set skip on $tap_ifs >=20 > where $tap_ifs is a macro containing four tap devices, doesn't do what's= =20 > needed.=20 Do the "set skip" option expands correctly (one tap if per line) ?=20 > In this context, igb0 is bridged with the tap devices. Traffic=20 > still gets hit by pf block rules on the host despite being for the vm > behind the tap device(s). Do you filter on your bridge if or igb0 ?=20 >=20 > Is a different approach needed?=20 Based on your context, I would do same as you.=20 Do you have a catch (block) all rule at then end ? Alternatively, I would try to have rules specifically for each interfaces y= ou have except for TAP IFs (and probably bridges). Some sort of "set skip" emulation. As for the rest, I can't answer. =20 > Do I need to use vlans? The bhyhe guests > need to have real routable IPs and both the host and the guests are on > the same subnet. The desired outcome was previously achieved with a > hardware firewall in front of the bhyve host. I'm not sure if this is > possible with freebsd's pf. Maybe it is with openbsd's? I understand > that we have pci passthru with bhyve+openbsd guests now. >=20 > thanks, K. From nobody Thu Nov 4 06:26:15 2021 X-Original-To: freebsd-pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 84791182907D for ; Thu, 4 Nov 2021 06:26:27 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: from mail-ua1-x933.google.com (mail-ua1-x933.google.com [IPv6:2607:f8b0:4864:20::933]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HlDDf2rkMz3n1q for ; Thu, 4 Nov 2021 06:26:26 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: by mail-ua1-x933.google.com with SMTP id i6so8918905uae.6 for ; Wed, 03 Nov 2021 23:26:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=XurHzoOq/9zKnsgqhkrGRtumu7Pj7UoVL6sxG2UWLaA=; b=UQuXNs+OZ0zgX/dQg0iMmzgpfOyCzsukgzDcoDZ2sMnb2cTdQN5MY56loquQXqKlKX V/PZ2c6Q8QWDnhcltJlV2PsV5LV5/8FHnyJnyileqFi6XGFd7p0RGTTc7FG0nUOiArF0 1HngHUlGMwtlpcdaTooxCnFugD2XpnZnmM63Atl48o9o2zEBwvzdm0qe70l+iixdpUVb 3p9LTsGsgcWXsFtVvIBG4JpKNrCz/Ev+2KU/XaOaGKorcMucx8YzK5ZWtKUBGXkGxAS0 cdnN8sMULQr55HCSQ5EQcJI/EHd70lZ4SVnLHgDcXB3it9GIKBMrIYpCQd5CMUJHiYsA T1/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=XurHzoOq/9zKnsgqhkrGRtumu7Pj7UoVL6sxG2UWLaA=; b=mHCGjFrMsILv83/zwMcJE7O+zH6lr7eHF6VOKtvrGZuEvpeFbS/WjxhhwAOgILrUSb /AapayaoeK8ok18+aPDkN6GIDyws9GBLmao+cRM8UTeVLDinO4NRAXl5rRcvIcnHBzYD ZgmispZrf7VoGUG09At4NWVa36wbc8RXCgufgJAmSYjo/Ikv0WI06zWUtbAC/bJBLwPA l7hYBJZkd8A5qXjyG0uOB4Xl07mF/HsfpVbTzTQ+pF0UgdbK7KHacPgCc5gV8hSY4bhl ZQTevsiDn4MH2GvsGHy4LuRBKK1PgXSSTDas+jejsGk/v86IRHlegtdrvGWhJHbyqyT6 6TNQ== X-Gm-Message-State: AOAM532ZJ/of0cViS3jAS0bV8bcouHxiv0SPmJvk5BAVHYY7JBnp4Nni q8rzVx3gAllOmzc3tfXfW7HbQvVDmux8ayAaTmtbQQs7slk= X-Google-Smtp-Source: ABdhPJwujJJ1QkVrmva8HqYWTyOCg46PQpb/eav19mH8xvLIpmB2ut3/lh3fOfCyPEP5gLGLUQfM4PN8cPLYJwqUAB0= X-Received: by 2002:a05:6102:5f2:: with SMTP id w18mr32577489vsf.6.1636007185727; Wed, 03 Nov 2021 23:26:25 -0700 (PDT) List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Date: Thu, 4 Nov 2021 09:26:15 +0300 Message-ID: Subject: matching receive interface and xmit interface in single rule To: freebsd-pf@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4HlDDf2rkMz3n1q X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=UQuXNs+O; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ozkankirik@gmail.com designates 2607:f8b0:4864:20::933 as permitted sender) smtp.mailfrom=ozkankirik@gmail.com X-Spamd-Result: default: False [-1.90 / 15.00]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pf@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_RHS_MATCH_FROMTLD(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::933:from]; NEURAL_HAM_SHORT(-0.90)[-0.898]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Hi, I'm looking for a solution to match a traffic received on igb0 and xmit on igb1. According to man page, ipfw(8) supports this syntax: ipfw add deny ip from any to any out recv ed0 xmit ed1 The recv interface can be tested on either incoming or outgoing packets, while the xmit interface can only be tested on outgoing packets. So out is required (and in is invalid) whenever xmit is used. I used an workaround for this requirement: - pass quick in on igb0 all keep state (if-bound) tag rule1_IN_IGB0 - pass quick out on igb1 all tagged rule1_IN_IGB0 keep state (if-bound) But this syntax has disadvantages: - if tags used for NAT, one of the tags will be lost Because of pf has only single tag support. - reading and writing of rules become complicated Is it possible to add support for this feature like ipfw or alternatively is it possible to have a separate tag for nat tag? Have a nice day Regards From nobody Thu Nov 4 13:00:17 2021 X-Original-To: pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5936C183D65A for ; Thu, 4 Nov 2021 13:00:29 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HlNzJ4vZxz4mYY for ; Thu, 4 Nov 2021 13:00:28 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: by mail-ed1-x535.google.com with SMTP id g14so21028305edz.2 for ; Thu, 04 Nov 2021 06:00:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuxpowered-net.20210112.gappssmtp.com; s=20210112; h=to:references:from:subject:message-id:date:user-agent:mime-version :in-reply-to; bh=Wr3czjD8C+TqABEdjmtS88VIcs0aGhY8SkwUYq+/tr0=; b=bDagGbQVUiLv4eb84sH18r64VuXDdbUsuQPjmPhsm555DrLmgrCE8efXsBT3BhNHUd +r7h1xIFKmhjBLuRO9LdmzmUtN7XN4qTjx4xClVQybFYMNdOK87AeCNMzHW2y880my5c nqBvOz9aEAwPP6RcWlwabkdco3RFsp5Z+0T/CXKuaY/2svMzd43BYL7isoETWYckGTbH CUwGMLByWmW10ZvhzVUIcj4PaoPlhN+ovcKbf4XRLl7tycftuppzk2LSRz+0AfLd1uVj Wj4isowUSgruR21ZyR1i9hcjiP/r4DgI9mHGiXmg1z+6APRnVVyOVWH9SYP+bO99dyDU e9Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:to:references:from:subject:message-id:date :user-agent:mime-version:in-reply-to; bh=Wr3czjD8C+TqABEdjmtS88VIcs0aGhY8SkwUYq+/tr0=; b=357IbOk1irKqfFXbBCcaC+P1esSUvRMC12i/mQS/dkAhKbX/+VAqPvZ2Jfy5Sv/yOk k31xI5XuTsz2OVBJWSclI5H3580VL0dlpdpge5X9V4v1Ji0I/vAXv6Veb1qEKFb9ggoc xwbP1G4xO11Ayz8sB5PpGeT3jV3pwF5WQAL9bZOAxmS21gJJWKtAv/DcLyW8MONzaQ3J 2VeHx8krLb6ETE3sFciO/0JP8yEwyyKJY06NkGj9zZqxB8abCqNJbAsN4qMyNNdCjacG u9IbU+lPDJzmMof/9znVr9u7EiXpiFmNeCMB83M2/TJnGmb/g36QqXnnW6822BZY3ejb avDQ== X-Gm-Message-State: AOAM530ea5GoiS450sGWGlnc+sZSDV/X30DepDqIgM98lWgmoehFT9aV xj+32szXlx37xKIAtgdKk40di6QCYnA/nw== X-Google-Smtp-Source: ABdhPJwzMKprqSwGPG/gPc54ZZCo81lfV2l4/yM8AvcVluqZq0xshgmw0SoFAz136BRT9XtspK74dg== X-Received: by 2002:a17:906:b88f:: with SMTP id hb15mr14578504ejb.91.1636030826404; Thu, 04 Nov 2021 06:00:26 -0700 (PDT) Received: from [172.17.100.137] ([212.48.107.10]) by smtp.gmail.com with ESMTPSA id cw10sm2745091ejc.80.2021.11.04.06.00.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 04 Nov 2021 06:00:25 -0700 (PDT) To: pf@freebsd.org References: From: Kajetan Staszkiewicz Subject: Re: "pfctl: Cannot allocate memory" issue with a large table Message-ID: Date: Thu, 4 Nov 2021 14:00:17 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G" X-Rspamd-Queue-Id: 4HlNzJ4vZxz4mYY X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuxpowered-net.20210112.gappssmtp.com header.s=20210112 header.b=bDagGbQV; dmarc=none; spf=pass (mx1.freebsd.org: domain of vegeta@tuxpowered.net designates 2a00:1450:4864:20::535 as permitted sender) smtp.mailfrom=vegeta@tuxpowered.net X-Spamd-Result: default: False [-3.23 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tuxpowered-net.20210112.gappssmtp.com:s=20210112]; NEURAL_HAM_MEDIUM(-0.63)[-0.630]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[pf@freebsd.org]; TO_DN_NONE(0.00)[]; HAS_ATTACHMENT(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[tuxpowered.net]; NEURAL_SPAM_SHORT(1.00)[1.000]; DKIM_TRACE(0.00)[tuxpowered-net.20210112.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::535:from]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G Content-Type: multipart/mixed; boundary="TbSppi6vxVmI4ZNSm4PnZIKVC608tU5bT"; protected-headers="v1" From: Kajetan Staszkiewicz To: pf@freebsd.org Message-ID: Subject: Re: "pfctl: Cannot allocate memory" issue with a large table References: In-Reply-To: --TbSppi6vxVmI4ZNSm4PnZIKVC608tU5bT Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 23.10.21 20:42, Marcel Bischoff wrote: > I often run into the "pfctl: Cannot allocate memory" error upon > replacing the table contents. Hi, I've encountered a similar issue after upgrading to FreeBSD 13.0. I have even cherry-picked https://github.com/freebsd/freebsd-src/commit/ea21980a3facfed4c2c6fd10d0f= 16276564fb540 which has not helped. I have a theory what is the problem here, but I lack detailed knowledge to confirm it. I have multiple Load Balancers running FreeBSD 11 or 13 and some of them run with with only 6 or 8GiB of memory installed. Each one shows 1-3GiB "wired" memory, <200MiB "active" memory and "inactive" slowly occupying all available memory within weeks after boot. Once there is only a few hundred MiB free memory, I can't reload the pf ruleset anymore on FreeBSD 13. Most of memory allocations in pf happens with M_NOWAIT flag. The aforementioned patch changes IOCTLs to request memory with M_WAITOK, but this does not change memory allocated for tables themselves. My guess is that when memory is full of inactive pages, it becomes impossible to allocate more UMA objects with M_NOWAIT, as it would require first getting rid of those pages (swapping them out? freeing them?). I'm unsure if this is due to changes in pf between 11 and 13, or rather increased memory pressure from other parts of system. I've always thought that it is beneficial to keep as much buffers / caches / inactive stuff in memory for better performance, but apparently it makes allocations which can't wait fail. Or at least that's my best guess, which somebody more experienced in in-kernel memory management (as I understand this would never be an issue in userspace!) should verify. --=20 | pozdrawiam / greetings | Powered by macOS, Debian and FreeBSD | | Kajetan Staszkiewicz | www: http://vegeta.tuxpowered.net | `------------------------^--------------------------------------' --TbSppi6vxVmI4ZNSm4PnZIKVC608tU5bT-- --yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wmMEABEIACMWIQSOEQZObv2B8mf0JbnjtFCvbXs6FAUCYYPZYQUDAAAAAAAKCRDjtFCvbXs6FAJ6 AKDuvPAFsr+wtI4tUWwK0YZnmRWFIwCcCJ/5ta5gAiROyX5uRVf+CDkQqRY= =AXFW -----END PGP SIGNATURE----- --yVor8nrdqT3VF5UI5lOZKKalDo4D7QR3G-- From nobody Sun Nov 7 21:00:38 2021 X-Original-To: pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C2C201840165 for ; Sun, 7 Nov 2021 21:00:40 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HnRV03BJ3z51Tb for ; Sun, 7 Nov 2021 21:00:40 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D9D2F24CE8 for ; Sun, 7 Nov 2021 21:00:38 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1A7L0cUQ095000 for ; Sun, 7 Nov 2021 21:00:38 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1A7L0coN094999 for pf@FreeBSD.org; Sun, 7 Nov 2021 21:00:38 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202111072100.1A7L0coN094999@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: pf@FreeBSD.org Subject: Problem reports for pf@FreeBSD.org that need special attention Date: Sun, 7 Nov 2021 21:00:38 +0000 List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16363188381.4F9f0db3.93418" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: Y --16363188381.4F9f0db3.93418 Date: Sun, 7 Nov 2021 21:00:38 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 237973 | pf: implement egress keyword to simplify rules ac 1 problems total for which you should take action. --16363188381.4F9f0db3.93418--