From owner-freebsd-ipfw@FreeBSD.ORG Mon May 12 06:40:03 2008 Return-Path: Delivered-To: freebsd-ipfw@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B19BF1065679 for ; Mon, 12 May 2008 06:40:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9E7128FC1B for ; Mon, 12 May 2008 06:40:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m4C6e3OZ012662 for ; Mon, 12 May 2008 06:40:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m4C6e3Uu012661; Mon, 12 May 2008 06:40:03 GMT (envelope-from gnats) Date: Mon, 12 May 2008 06:40:03 GMT Message-Id: <200805120640.m4C6e3Uu012661@freefall.freebsd.org> To: freebsd-ipfw@FreeBSD.org From: "Joost Bekkers" Cc: Subject: Re: kern/117234: [ipfw] [patch] ipfw send_pkt() and ipfw_tick() don't seem to support IPV6 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Joost Bekkers List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 06:40:03 -0000 The following reply was made to PR kern/117234; it has been noted by GNATS. From: "Joost Bekkers" To: "Max Laier" , bug-followup@FreeBSD.org Cc: john.w.court@nokia.com Subject: Re: kern/117234: [ipfw] [patch] ipfw send_pkt() and ipfw_tick() don't seem to support IPV6 Date: Mon, 12 May 2008 08:40:04 +0200 (CEST) Hello I've just tried the patch in this PR and found it not to work (yet). The keep-alive packets that are sent for IPv6 have their tcp port octets in the wrong order. Eg. if a dynamic rule exists for a connetion to port 4000 (0x0FA0), the keepalives are sent to 40975 (0xA00F) I haven't looked into where this difference between ipv4 and ipv6 originates, but forcing the byte-swap in send_pkt() makes everything work. I'd post the change I made, but I'm fairly sertain it's The Wrong Way (tm) to fix this. Greetings Joost. From owner-freebsd-ipfw@FreeBSD.ORG Mon May 12 11:06:59 2008 Return-Path: Delivered-To: freebsd-ipfw@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4D35F106567E for ; Mon, 12 May 2008 11:06:59 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 28E3F8FC1C for ; Mon, 12 May 2008 11:06:59 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m4CB6xq9038034 for ; Mon, 12 May 2008 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m4CB6wOW038030 for freebsd-ipfw@FreeBSD.org; Mon, 12 May 2008 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 12 May 2008 11:06:58 GMT Message-Id: <200805121106.m4CB6wOW038030@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ipfw@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-ipfw@FreeBSD.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2008 11:06:59 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/51274 ipfw [ipfw] [patch] ipfw2 create dynamic rules with parent o kern/73910 ipfw [ipfw] serious bug on forwarding of packets after NAT o kern/74104 ipfw [ipfw] ipfw2/1 conflict not detected or reported, manp o kern/88659 ipfw [modules] ipfw and ip6fw do not work properly as modul o kern/93300 ipfw [ipfw] ipfw pipe lost packets o kern/95084 ipfw [ipfw] [regression] [patch] IPFW2 ignores "recv/xmit/v o kern/97504 ipfw [ipfw] IPFW Rules bug o kern/97951 ipfw [ipfw] [patch] ipfw does not tie interface details to o kern/98831 ipfw [ipfw] ipfw has UDP hickups o kern/102471 ipfw [ipfw] [patch] add tos and dscp support o kern/103454 ipfw [ipfw] [patch] [request] add a facility to modify DF b o kern/106534 ipfw [ipfw] [panic] ipfw + dummynet o kern/112708 ipfw [ipfw] ipfw is seems to be broken to limit number of c o kern/117234 ipfw [ipfw] [patch] ipfw send_pkt() and ipfw_tick() don't s o kern/118993 ipfw [ipfw] page fault - probably it's a locking problem o conf/123119 ipfw [patch] rc script for ipfw does not handle IPv6 16 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- a kern/26534 ipfw [ipfw] Add an option to ipfw to log gid/uid of who cau o kern/46159 ipfw [ipfw] [patch] [request] ipfw dynamic rules lifetime f o kern/48172 ipfw [ipfw] [patch] ipfw does not log size and flags o kern/55984 ipfw [ipfw] [patch] time based firewalling support for ipfw o kern/60719 ipfw [ipfw] Headerless fragments generate cryptic error mes o kern/69963 ipfw [ipfw] install_state warning about already existing en o kern/71366 ipfw [ipfw] "ipfw fwd" sometimes rewrites destination mac a o kern/72987 ipfw [ipfw] ipfw/dummynet pipe/queue 'queue [BYTES]KBytes ( o bin/78785 ipfw [ipfw] [patch] ipfw verbosity locks machine if /etc/rc s kern/80642 ipfw [ipfw] [patch] ipfw small patch - new RULE OPTION o kern/82724 ipfw [ipfw] [patch] [request] Add setnexthop and defaultrou o kern/86957 ipfw [ipfw] [patch] ipfw mac logging o kern/87032 ipfw [ipfw] [patch] ipfw ioctl interface implementation o kern/91847 ipfw [ipfw] ipfw with vlanX as the device o kern/103328 ipfw [ipfw] [request] sugestions about ipfw table o kern/104682 ipfw [ipfw] [patch] Some minor language consistency fixes a o bin/104921 ipfw [patch] ipfw(8) sometimes treats ipv6 input as ipv4 (a o kern/105330 ipfw [ipfw] [patch] ipfw (dummynet) does not allow to set q o kern/107305 ipfw [ipfw] ipfw fwd doesn't seem to work o kern/111713 ipfw [dummynet] [request] Too few dummynet queue slots o kern/112561 ipfw [ipfw] ipfw fwd does not work with some TCP packets p kern/113388 ipfw [ipfw][patch] Addition actions with rules within speci o docs/113803 ipfw [patch] ipfw(8) - don't get bitten by the fwd rule o bin/115172 ipfw [patch] ipfw(8) list show some rules with a wrong form p kern/115755 ipfw [ipfw][patch] unify message and add a rule number wher o kern/116009 ipfw [ipfw] [patch] Ignore errors when loading ruleset from o kern/121122 ipfw [ipfw] [patch] add support to ToS IP PRECEDENCE fields o kern/121382 ipfw [dummynet]: 6.3-RELEASE-p1 page fault in dummynet (cor s kern/121807 ipfw [request] TCP and UDP port_table in ipfw o kern/122963 ipfw [ipfw] tcpdump does not show packets redirected by 'ip 30 problems total. From owner-freebsd-ipfw@FreeBSD.ORG Wed May 14 08:24:31 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4315F106566C for ; Wed, 14 May 2008 08:24:31 +0000 (UTC) (envelope-from madan.feedback@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.183]) by mx1.freebsd.org (Postfix) with ESMTP id 1BC618FC1A for ; Wed, 14 May 2008 08:24:31 +0000 (UTC) (envelope-from madan.feedback@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so4099139wah.3 for ; Wed, 14 May 2008 01:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=QFR4IVIsEJJO9aprFKc+Ro9erQMYjpt+pGyCQLABZnk=; b=j2S/z9uRVSrbWyXSSmuaDcLkqkTg8wKQs1Q9IH1Po/wJd+eiwWjK0bKGi2HCMH89t9qDIFRU/dbincGsMRuzBrArpHPN7QVr3XjgbEYupn9UiN6D10wMjHdsUH1u3HxIuHBWQugp3RORu2LwbWeGo0nYqzK+t94bWBBeOlXuvbc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=Gp/4KgZovqwZHOkO1pHsoFr9bbEQbOdfpPyX3OLqESlYbLJ8I8Go3hjlyyC11EM+M7t4IAy5H3B/5JzDwoEa+z229QO7FRq5G2tZ4ADQgAtQoxnk+WESBiJefCGhIlO+09BRoiPamPKjQRAEXvx2ekNnpJ8gEfZKj5GANH8zJqU= Received: by 10.114.66.8 with SMTP id o8mr786234waa.135.1210752013226; Wed, 14 May 2008 01:00:13 -0700 (PDT) Received: by 10.114.161.12 with HTTP; Wed, 14 May 2008 01:00:13 -0700 (PDT) Message-ID: <3a4237470805140100k4eeecb4aja4f6449b6a93ffb6@mail.gmail.com> Date: Wed, 14 May 2008 13:30:13 +0530 From: "Madan Thapa" To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: issues : FreeBSD kernel compile for ipfw support X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 08:24:31 -0000 Hi to all, I am freebsd newbie at the moment and want your help on the following issues : 1) I compiled bsd kernel for ipfw support using the doc at http://www.cyberciti.biz/faq/howto-setup-freebsd-ipfw-firewall/ and upon reboot, the system did not allow connections from outside. I could login to the server remotely via IPMI but not via ssh, although, ipfw entries were not setup in /etc/rc.conf to start on boot. So I ended up booting into old kernel ( generic ) , instead of IPFWKERNEL Now I want to know how to change the boot loader to boot the Generic kernel on subsequent reboots. Like in linux we do it in grub.conf 2) Can you also let me know the steps to add ipfw support in kernel? 3) How do I check that ipfw support is already there? # ipfw list will do ? Thanks From owner-freebsd-ipfw@FreeBSD.ORG Wed May 14 09:02:49 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97BB41065671 for ; Wed, 14 May 2008 09:02:49 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp7.yandex.ru (smtp7.yandex.ru [213.180.200.45]) by mx1.freebsd.org (Postfix) with ESMTP id E4D4F8FC1F for ; Wed, 14 May 2008 09:02:48 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([77.72.136.145]:4329 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S738796AbYENJCk (ORCPT ); Wed, 14 May 2008 13:02:40 +0400 X-Yandex-Spam: 1 X-Yandex-Front: smtp7 X-Yandex-TimeMark: 1210755760 X-Comment: RFC 2476 MSA function at smtp7.yandex.ru logged sender identity as: bu7cher Message-ID: <482AAAAD.9090809@yandex.ru> Date: Wed, 14 May 2008 13:02:37 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Madan Thapa References: <3a4237470805140100k4eeecb4aja4f6449b6a93ffb6@mail.gmail.com> In-Reply-To: <3a4237470805140100k4eeecb4aja4f6449b6a93ffb6@mail.gmail.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-ipfw@freebsd.org Subject: Re: issues : FreeBSD kernel compile for ipfw support X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 09:02:49 -0000 Madan Thapa wrote: > 1) I compiled bsd kernel for ipfw support using the doc at > http://www.cyberciti.biz/faq/howto-setup-freebsd-ipfw-firewall/ and upon I prefer an official documentation: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-ipfw.html > reboot, the system did not allow connections from outside. Yes, the default configuration blocks any packets. > I could login to > the server remotely via IPMI but not via ssh, although, ipfw entries were > not setup in /etc/rc.conf to start on boot. You compiled ipfw(9) into kernel and it works by default. > on subsequent reboots. Like in linux we do it in grub.conf You can install grub on the FreeBSD too. > 2) Can you also let me know the steps to add ipfw support in kernel? Read the Handbook's article. -- WBR, Andrey V. Elsukov From owner-freebsd-ipfw@FreeBSD.ORG Wed May 14 13:47:06 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E09B1065682 for ; Wed, 14 May 2008 13:47:06 +0000 (UTC) (envelope-from madan.feedback@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.177]) by mx1.freebsd.org (Postfix) with ESMTP id 452298FC0A for ; Wed, 14 May 2008 13:47:06 +0000 (UTC) (envelope-from madan.feedback@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so4222155wah.3 for ; Wed, 14 May 2008 06:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=sn0bU0+nbEf7qgT1ZGqK9L11PR0MMnVlvQk1eMItCyc=; b=YflxVdY4gXRpsbWdv26Lj45vfvgkLj5Ssw6HYDQ6jBw725cuPvqzcW9oMgAgR7X2HuLfWp73wEA5RanT7tHd3bCfkSbebPqejz2h7plW/Qr02dCcO/w8U4SlloywQo6TKKsb4a9M9ojOgbU0b+MGFHa+tUKb6JjBisw0FC4zzKc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=MRDYeWZtheuqxIeSAHno5j+M2DFTbXFhS93bODaTTKoNuDCpX3l18ZYHJIQvwNhLQ+yl3GZGhBABd+RmXQ+lO/Wx1vr0sUPyY61+L8BQUiw21v35MKH1LLvoUOIETZdS+zi4p9uTQvp8rMgHp9ujCWT5OMyg+oPkKI6juquOq7g= Received: by 10.114.66.8 with SMTP id o8mr1070055waa.135.1210772825565; Wed, 14 May 2008 06:47:05 -0700 (PDT) Received: by 10.114.161.12 with HTTP; Wed, 14 May 2008 06:47:05 -0700 (PDT) Message-ID: <3a4237470805140647u4d38ebfaw797ab004afd63912@mail.gmail.com> Date: Wed, 14 May 2008 19:17:05 +0530 From: "Madan Thapa" To: freebsd-ipfw@freebsd.org In-Reply-To: <482AAAAD.9090809@yandex.ru> MIME-Version: 1.0 References: <3a4237470805140100k4eeecb4aja4f6449b6a93ffb6@mail.gmail.com> <482AAAAD.9090809@yandex.ru> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: issues : FreeBSD kernel compile for ipfw support X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2008 13:47:06 -0000 Thanks a ton Andrey > WBR, Andrey V. Elsukov > From owner-freebsd-ipfw@FreeBSD.ORG Thu May 15 04:27:43 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63BAE106564A for ; Thu, 15 May 2008 04:27:43 +0000 (UTC) (envelope-from madan.feedback@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.182]) by mx1.freebsd.org (Postfix) with ESMTP id 39E458FC1B for ; Thu, 15 May 2008 04:27:43 +0000 (UTC) (envelope-from madan.feedback@gmail.com) Received: by wa-out-1112.google.com with SMTP id j4so251412wah.3 for ; Wed, 14 May 2008 21:27:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=+GQdf7Amgw/JxT6i+BgatwXWIe5Rd8XIPVsNe15O37c=; b=Bv57oH7qK8OGXg6I5DzZ/14d9wF+XjczSRhB0/tdG+ojog0X62dLI4eMye0VcQlHyqYWX611jmBPbvbTM4XNZJUQuVpHPSV415A5RC1cb0yOE/nCzLgcvDzD5Iq7VUkzcAexpYSHWMJJVRD1HqGU83jUNAr3uaswmxgW5FnusVk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=gfzR9jOY0S3IXE2gtqyLySmDZcj2oy8qrKtwTSOZRDKdszR++aB+0jf6DYqD/WnFJZ57WlNgu6bsKJPP5jYx/ukk68CXC6OPWvK1LEnUhK7+Zw95EYZn7PM0x1MkNOEW+bqyBEXNK1ezIWiJ1WA4cd3/uqPx60LaFV1ZdrsHPb0= Received: by 10.114.178.13 with SMTP id a13mr1984409waf.182.1210825661590; Wed, 14 May 2008 21:27:41 -0700 (PDT) Received: by 10.114.161.12 with HTTP; Wed, 14 May 2008 21:27:41 -0700 (PDT) Message-ID: <3a4237470805142127p2642590j6d39376701762ea9@mail.gmail.com> Date: Thu, 15 May 2008 09:57:41 +0530 From: "Madan Thapa" To: freebsd-ipfw@freebsd.org In-Reply-To: <482B7030.6090402@elischer.org> MIME-Version: 1.0 References: <3a4237470805140100k4eeecb4aja4f6449b6a93ffb6@mail.gmail.com> <482B7030.6090402@elischer.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: issues : FreeBSD kernel compile for ipfw support X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 04:27:43 -0000 Julian, Thank you very much for your help. ################################################################ you can do it via grub if you sodesire, or you can change the file > /boot/loader.conf to select a different kernel > (google should be able to tell you the right syntax) > > look for "loader.conf kernel" > > you can disable the firewall during boot by adding: > net.ip.fw.enable=0 > in a file called /etc/sysctl.conf > then it will not be turned on until you turn it on. > I suggest that you load it with rules that allow you to get into the > machine before you turn it on however.. > ################################################################ > >> >> >> From owner-freebsd-ipfw@FreeBSD.ORG Thu May 15 09:52:40 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2916B1065670; Thu, 15 May 2008 09:52:40 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp8.yandex.ru (smtp8.yandex.ru [213.180.200.213]) by mx1.freebsd.org (Postfix) with ESMTP id 473AD8FC38; Thu, 15 May 2008 09:52:39 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([77.72.136.145]:19184 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S7455954AbYEOJwd (ORCPT + 1 other); Thu, 15 May 2008 13:52:33 +0400 X-Yandex-Spam: 1 X-Yandex-Front: smtp8 X-Yandex-TimeMark: 1210845153 X-MsgDayCount: 3 X-Comment: RFC 2476 MSA function at smtp8.yandex.ru logged sender identity as: bu7cher Message-ID: <482C07DE.3090504@yandex.ru> Date: Thu, 15 May 2008 13:52:30 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Vivek Khera References: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> In-Reply-To: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-ipfw@freebsd.org, FreeBSD Stable Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 09:52:40 -0000 Vivek Khera wrote: > I had a box run out of dynamic state space yesterday. I found I can > increase the number of dynamic rules by increasing the sysctl parameter > net.inet.ip.fw.dyn_max. I can't find, however, how this affects memory > usage on the system. Is it dyanamically allocated and de-allocated, or > is it a static memory buffer? Each dynamic rule allocated dynamically. Be careful, too many dynamic rules will work very slow. -- WBR, Andrey V. Elsukov From owner-freebsd-ipfw@FreeBSD.ORG Thu May 15 10:19:21 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5D77106564A for ; Thu, 15 May 2008 10:19:21 +0000 (UTC) (envelope-from bms@FreeBSD.org) Received: from out1.smtp.messagingengine.com (out1.smtp.messagingengine.com [66.111.4.25]) by mx1.freebsd.org (Postfix) with ESMTP id 990E68FC13 for ; Thu, 15 May 2008 10:19:21 +0000 (UTC) (envelope-from bms@FreeBSD.org) Received: from compute1.internal (compute1.internal [10.202.2.41]) by out1.messagingengine.com (Postfix) with ESMTP id 781C71095EE; Thu, 15 May 2008 06:03:55 -0400 (EDT) Received: from heartbeat2.messagingengine.com ([10.202.2.161]) by compute1.internal (MEProxy); Thu, 15 May 2008 06:03:55 -0400 X-Sasl-enc: z0HN/iHOGaYXEwZU8WdSfCOJsTWNs5VfURBdU0IO4qd8 1210845835 Received: from empiric.lon.incunabulum.net (82-35-112-254.cable.ubr07.dals.blueyonder.co.uk [82.35.112.254]) by mail.messagingengine.com (Postfix) with ESMTPSA id BA0F224E61; Thu, 15 May 2008 06:03:54 -0400 (EDT) Message-ID: <482C0A89.104@FreeBSD.org> Date: Thu, 15 May 2008 11:03:53 +0100 From: "Bruce M. Simpson" User-Agent: Thunderbird 2.0.0.14 (X11/20080514) MIME-Version: 1.0 To: "Andrey V. Elsukov" References: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> <482C07DE.3090504@yandex.ru> In-Reply-To: <482C07DE.3090504@yandex.ru> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Vivek Khera , FreeBSD Stable , freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 10:19:21 -0000 Andrey V. Elsukov wrote: > Vivek Khera wrote: >> I had a box run out of dynamic state space yesterday. I found I can >> increase the number of dynamic rules by increasing the sysctl >> parameter net.inet.ip.fw.dyn_max. I can't find, however, how this >> affects memory usage on the system. Is it dyanamically allocated and >> de-allocated, or is it a static memory buffer? > > Each dynamic rule allocated dynamically. Be careful, too many dynamic > rules will work very slow. Got any figures for this? I took a quick glance and it looks like it just uses a hash over dst/src/dport/sport. If there are a lot of raw IP or ICMP flows then that's going to result in hash collisions. It might be a good project for someone to optimize if it isn't scaling for folk. "Bloomier" filters are probably worth a look -- bloom filters are a class of probabilistic hash which may return a false positive, "bloomier" filters are a refinement which tries to limit the false positives. Having said that the default tunable of 256 state entries is probably quite low for use cases other than "home/small office NAT gateway". cheers BMS From owner-freebsd-ipfw@FreeBSD.ORG Thu May 15 16:28:52 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51CCB1065671 for ; Thu, 15 May 2008 16:28:52 +0000 (UTC) (envelope-from vivek@khera.org) Received: from yertle.kcilink.com (myrtle.kcilink.com [66.250.193.116]) by mx1.freebsd.org (Postfix) with ESMTP id 26C118FC1C for ; Thu, 15 May 2008 16:28:51 +0000 (UTC) (envelope-from vivek@khera.org) Received: from host-121.int.kcilink.com (host-121.int.kcilink.com [192.168.7.121]) by yertle.kcilink.com (Postfix) with ESMTP id BA0DB8A0AD; Thu, 15 May 2008 12:09:39 -0400 (EDT) Message-Id: <6ADAB997-FAA4-43B8-AB57-3CC4A04F3700@khera.org> From: Vivek Khera To: FreeBSD Stable In-Reply-To: <482C0A89.104@FreeBSD.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Thu, 15 May 2008 12:09:39 -0400 References: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> <482C07DE.3090504@yandex.ru> <482C0A89.104@FreeBSD.org> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 16:28:52 -0000 On May 15, 2008, at 6:03 AM, Bruce M. Simpson wrote: > Having said that the default tunable of 256 state entries is > probably quite low for use cases other than "home/small office NAT > gateway". The deafult on my systems seems to be 4096. My steady state on a pretty popular web server is about 400, on a busy inbound mail server, around 800 states. I need to account for peaks much higher, though. Luckily most of my connections are short-lived. Thanks for the answers! From owner-freebsd-ipfw@FreeBSD.ORG Thu May 15 16:38:55 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10AEA10656AA; Thu, 15 May 2008 16:38:55 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id F030E8FC1C; Thu, 15 May 2008 16:38:54 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 567891CC038; Thu, 15 May 2008 09:20:56 -0700 (PDT) Date: Thu, 15 May 2008 09:20:56 -0700 From: Jeremy Chadwick To: "Bruce M. Simpson" Message-ID: <20080515162056.GA17187@eos.sc1.parodius.com> References: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> <482C07DE.3090504@yandex.ru> <482C0A89.104@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <482C0A89.104@FreeBSD.org> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Vivek Khera , "Andrey V. Elsukov" , FreeBSD Stable , freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 16:38:55 -0000 On Thu, May 15, 2008 at 11:03:53AM +0100, Bruce M. Simpson wrote: > Andrey V. Elsukov wrote: >> Vivek Khera wrote: >>> I had a box run out of dynamic state space yesterday. I found I can >>> increase the number of dynamic rules by increasing the sysctl parameter >>> net.inet.ip.fw.dyn_max. I can't find, however, how this affects memory >>> usage on the system. Is it dyanamically allocated and de-allocated, or >>> is it a static memory buffer? >> >> Each dynamic rule allocated dynamically. Be careful, too many dynamic >> rules will work very slow. > > Got any figures for this? I took a quick glance and it looks like it just > uses a hash over dst/src/dport/sport. If there are a lot of raw IP or ICMP > flows then that's going to result in hash collisions. > > It might be a good project for someone to optimize if it isn't scaling for > folk. "Bloomier" filters are probably worth a look -- bloom filters are a > class of probabilistic hash which may return a false positive, "bloomier" > filters are a refinement which tries to limit the false positives. > > Having said that the default tunable of 256 state entries is probably quite > low for use cases other than "home/small office NAT gateway". It's far too low for home/small office. Standard Linux NAT routers, such as the Linksys WRT54G/GL, come with a default state table count of 2048, and often is increased by third-party firmwares to 8192 based on justified necessity. Search for "conntrack" below: http://www.polarcloud.com/firmware 256 can easily be exhausted by more than one user loading multiple HTTP 1.0 web pages at one time (such is the case with many users now have browsers that load 7-8 web pages into separate tabs during startup). And if that's not enough reason, consider torrents, which is quite often what results in a home or office router exhausting its state table. Bottom line: the 256 default is too low. It needs to be increased to at least 2048. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-ipfw@FreeBSD.ORG Thu May 15 17:57:01 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1FD0B1065677 for ; Thu, 15 May 2008 17:57:01 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from mail.ciam.ru (ns.ciam.ru [213.247.195.75]) by mx1.freebsd.org (Postfix) with ESMTP id D15318FC16 for ; Thu, 15 May 2008 17:57:00 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from dhcp250-210.yandex.ru ([87.250.250.210]) by mail.ciam.ru with esmtpa (Exim 4.x) id 1Jwh6A-000FaT-VU for freebsd-ipfw@freebsd.org; Thu, 15 May 2008 21:18:35 +0400 Message-ID: <482C7043.3080901@FreeBSD.org> Date: Thu, 15 May 2008 21:17:55 +0400 From: Sergey Matveychuk User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-ipfw@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Strange swap values in top(1) on 7.0-RELEASE X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 May 2008 17:57:01 -0000 I see strange top(1) output for swap: Swap: 8064K Total, 8168K Used, K Free, 101% Inuse swapinfo output looks good: Device 1K-blocks Used Avail Capacity /dev/mirror/m1b 8192 8168 24 100% -- Dixi. Sem. From owner-freebsd-ipfw@FreeBSD.ORG Fri May 16 04:20:40 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26E591065675; Fri, 16 May 2008 04:20:40 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from gaia.nimnet.asn.au (nimbin.lnk.telstra.net [139.130.45.143]) by mx1.freebsd.org (Postfix) with ESMTP id 9B4168FC19; Fri, 16 May 2008 04:20:37 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (smithi@localhost) by gaia.nimnet.asn.au (8.8.8/8.8.8R1.5) with SMTP id NAA15526; Fri, 16 May 2008 13:56:50 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 16 May 2008 13:56:49 +1000 (EST) From: Ian Smith To: Jeremy Chadwick In-Reply-To: <20080515162056.GA17187@eos.sc1.parodius.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Vivek Khera , "Andrey V. Elsukov" , "Bruce M. Simpson" , freebsd-stable@freebsd.org, freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 04:20:40 -0000 On Thu, 15 May 2008, Jeremy Chadwick wrote: > On Thu, May 15, 2008 at 11:03:53AM +0100, Bruce M. Simpson wrote: > > Andrey V. Elsukov wrote: > >> Vivek Khera wrote: > >>> I had a box run out of dynamic state space yesterday. I found I can > >>> increase the number of dynamic rules by increasing the sysctl parameter > >>> net.inet.ip.fw.dyn_max. I can't find, however, how this affects memory > >>> usage on the system. Is it dyanamically allocated and de-allocated, or > >>> is it a static memory buffer? > >> > >> Each dynamic rule allocated dynamically. Be careful, too many dynamic > >> rules will work very slow. > > > > Got any figures for this? I took a quick glance and it looks like it just > > uses a hash over dst/src/dport/sport. If there are a lot of raw IP or ICMP > > flows then that's going to result in hash collisions. > > > > It might be a good project for someone to optimize if it isn't scaling for > > folk. "Bloomier" filters are probably worth a look -- bloom filters are a > > class of probabilistic hash which may return a false positive, "bloomier" > > filters are a refinement which tries to limit the false positives. > > > > Having said that the default tunable of 256 state entries is probably quite > > low for use cases other than "home/small office NAT gateway". > > It's far too low for home/small office. Standard Linux NAT routers, > such as the Linksys WRT54G/GL, come with a default state table count of > 2048, and often is increased by third-party firmwares to 8192 based on > justified necessity. Search for "conntrack" below: > > http://www.polarcloud.com/firmware > > 256 can easily be exhausted by more than one user loading multiple HTTP > 1.0 web pages at one time (such is the case with many users now have > browsers that load 7-8 web pages into separate tabs during startup). > > And if that's not enough reason, consider torrents, which is quite often > what results in a home or office router exhausting its state table. > > Bottom line: the 256 default is too low. It needs to be increased to at > least 2048. I think there may be some confusion in terms. Looking at defaults on my older 5.5 system - sure, call it a "home/small office NAT gateway": net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_count: 212 net.inet.ip.fw.dyn_max: 4096 net.inet.ip.fw.static_count: 153 What defaults to 256 is the number of hash table buckets, not the max number of dynamic rules, here 4096 (though the 5.5 manual says 8192). On hash collisions, a linked list is used for duplicate hashes of: i = (id->dst_ip) ^ (id->src_ip) ^ (id->dst_port) ^ (id->src_port); i &= (curr_dyn_buckets - 1); So while 256 may well be too few buckets for many systems, and like Bruce I wonder about the effectiveness of the xor hash for raw IP & ICMP and wouldn't mind seeing some stats on bucket use vs linked list lengths for various workloads, it doesn't determine the max no. of dynamic rules available, which is adjustable up without any apparent static memory allocation, and is moderated by the various expiry timeout sysctls. For reference, I admin a 4.8 filtering bridge with up to 20 boxes behind it, that has only very rarely reported exceeding the max no. of dynamic rules with the (4.8) default net.inet.ip.fw.dyn_max of 1000 .. however it only keeps state for UDP connections (and yes, it only ever hits that limit on torrents or skype, which are generally admin. prohib. :) cheers, Ian (not subscribed to -ipfw) From owner-freebsd-ipfw@FreeBSD.ORG Fri May 16 04:33:18 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54CFF1065670; Fri, 16 May 2008 04:33:18 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp6.yandex.ru (smtp6.yandex.ru [213.180.200.197]) by mx1.freebsd.org (Postfix) with ESMTP id 30F468FC2E; Fri, 16 May 2008 04:33:16 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([77.72.136.145]:47811 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S5473464AbYEPEdO (ORCPT + 2 others); Fri, 16 May 2008 08:33:14 +0400 X-Yandex-Spam: 1 X-Yandex-Front: smtp6 X-Yandex-TimeMark: 1210912394 X-MsgDayCount: 5 X-Comment: RFC 2476 MSA function at smtp6.yandex.ru logged sender identity as: bu7cher Message-ID: <482D0E87.6000003@yandex.ru> Date: Fri, 16 May 2008 08:33:11 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: "Bruce M. Simpson" References: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> <482C07DE.3090504@yandex.ru> <482C0A89.104@FreeBSD.org> In-Reply-To: <482C0A89.104@FreeBSD.org> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Vadim Goncharov , Vivek Khera , FreeBSD Stable , freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 04:33:18 -0000 Bruce M. Simpson wrote: > Got any figures for this? I took a quick glance and it looks like it > just uses a hash over dst/src/dport/sport. If there are a lot of raw IP > or ICMP flows then that's going to result in hash collisions. It's my guess, i haven't any figures.. Yes, hash collisions will trigger many searching in buckets lists. And increasing only dyn_max without increasing dyn_buckets will grow collisions. > It might be a good project for someone to optimize if it isn't scaling > for folk. "Bloomier" filters are probably worth a look -- bloom filters > are a class of probabilistic hash which may return a false positive, > "bloomier" filters are a refinement which tries to limit the false > positives. There were some ideas from Vadim Goncharov about rewriting dynamic rules implementation.. -- WBR, Andrey V. Elsukov From owner-freebsd-ipfw@FreeBSD.ORG Fri May 16 14:53:08 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BBB11065672; Fri, 16 May 2008 14:53:08 +0000 (UTC) (envelope-from vivek@khera.org) Received: from yertle.kcilink.com (thingy.kcilink.com [74.92.149.59]) by mx1.freebsd.org (Postfix) with ESMTP id 404C18FC2A; Fri, 16 May 2008 14:53:08 +0000 (UTC) (envelope-from vivek@khera.org) Received: from host-121.int.kcilink.com (host-121.int.kcilink.com [192.168.7.121]) by yertle.kcilink.com (Postfix) with ESMTP id 7AB048A0AD; Fri, 16 May 2008 10:53:07 -0400 (EDT) Message-Id: From: Vivek Khera To: FreeBSD Stable In-Reply-To: <6ADAB997-FAA4-43B8-AB57-3CC4A04F3700@khera.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Fri, 16 May 2008 10:53:07 -0400 References: <04EA1C34-AB7D-4A85-8A91-DED03E987706@khera.org> <482C07DE.3090504@yandex.ru> <482C0A89.104@FreeBSD.org> <6ADAB997-FAA4-43B8-AB57-3CC4A04F3700@khera.org> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-ipfw@freebsd.org Subject: Re: how much memory does increasing max rules for IPFW take up? X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2008 14:53:08 -0000 How are the buckets used? Are they hashed per rule number or some other mechanism? Nearly all of my states are from the same rule (eg, on a mail server for the SMTP port rule). How should I scale the buckets with the max rules? The default seems to be 4096 rules and 256 buckets. Should I maintain that ratio?