From owner-freebsd-pf@freebsd.org Mon Feb 26 11:21:52 2018 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E3E5F3E8DD for ; Mon, 26 Feb 2018 11:21:52 +0000 (UTC) (envelope-from Joe@stream-technologies.com) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30074.outbound.protection.outlook.com [40.107.3.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9761070460 for ; Mon, 26 Feb 2018 11:21:50 +0000 (UTC) (envelope-from Joe@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=61LoG4WhMJljHIHs9j86k7cpsFB4U1pbB38smdZPnEQ=; b=rfUILOZNlDSdvi2xM3uHgss01jzYAFXA+uuVTuFZR8OSX2H9HRlZrgJQlkzPYR3TQzeuBwYhq7xtGEs8dJjYGyAp+B8uNlZIG4ZP69pMIOOVPGTDWRwyg8mq6aJ94EXNv3OnlCwADAvB1hCFCHwy0Y0NSFaT+h1GKmwTVxdUTK8= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Joe@stream-technologies.com; Received: from [192.168.6.128] (212.250.79.109) by AM4PR07MB3411.eurprd07.prod.outlook.com (2603:10a6:205:a::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.548.6; Mon, 26 Feb 2018 11:21:47 +0000 Subject: Re: Kernel Panic To: Kristof Provost References: <5A842FC6.7020806@stream-technologies.com> <5A8443BF.8040208@stream-technologies.com> <5289570D-24E1-4292-B4D2-D2F67D7D2D4F@sigsegv.be> Cc: freebsd-pf@freebsd.org From: Joe Jones Message-ID: <5A93EDC9.7020407@stream-technologies.com> Date: Mon, 26 Feb 2018 11:21:45 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <5289570D-24E1-4292-B4D2-D2F67D7D2D4F@sigsegv.be> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [212.250.79.109] X-ClientProxiedBy: LNXP265CA0069.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:5d::33) To AM4PR07MB3411.eurprd07.prod.outlook.com (2603:10a6:205:a::32) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d31de72e-e1b5-4822-b111-08d57d0b1f9f X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:AM4PR07MB3411; X-Microsoft-Exchange-Diagnostics: 1; AM4PR07MB3411; 3:WgyeqxLJX5+AE5EWlPAdPJZyL+/vdu+5Lyt0KSer9XoFchFeMrU++MyDkO/N+wpwksggwFU95uXzv7Lo4v75EeYRiNh1lQlKCVBvYC4KdyV5acXAGJFjs7HLcos7wPGJ/I0ktMxXcZRTQ5c1xHeTk8kktYwI2rvjOTEtOA4ER9akhsPo4vsAni+iSWmMp0+8LAwd1rlKcHEMJ9C+JTIxUQZNyQxOxmmUmX0N8DyNcDycDXTv5/7tbsrGSIgME882; 25:paofbFiR6xZuA/xXK/8Q9GPiAXE4vR7AZLi+cO5IrH7QlbTv6MosgqaScu405NckvaHNl81EQT6/ReLWNHVcefSjbikuVvfA9qQeCpdBVYmWYVZp6RQzorcdvkRLkD44zWYHHrmU8k9ZhZWCyr5o7pM2lmb4BYlsbjMOoTpo45qw3U6c+NhLuqsLvYJsV6aa1M2EbAWlR0+A6hf0IjsHYkMq/k/VVLhNXU8DI43pDenyPEvG0yueKGaObyexWl59FzxCVAufJRx6i3Pqlycde2gAlyo4AwLgz+eq3V6f55Mcl/cVMj2ZHod6ruxyZLWZy4PCbvNU6eHLPNgY+q15yA==; 31:zhQrV3K1YxbImsMmgarl9jjXy/giLGZJ+7AuPBhdpA+PVan/IfRANV1uwj7/m7nTzltzk5fnq+xZ542aAfmoZDwlB0aAKfUH8BHoRlIlTIb5x6gq70pErn3q7OoivmlRvkCWRnuz//XhAUzGla2ZvXEkJhkcaSjwU4Ulau594q3dlpgpdfnjLt3UKiRscQ4dVRSxYMucAc/T7pt6bT6m/AoLZNdp3FDN/Sp2nUy5eXY= X-MS-TrafficTypeDiagnostic: AM4PR07MB3411: X-Microsoft-Exchange-Diagnostics: 1; AM4PR07MB3411; 20:dCuwXkiEKdDjmFN2XzTnx1l7AQCNHIFeAJYeOtA9+7e5YYK32MDbqc828bdEPsyrUDB+72xPilyMloOK0iQP7TCKH43K8RfSurjPdQeNRBhQZpsGlGkjJcnfwD2CEP1z1K1b+CNo/XO+zb358RiJyrEegn1dHweQqAXWAXipPhk=; 4:BGbCJR/OGCVU2m8fGkqXvfYBdb0XT5dsJhPZvDUWPakcd+U15UVjp27jitoB1N2iv0+UkjiJrmfip3L4kwUwLYAk5dT4ORiRYjkU1Dq5jzKiKLPQbQCHSDK31tVRS9N7/pqR+2sqyIvBLvgpnDLGJbuKj/RR7LmKHxA9ui0zhwKcF/YXfNa2ipUrurBv/RT0tPaLtUbmizl3387dluJLvVyUQaShCSZVtcUHcwDYhrb5DnZQoO+7lItFxjI0Hc19Pm55ayCQcmnQKRY852Bxag== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231220)(944501161)(3002001)(6041288)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:AM4PR07MB3411; BCL:0; PCL:0; RULEID:; SRVR:AM4PR07MB3411; X-Forefront-PRVS: 05954A7C45 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(346002)(39840400004)(366004)(39380400002)(376002)(396003)(199004)(189003)(80792005)(50466002)(6306002)(7736002)(16576012)(966005)(67846002)(8676002)(64126003)(575784001)(86362001)(81156014)(81166006)(72206003)(97736004)(65806001)(68736007)(229853002)(478600001)(305945005)(106356001)(105586002)(59896002)(66066001)(6916009)(65956001)(2950100002)(47776003)(53936002)(4326008)(33656002)(58126008)(6116002)(65816011)(76176011)(2906002)(87266011)(52116002)(6246003)(25786009)(3846002)(52146003)(2486003)(2870700001)(117156002)(8936002)(316002)(80316001)(59450400001)(23676004)(77096007)(53546011)(36756003)(386003)(221733001)(3480700004)(7116003)(6486002)(5660300001)(16526019)(26005)(186003)(93886005); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR07MB3411; H:[192.168.6.128]; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: stream-technologies.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtBTTRQUjA3TUIzNDExOzIzOld5d1hUNlNOZk8wU2VRK1M0bTlZaGl0allv?= =?utf-8?B?TVhVZTZiRXpCa1Y3SlprMENjMnk5WWlySEUzVDI0VC93VFhjeElkd2c0V0dk?= =?utf-8?B?YXZNRTNkZnJ6MTJlY0x5VWxva20rdFVybWtPK0xmVVhHT24xZjN5Tk0rYlhN?= =?utf-8?B?cG11TjkvR2lZNEt1cFVpbGdXajhxVEQ4L1hkMFh6QnF3K0hZMmNab0dBamdz?= =?utf-8?B?MWVUMENvYUs2ZDRoMElrTnNZRzVxeHpVV0g2WFZEM3NnL1NHeFpHZTVZNGlX?= =?utf-8?B?UXd2Q0ZZSE5PZWo1VnUvRGExWVBRL2tjZVBUTEc1dlo2b2FwSksvb0JiVngy?= =?utf-8?B?N25oR0ROdGUrdUhsUEI1bTVDczEvTzExUUZ0aGpvcW1IV3ZzTEJNWndVOHpJ?= =?utf-8?B?dElCS0tIaEhGN0ZnVDFrL3hMN1JEOXQ2SVZPeU84RWRkOXBYekVDUWdMTXdU?= =?utf-8?B?UUVpdjBWSHFGdUozRkNpamwrNWJ1WWdrbk5HcVlVTmVUN2Iybm40MWxRTkR2?= =?utf-8?B?SGRvazA0U3Z3WWJyVGM3WkNaTGpoSHVWMDZyYTQxUmRVNmhodnJhRDhFZGRS?= =?utf-8?B?ZldMSjlhYjR1bTZSdGRObkNnWFc1K1QwdlY4R0Q0T3lReElnS1dMT0tkU3Bq?= =?utf-8?B?V0ZOSnJWZlEyVFZsWktRTmFVRWpJZ2lOSVRFNFI0ZWRSZGN1ZVdJSlVCQ2ZM?= =?utf-8?B?Z1FtTm9weTI1dnJFNFNvSkxpamhmcUh3ckhzTnZqUk8rZW9oQ0doSS9xRmZu?= =?utf-8?B?aHZFY0pDN29pbzdabzJKQ2NWVXBPVkkybmVzS3hhYVh6dWRRaUt0a2VnNmFn?= =?utf-8?B?SWVjRjNEZXMwY2QvbXNkVjlTc1haOWs1QWROYWhIM3NVOElHajU1cUhEL0M4?= =?utf-8?B?SnQ5emRENkRTeWVCOGo0TTN3Y3RUVkpMV1I4b1Q1eGt2aU14dDN0bnREOVBV?= =?utf-8?B?SjNrR0lFL3NsNGlCSHFYVzI0YXd1YnBQOXdtdm9xVERpYVBGandlZlBXZUJm?= =?utf-8?B?NDdOZVpVUDdUbUROQkpUU2IzemVJbW5UOWxQKzZ2ZnRnS3ZvSFNJRHJmQTA5?= =?utf-8?B?SGt5dk0zSUZGL3ZML1NBei8rM2JnNDVvdW1DSm4vZzg1VTJHVGZ4R1FzalF2?= =?utf-8?B?UkVhblY2NVArNmc0bTd3YlFEakNvNkE5eHRzUG1qNldSRjE2b1RmWWlzYTM5?= =?utf-8?B?YWZEVnR2ZGdjUzdVaWl0enlTZG5WUXBtTVp2eUtkemMvYytOQ0cxTk1ad2Fw?= =?utf-8?B?VHg2N2FZNHZvMXVoYVpLdDZSUUtod3U3QzhRUVQ5V2N1QzRTcU9OZy9kbEFO?= =?utf-8?B?dDVqekZuTnlJMEhMclVHQmhmRGJoVGNUaERnZDgxWUV3ZHo5OUZBS1ZDb0gx?= =?utf-8?B?em1hMjIxMXRQNWlPbmU0Z3FiRC90eHl2SmhNQ1FaSFk1NExlRUM0NXpRbjVN?= =?utf-8?B?eEtzeisvdDNZdXE0K3R3OThvVjhoOXhXcUNrTG0zalk4eXJZQWFPSHhKVW4x?= =?utf-8?B?Kzg1UUMvTFUrNkZwZUk0U1dkU1oyZEFmUlpIQXlvN1hYWUpWQUJQS21NRWRD?= =?utf-8?B?R3pNeXl1Y0k0Q2VTMUpGTHhtb2V3RnphdXB6ZUZMM1BTa21uZ2FTai9RdUZE?= =?utf-8?B?eUdBTFhqeWNaZndxTEtRVjgvUXc4dEVvSExRWjRsaWtTNXgwaXNESzc3L2xK?= =?utf-8?B?YkNQeElicjVjeGZUemZBaGx6cFNVQ29XV2U5OW1VQXZKUjUyUWRoVGExdzVN?= =?utf-8?B?VmxHRGJ2ZGF0REhkTmk0Y08vb2pkdGZ2c29yYlB0MHhncXljblREbzVCSjht?= =?utf-8?B?ejkvcUtpczlHSWxNMnd2VktDS0Z2SXJReG9sK29lMVp1VER5bHhaYVBGOGJM?= =?utf-8?B?UnF5Q1hNUEJtMDl3YnBWYThSVnlaOUpUVWpZQnlZMVBtallZN3h2clhPcUxS?= =?utf-8?B?VFE0SVpOSnFFZVVjVzRXK1B3TnRVR0JYRENyVDdVVWZjNC95K2lUL01lN2RR?= =?utf-8?B?OEliNUFKRndXS29ud0FSOWcwZlBVZU1uODFaaTI2M1VVOHYwQ2pQdlJMd1B3?= =?utf-8?B?dXVLbjFsTkhFRElXTDZCNENXb0V5WmxGa0xxblhFUk5kNXpiUUkzRSttZ1Uw?= =?utf-8?B?VTUramNTYnZsRHlqTk9yQ2FHMTdJMHRvQWx0NFBlckxVQVRDOVhZdDRoL24v?= =?utf-8?B?QytTcmxrbWlhanUrc2tIUGhuUEJnPT0=?= X-Microsoft-Exchange-Diagnostics: 1; AM4PR07MB3411; 6:a/gWSX4rHWVQly7YQtHs2xoaz16JqY6bq1szF49h2Bhh/ifTqkmxZ+oEr2s17qsSDBG2rwTSGr1Ws2VKtCEIOkUqjvWNPxi229oMBB2znOksKz3FyALcE1isPOVEgBuFuL7vaS3nBD0EhfTIVmWqtt9EgQREEzoAytm5eg4OlIIqXvn5iy1D/sRf5llwA2f6JRtCEpwJWrb+H15pK21Qzbk5P3hkMNDx+cZnR7RliE4wmzux6SnhLNXBoUS+sxSa9b2Eu71GO5gvpy0FdBKJj87xzhdB1AmovQpK7rMcC3iYuCVhbaLz/bozQs8yKIkrz/D3exhOKylZHRhCYlK1mWYECTR4As7ICr7KWWkH5WE=; 5:J2QFcFWZC/ILxGUFmruauDiR4OGohl6d84oqLZIHQQU+Q5nYNu0q5UH6pQQ3RLi2jpDskXb/j+48pAE5sKU0a3mRAH3fsSYHLoVUGWq9m8SWCVfmrqF0Dk129N4RgSaAZkLUIwtxNYbRNXIIqjdJnYxzXHX5B7lBoB4e+p7jlsw=; 24:IdEm/5MdV0vOHO9YZzyQVyHv6Odq5bkUTt9Un98RIPGHngKcqi7YEtzuYob6vCNbSkuEqr8RNqiY5PQvG+sau8Mv7S9Qv0P3mCLIRIl9KeY=; 7:PfrC8zfvq0+7v6bqq7yuWFJSPjhu+py4yAjUkRm/MYJPc6cbACt7NFpw2ZW8X9fXgsjTEm3sxHSkM3ZUIYd0yf8y0saSO70VFQ3c/ZMLUkkyBRwgCab8d3WdDjG7USt+NeV69AefH8lKrPwwFlBK4UfNmSEw06KznwOdy9RwDx2RTfTp+NPwKaKnsyRM3cMeh8vPoBtI9ZMTofvfETKQfiBXG8ldV3oCFW4Fwyuoiexg3NvJnpCOMu5AIi3iCPsy SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2018 11:21:47.0484 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d31de72e-e1b5-4822-b111-08d57d0b1f9f X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 720fa073-5781-43bf-bc14-7bef2603ed21 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR07MB3411 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2018 11:21:52 -0000 Hi Kristof, we are not updating rules during the test although in production we will reload the rule set from time to time. We are constantly adding and removing from tables though, using the DIOCRADDADDRS and DIOCRDELADDRS ioctl, also DIOCKILLSTATES is being called a lot. These are all in response to RADIUS events. We tried using pfctl shell command rather than calling ioctl directly, to check that it wasn't a problem with how we are calling the ioctl. A little background. Our production system is running on 8.4 and has been stable for years. We are in the process of moving to 11.1 and are having big problems with stability when we allow customer traffic into the machine. At the moment we are using mirror ports on the switch to play live traffic into it. We're trying to work out the simplest configuration that causes a problem with a view to producing a good bug report. I have notices that the pfil interface https://www.freebsd.org/cgi/man.cgi?query=pfil&sektion=9 has locking in it which didn't exist in 8, I think it was introduced in 9? the locking functions appear in the man page in 10. I don't know if that interface is used directly by pf, but I'm guessing packet processing needs to be thread safe in a way it didn't in 8. Regards Joe Jones On 25/02/18 10:56, Kristof Provost wrote: > On 14 Feb 2018, at 19:57, Joe Jones wrote: >> On 14/02/18 13:09, Kristof Provost wrote: >>> On 14 Feb 2018, at 23:47, Joe Jones wrote: >>>> we are running test traffic through our system, after between 1 and >>>> 12 hours we get a kernel panic, always in the pfr_pool_get function >>>> in /usr/src/sys/netpfil/pf/pf_table.c line 2140. After a bit of >>>> investigation I confirmed that ke2 is set to null on line 2122. >>>> >>> It’d probably be interesting to know what the contents of uaddr/addr >>> is here. >>> From a very quick look at the code there’s supposed to be a route >>> lookup there, and I’d expect there to always be a result. The code >>> certainly expects it, because that looks to be what causes the panic. >>> >> >> (kgdb) p *uaddr >> No symbol "uaddr" in current context. >> >> (kgdb) p *addr >> $1 = { >> pfa = { >> v4 = { >> s_addr = 2016475826 >> }, >> v6 = { >> __u6_addr = { >> __u6_addr8 = 0xfffffe0000310d0c "��0x0\r1", >> __u6_addr16 = 0xfffffe0000310d0c, >> __u6_addr32 = 0xfffffe0000310d0c >> } >> }, >> addr8 = 0xfffffe0000310d0c "��0x0\r1", >> addr16 = 0xfffffe0000310d0c, >> addr32 = 0xfffffe0000310d0c >> } >> } >> > Interesting… That looks okay, so I have no idea why that lookup > returned NULL. > Are you modifying tables/rules at all during this test? > >> Am I right in thinking that's in network order. >> > I believe so, yes. > > Regards, > Kristof