From nobody Thu Feb 8 15:18:02 2024 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TW0wv4xLTz59sGR for ; Thu, 8 Feb 2024 15:18:07 +0000 (UTC) (envelope-from muhammad.waseem@sophos.com) Received: from id-euw1.prod.hydra.sophos.com (id-euw1.prod.hydra.sophos.com [198.154.180.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TW0wt3H1Qz4DK2 for ; Thu, 8 Feb 2024 15:18:06 +0000 (UTC) (envelope-from muhammad.waseem@sophos.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sophos.com header.s=sophosf69f0521910d49beb03f4ce823e25fdd header.b=WY5vGNpx; dkim=pass header.d=mail-dkim-eu-west-1.prod.hydra.sophos.com header.s=v1 header.b=7kOGEn7C; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=sophos.com; spf=pass (mx1.freebsd.org: domain of muhammad.waseem@sophos.com designates 198.154.180.3 as permitted sender) smtp.mailfrom=muhammad.waseem@sophos.com Received: from ip-172-19-0-242.eu-west-1.compute.internal (ip-172-19-0-242.eu-west-1.compute.internal [127.0.0.1]) by id-euw1.prod.hydra.sophos.com (Postfix) with ESMTP id 4TW0ws25S3zjWwy for ; Thu, 8 Feb 2024 15:18:05 +0000 (UTC) X-Sophos-Product-Type: Gateway X-Sophos-Email-ID: bf13eba9928b4eb9842188191931129d Received: from GBR01-CWX-obe.outbound.protection.outlook.com (mail-cwxgbr01lp2041.outbound.protection.outlook.com [104.47.85.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay-eu-west-1.prod.hydra.sophos.com (Postfix) with ESMTPS id 4TW0wq4FV6zvPry for ; Thu, 8 Feb 2024 15:18:03 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mia/7phe1HCvs0CxsG/n8GIsMjCMjcoQtYw+0QPohUAqGNMYBDyON31VftxQlmXEqoI0Gm6KzVq87uEIZD8qd1no9R/PXafrbiMWj1Kca6vt7+/uWKG0vfPsfHjOZqM+30nmpiH8ZgXqiuoFC/Ug3A1BGfH3VzwWAAhckP2KDvHFbBEhYuGmpcl+BpQVzUIJ7z4dGclDAgj5p1VDwNOiRAiumDiAJ8gvOMBjvkuVvFogDami3XEen/0iNFzStMABBIjv8VJ1j4XaOTNX8bsHJP9QyF0PbLimYqp1F3eO2RD+GkXNwqrAkaF93oLq6vSG87WlNsDIXg+NEJ7Dqa+hDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kg4xUXklqGzrJBVOuTDg74dOeyHjj0HIEkcEFxh6sms=; b=ZdtL2fGcYU7CDNEvJ856/742MlYvu4mwzy6y++6WoJo7afQBPErsVS+DupqOgnK0i5PZvXZYZ882NEp3Kpjlde6giHKHqjtZlfpo08bOornXPHJk6RWdWQoAbt6SuOaaGrYmLaSX+Xq9ROzYYzxubLkopG+nnfJ8Su2zCzCY4JPCD+MV4jgtTgcnSVeFALwX5Hx8SoLN/LGEkgxRJDIh8vu8FcP86xuQ7ZdpzXhGcwUStMrf3tFsFsXgzXGnzhzM8bzCvEy6SVBLuamgEl8mWGLnVqw60OpZskNcXTvMHFuj9Fz26fX3vUGN0+Sxw4Ug0MqwUMhnWnOwtAhxlBCRuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=sophos.com; dmarc=pass action=none header.from=sophos.com; dkim=pass header.d=sophos.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1707405482; s=sophosf69f0521910d49beb03f4ce823e25fdd; d=sophos.com; h=Content-Type:Date:Subject:To:From; bh=Kg4xUXklqGzrJBVOuTDg74dOeyHjj0HIEkcEFxh6sms=; b=WY5vGNpxkwGUoVJi0GkCYjusc+M0UVlslTf7EWIDBxj6bbGTwGGeZvN7H6vYibEg mMqlPLOks2SYOAwKtGgSY2IeBhbsxT0RiKo8dP6KT5/m4sF+joDq1C+GID0Yyn9c6U5 boR6AqRb66I6Brt306q/On7Lqb1EgE1SApVaKp6S3rXQvt8X0nBQNwGqY2zG7Gw6yaO V5SlPB272RttedYieCNxqTVE8pVlcpj3Jv0wSBSs15BIPmH6eDwCRSNmufRXwz3b2ol q5y1bRUIckKiShi8oghIO8xuxrqr0tdgRPJQROzhvg1WO/jiV7YImGFSpFLNc8hJnXl ByDWzMS6nQ== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1707405482; s=v1; d=mail-dkim-eu-west-1.prod.hydra.sophos.com; h=Content-Type:Date:Subject:To:From; bh=Kg4xUXklqGzrJBVOuTDg74dOeyHjj0HIEkcEFxh6sms=; b=7kOGEn7C6wzfcoB7bhOrnoSdEph/ctDoYK9psz+balV8HM0nlRbq1YyFkn9PCQEo FMD3bvuRVq2QjnCuY9nHble1hYFbF+GAZeYdc7ZegZ3/jhh6IB0Qx8dK4Ima6juhI3b NudRYsO9Q8GyEXO+h2WB6e7kYyDJq45fCRlTFlG37W0WsJkv4mFgEqVdbgw2VVBz9OA TOpsgg7fH5PATR8L2Wb0I5ICu4C2daq5sB+5knTlzse51JWoPJGAHGimt70QyNsHn0T cLahi7vkpynbzA/vcM4vo2AhMUAlW77JMaHgeZBXpTE92sI0uCLNdkZCqlj1f1HzNPq t5jsDUV1cw== Received: from CWXP265MB4796.GBRP265.PROD.OUTLOOK.COM (2603:10a6:400:153::12) by LO2P265MB5659.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:25f::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7270.24; Thu, 8 Feb 2024 15:18:02 +0000 Received: from CWXP265MB4796.GBRP265.PROD.OUTLOOK.COM ([fe80::ef23:c704:4d04:55fa]) by CWXP265MB4796.GBRP265.PROD.OUTLOOK.COM ([fe80::ef23:c704:4d04:55fa%4]) with mapi id 15.20.7249.041; Thu, 8 Feb 2024 15:18:02 +0000 From: Muhammad Waseem To: "freebsd-questions@freebsd.org" Subject: Seg Fault while upgrading freebsd TCP stack to v13.1 in mbuf chain Thread-Topic: Seg Fault while upgrading freebsd TCP stack to v13.1 in mbuf chain Thread-Index: Adpaofyk7FmBjylXRT+I87aknj3ktw== Date: Thu, 8 Feb 2024 15:18:02 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CWXP265MB4796:EE_|LO2P265MB5659:EE_ x-ms-office365-filtering-correlation-id: 473c3d95-1081-48c3-2740-08dc28b92438 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wvPtvGEb5GTmivced12RO9q0YDkOBZoOIGdgs8uKefqLYWqR37/7PUnkgimfWBHe6R1zmbH0onREANKg2JmXQpQ3RUaMenw8jiks+vLSix9kaSMwMqZV0si8AtW47VLPaR9SauxooXiewFYywqJRTbzNgbspx8tvqeFuJPnnvEyGcIv+igiXzuCRQO/UO3FL6KuIzk8hlAuovhWXMxdbC+QWWJ0a0VrakYYqEFUwqX1Yauq4BU2Z0Yt6rEXcAs/ZEq2250idu3f4J4LeFMALYfTJxwkGkNGHaBapxZdfQ14qmYp1Ndy/sBs1OrJyQ8nMMMqFb6DN49u4n5I28WBYqdkiKc0JILBulGzV6B5MIp1Vi/FNN6v1LyKjsptiW1hTozn4FKLCNs1n5ewNUMKW0f8lKSi9BU/FgLaeBXCLfd38F+JZOo1QV8hFIk2WARaFgZNpAHBFCfegZsa+/DKuwhi0CqlUkfU7ScQBqOffDFFSBYB/5ZSqWTRA4yrInWePg+gZjQI05CqE1gN8iQXitui2RzIU5yHlgFd4KpxuMyl/hN91BUovPVC0IiyZTxn+aI7whaRW4MY6YS4FmtuYi+ugxBWWfE5bDwPU9yMLDx5dZnx0JsU02BLm3E3gX9To x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CWXP265MB4796.GBRP265.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(366004)(346002)(396003)(376002)(136003)(230922051799003)(1800799012)(451199024)(186009)(64100799003)(5660300002)(52536014)(316002)(41300700001)(2906002)(83380400001)(38070700009)(26005)(33656002)(122000001)(86362001)(38100700002)(8936002)(6916009)(8676002)(66946007)(66476007)(66556008)(66446008)(64756008)(76116006)(6506007)(9686003)(478600001)(7696005)(71200400001)(55016003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LMd9g9+SfCOZFshDQ2kpixf5B01PDSmXrsyMH6ZwRNI/DOzp+GijX/qlGh86?= =?us-ascii?Q?EVxiukm8gsaQ3DCYpFNGpdAkmYpe9MqlU+H/SUQNdXq+UuXA3FPyvRZfBKA1?= =?us-ascii?Q?Qvqe85/pV4ePfKjYGlnwxFQd9nCmiQ3svpmIVgNNMOn3DJiL5CY4Ym382TfA?= =?us-ascii?Q?CVJaJu3sOKm6NX8VDhM/0b8CaFBwoZzHKB+5p5Nwc7wZLaUSetHep9GoO9Lv?= =?us-ascii?Q?+kHA1AHsTJenVtSoyC0XCsymdly/vFH00lxmPV1qJlffuqKJDFFUltZ+SgGR?= =?us-ascii?Q?m4ouSmU8eYWFAGEt69rVp+0F2O66T9UuIXSK0wB/WjipAgE+1cfFFDEqCd5V?= =?us-ascii?Q?/a+pTxPHiPPvg4uJhxVVKV7irkE9pQAwwqGUMqwJBkEoobgpC41nQBcKwyOt?= =?us-ascii?Q?AR50R0yRR7gdKrTOKkPicgYj4nitputse/t6MPe4OYDscGDGwV+EcfTLCoq8?= =?us-ascii?Q?UGXI11TIu7ccqtptVAnGH1+1iB/cweVSUJLllQTvzWxB79nPf8FX5lCBKrBz?= =?us-ascii?Q?0bJIj/EqLKUurl7ddLNUNcNfqC4M+uM5iU4ZMfbZsBReg8uLDn0uwf3snlaI?= =?us-ascii?Q?kHsjPSGhgZQGcDEpZ0ApOpRWN952yGUXkFcmJM2ArpiS2ROmQXLqeTOqVdg1?= =?us-ascii?Q?soIPAl4T2OLgSaw9/WVSR8DFP+6bnn0Z7gkWDUYpTcvAhusf41LI0qKtCfuL?= =?us-ascii?Q?99IUflDO5maFGXuFXUXaXjL6h/lN+NPzU83c2HX1JQ/Fp0w2V6eI+Qi9+2c7?= =?us-ascii?Q?tE3275rwjhRztsW8icgOBkrvqlbAxGIEqqq2qeTAcYVpSYf7BDvYvhVzaOmi?= =?us-ascii?Q?+9CzbPnHuPYDVpnQsL7mJKGYp6fmS4MU5aYlsb1Ipbbam+ogNL9KI/AsRGKN?= =?us-ascii?Q?AWWvBV8tIcHvVQLHunyv9VspBRXlc4O4iiCz0grrRRB4t4dUoAHTqojD5E15?= =?us-ascii?Q?UrVvf02jXbnczRTfGiktn6MVNEFpGUKYS9yY1LURxP2esdvcWpceNFes09Gx?= =?us-ascii?Q?79Hf+6Xf+RNwn6qkNVwmhTkwnUyfS2HXQLG44fm6DLRWbyAR1Hu2lT582XG3?= =?us-ascii?Q?qVKwjmCH+0CDgDXUhkNCjMt3qDT+s+CXKjqfMDDMQb9PTjvdcMQllbnEHDv0?= =?us-ascii?Q?u3LIjviYTWoqL0hGoX6oe1cj3thg9QSHijyWeaSgNPvQzY6GRTkDX14aoekw?= =?us-ascii?Q?BMgjQZAzQeuj1ENL+lT1GaxYONcFg0CIAlfzGw73qVRtKJBj/plDc/x24Jqp?= =?us-ascii?Q?s73ZUxAlUmXofhxX/FNtMkl2CdEMzLHjuAWQPCpTVLRwTN3AI5t8wJTKzXTQ?= =?us-ascii?Q?GeE9wWeWNkS0orkFYGjDSFvkJb9AhEyQNzUvcgm6cUtnqOk2pbUCPg5m/+RV?= =?us-ascii?Q?Pcfc5IiyY4jbILq/TGy7F8SvKqeSrERoTLtkdPPXz8/3ivuF0OBeyyzKSul1?= =?us-ascii?Q?XYLFiO9UDiURqyqIDpFCn/AfuuLxXFESnWs2AVXSlwDiRdIDnni9fh4H55xe?= =?us-ascii?Q?Gr9WfGvS9lmM/tc1bBwqoJlQv50hF0RAgtR895tN5MP2TMbzxkUD3RfjJL8m?= =?us-ascii?Q?+3e3MjvbQtC+BODBvCqkQmEALnjSDLeYskmn9HA0?= Content-Type: multipart/alternative; boundary="_000_CWXP265MB4796C339D93CED1682BD8A5E9F442CWXP265MB4796GBRP_" List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: o2kK6qWEeLzCkMSQxq+pbWUp6VOaMvfevUnX1wG8hz5E2mXJEl+UKmiAoCLBiQH03oROtdPtBT7krCEBO6fS/WVtDvaLKbBRBq62SQWQILUhQGCEGADe+pivUw5z/dWgbJ2XpTgzs77nIasN5fuEcwWwCueOY7gZz41VD/EkrsOapBs2sqRDSP+wFaWt+Hv0lAv0+PZTdt22ZDpHZhilaKyAi30Pf6Pq/r19Ce4KyVtuNUCtAlFzFNYxRTCs7kN1YQGJGifwMBE4wPExDUePQieR3almt/NdCvUdefatif30B1FkkyTwJmDfrcwLA4PqNpJdKJ5HQKbE2UKFB4/7yhodn/yMHP6NNRJ0HGYr6pz5bz1cHOQLxtb+IDTiczgLktfB/9Av7OGVx8pFE2ckINBxHh5gqsPorcAKZTme4Y0tM/1WU487let9zpVf4qAeE6BLpulS6UJtIyS5Q2qsq6PrF7oy2OfncTx08meas4DA/ZJ7robjuAfnU1BllGgUks0DYH+OZafJFGw7e+xl/hqXXwJNvtjJvQ8CrLfYFjjgBEoBCX5pFcB36NdPxXoV2x/CUwOwRV/XDs433ODEaSJAhs9M1zsmoeXv0NVQPY9WHlQFHfYFexwEeySdAj/6kgE7gI8GFoNVK+spCcaJkZCACJBWg1PFFd5ciiS27kzwTcQT5DkjoI03/MqCtmyo X-OriginatorOrg: sophos.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CWXP265MB4796.GBRP265.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 473c3d95-1081-48c3-2740-08dc28b92438 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Feb 2024 15:18:02.1399 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 358a41ff-46d9-49d3-a297-370d894eae6a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Fb4pBEzgMsntby5twildwhQCEFhalujyR0nbrFjJY5EZHNHrWRsg+Y58FU4l9BZre8PGqAM8cObhsc3JbnCfNcl9o/vyWbjlMI8lj8mTgtI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO2P265MB5659 X_Sophos_TLS_Connection: OPP_TLS_1_3 X_Sophos_TLS_Delivery: true X-Sophos-MH-Mail-Info-Key: NFRXMHdzMjVTM3pqV3d5LTE3Mi4xOS4wLjI0Mg== X-Spamd-Bar: ----- X-Spamd-Result: default: False [-5.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[sophos.com,quarantine]; RWL_MAILSPIKE_EXCELLENT(-0.40)[198.154.180.3:from]; R_DKIM_ALLOW(-0.20)[sophos.com:s=sophosf69f0521910d49beb03f4ce823e25fdd,mail-dkim-eu-west-1.prod.hydra.sophos.com:s=v1]; R_SPF_ALLOW(-0.20)[+ip4:198.154.180.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[198.154.180.3:from]; RCVD_IN_DNSWL_NONE(0.00)[104.47.85.41:received]; ASN(0.00)[asn:16509, ipnet:198.154.180.0/24, country:US]; DWL_DNSWL_NONE(0.00)[sophos.com:dkim]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MISSING_XM_UA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_EQ_ADDR_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[sophos.com:+,mail-dkim-eu-west-1.prod.hydra.sophos.com:+] X-Rspamd-Queue-Id: 4TW0wt3H1Qz4DK2 --_000_CWXP265MB4796C339D93CED1682BD8A5E9F442CWXP265MB4796GBRP_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable We are running a software that is used to operate middleboxes, and we have = the FreeBSD Network Stack for the TCP protocol implementation. It has serve= d us well, but we have not upgraded since 2013. In the past we have address= ed issues as and when we find them by applying patches from upstream, but t= hat won't work anymore and we decided to upgrade to 13.1. The upgrade was g= oing fine until, we were running smoke tests and we ran into a core. The particular smoke test in question uses Linux Traffic Control to add del= ay, and other impairments to the packet transmission. The exact options bei= ng: delay 10ms reorder 10% and we are using CURL to send a 512 MB randomly = generated file with a 45 second timeout window from the client. Even if we = expect the test to fail due to changes in the upgrade, it should not result= in a segfault. We primarily see the segmentation fault in this low memory machine and not = in other environments. We don't have a liberty to run memory analysis tool = on the environment where the core is reproduced. As for the details of the segmentation fault itself, its occurring in the m= buf chain. In the different tests we have run, the crash point is a differe= nt function, but usually occurs in these functions: 1. sbdrop_internal 2. tcp_m_copym 3. m_split (very rarely, it has also occured in) However, what's to note it always on trying to access a member of the curre= nt m buffer, e.g. m->m_len causes the crash or m->m_flag causes the crash. = I have tracked the faulty address that I get from these functions, to a soc= ket which is assigned from tcp_input_with_port() function from the inpcb st= ruct. The address is of course inaccessible in gdb. The faulty address belo= ngs to the mbuf chains in the so_snd socket buffer. It is usually the mbuf = in sb_sndptr. Either the first member itself or down the line. Although in = one or two cores, the same applies for the sb_mb mbuf chain (which I assume= is the main chain itself). From the addresses we can clearly see its a hea= p overflow, as I was able to go through sb_sndptr chain in one the cores un= til i found the faulty address. The last address before the faulty one is: 0x7f402a4d0700 after which comes= 0x9fff22eb779f. I also see this faulty address for the first time in the f= rame of the function tcp_input_with_port(), in inpcb struct(inp). The very = obvious difference between the two addresses and it show that somewhere whi= le accessing or assigning the mbuf, an overflow has occurred. These are mos= t common back trace: 1. sbdrop_internal() 2. sbdrop_locked() 3. tcp_do_segment() 4. tcp_input_with_port() 5. in_input() 5. netisr_dispatch_src() 6. ether_demux() 7. ether_input_internal() 8. ether_nh_input 9. netisr_dispatch_src() 10. netisr_dispatch() 11. ns_net_tcp_push_frame(). I have tried to track down the source of the faulty address further than tc= p_input_with_port() but with no avail. I only have cores available, and eve= n gdb blocks the seg fault from happening in the test. I have gone through = the code, and according to my meagre understanding, nothing indicates towar= ds a heap buffer overflow in any of the above functions. Any help, in point= ing to the right direction or anything else would be greatly appreciated. I= f you need any more information or a more appropriate mailing list, please = let me know. Thanks, Waseem --_000_CWXP265MB4796C339D93CED1682BD8A5E9F442CWXP265MB4796GBRP_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

We are running a software that is used to operate mi= ddleboxes, and we have the FreeBSD Network Stack for the TCP protocol imple= mentation. It has served us well, but we have not upgraded since 2013. In t= he past we have addressed issues as and when we find them by applying patches from upstream, but that won't wo= rk anymore and we decided to upgrade to 13.1. The upgrade was going fine un= til, we were running smoke tests and we ran into a core.

 

The particular smoke test in question uses Linux Tra= ffic Control to add delay, and other impairments to the packet transmission= . The exact options being: delay 10ms reorder 10% and we are using CURL to = send a 512 MB randomly generated file with a 45 second timeout window from the client. Even if we expect the tes= t to fail due to changes in the upgrade, it should not result in a segfault= .

 

We primarily see the segmentation fault in this low = memory machine and not in other environments. We don’t have a liberty= to run memory analysis tool on the environment where the core is reproduce= d.

 

As for the details of the segmentation fault itself,= its occurring in the mbuf chain. In the different tests we have run, the c= rash point is a different function, but usually occurs in these functions:<= o:p>

1. sbdrop_internal

2. tcp_m_copym

3. m_split (very rarely, it has also occured in)

 

However, what's to note it always on trying to acces= s a member of the current m buffer, e.g. m->m_len causes the crash or m-= >m_flag causes the crash. I have tracked the faulty address that I get f= rom these functions, to a socket which is assigned from tcp_input_with_port() function from the inpcb struct. The ad= dress is of course inaccessible in gdb. The faulty address belongs to the m= buf chains in the so_snd socket buffer. It is usually the mbuf in sb_sndptr= . Either the first member itself or down the line. Although in one or two cores, the same applies for the s= b_mb mbuf chain (which I assume is the main chain itself). From the address= es we can clearly see its a heap overflow, as I was able to go through sb_s= ndptr chain in one the cores until i found the faulty address.

 

The last address before the faulty one is: 0x7f402a4= d0700 after which comes 0x9fff22eb779f. I also see this faulty address for = the first time in the frame of the function tcp_input_with_port(), in inpcb= struct(inp). The very obvious difference between the two addresses and it show that somewhere while accessing or as= signing the mbuf, an overflow has occurred. These are most common back trac= e:

1. sbdrop_internal()

2. sbdrop_locked()

3. tcp_do_segment()

4. tcp_input_with_port()

5. in_input()

5. netisr_dispatch_src()

6. ether_demux()

7. ether_input_internal()

8. ether_nh_input

9. netisr_dispatch_src()

10. netisr_dispatch()

11. ns_net_tcp_push_frame().

 

I have tried to track down the source of the faulty = address further than tcp_input_with_port() but with no avail. I only have c= ores available, and even gdb blocks the seg fault from happening in the tes= t. I have gone through the code, and according to my meagre understanding, nothing indicates towards a heap buf= fer overflow in any of the above functions. Any help, in pointing to the ri= ght direction or anything else would be greatly appreciated. If you need an= y more information or a more appropriate mailing list, please let me know.

 

Thanks,

Waseem

--_000_CWXP265MB4796C339D93CED1682BD8A5E9F442CWXP265MB4796GBRP_--