From owner-freebsd-net@freebsd.org Mon Jan 1 22:05:13 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1A14E86087 for ; Mon, 1 Jan 2018 22:05:13 +0000 (UTC) (envelope-from charlie@atech.media) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 6B25180907 for ; Mon, 1 Jan 2018 22:05:13 +0000 (UTC) (envelope-from charlie@atech.media) Received: by mailman.ysv.freebsd.org (Postfix) id 67051E86086; Mon, 1 Jan 2018 22:05:13 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 668DFE86084 for ; Mon, 1 Jan 2018 22:05:13 +0000 (UTC) (envelope-from charlie@atech.media) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0070.outbound.protection.outlook.com [104.47.1.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A988D80906 for ; Mon, 1 Jan 2018 22:05:11 +0000 (UTC) (envelope-from charlie@atech.media) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=atech.media; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=cd4SGEbPqtqfZ6zFGgceNSdt9tF2A2BL8RI+Ednwe54=; b=fvIodqmcv+F9Ja5E51Pww8mUQp/TCUG1CSokBnwTaXAqVbG2QhjlBzuS5yxA7vNFGDqYjops1SKrhQ0ex95QV8A65INwYCiw+CJx5h62OHa2w9aOFetUVHDtGx7U22ZSNOhtK9a3DGztS8HitR2OfZ9vMJBSUO7tFWcI8ii+VnA= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=charlie@atech.media; Received: from [10.0.8.11] (185.102.133.45) by AM4PR05MB3492.eurprd05.prod.outlook.com (2603:10a6:205:6::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.366.8; Mon, 1 Jan 2018 22:05:08 +0000 Subject: Re: Linux netmap memory allocation To: Vincenzo Maffione Cc: "freebsd-net@freebsd.org" References: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> <6c5de1ed-0545-31b3-d0e2-4258fa4ccf1c@atech.media> From: Charlie Smurthwaite Message-ID: Date: Mon, 1 Jan 2018 22:05:04 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Originating-IP: [185.102.133.45] X-ClientProxiedBy: LNXP265CA0031.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:5c::19) To AM4PR05MB3492.eurprd05.prod.outlook.com (2603:10a6:205:6::33) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6e5e5521-0e0a-4824-2055-08d55163b87d X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4603075)(4627115)(201702281549075)(5600026)(4604075)(2017052603307)(7153060); SRVR:AM4PR05MB3492; X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB3492; 3:mKPRsV8Jn4wU4ii/FenHbjBRFWKnHnjyrm9xnuPQsVOdOOgWaMUMKyUbscLZq6rOjOnmASetQDyAJQrwZNPx0M0r1A4KmE+wDKz5x5E5LwCSpYI3PQxzz88msKP7g7Hk/RjnfX6Xyge4/1Tq/CzpIGvUCtdBc4nIq2lUJGdNzMkkDzhYAIxFFYfbigxq+bvGlwLdU64/iIm7gKTbqN+5pMaooyuH/6fPQiuL0OO9EftYPnq4seVwWEZBIIRLW/kY; 25:7NnkJh9gfe6qPYrdwgqev8Vw2LVl09TchRLSFgLeVwp6eEgCRCVmtxBjJrn/8s9YFtf+qJQNDEe4E1gSPFzQiwl0JfYYqlhAc77Nvq6zQTeu4PxHqOv/qB5DaR7wJdav3PBuPZlCBu+fiOt2oWgsQwAuAC5SmoysTw6mUwcYfHwIFp1yXV5VizF5Qw0E0FUNlWsr4xCImB12MeB2WZ60UlCn2tNBaL+Xh7Z6C8CHfzr7wnko8UcWpghq16Vqn+2aNlDkWMDFwihFjo+vHKJG6fCN+48i+GQlhpZrrzvxoMTYLf9EOssAmAl1qU6PKx14f1JI5XFOUdbNN6VkJd/ADg==; 31:WsIZ1dgiX87xXn14KcLcCnU3F9sbncm2XAyVi5WCLjACFLPC7ltDsM4V2n6meg/dOB7tuIyoo7B8nzrRcoQIHfyCKAmhKSJz4ZPOk07curcFWhWuvKJpk4Jx39nwQ8Fa/N34Lz1/gEHVTlBkJmHQJESAGLi7Vm4ZlTF62awqfrWLEYSO3YLpklYjk+io70i6ELbQ60PVv5v7oeHi3yJxHYn7zKb4Y34x/X/RgBOrKHs= X-MS-TrafficTypeDiagnostic: AM4PR05MB3492: X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB3492; 20:7Tmyhkuqc50YsGh77I3l4ljAy11O6KF50mDqoEvngTRI32zQmv8IksyurKsg0BK8msmkSVYf8Z243KNlwX7muJ5vDeFO2fw7NV4KKr4MW0X4vPv1rRQOwzFaX5nfmZVfECixiLu8N5PYlD5qCOr2oKWqoqBOOEM9GleLcOxIACR5oY5ydIlJQuRPCWNCbL2tDS8inPpJmEvArNTjSpO6VoL2AShhiqqX/KYAKXY+XfB8BslMb4hGhm3lTubHE5Xk; 4:IZcDK/pOPIj3PrXE35hmK7F3CxxpWJ+DVQEio/TQafBXsFUjAVPVKcOPq4OJ6FCM0CbgXfYCMxXBVkDTfkDTQGAXY3iBworjPjgplqEePAxTnAFCLYFEX9SnvM6wLu7CGyLTdBpOD8XzlNW1pnEreH4FVEtJpzfYe1Mlo1vXg5AXHoHQZxW1bElWqbh4y0rcIXc0RTbPESWxudoHIV/C1KC4oveOP/LJcAlkzoKfQtMsDespf/bdGVIqCzbjztM3X34gvAKty2jzZK9xFZ6VhHaMmFt2p3M6QEYMgmuvCY2RgzpvfPcZH1rNb8j8yvmI X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(166708455590820); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(93001095)(3231023)(944501075)(6041268)(20161123558120)(20161123560045)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(2016111802025)(6072148)(6043046)(201708071742011); SRVR:AM4PR05MB3492; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:AM4PR05MB3492; X-Forefront-PRVS: 0539EEBD11 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(346002)(396003)(366004)(376002)(39830400003)(39380400002)(189003)(199004)(52544003)(24454002)(43784003)(377424004)(68736007)(478600001)(7736002)(39060400002)(33896004)(386003)(4326008)(16526018)(3846002)(59450400001)(606006)(76176011)(6116002)(33964004)(31686004)(53546011)(105586002)(25786009)(8676002)(52116002)(966005)(106356001)(64126003)(6916009)(345774005)(65826007)(81166006)(316002)(83506002)(65806001)(8936002)(6666003)(2906002)(37036004)(81156014)(229853002)(97736004)(2950100002)(16586007)(3480700004)(65956001)(16576012)(6246003)(236005)(5660300001)(512874002)(86362001)(58126008)(66066001)(31696002)(77096006)(53936002)(93886005)(575784001)(54896002)(6306002)(6486002)(84326002)(46492003); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR05MB3492; H:[10.0.8.11]; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: atech.media does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM4PR05MB3492; 23:z1Yu81kNpramTmg6Oh/et5MpgM89nsOz4GzbU2M1K?= =?us-ascii?Q?WlBcA3XWPKQiDLFd76U324AWTK3ffid3eYM38StB9fZhpHU3MZAEKtbRMSRm?= =?us-ascii?Q?Af/oTm1qKSkZ+rwNlHx6ylL2QczmyyeGlFGh/hjN4hBrSiQIOqtXVzjYvsVL?= =?us-ascii?Q?9gOGFHKh0s3BY3zvtwAVo0yXfPt5YZNPn4b/pGQsr3tN6Ivj054EMFfSiNIn?= =?us-ascii?Q?XqZAJDvf41rNfgaMlX8moINmgW71qGVn3nUHrpergEsDYC8GDKap+Jdqmrp7?= =?us-ascii?Q?Gk/yghLYgzugHxSjvklp4BVmTe8T3xwvaq62H7X6Fd4UGayRKdQ4utxAeeKl?= =?us-ascii?Q?s+5CmchDJrsY+a7u+QCZx9Vv1sJZKmSDyNjR7XHs1+3RaTdtdwompo/r58mi?= =?us-ascii?Q?2Zk7iLkcHy2NIDRyVNxAfCYcaML3l4xyHHp381p2lYv/jMzfss4bGeRe8Jvk?= =?us-ascii?Q?xFuYpX59nR4Q9HOUBxrF0DipSfSxDfO0TfV5pxMq18SzBq/05694o3QrlzGD?= =?us-ascii?Q?kgZW6YgBiz+NWKrAC02P+c/xAnF11SXVrZVfh6xxU/fRrRFLhymiJTra/7c6?= =?us-ascii?Q?Janky4bjYyo7J/3c6Coa+j+r6AayguOaW1yfrkBG6146F4oRFo83Je3PsDBQ?= =?us-ascii?Q?GrikZBWthpO3MhKznvDUbz4DhvTFb64CyaAAXe0fXBUzc6Ccl1LJ/Uhi408Y?= =?us-ascii?Q?Yt2kppdC0UqPn23yeG68Lxzxz7Pke7UuHv2Rs+zwbyES6rnm4hKXL/pI1pwM?= =?us-ascii?Q?Y+gTEyl9uTDuD3oEDSLkZOvGfZWXSYznjO/LxNAiKER1Oy9Ev5l1cNYZ+zbv?= =?us-ascii?Q?cwYi/D9+tz2KA5Drp66UqswD7xuSWYnnGW9hgS2PZ/JhmRLdXJ3pYlD1OP7d?= =?us-ascii?Q?WykVxfDzcuaozjmkeIZzEz6W2rw01Mp3e3WcbtaYKbsFf10u4HrUdr4NajQc?= =?us-ascii?Q?M4lJqTVzMr3aaUKfJCSHMsQfCA1A+WGxm0XNKIKhTBy1prqA7B1IW0H1UKTz?= =?us-ascii?Q?c4uFLoIn25zKRAvCExG8UOPWL3KPov86QWWlxEFPALrOGotYUoOcB05J7+b5?= =?us-ascii?Q?5UEMxw+HBH/+NB8VQcZtUf6l/wiIskvUvZfdDpqvAgxXHfdjz70INipcUKNs?= =?us-ascii?Q?qbb7H7aIctXaGTUXzZmDN8ZeJ5uOnX1c4QbFOqodOj4gToUkdV4Gg5iTrueG?= =?us-ascii?Q?QBCRzV4wodT+lpkvYDEY8+BSw3c8dut0bd//qZbBCfEoJlAJfauz0HYa6E71?= =?us-ascii?Q?J1Z/9F1AC/mOLN1hBsKedLSuYFpYHP90BGfGPjnFMYlBiacAA3lzOqzwXVEU?= =?us-ascii?Q?//0k14HPNrbm7mh2BHnXFi0HJlw3MkMaCYxpixB7/fxIMzT/6EkVhI8ND/7t?= =?us-ascii?Q?9SLY9MbNrtkVFKwIH0P3hgHrh928aIVG9WmcreSyxryBZNC6Zg5tIBfDQbz1?= =?us-ascii?Q?Q15Ee7D3z/RSvMPAxCeQ3AbetLJwLhTGcUNs2tmBIY1BdUGa1M0TEGk1cPeu?= =?us-ascii?Q?2+v7q7SXQ5WB3jDEEEU1cNPhh8BvC2o6C9KYPL0q7cvIi5YBmIX+V0B2NkTt?= =?us-ascii?Q?lBFb/eDM0FR1ztt8NzSOUBrU2rB7dDvV7z3TS2uBRkFeozzio8WYFrczjPrL?= =?us-ascii?Q?4qs/TFXJJzpQBvq9eVwY4wdDatGMzHuajzOXJ6ePmk=3D?= X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB3492; 6:gg2w+kcmwEhz73ZeirG2OTyMNNDOcN9KKcHhHKMdwsSlQ+TgtctlQvRDZSXUuCapQuOywqF8fQs4npDqJT303lyIuUj6EEPVs5F5Chfzf5h+oS6eYC/k6totmX6pTfsI3stvZuHQ5EGy5K7pFJHl9f4O9O5Ci4IZqSbE17uoLndmSoDwXwootnk94megapAYsDYvaHIo8kmhJ9ckzoLiAN+g2JQvgcu+iBoT/8c8GDr724+x+F9tWWqrh99OqkmiUTMNUb5APtpHV9tRbzI04GW/UOOzv5qDCzRMFUVcbuqwI0bqVR/Tp+q6ri4JSUgpQR5Q9jRGLwEaD4QPONAR5GAQgZojDCjz9oSMthA/ZGM=; 5:CRv6z7l3BZ7ikvGd06ZYTZIvaU8h41biRx15RiY7DvSexjkjc+84um9nSa1kvVNSU35vU9SYgSHD23rzE+ePQ4ThswnXCcCYvXcfc+DS9ULUa0ZZPHpeQ33TO/gm2yHGLsnHV3mjpmRorzvi6GLY/b17Y38vGeqkbN6ZHRTW/7s=; 24:yGwDFABA1egnmR/J3WFWCz6G7rlVJF0VWNZ4gupDJFhBuh5u4TMXHjB0ewb1FQpBxBgDWiwLFy1IOiC6pxNeKqElVWQNBY/uTLY5IXYTF9Q=; 7:JI4IFy1Pvnz6SFmZBGhP7tSdfkOCaFOIkWbr7vbPSsZge4wQfFHfFiCR9z4bsUkzp2EheiIzjb1eH8KEoIrkD4XkubfPGnc+J/0Fof5QG+Qopm95eEqWg0mqUgndbCS/MqOVlLcOtTUExoeCVFoofyQmGCpkksE27V5jJpoxq3UfE2y5C7Z5a0sSlSzxqrvf/3q/fh7eYY4RBQRIPpdomxp9bwXCEe5jBAG9hBOT0VZDjHdQS146ZPYXjReM3lZI SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: atech.media X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jan 2018 22:05:08.0207 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6e5e5521-0e0a-4824-2055-08d55163b87d X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a8f6edf-720f-4e3d-b767-1360e39a8cdf X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB3492 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 22:05:13 -0000 On 01/01/18 21:05, Vincenzo Maffione wrote: 2018-01-01 17:14 GMT+01:00 Charlie Smurthwaite >: Hi, Thank you for your reply. I was able to resolve this. 1) I do indeed open one FD per NIC 2) I no longer specify nr_arg1, nr_arg2 nor nr_arg3. Instead I just verify = that all NICs return with identical nr_arg2 so that the memory is shared be= tween them. 3) I properly initialized my memory, my failure to do so was causing me a l= ot of confusion, The resulting memory space is large enough for all the NICs, and everything= works perfectly with zero-copy forwarding, great! The only thing I am still having trouble with is the ability to simultaneou= sly trigger a TX and an RX sync on all NICs. I have tried select, poll, and= epoll, and in all cases, RX rings are updated but TX rings are not and TX = packets are not pushed out (this occurs using both native and emulated netm= ap modes). I notice the documentation says "Note that on epoll and kqueue, = NETMAP_NO_TX_POLL and NETMAP_DO_RX_POLL only have an effect when some event= is posted for the file descriptor.", but the behaviour seems the same on p= oll and select as well as epoll, perhaps this is a linux-specific implement= ation detail? I have also found that all of these mechanisms seem to incur a very high co= st in terms of CPU time (making them no more efficient than busy waiting at= 1Mpps+). My current approach is as follows, but I feel like there should b= e a better option: for(int n=3D0; n web. https://at= ech.media This e-mail has been sent by aTech Media Limited (or one of its assoicated = group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). = Its contents are confidential therefore if you have received this message i= n error, we would appreciate it if you could let us know and delete the mes= sage. aTech Media Limited is a UK limited company, registration number 5523= 199. Dial 9 Communications Limited is a UK limited company, registration nu= mber 7740921. Viaduct Hosting Limited is a UK limited company, registration= number 8514362. All companies are registered at Unit 9 Winchester Place, N= orth Street, Poole, Dorset, BH15 1NX.