From owner-freebsd-net@freebsd.org Thu Dec 28 17:34:14 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C226CEAABA4 for ; Thu, 28 Dec 2017 17:34:14 +0000 (UTC) (envelope-from charlie@atech.media) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 90E093EE5 for ; Thu, 28 Dec 2017 17:34:14 +0000 (UTC) (envelope-from charlie@atech.media) Received: by mailman.ysv.freebsd.org (Postfix) id 8D000EAABA3; Thu, 28 Dec 2017 17:34:14 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C998EAABA2 for ; Thu, 28 Dec 2017 17:34:14 +0000 (UTC) (envelope-from charlie@atech.media) Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-eopbgr40058.outbound.protection.outlook.com [40.107.4.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D8AA13EE4 for ; Thu, 28 Dec 2017 17:34:12 +0000 (UTC) (envelope-from charlie@atech.media) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=atech.media; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fHYUqgdUZLGK8/369CP1I3Qv7aKqCCLhdHbQNY0vZtw=; b=CyyrMTUoLeUdu2IqqeCq3Lqvutf9DF/HzMCzSCVcZFfo/ak8j3hZt/1DzojAuAayC+A4bxwEoulmbBCOsS15JTOwt4RHDFZr1gBsmTaC5kmeYr9KvV539Hgcq58ocjnCobBJIi7Me72/mSecn9Gta4+GN5tVUQ++Ufteq6E1984= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=charlie@atech.media; Received: from [10.0.8.11] (185.102.133.45) by HE1PR05MB3498.eurprd05.prod.outlook.com (2603:10a6:7:32::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.345.14; Thu, 28 Dec 2017 17:34:09 +0000 From: Charlie Smurthwaite To: net@freebsd.org Subject: Linux netmap memory allocation Message-ID: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> Date: Thu, 28 Dec 2017 17:34:06 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.102.133.45] X-ClientProxiedBy: DB6P195CA0011.EURP195.PROD.OUTLOOK.COM (2603:10a6:4:cb::21) To HE1PR05MB3498.eurprd05.prod.outlook.com (2603:10a6:7:32::16) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8c723cae-cded-4a60-67ba-08d54e193435 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4603075)(4627115)(201702281549075)(5600026)(4604075)(2017052603307)(7153060); SRVR:HE1PR05MB3498; X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB3498; 3:f/KoSpUQFgQqxAQVNRKHH92dsiOSXjQLcktXvC8yce/w7MFjNkaAfCGY2JPxQ+3g0ehPtmQRlylOe1Z9Kvu9z100CuqeBgTA4OaEUsggFPvdD2oqGvN+UrgPFG/fCyZmqAcF4y+3+sb4nGgqkvptMGRXAgJ/M+Vzgz3f5gkVsmTsGTLo487fBqHcmJxr3LhSeQFuqbdhvppK+RGNB3NhlhiaTvwvH+lnrIcFJwYDAeaj5wNUMdFQcEAV8RmWMOSp; 25:yGwmbHvTw9dYbv19+yZTtBEvEcwyi5E9nNiGOprGhUuicyvAxylleyZgvwOu94k8wVBI2CMTbAiYkA7V9qoJutM/mBEzTl1SlPZ4+Rty19YwrOsnuUMNa0Qx6IgVOWKMKatJsuxShZlKYdD5pMiyIbDK2QN3Zsw+UTKFuH9S2zV6GrMho/mRCbfwwv9OL8baqmjd8A+bX2GtvuBzaNSe6ebTCa4zTjE8tWS09DcYZE5n1Io1O4POU7nkiSZD1zmZttBjjMuuLN0rE3vu4gR6REowpzeufRosuRH4viO+1D3x4cpMPw/QMhTJ5XsVF2tPPiDF7KI42mlq5f9v5pMK4w==; 31:jnIHilkOvwjlqqwJYVtFbbvyMnIMb49XOg+j6Ac1sIEm/bRFwXvC2iUfQTGm+ufM1QdUV+k+TVaE3mNaLVI3qogilLUCW0skof5j+6UMk6E4qTkrdEFQ5hG+YQS4BHCnQGGbHanWYJeQ7rIES/Tji/wpYxmLK6RaY8reJl//0uAqMXgiqNreIA6JrJyDwwohRQynXfCmsvz7S/9x/+ZNC6ajfLCnkGezCkQGL54hJtk= X-MS-TrafficTypeDiagnostic: HE1PR05MB3498: X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB3498; 20:oY9NUITQ0eneBWiz6+NoqkDlJ/MLjqWeWOiQp7eYvL81CQRSmRfIRfRtrSXkxNDodwioHQVsJQt3Wt6crPQvXILT0SPU06MNS46H6EWgSMfbnJTER/c5JKnOylPST4CQXQoI6NGFZHu/CvXkAY8c+tddw6we9VYr/OyDmNaWpmotwydfOMrqVg5Aqv49dqxKFCwLLXQQmVvDOmMW8AD/IQpiPjM55JyWDNzDGdPdE7QZS7eOuBIKY/UaMt/PgUth; 4:GtvdIjXK7GuNg9a5fFPuW+u4hDPjNWFW3yBvt50ih3rVDMg96USaGPXgcGoyPQ5XNdpLG+Mq6WaEkkx1fCaBf1gNM5u3E7Ihbp1TDAXHusn74PN1xuNNsVxq7e3pWDrVVcvi/QqTpr7s86gENB0OZT9S94zOz5IiwCBg+hk+XYj6GcSOfcP8OuQNIquTNzUzuGZLNQ6FF51LC2/qTwmUxrWar2ZZgbpra0upQYBxbsyVWknwd/Rp6Zk8AUqc7RISLjZ3qBzFHhUlyzYR2R5uSA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231023)(944501075)(3002001)(6041268)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(2016111802025)(20161123560045)(6072148)(6043046)(201708071742011); SRVR:HE1PR05MB3498; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:HE1PR05MB3498; X-Forefront-PRVS: 05352A48BE X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(376002)(39830400003)(39380400002)(346002)(366004)(396003)(52544003)(189003)(199004)(6666003)(16526018)(106356001)(316002)(2351001)(386003)(3480700004)(66066001)(65826007)(6306002)(65806001)(53936002)(2361001)(64126003)(966005)(5660300001)(97736004)(81156014)(305945005)(31686004)(23676004)(8676002)(2906002)(50466002)(83506002)(67846002)(81166006)(6916009)(105586002)(7736002)(33896004)(8936002)(58126008)(77096006)(16576012)(47776003)(8746002)(59450400001)(2486003)(65956001)(68736007)(478600001)(52146003)(6116002)(3846002)(25786009)(575784001)(31696002)(6486002)(52116002)(86362001)(46492003); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR05MB3498; H:[10.0.8.11]; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: atech.media does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtIRTFQUjA1TUIzNDk4OzIzOjNnV2MvVlhud2JiVDZreURUWDR5WVRwTWEz?= =?utf-8?B?U3d0Q2s5K0Q2bUZUbUZYemxCQWo2UXFUQnA1QndoaGVuMkRBQXdoeGJ2ZWlj?= =?utf-8?B?UTRvZHdQZTlVTno4NW5UYzMwVWxHWjdnZDhYNG9ISzdCMkk3UU1yMks2RllU?= =?utf-8?B?OWJETXNmUWdlbHFNZis0TTBWbDk4WkVVZ053bEE2dEgzL3huT2ZZUVlNNGlZ?= =?utf-8?B?R0ExVEp5VE9VdU5hekhSazBTa1FIUHFobFMyUk4wWS9JdjF0MU10SCtWanhP?= =?utf-8?B?Y3pCTjNwQktFZldCZGt2R20xempQOUNoTlkvK05JUi9BNW5vQlByTjd2Yyts?= =?utf-8?B?YTBvK2tsT1hGOFpDZzBEU1orQnlxczhOSnJaU1JnQ2JDc0dvRTV5dlMxejgr?= =?utf-8?B?R2dMWFlQaG1UNEJYUkJyV1U2clY0anlTNm5ITlltbVBlcE15OWhCbUloekZT?= =?utf-8?B?b243bExuWHg5c1NaMUk4ZTNCS1lkYnQ3dlJrT3V1RkI3T2owYm9ndUo3ZWp0?= =?utf-8?B?V1hQTTVaVDJJV1hTc0ZrbmgvKy94c0NTK0QzcWxlYWJvYnJDY0lHS2lBUTZ4?= =?utf-8?B?YjE0Z25QYzdpdTRpZS9uSUNvSExQb1pDSjdVY20vSUN5UnY2dHpWd1oxWUM4?= =?utf-8?B?QkxMdDZKQ1UzZTh4NU5qSW1hOVhQODF4dTRXaml2TnRYd0RKM0MzOVR4SEtG?= =?utf-8?B?R094bUxUSTdQS1dnMUlkWVFFVHdwQmZYejlGdkgwa2txVHd4N281dkZzVWNa?= =?utf-8?B?OUN1cXd2WlI1Z0VpSS83NSszc0tJL3BuTG11cUljVGR4ZDhSUmZwbFE3OEt2?= =?utf-8?B?WFVkckpqbU9KT3hERFN6TVhuU0pBR2hTU2RMWEVWOW1SVEp2SEtFNXlwbmRT?= =?utf-8?B?SmV3OWJxbHVTb2tzTmFUbXU5M0hJR2hZR1Eya3BxTUgxSFR0c0dreEpVVWpT?= =?utf-8?B?TlVGaDgxZ0tOTHJXeXE0bzAvUE9kMjFpU2sxcGRpYWowYlJvS1ZjVkEyR2Vz?= =?utf-8?B?ZE1jdUFQVXU0OUJseFovNGdIY1NycjVtUjNwV1ByazMxRkFtRkJHeDJjVDVu?= =?utf-8?B?T2NuajFlY2Eyb2Y0dlJVSWZVamY0Mkl1dHNzcHJjK0JDL2VtdjFONlMxZ29v?= =?utf-8?B?Znphc1JnbTFKTXkvbjF3bjR0VTgvOUgzaDZHQjhjL1ZOT2hhamw1RUZJNzVz?= =?utf-8?B?QnJlNXowTThoVXBDdGNHeWtSNGgrN0VpMFVWWkV4emc1ZU1yZzMwZ0c2dVE1?= =?utf-8?B?TXlRK0s2LzNwL1JWT2Fwa0x2VzdraE5ZOGYzL3BnWndEbXFwT2J6Y0RucXV3?= =?utf-8?B?MmRscFZTakNxSGpYbG15Z3JORGhEY1FaRGtrQjI1TmVNQ2dtRXl2WmtCN2ha?= =?utf-8?B?SVV5RXduMEtIWlZvbkg1bXBQbzVURGtxaDljWi9xaEhwVTFiMEl6ZVBvWXY2?= =?utf-8?B?bEtzRGZFT3g0RkFnYXpyeHM5QkNNakFrdVdrWHNVMzRHOWlTQ1Q4NVBiUnNY?= =?utf-8?B?WmpaSmVueEp4Y3g0TndZTUw1TlpPNjVyOVJOdmhqcGtBb054RVJ0QkZLRmlS?= =?utf-8?B?eTR3ZGhmK0pGckY4eEU4YTZ0RE0rbTRkM0dCWjVveHR3bm1EYS9ZRXFkbWVv?= =?utf-8?B?dG5FLzZoaU0zUzl4aEhjS09xam02SFcrdU1QTUdmR0hlaFA1SG52VkladWNw?= =?utf-8?B?UUFSNzkrQUl3SVVZMVpidWRTWm42WWh2QldGM2lxNGJkei83ei85MzhHT1Bx?= =?utf-8?B?eTZlQzkvL0QvaEFMWnBPM0N0Sjh6OExVRWxuOXBOR1FoSERpd1Rwb2xVODdQ?= =?utf-8?B?VExuSWgrL2UwZkFqY1Jhall2dmNjZEdRRjhEYkJpVDNydHJxelB6eUFLWlZk?= =?utf-8?Q?YvsLSarb5tkEvCZ5N3OSyTCjjx3T1B+0?= X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB3498; 6:sfNCRZxxHAc7gH8jFjYcs0l61Ys6P+VhuErEr/I76S2Ai83UnxC1uQ+z0KbI1w/QnHVM8a6W+FiSNsWQQQ0UYe7a/qdxGXYYhXgugLh0FFFkz2Qf32WHYGPzyscwbAnQkXf2P+zPECRhmCS7JHmQFB2wEkfnHdiDFVN+aazXLcVT0Usvbt+oFP3Eh5aT397TrqvIKK1+hs0L5Lfr+yr9yAcuL1eugAf3HwJvJbTe4QvlJUYAhmwPkpdwfq/Bb4cRFYxJp7/FX7+mY97fS+J5iUkcMMlDmTaOGvrjG5mt9hNXCvBWZiy6dYCE7FEIYiQ67gw8TO0SZ30lc+doxvNBlc+09Ln4RgOADNSR2NObai8=; 5:CcxqxPZTbPpxTD1hmpvvB3n712NUAMP8mfk94PZu+g2DVVlMyI5cPKNAg3tMD12z1p3ByDrQ/6WrVXlpd9qS3eVaW8ctTLxLM6Fsg7bbqHOQjbArbkTsPdrExBMG0TJ+a8syJszikQ6bHcQMQ1eXHcwvccPozLb1oVh1UeINFkA=; 24:WpRIFKdDFMNfcz9LDheuANSaM9XfXxAMjksuvhug6H2HtYHBiZVquf9sSKXUs8rQ/jZOqpAto7saXV0d0TybiRFmp8SRplcXJwe441cEfdw=; 7:D4+TLkSx9f1X2S7hToLZipcTMUeSwolZgPId7zATu+MuIm8ZrHUjHupFqK1kuyE6M5IK6CTKDFWS5z26R4LTfebqe2E36+pCWyN/WUq424xSk+GyMmXms2/Lq6TeVHgUZ2CjiJSlJxDJXjkhcJzIUK6/3c/qByxYkkT3l2nVQStih7fO0a2yMrqXpvr4MuVe3VGbkbCk2jsY/hhnFel6zXX0LalxhWk0vz3IRA0VkNPLNIr3iOE1JsdZXEvuoKgv SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: atech.media X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Dec 2017 17:34:09.7826 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8c723cae-cded-4a60-67ba-08d54e193435 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a8f6edf-720f-4e3d-b767-1360e39a8cdf X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR05MB3498 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Dec 2017 17:34:14 -0000 Hi, I'm just starting to use netmap and it is my intention to do zero-copy forwarding of frames between a large number of NICs. I am using Intel i350 (igb) on Linux. I therefore require a large memory area for rings and buffers. My calculation: 32 NICs * 2 rings (TX+RX) * 256 frames * 2048 bytes =3D 32MB I am currently having a problem (or perhaps just a misunderstanding) regarding allocation of this memory. I am attempting to use the following code: void thread_main(int thread_id) { struct nmreq req; // A struct for the netmap request int fd; // File descriptor for netmap socket void * mem; // Pointer to allocated memory area fd =3D open("/dev/netmap", 0); // Open a generic netmap socket strcpy(req.nr_name, "enp8s0f0"); // Copy NIC name into request req.nr_version =3D NETMAP_API; // Set version number req.nr_flags =3D NR_REG_ONE_NIC; // We will be using a single hw ring // Select ring 0, disable TX on poll req.nr_ringid =3D NETMAP_NO_TX_POLL | NETMAP_HW_RING | 0; // Ask for 64 additional rings to be allocated (32 * (TX+RX)) req.nr_arg1 =3D 64; // Allocate a separate memory area for each thread req.nr_arg2 =3D 10 + thread_id; // Ask for additional buffers (256 per ring) req.nr_arg3 =3D 64*256; // Initialize port ioctl(fd, NIOCREGIF, &req); // Check the allocated memory size printf("memsize: %u\n", req.nr_memsize); // Check the allocated memory area printf("nr_arg2: %u\n", req.nr_arg2); } The output is as follows: memsize: 4206859 nr_arg2: 10 This is far short of the amount of memory I am hoping to be allocated. Am I doing something wrong, or is this simply an indication that the driver is unwilling to allocate more than 4MB? A secondary (related) problem is that if I don't set arg1,arg2,arg3 in my code (ie they will be zero), then I get varying output (it varies between each of the following): memsize: 4206843 nr_arg2: 0 memsize: 343019520 nr_arg2: 1 Any pointers would be appreciated. Thanks! Charlie Charlie Smurthwaite Technical Director tel. email. charlie@atech.media web. https://a= tech.media This e-mail has been sent by aTech Media Limited (or one of its assoicated = group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). = Its contents are confidential therefore if you have received this message i= n error, we would appreciate it if you could let us know and delete the mes= sage. aTech Media Limited is a UK limited company, registration number 5523= 199. Dial 9 Communications Limited is a UK limited company, registration nu= mber 7740921. Viaduct Hosting Limited is a UK limited company, registration= number 8514362. All companies are registered at Unit 9 Winchester Place, N= orth Street, Poole, Dorset, BH15 1NX.