From owner-freebsd-net@freebsd.org Thu Dec 28 23:14:56 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31E1BE87649 for ; Thu, 28 Dec 2017 23:14:56 +0000 (UTC) (envelope-from charlie@atech.media) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id F32AC70C80 for ; Thu, 28 Dec 2017 23:14:55 +0000 (UTC) (envelope-from charlie@atech.media) Received: by mailman.ysv.freebsd.org (Postfix) id F28A8E87648; Thu, 28 Dec 2017 23:14:55 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2294E87647 for ; Thu, 28 Dec 2017 23:14:55 +0000 (UTC) (envelope-from charlie@atech.media) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0073.outbound.protection.outlook.com [104.47.1.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C4C070C7E for ; Thu, 28 Dec 2017 23:14:53 +0000 (UTC) (envelope-from charlie@atech.media) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=atech.media; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=jLafLtGj5a6oZ/NxEFEG8wXMbFML9mZjPLWXcMYoKzM=; b=Z9SXw1QffQdGjlzLxmdefxaRUTCrqefM/vnc274COx2M3REh6G/+og+OzxIM7VpdWkTH8jho32flnc/9OK71N+EDouy7NVo0Mi6uSF/Ok5uguEStpeX2UYBbd2p1YoinAknfA7n+nTna8qEHFRjE0VG1BAuJUQpzMqfG1IuGWc8= Received: from [10.0.8.11] (185.102.133.45) by VI1PR05MB3501.eurprd05.prod.outlook.com (2603:10a6:802:1e::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.345.14; Thu, 28 Dec 2017 23:14:50 +0000 Subject: Re: Linux netmap memory allocation From: Charlie Smurthwaite To: net@freebsd.org References: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> Message-ID: <682a059e-130a-99c2-b7e6-4a865098e965@atech.media> Date: Thu, 28 Dec 2017 23:14:46 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Originating-IP: [185.102.133.45] X-ClientProxiedBy: VI1PR08CA0097.eurprd08.prod.outlook.com (2603:10a6:800:d3::23) To VI1PR05MB3501.eurprd05.prod.outlook.com (2603:10a6:802:1e::31) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e05b2625-bc4d-4325-62fa-08d54e48cbc8 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4603075)(4627115)(201702281549075)(5600026)(4604075)(2017052603307)(7153060); SRVR:VI1PR05MB3501; X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB3501; 3:Ghmg0yShaAGc1XC9CJXmEmlSadoyamLXhofgB7o4+qnduRrFf4e30qM8qHnbNrDPiefI5uEd/WWSA39bKAYgEWnK/z5GV8iwJN63ritkh9Nb6ZCOC1is1k46uYFYIw0m+eaJd8jyTcfb1IHorxHLyzrDPFbr3dREGDLD86LTzAmga1lpf64Ox9hptJmQ812y1VemMPNxtLsE3Ar1JwAbiD2p+ZVmgfoBCs4hAZOKDxx2xWVA1GaClQx2emVGKqn8; 25:XeaLhRsEQzXqTnaeg6C8Q8DF2L+FKhC2HdQeNUBee1ZIstT3zNcQsevVBxW1kwlc5ri3fcJDQ2hkoNKt6RFvkSJEZtDvqwh41ge2mzXwlNYOiyQA6IkS7Pt5NmBJ37m8u3GArBQITaPKr+Tn3qbrkEtt+RJhOsEvIzT16BcMQ/Rkcg26O4GHAbsK35Gs1KjZEa/uGITi3iyMVdUW6l6bE7V4SqMcnexf8AOOMlQXEsoGvSJvCllhh1ntDhD5DqzKhhZCvhQQnkfaa/if8i+Hy0/SXfhyVQo1geZRzS/+Oe/WY9RcOsNVuUuk2Vudi3LEejLvOUnkZriparO2p6TquA==; 31:AQH9Cm+U0yVqv0jO/czzh4Vqul7ylAYrLd82W+OZMHsO1ZoE9qW7QmAXK0YdXPXOB96lyW7zjhAjQZLbbzzhuLH38SRGKTD61JskEkJmv9AnoZyLhkJLJhuPxp/tt/mM8LRjYhVCPBUvxIYfAkJ1L+SKsEgbQEIYUx0oAhEsydDwTJpcCiiPoKOblk89yL+AfR2r0DUQ/PAKU0/5Gv00xDXC+xaz1zyctB1QnnDJXR8= X-MS-TrafficTypeDiagnostic: VI1PR05MB3501: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=charlie@atech.media; X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB3501; 20:s3vE0ZoDtyQd1O54M2lWMPBuvpMAQr2IrGGIgRwTbZH+RqgP7u/qku/nMl3JMqw2obcjuRiUprKkODMiEtCeNZFOXkF4Cq45gP3qY+rW9K1AWqUNEoiXpwxjKWlmlBoxrNe3COw/NIN7aRFYU8P/frQd66ElebEqoMtSxH/9AKl59B/oZjKDW097x7jI4f6n1h1zyCjxptD6qyhCanXPdu7+ZxI2JFQsLj7sCqsDxlOgEuFSDoiK6bloBG9Mqzib; 4:eeCOhu1+Q4PxWE3PuX0i+/WY/LzWFOxnXZ+8g1viUlT0QmGAWbzXuuSMPb70ksbqdNGQqDtJyyIxJ6kQP1KCHtNXBl2OAdim2SbfTn7+oZiXnl9V0qeWwzT7Txx7u3DiNXi4/OLpySVnaCCxH3rNiyadm6KtKL+B+8xn7SBFs+4dk6KGjaZFnhyKXFrIR8AbyXDfRys1ZNmZ1TmdszcFxKaHqaFV3nuv8+5isIFrFz/SZpv79pqR14LwlAU+S660V+LFZnB4gCjU6iZARjx35Q== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(8121501046)(5005006)(10201501046)(3002001)(3231023)(944501075)(93006095)(93001095)(6041268)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(2016111802025)(20161123562045)(6043046)(6072148)(201708071742011); SRVR:VI1PR05MB3501; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:VI1PR05MB3501; X-Forefront-PRVS: 05352A48BE X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(366004)(376002)(39380400002)(396003)(346002)(39830400003)(52544003)(24454002)(189003)(199004)(7736002)(105586002)(53936002)(50466002)(64126003)(6306002)(2351001)(6246003)(106356001)(86362001)(16526018)(97736004)(229853002)(25786009)(478600001)(966005)(6486002)(6666003)(81156014)(2361001)(31696002)(31686004)(47776003)(8676002)(77096006)(65806001)(65956001)(66066001)(65826007)(2906002)(2950100002)(6916009)(5660300001)(6116002)(3846002)(575784001)(305945005)(2486003)(59450400001)(67846002)(68736007)(23676004)(76176011)(58126008)(52146003)(81166006)(3480700004)(8936002)(83506002)(52116002)(8746002)(33896004)(386003)(16576012)(316002)(53546011)(46492003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB3501; H:[10.0.8.11]; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: atech.media does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA1TUIzNTAxOzIzOlhnVWsra2V4djhwcFNER01HTTFWUEozTWtC?= =?utf-8?B?eGNRcjdPVy9HVEEwWGlKNWhZb2kvL3YybUFIZWNEbmIyeDh4VFQzUDNsT3B6?= =?utf-8?B?YWwxb2U0RGpkN3JnVzRoaW9ZNFp0eDN2NmJGcGpJVnhvRnNlaS84bm9lMWpQ?= =?utf-8?B?dFk0UXVnc1lpcnJrc2VrR05CVE83clpMZnpBOGNMTzIxTW0ydGQ4clUyc2xo?= =?utf-8?B?c2dWQWFnZU51VGFsbFBxWDYxYmRxTmNiMEJoazVwK1FMSlE3am9mc2c5NTZZ?= =?utf-8?B?RW1DU1U0dTB3QTNpRHhwZjJZK1dOOUROaFlmL3k5ejJqb1dPTTVYS1pPWEpa?= =?utf-8?B?YUNKem1UWnI1YlQ0MlJEQU9rKzBkUXVMQVdJeGxxVjVTUDFqRGJvV0ZVbWEy?= =?utf-8?B?VWlaQnN2R2srcGp5MUxjbXArbWpBV2ZDMTBLVEJZZlU2cUcvODZmWjZLQlhq?= =?utf-8?B?YWpTcEZyMjFFbUdDWXVva2UzUDlJbHNPd1Q5V2N4VXNlcWcvTGZuOFJyR2pK?= =?utf-8?B?b2krUzFsVlJtSW52R0RRcGFlWXdJQ09Vclk1ZGZjMVJWQzUrbC8weE81ME1S?= =?utf-8?B?a2ZiVkQzNFBjZ1o0eHVjVWUxOFN5Sk9RQm0wWThXSDdDWTJ6TGdlTFduVFZX?= =?utf-8?B?LzdGbEl1eGZnMHNHblhVQzBYYUFQZ2ZwdzJpZ3VkbjFkek95bVJZWmVHdnpj?= =?utf-8?B?eGdhc3hGeTVMZmxGM1NnZ3dhRW9aQlZvVUtaUkl5V2paR0E3ajRTREExMHBv?= =?utf-8?B?L0FNOUwvNlpWeFo1Z3ZWc0EwNWVqMTViVVlnQ1BMb2F4SVdnalhkMUlhTGxK?= =?utf-8?B?Z2poaHZFS2pBVDhLVHEvb1NoRk11VzFhWFFFZjlVYUVoYldzMGhNcGxJdElD?= =?utf-8?B?Y2hmenRTQ0JxOTU0czdHaWYwUTZBall0a0tKUUEvckFlUXh2R2NSUTZtckwz?= =?utf-8?B?ZG1sWUUremlvcFNtV1EwK0Z2WTFVcnFvWUVpSTMwRisvOUhyYW9ndlFWN3Fs?= =?utf-8?B?YjQwWXNsYTFGc3RBOTdHaHhiNGYrVzF0RXl2U3UwWDBqeTVoZXlyWVdtN29G?= =?utf-8?B?MVB4UW5kWEtiZ1VWOEFnbXhwTEtaZjgvYlRINjFzdk5YUWdic29CTXJFQmlV?= =?utf-8?B?bTdPbXk3UVUwRVBvV0E5YjZKckFsaFRuUlJBM01xaGpuZDZLdkxVUUh6ditV?= =?utf-8?B?UUJ4WTNPZVZkU0d0bjhmOFZtc00rQTVHT0JSeXQrbnp4aTcrM05wVGF2RHU0?= =?utf-8?B?eXo3QUl4VWJvR250ZWd2dlFrOWtxVnY1TVlTLzArK2VuSm1HNTJERWJyWDl4?= =?utf-8?B?aFpOTFNyemxGZXByNjViSFczVUEwcEk2dzlNZmZHYmZWdFFFY2c3NjBPaGpE?= =?utf-8?B?eHZEM0l3T2lUM3dab1d5bUdLb1h1c1NiSDdHU1NhTUZWOG8wMnA3UENvYk1C?= =?utf-8?B?ZDdXOHlwY0wva21KdFgwczdQSTVBdmd2RzBuM1cxeW1pRHJnWnVYRGhSdFYz?= =?utf-8?B?aU9NZXRsWm82eGJmbHpVWnBVWTRhdC9hWkQ1VHR0K3d4cUtGSFphNEI4VlZp?= =?utf-8?B?SUVQVWNlN01rWW1xRElCQ0pMS2c0RWtoYms3YThTaWRpRG5HUWI3MWRPbzN0?= =?utf-8?B?V1luMXFFQ0xTb3dXMnBBUll4bzJjaFd4NEV6YXBJbHhpNC9tWndmTEhyb2Vk?= =?utf-8?B?WGZuODdzeTdHUlRVbG9kcjF4cnlnNHQwUktiTlQ3ZXZRZzkwb0ZSZnp4Szlw?= =?utf-8?B?RmFsN1lEeG9DdzNyZG1JbGRDNHA0eTFUODQ3L3pUZFlsaHN5bG1zTldMTHMv?= =?utf-8?B?NGJtYXRUdXVlSTlydXplcCtFYmFoSmd6MVJ0czZyaVcvQWlXNDcvSXR4UWdC?= =?utf-8?B?SzRKZU9lNkhLWlpzaWc0bitNUFhaNjNmZVVoOXczZ2t2S3hLSm5OM1kvekZY?= =?utf-8?B?bVAvVTJMTzJSdTd3MWorUE5qKzVHcU1PTjJOWGRBUmhNNU5lY2FHU0l5cnVV?= =?utf-8?B?QVVPUlhKL3UxRjZ4R0Vlc1NZR3NBQkh0cUpvbjRRSGVNNkFSeUFWMFZZUkUr?= =?utf-8?Q?93BwU1nXnrrWZ4Q6CU7goQuDZ?= X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB3501; 6:LmPPLiYVfBmIEEjN/Y9mr76n/DoveLRsstNk9ZYHG7OMZ7hRsmC7LdpcB1GncbiIg0fpyth0YBAeGqyXHpiPDJzssLvKdccoPprVqc1icPL62NMrRQzQHC5jE5/hoLvFgVKsWTBtKt+PM5x60JSiGEMmtFfvciCqdaPEVRbF6jFAGJgIBYV6gZE9LvV7Wnd7W2XiF95YsgEHzF4iwZGy3w2BkXeX26La7S7W6li/7Ppl3agKJnleDu5mc+gMqbZHMTp+kc5P9tpgHmxtctsAzLHufl0+7kd82k1rtEhfEcR0ApTQ/riuV11rDvZPTrQuCdzbxYaTRczltR9YM2+gCYS7VLTqXLHS+Ad0ke0Lxu0=; 5:TOGTbZN7tV/NOf4bBUVnsqzkYpYouLQ8YMsQ+KNTW9Cof96clGQner/CM3cBFOMPjjcvGZkPQkazZiljSPeLlUO1Z9LGLIJDz4+surva/tHF2+LUTYDrisLxAHGe27j29Vlm4+cywn2Z3zhVbQMQvzlA0d8Bc6/tBi5HjNo8J7Y=; 24:bHy178MSWGdrmBu1V6IY+jlvfBCchZo4vPdDcza2rLpFh3nMl50xrlLFy56P+Ct4H6MP4/AOqAXKi5IzkK2d84Nz/cVpv0BrWdZqDQk4xag=; 7:Etkj5Nh/7GKjYVqGCQIQx5k9n4MK6hKhRB++Z19aAtjg5jNySw0j4mPk/ZDPqhXlJP9UeKQ9dpoZkNAKYhpa7E2OLg+sYnOCyryEV1f/dexorfrkTNRPB5+1GYQG++acqM3g0l863cT1BTBkzbEKGvhMXUO8tqPmQYVkTeyHuQN6r/RWr2eylQ4NATmOcGjh1EloQ+KsG41En0W4JzYv0opvEjBiZtOL83RIvd5JkH2CWZEGlC9vT8PjqTOTRrty SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: atech.media X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Dec 2017 23:14:50.3968 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e05b2625-bc4d-4325-62fa-08d54e48cbc8 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a8f6edf-720f-4e3d-b767-1360e39a8cdf X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB3501 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Dec 2017 23:14:56 -0000 Hi, To answer my own question, I have resolved this, there were seemingly 3 errors in my use of the API: 1) I did not initialize the req struct. Some uninitialized data were being passed to the ioctl. I feel silly. 2) Values greater then 1 do not seem to be accepted by nr_arg2, setting this to 1, or to 0 (which resolves to 1) works. 3) When fixing the above, the memory size is always 343019520, plenty for my requirement. Thanks, Charlie On 28/12/17 17:34, Charlie Smurthwaite wrote: > Hi, > > I'm just starting to use netmap and it is my intention to do zero-copy > forwarding of frames between a large number of NICs. I am using Intel > i350 (igb) on Linux. I therefore require a large memory area for rings > and buffers. > > My calculation: > 32 NICs * 2 rings (TX+RX) * 256 frames * 2048 bytes =3D 32MB > > I am currently having a problem (or perhaps just a misunderstanding) > regarding allocation of this memory. I am attempting to use the > following code: > > void thread_main(int thread_id) { > struct nmreq req; // A struct for the netmap request > int fd; // File descriptor for netmap socket > void * mem; // Pointer to allocated memory area > > fd =3D open("/dev/netmap", 0); // Open a generic netmap socket > strcpy(req.nr_name, "enp8s0f0"); // Copy NIC name into request > req.nr_version =3D NETMAP_API; // Set version number > req.nr_flags =3D NR_REG_ONE_NIC; // We will be using a single hw ring > > // Select ring 0, disable TX on poll > req.nr_ringid =3D NETMAP_NO_TX_POLL | NETMAP_HW_RING | 0; > > // Ask for 64 additional rings to be allocated (32 * (TX+RX)) > req.nr_arg1 =3D 64; > > // Allocate a separate memory area for each thread > req.nr_arg2 =3D 10 + thread_id; > > // Ask for additional buffers (256 per ring) > req.nr_arg3 =3D 64*256; > > // Initialize port > ioctl(fd, NIOCREGIF, &req); > > // Check the allocated memory size > printf("memsize: %u\n", req.nr_memsize); > // Check the allocated memory area > printf("nr_arg2: %u\n", req.nr_arg2); > } > > The output is as follows: > > memsize: 4206859 > nr_arg2: 10 > > This is far short of the amount of memory I am hoping to be allocated. > Am I doing something wrong, or is this simply an indication that the > driver is unwilling to allocate more than 4MB? > > A secondary (related) problem is that if I don't set arg1,arg2,arg3 in > my code (ie they will be zero), then I get varying output (it varies > between each of the following): > > memsize: 4206843 > nr_arg2: 0 > > memsize: 343019520 > nr_arg2: 1 > > Any pointers would be appreciated. Thanks! > > Charlie Charlie Smurthwaite Technical Director tel. email. charlie@atech.media web. https://a= tech.media This e-mail has been sent by aTech Media Limited (or one of its assoicated = group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). = Its contents are confidential therefore if you have received this message i= n error, we would appreciate it if you could let us know and delete the mes= sage. aTech Media Limited is a UK limited company, registration number 5523= 199. Dial 9 Communications Limited is a UK limited company, registration nu= mber 7740921. Viaduct Hosting Limited is a UK limited company, registration= number 8514362. All companies are registered at Unit 9 Winchester Place, N= orth Street, Poole, Dorset, BH15 1NX.