From owner-freebsd-net@freebsd.org Mon Jan 1 16:14:49 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8B30EB09B1 for ; Mon, 1 Jan 2018 16:14:49 +0000 (UTC) (envelope-from charlie@atech.media) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 83DAA71E77 for ; Mon, 1 Jan 2018 16:14:49 +0000 (UTC) (envelope-from charlie@atech.media) Received: by mailman.ysv.freebsd.org (Postfix) id 8035FEB09B0; Mon, 1 Jan 2018 16:14:49 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FBD7EB09AF for ; Mon, 1 Jan 2018 16:14:49 +0000 (UTC) (envelope-from charlie@atech.media) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50078.outbound.protection.outlook.com [40.107.5.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA4D271E76 for ; Mon, 1 Jan 2018 16:14:47 +0000 (UTC) (envelope-from charlie@atech.media) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=atech.media; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Tzs1JB1B1LpdYlWn1NNOGP0xbD2fy3LW3btHkc8K/0E=; b=XgIZIh645BM5IMOLaYtcg2hdRRUjCAZvxgpoS0qaBALs59ofy8xzKmvO877LuJpCTRjuqLJOLxdgsW40j53kxFejlcSWVlzKKDnjyHhiYqdm8y87cWObDHh3K1z/fOSjPuS0CzOgoTYTWYIk0rj+YGjBUC0qz3BC8i88gyXQ0MA= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=charlie@atech.media; Received: from [10.0.8.11] (185.102.133.45) by DB6PR05MB3494.eurprd05.prod.outlook.com (2603:10a6:6:1e::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.345.15; Mon, 1 Jan 2018 16:14:44 +0000 Subject: Re: Linux netmap memory allocation To: Vincenzo Maffione Cc: "freebsd-net@freebsd.org" References: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> From: Charlie Smurthwaite Message-ID: <6c5de1ed-0545-31b3-d0e2-4258fa4ccf1c@atech.media> Date: Mon, 1 Jan 2018 16:14:41 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Originating-IP: [185.102.133.45] X-ClientProxiedBy: LNXP265CA0012.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:5e::24) To DB6PR05MB3494.eurprd05.prod.outlook.com (2603:10a6:6:1e::33) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8d8afe69-370d-4b75-1896-08d55132c56e X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4603075)(4627115)(201702281549075)(5600026)(4604075)(2017052603307)(7153060); SRVR:DB6PR05MB3494; X-Microsoft-Exchange-Diagnostics: 1; DB6PR05MB3494; 3:J1tFE9/VNvotgitpDVrAPu22yYq5EUSZiZSMOXoSOmJkvpxs847vyelcPhyQA/m5qmnN/+9xKSWz0aQD+/j84dDkep7RyKXgvBTFQKGVqOQd1DTBwxsIPfBsNfrbGkhhXbjhhMSizSFoEdR8Q7ow/UGNf7lC3tPsv3mxa8biDYMkt1cYxZ8m1edCzNqCw+JtYXxJhUgMMybgmK+l2YcVuhoJ3yCq7OB+JqBYcerggM61MyyD8nHiQ1lZS7vmr1Nz; 25:6/RFVmodZ9c+vD4gJvSkgHtmt1iGbsI1iHsu0NHP+mkkL2cb5hPdvEvu7F1dqBia/+C7q0WtIv3Q1lpclE/T/onrKmsUP4bZaaB+LF82WOjqFKHY1Ondw0wr7dJDgcE6Xb77RvdqB4Tz3XmDT0/WQDps/tlMFZwqehy1YPOg8z1SDs+xINBhBgnR+dZ0jwHhvpyJD+lOfac9IAYZdVvBZEchLKCX/+5IljAof3TlWiutRraaeS2Qu2IzEDnD2/jeE7gfs08YSaK1Npv7v6F1frz3hhGI6gxB4fLF7cq/eZJR+ex/T2u2EBYWs4jApQKRt/Qajj7bzxnqtr/O1GGxWQ==; 31:hCP7MK2RT6H2pkbf8rHNkKkWa0876/UkJmLAE+asrKju7b9Zdid4MfFo2FDrluusKtOMUkOSigWKOC1fLyOasxNQgWVi0AvU6XSC+HJ5/walc/LOpb6bfpt6753/rZoGH4tGHyJDYvP3ftBdRh2u4D9MFxmozYuSeIgSQVb/iIdMT1WT5K3kKfLLK5l+3cZpCYoA9cD7qVFllmpPl32N0bq6W+WzI6l6SLKnQrrxYO4= X-MS-TrafficTypeDiagnostic: DB6PR05MB3494: X-Microsoft-Exchange-Diagnostics: 1; DB6PR05MB3494; 20:Ftoxu3XBA/pYJEHJW+FQlacEBSPOiAtgMgb4dkV5hVO5aoYFZYQE7MtEFvYvODP7l4QJEkG2Bti2cwjOak6jsDYpcV5cOgPaoaLgkdvVhUv+m0QlfXhAmdRMk4jcBJCMtRvzLPx0leHZLPbfTzqe9hxBQ9UB5N493NhQwVm7u31ENtbSGybAaTkaiFQydZxRzoSGzUVej9y+CnyRbiIqJU/7ft4lyNtNLxBOgjCjfKafimeIN28kweTJDWsXYAvj; 4:t7Md5WmmKQZCooVVGVcznCUUDCIboZ9JmfizrfdL/VcRiLRwsfi0csgr0W5PWMiqV5d74/0tWJlhHawVKwr3Hsu9a+iwfdB/J2+0gukqY89qWdzISUALWUZG0PFt1F3zOp9iQR361EHf/eOBck3n8g6uXyDQ6BtCtZmBuyfTye+U6xICm90C+5Km7FFh/GgwRkQcNIfxJOpYFz8Iq+L3dHB+QWhnNZJ/yryJX5smqzA+34SMHSSLhgEVy2UlsLy4B+be7/rjvEgweeGc3hLuy7bYa+0tKUeXescsvVDYM8GiUU6czbqt0HUt9HDC3sGMvSaKByDdt+DYiJkmJlSiuGqBUXTByAf55Yuiw6tBxA56IpU4xu8rfL7aJheVMcr0 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(166708455590820)(75325880899374)(83272782819144); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(3002001)(10201501046)(3231023)(944501075)(93006095)(93001095)(6041268)(20161123558120)(20161123564045)(20161123562045)(201703131423095)(20161123555045)(201703061421075)(2016111802025)(20161123560045)(6072148)(6043046)(201708071742011); SRVR:DB6PR05MB3494; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:DB6PR05MB3494; X-Forefront-PRVS: 0539EEBD11 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(346002)(376002)(39380400002)(366004)(52544003)(189003)(199004)(24454002)(377424004)(43784003)(68736007)(77096006)(6486002)(8676002)(81156014)(575784001)(8936002)(3480700004)(31696002)(508600001)(81166006)(86362001)(2906002)(3846002)(64126003)(6916009)(105586002)(6666003)(2950100002)(31686004)(606006)(106356001)(6116002)(53936002)(65826007)(5660300001)(33896004)(33964004)(6306002)(76176011)(16576012)(59450400001)(386003)(54896002)(84326002)(229853002)(39060400002)(345774005)(4326008)(52116002)(6246003)(16526018)(83506002)(97736004)(966005)(4001150100001)(7736002)(236005)(25786009)(53546011)(37036004)(58126008)(66066001)(65956001)(65806001)(16586007)(46492003); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6PR05MB3494; H:[10.0.8.11]; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: atech.media does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB6PR05MB3494; 23:R0mYLuGL0VLrxQ6bQsh1QIhcZ8/DeXvxr0TKrZn4p?= =?us-ascii?Q?bPwZUuoQl+9fWTlrlmB3W659BrkuOfV4d+x7TU3ddvr8Z+umzxbMtfsFYqeR?= =?us-ascii?Q?oYUhcppuvTyldsJ5xZMIt81mWmwU6XIoQYf+vWTcVKMP0tYM8izh0GeB2LXc?= =?us-ascii?Q?tQ/8zZtWFUf+yuHoKvqZg6snxYNsKFliAB/JmEQzLFNr+xmSl9RPuBzFOzfI?= =?us-ascii?Q?omQEN2I6k34+PpxOgv8IRfXnY2Qew2INdUsPdYtDLfxcUUCLr4phVBDiON/s?= =?us-ascii?Q?A8aEQU8RrFHflLh48I8gB3485bEzkxoe2uve/BNRAcAWST64oPth1rUCyYkv?= =?us-ascii?Q?TdGxt8Bovymu8jAsTWBqB56RNK7dK8aFdYqUelS7A68iY6CjQzzOIS/i7Mug?= =?us-ascii?Q?oMGfPxscjgyzg0zSXzXnY/EK2K3RhIrtzzT3flde95OgCBV8en23jGyZ2AxO?= =?us-ascii?Q?HcSi5lUFG8GsjwDU664MgDyUYSKFXqLl4GYUZTsxwhyKVbXbzmACNWC32kPL?= =?us-ascii?Q?No2nq5fm2sAHtS6ZZ3UAGcfvpJT8ubUqW4IXahwSF+mLm76TyP9dFRvhPAUU?= =?us-ascii?Q?knwBeBHaQN5T08FTsUMQv/IXB/Xt5SVhNjUESGnoHsyyjqhn/t6RmiBwzndR?= =?us-ascii?Q?MzEl1Kpdg3mhrlzygjEOQYw5s465A3GlCEuWu8PU6+b2bpJX/ijWt0B1PXxk?= =?us-ascii?Q?D//tyKPbAppOqodcy69VtIBFuzHlyhAky8qmcwyb2lGretd70aInrMzwia0h?= =?us-ascii?Q?2RlTg9TIoql9OBgasYPAVyY6YT/TXQ0KcSf0P/Nf0NXoSD/S4lDxewDgVKRD?= =?us-ascii?Q?Cn+NZjZ5n6QFpFg/jHqusB+Mu+/2Vf7MhQGyxZVvApDkgI0O0qeolm/eLYhZ?= =?us-ascii?Q?dONe9BcDOkX7txP7YE4IsUuLPY0pkxPLffvy0TC4StEgGov82c3ElySEKIH1?= =?us-ascii?Q?6GUfKqC0tAeL1rFEqzdX00yhfFyCoDA2Sk+hEVNa4qwwieAw2G7Jbh9EkOeb?= =?us-ascii?Q?dt1QnVF/fr8y/Pdiv6uNE40lxnPswT9mIO06bxI/7RPamE4t7h+UWnNFINDF?= =?us-ascii?Q?oK3L/Ae9lOLWkKWxKaTa74o0MA0gInA4W48X+RuWIDiBUqgR0tl1ZMNXeBa5?= =?us-ascii?Q?lIoy5JzrRIfiVyPiIpzYLVXvhAo0ZIIoYSHhtdCpKqKZxCopJVTL75jl0Q/K?= =?us-ascii?Q?GfyJ6yZ0mzfvagps5QbCZl0IIQaxVgl36vfoUuyQS0gfNKHHs20m7lzAzmCR?= =?us-ascii?Q?JG09Jwjjq0GQeWkYJLiwjnDiYNg+qiZI0+Rufuc9qHjRzm8ahQxOwfIRXO/Y?= =?us-ascii?Q?ly/YrI3MJHm4fXHZGx3x7k7XUDFs/j1zQpqhWxcgfTmpRajV7pifYDz6qCh9?= =?us-ascii?Q?7dSRiMiSopa1rwTpvQzGTB5wpZsdopgSoSLOVEqxRvf70cwRZH/q2EQUIFoB?= =?us-ascii?Q?MbOCSdths+Zwdw/aICB+XlJKFPcap+kkRZ9IiSxNnit9nXXzG0eOosrSKlcs?= =?us-ascii?Q?3KjYxSy+nWJkgAskeSQ0M3zyKgypeeszCUhz9y2p04Wke208cAE6A7WPrq3I?= =?us-ascii?Q?H0p/j64WnUbEVCzog=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB6PR05MB3494; 6:wf2bCoODQNdXGIv10RayeD4S3lqBrODCcYp/0nkvCB4TTOpChxsSJr8wSH3v0/Cps3EjmOAmSQZlMBzWEhMVZHchhDNldE0AeEbzHOJKHzL1hXV1MCV6bSrh5WIja/z8V4Gul6JllYzp22KjWG6DynRyc0xxTnuAW/sLShs9CpZMyEW/7NsrvAXwZmWtQb9Fs37LeLMumcRayJhHH4tz7rrQcIqgk/wMcpIw3cm7zlpt+4vwET+OgIDQYu8hJrRVJVMsZkOG2L6lmn+bKy0RGlbDySqgdVken6wQ39seTOtFnGnwkLNliNbrAfohldtda/MG77L0SEgVt/Jz1rDItALoqo5Z/3ln6W+cOvEXE0U=; 5:nEalYNQY+OI5/eLH3oh/1Bbx+rd4MLQ3jpfd1VBtjeaUUhOffCDapqDVQts0ZGx8s6V4R7wGSj9FdJfbAuuSLbxZPLH7N2LU4hZQ2q0lrW8i1Jj+bC0lYlET/rcX9DPCY8UBMWb5UyF7C4twVuoGS7ohbdWDoduKrg4BkfW0IzE=; 24:jJcNtFF+OpV0alAuPAD958kr5lvwujB8BAdwx4RbqNZTeIQP463WJ3YsLAvm/24gAwJJy1/AA4+RjMsSXRkBG0pcERPwTl90YBYk3Dixz0I=; 7:wX71JcpNpWgDaIXRY2tOlXkYsbhNKF6/a8dMJojAJHDR/d5pf0FEN6UwIjKrNqeuD5GXaWa45TbxZz5oqtV6RWLwxG0W/yUcJX6Yg8zlz/ZBEclE8ZzADLTVOQZDNrkjYAqI4co36nVJUW0B3Xeu5EDJHgXjHGNmtakKeSCNt+iUC+3yvEgI6NML32v3pRwqrfBaz60KH6KknheG3G8GBUL1uGtCCyXQTmnCgPwzigGK5WV7LRWJGxcaj7VNVEYI SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: atech.media X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jan 2018 16:14:44.4397 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8d8afe69-370d-4b75-1896-08d55132c56e X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 7a8f6edf-720f-4e3d-b767-1360e39a8cdf X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR05MB3494 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 16:14:49 -0000 Hi, Thank you for your reply. I was able to resolve this. 1) I do indeed open one FD per NIC 2) I no longer specify nr_arg1, nr_arg2 nor nr_arg3. Instead I just verify that all NICs return with identical nr_arg2 so that the memory is shared between them. 3) I properly initialized my memory, my failure to do so was causing me a lot of confusion, The resulting memory space is large enough for all the NICs, and everything works perfectly with zero-copy forwarding, great! The only thing I am still having trouble with is the ability to simultaneously trigger a TX and an RX sync on all NICs. I have tried select, poll, and epoll, and in all cases, RX rings are updated but TX rings are not and TX packets are not pushed out (this occurs using both native and emulated netmap modes). I notice the documentation says "Note that on epoll and kqueue, NETMAP_NO_TX_POLL and NETMAP_DO_RX_POLL only have an effect when some event is posted for the file descriptor.", but the behaviour seems the same on poll and select as well as epoll, perhaps this is a linux-specific implementation detail? I have also found that all of these mechanisms seem to incur a very high cost in terms of CPU time (making them no more efficient than busy waiting at 1Mpps+). My current approach is as follows, but I feel like there should be a better option:     for(int n=0; n Hi, >   If you have 32 NICs you should open 32 netmap file descriptors, (and > you should not specify 64 in nr_arg1 or 256 in nr_arg3, this is for > different usecases). Also, since you want to do zercopy you must not > specify a separate memory area (nr_arg2), but use the same one. > You may want to use the high level API nm_open() > https://github.com/luigirizzo/netmap/blob/master/sys/net/netmap_user.h#L307 > > You may also want to look at the netmap tutorial to get a better idea > of how the API works (https://github.com/vmaffione/netmap-tutorial). > > Cheers, >   Vincenzo > > 2017-12-28 18:34 GMT+01:00 Charlie Smurthwaite >: > > Hi, > > I'm just starting to use netmap and it is my intention to do zero-copy > forwarding of frames between a large number of NICs. I am using Intel > i350 (igb) on Linux. I therefore require a large memory area for rings > and buffers. > > My calculation: > 32 NICs * 2 rings (TX+RX) * 256 frames * 2048 bytes = 32MB > > I am currently having a problem (or perhaps just a misunderstanding) > regarding allocation of this memory. I am attempting to use the > following code: > > void thread_main(int thread_id) { >   struct nmreq req; // A struct for the netmap request >   int fd;           // File descriptor for netmap socket >   void * mem;       // Pointer to allocated memory area > >   fd = open("/dev/netmap", 0);     // Open a generic netmap socket >   strcpy(req.nr_name, "enp8s0f0"); // Copy NIC name into request >   req.nr_version = NETMAP_API;     // Set version number >   req.nr_flags = NR_REG_ONE_NIC;   // We will be using a single hw > ring > >   // Select ring 0, disable TX on poll >   req.nr_ringid = NETMAP_NO_TX_POLL | NETMAP_HW_RING | 0; > >   // Ask for 64 additional rings to be allocated (32 * (TX+RX)) >   req.nr_arg1 = 64; > >   // Allocate a separate memory area for each thread >   req.nr_arg2 = 10 + thread_id; > >   // Ask for additional buffers (256 per ring) >   req.nr_arg3 = 64*256; > >   // Initialize port >   ioctl(fd, NIOCREGIF, &req); > >   // Check the allocated memory size >   printf("memsize: %u\n", req.nr_memsize); >   // Check the allocated memory area >   printf("nr_arg2: %u\n", req.nr_arg2); > } > > The output is as follows: > > memsize: 4206859 > nr_arg2: 10 > > This is far short of the amount of memory I am hoping to be allocated. > Am I doing something wrong, or is this simply an indication that the > driver is unwilling to allocate more than 4MB? > > A secondary (related) problem is that if I don't set arg1,arg2,arg3 in > my code (ie they will be zero), then I get varying output (it varies > between each of the following): > > memsize: 4206843 > nr_arg2: 0 > > memsize: 343019520 > nr_arg2: 1 > > Any pointers would be appreciated. Thanks! > > Charlie > > > Charlie Smurthwaite > Technical Director > > tel.  email. charlie@atech.media > web. https://atech.media > > This e-mail has been sent by aTech Media Limited (or one of its > assoicated group companys, Dial 9 Communications Limited or > Viaduct Hosting Limited). Its contents are confidential therefore > if you have received this message in error, we would appreciate it > if you could let us know and delete the message. aTech Media > Limited is a UK limited company, registration number 5523199. Dial > 9 Communications Limited is a UK limited company, registration > number 7740921. Viaduct Hosting Limited is a UK limited company, > registration number 8514362. All companies are registered at Unit > 9 Winchester Place, North Street, Poole, Dorset, BH15 1NX. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > > > > -- > Vincenzo Maffione