From owner-freebsd-net@freebsd.org Mon Jan 1 15:40:31 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB4E9EAF0D0 for ; Mon, 1 Jan 2018 15:40:31 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B5A2970BBC for ; Mon, 1 Jan 2018 15:40:31 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B2466EAF0CE; Mon, 1 Jan 2018 15:40:31 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1E36EAF0CD for ; Mon, 1 Jan 2018 15:40:31 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-qt0-x22e.google.com (mail-qt0-x22e.google.com [IPv6:2607:f8b0:400d:c0d::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C2C470BBA for ; Mon, 1 Jan 2018 15:40:31 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-qt0-x22e.google.com with SMTP id a16so60479822qtj.3 for ; Mon, 01 Jan 2018 07:40:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=k0rCVedNhRNLz22UxZ1g6VOG+Ma7zP4R0CIjMstHZzw=; b=pzy16QOG/N0Tfy93CK94s800YHPe0ZFnxbf/w4PDQAuq2H5i/PieilY8aOrXnQYv6U SANsda5eZIm7Xbld2zZ2Fye9dRYEktQyHfsrdFkmNa0Cid2Dn587KRL8hZ4KqlpFKH9w hzdiLASKPdIJo5VHRojop051gd57ocEz1OCuTPvlv4N81qHQSP05968zgC/g63hjhCwE alEwTLcYLFMr0er8wNkQYsOZnyeNM3zesalqtB+3c3PWFMARKSwfY/mzuMFDJa5xlmuv xB2jbxGCEvb4SD3rGtnOLPkzetXA45nr81zbpd+qCLlEBNSL5CMEAkJvlMOa+GZgNsqf sIIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=k0rCVedNhRNLz22UxZ1g6VOG+Ma7zP4R0CIjMstHZzw=; b=ULaJh2jdXL5MyNmbCKSgW+O/ftI8127F7D7uvFuh6X7wEjnnIK3BZbB4af6MPl5mRl YzvYxWDrmPCta9CIov57JyVBqzntZ4t3LvtdLk8jcAdrsdadsHRGEaRT5aLKrRQlqeNW URV8SnS/it/QIQCCF5rEEY+aMiiNKTrRamfXBOPNvxJgKHiLtJXpklemzEe5nYUEPYj/ FV9PuEymbMcUX364JU12S+twO06FF7E2ZVSIwFTg2CLyHt68Xunbv2BzbbTJfrN47Rph r+CVwDyWkVgp9wBu0TTYotmkHZv5vV+8S78eA8D/zdu90dsTjPj5FUxvt0CUsK/T3qzL r+jg== X-Gm-Message-State: AKGB3mKrJX8Ld7scn2PQTjCKPOzlLY18DQlmv+ORJasLso+KRH8Qq/GH ZVxckUbx1BwGzQe2QfLlPRRlSLU5isUsh+zwrOZCOw== X-Google-Smtp-Source: ACJfBot09qksH3jgPD/fwpF/UuWn3qq9wcbrmRsRUC9izfGx68mOiYrNEf8b39bY3plbk9r10IpEbRC29pqkGStvwXA= X-Received: by 10.200.2.150 with SMTP id p22mr59152571qtg.328.1514821229652; Mon, 01 Jan 2018 07:40:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.174.5 with HTTP; Mon, 1 Jan 2018 07:40:29 -0800 (PST) In-Reply-To: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> References: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> From: Vincenzo Maffione Date: Mon, 1 Jan 2018 16:40:29 +0100 Message-ID: Subject: Re: Linux netmap memory allocation To: Charlie Smurthwaite Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 15:40:32 -0000 Hi, If you have 32 NICs you should open 32 netmap file descriptors, (and you should not specify 64 in nr_arg1 or 256 in nr_arg3, this is for different usecases). Also, since you want to do zercopy you must not specify a separate memory area (nr_arg2), but use the same one. You may want to use the high level API nm_open() https://github.com/luigirizzo/netmap/blob/master/sys/net/netmap_user.h#L307 You may also want to look at the netmap tutorial to get a better idea of how the API works (https://github.com/vmaffione/netmap-tutorial). Cheers, Vincenzo 2017-12-28 18:34 GMT+01:00 Charlie Smurthwaite : > Hi, > > I'm just starting to use netmap and it is my intention to do zero-copy > forwarding of frames between a large number of NICs. I am using Intel > i350 (igb) on Linux. I therefore require a large memory area for rings > and buffers. > > My calculation: > 32 NICs * 2 rings (TX+RX) * 256 frames * 2048 bytes = 32MB > > I am currently having a problem (or perhaps just a misunderstanding) > regarding allocation of this memory. I am attempting to use the > following code: > > void thread_main(int thread_id) { > struct nmreq req; // A struct for the netmap request > int fd; // File descriptor for netmap socket > void * mem; // Pointer to allocated memory area > > fd = open("/dev/netmap", 0); // Open a generic netmap socket > strcpy(req.nr_name, "enp8s0f0"); // Copy NIC name into request > req.nr_version = NETMAP_API; // Set version number > req.nr_flags = NR_REG_ONE_NIC; // We will be using a single hw ring > > // Select ring 0, disable TX on poll > req.nr_ringid = NETMAP_NO_TX_POLL | NETMAP_HW_RING | 0; > > // Ask for 64 additional rings to be allocated (32 * (TX+RX)) > req.nr_arg1 = 64; > > // Allocate a separate memory area for each thread > req.nr_arg2 = 10 + thread_id; > > // Ask for additional buffers (256 per ring) > req.nr_arg3 = 64*256; > > // Initialize port > ioctl(fd, NIOCREGIF, &req); > > // Check the allocated memory size > printf("memsize: %u\n", req.nr_memsize); > // Check the allocated memory area > printf("nr_arg2: %u\n", req.nr_arg2); > } > > The output is as follows: > > memsize: 4206859 > nr_arg2: 10 > > This is far short of the amount of memory I am hoping to be allocated. > Am I doing something wrong, or is this simply an indication that the > driver is unwilling to allocate more than 4MB? > > A secondary (related) problem is that if I don't set arg1,arg2,arg3 in > my code (ie they will be zero), then I get varying output (it varies > between each of the following): > > memsize: 4206843 > nr_arg2: 0 > > memsize: 343019520 > nr_arg2: 1 > > Any pointers would be appreciated. Thanks! > > Charlie > > > Charlie Smurthwaite > Technical Director > > tel. email. charlie@atech.media web. > https://atech.media > > This e-mail has been sent by aTech Media Limited (or one of its assoicated > group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). > Its contents are confidential therefore if you have received this message > in error, we would appreciate it if you could let us know and delete the > message. aTech Media Limited is a UK limited company, registration number > 5523199. Dial 9 Communications Limited is a UK limited company, > registration number 7740921. Viaduct Hosting Limited is a UK limited > company, registration number 8514362. All companies are registered at Unit > 9 Winchester Place, North Street, Poole, Dorset, BH15 1NX. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Vincenzo Maffione