Date: Thu, 28 Dec 2017 23:14:46 +0000 From: Charlie Smurthwaite <charlie@atech.media> To: net@freebsd.org Subject: Re: Linux netmap memory allocation Message-ID: <682a059e-130a-99c2-b7e6-4a865098e965@atech.media> In-Reply-To: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media> References: <7b85fc73-9cc8-0a60-5264-d26f47af5eae@atech.media>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
To answer my own question, I have resolved this, there were seemingly 3
errors in my use of the API:
1) I did not initialize the req struct. Some uninitialized data were
being passed to the ioctl. I feel silly.
2) Values greater then 1 do not seem to be accepted by nr_arg2, setting
this to 1, or to 0 (which resolves to 1) works.
3) When fixing the above, the memory size is always 343019520, plenty
for my requirement.
Thanks,
Charlie
On 28/12/17 17:34, Charlie Smurthwaite wrote:
> Hi,
>
> I'm just starting to use netmap and it is my intention to do zero-copy
> forwarding of frames between a large number of NICs. I am using Intel
> i350 (igb) on Linux. I therefore require a large memory area for rings
> and buffers.
>
> My calculation:
> 32 NICs * 2 rings (TX+RX) * 256 frames * 2048 bytes =3D 32MB
>
> I am currently having a problem (or perhaps just a misunderstanding)
> regarding allocation of this memory. I am attempting to use the
> following code:
>
> void thread_main(int thread_id) {
> struct nmreq req; // A struct for the netmap request
> int fd; // File descriptor for netmap socket
> void * mem; // Pointer to allocated memory area
>
> fd =3D open("/dev/netmap", 0); // Open a generic netmap socket
> strcpy(req.nr_name, "enp8s0f0"); // Copy NIC name into request
> req.nr_version =3D NETMAP_API; // Set version number
> req.nr_flags =3D NR_REG_ONE_NIC; // We will be using a single hw ring
>
> // Select ring 0, disable TX on poll
> req.nr_ringid =3D NETMAP_NO_TX_POLL | NETMAP_HW_RING | 0;
>
> // Ask for 64 additional rings to be allocated (32 * (TX+RX))
> req.nr_arg1 =3D 64;
>
> // Allocate a separate memory area for each thread
> req.nr_arg2 =3D 10 + thread_id;
>
> // Ask for additional buffers (256 per ring)
> req.nr_arg3 =3D 64*256;
>
> // Initialize port
> ioctl(fd, NIOCREGIF, &req);
>
> // Check the allocated memory size
> printf("memsize: %u\n", req.nr_memsize);
> // Check the allocated memory area
> printf("nr_arg2: %u\n", req.nr_arg2);
> }
>
> The output is as follows:
>
> memsize: 4206859
> nr_arg2: 10
>
> This is far short of the amount of memory I am hoping to be allocated.
> Am I doing something wrong, or is this simply an indication that the
> driver is unwilling to allocate more than 4MB?
>
> A secondary (related) problem is that if I don't set arg1,arg2,arg3 in
> my code (ie they will be zero), then I get varying output (it varies
> between each of the following):
>
> memsize: 4206843
> nr_arg2: 0
>
> memsize: 343019520
> nr_arg2: 1
>
> Any pointers would be appreciated. Thanks!
>
> Charlie
Charlie Smurthwaite
Technical Director
tel. email. charlie@atech.media<mailto:charlie@atech.media> web. https://a=
tech.media
This e-mail has been sent by aTech Media Limited (or one of its assoicated =
group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). =
Its contents are confidential therefore if you have received this message i=
n error, we would appreciate it if you could let us know and delete the mes=
sage. aTech Media Limited is a UK limited company, registration number 5523=
199. Dial 9 Communications Limited is a UK limited company, registration nu=
mber 7740921. Viaduct Hosting Limited is a UK limited company, registration=
number 8514362. All companies are registered at Unit 9 Winchester Place, N=
orth Street, Poole, Dorset, BH15 1NX.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?682a059e-130a-99c2-b7e6-4a865098e965>