Date: Tue, 6 Jan 2015 00:16:01 +0000 From: Carlos Ferreira <carlosmf.pt@gmail.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Regarding Netmap internal memory allocation. Message-ID: <CAJpYY6XYs7jDApz2L=dMjowhEFQyw5oGRG9K0JrLtXWdCs8HsQ@mail.gmail.com> In-Reply-To: <CAJpYY6U6d=ozhpKunRL4uw7fh%2B8D=P_x_UdUpKANTspVcfLwbg@mail.gmail.com> References: <CAJpYY6XmOnYX4bm=YyXr5hoiFLXhoEFXDa44Z_Fn1fHNj8G5_w@mail.gmail.com> <CAJpYY6UH4bU9xYBsAgZm6_h1no9TDPJ9J2TNp4V19rYGrRGuuw@mail.gmail.com> <CA%2BhQ2%2BiD=2g65ScRBzVVYGRu=jmxksCtMNmObTTnnXfG=Yuu3g@mail.gmail.com> <CAJpYY6UGuzVY%2B=_m2RhOfhR40NHxv6uM%2BfWeJpOGkj9DDXWKiw@mail.gmail.com> <CAJpYY6Xy1AZi%2Bb7Dm%2Bn-TnvJJzWEjzQC35%2BntbLm2f4DST=yjQ@mail.gmail.com> <CAJpYY6WGO%2BW8%2BpPfNk1G9WUAO_6DVZf5iqbw-nFP7wnhfxBHQw@mail.gmail.com> <CA%2BhQ2%2Bj5JkNn6H9cEdWtsfdv7B5XMto2TQgpYk%2Bvky7v0QQheA@mail.gmail.com> <CAJpYY6U=x_PAf0pKoWZTQwCfb0_kmra4Qb4TiUKgXKeTphXidw@mail.gmail.com> <CA%2BhQ2%2Bij-7wZrp%2BoLpVnTPFe-FNJV_ro8JxYQdPBTfKP2BMyPA@mail.gmail.com> <CAJpYY6U6d=ozhpKunRL4uw7fh%2B8D=P_x_UdUpKANTspVcfLwbg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
First of all, Happy New Year :)
Now, to more serious developments..
I was unable to reduce the netmap memory usage by manipulating the buf_num
value. The lowest I could get was around the 20MB which is still high. I'm
going to change the strategy and try to directly mmap the RX and TX buffers
from the interface sockets. The results are going to be worse and I know
that I'm not going to have Zero-Copy capability between interfaces but...
oh well, I guess at least I'm going to be able to do something better than
basic layer 2 sockets.
Thank you for the info regarding netmap.
On 30 December 2014 at 18:55, Carlos Ferreira <carlosmf.pt@gmail.com> wrote:
> Well... due to budget constraints I'm using USB 100Mb ports :)
> This is for experimental purposes only for now.
>
> Btw, can netmap work with wireless interfaces? I believe you once answered
> this question, but I could not find it in the mail list.
>
> On 30 December 2014 at 18:40, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
>
>> On Tue, Dec 30, 2014 at 6:38 PM, Carlos Ferreira <carlosmf.pt@gmail.com>
>> wrote:
>> > Ok, I'm having some trouble in tuning the amount of memory for netmap.
>> >
>> > I have been following the man page from FreeBSD in other to understand
>> the
>> > values at /sys/modules/netmap/parameters for linux but I'm having some
>> > trouble in understanding what each value actually means.
>> > For the following values:
>> >
>> > dev.netmap.ring_num: 200 -> Is this the number of rings in the Ring
>> Buffer
>> > Pool?
>>
>> yes. For interfaces with a single queue you need 4 rings (rx+tx for the
>> nic and
>> another rx+tx for the host port)
>>
>> > dev.netmap.ring_size: 36864 -> Is this value, the number of slots per
>> ring?
>>
>> this is the size in bytes of each ring. The number of slots is set by the
>> hardware (low end devices as in the openwrt devices will probably use
>> 256 or 512 slots, so 10-12k should suffice. But this is not worth
>> changing.
>>
>> Instead, you should reduce the number of buffers, though 8MB is only 4000
>> buffers and it is a bit on the low side for 5 ports.
>>
>> However, as far as I know most openwrt devices only have one physical NIC,
>> and a switch implementing various vlans.
>>
>> cheers
>> luigi
>>
>> >
>> > I'm trying to keep the amount of memory used by netmap as low as 4MB -
>> 8MB
>> > since I'm going to use only up to 4 NICs and one TAP.
>> >
>> > Thanks for the help!
>> >
>> >
>> > On 30 December 2014 at 16:16, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
>> >>
>> >> you can #undefine WITH_VALE.
>> >> But it is only 20K of code (and 150K of data structures, which you
>> >> can further reduce by lowering NM_BRIDGS).
>> >> The saving is probably not worth the effort.
>> >>
>> >> cheers
>> >> luigi
>> >>
>> >> On Tue, Dec 30, 2014 at 5:08 PM, Carlos Ferreira <
>> carlosmf.pt@gmail.com>
>> >> wrote:
>> >> > By the way, another question.
>> >> > Is there a way to not compile the code regarding the VALE switch? I'm
>> >> > only
>> >> > interested in using netmap with Tap Devices and NICs, so I was
>> hoping to
>> >> > save some memory.
>> >> >
>> >> > On 30 December 2014 at 15:47, Carlos Ferreira <carlosmf.pt@gmail.com
>> >
>> >> > wrote:
>> >> >>
>> >> >> You mean netmap_mem2.c ? It was there where I found the
>> >> >> NETMAP_BUF_MAX_NUM
>> >> >> define.
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 30 December 2014 at 15:43, Carlos Ferreira <
>> carlosmf.pt@gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Ok thanks. I was hoping not having to recompile the module, but
>> it's
>> >> >>> ok.
>> >> >>> Thank you for the info!
>> >> >>>
>> >> >>>
>> >> >>> On 30 December 2014 at 15:38, Luigi Rizzo <rizzo@iet.unipi.it>
>> wrote:
>> >> >>>>
>> >> >>>> you can reduce the amount of ram (buffers, mostly) by
>> >> >>>> tweaking the values in netmap_mem2.c ::
>> >> >>>> struct netmap_obj_params netmap_params[NETMAP_POOLS_NR] = {
>> >> >>>> ...
>> >> >>>> }
>> >> >>>>
>> >> >>>> or you can simply modify the constant
>> >> >>>>
>> >> >>>> netmap_mem2.h:#define NETMAP_BUF_MAX_NUM 20*4096*2
>> >> >>>>
>> >> >>>> to something smaller that suits an openwrt box
>> >> >>>> (in which i am very interested, as I'd like to deploy one of these
>> >> >>>> soon)
>> >> >>>>
>> >> >>>> cheers
>> >> >>>> luigi
>> >> >>>>
>> >> >>>>
>> >> >>>> On Tue, Dec 30, 2014 at 4:12 PM, Carlos Ferreira
>> >> >>>> <carlosmf.pt@gmail.com>
>> >> >>>> wrote:
>> >> >>>> > Update:
>> >> >>>> >
>> >> >>>> > I noticed that the netmap module was still crashing, after
>> >> >>>> > changing
>> >> >>>> > the
>> >> >>>> > OpenWRT VM ram to 256MB. I now raised to 1GB and it no longer
>> >> >>>> > crashed.
>> >> >>>> > The
>> >> >>>> > netmap module is now consuming about 350MB of Ram, which for my
>> >> >>>> > objectives
>> >> >>>> > is just too much...
>> >> >>>> >
>> >> >>>> > On 30 December 2014 at 14:06, Carlos Ferreira
>> >> >>>> > <carlosmf.pt@gmail.com>
>> >> >>>> > wrote:
>> >> >>>> >
>> >> >>>> >> To Luigi and to whom may be able to help
>> >> >>>> >>
>> >> >>>> >> Hello all.
>> >> >>>> >>
>> >> >>>> >> Is it possible to reduce the size of the memory buffer
>> allocated
>> >> >>>> >> by
>> >> >>>> >> the
>> >> >>>> >> netmap module?
>> >> >>>> >> I'm asking this because I was implementing some testing code,
>> >> >>>> >> using
>> >> >>>> >> NICs
>> >> >>>> >> and a Tap device in an OpenWRT VM with 64MB of RAM.
>> >> >>>> >> Because of the small RAM amount, the nm_open crashed when the
>> >> >>>> >> program
>> >> >>>> >> tried to netmap the tap device, after I previously netmapped
>> one
>> >> >>>> >> NIC
>> >> >>>> >> successfully.
>> >> >>>> >> After the crash, I bumped the VM RAM to 256MB and the test
>> program
>> >> >>>> >> ran
>> >> >>>> >> well, but not without me noticing that the VM RAM consumption
>> was
>> >> >>>> >> increased about 90 MB by netmap.
>> >> >>>> >>
>> >> >>>> >> Resuming, I want to know if there is a way to reduce the memory
>> >> >>>> >> buffer
>> >> >>>> >> allocation, without recompiling the netmap kernel module.
>> >> >>>> >>
>> >> >>>> >> Thank you for the attention.
>> >> >>>> >>
>> >> >>>> >> --
>> >> >>>> >>
>> >> >>>> >> Carlos Miguel Ferreira
>> >> >>>> >> Researcher at Telecommunications Institute
>> >> >>>> >> Aveiro - Portugal
>> >> >>>> >> Work E-mail - cmf@av.it.pt
>> >> >>>> >> Skype & GTalk -> carlosmf.pt@gmail.com
>> >> >>>> >> LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>> >> >>>> >>
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > --
>> >> >>>> >
>> >> >>>> > Carlos Miguel Ferreira
>> >> >>>> > Researcher at Telecommunications Institute
>> >> >>>> > Aveiro - Portugal
>> >> >>>> > Work E-mail - cmf@av.it.pt
>> >> >>>> > Skype & GTalk -> carlosmf.pt@gmail.com
>> >> >>>> > LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>> >> >>>> > _______________________________________________
>> >> >>>> > freebsd-net@freebsd.org mailing list
>> >> >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> >> >>>> > To unsubscribe, send any mail to
>> >> >>>> > "freebsd-net-unsubscribe@freebsd.org"
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> -----------------------------------------+-------------------------------
>> >> >>>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing.
>> >> >>>> dell'Informazione
>> >> >>>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
>> >> >>>> TEL +39-050-2211611 . via Diotisalvi 2
>> >> >>>> Mobile +39-338-6809875 . 56122 PISA (Italy)
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> -----------------------------------------+-------------------------------
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>>
>> >> >>> Carlos Miguel Ferreira
>> >> >>> Researcher at Telecommunications Institute
>> >> >>> Aveiro - Portugal
>> >> >>> Work E-mail - cmf@av.it.pt
>> >> >>> Skype & GTalk -> carlosmf.pt@gmail.com
>> >> >>> LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >>
>> >> >> Carlos Miguel Ferreira
>> >> >> Researcher at Telecommunications Institute
>> >> >> Aveiro - Portugal
>> >> >> Work E-mail - cmf@av.it.pt
>> >> >> Skype & GTalk -> carlosmf.pt@gmail.com
>> >> >> LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Carlos Miguel Ferreira
>> >> > Researcher at Telecommunications Institute
>> >> > Aveiro - Portugal
>> >> > Work E-mail - cmf@av.it.pt
>> >> > Skype & GTalk -> carlosmf.pt@gmail.com
>> >> > LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>> >>
>> >>
>> >>
>> >> --
>> >>
>> -----------------------------------------+-------------------------------
>> >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing.
>> dell'Informazione
>> >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
>> >> TEL +39-050-2211611 . via Diotisalvi 2
>> >> Mobile +39-338-6809875 . 56122 PISA (Italy)
>> >>
>> -----------------------------------------+-------------------------------
>> >
>> >
>> >
>> >
>> > --
>> >
>> > Carlos Miguel Ferreira
>> > Researcher at Telecommunications Institute
>> > Aveiro - Portugal
>> > Work E-mail - cmf@av.it.pt
>> > Skype & GTalk -> carlosmf.pt@gmail.com
>> > LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>>
>>
>>
>> --
>> -----------------------------------------+-------------------------------
>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione
>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
>> TEL +39-050-2211611 . via Diotisalvi 2
>> Mobile +39-338-6809875 . 56122 PISA (Italy)
>> -----------------------------------------+-------------------------------
>>
>
>
>
> --
>
> Carlos Miguel Ferreira
> Researcher at Telecommunications Institute
> Aveiro - Portugal
> Work E-mail - cmf@av.it.pt
> Skype & GTalk -> carlosmf.pt@gmail.com
> LinkedIn -> http://www.linkedin.com/in/carlosmferreira
>
--
Carlos Miguel Ferreira
Researcher at Telecommunications Institute
Aveiro - Portugal
Work E-mail - cmf@av.it.pt
Skype & GTalk -> carlosmf.pt@gmail.com
LinkedIn -> http://www.linkedin.com/in/carlosmferreira
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJpYY6XYs7jDApz2L=dMjowhEFQyw5oGRG9K0JrLtXWdCs8HsQ>