From owner-freebsd-net@freebsd.org Wed Mar 28 15:26:56 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA64EF689C6 for ; Wed, 28 Mar 2018 15:26:56 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-qk0-x22a.google.com (mail-qk0-x22a.google.com [IPv6:2607:f8b0:400d:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 424D6715E5 for ; Wed, 28 Mar 2018 15:26:56 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-qk0-x22a.google.com with SMTP id 132so2804534qkd.5 for ; Wed, 28 Mar 2018 08:26:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XP3zyOVab/a+FlrQ533P5icAQVWIdZf6uN4wRFzyi7w=; b=M8U8WFOjzbjnF1NU9cLg9OQbEq2l8zy5SyiXub9zVF6Q6Sv+U1BOHG7gMwU4NyCh+H i2WE9tMYf2kO3iBA/PwPTJvw3sA6ATnG8WyVujQI6cY82OIcOvJHmMxRHcZRQv08dfnK nZUEbR7VuaJTweBxm/OBbmuxRIcjKtTgCF7V+vJ2JjNP6zTCWhxB8vP7pEOANqB645t+ zg1Co9i5b7HyP/dku4s76JRarH1C1Lyd2BolMVqYqB1kRr5RXcWy8zysvcs8Ar63cG6W mScLJn64WfLkmJYRKuI9Vcz/HgmVAjG+MaACph4gknTDMoocKJFY+g2wwNK57Hr+YAIn oqkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XP3zyOVab/a+FlrQ533P5icAQVWIdZf6uN4wRFzyi7w=; b=lDBnAzFNThxwvzSXWkVxLQFhSwZcHqWJfOqRqfVloHmcgGnABkrt+xYQKyKmjnAaqC 5hh5Gjy0/5ETw5KwWEBNagDzG5RoXLeY+OertZzdboK80MCHlfY4v0qwho6h8xlLojym +5Cj1cjjEjErNJRwLn4rp9bh/ynC7bIVCSScO41hSDQYJQQeQ4/edkaIBFQrAxq79AcE YXzDPbyNHrd9k4OHPQ78ZN/kr/gbHRvoRxlegYVKoEBBAtUyDt97ohr6/n++H1FbcYe5 k18GSZqdm2QDOscSc96LbEiGXdIadyVhMHAgfBMftUftB+oOc01dq0XWQyzUeJfHfiCU cvFw== X-Gm-Message-State: ALQs6tDdqr9OKfGTCKUFN4ETmNnMojwMPahNshBrjbw6G1RQ5LLkI7ah wG6T1R0kJYibrJFq8z/sRE4HFPgi7hsLhGgJwdw= X-Google-Smtp-Source: AIpwx4/7/UU83brfj2tNqfVkDqpDcAScVHNTxCiKJEixJjP+D2HfY4ZDycP+NH6pk1uL84V5/Ju4pjCvnc+8aCjUbJA= X-Received: by 10.55.7.18 with SMTP id 18mr5716398qkh.249.1522250815443; Wed, 28 Mar 2018 08:26:55 -0700 (PDT) MIME-Version: 1.0 References: <092c8d937ff6471fb701d64dcff1c756@mbx01cmb01p.esentire.local> In-Reply-To: <092c8d937ff6471fb701d64dcff1c756@mbx01cmb01p.esentire.local> From: Vincenzo Maffione Date: Wed, 28 Mar 2018 15:26:44 +0000 Message-ID: Subject: Re: Netmap on Linux nm_open() fail when receive ring size is set to 4096 To: Ming Fu Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2018 15:26:57 -0000 Hi, You need to increase the value of some netmap module parameter. At least ring_size, maybe also buf_num. Then restart your netmap applications. Keep in mind that performance could worsen with more slots, because of increased cache thrashing. Cheers, Vincenzo On Wed, Mar 28, 2018, 3:50 PM Ming Fu wrote: > Hi, > > I was trying netmap on a Linux box with 128G of ram (64G per numa node). > If I set ixgbe interface to 4096 ring size, the nm_open will fail with > error "Cannot allocate memory". What can I tweak to make the card use > larger ring size? The following test was run after fresh reboot. > > $ ethtool -g enp5s0f0 > Ring parameters for enp5s0f0: > Pre-set maximums: > RX: 4096 > RX Mini: 0 > RX Jumbo: 0 > TX: 4096 > Current hardware settings: > RX: 512 > RX Mini: 0 > RX Jumbo: 0 > TX: 512 > > $ ethtool -G enp5s0f0 rx 1024 > $ ./nmtest -i enp5s0f0 > ^C > $ ethtool -G enp5s0f0 rx 2048 > $ nmtest -i enp5s0f0 > ^C > $ ethtool -G enp5s0f0 rx 4096 > $ nmtest -i enp5s0f0 > 816.039684 nm_open [945] NIOCREGIF failed: Cannot allocate memory > netmap:enp5s0f0 > fail to nm_open(netmap:enp5s0f0 ... ): Cannot allocate memory > > Thanks, > Ming > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >