From owner-freebsd-net@freebsd.org Thu Feb 16 09:34:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3AB97CE0DFF for ; Thu, 16 Feb 2017 09:34:49 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from smtp.unipi.it (smtp3.unipi.it [131.114.21.153]) by mx1.freebsd.org (Postfix) with ESMTP id D32541AD7 for ; Thu, 16 Feb 2017 09:34:47 +0000 (UTC) (envelope-from g.lettieri@iet.unipi.it) Received: from localhost (localhost [127.0.0.1]) by smtp.unipi.it (Postfix) with ESMTP id 765F0C1F73; Thu, 16 Feb 2017 10:25:09 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at unipi.it Received: from [10.216.1.207] (prova.iet.unipi.it [131.114.58.86]) (Authenticated User) by smtp.unipi.it (Postfix) with ESMTPSA id A38BEC19AD; Thu, 16 Feb 2017 10:25:08 +0100 (CET) Subject: Re: netmap error on -CURRENT with em and igb To: "Eggert, Lars" , "freebsd-net@FreeBSD.org" References: <8F640402-8F0B-407D-AB72-743A9AB50CA3@netapp.com> From: Giuseppe Lettieri Message-ID: <5dc6df40-f5f0-d9c2-ed3e-d543c0dd59e2@iet.unipi.it> Date: Thu, 16 Feb 2017 10:25:08 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <8F640402-8F0B-407D-AB72-743A9AB50CA3@netapp.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2017 09:34:49 -0000 Hi all, the "Operation not permitted" is coming from iflib_netmap_register: ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); ... IFDI_INIT(ctx); // for igb it calls em_if_init() ... return (ifp->if_drv_flags & IFF_DRV_RUNNING ? 0 : 1); the last line is meant to check that the initialization was successful, but em_if_init() is not setting IFF_DRV_RUNNING. Note that the iflib_netmap_register code, and the other netmap register functions which it mimicks, have other issues: - in case of failure on NIOCREGIF, they leaves the NETMAP_ON flag set even if the netmap data structures are deleted; - the 1 was intended as a generic indication of failure, but it translates to the rather confusing EPERM; - they check IFF_DRV_RUNNING flag outside of the lock; I don't know it this is safe or not. I can prepare a patch fixing these issues, but I don't know wether em_if_init() is supposed to set the IFF_DRV_RUNNING flag or not. Cheers, Giuseppe Il 16/02/2017 09:35, Eggert, Lars ha scritto: > Hi, > > I can't put em or igb interfaces into netmap mode on a recent -CURRENT (ix interfaces work on the same machines). Here are the pkt-gen and dmesg outputs: > > # sudo sysctl dev.netmap.admode=1 > # sudo sysctl dev.netmap.verbose=1 > > # sudo ./pkt-gen -i em1 > 790.411737 main [2274] interface is em1 > 790.411753 main [2397] running on 1 cpus (have 20) > 790.411823 extract_ip_range [369] range is 10.0.0.1:0 to 10.0.0.1:0 > 790.411828 extract_ip_range [369] range is 10.1.0.1:0 to 10.1.0.1:0 > 790.625117 nm_open [898] NIOCREGIF failed: Operation not permitted em1 > 790.625126 main [2479] Unable to open netmap:em1: Operation not permitted > 790.625129 main [2560] aborting > > dmesg: > 790.411868 [1776] netmap_interp_ringid em1: tx [0,2) rx [0,2) id 0 > 790.419431 [ 551] nm_mem_assign_group iommu_group 0 > 790.425548 [1148] netmap_config_obj_allocator objsize 1024 clustsize 4096 objects 4 > 790.434368 [1148] netmap_config_obj_allocator objsize 36864 clustsize 36864 objects 1 > 790.443390 [1148] netmap_config_obj_allocator objsize 2048 clustsize 4096 objects 2 > 790.452219 [1270] netmap_finalize_obj_allocator Pre-allocated 25 clusters (4/100KB) for 'netmap_if' > 790.463605 [1270] netmap_finalize_obj_allocator Pre-allocated 200 clusters (36/7200KB) for 'netmap_ring' > 790.534919 [1270] netmap_finalize_obj_allocator Pre-allocated 81920 clusters (4/327680KB) for 'netmap_buf' > 790.546113 [1377] netmap_mem_finalize_all interfaces 100 KB, rings 7200 KB, buffers 320 MB > 790.555873 [1380] netmap_mem_finalize_all Free buffers: 163838 > 790.605325 [1831] netmap_mem_global_deref active = 0 > 790.612967 [ 949] netmap_close dev 0xfffff8000a4d7c00 fflag 0x20003 devtype 8192 td 0xfffff8000b899000 > > # sudo ./pkt-gen -i igb1 > 094.077695 main [2274] interface is igb1 > 094.077711 main [2397] running on 1 cpus (have 10) > 094.077822 extract_ip_range [369] range is 10.0.0.1:0 to 10.0.0.1:0 > 094.077827 extract_ip_range [369] range is 10.1.0.1:0 to 10.1.0.1:0 > 094.280311 nm_open [898] NIOCREGIF failed: Operation not permitted igb1 > 094.280319 main [2479] Unable to open netmap:igb1: Operation not permitted > 094.280323 main [2560] aborting > > dmesg: > 094.077866 [1776] netmap_interp_ringid igb1: tx [0,8) rx [0,8) id 0 > 094.085425 [ 551] nm_mem_assign_group iommu_group 0 > 094.091449 [1148] netmap_config_obj_allocator objsize 1024 clustsize 4096 objects 4 > 094.100125 [1148] netmap_config_obj_allocator objsize 36864 clustsize 36864 objects 1 > 094.109004 [1148] netmap_config_obj_allocator objsize 2048 clustsize 4096 objects 2 > 094.117705 [1270] netmap_finalize_obj_allocator Pre-allocated 25 clusters (4/100KB) for 'netmap_if' > 094.128918 [1270] netmap_finalize_obj_allocator Pre-allocated 200 clusters (36/7200KB) for 'netmap_ring' > 094.199456 [1270] netmap_finalize_obj_allocator Pre-allocated 81920 clusters (4/327680KB) for 'netmap_buf' > 094.210482 [1377] netmap_mem_finalize_all interfaces 100 KB, rings 7200 KB, buffers 320 MB > 094.220078 [1380] netmap_mem_finalize_all Free buffers: 163838 > 094.260201 [1831] netmap_mem_global_deref active = 0 > 094.268364 [ 949] netmap_close dev 0xfffff8000a453000 fflag 0x20003 devtype 8192 td 0xfffff80014d45000 > > Lars > -- Dr. Ing. Giuseppe Lettieri Dipartimento di Ingegneria della Informazione Universita' di Pisa Largo Lucio Lazzarino 1, 56122 Pisa - Italy Ph. : (+39) 050-2217.649 (direct) .599 (switch) Fax : (+39) 050-2217.600 e-mail: g.lettieri@iet.unipi.it