From owner-freebsd-net@freebsd.org Wed Dec 7 01:36:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F1C4C6A4D8 for ; Wed, 7 Dec 2016 01:36:17 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A17EF0F for ; Wed, 7 Dec 2016 01:36:17 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: by mail-io0-x231.google.com with SMTP id l140so71763946iol.3 for ; Tue, 06 Dec 2016 17:36:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to; bh=XvHsgyLRrw/OwOd5gtET2Y3E3wped2lK8SwPl3W8WEM=; b=BwcrEcIwCZmIYtYVDnQN82uA4ZuAoE6nTNyau/l/00QdwTfffNkHpfZfmLk6fwL5JA rz/GdWzjPiHOfnsxQMoIcIMODLdFN9HvUZaPyK01wsbXn4sYzfZVJ7/VgpKPdw3T03v7 3IFbKTSrdrAB9oL87DqIzp/4FJ/Vy5ypn/oHm0f8jHc7F1cZXihm8DYoDIUssGuWSgP4 WYlUjvCynE5z6yCTvkb4rgSvTfHKzE5aTTuJhsDPzghh/1WxeiAy2vN7Kn3xqBCNxOXB zdu+Bn+PRphQch/dpnYdeh2zWjms2bUzbLyurSHrBU0ESVZl/ckGJPnT3PQif30qSfsf UN2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=XvHsgyLRrw/OwOd5gtET2Y3E3wped2lK8SwPl3W8WEM=; b=FBZHZctTJ8dB0lJrl5qIcLMjqCtZzTbjbkwLy6T9P3wexrGrUL1sH+IIW8GIBaLb90 3oUTSM39mQlOO0evJ3XqiZbGB2tRf8KSTAOnmoPyPndHp5fJZ2aPVqZZW6YtV6PpyvBh U9uxuBXku1chKfz7slcEdCafSUiN/pqwVvVRWN4rUmiiUdt8yzvRB9U60ERS+8IdIHLF RIos7WeMwFiaTJ79uKlIEkKzw1p+VILNHp7MHjwOVX76q3qYQJZ2Oo3lumkOaYK4sWMe NthY/ahpQj0PFmHe6fiWO/oBMMl5rbMnjQWRW9Hqp5KrApb7snhJ3kxB8dJExhLSFNOB yDbQ== X-Gm-Message-State: AKaTC01mQnkoabGira47bKY4Vag4izjErwYjTp36Qxp2fxhu5UPsEvZL8T6ItUQMX54i8V2PLGLzmOX3mwthZQ== X-Received: by 10.36.19.9 with SMTP id 9mr269135itz.32.1481074576525; Tue, 06 Dec 2016 17:36:16 -0800 (PST) MIME-Version: 1.0 Sender: sunxiaoye07@gmail.com Received: by 10.64.213.98 with HTTP; Tue, 6 Dec 2016 17:36:16 -0800 (PST) From: Xiaoye Sun Date: Tue, 6 Dec 2016 19:36:16 -0600 X-Google-Sender-Auth: ahI34y3frGQtiA59QNBzvj6udUY Message-ID: Subject: Can netmap be more efficient when it just does bridging between NIC and Linux kernal? To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2016 01:36:17 -0000 Hi, I am wondering if there a way to reduce the CPU usage of a netmap program similar to the bridge.c example. In my use case, I have a distributed application/framework (e.g. Spark or Hadoop) running on a cluster of machines (each of the machines runs Linux and has an Intel 10Gbps NIC). The application is both computation and network intensive. So there is a lot of data transfers between machines. I divide different data into two types (type 1 and type 2). Packets of type 1 data are sent through netmap (these packets don't go through Linux network stack). Packets of type 2 data are sent through Linux network stack. Both type 1 and type 2 data could be small or large. My netmap program runs on all the machines in the cluster and processes the packets of type 1 data (create, send, receive the packets) and forward packets of type 2 data between the NIC and the kernel by swapping the pointer to the NIC slot and the pointer to the kernel stack slot (similar to the bridge.c example in netmap repository). With my netmap program running on the machines, for an application having no type 1 data (netmap program behaves like a bridge which only does slot pointer swapping), the total running time of the application is longer than the case where no netmap program runs on the machines. It seems to me that the netmap program either slows down the network transfer for type 2 data, or it eats up too many CPU cycles and competes with the application process. However, with my netmap program running, iperf can reach 10Gbps bandwidth with 40-50% CPU usage on the netmap program (the netmap program is doing pointer swaping for iperf packets). I also found that after each poll returns, most of the time, the program might just swap one pointer, so there is a lot of system call overhead. Can anybody help me diagnose the source of the problem or is there a better way to write such program? I am wondering if there is a way to tuning the configuration so that the netmap program won't take up too much extra CPU when it runs like the bridge.c program. Best, Xiaoye