From owner-freebsd-current@FreeBSD.ORG Thu Sep 18 18:50:15 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 317FEAAC; Thu, 18 Sep 2014 18:50:15 +0000 (UTC) Received: from mail-la0-x22f.google.com (mail-la0-x22f.google.com [IPv6:2a00:1450:4010:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7ED762ED; Thu, 18 Sep 2014 18:50:14 +0000 (UTC) Received: by mail-la0-f47.google.com with SMTP id mc6so1780441lab.34 for ; Thu, 18 Sep 2014 11:50:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=nqutUqgD/dt5ZyXbg3fM7Rvo8N3VThzImDqy2HKmZ1k=; b=n6am/tpaC2gZKPC4ZxXO+X8XC6S1c/g/unAInMqscLNzf8Pe7JQjHT2Banq2W+ebmu KOIJ/30OTAvj+KAsCOBSzLCNFWa2r9MxM0ocHGEHG4IqRs0uPHwijFr8kTm0QSKbNOfJ sGewWbbnrF3rfUmOg4ptgSuii6jr2lNCiVjL/4MQhMwzuwyy3/dF/P66XYKjNGVC/s56 8gNcslMrGAqvqo4GmwerbePWms1ANJkvHRFppH/odQkzdHa1gBYP7L53jijtHxCo4J2Z qudFV1+WzrxWdilf8Kj/60jggIwFujJcnfMq9sBCRslxlPcjEtyf2opXi/HQrTeD9ZHO 9T+g== MIME-Version: 1.0 X-Received: by 10.112.48.100 with SMTP id k4mr1427780lbn.95.1411066212278; Thu, 18 Sep 2014 11:50:12 -0700 (PDT) Received: by 10.25.21.166 with HTTP; Thu, 18 Sep 2014 11:50:12 -0700 (PDT) In-Reply-To: References: Date: Thu, 18 Sep 2014 14:50:12 -0400 Message-ID: Subject: Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD From: Ryan Stone To: Stefano Garzarella Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" , George Neville-Neil , freebsd-current , Luigi Rizzo X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2014 18:50:15 -0000 On Wed, Sep 17, 2014 at 4:27 AM, Stefano Garzarella wrote: > Much of the advantage of TSO comes from crossing the network stack only > once per (large) segment instead of once per 1500-byte frame. > GSO does the same both for segmentation (TCP) and fragmentation (UDP) > by doing these operations as late as possible. My initial impression is that this is a layering violation. Code like this gives me pause: + eh = mtod(m, struct ether_vlan_header *); + if (eh->evl_encap_proto == htons(ETHERTYPE_VLAN)) { + eh_len = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; + } else { + eh_len = ETHER_HDR_LEN; + } + + return gso_dispatch(ifp, m, eh_len); If someone adds QinQ support, this code must be updated. When vxlan support comes in, we must update this code or else the outer UDP packet gets fragmented instead of the inner TCP payload being segmented. As more tunneling protocols get added to FreeBSD, the dispatch code for GSO gets uglier and uglier. It seems to me that the real problem that we are trying to solve is a lack of batching in the kernel. Currently the network stack operates on the mbuf (packet) boundary. It seems to me that we could introduce a "packet group" concept that is guaranteed to have the same L3 and L2 endpoint. In the transmit path, we would initially have a single (potentially oversized) packet in the group. When TCP segments the packet, it would add each packet to the packet group and pass it down the stack. Because we guarantee that the endpoints are the same for every packet in the group, the L3 code can do a single routing table lookup and the L2 code can do a single l2table lookup for the entire group. The disadvantages of packet groups would be that: a) You have touch a lot more code in a lot more places to take advantage of the concept. b) TSO inherently has the same layering problems. If we're going to solve the problem for tunneling protocols then GSO might well be able to take advantage of them.