From owner-freebsd-hackers@FreeBSD.ORG Thu Aug 28 04:43:44 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ECA47949 for ; Thu, 28 Aug 2014 04:43:43 +0000 (UTC) Received: from mail-ie0-f170.google.com (mail-ie0-f170.google.com [209.85.223.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B65431C24 for ; Thu, 28 Aug 2014 04:43:43 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id rl12so320034iec.29 for ; Wed, 27 Aug 2014 21:43:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=M3IaTfEcwc50SyGhE6LYcVZ4d4horODSwO4RSl47DEQ=; b=IV0JZPPqzPNQ6sNMduh5IZ0g964u//COai5aY1nRVfJJgPP03zJhOqVJRwzYdzggPd 1/QBXG49oWAOg8lv8uHXsobFk52iEhitUoQLey44KNM6y6raiT66Y/uxp6OrwNpPywxX iKIKk/GEbFWgbHAwLvJj3sbJdjhHiWFQ7QXfy3m20ZQPat6dav/Yclb4ElGZaB7G+CbH DMSS/TWebFc3bEKemit1cTohl+QhqU/rezTYeatoO7pQOXTQnN3y5EmV4OEfowJwa6aa UamaDkJMxHRlRIJW4m5mVCnhMzqfQcWNFUcre+wSJQRnmZp4U9sZdea9MW0H23mzSAIU X7EA== X-Gm-Message-State: ALoCoQn0mKPHysfmAhaC48GN7tYLIHU2hgV3sTfzrD/iuIfpN5H6d/Qf2mdxXtAaVh/INy4cL19j X-Received: by 10.42.67.133 with SMTP id t5mr2172398ici.62.1409200538167; Wed, 27 Aug 2014 21:35:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.29.20 with HTTP; Wed, 27 Aug 2014 21:35:17 -0700 (PDT) X-Originating-IP: [72.177.8.109] In-Reply-To: <53FE2B37.7050309@gameservers.com> References: <53FE2B37.7050309@gameservers.com> From: Bryan Venteicher Date: Wed, 27 Aug 2014 23:35:17 -0500 Message-ID: Subject: Re: Bug in FreeBSD VirtIO network driver (only with pf enabled) To: Brian Rak Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Aug 2014 04:43:44 -0000 On Wed, Aug 27, 2014 at 2:02 PM, Brian Rak wrote: > I have a FreeBSD 10 x64 guest installed inside a KVM instance on Linux. > When pf is active, and the server is sending data it causes the Linux hos= t > to report warnings related to GSO. > > I've talked to some Linux developers, and they believe it to be a bug > inside the FreeBSD VirtIO drivers. Based on what I'm seeing, I'm incline= d > to agree with them. > > Reproduction is mildly annoying, you need: > 1) A KVM guest setup with bridged networking, with FreeBSD running side > using the VirtIO network interface. (We tested with CentOS, but the exac= t > distribution should not matter) > 2) pf enabled (I'm using a single rule: "scrub in all") > 3) Some method of sending a bit of traffic from the guest (I use netcat) > > So, when I do this: > > # service pf start > # cat /root/test | nc vultr.com 80 > > The Linux kernel on the host will report: > > kernel: WARNING: CPU: 7 PID: 7772 at net/core/dev.c:2246 > skb_warn_bad_offload+0xc3/0xd0() > kernel: igb: caps=3D(0x0000000640114bb3, 0x0000000000000000) len=3D1498 > data_len=3D0 gso_size=3D1380 gso_type=3D5 ip_summed=3D0 > > If I do: > > # service pf stop > # cat /root/test | nc vultr.com 80 > > No such warning is reported. I can only reproduce this with pf enabled. > The contents of the /root/test don't matter, I'm using 4k of data from > /dev/urandom. The test file just needs to be bigger then the MTU of the > host's network interface. > > I was able to track this down to the virtio_net_hdr being sent by the > FreeBSD guest. With pf enabled, this outbound traffic has the following > header: > > flags =3D 0 > gso_type =3D VIRTIO_NET_HDR_GSO_TCPV4 > hdr_len =3D 66 > gso_size =3D 1440 > csum_start =3D 0 > csum_offset =3D 0 > > But, this is not a valid configuration. With VIRTIO_NET_HDR_GSO_TCPV4 > enabled, you should also be setting VIRTIO_NET_HDR_F_NEEDS_CSUM and > populating the csum_start and csum_offset fields. > > =E2=80=8BIt would appear that somewhere in pf the CSUM_TCP flag is getting = clear for CSUM_TSO packets. I don't believe the stack otherwise sets CSUM_TSO without CSUM_TCP. Now, it is probably safe to assume CSUM_TCP when CSUM_TSO is set: the mbuf's m_pkthdr.csum_data better be valid in such cases though. > http://www.spinics.net/lists/netdev/msg293976.html gives more detail on > this. I don't fully understand this, so I'd probably mangle the > explanation if I tried to give more detail. I can reproduce this at will > now, but fixing it is beyond my abilities. Is there a better place to > report this? I'm not entirely sure who is responsible for maintaining th= e > virtio driver. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " >