From owner-freebsd-xen@freebsd.org Thu Nov 5 17:08:36 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9FFBA2700D for ; Thu, 5 Nov 2015 17:08:36 +0000 (UTC) (envelope-from alans@spectralogic.com) Received: from mail1.bemta7.messagelabs.com (mail1.bemta7.messagelabs.com [216.82.254.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail1.bemta7.messagelabs.com", Issuer "VeriSign Class 3 International Server CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A59161867; Thu, 5 Nov 2015 17:08:36 +0000 (UTC) (envelope-from alans@spectralogic.com) Received: from [216.82.253.227] by server-11.bemta-7.messagelabs.com id 54/47-15271-09B8B365; Thu, 05 Nov 2015 17:02:08 +0000 X-Env-Sender: alans@spectralogic.com X-Msg-Ref: server-14.tower-170.messagelabs.com!1446742928!3453230!1 X-Originating-IP: [192.30.190.14] X-StarScan-Received: X-StarScan-Version: 7.19.2; banners=-,-,- X-VirusChecked: Checked Received: (qmail 30103 invoked from network); 5 Nov 2015 17:02:08 -0000 Received: from outmx1.spectralogic.com (HELO mail.spectralogic.com) (192.30.190.14) by server-14.tower-170.messagelabs.com with AES256-SHA encrypted SMTP; 5 Nov 2015 17:02:08 -0000 From: Alan Somers To: Larry Baird , Roger Pau Monn?? CC: "freebsd-xen@freebsd.org" , "ken@FreeBSD.org" Subject: Re: Checksum forwarding issue on XEN Thread-Topic: Checksum forwarding issue on XEN Thread-Index: AQHRF9zm9zZvkZi2QEuPJaWrDdnj/J6OC++AgAAQ9AA= Date: Thu, 5 Nov 2015 17:02:07 +0000 Message-ID: <563B8B72.5050800@spectralogic.com> References: <20151103201250.GA92469@gta.com> <563B72B2.6060308@citrix.com> <20151105160057.GA2268@gta.com> In-Reply-To: <20151105160057.GA2268@gta.com> Reply-To: Alan Somers Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 x-ms-exchange-transport-fromentityheader: Hosted Content-Type: text/plain; charset="Windows-1252" Content-ID: <389D9AE248AF5C4CB427288AA6767528@spectralogic.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2015 17:08:36 -0000 On 11/05/2015 09:00 AM, Larry Baird wrote: > Roger, > >> Adding the persons that contributed that code in case they can shed some >> light. Removing John Suykerbuyk's outdated address. I don't have a current=20 address for him. And replacing my spectralogic email address with my=20 FreeBSD one. >> >> El 03/11/15 a les 21.12, Larry Baird ha escrit: >>> Has anybody made any progress on "Bug 188261 - [xen] FreeBSD DomU PVHVM >>> guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host= ." >>> (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D188261)? >>> >>> The code for checksum calculation in the function xnb_add_mbuf_cksum() = looks >>> suspect. >>> >>> switch (iph->ip_p) { >>> case IPPROTO_TCP: >>> if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { >>> size_t tcplen =3D ntohs(iph->ip_len) - sizeof(= struct ip); >>> struct tcphdr *th =3D (struct tcphdr*)(iph + 1= ); >>> th->th_sum =3D in_pseudo(iph->ip_src.s_addr, >>> iph->ip_dst.s_addr, htons(IPPROTO_TCP + tc= plen)); >>> th->th_sum =3D in_cksum_skip(mbufc, >>> sizeof(struct ether_header) + ntohs(iph->i= p_len), >>> sizeof(struct ether_header) + (iph->ip_hl = << 2)); >>> } >>> break; >>> case IPPROTO_UDP: >>> if (mbufc->m_pkthdr.csum_flags & CSUM_IP_VALID) { >>> size_t udplen =3D ntohs(iph->ip_len) - sizeof(= struct ip); >>> struct udphdr *uh =3D (struct udphdr*)(iph + 1= ); >>> uh->uh_sum =3D in_pseudo(iph->ip_src.s_addr, >>> iph->ip_dst.s_addr, htons(IPPROTO_UDP + ud= plen)); >>> uh->uh_sum =3D in_cksum_skip(mbufc, >>> sizeof(struct ether_header) + ntohs(iph->i= p_len), >>> sizeof(struct ether_header) + (iph->ip_hl = << 2)); >>> } >>> break; >>> default: >>> break; >>> } >>> >>> >>> Both in_pseudo() and in_cksum_skip() set the same checksum. Does this >>> make since to anybody? Though it looks weird, I think it's actually correct. th->th_sum lies=20 within the packet that it inspected by in_cksum_skip. in_cksum_skip=20 skips the ethernet and IP headers, but not the TCP header. So it=20 consumes whatever value was previously set in th_sum. In any case, it=20 will be easy to check this code by running the builtin unit tests. Just=20 build the kernel with XNB_DEBUG defined, instantiate an xnb interface,=20 and do "sysctl dev.xnb.0.unit_test_results". >> The bug you are referring to affects FreeBSD when running as a guest >> using xen-netfront, but the code snipped above and the function >> referenced (xnb_add_mbuf_cksum) is only used on FreeBSD when running as >> a host (AKA Dom0) by xen-netback. >> >> TBH, I don't know that much about FreeBSD network subsystem to have an >> opinion, but it certainly looks weird. Patches are welcome :). > Xyper-V has a similar forward issue. I found they were misusing csum_flag= s > and were always attempting to do checksum offloading if CSUM_IP_VALID was > set. I have given them a patch that fixes the issue. I was hoping that > Xen's issue was similar. I found the issue above by looking at all uses > of csum_flags in sys/dev/xen. It is hard to tell what the correct fix > is, without fulling understand the protocal used when communicating betwe= en > backend and frontend of Xen. > > I am sure issue with XEN guest forwarding has to with checksum offloading= . > If I am not misinterpreting your comments, I can ignore code in netback a= nd > concentrate on code in netfront when trying to understand what is going w= rong. > > Thank you for some insite, > Larry > > FWIW, there is a performance problem with Xen and checksums. Whenever=20 FreeBSD is used as a dom0, checksum offloading should be disabled on the=20 netfronts. I don't know if the same problem exists on other OSes, but=20 it might. See xnb(4) for more details about this problem. -Alan=