From owner-freebsd-net@FreeBSD.ORG Wed Jul 7 20:50:49 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 77DE7106566C; Wed, 7 Jul 2010 20:50:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 14ECD8FC15; Wed, 7 Jul 2010 20:50:45 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o67KofEo012441 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 7 Jul 2010 23:50:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o67Kofms073822; Wed, 7 Jul 2010 23:50:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o67KofpY073821; Wed, 7 Jul 2010 23:50:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 7 Jul 2010 23:50:41 +0300 From: Kostik Belousov To: Ming Fu Message-ID: <20100707205041.GO13238@deviant.kiev.zoral.com.ua> References: <7C3D15DD6E8F464998CA1470D8A322F302BB9F72@ES02CO.wgti.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JpSMXXT8Tv61TEhN" Content-Disposition: inline In-Reply-To: <7C3D15DD6E8F464998CA1470D8A322F302BB9F72@ES02CO.wgti.net> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-net@freebsd.org, bz@freebsd.org, lstewart@freebsd.org Subject: Re: kern/123095 kern/131602 sendfile X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 20:50:49 -0000 --JpSMXXT8Tv61TEhN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jul 07, 2010 at 10:24:41AM -0700, Ming Fu wrote: > Hi, >=20 >=20 > I was trying to use sendfile and hit with problem very similar to the > 123095 and 131602.=20 > It seems that when the file is large enough (in megs), the file can be > corrupted even if it is open read-only and exist on disk as read-only > file, though the filesystem is mounted read-write. >=20 > I have a small program to reliably reproduce the problem. >=20 > ---------- corrupt.c ----------------- >=20 > #include > #include > #include > #include > #include > #include > #include > #include > #include > #include > main () { > int s, f; > struct sockaddr_in addr; > int flags; > char str[32]=3D"\r\n800\r\n"; > char *p =3D str; > struct stat sb; > int n; > fd_set wset; > int64_t size; > off_t sbytes; > off_t sent =3D 0; > int chunk; >=20 > s =3D socket(AF_INET, SOCK_STREAM, 0); > bzero(&addr, sizeof(addr)); > addr.sin_family =3D AF_INET; > addr.sin_port =3D htons(7000); > addr.sin_addr.s_addr =3D inet_addr("10.1.19.16"); >=20 > n =3D connect(s, (struct sockaddr *)&addr, sizeof (addr)); > if (n < 0) > warn ("fail to connect"); > flags =3D fcntl(s, F_GETFL); > flags |=3D O_NONBLOCK; > fcntl(s, F_SETFL); >=20 > f =3D open("large", O_RDONLY); > if (f<0) > warn("fail to open file"); > n =3D fstat(f, &sb); > if (n<0) > warn("fstat failed"); >=20 > size =3D sb.st_size; > chunk =3D 0; > while (size > 0) { > FD_ZERO(&wset); > FD_SET(s, &wset); > n =3D select(f+1, NULL, &wset, NULL, NULL); > if (n < 0) > continue; > if (chunk > 0) { > sbytes =3D 0; > n =3D sendfile(f, s, sent, chunk, NULL, &sbytes, > 0); > if (n < 0) > continue; > chunk -=3D sbytes; > size -=3D sbytes; > sent +=3D sbytes; > continue; > } > if (size > 2048) > chunk =3D 2048; > else > chunk =3D size; > n =3D sprintf(str, "\r\n%x\r\n", 2048); > p =3D str; > write(s, p, n); > } > } >=20 > ------------- end --------------------------------------------- >=20 > Run nc to receive the sendfile > $ nc -l 7000 >=20 > Copy a large from for sendfile to send > $ cp /usr/lib/libc_pic.a large >=20 > $ md5 large > MD5 (large) =3D 252def82f9d75df11df7123e9fd376f6 >=20 > $ cc -o co corrupt.c > $./co > $ md5 large=20 > MD5 (large) =3D 81ee84e55f4611434459f637c83b892e >=20 > I run this on 8.0-RELEASE. The same happens on 7.2 and 6.3. The disk are > SATA ide. I run all these command under unprivileged user account. I > also run the same program on several different hardware, the result is > the same. Although the corrupted file is not the same. The corruption > looks random to me. >=20 > I know a bit of the network side of FreeBSD kernel code, but the I have > no idea how the filesystem side work. I can dig a bit further if someone > give me a hint as where to look. Right, thank you for the easy way to reproduce it. I was able to trigger this as well. At the http://people.freebsd.org/~kib/vm/sf_buf_readonly.patch=20 is the patch to map some sf buffers readonly, in particular, for the pages that are mapped by sendfile(2). Sure enough, it triggered the panic immediately, with backtrace bcopy sbappendstream_locked tcp_do_segment tcp_input ip_input swi_net (swi because I tested over loopback). To be clear: the backtrace above points to the code path that causes modifications to the file object pages inserted (?) into the mbuf that are supposed to be immutable. Any help from tcp-clueful people is appreciated. --JpSMXXT8Tv61TEhN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkw06KEACgkQC3+MBN1Mb4g4HwCghePyV59D7goxxBJxXKQdFQvf 7RUAoOCBpPRNjNZfi3AbCuZxzqQmOfVj =YfsJ -----END PGP SIGNATURE----- --JpSMXXT8Tv61TEhN--