From owner-freebsd-net@FreeBSD.ORG Thu Mar 1 01:28:58 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4F2B5106566C for ; Thu, 1 Mar 2012 01:28:58 +0000 (UTC) (envelope-from puri.gaurav@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id CE4C08FC0A for ; Thu, 1 Mar 2012 01:28:57 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so57883bkc.13 for ; Wed, 29 Feb 2012 17:28:57 -0800 (PST) Received-SPF: pass (google.com: domain of puri.gaurav@gmail.com designates 10.205.133.13 as permitted sender) client-ip=10.205.133.13; Authentication-Results: mr.google.com; spf=pass (google.com: domain of puri.gaurav@gmail.com designates 10.205.133.13 as permitted sender) smtp.mail=puri.gaurav@gmail.com; dkim=pass header.i=puri.gaurav@gmail.com Received: from mr.google.com ([10.205.133.13]) by 10.205.133.13 with SMTP id hw13mr1474332bkc.25.1330565337002 (num_hops = 1); Wed, 29 Feb 2012 17:28:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=tXgeWh9SjvChGb3at1kQrQlQtJ+C3NIdhRQyfFinBAk=; b=umYn+wG2frtWDQT4IKEblw2C9nru+Ly/hgbqbM97Wn3eDkLTFQdlawG2qA0e3c8p3y CMC+KmIPm9e4/DBUD32/5Nu8xGc/RvLW6miZv7CfWfIqvhS0YfOYXjPXtA+cpc3jayU3 QkIFW1IxAGd+3IFQ4nT7O+Rfr7pG22TdlpyRU= MIME-Version: 1.0 Received: by 10.205.133.13 with SMTP id hw13mr1173016bkc.25.1330563961250; Wed, 29 Feb 2012 17:06:01 -0800 (PST) Received: by 10.205.24.132 with HTTP; Wed, 29 Feb 2012 17:06:01 -0800 (PST) Date: Wed, 29 Feb 2012 17:06:01 -0800 Message-ID: From: Gaurav Puri To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: FreeBSD7.2 http proxy not reading data off socket. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2012 01:28:58 -0000 We have a http proxy running on FreeBSD 7.2 The problem I am seeing is that the proxy accepts a connection from a client, in this case yy.94.74.40 and receives the GET request for disney.com. The application does not get the data off the socket for several seconds ~90 seconds when it actually responds back to the client. Netstat shows that the proxy which is listening on port 8080 has a full TCP listen queue. netstat -aLn | grep 8080 tcp4 1742/0/8192 127.0.0.1.8080 curl requests from the client result in delays of ~90 seconds. Packet catpure with relevant information. 05:46:19.357420 IP yy.94.74.40.16523 > xx.94.74.43.8080: Flags [S], seq 1120042155, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 40704817 ecr 0], length 0 05:46:19.357438 IP yy.94.74.43.8080 > xx.94.74.40.16523: Flags [S.], seq 3793473898, ack 1120042156, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 1031458393 ecr 40704817], length 0 05:46:19.357669 IP yy.94.74.40.16523 > xx.94.74.43.8080: Flags [.], ack 1, win 16471, options [nop,nop,TS val 40704817 ecr 1031458393], length 0 05:46:19.357684 IP yy.94.74.43.8080 > xx.94.74.40.16523: Flags [.], ack 1, win 16471, options [nop,nop,TS val 1031458393 ecr 40704817], length 0 05:46:19.357686 IP yy.94.74.40.16523 > xx.94.74.43.8080: Flags [P.], seq 1:286, ack 1, win 16471, options [nop,nop,TS val 40704817 ecr 1031458393], length 285 .... GET http://www.disney.com HTTP/1.1 05:46:19.357692 IP yy.94.74.43.8080 > xx.94.74.40.16523: Flags [.], ack 286, win 16435, options [nop,nop,TS val 1031458393 ecr 40704817], length 0 05:47:54.509303 IP yy.94.74.43.8080 > xx.94.74.40.16523: Flags [P.], seq 1:554, ack 286, win 16471, options [nop,nop,TS val 1031467800 ecr 40704817], length 553 ... HTTP/1.1 301 Moved Permanently netstat -an shows that the connection is established and then there is a long delay before the proxy process picks it off the socket. I also have ktrace output which shows kevent being called ~10 times a second but the data is not picked up off the socket. So event though kevent is getting called several times a second the proxy process does not get the data for quite sometime after. 6042 prox 1330523858.652689 CALL kevent(0xa,0x91ab4c8,0x20,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523858.798931 CALL kevent(0xa,0x91ab4c8,0x2e,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523858.894449 CALL kevent(0xa,0x91ab4c8,0x20,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523858.964911 CALL kevent(0xa,0x91ab4c8,0x14,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.051830 CALL kevent(0xa,0x91ab4c8,0xf,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.093017 CALL kevent(0xa,0x91ab4c8,0x26,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.137018 CALL kevent(0xa,0x91ab4c8,0x24,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.138366 CALL kevent(0xa,0x91ab4c8,0x19,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.332275 CALL kevent(0xa,0x91ab4c8,0x17,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.576070 CALL kevent(0xa,0x91ab4c8,0x1f,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.711850 CALL kevent(0xa,0x91ab4c8,0x21,0x91adcc8,0x200,0x80ee858) 6042 prox 1330523859.902660 CALL kevent(0xa,0x91ab4c8,0x2a,0x91adcc8,0x200,0x80ee858) Has anyone observed this behavior before and could suggest what else to look at on the appliance? -- ~Gaurav