From owner-svn-src-head@FreeBSD.ORG Sun Nov 23 03:58:20 2014 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E8277275; Sun, 23 Nov 2014 03:58:19 +0000 (UTC) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6624DDF7; Sun, 23 Nov 2014 03:58:19 +0000 (UTC) Received: by mail-la0-f43.google.com with SMTP id q1so6081719lam.2 for ; Sat, 22 Nov 2014 19:58:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=vQOZ8eJCrVmhzW9ICPRFQ27eeT30M81rk0lUL8YDy2c=; b=QBE3mx7lT+XcA/ep0n4LW2AmZrkSLnNXExSx0fVnXWdiqkpGSLg2ntDUYXZgpNxQ3g hxAcoUv9z294S5XyyP36PKoSGvI9ohKFSPWSc64NJC3NjdRE0Be/70WDZ+vu8tKlL/IT YYt0QVLhOjnfJziE4SyvhdFSied/e4b9p8Ww2JEjjTgN6XtmVKiCTCfhABx0PoaPbuOx eC6C5lHdqdLY1ba502woTFeb46oikjI+kjiax9ydPUWhfwCCOQZH+5SpRUzeE1Vo94Ww KY1u5MIGZ7hoxprV558PHw+58tIC4jSWir0nIHe7qWuL3pgdzDGEwWZLM180ehhiiRNj /8og== X-Received: by 10.152.8.82 with SMTP id p18mr13624388laa.25.1416715097460; Sat, 22 Nov 2014 19:58:17 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by mx.google.com with ESMTPSA id xe10sm2377481lbb.37.2014.11.22.19.58.15 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 22 Nov 2014 19:58:16 -0800 (PST) Sender: Alexander Motin Message-ID: <54715B56.8010905@FreeBSD.org> Date: Sun, 23 Nov 2014 05:58:14 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r274853 - in head: sys/dev/iscsi usr.sbin/ctld usr.sbin/iscsid References: <201411221509.sAMF9JIh016800@svn.freebsd.org> <20141122180318.GA3921@ox> In-Reply-To: <20141122180318.GA3921@ox> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2014 03:58:20 -0000 On 22.11.2014 20:03, Navdeep Parhar wrote: > On Sat, Nov 22, 2014 at 03:09:19PM +0000, Alexander Motin wrote: >> Author: mav >> Date: Sat Nov 22 15:09:18 2014 >> New Revision: 274853 >> URL: https://svnweb.freebsd.org/changeset/base/274853 >> >> Log: >> For both iSCSI initiator and target increase socket buffer sizes before >> establishing connection. >> >> This is a workaround for Chelsio TOE driver, that does not update socket >> buffer size in hardware after connection established, and unless that is >> done beforehand, kernel code will stuck, attempting to send/receive full >> PDU at once. > > cxgbe's TOE driver does update the rcv window in the hardware in > response to changes in the socket buffer size (see t4_rcvd in > cxgbe/tom/t4_cpl_io.c). > > The bug is that the driver tries to avoid sending too many small updates > to the chip, and instead tries to build up a balance of credits before > an update. This doesn't interact well with some applications that set > the sockbuf size to a high fixed value and then wait for the entire > buffer to fill up before draining it. The driver ends up waiting for > the application to drain the socket's receive buffer so that pru_rcvd > (t4_rcvd) can then send a big batch of credits to the chip, the > application is waiting for the receive to reach its high water mark > before draining the buffer, but the TOE on the chip has shut down the > receive window because it's out of credits. If you see the peer sending > zero window probes and the TOE refusing to budge, then the problem > you've observed is definitely the problem I just described. Yes, it looks like this. You may find the packet dump with the problem here: https://people.freebsd.org/~mav/rcvbuf.dump > I'll fix this in the TOE driver. I can send you an early patch in case > you'd like to try out the fix before it gets committed. Thank you. I'll be happy to try it. -- Alexander Motin