From owner-svn-src-all@FreeBSD.ORG Thu Jun 9 13:10:14 2011 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E29231065677; Thu, 9 Jun 2011 13:10:14 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id D92628FC12; Thu, 9 Jun 2011 13:10:13 +0000 (UTC) Received: by fxm11 with SMTP id 11so1440539fxm.13 for ; Thu, 09 Jun 2011 06:10:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:cc:subject:references:x-comment-to :sender:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=/jk4NV45LFCa49bAfXZm7WT3Hf+A68C1vsqxouv0l1E=; b=xpRUtogAOP8XoCVHjoajfY6QuZAD93jMRFvanx9hvJytgRNhJovVvg8kuxnZ/zlYnc nBW6JiJaH0l289TWWi86n4iErGsIdZnxZhjxVAfyi+k4YgaVeNhl4cnNw8hbvrlrD7f6 eUZ+anlcO3ldGrhO5iYUz48tvjFNU+9fmf8ts= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:sender:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=K0gltgUubEdpE38NJWi4Z2kfR4Vs9d27myx1LXS2VMKhaF7BQ9hgsszfV9El1ZujWW cyfIjn9INIKajmywOoZLuOJwnN7vgUq1tms9/yyh6tBnASXmkMzdt4lLM0UfKQheqyU0 Blg0zIdXqi8IqfspMqPA5KH9KWau7EYlCxIRA= Received: by 10.223.156.9 with SMTP id u9mr802093faw.70.1307625012597; Thu, 09 Jun 2011 06:10:12 -0700 (PDT) Received: from localhost ([95.69.172.154]) by mx.google.com with ESMTPS id y7sm652730fak.7.2011.06.09.06.10.09 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 09 Jun 2011 06:10:10 -0700 (PDT) From: Mikolaj Golub To: Maxim Sobolev References: <201106041601.p54G1Ut7016697@svn.freebsd.org> <4DEA653F.7070503@FreeBSD.org> <201106061057.p56Av3u7037614@kernblitz.nuclight.avtf.net> <4DED1CC5.1070001@FreeBSD.org> X-Comment-To: Maxim Sobolev Sender: Mikolaj Golub Date: Thu, 09 Jun 2011 16:10:08 +0300 In-Reply-To: <4DED1CC5.1070001@FreeBSD.org> (Maxim Sobolev's message of "Mon, 06 Jun 2011 11:30:29 -0700") Message-ID: <86wrgvkv67.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: vadim_nuclight@mail.ru, Kostik Belousov , svn-src-all@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: svn commit: r222688 - head/sbin/hastd X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jun 2011 13:10:15 -0000 On Mon, 06 Jun 2011 11:30:29 -0700 Maxim Sobolev wrote: MS> On 6/6/2011 3:57 AM, Vadim Goncharov wrote: >> Hi Maxim Sobolev! >> >> On Sat, 04 Jun 2011 10:02:55 -0700; Maxim Sobolev wrote: >> >>>> I don't know about the hast internal protocol but the above reads kind of >>>> wrong to me. >> >>> Hmm, not sure what exactly is wrong? Sender does 3 writes to the TCP >>> socket - 32k, 32k and 1071 bytes, while receiver does one >>> recv(MSG_WAITALL) with the size of 66607. So I suspect sender's kernel >>> does deliver two 32k packets and fills up receiver's buffer or >>> something. And the remaining 1071 bytes stay somewhere in sender's >>> kernel indefinitely, while recv() cannot complete in receiver's. Using >>> the same size when doing recv() solves the issue for me. With MSG_WAITALL, if data to receive are larger than receive buffer, after receiving some part of data it is drained to user buffer and the protocol is notified (sending window update) that there is some space in the receive buffer. So, normally, there should not be an issue with the scenario described above. But there was a race in soreceive_generic(), I believe I have fixed in r222454, when the connection could stall in sbwait. Do you still observe the issue with only r222454 applied? >> >> I'm also don't know the hast internal protocol, but the very need to adjust >> some *user* buffers while using _TCP_ is pretty strange: TCP doesn't depend on >> sender's behavior only. May be setsockopt(SO_RCVBUF) needs to be used. Also, >> why recv() is ever there on TCP, instead of read() ? Is that blocking or >> non-blocking read? In the latter case kqueue(2) is very usfeul. >> MS> MSG_WAITALL might be an issue here. I suspect receiver's kernel can't MS> dequeue two 32k packets until the last chunk arrives. I don't have a MS> time to look into it in detail unfortunately. Sorry, but I think your patch is wrong. If even it fixes the issue for you, actually I think it does not fix but hides a real problem we have to address. Receiving the whole chunk at once should be more effectively because we do one syscall instead of several. Also, if you receive in smaller chunks no need to set MSG_WAITALL at all. Besides, with your patch I am observing hangs on primary startup in init_remote->primary_connect->proto_connection_recv->proto_common_recv The primary worker process asks the parent to connect to the secondary. After establishing the connection the parent sends connection protocol name and descriptor to the worker (proto_connection_send/proto_connection_recv). The issue here is that in proto_connection_recv() the size of protoname is unknown, so it calls proto_common_recv() with size = 127, larger than protoname ("tcp"). It worked previously because after sending protoname proto_connection_send() sends the descriptor calling sendmsg(). This is data of different type and it makes recv() return although only 4 bytes of 127 requested were received. With your patch, after receiving these 4 bytes it returns back to recv() waiting for rest 123 bytes and gets stuck forever. Don't you observe this? It is strange, because for me it hangs on every start up. I am seeing this on yesterday current. -- Mikolaj Golub