Date: Mon, 27 Mar 2006 13:13:25 +0400 (MSD) From: Anton Yuzhaninov <citrin@rambler-co.ru> To: FreeBSD-gnats-submit@FreeBSD.org Cc: seanc@FreeBSD.org Subject: ports/94997: [patch] port databases/memcached performance (TCP_NODELAY) Message-ID: <200603270913.k2R9DPPk062044@park.rambler.ru> Resent-Message-ID: <200603270920.k2R9KDvn053563@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 94997 >Category: ports >Synopsis: [patch] port databases/memcached performance (TCP_NODELAY) >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-ports-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Mon Mar 27 09:20:13 GMT 2006 >Closed-Date: >Last-Modified: >Originator: Anton Yuzhaninov >Release: FreeBSD 6.1-BETA4 >Organization: "Rambler Internet Holding" OJSC >Environment: Server: FreeBSD clam4.rambler.ru 6.1-BETA4 FreeBSD 6.1-BETA4 #0: Tue Mar 14 13:59:38 UTC 2006 root@wv1u.samsco.home:/usr/obj/usr/src/sys/GENERIC i386 memcached-1.1.12_2 Client: FreeBSD mcgi18.rambler.ru 6.1-BETA4 FreeBSD 6.1-BETA4 #0: Tue Mar 14 13:59:38 UTC 2006 root@wv1u.samsco.home:/usr/obj/usr/src/sys/GENERIC i386 p5-Cache-Memcached-1.15 >Description: memcached has very big response time when reading stored objects size approximately from 1400 to 2800 - when reply from server sent in two ip packets. This caused by badly interaction between TCP delayed acknowledgements (enabled in FreeBSD by default) and Nagle's algorithm. Nagle's explanation of why interaction between delayed ACK and Nagle's algorithm is bad: http://developers.slashdot.org/comments.pl?sid=174457&threshold=1&commentsort=0&mode=thread&cid=14515105 So solution to improve latency is disable Nagle's algorithm for memcached via socket option TCP_NODELAY. This don't done by developers of memcached because on Linux TCP_CORK (analog of TCP_NOPUSH) not compatible with TCP_NODELAY. And on TCP_CORK show on Linux better performance than TCP_NODLEAY >From memcached Changelog: 2003-08-12 (Brad Fitzpatrick) * use TCP_CORK on Linux or TCP_PUSH on BSD * only use TCP_NODELAY when we don't have alternatives 2003-08-10 * disable Nagel's Algorithm (TCP_NODELAY) for better performance (avva) I run tests and it show, that TCP_NODELAY significantly decrease reading response time: http://citrin.ru/stuff/memcached/read.png and practically don't affect writing response time: http://citrin.ru/stuff/memcached/write.png Script used for this test: http://citrin.ru/stuff/memcached/bench.pl In result of using TCP_NODLEAY will be some network overhead, but in environment where usually used memcached (100Mb or 1Gb LAN) it negligible. >How-To-Repeat: Measure response time when reading from memcached objets size 1000, 2000 and 3000 bytes (or in more wide range of sizes), e. g. using this script: http://citrin.ru/stuff/memcached/test.pl >Fix: Apply this patch: --- memcached.c.orig Thu Mar 23 18:51:47 2006 +++ memcached.c Thu Mar 23 18:57:56 2006 @@ -1146,9 +1146,7 @@ setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, &flags, sizeof(flags)); setsockopt(sfd, SOL_SOCKET, SO_KEEPALIVE, &flags, sizeof(flags)); setsockopt(sfd, SOL_SOCKET, SO_LINGER, &ling, sizeof(ling)); -#if !defined(TCP_NOPUSH) setsockopt(sfd, IPPROTO_TCP, TCP_NODELAY, &flags, sizeof(flags)); -#endif /* * the memset call clears nonstandard fields in some impementations >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200603270913.k2R9DPPk062044>