From owner-freebsd-net@FreeBSD.ORG Mon Jul 4 20:33:42 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 744001065670 for ; Mon, 4 Jul 2011 20:33:42 +0000 (UTC) (envelope-from gygy@stsnet.ro) Received: from mail.stsnet.ro (mail.stsnet.ro [193.151.31.253]) by mx1.freebsd.org (Postfix) with ESMTP id E5C498FC13 for ; Mon, 4 Jul 2011 20:33:41 +0000 (UTC) Received: from mail.stsnet.ro (localhost.localdomain [127.0.0.1]) by mail.stsnet.ro (Postfix) with ESMTP id 37FD816DDA3; Mon, 4 Jul 2011 23:33:35 +0300 (EEST) Received: from localhost.localdomain [127.0.0.1] by BitDefender SMTP Proxy on localhost.localdomain [127.0.0.1] for localhost.localdomain [127.0.0.1]; Mon, 4 Jul 2011 23:33:35 +0300 (EEST) Received: from mail.stsnet.ro (localhost.localdomain [127.0.0.1]) (Authenticated sender: gygy) by mail.stsnet.ro (Postfix) with ESMTPA id 096AD16DDA0; Mon, 4 Jul 2011 23:33:35 +0300 (EEST) Received: from 188.26.158.192 (SquirrelMail authenticated user gygy) by mail.stsnet.ro with HTTP; Mon, 4 Jul 2011 23:33:35 +0300 Message-ID: In-Reply-To: <20110704191451.GA12372@rdtc.ru> References: <813678a855c90c49bf66c7084f88b45d.squirrel@mail.stsnet.ro> <20110704191451.GA12372@rdtc.ru> Date: Mon, 4 Jul 2011 23:33:35 +0300 From: "Adrian Minta" To: "Eugene Grosbein" User-Agent: SquirrelMail/1.4.20 MIME-Version: 1.0 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BitDefender-Spam: No (0) X-BitDefender-SpamStamp: v1, build 2.8.97.168971, SQMD Hits: none, bayes score: 500(0), pbayes score: 241(0), neunet score: 500(0), flags: [NN_LEGIT_SUMM_400_WORDS], SQMD: ca2a695e53dd054d805c5b76bb284c29.fuzzy.fzrbl.org, total: 0(775) X-BitDefender-CF-Stamp: none X-BitDefender-Scanner: Clean, Agent: BitDefender Smtp Proxy 3.1.0 on mail.stsnet.ro, sigver: 7.38159 Cc: freebsd-net@freebsd.org, Adrian Minta Subject: Re: FreeBSD 8.2 and MPD5 stability issues - update X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jul 2011 20:33:42 -0000 > You should study why existing connections break, > do clients disconnect themselves or server disconnect them? > You'll need turn off detailed logs, read mpd's documentation. > > Also, there are system-wide queues for NETGRAPH messages that can overflow > and that's bad thing. Check them out with command: > > vmstat -z | egrep 'ITEM|NetGraph' > > FAILURES column shows how many times NETGRAPH queues have been overflowed. > One may increase their LIMIT (second column in vmstat's output) > with /boot/loader.conf: > > net.graph.maxdata=65536 > net.graph.maxalloc=65536 > > Eugene Grosbein > > Back to "default" mpd for now. --- /boot/loader.conf --- kern.maxusers=1024 net.graph.maxdata=65536 net.graph.maxalloc=65536 kern.ipc.maxpipekva=320000000 ###8 cores maxthreads=8 net.isr.maxthreads=8 net.isr.bindthreads=1 net.isr.defaultqlimit=8192 net.isr.maxqlimit=10240 console="comconsole,vidconsole" hw.igb.rxd=4096 # IGB Tuning hw.igb.txd=4096 # IGB Tuning if_lagg_load="YES" --- /etc/sysctl.conf --- kern.maxfiles=250000 net.inet.ip.intr_queue_maxlen=10240 kern.ipc.nmbclusters=1280000 kern.ipc.somaxconn=32768 kern.ipc.maxsockbuf=128000000 kern.ipc.maxsockets=128000000 net.local.stream.recvspace=65536 net.local.stream.sendspace=65536 net.local.dgram.recvspace=8000 net.inet.udp.recvspace=262144 net.graph.maxdgram=10240000 net.graph.recvspace=10240000 kern.random.sys.harvest.ethernet=0 net.isr.direct=1 net.isr.direct_force=0 --- When failures start (1 sec interval): --- sessions: 5697 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 14, 8512, 11679211, 0 NetGraph data items: 104, 65540, 0, 899, 18656500, 0 sessions: 5695 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 14, 8512, 11698144, 0 NetGraph data items: 104, 65540, 0, 899, 18690476, 0 sessions: 5696 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 14, 8512, 11717921, 0 NetGraph data items: 104, 65540, 0, 899, 18726539, 0 sessions: 5697 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 8, 8518, 11736938, 0 NetGraph data items: 104, 65540, 0, 899, 18760816, 0 sessions: 5697 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 14, 8512, 11756221, 0 NetGraph data items: 104, 65540, 0, 899, 18796218, 0 sessions: 5697 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 11, 8515, 11775151, 0 NetGraph data items: 104, 65540, 0, 899, 18830631, 0 sessions: 5696 ITEM SIZE LIMIT USED FREE REQUESTS FAILURES NetGraph items: 104, 65540, 9, 8517, 11794388, 0 NetGraph data items: 104, 65540, 0, 899, 18865673, 0 --- mpd.log (via "log -ccp -chat -lcp") --- Jul 4 23:23:37 lns mpd: L2TP: Control connection 0x80a2aaf10 terminated: 0 (no more sessions exist in this tunnel) Jul 4 23:23:37 lns mpd: L2TP: Control connection 0x80a2a8810 terminated: 0 (no more sessions exist in this tunnel) Jul 4 23:23:37 lns mpd: [L19-6291] L2TP: Call #19 connected Jul 4 23:23:37 lns mpd: [L19-6291] Link: UP event Jul 4 23:23:37 lns mpd: [L19-6291] LCP: Up event Jul 4 23:23:37 lns mpd: [L19-6291] LCP: state change Starting --> Req-Sent Jul 4 23:23:37 lns mpd: [L19-6291] LCP: SendConfigReq #1 Jul 4 23:23:37 lns mpd: [L50-6292] L2TP: Call #50 connected Jul 4 23:23:37 lns mpd: [L50-6292] Link: UP event Jul 4 23:23:37 lns mpd: [L50-6292] LCP: Up event Jul 4 23:23:37 lns mpd: [L50-6292] LCP: state change Starting --> Req-Sent Jul 4 23:23:37 lns mpd: [L50-6292] LCP: SendConfigReq #1 Jul 4 23:23:37 lns mpd: [L7-6161] LCP: rec'd Configure Reject #1 (Req-Sent) Jul 4 23:23:37 lns mpd: [L7-6161] Wrong id#, expecting 4 Jul 4 23:23:37 lns mpd: [L50-5752] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L50-5752] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L6-6107] LCP: rec'd Configure Request #1 (Req-Sent) Jul 4 23:23:37 lns mpd: [L6-6107] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L6-6107] LCP: state change Req-Sent --> Ack-Sent Jul 4 23:23:37 lns mpd: [L25-6047] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L25-6047] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L1-5754] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L1-5754] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L26-6108] LCP: rec'd Configure Request #1 (Req-Sent) Jul 4 23:23:37 lns mpd: [L26-6108] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L26-6108] LCP: state change Req-Sent --> Ack-Sent Jul 4 23:23:37 lns mpd: [L1-6109] LCP: rec'd Configure Request #1 (Req-Sent) Jul 4 23:23:37 lns mpd: [L1-6109] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L1-6109] LCP: state change Req-Sent --> Ack-Sent Jul 4 23:23:37 lns mpd: [L3-5880] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L3-5880] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L50-6048] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L50-6048] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L19-5921] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L19-5921] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L25-5753] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L25-5753] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L35-6049] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L35-6049] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L3-5923] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L3-5923] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: L2TP: Control connection 0x80b15f610 10.42.1.19 1701 <-> 10.42.10.4 1701 connected Jul 4 23:23:37 lns mpd: Incoming L2TP packet from 10.42.10.1 1701 Jul 4 23:23:37 lns mpd: [L41-6185] LCP: SendConfigReq #4 Jul 4 23:23:37 lns mpd: [L48-6184] LCP: SendConfigReq #4 Jul 4 23:23:37 lns mpd: [L49-6130] LCP: SendConfigReq #5 Jul 4 23:23:37 lns mpd: [L26-6131] LCP: SendConfigReq #5 Jul 4 23:23:37 lns mpd: [L41-6127] LCP: SendConfigReq #5 Jul 4 23:23:37 lns mpd: [L30-6128] LCP: SendConfigReq #5 Jul 4 23:23:37 lns mpd: [L6-6129] LCP: SendConfigReq #5 Jul 4 23:23:37 lns mpd: [L31-6091] LCP: SendConfigReq #6 Jul 4 23:23:37 lns mpd: [L10-6050] LCP: SendConfigReq #7 Jul 4 23:23:37 lns mpd: [L21-6051] LCP: SendConfigReq #7 Jul 4 23:23:37 lns mpd: [L17-6014] LCP: SendConfigReq #8 Jul 4 23:23:37 lns mpd: [L47-6016] LCP: SendConfigReq #8 Jul 4 23:23:37 lns mpd: [L29-6015] LCP: SendConfigReq #8 Jul 4 23:23:37 lns mpd: [L3-5858] LCP: SendConfigReq #11 Jul 4 23:23:37 lns mpd: [L15-5937] LCP: SendConfigReq #14 Jul 4 23:23:37 lns mpd: [L26-5942] LCP: SendConfigReq #14 Jul 4 23:23:37 lns mpd: [L30-5944] LCP: SendConfigReq #14 Jul 4 23:23:37 lns mpd: [L22-5822] LCP: SendConfigReq #12 Jul 4 23:23:37 lns mpd: [L2-5821] LCP: SendConfigReq #12 Jul 4 23:23:37 lns mpd: [L50-5943] LCP: SendConfigReq #14 Jul 4 23:23:37 lns mpd: [L35-5941] LCP: SendConfigReq #14 Jul 4 23:23:37 lns mpd: [L25-5760] LCP: SendConfigReq #13 Jul 4 23:23:37 lns mpd: [L11-5945] LCP: SendConfigReq #14 Jul 4 23:23:37 lns mpd: [L4-5963] LCP: SendConfigReq #10 Jul 4 23:23:37 lns mpd: [L13-5962] LCP: SendConfigReq #10 Jul 4 23:23:37 lns mpd: [L35-6089] LCP: SendConfigReq #6 Jul 4 23:23:37 lns mpd: [L10-6088] LCP: SendConfigReq #6 Jul 4 23:23:37 lns mpd: [L43-6216] LCP: SendConfigReq #3 Jul 4 23:23:37 lns mpd: [L41-6217] LCP: SendConfigReq #3 Jul 4 23:23:37 lns mpd: L2TP: Incoming call #19 via connection 0x80b15f610 received Jul 4 23:23:37 lns mpd: [L19-6294] L2TP: Incoming call #19 via control connection 0x80b15f610 accepted Jul 4 23:23:37 lns mpd: [L14-6293] L2TP: Call #14 connected Jul 4 23:23:37 lns mpd: [L14-6293] Link: UP event Jul 4 23:23:37 lns mpd: [L14-6293] LCP: Up event Jul 4 23:23:37 lns mpd: [L14-6293] LCP: state change Starting --> Req-Sent Jul 4 23:23:37 lns mpd: [L14-6293] LCP: SendConfigReq #1 Jul 4 23:23:37 lns mpd: [L49-5991] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L49-5991] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L10-6050] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L10-6050] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L8-5924] LCP: rec'd Configure Request #1 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L8-5924] LCP: SendConfigAck #1 Jul 4 23:23:37 lns mpd: [L37-6123] LCP: rec'd Configure Reject #2 (Req-Sent) Jul 4 23:23:37 lns mpd: [L37-6123] Wrong id#, expecting 5 Jul 4 23:23:37 lns mpd: [L47-5867] LCP: rec'd Configure Reject #7 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L47-5867] Wrong id#, expecting 10 Jul 4 23:23:37 lns mpd: [L38-5964] LCP: rec'd Configure Reject #6 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L38-5964] Wrong id#, expecting 9 Jul 4 23:23:37 lns mpd: [L29-5965] LCP: rec'd Configure Reject #6 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L29-5965] Wrong id#, expecting 9 Jul 4 23:23:37 lns mpd: [L1-5783] LCP: rec'd Configure Reject #9 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L1-5783] Wrong id#, expecting 12 Jul 4 23:23:37 lns mpd: [L7-5828] LCP: rec'd Configure Reject #8 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L7-5828] Wrong id#, expecting 11 Jul 4 23:23:37 lns mpd: [L13-5868] LCP: rec'd Configure Reject #7 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L13-5868] Wrong id#, expecting 10 Jul 4 23:23:37 lns mpd: [L14-6003] LCP: rec'd Configure Reject #5 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L14-6003] Wrong id#, expecting 8 Jul 4 23:23:37 lns mpd: [L34-5947] LCP: rec'd Configure Reject #10 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L34-5947] Wrong id#, expecting 13 Jul 4 23:23:37 lns mpd: [L1-5948] LCP: rec'd Configure Reject #10 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L1-5948] Wrong id#, expecting 13 Jul 4 23:23:37 lns mpd: [L32-5826] LCP: rec'd Configure Reject #8 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L32-5826] Wrong id#, expecting 11 Jul 4 23:23:37 lns mpd: [L45-5946] LCP: rec'd Configure Reject #10 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L45-5946] Wrong id#, expecting 13 Jul 4 23:23:37 lns mpd: [L43-5830] LCP: rec'd Configure Reject #8 (Ack-Sent) Jul 4 23:23:37 lns mpd: [L43-5830] Wrong id#, expecting 11 Jul 4 23:23:37 lns mpd: L2TP: Control connection 0x80b15f010 10.42.1.50 1701 <-> 10.42.9.221 1701 connected Jul 4 23:23:37 lns mpd: [L23-6254] LCP: SendConfigReq #2 Jul 4 23:23:37 lns mpd: Incoming L2TP packet from 10.42.9.244 1701