From owner-freebsd-net@FreeBSD.ORG Wed Feb 25 07:38:28 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 828A7106566B for ; Wed, 25 Feb 2009 07:38:28 +0000 (UTC) (envelope-from siquijorphilips@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.170]) by mx1.freebsd.org (Postfix) with ESMTP id 59AFC8FC13 for ; Wed, 25 Feb 2009 07:38:28 +0000 (UTC) (envelope-from siquijorphilips@gmail.com) Received: by wf-out-1314.google.com with SMTP id 27so3044425wfd.7 for ; Tue, 24 Feb 2009 23:38:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=gRGcIt9IJ9FXwNq9TN8cHAHbrB/UWtw2VQyesROfBPs=; b=TkTzHaHHraUGA0AKSGt/V+lHM+XD2PHDFjDToeEP0W83UG9BEXhvlTyMjm3EKg8Wbl VppOyvaAGeOJZoeyX/+kbsWMTAVsoM+9FjUSuFHhA2Y+ua4tc2zepJ2OZ9zPuxe4ekZm 0yG8ICwuuGTDIES0LMt+j3Z2d+bKwQAtaap9o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=wyerLQgsio+RfQ7nstlrAHYkMrsCqqWeaXk1xhu9XQqw7WLHy8IiqXCyiU0z7AnHu4 aRhXMyuARgTfslNqraiec//guKOFVgXkMGmRneCj34EmXiZcMC2ULYeo2hWjAs3WCSco cWxRMZBx5SrPrAHwxxWiFbPhcyY7Mvsmf7l0o= MIME-Version: 1.0 Received: by 10.142.217.17 with SMTP id p17mr2991190wfg.32.1235546051790; Tue, 24 Feb 2009 23:14:11 -0800 (PST) Date: Wed, 25 Feb 2009 15:14:11 +0800 Message-ID: From: Siquijor Philips To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Questions on processing smaller frame size X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Feb 2009 07:38:28 -0000 Can someone explain why FreeBSD network throughput (both incoming and outgoing traffic) decreases when smaller frame size being processed? With smaller frame size, corresponding packet rate (packet/sec or pps) increases and experiencing dropped packets. What causes dropping of packets with small frame size? My system is running on a 2x Quad-Core Intel Xeon machine with FreeBSD-7.1 RELEASE using 1-Gigabit 2-port Chelsio NIC connected on a Spirent traffic generator. Smaller frame sizes are tested on 64-byte, 128-byte, 256-byte and 512-byte. Default frame size of 1500-byte which is the default of an Ethernet is working fine. Below are my data; # ifconfig -m cxgb0: flags=8843 metric 0 mtu 1500 options=5bb capabilities=7bb ether 00:07:43:05:7b:ac inet 192.168.0.1 netmask 0xffffe000 broadcast 192.168.255.255 inet6 fe80::207:43ff:fe05:7bac%cxgb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseTX ) status: active supported media: media autoselect media 1000baseTX mediaopt full-duplex media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP cxgb1: flags=8843 metric 0 mtu 1500 options=5bb capabilities=7bb ether 00:07:43:05:7b:ad inet 10.0.0.1 netmask 0xfff00000 broadcast 10.0.255.255 inet6 fe80::207:43ff:fe05:7bad%cxgb1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (1000baseTX ) status: active supported media: media autoselect media 1000baseTX mediaopt full-duplex media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP 1500-byte frame size -> OK --------------------- bwm-ng v0.6 (probing every 0.500s), press 'h' for help input: getifaddrs type: rate \ iface Rx Tx Total cxgb0: 938.58 Mb/s 938.58 Mb/s 1.83 Gb/s cxgb1: 938.28 Mb/s 938.58 Mb/s 1.83 Gb/s lo0: 0.00 b/s 0.00 b/s 0.00 b/s total: 1.83 Gb/s 1.83 Gb/s 3.67 Gb/s # netstat -I cxgb0 -w 1 -d input (cxgb0) output packets errs bytes packets errs bytes colls drops 82234 0 123059464 82229 0 123059464 0 0 82219 0 123031040 82240 0 123031040 0 0 82201 0 123017576 82187 0 123019072 0 0 82218 0 123025056 82242 0 123025056 0 0 82214 0 123028048 82247 0 123023560 0 0 82210 0 123023560 82209 0 123022064 0 0 82210 0 123020568 82244 0 123022064 0 0 82210 0 123023560 82226 0 123023560 0 0 82240 0 123057968 82208 0 123051984 0 0 82111 0 122870968 82161 0 122876952 0 0 82299 0 123158200 82312 0 123156704 0 0 82208 0 123013088 82226 0 123013088 0 0 82205 0 123022064 82229 0 123019072 0 0 82224 0 123028048 82213 0 123026552 0 0 82219 0 123025056 82252 0 123026552 0 0 82212 0 123023560 82218 0 123025056 0 0 82223 0 123026552 82231 0 123023560 0 0 82218 0 123023560 82212 0 123022064 0 0 82214 0 123028048 82252 0 123028048 0 0 82214 0 123023560 82226 0 123023560 0 0 # netstat -I cxgb1 -w 1 -d input (cxgb1) output packets errs bytes packets errs bytes colls drops 82220 0 122996632 82261 0 123029544 0 0 82211 0 122981672 82204 0 123022064 0 0 82207 0 122971200 82243 0 123022064 0 0 82215 0 122990648 82235 0 123022064 0 0 82220 0 122987656 82222 0 123029544 0 0 82209 0 122974192 82247 0 123025056 0 0 82215 0 122978680 82190 0 123025056 0 0 82226 0 122981672 82250 0 123023560 0 0 82214 0 122984664 82229 0 123022064 0 0 82217 0 122987656 82211 0 123025056 0 0 82214 0 122984664 82226 0 123023560 0 0 82224 0 122992144 82218 0 123023560 0 0 82208 0 122963720 82236 0 123025056 0 0 82218 0 122980176 82233 0 123022064 0 0 82217 0 122986160 82219 0 123028048 0 0 82215 0 122983168 82249 0 123023560 0 0 82208 0 122977184 82192 0 123025056 0 0 82214 0 122969704 82231 0 123019072 0 0 82214 0 122990648 82249 0 123031040 0 0 82210 0 122965216 82207 0 123019072 0 0 512-byte frame size: --------------------- bwm-ng v0.6 (probing every 0.500s), press 'h' for help input: getifaddrs type: rate | iface Rx Tx Total cxgb0: 911.85 Mb/s 407.88 Mb/s 1.29 Gb/s cxgb1: 898.39 Mb/s 318.05 Mb/s 1.19 Gb/s lo0: 0.00 b/s 0.00 b/s 0.00 b/s total: 1.77 Gb/s 725.93 Mb/s 2.48 Gb/s # netstat -I cxgb0 -w 1 -d input (cxgb0) output packets errs bytes packets errs bytes colls drops 234410 0 120264428 115185 0 58832496 0 120569 230622 0 118386352 106160 0 53905404 0 126996 232698 0 119392700 105872 0 53480716 0 129205 233396 0 119755412 111680 0 57011824 0 123326 232678 0 119324120 110720 0 56231028 0 123968 231795 0 118987824 108736 0 54822344 0 125823 232432 0 119239284 106464 0 54465220 0 127468 233480 0 119887492 112160 0 56965088 0 123223 231789 0 118927372 106816 0 53860700 0 128085 232554 0 119338852 106486 0 54217316 0 127951 232572 0 119318024 103850 0 52624736 0 131009 235524 0 120803924 117382 0 59691524 0 119868 229808 0 117925596 108954 0 55370476 0 123077 234391 0 120326912 105216 0 53702712 0 130974 230581 0 118323360 101312 0 51453796 0 131192 233676 0 119798592 105152 0 53400960 0 130652 232868 0 119469408 100992 0 51286664 0 133883 231428 0 118835932 110791 0 56069484 0 123184 234807 0 120577864 111481 0 56814212 0 125395 231241 0 118634256 107904 0 54803548 0 125396 # netstat -I cxgb1 -w 1 -d input (cxgb1) output packets errs bytes packets errs bytes colls drops 231385 0 117446044 79040 0 40151304 0 155399 231482 0 117471952 84192 0 42314876 0 150946 234274 0 118907560 82720 0 42464736 0 153623 229963 0 116715540 78656 0 39487856 0 154837 232107 0 117802152 79146 0 40233092 0 155670 231872 0 117672104 87101 0 44432220 0 147407 232063 0 117799612 80281 0 41005760 0 154019 232221 0 117875812 79574 0 40224456 0 155825 232181 0 117840252 86474 0 44109640 0 148175 233005 0 118255288 79648 0 40453056 0 155805 231327 0 117362224 84791 0 42597324 0 150323 233615 0 118578376 83305 0 42775632 0 152044 232620 0 118073932 88704 0 45054520 0 146961 231816 0 117679724 81312 0 41300908 0 153107 230951 0 117200680 85984 0 43672252 0 147643 232828 0 118150132 81984 0 41639236 0 153747 230936 0 117206776 81896 0 41540176 0 151850 232234 0 117866160 80696 0 40661844 0 154869 234465 0 119013224 90782 0 46159928 0 146240 230022 0 116739924 78224 0 39735760 0 154385 256-byte frame size: -------------------- bwm-ng v0.6 (probing every 0.500s), press 'h' for help input: getifaddrs type: rate / iface Rx Tx Total cxgb0: 516.81 Mb/s 89.50 Mb/s 606.31 Mb/s cxgb1: 510.47 Mb/s 0.00 b/s 510.47 Mb/s lo0: 0.00 b/s 0.00 b/s 0.00 b/s total: 1.00 Gb/s 89.50 Mb/s 1.09 Gb/s # netstat -I cxgb0 -w 1 -d input (cxgb0) output packets errs bytes packets errs bytes colls drops 262028 0 66449124 46674 0 11973780 0 215945 265254 0 67428648 45774 0 11543364 0 220970 263969 0 67080384 41728 0 10509156 0 224438 260856 0 66195864 48960 0 12336156 0 213067 280701 0 71610840 73376 0 18483696 0 207469 286981 0 73295964 101984 0 25687368 0 188828 291674 0 74534292 98048 0 24522624 0 197495 275561 0 70195608 64128 0 16142616 0 214413 299603 0 76662180 123520 0 31266900 0 178712 300192 0 76816656 127680 0 32177628 0 176512 301237 0 77091084 134968 0 33967584 0 170384 297626 0 76147344 127505 0 32120424 0 174518 272750 0 69459012 50775 0 12817224 0 223945 269337 0 68588604 48128 0 12125232 0 223996 272580 0 69435576 51840 0 13059396 0 223085 284996 0 72806580 88704 0 22343328 0 198919 269933 0 68693184 50304 0 12672072 0 222521 271766 0 69234984 61376 0 15460956 0 212545 279384 0 71270640 77920 0 19626768 0 204539 279604 0 71364132 80896 0 20377476 0 201143 # netstat -I cxgb1 -w 1 -d input (cxgb1) output packets errs bytes packets errs bytes colls drops 262519 0 66155544 1016 0 256032 0 264350 264428 0 66628548 2032 0 512316 0 267643 280279 0 70714980 0 0 0 0 286592 278263 0 70213248 0 0 0 0 282679 270101 0 68067468 0 0 0 0 273504 270874 0 68286708 0 0 0 0 274618 273371 0 68894280 0 0 0 0 277436 268266 0 67634532 0 0 0 0 272268 272676 0 68751144 0 0 0 0 276489 266928 0 67281984 0 0 0 0 269646 261617 0 65890692 0 0 0 0 264105 262947 0 66242484 0 0 0 0 265603 262054 0 66049956 0 0 0 0 264045 263196 0 66303720 0 0 0 0 267500 270744 0 68263020 0 0 0 0 275050 278268 0 70206444 0 0 0 0 283764 266665 0 67211676 0 0 0 0 268860 263425 0 66380076 0 0 0 0 266193 262424 0 66126060 0 0 0 0 265852 266928 0 67285512 0 0 0 0 271427 128-byte frame size: -------------------- bwm-ng v0.6 (probing every 0.500s), press 'h' for help input: getifaddrs type: rate | iface Rx Tx Total cxgb0: 253.88 Mb/s 39.65 Mb/s 293.54 Mb/s cxgb1: 251.18 Mb/s 3.79 Mb/s 254.97 Mb/s lo0: 0.00 b/s 0.00 b/s 0.00 b/s total: 505.06 Mb/s 43.44 Mb/s 548.51 Mb/s # netstat -I cxgb0 -w 1 -d input (cxgb0) output packets errs bytes packets errs bytes colls drops 272995 0 34151336 50464 0 6256048 0 222873 283224 0 35538152 71520 0 8865256 0 213793 282820 0 35460900 71680 0 8811688 0 214643 270570 0 33850884 48800 0 6041280 0 223482 268077 0 33469088 45440 0 5715780 0 225437 266423 0 33251840 52160 0 6466104 0 214914 267926 0 33494756 47680 0 5910212 0 221368 268975 0 33653476 44160 0 5474476 0 225070 273065 0 34201184 51072 0 6331936 0 223594 281093 0 35278868 70176 0 8698228 0 213052 290242 0 36473856 91424 0 11332732 0 201661 275838 0 34575664 59744 0 7405280 0 219405 271987 0 34020764 57984 0 7187412 0 216945 263805 0 32904640 50048 0 6204960 0 215690 266295 0 33242540 44192 0 5478940 0 223407 266219 0 33201992 45822 0 5629476 0 221918 264204 0 32959448 49090 0 6137256 0 214943 269254 0 33687948 43840 0 5435664 0 226363 270364 0 33849396 46016 0 5704992 0 225344 270689 0 33905196 50016 0 6199256 0 221875 # netstat -I cxgb1 -w 1 -d input (cxgb1) output packets errs bytes packets errs bytes colls drops 273869 0 33979720 0 0 0 0 278574 275475 0 34184444 0 0 0 0 280288 279181 0 34661100 0 0 0 0 284236 280953 0 34866816 0 0 0 0 286163 283458 0 35206824 0 0 0 0 289747 273164 0 33895400 0 0 0 0 277772 279365 0 34671392 0 0 0 0 283792 289865 0 35997696 0 0 0 0 296676 268840 0 33355256 0 0 0 0 273146 272989 0 33879528 0 0 0 0 277794 278536 0 34572192 0 0 0 0 283140 278738 0 34598108 0 0 0 0 284031 273043 0 33875436 0 0 0 0 277845 271792 0 33716592 0 0 0 0 275696 278252 0 34531644 0 0 0 0 283087 279277 0 34662216 0 0 0 0 284803 284664 0 35342356 0 0 0 0 291254 282126 0 35033100 0 0 0 0 288165 276860 0 34358044 0 0 0 0 281503 278605 0 34582236 0 0 0 0 284071 64-byte frame size: -------------------- bwm-ng v0.6 (probing every 0.500s), press 'h' for help input: getifaddrs type: rate - iface Rx Tx Total cxgb0: 126.25 Mb/s 37.31 Mb/s 163.56 Mb/s cxgb1: 124.84 Mb/s 6.42 Mb/s 131.26 Mb/s lo0: 0.00 b/s 0.00 b/s 0.00 b/s total: 251.09 Mb/s 43.73 Mb/s 294.82 Mb/s # netstat -I cxgb0 -w 1 -d input (cxgb0) output packets errs bytes packets errs bytes colls drops 271659 0 16462260 0 0 4754160 0 194931 288256 0 17544540 0 0 6974640 0 175370 284450 0 17308560 0 0 7179120 0 168900 278013 0 16872960 0 0 5713800 0 187902 269222 0 16316160 0 0 4139760 0 200472 277638 0 16865280 0 0 6381600 0 174917 275845 0 16747800 0 0 5573220 0 187276 270689 0 16414200 0 0 4324140 0 201575 264941 0 16023360 0 0 3646020 0 205483 274697 0 16660320 0 0 5223120 0 189366 272307 0 16506180 0 0 5105520 0 190638 272725 0 16535220 0 0 4841400 0 194902 273420 0 16582560 0 0 5320260 0 188407 272601 0 16535640 0 0 4854420 0 194869 273774 0 16602060 0 0 4998180 0 193173 274004 0 16624800 0 0 5591220 0 185281 270268 0 16377120 0 0 4653120 0 194185 270037 0 16367460 0 0 4345560 0 199280 269547 0 16324920 0 0 4917600 0 190163 272384 0 16512600 0 0 5144160 0 188393 # netstat -I cxgb1 -w 1 -d input (cxgb1) output packets errs bytes packets errs bytes colls drops 272277 0 16347120 0 0 0 0 276873 276497 0 16610580 0 0 0 0 281353 270445 0 16240320 0 0 0 0 274578 275056 0 16522080 0 0 0 0 279691 277819 0 16690500 0 0 0 0 283827 277585 0 16672740 0 0 0 0 280820 282193 0 16963740 0 0 0 0 287826 270417 0 16233420 0 0 121380 0 271558 268628 0 16127280 0 0 0 0 273754 280227 0 16836120 0 0 0 0 285798 267618 0 16059900 0 0 0 0 271038 274748 0 16502460 0 0 0 0 279588 276467 0 16606200 0 0 0 0 280183 273376 0 16416900 0 0 0 0 277882 269455 0 16176780 0 0 0 0 273417 273813 0 16442760 0 0 0 0 277677 273845 0 16445760 0 0 0 0 276387 276677 0 16621920 0 0 0 0 282040 275206 0 16532700 0 0 0 0 280072 271826 0 16327200 0 0 0 0 276746 Regards, Siquijor