Date: Tue, 25 Dec 2001 03:42:41 -0500 From: "ISPrime Support" <support@isprime.com> To: "ISPrime Support" <support@isprime.com>, <freebsd-stable@freebsd.org> Subject: Re: 4.5-PRERELEASE Major Stalling Message-ID: <000801c18d20$20bb72f0$4300a8c0@winter>
next in thread | raw e-mail | index | archive | help
This diff fixed it, for anyone curious, I don't know why a cvsup didnt get it: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/uipc_socket2.c.diff?r1=1. 78&r2=1.79&f=h --IS ----- Original Message ----- From: "ISPrime Support" <support@isprime.com> To: <freebsd-stable@freebsd.org> Sent: Tuesday, December 25, 2001 2:53 AM Subject: 4.5-PRERELEASE Major Stalling > I work at an ISP which hosts zfreehost.com (a pretty large freehost, pushing > about 300 megabit). > The zfreehost network has a few FBSD servers (all 4.4-STABLE except one > 4.5-PRERELEASE) running a custom in-house webserver, using a FBSD NFS > Storage backend. > Anyway, I upgraded one of the servers to 4.5-PRERELEASE just to test it out > for us, and I noticed there are MAJOR stalls in 4.5. After about 5 minutes > of running, all of the webserver processes (8 processes) are sitting in > state sbwait, doing nothing. in a truss, it like it stalls on the read() > immediately after accept() from the client socket (http). We run httpready > accept filtering, so when we do a read() it should never ever stall since it > wouldn't have accept()'d if there wasnt data to read(), or if it was ready > to close(), anyway -- I consider this to be a pretty major regression. At > the way this is running, we wont be able to run 4.5 on any of the production > servers, so I hope this can get fixed. > Here is what I see from top: > PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND > 266 nobody 2 0 992K 664K sbwait 0:01 0.63% 0.63% pimpd > 267 nobody 2 0 992K 664K sbwait 0:01 0.10% 0.10% pimpd > 264 nobody 2 0 992K 664K sbwait 0:01 0.05% 0.05% pimpd > 86 root 10 0 208K 80K nfsidl 0:00 0.05% 0.05% nfsiod > 123 root 28 0 2348K 1944K RUN 0:01 0.00% 0.00% sshd > 85 root 10 0 208K 80K nfsidl 0:01 0.00% 0.00% nfsiod > 269 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd > 262 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd > 268 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd > 263 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd > 265 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd > > This is what I see from ps auwx: > nobody 262 0.0 0.1 992 664 ?? Is 2:47AM 0:00.37 pimpd > nobody 263 0.0 0.1 992 664 ?? I 2:47AM 0:00.25 pimpd > nobody 264 0.0 0.1 992 664 ?? I 2:47AM 0:01.19 pimpd > nobody 265 0.0 0.1 992 664 ?? I 2:47AM 0:00.13 pimpd > nobody 266 0.0 0.1 992 664 ?? I 2:47AM 0:01.74 pimpd > nobody 267 0.0 0.1 992 664 ?? I 2:47AM 0:01.95 pimpd > nobody 268 0.0 0.1 992 664 ?? I 2:47AM 0:00.30 pimpd > nobody 269 0.0 0.1 992 664 ?? I 2:47AM 0:01.96 pimpd > > I did not enable fxp bundling. > > Here is sysctl -a: > kern.ostype: FreeBSD > kern.osrelease: 4.5-PRERELEASE > kern.osrevision: 199506 > kern.version: FreeBSD 4.5-PRERELEASE #0: Tue Dec 25 02:13:41 EST 2001 > root@zf3:/usr/src/sys/compile/ZFREE > > kern.maxvnodes: 68412 > kern.maxproc: 4116 > kern.maxfiles: 8232 > kern.argmax: 65536 > kern.securelevel: -1 > kern.hostname: zfreeVA3.isprime.com > kern.hostid: 0 > kern.clockrate: { hz = 100, tick = 10000, tickadj = 5, profhz = 1024, stathz > = 128 } > kern.posix1version: 199309 > kern.ngroups: 16 > kern.job_control: 1 > kern.saved_ids: 0 > kern.boottime: { sec = 1009265636, usec = 169977 } Tue Dec 25 02:33:56 2001 > kern.domainname: > kern.osreldate: 450000 > kern.bootfile: /kernel > kern.maxfilesperproc: 7408 > kern.maxprocperuid: 3704 > kern.dumpdev: > kern.ipc.maxsockbuf: 2097152 > kern.ipc.sockbuf_waste_factor: 8 > kern.ipc.somaxconn: 10240 > kern.ipc.max_linkhdr: 16 > kern.ipc.max_protohdr: 40 > kern.ipc.max_hdr: 56 > kern.ipc.max_datalen: 156 > kern.ipc.nmbclusters: 65536 > kern.ipc.semmap: 30 > kern.ipc.semmni: 10 > kern.ipc.semmns: 60 > kern.ipc.semmnu: 30 > kern.ipc.semmsl: 60 > kern.ipc.semopm: 100 > kern.ipc.semume: 10 > kern.ipc.semusz: 92 > kern.ipc.semvmx: 32767 > kern.ipc.semaem: 16384 > kern.ipc.shmmax: 409600000 > kern.ipc.shmmin: 1 > kern.ipc.shmmni: 192 > kern.ipc.shmseg: 128 > kern.ipc.shmall: 8192 > kern.ipc.shm_use_phys: 0 > kern.ipc.mbuf_wait: 32 > kern.ipc.mbtypes: 3589 3759 76 0 0 0 0 0 0 0 0 0 0 0 0 0 > kern.ipc.nmbufs: 262144 > kern.ipc.maxsockets: 65536 > kern.dummy: 0 > kern.ps_strings: 3217031152 > kern.usrstack: 3217031168 > kern.logsigexit: 1 > kern.cam.da.retry_count: 4 > kern.cam.da.default_timeout: 60 > kern.fallback_elf_brand: -1 > kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall > kern.module_path: /;/boot/;/modules/ > kern.acct_suspend: 2 > kern.acct_resume: 4 > kern.acct_chkfreq: 15 > kern.cp_time: 511 0 2075 2732 147211 > kern.timecounter.method: 0 > kern.timecounter.hardware: TSC > kern.openfiles: 52 > kern.kq_calloutmax: 4096 > kern.ps_arg_cache_limit: 256 > kern.ps_argsopen: 1 > kern.fast_vfork: 1 > kern.randompid: 0 > kern.maxusers: 256 > kern.ps_showallprocs: 1 > kern.shutdown.poweroff_delay: 5000 > kern.shutdown.kproc_shutdown_wait: 60 > kern.sugid_coredump: 1 > kern.coredump: 1 > kern.corefile: /tmp/%P.%N.core > kern.quantum: 100000 > kern.ccpu: 1948 > kern.fscale: 2048 > kern.devstat.numdevs: 3 > kern.devstat.generation: 3 > kern.devstat.version: 4 > kern.disks: da0 > kern.log_wakeups_per_second: 5 > kern.msgbuf: > kern.msgbuf_clear: 0 > kern.nselcoll: 0 > kern.consmute: 0 > kern.filedelay: 30 > kern.dirdelay: 29 > kern.metadelay: 28 > kern.minvnodes: 17103 > kern.chroot_allow_open_directories: 1 > vm.loadavg: { 0.01 0.04 0.01 } > vm.v_free_min: 1700 > vm.v_free_target: 7216 > vm.v_free_reserved: 416 > vm.v_inactive_target: 10824 > vm.v_cache_min: 7216 > vm.v_cache_max: 14432 > vm.v_pageout_free_min: 34 > vm.pageout_algorithm: 0 > vm.swap_enabled: 1 > vm.swap_async_max: 4 > vm.swap_idle_threshold1: 2 > vm.swap_idle_threshold2: 10 > vm.v_free_severe: 1058 > vm.stats.sys.v_swtch: 593028 > vm.stats.sys.v_trap: 83568 > vm.stats.sys.v_syscall: 1168881 > vm.stats.sys.v_intr: 1537519 > vm.stats.sys.v_soft: 137125 > vm.stats.vm.v_vm_faults: 84904 > vm.stats.vm.v_cow_faults: 6328 > vm.stats.vm.v_cow_optim: 0 > vm.stats.vm.v_zfod: 12331 > vm.stats.vm.v_ozfod: 12052 > vm.stats.vm.v_swapin: 0 > vm.stats.vm.v_swapout: 0 > vm.stats.vm.v_swappgsin: 0 > vm.stats.vm.v_swappgsout: 0 > vm.stats.vm.v_vnodein: 273 > vm.stats.vm.v_vnodeout: 0 > vm.stats.vm.v_vnodepgsin: 1986 > vm.stats.vm.v_vnodepgsout: 0 > vm.stats.vm.v_intrans: 0 > vm.stats.vm.v_reactivated: 119 > vm.stats.vm.v_pdwakeups: 0 > vm.stats.vm.v_pdpages: 0 > vm.stats.vm.v_dfree: 0 > vm.stats.vm.v_pfree: 15351 > vm.stats.vm.v_tfree: 20808 > vm.stats.vm.v_page_size: 4096 > vm.stats.vm.v_page_count: 257185 > vm.stats.vm.v_free_reserved: 416 > vm.stats.vm.v_free_target: 7216 > vm.stats.vm.v_free_min: 1700 > vm.stats.vm.v_free_count: 135965 > vm.stats.vm.v_wire_count: 29790 > vm.stats.vm.v_active_count: 1179 > vm.stats.vm.v_inactive_target: 10824 > vm.stats.vm.v_inactive_count: 90238 > vm.stats.vm.v_cache_count: 13 > vm.stats.vm.v_cache_min: 7216 > vm.stats.vm.v_cache_max: 14432 > vm.stats.vm.v_pageout_free_min: 34 > vm.stats.vm.v_interrupt_free_min: 2 > vm.stats.vm.v_forks: 320 > vm.stats.vm.v_vforks: 23 > vm.stats.vm.v_rforks: 0 > vm.stats.vm.v_kthreads: 4 > vm.stats.vm.v_forkpages: 20423 > vm.stats.vm.v_vforkpages: 1035 > vm.stats.vm.v_rforkpages: 0 > vm.stats.vm.v_kthreadpages: 0 > vm.stats.misc.zero_page_count: 135553 > vm.stats.misc.cnt_prezero: 203048 > vm.max_proc_mmap: 43690 > vm.max_launder: 32 > vm.pageout_stats_max: 7216 > vm.pageout_full_stats_interval: 20 > vm.pageout_stats_interval: 5 > vm.pageout_stats_free_max: 5 > vm.swap_idle_enabled: 0 > vm.defer_swapspace_pageouts: 0 > vm.disable_swapspace_pageouts: 0 > vm.zone: > ITEM SIZE LIMIT USED FREE REQUESTS > > PIPE: 160, 0, 0, 102, 158 > SWAPMETA: 160, 257185, 0, 0, 0 > ripcb: 192, 65536, 0, 21, 1 > syncache: 160, 15359, 16, 444, 94100 > tcpcb: 544, 65536, 1828, 6597, 92979 > udpcb: 192, 65536, 14, 28, 71 > unpcb: 64, 0, 2, 126, 28 > socket: 192, 65536, 3503, 4966, 93080 > KNOTE: 64, 0, 0, 128, 13 > NFSNODE: 352, 0, 39781, 17, 41857 > NFSMOUNT: 544, 0, 1, 6, 1 > VNODE: 192, 0, 40016, 52, 40016 > NAMEI: 1024, 0, 0, 16, 187371 > VMSPACE: 192, 0, 34, 30, 343 > PROC: 416, 0, 38, 60, 347 > DP fakepg: 64, 0, 0, 0, 0 > PV ENTRY: 28, 1347438, 19011, 243108, 106251 > MAP ENTRY: 48, 0, 391, 162, 15581 > KMAP ENTRY: 48, 64424, 116, 97, 53824 > MAP: 108, 0, 7, 3, 7 > VM OBJECT: 96, 0, 33375, 103, 37919 > vm.zone_kmem_pages: 1538 > vm.zone_kmem_kvaspace: 150491136 > vm.zone_kern_pages: 6330 > vm.kvm_size: 1069547520 > vm.kvm_free: 213909504 > vfs.nfs.realign_test: 216199 > vfs.nfs.realign_count: 0 > vfs.nfs.bufpackets: 4 > vfs.nfs.defect: 0 > vfs.nfs.diskless_valid: 0 > vfs.nfs.diskless_rootpath: > vfs.nfs.diskless_swappath: > vfs.nfs.access_cache_timeout: 600 > vfs.nfs.nfsv3_commit_on_close: 0 > vfs.numdirtybuffers: 3 > vfs.lodirtybuffers: 910 > vfs.hidirtybuffers: 1820 > vfs.numfreebuffers: 7200 > vfs.lofreebuffers: 405 > vfs.hifreebuffers: 810 > vfs.runningbufspace: 0 > vfs.lorunningspace: 524288 > vfs.hirunningspace: 1048576 > vfs.maxbufspace: 118013952 > vfs.hibufspace: 117358592 > vfs.lobufspace: 117293056 > vfs.bufspace: 117293056 > vfs.maxmallocbufspace: 5867929 > vfs.bufmallocspace: 129024 > vfs.getnewbufcalls: 58745 > vfs.getnewbufrestarts: 0 > vfs.vmiodirenable: 1 > vfs.bufdefragcnt: 0 > vfs.buffreekvacnt: 0 > vfs.bufreusecnt: 7159 > vfs.cache.numneg: 118 > vfs.cache.numcache: 40114 > vfs.cache.numcalls: 760858 > vfs.cache.dothits: 112 > vfs.cache.dotdothits: 13703 > vfs.cache.numchecks: 788978 > vfs.cache.nummiss: 45111 > vfs.cache.nummisszap: 8 > vfs.cache.numposzaps: 12 > vfs.cache.numposhits: 701263 > vfs.cache.numnegzaps: 2 > vfs.cache.numneghits: 647 > vfs.cache.numcwdcalls: 32 > vfs.cache.numcwdfail1: 0 > vfs.cache.numcwdfail2: 0 > vfs.cache.numcwdfail3: 0 > vfs.cache.numcwdfail4: 0 > vfs.cache.numcwdfound: 32 > vfs.cache.numfullpathcalls: 0 > vfs.cache.numfullpathfail1: 0 > vfs.cache.numfullpathfail2: 0 > vfs.cache.numfullpathfail3: 0 > vfs.cache.numfullpathfail4: 0 > vfs.cache.numfullpathfound: 0 > vfs.write_behind: 1 > vfs.reassignbufcalls: 493 > vfs.reassignbufloops: 0 > vfs.reassignbufsortgood: 174 > vfs.reassignbufsortbad: 75 > vfs.reassignbufmethod: 1 > vfs.nameileafonly: 0 > vfs.timestamp_precision: 0 > vfs.usermount: 0 > vfs.ffs.doreallocblks: 1 > vfs.ffs.doasyncfree: 1 > net.local.stream.sendspace: 8192 > net.local.stream.recvspace: 8192 > net.local.dgram.maxdgram: 2048 > net.local.dgram.recvspace: 4096 > net.local.inflight: 0 > net.inet.ip.portrange.lowfirst: 1023 > net.inet.ip.portrange.lowlast: 600 > net.inet.ip.portrange.first: 1024 > net.inet.ip.portrange.last: 8000 > net.inet.ip.portrange.hifirst: 49152 > net.inet.ip.portrange.hilast: 65535 > net.inet.ip.forwarding: 0 > net.inet.ip.redirect: 1 > net.inet.ip.ttl: 64 > net.inet.ip.rtexpire: 2400 > net.inet.ip.rtminexpire: 10 > net.inet.ip.rtmaxcache: 128 > net.inet.ip.sourceroute: 0 > net.inet.ip.intr_queue_maxlen: 100 > net.inet.ip.intr_queue_drops: 0 > net.inet.ip.accept_sourceroute: 0 > net.inet.ip.fastforwarding: 0 > net.inet.ip.keepfaith: 0 > net.inet.ip.subnets_are_local: 0 > net.inet.ip.maxfragpackets: 16384 > net.inet.ip.check_interface: 0 > net.inet.icmp.maskrepl: 0 > net.inet.icmp.icmplim: 4000 > net.inet.icmp.drop_redirect: 0 > net.inet.icmp.log_redirect: 0 > net.inet.icmp.bmcastecho: 0 > net.inet.tcp.rfc1323: 1 > net.inet.tcp.rfc1644: 1 > net.inet.tcp.mssdflt: 1460 > net.inet.tcp.keepidle: 7200000 > net.inet.tcp.keepintvl: 75000 > net.inet.tcp.sendspace: 131072 > net.inet.tcp.recvspace: 4096 > net.inet.tcp.keepinit: 75000 > net.inet.tcp.delacktime: 100 > net.inet.tcp.log_in_vain: 0 > net.inet.tcp.blackhole: 0 > net.inet.tcp.delayed_ack: 1 > net.inet.tcp.path_mtu_discovery: 1 > net.inet.tcp.slowstart_flightsize: 1 > net.inet.tcp.local_slowstart_flightsize: 4 > net.inet.tcp.newreno: 1 > net.inet.tcp.tcbhashsize: 512 > net.inet.tcp.do_tcpdrain: 1 > net.inet.tcp.pcbcount: 1827 > net.inet.tcp.icmp_may_rst: 1 > net.inet.tcp.strict_rfc1948: 0 > net.inet.tcp.isn_reseed_interval: 0 > net.inet.tcp.syncookies: 1 > net.inet.tcp.syncache.bucketlimit: 30 > net.inet.tcp.syncache.cachelimit: 15359 > net.inet.tcp.syncache.count: 16 > net.inet.tcp.syncache.hashsize: 512 > net.inet.tcp.syncache.rexmtlimit: 3 > net.inet.tcp.msl: 20000 > net.inet.tcp.always_keepalive: 1 > net.inet.udp.checksum: 1 > net.inet.udp.maxdgram: 9216 > net.inet.udp.recvspace: 41600 > net.inet.udp.log_in_vain: 0 > net.inet.udp.blackhole: 0 > net.inet.accf.unloadable: 0 > net.inet.accf.http.parsehttpversion: 0 > net.inet.raw.maxdgram: 8192 > net.inet.raw.recvspace: 8192 > net.link.generic.system.ifcount: 3 > net.link.ether.inet.prune_intvl: 300 > net.link.ether.inet.max_age: 1200 > net.link.ether.inet.host_down_time: 20 > net.link.ether.inet.maxtries: 5 > net.link.ether.inet.useloopback: 1 > net.link.ether.inet.proxyall: 0 > net.link.ether.inet.log_arp_wrong_iface: 1 > debug.elf_trace: 0 > debug.boothowto: -2147479552 > debug.free_devt: 0 > debug.fdexpand: 4 > debug.sizeof.vnode: 168 > debug.sizeof.proc: 408 > debug.sizeof.specinfo: 68 > debug.sizeof.disklabel: 276 > debug.sizeof.diskslices: 1820 > debug.sizeof.disk: 304 > debug.ttydebug: 0 > debug.nchash: 131071 > debug.ncnegfactor: 16 > debug.numneg: 118 > debug.numcache: 40114 > debug.vfscache: 1 > debug.vnsize: 168 > debug.ncsize: 36 > debug.disablecwd: 0 > debug.disablefullpath: 0 > debug.numvnodes: 40016 > debug.wantfreevnodes: 25 > debug.freevnodes: 26 > debug.rush_requests: 0 > debug.max_softdeps: 547296 > debug.tickdelay: 2 > debug.worklist_push: 0 > debug.blk_limit_push: 0 > debug.ino_limit_push: 0 > debug.blk_limit_hit: 0 > debug.ino_limit_hit: 0 > debug.sync_limit_hit: 0 > debug.indir_blk_ptrs: 0 > debug.inode_bitmap: 0 > debug.direct_blk_ptrs: 0 > debug.dir_entry: 0 > debug.dircheck: 0 > hw.machine: i386 > hw.model: Pentium III/Pentium III Xeon/Celeron > hw.ncpu: 1 > hw.byteorder: 1234 > hw.physmem: 1071288320 > hw.usermem: 949264384 > hw.pagesize: 4096 > hw.floatingpoint: 1 > hw.machine_arch: i386 > hw.instruction_sse: 0 > hw.availpages: 261379 > hw.fxp0.int_delay: 1000 > hw.fxp0.bundle_max: 6 > hw.fxp1.int_delay: 1000 > hw.fxp1.bundle_max: 6 > machdep.consdev: { major = 12, minor = 255 } > machdep.adjkerntz: 0 > machdep.disable_rtc_set: 0 > machdep.wall_cmos_clock: 0 > machdep.do_dump: 1 > machdep.enable_panic_key: 0 > machdep.ispc98: 0 > machdep.msgbuf: > machdep.msgbuf_clear: 0 > machdep.panic_on_nmi: 1 > machdep.i8254_freq: 1193182 > machdep.tsc_freq: 866427923 > user.cs_path: /usr/bin:/bin:/usr/sbin:/sbin: > user.bc_base_max: 99 > user.bc_dim_max: 2048 > user.bc_scale_max: 99 > user.bc_string_max: 1000 > user.coll_weights_max: 0 > user.expr_nest_max: 32 > user.line_max: 2048 > user.re_dup_max: 255 > user.posix2_version: 199212 > user.posix2_c_bind: 0 > user.posix2_c_dev: 0 > user.posix2_char_term: 0 > user.posix2_fort_dev: 0 > user.posix2_fort_run: 0 > user.posix2_localedef: 0 > user.posix2_sw_dev: 0 > user.posix2_upe: 0 > user.stream_max: 20 > user.tzname_max: 255 > p1003_1b.asynchronous_io: 0 > p1003_1b.mapped_files: 0 > p1003_1b.memlock: 0 > p1003_1b.memlock_range: 0 > p1003_1b.memory_protection: 0 > p1003_1b.message_passing: 0 > p1003_1b.prioritized_io: 0 > p1003_1b.priority_scheduling: 1 > p1003_1b.realtime_signals: 0 > p1003_1b.semaphores: 0 > p1003_1b.fsync: 0 > p1003_1b.shared_memory_objects: 0 > p1003_1b.synchronized_io: 0 > p1003_1b.timers: 0 > p1003_1b.aio_listio_max: 0 > p1003_1b.aio_max: 0 > p1003_1b.aio_prio_delta_max: 0 > p1003_1b.delaytimer_max: 0 > p1003_1b.mq_open_max: 0 > p1003_1b.pagesize: 4096 > p1003_1b.rtsig_max: 0 > p1003_1b.sem_nsems_max: 0 > p1003_1b.sem_value_max: 0 > p1003_1b.sigqueue_max: 0 > p1003_1b.timer_max: 0 > jail.set_hostname_allowed: 1 > jail.socket_unixiproute_only: 1 > jail.sysvipc_allowed: 0 > > If anyone has any ideas, please suggest. > > --Isprime Support > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?000801c18d20$20bb72f0$4300a8c0>