From owner-freebsd-stable Mon Dec 24 23:56:18 2001 Delivered-To: freebsd-stable@freebsd.org Received: from acura.isprime.com (acura.isprime.com [130.94.138.66]) by hub.freebsd.org (Postfix) with ESMTP id 55CF837B419 for ; Mon, 24 Dec 2001 23:55:59 -0800 (PST) Received: from winter (localhost [127.0.0.1]) by acura.isprime.com (8.11.3/8.11.2) with SMTP id fBP7twD99232 for ; Tue, 25 Dec 2001 02:55:58 -0500 (EST) Message-ID: <000501c18d19$4c8dba20$4300a8c0@winter> From: "ISPrime Support" To: Subject: 4.5-PRERELEASE Major Stalling Date: Tue, 25 Dec 2001 02:53:53 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 Disposition-Notification-To: "ISPrime Support" X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I work at an ISP which hosts zfreehost.com (a pretty large freehost, pushing about 300 megabit). The zfreehost network has a few FBSD servers (all 4.4-STABLE except one 4.5-PRERELEASE) running a custom in-house webserver, using a FBSD NFS Storage backend. Anyway, I upgraded one of the servers to 4.5-PRERELEASE just to test it out for us, and I noticed there are MAJOR stalls in 4.5. After about 5 minutes of running, all of the webserver processes (8 processes) are sitting in state sbwait, doing nothing. in a truss, it like it stalls on the read() immediately after accept() from the client socket (http). We run httpready accept filtering, so when we do a read() it should never ever stall since it wouldn't have accept()'d if there wasnt data to read(), or if it was ready to close(), anyway -- I consider this to be a pretty major regression. At the way this is running, we wont be able to run 4.5 on any of the production servers, so I hope this can get fixed. Here is what I see from top: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 266 nobody 2 0 992K 664K sbwait 0:01 0.63% 0.63% pimpd 267 nobody 2 0 992K 664K sbwait 0:01 0.10% 0.10% pimpd 264 nobody 2 0 992K 664K sbwait 0:01 0.05% 0.05% pimpd 86 root 10 0 208K 80K nfsidl 0:00 0.05% 0.05% nfsiod 123 root 28 0 2348K 1944K RUN 0:01 0.00% 0.00% sshd 85 root 10 0 208K 80K nfsidl 0:01 0.00% 0.00% nfsiod 269 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd 262 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd 268 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd 263 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd 265 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd This is what I see from ps auwx: nobody 262 0.0 0.1 992 664 ?? Is 2:47AM 0:00.37 pimpd nobody 263 0.0 0.1 992 664 ?? I 2:47AM 0:00.25 pimpd nobody 264 0.0 0.1 992 664 ?? I 2:47AM 0:01.19 pimpd nobody 265 0.0 0.1 992 664 ?? I 2:47AM 0:00.13 pimpd nobody 266 0.0 0.1 992 664 ?? I 2:47AM 0:01.74 pimpd nobody 267 0.0 0.1 992 664 ?? I 2:47AM 0:01.95 pimpd nobody 268 0.0 0.1 992 664 ?? I 2:47AM 0:00.30 pimpd nobody 269 0.0 0.1 992 664 ?? I 2:47AM 0:01.96 pimpd I did not enable fxp bundling. Here is sysctl -a: kern.ostype: FreeBSD kern.osrelease: 4.5-PRERELEASE kern.osrevision: 199506 kern.version: FreeBSD 4.5-PRERELEASE #0: Tue Dec 25 02:13:41 EST 2001 root@zf3:/usr/src/sys/compile/ZFREE kern.maxvnodes: 68412 kern.maxproc: 4116 kern.maxfiles: 8232 kern.argmax: 65536 kern.securelevel: -1 kern.hostname: zfreeVA3.isprime.com kern.hostid: 0 kern.clockrate: { hz = 100, tick = 10000, tickadj = 5, profhz = 1024, stathz = 128 } kern.posix1version: 199309 kern.ngroups: 16 kern.job_control: 1 kern.saved_ids: 0 kern.boottime: { sec = 1009265636, usec = 169977 } Tue Dec 25 02:33:56 2001 kern.domainname: kern.osreldate: 450000 kern.bootfile: /kernel kern.maxfilesperproc: 7408 kern.maxprocperuid: 3704 kern.dumpdev: kern.ipc.maxsockbuf: 2097152 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.somaxconn: 10240 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 40 kern.ipc.max_hdr: 56 kern.ipc.max_datalen: 156 kern.ipc.nmbclusters: 65536 kern.ipc.semmap: 30 kern.ipc.semmni: 10 kern.ipc.semmns: 60 kern.ipc.semmnu: 30 kern.ipc.semmsl: 60 kern.ipc.semopm: 100 kern.ipc.semume: 10 kern.ipc.semusz: 92 kern.ipc.semvmx: 32767 kern.ipc.semaem: 16384 kern.ipc.shmmax: 409600000 kern.ipc.shmmin: 1 kern.ipc.shmmni: 192 kern.ipc.shmseg: 128 kern.ipc.shmall: 8192 kern.ipc.shm_use_phys: 0 kern.ipc.mbuf_wait: 32 kern.ipc.mbtypes: 3589 3759 76 0 0 0 0 0 0 0 0 0 0 0 0 0 kern.ipc.nmbufs: 262144 kern.ipc.maxsockets: 65536 kern.dummy: 0 kern.ps_strings: 3217031152 kern.usrstack: 3217031168 kern.logsigexit: 1 kern.cam.da.retry_count: 4 kern.cam.da.default_timeout: 60 kern.fallback_elf_brand: -1 kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall kern.module_path: /;/boot/;/modules/ kern.acct_suspend: 2 kern.acct_resume: 4 kern.acct_chkfreq: 15 kern.cp_time: 511 0 2075 2732 147211 kern.timecounter.method: 0 kern.timecounter.hardware: TSC kern.openfiles: 52 kern.kq_calloutmax: 4096 kern.ps_arg_cache_limit: 256 kern.ps_argsopen: 1 kern.fast_vfork: 1 kern.randompid: 0 kern.maxusers: 256 kern.ps_showallprocs: 1 kern.shutdown.poweroff_delay: 5000 kern.shutdown.kproc_shutdown_wait: 60 kern.sugid_coredump: 1 kern.coredump: 1 kern.corefile: /tmp/%P.%N.core kern.quantum: 100000 kern.ccpu: 1948 kern.fscale: 2048 kern.devstat.numdevs: 3 kern.devstat.generation: 3 kern.devstat.version: 4 kern.disks: da0 kern.log_wakeups_per_second: 5 kern.msgbuf: kern.msgbuf_clear: 0 kern.nselcoll: 0 kern.consmute: 0 kern.filedelay: 30 kern.dirdelay: 29 kern.metadelay: 28 kern.minvnodes: 17103 kern.chroot_allow_open_directories: 1 vm.loadavg: { 0.01 0.04 0.01 } vm.v_free_min: 1700 vm.v_free_target: 7216 vm.v_free_reserved: 416 vm.v_inactive_target: 10824 vm.v_cache_min: 7216 vm.v_cache_max: 14432 vm.v_pageout_free_min: 34 vm.pageout_algorithm: 0 vm.swap_enabled: 1 vm.swap_async_max: 4 vm.swap_idle_threshold1: 2 vm.swap_idle_threshold2: 10 vm.v_free_severe: 1058 vm.stats.sys.v_swtch: 593028 vm.stats.sys.v_trap: 83568 vm.stats.sys.v_syscall: 1168881 vm.stats.sys.v_intr: 1537519 vm.stats.sys.v_soft: 137125 vm.stats.vm.v_vm_faults: 84904 vm.stats.vm.v_cow_faults: 6328 vm.stats.vm.v_cow_optim: 0 vm.stats.vm.v_zfod: 12331 vm.stats.vm.v_ozfod: 12052 vm.stats.vm.v_swapin: 0 vm.stats.vm.v_swapout: 0 vm.stats.vm.v_swappgsin: 0 vm.stats.vm.v_swappgsout: 0 vm.stats.vm.v_vnodein: 273 vm.stats.vm.v_vnodeout: 0 vm.stats.vm.v_vnodepgsin: 1986 vm.stats.vm.v_vnodepgsout: 0 vm.stats.vm.v_intrans: 0 vm.stats.vm.v_reactivated: 119 vm.stats.vm.v_pdwakeups: 0 vm.stats.vm.v_pdpages: 0 vm.stats.vm.v_dfree: 0 vm.stats.vm.v_pfree: 15351 vm.stats.vm.v_tfree: 20808 vm.stats.vm.v_page_size: 4096 vm.stats.vm.v_page_count: 257185 vm.stats.vm.v_free_reserved: 416 vm.stats.vm.v_free_target: 7216 vm.stats.vm.v_free_min: 1700 vm.stats.vm.v_free_count: 135965 vm.stats.vm.v_wire_count: 29790 vm.stats.vm.v_active_count: 1179 vm.stats.vm.v_inactive_target: 10824 vm.stats.vm.v_inactive_count: 90238 vm.stats.vm.v_cache_count: 13 vm.stats.vm.v_cache_min: 7216 vm.stats.vm.v_cache_max: 14432 vm.stats.vm.v_pageout_free_min: 34 vm.stats.vm.v_interrupt_free_min: 2 vm.stats.vm.v_forks: 320 vm.stats.vm.v_vforks: 23 vm.stats.vm.v_rforks: 0 vm.stats.vm.v_kthreads: 4 vm.stats.vm.v_forkpages: 20423 vm.stats.vm.v_vforkpages: 1035 vm.stats.vm.v_rforkpages: 0 vm.stats.vm.v_kthreadpages: 0 vm.stats.misc.zero_page_count: 135553 vm.stats.misc.cnt_prezero: 203048 vm.max_proc_mmap: 43690 vm.max_launder: 32 vm.pageout_stats_max: 7216 vm.pageout_full_stats_interval: 20 vm.pageout_stats_interval: 5 vm.pageout_stats_free_max: 5 vm.swap_idle_enabled: 0 vm.defer_swapspace_pageouts: 0 vm.disable_swapspace_pageouts: 0 vm.zone: ITEM SIZE LIMIT USED FREE REQUESTS PIPE: 160, 0, 0, 102, 158 SWAPMETA: 160, 257185, 0, 0, 0 ripcb: 192, 65536, 0, 21, 1 syncache: 160, 15359, 16, 444, 94100 tcpcb: 544, 65536, 1828, 6597, 92979 udpcb: 192, 65536, 14, 28, 71 unpcb: 64, 0, 2, 126, 28 socket: 192, 65536, 3503, 4966, 93080 KNOTE: 64, 0, 0, 128, 13 NFSNODE: 352, 0, 39781, 17, 41857 NFSMOUNT: 544, 0, 1, 6, 1 VNODE: 192, 0, 40016, 52, 40016 NAMEI: 1024, 0, 0, 16, 187371 VMSPACE: 192, 0, 34, 30, 343 PROC: 416, 0, 38, 60, 347 DP fakepg: 64, 0, 0, 0, 0 PV ENTRY: 28, 1347438, 19011, 243108, 106251 MAP ENTRY: 48, 0, 391, 162, 15581 KMAP ENTRY: 48, 64424, 116, 97, 53824 MAP: 108, 0, 7, 3, 7 VM OBJECT: 96, 0, 33375, 103, 37919 vm.zone_kmem_pages: 1538 vm.zone_kmem_kvaspace: 150491136 vm.zone_kern_pages: 6330 vm.kvm_size: 1069547520 vm.kvm_free: 213909504 vfs.nfs.realign_test: 216199 vfs.nfs.realign_count: 0 vfs.nfs.bufpackets: 4 vfs.nfs.defect: 0 vfs.nfs.diskless_valid: 0 vfs.nfs.diskless_rootpath: vfs.nfs.diskless_swappath: vfs.nfs.access_cache_timeout: 600 vfs.nfs.nfsv3_commit_on_close: 0 vfs.numdirtybuffers: 3 vfs.lodirtybuffers: 910 vfs.hidirtybuffers: 1820 vfs.numfreebuffers: 7200 vfs.lofreebuffers: 405 vfs.hifreebuffers: 810 vfs.runningbufspace: 0 vfs.lorunningspace: 524288 vfs.hirunningspace: 1048576 vfs.maxbufspace: 118013952 vfs.hibufspace: 117358592 vfs.lobufspace: 117293056 vfs.bufspace: 117293056 vfs.maxmallocbufspace: 5867929 vfs.bufmallocspace: 129024 vfs.getnewbufcalls: 58745 vfs.getnewbufrestarts: 0 vfs.vmiodirenable: 1 vfs.bufdefragcnt: 0 vfs.buffreekvacnt: 0 vfs.bufreusecnt: 7159 vfs.cache.numneg: 118 vfs.cache.numcache: 40114 vfs.cache.numcalls: 760858 vfs.cache.dothits: 112 vfs.cache.dotdothits: 13703 vfs.cache.numchecks: 788978 vfs.cache.nummiss: 45111 vfs.cache.nummisszap: 8 vfs.cache.numposzaps: 12 vfs.cache.numposhits: 701263 vfs.cache.numnegzaps: 2 vfs.cache.numneghits: 647 vfs.cache.numcwdcalls: 32 vfs.cache.numcwdfail1: 0 vfs.cache.numcwdfail2: 0 vfs.cache.numcwdfail3: 0 vfs.cache.numcwdfail4: 0 vfs.cache.numcwdfound: 32 vfs.cache.numfullpathcalls: 0 vfs.cache.numfullpathfail1: 0 vfs.cache.numfullpathfail2: 0 vfs.cache.numfullpathfail3: 0 vfs.cache.numfullpathfail4: 0 vfs.cache.numfullpathfound: 0 vfs.write_behind: 1 vfs.reassignbufcalls: 493 vfs.reassignbufloops: 0 vfs.reassignbufsortgood: 174 vfs.reassignbufsortbad: 75 vfs.reassignbufmethod: 1 vfs.nameileafonly: 0 vfs.timestamp_precision: 0 vfs.usermount: 0 vfs.ffs.doreallocblks: 1 vfs.ffs.doasyncfree: 1 net.local.stream.sendspace: 8192 net.local.stream.recvspace: 8192 net.local.dgram.maxdgram: 2048 net.local.dgram.recvspace: 4096 net.local.inflight: 0 net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.first: 1024 net.inet.ip.portrange.last: 8000 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.hilast: 65535 net.inet.ip.forwarding: 0 net.inet.ip.redirect: 1 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 2400 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 100 net.inet.ip.intr_queue_drops: 0 net.inet.ip.accept_sourceroute: 0 net.inet.ip.fastforwarding: 0 net.inet.ip.keepfaith: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.maxfragpackets: 16384 net.inet.ip.check_interface: 0 net.inet.icmp.maskrepl: 0 net.inet.icmp.icmplim: 4000 net.inet.icmp.drop_redirect: 0 net.inet.icmp.log_redirect: 0 net.inet.icmp.bmcastecho: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.rfc1644: 1 net.inet.tcp.mssdflt: 1460 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 131072 net.inet.tcp.recvspace: 4096 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.log_in_vain: 0 net.inet.tcp.blackhole: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.slowstart_flightsize: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.newreno: 1 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.pcbcount: 1827 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.strict_rfc1948: 0 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.syncookies: 1 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15359 net.inet.tcp.syncache.count: 16 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.msl: 20000 net.inet.tcp.always_keepalive: 1 net.inet.udp.checksum: 1 net.inet.udp.maxdgram: 9216 net.inet.udp.recvspace: 41600 net.inet.udp.log_in_vain: 0 net.inet.udp.blackhole: 0 net.inet.accf.unloadable: 0 net.inet.accf.http.parsehttpversion: 0 net.inet.raw.maxdgram: 8192 net.inet.raw.recvspace: 8192 net.link.generic.system.ifcount: 3 net.link.ether.inet.prune_intvl: 300 net.link.ether.inet.max_age: 1200 net.link.ether.inet.host_down_time: 20 net.link.ether.inet.maxtries: 5 net.link.ether.inet.useloopback: 1 net.link.ether.inet.proxyall: 0 net.link.ether.inet.log_arp_wrong_iface: 1 debug.elf_trace: 0 debug.boothowto: -2147479552 debug.free_devt: 0 debug.fdexpand: 4 debug.sizeof.vnode: 168 debug.sizeof.proc: 408 debug.sizeof.specinfo: 68 debug.sizeof.disklabel: 276 debug.sizeof.diskslices: 1820 debug.sizeof.disk: 304 debug.ttydebug: 0 debug.nchash: 131071 debug.ncnegfactor: 16 debug.numneg: 118 debug.numcache: 40114 debug.vfscache: 1 debug.vnsize: 168 debug.ncsize: 36 debug.disablecwd: 0 debug.disablefullpath: 0 debug.numvnodes: 40016 debug.wantfreevnodes: 25 debug.freevnodes: 26 debug.rush_requests: 0 debug.max_softdeps: 547296 debug.tickdelay: 2 debug.worklist_push: 0 debug.blk_limit_push: 0 debug.ino_limit_push: 0 debug.blk_limit_hit: 0 debug.ino_limit_hit: 0 debug.sync_limit_hit: 0 debug.indir_blk_ptrs: 0 debug.inode_bitmap: 0 debug.direct_blk_ptrs: 0 debug.dir_entry: 0 debug.dircheck: 0 hw.machine: i386 hw.model: Pentium III/Pentium III Xeon/Celeron hw.ncpu: 1 hw.byteorder: 1234 hw.physmem: 1071288320 hw.usermem: 949264384 hw.pagesize: 4096 hw.floatingpoint: 1 hw.machine_arch: i386 hw.instruction_sse: 0 hw.availpages: 261379 hw.fxp0.int_delay: 1000 hw.fxp0.bundle_max: 6 hw.fxp1.int_delay: 1000 hw.fxp1.bundle_max: 6 machdep.consdev: { major = 12, minor = 255 } machdep.adjkerntz: 0 machdep.disable_rtc_set: 0 machdep.wall_cmos_clock: 0 machdep.do_dump: 1 machdep.enable_panic_key: 0 machdep.ispc98: 0 machdep.msgbuf: machdep.msgbuf_clear: 0 machdep.panic_on_nmi: 1 machdep.i8254_freq: 1193182 machdep.tsc_freq: 866427923 user.cs_path: /usr/bin:/bin:/usr/sbin:/sbin: user.bc_base_max: 99 user.bc_dim_max: 2048 user.bc_scale_max: 99 user.bc_string_max: 1000 user.coll_weights_max: 0 user.expr_nest_max: 32 user.line_max: 2048 user.re_dup_max: 255 user.posix2_version: 199212 user.posix2_c_bind: 0 user.posix2_c_dev: 0 user.posix2_char_term: 0 user.posix2_fort_dev: 0 user.posix2_fort_run: 0 user.posix2_localedef: 0 user.posix2_sw_dev: 0 user.posix2_upe: 0 user.stream_max: 20 user.tzname_max: 255 p1003_1b.asynchronous_io: 0 p1003_1b.mapped_files: 0 p1003_1b.memlock: 0 p1003_1b.memlock_range: 0 p1003_1b.memory_protection: 0 p1003_1b.message_passing: 0 p1003_1b.prioritized_io: 0 p1003_1b.priority_scheduling: 1 p1003_1b.realtime_signals: 0 p1003_1b.semaphores: 0 p1003_1b.fsync: 0 p1003_1b.shared_memory_objects: 0 p1003_1b.synchronized_io: 0 p1003_1b.timers: 0 p1003_1b.aio_listio_max: 0 p1003_1b.aio_max: 0 p1003_1b.aio_prio_delta_max: 0 p1003_1b.delaytimer_max: 0 p1003_1b.mq_open_max: 0 p1003_1b.pagesize: 4096 p1003_1b.rtsig_max: 0 p1003_1b.sem_nsems_max: 0 p1003_1b.sem_value_max: 0 p1003_1b.sigqueue_max: 0 p1003_1b.timer_max: 0 jail.set_hostname_allowed: 1 jail.socket_unixiproute_only: 1 jail.sysvipc_allowed: 0 If anyone has any ideas, please suggest. --Isprime Support To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message