Date: Tue, 25 Dec 2001 02:53:53 -0500 From: "ISPrime Support" <support@isprime.com> To: <freebsd-stable@freebsd.org> Subject: 4.5-PRERELEASE Major Stalling Message-ID: <000501c18d19$4c8dba20$4300a8c0@winter>
next in thread | raw e-mail | index | archive | help
I work at an ISP which hosts zfreehost.com (a pretty large freehost, pushing
about 300 megabit).
The zfreehost network has a few FBSD servers (all 4.4-STABLE except one
4.5-PRERELEASE) running a custom in-house webserver, using a FBSD NFS
Storage backend.
Anyway, I upgraded one of the servers to 4.5-PRERELEASE just to test it out
for us, and I noticed there are MAJOR stalls in 4.5. After about 5 minutes
of running, all of the webserver processes (8 processes) are sitting in
state sbwait, doing nothing. in a truss, it like it stalls on the read()
immediately after accept() from the client socket (http). We run httpready
accept filtering, so when we do a read() it should never ever stall since it
wouldn't have accept()'d if there wasnt data to read(), or if it was ready
to close(), anyway -- I consider this to be a pretty major regression. At
the way this is running, we wont be able to run 4.5 on any of the production
servers, so I hope this can get fixed.
Here is what I see from top:
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
266 nobody 2 0 992K 664K sbwait 0:01 0.63% 0.63% pimpd
267 nobody 2 0 992K 664K sbwait 0:01 0.10% 0.10% pimpd
264 nobody 2 0 992K 664K sbwait 0:01 0.05% 0.05% pimpd
86 root 10 0 208K 80K nfsidl 0:00 0.05% 0.05% nfsiod
123 root 28 0 2348K 1944K RUN 0:01 0.00% 0.00% sshd
85 root 10 0 208K 80K nfsidl 0:01 0.00% 0.00% nfsiod
269 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
262 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
268 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
263 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
265 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
This is what I see from ps auwx:
nobody 262 0.0 0.1 992 664 ?? Is 2:47AM 0:00.37 pimpd
nobody 263 0.0 0.1 992 664 ?? I 2:47AM 0:00.25 pimpd
nobody 264 0.0 0.1 992 664 ?? I 2:47AM 0:01.19 pimpd
nobody 265 0.0 0.1 992 664 ?? I 2:47AM 0:00.13 pimpd
nobody 266 0.0 0.1 992 664 ?? I 2:47AM 0:01.74 pimpd
nobody 267 0.0 0.1 992 664 ?? I 2:47AM 0:01.95 pimpd
nobody 268 0.0 0.1 992 664 ?? I 2:47AM 0:00.30 pimpd
nobody 269 0.0 0.1 992 664 ?? I 2:47AM 0:01.96 pimpd
I did not enable fxp bundling.
Here is sysctl -a:
kern.ostype: FreeBSD
kern.osrelease: 4.5-PRERELEASE
kern.osrevision: 199506
kern.version: FreeBSD 4.5-PRERELEASE #0: Tue Dec 25 02:13:41 EST 2001
root@zf3:/usr/src/sys/compile/ZFREE
kern.maxvnodes: 68412
kern.maxproc: 4116
kern.maxfiles: 8232
kern.argmax: 65536
kern.securelevel: -1
kern.hostname: zfreeVA3.isprime.com
kern.hostid: 0
kern.clockrate: { hz = 100, tick = 10000, tickadj = 5, profhz = 1024, stathz
= 128 }
kern.posix1version: 199309
kern.ngroups: 16
kern.job_control: 1
kern.saved_ids: 0
kern.boottime: { sec = 1009265636, usec = 169977 } Tue Dec 25 02:33:56 2001
kern.domainname:
kern.osreldate: 450000
kern.bootfile: /kernel
kern.maxfilesperproc: 7408
kern.maxprocperuid: 3704
kern.dumpdev:
kern.ipc.maxsockbuf: 2097152
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 10240
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 40
kern.ipc.max_hdr: 56
kern.ipc.max_datalen: 156
kern.ipc.nmbclusters: 65536
kern.ipc.semmap: 30
kern.ipc.semmni: 10
kern.ipc.semmns: 60
kern.ipc.semmnu: 30
kern.ipc.semmsl: 60
kern.ipc.semopm: 100
kern.ipc.semume: 10
kern.ipc.semusz: 92
kern.ipc.semvmx: 32767
kern.ipc.semaem: 16384
kern.ipc.shmmax: 409600000
kern.ipc.shmmin: 1
kern.ipc.shmmni: 192
kern.ipc.shmseg: 128
kern.ipc.shmall: 8192
kern.ipc.shm_use_phys: 0
kern.ipc.mbuf_wait: 32
kern.ipc.mbtypes: 3589 3759 76 0 0 0 0 0 0 0 0 0 0 0 0 0
kern.ipc.nmbufs: 262144
kern.ipc.maxsockets: 65536
kern.dummy: 0
kern.ps_strings: 3217031152
kern.usrstack: 3217031168
kern.logsigexit: 1
kern.cam.da.retry_count: 4
kern.cam.da.default_timeout: 60
kern.fallback_elf_brand: -1
kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall
kern.module_path: /;/boot/;/modules/
kern.acct_suspend: 2
kern.acct_resume: 4
kern.acct_chkfreq: 15
kern.cp_time: 511 0 2075 2732 147211
kern.timecounter.method: 0
kern.timecounter.hardware: TSC
kern.openfiles: 52
kern.kq_calloutmax: 4096
kern.ps_arg_cache_limit: 256
kern.ps_argsopen: 1
kern.fast_vfork: 1
kern.randompid: 0
kern.maxusers: 256
kern.ps_showallprocs: 1
kern.shutdown.poweroff_delay: 5000
kern.shutdown.kproc_shutdown_wait: 60
kern.sugid_coredump: 1
kern.coredump: 1
kern.corefile: /tmp/%P.%N.core
kern.quantum: 100000
kern.ccpu: 1948
kern.fscale: 2048
kern.devstat.numdevs: 3
kern.devstat.generation: 3
kern.devstat.version: 4
kern.disks: da0
kern.log_wakeups_per_second: 5
kern.msgbuf:
kern.msgbuf_clear: 0
kern.nselcoll: 0
kern.consmute: 0
kern.filedelay: 30
kern.dirdelay: 29
kern.metadelay: 28
kern.minvnodes: 17103
kern.chroot_allow_open_directories: 1
vm.loadavg: { 0.01 0.04 0.01 }
vm.v_free_min: 1700
vm.v_free_target: 7216
vm.v_free_reserved: 416
vm.v_inactive_target: 10824
vm.v_cache_min: 7216
vm.v_cache_max: 14432
vm.v_pageout_free_min: 34
vm.pageout_algorithm: 0
vm.swap_enabled: 1
vm.swap_async_max: 4
vm.swap_idle_threshold1: 2
vm.swap_idle_threshold2: 10
vm.v_free_severe: 1058
vm.stats.sys.v_swtch: 593028
vm.stats.sys.v_trap: 83568
vm.stats.sys.v_syscall: 1168881
vm.stats.sys.v_intr: 1537519
vm.stats.sys.v_soft: 137125
vm.stats.vm.v_vm_faults: 84904
vm.stats.vm.v_cow_faults: 6328
vm.stats.vm.v_cow_optim: 0
vm.stats.vm.v_zfod: 12331
vm.stats.vm.v_ozfod: 12052
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_vnodein: 273
vm.stats.vm.v_vnodeout: 0
vm.stats.vm.v_vnodepgsin: 1986
vm.stats.vm.v_vnodepgsout: 0
vm.stats.vm.v_intrans: 0
vm.stats.vm.v_reactivated: 119
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_pdpages: 0
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_pfree: 15351
vm.stats.vm.v_tfree: 20808
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_page_count: 257185
vm.stats.vm.v_free_reserved: 416
vm.stats.vm.v_free_target: 7216
vm.stats.vm.v_free_min: 1700
vm.stats.vm.v_free_count: 135965
vm.stats.vm.v_wire_count: 29790
vm.stats.vm.v_active_count: 1179
vm.stats.vm.v_inactive_target: 10824
vm.stats.vm.v_inactive_count: 90238
vm.stats.vm.v_cache_count: 13
vm.stats.vm.v_cache_min: 7216
vm.stats.vm.v_cache_max: 14432
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_forks: 320
vm.stats.vm.v_vforks: 23
vm.stats.vm.v_rforks: 0
vm.stats.vm.v_kthreads: 4
vm.stats.vm.v_forkpages: 20423
vm.stats.vm.v_vforkpages: 1035
vm.stats.vm.v_rforkpages: 0
vm.stats.vm.v_kthreadpages: 0
vm.stats.misc.zero_page_count: 135553
vm.stats.misc.cnt_prezero: 203048
vm.max_proc_mmap: 43690
vm.max_launder: 32
vm.pageout_stats_max: 7216
vm.pageout_full_stats_interval: 20
vm.pageout_stats_interval: 5
vm.pageout_stats_free_max: 5
vm.swap_idle_enabled: 0
vm.defer_swapspace_pageouts: 0
vm.disable_swapspace_pageouts: 0
vm.zone:
ITEM SIZE LIMIT USED FREE REQUESTS
PIPE: 160, 0, 0, 102, 158
SWAPMETA: 160, 257185, 0, 0, 0
ripcb: 192, 65536, 0, 21, 1
syncache: 160, 15359, 16, 444, 94100
tcpcb: 544, 65536, 1828, 6597, 92979
udpcb: 192, 65536, 14, 28, 71
unpcb: 64, 0, 2, 126, 28
socket: 192, 65536, 3503, 4966, 93080
KNOTE: 64, 0, 0, 128, 13
NFSNODE: 352, 0, 39781, 17, 41857
NFSMOUNT: 544, 0, 1, 6, 1
VNODE: 192, 0, 40016, 52, 40016
NAMEI: 1024, 0, 0, 16, 187371
VMSPACE: 192, 0, 34, 30, 343
PROC: 416, 0, 38, 60, 347
DP fakepg: 64, 0, 0, 0, 0
PV ENTRY: 28, 1347438, 19011, 243108, 106251
MAP ENTRY: 48, 0, 391, 162, 15581
KMAP ENTRY: 48, 64424, 116, 97, 53824
MAP: 108, 0, 7, 3, 7
VM OBJECT: 96, 0, 33375, 103, 37919
vm.zone_kmem_pages: 1538
vm.zone_kmem_kvaspace: 150491136
vm.zone_kern_pages: 6330
vm.kvm_size: 1069547520
vm.kvm_free: 213909504
vfs.nfs.realign_test: 216199
vfs.nfs.realign_count: 0
vfs.nfs.bufpackets: 4
vfs.nfs.defect: 0
vfs.nfs.diskless_valid: 0
vfs.nfs.diskless_rootpath:
vfs.nfs.diskless_swappath:
vfs.nfs.access_cache_timeout: 600
vfs.nfs.nfsv3_commit_on_close: 0
vfs.numdirtybuffers: 3
vfs.lodirtybuffers: 910
vfs.hidirtybuffers: 1820
vfs.numfreebuffers: 7200
vfs.lofreebuffers: 405
vfs.hifreebuffers: 810
vfs.runningbufspace: 0
vfs.lorunningspace: 524288
vfs.hirunningspace: 1048576
vfs.maxbufspace: 118013952
vfs.hibufspace: 117358592
vfs.lobufspace: 117293056
vfs.bufspace: 117293056
vfs.maxmallocbufspace: 5867929
vfs.bufmallocspace: 129024
vfs.getnewbufcalls: 58745
vfs.getnewbufrestarts: 0
vfs.vmiodirenable: 1
vfs.bufdefragcnt: 0
vfs.buffreekvacnt: 0
vfs.bufreusecnt: 7159
vfs.cache.numneg: 118
vfs.cache.numcache: 40114
vfs.cache.numcalls: 760858
vfs.cache.dothits: 112
vfs.cache.dotdothits: 13703
vfs.cache.numchecks: 788978
vfs.cache.nummiss: 45111
vfs.cache.nummisszap: 8
vfs.cache.numposzaps: 12
vfs.cache.numposhits: 701263
vfs.cache.numnegzaps: 2
vfs.cache.numneghits: 647
vfs.cache.numcwdcalls: 32
vfs.cache.numcwdfail1: 0
vfs.cache.numcwdfail2: 0
vfs.cache.numcwdfail3: 0
vfs.cache.numcwdfail4: 0
vfs.cache.numcwdfound: 32
vfs.cache.numfullpathcalls: 0
vfs.cache.numfullpathfail1: 0
vfs.cache.numfullpathfail2: 0
vfs.cache.numfullpathfail3: 0
vfs.cache.numfullpathfail4: 0
vfs.cache.numfullpathfound: 0
vfs.write_behind: 1
vfs.reassignbufcalls: 493
vfs.reassignbufloops: 0
vfs.reassignbufsortgood: 174
vfs.reassignbufsortbad: 75
vfs.reassignbufmethod: 1
vfs.nameileafonly: 0
vfs.timestamp_precision: 0
vfs.usermount: 0
vfs.ffs.doreallocblks: 1
vfs.ffs.doasyncfree: 1
net.local.stream.sendspace: 8192
net.local.stream.recvspace: 8192
net.local.dgram.maxdgram: 2048
net.local.dgram.recvspace: 4096
net.local.inflight: 0
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 1024
net.inet.ip.portrange.last: 8000
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535
net.inet.ip.forwarding: 0
net.inet.ip.redirect: 1
net.inet.ip.ttl: 64
net.inet.ip.rtexpire: 2400
net.inet.ip.rtminexpire: 10
net.inet.ip.rtmaxcache: 128
net.inet.ip.sourceroute: 0
net.inet.ip.intr_queue_maxlen: 100
net.inet.ip.intr_queue_drops: 0
net.inet.ip.accept_sourceroute: 0
net.inet.ip.fastforwarding: 0
net.inet.ip.keepfaith: 0
net.inet.ip.subnets_are_local: 0
net.inet.ip.maxfragpackets: 16384
net.inet.ip.check_interface: 0
net.inet.icmp.maskrepl: 0
net.inet.icmp.icmplim: 4000
net.inet.icmp.drop_redirect: 0
net.inet.icmp.log_redirect: 0
net.inet.icmp.bmcastecho: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.rfc1644: 1
net.inet.tcp.mssdflt: 1460
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 4096
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.log_in_vain: 0
net.inet.tcp.blackhole: 0
net.inet.tcp.delayed_ack: 1
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.newreno: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.pcbcount: 1827
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.strict_rfc1948: 0
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.syncookies: 1
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncache.cachelimit: 15359
net.inet.tcp.syncache.count: 16
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.msl: 20000
net.inet.tcp.always_keepalive: 1
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 41600
net.inet.udp.log_in_vain: 0
net.inet.udp.blackhole: 0
net.inet.accf.unloadable: 0
net.inet.accf.http.parsehttpversion: 0
net.inet.raw.maxdgram: 8192
net.inet.raw.recvspace: 8192
net.link.generic.system.ifcount: 3
net.link.ether.inet.prune_intvl: 300
net.link.ether.inet.max_age: 1200
net.link.ether.inet.host_down_time: 20
net.link.ether.inet.maxtries: 5
net.link.ether.inet.useloopback: 1
net.link.ether.inet.proxyall: 0
net.link.ether.inet.log_arp_wrong_iface: 1
debug.elf_trace: 0
debug.boothowto: -2147479552
debug.free_devt: 0
debug.fdexpand: 4
debug.sizeof.vnode: 168
debug.sizeof.proc: 408
debug.sizeof.specinfo: 68
debug.sizeof.disklabel: 276
debug.sizeof.diskslices: 1820
debug.sizeof.disk: 304
debug.ttydebug: 0
debug.nchash: 131071
debug.ncnegfactor: 16
debug.numneg: 118
debug.numcache: 40114
debug.vfscache: 1
debug.vnsize: 168
debug.ncsize: 36
debug.disablecwd: 0
debug.disablefullpath: 0
debug.numvnodes: 40016
debug.wantfreevnodes: 25
debug.freevnodes: 26
debug.rush_requests: 0
debug.max_softdeps: 547296
debug.tickdelay: 2
debug.worklist_push: 0
debug.blk_limit_push: 0
debug.ino_limit_push: 0
debug.blk_limit_hit: 0
debug.ino_limit_hit: 0
debug.sync_limit_hit: 0
debug.indir_blk_ptrs: 0
debug.inode_bitmap: 0
debug.direct_blk_ptrs: 0
debug.dir_entry: 0
debug.dircheck: 0
hw.machine: i386
hw.model: Pentium III/Pentium III Xeon/Celeron
hw.ncpu: 1
hw.byteorder: 1234
hw.physmem: 1071288320
hw.usermem: 949264384
hw.pagesize: 4096
hw.floatingpoint: 1
hw.machine_arch: i386
hw.instruction_sse: 0
hw.availpages: 261379
hw.fxp0.int_delay: 1000
hw.fxp0.bundle_max: 6
hw.fxp1.int_delay: 1000
hw.fxp1.bundle_max: 6
machdep.consdev: { major = 12, minor = 255 }
machdep.adjkerntz: 0
machdep.disable_rtc_set: 0
machdep.wall_cmos_clock: 0
machdep.do_dump: 1
machdep.enable_panic_key: 0
machdep.ispc98: 0
machdep.msgbuf:
machdep.msgbuf_clear: 0
machdep.panic_on_nmi: 1
machdep.i8254_freq: 1193182
machdep.tsc_freq: 866427923
user.cs_path: /usr/bin:/bin:/usr/sbin:/sbin:
user.bc_base_max: 99
user.bc_dim_max: 2048
user.bc_scale_max: 99
user.bc_string_max: 1000
user.coll_weights_max: 0
user.expr_nest_max: 32
user.line_max: 2048
user.re_dup_max: 255
user.posix2_version: 199212
user.posix2_c_bind: 0
user.posix2_c_dev: 0
user.posix2_char_term: 0
user.posix2_fort_dev: 0
user.posix2_fort_run: 0
user.posix2_localedef: 0
user.posix2_sw_dev: 0
user.posix2_upe: 0
user.stream_max: 20
user.tzname_max: 255
p1003_1b.asynchronous_io: 0
p1003_1b.mapped_files: 0
p1003_1b.memlock: 0
p1003_1b.memlock_range: 0
p1003_1b.memory_protection: 0
p1003_1b.message_passing: 0
p1003_1b.prioritized_io: 0
p1003_1b.priority_scheduling: 1
p1003_1b.realtime_signals: 0
p1003_1b.semaphores: 0
p1003_1b.fsync: 0
p1003_1b.shared_memory_objects: 0
p1003_1b.synchronized_io: 0
p1003_1b.timers: 0
p1003_1b.aio_listio_max: 0
p1003_1b.aio_max: 0
p1003_1b.aio_prio_delta_max: 0
p1003_1b.delaytimer_max: 0
p1003_1b.mq_open_max: 0
p1003_1b.pagesize: 4096
p1003_1b.rtsig_max: 0
p1003_1b.sem_nsems_max: 0
p1003_1b.sem_value_max: 0
p1003_1b.sigqueue_max: 0
p1003_1b.timer_max: 0
jail.set_hostname_allowed: 1
jail.socket_unixiproute_only: 1
jail.sysvipc_allowed: 0
If anyone has any ideas, please suggest.
--Isprime Support
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?000501c18d19$4c8dba20$4300a8c0>