Date: Wed, 4 Jul 2001 23:47:12 +0300 From: "Vlad Marchenko" <vlad@infonet.com.ua> To: <freebsd-net@FreeBSD.ORG> Subject: Operation now in progress Message-ID: <023401c104ca$829d0d00$6a7a01d4@kuzya>
next in thread | raw e-mail | index | archive | help
Hello, We've got into problems that I have no idea how to solve. Our webcluster consist of one frontend server and several backend servers. FreeBSD 4.3-STABLE and apache 1.3.20 are used everywhere. When a rate of requests from frontend to backend server becomes too high (over 100 rps) backend server starting to behavior weird. This could be illustrated by generating heavy traffic from frontend to backend by ab test performance utility included in apache distribution: ------------ frontend# /usr/local/apache/bin/ab -c 40 -n 10000 "http://backend/page.html" This is ApacheBench, Version 1.3c <$Revision: 1.44 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2000 The Apache Group, http://www.apache.org/ Benchmarking backend (be patient)... Server timed out : Operation now in progress -------------- this makes 40 concurrent connections with total 10000 requests and as you see, it fails. When I see `top` on backend when ab is started - first several seconds it uses all CPU (0% idle), then it jumps to 100% idle and then ab reports "Server timed out: operation now in progress". There is _nothing_ in apache error.log. Nothing in /var/log/messages. `netstat -m` shows it has enough mbufs. it has enough memory. What's interesting - if I start simultaneously 4 ab "-c 10" each from 4 different hosts to test same backend (i.e. we have same 40 concurrent connections from backend point of view) = it works great. gives 50-60 rps per each started ab = 200+ rps total. I've tried "boa" webserver in place of apache - same problem. I've also tried to run backend on linux (RH 7.1) with same configured apache - runs fine, no "Operation now in progress" problem. This makes me think it's some OS (system) speciffic issue, not apache configuration problem. I would be very thankful to hear any ideas how to fix that. Thanks in advance. Below is config of my kernel and then /boot/loader.conf.local machine i386 cpu I686_CPU ident BACKEND maxusers 256 options SMP # Symmetric MultiProcessor Kernel options APIC_IO # Symmetric (APIC) I/O options INET #InterNETworking options FFS #Berkeley Fast Filesystem options FFS_ROOT #FFS usable as root device [keep this!] options SOFTUPDATES #Enable FFS soft updates support options CD9660 #ISO 9660 Filesystem options CD9660_ROOT #CD-ROM usable as root, CD9660 required options PROCFS #Process filesystem options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!] options SCSI_DELAY=5000 #Delay (in ms) before probing SCSI options UCONSOLE #Allow users to grab the console options USERCONFIG #boot -c editor options VISUAL_USERCONFIG #visual boot -c editor options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options P1003_1B #Posix P1003_1B real-time extensions options _KPOSIX_PRIORITY_SCHEDULING options KBD_INSTALL_CDEV # install a CDEV entry in /dev options IPFIREWALL options IPFIREWALL_VERBOSE device isa device pci device fdc0 at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 device scbus # SCSI bus (required) device da # Direct Access (disks) device pass # Passthrough device (direct SCSI access) device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 flags 0x1 device psm0 at atkbdc? irq 12 device vga0 at isa? device sc0 at isa? flags 0x100 device npx0 at nexus? port IO_NPX irq 13 device apm0 at nexus? disable flags 0x20 # Advanced Power Management device sio0 at isa? port IO_COM1 flags 0x10 irq 4 device sio1 at isa? port IO_COM2 irq 3 device miibus # MII bus support device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') pseudo-device splash pseudo-device snp 3 pseudo-device loop # Network loopback pseudo-device ether # Ethernet support pseudo-device tun # Packet tunnel. pseudo-device pty # Pseudo-ttys (telnet etc) pseudo-device md # Memory "disks" pseudo-device bpf #Berkeley packet filter /boot/loader.conf.local verbose_loading="YES" kern.vm.kmem.size="268435456" # size of kernel memory (bytes) kern.vm.pmap.shpgperproc="1024" # shared pages per process kern.nbuf="16384" # number of pages for buffers kern.maxfiles="16384" # max number of open files kern.maxproc="4096" # max number of processes kern.ncallout="32768" # callout limit kern.ipc.maxsockets="32768" # max number of sockets kern.ipc.nmbclusters="32768" # number of mbuf clusters kern.ipc.nmbufs="65536" # number of mbufs kern.ipc.nsfbufs="4096" # number of sendfile bufs net.inet.tcp.tcbhashsize="16384" # size of tcp pcb hash tables -- Vlad Marchenko To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?023401c104ca$829d0d00$6a7a01d4>