Date: Mon, 12 Feb 2018 09:38:45 -0800 From: =?utf-8?Q?Ask_Bj=C3=B8rn_Hansen?= <ask@develooper.com> To: Eugene Grosbein <eugen@grosbein.net> Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD on 64MB memory Message-ID: <C2042F56-2EB7-47B9-92C5-52DA21CA3132@develooper.com> In-Reply-To: <5A8123CE.9050609@grosbein.net> References: <5FB97479-C49D-4C6E-8416-015ECA656C14@develooper.com> <5A8123CE.9050609@grosbein.net>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Feb 11, 2018, at 9:19 PM, Eugene Grosbein <eugen@grosbein.net> = wrote: >=20 > 12.02.2018 11:56, Ask Bj=C3=B8rn Hansen =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> Hi, >>=20 >> I have an old Soekris system with 64MB memory that I upgraded from = 10.3 to 11.1 recently. Since then it=E2=80=99s started hanging every few = days. >>=20 >> Today I happened to have a =E2=80=9Ctop=E2=80=9D instance running on = the serial console. The system is minimally responsive to the network = (ICMP and CARP are working, but no services). >>=20 >>> =46rom the top output it=E2=80=99s not clear what resource it=E2=80=99= s out of. >=20 > I suspect it is out of many types of kernel memory including mbuf = clusters, > hence no working TCP/UDP but ICMP works. That makes sense. The console locks up, too (as soon as I ctrl-c=E2=80=99e= d the running top process the console was frozen). >> There=E2=80=99s no swap configured, but that what it looks like = it=E2=80=99s trying to do?=20 >>=20 >> The =E2=80=98pf purge=E2=80=99 process is suspicious. There are no pf = rules configured on the system (it should be all disabled). >>=20 >> Any suggestions? (Other than =E2=80=9Cseriously =E2=80=A6 64MB = memory?!=E2=80=9D). >=20 > Please show output of commands: >=20 > grep memory /var/run/dmesg.boot real memory =3D 67108864 (64 MB) avail memory =3D 42098688 (40 MB) The 24MB are for the kernel? I wonder my 11.1 kernel is less = discriminating with what I compiled in... > top -ores -d1 Shortly after boot: last pid: 1008; load averages: 0.57, 0.62, 0.53 up 0+00:19:31 = 06:24:50 8 processes: 1 running, 7 sleeping CPU: % user, % nice, % system, % interrupt, % idle Mem: 9084K Active, 3644K Inact, 29M Wired, 4862K Buf, 492K Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 911 root 1 22 0 8816K 8844K select 0:39 4.20% ntpd 959 root 1 52 0 10756K 5196K select 0:00 0.00% sshd 709 root 1 20 0 7300K 3224K select 0:00 0.00% devd 1002 root 1 20 0 6712K 2876K pause 0:01 0.49% csh 1008 root 1 49 0 7272K 2356K RUN 0:00 0.00% top 991 root 1 26 0 6400K 2196K wait 0:01 0.00% login 985 root 1 20 0 5952K 1848K nanslp 0:00 0.00% cron 788 root 1 20 0 5920K 1776K select 0:01 0.00% syslogd > sysctl kern.ipc.nmbclusters kern.ipc.nmbclusters: 898 kern.ipc.nmbufs: 5745 kern.ipc.maxmbufmem: 7350272 > It would be also very useful to obtain output of "vmstat -z" in a = moment of breakage. I made it run on the console every 30 seconds, this is the last one I = got. For comparison there=E2=80=99s a =E2=80=9Cshortly after boot=E2=80=9D= version below the =3D=3D=3D=E2=80=99s. Ask Shortly before it stopped responding: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 256, 0, 128, 7, 128, 0, 0 UMA Zones: 408, 0, 129, 6, 129, 0, 0 UMA Slabs: 56, 0, 1655, 73, 20845, 0, 0 UMA Hash: 128, 0, 11, 20, 14, 0, 0 4 Bucket: 16, 0, 0, 504, 364, 0, 0 6 Bucket: 24, 0, 0, 336, 118, 0, 0 8 Bucket: 32, 0, 0, 378, 54150, 0, 0 12 Bucket: 48, 0, 0, 0, 0, 0, 0 16 Bucket: 64, 0, 2, 313, 1840, 8, 0 32 Bucket: 128, 0, 3, 152, 1516, 0, 0 64 Bucket: 256, 0, 1, 74, 221, 0, 0 128 Bucket: 512, 0, 4, 28, 1324, 0, 0 256 Bucket: 1024, 0, 3, 13, 3917, 4, 0 vmem btag: 28, 0, 3169, 143, 3847, 56, 0 VM OBJECT: 148, 0, 1208, 115, 38997, 0, 0 RADIX NODE: 44, 10738, 1749, 344, 88089, 0, 0 MAP: 140, 0, 3, 81, 3, 0, 0 KMAP ENTRY: 72, 0, 5, 163, 5, 0, 0 MAP ENTRY: 72, 0, 244, 148, 89326, 0, 0 VMSPACE: 232, 0, 10, 75, 2062, 0, 0 fakepg: 68, 0, 0, 0, 0, 0, 0 mt_zone: 2060, 0, 352, 0, 352, 0, 0 16: 16, 0, 1159, 353, 813531, 0, 0 32: 32, 0, 957, 303, 23866, 0, 0 64: 64, 0, 1385, 253, 29315, 0, 0 128: 128, 0, 734, 134, 14162, 0, 0 256: 256, 0, 378, 57, 5062, 0, 0 512: 512, 0, 57, 7, 4807, 0, 0 1024: 1024, 0, 31, 33, 539, 0, 0 2048: 2048, 0, 208, 2, 2908, 0, 0 4096: 4096, 0, 86, 0, 2165, 0, 0 8192: 8192, 0, 0, 0, 0, 0, 0 16384: 16384, 0, 0, 0, 0, 0, 0 32768: 32768, 0, 0, 0, 0, 0, 0 65536: 65536, 0, 0, 0, 0, 0, 0 SLEEPQUEUE: 44, 0, 85, 293, 85, 0, 0 64 pcpu: 8, 0, 2642, 174, 2714, 0, 0 ptr pcpu: 4, 0, 0, 0, 0, 0, 0 Files: 56, 0, 42, 318, 18945, 0, 0 filedesc0: 888, 0, 34, 6, 2085, 0, 0 rl_entry: 28, 0, 23, 265, 23, 0, 0 TURNSTILE: 72, 0, 85, 132, 85, 0, 0 umtx pi: 52, 0, 0, 0, 0, 0, 0 umtx_shm: 52, 0, 0, 0, 0, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 916, 0, 33, 15, 2084, 0, 0 THREAD: 904, 0, 70, 14, 71, 0, 0 cpuset: 40, 0, 49, 52, 49, 0, 0 audit_record: 1112, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 5745, 259, 0, 418909, 2, 0 mbuf: 256, 5745, 263, 273, 773249, 18, 0 mbuf_cluster: 2048, 898, 259, 131, 204839,1868, 0 mbuf_jumbo_page: 4096, 448, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 132, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 74, 0, 0, 0, 0, 0 g_bio: 264, 0, 0, 0, 13391, 0, 0 ttyinq: 152, 0, 15, 37, 60, 0, 0 ttyoutq: 256, 0, 8, 22, 32, 0, 0 nvme_request: 100, 0, 0, 0, 0, 0, 0 cryptop: 60, 0, 0, 0, 0, 0, 0 cryptodesc: 56, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 16, 0, 0, 0, 0, 0, 0 FPU_save_area: 512, 0, 0, 0, 0, 0, 0 VNODE: 284, 0, 1134, 28, 1170, 0, 0 VNODEPOLL: 64, 0, 0, 0, 0, 0, 0 BUF TRIE: 44, 0, 52, 676, 2310, 0, 0 NAMEI: 1024, 0, 0, 16, 55376, 0, 0 rentr: 12, 0, 0, 0, 0, 0, 0 S VFS Cache: 72, 0, 977, 143, 2862, 0, 0 STS VFS Cache: 92, 0, 0, 0, 0, 0, 0 L VFS Cache: 292, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 312, 0, 0, 0, 0, 0, 0 NCLNODE: 360, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 58, 0, 0 Mountpoints: 672, 0, 4, 14, 5, 0, 0 pipe: 416, 0, 0, 0, 1030, 0, 0 procdesc: 80, 0, 0, 0, 0, 0, 0 AIO: 128, 0, 0, 0, 0, 0, 0 AIOP: 16, 0, 0, 0, 0, 0, 0 AIOCB: 428, 0, 0, 0, 0, 0, 0 AIOL: 64, 0, 0, 0, 0, 0, 0 AIOLIO: 172, 0, 0, 0, 0, 0, 0 ksiginfo: 80, 0, 166, 134, 17061, 0, 0 itimer: 232, 0, 0, 0, 0, 0, 0 bridge_rtnode: 36, 0, 0, 0, 0, 0, 0 KNOTE: 72, 0, 0, 0, 0, 0, 0 socket: 524, 1393, 20, 15, 3048, 0, 0 unpcb: 172, 1403, 6, 17, 2053, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0 ipq: 32, 126, 0, 0, 0, 0, 0 udp_inpcb: 304, 1404, 11, 15, 925, 0, 0 udpcb: 20, 1414, 11, 191, 925, 0, 0 tcp_inpcb: 304, 1404, 2, 11, 61, 0, 0 tcpcb: 772, 1395, 2, 3, 61, 0, 0 tcptw: 60, 335, 0, 0, 0, 0, 0 syncache: 128, 15376, 0, 0, 54, 12, 0 hostcache: 76, 15370, 0, 0, 0, 0, 0 sackhole: 20, 0, 0, 0, 0, 0, 0 tcpreass: 20, 202, 0, 0, 0, 0, 0 sctp_ep: 1052, 1395, 0, 0, 0, 0, 0 sctp_asoc: 1656, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80136, 0, 0, 5, 0, 0 sctp_raddr: 540, 80003, 0, 0, 0, 0, 0 sctp_chunk: 104, 400026, 0, 0, 0, 0, 0 sctp_readq: 108, 400007, 0, 0, 0, 0, 0 sctp_stream_msg_out: 72, 400008, 0, 0, 0, 0, 0 sctp_asconf: 24, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400008, 0, 0, 0, 0, 0 udplite_inpcb: 304, 1404, 0, 0, 0, 0, 0 ripcb: 304, 1404, 0, 0, 0, 0, 0 rtentry: 112, 0, 42, 30, 50, 0, 0 pf mtags: 32, 0, 0, 0, 0, 0, 0 pf states: 208, 10013, 0, 0, 0, 0, 0 pf state keys: 64, 0, 0, 0, 0, 0, 0 pf source nodes: 116, 10030, 0, 0, 0, 0, 0 pf table entries: 92, 0, 0, 0, 0, 0, 0 pf table counters: 64, 0, 0, 0, 0, 0, 0 pf frags: 80, 0, 0, 0, 0, 0, 0 pf frag entries: 24, 5040, 0, 0, 0, 0, 0 pf state scrubs: 28, 0, 0, 0, 0, 0, 0 IPFW counters: 16, 0, 1, 63, 1, 0, 0 IPFW dynamic rule: 112, 16416, 0, 0, 0, 0, 0 divcb: 304, 1404, 0, 0, 0, 0, 0 selfd: 32, 0, 38, 340, 2106598, 0, 0 SWAPMETA: 276, 5390, 0, 0, 0, 0, 0 FFS inode: 112, 0, 1109, 151, 1144, 0, 0 FFS1 dinode: 128, 0, 632, 81, 662, 0, 0 FFS2 dinode: 256, 0, 477, 33, 482, 0, 0 md0: 512, 0, 5042, 14, 5050, 0, 0 md1: 512, 0, 4298, 6, 4299, 0, 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D ~25 minutes after boot: ntp1.us.grundclock.com# vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 256, 0, 128, 7, 128, 0, 0 UMA Zones: 408, 0, 129, 6, 129, 0, 0 UMA Slabs: 56, 0, 1658, 70, 2158, 0, 0 UMA Hash: 128, 0, 11, 20, 14, 0, 0 4 Bucket: 16, 0, 1, 503, 254, 0, 0 6 Bucket: 24, 0, 0, 336, 118, 0, 0 8 Bucket: 32, 0, 2, 376, 149, 0, 0 12 Bucket: 48, 0, 0, 0, 0, 0, 0 16 Bucket: 64, 0, 5, 310, 35, 8, 0 32 Bucket: 128, 0, 6, 149, 396, 0, 0 64 Bucket: 256, 0, 3, 72, 47, 0, 0 128 Bucket: 512, 0, 9, 23, 52, 0, 0 256 Bucket: 1024, 0, 15, 13, 110, 0, 0 vmem btag: 28, 0, 3101, 67, 3133, 22, 0 VM OBJECT: 148, 0, 1187, 109, 12302, 0, 0 RADIX NODE: 44, 10738, 1712, 381, 33112, 0, 0 MAP: 140, 0, 3, 81, 3, 0, 0 KMAP ENTRY: 72, 0, 5, 163, 5, 0, 0 MAP ENTRY: 72, 0, 227, 165, 27414, 0, 0 VMSPACE: 232, 0, 9, 76, 999, 0, 0 fakepg: 68, 0, 0, 0, 0, 0, 0 mt_zone: 2060, 0, 352, 0, 352, 0, 0 16: 16, 0, 1163, 349, 61329, 0, 0 32: 32, 0, 955, 305, 5553, 0, 0 64: 64, 0, 1366, 272, 20940, 0, 0 128: 128, 0, 726, 142, 13237, 0, 0 256: 256, 0, 374, 61, 1906, 0, 0 512: 512, 0, 57, 31, 1553, 0, 0 1024: 1024, 0, 31, 33, 526, 0, 0 2048: 2048, 0, 208, 6, 2703, 0, 0 4096: 4096, 0, 85, 2, 1101, 0, 0 8192: 8192, 0, 0, 0, 0, 0, 0 16384: 16384, 0, 0, 0, 0, 0, 0 32768: 32768, 0, 0, 0, 0, 0, 0 65536: 65536, 0, 0, 0, 0, 0, 0 SLEEPQUEUE: 44, 0, 77, 301, 77, 0, 0 64 pcpu: 8, 0, 2642, 174, 2714, 0, 0 ptr pcpu: 4, 0, 0, 0, 0, 0, 0 Files: 56, 0, 42, 318, 8074, 0, 0 filedesc0: 888, 0, 33, 15, 1022, 0, 0 rl_entry: 28, 0, 16, 272, 16, 0, 0 TURNSTILE: 72, 0, 77, 140, 77, 0, 0 umtx pi: 52, 0, 0, 0, 0, 0, 0 umtx_shm: 52, 0, 0, 0, 0, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 916, 0, 32, 12, 1021, 0, 0 THREAD: 904, 0, 62, 14, 63, 0, 0 cpuset: 40, 0, 49, 52, 49, 0, 0 audit_record: 1112, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 5745, 128, 253, 32698, 0, 0 mbuf: 256, 5745, 1, 263, 57270, 0, 0 mbuf_cluster: 2048, 898, 381, 3, 726, 0, 0 mbuf_jumbo_page: 4096, 448, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 132, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 74, 0, 0, 0, 0, 0 g_bio: 264, 0, 0, 30, 11703, 0, 0 ttyinq: 152, 0, 15, 37, 60, 0, 0 ttyoutq: 256, 0, 8, 22, 32, 0, 0 nvme_request: 100, 0, 0, 0, 0, 0, 0 cryptop: 60, 0, 0, 0, 0, 0, 0 cryptodesc: 56, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 16, 0, 0, 0, 0, 0, 0 FPU_save_area: 512, 0, 0, 0, 0, 0, 0 VNODE: 284, 0, 1130, 32, 1165, 0, 0 VNODEPOLL: 64, 0, 0, 0, 0, 0, 0 BUF TRIE: 44, 0, 51, 677, 1736, 0, 0 NAMEI: 1024, 0, 0, 16, 21104, 0, 0 rentr: 12, 0, 0, 0, 0, 0, 0 S VFS Cache: 72, 0, 972, 148, 2134, 0, 0 STS VFS Cache: 92, 0, 0, 0, 0, 0, 0 L VFS Cache: 292, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 312, 0, 0, 0, 0, 0, 0 NCLNODE: 360, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 9, 11, 27, 0, 0 Mountpoints: 672, 0, 4, 14, 5, 0, 0 pipe: 416, 0, 0, 9, 534, 0, 0 procdesc: 80, 0, 0, 0, 0, 0, 0 AIO: 128, 0, 0, 0, 0, 0, 0 AIOP: 16, 0, 0, 0, 0, 0, 0 AIOCB: 428, 0, 0, 0, 0, 0, 0 AIOL: 64, 0, 0, 0, 0, 0, 0 AIOLIO: 172, 0, 0, 0, 0, 0, 0 ksiginfo: 80, 0, 30, 170, 1209, 0, 0 itimer: 232, 0, 0, 0, 0, 0, 0 bridge_rtnode: 36, 0, 0, 0, 0, 0, 0 KNOTE: 72, 0, 0, 0, 0, 0, 0 socket: 524, 1393, 20, 8, 2030, 0, 0 unpcb: 172, 1403, 6, 17, 1748, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0 ipq: 32, 126, 0, 0, 0, 0, 0 udp_inpcb: 304, 1404, 11, 15, 262, 0, 0 udpcb: 20, 1414, 11, 191, 262, 0, 0 tcp_inpcb: 304, 1404, 2, 11, 11, 0, 0 tcpcb: 772, 1395, 2, 3, 11, 0, 0 tcptw: 60, 335, 0, 0, 0, 0, 0 syncache: 128, 15376, 0, 0, 4, 0, 0 hostcache: 76, 15370, 0, 0, 0, 0, 0 sackhole: 20, 0, 0, 0, 0, 0, 0 tcpreass: 20, 202, 0, 0, 0, 0, 0 sctp_ep: 1052, 1395, 0, 0, 0, 0, 0 sctp_asoc: 1656, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80136, 0, 0, 5, 0, 0 sctp_raddr: 540, 80003, 0, 0, 0, 0, 0 sctp_chunk: 104, 400026, 0, 0, 0, 0, 0 sctp_readq: 108, 400007, 0, 0, 0, 0, 0 sctp_stream_msg_out: 72, 400008, 0, 0, 0, 0, 0 sctp_asconf: 24, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400008, 0, 0, 0, 0, 0 udplite_inpcb: 304, 1404, 0, 0, 0, 0, 0 ripcb: 304, 1404, 0, 0, 0, 0, 0 rtentry: 112, 0, 42, 30, 50, 0, 0 pf mtags: 32, 0, 0, 0, 0, 0, 0 pf states: 208, 10013, 0, 0, 0, 0, 0 pf state keys: 64, 0, 0, 0, 0, 0, 0 pf source nodes: 116, 10030, 0, 0, 0, 0, 0 pf table entries: 92, 0, 0, 0, 0, 0, 0 pf table counters: 64, 0, 0, 0, 0, 0, 0 pf frags: 80, 0, 0, 0, 0, 0, 0 pf frag entries: 24, 5040, 0, 0, 0, 0, 0 pf state scrubs: 28, 0, 0, 0, 0, 0, 0 IPFW counters: 16, 0, 1, 63, 1, 0, 0 IPFW dynamic rule: 112, 16416, 0, 0, 0, 0, 0 divcb: 304, 1404, 0, 0, 0, 0, 0 selfd: 32, 0, 32, 220, 150507, 0, 0 SWAPMETA: 276, 5390, 0, 0, 0, 0, 0 FFS inode: 112, 0, 1105, 155, 1139, 0, 0 FFS1 dinode: 128, 0, 628, 147, 658, 0, 0 FFS2 dinode: 256, 0, 477, 18, 481, 0, 0 md0: 512, 0, 5041, 7, 5049, 0, 0 md1: 512, 0, 4266, 6, 4267, 0, 0=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C2042F56-2EB7-47B9-92C5-52DA21CA3132>