Date: Sat, 4 Jan 2020 23:35:33 +0100 (CET) From: Wojciech Puchar <wojtek@puchar.net> To: freebsd-hackers@freebsd.org Subject: processes are killed because of out of swap space Message-ID: <alpine.BSF.2.20.2001042332540.20033@puchar.net>
next in thread | raw e-mail | index | archive | help
when i try to use more virtual memory (tested by putting files to tmpfs
/tmp).
like that
pid 16977 (bhyve), jid 0, uid 0, was killed: out of swap space
the problem is that it's less than 10GB swap used while i have 120GB
available.
before processed begin to be killed system stalls for a while.
any idea what to check?
below my sysctl -a vm
vm.vmtotal:
System wide totals computed every five seconds: (values in kilobytes)
===============================================
Processes: (RUNQ: 2 Disk Wait: 13 Page Wait: 0 Sleep: 614)
Virtual Memory: (Total: 35115848K Active: 34336216K)
Real Memory: (Total: 17559404K Active: 17498736K)
Shared Virtual Memory: (Total: 819960K Active: 231808K)
Shared Real Memory: (Total: 117712K Active: 59860K)
Free Memory: 707048K
vm.loadavg: { 1,56 1,61 1,82 }
vm.v_free_min: 51326
vm.v_free_target: 173333
vm.v_free_reserved: 10657
vm.v_inactive_target: 259999
vm.v_pageout_free_min: 34
vm.swap_enabled: 1
vm.overcommit: 1
vm.kvm_free: 2160297242624
vm.kvm_size: 2199023251456
vm.pmap.pdpe.demotions: 3
vm.pmap.pde.promotions: 60330
vm.pmap.pde.p_failures: 42765
vm.pmap.pde.mappings: 14905
vm.pmap.pde.demotions: 15645
vm.pmap.allow_2m_x_ept: 1
vm.pmap.pcid_save_cnt: 142000859
vm.pmap.pti: 1
vm.pmap.invpcid_works: 1
vm.pmap.pcid_enabled: 1
vm.pmap.pg_ps_enabled: 1
vm.pmap.pat_works: 1
vm.idlezero_enable: 0
vm.swap_idle_threshold2: 360
vm.swap_idle_threshold1: 180
vm.swap_idle_enabled: 1
vm.reserv.reclaimed: 10817
vm.reserv.partpopq:
LEVEL SIZE NUMBER
-1: 0K, 0
vm.reserv.fullpop: 380
vm.reserv.freed: 97537
vm.reserv.broken: 11002
vm.default_policy: rr
vm.ndomains: 1
vm.phys_segs:
SEGMENT 0:
start: 0x10000
end: 0x96000
domain: 0
free list: 0xffffffff80d5bf10
SEGMENT 1:
start: 0x100000
end: 0x200000
domain: 0
free list: 0xffffffff80d5bf10
SEGMENT 2:
start: 0x200000
end: 0x1000000
domain: 0
free list: 0xffffffff80d5bf10
SEGMENT 3:
start: 0x1000000
end: 0x1372000
domain: 0
free list: 0xffffffff80d5bca0
SEGMENT 4:
start: 0x1378000
end: 0x13a3000
domain: 0
free list: 0xffffffff80d5bca0
SEGMENT 5:
start: 0x1400000
end: 0x791c9000
domain: 0
free list: 0xffffffff80d5bca0
SEGMENT 6:
start: 0x100000000
end: 0x849e72000
domain: 0
free list: 0xffffffff80d5bca0
vm.phys_free:
DOMAIN 0:
FREE LIST 0:
ORDER (SIZE) | NUMBER
| POOL 0 | POOL 1
-- -- -- -- -- --
12 ( 16384K) | 0 | 0
11 ( 8192K) | 0 | 0
10 ( 4096K) | 0 | 0
9 ( 2048K) | 0 | 0
8 ( 1024K) | 0 | 0
7 ( 512K) | 0 | 0
6 ( 256K) | 52 | 1
5 ( 128K) | 462 | 6
4 ( 64K) | 2152 | 9
3 ( 32K) | 9950 | 36
2 ( 16K) | 10623 | 125
1 ( 8K) | 182 | 121
0 ( 4K) | 0 | 3
FREE LIST 1:
ORDER (SIZE) | NUMBER
| POOL 0 | POOL 1
-- -- -- -- -- --
12 ( 16384K) | 0 | 0
11 ( 8192K) | 0 | 0
10 ( 4096K) | 0 | 0
9 ( 2048K) | 0 | 0
8 ( 1024K) | 0 | 0
7 ( 512K) | 0 | 0
6 ( 256K) | 0 | 0
5 ( 128K) | 0 | 0
4 ( 64K) | 8 | 0
3 ( 32K) | 9 | 0
2 ( 16K) | 15 | 0
1 ( 8K) | 18 | 0
0 ( 4K) | 21 | 0
vm.max_wired: 2659648
vm.background_launder_max: 20480
vm.background_launder_rate: 4096
vm.background_launder_target: 12200
vm.act_scan_laundry_weight: 3
vm.pageout_oom_seq: 12
vm.pageout_lock_miss: 0
vm.disable_swapspace_pageouts: 0
vm.lowmem_period: 1
vm.pageout_update_period: 600
vm.pageout_wakeup_thresh: 56452
vm.panic_on_oom: 0
vm.page_blacklist:
vm.tryrelock_restart: 31748
vm.boot_pages: 64
vm.old_msync: 0
vm.mincore_mapped: 1
vm.old_mlock: 0
vm.stats.object.bypasses: 18723
vm.stats.object.collapses: 78824
vm.stats.misc.cnt_prezero: 0
vm.stats.misc.zero_page_count: 325
vm.stats.vm.v_kthreadpages: 0
vm.stats.vm.v_rforkpages: 100
vm.stats.vm.v_vforkpages: 52157
vm.stats.vm.v_forkpages: 1018587
vm.stats.vm.v_kthreads: 103
vm.stats.vm.v_rforks: 2
vm.stats.vm.v_vforks: 1484
vm.stats.vm.v_forks: 18704
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_cache_count: 0
vm.stats.vm.v_laundry_count: 841044
vm.stats.vm.v_inactive_count: 2695784
vm.stats.vm.v_inactive_target: 259999
vm.stats.vm.v_active_count: 3536963
vm.stats.vm.v_wire_count: 883581
vm.stats.vm.v_free_count: 176747
vm.stats.vm.v_free_min: 51326
vm.stats.vm.v_free_target: 173333
vm.stats.vm.v_free_reserved: 10657
vm.stats.vm.v_page_count: 8134081
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_tfree: 714905915
vm.stats.vm.v_pfree: 38188448
vm.stats.vm.v_dfree: 139195941
vm.stats.vm.v_tcached: 0
vm.stats.vm.v_pdshortfalls: 114
vm.stats.vm.v_pdpages: 381445429
vm.stats.vm.v_pdwakeups: 1093
vm.stats.vm.v_reactivated: 1829047
vm.stats.vm.v_intrans: 1165647
vm.stats.vm.v_vnodepgsout: 780
vm.stats.vm.v_vnodepgsin: 407218
vm.stats.vm.v_vnodeout: 383
vm.stats.vm.v_vnodein: 74309
vm.stats.vm.v_swappgsout: 11838348
vm.stats.vm.v_swappgsin: 2406191
vm.stats.vm.v_swapout: 825054
vm.stats.vm.v_swapin: 1142018
vm.stats.vm.v_ozfod: 1020
vm.stats.vm.v_zfod: 33027903
vm.stats.vm.v_cow_optim: 2572
vm.stats.vm.v_cow_faults: 2530416
vm.stats.vm.v_io_faults: 310268
vm.stats.vm.v_vm_faults: 44268076
vm.stats.sys.v_soft: 5808251
vm.stats.sys.v_intr: 8974021
vm.stats.sys.v_syscall: 251679096
vm.stats.sys.v_trap: 272835752
vm.stats.sys.v_swtch: 260966303
vm.v_free_severe: 30991
vm.max_kernel_address: 18446744073709547520
vm.min_kernel_address: 18446741874686296064
vm.kstacks: 1130
vm.kstack_cache_size: 128
vm.zone_warnings: 1
vm.zone_count: 120
vm.nswapdev: 1
vm.dmmax: 32
vm.swap_fragmentation:
Free space on device gpt/swap.eli:
number of maximal free ranges: 20935
largest free range: 20512000
average maximal free range size: 1418
number of maximal free ranges of different sizes:
count | size range
----- | ----------
12501 | 1
2117 | 2
1473 | 3 to 4
789 | 5 to 7
658 | 8 to 12
593 | 13 to 20
906 | 21 to 33
298 | 34 to 54
298 | 55 to 88
240 | 89 to 143
191 | 144 to 232
176 | 233 to 376
137 | 377 to 609
92 | 610 to 986
105 | 987 to 1596
91 | 1597 to 2583
76 | 2584 to 4180
51 | 4181 to 6764
37 | 6765 to 10945
33 | 10946 to 17710
27 | 17711 to 28656
10 | 28657 to 46367
14 | 46368 to 75024
9 | 75025 to 121392
3 | 121393 to 196417
2 | 196418 to 317810
4 | 317811 to 514228
3 | 514229 to 832039
1 | 20512000
vm.swap_async_max: 4
vm.swap_maxpages: 65072752
vm.swzone: 553118392
vm.swap_reserved: 11839164416
vm.swap_total: 128849018880
vm.phys_pager_cluster: 1024
vm.kmem_map_free: 32650514432
vm.kmem_map_size: 666681344
vm.kmem_size_scale: 1
vm.kmem_size_max: 1319413950874
vm.kmem_size_min: 0
vm.kmem_zmax: 65536
vm.kmem_size: 33317195776
vm.md_malloc_wait: 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.20.2001042332540.20033>