From owner-freebsd-fs@freebsd.org Tue Nov 22 07:09:31 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E966C4E83C for ; Tue, 22 Nov 2016 07:09:31 +0000 (UTC) (envelope-from jamesmorris8@outlook.com) Received: from BLU004-OMC4S20.hotmail.com (blu004-omc4s20.hotmail.com [65.55.111.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3370A1B9F for ; Tue, 22 Nov 2016 07:09:30 +0000 (UTC) (envelope-from jamesmorris8@outlook.com) Received: from IND01-MA1-obe.outbound.protection.outlook.com ([65.55.111.137]) by BLU004-OMC4S20.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Mon, 21 Nov 2016 23:08:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=JiCP0Zn9cQPUMLpTI2CV7DgEFKbGI0H/vIU+arehFhM=; b=uwz+XKwA+7o6z2/zBdUMOrDfzPFJJnry86dvbZYoK7EuwyLE/RRUqtfmn6QcbeqYHJJAQdOvB/VRBYeSgNatfPKjMEXJEXodGkuAKmogxKAcSOnMrXWP8OoZQ+yoms6cvsp7+bgMRAWoD/3trJabqvCau/e9FrdLKBfkbMpO1gjFi74e3UnReKD7rqYg2FXCgFHeyoGwSgp/t2rG9Aniw+ZDUWmv8Da+J8HP6XV0etOCx2QIxyZyjxaJ8oYWzrqrSYmG09yGrByFaSpV+K3nJ60vPDpqM7ndtGBS//0fb481e5cqMgaX7B8E6xV3DPAPUGB8+Q77DK/fHmqlY0XR+g== Received: from MA1IND01FT003.eop-IND01.prod.protection.outlook.com (10.152.200.58) by MA1IND01HT003.eop-IND01.prod.protection.outlook.com (10.152.200.128) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.721.5; Tue, 22 Nov 2016 07:08:13 +0000 Received: from BM1PR01MB0209.INDPRD01.PROD.OUTLOOK.COM (10.152.200.51) by MA1IND01FT003.mail.protection.outlook.com (10.152.200.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.721.5 via Frontend Transport; Tue, 22 Nov 2016 07:08:13 +0000 Received: from BM1PR01MB0209.INDPRD01.PROD.OUTLOOK.COM ([10.164.129.147]) by BM1PR01MB0209.INDPRD01.PROD.OUTLOOK.COM ([10.164.129.147]) with mapi id 15.01.0734.007; Tue, 22 Nov 2016 07:08:12 +0000 From: James Morris To: freebsd-fs Subject: Unresponsive system, ZFS ARK forcing proceses into swap Thread-Topic: Unresponsive system, ZFS ARK forcing proceses into swap Thread-Index: AQHSRIxYt2P25AuZpEeTIliP4ppnMg== Date: Tue, 22 Nov 2016 07:08:11 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=outlook.com; x-incomingtopheadermarker: OriginalChecksum:; UpperCasedChecksum:; SizeAsReceived:7243; Count:36 x-tmn: [6cGT0ue668HrrHxN20sAyrrkOJxLMd5G] x-incomingheadercount: 36 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1; MA1IND01HT003; 5:JfJtlzaGPPh5ZOd4ddJ6D5gxKW7JF/01HnVh++P7YPF6U/v4iPLuhPBYlkyI8hEdmz46Zdqabgi+XnteA51mTk+IDdeDSPNdsZZJ7urZN0K5LXXZ58WvZ4/xez7evj5AYoBp5LNvwg7TJWDevuvOig==; 24:NQ9AOtQjjkWLxMY5MIZD31+05sVUsF3RRXJuAvrIIzMPZT6HT7/bV4khxDbQ+JCebJmdyNwzz5FYgvkP3kvEZhbXUYPlfThNUKnBHI+q4M0=; 7:bO7yg4fcq2ERlEE+fnQQpkgSnEBK/h16VsswQ86fd8dOxS1HrHvcXfHz63ZRUocHANqGoVpefImiUqjwZueqfvJwh2tyWnb9Y0l67xPTcz4/lqv0YeUJDKf8665ZeIKn5UlL1IA+eRABxaEXSIhUEGMvuzUIedTDG0gHytVq6qO6UDqmap7OXg/DaW/C9rMjjHPzoC291uVmZlXhwpMB/Bv5D6eJr+3RiK+U3ZVhPB8WB7kzinggH0gFZ1IuBM8AGGSNKEdOqq+hSNi54NZbhOJZ9mXuru7faAsO6CB3oeNNw1aihCl4XVmAQrE8iV/GEfhx0rlwnHrtUdqIpmodgfZBNYVkQV0IG/YSpH4cncU= x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(10019020)(98900003); DIR:OUT; SFP:1102; SCL:1; SRVR:MA1IND01HT003; H:BM1PR01MB0209.INDPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; x-ms-office365-filtering-correlation-id: e22895eb-573d-4f45-5ad3-08d412a652c1 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(1601124038)(1603103113)(1603101340)(1601125047); SRVR:MA1IND01HT003; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(432015012)(82015046); SRVR:MA1IND01HT003; BCL:0; PCL:0; RULEID:; SRVR:MA1IND01HT003; x-forefront-prvs: 0134AD334F spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Nov 2016 07:08:11.8829 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: MA1IND01HT003 X-OriginalArrivalTime: 22 Nov 2016 07:08:24.0969 (UTC) FILETIME=[3785BF90:01D2448F] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 07:09:31 -0000 Hi, I have a freebsd 10.3 zfs server running with multiple zfs jails. I run samba, syslog-ng, a web server and a few assorted other services in e= ach jail. The machine is not used much, but I have noticed that I can not ssh into th= e base system or reach a web server that I run on the base system after a f= ew days of running. On connecting via the console I notice that arc is taking up a bulk of the = ram and it seems processes like sshd and nginx are frozen. Also the swap space is utilised and the ARC is taking up the majority of my= RAM. When I try and run an interactive command like top, less, ... It loads, but= the system takes exceedingly long 10min+ to close these. The same symptoms are true when I try to use ^c to quit a running process l= ike ping. Any help triaging this issue will be much appreciated. I believe that the ZFS ARC has something to do with it, but I do not want t= o turn it off altogether. Is there a way to make sure ZFS does not push my other processes (sshd, ngi= nx, ...) into swap where they are unresponsive. Output of top while in this state is below last pid: 31465; load averages: 0.43, 0.56, 0.54 up 1+08:18:48 13:5= 9:44 319 processes: 1 running, 308 sleeping, 1 stopped, 3 zombie, 6 waiting Mem: 82M Active, 336K Inact, 15G Wired, 76M Cache, 12M Free ARC: 12G Total, 3503M MFU, 26M MRU, 13M Anon, 22M Header, 8572M Other Swap: 2048M Total, 1064M Used, 984M Free, 51% Inuse Thanks in advance, James=