From owner-freebsd-hackers@freebsd.org Mon Mar 21 16:02:09 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B5A1AD856C for ; Mon, 21 Mar 2016 16:02:09 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C9FF2328 for ; Mon, 21 Mar 2016 16:02:08 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-ig0-x22d.google.com with SMTP id kc10so70853250igb.0 for ; Mon, 21 Mar 2016 09:02:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=KVn5IxpC7MBCOPM2jyGSHCCAW28RwZRIS4hnERgQKbM=; b=h4kkJTLXqF3w9t5mXe3TDXa3KcIdaz2d2iqNdn6WQWh7EQjNjWwUYTf/I23pNHiBK1 2DHcsC8L060VLTBDfMDqriBPG9CeYl+OFP0SHI4IOKdFlCjsnD+TRRfjKA/9KT0kw1U6 YIRkcfhrxgfixizRaO3fwRTWCPNpa/2TSUPM0NZoNawBnnMnq0RPOShHGPaIcdqoGYK4 sNf//041belLUIF+mscdPePgFY5xcuykoL2dLVMJNDKJeeimQHZQzqrCOkugoZ8CDnZd J0z0FdmGRupwJvV5Ged/4Ic5Ah8OSrkUSIxpuIFO1U3d9TkvSx9J2C2CgrpUattscQ+z UmDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=KVn5IxpC7MBCOPM2jyGSHCCAW28RwZRIS4hnERgQKbM=; b=Dki3+WwouRNiICr0XNilRVp6mH5r2srqnQpWxdRs2uX95rL//ad+lQd14D4Hs+jIfN iP+5lmcOLQ3JkqTffcYJy+TNfKz90BrIv5nWHN7GML4Xx6K+EG2Xz+MViHl+JvSn7KFf vrGPxgcxJwCuxWwSkB0qKQXgY3aQX7nQVz790KOpcGN34n+s52OJ3g6ZKTUsrhe7cl7b LbsW7fcVoDc1oBGXfht2MYOQJS7KpYHcBq8lOVFAj5/vKKibKW072AUiMNoicng5MMYK nKxfHWUrQ7BDv5OyJldCfgNb9svwX5/mZlYIrcVQG6NKWbqMDhhKJGvb5RnCLuBqjF+u 9tAQ== X-Gm-Message-State: AD7BkJKVaa4M0qV8zLaOv8E9cSZZs5F3ibdDcK4yejlZGIF62d3xkl26+QJnOtTly+H9FFzfDX2xxK5/6HokEQ== MIME-Version: 1.0 X-Received: by 10.50.61.232 with SMTP id t8mr13566559igr.83.1458576128227; Mon, 21 Mar 2016 09:02:08 -0700 (PDT) Received: by 10.107.6.32 with HTTP; Mon, 21 Mar 2016 09:02:08 -0700 (PDT) In-Reply-To: <201603210349.u2L3nuGu027447@hypatia.MAE.cwru.edu> References: <201603210349.u2L3nuGu027447@hypatia.MAE.cwru.edu> Date: Mon, 21 Mar 2016 12:02:08 -0400 Message-ID: Subject: Re: write(2) returns ERR#12 'Cannot allocate memory' ? From: Ryan Stone To: X9Y9NWTX8KC@falstaff.mae.cwru.edu Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Mar 2016 16:02:09 -0000 You could try running this (lightly tested) DTrace script to narrow down where the error comes from. Be warned that there can be false positives. I tried to filter out the ones I saw in my quick tests but you may see others. #!/usr/sbin/dtrace -s #pragma D option quiet syscall::write:entry { self->in_write = 1; } fbt::lapic_handle_timer:entry { self->in_timer = 1; } fbt::lapic_handle_timer:return { self->in_timer = 0; } fbt:::return / self->in_write && !self->in_timer && probefunc != "tcp_addoptions" && probefunc != "maybe_yield" && probefunc != "sched_pickcpu" && arg1 == 12 / { printf("Returned %d from %s\n", arg1, probefunc); } syscall::write:return { self->in_write = 0; } On Sun, Mar 20, 2016 at 11:49 PM, Jim Berilla < X9Y9NWTX8KC@falstaff.mae.cwru.edu> wrote: > I've been trying to use the program gdrive-freebsd-x64 (downloaded from > github) to upload some files to Google Drive. One of the files is 140 GB > in size. Out of four tries, it has failed all four times, after anywhere > between 27 and 121 GB transferred. > > The error message is: > > |> Failed to upload file: \ > |> Post https://www.googleapis.com/upload/drive/v3/files?alt=[...]: \ > |> write tcp 129.22.152.23:15810->216.58.216.234:443: \ > |> write: cannot allocate memory > > On the last attempt, I ran truss on the process to see what it was doing. > Here is the output around the time of failure: > > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\ti"...,4125) = 4125 (0x101d) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\tj."...,4125) = 4125 (0x101d) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\tk"...,4125) = 4125 (0x101d) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd6dcd,0x800bebd20},64,0x0) = > 1 (0x1) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd68c4,0x800bebd20},64,0x0) = > 1 (0x1) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\tlm:"...,4125) = 4125 (0x101d) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd5db0,0x800bebd20},64,0x0) = > 1 (0x1) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd5db0,0x800bebd20},64,0x0) = > 1 (0x1) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\tmR"...,4125) = 4125 (0x101d) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\tn"...,4125) = 4125 (0x101d) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd5db0,0x800bebd20},64,0x0) = > 1 (0x1) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\toOQ"...,4125) = 4125 (0x101d) > |> write(3,"\^W\^C\^C\^P\^X\0\0\0\0\0\t\tp`"...,4125) ERR#12 'Cannot > allocate memory' > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd488a,0x800bebd20},64,0x0) = > 1 (0x1) > |> _umtx_op(0xc8204c4510,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x0,0x0) = 0 (0x0) > |> _umtx_op(0xc8204c4510,UMTX_OP_WAKE_PRIVATE,0x1,0x0,0x0) = 0 (0x0) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd4d93,0x800bebd20},64,0x0) = > 1 (0x1) > |> nanosleep({0.000100000 }) = 0 (0x0) > |> > kevent(4,0x0,0,{0x3,EVFILT_WRITE,EV_CLEAR,0,0xd8e07,0x800bebd20},64,0x0) = > 1 (0x1) > |> write(3,"\^U\^C\^C\0\^Z\0\0\0\0\0\t\tqn"...,31) = 31 (0x1f) > > If I read that correctly, the write(2) system call is returning a value > of 12 (ENOMEM). That is not a value listed under ERRORS in the man page > for that system call. > > I assume FD 3 is a socket opened to one of Google's machines. > > This system has 128 GB of physical memory, and I added 256 GB of swap > after the second failed attempt. top showed the process had a SIZE of > about 50 MB. Nothing[1] else shows evidence of memory starvation. As > far as I can tell, there is always at least a couple GB free memory, > plus another 40 to 80 GB in the ARC. Not to mention 238 GB of free > swap now. > > Interesting that there is a successful write to the same FD about seven > lines after the failed one. > > On the problem machine, uname -a says: > > |> FreeBSD kirara 10.2-RELEASE-p7 FreeBSD 10.2-RELEASE-p7 #0: \ > |> Mon Nov 2 14:19:39 UTC 2015 \ > |> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > > On a Solaris 11 system, the transfer worked the first time. > > On a FreeBSD 10.1-RELEASE-p16 system with only 4 GB of memory, it also > worked the first time, using the exact same binary as used on the problem > machine. > > I'm familiar with C programming, some on the system level. But I am new > to FreeBSD and its file hierarchy. > > If someone can point me to the right file, I'd be glad to examine the > code and try to figure out how it can return this error value. If > anyone has any thoughts on this problem, I can explore them. > > Thanks. > > [1] Or so I thought. After starting to look into this problem, I was > logged in remotely to this machine via SSH, and suddenly the terminal > window closed without warning. Looking through /var/log/messages, I > find: > > |> Mar 11 10:48:26 kirara sshd[31353]: fatal: Write failed: Cannot > allocate memory > > -- > Jim Berilla, X9Y9NWTX8KC@falstaff.MAE.cwru.edu, +1 216 368 6776 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >