From nobody Mon Jan 23 01:54:31 2023 X-Original-To: freebsd-dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P0Y7Y4vGTz316hZ for ; Mon, 23 Jan 2023 01:54:33 +0000 (UTC) (envelope-from cracauer@cons.org) Received: from koef.zs64.net (koef.zs64.net [IPv6:2a00:14b0:4200:32e0::1e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P0Y7X5qQzz3Myx for ; Mon, 23 Jan 2023 01:54:32 +0000 (UTC) (envelope-from cracauer@cons.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of cracauer@cons.org designates 2a00:14b0:4200:32e0::1e6 as permitted sender) smtp.mailfrom=cracauer@cons.org; dmarc=none Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by 0ons.org (8.16.1/8.15.2) with ESMTP id 30N1sVKc096136 for ; Mon, 23 Jan 2023 01:54:31 GMT (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.16.1/8.15.2/Submit) id 30N1sVcl096135 for freebsd-dtrace@freebsd.org; Sun, 22 Jan 2023 20:54:31 -0500 (EST) (envelope-from cracauer) Date: Sun, 22 Jan 2023 20:54:31 -0500 From: Martin Cracauer To: freebsd-dtrace@freebsd.org Subject: DTrace - capturing two userspace strack frames on top of system call Message-ID: List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip6:2a00:14b0:4200:32e0::1e6:c]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-dtrace@freebsd.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[cons.org]; ASN(0.00)[asn:13135, ipnet:2a00:14b0::/32, country:DE]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[cracauer]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-dtrace@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4P0Y7X5qQzz3Myx X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N I want to capture the (userland) stack trace on top of the write(2) system call. I seem to have some difficulty switching from kernel to user mode here. For every write system call I want to print the calling userlevel frames. I can't care whether they are individually printed or counted. Here is what I think should do it: syscall::write*:entry /arg1/ { @traces[ustack()] = count(); } However, I get one error each for each write call: dtrace: error on enabled probe ID 2 (ID 56902: syscall:freebsd:write:entry): invalid address (0x0) in action #2 This gives the same error: syscall::write*:entry /arg1/ { ustack(); } %% If I use system stackframes it works, but of course it doesn't print the calling frames: syscall::write*:entry /arg1/ { @traces[stack()] = count(); } dtrace: script 'stack-to-write.dtrace' matched 3 probes dtrace: buffer size lowered to 2m dtrace: aggregation size lowered to 2m dtrace: pid 11790 has exited kernel`handle_el0_sync+0x40 136 %% Is what I am trying to do even possible? Can I mix kernel and userlevel space like this? Any other ideas? I could brute-force it with LD_PRELOAD overloading of write(2), but dtrace would be more elegant. Thanks in advance Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ From nobody Mon Jan 23 03:04:50 2023 X-Original-To: freebsd-dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P0Zhj0h4Mz3b96M for ; Mon, 23 Jan 2023 03:04:53 +0000 (UTC) (envelope-from cracauer@cons.org) Received: from koef.zs64.net (koef.zs64.net [IPv6:2a00:14b0:4200:32e0::1e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P0Zhh0sQ0z3RH7 for ; Mon, 23 Jan 2023 03:04:51 +0000 (UTC) (envelope-from cracauer@cons.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of cracauer@cons.org designates 2a00:14b0:4200:32e0::1e6 as permitted sender) smtp.mailfrom=cracauer@cons.org; dmarc=none Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by 0ons.org (8.16.1/8.15.2) with ESMTP id 30N34oZ1002478 for ; Mon, 23 Jan 2023 03:04:50 GMT (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.16.1/8.15.2/Submit) id 30N34oLE002477 for freebsd-dtrace@freebsd.org; Sun, 22 Jan 2023 22:04:50 -0500 (EST) (envelope-from cracauer) Date: Sun, 22 Jan 2023 22:04:50 -0500 From: Martin Cracauer To: freebsd-dtrace@freebsd.org Subject: Re: DTrace - capturing two userspace strack frames on top of system call Message-ID: References: List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip6:2a00:14b0:4200:32e0::1e6]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:13135, ipnet:2a00:14b0::/32, country:DE]; MLMMJ_DEST(0.00)[freebsd-dtrace@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[cons.org]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[cracauer]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-dtrace@freebsd.org]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4P0Zhh0sQ0z3RH7 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Actually the error only appears on arm64 for me. I moved the script to amd64 and it works as I thought. Martin Martin Cracauer wrote on Sun, Jan 22, 2023 at 08:54:31PM -0500: > I want to capture the (userland) stack trace on top of the write(2) > system call. I seem to have some difficulty switching from kernel to > user mode here. For every write system call I want to print the > calling userlevel frames. I can't care whether they are individually > printed or counted. > > Here is what I think should do it: > syscall::write*:entry /arg1/ { @traces[ustack()] = count(); } > > However, I get one error each for each write call: > dtrace: error on enabled probe ID 2 (ID 56902: > syscall:freebsd:write:entry): invalid address (0x0) in action #2 > > This gives the same error: > syscall::write*:entry /arg1/ { ustack(); } > > > %% > > If I use system stackframes it works, but of course it doesn't print > the calling frames: > > syscall::write*:entry /arg1/ { @traces[stack()] = count(); } > > dtrace: script 'stack-to-write.dtrace' matched 3 probes > dtrace: buffer size lowered to 2m > dtrace: aggregation size lowered to 2m > dtrace: pid 11790 has exited > > > kernel`handle_el0_sync+0x40 > 136 > > %% > > Is what I am trying to do even possible? Can I mix kernel and > userlevel space like this? > > Any other ideas? I could brute-force it with LD_PRELOAD overloading > of write(2), but dtrace would be more elegant. > > Thanks in advance > Martin > -- > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > Martin Cracauer http://www.cons.org/cracauer/ -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ From nobody Mon Jan 23 16:52:12 2023 X-Original-To: freebsd-dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P0x3N3Rf2z30wwg for ; Mon, 23 Jan 2023 16:52:16 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-il1-x130.google.com (mail-il1-x130.google.com [IPv6:2607:f8b0:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P0x3N1Mn9z41rG for ; Mon, 23 Jan 2023 16:52:16 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-il1-x130.google.com with SMTP id p12so6187440ilq.10 for ; Mon, 23 Jan 2023 08:52:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=mVLWiaSaIQ5VenQvdTfPesu0k2LBEKrVplzx37DsuV8=; b=JROMsHMHWAW3osuhiABMR0MwKYf/TNUwjPTcLtlwl1HJM4Oqpgk69zoXcrusJrIhUT 2eehQXJttgr8RBLxRk6O26WtIAJwqKyKKGmh1hGT1EEV3KEZEjwAtI0vpP7UK50hR1cT SPFnHXkDoKwQirt6dVDTkUBSZ9I7wlSDEw+InIXPUxzANmPXkL9IAPNToDVLA8/w82oO E1DkpDKgE47YNkHD2FXl1v3JmEbef3sl4JbAq2xse5UJZXLMiQHNNfl2lr8CgXuSBFJi je8Nft8LlNOwjtULeiW2Y/WP7/NH3tvrXo82Z2UDDAeZmBmaG5LLGUSb6PKsZlIFdOv7 3Hyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mVLWiaSaIQ5VenQvdTfPesu0k2LBEKrVplzx37DsuV8=; b=g3w2/FMMa9P+C5M48Hkt8CBcBeeYRe/AMl37FPIFnMohaFKY5wmldoy2eXKsqYAdFg +DiVb2+rF2mVUINmoSveFR0fOu4hQuJ32tosqo05LAJy1wVkpVt3aq9uv47+AJb5QXVa WLqFuHQMPngjVSRSNCTUKWy/sq6d5mv+7h85svCOaDuFGQCY3qn24XZMqpjpvne51cV0 +65x0DbUMijxCrpoy91q8aeMOmO+IsGzRTR4OsYnOPejogkVuAjdSYPnfNaaO5r9vrUb H0BYXghZ9Mxu0bIhAyZB8uqYtYwFqOFp71oeN78AK6VxZvgIOhor+Cl4/t2I9Ceo/hDg svvw== X-Gm-Message-State: AFqh2kqPF8kQnjtBEc7eV9VuxImxmrBrMCklYcjKGmKlelfoduaUaesl 1PLeBlhwM2PXPVLQsNfOnOhHIUZcqZ8= X-Google-Smtp-Source: AMrXdXtnSo4esuj5NqA2UxpfSgz48rhoz5bbwdLvusFwOF4E+XpstxjsBY26LZKcy4pYm79rxd9nIw== X-Received: by 2002:a92:c749:0:b0:30e:f331:d529 with SMTP id y9-20020a92c749000000b0030ef331d529mr23780996ilp.1.1674492735250; Mon, 23 Jan 2023 08:52:15 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id s2-20020a02cf22000000b003a47dd62351sm7976096jar.144.2023.01.23.08.52.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 08:52:14 -0800 (PST) Date: Mon, 23 Jan 2023 11:52:12 -0500 From: Mark Johnston To: Martin Cracauer Cc: freebsd-dtrace@freebsd.org Subject: Re: DTrace - capturing two userspace strack frames on top of system call Message-ID: References: List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4P0x3N1Mn9z41rG X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sun, Jan 22, 2023 at 10:04:50PM -0500, Martin Cracauer wrote: > Actually the error only appears on arm64 for me. I moved the script > to amd64 and it works as I thought. Support for userspace tracing on arm64 is definitely less mature than on amd64, so this isn't too surprising, unfortunately. Is the problem reproducible with a trivial program compiled with -fno-omit-frame-pointer? > Martin > > Martin Cracauer wrote on Sun, Jan 22, 2023 at 08:54:31PM -0500: > > I want to capture the (userland) stack trace on top of the write(2) > > system call. I seem to have some difficulty switching from kernel to > > user mode here. For every write system call I want to print the > > calling userlevel frames. I can't care whether they are individually > > printed or counted. > > > > Here is what I think should do it: > > syscall::write*:entry /arg1/ { @traces[ustack()] = count(); } > > > > However, I get one error each for each write call: > > dtrace: error on enabled probe ID 2 (ID 56902: > > syscall:freebsd:write:entry): invalid address (0x0) in action #2 > > > > This gives the same error: > > syscall::write*:entry /arg1/ { ustack(); } > > > > > > %% > > > > If I use system stackframes it works, but of course it doesn't print > > the calling frames: > > > > syscall::write*:entry /arg1/ { @traces[stack()] = count(); } > > > > dtrace: script 'stack-to-write.dtrace' matched 3 probes > > dtrace: buffer size lowered to 2m > > dtrace: aggregation size lowered to 2m > > dtrace: pid 11790 has exited > > > > > > kernel`handle_el0_sync+0x40 > > 136 > > > > %% > > > > Is what I am trying to do even possible? Can I mix kernel and > > userlevel space like this? > > > > Any other ideas? I could brute-force it with LD_PRELOAD overloading > > of write(2), but dtrace would be more elegant. > > > > Thanks in advance > > Martin > > -- > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > Martin Cracauer http://www.cons.org/cracauer/ > > -- > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > Martin Cracauer http://www.cons.org/cracauer/ > From nobody Mon Jan 23 21:54:05 2023 X-Original-To: freebsd-dtrace@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P13lk0bV3z3bVmS for ; Mon, 23 Jan 2023 21:54:10 +0000 (UTC) (envelope-from cracauer@cons.org) Received: from koef.zs64.net (koef.zs64.net [IPv6:2a00:14b0:4200:32e0::1e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P13lj4c11z3HG8; Mon, 23 Jan 2023 21:54:09 +0000 (UTC) (envelope-from cracauer@cons.org) Authentication-Results: mx1.freebsd.org; none Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by 0ons.org (8.16.1/8.15.2) with ESMTP id 30NLs5K0028351; Mon, 23 Jan 2023 21:54:05 GMT (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.16.1/8.15.2/Submit) id 30NLs5Jd028350; Mon, 23 Jan 2023 16:54:05 -0500 (EST) (envelope-from cracauer) Date: Mon, 23 Jan 2023 16:54:05 -0500 From: Martin Cracauer To: Mark Johnston Cc: Martin Cracauer , freebsd-dtrace@freebsd.org Subject: Re: DTrace - capturing two userspace strack frames on top of system call Message-ID: References: List-Id: A discussion list for developers working on DTrace in FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-dtrace List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-dtrace@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4P13lj4c11z3HG8 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:13135, ipnet:2a00:14b0::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Mark Johnston wrote on Mon, Jan 23, 2023 at 11:52:12AM -0500: > On Sun, Jan 22, 2023 at 10:04:50PM -0500, Martin Cracauer wrote: > > Actually the error only appears on arm64 for me. I moved the script > > to amd64 and it works as I thought. > > Support for userspace tracing on arm64 is definitely less mature than on > amd64, so this isn't too surprising, unfortunately. > > Is the problem reproducible with a trivial program compiled with > -fno-omit-frame-pointer? Yes: $ gcc -O2 -fno-omit-frame-pointer -Wall -Werror -o crasparse crasparse.c # dtrace -n 'syscall::write*:entry /arg1/ { @traces[ustack()] = count(); }' -c ./crasparse dtrace: error on enabled probe ID 1 (ID 57136: syscall:freebsd:writev:entry): invalid address (0x0) in action #2 [...] Should I make a bugzilla entry out of this? Martin > > Martin Cracauer wrote on Sun, Jan 22, 2023 at 08:54:31PM -0500: > > > I want to capture the (userland) stack trace on top of the write(2) > > > system call. I seem to have some difficulty switching from kernel to > > > user mode here. For every write system call I want to print the > > > calling userlevel frames. I can't care whether they are individually > > > printed or counted. > > > > > > Here is what I think should do it: > > > syscall::write*:entry /arg1/ { @traces[ustack()] = count(); } > > > > > > However, I get one error each for each write call: > > > dtrace: error on enabled probe ID 2 (ID 56902: > > > syscall:freebsd:write:entry): invalid address (0x0) in action #2 > > > > > > This gives the same error: > > > syscall::write*:entry /arg1/ { ustack(); } > > > > > > > > > %% > > > > > > If I use system stackframes it works, but of course it doesn't print > > > the calling frames: > > > > > > syscall::write*:entry /arg1/ { @traces[stack()] = count(); } > > > > > > dtrace: script 'stack-to-write.dtrace' matched 3 probes > > > dtrace: buffer size lowered to 2m > > > dtrace: aggregation size lowered to 2m > > > dtrace: pid 11790 has exited > > > > > > > > > kernel`handle_el0_sync+0x40 > > > 136 > > > > > > %% > > > > > > Is what I am trying to do even possible? Can I mix kernel and > > > userlevel space like this? > > > > > > Any other ideas? I could brute-force it with LD_PRELOAD overloading > > > of write(2), but dtrace would be more elegant. > > > > > > Thanks in advance > > > Martin > > > -- > > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > > Martin Cracauer http://www.cons.org/cracauer/ > > > > -- > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > Martin Cracauer http://www.cons.org/cracauer/ > > -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/