From owner-soc-status@FreeBSD.ORG Mon Jun 2 13:04:34 2014 Return-Path: Delivered-To: soc-status@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C81AB78C; Mon, 2 Jun 2014 13:04:34 +0000 (UTC) Received: from mx1.mail.bg (mx1.mail.bg [IPv6:2001:67c:16b8:1::2:17]) by mx1.freebsd.org (Postfix) with ESMTP id 46E6922D5; Mon, 2 Jun 2014 13:04:34 +0000 (UTC) Received: from [10.1.1.159] (unknown [95.87.254.225]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.mail.bg (Postfix) with ESMTPSA id 263DB6003F3B; Mon, 2 Jun 2014 16:04:33 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=mail.bg; s=default; t=1401714273; bh=C7PM38EZkdTg+Os81THYrrhRqvW5Xdae48bmXhYcviQ=; h=Subject:Mime-Version:Content-Type:From:In-Reply-To:Date:Cc: Content-Transfer-Encoding:Message-Id:References:To; b=w4kzhkS2zenCqAjfFkqwgYtPiyRKV6l87U2fOoXedjPKsUNPh85j3YZRpxnFWUx3l MrYeE6scZegh+aXGS5npO5DSaj3VuhZHIIEbPkNX73jqtfAnWIWC1CkSMDDtUxMxU3 Dr3HSIDmqLRzSuY+e9EMCCEJK9Kg39PtmwzRh0y0= Subject: Re: [Machine readable output from userland utilities] report Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: Zaro Korchev In-Reply-To: <1ED224CA-9E91-459F-9ADE-E614755D1AA2@gmail.com> Date: Mon, 2 Jun 2014 16:04:30 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <82CCE1ED-525A-4F24-9911-BE3A26D2FC4F@mail.bg> References: <8D1B686D-1AAA-4E07-9270-E42699110561@mail.bg> <4890861C-FC91-445D-AE9B-31CD5FDFD0A9@theravensnest.org> <15BC1D7C-B909-48DB-AB6D-FF0F0F9C2B0A@mail.bg> <899129C5-977C-4CE7-A873-460D69D6EA85@theravensnest.org> <5385BD49.5020602@FreeBSD.org> <105B3699-2A41-4384-8D02-523D8445436B@mail.bg> <1ED224CA-9E91-459F-9ADE-E614755D1AA2@gmail.com> To: Jonathan Anderson X-Mailer: Apple Mail (2.1283) Cc: Vsevolod Stakhov , Eitan Adler , David Chisnall , Pawel Jakub Dawidek , soc-status@freebsd.org, Jonathan Anderson X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2014 13:04:35 -0000 Hi Jonathan I just saw your email. I must have missed it earlier. Sorry about that. The functions were just some example names. I forgot to put the sol_ = prefix in all of them. I understand your concerns about multi-threading. The idea is to have = functions that serialize the object in an allocated buffer as it is = constructed. Here is a more detailed example of what I mean: char *ucldump_object_start(const struct ucldump *restrict ctx, char = *buf, size_t buf_len) { if (ctx->format =3D=3D UCL_EMIT_JSON_COMPACT) { if (buf_len < 1) return 0; buf[0] =3D '{'; return buf + 1; } // etc. } char *ucldump_object_key(const struct ucldump *restrict ctx, char = *restrict buf, size_t buf_len, const char *restrict key, size_t key_len) { if (ctx->format =3D=3D UCL_EMIT_JSON_COMPACT) { // Make sure there is enough space for // "key": if (buf_len < (1 + key_len + 1 + 1)) return 0; *buf++ =3D '"'; *buf++ =3D '"'; memcpy(buf, key, key_len); buf +=3D key_len; *buf++ =3D ':'; return buf; } // etc. } This is still just an example. I don't know what will be the most = appropriate way to implement this. The idea is that for JSON and YAML sol_object_start() can call = ucldump_object_start(), sol_object_key can call ucldump_object_key() and = so on. This way UCL can take care of the exact output format while SOL = provides a uniform API for all formats (including other formats like XML = that can use a different backend). Since the serialized data is written to an allocated buffer, such = functions will not be an issue for multi-threading. Multi-threaded = applications just need to use separate buffers and separate = files/sockets/pipes/whatever (which they must do anyway). Zaro Le 28 May 2014 =E0 15:16, Jonathan Anderson a =E9crit : > On 28 May 2014, at 9:36, Zaro Korchev wrote: >> The idea is to serialize the object as it is constructed. That way it = is not necessary to keep all the data in memory at any given moment. = This also allows to create pipeline when the second application starts = consuming output before the first one finished producing it. >>=20 >> for example: >> object_start(buf, ...) writes to buf: >> { >> object_key(buf, ..., "foo") writes to buf: >> "foo" >> sol_object_integer(buf, ..., 42) writes to buf: >> : 42 >> sol_object_end(buf, ...) writes to buf: >> } >=20 > I notice that you have a mix of "object_" and "sol_object_" calls: are = these all purely serialization functions (and should they all be = prefixed with "sol_")? It would be unfortunate to commit to a stateful = object *creation* API, as that would make things difficult for = multithreaded work (e.g. each thread creates an object, each of which is = added to the top-level object as it is completed rather than in lexical = order). >=20 >=20 > Jon > -- > Jonathan Anderson >=20 > jonathan@FreeBSD.org > http://freebsd.org/~jonathan/ >=20