From owner-freebsd-current@freebsd.org Fri Jan 22 17:08:56 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE081A8D12A; Fri, 22 Jan 2016 17:08:56 +0000 (UTC) (envelope-from anton.rang@isilon.com) Received: from mailuogwdur.emc.com (mailuogwdur.emc.com [128.221.224.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailuogwprd51.lss.emc.com", Issuer "RSA Corporate Server CA v2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B6091539; Fri, 22 Jan 2016 17:08:56 +0000 (UTC) (envelope-from anton.rang@isilon.com) Received: from maildlpprd52.lss.emc.com (maildlpprd52.lss.emc.com [10.106.48.156]) by mailuogwprd51.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id u0MH8jPQ023465 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 22 Jan 2016 12:08:46 -0500 X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd51.lss.emc.com u0MH8jPQ023465 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=isilon.com; s=jan2013; t=1453482526; bh=oARnOmVavDJPlXFZZ37Qc5XncHU=; h=From:To:CC:Subject:Date:Message-ID:References:In-Reply-To: Content-Type:Content-Transfer-Encoding:MIME-Version; b=figiBiZKDexuWu9jSgkwCmt7fg0ngzECxuBX+R0w45XG6/SbeQsNtjEkUA1NdS0dZ bdZjydq591w6Xn+M2rIWslCUFXAsiCLmWWo0qfXnRmwyUDCFsRbSEby+lLDQzm9DKI +r12KcZx93cz+pDCnRccFFI9I8v+UPrtPOi/BN9A= X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd51.lss.emc.com u0MH8jPQ023465 Received: from mailusrhubprd04.lss.emc.com (mailusrhubprd04.lss.emc.com [10.253.24.22]) by maildlpprd52.lss.emc.com (RSA Interceptor); Fri, 22 Jan 2016 12:08:25 -0500 Received: from MXHUB217.corp.emc.com (MXHUB217.corp.emc.com [10.253.68.87]) by mailusrhubprd04.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id u0MH8XOn020707 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=FAIL); Fri, 22 Jan 2016 12:08:33 -0500 Received: from MX104CL01.corp.emc.com ([169.254.7.159]) by MXHUB217.corp.emc.com ([10.253.68.87]) with mapi id 14.03.0266.001; Fri, 22 Jan 2016 12:08:32 -0500 From: "Rang, Anton" To: Mathieu Prevot , NGie Cooper CC: Jan Bramkamp , "freebsd-hackers@freebsd.org" , "freebsd-current@freebsd.org" Subject: RE: IoT OS Thread-Topic: IoT OS Thread-Index: AQHRVGer4Z6ZjBEe40iiSuO3vjQwlJ8GfjEAgAABBYCAABhHgIABKbng Date: Fri, 22 Jan 2016 17:08:32 +0000 Message-ID: References: <56A10892.2090308@rlwinm.de> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.13.55.14] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Sentrion-Hostname: mailusrhubprd04.lss.emc.com X-RSA-Classifications: public X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 17:08:56 -0000 >Say all objects are connected peer to peer with wifi, some of them are con= nected to internet through gsm network or wifi to a box. >These object are moving in space, and for some reasons, connections are dy= namical and can be severely impaired or lost. > >They have incoming local streams of data (eg HD videos, accelerometer, GPS= , other wifi and gsm signals, etc). > >I would like to abstract the CPU layer, storage layer, and internet connec= tion so that in realtime results of one of my objects are saved >if this object dies, so that if one of the object giving internet access t= o the group loose its connection, the redundancy allows the group of object= not to lose internet connection. > >Can I consider these as different load balancing layers ? Do you recommend= to implement this at the kernel layer or at an API layer ? >Can I see that as a lightweight cluster ? > >I think the API is more flexible, especially if I have an heterogeneous (b= y CPU, OS) set of connected object. However, working at the kernel level al= lows existing programs not to be rewritten. >What are your thoughts ? =3D=3D=3D OK, I think I understand your question now. This isn't the right list for it, though I'm not sure where the right place= to go would be -- it's not FreeBSD-specific, in any case. There are academ= ic research groups looking into this type of problem; for instance, in the = area of sensor networks (ACM Transactions on Sensor Networks covers some of= these areas). There may be USENET groups which cover this area. To cover your three areas, which I think require somewhat different solutio= ns -- (a) CPU layer. I don't really recommend trying to abstract this. You coul= d use a virtual machine to hide the underlying architecture, and checkpoint= state periodically, but this is likely to slow down execution too much to = be useful. If the issue that a service may become unavailable, I'd recomme= nd a middleware layer which can detect this and recover by starting a new i= nstance of the service. Middleware layers like ZeroMQ, and clustering softw= are, may be a useful starting point. This does mean that stateful connecti= ons (like reading a video stream) won't recover cleanly, though; the client= would need to reconnect to attach to the new instance of the service. If = you really need that, it's going to be hard. (b) Storage layer. Look into highly-available clustered storage solutions.= If you can use key-value or some other simplified storage model, do it. = There are clustered file systems but probably none freely available that wo= uld work on the scale you envision and give decent performance. There are = more alternatives if you're flexible about the format in which you're stori= ng data (e.g. replicated object stores). (c) Networking layer; or internet. If you can drop & re-establish a connect= ion, or if every node has its own IP address (IPv6), this should be pretty = straightforward; software could detect loss of connection and change the ro= uting used to go through a different system. If not, you'll be a bit limite= d since mirroring TCP state between nodes would be too slow. This is a case= where the existing operating system kernels are likely to do most of what = you need; you simply need to add a layer to detect routing problems and sel= ect a new internet gateway appropriately. I'd avoid implementing any clustering within the kernel, in part because if= you have a wide variety of objects you may not want the same kernel on all= of them, and in part because debugging & recovery is much harder. You're u= nlikely to want to run most existing software on such a system anyway (espe= cially if they have relatively weak processors); you're better off writing = to a set of clustering APIs for storage and state, at least. For networking= , as mentioned, you can likely use the existing TCP stack & just add contro= ls to redirect traffic as needed. -- Anton