[Beowulf] Automatically replication of directories among nodes

[Ellis H. Wilson III]

On 11/25/20 12:28 PM, Lux, Jim (US 7140) via Beowulf wrote:
> It’s kind of a cluster, but not exactly HPC.
> 
> What I have is 3 rPi computers, A,B, and C, and what I’d like to do is 
> keep the desktop and some data directories on all of them synchronized.  
> So if on node A, I add something to A:~/Desktop, it (in short order) 
> winds up in the ~/Desktop directory on the other 2 machines.
> 
> It’s easy if I’m doing it from another computer – pdsh and similar do a 
> great job turning a single command into 3 (or N).
> 
> But what if it’s on the node itself.  I thought about something like 
> rsync running every second or 10 seconds, or whatever.
> 
> But maybe there’s a clever-er way. The network connection isn’t perfect, 
> so a “map it to a shared network drive” approach doesn’t work, and 
> there’s no guarantee that the state is the same on all machines (i.e. 
> one might drop off and reset, and be way behind the other two).  And, 
> the changes might come from any source (i.e. I can’t run all file 
> changes through some single entry point that does a pdsh like “write 3 
> times”)

You're getting at the meat of what network-attached storage has to solve 
all the time -- client consistency and coherency.  What you're 
requirements are for that drives what network-attached filesystem you 
use.  In the case of 3 raspberry pi computers, solving the network 
problem and just using NFS is by far the simplest and cheapest solution. 
  One of the rpis can even be your NFS server if you like, and a 1GbE 
switch is in the low double digits US currency.

Nevertheless, if you really insist on eventually consistent behavior, 
know that you're entering into the realm of the unknown.  What happens 
if rpiA is offline, writes to a file, and then rpiB reads from that file 
some time later?  What's expected at that time by that application? 
What if whole directories are deleted?  Or recreated with the same name? 
  How do hardlinks function when rpis disagree over what they should be 
linked to?  The list goes on and on.

If your datasets are truly discrete then don't bother replicating to the 
other rpis, just replicate to discrete folders on some shared NFS when 
networking is available again using one-way rsync.  This way there's no 
coherency/consistency issues.

The only last thing I'll say if you really want some kind of clumsy 
solution to this is to look at ownCloud and available offline 
directories.  You will still have consistency/coherency problems, but at 
least there will be some other framework managing them for you and a 
centralized store that manages source of truth for your data.

In industrial control scenarios changes are made at some central 
location and then distributed throughout the system simultaneously. 
Even IOT works this way -- the other nodes don't replicate data sent by 
the others even though the data might flow through them.  The data and 
controls are really solely sent from and stored at some central device 
(such as the openHAB rpi in my house to my various IOT devices).

Best,

ellis