[Beowulf] File sync across rPis

[Lux, Jim (US 7140)]

Based on the suggestions, first, I tried rsync – it sort of works, but it’s a lot of work to do the integration with cron, etc.; and there’s some subtle stuff with scheduling that I ran into (you really need to have some randomization in the times when it runs, or you wind up running two rsyncs at the same time).

I tried syncthing, also based on suggestions – it works great. Trivial install, so far (”apt-get install syncthing”) .  I was hoping I’d be able to stay out of GUI-land (i.e. I’d rather do configuration changes with ssh than vnc, and, when replicating it to new machines, it’s more controllable if it’s a script, and I can use pdsh, for instance), but trying to figure out the somewhat complex xml config files – GUI did make it easy, so there we are. Worst comes to worst, I’ll fire up the GUI on each one in sequence.

But so far, it seems work fine. I was concerned about resources, but for the small number of files I’m working with (dozens, not thousands) the time required to do the hashing, etc. is tiny (as in <0.1% CPU)

There is a tricky aspect to syncthing – having to do with the timestamps on the files, and whether your computer comes up with the “right time” (RPi do not have a battery backed up real time clock, so it has to “get time” from NTP, and I’ve not got my in-house NTP server running, yet)

I seriously looked at glusterfs – which looks very cool, and is sort of the next level of seamlessness – But it seemed a bit heavyweight for this application.  I may still look into it, though – I’ve got some ideas which might benefit from that almost “multiported, redundant storage” view.



Jim Lux
SunRISE Project Manager
Jet Propulsion Lab
+1(818)354-2075 (office)
+1(818)395-2714 (cell)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20201127/8963e3aa/attachment.htm>