<div dir="ltr">Igor, if there are any papers published on what you are doing with these images I would be very interested.<div>I went to the new London HPC and AI Meetup on Thursday, one talk was by Odin Vision which was excellent.</div><div>Recommend the new Meetup to anyone in the area. Next meeting 21st August.</div><div><br></div><div>And a plug to Verne Global - they provided free Icelandic beer.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 29 Jun 2019 at 05:43, INKozin via Beowulf <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Converting the files to TF records or similar would be one obvious approach if you are concerned about meta data. But then I d understand why some people would not want that (size, augmentation process). I assume you are are doing the training in a distributed fashion using MPI via Horovod or similar and it might be tempting to do file partitioning across the nodes. However doing so introduces a bias into minibatches (and custom preprocessing). If you partition carefully by mapping classes to nodes it may work but I also understand why some wouldn't be totally happy with that. Ive trained keras/TF/horovod models on imagenet using up to 6 nodes each with four p100/v100 and it worked reasonably well. As the training still took a few days copying to local NVMe disks was a good option.<div dir="auto">Hth</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 28 Jun 2019, 18:47 Mark Hahn, <<a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br>
I wonder if anyone has comments on ways to avoid metadata bottlenecks<br>
for certain kinds of small-io-intensive jobs. For instance, ML on imagenet,<br>
which seems to be a massive collection of trivial-sized files.<br>
<br>
A good answer is "beef up your MD server, since it helps everyone".<br>
That's a bit naive, though (no money-trees here.)<br>
<br>
How about things like putting the dataset into squashfs or some other <br>
image that can be loop-mounted on demand? sqlite? perhaps even a format<br>
that can simply be mmaped as a whole?<br>
<br>
personally, I tend to dislike the approach of having a job stage tons of<br>
stuff onto node storage (when it exists) simply because that guarantees a<br>
waste of cpu/gpu/memory resources for however long the stagein takes...<br>
<br>
thanks, mark hahn.<br>
-- <br>
operator may differ from spokesperson. <a href="mailto:hahn@mcmaster.ca" rel="noreferrer" target="_blank">hahn@mcmaster.ca</a><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" rel="noreferrer" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote></div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote></div>