<div dir="ltr">Hi Michael,<div>Your speed suspiciously looks like the maximum HDD speed. How many stripes is the directory set to on the source and destinations? This is a common problem for my researchers as they don't understand lustre. They expect to have single stream bandwidth on multi GB/s and get about the speed of a single disk. </div><div><br></div><div>The other common cause for this speed is hitting the CPU maximum as most of transfer parts are single threaded. Have you looked at the cpu performance with htop? It will show systime as well so that can help isolate whether it is the kernel slowing the process down.</div><div> <br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Cheers,<br><br>Lance<br>--<br>Dr Lance Wilson<br>Characterisation Virtual Laboratory (CVL) Coordinator &</div><div dir="ltr">Senior HPC Consultant</div><div>Ph: 03 99055942 (+61 3 99055942)</div><div dir="ltr">Mobile: 0437414123 (+61 4 3741 4123)</div><div dir="ltr">Multi-modal Australian ScienceS Imaging and Visualisation Environment<br>(<a href="http://www.massive.org.au/" rel="noreferrer" style="color:rgb(17,85,204)" target="_blank">www.massive.org.au</a>)<br>Monash University<br></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 3 Jan 2020 at 05:12, Jonathan Aquilina <<a href="mailto:jaquilina@eagleeyet.net">jaquilina@eagleeyet.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>
<div style="direction:ltr">Aren’t you going to have issues with I/O contention if you are copying on the same machine but different directories? Is the transfer even going over the link</div>
</div>
<div><br>
</div>
<div>
<div style="direction:ltr">Regards,</div>
<div style="direction:ltr">Jonathan Aquilina</div>
<div style="direction:ltr">Owner managing director</div>
<div><br>
</div>
<div style="direction:ltr">Phone (356) 20330099</div>
<div style="direction:ltr">Mobile (356) 79957942</div>
<div><br>
</div>
<div style="direction:ltr">Email <a href="mailto:sales@eagleeyet.net" target="_blank">sales@eagleeyet.net</a></div>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-7088170960012063552divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Beowulf <<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">beowulf-bounces@beowulf.org</a>> on behalf of Jonathan Engwall <<a href="mailto:engwalljonathanthereal@gmail.com" target="_blank">engwalljonathanthereal@gmail.com</a>><br>
<b>Sent:</b> Thursday, January 2, 2020 5:56:25 PM<br>
<b>To:</b> Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com" target="_blank">mdidomenico4@gmail.com</a>><br>
<b>Cc:</b> Beowulf Mailing List <<a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a>><br>
<b>Subject:</b> Re: [Beowulf] 10G and rsync</font>
<div> </div>
</div>
<div>
<div dir="auto">The whitepaper was interesting. Single core VMs might be your best bet.</div>
<br>
<div>
<div dir="ltr">On Thu, Jan 2, 2020, 8:48 AM Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com" target="_blank">mdidomenico4@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
i'll check it, but keep in mind. i'm not copying files between two<br>
servers, but rather between two directories on the same server.<br>
<br>
ideally if rsync is still using ssh under the covers in my scenario,<br>
i'm hopeful hpn-ssh might alleviate the bottleneck condition. if it's<br>
not i'm back to square one.<br>
<br>
On Thu, Jan 2, 2020 at 11:42 AM Alex Chekholko <<a href="mailto:alex@calicolabs.com" rel="noreferrer" target="_blank">alex@calicolabs.com</a>> wrote:<br>
><br>
> Hi Michael,<br>
><br>
> I would recommend trying 'bbcp' before 'hpn-ssh' as the latter will really only benefit you for high-latency links, e.g. across country.<br>
><br>
> Put the bbcp binary on both sides and try it out. If you don't have a way to install bbcp into a system $PATH, you can specify the absolute path to the binary. Random link with examples here:<br>
> <a href="https://www.nics.tennessee.edu/computing-resources/data-transfer/bbcp" rel="noreferrer noreferrer" target="_blank">
https://www.nics.tennessee.edu/computing-resources/data-transfer/bbcp</a><br>
><br>
> Regards,<br>
> Alex<br>
><br>
> On Thu, Jan 2, 2020 at 8:32 AM Michael Di Domenico <<a href="mailto:mdidomenico4@gmail.com" rel="noreferrer" target="_blank">mdidomenico4@gmail.com</a>> wrote:<br>
>><br>
>> just to further the discussion and for everyone's education i found<br>
>> this whitepaper, which seems to confirm what i see<br>
>><br>
>> <a href="https://www.intel.com/content/dam/support/us/en/documents/network/sb/fedexcasestudyfinal.pdf" rel="noreferrer noreferrer" target="_blank">
https://www.intel.com/content/dam/support/us/en/documents/network/sb/fedexcasestudyfinal.pdf</a><br>
>><br>
>> maybe hpn-ssh is something i can work into my process<br>
>><br>
>><br>
>> On Thu, Jan 2, 2020 at 10:26 AM Michael Di Domenico<br>
>> <<a href="mailto:mdidomenico4@gmail.com" rel="noreferrer" target="_blank">mdidomenico4@gmail.com</a>> wrote:<br>
>> ><br>
>> > does anyone know or has anyone gotten rsync to push wire speed<br>
>> > transfers of big files over 10G links? i'm trying to sync a directory<br>
>> > with several large files. the data is coming from local disk to a<br>
>> > lustre filesystem. i'm not using ssh in this case. i have 10G<br>
>> > ethernet between both machines. both end points have more then<br>
>> > enough spindles to handle 900MB/sec.<br>
>> ><br>
>> > i'm using 'rsync -rav --progress --stats -x --inplace<br>
>> > --compress-level=0 /dir1/ /dir2/' but each file (which is 100's of<br>
>> > GB's) is getting choked at 100MB/sec<br>
>> ><br>
>> > running iperf and dd between the client and the lustre hits 900MB/sec,<br>
>> > so i fully believe this is an rsync limitation.<br>
>> ><br>
>> > googling around hasn't lent any solid advice, most of the articles are<br>
>> > people that don't check the network first...<br>
>> ><br>
>> > with the prevalence of 10G these days, i'm surprised this hasn't come<br>
>> > up before, or my google-fu really stinks. which doesn't bode well<br>
>> > given its the first work day of 2020 :(<br>
>> _______________________________________________<br>
>> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" rel="noreferrer" target="_blank">
Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
>> To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer noreferrer" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" rel="noreferrer" target="_blank">
Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer noreferrer" target="_blank">
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a><br>
</blockquote></div>