[Beowulf] Paper describing Google's queuing system "Borg"

[Adam DeConinck]

> Is Omega the successor?

John Wilkes from Google gave a presentation on Omega at Lisa '13 (video @
https://www.usenix.org/cluster-management-google).  If I recall correctly,
he says in the talk that Omega was developed as a potential successor to
their current cluster manager, but that they were still trialing the system
and hadn't decided yet whether to switch. Or something to that effect -- I
don't have time this morning to re-watch the video.

I haven't seen any publications which definitively say whether Google has
chosen to move to Omega yet, but then, there wouldn't be. ;-)

The Borg paper also says:

> This is quite similar in spirit to the optimistic concurrency control
used in Omega [69], and indeed we recently added the ability for Borg to
use different schedulers for different workload types. (pg 5, section 3.4)

My understanding of Omega is that its major new feature is the ability to
provide optimistic resource allocations to several different schedulers. So
I wonder if Google has simply decided to backport the new, desired features
of Omega back to Borg.


On Tue, Apr 21, 2015 at 7:51 AM, Scott Atchley <e.scott.atchley at gmail.com>
wrote:

> Is Omega the successor? The Borg paper mentions Omega :
>
>   Omega [69] supports multiple parallel, specialized “verti- cals” that
> are each roughly equivalent to a Borgmaster minus its persistent store and
> link shards. Omega schedulers use optimistic concurrency control to
> manipulate a shared repre- sentation of desired and observed cell state
> stored in a cen- tral persistent store, which is synced to/from the
> Borglets by a separate link component. The Omega architecture was de-
> signed to support multiple distinct workloads that have their
>
> own application-specific RPC interface, state machines, and scheduling
> policies (e.g., long-running servers, batch jobs from various frameworks,
> infrastructure services like clus- ter storage systems, virtual machines
> from the Google Cloud Platform). On the other hand, Borg offers a “one size
> fits all” RPC interface, state machine semantics, and scheduler pol- icy,
> which have grown in size and complexity over time as a result of needing to
> support many disparate workloads, and scalability has not yet been a
> problem (§3.4).
>
>
>
> On Thu, Apr 16, 2015 at 12:20 PM, Deepak Singh <mndoci at gmail.com> wrote:
>
>> Great to see something about the Borg design out there.  Google have also
>> written about the successor to Borg, a framework called Omega.
>>
>> http://research.google.com/pubs/pub41684.html
>>
>> PDF: http://research.google.com/pubs/archive/41684.pdf
>>
>>
>> On Thu, Apr 16, 2015 at 6:53 AM Chris Samuel <samuel at unimelb.edu.au>
>> wrote:
>>
>>> Hi all,
>>>
>>> This is a very recent (2015) paper describing the queuing system used by
>>> Google internally, called "Borg".
>>>
>>> http://research.google.com/pubs/pub43438.html
>>>
>>> Full PDF available from there.
>>>
>>> Thought it might interest some folks!
>>>
>>> All the best,
>>> Chris
>>> --
>>>  Christopher Samuel        Senior Systems Administrator
>>>  VLSCI - Victorian Life Sciences Computation Initiative
>>>  Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
>>>  http://www.vlsci.org.au/      http://twitter.com/vlsci
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20150421/889814c2/attachment.html>