[Beowulf] Pricing and Trading Networks: Down is Up, Left is Right

[Vincent Diepeveen]

On Feb 17, 2012, at 7:05 PM, Lux, Jim (337C) wrote:

>
>
> On 2/17/12 7:42 AM, "Vincent Diepeveen" <diep at xs4all.nl> wrote:
>
>>
>> On Feb 17, 2012, at 3:12 PM, Lux, Jim (337C) wrote:
>>
>>
>>> If so, I'd go for that.. The population of FPGA developers is  
>>> probably
>>> 1/100 the number of conventional Von Neuman machine developers (in
>>> whatever language).
>>>
>>> Interestingly, such a scarcity does not translate to 100x higher  
>>> pay.
>>
>> I'm not a  magnificent FPGA developer - i'd say first speedup the
>> software - there is so many traders
>> real slow there. But they don't even are interested IN THAT.
>>
>>> Most of the surveys show that in terms of median compensation FPGA
>>> designers get maybe 30-40% more than software developers.
>>
>> The rate for development is $1 an hour in india. If you deal with
>> major companies in India it's $2.50 an hour
>> including everything.
>
>
> I think you're a bit low there.  People I know who are contracting
> off-shore development say that the net cost (to the US firm) is  
> about 1/4
> and 1/3 what the equivalent person would cost in the US.  (and the  
> price
> is rising)

Well they are wrong then. Usually they also count in the costs at the  
location which is a staff
at the spot, so not in India.

The real costs in India you can get it for are $2.50 an hour.
This is *big* consultancy companies.

The consultants who're independant are usually rates of around $1 -  
$2 an hour.

Philippines sits around $1.11 an hour for consultants.

If i look to actual produced work in India you see they're very good  
in trying to get more work,
like additional pay for extra features - so most projects budgeted at  
X hours usually end up in 2X;
of course you and i know that doesn't mean it took 2X :)

The actual developers working for such companies complain loud  
usually about their salary.
Yet also for them getting work as a consultant is very difficult.

They sit around $150 a month.

This is *normal* development.

As you know i'm more involved in mass market products, usually that's  
the better developers on this planet; nothing as complicated
as producing a good mass market product as it has to work everywhere.

In India rates for that are open market and also go up rapidly. I've  
even heard of some who got a $1000 a month there in India.

But this is really 1 in a 100 developers.

>
> You can hire very low level people quite inexpensively on a bare  
> contract,
> but you spend more managing them, and compensating for the incredible
> defect density.

It was indeed the habit to have the managers over here. However times  
have changed.

Nowadays management also sits in Asia.

>
> And I doubt that good FPGA folks are as thick on the ground in low  
> cost
> places as they are in the US or Europe.

We were discussing normal development and i noted that normal  
development happens in India
for $1 an hour for independants and $2.50 including management  
overhead and everything an hour,
by the major consultancy companies.

If i see how much power Bulldozer CPU eat, then i can assure you that  
India is about the last spot on the planet where i'd
have develop a FPGA for trading :)

To start with you're not gonna ship a development board to India -  
it's gonna disappear without anyone knowing and without
anyone who can be blamed.

I remember how i shipped some stuff to India.

0% arrived.

Tracking codes - forget it - that's just a number from the west -  
something utmost useless in India.

>
>
>>>>
>>>> Getting a FPGA a tad higher clocked from Xilinx, say the first  
>>>> sample
>>>> at 22 nm, is probably not cheap either.
>>>
>>> The biggest, baddest Virtex 7 (XC7V2000TL2FLG1925E) is a mere
>>> $132k, qty 1
>>> (drops to $127k in qty 1000)
>>> (16 week lead time)
>>>
>>> 2 million logic cells 70 Mb onchip block ram, 3600 dsp slices,  
>>> 28Gb/s
>>> transceivers, etc.etc.etc.
>>>
>>
>> how high that would clock?
> It's a bit tricky when talking clock rates in FPGAs.. Most designs are
> only regionally synchronous, so you need to take into account  
> propagation
> delays across the chip if you want max performance.
>
> You feed a low speed clock (few hundred MHz) into the chip and it gets
> multiplied up in onchip DPLLs/DCMs.
> The MMCM takes 1066 Mhz max input.

I'm guessing around a 2.5Ghz.

We know they expected some boards to be able to clock around 1.7Ghz.

>
> There's also different clocks for "entirely on chip" and "going off  
> chip",
> particularly for things like the GigE or PCI-X interfaces (which have
> their own clocks)  Likewise these things have built in interfaces  
> to RAM
> (so you can get like 1900 Mb/s to DDR ram, running the core at 2V).
>
> It's more like doing logic designs.. Propagation delays from D to O
> through combinatorial logic look like they're in the 0.09 ns  
> range.  CLB
> flipflops have a setup time of 0.04ns and hold time of 0.13 ns, so  
> that
> kind of looks like you could toggle at around 2 Ghz
>
> There's plenty of documentation out there, but it's not as simple  
> as "I'm
> running the CPU at 3 GHz"
>

I've not seen any actual designs - they are utmost top secret - no  
military secret is as secret as what they use
in the datacenters in own designs - besides that the biggest military  
secrets, like when a war is gonna happen,
you roughly can predict quite well if you watch the news.

Forget DDR ram, DDR ram really is too slow in latency also forget  
about pci-x.
I'd say also forget about pci-e - think of custom mainboards.

On the FPGA card there's gonna be massive SRAM besides a solarflare NIC.

You need quite a lot actually as the entire datafeed has to get  
streamed to the rest of the mainboard
for storage and more complicated trading analysis i'd suppose.

I would guess the chip can do the simple trading decisions in a  
pretty simple manner.
This keeps the chipdesign relative simple and you can focus upon  
clocking it high.

The rest you want to do in software. Losing the pci-e latencies then  
is crucial as that's a big bottleneck.

Also forget normal RAM. No nothing DDR3. Just SRAM and many dozens of  
gigabytes of it as we want to keep the entire day preferably in RAM
and our analysis are nonstop hammering onto the SRAM so basically the  
bandwidth to the SRAM determines how fast our analysis will be.

The simple trading decisions already get done by the FPGA of course.

But well i guess they probably tried to get ALL trading decisions  
inside the fpga - who knows.

As for the mainboards, to quote someone : "there are some very custom  
designs out there".

Yet of course you can do real well in software already for a fraction  
of that budget.


>
>>
>> note you also need to integrate the fastest 10 gigabit nic - right
>> now that's solarflare.
>
>
> Do you? Why not use some other interconnect.

Because the datacenter of the exchange dictates to you what sort of  
protocol they use. Several are now exclusively solarflare.
FTSE has gone to infiniband - but i'm not sure whether that's their  
internal machines only.

FTSE is only limited interesting of course - Chicago is more  
interesting :)

Realize also by the way that you have many connections. Really a lot.  
Market A comes from IP adres X, Market B from IP adres Y and so on,
we speak of hundreds of IP adresses you have to connect to  
simultaneously.

You get those adresses in a dynamic manner, say in a XML type manner.

It's a total mess they created and they love create a mess as that  
means more work and work means you can make money from rich traders.

They really overcomplicated it all.

Also realize at any moment they can change the messages - the way how  
the messages look like you get in a dynamic manner in XML.
There is nothing really hard defined.

That's why all that generic software is trading at such slow speeds!
No nothing hardcoded.

Basically the whole protocol is NOT designed for speed.

Calling it a spaghettidesign would be a compliment. The financials  
LOVE to just add to the mess and never 'fix' things.

See it as adding components to the space shuttle and then pray it  
still works. Actually to get back to the factual criticism in that  
article -
the space shuttle was intended to work correctly - the exchanges do  
not give the garantuee that you get the information at all :)

Only the trading is TCP, the crucial stream what bids and offers  
there are, they are in RAW format and just 1 channel is gonna be fast
enough for you to try follow it. If that channel doesn't have the  
correct info - bad luck for you.

That bad luck of course happens during big surges - we all realize that.

This lossy way of information is the underlying method based upon  
which decisions get taken which have major implications :)

There is actually from the FIX/FAST community meetings on the  
protocol at a regular interval. To just show your company name
you have to pay a $25k. To also display it at the conference is a big  
multiple of that.

The next one is scheduled for London at the 13th of March 2012:  
www.fixprotocol.org


>>
>>>
>>> To put it in a box with power supply and interfaces probably would
>>> set you
>>> back a good chunk of a million dollars.
>>>
>>
>> if you'd do it in a simple manner sure - but if you're already on
>> that path of speed and your main income is based upon speed you
>> want something faster than what's online there of course.
>>
>> You phone xilinx and intel and demand they print fpga's before
>> printing cpu's in a new proces technology, and clock 'em as high as
>> possible.
>
>
> I seriously doubt that even a huge customer is going to change  
> Xilinx's
> plans.  They basically push the technology to what they can do  
> constrained
> by manufacturability.
>
> Then you have all the thermal issues to worry about.  These big  
> parts can
> dissipate more heat than you can get out through the package/pins,  
> or even
> spread across the die.
>

It has crossed my mind for just a second that if 1 government would  
put 1 team together and fund it,
and have them produce a magnificent trading solution,
which for example plugs in flawless into the IBM websphere,
and have their own traders buy this, especially small ones with just  
ties to their own nation, that in theory things still are total legal.

A team, of course so called under a company name, produces a product,  
and traders buy this in from this company.

The only thing is they don't tell around they do business with this  
company - only their accountant sees that name - things stil  
perfectly legal.

Now some of those traders will make a profit and others will lose  
some; statistically they wil however perform a lot better than they  
used to do.

That means that a big flow of cash moves towards 1 nation.

Of course you can also do this just in software - all you have to  
beat is IBM, which is freaking peanuts for a good programmer.

The only important thing is that they're with that fastest 10%. No  
need to have the fastest solution there - let some hedgefunds
design that (they already did).

It's just a throught experiment, but it sure would basically win a  
lot of money from abroad to your own nation; in the end a big part of  
that
then flows into your own nation and pays for things.

Do not think in small numbers here - they trade so much money daily,  
just a small statistical advantage as your software is faster
than it would otherwise have been, is having a huge impact.

>
>> guess it's around a 4 million a year minimum.
>>
>> That's not a problem for the hedgefunds to pay actually.
>
> I suspect that money isn't the scarce resource, people are.  There  
> aren't
> many people in the world who can effectively do this kind of thing.  
> (For
> the kind of thing you're talking about, it's probably in the 100s,  
> maybe
> 1000s, total, worldwide)

Well i'm sure there aren't many, but i'm very sure they aren't trying  
to get the best programmers - they mainly hire JAVA coders everywhere.

>
>>
>> For me it's incredible that there is so little jobs actually for
>> trading in imperative languages and basically 99.9% of all jobs are
>> Java there.
>
>
> Why is this amazing?

Suppose you only hire people who are lefthanded. That's basically  
what they're doing.

Because speed is everything at the exchanges now and you aren't gonna  
get the best people this way,
as they shut out the biggest experts basically, with a few exceptions.

> The vast majority of money spent on software is
> spent on run of the mill, mundane chores like payroll accounting,
> inventory control, processing consumer transactions, etc.
>
> So there's a huge population of people to draw from.
>

> If you're looking for top people, and you need some number of them,  
> you're
> better off taking the 3 or 4 sigma from the mean people from a huge
> population, than taking the 1 sigma people from a small population.
>

They just draw out of a small population of usually very bad  
programmers who studied finance.

How do you get the best software engineers then for your trading  
application?

>
>
>>
>> I just simply would never do business with jpmorgan chase. they seem
>> to hire 0 persons who are busy imperative or even in C++.
>
>
> So what?  That's just a personal preference on your part.  How do  
> you know
> they aren't hiring those people through some other channel?

And what would that 'other channel' be then?

Vaste majority simply isn't doing this.

>
> You don't see a lot of ads out there for FORTRAN programmers, but  
> here at
> JPL, about 25% of the software work that's being done is in  
> FORTRAN.  And

No matter how genius you are, if all you do is write fortran, and  
didn't study finance,
then you are not allowed to write a trading application, AS YOU WON'T  
GET HIRED :)

> of the people doing software at JPL, more than half do NOT have  
> degrees in
> CS or even EE, and I'd venture that they were not hired as "software
> developers":  more likely they were hired for their domain specific
> knowledge.
>
>> Basically means that the entire market of genius guys who know how to
>> beat you in game tree search, which is about all artificial
>> intelligence experts,
>> they're shut out from majority of companies to get a job there.
>
> Definitely not.  If you're in that tippy top 0.1%, you're not  
> getting jobs
> by throwing your resume over the transom.  You're getting a job  
> because
> you know someone or someone came to know of you through other means.
>

You seem to be the expert in hiring people :)

> I didn't get my job at JPL by submitting a resume, and I think  
> that's true
> of the vast majority of people here. It was also true of my last job,
> doing special effects work.  And in fact, now that I think back, I  
> think I
> have had only one job which was a resume in response to an ad.
>

When you got there, there was a circle at your resume around the word  
'NASA'

And nothing else mattered i bet :)

>
>
>>
>> The way to get a job is to be young and have a degree in economics.
>
> Depends on what job you want.
>
> A good fraction of the technical degree grads at MIT are being  
> hired by
> the finance industry. This concerns people like us at JPL, because we
> can't offer competitive pay and benefits, and on a personal note, I  
> think
> it's a shame that they're probably not going to be using their  
> skills in
> the field in which they were actually trained.
>
>
> But the way to get a good job has always been, and will always be,  
> to know
> someone.
>
> There have been *numerous* well controlled studies that looked at  
> hiring
> behavior, and regardless of what the recruiting people say, in reality
> managers make their decisions on the same few factors, and  
> "recommendation
> of coworker or industry colleague" is right up there in the top few.
>
>
>>
>
>