>> If I had just 2 fat nodes could I connect the HCAs directly to each other >> and avoid the switch costs? > > Yes. With 3 nodes it might be cheaper having 2 HCA per node, 6 cables, > than the switch solution. with 3 nodes, each with a two ports, wouldn't you need just 3 cables? how is routing controlled in switchless configs? does IB have node-level forwarding?