Hi, I'm Nick from Stack Overflow! Hope I can clarify some bits here:
While we tried this, we didn't end up going with the 4 GPU in 1U config. Luckily we were able to test various configurations in Dell's test lab. The way you fit 4 in a 1U server is in a Dell C4130 chassis. The 4 cards are not consumer cards, but the professional Tesla variants. They're higher power, more memory, and more importantly: come in both airflow directions. The 4 cards are installed across the front bezel, and the airflow is reverse of what you'd buy in any consumer card.
Here's my tweet stream from first researching it, which includes pictures. Here's a clear view:
Note that every other card is upside down, but those are 300W cards. It takes dual 1600W PSUs to power the rig.
What we actually ended up going with (we're finally ordering hardware this week for many things) is a Dell R730 chassis. Since we can put cards in the back of it, instead of doing Tesla cards which are about $5,000 a piece in a server that starts far higher (IIRC $20K with a single card was our starting point), we can use consumer cards instead. In a R730 (a 2U server), one PCIe config allows 2 full-height cards in the back. We can use consumer GTX 1080s, which are $700 a piece and about 60-65% as fast for our use case. Marc Gravell has detailed test data we can share, we tested each scenario with singles and multiples of K40s, K80s, M40s, GTX 980s, and GTX 1080s.
In a R730, we can't order it with the higher than 140W TDP processors and the GPU config we want because of how Dell limits thermal configuration. But they don't take into account a GTX 1080 has a 180W TDP and not a 300W TDP of the Tesla models they sell (remember, times 2, so 240W less cooling needed). Given this, we couldn't (by default) directly order the Intel E5-2687W v4 processors we want (which are 160W TDP) for the workloads on these boxes. Luckily, we can work around this by just ordering the server and GPU kit separate and we'll install it ourselves.
The first server should be in place in the February/March timeframe if all goes well, and we'll of course share the adventure. Happy to answer more questions as they arise.
Here are some extra informational links: