8 GPU GeForce Titan Tyan System

May 24, 2013 - 20 comments

A box arrived a few days ago at work.

The box contained a little supercomputer comprised of 8 GeForce Titan GPUs in a Tyan FT77A platform.

The system fits nicely in our server room.

And the 8 GPUs light up too!

20 Comments

#1 efo wrote on July 3, 2013:

It would be great if you could post some rendering benchmarks on v-ray or anything that utilize multiple GPU’s. thanks.

#2 Rob wrote on July 4, 2013:

Hey, this machine looks awesome. Any stability issues when running all 8 GPUs at the same time?

Also, is there any room for additional PCIe cards when you have 8 GPUs installed? It look like the other PCIe slots are obstructed by the GPUs…..

#3 Sebastian Schaetz wrote on July 10, 2013:

Hi there,

I don’t have any performance measurements that I can share right now. Compared to an 8 GPU system with PCIe 2.0 and GTX 580 it is significantly faster. No quantum leap but faster.

If 8 GPUs are installed there is no room for an additional PCIe card. You can remove one or two cards and put your PCIe card (Infiniband?) in there.

Cheers,
Sebastian

#4 Nick wrote on August 20, 2013:

Hi, impressive setup! I’ve been looking for a GPU expander that uses Gen3 with 8 double sized gpu slots. Did you install the Titan’s yourself or did the company you purchased it form install them? Can you make a recommendation?
Thanks!
Nick

#5 Sebastian Schaetz wrote on August 28, 2013:

Hi Nick,

companies usually have to be convinced to sell Titans for such systems, they will give you numerous reasons why it is a bad idea. Thy will cite warranty, ECC, double precision etc. But the fact of the matter is, consumer grade cards are faster due to their less conservative clock settings. And they are cheaper.
I would not recommend them for 24/7 simulation calculations often done in HPC but for applications with real-time constraints they are quite useful.
Shoot me an e-mail if you want a recommendation about where to buy them (in Germany).

Cheers,
Sebastian

#6 Boris wrote on November 4, 2013:

Just curious, what OS are you using on it? I have the same chassis with 8x GTX780 cards, and I can only get 7 to work with Windows 7 or 8.1 – 8th card to come up always gives error 43.

#7 Sebastian Schaetz wrote on November 4, 2013:

Hi Boris,

I heared that before. On our system all 8 GPUs are working. We’re running Ubuntu 12.04, newest drivers, CUDA 5.5.

Cheers,
Sebastian

#8 ManC wrote on November 16, 2013:

Hi,
Awesome rig you have their dude! very envious.
Your email is’nt obviously available on your site/blog (intentional probably), would you mind emailing me where you purchased this unit from?

@Boris, If you ever come across a solution to your problem please share it, I can see myself running to this hurdle to.

#9 Sebastian Schaetz wrote on November 16, 2013:

ManC,

I sent you an e-mail. You can reach me at echelon.blog@gmail.com.

Cheers,
Sebastian

#10 zzz1000 wrote on November 27, 2013:

Hi Boris, did you solve your problems? I suggest you replace one of your GTX780s with a Titan or GTX780 Ti and see if it does the trick.
BTW, why not use Linux? Linux is able to handle more GPUs. Recently I have built a 16-GPU rig with NVIDIA cards, and it just works fine under Linux.

#11 Sebastian Schaetz wrote on November 27, 2013:

A 16 card rig sound intriguing. Can you go into some details about that system.
I agree about Linux. It is well supported and all the tools are readily available.

#12 zzz1000 wrote on November 28, 2013:

Hi Sebastian Schaetz,
Yes, and I think building a rig with such a large number of gpus is very challenging and interesting.
Note that my rig has only 11 cards, not 16. But five of them are dual-gpu cards, so the total number of gpus is exactly 16.
The motherboard I used is a Supermicro X9DRX+-F, which has 11 PCI-e x8 slots (but no any x16 slots). PCI-e extenders are required to connect the cards to the board properly.
The reason why only 5 dual-gpu cards used is just because I have no enough dual-gpu cards at hand. Two more dual-gpu cards were ordered today and I’m curious to see if 17 or 18 gpus on a single system will work.
To save money, I only used some cheap cards to build the rig. The 11 cards are composed of 6x GTX660 Ti and 5x GTX295, devicequery and nvidia-smi information for the rig can be found here:
https://devtalk.nvidia.com/default/topic/572651/issue-with-more-than-8-gpus/ (9#)

#13 Sebastian Schaetz wrote on November 28, 2013:

Hi zzz1000,

thanks for the detailed info about your setup. Sounds awesome! Looking forward to hearing more about your experiences with even more cards! Might I ask what you’re doing with your system? Our system is used for medical imaging – we reconstruct images with the 8 GPU rig in (almost) real-time.

Best,
Sebastian

#14 zzz1000 wrote on November 29, 2013:

Hi, my server is used for CUDA software development, but I put so many GPU cards in it is mainly just for fun. I’m curious if there is really a limit for 16 GPUs.
And now, the answer is out there. My rig has been upgraded to 18 GPUs and it just works fine!
For more detailed info, see:
https://devtalk.nvidia.com/default/topic/649542/cuda-setup-and-installation/18-gpus-in-a-single-rig-and-it-works/

#15 Sebastian Schaetz wrote on December 2, 2013:

Impressive zzz1000! To keep in the spirit of this post, can you share some pictures maybe?

#16 Williams wrote on February 1, 2014:

Great blog/post. Would you guys tell me something about power consumption and stability of such a system? Could it be used on a cluster, up 24/7 . How long the cards survive?
We have a cluster. The vendor installed some GPUs and it usually presented problems. We had to call the vendor several times to fix the installation. So I guess that a proper/robust installation of GPUs is not that simple.

thanks

#17 Sebastian Schaetz wrote on February 1, 2014:

Hi Williams,
the power consumption of this system is not higher than 2kW if all 8 GPUs are running. We had some stability problems in the beginning but they seem to be mostly gone at the moment. Like you, we had some intensive interaction with the vendor for replacements, software updates (Nvidia driver + BIOS) and configuration settings. All in all not a nice experience. But in my experience, this stabilizes after some time and the systems then run reliably for about 2 years or more.

I can however not attest to how stable they are in a 24/7 setting. Our systems run 24/7 but computation on GPUs is only done in small bursts throughout the day. There are however some computing centers that run multi-GPU systems (not 8 but 2 or 3 per node) with consumer grade cards quite successfully. There are probably more problems than with professional cards but they are also a lot more expensive.

What kind of system are you using? And what for?
Cheers,
Sebastian

#18 Robin Lam wrote on November 18, 2015:

Hi Boris,

I am having a very similar issue with 8 x GTX 980 Tis. Did you ever get it to work with 8 cards in Windows 7 x64?

It would be very interesting to know if you got it working and how?

Many thanks

Robin.

PS: I also sent a PM to Sebastian.

#19 Roman wrote on November 17, 2016:

Hi,

I understand it can be late to ask, however, I would try :-)

Could you please explain how you deal with the embedded graphics? Do you disable it or, in contrary, you keep it enabled and you can use it as a normal card display can be connected to, thus leaving 8 GPUs solely for calculations?

Thank you.

#20 Stephen Balaban wrote on December 12, 2017:

TYAN vs SuperMicro. So this TYAN system has a dual PCIe root complex, i.e. four GPUs under CPU 1 and four GPUs under CPU 2. How do you like the TYAN platform compared with something like a SuperMicro 4208GR-TRT2? At Lambda Labs, we use the SuperMicro platform but are considering switching. It’s a bit unfortunate that the TYAN doesn’t have a single root complex. (Accomplished with two PEX 8696 PLX Switches.)

Sorry, the comment form is closed at this time.