9+ GPU machine

Moderator: BarsMonster

Post Reply [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable
D3ad0ne
Posts: 111
Joined: Fri Sep 04, 2009 5:37 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

9+ GPU machine

Post by D3ad0ne » Fri Dec 04, 2009 4:55 pm

I thought I would create a new thread instead of posting in the 'how high can you get thread'. I have been working on making a +9 GPU machine. I was waiting on a PCI-e extension cable to test this out. It finally arrived today and I popped in the 9th card. I had been getting conflicting comments that it would not work due to bios, os, or driver issues. Well I am happy to say that as soon as windows booted it identified the device and installed the drivers. I am still testing but it appears that all my GPU hash cracking utilities work and fully utilize all 9 GPU's, except BarsWF v.8, I receive invalid device pointer errors. I now have on order two more cables that will allow me to use all 7 PCI-e slots. But for anyone that is curious if this type of setup will work the answer is yes, it can be done.
dev_man_9_GPU.JPG
dev_man_9_GPU.JPG (29.96 KiB) Viewed 11406 times
ighash_9_GPU.JPG
BarsWF doesn't like having more GPU's than CPU's the numbers bounce around a lot, and I get really bad speed with all 8 cores being used. --cpu_6 helped out a bit.
BarsWF_64_9_GPU.JPG
BarsWF_64_9_GPU.JPG (45.56 KiB) Viewed 11406 times

IvanG
Posts: 89
Joined: Wed May 06, 2009 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by IvanG » Fri Dec 04, 2009 7:27 pm

Congrats on building such system :). Never seen anything like this.

Anyway, results aren't looking good for me -- having 4xGTX295 it should be ~8*705M = 5640M for single MD5 with ighashgpu. But your screenshot shows only 5093M. 3 out of 8 GTX295 cores running non optimally and that can be caused by some hardware conflicts. Bars's result for 9xGPU also lower than it should be for 8x. Definitely more tests are needed.

D3ad0ne
Posts: 111
Joined: Fri Sep 04, 2009 5:37 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by D3ad0ne » Fri Dec 04, 2009 9:24 pm

3 out of 8 GTX295 cores running non optimally and that can be caused by some hardware conflicts. Bars's result for 9xGPU also lower than it should be for 8x.
You are correct. I didn't have a chance to look into it to much at the time. With only 8 GPU's I normally get over 6B/sec with bars, however this was just a quick picture snap to show that it works. It usually take a min or so to stablize. GPU2 in the pictures is the 8800GTX it should be getting around 400M/sec. But I recived an error at boot up that said that it may not be reciveing enough power, and that performance will be reduced to protect the card so I figured that was why it was only getting 50M/sec. I'll work on it more later tonight and see if I can get the bugs worked out.

D3ad0ne
Posts: 111
Joined: Fri Sep 04, 2009 5:37 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by D3ad0ne » Sat Dec 05, 2009 12:32 am

Well apparently I was having a power issue, I used alternate power source for the extra card. Now it is using the full use of the 8800GTX :). On bars I had to use --cpu_n 6 in order for it to be stable enough for it to keep the GPU's fully tasked. IGhash is actually slower now that they are all working. Once again I think it is over tasking the CPU. Now I just need the dang i9's to come out. :(
BarsWF_64_9_GPU2.JPG
BarsWF_64_9_GPU2.JPG (45.61 KiB) Viewed 11369 times

IvanG
Posts: 89
Joined: Wed May 06, 2009 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by IvanG » Sat Dec 05, 2009 12:53 am

D3ad0ne wrote:IGhash is actually slower now that they are all working. Once again I think it is over tasking the CPU. Now I just need the dang i9's to come out. :(
Which is strange in fact as without /cpudontcare switch it shouldn't use CPU at all...

User avatar
LordMike
Posts: 184
Joined: Tue Feb 10, 2009 8:34 am
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by LordMike » Sat Dec 05, 2009 5:07 pm

Fascinating... :D

User avatar
BarsMonster
Site Admin
Posts: 1118
Joined: Wed Oct 01, 2008 7:58 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by BarsMonster » Sat Dec 05, 2009 9:35 pm

I do not think it is limited by CPU, must be driver issue or you have a bunch of cards on a single 1x PCI-E line.
Could you try --gpu_time=1500 and larger charset (aA0~) ? Does it makes a difference?

D3ad0ne
Posts: 111
Joined: Fri Sep 04, 2009 5:37 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by D3ad0ne » Sun Dec 06, 2009 5:21 am

Could you try --gpu_time=1500 and larger charset (aA0~)
Gave it a try, the screen refresh is slower, but speeds are about the same.

GPU-Z says that all cards are running @ 16x except for the 8800 GTX is running at 4X. Which I think is weird cause it is suppose to go to 8X for that slot and the next one according to the motherboard manual.

There is definitely something wrong somewhere. Unfortunately I'm not sure what.

Crucifix wrote:
I dunno if CUDA has any specific GPU limitation.

I know for a fact Windows will not boot if there are more than 8 active GPUs.
I would like to point out that I have to wait about 45 seconds for the windows logon screen to come up. I sits at a blank screen. That maybe why Crucifix said "I know for a fact windows will not boot" Perhaps the person didn't wait long enough at boot time to see if it would work.

User avatar
LordMike
Posts: 184
Joined: Tue Feb 10, 2009 8:34 am
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by LordMike » Sun Dec 06, 2009 9:52 pm

D3ad0ne wrote:
I dunno if CUDA has any specific GPU limitation.

I know for a fact Windows will not boot if there are more than 8 active GPUs.
I would like to point out that I have to wait about 45 seconds for the windows logon screen to come up. I sits at a blank screen. That maybe why Crucifix said "I know for a fact windows will not boot" Perhaps the person didn't wait long enough at boot time to see if it would work.
Uh.. Point of fact... If by 'blank screen' you mean the windows blue background.. Then it *has* booted.. :P
But I s'pose you mean black screen :P.. Other would seem rather dumb.. :P

D3ad0ne
Posts: 111
Joined: Fri Sep 04, 2009 5:37 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by D3ad0ne » Mon Dec 07, 2009 1:35 pm

Uh.. Point of fact... If by 'blank screen' you mean the windows blue background.. Then it *has* booted..
Haha, yea this is between windows XP logo that has the scrolling blue bar and the actual password logon screen. The screen is black, and the monitor light is green, that means that it is getting a signal but the system appears to be locked up. I initally thought something was wrong, I stepped away for a minute and came back and it was at the logon screen. Perhaps it just takes a while for the drivers to identify all the hardware devices, I really donno.

User avatar
LordMike
Posts: 184
Joined: Tue Feb 10, 2009 8:34 am
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by LordMike » Tue Dec 08, 2009 7:59 am

D3ad0ne wrote:
Uh.. Point of fact... If by 'blank screen' you mean the windows blue background.. Then it *has* booted..
Haha, yea this is between windows XP logo that has the scrolling blue bar and the actual password logon screen. The screen is black, and the monitor light is green, that means that it is getting a signal but the system appears to be locked up. I initally thought something was wrong, I stepped away for a minute and came back and it was at the logon screen. Perhaps it just takes a while for the drivers to identify all the hardware devices, I really donno.
Official windows code!!!

Code: Select all

if (devices)
{
  loadtime += 10000 * devices.length;
}

IvanG
Posts: 89
Joined: Wed May 06, 2009 12:42 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by IvanG » Wed Dec 16, 2009 11:26 am

http://fastra2.ua.ac.be/
The FASTRA II design contains six NVIDIA GTX295 dual-GPU cards, and one GTX275 single-GPU card.
With the help from ASUS they got modified BIOS :).

D3ad0ne
Posts: 111
Joined: Fri Sep 04, 2009 5:37 pm
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Re: 9+ GPU machine

Post by D3ad0ne » Wed Dec 16, 2009 2:47 pm

Thanks for the link Ivan. It's funny that they point out:
Although the system is up and running, we are still experiencing software stability issues, probably caused by an incompatibility between the video drivers and the BIOS and Linux modifications we had to use
So I'm glad to see I'm not the only one that has some problems. Seems like they are only using it for boring medical stuff though. Right now I'm not buying any more GTX 295's to add to the machine. Coolness factor aside I don't want to deal with custom bios's and software issues. So my efforts are saving up money for the 300 series cards.

Post Reply
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable
[phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1266: count(): Parameter must be an array or an object that implements Countable

Who is online

Users browsing this forum: No registered users and 1 guest