Re: Parallel Application Processing

From: Holger Baer <holger.baer_at_science-computing.de>
Date: Wed, 10 Aug 2005 11:43:06 +0200
Message-ID: <ddci7a$i5u$1@news.BelWue.DE>

Billy wrote:
> Holger Baer wrote:
>
>

[...]

>

>>According to Cary Milsap (read Batch Queue discipline and Batch Queue
>>and the magic of 2 available at hotsos.com after registration) you shouldn't
>>run more jobs than 2 times the number of CPUs in parallel.

>
>
> Holger, I will take that as a rule of thumb as it depends on whether
> the processes are CPU bound or not. Unix is capable of running 1000's
> of processes in parallel. Windows is capable of running 1000's of
> threads in parallel. It is the actual work done in those that governs
> just how many can be effectively run in parallel.
>

That goes without saying. But the point is, that in a typical batch environment CPU and disk usage aproximately level out, so while one process is using CPU the other one is waiting for disk. And I talked *specifically* about batch processing, which means the processes are actually doing something - not some threads that are just hanging around. :-)

As for the scalability of this solution: You just have to break it up into enough pieces, set the job_queue_processes accordingly and Bob's your uncle.

Cheers
Holger Received on Wed Aug 10 2005 - 04:43:06 CDT