Re: RAC or Large SMP...?

Robert Klemme <shortcutter_at_googlemail.com> writes:

> On 10.10.2008 08:38, Tim X wrote:
>
>> There is a growing argument that one of the main limitations of
>> multi-core systems isn't at the hardware level, but rather, at the
>> software level. In particular, the lack of good support in many
>> languages for writing software that is able to exploit parallel
>> processing efficiently. The few languages that provide such support
>> often rely heavily on the ability of the programmer to understand the
>> complexities and have the skills to adopt algorithms suitable for such
>> processing. such programmers are rare, particularly in the current era
>> of out sourcing and commodity development where the emphasis tends to be
>> towards programmers that are API driven code monkeys that have little
>> depth of knowledge regarding algorithms and data structures generally
>> and probably little, if any, knowledge of issues associated with
>> parallel processing.
>
> I believe this analysis is spot on, but...
>
>> There are some very interesting approaches being
>> developed and I'm confident that in time, many of these considerations
>> will be handled by clever compilers and sophisticated language support
>> and APIs that are multi-core aware, but thats probably a way off yet.
>
> ... I am not too optimistic here. I do believe that we will see some
> improvements over time (for example, since Java 5 there are more
> abstractions of concurrency constructs which help building concurrent
> applications that do scale better). But I do not see that we will come to
> a situation where the programmer provides a sequential problem description
> and gets parallel execution. For a very long time the people implementing
> parallelized applications will have to do the analysis and design work that
> determines which parts can be done concurrently and how synchronization
> must take place. And from my experience this is the crucial part in
> getting scalability through concurrency into an application.
>

Yes, I agree. I probably wsn't clear enough. I don't think we are ever likely to reach the point where programmers can stay in a sequencial mindset and the tools will take care of the rest. Programmers will need to change the way they think about problems and learn to apply ADTs and develop algorithms that are able to exploit the parallelism. My amin point was that even those programmers that are able to think that way are somewhat constrained by current language and compiler technology that make it quite complex and difficult to express parallelism in a way that is concise and clear. Furthermore, due to the lack of adequately expressive constructs, being able to reacj a satisfactory level of confidence re: correctness etc is very difficult. to some degree, what is already available seems very much like a poorly fitting kludge bolted onto languages that were primarily designed for sequencial non-parallel environments.

I listened to a talk by 'Matz' (Ruby) who said that what he wanted to achieve with Ruby was a language that had a close fit with how he thought about problems. His arguement was that this allowed him to express his problems in a way that was clearer and faster because he didn't have to translate his mental model into a program model. to some extent, I think this applies with parallel programming. currently, those programmers that have the knowledge/skills to think about problems in a parallel fashion are constrained by the fact they have to then translate their mental model into a language that was designed primarily for sequencial problem solutions and which has cumbersome 'enhancements' designed to provide a safety net when operating in a parallel environment, but which often don't fit 'naturally' with the language. I'm talking about mainstream languages rather than specialised languages like Occum or other langauges that have been designed specifically for developing applications that can exploit parallelism.

I've actually been surprised in the past by the degree to which most programmers have a 'sequencial' mindset. for example, I was surprised at the number of programmers that got 'stung' in the early days of the web because they had an assumption that visitors would arrive in a sequencial manner. I still come across 'bugs' that are primarily due to such thinking. for example, the number of web page counters that are incorrect because they assume that no other process will access their datafile in between reading the number and writing back a new number that is incremented by 1. I often have to explain to novice programmers that in many situations you cannot assume sequencial access to shared resources. You even see similar issues with many database applications. How often do we see confused questions regarding locks, commits, rollbacks etc.

I think I must have been very very lucky. The second course I did in computing science was on parallel programming. It was back in the late 80s and I was lucky enough to have access to a simulated hypercube. It was a special PC board that cost a rediculous amount of money (even for back then when a PC with a 20Mb hard drive and EGA graphics that ran at 12Mhz cost over $6k. I think the bord cost something like $30k or more. (this was when I purchased my first house, which cost only $60k and which I sold a couple of years back for $300k) You had a buggy DOS (I think it was only DOS 1 or 2, most PCs were DOS 3.3 from memory) and a hacked C compiler that had some additional features that made it easy to address specific nodes in the hypercube and some basic suppoort for semaphores and mutex type concepts. As primitive as it was, it was a hell of a lot of fun. You could develop interesting sorting algorithms that addressed specific nodes in the hypercube using a mitmask addressing scheme. However, once you began to get even slightly complex, it became extremely difficult to map your mental model to the software level and debugging problems was a complete nightmare! Still, I think having done that course put me in a very good position for much of the stuff I worked on later in life.

There is some interesting work I've seen passing references to that looks quite promising, especially in the area of providing safe access to resources in a predictable way. For example, I think Microsoft has been doing some work on a sophisticated optimistic memory model in which access is controlled by the udnerlying framework and which aims to eliminate the programmer from having to worry about such issues. The model is very similar to how oracle handles such issues and enables consistent views of the data. in fact, I think from memory that they said the model they were developing was based on how databases have addressed the need to provide consistent views of data in an environment with many clients. It is technologies like this that will enable programmers to devleop parallel programs that don't become so complex that either they never get out of the bug squashing stage or when they do, they die due to maintenance problems.

Of course, all of this may become completely mute when quantum computing comes into play and all our software is in every possible state at every possible moment. Now that will be a paradigm that is likely to melt a few programmer minds that have grown up in the von neumann world! Luckily, I expect to be retired by then.

Tim

-- 
tcross (at) rapttech dot com dot au