Parallel Computing: The New Old Thing

Wednesday, April 1, 2009 – 2:33 PM


This is one of several forthcoming posts on concurrency and parallel computing. Much of the content may end up in a white paper I’m writing to help developers get to grips with concurrent or parallel computing. I’d love to hear your feedback on this. If you think this is helpful or could be improved then post a comment below.

What if someone told you that your application isn’t going to run faster on the next generation of hardware? What if they said that it might actually run slower? What if they told you that to harness the power in future generations of hardware your applications are going to have to execute code in parallel?

Executing in parallel means different parts of your application are running concurrently. Many applications do this today by using background threads to do work while their main UI thread continues to respond to user input. But parallel computing usually means a lot more concurrency that this. Applications might have tens or hundreds of concurrent operations working together on the same task, not just a couple of worker threads.

The IBM SP2. A parallel high performance computer that looked the part. Parallel computing has been around for a long time. For most of that time it’s been talked about as the next big thing. Like pen computing it was going to be “huge next year”. I was playing around with parallel machines back in the early 90s and while they held their own in certain specialized fields, like scientific modeling and engineering, they didn’t really move into the mainstream. Yes, they moved into other fields like financial modeling and graphics but hardly became a mainstream or desktop phenomenon.

Why was this? Two reasons really. The hardware was costly and specialized. The first parallel computers used custom built hardware and were very expensive. The machine I wrote applications for in 1990 used custom hardware and had a proprietary operating system and development language and would have cost over $100k in today’s money. Over time this changed somewhat with the market moving to clusters of Windows or Linux machines which could be assembled from off the shelf hardware and use common libraries or compiler extensions making the applications more portable.

What hasn’t changed was that writing software for these parallel machines is still a specialized field and for many the investment in porting an existing application to a different hardware platform and rewriting it to take advantage of a parallel architecture wasn’t worth the effort. Especially when the performance of regular desktop machines keeps increasing. Up until now.

The end of the free lunch

What’s changed today? Well firstly the “free lunch” provided by ever rising processor clock speeds is pretty much coming to an end. Clock speeds are no longer rising (see graph). So the days of your application running twice as fast when recompiled on the next generation of processors is rapidly coming to a close. Moore’s Law will still holds, so for a few more generations the number of transistors will continue to double. But that just means more transistors running at the same speed.

The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software - Herb Sutter (Mar 2005)

So how do you take advantage this? The answer is concurrency. Add more cores on the same processor die and allow instructions to execute in parallel on each core. Two cores aren’t typically going to run twice as fast as one core but a reasonably written application might see a x1.5 improvement over a single core running at the same speed. This can vary a lot depending but in general two cores can provide more processing power than one core but not twice as much.

What’s even more interesting is that this is here today. The computer you’re reading this on now is probably running code in parallel on two or more cores. Multi-core x86 processors started appearing in 2004 and today pretty much any new machine sold will have at least two cores. The latest generation of desktop applications – like Office3 Excel 2007 and AutoDesk – have already been modified to take advantage of multi-core hardware.

Time for some terminology. Multi-core, manycore, what’s the difference?

“In general, a ‘multi-core’ chip refers to eight or less homogeneous cores in one microprocessor package, whereas a ‘manycore’ chip has more than eight possibly heterogeneous cores in one microprocessor package. In a manycore system, all cores share the resources and services, including memory and disk access, provided by the operating system. Microsoft and industry partners anticipate the advent of affordable general-purpose ‘manycore’ systems in a few years.” 

The Manycore Shift, Microsoft (Nov 2007)

So we have multi-core on the desktop today. In some ways we have manycore is too. Some sections of the scientific and engineering communities are already using the parallel processing capabilities of graphics cards–which contain hundreds of cores–as a compute resource for certain types of problems. There are some great examples of this on Nvidia’s CUDA Zone web site.


So the hardware is here but what about the software? In order to take advantage of this new parallel world applications need to be redesigned and the work distributed across the available cores. None of this makes any difference if only a handful of people can actually write programs that run on these new multi or many core processors. The changes in hardware are starting to drive new investments in software too.

“The software development industry is taking strides to make parallelism more accessible to all developers, and Microsoft is helping to lead the way. With Microsoft Visual Studio 2010, Microsoft is delivering the first wave of powerful developer tools that ease the transition to parallel code.”

Taking Parallelism Mainstream, Microsoft (Oct 2008)

So that sounds pretty serious, and if you take a look there’s a lot coming in the Visual Studio 2010 timeframe. The pretty diagram below shows libraries for both managed and unmanaged code, tooling support in the form of debuggers and profilers as well as investments at the OS and hardware level.

Taking Parallelism Mainstream - Microsoft (Oct 2008)

Really this is about raising the level of abstraction and tooling when it comes to writing concurrent code. This is a good thing. Most significant advances in software development have been about providing more abstraction to allow developers to think about their customer’s problem not their implementation problem. Assembler code gave way to languages like C, managed languages like C# raised the level of abstraction yet again. Today developers think a lot less about memory management and memory locations and more about data structures they defined.

Similarly when it comes to writing concurrent applications it would be a lot better if developers didn’t have to think about threads and threading all the time. You can write code today that takes advantage multi-core processors but it’s a lot of work, thinking about threads, the flow of execution across them and their use of shared memory.

What’s next?

I’d like to consider all this from my perspective as a development manager and lead. What’s the bigger picture? How does this play into application development today?  What are the proven practices and patterns which may help? How do can you get the other developers on the team up to speed? Do you have to? What parts of the application should be parallelized? How should you go about it?

This might seem like a tall order and very speculative given that everything in the diagram above is still yet to ship in beta. The key here is that the while all of this is new to most PC developers the High Performance Computing community has been working on this for many years and figuring out what works and what doesn’t. Amazingly some of the things they figured out twenty years ago are still relevant today. 


So while I work on part 2 here are some of the resources I found useful so far (in no real order):

There’s also some good stuff on what might happen in the future and how the level of abstraction may rise still further.

  1. 3 Responses to “Parallel Computing: The New Old Thing”

  2. Would I be right in thinking that all this talk of Lambda Expressions in LINQ is, in part, softening us up for some serious functional programming goodness?

    By Neil on Apr 14, 2009

  3. Neil,

    Well it wasn’t until someone gave me a book on F# for Scientists. Now it might be :).


    By Ade Miller on Apr 14, 2009

  1. 1 Trackback(s)

  2. Sep 22, 2009: PDC 2009: Patterns of Parallel Programming Resources | #2782 - Thinking about agile (small 'a') software development, patterns and practices for building Microsoft .NET applications.

Sorry, comments for this entry are closed at this time.