Speaking at BUILD: Analyzing “big data” with LINQ to HPC

Thursday, September 15, 2011 – 12:41 PM

Go to the BUILD site.For those of you who haven’t figured it out yet I’ll be speaking at BUILD on Friday afternoon.  (2pm session SAC-453T in room 210b).

Analyzing “big data” with LINQ to HPC

"Big data" refers to unstructured data sets so large that they cannot be analyzed using traditional database tools. Today, big data are becoming more common; it is prevalent not just in Web traffic, but also in industries like oil & gas, finance and manufacturing.

Based on Microsoft Research’s Dryad project, LINQ to HPC is a programming model and distributed runtime for building analysis solutions for big data. It goes beyond MapReduce and leverages the LINQ programming model and HPC scheduler to execute optimized query graphs across a cluster of machines.

In this session, you will learn how to use LINQ to HPC on both Windows Azure and an on-premise Windows cluster to build analytic apps that deal with big data. These apps will be able to scale out to hundreds of machines without having to deal with scheduling, data replication and node failure complexities generally associated with programming a large, distributed data-parallel system.

You can find out more about the session on the BUILD web site, this includes the video and slide decks etc.

For those of you who want the code then the Beta 2 release of LINQ to HPC can be downloaded from the HPC Connect site. This includes most of the sample code I show in the talk (with the exception of the back testing example).