Map Reduce is dead, long live Spark!
That’s the impression I, and I think most people attending the conference, walked away with after Strata NY 2014. Most of the interesting presentations were centered on Spark. Only corporate IT presentations about “in progress hadoop implementations” were about Map Reduce.
So who’s working on Spark? Cool startups and vendors (preparing for enterprise IT departments to move on to Spark in a year or two).
Who’s working on Map Reduce? Corporate IT departments migrating off legacy BI systems onto the promised land of Hadoop (dream come true, or nightmare around the corner, not sure which one it will be for people).
It makes sense. Map Reduce has been tested and is ‘safe’ now for enterprise IT teams to start deploying it into production systems. Spark is still very new and untested. Too risky for a Fortune 500 to dive into replacing legacy systems with a still-in-diapers open source software “solution.” Nonetheless, I am sure every technical worker will be drooling to “prototype” or create proof of concepts with Spark after this conference.
David Mantilla 