Apache™ Hadoop® is an open-source software platform that allows hundreds of terabytes—or even petabytes—of data to be processed and analyzed. In a world where more data is being created every day, Hadoop lets businesses and organizations find value in that data, which may have been languishing as an untapped resource until now.
As Hadoop grows, so too does the annual Hadoop Summit. This year’s event, which took place June 26-27 in San Jose, had close to 3,000 technical people gathered together to hear more about how Hadoop is being used to harness the exponential growth in data and information about the future of Hadoop.
“There is no doubt that Hadoop is becoming widely accepted in the community and in the enterprise,” says Abhijit Pol, Chief Architect at Rocket Fuel. “And the growth of the annual Hadoop Summit is basically mirroring that popularity.”
Pol explains why Rocket Fuel agreed to be one of the sponsors of this year’s Hadoop Summit. “We use Hadoop at Rocket Fuel, and we are proud to contribute to the Hadoop ecosystem,” he says. “This year we weren’t showcasing a product, but simply wanted to be at the Summit because Rocket Fuel not only uses Hadoop, but also provides feedback to the community on how we can all work together to improve Hadoop and make it better going forward.”
Rocket Fuel was founded on the vision of using big data and artificial intelligence to advance digital advertising. The company uses artificial intelligence for programmatic buying in identifying buying opportunities at the moment they occur, an activity that generates a tremendous amount of data. “We went with Hadoop because it aligns well with the vision we have,” says Pol. “We wanted to establish an architecture that is very easy to scale, is distributed, and—more importantly—can evolve as the business needs of our customers expand and grow.”
At the Summit, various members of the open-source community submitted papers on how they are using Hadoop, and shared stories about the interesting and challenging problems they are solving. Pol also submitted a paper detailing how Rocket Fuel deals with billions of events per day, and several petabytes of data every month, using Hadoop to effectively deal with that data volume.
Pol’s paper explained the evolution of big-data architecture at Rocket Fuel. He described how the architecture has matured over four years on the Hadoop ecosystem, and how it is expected to evolve going forward. The paper also presented best practices in data pipeline design.
“Even though it took place on the last day of the event, my session was well attended,” says Pol. “We also had a large number of people stop by the Rocket Fuel booth to find out more about our company and what we do with big data. The Summit had very lively atmosphere, with all kinds of technical people interacting and trying to understand the next generation of big-data technology.”
He adds that Rocket Fuel remains committed to using Hadoop. “Not only were we early adopters of Hadoop, we also evolved and grew with it. We are committed to both continuing to use Hadoop and its ecosystem as well as to contributing to it on an ongoing basis.”