Join the web’s most supportive community of creators and get high-quality tools for hosting, sharing, and streaming videos in gorgeous HD with no ads.

Keynote day 2 of Berlin Buzzwords 2012. More Info: http://berlinbuzzwords.de/sessions/keynote-0

Alex Lloyd - Building Spanner

Cascalog is a data processing library for Clojure - a Lisp dialect for the JVM. Cascalog mixes functional and logic programming. It makes data processing code very concise, easier to grasp and reason about. Being just a Clojure library, all of Clojures&#039; features are just a keystroke away - no need to learn another custom language (like Pig or Hive). This also allows for powerful abstraction and composition capabilities. Live Demo - Parens abound!  More Info: http://berlinbuzzwords.de/sessions/introducing-cascalog-functional-data-processing-hadoop

Stefan H&uuml;bner - Introducing Cascalog: Functional Data Processing for Hadoop

Using Hadoop based business intelligence analytics, we analyzed the Hadoop source code and its development over time and found some interesting and fun facts we want to share with the community.  This talk will illustrate text and related analytics with Hadoop on Hadoop to reveal the true hidden secrets of the elephant. This entertaining session highlights the value of data correlation across multiple datasets and the visualization of those correlations to reveal hidden data relationships.  More Info: http://berlinbuzzwords.de/sessions/analyzing-hadoop-source-code-hadoop

Peter Voss - Analyzing Hadoop Source Code with Hadoop

Bringing Together Millions of Publishers and Thousands of Advertisers by Zanox Reporting Systems built on top of Hadoop and Lucene Technologies Dr. Dragan Milosevic, Senior Hadoop Architect, dragan.milosevic@zanox.com Zanox is company that deals with the huge amount of data. Currently our tracking gets more than 2 million sales, 30 million clicks and almost 1 billion views every day. Huge amount of data are also coming from search-engines that are providing information about related costs. The challenge is to join and summarise those huge sets of data and provide valuable tracking and cost statistics to more than 1 million publishers and 10 thousands advertisers. They will use our analyses to successfully drive their business by knowing which launched campaigns are well performing and how others can be changed to bring even more money. This talk will have two parts. The first part shows how Hadoop helps in analysing available data and saving valuable results in Lucene indexes. The second part describes Lucene search infrastructure that is able to efficiently generate reports in real time for publishers and advertisers. Challenges to be solved during Hadoop processing can be summarised as follows: (1) Joining several huge sets of data, namely the data that are tracked by Zanox, costs data coming from search-engines and master-data about publishers and advertisers. We have developed unique approach that combines map-side and reduce-side joins, trying to combine the best of both techniques. It does sampling to identify records which have join-key that is frequent. Those records will be joined on a map-side and directly written to output without loading map-reduce pipeline. Only records whose join-key is infrequent will be propagated to reducers in order to be joined there. Provided experiments have shown significant gains compared to pure reduce-side join mainly due to the fact that more than 80% of records can be joined on a map-side, resulting that only one fifth of records have to be sorted and joined on a reduce-side. (2) Several aggregation jobs are found to be very important for us, mainly due to the huge number of records that have to be summarised. We have made many experiments aiming to speed-up map-reduce engine, being in our particular case slow mainly due to the fact that sorting on a reduce-side has to wait until all map-tasks are not completed. In the case where thousands of map-tasks have to be executed, reduce tasks are waiting a significant amount of time. Our solution is very simple, and it is based on replacing one huge job with several smaller ones that are responsible only for the selected part of input data. Because smaller jobs have much less map-tasks that have to be finished before reduce-tasks can start with expensive sorting and aggregation operations, we have managed to earlier activate resources that are responsible for reduce tasks and on average to speed-up complete aggregation for more than 30%. (3) Lucene indexes save every aggregated result produced by Hadoop jobs to make them real-time available to publishers and advertisers. Created indexes represent different views to aggregated data, being responsible to speed-up the processing of queries. We have optimised indexing performance by (a) building different aggregations of every record simultaneously, (b) using intelligent partitioners that are sending the semantically same aggregated-versions of records to same reducers, and (c) taking care of producing medium-size indexes that can be generated completely in memory on a reduce-side. Lucene search infrastructure is dealing with the real-time generation of reports on a request basis. Because the response time plays very important role while serving online requests, the architecture of our search-backend is optimised by following actions: (1) Indexing data with different aggregation granularities to optimise the execution of various types of queries. Ideally the requested information is already directly indexed in which case it is only necessary to find needed record and to deliver it to publisher or advertiser. For example, one publisher might be interested in knowing the number of views on a particular day. If such an aggregation is already pre-computed by Hadoop and directly available in Lucene index, reporting is simply searching for a record that corresponds to given publisher and day. Those reports can be generated in a matter of only several milliseconds. (2) Estimate costs of processing each received query to select the best available index and to optimise response-time for report-generation. Due to the great flexibility that publishers and advertisers have in building their own customised queries, it is not possible to pre-compute all possible aggregation-levels for every single query that can be received. Our approach is capable of estimating costs of processing each received query by every available index type. The index that has the closest aggregation-level will be chosen, because it guarantees that the amount of aggregation that has to be performed on-the-fly is the smallest. For example, if one advertiser wants to know the number of clicks in the particular time-period, the given time-period will be first split on days, months and years. Indexes with year-aggregation are preferred over month and day-aggregations, due to having smaller on-the-fly aggregation requirements. Consequently the most expensive day-aggregation will be used only for the parts of the requested time-period where month and year aggregations are not applicable. (3) Profit-aware combination of memory and file-system indexes provides nice business models by saving more important data in memory indexes and others in file-system ones. In practice this means that the important publishers and advertisers are getting more space in memory indexes and therefore they will get their reports faster. The implemented solution can be nicely used to reserve the amount of memory indexes to be proportional to paid price margins. Those two parts of our reporting-system have already a long history behind. System is productive since 2009, and one of its first versions has been already presented on Hadoop-Get-Together in March 2010 (http://vimeo.com/10201534). Its enhancements made in the last two years are here briefly summarised. If both parts of the system are to be presented 40 minute slot will be more than welcome. Never the less, we can always focus presentation either on Hadoop or Lucene part, in which case it will be possible to fit everything in 20 minutes.  More Info &amp; Slides: http://berlinbuzzwords.de/sessions/bringing-together-millions-publishers-and-thousands-advertisers-zanox-reporting-systems-bui

Dragan Milosevic - Bringing Together Millions of Publishers and Thousands of Advertisers

Hadoop is the core of a growing ecosystem of software components. Hadoop itself only consists of a distributed file system (HDFS) and a functional programming layer (MapReduce) to process stored data. The Hadoop Ecosystem developed tools around these basic components to interact with Hadoop more easily and integrate it into other environments. This talk gives a brief introduction into Hadoop, and then covers the most important tools tools around it. When possible, demos will be presented. Sqoop &amp; Flume - Getting Data into HDFS from Logs and Databases Hive &amp; Pig - High Level Access to MapReduce HBase - A random access, low latency key/value store on top of Hadoop Mahout - A Machine Learning Library for MapReduce Whirr - Install a Hadoop Cluster on external cloud providers Oozie - A workflow engine to automatically submit Hadoop jobs and some more tools The target audience is new users of Hadoop to get an overview of the many available tools around.  More Info: http://berlinbuzzwords.de/sessions/hadoop-ecosystem

Kai Voigt - Hadoop Ecosystem

PLEASE NOTE: Because of technical problems the first few minutes of this recording are missing. We are very sorry and apologize for this.  Apache Giraph is a an implementation of Google&#039;s Pregel system for working with large scale graphs, going beyond the Map-Reduce model, with an approach tailored for graph algorithms. Whereas graph algorithms must be shoehorned into the Map-Reduce model, within Giraph they are expressed naturally by modeling the behavior of vertices across synchronized steps. Important algorithms such as PageRank become trivially simple, with orders of magnitude improvement over traditional Map-Reduce approaches. Giraph was originally developed at Yahoo! Research and is now an Apache project with a diverse and growing community, including committers from LinkedIn, Facebook, Twitter and several universities.  More Info: http://berlinbuzzwords.de/sessions/large-scale-graph-computation-apache-giraph

Jakob Homan - Large scale graph computation with Apache Giraph

Apache Hive provides a convenient SQL-based query language to data stored in HDFS. HDFS provides highly scaleable bandwidth to the data, but does not support arbitrary writes. One of Hortonworks&#039; customers needs to store a high volume of customer data (&gt; 1 TB/day) and that data contains a high percentage (15%) of record updates distributed across years. In many high-update use-cases, HBase would suffice, but the current lack of push down filters from Hive into HBase and HBase&#039;s single level keys make it too expensive. Our solution is to use a custom record reader that stores the edit records as separate HDFS files and synthesizes the current set of records dynamically as the table is read. This provides an economical solution to their need that works within the framework provided by Hive. We believe this use case applies to many Hive users and plan to develop and open source a reusable solution.  More Info: http://berlinbuzzwords.de/sessions/processing-high-volume-updates-hive

Owen O&#039;Malley - Processing High Volume Updates in Hive

Keynote day one of Berlin Buzzwords 2012. More Info: http://berlinbuzzwords.de/sessions/keynote

Leslie Hawthorn - How to make your project blossom

Hadoop has proven to be an invaluable tool for many companies over the past few years. Yet it has it&#039;s ways and knowing them up front can safe valuable time. This session is a run down of the ever recurring lessons learned from running various Hadoop clusters in production since version 0.15. What to expect from Hadoop - and what not? How to integrate Hadoop into exiting infrastructure? Which data formats to use? What compression? Small files vs big files? Append or not? Essential configuration and operations tips. What about querying all the data? The project, the community and pointers to interesting projects that complement the Hadoop experience. This session varies from high level discussions to quite technical details. Check out the slides of the talk here: http://slidesha.re/bbuzz-hadoop-lessons-learned  More Info: http://berlinbuzzwords.de/sessions/hadoop-lessons-learned

Torsten Curdt - Hadoop Lessons Learned

HCatalog is a table and storage management system for Hadoop that makes it easy for multiple data processing tools to interact with the same data. Since it provides a single data model, it allows users to pick the best tool for their job, whether it is Pig, Hive, or MapReduce, and share with other users who make different choices. Its table abstraction allows users to read and write data without being aware of where on HDFS the data is stored and what format is used (text, sequence file, RCFile, etc.). The table abstraction and shared data model also allow schemas to be known and checked as part of program development. As data changes over time, in format or storage requirements, this can be managed without requiring changes to existing data consumers&#039; programs. This talk will cover interfaces for Pig, Hive, and MapReduce users as well as an overview of current work and the roadmap for future development.  More Info: http://berlinbuzzwords.de/sessions/hcatalog-table-management-hadoop

Browser Fingerprint