This section should teach participants:

1. What Hadoop is
2. Current R/Hadoop integrations
3. When to use R with Hadoop (guidelines)
4. How to use R with Hadoop (lab)

Key ideas: enables distributed computing; open source; widely used

• older, more mature "cloud computing" technology

The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage…

Source: Apache Software Foundation - What is Apache Hadoop?

Two of Hadoop's main features are particularly relevant to this talk:

Distributed computing is analogous to parallel computing

Parallel - multiple processors run code

Distributed - multiple computers run code

Size matters

• Parallelization occurs when data is stored on multiple computers
• Files are typically split into 64MB or 128MB chunks
• Small files won't parallelize

MapReduce programs have 3 main stages

1. Map: Apply a function to extract and group data
2. Shuffle/sort: Sort the function's outputs
3. Reduce: Compute summaries of the grouped outputs

• Extract data
• Group data
• Compute group summaries

"Everything else"

• Iterative algorithms
• Multi-step workflows

Integration purposes

• Let people use Hadoop to execute R code
• Let people use R to access data stored in Hadoop

Your computing needs align with natural strengths of R and Hadoop

Evaluate alignment with the following factors:

1. Download lab materials from web (links on handout)
   • R script
   • Lab instructions
   • Windows only: PuTTY or other ssh client
   • Optional: presentation slides
2. Connect to TheCantina wireless network

1. Present an example of a problem to integrate
2. Connect to Hadoop via R
3. Work through a basic integration
4. Modify the analysis on your own

• A car insurance company launched a small pilot study to evaluate a new program they are considering offering to all of their customers. At the end of the study the participants were asked whether or not they would like to stay enrolled in the offering.

• The company would like to use the participants' demographic information and their feedback to help predict whether the program can be profitable if offered to all customers.

• For marketing purposes, they are additionally interested in knowing if the program is very popular with specific subsets of their customers.

Business questions:

• Keep the program?
• Focus marketing to specific groups?

1. In R, build a logistic regression model from the pilot study data
2. PARALLEL Use RHadoop to use the regression model to predict which customers would like in the program (map) and combine the results (reduce)
3. Use RHadoop to retrieve the summary data and conduct follow-on analyses in R or build charts and tables to help present the data

This will demonstrate…

• Applying prediction and classification models to Hadoop data (customers)
• Analyzing small data stored in Hadoop (summary data)

• Log in to a server that can submit MapReduce jobs to Hadoop

  ssh cloudera@192.168.1.105    (password: cloudera)

• Start R

  R

• Within R…
  • Load key RHadoop library

    library(rmr2)

  • Tell RHadoop where to find Hadoop commands

    Sys.setenv(HADOOP_CMD='/usr/bin/hadoop')
Sys.setenv(HADOOP_STREAMING='/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming.jar')

• Open "The Hadoop UI" (HUE)
  • HUE landing page: http://192.168.1.105:8888/
    • Username and password: cloudera
  • View status of MapReduce jobs: http://192.168.1.105:8888/jobbrowser/
  • View contents of hdfs: http://192.168.1.105:8888/filebrowser/#/

• Load data from pilot program

  pilot.path = '/CSP/data/insurance/pilotProgram_results.csv'
  pilot = from.dfs(pilot.path, format='csv')
  pilot.data = pilot$val
  colnames.path = '/CSP/data/insurance/columnNames.csv'
  data.colnames = from.dfs(colnames.path, format='csv')
  colnames(pilot.data) = t(data.colnames$val)

• Fit logistic regression model to data

  pilot.fitted = glm(stay ~ ., binomial, pilot.data)

• Reduce model and view results

  library(MASS)
  pilot.fitted.reduced = stepAIC(pilot.fitted)
  summary(pilot.fitted.reduced)

• Write mapper

  predictor.names = t(data.colnames$val)[-1]
  predictor.levels = lapply(pilot.data, levels)[-1]
  predictors.count = length(predictor.levels)
  not.null = function(x) { !is.null(x) }
  columns.forFactors = (1:predictors.count)[sapply(predictor.levels, not.null)]

  prediction.mapper = function(key, customer){ 
        colnames(customer) = predictor.names
        for(factorCol in columns.forFactors) {
          unseenLevels = which(!(customer[,factorCol] %in% predictor.levels[[factorCol]]))
          customer[unseenLevels, factorCol] = NA
        }
        customer.pred = plogis(predict.glm(pilot.fitted.reduced, customer))
        keyval(paste(as.character(customer$region), as.character(customer$gender)),
               ifelse(customer.pred>.5, 1, 0))
  }

• Write reducer

  counting.reducer = function(k, vv) { keyval(k, sum(vv, na.rm=T)/length(vv)) }

• Set up and execute MapReduce job

  mapred.result = mapreduce(
        input = '/CSP/data/insurance/customer_profiles.csv',
        input.format = 'csv',
        output.format = 'csv',
        map = prediction.mapper,
        reduce = counting.reducer
  )

• Retrieve analytic results

  mapred.result.data = from.dfs(mapred.result(), format='csv')

• Use a different regression/classification model

• Compute more detailed summaries of customer predictions
  • E.g., Group by gender, age, and region
  • E.g., Work with raw estimates and compute averages or variances instead of percentages
• Identify model weaknesses
  • What factor levels are present in customer records but not in pilot records?
  • Which or how many customers have these types of levels?
• Explore functions and objects in rmr2
  • Look at mapred.result
  • Write data to hdfs in different formats

• Practical computing requires balancing computing speed, programming efficiency, and personal comfort. R/Hadoop integrations offer more ways to achieve balance.

• R/Hadoop integrations give practitioners opportunities to use strengths of both technologies.

• R/Hadoop integrations give R programmers access to "non-R" technologies.

R/Hadoop integrations

• RHadoop documentation and examples on github
• RHIPE's example analysis of airplane dataset
• Any online tutorial for logistic regression or k-means via R/Hadoop
• RevolutionAnalytics' integration of biglm with RHadoop

Hadoop

• Hadoop documentation by Apache, Hortonworks, or others

Logos/Graphics:

• Apache Software Foundation
• Amazon Web Services
• HortonWorks

• Parallelization fully uses computational resources and saves time

• Parts of many statistical analyses are parallelizable

• Many options for parallelizing R code - use what works for you!