Introducing Boardgame Recommendation website built with Shiny

My family and I love boardgames.  We are also on the lookout for new ones that would fit with the ones we already like to play.  So I decided to create a boardgame recommendation website

https://larrydag.shinyapps.io/boardgame_reco/

I built it using R.  The recommendation engine uses a very simple collaborative filtering algortihm based on correlation scores from other boardgame players collection lists.  The collections are gathered using the API from BoardgameGeek.com.  It is very much in a beta project phase as I just wanted to get something built to get working.

I also wanted another project to build in Shiny.  I really like how easy it is to publish R projects with Shiny.

Some of the features include:

• Ability to enter your own collection
• Get recommendation on your collection
• Amazon link to buy boardgame that is recommended

Its a work in progress.  There is much to clean up and to make more presentable.  Please take a look and offer comments to help improve the website. 

Microsoft allows trying its new SQL Server for Linux

This is a great time to have tools to do data analysis.  Microsoft is now allowing evaluations of its new SQL Server.  This new SQL Server can now be deployed to a Linux OS environment.  This is big news especially for those that have followed this blog through the years.

The new SQL Server version is also carries new analytic tools such as R.  From this Techcrunch announcement

The new version of SQL Server will include improved support for R Services and a number of new machine learning and deep neural networking features.

These are exciting times in deed.  I hope Microsoft continues this path to bringing new tools to the market.

Microsoft releases LightGBM

Microsoft has been really increasing their development of tools in the predictive analytics and machine learning space.  Another such tool they released recently is LightGBM.  From the Github site...

LightGBM is a fast, distributed, high performance gradient boosting (GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Microsoft is definitely increasing their attempts to capitalize on the machine learning and big data movement.  I hope they continue to develop tools such as LightGBM and R with SQL Server.

Statistical Analysis and Data Mining hot on LinkedIn for 2014

Statistical Analysis and Data Mining are considered the hottest skills on LinkedIn for 2014.  According to their report from analyzing jobs and recruiters on the LinkedIn website.  I would say its safe to say that it will continue to be hot for 2015.

If you are looking to get your skills honed up I would suggest looking at ComputerWorld's Beginners Guide to R.  It looks like a complete tutorial and is indexed rather well.

Interesting high contrast plots in R

I was inspired by this blog post and thought I could do the same thing in R.  Well I posted the code in Google+


Here are my results.  Not bad.

Introducing the Shiny App DThiring

Well it has a been a long time since I have written anything on this blog.  I am long overdue.  I've been terribly busy learning new things and getting on with life.  One of the things I have learned is building R applications using Shiny developed by RStudio.   The folks at RStudio have also created a way to deploy Shiny apps using Shinyapps.io.  Follow the link to DThiring

http://larrydag.shinyapps.io/dthiring/

I'm a big fan of Data Tau which is a Data Science equivalent to Hacker News.  Like Hacker News, Data Tau has a Who's Hiring comment on the first of every month.  It is a good resource for those looking to see relevant jobs in the Data Science world.  Well someone created a hiring listings aggregator called http://hnhiring.me/.  I decided I wanted to see if I could build a similar application using R and Shiny.

I will be posting the source code for this app to github in the near future.

If you have any ideas on how to improve this application let me know in this comment section.

Google Statistician uses R and other programming tools

A great interview on the Simply Statistics blog with Google's Nick Chamandy, Phd in Statistics.  Explains that he mainly uses R among other tools to perform his work at Google.  Also of note is the active data science community within Google that uses R as well as some other interesting tools.  Note that they use a lot of data at Google, understandably, and that R usually can not handle the size.  They do a lot of ad hoc reduction of the data with tools like map reduce, Go, and even an R API.  I would love to see how they use the R API to assimilate data.

An interesting insight from the interview is the amount of programming done by the Statisticians.  It seems the culture at Google is to foster autonomy and let the modelers develop their own data manipulation from the raw data.  This requires a broader skillset beyond the statistical analysis tools.

I've found in my work that having knowledge in many tools like R,  CPLEX, and GLPK allows me to be a more effective in my work.  Recently I've been learning a lot of SQL using the PostgreSQL platform.  The tools of SQL combined with statistical tools like R make for a very strong combination.  I'm very agile in my work and can do a varied number of decision analysis.

Popularity of R continues

No doubt those that read my blog know that the tools I use to do my Industrial Engineering and Operations Research work heavily rely on the open source side of software.  That is why I try to support as many open source projects such as COIN-OR, GLPK, and OpenOpt.  One tool that I love to perform Applied Math and Statistics is the statistical computing platform R.  So it comes as no surprise that I like to see how R is growing and its popularity among programmers.

A recent blog from RedMonk produced results of a programming language popularity study.  The study involved ranking popularity using common social media online sites such as Stack Overflow and GitHub.  These sites draw in a lot of programmers for their popularity around Q&A and code review.  I was surprised to see that R ranks highly compared to some very prominant programming languages.

Also interesting to note that the only other "Data Science" type of programming language I could find was Matlab.  As far as I could tell SAS, S, SPSS, Stata are still rather popular but apparently not among the programming community.

National Registry of Exonerations charts with R

According to recent news (dallasnews.com) there is a new release of a public national database for wrongful convictions.  There are plenty of details in the public list including Age, Race, and how the conviction was overturned.  According to the database it seems that most of the convictions were overturned due to DNA evidence.

I thought it would be interesting to plot summaries of the details using the open source statistical computing environment R Project.  The following are the plots from the National Registry of Exonerations database.

 

Here is the R code used to create the above pie charts.


# National Registry of Exonerations
# pie charts

library(XML)

u <- "http://www.law.umich.edu/special/exoneration/Pages/detaillist.aspx"

listu <- readHTMLTable(u)

exondf <- listu[[7]]
data <- exondf[24:nrow(exondf),]
names(data) <- as.character(unlist(exondf[4,]))

# transform data
data$Age <- droplevels(data$Age)
data$Race <- droplevels(data$Race)
data$State <- droplevels(data$State)
data$Crime <- droplevels(data$Crime)
data$Sentence <- droplevels(data$Sentence)
data$Convicted <- droplevels(data$Convicted)
data$Exonerated <- droplevels(data$Exonerated)

data$AgeCNV <- as.numeric(as.character(data$Age))
data$ConvictedCNV <- as.numeric(as.character(data$Convicted))
data$ExoneratedCNV <- as.numeric(as.character(data$Exonerated))

data$AgeCNV_floor <- floor(data$AgeCNV/10)*10
data$ConfinedYrs <- data$ExoneratedCNV - data$ConvictedCNV
data$ConfinedYrs_floor <- floor(data$ConfinedYrs/5)*5

# plot pie charts

LABELS <- c("10-19","20-29","30-39","40-49","50-59","60-69","")
pie(table(data$AgeCNV_floor), labels=LABELS, main="Age Exonerated")

pie(table(data$Race), main="Race")

pie(tail(sort(table(data$State)),10), main="Top 10 States")

LABELS <- c("0-4","5-9","10-14","15-19","20-24","25-29","30-34","35+")
pie(table(data$ConfinedYrs_floor), labels=LABELS, main="Years Confined")

R graphic used for Facebook IPO

Apparently former Facebook intern, Paul Butler,  graphic of the Facebook social network graph is being used for Facebook's IPO.  The social network graphic is featured on Page 7 of the IPO filing.  His graphic was featured on mashable and R-bloggers not too long ago.  The graphic is of Facebook connections between city centers around the world.  Paul used an ingenious method of color transparency and great circle arcs to display the social network graph.

This is just one of the really cool things you can do with R.  Not only is R used as a visual medium but also to calculate the great circle paths.  This is really neat to see R in such a high profile setting.  If you want to learn more about R you can read an IEORTools post about R links for beginners on World Statistics Day.  Also there are many books that you can buy on R programming at the IEORTools Online Store.

Visualizing categorical data in R

I came across an interesting SAS macro that was used for visualizing log odds relationships of data.  This type of chart is helpful for visualizing the relationship between a binary dependent variable and a continuous independent variable.  I don't use SAS on a daily basis as I prefer to use R.  So I got to thinking that I could recreate this macro using only R.  I thought this would be a good tutorial for R on developing functions, using different plot techniques, and overlapping chart types.

The following picture is the result of the logodds function in R.  The chart is really close but not quite exact.  For the histogram points I decided to use the default squares of the stripchart plot and used a grey color to make it look a little faded.

The following is the R script.

logoddsFnc <- function(data_ind, data_dep, ind_varname, min.count=1){

  # Assumptions: x & y are numeric vectors of the same
  # length, y is 0/1 varible.  This returns a vector
  # of breaks of the x variable where each bin has at
  # least min.countnumber of y's
  bin.by.other.count <- function(x, other, min.cnt=1) {
    csum <- cumsum(tapply(other, x, sum))
    breaks <- numeric(0)
 
    i <- 1
    breaks[i] <- as.numeric(names(csum)[1])
    cursum <- csum[1]
 
    for ( a in names(csum) ) {
      if ( csum[a] - cursum >= min.cnt ) {
        i <- i + 1
        breaks[i] <- as.numeric(a)
        cursum <- csum[a]
      }
    }
 
    breaks
  }
 
  brks <- bin.by.other.count(data_ind, data_dep, min.cnt=min.count)
 
  # Visualizing binary categorical data
  var_cut <- cut(data_ind, breaks=brks, include.lowest=T)
  var_mean <- tapply(data_dep, var_cut, mean)
  var_median <- tapply(data_ind, var_cut, median)
 
  mydf <- data.frame(ind=data_ind, dep=data_dep)
  fit <- glm(dep ~ ind, data=mydf, family=binomial())
  pred <- predict(fit, data.frame(ind=min(data_ind):max(data_ind)),
          type="response", se.fit=T)
 
  # Plot
  plot(x=var_median, y=var_mean, ylim=c(0,1.15),
       xlab=ind_varname, ylab="Exp Prob", pch=21, bg="black")
  stripchart(data_ind[data_dep==0], method="stack",
             at=0, add=T, col="grey")
  stripchart(data_ind[data_dep==1], method="stack",
             at=1, add=T, col="grey")
 
  lines(x=min(data_ind):max(data_ind),
        y=pred$fit, col="blue", lwd=2)
  lines(lowess(x=var_median,
               y=var_mean, f=.30), col="red")
 
  lines(x=min(data_ind):max(data_ind),
        y=pred$fit - 1.96*pred$se.fit, lty=2, col="blue")
  lines(x=min(data_ind):max(data_ind),
        y=pred$fit + 1.96*pred$se.fit, lty=2, col="blue")
}



 

logoddsFnc(icu$age, icu$died, "age", min.count=3)



The ICU data  for this example can be found in the R package "vcdExtra".  Special thanks to David of Univ. of Dallas for providing me with a way to develop breaks in the independent variable as seen by the bin.by.other.count function. 

The author of the SAS macro is also the author of Visualizing Categorical Data by M. Friendly which is a great reference for analyzing and visualizing data in factored groups.

R Tutorial: Add confidence intervals to dotchart

Recently I was working on a data visualization project.  I wanted to visualize summary statistics by category of the data.  Specifically I wanted to see a simple dispersion of data with confidence intervals for each category of data. 

R is my tool of choice for data visualization.  My audience was a general audience so I didn't want to use boxplots or other density types of visualization methods.  I wanted a simple mean and 95% (~ roughly 2 standard deviations) confidence around the mean.  My method of choice was to use the dotchart function.  Yet that function is limited to showing the data points and not the dispersion of the data.  So I needed to layer in the confidence intervals. 

The great thing about R is that the functions and objects are pretty much layered.  I can create one R object and add to it as I see fit.  This is mainly true with most plotting functions in R.  I knew that I could use the lines function to add lines to an existing plot.  This method worked great for my simplistic plot and adds another tool to my R toolbox.

Here is the example dotchart with confidence intervals R script using the "mtcars" dataset that is provided with any R installation.

### Create data frame with mean and std dev
x <- data.frame(mean=tapply(mtcars$mpg, list(mtcars$cyl), mean), sd=tapply(mtcars$mpg, list(mtcars$cyl), sd) )

###  Add lower and upper levels of confidence intervals
x$LL <- x$mean-2*x$sd
x$UL <- x$mean+2*x$sd

### plot dotchart with confidence intervals

title <- "MPG by Num. of Cylinders with 95% Confidence Intervals"

dotchart(x$mean, col="blue", xlim=c(floor(min(x$LL)/10)*10, ceiling(max(x$UL)/10)*10), main=title )

for (i in 1:nrow(x)){
    lines(x=c(x$LL[i],x$UL[i]), y=c(i,i))
}
grid()






And here is the example of the finished product.

Open Source replacements for Operations Research and Analytics Software

I was reading an article from Datamation on 70 Open Source Replacements for Small Business when I noticed a glaring omission.  Where are the software applications for Operations Research and Analytics?  So here is my best addendum to this article that should complete what small business should know about Open Source analytics productivity software.

Statistics and Computation

1.  R Project

Replaces: SAS, SPSS

R is a free and open source statistical computing environment that holds its own against some of the most established proprietary statistical environments.  R is available on all operating systems and is free for download.  R also has a community driven library of add-on packages that are also freely available and cover almost any statistical, mathematical, or optimization need.

Also a great reference manual for those switching from SAS to R is SAS and R: Data Management, Statistical Analysis, and Graphics




2.  RapidMiner

Replaces:  KnowledgeSEEKER

RapidMiner is a data mining software with a graphical front-end.  RapidMiner is suitable for most data mining and data transformation needs.


Mathematical Programming and Optimization

3.  GLPK

Replaces:  AMPL

GLPK is a GNU/free software linear programming software kit.  GLPK is intended for large-scale linear programming, mixed integer programming.  GLPK is based on GNU MathProg (or GMPL) which is considered a subset of the AMPL syntax.  GLPK also has its own solver.

4.  Symphony

Replaces: CPLEX, Gurobi

Symphony is a mixed integer linear programming solver developed under COIN-OR.  Symphony is a flexible framework that offers many methods to customize solver capabilities given problem sets.


5.  OpenSolver

Replaces:  Excel Solver

OpenSolver is a linear an integer optimizer alternative to the Excel Solver in Microsoft Excel.  OpenSolver is based on the COIN-OR CBC engine.  Unlike the Excel Solver there is no software limits to the size of the problem that can be solved.

R again in Google Summer of Code

I'm a big fan of the Google Summer of Code.  It brings great projects together with a learning opportunity for students.  Once again the R Project was selected to be part of the Google Summer of Code in 2011.  Some other notable mathematical and statistics projects with R include Shogun Machine Learning, SymPy, Gambit,  Computational Geometry Algorithms Lab, Orange, and Computational Science and Engineering.

The Google Summer of Code has really grown over the years.  I'm glad to see that these open source initiatives really help teach our younger generation. 

IBM has a Natural Language Purpose

I wanted to write a blog post about the advancements of Natural Language Processing in light of the performance of IBM's Watson on the Jeopardy challenge last week.  Natural Language Processing is the science of transforming and interpreting human spoken and written language by artificial means.  Generally this type of study has been limited to academic research due to the high computing power demands.  Now there are even open source software implementations, including many R Natural Language Processing packages.  There is a lot to write about the new advances in NLP.

Instead I came across an interesting editorial on the cheap publicity stunt that is IBM's Watson.  At first I thought the article was a comedy that would make fun of Watson's errors on Jeopardy.  Then I realized the author Colby Cosh is not jesting at all.  This should not be news to me.  The field of Operations Research, which was definitely used to help develop Watson, is a widely misunderstood field.  Cosh has a hard time understanding why IBM would want to develop such a stunt to compete against humans.  Cosh seems to think that the only gain is IBM's shareholders.  I can assure you that if IBM wanted to make money on this venture they would have created a computer that would compete on American Idol.  Jeopardy is no ratings juggernaut in the US.

So what purpose would IBM have for competing on Jeopardy.  Perhaps the idea of "competition" is misleading.  In my eyes I was not seeing if a computer can beat humans in a battle of wits.  I was seeing if a device could interpret, process, and return meaningful information on the same level as human interpretation.  Natural Language Processing is like code breaking.  Similarly mathematics, physics, natural science are like codes to mathematicians, scientists, and engineers.  It is the process of trying to decipher and interpret our natural surroundings.  Language is no different.  I can see it easy for Cosh to think that the sole idea of the competition is to beat humans.  The purpose was simply to decipher the natural language code.  In a better understanding of natural language we can then understand our surroundings a little better.

So why the hype with a computer?

"So why, one might ask, are we still throwing computer power at such tightly delimited tasks,..."

The answer can be found already in the field of Operations Research and Management Science.  Perhaps Cash has purchased a plane ticket in the past few years.  He might have noticed that air transportation has become very affordable due to competitive pricing.  A lot of that is due to optimization and revenue management algorithms in the airline industry.  Perhaps he noticed the increase in quality, service, and price of privatized parcel postage.  The science of better decision making and transportation algorithms have greatly improved supply chain and delivery efficiency.  The list can go on and on.  Artificial Intelligence is probably a poor way of describing computer optimization and machine learning science.  Artificial Intelligence is not going to replace human intelligence but only help improve the human based decisions that we make every day.  IBM has already stated that they wish to improve the medical field with Watson.  Medical diagnosis requires vast amounts of information and Watson can help decipher medical journals, texts, and resources within seconds.  Applications of Watson could be used in third world countries where medical resources are scarce.

I will be looking forward to IBM's advancement with Natural Language Processing.  This offers a new venture into better decision sciences.  Perhaps "smacking into the limits" of artificial intelligence will create a better life for those that use human intelligence every day.

Video of Joy of Stats by Hans Rosling

The Joy of Stats narrated by Hans Rosling was just produced by BBC and shown to their audience.  Hans Rosling via gapminder.org was kind enough to post the full hour video of the documentary about joys of statistics.  The video is posted on YouTube and is available to anyone.

http://www.gapminder.org/videos/the-joy-of-stats/

Hans Rosling's passion for statistics is infectious.  He definitely has a joy about him that persuades the viewer to really enjoy finding new and invigorating ways to explore data.  Now for me this is not hard to do as I love data and analyzing.  Yet for many in the world mathematics, let alone statistics, is considered a universe all unto its own that they dare not search.  Hans breaks down that barrier with The Joy of Stats.  No matter your educational interests or background I find it very hard to ignore his plea that statistics is not boring, and dare I say it, sexy. 

If you are interested in this video as a eulogy to statistics you would also enjoy Dr. Robert Lewis's essay on Mathematics.  Both of these works explain how a world without number analysis is merely a world not worth living.  There is so much to explore in so little time.  I am so happy that I decided to take a career in Engineering and Operations Research to help the world one datum at a time.

Where to find good data sets

O'Reilly Media has been a big advocate of Open Data and believes that is where a lot of computing is going to be headed in the future.  I think they are definitely on to something.  Yet the future could be now.  There is a lot of opportunities to find good data sources immediately.  One of my favorite blogs, OReilly Radar, has an article by Edd Dumbill on Where To Find Data.  There is plenty of good data available on the internet for download to explore and mine new information.  These places not only offer great sources of data but many of them offer an API to allow quick and seamless access.  Below is a link summary from the article.

Freebase

An all-things graph database.  The website focuses on trends of certain cultural and interest topics.

Amazon Public Data Sets

Amazon is probably considered the cloud computing mecca next to Google.  Amazon Web Services offers a lot.  One of which is storage of public data sets.  They offer a huge variety of public data.

Windows Azure Data Marketplace

Surprisingly Microsoft has an open data protocol data source.  This data market offers quite a few points of interest data sets.

Yahoo Query Language

YQL is an interesting API that is very similar to SQL.  YQL is essentially a language that allows to grab data from cloud services.  This could be very handy to grabbing data quickly and dynamically.  YQL offers to connect to a lot of data sources as well.

Infochimps

Infochimps is a data marketplace warehouse.  They offer to host, sell, and distribute data sets.  Some of their data comes at a cost but a lot of their data is free as well.  This is an interesting startup and will be very interesting to follow their growth.  Also there is a new Infochimps R package that uses their API to gather data and process Infochimps data.

DBpedia

DBpedia is a wikipedia for data sets.  In fact the data itself comes from Wikipedia. 


Some other sources not from the article include the World Bank open data and the U.S. Census data.

Big Data Logistic Regression with R and ODBC

Recently I've been doing a lot of work with predictive models using logistic regression.  Logistic regression is great for determing probable outcomes of a independent binary target variable.  R is a great tool for accomplishing this task.  Often times I will use the base function glm to develop a model.  Yet there are times, due to the hardware or software memory restrictions, that the usual glm function is not enough get the job done.

A great alternative to performing usual logistic regression analyses on big data is using the biglm package.  Biglm performs the same regression optimization but processes the data in "chunks" at a time.  This allows R to only perform calculations on smaller data sets without the need for large memory allocations to the computer.  Biglm also has an interesting option that it not only can perform calculations on imported dataframes and text files but also database connectivity.  This is where the helpful package RODBC comes in to the aid.

I have be looking all over the R support lists and blogs in hopes of finding a good tutorial using biglm and RODBC.  I was not successful yet I was able to find out how to perform this myself.

INFORMS Data Mining Competition leaders used Open Source software

The results of 2010 INFORMS data mining competition just recently finished.  The leaders were presented at the 2010 Annual INFORMS Conference.  The 2010 INFORMS data mining competition goal was to determine short term movements in stock prices.  You may recall that IEOR Tools competed in this competition with not too glamorous results at the end.  There was a lot to learn from this competition.  Firstly that it seems trading price movement can be correlated to lags in prices very well.  Most of the top leaderboard finishers used future information to determine an appropriate lag in the price movement.

The next most interesting thing is that all top 3 finishers used free and open source software as tools for the competition.  Two of the leaders used R and the second place finisher used Python, namely SciPy.  This should not be surprising to most people in the analytics community.  Open source software has been making inroads for quite a while.  The R-Project has been getting a lot of interesting press lately especially in enterprise business circles.  Python is an objective oriented programming language that is getting more popular.  Python's popularity seems to be to its ease of use and how quickly it can be learned and implemented.

The presenters of the 2010 INFORMS data mining competition were kind to post the methods of the Top 3 competitors.  Each method is an interesting read on how they were able to use the open source tools to get predictive results of stock price movements.

If you are interesting in learning more about R as a tool I recommend a new book by Luis Torgo "Data Mining with R: Learning with Case Studies".

This book is one of the first in its kind of showing R methodologies with real life applications.  I am intending to get the book and hopefully have a review on it in the near future.  I am already hearing good things about it.

R references for handling Big data

The Dallas R User Group had a meeting over the weekend.  One of the discussions is the memory limitations with R.  This is a common subject among the R community and R User Groups.  There has been a lot of strides recently in allowing R to stretch its memory limitations.  I thought I would compile and share some of the best resources I have found to remedy the big data issue.

CRAN Packages
ff
This package allocates hard disk space to big data vectors.

bigmemory
This package allocates points to unused memory or points to a swap file.

Blog Articles
Taking R to the Limit:  Parallelism and Big Data

Hitting the Big DataCeiling Limit in R
While this is not a helpful article for big data it does show some of the issues R current faces.  Namely the issue of that lack of a "int64" or Long Long data type memory allocation.

Enterprise Software
Revolution R Enterprise
Revolution Analytics is creating enterprise software around R to tackle issues of big data, parallelism and threaded computing in order to speed up large data processing and analytics.