Top 50 Statistics blogs of 2011

TheBestColleges.org published their list of the top 50 statistics blogs.  This is a really good resource list of statistical analysis and news.

Machine Learning for everyone

Well maybe mostly everyone. Have  you been interested in gaining knowledge in the latest craze of artificial intelligence and computing?  Then go no further than Stanford's Machine Learning course which is now open enrollment to everyone!  Andrew Ng is back to provide the world with knowledge about Machine Learning for the entire masses. 

Per Stanford's website, Machine Learning is data mining and statistical pattern recognition.  Mostly it is applying mathematical and statistical methods to draw out information behaviors from data sources.  So do you want to invent the next Netflix, Amazon or Google?  This is the course for you.

If you do not want to enroll in the Machine Learning class you could always watch some of the older lectures online.  Andrew Ng provides plenty of information from past lectures with student contributed projects.  The CS 229 website is worth a look for a punch of Machine Learning related resources.

Problems with data visualizations followup

On a recent article post I was showing a bad data visualization chart from The Economist.  While reading over Slashdot I found a similar bad data visualization article about bad visualizations from BP and GE as presented by Stephen Few's blog.  No doubt a lot of people share in the same frustration.

Now that we are in the Insight Age it seems that we will continually question and interpret how data will be presented to us.  We are now data rich but knowledge poor.  I believe there is going to be vast new opportunities to help disseminate the data.  Perhaps even ways to help visualize the data as well. 

I strongly suggest reading Stephen Few's blog.  It is an interesting read on how data visualization can be used poorly.  He even shows examples on how to do it correctly.

How not to do data visualization

I was glancing over Hacker News and came across an article from the Economist Daily Chart blog.  The daily chart was about nations debt management.  The following can be shown here.

Seems innocent enough.  It shows in declining order the debt per nation.  What a second?  Why is Ireland have more debt than USA? After reading the article more thoroughly it looks like it is a percentage of GDP.  What a third time?  Is the bar graph the percentage of GDP or the number in the white box a percentage of GDP?  And how does this relate to debt management?  So apparently in the article it explains the change in primary balance for each nation to be 60% of GDP.  So the bar graph is a % change of GDP to get to 60% of GDP.  Are we crystal?  I'm not sure I totally understand but that is my basic understanding.

Data visualization is important in Analytics and Operations Research.  We need to model real world applications quite a lot.  Often times there is no better way to do this than to use a chart or graph.  The real art is conveying the crux of the message to the recipient.  There is an internet meme devoted to the art of bad chart making.  I feel bad using the Economist as an example because after all I did finally (I think) come away with the right idea.  But still notice how there are no data or axis labels across the top of the chart.  Also the numbers in the white boxes are not given any units.  I'm still not sure if those numbers in the white box are a percentage or a debt value.  Sometimes the visual art clutters the real message.  It is important to make sure that recipient has the right frame of reference and can understand each graphic and label.

Moneyball coming to the big screen

Recently I found out that the book Moneyball by Michael Lewis will be shown as a motion picture.  The Moneyball trailer can be viewed online.  In case you have never heard of Michael Lewis then you might have heard about the movie "The Blind Side" which he also wrote the accompanying book.  The book Moneyball is about the Oakland Athletics and how they used analytics and mathematcal know-how to turn around a professional baseball franchise.

The story centers around Billy Beane which is played by Brad Pitt in the movie.  Billy Beane is a professional ballplayer turned General Manager.  Billy Beane inherits the top organizational management job for the losing Oakland Athletics.  He is immediately frustrated with the same old losing ways and believes he needs to shake up the system.  He finds out about the curious world of baseball analytics or otherwise know as sabermetrics and hires a curious crew of young mathematically gifted folks.

The story is fascinating even if you are not a fan of baseball.  The use of mathematics to help make business decisions is nothing new.  Yet employing this analytics method to an industry that is deep rooted in old ways and practices is intriguing.  Changing the ways of the "good ole boy" network requires risk, knowledge, and sometimes good fortune.  This can translate to almost any industry or even organization.  I am most definitely looking forward to seeing this movie.

R Tutorial: Add confidence intervals to dotchart

Recently I was working on a data visualization project.  I wanted to visualize summary statistics by category of the data.  Specifically I wanted to see a simple dispersion of data with confidence intervals for each category of data. 

R is my tool of choice for data visualization.  My audience was a general audience so I didn't want to use boxplots or other density types of visualization methods.  I wanted a simple mean and 95% (~ roughly 2 standard deviations) confidence around the mean.  My method of choice was to use the dotchart function.  Yet that function is limited to showing the data points and not the dispersion of the data.  So I needed to layer in the confidence intervals. 

The great thing about R is that the functions and objects are pretty much layered.  I can create one R object and add to it as I see fit.  This is mainly true with most plotting functions in R.  I knew that I could use the lines function to add lines to an existing plot.  This method worked great for my simplistic plot and adds another tool to my R toolbox.

Here is the example dotchart with confidence intervals R script using the "mtcars" dataset that is provided with any R installation.

### Create data frame with mean and std dev
x <- data.frame(mean=tapply(mtcars$mpg, list(mtcars$cyl), mean), sd=tapply(mtcars$mpg, list(mtcars$cyl), sd) )

###  Add lower and upper levels of confidence intervals
x$LL <- x$mean-2*x$sd
x$UL <- x$mean+2*x$sd

### plot dotchart with confidence intervals

title <- "MPG by Num. of Cylinders with 95% Confidence Intervals"

dotchart(x$mean, col="blue", xlim=c(floor(min(x$LL)/10)*10, ceiling(max(x$UL)/10)*10), main=title )

for (i in 1:nrow(x)){
    lines(x=c(x$LL[i],x$UL[i]), y=c(i,i))
}
grid()






And here is the example of the finished product.

Open Source replacements for Operations Research and Analytics Software

I was reading an article from Datamation on 70 Open Source Replacements for Small Business when I noticed a glaring omission.  Where are the software applications for Operations Research and Analytics?  So here is my best addendum to this article that should complete what small business should know about Open Source analytics productivity software.

Statistics and Computation

1.  R Project

Replaces: SAS, SPSS

R is a free and open source statistical computing environment that holds its own against some of the most established proprietary statistical environments.  R is available on all operating systems and is free for download.  R also has a community driven library of add-on packages that are also freely available and cover almost any statistical, mathematical, or optimization need.

Also a great reference manual for those switching from SAS to R is SAS and R: Data Management, Statistical Analysis, and Graphics




2.  RapidMiner

Replaces:  KnowledgeSEEKER

RapidMiner is a data mining software with a graphical front-end.  RapidMiner is suitable for most data mining and data transformation needs.


Mathematical Programming and Optimization

3.  GLPK

Replaces:  AMPL

GLPK is a GNU/free software linear programming software kit.  GLPK is intended for large-scale linear programming, mixed integer programming.  GLPK is based on GNU MathProg (or GMPL) which is considered a subset of the AMPL syntax.  GLPK also has its own solver.

4.  Symphony

Replaces: CPLEX, Gurobi

Symphony is a mixed integer linear programming solver developed under COIN-OR.  Symphony is a flexible framework that offers many methods to customize solver capabilities given problem sets.


5.  OpenSolver

Replaces:  Excel Solver

OpenSolver is a linear an integer optimizer alternative to the Excel Solver in Microsoft Excel.  OpenSolver is based on the COIN-OR CBC engine.  Unlike the Excel Solver there is no software limits to the size of the problem that can be solved.

Baseball and Decision Analytics

When its spring time it means that baseball season is getting ready to get started.  On the last day of March is when the 2011 Major League Baseball season gets going.  Baseball is as American as apple pie and almost every baseball enthusiast has something to say about the game.  Analytics professionals are not far behind when it comes to opinions on baseball. 

Baseball is definitely a numbers game.  Mathematicians have been studying baseball for as long as the game itself has been played.  One of the first notable baseball analysts to apply decision analysis was Bill James.  Bill coined the study of baseball analysis as sabermetrics which is taken from the acronym of the Society of American Baseball Research.  More recently baseball decision analysis has found its way to the Major League Baseball teams management offices.  Popular books such as Moneyball by Michael Lewis and The Extra 2%: How Wall Street Strategies Took a Major League Baseball Team from Worst to First by Jonah Keri have shown how major league management turned around poor performaning clubs into championship contenders.  The mathematics behind their decision analysis can be described best by Wayne Winston's book called Mathletics: How Gamblers, Managers, and Sports Enthusiasts Use Mathematics in Baseball, Basketball, and Football.

Baseball decision analysis has grown up since Bill James devised the batting average.  Now baseball decision analysis uses techniques such as replacement value.  The Value Over Replacement determines the value of a player given that player would be replaced by an average or run-of-the-mill at the given player's position.  Value Over Replacement was made popular by Keith Woolner, the author of the Baseball Prospectus 2011.  At first the value, which is usually offensive value, was to determine how many runs a player could produce over an average player.  Now value over replacement methodologies determine how many wins a player can generate for their respective team.  One of the best sites to give WAR analysis, or Wins Over Replacement, is Fangraphs.  Fangraphs has about every major statistic on baseball available for the baseball enthusiast.  In fact they even have heat maps for pitch location.  Ready to manage your own team yet?

Pitch location heat map from Fangraphs.com

Of course all of this decision analysis would not be possible without the numbers.  One of the best places for baseball data is Baseball-Reference.com.  Just about every data point on baseball can be mined from the site and downloaded.  So if you have a craving to create your own baseball metric or analytics strategy there should be nothing stopping you.

This is another post in the INFORMS Online Blog Challenge.  This month is O.R. and Sports. 

R again in Google Summer of Code

I'm a big fan of the Google Summer of Code.  It brings great projects together with a learning opportunity for students.  Once again the R Project was selected to be part of the Google Summer of Code in 2011.  Some other notable mathematical and statistics projects with R include Shogun Machine Learning, SymPy, Gambit,  Computational Geometry Algorithms Lab, Orange, and Computational Science and Engineering.

The Google Summer of Code has really grown over the years.  I'm glad to see that these open source initiatives really help teach our younger generation. 

Statistics blog roundup from the Internet

For those of us that want to keep up with the latest in the world of statistics here is a great list of 40 statistical blogs on the internet today from bschool.com.  Some of the blogs even include R (link) which is my favorite statistical computing environment. 

I have a few that I read on an occasional basis.   Those include

R-Bloggers - an R blog aggregator that is very useful for R enthusiasists

Revolutions - blog by David Smith of Revolution Analytics, an enterprise ready R support and developer company

The Numbers Guy - Wall Street Journal blog about, you guessed it, numbers and statistics.

As you can tell I have been doing a lot of statistical work lately.  Most of my efforts have been in the realm of predictive analytics.  These blogs have come in useful in finding valuable references for predictive analytics.  One book that is often mentioned in these statistical blogs is Applied logistic regression (Wiley Series in probability and statistics).  I highly recommend this book as a statistical reference for logistic regression analysis.

Applied Logistic Regression takes the reader through an understanding logistic regression model and its usage for modern prediction analysis.  The book starts with and understanding of log odds and introduction to the logistic regression model.  Then the book goes into detail of the model parameters, coefficients, and estimation.  Strategies for model building and case studies are also referred in this book.

Video of Joy of Stats by Hans Rosling

The Joy of Stats narrated by Hans Rosling was just produced by BBC and shown to their audience.  Hans Rosling via gapminder.org was kind enough to post the full hour video of the documentary about joys of statistics.  The video is posted on YouTube and is available to anyone.

http://www.gapminder.org/videos/the-joy-of-stats/

Hans Rosling's passion for statistics is infectious.  He definitely has a joy about him that persuades the viewer to really enjoy finding new and invigorating ways to explore data.  Now for me this is not hard to do as I love data and analyzing.  Yet for many in the world mathematics, let alone statistics, is considered a universe all unto its own that they dare not search.  Hans breaks down that barrier with The Joy of Stats.  No matter your educational interests or background I find it very hard to ignore his plea that statistics is not boring, and dare I say it, sexy. 

If you are interested in this video as a eulogy to statistics you would also enjoy Dr. Robert Lewis's essay on Mathematics.  Both of these works explain how a world without number analysis is merely a world not worth living.  There is so much to explore in so little time.  I am so happy that I decided to take a career in Engineering and Operations Research to help the world one datum at a time.

Where to find good data sets

O'Reilly Media has been a big advocate of Open Data and believes that is where a lot of computing is going to be headed in the future.  I think they are definitely on to something.  Yet the future could be now.  There is a lot of opportunities to find good data sources immediately.  One of my favorite blogs, OReilly Radar, has an article by Edd Dumbill on Where To Find Data.  There is plenty of good data available on the internet for download to explore and mine new information.  These places not only offer great sources of data but many of them offer an API to allow quick and seamless access.  Below is a link summary from the article.

Freebase

An all-things graph database.  The website focuses on trends of certain cultural and interest topics.

Amazon Public Data Sets

Amazon is probably considered the cloud computing mecca next to Google.  Amazon Web Services offers a lot.  One of which is storage of public data sets.  They offer a huge variety of public data.

Windows Azure Data Marketplace

Surprisingly Microsoft has an open data protocol data source.  This data market offers quite a few points of interest data sets.

Yahoo Query Language

YQL is an interesting API that is very similar to SQL.  YQL is essentially a language that allows to grab data from cloud services.  This could be very handy to grabbing data quickly and dynamically.  YQL offers to connect to a lot of data sources as well.

Infochimps

Infochimps is a data marketplace warehouse.  They offer to host, sell, and distribute data sets.  Some of their data comes at a cost but a lot of their data is free as well.  This is an interesting startup and will be very interesting to follow their growth.  Also there is a new Infochimps R package that uses their API to gather data and process Infochimps data.

DBpedia

DBpedia is a wikipedia for data sets.  In fact the data itself comes from Wikipedia. 


Some other sources not from the article include the World Bank open data and the U.S. Census data.

Big Data Logistic Regression with R and ODBC

Recently I've been doing a lot of work with predictive models using logistic regression.  Logistic regression is great for determing probable outcomes of a independent binary target variable.  R is a great tool for accomplishing this task.  Often times I will use the base function glm to develop a model.  Yet there are times, due to the hardware or software memory restrictions, that the usual glm function is not enough get the job done.

A great alternative to performing usual logistic regression analyses on big data is using the biglm package.  Biglm performs the same regression optimization but processes the data in "chunks" at a time.  This allows R to only perform calculations on smaller data sets without the need for large memory allocations to the computer.  Biglm also has an interesting option that it not only can perform calculations on imported dataframes and text files but also database connectivity.  This is where the helpful package RODBC comes in to the aid.

I have be looking all over the R support lists and blogs in hopes of finding a good tutorial using biglm and RODBC.  I was not successful yet I was able to find out how to perform this myself.

R references for handling Big data

The Dallas R User Group had a meeting over the weekend.  One of the discussions is the memory limitations with R.  This is a common subject among the R community and R User Groups.  There has been a lot of strides recently in allowing R to stretch its memory limitations.  I thought I would compile and share some of the best resources I have found to remedy the big data issue.

CRAN Packages
ff
This package allocates hard disk space to big data vectors.

bigmemory
This package allocates points to unused memory or points to a swap file.

Blog Articles
Taking R to the Limit:  Parallelism and Big Data

Hitting the Big DataCeiling Limit in R
While this is not a helpful article for big data it does show some of the issues R current faces.  Namely the issue of that lack of a "int64" or Long Long data type memory allocation.

Enterprise Software
Revolution R Enterprise
Revolution Analytics is creating enterprise software around R to tackle issues of big data, parallelism and threaded computing in order to speed up large data processing and analytics.

R Links for the Beginner on World Statistics Day

In honor of the first World Statistics Day I thought I would share some of my favorite R links.  R is a free software statistical computing environment for performing all sorts of data and mathematical manipulation.

Introduction and Tutorials

R Tutorial Series and Introduction
Burns Statistics Tutorials
Introductory R Tutorials
Learning R

Videos about R

Videos on Data Analysis with R

Graphics with R

Advanced Graphics
R Graph Gallery

Data mining competition with R

There is a new data mining competition aimed at predicting preferred data mining tools in R via dataists.com.   The concept of the competition is to try to determine which R packages are preferred in the R community via their CRAN package libraries.  The developers of this new competition are also in the R community with the NY R Users Group.

I am a user of R and I am also a member of the Dallas R Users Group.  As you can imagine I find this competition very interesting because I could benefit greatly from knowing the preferred methods of implementation in the R community.  This can also be a very interesting exercise to determine preferred modeling methods.  I believe this competition will give an insight into the most common methods to apply statistical computing in the community today.

R is getting a lot of press lately.  Revolution Analytics just released the first part of a series of articles on the impacts of R and why it is hot.  R as a statistical and optimization tool is really making a play in the business community.

I am a big fan of these data mining competitions as I've written previously.  It is really interesting what you can learn about the world from the competitions especially if you do not have previous knowledge of the subject matter.  I will try to bring up these competitions from time to time.  I would be interested to know if any IEOR Tools readers have participated in these competitions and what was their experience.

Kaggle introduces new Chess rating competition

Kaggle, home of the statistics and predictive modeling competitions, is introducing its latest contest Elo Versus The Rest of The World.  The competition is being organized by Jeff Sonas who is a chess-metrics afficionado himself.  Jeff describes his history with rating chess players and why he wanted to start such a competition with Kaggle.

This looks to be a really interesting modeling competition with already more than 40 submissions in the leaderboard.  The interesting note about this competition is that the Elo rating system itself is going to be making an appearance on the leaderboard.  This means that if no one beats the Elo system than there is no declared winner.  Although it looks like someone has beaten Elo at its game already.  Elo will be on the leaderboard as a benchmark to make sure that the competition is proving its worth.

I hope to get a chance to make an appearance on the leaderboard.  I am involved in Kaggle's INFORMS 2010 Data Mining contest.  I'm barely hanging on to the top 10 in that competition.  There are some pretty good models to compete against in that group. 

R IDE for Linux and Gnome

I have been using R in my work recently.  I have also been using R at home to do some tinkering.  In my work environment I use Windows (none too pleased).  I find using the regular R console with Textpad makes for a good Windows development environment.  I haven't been able to replicate this at my home.  At my home I have Ubuntu as my operating system.  I have been searching for a comparable R environment for my home.

That is until now.  The statistics blog at Stattler.com did some research on this very topic of R and Ubuntu.  They found a plugin for R with the text editor Gedit that works wonders.  The plugin is called Rgedit and is very easy to install.  Stattler offers a simple instruction for installation.  Also Stattler has a great review of the Rgedit plugin. Rgedit is very similar in layout to usual gedit text editor except it splits the panes of the screen for code and R output.

Some of the highlights of the Rgedit plugin include:

• Split screen of panes and can be turned on and off
• Syntax highlighting specific to the R code
• Single line or batch processing of R scripts
• Multiple R workspaces can be run
• Shortcut keys can be created and customized

This plugin suits my needs just fine for my Ubuntu uses with R.  There are many other IDEs for R that you may find suit your needs better.  The the beauty of open source software is that there never seems to be a shortage of options.

Audio of Richard Stallman keynote at useR2010

useR 2010 Conference

The topic of open source and free software licensing can be a very confusing topic.  In the Operations Research world it is no different.  There are a lot of players in the mathematical programming software world that are vying for attention that include both proprietary and free software.  Insight into the world of free software really requires immersion into using the free software products and finding how it can apply to daily application.  Another good way to understand free software is to get it from the founders of the movement.  Richard Stallman is considered the father of the free software movement and you can find a lot of good material online based on his work.  The useR2010 conference, the annual conference for R project for statistical computing, just completed this past week and the final keynote was given by Richard Stallman.

The R-statistics blog was kind enough to post an audio of the keynote address by Richard Stallman at useR2010.  Richard is not your typical stereotype of a computer geek.  He may look the part but Richard does not pull any punches in his presentation of the free software movement and it's ideology.  Richard's talk discusses the history of the free software movement, the Gnu Public License, and his history of dealing with free software. 

Why was Richard giving the keynote address when he doesn't have a statistics background?  Well the R statistical computing software platform is licensed under the GPL, Gnu Public License.  R is free to use, distrubute, modify and improve as long as it's code is given credit to it's creator.  This is much of what the GPL represents.  Listen to the audio by Richard to really understand his passion for free software and what it means to him and the software world.

Podcast with Revloutions Analytics CEO Norman Nie

Through the Revolutions blog there is a really interesting podcast about R by Internet Evolution Radio interviewing Norman Nie.  Norman Nie is the CEO of Revolutions Analytics which I posted about in the past on how Revelotion Analytics is going to take R commercial.

In this podcast Norman is asked a lot of interesting questions about R and the statistical modeling enterprise in general.  They discuss his past with SPSS.  They also discuss the advantages of using an Open Source software versus a proprietary platform.  The interview gets really interesting when they discuss how statistical data is important to enterprise business and how a lot of organizations get it wrong.

If you are new to R and want to know more about its capabilities this is a great podcast.