Statistical Analysis and Data Mining hot on LinkedIn for 2014

Statistical Analysis and Data Mining are considered the hottest skills on LinkedIn for 2014.  According to their report from analyzing jobs and recruiters on the LinkedIn website.  I would say its safe to say that it will continue to be hot for 2015.

If you are looking to get your skills honed up I would suggest looking at ComputerWorld's Beginners Guide to R.  It looks like a complete tutorial and is indexed rather well.

Data mining the Federal Reserve

The Federal Reserve now has the ability to have its data programmatically retrieved.  The St. Louis Fed Web Services allows programmers and data scientists to retrieve key economic data from their libraries.  I have not had a chance to peruse the site at all but this can be a really interesting source of data.  The age of Open Data is really upon us.  This can lead to some really interesting research for professionals and amateur scientists.

Machine Learning for everyone

Well maybe mostly everyone. Have  you been interested in gaining knowledge in the latest craze of artificial intelligence and computing?  Then go no further than Stanford's Machine Learning course which is now open enrollment to everyone!  Andrew Ng is back to provide the world with knowledge about Machine Learning for the entire masses. 

Per Stanford's website, Machine Learning is data mining and statistical pattern recognition.  Mostly it is applying mathematical and statistical methods to draw out information behaviors from data sources.  So do you want to invent the next Netflix, Amazon or Google?  This is the course for you.

If you do not want to enroll in the Machine Learning class you could always watch some of the older lectures online.  Andrew Ng provides plenty of information from past lectures with student contributed projects.  The CS 229 website is worth a look for a punch of Machine Learning related resources.

IBM has a Natural Language Purpose

I wanted to write a blog post about the advancements of Natural Language Processing in light of the performance of IBM's Watson on the Jeopardy challenge last week.  Natural Language Processing is the science of transforming and interpreting human spoken and written language by artificial means.  Generally this type of study has been limited to academic research due to the high computing power demands.  Now there are even open source software implementations, including many R Natural Language Processing packages.  There is a lot to write about the new advances in NLP.

Instead I came across an interesting editorial on the cheap publicity stunt that is IBM's Watson.  At first I thought the article was a comedy that would make fun of Watson's errors on Jeopardy.  Then I realized the author Colby Cosh is not jesting at all.  This should not be news to me.  The field of Operations Research, which was definitely used to help develop Watson, is a widely misunderstood field.  Cosh has a hard time understanding why IBM would want to develop such a stunt to compete against humans.  Cosh seems to think that the only gain is IBM's shareholders.  I can assure you that if IBM wanted to make money on this venture they would have created a computer that would compete on American Idol.  Jeopardy is no ratings juggernaut in the US.

So what purpose would IBM have for competing on Jeopardy.  Perhaps the idea of "competition" is misleading.  In my eyes I was not seeing if a computer can beat humans in a battle of wits.  I was seeing if a device could interpret, process, and return meaningful information on the same level as human interpretation.  Natural Language Processing is like code breaking.  Similarly mathematics, physics, natural science are like codes to mathematicians, scientists, and engineers.  It is the process of trying to decipher and interpret our natural surroundings.  Language is no different.  I can see it easy for Cosh to think that the sole idea of the competition is to beat humans.  The purpose was simply to decipher the natural language code.  In a better understanding of natural language we can then understand our surroundings a little better.

So why the hype with a computer?

"So why, one might ask, are we still throwing computer power at such tightly delimited tasks,..."

The answer can be found already in the field of Operations Research and Management Science.  Perhaps Cash has purchased a plane ticket in the past few years.  He might have noticed that air transportation has become very affordable due to competitive pricing.  A lot of that is due to optimization and revenue management algorithms in the airline industry.  Perhaps he noticed the increase in quality, service, and price of privatized parcel postage.  The science of better decision making and transportation algorithms have greatly improved supply chain and delivery efficiency.  The list can go on and on.  Artificial Intelligence is probably a poor way of describing computer optimization and machine learning science.  Artificial Intelligence is not going to replace human intelligence but only help improve the human based decisions that we make every day.  IBM has already stated that they wish to improve the medical field with Watson.  Medical diagnosis requires vast amounts of information and Watson can help decipher medical journals, texts, and resources within seconds.  Applications of Watson could be used in third world countries where medical resources are scarce.

I will be looking forward to IBM's advancement with Natural Language Processing.  This offers a new venture into better decision sciences.  Perhaps "smacking into the limits" of artificial intelligence will create a better life for those that use human intelligence every day.

Science of Matchmaking

The science of matchmaking has seen serious growth in the last few years.  What exactly is so scientific about matchmaking anyway?  The goal of any commercial enterprise (and some public organizations) is to match products or services to the demand of consumers.  The idea of matching consumers with products and services is not new.  Matchmaking is essentially the business art of Marketing.  The science behind the matchmaking has seen the most advancement and improvement in recent time.  Generally speaking computing power has made the difference for the technologically leap forward.  Millions of points of observations and data can be sifted and combed with great ease as compared to even just a decade ago.  There is a scientific magic fiddling to the matchmaking phenomenon (sorry, bad Fiddler On The Roof pun). 


Mathematics of Matchmaking

I'm not sure I can cover all of the math behind the science of matchmaking.  I thought it best to describe an example with the company Netflix.  Netflix wants to make the decision process of selecting movies for its customers easier.  Netflix developed an algorithm to match customers' interests in movies.  In fact they even decided to farm out an improvement to the algorithm in a worldwide contest.  So how does the Netflix algorithm work?  There is a lot of math behind the algorithm but it essentially comes down to finding common features in the customer and movie data.  The customers give Netflix a clue to the features they want by ranking movies the customers enjoy.  This then becomes the dependent variables in the algorithm formulation.  Then the algorithms churn out likely matches based on common feature sets. 

Perhaps one of the best writings on this subject was given by Simon Funk on his blog about his Netflix Contest adventures.  Simon thought a creative way to find features would be to use the matrix transformation process of Singular Value Decomposition.  Traditionally Singular Value Decomposition was used in the microelectronics industry to improve digital signal processing.  Simon wrote up an easy solution for matchmaking movie features with the SVD method which spurned a wave of enthusiasm in the Netflix Contest entrants.


Finding feature sets is not exclusively in the realm of Linear Algebra.  There are also methods of clustering, regression, support vector machine, neural networks, bayesian networks, and decision trees just to name a few.  The science of matchmaking is closely related to artificial intelligence and is commonly referred to machine learning.  Machine learning is using algorithms and mathematical methods to evolve and generate behaviors from data in order to solve a problem.


Processing the Matchmaking Data

The science of matchmaking would not be complete without the data.  The advent of the internet has opened a lot of new enterprises that makes use of millions of data observations.  These internet companies have a lot of data to process in huge server arrays that will make even the ENIAC envious.  So how do these companies process all of this matchmaking data with their matchmaking algorithms?  The basic answer is to break it down into manageable chunks.  Perhaps no greater example is Google and their MapReduce methods.  MapReduce is a software framework process that takes a large computing need and breaks it down into a distributed network that is more manageable.  The first step in the MapReduce process is to Map the data.  The Map process is to organize and distribute the data to computing nodes, usually a huge cluster.  The Reduce process is to apply the algorithm or learning process to a node in the network and determine an answer to the data its given.  This essentially gives it a local optimum.  This process is iterated until a globally learned optimum is achieved.  This is a very cut and dry description but you get the idea.


The MapReduce software framework is proprietary to Google.  That has not stopped software enthusiasts.  An open source MapReduce method was created called Hadoop and is growing into a stronger user supported community.


So what can be used with the science of matchmaking?  Really anything the heart desires (okay, again, that was bad).  Amazon.com uses recommendation algorithms for its books and products.   Online dating sites (how appropriate) uses matchmaking methods for matching interested daters.  Search engines like Google uses matchmaking algorithms, known as PageRank, to match search keywords with websites.  As you can tell these types of enterprises are doing very well thanks to the science of matchmaking.



This article is part of the INFORMS Online blog challenge.  February's blog challenge is Operations Research and Love.

Data Mining Books List

I came across a great list of Data Mining Books while perusing around the internet.  The list is maintained by Kurt Thearling who is Director and Chief Scientist in various organizations helping to develop their Analytics and Engineering groups.  Kurt has written some white papers on the subject of Data Mining and has also been featured on NPR.  Kurt's NPR piece was about data mining and privacy which is obviously a big subject in our Facebook society today.

I believe this is probably one the the most comprehensive lists of Data Mining books available.  If you are interested in obtaining one of these books please be sure to peruse the the new IEOR Tools Online Store Data Mining section.  There you can find books and references on Data Mining with varying levels from introductory to advanced applications.

Video of Joy of Stats by Hans Rosling

The Joy of Stats narrated by Hans Rosling was just produced by BBC and shown to their audience.  Hans Rosling via gapminder.org was kind enough to post the full hour video of the documentary about joys of statistics.  The video is posted on YouTube and is available to anyone.

http://www.gapminder.org/videos/the-joy-of-stats/

Hans Rosling's passion for statistics is infectious.  He definitely has a joy about him that persuades the viewer to really enjoy finding new and invigorating ways to explore data.  Now for me this is not hard to do as I love data and analyzing.  Yet for many in the world mathematics, let alone statistics, is considered a universe all unto its own that they dare not search.  Hans breaks down that barrier with The Joy of Stats.  No matter your educational interests or background I find it very hard to ignore his plea that statistics is not boring, and dare I say it, sexy. 

If you are interested in this video as a eulogy to statistics you would also enjoy Dr. Robert Lewis's essay on Mathematics.  Both of these works explain how a world without number analysis is merely a world not worth living.  There is so much to explore in so little time.  I am so happy that I decided to take a career in Engineering and Operations Research to help the world one datum at a time.

2 years of blogging with IEOR Tools

I forgot I was going to make a mention but on Nov. 21 was officially two years of blogging about Industrial Engineering and Operations Research Tools.  I have really enjoyed writing about this space and reading all of the contributions.  I have no intentions of quitting and hope to make many more contributions.

An update to the blog is that I'm starting to contribute Amazon content to the site.  Amazon has been a valuable resource for linking books on content matter.  I've thought about adding a website that will be a "store" or compilation of some of the better resources with Amazon being a partner.  I thought I would bring this up with the readers first to see if this would be a valuable addition to this blog.  It would be a clearinghouse or aggregator for all the best tools and resources in Operations Research, Industrial Engineering, Analytics and Data Mining.  I'm not sure there is anything on the internet besides doing searches in Google or Amazon.  I hope the site would be nice layout to help easily find resources.

Since it is the holiday season I would like to send my warmest regards to all those reading.  I thank you so much for your readership.  I wish you and your family a safe and happy holidays. 

INFORMS Data Mining Competition leaders used Open Source software

The results of 2010 INFORMS data mining competition just recently finished.  The leaders were presented at the 2010 Annual INFORMS Conference.  The 2010 INFORMS data mining competition goal was to determine short term movements in stock prices.  You may recall that IEOR Tools competed in this competition with not too glamorous results at the end.  There was a lot to learn from this competition.  Firstly that it seems trading price movement can be correlated to lags in prices very well.  Most of the top leaderboard finishers used future information to determine an appropriate lag in the price movement.

The next most interesting thing is that all top 3 finishers used free and open source software as tools for the competition.  Two of the leaders used R and the second place finisher used Python, namely SciPy.  This should not be surprising to most people in the analytics community.  Open source software has been making inroads for quite a while.  The R-Project has been getting a lot of interesting press lately especially in enterprise business circles.  Python is an objective oriented programming language that is getting more popular.  Python's popularity seems to be to its ease of use and how quickly it can be learned and implemented.

The presenters of the 2010 INFORMS data mining competition were kind to post the methods of the Top 3 competitors.  Each method is an interesting read on how they were able to use the open source tools to get predictive results of stock price movements.

If you are interesting in learning more about R as a tool I recommend a new book by Luis Torgo "Data Mining with R: Learning with Case Studies".

This book is one of the first in its kind of showing R methodologies with real life applications.  I am intending to get the book and hopefully have a review on it in the near future.  I am already hearing good things about it.

Data mining competition with R

There is a new data mining competition aimed at predicting preferred data mining tools in R via dataists.com.   The concept of the competition is to try to determine which R packages are preferred in the R community via their CRAN package libraries.  The developers of this new competition are also in the R community with the NY R Users Group.

I am a user of R and I am also a member of the Dallas R Users Group.  As you can imagine I find this competition very interesting because I could benefit greatly from knowing the preferred methods of implementation in the R community.  This can also be a very interesting exercise to determine preferred modeling methods.  I believe this competition will give an insight into the most common methods to apply statistical computing in the community today.

R is getting a lot of press lately.  Revolution Analytics just released the first part of a series of articles on the impacts of R and why it is hot.  R as a statistical and optimization tool is really making a play in the business community.

I am a big fan of these data mining competitions as I've written previously.  It is really interesting what you can learn about the world from the competitions especially if you do not have previous knowledge of the subject matter.  I will try to bring up these competitions from time to time.  I would be interested to know if any IEOR Tools readers have participated in these competitions and what was their experience.

Current Data Mining and Analytics Challenges

I love the Data Mining and Analytics Challenges.  There tends to be so much collaboration and open knowledge especially if the challenge has an affiliated forum.  There really is so much to learn and the challenges offer a great way to bring all of the resources and knowledge together.  Here is a list of the current challenges underway in the Data Mining community.

1. Kaggle is hosting a three competions.   Tourism Forecasting part one challenges to predict 581 tourism-related time series. Chess Ratings - Elo vs the rest of the World is trying to determine a chess rating system that is better than the current Elo rating system. INFORMS Data Mining Contest challenges to predict intra-day stock price movements based on experts predictions, sector data, and other indicators.
2. TunedIT is another competition hosting organization.  Currently TunedIT is hosting the e-LICO mutli-omics prediction challenge with background knowledge on Obstructive Nephropathy.  Yes, I had to look it up too.
3. UC San Diego is hosting the 2010 UC San Diego Data Mining Contest.  This is a two task contest which tries to predict e-tailer's data on consumer and non-consumer information.  The two tasks are a binary preditor and a boolean-transformed predictor.
   
   
   

IEOR Tools Tutorial: Learning XML with R

I have been using a lot of R lately in my work.  R (main site) is an open source statistical computing platform.  Saying R is only used for statistics does not do it justice.  I am finding it to be a really powerful statistical and optimization computing platform.  There seems to be no task that can not be accomplished.  Lately I've been curious about measuring performance with my blog and how it compares to other blogs.  So I thought I would use this opportunity to show how I performed this in R.  I want to rank Operations Research blogs using the Alexa ranking system.  Unfortunately Alexa does not have a search function for Operations Research blogs so I am going to have to get the information myself using R.

This R tutorial is going to be using the package XML.  Packages are used in R to perform specific computational needs that the base R platform can not accomplish on its own.  There are several different packages that can be loaded into R to perform a wide variety of problem instances. 

Kaggle introduces new Chess rating competition

Kaggle, home of the statistics and predictive modeling competitions, is introducing its latest contest Elo Versus The Rest of The World.  The competition is being organized by Jeff Sonas who is a chess-metrics afficionado himself.  Jeff describes his history with rating chess players and why he wanted to start such a competition with Kaggle.

This looks to be a really interesting modeling competition with already more than 40 submissions in the leaderboard.  The interesting note about this competition is that the Elo rating system itself is going to be making an appearance on the leaderboard.  This means that if no one beats the Elo system than there is no declared winner.  Although it looks like someone has beaten Elo at its game already.  Elo will be on the leaderboard as a benchmark to make sure that the competition is proving its worth.

I hope to get a chance to make an appearance on the leaderboard.  I am involved in Kaggle's INFORMS 2010 Data Mining contest.  I'm barely hanging on to the top 10 in that competition.  There are some pretty good models to compete against in that group. 

R IDE for Linux and Gnome

I have been using R in my work recently.  I have also been using R at home to do some tinkering.  In my work environment I use Windows (none too pleased).  I find using the regular R console with Textpad makes for a good Windows development environment.  I haven't been able to replicate this at my home.  At my home I have Ubuntu as my operating system.  I have been searching for a comparable R environment for my home.

That is until now.  The statistics blog at Stattler.com did some research on this very topic of R and Ubuntu.  They found a plugin for R with the text editor Gedit that works wonders.  The plugin is called Rgedit and is very easy to install.  Stattler offers a simple instruction for installation.  Also Stattler has a great review of the Rgedit plugin. Rgedit is very similar in layout to usual gedit text editor except it splits the panes of the screen for code and R output.

Some of the highlights of the Rgedit plugin include:

• Split screen of panes and can be turned on and off
• Syntax highlighting specific to the R code
• Single line or batch processing of R scripts
• Multiple R workspaces can be run
• Shortcut keys can be created and customized

This plugin suits my needs just fine for my Ubuntu uses with R.  There are many other IDEs for R that you may find suit your needs better.  The the beauty of open source software is that there never seems to be a shortage of options.

Eight Data Mining Social Networking Groups

 
Visit AnalyticBridge

Networking is an essential part of career management for any professional.  The relationships we develop can have great impact on our career direction and growth.  I tell young professionals all the time that their best asset in career growth is their professional network.  I tell them to start early and maintain the network continually.  The advent of LinkedIn and other internet social networking sites has made that task easier.

Vincent Granville at AnalyticBridge.com has compiled a list of 8 data mining social networking groups with more than 2000 members.  These groups are easy to become as member as all of them are associated with LinkedIn.  If anyone is finding anyone in the data mining community more than likely they will be a member in these groups.

I have found a couple of different jobs through LinkedIn in the past.  I have found hiring managers as well as peers that I would be potentially working with closely.  I found that to be a great benefit before the interview process.  In fact I even would contact some of the peers in the group to get a pre-interview idea of where I was working and the idiosyncrasies of the organization.  The important thing is using your professional network to maximize your career productivity.

Kaggle hosting INFORMS 2010 Data Mining Contest

Kaggle is hosting the 2010 INFORMS Data Mining Contest.  The goal of this years INFORMS Data Mining Contest is to predict intra-day stock price movements.  All data and submission guidelines are provided on the Kaggle website.  Entries that are submitted are immediately scored and evaluated by an AUC calculation.  The leading AUC score by the end of the contest is going to be honored as the annual INFORMS meeting which is in Austin, Texas (Nov. 7-10).

There is already a lot of good discussions of modeling techniques.  Mark started off with a question on OR-Exchange about modeling methods for the INFORMS contest.   Since the data is a binary categorical target his preferred method was using Logistic Regression.  Mark provides example R code to provide collaborative input to the contest.  I followed suit and provided an IEORTools entry to the contest.  I used the same methods of Logistic Regression.  I also did some variable analysis using the rpart package in R to develop a decision tree.  After pulling some variables that were not significant I was able to get on the leaderboard with Mark.  The pictured leaderboard is of June 28. 

There is also some good discussion on the Kaggle website contest forum.  Posted on the forum one entrant suggested possible variables to use in a Logistic Regression model which is very beneficial.

I really like to see this collaborate effort to modeling.  This was one of the qualities I really enjoyed in the Netflix Prize.  I hope Kaggle and INFORMS continues to provide these fun and thought provoking contests.

R package for World Bank Data

A little while ago I posted about how the World Bank data is open to the public for research.  This apparently is just the beginning of what is possible with having free access to a lot of really good data sets on socio-economic information.  R-chart blog just recently posted saying that an R package was developed as an API to access the World Bank data.

This opens up a lot more data mining opportunities and could just be the start of some great analytic research.  I'm really looking forward to seeing what some of the great R minds will find with the World Bank data at their fingertips.  Since R is freely available anyway this merger makes sense on all sorts of levels.  Happy data mining!


 
 Software for Data Analysis: Programming with R (Statistics and Computing)
 

Analytics and FIFA World Cup

What would the FIFA World Cup be without the prognosticators?  You can be assured that the Analytics community is not far away from the scene.  There are plenty of places to find on the web predictions and analytics of the 2010 FIFA World Cup.  Here are some of places on the web where you can find all of your World Cup analytics interests.

Wayne Winston is posting some predictions and rankings on his blog mathletics.  If you are a fan of sports and analytics (i.e. Moneyball) than you would love Wayne Winston's blog.   Wayne does predictions for professional and collegiate sports in basketball, football, baseball, and soccer.

Blog posts on AnalyticBridge tells that big financial institutions are using quantitative financial instruments used in credit swaps and and debt obligations for predicting World Cup outcomes.  It is a corporate financial challenge that is trying to predict with country will go the farthest in the World Cup.  Let's hope its not the same models that were used to predict mortgage backed securities from a few years back.

Spotfire's blog has an entry about providing World Cup data all the way back to 1930.  TIBCO, Spotfire's parent, is providing analytic data from the World Cup including all sorts of statistics.  Analysts can get scores, goals, penalties, attendance, and other data points.  The online app that TIBCO provides also has nifty charts to compare different countries performance. 

More Analytic Competitions

In a follow-up to a previous IEOR Tools blog post on Predictive Modeling and Recommendation Challenges there is another organization opening up analytic competitions. Kaggle is an organization that is trying to bring together the best prediction modeling and statistical talent vis-a-vis analytic competitions.

http://kaggle.com/

Kaggle is also encouraging organizations to host a competition on their platform. They want to encourage companies to use them to find top notch predictive analysts.

There are two types of competitions promoted by Kaggle. The two kinds are predicting the future and predicting the past. From Kaggle's website...

The platform allows companies, researchers, governments and other organizations to post their problems and have statisticians worldwide compete to predict the future (produce the best forecasts) or predict the past (find the best insights hiding in your data).

The current contest is a European Voting.

Kaggle is taking advantage of the Netflix Prize and its success. The hope is that Kaggle can be a platform to bring these competitions together. It will be interesting to follow Kaggle to see if there is success in these open competitions for analytics. The results of the Netflix Prize seems to think that there will be a good indication.

World Bank opens data to the public for open research

Press release this week from the World Bank Group states the World Bank will release free access to data. According to the article there is over 2,000 financial, business, health, economic and human development statistics available for free to research.

The World Bank has created a new website to access the free data at data.worldbank.org. Skimming over the Data Catalog shows a great amount of variety in the data sets. There are tables on Global Finance, Education Statistics, Poverty in developing countries, Gender, Business, Debt, Governance just to name a few.

This is very encouraging that the World Bank will offer data openly in this manner. Openness can be a great asset to the research community and help drive improvements and reform where needed. I definitely cheer the World Bank for allowing this data to become public.