Maximize Productivity with Industrial Engineer and Operations Research Tools

Should science be open

2012-01-12T06:13:00.000-08:00

Two interesting articles appeared this week in some blogs I frequent about technology and science. The first is an Op-ed in the New York Times titled Research Bought, Then Paid For and the next is Open Science: why is it so hard? The two articles are a different take on the idea that scientific findings should be open for everyone. Someone who is outside the scientific community might think that statement is silly. Of course science is open. No one has a copyright or a monopoly on scientific or mathematical discoveries. Yet that is not the real issue. The real issue is the access to those scientific discoveries. In some cases the scientific discoveries are paid for by public subsidies.

The main focus of those two articles is that science has been hijacked by the publishers. The articles even go so far as saying the hijacking is a monopoly of sorts. I think monopoly is too strong of an analogy but the publishers do have a lot of control. The control is mostly about access to the science. The publishers own the copyright and can limit access to anyone unless a fee is paid. A lot of the times these fees are rather high. Now it looks like with the Research Works Act the access to publicly funded scientific research will be limited as well. Access to the science is the crux of the debate.

Academics rely on publishing of their scientific findings for further funding of their research. It is part of the academic circle of life. Publishing begets more funding which begets more publishing and the cycle continues. I do believe academic community deserves to get compensated for their research. I'm not sure how much residual income they get other than peer review notoriety from their published content. Publishers seem, again, to have a lot of the control.

I am not an academic researcher. My work is trying to help organizations better themselves by using the learning, skills, and knowledge I have acquired through the years as an Operations Research professional. I try to keep up to date on the latest research and methods by studying journals, networking with colleagues, and reading articles. I rely on scientific access quite a bit in staying up to date with the latest findings. I rely on the academic community so I can improve my knowledge and skills. Yet it seems very difficult for my to gain access to a lot of good research. There has to be a common ground for access to the science. I wish I had a simple solution to this issue but it seems very large and very complicated. There are a lot of interactions that I am sure I am glossing over. Yet I am a big fan of the idea of Open Science.

There are some publishers that do understand this problem. INFORMS seems to get this issue rather well. They do not charge a lot for their journals. In fact as part of membership INFORMS allows two free subscriptions to any journal of your choosing. In addition to that the PubsOnLine Suite is available for $99 which is 12 journals for a whole year. That is a bargain compared to some other publishers. So not all publishers are pure evil. There are some good ones.

IEORTools.com Resources added

2012-01-02T14:19:00.000-08:00

I've decided to spruce up my personal website IEORTools.com. I want to add some additional resources to it along with the book store. Most of the content will be relevant reference links to Industrial Engineering and Operations Research professionals.

The first thing I did was added a Resources side menu. The Resources side menu will link to relevant resource sections. So far I have created the following resources

These links are a collection of resources that I have accumulated over the years. The links are a great reference and hopefully I can build them up more. I'm going to be creating more content on ieortools.com site as opposed to the blog because I'm just running out of room.

Most popular 2011 IEOR Tools blog articles

2011-12-30T05:39:00.000-08:00

The most popular IEOR Tools blog articles of 2011. It is time for reflection and I like to do this every year. It gives me perspective about what is being read. It is also an interesting look at our interests. This year seems to be about our thirst for software tools and how to use them. Also books are still big for reference materials.

Visualizing categorical data in R

2011-12-21T07:29:00.000-08:00

I came across an interesting SAS macro that was used for visualizing log odds relationships of data. This type of chart is helpful for visualizing the relationship between a binary dependent variable and a continuous independent variable. I don't use SAS on a daily basis as I prefer to use R. So I got to thinking that I could recreate this macro using only R. I thought this would be a good tutorial for R on developing functions, using different plot techniques, and overlapping chart types.

The following picture is the result of the logodds function in R. The chart is really close but not quite exact. For the histogram points I decided to use the default squares of the stripchart plot and used a grey color to make it look a little faded.

The following is the R script.

logoddsFnc <- function(data_ind, data_dep, ind_varname, min.count=1){

# Assumptions: x & y are numeric vectors of the same
# length, y is 0/1 varible. This returns a vector
# of breaks of the x variable where each bin has at
# least min.countnumber of y's
bin.by.other.count <- function(x, other, min.cnt=1) {
    csum <- cumsum(tapply(other, x, sum))
    breaks <- numeric(0)

    i <- 1
    breaks[i] <- as.numeric(names(csum)[1])
    cursum <- csum[1]

    for ( a in names(csum) ) {
      if ( csum[a] - cursum >= min.cnt ) {
        i <- i + 1
        breaks[i] <- as.numeric(a)
        cursum <- csum[a]
      }
    }

    breaks
}

brks <- bin.by.other.count(data_ind, data_dep, min.cnt=min.count)

# Visualizing binary categorical data
var_cut <- cut(data_ind, breaks=brks, include.lowest=T)
var_mean <- tapply(data_dep, var_cut, mean)
var_median <- tapply(data_ind, var_cut, median)

mydf <- data.frame(ind=data_ind, dep=data_dep)
fit <- glm(dep ~ ind, data=mydf, family=binomial())
pred <- predict(fit, data.frame(ind=min(data_ind):max(data_ind)),
        type="response", se.fit=T)

# Plot
plot(x=var_median, y=var_mean, ylim=c(0,1.15),
       xlab=ind_varname, ylab="Exp Prob", pch=21, bg="black")
stripchart(data_ind[data_dep==0], method="stack",
             at=0, add=T, col="grey")
stripchart(data_ind[data_dep==1], method="stack",
             at=1, add=T, col="grey")

lines(x=min(data_ind):max(data_ind),
        y=pred$fit, col="blue", lwd=2)
lines(lowess(x=var_median,
               y=var_mean, f=.30), col="red")

lines(x=min(data_ind):max(data_ind),
        y=pred$fit - 1.96*pred$se.fit, lty=2, col="blue")
lines(x=min(data_ind):max(data_ind),
        y=pred$fit + 1.96*pred$se.fit, lty=2, col="blue")
}

logoddsFnc(icu$age, icu$died, "age", min.count=3)

The ICU data for this example can be found in the R package "vcdExtra". Special thanks to David of Univ. of Dallas for providing me with a way to develop breaks in the independent variable as seen by the bin.by.other.count function.

The author of the SAS macro is also the author of Visualizing Categorical Data by M. Friendly which is a great reference for analyzing and visualizing data in factored groups.

OpenOpt Suite 0.37

2011-12-15T08:48:00.000-08:00

Hi all,
I'm glad to inform you about new release 0.37 (2011-Dec-15) of our free software:

OpenOpt (numerical optimization):

IPOPT initialization time gap (time till first iteration) for FuncDesigner models has been decreased

Some improvements and bugfixes for interalg, especially for "search all SNLE solutions" mode (Systems of Non Linear Equations)

Eigenvalue problems (EIG) (in both OpenOpt and FuncDesigner)

Equality constraints for GLP (global) solver de

Some changes for goldenSection ftol stop criterion

FuncDesigner:

Major sparse Automatic differentiation improvements for badly-vectorized or unvectorized problems with lots of constraints (except of box bounds); some problems now work many times or orders faster (of course not faster than vectorized problems with insufficient number of variable arrays). It is recommended to retest your large-scale problems with useSparse = 'auto' | True| False

Two new methods for splines to check their quality: plot and residual

Solving ODE dy/dt = f(t) with specifiable accuracy by interalg

Speedup for solving 1-dimensional IP by interalg

SpaceFuncs and DerApproximator:

Some code cleanup

You may trace OpenOpt development information in our recently created entries in Twitter and Facebook, see http://openopt.org for details.

Regards, Dmitrey.

Expanded list of online courses for data analysis

2011-12-15T06:17:00.000-08:00

The folks at Stanford have been really busy putting together online curriculum for the world to learn. This is a followup to a previous post Machine Learning for everyone. Stanford has included a bunch of other courses that they will promote online.

Some of the interesting courses are

Model Thinking
Natural Language Processing
Game Theory
Design and Analysis of Algorithms

Apart from the promoted online courses from Stanford there is also other courses of interest that are not promoted but still access to lectures and notes. Notable courses from the Stanford School of Engineering include the following.

Introduction to Linear Dynamical Systems
Convex Optimization I
Convex Optimization II

Stanford isn't the only school that is promoting their lectures online for free use. A lot of schools are promiting open learning and collaboration through what is called Open Courseware. Some notable schools inlcuded.

As an analytics professional for many years I've found honing your skills to be very important for your career. Now more than ever it is easier to do with schools opening up their classes for everyone. I strongly recommend finding areas of expertise that you are passionate about or want to learn and find the schools that promote them online.

My learning as a Data Scientist

2011-11-10T06:24:00.000-08:00

So apparently the new en-vogue title is Data Scientist. I can now include that to my already expanding list of titles. In the past I've been known as an Engineer, Operations Analyst, Production Control Specialist, and an Analytics Analyst. Now I'm considered a Data Scientist. It's all the same to me. My training and expertise has allowed me to problem solve many challenges within organizations. The title doesn't matter. There are opportunities for people with my skill set.

A recent blog post by Kontagent Kaleidoscope about Big Data is Useless without Science got me thinking about my role as a self-proclaimed Data Scientist. The blog article points out a need for the science of better decision making. Organizations are looking for people to help them turn their data mines into information gold. I've definitely learned a lot over the years as a Data Scientist and I thought I would list some of those learnings.

1. Organizations Don't Know What a Data Scientist Can Do

The idea here is marketing your own talents. The Data Scientist needs to put their methods and work out there for the organization to see and touch. This means working with the peers and management in the organization. The Data Scientist needs to be able to eloquently relate methods, problems, challenges and how they can be solved. Important skills here are personal marketing and communication. I know this goes against the grain of many numbers geeks like me.

2. Problems Don't Solve Themselves

Opportunities for solving real problems in an organization are always around. The trick is being in the right place at the right time to be able to solve those problems. Organizations have hoarded a lot of data and many times they don't even remember why. The Data Scientist needs to turn into a Data Detective. Explore all aspects of the organization. Interview different departments and see how they tick and ask questions like "What keeps you up at night about your job?". I was often surprised how a simple solution would go a long way to helping someone else out. This develops true collaboration and leads to bigger problems to solve.

3. Always Continue to Learn New Things

The world is constantly evolving and there are always new tools, tricks, methods, algorithms, software and mechanisms. The Data Scientist needs to be able to adapt to new technologies. I've found its best to stay current with whats new in order to stay sharp and meet new demands. The internet can be your friend. Even keeping up with a favorite list of blogs can help with staying current. Times change and so do organization's needs. Perhaps this is just me but I love learning new things as it creates a fun diversion and improves my skill sets.

Top 50 Statistics blogs of 2011

2011-10-11T05:33:00.000-07:00

TheBestColleges.org published their list of the top 50 statistics blogs. This is a really good resource list of statistical analysis and news.

Data mining the Federal Reserve

2011-10-03T06:03:00.000-07:00

The Federal Reserve now has the ability to have its data programmatically retrieved. The St. Louis Fed Web Services allows programmers and data scientists to retrieve key economic data from their libraries. I have not had a chance to peruse the site at all but this can be a really interesting source of data. The age of Open Data is really upon us. This can lead to some really interesting research for professionals and amateur scientists.

Machine Learning for everyone

2011-09-26T06:23:00.000-07:00

Well maybe mostly everyone. Have you been interested in gaining knowledge in the latest craze of artificial intelligence and computing? Then go no further than Stanford's Machine Learning course which is now open enrollment to everyone! Andrew Ng is back to provide the world with knowledge about Machine Learning for the entire masses.

Per Stanford's website, Machine Learning is data mining and statistical pattern recognition. Mostly it is applying mathematical and statistical methods to draw out information behaviors from data sources. So do you want to invent the next Netflix, Amazon or Google? This is the course for you.

If you do not want to enroll in the Machine Learning class you could always watch some of the older lectures online. Andrew Ng provides plenty of information from past lectures with student contributed projects. The CS 229 website is worth a look for a punch of Machine Learning related resources.

Data Driven Success in Professional Baseball

2011-09-23T09:34:00.000-07:00

An interesting article from Data Center Knowledge about the presentation Paul DePodesta gave at the Strata Summit. Paul DePodesta is known for bringing mathematic and analytical know-how to Billy Beane and the major league professional baseball team Oakland Athletics. His story was accounted by Michael Lewis in the book "Moneyball" and is being portrayed with the same name on the big screen opening this weekend.

I really liked this quote from Paul in the article.

We didn’t solve baseball. But we reduced the inefficiency of our decision making.

Is that not the sort of things that an analytical professional or an Operations Researcher ultimately tries to do? Operations Research is not the art of creating anything new. It is the art of creating existing things better. All decision making is inefficient to some point. Even the right decision can be inefficient on some level. Decisions are full of balancing acts of constraints and feasibility.

Also this proves that no industry or organization is absent of a need for efficient decision making. Even baseball can us a dose of improved decision analysis. Whether is scheduling the league or determining the best pitcher for their value. Sports has definitely come into their own with decision analytics. I'm eager to watch Paul's career and wonder if analytics is taking it to the next level.

OpenOpt Suite 0.36

2011-09-15T10:32:00.000-07:00

New release of the free BSD-licensed software OpenOpt Suite is out:

OpenOpt:

* Now solver interalg can handle all types of constraints and integration problems

* Some minor improvements and code cleanup

FuncDesigner:

* Interval analysis now can involve min, max and 1-d monotone splines R -> R of 1st and 3rd order

* Some bugfixes and improvements

SpaceFuncs:

* Some minor changes

DerApproximator:

* Some improvements for obtaining derivatives in points from R^n where left or right derivative for a variable is absent, especially for stencil > 1

See http://openopt.org for more details.

Physicist cuts airplane boarding time in half

2011-08-31T09:50:00.000-07:00

I have always been fascinated with the airplane boarding problem. I wish I was in the airline industry because I would love to tackle this problem. I used to travel a lot for my job and I would marvel at how inefficient the time it took to board an airliner. My first inclination is to redesign the plane (and the airport jetway) to include exits at middle and rear of the plane to go along with the forward exit. Yet I never put my ideas to paper and tried to calculate efficiency gains. There have been a lot of ideas try to find the optimal boarding arrangement. Would you believe that random boarding, i.e. Southwest Airlines, is a more optimal boarding procedure then the current row assignment method?

Yet a physicist from Fermilab, Jason Steffen, did have some interesting ideas to improve the existing airplane boarding procedures. By using Monte Carlo simulations to measure efficiency and test his ideas he was able to improve airplane boarding by as much as half the time. From the article, his methods were to using sections of window seats first but alternate aisles so passengers would not interfere with each other.

This is a very clever idea. Yet I found one flaw that may not have been assumed in his study. I've noticed that overhead space is a premium for passengers, especially for business travelers. Business travelers often bring two carry-on bags. These bags tend to fill up the overhead bins rather quickly. When the overhead bins fill up then passengers have to search in the aisles looking for available space for their bags. This creates a bottleneck and queues develop for the other boarding passengers. It seems to me that Jason's study makes an assumption that all overhead bins would be available at time of boarding. If in fact alternating rows are used in his model than overhead bins might become filled to capacity before passengers board and create more bottlenecks. Its just one theory that would be worth investigating before Jason's procedures are implemented.

I applaud Dr. Steffen's studies and finds in the airplane boarding problem. It is a fascinating problem as most of us have encountered airplane boarding from time to time. For more information on his methods you can read about Jason's work airplane boarding, which is very fascinating, on his website.

What did we learn from the Space Shuttle program

2011-07-25T06:41:00.000-07:00

The NASA Space Shuttle program ended this past week with STS-135. I still remember watching the shuttle launches as a boy. They filled my head with dreams of space exploration and new discoveries. As I grew older I took an interest in engineering and studied to be one in a university. There I discovered the enormity of the engineering marvel that was the Space Shuttle program.

Discover published an article this week on what was the debacle of the Space Shuttle program. A lot of good and interesting points made by Amos Zeeberg in this article. The Space Shuttle was originally designed to be a cost effective way of getting man and technology into space. The program definitely did not deliver on that promise or projection. Also the Space Shuttle was considered to have only a risk of failure of 1 in 100,000. I don't know if that is remotely true. As we unfortunately know the true risk of failure was 2 in 135. Space travel is risky no matter how it is done.

So from an engineer's point of view, albeit one that was not involved with the space program, what can we really learn from the Shuttle Program. I believe applying Industrial Engineer and Operations Research principles we could come to some conclusions. I don't personally think the Shuttle missions were a total debacle. As an Engineer there is always something to learn even if there is a failure. Edison said it best that he didn't fail 1000 times trying to develop a light bulb, only he learned 1000 ways on how not to build one.

Firstly, risk needs to be measured from a micro and macro perspective. There are many systems that lead to failure. Each system has a life all of its own. The risk could be as simple as an O-Ring to as complicated as a practical study of landing on the Moon. Risks can be measured and weighed from different perspectives of time, cost, and quality of delivery of promise. When all risks are measured than perspective can be put into place as to delivery of a promise. Perhaps the Shuttle program didn't deliver on all promises. Yet it did prove many things that reusable vehicles were ahead of its time. We can learn a lot from the Shuttle Program on examining risks of promise and making sure that we evaluate different objectives and goals.

Secondly, engineering and management should be a cultivated relationship that needs to understand each others' strengths and goals. Engineering has the design in its best interest. Management has the mission in its best interest. The design and mission are unique and have there own set of goals. Yes there are going to be risks weighed in both the design and mission. The complexity is when merging the risks of the design and mission together. The magnitude of the NASA Space Shuttle Program magnified the relationship between engineering and management. The best and the worst was brought to light. The engineering marvel of creating a reusable vehicle is magnificent. The managerial feat of sending man into space with a reusable vehicle on more than 100 missions is not insignificant. The importance of merging design and mission together was a great learning experience with the Space Shuttle program. We have already seen fruits of that success. Missions to Mars and beyond the Solar System have proven that success.

The NASA Shuttle Program was not an outright debacle. There was a lot to learn from the process. No it did not deliver on all initial expectations. Yet it did deliver on this young boy's dreams of discovery and knowledge. Once an Engineer, always an Engineer. I hope that we will never cease to learn and improve from our failures.

Problems with data visualizations followup

2011-07-04T05:36:00.000-07:00

On a recent article post I was showing a bad data visualization chart from The Economist. While reading over Slashdot I found a similar bad data visualization article about bad visualizations from BP and GE as presented by Stephen Few's blog. No doubt a lot of people share in the same frustration.

Now that we are in the Insight Age it seems that we will continually question and interpret how data will be presented to us. We are now data rich but knowledge poor. I believe there is going to be vast new opportunities to help disseminate the data. Perhaps even ways to help visualize the data as well.

I strongly suggest reading Stephen Few's blog. It is an interesting read on how data visualization can be used poorly. He even shows examples on how to do it correctly.

How not to do data visualization

2011-06-30T06:24:00.000-07:00

I was glancing over Hacker News and came across an article from the Economist Daily Chart blog. The daily chart was about nations debt management. The following can be shown here.

Seems innocent enough. It shows in declining order the debt per nation. What a second? Why is Ireland have more debt than USA? After reading the article more thoroughly it looks like it is a percentage of GDP. What a third time? Is the bar graph the percentage of GDP or the number in the white box a percentage of GDP? And how does this relate to debt management? So apparently in the article it explains the change in primary balance for each nation to be 60% of GDP. So the bar graph is a % change of GDP to get to 60% of GDP. Are we crystal? I'm not sure I totally understand but that is my basic understanding.

Data visualization is important in Analytics and Operations Research. We need to model real world applications quite a lot. Often times there is no better way to do this than to use a chart or graph. The real art is conveying the crux of the message to the recipient. There is an internet meme devoted to the art of bad chart making. I feel bad using the Economist as an example because after all I did finally (I think) come away with the right idea. But still notice how there are no data or axis labels across the top of the chart. Also the numbers in the white boxes are not given any units. I'm still not sure if those numbers in the white box are a percentage or a debt value. Sometimes the visual art clutters the real message. It is important to make sure that recipient has the right frame of reference and can understand each graphic and label.

Recommended Machine Learning blogs

2011-06-26T05:25:00.000-07:00

I happened upon an OSQA site called metaoptimize about recommended Machine Learning blogs. There were a lot of blogs listed on this site that I had not seen before so it really got me interested.

Good Machine Learning blogs

Machine Learning is the scientific process of developing algorithms for computers to evolve based on empirical data. For instance one may develop a decision tree that helps predict a certain behavior from a data set. The decision tree itself is just a method to predict behavior. Yet perhaps more data can be acquired and more behaviors can be realized. Then the decision tree is computed again based on the newer data (and perhaps combined with the older). New behaviors are learned from the newer data and a new implementation of the decision tree is evolved for new behaviors. This process becomes algorithmic and continues.

Machine Learning developed out of the field of Artificial Intelligence. The idea of having computers learn has been around since as long as computers itself. Machine Learning is really starting to develop as computing power has caught up to theory. Machine Learning has a lot of uses and may be used by some of your favorite computer applications. Some examples include product recommendation systems like Amazon or Netflix, search engines like Google or Bing. Machine Learning is seeing practical uses in many places and its only just touching the surface.

Moneyball coming to the big screen

2011-06-20T06:08:00.000-07:00

Recently I found out that the book Moneyball by Michael Lewis will be shown as a motion picture. The Moneyball trailer can be viewed online. In case you have never heard of Michael Lewis then you might have heard about the movie "The Blind Side" which he also wrote the accompanying book. The book Moneyball is about the Oakland Athletics and how they used analytics and mathematcal know-how to turn around a professional baseball franchise.

The story centers around Billy Beane which is played by Brad Pitt in the movie. Billy Beane is a professional ballplayer turned General Manager. Billy Beane inherits the top organizational management job for the losing Oakland Athletics. He is immediately frustrated with the same old losing ways and believes he needs to shake up the system. He finds out about the curious world of baseball analytics or otherwise know as sabermetrics and hires a curious crew of young mathematically gifted folks.

The story is fascinating even if you are not a fan of baseball. The use of mathematics to help make business decisions is nothing new. Yet employing this analytics method to an industry that is deep rooted in old ways and practices is intriguing. Changing the ways of the "good ole boy" network requires risk, knowledge, and sometimes good fortune. This can translate to almost any industry or even organization. I am most definitely looking forward to seeing this movie.

OpenOpt Suite 0.34

2011-06-16T04:40:00.000-07:00

I'm glad to inform you about new quarterly release 0.34 of our free OOSuite package software (OpenOpt, FuncDesigner, SpaceFuncs, DerApproximator) .

Main changes:

* Python 3 compatibility

* Lots of improvements and speedup for interval calculations

* Now interalg can obtain all solutions of nonlinear equation (example) or systems of them (example) in the involved box lb_i <= x_i <= ub_i (bounds can be very large), possibly constrained (e.g. sin(x) + cos(y+x) > 0.5 or [sin(i*x) + y/i < i for i in range(100)] )

* Many other improvements and speedup for interalg

Regards, D.

Analytics geeks win NBA championships

2011-06-13T06:20:00.000-07:00

The Dallas Mavericks win their franchise first NBA Title. They won their first championship by beating teams that everyone thought they could not beat. The Mavericks were able to beat juggernauts like the Los Angeles Lakers, a fast paced Portland Trailblazers team, up-and-coming youngster superstars in the Oklahoma City Thunder and of course the Big Three from the Miami Heat. As good as the Mavericks were executing on the basketball court they were equally as good executing a between-the-ears approach to basketball. The Mavericks were able to win the game by studying the numbers of professional basketball. Some of the champions on the Mavericks team may not be able to hit even 10% of three pointers but they sure know how to analyze a winning combination.

The analytics culture starts with Dallas Mavericks owner Mark Cuban. According to ESPN when Mark Cuban was looking for a coach he studied games and found out that Rick Carlisle used the most efficient lineups most frequently. Mark Cuban hiring Rick Carlisle to coach the Mavericks was a no-brainer because the numbers do not lie. As for Rick Carlisle, he is known for being a very cerebral coach and very handy with crunching NBA statistics as well.

Another known fact about the Dallas Mavericks is that they use an analytics staff to gain a competitive edge. Most recently they have retained the NBA analytics stat guru Roland Beech of 82games.com. In the past they had used the services of Wayne Winston, an Operations Research professor, to help analyze their lineups to be more competitive.

Mark Cuban gives a lot of attention to the geeks for Mavericks winning. From the ESPN article

I give a lot of credit to Coach Carlisle for putting Roland on the bench and interfacing with him, and making sure we understood exactly what was going on. Knowing what lineups work, what the issues were in terms of play calls and training.

That is a lot of brainpower on the bench in every game. It is good to see the geeks get their due. Way to go Mavericks and looking forward to seeing what the geeks put on the court next season!

Sports analytics summer blog reading recommendations

2011-05-23T17:04:00.000-07:00

The dog days of summer are almost here and if you are a sports fan it can be long. Only baseball and soccer endure the summer seasons in the U.S. Even if you are a die hard baseball or soccer fan the season itself can seem to last forever. Now is the perfect time to get caught up in the analytics of your favorite spectator sports. The following is some of my favorite sports analytics blogs and reading material.

Baseball
FanGraphs.com
FanGraphs is the all everything baseball numbers website. The best thing that FanGraphs is known for is having a complete database of baseball players metrics. One of my favorite metrics in baseball is WAR or Wins Above Replacement. If that is not enough they even have heat maps of strike zone pitching locations. Tracking your favorite team has never been more analytically exciting.

Football
AdvancedNFLstats.com
Advanced NFL Stats is the best NFL analytics blog out there right now. Similar to FanGraphs there is a complete database of NFL offense and defense metrics. Advanced NFL Stats also does a good job of explaining the numbers behind the measurements. Football is no easy task to analyze team and player performance. This site does an excellent job of both. Also Advanced NFL Stats is keeping a database of play-by-play data.

Drive-by-Football
The up and comer of the NFL analytics blogs is Drive-By Football. Drive-By does a great job of explaining some of the harder math around determing team and player efficiency. One of the most interesting features is the Markov Chain Drive calculator which calculates likelihood of scoring scenarios drive-by-drive hence the name of the website.

Basketball
Wayne Winston blog
This blog's primary focus is on Basketball, specifically the NBA. Wayne Winston is definitely known as a prolific Operations Research professor. You may not know is that Wayne Winston consulted the Dallas Mavericks and other sports teams to help improve their franchises. Wayne talks about other sports from time to time as well. If you have not read Wayne Winston's book Mathletics: How Gamblers, Managers, and Sports Enthusiasts Use Mathematics in Baseball, Basketball, and Football you are in for an analytical treat. Wayne analyzes the why and how of measuring professional sports efficiency and winning.

In Memorium of Dr. Paul Jensen

2011-05-17T17:13:00.000-07:00

I received discouraging news last week that we lost a colleague in the Operations Research and INFORMS community. Dr. Paul Jensen passed away peacefully on April 4, 2011. Dr. Jensen served a number of years at the Univ. of Texas in Austin as a great contributor to the Operations Research community and researcher. As recently as 2007 Dr. Jensen was awarded the INFORMS Prize for the Teaching of ORMS Practice.

I unfortunately did not know Dr. Jensen personally. I was first introduced to his ORMM website through my graduate courses at SMU. The ORMM website is a great resource to help educate the principles of Operations Research methods. I was also able to use some of his Excel modeling add-ons in practice to demonstrate optimization problems.

Dr. James Cochran is going to hold a special session in memorium of Dr. Jensen. This message from Dr. Cochran was sent on Dr. Jensen's ORMM mailing list.

Dear friends and colleagues,

Paul was a good friend and colleague. I know each of us will miss him (as will many other friends throughout the OR community) and each of us is very sorry for the loss suffered by Margaret and the rest of Paul's family.

I will chair a special INFORM-ED sponsored session in Paul's memory at the 2011 INFORMS Conference in Charlotte (November 13-16). Several of Paul's many friends will speak on his contributions to operations research education and share personal stories and remembrances about Paul. Margaret and Paul's children will be invited to attend, and I hope each of you will also be able to attend (I'll try to reserve some time at the end of the session during which members of the audience will have an opportunity to share their thoughts).

INFORMS Transactions on Education (the online journal for which I am Editor in Chief) will also publish a special issue devoted to Paul's influence on OR education. Dave Morton has kindly agreed to edit this special issue, so I am certain it will be a fine tribute to Paul.

Sincerely,

Jim

Welcome to the Insight Age

2011-05-16T21:23:00.000-07:00

We are in the midst of the Insight Age. In other words the end of the Information Age. This has been explained on HPCwire quoting HP Labs distinguished technologist Parthasarathy Ranganathan. No longer are we seeking ways to process information. We are seeking ways to disseminate and draw conclusions from the information we already hold.

Does this sound familiar to anyone in Operations Research? It should because this is what Operations Research has been doing for years. I think I sound like a broken record sometimes. Yet I guess the story needs to be told again. But perhaps I'm being a little too snarky. It could just mean that the Information Age is catching up to the decision science analysts.

The crux of the article is technology meeting the demands of information overload. Yet that is not what the definition of insight is to me. Insight is drawing conclusions based on the evidence. The Operations Research analyst will undoubtedly be well prepared for this evolutionary advancement. I'm sure HP is aware that technology alone will not help the Insight Age revolution.

I hope we've all seen this new age coming. The Insight Age is here and is ready to be tackled. My next inclination is to think what will define the Insight Age. The Information Age was defined by the internet, computing power, and globalization. My prognostication to define the Insight Age is open data and decision science. Open data is about having no barriers to information. Data will be freely accessible and easy to disseminate. Decision science is already here and will make an even bigger impact. Machine Learning, Artificial Intelligence, Optimization Algorithms will all be the cogs of the Insight Age mechanism.

Insight Age is such a fitting name. I'm really liking it the more I think about it. I'm going to try to remember that in some of my future conversations.

R Tutorial: Add confidence intervals to dotchart

2011-05-15T21:20:00.000-07:00

Recently I was working on a data visualization project. I wanted to visualize summary statistics by category of the data. Specifically I wanted to see a simple dispersion of data with confidence intervals for each category of data.

R is my tool of choice for data visualization. My audience was a general audience so I didn't want to use boxplots or other density types of visualization methods. I wanted a simple mean and 95% (~ roughly 2 standard deviations) confidence around the mean. My method of choice was to use the dotchart function. Yet that function is limited to showing the data points and not the dispersion of the data. So I needed to layer in the confidence intervals.

The great thing about R is that the functions and objects are pretty much layered. I can create one R object and add to it as I see fit. This is mainly true with most plotting functions in R. I knew that I could use the lines function to add lines to an existing plot. This method worked great for my simplistic plot and adds another tool to my R toolbox.

Here is the example dotchart with confidence intervals R script using the "mtcars" dataset that is provided with any R installation.

### Create data frame with mean and std dev
x <- data.frame(mean=tapply(mtcars$mpg, list(mtcars$cyl), mean), sd=tapply(mtcars$mpg, list(mtcars$cyl), sd) )

### Add lower and upper levels of confidence intervals
x$LL <- x$mean-2*x$sd
x$UL <- x$mean+2*x$sd

### plot dotchart with confidence intervals

title <- "MPG by Num. of Cylinders with 95% Confidence Intervals"

dotchart(x$mean, col="blue", xlim=c(floor(min(x$LL)/10)*10, ceiling(max(x$UL)/10)*10), main=title )

for (i in 1:nrow(x)){
lines(x=c(x$LL[i],x$UL[i]), y=c(i,i))
}
grid()

And here is the example of the finished product.

Google funding research to measure regret

2011-05-03T06:38:00.000-07:00

According to an article in Mashable Google is funding Artificial Intelligence research at Tel Aviv University that will help determine if computers could be taught regret. My first inclination is to wonder if this is really anything new. Linear programming itself is all about regret or, in financial terms, opportunity cost. From the article it describes the research is about how to

measure the distance between a desired outcome and the actual outcome, which can be interpreted as “virtual regret.”

That sounds a lot like mathematical programming to me. So what is so different about the Tel Aviv Universtity findings? Apparently its not something new with the algorithms but more or less new with how the data is processed. Dr. Yishay Mansour explains that they will be using machine learning methodologies to look at all the relevant variables in advance of making informed decisions. This sounds more like this research is in the realm of how to understand large amounts of data and processing it into usable information.

Big data is a huge problem in the data rich but information lacking internet environment that we face today. There is a lot of data handled by organizations but they need to know what to do with it. Today's Operations Research professional should be perched to swoop in an help this issue. Organizations are data rich but lack the focus to apply it to meaningful decision analysis. I'm hoping that this is only going to lead to a big watershed moment for the Operations Research community.

Open Source replacements for Operations Research and Analytics Software

2011-04-21T11:00:00.000-07:00

I was reading an article from Datamation on 70 Open Source Replacements for Small Business when I noticed a glaring omission. Where are the software applications for Operations Research and Analytics? So here is my best addendum to this article that should complete what small business should know about Open Source analytics productivity software.

Statistics and Computation

1. R Project

Replaces: SAS, SPSS

R is a free and open source statistical computing environment that holds its own against some of the most established proprietary statistical environments. R is available on all operating systems and is free for download. R also has a community driven library of add-on packages that are also freely available and cover almost any statistical, mathematical, or optimization need.

Also a great reference manual for those switching from SAS to R is SAS and R: Data Management, Statistical Analysis, and Graphics

2. RapidMiner

Replaces: KnowledgeSEEKER

RapidMiner is a data mining software with a graphical front-end. RapidMiner is suitable for most data mining and data transformation needs.

Mathematical Programming and Optimization

3. GLPK

Replaces: AMPL

GLPK is a GNU/free software linear programming software kit. GLPK is intended for large-scale linear programming, mixed integer programming. GLPK is based on GNU MathProg (or GMPL) which is considered a subset of the AMPL syntax. GLPK also has its own solver.

4. Symphony

Replaces: CPLEX, Gurobi

Symphony is a mixed integer linear programming solver developed under COIN-OR. Symphony is a flexible framework that offers many methods to customize solver capabilities given problem sets.

5. OpenSolver

Replaces: Excel Solver

OpenSolver is a linear an integer optimizer alternative to the Excel Solver in Microsoft Excel. OpenSolver is based on the COIN-OR CBC engine. Unlike the Excel Solver there is no software limits to the size of the problem that can be solved.

OReilly - Quiet Rise of Machine Learning

2011-04-12T05:50:00.000-07:00

An interesting article from the OReilly Radar blog by Jenn Webb on the Quiet Rise of Machine Learning. I love the insight from the article that decision sciences like Operations Research are getting more mainstream. Machine Learning is that one science of mixing data mining and predictive analytics. The methodologies of machine learning is nothing new. Computers have been able to keep up with the mathematics and now skilled scientists are using these techniques all over industry.

Yet to me what is interesting in this article is that implies that machine learning sciences is rising from basically nothingness. As if this is some sort of new fangled technology developed by IBM for a special man vs. machine Jeopardy act. I guess I'm a little too close to the Operations Research community to know where the roots really lie. For one I'm happy that decision sciences like machine learning are getting more and more recognition. On the other hand I'm thinking "Where have you been since World War II?". I guess I'm a little too cynical lately.

I love the OReilly Radar blog as it seems more and more articles are about the promise of data analytics. I guess I'm just wishing for a little more investigative reporting. In fact I think it would benefit INFORMS if they partnered with OReilly Media. OReilly definitely has a focus on analytics now and INFORMS is prime to provide a lot of great content for discussion.

Baseball and Decision Analytics

2011-03-30T18:52:00.000-07:00

When its spring time it means that baseball season is getting ready to get started. On the last day of March is when the 2011 Major League Baseball season gets going. Baseball is as American as apple pie and almost every baseball enthusiast has something to say about the game. Analytics professionals are not far behind when it comes to opinions on baseball.

Baseball is definitely a numbers game. Mathematicians have been studying baseball for as long as the game itself has been played. One of the first notable baseball analysts to apply decision analysis was Bill James. Bill coined the study of baseball analysis as sabermetrics which is taken from the acronym of the Society of American Baseball Research. More recently baseball decision analysis has found its way to the Major League Baseball teams management offices. Popular books such as Moneyball by Michael Lewis and The Extra 2%: How Wall Street Strategies Took a Major League Baseball Team from Worst to First by Jonah Keri have shown how major league management turned around poor performaning clubs into championship contenders. The mathematics behind their decision analysis can be described best by Wayne Winston's book called Mathletics: How Gamblers, Managers, and Sports Enthusiasts Use Mathematics in Baseball, Basketball, and Football.

Baseball decision analysis has grown up since Bill James devised the batting average. Now baseball decision analysis uses techniques such as replacement value. The Value Over Replacement determines the value of a player given that player would be replaced by an average or run-of-the-mill at the given player's position. Value Over Replacement was made popular by Keith Woolner, the author of the Baseball Prospectus 2011. At first the value, which is usually offensive value, was to determine how many runs a player could produce over an average player. Now value over replacement methodologies determine how many wins a player can generate for their respective team. One of the best sites to give WAR analysis, or Wins Over Replacement, is Fangraphs. Fangraphs has about every major statistic on baseball available for the baseball enthusiast. In fact they even have heat maps for pitch location. Ready to manage your own team yet?

Pitch location heat map from Fangraphs.com

Of course all of this decision analysis would not be possible without the numbers. One of the best places for baseball data is Baseball-Reference.com. Just about every data point on baseball can be mined from the site and downloaded. So if you have a craving to create your own baseball metric or analytics strategy there should be nothing stopping you.

This is another post in the INFORMS Online Blog Challenge. This month is O.R. and Sports.

R again in Google Summer of Code

2011-03-22T20:54:00.000-07:00

I'm a big fan of the Google Summer of Code. It brings great projects together with a learning opportunity for students. Once again the R Project was selected to be part of the Google Summer of Code in 2011. Some other notable mathematical and statistics projects with R include Shogun Machine Learning, SymPy, Gambit, Computational Geometry Algorithms Lab, Orange, and Computational Science and Engineering.

The Google Summer of Code has really grown over the years. I'm glad to see that these open source initiatives really help teach our younger generation.

OpenOpt Suite release 0.33

2011-03-16T02:46:00.000-07:00

New release 0.33 of OpenOpt Suite is out:

OpenOpt:

cplex has been connected
New global solver interalg with guarantied precision, competitor to LGO, BARON, MATLAB's intsolver and Direct (also can work in inexact mode), can work with non-Lipschitz and even some discontinuous functions
New solver amsg2p for unconstrained medium-scaled NLP and NSP

FuncDesigner:

Essential speedup for automatic differentiation when vector-variables are involved, for both dense and sparse cases
Solving MINLP became available
Add uncertainty analysis
Add interval analysis
Now you can solve systems of equations with automatic determination is the system linear or nonlinear (subjected to given set of free or fixed variables)
FD Funcs min and max can work on lists of oofuns
Bugfix for sparse SLE (system of linear equations), that slowed down computations and demanded more memory
New oofuns angle, cross
Using OpenOpt result(oovars) is available, also, start points with oovars() now can be assigned easier

SpaceFuncs (2D, 3D, N-dimensional geometric package with abilities for parametrized calculations, solving systems of geometric equations and numerical optimization with automatic differentiation):

Some bugfixes

DerApproximator:

Adjusted with some changes in FuncDesigner

For more details visit http://openopt.org.

IBM has a Natural Language Purpose

2011-02-19T05:58:00.000-08:00

I wanted to write a blog post about the advancements of Natural Language Processing in light of the performance of IBM's Watson on the Jeopardy challenge last week. Natural Language Processing is the science of transforming and interpreting human spoken and written language by artificial means. Generally this type of study has been limited to academic research due to the high computing power demands. Now there are even open source software implementations, including many R Natural Language Processing packages. There is a lot to write about the new advances in NLP.

Instead I came across an interesting editorial on the cheap publicity stunt that is IBM's Watson. At first I thought the article was a comedy that would make fun of Watson's errors on Jeopardy. Then I realized the author Colby Cosh is not jesting at all. This should not be news to me. The field of Operations Research, which was definitely used to help develop Watson, is a widely misunderstood field. Cosh has a hard time understanding why IBM would want to develop such a stunt to compete against humans. Cosh seems to think that the only gain is IBM's shareholders. I can assure you that if IBM wanted to make money on this venture they would have created a computer that would compete on American Idol. Jeopardy is no ratings juggernaut in the US.

So what purpose would IBM have for competing on Jeopardy. Perhaps the idea of "competition" is misleading. In my eyes I was not seeing if a computer can beat humans in a battle of wits. I was seeing if a device could interpret, process, and return meaningful information on the same level as human interpretation. Natural Language Processing is like code breaking. Similarly mathematics, physics, natural science are like codes to mathematicians, scientists, and engineers. It is the process of trying to decipher and interpret our natural surroundings. Language is no different. I can see it easy for Cosh to think that the sole idea of the competition is to beat humans. The purpose was simply to decipher the natural language code. In a better understanding of natural language we can then understand our surroundings a little better.

So why the hype with a computer?

"So why, one might ask, are we still throwing computer power at such tightly delimited tasks,..."

The answer can be found already in the field of Operations Research and Management Science. Perhaps Cash has purchased a plane ticket in the past few years. He might have noticed that air transportation has become very affordable due to competitive pricing. A lot of that is due to optimization and revenue management algorithms in the airline industry. Perhaps he noticed the increase in quality, service, and price of privatized parcel postage. The science of better decision making and transportation algorithms have greatly improved supply chain and delivery efficiency. The list can go on and on. Artificial Intelligence is probably a poor way of describing computer optimization and machine learning science. Artificial Intelligence is not going to replace human intelligence but only help improve the human based decisions that we make every day. IBM has already stated that they wish to improve the medical field with Watson. Medical diagnosis requires vast amounts of information and Watson can help decipher medical journals, texts, and resources within seconds. Applications of Watson could be used in third world countries where medical resources are scarce.

I will be looking forward to IBM's advancement with Natural Language Processing. This offers a new venture into better decision sciences. Perhaps "smacking into the limits" of artificial intelligence will create a better life for those that use human intelligence every day.

Question and Answer sites for Analytics and Operations Research

2011-02-16T05:46:00.000-08:00

This post is inspired from a similar post on Jeremy Anglin's blog about statistics question and answer sites. I thought it would be a good idea to list some of the Operations Research and Analytics focused question and answer sites. Some of these

Operations Research
OR-Exchange
This site is a stackexchange Q&A Site started by Michael Trick. A crowd sourcing question and answer resource for anything Operations Research related. I believe the best one available for Operations Research.

Numerical Optimization Forum
I find it unfortunate that there is so little forums based on Operations Research. This is one of the few and it is a good one. It is moderated by IEOR Tools contributor Dmitrey.

I'm purposely leaving out the sci.ops-research Usenet group because I believe its fallen into disarray with spam content.

Math/Statistics
Cross Validated
My favorite stack-exchange site dedicated to statistics.

Math
Math Overflow

Software
StackOverflow - R tag
StackOverlow - SQL tag

Mailing Lists
Mailing lists do not get as much notoriety as they once did. Maybe because there are so many other options on the internet for getting information. I still think they are a valuable resource and a good online community.

R Help Mailing List   http://www.r-project.org/mail.html
GLPK Mailing List   http://lists.gnu.org/mailman/listinfo/help-glpk
COIN-OR Mailing List(s) http://list.coin-or.org/mailman/listinfo/

Beta StackExchange Sites
These sites might be of interesting to the Operations Research community. They are not live yet but are looking to generate a following.

http://area51.stackexchange.com/proposals/28815/computational-science
http://area51.stackexchange.com/proposals/1907/numerical-modeling-and-simulation
http://area51.stackexchange.com/proposals/27706/engineering-and-applied-sciences
http://area51.stackexchange.com/proposals/26434/machine-learning
http://area51.stackexchange.com/proposals/24602/data-capture-analysis
http://area51.stackexchange.com/proposals/22964/sas-programming-language
http://area51.stackexchange.com/proposals/18584/engineering-and-scientific-software-tools
http://area51.stackexchange.com/proposals/15237/r-statistical-package
http://area51.stackexchange.com/proposals/9218/operations-research (Interesting. Do they know about OR-Exchange?)

I would love to see more examples that I can include in this list.

UPDATE:

I forgot to add O'Reilly's Q&A site with the R tag. http://answers.oreilly.com/tag/R

Science of Matchmaking

2011-02-11T18:55:00.000-08:00

The science of matchmaking has seen serious growth in the last few years. What exactly is so scientific about matchmaking anyway? The goal of any commercial enterprise (and some public organizations) is to match products or services to the demand of consumers. The idea of matching consumers with products and services is not new. Matchmaking is essentially the business art of Marketing. The science behind the matchmaking has seen the most advancement and improvement in recent time. Generally speaking computing power has made the difference for the technologically leap forward. Millions of points of observations and data can be sifted and combed with great ease as compared to even just a decade ago. There is a scientific magic fiddling to the matchmaking phenomenon (sorry, bad Fiddler On The Roof pun).

Mathematics of Matchmaking

I'm not sure I can cover all of the math behind the science of matchmaking. I thought it best to describe an example with the company Netflix. Netflix wants to make the decision process of selecting movies for its customers easier. Netflix developed an algorithm to match customers' interests in movies. In fact they even decided to farm out an improvement to the algorithm in a worldwide contest. So how does the Netflix algorithm work? There is a lot of math behind the algorithm but it essentially comes down to finding common features in the customer and movie data. The customers give Netflix a clue to the features they want by ranking movies the customers enjoy. This then becomes the dependent variables in the algorithm formulation. Then the algorithms churn out likely matches based on common feature sets.

Perhaps one of the best writings on this subject was given by Simon Funk on his blog about his Netflix Contest adventures. Simon thought a creative way to find features would be to use the matrix transformation process of Singular Value Decomposition. Traditionally Singular Value Decomposition was used in the microelectronics industry to improve digital signal processing. Simon wrote up an easy solution for matchmaking movie features with the SVD method which spurned a wave of enthusiasm in the Netflix Contest entrants.

Finding feature sets is not exclusively in the realm of Linear Algebra. There are also methods of clustering, regression, support vector machine, neural networks, bayesian networks, and decision trees just to name a few. The science of matchmaking is closely related to artificial intelligence and is commonly referred to machine learning. Machine learning is using algorithms and mathematical methods to evolve and generate behaviors from data in order to solve a problem.

Processing the Matchmaking Data

The science of matchmaking would not be complete without the data. The advent of the internet has opened a lot of new enterprises that makes use of millions of data observations. These internet companies have a lot of data to process in huge server arrays that will make even the ENIAC envious. So how do these companies process all of this matchmaking data with their matchmaking algorithms? The basic answer is to break it down into manageable chunks. Perhaps no greater example is Google and their MapReduce methods. MapReduce is a software framework process that takes a large computing need and breaks it down into a distributed network that is more manageable. The first step in the MapReduce process is to Map the data. The Map process is to organize and distribute the data to computing nodes, usually a huge cluster. The Reduce process is to apply the algorithm or learning process to a node in the network and determine an answer to the data its given. This essentially gives it a local optimum. This process is iterated until a globally learned optimum is achieved. This is a very cut and dry description but you get the idea.

The MapReduce software framework is proprietary to Google. That has not stopped software enthusiasts. An open source MapReduce method was created called Hadoop and is growing into a stronger user supported community.

So what can be used with the science of matchmaking? Really anything the heart desires (okay, again, that was bad). Amazon.com uses recommendation algorithms for its books and products. Online dating sites (how appropriate) uses matchmaking methods for matching interested daters. Search engines like Google uses matchmaking algorithms, known as PageRank, to match search keywords with websites. As you can tell these types of enterprises are doing very well thanks to the science of matchmaking.

This article is part of the INFORMS Online blog challenge. February's blog challenge is Operations Research and Love.

Data Mining Books List

2011-02-09T06:42:00.000-08:00

I came across a great list of Data Mining Books while perusing around the internet. The list is maintained by Kurt Thearling who is Director and Chief Scientist in various organizations helping to develop their Analytics and Engineering groups. Kurt has written some white papers on the subject of Data Mining and has also been featured on NPR. Kurt's NPR piece was about data mining and privacy which is obviously a big subject in our Facebook society today.

I believe this is probably one the the most comprehensive lists of Data Mining books available. If you are interested in obtaining one of these books please be sure to peruse the the new IEOR Tools Online Store Data Mining section. There you can find books and references on Data Mining with varying levels from introductory to advanced applications.

Operations Research and gerrymandering

2011-01-27T20:58:00.000-08:00

INFORMS blog challenge for January involves Operations Research and politics. While public debate on politics will never change perhaps one thing that can change is the involvement better decision making to solving some of the political discourse.

One area of politics is the topic of gerrymandering. For those that are not familiar to gerrymandering it is the process of resetting electoral boundaries for voting purposes. Gerrymandering is a hot topic around any election because it is usually the party in power controls the rights to reset the electoral boundaries. This leads to an obvious advantage to the party as they can maintain a seat in a legislature with setting boundaries based on past voting behavior.

Operations Research can be a valuable asset to the process of redistricting. In fact Operations Research has been very much involved in redistricting for at least 50 years. Decisions to draw electoral lines can follow any number of constructions including demographics, population centers, municipality boundaries or industry types. As information abounds more freely there is more opportunity to use it for decision making. It seems every new census brings more available data. The growth of the internet has allowed information to be available more openly. Opportunities should grow in Operations Research to provide redistricting decision makers the information for informed analysis.

Perhaps one of the better uses of Operations Research could be the ethical context of the gerrymandering debate. I have often heard it debated that Operations Research may have created the politically polarized country we have today. The same tools of Operations Research could be used to allow transparency in the redistricting process. It could be useful for citizens to know how probable or likely outcomes of elections based on redistricting suggestions. Websites like OpenSecrets.org shows how money influences party affiliation and elections. Perhaps similar websites can emerge on electoral districts and the legislation that helped create them.

The debate of gerrymandering will last for centuries I am sure. I believe Operations Research can play a vital part in the debate. Information is more open and easy to access than ever. Let's use that to our best ability and help inform the voting electorate.

Statistics blog roundup from the Internet

2011-01-21T06:03:00.000-08:00

For those of us that want to keep up with the latest in the world of statistics here is a great list of 40 statistical blogs on the internet today from bschool.com. Some of the blogs even include R (link) which is my favorite statistical computing environment.

I have a few that I read on an occasional basis. Those include

R-Bloggers - an R blog aggregator that is very useful for R enthusiasists

Revolutions - blog by David Smith of Revolution Analytics, an enterprise ready R support and developer company

The Numbers Guy - Wall Street Journal blog about, you guessed it, numbers and statistics.

As you can tell I have been doing a lot of statistical work lately. Most of my efforts have been in the realm of predictive analytics. These blogs have come in useful in finding valuable references for predictive analytics. One book that is often mentioned in these statistical blogs is Applied logistic regression (Wiley Series in probability and statistics). I highly recommend this book as a statistical reference for logistic regression analysis.

Applied Logistic Regression takes the reader through an understanding logistic regression model and its usage for modern prediction analysis. The book starts with and understanding of log odds and introduction to the logistic regression model. Then the book goes into detail of the model parameters, coefficients, and estimation. Strategies for model building and case studies are also referred in this book.

Video of Joy of Stats by Hans Rosling

2010-12-31T13:29:00.000-08:00

The Joy of Stats narrated by Hans Rosling was just produced by BBC and shown to their audience. Hans Rosling via gapminder.org was kind enough to post the full hour video of the documentary about joys of statistics. The video is posted on YouTube and is available to anyone.

http://www.gapminder.org/videos/the-joy-of-stats/

Hans Rosling's passion for statistics is infectious. He definitely has a joy about him that persuades the viewer to really enjoy finding new and invigorating ways to explore data. Now for me this is not hard to do as I love data and analyzing. Yet for many in the world mathematics, let alone statistics, is considered a universe all unto its own that they dare not search. Hans breaks down that barrier with The Joy of Stats. No matter your educational interests or background I find it very hard to ignore his plea that statistics is not boring, and dare I say it, sexy.

If you are interested in this video as a eulogy to statistics you would also enjoy Dr. Robert Lewis's essay on Mathematics. Both of these works explain how a world without number analysis is merely a world not worth living. There is so much to explore in so little time. I am so happy that I decided to take a career in Engineering and Operations Research to help the world one datum at a time.

An essay on Mathematics and education

2010-12-22T20:07:00.000-08:00

I just read an essay on Mathematics by Dr. Robert Lewis named "What Math?" This has to be the single best essay I have ever read on Mathematics and why it is so important. I can imagine Dr. Lewis has been peppered with cynic criticism about understanding math just as much as any of us in our analytical profession. Dr. Lewis does a superb job of explaining the essence of why those of us that use math love it so much.

Educating Math to our newer generations is definitely a concern. I really like how Dr. Lewis explains that education is not just about transfer of information but the understanding of underlying principles of specific knowledge. The parables are a very clever device to relay those principles of math.

I also love how he portrays Math as not just a device for the technologically minded but also for the liberal arts. Dr. Lewis conveys that Math is not merely knowing numbers but the processes of finding solutions. My own example is when people often ask me how I am so good at math. I usually tell them its just like learning a language. Once you understand the language and are fluent then you can start applying it in everyday life. Math is a language to learn just as much a foreign language. It may take some time to learn but it will take a lifetime to master.

I highly recommend reading this essay. I also recommend saving this essay for our future generations, teachers, educators, family members, and friends. This essay can be used to help bridge understanding that may be missing from our own words.

New OpenOpt/FuncDesigner quarterly release

2010-12-16T23:33:00.000-08:00

New OpenOpt and FuncDesigner quarterly release is out: 0.32.

OpenOpt:
* New class: LCP (and related solver)
* New QP solver: qlcp
* New NLP solver: sqlcp
* New large-scale NSP solver gsubg. Currently it still requires lots of improvements (especially for constraints - their handling is very premature yet and often fails), but since the solver sometimes already works better than ipopt, algencan and other competitors it was tried with, I decided to include the one into the release.
* Now SOCP can handle Ax <= b constraints (and bugfix for handling lb <= x <= ub has been committed) * Some other fixes and improvements

FuncDesigner:
* Add new functions removeAttachedConstraints, min and max
* Systems of nonlinear equations: possibility to assign personal tolerance for an equation
* Some fixes and improvements (especially for automatic differentiation)

See also: Full Changelog, Some Applications, Future Plans

Where to find good data sets

2010-12-16T06:26:00.000-08:00

O'Reilly Media has been a big advocate of Open Data and believes that is where a lot of computing is going to be headed in the future. I think they are definitely on to something. Yet the future could be now. There is a lot of opportunities to find good data sources immediately. One of my favorite blogs, OReilly Radar, has an article by Edd Dumbill on Where To Find Data. There is plenty of good data available on the internet for download to explore and mine new information. These places not only offer great sources of data but many of them offer an API to allow quick and seamless access. Below is a link summary from the article.

Freebase

An all-things graph database. The website focuses on trends of certain cultural and interest topics.

Amazon Public Data Sets

Amazon is probably considered the cloud computing mecca next to Google. Amazon Web Services offers a lot. One of which is storage of public data sets. They offer a huge variety of public data.

Windows Azure Data Marketplace

Surprisingly Microsoft has an open data protocol data source. This data market offers quite a few points of interest data sets.

Yahoo Query Language

YQL is an interesting API that is very similar to SQL. YQL is essentially a language that allows to grab data from cloud services. This could be very handy to grabbing data quickly and dynamically. YQL offers to connect to a lot of data sources as well.

Infochimps

Infochimps is a data marketplace warehouse. They offer to host, sell, and distribute data sets. Some of their data comes at a cost but a lot of their data is free as well. This is an interesting startup and will be very interesting to follow their growth. Also there is a new Infochimps R package that uses their API to gather data and process Infochimps data.

DBpedia

DBpedia is a wikipedia for data sets. In fact the data itself comes from Wikipedia.

Some other sources not from the article include the World Bank open data and the U.S. Census data.

Shortest past algorithm solved by ants?

2010-12-12T15:31:00.000-08:00

University of Syndey researchers are working on the next greatest optimization algorithms. You would think they would be hunkered down in the math or computer science departments working with large multi-core processors. Yet Chris Reid and Madeleine Beekman, working with David Sumpter of Uppsala Univ., are studying how ants solve the shortest path problem. By studying how ants solve complex and dynamic problems such as getting food back to their colony they could unravel some new and innovative ways to solve routing problems. The researchers published their results in Journal of Experimental Biology.

There has already been some algorithms developed out of studying the ants. One method is the Ant Colony Optimisation (ACO) algorithms. Ants solve the complex problem of shortest path by communicating to other ants in the colony by pheromone trails. Each ant leaves a pheromone trail as a signal back to a following ant. The trail has a certain "optimal path" signal telling other ants the best way to get to the intended destination.

It would be really interesting to find out that the best shortest path algorithm might have been literally under our noses the entire time. This will be an interesting study to follow for the Operations Research community.

2 years of blogging with IEOR Tools

2010-12-08T19:33:00.000-08:00

I forgot I was going to make a mention but on Nov. 21 was officially two years of blogging about Industrial Engineering and Operations Research Tools. I have really enjoyed writing about this space and reading all of the contributions. I have no intentions of quitting and hope to make many more contributions.

An update to the blog is that I'm starting to contribute Amazon content to the site. Amazon has been a valuable resource for linking books on content matter. I've thought about adding a website that will be a "store" or compilation of some of the better resources with Amazon being a partner. I thought I would bring this up with the readers first to see if this would be a valuable addition to this blog. It would be a clearinghouse or aggregator for all the best tools and resources in Operations Research, Industrial Engineering, Analytics and Data Mining. I'm not sure there is anything on the internet besides doing searches in Google or Amazon. I hope the site would be nice layout to help easily find resources.

Since it is the holiday season I would like to send my warmest regards to all those reading. I thank you so much for your readership. I wish you and your family a safe and happy holidays.

Big Data Logistic Regression with R and ODBC

2010-12-07T19:02:00.000-08:00

Recently I've been doing a lot of work with predictive models using logistic regression. Logistic regression is great for determing probable outcomes of a independent binary target variable. R is a great tool for accomplishing this task. Often times I will use the base function glm to develop a model. Yet there are times, due to the hardware or software memory restrictions, that the usual glm function is not enough get the job done.

A great alternative to performing usual logistic regression analyses on big data is using the biglm package. Biglm performs the same regression optimization but processes the data in "chunks" at a time. This allows R to only perform calculations on smaller data sets without the need for large memory allocations to the computer. Biglm also has an interesting option that it not only can perform calculations on imported dataframes and text files but also database connectivity. This is where the helpful package RODBC comes in to the aid.

I have be looking all over the R support lists and blogs in hopes of finding a good tutorial using biglm and RODBC. I was not successful yet I was able to find out how to perform this myself.

The first step is to establish an ODBC source to a database. In this example I am using a Windows OS environment and connecting to a MS SQL Server. An odbc source must first be setup on the computer. This is usually done in the Windows Control Panel. Once that is done then RODBC can be used to establish a connection. My example is an odbc data source name called "sqlserver".

library(RODBC)
myconn <- odbcConnect(sqlserver)

Now an ODBC connection object is established. Queries can now be submitted to the SQL Server via the sqlQuery function which is what we will use as the data source. SQL scripts can be the typical select statements.

sqlqry <- "select myvars, targetvar from mytable"

Next is to use the bigglm function to perform the logistic regression.

library(biglm)

fit <- bigglm(targetvar ~ myvars, data=sqlQuery(myconn, sqlqry), family=binomial(), chunksize=100, maxit=10)

summary(fit)

The data is being pulled from the SQL Server via the sqlQuery function from the RODBC package. The bigglm will recognize the sqlQuery data as a dataframe. The chunksize specifies the number of lines to process at any time. The maxit value specifies the maximum number of Fisher scoring iterations.

This big data method is not difficult once the SQL Server connection is setup. You will notice that R will not have the memory limitations while performing logistic regressions. For more information on regression modeling I recommend getting Applied logistic regression (Wiley Series in probability and statistics). This book has been an essential for statistics and applied regression practitioners. Other good resources are Logistic Regression Models (Chapman & Hall/CRC Texts in Statistical Science) and Modern Regression Techniques Using R: A Practical Guide.

INFORMS Data Mining Competition leaders used Open Source software

2010-11-29T07:10:00.000-08:00

The results of 2010 INFORMS data mining competition just recently finished. The leaders were presented at the 2010 Annual INFORMS Conference. The 2010 INFORMS data mining competition goal was to determine short term movements in stock prices. You may recall that IEOR Tools competed in this competition with not too glamorous results at the end. There was a lot to learn from this competition. Firstly that it seems trading price movement can be correlated to lags in prices very well. Most of the top leaderboard finishers used future information to determine an appropriate lag in the price movement.

The next most interesting thing is that all top 3 finishers used free and open source software as tools for the competition. Two of the leaders used R and the second place finisher used Python, namely SciPy. This should not be surprising to most people in the analytics community. Open source software has been making inroads for quite a while. The R-Project has been getting a lot of interesting press lately especially in enterprise business circles. Python is an objective oriented programming language that is getting more popular. Python's popularity seems to be to its ease of use and how quickly it can be learned and implemented.

The presenters of the 2010 INFORMS data mining competition were kind to post the methods of the Top 3 competitors. Each method is an interesting read on how they were able to use the open source tools to get predictive results of stock price movements.

If you are interesting in learning more about R as a tool I recommend a new book by Luis Torgo "Data Mining with R: Learning with Case Studies".

This book is one of the first in its kind of showing R methodologies with real life applications. I am intending to get the book and hopefully have a review on it in the near future. I am already hearing good things about it.

R references for handling Big data

2010-10-27T12:06:00.000-07:00

The Dallas R User Group had a meeting over the weekend. One of the discussions is the memory limitations with R. This is a common subject among the R community and R User Groups. There has been a lot of strides recently in allowing R to stretch its memory limitations. I thought I would compile and share some of the best resources I have found to remedy the big data issue.

CRAN Packages
ff
This package allocates hard disk space to big data vectors.

bigmemory
This package allocates points to unused memory or points to a swap file.

Blog Articles
Taking R to the Limit: Parallelism and Big Data

Hitting the Big DataCeiling Limit in R
While this is not a helpful article for big data it does show some of the issues R current faces. Namely the issue of that lack of a "int64" or Long Long data type memory allocation.

Enterprise Software
Revolution R Enterprise
Revolution Analytics is creating enterprise software around R to tackle issues of big data, parallelism and threaded computing in order to speed up large data processing and analytics.

R Links for the Beginner on World Statistics Day

2010-10-20T06:32:00.000-07:00

In honor of the first World Statistics Day I thought I would share some of my favorite R links. R is a free software statistical computing environment for performing all sorts of data and mathematical manipulation.

Introduction and Tutorials

R Tutorial Series and Introduction
Burns Statistics Tutorials
Introductory R Tutorials
Learning R

Videos about R

Videos on Data Analysis with R

Graphics with R

Advanced Graphics
R Graph Gallery

Data mining competition with R

2010-10-08T06:31:00.000-07:00

There is a new data mining competition aimed at predicting preferred data mining tools in R via dataists.com. The concept of the competition is to try to determine which R packages are preferred in the R community via their CRAN package libraries. The developers of this new competition are also in the R community with the NY R Users Group.

I am a user of R and I am also a member of the Dallas R Users Group. As you can imagine I find this competition very interesting because I could benefit greatly from knowing the preferred methods of implementation in the R community. This can also be a very interesting exercise to determine preferred modeling methods. I believe this competition will give an insight into the most common methods to apply statistical computing in the community today.

R is getting a lot of press lately. Revolution Analytics just released the first part of a series of articles on the impacts of R and why it is hot. R as a statistical and optimization tool is really making a play in the business community.

I am a big fan of these data mining competitions as I've written previously. It is really interesting what you can learn about the world from the competitions especially if you do not have previous knowledge of the subject matter. I will try to bring up these competitions from time to time. I would be interested to know if any IEOR Tools readers have participated in these competitions and what was their experience.

IBM's furious Analytics aquisitions

2010-09-21T06:05:00.000-07:00

Is anyone keeping up with IBM and their propensity to obtain Analytics based companies? Let's see if we can do a recap of IBM in the news in the last couple of years.

Acquires COGNOS for $5.0 billion in November 2007
Acquires ILOG for $0.34 billion in July 2008
Acquires SPSS for $1.2 billion in the same month July 2008
Acquires Netezza for $1.7 billion in September 2010

In actuality there is an estimated $12 billion dollars of 23 Analytics based companies that IBM has acquired in the last few years. That is quite a leap for a hardware/software/IT company. IEOR Tools has talked about IBM's emergence as an analytics company before with new analytic centers and acquisitions. I think its safe to assume that IBM is the de-facto analytics champion in the world right now.

So what does this mean for Operations Research and their professionals? I believe it means the sky is the limit now. This is a grand opportunity for the Operations Research community. In fact I would even say that if INFORMS does not take advantage of the recent demand for analytics and decision sciences then they are missing this big picture. Jobs should be plentiful in the foreseeable future. There should be plenty of work to keep management happy and help drive value into organizations. This may even be the dawn of a new day for Operations Research and Analytics. There is so much buzz now it will leave a ringing in your ears. Sure I might be a little optimistic but I think any news is good news right now in this economy.

I also believe that IBM is not done. I think IBM is going to evolve even more in the Analytics realm. Perhaps getting more involved in the software within Operations Research and statistics. Its just a guess but who knows if SAS, Matlab, or even contributing to open source projects like R, RapidMiner or Weka. This is an exciting time none of the less for Operations Research.

Current Data Mining and Analytics Challenges

2010-09-16T06:42:00.000-07:00

I love the Data Mining and Analytics Challenges. There tends to be so much collaboration and open knowledge especially if the challenge has an affiliated forum. There really is so much to learn and the challenges offer a great way to bring all of the resources and knowledge together. Here is a list of the current challenges underway in the Data Mining community.

Kaggle is hosting a three competions. Tourism Forecasting part one challenges to predict 581 tourism-related time series. Chess Ratings - Elo vs the rest of the World is trying to determine a chess rating system that is better than the current Elo rating system. INFORMS Data Mining Contest challenges to predict intra-day stock price movements based on experts predictions, sector data, and other indicators.
TunedIT is another competition hosting organization. Currently TunedIT is hosting the e-LICO mutli-omics prediction challenge with background knowledge on Obstructive Nephropathy. Yes, I had to look it up too.
UC San Diego is hosting the 2010 UC San Diego Data Mining Contest. This is a two task contest which tries to predict e-tailer's data on consumer and non-consumer information. The two tasks are a binary preditor and a boolean-transformed predictor.

OpenOpt release 0.31

2010-09-15T02:19:00.000-07:00

OpenOpt 0.31:

Lots of new NLP, NSP (nonsmooth) and GLP (global) solvers from nlopt have been connected
New LP solver: pclp (very premature, but permissive license (BSD) and pure Python implementation)
Some bugfixes (mostly wrt using sparse matrices) and code cleanup

FuncDesigner 0.21:

New features: Integration, Translator
Some speedup for functions evaluation, automatic differentiation, optimization problems with some fixed variables
New parameter useSparse for optimization probs and automatic differentiation (sometimes autoselect works prematurely)
Some bugfixes (mostly wrt using sparse matrices) and code cleanup

DerApproximator 0.21:

Add parameter exactShape (prevents flattering operations on result)

Welcome to our homepage: http://openopt.org

Computer languages and Applied Math

2010-09-08T06:04:00.000-07:00

There is no question that computer languages have helped pushed the envelope for applied mathematics. It is hard to imagine where we would be without airline scheduling, supply chain management, or inventory control if it were not for all of the great advances in optimization and statistical computing. I have thought a lot about the convergence of computing and Operations Research. In fact I brought up a discussion on the topic on OR-Exchange with the question "Is programming skills a requirement for today's OR practitioner?" You would think with all of the advances in computing that programming would be simpler but that is not the case.

There is an interesting debate in the R-project community about the shortcomings of the R language. Xi'an Og posted a discussion on R shortcomings re-posted from another blog. The consensus of the R community seems to be that R is an inferior language but has a brilliant library of resources. So where does that leave the practioner? Does the practioner need to update their coding skills and develop something better in another computer language? I find it really interesting that some of the first solutions to this debate is to scrap everything and start over.

I don't think this debate is ever going to change. The computer is always going to be a valuable tool for the Operations Research practitioner. The tools we use to complete our daily tasks need to ubiquitous but also readily available. Let's just say that the slide rule is not going to be making any sort of comeback.

I believe that the Open Source model has a real advantage here over the proprietary counterparts in this debate. The community has a lot of input into Open Source software. It is often called a meritocracy. The best solutions continue while those that do not go away in obscurity in the Open Source model. This is one of the reasons why I advocate Open Source software. In the end I think R is going to be fine. There will be advances, possible even forks of the software, but there will always be progress. The only limitations seem to be of what we could dream.

What did the new PvsNP proof prove?

2010-08-20T06:56:00.000-07:00

I normally don't like to blog about mathematical theory. I usually leave that for the smart people and theorists. Yet there is an interesting article out this week from the Science section of the New York Times about the new PvsNP proof from Vinay Deolalikar. The article is not your typical topical subject matter about what the mathematicians are working on next. The article instead is about the explosion of activity and dialogue on the internet and around the world about this proof in the mathematical community. The author is suggesting that the likes of which have not been seen before with these types of theoretical discussions. I would have to agree with the author. I also find it very ironic that the Old Grey Lady is reporting on this as well. Since really the only thing that can really be proven, as this article suggests, is that the old media is nearly dead and the new media has supplanted it.

This brings another interesting thought to how problems are solved now. The mathematical community is closer now than it ever has been. This is the age of online crowd sourcing. If I have an question about Operations Research I go to OR-Exchange. If I am looking for a professional network contact I go to LinkedIn or INFORMS. If I need to read about the interests of the Operations Research communities I will go to their blogs. The convergence of ideas, thoughts, and knowledge is closer now and is only going to get even closer.

This article is one of the reasons why I am such an advocate of open source software with organizations like GNU and COIN-OR. Open source brings the best of bringing thoughts and ideas together to create a quality product. Sure there are licensing issues. This article is a good metaphor in that software licensing is like the "old media". Licensing is trying to catch up with the new technology but there are still a lot of kinks to work out. There are even suggestions now that software patents should even eliminated. I'm not sure what will happen but I do know that open source software is driving a lot of innovation in a much shorter time frame.

So yes I find it ironic that the New York Times is reporting on this proof as if it is new news. Maybe I'm just too close to the subject so I understand it a little better than the rest of the New York Times readers. Yet if you are anywhere near the mathematical world you would have already seen the proof and had your own conjectures. Even if that is the case we can prove now that information and knowledge is faster and easier to obtain than ever.

IEOR Tools Tutorial: Learning XML with R

2010-08-16T18:59:00.000-07:00

I have been using a lot of R lately in my work. R (main site) is an open source statistical computing platform. Saying R is only used for statistics does not do it justice. I am finding it to be a really powerful statistical and optimization computing platform. There seems to be no task that can not be accomplished. Lately I've been curious about measuring performance with my blog and how it compares to other blogs. So I thought I would use this opportunity to show how I performed this in R. I want to rank Operations Research blogs using the Alexa ranking system. Unfortunately Alexa does not have a search function for Operations Research blogs so I am going to have to get the information myself using R.

This R tutorial is going to be using the package XML. Packages are used in R to perform specific computational needs that the base R platform can not accomplish on its own. There are several different packages that can be loaded into R to perform a wide variety of problem instances.

The first step is to load the XML package into the current R workspace. If you do not have the XML package installed on your computer you will have to get it installed from the CRAN repositories.

After loading the XML package is where the problem set programming begins. I will need to save into the workspace the url of the Alexa information. Once I have the variables then I can move onto using the XML package to gather the information.

The main functions used in the XML package are htmlTreeParse, getNodeSet, and readHTMLTable. htmTreeParse grabs the XML code from the URL and stores it into an XML readable format. getNodeSet is a retrieval function that grabs only the data you specifify. In this instance it is looking for the XML nodes of dir and table with a id value equal to siteStats. The readHTMLTable takes the siteStats information and creates a table of data values.

While gathering the Alexa information with XML I'm also going to have to format the data into a readable structure. This will require tabulating and text string manipulation. Notice the use of the functions table, strsplit, and gsub to format the data. All of this is performed in a for loop that performs all of XML and text formatting one URL at a time. I've also created a data frame to place all of the relevant information into a readable table.

The following is the R code.

library(XML)

urlbeg <- "http://www.alexa.com/siteinfo/"

urllist <- c(
"industrialengineertools.blogspot.com",
"punkrockor.wordpress.com",
"thinkor.org",
"john-poppelaars.blogspot.com",
"bit-player.org",
"opsres.wordpress.com",
"orbythebeach.wordpress.com",
"spokutta.wordpress.com",
"engineered.typepad.com",
"bernoulli-on-business.blogspot.com",
"greenor.wordpress.com",
"fmwaves.kproductivity.com",
"blog.intechne.com",
"jimorlin.wordpress.com",
"jtonedm.com",
"mswd.wordpress.com",
"www.hakank.org",
"optandor.com",
"stochastix.wordpress.com",
"restart2.blogspot.com",
"scottaaronson.com",
"ateji.blogspot.com",
"geomblog.blogspot.com",
"ormsblog.com",
"wehart.blogspot.com",
"yetanothermathprogrammingconsultant.blogspot.com",
"annanagurney.blogspot.com",
"healthyalgorithms.wordpress.com",
"iaoreditor.blogspot.com",
"openresearch.wordpress.com",
"ornotes.blogspot.com",
"reflectionsonor.wordpress.com",
"arandomforest.com",
"analytics-magazine.com",
"hsimonis.wordpress.com",
"cpstandard.wordpress.com",
"blog.athico.com",
"dualnoise.blogspot.com",
"geneticargonaut.blogspot.com",
"john-raffensperger.blogspot.com",
"orforum.blog.informs.org",
"orinanobworld.blogspot.com",
"www.or-exchange.com",
"pomsblog.wordpress.com",
"research-reflections.blogspot.com",
"www.scienceofbetter.org",
"operationsroom.wordpress.com"
)

ORrank <- data.frame()

for (i in c(1:length(urllist)) ){
    url <- paste(urlbeg, urllist[i], sep="")
    doc <- htmlTreeParse(url, useInternalNodes=T)

    nset <- getNodeSet(doc, "//div/table[@id='siteStats']")

    tables <- lapply(nset, readHTMLTable)

    rankstr <- tables[[1]][2]
    rankstrdf <- strsplit(as.character(rankstr$V2), "\n")
    rank <- gsub(" ","",rankstrdf[[1]][1])
    rank <- as.numeric(gsub(",","",rank))
    tmpdf <- data.frame(ORblog=urllist[i], AlexaRank=rank)

    ORrank <- rbind(ORrank, tmpdf)

    rm(url)
    rm(doc)
    rm(nset)
    rm(tables)
    rm(rankstr)
    rm(rankstrdf)
    rm(rank)
    rm(tmpdf)
}
rm(i)

ORrank <- ORrank[order(ORrank$AlexaRank),]
rownames(ORrank) <- 1:nrow(ORrank)
print(ORrank)

Here is a final output from the ORrank data frame.

                                             ORblog AlexaRank
1                          orforum.blog.informs.org    154736
2                                 scottaaronson.com    308410
3                                    bit-player.org   1444318
4                                   blog.athico.com   1484646
5                                       jtonedm.com   1504658
6                      operationsroom.wordpress.com   1631529
7                             geomblog.blogspot.com   1711672
8                                    www.hakank.org   1955830
9                           www.scienceofbetter.org   2550459
10                           engineered.typepad.com   2625563
11                         stochastix.wordpress.com   3002085
12                         punkrockor.wordpress.com   3303052
13                       openresearch.wordpress.com   3811636
14                           hsimonis.wordpress.com   4068033
15                        fmwaves.kproductivity.com   4281627
16                        annanagurney.blogspot.com   5047922
17                              www.or-exchange.com   6052089
18                                      thinkor.org   6134442
19                           analytics-magazine.com   6674061
20                  healthyalgorithms.wordpress.com   7373428
21                  john-raffensperger.blogspot.com   8516473
22                            greenor.wordpress.com   8666209
23                       orbythebeach.wordpress.com   9437585
24                                arandomforest.com 12225347
25                               mswd.wordpress.com 12571553
26                                blog.intechne.com 13784064
27                           spokutta.wordpress.com 15236071
28                         cpstandard.wordpress.com 19401625
29                     geneticargonaut.blogspot.com 20064295
30                                     ormsblog.com 21294575
31                              wehart.blogspot.com 22329286
32 yetanothermathprogrammingconsultant.blogspot.com 24431355
33                           dualnoise.blogspot.com 25165358
34                               ateji.blogspot.com 25304653
35                    reflectionsonor.wordpress.com 27537074
36             industrialengineertools.blogspot.com        NA
37                     john-poppelaars.blogspot.com        NA
38                             opsres.wordpress.com        NA
39               bernoulli-on-business.blogspot.com        NA
40                           jimorlin.wordpress.com        NA
41                                     optandor.com        NA
42                            restart2.blogspot.com        NA
43                          iaoreditor.blogspot.com        NA
44                             ornotes.blogspot.com        NA
45                       orinanobworld.blogspot.com        NA
46                           pomsblog.wordpress.com        NA
47                research-reflections.blogspot.com        NA

Not exactly in the friendliest of formats but it does the trick. I hope that this will help others who wish to use the powerful XML package with R. I know I have definitely learned a lot about XML in the process. I also found out that I have a lot more work to do with my blog.

Note: If you are wondering where Michael Trick's blog is located there is a reason. Unfortunately his blog and some others are in a sub-domain of another url not affiliated with his blog. This means Alexa can not rank it compared to blogs with a primary domain. Yet everyone in the Operations Research community knows where Michael's blog ranks anyway.

Favorite Operations Research books from OR-Exchange

2010-08-08T12:55:00.000-07:00

A while ago I posted a question on OR-Exchange about some of the favorite Operations Research books that OR-Exchange members like to reference. I was rather pleased with the response. Of course there are great books on the subject of Operations Research. The best part of OR-Exchange allows for the community to vote up the favorites. A lot of these books are just plain good to have in your desk drawer or in your work study. I have to admit that I have not read all of these books. So this gives me a good excuse to go get them and perhaps offer up some reviews in the future.

So in order of OR-Exchange votes here are the favorite Operations Research books.

1. Applied Mathematical Programming by Bradley, Hax, Magnanti.

Applied Mathematical Programming

Also available at http://web.mit.edu/15.053/www/ but if you like it you might want to give it a purchase.

2. Network Flows: Theory, Algorithms, and Applications by Ahuja, Magnanti, Orlin

3. Linear Programming by Chvatal

4. Model Building in Mathematical Programming by Williams

5. Introduction to Operations Research by Hillier, Lieberman

6. 50 Years of Integer Programming by Juenger, Liebling, Naddef, Nemhauser, Pulleyblank, Reinelt, Rinaldi, Wolsey

7. The Traveling Salesman Problem: A Computational Study by Applegate, Bixby, Chvatal, Cook

8. Tabu Search by Glover, Laguna

9. Prisoner's Dilemma by Poundstone

10. Serious Play by Schrage

11. The Fifth Discipline by Senge

12. The Predictioneer's Game by Mesquita

13. Optimization Algorithms for Networks and Graphs by Evans, Minieka

Kaggle introduces new Chess rating competition

2010-08-05T17:32:00.000-07:00

Kaggle, home of the statistics and predictive modeling competitions, is introducing its latest contest Elo Versus The Rest of The World. The competition is being organized by Jeff Sonas who is a chess-metrics afficionado himself. Jeff describes his history with rating chess players and why he wanted to start such a competition with Kaggle.

This looks to be a really interesting modeling competition with already more than 40 submissions in the leaderboard. The interesting note about this competition is that the Elo rating system itself is going to be making an appearance on the leaderboard. This means that if no one beats the Elo system than there is no declared winner. Although it looks like someone has beaten Elo at its game already. Elo will be on the leaderboard as a benchmark to make sure that the competition is proving its worth.

I hope to get a chance to make an appearance on the leaderboard. I am involved in Kaggle's INFORMS 2010 Data Mining contest. I'm barely hanging on to the top 10 in that competition. There are some pretty good models to compete against in that group.

New Look for IEOR Tools

2010-08-03T19:51:00.000-07:00

I think I'm following the trend of a lot of OR bloggers and updating the look of the site. I'm not sure if it is a more modern Web 2.0 look but at least its a little more refined. I finally got around to fixing the font on the title. Anyway let me know if you like it.

R IDE for Linux and Gnome

2010-08-03T06:45:00.000-07:00

I have been using R in my work recently. I have also been using R at home to do some tinkering. In my work environment I use Windows (none too pleased). I find using the regular R console with Textpad makes for a good Windows development environment. I haven't been able to replicate this at my home. At my home I have Ubuntu as my operating system. I have been searching for a comparable R environment for my home.

That is until now. The statistics blog at Stattler.com did some research on this very topic of R and Ubuntu. They found a plugin for R with the text editor Gedit that works wonders. The plugin is called Rgedit and is very easy to install. Stattler offers a simple instruction for installation. Also Stattler has a great review of the Rgedit plugin. Rgedit is very similar in layout to usual gedit text editor except it splits the panes of the screen for code and R output.

Some of the highlights of the Rgedit plugin include:

Split screen of panes and can be turned on and off
Syntax highlighting specific to the R code
Single line or batch processing of R scripts
Multiple R workspaces can be run
Shortcut keys can be created and customized

This plugin suits my needs just fine for my Ubuntu uses with R. There are many other IDEs for R that you may find suit your needs better. The the beauty of open source software is that there never seems to be a shortage of options.

Eight Data Mining Social Networking Groups

2010-07-31T16:39:00.000-07:00

Visit AnalyticBridge

Networking is an essential part of career management for any professional. The relationships we develop can have great impact on our career direction and growth. I tell young professionals all the time that their best asset in career growth is their professional network. I tell them to start early and maintain the network continually. The advent of LinkedIn and other internet social networking sites has made that task easier.

Vincent Granville at AnalyticBridge.com has compiled a list of 8 data mining social networking groups with more than 2000 members. These groups are easy to become as member as all of them are associated with LinkedIn. If anyone is finding anyone in the data mining community more than likely they will be a member in these groups.

I have found a couple of different jobs through LinkedIn in the past. I have found hiring managers as well as peers that I would be potentially working with closely. I found that to be a great benefit before the interview process. In fact I even would contact some of the peers in the group to get a pre-interview idea of where I was working and the idiosyncrasies of the organization. The important thing is using your professional network to maximize your career productivity.

Audio of Richard Stallman keynote at useR2010

2010-07-27T06:17:00.000-07:00

useR 2010 Conference

The topic of open source and free software licensing can be a very confusing topic. In the Operations Research world it is no different. There are a lot of players in the mathematical programming software world that are vying for attention that include both proprietary and free software. Insight into the world of free software really requires immersion into using the free software products and finding how it can apply to daily application. Another good way to understand free software is to get it from the founders of the movement. Richard Stallman is considered the father of the free software movement and you can find a lot of good material online based on his work. The useR2010 conference, the annual conference for R project for statistical computing, just completed this past week and the final keynote was given by Richard Stallman.

The R-statistics blog was kind enough to post an audio of the keynote address by Richard Stallman at useR2010. Richard is not your typical stereotype of a computer geek. He may look the part but Richard does not pull any punches in his presentation of the free software movement and it's ideology. Richard's talk discusses the history of the free software movement, the Gnu Public License, and his history of dealing with free software.

Why was Richard giving the keynote address when he doesn't have a statistics background? Well the R statistical computing software platform is licensed under the GPL, Gnu Public License. R is free to use, distrubute, modify and improve as long as it's code is given credit to it's creator. This is much of what the GPL represents. Listen to the audio by Richard to really understand his passion for free software and what it means to him and the software world.

Podcast with Revloutions Analytics CEO Norman Nie

2010-07-14T06:24:00.000-07:00

Through the Revolutions blog there is a really interesting podcast about R by Internet Evolution Radio interviewing Norman Nie. Norman Nie is the CEO of Revolutions Analytics which I posted about in the past on how Revelotion Analytics is going to take R commercial.

In this podcast Norman is asked a lot of interesting questions about R and the statistical modeling enterprise in general. They discuss his past with SPSS. They also discuss the advantages of using an Open Source software versus a proprietary platform. The interview gets really interesting when they discuss how statistical data is important to enterprise business and how a lot of organizations get it wrong.

If you are new to R and want to know more about its capabilities this is a great podcast.

OpenGamma startup claims Open Source Financial Analytics

2010-07-13T06:27:00.000-07:00

According to h-online.com a new startup firm, OpenGamma, is preparing to launch its Open Source Financial Analytics solutions. OpenGamma is a London based firm that will specialize in risk management and financial markets by providing software architecture. From the article, this is a quote from the CEO Kyle Wylie

"Our goal in building OpenGamma isn't just to build an open source technology" said Wylie, "Our goal is to build the best platform for financial analytics and risk management possible". The platform will be made available under a "commercial friendly open source licence"

OpenGamma is going to provide several solutions based on its Open Architecture software platform. According to their website they will be providing batch risk systems, commercial trading, bespoke trading, and event-driven alert systems. The companies main moniker and selling point is that all of their software code with be Open. This means that companies will have the flexibility to not only debug but potential contribute back to the project, in theory.

This sounds like a brave yet prudent business venture. I think OpenGamma could be wildly successful with their Open Architecture platform. They can be especially successful if they allow the financial analytics community to contribute back to their software platforms. I believe we will be seeing more companies and startups like this in the future in Analytics. Perhaps there is an Open Source Operations Research platform on the horizon.

Open source solver for Excel

2010-07-07T10:30:00.000-07:00

Thanks to a post by Michael Trick we find that the open source solver community has an new platform with Excel. Although its not a replacement for Solver the OpenSolver does offer a lot of benefits that the existing Excel optimization platform does not perform. OpenSolver is an extender of the existing Solver. You will still need to use Solver to develop the model. Yet OpenSolver can take over from there. Some of the benefits include

COIN-OR CBC optimization engine to perform the calculations
Compatible to existing Solver models
No artificial limits to the size of the problem (huge win here!)

Some of the disadvantages are that it does not solve non-linear models. Also it does not run as a stand-alone plug-in as the current Solver. I believe that OpenSolver is only developed for the Excel 2007 platform. OpenSolver was developed by Andrew Mason and is licensed under GPL. OpenSolver is free to distribute and download.

I'm hoping to give this a try soon. I'm really encouraged by OpenSolver because I always thought the current Solver was very limited. Looking forward to great things from OpenSolver. For other ideas about Open Source solvers with spreadsheets be sure to look at Open Office Calc.

Kaggle hosting INFORMS 2010 Data Mining Contest

2010-06-29T10:53:00.000-07:00

Kaggle is hosting the 2010 INFORMS Data Mining Contest. The goal of this years INFORMS Data Mining Contest is to predict intra-day stock price movements. All data and submission guidelines are provided on the Kaggle website. Entries that are submitted are immediately scored and evaluated by an AUC calculation. The leading AUC score by the end of the contest is going to be honored as the annual INFORMS meeting which is in Austin, Texas (Nov. 7-10).

There is already a lot of good discussions of modeling techniques. Mark started off with a question on OR-Exchange about modeling methods for the INFORMS contest. Since the data is a binary categorical target his preferred method was using Logistic Regression. Mark provides example R code to provide collaborative input to the contest. I followed suit and provided an IEORTools entry to the contest. I used the same methods of Logistic Regression. I also did some variable analysis using the rpart package in R to develop a decision tree. After pulling some variables that were not significant I was able to get on the leaderboard with Mark. The pictured leaderboard is of June 28.

There is also some good discussion on the Kaggle website contest forum. Posted on the forum one entrant suggested possible variables to use in a Logistic Regression model which is very beneficial.

I really like to see this collaborate effort to modeling. This was one of the qualities I really enjoyed in the Netflix Prize. I hope Kaggle and INFORMS continues to provide these fun and thought provoking contests.

U.S. SEC endorses Python to fix financial problems

2010-06-25T17:13:00.000-07:00

News from PCWorld is mentioning that ActivePython, the software distrbution from ActiveState, is going to include numerical, scientific, and optimization software with its current software bundle. The numerical and optimization software it is going to include is Python based NumPy, SciPy, and matplotlib. All of the new software is open source and available for free download.

Apparently this is in anticipation to the new U.S. financial rules from the U.S. Securities and Exchange Commission. On April 7, 2010 the U.S. S.E.C. proposed new rules for Asset-Backed Securities that will enable the markets to run efficiently and fairly. On the first page of the released documents from the S.E.C. they mention the use of Python. That is a nice shocker to us open source advocates.

Python is a great computing language. It is really easy to learn compared to the other languages such as C. Perhaps the U.S. S.E.C. thought it would be the best choice because of its ease of use and abundance of software packages. This is really interesting news and hopefully we will be hearing more about it in the near future.

R package for World Bank Data

2010-06-24T06:02:00.000-07:00

A little while ago I posted about how the World Bank data is open to the public for research. This apparently is just the beginning of what is possible with having free access to a lot of really good data sets on socio-economic information. R-chart blog just recently posted saying that an R package was developed as an API to access the World Bank data.

This opens up a lot more data mining opportunities and could just be the start of some great analytic research. I'm really looking forward to seeing what some of the great R minds will find with the World Bank data at their fingertips. Since R is freely available anyway this merger makes sense on all sorts of levels. Happy data mining!

Software for Data Analysis: Programming with R (Statistics and Computing)

Analytics and FIFA World Cup

2010-06-16T19:16:00.000-07:00

What would the FIFA World Cup be without the prognosticators? You can be assured that the Analytics community is not far away from the scene. There are plenty of places to find on the web predictions and analytics of the 2010 FIFA World Cup. Here are some of places on the web where you can find all of your World Cup analytics interests.

Wayne Winston is posting some predictions and rankings on his blog mathletics. If you are a fan of sports and analytics (i.e. Moneyball) than you would love Wayne Winston's blog. Wayne does predictions for professional and collegiate sports in basketball, football, baseball, and soccer.

Blog posts on AnalyticBridge tells that big financial institutions are using quantitative financial instruments used in credit swaps and and debt obligations for predicting World Cup outcomes. It is a corporate financial challenge that is trying to predict with country will go the farthest in the World Cup. Let's hope its not the same models that were used to predict mortgage backed securities from a few years back.

Spotfire's blog has an entry about providing World Cup data all the way back to 1930. TIBCO, Spotfire's parent, is providing analytic data from the World Cup including all sorts of statistics. Analysts can get scores, goals, penalties, attendance, and other data points. The online app that TIBCO provides also has nifty charts to compare different countries performance.

OpenOpt release 0.29

2010-06-15T23:54:00.000-07:00

New OpenOpt Suite release is out. This is free (license: BSD) and cross-platform (Linux, Windows, Mac etc) Python language modules for numerical optimization, automatic differentiation, solving systems of linear/nonlinear/ordinary differential equations etc. It is published quarterly since 2007, already has some essential applications and expected to become even more popular with Python release 3.3, where dynamic compilation will be implemented.

OpenOpt 0.29:
* Some minor bugfixes
* Some improvements for handling sparse matrices
ralg:
* Bugfix for problems with nonlinear equality constraints
* Major changes for problems with nConstraints>1

FuncDesigner 0.19:
* Some improvements for automatic differentiation
* New feature: attached constraints
* New feature: oosystem
* Now you can model & solve ODE systems

DerApproximator 0.19:
* Function get_d2
* Add new stencil

You can try it online via our Sage-server.
See also: Full Changelog, Future Plans

OR-Exchange confession

2010-06-01T19:51:00.001-07:00

I have a blogger confession to make about OR-Exchange. I am addicted. I think its the first Operations Research related social network that has me really hooked. I confess that I check it daily. Yes you can see that I've earned a silver badge for my continual obedience. Is this OR's Farmville? Well at least it is for me.

The premise for OR-Exchange is really simple. Think of an Operations Research related question that bugs you, puzzles you, or simply just want to get peer feedback. Shortly after, and I mean shortly, you will be barraged by answers from like minded individuals. The perfect storm that the Web 2.0 wants to fuel.

In my mind the beauty of OR-Exchange is that it is not any normal social network. This is a social network of peers that understand my issues, problems and concerns. Maybe its just my Generation X upbringing that requires instant gratification. Yet I don't need a whole lot of stimulating from other social networks. In fact I'm pretty much done with most of the others. The online media I keep going back to are the ones associated with my interests and for me that is OR-Exchange right now.

I love the feedback from the folks at OR-Exchange. Good, bad or indifferent it brings perspectives that I often don't get in my circles. In my present work I don't often get to chat up Operations Research with my co-workers. I'm one of two employees that has any knowledge of what is Operations Research. I guess that's where it benefits me. I'm hoping that it benefits others like us that either has to wait a year to go to an INFORMS conference. I am active in my local INFORMS Chapter but most of it is topical speeches and programs. OR-Exchange is more of an outlet which has filled a void for me.

I hope the Operations Research community can take on to OR-Exchange. I believe there can be only more good as more users come online. Please help preach its worth if you are using it. Perhaps there are many more unanswered questions in the Operations Research community.

My 5 favorite Operations Research blogs

2010-05-13T17:06:00.000-07:00

I do a lot of blogging in my spare time. I especially like to read up on blogs that really interest me. My passion is what I do for a living which is, of course, Operations Research and Industrial Engineering. I am often amazed of the great writing and resources available in the online Operations Research community. So I thought I would share on this blog my 5 favorite blogs that I am usually checking every day. These blogs are not listed in any particular order.

1. Michael Trick's Operations Research blog

You can pretty much argue that Michael is the "Father of Operations Research blogs". Michael does a great job of mixing academia and real world applications of Operations Research. Often times the comment section is worth the read with great contributors to his blog.

2. Thoughts on business, engineering, and higher education by Aurelie Thiele

This blog by Auriele is probably some of the best writing in the Operations Research blogs. I particularly love the issues that Auriele presents on a weekly basis. In fact I'm outright jealous of Auriele's insight. This blog is just a flat out good read.

3. Punk Rock Operations Research

Punk Rock O.R.'s writer Laura McLay is another good Operations Research blogger that mixes academia and real life OR interests. I enjoy Laura's commentary on a lot of issues that you might not expect in mainstream OR applications. I especially like Laura's interests in sports.

4. Sebastian Pokutta's Blog

This blog may not be one of the most popular blogs but I really like Sebastian's Operations Research blog. Maybe the fact that I really relate to Sebastian's ideas and his endorsement of open source software in Operations Research. Sebastian finds really good nuggets in the OR world that you don't often see on other blogs.

5. ThinkOR

This blog is perhaps one of the best in writing real life Operations Research examples. I really enjoy the thoughtfulness of this blog and writing of this blog. I enjoy reading about ThinkOR's writing style of sifting through real world problems and dictating possible solutions.

Algorithms and Wall Street

2010-05-10T07:54:00.000-07:00

The crazy events of Wall Street last week sent off a huge wave of confusion as to the events that led to the sudden drop in stock prices. At first it was thought to be a "fat finger" that cause the decline of major stock indexes. Now the focus is on the large trading farms of computers that are said to make trades by specific rules and algorithms. Now there is a question as to what are the underlying algorithms that these computers are trading. What was thought to be a no brainer of setting trades at the speed of electrons to make a more efficient market is now all being thrown into question.

I do not claim to understand the rules or algorithms that are programmed into these trading computers. Wall Street trading is not my area of expertise. Although I am curious at this overall crisis and how it could be the result of supposed computer rules. The U.S. government is interested also as they are investigating what caused the sudden drop. Can algorithms imposed to trade on a whim cause that much market capitalization to drop out so suddenly. There are claims that market values dropped by nearly 100% on long established companies like Accenture.

I'm definitely going to be following this story closely. I'm curious what the SEC is going to find in their investigation. I'm going to reserve my opinions until more facts are brought forth. Perhaps we may never really know what caused this crisis. I would hope that it is something the Operations Research community could learn. We know that algorithms can be developed to provide great benefits to people and organizations. Yet we hardly ever hear of the times when they can cause great trouble. We can learn from those bad implementations of algorithms. Usually at the heart of it is not so much a bad algorithm but the underlying assumptions of the model. We should know this all to well with the recent mortgage crisis. Perhaps this road to recovery out of this current recession is going to take a lot more time than we thought.

R has a revolutionary commercial launch

2010-05-06T20:18:00.000-07:00

R is going commercial and mainstream thanks to Revolution Analytics. Revolution Analytics, formerly REvolution Computing, is going to take R to the next level in predictive analytics and data mining for enterprise business. Many in the OR blog sphere is reporting on this move as it can mean big changes to the statistical enterprise software market.

Revolution Analytics is going to bridge the academic and business divide by providing solutions that were considered limitations to R in the past. They will be focusing on software enhancements that will be able to handle larger datasets. There is going to be better use of multi-core processing power. There is also going to be improvements to user-interfaces for business analysts.

R is a free and open source software environment for statistical computing and data visualization. I think it is too early to tell what this announcement is going to mean for the statistical enterprise software market. Revolution Analytics has already said that they will be mixing proprietary methods with R. This will be interesting to see how the R community embraces that relationship. R is licensed under the Gnu Public License which is supported by the Free Software Foundation. That is a crowd that does not take too kindly to proprietary software and patents. It will be interesting to follow Revolution Analytics and how they are able to implement their roadmap.

Railways improved by mobilizing Operations Research

2010-05-04T05:24:00.000-07:00

A great article by PhysOrg.com on the values of Operations Research implemented in the Dutch railways. Improvements were able to be realized in train arrivals, passenger utilization in the cars, and operating profit. Many countries were impacted by the improved railway service all across Europe including Netherlands, Germany, and Switzerland among others.

The team that implemented the Operations Research strategies for the railway improvement project is lead by Christos Zaroliagis, a professor of Computer Science and Informatics at University of Patras. Christos was part of the team that earned the 2008 Edelman Prize from INFORMS for "The New Dutch Timetable: The O.R. Revolution." The team of the ARRIVAL project is a consortium of several researchers from many European countries.

This is a great example of Operations Research in practice and how OR continues to improve the operations and lives of organizations. I really like sharing stories like this because I don't feel they often get their due respect. There is a lot of research and planning in the background of a good research project, let alone Operations Research, that does not get noticed.

Operations Research courses via Open Courseware Consortium

2010-04-28T11:20:00.000-07:00

Open Courseware is making it possible for anyone with an internet connection to find subject knowledge from many different academic institutions. The best part for knowledge seekers is that it is free and open to the public. Perhaps you would like to know about Quantum Physics or English Literature. There is a very good chance that lectures, notes, exams and class references will be available in a particular subject.

Operations Research is no different to Open Courseware. In fact there is an increasing amount of Operations Research, Management Science, Supply Chain, and Applied Mathematics available on the internet. The Open Courseware Consortium is one way to find open courses on the internet in Operations Research. The consortium themselves are trying to promote themselves with membership but to search for courses is free to the public by the institution. IEOR Tools has featured this in previous blog post about open courseware.

More Analytic Competitions

2010-04-27T06:44:00.000-07:00

In a follow-up to a previous IEOR Tools blog post on Predictive Modeling and Recommendation Challenges there is another organization opening up analytic competitions. Kaggle is an organization that is trying to bring together the best prediction modeling and statistical talent vis-a-vis analytic competitions.

http://kaggle.com/

Kaggle is also encouraging organizations to host a competition on their platform. They want to encourage companies to use them to find top notch predictive analysts.

There are two types of competitions promoted by Kaggle. The two kinds are predicting the future and predicting the past. From Kaggle's website...

The platform allows companies, researchers, governments and other organizations to post their problems and have statisticians worldwide compete to predict the future (produce the best forecasts) or predict the past (find the best insights hiding in your data).

The current contest is a European Voting.

Kaggle is taking advantage of the Netflix Prize and its success. The hope is that Kaggle can be a platform to bring these competitions together. It will be interesting to follow Kaggle to see if there is success in these open competitions for analytics. The results of the Netflix Prize seems to think that there will be a good indication.

World Bank opens data to the public for open research

2010-04-23T05:38:00.000-07:00

Press release this week from the World Bank Group states the World Bank will release free access to data. According to the article there is over 2,000 financial, business, health, economic and human development statistics available for free to research.

The World Bank has created a new website to access the free data at data.worldbank.org. Skimming over the Data Catalog shows a great amount of variety in the data sets. There are tables on Global Finance, Education Statistics, Poverty in developing countries, Gender, Business, Debt, Governance just to name a few.

This is very encouraging that the World Bank will offer data openly in this manner. Openness can be a great asset to the research community and help drive improvements and reform where needed. I definitely cheer the World Bank for allowing this data to become public.

OR-Exchange in the top 50

2010-04-18T15:14:00.001-07:00

Per Stack Exchange's directory of sites OR-Exchange has moved into the Top 50. This is good news for those in favor of keeping OR-Exchange a resource for the Operations Research community. There have been a lot of good questions asked this week. Although I threw a proverbial OR question about OR solvers some of the good ones include...

What file format for problem definition is suitable for OR-Exchange?

References for conjecture: Any regression can be translated into a math model

Please keep the questions and answers coming for OR-Exchange. For those that don't know about OR-Exchange I had a recent blog post about promoting this website for the Operations Research community.

OR-Exchange Needs Your Help

2010-04-15T02:59:00.000-07:00

Michael Trick, maintainer of the OR-Exchange site, is finding himself in a predicament. Apparently the owners of Stack Exchange that make OR-Exchange possible are changing their terms of service and providing an ultimatum to the lower trafficked pages. OR-Exchange will have to shut down if it can not meet the new requirements.

I'm a fan of OR-Exchange. OR-Exchange is a great place to share ideas with the Operations Research community and find answers to questions. OR-Exchange is in the model of digg.com and reddit.com where you can vote up the questions and answers that you find favorable. That means value content will always rise to the top for easy dissemination.

Some people may argue that there is already a lot of information on the internet with Operations Research and that I can't argue against. The value of OR-Exchange is the dynamic collaboration with the Operations Research community. There can be value in posting Q&A topics and getting the community to answer to vote up and comment.

I urge you to give OR-Exchange a try to help promote this project. Otherwise we might lose a great resource.

Math is the New Cool for Employers

2010-04-14T06:38:00.000-07:00

While parusing AnalyticBridge, Vincent Granville posted an article from the Wall Street Journal online that New Hiring Formula Values Math Pros. The WSJ article states that more and more companies are looking to statistics, data mining, and machine learning experts. Computer Science is waning compared to analytic experts. From the article...

The most desirable candidates, employers say, can have a variety of experience and educational backgrounds. Companies say specific degrees are less important than a focus on data-mining techniques.

This is definitely a trend I've been seeing. Companies want to see value delivered from their employees instead of just data management. I'm encouraged for the Industrial Engineering and Operations Research field and am looking forward to see how it takes off.

Anyone interested in a good RUG?

2010-04-12T07:31:00.001-07:00

R User Groups are popping up around the country. In Dallas there is a new R User Group as told by David at the REvolutions blog. If the Larry in the article sounds familiar than you are right! Here is where you can sign up for the RUG in Dallas.

http://tech.groups.yahoo.com/group/Dallas_RUG/

Also in Chicago they are getting things going with their own R User Group. REvolutions blog chimes in as well in an announcement for the windy city R users.

For those that don't know, R Project is a statistical computing environment very similar to S+ and SAS. It is free and open source and contains hundreds of free libraries and packages for statistical, optimization, predictive analytics, and data mining computing.

If you would like to get more interested in R in your region take a look at the REvolutions blog. REvolutions lists R User Groups all around the world. And if one is not in your area go ahead and get one started. It is a great way to network with professionals in your discipline.

DataMiningTools.com devoted to sharing data mining resources

2010-04-06T09:46:00.000-07:00

DataMiningTools.com is an up and coming website devoted to all things data mining. There is a lot of tutorials, videos, reviews, and recommendations for quality tools of the data mining trade. There is even a feature for Open Source tools which definitely gets my attention.

DataMiningTools.com seems to be to data mining what IEOR Tools is to Industrial Engineering and Operations Research. I really like the presentation of the content. The links are tagged really well and are easy to find relevant resources for data mining. In the future I hope to feature some of the tutorials from this website.

One area you may to look at is the R Project tutorials which has my interest as of late.

New IEOR Tools contributor - developer of OpenOpt

2010-03-26T10:44:00.000-07:00

I would like to welcome Dmitrey, developer of the open source optimization software suite, OpenOpt and maintainer of OpenOpt.org. Dmitrey will contribute from time to time about OpenOpt and software releases.

When I started the IEOR Tools blog I wanted it to be a forum for discussion, evaluation, and peer review of the tools available to the Industrial Engineering and Operations Research community. I believe that the best opportunity for continued development and research of tools of the trade is in the Open Source community. I hope that this blog can be a conduit for continued development and opportunity.

I'm excited for the contributions that Dmitrey will provide with OpenOpt and other open source developments.

Timetric, startup grearing up for a new statistics presentation platform

2010-03-25T10:17:00.000-07:00

Techcrunch is reporting that Timetric has closed seed funding for an innovative way to present statistics online. From the article, the money is going to be used to help get their statistical platform rolled out. Timetric's claim is that it will be able to present statistics in a more "useful" manner. Timetric already is working with online gurus such as Guardian and United Business Media.

It seems to me that this may be competing with the Wolphram Alphra platform. Another way to present public data on the internet. Yet this is also showing an increased trend in the importance of data visualization. I'm encouraged to see there is competitors in the data visualization market. It definitely shows that there is a need for that data management and presenting it in a meaningful way. Sounds a lot like Operations Research to me!

I really like the layout of the website. Any website that focuses on visualization should be sharp and refined. Curious to see how Timetric proves out in the future.

R Project in Google Summer of Code 2010

2010-03-19T05:37:00.001-07:00

R Project, the open source statistical and mathematical computing environment, is going to be a part of the Google Summer of Code 2010. There is a an R Wiki page devoted to topics and projects within the R Project for the Google Summer of Code. The assortment of projects range from mathematical and statistical research oriented to computer API and interface.

The Google Summer of Code is a student internship program that provides stipends to develop free and open source software around the globe. The GSOC has been in existence since 2005 and has allowed thousands of students to work on hundreds of computing projects of interest. For a complete list of the GSOC open source organizations can be found on their project site.

10 Linux Productivity Tools

2010-03-18T15:12:00.000-07:00

For anyone who reads this blog it is no mystery that I am an Open Source and Free Software advocate. I have my reasons, which are many, I have previously posted on the IEOR Tools blog. That being said I like to find software for my favorite free operating system Linux. I know, Linux by itself is not an operating system but the kernel. I'm just referring to the "flavor" of the operating system which uses Linux.

There are some great productivity tools for Linux that can help any Analyst or Engineer. Linux.com does a great job of reviewing some of the best available productivity tools for Linux. I am a big fan of Kontact, Ocular, and Kivio. I've mentioned before that Kivio is a great free software diagramming tool.

These applications can help improve productivity in the Linux environment. Often times folks feel the Windows corners the market on these types of applications but that is not often the case.

Predictive Modeling/Recommendation Challenges

2010-03-15T06:13:00.000-07:00

The absence of the Netflix Prize may leave a gaping hole in some of our leisure activities. Well maybe not that big of a hole but at least some of our thought provoking lapses of time. If you are an avid modeler and really want to stretch your data mining and predictive modeling skills to the limit than there are other ways to get that accomplished.

Here are some other notable predictive modeling contests available compliments of KDnuggets.

http://www.kdnuggets.com/datasets/competitions.html

A notable competition is the Analytics X Prize which aims at solving social problems within our world. The current prize is predicting homicide rates in Philadelphia. A bit morbid but may prove useful to municipalities across the world.

Also Yahoo has a collaborative learning or recommendation prize of their own. Yahoo Learning to Rank Challenge allows modelers to benchmark their ranking algorithms against the world. Must act quick because the challenge ends in June 2010.

Netflix scraps Netflix Prize II in lieu of lawsuit

2010-03-12T14:08:00.000-08:00

Unfortunately for the prediction and mathematical modeling community Netflix has decided to scrap the sequel to the Netflix Prize. The matter came to a conclusion after a lawsuit was filed against Netflix about the public access to its member's ratings data. For the original Netflix prize the member ratings data was made anonymous. Yet the lawsuit is claiming that the ratings are so good that (from the article)

improvements made to the recommendation engine made it easier to identify people through supposedly anonymous information.

I guess the modeling community gets a +1 for great improvements.

I was a big fan of the Netflix Prize even if it only brought marginal improvements to the actual recommendation system. The shared knowledge and collaborative spirit was impressive. It doesn't sound like Netflix is going to go out on a limb and suggest a new contest. So it looks like this might be the end of the contest. This is very unfortunate because I was hoping this would spark a lot of companies trying these types of contests.

This also brings up a good point about making data anonymous. There are lots of ways to get this done. Please share ways that you're modeling for business, academic, or clientele that required making data anonymous.

Free e-books on Mathematics

2010-03-04T08:20:00.000-08:00

Thanks to Jeromy Anglim on his blog he found a great link for free e-books on mathematics from e-bookdirectory.com. There are quite a few e-books in this link which can be a great resource for rehashing and learning some new methods.

Some noted e-books in mathematics that I found to be interesting

Algorithms by Ian Craw, John Pulham
An Introduction to R by W. N. Venables, D. M. Smith
Engineering Mathematics by Ian Craw, Stuart Dagger, John Pulham
Statistics for Business and Economics by Marcelo Fernandes

Jeromy Anglim's blog is a very interesting resource for statistics, data mining, and R. Be sure to read it for its wonderful insights.

Gnumeric 1.10 released

2010-02-22T05:58:00.000-08:00

OStatic.com has a review of the new Gnumeric 1.10 release. For those that don't know Gnumeric is an open source spreadsheet application built specifically for the GNOME desktop environment. Gnome can use a wide variety of spreadsheet formats including Microsoft Excel files, Lotus 1-2-3, Applix, Sylk, XBase, Open Office, Quattro Pro, Dif, Plan Perfect, and Oleo. Gnumeric's files is based on the Open Document Format.

The new 1.10 release removes the 65000 line restriction that was so common to many spreadsheet programs. From the Ostatic review...

The rest is all good news, though. Users will find plenty of improvements in Gnumeric 1.10, including better graphs with new plot types, about 40 new functions, and performance improvements for larger spreadsheets.

Gnumeric can be a great alternative to other spreadsheet programs.

R Project named in Intelligent Enterprise 2010 Editor's Choice Awards

2010-02-18T02:03:00.000-08:00

News from Revolutions blog says that R Project was named to the Intelligent Enterprise 2010 Editor's Choice Awards. They also state that the R Project is also one of the twelve companies to watch in Business Intelligence. For those that use R it should come as no surprise the increased notoriety that R is receiving.

Other Open Source selections from the Intelligent Enterprise 2010 Editor's Choice Awards include Apache (open source web server), Jaspersoft (open source business intelligence), and Eclipse Foundation (open source software development).

A review of website Programming R

2010-02-17T05:55:00.000-08:00

Lately I have been getting deeply involved in the statistics world of R. I have recently taken a job in online marketing and I am finding R to be a very useful tool. The first thing I did was to try to find tools of the R trade. I am finding that there are a lot of great tools for R available on the internet.

One of those online tools is the website Programming R. Programming R seems to be a fairly new website that seems to be only around for about a year. The promise of Programming R is it's dedication to R users from beginner to advanced. I find the articles written for Programming R to be very concise and well written. Programming R also provides book reviews for R. This is a very helpful tool for the beginner user.

One interesting section of the Programming R website is its devotion to R consultants and R jobs. The R project itself promotes these but its refreshing to see an independent website to promote statistical jobs devoted to R.

There is also a web forum which is always useful. Unfortunately its not very featured by its users for some reason. Hopefully as the website grows the forum will be used more often.

Overall I really like Programming R for its writing and content. I recommend new and veteran R users to seek Programming R as a resource.

When to use Excel and when to use R

2010-02-01T09:11:00.000-08:00

There is a great post in O'Reilly's Answers that talks about when to use Excel and when to use R. I have been using a lot of R lately to perform some data analysis and logistic regression. R is a great tool for statistical analysis. R is also free and open source.

I am going to be doing some more blogging about statistics and using R (R-project homepage). There is a huge amount of uses with R and I'm sure I will only hit the tip of the iceberg. If you have experience in R please let me know and I would love to share your experiences.

8 Open Source Business Intelligence Software

2010-01-05T05:22:00.000-08:00

Business Intelligence software has become a necessity for data rich companies. Business Intelligence software is the software that handles the large amounts of data and transforms it into meaningful information. Of course a lot of the transformation requires technical know-how such as data mining and operations research. Linux Links has put together the top Free and Open Source Business Intelligence software available. Some I have already linked before but there are new ones on this list I was not aware.

Business Intelligence one of hottest skills for 2010

2009-12-31T15:12:00.000-08:00

Computerworld takes a look at the main needs for 2010 and finds that Business Intelligence will be a hot skill. This article is mainly about the IT world but it reflects a growing need for data transformation and data mining in business today. There is a lot that goes into Business Intelligence other than just setting up an enterprise ready database system. The article mentions that there will be a need for analysis and decision making. From the article...

More important than a BI expert, though, are programmer/analysts who can relate the nitty-gritty of data tables, database joins and data structure to business requirements. "That's what I'm finding is more valuable to us at this stage in getting BI established and used by the business

Data analysis and processing information is only going to grow in this data rich economy. Organizations are going to be looking for people who can not just sift through the data but also relate it to business decision analysis. I believe that Operations Research could really see a "re-birth" if you will. I'm going to be looking forward to 2010.

Learn to program in Python by making games

2009-12-29T19:49:00.000-08:00

Python is a great programming language for applying some basic mathematical programming. Here is an article via Makezine.com blog about how to learn Python by writing games. This is a follow up to my last article on Artificial Intelligence with Python. Learning a new programming language can be a bit dull. I thought it might be more fun to learn how to write games in Python. Perhaps this is something to do for while off for the New Year.

Artificial Intelligence with Python

2009-12-18T08:25:00.000-08:00

While stumbling across cyberspace looking for some interesting Python tools and tutorials I found this rather interesting webcast. The video is of Raymond Hettinger at PyCon 2009. Raymond describes the usefulness of Python with applying artificial intelligence and data mining. This talk is very interesting to see how useful of a tool Python can be to performing some relevant Operations Research tasks specifically with data manipulation and learning. As I have mentioned before Open Source software offers a lot in the way of Operations Research tools. I hope you enjoy it as much as I did.

Source:
http://blip.tv/file/1947373

Weekend fun: The perfect way to slice pizza

2009-12-12T05:17:00.000-08:00

The New Scientist has a fun article about slicing perfect pizza, mathematician style. The basic premise of the article is to find out if there is an optimal way to slice the pizza so that the pie can be distributed among diners equally. Rick Mabry and Paul Deiermann of LSU have been trying to prove the hypothesis of the equitable slices. Of course it comes as no surprise as it all comes down to the cut. I'm not really sure what makes me more hungry, reading about proofs or about the pizza? Either way you slice it looks like fun to me.

Flaw of Averages followup: The Average Lie

2009-11-20T10:18:00.000-08:00

The Simple Dollar has a great article about the The Average Lie. The reason why I bring up this article because a recent IEOR Tools blog commentator opined in an earlier post on the Flaw of Averages by Sam Savage. The commentator said Sam didn't bring up any good examples. Actually I think Sam did bring up a great example in personal finance of the fallacy of assuming x% average growth rate for the life of an investment vehicle. This article by The Simple Dollar is a great illustration of Sam's example.

After posting that article I could think of many different examples of the Flaw of Averages. Some that come to mind in industrial operations include average rate of production, average time to complete a task, average demand for a product. I have had to deal with each of these scenarios often having to come up with better suggestions to management to deal with the operational challenge other than using an average value. I would love to hear other great examples.

Social Networks and Operations Research

2009-11-12T09:30:00.000-08:00

Based on a request from a blog reader Michael Trick has set up the OR-Exchange as a social networking way to get Operations Research related questions and answers. It is an experiment to see if it would be beneficial to the Operations Research community. For those of us that have used the common Usenet groups in the past we have recently seen it degrade into a chasm of spam. Hopefully with these new networking sites this will help remedy that networking need in the Operations Research community.

On another note Analytic Bridge and LinkedIn is also a very good networking and discussion social networking tool. There are many "Groups" that one can join at LinkedIn to join with common Operations Research interests. Here are links to the following social networking sites devoted to all things Operations Research.

LinkedIn INFORMS group

LinkedIn DataShaping Advanved Analytics group

Analytic Bridge

Flaw of Averages recap and INFORMS podcast

2009-11-10T13:42:00.001-08:00

I talked about an article about Sam Savage and his new book the Flaw of Averages not too long ago. This past Friday INFORMS released a new podcast in their series the Science of Better interviewing Sam Savage. This is a fascinating interview about the theory in his book and the, unfortunate often, over simplification of decision analysis based on a simple average of a data population. I strongly suggest listening to this podcast as Sam has a unique way of presenting his theories.

Maximize Productivity with Industrial Engineer and Operations Research Tools

Should science be open

IEORTools.com Resources added

Most popular 2011 IEOR Tools blog articles

Visualizing categorical data in R

OpenOpt Suite 0.37

Expanded list of online courses for data analysis

My learning as a Data Scientist

Top 50 Statistics blogs of 2011

Data mining the Federal Reserve

Machine Learning for everyone

Data Driven Success in Professional Baseball

OpenOpt Suite 0.36

Physicist cuts airplane boarding time in half

What did we learn from the Space Shuttle program

Problems with data visualizations followup

How not to do data visualization

Recommended Machine Learning blogs

Moneyball coming to the big screen

OpenOpt Suite 0.34

Analytics geeks win NBA championships

Sports analytics summer blog reading recommendations

In Memorium of Dr. Paul Jensen

Welcome to the Insight Age

R Tutorial: Add confidence intervals to dotchart

Google funding research to measure regret

Open Source replacements for Operations Research and Analytics Software

OReilly - Quiet Rise of Machine Learning

Baseball and Decision Analytics

R again in Google Summer of Code

OpenOpt Suite release 0.33

IBM has a Natural Language Purpose

Question and Answer sites for Analytics and Operations Research

Science of Matchmaking

Data Mining Books List

Operations Research and gerrymandering

Statistics blog roundup from the Internet

Video of Joy of Stats by Hans Rosling

Top IEORTools Blog Articles of 2010

An essay on Mathematics and education

New OpenOpt/FuncDesigner quarterly release

Where to find good data sets

Shortest past algorithm solved by ants?

2 years of blogging with IEOR Tools

Big Data Logistic Regression with R and ODBC

INFORMS Data Mining Competition leaders used Open Source software

R references for handling Big data

R Links for the Beginner on World Statistics Day

Data mining competition with R

IBM's furious Analytics aquisitions

Current Data Mining and Analytics Challenges

OpenOpt release 0.31

Computer languages and Applied Math

What did the new PvsNP proof prove?

IEOR Tools Tutorial: Learning XML with R

Favorite Operations Research books from OR-Exchange

Kaggle introduces new Chess rating competition

New Look for IEOR Tools

R IDE for Linux and Gnome

Eight Data Mining Social Networking Groups

Audio of Richard Stallman keynote at useR2010

Podcast with Revloutions Analytics CEO Norman Nie

OpenGamma startup claims Open Source Financial Analytics

Open source solver for Excel

Kaggle hosting INFORMS 2010 Data Mining Contest

U.S. SEC endorses Python to fix financial problems

R package for World Bank Data

Analytics and FIFA World Cup

OpenOpt release 0.29

OR-Exchange confession

My 5 favorite Operations Research blogs

Algorithms and Wall Street

R has a revolutionary commercial launch

Railways improved by mobilizing Operations Research

Operations Research courses via Open Courseware Consortium

More Analytic Competitions

World Bank opens data to the public for open research

OR-Exchange in the top 50

OR-Exchange Needs Your Help

Math is the New Cool for Employers