Introducing the Shiny App DThiring

Well it has a been a long time since I have written anything on this blog.  I am long overdue.  I've been terribly busy learning new things and getting on with life.  One of the things I have learned is building R applications using Shiny developed by RStudio.   The folks at RStudio have also created a way to deploy Shiny apps using Shinyapps.io.  Follow the link to DThiring

http://larrydag.shinyapps.io/dthiring/

I'm a big fan of Data Tau which is a Data Science equivalent to Hacker News.  Like Hacker News, Data Tau has a Who's Hiring comment on the first of every month.  It is a good resource for those looking to see relevant jobs in the Data Science world.  Well someone created a hiring listings aggregator called http://hnhiring.me/.  I decided I wanted to see if I could build a similar application using R and Shiny.

I will be posting the source code for this app to github in the near future.

If you have any ideas on how to improve this application let me know in this comment section.

Google Statistician uses R and other programming tools

A great interview on the Simply Statistics blog with Google's Nick Chamandy, Phd in Statistics.  Explains that he mainly uses R among other tools to perform his work at Google.  Also of note is the active data science community within Google that uses R as well as some other interesting tools.  Note that they use a lot of data at Google, understandably, and that R usually can not handle the size.  They do a lot of ad hoc reduction of the data with tools like map reduce, Go, and even an R API.  I would love to see how they use the R API to assimilate data.

An interesting insight from the interview is the amount of programming done by the Statisticians.  It seems the culture at Google is to foster autonomy and let the modelers develop their own data manipulation from the raw data.  This requires a broader skillset beyond the statistical analysis tools.

I've found in my work that having knowledge in many tools like R,  CPLEX, and GLPK allows me to be a more effective in my work.  Recently I've been learning a lot of SQL using the PostgreSQL platform.  The tools of SQL combined with statistical tools like R make for a very strong combination.  I'm very agile in my work and can do a varied number of decision analysis.

Popularity of R continues

No doubt those that read my blog know that the tools I use to do my Industrial Engineering and Operations Research work heavily rely on the open source side of software.  That is why I try to support as many open source projects such as COIN-OR, GLPK, and OpenOpt.  One tool that I love to perform Applied Math and Statistics is the statistical computing platform R.  So it comes as no surprise that I like to see how R is growing and its popularity among programmers.

A recent blog from RedMonk produced results of a programming language popularity study.  The study involved ranking popularity using common social media online sites such as Stack Overflow and GitHub.  These sites draw in a lot of programmers for their popularity around Q&A and code review.  I was surprised to see that R ranks highly compared to some very prominant programming languages.

Also interesting to note that the only other "Data Science" type of programming language I could find was Matlab.  As far as I could tell SAS, S, SPSS, Stata are still rather popular but apparently not among the programming community.

My learning as a Data Scientist

So apparently the new en-vogue title is Data Scientist.  I can now include that to my already expanding list of titles.  In the past I've been known as an Engineer, Operations Analyst, Production Control Specialist, and an Analytics Analyst.  Now I'm considered a Data Scientist.  It's all the same to me.  My training and expertise has allowed me to problem solve many challenges within organizations.  The title doesn't matter.  There are opportunities for people with my skill set.

A recent blog post by Kontagent Kaleidoscope about Big Data is Useless without Science got me thinking about my role as a self-proclaimed Data Scientist.  The blog article points out a need for the science of better decision making.  Organizations are looking for people to help them turn their data mines into information gold.  I've definitely learned a lot over the years as a Data Scientist and I thought I would list some of those learnings.

1.  Organizations Don't Know What a Data Scientist Can Do

The idea here is marketing your own talents.  The Data Scientist needs to put their methods and work out there for the organization to see and touch.  This means working with the peers and management in the organization.  The Data Scientist needs to be able to eloquently relate methods, problems, challenges and how they can be solved.  Important skills here are personal marketing and communication.  I know this goes against the grain of many numbers geeks like me.

2.  Problems Don't Solve Themselves

Opportunities for solving real problems in an organization are always around.  The trick is being in the right place at the right time to be able to solve those problems.  Organizations have hoarded a lot of data and many times they don't even remember why.  The Data Scientist needs to turn into a Data Detective.  Explore all aspects of the organization.  Interview different departments and see how they tick and ask questions like "What keeps you up at night about your job?".  I was often surprised how a simple solution would go a long way to helping someone else out.  This develops true collaboration and leads to bigger problems to solve.

3.  Always Continue to Learn New Things

The world is constantly evolving and there are always new tools, tricks, methods, algorithms, software and mechanisms.  The Data Scientist needs to be able to adapt to new technologies.  I've found its best to stay current with whats new in order to stay sharp and meet new demands.  The internet can be your friend.  Even keeping up with a favorite list of blogs can help with staying current.  Times change and so do organization's needs.  Perhaps this is just me but I love learning new things as it creates a fun diversion and improves my skill sets.