Question lists for the process of data visualization

Notes taken from this Tableau whitepaper: questions to ask before and throughout the data visualization procedure

Part 1: Overall questions

• Who is your audience?
• What questions do they have?
• What answers do you find for them?
• What other questions does it inspire?
• What conversations will result?

Part 2: First pass questions

• Does this visualization answer all of your questions?
• Is the purpose of the visualization clearly explained in its title or surrounding text?
• Can you understand the visualization in 30 seconds or less, without additional information?
• Does your visualization include a title? Is that title simple, informative, and eye-catching?
• Does your visualization include subtitles to guide your viewers?

Part 3: Finding the right chart type for your analysis

• What types of analysis are you performing?
• Have you selected the most suitable chart type(s) for your types of analysis?
• Have you considered alternative chart types that could work better than the ones you have chosen?

Part 4: Checking if your views are effective

• Are your most important data shown on the X- and Y-axes and your less important data encoded in color or shape attributes?
• Are your views oriented intuitively – do they cater to the way your viewers read and perceive data?
• Have you limited the number of measure or dimensions in a single view so that your users can see your data?
• Have you limited your usage of colors and shapes so that your users can distinguish them and see patterns?

Part 5: Is your dashboard holistic?

• Do all your views fit together to tell a single story?
• Do all your views flow well from one to the next? Are they in a good order?
• Do your most important views appear in the top or top-left corner?
• Are secondary elements in your dashboard placed well so they support the views without interrupting them?
• Are your filters in the right locations?
• Do your filters work correctly? Do views become blank or downright confusing if you apply a filter?
• Do your filters apply to the right scope?
• Are your filter titles informative? Can viewers easily understand how to interactive with your filters?
• Are your legends close to the views they apply to?
• Is your legend highlight button set to “on” or “off” according to your preference?
• Do you have filter, highlight or URL actions? If so, do they work?
• Are your legends and filters grouped and placed intuitively?
• Do you have scrollbars in your views? If so, are they acceptable ones? Are your views scrunched?
• Do your views fit consistently well when you apply filters?

Part 6: Did you perfect your work?

• Do all the colors on your dashboard go together without clashing?
• Do you have less than 7-10 colors on your dashboards?
• Do you use fonts consistently in all of your views and no more than three different fonts on one dashboard?
• Are you labels clear and concise? Are they placed optimally to help guide your viewers? Make sure subtitles are formatted to be subordinate to the main title.
• Are you tooltips informative? Do they have the right format so that they’re easy for viewers to use?

News of the week: first female Fields Medalist

http://www.theguardian.com/science/2014/aug/13/fields-medal-mathematics-prize-woman-maryam-mirzakhani?CMP=fb_gu

No Free Lunch

No Free Lunch is a theorem:

• wikipedia
• Original paper

Python for Data Analysis

Python is known to be the go-to language for data analysis, and for my purposes, it naturally become the first thing I want to attack. For my daily job I needed to parse hundreds of thousands flat json files into tabular format and then to message them into data source that I can work on. Pandas has played an important role in all these, and it helped me getting things done by knowing minimum about Python in general. This book an automatically recommendation for people who are about to use Python for data analysis. I just want to point out a couple of caveat I’ve found during my own learning.

First of all, I was curbed quite a few times by performance issues for I converted lists and records too frequently into data frame. Even though data frame created using Pandas is already thousands (or more) times faster than those created in R, it is still a slow process in nature.

Second, I started reading this book before I started working on my project, and installed EPD (version 2.7.3) as the main python on my computer (a Mac). This is not a great choice, and I should have just installed the official Mac version of Python 2.7.8 instead. It has given me a lot of pain in package management. So here’s a friendly advice for those who are about to use this book as the first step into Python. 🙂

In fact, I didn’t quite finish reading this book (and am probably not going to). The book’s instructions worked for its own data set, but not for mine. The json file it was dealing with was in different format than the ones I got, and I got stuck on my first step, and turned to the internet for solutions. Things happened naturally from there – for all the questions I had to get the project started, I could find answers online all over the place online. Also, the methods the book introduced for munging the data only handles “small” data well enough, and may turn out to be a terrible solution to handle “big” data – it’d never finish! In reality, using Python for data analysis does not equal to turning everything into a data frame. I learned my hard lesson(s) about that. Apparently, the book went through the pandas package in a hands-on way without providing too much insights into how data should be handled and why, as to which performance issues the user should keep in mind. That doesn’t make this book lesser instructional, but it’s worth noting.

My first publication in mathematics

I haven’t published a lot in my life, not to mention in mathematics. It has once been my passion to become a writer of sorts, and that dream silently faded away amongst a ton of other dreams. However, it is forever memorable to me when my first publication in mathematics happened – when I was in the seventh grade, second year into the middle school.

Ever since I was small, I’ve been led by my father to habitually learn and think on my own. He encouraged me taking my interests as far as I’d like, on whichever subjects. One of the very first things he told me was, “Curiosity is the best teacher.” – The true meaning of this saying had stayed with me since then.

Before seventh grade, I’d grown curious about mathematics, and had been indulging in the fun of mathematical discovery for a while. I taught myself algebra by going through textbooks, and the incident happened not long after that. One day I was going through the derivation of . I suddenly became curious so that I derived the formula for by brutal force. There I found a pattern: the square of the sum of terms is equal to the sum of the squares plus two times their pair-wise products! I quickly jotted down notes for deriving this for the square of terms, and the finding is confirmed!

I was so excited that I told this discovery to my father soon after. He said:”Why don’t you write it down, as thoroughly and convincingly as possible?” I took this very interesting and exciting challenge happily, and went through pains to explain the logic of the derivation of this formula to the best of my abilities. Needless to say, bearing little idea about proper mathematical terminologies, not even hearing about the thing called “mathematical deduction” – which came to me only one year later as a huge realization – my writing at that time was silly and clumsy. I was like a child who just learned how to run shortly after learning how to walk. The fun was beyond everything.

After I was done, my father seemed delighted. To him, the joy is about watching his kid learning how to run on her own shortly after she just learned how to walk, I guess?

Weeks after that, when I had already forgotten about this, my father came home one day saying:”V, your article is published!” – He submitted my silly writing to a magazine called “Mathematics for Middle School Students” and it got accepted! Hence, my first publication in mathematics.

Truth is, my father is the best educator in the world. There’s simply no doubt about that.

Recent Tips with Tableau

R implementation in Tableau 8.1

Tableau 8.1 had the new functionality of integrating R scripting. Since I only started using Tableau two months ago and got rather serious with it two weeks ago, I picked this up only recently. In case you want to do the same, this webpage: R Implementation Notes is really all you need to go through – I found some other webpages on this topic rather confusing.

After successfully installing and running Rserve, you might want to check if it really works. For that purpose, I found this incredibly helpful blog of Bora Beran – if you really just want to grab some test files to see how it works, you can find all Tableau workbooks that Bora Beran referenced in his blog shared in the cloud.

Tableau 8.2 Beta

As far as I can gather, Tableau 8.2 has two new offerings: Tableau on a Mac, and Story Points – I’ve tested installing Tableau 8.2 beta on my Mac, and it works seamlessly. I haven’t found anything strange by switching from Windows to Mac for Tableau, except that it did run a bit slow.. but then, I was working on a relatively large data set (1G in csv format, which is compressed into 64M Tableau package workbook) on a Mac with 4G RAM – The new functionality of Story Points is very easy to use. It is as intuitive as worksheets and dashboards.

Testing is easy – if you could somehow successfully signup for an account with the Tableau beta program (the hardest part), you can download a beta version that you can use before it expires in a couple of weeks.

Tableau Conference 2014

It would be great if I could attend the Tableau conference in September – if only to see Hans Rosling speaking.

Happy analyzing!

Difference between Data Mining and Predictive Modeling

Data mining: this is cute! Predictive modeling: this will turn into revenue!

The blog that inspired me is talking about how non-predictive the recent “love curve” by Facebook’s data science team is. Interesting enough, I had the same thoughts when seeing that chart but never went so far as to try to prove its uselessness using “scientific method”.

For one thing, I fell victim of Bulverism, since I was thinking that FB’s chart was not meaningful without even trying to prove that – as cited by C. S. Lewis by Greg Mankiw:

In other words, you must show that a man is wrong before you start explaining why he is wrong. The modern method [Note: This essay was written in 1941.] is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became to be so silly. In the course of the last fifteen years I have found this vice so common that I have had to invent a name for it. I call it “Bulverism.” Some day I am going the write the biography of its imaginary inventor, Ezekiel Bulver, whose destiny was determined at the age of five when he heard his mother say to his father – who had been maintaining that two sides of a triangle were together greater than the third – “Oh, you say that because you are a man.” “At that moment,” E. Bulver assures us, “there flashed across my opening mind the great truth that refutation is no necessary part of argument. Assume your opponent is wrong, and then explain his error, and the world will be at your feet. Attempt to prove that he is wrong or (worse still) try to find out whether he is wrong or right, and the national dynamism of our age will thrust you to the wall.” That is how Bulver became one of the makers of the Twentieth Century.

Recommend a blog: Farnam Street

Even though this blog has nothing to do with data, and that I’m not too fond of chicken soup style – cheap talk of business successes and blah blah – I’ve so far found each post on this blog pleasant to read.

For example, today’s title is: Average is Over.

Two Egg Problem

I encountered this interview question once: Two Egg Problem, which states:

You are given two eggs, and access to a 100-story building. Both eggs are identical. The aim is to find out the highest floor from which an egg will not break when dropped out of a window from that floor. If an egg is dropped and does not break, it is undamaged and can be dropped again. However, once an egg is broken, that’s it for that egg.

If an egg breaks when dropped from floor n, then it would also have broken from any floor above that. If an egg survives a fall, then it will survive any fall shorter than that.

The question is: What strategy should you adopt to minimize the number egg drops it takes to find the solution?. (And what is the worst case for the number of drops it will take?)

Recommend a blog: The Functional Art

This blog: The Functional Art is very much worth recommending – chances are you’ve already heard of it – a blog of such quality is hard to miss. For example, this article led me into the door of understanding the rival between two legendary data scientists: Richard Saul Wurman, and Edward R. Tufte. Great reading!

I’ve got the book with the same title for a while, though I haven’t got a chance to start reading it.