At App.net we collect a metric ton of stats. When I build anything I try to collect some stats. We all must do that. Why not right, stats are fun. The hard part comes when you need to formulate those stats into information. It's not easy. Slowly, I learned what to watch, what was important. It's hard earned intuition, but intuition is no substitute for a well structured argument. Which is why I loved this short piece by Edward Tufte.
I could really could pull a quotes from any part but here are my highlights.
The purpose of analytical displays of information is to assist thinking about evidence.
I think this is the baseline. Only show what you need to help you understand what's going on.
Analytical graphics should be constructed to serve the fundamental cognitive tasks in reasoning about evidence: describing the data, making comparisons, understanding causality, assessing credibility of data and analysis. Thus the logic of design replicates the logic of analysis; design reasoning emulates evidence reasoning.
Don't just show graphs because they look pretty. Hopefully you have graphs that help you understand whats working, and whats not. In some sense our graphs at the office have become like an outboard 6th sense. I sometimes react to graphs physically before I have consciously understood what has happened.
In working on 4 books on analytical design, I have often turned to the human factors literature, and then left in despair, finding few examples or ideas (beyond common-sensical) that were useful in my own work. This contrasts to the work of scientists, artists, art historians, and architects--work overflowing with ideas about evidence, seeing, and the craft of making analytical displays.
While I don't have such a hard-line stance about the art of things, I definitely feel there is a lot of BS when it comes to design.
Anyway read it for your self. It's good.