Parallels between data science and traditional scientific observation


Reading The Art of Observation and Why Genius Lies in the Selection of What Is Worth Observing on brainpickings (which seems to be the source of much inspiration for me at the moment) I was struck by the parallels between what data scientists do at tech companies and the traditional scientific observation.

Hence these descriptions of scientific observation are relevant for data scientists and their managers.

  1. There are two types of observations – (a) spontaneous or passive observations which are unexpected; and (b) induced or active observations which are deliberately sought, usually on account of an hypothesis
  2. One cannot observe everything closely, therefore one must discriminate and try to select the significant.
  3. Most of the knowledge and much of the genius of the research worker lie behind his selection of what is worth observing. It is a crucial choice, often determining the success or failure of months of work, often differentiating the brilliant discoverer from the … plodder. [NB – track the right metrics]
  4. Powers of observation can be developed by cultivating the habit of watching things with an active, enquiring mind. It is no exaggeration to say that well developed habits of observation are more important in research than large accumulations of academic learning.
  5. Effective scientific observation also requires a good background, for only by being familiar with the usual can we notice something as being unusual or unexplained. [NB – know your benchmarks]