How to perform efficient experiments and to document useful results?
Time to Read 3 min
I encountered the situation time and again: a big heap of plots or a directory full of files, the results of many trials and experiments. Neither I who generated the results nor anybody else knew how to identify how the results were achieved. What was the hardware configuration, what the software version? Many times the stored results were utterly useless because it was not clear why and how they were generated.
The problem was usually not only the results, but also the trials themselves that generated them. There was some idea what to achieve, but no clear goals. So a lot of the work was not as efficient as it could have been.
A Template to Make Experiments Useful
To avoid these situations, we introduced a template that clearly structures the process of measurements, simulations, software behavior tests etc., "experiments" in short.
What are Possible solutions?
One could think of complex data management databases, there exists even software to manage measurement data. But they still need a lot of meta-information added to the measurement itself.
The very compact solution is the A3 report of Lean fame. The web is full of examples and references for different applications. For decisions it makes sense to compress the information on one single page, but in many cases there is not enough space to document all the information needed to make the experiment useful. Our template stays close to the A3 report structure, so if needed A3 reports can easily be condensed from it.
By filling out the template from the beginning of the experiment, the user is guided through the complete process
This is the part that is maybe less useful when re-using the results later, but it is very useful to clarify the thoughts of the engineer performing the experiment:
What is the starting point? Why do we do this? These are simple, but powerful questions to be answered before the experiment.
What is the hypothesis? What do you want to find out? How do you want to find it out? Also here, it helps to stop and write down the purpose of the work I will do. To think about this beforehand also helps to execute the experiment in an efficient way.
In an engineering context with configurations and versions that are constantly changing, it is important to document the exact configuration with which the experiment was conducted, the baseline. The same applies to the setup that states which instrument and tools were used in which configuration to achieve the result. Also the conditions can be important: what was the temperature, the humidity etc. Many a measurement in a dry heated room in winter could not be reproduced on a damp summer day, simple things like different electrostatic behavior can lead to interesting "software" effects!
Only after all this is documented, the results are written or better plotted, together with detailed information about the meaning of each value.
Last comes the most important point:
The analysis of the data, their effects and their root causes. Does the hypothesis still hold?
As an experiment makes no sense when nothing is learned from it, consequences in the form of actions must be documented, including the persons and roles that are affected.
But this takes too long for me!
You are completely right, it makes no sense to go through the whole process and fill out the complete template for every voltage measured. If we think a measurement is so important or useful in the future that we document it, we always do it completely. Otherwise we just throw it away, as without full-blown documentation the best scope screenshot, the nicest debug log, the most sophisticated failure time simulation will be useless.
Beautifully, the experiment process and the template lead to a "natural selection" of the measurements to be stored: if something is really worth to be stored, it is also worth the documentation effort.
And yes, one last thing: the whole process through which the engineer is guided is the PDCA cycle from my last post (Plan: starting point/ hypothesis, Do: experiment, Check: analysis, Act: consequences)!