Plan, plan and plan again: A practical guide to planning experiments

Evolution of an experimental plan

Published 13th February 2017 by Andy Connelly. Last updated 9th May 2017 by Andy Connelly

Introduction

The plan is key. Before you launch into a long set of experiments; plan, plan and plan again. It is where you set out the question you want to answer and how you are going to answer it. In other words, the theory and the observations you are going to make. The plan (and often even the question) will have to be refined along the way through laboratory and desk work but you need to start each set of experiments with a clear idea of what data you will collect and, equally importantly, what you will do with that data. There are many stages in the planning stage.

DISCLAIMER: I am not an expert on experimental design. The content of this blog is what I have discovered through my efforts to understand the subject. I have done my best to make the information here in as accurate as possible. If you spot any errors or admissions, or have any comments, please let me know.

Variables

Variables are the factors in your experiment you will change, control, or measure (e.g. temperature, concentration, etc.) (see Figure 1) If you are not sure what you are looking for, or wanting to know, then you will need refine your question. Analytical chemistry is like talking to a very pedantic teenager. Stupid question in, stupid answer out!

Always make sure you know what question you want the data to answer before you start. However, make sure you don’t let your prediction cloud your experimental judgement.

Start by brainstorming your ideas. Think about:

  • The variables are you measuring (dependent variables)
  • The variables are you going to vary (independent variables)
  • The variables are you going to control (control variables)
  • The levels you want to measure (e.g. 1-20ppm, 50-55ppb, 500-600°C, etc.)
  • The equipment you’ll need and what is available
  • The time (and money) available
  • How long your samples will last once collected
  • Etc….
Experimental design of a cake
Figure 1: Experimental design of a cake (from https://www.moresteam.com/toolbox/pics/4081.gif). Arrow added by the author here to show feedback into new set of experiments to improve the outcomes – or because you want more cake!

It is a good idea to put all possible variables down at first and then prioritise. Decide which you will vary, measure, and control but also which will have an effect – some variables you think are important may not be relevant to your experiments – see Figure 2.

Evolution of an experimental plan
Figure 2: Evolution of an experimental plan (yours might need to be a bit more detailed!).

Quantitative analysis

To gain quantitative results you will also need to decide exactly what values you want to measure, the range of those values, and how much uncertainty (the level of precision) you can accept in the measurement.

This may sound strange but even quantitative property measurements do not give the true value of that property; they only ever give an estimation of that value. We can improve that estimate by repeat measurements but it can still only ever be an estimate. For this reason, we must specify the measurement uncertainty in all your work. Without this, your results are meaningless as readers have no idea how much they can trust your value. For example, is your result 50ppm or is it 50ppm ±100ppm?

There are two important definitions to introduce here.

  • Error: how far a single measurement is from the hypothetical “true” value. Errors can be gross, systematic, or random.
  • Uncertainty: the range of values within which you are confident, to a certain level (e.g. 95%), the true value sits.

You need to think about how you are going to calculate and present your uncertainties. Showing your uncertainties demonstrates that you are aware of the errors in your measurements and that you have taken them into account. It also allows you to monitor those errors and look at ways to reduce them and so improving the quality of your data. We must show the reader that the results we are trying to convince them of are statistically robust and not just random chance. Figure 3 & 4 show good, and bad, approaches to this.

A bad approach to quantitative analysis.
Figure 3: Diagram showing a bad approach to quantitative analysis.
A good approach to quantitative analysis.
Figure 4: A better approach to quantitative analysis.

Once you have narrowed down the question, brainstormed your variables, and decided what data type, range, and uncertainty you are happy with; you need to design the experiments that will answer the question.

Experimental design

The idea behind experimental design is to answer your question as efficiently as possible while collecting robust data that is suitable for your needs. This is particularly important in medical sciences due to issues like the placebo effect. Luckily, in laboratory based physical science, issues are generally (although not always) less complicated.

The most common approach to experimental design in chemistry is one-at-a-time design where we investigate individual factors while others are held constant. This is very effective in many situations and is formally known as (full) factorial design. In full factorial experiments two or more factors are varied across all their possible values such that all possible combinations of these values are tested. An example of this is shown in Table 1. With this method the whole of phase space is explored with all variations attempted while keeping other potential variables constant throughout (e.g. time).

The number of trials/experiments required can be easily calculated. For example, an experiment with three factors each with two levels gives a total of 8 (2^3=8) experimental conditions so 8 trials are required. Similarly, two factors, each with three levels requires 9 trials (3^2=9).

full factorial design
Table 1: An example of full factorial design. Three factors: one factor with three possible values (3^1) and two factors with two possible values (2^2) so requiring 12 trials (2^2 x 3^1).

Clearly, full factorial design is fantastic for certain situations; however, as the number of factors, or the number of values each factor can take, increases the number of trials you have to run increases dramatically. For this reason various other systems have been developed. Some common ones include:

The power of these methods is in reducing the number of experiments required to achieve the same aim. They do this through various clever mathematical systems for reducing the number of experiments while still sampling across the system of interest. One method that is worth mentioning as an example is Plackett-Burman.

Plackett-Burman

In a full factorial design experiment with five variables each with two values would give a total of 3^2 experiments (25). In Plackett-Burman you would need a minimum of 12 experiments! The system works by carrying out selective experiments and estimating the effect of each variable on the outcomes.

It is particularly helpful to use Plackett-Burman design:

  • In screening – when looking across a wide range of variables and seeing which are the most important
  • When neglecting higher order interactions is possible – i.e. the variables do not interact much
  • In two-level multi-factor experiments – each variable takes only two possible values
  • When there are more than four factors (if there are between two to four variables, a full factorial can be performed)

There are also some drawbacks to using Plackett-Burman designs:

  • They do not verify if the effect of one factor depends on another factor.
  • If you run the smallest design you can, it does not follow that enough data has been collected to know what those effects are precisely.

There is not space to go into detail here but there are many books available on this, and related, techniques (see Further Reading section below).

Optimisation

Another area of experimental design is ‘Optimisation’. This type of experimental design is very useful if you are looking to optimise a process or a signal from a piece of equipment. It allows you to optimise with the minimum number of experiments. Examples are: Simplex (see Figure 3) and Simulated Annealing.

Sequential Simplex Optimization
Figure 3: Sequential Simplex Optimization: a method of searching phase space for the “optimal” value. (After http://www.engineeredsoftware.com/sso.asp)

Practicalities

Once you have thought through the variables and experimental design there are various practical issues to consider in terms of the equipment you might want to use (see Figure 4).

Choice of equipment
Figure 4: Venn diagram showing the key decisions when choosing a technique for your experiment.

In more detail:

  • What?
    • What do you want to measure?
    • What do you want to control/vary?
    • How much sample it will take for your specific samples?
    • What level of precision/accuracy do you need?
  • Where?
    • Is the equipment/technique available in your labs?
    • Is the equipment/technique available suitable for the analysis of your samples?
    • Can the equipment detect the levels you need it to? (LOD)
    • What is the cost of the analysis (both in time and money)?
  • How?
    • Has a method already been developed on the equipment you are interested in?
    • What effect will your samples have on the measurement (e.g. if your samples are corrosive they may damage the equipment, will the samples interfere with the measurement?)
    • How long will your samples survive in their current form?
    • How can you be use you are measuring what you think you are measuring?
  • Who?
    • Have you talked to the technician?!
    • Have you asked to see if anyone has done this type of thing before?
  • Why?
    • If you are asking this already… it is not a good sign!

You need to consider these questions BEFORE you start your experiments. Otherwise, you might be wasting your time! This is, by no means, a complete list…

Conclusion

There are many aspects to planning your experiments. Some planning takes place in the laboratory, some at your desk, it will depend on your project and your approach to the project. However, the most important thing is that you have an open mind to changing your plan if things are not working. The more time you spend at the beginning planning the less likely that is to happen but it is called experiment for a reason…you’re never quiet sure what is going to come out the other side!

 Further reading

  • Statistics and Chemometrics for Analytical Chemistry, Miller & Miller, 5th ed. Pearson (2005)
  • Data analysis for chemistry: An introductory guide for students and laboratory scientists, Hibbert & Gooding, 2006
  • Statistics: A guide to the use of statistical methods in the physical sciences. Roger Barlow, John Wiley & Sons, 1989.
  • Statistics without tears: A primer for Non-mathematicians, Derek Rowntree, Penguin Books, 1981.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s