The 'Freezer' Project

Further to the Boston meeting entitled: Strengthening the Use of Mathematical Models in Community Trials, a proposal for a method of validating mathematical models models in this setting was characterised.

Background

Mathematical models are often used to anticipate the impact of HIV prevention interventions but their ability to do this reliably has never been objectively tested. Several CRCTs will be initiated in 2013 that will measure the impact of HIV prevention interventions. Mathematical models have been used to predict the magnitude of these specific interventions and those calculations have informed the design of the studies. It should therefore be possible to store those predictions, and the models that were used to generate them, so that they can be compared with actual empirical observation at the end of the trials. Although this provides only a limited test of the capability of models -- in particular, it will only be possible to examine the short-term impact of the intervention -- we believe it is a valuable opportunity to challenge mathematical models.

Models are increasingly being used in the biomedical arena where validation standards are currently less rigorous and a methodology to quantify credibility is greatly sort after (1). The most important objective of this exercise is to begin to understand how model projections can succeed or fail in projecting the impact of interventions in CRCTs rather than to identify a ‘winning’ model. 

The 'Freezer' Protocol

The protocol developed by the Secretariat outlines a method by which these models can be validated. The three main hypotheses are listed below:

1. Do mathematical models provide the correct guidance about the impact of the intervention in the design phase (and interim phase, if appropriate) of the trials? 

2. Can the models predict the short-term impact of interventions once they are updated with information about particular aspects of the system that was not known when the models were first developed? For example, coverage of interventions, effects of intervention, of epidemiological data for the setting. 

3. Can our very best, fully-updated models, predict the short-term impact of interventions? 

The full protocol is available on the right hand side of this page.