An open source site for systematic reviews

Today saw the launch of a fab website called Randomise Me that “is a platform that enables individuals and organisations to run trials of varying sizes.”

The site is great because it enables anyone, anywhere to perform RCTs – for free. As far as I’m concerned, breaking down the barriers to ‘academic’ things like this is something to be applauded.

Having been involved in carrying out a couple of systematic reviews I think there’s a massive opportunity to develop a similar website that would enable anyone to do them.

I tweeted about this earlier and a couple of very kind people have come forward to say they might be able to help with such a project. I therefore thought I’d write a quick blog to describe what I think would be involved. There are a few steps for carrying out systematic reviews, but broadly speaking, the ones that require some form of software are:

  1. Collation of references – gathering together all the abstracts found by a literature search into one place.
  2. De-duplication of references – there will almost certainly be multiple copies of the same paper found by the literature search, so it’s helpful to be able to remove these duplicates before the next step, which is…
  3. Screening of abstracts and then full text papers. This step requires the creation of a list criteria against which each  paper is assessed and is then either marked as being suitable for inclusion or not.
  4. Data extraction – for the final set of studies included in the review, this step requires taking the data of interest out of each paper (using some sort of form) and then collating the data from each form into a new table.
  5. Assessment of bias of the included studies – again this sounds fancy but software-wise it just requires the creation of a data entry form for each study similar to the previous step.

The added complexity is that steps three to five above are typically done by two, and sometimes three people with reconciliation between the individual decisions.

I might have missed something from the list above, but I’m deliberately leaving out analysis steps (like meta analysis) because there is a great open source statistical package called R and I’m almost certain it would be possible to integrate with this.

At the moment many people create Microsoft Access databases to perform their systematic reviews. The main drawback to this is that it requires a license, knowledge of how to use Access, and steps three to five above are a real pain in the backside to do in a desktop based database if more than one person is involved.

In terms of what’s out there already, there are some good websites, but they’re either prohibitively expensive or don’t really do the full job, including:

  • EPPI reviewer is what I’ve been using and it’s pretty good and is able to do all of the steps above, but costs at least £55 a month if you’re double entry screening which is obviously a major barrier for most people. 
  • Distiller SR looks comprehensive but this is even more expensive than EPPI reviewer.
  • The Systematic Review Data Repository looks like it’s free, but I’ve never used it and it only seems to do step four above.
  • EROS was recently recommended but it doesn’t seem to be free and I can’t work out what it does which isn’t a great start.

Writing this post has made me realise there’s probably a good reason that no one has put together an open source site for performing systematic reviews  –  it’s no simple task!

Still, I stand by the idea that it would be a game changer if anyone, anywhere, could access the software to perform systematic reviews. If you’re interested in a project like this then drop me an email (address here) or post a note below as I’d be keen to help create


Leave a Reply

Your email address will not be published. Required fields are marked *