Wikipedia vs. Academic Papers – a Middle Ground

We’re trialling an experiment until the end of February. Can we assemble a review of an area of science on a wiki, allowing anyone to contribute, and then publish that in a peer-reviewed academic journal? (early description of this on G+)

Wikipedia has a great deal of useful information, but its coverage of some academic areas can be patchy – it naturally depends who has contributed. With many articles the reader will be unable to judge whether the article is complete and accurate according to the current knowledge of the field. By contrast, many academic articles in journals are not open access, reducing the readership. Sometimes articles are written with a significant bias towards the authors’ work, and indeed sometimes that is the explicit purpose of the article. It’s very rare for leaders in a field to get together to co-author a review.

Wikipedia also has a suspect reputation in academia – we complain about its shortcomings, do little about it, and then all go and use it when nobody is looking.

What if we could assemble a paper openly, using a consortium of interested people, and then at a point where everyone is happy that the article is complete, we submit it for peer-review and publication in an open-access journal? That would have the disadvantage of killing off future edits (because the paper was reviewed as a static object) but the great advantage of producing a citeable article that would be curated by the journal. One could donate the final version to a relevant Wikipedia page, but the article is guaranteed as peer-reviewed, and has a DOI, etc.

We attempted this with our resolution paper last year. The research article we wrote was based on public-domain knowledge arising from our open online electronic lab notebooks. It was written on a wiki where anyone could contribute. It was submitted to a journal, reviewed anonymously and then published. In that case nobody outside the core team (the listed authors) contributed to the writing. They could have, but in the end did not.

The chemistry in question was, to a first approximation, solved. What we’d like to do is find a second solution to the problem that uses more sophisticated science (which is more expensive, perhaps prohibitively, but that’s OK at the moment). The area of science that is relevant is a certain chemical reaction – the catalytic, asymmetric Pictet-Spengler reaction. In the course of doing this research we’ve been looking back over what’s known of this reaction and when one does this one can’t help thinking about writing a review. We thought it would be interesting to generate such a review in the open, allowing anyone else with an interest to participate.

The current draft is here. The talk page contains what needs to be done and allows a trail of author interactions. Two of my students and I have to date done most of the work. Three other people have expressed firm intentions to edit the document and add sections. We have applied a time limit to the writing process of the end of February because that’s when semester starts in Sydney and it would be nice to be finished with it by then. A Dropbox folder contains all the relevant things – pdfs of relevant papers and the raw files used to generate the diagrams. People working on the paper have access to that.

This experiment in writing chimes with what a few others have been saying about the review process (my summary with links is here). Tim Gowers, for example, has been talking of a possible website where papers could be deposited as drafts (arXiv style) and then people be allowed to review them for improvement before submission. There is an interesting discussion to be had there about a “currency” of peer-review – that if someone undertook some editing of an article, could their contribution be reciprocated, maybe quantitatively? This goes to forming an incentive to participate in peer review, and also goes to how we could build reputation in academic circles – an attractive idea for younger scientists who would like to engage in writing papers and building careers, but for whom there is no clear way to do so because they lack the usual academic monikers. Currently professors do peer review (or are meant to) yet have no great incentive to do so beyond the better parts of their nature, and graduate students may be involved but receive no recognition outside their groups. Can lessons learned in open source software construction be applied to writing papers? How would the infinite gradations of expertise work? Would there need to be restrictions on permissions to authorship based on one’s current standing in such a system? These are really interesting questions that go to the future of how we write papers, so this experiment is an attempt to look at those things.

Fundamentally what we’re proposing is simple – people get together to write. Everything can be checked and edited. We have instituted a “quality control” section in the talk page to ensure that everything written is checked by at least one person, who identifies themselves. I have assigned myself as corresponding author to ensure the buck stops with someone. I am keen, for example, for the review to be well-written besides just being correct. I have read a lot of perfectly correct reviews which often require a lot of coffee.

The challenge of this model is this: How much work constitutes authorship? I don’t know. That’s the purpose of the experiment. At this stage, with a small number of participants, I can do this manually. If I think someone has contributed significantly, I will allow that person to be named. If the contributions are minor, that person will be “acknowledged”. It is possible for a student to be named, but not their professor (very much not what happens in my discipline) but I am making it clear to anyone volunteering that I have to check that with whomever is paying their salary, because I’m polite like that. It is certainly possible that if someone does a very large amount of work on the review, taking it in new directions or rewriting large chunks for the better, that that person should become (a) corresponding author, because being corresponding author means taking ownership of the end result.

It’s certainly been an interesting process thus far (20 days). Writing in this way feels “live” – edits can come at any time, from anywhere. There is a history of the page so things can be tracked and undone. The writing has gone quickly, apart from in the last five days where my students and I each independently became involved in other things that took us away from writing. But the fundamental aim is: to produce a complete, high-quality review of an area. I am very interested to see if that is what happens.

Where to submit? I’m considering the small number of options in chemistry. PLoS ONE does not publish reviews.

I’d also be grateful for any related examples of distributed paper writing where the draft assembled in the open then went to peer review. I am assuming that the papers arising from the Polymath project (rather than just the project itself) were constructed by multiple people, but if anyone knows of other cases, please say.