Anatomy of an Open Science Paper

We use lab notebooks to record research. Why not publish lab notebooks alongside papers?

My lab just published a paper on some chemical methodology towards some potential tuberculosis drugs we finished last year. The chemistry was carried out by Kat Badiola, with bio testing courtesy of my colleague Jamie Triccas at Sydney Uni.

In some senses it’s a traditional paper, but it has an interesting feature.

Standard, brief papers are like press releases, conveying the choicest bits of the science. Most manuscripts we often read are a little like this.

Which is useful but of course ridiculous – like icing with no cake. So we often prefer to publish larger pieces of work, where the choicest bits are accompanied by a “Supporting Information” section – stuff you consult if you want to know more. Usually this is a PDF file, usually containing pictures of datasets – in Chemistry this is usually copies of NMR spectra, for example.

Cake
(via https://www.flickr.com/photos/pejrm/7589142384/in/photostream/ (CC BY-NC-SA 2.0))

Which is also ridiculous on its own, since it’s 2014 and megabytes are cheaper than milk now, so we’re able to include raw data files too (though people seldom do, I don’t know why) and we’ve done so with this paper. So you can download the files and reprocess spectra for yourself if you like. Useful for all sorts of obvious reasons I won’t list here.

Which is great but not enough.

What about all the “failed” reactions? What about the repeats, so that one can assess reproducibility? What about the comments and ideas we all put in lab notebooks, such as things to try next? What about the strands of the projects that didn’t quite work out, but may yield to another investigator? What about photos of the science (genuinely useful in organic synthesis), or other data that would help someone wanting to take this research on?

In other words, wouldn’t it be useful to include the lab notebook as part of the paper?

That’s what we’ve done here. The electronic lab notebook Kat used – it’s been zipped up and put on our Uni’s institutional repository. You can download it, unzip it and explore it as a snapshot using a web browser – all the links work, and every experiment is in there.

Components of a Paper

Useful, and easy to do. Very easy for open projects, since you are already sharing everything so why not zip up the notebooks and include them? It’s probably more difficult for closed projects only because if a lab notebook is closed it’s probably not necessarily in the best shape to be shared with others – openness in record keeping can encourage (not guarantee) a way or recording activity that is understandable by others outside the team. Including the notebook is essentially impossible if you’re still using a paper lab notebook (for shame!)

Thanks to the Labtrove team at the Uni of Southampton for capturing the snapshot and generally being great.

(Related links: Some suggestions as to how we might change journal articles in future are here. More on the browser-based notebook Labtrove here and here. Please comment below if I’ve missed anything.)

Crowdsourcing Drug Discovery

Open Source Malaria has completed an experiment in crowdsourcing for open drug discovery.

Identifying and developing medicines is a labour-intensive process, particularly in the discovery and optimization phases, and most particularly in the physical preparation of samples of new molecules for testing: a phase that consumes large amounts of time and money and is often a roadblock. One of the obvious things to do is to crowdsource the synthetic chemistry using students.

In open source projects there is the tantalizing additional possibility that student teams can assemble to work on those problems because what’s needed can be fully described in the open. The openness means that the teams could learn from each other, share data and receive peer review and mentorship from interested experts based elsewhere, such as in the pharmaceutical industry.

There have obviously been examples of crowdsourcing in various scientific arenas such as genomics (e.g., 1, 2) and there have been closed groups of students operating in the area of drug synthesis. There are examples of crowdsourcing initiatives in attempts to identify biologically active natural products (1, 2), Joerg Bentzien ran a Kaggle competition in in silico small molecule modeling, Urmi Bajpai is working with students in her lab on some biochemical projects (see also this earlier story) and there are other preparative activities of many kinds dating back to things as diverse as the AIDS quilts. I wasn’t aware of any students participating in crowdsourced synthetic chemistry as part of a project that was open source, though Patrick Thomson provided a spectacular example of how openness can lead to high quality scientific contributions from individuals. Mass recruitment of synthetic expertise is going to be one cornerstone of any scaled-up vision for Open Source Pharma, and I was very keen to see if we could complete this exercise in OSM as a precedent, and maybe learn how to make sure it works most effectively.

We just completed this precedent. Around 50 students from Lawrence University in the US midwest worked on the synthesis of six new analogs in the “Near Neighbour” (NN) branch of Series 1 of OSM. The compounds were mailed to a different lab in the US for biological evaluation. Active compounds were discovered – one of them new to the project and quite potent. I find the closure of this loop tremendously exciting.

Data for the Lawrence University Compounds – check out the red value for OSM-A-3!

Here’s how it worked. I was contacted a few days before the end of 2012 by two US-based academics asking whether their undergraduate cohorts could contribute to OSM. This had been an aim of the project way back in the design phase and not something we’d gotten round to. For one of the academics, Stefan Debbert, Murray Robertson and I wrote up a description of a synthetic route we’d been using for the NN compounds. The chemistry is not trivial, but possesses the advantage that the compounds are typically solids that can be easily purified and are stable under ambient conditions. Stefan looked into this and came up with some analog structures he felt his students would be able to access, and without any further input from us he got this working in his lab class in early 2013. Around June/July he wrote to say his students had finished the syntheses, and in November he uploaded all the data to the online lab book. After some quality control checking of the data by others, the compounds were shipped to Kip Guy’s lab at St Jude’s where Julie Clark performed the assessment of potency and the data were put up online last month, thereby closing the loop on this part of the project. This whole process took longer than it should have since we were all feeling our way a little on this one.

The data have come in just in time for the writing of the first OSM paper, which is at an advanced stage. We are now faced with the interesting challenge of how to credit this student cohort with authorship. What a great problem to have.

A Student Hard at Work in the Debbert Undergrad Lab

The take-home: a cohort of undergraduates successfully made new molecules that are potent in killing the malaria parasite. The data are valuable as part of a larger project of current research that will be published in the peer-reviewed literature.

 The students were able to make an impact because:

1) A current/future research need was openly described

2) The rules behind the project clearly stated that anyone could participate

3) The student team was locally and carefully mentored by a dedicated individual (Stefan) willing to engage in an unusual activity

4) The details of what was needed were described in full, i.e. previous, related research that made it clear the activity would not be open-ended

5) The outputs were actively used, i.e. data deposited in the relevant lab notebook for the benefit of the wider project and molecules tested by a laboratory willing to do so.

This is scaleable. Any team of undergraduates can engage in real research in this way, and perform scientific activities that are valuable to a “live” project and which might result in the undergraduates being able to publish their work. The work could be out-of-hours or, more excitingly (and as in Stefan’s case) part of a formal lab class for which the students receive credit – more on this below. I don’t know about you, but if I’d had the chance to make real molecules for a real research project as an undergrad, I would have found that quite motivating.

One of the neatest aspects about doing this openly is the quality control. A cynical onlooker might say “Well, these are just a bunch of undergrads – how can we trust the work?” You don’t have to. In an open source project, you have access to the raw data, so you can check the quality for yourself. Many people don’t trust data in synthetic chemistry journals in any case (trust is a problem across the discipline) but raw data in open projects solves this problem in as much as it can be solved.

So what would I change next time?

1) Fully Independent Contribution. The collaboration was set up because Stefan contacted me to ask if he could help out. That’s fine, but in an ideal world it would be clear to a lab director what was required without asking – that’s the aim of the Github To Do List that has since been introduced to OSM. But even there it’s sometimes not obvious what’s required and I think we can use Github more effectively. As much as possible in an open project one needs to promote independent contributions and an interlinked mentorship structure (the “Linus doesn’t scale” problem). Having said that, after the initial communications, Stefan ran the whole class independently, which is a testament to his achievement.

The OSM Consortium has a public to do list

2) Direct Data Deposition. The students did not deposit data into the lab notebook directly – Stefan himself aggregated the work and deposited the data for each compound himself. I think this arose because using the lab notebook presented a barrier to participation that was just slightly too large, and there was a concern that students might copy each other. I want to make sure the lab notebook is so easy to use that next time the students are happy to put the data up themselves as they are working – which is the standard practice in OSM. This means making sure that everyone understands that the data they are reading is like any data in any lab book and just because it’s on the internet doesn’t mean it’s correct or final.

One of Stefan’s ELN Entries

3) Global Lab Buddies. Ideally we would have two teams in different places communicating with each other on a synthetic route and not via the central project hub. I have in mind something that ought to be of interest to funding agencies like Fogarty, USAID or Wellcome – a lab class in a developed country pursues a synthetic route and a lab class in a developing nation pursues the same or related experiments, with direct collaboration between students and with mentorship in both places. This has the potential to be an inexpensive way of creating a “lab buddy” scheme where students can share what they learn with each other directly (with only light guidance from a PI, so that the collaboration scales well) and there is no need to spend money moving people around or organizing bench fees; effective lab-based crowdsourcing is significant in terms of how we think about organizing “networks” between universities (e.g. the APRU or WUN) – you can do a lot more for a lot less money if you don’t have to buy airplane tickets.

(CC image from https://www.flickr.com/photos/zachstern/2852218181/sizes/m/)

The experience makes me wonder about best practice:

1) Mentorship. How many mentors are needed for a class? There needs to be a committed local champion like Stefan, and the class size can be left up to the lab director, who also manages local health/safety requirements. There needs to be a central mentor (me, in this case) to help with any snafus. I think there is real potential here for additional mentorship from pharma professionals who could ensure that the molecules being made are reasonable for a drug discovery/development project. Such mentorship could be provided directly pro bono, or organized through a PDP (e.g. MMV, DNDi) or learned organization (e.g. the RSC). Pharma experts in a given location could contribute to their community in this way – e.g. Merck could mentor students working in schools in New Jersey.

2) Replication. It makes sense to me for two students to run each experiment, as a form of replication check. This is real research, so there should be controls, particularly if the students are not very experienced. It’s also probably a good idea for the students to make known compounds as part of the synthetic cluster. The need for positive controls of this kind is particularly keen in open projects since biological evaluation may well be performed in different labs where protocols will inevitably differ. A modest degree of replication makes everyone happier with the numbers ultimately obtained.

3) End-to-End. The work the students do needs to be incorporated into the larger project (at least on a wiki or in a paper) which may require that the molecules are evaluated in some way. If this is needed, then it’s crucial that it happens, so that student effort is not wasted or merely archived. Never ask a crowd to do anything that is not then used. This needs to be factored in to the planning of project resources and means that before the synthesis starts, there needs to be a commitment from somewhere that the molecules will be taken on (in this case, to kill a nasty).

4) How to Assess – or The Brown Oil Problem. I think the greatest power of crowdsourcing lab work lies in incorporating this into the undergraduate curriculum. Imagine that you’re planning a lab class, and you check online to find 38 different current project needs in open malaria, TB and Ebola projects. Let’s say the Open Source Pharma vision is made real and there is a repository of such projects that are known to be active that very day. This is not very far-fetched. You would, I think, be very keen to use one of these real research needs in your class, to fire up your students about how cool research is. If the syntheses were well-designed you could get started as soon as the starting materials were in. But the complication is: How to Assess the Project? What happens if a student fails to prepare a molecule that ought, in theory, to be preparable? How do we assess a project outcome if the outcome is novel, as part of a research project? I have no good answer to this, but feel that there are steps that can be taken:

i) Clarity of Design. The designer of the synthetic route to be undertaken must provide as much detail as possible about the ease of the synthesis. This honesty is simple in open projects – in the NN synthesis undertaken by Stefan it was possible for him to see all of the mis-steps and failures that the OSM team had wrestled with previously, and it was therefore possible for us to provide a lot of advice about key steps. Students themselves can even cite previous attempts as mitigating circumstances (“Though the molecule could not be prepared it was noted that Mat Todd similarly failed to generate this key intermediate in experiment 34-6 (13th Oct 2012)…” etc)

ii) Emphasis on Approach. The marking scheme used by the lab director needs to focus more on the approach used by the student and the quality of the scientific record produced than on the final outcome. This is in any case good academic practice. There remains an issue of plagiarism if students are posting items into the public domain, but one has to wonder about the value of having the students write reports that are vulnerable to this in the modern age.

iii) Time-resistance. The marking scheme needs to be immune to later discoveries. If it is ultimately found that a particular compound is unstable, a student who has earlier made this compound and found the same phenomenon cannot have their efforts re-marked, just because this makes the assessment too conditional and complex.

OSM is supported financially and scientifically by the Medicines for Malaria Venture and the Australian Government

If people want to get involved in this kind of activity, and it’s not clear from the OSM To Do List or the wiki what is needed, then don’t hesitate to get in touch directly. It’d be great to generate some more synthetic schemes suitable for students. If you are planning a research project yourselves, and are writing a proposal, consider including some element of crowdsourcing+open source so we can have lots more students contributing to real research in the public domain.

An Example of Open Source Drug Discovery

“Open Source Drug Discovery? How does that work?” I am asked this quite a lot. There are some principles and core practices that are involved, embodied in the Six Laws, but those are quite hifalutin. Let me give a practical example.

The main point is that everything is open, so anyone can come along, make molecules, do biology, do informatics, contribute data to the lab notebooks, coordinate the research and generally contribute in any number of ways for which there are simple technological solutions. All sorts of inputs are possible, including experimental ones.

Earlier this year there was a need to make a bunch of molecules as part of the first medicinal chemistry campaign in the open source malaria project. The hit molecule that was the starting point had an ester in it, and initial results had shown that the ester was a problem. There were several ester replacements that needed to be synthesized. Many of these had been made. A few had either not been tackled or were proving difficult to make. Alice Williamson made up a “Wanted” poster to highlight that anyone could have a go at making them (see below, and in these nice images of the process of collaboration that Alice also made).

 

 

Because these difficulties were being tackled in real time, other scientists could read about them and suggest solutions. Patrick Thomson, an Edinburgh PhD student, was reading this and decided to join in. He got going in the lab on the sulfonamide sE, started posting his data to the same lab notebook the Sydney team were using (to make it easier to compare and share data) and started his own blog to help marshall and summarize results. He corresponded with a student in my lab in Sydney to overcome synthetic hurdles. Several of us worked together to ensure quality control of the data – simple in the digital age.

 

And Patrick nailed it. He made the compound. He then mailed it off for biological evaluation. Not to Sydney, which would be a waste of good funds for freight charges, but to a local lab at The University of Dundee. With the appropriate controls and sharing of all data, testing of compounds in different labs can be trusted to deliver a consistent picture.

 

In the event, Patrick’s molecules (he tested 3) were inactive. This is a shame, but at the same time the ester/sulfonamide replacement is an obvious one to make and the data enhance the paper where these results will go, on which Patrick will be an author.

Patrick showed great initiative here, and philosophical agility – he got involved with something unfamiliar where he had not met the people he’d be working with. Maybe it’s because he agreed with the principles of openness, or maybe it was because he saw that all project data were being shared so trust wasn’t an issue – he could see the project involved quality science. Either way, Patrick has provided an excellent example of how open source experimental lab-based research can work on a practical level.

Which makes one think of all sorts of added benefits of working in a genuinely open manner. One is that open source research provides a low-tech infrastructure for researchers from developing (endemic) countries to get involved in collaborative projects with their peers in the developed world. There are no expensive online tools, no subscription fees for software. Researchers carry out their work in local labs, and share all the data online using existing technology solutions. Crucially the mechanism scales because researchers interact directly, rather than having to pass everything by a bottleneck professor. There is no monolithic “aid” program needed, just the means for people to work together effectively and efficiently. The involvement of endemic labs remains to my mind a top priority target for open research – the barrier is identifying the labs with the right level of chemical facilities (if anyone has any ideas…).

Today the Open Source Malaria landing page is launched – the project used to be called Open Source Drug Discovery for Malaria, but the name was changed partly to highlight more the real essence of the project – open source. Much of the research is about basic synthetic chemistry, or biology and yes it’s aimed at finding new medicines for malaria, but at the same time the power of the approach lies in how it changes the way we do basic science. The new website highlights the most productive tools used thus far – the lab notebooks, the Github To Do list, the Twitter account, the wiki.

 

 

With these simple tools, Patrick was able to make an important, material contribution to an experimental research project without ever leaving his Wimbledon-winning home country. Anyone else can do the same if the inviolable principles of open source are agreed upon – No Need to Ask.

 

Companies and Open Science

There is an argument that says we should not be making simple compounds in academic research labs, but rather using specialist services to make molecules with which we then do interesting science. There is a lot of truth to that, particularly with medchem projects involving structures that are simple, or for which there is synthetic precedent. We’re making compounds in my lab as part of the open source drug discovery for malaria project because we’re trying to drive a unique way of doing things in the open and show that that works, but if everyone started doing open source drug discovery, and it became the standard way of doing things, I wonder if the process would be more efficient if we used a network of CROs, or even a set of super-labs specializing in synthesis (a proposal I’m thinking about, but I can’t decide whether it should be government- or industry funded). I think Elizabeth Iorns of Science Exchange has mentioned this idea too.

The potential for CROs to make an impact on open projects is great. This is because an open source project consists of a nimble network. In regular research projects, funded by government grants, one will typically define the project structure at the start, cost up the resources, and then carry out that research in a manner that is closely related to the proposal. Naturally there will be variations in science and approach, but there are usually not huge variations in the structure of the team. Indeed, if there is no clear plan of how to carry out the research at the outset, that can be viewed dimly by referees. If the structure of the project changes dramatically during the funding period that can often require explicit reasons in end-of-project reports. There is thus some incentive in keeping the project as-designed. There may also be professional or emotional ties that mean a team is kept together even if it turns out it’s not the best team for the job.

In contrast open source projects demand a nimbleness – that people get involved if needed, and that the required people may not be those defined in advance. I have found this challenging to explain in grant proposals, where I have tended to define project participants at the outset, and mentioned that the team will adapt and change in a process that is not fully controlled. I try not to highlight it too much, even though I know the ever-shifting project structure is one of the great novelties and strengths.

So open projects are strong because of a loose set of transient connections. Interestingly this kind of network was recently shown to result in more cooperative behavior than networks with strong, more permanent connections between members. It’s a model, but an interesting one that to some extent mirrors what we’ve seen in our open science projects.

Nimble projects can use small inputs from a number of people if needed, and those people, once they have contributed, have no obligation to continue to contribute. An open structure relies on enthusiasm and available resources and is very good at accepting small components from many people. This kind of project takes much more coordination, and is more complex, but should ultimately be more efficient provided there is strong, open management.

So if we want to evaluate a series of compounds in a medchem program, for example, where the compounds are not necessarily methyl-futile analogs, but consist of different structures that may require different kinds of chemistry, it makes more sense to break up the work involved, and have several labs make a small number of compounds in parallel. It makes particular sense if there is a lab somewhere with expertise in the synthesis of a particular structure class, for example, who might be able to make that kind of structure quickly.

CROs are set up perfectly for this, since they could make a single compound for a project during a short (1-2 week) downtime for a lab member, and then once that compound has been made they are not required to be involved any further. Rather like a bit torrent client that uses spare capacity in short bursts.

The CRO marketplace is crowded and, I’m guessing, working with tight profit margins. Naturally companies are under pressure to attend to their core business. So why would a CRO want to contribute to an open source project? Several reasons:

1) PR argument 1: If the project is philanthropic, the company gains a positive public image for contributing to the solution of that problem.

2) PR argument 2: By performing the work in the open, where all data are shared in real time, a company can demonstrate its technical prowess, i.e. onlookers can see how fast a company can work, and the quality of the work in detail and in a real case scenario. Big pharma could observe the quality before contracting the company for new work.

3) Employee satisfaction I: I believe (without data) that people are people and that employees working on a philanthropic project gain professional satisfaction from doing so.

4) Employee satisfaction II: Scientists like to solve problems, and a challenge in parallel to one’s paid work, particularly if part of a team effort, is satisfying. Those taking part will likely secure publications.

5) Exploitation of Unavoidable Downtime: Even an efficiently-run CRO will have periods of a week or so where an employee is not fully occupied. That spare time can be put to good use with the kinds of small contributions needed by open projects.

6) Philosophical Agility: If a marketplace is tough, an edge is given to those companies who can distinguish themselves. Taking part in an open research project, which is an unusual thing to do, demonstrates a company is happy to think laterally and engage with ideas outside the comfort zone of their competitors.

These arguments mirror those that I presume operate in what is an excellent model for what I’m proposing – the pro bono work performed by law firms. This is a major component of the legal profession. Now in many cases the big pro bono cases are those taken on by the big firms, and the analogous situation is the fantastic work that many of the big pharma companies are doing in philanthropic work for neglected tropical diseases – impressive and little-known by the public. One might argue that the CRO sector, which is more under pressure, cannot afford to contribute this way. I disagree, and return to the idea that an open project can be advanced by small contributions, appropriately coordinated. Rather than a CRO being locked into a collaborative arrangement for a long time, a CRO could be engaged in an open project (to make a molecule or acquire some data) for days or weeks before moving on to something more aligned with core business.

What’s needed? Open data and an open research method will allow people to find and see what a project needs in a very responsive manner. But it’s imperfect since we may not search for the right thing, or the project participants may not record/coordinate their data in a way that is easily found. Perhaps instead we need a Clearing House for Contributions to Research. A place where people can say what work they need doing, and the clearing house tries to find companies who can do that work. Such a thing exists in law – PILCH – that matches those seeking pro bono work with those companies best able to provide that work. This mirrors the idea of the Molecular Craigslist I floated – where people could upload structures of molecules they need, and people can bid to provide those, i.e. needs-driven, not supplier-driven.

Science Exchange and Assay Depot are like this, but again the process is not open (as far as I know). Thus I can request a service and receive quotes/offers (though to be clear I’ve never tried that) but I suspect the bidding process is not an open market auction-style. I’d like to see an open variant where there is a genuine process of bidding, in case someone wants to provide a molecule or service for free, for example, or in case the open procurement process itself could stimulate interest in contributing to the project. Perhaps Science Exchange and others could have an option to make the matchmaking process open. i.e. I need X, who would like to provide it?

To date I have pitched the idea of contributing to the open malaria project to three CROs. One didn’t reply. One said they weren’t interested (but from the response I suspect my request was not well understood) and one said the following, to paraphrase: “Thank you for your mail. This request is very innovative. We are not interested in taking part because no other CRO is currently taking part”. I found this last reply quite interesting because the logic to me was the reverse of what I had expected. To me, it would have been a plus that nobody else was doing open source work, since were I to take part that would distinguish my company from all the others.

So we’re going to be sending out more requests to CROs to make molecules as part of the osdd malaria project – at the time of writing there are several that are still needed. If my arguments above are right, there should be interest from the CRO sector. The question is how best to engage with that sector.