Transparent Programming: Important Habits for Reproducibility and Research Integrity
To you for being here!
To the Society for Medical Decision Making for supporting the development of this workshop
To Madison Coots for helping develop this workshop!
You can find all workshop information at https://code4mdm.github.io.
This code is a kludgly, ugly, inefficient mess. (…) It is probably riddled with problems, mistakes, bugs, inefficiencies, vestigial code stubs, etc etc. (…) I am as confident as I am capable of being that all of the factual claims that were made in the manuscript are accurate.
We need to do more: we need to inspire trust. - The code is correct (and I have made it easy for you/someone to check); - My workflow is robust; - My workflow itself is accessible, and I will be guiding you through it.
What do you need to execute this project? Where do you start?
Demonstrate a trustworthy workflow.
Automated analyses trace your steps, and prevent human error (or at the very least: document it).
Share your data, release your code, publish your findings.
We will take you through a workflow (in a broad sense!)
Setting up a project
Establishing a robust backup / version control system
Writing documentation
Making your project accessible
Our group has many different abilities and experiences. We hope you will value this as much as we do!
Be prepared to work with others around you.
Ask questions when you have them!
Feel free and safe to share your expertise and experiences.
We want to teach you good habits that will make your work more accessible, trustworthy, and reproducible by others. In doing so, we have tried to identify those habits that are a good return on investment: meaning, they save you time in the not-so-long run.
A research compendium is a collection of all digital parts of a research project including data, code, texts (…). The collection is created in such a way that reproducing all results is straightforward.
Source: The Turing Way
Contain your project in a single recognizable folder
Distinguish folder types, name them accordingly:
Initialize a README file, document your project
Choose a license
Publish your project.
You can set up a project template using a nifty tool called cookiecutter.
First, ensure you have cookiecutter installed:
or with conda:
or try the following alternatives:
A command-line utility that creates projects from cookiecutters (project templates). e.g. creating a Python package project from a Python package project template.
There are MANY templates available for your purposes. Take a look!
We have chosed a designed a template based on my own personal preferences. Go ahead and install it!
Answer the questions cookiecutter asks you, and browse the resulting project to see where your answers ended up.
Your project should be transportable between computers.
For this reason, you should use relative paths only: compare
/Users/jacob/Dropbox/data/zincfinger.json
./data/zincfinger.json
./
means: in this folder
../
means: one folder up
Copyright is implicit; others cannot use your code without your permission.
Licensing gives that permission, and its boundaries and conditions.
Choosing a license early on means being aware of your license as the project proceeds (and not creating conflicts).
There are over 80 OSI-approved licenses (and many, many others) to choose from.
This is one I like to use:
What is important to you? What does your lab use? Choose your own license!
Uh… Isn’t ‘publication’ the thing you do… at the end?
No! Publishing your project at an early stage - forces you to consider readability throughout - minimizes the mess you have to deal with when you (finally) decide to publish - allows collaboration and support - facilitates sharing and re-use.
But what if someone scoops my code! I’m a revolutionary, they will steal my ideas!
If you are super paranoid, you can always opt for a private repository. It is your work & up to you. But consider the advantages!
If you have sensitive data…
Don’t include your data in your software repository (that’s not what they are for anyway).
Consider generating simulated data so your code can run regardless.
And for all data:
synergistic with version control software git
makes history public and accessible (eek!)
allows publication of different releases
provides a platform for interaction and collaboration
(or other stable repository, like the OSF) - direct archiving supported from github to zenodo