Halfpipe

ATAC-Seq Pipeline implemented as a webinterface.

Setup

Halfpipe requires Python3 and uses the webframework Django.

Make sure you have the following dependencies installed:

• Postgresql
• Redis

To install them on Ubuntu/Debian use:

apt-get install postgresql postgresql-contrib libpq-dev redis-server

Installation under MacOS requires homebrew:

brew install postgres && brew install redis

Database setup

Create a database superuser with the same username as your current user:

sudo -u postgres createuser --interactive

To create database for the project use:

createdb halfpipe

Storage backend setup

File storage is done with a S3-compatible storage API called minio.

On MacOS there is a homebrew package available: m̀inio/stable/minio. Linux users can download the binary directly with:

# Download + make exec
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio

Then minio can be started with:

./minio server shared

Other dependencies

Python dependencies are managed via pip. It is however advised to install those dependencies into an virtual environment. This can be created with:

python3 -m venv venv

To activate the environment use:

source venv/bin/activate

This needs to be done in every shell that should use our newly created environment.

For development purposes we suggest installing the Python dependencies in the requirements/testing.txt file:

pip install -r requirements/testing.txt

In order to install javascript and css dependencies use:

python manage.py install_dependencies requirements/static.json

Running

Jobs are managed with Celery, a distributed Task-Queue for Python. Start a worker with:

celery -A halfpipe worker --task-events

To inspect the job queue and status use celery -A halfpipe events Keep in mind that the worker needs to be restarted in order to reflect code changes.

The Django Webserver can be started with:

python manage.py runserver

Changes on schema require running python manage.py migrate to update the database. For more information see Django's model documentation.

Testing and best practices

Halfpipe uses Pytest to run tests:

pytest

The Python code style conforms to Black. To format all Python code use:

black .

Bowtie Indices

Halfpipe does not ship with a index mapping, but includes documentation on how to obtain them. Follow the instructions in halfpipe/fixtures/index_mapping.yaml and install the fixtures with:

python manage.py loaddata halfpipe/fixtures/index_mapping.yaml

Project organisation

TODO (till)

Troubleshooting

TODO (till)