HFOS Documentation¶

Architecture¶

The system consists of two parts: 1. A backend written in Python. It handles communications, data handling and other general services provided by independent modules. 2. To communicate with users, a HTML5 based Frontend is deployed to most modern web browser capable clients.

Meshed Operation¶

The cloud/server-less mesh operation enables local independence and adaptability regarding network environments.

No platform specifics¶

It also eliminates the need to write platform specific applications (e.g. native Android or other mobile platform applications)

Install script¶

The script currently only supports Debian based systems.

To use the install script, get the source code (see Getting the source) if you don’t have it already, then invoke the script with root permissions:

$ sudo ./install

If you run into trouble or get any unexpected errors, try the complex installation procedure.

Docker¶

We’re providing a Docker image for installation.

The command to get the current testing release is:

$ docker run -i -t hackerfleet/hfos hfos_launcher.py

Planned Installations¶

• We’re planning to offer ready-made SD card images for various embedded systems.
• A custom NixOS system is planned as well.

Backend¶

HFOS’ backend has a few dependencies:

• Python: >= 3.3 (or possibly pypy >= 2.0)
• Database: MongoDb

A few more dependencies like nginx, and some python packages provided per distribution are recommended, but not strictly necessary.

The HFOS Python package additionally installs a few pure Python libraries:

• Circuits
• Click and a few supporting packages
• PyMongo
• PyOpenSSL
• PyStache
• JSONSchema
• DPath
• DeepDiff

Frontend¶

The frontend is built with

• node
• npm

and others. The detailed list can be found in frontend/package.json after pulling the frontend git submodule.

Latest Stable Release¶

By design, there is currently no stable release planned.

The latest stable releases (if there should ever be one) could be downloaded from the Releases page (specifically the Tags tab).

Latest Development Source Code¶

We use Git for source control and code sharing.

The latest development branch can be cloned using the following command:

$ git clone https://github.com/hackerfleet/hfos.git

For further instructions on how to use Git, please refer to the Git Website.

Preparation¶

These instructions are for Debian or Ubuntu based systems. Installation on other distributions is possible.

Before doing anything with HFOS, be sure you have all the dependencies installed via your distribution’s package manager.

For Debian Unstable use this:

$ sudo apt-get install nginx mongodb python3.5 python3-pip python3-grib \
                       python3-bson python3-pymongo python3-serial

If you want (and can, depending on your platform/distribution), install the mongo and bson extensions for speedups:

$ sudo apt-get install python3-pymongo-ext python3-bson-ext

The system will need to get a bunch of more dependencies via npm to set up the frontend, so install npm and if necessary the nodejs-legacy-symlink package:

$ sudo apt-get install npm nodejs
$ sudo npm install npm@4.2.0 -g

If you want to install the full development dependencies to write documentation as well, you need to install the enchant package:

$ sudo apt-get install enchant

In case you want to use raster (or in future: vector) charts in HFOS’ map module, you’ll need to install libgdal and its binaries:

$ sudo apt-get install gdal-bin python-gdal

Note, that it is necessary to install python-gdal 2.7 - not the python3 variant, as the scripts are not included in that.

Getting the source¶

To initially obtain the development source code if you don’t have it already, use git thus:

$ mkdir ~/src
$ cd ~/src
$ git clone https://github.com/hackerfleet/hfos
$ cd hfos
$ git submodule init
$ git submodule update
$ git pull
$ cd frontend
$ git pull

Backend¶

The manage tool usually can install everything you need. It starts by adding a new system user for HFOS and generating a (currently only self signed) certificate.

The process also involves installing the supplied modules, getting the frontend dependencies, building and installing the documentation, etc.

It also creates two folders in /var (lib/hfos and cache/hfos) for hfos’ tile- cache and other stuff as well as install basic default provisions into the database.

Finally, it installs and activates a systemd and nginx service script to launch HFOS on bootup and make it available to users.

$ virtualenv -p /usr/bin/python3.5 --system-site-packages venv
$ source venv/bin/activate
$ pip install -Ur requirements.txt
$ python setup.py develop
$ sudo venv/bin/python hfos_manage.py install all

If you want to develop (documentation) as well, you’ll need to use the requirements-dev.txt instead of the normal one.

If you want to manually start HFOS, invoke the launcher thus:

$ sudo ./venv/bin/python hfos_launcher.py

Running the launcher as root to be able to open ports below 1024 should be safe, as it drops its root privileges, unless you specify –insecure, which is strongly discouraged and only meant for development purposes. The default is to use port 8055 and relay that with the supplied nginx site definition

Documentation¶

The documentation is available online on ReadTheDocs.org. If you wish to build and install the included documentation for offline use, run these commands:

$ sudo ./venv/bin/python hfos_manage.py install docs

This installs all necessary documentation tools and copies the files to the expected HFOS web data folder.

You can also build the PDF file (and various other formats) by using the Makefile inside the docs directory.

$ cd docs
$ make pdf

Just running make without arguments gives you a list of the other available documentation formats.

Installing from a Source Package¶

If you have downloaded a source archive, this applies to you.

$ python setup.py install

For other installation options see:

$ python setup.py --help install

Installing from the Development Repository¶

If you have cloned the source code repository, this applies to you.

If you have cloned the development repository, it is recommended that you use setuptools and use the following command:

$ python setup.py develop

This will allow you to regularly update your copy of the hfos development repository by simply performing the following in the hfos working directory:

$ git pull -u
$ cd frontend
$ git pull -u

Windows & OS X installation notes¶

These instructions are WiP. The easiest way to get HFOS on Win7 or newer is to install and user Docker or a virtual machine

To install on Windows, you’ll need to install these packages first:

• Python 3.5 https://www.python.org/downloads/windows/
• MongoDB https://www.mongodb.org/downloads#production
• pymongo
• numpy

Platform specific¶

There are some collected instructions for various hardware platforms:

Communication¶

• IRC Channel on the FreeNode IRC Network
• Issue Tracker

Standards¶

We use the following coding standard:

• PEP-008

We also lint our codebase with the following tools:

• pyflakes
• pep8
• mccabe

Please ensure your Development IDE or Editor has the above linters and checkers in place and enabled.

Alternatively you can use the following command line tool:

• flake8

Tools¶

We use the following tools to develop HFOS and share code:

• Code Sharing: Git
• Code Hosting and Bug Reporting: GitHub
• Issue Tracker: Issue Tracker
• Documentation Hosting: Read the Docs
• Package Hosting: Python Package Index (PyPi)
• Docker Hub Automated Builds: Dockerhub
• Continuous Integration: Travis CI
• Code Quality: Landscape
• Frontend Testing: Browserstack
• Translations: Transifex

We strongly suggest familiarizing with all of them, to make sure you understand our CI.

Big thanks to all of these magnificent and free-for-opensource services!

Submitting Bug Reports¶

We welcome all bug reports. We do however prefer bug reports in a clear and concise form with repeatable steps. One of the best ways you can report a bug to us is by writing a unit test (//similar to the ones in our tests//) so that we can verify the bug, fix it and commit the fix along with the test.

To submit a bug report, please Create an Issue

Writing new tests¶

We’re not perfect, and we’re still writing more tests to ensure quality code. If you’d like to help, please Fork HFOS, write more tests that cover more of our code base and submit a Pull Request. Many Thanks!

Adding New Features¶

If you’d like to see a new feature added to HFOS, then we’d like to hear about it~ We would like to see some discussion around any new features as well as valid use-cases. To start the discussions off, please either:

• Chat with us on #hackerfleet on the FreeNode IRC Network

  or

• Create an Issue

Prerequisites¶

It is highly recommended that you install and use virtualenv for all your Python development and production deployments (not just HFOS).

It is also convenient to install and use the accompanying shell scripts and tools virtualenvwrapper which adds a nice set of workflows and functions useful for both development and deployments.

$ pip install -U virtualenvwrapper
$ source $(which virtualenvwrapper.sh)

In addition to the above recommendations you must also have a Git client installed and ready to use as well as your Editor/IDE of choice ready to use.

Getting Started¶

1. Fork HFOS (if you haven’t done so already)
2. Clone your forked repository using Git
3. Create a new virtual environment using virtualenvwrapper
4. Install the Development Requirements
5. Install HFOS in “develop” mode

And you’re done!

Example:

$ git clone git@github.com:yourgithubaccount/hfos.git
$ cd hfos
$ mkvirtualenv hfos
$ pip install -r requirements-dev.txt
$ python setup.py develop

Software Development Life Cycle (SDLC)¶

We employ the use of the SCRUM Agile Process and use our Issue Tracker to track features, bugs, chores and releases. If you wish to contribute to HFOS, please familiarize yourself with SCRUM and GitHub’s Issue Tracker.

Bug Reports¶

• New Bug Reports are submitted via: https://github.com/hackerfleet/hfos/issues

• Confirmation and Discussion of all New Bug Reports.

• Once confirmed, a new Bug is raised in our Issue Tracker

• An appropriate milestone will be set (depending on current milestone’s schedule and resources)

• A unit test developed that demonstrates the bug’s failure.

• A fix developed that passes the unit test and breaks no others.

• A New Pull Request created with the fix.

  This should contain: - A new or modified unit test. - A patch that fixes the bug ensuring all unit tests pass. - The Change Log updated. - Appropriate documentation updated.

• The Pull Request is reviewed and approved by at least two other developers.

Feature Requests¶

• New Feature Requests are submitted via: https://github.com/hackerfleet/hfos/issues

• Confirmation and Discussion of all New Feature Requests.

• Once confirmed, a new Feature is raised in our Issue Tracker

• An appropriate milestone will be set (depending on current milestone’s schedule and resources)

• A unit test developed that demonstrates the new feature.

• The new feature developed that passes the unit test and breaks no others.

• A New Pull Request created with the fix.

  This must contains: - A new or modified unit test. - A patch that implements the new feature ensuring all unit tests pass. - The Change Log updated. - Appropriate documentation updated.

• The Pull Request is reviewed and approved by at least two other developers.

Writing new Code¶

• Submit a New Issue

• Write your code.

• Use flake8 to ensure code quality.

• Run the tests:

  $ tox
  

• Ensure any new or modified code does not break existing unit tests.

• Update any relevant doc strings or documentation.

• Update the Change Log appropriately.

• Submit a New Pull Request.

Running the Tests¶

To run the tests you will need the following installed:

• tox installed as well as
• pytest-cov
• pytest

All of these can be installed via pip install -r requirements-dev.txt.

Please also ensure - if you can - that you you have all supported versions of Python that HFOS supports installed in your local environment.

To run the tests:

$ tox

Cyclomatic Complexity¶

• Code Complexity shall not exceed 10

  See: Limiting Cyclomatic Complexity

Coding Style¶

• Code shall confirm to the PEP8 Style Guide.

• Doc Strings shall confirm to the PEP257 Convention.

Revision History¶

• Commits shall be small tangible pieces of work. - Each commit must be concise and manageable. - Large changes are to be done over smaller commits.
• There shall be no commit squashing.
• Rebase your changes as often as you can.

Unit Tests¶

• Every new feature and bug fix must be accompanied with a unit test. (The only exception to this are minor trivial changes).