Simphony-remote¶

The Simphony-remote is web service that allows users to start and work with simphony enabled environments remotely.

Key provided features:

Isolated working environments using docker containers.

No install remote access through a web browser.

Sharing of working sessions.

Based on community supported open source initiatives (JupyterHub)

Acknowledgments¶

This software is developed under the SimPhoNy project, an EU-project funded by the 7th Framework Programme (Project number 604005) under the call NMP.2013.1.4-1: “Development of an integrated multi-scale modelling environment for nanomaterials and systems by design”.

Contents¶

This documentation is for simphony-remote 2.1 (2.1.0)

Deployment¶

Single Machine¶

Deployment of the complete system in a single machine/VM.

Note

The following instructions assume a clean up-to-date ubuntu 14.04 system.

Retrieve the single user session manager:

git clone https://github.com/simphony/simphony-remote

Make sure that you are obtaining a recent version of Docker, at least 1.12. Full instructions available at the Docker website. A Makefile rule is provided for convenience. NOTE: this overwrites the docker.list file you might have setup in your /etc/apt/sources.d/ directory. You might be prompted for the root password to execute this:
```
make deps
```
Make sure your docker server is running, and your user is allowed to connect to the docker server (check accessibility of /var/run/docker.sock). You obtain this by running:
```
sudo service docker start
sudo addgroup your_username docker
```
and logging out and in again. Check if your docker server is operative by running:
```
docker info
```
Create and activate a virtual environment, then set the appropriate PATH for the node modules:
```
make venv
. venv/bin/activate
export PATH=`node bin`:$PATH
```
Install the python dependencies:
```
make pythondeps
```
And install the package itself:
```
make install
```
Generate the SSL certificates if you do not already have them. The resulting certificates will have names test.* because they are self-signed and are not supposed to be used for production. A CA-signed certificate should be obtained instead. The certificates will be created in the jupyterhub directory:
```
make certs
```
Create the database. By default, this is a sqlite file:
```
make db
```
Change dir into jupyterhub:
```
cd ./jupyterhub
```
and verify that jupyterhub_config.py is correct for your deployment machine setup (see Configuration).
If you are using virtual users (users that are not present on the system) you need to create a temporary space where the virtual user homes are created:
```
mkdir /tmp/remoteapp
```
You can now start the service:
```
sh start.sh
```
Visit the site at:
```
https://127.0.0.1:8000
```

Setup docker containers¶

Compatible docker containers can be found in DockerHub. Refer to the documentation of simphony-remote-docker repository to deploy the images.

Setup Database Accounting¶

A database is needed for managing the remote applications available for each user. Note that this database is in addition to the database created or used by JupyterHub.

Various accounting sources are supported:

Default sqlite database

remoteappmanager by default uses a sqlite database remoteappmanager.db in the current work directory. The remoteappdb command-line tool is provided for setting up the database. Please refer to the Utilities section for details on the use of this program.
Other DBAPI implementations and databases

For database implementation supported by SQLAlchemy, you may configure remoteappmanager to use remoteappmanager.db.orm.AppAccounting. Please also refer to Configure the remoteappmanager for details on setting up the accounting class.

Note

The use of databases other than sqlite3 is not tested
CSV file

You may configurate remoteappmanager to use a CSV file as its database. Please refer to Configure the remoteappmanager for details on setting up the accounting class to use remoteappmanager.db.csv_db.CSVAccounting.
Others

Any arbitrary database implementation may be used as long as an accounting class compliant to the API of remoteappmanager.db.interfaces.ABCAccounting is provided. Please also refer to Configure the remoteappmanager for details on setting up the accounting class.

Start JupyterHub¶

Start jupyterhub by invoking the start script:
```
sh start.sh
```
Note

If you want to keep the application running, use screen to start a detachable terminal.
Note

Running on OSX or with a separate docker machine requires that the appropriate environment variables are set before starting jupyterhub. refer to the command docker-machine env to setup the appropriate environment. In general, invoking:
```
eval `docker-machine env`
```
will enable the appropriate environment. On Linux, by default the host machine and the docker machine coincide, so this step is not needed.
JupyterHub is now running at https://localhost:8000

Configuration¶

Configure the spawner¶

The jupyterhub configuration is documented in the jupyterhub documentation. The important difference is the spawner to use, which is configured as:

c.JupyterHub.spawner_class = 'remoteappmanager.jupyterhub.spawners.SystemUserSpawner'
# or
# c.JupyterHub.spawner_class = 'remoteappmanager.jupyterhub.spawners.VirtualUserSpawner'

in the jupyterhub_config.py file.

Please refer to remoteappmanager.jupyterhub.spawners for the available spawners in this project.

Configure the authenticator and the admin user¶

Different authenticators can be plugged into jupyterhub. In the configuration file, the following entry will change the authenticator:

c.JupyterHub.authenticator_class = ('remoteappmanager.jupyterhub.auth.WorldAuthenticator')

WorldAuthenticator will allow any user to pass authentication. Use this authenticator only for testing purposes.

Administration capabilities are decided by jupyterhub, not remoteappmanager. jupyterhub_config.py allows to setup admin users with the following entry:

c.Authenticator.admin_users = {"admin"}

Note that the entry must be a python set. Users in this set will, once logged in, reach an administrative interface, instead of the docker application management.

Configure the remoteappmanager¶

Configuration of the remote application is performed from two sources.

the command line, specified by the Spawner.
a config file. The location of this file is specified as part of the command line options.

Their options are fully disjoint, and they configure different aspects of the application: Command line options are dynamically decided according to the user that requests the spawn; Config file options are general in nature, and allow the remoteappmanager to perform adequately against the current docker setup.

Command line options

--base_urlpath                   The base url where the server resides
--config_file                    The path of the configuration file
--cookie_name                    The cookie name for authentication
--hub_api_url                    The url of the jupyterhub REST API
--hub_host                       The url of the jupyterhub server
--hub_prefix                     The url prefix of the jupyterhub
--ip                             The IP address to bind
--port                           Port at which to spawn
--proxy_api_url                  The url of the reverse proxy API
--user                           The user as specified at the jupyterhub login

When remoteappmanager is started from jupyterhub using the spawner, all the command line options are filled in automatically.

Config file

The remoteappmanager has a number of parameters configurable via a config file. The path of the config file should be specified in the spawner in jupyterhub_config.py:
```
c.SystemUserSpawner.config_file_path = "/path/to/config.py"
```
Please refer to remoteappmanager.file_config.FileConfig for the configurable parameters. Note that this config file will be used by all remoteappmanagers for any user.

For example, to use CSV as the database, /path/to/config.py would contain the followings:
```
database_class = 'remoteappmanager.db.csv_db.CSVDatabase'
database_kwargs = {'url': '/path/to/csv_file'}
```

Administration¶

As specified in the deployment section, the authenticator will grant administrative rights to users in the specified set. Once logged in, an administrative user will be served by a different application, where it can add or remove users, applications, and authorize users to run specific applications. It is also possible to stop currently running containers.

It is important to note that the administrative interface works only with accounting backends supporting addition and removal. More specifically, it does not support the CSV backend. Read operations are supported, but write operations will be denied.

Utilities¶

Simphony remote comes with two utility scripts:

remoteappdb: Allows to add new applications, create new users, and specify permissions between users and applications in a database from the command line. It is targeted at system administrators.

remoteapprest: Allows to start, stop, inquire running containers from the command line.

Remoteappdb¶

Note: As of version 0.9.0 the management functionality of this utility is also covered by the administrative web interface.

The script is aimed at system administrators using the database (by default, a sqlite database) to perform accounting of users and applications.

The database must be first initialized with the init command:

remoteappdb ~/remoteappmanager.db init

Once initialized, the database content is ready to be configured. New applications are registered with app create. The image name must match the image name in docker:

remoteappdb ~/remoteappmanager.db app create myimage

The option –verify can be used to validate the image name against docker.

You can also create users with the user create command:

remoteappdb ~/remoteappmanager.db user create myuser

An application will not be visible not can be started by a user until permission is granted. To grant permission, use the app grant command:

remoteappdb ~/remoteappmanager.db app create myimage myuser

By default, this command will grant no special options. It is however possible to specify a different running policy, like for example mounting a common home directory, with the following options:

--allow-home   Enable mounting of home directory
--allow-view   Enable third-party visibility of the running container.
--volume TEXT  Application data volume, format=SOURCE:TARGET:MODE, where
               mode is 'ro' or 'rw'.

Note that you can grant access to the same application with multiple, different policies. Each application and policy will appear as a separate option in the user choice of runnable applications.

The script provides additional functionality to inquire the current state of the database, such as listing the current users, applications, revoke permissions, remove applications and so on.

Remoteapprest¶

This script is experimental and exploits the REST API provided by the server to allow inquiring, starting, and stopping containers from the command line.

Before using the CLI, you need to authenticate against the jupyterhub server with the login command:

remoteapprest login http://jupyterhubserver.example/

You will be inquired about username and password. Once sucessfully logged in, your credentials will be stored in a file .remoteapprest in your home directory. Note that your password will not be saved, only an authentication token.

Once logged in, you can inquire about the available applications by issuing:

remoteapprest app available

Note that you don’t need to specify the endpoint. This command will show you a list of the available applications, preceded by a unique identifier:

6dbe8e166c94b0b4b36a2d961586acc0: myapplication

This identifier can be used to start a new container, using the following command:

remoteapprest app start 6dbe8e166c94b0b4b36a2d961586acc0

The application will run, and can be seen with:

remoteapprest app running

83c18fcd833595a571d556a5e6c253f8: myapplication

Which will show a different identifier for this running instance. Finally, the application can be stopped using the stop command:

remoteapprest app stop 83c18fcd833595a571d556a5e6c253f8

Design¶

Simphony remote is derived from the Jupyterhub design with a custom single user application that manages the available docker images for each user (Fig 1).

Components¶

Figure. 1: Component diagram a basic remote app server based on the Jupyter hub infrastructure.

Reverse proxy: Proxy: the public facing part of the server that uses a dynamic proxy to route HTTP requests to the Hub and Single User Servers.
Jupyter Hub: manages user accounts and authentication and coordinates Single Users Servers using a Spawner.
Single user manager: A web server to manage the images and the active sessions. There is one such server for each authenticated user.
Docker: The docker engine managing the docker containers.

Note

Docker containers are connected via HTTP. HTTPS encryption is only provided by the proxy.

Usecases¶

Figure 2: Basic usecases

A Scientist should be able to:

Login
Inspect the available docker images
Start a new session
Stop a running session
Share a session with another user

Furthermore an Administrator should be able to:

Build compatible docker images
Upload docker images.

The above design as of version 0.1 supports the usecases (see example in Fig 3):

Figure 3: Sequence diagram of an authorised user starting a remote session.

Developer documentation¶

RemoteAppManager¶

The main tornado web application that manages the containers (docker applications) for each user.

Docker image specifications¶

Docker images compliant to the simphony-remote application define a protocol through docker LABEL and environment variables.

Labels¶

Labels are defined with the prefix namespace:

eu.simphony-project.docker

The following labels are currently defined. Their definition can be found in remoteappmanager.docker.docker_labels

ui_name: the UI visible name of the image

icon_128: a base64 encoded png image that will result as an icon

description: a user-readable description of the image

type: a string identifying the type of the container, depending on the original base image (vncapp or webapp)

env: subnamespace for accepted environment variables. See below.

The env is a subnamespace defining the environment variables the image internals can understand. This does not mean that they are the only ones that will be passed to the image.

The naming strategy works around the docker label restrictions of having kebab case vs envvars that are traditionally MACRO_CASE. Additionally, it allows new variables to be added by layers without having to know the variables understood by the base layer.

The strategy is as follows: the name after the env will be converted to uppercase and dashes converted to underscores. For example:

env.x11-width -> container accepts and understands envvar X11_WIDTH

the value of the label is currently unused, and should be left empty.

If your application uses variables with a different convention, or uses double underscores, you will have to define an auxilliary variable and transfer the value in the image startup scripts.

Currently reserved env keys:

x11-width: for the VNC images, the X11 width

x11-height: for the VNC images, the X11 height

x11-depth: for the VNC images, the X11 depth (currently unused, fixed at 16)

Container Labels¶

When a container is started, the following labels will be added:

url_id: unique identifier that ends up in the URL when the user is redirected

mapping_id: a unique key identifying the combination of image and policy used to start the container.

user: the user that started the container

Environment variables¶

The following environment variables are passed at container startup:

JPY_USER: the username used to login to the Jupyterhub frontend. Can be an email address, or anything else your authenticator accepts.

JPY_BASE_USER_URL: The base URL _path_ where the user has its service.

USER: A unix-likable username to create the container user.

URL_ID: a unique key assigned to the container that will end up in the user-exposed URL to reach the container.

If the image accepts additional envvars (through the env labels mechanism outlined above) these variables will be passed through the configurables mechanism: special variables are recognized and exposed to the user as a configurable UI, then passed to the container at startup. See the reserved env labels for details.