The trains-server is the backend service infrastructure for TRAINS. It allows multiple users to collaborate and manage their experiments. By default, TRAINS is set up to work with the TRAINS demo server, which is open to anyone and resets periodically. In order to host your own server, you will need to install trains-server and point TRAINS to it.

trains-server contains the following components:

• The TRAINS Web-App, a single-page UI for experiment management and browsing
• RESTful API for:
  • Documenting and logging experiment information, statistics and results
  • Querying experiments history, logs and results
• Locally-hosted file server for storing images and models making them easily accessible using the Web-App

You can quickly setup your trains-server using a pre-built Docker image (see Installation).

When new releases are available, you can upgrade your pre-built Docker image (see Upgrade).

The trains-server's code is freely available here.

Use our pre-installed Amazon Machine Image for easy deployment in AWS.

Details and instructions can be found here.

This section contains the instructions to setup and launch a pre-built Docker image for the trains-server. This is the quickest way to get started with your own server. Alternatively, you can build the entire trains-server architecture using the code available in our repositories.

Please Note:

• This Docker image was tested with Linux, only. For Windows users, we recommend running the server on a Linux virtual machine.

• All command-line instructions below assume you're using bash.

Make sure you are logged in as a user with sudo privileges.

In order to run the pre-packaged trains-server, install Docker.

• See Supported platforms in the Docker documentation for instructions

• For example, to install in Ubuntu / Mint (x86_64/amd64):

  sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
  . /etc/os-release
  sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $UBUNTU_CODENAME stable"
  sudo apt-get update
  sudo apt-get install -y docker-ce

To run the ElasticSearch Docker container, setup the Docker daemon by modifying the default values required by Elastic in your Docker configuration file (see Notes for production use and defaults). We provide instructions for the most common Docker configuration files.

Edit or create the Docker configuration file:

• If your system contains a /etc/sysconfig/docker Docker configuration file, edit it.

  Add the options in quotes to the available arguments in the OPTIONS section:

  OPTIONS="--default-ulimit nofile=1024:65536 --default-ulimit memlock=-1:-1"

• Otherwise, edit /etc/docker/daemon.json (if it exists) or create it (if it does not exist).

  Add or modify the defaults-ulimits section as shown below. Be sure the defaults-ulimits section contains the nofile and memlock sub-sections and values shown.

  Note: Your configuration file may contain other sections. If so, confirm that the sections are separated by commas (valid JSON format). For more information about Docker configuration files, see Daemon configuration file in the Docker documentation.

  The trains-server required defaults values are:

  {
      "default-ulimits": {
          "nofile": {
              "name": "nofile",
              "hard": 65536,
              "soft": 1024
          },
          "memlock":
          {
              "name": "memlock",
              "soft": -1,
              "hard": -1
          }
      }
  }

After modifying the configuration file, restart the Docker daemon:

sudo service docker stop
sudo service docker start

The maximum number of memory map areas a process can use is defined using the vm.max_map_count kernel setting.

Elastic requires that vm.max_map_count is at least 262144 (see Production mode).

• For CentOS 7, Ubuntu 16.04, Mint 18.3, Ubuntu 18.04 and Mint 19 users, we tested the following commands to set vm.max_map_count:

  sudo echo "vm.max_map_count=262144" > /tmp/99-trains.conf
  sudo mv /tmp/99-trains.conf /etc/sysctl.d/99-trains.conf
  sudo sysctl -w vm.max_map_count=262144

• For information about setting this parameter on other systems, see the elastic documentation.

Choose a directory on your system in which all data maintained by the trains-server is stored. Create this directory, and set its owner and group to uid 1000. The data stored in this directory will include the database, uploaded files and logs.

For example, if your data directory is /opt/trains, then use the following command:

sudo mkdir -p /opt/trains/data/elastic && sudo chown -R 1000:1000 /opt/trains

Note:

• If your data directory is not /opt/trains, please find and replace /opt/trains in the following commands with your data directory path

• Make sure ports 8008, 8080 and 8081 are not in use before starting the docker containers, as the containers will fail to initialize if these ports are already taken. If the following commands shows no output, the ports are available:

  sudo netstat -tplna | egrep "8008|8080|8081"

To launch the Docker containers, use the following commands:

sudo docker run -d --restart="always" --name="trains-elastic" -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -e "bootstrap.memory_lock=true" -e "cluster.name=trains" -e "discovery.zen.minimum_master_nodes=1" -e "node.name=trains" -e "script.inline=true" -e "script.update=true" -e "thread_pool.bulk.queue_size=2000" -e "thread_pool.search.queue_size=10000" -e "xpack.security.enabled=false" -e "xpack.monitoring.enabled=false" -e "cluster.routing.allocation.node_initial_primaries_recoveries=500" -e "node.ingest=true" -e "http.compression_level=7" -e "reindex.remote.whitelist=*.*" -e "script.painless.regex.enabled=true" --network="host" -v /opt/trains/data/elastic:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.6.16

sudo docker run -d --restart="always" --name="trains-mongo" -v /opt/trains/data/mongo/db:/data/db -v /opt/trains/data/mongo/configdb:/data/configdb --network="host" mongo:3.6.5

sudo docker run -d --restart="always" --name="trains-fileserver" --network="host" -v /opt/trains/logs:/var/log/trains -v /opt/trains/data/fileserver:/mnt/fileserver allegroai/trains:latest fileserver

sudo docker run -d --restart="always" --name="trains-apiserver" --network="host" -v /opt/trains/logs:/var/log/trains allegroai/trains:latest apiserver

sudo docker run -d --restart="always" --name="trains-webserver" --network="host" -v /opt/trains/logs:/var/log/trains allegroai/trains:latest webserver

After the trains-server Dockers are up, the following are available:

• API server on port 8008
• Web server on port 8080
• File server on port 8081

Once you have installed the trains-server, make sure to configure trains to use your locally installed server (and not the demo server).

If you have already installed trains, run the trains-init command for an interactive setup or edit your trains.conf file and make sure the api.host value is configured as follows:

api {
    host: "http://localhost:8008"
}

See Installing and Configuring TRAINS for more details.

Now that the trains-server is installed, and TRAINS is configured to use it, you can use TRAINS in your experiments and view them in the web server, for example http://localhost:8080

We are constantly updating, improving and adding to the trains-server. New releases will include new pre-built Docker images. When we release a new version and include a new pre-built Docker image for it, upgrade as follows:

1. Shut down and remove each of your Docker instances using the following commands:

    sudo docker stop <docker-name>
    sudo docker rm -v <docker-name>
   

   The Docker names are (see Launching Docker Containers):

   • trains-elastic
   • trains-mongo
   • trains-fileserver
   • trains-apiserver
   • trains-webserver

2. We highly recommend backing up your data directory!. A simple way to do that is using tar:

   For example, if your data directory is /opt/trains, use the following command:

    sudo tar czvf ~/trains_backup.tgz /opt/trains/data
   

   This back ups all data to an archive in your home directory.

   To restore this example backup, use the following command:

    sudo rm -R /opt/trains/data
    sudo tar -xzf ~/trains_backup.tgz -C /opt/trains/data
   

3. Launch the newly released Docker image (see Launching Docker Containers).

Server Side Public License v1.0

trains-server relies on both MongoDB and ElasticSearch. With the recent changes in both MongoDB's and ElasticSearch's OSS license, we feel it is our responsibility as a member of the community to support the projects we love and cherish. We believe the cause for the license change in both cases is more than just, and chose SSPL because it is the more general and flexible of the two licenses.

This is our way to say - we support you guys!