Using the CLI

ChemCLI, short for Chemotion CLI, is a tool that helps administrators manage Chemotion ELN on a machine. The goal is to make installation and maintenance of (multiple instances of) Chemotion as easy as possible.

Compatibility with Chemotion ELN

The ChemCLI tool supports the last three major releases and last three minor releases of the ELN. However, it is possible that certain versions are not available for installation based on user feedback.

Releases older than 1 year are generally not included in the CLI.

Concept for chemCLI

The commands have the following general layout:

general: cli-executable  <resource>  <command>  <flags>
         └─────┬──────┘  └───┬────┘  └───┬───┘  └──┬──┘
example:    chemCLI       instance     logs     --all

The following features have been implemented:

• ✔ Installation & Deployment: ./chemCLI > install installs a production instance that is ready to use.
• ✔ Instance life cycle commands: ./chemCLI > on|off|restart and ./chemCLI > instance > stats|ping|logs.
• ✔ Upgrade: ./chemCLI > instance > upgrade to upgrade an existing Chemotion instance.
• ✔ Backups: ./chemCLI > instance > backup to save the data associated with an instance.
• ✔ Restore: ./chemCLI > instance > restore to restore the data associated with an instance into a new instance.
• ✔ Multiple instances: ./chemCLI > instance > new|list|switch|remove can be used to manage multiple instances.
• ✔ Administrative consoles: ./chemCLI > instance > consoles > <console_name> to drop into shell, postgreSQL and rails consoles.
• ✔ Users: ./chemCLI > instance > users > create|list|update|describe|delete to manage users in an instance. Particularly, ./chemCLI > instance > users > update can be used to modify the password for users who forget theirs.

Download

Getting the tool

The ChemCLI tool is a binary file called chemCLI and needs no installation. The only prerequisite is the installation of Docker Engine on Linux. In the rare scenario that the server is running Windows or Mac, the administrator can make use of Docker Desktop, which comes with a GUI. Builds for the following systems are available:

Download Links

Depending on the OS, go to GitHub and download the latest release of the CLI.

• Linux, amd64
• Windows, amd64
• macOS, apple-silicon
• macOS, amd64

Please be sure that both the docker and docker compose commands are available. This should be the case if Docker Desktop is installed. If only Docker Engine is installed, then please make sure that docker compose is also available as a command (as opposed to docker-compose). On Linux, it might be necessary to install the docker-compose-plugin to achieve this.

These binary builds should not rely on libraries of the underlying operating system: if they still do not work on a system for some reason, please create an issue here and the Chemotion team will try to provide a binary build as soon as possible. Alternatively, users can always compile the go source code on their own.

Making it an executable

OS	How to make it an executable	How to run the executable
Linux (Ubuntu, WSL etc.)	chmod u+x chemCLI	./chemCLI
Windows (Powershell)*	(nothing to do)	.\chemCLI.exe
macOS (intel/amd64)^	chmod u+x chemCLI.amd.osx	./chemCLI.amd.osx
macOS (apple-silicon)^	chmod u+x chemCLI.arm.osx	./chemCLI.arm.osx

*On Windows, it is recommended to use Powershell 7 instead of the one provided natively (confusingly called Windows Powershell). In any case, it is necessary to have pwsh in the $PATH.

^On macOS, if there is a security pop-up when running the command, please also Allow the executable in System Preferences > Security & Privacy.

Usage: Setting up an instance

Please Note

• All commands here, and all the documentation of the tool, use ./chemCLI when describing how to run the executable. However, the specific command to run the executable is given in the table above.
• If possible, do not rename the executable, or rename/remove files and folders created by it. All reasonable operations can be done using chemCLI; manual operations might break chemCLI's ability to understand how things are laid out on the machine.
• Everything happens in the folder (and subfolders) where ./chemCLI is executed. All files and folders are expected to be there; otherwise failures can happen. The user executing ChemCLI is expected to have all file permissions for this folder.
• If docker-compose.yml needs to be modified for some reason, choose to modify docker-compose.cli.yml instead. The docker-compose.cli.yml is (in most cases) not modified during upgrades. On the other hand, docker-compose.yml is replaced during upgrades.

Make a dedicated folder

Make a folder to store the installation(s) of Chemotion ELN. Ideally this folder should be on the largest drive (in terms of free space) of the system. Remember that Chemotion also uses space via Docker (Docker containers, volumes etc.); therefore, make sure that the system partition has abundant free space.

Install your first instance

To begin the installation, run the executable (./chemCLI) and follow the prompt. The first installation can take a really long time (15-30 minutes depending on download and processor speeds). Please be aware that instance names must be lowercase and cannot contain periods (.).

This will create the first (production-grade) instance of Chemotion on the system. Generally, this suffices for using Chemotion in a single scientific group/lab. By default

• this first instance will be available on port 4000
• this first instance will be the selected instance.

⚠️ chem_cli.yml: Installation also creates a file called chem_cli.yml. This file is critical as it contains information regarding existing installations. Removing the file will render chemCLI clueless about existing installations and it will behave as if Chemotion was never installed. Please do not remove the file. Ideally there should be no need to modify it manually.

Set up a Reverse-Proxy

The Chemotion team strongly recommends getting in touch with the IT services for this part of the installation. Getting it wrong can have security implications for the data/network.

The ELN runs inside a container and is therefore not available on the computer's network; other than on one exposed port. To make the installation available on the network, the container's port should then be linked to an external network. The Chemotion team suggests setting up a reverse-proxy service. Such a service forwards a domain name (e.g. https://chemotion.dept.uni.de) to the ELN so that users can visit the domain name to use the ELN. A rough guideline on how to do so is provided here.

The Chemotion team suggests running an nginx reverse-proxy server if one is not already in place. To use Shibboleth, please use Apache. Administrators familiar with the process can also choose to run something similar, like Traefik.

The simplest way to set up nginx is as follows:

1. Include nginx as a service (in the form of a container) in the docker-compose.cli.yml file used to run Chemotion ELN. The service's description should look as follows:

# This block is to be included in the `services` block.
nginx:
  image: nginx
  restart: always
  volumes:
    - ./nginx.conf:/etc/nginx/conf.d/nginx.conf:ro
    - ./ssl/:/ssl/:ro
  ports:
    - "443:443"
  networks:
    - proxy

Here a new network called proxy is defined, which is used to bridge the ELN's network and the host's network. To use it successfully, please also include the following at the end of the file:

networks:
  proxy:
    external: true

This declares the proxy network as external i.e. it is allowed to talk to the host's network.

To link this proxy network with the ELN's network, include it as a key-value pair for the eln service i.e. in the docker-compose.cli.yml file, under the service called eln, do the following:

services:
  ... # other entries
  eln:
    ... # other entries
    labels: # labels added by chemCLI
      - net.chemotion.cli.project=...
    networks: # <--- this block is what needs to be included
      - proxy

Additionally, create such a network explicitly by running the following command in a shell:

docker network create proxy

2. Then, create the nginx.conf file in the same folder as the docker-compose file. The contents of the file should look as follows:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name chemotion.dept.uni.de;

    ssl_certificate /ssl/chemotion.dept.uni.de.crt;
    ssl_certificate_key /ssl/chemotion.dept.uni.de.key;

    resolver 127.0.0.11; # this bit is very important as it allows nginx and docker to cooperate

    add_header Content-Security-Policy "default-src 'self' 'unsafe-inline' 'unsafe-eval' ; worker-src 'self' blob: ; connect-src 'self' ws: blob: commonchemistry.cas.org dx.doi.org doi.org api.crossref.org; font-src 'self' data: ; frame-src 'self' nmrium.nmrxiv.org ; img-src 'self' data: ; upgrade-insecure-requests;" ;

    location / {
        proxy_pass http://eln:4000;
    }
}

Here, the configuration accesses the files ....crt and ....key. These files are certificates that allow the connection to use encryption (i.e. use https). These certificate files should be provided by the IT department, or a service like Let's Encrypt can be used to generate them as long as the IT and firewall policies allow that. Please put these files in a folder called ssl and put this folder in the folder containing the docker-compose.yml (and `docker-compose.cli.yml) file. The add_header block allows access to external services e.g. it allows commonchemistry.cas.org (for creating samples by cas-rn), as well as dx.doi.org, doi.org, and api.crossref.org (to fetch literature references by doi).

Further information on how to configure nginx, please refer to tutorials online (e.g. here and here).

To summarize, make sure that:

• The reverse proxy is set up properly (i.e. DNS resolves the URL to the server's IP address).
• Any potential firewall/networking rules do not get in the way.
• Docker exposes and listens on the correct ports.
• The ELN is running.

Usage: Managing Instance(s)

The selected instance

Once multiple instances of Chemotion are installed, the actions of chemCLI pertain to only one of them i.e. only one is actively operated when a command in the ./chemCLI > instance section is run. This instance is referred to as the selected instance and its name is stored in a local file (chem_cli.yml). Use ./chemCLI instance switch to switch to another instance when more than one instance exists.

Users can also select an instance temporarily by giving its name to the CLI as a flag e.g. ./chemCLI instance status -i my-other-instance.

Start and Stop Chemotion

To turn on, and off, the selected instance, issue the commands:

• ./chemCLI on, and
• ./chemCLI off.

Backup an instance

WARNINGS

This backup process does not backup data that resides outside Docker volumes e.g. services and shared folders. These folders need to be backed up separately.

By default, backups are created in a folder that will be removed if the instance is removed or if Chemotion is uninstalled (using ChemCLI). Remember to move the backups before running remove/uninstall operations.

An instance of the ELN installed using the CLI can be backed up by running the following command: ./chemCLI instance backup -i <name_of_instance> -q. This command must be run individually for every instance of Chemotion ELN present. Two new backup files are created for each execution of the command inside the instances/<name_of_instance-xxxxxxx>/shared/backup folder. (These two files can be used to restore data into a new instance using the ./chemCLI instance restore command.)

If running this as a cron job, remember to change into the folder where the ChemCLI executable exists. To do this, include cd path/to/the/folder in the job script. Therefore an example crontab file that runs at 03:00 on every day-of-week from Tuesday through Saturday would look as follows:

0 3 * * 2-6 cd /home/admin/installations/chemotion_ELN && ./chemCLI instance backup -i prodinstance -q

Please also refer to the notes on manual installation for a better understanding of the backup process.

Upgrading an instance (for ELN versions 1.3 and above)

As long as an instance of Chemotion was installed using this tool, the upgrade process is quite straightforward:

• First make sure that the latest version of this tool is available.
• Prepare for the update by running ./chemCLI > instance > upgrade > pull image only. This will download the latest Chemotion image from the internet if not already present on the system. Doing this saves time later (during scheduled downtime).
• Schedule a downtime of at least 15 minutes; more if there is a lot of data that needs to be backed up. During the downtime, run ./chemCLI > instance > all actions to backup the data followed by an upgrade of the instance.

When upgrading from ELN version 1.3, please create a backup using chemCLI version 0.2.2 or above. This is because the data backup script provided inside the container is broken and this is fixed by the chemCLI tool.

Uninstallation

⚠️ be sure about the intended action!

Everything created by chemCLI can be uninstalled by running: ./chemCLI > advanced > uninstall. Finally, the downloaded binary itself can simply be deleted. To reiterate: By default, backups are created in a folder that will be removed if the instance is removed or if Chemotion is uninstalled (using ChemCLI). Remember to move the backups before running remove/uninstall operations.

Updating the Tool

chemCLI (version 0.2 onwards) is configured to check for new releases of itself every 24 hours with the servers of GitHub. If a new version is available, it will provide a notification. The tool can then be updated by going to ./chemCLI > advanced > update - chemCLI.

Respecting privacy: This automatic checking can be disabled for the next 100 years by running ./chemCLI advanced update --disable-autocheck.

Updating from version 0.1.x

For chemCLI version 0.1 (then called chemotion CLI), please do the following to update to the latest version:

• Download the latest version and make it an executable.
• Create copy of the file chemotion-cli.yml as chem_cli.yml by executing: cp chemotion-cli.yml chem_cli.yml.
• Run the new executable (./chemCLI). It should guide you through an automated update. After a successful update, it is safe to remove chemotion-cli.yml and chemotion-cli.log files.
• This update does not upgrade the instances.

Please note that support for version 0.1.x has been completely deprecated on 31.12.2023.

Advanced Usage

Using flags

chemCLI has a lot to offer, with a few advanced features available exclusively via flags. Use the --help option at the end of the command and its subcommands to explore more.

A particular construct worth noting is using the ./chemCLI instance restore command to create the first instance while restoring data from a previous instance into it. This can be useful for moving from non-docker and/or non-chemCLI based installations to a chemCLI managed installation.

For example:

./chemCLI instance restore --name first-instance --data /absolute/path/to/backup.data.tar.gz --db /absolute/path/to/backup.sql.gz --suffix 4cfcfd0c --address https://chemotion.myuni.de --use https://github.com/Chemotion/ChemCLI/releases/latest/download/docker-compose.yml

Silent and Debug Use

Almost all features of chemCLI can be used in silent mode i.e. without any input/interaction from user as long as all required pieces of information have been provided using flags. In silent mode, most of the output from the CLI (but not that of docker) is logged only in the log file, and not put on screen.

To use chemCLI in silent mode, add the flag -q/--quiet to your command. The CLI will then use default values and other flags to try and accomplish the action. Examples:

./chemCLI -q instance new --name second-instance --address https://myuni.de:3000 --use ../my/path/docker-compose.yml

./chemCLI -q instance backup # for running backups silently

Similarly, the CLI can be run in Debug mode when an error is encountered. This produces a very detailed log file containing a trace of the actions undertaken. Reporting the error and sending the log file to the Chemotion team helps a lot with troubleshooting. Please audit the debug file for any personal information (such as username etc.) before sending it.

Known limitations and bugs

1. The CLI must be updated manually for versions 0.2.11 and lower. To do so, go to Releases and download the required executable. See Updating the Tool for more.

   Manually updating the tool will trigger a warning about version mismatch whenever ChemCLI is run. To get around it, run ./chemCLI advanced update --force once after putting the new executable in place.

2. Does a bug like the following appear when trying to update an instance?

   pg_dump: error: connection to server at "db" (172.21.0.2), port 5432 failed: FATAL:  database "chemotion" does not exist

   Then please have a look at the docker-compose.cli.yml files in the instances folder. The section services.executor.image should be changed so that it matches services.eln.image in the docker-compose.yml file.

3. Please note that support for version 0.1.x of the CLI has been completely deprecated on 31.12.2023. The code that helps users migrate from version 0.1 to 0.2 will be removed in the next major release of the CLI.

:::