Skip to main content
Version: ELN v4.x

Upgrading

Upgrading an existing instance is covered by the tool it was installed with: see Using the CLI → Upgrading an instance for a ChemCLI install, and the image tag for a manual Docker install. This page collects what is specific to a particular version step — the things that need an action beyond pulling a new image.

It is the same page in every version of this documentation, so that an administrator can read the whole path at once, including a jump across more than one version.

Before any upgrade​

  1. Back up the database, and the file volumes with it — the attachments and public/safety_sheets. Several migrations cannot be undone, and some of them change files rather than rows.
  2. Rehearse a major step on a copy of the production database. This matters most for a jump across two versions, where the migrations run in an order no single step ever produces.
  3. Mount the file storage while the migration runs. More than one migration reads or rewrites stored files, and anything it cannot reach is skipped.

With a ChemCLI or a manual Docker install, database migrations run automatically when the container starts. A source install runs bundle exec rake db:migrate as part of its own upgrade. Nothing else on this page has to be run by hand unless it says so.

Which path applies​

RunningGoing toRead
2.x3.xFrom version 2 to version 3
3.x, before 3.1.43.1.4items 12 and 13 of From version 2 to version 3
3.x4.xFrom version 3 to version 4
2.x4.x, in one stepboth sections above, then From version 2 directly to version 4
4.0.0 or 4.0.14.0.2the items in From version 3 to version 4 marked as changed in 4.0.1 or 4.0.2

This page is written for the versions ChemCLI currently offers: 3.1.4 for version 3 and 4.0.2 for version 4.

From version 2 to version 3​

Version 3 has no breaking-change list of its own; everything below comes from what the upgrade does. The first two items are the ones that stop an upgrade or lose something.

1. The ELN and worker do not start​

Symptom. Nothing comes up after the upgrade. The log shows KeyError: key not found: "OTP_SECRET_KEY". No migration runs, so the database is untouched.

Cause. From version 3.1.0 two-factor sign-in needs a server-side key, and it is read with no default. Version 3.0.0 does not need it.

Fix. Set OTP_SECRET_KEY to a unique string of at least 32 random characters, on the eln and worker containers and on anything else that runs rake tasks. Never use an example value, and never change it afterwards: a new value switches off two-factor login for every user who had enabled it, and each of them has to enable it again.

Applies to manual Docker and source installs. ChemCLI adds one during the upgrade only when the setting is missing altogether, so check the value on a ChemCLI install as well.

Symptom. A chemical's stored safety data sheet opens nothing — a 404 — after the upgrade.

Cause. A migration moves and renames every stored sheet under public/safety_sheets/<vendor>/ and rewrites the stored links. The rewritten links do not always match the new file names.

Fix. Before upgrading, back up public/safety_sheets/. The migration also copies it to public/safety_sheets_backup once, but do not rely on that alone. Afterwards, where a link fails, fetch or upload that sheet again from the chemical's inventory tab.

Applies to any instance with stored sheets.

3. Structures render differently, or worse​

Symptom. Structures from imports, and polymer or surface structures, look different or coarser than before. The log may show IndigoService Error:.

Cause. The old ketcherails renderer is gone. Server-side rendering now prefers the optional Indigo service for those cases, falling back to ketchersvc and then to OpenBabel.

Fix. Run the indigo_service container and create config/indigo_service.yml:

:indigo_service_url: 'http://indigo_service/'

The trailing slash is required. A ChemCLI installation already includes the service.

Applies to manual Docker and source installs.

4. Structure rendering falls back to OpenBabel everywhere​

Symptom. Rendering is poor across the board, not only for imports.

Cause. The example URL in config/ketcher_service.yml moved to port 4000.

Fix. Point it at the running service — http://ketchersvc:4000/render, where it used to be port 9000.

Applies to manual Docker, and source installs keeping their own file.

5. The structure editor does not load​

Symptom. The editor pane stays empty.

Cause. In config/structure_editors.yml the key ketcher2 became ketcher, and its path became /editors/ketcher/standalone/index.html.

Fix. Edit the file to match.

Applies to manual Docker, and source installs keeping their own file.

6. A third-party editor stays blank​

Symptom. ChemDraw JS or Marvin JS, loaded from another host, shows nothing. The browser console reports a Content-Security-Policy script-src-elem error.

Cause. Version 3 restricts which scripts a page may load.

Fix. Host the editor's JavaScript under the ELN's own public/, or allow its origin in the policy.

Applies to any install with additional editors.

7. The upgrade takes far longer than usual​

Symptom. A long migration, and noticeably more disk used afterwards.

Cause. A migration regenerates every image and PDF thumbnail at a larger size — 800 px, 200 dpi — and the new indexes lock large tables while they build.

Fix. Nothing to correct; plan the downtime, and keep the storage mounted so the thumbnails can be written. Failures are logged per file and skipped.

Applies to all install methods.

8. Scripts reading device_type fail​

Symptom. A local query or report errors on device_descriptions.

Cause. device_type became device_class, and device_type_detail became device_class_detail.

Fix. Update the queries.

Applies to anyone querying the database directly.

9. Four API endpoints changed​

Symptom. An integration fails on chemicals, device descriptions, OnlyOffice or editor options.

Fix. Update the client:

Version 2Version 3
PUT /chemicals/:sample_idPUT /chemicals, with sample_id in the body
PUT /device_descriptions/byontology/:idremoved — use GET /device_descriptions/ontologies
POST /editor/startGET /editor/:id/start
…/profiles/editors/ketcher2-options…/profiles/editors/ketcher-options

Applies to API clients.

10. Automatic inbox transfer arrives, switched off​

Symptom. Nothing appears to change, but a setting users have not seen before is now in their profile.

Cause. Automatic transfer of inbox files is new in version 3. Existing profiles are given the setting switched off, with manual transfer on, so the behaviour users already know is kept.

Fix. Nothing to correct; users who want it can turn it on themselves.

Applies to all install methods.

11. Rich text may lose images and formatting​

Symptom. Images, links or coloured text pasted into a description, a research plan or another rich-text field no longer appear after the upgrade. Editing and saving such a field can remove them for good.

Cause. From version 3.1.1 the rich-text editors accept a reduced set of formats. Content outside that set is not carried through.

Fix. No repair is available, so treat this as a precaution rather than a step: before upgrading, look over any rich text that matters — long descriptions and research plans especially — and keep the database backup until it has been checked. Avoid editing and saving an affected field until its content has been confirmed, since saving is what makes the loss permanent.

Applies to all install methods. This was found by reading the code rather than by observing an upgrade, so the exact set of formats affected is not certain; images are the clearest case.

12. Outgoing mail behaves differently​

Symptom. Mail now goes out where an unsupported SMTP_SSL_MODE value used to make every attempt fail, and a slow authenticating relay no longer times out. Mail that relied on a blank variable may behave differently.

Cause. Version 3.1.4 reads the mail settings more strictly. Variables left blank are dropped, so their defaults apply. SMTP_SSL_MODE accepts only none and peer, and any other value is now ignored instead of breaking delivery. When SMTP_DOMAIN is unset, the HELO name falls back to the host part of PUBLIC_URL. Two new settings, SMTP_OPEN_TIMEOUT and SMTP_READ_TIMEOUT, default to 10 and 30 seconds; both used to be 5, which an authenticating relay often exceeded.

Fix. Review the SMTP_* settings against Configuration, and send a test mail after upgrading — a password reset will do.

Applies to all install methods.

13. Deleted devices lose their abbreviation and email​

Symptom. None in the interface. A query against devices finds name_abbreviation and email empty on deleted devices.

Cause. A migration in version 3.1.4 clears both fields on every deleted device, so that a new device can reuse them. It cannot be undone.

Fix. Nothing to correct. If those values matter to a local record, export them before upgrading.

Applies to anyone querying the database directly.

Two things to check before starting​

Keep config/storage.yml. Version 3 still refuses to boot without it.

Check for pre-2021 attachments. The old attachment storage code is gone from version 3, so an instance that was ever below 1.5.0 should run this while still on version 2:

SELECT count(*) FROM attachments
WHERE attachment_data IS NULL AND deleted_at IS NULL;

The answer should be 0. If it is not, repair those attachments on version 2 first. Do not go from a version below 1.5.0 straight to version 3.

No job queue or scheduled-task variable changed between version 2 and version 3.

From version 3 to version 4​

Version 4.0.0 rebuilt collection sharing, merged the PubChem background jobs into one, and moved several API paths. Most instances need no action, but four of the changes below are silent: nothing fails visibly, and the symptom appears days later.

Coming from a 3.x release before 3.1.4, items 12 and 13 of the version 2 to version 3 section apply as well: version 4.0.2 contains both changes.

Take a database backup first

The sharing migration cannot be rolled back. Its down step restores the table structure but not the data, and several later migrations cannot be undone either. Some refuse to run backwards and say so. Others are more treacherous: rolling one back appears to succeed — the migration is marked as undone — while the change it made to the data stays, and can no longer be told apart from what was there before. A failed upgrade is recoverable only from a backup.

Version 4.0.2 adds one more irreversible change: samples shared between reactions are split, as described in item 10 below.

Run the migration on a host where the attachment storage is mounted. One migration repairs image derivatives in place, and entries it cannot reach are dropped and regenerated the next time a preview is requested.

1. The PubChem queues became one​

Symptom. Imported molecules never receive a name, a PubChem ID or safety (GHS) data, and samples made from them keep their sum formula as a name. No error appears anywhere.

Cause. The three jobs behind pubchem, pubchemLcss and single_pubchem_lcss were merged into a single job on the queue pubchem_lookup. A worker started with an explicit queue list no longer matches, so the jobs are queued and never run.

Fix. In DELAYED_JOB_ARGS, replace pubchem,pubchemLcss,single_pubchem_lcss with pubchem_lookup, or add a catch-all pool:

DELAYED_JOB_ARGS="--pool=transfer_to_repo --pool=collect_data --pool=* "

Applies to instances whose worker runs a queue allowlist. No action is needed if DELAYED_JOB_ARGS is unset, or already ends in --pool=* — which is the default for a ChemCLI install and for the documented example. A catch-all is the safer setting in any case, because some queue names are generated at run time (report_<id>, remove_files_<job_id>) and an allowlist cannot name them.

Old rows left behind by the removed jobs are deleted automatically when the application boots. They do not need cleaning out by hand, and no work is lost: the weekly sweep works out again what is still missing.

2. CRON_CONFIG_PC_LCSS is ignored​

Symptom. Safety-data lookups run again on an instance where they had been switched off.

Cause. The variable no longer exists. CRON_CONFIG_PC_CID now schedules the single combined sweep, and an instance that had disabled only the LCSS sweep is left with the combined one enabled. The application logs a warning at boot if the obsolete variable is still set.

Fix. Remove CRON_CONFIG_PC_LCSS. To stop the recurring sweep, set CRON_CONFIG_PC_CID=disabled. Note that this stops only the sweep — a lookup is still queued for each molecule as it is created.

Applies to all install methods.

3. The sharing tables were replaced​

Symptom. A local script, report or reporting query fails with relation "sync_collections_users" does not exist, or a comparable column … does not exist.

Cause. Sharing was rebuilt. The table sync_collections_users, the view v_samples_collections and thirteen sharing columns on collections were dropped, and shares now live in collection_shares.

Fix. Rewrite local queries against collection_shares — which carries collection_id, shared_with_id, permission_level and the per-type detail levels — and against collections.user_id, which now always holds the real owner.

Applies to anyone querying the database directly. The ELN itself and its own interface are unaffected.

note

Two SQL functions, collection_shared_names and detail_level_for_sample, are left behind in an upgraded database and still reference the dropped table. Nothing in the ELN calls them. They fail if something else does.

4. Collections owned by a group account become unreachable​

This is the most serious item on the page, and the only one with no fix in the interface.

Symptom. After the upgrade, members of a group can no longer add elements to, or export, a collection that belonged to the group.

Cause. Collection ownership is now personal. Members reach a group's collections only through a share addressed to the group. Shares to a group made in version 3 were converted and keep working. A collection owned by the group account itself, with no share on it, is now reachable by nobody — and because administering a share requires owning the collection or holding Manage shares on it, neither of which anyone now has, and because a group account cannot open the workspace, no screen in the ELN can repair it.

Fix. After upgrading, list the affected collections:

SELECT c.id, c.label, u.name_abbreviation
FROM collections c
JOIN users u ON u.id = c.user_id
WHERE u.type = 'Group'
AND c.deleted_at IS NULL
AND c.is_locked IS NOT TRUE
AND NOT EXISTS (SELECT 1 FROM collection_shares cs WHERE cs.collection_id = c.id);

Run it after the upgrade, not before. Before the migration it also returns the version 3 shares-to-groups, which the upgrade converts by itself.

Each row then has to be repaired from the Rails console or in SQL, either by creating a share addressed to the group, or by setting collections.user_id to a person who can then share it.

Applies to all install methods, and only to instances that used group accounts.

The migration also soft-deletes the empty, automatically created All and chemotion-repository.net collections belonging to group accounts. Any that still hold content are kept, and their identifiers are printed in the migration output on lines beginning "keeping group" — worth reading once and deciding on by hand.

5. Some API paths moved​

Symptom. An integration receives 404, or another 4xx response, on calls concerning user labels or groups.

Cause. Those routes were rebuilt and no alias was left behind.

Fix. Update the client:

Version 3Version 4
GET /api/v1/users/list_labelsGET /api/v1/user_labels/list_labels
PUT /api/v1/users/save_labelPUT /api/v1/user_labels/save_label
POST /api/v1/groups/create, body wrapped in group_paramPOST /api/v1/groups, body flat
GET /api/v1/groups/qrycurrentGET /api/v1/groups
PUT /api/v1/groups/upd/:id with destroy_groupDELETE /api/v1/groups/:id
PUT /api/v1/groups/upd/:id with add_usersPOST /api/v1/groups/:id/members, body {user_ids: […]}
PUT /api/v1/groups/upd/:id with rm_usersDELETE /api/v1/groups/:id/members/:user_id, one per call
PUT /api/v1/groups/upd/:id with add_adminPOST /api/v1/groups/:id/admins/:user_id, one per call
PUT /api/v1/groups/upd/:id with rm_adminDELETE /api/v1/groups/:id/admins/:user_id, one per call
GET /api/v1/groups/queryCurrentDevicesGET /api/v1/users/devices
GET /api/v1/groups/deviceMetadata/:device_idGET /api/v1/devices/:device_id/metadata

One response change comes with this: group validation errors now return 422 where they used to return 404. A new bulk endpoint, POST /api/v1/user_labels/bulk, has no version 3 equivalent.

Applies to external scripts and clients. The ELN's own interface is updated.

6. Long jobs are stopped after an hour​

Symptom. A very large import stops partway and fails. Exports may be tried again; an import is not.

Cause. The limit on a single background job is now 60 minutes. It was four hours.

Fix. None available without changing code. Split a very large import into smaller files.

Applies to all install methods.

7. Weighing Tasks disappears​

Symptom. The Weighing Tasks button is gone from the top bar after the upgrade, for everyone. Sample Explorer, which users may have been told to expect in version 4, is not there either.

Cause. Version 4 puts both behind switches in config/ui_components.yml, and in production both ship switched off. Weighing Tasks worked in version 3 without any switch, so upgrading takes it away until an administrator turns it back on. Sample Explorer is genuinely new, and off for the same reason.

Fix. Set them to true in config/ui_components.yml and restart:

:weighing_tasks: true
:sample_explorer: true

The file is created from its example on first boot if it is missing. Only the literal true counts.

Applies to all install methods.

8. Saved wellplate import templates stop working​

Symptom. Importing a wellplate spreadsheet that worked before the upgrade fails with 'Molarity' must be in cell E1., usually followed by further errors about the readout columns.

Cause. The template gained a Molarity column at E, so every readout column moved one place to the right. A spreadsheet saved from version 3 has the old layout.

Fix. Download a fresh template from the wellplate's Attachments tab — the wellplate has to be saved first — and transfer the data into it, or insert a column at E headed Molarity and leave it empty. Previously imported data is unaffected.

Applies to anyone who keeps their own copy of the import template.

9. Safety data sheet lookups fail behind a proxy​

Symptom. Searching for a safety data sheet returns nothing, on an instance whose outbound traffic passes through a proxy or a restrictive firewall.

Cause. Version 4 restricts the vendor lookups to a fixed list of https hosts.

Fix. Allow outbound https to sigmaaldrich.com, fishersci.com and thermofisher.com.

Applies to instances with filtered outbound access. Nothing to do where outbound traffic is unrestricted.

10. Reactions no longer share a sample (changed in 4.0.2)​

Symptom. After the upgrade, a reaction that used to share a sample with an earlier reaction lists a new split of it instead, labelled with an extra counter — ABC-12-1, ABC-12-2 — and with no analyses. The analyses are still on the original sample, in the earlier reaction.

Cause. A sample now belongs to at most one reaction. The migration keeps each shared sample with the reaction created first, and gives every other reaction a new split of it, as the reaction editor does when a sample is dropped onto a reaction. Name, external label and composition are copied, and the reaction's variations are moved to the new sample with their values; analyses are not copied. The same migration also returns a link that an earlier fault had moved from one reaction to another, and removes a sample listed twice in the same role of one reaction.

Fix. Nothing to correct, but it cannot be undone, so take the backup. The migration log reports how many links it returned, removed and split. Tell users before upgrading, so that a new split in an old reaction is not a surprise. If any split fails, the migration stops and names the links concerned.

Applies to all install methods.

11. The Variations tab fails on older reactions (changed in 4.0.1)​

Symptom. On a reaction saved before 4.0.1, the Variations tab does not open, or shows columns for materials the reaction no longer has.

Cause. Variations stored keys naming materials, and some of those keys no longer match the reaction's materials.

Fix. Run the repair task after upgrading. It reports what it would change and changes nothing:

rake reactions:repair_variation_keys

To apply the repair:

rake reactions:repair_variation_keys'[false]'

Run it inside the ELN container or with bundle exec, as described under After reaching version 4.

Applies to all install methods, and only to instances whose reactions use variations.

12. Bookmarks to the ChemSpectra pages fail (changed in 4.0.1)​

Symptom. A bookmark to /chemspectra or /chemspectra-editor returns not found.

Cause. The standalone ChemSpectra pages and their API routes are removed.

Fix. Open ChemSpectra from an analysis instead.

Applies to all install methods.

13. Sending an IconNMR configuration from the ELN is gone (changed in 4.0.1)​

Symptom. The legacy upload of an IconNMR experiment configuration from the ELN no longer exists.

Cause. The legacy device code was removed, and with it the only use of the global SFTP_HOST, SFTP_USER, SFTP_PASSWORD, SFTP_PORT, SFTP_KEYS and SFTP_UPLOAD_FOLDER variables.

Fix. None in the ELN; the variables can be removed. Device data collection is unaffected: it uses each device's own settings, on the device's Data Collector tab under Admin → Devices.

Applies to instances that sent IconNMR configurations from the ELN.

14. Analysis previews stay on an old image (changed in 4.0.1)​

Symptom. An analysis keeps showing the same preview image after new files are uploaded to it.

Cause. Before 4.0.1, saving a dataset fixed its first thumbnailed file as the analysis's preview. The default now follows new uploads, preferring a file whose name contains "combined", else the most recently updated one. The upgrade does not change previews that were fixed before.

Fix. In the analysis's image dialog, choose another image, or click Use default to return to the automatic choice.

Applies to all install methods.

From version 2 directly to version 4​

The jump works. Nothing forces a stop at version 3: version 4 carries every migration needed, nothing that only version 3 provided was removed, and the sharing rebuild handles version 2's sharing data. ChemCLI offers an installed 2.2.0 both 3.1.4 and 4.0.2.

Rehearse this one on a copy

Version 2.2.0 is a side branch of the history, so in a direct jump some version 4 migrations run before some version 3 ones — an order that neither single step ever produces. Two of those interleavings could not be verified.

Run the upgrade against a copy of the production database first. Stopping at 3.1.4 and upgrading again avoids the risk entirely, at the cost of one more upgrade.

Everything in the two sections above applies. In a single jump, do it in this order:

  1. On version 2, run the pre-2021 attachment check above. The answer must be 0.
  2. Back up the database and every file volume, public/safety_sheets included.
  3. Set OTP_SECRET_KEY on the eln, worker and executor containers, as in item 1 of the version 2 to version 3 section.
  4. Apply the configuration items — items 3 to 6 and 12 of the version 2 section, and from the version 3 section the PubChem queue change and any script reading the old sharing tables.
  5. Start version 4 with the file volumes mounted. Sixty-eight migrations run.
  6. Read the migration log for the collection-share counts, the tree-check line and the reaction-sample counts, then work through the group-collections check in item 4 of the version 3 section.
  7. Run the variation-key repair from item 11 of the version 3 section, if reactions use variations.
  8. Optionally, run the duplicate-attachment cleanup described under After reaching version 4 below.

The safety data sheet link problem — item 2 of the version 2 section — applies to the jump as well.

After reaching version 4​

One optional cleanup is available, whichever path was taken. It removes duplicate spectra attachments created before the fault that produced them was corrected:

rake attachments:cleanup_duplicate_derived_attachments

It reports what it would do and changes nothing. To apply it:

rake attachments:cleanup_duplicate_derived_attachments'[false]'

On a Docker install these run inside the ELN container, as docker exec -it <eln container> bundle exec rake …; on a source install, prefix them with bundle exec.

Upgrading with the CLI​

Upgrade with chemCLI instance upgrade. ChemCLI 0.2.32 offers every supported version newer than the running one — currently 3.1.4 and 4.0.2 — so a version 2 or 3 instance moves to 4.0.2 in one step. Choose all actions to pull the images, back up and upgrade. The first start after the upgrade migrates the database and can take up to 20 minutes on a large instance.

Run without a terminal, or with --use <compose URL>, the CLI offers no menu: it uses the given compose file, or by default the newest release's.

The steps on this page still apply — they concern the ELN itself rather than the tool that deploys it. The CLI's own worker serves every queue, so the PubChem queue change needs no action. Make sure OTP_SECRET_KEY is set to a unique value of at least 32 characters, as in item 1 of the version 2 to version 3 section. The rehearsal advice above still applies to a jump from version 2.