Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Run watchtower next to PMM Server container in AMI

Description

What should be done:

We need to run watchtower container next to PMM Server container in AMI/OVF deployments.

We need to enable podman socket support and watchtower container should have access to podman socket.

Added value:

Ability to update using UI button in AMI/OVF installations.

Suggested implementation / options:

Out of scope:

Details:

We are going to prepare our own fork of , so that one should be started next to PMM Server.

This is how I run it locally

 

  1. Watchtower and PMM Server should run on the same network or at least PMM should be able to connect watchtower

  2. Watchtower shouldn’t be available anywhere outside of docker network or localhost

  3. PMM Server container has 2 new environment variables

    2. PMM_WATCHTOWER_HOST is used to connect to watchtower container and should include schema, host and port as in example above

    3. PMM_WATCHTOWER_TOKEN should be the same as WATCHTOWER_HTTP_API_TOKEN in watchtower container

How to test

  1. Setup PMM with AMI and OVF

  2. SSH to instance

  3. Check that Watchtower is running

  4. Check that Watchtower and PMM Server is on the same network

  5. Check that PMM has correct Watchtower host set as ENV variable

  6. Check that Watchtower and PMM use the same API token

How to document

None

Activity

Show:

Ondrej Patocka July 9, 2024 at 9:23 AM

How to build from separate branch info would be handy

talha.rizwan May 20, 2024 at 11:54 AM

I've generated a new AMI: ami-011c01a7ab02527a3. This image uses Watchtower and PMM-Server containers without systemd. However, when attempting to upgrade PMM-Server via the UI, I encountered the following error in the Watchtower logs:

Here is the actual error:

The error seems related to a lock mismatch between the container and the volume during the update process.

I've also encountered errors while attempting to use a bind mount directory instead of a volume for PMM Server. Below are the details of the commands run and the resulting errors:

After creating the necessary files, I ran the command again:

After addressing the missing files, the command was executed again, but issues persisted:

I accessed the container to further investigate:

Nginx, Grafana, and PostgreSQL failed due to missing directories. Why is PMM Server expecting these directories?

talha.rizwan May 9, 2024 at 12:49 PM

I've generated a new AMI image ami-07150e831dfbd758c. Could someone kindly test it out using the pmm3-ami-staging-start pipeline?

The pmm-server.service unit file retrieves image and tag values from the ~/.config/systemd/user/pmm-server.env file.

However, there are issues with the services for pmm-server and watchtower out of the box. I'm encountering the following problems:

Moreover, aardvark-dns in rootless mode isn't behaving as expected with systemd. Unfortunately, a solution for this issue isn't readily available. To make the services functional, execute the following commands:

After completing these steps, attempt to update the pmm-server via the UI. Please note, there are several edge cases, sometimes the update succeeds, and sometimes it fails. For instance, you might encounter an error like:

In scenarios where the image tag differs from the updated version, if watchtower successfully updates the container, the service may fail. Additionally, sometimes the pmm-server service re-executes the container with the old image tag (if it's different), causing the watchtower update to fail.

I'm seeking assistance in testing and identifying other potential cases. In the interim, I'll develop a new AMI image that doesn't rely on systemd.

Nurlan Moldomurov April 30, 2024 at 9:44 AM

Hi , as I said in the call the idea is to use file in /srv as an envfile in systemd unit and use docker tag value from that file in podman command.

Nurlan Moldomurov April 30, 2024 at 6:46 AM

yes, let’s try without systemd. is there an option for podman to always restart without systemd?

Done

Details

Assignee

Reporter

Priority

Components

Labels

Needs QA

Yes

Needs Doc

No

Planned Version/s

Fix versions

Story Points

Parent

Smart Checklist

Created February 14, 2024 at 1:20 PM
Updated December 2, 2024 at 3:05 PM
Resolved August 3, 2024 at 8:13 AM