AMI appears to break with an additional volume

Description

Impact on the user
PMM is unusable

Steps to reproduce

  • Have an existing snapshot that you would like to have access to

  • Create a new AMI instance and add an additional volume using the extra snapshot

  • Wait for the instance to finish initialising

  • Attempt to access PMM via the UI

Actual result
A broken instance is created.

Expected Result
The extra volume is left alone

Workaround
N/A

Details
When wanting to test an AMI upgrade where the active AMI would send alerts, or potentially cause other issues, a reasonable option is to create a new instance with the extra volume for administrator use via SSH.

$ sudo pvs PV VG Fmt Attr PSize PFree /dev/nvme1n1 DataVG lvm2 a-- <100.00g 0 /dev/nvme2n1 DataVG lvm2 a-- <100.00g 0 $ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert DataLV DataVG Vwi---tz-- 80.00g ThinPool DataLV DataVG Vwi---tz-- 80.00g ThinPool ThinPool DataVG twi---tz-- 99.80g ThinPool DataVG twi---tz-- 99.80g $ sudo lsblk --ascii --nodeps -o KNAME,NAME,MODEL,SIZE,ROTA,SCHED,LABEL KNAME NAME MODEL SIZE ROTA SCHED LABEL nvme0n1 nvme0n1 Amazon Elastic Block Store 8G 0 none nvme1n1 nvme1n1 Amazon Elastic Block Store 100G 0 none nvme2n1 nvme2n1 Amazon Elastic Block Store 100G 0 none $ ls -la /dev/disk/ total 0 drwxr-xr-x. 5 root root 100 Jul 29 09:24 . drwxr-xr-x. 16 root root 2760 Jul 29 09:24 .. drwxr-xr-x. 2 root root 240 Jul 29 09:24 by-id drwxr-xr-x. 2 root root 120 Jul 29 09:24 by-path drwxr-xr-x. 2 root root 60 Jul 29 09:24 by-uuid $ grep LABEL /etc/fstab LABEL=DATA /srv xfs defaults,nofail 0 0 $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 09:24 ? 00:00:04 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 root 2 0 0 09:24 ? 00:00:00 [kthreadd] root 4 2 0 09:24 ? 00:00:00 [kworker/0:0H] root 6 2 0 09:24 ? 00:00:00 [ksoftirqd/0] root 7 2 0 09:24 ? 00:00:00 [migration/0] root 8 2 0 09:24 ? 00:00:00 [rcu_bh] root 9 2 0 09:24 ? 00:00:00 [rcu_sched] root 10 2 0 09:24 ? 00:00:00 [lru-add-drain] root 11 2 0 09:24 ? 00:00:00 [watchdog/0] root 12 2 0 09:24 ? 00:00:00 [watchdog/1] root 13 2 0 09:24 ? 00:00:00 [migration/1] root 14 2 0 09:24 ? 00:00:00 [ksoftirqd/1] root 16 2 0 09:24 ? 00:00:00 [kworker/1:0H] root 18 2 0 09:24 ? 00:00:00 [kdevtmpfs] root 19 2 0 09:24 ? 00:00:00 [netns] root 20 2 0 09:24 ? 00:00:00 [khungtaskd] root 21 2 0 09:24 ? 00:00:00 [writeback] root 22 2 0 09:24 ? 00:00:00 [kintegrityd] root 23 2 0 09:24 ? 00:00:00 [bioset] root 24 2 0 09:24 ? 00:00:00 [bioset] root 25 2 0 09:24 ? 00:00:00 [bioset] root 26 2 0 09:24 ? 00:00:00 [kblockd] root 27 2 0 09:24 ? 00:00:00 [md] root 28 2 0 09:24 ? 00:00:00 [edac-poller] root 29 2 0 09:24 ? 00:00:00 [watchdogd] root 35 2 0 09:24 ? 00:00:00 [kswapd0] root 36 2 0 09:24 ? 00:00:00 [ksmd] root 37 2 0 09:24 ? 00:00:00 [khugepaged] root 38 2 0 09:24 ? 00:00:00 [crypto] root 46 2 0 09:24 ? 00:00:00 [kthrotld] root 47 2 0 09:24 ? 00:00:00 [kworker/u4:1] root 48 2 0 09:24 ? 00:00:00 [kmpath_rdacd] root 49 2 0 09:24 ? 00:00:00 [kaluad] root 50 2 0 09:24 ? 00:00:00 [kpsmoused] root 52 2 0 09:24 ? 00:00:00 [ipv6_addrconf] root 66 2 0 09:24 ? 00:00:00 [deferwq] root 120 2 0 09:24 ? 00:00:00 [kauditd] root 185 2 0 09:24 ? 00:00:00 [rpciod] root 186 2 0 09:24 ? 00:00:00 [xprtiod] root 257 2 0 09:24 ? 00:00:00 [nvme-wq] root 260 2 0 09:24 ? 00:00:00 [nvme-reset-wq] root 261 2 0 09:24 ? 00:00:00 [nvme-delete-wq] root 262 2 0 09:24 ? 00:00:00 [ena] root 269 2 0 09:24 ? 00:00:00 [kworker/u4:4] root 282 2 0 09:24 ? 00:00:00 [bioset] root 283 2 0 09:24 ? 00:00:00 [xfsalloc] root 284 2 0 09:24 ? 00:00:00 [xfs_mru_cache] root 285 2 0 09:24 ? 00:00:00 [xfs-buf/nvme0n1] root 286 2 0 09:24 ? 00:00:00 [xfs-data/nvme0n] root 287 2 0 09:24 ? 00:00:00 [xfs-conv/nvme0n] root 288 2 0 09:24 ? 00:00:00 [xfs-cil/nvme0n1] root 289 2 0 09:24 ? 00:00:00 [xfs-reclaim/nvm] root 290 2 0 09:24 ? 00:00:00 [xfs-log/nvme0n1] root 291 2 0 09:24 ? 00:00:00 [xfs-eofblocks/n] root 292 2 0 09:24 ? 00:00:02 [xfsaild/nvme0n1] root 293 2 0 09:24 ? 00:00:00 [kworker/1:1H] root 294 2 0 09:24 ? 00:00:00 [kworker/0:1H] root 407 1 0 09:24 ? 00:00:02 /usr/lib/systemd/systemd-journald root 444 1 0 09:24 ? 00:00:00 /usr/sbin/lvmetad -f root 445 1 0 09:24 ? 00:00:00 /usr/lib/systemd/systemd-udevd root 533 2 0 09:24 ? 00:00:00 [nfit] root 541 1 0 09:24 ? 00:00:00 /sbin/auditd root 568 1 0 09:24 ? 00:00:00 /usr/sbin/irqbalance --foreground dbus 570 1 0 09:24 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation rpc 574 1 0 09:24 ? 00:00:00 /sbin/rpcbind -w root 576 1 0 09:24 ? 00:00:00 /usr/lib/systemd/systemd-logind polkitd 577 1 0 09:24 ? 00:00:00 /usr/lib/polkit-1/polkitd --no-debug root 590 1 0 09:24 ? 00:00:00 /usr/sbin/gssproxy -D root 600 1 0 09:24 ? 00:00:00 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid root 940 1 0 09:25 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H ip-10-0-1-58 eth0 root 999 1 0 09:25 ? 00:00:00 /usr/bin/python2 -Es /usr/sbin/tuned -l -P root 1091 1 0 09:25 tty1 00:00:00 /sbin/agetty --noclear tty1 linux root 1098 1 0 09:25 ? 00:00:00 /usr/sbin/crond -n root 1100 1 0 09:25 ttyS0 00:00:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 chrony 1105 1 0 09:25 ? 00:00:00 /usr/sbin/chronyd root 1225 1 0 09:25 ? 00:00:00 /usr/sbin/sshd -D root 7605 2 0 09:35 ? 00:00:00 [kworker/0:0] root 7753 2 0 09:45 ? 00:00:00 [kworker/0:2] root 8023 1 0 10:01 ? 00:00:00 /usr/sbin/anacron -s root 18613 1225 0 10:41 ? 00:00:00 sshd: admin [priv] admin 18616 18613 0 10:41 ? 00:00:00 sshd: admin@pts/0 admin 18617 18616 0 10:41 pts/0 00:00:00 -bash root 18825 2 0 10:52 ? 00:00:00 [kworker/1:2] root 18936 2 0 10:57 ? 00:00:00 [kworker/1:1] admin 19141 18617 0 11:08 pts/0 00:00:00 ps -ef

How to test

None

How to document

None

Smart Checklist

Activity

Show:

C W July 29, 2022 at 11:27 AM
Edited

Additional issue:

$ sudo yum check-update Loaded plugins: fastestmirror Determining fastest mirrors Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=genclo error was 14: curl#7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable" One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true Cannot find a valid baseurl for repo: base/7/x86_64

I didn't notice any information about restrictions about how you can secure your instance, are there any? Not allowing HTTP traffic is an example of how it might be restricted

Details

Assignee

Reporter

Priority

Components

Labels

Needs QA

Yes

Needs Doc

No

Affects versions

Smart Checklist

Created July 29, 2022 at 11:08 AM
Updated March 6, 2024 at 12:53 AM

Flag notifications