AMI appears to break with an additional volume
General
Escalation
General
Escalation
Description
How to test
None
How to document
None
Smart Checklist
Activity
Show:
C W July 29, 2022 at 11:27 AMEdited
Additional issue:
$ sudo yum check-update
Loaded plugins: fastestmirror
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=genclo error was
14: curl#7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"
One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot find a valid baseurl for repo: base/7/x86_64@Nikita Beletskii I didn't notice any information about restrictions about how you can secure your instance, are there any? Not allowing HTTP traffic is an example of how it might be restricted
Impact on the user
PMM is unusable
Steps to reproduce
Have an existing snapshot that you would like to have access to
Create a new AMI instance and add an additional volume using the extra snapshot
Wait for the instance to finish initialising
Attempt to access PMM via the UI
Actual result
A broken instance is created.
Expected Result
The extra volume is left alone
Workaround
N/A
Details
When wanting to test an AMI upgrade where the active AMI would send alerts, or potentially cause other issues, a reasonable option is to create a new instance with the extra volume for administrator use via SSH.
$ sudo pvs PV VG Fmt Attr PSize PFree /dev/nvme1n1 DataVG lvm2 a-- <100.00g 0 /dev/nvme2n1 DataVG lvm2 a-- <100.00g 0 $ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert DataLV DataVG Vwi---tz-- 80.00g ThinPool DataLV DataVG Vwi---tz-- 80.00g ThinPool ThinPool DataVG twi---tz-- 99.80g ThinPool DataVG twi---tz-- 99.80g $ sudo lsblk --ascii --nodeps -o KNAME,NAME,MODEL,SIZE,ROTA,SCHED,LABEL KNAME NAME MODEL SIZE ROTA SCHED LABEL nvme0n1 nvme0n1 Amazon Elastic Block Store 8G 0 none nvme1n1 nvme1n1 Amazon Elastic Block Store 100G 0 none nvme2n1 nvme2n1 Amazon Elastic Block Store 100G 0 none $ ls -la /dev/disk/ total 0 drwxr-xr-x. 5 root root 100 Jul 29 09:24 . drwxr-xr-x. 16 root root 2760 Jul 29 09:24 .. drwxr-xr-x. 2 root root 240 Jul 29 09:24 by-id drwxr-xr-x. 2 root root 120 Jul 29 09:24 by-path drwxr-xr-x. 2 root root 60 Jul 29 09:24 by-uuid $ grep LABEL /etc/fstab LABEL=DATA /srv xfs defaults,nofail 0 0 $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 09:24 ? 00:00:04 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 root 2 0 0 09:24 ? 00:00:00 [kthreadd] root 4 2 0 09:24 ? 00:00:00 [kworker/0:0H] root 6 2 0 09:24 ? 00:00:00 [ksoftirqd/0] root 7 2 0 09:24 ? 00:00:00 [migration/0] root 8 2 0 09:24 ? 00:00:00 [rcu_bh] root 9 2 0 09:24 ? 00:00:00 [rcu_sched] root 10 2 0 09:24 ? 00:00:00 [lru-add-drain] root 11 2 0 09:24 ? 00:00:00 [watchdog/0] root 12 2 0 09:24 ? 00:00:00 [watchdog/1] root 13 2 0 09:24 ? 00:00:00 [migration/1] root 14 2 0 09:24 ? 00:00:00 [ksoftirqd/1] root 16 2 0 09:24 ? 00:00:00 [kworker/1:0H] root 18 2 0 09:24 ? 00:00:00 [kdevtmpfs] root 19 2 0 09:24 ? 00:00:00 [netns] root 20 2 0 09:24 ? 00:00:00 [khungtaskd] root 21 2 0 09:24 ? 00:00:00 [writeback] root 22 2 0 09:24 ? 00:00:00 [kintegrityd] root 23 2 0 09:24 ? 00:00:00 [bioset] root 24 2 0 09:24 ? 00:00:00 [bioset] root 25 2 0 09:24 ? 00:00:00 [bioset] root 26 2 0 09:24 ? 00:00:00 [kblockd] root 27 2 0 09:24 ? 00:00:00 [md] root 28 2 0 09:24 ? 00:00:00 [edac-poller] root 29 2 0 09:24 ? 00:00:00 [watchdogd] root 35 2 0 09:24 ? 00:00:00 [kswapd0] root 36 2 0 09:24 ? 00:00:00 [ksmd] root 37 2 0 09:24 ? 00:00:00 [khugepaged] root 38 2 0 09:24 ? 00:00:00 [crypto] root 46 2 0 09:24 ? 00:00:00 [kthrotld] root 47 2 0 09:24 ? 00:00:00 [kworker/u4:1] root 48 2 0 09:24 ? 00:00:00 [kmpath_rdacd] root 49 2 0 09:24 ? 00:00:00 [kaluad] root 50 2 0 09:24 ? 00:00:00 [kpsmoused] root 52 2 0 09:24 ? 00:00:00 [ipv6_addrconf] root 66 2 0 09:24 ? 00:00:00 [deferwq] root 120 2 0 09:24 ? 00:00:00 [kauditd] root 185 2 0 09:24 ? 00:00:00 [rpciod] root 186 2 0 09:24 ? 00:00:00 [xprtiod] root 257 2 0 09:24 ? 00:00:00 [nvme-wq] root 260 2 0 09:24 ? 00:00:00 [nvme-reset-wq] root 261 2 0 09:24 ? 00:00:00 [nvme-delete-wq] root 262 2 0 09:24 ? 00:00:00 [ena] root 269 2 0 09:24 ? 00:00:00 [kworker/u4:4] root 282 2 0 09:24 ? 00:00:00 [bioset] root 283 2 0 09:24 ? 00:00:00 [xfsalloc] root 284 2 0 09:24 ? 00:00:00 [xfs_mru_cache] root 285 2 0 09:24 ? 00:00:00 [xfs-buf/nvme0n1] root 286 2 0 09:24 ? 00:00:00 [xfs-data/nvme0n] root 287 2 0 09:24 ? 00:00:00 [xfs-conv/nvme0n] root 288 2 0 09:24 ? 00:00:00 [xfs-cil/nvme0n1] root 289 2 0 09:24 ? 00:00:00 [xfs-reclaim/nvm] root 290 2 0 09:24 ? 00:00:00 [xfs-log/nvme0n1] root 291 2 0 09:24 ? 00:00:00 [xfs-eofblocks/n] root 292 2 0 09:24 ? 00:00:02 [xfsaild/nvme0n1] root 293 2 0 09:24 ? 00:00:00 [kworker/1:1H] root 294 2 0 09:24 ? 00:00:00 [kworker/0:1H] root 407 1 0 09:24 ? 00:00:02 /usr/lib/systemd/systemd-journald root 444 1 0 09:24 ? 00:00:00 /usr/sbin/lvmetad -f root 445 1 0 09:24 ? 00:00:00 /usr/lib/systemd/systemd-udevd root 533 2 0 09:24 ? 00:00:00 [nfit] root 541 1 0 09:24 ? 00:00:00 /sbin/auditd root 568 1 0 09:24 ? 00:00:00 /usr/sbin/irqbalance --foreground dbus 570 1 0 09:24 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation rpc 574 1 0 09:24 ? 00:00:00 /sbin/rpcbind -w root 576 1 0 09:24 ? 00:00:00 /usr/lib/systemd/systemd-logind polkitd 577 1 0 09:24 ? 00:00:00 /usr/lib/polkit-1/polkitd --no-debug root 590 1 0 09:24 ? 00:00:00 /usr/sbin/gssproxy -D root 600 1 0 09:24 ? 00:00:00 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid root 940 1 0 09:25 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H ip-10-0-1-58 eth0 root 999 1 0 09:25 ? 00:00:00 /usr/bin/python2 -Es /usr/sbin/tuned -l -P root 1091 1 0 09:25 tty1 00:00:00 /sbin/agetty --noclear tty1 linux root 1098 1 0 09:25 ? 00:00:00 /usr/sbin/crond -n root 1100 1 0 09:25 ttyS0 00:00:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 chrony 1105 1 0 09:25 ? 00:00:00 /usr/sbin/chronyd root 1225 1 0 09:25 ? 00:00:00 /usr/sbin/sshd -D root 7605 2 0 09:35 ? 00:00:00 [kworker/0:0] root 7753 2 0 09:45 ? 00:00:00 [kworker/0:2] root 8023 1 0 10:01 ? 00:00:00 /usr/sbin/anacron -s root 18613 1225 0 10:41 ? 00:00:00 sshd: admin [priv] admin 18616 18613 0 10:41 ? 00:00:00 sshd: admin@pts/0 admin 18617 18616 0 10:41 pts/0 00:00:00 -bash root 18825 2 0 10:52 ? 00:00:00 [kworker/1:2] root 18936 2 0 10:57 ? 00:00:00 [kworker/1:1] admin 19141 18617 0 11:08 pts/0 00:00:00 ps -ef