Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Introduce Azure CLI for checking if backup objects/folders exist in Azure storage

Description

We are make http requests (curl) to the Azure API for checking if objects exist in the Azure blob containers.

https://github.com/percona/percona-docker/blob/6e34153e77cadb30554ed4c33eb2428c6b198279/percona-xtradb-cluster-8.0-backup/lib/pxc/backup.sh#L36-L84

(Sharing 8.0 backup version, but the same code exists in other versions as well, e.g 5.7 and 8.4)

e.g.

azure_auth_header_file() { local params="$1" local request_date="$2" local hex_tmp local signature_tmp local auth_header_tmp local resource local string_to_sign local decoded_key hex_tmp=$(mktemp) signature_tmp=$(mktemp) auth_header_tmp=$(mktemp) decoded_key=$(echo -n "$AZURE_ACCESS_KEY" | base64 -d | hexdump -ve '1/1 "%02x"') echo -n "$decoded_key" >"$hex_tmp" resource="/$AZURE_STORAGE_ACCOUNT/$AZURE_CONTAINER_NAME" string_to_sign=$(printf "GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:%s\nx-ms-version:2021-06-08\n%s\n%s" \ "$request_date" \ "$resource" \ "$params") printf "%s" "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$(cat "$hex_tmp")" -binary | base64 >"$signature_tmp" echo -n "Authorization: SharedKey $AZURE_STORAGE_ACCOUNT:$(cat "$signature_tmp")" >"$auth_header_tmp" echo "$auth_header_tmp" } is_object_exist_azure() { object="$1" { set +x; } 2>/dev/null connection_string="$ENDPOINT/$AZURE_CONTAINER_NAME?comp=list&restype=container&prefix=$object" request_date=$(LC_ALL=en_US.utf8 TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z") header_version="x-ms-version: 2021-06-08" header_date="x-ms-date: $request_date" header_auth_file=$(azure_auth_header_file "$(printf 'comp:list\nprefix:%s\nrestype:container' "$object")" "$request_date") response=$(curl -s -H "$header_version" -H "$header_date" -H "@$header_auth_file" "${connection_string}") res=$(echo "$response" | grep "<Blob>") set -x if [[ ${#res} -ne 0 ]]; then return 1 fi return 0 }

This implementation is hard to maintain and it is error prone. We can switch to Azure CLI for checking if an object exist in the blob container.

some example azure cli commands that we can consider

for listing everything in a prefix:

az storage blob list \ --account-name "engk8soperators" \ --container-name "operator-testing" \ --auth-mode key \ --account-key <SECRET KEY> \ --query "[?starts_with(name, 'cluster1-2025-04-01-08:57:45-full/')].name"
Screenshot 2025-04-02 at 12.40.44 AM.png

Check if a particular file exists in the blob container:

az storage blob exists \ --account-name "engk8soperators" \ --container-name "operator-testing" \ --name "cluster1-2025-04-01-08:57:45-full/backup-my.cnf.00000000000000000000" \ --auth-mode key \ --account-key <SECRET KEY>
Screenshot 2025-04-02 at 12.42.22 AM.png

IMPORTANT NOTE: We need to ask the release team to add azure cli to pxb image so we can start using it

Environment

None

Attachments

2
  • 01 Apr 2025, 09:43 PM
  • 01 Apr 2025, 09:43 PM

Activity

Details

Assignee

Reporter

Needs QA

Yes

Fix versions

Priority

Smart Checklist

Created 2 days ago
Updated 2 days ago

Flag notifications