PXC 8 crashes on startup when upgrading from 5.7

General

Escalation

General

Escalation

Description

2020-05-12T16:54:40.776149Z 0 [Warning] [MY-000000] [WSREP] Option --wsrep-causal-reads is deprecated. Please start using --wsrep-sync-wait.
2020-05-12T16:54:40.776155Z 0 [Warning] [MY-000000] [WSREP] --wsrep-causal-reads=ON takes precedence over --wsrep-sync-wait=0. WSREP_SYNC_WAIT_BEFORE_READ is on
2020-05-12T16:54:40.777245Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18-9) starting as process 1583
2020-05-12T16:54:40.778935Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2020-05-12T16:54:40.778968Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib/galera4/libgalera_smm.so'
2020-05-12T16:54:40.780326Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy <info@codership.com> loaded successfully.
2020-05-12T16:54:40.780364Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.
2020-05-12T16:54:40.780779Z 0 [Note] [MY-000000] [Galera] Found saved state: db204094-9470-11ea-9032-4277165867e8:-1, safe_to_bootstrap: 1
2020-05-12T16:54:40.780885Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: db204094-9470-11ea-9032-4277165867e8
Seqno: -1 - -1
Offset: -1
Synced: 0
2020-05-12T16:54:40.780945Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: db204094-9470-11ea-9032-4277165867e8, offset: -1
2020-05-12T16:54:40.781030Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete.
2020-05-12T16:54:40.781087Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete.
2020-05-12T16:54:40.781153Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1
2020-05-12T16:54:40.781202Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/168 bytes) complete.
2020-05-12T16:54:40.781237Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (168/168 bytes) complete.
2020-05-12T16:54:40.781271Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers
2020-05-12T16:54:40.781307Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 168/2147483648
2020-05-12T16:54:40.785202Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /data/mysql/; base_host = 10.233.120.174; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; 
2020-05-12T16:54:40.852007Z 0 [Note] [MY-000000] [WSREP] Starting replication
2020-05-12T16:54:40.852102Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
2020-05-12T16:54:40.852155Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2020-05-12T16:54:40.852250Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2020-05-12T16:54:40.852396Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2020-05-12T16:54:40.852491Z 0 [Note] [MY-000000] [Galera] initializing ssl context
2020-05-12T16:54:40.852665Z 0 [Note] [MY-000000] [Galera] backend: asio
2020-05-12T16:54:40.852800Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 
2020-05-12T16:54:40.852929Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk successful
2020-05-12T16:54:40.871255Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2020-05-12T16:54:40.871397Z 0 [Note] [MY-000000] [Galera] (ee952e10, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2020-05-12T16:54:40.871471Z 0 [Note] [MY-000000] [Galera] (ee952e10, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2020-05-12T16:54:40.871725Z 0 [Note] [MY-000000] [Galera] EVS version 1
2020-05-12T16:54:40.871849Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'xtradb'
2020-05-12T16:54:40.871935Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
2020-05-12T16:54:40.872201Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
2020-05-12T16:54:40.872285Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
2020-05-12T16:54:40.872376Z 0 [Note] [MY-000000] [Galera] Node ee952e10 state primary
2020-05-12T16:54:40.872476Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,ee952e10,9)
memb {
 ee952e10,0
 }
joined {
 }
left {
 }
partitioned {
 }
)
2020-05-12T16:54:40.872555Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2020-05-12T16:54:40.885288Z 0 [Note] [MY-000000] [Galera] clear restored view
2020-05-12T16:54:40.885443Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2020-05-12T16:54:40.885636Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2020-05-12T16:54:40.885871Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
2020-05-12T16:54:40.886047Z 0 [Note] [MY-000000] [Galera] Opened channel 'xtradb'
2020-05-12T16:54:40.886237Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2020-05-12T16:54:40.886449Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2020-05-12T16:54:40.886592Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 45a4c967-9471-11ea-a66d-130a23cd9c77
2020-05-12T16:54:40.886710Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2020-05-12T16:54:40.886877Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 45a4d51f-9471-11ea-a656-a3d6596e941c
2020-05-12T16:54:40.887027Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 45a4d51f-9471-11ea-a656-a3d6596e941c
2020-05-12T16:54:40.887160Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 45a4d51f-9471-11ea-a656-a3d6596e941c from 0 (node01)
2020-05-12T16:54:40.887348Z 0 [Note] [MY-000000] [Galera] Quorum results:
 version = 6,
 component = PRIMARY,
 conf_id = 0,
 members = 1/1 (primary/total),
 act_id = 0,
 last_appl. = 0,
 protocols = 2/10/4 (gcs/repl/appl),
 vote policy= 0,
 group UUID = 45a4c967-9471-11ea-a66d-130a23cd9c77
2020-05-12T16:54:40.887502Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2020-05-12T16:54:40.887660Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (1)
2020-05-12T16:54:40.887802Z 0 [Note] [MY-000000] [Galera] Member 0.0 (node01) synced with group.
2020-05-12T16:54:40.887969Z 2 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered
2020-05-12T16:54:40.888189Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 1)
2020-05-12T16:54:40.888321Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1
2020-05-12T16:54:40.888541Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)
2020-05-12T16:54:40.888771Z 2 [Note] [MY-000000] [Galera] ####### My UUID: ee952e10-946f-11ea-b21c-3205d47b4024
2020-05-12T16:54:40.888997Z 2 [Note] [MY-000000] [Galera] Server node01 connected to cluster at position 45a4c967-9471-11ea-a66d-130a23cd9c77:1 with ID ee952e10-946f-11ea-b21c-3205d47b4024
2020-05-12T16:54:40.889227Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
[wsrep_notify] called with args : --status connected
[wsrep_notify] Status Update xtradb-bootstrap-57ffc49947-c96v4 status=connected
2020-05-12T16:54:40.898326Z 2 [Note] [MY-000000] [Galera] ####### ST not required
2020-05-12T16:54:40.898565Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
2020-05-12T16:54:40.898832Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2020-05-12T16:54:40.899149Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
2020-05-12T16:54:40.899363Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 1
2020-05-12T16:54:40.899570Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2020-05-12T16:54:40.899868Z 2 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1
2020-05-12T16:54:40.915639Z 2 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped
2020-05-12T16:54:40.915837Z 2 [Note] [MY-000000] [Galera] GCache history reset: old(db204094-9470-11ea-9032-4277165867e8:1 -> 45a4c967-9471-11ea-a66d-130a23cd9c77:0
2020-05-12T16:54:43.003023Z 2 [Note] [MY-000000] [Galera] Flushing memory map to disk...
2020-05-12T16:54:45.378325Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 168 bytes
2020-05-12T16:54:45.378509Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers
2020-05-12T16:54:45.388775Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 1
2020-05-12T16:54:45.388951Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1
2020-05-12T16:54:45.389113Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2020-05-12T16:54:45.389255Z 2 [Note] [MY-000000] [Galera] ================================================
View:
 id: 45a4c967-9471-11ea-a66d-130a23cd9c77:1
 status: primary
 protocol_version: 4
 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 final: no
 own_index: 0
 members(1):
 0: ee952e10-946f-11ea-b21c-3205d47b4024, node01
=================================================
2020-05-12T16:54:45.389383Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
[wsrep_notify] called with args : --status joiner
[wsrep_notify] Status Update xtradb-bootstrap-57ffc49947-c96v4 status=joiner
2020-05-12T16:54:45.395452Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
[wsrep_notify] called with args : --status initializing
[wsrep_notify] Status Update xtradb-bootstrap-57ffc49947-c96v4 status=initializing
2020-05-12T16:54:45.405690Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2020-05-12T16:54:45.586878Z 3 [Note] [MY-000000] [WSREP] Initiating SST cancellation
16:54:45 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x55a758bb7260
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0bb51cdce0 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55a7566fd161]
/usr/sbin/mysqld(handle_fatal_signal+0x323) [0x55a755764e93]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f0c4be413c0]
/usr/sbin/mysqld(log_online_follow_redo_log_one_pass()+0xec) [0x55a756899b0c]
/usr/sbin/mysqld(srv_start(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x30ae) [0x55a7569b326e]
/usr/sbin/mysqld(+0x2336deb) [0x55a7567e9deb]
/usr/sbin/mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x9a) [0x55a756495aaa]
/usr/sbin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x3fc) [0x55a7566c761c]
/usr/sbin/mysqld(+0x13bed18) [0x55a755871d18]
/usr/sbin/mysqld(+0x2785eba) [0x55a756c38eba]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f0c4be35609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f0c4b611103]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 3
Status: NOT_KILLED
You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.

Environment

None

Linked issues

is triggered by

PS-7057

PS 5.7 to 8.0 upgrade failed with crash when innodb_track_changed_pages =1

Smart Checklist

Activity

Show:

Julia Vural March 4, 2025 at 9:27 PM

It appears that this issue is no longer being worked on, so we are closing it for housekeeping purposes. If you believe the issue still exists, please open a new ticket after confirming it's present in the latest release.

Lalit Choudhary May 14, 2020 at 8:45 AM

The original issue in Percona server. Ref: https://jira.percona.com/browse/PS-7057

Cameron Braid May 13, 2020 at 2:56 PM

That fixes it

I first only set innodb_track_changed_pages=0 in v5.7 and it crashed, but when also setting innodb_track_changed_pages=0 in v8 it worked

xtradb-bootstrap-6d6b956bb9-x6hkx xtradb =================================================
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:52:12.748823Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] called with args : --status joiner
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-x6hkx status=joiner
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:52:12.773116Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] called with args : --status initializing
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-x6hkx status=initializing
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:52:12.786986Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 100 200 300 400 500
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 100 200 300 400 500
xtradb-bootstrap-6d6b956bb9-x6hkx prom-mysqld-exporter time="2020-05-13T14:52:24Z" level=error msg="Error pinging mysqld: dial tcp 127.0.0.1:3306: getsockopt: connection refused" source="mysqld_exporter.go:247" 
xtradb-bootstrap-6d6b956bb9-x6hkx prom-mysqld-exporter time="2020-05-13T14:52:39Z" level=error msg="Error pinging mysqld: dial tcp 127.0.0.1:3306: getsockopt: connection refused" source="mysqld_exporter.go:247" 
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:52:43.992809Z 4 [System] [MY-011003] [Server] Finished populating Data Dictionary tables with data.
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:52:47.093866Z 4 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:52:48.214273Z 7 [System] [MY-013381] [Server] Server upgrade from '50700' to '80018' started.
xtradb-bootstrap-6d6b956bb9-x6hkx prom-mysqld-exporter time="2020-05-13T14:52:54Z" level=error msg="Error pinging mysqld: dial tcp 127.0.0.1:3306: getsockopt: connection refused" source="mysqld_exporter.go:247" 
xtradb-bootstrap-6d6b956bb9-x6hkx prom-mysqld-exporter time="2020-05-13T14:53:09Z" level=error msg="Error pinging mysqld: dial tcp 127.0.0.1:3306: getsockopt: connection refused" source="mysqld_exporter.go:247" 
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.157577Z 7 [System] [MY-013381] [Server] Server upgrade from '50700' to '80018' completed.
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.252716Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.264615Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/data/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.281300Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.281321Z 0 [Note] [MY-000000] [Galera] Server initialized
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.281328Z 0 [Note] [MY-000000] [WSREP] Server status change initializing -> initialized
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] called with args : --status initialized
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-x6hkx status=initialized
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.290663Z 2 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301380Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301478Z 12 [Note] [MY-000000] [WSREP] Starting applier thread 12
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301563Z 15 [Note] [MY-000000] [WSREP] Starting applier thread 15
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301585Z 16 [Note] [MY-000000] [WSREP] Starting applier thread 16
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301564Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.18-9' socket: '/data/mysql/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel9, Revision 6490e00, WSREP version 26.4.3.
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301659Z 17 [Note] [MY-000000] [WSREP] Starting applier thread 17
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.301486Z 13 [Note] [MY-000000] [WSREP] Starting applier thread 13
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.302295Z 18 [Note] [MY-000000] [WSREP] Starting applier thread 18
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.302419Z 19 [Note] [MY-000000] [WSREP] Starting applier thread 19
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.302454Z 20 [Note] [MY-000000] [WSREP] Starting applier thread 20
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.302707Z 10 [Note] [MY-000000] [WSREP] Cluster table is empty, not recovering transactions
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.302768Z 2 [Note] [MY-000000] [WSREP] Server status change initialized -> joined
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] called with args : --status joined
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-x6hkx status=joined
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] called with args : --status joined --uuid 510e9c1c-9529-11ea-beb3-3a8d4bf9ab76 --primary yes --index 0 --members fa3e4d49-9528-11ea-92e9-43209c6144ef/node01/10.233.121.59:3306
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-x6hkx status=joined
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Primary Update xtradb-bootstrap-6d6b956bb9-x6hkx primary=yes
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.336338Z 2 [Note] [MY-000000] [Galera] Server node01 synced with group
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.336376Z 2 [Note] [MY-000000] [WSREP] Server status change joined -> synced
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.336382Z 2 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] called with args : --status synced
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-x6hkx status=synced
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.436502Z 0 [ERROR] [MY-011300] [Server] Plugin mysqlx reported: 'Setup of socket: '/var/run/mysqld/mysqlx.sock' failed, can't create lock file /var/run/mysqld/mysqlx.sock.lock'
xtradb-bootstrap-6d6b956bb9-x6hkx xtradb 2020-05-13T14:53:11.436525Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060

Lalit Choudhary May 13, 2020 at 2:33 PM

@Cameron Braid

Thank you for the test.

Could you please do 1 more test by disabling innodb_track_changed_pages option.

In my.cnf

innodb_track_changed_pages=0

Restart PXC MySQL node.

and then try shutdown and upgrade.

Cameron Braid May 13, 2020 at 2:25 PM

Nope, definately a clean shutdown, same issue

xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:05.032194Z 0 [Note] Shutting down plugin 'InnoDB'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:05.032244Z 0 [Note] InnoDB: FTS optimize thread exiting.
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:05.036092Z 0 [Note] InnoDB: Starting shutdown...
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:05.140175Z 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql/ib_buffer_pool
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:05.140659Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 200514 0:21:05
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:05.744308Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.640786Z 0 [Note] InnoDB: Shutdown completed; log sequence number 19741531522
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641172Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641190Z 0 [Note] Shutting down plugin 'CSV'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641196Z 0 [Note] Shutting down plugin 'MEMORY'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641200Z 0 [Note] Shutting down plugin 'MyISAM'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641208Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641230Z 0 [Note] Shutting down plugin 'sha256_password'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641233Z 0 [Note] Shutting down plugin 'mysql_native_password'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641237Z 0 [Note] Shutting down plugin 'wsrep'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.641363Z 0 [Note] Shutting down plugin 'binlog'
xtradb-bootstrap-5dd9dc8d4f-4zxj2 xtradb 2020-05-13T14:21:07.643455Z 0 [Note] mysqld: Shutdown complete

same crash

xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 2020-05-13T14:24:19.680949Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb [wsrep_notify] called with args : --status joiner
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-gn7q2 status=joiner
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 2020-05-13T14:24:19.711666Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb [wsrep_notify] called with args : --status initializing
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb [wsrep_notify] Status Update xtradb-bootstrap-6d6b956bb9-gn7q2 status=initializing
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 2020-05-13T14:24:19.724918Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 2020-05-13T14:24:19.987742Z 3 [Note] [MY-000000] [WSREP] Initiating SST cancellation
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 14:24:19 UTC - mysqld got signal 11 ;
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Thread pointer: 0x560cd2bdf7f0
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Attempting backtrace. You can use the following information to find out
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb where mysqld died. If you see no messages after this, something went
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb terribly wrong...
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb stack_bottom = 7fdf6d13fce0 thread_stack 0x46000
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x560cd0530161]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(handle_fatal_signal+0x323) [0x560ccf597e93]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fe003db33c0]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(log_online_follow_redo_log_one_pass()+0xec) [0x560cd06ccb0c]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(srv_start(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x30ae) [0x560cd07e626e]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(+0x2336deb) [0x560cd061cdeb]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x9a) [0x560cd02c8aaa]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x3fc) [0x560cd04fa61c]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(+0x13bed18) [0x560ccf6a4d18]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /usr/sbin/mysqld(+0x2785eba) [0x560cd0a6beba]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fe003da7609]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fe003583103]
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Trying to get some variables.
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Some pointers may be invalid and cause the dump to abort.
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Query (0): Connection ID (thread ID): 3
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb Status: NOT_KILLED
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb 
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb You may download the Percona XtraDB Cluster operations manual by visiting
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb http://www.percona.com/software/percona-xtradb-cluster/. You may find information
xtradb-bootstrap-6d6b956bb9-gn7q2 xtradb in the manual which will help you identify the cause of the crash.

Won't Do

Details

Assignee

Unassigned

Reporter

Cameron Braid

Labels

upgrade

Affects versions

8.0.18-9.3

Priority

Medium

Smart Checklist

Created May 12, 2020 at 4:56 PM

Updated March 4, 2025 at 9:27 PM

Resolved March 4, 2025 at 9:27 PM

Configure

Percona XtraDB Cluster

PXC 8 crashes on startup when upgrading from 5.7

Description

Environment

Linked issues

is triggered by

Smart Checklist

Activity

Julia Vural March 4, 2025 at 9:27 PM

Lalit Choudhary May 14, 2020 at 8:45 AM

Cameron Braid May 13, 2020 at 2:56 PM

Lalit Choudhary May 13, 2020 at 2:33 PM

Cameron Braid May 13, 2020 at 2:25 PM

Details

Assignee

Reporter

Labels

Affects versions

Priority

Smart Checklist

Flag notifications

Something's gone wrong