rocksdb.persistent_cache failing

General

Escalation

General

Escalation

Description

mysqld: /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc:572: static rocksdb::Slice rocksdb::BlockBasedTable::GetCacheKey(const char*, size_t, const rocksdb::BlockHandle&, char*): Assertion `cache_key_prefix_size != 0' failed.

...

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/ssd/percona/myrocks/percona-server-install-5.7/bin/mysqld(my_print_stacktrace+0x38)[0x18ad211]
/ssd/percona/myrocks/percona-server-install-5.7/bin/mysqld(handle_fatal_signal+0x411)[0xeaeada]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f6850b11330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f684ff4fc37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f684ff53028]
/lib/x86_64-linux-gnu/libc.so.6(+0x2fbf6)[0x7f684ff48bf6]
/lib/x86_64-linux-gnu/libc.so.6(+0x2fca2)[0x7f684ff48ca2]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb15BlockBasedTable11GetCacheKeyEPKcmRKNS_11BlockHandleEPc+0x64)[0x7f682d8ff52a]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb21PersistentCacheHelper13LookupRawPageERKNS_22PersistentCacheOptionsERKNS_11BlockHandleEPSt10unique_ptrIA_cSt14default_deleteIS8_EEm+0xf8)[0x7f682d955b72]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb17ReadBlockContentsEPNS_22RandomAccessFileReaderERKNS_6FooterERKNS_11ReadOptionsERKNS_11BlockHandleEPNS_13BlockContentsERKNS_18ImmutableCFOptionsEbRKNS_5SliceERKNS_22PersistentCacheOptionsE+0x2c3)[0x7f682d929609]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(+0x979b81)[0x7f682d8feb81]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb15BlockBasedTable13ReadMetaBlockEPNS0_3RepEPSt10unique_ptrINS_5BlockESt14default_deleteIS4_EEPS3_INS_16InternalIteratorES5_IS9_EE+0xf4)[0x7f682d900c08]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb15BlockBasedTable4OpenERKNS_18ImmutableCFOptionsERKNS_10EnvOptionsERKNS_22BlockBasedTableOptionsERKNS_21InternalKeyComparatorEOSt10unique_ptrINS_22RandomAccessFileReaderESt14default_deleteISE_EEmPSD_INS_11TableReaderESF_ISJ_EEbbi+0x3a0)[0x7f682d8ff95c]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZNK7rocksdb22BlockBasedTableFactory14NewTableReaderERKNS_18TableReaderOptionsEOSt10unique_ptrINS_22RandomAccessFileReaderESt14default_deleteIS5_EEmPS4_INS_11TableReaderES6_ISA_EEb+0xa2)[0x7f682d8fbc50]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb10TableCache14GetTableReaderERKNS_10EnvOptionsERKNS_21InternalKeyComparatorERKNS_14FileDescriptorEbmbPNS_13HistogramImplEPSt10unique_ptrINS_11TableReaderESt14default_deleteISD_EEbib+0x321)[0x7f682d87a28f]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZN7rocksdb10TableCache9FindTableERKNS_10EnvOptionsERKNS_21InternalKeyComparatorERKNS_14FileDescriptorEPPNS_5Cache6HandleEbbPNS_13HistogramImplEbib+0x230)[0x7f682d87a686]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZZN7rocksdb14VersionBuilder3Rep17LoadTableHandlersEPNS_13InternalStatsEibENKUlvE_clEv+0x12d)[0x7f682d887ab7]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZNSt17_Function_handlerIFvvEZN7rocksdb14VersionBuilder3Rep17LoadTableHandlersEPNS1_13InternalStatsEibEUlvE_E9_M_invokeERKSt9_Any_data+0x20)[0x7f682d889304]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZNKSt8functionIFvvEEclEv+0x32)[0x7f682d8887ec]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZNSt12_Bind_simpleIFSt8functionIFvvEEvEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE+0x28)[0x7f682d88d308]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZNSt12_Bind_simpleIFSt8functionIFvvEEvEEclEv+0x1b)[0x7f682d88d25f]
/ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvvEEvEEE6_M_runEv+0x1c)[0x7f682d88d1f8]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60)[0x7f68508aea60]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f6850b09184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6850016bed]

Core was generated by `/ssd/percona/myrocks/percona-server-install-5.7/bin/mysqld --defaults-group-suf'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f6850b0e611 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61  ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
#0  0x00007f6850b0e611 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000018ad2a1 in my_write_core (sig=6) at /ssd/percona/myrocks/percona-server-5.7/mysys/stacktrace.c:249
#2  0x0000000000eaecd1 in handle_fatal_signal (sig=6) at /ssd/percona/myrocks/percona-server-5.7/sql/signal_handler.cc:223
#3  <signal handler called>
#4  0x00007f684ff4fc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f684ff53028 in __GI_abort () at abort.c:89
#6  0x00007f684ff48bf6 in __assert_fail_base (fmt=0x7f685009cbf8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f682db8622f "cache_key_prefix_size != 0", file=file@entry=0x7f682db861b8 "/ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc", line=line@entry=572, function=function@entry=0x7f682db87e00 <rocksdb::BlockBasedTable::GetCacheKey(char const*, unsigned long, rocksdb::BlockHandle const&, char*)::__PRETTY_FUNCTION__> "static rocksdb::Slice rocksdb::BlockBasedTable::GetCacheKey(const char*, size_t, const rocksdb::BlockHandle&, char*)") at assert.c:92
#7  0x00007f684ff48ca2 in __GI___assert_fail (assertion=0x7f682db8622f "cache_key_prefix_size != 0", file=0x7f682db861b8 "/ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc", line=572, function=0x7f682db87e00 <rocksdb::BlockBasedTable::GetCacheKey(char const*, unsigned long, rocksdb::BlockHandle const&, char*)::__PRETTY_FUNCTION__> "static rocksdb::Slice rocksdb::BlockBasedTable::GetCacheKey(const char*, size_t, const rocksdb::BlockHandle&, char*)") at assert.c:101
#8  0x00007f682d8ff52a in rocksdb::BlockBasedTable::GetCacheKey (cache_key_prefix=0x7f6850b003d8 <std::string::_Rep::_S_empty_rep_storage+24> "", cache_key_prefix_size=0, handle=..., cache_key=0x7f6804ff7160 "") at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc:572
#9  0x00007f682d955b72 in rocksdb::PersistentCacheHelper::LookupRawPage (cache_options=..., handle=..., raw_data=0x7f6804ff7230, raw_data_size=37) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/persistent_cache_helper.cc:57
#10 0x00007f682d929609 in rocksdb::ReadBlockContents (file=0x7f680c000d70, footer=..., read_options=..., handle=..., contents=0x7f6804ff8710, ioptions=..., decompression_requested=true, compression_dict=..., cache_options=...) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/format.cc:344
#11 0x00007f682d8feb81 in rocksdb::(anonymous namespace)::ReadBlockFromFile (file=0x7f680c000d70, footer=..., options=..., handle=..., result=0x7f6804ff87a0, ioptions=..., do_uncompress=true, compression_dict=..., cache_options=..., global_seqno=18446744073709551615, read_amp_bytes_per_bit=0) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc:77
#12 0x00007f682d900c08 in rocksdb::BlockBasedTable::ReadMetaBlock (rep=0x7f680c001590, meta_block=0x7f6804ff88b0, iter=0x7f6804ff88c0) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc:855
#13 0x00007f682d8ff95c in rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool, bool, int) (ioptions=..., env_options=..., table_options=..., internal_comparator=..., file=<unknown type in /ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so, CU 0x1a8466e, DIE 0x1ad8a6f>, file_size=1043, table_reader=0x7f6804ff8ce0, prefetch_index_and_filter_in_cache=false, skip_filters=false, level=0) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_reader.cc:631
#14 0x00007f682d8fbc50 in rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool) const (this=0x250f90e0, table_reader_options=..., file=<unknown type in /ssd/percona/myrocks/percona-server-install-5.7/lib/mysql/plugin/ha_rocksdb.so, CU 0x1a52f30, DIE 0x1a81cd8>, file_size=1043, table_reader=0x7f6804ff8ce0, prefetch_index_and_filter_in_cache=false) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/table/block_based_table_factory.cc:59
#15 0x00007f682d87a28f in rocksdb::TableCache::GetTableReader (this=0x2510a3b0, env_options=..., internal_comparator=..., fd=..., sequential_mode=false, readahead=0, record_read_stats=true, file_read_hist=0x25112af0, table_reader=0x7f6804ff8ce0, skip_filters=false, level=0, prefetch_index_and_filter_in_cache=false) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/db/table_cache.cc:111
#16 0x00007f682d87a686 in rocksdb::TableCache::FindTable (this=0x2510a3b0, env_options=..., internal_comparator=..., fd=..., handle=0x251172d8, no_io=false, record_read_stats=true, file_read_hist=0x25112af0, skip_filters=false, level=0, prefetch_index_and_filter_in_cache=false) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/db/table_cache.cc:148
#17 0x00007f682d887ab7 in rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*, int, bool)::{lambda()#1}::operator()() const (__closure=0x25117230) at /ssd/percona/myrocks/percona-server-5.7/storage/rocksdb/rocksdb/db/version_builder.cc:350
#18 0x00007f682d889304 in std::_Function_handler<void (), rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*, int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/4.8/functional:2071
#19 0x00007f682d8887ec in std::function<void ()>::operator()() const (this=0x2510b4a0) at /usr/include/c++/4.8/functional:2471
#20 0x00007f682d88d308 in std::_Bind_simple<std::function<void ()> ()>::_M_invoke<>(std::_Index_tuple<>) (this=0x2510b4a0) at /usr/include/c++/4.8/functional:1732
#21 0x00007f682d88d25f in std::_Bind_simple<std::function<void ()> ()>::operator()() (this=0x2510b4a0) at /usr/include/c++/4.8/functional:1720
#22 0x00007f682d88d1f8 in std::thread::_Impl<std::_Bind_simple<std::function<void ()> ()> >::_M_run() (this=0x2510b488) at /usr/include/c++/4.8/thread:115
#23 0x00007f68508aea60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#24 0x00007f6850b09184 in start_thread (arg=0x7f6804ff9700) at pthread_create.c:312
#25 0x00007f6850016bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Environment

None

Smart Checklist

Activity

Show:

George Lorch November 21, 2019 at 9:28 PM

incomplete and no activity for > 90 days

Laurynas Biveinis January 10, 2019 at 3:16 PM

@George Lorch is this test still failing?

George Lorch August 11, 2017 at 9:40 PM

This may actually be due to both https://perconadev.atlassian.net/browse/MYR-163#icft=MYR-163 and https://perconadev.atlassian.net/browse/MYR-164#icft=MYR-164

Will watch this after these land and close if test stabilizes.

Not a Bug

Details
Assignee
Unassigned
Reporter
George Lorch(Deactivated)
Labels
cimtr
Components
MyRocks
Affects versions
5.7.18-14
Priority
High

Smart Checklist

Created June 20, 2017 at 8:40 PM

Updated March 6, 2024 at 1:26 PM

Resolved November 21, 2019 at 9:28 PM

rocksdb.persistent_cache failing

Description

Environment

Smart Checklist

Activity

George Lorch November 21, 2019 at 9:28 PM

Laurynas Biveinis January 10, 2019 at 3:16 PM

George Lorch August 11, 2017 at 9:40 PM

Details
Assignee
Unassigned
Reporter
George Lorch(Deactivated)
Labels
cimtr
Components
MyRocks
Affects versions
5.7.18-14
Priority
High

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

Smart Checklist

Smart Checklist

Flag notifications

Something's gone wrong

Something's gone wrong

rocksdb.persistent_cache failing

Description

Environment

Smart Checklist

Activity

George Lorch November 21, 2019 at 9:28 PM

Laurynas Biveinis January 10, 2019 at 3:16 PM

George Lorch August 11, 2017 at 9:40 PM

DetailsAssigneeUnassignedUnassignedReporterGeorge LorchGeorge Lorch(Deactivated)LabelscimtrComponentsMyRocksAffects versions5.7.18-14PriorityHigh

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

Smart ChecklistOpen Smart Checklist

Smart Checklist

Flag notifications

Something's gone wrong

Something's gone wrong

Details
Assignee
Unassigned
Reporter
George Lorch(Deactivated)
Labels
cimtr
Components
MyRocks
Affects versions
5.7.18-14
Priority
High

Smart Checklist