Make MyRocks copy ALTERs optionally use bulk load semantics
General
Escalation
General
Escalation
Description
A typical problem when migrating tables from InnoDB to MyRocks and performing ALTER TABLEs which make a copy of the table is excessive memory consumption due to MyRocks/RocksDB transactional memory requirements. Some of this can be mitigated with MyRocks WriteBatch settings, but, not entirely due to memory required by RocksDB point locking scheme.
To help solve this issue and reduce the chances of OOM or severe performance degradation during the copy, we should (optionally controlled) utilize the MyRocks bulk loading and commit in the middle mechanisms. Since copying ALTERs is executed entirely from the server layer with minimal support from the storage engines ALTER code, we may want to extend the storage engine interface to expose an optional pre and post table copy call that would allow a MyRocks specific implementation of these to enable the bulk loading and commit in the middle functionality during the table copy. There are likely other ways to achieve the solution so we need to investigate and propose a good solution.
A typical problem when migrating tables from InnoDB to MyRocks and performing ALTER TABLEs which make a copy of the table is excessive memory consumption due to MyRocks/RocksDB transactional memory requirements. Some of this can be mitigated with MyRocks WriteBatch settings, but, not entirely due to memory required by RocksDB point locking scheme.
To help solve this issue and reduce the chances of OOM or severe performance degradation during the copy, we should (optionally controlled) utilize the MyRocks bulk loading and commit in the middle mechanisms. Since copying ALTERs is executed entirely from the server layer with minimal support from the storage engines ALTER code, we may want to extend the storage engine interface to expose an optional pre and post table copy call that would allow a MyRocks specific implementation of these to enable the bulk loading and commit in the middle functionality during the table copy. There are likely other ways to achieve the solution so we need to investigate and propose a good solution.