[To dev/1.3] Repair lost data partitions by providing a new SQL statement - #18472
Open
zerolbsony wants to merge 13 commits into
Open
[To dev/1.3] Repair lost data partitions by providing a new SQL statement#18472zerolbsony wants to merge 13 commits into
zerolbsony wants to merge 13 commits into
Conversation
* Come true all rpc interfaces in the DataNode, and partial features in the DataPartitionTableIntegrityCheckProcedure. * Debugged and verified all key logic in the procedure. * Correct dataPartitionTables and finalDataPartitionTable serialization; Adjust method that record the earliest timeslot id for every database * Correct heartbeat logic when data partition table is generating; Remove two unit testes only run successful in local environment * Use StorageEngine.getInstance().getAllDataRegions() to get Data Partition Information instead of scanning data directories in the DataNode; Correct the logic that retry after the step failed; Correct skipDataNodes and failedDataNodes serialization and deserialization. * Adjust the default value is 1 min * Adjust the default value is 1 min * Append a description about the unit * use the spotless command to format code * Avoid writing duplicate values * Fix bug when get no data partition table in the ConfigNode. * Add a license description. * mvn spotless:apply * Fix problems based on review opinions * Remove some unuseful functions * Find the earliest time slot in the ConfigNode * Resolve the problem that data partition table generation is not completed in the progress caused by rpc timeout * Correct need to redirect the target step. * Integrate the merge logic into the DataPartitionTable class * Support multiple databases data partition tables are repaired * Fix bugs * Fix * Compare startTime between ConfigNode and DataNode, if the larger one is from ConfigNode, the data partition table lost. * Change to 10 seconds. * Correct the input parameter name * Changed to the local variable. * Fix all opinions * spotless code * license * Use our standard binary written function instead of stream.UTF(); The overflow problem still exist, that's no way to resolve, previously, the user set timePartitionOrigin to Long.MIN_VALUE. In this case, adding it to partitionId = -1 will indeed cause an overflow. However, the partition table in the system only accepts timestamps of the long type and does not support bigint timestamps. Therefore, if an overflow actually occurs, we have to accept the outcome where the program is interrupted by an exception being thrown. * Fix some opinions * Fix another lots of opinions * Optimize codes * Skip current loop when no time slot info is found in the seq and unseq directory * Fix some opinions * Use the Google guava RateLimiter instead of LeakyBucketRateLimiter; Adjust new method to compute progress of data partition table generation * Fix * Fix bug * Fix serialize bugs (cherry picked from commit 402abdd)
…Exception (apache#17363) (cherry picked from commit fd3726f)
…TableIntegrityCheckProcedure (apache#17369) (cherry picked from commit 3b02d32)
…g data partition table (apache#17416) (cherry picked from commit 436524f)
…quest frequently after one of DNs in the cluster is down up (apache#17461) (cherry picked from commit 5b642df)
…re complete (apache#17476) (cherry picked from commit d44046c)
…rrent progress if DN is restarted before the data partition generation complete (apache#17491) (cherry picked from commit b4ce8da)
… ConfigNode start up. (apache#17506) (cherry picked from commit 89c2af8)
(cherry picked from commit b7eb193)
…nt (apache#17637) (cherry picked from commit 522c31c)
…o consensus when not found any data partition in the CN (apache#18104) (cherry picked from commit 453cb6e)
…che#18221) (cherry picked from commit 9ae6674)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.