New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature-wip](merge-on-write) MOW table support different primary keys and sort keys #24788

Merged

dataroaring merged 38 commits into apache:master from mymeiyi:mow-master-0922-tmp-4

Nov 24, 2023

Contributor

mymeiyi commented Sep 22, 2023 •

edited

Loading

Proposed changes

MOW tables support define different primary keys and sort keys like this:

       CREATE TABLE `tableName` (
                `c_custkey` int(11) NOT NULL COMMENT "",
                `c_name` varchar(26) NOT NULL COMMENT "",
                `c_address` varchar(41) NOT NULL COMMENT "",
                `c_city` varchar(11) NOT NULL COMMENT ""
        )
        UNIQUE KEY (`c_custkey`)
        CLUSTER BY (`c_name`, `c_city`, `c_address`)
        DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
        PROPERTIES (
            "replication_num" = "1",
            "enable_unique_key_merge_on_write" = "true"
        );

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions bot reviewed

View reviewed changes

Contributor

github-actions bot left a comment

clang-tidy made some suggestions

be/src/olap/memtable.cpp

+                  vectorized::MutableBlock mutable_block =
+                          vectorized::MutableBlock::build_mutable_block(&in_block);
+                  std::vector<RowInBlock*> row_in_blocks;

Contributor

github-actions bot Sep 22, 2023

warning: variable 'row_in_blocks' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                std::vector<RowInBlock*> row_in_blocks;
          
                std::vector<RowInBlock*> row_in_blocks = 0;

be/src/olap/memtable.cpp

+                  in_block = mutable_block.to_block();
+                  SCOPED_RAW_TIMER(&_stat.put_into_output_ns);
+                  std::vector<int> row_pos_vec;

Contributor

github-actions bot Sep 22, 2023

warning: variable 'row_pos_vec' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                std::vector<int> row_pos_vec;
          
                std::vector<int> row_pos_vec = 0;

be/src/olap/rowset/segment_v2/segment.cpp

               Status Segment::lookup_row_key(const Slice& key, bool with_seq_col, RowLocation* row_location) {
                   RETURN_IF_ERROR(load_pk_index_and_bf());
                   bool has_seq_col = _tablet_schema->has_sequence_col();
+                  bool has_rowid = !_tablet_schema->cluster_key_idxes().empty();

Contributor

github-actions bot Sep 22, 2023

warning: variable 'has_rowid' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                bool has_rowid = !_tablet_schema->cluster_key_idxes().empty();
          
                bool has_rowid = false = !_tablet_schema->cluster_key_idxes().empty();

be/src/olap/rowset/segment_v2/segment_writer.cpp

                                                                   converted_result.second->get_data(), num_rows));
                   }
                   if (_has_key) {
-                      if (_tablet_schema->keys_type() == UNIQUE_KEYS && _opts.enable_unique_key_merge_on_write) {
+                      bool need_primary_key_indexes = (_tablet_schema->keys_type() == UNIQUE_KEYS &&

Contributor

github-actions bot Sep 22, 2023

warning: variable 'need_primary_key_indexes' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                    bool need_primary_key_indexes = (_tablet_schema->keys_type() == UNIQUE_KEYS &&
          
                    bool need_primary_key_indexes = false = (_tablet_schema->keys_type() == UNIQUE_KEYS &&

be/src/olap/rowset/segment_v2/segment_writer.cpp Show resolved Hide resolved

be/src/olap/rowset/segment_v2/segment_writer.cpp Outdated

+                          if (!need_short_key_indexes) {
+                              std::string last_key;
+                              for (size_t pos = 0; pos < num_rows; pos++) {
+                                  std::string key = _full_encode_keys(key_columns, pos);

Contributor

github-actions bot Sep 22, 2023

warning: variable 'key' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                                std::string key = _full_encode_keys(key_columns, pos);
          
                                std::string key = 0 = _full_encode_keys(key_columns, pos);

be/src/olap/rowset/segment_v2/segment_writer.cpp Outdated

+                              std::vector<std::string> primary_keys;
+                              // keep primary keys in memory
+                              for (uint32_t pos = 0; pos < num_rows; pos++) {
+                                  std::string key =

Contributor

github-actions bot Sep 22, 2023

warning: variable 'key' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                                std::string key =
          
                                std::string key = 0 =

be/src/olap/rowset/segment_v2/segment_writer.cpp Outdated

+                              // sort primary keys
+                              std::sort(primary_keys.begin(), primary_keys.end());
+                              // write primary keys
+                              std::string last_key;

Contributor

github-actions bot Sep 22, 2023

warning: variable 'last_key' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                            std::string last_key;
          
                            std::string last_key = 0;

be/src/olap/tablet_schema.h

@@ @@ -248,6 +248,7 @@ class TabletSchema { @@
                   std::vector<TabletColumn>& mutable_columns();
                   size_t num_columns() const { return _num_columns; }
                   size_t num_key_columns() const { return _num_key_columns; }
+                  std::vector<uint32_t> cluster_key_idxes() const { return _cluster_key_idxes; }

Contributor

github-actions bot Sep 22, 2023

warning: function 'cluster_key_idxes' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change

      
                std::vector<uint32_t> cluster_key_idxes() const { return _cluster_key_idxes; }
          
                [[nodiscard]] std::vector<uint32_t> cluster_key_idxes() const { return _cluster_key_idxes; }

be/test/olap/primary_key_index_test.cpp

@@ @@ -57,7 +57,7 @@ TEST_F(PrimaryKeyIndexTest, builder) { @@
                   auto fs = io::global_local_filesystem();
                   EXPECT_TRUE(fs->create_file(filename, &file_writer).ok());
-                  PrimaryKeyIndexBuilder builder(file_writer.get(), 0);
+                  PrimaryKeyIndexBuilder builder(file_writer.get(), 0, 0);

Contributor

github-actions bot Sep 22, 2023

warning: variable 'builder' is not initialized [cppcoreguidelines-init-variables]

Suggested change

      
                PrimaryKeyIndexBuilder builder(file_writer.get(), 0, 0);
          
                PrimaryKeyIndexBuilder builder = 0(file_writer.get(), 0, 0);

Contributor Author

mymeiyi commented Sep 22, 2023

run buildall

doris-robot commented Sep 22, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.23 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162319842 Bytes

doris-robot commented Sep 22, 2023

TeamCity be ut coverage result:
Function Coverage: 36.33% (8024/22088)
Line Coverage: 28.63% (64475/225202)
Region Coverage: 27.55% (33500/121592)
Branch Coverage: 24.22% (17148/70804)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d90f7e0ff591e36e4cec0bc1756285773c7974d_7d90f7e0ff591e36e4cec0bc1756285773c7974d/report/index.html

Contributor Author

mymeiyi commented Sep 22, 2023

run buildall

doris-robot commented Sep 22, 2023

TeamCity be ut coverage result:
Function Coverage: 36.32% (8024/22090)
Line Coverage: 28.63% (64474/225231)
Region Coverage: 27.54% (33489/121597)
Branch Coverage: 24.21% (17146/70808)
Coverage Report: http://coverage.selectdb-in.cc/coverage/114115ea785c2101421b29c4955d3efff3f1999b_114115ea785c2101421b29c4955d3efff3f1999b/report/index.html

doris-robot commented Sep 22, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.67 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162157364 Bytes

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from 114115e to b28cd8e Compare

September 25, 2023 12:05

Contributor Author

mymeiyi commented Sep 25, 2023

run buildall

doris-robot commented Sep 25, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.79 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162173679 Bytes

doris-robot commented Sep 25, 2023

TeamCity be ut coverage result:
Function Coverage: 36.31% (8112/22343)
Line Coverage: 28.42% (64914/228385)
Region Coverage: 27.32% (33662/123231)
Branch Coverage: 23.98% (17194/71716)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b28cd8ec933dd28f427680844d09957b90455312_b28cd8ec933dd28f427680844d09957b90455312/report/index.html

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from b28cd8e to b254d4a Compare

September 26, 2023 11:37

Contributor Author

mymeiyi commented Sep 26, 2023

run buildall

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from b254d4a to d44818c Compare

September 28, 2023 10:07

mymeiyi force-pushed the mow-master-0922-tmp-4 branch 3 times, most recently from b68e609 to f20bede Compare

October 12, 2023 09:49

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from f20bede to 8d367cc Compare

October 16, 2023 03:52

github-actions bot reviewed

View reviewed changes

Contributor

github-actions bot left a comment

clang-tidy made some suggestions

be/src/olap/delete_bitmap_calculator.h

@@ @@ -91,7 +94,7 @@ class MergeIndexDeleteBitmapCalculator { @@
                   MergeIndexDeleteBitmapCalculator() = default;
                   Status init(RowsetId rowset_id, std::vector<SegmentSharedPtr> const& segments,
-                              size_t seq_col_length = 0, size_t max_batch_size = 1024);
+                              size_t seq_col_length = 0, size_t rowid_length = 0, size_t max_batch_size = 1024);

Contributor

github-actions bot Oct 16, 2023

warning: 1024 is a magic number; consider replacing it with a named constant [readability-magic-numbers]

                size_t seq_col_length = 0, size_t rowid_length = 0, size_t max_batch_size = 1024);
                                                                                            ^

be/src/olap/rowset/segment_v2/segment_writer.h

                   // used for unique-key with merge on write
                   void _encode_seq_column(const vectorized::IOlapColumnDataAccessor* seq_column, size_t pos,
                                           string* encoded_keys);
+                  void _encode_rowid(const uint32_t rowid, string* encoded_keys);

Contributor

github-actions bot Oct 16, 2023

warning: parameter 'rowid' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

      
                void _encode_rowid(const uint32_t rowid, string* encoded_keys);
          
                void _encode_rowid(uint32_t rowid, string* encoded_keys);

mymeiyi force-pushed the mow-master-0922-tmp-4 branch 2 times, most recently from 293685a to b1a84a5 Compare

October 26, 2023 06:05

github-actions bot added the meta-change label

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from b1a84a5 to 4967df3 Compare

October 30, 2023 07:14

Contributor Author

mymeiyi commented Oct 30, 2023

run buildall

doris-robot commented Oct 30, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.64 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162043042 Bytes

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from 4967df3 to 7684b08 Compare

October 31, 2023 03:09

Contributor Author

mymeiyi commented Oct 31, 2023

run buildall

doris-robot commented Oct 31, 2023

TeamCity be ut coverage result:
Function Coverage: 37.37% (8472/22672)
Line Coverage: 29.66% (68619/231325)
Region Coverage: 28.26% (35666/126198)
Branch Coverage: 25.14% (18285/72728)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7684b080dbe08b11231a6c230fe1588fcdd60a75_7684b080dbe08b11231a6c230fe1588fcdd60a75/report/index.html

doris-robot commented Oct 31, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.98 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162149380 Bytes

Contributor Author

mymeiyi commented Oct 31, 2023

run buildall

mymeiyi added 16 commits

November 24, 2023 10:16


          support vertical compaction

5ecf38a


          Fix vertical compaction

cc3b92d


          rebase master

052c927


          fix comments

a68ecd0


          disable test_delete_sign_delete_bitmap

de54393

fix

ebf35e0


          fix compile

bb75b95


          skip vertical segment writer

f29ce66


          some fix

a3618f4


          modify by comments

db1b557


          fix DeleteBitmapCalculatorTest

1d1f6c5


          Fix segment compaction


          Fix test_point_query_cluster_key

80bbd69


          Fix test_point_query_cluster_key

fbb6aa8


          Fix test_point_query_cluster_key

520a15c


          Fix test_point_query_cluster_key

378c20a

mymeiyi dismissed stale reviews from zhannngchen and dataroaring via

378c20a

November 24, 2023 02:21

mymeiyi force-pushed the mow-master-0922-tmp-4 branch from eef98b3 to 378c20a Compare

November 24, 2023 02:21

Contributor Author

mymeiyi commented Nov 24, 2023

run buildall

github-actions bot removed the approved label

doris-robot commented Nov 24, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 378c20a7f4f6bb8147ab75198eaf52eadb7f51f4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4940	4656	4630	4630
q2	361	151	162	151
q3	2042	1951	1967	1951
q4	1386	1266	1228	1228
q5	3921	3937	4031	3937
q6	255	132	133	132
q7	1452	890	885	885
q8	2768	2789	2773	2773
q9	31410	9463	9543	9463
q10	3458	3549	3520	3520
q11	376	238	249	238
q12	474	300	289	289
q13	4549	3827	3814	3814
q14	325	291	278	278
q15	587	533	525	525
q16	668	598	583	583
q17	1125	968	926	926
q18	7800	7284	7324	7284
q19	1702	1657	1674	1657
q20	545	309	292	292
q21	4606	4052	3986	3986
q22	484	384	376	376
Total cold run time: 75234 ms
Total hot run time: 48918 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4634	4565	4561	4561
q2	350	240	259	240
q3	4029	3998	3997	3997
q4	2701	2707	2711	2707
q5	9663	9562	9617	9562
q6	242	124	128	124
q7	3040	2479	2497	2479
q8	4402	4403	4429	4403
q9	12982	12817	12840	12817
q10	4055	4206	4196	4196
q11	767	667	745	667
q12	975	803	841	803
q13	4308	3582	3562	3562
q14	378	364	357	357
q15	574	518	522	518
q16	735	656	690	656
q17	3903	3913	3849	3849
q18	9634	8986	8909	8909
q19	1820	1788	1790	1788
q20	2376	2052	2043	2043
q21	8934	8669	8549	8549
q22	934	815	804	804
Total cold run time: 81436 ms
Total hot run time: 77591 ms

doris-robot commented Nov 24, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.21 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 31.4 seconds inserted 10000000 Rows, about 318K ops/s
storage size: 17099577551 Bytes

dataroaring approved these changes

View reviewed changes

Contributor

dataroaring left a comment

LGTM

zhannngchen approved these changes

View reviewed changes

Contributor

zhannngchen left a comment

LGTM

github-actions bot added the approved label

Contributor

github-actions bot commented Nov 24, 2023

PR approved by at least one committer and no changes requested.

dataroaring merged commit 553e4a8 into apache:master

19 of 20 checks passed

seawinde pushed a commit to seawinde/doris that referenced this pull request


          [feature-wip](merge-on-write) MOW table support different primary key…

95e212b

…s and sort keys (apache#24788)

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request


          [feature-wip](merge-on-write) MOW table support different primary key…

6d70532

…s and sort keys (apache#24788)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved meta-change reviewed