cherry pick upstream release-7.5 commits for compatibility (tikv#1753)

* charset: support utf8mb4_0900_ai_ci collation (tikv#15329) close tikv#15328 Signed-off-by: Yang Keao <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * charset: implement utf8mb4_0900_bin collation (tikv#15398) close tikv#15397 Signed-off-by: Yang Keao <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * expr: support json function member_of (tikv#15391) close tikv#15388 Signed-off-by: wshwsh12 <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * storage: avoid duplicated Instant:now (tikv#15489) close tikv#15490 avoid duplicated Instant:now Signed-off-by: SpadeA-Tang <[email protected]> * coprocessor: use the deadline in kvrpcpb::Context (tikv#15564) close tikv#15565 Signed-off-by: lance6716 <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * tidb_query_datatype,collation: remove utf8mb4_0900_bin from need_restored_data (tikv#15572) close tikv#15571 Signed-off-by: Yang Keao <[email protected]> * expr: fix wrong result of 0 / decimal and 0 % decimal (tikv#15675) close tikv#15631 Signed-off-by: SeaRise <[email protected]> * coprocessor: do not treat deadline exceeded error as other error (tikv#15709) ref tikv#15566 Signed-off-by: glorv <[email protected]> * tidb_query_expr: fix the behavior of `field` function (tikv#15879) (tikv#15884) close tikv#15878 Signed-off-by: Yang Keao <[email protected]> Co-authored-by: Yang Keao <[email protected]> * executor: omit truncating error when handling decimal multiplication in arithmetic operators (tikv#16187) (tikv#16277) close tikv#16268, close pingcap/tidb#48332 omit truncating error when handling decimal multiplication in arithmetic operators Signed-off-by: TONG, Zhigao <[email protected]> Co-authored-by: TONG, Zhigao <[email protected]> Co-authored-by: tonyxuqqi <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * *: uniform deadline exceeded error in cop response (tikv#16155) (tikv#16502) close tikv#16154 Signed-off-by: crazycs520 <[email protected]> * copr, json: handle u64 json correctly (tikv#16513) (tikv#16538) close tikv#16512 Signed-off-by: Yang Keao <[email protected]> Co-authored-by: Yang Keao <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * expr: fix the wrong decimal's result frac in some cases (tikv#16914) (tikv#16919) close tikv#16913 Fix the wrong decimal's result frac in some cases. Signed-off-by: gengliqi <[email protected]> Co-authored-by: gengliqi <[email protected]> * server: change the log level to debug for cop error response (tikv#15882) (tikv#16927) ref tikv#15881 Change the coprocessor error response log level to DEBUG Signed-off-by: ti-chi-bot <[email protected]> Signed-off-by: cfzjywxk <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * expression: fix a `cast_string_to_decimal` truncate bug (tikv#16963) (tikv#16985) close tikv#16962 fix a `cast_string_to_decimal` truncate bug Signed-off-by: gengliqi <[email protected]> Co-authored-by: gengliqi <[email protected]> * expression: fix overflow panic in `conv` (tikv#16970) (tikv#16980) close tikv#16969 fix overflow panic in `conv` Signed-off-by: gengliqi <[email protected]> Co-authored-by: gengliqi <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * copr: don't resize binary opaque when the type flen is unspecified (tikv#16617) (tikv#16713) close tikv#16616 Signed-off-by: Yang Keao <[email protected]> Co-authored-by: Yang Keao <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * grpc: set compression args for TiKV service (tikv#17180) (tikv#17263) close tikv#17176 Set compression arguments for TiKV service. The compression arguments are loaded from TiKV config. It will affect TiKV's response to TiDB. Signed-off-by: ekexium <[email protected]> Co-authored-by: ekexium <[email protected]> Signed-off-by: Evan Zhou <[email protected]> * fix format Signed-off-by: Evan Zhou <[email protected]> * build image for e2e Signed-off-by: zeminzhou <[email protected]> --------- Signed-off-by: SpadeA-Tang <[email protected]> Signed-off-by: Yang Keao <[email protected]> Signed-off-by: SeaRise <[email protected]> Signed-off-by: glorv <[email protected]> Signed-off-by: crazycs520 <[email protected]> Signed-off-by: Evan Zhou <[email protected]> Signed-off-by: zeminzhou <[email protected]> Co-authored-by: YangKeao <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: Shenghui Wu <[email protected]> Co-authored-by: Spade A <[email protected]> Co-authored-by: lance6716 <[email protected]> Co-authored-by: SeaRise <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: TONG, Zhigao <[email protected]> Co-authored-by: tonyxuqqi <[email protected]> Co-authored-by: crazycs <[email protected]> Co-authored-by: gengliqi <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: cfzjywxk <[email protected]> Co-authored-by: zzm <[email protected]> Co-authored-by: zeminzhou <[email protected]>
iosmanthus · Sep 10, 2024 · 3922ba8 · 3922ba8
1 parent 4d1e5b2
commit 3922ba8
Show file tree

Hide file tree

Showing 32 changed files with 13,395 additions and 361 deletions.
diff --git a/.github/workflows/cd.yml b/.github/workflows/cd.yml
@@ -94,10 +94,10 @@ jobs:
         if: ${{ inputs.update_daily_run_image || github.ref == 'refs/heads/cloud-engine' }}
         run: |
           # Remove the local manifest entry if it exists
-          docker manifest rm hub.pingcap.net/keyspace/tikv-cse:v6.6.0 || true
+          docker manifest rm hub.pingcap.net/keyspace/tikv-cse:v7.5.0 || true
           
-          docker manifest create hub.pingcap.net/keyspace/tikv-cse:v6.6.0 \
+          docker manifest create hub.pingcap.net/keyspace/tikv-cse:v7.5.0 \
           --amend hub.pingcap.net/sunxiaoguang/serverless/tikv:${{ github.sha }}-amd64 \
           --amend hub.pingcap.net/sunxiaoguang/serverless/tikv:${{ github.sha }}-arm64
           
-          docker manifest push hub.pingcap.net/keyspace/tikv-cse:v6.6.0
+          docker manifest push hub.pingcap.net/keyspace/tikv-cse:v7.5.0
diff --git a/components/cloud_server/src/server.rs b/components/cloud_server/src/server.rs
@@ -204,6 +204,8 @@ impl<T: RaftStoreRouter + Unpin, S: StoreAddrResolver + 'static> Server<T, S> {
             .http2_max_ping_strikes(i32::MAX) // For pings without data from clients.
             .keepalive_time(cfg.value().grpc_keepalive_time.into())
             .keepalive_timeout(cfg.value().grpc_keepalive_timeout.into())
+            .default_compression_algorithm(cfg.value().grpc_compression_algorithm())
+            .default_gzip_compression_level(cfg.value().grpc_gzip_compression_level)
             .build_args();
         let health_service = HealthService::default();
         let builder = {

diff --git a/components/rfstore/src/errors.rs b/components/rfstore/src/errors.rs
@@ -7,7 +7,7 @@ use kvproto::{errorpb, metapb};
 use protobuf::ProtobufError;
 use raftstore::coprocessor::Error as CopError;
 use thiserror::Error;
-use tikv_util::codec;
+use tikv_util::{codec, deadline::set_deadline_exceeded_busy_error};
 
 use crate::store::PeerMsg;
 
@@ -260,6 +260,9 @@ impl From<Error> for errorpb::Error {
                     INGEST_OVERLAP_ERROR_TAG, region_id, existed_file_id, ingest_file_id,
                 ));
             }
+            Error::DeadlineExceeded => {
+                set_deadline_exceeded_busy_error(&mut errorpb);
+            }
             _ => {}
         };
 

diff --git a/components/tidb_query_datatype/src/codec/collation/collator/mod.rs b/components/tidb_query_datatype/src/codec/collation/collator/mod.rs
@@ -5,7 +5,7 @@ mod gbk_collation;
 mod latin1_bin;
 mod utf8mb4_binary;
 mod utf8mb4_general_ci;
-mod utf8mb4_unicode_ci;
+mod utf8mb4_uca;
 
 use std::{
     cmp::Ordering,
@@ -19,7 +19,7 @@ pub use gbk_collation::*;
 pub use latin1_bin::*;
 pub use utf8mb4_binary::*;
 pub use utf8mb4_general_ci::*;
-pub use utf8mb4_unicode_ci::*;
+pub use utf8mb4_uca::*;
 
 use super::{charset::*, Collator};
 use crate::codec::Result;
@@ -43,6 +43,8 @@ mod tests {
             (Collation::Latin1Bin, 4),
             (Collation::GbkBin, 5),
             (Collation::GbkChineseCi, 6),
+            (Collation::Utf8Mb40900AiCi, 7),
+            (Collation::Utf8Mb40900Bin, 8),
         ];
         let cases = vec![
             // (sa, sb, [Utf8Mb4Bin, Utf8Mb4BinNoPadding, Utf8Mb4GeneralCi, Utf8Mb4UnicodeCi,
@@ -58,6 +60,8 @@ mod tests {
                     Ordering::Equal,
                     Ordering::Equal,
                     Ordering::Equal,
+                    Ordering::Equal,
+                    Ordering::Equal,
                 ],
             ),
             (
@@ -71,6 +75,8 @@ mod tests {
                     Ordering::Equal,
                     Ordering::Equal,
                     Ordering::Equal,
+                    Ordering::Less,
+                    Ordering::Less,
                 ],
             ),
             (
@@ -84,6 +90,8 @@ mod tests {
                     Ordering::Greater,
                     Ordering::Greater,
                     Ordering::Equal,
+                    Ordering::Less,
+                    Ordering::Greater,
                 ],
             ),
             (
@@ -97,6 +105,8 @@ mod tests {
                     Ordering::Greater,
                     Ordering::Greater,
                     Ordering::Greater,
+                    Ordering::Greater,
+                    Ordering::Greater,
                 ],
             ),
             (
@@ -110,6 +120,8 @@ mod tests {
                     Ordering::Less,
                     Ordering::Less,
                     Ordering::Less,
+                    Ordering::Less,
+                    Ordering::Less,
                 ],
             ),
             (
@@ -123,6 +135,8 @@ mod tests {
                     Ordering::Less,
                     Ordering::Less,
                     Ordering::Less,
+                    Ordering::Equal,
+                    Ordering::Less,
                 ],
             ),
             (
@@ -136,6 +150,8 @@ mod tests {
                     Ordering::Less,
                     Ordering::Less,
                     Ordering::Less,
+                    Ordering::Greater,
+                    Ordering::Less,
                 ],
             ),
             (
@@ -149,6 +165,8 @@ mod tests {
                     Ordering::Greater,
                     Ordering::Less,
                     Ordering::Less,
+                    Ordering::Equal,
+                    Ordering::Greater,
                 ],
             ),
             (
@@ -162,6 +180,8 @@ mod tests {
                     Ordering::Less,
                     Ordering::Greater,
                     Ordering::Greater,
+                    Ordering::Less,
+                    Ordering::Less,
                 ],
             ),
             (
@@ -175,6 +195,8 @@ mod tests {
                     Ordering::Less,
                     Ordering::Less,
                     Ordering::Less,
+                    Ordering::Less,
+                    Ordering::Less,
                 ],
             ),
         ];
@@ -231,6 +253,8 @@ mod tests {
             (Collation::Latin1Bin, 4),
             (Collation::GbkBin, 5),
             (Collation::GbkChineseCi, 6),
+            (Collation::Utf8Mb40900AiCi, 7),
+            (Collation::Utf8Mb40900Bin, 8),
         ];
         let cases = vec![
             // (str, [Utf8Mb4Bin, Utf8Mb4BinNoPadding, Utf8Mb4GeneralCi, Utf8Mb4UnicodeCi, Latin1,
@@ -245,6 +269,8 @@ mod tests {
                     vec![0x61],
                     vec![0x61],
                     vec![0x41],
+                    vec![0x1C, 0x47],
+                    vec![0x61],
                 ],
             ),
             (
@@ -257,6 +283,8 @@ mod tests {
                     vec![0x41],
                     vec![0x41],
                     vec![0x41],
+                    vec![0x1C, 0x47, 0x2, 0x9],
+                    vec![0x41, 0x20],
                 ],
             ),
             (
@@ -269,6 +297,8 @@ mod tests {
                     vec![0x41],
                     vec![0x41],
                     vec![0x41],
+                    vec![0x1C, 0x47],
+                    vec![0x41],
                 ],
             ),
             (
@@ -281,6 +311,8 @@ mod tests {
                     vec![0xF0, 0x9F, 0x98, 0x83],
                     vec![0x3F],
                     vec![0x3F],
+                    vec![0x15, 0xFE],
+                    vec![0xF0, 0x9F, 0x98, 0x83],
                 ],
             ),
             (
@@ -321,6 +353,17 @@ mod tests {
                         0x46, 0x4f, 0x4f, 0x20, 0x3f, 0x20, 0x42, 0x41, 0x52, 0x20, 0x3f, 0x20,
                         0x42, 0x41, 0x5a, 0x20, 0x3f, 0x20, 0x51, 0x55, 0x58,
                     ],
+                    vec![
+                        0x1C, 0xE5, 0x1D, 0xDD, 0x1D, 0xDD, 0x2, 0x9, 0x5, 0x84, 0x2, 0x9, 0x1C,
+                        0x60, 0x1C, 0x47, 0x1E, 0x33, 0x2, 0x9, 0xE, 0xF0, 0x2, 0x9, 0x1C, 0x60,
+                        0x1C, 0x47, 0x1F, 0x21, 0x2, 0x9, 0x9, 0x1B, 0x2, 0x9, 0x1E, 0x21, 0x1E,
+                        0xB5, 0x1E, 0xFF,
+                    ],
+                    vec![
+                        0x46, 0x6F, 0x6F, 0x20, 0xC2, 0xA9, 0x20, 0x62, 0x61, 0x72, 0x20, 0xF0,
+                        0x9D, 0x8C, 0x86, 0x20, 0x62, 0x61, 0x7A, 0x20, 0xE2, 0x98, 0x83, 0x20,
+                        0x71, 0x75, 0x78,
+                    ],
                 ],
             ),
             (
@@ -336,6 +379,11 @@ mod tests {
                     vec![0xEF, 0xB7, 0xBB],
                     vec![0x3f],
                     vec![0x3f],
+                    vec![
+                        0x23, 0x25, 0x23, 0x9C, 0x2, 0x9, 0x23, 0x25, 0x23, 0x9C, 0x23, 0xB, 0x23,
+                        0x9C, 0x23, 0xB1,
+                    ],
+                    vec![0xEF, 0xB7, 0xBB],
                 ],
             ),
             (
@@ -348,6 +396,8 @@ mod tests {
                     vec![0xE4, 0xB8, 0xAD, 0xE6, 0x96, 0x87],
                     vec![0xD6, 0xD0, 0xCE, 0xC4],
                     vec![0xD3, 0x21, 0xC1, 0xAD],
+                    vec![0xFB, 0x40, 0xCE, 0x2D, 0xFB, 0x40, 0xE5, 0x87],
+                    vec![0xE4, 0xB8, 0xAD, 0xE6, 0x96, 0x87],
                 ],
             ),
         ];

diff --git a/.../collation/collator/utf8mb4_unicode_ci.rs → ...llation/collator/utf8mb4_uca/data_0400.rs b/.../collation/collator/utf8mb4_unicode_ci.rs → ...llation/collator/utf8mb4_uca/data_0400.rs
@@ -1,20 +1,22 @@
 // Copyright 2020 TiKV Project Authors. Licensed under Apache-2.0.
 
-use super::*;
+// Created from https://www.unicode.org/Public/UCA/4.0.0/allkeys-4.0.0.txt
 
-/// Collator for `utf8mb4_unicode_ci` collation with padding behavior (trims
-/// right spaces).
-#[derive(Debug)]
-pub struct CollatorUtf8Mb4UnicodeCi;
+use super::{super::PADDING_SPACE, UnicodeVersion};
 
-impl Collator for CollatorUtf8Mb4UnicodeCi {
-    type Charset = CharsetUtf8mb4;
-    type Weight = u128;
+static LONG_RUNE: u64 = 0xFFFD;
 
-    const IS_CASE_INSENSITIVE: bool = true;
+#[derive(Debug)]
+pub struct Unicode0400 {}
+
+impl UnicodeVersion for Unicode0400 {
+    #[inline]
+    fn preprocess(s: &str) -> &str {
+        s.trim_end_matches(PADDING_SPACE)
+    }
 
     #[inline]
-    fn char_weight(ch: char) -> Self::Weight {
+    fn char_weight(ch: char) -> u128 {
         let r = ch as usize;
         if r > 0xFFFF {
             return 0xFFFD;
@@ -27,87 +29,8 @@ impl Collator for CollatorUtf8Mb4UnicodeCi {
 
         u as u128
     }
-
-    #[inline]
-    fn write_sort_key<W: BufferWriter>(writer: &mut W, bstr: &[u8]) -> Result<usize> {
-        let s = str::from_utf8(bstr)?.trim_end_matches(PADDING_SPACE);
-        let mut n = 0;
-        for ch in s.chars() {
-            let mut weight = Self::char_weight(ch);
-            while weight != 0 {
-                writer.write_u16_be((weight & 0xFFFF) as u16)?;
-                n += 1;
-                weight >>= 16
-            }
-        }
-        Ok(n * std::mem::size_of::<u16>())
-    }
-
-    #[inline]
-    fn sort_compare(a: &[u8], b: &[u8]) -> Result<Ordering> {
-        let mut ca = str::from_utf8(a)?.trim_end_matches(PADDING_SPACE).chars();
-        let mut cb = str::from_utf8(b)?.trim_end_matches(PADDING_SPACE).chars();
-        let mut an = 0;
-        let mut bn = 0;
-
-        loop {
-            if an == 0 {
-                for ach in &mut ca {
-                    an = Self::char_weight(ach);
-                    if an != 0 {
-                        break;
-                    }
-                }
-            }
-
-            if bn == 0 {
-                for bch in &mut cb {
-                    bn = Self::char_weight(bch);
-                    if bn != 0 {
-                        break;
-                    }
-                }
-            }
-
-            if an == 0 || bn == 0 {
-                return Ok(an.cmp(&bn));
-            }
-
-            if an == bn {
-                an = 0;
-                bn = 0;
-                continue;
-            }
-
-            while an != 0 && bn != 0 {
-                if (an ^ bn) & 0xFFFF == 0 {
-                    an >>= 16;
-                    bn >>= 16;
-                } else {
-                    return Ok((an & 0xFFFF).cmp(&(bn & 0xFFFF)));
-                }
-            }
-        }
-    }
-
-    #[inline]
-    fn sort_hash<H: Hasher>(state: &mut H, bstr: &[u8]) -> Result<()> {
-        let s = str::from_utf8(bstr)?.trim_end_matches(PADDING_SPACE);
-        for ch in s.chars() {
-            let mut weight = Self::char_weight(ch);
-            while weight != 0 {
-                (weight & 0xFFFF).hash(state);
-                weight >>= 16;
-            }
-        }
-        Ok(())
-    }
 }
 
-// Created from https://www.unicode.org/Public/UCA/4.0.0/allkeys-4.0.0.txt
-
-static LONG_RUNE: u64 = 0xFFFD;
-
 #[inline]
 fn map_long_rune(r: usize) -> u128 {
     match r {