State difference at mainnet's block 3474317

[roman-khimov]

block 3474317: value mismatch for key d471621ecc3fb072a3a754254ba00b8ed0affa2f6c6f636b7570735fa48c1ae6617e50930047e5a385624b15f8872c8802b514030001: 001f80040203b814030203b51403020500e40b5402820102010d000500e40b540200 vs 001c800400000203b51403020500e40b5402820102010d000500e40b540200
block 3474317: value mismatch for key d471621ecc3fb072a3a754254ba00b8ed0affa2f6c6f636b7570735fa48c1ae6617e50930047e5a385624b15f8872c8802b614030001: 001f800402031115030203b61403020500e40b5402820102010d000500e40b540200 vs 001c800400000203b61403020500e40b5402820102010d000500e40b540200
block 3474317: value mismatch for key d471621ecc3fb072a3a754254ba00b8ed0affa2f6c6f636b7570735fa48c1ae6617e50930047e5a385624b15f8872c8802b714030001: 001c800400000203b71403020500e40b5402820102010d000500e40b540200 vs 001f80040203b814030203b71403020500e40b5402820102010d000500e40b540200
block 3474317: value mismatch for key d471621ecc3fb072a3a754254ba00b8ed0affa2f6c6f636b7570735fa48c1ae6617e50930047e5a385624b15f8872c8802b814030001: 001d80040201000203b81403020500e40b5402820102010d000500e40b540200 vs 001f800402031115030203b81403020500e40b5402820102010d000500e40b540200

It comes from contract 0x2ffaafd08e0ba04b2554a7a372b03fcc1e6271d4 (which I wasn't able to find the source for, unfortunately). And it relates to transferWithLockupPeriod invocations of it, one of which actually differs a bit in terms of gas spent (though it does successfully transfer the asset), C#:

{
   "jsonrpc" : "2.0",
   "id" : 1,
   "result" : {
      "executions" : [
         {
            "contract" : "0x9ebf5fab541b10a9aedecd503c118c15c0e70f58",
            "trigger" : "Application",
            "gas_consumed" : "6.849",
            "vmstate" : "HALT",
            "notifications" : [
               {
                  "state" : {
                     "value" : [
                        {
                           "type" : "ByteArray",
                           "value" : "7472616e73666572"
                        },
                        {
                           "value" : "7d2185c97fa43cb41c5617941c6b68d146a84ae5",
                           "type" : "ByteArray"
                        },
                        {
                           "value" : "a48c1ae6617e509347e5a385624b15f8872c8802",
                           "type" : "ByteArray"
                        },
                        {
                           "type" : "ByteArray",
                           "value" : "00e40b5402"
                        }
                     ],
                     "type" : "Array"
                  },
                  "contract" : "0x2ffaafd08e0ba04b2554a7a372b03fcc1e6271d4"
               }
            ],
            "stack" : [
               {
                  "value" : "1",
                  "type" : "Integer"
               }
            ]
         }
      ],
      "txid" : "0x0fbe86245909655d4815962cc2092267303993133e1f133a911dcf70a0603aeb"
   }
}

neo-go:

{
   "result" : {
      "txid" : "0x0fbe86245909655d4815962cc2092267303993133e1f133a911dcf70a0603aeb",
      "executions" : [
         {
            "gas_consumed" : "6.842",
            "trigger" : "Application",
            "contract" : "0x9ebf5fab541b10a9aedecd503c118c15c0e70f58",
            "vmstate" : "HALT",
            "notifications" : [
               {
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "type" : "ByteArray",
                           "value" : "7472616e73666572"
                        },
                        {
                           "type" : "ByteArray",
                           "value" : "7d2185c97fa43cb41c5617941c6b68d146a84ae5"
                        },
                        {
                           "type" : "ByteArray",
                           "value" : "a48c1ae6617e509347e5a385624b15f8872c8802"
                        },
                        {
                           "value" : "00e40b5402",
                           "type" : "ByteArray"
                        }
                     ]
                  },
                  "contract" : "0x2ffaafd08e0ba04b2554a7a372b03fcc1e6271d4"
               }
            ],
            "stack" : [
               {
                  "type" : "Integer",
                  "value" : "1"
               }
            ]
         }
      ]
   },
   "id" : 1,
   "jsonrpc" : "2.0"
}

I thought that it might be because of different key order in Storage.Find results, but either it's not the case or the order is still not quite right after this change (it doesn't solve the problem):

--- a/pkg/core/interop_neo.go
+++ b/pkg/core/interop_neo.go
@@ -4,6 +4,7 @@ import (
        "errors"
        "fmt"
        "math"
+       "sort"
        "strings"
 
        "github.com/nspcc-dev/neo-go/pkg/core/state"
@@ -450,13 +451,18 @@ func (ic *interopContext) storageFind(v *vm.VM) error {
                return err
        }
 
-       filteredMap := vm.NewMapItem()
-       for k, v := range siMap {
+       keys := make([]string, 0, len(siMap))
+       for k := range siMap {
                if strings.HasPrefix(k, prefix) {
-                       filteredMap.Add(vm.NewByteArrayItem([]byte(k)),
-                               vm.NewByteArrayItem(v.Value))
+                       keys = append(keys, k)
                }
        }
+       sort.Strings(keys)
+       filteredMap := vm.NewMapItem()
+       for _, k := range keys {
+               filteredMap.Add(vm.NewByteArrayItem([]byte(k)),
+                       vm.NewByteArrayItem(siMap[k].Value))
+       }
 
        item := vm.NewMapIterator(filteredMap)
        v.Estack().PushVal(item)

The key in question is easily parsed like this (notice "lockup_" prefix and some sort of sequencing is being done here in the fourth component (b51403/b61403/b71403/b81403 in different keys)):

d471621ecc3fb072a3a754254ba00b8ed0affa2f 6c6f636b7570735f a48c1ae6617e50930047e5a385624b15f8872c8802 b51403 0001 

Values:

                key?       amount         map             amount
001c 80 04 0000 0203b51403 020500e40b5402 82 01 02010d 000500e40b540200
001f 80 04 0203b81403 0203b51403 020500e40b5402 82 01 02010d 000500e40b540200

[roman-khimov]

And BTW, this difference in 7 instructions may be related to one excessive loop iteration here:

4536     DUPFROMALTSTACK                                                                                                                                                                                        
4537     PUSH9                                                                                                                                                                                                  
4538     PICKITEM                                                                                                                                                                                               
4539     DUPFROMALTSTACK                                                                                                                                                                                        
4540     PUSH6                                                                                                                                                                                                  
4541     PICKITEM                                                                                                                                                                                               
4542     PUSH2                                                                                                                                                                                                  
4543     PICKITEM                                                                                                                                                                                               
4544     NUMEQUAL                                                                                                                                                                                               
4545     JMPIF              4611 (66/4200)                                                                                                                                                                      
4548     DUPFROMALTSTACK                                                                                                                                                                                        
4549     PUSH6                                                                                                                                                                                                  
4550     PICKITEM                                                                                                                                                                                               
4551     PUSH3                                                                                                                                                                                                  
4552     PICKITEM                                                                                                                                                                                               
4553     DUPFROMALTSTACK                                                                                                                                                                                        
4554     PUSH14                                                                                                                                                                                                 
4555     PICKITEM                                                                                                                                                                                               
4556     HASKEY                                                                                                                                                                                                 
4557     JMPIFNOT           4592 (35/2300)                                                                                                                                                                      
4560     DUPFROMALTSTACK                                                                                                                                                                                        
4561     PUSH10                                                                                                                                                                                                 
4562     PICKITEM                                                                                                                                                                                               
4563     DUPFROMALTSTACK                                                                                                                                                                                        
4564     PUSH14                                                                                                                                                                                                 
4565     PICKITEM                                                                                                                                                                                               
4566     DUPFROMALTSTACK                                                                                                                                                                                        
4567     PUSH6                                                                                                                                                                                                  
4568     PICKITEM                                                                                                                                                                                               
4569     PUSH3                                                                                                                                                                                                  
4570     PICKITEM                                                                                                                                                                                               
4571     DUPFROMALTSTACK                                                                                                                                                                                        
4572     PUSH14                                                                                                                                                                                                 
4573     PICKITEM                                                                                                                                                                                               
4574     PICKITEM                                                                                                                                                                                               
4575     SETITEM                                                                                                                                                                                                
4576     DUPFROMALTSTACK                                                                                                                                                                                        
4577     PUSH9                                                                                                                                                                                                  
4578     PICKITEM                                                                                                                                                                                               
4579     DUPFROMALTSTACK                                                                                                                                                                                        
4580     PUSH10                                                                                                                                                                                                 
4581     PICKITEM                                                                                                                                                                                               
4582     DUPFROMALTSTACK                                                                                                                                                                                        
4583     PUSH14                                                                                                                                                                                                 
4584     PICKITEM                                                                                                                                                                                               
4585     PICKITEM                                                                                                                                                                                               
4586     ADD                                                                                                                                                                                                    
4587     DUPFROMALTSTACK                                                                                                                                                                                        
4588     PUSH9                                                                                                                                                                                                  
4589     PUSH2                                                                                                                                                                                                  
4590     ROLL                                                                                                                                                                                                   
4591     SETITEM                                                                                                                                                                                                
4592     DUPFROMALTSTACK                                                                                                                                                                                        
4593     PUSH14                                                                                                                                                                                                 
4594     PICKITEM                                                                                                                                                                                               
4595     PUSH1                                                                                                                                                                                                  
4596     ADD                                                                                                                                                                                                    
4597     DUPFROMALTSTACK                                                                                                                                                                                        
4598     PUSH14                                                                                                                                                                                                 
4599     PUSH2                                                                                                                                                                                                  
4600     ROLL                                                                                                                                                                                                   
4601     SETITEM                                                                                                                                                                                                
4602     DUPFROMALTSTACK                                                                                                                                                                                        
4603     PUSH14                                                                                                                                                                                                 
4604     PICKITEM                                                                                                                                                                                               
4605     PUSHBYTES1         1f ("\x1f")                                                                                                                                                                         
4607     LTE                                                                                                                                                                                                    
4608     JMPIF              4536 (-72/b8ff)                                                                                                                                                                     
4611     DUPFROMALTSTACK                                                                                                                                                                                        
...

There are exactly seven gas-counted instructions between 4536 and 4545.

[roman-khimov]

Contract itself for quick reference:

{
   "id" : 1,
   "result" : {
      "email" : "[email protected]",
      "author" : "Time Innovation Pte. Ltd.",
      "description" : "https://timeinnovation.io",
      "version" : 0,
      "name" : "TimeCoin",
      "properties" : {
         "dynamic_invoke" : false,
         "storage" : true,
         "is_payable" : false
      },
      "parameters" : [
         "String",
         "Array"
      ],
      "hash" : "0x2ffaafd08e0ba04b2554a7a372b03fcc1e6271d4",
      "returntype" : "ByteArray",
      "code_version" : "1.0.0",
      "script" : "54c56b6c766b00527ac46c766b51527ac46168164e656f2e52756e74696d652e4765745472696767657260907c907c9e6316016165d9016a52527ac4006a53527ac4622a006a52c36a53c3c36a00c3876412006a00c36a51c3617c650b02616c75666a53c351936a53527ac46a53c36a52c3c09f63d1ff61652c196a52527ac4006a53527ac4622a006a52c36a53c3c36a00c3876412006a00c36a51c3617c657b19616c75666a53c351936a53527ac46a53c36a52c3c09f63d1ff616586166a52527ac4006a53527ac4622a006a52c36a53c3c36a00c3876412006a00c36a51c3617c659816616c75666a53c351936a53527ac46a53c36a52c3c09f63d1ff61654d066a52527ac4006a53527ac4622a006a52c36a53c3c36a00c3876412006a00c36a51c3617c654606616c75666a53c351936a53527ac46a53c36a52c3c09f63d1ff6227006168164e656f2e52756e74696d652e47657454726967676572630b006165c812616c756600616c756600c56b0854696d65436f696e616c756600c56b03544d43616c756600c56b58616c756600c56b137b224e45502d35222c20224e45502d3130227d616c756651c56b6c766b00527ac46a00c3c063080000616c75666a00c36168174e656f2e52756e74696d652e446573657269616c697a65616c756651c56b6c766b00527ac46a00c36168154e656f2e52756e74696d652e53657269616c697a65616c756600c56b54c57600610e636f6e74726163744465706c6f79c47651610f636f6e74726163744d696772617465c47652611275706461746541646d696e41646472657373c47653610d7365744f7463417070726f7665c4616c756653c56b6c766b00527ac46c766b51527ac46165b311635300384f7065726174696f6e206661696c65642e20636f6e747261637420696e766f6b6572206973206e6f742041646d696e697374726174696f7261680f4e656f2e52756e74696d652e4c6f6700616c75666a00c3610e636f6e74726163744465706c6f7987640b006165bb00616c75666a00c3610f636f6e74726163744d69677261746587641f006a51c359617c65cc1063080000616c75666a51c361655c02616c75666a00c3611275706461746541646d696e41646472657373876421006a51c351617c65951063080000616c75666a51c300c361658d01616c75666a00c3610d7365744f7463417070726f766587642f006a51c352617c65611063080000616c75666a51c300c36a51c351c36a52527ac46a52c3617c651b03616c756600616c756651c56b61651411644f0034436f6e74726163744465706c6f79206973206661696c65642e20636f6e747261637420616c7265616479206465706c6f7965642e61680f4e656f2e52756e74696d652e4c6f6700616c75666165fa106a00527ac46168164e656f2e53746f726167652e476574436f6e746578746165191d6a00c3615272680f4e656f2e53746f726167652e5075746168164e656f2e53746f726167652e476574436f6e746578746165fd1c0800006c98fdf0e107615272680f4e656f2e53746f726167652e50757461147d2185c97fa43cb41c5617941c6b68d146a84ae561655b007561147d2185c97fa43cb41c5617941c6b68d146a84ae50800006c98fdf0e107617c651717751a436f6e74726163744465706c6f7920697320737563636573732e61680f4e656f2e52756e74696d652e4c6f6751616c756651c56b6c766b00527ac46a00c3c001149c634c003155706461746541646d696e41646472657373206661696c65642e206e65774164647265737320697320696e76616c69642e61680f4e656f2e52756e74696d652e4c6f6700616c75666168164e656f2e53746f726167652e476574436f6e746578746165c71b6a00c3615272680f4e656f2e53746f726167652e50757451616c75665ac56b6c766b00527ac461682d53797374656d2e457865637574696f6e456e67696e652e476574457865637574696e675363726970744861736861681a4e656f2e426c6f636b636861696e2e476574436f6e74726163746168164e656f2e436f6e74726163742e4765745363726970746a00c300c36a51527ac46a51c3907c907c9e63080000616c75666a00c351c36a52527ac46a00c352c36a53527ac46a00c353c36a54527ac46a00c354c36a55527ac46a00c355c36a56527ac46a00c356c36a57527ac46a00c357c36a58527ac46a00c358c36a59527ac46a51c36a52c36a53c36a54c36a55c36a56c36a57c36a58c36a59c361587951795a727551727557795279597275527275567953795872755372755579547957727554727568144e656f2e436f6e74726163742e4d6967726174657551616c756653c56b6c766b00527ac46c766b51527ac46a00c3c001149c634200275365744f7463417070726f76652066616c65642e206164647265737320697320696e76616c696461680f4e656f2e52756e74696d652e4c6f6700616c75666168164e656f2e53746f726167652e476574436f6e7465787461655f1a617c656e1a6a51c363070000620400516a52527ac46a00c36a52c361527265781a51616c756600c56b51c576006112737570706f727465645374616e6461726473c4616c756652c56b6c766b00527ac46c766b51527ac46a00c36112737570706f727465645374616e646172647387640b006165f8f9616c756600616c756653c56b6c766b00527ac46c766b51527ac46a00c3c001149c633f002453657442616c616e63652066616c65642e206164647265737320697320696e76616c696461680f4e656f2e52756e74696d652e4c6f6700616c75666168164e656f2e53746f726167652e476574436f6e7465787461654c19617c657b196a52527ac46a51c300a16411006a52c36a00c3617c65d5196212006a52c36a00c36a51c361527265771951616c75665bc56b6c766b00527ac4006a51527ac46165930c6165e60c6a52527ac46a52c300c30164956a52c351c3936a53527ac46a52c352c36a54527ac46168164e656f2e53746f726167652e476574436f6e746578746165d1186a00c37e617c68104e656f2e53746f726167652e46696e646a55527ac462c2016a55c36168124e656f2e4974657261746f722e56616c75656168174e656f2e52756e74696d652e446573657269616c697a656a56527ac46a56c351c36a53c39f6443006168164e656f2e53746f726167652e476574436f6e74657874616551186a00c37e6a56c351c37e617c68124e656f2e53746f726167652e44656c657465623f016a56c351c36a53c39c641b01006a57527ac46a56c353c3cc6a58527ac4006a59527ac46241006a58c36a59c3c36a5a527ac46a5ac36a54c3a1640f006a56c353c36a5ac3ca6215006a57c36a56c353c36a5ac3c3936a57527ac46a59c351936a59527ac46a59c36a58c3c09f63baff6a57c36a56c352c39e64ac006a56c36a57c3527cc46a56c352c300a16443006168164e656f2e53746f726167652e476574436f6e74657874616583176a00c37e6a56c351c37e617c68124e656f2e53746f726167652e44656c6574656259006168164e656f2e53746f726167652e476574436f6e74657874616543176a00c37e6a56c351c37e6a56c36168154e656f2e52756e74696d652e53657269616c697a65615272680f4e656f2e53746f726167652e5075746a56c352c300a16311006a51c36a56c352c3936a51527ac46a55c36168114e656f2e4974657261746f722e4e657874632afe6a51c3009f64080000616c75666a51c3616c75660114c56b6c766b00527ac46c766b51527ac46a51c300a164080000616c75666165200a6165730a6a52527ac46a52c352c36a53527ac46a52c300c30164956a52c351c3936a54527ac4006a55527ac4006a56527ac4c76a57527ac46168164e656f2e53746f726167652e476574436f6e7465787461654c166a00c37e617c68104e656f2e53746f726167652e46696e646a58527ac46282016a58c36168124e656f2e4974657261746f722e56616c75656168174e656f2e52756e74696d652e446573657269616c697a656a59527ac46a59c351c36a54c39f6443006168164e656f2e53746f726167652e476574436f6e746578746165cc156a00c37e6a59c351c37e617c68124e656f2e53746f726167652e44656c65746562ff006a59c351c36a54c39c64bb000075006a5a527ac46a59c353c3cc6a5b527ac4006a5c527ac46241006a5bc36a5cc3c36a5d527ac46a5dc36a53c3a1640f006a59c353c36a5dc3ca6215006a5ac36a59c353c36a5dc3c3936a5a527ac46a5cc351936a5c527ac46a5cc36a5bc3c09f63baff6a59c36a5ac3527cc46a59c352c300a16443006168164e656f2e53746f726167652e476574436f6e74657874616508156a00c37e6a59c351c37e617c68124e656f2e53746f726167652e44656c657465623b006a55c3009c630f006a55c36a59c351c3a0640d006a59c351c36a55527ac46a56c36a59c352c3936a56527ac46a57c36a59c351c36a59c3c46a58c36168114e656f2e4974657261746f722e4e657874636afe6a51c300a164080051616c75666a51c36a56c3a264fb016a57c3cc6a5b527ac4006a5c527ac46254006a5bc36a5cc3c36a5e527ac46168164e656f2e53746f726167652e476574436f6e74657874616541146a00c37e6a5ec37e617c68124e656f2e53746f726167652e44656c6574656a5cc351936a5c527ac46a5cc36a5bc3c09f63a7ff51616c75666a57c36a55c3c36a5f527ac46a51c36a5fc352c39400a26451006a51c36a5fc352c3946a51527ac46168164e656f2e53746f726167652e476574436f6e746578746165c4136a00c37e6a5fc351c37e617c68124e656f2e53746f726167652e44656c6574656214016a5fc37652c36a51c394527cc46a5fc353c3cc6a60527ac4006a0111527ac4628b006a51c36a5fc353c36a60c36a0111c3c3c39400a2642b006a51c36a5fc353c36a60c36a0111c3c3c3946a51527ac46a5fc353c36a60c36a0111c3c3ca6238006a5fc353c36a0112527ac46a60c36a0111c3c36a0113527ac46a0112c36a0113c36a0112c36a0113c3c36a51c394c4006a51527ac46a51c300a1631b006a0111c351936a0111527ac46a0111c36a60c3c09f636fff6168164e656f2e53746f726167652e476574436f6e746578746165ce126a00c37e6a5fc351c37e6a5fc36168154e656f2e52756e74696d652e53657269616c697a65615272680f4e656f2e53746f726167652e50757451616c75666a5fc300c300946a55527ac46a55c3009e6376fe51616c756656c56b6c766b00527ac46c766b51527ac46c766b52527ac46a52c361652e067600c36a53527ac47651c36a54527ac452c36a55527ac46a00c36a51c36a53c36a54c36a55c3615479517956727551727553795279557275527275650700616c75665fc56b6c766b00527ac46c766b51527ac46c766b52527ac46c766b53527ac46c766b54527ac46a52c30164956a53c3936a55527ac46154c56a56527ac46168164e656f2e53746f726167652e476574436f6e746578746165bc116a00c37e6a55c37e617c680f4e656f2e53746f726167652e4765746a57527ac46a57c3c06428006a57c36168174e656f2e52756e74696d652e446573657269616c697a656a56527ac4621c006154c57600007cc4766a55c3517cc47600527cc46a56527ac46168164e656f2e53746f726167652e476574436f6e7465787461653a116a00c37e617c68104e656f2e53746f726167652e46696e646a58527ac462cb006a58c36168124e656f2e4974657261746f722e56616c75656168174e656f2e52756e74696d652e446573657269616c697a656a5b527ac46a5bc351c36a55c3a26388006a5bc300c3009c630f006a55c36a5bc300c39f6472006a56c36a5bc300c30094007cc46a5bc36a55c3007cc46168164e656f2e53746f726167652e476574436f6e7465787461658e106a00c37e6a5bc351c37e6a5bc36168154e656f2e52756e74696d652e53657269616c697a65615272680f4e656f2e53746f726167652e507574621d006a58c36168114e656f2e4974657261746f722e4e6578746321ff006a59527ac4c76a5a527ac46a56c37652c36a51c393527cc46a56c352c36a51c39c6410006a5ac36a54c36a51c3c46296006a56c353c36a54c3cb642a006a56c353c36a5c527ac46a54c36a5d527ac46a5cc36a5dc36a5cc36a5dc3c36a51c393c4620f006a56c353c36a54c36a51c3c4516a5e527ac46245006a59c36a56c352c39c6342006a56c353c36a5ec3cb6423006a5ac36a5ec36a56c353c36a5ec3c3c46a59c36a5ac36a5ec3c3936a59527ac46a5ec351936a5e527ac46a5ec3011fa163b8ff6a56c36a5ac3537cc46168164e656f2e53746f726167652e476574436f6e7465787461654d0f6a00c37e6a55c37e6a56c36168154e656f2e52756e74696d652e53657269616c697a65615272680f4e656f2e53746f726167652e50757451616c756656c56b6c766b00527ac46c766b51527ac4c76a52527ac46168164e656f2e53746f726167652e476574436f6e746578746165dd0e6a00c37e617c68104e656f2e53746f726167652e46696e646a53527ac4006a54527ac4625c006a53c36168124e656f2e4974657261746f722e56616c75656168174e656f2e52756e74696d652e446573657269616c697a656a55527ac46a51c36a55c351c3a1640f006a52c36a55c351c36a55c3c46a54c351936a54527ac46a53c36168114e656f2e4974657261746f722e4e6578746390ff6a52c3616c756651c56b6c766b00527ac46a00c3c001149c635400395665726966794973417070726f7665644f74632066616c65642e202866726f6d207c20746f29206164647265737320697320696e76616c696461680f4e656f2e52756e74696d652e4c6f6700616c75666168164e656f2e53746f726167652e476574436f6e746578746165cb0d617c65da0d6a00c3617c65880e00a164080000616c756651616c756652c56b6c766b00527ac46c766b51527ac46a00c3c06a51c3907c907c9e63080051616c756617496e76616c696420417267756d656e74204c656e67746861680f4e656f2e52756e74696d652e4c6f6700616c756600c56b61658a006453006168164e656f2e53746f726167652e476574436f6e746578746165cf0c617c680f4e656f2e53746f726167652e4765746168184e656f2e52756e74696d652e436865636b5769746e657373616c756661147d2185c97fa43cb41c5617941c6b68d146a84ae56168184e656f2e52756e74696d652e436865636b5769746e657373616c756600c56b6168164e656f2e53746f726167652e476574436f6e7465787461655d0c617c680f4e656f2e53746f726167652e47657400a0616c756600c56b6168184e656f2e426c6f636b636861696e2e4765744865696768746168184e656f2e426c6f636b636861696e2e4765744865616465726168174e656f2e4865616465722e47657454696d657374616d70616c75665bc56b6c766b00527ac453c56a51527ac45dc5760000c47651011fc47652013bc47653015ac476540178c47655029700c4765602b500c4765702d400c4765802f300c47659021101c4765a023001c4765b024e01c4765c026d01c46a52527ac45dc5760000c47651011fc47652013cc47653015bc476540179c47655029800c4765602b600c4765702d500c4765802f400c47659021201c4765a023101c4765b024f01c4765c026e01c46a53527ac46a00c30500f791770e9303805101966a54527ac46a54c303b13a02966a55527ac46a54c36a55c303b13a0295946a54527ac46a54c303ac8e00966a56527ac46a56c354907c907c9e630900536a56527ac46a54c36a56c303ac8e0095946a54527ac46a54c302b505966a57527ac46a54c36a57c302b50595946a54527ac46a54c3026d01966a58527ac46a58c354907c907c9e630900536a58527ac46a51c3006a55c3029001956a56c3016495936a57c35495936a58c3935193c46a54c36a58c3026d0195946a54527ac46a58c353907c907c9e631c006a57c30118907c907c9e630b006a56c3539c62080051620400006309006a52c36206006a53c36a59527ac46a54c356996a5a527ac4620d006a5ac351936a5a527ac46a54c36a59c36a5ac3c3a263ebff6a51c3516a5ac3c46a51c3526a54c36a59c36a5ac35194c3945193c46a51c3616c756600c56b52c5760061187472616e73666572576974684c6f636b7570506572696f64c47651610e6765744c6f636b75704172726179c4616c75665ec56b6c766b00527ac46c766b51527ac46a00c361187472616e73666572576974684c6f636b7570506572696f6487645c006a51c354617c6524fc63080000616c75666a51c300c36a51c351c36a52527ac46a51c352c36a53527ac46a51c353c36a54527ac46a52c36a53c36a54c361537951795572755172755279527954727552727565fd00616c75666a00c3610e6765744c6f636b757041727261798764dd006a51c351617c65b4fb63080000616c75666a51c300c36165befc616511fd6a55527ac46a55c300c30164956a55c351c3936a56527ac4c76a57527ac46a56c3617c6507facd6a58527ac4006a59527ac46277006a58c36a59c3c36a5a527ac4006a5b527ac4624a006a5ac353c3cc6a5bc3c36a5c527ac46a5ac351c30164956a5cc3936a5d527ac46a57c36a5dc352c576006a5dc3c476516a5ac353c36a5cc3c30094c4c46a5bc351936a5b527ac46a5bc36a5ac353c3ccc09f63aeff6a59c351936a59527ac46a59c36a58c3c09f6384ff6a57c3cd616c756600616c756655c56b6c766b00527ac46c766b51527ac46c766b52527ac46c766b53527ac46165d6fb6a54527ac46a53c36a54c3a1645100365472616e73666572576974684c6f636b7570506572696f64206661696c65642e20706572696f64203c3d2063757272656e7454696d6561680f4e656f2e52756e74696d652e4c6f6700616c75666a00c36a51c36a52c36a53c361537951795572755172755279527954727552727565f504616c756600c56b56c5760061046e616d65c47651610673796d626f6cc47652610b746f74616c537570706c79c476536108646563696d616c73c47654610962616c616e63654f66c4765561087472616e73666572c4616c756600c56b616572e7616c756600c56b616577e7616c756600c56b616577e7616c756654c56b6c766b00527ac46c766b51527ac46a00c361046e616d6587640b006165c0ff616c75666a00c36108646563696d616c7387640b006165bdff616c75666a00c3610b746f74616c537570706c7987640b0061659e00616c75666a00c3610673796d626f6c87640b0061657fff616c75666a00c3610962616c616e63654f66876421006a51c351617c6533f963080000616c75666a51c300c361658e00616c75666a00c361087472616e7366657287643d006a51c353617c6504f963080000616c75666a51c300c36a51c351c36a52527ac46a51c352c36a53527ac46a52c36a53c3615272651d01616c756600616c756600c56b6168164e656f2e53746f726167652e476574436f6e7465787461652206617c680f4e656f2e53746f726167652e476574616c756651c56b6c766b00527ac46a00c3c001149c633f002442616c616e63654f66206973206661696c65642e20696e76616c6964206164647265737361680f4e656f2e52756e74696d652e4c6f6700616c75666168164e656f2e53746f726167652e476574436f6e746578746165b605617c65e5056a00c3617c659306616c756653c56b6c766b00527ac46c766b51527ac46a00c361656dff6a51c3936a52527ac46a00c36a52c3617c65bceb7561006a00c36a51c3615272087472616e7366657254c168124e656f2e52756e74696d652e4e6f7469667951616c756658c56b6c766b00527ac46c766b51527ac46c766b52527ac46a00c3c00114907c907c9e630d006a51c3c001149c6348002d5472616e73666572206973206661696c65642e202866726f6d7c746f292061646472657320696e76616c69642e61680f4e656f2e52756e74696d652e4c6f6700616c75666a52c3009f6445002a5472616e73666572206973206661696c65642e207472616e736665722076616c7565206973203c3d203061680f4e656f2e52756e74696d652e4c6f6700616c75666a00c36168184e656f2e52756e74696d652e436865636b5769746e6573736346002b5472616e73666572206661696c65642e20636f6e747261637420696e766f6b65722069732076616c69642e61680f4e656f2e52756e74696d652e4c6f6700616c75666a51c36165f5f56a53527ac46a00c36165f3fd6a54527ac4006a55527ac46a53c3640e006a54c36a55527ac4621b006a00c36165dbea6a57527ac46a54c36a57c3946a55527ac46a55c36a52c39f644200275472616e73666572206973206661696c65642e2066726f6d42616c616e6365203c2076616c756561680f4e656f2e52756e74696d652e4c6f6700616c75666a00c36a51c39c630b006a52c3009c644400295472616e73666572206661696c65642e2066726f6d203d3d20746f206f722076616c7565203d3d203061680f4e656f2e52756e74696d652e4c6f6751616c75666a53c3640f006a00c36a52c3617c6581ec756a00c36a54c36a52c394617c6565e9756a51c36165fafc6a52c3936a56527ac46a51c36a56c3617c6549e975616a00c36a51c36a52c3615272087472616e7366657254c168124e656f2e52756e74696d652e4e6f7469667951616c756659c56b6c766b00527ac46c766b51527ac46c766b52527ac46c766b53527ac46a00c3c00114907c907c9e630d006a51c3c001149c6348002d5472616e73666572206973206661696c65642e202866726f6d7c746f292061646472657320696e76616c69642e61680f4e656f2e52756e74696d652e4c6f6700616c75666a52c3009f6445002a5472616e73666572206973206661696c65642e207472616e736665722076616c7565206973203c3d203061680f4e656f2e52756e74696d652e4c6f6700616c75666a00c36168184e656f2e52756e74696d652e436865636b5769746e6573736346002b5472616e73666572206661696c65642e20636f6e747261637420696e766f6b65722069732076616c69642e61680f4e656f2e52756e74696d652e4c6f6700616c75666a51c3616579f36a54527ac46a00c3616577fb6a55527ac4006a56527ac46a54c3640e006a55c36a56527ac4621b006a00c361655fe86a58527ac46a55c36a58c3946a56527ac46a56c36a52c39f644200275472616e73666572206973206661696c65642e2066726f6d42616c616e6365203c2076616c756561680f4e656f2e52756e74696d652e4c6f6700616c75666a00c36a51c39c630b006a52c3009c644400295472616e73666572206661696c65642e2066726f6d203d3d20746f206f722076616c7565203d3d203061680f4e656f2e52756e74696d652e4c6f6751616c75666a54c3640f006a00c36a52c3617c6505ea756a00c36a55c36a52c394617c65e9e6756a51c361657efa6a52c3936a57527ac46a51c36a57c3617c65cde6756a51c36a52c36a53c36152726516ee75616a00c36a51c36a52c3615272087472616e7366657254c168124e656f2e52756e74696d652e4e6f7469667951616c756600c56b0d636f6e747261637441646d696e616c756600c56b10636f6e7472616374496e697454696d65616c756600c56b10746f74616c546f6b656e537570706c79616c756600c56b0862616c616e63655f616c756600c56b086c6f636b7570735f616c756600c56b0c6f7463417070726f7665645f616c756652c56b6c766b00527ac46c766b51527ac46152c5766a00c3007cc4766a51c3517cc4616c756654c56b6c766b00527ac46c766b51527ac46c766b52527ac46a00c351c301007e6a51c37e6a53527ac46a00c300c36a53c36a52c3615272680f4e656f2e53746f726167652e507574616c756653c56b6c766b00527ac46c766b51527ac46a00c351c301007e6a51c37e6a52527ac46a00c300c36a52c3617c68124e656f2e53746f726167652e44656c657465616c756653c56b6c766b00527ac46c766b51527ac46a00c351c301007e6a51c37e6a52527ac46a00c300c36a52c3617c680f4e656f2e53746f726167652e476574616c7566"
   },
   "jsonrpc" : "2.0"
}

[roman-khimov]

Log around that block:

2020-04-03T11:00:36.569+0300    INFO    blockchain persist completed    {"persistedBlocks": 45, "persistedKeys": 2173, "headerHeight": 3474286, "blockHeight": 3474285, "took": "185.706934ms"}
2020-04-03T11:00:37.201+0300    INFO    runtime log     {"script": "8d085e441a6e2e751e60146b9da2662b5afcc0c9", "logs": "\"Transfer() empty transfer amount or from==to\""}
2020-04-03T11:00:37.211+0300    INFO    runtime log     {"script": "3fbc607c12c28736343224a4b4d8f513a5c27ca8", "logs": "\"insufficient funds\""}
2020-04-03T11:00:37.778+0300    INFO    blockchain persist completed    {"persistedBlocks": 50, "persistedKeys": 1908, "headerHeight": 3474336, "blockHeight": 3474335, "took": "208.520317ms"}

[fyrchik]

b51403 (b6.., b7..) is 201909 (..10, ..11) which looks like year + month.
111503 is 202001 btw.

[roman-khimov]

I'd also suspected Neo.Blockchain.GetHeight/Neo.Blockchain.GetHeader/Neo.Header.GetTimestamp syscalls sequence (maybe getting the wrong height/header/time), but those seem to be fine, the height corresponds to already persisted height and not the block being persisted, so everything else should match too.

[roman-khimov]

Any progress here?

[roman-khimov]

A list of syscalls this contract makes:

"Neo.Blockchain.GetContract"
"Neo.Blockchain.GetHeader"
"Neo.Blockchain.GetHeight"
"Neo.Contract.GetScript"
"Neo.Contract.Migrate"
"Neo.Header.GetTimestamp"
"Neo.Iterator.Next"
"Neo.Iterator.Value"
"Neo.Runtime.CheckWitness"
"Neo.Runtime.Deserialize"
"Neo.Runtime.GetTrigger"
"Neo.Runtime.Log"
"Neo.Runtime.Notify"
"Neo.Runtime.Serialize"
"Neo.Storage.Delete"
"Neo.Storage.Find"
"Neo.Storage.Get"
"Neo.Storage.GetContext"
"Neo.Storage.Put"
"System.ExecutionEngine.GetExecutingScriptHash"

We know that we have exactly the same storage state before 3474317, so I can only suspect Find and iterations, but that was already tried.

At the same time it's not hard to get a full trace of this contract execution from our VM just to see how those values are formed and maybe that would give some clue. Of course ideally we should compare this trace with C# VM trace, but that's a bit more complicated.

[fyrchik]

I have checked the order of storage.Find iterator:

1. It is ordered by keys.
2. Removing values while holding using an iterator does not affect anything.

There is at least one function 4709--4920 which returns a Map item.
Then a VALUES is called on it at 6238.
This is where we can return wrong order of elements, as C# node does not use any ordering in NEO2.0. This can explain why there is one more loop iteration.

[roman-khimov]

C# node does not use any ordering in NEO2.0

Any guaranteed ordering, true, but at the same time our current 3.0-compatible ordering is in fact compatible with neo-storage-audit at least up to this block (that's why I've not yet release go version of compare-dumps, it was intended to compare deserialized maps, but in fact this deserialization is not yet needed). Maybe of course that's where it finally fails...

[roman-khimov]

Side note: I'm not sure how storage key serialization affects Find ordering, see neo-project/neo#793. Maybe that's the difference between what I've tried here and what really needs to be done.

[roman-khimov]

FYI: neo-project/neo#946 and neo-project/neo#950, though it looks like 2.x doesn't have it.

[fyrchik]

After I made Storage.Find return items in reverse lexicographic order, state is equal until block 3563028 in this transaction https://neoscan.io/transaction/9fe1c215386d89f6e82a0f0687821bf5978a9181746f01533b341c0a92d6c981
Fun fact: it is ChronoCoin (this task is about TimeCoin contract) and invoked method is transferWithLockupPeriod.
Application logs are equal, but storage differs.

[roman-khimov]

Can you share state differences? Are they caused by the same ordering issue (do they react to any changes wrt this issue)?

[fyrchik]

I think so, because it is a similar contract, it also uses iterator returned from storage.Find.
Our:

"state": "Changed",
"key": "deb3fc4d7571d3ea0ca4ae9112baf7a9e1e375c86c6f636b7570735f2c51adddf51d59f100bd4d3678e228815b8d665f70b614030001",
"value": "002180040203b714030203b61403020640715b5bb5008201020113000640715b5bb50000"

Their (note that the key is also different):

"state": "Changed",
"key": "deb3fc4d7571d3ea0ca4ae9112baf7a9e1e375c86c6f636b7570735f2c51adddf51d59f100bd4d3678e228815b8d665f70b514030001",
"value": "002180040203b714030203b51403020640715b5bb5008201020113000640715b5bb50000"

[fyrchik]

Well I have tried:
https://github.com/neo-project/neo/blob/master-2.x/neo/IO/Caching/DataCache.cs#L121

1. Reordering Seek in MemcachedStore.Seek() (MemoryStore and persistent store).
2. Sorting results in different orders (in every possible way, but see previous post).
3. Caching results of storage.Get and seeking them together with Put items.

Combined 1, 2, 3 in every possible combination without any success. May be I am just tired and missed something.

[roman-khimov]

My biggest fear wrt this bug is inter-transaction or inter-block caching, I've not looked at C# node's caching system close enough to know whether and how some accesses by previous blocks/transactions affect caching and Storage.Find results.

But we have a cache instance for transaction, I'd probably try to start with that. Log every put/get happening for our contracts, then look at Find accesses, the number of returned elements, possible combinations of their order (luckily there are not that many elements), maybe see which combinations work and which don't for any given block (3474317, 3563028, any subsequent that exhibits this behaviour), then try to figure out what's the common pattern. In short --- get more data for each problematic block (if we can go through them in some way). If we're to suppose that this order in not dependent on previous transactions/block then this approach should give the result.

[roman-khimov]

It's gone and we have more interesting bugs now.