Merge branch 'master' into release-0.16.0-promote

CHANGELOG.md

@@ -2,6 +2,26 @@
 
 # UNRELEASED
 
+### fix!: always fetch did file from canister when making canister calls
+
+`dfx canister call` will always fetch did file from the canister metadata. This is especially helpful for calling remote canisters. It's a breaking change in the sense that if the canister doesn't have the `candid:service` metadata, we will not read the local did file from build artifact, and dfx will issue a warning in this case to encourage canister developers to put the did file into canister metadata.
+
+### fix: support `import` for local did file
+
+If the local did file contains `import` or init args, dfx will rewrite the did file when storing in canister metadata.
+Due to current limitations of the Candid parser, comments will be dropped during rewriting. 
+If the local did file doesn't contain `import` or init args, we will not perform the rewriting, thus preserving the comments.
+
+### fix: subtyping check reports the special opt rule as error
+
+### fix: can now run several dfx canister commands outside of a project
+
+The following commands now work outside of a project:
+- `dfx canister start <specific canister id>`
+- `dfx canister stop <specific canister id>`
+- `dfx canister deposit-cycles <amount> <specific canister id>`
+- `dfx canister uninstall-code <specific canister id>`
+
 # 0.16.0
 
 ### feat: large canister modules now supported
@@ -16,6 +36,13 @@ Providers can provide the hash through `wasm_hash_url` instead of hard coding th
 
 If the hash of downloaded wasm doesn’t match the provided hash (`wasm_hash`, `wasm_hash_url` or read from mainnet state tree), dfx deps won’t abort. Instead, it will print a warning message.
 
+### feat: create canister on specific subnets or subnet types
+
+`dfx deploy`, `dfx canister create`, and `dfx ledger create-canister` now support the option `--subnet <subnet principal>` to create canisters on specific subnets.
+
+`dfx canister create` and `dfx deploy` now support the option `--subnet-type <subnet type>` to create canisters on a random subnet of a certain type.
+Use `dfx ledger show-subnet-types` to list the available subnet types
+
 ### feat!: update `dfx cycles` commands with mainnet `cycles-ledger` canister ID
 
 The `dfx cycles` command no longer needs nor accepts the `--cycles-ledger-canister-id <canister id>` parameter.

docs/cli-reference/dfx-canister.md

@@ -220,6 +220,8 @@ You can use the following options with the `dfx canister create` command.
 | `--no-wallet`                             | Performs the call with the user Identity as the Sender of messages. Bypasses the Wallet canister. Enabled by default.                                                                                                                                                                                                                                                                    |
 | `--with-cycles <number-of-cycles>`        | Specifies the initial cycle balance to deposit into the newly created canister. The specified amount needs to take the canister create fee into account. This amount is deducted from the wallet's cycle balance.                                                                                                                                                                        |
 | `--specified-id <PRINCIPAL>`              | Attempts to create the canister with this Canister ID |
+| `--subnet-type <subnet-type>`             | Specify the subnet type to create the canister on. If no subnet type is provided, the canister will be created on a random default application subnet.                      |
+| `--subnet <subnet-principal>`             | Specify the subnet to create the canister on. If no subnet is provided, the canister will be created on a random default application subnet.                                |
 
 ### Arguments
 

docs/cli-reference/dfx-deploy.md

@@ -30,6 +30,8 @@ You can use the following options with the `dfx deploy` command.
 | `--specified-id <PRINCIPAL>`       | Attempts to create the canister with this Canister ID                                                                                 |
 | `--by-proposal`                    | Upload proposed changed assets, but do not commit them.  Follow up by calling either commit_proposed_batch() or delete_batch().                                             |
 | `--compute-evidence`               | Build a frontend canister, determine batch operations required to synchronize asset canister contents, and compute a hash over those operations.  Displays this hash ("evidence"), which should match the evidence displayed by `dfx deploy --by-proposal`. |
+| `--subnet-type <subnet-type>`      | Specify the subnet type to create the canister on. If no subnet type is provided, the canister will be created on a random default application subnet.                      |
+| `--subnet <subnet-principal>`      | Specify the subnet to create the canister on. If no subnet is provided, the canister will be created on a random default application subnet.                                |
 
 ### Arguments
 

docs/cli-reference/dfx-identity.md

@@ -22,6 +22,7 @@ For reference information and examples that illustrate using `dfx identity` comm
 | [`get-principal`](#dfx-identity-get-principal) | Shows the textual representation of the principal associated with the current identity.                                   |
 | [`get-wallet`](#dfx-identity-get-wallet)       | Shows the canister identifier for the wallet associated with your current identity principal.                             |
 | `help`                                          | Displays this usage message or the help of the given subcommand(s).                                                       |
+| [`export`](#dfx-identity-export)               | Exports the PEM definition for an identity. |
 | [`import`](#dfx-identity-import)               | Creates a new identity by importing a PEM file that contains the key information or security certificate for a principal. |
 | [`list`](#dfx-identity-list)                   | Lists existing identities.                                                                                                |
 | [`new`](#dfx-identity-new)                     | Creates a new identity.                                                                                                   |
@@ -110,6 +111,25 @@ To display the canister identifier for the wallet canister associated with your
 dfx identity get-wallet --network=https://192.168.74.4
 ```
 
+## dfx identity export
+
+Use the `dfx identity export` command to export the PEM definition of an existing identity.  You can import this definition elsewhere using [`dfx identity import`](#dfx-identity-import).
+
+### Basic usage
+
+``` bash
+dfx identity export identity-name
+```
+
+### Examples
+
+The following will export a PEM file for an existing identity.  If you transferred this file to another computer, the
+example below for `dfx identity import` would import it.
+
+``` bash
+dfx identity export alice >generated-id.pem
+```
+
 ## dfx identity import
 
 Use the `dfx identity import` command to create a user identity by importing the user’s key information or security certificate from a PEM file.

docs/cli-reference/dfx-ledger.md

@@ -121,6 +121,7 @@ You can specify the following argument for the `dfx ledger create-canister` comm
 | `--icp <icp>`                 | Specify ICP tokens as a whole number. You can use this option on its own or in conjunction with `--e8s`.                                                                                                                                             |
 | `--max-fee <max-fee>`         | Specify a maximum transaction fee. The default is 10000 e8s.                                                                                                                                                                                         |
 | `--subnet-type <subnet-type>` | Specify the optional subnet type to create the canister on. If no subnet type is provided, the canister will be created on a random default application subnet.                                                                                      |
+| `--subnet <subnet-principal>` | Specify the optional subnet to create the canister on. If no subnet is provided, the canister will be created on a random default application subnet.                                                                                                |
 | `--created-at-time <timestamp>`| Specify the timestamp-nanoseconds for the `created_at_time` field on the ledger transfer request. Useful for controlling transaction-de-duplication. https://internetcomputer.org/docs/current/developer-docs/integrations/icrc-1/#transaction-deduplication-  |
 
 ### Examples

e2e/assets/cycles-ledger/dfx.json

@@ -5,10 +5,10 @@
       "wasm": "cycles-ledger.wasm.gz",
       "candid": "cycles-ledger.did"
     },
-    "cycles-depositor": {
+    "depositor": {
       "type": "custom",
-      "wasm": "cycles-depositor.wasm.gz",
-      "candid": "cycles-depositor.did"
+      "wasm": "depositor.wasm.gz",
+      "candid": "depositor.did"
     }
   }
-}
+}

e2e/assets/fake_cmc/dfx.json

@@ -0,0 +1,9 @@
+{
+  "canisters": {
+    "fake-cmc": {
+      "type": "custom",
+      "wasm": "fake-cmc.wasm.gz",
+      "candid": "fake-cmc.did"
+    }
+  }
+}

e2e/assets/prebuilt_custom_canister/dfx.json

@@ -29,6 +29,15 @@
       "candid": "custom_with_build_step.did",
       "wasm": "main.wasm",
       "build": "echo just a build step"
+    },
+    "prebuilt_local_import": {
+       "type": "custom",
+       "candid": "local_import.did",
+       "wasm": "main.wasm",
+       "build": [],
+       "metadata": [
+         { "name": "candid:service" }
+       ]
     }
   },
   "networks": {

e2e/assets/prebuilt_custom_canister/local_import.did

@@ -0,0 +1,2 @@
+import service "prebuilt_custom_no_build.did";
+

e2e/assets/upgrade/v1.mo

@@ -4,6 +4,9 @@ actor {
     state += 1;
     return state;
   };
+  public func f() : async ?Int {
+    return ?42;
+  };
   public query func read() : async Int { return state; };
 }
 

e2e/assets/upgrade/v2.mo

@@ -6,6 +6,9 @@ actor {
     new_state += 1;
     return new_state;
   };
+  public func f() : async ?Int {
+    return ?42;
+  };
   public query func read() : async Nat { return new_state; };
 }
 

e2e/assets/upgrade/v2_bad.mo

@@ -4,6 +4,9 @@ actor {
     state += 1;
     return state;
   };
+  public func f() : async ?Int {
+    return ?42;
+  };
   public query func read() : async Nat { return state; };
 }
 

e2e/assets/upgrade/v3_bad.mo

@@ -4,6 +4,9 @@ actor {
     state += 1;
     return state;
   };
+  public func f() : async ?Int {
+    return ?42;
+  };
   public query func read2() : async Int { return state; };
 }
 

e2e/assets/upgrade/v4_bad.mo

@@ -0,0 +1,12 @@
+actor {
+  stable var state : Int = 0;
+  public func inc() : async Int {
+    state += 1;
+    return state;
+  };
+  public func f() : async ?Text {
+    return ?"";
+  };
+  public query func read() : async Int { return state; };
+}
+

e2e/tests-dfx/assetscanister.bash

@@ -266,7 +266,7 @@ check_permission_failure() {
   FE_CANISTER_ID="$(dfx canister id e2e_project_frontend)"
   rm .dfx/local/canister_ids.json
   assert_command_fail dfx canister call "$FE_CANISTER_ID" validate_revoke_permission "(record { of_principal=principal \"$PREPARE_PRINCIPAL\"; permission = variant { FlyBeFree }; })"
-  assert_contains "trapped"
+  assert_contains "FlyBeFree not found"
 }
 
 @test "access control - fine-grained" {
@@ -905,24 +905,29 @@ check_permission_failure() {
   dfx canister install e2e_project_frontend
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("/binary/noise.txt")' --output idl
-  assert_eq '(blob "\b8\01\20\80\0a\77\31\32\20\00\78\79\0a\4b\4c\0b\0a\6a\6b")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "\b8\01\20\80\0a\77\31\32\20\00\78\79\0a\4b\4c\0b\0a\6a\6b")' "$stdout"
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("/text-with-newlines.txt")' --output idl
-  assert_eq '(blob "cherries\0ait\27s cherry season\0aCHERRIES")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "cherries\0ait\27s cherry season\0aCHERRIES")' "$stdout"
 
   assert_command dfx canister call --update e2e_project_frontend store '(record{key="AA"; content_type="text/plain"; content_encoding="identity"; content=blob "hello, world!"})'
   assert_eq '()'
   assert_command dfx canister call --update e2e_project_frontend store '(record{key="B"; content_type="application/octet-stream"; content_encoding="identity"; content=blob"XWV"})'
   assert_eq '()'
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("B")' --output idl
-  assert_eq '(blob "XWV")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "XWV")' "$stdout"
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("AA")' --output idl
-  assert_eq '(blob "hello, world!")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "hello, world!")' "$stdout"
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("B")' --output idl
-  assert_eq '(blob "XWV")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "XWV")' "$stdout"
 
   assert_command_fail dfx canister call --query e2e_project_frontend retrieve '("C")'
 }

e2e/tests-dfx/build_granular.bash

@@ -67,10 +67,12 @@ teardown() {
   dfx canister install e2e_project_frontend
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("/binary/noise.txt")' --output idl
-  assert_eq '(blob "\b8\01\20\80\0a\77\31\32\20\00\78\79\0a\4b\4c\0b\0a\6a\6b")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "\b8\01\20\80\0a\77\31\32\20\00\78\79\0a\4b\4c\0b\0a\6a\6b")' "$stdout"
 
   assert_command dfx canister call --query e2e_project_frontend retrieve '("/text-with-newlines.txt")' --output idl
-  assert_eq '(blob "cherries\0ait\27s cherry season\0aCHERRIES")'
+  # shellcheck disable=SC2154
+  assert_eq '(blob "cherries\0ait\27s cherry season\0aCHERRIES")' "$stdout"
 }
 
 @test "cyclic dependencies are detected" {

e2e/tests-dfx/call.bash

@@ -26,9 +26,9 @@ teardown() {
   CANISTER_ID=$(dfx canister id hello_backend)
   rm .dfx/local/canister_ids.json
 
-  # if no candid file known, then no field names
+  # given a canister id, fetch the did file from metadata
   assert_command dfx canister call "$CANISTER_ID" make_struct '("A", "B")'
-  assert_eq '(record { 99 = "A"; 100 = "B" })'
+  assert_eq '(record { c = "A"; d = "B" })'
 
   # if passing the candid file, field names available
   assert_command dfx canister call --candid .dfx/local/canisters/hello_backend/hello_backend.did "$CANISTER_ID" make_struct '("A", "B")'

e2e/tests-dfx/create.bash

@@ -1,6 +1,7 @@
 #!/usr/bin/env bats
 
 load ../utils/_
+load ../utils/cycles-ledger
 
 setup() {
   standard_setup
@@ -269,3 +270,28 @@ teardown() {
   assert_command dfx wallet upgrade --identity alice
   assert_command dfx canister create --all --controller alice --controller bob --identity alice
 }
+
+@test "create canister - subnet targetting" {
+  # fake cmc setup
+  cd ..
+  dfx_new fake_cmc
+  install_asset fake_cmc
+  install_cycles_ledger_canisters
+  dfx_start
+  assert_command dfx deploy fake-cmc --specified-id "rkp4c-7iaaa-aaaaa-aaaca-cai" # CMC canister id
+  cd ../e2e_project
+
+  # use --subnet <principal>
+  SUBNET_ID="5kdm2-62fc6-fwnja-hutkz-ycsnm-4z33i-woh43-4cenu-ev7mi-gii6t-4ae" # a random, valid principal
+  assert_command dfx canister create e2e_project_backend --subnet "$SUBNET_ID"
+  cd ../fake_cmc
+  assert_command dfx canister call fake-cmc last_create_canister_args
+  assert_contains "subnet = principal \"$SUBNET_ID\";"
+
+  # use --subnet-type
+  cd ../e2e_project
+  assert_command dfx canister create e2e_project_frontend --subnet-type custom_subnet_type
+  cd ../fake_cmc
+  assert_command dfx canister call fake-cmc last_create_canister_args
+  assert_contains 'subnet_type = opt "custom_subnet_type"'
+}