diff --git a/.github/scripts/sync-github-releases.sh b/.github/scripts/sync-github-releases.sh index c5c6f97ed6d8..48df370b2216 100755 --- a/.github/scripts/sync-github-releases.sh +++ b/.github/scripts/sync-github-releases.sh @@ -49,7 +49,7 @@ CREATE_RELEASE=() TAG_URL="${REPO}/tags?per_page=100" ## This function is used to loop over the pagianated results from github tags -## It sets TAGS to be the the json from the current page of tags +## It sets TAGS to be the json from the current page of tags _next_page() { TAGS=$(curl -s "${HDR[@]}" ${TAG_URL}) ## In the "Link:" header from github we get the link for the next page. diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml index 4f57b16ee6d1..d6438c9ed37c 100644 --- a/.github/workflows/main.yaml +++ b/.github/workflows/main.yaml @@ -21,8 +21,9 @@ on: schedule: - cron: 0 1 * * * +## We cancel any multiple runs from PRs, while runs from tags/branches are allowed concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.sha }} + group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.pull_request.number || github.sha }} cancel-in-progress: true env: diff --git a/HOWTO/DEVELOPMENT.md b/HOWTO/DEVELOPMENT.md index 3a12177f31a1..dfe0e3a0d868 100644 --- a/HOWTO/DEVELOPMENT.md +++ b/HOWTO/DEVELOPMENT.md @@ -350,7 +350,7 @@ switches are: * Set environment variable `VALGRIND_MISC_FLAGS` for any extra valgrind flags you want to pass. * -asan * Start [Clang Address Sanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) - with the the correct settings and use the `asan` [type](types-and-flavors). + with the correct settings and use the `asan` [type](types-and-flavors). * Set environment variable `ASAN_LOG_DIR` to where you want the logs. * Set environment variable `ASAN_OPTIONS` for any extra asan options you want to pass. * -gcov diff --git a/HOWTO/INSTALL-IOS.md b/HOWTO/INSTALL-IOS.md index 3ec368ecc70a..9a8d38e9ac5d 100644 --- a/HOWTO/INSTALL-IOS.md +++ b/HOWTO/INSTALL-IOS.md @@ -17,7 +17,7 @@ In order to inform the build system that it should generate the static linkable ### Configure Erlang/OTP ### - To build without without OpenSSL support, run `configure` like this: + To build without OpenSSL support, run `configure` like this: $ ./otp_build configure \ --xcomp-conf=./xcomp/erl-xcomp-arm64-ios.conf \ diff --git a/HOWTO/MARKDOWN.md b/HOWTO/MARKDOWN.md index 91b5526bf925..5cc5b79adda5 100644 --- a/HOWTO/MARKDOWN.md +++ b/HOWTO/MARKDOWN.md @@ -183,7 +183,7 @@ places. Appropriate attributes to the `X` tag will also be generated. contain information from a \%CopyrightBegin\%, \%CopyrightEnd\% block if such exist (see below). -* A level `X` heading where `1 < X <= 6` will cause the the following +* A level `X` heading where `1 < X <= 6` will cause the following to be generated:
@@ -209,7 +209,7 @@ places. Appropriate attributes to the `X` tag will also be generated. other documents. That is, *be careful* when changing headings in an existing document. -* A level `X` heading where `6 < X` will cause the the following +* A level `X` heading where `6 < X` will cause the following to be generated: @@ -247,7 +247,7 @@ Copyright and License %CopyrightBegin% -Copyright Ericsson AB 2010-2021. All Rights Reserved. +Copyright Ericsson AB 2010-2023. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/bootstrap/bin/no_dot_erlang.boot b/bootstrap/bin/no_dot_erlang.boot index 3d14d2b44706..01b855e6970b 100644 Binary files a/bootstrap/bin/no_dot_erlang.boot and b/bootstrap/bin/no_dot_erlang.boot differ diff --git a/bootstrap/bin/start.boot b/bootstrap/bin/start.boot index 3d14d2b44706..01b855e6970b 100644 Binary files a/bootstrap/bin/start.boot and b/bootstrap/bin/start.boot differ diff --git a/bootstrap/bin/start_clean.boot b/bootstrap/bin/start_clean.boot index 3d14d2b44706..01b855e6970b 100644 Binary files a/bootstrap/bin/start_clean.boot and b/bootstrap/bin/start_clean.boot differ diff --git a/bootstrap/lib/compiler/ebin/beam_asm.beam b/bootstrap/lib/compiler/ebin/beam_asm.beam index 4fe7fdc6fd9c..f875cbdfa9db 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_asm.beam and b/bootstrap/lib/compiler/ebin/beam_asm.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_block.beam b/bootstrap/lib/compiler/ebin/beam_block.beam index adf08ad7208d..c352fe2e0c6f 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_block.beam and b/bootstrap/lib/compiler/ebin/beam_block.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_clean.beam b/bootstrap/lib/compiler/ebin/beam_clean.beam index 2379115a48db..951bd46f14ea 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_clean.beam and b/bootstrap/lib/compiler/ebin/beam_clean.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_core_to_ssa.beam b/bootstrap/lib/compiler/ebin/beam_core_to_ssa.beam index f1d1b936ec02..242d57c4de78 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_core_to_ssa.beam and b/bootstrap/lib/compiler/ebin/beam_core_to_ssa.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_dict.beam b/bootstrap/lib/compiler/ebin/beam_dict.beam index edfd03f4abb1..11a1934b4430 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_dict.beam and b/bootstrap/lib/compiler/ebin/beam_dict.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_disasm.beam b/bootstrap/lib/compiler/ebin/beam_disasm.beam index 886851b0869e..93f910a17406 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_disasm.beam and b/bootstrap/lib/compiler/ebin/beam_disasm.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_flatten.beam b/bootstrap/lib/compiler/ebin/beam_flatten.beam index cc6740a0ea39..19d0e71bfc18 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_flatten.beam and b/bootstrap/lib/compiler/ebin/beam_flatten.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_jump.beam b/bootstrap/lib/compiler/ebin/beam_jump.beam index 558708951245..66c75b5d3f48 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_jump.beam and b/bootstrap/lib/compiler/ebin/beam_jump.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_opcodes.beam b/bootstrap/lib/compiler/ebin/beam_opcodes.beam index 0dd64db55bee..0f2f9adaf193 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_opcodes.beam and b/bootstrap/lib/compiler/ebin/beam_opcodes.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_alias.beam b/bootstrap/lib/compiler/ebin/beam_ssa_alias.beam index ece709f981ad..ae3c99eb4a51 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_alias.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_alias.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_codegen.beam b/bootstrap/lib/compiler/ebin/beam_ssa_codegen.beam index 716d6fd15994..1e478db4f34f 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_codegen.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_codegen.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_dead.beam b/bootstrap/lib/compiler/ebin/beam_ssa_dead.beam index 9848bf4dec91..8a554108d3e0 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_dead.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_dead.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_lint.beam b/bootstrap/lib/compiler/ebin/beam_ssa_lint.beam index 1fbd08ded684..2569facf154c 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_lint.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_lint.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_opt.beam b/bootstrap/lib/compiler/ebin/beam_ssa_opt.beam index 7c2373ee650a..39914638b7a9 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_opt.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_opt.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_pre_codegen.beam b/bootstrap/lib/compiler/ebin/beam_ssa_pre_codegen.beam index 38c4825dcff8..cd9675a07464 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_pre_codegen.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_pre_codegen.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_share.beam b/bootstrap/lib/compiler/ebin/beam_ssa_share.beam index 793d26d3661f..27b36c3c1403 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_share.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_share.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_ssa_type.beam b/bootstrap/lib/compiler/ebin/beam_ssa_type.beam index 232027d457c1..f76a158383ae 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_ssa_type.beam and b/bootstrap/lib/compiler/ebin/beam_ssa_type.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_trim.beam b/bootstrap/lib/compiler/ebin/beam_trim.beam index 6adca21ec07d..934f12a5dd29 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_trim.beam and b/bootstrap/lib/compiler/ebin/beam_trim.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_types.beam b/bootstrap/lib/compiler/ebin/beam_types.beam index 7199eb4a8891..367a31dea966 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_types.beam and b/bootstrap/lib/compiler/ebin/beam_types.beam differ diff --git a/bootstrap/lib/compiler/ebin/beam_validator.beam b/bootstrap/lib/compiler/ebin/beam_validator.beam index d0b4f226abcb..243a8eae6e62 100644 Binary files a/bootstrap/lib/compiler/ebin/beam_validator.beam and b/bootstrap/lib/compiler/ebin/beam_validator.beam differ diff --git a/bootstrap/lib/compiler/ebin/cerl_inline.beam b/bootstrap/lib/compiler/ebin/cerl_inline.beam index fa0a9763e627..9b569f80d49a 100644 Binary files a/bootstrap/lib/compiler/ebin/cerl_inline.beam and b/bootstrap/lib/compiler/ebin/cerl_inline.beam differ diff --git a/bootstrap/lib/compiler/ebin/compile.beam b/bootstrap/lib/compiler/ebin/compile.beam index a223fa123917..8de3b85fc2bb 100644 Binary files a/bootstrap/lib/compiler/ebin/compile.beam and b/bootstrap/lib/compiler/ebin/compile.beam differ diff --git a/bootstrap/lib/compiler/ebin/core_parse.beam b/bootstrap/lib/compiler/ebin/core_parse.beam index c46d70f2b2ed..c6070109d663 100644 Binary files a/bootstrap/lib/compiler/ebin/core_parse.beam and b/bootstrap/lib/compiler/ebin/core_parse.beam differ diff --git a/bootstrap/lib/compiler/ebin/core_pp.beam b/bootstrap/lib/compiler/ebin/core_pp.beam index 0ea9976936b0..bc0755728b42 100644 Binary files a/bootstrap/lib/compiler/ebin/core_pp.beam and b/bootstrap/lib/compiler/ebin/core_pp.beam differ diff --git a/bootstrap/lib/compiler/ebin/rec_env.beam b/bootstrap/lib/compiler/ebin/rec_env.beam index bddf8e7b775c..ea128c7489ef 100644 Binary files a/bootstrap/lib/compiler/ebin/rec_env.beam and b/bootstrap/lib/compiler/ebin/rec_env.beam differ diff --git a/bootstrap/lib/compiler/ebin/sys_core_fold.beam b/bootstrap/lib/compiler/ebin/sys_core_fold.beam index d2f73af14933..f2f6b017e97a 100644 Binary files a/bootstrap/lib/compiler/ebin/sys_core_fold.beam and b/bootstrap/lib/compiler/ebin/sys_core_fold.beam differ diff --git a/bootstrap/lib/compiler/ebin/sys_core_inline.beam b/bootstrap/lib/compiler/ebin/sys_core_inline.beam index 5ecd606b9f84..b811983c1cca 100644 Binary files a/bootstrap/lib/compiler/ebin/sys_core_inline.beam and b/bootstrap/lib/compiler/ebin/sys_core_inline.beam differ diff --git a/bootstrap/lib/compiler/ebin/sys_coverage.beam b/bootstrap/lib/compiler/ebin/sys_coverage.beam new file mode 100644 index 000000000000..47a7eb8b492a Binary files /dev/null and b/bootstrap/lib/compiler/ebin/sys_coverage.beam differ diff --git a/bootstrap/lib/compiler/ebin/v3_core.beam b/bootstrap/lib/compiler/ebin/v3_core.beam index 06d32b178c8b..0838e0f309f1 100644 Binary files a/bootstrap/lib/compiler/ebin/v3_core.beam and b/bootstrap/lib/compiler/ebin/v3_core.beam differ diff --git a/bootstrap/lib/kernel/ebin/application_controller.beam b/bootstrap/lib/kernel/ebin/application_controller.beam index 6ea3ba586419..7a1a07068676 100644 Binary files a/bootstrap/lib/kernel/ebin/application_controller.beam and b/bootstrap/lib/kernel/ebin/application_controller.beam differ diff --git a/bootstrap/lib/kernel/ebin/application_master.beam b/bootstrap/lib/kernel/ebin/application_master.beam index 925cef0803be..f63e296b8028 100644 Binary files a/bootstrap/lib/kernel/ebin/application_master.beam and b/bootstrap/lib/kernel/ebin/application_master.beam differ diff --git a/bootstrap/lib/kernel/ebin/code.beam b/bootstrap/lib/kernel/ebin/code.beam index b2dec0ddf904..8cf863fe3bb0 100644 Binary files a/bootstrap/lib/kernel/ebin/code.beam and b/bootstrap/lib/kernel/ebin/code.beam differ diff --git a/bootstrap/lib/kernel/ebin/code_server.beam b/bootstrap/lib/kernel/ebin/code_server.beam index 89eb91230eb5..c864c58a3ede 100644 Binary files a/bootstrap/lib/kernel/ebin/code_server.beam and b/bootstrap/lib/kernel/ebin/code_server.beam differ diff --git a/bootstrap/lib/kernel/ebin/disk_log.beam b/bootstrap/lib/kernel/ebin/disk_log.beam index 5af893b557ed..00bc12a8192c 100644 Binary files a/bootstrap/lib/kernel/ebin/disk_log.beam and b/bootstrap/lib/kernel/ebin/disk_log.beam differ diff --git a/bootstrap/lib/kernel/ebin/dist_ac.beam b/bootstrap/lib/kernel/ebin/dist_ac.beam index 44e4bb784c33..ec03e321d0cd 100644 Binary files a/bootstrap/lib/kernel/ebin/dist_ac.beam and b/bootstrap/lib/kernel/ebin/dist_ac.beam differ diff --git a/bootstrap/lib/kernel/ebin/erl_erts_errors.beam b/bootstrap/lib/kernel/ebin/erl_erts_errors.beam index e7adb3aa8158..ad041d5d4ce7 100644 Binary files a/bootstrap/lib/kernel/ebin/erl_erts_errors.beam and b/bootstrap/lib/kernel/ebin/erl_erts_errors.beam differ diff --git a/bootstrap/lib/kernel/ebin/erl_kernel_errors.beam b/bootstrap/lib/kernel/ebin/erl_kernel_errors.beam index 3e58b02f2a97..9f2d7a1c2981 100644 Binary files a/bootstrap/lib/kernel/ebin/erl_kernel_errors.beam and b/bootstrap/lib/kernel/ebin/erl_kernel_errors.beam differ diff --git a/bootstrap/lib/kernel/ebin/erts_debug.beam b/bootstrap/lib/kernel/ebin/erts_debug.beam index 560ba7b86517..2254e1f3f5d5 100644 Binary files a/bootstrap/lib/kernel/ebin/erts_debug.beam and b/bootstrap/lib/kernel/ebin/erts_debug.beam differ diff --git a/bootstrap/lib/kernel/ebin/file.beam b/bootstrap/lib/kernel/ebin/file.beam index f1699210418c..4b64642dea11 100644 Binary files a/bootstrap/lib/kernel/ebin/file.beam and b/bootstrap/lib/kernel/ebin/file.beam differ diff --git a/bootstrap/lib/kernel/ebin/gen_tcp_socket.beam b/bootstrap/lib/kernel/ebin/gen_tcp_socket.beam index 85fcde7e9ae9..dcb49747a602 100644 Binary files a/bootstrap/lib/kernel/ebin/gen_tcp_socket.beam and b/bootstrap/lib/kernel/ebin/gen_tcp_socket.beam differ diff --git a/bootstrap/lib/kernel/ebin/gen_udp_socket.beam b/bootstrap/lib/kernel/ebin/gen_udp_socket.beam index 26a60ca6491a..2f3790693d5e 100644 Binary files a/bootstrap/lib/kernel/ebin/gen_udp_socket.beam and b/bootstrap/lib/kernel/ebin/gen_udp_socket.beam differ diff --git a/bootstrap/lib/kernel/ebin/global.beam b/bootstrap/lib/kernel/ebin/global.beam index cbd5ec5a3f66..81ee2d89221c 100644 Binary files a/bootstrap/lib/kernel/ebin/global.beam and b/bootstrap/lib/kernel/ebin/global.beam differ diff --git a/bootstrap/lib/kernel/ebin/group.beam b/bootstrap/lib/kernel/ebin/group.beam index ea5d8d480289..493c947ba3dd 100644 Binary files a/bootstrap/lib/kernel/ebin/group.beam and b/bootstrap/lib/kernel/ebin/group.beam differ diff --git a/bootstrap/lib/kernel/ebin/inet.beam b/bootstrap/lib/kernel/ebin/inet.beam index ac9e5e3d1b7c..c500ece2dad8 100644 Binary files a/bootstrap/lib/kernel/ebin/inet.beam and b/bootstrap/lib/kernel/ebin/inet.beam differ diff --git a/bootstrap/lib/kernel/ebin/inet_config.beam b/bootstrap/lib/kernel/ebin/inet_config.beam index 3e306ebddbe4..a8aba0382b18 100644 Binary files a/bootstrap/lib/kernel/ebin/inet_config.beam and b/bootstrap/lib/kernel/ebin/inet_config.beam differ diff --git a/bootstrap/lib/kernel/ebin/inet_db.beam b/bootstrap/lib/kernel/ebin/inet_db.beam index a05d069ef8f2..aeb88a0f0bd3 100644 Binary files a/bootstrap/lib/kernel/ebin/inet_db.beam and b/bootstrap/lib/kernel/ebin/inet_db.beam differ diff --git a/bootstrap/lib/kernel/ebin/inet_dns.beam b/bootstrap/lib/kernel/ebin/inet_dns.beam index 69b85c1263c5..7b5426df736b 100644 Binary files a/bootstrap/lib/kernel/ebin/inet_dns.beam and b/bootstrap/lib/kernel/ebin/inet_dns.beam differ diff --git a/bootstrap/lib/kernel/ebin/inet_gethost_native.beam b/bootstrap/lib/kernel/ebin/inet_gethost_native.beam index fb5256782bc2..4aababdfbdd4 100644 Binary files a/bootstrap/lib/kernel/ebin/inet_gethost_native.beam and b/bootstrap/lib/kernel/ebin/inet_gethost_native.beam differ diff --git a/bootstrap/lib/kernel/ebin/inet_res.beam b/bootstrap/lib/kernel/ebin/inet_res.beam index ac35d0548543..e5f7466dbf33 100644 Binary files a/bootstrap/lib/kernel/ebin/inet_res.beam and b/bootstrap/lib/kernel/ebin/inet_res.beam differ diff --git a/bootstrap/lib/kernel/ebin/logger.beam b/bootstrap/lib/kernel/ebin/logger.beam index 96b55ffcf61d..3a951956b5f3 100644 Binary files a/bootstrap/lib/kernel/ebin/logger.beam and b/bootstrap/lib/kernel/ebin/logger.beam differ diff --git a/bootstrap/lib/kernel/ebin/logger_simple_h.beam b/bootstrap/lib/kernel/ebin/logger_simple_h.beam index 5aa8b3c80caa..f1240f9d989b 100644 Binary files a/bootstrap/lib/kernel/ebin/logger_simple_h.beam and b/bootstrap/lib/kernel/ebin/logger_simple_h.beam differ diff --git a/bootstrap/lib/kernel/ebin/logger_std_h.beam b/bootstrap/lib/kernel/ebin/logger_std_h.beam index 747efcaff220..42c36ce1c8a5 100644 Binary files a/bootstrap/lib/kernel/ebin/logger_std_h.beam and b/bootstrap/lib/kernel/ebin/logger_std_h.beam differ diff --git a/bootstrap/lib/kernel/ebin/net_kernel.beam b/bootstrap/lib/kernel/ebin/net_kernel.beam index c68df327fecb..b7393a53ee4c 100644 Binary files a/bootstrap/lib/kernel/ebin/net_kernel.beam and b/bootstrap/lib/kernel/ebin/net_kernel.beam differ diff --git a/bootstrap/lib/kernel/ebin/pg.beam b/bootstrap/lib/kernel/ebin/pg.beam index 356e58617760..78006417233e 100644 Binary files a/bootstrap/lib/kernel/ebin/pg.beam and b/bootstrap/lib/kernel/ebin/pg.beam differ diff --git a/bootstrap/lib/kernel/ebin/prim_tty.beam b/bootstrap/lib/kernel/ebin/prim_tty.beam index 1ecbbb08edb3..61a8a2f3ec2d 100644 Binary files a/bootstrap/lib/kernel/ebin/prim_tty.beam and b/bootstrap/lib/kernel/ebin/prim_tty.beam differ diff --git a/bootstrap/lib/kernel/ebin/socket.beam b/bootstrap/lib/kernel/ebin/socket.beam index 4b1063641985..bfcc7802d067 100644 Binary files a/bootstrap/lib/kernel/ebin/socket.beam and b/bootstrap/lib/kernel/ebin/socket.beam differ diff --git a/bootstrap/lib/kernel/ebin/user_drv.beam b/bootstrap/lib/kernel/ebin/user_drv.beam index 96b25a376bd5..4c4fe20a187b 100644 Binary files a/bootstrap/lib/kernel/ebin/user_drv.beam and b/bootstrap/lib/kernel/ebin/user_drv.beam differ diff --git a/bootstrap/lib/stdlib/ebin/argparse.beam b/bootstrap/lib/stdlib/ebin/argparse.beam index 60e604be6560..a6358a5d0372 100644 Binary files a/bootstrap/lib/stdlib/ebin/argparse.beam and b/bootstrap/lib/stdlib/ebin/argparse.beam differ diff --git a/bootstrap/lib/stdlib/ebin/c.beam b/bootstrap/lib/stdlib/ebin/c.beam index b9bbddb64fa1..c685420862b4 100644 Binary files a/bootstrap/lib/stdlib/ebin/c.beam and b/bootstrap/lib/stdlib/ebin/c.beam differ diff --git a/bootstrap/lib/stdlib/ebin/edlin.beam b/bootstrap/lib/stdlib/ebin/edlin.beam index 99a0e06b88d2..dbbfcbab1b88 100644 Binary files a/bootstrap/lib/stdlib/ebin/edlin.beam and b/bootstrap/lib/stdlib/ebin/edlin.beam differ diff --git a/bootstrap/lib/stdlib/ebin/edlin_context.beam b/bootstrap/lib/stdlib/ebin/edlin_context.beam index f229f29fd2da..4c22bdd61563 100644 Binary files a/bootstrap/lib/stdlib/ebin/edlin_context.beam and b/bootstrap/lib/stdlib/ebin/edlin_context.beam differ diff --git a/bootstrap/lib/stdlib/ebin/edlin_expand.beam b/bootstrap/lib/stdlib/ebin/edlin_expand.beam index 4dc492ac30f0..2dc2526324ab 100644 Binary files a/bootstrap/lib/stdlib/ebin/edlin_expand.beam and b/bootstrap/lib/stdlib/ebin/edlin_expand.beam differ diff --git a/bootstrap/lib/stdlib/ebin/edlin_key.beam b/bootstrap/lib/stdlib/ebin/edlin_key.beam index 11bed2a7b572..85a0ab5e8549 100644 Binary files a/bootstrap/lib/stdlib/ebin/edlin_key.beam and b/bootstrap/lib/stdlib/ebin/edlin_key.beam differ diff --git a/bootstrap/lib/stdlib/ebin/epp.beam b/bootstrap/lib/stdlib/ebin/epp.beam index 6bd7f3c49955..fd2707676d57 100644 Binary files a/bootstrap/lib/stdlib/ebin/epp.beam and b/bootstrap/lib/stdlib/ebin/epp.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_error.beam b/bootstrap/lib/stdlib/ebin/erl_error.beam index a166c1815be3..38a52c800484 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_error.beam and b/bootstrap/lib/stdlib/ebin/erl_error.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_eval.beam b/bootstrap/lib/stdlib/ebin/erl_eval.beam index 687077d2f19e..7bc3e5e2b324 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_eval.beam and b/bootstrap/lib/stdlib/ebin/erl_eval.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_expand_records.beam b/bootstrap/lib/stdlib/ebin/erl_expand_records.beam index 800aafd15a55..822f90595968 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_expand_records.beam and b/bootstrap/lib/stdlib/ebin/erl_expand_records.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_features.beam b/bootstrap/lib/stdlib/ebin/erl_features.beam index 5c4fcc7e56de..e4857ace6c84 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_features.beam and b/bootstrap/lib/stdlib/ebin/erl_features.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_lint.beam b/bootstrap/lib/stdlib/ebin/erl_lint.beam index f1f3aaf926d5..f9f1399ab1d7 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_lint.beam and b/bootstrap/lib/stdlib/ebin/erl_lint.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_parse.beam b/bootstrap/lib/stdlib/ebin/erl_parse.beam index 5e41cb0186b1..919a12d0a17c 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_parse.beam and b/bootstrap/lib/stdlib/ebin/erl_parse.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_pp.beam b/bootstrap/lib/stdlib/ebin/erl_pp.beam index cf61971647b6..4526e5b44ce0 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_pp.beam and b/bootstrap/lib/stdlib/ebin/erl_pp.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_scan.beam b/bootstrap/lib/stdlib/ebin/erl_scan.beam index 1b76ff6a83fe..ee96522758a8 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_scan.beam and b/bootstrap/lib/stdlib/ebin/erl_scan.beam differ diff --git a/bootstrap/lib/stdlib/ebin/erl_stdlib_errors.beam b/bootstrap/lib/stdlib/ebin/erl_stdlib_errors.beam index dbec403bb20e..2571094fac21 100644 Binary files a/bootstrap/lib/stdlib/ebin/erl_stdlib_errors.beam and b/bootstrap/lib/stdlib/ebin/erl_stdlib_errors.beam differ diff --git a/bootstrap/lib/stdlib/ebin/escript.beam b/bootstrap/lib/stdlib/ebin/escript.beam index eff3d1007517..d9899ace7f82 100644 Binary files a/bootstrap/lib/stdlib/ebin/escript.beam and b/bootstrap/lib/stdlib/ebin/escript.beam differ diff --git a/bootstrap/lib/stdlib/ebin/file_sorter.beam b/bootstrap/lib/stdlib/ebin/file_sorter.beam index 6abf0e6f176d..2a42ad58b990 100644 Binary files a/bootstrap/lib/stdlib/ebin/file_sorter.beam and b/bootstrap/lib/stdlib/ebin/file_sorter.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gb_sets.beam b/bootstrap/lib/stdlib/ebin/gb_sets.beam index 752ca8a0623d..4c8619d087c9 100644 Binary files a/bootstrap/lib/stdlib/ebin/gb_sets.beam and b/bootstrap/lib/stdlib/ebin/gb_sets.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gb_trees.beam b/bootstrap/lib/stdlib/ebin/gb_trees.beam index 8aa75b7d0be1..1f17fba381c0 100644 Binary files a/bootstrap/lib/stdlib/ebin/gb_trees.beam and b/bootstrap/lib/stdlib/ebin/gb_trees.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gen.beam b/bootstrap/lib/stdlib/ebin/gen.beam index 2b1db2c1f789..dc4446b1ff02 100644 Binary files a/bootstrap/lib/stdlib/ebin/gen.beam and b/bootstrap/lib/stdlib/ebin/gen.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gen_event.beam b/bootstrap/lib/stdlib/ebin/gen_event.beam index a65c9e4718d2..9816c204e54c 100644 Binary files a/bootstrap/lib/stdlib/ebin/gen_event.beam and b/bootstrap/lib/stdlib/ebin/gen_event.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gen_fsm.beam b/bootstrap/lib/stdlib/ebin/gen_fsm.beam index cdabaa74f034..aefb890b0006 100644 Binary files a/bootstrap/lib/stdlib/ebin/gen_fsm.beam and b/bootstrap/lib/stdlib/ebin/gen_fsm.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gen_server.beam b/bootstrap/lib/stdlib/ebin/gen_server.beam index a018a70a0ed6..dd155b19ca54 100644 Binary files a/bootstrap/lib/stdlib/ebin/gen_server.beam and b/bootstrap/lib/stdlib/ebin/gen_server.beam differ diff --git a/bootstrap/lib/stdlib/ebin/gen_statem.beam b/bootstrap/lib/stdlib/ebin/gen_statem.beam index 61eb6b89aa01..146dcbc33421 100644 Binary files a/bootstrap/lib/stdlib/ebin/gen_statem.beam and b/bootstrap/lib/stdlib/ebin/gen_statem.beam differ diff --git a/bootstrap/lib/stdlib/ebin/lists.beam b/bootstrap/lib/stdlib/ebin/lists.beam index 067134d28f6c..9a830e809957 100644 Binary files a/bootstrap/lib/stdlib/ebin/lists.beam and b/bootstrap/lib/stdlib/ebin/lists.beam differ diff --git a/bootstrap/lib/stdlib/ebin/maps.beam b/bootstrap/lib/stdlib/ebin/maps.beam index 3dbfa76f984b..318e091fc39c 100644 Binary files a/bootstrap/lib/stdlib/ebin/maps.beam and b/bootstrap/lib/stdlib/ebin/maps.beam differ diff --git a/bootstrap/lib/stdlib/ebin/proc_lib.beam b/bootstrap/lib/stdlib/ebin/proc_lib.beam index 16ad2ff56269..a7e63f4dd840 100644 Binary files a/bootstrap/lib/stdlib/ebin/proc_lib.beam and b/bootstrap/lib/stdlib/ebin/proc_lib.beam differ diff --git a/bootstrap/lib/stdlib/ebin/proplists.beam b/bootstrap/lib/stdlib/ebin/proplists.beam index de05dc9f9713..b7a921734d04 100644 Binary files a/bootstrap/lib/stdlib/ebin/proplists.beam and b/bootstrap/lib/stdlib/ebin/proplists.beam differ diff --git a/bootstrap/lib/stdlib/ebin/qlc.beam b/bootstrap/lib/stdlib/ebin/qlc.beam index 1971e80a6b8f..69c28cc56d95 100644 Binary files a/bootstrap/lib/stdlib/ebin/qlc.beam and b/bootstrap/lib/stdlib/ebin/qlc.beam differ diff --git a/bootstrap/lib/stdlib/ebin/qlc_pt.beam b/bootstrap/lib/stdlib/ebin/qlc_pt.beam index 656f664fae68..c9e945208ef9 100644 Binary files a/bootstrap/lib/stdlib/ebin/qlc_pt.beam and b/bootstrap/lib/stdlib/ebin/qlc_pt.beam differ diff --git a/bootstrap/lib/stdlib/ebin/shell.beam b/bootstrap/lib/stdlib/ebin/shell.beam index 78fa66ad43b2..8e99f2b243ea 100644 Binary files a/bootstrap/lib/stdlib/ebin/shell.beam and b/bootstrap/lib/stdlib/ebin/shell.beam differ diff --git a/bootstrap/lib/stdlib/ebin/shell_default.beam b/bootstrap/lib/stdlib/ebin/shell_default.beam index 8ebf59566806..6fb5f9ca4b80 100644 Binary files a/bootstrap/lib/stdlib/ebin/shell_default.beam and b/bootstrap/lib/stdlib/ebin/shell_default.beam differ diff --git a/bootstrap/lib/stdlib/ebin/sofs.beam b/bootstrap/lib/stdlib/ebin/sofs.beam index 10c3eb52e28a..f71b42b3a33b 100644 Binary files a/bootstrap/lib/stdlib/ebin/sofs.beam and b/bootstrap/lib/stdlib/ebin/sofs.beam differ diff --git a/bootstrap/lib/stdlib/ebin/string.beam b/bootstrap/lib/stdlib/ebin/string.beam index b5da368090a7..70fe45ba6c33 100644 Binary files a/bootstrap/lib/stdlib/ebin/string.beam and b/bootstrap/lib/stdlib/ebin/string.beam differ diff --git a/bootstrap/lib/stdlib/ebin/supervisor.beam b/bootstrap/lib/stdlib/ebin/supervisor.beam index ff9960f6f3bc..63d05fcdb6b7 100644 Binary files a/bootstrap/lib/stdlib/ebin/supervisor.beam and b/bootstrap/lib/stdlib/ebin/supervisor.beam differ diff --git a/bootstrap/lib/stdlib/ebin/timer.beam b/bootstrap/lib/stdlib/ebin/timer.beam index 45a7606d0a41..69748985d021 100644 Binary files a/bootstrap/lib/stdlib/ebin/timer.beam and b/bootstrap/lib/stdlib/ebin/timer.beam differ diff --git a/bootstrap/lib/stdlib/ebin/uri_string.beam b/bootstrap/lib/stdlib/ebin/uri_string.beam index 25dfa07af141..25adce194c70 100644 Binary files a/bootstrap/lib/stdlib/ebin/uri_string.beam and b/bootstrap/lib/stdlib/ebin/uri_string.beam differ diff --git a/bootstrap/lib/stdlib/ebin/zip.beam b/bootstrap/lib/stdlib/ebin/zip.beam index d1ee1a010c93..30204589dae3 100644 Binary files a/bootstrap/lib/stdlib/ebin/zip.beam and b/bootstrap/lib/stdlib/ebin/zip.beam differ diff --git a/erts/doc/src/alt_dist.xml b/erts/doc/src/alt_dist.xml index 09ac9ee5dfe9..91106ea43185 100644 --- a/erts/doc/src/alt_dist.xml +++ b/erts/doc/src/alt_dist.xml @@ -4,7 +4,7 @@
- 20002022 + 20002023 Ericsson AB. All Rights Reserved. diff --git a/erts/doc/src/erl_dist_protocol.xml b/erts/doc/src/erl_dist_protocol.xml index 2baaf5f3d25e..e2bf7c447e9c 100644 --- a/erts/doc/src/erl_dist_protocol.xml +++ b/erts/doc/src/erl_dist_protocol.xml @@ -1471,7 +1471,7 @@ DiB == gen_digest(ChA, ICA)?

Process identifier of the group leader of the newly created process.

 {Module :: atom(), Function :: atom(), Arity :: integer() >= 0} -

Entry point for the the new process.

 +

Entry point for the new process.

 OptList :: [term()]

A proper list of spawn options to use when spawning.

 ArgList :: [term()] diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml index 3c6a1c594ddc..a8f59abbda63 100644 --- a/erts/doc/src/erl_nif.xml +++ b/erts/doc/src/erl_nif.xml @@ -1112,7 +1112,7 @@ typedef struct { size_tenif_binary_to_term(ErlNifEnv *env, const unsigned char* data, size_t size, ERL_NIF_TERM *term, - ErlNifBinaryToTerm opts) + unsigned int opts) Create a term from the external format.

Creates a term that is the result of decoding the binary data at diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index 8ba476e57c49..54da652bb735 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -8728,13 +8728,13 @@ true corresponds to scheduler number 1 and so on. If support for dirty schedulers exist, an element with the value for the dirty CPU run queue and its associated dirty CPU schedulers - follow and then as last element the value for the the dirty + follow and then as last element the value for the dirty IO run queue and its associated dirty IO schedulers follow. The information is not gathered atomically. That is, the result is not necessarily a consistent snapshot of the state, but instead quite efficiently gathered.

Each normal scheduler has one run queue that it - manages. If dirty schedulers schedulers are supported, all + manages. If dirty schedulers are supported, all dirty CPU schedulers share one run queue, and all dirty IO schedulers share one run queue. That is, we have multiple normal run queues, one dirty CPU run queue and one dirty @@ -9013,7 +9013,7 @@ lists:map( necessarily a consistent snapshot of the state, but instead quite efficiently gathered.

Each normal scheduler has one run queue that it - manages. If dirty schedulers schedulers are supported, all + manages. If dirty schedulers are supported, all dirty CPU schedulers share one run queue, and all dirty IO schedulers share one run queue. That is, we have multiple normal run queues, one dirty CPU run queue and one dirty diff --git a/erts/doc/src/init.xml b/erts/doc/src/init.xml index 94d424140393..a5110ac354e8 100644 --- a/erts/doc/src/init.xml +++ b/erts/doc/src/init.xml @@ -243,8 +243,7 @@

The support for loading of code from archive files is experimental. The only purpose of releasing it before it is ready is to obtain early feedback. The file format, semantics, - interfaces, and so on, can be changed in a future release. The - -code_path_choice flag is also experimental.

+ interfaces, and so on, can be changed in a future release.

The init module interprets the following command-line flags:

@@ -271,13 +270,8 @@ and an ebin directory in an archive file.

-

This flag is particular - useful when you want to elaborate with code loading from - archives without editing the boot script. For more - information about interpretation of boot scripts, see - script(4). - The flag has also a similar effect on how the code server works; see - code(3).

+

It defaults to strict from OTP 27 and this option + is scheduled for removal in OTP 28.

-epmd_module Module diff --git a/erts/doc/src/notes.xml b/erts/doc/src/notes.xml index ba152477e3ce..e907c6521162 100644 --- a/erts/doc/src/notes.xml +++ b/erts/doc/src/notes.xml @@ -31,6 +31,193 @@

This document describes the changes made to the ERTS application.

+
Erts 14.2.1 + +
Fixed Bugs and Malfunctions + + +

+ Removed unnecessary PCRE source tar-ball.

+

+ Own Id: OTP-18902

+ + +
+ +
+ +
Erts 14.2 + +
Fixed Bugs and Malfunctions + + +

+ Fix erl.exe to to restore the console to its original + state when exiting. This bug was introduced in OTP 26.0 + and only happens when erl.exe is run in cmd.exe.

+

+ Own Id: OTP-18751 Aux Id: GH-7621 GH-7548

+ + +

+ Fix faulty debug assert when page size is larger than + 16kb, like on PowerPC. Did crash debug VM directly at + start.

+

+ Own Id: OTP-18802

+ + +

zlib will no longer randomly return garbage + (negative) Adler32 checksums.

+

+ Own Id: OTP-18811 Aux Id: ERIERL-994

+ + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +

+ A process with message_queue_data configured as + off_heap could end up in an inconsistent state + when being receive traced, inspected using + process_info/2 with the message_queue_len + item, or inspected using the break menu (CTRL-C). When it + ended up in this inconsistent state, it was not enqueued + into a run queue even though it was set in a runnable + state.This also effected signals being sent to the + process after it had gotten into this inconsistent state, + in such a way that it was from this point not possible to + communicate with it.

+

+ Own Id: OTP-18838 Aux Id: PR-7822, GH-7801

+ + +

+ A race occurring when a process was selected for dirty + execution simultaneously as it was scheduled for handling + a signal could cause the process to end up in an + inconsistent state. When it ended up in this inconsistent + state, it was not enqueued into a run queue even though + it was set in a runnable state. This also effected + signals being sent to the process after it had gotten + into this inconsistent state, in such a way that it was + from this point not possible to communicate with it.

+

+ Own Id: OTP-18839 Aux Id: PR-7822, GH-7801

+ + +

+ When a process had to to wait in the run queue for a long + time before being selected for dirty execution, it could + not receive signals. This caused inspection of such a + process, for example using process_info/2, to take + a long time.

+

+ This issue was introduced in OTP 25.3.2.6 and 26.1 when + fixing an issue where a constant flow of signals + prevented a process from being able to execute dirty.

+

+ Own Id: OTP-18841 Aux Id: PR-7822, GH-7801, OTP-18737

+ + +

Fixed a bug in the JIT that miscompiled large + select_val instructions.

+

+ Own Id: OTP-18842

+ + +

+ Fix bug on Windows where large writes to + standard_io could cause duplicate data to be + written.

+

+ Own Id: OTP-18871 Aux Id: GH-7838

+ + +

+ The struct ip_mreqn field imr_ifindex had + got an incorrect byte order conversion that has been + corrected.

+

+ Own Id: OTP-18880 Aux Id: GH-7736, PR-7761

+ + +

+ On OTP 24 and OTP 25, incoming distributed messages + larger than 64 KiB sent using an alias leaked memory if + the alias had been removed prior to entering the node. + This issue was not present on OTP 26.

+

+ Incoming distributed messages larger than 64 KiB sent + using an alias which had been removed on the receiving + node could crash the node. This crash was quite unlikely + on OTP 24 and OTP 25, but very likely on OTP 26.

+

+ 'DOWN' signals with exit reason larger than 64 KiB + directed towards a process on a node with a not matching + creation leaked memory on the receiving node. Such + signals should however be very rare.

+

+ Own Id: OTP-18885 Aux Id: GH-7834, GH-7890, PR-7915

+ + +
+ + +
Improvements and New Features + + +

+ Add Windows support for DGRAM socket connect.

+

+ Own Id: OTP-18762

+ + +

+ process_info/2 + now supports lookup of values for specific keys in the + process dictionary. For example, {{dictionary, Key}, + Value} = process_info(Pid, {dictionary, Key}).

+

+ Own Id: OTP-18765 Aux Id: PR-7707

+ + +

+ Removed unnecessary regexp library used when generating + yielding BIFs.

+

+ Own Id: OTP-18830 Aux Id: PR-7823

+ + +

+ Fix tty restore when +Bc is used.

+

+ Own Id: OTP-18872 Aux Id: GH-7832

+ + +

+ Replaced old md5 implementation with an implementation + from OpenSSL.

+

+ Own Id: OTP-18877

+ + +

+ Removed unused makewhatis script.

+

+ Own Id: OTP-18899

+ + +
+ +
+
Erts 14.1.1
Fixed Bugs and Malfunctions @@ -985,6 +1172,124 @@
+
Erts 13.2.2.5 + +
Fixed Bugs and Malfunctions + + +

+ Fix faulty debug assert when page size is larger than + 16kb, like on PowerPC. Did crash debug VM directly at + start.

+

+ Own Id: OTP-18802

+ + +

+ A process with message_queue_data configured as + off_heap could end up in an inconsistent state + when being receive traced, inspected using + process_info/2 with the message_queue_len + item, or inspected using the break menu (CTRL-C). When it + ended up in this inconsistent state, it was not enqueued + into a run queue even though it was set in a runnable + state.This also effected signals being sent to the + process after it had gotten into this inconsistent state, + in such a way that it was from this point not possible to + communicate with it.

+

+ Own Id: OTP-18838 Aux Id: PR-7822, GH-7801

+ + +

+ A race occurring when a process was selected for dirty + execution simultaneously as it was scheduled for handling + a signal could cause the process to end up in an + inconsistent state. When it ended up in this inconsistent + state, it was not enqueued into a run queue even though + it was set in a runnable state. This also effected + signals being sent to the process after it had gotten + into this inconsistent state, in such a way that it was + from this point not possible to communicate with it.

+

+ Own Id: OTP-18839 Aux Id: PR-7822, GH-7801

+ + +

+ When a process had to to wait in the run queue for a long + time before being selected for dirty execution, it could + not receive signals. This caused inspection of such a + process, for example using process_info/2, to take + a long time.

+

+ This issue was introduced in OTP 25.3.2.6 and 26.1 when + fixing an issue where a constant flow of signals + prevented a process from being able to execute dirty.

+

+ Own Id: OTP-18841 Aux Id: PR-7822, GH-7801, OTP-18737

+ + +

Fixed a bug in the JIT that miscompiled large + select_val instructions.

+

+ Own Id: OTP-18842

+ + +

+ On OTP 24 and OTP 25, incoming distributed messages + larger than 64 KiB sent using an alias leaked memory if + the alias had been removed prior to entering the node. + This issue was not present on OTP 26.

+

+ Incoming distributed messages larger than 64 KiB sent + using an alias which had been removed on the receiving + node could crash the node. This crash was quite unlikely + on OTP 24 and OTP 25, but very likely on OTP 26.

+

+ 'DOWN' signals with exit reason larger than 64 KiB + directed towards a process on a node with a not matching + creation leaked memory on the receiving node. Such + signals should however be very rare.

+

+ Own Id: OTP-18885 Aux Id: GH-7834, GH-7890, PR-7915

+ + +

+ Removed unnecessary PCRE source tar-ball.

+

+ Own Id: OTP-18902

+ + +
+ + +
Improvements and New Features + + +

+ Removed unnecessary regexp library used when generating + yielding BIFs.

+

+ Own Id: OTP-18830 Aux Id: PR-7823

+ + +

+ Replaced old md5 implementation with an implementation + from OpenSSL.

+

+ Own Id: OTP-18877

+ + +

+ Removed unused makewhatis script.

+

+ Own Id: OTP-18899

+ + +
+ +
+
Erts 13.2.2.4
Fixed Bugs and Malfunctions @@ -2623,6 +2928,74 @@
+
Erts 12.3.2.15 + +
Fixed Bugs and Malfunctions + + +

+ Fix faulty debug assert when page size is larger than + 16kb, like on PowerPC. Did crash debug VM directly at + start.

+

+ Own Id: OTP-18802

+ + +

+ On OTP 24 and OTP 25, incoming distributed messages + larger than 64 KiB sent using an alias leaked memory if + the alias had been removed prior to entering the node. + This issue was not present on OTP 26.

+

+ Incoming distributed messages larger than 64 KiB sent + using an alias which had been removed on the receiving + node could crash the node. This crash was quite unlikely + on OTP 24 and OTP 25, but very likely on OTP 26.

+

+ 'DOWN' signals with exit reason larger than 64 KiB + directed towards a process on a node with a not matching + creation leaked memory on the receiving node. Such + signals should however be very rare.

+

+ Own Id: OTP-18885 Aux Id: GH-7834, GH-7890, PR-7915

+ + +

+ Removed unnecessary PCRE source tar-ball.

+

+ Own Id: OTP-18902

+ + +
+ + +
Improvements and New Features + + +

+ Removed unnecessary regexp library used when generating + yielding BIFs.

+

+ Own Id: OTP-18830 Aux Id: PR-7823

+ + +

+ Replaced old md5 implementation with an implementation + from OpenSSL.

+

+ Own Id: OTP-18877

+ + +

+ Removed unused makewhatis script.

+

+ Own Id: OTP-18899

+ + +
+ +
+
Erts 12.3.2.14
Fixed Bugs and Malfunctions @@ -4587,7 +4960,7 @@ map in a single pass.

maps:from_keys/2 constructs a map from a list of - keys and a single value and can be used to to optimize + keys and a single value and can be used to optimize sets operations such as from_list/1, filter/2, intersection/2, and subtract/2.

@@ -8336,7 +8709,7 @@

The Kernel application's User's Guide now contain a - Logger Cookbook with with common usage patterns.

+ Logger Cookbook with common usage patterns.

Own Id: OTP-16208

@@ -12135,7 +12508,7 @@ been called, all exceptions will now contain the entry for one function (despite the zero). It used to be that a hand-made stack backtrace passed to - erlang:raise/3 would be be truncated to an empty + erlang:raise/3 would be truncated to an empty list.

Own Id: OTP-15026

@@ -20798,7 +21171,7 @@

The previous default of a maximum of 32768 simultaneous processes has been raised to 262144. This value can be - changed using the the +P command line flag of erl(1). Note that the value passed now is considered as a hint, and that actual @@ -21602,7 +21975,7 @@

Calling trace_info/2 asking for information about a - function that had native could could crash the run-time + function that had native could crash the run-time system.

Own Id: OTP-9886

@@ -25413,7 +25786,7 @@

Emulator flags in an escript were earlier inherited to - emulators started from from the emulator running the + emulators started from the emulator running the escript. For example when an escript invoked os:cmd("erl"), the new emulator were given erroneous emulator flags. This bug has now been fixed

@@ -26873,7 +27246,7 @@ In the section about binary construction, the reference manual now mentions what happens when an integer value does not fit into an integer segment of size N (namely, - that the N least significant bits will be put into into + that the N least significant bits will be put into the binary and that the most significant bits will be silently discarded). (Thanks to Edwin Fine.)

@@ -29653,7 +30026,7 @@

The undocumented option (for the - module) did not not work correctly when + module) did not work correctly when there were multiple continuation lines. (Thanks to Per Hedeland.)

Own Id: OTP-5945

diff --git a/erts/doc/src/tty.xml b/erts/doc/src/tty.xml index a14e3a2b1de9..054e9ab219d6 100644 --- a/erts/doc/src/tty.xml +++ b/erts/doc/src/tty.xml @@ -229,6 +229,18 @@ erl M-c Clear current expression + + M-h + Display help for the module or function closest on the left of the cursor. + + + PageUp + Scroll the expand, search or help buffer 5 lines upwards. + + + PageDown + Scroll the expand, search or help buffer 5 lines downwards. + tty Text Editing
diff --git a/erts/emulator/beam/beam_bif_load.c b/erts/emulator/beam/beam_bif_load.c index 27d607abf511..7fe9e9960c69 100644 --- a/erts/emulator/beam/beam_bif_load.c +++ b/erts/emulator/beam/beam_bif_load.c @@ -1224,19 +1224,21 @@ any_heap_refs(Eterm* start, Eterm* end, char* mod_start, Uint mod_size) break; case TAG_PRIMARY_HEADER: if (!header_is_transparent(val)) { - Eterm* new_p; - if (header_is_bin_matchstate(val)) { - ErlBinMatchState *ms = (ErlBinMatchState*) p; - ErlBinMatchBuffer *mb = &(ms->mb); - if (ErtsInArea(mb->orig, mod_start, mod_size)) { + Eterm* new_p; + + if (val == HEADER_SUB_BITS) { + ErlSubBits *sb = (ErlSubBits*) p; + + if (ErtsInArea(sb->orig, mod_start, mod_size)) { return 1; } } - new_p = p + thing_arityval(val); - ASSERT(start <= new_p && new_p < end); - p = new_p; - } - } + + new_p = p + thing_arityval(val); + ASSERT(start <= new_p && new_p < end); + p = new_p; + } + } } return 0; } diff --git a/erts/emulator/beam/beam_code.h b/erts/emulator/beam/beam_code.h index 720ae41b204d..0cb6896641c0 100644 --- a/erts/emulator/beam/beam_code.h +++ b/erts/emulator/beam/beam_code.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2020-2021. All Rights Reserved. + * Copyright Ericsson AB 2020-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/beam/beam_common.c b/erts/emulator/beam/beam_common.c index 57d179f5d79c..0f2d4aff094a 100644 --- a/erts/emulator/beam/beam_common.c +++ b/erts/emulator/beam/beam_common.c @@ -1730,7 +1730,12 @@ call_fun(Process* p, /* Current process. */ if (ERTS_LIKELY(code_ptr != beam_unloaded_fun && fun_arity(funp) == arity)) { - for (int i = 0, num_free = fun_num_free(funp); i < num_free; i++) { + /* Copy the free variables, skipping the FunRef in the environment. + * + * Note that we avoid using fun_num_free as it asserts that the + * argument is a local function, which we don't need to care about + * here. */ + for (int i = 0, num_free = fun_env_size(funp) - 1; i < num_free; i++) { reg[i + arity] = funp->env[i]; } @@ -1739,7 +1744,6 @@ call_fun(Process* p, /* Current process. */ DTRACE_LOCAL_CALL(p, erts_code_to_codemfa(code_ptr)); } else { Export *ep = funp->entry.exp; - ASSERT(is_external_fun(funp) && funp->next == NULL); DTRACE_GLOBAL_CALL(p, &ep->info.mfa); } #endif diff --git a/erts/emulator/beam/beam_file.c b/erts/emulator/beam/beam_file.c index 770282e271a8..7487fe16d643 100644 --- a/erts/emulator/beam/beam_file.c +++ b/erts/emulator/beam/beam_file.c @@ -1531,7 +1531,8 @@ static int marshal_allocation_list(BeamReader *reader, Sint *res) { LoadAssert(beamreader_read_tagged(reader, &val)); LoadAssert(val.tag == TAG_u); - LoadAssert(val.word_value <= ERTS_SINT32_MAX / FLOAT_SIZE_OBJECT); + LoadAssert(val.word_value <= ERTS_SINT32_MAX / + MAX(FLOAT_SIZE_OBJECT, ERL_FUN_SIZE + 1)); number = val.word_value; switch(kind) { @@ -1544,8 +1545,11 @@ static int marshal_allocation_list(BeamReader *reader, Sint *res) { sum += FLOAT_SIZE_OBJECT * number; break; case 2: - LoadAssert(sum <= (ERTS_SINT32_MAX - ERL_FUN_SIZE * number)); - sum += ERL_FUN_SIZE * number; + LoadAssert(sum <= (ERTS_SINT32_MAX - (ERL_FUN_SIZE + 1) * number)); + + /* This is always a local fun, so we need to add one word to + * reserve space for its `FunRef`. */ + sum += (ERL_FUN_SIZE + 1) * number; break; default: LoadError("Invalid allocation tag"); diff --git a/erts/emulator/beam/beam_load.c b/erts/emulator/beam/beam_load.c index c0c360836afc..6a9498c7de53 100644 --- a/erts/emulator/beam/beam_load.c +++ b/erts/emulator/beam/beam_load.c @@ -662,16 +662,21 @@ erts_release_literal_area(ErtsLiteralArea* literal_area) erts_bin_release(bin); break; } - case FUN_SUBTAG: + case FUN_REF_SUBTAG: { - /* We _KNOW_ that this is a local fun, otherwise it would not - * be part of the off-heap list. */ - ErlFunEntry* fe = ((ErlFunThing*)oh)->entry.fun; - - ASSERT(is_local_fun((ErlFunThing*)oh)); - - if (erts_refc_dectest(&fe->refc, 0) == 0) { - erts_erase_fun_entry(fe); + ErlFunEntry* fe = ((FunRef*)oh)->entry; + + /* All fun entries are NULL during module loading, before the + * code is finalized, so we need to tolerate it to avoid + * crashing in the prepared code destructor. + * + * Strictly speaking it would be nice to crash when we see this + * outside of loading, but it's too complicated to keep track + * of whether we are. */ + if (fe != NULL) { + if (erts_refc_dectest(&fe->refc, 0) == 0) { + erts_erase_fun_entry(fe); + } } break; } diff --git a/erts/emulator/beam/bif.tab b/erts/emulator/beam/bif.tab index 614e8357c832..99344495c04e 100644 --- a/erts/emulator/beam/bif.tab +++ b/erts/emulator/beam/bif.tab @@ -793,3 +793,4 @@ bif code:get_coverage_mode/1 bif code:get_coverage/2 bif code:reset_coverage/1 bif code:set_coverage_mode/1 +bif ets:update_element/4 diff --git a/erts/emulator/beam/code_ix.h b/erts/emulator/beam/code_ix.h index 1c0472152306..0ddb2b1b8f1e 100644 --- a/erts/emulator/beam/code_ix.h +++ b/erts/emulator/beam/code_ix.h @@ -277,12 +277,8 @@ int erts_has_code_stage_permission(void); int erts_has_code_mod_permission(void); #endif -/* module/function/arity can be NIL/NIL/-1 when the MFA is pointing to some - invalid code, for instance unloaded_fun. */ #define ASSERT_MFA(MFA) \ - ASSERT((is_atom((MFA)->module) || is_nil((MFA)->module)) && \ - (is_atom((MFA)->function) || is_nil((MFA)->function)) && \ - (((MFA)->arity >= 0 && (MFA)->arity < 1024) || (MFA)->arity == -1)) + ASSERT(is_atom((MFA)->module) && is_atom((MFA)->function)) extern erts_atomic32_t the_active_code_index; extern erts_atomic32_t the_staging_code_index; diff --git a/erts/emulator/beam/copy.c b/erts/emulator/beam/copy.c index b6c50a8c6ea1..89c5ed08451c 100644 --- a/erts/emulator/beam/copy.c +++ b/erts/emulator/beam/copy.c @@ -36,6 +36,12 @@ #include "dtrace-wrapper.h" #include "erl_global_literals.h" +/* The shared_xyz functions temporarily use the primary tag bits of the header + * word to store whether the term has been visited/processed before, preventing + * us from directly comparing the header to constants (e.g. HEADER_BIN_REF) and + * breaking an assertion in * `thing_subtag`. */ +#define shared_thing_subtag(Header) _unchecked_thing_subtag(Header) + static void move_one_frag(Eterm** hpp, ErlHeapFragment*, ErlOffHeap*, int); /* @@ -137,21 +143,24 @@ Uint size_object_x(Eterm obj, erts_literal_area_t *litopt) } obj = *++ptr; break; - case FUN_SUBTAG: - { + case FUN_REF_SUBTAG: + sum += ERL_FUN_REF_SIZE; + goto pop_next; + case FUN_SUBTAG: + { const ErlFunThing* funp = (ErlFunThing*)fun_val(obj); ASSERT(ERL_FUN_SIZE == (1 + thing_arityval(hdr))); - sum += ERL_FUN_SIZE + fun_num_free(funp); + sum += ERL_FUN_SIZE + fun_env_size(funp); - for (int i = 1; i < fun_num_free(funp); i++) { + for (int i = 1; i < fun_env_size(funp); i++) { obj = funp->env[i]; if (!IS_CONST(obj)) { ESTACK_PUSH(s, obj); } } - if (fun_num_free(funp) > 0) { + if (fun_env_size(funp) > 0) { obj = funp->env[0]; break; } @@ -207,19 +216,29 @@ Uint size_object_x(Eterm obj, erts_literal_area_t *litopt) break; case SUB_BITS_SUBTAG: { - /* Note that we copy the structure verbatim even if the - * size is lower than the off-heap limit as it was most - * likely created that way on purpose. - * - * We also include the size of the attached BinRef to - * save us another lap through the main loop. */ - sum += ERL_REFC_BITS_SIZE; + const ErlSubBits *sb = (ErlSubBits*)boxed_val(obj); + const Eterm *underlying = boxed_val(sb->orig); + + if (*underlying == HEADER_BIN_REF) { + /* Note that we copy the structure verbatim even if + * the size is lower than the off-heap limit as it + * was most likely created that way on purpose. + * + * We also include the size of the attached BinRef + * to save us another lap through the main loop. */ + sum += ERL_REFC_BITS_SIZE; + } else { + /* When an ErlSubBits is used as a match context it + * may refer to an on-heap bitstring instead of a + * BinRef, in which case we need to copy the + * underlying data instead of the match context. */ + ASSERT(erl_sub_bits_is_match_context(sb)); + sum += heap_bits_size(sb->end - sb->start); + } + goto pop_next; } break; - case BIN_MATCHSTATE_SUBTAG: - erts_exit(ERTS_ABORT_EXIT, - "size_object: matchstate term not allowed"); default: sum += thing_arityval(hdr) + 1; goto pop_next; @@ -377,13 +396,16 @@ Uint size_shared(Eterm obj) } goto pop_next; } - case FUN_SUBTAG: { + case FUN_REF_SUBTAG: + sum += ERL_FUN_REF_SIZE; + goto pop_next; + case FUN_SUBTAG: { const ErlFunThing* funp = (ErlFunThing *) ptr; ASSERT(ERL_FUN_SIZE == (1 + thing_arityval(hdr))); - sum += ERL_FUN_SIZE + fun_num_free(funp); + sum += ERL_FUN_SIZE + fun_env_size(funp); - for (int i = 0; i < fun_num_free(funp); i++) { + for (int i = 0; i < fun_env_size(funp); i++) { obj = funp->env[i]; if (!IS_CONST(obj)) { EQUEUE_PUT(s, obj); @@ -398,8 +420,21 @@ Uint size_shared(Eterm obj) } case SUB_BITS_SUBTAG: { const ErlSubBits *sb = (ErlSubBits*)ptr; - EQUEUE_PUT(s, sb->orig); - sum += ERL_SUB_BITS_SIZE; + const Eterm *underlying = boxed_val(sb->orig); + + if (shared_thing_subtag(*underlying) == BIN_REF_SUBTAG) { + EQUEUE_PUT(s, sb->orig); + sum += ERL_SUB_BITS_SIZE; + } else { + /* This is a match context referring to a heap bitstring. + * As this is fairly rare and sharing multiple instances of + * the same match context is rarer still, we'll ignore any + * potential sharing here and assume that a blunt copy will + * be made. */ + ASSERT(erl_sub_bits_is_match_context(sb)); + sum += heap_bits_size(sb->end - sb->start); + } + goto pop_next; } case MAP_SUBTAG: @@ -436,9 +471,6 @@ Uint size_shared(Eterm obj) default: erts_exit(ERTS_ABORT_EXIT, "size_shared: bad hashmap type %d\n", MAP_HEADER_TYPE(hdr)); } - case BIN_MATCHSTATE_SUBTAG: - erts_exit(ERTS_ABORT_EXIT, - "size_shared: matchstate term not allowed"); default: sum += thing_arityval(hdr) + 1; goto pop_next; @@ -526,10 +558,10 @@ Uint size_shared(Eterm obj) } goto cleanup_next; } - case FUN_SUBTAG: { + case FUN_SUBTAG: { const ErlFunThing *funp = (ErlFunThing *) ptr; - for (int i = 0; i < fun_num_free(funp); i++) { + for (int i = 0; i < fun_env_size(funp); i++) { obj = funp->env[i]; if (!IS_CONST(obj)) { EQUEUE_PUT_UNCHECKED(s, obj); @@ -539,7 +571,13 @@ Uint size_shared(Eterm obj) } case SUB_BITS_SUBTAG: { const ErlSubBits *sb = (ErlSubBits*)ptr; - EQUEUE_PUT_UNCHECKED(s, sb->orig); + const Eterm *underlying = boxed_val(sb->orig); + + /* We only visit BinRefs: see comment above on match + * contexts. */ + if (shared_thing_subtag(*underlying) == BIN_REF_SUBTAG) { + EQUEUE_PUT_UNCHECKED(s, sb->orig); + } goto cleanup_next; } case MAP_SUBTAG: @@ -756,62 +794,101 @@ Eterm copy_struct_x(Eterm obj, Uint sz, Eterm** hpp, ErlOffHeap* off_heap, break; case SUB_BITS_SUBTAG: { - ErlSubBits *from_sb, *to_sb; - BinRef *from_br, *to_br; + ErlSubBits *from_sb = (ErlSubBits*)objp; + Eterm *underlying = boxed_val(from_sb->orig); - /* As BinRefs are only reachable through ErlSubBits and we - * aren't doing a shared copy, and certain functions like - * copy_ets_element assume that outer objects appear before - * inner ones on the heap, we'll handle them together - * here. */ - hbot -= ERL_BIN_REF_SIZE; - to_br = (BinRef*)hbot; - hbot -= ERL_SUB_BITS_SIZE; - to_sb = (ErlSubBits*)hbot; + if (*underlying == HEADER_BIN_REF) { + BinRef *from_br, *to_br; + ErlSubBits *to_sb; - from_sb = (ErlSubBits*)objp; - from_br = (BinRef*)boxed_val(from_sb->orig); + from_br = (BinRef*)underlying; - ASSERT(from_br->thing_word == HEADER_BIN_REF); - erts_pin_writable_binary(from_br); + /* As BinRefs are only reachable through ErlSubBits and + * we aren't doing a shared copy, and certain functions + * like copy_ets_element assume that outer objects + * appear before inner ones on the heap, we'll handle + * them together here. */ + hbot -= ERL_BIN_REF_SIZE; + to_br = (BinRef*)hbot; + hbot -= ERL_SUB_BITS_SIZE; + to_sb = (ErlSubBits*)hbot; - erts_refc_inc(&(from_br->val)->intern.refc, 2); + ASSERT(from_br->thing_word == HEADER_BIN_REF); + erts_pin_writable_binary(from_sb, from_br); - *to_sb = *from_sb; - *to_br = *from_br; + erts_refc_inc(&(from_br->val)->intern.refc, 2); - to_br->next = off_heap->first; - off_heap->first = (struct erl_off_heap_header*)to_br; - ERTS_BR_OVERHEAD(off_heap, to_br); + *to_sb = *from_sb; + *to_br = *from_br; - to_sb->orig = make_bitstring(to_br); - *argp = make_bitstring(to_sb); + ASSERT(erl_sub_bits_is_normal(to_sb)); + to_sb->orig = make_boxed((Eterm*)to_br); + + to_br->next = off_heap->first; + off_heap->first = (struct erl_off_heap_header*)to_br; + ERTS_BR_OVERHEAD(off_heap, to_br); + + *argp = make_bitstring(to_sb); + } else { + /* When an ErlSubBits is used as a match context it may + * refer to an on-heap bitstring instead of a BinRef, + * in which case we need to copy the underlying data + * instead of the match context. */ + Uint size = from_sb->end - from_sb->start; + + ASSERT(erl_sub_bits_is_match_context(from_sb)); + + hbot -= heap_bits_size(size); + *argp = HEAP_BITSTRING(hbot, + erl_sub_bits_get_base(from_sb), + from_sb->start, + size); + } } break; - case FUN_SUBTAG: - { + case FUN_REF_SUBTAG: + { + const FunRef *src_ref = (const FunRef *)objp; + FunRef *dst_ref; + + hbot -= ERL_FUN_REF_SIZE; + dst_ref = (FunRef *)hbot; + + dst_ref->thing_word = HEADER_FUN_REF; + dst_ref->entry = src_ref->entry; + + dst_ref->next = off_heap->first; + off_heap->first = (struct erl_off_heap_header*)dst_ref; + + /* All fun entries are NULL during module loading, before + * the code is finalized. + * + * Strictly speaking it would be nice to crash when we see + * this outside of loading, but it's too complicated to + * keep track of whether we are. */ + if (dst_ref->entry != NULL) { + erts_refc_inc(&(dst_ref->entry)->refc, 2); + } + + *argp = make_boxed((Eterm*)dst_ref); + } + break; + case FUN_SUBTAG: + { const ErlFunThing *src_fun = (const ErlFunThing *)objp; ErlFunThing *dst_fun = (ErlFunThing *)htop; *dst_fun = *src_fun; - for (int i = 0; i < fun_num_free(dst_fun); i++) { + for (int i = 0; i < fun_env_size(dst_fun); i++) { dst_fun->env[i] = src_fun->env[i]; } ASSERT(&htop[ERL_FUN_SIZE] == &dst_fun->env[0]); - htop = &dst_fun->env[fun_num_free(dst_fun)]; + htop = &dst_fun->env[fun_env_size(dst_fun)]; *argp = make_fun(dst_fun); - - if (is_local_fun(dst_fun)) { - dst_fun->next = off_heap->first; - off_heap->first = (struct erl_off_heap_header*)dst_fun; - erts_refc_inc(&dst_fun->entry.fun->refc, 2); - } else { - ASSERT(is_external_fun(dst_fun) && dst_fun->next == NULL); - } - } - break; + } + break; case EXTERNAL_PID_SUBTAG: case EXTERNAL_PORT_SUBTAG: case EXTERNAL_REF_SUBTAG: @@ -863,9 +940,6 @@ Eterm copy_struct_x(Eterm obj, Uint sz, Eterm** hpp, ErlOffHeap* off_heap, erts_exit(ERTS_ABORT_EXIT, "copy_struct: bad hashmap type %d\n", MAP_HEADER_TYPE(hdr)); } break; - case BIN_MATCHSTATE_SUBTAG: - erts_exit(ERTS_ABORT_EXIT, - "copy_struct: matchstate term not allowed"); case REF_SUBTAG: if (is_magic_ref_thing(objp)) { ErtsMRefThing *mreft = (ErtsMRefThing *) objp; @@ -904,11 +978,12 @@ Eterm copy_struct_x(Eterm obj, Uint sz, Eterm** hpp, ErlOffHeap* off_heap, " not equal to copy %T\n", org_obj, res); } - if (htop != hbot) + if (htop != hbot) { erts_exit(ERTS_ABORT_EXIT, "Internal error in copy_struct() when copying %T:" " htop=%p != hbot=%p (sz=%beu)\n", org_obj, htop, hbot, org_sz); + } #else if (htop > hbot) { erts_exit(ERTS_ABORT_EXIT, @@ -1188,9 +1263,9 @@ Uint copy_shared_calculate(Eterm obj, erts_shcopy_t *info) const ErlFunThing* funp = (ErlFunThing *) ptr; ASSERT(ERL_FUN_SIZE == (1 + thing_arityval(hdr))); - sum += ERL_FUN_SIZE + fun_num_free(funp); + sum += ERL_FUN_SIZE + fun_env_size(funp); - for (int i = 0; i < fun_num_free(funp); i++) { + for (int i = 0; i < fun_env_size(funp); i++) { obj = funp->env[i]; if (!IS_CONST(obj)) { EQUEUE_PUT(s, obj); @@ -1200,14 +1275,37 @@ Uint copy_shared_calculate(Eterm obj, erts_shcopy_t *info) goto pop_next; } case BIN_REF_SUBTAG: { - erts_pin_writable_binary((BinRef*)ptr); sum += ERL_BIN_REF_SIZE; goto pop_next; } case SUB_BITS_SUBTAG: { - const ErlSubBits *sb = (ErlSubBits*)ptr; - EQUEUE_PUT(s, sb->orig); - sum += ERL_SUB_BITS_SIZE; + ErlSubBits *sb = (ErlSubBits*)ptr; + Eterm *underlying = boxed_val(sb->orig); + Eterm orig_hdr = *underlying; + + switch (primary_tag(orig_hdr)) { + case BOXED_SHARED_UNPROCESSED: + case BOXED_SHARED_PROCESSED: + orig_hdr = SHTABLE_X(t, orig_hdr >> _TAG_PRIMARY_SIZE); + break; + } + + if (shared_thing_subtag(orig_hdr) == BIN_REF_SUBTAG) { + erts_pin_writable_binary(sb, (BinRef*)underlying); + + EQUEUE_PUT(s, sb->orig); + sum += ERL_SUB_BITS_SIZE; + } else { + /* This is a match context referring to an on-heap + * bitstring. + * + * As this is fairly rare and sharing multiple instances of + * the same match context is rarer still, we'll make a + * blunt copy that ignores sharing. */ + ASSERT(erl_sub_bits_is_match_context(sb)); + sum += heap_bits_size(sb->end - sb->start); + } + goto pop_next; } case MAP_SUBTAG: @@ -1247,9 +1345,6 @@ Uint copy_shared_calculate(Eterm obj, erts_shcopy_t *info) default: erts_exit(ERTS_ABORT_EXIT, "copy_shared_calculate: bad hashmap type %d\n", MAP_HEADER_TYPE(hdr)); } - case BIN_MATCHSTATE_SUBTAG: - erts_exit(ERTS_ABORT_EXIT, - "size_shared: matchstate term not allowed"); default: sum += thing_arityval(hdr) + 1; goto pop_next; @@ -1489,7 +1584,23 @@ Uint copy_shared_perform_x(Eterm obj, Uint size, erts_shcopy_t *info, } goto cleanup_next; } - case FUN_SUBTAG: { + case FUN_REF_SUBTAG: + { + const FunRef *src_ref = (const FunRef *)ptr; + FunRef *dst_ref = (FunRef *)hp; + + dst_ref->thing_word = HEADER_FUN_REF; + dst_ref->entry = src_ref->entry; + + dst_ref->next = off_heap->first; + off_heap->first = (struct erl_off_heap_header*)dst_ref; + erts_refc_inc(&(dst_ref->entry)->refc, 2); + + *resp = make_boxed((Eterm*)dst_ref); + hp += ERL_FUN_REF_SIZE; + } + goto cleanup_next; + case FUN_SUBTAG: { const ErlFunThing *src_fun = (const ErlFunThing *)ptr; ErlFunThing *dst_fun = (ErlFunThing *)hp; @@ -1499,7 +1610,7 @@ Uint copy_shared_perform_x(Eterm obj, Uint size, erts_shcopy_t *info, * restore it. */ dst_fun->thing_word = hdr; - for (int i = 0; i < fun_num_free(dst_fun); i++) { + for (int i = 0; i < fun_env_size(dst_fun); i++) { obj = src_fun->env[i]; if (!IS_CONST(obj)) { @@ -1511,20 +1622,12 @@ Uint copy_shared_perform_x(Eterm obj, Uint size, erts_shcopy_t *info, } ASSERT(&hp[ERL_FUN_SIZE] == &dst_fun->env[0]); - hp = &dst_fun->env[fun_num_free(dst_fun)]; + hp = &dst_fun->env[fun_env_size(dst_fun)]; *resp = make_fun(dst_fun); - if (is_local_fun(dst_fun)) { - dst_fun->next = off_heap->first; - off_heap->first = (struct erl_off_heap_header*) dst_fun; - erts_refc_inc(&dst_fun->entry.fun->refc, 2); - } else { - ASSERT(is_external_fun(dst_fun) && dst_fun->next == NULL); - } - - goto cleanup_next; - } - case MAP_SUBTAG: + goto cleanup_next; + } + case MAP_SUBTAG: *resp = make_flatmap(hp); *hp++ = hdr; switch (MAP_HEADER_TYPE(hdr)) { @@ -1589,20 +1692,46 @@ Uint copy_shared_perform_x(Eterm obj, Uint size, erts_shcopy_t *info, goto cleanup_next; } case SUB_BITS_SUBTAG: { - const ErlSubBits *from_sb; - ErlSubBits *to_sb; + const ErlSubBits *from_sb = (ErlSubBits*)ptr; + const Eterm *underlying = boxed_val(from_sb->orig); + Eterm orig_hdr = *underlying; + + switch (primary_tag(orig_hdr)) { + case BOXED_SHARED_UNPROCESSED: + case BOXED_SHARED_PROCESSED: + orig_hdr = SHTABLE_X(t, orig_hdr >> _TAG_PRIMARY_SIZE); + break; + } - from_sb = (ErlSubBits*)ptr; - to_sb = (ErlSubBits*)hp; + if (shared_thing_subtag(orig_hdr) == BIN_REF_SUBTAG) { + ErlSubBits *to_sb; - EQUEUE_PUT_UNCHECKED(s, from_sb->orig); + to_sb = (ErlSubBits*)hp; - *to_sb = *from_sb; - to_sb->thing_word = hdr; - to_sb->orig = HEAP_ELEM_TO_BE_FILLED; + EQUEUE_PUT_UNCHECKED(s, from_sb->orig); - *resp = make_bitstring(to_sb); - hp += ERL_SUB_BITS_SIZE; + *to_sb = *from_sb; + to_sb->thing_word = hdr; + to_sb->orig = HEAP_ELEM_TO_BE_FILLED; + + *resp = make_bitstring(to_sb); + hp += ERL_SUB_BITS_SIZE; + } else { + /* This is a match context referring to an on-heap + * bitstring. + * + * As this is fairly rare and sharing multiple instances of + * the same match context is rarer still, we'll make a + * blunt copy that ignores sharing. */ + Uint size = from_sb->end - from_sb->start; + ASSERT(erl_sub_bits_is_match_context(from_sb)); + + hbot -= heap_bits_size(size); + *resp = HEAP_BITSTRING(hbot, + erl_sub_bits_get_base(from_sb), + from_sb->start, + size); + } goto cleanup_next; } @@ -1676,7 +1805,7 @@ Uint copy_shared_perform_x(Eterm obj, Uint size, erts_shcopy_t *info, const ErlFunThing* funp = (ErlFunThing *) hscan; ASSERT(ERL_FUN_SIZE == (1 + thing_arityval(*hscan))); hscan += ERL_FUN_SIZE; - remaining = fun_num_free(funp); + remaining = fun_env_size(funp); break; } case MAP_SUBTAG: @@ -1700,11 +1829,17 @@ Uint copy_shared_perform_x(Eterm obj, Uint size, erts_shcopy_t *info, MAP_HEADER_TYPE(*hscan)); } break; - case SUB_BITS_SUBTAG: - /* Make sure that the `orig` field is scanned. */ - hscan += ERL_SUB_BITS_SIZE - 1; - remaining = 1; + case SUB_BITS_SUBTAG: { + const ErlSubBits *sb = (ErlSubBits*)hscan; + if (sb->orig == HEAP_ELEM_TO_BE_FILLED) { + hscan += offsetof(ErlSubBits, orig) / sizeof(Eterm); + remaining = 1; + } else { + ASSERT(erl_sub_bits_is_match_context(sb)); + hscan += ERL_SUB_BITS_SIZE; + } break; + } default: hscan += 1 + thing_arityval(*hscan); break; @@ -1847,25 +1982,34 @@ Eterm* copy_shallow_x(Eterm *ERTS_RESTRICT ptr, Uint sz, Eterm **hpp, switch (val & _HEADER_SUBTAG_MASK) { case ARITYVAL_SUBTAG: break; - case BIN_REF_SUBTAG: + case SUB_BITS_SUBTAG: + { + const ErlSubBits *sb = (ErlSubBits*)(&tp[-1]); + const Uint orig_offset = ERL_SUB_BITS_SIZE - 2; + + ASSERT(erl_sub_bits_is_normal(sb)); + + sys_memcpy(hp, tp, orig_offset * sizeof(Eterm)); + hp[orig_offset] = byte_offset_ptr(sb->orig, offs); + + hp += orig_offset + 1; + tp += orig_offset + 1; + + sz -= ERL_SUB_BITS_SIZE - 1; + } + break; + case BIN_REF_SUBTAG: { BinRef *br = (BinRef*)(&tp[-1]); erts_refc_inc(&(br->val)->intern.refc, 2); ERTS_BR_OVERHEAD(off_heap, br); + goto off_heap_common; } - goto off_heap_common; - - case FUN_SUBTAG: + case FUN_REF_SUBTAG: { - ErlFunThing* funp = (ErlFunThing *) (tp-1); - - if (is_local_fun(funp)) { - erts_refc_inc(&funp->entry.fun->refc, 2); - goto off_heap_common; - } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); - goto default_copy; - } + FunRef *refp = (FunRef *) (tp-1); + erts_refc_inc(&(refp->entry)->refc, 2); + goto off_heap_common; } case EXTERNAL_PID_SUBTAG: case EXTERNAL_PORT_SUBTAG: @@ -1900,7 +2044,6 @@ Eterm* copy_shallow_x(Eterm *ERTS_RESTRICT ptr, Uint sz, Eterm **hpp, } /* Fall through... */ } - default_copy: default: { int tari = header_arity(val); @@ -1972,11 +2115,16 @@ void erts_move_multi_frags(Eterm** hpp, ErlOffHeap* off_heap, ErlHeapFragment* f } break; case TAG_PRIMARY_HEADER: - if (header_is_thing(gval)) { - hp += thing_arityval(gval); - } - break; - } + if (gval == HEADER_SUB_BITS) { + /* Tag the `orig` field as a literal. It's the last field + * inside the thing structure so we can handle it by pretending + * it's not part of the thing. */ + hp += thing_arityval(gval) - 1; + } else if (header_is_thing(gval)) { + hp += thing_arityval(gval); + } + break; + } } for (i=0; inext = off_heap->first; off_heap->first = hdr; break; - case FUN_SUBTAG: - { - const ErlFunThing *funp = (ErlFunThing *) hdr; - - if (is_local_fun(funp)) { - hdr->next = off_heap->first; - off_heap->first = hdr; - } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); - } - } - break; } } else { /* must be a cons cell */ ASSERT(ptr+1 < end); diff --git a/erts/emulator/beam/dist.c b/erts/emulator/beam/dist.c index 1d3fc46cc5b3..59281208a60e 100644 --- a/erts/emulator/beam/dist.c +++ b/erts/emulator/beam/dist.c @@ -2587,11 +2587,12 @@ int erts_net_message(Port *prt, if (!is_external_pid(watcher)) goto invalid_message; if (erts_this_dist_entry == external_pid_dist_entry(watcher)) - break; + goto monitored_process_not_alive; goto invalid_message; } if (!erts_proc_lookup(watcher)) { + monitored_process_not_alive: if (ede_hfrag != NULL) { erts_free_dist_ext_copy(erts_get_dist_ext(ede_hfrag)); free_message_buffer(ede_hfrag); diff --git a/erts/emulator/beam/emu/bs_instrs.tab b/erts/emulator/beam/emu/bs_instrs.tab index bd3e8f58d1bb..0374fb6069b4 100644 --- a/erts/emulator/beam/emu/bs_instrs.tab +++ b/erts/emulator/beam/emu/bs_instrs.tab @@ -148,14 +148,15 @@ i_bs_get_binary_all2.fetch(Ctx) { } i_bs_get_binary_all2.execute(Fail, Live, Unit, Dst) { - ErlBinMatchBuffer *_mb; + ErlSubBits *sb; Eterm _result; $GC_TEST_PRESERVE(BUILD_SUB_BITSTRING_HEAP_NEED, $Live, context); - _mb = ms_matchbuffer(context); - if (((_mb->size - _mb->offset) % $Unit) == 0) { + sb = (ErlSubBits*)bitstring_val(context); + + if (((sb->end - sb->start) % $Unit) == 0) { LIGHT_SWAPOUT; - _result = erts_bs_get_binary_all_2(c_p, _mb); + _result = erts_bs_get_binary_all_2(c_p, sb); LIGHT_SWAPIN; HEAP_SPACE_VERIFIED(0); ASSERT(is_value(_result)); @@ -177,14 +178,14 @@ i_bs_get_binary2.fetch(Ctx) { } i_bs_get_binary2.execute(Fail, Live, Sz, Flags, Dst) { - ErlBinMatchBuffer *_mb; + ErlSubBits *sb; Eterm _result; Uint _size; $BS_GET_FIELD_SIZE($Sz, (($Flags) >> 3), $FAIL($Fail), _size); $GC_TEST_PRESERVE(BUILD_SUB_BITSTRING_HEAP_NEED, $Live, context); - _mb = ms_matchbuffer(context); + sb = (ErlSubBits*)bitstring_val(context); LIGHT_SWAPOUT; - _result = erts_bs_get_binary_2(c_p, _size, $Flags, _mb); + _result = erts_bs_get_binary_2(c_p, _size, $Flags, sb); LIGHT_SWAPIN; HEAP_SPACE_VERIFIED(0); if (is_non_value(_result)) { @@ -206,12 +207,12 @@ i_bs_get_binary_imm2.fetch(Ctx) { } i_bs_get_binary_imm2.execute(Fail, Live, Sz, Flags, Dst) { - ErlBinMatchBuffer *_mb; + ErlSubBits *sb; Eterm _result; $GC_TEST_PRESERVE(BUILD_SUB_BITSTRING_HEAP_NEED, $Live, context); - _mb = ms_matchbuffer(context); + sb = (ErlSubBits*)bitstring_val(context); LIGHT_SWAPOUT; - _result = erts_bs_get_binary_2(c_p, $Sz, $Flags, _mb); + _result = erts_bs_get_binary_2(c_p, $Sz, $Flags, sb); LIGHT_SWAPIN; HEAP_SPACE_VERIFIED(0); if (is_non_value(_result)) { @@ -232,7 +233,7 @@ i_bs_get_float2.fetch(Ctx) { } i_bs_get_float2.execute(Fail, Live, Sz, Flags, Dst) { - ErlBinMatchBuffer *_mb; + ErlSubBits *sb; Eterm _result; Sint _size; @@ -241,9 +242,9 @@ i_bs_get_float2.execute(Fail, Live, Sz, Flags, Dst) { } _size *= (($Flags) >> 3); $GC_TEST_PRESERVE(FLOAT_SIZE_OBJECT, $Live, context); - _mb = ms_matchbuffer(context); + sb = (ErlSubBits*)bitstring_val(context); LIGHT_SWAPOUT; - _result = erts_bs_get_float_2(c_p, _size, ($Flags), _mb); + _result = erts_bs_get_float_2(c_p, _size, ($Flags), sb); LIGHT_SWAPIN; HEAP_SPACE_VERIFIED(0); if (is_non_value(_result)) { @@ -266,27 +267,25 @@ i_bs_skip_bits2.fetch(Ctx, Bits) { } i_bs_skip_bits2.execute(Fail, Unit) { - ErlBinMatchBuffer *_mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); size_t new_offset; Uint _size; - _mb = ms_matchbuffer(context); $BS_GET_FIELD_SIZE(bits, $Unit, $FAIL($Fail), _size); - new_offset = _mb->offset + _size; - if (new_offset <= _mb->size) { - _mb->offset = new_offset; + new_offset = sb->start + _size; + if (new_offset <= sb->end) { + sb->start = new_offset; } else { $FAIL($Fail); } } i_bs_skip_bits_imm2(Fail, Ms, Bits) { - ErlBinMatchBuffer *_mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ms); size_t new_offset; - _mb = ms_matchbuffer($Ms); - new_offset = _mb->offset + ($Bits); - if (new_offset <= _mb->size) { - _mb->offset = new_offset; + new_offset = sb->start + ($Bits); + if (new_offset <= sb->end) { + sb->start = new_offset; } else { $FAIL($Fail); } @@ -446,7 +445,7 @@ bs_init.execute(Live, Dst) { HTOP += bin_need; hb->thing_word = header_heap_bits(num_bits); - ERTS_SET_HB_SIZE(hb, num_bits); + hb->size = num_bits; erts_current_bin = (byte *) hb->data; new_binary = make_bitstring(hb); @@ -554,7 +553,7 @@ bs_init_bits.execute(Live, Dst) { HTOP += heap_bits_size(num_bits); hb->thing_word = header_heap_bits(num_bits); - ERTS_SET_HB_SIZE(hb, num_bits); + hb->size = num_bits; erts_current_bin = (byte*)hb->data; new_binary = make_bitstring(hb); @@ -808,12 +807,10 @@ i_bs_validate_unicode_retract(Fail, Src, Ms) { Eterm i = $Src; if (is_not_small(i) || i > make_small(0x10FFFFUL) || (make_small(0xD800UL) <= i && i <= make_small(0xDFFFUL))) { - Eterm ms = $Ms; /* Match context */ - ErlBinMatchBuffer* mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ms); /* Invalid value. Retract the position in the binary. */ - mb = ms_matchbuffer(ms); - mb->offset -= 32; + sb->start -= 32; $BADARG($Fail); } } @@ -1096,7 +1093,7 @@ i_bs_create_bin(Fail, Alloc, Live, Dst, N) { hb = (ErlHeapBits *) HTOP; HTOP += heap_bits_size(num_bits); hb->thing_word = header_heap_bits(num_bits); - ERTS_SET_HB_SIZE(hb, num_bits); + hb->size = num_bits; erts_current_bin = (byte *) hb->data; new_binary = make_bitstring(hb); } else { @@ -1240,33 +1237,29 @@ i_bs_create_bin(Fail, Alloc, Live, Dst, N) { // bs_test_zero_tail2(Fail, Ctx) { - ErlBinMatchBuffer *_mb; - _mb = (ErlBinMatchBuffer*) ms_matchbuffer($Ctx); - if (_mb->size != _mb->offset) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); + if (sb->end != sb->start) { $FAIL($Fail); } } bs_test_tail_imm2(Fail, Ctx, Offset) { - ErlBinMatchBuffer *_mb; - _mb = ms_matchbuffer($Ctx); - if (_mb->size - _mb->offset != $Offset) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); + if (sb->end - sb->start != $Offset) { $FAIL($Fail); } } bs_test_unit(Fail, Ctx, Unit) { - ErlBinMatchBuffer *_mb; - _mb = ms_matchbuffer($Ctx); - if ((_mb->size - _mb->offset) % $Unit) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); + if ((sb->end - sb->start) % $Unit) { $FAIL($Fail); } } bs_test_unit8(Fail, Ctx) { - ErlBinMatchBuffer *_mb; - _mb = ms_matchbuffer($Ctx); - if ((_mb->size - _mb->offset) & 7) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); + if ((sb->end - sb->start) & 7) { $FAIL($Fail); } } @@ -1282,17 +1275,18 @@ i_bs_get_integer_8.fetch(Ctx) { } i_bs_get_integer_8.execute(Fail, Dst) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Eterm _result; - ErlBinMatchBuffer* _mb = ms_matchbuffer(context); - if (_mb->size - _mb->offset < 8) { + if (sb->end - sb->start < 8) { $FAIL($Fail); } - if (BIT_OFFSET(_mb->offset) != 0) { - _result = erts_bs_get_integer_2(c_p, 8, 0, _mb); + if (BIT_OFFSET(sb->start) != 0) { + _result = erts_bs_get_integer_2(c_p, 8, 0, sb); } else { - _result = make_small(_mb->base[BYTE_OFFSET(_mb->offset)]); - _mb->offset += 8; + _result = make_small(*(erl_sub_bits_get_base(sb) + + BYTE_OFFSET(sb->start))); + sb->start += 8; } $Dst = _result; } @@ -1308,17 +1302,18 @@ i_bs_get_integer_16.fetch(Ctx) { } i_bs_get_integer_16.execute(Fail, Dst) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Eterm _result; - ErlBinMatchBuffer* _mb = ms_matchbuffer(context); - if (_mb->size - _mb->offset < 16) { + if (sb->end - sb->start < 16) { $FAIL($Fail); } - if (BIT_OFFSET(_mb->offset) != 0) { - _result = erts_bs_get_integer_2(c_p, 16, 0, _mb); + if (BIT_OFFSET(sb->start) != 0) { + _result = erts_bs_get_integer_2(c_p, 16, 0, sb); } else { - _result = make_small(get_int16(_mb->base+BYTE_OFFSET(_mb->offset))); - _mb->offset += 16; + _result = make_small(get_int16(erl_sub_bits_get_base(sb) + + BYTE_OFFSET(sb->start))); + sb->start += 16; } $Dst = _result; } @@ -1335,18 +1330,19 @@ i_bs_get_integer_32.fetch(Ctx) { } i_bs_get_integer_32.execute(Fail, Dst) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Uint32 _integer; - ErlBinMatchBuffer* _mb = ms_matchbuffer(context); - if (_mb->size - _mb->offset < 32) { + if (sb->end - sb->start < 32) { $FAIL($Fail); } - if (BIT_OFFSET(_mb->offset) != 0) { - _integer = erts_bs_get_unaligned_uint32(_mb); + if (BIT_OFFSET(sb->start) != 0) { + _integer = erts_bs_get_unaligned_uint32(sb); } else { - _integer = get_int32(_mb->base + _mb->offset/8); + _integer = get_int32(erl_sub_bits_get_base(sb) + + BYTE_OFFSET(sb->start)); } - _mb->offset += 32; + sb->start += 32; $Dst = make_small(_integer); } %endif @@ -1359,8 +1355,8 @@ bs_get_integer.head() { } bs_get_integer.fetch(Ctx, Size, Live) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); Uint wordsneeded; - ErlBinMatchBuffer* mb; Ms = $Ctx; Sz = $Size; wordsneeded = 1+WSIZE(NBYTES(Sz)); @@ -1369,8 +1365,7 @@ bs_get_integer.fetch(Ctx, Size, Live) { * and then realize we don't need the allocated space (if the * op fails). */ - mb = ms_matchbuffer(Ms); - if (mb->size - mb->offset >= Sz) { + if (sb->end - sb->start >= Sz) { $GC_TEST_PRESERVE(wordsneeded, $Live, Ms); } } @@ -1381,12 +1376,11 @@ bs_get_integer.fetch_small(Ctx, Size) { } bs_get_integer.execute(Fail, Flags, Dst) { - ErlBinMatchBuffer* mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(Ms); Eterm result; - mb = ms_matchbuffer(Ms); LIGHT_SWAPOUT; - result = erts_bs_get_integer_2(c_p, Sz, $Flags, mb); + result = erts_bs_get_integer_2(c_p, Sz, $Flags, sb); LIGHT_SWAPIN; HEAP_SPACE_VERIFIED(0); if (is_non_value(result)) { @@ -1406,9 +1400,9 @@ i_bs_get_integer.fetch(Ctx) { } i_bs_get_integer.execute(Fail, Live, FlagsAndUnit, Sz, Dst) { + ErlSubBits *sb; Uint flags; Uint size; - ErlBinMatchBuffer* mb; Eterm result; flags = $FlagsAndUnit; @@ -1419,21 +1413,24 @@ i_bs_get_integer.execute(Fail, Live, FlagsAndUnit, Sz, Dst) { * We do not want a gc and then realize we don't need * the allocated space (i.e. if the op fails). * - * Remember to re-acquire the matchbuffer after gc. + * Remember to re-acquire the match context after gc. */ - mb = ms_matchbuffer(context); - if (mb->size - mb->offset < size) { + sb = (ErlSubBits*)bitstring_val(context); + if (sb->end - sb->start < size) { $FAIL($Fail); } wordsneeded = 1+WSIZE(NBYTES((Uint) size)); $GC_TEST_PRESERVE(wordsneeded, $Live, context); $REFRESH_GEN_DEST(); } - mb = ms_matchbuffer(context); + + sb = (ErlSubBits*)bitstring_val(context); + LIGHT_SWAPOUT; - result = erts_bs_get_integer_2(c_p, size, flags, mb); + result = erts_bs_get_integer_2(c_p, size, flags, sb); LIGHT_SWAPIN; + HEAP_SPACE_VERIFIED(0); if (is_non_value(result)) { $FAIL($Fail); @@ -1452,21 +1449,21 @@ i_bs_get_utf8.fetch(Ctx) { } i_bs_get_utf8.execute(Fail, Dst) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Eterm result; - ErlBinMatchBuffer* mb = ms_matchbuffer(context); - if (mb->size - mb->offset < 8) { + if (sb->end - sb->start < 8) { $FAIL($Fail); } - if (BIT_OFFSET(mb->offset) != 0) { - result = erts_bs_get_utf8(mb); + if (BIT_OFFSET(sb->start) != 0) { + result = erts_bs_get_utf8(sb); } else { - byte b = mb->base[BYTE_OFFSET(mb->offset)]; + byte b = *(erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start)); if (b < 128) { result = make_small(b); - mb->offset += 8; + sb->start += 8; } else { - result = erts_bs_get_utf8(mb); + result = erts_bs_get_utf8(sb); } } if (is_non_value(result)) { @@ -1487,8 +1484,8 @@ i_bs_get_utf16.fetch(Ctx) { } i_bs_get_utf16.execute(Fail, Flags, Dst) { - ErlBinMatchBuffer* mb = ms_matchbuffer(context); - Eterm result = erts_bs_get_utf16(mb, $Flags); + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); + Eterm result = erts_bs_get_utf16(sb, $Flags); if (is_non_value(result)) { $FAIL($Fail); @@ -1498,24 +1495,28 @@ i_bs_get_utf16.execute(Fail, Flags, Dst) { } i_bs_match_string(Ctx, Fail, Bits, Ptr) { + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); byte* bytes = (byte *) $Ptr; - Uint bits = $Bits; - ErlBinMatchBuffer* mb; + Uint size = $Bits; Uint offs; - mb = ms_matchbuffer($Ctx); - if (mb->size - mb->offset < bits) { + if (sb->end - sb->start < size) { $FAIL($Fail); } - offs = mb->offset & 7; - if (offs == 0 && (bits & 7) == 0) { - if (sys_memcmp(bytes, mb->base+(mb->offset>>3), bits>>3)) { + offs = BIT_OFFSET(sb->start); + if (offs == 0 && TAIL_BITS(size) == 0) { + if (sys_memcmp(bytes, + erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start), + BYTE_SIZE(size))) { $FAIL($Fail); } - } else if (erts_cmp_bits(bytes, 0, mb->base+(mb->offset>>3), mb->offset & 7, bits)) { + } else if (erts_cmp_bits(bytes, 0, + erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start), + BIT_OFFSET(sb->start), + size)) { $FAIL($Fail); } - mb->offset += bits; + sb->start += size; } bs_get_tail := bs_get_tail.fetch.execute; @@ -1529,23 +1530,21 @@ bs_get_tail.fetch(Src) { } bs_get_tail.execute(Dst, Live) { - ErlBinMatchBuffer* mb; Eterm bin, *htop; - - ASSERT(header_is_bin_matchstate(*boxed_val(context))); + ErlSubBits *sb; $GC_TEST_PRESERVE(BUILD_SUB_BITSTRING_HEAP_NEED, $Live, context); htop = HTOP; - mb = ms_matchbuffer(context); + sb = (ErlSubBits*)bitstring_val(context); bin = erts_build_sub_bitstring(&htop, - mb->orig & TAG_PTR_MASK__, - (BinRef*)boxed_val(mb->orig), - mb->base, - mb->offset, - mb->size - mb->offset); + sb->orig & TAG_PTR_MASK__, + (BinRef*)boxed_val(sb->orig), + erl_sub_bits_get_base(sb), + sb->start, + sb->end - sb->start); HTOP = htop; $REFRESH_GEN_DEST(); @@ -1566,8 +1565,8 @@ i_bs_start_match3_gp.fetch(Src) { } i_bs_start_match3_gp.execute(Live, Fail, Dst, Pos) { - Eterm header; Uint position, live; + Eterm header; live = $Live; @@ -1577,30 +1576,32 @@ i_bs_start_match3_gp.execute(Live, Fail, Dst, Pos) { header = *boxed_val(context); - if (header_is_bin_matchstate(header)) { - ErlBinMatchBuffer *mb; + if (is_bitstring_header(header)) { + ErlSubBits *sb; + int reuse = 0; - ASSERT(HEADER_NUM_SLOTS(header) == 0); + if (header == HEADER_SUB_BITS) { + sb = (ErlSubBits*)bitstring_val(context); + reuse = erl_sub_bits_is_match_context(sb); + } - mb = ms_matchbuffer(context); - position = mb->offset; + if (!reuse) { + $GC_TEST_PRESERVE(ERL_SUB_BITS_SIZE, live, context); - $Dst = context; - } else if (is_bitstring_header(header)) { - ErlBinMatchState *ms; - - $GC_TEST_PRESERVE(ERL_BIN_MATCHSTATE_SIZE(0), live, context); - HEAP_TOP(c_p) = HTOP; + HEAP_TOP(c_p) = HTOP; #ifdef DEBUG - c_p->stop = E; /* Needed for checking in HeapOnlyAlloc(). */ + c_p->stop = E; /* Needed for checking in HeapOnlyAlloc(). */ #endif - ms = erts_bs_start_match_3(c_p, context); - HTOP = HEAP_TOP(c_p); - HEAP_SPACE_VERIFIED(0); + sb = erts_bs_start_match_3(c_p, context); + HTOP = HEAP_TOP(c_p); + HEAP_SPACE_VERIFIED(0); - $REFRESH_GEN_DEST(); - $Dst = make_matchstate(ms); - position = ms->mb.offset; + $REFRESH_GEN_DEST(); + context = make_bitstring(sb); + } + + position = sb->start; + $Dst = context; } else { $FAIL($Fail); } @@ -1609,85 +1610,19 @@ i_bs_start_match3_gp.execute(Live, Fail, Dst, Pos) { $Pos = make_small(position); } -i_bs_start_match3 := i_bs_start_match3.fetch.execute; - -i_bs_start_match3.head() { - Eterm context; -} - -i_bs_start_match3.fetch(Src) { - context = $Src; -} - -i_bs_start_match3.execute(Live, Fail, Dst) { - Eterm header; - Uint live; - - live = $Live; - - if (!is_boxed(context)) { - $FAIL($Fail); - } - - header = *boxed_val(context); - - if (header_is_bin_matchstate(header)) { - ASSERT(HEADER_NUM_SLOTS(header) == 0); - $Dst = context; - } else if (is_bitstring_header(header)) { - ErlBinMatchState *ms; - - $GC_TEST_PRESERVE(ERL_BIN_MATCHSTATE_SIZE(0), live, context); - HEAP_TOP(c_p) = HTOP; -#ifdef DEBUG - c_p->stop = E; /* Needed for checking in HeapOnlyAlloc(). */ -#endif - ms = erts_bs_start_match_3(c_p, context); - HTOP = HEAP_TOP(c_p); - HEAP_SPACE_VERIFIED(0); - - $REFRESH_GEN_DEST(); - $Dst = make_matchstate(ms); - } else { - $FAIL($Fail); - } -} - bs_set_position(Ctx, Pos) { - ErlBinMatchBuffer* mb; - Eterm context; - - context = $Ctx; - ASSERT(header_is_bin_matchstate(*boxed_val(context))); - - mb = ms_matchbuffer(context); - mb->offset = unsigned_val($Pos); + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); + sb->start = unsigned_val($Pos); } i_bs_get_position(Ctx, Dst) { - ErlBinMatchBuffer* mb; - Eterm context; - - context = $Ctx; - ASSERT(header_is_bin_matchstate(*boxed_val(context))); + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); - mb = ms_matchbuffer(context); - $Dst = make_small(mb->offset); + $Dst = make_small(sb->start); } %else -# -# Unlike their 64-bit counterparts, the 32-bit position instructions operate on -# an offset from the "base position" of the context because storing raw -# positions would lead to the creation of far too many bigints. -# -# When a match context is reused we check whether its position fits into an -# immediate, and create a new match context if it does not. This means we only -# have to allocate stuff roughly once every 16MB rather than every time we -# match at a position beyond 16MB. -# - bs_set_position := bs_set_position.fetch.execute; bs_set_position.head() { @@ -1700,16 +1635,13 @@ bs_set_position.fetch(Ctx, Pos) { } bs_set_position.execute() { - ErlBinMatchState *ms; - - ASSERT(header_is_bin_matchstate(*boxed_val(context))); - ms = (ErlBinMatchState*)boxed_val(context); + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); if (ERTS_LIKELY(is_small(position))) { - ms->mb.offset = ms->save_offset[0] + unsigned_val(position); + sb->start = unsigned_val(position); } else { ASSERT(is_big(position)); - ms->mb.offset = ms->save_offset[0] + *BIG_V(big_val(position)); + sb->start = *BIG_V(big_val(position)); } } @@ -1724,13 +1656,8 @@ bs_get_position.fetch(Ctx) { } bs_get_position.execute(Dst, Live) { - ErlBinMatchState *ms; - Uint position; - - ASSERT(header_is_bin_matchstate(*boxed_val(context))); - ms = (ErlBinMatchState*)boxed_val(context); - - position = ms->mb.offset - ms->save_offset[0]; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); + Uint position = sb->start; if (ERTS_LIKELY(IS_USMALL(0, position))) { $Dst = make_small(position); @@ -1750,6 +1677,8 @@ bs_get_position.execute(Dst, Live) { } } +%endif + i_bs_start_match3 := i_bs_start_match3.fetch.execute; i_bs_start_match3.head() { @@ -1772,60 +1701,36 @@ i_bs_start_match3.execute(Live, Fail, Dst) { header = *boxed_val(context); - if (header_is_bin_matchstate(header)) { - ErlBinMatchState *current_ms; - Uint position; - - ASSERT(HEADER_NUM_SLOTS(header) == 1); - - current_ms = (ErlBinMatchState*)boxed_val(context); - position = current_ms->mb.offset - current_ms->save_offset[0]; - - if (ERTS_LIKELY(IS_USMALL(0, position))) { - $Dst = context; - } else { - ErlBinMatchState *new_ms; - - $GC_TEST_PRESERVE(ERL_BIN_MATCHSTATE_SIZE(1), live, context); - current_ms = (ErlBinMatchState*)boxed_val(context); + if (is_bitstring_header(header)) { + ErlSubBits *sb; + int reuse = 0; - new_ms = (ErlBinMatchState*)HTOP; - HTOP += ERL_BIN_MATCHSTATE_SIZE(1); - - new_ms->thing_word = HEADER_BIN_MATCHSTATE(1); - new_ms->save_offset[0] = current_ms->mb.offset; - new_ms->mb = current_ms->mb; - - $REFRESH_GEN_DEST(); - $Dst = make_matchstate(new_ms); + if (header == HEADER_SUB_BITS) { + sb = (ErlSubBits*)bitstring_val(context); + reuse = erl_sub_bits_is_match_context(sb); } - } else if (is_bitstring_header(header)) { - Eterm result; - $GC_TEST_PRESERVE(ERL_BIN_MATCHSTATE_SIZE(1), live, context); - HEAP_TOP(c_p) = HTOP; + if (!reuse) { + $GC_TEST_PRESERVE(ERL_SUB_BITS_SIZE, live, context); + HEAP_TOP(c_p) = HTOP; #ifdef DEBUG - c_p->stop = E; /* Needed for checking in HeapOnlyAlloc(). */ + c_p->stop = E; /* Needed for checking in HeapOnlyAlloc(). */ #endif + sb = erts_bs_start_match_3(c_p, context); + HTOP = HEAP_TOP(c_p); + HEAP_SPACE_VERIFIED(0); - /* We intentionally use erts_bs_start_match_2 so that we can use - * save_offset as a base for all saved positions on this context, - * allowing us to avoid bigints for much longer. */ - result = erts_bs_start_match_2(c_p, context, 1); - - HTOP = HEAP_TOP(c_p); - HEAP_SPACE_VERIFIED(0); + $REFRESH_GEN_DEST(); + context = make_bitstring(sb); + } - $REFRESH_GEN_DEST(); - $Dst = result; + $Dst = context; } else { $FAIL($Fail); } } -%endif - // // New instructions introduced in OTP 26 for matching of integers and // binaries of fixed sizes follow. @@ -1846,9 +1751,9 @@ i_bs_ensure_bits.fetch(Src) { } i_bs_ensure_bits.execute(NumBits, Fail) { - ErlBinMatchBuffer* mb = ms_matchbuffer(context); + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Uint size = $NumBits; - if (mb->size - mb->offset < size) { + if (sb->end - sb->start < size) { $FAIL($Fail); } } @@ -1868,11 +1773,11 @@ i_bs_ensure_bits_unit.fetch(Src) { } i_bs_ensure_bits_unit.execute(NumBits, Unit, Fail) { - ErlBinMatchBuffer* mb = ms_matchbuffer(context); + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Uint size = $NumBits; Uint diff; - if ((diff = mb->size - mb->offset) < size) { + if ((diff = sb->end - sb->start) < size) { $FAIL($Fail); } if ((diff - size) % $Unit != 0) { @@ -1889,24 +1794,24 @@ i_bs_read_bits := i_bs_read_bits.fetch.execute; i_bs_ensure_bits_read := i_bs_read_bits.fetch.ensure_bits.execute; i_bs_read_bits.head() { - ErlBinMatchBuffer* mb; + ErlSubBits *sb; Uint size; } i_bs_read_bits.fetch(Src, NumBits) { - mb = ms_matchbuffer($Src); + sb = (ErlSubBits*)bitstring_val($Src); size = $NumBits; } i_bs_read_bits.ensure_bits(Fail) { - if (mb->size - mb->offset < size) { + if (sb->end - sb->start < size) { $FAIL($Fail); } } i_bs_read_bits.execute() { - byte *byte_ptr; - Uint bit_offset = mb->offset % 8; + const byte *byte_ptr; + Uint bit_offset = sb->start % 8; Uint num_bytes_to_read = (size + 7) / 8; Uint num_partial = size % 8; @@ -1916,8 +1821,8 @@ i_bs_read_bits.execute() { } bitdata = 0; - byte_ptr = mb->base + (mb->offset >> 3); - mb->offset += size; + byte_ptr = erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start); + sb->start += size; switch (num_bytes_to_read) { #ifdef ARCH_64 case 9: @@ -1975,13 +1880,13 @@ i_bs_extract_integer(NumBits, Dst) { // i_bs_read_integer_8 Ctx Dst i_bs_read_integer_8(Ctx, Dst) { - ErlBinMatchBuffer* mb = ms_matchbuffer($Ctx); + ErlSubBits *sb = (ErlSubBits*)bitstring_val($Ctx); byte *byte_ptr; - Uint bit_offset = mb->offset % 8; + Uint bit_offset = sb->start % 8; Eterm result; - byte_ptr = mb->base + (mb->offset >> 3); - mb->offset += 8; + byte_ptr = erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start); + sb->start += 8; result = byte_ptr[0]; if (bit_offset != 0) { result = result << 8 | byte_ptr[1]; @@ -2006,15 +1911,15 @@ i_bs_get_fixed_integer.fetch(Src) { } i_bs_get_fixed_integer.execute(Size, Flags, Dst) { - ErlBinMatchBuffer* mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Uint size = $Size; Eterm result; - mb = ms_matchbuffer(context); LIGHT_SWAPOUT; - result = erts_bs_get_integer_2(c_p, size, $Flags, mb); + result = erts_bs_get_integer_2(c_p, size, $Flags, sb); LIGHT_SWAPIN; HEAP_SPACE_VERIFIED(0); + $Dst = result; } @@ -2033,24 +1938,21 @@ i_bs_get_fixed_binary.fetch(Src) { } i_bs_get_fixed_binary.execute(Size, Dst) { - ErlBinMatchBuffer* mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Uint size = $Size; Eterm* htop; Eterm result; - ASSERT(header_is_bin_matchstate(*boxed_val(context))); - htop = HTOP; - mb = ms_matchbuffer(context); result = erts_build_sub_bitstring(&htop, - mb->orig & TAG_PTR_MASK__, - (BinRef*)boxed_val(mb->orig), - mb->base, - mb->offset, + sb->orig & TAG_PTR_MASK__, + (BinRef*)boxed_val(sb->orig), + erl_sub_bits_get_base(sb), + sb->start, size); HTOP = htop; - mb->offset += size; + sb->start += size; $Dst = result; } @@ -2070,20 +1972,17 @@ i_bs_get_tail.fetch(Src) { } i_bs_get_tail.execute(Dst) { - ErlBinMatchBuffer* mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Eterm* htop; Eterm result; - ASSERT(header_is_bin_matchstate(*boxed_val(context))); - htop = HTOP; - mb = ms_matchbuffer(context); result = erts_build_sub_bitstring(&htop, - mb->orig & TAG_PTR_MASK__, - (BinRef*)boxed_val(mb->orig), - mb->base, - mb->offset, - mb->size - mb->offset); + sb->orig & TAG_PTR_MASK__, + (BinRef*)boxed_val(sb->orig), + erl_sub_bits_get_base(sb), + sb->start, + sb->end - sb->start); HTOP = htop; $Dst = result; @@ -2104,12 +2003,10 @@ i_bs_skip.fetch(Src) { } i_bs_skip.execute(Size) { - ErlBinMatchBuffer* mb; + ErlSubBits *sb = (ErlSubBits*)bitstring_val(context); Uint size = $Size; - ASSERT(header_is_bin_matchstate(*boxed_val(context))); - mb = ms_matchbuffer(context); - mb->offset += size; + sb->start += size; } // i_bs_drop Size diff --git a/erts/emulator/beam/emu/emu_load.c b/erts/emulator/beam/emu/emu_load.c index 16ec0aa4c49c..7ca930eb5815 100644 --- a/erts/emulator/beam/emu/emu_load.c +++ b/erts/emulator/beam/emu/emu_load.c @@ -93,10 +93,10 @@ int beam_load_prepare_emit(LoaderState *stp) { init_label(&stp->labels[i]); } - stp->lambda_literals = erts_alloc(ERTS_ALC_T_PREPARED_CODE, - stp->beam.lambdas.count * sizeof(SWord)); + stp->fun_refs = erts_alloc(ERTS_ALC_T_PREPARED_CODE, + stp->beam.lambdas.count * sizeof(SWord)); for (i = 0; i < stp->beam.lambdas.count; i++) { - stp->lambda_literals[i] = ERTS_SWORD_MAX; + stp->fun_refs[i] = ERTS_SWORD_MAX; } stp->import_patches = @@ -190,9 +190,9 @@ int beam_load_prepared_dtor(Binary* magic) stp->labels = NULL; } - if (stp->lambda_literals != NULL) { - erts_free(ERTS_ALC_T_PREPARED_CODE, (void *)stp->lambda_literals); - stp->lambda_literals = NULL; + if (stp->fun_refs != NULL) { + erts_free(ERTS_ALC_T_PREPARED_CODE, (void *)stp->fun_refs); + stp->fun_refs = NULL; } if (stp->import_patches != NULL) { @@ -623,20 +623,19 @@ void beam_load_finalize_code(LoaderState* stp, struct erl_module_instance* inst_ */ if (stp->beam.lambdas.count) { BeamFile_LambdaTable *lambda_table; - ErtsLiteralArea *literal_area; ErlFunEntry **fun_entries; - LambdaPatch* lp; - int i; + LambdaPatch *lp; lambda_table = &stp->beam.lambdas; + fun_entries = erts_alloc(ERTS_ALC_T_LOADER_TMP, sizeof(ErlFunEntry*) * lambda_table->count); - literal_area = (stp->code_hdr)->literal_area; - - for (i = 0; i < lambda_table->count; i++) { + for (int i = 0; i < lambda_table->count; i++) { BeamFile_LambdaEntry *lambda; ErlFunEntry *fun_entry; + FunRef *fun_refp; + Eterm fun_ref; lambda = &lambda_table->entries[i]; @@ -648,33 +647,31 @@ void beam_load_finalize_code(LoaderState* stp, struct erl_module_instance* inst_ lambda->arity - lambda->num_free); fun_entries[i] = fun_entry; - if (erts_is_fun_loaded(fun_entry, staging_ix)) { - /* We've reloaded a module over itself and inherited the old - * instance's fun entries, so we need to undo the reference - * bump in `erts_put_fun_entry2` to make fun purging work. */ - erts_refc_dectest(&fun_entry->refc, 1); - } - - /* Finalize the literal we've created for this lambda, if any, - * converting it from an external fun to a local one with the newly - * created fun entry. */ - if (stp->lambda_literals[i] != ERTS_SWORD_MAX) { - ErlFunThing *funp; - Eterm literal; - - literal = beamfile_get_literal(&stp->beam, - stp->lambda_literals[i]); - funp = (ErlFunThing *)fun_val(literal); + fun_ref = beamfile_get_literal(&stp->beam, stp->fun_refs[i]); + /* If there are no free variables, the literal refers to an + * ErlFunThing that needs to be fixed up before we process the + * FunRef. */ + if (lambda->num_free == 0) { + ErlFunThing *funp = (ErlFunThing*)boxed_val(fun_ref); + ASSERT(funp->entry.fun == NULL); funp->entry.fun = fun_entry; + fun_ref = funp->env[0]; + } - funp->next = literal_area->off_heap; - literal_area->off_heap = (struct erl_off_heap_header *)funp; - - ASSERT(funp->thing_word & (1 << FUN_HEADER_EXTERNAL_OFFS)); - funp->thing_word &= ~(1 << FUN_HEADER_EXTERNAL_OFFS); - - erts_refc_inc(&fun_entry->refc, 2); + /* Patch up the fun reference literal. */ + fun_refp = (FunRef*)boxed_val(fun_ref); + fun_refp->entry = fun_entry; + + /* Bump the reference count: this could not be done when copying + * the literal as we had no idea which entry it belonged to. + * + * We also need to parry an annoying wrinkle: when reloading a + * module over itself, we inherit the old instance's fun entries, + * and thus have to cancel the reference bump in + * `erts_put_fun_entry2` to make fun purging work. */ + if (!erts_is_fun_loaded(fun_entry, staging_ix)) { + erts_refc_inctest(&fun_entry->refc, 1); } erts_set_fun_code(fun_entry, diff --git a/erts/emulator/beam/emu/instrs.tab b/erts/emulator/beam/emu/instrs.tab index bdf0e1e3a81d..5238fdd5dcec 100644 --- a/erts/emulator/beam/emu/instrs.tab +++ b/erts/emulator/beam/emu/instrs.tab @@ -541,16 +541,35 @@ i_init3(Y1, Y2, Y3) { } i_make_fun3(FunP, Dst, Arity, NumFree) { - ErlFunThing* funp; ErlFunEntry *fe = (ErlFunEntry *) $FunP; int i, num_free = $NumFree; + ErlFunThing* funp; //| -no_next - SWAPOUT; - funp = erts_new_local_fun_thing(c_p, fe, $Arity, num_free); - SWAPIN; + + /* We add one word to account for the `FunRef` keeping our fun entry + * alive, it's kept as part of the environment. */ + funp = (ErlFunThing*)HTOP; + HTOP += ERL_FUN_SIZE + num_free + 1; + + funp->thing_word = MAKE_FUN_HEADER($Arity, num_free, 0); + funp->entry.fun = fe; + +#ifdef DEBUG + { + /* Note that `mfa` may be NULL if the fun is currently being purged. We + * ignore this since it's not an error and we only need `mfa` to + * sanity-check the arity at this point. If the fun is called while in + * this state, the `error_handler` module will take care of it. */ + const ErtsCodeMFA *mfa = erts_get_fun_mfa(fe, erts_active_code_ix()); + ASSERT(!mfa || fun_arity(funp) == mfa->arity - num_free); + ASSERT($Arity == fe->arity); + } +#endif + I = $NEXT_INSTRUCTION; - for (i = 0; i < num_free; i++) { - Eterm term = *I++; + + for (i = 0; i < (num_free + 1); i++) { + Eterm term = *I++; switch (loader_tag(term)) { case LOADER_X_REG: term = x(loader_x_reg_index(term)); @@ -559,8 +578,9 @@ i_make_fun3(FunP, Dst, Arity, NumFree) { term = y(loader_y_reg_index(term)); break; } - funp->env[i] = term; + funp->env[i] = term; } + $Dst = make_fun(funp); Goto(*I); } @@ -729,7 +749,7 @@ put_tuple2(Dst, Arity) { /* * If operands are not packed (in the 32-bit VM), - * is is not safe to use $Dst directly after I + * it is not safe to use $Dst directly after I * has been updated. */ Eterm* dst_ptr = &($Dst); diff --git a/erts/emulator/beam/emu/load.h b/erts/emulator/beam/emu/load.h index 6de420a5dc31..79b04a3fb89e 100644 --- a/erts/emulator/beam/emu/load.h +++ b/erts/emulator/beam/emu/load.h @@ -142,10 +142,11 @@ struct LoaderState_ { unsigned int current_li; /* Current line instruction */ unsigned int* func_line; /* Mapping from function to first line instr */ - /* Translates lambda indexes to their literals, if any. Lambdas that lack - * a literal (for example if they have an environment) are represented by - * ERTS_SWORD_MAX. */ - SWord *lambda_literals; + /* Translates lambda indexes to the literal holding their FunRef. + * + * Lambdas that lack an environment are represented by an ErlFunThing that + * is immediately followed by an FunRef. */ + SWord *fun_refs; int otp_20_or_higher; diff --git a/erts/emulator/beam/erl_alloc.h b/erts/emulator/beam/erl_alloc.h index dc931e825450..c675fe3f8800 100644 --- a/erts/emulator/beam/erl_alloc.h +++ b/erts/emulator/beam/erl_alloc.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2002-2022. All Rights Reserved. + * Copyright Ericsson AB 2002-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/beam/erl_bif_chksum.c b/erts/emulator/beam/erl_bif_chksum.c index 66f7992c2ef3..dafe3451ee8e 100644 --- a/erts/emulator/beam/erl_bif_chksum.c +++ b/erts/emulator/beam/erl_bif_chksum.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2008-2021. All Rights Reserved. + * Copyright Ericsson AB 2008-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/beam/erl_bif_info.c b/erts/emulator/beam/erl_bif_info.c index 5ed0c624ef7e..754cc0588f3b 100644 --- a/erts/emulator/beam/erl_bif_info.c +++ b/erts/emulator/beam/erl_bif_info.c @@ -3815,7 +3815,6 @@ fun_info_2(BIF_ALIST_2) fe = funp->entry.fun; mfa = erts_get_fun_mfa(fe, erts_active_code_ix()); } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); mfa = &(funp->entry.exp)->info.mfa; fe = NULL; } @@ -3855,13 +3854,12 @@ fun_info_2(BIF_ALIST_2) break; case am_env: { - Uint num_free = fun_num_free(funp); - int i; + int num_free = is_local_fun(funp) ? fun_num_free(funp) : 0; hp = HAlloc(p, 3 + 2 * num_free); val = NIL; - for (i = num_free - 1; i >= 0; i--) { + for (int i = num_free - 1; i >= 0; i--) { val = CONS(hp, funp->env[i], val); hp += 2; } @@ -3919,7 +3917,6 @@ fun_info_mfa_1(BIF_ALIST_1) make_small(fun_arity(funp)))); } } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); mfa = &(funp->entry.exp)->info.mfa; } diff --git a/erts/emulator/beam/erl_bits.c b/erts/emulator/beam/erl_bits.c index 2727cc717e78..38c4de7d101f 100644 --- a/erts/emulator/beam/erl_bits.c +++ b/erts/emulator/beam/erl_bits.c @@ -98,81 +98,58 @@ static byte get_bit(byte b, size_t a_offs); } \ }while(0) \ -Eterm -erts_bs_start_match_2(Process *p, Eterm bin, Uint Max) -{ - ErlBinMatchState *ms; - Uint offset, size; - byte *base; - Eterm br_flags; - BinRef *br; - Uint* hp; - - ASSERT(is_bitstring(bin)); - - hp = HeapOnlyAlloc(p, ERL_BIN_MATCHSTATE_SIZE(Max)); - ms = (ErlBinMatchState *) hp; - - ERTS_PIN_BITSTRING(bin, br_flags, br, base, offset, size); - - ms->thing_word = HEADER_BIN_MATCHSTATE(Max); - (ms->mb).orig = br ? ((Eterm)br | br_flags) : bin; - (ms->mb).base = base; - (ms->mb).offset = ms->save_offset[0] = offset; - (ms->mb).size = offset + size; - - return make_matchstate(ms); -} - -ErlBinMatchState *erts_bs_start_match_3(Process *p, Eterm bin) +ErlSubBits *erts_bs_start_match_3(Process *p, Eterm bin) { - ErlBinMatchState *ms; + ErlSubBits *sb; Uint offset, size; byte *base; Eterm br_flags; BinRef *br; - Uint* hp; + Uint *hp; ASSERT(is_bitstring(bin)); - hp = HeapOnlyAlloc(p, ERL_BIN_MATCHSTATE_SIZE(0)); - ms = (ErlBinMatchState *) hp; + hp = HeapOnlyAlloc(p, ERL_SUB_BITS_SIZE); + sb = (ErlSubBits *) hp; ERTS_PIN_BITSTRING(bin, br_flags, br, base, offset, size); - ms->thing_word = HEADER_BIN_MATCHSTATE(0); - (ms->mb).orig = br ? ((Eterm)br | br_flags) : bin; - (ms->mb).base = base; - (ms->mb).offset = offset; - (ms->mb).size = offset + size; + erl_sub_bits_init(sb, + ERL_SUB_BITS_FLAGS_MATCH_CONTEXT, + br ? ((Eterm)br | br_flags) : bin, + base, + offset, + size); - return ms; + return sb; } #ifdef DEBUG # define CHECK_MATCH_BUFFER(MB) check_match_buffer(MB) -static void check_match_buffer(const ErlBinMatchBuffer *mb) +static void check_match_buffer(const ErlSubBits *sb) { - Eterm *unboxed = boxed_val(mb->orig); - const byte *base; + Eterm *unboxed = boxed_val(sb->orig); + const byte *match_base = erl_sub_bits_get_base(sb); + const byte *orig_base; Uint size; if (*unboxed == HEADER_BIN_REF) { const BinRef *br = (BinRef*)unboxed; - size = NBITS((br->val)->orig_size); - base = (byte*)br->bytes; ASSERT(!((br->val)->intern.flags & (BIN_FLAG_WRITABLE | BIN_FLAG_ACTIVE_WRITER))); + size = NBITS((br->val)->orig_size); + orig_base = (byte*)(sb->base_flags & ~ERL_SUB_BITS_FLAG_MASK); } else { const ErlHeapBits *hb = (ErlHeapBits*)unboxed; size = hb->size; - base = (byte*)hb->data; + orig_base = (byte*)hb->data; } - ASSERT(mb->size <= size); - ASSERT(mb->base >= base && mb->base <= (base + NBYTES(size))); - ASSERT(mb->offset <= (size - NBITS(mb->base - base))); + ASSERT(sb->end >= sb->start); + ASSERT(size >= (sb->end - sb->start)); + ASSERT(match_base >= orig_base && match_base <= (orig_base + NBYTES(size))); + ASSERT(sb->start <= (size - NBITS(match_base - orig_base))); } #else # define CHECK_MATCH_BUFFER(MB) @@ -180,7 +157,7 @@ static void check_match_buffer(const ErlBinMatchBuffer *mb) Eterm erts_bs_get_integer_2( -Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) +Process *p, Uint num_bits, unsigned flags, ErlSubBits *sb) { Uint bytes; Uint bits; @@ -199,8 +176,8 @@ Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) return SMALL_ZERO; } - CHECK_MATCH_BUFFER(mb); - if (mb->size - mb->offset < num_bits) { /* Asked for too many bits. */ + CHECK_MATCH_BUFFER(sb); + if (sb->end - sb->start < num_bits) { /* Asked for too many bits. */ return THE_NON_VALUE; } @@ -208,14 +185,14 @@ Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) * Special cases for field sizes up to the size of Uint. */ - if (num_bits <= 8-(offs = BIT_OFFSET(mb->offset))) { + if (num_bits <= 8-(offs = BIT_OFFSET(sb->start))) { /* * All bits are in one byte in the binary. We only need * shift them right and mask them. */ - Uint b = mb->base[BYTE_OFFSET(mb->offset)]; + Uint b = *(erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start)); Uint mask = MAKE_MASK(num_bits); - mb->offset += num_bits; + sb->start += num_bits; b >>= 8 - offs - num_bits; b &= mask; if ((flags & BSF_SIGNED) && b >> (num_bits-1)) { @@ -228,10 +205,11 @@ Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) * combine the bytes to a word first, and then shift right and * mask to extract the bits. */ - Uint byte_offset = BYTE_OFFSET(mb->offset); - Uint w = mb->base[byte_offset] << 8 | mb->base[byte_offset+1]; + Uint byte_offset = BYTE_OFFSET(sb->start); + const byte* bp = erl_sub_bits_get_base(sb) + byte_offset; + Uint w = bp[0] << 8 | bp[1]; Uint mask = MAKE_MASK(num_bits); - mb->offset += num_bits; + sb->start += num_bits; w >>= 16 - offs - num_bits; w &= mask; if ((flags & BSF_SIGNED) && w >> (num_bits-1)) { @@ -243,12 +221,12 @@ Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) * Handle field sizes from 9 up to SMALL_BITS-1 bits, big-endian, * stored in at least two bytes. */ - byte* bp = mb->base + BYTE_OFFSET(mb->offset); + const byte* bp = erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start); Uint n; Uint w; n = num_bits; - mb->offset += num_bits; + sb->start += num_bits; /* * Handle the most signicant byte if it contains 1 to 7 bits. @@ -326,13 +304,15 @@ Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) */ if (flags & BSF_LITTLE) { - erts_copy_bits(mb->base, mb->offset, 1, LSB, 0, 1, num_bits); + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, + LSB, 0, 1, num_bits); *MSB >>= offs; /* adjust msb */ } else { *MSB = 0; - erts_copy_bits(mb->base, mb->offset, 1, MSB, offs, -1, num_bits); + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, + MSB, offs, -1, num_bits); } - mb->offset += num_bits; + sb->start += num_bits; /* * Get the sign bit. @@ -436,12 +416,12 @@ Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) } Eterm -erts_bs_get_binary_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) +erts_bs_get_binary_2(Process *p, Uint num_bits, unsigned flags, ErlSubBits *sb) { Eterm result; - CHECK_MATCH_BUFFER(mb); - if (mb->size - mb->offset < num_bits) { + CHECK_MATCH_BUFFER(sb); + if (sb->end - sb->start < num_bits) { /* Asked for too many bits. */ return THE_NON_VALUE; } @@ -451,18 +431,18 @@ erts_bs_get_binary_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffe */ result = erts_build_sub_bitstring(&HEAP_TOP(p), - mb->orig & TAG_PTR_MASK__, - (BinRef*)boxed_val(mb->orig), - mb->base, - mb->offset, num_bits); + sb->orig & TAG_PTR_MASK__, + (BinRef*)boxed_val(sb->orig), + erl_sub_bits_get_base(sb), + sb->start, num_bits); - mb->offset += num_bits; + sb->start += num_bits; return result; } Eterm -erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb) +erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlSubBits *sb) { Eterm* hp; erlfp16 f16; @@ -471,14 +451,14 @@ erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer byte* fptr; FloatDef f; - CHECK_MATCH_BUFFER(mb); + CHECK_MATCH_BUFFER(sb); if (num_bits == 0) { f.fd = 0.0; hp = HeapOnlyAlloc(p, FLOAT_SIZE_OBJECT); PUT_DOUBLE(f, hp); return make_float(hp); } - if (mb->size - mb->offset < num_bits) { /* Asked for too many bits. */ + if (sb->end - sb->start < num_bits) { /* Asked for too many bits. */ return THE_NON_VALUE; } if (num_bits == 16) { @@ -492,11 +472,11 @@ erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer } if (BIT_IS_MACHINE_ENDIAN(flags)) { - erts_copy_bits(mb->base, mb->offset, 1, + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, fptr, 0, 1, num_bits); } else { - erts_copy_bits(mb->base, mb->offset, 1, + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, fptr + NBYTES(num_bits) - 1, 0, -1, num_bits); } @@ -519,28 +499,28 @@ erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer f.fd = f64; #endif } - mb->offset += num_bits; + sb->start += num_bits; hp = HeapOnlyAlloc(p, FLOAT_SIZE_OBJECT); PUT_DOUBLE(f, hp); return make_float(hp); } Eterm -erts_bs_get_binary_all_2(Process *p, ErlBinMatchBuffer* mb) +erts_bs_get_binary_all_2(Process *p, ErlSubBits *sb) { Uint bit_size; Eterm result; - CHECK_MATCH_BUFFER(mb); - bit_size = mb->size - mb->offset; + CHECK_MATCH_BUFFER(sb); + bit_size = sb->end - sb->start; result = erts_build_sub_bitstring(&HEAP_TOP(p), - mb->orig & TAG_PTR_MASK__, - (BinRef*)boxed_val(mb->orig), - mb->base, - mb->offset, bit_size); + sb->orig & TAG_PTR_MASK__, + (BinRef*)boxed_val(sb->orig), + erl_sub_bits_get_base(sb), + sb->start, bit_size); - mb->offset = mb->size; + sb->start = sb->end; return result; } @@ -1448,6 +1428,7 @@ static void build_writable_bitstring(Process *p, Eterm **hpp, Binary *bin, + Uint current_size, Uint apparent_size, BinRef **brp, ErlSubBits **sbp) @@ -1469,11 +1450,12 @@ build_writable_bitstring(Process *p, MSO(p).overhead += apparent_size / (sizeof(Eterm) * 8); - br->bytes = (byte*)bin->orig_bytes; - - sb->thing_word = HEADER_SUB_BITS; - sb->is_writable = 1; - sb->orig = make_bitstring(br); + erl_sub_bits_init(sb, + ERL_SUB_BITS_FLAGS_WRITABLE, + make_boxed((Eterm*)br), + &bin->orig_bytes[0], + 0, + current_size); *brp = br; *sbp = sb; @@ -1542,8 +1524,8 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, goto not_writable; } sb = (ErlSubBits*)ptr; - if (!sb->is_writable) { - goto not_writable; + if (!erl_sub_bits_is_writable(sb)) { + goto not_writable; } br = (BinRef *) boxed_val(sb->orig); @@ -1557,8 +1539,8 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, /* * OK, the binary is writable. */ - - erts_bin_offset = sb->size; + ASSERT(sb->start == 0); + erts_bin_offset = sb->end; if (unit > 1) { if ((unit == 8 && (erts_bin_offset & 7) != 0) || (unit != 8 && (erts_bin_offset % unit) != 0)) { @@ -1584,9 +1566,9 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, used_size_in_bits = erts_bin_offset + build_size_in_bits; /* Make sure that no one else can append to the incoming bitstring. */ - sb->is_writable = 0; + erl_sub_bits_clear_writable(sb); - update_wb_overhead(c_p, br, sb->size, used_size_in_bits); + update_wb_overhead(c_p, br, sb->end, used_size_in_bits); binp->intern.flags |= BIN_FLAG_ACTIVE_WRITER; /* Reallocate the underlying binary if it is too small. */ @@ -1595,13 +1577,12 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, binp = erts_bin_realloc(binp, new_size); br->val = binp; - br->bytes = (byte*)binp->orig_bytes; BUMP_REDS(c_p, erts_bin_offset / BITS_PER_REDUCTION); } binp->intern.apparent_size = NBYTES(used_size_in_bits); - erts_current_bin = br->bytes; + erts_current_bin = (byte*)binp->orig_bytes; /* Allocate heap space and build a new sub binary. */ reg[live] = sb->orig; @@ -1611,12 +1592,15 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, (void)erts_garbage_collect(c_p, heap_need, reg, live + 1); } - sb = (ErlSubBits *) c_p->htop; + sb = (ErlSubBits*)c_p->htop; c_p->htop += ERL_SUB_BITS_SIZE; - sb->thing_word = HEADER_SUB_BITS; - ERTS_SET_SB_RANGE(sb, 0, used_size_in_bits); - sb->is_writable = 1; - sb->orig = reg[live]; + + erl_sub_bits_init(sb, + ERL_SUB_BITS_FLAGS_WRITABLE, + reg[live], + erts_current_bin, + 0, + used_size_in_bits); return make_bitstring(sb); @@ -1638,7 +1622,7 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, bin = reg[live]; } - /* Calculate sizes. The size of the new binary, is the sum of the + /* Calculate sizes. The size of the new binary is the sum of the * build size and the size of the old binary. Allow some room * for growing. */ ERTS_GET_BITSTRING(bin, src_bytes, src_offset, src_size); @@ -1671,6 +1655,7 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, &c_p->htop, erts_bin_nrml_alloc(MAX(alloc_size, 256)), used_size_in_bits, + used_size_in_bits, &br, &sb); @@ -1684,8 +1669,6 @@ erts_bs_append_checked(Process* c_p, Eterm* reg, Uint live, src_size); BUMP_REDS(c_p, src_size / BITS_PER_REDUCTION); - ERTS_SET_SB_RANGE(sb, 0, used_size_in_bits); - return make_bitstring(sb); } } @@ -1732,7 +1715,8 @@ erts_bs_private_append_checked(Process* p, Eterm bin, Uint build_size_in_bits, U ASSERT(br->thing_word == HEADER_BIN_REF); /* Calculate new size in bits. */ - erts_bin_offset = sb->size; + ASSERT(sb->start == 0); + erts_bin_offset = sb->end; if((ERTS_UINT_MAX - build_size_in_bits) < erts_bin_offset) { p->fvalue = am_size; @@ -1743,7 +1727,7 @@ erts_bs_private_append_checked(Process* p, Eterm bin, Uint build_size_in_bits, U refc_binary = br->val; new_position = erts_bin_offset + build_size_in_bits; - update_wb_overhead(p, br, sb->size, new_position); + update_wb_overhead(p, br, sb->end, new_position); used_size = NBYTES(new_position); new_size = GROW_PROC_BIN_SIZE(used_size); @@ -1752,15 +1736,20 @@ erts_bs_private_append_checked(Process* p, Eterm bin, Uint build_size_in_bits, U /* This is the normal case - the binary is writable. There are no other * references to the binary, so it is safe to reallocate it when it's * too small. */ - ASSERT(sb->is_writable); + ASSERT(erl_sub_bits_is_writable(sb)); ASSERT(erts_refc_read(&refc_binary->intern.refc, 1) == 1); if (refc_binary->orig_size < used_size) { refc_binary = erts_bin_realloc(refc_binary, new_size); br->val = refc_binary; - br->bytes = (byte*)refc_binary->orig_bytes; BUMP_REDS(p, erts_bin_offset / BITS_PER_REDUCTION); } + + ASSERT(sb->start == 0); + sb->end = new_position; + + refc_binary->intern.flags |= BIN_FLAG_ACTIVE_WRITER; + refc_binary->intern.apparent_size = used_size; } else { /* The binary is NOT writable. The only way that this can happen is * when call tracing is turned on, which means that a trace process now @@ -1776,7 +1765,13 @@ erts_bs_private_append_checked(Process* p, Eterm bin, Uint build_size_in_bits, U Binary *new_binary = erts_bin_nrml_alloc(new_size); Eterm *hp = HeapFragOnlyAlloc(p, ERL_REFC_BITS_SIZE); - build_writable_bitstring(p, &hp, new_binary, new_position, &br, &sb); + build_writable_bitstring(p, + &hp, + new_binary, + new_position, + new_position, + &br, + &sb); sys_memcpy(new_binary->orig_bytes, refc_binary->orig_bytes, @@ -1786,20 +1781,17 @@ erts_bs_private_append_checked(Process* p, Eterm bin, Uint build_size_in_bits, U refc_binary = new_binary; } - refc_binary->intern.flags = BIN_FLAG_WRITABLE | BIN_FLAG_ACTIVE_WRITER; - refc_binary->intern.apparent_size = used_size; - - ERTS_SET_SB_RANGE(sb, 0, new_position); + ASSERT(refc_binary->intern.flags & BIN_FLAG_WRITABLE); + erts_current_bin = (byte*)&refc_binary->orig_bytes[0]; - erts_current_bin = br->bytes; - - return bin; + return make_bitstring(sb); } Eterm erts_bs_init_writable(Process* p, Eterm sz) { Uint bin_size = 1024; + Binary *refc_binary; ErlSubBits *sb; BinRef *br; @@ -1814,50 +1806,51 @@ erts_bs_init_writable(Process* p, Eterm sz) (void)erts_garbage_collect(p, ERL_REFC_BITS_SIZE, NULL, 0); } + refc_binary = erts_bin_nrml_alloc(bin_size); build_writable_bitstring(p, &p->htop, - erts_bin_nrml_alloc(bin_size), + refc_binary, + 0, bin_size * 8, &br, &sb); - ERTS_SET_SB_RANGE(sb, 0, 0); (void)br; return make_bitstring(sb); } -int erts_pin_writable_binary(BinRef *br) { - enum binary_flags flags; - Binary *refc_binary; +void erts_pin_writable_binary(ErlSubBits *sb, BinRef *br) { + if (erl_sub_bits_was_writable(sb)) { + enum binary_flags flags; + Binary *refc_binary; - refc_binary = br->val; - flags = refc_binary->intern.flags; + refc_binary = br->val; + flags = refc_binary->intern.flags; - if (flags & (BIN_FLAG_WRITABLE | BIN_FLAG_ACTIVE_WRITER)) { - Uint apparent_size = refc_binary->intern.apparent_size; + if (flags & (BIN_FLAG_WRITABLE | BIN_FLAG_ACTIVE_WRITER)) { + Uint apparent_size = refc_binary->intern.apparent_size; - ASSERT(refc_binary->orig_size >= apparent_size); - ASSERT((flags & ~(BIN_FLAG_WRITABLE | BIN_FLAG_ACTIVE_WRITER)) == 0); - ASSERT(erts_refc_read(&refc_binary->intern.refc, 1) == 1); + ASSERT(refc_binary->orig_size >= apparent_size); + ASSERT(!(flags & ~(BIN_FLAG_WRITABLE | BIN_FLAG_ACTIVE_WRITER))); + ASSERT(erts_refc_read(&refc_binary->intern.refc, 1) == 1); - refc_binary->intern.flags = 0; + refc_binary->intern.flags = 0; - /* Our allocators are 8 byte aligned, i.e., shrinking with less than 8 - * bytes will have no real effect */ - if (refc_binary->orig_size - apparent_size >= 8) { - refc_binary = erts_bin_realloc(refc_binary, apparent_size); - - br->val = refc_binary; - br->bytes = (byte*)refc_binary->orig_bytes; - - return 1; + /* Our allocators are 8 byte aligned, i.e., shrinking with less + * than 8 bytes will have no real effect */ + if (refc_binary->orig_size - apparent_size >= 8) { + refc_binary = erts_bin_realloc(refc_binary, apparent_size); + br->val = refc_binary; + } } + + sb->base_flags = (UWord)refc_binary->orig_bytes; } - return 0; + ASSERT(erl_sub_bits_is_normal(sb)); } Uint32 -erts_bs_get_unaligned_uint32(ErlBinMatchBuffer* mb) +erts_bs_get_unaligned_uint32(ErlSubBits* sb) { Uint bytes; Uint offs; @@ -1865,9 +1858,9 @@ erts_bs_get_unaligned_uint32(ErlBinMatchBuffer* mb) byte* LSB; byte* MSB; - CHECK_MATCH_BUFFER(mb); - ASSERT((mb->offset & 7) != 0); - ASSERT(mb->size - mb->offset >= 32); + CHECK_MATCH_BUFFER(sb); + ASSERT((sb->start & 7) != 0); + ASSERT(sb->end - sb->start >= 32); bytes = 4; offs = 0; @@ -1876,14 +1869,14 @@ erts_bs_get_unaligned_uint32(ErlBinMatchBuffer* mb) MSB = LSB + bytes - 1; *MSB = 0; - erts_copy_bits(mb->base, mb->offset, 1, MSB, offs, -1, 32); + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, MSB, offs, -1, 32); return LSB[0] | (LSB[1]<<8) | (LSB[2]<<16) | (LSB[3]<<24); } static void -erts_align_utf8_bytes(ErlBinMatchBuffer* mb, byte* buf) +erts_align_utf8_bytes(ErlSubBits *sb, byte* buf) { - Uint bits = mb->size - mb->offset; + Uint bits = sb->end - sb->start; /* * Copy up to 4 bytes into the supplied buffer. @@ -1899,15 +1892,15 @@ erts_align_utf8_bytes(ErlBinMatchBuffer* mb, byte* buf) } else { bits = 16; } - erts_copy_bits(mb->base, mb->offset, 1, buf, 0, 1, bits); + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, buf, 0, 1, bits); } Eterm -erts_bs_get_utf8(ErlBinMatchBuffer* mb) +erts_bs_get_utf8(ErlSubBits *sb) { Eterm result; Uint remaining_bits; - byte* pos; + const byte *pos; byte tmp_buf[4]; Eterm a, b, c; @@ -1925,22 +1918,22 @@ erts_bs_get_utf8(ErlBinMatchBuffer* mb) 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 3,3,3,3,3,3,3,3,9,9,9,9,9,9,9,9 }; - CHECK_MATCH_BUFFER(mb); + CHECK_MATCH_BUFFER(sb); - if ((remaining_bits = mb->size - mb->offset) < 8) { + if ((remaining_bits = sb->end - sb->start) < 8) { return THE_NON_VALUE; } - if (BIT_OFFSET(mb->offset) == 0) { - pos = mb->base + BYTE_OFFSET(mb->offset); + if (BIT_OFFSET(sb->start) == 0) { + pos = erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start); } else { - erts_align_utf8_bytes(mb, tmp_buf); + erts_align_utf8_bytes(sb, tmp_buf); pos = tmp_buf; } result = pos[0]; switch (erts_trailing_bytes_for_utf8[result]) { case 0: /* One byte only */ - mb->offset += 8; + sb->start += 8; break; case 1: /* Two bytes */ @@ -1952,7 +1945,7 @@ erts_bs_get_utf8(ErlBinMatchBuffer* mb) return THE_NON_VALUE; } result = (result << 6) + a - (Eterm) 0x00003080UL; - mb->offset += 16; + sb->start += 16; break; case 2: /* Three bytes */ @@ -1969,7 +1962,7 @@ erts_bs_get_utf8(ErlBinMatchBuffer* mb) if (0xD800 <= result && result <= 0xDFFF) { return THE_NON_VALUE; } - mb->offset += 24; + sb->start += 24; break; case 3: /* Four bytes */ @@ -1989,7 +1982,7 @@ erts_bs_get_utf8(ErlBinMatchBuffer* mb) if (result > 0x10FFFF) { return THE_NON_VALUE; } - mb->offset += 32; + sb->start += 32; break; default: return THE_NON_VALUE; @@ -1998,10 +1991,10 @@ erts_bs_get_utf8(ErlBinMatchBuffer* mb) } Eterm -erts_bs_get_utf16(ErlBinMatchBuffer* mb, Uint flags) +erts_bs_get_utf16(ErlSubBits *sb, Uint flags) { Uint bit_offset; - Uint num_bits = mb->size - mb->offset; + Uint num_bits = sb->end - sb->start; byte* src; byte tmp_buf[4]; Uint16 w1; @@ -2011,20 +2004,21 @@ erts_bs_get_utf16(ErlBinMatchBuffer* mb, Uint flags) return THE_NON_VALUE; } - CHECK_MATCH_BUFFER(mb); + CHECK_MATCH_BUFFER(sb); /* * Set up the pointer to the source bytes. */ - if ((bit_offset = BIT_OFFSET(mb->offset)) == 0) { + if ((bit_offset = BIT_OFFSET(sb->start)) == 0) { /* We can access the binary directly because the bytes are aligned. */ - src = mb->base + BYTE_OFFSET(mb->offset); + src = erl_sub_bits_get_base(sb) + BYTE_OFFSET(sb->start); } else { /* * We must copy the data to a temporary buffer. If possible, * get 4 bytes, otherwise two bytes. */ Uint n = num_bits < 32 ? 16 : 32; - erts_copy_bits(mb->base, mb->offset, 1, tmp_buf, 0, 1, n); + erts_copy_bits(erl_sub_bits_get_base(sb), sb->start, 1, + tmp_buf, 0, 1, n); src = tmp_buf; } @@ -2037,7 +2031,7 @@ erts_bs_get_utf16(ErlBinMatchBuffer* mb, Uint flags) w1 = (src[0] << 8) | src[1]; } if (w1 < 0xD800 || w1 > 0xDFFF) { - mb->offset += 16; + sb->start += 16; return make_small(w1); } else if (w1 > 0xDBFF) { return THE_NON_VALUE; @@ -2056,7 +2050,7 @@ erts_bs_get_utf16(ErlBinMatchBuffer* mb, Uint flags) if (!(0xDC00 <= w2 && w2 <= 0xDFFF)) { return THE_NON_VALUE; } - mb->offset += 32; + sb->start += 32; return make_small((((w1 & 0x3FF) << 10) | (w2 & 0x3FF)) + 0x10000UL); } @@ -2304,15 +2298,25 @@ Eterm erts_build_sub_bitstring(Eterm **hp, return result; } else { ErlSubBits *sb = (ErlSubBits*)*hp; - *hp += ERL_SUB_BITS_SIZE; + UWord flags = 0; ASSERT(br && ((br_flags & _TAG_PRIMARY_MASK) == TAG_PRIMARY_BOXED)); - sb->thing_word = HEADER_SUB_BITS; - ERTS_SET_SB_RANGE(sb, offset, size); - sb->orig = ((Eterm)br) | br_flags; - sb->is_writable = 0; + /* If the underlying binary is writable, we have to mark the result as + * volatile. We can skip this indirection once the flags move into + * br_flags. */ + if ((br->val)->intern.flags & BIN_FLAG_WRITABLE) { + flags |= ERL_SUB_BITS_FLAG_VOLATILE; + } + + erl_sub_bits_init(sb, + flags, + ((Eterm)br) | br_flags, + base, + offset, + size); + *hp += ERL_SUB_BITS_SIZE; return make_bitstring(sb); } } @@ -2327,18 +2331,26 @@ Eterm erts_wrap_refc_bitstring(struct erl_off_heap_header **oh, { ErlSubBits *sb = (ErlSubBits*)&(*hpp)[ERL_BIN_REF_SIZE]; BinRef *br = (BinRef*)*hpp; + UWord flags = 0; ASSERT(bin != NULL); br->thing_word = HEADER_BIN_REF; br->next = (*oh); br->val = bin; - br->bytes = bytes; - sb->thing_word = HEADER_SUB_BITS; - sb->is_writable = 0; - sb->orig = make_boxed((Eterm*)br); - ERTS_SET_SB_RANGE(sb, offset, size); + /* If the underlying binary is writable, we have to mark the result as + * volatile. */ + if (bin->intern.flags & BIN_FLAG_WRITABLE) { + flags |= ERL_SUB_BITS_FLAG_VOLATILE; + } + + erl_sub_bits_init(sb, + flags, + make_boxed((Eterm*)br), + bytes, + offset, + size); *oh = (struct erl_off_heap_header*)br; *overhead += size / NBITS(sizeof(Eterm)); @@ -2400,7 +2412,7 @@ erts_hfact_new_bitstring(ErtsHeapFactory *hfact, Uint reserve_size, reserve_size); hb->thing_word = header_heap_bits(size); - ERTS_SET_HB_SIZE(hb, size); + hb->size = size; *datap = (byte*)hb->data; @@ -2451,7 +2463,7 @@ erts_new_bitstring_refc(Process *p, Uint size, Binary **binp, byte **datap) hb = (ErlHeapBits *)HAlloc(p, heap_bits_size(size)); hb->thing_word = header_heap_bits(size); - ERTS_SET_HB_SIZE(hb, size); + hb->size = size; *datap = (byte*)hb->data; @@ -2521,19 +2533,21 @@ erts_shrink_binary_term(Eterm binary, size_t size) if (thing_subtag(*ptr) == HEAP_BITS_SUBTAG) { ErlHeapBits *hb = (ErlHeapBits*)ptr; ASSERT(TAIL_BITS(hb->size) == 0 && hb->size >= NBITS(size)); - ERTS_SET_HB_SIZE(hb, NBITS(size)); + hb->size = NBITS(size); } else { ErlSubBits *sb = (ErlSubBits*)ptr; BinRef *br = (BinRef*)boxed_val(sb->orig); - ASSERT(TAIL_BITS(sb->size) == 0 && sb->size >= NBITS(size)); - ERTS_SET_SB_RANGE(sb, sb->offs, NBITS(size)); + ASSERT(erl_sub_bits_is_normal(sb)); + ASSERT(TAIL_BITS(sb->end) == 0 && sb->end >= NBITS(size)); + ASSERT(sb->start == 0); + sb->end = NBITS(size); /* Our allocators are 8-byte aligned, so don't bother reallocating for * differences smaller than that. */ - if (size < (NBYTES(sb->size) + 8)) { + if (size < (NBYTES(sb->end) + 8)) { br->val = erts_bin_realloc(br->val, size); - br->bytes = (byte*)(br->val)->orig_bytes; + sb->base_flags = (UWord)&(br->val)->orig_bytes[0]; } } diff --git a/erts/emulator/beam/erl_bits.h b/erts/emulator/beam/erl_bits.h index 276d292cdb55..bd64b9c5d9fd 100644 --- a/erts/emulator/beam/erl_bits.h +++ b/erts/emulator/beam/erl_bits.h @@ -21,42 +21,6 @@ #ifndef __ERL_BITS_H__ #define __ERL_BITS_H__ -/* ************************************************************************* */ - -/** @brief This structure represents a binary to be matched, we plan to replace - * this with ErlSubBits in the near future. */ -typedef struct erl_bin_match_buffer { - Eterm orig; /* Original binary term. */ - byte* base; /* Current position in binary. */ - Uint offset; /* Offset in bits. */ - size_t size; /* Size of binary in bits. */ -} ErlBinMatchBuffer; - -typedef struct erl_bin_match_struct { - Eterm thing_word; - ErlBinMatchBuffer mb; /* Present match buffer */ - Eterm save_offset[1]; /* Saved offsets, only valid for contexts - * created through bs_start_match2. */ -} ErlBinMatchState; - -#define ERL_BIN_MATCHSTATE_SIZE(_Max) \ - ((offsetof(ErlBinMatchState, save_offset) + (_Max)*sizeof(Eterm))/sizeof(Eterm)) -#define HEADER_BIN_MATCHSTATE(_Max) \ - _make_header(ERL_BIN_MATCHSTATE_SIZE((_Max)) - 1, _TAG_HEADER_BIN_MATCHSTATE) -#define HEADER_NUM_SLOTS(hdr) \ - (header_arity(hdr) - (offsetof(ErlBinMatchState, save_offset) / sizeof(Eterm)) + 1) - -#define make_matchstate(_Ms) make_boxed((Eterm*)(_Ms)) -#define ms_matchbuffer(_Ms) &(((ErlBinMatchState*) boxed_val(_Ms))->mb) - -#define matchbuffer_base(Bin) \ - (*boxed_val(Bin) == HEADER_BIN_REF ? \ - ((BinRef*)boxed_val(Bin))->bytes : \ - (ASSERT(thing_subtag(*boxed_val(Bin)) == HEAP_BITS_SUBTAG), \ - (byte*)(&(((ErlHeapBits*)boxed_val(Bin))->data)))) - -/* ************************************************************************* */ - /** @brief returns the number of bytes needed to store \c x bits. */ #define NBYTES(x) (((Uint64)(x) + (Uint64) 7) >> 3) /** @brief returns the number of bits there are in \c x bytes. */ @@ -71,24 +35,85 @@ typedef struct erl_bin_match_struct { /* ************************************************************************* */ -#define bitstring_size(Bin) (bitstring_val(Bin)[1]) +#define bitstring_size(Bin) \ + ((bitstring_val(Bin)[0]) == HEADER_SUB_BITS ? \ + (((ErlSubBits*)bitstring_val(Bin))->end - \ + ((ErlSubBits*)bitstring_val(Bin))->start) : \ + (((ErlHeapBits*)bitstring_val(Bin))->size)) -/* This structure represents the term form of an off-heap bitstring. - * - * Note: The last field (orig) is not counted in arityval in the header to - * simplify garbage collection. */ +/* This structure represents either the term form of an off-heap bitstring, or + * a bitstring match context. */ typedef struct erl_sub_bits { - Eterm thing_word; /* Subtag SUB_BITS_SUBTAG. */ - Uint size; /* Size in bits. */ - Uint offs; /* Offset in bits. */ - byte is_writable; /* The underlying Binary* is writable */ - Eterm orig; /* Boxed BinRef* */ + /* Subtag SUB_BITS_SUBTAG. */ + Eterm thing_word; + + /* Combined base pointer and flag field, making room for the flags by + * truncating the lowest 2 bits from the base pointer and compensating by + * bumping start/end to match. + * + * This means that the base pointer returned from ERTS_GET_BITSTRING and + * friends is _ONLY_ valid when adjusted by the provided offset. + * + * This is placed first to simplify checking whether this is a viable + * match context. */ + UWord base_flags; + + /* Start and end offset in bits. */ + Uint start; + Uint end; + + /* Tagged pointer to BinRef or ErlHeapBits, the latter is valid iff this is + * a match context. */ + Eterm orig; } ErlSubBits; +#define ERL_SUB_BITS_FLAGS_MATCH_CONTEXT \ + ERL_SUB_BITS_FLAG_MUTABLE +#define ERL_SUB_BITS_FLAGS_WRITABLE \ + (ERL_SUB_BITS_FLAG_MUTABLE | ERL_SUB_BITS_FLAG_VOLATILE) + +/* Whether it's safe to modify the SB itself, for example match contexts or + * active writable binaries (erts_bs_private_append). */ +#define ERL_SUB_BITS_FLAG_MUTABLE ((UWord)(1 << 0)) +/* Whether the underlying data can move around, for example if the SB is (or + * was) a writable bitstring. */ +#define ERL_SUB_BITS_FLAG_VOLATILE ((UWord)(1 << 1)) +#define ERL_SUB_BITS_FLAG_MASK ((UWord)3) + +ERTS_GLB_INLINE void +erl_sub_bits_init(ErlSubBits *sb, UWord flags, Eterm orig, const void *base, + Uint offset, Uint size); + +ERTS_GLB_INLINE void +erl_sub_bits_update_moved(ErlSubBits *sb, Eterm orig); + +#define erl_sub_bits_is_normal(SubBits) \ + (erl_sub_bits_get_flags(SubBits) == 0) + +#define erl_sub_bits_is_match_context(SubBits) \ + (erl_sub_bits_get_flags(SubBits) == ERL_SUB_BITS_FLAG_MUTABLE) + +#define erl_sub_bits_is_writable(SubBits) \ + (erl_sub_bits_get_flags(SubBits) == \ + (ERL_SUB_BITS_FLAG_MUTABLE | ERL_SUB_BITS_FLAG_VOLATILE)) + +#define erl_sub_bits_was_writable(SubBits) \ + (erl_sub_bits_get_flags(SubBits) & ERL_SUB_BITS_FLAG_VOLATILE) + +#define erl_sub_bits_clear_writable(SubBits) \ + (ASSERT(erl_sub_bits_is_writable(SubBits)), \ + (SubBits)->base_flags &= ~ERL_SUB_BITS_FLAG_MUTABLE) + +#define erl_sub_bits_get_flags(SubBits) \ + ((SubBits)->base_flags & ERL_SUB_BITS_FLAG_MASK) + +#define erl_sub_bits_get_base(SubBits) \ + ((byte*)((SubBits)->base_flags & ~ERL_SUB_BITS_FLAG_MASK)) + /** @brief The size in words of an ErlSubBits. */ #define ERL_SUB_BITS_SIZE (sizeof(ErlSubBits) / sizeof(Eterm)) -#define HEADER_SUB_BITS _make_header(ERL_SUB_BITS_SIZE-2,_TAG_HEADER_SUB_BITS) +#define HEADER_SUB_BITS _make_header(ERL_SUB_BITS_SIZE-1,_TAG_HEADER_SUB_BITS) /** @brief A handle to an off-heap binary. While terms internally, these can * only be referred to by sub-bitstrings, and should never be exposed to the @@ -97,10 +122,8 @@ typedef struct bin_ref { Eterm thing_word; /* Subtag BIN_REF_SUBTAG. */ Binary *val; /* Pointer to Binary structure. */ struct erl_off_heap_header *next; - byte *bytes; /* Pointer to the actual data bytes. */ } BinRef; -/* process binaries stuff (special case of binaries) */ #define HEADER_BIN_REF _make_header(ERL_BIN_REF_SIZE-1,_TAG_HEADER_BIN_REF) /** @brief The size in words of a BinRef. */ @@ -198,14 +221,14 @@ struct erl_bits_state { /* Helpers for the bitstring syntax */ Eterm erts_bs_start_match_2(Process *p, Eterm Bin, Uint Max); -ErlBinMatchState *erts_bs_start_match_3(Process *p, Eterm Bin); -Eterm erts_bs_get_integer_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb); -Eterm erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb); +ErlSubBits *erts_bs_start_match_3(Process *p, Eterm Bin); +Eterm erts_bs_get_integer_2(Process *p, Uint num_bits, unsigned flags, ErlSubBits* sb); +Eterm erts_bs_get_float_2(Process *p, Uint num_bits, unsigned flags, ErlSubBits* sb); /* These will create heap binaries when appropriate, so they require free space * up to BUILD_SUB_BITSTRING_HEAP_NEED. */ -Eterm erts_bs_get_binary_2(Process *p, Uint num_bits, unsigned flags, ErlBinMatchBuffer* mb); -Eterm erts_bs_get_binary_all_2(Process *p, ErlBinMatchBuffer* mb); +Eterm erts_bs_get_binary_2(Process *p, Uint num_bits, unsigned flags, ErlSubBits* sb); +Eterm erts_bs_get_binary_all_2(Process *p, ErlSubBits* sb); /* Binary construction, new instruction set. */ int erts_new_bs_put_integer(ERL_BITS_PROTO_3(Eterm Integer, Uint num_bits, unsigned flags)); @@ -216,9 +239,9 @@ int erts_new_bs_put_binary_all(Process *c_p, Eterm Bin, Uint unit); Eterm erts_new_bs_put_float(Process *c_p, Eterm Float, Uint num_bits, int flags); void erts_new_bs_put_string(ERL_BITS_PROTO_2(byte* iptr, Uint num_bytes)); -Uint32 erts_bs_get_unaligned_uint32(ErlBinMatchBuffer* mb); -Eterm erts_bs_get_utf8(ErlBinMatchBuffer* mb); -Eterm erts_bs_get_utf16(ErlBinMatchBuffer* mb, Uint flags); +Uint32 erts_bs_get_unaligned_uint32(ErlSubBits* sb); +Eterm erts_bs_get_utf8(ErlSubBits* sb); +Eterm erts_bs_get_utf16(ErlSubBits* sb, Uint flags); Eterm erts_bs_append(Process* p, Eterm* reg, Uint live, Eterm build_size_term, Uint extra_words, Uint unit); Eterm erts_bs_append_checked(Process* p, Eterm* reg, Uint live, Uint size, @@ -254,7 +277,7 @@ int erts_cmp_bits__(const byte* a_ptr, /** @brief Pins an off-heap binary in place, ensuring that it cannot be moved * by the writable-binary optimization. */ -int erts_pin_writable_binary(BinRef *br); +void erts_pin_writable_binary(ErlSubBits *sb, BinRef *br); /* Calculate the heap space for a binary extracted by * erts_build_sub_bitstring(). */ @@ -322,25 +345,18 @@ Eterm erts_wrap_refc_bitstring(struct erl_off_heap_header **oh, (oh)->overhead += ((br)->val)->orig_size / sizeof(Eterm); \ } while(0) -#define ERTS_SET_HB_SIZE(hb, bit_size) \ - do { \ - Uint __bit_size = (bit_size); \ - (hb)->size = __bit_size; \ - } while(0) - -#define ERTS_SET_SB_RANGE(sb, bit_offset, bit_size) \ - do { \ - Uint __bit_size = (bit_size); \ - Uint __bit_offset = (bit_offset); \ - (sb)->size = __bit_size; \ - (sb)->offs = __bit_offset; \ - } while(0) - -/** @brief Extracts a window into the given bitstring. */ +/** @brief Extracts a window into the given bitstring. + * + * This works on _ALL_ bitstrings, including mutable ones like match contexts + * or writable binaries (where the returned values reflect the current state). + * + * In most routines that only look at the data, this lets us gloss over + * the fact that these terms are special, making it safe for the compiler to + * pass them to any routine known not to alias `Bin` itself. */ #define ERTS_GET_BITSTRING(Bin, \ Base, \ - BitOffset, \ - BitSize) \ + Offset, \ + Size) \ do { \ ERTS_DECLARE_DUMMY(const BinRef *_unused_br); \ ERTS_DECLARE_DUMMY(Eterm _unused_br_tag); \ @@ -348,8 +364,8 @@ Eterm erts_wrap_refc_bitstring(struct erl_off_heap_header **oh, _unused_br_tag, \ _unused_br, \ Base, \ - BitOffset, \ - BitSize); \ + Offset, \ + Size); \ } while (0) /** @brief As \c ERTS_GET_BITSTRING but also extracts the underlying binary @@ -357,18 +373,32 @@ Eterm erts_wrap_refc_bitstring(struct erl_off_heap_header **oh, #define ERTS_GET_BITSTRING_REF(Bin, RefFlags, Ref, Base, Offset, Size) \ do { \ const Eterm *_unboxed = bitstring_val(Bin); \ - Size = _unboxed[1]; \ if (*_unboxed == HEADER_SUB_BITS) { \ ErlSubBits* _sb = (ErlSubBits*)_unboxed; \ - BinRef *_br = ((BinRef*)boxed_val(_sb->orig)); \ - ASSERT(_br->thing_word == HEADER_BIN_REF); \ - Base = &_br->bytes[0]; \ + BinRef *_br; \ + _unboxed = boxed_val(_sb->orig); \ + /* Match contexts may refer to on-heap bitstrings */ \ + _br = (*_unboxed == HEADER_BIN_REF) ? (BinRef*)_unboxed : NULL; \ + ASSERT(erl_sub_bits_is_match_context(_sb) || _br); \ + if (ERTS_LIKELY(!erl_sub_bits_was_writable(_sb))) { \ + Base = (byte*)(_sb->base_flags & ~ERL_SUB_BITS_FLAG_MASK); \ + } else { \ + /* If the source was writable its underlying binary may */ \ + /* move at any point, so we need to follow it instead of */ \ + /* using the stored pointer. */ \ + Base = (byte*)((_br->val)->orig_bytes); \ + if (!((_br->val)->intern.flags & BIN_FLAG_WRITABLE)) { \ + _sb->base_flags = (UWord)Base; \ + } \ + } \ + Offset = _sb->start; \ + Size = _sb->end - _sb->start; \ RefFlags = _sb->orig & TAG_PTR_MASK__; \ Ref = _br; \ - Offset = _sb->offs; \ } else { \ const ErlHeapBits *_hb = ((ErlHeapBits*)_unboxed); \ Base = (byte*)&_hb->data[0]; \ + Size = _hb->size; \ Offset = 0; \ RefFlags = 0; \ Ref = NULL; \ @@ -380,21 +410,28 @@ Eterm erts_wrap_refc_bitstring(struct erl_off_heap_header **oh, #define ERTS_PIN_BITSTRING(Bin, RefFlags, Ref, Base, Offset, Size) \ do { \ const Eterm *_unboxed = bitstring_val(Bin); \ - Size = _unboxed[1]; \ if (*_unboxed == HEADER_SUB_BITS) { \ ErlSubBits* _sb = (ErlSubBits*)_unboxed; \ - BinRef *_br = ((BinRef*)boxed_val(_sb->orig)); \ - ASSERT(_br->thing_word == HEADER_BIN_REF); \ - erts_pin_writable_binary(_br); \ - Base = &_br->bytes[0]; \ - Offset = _sb->offs; \ + BinRef *_br; \ + _unboxed = boxed_val(_sb->orig); \ + /* Match contexts may refer to on-heap bitstrings */ \ + _br = (*_unboxed == HEADER_BIN_REF) ? (BinRef*)_unboxed : NULL; \ + ASSERT(erl_sub_bits_is_match_context(_sb) || _br); \ + if (ERTS_LIKELY(_br != NULL)) { \ + /* Pinning updates base and flags when necessary. */ \ + erts_pin_writable_binary(_sb, _br); \ + } \ + Base = (byte*)(_sb->base_flags & ~ERL_SUB_BITS_FLAG_MASK); \ + Offset = _sb->start; \ + Size = _sb->end - _sb->start; \ RefFlags = _sb->orig & TAG_PTR_MASK__; \ Ref = _br; \ } else { \ const ErlHeapBits *_hb = ((ErlHeapBits*)_unboxed); \ Base = (byte*)&_hb->data[0]; \ - Ref = NULL; \ + Size = _hb->size; \ Offset = 0; \ + Ref = NULL; \ } \ } while (0) @@ -445,6 +482,57 @@ erts_free_aligned_binary_bytes(const byte* buf); #if ERTS_GLB_INLINE_INCL_FUNC_DEF +ERTS_GLB_INLINE void +erl_sub_bits_init(ErlSubBits *sb, UWord flags, Eterm orig, const void *base, + Uint offset, Uint size) +{ + /* The data pointers that we produce are all aligned, but unaligned ones + * can sneak in through resource binaries or the likes, in which case we + * cannot use the lower bits for storage. + * + * To handle this corner case, we'll adjust the incoming pointer and shift + * the start of the bitstring to fit. */ + Uint adjustment = (UWord)base & ERL_SUB_BITS_FLAG_MASK; + + ASSERT(is_boxed(orig)); + ASSERT(!(flags & ~ERL_SUB_BITS_FLAG_MASK)); + +#ifdef DEBUG + if (*boxed_val(orig) == HEADER_BIN_REF) { + Binary *bin = ((BinRef*)boxed_val(orig))->val; + ASSERT((flags & ERL_SUB_BITS_FLAG_VOLATILE) || + !(bin->intern.flags & BIN_FLAG_WRITABLE)); + } else { + ASSERT(flags == ERL_SUB_BITS_FLAGS_MATCH_CONTEXT); + } +#endif + + sb->thing_word = HEADER_SUB_BITS; + sb->start = offset + adjustment * 8; + sb->end = sb->start + size; + sb->base_flags = ((UWord)base - adjustment) | flags; + sb->orig = orig; +} + +ERTS_GLB_INLINE void +erl_sub_bits_update_moved(ErlSubBits *sb, Eterm orig) +{ + Eterm *ptr = ptr_val(orig); + + if (thing_subtag(*ptr) == HEAP_BITS_SUBTAG) { + UWord new_base = (UWord)&((ErlHeapBits*)ptr)->data; + + /* Only match contexts can refer to on-heap bitstrings, ordinary + * ErlSubBits _must_ point to a BinRef at all times. */ + ASSERT(erl_sub_bits_is_match_context(sb)); + + /* Heap alignment guarantees that we don't need to adjust offset + * according to the base here. */ + ASSERT(!(new_base & ERL_SUB_BITS_FLAG_MASK)); + sb->base_flags = new_base | (sb->base_flags & ERL_SUB_BITS_FLAG_MASK); + } +} + ERTS_GLB_INLINE void copy_binary_to_buffer(byte *dst_base, Uint dst_offset, const byte *src_base, Uint src_offset, diff --git a/erts/emulator/beam/erl_db.c b/erts/emulator/beam/erl_db.c index 8f5e1a9543d2..009a0d2dd45e 100644 --- a/erts/emulator/beam/erl_db.c +++ b/erts/emulator/beam/erl_db.c @@ -1203,35 +1203,29 @@ BIF_RETTYPE ets_take_2(BIF_ALIST_2) BIF_RET(ret); } -/* -** update_element(Tab, Key, {Pos, Value}) -** update_element(Tab, Key, [{Pos, Value}]) -*/ -BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) +static BIF_RETTYPE do_update_element(Process *p, DbTable *tb, + Eterm key, Eterm pos_val, Eterm default_obj) { - DbTable* tb; int cret = DB_ERROR_BADITEM; Eterm list; Eterm iter; - DeclareTmpHeap(cell,2,BIF_P); + DeclareTmpHeap(cell,2,p); DbUpdateHandle handle; - DB_BIF_GET_TABLE(tb, DB_WRITE, LCK_WRITE_REC, BIF_ets_update_element_3); - - UseTmpHeap(2,BIF_P); + UseTmpHeap(2,p); if (!(tb->common.status & (DB_SET | DB_ORDERED_SET | DB_CA_ORDERED_SET))) { - BIF_P->fvalue = EXI_TAB_TYPE; + p->fvalue = EXI_TAB_TYPE; cret = DB_ERROR_BADPARAM; goto bail_out; } - if (is_tuple(BIF_ARG_3)) { - list = CONS(cell, BIF_ARG_3, NIL); + if (is_tuple(pos_val)) { + list = CONS(cell, pos_val, NIL); } else { - list = BIF_ARG_3; + list = pos_val; } - if (!tb->common.meth->db_lookup_dbterm(BIF_P, tb, BIF_ARG_2, THE_NON_VALUE, &handle)) { + if (!tb->common.meth->db_lookup_dbterm(p, tb, key, default_obj, &handle)) { cret = DB_ERROR_BADKEY; goto bail_out; } @@ -1256,12 +1250,13 @@ BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) } position = signed_val(pvp[1]); if (position == tb->common.keypos) { - BIF_P->fvalue = EXI_KEY_POS; + p->fvalue = EXI_KEY_POS; cret = DB_ERROR_UNSPEC; goto finalize; } - if (position < 1 || position == tb->common.keypos || - position > arityval(handle.dbterm->tpl[0])) { + if (position < 1 || position > arityval(handle.dbterm->tpl[0])) { + p->fvalue = EXI_POSITION; + cret = DB_ERROR_UNSPEC; goto finalize; } } @@ -1278,7 +1273,7 @@ BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) tb->common.meth->db_finalize_dbterm(cret, &handle); bail_out: - UnUseTmpHeap(2,BIF_P); + UnUseTmpHeap(2,p); db_unlock(tb, LCK_WRITE_REC); switch (cret) { @@ -1287,13 +1282,44 @@ BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) case DB_ERROR_BADKEY: BIF_RET(am_false); case DB_ERROR_SYSRES: - BIF_ERROR(BIF_P, SYSTEM_LIMIT); + BIF_ERROR(p, SYSTEM_LIMIT); case DB_ERROR_UNSPEC: - BIF_ERROR(BIF_P, BADARG | EXF_HAS_EXT_INFO); + BIF_ERROR(p, BADARG | EXF_HAS_EXT_INFO); default: break; } - BIF_ERROR(BIF_P, BADARG); + BIF_ERROR(p, BADARG); +} + +/* +** update_element(Tab, Key, {Pos, Value}) +** update_element(Tab, Key, [{Pos, Value}]) +*/ +BIF_RETTYPE ets_update_element_3(BIF_ALIST_3) +{ + DbTable* tb; + + DB_BIF_GET_TABLE(tb, DB_WRITE, LCK_WRITE_REC, BIF_ets_update_element_3); + + return do_update_element(BIF_P, tb, BIF_ARG_2, BIF_ARG_3, THE_NON_VALUE); +} + +/* +** update_element(Tab, Key, {Pos, Value}, Default) +** update_element(Tab, Key, [{Pos, Value}], Default) +*/ +BIF_RETTYPE ets_update_element_4(BIF_ALIST_4) +{ + DbTable* tb; + + DB_BIF_GET_TABLE(tb, DB_WRITE, LCK_WRITE_REC, BIF_ets_update_element_4); + + if (is_not_tuple(BIF_ARG_4)) { + db_unlock(tb, LCK_WRITE_REC); + BIF_ERROR(BIF_P, BADARG); + } + + return do_update_element(BIF_P, tb, BIF_ARG_2, BIF_ARG_3, BIF_ARG_4); } static BIF_RETTYPE diff --git a/erts/emulator/beam/erl_db_util.c b/erts/emulator/beam/erl_db_util.c index d905bda75896..e0122bbc71cb 100644 --- a/erts/emulator/beam/erl_db_util.c +++ b/erts/emulator/beam/erl_db_util.c @@ -3743,12 +3743,9 @@ void db_cleanup_offheap_comp(DbTerm* obj) case BIN_REF_SUBTAG: erts_bin_release(u.br->val); break; - case FUN_SUBTAG: - /* We _KNOW_ that this is a local fun, otherwise it would not - * be part of the off-heap list. */ - ASSERT(is_local_fun(u.fun)); - if (erts_refc_dectest(&u.fun->entry.fun->refc, 0) == 0) { - erts_erase_fun_entry(u.fun->entry.fun); + case FUN_REF_SUBTAG: + if (erts_refc_dectest(&(u.fref->entry)->refc, 0) == 0) { + erts_erase_fun_entry(u.fref->entry); } break; case REF_SUBTAG: diff --git a/erts/emulator/beam/erl_debug.c b/erts/emulator/beam/erl_debug.c index f11a3fe1f1d4..17755c152816 100644 --- a/erts/emulator/beam/erl_debug.c +++ b/erts/emulator/beam/erl_debug.c @@ -189,12 +189,12 @@ pdisplay1(fmtfn_t to, void *to_arg, Process* p, Eterm obj) case BITSTRING_DEF: erts_print(to, to_arg, "#Bin"); break; - case MATCHSTATE_DEF: - erts_print(to, to_arg, "#Matchstate"); - break; case BIN_REF_DEF: erts_print(to, to_arg, "#BinRef"); break; + case FUN_REF_DEF: + erts_print(to, to_arg, "#FunRef"); + break; default: erts_print(to, to_arg, "unknown object %x", obj); } diff --git a/erts/emulator/beam/erl_etp.c b/erts/emulator/beam/erl_etp.c index 0d36f198300b..1654304c9bd0 100644 --- a/erts/emulator/beam/erl_etp.c +++ b/erts/emulator/beam/erl_etp.c @@ -92,13 +92,13 @@ const Eterm etp_hole_marker = 0; #endif const Eterm etp_arityval_subtag = ARITYVAL_SUBTAG; -const Eterm etp_bin_matchstate_subtag = BIN_MATCHSTATE_SUBTAG; const Eterm etp_big_tag_mask = _BIG_TAG_MASK; const Eterm etp_big_sign_bit = _BIG_SIGN_BIT; const Eterm etp_pos_big_subtag = POS_BIG_SUBTAG; const Eterm etp_neg_big_subtag = NEG_BIG_SUBTAG; const Eterm etp_ref_subtag = REF_SUBTAG; const Eterm etp_fun_subtag = FUN_SUBTAG; +const Eterm etp_fun_ref_subtag = FUN_REF_SUBTAG; const Eterm etp_float_subtag = FLOAT_SUBTAG; const Eterm etp_bitstring_tag_mask = _BITSTRING_TAG_MASK; const Eterm etp_heap_bits_subtag = HEAP_BITS_SUBTAG; @@ -122,7 +122,6 @@ const Eterm etp_tag_header_sub_bits = _TAG_HEADER_SUB_BITS; const Eterm etp_tag_header_external_pid = _TAG_HEADER_EXTERNAL_PID; const Eterm etp_tag_header_external_port = _TAG_HEADER_EXTERNAL_PORT; const Eterm etp_tag_header_external_ref = _TAG_HEADER_EXTERNAL_REF; -const Eterm etp_tag_header_bin_matchstate = _TAG_HEADER_BIN_MATCHSTATE; const Eterm etp_tag_header_map = _TAG_HEADER_MAP; const Eterm etp_tag_header_mask = _TAG_HEADER_MASK; diff --git a/erts/emulator/beam/erl_flxctr.h b/erts/emulator/beam/erl_flxctr.h index 6109705b4516..6065a5cb3ae4 100644 --- a/erts/emulator/beam/erl_flxctr.h +++ b/erts/emulator/beam/erl_flxctr.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2019-2021. All Rights Reserved. + * Copyright Ericsson AB 2019-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -290,7 +290,7 @@ int erts_flxctr_suspend_until_thr_prg_if_snapshot_ongoing(ErtsFlxCtr* c, Process /** * @brief This function returns the number of bytes that are allocated - * for for the given FlxCtr. + * for the given FlxCtr. * * @return nr of bytes allocated for the FlxCtr */ diff --git a/erts/emulator/beam/erl_fun.c b/erts/emulator/beam/erl_fun.c index 2fa939c70b3b..2f2585658ccd 100644 --- a/erts/emulator/beam/erl_fun.c +++ b/erts/emulator/beam/erl_fun.c @@ -147,7 +147,8 @@ erts_put_fun_entry2(Eterm mod, int old_uniq, int old_index, return &fc->entry; } -const ErtsCodeMFA *erts_get_fun_mfa(const ErlFunEntry *fe, ErtsCodeIndex ix) { +const ErtsCodeMFA *erts_get_fun_mfa(const ErlFunEntry *fe, ErtsCodeIndex ix) +{ ErtsCodePtr address = fe->dispatch.addresses[ix]; if (address != beam_unloaded_fun) { @@ -157,13 +158,15 @@ const ErtsCodeMFA *erts_get_fun_mfa(const ErlFunEntry *fe, ErtsCodeIndex ix) { return NULL; } -void erts_set_fun_code(ErlFunEntry *fe, ErtsCodeIndex ix, ErtsCodePtr address) { +void erts_set_fun_code(ErlFunEntry *fe, ErtsCodeIndex ix, ErtsCodePtr address) +{ /* Fun entries MUST NOT be updated during a purge! */ ASSERT(fe->pend_purge_address == NULL); fe->dispatch.addresses[ix] = address; } -int erts_is_fun_loaded(const ErlFunEntry* fe, ErtsCodeIndex ix) { +int erts_is_fun_loaded(const ErlFunEntry* fe, ErtsCodeIndex ix) +{ return fe->dispatch.addresses[ix] != beam_unloaded_fun; } @@ -296,37 +299,6 @@ erts_fun_purge_complete(ErlFunEntry **funs, Uint no) ERTS_THR_WRITE_MEMORY_BARRIER; } -ErlFunThing *erts_new_local_fun_thing(Process *p, ErlFunEntry *fe, - int arity, int num_free) -{ - ErlFunThing *funp; - - funp = (ErlFunThing*) p->htop; - p->htop += ERL_FUN_SIZE + num_free; - erts_refc_inc(&fe->refc, 2); - - funp->thing_word = MAKE_FUN_HEADER(arity, num_free, 0); - funp->entry.fun = fe; - - funp->next = MSO(p).first; - MSO(p).first = (struct erl_off_heap_header*) funp; - -#ifdef DEBUG - { - /* Note that `mfa` may be NULL if the fun is currently being purged. We - * ignore this since it's not an error and we only need `mfa` to - * sanity-check the arity at this point. If the fun is called while in - * this state, the `error_handler` module will take care of it. */ - const ErtsCodeMFA *mfa = erts_get_fun_mfa(fe, erts_active_code_ix()); - ASSERT(!mfa || fun_arity(funp) == mfa->arity - num_free); - ASSERT(arity == fe->arity); - } -#endif - - return funp; -} - - struct dump_fun_foreach_args { fmtfn_t to; void *to_arg; @@ -379,9 +351,7 @@ fun_cmp(ErlFunEntryContainer* obj1, ErlFunEntryContainer* obj2) ErlFunEntry* fe1 = &obj1->entry; ErlFunEntry* fe2 = &obj2->entry; - return !(fe1->old_index == fe2->old_index && - fe1->old_uniq == fe2->old_uniq && - fe1->module == fe2->module && + return !(fe1->module == fe2->module && fe1->index == fe2->index && fe1->arity == fe2->arity && !sys_memcmp(fe1->uniq, fe2->uniq, sizeof(fe1->uniq))); diff --git a/erts/emulator/beam/erl_fun.h b/erts/emulator/beam/erl_fun.h index 705f1ceffe7a..2dce49180799 100644 --- a/erts/emulator/beam/erl_fun.h +++ b/erts/emulator/beam/erl_fun.h @@ -42,17 +42,34 @@ typedef struct erl_fun_entry { int old_index; /* Old style index */ erts_refc_t refc; /* Reference count: One for code + one for - * each fun object in each process. */ + * each FunRef. */ ErtsCodePtr pend_purge_address; /* Address during a pending purge */ } ErlFunEntry; +/* Reference-holding structure for funs. As these normally live in the literal + * area of their module instance and are shared with all lambdas pointing to + * the same function, having it separated like this saves us from having to + * reference-count every single lambda. + * + * These references copied onto the heap under some circumstances, for example + * trips through ETS or the external term format, but generally they'll live + * off-heap. */ +typedef struct erl_fun_ref { + Eterm thing_word; + ErlFunEntry *entry; + struct erl_off_heap_header *next; +} FunRef; + +#define ERL_FUN_REF_SIZE ((sizeof(FunRef)/sizeof(Eterm))) +#define HEADER_FUN_REF _make_header(ERL_FUN_REF_SIZE-1,_TAG_HEADER_FUN_REF) + /* This structure represents a 'fun' (lambda), whether local or external. It is * stored on process heaps, and has variable size depending on the size of the * environment. */ typedef struct erl_fun_thing { - /* The header contains FUN_SUBTAG, arity, number of free variables, and - * whether this is an external fun. */ + /* The header contains FUN_SUBTAG, arity, and the size of the environment, + * the latter being zero for external funs and non-zero for local ones. */ Eterm thing_word; union { @@ -61,39 +78,35 @@ typedef struct erl_fun_thing { * pointer to improve performance. */ ErtsDispatchable *disp; - /* Pointer to function entry, valid iff the external bit is clear.*/ + /* Pointer to function entry, valid iff this is local fun. */ ErlFunEntry *fun; - /* Pointer to export entry, valid iff the external bit is set.*/ + /* Pointer to export entry, valid iff this is an external fun. */ Export *exp; } entry; - /* Next off-heap object, must be NULL when this is an external fun. */ - struct erl_off_heap_header *next; - - /* Environment (free variables), may be compound terms. */ + /* Environment (free variables), may be compound terms. + * + * External funs lack this altogether, and local funs _always_ reference a + * `FunRef` just past the last free variable. This ensures that the + * `ErlFunEntry` above will always be valid. */ Eterm env[]; } ErlFunThing; -#define is_external_fun(FunThing) \ - (!!(((FunThing)->thing_word >> FUN_HEADER_EXTERNAL_OFFS) & 1)) -#define is_local_fun(FunThing) \ - (!(is_external_fun(FunThing))) +#define is_external_fun(FunThing) (fun_env_size(FunThing) == 0) +#define is_local_fun(FunThing) (!is_external_fun(FunThing)) #define fun_arity(FunThing) \ (((FunThing)->thing_word >> FUN_HEADER_ARITY_OFFS) & 0xFF) +#define fun_env_size(FunThing) \ + ((FunThing)->thing_word >> FUN_HEADER_ENV_SIZE_OFFS) #define fun_num_free(FunThing) \ - (((FunThing)->thing_word >> FUN_HEADER_NUM_FREE_OFFS) & 0xFF) + (ASSERT(is_local_fun(FunThing)), fun_env_size(FunThing) - 1) /* ERL_FUN_SIZE does _not_ include space for the environment which is a * C99-style flexible array */ #define ERL_FUN_SIZE ((sizeof(ErlFunThing)/sizeof(Eterm))) -ErlFunThing *erts_new_local_fun_thing(Process *p, - ErlFunEntry *fe, - int arity, - int num_free); - void erts_init_fun_table(void); void erts_fun_info(fmtfn_t, void *); int erts_fun_table_sz(void); diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c index da0da58bb371..73944331b86f 100644 --- a/erts/emulator/beam/erl_gc.c +++ b/erts/emulator/beam/erl_gc.c @@ -243,7 +243,7 @@ erts_init_gc(void) ERTS_CT_ASSERT(offsetof(BinRef,thing_word) == offsetof(ErlFunThing,thing_word)); ERTS_CT_ASSERT(offsetof(BinRef,thing_word) == offsetof(ExternalThing,header)); ERTS_CT_ASSERT(offsetof(BinRef,next) == offsetof(struct erl_off_heap_header,next)); - ERTS_CT_ASSERT(offsetof(BinRef,next) == offsetof(ErlFunThing,next)); + ERTS_CT_ASSERT(offsetof(BinRef,next) == offsetof(FunRef,next)); ERTS_CT_ASSERT(offsetof(BinRef,next) == offsetof(ExternalThing,next)); erts_test_long_gc_sleep = 0; @@ -1314,12 +1314,9 @@ erts_garbage_collect_literals(Process* p, Eterm* literals, erts_refc_inc(&refc_binary->intern.refc, 2); break; } - case FUN_SUBTAG: + case FUN_REF_SUBTAG: { - /* We _KNOW_ that this is a local fun, otherwise it would - * not be part of the off-heap list. */ - ErlFunEntry *fe = ((ErlFunThing*)ptr)->entry.fun; - ASSERT(is_local_fun((ErlFunThing*)ptr)); + ErlFunEntry *fe = ((FunRef*)ptr)->entry; erts_refc_inc(&fe->refc, 2); break; } @@ -1728,26 +1725,27 @@ do_minor(Process *p, ErlHeapFragment *live_hf_end, break; } case TAG_PRIMARY_HEADER: { - if (!header_is_thing(gval)) - n_hp++; - else { - if (header_is_bin_matchstate(gval)) { - ErlBinMatchState *ms = (ErlBinMatchState*) n_hp; - ErlBinMatchBuffer *mb = &(ms->mb); - Eterm* origptr = &(mb->orig); - ptr = boxed_val(*origptr); - val = *ptr; - if (IS_MOVED_BOXED(val)) { - *origptr = val; - mb->base = matchbuffer_base(val); - } else if (ErtsInArea(ptr, mature, mature_size)) { - move_boxed(ptr,val,&old_htop,origptr); - mb->base = matchbuffer_base(mb->orig); - } else if (ErtsInYoungGen(*origptr, ptr, oh, oh_size)) { - move_boxed(ptr,val,&n_htop,origptr); - mb->base = matchbuffer_base(mb->orig); - } - } + if (!header_is_thing(gval)) { + n_hp++; + } else { + if (gval == HEADER_SUB_BITS) { + ErlSubBits *sb = (ErlSubBits*)n_hp; + Eterm *underlying = &sb->orig; + + ptr = boxed_val(*underlying); + val = *ptr; + + if (IS_MOVED_BOXED(val)) { + *underlying = val; + erl_sub_bits_update_moved(sb, val); + } else if (ErtsInArea(ptr, mature, mature_size)) { + move_boxed(ptr, val, &old_htop, underlying); + erl_sub_bits_update_moved(sb, sb->orig); + } else if (ErtsInYoungGen(*underlying, ptr, oh, oh_size)) { + move_boxed(ptr, val, &n_htop, underlying); + erl_sub_bits_update_moved(sb, sb->orig); + } + } n_hp += (thing_arityval(gval)+1); } break; @@ -2286,26 +2284,26 @@ sweep(Eterm *n_hp, Eterm *n_htop, break; } case TAG_PRIMARY_HEADER: { - if (!header_is_thing(gval)) { - n_hp++; - } else { - if (header_is_bin_matchstate(gval)) { - ErlBinMatchState *ms = (ErlBinMatchState*) n_hp; - ErlBinMatchBuffer *mb = &(ms->mb); - Eterm* origptr; - origptr = &(mb->orig); - ptr = boxed_val(*origptr); - val = *ptr; - if (IS_MOVED_BOXED(val)) { - *origptr = val; - mb->base = matchbuffer_base(*origptr); - } else if (ERTS_IS_IN_SWEEP_AREA(*origptr, ptr)) { - move_boxed(ptr,val,&n_htop,origptr); - mb->base = matchbuffer_base(*origptr); - } - } - n_hp += (thing_arityval(gval)+1); - } + if (!header_is_thing(gval)) { + n_hp++; + } else { + if (gval == HEADER_SUB_BITS) { + ErlSubBits *sb = (ErlSubBits*)n_hp; + Eterm *underlying = &sb->orig; + + ptr = boxed_val(*underlying); + val = *ptr; + + if (IS_MOVED_BOXED(val)) { + *underlying = val; + erl_sub_bits_update_moved(sb, *underlying); + } else if (ERTS_IS_IN_SWEEP_AREA(*underlying, ptr)) { + move_boxed(ptr, val, &n_htop, underlying); + erl_sub_bits_update_moved(sb, *underlying); + } + } + n_hp += (thing_arityval(gval)+1); + } break; } default: @@ -2382,26 +2380,26 @@ sweep_literals_to_old_heap(Eterm* heap_ptr, Eterm* heap_end, Eterm* htop, break; } case TAG_PRIMARY_HEADER: { - if (!header_is_thing(gval)) { - heap_ptr++; - } else { - if (header_is_bin_matchstate(gval)) { - ErlBinMatchState *ms = (ErlBinMatchState*) heap_ptr; - ErlBinMatchBuffer *mb = &(ms->mb); - Eterm* origptr; - origptr = &(mb->orig); - ptr = boxed_val(*origptr); - val = *ptr; - if (IS_MOVED_BOXED(val)) { - *origptr = val; - mb->base = matchbuffer_base(*origptr); - } else if (ErtsInArea(ptr, src, src_size)) { - move_boxed(ptr,val,&htop,origptr); - mb->base = matchbuffer_base(*origptr); - } - } - heap_ptr += (thing_arityval(gval)+1); - } + if (!header_is_thing(gval)) { + heap_ptr++; + } else { + if (gval == HEADER_SUB_BITS) { + ErlSubBits *sb = (ErlSubBits*)heap_ptr; + Eterm *underlying = &sb->orig; + + ptr = boxed_val(*underlying); + val = *ptr; + + if (IS_MOVED_BOXED(val)) { + *underlying = val; + erl_sub_bits_update_moved(sb, *underlying); + } else if (ErtsInArea(ptr, src, src_size)) { + move_boxed(ptr, val, &htop, underlying); + erl_sub_bits_update_moved(sb, *underlying); + } + } + heap_ptr += (thing_arityval(gval)+1); + } break; } default: @@ -2522,20 +2520,19 @@ erts_copy_one_frag(Eterm** hpp, ErlOffHeap* off_heap, case EXTERNAL_PID_SUBTAG: case EXTERNAL_PORT_SUBTAG: case EXTERNAL_REF_SUBTAG: + case FUN_REF_SUBTAG: oh = (struct erl_off_heap_header*) (hp-1); cpy_sz = thing_arityval(val); goto cpy_words; - case FUN_SUBTAG: + case SUB_BITS_SUBTAG: { - ErlFunThing *funp = (ErlFunThing*) (fhp - 1); - - if (is_local_fun(funp)) { - oh = (struct erl_off_heap_header*) (hp - 1); - } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); - } + /* Match contexts and writable binaries should never be present + * in signals. */ + ASSERT(erl_sub_bits_is_normal((ErlSubBits*)(fhp - 1))); - cpy_sz = thing_arityval(val); + /* Offset the `orig` field, as it's the last field inside the + * thing we can handle it by pretending it's not part of it. */ + cpy_sz = thing_arityval(val) - 1; goto cpy_words; } default: @@ -2931,7 +2928,6 @@ shrink_writable_binary(BinRef *br, Uint leave_unused) binary = erts_bin_realloc(binary, new_size); br->val = binary; - br->bytes = (byte*)binary->orig_bytes; } } @@ -3046,21 +3042,14 @@ sweep_off_heap(Process *p, int fullsweep) erts_bin_release(((BinRef*)ptr)->val); break; } - case FUN_SUBTAG: - { - ErlFunThing* funp = ((ErlFunThing*)ptr); - - if (is_local_fun(funp)) { - ErlFunEntry* fe = funp->entry.fun; - - if (erts_refc_dectest(&fe->refc, 0) == 0) { - erts_erase_fun_entry(fe); - } - } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); + case FUN_REF_SUBTAG: + { + FunRef *refp = ((FunRef*)ptr); + if (erts_refc_dectest(&(refp->entry)->refc, 0) == 0) { + erts_erase_fun_entry(refp->entry); } - break; - } + break; + } case REF_SUBTAG: { ErtsMagicBinary *bptr; @@ -3103,7 +3092,7 @@ sweep_off_heap(Process *p, int fullsweep) break; } default: - ASSERT(is_fun_header(ptr->thing_word) || + ASSERT(ptr->thing_word == HEADER_FUN_REF || is_external_header(ptr->thing_word) || is_magic_ref_thing(ptr)); break; @@ -3289,10 +3278,10 @@ offset_heap(Eterm* hp, Uint sz, Sint offs, char* area, Uint area_size) if (!is_magic_ref_thing(hp)) break; case BIN_REF_SUBTAG: - case FUN_SUBTAG: case EXTERNAL_PID_SUBTAG: case EXTERNAL_PORT_SUBTAG: case EXTERNAL_REF_SUBTAG: + case FUN_REF_SUBTAG: { struct erl_off_heap_header* oh = (struct erl_off_heap_header*) hp; @@ -3311,16 +3300,16 @@ offset_heap(Eterm* hp, Uint sz, Sint offs, char* area, Uint area_size) } } break; - case BIN_MATCHSTATE_SUBTAG: - { - ErlBinMatchState *ms = (ErlBinMatchState*) hp; - ErlBinMatchBuffer *mb = &(ms->mb); - if (ErtsInArea(ptr_val(mb->orig), area, area_size)) { - mb->orig = offset_ptr(mb->orig, offs); - mb->base = matchbuffer_base(mb->orig); - } - } - break; + case SUB_BITS_SUBTAG: + { + ErlSubBits *sb = (ErlSubBits*) hp; + + if (ErtsInArea(ptr_val(sb->orig), area, area_size)) { + sb->orig = offset_ptr(sb->orig, offs); + erl_sub_bits_update_moved(sb, sb->orig); + } + } + break; } sz -= tari; hp += tari + 1; @@ -3981,12 +3970,10 @@ check_all_heap_terms_in_range(int (*check_eterm)(Eterm), case ARITYVAL_SUBTAG: break; case BIN_REF_SUBTAG: - goto off_heap_common; - case FUN_SUBTAG: - goto off_heap_common; case EXTERNAL_PID_SUBTAG: case EXTERNAL_PORT_SUBTAG: case EXTERNAL_REF_SUBTAG: + case FUN_REF_SUBTAG: off_heap_common: { int tari = thing_arityval(val); @@ -4133,20 +4120,14 @@ erts_check_off_heap2(Process *p, Eterm *htop) case BIN_REF_SUBTAG: refc = erts_refc_read(&(u.br->val)->intern.refc, 1); break; - case FUN_SUBTAG: - if (is_local_fun(u.fun)) { - refc = erts_refc_read(&u.fun->entry.fun->refc, 1); - } else { - /* Export fun, fake a valid refc. */ - ASSERT(is_external_fun(u.fun) && u.fun->next == NULL); - refc = 1; - } - break; case EXTERNAL_PID_SUBTAG: case EXTERNAL_PORT_SUBTAG: case EXTERNAL_REF_SUBTAG: refc = erts_refc_read(&u.ext->node->refc, 1); break; + case FUN_REF_SUBTAG: + refc = erts_refc_read(&(u.fref->entry)->refc, 1); + break; case REF_SUBTAG: ASSERT(is_magic_ref_thing(u.hdr)); refc = erts_refc_read(&u.mref->mb->intern.refc, 1); diff --git a/erts/emulator/beam/erl_gc.h b/erts/emulator/beam/erl_gc.h index ab34d89bbc8e..ac8c772677bf 100644 --- a/erts/emulator/beam/erl_gc.h +++ b/erts/emulator/beam/erl_gc.h @@ -66,20 +66,26 @@ ERTS_GLB_INLINE Eterm* move_boxed(Eterm *ERTS_RESTRICT ptr, Eterm hdr, Eterm **h ASSERT(is_header(hdr)); nelts = header_arity(hdr); switch ((hdr) & _HEADER_SUBTAG_MASK) { - case SUB_BITS_SUBTAG: - nelts++; - break; case MAP_SUBTAG: - if (is_flatmap_header(hdr)) nelts+=flatmap_get_size(ptr) + 1; - else nelts += hashmap_bitcount(MAP_HEADER_VAL(hdr)); - break; - case FUN_SUBTAG: nelts+=fun_num_free((ErlFunThing*)(ptr)); break; + if (is_flatmap_header(hdr)) { + nelts += flatmap_get_size(ptr) + 1; + } else { + nelts += hashmap_bitcount(MAP_HEADER_VAL(hdr)); + } + break; + case FUN_SUBTAG: + nelts += fun_env_size((ErlFunThing*)(ptr)); + break; } + gval = make_boxed(htop); *orig = gval; *htop++ = hdr; *ptr++ = gval; - while (nelts--) *htop++ = *ptr++; + + while (nelts--) { + *htop++ = *ptr++; + } *hpp = htop; return ptr; diff --git a/erts/emulator/beam/erl_io_queue.c b/erts/emulator/beam/erl_io_queue.c index cb9d66e81c30..e6d40ace405c 100644 --- a/erts/emulator/beam/erl_io_queue.c +++ b/erts/emulator/beam/erl_io_queue.c @@ -731,8 +731,8 @@ erts_ioq_iolist_vec_len(Eterm obj, int* vsize, Uint* csize, } typedef struct { + Eterm *result_tail; Eterm result_head; - Eterm result_tail; Eterm input_list; struct erl_off_heap_header **off_heap_tail; @@ -786,8 +786,8 @@ static int iol2v_state_destructor(Binary *data) { static void iol2v_init(iol2v_state_t *state, Process *process, Eterm input) { state->process = process; + state->result_tail = &state->result_head; state->result_head = NIL; - state->result_tail = NIL; state->input_list = input; state->magic_reference = NIL; @@ -802,24 +802,33 @@ static void iol2v_init(iol2v_state_t *state, Process *process, Eterm input) { CLEAR_SAVED_ESTACK(&state->estack); } +static void iol2v_finish_accumulator(iol2v_state_t *state) { + Binary *accumulator; + BinRef *acc_ref; + Uint size; + + accumulator = state->acc; + state->acc = NULL; + + acc_ref = state->acc_ref; + ASSERT(acc_ref->val == accumulator); + + /* Our allocators are 8-byte aligned, so don't bother reallocating for + * differences smaller than that. */ + size = state->acc_offset + state->acc_size; + if (size < (accumulator->orig_size + 8)) { + acc_ref->val = erts_bin_realloc(accumulator, size); + } +} + /* Destructively enqueues a term to the result list, saving us the hassle of * having to reverse it later. This is safe since GC is disabled and we never * leak the unfinished term to the outside. */ static void iol2v_enqueue_result(iol2v_state_t *state, Eterm term) { - Eterm prev_tail; - Eterm *hp; + Eterm *hp = HAlloc(state->process, 2); - prev_tail = state->result_tail; - - hp = HAlloc(state->process, 2); - state->result_tail = CONS(hp, term, NIL); - - if(prev_tail != NIL) { - Eterm *prev_cell = list_val(prev_tail); - CDR(prev_cell) = state->result_tail; - } else { - state->result_head = state->result_tail; - } + *state->result_tail = CONS(hp, term, NIL); + state->result_tail = &hp[1]; state->reds_spent += 1; } @@ -830,20 +839,21 @@ static Eterm iol2v_extract_acc_term(iol2v_state_t *state) { hp = HAlloc(state->process, ERL_SUB_BITS_SIZE); + /* We mark all our produced binaries as volatile as they may be reallocated + * when we shrink the final accumulator. As the underlying binary isn't + * writable, they will be lazily made non-volatile the first time they are + * accessed. + * + * (The base pointer is left NULL to catch errors in the aforementioned + * code) */ sb = (ErlSubBits*)hp; - sb->thing_word = HEADER_SUB_BITS; - ERTS_SET_SB_RANGE(sb, + erl_sub_bits_init(sb, + ERL_SUB_BITS_FLAG_VOLATILE, + make_boxed((Eterm*)state->acc_ref), + NULL, NBITS(state->acc_offset), NBITS(state->acc_size)); - sb->orig = make_bitstring(state->acc_ref); - /* TODO: once we've moved base pointers from BinRefs to sub-binaries, link - * ourselves to a sub-binary list so that we can set our pointer after - * reallocating the accumulator to its actual size. - * - * The annoying part is that this is only necessary for the _very last_ - * accumulator, as that's the only one to be shrunk. I wonder if there's - * anything clever we can do with that knowledge. */ state->acc_offset += state->acc_size; state->acc_size = 0; @@ -868,6 +878,7 @@ static Uint iol2v_expand_acc(iol2v_state_t *state, UWord extra) { } iol2v_enqueue_result(state, iol2v_extract_acc_term(state)); + iol2v_finish_accumulator(state); } refc_binary = erts_bin_nrml_alloc(MAX(extra, IOL2V_ACC_SIZE)); @@ -875,7 +886,6 @@ static Uint iol2v_expand_acc(iol2v_state_t *state, UWord extra) { br = (BinRef*)HAlloc(state->process, ERL_BIN_REF_SIZE); br->thing_word = HEADER_BIN_REF; br->val = refc_binary; - br->bytes = (byte*)refc_binary->orig_bytes; (*state->off_heap_tail) = (struct erl_off_heap_header*)br; state->off_heap_tail = &br->next; @@ -1027,6 +1037,10 @@ static BIF_RETTYPE iol2v_yield(iol2v_state_t *state) { boxed_state->magic_reference = erts_mk_magic_ref(&hp, &MSO(boxed_state->process), magic_binary); + if (state->result_tail == &state->result_head) { + boxed_state->result_tail = &boxed_state->result_head; + } + if (state->off_heap_tail == &state->off_heap.first) { boxed_state->off_heap_tail = &boxed_state->off_heap.first; } @@ -1125,6 +1139,8 @@ static BIF_RETTYPE iol2v_continue(iol2v_state_t *state) { } if (state->acc) { + iol2v_finish_accumulator(state); + /* Link the state's off-heap list into the process. */ ASSERT(state->off_heap.first != NULL && state->acc_ref != NULL); *(state->off_heap_tail) = MSO(state->process).first; diff --git a/erts/emulator/beam/erl_map.c b/erts/emulator/beam/erl_map.c index b5a2d333ce7d..61fd28478c57 100644 --- a/erts/emulator/beam/erl_map.c +++ b/erts/emulator/beam/erl_map.c @@ -3361,7 +3361,6 @@ BIF_RETTYPE erts_internal_term_type_1(BIF_ALIST_1) { if (is_local_fun(funp)) { BIF_RET(ERTS_MAKE_AM("fun")); } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); BIF_RET(ERTS_MAKE_AM("export")); } } @@ -3383,8 +3382,6 @@ BIF_RETTYPE erts_internal_term_type_1(BIF_ALIST_1) { BIF_RET(ERTS_MAKE_AM("heap_binary")); case SUB_BITS_SUBTAG: BIF_RET(ERTS_MAKE_AM("sub_binary")); - case BIN_MATCHSTATE_SUBTAG: - BIF_RET(ERTS_MAKE_AM("matchstate")); case POS_BIG_SUBTAG: case NEG_BIG_SUBTAG: BIF_RET(ERTS_MAKE_AM("bignum")); diff --git a/erts/emulator/beam/erl_message.c b/erts/emulator/beam/erl_message.c index a1c30ae4a758..6319cf7ee136 100644 --- a/erts/emulator/beam/erl_message.c +++ b/erts/emulator/beam/erl_message.c @@ -170,12 +170,9 @@ erts_cleanup_offheap_list(struct erl_off_heap_header* first) case BIN_REF_SUBTAG: erts_bin_release(u.br->val); break; - case FUN_SUBTAG: - /* We _KNOW_ that this is a local fun, otherwise it would not - * be part of the off-heap list. */ - ASSERT(is_local_fun(u.fun)); - if (erts_refc_dectest(&u.fun->entry.fun->refc, 0) == 0) { - erts_erase_fun_entry(u.fun->entry.fun); + case FUN_REF_SUBTAG: + if (erts_refc_dectest(&(u.fref->entry)->refc, 0) == 0) { + erts_erase_fun_entry(u.fref->entry); } break; case REF_SUBTAG: diff --git a/erts/emulator/beam/erl_msacc.h b/erts/emulator/beam/erl_msacc.h index 1907f600f51a..56f74c98d4e6 100644 --- a/erts/emulator/beam/erl_msacc.h +++ b/erts/emulator/beam/erl_msacc.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2014-2022. All Rights Reserved. + * Copyright Ericsson AB 2014-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -150,7 +150,7 @@ struct erl_msacc_t_ { Eterm id; char *type; - /* the the values below are protected by mtx iff unmanaged = 1 */ + /* the values below are protected by mtx iff unmanaged = 1 */ ErtsSysPerfCounter perf_counter; Uint state; ErtsMsAccPerfCntr counters[]; diff --git a/erts/emulator/beam/erl_nif.c b/erts/emulator/beam/erl_nif.c index 92599169efd2..00892c2f3bbd 100644 --- a/erts/emulator/beam/erl_nif.c +++ b/erts/emulator/beam/erl_nif.c @@ -1488,7 +1488,7 @@ size_t enif_binary_to_term(ErlNifEnv *dst_env, const unsigned char* data, size_t data_sz, ERL_NIF_TERM *term, - ErlNifBinaryToTerm opts) + unsigned int opts) { Sint size; ErtsHeapFactory factory; @@ -4160,10 +4160,7 @@ static int fill_iovec_with_slice(ErlNifEnv *env, br->next = MSO(env->proc).first; MSO(env->proc).first = (struct erl_off_heap_header*)br; br->val = bin; - br->bytes = (byte*) bin->orig_bytes; ERTS_BR_OVERHEAD(&MSO(env->proc), br); - - return make_bitstring(br); } } diff --git a/erts/emulator/beam/erl_nif_api_funcs.h b/erts/emulator/beam/erl_nif_api_funcs.h index 998bfebd8baa..7d5d037a317f 100644 --- a/erts/emulator/beam/erl_nif_api_funcs.h +++ b/erts/emulator/beam/erl_nif_api_funcs.h @@ -649,13 +649,13 @@ static ERL_NIF_INLINE ERL_NIF_TERM enif_make_list9(ErlNifEnv* env, # define enif_make_pid(ENV, PID) ((void)(ENV),(const ERL_NIF_TERM)((PID)->pid)) # define enif_compare_pids(A, B) (enif_compare((A)->pid,(B)->pid)) # define enif_select_read(ENV, E, OBJ, PID, MSG, MSG_ENV) \ - enif_select_x(ENV, E, ERL_NIF_SELECT_READ | ERL_NIF_SELECT_CUSTOM_MSG, \ + enif_select_x(ENV, E, (enum ErlNifSelectFlags)(ERL_NIF_SELECT_READ | ERL_NIF_SELECT_CUSTOM_MSG), \ OBJ, PID, MSG, MSG_ENV) # define enif_select_write(ENV, E, OBJ, PID, MSG, MSG_ENV) \ - enif_select_x(ENV, E, ERL_NIF_SELECT_WRITE | ERL_NIF_SELECT_CUSTOM_MSG, \ + enif_select_x(ENV, E, (enum ErlNifSelectFlags)(ERL_NIF_SELECT_WRITE | ERL_NIF_SELECT_CUSTOM_MSG), \ OBJ, PID, MSG, MSG_ENV) # define enif_select_error(ENV, E, OBJ, PID, MSG, MSG_ENV) \ - enif_select_x(ENV, E, ERL_NIF_SELECT_ERROR | ERL_NIF_SELECT_CUSTOM_MSG, \ + enif_select_x(ENV, E, (enum ErlNifSelectFlags)(ERL_NIF_SELECT_ERROR | ERL_NIF_SELECT_CUSTOM_MSG), \ OBJ, PID, MSG, MSG_ENV) #if SIZEOF_LONG == 8 diff --git a/erts/emulator/beam/erl_node_tables.c b/erts/emulator/beam/erl_node_tables.c index 514fc00fb0f4..5f6ffa58be41 100644 --- a/erts/emulator/beam/erl_node_tables.c +++ b/erts/emulator/beam/erl_node_tables.c @@ -1570,7 +1570,7 @@ insert_offheap(ErlOffHeap *oh, int type, Eterm id) } break; case BIN_REF_SUBTAG: - case FUN_SUBTAG: + case FUN_REF_SUBTAG: break; /* No need to */ default: ASSERT(is_external_header(u.hdr->thing_word)); diff --git a/erts/emulator/beam/erl_node_tables.h b/erts/emulator/beam/erl_node_tables.h index 56696586c606..bde12fcfacb7 100644 --- a/erts/emulator/beam/erl_node_tables.h +++ b/erts/emulator/beam/erl_node_tables.h @@ -196,7 +196,7 @@ Vs = [begin [Val, _, _, _, What] = All = string:lexemes(Ln, " "),{Val,What,All} Accs = lists:foldl(fun({V,<<"ERL_NODE_INC">>,_},M) -> Val = maps:get(V,M,0), M#{ V => Val + 1 }; ({V,<<"ERL_NODE_DEC">>,_},M) -> Val = maps:get(V,M,0), M#{ V => Val - 1 } end, #{}, Vs). lists:usort(lists:filter(fun({V,N}) -> N /= 0 end, maps:to_list(Accs))). - * There are bound to be bugs in the the instrumentation code, but + * There are bound to be bugs in the instrumentation code, but * at least this is a place to start when hunting refc bugs. * */ diff --git a/erts/emulator/beam/erl_posix_str.c b/erts/emulator/beam/erl_posix_str.c index c57c269095a9..ad596bc726aa 100644 --- a/erts/emulator/beam/erl_posix_str.c +++ b/erts/emulator/beam/erl_posix_str.c @@ -12,6 +12,50 @@ * of this file, and for a DISCLAIMER OF ALL WARRANTIES. * * SCCS: @(#) tclPosixStr.c 1.32 96/10/10 10:09:42 + +No "license.terms" file is included; instead, the content of it +follows here: + +This software is copyrighted by the Regents of the University of +California, Sun Microsystems, Inc., and other parties. The following +terms apply to all files associated with the software unless explicitly +disclaimed in individual files. + +The authors hereby grant permission to use, copy, modify, distribute, +and license this software and its documentation for any purpose, provided +that existing copyright notices are retained in all copies and that this +notice is included verbatim in any distributions. No written agreement, +license, or royalty fee is required for any of the authorized uses. +Modifications to this software may be copyrighted by their authors +and need not follow the licensing terms described here, provided that +the new terms are clearly indicated on the first page of each file where +they apply. + +IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY +FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR ANY +DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES, +INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. THIS SOFTWARE +IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE +NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR +MODIFICATIONS. + +GOVERNMENT USE: If you are acquiring this software on behalf of the +U.S. government, the Government shall have only "Restricted Rights" +in the software and related documentation as defined in the Federal +Acquisition Regulations (FARs) in Clause 52.227.19 (c) (2). If you +are acquiring the software on behalf of the Department of Defense, the +software shall be classified as "Commercial Computer Software" and the +Government shall have only "Restricted Rights" as defined in Clause +252.227-7013 (c) (1) of DFARs. Notwithstanding the foregoing, the +authors grant the U.S. Government and others acting in its behalf +permission to use and distribute the software in accordance with the +terms specified in this license. + */ /* %ExternalCopyright% */ diff --git a/erts/emulator/beam/erl_printf_term.c b/erts/emulator/beam/erl_printf_term.c index 04778536ff83..b4b1c0bceb5c 100644 --- a/erts/emulator/beam/erl_printf_term.c +++ b/erts/emulator/beam/erl_printf_term.c @@ -670,8 +670,6 @@ print_term(fmtfn_t fn, void* arg, Eterm obj, long *dcount) { long tdcount; int tres; - ASSERT(is_external_fun(funp) && funp->next == NULL); - PRINT_STRING(res, fn, arg, "fun "); /* We pass a temporary 'dcount' and adjust the real one @@ -772,12 +770,12 @@ print_term(fmtfn_t fn, void* arg, Eterm obj, long *dcount) { } break; } - case MATCHSTATE_DEF: - PRINT_STRING(res, fn, arg, "#MatchState"); - break; - case BIN_REF_DEF: - PRINT_STRING(res, fn, arg, "#BinRef"); - break; + case BIN_REF_DEF: + PRINT_STRING(res, fn, arg, "#BinRef"); + break; + case FUN_REF_DEF: + PRINT_STRING(res, fn, arg, "#FunRef"); + break; default: PRINT_STRING(res, fn, arg, "flags : (Uint16) 0; if (!(flags & ERTS_ML_STATE_ALIAS_MASK) @@ -5548,17 +5552,13 @@ handle_alias_message(Process *c_p, ErtsSigRecvTracing *tracing, * drop message... */ remove_nm_sig(c_p, sig, next_nm_sig); - /* restored as message... */ - ERL_MESSAGE_TERM(sig) = msg; - if (type == ERTS_SIG_Q_TYPE_DIST) - sig->data.heap_frag = &sig->hfrag; - else - sig->data.attached = data_attached; sig->next = NULL;; erts_cleanup_messages(sig); return 2; } + ERL_MESSAGE_FROM(sig) = from; + if ((flags & ERTS_ML_STATE_ALIAS_MASK) == ERTS_ML_STATE_ALIAS_ONCE) { mon->flags &= ~ERTS_ML_STATE_ALIAS_MASK; @@ -7366,12 +7366,11 @@ area_literal_size(Eterm* start, Eterm* end, char* lit_start, Uint lit_size) break; case TAG_PRIMARY_HEADER: if (!header_is_transparent(val)) { - Eterm* new_p; - if (header_is_bin_matchstate(val)) { - ErlBinMatchState *ms = (ErlBinMatchState*) p; - ErlBinMatchBuffer *mb = &(ms->mb); - if (ErtsInArea(mb->orig, lit_start, lit_size)) { - sz += size_object(mb->orig); + Eterm *new_p; + if (val == HEADER_SUB_BITS) { + ErlSubBits *sb = (ErlSubBits*) p; + if (ErtsInArea(sb->orig, lit_start, lit_size)) { + sz += size_object(sb->orig); } } new_p = p + thing_arityval(val); @@ -7405,16 +7404,20 @@ area_literal_copy(Eterm **hpp, ErlOffHeap *ohp, case TAG_PRIMARY_HEADER: if (!header_is_transparent(val)) { Eterm* new_p; - /* matchstate in message, not possible. */ - if (header_is_bin_matchstate(val)) { - ErlBinMatchState *ms = (ErlBinMatchState*) p; - ErlBinMatchBuffer *mb = &(ms->mb); - if (ErtsInArea(mb->orig, lit_start, lit_size)) { - sz = size_object(mb->orig); - mb->orig = copy_struct(mb->orig, sz, hpp, ohp); - } + + if (val == HEADER_SUB_BITS) { + /* Match contexts and writable binaries should never be + * present in signals. */ + ASSERT(erl_sub_bits_is_normal((ErlSubBits*)p)); + + /* Make sure to copy the `orig` field if needed. It's the + * last field inside the thing structure so we can handle + * it by pretending it's not part of the thing. */ + new_p = p + thing_arityval(val) - 1; + } else { + new_p = p + thing_arityval(val); } - new_p = p + thing_arityval(val); + ASSERT(start <= new_p && new_p < end); p = new_p; } diff --git a/erts/emulator/beam/erl_proc_sig_queue.h b/erts/emulator/beam/erl_proc_sig_queue.h index 1caab5bb017d..82c49a028ae8 100644 --- a/erts/emulator/beam/erl_proc_sig_queue.h +++ b/erts/emulator/beam/erl_proc_sig_queue.h @@ -216,30 +216,42 @@ typedef struct { ERTS_SIG_Q_TYPE_MAX #define ERTS_SIG_IS_DIST_ALIAS_MSG_TAG(Tag) \ - ((Tag) == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ - ERTS_SIG_Q_TYPE_DIST, \ - 0)) + (((Tag) & (ERTS_PROC_SIG_TYPE_MASK \ + | ERTS_PROC_SIG_OP_MASK \ + | ERTS_PROC_SIG_BASE_TAG_MASK)) \ + == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ + ERTS_SIG_Q_TYPE_DIST, \ + 0)) #define ERTS_SIG_IS_DIST_ALIAS_MSG(sig) \ ERTS_SIG_IS_DIST_ALIAS_MSG_TAG(((ErtsSignal *) (sig))->common.tag) #define ERTS_SIG_IS_OFF_HEAP_ALIAS_MSG_TAG(Tag) \ - ((Tag) == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ - ERTS_SIG_Q_TYPE_OFF_HEAP, \ - 0)) + (((Tag) & (ERTS_PROC_SIG_TYPE_MASK \ + | ERTS_PROC_SIG_OP_MASK \ + | ERTS_PROC_SIG_BASE_TAG_MASK)) \ + == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ + ERTS_SIG_Q_TYPE_OFF_HEAP, \ + 0)) #define ERTS_SIG_IS_OFF_HEAP_ALIAS_MSG(sig) \ ERTS_SIG_IS_OFF_HEAP_ALIAS_MSG_TAG(((ErtsSignal *) (sig))->common.tag) #define ERTS_SIG_IS_HEAP_ALIAS_MSG_TAG(Tag) \ - ((Tag) == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ - ERTS_SIG_Q_TYPE_HEAP, \ - 0)) + (((Tag) & (ERTS_PROC_SIG_TYPE_MASK \ + | ERTS_PROC_SIG_OP_MASK \ + | ERTS_PROC_SIG_BASE_TAG_MASK)) \ + == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ + ERTS_SIG_Q_TYPE_HEAP, \ + 0)) #define ERTS_SIG_IS_HEAP_ALIAS_MSG(sig) \ ERTS_SIG_IS_HEAP_ALIAS_MSG_TAG(((ErtsSignal *) (sig))->common.tag) #define ERTS_SIG_IS_HEAP_FRAG_ALIAS_MSG_TAG(Tag) \ - ((Tag) == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ - ERTS_SIG_Q_TYPE_HEAP_FRAG, \ - 0)) + (((Tag) & (ERTS_PROC_SIG_TYPE_MASK \ + | ERTS_PROC_SIG_OP_MASK \ + | ERTS_PROC_SIG_BASE_TAG_MASK)) \ + == ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_ALIAS_MSG, \ + ERTS_SIG_Q_TYPE_HEAP_FRAG, \ + 0)) #define ERTS_SIG_IS_HEAP_FRAG_ALIAS_MSG(sig) \ ERTS_SIG_IS_HEAP_FRAG_ALIAS_MSG_TAG(((ErtsSignal *) (sig))->common.tag) @@ -358,6 +370,15 @@ int erts_proc_sig_queue_force_buffers(Process*); << ERTS_SIG_Q_XTRA_SHIFT), \ _TAG_HEADER_EXTERNAL_PID)) +#define ERTS_PROC_SIG_BASE_TAG_MASK \ + ((1 << _HEADER_ARITY_OFFS)-1) +#define ERTS_PROC_SIG_OP_MASK \ + (ERTS_SIG_Q_OP_MASK << (ERTS_SIG_Q_OP_SHIFT + _HEADER_ARITY_OFFS)) +#define ERTS_PROC_SIG_TYPE_MASK \ + (ERTS_SIG_Q_TYPE_MASK << (ERTS_SIG_Q_TYPE_SHIFT + _HEADER_ARITY_OFFS)) +#define ERTS_PROC_SIG_XTRA_MASK \ + (ERTS_SIG_Q_XTRA_MASK << (ERTS_SIG_Q_XTRA_SHIFT + _HEADER_ARITY_OFFS)) + struct dist_entry_; #define ERTS_PROC_HAS_INCOMING_SIGNALS(P) \ @@ -984,7 +1005,7 @@ erts_proc_sig_send_sync_suspend(Process *c_p, Eterm to, * term returned by 'func'. If the return value of * 'func' is not an immediate term, 'func' has to * allocate a heap fragment where the result is stored - * and update the the heap fragment pointer pointer + * and update the heap fragment pointer pointer * passed as third argument to point to it. * * If this function returns a reference, 'func' will @@ -1049,7 +1070,7 @@ erts_proc_sig_send_rpc_request(Process *c_p, * term returned by 'func'. If the return value of * 'func' is not an immediate term, 'func' has to * allocate a heap fragment where the result is stored - * and update the the heap fragment pointer pointer + * and update the heap fragment pointer pointer * passed as third argument to point to it. * * If this function returns a reference, 'func' will diff --git a/erts/emulator/beam/erl_process.h b/erts/emulator/beam/erl_process.h index 44bc0b306935..d005ea0792ba 100644 --- a/erts/emulator/beam/erl_process.h +++ b/erts/emulator/beam/erl_process.h @@ -1684,7 +1684,7 @@ extern int erts_system_profile_ts_type; # define F_INITIAL_TRACE_FLAGS 0 #endif -/* F_TIMESTAMP_MASK is a bit-field of all all timestamp types */ +/* F_TIMESTAMP_MASK is a bit-field of all timestamp types */ #define F_TIMESTAMP_MASK \ (ERTS_TRACE_TS_TYPE_MASK << ERTS_TRACE_FLAGS_TS_TYPE_SHIFT) diff --git a/erts/emulator/beam/erl_process_dump.c b/erts/emulator/beam/erl_process_dump.c index be9b38d02e37..ea6e6522e96c 100644 --- a/erts/emulator/beam/erl_process_dump.c +++ b/erts/emulator/beam/erl_process_dump.c @@ -629,7 +629,11 @@ heap_dump(fmtfn_t to, void *to_arg, Eterm x) *ptr = OUR_NIL; } else if (hdr == HEADER_BIN_REF) { dump_bin_ref(to, to_arg, (BinRef*)ptr); - *ptr = OUR_NIL; + *ptr = OUR_NIL; + } else if (hdr == HEADER_FUN_REF) { + const FunRef *fref = (FunRef*)ptr; + erts_print(to, to_arg, "Rf" PTR_FMT "\n", fref->entry); + *ptr = OUR_NIL; } else if (is_external_pid_header(hdr)) { erts_print(to, to_arg, "P%T\n", x); *ptr = OUR_NIL; @@ -758,10 +762,10 @@ dump_externally(fmtfn_t to, void *to_arg, Eterm term) * The crashdump_viewer does not allow inspection of them anyway. */ ErlFunThing* funp = (ErlFunThing *) fun_val(term); - Uint num_free = fun_num_free(funp); + Uint env_size = fun_env_size(funp); Uint i; - for (i = 0; i < num_free; i++) { + for (i = 0; i < env_size; i++) { funp->env[i] = NIL; } } @@ -982,6 +986,9 @@ dump_module_literals(fmtfn_t to, void *to_arg, ErtsLiteralArea* lit_area) } } else if (w == HEADER_BIN_REF) { dump_bin_ref(to, to_arg, (BinRef*)htop); + } else if (w == HEADER_FUN_REF) { + const FunRef *fref = (FunRef*)htop; + erts_print(to, to_arg, "Rf" PTR_FMT "\n", fref->entry); } else if (is_map_header(w)) { if (is_flatmap_header(w)) { flatmap_t* fmp = (flatmap_t *) flatmap_val(term); @@ -1039,7 +1046,10 @@ dump_module_literals(fmtfn_t to, void *to_arg, ErtsLiteralArea* lit_area) size = 1 + header_arity(w); switch (w & _HEADER_SUBTAG_MASK) { case FUN_SUBTAG: - ASSERT(fun_num_free((ErlFunThing*)(htop)) == 0); + { + const ErlFunThing *funp = (ErlFunThing*)htop; + size += fun_env_size(funp); + } break; case MAP_SUBTAG: if (is_flatmap_header(w)) { @@ -1048,9 +1058,6 @@ dump_module_literals(fmtfn_t to, void *to_arg, ErtsLiteralArea* lit_area) size += hashmap_bitcount(MAP_HEADER_VAL(w)); } break; - case SUB_BITS_SUBTAG: - size += 1; - break; } break; default: diff --git a/erts/emulator/beam/erl_term.c b/erts/emulator/beam/erl_term.c index 4f7a43f9712e..4bb31e3230b9 100644 --- a/erts/emulator/beam/erl_term.c +++ b/erts/emulator/beam/erl_term.c @@ -115,10 +115,10 @@ erts_term_init(void) /* Check that the tag masks cannot confuse tags outside of their * category. */ const Eterm tags[] = {ARITYVAL_SUBTAG, - BIN_MATCHSTATE_SUBTAG, POS_BIG_SUBTAG, NEG_BIG_SUBTAG, REF_SUBTAG, + FUN_REF_SUBTAG, FUN_SUBTAG, FLOAT_SUBTAG, HEAP_BITS_SUBTAG, diff --git a/erts/emulator/beam/erl_term.h b/erts/emulator/beam/erl_term.h index a48a2cc7bdb5..0e5416e658ce 100644 --- a/erts/emulator/beam/erl_term.h +++ b/erts/emulator/beam/erl_term.h @@ -99,7 +99,7 @@ struct erl_node_; /* Declared in erl_node_tables.h */ * HEADER tags: * * 0000 ARITYVAL - * 0001 BINARY_AGGREGATE | + * 0001 FUN_REF | * 001x BIGNUM with sign bit | * 0100 REF | * 0101 FUN | THINGS @@ -118,13 +118,13 @@ struct erl_node_; /* Declared in erl_node_tables.h */ * * - The tag is zero for arityval and non-zero for thing headers. * - A single bit differentiates between positive and negative bignums. - * - If more tags are needed, the REF and and EXTERNAL_REF tags could probably + * - If more tags are needed, the REF and EXTERNAL_REF tags could probably * be combined to one tag. * * XXX: globally replace XXX_SUBTAG with TAG_HEADER_XXX */ #define ARITYVAL_SUBTAG (0x0 << _TAG_PRIMARY_SIZE) /* TUPLE */ -#define BIN_MATCHSTATE_SUBTAG (0x1 << _TAG_PRIMARY_SIZE) +#define FUN_REF_SUBTAG (0x1 << _TAG_PRIMARY_SIZE) /* FUN_REF */ #define _BIG_TAG_MASK (~(0x1 << _TAG_PRIMARY_SIZE) & _TAG_HEADER_MASK) #define _BIG_SIGN_BIT (0x1 << _TAG_PRIMARY_SIZE) #define POS_BIG_SUBTAG (0x2 << _TAG_PRIMARY_SIZE) /* BIGNUM */ @@ -144,7 +144,8 @@ struct erl_node_; /* Declared in erl_node_tables.h */ /* _EXTERNAL_TAG_MASK requires that 0xF is reserved for external terms. */ #define _TAG_HEADER_ARITYVAL (TAG_PRIMARY_HEADER|ARITYVAL_SUBTAG) -#define _TAG_HEADER_FUN (TAG_PRIMARY_HEADER|FUN_SUBTAG) +#define _TAG_HEADER_FUN_REF (TAG_PRIMARY_HEADER|FUN_REF_SUBTAG) +#define _TAG_HEADER_FUN (TAG_PRIMARY_HEADER|FUN_SUBTAG) #define _TAG_HEADER_POS_BIG (TAG_PRIMARY_HEADER|POS_BIG_SUBTAG) #define _TAG_HEADER_NEG_BIG (TAG_PRIMARY_HEADER|NEG_BIG_SUBTAG) #define _TAG_HEADER_FLOAT (TAG_PRIMARY_HEADER|FLOAT_SUBTAG) @@ -155,7 +156,6 @@ struct erl_node_; /* Declared in erl_node_tables.h */ #define _TAG_HEADER_EXTERNAL_PID (TAG_PRIMARY_HEADER|EXTERNAL_PID_SUBTAG) #define _TAG_HEADER_EXTERNAL_PORT (TAG_PRIMARY_HEADER|EXTERNAL_PORT_SUBTAG) #define _TAG_HEADER_EXTERNAL_REF (TAG_PRIMARY_HEADER|EXTERNAL_REF_SUBTAG) -#define _TAG_HEADER_BIN_MATCHSTATE (TAG_PRIMARY_HEADER|BIN_MATCHSTATE_SUBTAG) #define _TAG_HEADER_MAP (TAG_PRIMARY_HEADER|MAP_SUBTAG) @@ -167,7 +167,6 @@ struct erl_node_; /* Declared in erl_node_tables.h */ (((x) & (_HEADER_SUBTAG_MASK)) == ARITYVAL_SUBTAG) #define header_is_arityval(x) (((x) & _HEADER_SUBTAG_MASK) == ARITYVAL_SUBTAG) #define header_is_thing(x) (!header_is_transparent((x))) -#define header_is_bin_matchstate(x) ((((x) & (_HEADER_SUBTAG_MASK)) == BIN_MATCHSTATE_SUBTAG)) #define _CPMASK 0x3 @@ -379,6 +378,8 @@ _ET_DECLARE_CHECKED(Uint,thing_subtag,Eterm) #define is_bitstring_header(x) \ (((x) & (_BITSTRING_TAG_MASK)) == _TAG_HEADER_HEAP_BITS) +#define is_bin_ref(x) \ + (is_boxed((x)) && *boxed_val(x) == HEADER_BIN_REF) #define make_bitstring(x) make_boxed((Eterm*)(x)) #define is_bitstring(x) (is_boxed((x)) && is_bitstring_header(*boxed_val((x)))) #define is_not_bitstring(x) (!is_bitstring((x))) @@ -393,27 +394,31 @@ _ET_DECLARE_CHECKED(Eterm*,bitstring_val,Wterm) * * aaaaaaaaaaaaaaaa aaaaaaaaaatttt00 arity:26, tag:4 * - * Since the arity and number of free variables are both limited to 255, and we - * only need one bit to signify whether the fun is local or external, we can - * fit all of that information in the header word. + * Since the arity and number of free variables are both limited to 255, we can + * fit them both into the header word. * - * 0000000effffffff aaaaaaaa00010100 external:1, free:8, arity:8 + * 0000000eeeeeeeee aaaaaaaa00010100 environment_size:9, arity:8 * * Note that the lowest byte contains only the function subtag, and the next * byte after that contains only the arity. This lets us combine the type * and/or arity check into a single comparison without masking, by using 8- or - * 16-bit operations on the header word. */ + * 16-bit operations on the header word. + * + * Furthermore, as local funs always have at least one environment variable as + * the `FunRef` is treated as part of the environment, we can distinguish + * them by checking whether the environment size is zero. */ #define FUN_HEADER_ARITY_OFFS (_HEADER_ARITY_OFFS + 2) -#define FUN_HEADER_NUM_FREE_OFFS (FUN_HEADER_ARITY_OFFS + 8) -#define FUN_HEADER_EXTERNAL_OFFS (FUN_HEADER_NUM_FREE_OFFS + 8) +#define FUN_HEADER_ENV_SIZE_OFFS (FUN_HEADER_ARITY_OFFS + 8) #define MAKE_FUN_HEADER(Arity, NumFree, External) \ - (_TAG_HEADER_FUN | \ + (ASSERT((!(External)) || ((NumFree) == 0)), \ + (_TAG_HEADER_FUN | \ (((Arity)) << FUN_HEADER_ARITY_OFFS) | \ - (((NumFree)) << FUN_HEADER_NUM_FREE_OFFS) | \ - ((!!(External)) << FUN_HEADER_EXTERNAL_OFFS)) + (((NumFree + !External)) << FUN_HEADER_ENV_SIZE_OFFS))) +#define is_fun_ref(x) \ + (is_boxed((x)) && *boxed_val(x) == HEADER_FUN_REF) #define is_fun_header(x) (((x) & _HEADER_SUBTAG_MASK) == FUN_SUBTAG) #define make_fun(x) make_boxed((Eterm*)(x)) #define is_any_fun(x) (is_boxed((x)) && is_fun_header(*boxed_val((x)))) @@ -1421,8 +1426,8 @@ _ET_DECLARE_CHECKED(Uint,loader_y_reg_index,Uint) #define FLOAT_DEF 0xe #define BIG_DEF 0xf #define SMALL_DEF 0x10 -#define MATCHSTATE_DEF 0x11 /* not a "real" term */ -#define BIN_REF_DEF 0x12 /* not a "real" term */ +#define BIN_REF_DEF 0x11 /* not a "real" term */ +#define FUN_REF_DEF 0x12 /* not a "real" term */ #define FIRST_VACANT_TAG_DEF 0x13 @@ -1490,9 +1495,9 @@ ERTS_GLB_INLINE unsigned tag_val_def(Wterm x) case (_TAG_HEADER_EXTERNAL_REF >> _TAG_PRIMARY_SIZE): return EXTERNAL_REF_DEF; case (_TAG_HEADER_MAP >> _TAG_PRIMARY_SIZE): return MAP_DEF; case (_TAG_HEADER_BIN_REF >> _TAG_PRIMARY_SIZE): return BIN_REF_DEF; + case (_TAG_HEADER_FUN_REF >> _TAG_PRIMARY_SIZE): return FUN_REF_DEF; case (_TAG_HEADER_HEAP_BITS >> _TAG_PRIMARY_SIZE): return BITSTRING_DEF; case (_TAG_HEADER_SUB_BITS >> _TAG_PRIMARY_SIZE): return BITSTRING_DEF; - case (_TAG_HEADER_BIN_MATCHSTATE >> _TAG_PRIMARY_SIZE): return MATCHSTATE_DEF; } break; diff --git a/erts/emulator/beam/erl_term_hashing.c b/erts/emulator/beam/erl_term_hashing.c index 072e109fbe6b..796dd9574b0c 100644 --- a/erts/emulator/beam/erl_term_hashing.c +++ b/erts/emulator/beam/erl_term_hashing.c @@ -257,8 +257,6 @@ Uint32 make_hash(Eterm term_arg) } else { const ErtsCodeMFA *mfa = &funp->entry.exp->info.mfa; - ASSERT(is_external_fun(funp) && funp->next == NULL); - hash = hash * FUNNY_NUMBER11 + mfa->arity; hash = hash*FUNNY_NUMBER1 + (atom_tab(atom_val(mfa->module))->slot.bucket.hvalue); @@ -1189,8 +1187,6 @@ make_hash2_helper(Eterm term_param, const int can_trap, Eterm* state_mref_write_ } else { Export *ep = funp->entry.exp; - ASSERT(is_external_fun(funp) && funp->next == NULL); - UINT32_HASH_2 (ep->info.mfa.arity, atom_tab(atom_val(ep->info.mfa.module))->slot.bucket.hvalue, @@ -1924,8 +1920,6 @@ make_internal_hash(Eterm term, erts_ihash_t salt) goto pop_next; } else { - ASSERT(is_external_fun(funp) && funp->next == NULL); - /* Assumes Export entries never move */ IHASH_MIX_ALPHA(IHASH_TYPE_EXTERNAL_FUN); IHASH_MIX_BETA((UWord)funp->entry.exp); @@ -1944,8 +1938,7 @@ make_internal_hash(Eterm term, erts_ihash_t salt) ERTS_GET_BITSTRING(term, data, offset, size); IHASH_MIX_ALPHA(IHASH_TYPE_BINARY); - IHASH_MIX_BETA(offset); - IHASH_MIX_ALPHA(size); + IHASH_MIX_BETA(size); if (size > 0) { const byte *bytes = data; diff --git a/erts/emulator/beam/erl_trace.c b/erts/emulator/beam/erl_trace.c index cca8c6a57091..6cc863ca0697 100644 --- a/erts/emulator/beam/erl_trace.c +++ b/erts/emulator/beam/erl_trace.c @@ -1093,16 +1093,15 @@ Uint32 erts_call_trace(Process* p, ErtsCodeInfo *info, Binary *match_spec, Eterm* args, int local, ErtsTracer *tracer) { + const int arity = info->mfa.arity; Eterm* hp; Eterm mfa_tuple; - int arity; int i; Uint32 return_flags; Eterm pam_result = am_true; Uint32 meta_flags; Uint32 *tracee_flags; ErtsTracerNif *tnif = NULL; - Eterm transformed_args[MAX_ARG]; ErtsTracer pre_ms_tracer = erts_tracer_nil; ERTS_LC_ASSERT(erts_proc_lc_my_proc_locks(p) & ERTS_PROC_LOCK_MAIN); @@ -1163,51 +1162,6 @@ erts_call_trace(Process* p, ErtsCodeInfo *info, Binary *match_spec, } } - /* - * Because of the delayed sub-binary creation optimization introduced in - * R12B, (at most) one of arguments can be a match context instead of - * a binary. Since we don't want to handle match contexts in utility functions - * such as size_object() and copy_struct(), we must make sure that we - * temporarily convert any match contexts to sub binaries. - */ - arity = info->mfa.arity; - for (i = 0; i < arity; i++) { - Eterm arg = args[i]; - - if (is_boxed(arg) && header_is_bin_matchstate(*boxed_val(arg))) { - ErlBinMatchState* ms = (ErlBinMatchState *)boxed_val(arg); - ErlBinMatchBuffer* mb = &ms->mb; - - if (is_bitstring(mb->orig)) { - arg = erts_make_sub_bitstring(p, - mb->orig, - mb->offset, - mb->size - mb->offset); - } else { - Binary *refc_binary; - BinRef *br; - Eterm *hp; - - br = (BinRef*)boxed_val(mb->orig); - refc_binary = br->val; - - erts_refc_inctest(&refc_binary->intern.refc, 1); - - hp = HAlloc(p, ERL_REFC_BITS_SIZE); - arg = erts_wrap_refc_bitstring(&MSO(p).first, - &MSO(p).overhead, - &hp, - refc_binary, - mb->base, - mb->offset, - mb->size - mb->offset); - } - } - - transformed_args[i] = arg; - } - args = transformed_args; - /* * If there is a PAM program, run it. Return if it fails. * diff --git a/erts/emulator/beam/export.c b/erts/emulator/beam/export.c index ef82661e9967..dd1f1ed258c7 100644 --- a/erts/emulator/beam/export.c +++ b/erts/emulator/beam/export.c @@ -189,7 +189,6 @@ static void create_shared_lambda(Export *export) lambda->thing_word = MAKE_FUN_HEADER(export->info.mfa.arity, 0, 1); lambda->entry.exp = export; - lambda->next = NULL; export->lambda = make_fun(lambda); diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c index 4cbe0eecade0..f185735b504e 100644 --- a/erts/emulator/beam/external.c +++ b/erts/emulator/beam/external.c @@ -2642,7 +2642,6 @@ static Eterm erts_term_to_binary_int(Process* p, Sint bif_ix, Eterm Term, Eterm result_ref = (BinRef*)hp; result_ref->thing_word = HEADER_BIN_REF; result_ref->val = result_bin; - result_ref->bytes = (byte*)result_bin->orig_bytes; hp += ERL_BIN_REF_SIZE; referenced_cbin = 0; @@ -2666,16 +2665,17 @@ static Eterm erts_term_to_binary_int(Process* p, Sint bif_ix, Eterm Term, Eterm /* If the term refers to the entire segment, we can * use it as is. Otherwise we need to return a * shrunken copy. */ - if (from_sb->size != NBITS(iovp->iov_len)) { + if (NBITS(iovp->iov_len) != (from_sb->end - + from_sb->start)) { ErlSubBits *to_sb = (ErlSubBits*)hp; segment = make_bitstring(to_sb); hp += ERL_SUB_BITS_SIZE; *to_sb = *from_sb; - to_sb->size = NBITS(iovp->iov_len); + to_sb->end = to_sb->start + NBITS(iovp->iov_len); - ASSERT(to_sb->size < from_sb->size); + ASSERT(to_sb->end < from_sb->end); } } else { /* We don't have a term and need to create one now, @@ -2697,11 +2697,12 @@ static Eterm erts_term_to_binary_int(Process* p, Sint bif_ix, Eterm Term, Eterm ASSERT(IS_BINARY_SIZE_OK(iov_offset)); - sb->thing_word = HEADER_SUB_BITS; - ERTS_SET_SB_RANGE(sb, + erl_sub_bits_init(sb, + 0, + make_boxed((Eterm*)result_ref), + result_bin->orig_bytes, NBITS(iov_offset), NBITS(iovp->iov_len)); - sb->orig = make_bitstring(result_ref); segment = make_bitstring(sb); hp += ERL_SUB_BITS_SIZE; @@ -3976,8 +3977,6 @@ enc_term_int(TTBEncodeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, byte* ep, } else { Export *exp = funp->entry.exp; - ASSERT(is_external_fun(funp) && funp->next == NULL); - *ep++ = EXPORT_EXT; ep = enc_atom(acmp, exp->info.mfa.module, ep, dflags); ep = enc_atom(acmp, exp->info.mfa.function, ep, dflags); @@ -5001,7 +5000,8 @@ dec_term(ErtsDistExternal *edep, break; case NEW_FUN_EXT: { - ErlFunThing* funp = (ErlFunThing *) hp; + ErlFunThing *funp; + FunRef *refp; Uint arity; Eterm module; const byte* uniq; @@ -5020,10 +5020,20 @@ dec_term(ErtsDistExternal *edep, ep += 4; num_free = get_int32(ep); ep += 4; - hp += ERL_FUN_SIZE; - hp += num_free; - funp->thing_word = MAKE_FUN_HEADER(arity, num_free, 0); - *objp = make_fun(funp); + + refp = (FunRef*)&hp[0]; + funp = (ErlFunThing*)&hp[ERL_FUN_REF_SIZE]; + + refp->thing_word = HEADER_FUN_REF; + funp->thing_word = MAKE_FUN_HEADER(arity, num_free, 0); + *objp = make_fun(funp); + + hp += ERL_FUN_REF_SIZE + ERL_FUN_SIZE; + + /* Fun references are stored just past the end of the free + * variables. */ + funp->env[num_free] = make_boxed((Eterm*)refp); + hp += num_free + 1; /* Module */ if ((ep = dec_atom(edep, ep, &module, 0)) == NULL) { @@ -5057,17 +5067,17 @@ dec_term(ErtsDistExternal *edep, goto error; } - /* - * It is safe to link the fun into the fun list only when - * no more validity tests can fail. - */ - funp->next = factory->off_heap->first; - factory->off_heap->first = (struct erl_off_heap_header*)funp; + /* It is safe to link the fun into the fun list only when no + * more validity tests can fail. */ + refp->next = factory->off_heap->first; + factory->off_heap->first = (struct erl_off_heap_header*)refp; funp->entry.fun = erts_put_fun_entry2(module, old_uniq, old_index, uniq, index, arity); - hp = factory->hp; + refp->entry = funp->entry.fun; + + hp = factory->hp; /* Environment */ for (i = num_free-1; i >= 0; i--) { @@ -5116,6 +5126,8 @@ dec_term(ErtsDistExternal *edep, sys_memcpy(sb, ep, sizeof(ErlSubBits) + sizeof(BinRef)); ep += sizeof(ErlSubBits) + sizeof(BinRef); + + sb->orig = make_boxed((Eterm*)br); } else { /* The encoded bitstring can be described entirely from the * wrapped Binary* object, so we've skipped encoding an @@ -5123,9 +5135,12 @@ dec_term(ErtsDistExternal *edep, sys_memcpy(br, ep, sizeof(BinRef)); ep += sizeof(BinRef); - sb->thing_word = HEADER_SUB_BITS; - ERTS_SET_SB_RANGE(sb, 0, NBITS((br->val)->orig_size)); - sb->is_writable = 0; + erl_sub_bits_init(sb, + 0, + make_boxed((Eterm*)br), + &(br->val)->orig_bytes[0], + 0, + NBITS((br->val)->orig_size)); } erts_refc_inc(&(br->val)->intern.refc, 1); @@ -5133,8 +5148,6 @@ dec_term(ErtsDistExternal *edep, br->next = (factory->off_heap)->first; (factory->off_heap)->first = (struct erl_off_heap_header*)br; ERTS_BR_OVERHEAD(factory->off_heap, br); - - sb->orig = make_boxed((Eterm*)br); *objp = make_bitstring(sb); break; } @@ -5689,8 +5702,6 @@ encode_size_struct_int(TTBSizeContext* ctx, ErtsAtomCacheMap *acmp, Eterm obj, } else { Export* ep = funp->entry.exp; - ASSERT(is_external_fun(funp) && funp->next == NULL); - result += 1; result += encode_atom_size(acmp, ep->info.mfa.module, dflags); result += encode_atom_size(acmp, ep->info.mfa.function, dflags); @@ -6140,7 +6151,7 @@ decoded_size(const byte *ep, const byte* endp, int internal_tags, B2TContext* ct goto error; } ADDTERMS(4 + num_free); - heap_size += ERL_FUN_SIZE + num_free; + heap_size += ERL_FUN_REF_SIZE + ERL_FUN_SIZE + num_free + 1; break; } case FUN_EXT: diff --git a/erts/emulator/beam/generators.tab b/erts/emulator/beam/generators.tab index 0a0d1562f04c..bdae340b68c9 100644 --- a/erts/emulator/beam/generators.tab +++ b/erts/emulator/beam/generators.tab @@ -301,38 +301,75 @@ gen.skip_bits2(Fail, Ms, Size, Unit, Flags) { } // Creates an instruction that moves a literal lambda to a register. -MakeLiteralLambda(Op, Index, DstType, DstVal) { - BeamFile_LambdaEntry *entry = &S->beam.lambdas.entries[Idx.val]; +MakeLiteralFun(Op, Index, Arity, Dst) { SWord literal; - ASSERT(entry->num_free == 0); - /* If we haven't already done so, we need to create a placeholder for the * lambda. */ - if (S->lambda_literals[$Index] == ERTS_SWORD_MAX) { - Eterm tmp_hp[ERL_FUN_SIZE]; + literal = S->fun_refs[$Index]; + if (literal == ERTS_SWORD_MAX) { + Eterm tmp_hp[ERL_FUN_SIZE + ERL_FUN_REF_SIZE + 1]; ErlFunThing *funp; + FunRef *refp; + + /* +1 to skip funp->env[0] */ + refp = (FunRef*)&tmp_hp[ERL_FUN_SIZE + 1]; + refp->thing_word = HEADER_FUN_REF; + refp->entry = NULL; + refp->next = NULL; - /* The placeholder is an external fun with the export field set to - * NULL, preventing it from colliding with external fun literals - * created by the user. We also disable deduplication to prevent it - * from colliding with other placeholder lambdas of the same arity. */ funp = (ErlFunThing*)tmp_hp; - funp->thing_word = MAKE_FUN_HEADER(entry->arity, 0, 1); + funp->thing_word = MAKE_FUN_HEADER($Arity, 0, 0); funp->entry.exp = NULL; - funp->next = NULL; + funp->env[0] = make_boxed((Eterm*)refp); - literal = beamfile_add_literal(&S->beam, make_fun(tmp_hp), 0); - S->lambda_literals[$Index] = literal; - } else { - literal = S->lambda_literals[$Index]; + literal = beamfile_add_literal(&S->beam, make_fun((Eterm*)funp), 0); + S->fun_refs[$Index] = literal; } $BeamOpNameArity($Op, move, 2); ($Op)->a[0].type = TAG_q; ($Op)->a[0].val = literal; - ($Op)->a[1].type = $DstType; - ($Op)->a[1].val = $DstVal; + ($Op)->a[1] = $Dst; +} + +MakeFun(Op, Index, Arity, NumFree, Env, Dst) { + SWord literal; + + $BeamOpNameArity(op, i_make_fun3, 4); + $BeamOpArity(op, 5 + $NumFree); + + op->a[0].type = TAG_u; + op->a[0].val = $Index; + op->a[1] = $Dst; + op->a[2].type = TAG_u; + op->a[2].val = $Arity - $NumFree; + op->a[3].type = TAG_u; + op->a[3].val = $NumFree; + + for (int i = 0; i < $NumFree; i++) { + op->a[4 + i] = $Env[i]; + } + + /* Create a FunRef if we haven't already done so (may be shared with other + * make_fun3 instructions). */ + literal = S->fun_refs[$Index]; + if (literal == ERTS_SWORD_MAX) { + Eterm tmp_hp[ERL_FUN_REF_SIZE]; + FunRef *refp = (FunRef*)tmp_hp; + + refp->thing_word = HEADER_FUN_REF; + refp->entry = NULL; + refp->next = NULL; + + literal = beamfile_add_literal(&S->beam, make_boxed((Eterm*)refp), 0); + S->fun_refs[$Index] = literal; + } + + /* Add the FunRef as part of the environment, just past the free + * variables. */ + op->a[4 + $NumFree].type = TAG_q; + op->a[4 + $NumFree].val = literal; } gen.make_fun3(Idx, Dst, NumFree, Env) { @@ -344,25 +381,10 @@ gen.make_fun3(Idx, Dst, NumFree, Env) { BeamFile_LambdaEntry *entry = &S->beam.lambdas.entries[Idx.val]; if (NumFree.val == entry->num_free) { - int i; - - if (NumFree.val == 0 && erts_init_process_id != ERTS_INVALID_PID) { - $MakeLiteralLambda(op, Idx.val, Dst.type, Dst.val); + if (entry->num_free == 0) { + $MakeLiteralFun(op, Idx.val, entry->arity, Dst); } else { - $BeamOpNameArity(op, i_make_fun3, 4); - $BeamOpArity(op, 4 + NumFree.val); - - op->a[0].type = TAG_u; - op->a[0].val = Idx.val; - op->a[1] = Dst; - op->a[2].type = TAG_u; - op->a[2].val = entry->arity - entry->num_free; - op->a[3].type = TAG_u; - op->a[3].val = entry->num_free; - - for (i = 0; i < NumFree.val; i++) { - op->a[i+4] = Env[i]; - } + $MakeFun(op, Idx.val, entry->arity, entry->num_free, Env, Dst); } return op; diff --git a/erts/emulator/beam/global.h b/erts/emulator/beam/global.h index a92ed632bc14..8f9e3b78381b 100644 --- a/erts/emulator/beam/global.h +++ b/erts/emulator/beam/global.h @@ -273,7 +273,7 @@ extern Eterm erts_ddll_monitor_driver(Process *p, union erl_off_heap_ptr { struct erl_off_heap_header* hdr; BinRef *br; - struct erl_fun_thing* fun; + struct erl_fun_ref* fref; struct external_thing_* ext; ErtsMRefThing *mref; Eterm* ep; diff --git a/erts/emulator/beam/jit/arm/beam_asm.hpp b/erts/emulator/beam/jit/arm/beam_asm.hpp index 15c767a0dcda..d2a219cd0359 100644 --- a/erts/emulator/beam/jit/arm/beam_asm.hpp +++ b/erts/emulator/beam/jit/arm/beam_asm.hpp @@ -842,8 +842,9 @@ class BeamModuleAssembler : public BeamAssembler, /* Save the last known unreachable position. */ size_t last_unreachable_offset = 0; - /* Mark this point unreachable. Use at the end of a BEAM - * instruction. */ + /* Mark this point unreachable. This must be placed at the very end when + * used in a BEAM instruction, and should not be used in helper + * functions. */ void mark_unreachable() { last_unreachable_offset = a.offset(); } diff --git a/erts/emulator/beam/jit/arm/beam_asm_global.cpp b/erts/emulator/beam/jit/arm/beam_asm_global.cpp index 2e93cf9ee68d..84cbf0a51568 100644 --- a/erts/emulator/beam/jit/arm/beam_asm_global.cpp +++ b/erts/emulator/beam/jit/arm/beam_asm_global.cpp @@ -247,8 +247,6 @@ void BeamModuleAssembler::emit_raise_exception(const ErtsCodeMFA *exp) { fragment_call(ga->get_raise_exception_null_exp()); } - mark_unreachable(); - /* `line` instructions need to know the latest offset that may throw an * exception. See the `line` instruction for details. */ last_error_offset = a.offset(); diff --git a/erts/emulator/beam/jit/arm/instr_bs.cpp b/erts/emulator/beam/jit/arm/instr_bs.cpp index ef7715f4fa54..3fa07e9ef7f2 100644 --- a/erts/emulator/beam/jit/arm/instr_bs.cpp +++ b/erts/emulator/beam/jit/arm/instr_bs.cpp @@ -545,35 +545,41 @@ void BeamModuleAssembler::emit_i_bs_start_match3(const ArgRegister &Src, * free to skip the binary tests. */ } - arm::Gp boxed_ptr = emit_ptr_val(TMP1, ARG2); - a.ldur(TMP1, emit_boxed_val(boxed_ptr, 0)); + emit_untag_ptr(TMP1, ARG2); - a.and_(TMP1, TMP1, imm(_HEADER_SUBTAG_MASK)); - a.cmp(TMP1, imm(BIN_MATCHSTATE_SUBTAG)); + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, thing_word, base_flags); + a.ldp(TMP2, TMP3, arm::Mem(TMP1)); + + ERTS_CT_ASSERT((HEADER_SUB_BITS & _TAG_PRIMARY_MASK) == 0 && + (ERL_SUB_BITS_FLAG_MASK == _TAG_PRIMARY_MASK)); + a.bfi(TMP2, TMP3, imm(0), imm(2)); + a.cmp(TMP2, imm(HEADER_SUB_BITS | ERL_SUB_BITS_FLAGS_MATCH_CONTEXT)); a.b_eq(next); - if (Fail.get() != 0) { - const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; - ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); - ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == (_TAG_HEADER_HEAP_BITS & mask)); + { + if (Fail.get() != 0) { + const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; + ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); + ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == + (_TAG_HEADER_HEAP_BITS & mask)); - comment("is_bitstring_header"); - a.and_(TMP1, TMP1, imm(mask)); - a.cmp(TMP1, imm(_TAG_HEADER_HEAP_BITS)); - a.b_ne(resolve_beam_label(Fail, disp1MB)); - } + a.and_(TMP2, TMP2, imm(mask)); + a.cmp(TMP2, imm(_TAG_HEADER_HEAP_BITS)); + a.b_ne(resolve_beam_label(Fail, disp1MB)); + } - emit_gc_test_preserve(ArgWord(ERL_BIN_MATCHSTATE_SIZE(0)), Live, Src, ARG2); + emit_gc_test_preserve(ArgWord(ERL_SUB_BITS_SIZE), Live, Src, ARG2); - emit_enter_runtime(Live.get()); + emit_enter_runtime(Live.get()); - a.mov(ARG1, c_p); - /* ARG2 was set above */ - runtime_call<2>(erts_bs_start_match_3); + a.mov(ARG1, c_p); + /* ARG2 was set above */ + runtime_call<2>(erts_bs_start_match_3); - emit_leave_runtime(Live.get()); + emit_leave_runtime(Live.get()); - a.add(ARG2, ARG1, imm(TAG_PRIMARY_BOXED)); + a.add(ARG2, ARG1, imm(TAG_PRIMARY_BOXED)); + } a.bind(next); mov_arg(Dst, ARG2); @@ -583,27 +589,25 @@ void BeamModuleAssembler::emit_i_bs_match_string(const ArgRegister &Ctx, const ArgLabel &Fail, const ArgWord &Bits, const ArgBytePtr &Ptr) { - const int position_offset = offsetof(ErlBinMatchState, mb.offset); - const int size_offset = offsetof(ErlBinMatchState, mb.size); - const int base_offset = offsetof(ErlBinMatchState, mb.base); - const auto size = Bits.get(); { auto ctx_reg = load_source(Ctx, TMP1); - a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, position_offset)); - a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, size_offset)); + emit_untag_ptr(TMP5, ctx_reg.reg); + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(TMP5, offsetof(ErlSubBits, start))); add(TMP4, TMP2, size); a.cmp(TMP4, TMP3); a.b_hi(resolve_beam_label(Fail, disp1MB)); - /* ARG4 = mb->offset & 7 */ - a.and_(ARG4, TMP2, imm(7)); - - /* ARG3 = mb->base + (mb->offset >> 3) */ - a.ldur(TMP1, emit_boxed_val(ctx_reg.reg, base_offset)); + /* ARG3 = (sb->base_flags & ~mask) + (sb->start >> 3) */ + a.ldr(TMP1, arm::Mem(TMP5, offsetof(ErlSubBits, base_flags))); + a.and_(TMP1, TMP1, imm(~ERL_SUB_BITS_FLAG_MASK)); a.add(ARG3, TMP1, TMP2, arm::lsr(3)); + + /* ARG4 = sb->start & 7 */ + a.and_(ARG4, TMP2, imm(7)); } emit_enter_runtime(); @@ -619,21 +623,21 @@ void BeamModuleAssembler::emit_i_bs_match_string(const ArgRegister &Ctx, { auto ctx_reg = load_source(Ctx, TMP1); - a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, position_offset)); + a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, offsetof(ErlSubBits, start))); add(TMP2, TMP2, size); - a.stur(TMP2, emit_boxed_val(ctx_reg.reg, position_offset)); + a.stur(TMP2, emit_boxed_val(ctx_reg.reg, offsetof(ErlSubBits, start))); } } void BeamModuleAssembler::emit_i_bs_get_position(const ArgRegister &Ctx, const ArgRegister &Dst) { - const int position_offset = offsetof(ErlBinMatchState, mb.offset); + const int start_offset = offsetof(ErlSubBits, start); auto ctx_reg = load_source(Ctx, TMP1); auto dst_reg = init_destination(Dst, TMP2); /* Match contexts can never be literals, so we can skip clearing literal * tags. */ - a.ldur(dst_reg.reg, emit_boxed_val(ctx_reg.reg, position_offset)); + a.ldur(dst_reg.reg, emit_boxed_val(ctx_reg.reg, start_offset)); a.lsl(dst_reg.reg, dst_reg.reg, imm(_TAG_IMMED1_SIZE)); a.orr(dst_reg.reg, dst_reg.reg, imm(_TAG_IMMED1_SMALL)); @@ -729,15 +733,15 @@ void BeamModuleAssembler::emit_bs_get_integer2(const ArgLabel &Fail, void BeamModuleAssembler::emit_bs_test_tail2(const ArgLabel &Fail, const ArgRegister &Ctx, const ArgWord &Offset) { - const int position_offset = offsetof(ErlBinMatchState, mb.offset); - const int size_offset = offsetof(ErlBinMatchState, mb.size); + const int start_offset = offsetof(ErlSubBits, start); + const int end_offset = offsetof(ErlSubBits, end); auto ctx_reg = load_source(Ctx, TMP1); ASSERT(Offset.isWord()); - a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, size_offset)); - a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, position_offset)); + a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, end_offset)); + a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, start_offset)); a.sub(TMP2, TMP2, TMP3); if (Offset.get() != 0) { @@ -750,11 +754,11 @@ void BeamModuleAssembler::emit_bs_test_tail2(const ArgLabel &Fail, void BeamModuleAssembler::emit_bs_set_position(const ArgRegister &Ctx, const ArgRegister &Pos) { - const int position_offset = offsetof(ErlBinMatchState, mb.offset); + const int start_offset = offsetof(ErlSubBits, start); auto [ctx, pos] = load_sources(Ctx, TMP1, Pos, TMP2); a.lsr(TMP2, pos.reg, imm(_TAG_IMMED1_SIZE)); - a.stur(TMP2, emit_boxed_val(ctx.reg, position_offset)); + a.stur(TMP2, emit_boxed_val(ctx.reg, start_offset)); } void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, @@ -772,11 +776,10 @@ void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, ARG1); /* Make field fetching slightly more compact by pre-loading the match - * buffer into the right argument slot for `erts_bs_get_binary_all_2`. */ - lea(ARG2, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb))); - - ERTS_CT_ASSERT_FIELD_PAIR(ErlBinMatchBuffer, offset, size); - a.ldp(TMP2, TMP3, arm::Mem(ARG2, offsetof(ErlBinMatchBuffer, offset))); + * context into the right argument slot for `erts_bs_get_binary_all_2`. */ + emit_untag_ptr(ARG2, ARG1); + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(ARG2, offsetof(ErlSubBits, start))); /* Remainder = Size - Offset */ a.sub(TMP1, TMP3, TMP2); @@ -809,22 +812,22 @@ void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, } void BeamGlobalAssembler::emit_bs_get_tail_shared() { - lea(TMP1, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb))); + emit_untag_ptr(TMP1, ARG1); - ERTS_CT_ASSERT_FIELD_PAIR(ErlBinMatchBuffer, orig, base); - a.ldp(ARG2, ARG4, arm::Mem(TMP1, offsetof(ErlBinMatchBuffer, orig))); + a.ldr(ARG4, arm::Mem(TMP1, offsetof(ErlSubBits, base_flags))); + a.ldr(ARG2, arm::Mem(TMP1, offsetof(ErlSubBits, orig))); + a.and_(ARG4, ARG4, imm(~ERL_SUB_BITS_FLAG_MASK)); a.and_(ARG3, ARG2, imm(~TAG_PTR_MASK__)); a.and_(ARG2, ARG2, imm(TAG_PTR_MASK__)); - ERTS_CT_ASSERT_FIELD_PAIR(ErlBinMatchBuffer, offset, size); - a.ldp(ARG5, TMP1, arm::Mem(TMP1, offsetof(ErlBinMatchBuffer, offset))); + /* Extracted size = sb->end - sb->start */ + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(ARG5, TMP1, arm::Mem(TMP1, offsetof(ErlSubBits, start))); + a.sub(ARG6, TMP1, ARG5); lea(ARG1, arm::Mem(c_p, offsetof(Process, htop))); - /* Extracted size = mb->size - mb->offset */ - a.sub(ARG6, TMP1, ARG5); - emit_enter_runtime_frame(); emit_enter_runtime(); @@ -854,19 +857,19 @@ void BeamModuleAssembler::emit_bs_get_tail(const ArgRegister &Ctx, /* Bits to skip are passed in ARG1 */ void BeamModuleAssembler::emit_bs_skip_bits(const ArgLabel &Fail, const ArgRegister &Ctx) { - const int position_offset = offsetof(ErlBinMatchState, mb.offset); - const int size_offset = offsetof(ErlBinMatchState, mb.size); + const int start_offset = offsetof(ErlSubBits, start); + const int end_offset = offsetof(ErlSubBits, end); auto ctx_reg = load_source(Ctx, TMP1); - a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, position_offset)); - a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, size_offset)); + a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, start_offset)); + a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, end_offset)); a.add(TMP2, TMP2, ARG1); a.cmp(TMP2, TMP3); a.b_hi(resolve_beam_label(Fail, disp1MB)); - a.stur(TMP2, emit_boxed_val(ctx_reg.reg, position_offset)); + a.stur(TMP2, emit_boxed_val(ctx_reg.reg, start_offset)); } void BeamModuleAssembler::emit_i_bs_skip_bits2(const ArgRegister &Ctx, @@ -885,18 +888,17 @@ void BeamModuleAssembler::emit_i_bs_skip_bits2(const ArgRegister &Ctx, if (!can_fail && Unit.get() == 1) { comment("simplified skipping because the types are known"); - const int position_offset = offsetof(ErlBinMatchState, mb.offset); - const int size_offset = offsetof(ErlBinMatchState, mb.size); auto [ctx, size] = load_sources(Ctx, TMP1, Size, TMP2); - a.ldur(TMP3, emit_boxed_val(ctx.reg, position_offset)); - a.ldur(TMP4, emit_boxed_val(ctx.reg, size_offset)); + emit_untag_ptr(TMP5, ctx.reg); + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP3, TMP4, arm::Mem(TMP5, offsetof(ErlSubBits, start))); a.add(TMP3, TMP3, size.reg, arm::lsr(_TAG_IMMED1_SIZE)); a.cmp(TMP3, TMP4); a.b_hi(resolve_beam_label(Fail, disp1MB)); - a.stur(TMP3, emit_boxed_val(ctx.reg, position_offset)); + a.str(TMP3, arm::Mem(TMP5, offsetof(ErlSubBits, start))); } else if (emit_bs_get_field_size(Size, Unit.get(), fail, ARG1) >= 0) { emit_bs_skip_bits(Fail, Ctx); } @@ -931,7 +933,7 @@ void BeamModuleAssembler::emit_i_bs_get_binary2(const ArgRegister &Ctx, Ctx, ARG4); - lea(ARG4, emit_boxed_val(ARG4, offsetof(ErlBinMatchState, mb))); + emit_untag_ptr(ARG4, ARG4); emit_enter_runtime(Live.get()); @@ -965,7 +967,7 @@ void BeamModuleAssembler::emit_i_bs_get_float2(const ArgRegister &Ctx, emit_gc_test_preserve(ArgWord(FLOAT_SIZE_OBJECT), Live, Ctx, ARG4); if (emit_bs_get_field_size(Sz, unit, fail, ARG2) >= 0) { - lea(ARG4, emit_boxed_val(ARG4, offsetof(ErlBinMatchState, mb))); + emit_untag_ptr(ARG4, ARG4); emit_enter_runtime(Live.get()); @@ -1041,9 +1043,9 @@ void BeamModuleAssembler::emit_i_bs_put_utf8(const ArgLabel &Fail, * return value. */ void BeamGlobalAssembler::emit_bs_get_utf8_short_shared() { - const int position_offset = offsetof(ErlBinMatchBuffer, offset); + const int start_offset = offsetof(ErlSubBits, start); - const arm::Gp match_state = ARG1; + const arm::Gp match_context = ARG1; const arm::Gp bitdata = ARG2; const arm::Gp bin_position = ARG3; const arm::Gp bin_base = ARG4; @@ -1111,7 +1113,7 @@ void BeamGlobalAssembler::emit_bs_get_utf8_short_shared() { /* Handle plain old ASCII (code point < 128). */ a.bind(ascii); a.add(bin_position, bin_position, imm(8)); - a.str(bin_position, arm::Mem(match_state, position_offset)); + a.str(bin_position, arm::Mem(match_context, start_offset)); a.mov(ARG1, imm(_TAG_IMMED1_SMALL)); a.orr(ARG1, ARG1, bitdata, arm::lsr(56 - _TAG_IMMED1_SIZE)); a.ret(a64::x30); @@ -1132,9 +1134,9 @@ void BeamGlobalAssembler::emit_bs_get_utf8_short_shared() { * failure, ARG1 contains an invalid term where the tags bits are zero. */ void BeamGlobalAssembler::emit_bs_get_utf8_shared() { - const int position_offset = offsetof(ErlBinMatchBuffer, offset); + const int start_offset = offsetof(ErlSubBits, start); - const arm::Gp match_state = ARG1; + const arm::Gp match_context = ARG1; const arm::Gp bitdata = ARG2; const arm::Gp bin_position = ARG3; @@ -1249,7 +1251,7 @@ void BeamGlobalAssembler::emit_bs_get_utf8_shared() { a.csel(TMP2, TMP2, ZERO, imm(arm::CondCode::kLS)); a.add(bin_position, bin_position, byte_count, arm::lsl(3)); - a.str(bin_position, arm::Mem(match_state, position_offset)); + a.str(bin_position, arm::Mem(match_context, start_offset)); a.orr(ARG1, TMP2, bitdata, arm::lsl(_TAG_IMMED1_SIZE)); a.ret(a64::x30); @@ -1257,10 +1259,10 @@ void BeamGlobalAssembler::emit_bs_get_utf8_shared() { void BeamModuleAssembler::emit_bs_get_utf8(const ArgRegister &Ctx, const ArgLabel &Fail) { - const int base_offset = offsetof(ErlBinMatchBuffer, base); - const int position_offset = offsetof(ErlBinMatchBuffer, offset); + const int base_offset = offsetof(ErlSubBits, base_flags); + const int start_offset = offsetof(ErlSubBits, start); - const arm::Gp match_state = ARG1; + const arm::Gp match_context = ARG1; const arm::Gp bitdata = ARG2; const arm::Gp bin_position = ARG3; const arm::Gp bin_base = ARG4; @@ -1273,10 +1275,12 @@ void BeamModuleAssembler::emit_bs_get_utf8(const ArgRegister &Ctx, Label check = a.newLabel(); Label done = a.newLabel(); - lea(match_state, emit_boxed_val(ctx.reg, offsetof(ErlBinMatchState, mb))); - ERTS_CT_ASSERT_FIELD_PAIR(ErlBinMatchBuffer, offset, size); - a.ldp(bin_position, bin_size, arm::Mem(ARG1, position_offset)); + emit_untag_ptr(match_context, ctx.reg); + + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(bin_position, bin_size, arm::Mem(ARG1, start_offset)); a.ldr(bin_base, arm::Mem(ARG1, base_offset)); + a.and_(bin_base, bin_base, imm(~ERL_SUB_BITS_FLAG_MASK)); a.sub(bitdata, bin_size, bin_position); a.cmp(bitdata, imm(32)); a.b_lo(fallback); @@ -1286,7 +1290,7 @@ void BeamModuleAssembler::emit_bs_get_utf8(const ArgRegister &Ctx, /* Handle plain old ASCII (code point < 128). */ a.add(bin_position, bin_position, imm(8)); - a.str(bin_position, arm::Mem(ARG1, position_offset)); + a.str(bin_position, arm::Mem(ARG1, start_offset)); a.mov(ARG1, imm(_TAG_IMMED1_SMALL)); a.orr(ARG1, ARG1, bitdata, arm::lsr(56 - _TAG_IMMED1_SIZE)); a.b(done); @@ -1367,7 +1371,7 @@ void BeamModuleAssembler::emit_bs_get_utf16(const ArgRegister &Ctx, const ArgWord &Flags) { auto ctx_reg = load_source(Ctx, TMP1); - lea(ARG1, emit_boxed_val(ctx_reg.reg, offsetof(ErlBinMatchState, mb))); + emit_untag_ptr(ARG1, ctx_reg.reg); emit_enter_runtime(); @@ -1449,12 +1453,12 @@ void BeamModuleAssembler::emit_i_bs_validate_unicode_retract( a.bind(fail); { - const int position_offset = offsetof(ErlBinMatchState, mb.offset); + const int start_offset = offsetof(ErlSubBits, start); auto ctx_reg = load_source(Ms, TMP2); - a.ldur(TMP1, emit_boxed_val(ctx_reg.reg, position_offset)); + a.ldur(TMP1, emit_boxed_val(ctx_reg.reg, start_offset)); a.sub(TMP1, TMP1, imm(32)); - a.stur(TMP1, emit_boxed_val(ctx_reg.reg, position_offset)); + a.stur(TMP1, emit_boxed_val(ctx_reg.reg, start_offset)); if (Fail.get() != 0) { a.b(resolve_beam_label(Fail, disp128MB)); @@ -1472,10 +1476,8 @@ void BeamModuleAssembler::emit_bs_test_unit(const ArgLabel &Fail, auto ctx_reg = load_source(Ctx, TMP1); unsigned int unit = Unit.get(); - a.ldur(TMP2, - emit_boxed_val(ctx_reg.reg, offsetof(ErlBinMatchState, mb.size))); - a.ldur(TMP3, - emit_boxed_val(ctx_reg.reg, offsetof(ErlBinMatchState, mb.offset))); + a.ldur(TMP2, emit_boxed_val(ctx_reg.reg, offsetof(ErlSubBits, end))); + a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, offsetof(ErlSubBits, start))); a.sub(TMP1, TMP2, TMP3); @@ -2363,22 +2365,10 @@ void BeamModuleAssembler::emit_i_bs_create_bin(const ArgLabel &Fail, if (seg.size.isAtom() && seg.size.as().get() == am_all && seg.type == am_binary) { comment("size of an entire bitstring"); - if (exact_type(seg.src)) { - auto src = load_source(seg.src, ARG1); - arm::Gp boxed_ptr = emit_ptr_val(ARG1, src.reg); - - comment("inlined size code because the value is always " - "a bitstring"); - - a.ldur(TMP2, emit_boxed_val(boxed_ptr, sizeof(Eterm))); - } else { - const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; - ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); - ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == - (_TAG_HEADER_HEAP_BITS & mask)); - mov_arg(ARG1, seg.src); + mov_arg(ARG1, seg.src); + if (!exact_type(seg.src)) { /* Note: ARG1 must remain equal to seg.src here for error * reporting to work. */ if (Fail.get() == 0) { @@ -2390,17 +2380,38 @@ void BeamModuleAssembler::emit_i_bs_create_bin(const ArgLabel &Fail, } emit_is_boxed(resolve_label(error, dispUnknown), seg.src, ARG1); - emit_untag_ptr(TMP3, ARG1); + } + + emit_untag_ptr(TMP4, ARG1); - ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); - ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, thing_word, size); - a.ldp(TMP1, TMP2, arm::Mem(TMP3)); - a.and_(TMP1, TMP1, imm(mask)); - a.cmp(TMP1, imm(_TAG_HEADER_HEAP_BITS)); + ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); + a.ldp(TMP1, TMP2, arm::Mem(TMP4)); + if (masked_types(seg.src) == + BeamTypeId::Bitstring) { + comment("optimized size code because the value is always a " + "bitstring when boxed"); + } else { + const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; + ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); + ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == + (_TAG_HEADER_HEAP_BITS & mask)); + + a.and_(TMP3, TMP1, imm(mask)); + a.cmp(TMP3, imm(_TAG_HEADER_HEAP_BITS)); a.b_ne(resolve_label(error, disp1MB)); } + Label not_sub_bits = a.newLabel(); + a.cmp(TMP1, imm(HEADER_SUB_BITS)); + a.b_ne(not_sub_bits); + { + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(TMP4, offsetof(ErlSubBits, start))); + a.sub(TMP2, TMP3, TMP2); + } + a.bind(not_sub_bits); + a.add(sizeReg, sizeReg, TMP2); } else if (seg.unit != 0) { bool can_fail = true; @@ -3817,10 +3828,10 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, ArgWord const &Need, ArgWord const &Live, Span const &List) { - const int orig_offset = offsetof(ErlBinMatchState, mb.orig); - const int base_offset = offsetof(ErlBinMatchState, mb.base); - const int position_offset = offsetof(ErlBinMatchState, mb.offset); - const int size_offset = offsetof(ErlBinMatchState, mb.size); + const int orig_offset = offsetof(ErlSubBits, orig); + const int base_offset = offsetof(ErlSubBits, base_flags); + const int start_offset = offsetof(ErlSubBits, start); + const int end_offset = offsetof(ErlSubBits, end); std::vector segments; @@ -3912,8 +3923,8 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, auto stride = seg.size; auto unit = seg.unit; - a.ldur(bin_position, emit_boxed_val(ctx_reg.reg, position_offset)); - a.ldur(bin_size, emit_boxed_val(ctx_reg.reg, size_offset)); + a.ldur(bin_position, emit_boxed_val(ctx_reg.reg, start_offset)); + a.ldur(bin_size, emit_boxed_val(ctx_reg.reg, end_offset)); a.sub(TMP5, bin_size, bin_position); if (stride != 0) { cmp(TMP5, stride); @@ -3946,8 +3957,8 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, auto ctx_reg = load_source(Ctx, TMP1); auto size = seg.size; - a.ldur(bin_position, emit_boxed_val(ctx_reg.reg, position_offset)); - a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, size_offset)); + a.ldur(bin_position, emit_boxed_val(ctx_reg.reg, start_offset)); + a.ldur(TMP3, emit_boxed_val(ctx_reg.reg, end_offset)); if (size != 0) { a.sub(TMP1, TMP3, bin_position); cmp(TMP1, size); @@ -3990,16 +4001,17 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, auto ctx = load_source(Ctx, ARG1); if (!position_is_valid) { - a.ldur(bin_position, - emit_boxed_val(ctx.reg, position_offset)); + a.ldur(bin_position, emit_boxed_val(ctx.reg, start_offset)); position_is_valid = true; } + a.ldur(bin_base, emit_boxed_val(ctx.reg, base_offset)); + a.and_(bin_base, bin_base, imm(~ERL_SUB_BITS_FLAG_MASK)); emit_read_bits(seg.size, bin_base, bin_position, bitdata); a.add(bin_position, bin_position, imm(seg.size)); - a.stur(bin_position, emit_boxed_val(ctx.reg, position_offset)); + a.stur(bin_position, emit_boxed_val(ctx.reg, start_offset)); } break; } @@ -4038,7 +4050,7 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, a.mov(ARG1, c_p); a.mov(ARG2, bits); a.mov(ARG3, flags); - lea(ARG4, emit_boxed_val(ctx.reg, offsetof(ErlBinMatchState, mb))); + emit_untag_ptr(ARG4, ctx.reg); if (bits >= SMALL_BITS) { emit_enter_runtime(live); @@ -4069,10 +4081,11 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, a.and_(ARG3, ARG2, imm(~TAG_PTR_MASK__)); a.and_(ARG2, ARG2, imm(TAG_PTR_MASK__)); a.ldur(ARG4, emit_boxed_val(ctx.reg, base_offset)); - a.ldur(ARG5, emit_boxed_val(ctx.reg, position_offset)); + a.and_(ARG4, ARG4, imm(~ERL_SUB_BITS_FLAG_MASK)); + a.ldur(ARG5, emit_boxed_val(ctx.reg, start_offset)); mov_imm(ARG6, seg.size); a.add(TMP2, ARG5, ARG6); - a.stur(TMP2, emit_boxed_val(ctx.reg, position_offset)); + a.stur(TMP2, emit_boxed_val(ctx.reg, start_offset)); emit_enter_runtime( Live.as().get()); @@ -4099,11 +4112,11 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, comment("skip %ld", seg.size); auto ctx = load_source(Ctx, TMP1); if (!position_is_valid) { - a.ldur(bin_position, emit_boxed_val(ctx.reg, position_offset)); + a.ldur(bin_position, emit_boxed_val(ctx.reg, start_offset)); position_is_valid = true; } add(bin_position, bin_position, seg.size); - a.stur(bin_position, emit_boxed_val(ctx.reg, position_offset)); + a.stur(bin_position, emit_boxed_val(ctx.reg, start_offset)); break; } case BsmSegment::action::DROP: diff --git a/erts/emulator/beam/jit/arm/instr_common.cpp b/erts/emulator/beam/jit/arm/instr_common.cpp index 6e97e3a509b4..2cb0992efbfb 100644 --- a/erts/emulator/beam/jit/arm/instr_common.cpp +++ b/erts/emulator/beam/jit/arm/instr_common.cpp @@ -973,24 +973,26 @@ void BeamModuleAssembler::emit_is_binary(const ArgLabel &Fail, auto src = load_source(Src, ARG1); emit_is_boxed(resolve_beam_label(Fail, dispUnknown), Src, src.reg); + emit_untag_ptr(ARG1, src.reg); - if (masked_types(Src) == BeamTypeId::Bitstring) { - arm::Gp boxed_ptr = emit_ptr_val(ARG1, src.reg); + ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); + a.ldp(TMP1, TMP2, arm::Mem(ARG1)); + Label not_sub_bits = a.newLabel(); + a.cmp(TMP1, imm(HEADER_SUB_BITS)); + a.b_ne(not_sub_bits); + { + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(ARG1, offsetof(ErlSubBits, start))); + a.sub(TMP2, TMP3, TMP2); + } + a.bind(not_sub_bits); + + if (masked_types(Src) == BeamTypeId::Bitstring) { comment("skipped header test since we know it's a bitstring when " "boxed"); - - ERTS_CT_ASSERT(offsetof(ErlHeapBits, size) == sizeof(Eterm)); - ERTS_CT_ASSERT(offsetof(ErlSubBits, size) == sizeof(Eterm)); - a.ldur(TMP2, emit_boxed_val(boxed_ptr, sizeof(Eterm))); a.tst(TMP2, imm(7)); } else { - emit_untag_ptr(ARG1, src.reg); - - ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); - ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, thing_word, size); - a.ldp(TMP1, TMP2, arm::Mem(ARG1)); - const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == (_TAG_HEADER_HEAP_BITS & mask)); @@ -1449,19 +1451,6 @@ void BeamModuleAssembler::emit_is_eq_exact(const ArgLabel &Fail, const ArgSource &Y) { auto x = load_source(X, ARG1); - if (exact_type(X) && Y.isLiteral()) { - Eterm literal = beamfile_get_literal(beam, Y.as().get()); - - if (is_bitstring(literal) && bitstring_size(literal) == 0) { - comment("simplified equality test with empty binary"); - - arm::Gp boxed_ptr = emit_ptr_val(ARG1, x.reg); - a.ldur(TMP1, emit_boxed_val(boxed_ptr, sizeof(Eterm))); - a.cbnz(TMP1, resolve_beam_label(Fail, disp1MB)); - return; - } - } - /* If either argument is known to be an immediate, we can fail immediately * if they're not equal. */ if (always_immediate(X) || always_immediate(Y)) { @@ -1522,20 +1511,6 @@ void BeamModuleAssembler::emit_is_ne_exact(const ArgLabel &Fail, const ArgSource &Y) { auto x = load_source(X, ARG1); - if (exact_type(X) && Y.isLiteral()) { - Eterm literal = beamfile_get_literal(beam, Y.as().get()); - - if (is_bitstring(literal) && bitstring_size(literal) == 0) { - arm::Gp boxed_ptr = emit_ptr_val(ARG1, x.reg); - - comment("simplified non-equality test with empty binary"); - a.ldur(TMP1, emit_boxed_val(boxed_ptr, sizeof(Eterm))); - a.cbz(TMP1, resolve_beam_label(Fail, disp1MB)); - - return; - } - } - /* If either argument is known to be an immediate, we can fail immediately * if they're equal. */ if (always_immediate(X) || always_immediate(Y)) { @@ -2322,22 +2297,27 @@ void BeamModuleAssembler::emit_is_int_ge(ArgLabel const &Fail, void BeamModuleAssembler::emit_badmatch(const ArgSource &Src) { emit_error(BADMATCH, Src); + mark_unreachable(); } void BeamModuleAssembler::emit_case_end(const ArgSource &Src) { emit_error(EXC_CASE_CLAUSE, Src); + mark_unreachable(); } void BeamModuleAssembler::emit_system_limit_body() { emit_error(SYSTEM_LIMIT); + mark_unreachable(); } void BeamModuleAssembler::emit_if_end() { emit_error(EXC_IF_CLAUSE); + mark_unreachable(); } void BeamModuleAssembler::emit_badrecord(const ArgSource &Src) { emit_error(EXC_BADRECORD, Src); + mark_unreachable(); } void BeamModuleAssembler::emit_catch(const ArgYRegister &Y, diff --git a/erts/emulator/beam/jit/arm/instr_fun.cpp b/erts/emulator/beam/jit/arm/instr_fun.cpp index 61cded38e755..b7ad0ffa8e66 100644 --- a/erts/emulator/beam/jit/arm/instr_fun.cpp +++ b/erts/emulator/beam/jit/arm/instr_fun.cpp @@ -205,45 +205,40 @@ void BeamModuleAssembler::emit_i_make_fun3(const ArgLambda &Lambda, const ArgWord &Arity, const ArgWord &NumFree, const Span &env) { - const ssize_t num_free = NumFree.get(); - ssize_t i; + Uint i = 0; - ASSERT(num_free == (ssize_t)env.size()); + ASSERT((NumFree.get() + 1) == env.size() && + (NumFree.get() + Arity.get()) < MAX_ARG); - a.mov(ARG1, c_p); - mov_arg(ARG2, Lambda); - mov_arg(ARG3, Arity); - mov_arg(ARG4, NumFree); - - emit_enter_runtime(); - - runtime_call<4>(erts_new_local_fun_thing); + mov_arg(TMP2, Lambda); - emit_leave_runtime(); - - if (num_free) { - comment("Move fun environment"); - } + comment("Create fun thing"); + mov_imm(TMP1, MAKE_FUN_HEADER(Arity.get(), NumFree.get(), 0)); + ERTS_CT_ASSERT_FIELD_PAIR(ErlFunThing, thing_word, entry.fun); + a.stp(TMP1, TMP2, arm::Mem(HTOP, offsetof(ErlFunThing, thing_word))); - for (i = 0; i < num_free - 1; i += 2) { - ssize_t offset = offsetof(ErlFunThing, env) + i * sizeof(Eterm); + comment("Move fun environment"); + while (i < env.size() - 1) { + int offset = offsetof(ErlFunThing, env) + i * sizeof(Eterm); if ((i % 128) == 0) { check_pending_stubs(); } auto [first, second] = load_sources(env[i], TMP1, env[i + 1], TMP2); - safe_stp(first.reg, second.reg, arm::Mem(ARG1, offset)); + safe_stp(first.reg, second.reg, arm::Mem(HTOP, offset)); + i += 2; } - if (i < num_free) { - ssize_t offset = offsetof(ErlFunThing, env) + i * sizeof(Eterm); - mov_arg(arm::Mem(ARG1, offset), env[i]); + if (i < env.size()) { + int offset = offsetof(ErlFunThing, env) + i * sizeof(Eterm); + mov_arg(arm::Mem(HTOP, offset), env[i]); } comment("Create boxed ptr"); auto dst = init_destination(Dst, TMP1); - a.orr(dst.reg, ARG1, imm(TAG_PRIMARY_BOXED)); + a.orr(dst.reg, HTOP, imm(TAG_PRIMARY_BOXED)); + add(HTOP, HTOP, (ERL_FUN_SIZE + env.size()) * sizeof(Eterm)); flush_var(dst); } diff --git a/erts/emulator/beam/jit/arm/instr_guard_bifs.cpp b/erts/emulator/beam/jit/arm/instr_guard_bifs.cpp index 9b7c59bdda5b..9bc3e540f914 100644 --- a/erts/emulator/beam/jit/arm/instr_guard_bifs.cpp +++ b/erts/emulator/beam/jit/arm/instr_guard_bifs.cpp @@ -339,9 +339,18 @@ void BeamGlobalAssembler::emit_bif_bit_size_helper(Label error) { emit_untag_ptr(TMP3, ARG1); ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); - ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, thing_word, size); a.ldp(TMP1, TMP2, arm::Mem(TMP3)); + Label not_sub_bits = a.newLabel(); + a.cmp(TMP1, imm(HEADER_SUB_BITS)); + a.b_ne(not_sub_bits); + { + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(TMP3, offsetof(ErlSubBits, start))); + a.sub(TMP2, TMP3, TMP2); + } + a.bind(not_sub_bits); + const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == (_TAG_HEADER_HEAP_BITS & mask)); @@ -380,18 +389,26 @@ void BeamModuleAssembler::emit_bif_bit_size(const ArgLabel &Fail, mov_var(dst, ARG1); } else { emit_is_boxed(resolve_beam_label(Fail, dispUnknown), Src, src.reg); + emit_untag_ptr(ARG1, src.reg); + + ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); + a.ldp(TMP1, TMP2, arm::Mem(ARG1)); + + Label not_sub_bits = a.newLabel(); + a.cmp(TMP1, imm(HEADER_SUB_BITS)); + a.b_ne(not_sub_bits); + { + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(ARG1, offsetof(ErlSubBits, start))); + a.sub(TMP2, TMP3, TMP2); + } + a.bind(not_sub_bits); if (masked_types(Src) == BeamTypeId::Bitstring) { - arm::Gp boxed_ptr = emit_ptr_val(ARG1, src.reg); - comment("skipped header test since we know it's a bitstring when " "boxed"); - a.ldur(TMP2, emit_boxed_val(boxed_ptr, sizeof(Eterm))); } else { - emit_untag_ptr(ARG1, src.reg); - a.ldp(TMP1, TMP2, arm::Mem(ARG1)); - const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == @@ -445,18 +462,26 @@ void BeamModuleAssembler::emit_bif_byte_size(const ArgLabel &Fail, mov_var(dst, ARG1); } else { emit_is_boxed(resolve_beam_label(Fail, dispUnknown), Src, src.reg); + emit_untag_ptr(ARG1, src.reg); + + ERTS_CT_ASSERT_FIELD_PAIR(ErlHeapBits, thing_word, size); + a.ldp(TMP1, TMP2, arm::Mem(ARG1)); + + Label not_sub_bits = a.newLabel(); + a.cmp(TMP1, imm(HEADER_SUB_BITS)); + a.b_ne(not_sub_bits); + { + ERTS_CT_ASSERT_FIELD_PAIR(ErlSubBits, start, end); + a.ldp(TMP2, TMP3, arm::Mem(ARG1, offsetof(ErlSubBits, start))); + a.sub(TMP2, TMP3, TMP2); + } + a.bind(not_sub_bits); if (masked_types(Src) == BeamTypeId::Bitstring) { - arm::Gp boxed_ptr = emit_ptr_val(ARG1, src.reg); - comment("skipped header test since we know it's a bitstring when " "boxed"); - a.ldur(TMP2, emit_boxed_val(boxed_ptr, sizeof(Eterm))); } else { - emit_untag_ptr(ARG1, src.reg); - a.ldp(TMP1, TMP2, arm::Mem(ARG1)); - const auto mask = _BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK; ERTS_CT_ASSERT(TAG_PRIMARY_HEADER == 0); ERTS_CT_ASSERT(_TAG_HEADER_HEAP_BITS == diff --git a/erts/emulator/beam/jit/arm/instr_trace.cpp b/erts/emulator/beam/jit/arm/instr_trace.cpp index 1db21b74460f..f07648127cec 100644 --- a/erts/emulator/beam/jit/arm/instr_trace.cpp +++ b/erts/emulator/beam/jit/arm/instr_trace.cpp @@ -215,4 +215,5 @@ void BeamModuleAssembler::emit_i_hibernate() { a.bind(error); emit_raise_exception(&BIF_TRAP_EXPORT(BIF_hibernate_3)->info.mfa); + mark_unreachable(); } diff --git a/erts/emulator/beam/jit/arm/ops.tab b/erts/emulator/beam/jit/arm/ops.tab index f3c99bac9f2c..ef0c1c840be3 100644 --- a/erts/emulator/beam/jit/arm/ops.tab +++ b/erts/emulator/beam/jit/arm/ops.tab @@ -258,7 +258,7 @@ i_get_tuple_element Tuple Pos Tuple2 | current_tuple Tuple3 | equal(Tuple, Tuple2) | equal(Tuple, Tuple3) => i_get_tuple_element Tuple Pos Tuple -# This is a current_tuple instruction instruction not followed by +# This is a current_tuple instruction not followed by # get_tuple_element. Invalidate the current tuple pointer. current_tuple Tuple => _ diff --git a/erts/emulator/beam/jit/asm_load.c b/erts/emulator/beam/jit/asm_load.c index 52299031c007..bf97d3298751 100644 --- a/erts/emulator/beam/jit/asm_load.c +++ b/erts/emulator/beam/jit/asm_load.c @@ -128,15 +128,15 @@ int beam_load_prepare_emit(LoaderState *stp) { init_label(&stp->labels[i]); } - stp->lambda_literals = erts_alloc(ERTS_ALC_T_PREPARED_CODE, - stp->beam.lambdas.count * sizeof(SWord)); + stp->fun_refs = erts_alloc(ERTS_ALC_T_PREPARED_CODE, + stp->beam.lambdas.count * sizeof(SWord)); for (i = 0; i < stp->beam.lambdas.count; i++) { BeamFile_LambdaEntry *lambda = &stp->beam.lambdas.entries[i]; if (stp->labels[lambda->label].lambda_index == INVALID_LAMBDA_INDEX) { stp->labels[lambda->label].lambda_index = i; - stp->lambda_literals[i] = ERTS_SWORD_MAX; + stp->fun_refs[i] = ERTS_SWORD_MAX; } else { beam_load_report_error(__LINE__, stp, @@ -222,6 +222,14 @@ int beam_load_prepared_dtor(Binary *magic) { erts_free(ERTS_ALC_T_PREPARED_CODE, hdr->are_nifs); hdr->are_nifs = NULL; } + if (hdr->coverage) { + erts_free(ERTS_ALC_T_CODE_COVERAGE, hdr->coverage); + hdr->coverage = NULL; + } + if (hdr->line_coverage_valid) { + erts_free(ERTS_ALC_T_CODE_COVERAGE, hdr->line_coverage_valid); + hdr->line_coverage_valid = NULL; + } erts_free(ERTS_ALC_T_PREPARED_CODE, hdr); stp->load_hdr = NULL; @@ -232,9 +240,9 @@ int beam_load_prepared_dtor(Binary *magic) { stp->labels = NULL; } - if (stp->lambda_literals) { - erts_free(ERTS_ALC_T_PREPARED_CODE, (void *)stp->lambda_literals); - stp->lambda_literals = NULL; + if (stp->fun_refs) { + erts_free(ERTS_ALC_T_PREPARED_CODE, (void *)stp->fun_refs); + stp->fun_refs = NULL; } if (stp->bif_imports) { @@ -871,6 +879,12 @@ int beam_load_finish_emit(LoaderState *stp) { (const char *)stp->beam.checksum, sizeof(stp->beam.checksum)); + /* Transfer ownership of the coverage tables to the prepared code. */ + stp->load_hdr->coverage = stp->coverage; + stp->load_hdr->line_coverage_valid = stp->line_coverage_valid; + stp->coverage = NULL; + stp->line_coverage_valid = NULL; + /* Move the code to its final location. */ beamasm_codegen(stp->ba, &stp->executable_region, @@ -887,13 +901,6 @@ int beam_load_finish_emit(LoaderState *stp) { stp->code_hdr = code_hdr_ro; stp->loaded_size = module_size; - /* Transfer ownership of the coverage tables to the loaded code. */ - code_hdr_rw->coverage = stp->coverage; - code_hdr_rw->line_coverage_valid = stp->line_coverage_valid; - - stp->coverage = NULL; - stp->line_coverage_valid = NULL; - /* * Place the literals in their own allocated heap (for fast range check) * and fix up all instructions that refer to it. @@ -1006,7 +1013,7 @@ int beam_load_finish_emit(LoaderState *stp) { void beam_load_finalize_code(LoaderState *stp, struct erl_module_instance *inst_p) { ErtsCodeIndex staging_ix; - int code_size, i; + int code_size; ERTS_LC_ASSERT(erts_initialized == 0 || erts_has_code_load_permission() || erts_thr_progress_is_blocking()); @@ -1042,7 +1049,7 @@ void beam_load_finalize_code(LoaderState *stp, /* Exported functions */ staging_ix = erts_staging_code_ix(); - for (i = 0; i < stp->beam.exports.count; i++) { + for (int i = 0; i < stp->beam.exports.count; i++) { BeamFile_ExportEntry *entry = &stp->beam.exports.entries[i]; ErtsCodePtr address; Export *ep; @@ -1062,7 +1069,7 @@ void beam_load_finalize_code(LoaderState *stp, /* Patch external function calls, this is done after exporting functions as * the module may remote-call itself*/ - for (i = 0; i < stp->beam.imports.count; i++) { + for (int i = 0; i < stp->beam.imports.count; i++) { BeamFile_ImportEntry *entry = &stp->beam.imports.entries[i]; Export *import; @@ -1073,12 +1080,13 @@ void beam_load_finalize_code(LoaderState *stp, /* Patch fun creation. */ if (stp->beam.lambdas.count) { - ErtsLiteralArea *literal_area = (stp->code_hdr)->literal_area; BeamFile_LambdaTable *lambda_table = &stp->beam.lambdas; - for (i = 0; i < lambda_table->count; i++) { + for (int i = 0; i < lambda_table->count; i++) { BeamFile_LambdaEntry *lambda; ErlFunEntry *fun_entry; + FunRef *fun_refp; + Eterm fun_ref; lambda = &lambda_table->entries[i]; @@ -1089,39 +1097,37 @@ void beam_load_finalize_code(LoaderState *stp, lambda->index, lambda->arity - lambda->num_free); - if (erts_is_fun_loaded(fun_entry, staging_ix)) { - /* We've reloaded a module over itself and inherited the old - * instance's fun entries, so we need to undo the reference - * bump in `erts_put_fun_entry2` to make fun purging work. */ - erts_refc_dectest(&fun_entry->refc, 1); + ASSERT(stp->fun_refs[i] != ERTS_SWORD_MAX); + fun_ref = beamfile_get_literal(&stp->beam, stp->fun_refs[i]); + + /* If there are no free variables, the literal refers to an + * ErlFunThing that needs to be fixed up before we process the + * FunRef. */ + if (lambda->num_free == 0) { + ErlFunThing *funp = (ErlFunThing *)boxed_val(fun_ref); + ASSERT(fun_env_size(funp) == 1 && funp->entry.fun == NULL); + funp->entry.fun = fun_entry; + fun_ref = funp->env[0]; + } + + fun_refp = (FunRef *)boxed_val(fun_ref); + fun_refp->entry = fun_entry; + + /* Bump the reference count: this could not be done when copying + * the literal as we had no idea which entry it belonged to. + * + * We also need to parry an annoying wrinkle: when reloading a + * module over itself, we inherit the old instance's fun entries, + * and thus have to cancel the reference bump in + * `erts_put_fun_entry2` to make fun purging work. */ + if (!erts_is_fun_loaded(fun_entry, staging_ix)) { + erts_refc_inctest(&fun_entry->refc, 1); } erts_set_fun_code(fun_entry, staging_ix, beamasm_get_lambda(stp->ba, i)); - /* Finalize the literal we've created for this lambda, if any, - * converting it from an external fun to a local one with the newly - * created fun entry. */ - if (stp->lambda_literals[i] != ERTS_SWORD_MAX) { - ErlFunThing *funp; - Eterm literal; - - literal = beamfile_get_literal(&stp->beam, - stp->lambda_literals[i]); - funp = (ErlFunThing *)fun_val(literal); - - funp->entry.fun = fun_entry; - - funp->next = literal_area->off_heap; - literal_area->off_heap = (struct erl_off_heap_header *)funp; - - ASSERT(funp->thing_word & (1 << FUN_HEADER_EXTERNAL_OFFS)); - funp->thing_word &= ~(1 << FUN_HEADER_EXTERNAL_OFFS); - - erts_refc_inc(&fun_entry->refc, 2); - } - beamasm_patch_lambda(stp->ba, stp->writable_region, i, fun_entry); } } @@ -1134,6 +1140,8 @@ void beam_load_finalize_code(LoaderState *stp, /* Prevent literals and code from being freed. */ (stp->load_hdr)->literal_area = NULL; stp->load_hdr->are_nifs = NULL; + stp->load_hdr->coverage = NULL; + stp->load_hdr->line_coverage_valid = NULL; stp->executable_region = NULL; stp->writable_region = NULL; stp->code_hdr = NULL; diff --git a/erts/emulator/beam/jit/beam_jit_common.cpp b/erts/emulator/beam/jit/beam_jit_common.cpp index e70c838814ef..a65fa547de02 100644 --- a/erts/emulator/beam/jit/beam_jit_common.cpp +++ b/erts/emulator/beam/jit/beam_jit_common.cpp @@ -569,14 +569,6 @@ void BeamModuleAssembler::patchStrings(char *rw_base, #if defined(DEBUG) && defined(JIT_HARD_DEBUG) void beam_jit_validate_term(Eterm term) { - if (is_boxed(term)) { - Eterm header = *boxed_val(term); - - if (header_is_bin_matchstate(header)) { - return; - } - } - size_object_x(term, NULL); } #endif @@ -829,7 +821,7 @@ Eterm beam_jit_bs_init(Process *c_p, c_p->htop += bin_need; hb->thing_word = header_heap_bits(num_bits); - ERTS_SET_HB_SIZE(hb, num_bits); + hb->size = num_bits; erts_current_bin = (byte *)hb->data; return make_bitstring(hb); @@ -877,7 +869,7 @@ Eterm beam_jit_bs_init_bits(Process *c_p, c_p->htop += heap_bits_size(num_bits); hb->thing_word = header_heap_bits(num_bits); - ERTS_SET_HB_SIZE(hb, num_bits); + hb->size = num_bits; erts_current_bin = (byte *)hb->data; return make_bitstring(hb); @@ -906,7 +898,7 @@ Eterm beam_jit_bs_get_integer(Process *c_p, Uint flags, Uint size, Uint Live) { - ErlBinMatchBuffer *mb; + ErlSubBits *sb; if (size >= SMALL_BITS) { Uint wordsneeded; @@ -917,8 +909,8 @@ Eterm beam_jit_bs_get_integer(Process *c_p, * * Remember to re-acquire the matchbuffer after gc. */ - mb = ms_matchbuffer(context); - if (mb->size - mb->offset < size) { + sb = (ErlSubBits *)bitstring_val(context); + if (sb->end - sb->start < size) { return THE_NON_VALUE; } @@ -928,8 +920,8 @@ Eterm beam_jit_bs_get_integer(Process *c_p, context = reg[Live]; } - mb = ms_matchbuffer(context); - return erts_bs_get_integer_2(c_p, size, flags, mb); + sb = (ErlSubBits *)bitstring_val(context); + return erts_bs_get_integer_2(c_p, size, flags, sb); } void beam_jit_bs_construct_fail_info(Process *c_p, diff --git a/erts/emulator/beam/jit/load.h b/erts/emulator/beam/jit/load.h index 54fb2b921338..7b5d25f4d939 100644 --- a/erts/emulator/beam/jit/load.h +++ b/erts/emulator/beam/jit/load.h @@ -88,10 +88,11 @@ struct LoaderState_ { void *coverage; byte *line_coverage_valid; - /* Translates lambda indexes to their literals, if any. Lambdas that lack - * a literal (for example if they have an environment) are represented by - * ERTS_SWORD_MAX. */ - SWord *lambda_literals; + /* Translates lambda indexes to the literal holding their FunRef. + * + * Lambdas that lack an environment are represented by an ErlFunThing that + * is immediately followed by an FunRef. */ + SWord *fun_refs; void *ba; /* Assembler used to create x86 assembly */ diff --git a/erts/emulator/beam/jit/x86/instr_bs.cpp b/erts/emulator/beam/jit/x86/instr_bs.cpp index eb43b514c5f5..fd3f32ba7e42 100644 --- a/erts/emulator/beam/jit/x86/instr_bs.cpp +++ b/erts/emulator/beam/jit/x86/instr_bs.cpp @@ -577,34 +577,48 @@ void BeamModuleAssembler::emit_i_bs_start_match3(const ArgRegister &Src, } x86::Gp boxed_ptr = emit_ptr_val(ARG1, ARG2); - a.mov(RETd, emit_boxed_val(boxed_ptr, 0, sizeof(Uint32))); - a.and_(RETb, imm(_HEADER_SUBTAG_MASK)); - a.cmp(RETb, imm(BIN_MATCHSTATE_SUBTAG)); -#ifdef JIT_HARD_DEBUG - a.je(next); + /* Boxed terms have at least one word past the header, so we can + * speculatively load base_flags together with the thing_word. */ + ERTS_CT_ASSERT(offsetof(ErlSubBits, thing_word) == 0 && + offsetof(ErlSubBits, base_flags) == sizeof(Eterm)); + a.mov(RETd, + emit_boxed_val(boxed_ptr, + offsetof(ErlSubBits, base_flags), + sizeof(Uint32))); + + ERTS_CT_ASSERT((HEADER_SUB_BITS & _TAG_PRIMARY_MASK) == 0 && + (ERL_SUB_BITS_FLAG_MASK == _TAG_PRIMARY_MASK)); + a.and_(RETd, imm(ERL_SUB_BITS_FLAG_MASK)); + a.xor_(RETd, imm(HEADER_SUB_BITS | ERL_SUB_BITS_FLAGS_MATCH_CONTEXT)); + a.xor_(RETd, emit_boxed_val(boxed_ptr, 0, sizeof(Uint32))); +#ifdef HARD_DEBUG + a.jz(next); #else - a.short_().je(next); + a.short_().jz(next); #endif - if (Fail.get() != 0) { - comment("is_bitstring_header"); - a.and_(RETb, imm(_BITSTRING_TAG_MASK)); - a.cmp(RETb, imm(_TAG_HEADER_HEAP_BITS)); - a.jne(resolve_beam_label(Fail)); - } + { + /* RETd now contains the thing_word and flag bits that differ. The + * latter can be ignored, and we can check whether this is a bitstring + * by testing whether all bits within the bitstring mask are zero. */ + if (Fail.get() != 0) { + a.and_(RETd, imm(_BITSTRING_TAG_MASK & ~_TAG_PRIMARY_MASK)); + a.jnz(resolve_beam_label(Fail)); + } - emit_gc_test_preserve(ArgWord(ERL_BIN_MATCHSTATE_SIZE(0)), Live, Src, ARG2); + emit_gc_test_preserve(ArgWord(ERL_SUB_BITS_SIZE), Live, Src, ARG2); - emit_enter_runtime(); + emit_enter_runtime(); - a.mov(ARG1, c_p); - /* ARG2 was set above */ - runtime_call<2>(erts_bs_start_match_3); + a.mov(ARG1, c_p); + /* ARG2 was set above */ + runtime_call<2>(erts_bs_start_match_3); - emit_leave_runtime(); + emit_leave_runtime(); - a.lea(ARG2, x86::qword_ptr(RET, TAG_PRIMARY_BOXED)); + a.lea(ARG2, x86::qword_ptr(RET, TAG_PRIMARY_BOXED)); + } a.bind(next); mov_arg(Dst, ARG2); @@ -619,21 +633,25 @@ void BeamModuleAssembler::emit_i_bs_match_string(const ArgRegister &Ctx, mov_arg(ARG1, Ctx); - a.mov(ARG2, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); + a.mov(ARG2, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); a.lea(ARG3, x86::qword_ptr(ARG2, size)); - a.cmp(ARG3, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.size))); + a.cmp(ARG3, emit_boxed_val(ARG1, offsetof(ErlSubBits, end))); a.ja(fail); a.mov(TMP_MEM1q, ARG1); - /* ARG4 = mb->offset & 7 */ + /* ARG4 = sb->start & 7 */ a.mov(ARG4, ARG2); a.and_(ARG4, imm(7)); - /* ARG3 = mb->base + (mb->offset >> 3) */ + /* ARG3 = sb->base_flags + (sb->start >> 3) - match_ctx_flag */ a.shr(ARG2, imm(3)); - a.mov(ARG3, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.base))); - a.add(ARG3, ARG2); + a.mov(ARG3, emit_boxed_val(ARG1, offsetof(ErlSubBits, base_flags))); + a.lea(ARG3, + x86::qword_ptr(ARG3, + ARG2, + 0, + -(Sint)ERL_SUB_BITS_FLAGS_MATCH_CONTEXT)); emit_enter_runtime(); @@ -648,8 +666,7 @@ void BeamModuleAssembler::emit_i_bs_match_string(const ArgRegister &Ctx, a.jne(fail); a.mov(ARG1, TMP_MEM1q); - a.add(emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset)), - imm(size)); + a.add(emit_boxed_val(ARG1, offsetof(ErlSubBits, start)), imm(size)); } void BeamModuleAssembler::emit_i_bs_get_position(const ArgRegister &Ctx, @@ -658,7 +675,7 @@ void BeamModuleAssembler::emit_i_bs_get_position(const ArgRegister &Ctx, /* Match contexts can never be literals, so we can skip clearing literal * tags. */ - a.mov(ARG1, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); + a.mov(ARG1, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); a.sal(ARG1, imm(_TAG_IMMED1_SIZE)); a.or_(ARG1, imm(_TAG_IMMED1_SMALL)); @@ -758,8 +775,8 @@ void BeamModuleAssembler::emit_bs_test_tail2(const ArgLabel &Fail, const ArgWord &Offset) { mov_arg(ARG1, Ctx); - a.mov(ARG2, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.size))); - a.sub(ARG2, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); + a.mov(ARG2, emit_boxed_val(ARG1, offsetof(ErlSubBits, end))); + a.sub(ARG2, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); if (Offset.get() != 0) { if (Support::isInt32(Offset.get())) { @@ -779,7 +796,7 @@ void BeamModuleAssembler::emit_bs_set_position(const ArgRegister &Ctx, mov_arg(ARG2, Pos); a.sar(ARG2, imm(_TAG_IMMED1_SIZE)); - a.mov(emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset)), ARG2); + a.mov(emit_boxed_val(ARG1, offsetof(ErlSubBits, start)), ARG2); } void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, @@ -789,6 +806,8 @@ void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, const ArgRegister &Dst) { unsigned unit = Unit.get(); + /* The division below clobbers RAX:RDX, place the context in ARG1 which is + * neither on all supported platforms. */ mov_arg(ARG1, Ctx); emit_gc_test_preserve(ArgWord(BUILD_SUB_BITSTRING_HEAP_NEED), @@ -796,11 +815,10 @@ void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, Ctx, ARG1); - a.mov(RET, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.size))); - a.sub(RET, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); + a.mov(RET, emit_boxed_val(ARG1, offsetof(ErlSubBits, end))); + a.sub(RET, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); if ((unit & (unit - 1))) { - /* Clobbers ARG3 */ a.cqo(); mov_imm(ARG4, unit); a.div(ARG4); @@ -813,7 +831,7 @@ void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, emit_enter_runtime(); - a.lea(ARG2, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb))); + a.lea(ARG2, emit_boxed_val(ARG1)); a.mov(ARG1, c_p); runtime_call<2>(erts_bs_get_binary_all_2); @@ -825,18 +843,19 @@ void BeamModuleAssembler::emit_i_bs_get_binary_all2(const ArgRegister &Ctx, void BeamGlobalAssembler::emit_bs_get_tail_shared() { emit_enter_runtime(); - a.mov(ARG2, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.orig))); + a.mov(ARG2, emit_boxed_val(ARG1, offsetof(ErlSubBits, orig))); a.mov(ARG3, ARG2); /* ARG2 = tag bits of mb.orig, ARG3 = mb.orig without tag bits */ a.and_(ARG2, imm(TAG_PTR_MASK__)); a.and_(ARG3, imm(~TAG_PTR_MASK__)); - a.mov(ARG4, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.base))); - a.mov(ARG5, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); + a.mov(ARG4, emit_boxed_val(ARG1, offsetof(ErlSubBits, base_flags))); + a.and_(ARG4, imm(~ERL_SUB_BITS_FLAG_MASK)); + a.mov(ARG5, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); - /* Extracted size = mb->size - mb->offset */ - a.mov(ARG6, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.size))); + /* Extracted size = end - start */ + a.mov(ARG6, emit_boxed_val(ARG1, offsetof(ErlSubBits, end))); a.sub(ARG6, ARG5); a.lea(ARG1, x86::qword_ptr(c_p, offsetof(Process, htop))); @@ -867,11 +886,11 @@ void BeamModuleAssembler::emit_bs_skip_bits(const ArgLabel &Fail, const ArgRegister &Ctx) { mov_arg(ARG1, Ctx); - a.add(RET, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); - a.cmp(RET, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.size))); + a.add(RET, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); + a.cmp(RET, emit_boxed_val(ARG1, offsetof(ErlSubBits, end))); a.ja(resolve_beam_label(Fail)); - a.mov(emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset)), RET); + a.mov(emit_boxed_val(ARG1, offsetof(ErlSubBits, start)), RET); } void BeamModuleAssembler::emit_i_bs_skip_bits2(const ArgRegister &Ctx, @@ -922,7 +941,7 @@ void BeamModuleAssembler::emit_i_bs_get_binary2(const ArgRegister &Ctx, a.mov(ARG1, c_p); a.mov(ARG2, TMP_MEM1q); mov_imm(ARG3, Flags.get()); - a.lea(ARG4, emit_boxed_val(ARG4, offsetof(ErlBinMatchState, mb))); + a.sub(ARG4, imm(TAG_PRIMARY_BOXED)); runtime_call<4>(erts_bs_get_binary_2); emit_leave_runtime(); @@ -954,8 +973,9 @@ void BeamModuleAssembler::emit_i_bs_get_float2(const ArgRegister &Ctx, emit_enter_runtime(); a.mov(ARG1, c_p); + /* ARG2 set above */ mov_imm(ARG3, Flags.get()); - a.lea(ARG4, emit_boxed_val(ARG4, offsetof(ErlBinMatchState, mb))); + a.sub(ARG4, imm(TAG_PRIMARY_BOXED)); runtime_call<4>(erts_bs_get_float_2); emit_leave_runtime(); @@ -1029,7 +1049,7 @@ void BeamModuleAssembler::emit_i_bs_put_utf8(const ArgLabel &Fail, * return value. */ void BeamGlobalAssembler::emit_bs_get_utf8_short_shared() { - const int position_offset = offsetof(ErlBinMatchBuffer, offset); + const int start_offset = offsetof(ErlSubBits, start); const x86::Gp ctx = ARG1; const x86::Gp bin_position = ARG2; @@ -1151,7 +1171,7 @@ void BeamGlobalAssembler::emit_bs_get_utf8_short_shared() { /* Handle plain old ASCII (code point < 128). */ a.bind(ascii); - a.add(x86::qword_ptr(ctx, position_offset), imm(8)); + a.add(x86::qword_ptr(ctx, start_offset), imm(8)); a.shr(RET, imm(56 - _TAG_IMMED1_SIZE)); a.or_(RET, imm(_TAG_IMMED1_SMALL)); /* Always clears ZF. */ a.ret(); @@ -1305,7 +1325,7 @@ void BeamGlobalAssembler::emit_bs_get_utf8_shared() { a.cmp(byte_count_d, imm(2)); a.ja(error); - a.mov(x86::qword_ptr(ARG1, offsetof(ErlBinMatchBuffer, offset)), ARG3); + a.mov(x86::qword_ptr(ARG1, offsetof(ErlSubBits, start)), ARG3); a.shl(RETd, imm(_TAG_IMMED1_SIZE)); a.or_(RETd, imm(_TAG_IMMED1_SMALL)); /* Always clears ZF. */ @@ -1322,9 +1342,9 @@ void BeamGlobalAssembler::emit_bs_get_utf8_shared() { void BeamModuleAssembler::emit_bs_get_utf8(const ArgRegister &Ctx, const ArgLabel &Fail) { - const int base_offset = offsetof(ErlBinMatchBuffer, base); - const int position_offset = offsetof(ErlBinMatchBuffer, offset); - const int size_offset = offsetof(ErlBinMatchBuffer, size); + const int base_offset = offsetof(ErlSubBits, base_flags); + const int start_offset = offsetof(ErlSubBits, start); + const int end_offset = offsetof(ErlSubBits, end); const x86::Gp ctx = ARG1; const x86::Gp bin_position = ARG2; @@ -1334,11 +1354,12 @@ void BeamModuleAssembler::emit_bs_get_utf8(const ArgRegister &Ctx, check = a.newLabel(), done = a.newLabel(); mov_arg(ctx, Ctx); - a.lea(ctx, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb))); - a.mov(bin_position, x86::qword_ptr(ctx, position_offset)); - a.mov(RET, x86::qword_ptr(ctx, size_offset)); + a.sub(ctx, imm(TAG_PRIMARY_BOXED)); + a.mov(bin_position, x86::qword_ptr(ctx, start_offset)); + a.mov(RET, x86::qword_ptr(ctx, end_offset)); a.mov(bin_base, x86::qword_ptr(ctx, base_offset)); + a.and_(bin_base, imm(~ERL_SUB_BITS_FLAG_MASK)); a.sub(RET, bin_position); a.cmp(RET, imm(32)); a.short_().jb(fallback); @@ -1362,7 +1383,7 @@ void BeamModuleAssembler::emit_bs_get_utf8(const ArgRegister &Ctx, a.short_().js(multi_byte); /* Handle plain old ASCII (code point < 128). */ - a.add(x86::qword_ptr(ctx, position_offset), imm(8)); + a.add(x86::qword_ptr(ctx, start_offset), imm(8)); a.shr(RETd, imm(24 - _TAG_IMMED1_SIZE)); a.or_(RETd, imm(_TAG_IMMED1_SMALL)); a.short_().jmp(done); @@ -1459,7 +1480,7 @@ void BeamModuleAssembler::emit_bs_get_utf16(const ArgRegister &Ctx, emit_enter_runtime(); - a.lea(ARG1, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb))); + a.sub(ARG1, imm(TAG_PRIMARY_BOXED)); mov_imm(ARG2, Flags.get()); runtime_call<2>(erts_bs_get_utf16); @@ -1536,8 +1557,7 @@ void BeamModuleAssembler::emit_i_bs_validate_unicode_retract( { mov_arg(ARG1, Ms); - a.sub(emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset)), - imm(32)); + a.sub(emit_boxed_val(ARG1, offsetof(ErlSubBits, start)), imm(32)); if (Fail.get() != 0) { a.jmp(resolve_beam_label(Fail)); @@ -1556,8 +1576,8 @@ void BeamModuleAssembler::emit_bs_test_unit(const ArgLabel &Fail, mov_arg(ARG1, Ctx); - a.mov(RET, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.size))); - a.sub(RET, emit_boxed_val(ARG1, offsetof(ErlBinMatchState, mb.offset))); + a.mov(RET, emit_boxed_val(ARG1, offsetof(ErlSubBits, end))); + a.sub(RET, emit_boxed_val(ARG1, offsetof(ErlSubBits, start))); if ((unit & (unit - 1))) { /* Clobbers ARG3 */ @@ -2564,13 +2584,7 @@ void BeamModuleAssembler::emit_i_bs_create_bin(const ArgLabel &Fail, runtime_entered = bs_maybe_enter_runtime(runtime_entered); mov_arg(ARG1, seg.src); - if (exact_type(seg.src)) { - x86::Gp boxed_ptr = emit_ptr_val(ARG1, ARG1); - - comment("optimized size code because the value is always a " - "bitstring"); - a.add(sizeReg, emit_boxed_val(boxed_ptr, sizeof(Eterm))); - } else { + if (!exact_type(seg.src)) { /* Note: ARG1 must remain equal to seg.src here for error * reporting to work. */ if (Fail.get() == 0) { @@ -2582,15 +2596,48 @@ void BeamModuleAssembler::emit_i_bs_create_bin(const ArgLabel &Fail, } emit_is_boxed(error, seg.src, ARG1); + } + + Label next = a.newLabel(), not_sub_bits = a.newLabel(); + + x86::Gp boxed_ptr = emit_ptr_val(ARG3, ARG1); + a.mov(RETd, emit_boxed_val(boxed_ptr, 0, sizeof(Uint32))); + + /* Speculatively load heap bits size, this is safe since all boxed + * terms have at least one data word. */ + ERTS_CT_ASSERT(offsetof(ErlHeapBits, size) == sizeof(Eterm)); + a.mov(ARG2d, + emit_boxed_val(boxed_ptr, + offsetof(ErlHeapBits, size), + sizeof(Uint32))); + + a.cmp(RETd, imm(HEADER_SUB_BITS)); + a.short_().jne(not_sub_bits); + { + a.mov(ARG2, + emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, end))); + a.sub(ARG2, + emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, start))); + + if (masked_types(seg.src) == + BeamTypeId::Bitstring) { + comment("optimized size code because the value is always " + "a bitstring when boxed"); + } else { + a.short_().jmp(next); + } + } + a.bind(not_sub_bits); - x86::Gp boxed_ptr = emit_ptr_val(ARG2, ARG1); - a.mov(RETd, emit_boxed_val(boxed_ptr, 0, sizeof(Uint32))); + if (masked_types(seg.src) != + BeamTypeId::Bitstring) { a.and_(RETb, imm(_BITSTRING_TAG_MASK)); a.cmp(RETb, imm(_TAG_HEADER_HEAP_BITS)); a.jne(error); - - a.add(sizeReg, emit_boxed_val(boxed_ptr, sizeof(Eterm))); } + + a.bind(next); + a.add(sizeReg, ARG2); } else if (seg.unit != 0) { bool can_fail = true; comment("size binary/integer/float/string"); @@ -3988,10 +4035,10 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, ArgWord const &Need, ArgWord const &Live, Span const &List) { - const int orig_offset = offsetof(ErlBinMatchState, mb.orig); - const int base_offset = offsetof(ErlBinMatchState, mb.base); - const int position_offset = offsetof(ErlBinMatchState, mb.offset); - const int size_offset = offsetof(ErlBinMatchState, mb.size); + const int orig_offset = offsetof(ErlSubBits, orig); + const int base_offset = offsetof(ErlSubBits, base_flags); + const int start_offset = offsetof(ErlSubBits, start); + const int end_offset = offsetof(ErlSubBits, end); std::vector segments; @@ -4110,17 +4157,17 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, comment("ensure_at_least %ld %ld", size, seg.unit); mov_arg(ctx, Ctx); if (unit == 1) { - a.mov(bin_position, emit_boxed_val(ctx, position_offset)); + a.mov(bin_position, emit_boxed_val(ctx, start_offset)); a.lea(RET, qword_ptr(bin_position, size)); - a.cmp(RET, emit_boxed_val(ctx, size_offset)); + a.cmp(RET, emit_boxed_val(ctx, end_offset)); a.ja(resolve_beam_label(Fail)); } else if (size == 0 && next_instr_clobbers) { - a.mov(RET, emit_boxed_val(ctx, size_offset)); - a.sub(RET, emit_boxed_val(ctx, position_offset)); + a.mov(RET, emit_boxed_val(ctx, end_offset)); + a.sub(RET, emit_boxed_val(ctx, start_offset)); is_ctx_valid = is_position_valid = false; } else { - a.mov(RET, emit_boxed_val(ctx, size_offset)); - a.mov(bin_position, emit_boxed_val(ctx, position_offset)); + a.mov(RET, emit_boxed_val(ctx, end_offset)); + a.mov(bin_position, emit_boxed_val(ctx, start_offset)); a.sub(RET, bin_position); cmp(RET, size, tmp); a.jl(resolve_beam_label(Fail)); @@ -4152,12 +4199,12 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, comment("ensure_exactly %ld", size); mov_arg(ctx, Ctx); - a.mov(RET, emit_boxed_val(ctx, size_offset)); + a.mov(RET, emit_boxed_val(ctx, end_offset)); if (next_instr_clobbers) { - a.sub(RET, emit_boxed_val(ctx, position_offset)); + a.sub(RET, emit_boxed_val(ctx, start_offset)); is_ctx_valid = is_position_valid = false; } else { - a.mov(bin_position, emit_boxed_val(ctx, position_offset)); + a.mov(bin_position, emit_boxed_val(ctx, start_offset)); a.sub(RET, bin_position); is_ctx_valid = is_position_valid = true; } @@ -4214,11 +4261,12 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, is_ctx_valid = true; } if (!is_position_valid) { - a.mov(bin_position, emit_boxed_val(ctx, position_offset)); + a.mov(bin_position, emit_boxed_val(ctx, start_offset)); is_position_valid = true; } a.mov(bin_base, emit_boxed_val(ctx, base_offset)); - a.add(emit_boxed_val(ctx, position_offset), imm(seg.size)); + a.and_(bin_base, imm(~ERL_SUB_BITS_FLAG_MASK)); + a.add(emit_boxed_val(ctx, start_offset), imm(seg.size)); emit_read_bits(seg.size, bin_base, bin_position, bitdata); } @@ -4269,12 +4317,13 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, is_ctx_valid = true; } if (!is_position_valid) { - a.mov(bin_position, emit_boxed_val(ctx, position_offset)); + a.mov(bin_position, emit_boxed_val(ctx, start_offset)); is_position_valid = true; } a.mov(bin_base, emit_boxed_val(ctx, base_offset)); - a.add(emit_boxed_val(ctx, position_offset), imm(seg.size)); + a.and_(bin_base, imm(~ERL_SUB_BITS_FLAG_MASK)); + a.add(emit_boxed_val(ctx, start_offset), imm(seg.size)); emit_read_integer(bin_base, bin_position, tmp, flags, bits, Dst); is_position_valid = false; @@ -4287,11 +4336,11 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, comment("get integer %ld", bits); if (!is_ctx_valid) { - mov_arg(ctx, Ctx); + mov_arg(ARG4, Ctx); + } else if (ctx != ARG4) { + a.mov(ARG4, ctx); } - a.lea(ARG4, emit_boxed_val(ctx, offsetof(ErlBinMatchState, mb))); - if (bits >= SMALL_BITS) { emit_enter_runtime(); @@ -4302,7 +4351,7 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, a.mov(ARG1, c_p); a.mov(ARG2, bits); a.mov(ARG3, flags); - /* ARG4 set above */ + a.sub(ARG4, imm(TAG_PRIMARY_BOXED)); runtime_call<4>(erts_bs_get_integer_2); if (bits >= SMALL_BITS) { @@ -4331,9 +4380,10 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, a.and_(ARG2, imm(TAG_PTR_MASK__)); a.and_(ARG3, imm(~TAG_PTR_MASK__)); a.mov(ARG4, emit_boxed_val(RET, base_offset)); - a.mov(ARG5, emit_boxed_val(RET, position_offset)); + a.and_(ARG4, imm(~ERL_SUB_BITS_FLAG_MASK)); + a.mov(ARG5, emit_boxed_val(RET, start_offset)); mov_imm(ARG6, seg.size); - a.add(emit_boxed_val(RET, position_offset), ARG6); + a.add(emit_boxed_val(RET, start_offset), ARG6); runtime_call<6>(erts_build_sub_bitstring); @@ -4364,7 +4414,7 @@ void BeamModuleAssembler::emit_i_bs_match_test_heap(ArgLabel const &Fail, /* The compiler limits the size of any segment in a bs_match * instruction to 24 bits. */ ASSERT((seg.size >> 24) == 0); - a.add(emit_boxed_val(ctx, position_offset), imm(seg.size)); + a.add(emit_boxed_val(ctx, start_offset), imm(seg.size)); is_position_valid = false; break; } diff --git a/erts/emulator/beam/jit/x86/instr_common.cpp b/erts/emulator/beam/jit/x86/instr_common.cpp index 490cba278019..bc19949492c2 100644 --- a/erts/emulator/beam/jit/x86/instr_common.cpp +++ b/erts/emulator/beam/jit/x86/instr_common.cpp @@ -961,34 +961,42 @@ void BeamModuleAssembler::emit_is_bitstring(const ArgLabel &Fail, void BeamModuleAssembler::emit_is_binary(const ArgLabel &Fail, const ArgSource &Src) { + Label not_sub_bits = a.newLabel(); + mov_arg(ARG1, Src); emit_is_boxed(resolve_beam_label(Fail), Src, ARG1); x86::Gp boxed_ptr = emit_ptr_val(ARG1, ARG1); + ERTS_CT_ASSERT(offsetof(ErlHeapBits, size) == sizeof(Eterm)); + a.mov(RET, emit_boxed_val(boxed_ptr, offsetof(ErlHeapBits, size))); + + a.mov(ARG2d, emit_boxed_val(boxed_ptr)); + a.cmp(ARG2d, imm(HEADER_SUB_BITS)); + a.short_().jne(not_sub_bits); + { + a.mov(RET, emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, end))); + a.sub(RET, emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, start))); + } + a.bind(not_sub_bits); + + /* Shift out all but the lowest three bits from the size, leaving a + * non-zero value if it's not evenly divisible by 8. + * + * This is used to combine the size and header checks, where OR-ing the + * shifted size into the header word forces the check to fail when we have + * a non-binary bitstring. */ + ERTS_CT_ASSERT((7u << (32 - 3)) > _BITSTRING_TAG_MASK); + a.shl(RETd, imm(32 - 3)); + if (masked_types(Src) == BeamTypeId::Bitstring) { comment("skipped header test since we know it's a bitstring when " "boxed"); - ERTS_CT_ASSERT(offsetof(ErlHeapBits, size) == sizeof(Eterm)); - ERTS_CT_ASSERT(offsetof(ErlSubBits, size) == sizeof(Eterm)); - a.test(emit_boxed_val(boxed_ptr, sizeof(Eterm), 1), imm(7)); } else { - a.mov(RETd, emit_boxed_val(boxed_ptr)); - a.and_(RETd, imm(_BITSTRING_TAG_MASK)); - - /* Load the size in bits into ARG1d, then shift out all but the lowest - * three bits, leaving a non-zero value if the size is not evenly - * divisible by 8. - * - * Thus, OR-ing this value into the header word forces the check to - * fail when we have a non-binary bitstring. */ - a.mov(ARG1d, emit_boxed_val(boxed_ptr, sizeof(Eterm), sizeof(Uint32))); - a.shl(ARG1d, imm(32 - 3)); - - ERTS_CT_ASSERT((7u << (32 - 3)) > _BITSTRING_TAG_MASK); - a.or_(RETd, ARG1d); - a.cmp(RETd, imm(_TAG_HEADER_HEAP_BITS)); + a.and_(ARG2d, imm(_BITSTRING_TAG_MASK)); + a.or_(ARG2d, RETd); + a.cmp(ARG2d, imm(_TAG_HEADER_HEAP_BITS)); } a.jne(resolve_beam_label(Fail)); @@ -1396,20 +1404,6 @@ void BeamModuleAssembler::emit_i_test_arity(const ArgLabel &Fail, void BeamModuleAssembler::emit_is_eq_exact(const ArgLabel &Fail, const ArgSource &X, const ArgSource &Y) { - if (exact_type(X) && Y.isLiteral()) { - Eterm literal = beamfile_get_literal(beam, Y.as().get()); - - if (is_bitstring(literal) && bitstring_size(literal) == 0) { - mov_arg(RET, X); - - x86::Gp boxed_ptr = emit_ptr_val(RET, RET); - - comment("simplified equality test with empty bitstring"); - a.cmp(emit_boxed_val(boxed_ptr, sizeof(Eterm)), 0); - a.jne(resolve_beam_label(Fail)); - } - } - /* If one argument is known to be an immediate, we can fail * immediately if they're not equal. */ if (X.isRegister() && always_immediate(Y)) { @@ -1474,22 +1468,6 @@ void BeamModuleAssembler::emit_is_eq_exact(const ArgLabel &Fail, void BeamModuleAssembler::emit_is_ne_exact(const ArgLabel &Fail, const ArgSource &X, const ArgSource &Y) { - if (exact_type(X) && Y.isLiteral()) { - Eterm literal = beamfile_get_literal(beam, Y.as().get()); - - if (is_bitstring(literal) && bitstring_size(literal) == 0) { - mov_arg(RET, X); - - x86::Gp boxed_ptr = emit_ptr_val(RET, RET); - - comment("simplified non-equality test with empty bitstring"); - a.cmp(emit_boxed_val(boxed_ptr, sizeof(Eterm)), 0); - a.je(resolve_beam_label(Fail)); - - return; - } - } - /* If one argument is known to be an immediate, we can fail * immediately if they're equal. */ if (X.isRegister() && always_immediate(Y)) { diff --git a/erts/emulator/beam/jit/x86/instr_fun.cpp b/erts/emulator/beam/jit/x86/instr_fun.cpp index e621d2529fd5..fe825ce1e75f 100644 --- a/erts/emulator/beam/jit/x86/instr_fun.cpp +++ b/erts/emulator/beam/jit/x86/instr_fun.cpp @@ -189,28 +189,24 @@ void BeamModuleAssembler::emit_i_make_fun3(const ArgLambda &Lambda, const ArgWord &Arity, const ArgWord &NumFree, const Span &env) { - size_t num_free = env.size(); + ASSERT((NumFree.get() + 1) == env.size() && + (NumFree.get() + Arity.get()) < MAX_ARG); - ASSERT(NumFree.get() == num_free); + mov_arg(RET, Lambda); - mov_arg(ARG2, Lambda); - mov_arg(ARG3, Arity); - mov_arg(ARG4, NumFree); - - emit_enter_runtime(); - - a.mov(ARG1, c_p); - runtime_call<4>(erts_new_local_fun_thing); - - emit_leave_runtime(); + comment("Create fun thing"); + a.mov(x86::qword_ptr(HTOP, offsetof(ErlFunThing, thing_word)), + imm(MAKE_FUN_HEADER(Arity.get(), NumFree.get(), 0))); + a.mov(x86::qword_ptr(HTOP, offsetof(ErlFunThing, entry.fun)), RET); comment("Move fun environment"); - for (unsigned i = 0; i < num_free; i++) { - const ArgVal &next = i + 1 < num_free ? env[i + 1] : ArgNil(); + for (Uint i = 0; i < env.size(); i++) { + const ArgVal &next = (i + 1) < env.size() ? env[i + 1] : ArgNil(); + switch (ArgVal::memory_relation(env[i], next)) { case ArgVal::Relation::consecutive: { x86::Mem src_ptr = getArgRef(env[i].as(), 16); - x86::Mem dst_ptr = x86::xmmword_ptr(RET, + x86::Mem dst_ptr = x86::xmmword_ptr(HTOP, offsetof(ErlFunThing, env) + i * sizeof(Eterm)); comment("(moving two items)"); @@ -224,7 +220,7 @@ void BeamModuleAssembler::emit_i_make_fun3(const ArgLambda &Lambda, goto fallback; } x86::Mem src_ptr = getArgRef(env[i + 1].as(), 16); - x86::Mem dst_ptr = x86::xmmword_ptr(RET, + x86::Mem dst_ptr = x86::xmmword_ptr(HTOP, offsetof(ErlFunThing, env) + i * sizeof(Eterm)); comment("(moving and swapping two items)"); @@ -235,7 +231,7 @@ void BeamModuleAssembler::emit_i_make_fun3(const ArgLambda &Lambda, } case ArgVal::Relation::none: fallback: - mov_arg(x86::qword_ptr(RET, + mov_arg(x86::qword_ptr(HTOP, offsetof(ErlFunThing, env) + i * sizeof(Eterm)), env[i]); @@ -244,7 +240,8 @@ void BeamModuleAssembler::emit_i_make_fun3(const ArgLambda &Lambda, } comment("Create boxed ptr"); - a.or_(RETb, TAG_PRIMARY_BOXED); + a.lea(RET, x86::qword_ptr(HTOP, TAG_PRIMARY_BOXED)); + a.add(HTOP, imm((ERL_FUN_SIZE + env.size()) * sizeof(Eterm))); mov_arg(Dst, RET); } diff --git a/erts/emulator/beam/jit/x86/instr_guard_bifs.cpp b/erts/emulator/beam/jit/x86/instr_guard_bifs.cpp index b44e5900948c..afaeee6fddbc 100644 --- a/erts/emulator/beam/jit/x86/instr_guard_bifs.cpp +++ b/erts/emulator/beam/jit/x86/instr_guard_bifs.cpp @@ -209,8 +209,17 @@ void BeamModuleAssembler::emit_bif_bit_size(const ArgWord &Bif, x86::Gp boxed_ptr = emit_ptr_val(ARG2, ARG2); ERTS_CT_ASSERT(offsetof(ErlHeapBits, size) == sizeof(Eterm)); - ERTS_CT_ASSERT(offsetof(ErlSubBits, size) == sizeof(Eterm)); a.mov(ARG1, emit_boxed_val(boxed_ptr, sizeof(Eterm))); + + Label not_sub_bits = a.newLabel(); + a.cmp(emit_boxed_val(boxed_ptr), imm(HEADER_SUB_BITS)); + a.short_().jne(not_sub_bits); + { + a.mov(ARG1, emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, end))); + a.sub(ARG1, emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, start))); + } + a.bind(not_sub_bits); + a.shl(ARG1, imm(_TAG_IMMED1_SIZE)); a.or_(ARG1, imm(_TAG_IMMED1_SMALL)); @@ -237,8 +246,16 @@ void BeamModuleAssembler::emit_bif_byte_size(const ArgWord &Bif, x86::Gp boxed_ptr = emit_ptr_val(ARG2, ARG2); ERTS_CT_ASSERT(offsetof(ErlHeapBits, size) == sizeof(Eterm)); - ERTS_CT_ASSERT(offsetof(ErlSubBits, size) == sizeof(Eterm)); - a.mov(ARG1, emit_boxed_val(boxed_ptr, sizeof(Eterm))); + a.mov(ARG1, emit_boxed_val(boxed_ptr, offsetof(ErlHeapBits, size))); + + Label not_sub_bits = a.newLabel(); + a.cmp(emit_boxed_val(boxed_ptr), imm(HEADER_SUB_BITS)); + a.short_().jne(not_sub_bits); + { + a.mov(ARG1, emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, end))); + a.sub(ARG1, emit_boxed_val(boxed_ptr, offsetof(ErlSubBits, start))); + } + a.bind(not_sub_bits); /* Round up to the nearest byte. */ a.add(ARG1, imm(7)); diff --git a/erts/emulator/beam/jit/x86/ops.tab b/erts/emulator/beam/jit/x86/ops.tab index b835114b2aa6..8969eba81ba9 100644 --- a/erts/emulator/beam/jit/x86/ops.tab +++ b/erts/emulator/beam/jit/x86/ops.tab @@ -260,7 +260,7 @@ i_get_tuple_element Tuple Pos Tuple2 | current_tuple Tuple3 | equal(Tuple, Tuple2) | equal(Tuple, Tuple3) => i_get_tuple_element Tuple Pos Tuple -# This is a current_tuple instruction instruction not followed by +# This is a current_tuple instruction not followed by # get_tuple_element. Invalidate the current tuple pointer. current_tuple Tuple => _ diff --git a/erts/emulator/beam/utils.c b/erts/emulator/beam/utils.c index 6823e378a2df..e3bd8df1f88e 100644 --- a/erts/emulator/beam/utils.c +++ b/erts/emulator/beam/utils.c @@ -1177,9 +1177,8 @@ int eq(Eterm a, Eterm b) ++bb; goto term_array; } - case BIN_REF_SUBTAG: - case HEAP_BITS_SUBTAG: - case SUB_BITS_SUBTAG: + case HEAP_BITS_SUBTAG: + case SUB_BITS_SUBTAG: { Uint a_offset, a_size; byte* a_base; @@ -1203,6 +1202,38 @@ int eq(Eterm a, Eterm b) break; /* not equal */ } +#ifdef DEBUG + /* When copying terms, the debug emulator may check that the copy + * is strictly equal to the source. As non-term heap objects may be + * copied under certain circumstances (e.g. naked fun references to + * a module literal area), we'll add a conservative implementation + * to cover direct equality checks of non-term heap objects. + * + * Note that we only need this to handle `eq(FunRef, FunRef)` and + * the like: we do not visit the FunRef or BinRef of any term we + * see while testing equality, so we should never land here under + * under normal circumstances. */ + case BIN_REF_SUBTAG: + if (is_bin_ref(b)) { + BinRef *r1 = (BinRef*)boxed_val(a); + BinRef *r2 = (BinRef*)boxed_val(b); + + if (r1->val == r2->val) { + goto pop_next; + } + } + break; /* not equal */ + case FUN_REF_SUBTAG: + if (is_fun_ref(b)) { + FunRef *r1 = (FunRef*)boxed_val(a); + FunRef *r2 = (FunRef*)boxed_val(b); + + if (r1->entry == r2->entry) { + goto pop_next; + } + } + break; /* not equal */ +#endif case FUN_SUBTAG: { ErlFunThing* f1; @@ -1220,18 +1251,12 @@ int eq(Eterm a, Eterm b) } if (is_local_fun(f1) && is_local_fun(f2)) { - ErlFunEntry *fe1, *fe2; - - fe1 = f1->entry.fun; - fe2 = f2->entry.fun; - - if (fe1->module != fe2->module || - fe1->index != fe2->index || - fe1->old_uniq != fe2->old_uniq) { + if (f1->entry.fun != f2->entry.fun) { goto not_equal; } - if ((sz = fun_num_free(f1)) == 0) { + sz = fun_num_free(f1); + if (sz == 0) { goto pop_next; } @@ -2036,7 +2061,6 @@ Sint erts_cmp_compound(Eterm a, Eterm b, int exact, int eq_only) goto mixed_types; } ON_CMP_GOTO(big_comp(a, b)); - case (_TAG_HEADER_FUN >> _TAG_PRIMARY_SIZE): if (is_not_any_fun(b)) { a_tag = FUN_DEF; @@ -2051,20 +2075,22 @@ Sint erts_cmp_compound(Eterm a, Eterm b, int exact, int eq_only) Sint diff; - diff = erts_cmp_atoms(fe1->module, (fe2)->module); + if (fe1 != fe2) { + diff = erts_cmp_atoms(fe1->module, (fe2)->module); - if (diff != 0) { - RETURN_NEQ(diff); - } + if (diff != 0) { + RETURN_NEQ(diff); + } - diff = fe1->index - fe2->index; - if (diff != 0) { - RETURN_NEQ(diff); - } + diff = fe1->index - fe2->index; + if (diff != 0) { + RETURN_NEQ(diff); + } - diff = fe1->old_uniq - fe2->old_uniq; - if (diff != 0) { - RETURN_NEQ(diff); + diff = fe1->old_uniq - fe2->old_uniq; + if (diff != 0) { + RETURN_NEQ(diff); + } } diff = fun_num_free(f1) - fun_num_free(f2); diff --git a/erts/emulator/internal_doc/AutomaticYieldingOfCCode.md b/erts/emulator/internal_doc/AutomaticYieldingOfCCode.md index 9ee24595d8e8..e20cc3de75ae 100644 --- a/erts/emulator/internal_doc/AutomaticYieldingOfCCode.md +++ b/erts/emulator/internal_doc/AutomaticYieldingOfCCode.md @@ -106,7 +106,7 @@ belong. For example, the functions for the map BIFs are placed in `erl_map.c` together with the other map-related functions. When building, YCF is invoked to generate the transformed versions of the functions into a header file that is included in the source file that -contains the the non-transformed version of the function (search for +contains the non-transformed version of the function (search for YCF in `$ERL_TOP/erts/emulator/Makefile.in` to see examples of how YCF can be invoked). diff --git a/erts/emulator/internal_doc/PTables.md b/erts/emulator/internal_doc/PTables.md index 6b316eaa7e76..dbb811a2400c 100644 --- a/erts/emulator/internal_doc/PTables.md +++ b/erts/emulator/internal_doc/PTables.md @@ -238,7 +238,7 @@ When we insert a new element in the table we do the following: be done before we publish the structure in the table. This, for example, includes storing the identifier in the process structure. -6. Now we can publish the structure in the table by writing the the +6. Now we can publish the structure in the table by writing the pointer to the process structure in the slot previously reserved in 3. diff --git a/erts/emulator/internal_doc/Tracing.md b/erts/emulator/internal_doc/Tracing.md index 28c973c26409..d270b7f2180c 100644 --- a/erts/emulator/internal_doc/Tracing.md +++ b/erts/emulator/internal_doc/Tracing.md @@ -153,7 +153,7 @@ through when adding a new breakpoint. 9. Wait for thread progress. 10. Prepare for next call to `trace_pattern` by updating the new staging part - (the old active) of the breakpoint to be identic to the the new active part. + (the old active) of the breakpoint to be identic to the new active part. 11. Release code modification permission and return from `trace_pattern`. diff --git a/erts/emulator/internal_doc/beam_makeops.md b/erts/emulator/internal_doc/beam_makeops.md index 9c127517dc34..64b81262da53 100644 --- a/erts/emulator/internal_doc/beam_makeops.md +++ b/erts/emulator/internal_doc/beam_makeops.md @@ -1071,8 +1071,8 @@ Here is first an example how it is used: is_map Fail Lit=q | literal_is_map(Lit) => _ If the `Lit` operand is a literal, then the `literal_is_map()` -predicate is called to determine wheter is is a map literal. -It it is, the instruction is not needed and can be removed. +predicate is called to determine whether it is a map literal. +If it is, the instruction is not needed and can be removed. `literal_is_map()` is implemented like this (in `emu/predicates.tab`): @@ -1682,7 +1682,7 @@ similar to this: #### Variable number of operands #### -Here follows an example of how to to handle an instruction with a variable number +Here follows an example of how to handle an instruction with a variable number of operands for the interpreter. Here is the instruction definition in `emu/ops.tab`: put_tuple2 xy I * diff --git a/erts/emulator/nifs/common/prim_net_nif.c b/erts/emulator/nifs/common/prim_net_nif.c index 299b96b60572..8655c561e394 100644 --- a/erts/emulator/nifs/common/prim_net_nif.c +++ b/erts/emulator/nifs/common/prim_net_nif.c @@ -3648,7 +3648,7 @@ void encode_adapter_index_map(ErlNifEnv* env, * nif_get_ip_address_table * * Description: - * Get ip address table table. + * Get ip address table. * This is a windows only function! * * Active Interfaces? diff --git a/erts/emulator/nifs/common/zlib_nif.c b/erts/emulator/nifs/common/zlib_nif.c index 92df1f48e10d..dad50f1c2f2b 100644 --- a/erts/emulator/nifs/common/zlib_nif.c +++ b/erts/emulator/nifs/common/zlib_nif.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson 2017-2021. All Rights Reserved. + * Copyright Ericsson 2017-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/pcre/pcre-8.44.tar.bz2 b/erts/emulator/pcre/pcre-8.44.tar.bz2 deleted file mode 100644 index dc978b77a7bb..000000000000 Binary files a/erts/emulator/pcre/pcre-8.44.tar.bz2 and /dev/null differ diff --git a/erts/emulator/sys/common/erl_mseg.c b/erts/emulator/sys/common/erl_mseg.c index 75d84758c6ba..98f45e0f994a 100644 --- a/erts/emulator/sys/common/erl_mseg.c +++ b/erts/emulator/sys/common/erl_mseg.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2002-2022. All Rights Reserved. + * Copyright Ericsson AB 2002-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/sys/common/erl_osenv.h b/erts/emulator/sys/common/erl_osenv.h index 9b592c050758..ca65e2b5ffa8 100644 --- a/erts/emulator/sys/common/erl_osenv.h +++ b/erts/emulator/sys/common/erl_osenv.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2017-2021. All Rights Reserved. + * Copyright Ericsson AB 2017-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -78,7 +78,7 @@ int erts_osenv_unset_term(erts_osenv_t *env, Eterm key); * @param value [in,out] The buffer to copy the value into, may be NULL if you * only wish to query presence. * - * @return 1 on success, 0 if the key couldn't be found, and -1 if if the value + * @return 1 on success, 0 if the key couldn't be found, and -1 if the value * didn't fit into the buffer. */ int erts_osenv_get_native(const erts_osenv_t *env, const erts_osenv_data_t *key, erts_osenv_data_t *value); diff --git a/erts/emulator/sys/unix/erl_child_setup.c b/erts/emulator/sys/unix/erl_child_setup.c index fcae10e3ee4d..72a7f77b2e6c 100644 --- a/erts/emulator/sys/unix/erl_child_setup.c +++ b/erts/emulator/sys/unix/erl_child_setup.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2002-2022. All Rights Reserved. + * Copyright Ericsson AB 2002-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/sys/unix/sys_drivers.c b/erts/emulator/sys/unix/sys_drivers.c index 493e5597bcff..9013a7475f30 100644 --- a/erts/emulator/sys/unix/sys_drivers.c +++ b/erts/emulator/sys/unix/sys_drivers.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 1996-2020. All Rights Reserved. + * Copyright Ericsson AB 1996-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/alloc_SUITE_data/migration.c b/erts/emulator/test/alloc_SUITE_data/migration.c index b51a8bd27b07..b419fc36ec9c 100644 --- a/erts/emulator/test/alloc_SUITE_data/migration.c +++ b/erts/emulator/test/alloc_SUITE_data/migration.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2014-2018. All Rights Reserved. + * Copyright Ericsson AB 2014-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/beam_SUITE.erl b/erts/emulator/test/beam_SUITE.erl index d3a0bf65e9cc..ad2ff6c8e8de 100644 --- a/erts/emulator/test/beam_SUITE.erl +++ b/erts/emulator/test/beam_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1998-2021. All Rights Reserved. +%% Copyright Ericsson AB 1998-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/big_SUITE.erl b/erts/emulator/test/big_SUITE.erl index 635abc880087..5e121e30c570 100644 --- a/erts/emulator/test/big_SUITE.erl +++ b/erts/emulator/test/big_SUITE.erl @@ -227,7 +227,7 @@ test_squaring(I) -> Sqr = I * I, %% This expression will be multiplied in the usual way, because - %% the the two operands for '*' are stored at different addresses. + %% the two operands for '*' are stored at different addresses. Sqr = I * ((I + id(1)) - id(1)), ok. diff --git a/erts/emulator/test/bs_bincomp_SUITE.erl b/erts/emulator/test/bs_bincomp_SUITE.erl index 5d74ff5b4199..64f1e8e8752b 100644 --- a/erts/emulator/test/bs_bincomp_SUITE.erl +++ b/erts/emulator/test/bs_bincomp_SUITE.erl @@ -141,7 +141,7 @@ random_binary() -> << <<($a + rand:uniform($z - $a)):8>> || _ <- Seq >>. random_binaries(N) when N > 0 -> - random_binary(), + 80 = bit_size(random_binary()), random_binaries(N - 1); random_binaries(_) -> ok. diff --git a/erts/emulator/test/bs_match_int_SUITE.erl b/erts/emulator/test/bs_match_int_SUITE.erl index 3a7ba43b2981..dc285dd80b4a 100644 --- a/erts/emulator/test/bs_match_int_SUITE.erl +++ b/erts/emulator/test/bs_match_int_SUITE.erl @@ -112,7 +112,7 @@ get_int(Bin0) -> %% Note that it has become impossible to create a byte-sized sub %% binary (see erts_build_sub_bitstring() in erl_bits.c) of size 64 %% or less. Therefore, to be able to create an unaligned binary, - %% we'll need to base it on on a binary with more than 64 bytes. + %% we'll need to base it on a binary with more than 64 bytes. Size = bit_size(Bin0), Filler = rand:bytes(65), UnsignedBigBin = id(<>), diff --git a/erts/emulator/test/bs_utf_SUITE.erl b/erts/emulator/test/bs_utf_SUITE.erl index ec15c1b312bb..973b99e71b26 100644 --- a/erts/emulator/test/bs_utf_SUITE.erl +++ b/erts/emulator/test/bs_utf_SUITE.erl @@ -67,7 +67,7 @@ do_utf8_roundtrip(First, Last) when First =< Last -> %% Here a heap binary and a sub binary will be allocated. If the %% write in the utf8 segment extends beyond the end of heap binary, - %% it will will overwrite the header for the sub binary. + %% it will overwrite the header for the sub binary. <<-1:(64-9)/signed,Bin/binary>> = id(<<-1:(64-9),First/utf8>>), <<-1:63/signed,Bin/binary>> = id(<<-1:63,First/utf8>>), diff --git a/erts/emulator/test/code_SUITE.erl b/erts/emulator/test/code_SUITE.erl index f74df5ca7133..b95c4074b953 100644 --- a/erts/emulator/test/code_SUITE.erl +++ b/erts/emulator/test/code_SUITE.erl @@ -589,7 +589,7 @@ constant_refc_binaries_test(Config) when is_list(Config) -> Bef = memory_binary(), io:format("Binary data (bytes) before test: ~p\n", [Bef]), - %% Compile the the literals module. + %% Compile the literals module. Data = proplists:get_value(data_dir, Config), File = filename:join(Data, "literals"), {ok,literals,Code} = compile:file(File, [report,binary]), @@ -858,7 +858,7 @@ t_copy_literals(Config) when is_list(Config) -> run_sys_proc_test(fun t_copy_literals_test/1, Config). t_copy_literals_test(Config) when is_list(Config) -> - %% Compile the the literals module. + %% Compile the literals module. Data = proplists:get_value(data_dir, Config), File = filename:join(Data, "literals"), {ok,literals,Code} = compile:file(File, [report,binary]), diff --git a/erts/emulator/test/dirty_nif_SUITE.erl b/erts/emulator/test/dirty_nif_SUITE.erl index 5069dfa5b9ce..632d289c5708 100644 --- a/erts/emulator/test/dirty_nif_SUITE.erl +++ b/erts/emulator/test/dirty_nif_SUITE.erl @@ -520,7 +520,7 @@ dirty_nif_send_traced(Config) when is_list(Config) -> true = Time2 >= 1900, %% Make sure that the send trace is - %% in between an in and and out trace + %% in between an in and an out trace (fun F() -> %% We got an in trace, look for out or send {trace,Sndr,in,_} = recv_trace_from(Sndr), diff --git a/erts/emulator/test/distribution_SUITE.erl b/erts/emulator/test/distribution_SUITE.erl index 144f9ec3b8d9..a02988c90bd8 100644 --- a/erts/emulator/test/distribution_SUITE.erl +++ b/erts/emulator/test/distribution_SUITE.erl @@ -1162,7 +1162,7 @@ roundtrip(Term) -> exit(Term). %% Test that the smallest external term [] aka NIL can be sent to -%% another node node and back again. +%% another node and back again. nil_roundtrip(Config) when is_list(Config) -> process_flag(trap_exit, true), {ok, Peer, Node} = ?CT_PEER(), diff --git a/erts/emulator/test/erts_debug_SUITE.erl b/erts/emulator/test/erts_debug_SUITE.erl index 692b225152ad..b24b590fa2aa 100644 --- a/erts/emulator/test/erts_debug_SUITE.erl +++ b/erts/emulator/test/erts_debug_SUITE.erl @@ -82,7 +82,10 @@ test_size(Config) when is_list(Config) -> %% Fun environment size = 0 (the smallest fun possible) SimplestFun = fun() -> ok end, - FunSz0 = 3, + + %% 2 words for the fun, 1 word to point at the off-heap reference, and + %% 3 words for the off-heap reference itself. The actual on-heap size is 3. + FunSz0 = 6, FunSz0 = do_test_size(SimplestFun), %% Fun environment size = 1 @@ -95,8 +98,9 @@ test_size(Config) when is_list(Config) -> FunSz1 = do_test_size(fun() -> ConsCell1 end) - do_test_size(ConsCell1), - %% External funs are the same size as local ones without environment - FunSz0 = do_test_size(fun lists:sort/1), + %% External funs are always 2 words (they're also always stored off-heap, + %% so the effective size is zero). + 2 = do_test_size(fun lists:sort/1), Arch = 8 * erlang:system_info({wordsize, external}), case {Arch, do_test_size(mk_ext_pid({a@b, 1}, 17, 42))} of @@ -122,9 +126,9 @@ test_size(Config) when is_list(Config) -> {32, 18} -> ok; {64, 10} -> ok end, - 9 = do_test_size(<<0:(8*65)>>), % ErlSubBits + BinRef + 8 = do_test_size(<<0:(8*65)>>), % ErlSubBits + BinRef 3 = do_test_size(<<5:7>>), % ErlHeapBits - 9 = do_test_size(<<0:(8*80+1)>>), % ErlSubBits + BinRef + 8 = do_test_size(<<0:(8*80+1)>>), % ErlSubBits + BinRef %% Test shared data structures. do_test_size([ConsCell1|ConsCell1], diff --git a/erts/emulator/test/fun_SUITE.erl b/erts/emulator/test/fun_SUITE.erl index 113ceb3690ce..f86b77a650c5 100644 --- a/erts/emulator/test/fun_SUITE.erl +++ b/erts/emulator/test/fun_SUITE.erl @@ -508,13 +508,19 @@ bad_md5(Bad) -> {'EXIT',{badarg,_}} = (catch erlang:md5(Bad)). refc(Config) when is_list(Config) -> + %% As the fun entry is owned by the fun's shared reference holder and not + %% the fun itself, its reference count should be generally be unchanged + %% regardless of how many copies we create, and on which process we do so. + %% + %% Only certain operations that break the sharing of the literal reference + %% holder should have an impact. F1 = fun_factory(2), {refc,2} = erlang:fun_info(F1, refc), F2 = fun_factory(42), - {refc,3} = erlang:fun_info(F1, refc), + {refc,2} = erlang:fun_info(F1, refc), process_flag(trap_exit, true), - Pid = spawn_link(fun() -> {refc,4} = erlang:fun_info(F1, refc) end), + Pid = spawn_link(fun() -> {refc,2} = erlang:fun_info(F1, refc) end), receive {'EXIT',Pid,normal} -> ok; Other -> ct:fail({unexpected,Other}) @@ -522,13 +528,20 @@ refc(Config) when is_list(Config) -> process_flag(trap_exit, false), %% Wait to make sure that the process has terminated completely. receive after 1 -> ok end, - {refc,3} = erlang:fun_info(F1, refc), + {refc,2} = erlang:fun_info(F1, refc), + + %% Force a copy of the underlying reference holder by passing through the + %% external term format. + F3 = binary_to_term(term_to_binary(F1)), + 3 = fun_refc(F1), + 3 = fun_refc(F3), %% Garbage collect. Only the F2 fun will be left. 7 = F1(5), true = erlang:garbage_collect(), 40 = F2(-2), {refc,2} = erlang:fun_info(F2, refc), + ok. fun_factory(Const) -> diff --git a/erts/emulator/test/list_bif_SUITE.erl b/erts/emulator/test/list_bif_SUITE.erl index 2e43e9ad5a69..8951a3e806fa 100644 --- a/erts/emulator/test/list_bif_SUITE.erl +++ b/erts/emulator/test/list_bif_SUITE.erl @@ -60,10 +60,13 @@ t_list_to_integer(Config) when is_list(Config) -> {error,badarg} = string:to_integer($A), %% System limit. - Digits = lists:duplicate(11_000_000, $9), + Digits = lists:duplicate(3_000_000, $9), {'EXIT',{system_limit,_}} = catch list_to_integer(Digits), + _ = erlang:garbage_collect(), {'EXIT',{system_limit,_}} = catch list_to_integer(Digits, 16), + _ = erlang:garbage_collect(), {error,system_limit} = string:to_integer(Digits), + _ = erlang:garbage_collect(), ok. diff --git a/erts/emulator/test/map_SUITE.erl b/erts/emulator/test/map_SUITE.erl index e034714de21e..a7373541871f 100644 --- a/erts/emulator/test/map_SUITE.erl +++ b/erts/emulator/test/map_SUITE.erl @@ -2994,7 +2994,7 @@ t_maps_without(_Config) -> %% MISC -%% Verify that the the number of nodes in hashmaps +%% Verify that the number of nodes in hashmaps %% of different types and sizes does not deviate too %% much from the theoretical model. %% For debug with DBG_HASHMAP_COLLISION_BONANZA the test will expect diff --git a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c index 5ae97e529d54..fb7fe70dfe45 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_SUITE.c +++ b/erts/emulator/test/nif_SUITE_data/nif_SUITE.c @@ -2558,8 +2558,7 @@ static ERL_NIF_TERM binary_to_term(ErlNifEnv* env, int argc, const ERL_NIF_TERM /* build dummy heap term first to provoke OTP-15080 */ dummy = enif_make_list_cell(msg_env, atom_true, atom_false); - ret = enif_binary_to_term(msg_env, bin.data, bin.size, &term, - (ErlNifBinaryToTerm)opts); + ret = enif_binary_to_term(msg_env, bin.data, bin.size, &term, opts); if (!ret) return atom_false; diff --git a/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_drv_nif.h b/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_drv_nif.h index d3b163617c8e..939676399112 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_drv_nif.h +++ b/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_drv_nif.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2010-2017. All Rights Reserved. + * Copyright Ericsson AB 2010-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif.h b/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif.h index 09460120cbaa..4698305003f1 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif.h +++ b/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2009-2021. All Rights Reserved. + * Copyright Ericsson AB 2009-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif_api_funcs.h b/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif_api_funcs.h index 641ddb51a2b7..90495bbffe24 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif_api_funcs.h +++ b/erts/emulator/test/nif_SUITE_data/nif_api_2_0/erl_nif_api_funcs.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2009-2017. All Rights Reserved. + * Copyright Ericsson AB 2009-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_drv_nif.h b/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_drv_nif.h index d3b163617c8e..939676399112 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_drv_nif.h +++ b/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_drv_nif.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2010-2017. All Rights Reserved. + * Copyright Ericsson AB 2010-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif.h b/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif.h index 027187557343..a59d2e09ba2a 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif.h +++ b/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2009-2017. All Rights Reserved. + * Copyright Ericsson AB 2009-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif_api_funcs.h b/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif_api_funcs.h index 4fa2876070ea..8def252bd60d 100644 --- a/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif_api_funcs.h +++ b/erts/emulator/test/nif_SUITE_data/nif_api_2_4/erl_nif_api_funcs.h @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2009-2017. All Rights Reserved. + * Copyright Ericsson AB 2009-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/emulator/test/num_bif_SUITE.erl b/erts/emulator/test/num_bif_SUITE.erl index b986eb25b8e4..f0080223d5c5 100644 --- a/erts/emulator/test/num_bif_SUITE.erl +++ b/erts/emulator/test/num_bif_SUITE.erl @@ -673,10 +673,13 @@ t_string_to_integer(Config) when is_list(Config) -> ]), %% System limit - Digits = lists:duplicate(11_000_000, $9), + Digits = lists:duplicate(3_000_000, $9), {'EXIT',{system_limit,_}} = catch list_to_integer(Digits), + _ = erlang:garbage_collect(), {'EXIT',{system_limit,_}} = catch list_to_integer(Digits, 16), + _ = erlang:garbage_collect(), {error,system_limit} = string:to_integer(Digits), + _ = erlang:garbage_collect(), ok. diff --git a/erts/emulator/test/process_SUITE.erl b/erts/emulator/test/process_SUITE.erl index 403a6afceb58..cd096f1a2390 100644 --- a/erts/emulator/test/process_SUITE.erl +++ b/erts/emulator/test/process_SUITE.erl @@ -96,6 +96,8 @@ spawn_request_reply_option/1, dist_spawn_arg_list_mixup/1, alias_bif/1, + dist_frag_alias/1, + dist_frag_unaliased/1, monitor_alias/1, spawn_monitor_alias/1, demonitor_aliasmonitor/1, @@ -193,7 +195,8 @@ groups() -> otp_16436, otp_16642]}, {alias, [], [alias_bif, monitor_alias, spawn_monitor_alias, - demonitor_aliasmonitor, down_aliasmonitor]}]. + demonitor_aliasmonitor, down_aliasmonitor, + dist_frag_alias, dist_frag_unaliased]}]. init_per_suite(Config) -> A0 = case application:start(sasl) of @@ -5080,7 +5083,84 @@ alias_bif_test(Node) -> end), [{A3,1},{'DOWN', M3, _, _, _}] = recv_msgs(2), ok. - + +dist_frag_alias(Config) when is_list(Config) -> + Tester = self(), + {ok, Peer, Node} = ?CT_PEER(), + {P,M} = spawn_monitor(Node, + fun () -> + Alias = alias(), + Tester ! {alias, Alias}, + receive + {data, Data} -> + garbage_collect(), + Tester ! {received_data, Data} + end, + exit(end_of_test) + end), + Data = term_to_binary(lists:seq(1, 1000000)), + receive + {alias, Alias} -> + Alias ! {data, Data}, + receive + {received_data, RecvData} -> + Data = RecvData; + {'DOWN', M, process, P, R2} -> + ct:fail(R2) + end; + {'DOWN', M, process, P, R1} -> + ct:fail(R1) + end, + receive + {'DOWN', M, process, P, R3} -> + end_of_test = R3 + end, + peer:stop(Peer), + ok. + +dist_frag_unaliased(Config) when is_list(Config) -> + %% Leak fixed by PR-7915 would have been detected using asan or valgrind + %% when running this test... + Tester = self(), + {ok, Peer, Node} = ?CT_PEER(), + {P,M} = spawn_monitor(Node, + fun () -> + Alias = alias(), + Tester ! {alias, Alias}, + receive + {data, Data} -> + garbage_collect(), + unalias(Alias), + Tester ! {received_data, Data}, + receive + {data, _Data} -> + exit(received_data_again); + end_of_test -> + exit(end_of_test) + end + end + end), + Data = term_to_binary(lists:seq(1, 1000000)), + receive + {alias, Alias} -> + Alias ! {data, Data}, + receive + {received_data, RecvData} -> + Data = RecvData; + {'DOWN', M, process, P, R2} -> + ct:fail(R2) + end, + Alias ! {data, Data}, + P ! end_of_test; + {'DOWN', M, process, P, R1} -> + ct:fail(R1) + end, + receive + {'DOWN', M, process, P, R3} -> + end_of_test = R3 + end, + peer:stop(Peer), + ok. monitor_alias(Config) when is_list(Config) -> monitor_alias_test(node()), diff --git a/erts/emulator/test/property_test/phash2_properties.erl b/erts/emulator/test/property_test/phash2_properties.erl index f9f9e1941715..f6dea4fbad69 100644 --- a/erts/emulator/test/property_test/phash2_properties.erl +++ b/erts/emulator/test/property_test/phash2_properties.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2019-2019. All Rights Reserved. +%% Copyright Ericsson AB 2019-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/erts/etc/common/Makefile.in b/erts/etc/common/Makefile.in index 3c5a5cee64ba..42af9c87dca5 100644 --- a/erts/etc/common/Makefile.in +++ b/erts/etc/common/Makefile.in @@ -200,7 +200,7 @@ INSTALL_EMBEDDED_PROGS += \ INSTALL_EMBEDDED_DATA = $(UXETC)/start.src $(UXETC)/start_erl.src INSTALL_TOP = Install INSTALL_TOP_BIN = -INSTALL_MISC = $(UXETC)/format_man_pages $(UXETC)/makewhatis +INSTALL_MISC = $(UXETC)/format_man_pages INSTALL_SRC = $(UXETC)/setuid_socket_wrap.c #delivered as an example ERLEXECDIR = . INSTALL_LIBS = diff --git a/erts/etc/unix/RELNOTES b/erts/etc/unix/RELNOTES index 97aec2b10079..255d15be75a7 100644 --- a/erts/etc/unix/RELNOTES +++ b/erts/etc/unix/RELNOTES @@ -1,7 +1,7 @@ %CopyrightBegin% - Copyright Ericsson AB 1996-2021. All Rights Reserved. + Copyright Ericsson AB 1996-2023. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -236,7 +236,7 @@ the 4.2 system, but there is a number of incompatibilities. -- eprof, is a (tty-based) tool for real-time profiling, see eprof(3). --- dbg, is a (tty-based) interface to the the trace/3 BIF, see dbg(3). +-- dbg, is a (tty-based) interface to the trace/3 BIF, see dbg(3). -- pman, is a (pxw-based) interface to the trace/3 BIF. diff --git a/erts/etc/unix/etp-commands.in b/erts/etc/unix/etp-commands.in index a62f908b2a2a..783a6e22844f 100644 --- a/erts/etc/unix/etp-commands.in +++ b/erts/etc/unix/etp-commands.in @@ -2,7 +2,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2005-2022. All Rights Reserved. +# Copyright Ericsson AB 2005-2023. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -425,16 +425,20 @@ define etp-boxed-immediate-1 if ($etp_boxed_immediate_h == etp_fun_subtag) printf "#Fun<" else - if ($etp_boxed_immediate_h == etp_bin_ref_subtag) - printf "#BinRef<" + if ($etp_boxed_immediate_h == etp_fun_ref_subtag) + printf "#FunRef<" else - if ($etp_boxed_immediate_h == etp_heap_bits_subtag) - printf "#HeapBits<" - else - if ($etp_boxed_immediate_h == etp_sub_bits_subtag) - printf "#SubBits<" - else - printf "#Header%X<", $etp_boxed_immediate_h + if ($etp_boxed_immediate_h == etp_bin_ref_subtag) + printf "#BinRef<" + else + if ($etp_boxed_immediate_h == etp_heap_bits_subtag) + printf "#HeapBits<" + else + if ($etp_boxed_immediate_h == etp_sub_bits_subtag) + printf "#SubBits<" + else + printf "#Header%X<", $etp_boxed_immediate_h + end end end end @@ -4394,7 +4398,7 @@ document etp-search-alloc %--------------------------------------------------------------------------- % etp-search-heaps Eterm % -% Search all internal allocator memory blocks for for the specified Eterm. +% Search all internal allocator memory blocks for the specified Eterm. %--------------------------------------------------------------------------- end diff --git a/erts/etc/unix/etp.py b/erts/etc/unix/etp.py index 5fbc275a1799..6d9339fe57da 100644 --- a/erts/etc/unix/etp.py +++ b/erts/etc/unix/etp.py @@ -415,6 +415,8 @@ def boxed(valobj, depth = float('inf')): return '#Ref' if masked_hdr == c('FUN_SUBTAG'): return '#Fun' + if masked_hdr == c('FUN_REF_SUBTAG'): + return '#FunRef' if masked_hdr == c('HEAP_BITS_SUBTAG'): return '#HeapBits' if masked_hdr == c('SUB_BITS_SUBTAG'): diff --git a/erts/etc/unix/makewhatis b/erts/etc/unix/makewhatis deleted file mode 100644 index 0d047fe8c465..000000000000 --- a/erts/etc/unix/makewhatis +++ /dev/null @@ -1,327 +0,0 @@ -#!/bin/sh -# makewhatis: create the whatis database -# Created: Sun Jun 14 10:49:37 1992 -# Revised: Sat Jan 8 14:12:37 1994 by faith@cs.unc.edu -# Revised: Sat Mar 23 17:56:18 1996 by micheal@actrix.gen.nz -# Copyright 1992, 1993, 1994 Rickard E. Faith (faith@cs.unc.edu) -# May be freely distributed and modified as long as copyright is retained. -# -# Wed Dec 23 13:27:50 1992: Rik Faith (faith@cs.unc.edu) applied changes -# based on Mitchum DSouza (mitchum.dsouza@mrc-apu.cam.ac.uk) cat patches. -# Also, cleaned up code and make it work with NET-2 doc pages. -# -# makewhatis-1.4: aeb 940802, 941007, 950417 -# Fixed so that the -c option works correctly for the cat pages -# on my machine. Fix for -u by Nan Zou (nan@ksu.ksu.edu). -# Many minor changes. -# The -s option is undocumented, and may well disappear again. -# -# Sat Mar 23 1996: Michael Hamilton (michael@actrix.gen.nz). -# I changed the script to invoke gawk only once for each directory tree. -# This speeds things up considerably (from 30 minutes down to 1.5 minutes -# on my 486DX66). -# 960401 - aeb: slight adaptation to work correctly with cat pages. -# 960510 - added fixes by brennan@raven.ca.boeing.com, author of mawk. -# 971012 - replaced "test -z" - it doesn't work on SunOS 4.1.3_U1. -# 980710 - be more careful with TMPFILE -# -# Note for Slackware users: "makewhatis -v -w -c" will work. - -# %ExternalCopyright% -PATH=/usr/bin:/bin - -DEFMANPATH=/usr/man -DEFCATPATH=/usr/man/preformat:/usr/man - -# Find a place for our temporary files. If security is not a concern, use -# TMPFILE=/tmp/whatis$$; TMPFILEDIR=none -# Of course makewhatis should only have the required permissions -# (for reading and writing directories like /usr/man). -# We try here to be careful (and avoid preconstructed symlinks) -# in case makewhatis is run as root, by creating a subdirectory of /tmp. -# If that fails we use $HOME. -# The code below uses test -O which doesn't work on all systems. -TMPFILE=$HOME/whatis$$ -TMPFILEDIR=/tmp/whatis$$ -if [ ! -d $TMPFILEDIR ]; then - mkdir $TMPFILEDIR - chmod 0700 $TMPFILEDIR - if [ -O $TMPFILEDIR ]; then - TMPFILE=$TMPFILEDIR/w - fi -fi - -topath=manpath - -defmanpath=$DEFMANPATH -defcatpath= - -sections="1 2 3 4 5 6 7 8 9 n l" - -for name in $* -do -if [ -n "$setsections" ]; then - setsections= - sections=$name - continue -fi -case $name in - -c) topath=catpath - defmanpath= - defcatpath=$DEFCATPATH - continue;; - -s) setsections=1 - continue;; - -u) findarg="-ctime 0" - update=1 - continue;; - -v) verbose=1 - continue;; - -w) manpath=`man --path` - continue;; - -*) echo "Usage: makewhatis [-u] [-v] [-w] [manpath] [-c [catpath]]" - echo " This will build the whatis database for the man pages" - echo " found in manpath and the cat pages found in catpath." - echo " -u: update database with new pages" - echo " -v: verbose" - echo " -w: use manpath obtained from \`man --path\`" - echo " [manpath]: man directories (default: $DEFMANPATH)" - echo " [catpath]: cat directories (default: the first existing" - echo " directory in $DEFCATPATH)" - exit;; - *) if [ -d $name ] - then - eval $topath="\$$topath":$name - else - echo "No such directory $name" - exit - fi;; -esac -done - -manpath=`echo ${manpath-$defmanpath} | tr : ' '` -if [ x"$catpath" = x ]; then - for d in `echo $defcatpath | tr : ' '` - do - if [ -d $d ]; then catpath=$d; break; fi - done -fi -catpath=`echo ${catpath} | tr : ' '` - -# first truncate all the whatis files that will be created new, -# then only update - we might visit the same directory twice -if [ x$update = x ]; then - for pages in man cat - do - eval path="\$$pages"path - for mandir in $path - do - cp /dev/null $mandir/whatis - done - done -fi - -for pages in man cat -do - export pages - eval path="\$$pages"path - for mandir in $path - do - if [ x$verbose != x ]; then - echo "about to enter $mandir" > /dev/tty - fi - if [ -s ${mandir}/whatis -a $pages = man ]; then - if [ x$verbose != x ]; then - echo skipping $mandir - we did it already > /dev/tty - fi - else - here=`pwd` - cd $mandir - for i in $sections - do - if [ -d ${pages}$i ] - then - cd ${pages}$i - section=$i - export section verbose - find . -name '*' $findarg -print | /usr/bin/gawk ' - - function readline() { - if (use_zcat) { - result = (pipe_cmd | getline); - if (result < 0) { - print "Pipe error: " pipe_cmd " " ERRNO > "/dev/stderr"; - } - } else { - result = (getline < filename); - if (result < 0) { - print "Read file error: " filename " " ERRNO > "/dev/stderr"; - } - } - return result; - } - - function closeline() { - if (use_zcat) { - return close(pipe_cmd); - } else { - return close(filename); - } - } - - function do_one() { - after = 0; insh = 0; thisjoin = 1; charct = 0; - - if (verbose) { - print "adding " filename > "/dev/tty" - } - - use_zcat = (filename ~ /\.Z$/ || filename ~ /\.z$/ || - filename ~ /\.gz$/); - match(filename, "/[^/]+$"); - progname = substr(filename, RSTART + 1, RLENGTH - 1); - if (match(progname, "\\." section "[A-Za-z]+")) { - actual_section = substr(progname, RSTART + 1, RLENGTH - 1); - } else { - actual_section = section; - } - sub(/\..*/, "", progname); - if (use_zcat) { - pipe_cmd = "zcat " filename; - } - - while (readline() > 0) { - gsub(/.\b/, ""); - if (($1 ~ /^\.[Ss][Hh]/ && $2 ~ /[Nn][Aa][Mm][Ee]/) || - (pages == "cat" && $1 ~ /^NAME/)) { - if (!insh) - insh = 1; - else { - printf "\n"; - closeline(); - return; - } - } else if (insh) { - if ($1 ~ /^\.[Ss][HhYS]/ || - (pages == "cat" && - ($1 ~ /^S[yYeE]/ || $1 ~ /^DESCRIPTION/ || - $1 ~ /^COMMAND/ || $1 ~ /^OVERVIEW/ || - $1 ~ /^STRUCTURES/ || $1 ~ /^INTRODUCTION/))) { - # end insh for Synopsis, Syntax, but also for - # DESCRIPTION (e.g., XFree86.1x), - # COMMAND (e.g., xspread.1) - # OVERVIEW (e.g., TclCommandWriting.3) - # STRUCTURES (e.g., XEvent.3x) - # INTRODUCTION (e.g., TclX.n) - printf "\n"; - closeline(); - return; - } else { # derived from Tom Christiansen perl script - if (!after && $0 ~ progname"-") { # Fix old cat pages - sub(progname"-", progname" - "); - } - gsub(/ /, " "); # Translate tabs to spaces - gsub(/ +/, " "); # Collapse spaces - gsub(/ *, */, ", "); # Fix comma spacings - sub(/^ /, ""); # Kill initial spaces - sub(/ $/, ""); # Kill trailing spaces - sub(/__+/, "_"); # Collapse underscores - if ($0 ~ /[^ ]-$/) { - sub(/-$/, ""); # Handle Hyphenations - nextjoin = 1; - } else - nextjoin = 0; - sub(/^.[IB] /, ""); # Kill bold and italics - sub(/^.Nm /, ""); # Kill bold - sub(/^.Tn /, ""); # Kill normal - sub(/^.Li /, ""); # Kill .Li - sub(/^.Dq /, ""); # Kill .Dq - sub(/^.Nd */, "- "); # Convert .Nd to dash - gsub(/\\f[PRIB0123]/, ""); # Kill font changes - gsub(/\\s[-+0-9]*/, ""); # Kill size changes - gsub(/\\&/, ""); # Kill \& - gsub(/\\\((ru|ul)/, "_"); # Translate - gsub(/\\\((mi|hy|em)/, "-"); # Translate - gsub(/\\\*\(../, ""); # Kill troff strings - sub(/^\.\\\".*/, ""); # Kill comments - gsub(/\\/, ""); # Kill all backslashes - if ($1 ~ /^\.../ || $1 == "") { - if (after && !needmore) { - printf "\n"; - thisjoin = 1; - charct = 0; - after = 0; - } - } else { - if ($0 ~ /^- /) { - sub("- ", " - "); - } else if (!thisjoin && $0 !~ /^- /) { - printf " "; - charct += 1; - } - thisjoin = nextjoin; - if ($0 !~ / - / && $0 !~ / -$/ && $0 !~ /^- /) { - printf "%s", $0; - charct += length(); - needmore = 0; - } else { - after = 1 - if ($0 ~ / - /) { - where = match( $0 , / - /); - } else if ($0 ~ / -$/) { - where = match( $0, / -$/); - } else { - where = 1; - } - if ((width = 20-charct) < 0) width=0 - printf "%-*s", width, sprintf( "%s (%s)", - substr( $0, 1, where-1 ), actual_section ); - printf "%s", substr( $0, where ) - if ($0 ~ /- *$/) { - needmore = 1; - } else { - needmore = 0; - } - } - } - } - } - } - closeline(); - } - - { # Main action - process each filename read in. - filename = $0; - do_one(); - } - ' pages=$pages section=$section verbose=$verbose - cd .. - fi - done > $TMPFILE - - cd $here - - # kludge for Slackware's /usr/man/preformat - if [ $mandir = /usr/man/preformat ] - then - mandir1=/usr/man - else - mandir1=$mandir - fi - - if [ -f ${mandir1}/whatis ] - then - cat ${mandir1}/whatis >> $TMPFILE - fi - sed '/^$/d' < $TMPFILE | sort | uniq > ${mandir1}/whatis - - chmod 644 ${mandir1}/whatis - rm $TMPFILE - fi - done -done - -# remove the dir if we created it -if [ $TMPFILE = $TMPFILEDIR/w ]; then - rmdir $TMPFILEDIR -fi diff --git a/erts/lib_src/yielding_c_fun/ycf_lexer.c b/erts/lib_src/yielding_c_fun/ycf_lexer.c index be4112dca1dd..dd78cc7903e4 100644 --- a/erts/lib_src/yielding_c_fun/ycf_lexer.c +++ b/erts/lib_src/yielding_c_fun/ycf_lexer.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB and Kjell Winblad 2019-2021. All Rights Reserved. + * Copyright Ericsson AB and Kjell Winblad 2019-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/erts/preloaded/ebin/atomics.beam b/erts/preloaded/ebin/atomics.beam index 46816a472d28..164e1b96ea94 100644 Binary files a/erts/preloaded/ebin/atomics.beam and b/erts/preloaded/ebin/atomics.beam differ diff --git a/erts/preloaded/ebin/counters.beam b/erts/preloaded/ebin/counters.beam index d693a3bb8ba9..38083d17f4ea 100644 Binary files a/erts/preloaded/ebin/counters.beam and b/erts/preloaded/ebin/counters.beam differ diff --git a/erts/preloaded/ebin/erl_init.beam b/erts/preloaded/ebin/erl_init.beam index cb7e6e25856b..b02ef52d0621 100644 Binary files a/erts/preloaded/ebin/erl_init.beam and b/erts/preloaded/ebin/erl_init.beam differ diff --git a/erts/preloaded/ebin/erl_prim_loader.beam b/erts/preloaded/ebin/erl_prim_loader.beam index a46d58d7a521..4c7ace4cce69 100644 Binary files a/erts/preloaded/ebin/erl_prim_loader.beam and b/erts/preloaded/ebin/erl_prim_loader.beam differ diff --git a/erts/preloaded/ebin/erl_tracer.beam b/erts/preloaded/ebin/erl_tracer.beam index 662ef712eab8..5f26673a161d 100644 Binary files a/erts/preloaded/ebin/erl_tracer.beam and b/erts/preloaded/ebin/erl_tracer.beam differ diff --git a/erts/preloaded/ebin/erlang.beam b/erts/preloaded/ebin/erlang.beam index 2581c9d88d02..6396c379fee0 100644 Binary files a/erts/preloaded/ebin/erlang.beam and b/erts/preloaded/ebin/erlang.beam differ diff --git a/erts/preloaded/ebin/erts_code_purger.beam b/erts/preloaded/ebin/erts_code_purger.beam index 04fc2208bac2..89b1b1e551a6 100644 Binary files a/erts/preloaded/ebin/erts_code_purger.beam and b/erts/preloaded/ebin/erts_code_purger.beam differ diff --git a/erts/preloaded/ebin/erts_dirty_process_signal_handler.beam b/erts/preloaded/ebin/erts_dirty_process_signal_handler.beam index 24bcc53c3ecf..9fd774fb6b05 100644 Binary files a/erts/preloaded/ebin/erts_dirty_process_signal_handler.beam and b/erts/preloaded/ebin/erts_dirty_process_signal_handler.beam differ diff --git a/erts/preloaded/ebin/erts_internal.beam b/erts/preloaded/ebin/erts_internal.beam index 62694988dc24..1cb7d6bbd047 100644 Binary files a/erts/preloaded/ebin/erts_internal.beam and b/erts/preloaded/ebin/erts_internal.beam differ diff --git a/erts/preloaded/ebin/erts_literal_area_collector.beam b/erts/preloaded/ebin/erts_literal_area_collector.beam index 54c0f4c819a0..31a47701e0f5 100644 Binary files a/erts/preloaded/ebin/erts_literal_area_collector.beam and b/erts/preloaded/ebin/erts_literal_area_collector.beam differ diff --git a/erts/preloaded/ebin/init.beam b/erts/preloaded/ebin/init.beam index 15f75ff4c4d4..8f90d3c230d2 100644 Binary files a/erts/preloaded/ebin/init.beam and b/erts/preloaded/ebin/init.beam differ diff --git a/erts/preloaded/ebin/persistent_term.beam b/erts/preloaded/ebin/persistent_term.beam index 461882747936..04027703c16c 100644 Binary files a/erts/preloaded/ebin/persistent_term.beam and b/erts/preloaded/ebin/persistent_term.beam differ diff --git a/erts/preloaded/ebin/prim_buffer.beam b/erts/preloaded/ebin/prim_buffer.beam index fa74efcfeb58..9cf6bb621816 100644 Binary files a/erts/preloaded/ebin/prim_buffer.beam and b/erts/preloaded/ebin/prim_buffer.beam differ diff --git a/erts/preloaded/ebin/prim_eval.beam b/erts/preloaded/ebin/prim_eval.beam index 8a5c4ee49d3f..c51f6ccb17b9 100644 Binary files a/erts/preloaded/ebin/prim_eval.beam and b/erts/preloaded/ebin/prim_eval.beam differ diff --git a/erts/preloaded/ebin/prim_file.beam b/erts/preloaded/ebin/prim_file.beam index 3b4d892eed1f..4800e7355827 100644 Binary files a/erts/preloaded/ebin/prim_file.beam and b/erts/preloaded/ebin/prim_file.beam differ diff --git a/erts/preloaded/ebin/prim_inet.beam b/erts/preloaded/ebin/prim_inet.beam index 883010b373ef..5f8d5b8709f3 100644 Binary files a/erts/preloaded/ebin/prim_inet.beam and b/erts/preloaded/ebin/prim_inet.beam differ diff --git a/erts/preloaded/ebin/prim_net.beam b/erts/preloaded/ebin/prim_net.beam index 0495ec0d779b..ba4d62024b63 100644 Binary files a/erts/preloaded/ebin/prim_net.beam and b/erts/preloaded/ebin/prim_net.beam differ diff --git a/erts/preloaded/ebin/prim_socket.beam b/erts/preloaded/ebin/prim_socket.beam index 26c2cacb48d9..dc896ea2dcd5 100644 Binary files a/erts/preloaded/ebin/prim_socket.beam and b/erts/preloaded/ebin/prim_socket.beam differ diff --git a/erts/preloaded/ebin/prim_zip.beam b/erts/preloaded/ebin/prim_zip.beam index a5d1a6553b4d..410f5d1a1c33 100644 Binary files a/erts/preloaded/ebin/prim_zip.beam and b/erts/preloaded/ebin/prim_zip.beam differ diff --git a/erts/preloaded/ebin/socket_registry.beam b/erts/preloaded/ebin/socket_registry.beam index 55b4305b1c25..ca890c5ee149 100644 Binary files a/erts/preloaded/ebin/socket_registry.beam and b/erts/preloaded/ebin/socket_registry.beam differ diff --git a/erts/preloaded/ebin/zlib.beam b/erts/preloaded/ebin/zlib.beam index fc1d029f27bd..58bcc5f30c1c 100644 Binary files a/erts/preloaded/ebin/zlib.beam and b/erts/preloaded/ebin/zlib.beam differ diff --git a/erts/preloaded/src/init.erl b/erts/preloaded/src/init.erl index f4f3bfdb98dd..a8bff9d252f5 100644 --- a/erts/preloaded/src/init.erl +++ b/erts/preloaded/src/init.erl @@ -321,7 +321,7 @@ code_path_choice() -> {ok,[["relaxed"]]} -> relaxed; _Else -> - relaxed + strict end. boot(Start,Flags,Args) -> diff --git a/erts/test/parallel_messages_SUITE_data/visualize_throughput.html b/erts/test/parallel_messages_SUITE_data/visualize_throughput.html index c4ed94b679a8..9f4ebed4e9dc 100644 --- a/erts/test/parallel_messages_SUITE_data/visualize_throughput.html +++ b/erts/test/parallel_messages_SUITE_data/visualize_throughput.html @@ -4,7 +4,7 @@ - + @@ -38,7 +38,7 @@

Message Send/Receive Benchmark Result Viewer

Note that one can paste results from several benchmark runs into the field below. Results from the same scenario but from different benchmark runs will be relabeled and plotted in the same graph automatically.

- Note also that that lines can be hidden by clicking on the corresponding label. + Note also that lines can be hidden by clicking on the corresponding label.

Paste the generated data in the field below and press the Render button:
diff --git a/erts/vsn.mk b/erts/vsn.mk index 6d7874ce9ab4..8a2ae91fa57f 100644 --- a/erts/vsn.mk +++ b/erts/vsn.mk @@ -18,7 +18,7 @@ # %CopyrightEnd% # -VSN = 14.1.1 +VSN = 14.2.1 # Port number 4365 in 4.2 # Port number 4366 in 4.3 diff --git a/lib/asn1/doc/src/GUI.asn1 b/lib/asn1/doc/src/GUI.asn1 new file mode 100644 index 000000000000..b05ecb217010 --- /dev/null +++ b/lib/asn1/doc/src/GUI.asn1 @@ -0,0 +1,31 @@ +GUI DEFINITIONS AUTOMATIC TAGS ::= BEGIN + + Action ::= SEQUENCE { + number INTEGER DEFAULT 15, + handle Handle DEFAULT {number 12, on TRUE} + } + + Key ::= Button + Handle ::= Key + + Button ::= SEQUENCE { + number INTEGER, + on BOOLEAN + } + + Window ::= CHOICE { + vsn INTEGER, + status Status + } + + Status ::= SEQUENCE { + state INTEGER, + buttonList SEQUENCE OF Button, + enabled BOOLEAN OPTIONAL, + actions CHOICE { + possibleActions SEQUENCE OF Action, + noOfActions INTEGER + } + } + +END diff --git a/lib/asn1/doc/src/GUI.asn1config b/lib/asn1/doc/src/GUI.asn1config new file mode 100644 index 000000000000..ebcb6295581c --- /dev/null +++ b/lib/asn1/doc/src/GUI.asn1config @@ -0,0 +1,6 @@ +{exclusive_decode, + {'GUI', + [{decode_Window_exclusive, + ['Window',[{status,[{buttonList,parts},{actions,undecoded}]}]]}, + {decode_Button_exclusive, + ['Button',[{number,undecoded}]]}]}}. diff --git a/lib/asn1/doc/src/Seq.asn b/lib/asn1/doc/src/Seq.asn deleted file mode 100644 index 2f2c48cf025d..000000000000 --- a/lib/asn1/doc/src/Seq.asn +++ /dev/null @@ -1,37 +0,0 @@ -GUI DEFINITIONS AUTOMATIC TAGS ::= - -BEGIN - -Action ::= SEQUENCE - { - number INTEGER DEFAULT 15, - handle [0] Handle DEFAULT {number 12, on TRUE} - } - -Key ::= [11] EXPLICIT Button -Handle ::= [12] Key -Button ::= SEQUENCE - { - number INTEGER, - on BOOLEAN - } - -Window ::= CHOICE - { - vsn INTEGER, - status E - } - -Status ::= SEQUENCE - { - state INTEGER, - buttonList SEQUENCE OF Button, - enabled BOOLEAN OPTIONAL, - actions CHOICE { - possibleActions SEQUENCE OF Action, - noOfActions INTEGER - } - } - - -END diff --git a/lib/asn1/doc/src/Seq.asn1config b/lib/asn1/doc/src/Seq.asn1config deleted file mode 100644 index 571cf4cd32b7..000000000000 --- a/lib/asn1/doc/src/Seq.asn1config +++ /dev/null @@ -1,3 +0,0 @@ -{exclusive_decode,{'GUI', - [{decode_Window_exclusive,['Window',[{status,[{buttonList,parts},{actions,undecoded}]}]]}, - {decode_Button_exclusive,['Button',[{number,undecoded}]]}]}}. diff --git a/lib/asn1/doc/src/asn1_spec.xmlsrc b/lib/asn1/doc/src/asn1_spec.xmlsrc index fd30d34a34d3..6f5ba5d2e3f3 100644 --- a/lib/asn1/doc/src/asn1_spec.xmlsrc +++ b/lib/asn1/doc/src/asn1_spec.xmlsrc @@ -4,7 +4,7 @@
- 20032021 + 20032023 Ericsson AB. All Rights Reserved. @@ -30,19 +30,18 @@ asn1_spec.xml
-

When performance is of highest priority and you are interested in - a limited part of the ASN.1 encoded message before deciding what - to do with the rest of it, an option is to decode only this small - part. The situation can be a server that has to decide the - addressee of a message. The addressee can be interested in - the entire message, but the server can be a bottleneck that you want - to spare any unnecessary load.

-

Instead of making two complete decodes (the normal case of +

When performance is of highest priority and one is interested in + a limited part of the ASN.1 encoded message before deciding what to + do with the rest of it, an option is to decode only a part of the + message. This situation can be a server that has to decide the + addressee of a message. The addressee can be interested in the + entire message, but the server can be a bottleneck that you want to + spare any unnecessary load.

+

Instead of making two complete decodes (the normal case of decode), one in the server and one in the addressee, it is only - necessary to make one specialized decode(in the server) - and another complete decode(in the addressee). This section - describes the following two specialized decodes, which support - to solve this and similar problems:

+ necessary to make one specialized decode (in the server) + and another complete decode (in the addressee). This section + describes the following specialized decode functionality:

Exclusive decode Selected decode @@ -52,16 +51,11 @@
Exclusive Decode -

The basic idea with exclusive - decode is to specify which parts of the message you want to - exclude from being decoded. These parts remain encoded and are - returned in the value structure as binaries. They can be decoded - in turn by passing them to a certain decode_part/2 - function. The performance gain is high for large messages. - You can do an exclusive decode and later one or more - decodes of the parts, or a second complete decode instead of two or - more complete decodes. -

+

The basic idea with exclusive decode is to specify which parts + of the message you want to exclude from being decoded. These parts + remain encoded and are returned in the value structure as + binaries. The undecoded parts can be decoded later by calling the + decode_part/2 function.

Procedure @@ -105,17 +99,17 @@ decode_exclusive and an ASN.1 encoded message Bin is to be exclusive decoded, the call is as follows:

-{ok,Excl_Message} = 'MyModule':decode_exclusive(Bin)
+{ok,ExclMessage} = 'MyModule':decode_exclusive(Bin) -

The result Excl_Message has the same structure as a +

The result ExclMessage has the same structure as a complete decode would have, except for the parts of the top type that were not decoded. The undecoded parts are on their places - in the structure on format {Type_Key,Undecoded_Value}. + in the structure on format {TypeKey,UndecodedValue}.

Each undecoded part that is to be decoded must be fed into function decode_part/2 as follows:

-{ok,Part_Message} = 'MyModule':decode_part(Type_Key,Undecoded_Value)
+{ok,PartMessage} = 'MyModule':decode_part(TypeKey, UndecodedValue)
@@ -124,54 +118,62 @@

This instruction is written in the configuration file in the following format:

-Exclusive_Decode_Instruction = {exclusive_decode,{Module_Name,Decode_Instructions}}.
+ExclusiveDecodeInstruction = {exclusive_decode,{ModuleName,DecodeInstructions}}.
 
-Module_Name = atom()
+ModuleName = atom()
 
-Decode_Instructions = [Decode_Instruction]+
+DecodeInstructions = [DecodeInstruction]+
 
-Decode_Instruction = {Exclusive_Decode_Function_Name,Type_List}
+DecodeInstruction = {ExclusiveDecodeFunctionName,TypeList}
 
-Exclusive_Decode_Function_Name = atom()
+ExclusiveDecodeFunctionName = atom()
 
-Type_List = [Top_Type,Element_List]
+TypeList = [TopType,ElementList]
 
-Element_List = [Element]+
+ElementList = [Element]+
 
 Element = {Name,parts} |
           {Name,undecoded} |
-          {Name,Element_List}
+          {Name,ElementList}
 
-Top_Type = atom()
+TopType = atom()
 
 Name = atom()

The instruction must be a valid Erlang term ended by a dot.

-

In Type_List the "path" from the top type to each - undecoded subcomponents is described. The top type of the path is - an atom, the name of it. The action on each component/type that - follows is described by one of - {Name,parts}, {Name,undecoded}, {Name,Element_List}.

-

The use and effect of the actions are as follows: -

+

In TypeList the path from the top type to each + undecoded subcomponent is described. TopType is the name + of a top-level type in the ASN.1 specification. The action for + each component in ElementList is described by one of:

+ - {Name,undecoded} - Tells that the element is left - undecoded during the exclusive decode. The type of Name - can be any ASN.1 type. The value of element Name is - returned as a tuple (as mentioned in the previous section) in - the value structure of the top type. - {Name,parts} - The type of Name can be one of - SEQUENCE OF or SET OF. The action implies that - the different components of Name are left undecoded. The - value of Name is returned as a tuple (as mentioned in - the previous section) where the second element is a list of - binaries. This is because the representation of a SEQUENCE OF - or a SET OF in Erlang is a list of its internal type. Any - of the elements in this list or the entire list can be decoded by - function decode_part. - {Name,Element_List} - This action is used when one or - more of the subtypes of Name is exclusive decoded. + {Name,parts} + {Name,undecoded} + {Name,ElementList} +

The use and effect of the actions are as follows:

+ + {Name,undecoded} + Leaves the element undecoded. The type of Name can + be any ASN.1 type. The value of element Name is + returned as a tuple (as mentioned in the previous section) in + the value structure of the top type. + + {Name,parts} + The type of Name must be either SEQUENCE OF + or SET OF. The action implies that the different + components of Name are left undecoded. The value of + Name is returned as a tuple (as mentioned in the + previous section) where the second element is a list of + binaries. This is because the representation of a SEQUENCE + OF or a SET OF in Erlang is a list of its internal + type. Any of the elements in this list or the entire list can + be decoded by function decodepart. + + {Name,ElementList} + This action is used when one or more of the subtypes of + Name is exclusively decoded. +

Name in these actions can be a component name of a SEQUENCE OF or a SET OF, or a name of an alternative in a CHOICE. @@ -183,12 +185,12 @@ Name = atom()

In this examples, the definitions from the following ASN.1 specification are used:

- +

If Button is a top type and it is needed to exclude - component number from decode, Type_List in the + component number from decode, TypeList in the instruction in the configuration file is ['Button',[{number,undecoded}]]. If you call the decode - function decode_Button_exclusive, Decode_Instruction is + function decode_Button_exclusive, DecodeInstruction is {decode_Button_exclusive,['Button',[{number,undecoded}]]}.

Another top type is Window whose subcomponent @@ -196,7 +198,7 @@ Name = atom() buttonList are to be left undecoded. For this type, the function is named decode__Window_exclusive. The complete Exclusive_Decode_Instruction configuration is as follows:

- +

The following figure shows the bytes of a Window:status message. The components buttonList and actions are excluded from decode. Only state and enabled are decoded @@ -206,115 +208,91 @@ Name = atom() Bytes of a Window:status Message

-

Compiling GUI.asn including the configuration file is done - as follows:

+

Here follows an example of how the module. Note that option no_ok_wrapper + is used to make the make example more concise.

-unix> erlc -bber +asn1config GUI.asn
-
-erlang> asn1ct:compile('GUI', [ber,asn1config]).
-

The module can be used as follows:

- 
-1> Button_Msg = {'Button',123,true}.
-{'Button',123,true}
-2> {ok,Button_Bytes} = 'GUI':encode('Button',Button_Msg).
-{ok,[<<48>>,
-     [6],
-     [<<128>>,
-      [1],
-      123],
-     [<<129>>,
-      [1],
-      255]]}
-3> {ok,Exclusive_Msg_Button} = 'GUI':decode_Button_exclusive(list_to_binary(Button_Bytes)).
-{ok,{'Button',{'Button_number',<<28,1,123>>},
-         true}}
-4> 'GUI':decode_part('Button_number',<<128,1,123>>).
-{ok,123}
-5> Window_Msg = 
-{'Window',{status,{'Status',35,
-              [{'Button',3,true},
-               {'Button',4,false},
-               {'Button',5,true},
-               {'Button',6,true},
-               {'Button',7,false},
-               {'Button',8,true},
-               {'Button',9,true},
-               {'Button',10,false},
-               {'Button',11,true},
-               {'Button',12,true},
-               {'Button',13,false},
-               {'Button',14,true}],
-              false,
-              {possibleActions,[{'Action',16,{'Button',17,true}}]}}}}. 
-{'Window',{status,{'Status',35,
-              [{'Button',3,true},
-               {'Button',4,false},
-               {'Button',5,true},
-               {'Button',6,true},
-               {'Button',7,false},
-               {'Button',8,true},
-               {'Button',9,true},
-               {'Button',10,false},
-               {'Button',11,true},
-               {'Button',12,true},
-               {'Button',13,false},
-               {'Button',14,true}],
-              false,
-              {possibleActions,[{'Action',16,{'Button',17,true}}]}}}}
-6> {ok,Window_Bytes}='GUI':encode('Window',Window_Msg).
-{ok,[<<161>>,
-     [127],
-     [<<128>>, ...
-
-
-8> {ok,{status,{'Status',Int,{Type_Key_SeqOf,Val_SEQOF},
-BoolOpt,{Type_Key_Choice,Val_Choice}}}}=
-'GUI':decode_Window_status_exclusive(list_to_binary(Window_Bytes)).
-{ok,{status,{'Status',35,
-        {'Status_buttonList',[<<48,6,128,1,3,129,1,255>>,
-                              <<48,6,128,1,4,129,1,0>>,
-                              <<48,6,128,1,5,129,1,255>>,
-                              <<48,6,128,1,6,129,1,255>>,
-                              <<48,6,128,1,7,129,1,0>>,
-                              <<48,6,128,1,8,129,1,255>>,
-                              <<48,6,128,1,9,129,1,255>>,
-                              <<48,6,128,1,10,129,1,0>>,
-                              <<48,6,128,1,11,129,1,255>>,
-                              <<48,6,128,1,12,129,1,255>>,
-                              <<48,6,128,1,13,129,1,0>>,
-                              <<48,6,128,1,14,129,1,255>>]},
-        false,
-        {'Status_actions',
-<<163,21,160,19,48,17,2,1,16,160,12,172,10,171,8,48,6,128,1,...>>}}}}
-10> 'GUI':decode_part(Type_Key_SeqOf,Val_SEQOF).
-{ok,[{'Button',3,true},
-     {'Button',4,false},
-     {'Button',5,true},
-     {'Button',6,true},
-     {'Button',7,false},
-     {'Button',8,true},
-     {'Button',9,true},
-     {'Button',10,false},
-     {'Button',11,true},
-     {'Button',12,true},
-     {'Button',13,false},
-     {'Button',14,true}]}
-11> 'GUI':decode_part(Type_Key_SeqOf,hd(Val_SEQOF)).
-{ok,{'Button',3,true}}
-12> 'GUI':decode_part(Type_Key_Choice,Val_Choice).  
-{ok,{possibleActions,[{'Action',16,{'Button',17,true}}]}}
+1> asn1ct:compile('GUI', [ber,asn1config,no_ok_wrapper]). +ok +2> rr('GUI'). +['Action','Button','Status'] +3> ButtonMsg = #'Button'{number=123,on=true}. +#'Button'{number = 123,on = true} +4> ButtonBytes = 'GUI':encode('Button', ButtonMsg). +<<48,6,128,1,123,129,1,255>> +5> ExclusiveMsgButton = 'GUI':decode_Button_exclusive(ButtonBytes). +#'Button'{number = {'Button_number',<<128,1,123>>}, + on = true} +6> {UndecKey,UndecBytes} = ExclusiveMsgButton#'Button'.number. +{'Button_number',<<128,1,123>>} +7> 'GUI':decode_part(UndecKey, UndecBytes). +123 +8> WindowMsg = +{status,{'Status',35, + [{'Button',3,true}, + {'Button',4,false}, + {'Button',5,true}, + {'Button',6,true}, + {'Button',7,false}], + false, + {possibleActions,[{'Action',16,{'Button',17,true}}]}}}. +{status,#'Status'{state = 35, + buttonList = [#'Button'{number = 3,on = true}, + #'Button'{number = 4,on = false}, + #'Button'{number = 5,on = true}, + #'Button'{number = 6,on = true}, + #'Button'{number = 7,on = false}], + enabled = false, + actions = {possibleActions,[#'Action'{number = 16, + handle = #'Button'{number = 17,on = true}}]}}} +9> WindowBytes = 'GUI':encode('Window', WindowMsg). +<<161,65,128,1,35,161,40,48,6,128,1,3,129,1,255,48,6,128, + 1,4,129,1,0,48,6,128,1,5,129,...>> +10> {status,#'Status'{buttonList={UndecWindowKey,UndecWindowParts}}} = +'GUI':decode_Window_exclusive(WindowBytes). +{status,#'Status'{state = 35, + buttonList = {'Status_buttonList',[<<48,6,128,1,3,129,1, + 255>>, + <<48,6,128,1,4,129,1,0>>, + <<48,6,128,1,5,129,1,255>>, + <<48,6,128,1,6,129,1,255>>, + <<48,6,128,1,7,129,1,0>>]}, + enabled = false, + actions = {'Status_actions',<<163,15,160,13,48,11,128, + 1,16,161,6,128,1,17,129, + 1,255>>}}} +11> 'GUI':decode_part(UndecWindowKey, UndecWindowParts). +[#'Button'{number = 3,on = true}, + #'Button'{number = 4,on = false}, + #'Button'{number = 5,on = true}, + #'Button'{number = 6,on = true}, + #'Button'{number = 7,on = false}] +12> 'GUI':decode_part(UndecWindowKey, hd(UndecWindowParts)). +#'Button'{number = 3,on = true} +13> {status,#'Status'{actions={ChoiceKey,ChoiceUndec}}} = v(10). +{status,#'Status'{state = 35, + buttonList = {'Status_buttonList',[<<48,6,128,1,3,129,1, + 255>>, + <<48,6,128,1,4,129,1,0>>, + <<48,6,128,1,5,129,1,255>>, + <<48,6,128,1,6,129,1,255>>, + <<48,6,128,1,7,129,1,0>>]}, + enabled = false, + actions = {'Status_actions',<<163,15,160,13,48,11,128, + 1,16,161,6,128,1,17,129, + 1,255>>}}} +14> 'GUI':decode_part(ChoiceKey, ChoiceUndec). +{possibleActions,[#'Action'{number = 16, + handle = #'Button'{number = 17,on = true}}]}
Selective Decode -

This specialized decode decodes a subtype of a - constructed value and is the fastest method to extract a - subvalue. This decode is typically used when you want to - inspect, for example, a version number, to be able to decide what - to do with the entire value. The result is returned as - {ok,Value} or {error,Reason}. -

+

Selective decode decodes a single subtype of a constructed + value. This is the fastest method to extract a subvalue. Selective + decode is typically used when one want to inspect, for example, a + version number to be able to decide what to do with the entire + value.

Procedure @@ -343,13 +321,11 @@ BoolOpt,{Type_Key_Choice,Val_Choice}}}}=
User Interface

The only new user interface function is the one provided by the - user in the configuration file. The function is started by - the ModuleName:FunctionName notation. -

+ user in the configuration file.

For example, if the configuration file includes the specification {selective_decode,{'ModuleName',[{selected_decode_Window,TypeList}]}} do the selective decode by - {ok,Result}='ModuleName':selected_decode_Window(EncodedBinary).

+ {ok,Result} = 'ModuleName':selected_decode_Window(EncodedBinary).

@@ -358,42 +334,42 @@ BoolOpt,{Type_Key_Choice,Val_Choice}}}}=

One or more selective decode functions can be described in a configuration file. Use the following notation:

-Selective_Decode_Instruction = {selective_decode,{Module_Name,Decode_Instructions}}.
+SelectiveDecodeInstruction = {selective_decode,{ModuleName,DecodeInstructions}}.
 
-Module_Name = atom()
+ModuleName = atom()
 
-Decode_Instructions = [Decode_Instruction]+
+DecodeInstructions = [DecodeInstruction]+
 
-Decode_Instruction = {Selective_Decode_Function_Name,Type_List}
+DecodeInstruction = {SelectiveDecodeFunctionName,TypeList}
 
-Selective_Decode_Function_Name = atom()
+SelectiveDecodeFunctionName = atom()
 
-Type_List = [Top_Type|Element_List]
+TypeList = [TopType|ElementList]
 
-Element_List = Name|List_Selector
+ElementList = Name|ListSelector
 
 Name = atom()
 
-List_Selector = [integer()]
+ListSelector = [integer()]

The instruction must be a valid Erlang term ended by a dot.

- Module_Name is the same as the name of the ASN.1 + ModuleName is the same as the name of the ASN.1 specification, but without the extension. - Decode_Instruction is a tuple with your chosen + DecodeInstruction is a tuple with your chosen function name and the components from the top type that leads to the single type you want to decode. Ensure to choose a name of your function that is not the same as any of the generated functions. - The first element of Type_List is the top type of the - encoded message. In Element_List, it is followed by + The first element of TypeList is the top type of the + encoded message. In ElementList, it is followed by each of the component names that leads to selected type. - Each name in Element_List must be a constructed type + Each name in ElementList must be a constructed type except the last name, which can be any type. - List_Selector makes it possible to choose one of the - encoded components in a a SEQUENCE OF or a SET OF. + ListSelector makes it possible to choose one of the + encoded components in a SEQUENCE OF or a SET OF. It is also possible to go further in that component and pick a - subtype of that to decode. So, in the Type_List: + subtype of that to decode. So, in the TypeList: ['Window',status,buttonList,[1],number], component buttonList must be of type SEQUENCE OF or SET OF. @@ -407,7 +383,7 @@ List_Selector = [integer()]
- Another Example + Example

In this example, the same ASN.1 specification as in Section Writing an Exclusive Decode Instruction is used. The following is a valid selective decode instruction:

@@ -438,15 +414,14 @@ List_Selector = [integer()] value 4711 is to be picked by selected_decode_Action. In an Erlang terminal it looks as follows:

-ValAction = {'Action',17,{'Button',4711,false}}.
+1> asn1ct:compile('GUI', [ber,asn1config,no_ok_wrapper]).
+ok
+2> ValAction = {'Action',17,{'Button',4711,false}}.
 {'Action',17,{'Button',4711,false}}
-7> {ok,Bytes}='GUI':encode('Action',ValAction).
-...
-8> BinBytes = list_to_binary(Bytes).
+3> Bytes = 'GUI':encode('Action',ValAction).
 <<48,18,2,1,17,160,13,172,11,171,9,48,7,128,2,18,103,129,1,0>>
-9> 'GUI':selected_decode_Action(BinBytes).
-{ok,4711}
-10>
+4> 'GUI':selected_decode_Action(Bytes). +4711

The third instruction, ['Window',status,actions,possibleActions,[1],handle,number], works as follows:

@@ -482,7 +457,7 @@ ValAction = {'Action',17,{'Button',4711,false}}. selected_decode_Window1 decodes the intended subvalue of value Val:

-1> Val = {'Window',{status,{'Status',12,
+1> Val = {status,{'Status',12,
                     [{'Button',13,true},
                      {'Button',14,false},
                      {'Button',15,true},
@@ -490,355 +465,14 @@ ValAction = {'Action',17,{'Button',4711,false}}.
                     true,
                     {possibleActions,[{'Action',17,{'Button',18,false}},
                                       {'Action',19,{'Button',20,true}},
-                                      {'Action',21,{'Button',22,false}}]}}}}
-2> {ok,Bytes}='GUI':encode('Window',Val).
-...
-3> Bin = list_to_binary(Bytes).
-<<161,101,128,1,12,161,32,48,6,128,1,13,129,1,255,48,6,128,1,14,129,1,0,48,6,128,1,15,129,...>>
-4> 'GUI':selected_decode_Window1(Bin).
-{ok,13}
-5> 'GUI':selected_decode_Window2(Bin).
-{ok,18}
-

Notice that the value fed into the selective decode - functions must be a binary. -

-
-
- -
- Performance -

To give an indication on the possible performance gain using - the specialized decodes, some measures have been performed. The - relative figures in the outcome between selective, exclusive, and - complete decode (the normal case) depend on the structure of - the type, the size of the message, and on what level the - selective and exclusive decodes are specified. -

- -
- ASN.1 Specifications, Messages, and Configuration -

The specifications GUI and - MEDIA-GATEWAY-CONTROL - were used in the test. -

-

For the GUI specification the configuration was as follows:

- 
-{selective_decode,
-  {'GUI',
-    [{selected_decode_Window1,
-         ['Window',
-          status,buttonList,
-          [1],
-          number]},
-     {selected_decode_Window2,
-         ['Window',
-          status,
-          actions,
-          possibleActions,
-          [1],
-          handle,number]}]}}.
-     {exclusive_decode,
-         {'GUI',
-            [{decode_Window_status_exclusive,
-                ['Window',
-                 [{status,
-                     [{buttonList,parts},
-                      {actions,undecoded}]}]]}]}}.
-

The MEDIA-GATEWAY-CONTROL configuration was as follows:

- 
-{exclusive_decode,
-  {'MEDIA-GATEWAY-CONTROL',
-    [{decode_MegacoMessage_exclusive,
-        ['MegacoMessage',
-         [{authHeader,undecoded},
-          {mess,
-             [{mId,undecoded},
-              {messageBody,undecoded}]}]]}]}}.
-{selective_decode,
-  {'MEDIA-GATEWAY-CONTROL',
-    [{decode_MegacoMessage_selective,
-         ['MegacoMessage',mess,version]}]}}.
-

The corresponding values were as follows:

- 
-{'Window',{status,{'Status',12,
-              [{'Button',13,true},
-               {'Button',14,false},
-               {'Button',15,true},
-               {'Button',16,false},
-               {'Button',13,true},
-               {'Button',14,false},
-               {'Button',15,true},
-               {'Button',16,false},
-               {'Button',13,true},
-               {'Button',14,false},
-               {'Button',15,true},
-               {'Button',16,false}],
-              true,
-              {possibleActions,
-                 [{'Action',17,{'Button',18,false}},
-                  {'Action',19,{'Button',20,true}},
-                  {'Action',21,{'Button',22,false}},
-                  {'Action',17,{'Button',18,false}},
-                  {'Action',19,{'Button',20,true}},
-                  {'Action',21,{'Button',22,false}},
-                  {'Action',17,{'Button',18,false}},
-                  {'Action',19,{'Button',20,true}},
-                  {'Action',21,{'Button',22,false}},
-                  {'Action',17,{'Button',18,false}},
-                  {'Action',19,{'Button',20,true}},
-                  {'Action',21,{'Button',22,false}},
-                  {'Action',17,{'Button',18,false}},
-                  {'Action',19,{'Button',20,true}},
-                  {'Action',21,{'Button',22,false}},
-                  {'Action',17,{'Button',18,false}},
-                  {'Action',19,{'Button',20,true}},
-                  {'Action',21,{'Button',22,false}}]}}}}
-
-
-{'MegacoMessage',asn1_NOVALUE,
-  {'Message',1,
-    {ip4Address,
-      {'IP4Address',[125,125,125,111],55555}},
-  {transactions,
-    [{transactionReply,
-      {'TransactionReply',50007,asn1_NOVALUE,
-       {actionReplies,
-        [{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,
-          [{auditValueReply,{auditResult,{'AuditResult',
-            {'TerminationID',[],[255,255,255]},
-             [{mediaDescriptor,
-               {'MediaDescriptor',asn1_NOVALUE,
-                {multiStream,
-                 [{'StreamDescriptor',1,
-                   {'StreamParms',
-                    {'LocalControlDescriptor',
-                     sendRecv,
-                     asn1_NOVALUE,
-                     asn1_NOVALUE,
-                     [{'PropertyParm',
-                       [0,11,0,7],
-                       [[52,48]],
-                       asn1_NOVALUE}]},
-                    {'LocalRemoteDescriptor',
-                     [[{'PropertyParm',
-                        [0,0,176,1],
-                        [[48]],
-                        asn1_NOVALUE},
-                       {'PropertyParm',
-                         [0,0,176,8],
-                         [[73,78,32,73,80,52,32,49,50,53,46,49,
-                           50,53,46,49,50,53,46,49,49,49]],
-                         asn1_NOVALUE},
-                       {'PropertyParm',
-                         [0,0,176,15],
-                         [[97,117,100,105,111,32,49,49,49,49,32,
-                           82,84,80,47,65,86,80,32,32,52]],
-                         asn1_NOVALUE},
-                       {'PropertyParm',
-                         [0,0,176,12],
-                         [[112,116,105,109,101,58,51,48]],
-                         asn1_NOVALUE}]]},
-                    {'LocalRemoteDescriptor',
-                     [[{'PropertyParm',
-                         [0,0,176,1],
-                         [[48]],
-                         asn1_NOVALUE},
-                       {'PropertyParm',
-                         [0,0,176,8],
-                         [[73,78,32,73,80,52,32,49,50,52,46,49,50,
-                           52,46,49,50,52,46,50,50,50]],
-                         asn1_NOVALUE},
-                       {'PropertyParm',
-                         [0,0,176,15],
-                         [[97,117,100,105,111,32,50,50,50,50,32,82,
-                           84,80,47,65,86,80,32,32,52]],
-                         asn1_NOVALUE},
-                       {'PropertyParm',
-                         [0,0,176,12],
-                         [[112,116,105,109,101,58,51,48]],
-                         asn1_NOVALUE}]]}}}]}}},
-              {packagesDescriptor,
-               [{'PackagesItem',[0,11],1},
-                {'PackagesItem',[0,11],1}]},
-              {statisticsDescriptor,
-               [{'StatisticsParameter',[0,12,0,4],[[49,50,48,48]]},
-                {'StatisticsParameter',[0,11,0,2],[[54,50,51,48,48]]},
-                {'StatisticsParameter',[0,12,0,5],[[55,48,48]]},
-                {'StatisticsParameter',[0,11,0,3],[[52,53,49,48,48]]},
-                {'StatisticsParameter',[0,12,0,6],[[48,46,50]]},
-                {'StatisticsParameter',[0,12,0,7],[[50,48]]},
-                {'StatisticsParameter',[0,12,0,8],[[52,48]]}]}]}}}]}]}}}]}}}
-

The size of the encoded values was 458 bytes for GUI and 464 - bytes for MEDIA-GATEWAY-CONTROL. -

-
- -
- Results -

The ASN.1 specifications in the test were compiled with options - ber_bin, optimize, driver and asn1config. Omitting - option driver gives - higher values for decode and decode_part. These tests have - not been rerun using NIFs, but are expected to perform about 5% better - than the linked-in driver. -

-

The test program runs 10000 decodes on the value, resulting - in an output with the elapsed time in microseconds for the - total number of decodes. -

- - - Function - Time (microseconds) - Decode Type - ASN.1 Specification - % of Time versus Complete Decode - - - decode_MegacoMessage_selective/1 - 374045 - Selective - MEDIA-GATEWAY-CONTROL - 8.3 - - - decode_MegacoMessage_exclusive/1 - 621107 - Exclusive - MEDIA-GATEWAY-CONTROL - 13.8 - - - decode/2 - 4507457 - Complete - MEDIA-GATEWAY-CONTROL - 100 - - - selected_decode_Window1/1 - 449585 - Selective - GUI - 7.6 - - - selected_decode_Window2/1 - 890666 - Selective - GUI - 15.1 - - - decode_Window_status_exclusive/1 - 1251878 - Exclusive - GUI - 21.3 - - - decode/2 - 5889197 - Complete - GUI - 100 - - Results of Complete, Exclusive, and Selective Decode -
-

It is also of interest to know the relation is between - a complete decode, an exclusive decode followed by - decode_part of the excluded parts, and a selective decode - followed by a complete decode. Some situations can be compared to - this simulation, for example, inspect a subvalue and later inspect - the entire value. The following table shows figures from this - test. The number of loops and the time unit are the same as in the - previous test. -

- - - Actions - Function     - Time (microseconds) - ASN.1 Specification - % of Time vs. Complete Decode - - - Complete - decode/2 - 4507457 - MEDIA-GATEWAY-CONTROL - 100 - - - Selective and Complete - decode_­MegacoMessage_­selective/1 - 4881502 - MEDIA-GATEWAY-CONTROL - 108.3 - - - Exclusive and decode_part - decode_­MegacoMessage_­exclusive/1 - 5481034 - MEDIA-GATEWAY-CONTROL - 112.3 - - - Complete - decode/2 - 5889197 - GUI - 100 - - - Selective and Complete - selected_­decode_­Window1/1 - 6337636 - GUI - 107.6 - - - Selective and Complete - selected_­decode_­Window2/1 - 6795319 - GUI - 115.4 - - - Exclusive and decode_part - decode_­Window_­status_­exclusive/1 - 6249200 - GUI - 106.1 - - Results of Complete, Exclusive + decode_part, and Selective + complete decodes -
-

Other ASN.1 types and values can differ much from these - figures. It is therefore important that you, in every case where - you intend to use either of these decodes, perform some tests - that show if you will benefit your purpose. -

-
- -
- Final Remarks - - The gain of using selective and exclusive decode instead of a - complete decode is greater the bigger the value and the - less deep in the structure you have to decode. - Use selective decode instead of exclusive decode if you are - interested in only a single subvalue. - Exclusive decode followed by - decode_part decodes is attractive if the parts are sent - to different servers for decoding, or if you in some cases are not - interested in all parts. - The fastest selective decode is when the decoded type is a - primitive type and not so deep in the structure of the top - type. selected_decode_Window2 decodes a high constructed - value, which explains why this operation is relatively slow. - It can vary from case to case which combination of - selective/complete decode or exclusive/part decode is the fastest. - + {'Action',21,{'Button',22,false}}]}}}. +2> Bin = 'GUI':encode('Window',Val). +<<161,89,128,1,12,161,32,48,6,128,1,13,129,1,255,48,6,128, + 1,14,129,1,0,48,6,128,1,15,129,...>> +4> 'GUI':selected_decode_Window1(Bin). +13 +5> 'GUI':selected_decode_Window2(Bin). +18
diff --git a/lib/asn1/doc/src/notes.xml b/lib/asn1/doc/src/notes.xml index c53f969384a0..53e782d56918 100644 --- a/lib/asn1/doc/src/notes.xml +++ b/lib/asn1/doc/src/notes.xml @@ -4,7 +4,7 @@
- 20042022 + 20042023 Ericsson AB. All Rights Reserved. @@ -32,6 +32,22 @@

This document describes the changes made to the asn1 application.

+
Asn1 5.2.1 + +
Fixed Bugs and Malfunctions + + +

+ Fix benign warning from gcc 11 about mismatching call to + free().

+

+ Own Id: OTP-18844

+ + +
+ +
+
Asn1 5.2
Fixed Bugs and Malfunctions @@ -107,6 +123,22 @@
+
Asn1 5.0.21.1 + +
Fixed Bugs and Malfunctions + + +

+ Fix benign warning from gcc 11 about mismatching call to + free().

+

+ Own Id: OTP-18844

+ + +
+ +
+
Asn1 5.0.21
Fixed Bugs and Malfunctions @@ -158,6 +190,22 @@
+
Asn1 5.0.18.2 + +
Fixed Bugs and Malfunctions + + +

+ Fix benign warning from gcc 11 about mismatching call to + free().

+

+ Own Id: OTP-18844

+ + +
+ +
+
Asn1 5.0.18.1
Fixed Bugs and Malfunctions @@ -723,7 +771,7 @@

- Many bugs have been eliminated in the the ASN.1 compiler + Many bugs have been eliminated in the ASN.1 compiler so that it can now successfully compile many more ASN.1 specifications. Error messages have also been improved.

@@ -977,7 +1025,7 @@

The ASN.1 compiler would fail to compile a constraint - with values given for for the extension part (such as + with values given for the extension part (such as INTEGER (1..10, ..., 11..20)).

Own Id: OTP-11504

diff --git a/lib/asn1/src/Makefile b/lib/asn1/src/Makefile index 9e13d02c8a86..06329840c4fb 100644 --- a/lib/asn1/src/Makefile +++ b/lib/asn1/src/Makefile @@ -66,6 +66,7 @@ CT_MODULES= \ asn1ct_tok \ asn1ct_parser2 \ asn1ct_table \ + asn1ct_partial_decode \ $(EVAL_CT_MODULES) RT_MODULES= \ diff --git a/lib/asn1/src/asn1.app.src b/lib/asn1/src/asn1.app.src index 12a6de88bc18..793b70df4e02 100644 --- a/lib/asn1/src/asn1.app.src +++ b/lib/asn1/src/asn1.app.src @@ -10,5 +10,5 @@ ]}, {env, []}, {applications, [kernel, stdlib]}, - {runtime_dependencies, ["stdlib-3.13","kernel-7.0","erts-11.0"]} + {runtime_dependencies, ["stdlib-5.0","kernel-9.0","erts-14.0"]} ]}. diff --git a/lib/asn1/src/asn1_db.erl b/lib/asn1/src/asn1_db.erl index c7799d076227..a8101edb27e6 100644 --- a/lib/asn1/src/asn1_db.erl +++ b/lib/asn1/src/asn1_db.erl @@ -106,7 +106,9 @@ loop(#state{parent = Parent, monitor = MRef, table = Table, loop(State); {save, OutFile, Mod} -> Mtab = ets:lookup_element(Table, Mod, 2), - TempFile = OutFile ++ ".#temp", + TempFile = OutFile ++ + integer_to_list(erlang:unique_integer([positive])) ++ + ".#temp", ok = ets:tab2file(Mtab, TempFile), ok = file:rename(TempFile, OutFile), loop(State); diff --git a/lib/asn1/src/asn1ct.erl b/lib/asn1/src/asn1ct.erl index 3788f7778923..54436e5b9724 100644 --- a/lib/asn1/src/asn1ct.erl +++ b/lib/asn1/src/asn1ct.erl @@ -42,14 +42,13 @@ maybe_rename_function/3,current_sindex/0, set_current_sindex/1,maybe_saved_sindex/2, parse_and_save/2,verbose/3,warning/3,warning/4,error/3,format_error/1]). +-export([save_config/2,save_gen_state/2,save_gen_state/3]). -export([get_bit_string_format/0,use_legacy_types/0]). -include("asn1_records.hrl"). -include_lib("stdlib/include/erl_compile.hrl"). -include_lib("kernel/include/file.hrl"). --import(asn1ct_gen_ber_bin_v2,[encode_tag_val/3,decode_class/1]). - -ifndef(vsn). -define(vsn,"0.0.1"). -endif. @@ -59,24 +58,6 @@ -define(dupl_equaldefs,2). -define(dupl_eqdefs_uniquedefs,?dupl_equaldefs bor ?dupl_uniquedefs). --define(CONSTRUCTED, 2#00100000). - -%% macros used for partial decode commands --define(CHOOSEN,choosen). --define(SKIP,skip). --define(SKIP_OPTIONAL,skip_optional). - -%% macros used for partial incomplete decode commands --define(MANDATORY,mandatory). --define(DEFAULT,default). --define(OPTIONAL,opt). --define(OPTIONAL_UNDECODED,opt_undec). --define(PARTS,parts). --define(UNDECODED,undec). --define(ALTERNATIVE,alt). --define(ALTERNATIVE_UNDECODED,alt_undec). --define(ALTERNATIVE_PARTS,alt_parts). - %% Removed functions -removed({decode,'_',"use Mod:decode/2 instead"}). @@ -259,8 +240,12 @@ abs_listing(#st{code={M,_},outfile=OutFile}) -> generate_pass(#st{code=Code,outfile=OutFile,erule=Erule,opts=Opts}=St0) -> St = St0#st{code=undefined}, %Reclaim heap space - generate(Code, OutFile, Erule, Opts), - {ok,St}. + case generate(Code, OutFile, Erule, Opts) of + ok -> + {ok,St}; + {error,Errors} -> + {error,St#st{error=Errors}} + end. compile_pass(#st{outfile=OutFile,opts=Opts0}=St) -> asn1_db:dbstop(), %Reclaim memory. @@ -335,11 +320,21 @@ clean_errors(Errors) when is_list(Errors) -> {Structured,Structured ++ AdHoc}; clean_errors(AdHoc) -> {[],AdHoc}. -print_structured_errors([_|_]=Errors) -> - _ = [io:format("~ts:~w: ~ts\n", [F,L,M:format_error(E)]) || - {structured_error,{F,L},M,E} <- Errors], - ok; -print_structured_errors(_) -> ok. +print_structured_errors(Errors) when is_list(Errors) -> + _ = [print_structured_error(F, M, E) || + {structured_error,F,M,E} <- Errors], + ok. + +print_structured_error(F, M, Error) -> + Formatted = M:format_error(Error), + case F of + none -> + io:format("~ts\n", [Formatted]); + {File,none} -> + io:format("~ts: ~ts\n", [File,Formatted]); + {File,Line} when is_integer(Line) -> + io:format("~ts:~p: ~ts\n", [File,Line,Formatted]) + end. compile1(File, #st{opts=Opts}=St0) -> compiler_verbose(File, Opts), @@ -864,21 +859,17 @@ generate({M,CodeTuple}, OutFile, EncodingRule, Options) -> check_maps_option(Gen), %% create decoding function names and taglists for partial decode - try - specialized_decode_prepare(Gen, M) - catch - throw:{error, Reason} -> - warning("Error in configuration file: ~n~p~n", - [Reason], Options, - "Error in configuration file") - end, - - asn1ct_gen:pgen(OutFile, Gen, Code), - cleanup_bit_string_format(), - erase(tlv_format), % used in ber - erase(class_default_type),% used in ber - asn1ct_table:delete(check_functions), - ok. + case specialized_decode_prepare(Gen, M) of + {error,_}=Error -> + Error; + ok -> + asn1ct_gen:pgen(OutFile, Gen, Code), + cleanup_bit_string_format(), + erase(tlv_format), % used in ber + erase(class_default_type), % used in ber + asn1ct_table:delete(check_functions), + ok + end. init_gen_record(EncodingRule, Options) -> Erule = case EncodingRule of @@ -1446,205 +1437,27 @@ prepare_bytes(Bytes) -> list_to_binary(Bytes). vsn() -> ?vsn. -specialized_decode_prepare(#gen{erule=ber,options=Options}=Gen, M) -> +specialized_decode_prepare(#gen{erule=ber,options=Options}=Gen, #module{name=Mod}) -> case lists:member(asn1config, Options) of - true -> - special_decode_prepare_1(Gen, M); - false -> - ok + true -> + case read_config_file(Gen, Mod) of + {ok,ConfigName,ConfigItems} -> + try + asn1ct_partial_decode:prepare(ConfigItems, Mod) + catch + throw:{structured_error,Error} -> + {error,[{structured_error,{ConfigName,none}, + asn1ct_partial_decode,Error}]} + end; + no_config_file -> + ok + end; + false -> + ok end; specialized_decode_prepare(_, _) -> ok. -%% Reads the configuration file if it exists and stores information -%% about partial decode and incomplete decode -special_decode_prepare_1(#gen{options=Options}=Gen, M) -> - %% read configure file - ModName = case lists:keyfind(asn1config, 1, Options) of - {_,MName} -> MName; - false -> M#module.name - end, -%% io:format("ModName: ~p~nM#module.name: ~p~n~n",[ModName,M#module.name]), - case read_config_file(Gen, ModName) of - no_config_file -> - ok; - CfgList -> - SelectedDecode = get_config_info(CfgList,selective_decode), - ExclusiveDecode = get_config_info(CfgList,exclusive_decode), - CommandList = create_partial_decode_gen_info(M#module.name, - SelectedDecode), - %% To convert CommandList to a proper list for the driver change - %% the list:[[choosen,Tag1],skip,[skip_optional,Tag2]] to L = - %% [5,2,Tag1,0,1,Tag2] where 5 is the length, and call - %% port_control(asn1_driver_port,3,[L| Bin]) - save_config(partial_decode,CommandList), - save_gen_state(selective_decode,SelectedDecode), - CommandList2 = create_partial_inc_decode_gen_info(M#module.name, - ExclusiveDecode), - Part_inc_tlv_tags = tlv_tags(CommandList2), - save_config(partial_incomplete_decode,Part_inc_tlv_tags), - save_gen_state(exclusive_decode,ExclusiveDecode,Part_inc_tlv_tags) - end. - -%% create_partial_inc_decode_gen_info/2 -%% -%% Creates a list of tags out of the information in TypeNameList that -%% tells which value will be incomplete decoded, i.e. each end -%% component/type in TypeNameList. The significant types/components in -%% the path from the toptype must be specified in the -%% TypeNameList. Significant elements are all constructed types that -%% branches the path to the leaf and the leaf it self. -%% -%% Returns a list of elements, where an element may be one of -%% mandatory|[opt,Tag]|[bin,Tag]. mandatory correspond to a mandatory -%% element that shall be decoded as usual. [opt,Tag] matches an -%% OPTIONAL or DEFAULT element that shall be decoded as -%% usual. [bin,Tag] corresponds to an element, mandatory, OPTIONAL or -%% DEFAULT, that shall be left encoded (incomplete decoded). -create_partial_inc_decode_gen_info(ModName,{Mod,[{Name,L}|Ls]}) when is_list(L) -> - TopTypeName = partial_inc_dec_toptype(L), - [{Name,TopTypeName, - create_partial_inc_decode_gen_info1(ModName,TopTypeName,{Mod,L})}| - create_partial_inc_decode_gen_info(ModName,{Mod,Ls})]; -create_partial_inc_decode_gen_info(_,{_,[]}) -> - []; -create_partial_inc_decode_gen_info(_,[]) -> - []. - -create_partial_inc_decode_gen_info1(ModName,TopTypeName,{ModName, - [_TopType|Rest]}) -> - case asn1_db:dbget(ModName,TopTypeName) of - #typedef{typespec=TS} -> - TagCommand = get_tag_command(TS,?MANDATORY,mandatory), - create_pdec_inc_command(ModName,get_components(TS#type.def), - Rest,[TagCommand]); - _ -> - throw({error,{"wrong type list in asn1 config file", - TopTypeName}}) - end; -create_partial_inc_decode_gen_info1(M1,_,{M2,_}) when M1 /= M2 -> - throw({error,{"wrong module name in asn1 config file", - M2}}); -create_partial_inc_decode_gen_info1(_,_,TNL) -> - throw({error,{"wrong type list in asn1 config file", - TNL}}). - -%% -%% Only when there is a 'ComponentType' the config data C1 may be a -%% list, where the incomplete decode is branched. So, C1 may be a -%% list, a "binary tuple", a "parts tuple" or an atom. The second -%% element of a binary tuple and a parts tuple is an atom. -create_pdec_inc_command(_ModName,_,[],Acc) -> - lists:reverse(Acc); -create_pdec_inc_command(ModName,{Comps1,Comps2},TNL,Acc) - when is_list(Comps1),is_list(Comps2) -> - create_pdec_inc_command(ModName,Comps1 ++ Comps2,TNL,Acc); -%% The following two clauses match on the type after the top -%% type. This one if the top type had no tag, i.e. a CHOICE. -create_pdec_inc_command(ModN,Clist,[CL|_Rest],[[]]) when is_list(CL) -> - create_pdec_inc_command(ModN,Clist,CL,[]); -create_pdec_inc_command(ModN,Clist,[CL|_Rest],Acc) when is_list(CL) -> - InnerDirectives=create_pdec_inc_command(ModN,Clist,CL,[]), - lists:reverse([InnerDirectives|Acc]); -create_pdec_inc_command(ModName, - CList=[#'ComponentType'{name=Name,typespec=TS, - prop=Prop}|Comps], - TNL=[C1|Cs],Acc) -> - case C1 of - {Name,undecoded} -> - TagCommand = get_tag_command(TS,?UNDECODED,Prop), - create_pdec_inc_command(ModName,Comps,Cs,concat_sequential(TagCommand,Acc)); - {Name,parts} -> - TagCommand = get_tag_command(TS,?PARTS,Prop), - create_pdec_inc_command(ModName,Comps,Cs,concat_sequential(TagCommand,Acc)); - L when is_list(L) -> - %% I guess this never happens due to previous clause. - %% This case is only possible as the first element after - %% the top type element, when top type is SEGUENCE or SET. - %% Follow each element in L. Must note every tag on the - %% way until the last command is reached, but it ought to - %% be enough to have a "complete" or "complete optional" - %% command for each component that is not specified in the - %% config file. Then in the TLV decode the components with - %% a "complete" command will be decoded by an ordinary TLV - %% decode. - create_pdec_inc_command(ModName,CList,L,Acc); - {Name,RestPartsList} when is_list(RestPartsList) -> - %% Same as previous, but this may occur at any place in - %% the structure. The previous is only possible as the - %% second element. - case get_tag_command(TS,?MANDATORY,Prop) of - ?MANDATORY -> - InnerDirectives= - create_pdec_inc_command(ModName,TS#type.def, - RestPartsList,[]), - create_pdec_inc_command(ModName,Comps,Cs, - [[?MANDATORY,InnerDirectives]|Acc]); - [Opt,EncTag] -> - InnerDirectives = - create_pdec_inc_command(ModName,TS#type.def, - RestPartsList,[]), - create_pdec_inc_command(ModName,Comps,Cs, - [[Opt,EncTag,InnerDirectives]|Acc]) - end; - _ -> - %% this component may not be in the config list - TagCommand = get_tag_command(TS,?MANDATORY,Prop), - create_pdec_inc_command(ModName,Comps,TNL,concat_sequential(TagCommand,Acc)) - end; -create_pdec_inc_command(ModName, - {'CHOICE',[#'ComponentType'{name=C1, - typespec=TS, - prop=Prop}|Comps]}, - [{C1,Directive}|Rest],Acc) -> - case Directive of - List when is_list(List) -> - TagCommand = get_tag_command(TS,?ALTERNATIVE,Prop), - CompAcc = - create_pdec_inc_command(ModName, - get_components(TS#type.def),List,[]), - NewAcc = case TagCommand of - [Command,Tag] when is_atom(Command) -> - [[Command,Tag,CompAcc]|Acc]; - [L1,_L2|Rest] when is_list(L1) -> - case lists:reverse(TagCommand) of - [Atom|Comms] when is_atom(Atom) -> - [concat_sequential(lists:reverse(Comms), - [Atom,CompAcc])|Acc]; - [[Command2,Tag2]|Comms] -> - [concat_sequential(lists:reverse(Comms), - [[Command2,Tag2,CompAcc]])|Acc] - end - end, - create_pdec_inc_command(ModName,{'CHOICE',Comps},Rest, - NewAcc); - undecoded -> - TagCommand = get_tag_command(TS,?ALTERNATIVE_UNDECODED,Prop), - create_pdec_inc_command(ModName,{'CHOICE',Comps},Rest, - concat_sequential(TagCommand,Acc)); - parts -> - TagCommand = get_tag_command(TS,?ALTERNATIVE_PARTS,Prop), - create_pdec_inc_command(ModName,{'CHOICE',Comps},Rest, - concat_sequential(TagCommand,Acc)) - end; -create_pdec_inc_command(ModName, - {'CHOICE',[#'ComponentType'{typespec=TS, - prop=Prop}|Comps]}, - TNL,Acc) -> - TagCommand = get_tag_command(TS,?ALTERNATIVE,Prop), - create_pdec_inc_command(ModName,{'CHOICE',Comps},TNL, - concat_sequential(TagCommand,Acc)); -create_pdec_inc_command(M,{'CHOICE',{Cs1,Cs2}},TNL,Acc) - when is_list(Cs1),is_list(Cs2) -> - create_pdec_inc_command(M,{'CHOICE',Cs1 ++ Cs2},TNL,Acc); -create_pdec_inc_command(ModName,#'Externaltypereference'{module=M,type=Name}, - TNL,Acc) -> - #type{def=Def} = get_referenced_type(M,Name), - create_pdec_inc_command(ModName,get_components(Def),TNL,Acc); -create_pdec_inc_command(_,_,TNL,_) -> - throw({error,{"unexpected error when creating partial " - "decode command",TNL}}). - partial_inc_dec_toptype([T|_]) when is_atom(T) -> T; partial_inc_dec_toptype([{T,_}|_]) when is_atom(T) -> @@ -1654,272 +1467,12 @@ partial_inc_dec_toptype([L|_]) when is_list(L) -> partial_inc_dec_toptype(_) -> throw({error,{"no top type found for partial incomplete decode"}}). - -%% Creates a list of tags out of the information in TypeList and Types -%% that tells which value will be decoded. Each constructed type that -%% is in the TypeList will get a "choosen" command. Only the last -%% type/component in the TypeList may be a primitive type. Components -%% "on the way" to the final element may get the "skip" or the -%% "skip_optional" command. -%% CommandList = [Elements] -%% Elements = {choosen,Tag}|{skip_optional,Tag}|skip -%% Tag is a binary with the tag BER encoded. -create_partial_decode_gen_info(ModName,{ModName,TypeLists}) -> - [create_partial_decode_gen_info1(ModName,TL) || TL <- TypeLists]; -create_partial_decode_gen_info(_,[]) -> - []; -create_partial_decode_gen_info(_M1,{M2,_}) -> - throw({error,{"wrong module name in asn1 config file", - M2}}). - -create_partial_decode_gen_info1(ModName,{FuncName,TypeList}) -> - case TypeList of - [TopType|Rest] -> - case asn1_db:dbget(ModName,TopType) of - #typedef{typespec=TS} -> - TagCommand = get_tag_command(TS,?CHOOSEN), - Ret=create_pdec_command(ModName, - get_components(TS#type.def), - Rest,concat_tags(TagCommand,[])), - {FuncName,Ret}; - _ -> - throw({error,{"wrong type list in asn1 config file", - TypeList}}) - end; - _ -> - [] - end; -create_partial_decode_gen_info1(_,_) -> - ok. - -%% create_pdec_command/4 for each name (type or component) in the -%% third argument, TypeNameList, a command is created. The command has -%% information whether the component/type shall be skipped, looked -%% into or returned. The list of commands is returned. -create_pdec_command(_ModName,_,[],Acc) -> - Remove_empty_lists = - fun([[]|L],Res,Fun) -> - Fun(L,Res,Fun); - ([],Res,_) -> - Res; - ([H|L],Res,Fun) -> - Fun(L,[H|Res],Fun) - end, - Remove_empty_lists(Acc,[],Remove_empty_lists); -create_pdec_command(ModName,[#'ComponentType'{name=C1,typespec=TS}|_Comps], - [C1|Cs],Acc) -> - %% this component is a constructed type or the last in the - %% TypeNameList otherwise the config spec is wrong - TagCommand = get_tag_command(TS,?CHOOSEN), - create_pdec_command(ModName,get_components(TS#type.def), - Cs,concat_tags(TagCommand,Acc)); -create_pdec_command(ModName,[#'ComponentType'{typespec=TS, - prop=Prop}|Comps], - [C2|Cs],Acc) -> - TagCommand = - case Prop of - mandatory -> - get_tag_command(TS,?SKIP); - _ -> - get_tag_command(TS,?SKIP_OPTIONAL) - end, - create_pdec_command(ModName,Comps,[C2|Cs],concat_tags(TagCommand,Acc)); -create_pdec_command(ModName,{'CHOICE',[Comp=#'ComponentType'{name=C1}|_]},TNL=[C1|_Cs],Acc) -> - create_pdec_command(ModName,[Comp],TNL,Acc); -create_pdec_command(ModName,{'CHOICE',[#'ComponentType'{}|Comps]},TNL,Acc) -> - create_pdec_command(ModName,{'CHOICE',Comps},TNL,Acc); -create_pdec_command(ModName,{'CHOICE',{Cs1,Cs2}},TNL,Acc) - when is_list(Cs1),is_list(Cs2) -> - create_pdec_command(ModName,{'CHOICE',Cs1 ++ Cs2},TNL,Acc); -create_pdec_command(ModName,#'Externaltypereference'{module=M,type=C1}, - TypeNameList,Acc) -> - #type{def=Def} = get_referenced_type(M,C1), - create_pdec_command(ModName,get_components(Def),TypeNameList, - Acc); -create_pdec_command(ModName,TS=#type{def=Def},[C1|Cs],Acc) -> - %% This case when we got the "components" of a SEQUENCE/SET OF - case C1 of - [1] -> - %% A list with an integer is the only valid option in a 'S - %% OF', the other valid option would be an empty - %% TypeNameList saying that the entire 'S OF' will be - %% decoded. - TagCommand = get_tag_command(TS,?CHOOSEN), - create_pdec_command(ModName,Def,Cs,concat_tags(TagCommand,Acc)); - [N] when is_integer(N) -> - TagCommand = get_tag_command(TS,?SKIP), - create_pdec_command(ModName,Def,[[N-1]|Cs], - concat_tags(TagCommand,Acc)); - Err -> - throw({error,{"unexpected error when creating partial " - "decode command",Err}}) - end; -create_pdec_command(_,_,TNL,_) -> - throw({error,{"unexpected error when creating partial " - "decode command",TNL}}). - -get_components(#'SEQUENCE'{components={C1,C2}}) when is_list(C1),is_list(C2) -> - C1++C2; -get_components(#'SEQUENCE'{components=Components}) -> - Components; -get_components(#'SET'{components={C1,C2}}) when is_list(C1),is_list(C2) -> - C1++C2; -get_components(#'SET'{components=Components}) -> - Components; -get_components({'SEQUENCE OF',Components}) -> - Components; -get_components({'SET OF',Components}) -> - Components; -get_components(Def) -> - Def. - -concat_sequential(L=[A,B],Acc) when is_atom(A),is_binary(B) -> - [L|Acc]; -concat_sequential(L,Acc) when is_list(L) -> - concat_sequential1(lists:reverse(L),Acc); -concat_sequential(A,Acc) -> - [A|Acc]. -concat_sequential1([],Acc) -> - Acc; -concat_sequential1([[]],Acc) -> - Acc; -concat_sequential1([El|RestEl],Acc) when is_list(El) -> - concat_sequential1(RestEl,[El|Acc]); -concat_sequential1([mandatory|RestEl],Acc) -> - concat_sequential1(RestEl,[mandatory|Acc]); -concat_sequential1(L,Acc) -> - [L|Acc]. - - -many_tags([?SKIP])-> - false; -many_tags([?SKIP_OPTIONAL,_]) -> - false; -many_tags([?CHOOSEN,_]) -> - false; -many_tags(_) -> - true. - -concat_tags(Ts,Acc) -> - case many_tags(Ts) of - true when is_list(Ts) -> - lists:reverse(Ts)++Acc; - true -> - [Ts|Acc]; - false -> - [Ts|Acc] - end. -%% get_tag_command(Type,Command) - -%% Type is the type that has information about the tag Command tells -%% what to do with the encoded value with the tag of Type when -%% decoding. -get_tag_command(#type{tag=[]},_) -> - []; -%% SKIP and SKIP_OPTIONAL shall return only one tag command regardless -get_tag_command(#type{},?SKIP) -> - ?SKIP; -get_tag_command(#type{tag=Tags},?SKIP_OPTIONAL) -> - Tag=hd(Tags), - [?SKIP_OPTIONAL,encode_tag_val(decode_class(Tag#tag.class), - Tag#tag.form,Tag#tag.number)]; -get_tag_command(#type{tag=[Tag]},Command) -> - %% encode the tag according to BER - [Command,encode_tag_val(decode_class(Tag#tag.class),Tag#tag.form, - Tag#tag.number)]; -get_tag_command(T=#type{tag=[Tag|Tags]},Command) -> - TC = get_tag_command(T#type{tag=[Tag]},Command), - TCs = get_tag_command(T#type{tag=Tags},Command), - case many_tags(TCs) of - true when is_list(TCs) -> - [TC|TCs]; - _ -> [TC|[TCs]] - end. - -%% get_tag_command/3 used by create_pdec_inc_command -get_tag_command(#type{tag=[]},_,_) -> - []; -get_tag_command(#type{tag=[Tag]},?MANDATORY,Prop) -> - case Prop of - mandatory -> - ?MANDATORY; - {'DEFAULT',_} -> - [?DEFAULT,encode_tag_val(decode_class(Tag#tag.class), - Tag#tag.form,Tag#tag.number)]; - _ -> [?OPTIONAL,encode_tag_val(decode_class(Tag#tag.class), - Tag#tag.form,Tag#tag.number)] - end; -get_tag_command(#type{tag=[Tag]},Command,Prop) -> - [anonymous_dec_command(Command,Prop),encode_tag_val(decode_class(Tag#tag.class),Tag#tag.form, Tag#tag.number)]; -get_tag_command(#type{tag=Tag},Command,Prop) when is_record(Tag,tag) -> - get_tag_command(#type{tag=[Tag]},Command,Prop); -get_tag_command(T=#type{tag=[Tag|Tags]},Command,Prop) -> - [get_tag_command(T#type{tag=[Tag]},Command,Prop)|[ - get_tag_command(T#type{tag=Tags},Command,Prop)]]. - -anonymous_dec_command(?UNDECODED,'OPTIONAL') -> - ?OPTIONAL_UNDECODED; -anonymous_dec_command(Command,_) -> - Command. - -get_referenced_type(M,Name) -> - case asn1_db:dbget(M,Name) of - #typedef{typespec=TS} -> - case TS of - #type{def=#'Externaltypereference'{module=M2,type=Name2}} -> - %% The tags have already been taken care of in the - %% first reference where they were gathered in a - %% list of tags. - get_referenced_type(M2,Name2); - #type{} -> TS; - _ -> - throw({error,{"unexpected element when" - " fetching referenced type",TS}}) - end; - T -> - throw({error,{"unexpected element when fetching " - "referenced type",T}}) - end. - - -tlv_tags([]) -> - []; -tlv_tags([mandatory|Rest]) -> - [mandatory|tlv_tags(Rest)]; -tlv_tags([[Command,Tag]|Rest]) when is_atom(Command),is_binary(Tag) -> - [[Command,tlv_tag(Tag)]|tlv_tags(Rest)]; -tlv_tags([[Command,Directives]|Rest]) when is_atom(Command),is_list(Directives) -> - [[Command,tlv_tags(Directives)]|tlv_tags(Rest)]; -%% remove all empty lists -tlv_tags([[]|Rest]) -> - tlv_tags(Rest); -tlv_tags([{Name,TopType,L1}|Rest]) when is_list(L1),is_atom(TopType) -> - [{Name,TopType,tlv_tags(L1)}|tlv_tags(Rest)]; -tlv_tags([[Command,Tag,L1]|Rest]) when is_list(L1),is_binary(Tag) -> - [[Command,tlv_tag(Tag),tlv_tags(L1)]|tlv_tags(Rest)]; -tlv_tags([[mandatory|Rest]]) -> - [[mandatory|tlv_tags(Rest)]]; -tlv_tags([L=[L1|_]|Rest]) when is_list(L1) -> - [tlv_tags(L)|tlv_tags(Rest)]. - -tlv_tag(<>) when TagNo < 31 -> - (Cl bsl 16) + TagNo; -tlv_tag(<>) -> - (Cl bsl 16) + TagNo; -tlv_tag(<>) -> - TagNo = tlv_tag1(Buffer,0), - (Cl bsl 16) + TagNo. -tlv_tag1(<<0:1,PartialTag:7>>,Acc) -> - (Acc bsl 7) bor PartialTag; -tlv_tag1(<<1:1,PartialTag:7,Buffer/binary>>,Acc) -> - tlv_tag1(Buffer,(Acc bsl 7) bor PartialTag). - %% Reads the content from the configuration file and returns the %% selected part chosen by InfoType. Assumes that the config file %% content is an Erlang term. read_config_file_info(ModuleName, InfoType) when is_atom(InfoType) -> Name = ensure_ext(ModuleName, ".asn1config"), - CfgList = read_config_file0(Name, []), + {ok,_,CfgList} = read_config_file0(Name, []), get_config_info(CfgList, InfoType). read_config_file(#gen{options=Options}, ModuleName) -> @@ -1927,12 +1480,13 @@ read_config_file(#gen{options=Options}, ModuleName) -> Includes = [I || {i,I} <- Options], read_config_file0(Name, ["."|Includes]). -read_config_file0(Name, [D|Dirs]) -> - case file:consult(filename:join(D, Name)) of +read_config_file0(Name0, [Dir|Dirs]) -> + Name = filename:join(Dir, Name0), + case file:consult(Name) of {ok,CfgList} -> - CfgList; + {ok,Name,CfgList}; {error,enoent} -> - read_config_file0(Name, Dirs); + read_config_file0(Name0, Dirs); {error,Reason} -> Error = "error reading asn1 config file: " ++ file:format_error(Reason), @@ -2396,7 +1950,7 @@ verbose(Format, Args, S) -> end. format_error({write_error,File,Reason}) -> - io_lib:format("writing output file ~s failed: ~s", + io_lib:format(<<"writing output file ~ts failed: ~s">>, [File,file:format_error(Reason)]). is_error(#state{options=Opts}) -> diff --git a/lib/asn1/src/asn1ct_check.erl b/lib/asn1/src/asn1ct_check.erl index f1e8a1912895..4c7f63ee92ee 100644 --- a/lib/asn1/src/asn1ct_check.erl +++ b/lib/asn1/src/asn1ct_check.erl @@ -4934,7 +4934,7 @@ componentrelation_leadingattr(S,CompList) -> %% get_simple_table_if_used/2 should find out whether there are any %% component relation constraints in the entire tree of Cs1 that %% relates to this level. It returns information about the simple - %% table constraint necessary for the the call to + %% table constraint necessary for the call to %% componentrelation_leadingattr/6. The step when the leading %% attribute and the syntax tree is modified to support the code %% generating. diff --git a/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl b/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl index 84680107269f..d6049a12a345 100644 --- a/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl +++ b/lib/asn1/src/asn1ct_constructed_ber_bin_v2.erl @@ -1194,14 +1194,23 @@ gen_dec_line(Erules,TopType,Cname,CTags,Type,OptOrMand,DecObjInf) -> Pdec; _ -> - emit(["[{",{asis,FirstTag}, - ",",{curr,v},"}|Temp", - {curr,tlv}, - "] ->",nl]), + DecTag = + case asn1ct:get_gen_state_field(namelist) of + [{Cname,undecoded}|_] -> + emit(["[",{curr,v},"|Temp",{curr,tlv},"] ", + "when is_binary(",{curr,v},") ->",nl]), + Tag; + _ -> + emit(["[{",{asis,FirstTag}, + ",",{curr,v},"}|Temp", + {curr,tlv}, + "] ->",nl]), + RestTag + end, emit([indent(4),"{"]), Pdec= gen_dec_call(InnerType,Erules,TopType,Cname, - Type,BytesVar,RestTag,mandatory, + Type,BytesVar,DecTag,mandatory, ", mandatory, ",DecObjInf, OptOrMand), @@ -1358,10 +1367,9 @@ gen_dec_call1(WhatKind, _, TopType, Cname, Type, BytesVar, Tag) -> %% This is to prepare SEQUENCE OF value in %% partial incomplete decode for a later %% part-decode, i.e. skip %% the tag. - asn1ct:add_generated_refed_func({[Cname|TopType], - parts, - [],Type}), - emit(["{'",asn1ct_gen:list2name([Cname|TopType]),"',"]), + Id = [parts,Cname|TopType], + asn1ct:add_generated_refed_func({Id,parts,[],Type}), + emit(["{'",asn1ct_gen:list2name(Id),"',"]), asn1ct_func:need({ber,match_tags,2}), EmitDecFunCall("match_tags"), emit("}"); diff --git a/lib/asn1/src/asn1ct_gen.erl b/lib/asn1/src/asn1ct_gen.erl index 5305260f95c7..308781491ac2 100644 --- a/lib/asn1/src/asn1ct_gen.erl +++ b/lib/asn1/src/asn1ct_gen.erl @@ -295,7 +295,7 @@ pgen_partial_types1(Erules,[{FuncName,[TopType|RestTypes]}|Rest]) -> CurrMod = get(currmod), TypeDef = asn1_db:dbget(CurrMod,TopType), traverse_type_structure(Erules,TypeDef,RestTypes,FuncName, - TypeDef#typedef.name), + [TypeDef#typedef.name]), pgen_partial_types1(Erules,Rest); pgen_partial_types1(_,[]) -> ok; @@ -479,24 +479,20 @@ pgen_partial_incomplete_decode1(#gen{erule=ber}) -> gen_part_decode_funcs(GeneratedFs,0); pgen_partial_incomplete_decode1(#gen{}) -> ok. -emit_partial_incomplete_decode({FuncName,TopType,Pattern}) -> +emit_partial_incomplete_decode({FuncName,TopType,Pattern}) + when is_atom(TopType) -> TypePattern = asn1ct:get_gen_state_field(inc_type_pattern), - TPattern = - case lists:keysearch(FuncName,1,TypePattern) of - {value,{_,TP}} -> TP; - _ -> exit({error,{asn1_internal_error,exclusive_decode}}) - end, + {_,TPattern} = lists:keyfind(FuncName, 1, TypePattern), TopTypeName = - case asn1ct:maybe_saved_sindex(TopType,TPattern) of - I when is_integer(I),I>0 -> - lists:concat([TopType,"_",I]); - _ -> - atom_to_list(TopType) - end, + case asn1ct:maybe_saved_sindex(TopType, TPattern) of + I when is_integer(I), I > 0 -> + list_to_atom(lists:concat([TopType,"_",I])); + _ -> + TopType + end, emit([{asis,FuncName},"(Bytes) ->",nl, - " decode_partial_incomplete('",TopTypeName,"',Bytes,",{asis,Pattern},").",nl]); -emit_partial_incomplete_decode(D) -> - throw({error,{asn1,{"bad data in asn1config file",D}}}). + " decode_partial_incomplete(",{asis,TopTypeName},", Bytes, ", + {asis,Pattern},").",nl]). gen_part_decode_funcs([Data={Name,_,_,Type}|GeneratedFs],N) -> InnerType = @@ -507,12 +503,18 @@ gen_part_decode_funcs([Data={Name,_,_,Type}|GeneratedFs],N) -> get_inner(Type#type.def) end, WhatKind = type(InnerType), - TypeName=list2name(Name), + DispatchId = list_to_atom(list2name(Name)), + TypeName = case Name of + [parts|TypeName0] -> + list2name(TypeName0); + _ -> + list2name(Name) + end, if N > 0 -> emit([";",nl]); true -> ok end, - emit(["decode_inc_disp('",TypeName,"',Data) ->",nl]), + emit(["decode_inc_disp(",{asis,DispatchId},",Data) ->",nl]), gen_part_decode_funcs(WhatKind,TypeName,Data), gen_part_decode_funcs(GeneratedFs,N+1); gen_part_decode_funcs([_H|T],N) -> @@ -779,7 +781,7 @@ pgen_dispatcher(Gen, Types) -> emit(["try ",Call," of",nl, " Bytes ->",nl, " {ok,Bytes}",nl, - try_catch()]) + try_catch(),".",nl]) end, emit([nl,nl]), @@ -794,7 +796,7 @@ pgen_dispatcher(Gen, Types) -> emit(["try ",JerCall," of",nl, " Bytes ->",nl, " {ok,Bytes}",nl, - try_catch()]) + try_catch(),".",nl]) end, emit([nl,nl]); false -> @@ -854,7 +856,7 @@ pgen_dispatcher(Gen, Types) -> case NoOkWrapper of false -> - emit([nl,try_catch(),nl,nl]); + emit([nl,try_catch(),".",nl,nl]); true -> emit([".",nl,nl]) end, @@ -874,7 +876,7 @@ pgen_dispatcher(Gen, Types) -> result_line(false, ["Result"]), case NoOkWrapper of false -> - emit([nl,try_catch(),nl,nl]); + emit([nl,try_catch(),".",nl,nl]); true -> emit([".",nl,nl]) end; @@ -884,7 +886,7 @@ pgen_dispatcher(Gen, Types) -> %% REST of MODULE - gen_decode_partial_incomplete(Gen), + gen_decode_partial_incomplete(Gen, NoOkWrapper), gen_partial_inc_dispatcher(Gen), case Gen of @@ -916,7 +918,7 @@ try_catch() -> " Reason ->",nl, " {error,{asn1,{Reason,Stk}}}",nl, " end",nl, - "end."]. + "end"]. gen_info_functions(Gen) -> Erule = case Gen of @@ -938,7 +940,7 @@ gen_info_functions(Gen) -> "legacy_erlang_types() -> ", {asis,asn1ct:use_legacy_types()},".",nl,nl]). -gen_decode_partial_incomplete(#gen{erule=ber}) -> +gen_decode_partial_incomplete(#gen{erule=ber}, NoOkWrapper) -> case {asn1ct:read_config_data(partial_incomplete_decode), asn1ct:get_gen_state_field(inc_type_pattern)} of {undefined,_} -> @@ -946,37 +948,44 @@ gen_decode_partial_incomplete(#gen{erule=ber}) -> {_,undefined} -> ok; _ -> - EmitCaseClauses = - fun() -> - emit([" {'EXIT',{error,Reason}} ->",nl, - " {error,Reason};",nl, - " {'EXIT',Reason} ->",nl, - " {error,{asn1,Reason}};",nl, - " Result ->",nl, - " {ok,Result}",nl, - " end"]) - end, - emit(["decode_partial_incomplete(Type,Data0,", - "Pattern) ->",nl]), - emit([" {Data,_RestBin} =",nl, - " ",{call,ber,decode_primitive_incomplete, - ["Pattern","Data0"]},com,nl, - " case catch decode_partial_inc_disp(Type,", - "Data) of",nl]), - EmitCaseClauses(), - emit([".",nl,nl]), - emit(["decode_part(Type, Data0) " - "when is_binary(Data0) ->",nl]), - emit([" case catch decode_inc_disp(Type,element(1, ", - {call,ber,ber_decode_nif,["Data0"]},")) of",nl]), - EmitCaseClauses(), - emit([";",nl]), - emit(["decode_part(Type, Data0) ->",nl]), - emit([" case catch decode_inc_disp(Type, Data0) of",nl]), - EmitCaseClauses(), - emit([".",nl,nl]) + emit(["decode_partial_incomplete(Type, Data0, Pattern) ->",nl, + " {Data,_RestBin} =",nl, + " ",{call,ber,decode_primitive_incomplete, + ["Pattern","Data0"]},com,nl]), + case NoOkWrapper of + true -> + emit([" decode_partial_inc_disp(Type, Data)",nl]); + false -> + emit([" try {ok,decode_partial_inc_disp(Type, Data)}",nl, + try_catch()]) + end, + emit([".",nl,nl]), + + emit(["decode_part(Type, Data0) when is_binary(Data0) ->",nl]), + case NoOkWrapper of + true -> + emit([" decode_inc_disp(Type, element(1, ", + {call,ber,ber_decode_nif,["Data0"]}, + "))",nl]); + false -> + emit([" try {ok,decode_inc_disp(Type, element(1, ", + {call,ber,ber_decode_nif,["Data0"]}, + "))}",nl, + try_catch()]) + end, + emit([";",nl]), + + emit(["decode_part(Type, Data0) ->",nl]), + case NoOkWrapper of + true -> + emit([" decode_inc_disp(Type, Data0)"]); + false -> + emit([" try {ok,decode_inc_disp(Type, Data0)}",nl, + try_catch()]) + end, + emit([".",nl,nl]) end; -gen_decode_partial_incomplete(#gen{}) -> +gen_decode_partial_incomplete(#gen{}, _) -> ok. gen_partial_inc_dispatcher(#gen{erule=ber}) -> diff --git a/lib/asn1/src/asn1ct_gen_ber_bin_v2.erl b/lib/asn1/src/asn1ct_gen_ber_bin_v2.erl index 0e7d07e92acd..1f0ac295828b 100644 --- a/lib/asn1/src/asn1ct_gen_ber_bin_v2.erl +++ b/lib/asn1/src/asn1ct_gen_ber_bin_v2.erl @@ -30,7 +30,7 @@ -export([gen_encode_prim/4]). -export([gen_dec_prim/3]). -export([gen_objectset_code/2, gen_obj_code/3]). --export([encode_tag_val/3]). +-export([encode_tag_val/3,tag_to_integer/1]). -export([gen_inc_decode/2,gen_decode_selected/3]). -export([extaddgroup2sequence/1]). -export([dialyzer_suppressions/1]). @@ -72,16 +72,33 @@ dialyzer_suppressions(_) -> false -> ok; true -> suppress({ber,encode_bit_string,4}) end, - suppress({ber,decode_selective,2}), + + %% The `no_match` option for dialyzer will suppress clauses whose + %% patterns will never match. Here we must ensure that dialyzer + %% sees calls to all helper functions to avoid warnings for + %% functions that will never be called. + %% + %% We provide argument lists that avoid dialyzer warnings, + %% but stills allows the compiler to do some optimizations. + + Args1 = ["element(5, Arg)", "[skip,{skip_optional,<<0>>},{chosen,<<0>>}]"], + suppress({ber,decode_selective,2}, Args1), + + Args2 = ["[{undecoded,0},{alt_parts,0}]", "element(6, Arg)"], + suppress({ber,decode_primitive_incomplete,2}, Args2), + emit([" ok.",nl]). -suppress({M,F,A}=MFA) -> +suppress({_,_,A}=MFA) -> + Args = [lists:concat(["element(",I,", Arg)"]) || I <- lists:seq(1, A)], + suppress(MFA, Args). + +suppress({M,F,_}=MFA, Args) -> case asn1ct_func:is_used(MFA) of false -> ok; true -> - Args = [lists:concat(["element(",I,", Arg)"]) || I <- lists:seq(1, A)], - emit([" ",{call,M,F,Args},com,nl]) + emit([" _ = ",{call,M,F,Args},com,nl]) end. %%=============================================================================== @@ -353,19 +370,25 @@ gen_inc_decode(Erules,Type) when is_record(Type,typedef) -> gen_decode_selected(Erules,Type,FuncName) -> emit([FuncName,"(Bin) ->",nl]), Patterns = asn1ct:read_config_data(partial_decode), - Pattern = - case lists:keysearch(FuncName,1,Patterns) of - {value,{_,P}} -> P; - false -> exit({error,{internal,no_pattern_saved}}) - end, + {_,Pattern} = lists:keyfind(FuncName, 1, Patterns), emit([" case ",{call,ber,decode_selective, [{asis,Pattern},"Bin"]}," of",nl, " {ok,Bin2} when is_binary(Bin2) ->",nl, " {Tlv,_} = ", {call,ber,ber_decode_nif,["Bin2"]},com,nl]), - emit("{ok,"), - gen_decode_selected_type(Erules,Type), - emit(["};",nl," Err -> exit({error,{selective_decode,Err}})",nl, - " end.",nl]). + NoOkWrapper = proplists:get_bool(no_ok_wrapper, Erules#gen.options), + case NoOkWrapper of + true -> ok; + false -> emit("{ok,") + end, + gen_decode_selected_type(Erules, Type), + case NoOkWrapper of + true -> + ok; + false -> + emit(["};",nl, + " Err -> exit({error,{selective_decode,Err}})"]) + end, + emit([" end.",nl]). gen_decode_selected_type(_Erules,TypeDef) -> Def = TypeDef#typedef.typespec, @@ -382,14 +405,9 @@ gen_decode_selected_type(_Erules,TypeDef) -> asn1ct_name:new(len), gen_dec_prim(Def, BytesVar, Tag); {constructed,bif} -> - TopType = case TypeDef#typedef.name of - A when is_atom(A) -> [A]; - N -> N - end, - DecFunName = lists:concat(["'",dec,"_", - asn1ct_gen:list2name(TopType),"'"]), - emit([DecFunName,"(",BytesVar, - ", ",{asis,Tag},")"]); + TopType = TypeDef#typedef.name, + DecFunName = dec_func(asn1ct_gen:list2name(TopType)), + emit([DecFunName,"(",BytesVar,", ",{asis,Tag},")"]); TheType -> DecFunName = mkfuncname(TheType,dec), emit([DecFunName,"(",BytesVar, @@ -1527,6 +1545,10 @@ get_object_field(Name,ObjectFields) -> false -> false end. +tag_to_integer(#tag{class=Class,number=N}) + when is_integer(N), 0 =< N, N =< 1 bsl 16 -> + decode_class(Class) bsl 10 bor N. + %%encode_tag(TagClass(?UNI, APP etc), Form (?PRIM etx), TagInteger) -> %% 8bit Int | binary encode_tag_val(Class, Form, TagNo) when (TagNo =< 30) -> diff --git a/lib/asn1/src/asn1ct_imm.erl b/lib/asn1/src/asn1ct_imm.erl index c2ec27e19592..1f1cf81bd4fb 100644 --- a/lib/asn1/src/asn1ct_imm.erl +++ b/lib/asn1/src/asn1ct_imm.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2012-2022. All Rights Reserved. +%% Copyright Ericsson AB 2012-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -498,7 +498,7 @@ is_aligned(k_m_string, _Lb, Ub) -> Ub >= 16. %%% -%%% Generating the intermediate format format for decoding. +%%% Generating the intermediate format for decoding. %%% dec_string(Sv, U, Aligned0, T) when is_integer(Sv) -> @@ -1791,7 +1791,7 @@ enc_pre_cg_nonbuilding(Imm, _) -> Imm. %%% an expensive complete/1 implementation). If we can be sure that %%% complete/1 will be called with an iolist (no 'align' atoms or %%% bitstrings in the list), we can call iolist_to_binary/1 -%%% instead. If the list may include bitstrings, we can can call +%%% instead. If the list may include bitstrings, we can call %%% list_to_bitstring/1 (note that list_to_bitstring/1 does not accept %%% a binary or bitstring, so we MUST be sure that we only pass it a %%% list). If complete/1 is called with a binary, we can omit the @@ -2199,7 +2199,7 @@ propagate({var,Var}, Propagate, #ost{sym=Sym0}=St) when is_function(Propagate, 2 %%% any Anything. %%% %%% align Basically iodata, but the list may contain bitstrings -%%% and the the atom 'align'. Can be passed to complete/1 +%%% and the atom 'align'. Can be passed to complete/1 %%% to construct a binary. Only used for aligned PER (per). %%% %%% bitstring An Erlang bitstring. diff --git a/lib/asn1/src/asn1ct_parser2.erl b/lib/asn1/src/asn1ct_parser2.erl index 2bdc284b1271..a86fd7f3399c 100644 --- a/lib/asn1/src/asn1ct_parser2.erl +++ b/lib/asn1/src/asn1ct_parser2.erl @@ -2,7 +2,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2000-2021. All Rights Reserved. +%% Copyright Ericsson AB 2000-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -283,7 +283,7 @@ parse_Assignment([{typereference,_,_},{'{',_}|_]=Tokens) -> %% 2) ValueSet{...} Type ::= ... %% ObjectSet{...} CLASS-NAME ::= CLASS {...} %% 3) CLASS-NAME{...} ::= CLASS {...} - %% A parameterized value set and and a parameterized object set + %% A parameterized value set and a parameterized object set %% cannot be distinguished from each other without type information. Flist = [fun parse_ParameterizedTypeAssignment/1, fun parse_ParameterizedValueSetTypeAssignment/1, diff --git a/lib/asn1/src/asn1ct_partial_decode.erl b/lib/asn1/src/asn1ct_partial_decode.erl new file mode 100644 index 000000000000..c4df9be4df87 --- /dev/null +++ b/lib/asn1/src/asn1ct_partial_decode.erl @@ -0,0 +1,396 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2023. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +-module(asn1ct_partial_decode). +-export([prepare/2,format_error/1]). + +-include("asn1_records.hrl"). + +prepare(Items, Mod) -> + _ = [prepare_item(Item, Mod) || Item <- Items], + ok. + +format_error({bad_decode_instruction,Term}) -> + io_lib:format(<<"badly formed exclusive decode instruction: ~p">>, + [Term]); +format_error({bad_exclusive_decode,Term}) -> + io_lib:format(<<"badly formed exclusive decode instructions: ~p">>, + [Term]); +format_error({bad_module_name,Mod,ShouldBe}) -> + io_lib:format(<<"the module name is ~p; expected it to be the same as the name" + " of the ASN.1 module (~p)">>, + [Mod,ShouldBe]); +format_error({bad_selective_decode,Term}) -> + io_lib:format(<<"badly formed selective decode instructions: ~p">>, + [Term]); +format_error({bad_selective_decode_element,Term}) -> + io_lib:format(<<"badly formed element in selective decode instruction: ~p">>, + [Term]); +format_error({bad_selective_decode_type_list,Term}) -> + io_lib:format(<<"badly formed type list in selective decode instruction: ~p">>, + [Term]); +format_error({stepping_into_primitive,Path}) -> + io_lib:format(<<"the tail end of selective decode instructions attempts to ", + "step into a primitive type:\n ~p">>, + [Path]); +format_error({undefined_name,Type}) -> + io_lib:format(<<"name ~p not found">>, [Type]); +format_error({undefined_type,Type}) -> + io_lib:format(<<"type ~p does not exist">>, [Type]). + +%%% +%%% Common macros. +%%% + +-define(ASN1CT_GEN_BER, asn1ct_gen_ber_bin_v2). + +%%% +%%% Start of local functions. +%%% + +prepare_item({selective_decode,SelectedDecode}, Mod) -> + CommandList = selective_decode(Mod, SelectedDecode), + asn1ct:save_config(partial_decode, CommandList), + asn1ct:save_gen_state(selective_decode, SelectedDecode), + ok; +prepare_item({exclusive_decode,ExclusiveDecode}, Mod) -> + ExclusiveCommands = exclusive_decode(Mod, ExclusiveDecode), + asn1ct:save_config(partial_incomplete_decode, ExclusiveCommands), + asn1ct:save_gen_state(exclusive_decode, ExclusiveDecode, ExclusiveCommands), + ok. + +%%% +%%% Handle exclusive decode. +%%% + +-define(MANDATORY, mandatory). +-define(DEFAULT, default). +-define(DEFAULT_UNDECODED, default_undecoded). +-define(OPTIONAL, opt). +-define(OPTIONAL_UNDECODED, opt_undecoded). +-define(PARTS, parts). +-define(UNDECODED, undecoded). +-define(ALTERNATIVE, alt). +-define(ALTERNATIVE_UNDECODED, alt_undecoded). +-define(ALTERNATIVE_PARTS, alt_parts). + +exclusive_decode(Mod, {ModI,Instructions}) when is_list(Instructions) -> + if + Mod =:= ModI -> + exclusive_decode_1(Mod, Instructions); + true -> + cfg_error({bad_module_name,ModI,Mod}) + end; +exclusive_decode(_Mod, Term) -> + cfg_error({bad_exclusive_decode,Term}). + +exclusive_decode_1(Mod, [{FunName,[TopType,Directives0]}|Is]) + when is_atom(FunName), is_atom(TopType), is_list(Directives0) -> + Directives = exclusive_decode_map(Directives0), + [{FunName,TopType,exclusive_decode_2(Mod, TopType, Directives)} | + exclusive_decode_1(Mod, Is)]; +exclusive_decode_1(_Mod, []) -> + []; +exclusive_decode_1(_Mod, Term) -> + cfg_error({bad_exclusive_decode,Term}). + +exclusive_decode_2(ModName, TopType, Directives) -> + case asn1_db:dbget(ModName, TopType) of + #typedef{typespec=TS} -> + Acc = get_tag_command(TS, ?MANDATORY, mandatory), + exclusive_decode_command(get_components(TS#type.def), + Directives, Acc); + undefined -> + cfg_error({undefined_type,TopType}) + end. + +exclusive_decode_map(Commands) -> + exclusive_decode_map(Commands, []). + +exclusive_decode_map([H|T], Acc) -> + case H of + {Name,Command0} when is_atom(Name) -> + Command = + if + Command0 =:= ?UNDECODED; Command0 =:= ?PARTS -> + Command0; + is_list(Command0) -> + exclusive_decode_map(Command0, []); + true -> + cfg_error({bad_decode_instruction,H}) + end, + exclusive_decode_map(T, [{Name,Command}|Acc]); + _ -> + cfg_error({bad_decode_instruction,H}) + end; +exclusive_decode_map([], Acc) -> + maps:from_list(Acc). + +exclusive_decode_command(_, Commands, Acc) when map_size(Commands) =:= 0 -> + lists:reverse(Acc); +exclusive_decode_command([#'ComponentType'{name=Name,typespec=TS, + prop=Prop}|Comps], + Commands0, Acc) -> + case maps:take(Name, Commands0) of + {Command,Commands} when is_atom(Command) -> + TagCommands = get_tag_command(TS, Command, Prop), + exclusive_decode_command(Comps, Commands, TagCommands++Acc); + {InnerCommands0,Commands} when is_map(Commands0) -> + InnerCommands = exclusive_decode_command(TS#type.def, + InnerCommands0, []), + case get_tag_command(TS, ?MANDATORY, Prop) of + [?MANDATORY] -> + exclusive_decode_command(Comps, Commands, + [{?MANDATORY,InnerCommands}|Acc]); + [{Opt,EncTag}] -> + exclusive_decode_command(Comps, Commands, + [{Opt,EncTag,InnerCommands}|Acc]) + end; + error -> + case get_tag_command(TS, ?MANDATORY, Prop) of + [] -> + case TS of + #type{def=#'Externaltypereference'{}} -> + exclusive_decode_command(Comps, Commands0, [mandatory|Acc]); + _ -> + exclusive_decode_command(Comps, Commands0, Acc) + end; + [_|_]=TagCommands -> + exclusive_decode_command(Comps, Commands0, TagCommands ++ Acc) + end + end; +exclusive_decode_command({'CHOICE',[_|_]=Cs}, Commands, Acc) -> + exclusive_decode_choice_cs(Cs, Commands, Acc); +exclusive_decode_command({'CHOICE',{Cs1,Cs2}}, Commands, Acc) + when is_list(Cs1), is_list(Cs2) -> + exclusive_decode_choice_cs(Cs1 ++ Cs2, Commands, Acc); +exclusive_decode_command(#'Externaltypereference'{module=M,type=Name}, + Commands, Acc) -> + #type{def=Def} = get_referenced_type(M, Name), + exclusive_decode_command(get_components(Def), Commands, Acc); +exclusive_decode_command([], Commands, _) -> + [{Name,_}|_] = maps:to_list(maps:iterator(Commands, ordered)), + cfg_error({undefined_name,Name}). + +exclusive_decode_choice_cs(_, Commands, Acc) when map_size(Commands) =:= 0 -> + lists:reverse(Acc); +exclusive_decode_choice_cs([#'ComponentType'{name=Name,typespec=TS}|Cs], + Commands0, Acc) -> + case maps:take(Name, Commands0) of + {Inner,Commands} -> + case Inner of + ?UNDECODED -> + TagCommands = get_tag_command(TS, ?ALTERNATIVE_UNDECODED, mandatory), + exclusive_decode_choice_cs(Cs, Commands, TagCommands ++ Acc); + ?PARTS -> + TagCommands = get_tag_command(TS, ?ALTERNATIVE_PARTS, mandatory), + exclusive_decode_choice_cs(Cs, Commands, TagCommands ++ Acc); + _ when is_map(Inner) -> + [{Command,Tag}] = get_tag_command(TS, ?ALTERNATIVE, mandatory), + CompAcc = exclusive_decode_command(get_components(TS#type.def), Inner, []), + exclusive_decode_choice_cs(Cs, Commands, [{Command,Tag,CompAcc}|Acc]) + end; + error -> + TagCommands = get_tag_command(TS, ?ALTERNATIVE, mandatory), + exclusive_decode_choice_cs(Cs, Commands0, TagCommands ++ Acc) + end. + +get_tag_command(#type{tag=[]}, _, _) -> + []; +get_tag_command(#type{tag=[Tag]}, ?MANDATORY, Prop) -> + [case Prop of + mandatory -> + ?MANDATORY; + {'DEFAULT',_} -> + {?DEFAULT,?ASN1CT_GEN_BER:tag_to_integer(Tag)}; + _ -> + {?OPTIONAL,?ASN1CT_GEN_BER:tag_to_integer(Tag)} + end]; +get_tag_command(#type{tag=[_|_]=Tags}, ?PARTS=Command, Prop) -> + [{anonymous_dec_command(Command, Prop), + [?ASN1CT_GEN_BER:tag_to_integer(Tag) || Tag <- Tags]}]; +get_tag_command(#type{tag=[_|_]=Tags}, ?UNDECODED=Command, Prop) -> + [{anonymous_dec_command(Command, Prop), + [?ASN1CT_GEN_BER:tag_to_integer(Tag) || Tag <- Tags]}]; +get_tag_command(#type{tag=[Tag]}, Command, Prop) -> + [{anonymous_dec_command(Command, Prop), + ?ASN1CT_GEN_BER:tag_to_integer(Tag)}]; +get_tag_command(#type{tag=[_|_]=Tags}=Type, Command, Prop) -> + lists:reverse([hd(get_tag_command(Type#type{tag=[Tag]}, Command, Prop)) || + Tag <- Tags]). + +anonymous_dec_command(?UNDECODED, 'OPTIONAL') -> + ?OPTIONAL_UNDECODED; +anonymous_dec_command(?UNDECODED, {'DEFAULT',_}) -> + ?DEFAULT_UNDECODED; +anonymous_dec_command(Command,_) -> + Command. + +%%% +%%% Selective decode. +%%% + +-define(CHOSEN, chosen). +-define(SKIP, skip). +-define(SKIP_OPTIONAL, skip_optional). + +selective_decode(Mod, {ModI,TypeLists}) -> + if + Mod =:= ModI -> + selective_decode1(Mod, TypeLists); + true -> + cfg_error({bad_module_name,ModI,Mod}) + end; +selective_decode(_, Bad) -> + cfg_error({bad_selective_decode,Bad}). + +selective_decode1(Mod, [TL|TypeLists]) -> + [selective_decode2(Mod, TL) | + selective_decode1(Mod, TypeLists)]; +selective_decode1(_, []) -> + []; +selective_decode1(_, Bad) -> + cfg_error({bad_selective_decode,Bad}). + +selective_decode2(ModName, {FuncName,TypeList}) -> + case TypeList of + [TopType|Types] -> + case asn1_db:dbget(ModName, TopType) of + #typedef{typespec=TS} -> + TagCommand = get_tag_command(TS, ?CHOSEN), + Ret = selective_decode_command(get_components(TS#type.def), + Types, concat_tags(TagCommand, [])), + {FuncName,Ret}; + undefined -> + cfg_error({undefined_type,TopType}) + end; + _ -> + cfg_error({bad_selective_decode_type_list,TypeList}) + end; +selective_decode2(_, Bad) -> + cfg_error({bad_selective_decode,Bad}). + +selective_decode_command(_, [], Acc) -> + lists:reverse(Acc); +selective_decode_command([#'ComponentType'{name=Name,typespec=TS}|_], + [Name], Acc) -> + TagCommand = get_tag_command(TS, ?CHOSEN), + lists:reverse(concat_tags(TagCommand, Acc)); +selective_decode_command([#'ComponentType'{name=Name,typespec=TS}|_], + [Name|Cs], Acc) -> + case asn1ct_gen:type(asn1ct_gen:get_inner(TS#type.def)) of + {primitive,bif} -> + cfg_error({stepping_into_primitive,[Name|Cs]}); + _ -> + TagCommand = get_tag_command(TS, ?CHOSEN), + selective_decode_command(get_components(TS#type.def), + Cs, concat_tags(TagCommand, Acc)) + end; +selective_decode_command([#'ComponentType'{typespec=TS, + prop=Prop}|Comps], + [_|_]=Cs, Acc) -> + TagCommand = case Prop of + mandatory -> + get_tag_command(TS, ?SKIP); + _ -> + get_tag_command(TS, ?SKIP_OPTIONAL) + end, + selective_decode_command(Comps, Cs, concat_tags(TagCommand, Acc)); +selective_decode_command({'CHOICE',[_|_]=Cs}, [Name|_]=TNL, Acc) -> + case lists:keyfind(Name, #'ComponentType'.name, Cs) of + #'ComponentType'{}=C -> + selective_decode_command([C], TNL, Acc); + false -> + cfg_error({undefined_name,Name}) + end; +selective_decode_command({'CHOICE',{Cs1,Cs2}}, TNL, Acc) + when is_list(Cs1), is_list(Cs2) -> + selective_decode_command({'CHOICE',Cs1 ++ Cs2}, TNL, Acc); +selective_decode_command(#'Externaltypereference'{module=M,type=C1}, + TypeNameList, Acc) -> + #type{def=Def} = get_referenced_type(M, C1), + selective_decode_command(get_components(Def), TypeNameList, Acc); +selective_decode_command(#type{def=Def}=TS, [C1|Cs], Acc0) -> + case C1 of + [N] when is_integer(N), N >= 1 -> + SkipTags = lists:duplicate(N - 1, ?SKIP), + Acc = SkipTags ++ Acc0, + TagCommand = get_tag_command(TS, ?CHOSEN), + selective_decode_command(Def, Cs, concat_tags(TagCommand, Acc)); + Bad -> + cfg_error({bad_selective_decode_element,Bad}) + end; +selective_decode_command(_, [Name], _) -> + cfg_error({undefined_name,Name}). + +get_tag_command(#type{tag=[]}, _) -> + []; +get_tag_command(#type{}, ?SKIP) -> + [?SKIP]; +get_tag_command(#type{tag=[Tag|_]}, ?SKIP_OPTIONAL) -> + #tag{class=Class,form=Form,number=TagNo} = Tag, + [{?SKIP_OPTIONAL, + ?ASN1CT_GEN_BER:encode_tag_val(?ASN1CT_GEN_BER:decode_class(Class), + Form, TagNo)}]; +get_tag_command(#type{tag=[Tag]}, Command) -> + #tag{class=Class,form=Form,number=TagNo} = Tag, + [{Command, + ?ASN1CT_GEN_BER:encode_tag_val(?ASN1CT_GEN_BER:decode_class(Class), + Form, TagNo)}]; +get_tag_command(T=#type{tag=[Tag|Tags]}, Command) -> + TC = get_tag_command(T#type{tag=[Tag]}, Command), + TCs = get_tag_command(T#type{tag=Tags}, Command), + TC ++ TCs. + +concat_tags(Ts, Acc) when is_list(Ts) -> + lists:reverse(Ts, Acc). + +%%% +%%% Common utilities. +%%% + +get_components(#'SEQUENCE'{components={Cs1,Cs2}}) when is_list(Cs1), is_list(Cs2) -> + Cs1 ++ Cs2; +get_components(#'SEQUENCE'{components=Cs}) when is_list(Cs) -> + Cs; +get_components(#'SET'{components={Cs1,Cs2}}) when is_list(Cs1), is_list(Cs2) -> + Cs1 ++ Cs2; +get_components(#'SET'{components=Cs}) when is_list(Cs) -> + Cs; +get_components({'SEQUENCE OF',#type{}=Component})-> + Component; +get_components({'SET OF',#type{}=Component}) -> + Component; +get_components(Def) -> + Def. + +get_referenced_type(M0, Name0) -> + #typedef{typespec=TS} = asn1_db:dbget(M0, Name0), + case TS of + #type{def=#'Externaltypereference'{module=M,type=Name}} -> + %% The tags have already been taken care of in the first + %% reference where they were gathered in a list of tags. + get_referenced_type(M, Name); + #type{} -> + TS + end. + +cfg_error(Error) -> + throw({structured_error,Error}). diff --git a/lib/asn1/src/asn1rtt_ber.erl b/lib/asn1/src/asn1rtt_ber.erl index f2458d25c538..13df1c73276e 100644 --- a/lib/asn1/src/asn1rtt_ber.erl +++ b/lib/asn1/src/asn1rtt_ber.erl @@ -160,22 +160,29 @@ decode_constructed_indefinite(Bin,Acc) -> %% decode_primitive_incomplete/2 decodes an encoded message incomplete %% by help of the pattern attribute (first argument). -decode_primitive_incomplete([[default,TagNo]],Bin) -> %default +decode_primitive_incomplete([{default,TagNo}], Bin) -> case decode_tag_and_length(Bin) of {Form,TagNo,V,Rest} -> - decode_incomplete2(Form,TagNo,V,[],Rest); + decode_incomplete2(Form, TagNo, V, [], Rest); _ -> asn1_NOVALUE end; -decode_primitive_incomplete([[default,TagNo,Directives]],Bin) -> +decode_primitive_incomplete([{default,TagNo,Directives}], Bin) -> %% default, constructed type, Directives points into this type case decode_tag_and_length(Bin) of {Form,TagNo,V,Rest} -> - decode_incomplete2(Form,TagNo,V,Directives,Rest); + decode_incomplete2(Form, TagNo, V, Directives, Rest); + _ -> + asn1_NOVALUE + end; +decode_primitive_incomplete([{default_undecoded,[Tag|_]}], Bin) -> + case decode_tag_and_length(Bin) of + {_,Tag,_,_} -> + decode_incomplete_bin(Bin); _ -> asn1_NOVALUE end; -decode_primitive_incomplete([[opt,TagNo]],Bin) -> +decode_primitive_incomplete([{opt,TagNo}],Bin) -> %% optional case decode_tag_and_length(Bin) of {Form,TagNo,V,Rest} -> @@ -183,74 +190,79 @@ decode_primitive_incomplete([[opt,TagNo]],Bin) -> _ -> asn1_NOVALUE end; -decode_primitive_incomplete([[opt,TagNo,Directives]],Bin) -> +decode_primitive_incomplete([{opt,TagNo,Directives}], Bin) -> %% optional case decode_tag_and_length(Bin) of {Form,TagNo,V,Rest} -> - decode_incomplete2(Form,TagNo,V,Directives,Rest); + decode_incomplete2(Form, TagNo, V, Directives, Rest); _ -> asn1_NOVALUE end; %% An optional that shall be undecoded -decode_primitive_incomplete([[opt_undec,Tag]],Bin) -> +decode_primitive_incomplete([{opt_undecoded,[Tag|_]}], Bin) -> case decode_tag_and_length(Bin) of {_,Tag,_,_} -> - decode_incomplete_bin(Bin); + decode_incomplete_bin(Bin); _ -> asn1_NOVALUE end; %% A choice alternative that shall be undecoded -decode_primitive_incomplete([[alt_undec,TagNo]|RestAlts],Bin) -> +decode_primitive_incomplete([{alt_undecoded,TagNo}|RestAlts], Bin) -> case decode_tag_and_length(Bin) of {_,TagNo,_,_} -> decode_incomplete_bin(Bin); _ -> - decode_primitive_incomplete(RestAlts,Bin) + decode_primitive_incomplete(RestAlts, Bin) end; -decode_primitive_incomplete([[alt,TagNo]|RestAlts],Bin) -> +decode_primitive_incomplete([{alt,TagNo}|RestAlts], Bin) -> case decode_tag_and_length(Bin) of {_Form,TagNo,V,Rest} -> {{TagNo,V},Rest}; _ -> decode_primitive_incomplete(RestAlts,Bin) end; -decode_primitive_incomplete([[alt,TagNo,Directives]|RestAlts],Bin) -> +decode_primitive_incomplete([{alt,TagNo,Directives}|RestAlts], Bin) -> case decode_tag_and_length(Bin) of {Form,TagNo,V,Rest} -> decode_incomplete2(Form,TagNo,V,Directives,Rest); _ -> decode_primitive_incomplete(RestAlts,Bin) end; -decode_primitive_incomplete([[alt_parts,TagNo]],Bin) -> +decode_primitive_incomplete([{alt_parts,TagNo}], Bin) -> case decode_tag_and_length(Bin) of {_Form,TagNo,V,Rest} -> {{TagNo,V},Rest}; _ -> asn1_NOVALUE end; -decode_primitive_incomplete([[alt_parts,TagNo]|RestAlts],Bin) -> +decode_primitive_incomplete([{alt_parts,TagNo}|RestAlts], Bin) -> case decode_tag_and_length(Bin) of {_Form,TagNo,V,Rest} -> {{TagNo,decode_parts_incomplete(V)},Rest}; _ -> - decode_primitive_incomplete(RestAlts,Bin) + decode_primitive_incomplete(RestAlts, Bin) end; -decode_primitive_incomplete([[undec,_TagNo]|_RestTag],Bin) -> - %% incomlete decode +decode_primitive_incomplete([{undecoded,_TagNo}|_RestTag], Bin) -> decode_incomplete_bin(Bin); -decode_primitive_incomplete([[parts,TagNo]|_RestTag],Bin) -> +decode_primitive_incomplete([{parts,[TagNo|MoreTags]}|_RestTag], Bin) -> case decode_tag_and_length(Bin) of {_Form,TagNo,V,Rest} -> - {{TagNo,decode_parts_incomplete(V)},Rest}; + case MoreTags of + [] -> + {{TagNo,decode_parts_incomplete(V)},Rest}; + [TagNo2] -> + {_,TagNo2,V2,<<>>} = decode_tag_and_length(V), + {{TagNo,{TagNo2,decode_parts_incomplete(V2)}},Rest} + end; Err -> {error,{asn1,"tag failure",TagNo,Err}} end; -decode_primitive_incomplete([mandatory|RestTag],Bin) -> +decode_primitive_incomplete([mandatory|RestTag], Bin) -> {Form,TagNo,V,Rest} = decode_tag_and_length(Bin), decode_incomplete2(Form,TagNo,V,RestTag,Rest); %% A choice that is a toptype or a mandatory component of a %% SEQUENCE or SET. -decode_primitive_incomplete([[mandatory|Directives]],Bin) -> +decode_primitive_incomplete([{mandatory,Directives}], Bin) -> {Form,TagNo,V,Rest} = decode_tag_and_length(Bin), decode_incomplete2(Form,TagNo,V,Directives,Rest); decode_primitive_incomplete([],Bin) -> @@ -269,7 +281,7 @@ decode_parts_incomplete(Bin) -> %% decode_incomplete2 checks if V is a value of a constructed or -%% primitive type, and continues the decode propeerly. +%% primitive type, and continues the decode properly. decode_incomplete2(_Form=2,TagNo,V,TagMatch,_) -> %% constructed indefinite length {Vlist,Rest2} = decode_constr_indef_incomplete(TagMatch,V,[]), @@ -288,16 +300,16 @@ decode_constructed_incomplete(_TagMatch,<<>>) -> decode_constructed_incomplete([mandatory|RestTag],Bin) -> {Tlv,Rest} = decode_primitive(Bin), [Tlv|decode_constructed_incomplete(RestTag,Rest)]; -decode_constructed_incomplete(Directives=[[Alt,_]|_],Bin) - when Alt =:= alt_undec; Alt =:= alt; Alt =:= alt_parts -> +decode_constructed_incomplete([{Alt,_}|_]=Directives, Bin) + when Alt =:= alt_undecoded; Alt =:= alt; Alt =:= alt_parts -> {_Form,TagNo,V,Rest} = decode_tag_and_length(Bin), case incomplete_choice_alt(TagNo, Directives) of - {alt_undec,_} -> + {alt_undecoded,_} -> LenA = byte_size(Bin) - byte_size(Rest), <> = Bin, A; {alt,InnerDirectives} -> - {Tlv,Rest} = decode_primitive_incomplete(InnerDirectives,V), + {Tlv,Rest} = decode_primitive_incomplete(InnerDirectives, V), {TagNo,Tlv}; {alt_parts,_} -> [{TagNo,decode_parts_incomplete(V)}]; @@ -305,7 +317,7 @@ decode_constructed_incomplete(Directives=[[Alt,_]|_],Bin) %% if a choice alternative was encoded that %% was not specified in the config file, %% thus decode component anonomous. - {Tlv,_}=decode_primitive(Bin), + {Tlv,_} = decode_primitive(Bin), Tlv end; decode_constructed_incomplete([TagNo|RestTag],Bin) -> @@ -337,12 +349,12 @@ decode_incomplete_bin(Bin) -> <> = Bin, {IncBin,Ret}. -incomplete_choice_alt(TagNo,[[Alt,TagNo]|Directives]) -> +incomplete_choice_alt(TagNo, [{Alt,TagNo}|Directives]) -> {Alt,Directives}; -incomplete_choice_alt(TagNo,[D]) when is_list(D) -> - incomplete_choice_alt(TagNo,D); -incomplete_choice_alt(TagNo,[_H|Directives]) -> - incomplete_choice_alt(TagNo,Directives); +incomplete_choice_alt(TagNo, [D]) when is_list(D) -> + incomplete_choice_alt(TagNo, D); +incomplete_choice_alt(TagNo, [_H|Directives]) -> + incomplete_choice_alt(TagNo, Directives); incomplete_choice_alt(_,[]) -> no_match. @@ -353,35 +365,35 @@ incomplete_choice_alt(_,[]) -> %% Returns {ok,Value} or {error,Reason} %% Value is a binary that in turn must be decoded to get the decoded %% value. -decode_selective([],Binary) -> +decode_selective([], Binary) -> {ok,Binary}; -decode_selective([skip|RestPattern],Binary)-> - {ok,RestBinary}=skip_tag(Binary), - {ok,RestBinary2}=skip_length_and_value(RestBinary), - decode_selective(RestPattern,RestBinary2); -decode_selective([[skip_optional,Tag]|RestPattern],Binary) -> - case skip_optional_tag(Tag,Binary) of - {ok,RestBinary} -> - {ok,RestBinary2}=skip_length_and_value(RestBinary), - decode_selective(RestPattern,RestBinary2); - missing -> - decode_selective(RestPattern,Binary) +decode_selective([skip|RestPattern], Binary)-> + {ok,RestBinary} = skip_tag(Binary), + {ok,RestBinary2} = skip_length_and_value(RestBinary), + decode_selective(RestPattern, RestBinary2); +decode_selective([{skip_optional,Tag}|RestPattern], Binary) -> + case skip_optional_tag(Tag, Binary) of + {ok,RestBinary} -> + {ok,RestBinary2} = skip_length_and_value(RestBinary), + decode_selective(RestPattern, RestBinary2); + missing -> + decode_selective(RestPattern, Binary) end; -decode_selective([[choosen,Tag]],Binary) -> - return_value(Tag,Binary); -decode_selective([[choosen,Tag]|RestPattern],Binary) -> - case skip_optional_tag(Tag,Binary) of - {ok,RestBinary} -> - {ok,Value} = get_value(RestBinary), - decode_selective(RestPattern,Value); - missing -> - {ok,<<>>} +decode_selective([{chosen,Tag}], Binary) -> + return_value(Tag, Binary); +decode_selective([{chosen,Tag}|RestPattern], Binary) -> + case skip_optional_tag(Tag, Binary) of + {ok,RestBinary} -> + {ok,Value} = get_value(RestBinary), + decode_selective(RestPattern,Value); + missing -> + {ok,<<>>} end; decode_selective(P,_) -> {error,{asn1,{partial_decode,"bad pattern",P}}}. return_value(Tag,Binary) -> - {ok,{Tag,RestBinary}}=get_tag(Binary), + {ok,{Tag,RestBinary}} = get_tag(Binary), {ok,{LenVal,_RestBinary2}} = get_length_and_value(RestBinary), {ok,<>}. diff --git a/lib/asn1/test/asn1_SUITE.erl b/lib/asn1/test/asn1_SUITE.erl index e531b346f85a..e4e3b0b7679c 100644 --- a/lib/asn1/test/asn1_SUITE.erl +++ b/lib/asn1/test/asn1_SUITE.erl @@ -965,11 +965,15 @@ specialized_decodes(Config, Rule, Opts) -> "PartialDecSeq2.asn", "PartialDecSeq3.asn", "PartialDecMyHTTP.asn", - "MEDIA-GATEWAY-CONTROL.asn", "P-Record", - "PartialDecChoExtension.asn"], + "PartialDecChoExtension.asn", + "OCSP-2013-88.asn1", + "PKIX1Explicit88.asn1"], Config, - [Rule,legacy_erlang_types,asn1config|Opts]), + [Rule,asn1config|Opts]), + asn1_test_lib:compile("MEDIA-GATEWAY-CONTROL.asn", + Config, + [Rule,legacy_erlang_types,asn1config|Opts]), test_partial_incomplete_decode:test(Config), test_selective_decode:test(). diff --git a/lib/asn1/test/asn1_SUITE_data/BasicOCSPResponse.ber b/lib/asn1/test/asn1_SUITE_data/BasicOCSPResponse.ber new file mode 100644 index 000000000000..016ce972c3c4 Binary files /dev/null and b/lib/asn1/test/asn1_SUITE_data/BasicOCSPResponse.ber differ diff --git a/lib/asn1/test/asn1_SUITE_data/MEDIA-GATEWAY-CONTROL.asn1config b/lib/asn1/test/asn1_SUITE_data/MEDIA-GATEWAY-CONTROL.asn1config index b7dba3c95c8b..cff991dbc3b0 100644 --- a/lib/asn1/test/asn1_SUITE_data/MEDIA-GATEWAY-CONTROL.asn1config +++ b/lib/asn1/test/asn1_SUITE_data/MEDIA-GATEWAY-CONTROL.asn1config @@ -1,7 +1,9 @@ +%% -*- erlang -*- {exclusive_decode, - {'MEDIA-GATEWAY-CONTROL', - [{decode_MegacoMessage_exclusive,['MegacoMessage',[{authHeader,undecoded},{mess,[{mId,undecoded},{messageBody,undecoded}]}]]}, - {decode_Message_version,['Message',[{mId,undecoded},{messageBody,undecoded}]]}]}}. + {'MEDIA-GATEWAY-CONTROL', + [{decode_MegacoMessage_exclusive, + ['MegacoMessage',[{authHeader,undecoded},{mess,[{mId,undecoded},{messageBody,undecoded}]}]]}, + {decode_Message_version,['Message',[{mId,undecoded},{messageBody,undecoded}]]}]}}. {selective_decode, - {'MEDIA-GATEWAY-CONTROL', - [{decode_MegacoMessage_selective,['MegacoMessage',mess,version]}]}}. + {'MEDIA-GATEWAY-CONTROL', + [{decode_MegacoMessage_selective,['MegacoMessage',mess,version]}]}}. diff --git a/lib/asn1/test/asn1_SUITE_data/OCSP-2013-88.asn1 b/lib/asn1/test/asn1_SUITE_data/OCSP-2013-88.asn1 new file mode 100644 index 000000000000..32b1eed96224 --- /dev/null +++ b/lib/asn1/test/asn1_SUITE_data/OCSP-2013-88.asn1 @@ -0,0 +1,149 @@ +-- OCSP definition from RFC6960, 1998 Syntax + +OCSP-2013-88 { + iso(1) identified-organization(3) dod(6) internet(1) + security(5) mechanisms(5) pkix(7) id-mod(0) + id-mod-ocsp-2013-88(81) +} + +DEFINITIONS EXPLICIT TAGS ::= + +BEGIN + +IMPORTS + + -- PKIX Certificate Extensions + AuthorityInfoAccessSyntax, CRLReason, GeneralName + FROM PKIX1Implicit88 { iso(1) identified-organization(3) + dod(6) internet(1) security(5) mechanisms(5) pkix(7) + id-mod(0) id-pkix1-implicit(19) } + + Name, CertificateSerialNumber, Extensions, + id-kp, id-ad-ocsp, Certificate, AlgorithmIdentifier + FROM PKIX1Explicit88 { iso(1) identified-organization(3) + dod(6) internet(1) security(5) mechanisms(5) pkix(7) + id-mod(0) id-pkix1-explicit(18) }; + +OCSPRequest ::= SEQUENCE { + tbsRequest TBSRequest, + optionalSignature [0] EXPLICIT Signature OPTIONAL } + +TBSRequest ::= SEQUENCE { + version [0] EXPLICIT Version DEFAULT v1, + requestorName [1] EXPLICIT GeneralName OPTIONAL, + requestList SEQUENCE OF Request, + requestExtensions [2] EXPLICIT Extensions OPTIONAL } + +Signature ::= SEQUENCE { + signatureAlgorithm AlgorithmIdentifier, + signature BIT STRING, + certs [0] EXPLICIT SEQUENCE OF Certificate OPTIONAL } + +Version ::= INTEGER { v1(0) } + +Request ::= SEQUENCE { + reqCert CertID, + singleRequestExtensions [0] EXPLICIT Extensions OPTIONAL } + +CertID ::= SEQUENCE { + hashAlgorithm AlgorithmIdentifier, + issuerNameHash OCTET STRING, -- Hash of issuer's DN + issuerKeyHash OCTET STRING, -- Hash of issuer's public key + serialNumber CertificateSerialNumber } + +OCSPResponse ::= SEQUENCE { + responseStatus OCSPResponseStatus, + responseBytes [0] EXPLICIT ResponseBytes OPTIONAL } + +OCSPResponseStatus ::= ENUMERATED { + successful (0), -- Response has valid confirmations + malformedRequest (1), -- Illegal confirmation request + internalError (2), -- Internal error in issuer + tryLater (3), -- Try again later + -- (4) is not used + sigRequired (5), -- Must sign the request + unauthorized (6) -- Request unauthorized +} + +ResponseBytes ::= SEQUENCE { + responseType OBJECT IDENTIFIER, + response OCTET STRING } + +BasicOCSPResponse ::= SEQUENCE { + tbsResponseData ResponseData, + signatureAlgorithm AlgorithmIdentifier, + signature BIT STRING, + certs [0] EXPLICIT SEQUENCE OF Certificate OPTIONAL } + +ResponseData ::= SEQUENCE { + version [0] EXPLICIT Version DEFAULT v1, + responderID ResponderID, + producedAt GeneralizedTime, + responses SEQUENCE OF SingleResponse, + responseExtensions [1] EXPLICIT Extensions OPTIONAL } + +ResponderID ::= CHOICE { + byName [1] Name, + byKey [2] KeyHash } + +KeyHash ::= OCTET STRING -- SHA-1 hash of responder's public key + -- (i.e., the SHA-1 hash of the value of the + -- BIT STRING subjectPublicKey [excluding + -- the tag, length, and number of unused + -- bits] in the responder's certificate) + +SingleResponse ::= SEQUENCE { + certID CertID, + certStatus CertStatus, + thisUpdate GeneralizedTime, + nextUpdate [0] EXPLICIT GeneralizedTime OPTIONAL, + singleExtensions [1] EXPLICIT Extensions OPTIONAL } + +CertStatus ::= CHOICE { + good [0] IMPLICIT NULL, + revoked [1] IMPLICIT RevokedInfo, + unknown [2] IMPLICIT UnknownInfo } + +RevokedInfo ::= SEQUENCE { + revocationTime GeneralizedTime, + revocationReason [0] EXPLICIT CRLReason OPTIONAL } + +UnknownInfo ::= NULL + +ArchiveCutoff ::= GeneralizedTime + +AcceptableResponses ::= SEQUENCE OF OBJECT IDENTIFIER + +ServiceLocator ::= SEQUENCE { + issuer Name, + locator AuthorityInfoAccessSyntax } + +CrlID ::= SEQUENCE { + crlUrl [0] EXPLICIT IA5String OPTIONAL, + crlNum [1] EXPLICIT INTEGER OPTIONAL, + crlTime [2] EXPLICIT GeneralizedTime OPTIONAL } + +PreferredSignatureAlgorithms ::= SEQUENCE OF PreferredSignatureAlgorithm + +PreferredSignatureAlgorithm ::= SEQUENCE { + sigIdentifier AlgorithmIdentifier, + certIdentifier AlgorithmIdentifier OPTIONAL } + +Nonce ::= OCTET STRING + +-- Object Identifiers + +-- Already defined in PKIX1Implicit88 +--id-kp-OCSPSigning OBJECT IDENTIFIER ::= { id-kp 9 } +id-pkix-ocsp OBJECT IDENTIFIER ::= { id-ad-ocsp } +id-pkix-ocsp-basic OBJECT IDENTIFIER ::= { id-pkix-ocsp 1 } +id-pkix-ocsp-nonce OBJECT IDENTIFIER ::= { id-pkix-ocsp 2 } +id-pkix-ocsp-crl OBJECT IDENTIFIER ::= { id-pkix-ocsp 3 } +id-pkix-ocsp-response OBJECT IDENTIFIER ::= { id-pkix-ocsp 4 } +id-pkix-ocsp-nocheck OBJECT IDENTIFIER ::= { id-pkix-ocsp 5 } +id-pkix-ocsp-archive-cutoff OBJECT IDENTIFIER ::= { id-pkix-ocsp 6 } +id-pkix-ocsp-service-locator OBJECT IDENTIFIER ::= { id-pkix-ocsp 7 } +id-pkix-ocsp-pref-sig-algs OBJECT IDENTIFIER ::= { id-pkix-ocsp 8 } +id-pkix-ocsp-extended-revoke OBJECT IDENTIFIER ::= { id-pkix-ocsp 9 } + +END diff --git a/lib/asn1/test/asn1_SUITE_data/OCSP-2013-88.asn1config b/lib/asn1/test/asn1_SUITE_data/OCSP-2013-88.asn1config new file mode 100644 index 000000000000..45aadf179d00 --- /dev/null +++ b/lib/asn1/test/asn1_SUITE_data/OCSP-2013-88.asn1config @@ -0,0 +1,23 @@ +%% -*- erlang -*- +{exclusive_decode, + {'OCSP-2013-88', + [ + {decode_version_undec, + ['BasicOCSPResponse',[{tbsResponseData,[{version,undecoded}]}]]}, + {decode_responderID_undec, + ['BasicOCSPResponse',[{tbsResponseData,[{responderID,undecoded}]}]]}, + {decode_producedAt_undec, + ['BasicOCSPResponse',[{tbsResponseData,[{producedAt,undecoded}]}]]}, + {decode_responses_undec, + ['BasicOCSPResponse',[{tbsResponseData,[{responses,undecoded}]}]]}, + {decode_responses_parts, + ['BasicOCSPResponse',[{tbsResponseData,[{responses,parts}]}]]}, + {decode_tbsResponseData_undec, + ['BasicOCSPResponse',[{tbsResponseData,undecoded}]]}, + {decode_BasicOCSPResponse_signature_undec, + ['BasicOCSPResponse',[{signature,undecoded}]]}, + {decode_BasicOCSPResponse_certs_undec, + ['BasicOCSPResponse',[{certs,undecoded}]]}, + {decode_BasicOCSPResponse_certs_parts, + ['BasicOCSPResponse',[{certs,parts}]]} + ]}}. diff --git a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq.asn1config b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq.asn1config index 0d91e0c3b3dd..bf388c0754ad 100644 --- a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq.asn1config +++ b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq.asn1config @@ -1,14 +1,19 @@ -{selective_decode,{'PartialDecSeq', - [{selected_decode_F1,['F',fb,b,[1],a]}, - {selected_decode_F2,['F',fb,b]}, - {selected_decode_F3,['F',fb,b,[1]]}, - {selected_decode_F4,['F',fb,d,da,[1],b,a]}, - {selected_decode_E1,['E',d,dc,dcc,dcca]}]}}. -{exclusive_decode,{'PartialDecSeq', - [{decode_F_fb_incomplete,['F',[{fb,[{b,parts},{d,undecoded}]}]]}, - {decode_D_incomplete,['D',[{a,undecoded}]]}, - {decode_F_fb_exclusive2,['F',[{fb,[{b,parts},{d,[{da,parts}]}]}]]}, {decode_F_fb_exclusive3,['F',[{fb,[{b,parts},{d,[{da,parts},{dc,[{dcc,undecoded}]}]}]}]]}]}}. -{module_name,'Seq.asn1'}. -{compile_options,[ber,debug_info]}. -{multifile_compile,['M1.asn','M2.asn']}. - +%% -*- erlang -*- +{selective_decode, + {'PartialDecSeq', + [{selected_decode_F1_1,['F',fb,b,[1],a]}, + {selected_decode_F1_2,['F',fb,b,[2],a]}, + {selected_decode_F2,['F',fb,b]}, + {selected_decode_F3,['F',fb,b,[1]]}, + {selected_decode_F4,['F',fb,d,da,[1],b,a]}, + {selected_decode_F5,['F',fb]}, + {selected_decode_E1,['E',d,dc,dcc,dcca]}, + {selected_decode_E2,['E',b]} + ]}}. +{exclusive_decode, + {'PartialDecSeq', + [{decode_E_b_incomplete,['E',[{b,parts}]]}, + {decode_F_fb_incomplete,['F',[{fb,[{b,parts},{d,undecoded}]}]]}, + {decode_D_incomplete,['D',[{a,undecoded}]]}, + {decode_F_fb_exclusive2,['F',[{fb,[{b,parts},{d,[{da,parts}]}]}]]}, + {decode_F_fb_exclusive3,['F',[{fb,[{b,parts},{d,[{da,parts},{dc,[{dcc,undecoded}]}]}]}]]}]}}. diff --git a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn index 2e77d250d268..33a88048b1e3 100644 --- a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn +++ b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn @@ -8,6 +8,13 @@ B ::= CHOICE { c S } +Bext ::= CHOICE { + a INTEGER, + b SEQUENCE {aa INTEGER, ba INTEGER}, + ..., + c S +} + S ::= SEQUENCE { a BOOLEAN, b BOOLEAN @@ -26,4 +33,14 @@ D ::= SEQUENCE { b BOOLEAN } +SeqChoice ::= SEQUENCE { + c CHOICE { + b BOOLEAN, + i INTEGER, + s VisibleString + }, + d OCTET STRING +} + + END diff --git a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn1config b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn1config index d9967d995867..776c106ab475 100644 --- a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn1config +++ b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq2.asn1config @@ -1,4 +1,15 @@ -%{partial_decode,{{module,'Seq2'},['F',fb,b,[1],a]}}. -{exclusive_decode,{'PartialDecSeq2', - [{decode_A_c_b_incomplete,['A',[{c,[{a,undecoded},{b,undecoded}]}]]}, - {decode_D_incomplete,['D',[{a,undecoded}]]}]}}. +%% -*- erlang -*- +{exclusive_decode, + {'PartialDecSeq2', + [{decode_A_a_incomplete,['A',[{a,undecoded}]]}, + {decode_A_c_b_incomplete,['A',[{c,[{a,undecoded},{b,undecoded}]}]]}, + {decode_D_incomplete,['D',[{a,undecoded}]]}, + {decode_Bext_c_incomplete,['Bext',[{c,undecoded}]]}, + {decode_Bext_c_b_incomplete,['Bext',[{c,[{b,undecoded}]}]]}, + {decode_SeqChoice_c_b_d_incomplete, + ['SeqChoice',[{c,[{b,undecoded}]}, + {d,undecoded}]]}, + {decode_SeqChoice_c_bis_incomplete, + ['SeqChoice', + [{c,[{b,undecoded},{i,undecoded},{s,undecoded}]}]]} + ]}}. diff --git a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq3.asn1config b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq3.asn1config index 44d22aa1d426..72423d682d24 100644 --- a/lib/asn1/test/asn1_SUITE_data/PartialDecSeq3.asn1config +++ b/lib/asn1/test/asn1_SUITE_data/PartialDecSeq3.asn1config @@ -1,2 +1,7 @@ -{exclusive_decode,{'PartialDecSeq3', - [{decode_S1_incomplete,['S1',[{b,[{c,parts}]},{c,[{a,parts}]},{d,parts}]]}]}}. +%% -*- erlang -*- +{exclusive_decode, + {'PartialDecSeq3', + [{decode_S1_incomplete,['S1',[{b,[{c,parts}]},{c,[{a,parts}]},{d,parts}]]}, + {decode_S1_b_incomplete,['S1',[{b,undecoded}]]}, + {decode_S3_second,['S3',[{second,undecoded}]]} + ]}}. diff --git a/lib/asn1/test/asn1_SUITE_data/TCAPPackage.asn1config b/lib/asn1/test/asn1_SUITE_data/TCAPPackage.asn1config index b0ccd7d34cc1..74c95105d1b1 100644 --- a/lib/asn1/test/asn1_SUITE_data/TCAPPackage.asn1config +++ b/lib/asn1/test/asn1_SUITE_data/TCAPPackage.asn1config @@ -1,12 +1,15 @@ -{exclusive_decode, {'TCAPPackage', - [{decode_PackageType, ['PackageType', - [{unidirectional, undecoded}, - {queryWithPerm, [{componentPortion, parts}]}, - {queryWithoutPerm, undecoded}, - {response, [{componentPortion, parts}]}, - {conversationWithPerm, undecoded}, - {conversationWithoutPerm, undecoded}, - {abort, undecoded}]]}, - {decode_TransactionPDU, ['TransactionPDU', - [{componentPortion, parts}]]}]}}. - +%% -*- erlang -*- +{exclusive_decode, + {'TCAPPackage', + [{decode_PackageType, + ['PackageType', + [{unidirectional, undecoded}, + {queryWithPerm, [{componentPortion, parts}]}, + {queryWithoutPerm, undecoded}, + {response, [{componentPortion, parts}]}, + {conversationWithPerm, undecoded}, + {conversationWithoutPerm, undecoded}, + {abort, undecoded}]]}, + {decode_TransactionPDU, + ['TransactionPDU', + [{componentPortion, parts}]]}]}}. diff --git a/lib/asn1/test/asn1_app_SUITE.erl b/lib/asn1/test/asn1_app_SUITE.erl index 12b0b509ebb4..1cbac5a62cb8 100644 --- a/lib/asn1/test/asn1_app_SUITE.erl +++ b/lib/asn1/test/asn1_app_SUITE.erl @@ -133,7 +133,8 @@ check_asn1ct_modules(Extra) -> asn1ct_gen_ber_bin_v2,asn1ct_value, asn1ct_tok,asn1ct_parser2,asn1ct_table, asn1ct_imm,asn1ct_func,asn1ct_rtt, - asn1ct_eval_ext,asn1ct_gen_jer], + asn1ct_eval_ext,asn1ct_gen_jer, + asn1ct_partial_decode], case Extra -- ASN1CTMods of [] -> ok; diff --git a/lib/asn1/test/error_SUITE.erl b/lib/asn1/test/error_SUITE.erl index aa2b36bbd7e7..eeb83deb08ec 100644 --- a/lib/asn1/test/error_SUITE.erl +++ b/lib/asn1/test/error_SUITE.erl @@ -20,7 +20,9 @@ -module(error_SUITE). -export([suite/0,all/0,groups/0, - already_defined/1,bitstrings/1, + already_defined/1, + bad_config_exclusive/1,bad_config_selective/1, + bitstrings/1, classes/1,constraints/1,constructed/1,enumerated/1, imports_exports/1,instance_of/1,integers/1,objects/1, object_field_extraction/1,oids/1,rel_oids/1, @@ -38,6 +40,8 @@ groups() -> [{p,parallel(), [already_defined, bitstrings, + bad_config_exclusive, + bad_config_selective, classes, constraints, constructed, @@ -91,6 +95,99 @@ already_defined(Config) -> } = run(P, Config), ok. +bad_config_exclusive(Config) -> + M = 'BadConfigExclusive', + P = {M, + <<"BadConfigExclusive DEFINITIONS AUTOMATIC TAGS ::= BEGIN + Seq ::= SEQUENCE { + a INTEGER, + b SEQUENCE OF INTEGER, + c BOOLEAN + } + END\n">>}, + + Conf1 = [{exclusive_decode,{'WrongModuleName',[whatever]}}], + {error,{bad_module_name,'WrongModuleName',M}} = run(P, Config, Conf1), + + Conf2 = [{exclusive_decode,wrong}], + {error,{bad_exclusive_decode,wrong}} = run(P, Config, Conf2), + + Conf3 = [{exclusive_decode,{M,[wrong]}}], + {error,{bad_exclusive_decode,[wrong]}} = run(P, Config, Conf3), + + Conf4 = [{exclusive_decode,{M,{42,[top_type,[]]}}}], + {error,{bad_exclusive_decode,_}} = run(P, Config, Conf4), + + Conf5 = [{exclusive_decode, + {M, [{exclusive_Seq, ['TopType', [{b,parts}]]}]}}], + {error,{undefined_type,'TopType'}} = run(P, Config, Conf5), + + Conf6 = [{exclusive_decode, + {M, [{exclusive_Seq, ['Seq', [{d,parts}]]}]}}], + {error,{undefined_name,d}} = run(P, Config, Conf6), + + Conf7 = [{exclusive_decode, + {M, [{exclusive_Seq, ['Seq', [{b,whatever}]]}]}}], + {error,{bad_decode_instruction,{b,whatever}}} = run(P, Config, Conf7), + + Conf8 = [{exclusive_decode, + {M, [{exclusive_Seq, ['Seq', [{a,b,c}]]}]}}], + {error,{bad_decode_instruction,{a,b,c}}} = run(P, Config, Conf8), + + ok. + +bad_config_selective(Config) -> + M = 'BadConfigSelective', + P = {M, + <<"BadConfigSelective DEFINITIONS AUTOMATIC TAGS ::= BEGIN + Seq ::= SEQUENCE { + a INTEGER, + b SEQUENCE OF INTEGER, + c BOOLEAN, + cc CHOICE { + ca INTEGER, + cb BOOLEAN + } + } + END\n">>}, + + Conf1 = [{selective_decode,{'WrongModuleName',[whatever]}}], + {error,{bad_module_name,'WrongModuleName',M}} = run(P, Config, Conf1), + + Conf2 = [{selective_decode,wrong}], + {error,{bad_selective_decode,wrong}} = run(P, Config, Conf2), + + Conf3 = [{selective_decode,{M,[wrong]}}], + {error,{bad_selective_decode,wrong}} = run(P, Config, Conf3), + + Conf4 = [{selective_decode,{M,[{f,not_list}]}}], + {error,{bad_selective_decode_type_list,not_list}} = run(P, Config, Conf4), + + Conf5 = [{selective_decode,{M,{42,[top_type,[]]}}}], + {error,{bad_selective_decode,{42,_}}} = run(P, Config, Conf5), + + Conf6 = [{selective_decode, + {M, [{only_Seq_b, ['TopType', [{b,parts}]]}]}}], + {error,{undefined_type,'TopType'}} = run(P, Config, Conf6), + + Conf7 = [{selective_decode, + {M, [{only_Seq_b, ['Seq', d]}]}}], + {error,{undefined_name,d}} = run(P, Config, Conf7), + + Conf8 = [{selective_decode, + {M, [{only_Seq_b, ['Seq', b, whatever]}]}}], + {error,{bad_selective_decode_element,whatever}} = run(P, Config, Conf8), + + Conf9 = [{selective_decode, + {M, [{only_Seq_something, ['Seq', cc, whatever]}]}}], + {error,{undefined_name,whatever}} = run(P, Config, Conf9), + + Conf10 = [{selective_decode, + {M, [{only_Seq_something, ['Seq', a, x]}]}}], + {error,{stepping_into_primitive,[a,x]}} = run(P, Config, Conf10), + + ok. + bitstrings(Config) -> M = 'Bitstrings', P = {M, @@ -928,7 +1025,6 @@ values(Config) -> } = run(P, Config), ok. - run({Mod,Spec}, Config) -> Base = atom_to_list(Mod) ++ ".asn1", File = filename:join(proplists:get_value(priv_dir, Config), Base), @@ -936,3 +1032,22 @@ run({Mod,Spec}, Config) -> Include = filename:join(filename:dirname(Include0), "asn1_SUITE_data"), ok = file:write_file(File, Spec), asn1ct:compile(File, [{i, Include}]). + +run({Mod,Spec}, Config, Asn1ConfigTerm) -> + PrivDir = proplists:get_value(priv_dir, Config), + + Asn1ConfigFile = filename:join(PrivDir, atom_to_list(Mod) ++ ".asn1config"), + Asn1ConfigData = [io_lib:format("~p. \n", [Term]) || Term <- Asn1ConfigTerm], + ok = file:write_file(Asn1ConfigFile, Asn1ConfigData), + + Base = atom_to_list(Mod) ++ ".asn1", + File = filename:join(PrivDir, Base), + ok = file:write_file(File, Spec), + + case asn1ct:compile(File, [{i, PrivDir}, asn1config]) of + {error,[{structured_error,{Asn1ConfigFile,none}, + asn1ct_partial_decode,Error}]} -> + {error,Error}; + Other -> + Other + end. diff --git a/lib/asn1/test/test_partial_incomplete_decode.erl b/lib/asn1/test/test_partial_incomplete_decode.erl index 7c5cfab10a82..f4c46f3cb087 100644 --- a/lib/asn1/test/test_partial_incomplete_decode.erl +++ b/lib/asn1/test/test_partial_incomplete_decode.erl @@ -25,49 +25,89 @@ -include_lib("common_test/include/ct.hrl"). test(Config) -> + DataDir = proplists:get_value(data_dir, Config), + test_PartialDecSeq(), + test_PartialDecSeq2(), + test_PartialDecSeq3(), + test_MyHTTPMsg(), + test_megaco(DataDir), + test_OCSP(DataDir), + ok. + +test_PartialDecSeq() -> + M = 'PartialDecSeq', + FMsg = msg('F'), - Bytes1 = roundtrip('PartialDecSeq', 'F', FMsg), - {ok,IncFMsg} = 'PartialDecSeq':decode_F_fb_incomplete(Bytes1), - decode_parts('F', IncFMsg), - {ok,IncF2Msg} = 'PartialDecSeq':decode_F_fb_exclusive2(Bytes1), - decode_parts('F2', IncF2Msg), - + test_exclusive(fun M:decode_F_fb_incomplete/1, 'F', FMsg), + DMsg = msg('D'), - Bytes2 = roundtrip('PartialDecSeq', 'D', DMsg), - {ok,IncDMsg} = 'PartialDecSeq':decode_D_incomplete(Bytes2), - decode_parts('D', IncDMsg), + test_exclusive(fun M:decode_D_incomplete/1, 'D', DMsg), F3Msg = msg('F3'), - BytesF3 = roundtrip('PartialDecSeq', 'F', F3Msg), - {ok,IncF3Msg} = 'PartialDecSeq':decode_F_fb_exclusive3(BytesF3), - decode_parts('F3', IncF3Msg), - - AMsg = msg('A'), - Bytes3 = roundtrip('PartialDecSeq2', 'A', AMsg), - {ok,IncFMsg3} = 'PartialDecSeq2':decode_A_c_b_incomplete(Bytes3), - decode_parts('A', IncFMsg3), - - MyHTTPMsg = msg('GetRequest'), - Bytes4 = roundtrip('PartialDecMyHTTP', 'GetRequest', MyHTTPMsg), - {ok,IncFMsg4} = 'PartialDecMyHTTP':decode_GetRequest_incomplete(Bytes4), - decode_parts('GetRequest', IncFMsg4), - + test_exclusive(fun M:decode_F_fb_exclusive3/1, 'F', F3Msg), + + EMsg = msg('E'), + test_exclusive(fun M:decode_E_b_incomplete/1, 'E', EMsg), + + ok. + +test_PartialDecSeq2() -> + M = 'PartialDecSeq2', + + %% Test DEFAULT value. + AMsg1 = msg('A_1'), + AMsg1Encoded = roundtrip(M, 'A', AMsg1), + {ok,AMsg1} = M:decode_A_a_incomplete(AMsg1Encoded), + + AMsg2 = msg('A_2'), + test_exclusive(fun M:decode_A_a_incomplete/1, 'A', AMsg2), + test_exclusive(fun M:decode_A_c_b_incomplete/1, 'A', AMsg2), + + SMsg = {'S',true,false}, + BextMsg = {c,SMsg}, + + test_exclusive(fun M:decode_Bext_c_incomplete/1, 'Bext', BextMsg), + test_exclusive(fun M:decode_Bext_c_b_incomplete/1, 'Bext', BextMsg), + + T = 'SeqChoice', + + SeqChoiceMsg1 = {'SeqChoice',{b,true},<<"abc">>}, + test_exclusive(fun M:decode_SeqChoice_c_b_d_incomplete/1, T, SeqChoiceMsg1), + + test_exclusive(fun M:decode_SeqChoice_c_bis_incomplete/1, T, SeqChoiceMsg1), + + SeqChoiceMsg2 = {'SeqChoice',{i,42},<<"cde">>}, + test_exclusive(fun M:decode_SeqChoice_c_bis_incomplete/1, T, SeqChoiceMsg2), + + SeqChoiceMsg3 = {'SeqChoice',{s,"xyz"},<<"fgh">>}, + test_exclusive(fun M:decode_SeqChoice_c_bis_incomplete/1, T, SeqChoiceMsg3), + + ok. + +test_PartialDecSeq3() -> + M = 'PartialDecSeq3', + MsgS1_1 = msg('S1_1'), - Bytes5 = roundtrip('PartialDecSeq3', 'S1', MsgS1_1), - {ok,IncFMsg5} = 'PartialDecSeq3':decode_S1_incomplete(Bytes5), - decode_parts('S1_1', IncFMsg5), + test_exclusive(fun M:decode_S1_incomplete/1, 'S1', MsgS1_1), + test_exclusive(fun M:decode_S1_b_incomplete/1, 'S1', MsgS1_1), MsgS1_2 = msg('S1_2'), - Bytes6 = roundtrip('PartialDecSeq3', 'S1', MsgS1_2), - {ok,IncFMsg6} = 'PartialDecSeq3':decode_S1_incomplete(Bytes6), - decode_parts('S1_2', IncFMsg6), + test_exclusive(fun M:decode_S1_incomplete/1, 'S1', MsgS1_2), + + MsgS3 = msg('S3'), + test_exclusive(fun M:decode_S3_second/1, 'S3', MsgS3), - %% test of MEDIA-GATEWAY-CONTROL - test_megaco(Config), ok. -test_megaco(Config) -> - DataDir = proplists:get_value(data_dir, Config), +test_MyHTTPMsg() -> + MyHTTPMsg = msg('GetRequest'), + Bytes1 = roundtrip('PartialDecMyHTTP', 'GetRequest', MyHTTPMsg), + {ok,IncFMsg4} = 'PartialDecMyHTTP':decode_GetRequest_incomplete(Bytes1), + decode_parts('GetRequest', IncFMsg4), + + ok. + +test_megaco(DataDir) -> Files = filelib:wildcard(filename:join([DataDir,megacomessages,"*.val"])), Mod = 'MEDIA-GATEWAY-CONTROL', lists:foreach(fun(File) -> @@ -87,34 +127,67 @@ exclusive_decode(Bin,F) -> {ok,_} = Mod:decode_part(MsgMBodyKey,MsgMBody), ok. -decode_parts('F',PartDecMsg) -> - {fb,{'E',35,{NameE_b,ListBinE_b},false,{NameE_d,BinE_d}}} = PartDecMsg, - {ok,[{'D',3,true}|_]} = 'PartialDecSeq':decode_part(NameE_b,ListBinE_b), - {ok,{'D',3,true}} = 'PartialDecSeq':decode_part(NameE_b, - hd(ListBinE_b)), - {ok,{da,[{'A',16,{'D',17,true}}]}} = - 'PartialDecSeq':decode_part(NameE_d,BinE_d), - ok; -decode_parts('F2',PartDecMsg) -> - {fb,{'E',35,{E_bkey,E_b},false,{da,{E_d_akey,E_d_a}}}} = PartDecMsg, - {ok,[{'D',3,true},{'D',4,false},{'D',5,true},{'D',6,true},{'D',7,false},{'D',8,true},{'D',9,true},{'D',10,false},{'D',11,true},{'D',12,true},{'D',13,false},{'D',14,true}]} = 'PartialDecSeq':decode_part(E_bkey,E_b), - {ok,[{'A',16,{'D',17,true}}]} = 'PartialDecSeq':decode_part(E_d_akey,E_d_a); +test_OCSP(DataDir) -> + Mod = 'OCSP-2013-88', + + ResponseData = {'ResponseData', + v1, %Version + {byKey,<<"key hash">>}, + "factory", + [], + asn1_NOVALUE}, + + Type = 'BasicOCSPResponse', + + BasicMsg = {Type, + ResponseData, + {'AlgorithmIdentifier',Mod:'id-pkix-ocsp-basic'(),asn1_NOVALUE}, + <<"signature">>, + []}, + + test_exclusive(fun Mod:decode_version_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_responderID_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_producedAt_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_responses_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_responses_parts/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_tbsResponseData_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_BasicOCSPResponse_signature_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_BasicOCSPResponse_certs_undec/1, Type, BasicMsg), + test_exclusive(fun Mod:decode_BasicOCSPResponse_certs_parts/1, Type, BasicMsg), + + %% Test undecoded/parts for an absent element. + MsgWithoutCerts = + {Type, + ResponseData, + {'AlgorithmIdentifier',Mod:'id-pkix-ocsp-basic'(),asn1_NOVALUE}, + <<"signature">>, + asn1_NOVALUE}, + {ok,Enc} = Mod:encode(Type, MsgWithoutCerts), + {ok,MsgWithoutCerts} = Mod:decode_BasicOCSPResponse_certs_undec(Enc), + {ok,MsgWithoutCerts} = Mod:decode_BasicOCSPResponse_certs_parts(Enc), + + DataFileName = filename:join(DataDir, "BasicOCSPResponse.ber"), + {ok,CannedData} = file:read_file(DataFileName), + {ok,HugeMsg} = Mod:decode('BasicOCSPResponse', CannedData), + + %% Decode version with a default value. + {ok,HugeMsg} = Mod:decode_version_undec(CannedData), + + test_exclusive(fun Mod:decode_responderID_undec/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_producedAt_undec/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_responses_undec/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_responses_parts/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_tbsResponseData_undec/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_BasicOCSPResponse_signature_undec/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_BasicOCSPResponse_certs_undec/1, Type, HugeMsg), + test_exclusive(fun Mod:decode_BasicOCSPResponse_certs_parts/1, Type, HugeMsg), + + ok. decode_parts('F3',PartDecMsg) -> {fb,{'E',10,{E_bkey,E_b},false,{dc,{'E_d_dc',13,true,{E_d_dc_dcckey,E_d_dc_dcc}}}}} = PartDecMsg, {ok,[{'D',11,true},{'D',12,false}]} = 'PartialDecSeq':decode_part(E_bkey,E_b), {ok,{'E_d_dc_dcc',14,15}} = 'PartialDecSeq':decode_part(E_d_dc_dcckey,E_d_dc_dcc); - - -decode_parts('D',PartDecMsg) -> - {'D',{NameD_a,BinD_a},true} = PartDecMsg, - {ok,123} = 'PartialDecSeq':decode_part(NameD_a,BinD_a), - ok; -decode_parts('A',PartDecMsg) -> - {'A',12,{c,{'S',true,false}},{b,{NameA_c_b,BinA_c_b}}} = PartDecMsg, - {ok,{'A_c_b',false,false}} = - 'PartialDecSeq2':decode_part(NameA_c_b,BinA_c_b), - ok; decode_parts('GetRequest',PartDecMsg) -> {'GetRequest',true,false, {'AcceptTypes',[html,'plain-text',gif,jpeg], @@ -126,34 +199,17 @@ decode_parts('GetRequest',PartDecMsg) -> {ok,"hell"} = 'PartialDecMyHTTP':decode_part(NameAcceptTypes_others, hd(ListBinAcceptTypes_others)), - ok; -decode_parts('S1_1',PartDecMsg) -> - {'S1',14,{'S2',false,12,{NameS2c,BinS2c}}, - {_,{NameS1c_a,ListBinS1c_a}},{NameS1d,BinS1d}} = PartDecMsg, - {ok,[{'S3',10,"PrintableString","OCTETSTRING", - [one,two,three,four]}|_Rest1]} = - 'PartialDecSeq3':decode_part(NameS2c,BinS2c), - {ok,[{'S3',10,"PrintableString","OCTETSTRING", - [one,two,three,four]}|_Rest2]} = - 'PartialDecSeq3':decode_part(NameS1c_a,ListBinS1c_a), - {ok,{'S3',10,"PrintableString","OCTETSTRING", - [one,two,three,four]}} = - 'PartialDecSeq3':decode_part(NameS1c_a,hd(ListBinS1c_a)), - {ok,[{'Name',"Hans","HCA","Andersen"}|_Rest3]} = - 'PartialDecSeq3':decode_part(NameS1d,BinS1d), - ok; -decode_parts('S1_2',PartDecMsg) -> - {'S1',14,{'S2',false,12,_S2c},S1c_b,{NameS1d,BinS1d}} = PartDecMsg, - {b,{'C1_b',11,true, - {'S4',{'Name',"Hans","HCA","Andersen"},"MSc"}}}=S1c_b, - {ok,[{'Name',"Hans","HCA","Andersen"}|_Rest3]} = - 'PartialDecSeq3':decode_part(NameS1d,BinS1d), ok. - - +msg('E') -> + {'E',35,msg('D_many'),false,{da,[{'A',16,{'D',17,true}}]}}; + +msg('D_many') -> + [{'D',3,true},{'D',4,false},{'D',5,true},{'D',6,true},{'D',7,false},{'D',8,true},{'D',9,true}, + {'D',10,false},{'D',11,true},{'D',12,true},{'D',13,false},{'D',14,true}]; + msg('F') -> - {fb,{'E',35,[{'D',3,true},{'D',4,false},{'D',5,true},{'D',6,true},{'D',7,false},{'D',8,true},{'D',9,true},{'D',10,false},{'D',11,true},{'D',12,true},{'D',13,false},{'D',14,true}],false,{da,[{'A',16,{'D',17,true}}]}}}; + {fb,msg('E')}; msg('F3') -> {fb,{'E',10,[{'D',11,true},{'D',12,false}],false,{dc,{'E_d_dc',13,true,{'E_d_dc_dcc',14,15}}}}}; @@ -161,8 +217,10 @@ msg('F3') -> msg('D') -> {'D',123,true}; -msg('A') -> - {'A',12,{c,{'S',true,false}},{b,{'A_c_b',false,false}}}; +msg('A_1') -> + {'A',15,{c,{'S',true,false}},{b,{'A_c_b',false,false}}}; +msg('A_2') -> + {'A',42,{c,{'S',true,false}},{b,{'A_c_b',false,false}}}; msg('GetRequest') -> {'GetRequest',true,false, @@ -181,7 +239,7 @@ msg('C1_a') -> msg('C1_b') -> {b,{'C1_b',11,true,msg('S4')}}; msg('S3') -> - {'S3',10,"PrintableString","OCTETSTRING",[one,two,three,four]}; + {'S3',10,"PrintableString",<<"OCTETSTRING">>,[one,two,three,four]}; msg('S4') -> {'S4',msg('Name'),"MSc"}; msg('SO1') -> @@ -191,3 +249,52 @@ msg('Name') -> roundtrip(M, T, V) -> asn1_test_lib:roundtrip_enc(M, T, V). + +test_exclusive(DecodeFun, Type, Msg) -> + {module,Mod} = erlang:fun_info(DecodeFun, module), + Encoded = roundtrip(Mod, Type, Msg), + case DecodeFun(Encoded) of + {ok,Msg} -> + error({should_be_different,Msg}); + {ok,Decoded} -> + case dec_parts(Decoded, Msg, Mod) of + Msg -> + ok; + OtherMsg -> + io:format(""" + Partial decoding: + ~p + + Expected: + ~p + + Got: + ~p + """, [Decoded,Msg,OtherMsg]), + error(full_and_partial_decode_differ) + end + end. + +dec_parts(Same, Same, _Mod) -> + Same; +dec_parts({Name,Parts}, Expected, Mod) when is_atom(Name), is_list(Parts), is_list(Expected) -> + [begin + {ok,Dec} = Mod:decode_part(Name, Bin), + Dec + end || Bin <- Parts]; +dec_parts({Name,Undec}, _Expected, Mod) when is_atom(Name), is_binary(Undec) -> + {ok,Dec} = Mod:decode_part(Name, Undec), + Dec; +dec_parts({Name,{Tag,_}=Undec}, _Expected, Mod) when is_atom(Name), is_integer(Tag) -> + {ok,Dec} = Mod:decode_part(Name, Undec), + Dec; +dec_parts(Tuple0, Expected, Mod) when is_tuple(Tuple0), is_tuple(Expected) -> + Tuple = dec_parts_list(tuple_to_list(Tuple0), tuple_to_list(Expected), Mod), + list_to_tuple(Tuple); +dec_parts(List, Expected, Mod) when is_list(List), is_list(Expected) -> + dec_parts_list(List, Expected, Mod). + +dec_parts_list([H1|T1], [H2|T2], Mod) -> + [dec_parts(H1, H2, Mod) | dec_parts_list(T1, T2, Mod)]; +dec_parts_list([], [], _Mod) -> + []. diff --git a/lib/asn1/test/test_selective_decode.erl b/lib/asn1/test/test_selective_decode.erl index 3220ccc2ed09..5953292f0f86 100644 --- a/lib/asn1/test/test_selective_decode.erl +++ b/lib/asn1/test/test_selective_decode.erl @@ -26,17 +26,22 @@ test() -> FMsg = msg('F'), Bytes = roundtrip('PartialDecSeq', 'F', FMsg), - {ok,3} = 'PartialDecSeq':selected_decode_F1(Bytes), + {ok,3} = 'PartialDecSeq':selected_decode_F1_1(Bytes), + {ok,4} = 'PartialDecSeq':selected_decode_F1_2(Bytes), {ok,[{'D',3,true},{'D',4,false},{'D',5,true},{'D',6,true}, {'D',7,false},{'D',8,true},{'D',9,true},{'D',10,false}, {'D',11,true},{'D',12,true},{'D',13,false},{'D',14,true}]} = 'PartialDecSeq':selected_decode_F2(Bytes), {ok,{'D',3,true}} = 'PartialDecSeq':selected_decode_F3(Bytes), {ok,17} = 'PartialDecSeq':selected_decode_F4(Bytes), - + E_from_F = element(2, FMsg), + {ok,E_from_F} = 'PartialDecSeq':selected_decode_F5(Bytes), + EMsg = msg('E'), Bytes2 = roundtrip('PartialDecSeq', 'E', EMsg), {ok,14} = 'PartialDecSeq':selected_decode_E1(Bytes2), + Emsg_b = msg('E_b'), + {ok,Emsg_b} = 'PartialDecSeq':selected_decode_E2(Bytes2), MGCMsg = msg('M-G-C'), Bytes3 = roundtrip('MEDIA-GATEWAY-CONTROL', 'MegacoMessage', MGCMsg), @@ -61,7 +66,10 @@ msg('F') -> {fb,{'E',35,[{'D',3,true},{'D',4,false},{'D',5,true},{'D',6,true},{'D',7,false},{'D',8,true},{'D',9,true},{'D',10,false},{'D',11,true},{'D',12,true},{'D',13,false},{'D',14,true}],false,{da,[{'A',16,{'D',17,true}}]}}}; msg('E') -> - {'E',10,[{'D',11,true},{'D',12,false}],false,{dc,{'E_d_dc',13,true,{'E_d_dc_dcc',14,15}}}}; + {'E',10,msg('E_b'),false,{dc,{'E_d_dc',13,true,{'E_d_dc_dcc',14,15}}}}; + +msg('E_b') -> + [{'D',11,true},{'D',12,false}]; msg('M-G-C') -> {'MegacoMessage',asn1_NOVALUE,{'Message',1,{ip4Address,{'IP4Address',[125,125,125,111],55555}},{transactions,[{transactionReply,{'TransactionReply',50007,asn1_NOVALUE,{actionReplies,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,[{auditValueReply,{auditResult,{'AuditResult',{'TerminationID',[],[255,255,255]},[{mediaDescriptor,{'MediaDescriptor',asn1_NOVALUE,{multiStream,[{'StreamDescriptor',1,{'StreamParms',{'LocalControlDescriptor',sendRecv,asn1_NOVALUE,asn1_NOVALUE,[{'PropertyParm',[0,11,0,7],[[52,48]],asn1_NOVALUE}]},{'LocalRemoteDescriptor',[[{'PropertyParm',[0,0,176,1],[[48]],asn1_NOVALUE},{'PropertyParm',[0,0,176,8],[[73,78,32,73,80,52,32,49,50,53,46,49,50,53,46,49,50,53,46,49,49,49]],asn1_NOVALUE},{'PropertyParm',[0,0,176,15],[[97,117,100,105,111,32,49,49,49,49,32,82,84,80,47,65,86,80,32,32,52]],asn1_NOVALUE},{'PropertyParm',[0,0,176,12],[[112,116,105,109,101,58,51,48]],asn1_NOVALUE}]]},{'LocalRemoteDescriptor',[[{'PropertyParm',[0,0,176,1],[[48]],asn1_NOVALUE},{'PropertyParm',[0,0,176,8],[[73,78,32,73,80,52,32,49,50,52,46,49,50,52,46,49,50,52,46,50,50,50]],asn1_NOVALUE},{'PropertyParm',[0,0,176,15],[[97,117,100,105,111,32,50,50,50,50,32,82,84,80,47,65,86,80,32,32,52]],asn1_NOVALUE},{'PropertyParm',[0,0,176,12],[[112,116,105,109,101,58,51,48]],asn1_NOVALUE}]]}}}]}}},{packagesDescriptor,[{'PackagesItem',[0,11],1},{'PackagesItem',[0,11],1}]},{statisticsDescriptor,[{'StatisticsParameter',[0,12,0,4],[[49,50,48,48]]},{'StatisticsParameter',[0,11,0,2],[[54,50,51,48,48]]},{'StatisticsParameter',[0,12,0,5],[[55,48,48]]},{'StatisticsParameter',[0,11,0,3],[[52,53,49,48,48]]},{'StatisticsParameter',[0,12,0,6],[[48,46,50]]},{'StatisticsParameter',[0,12,0,7],[[50,48]]},{'StatisticsParameter',[0,12,0,8],[[52,48]]}]}]}}}]}]}}}]}}}; diff --git a/lib/asn1/test/test_special_decode_performance.erl b/lib/asn1/test/test_special_decode_performance.erl index 35c396575b50..6814bbfaf3c4 100644 --- a/lib/asn1/test/test_special_decode_performance.erl +++ b/lib/asn1/test/test_special_decode_performance.erl @@ -120,7 +120,7 @@ loop2(Mod,FS,Bin,N) -> get_selective_funcs('PartialDecSeq') -> % [selected_decode_F1,selected_decode_F2,selected_decode_F3,selected_decode_F4]; - [selected_decode_F1,selected_decode_F3,selected_decode_F4]; + [selected_decode_F1_1,selected_decode_F3,selected_decode_F4]; get_selective_funcs('MEDIA-GATEWAY-CONTROL') -> [decode_MegacoMessage_selective]. diff --git a/lib/asn1/vsn.mk b/lib/asn1/vsn.mk index 6dc4dc0c87d2..c27b02997eeb 100644 --- a/lib/asn1/vsn.mk +++ b/lib/asn1/vsn.mk @@ -1 +1 @@ -ASN1_VSN = 5.2 +ASN1_VSN = 5.2.1 diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml index 5f4da5f44b36..5b6b1737d516 100644 --- a/lib/common_test/doc/src/basics_chapter.xml +++ b/lib/common_test/doc/src/basics_chapter.xml @@ -4,7 +4,7 @@
- 20032022 + 20032023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index a74777c56e25..7267c32249ef 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -4,7 +4,7 @@
- 20032020 + 20032023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/config_file_chapter.xml b/lib/common_test/doc/src/config_file_chapter.xml index 2ab9259dfbff..ebd02687b78d 100644 --- a/lib/common_test/doc/src/config_file_chapter.xml +++ b/lib/common_test/doc/src/config_file_chapter.xml @@ -4,7 +4,7 @@
- 20042021 + 20042023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/ct_cover.xml b/lib/common_test/doc/src/ct_cover.xml index 600e3e51d623..69c80e80d185 100644 --- a/lib/common_test/doc/src/ct_cover.xml +++ b/lib/common_test/doc/src/ct_cover.xml @@ -4,7 +4,7 @@
- 20102020 + 20102023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml index 7458533984e6..07901b63b143 100644 --- a/lib/common_test/doc/src/dependencies_chapter.xml +++ b/lib/common_test/doc/src/dependencies_chapter.xml @@ -4,7 +4,7 @@
- 20062021 + 20062023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/example_chapter.xml b/lib/common_test/doc/src/example_chapter.xml index e355e6ae7ce7..84f9fb22793c 100644 --- a/lib/common_test/doc/src/example_chapter.xml +++ b/lib/common_test/doc/src/example_chapter.xml @@ -4,7 +4,7 @@
- 20032016 + 20032023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/getting_started_chapter.xml b/lib/common_test/doc/src/getting_started_chapter.xml index 0a12a9653695..3b5143e7f7ae 100644 --- a/lib/common_test/doc/src/getting_started_chapter.xml +++ b/lib/common_test/doc/src/getting_started_chapter.xml @@ -4,7 +4,7 @@
- 20072021 + 20072023 Ericsson AB. All Rights Reserved. diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index e304a309c9ca..bee62838e935 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -33,6 +33,51 @@ notes.xml
+
Common_Test 1.26 + +
Fixed Bugs and Malfunctions + + +

+ With this change, common_test returns an error when suite + with a badly defined group is executed.

+

+ *** POTENTIAL INCOMPATIBILITY ***

+

+ Own Id: OTP-18728 Aux Id: PR-7487, PR-7674

+ + +

+ With this change, stylesheet option is applied to all + HTML report pages.

+

+ Own Id: OTP-18760

+ + +

+ Update all <tt> html tags to be <code> + instead.

+

+ Own Id: OTP-18799 Aux Id: PR-7695

+ + +
+ + +
Improvements and New Features + + +

+ This change fixes docs, so that historically deprecated + ?config macro is no longer recommended to be used.

+

+ Own Id: OTP-18858 Aux Id: PR-7825

+ + +
+ +
+
Common_Test 1.25.1
Fixed Bugs and Malfunctions @@ -3991,7 +4036,7 @@

- If a timetrap timeout occurred during execution of of a + If a timetrap timeout occurred during execution of a function in a lib module (i.e. a function called directly or indirectly from a test case), the Suite argument in the end_tc/3 framework callback function would not @@ -5082,7 +5127,7 @@

The rx library, included with common_test, failed to - build on on some architectures because the -fPIC compiler + build on some architectures because the -fPIC compiler option was missing.

Own Id: OTP-7111

diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index b84c54041bc8..6063127bf0a0 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -817,7 +817,7 @@

In the data directory, data_dir, the test module has its own files needed for the testing. The name of data_dir - is the the name of the test suite followed by "_data". + is the name of the test suite followed by "_data". For example, "some_path/foo_SUITE.beam" has the data directory "some_path/foo_SUITE_data/". Use this directory for portability, that is, to avoid hardcoding directory names in your suite. As the data diff --git a/lib/common_test/src/ct_logs.erl b/lib/common_test/src/ct_logs.erl index 07108cf7c90d..063259a2b43a 100644 --- a/lib/common_test/src/ct_logs.erl +++ b/lib/common_test/src/ct_logs.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2003-2021. All Rights Reserved. +%% Copyright Ericsson AB 2003-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/common_test/src/ct_master.erl b/lib/common_test/src/ct_master.erl index 52e82078eaee..fd0f7ca18c8c 100644 --- a/lib/common_test/src/ct_master.erl +++ b/lib/common_test/src/ct_master.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2006-2018. All Rights Reserved. +%% Copyright Ericsson AB 2006-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/common_test/src/erl2html2.erl b/lib/common_test/src/erl2html2.erl index cb000bafd62a..05cb807ec4f1 100644 --- a/lib/common_test/src/erl2html2.erl +++ b/lib/common_test/src/erl2html2.erl @@ -274,7 +274,13 @@ possibly_enhance(Str,false) -> %%%----------------------------------------------------------------- %%% End of the file footer() -> - "". + %% If the URL has an anchor part at the end (# with line number), + %% color that line to make it easier to find on the screen. + "\n". %%%----------------------------------------------------------------- %%% Read encoding from source file diff --git a/lib/common_test/src/test_server.erl b/lib/common_test/src/test_server.erl index 7e3b0d995c42..3301840dc3a6 100644 --- a/lib/common_test/src/test_server.erl +++ b/lib/common_test/src/test_server.erl @@ -466,7 +466,7 @@ run_test_case_msgloop(#st{ref=Ref,pid=Pid,end_conf_pid=EndConfPid0}=St0) -> From ! {self(),Tag,ok}, run_test_case_msgloop(St); {abort_current_testcase,_,_}=Abort when St0#st.status =:= starting -> - %% we're in init phase, must must postpone this operation + %% we're in init phase, must postpone this operation %% until test case execution is in progress (or FW:init_tc %% gets killed) self() ! Abort, diff --git a/lib/common_test/src/test_server_ctrl.erl b/lib/common_test/src/test_server_ctrl.erl index 35fb69698adf..e5a361bd332d 100644 --- a/lib/common_test/src/test_server_ctrl.erl +++ b/lib/common_test/src/test_server_ctrl.erl @@ -4632,7 +4632,7 @@ update_config(Config, []) -> %% simple list of test cases to call, when executing the test suite. %% %% CurMod is the "current" module, that is, the module the last instruction -%% was read from. May be be set to 'none' initially. +%% was read from. May be set to 'none' initially. %% %% SkipList is the list of test cases to skip and requirements to deny. %% diff --git a/lib/common_test/test/ct_misc_1_SUITE.erl b/lib/common_test/test/ct_misc_1_SUITE.erl index f1d783fc8da5..8a305b820572 100644 --- a/lib/common_test/test/ct_misc_1_SUITE.erl +++ b/lib/common_test/test/ct_misc_1_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2016. All Rights Reserved. +%% Copyright Ericsson AB 2010-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/common_test/test/ct_misc_1_SUITE_data/bad_groups_SUITE.erl b/lib/common_test/test/ct_misc_1_SUITE_data/bad_groups_SUITE.erl index 8c5180ddf7a1..06d5daaea5c6 100644 --- a/lib/common_test/test/ct_misc_1_SUITE_data/bad_groups_SUITE.erl +++ b/lib/common_test/test/ct_misc_1_SUITE_data/bad_groups_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2009-2016. All Rights Reserved. +%% Copyright Ericsson AB 2009-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/common_test/test/erl2html2_SUITE_data/m1.erl b/lib/common_test/test/erl2html2_SUITE_data/m1.erl index 1d405963a5c3..ed01f62fdcfa 100644 --- a/lib/common_test/test/erl2html2_SUITE_data/m1.erl +++ b/lib/common_test/test/erl2html2_SUITE_data/m1.erl @@ -43,7 +43,7 @@ ok. % indentation error, OTP-9710 %% Function inside macro definition ?MACRO_DEFINING_A_FUNCTION. -%% Two function one one line +%% Two function on one line quuux() -> ok. quuuux() -> ok. %% do_something/0 does something diff --git a/lib/common_test/vsn.mk b/lib/common_test/vsn.mk index 16f1df9645a2..fc57703c9e7a 100644 --- a/lib/common_test/vsn.mk +++ b/lib/common_test/vsn.mk @@ -1 +1 @@ -COMMON_TEST_VSN = 1.25.1 +COMMON_TEST_VSN = 1.26 diff --git a/lib/compiler/src/beam_block.erl b/lib/compiler/src/beam_block.erl index 9cd6d93783a4..9214c15ec0f9 100644 --- a/lib/compiler/src/beam_block.erl +++ b/lib/compiler/src/beam_block.erl @@ -131,7 +131,8 @@ blockify([I|Is0]=IsAll, Acc) -> error -> blockify(Is0, [I|Acc]); Instr when is_tuple(Instr) -> {Block0,Is} = collect_block(IsAll), - Block = sort_moves(Block0), + Block1 = sort_moves(Block0), + Block = swap_opt_block(Block1, []), blockify(Is, [{block,Block}|Acc]) end; blockify([], Acc) -> reverse(Acc). @@ -170,6 +171,10 @@ collect({put_map,{f,0},Op,S,D,R,{list,Puts}}) -> collect({fmove,S,D}) -> {set,[D],[S],fmove}; collect({fconv,S,D}) -> {set,[D],[S],fconv}; collect({executable_line,Line}) -> {set,[],[],{executable_line,Line}}; +collect({swap,D1,D2}) -> + Regs = [D1,D2], + {set,Regs,Regs,swap}; +collect({make_fun3,F,I,U,D,{list,Ss}}) -> {set,[D],Ss,{make_fun3,F,I,U}}; collect(_) -> error. %% embed_lines([Instruction]) -> [Instruction] @@ -223,6 +228,51 @@ sort_on_yreg([{set,[Dst],[Src],move}|_]=Moves) -> keysort(3, Moves) end. +%% Attempt to replace a swap instruction with a move instruction. +swap_opt_block([{set,[D1,D2],_,swap}=I|Is], [{set,[Dst],Ss,Op}|Acc]=Acc0) -> + case Op of + {get_tuple_element,_} -> + %% Don't separate from other get_tuple_element instructions or + %% tuple testing instructions. + swap_opt_block(Is, [I|Acc0]); + {alloc,_,_} -> + %% Potentially unsafe. + swap_opt_block(Is, [I|Acc0]); + get_hd -> + %% Don't separate from get_tl. + swap_opt_block(Is, [I|Acc0]); + get_tl -> + %% Don't separate from get_hd. + swap_opt_block(Is, [I|Acc0]); + _ -> + case is_used(Dst, Ss) of + true -> + swap_opt_block(Is, [I|Acc0]); + false -> + OtherDst = case Dst of + D1 -> D2; + D2 -> D1; + _ -> none + end, + case OtherDst of + none -> + swap_opt_block(Is, [I|Acc0]); + _ -> + swap_opt_block(Is, [{set,[OtherDst],Ss,Op}, + {set,[Dst],[OtherDst],move}|Acc]) + end + end + end; +swap_opt_block([I|Is], Acc) -> + swap_opt_block(Is, [I|Acc]); +swap_opt_block([], Acc) -> + reverse(Acc). + +is_used(D, [D|_]) -> true; +is_used(D, [{tr,D,_}|_]) -> true; +is_used(D, [_|As]) -> is_used(D, As); +is_used(_, []) -> false. + %%% %%% Coalesce adjacent get_map_elements and has_map_fields instructions. %%% diff --git a/lib/compiler/src/beam_clean.erl b/lib/compiler/src/beam_clean.erl index 54cb9ec30947..3cc6001881ca 100644 --- a/lib/compiler/src/beam_clean.erl +++ b/lib/compiler/src/beam_clean.erl @@ -23,19 +23,22 @@ -export([module/2]). +-import(lists, [reverse/1]). + -spec module(beam_utils:module_code(), [compile:option()]) -> {'ok',beam_utils:module_code()}. module({Mod,Exp,Attr,Fs0,_}, Opts) -> - Order = [Lbl || {function,_,_,Lbl,_} <- Fs0], - All = #{Lbl => Func || {function,_,_,Lbl,_}=Func <- Fs0}, - WorkList = rootset(Fs0, Exp, Attr), + Fs1 = move_out_funs(Fs0), + Order = [Lbl || {function,_,_,Lbl,_} <- Fs1], + All = #{Lbl => Func || {function,_,_,Lbl,_}=Func <- Fs1}, + WorkList = rootset(Fs1, Exp, Attr), Used = find_all_used(WorkList, All, sets:from_list(WorkList, [{version, 2}])), - Fs1 = remove_unused(Order, Used, All), - {Fs2,Lc} = clean_labels(Fs1), - Fs3 = fix_bs_create_bin(Fs2, Opts), - Fs4 = fix_badrecord(Fs3, Opts), - Fs = maybe_remove_lines(Fs4, Opts), + Fs2 = remove_unused(Order, Used, All), + {Fs3,Lc} = clean_labels(Fs2), + Fs4 = fix_bs_create_bin(Fs3, Opts), + Fs5 = fix_badrecord(Fs4, Opts), + Fs = maybe_remove_lines(Fs5, Opts), {ok,{Mod,Exp,Attr,Fs,Lc}}. %% Determine the rootset, i.e. exported functions and @@ -78,6 +81,31 @@ add_to_work_list(F, {Fs,Used}=Sets) -> false -> {[F|Fs],sets:add_element(F, Used)} end. +%% Move out make_fun3 instructions from blocks. That is necessary because +%% they contain labels that must be seen and renumbered. + +move_out_funs([{function,Name,Arity,Entry,Is0}|Fs]) -> + Is = move_out_funs_is(Is0), + [{function,Name,Arity,Entry,Is}|move_out_funs(Fs)]; +move_out_funs([]) -> []. + +move_out_funs_is([{block,Bl}|Is]) -> + move_out_funs_block(Bl, Is, []); +move_out_funs_is([I|Is]) -> + [I|move_out_funs_is(Is)]; +move_out_funs_is([]) -> []. + +move_out_funs_block([{set,[D],Ss,{make_fun3,F,I,U}}|Bl], Is, Acc) -> + make_block(Acc) ++ + [{make_fun3,F,I,U,D,{list,Ss}} | + move_out_funs_block(Bl, Is, [])]; +move_out_funs_block([B|Bl], Is, Acc) -> + move_out_funs_block(Bl, Is, [B|Acc]); +move_out_funs_block([], Is, Acc) -> + make_block(Acc) ++ move_out_funs_is(Is). + +make_block([_|_]=Is) -> [{block,reverse(Is)}]; +make_block([]) -> []. %%% %%% Coalesce adjacent labels. Renumber all labels to eliminate gaps. diff --git a/lib/compiler/src/beam_flatten.erl b/lib/compiler/src/beam_flatten.erl index 9a29b1ad503e..de11f6300970 100644 --- a/lib/compiler/src/beam_flatten.erl +++ b/lib/compiler/src/beam_flatten.erl @@ -63,7 +63,8 @@ norm({set,[D],[S|Puts],{alloc,R,{put_map,Op,F}}}) -> {put_map,F,Op,S,D,R,{list,Puts}}; norm({set,[],[],remove_message}) -> remove_message; norm({set,[],[],{line,_}=Line}) -> Line; -norm({set,[],[],{executable_line,_}=Line}) -> Line. +norm({set,[],[],{executable_line,_}=Line}) -> Line; +norm({set,[D1,D2],[D1,D2],swap}) -> {swap,D1,D2}. norm_allocate({_Zero,nostack,Nh,[]}, Regs) -> [{test_heap,Nh,Regs}]; diff --git a/lib/compiler/src/beam_trim.erl b/lib/compiler/src/beam_trim.erl index b5ff21bbaca1..bfc6992aac56 100644 --- a/lib/compiler/src/beam_trim.erl +++ b/lib/compiler/src/beam_trim.erl @@ -292,9 +292,6 @@ remap([{recv_marker_clear,Ref}|Is], Remap) -> remap([{recv_marker_reserve,Mark}|Is], Remap) -> I = {recv_marker_reserve,remap_arg(Mark, Remap)}, [I|remap(Is, Remap)]; -remap([{swap,Reg1,Reg2}|Is], Remap) -> - I = {swap,remap_arg(Reg1, Remap),remap_arg(Reg2, Remap)}, - [I|remap(Is, Remap)]; remap([{test,Name,Fail,Ss}|Is], Remap) -> I = {test,Name,Fail,remap_args(Ss, Remap)}, [I|remap(Is, Remap)]; @@ -549,12 +546,6 @@ do_usage([{recv_marker_reserve,Src}|Is], Safe, Regs0, Ns, Acc) -> Regs = ordsets:union(Regs0, yregs([Src])), U = {Regs,Ns}, do_usage(Is, Safe, Regs, Ns, [U|Acc]); -do_usage([{swap,R1,R2}|Is], Safe, Regs0, Ns0, Acc) -> - Ds = yregs([R1,R2]), - Regs = ordsets:union(Regs0, Ds), - Ns = ordsets:union(Ns0, Ds), - U = {Regs,Ns}, - do_usage(Is, Safe, Regs, Ns, [U|Acc]); do_usage([{test,_,Fail,Ss}|Is], Safe, Regs0, Ns, Acc) -> case is_safe_branch(Fail, Safe) of true -> diff --git a/lib/compiler/src/beam_types.erl b/lib/compiler/src/beam_types.erl index 3fc2476898e1..b81bbc06164d 100644 --- a/lib/compiler/src/beam_types.erl +++ b/lib/compiler/src/beam_types.erl @@ -45,7 +45,6 @@ make_boolean/0, make_cons/2, make_float/1, - make_float/2, make_integer/1, make_integer/2]). @@ -169,10 +168,16 @@ mts_records([{Key, A} | RsA], [{Key, B} | RsB], Acc) -> none -> mts_records(RsA, RsB, Acc); T -> mts_records(RsA, RsB, [{Key, T} | Acc]) end; -mts_records([{KeyA, _} | _ ]=RsA, [{KeyB, _} | RsB], Acc) when KeyA > KeyB -> - mts_records(RsA, RsB, Acc); -mts_records([{KeyA, _} | RsA], [{KeyB, _} | _] = RsB, Acc) when KeyA < KeyB -> - mts_records(RsA, RsB, Acc); +mts_records([{KeyA, _} | _]=RsA, [{KeyB, _} | _]=RsB, Acc) -> + %% We must use total ordering rather than plain '<' as -0.0 differs from + %% +0.0 + case total_compare(KeyA, KeyB, fun erlang:'<'/2) of + true -> + mts_records(tl(RsA), RsB, Acc); + false -> + true = KeyA =/= KeyB, %Assertion. + mts_records(RsA, tl(RsB), Acc) + end; mts_records(_RsA, [], [_|_]=Acc) -> reverse(Acc); mts_records([], _RsB, [_|_]=Acc) -> @@ -320,10 +325,16 @@ jts_records(RsA, RsB, N, Acc) when N > ?TUPLE_SET_LIMIT -> #t_tuple{} = normalize_tuple_set(Acc, B); jts_records([{Key, A} | RsA], [{Key, B} | RsB], N, Acc) -> jts_records(RsA, RsB, N + 1, [{Key, lub(A, B)} | Acc]); -jts_records([{KeyA, _} | _]=RsA, [{KeyB, B} | RsB], N, Acc) when KeyA > KeyB -> - jts_records(RsA, RsB, N + 1, [{KeyB, B} | Acc]); -jts_records([{KeyA, A} | RsA], [{KeyB, _} | _] = RsB, N, Acc) when KeyA < KeyB -> - jts_records(RsA, RsB, N + 1, [{KeyA, A} | Acc]); +jts_records([{KeyA, A} | _]=RsA, [{KeyB, B} | _]=RsB, N, Acc) -> + %% We must use total ordering rather than plain '<' as -0.0 differs from + %% +0.0 + case total_compare(KeyA, KeyB, fun erlang:'<'/2) of + true -> + jts_records(tl(RsA), RsB, N + 1, [{KeyA, A} | Acc]); + false -> + true = KeyA =/= KeyB, %Assertion. + jts_records(RsA, tl(RsB), N + 1, [{KeyB, B} | Acc]) + end; jts_records([{KeyA, A} | RsA], [], N, Acc) -> jts_records(RsA, [], N + 1, [{KeyA, A} | Acc]); jts_records([], [{KeyB, B} | RsB], N, Acc) -> @@ -479,8 +490,7 @@ is_bs_matchable_type(Type) -> Result :: {ok, term()} | error. get_singleton_value(#t_atom{elements=[Atom]}) -> {ok, Atom}; -get_singleton_value(#t_float{elements={Float,Float}}) when Float /= 0 -> - %% 0.0 is not actually a singleton as it has two encodings: 0.0 and -0.0 +get_singleton_value(#t_float{elements={Float,Float}}) -> {ok, Float}; get_singleton_value(#t_integer{elements={Int,Int}}) -> {ok, Int}; @@ -697,11 +707,7 @@ make_cons(Head0, Tail) -> -spec make_float(float()) -> type(). make_float(Float) when is_float(Float) -> - make_float(Float, Float). - --spec make_float(float(), float()) -> type(). -make_float(Min, Max) when is_float(Min), is_float(Max), Min =< Max -> - #t_float{elements={Min, Max}}. + #t_float{elements={Float,Float}}. -spec make_integer(integer()) -> type(). make_integer(Int) when is_integer(Int) -> @@ -882,7 +888,7 @@ glb(#t_integer{elements=R1}, #t_integer{elements=R2}) -> glb(#t_integer{elements=R1}, #t_number{elements=R2}) -> integer_from_range(glb_ranges(R1, R2)); glb(#t_float{elements=R1}, #t_number{elements=R2}) -> - float_from_range(glb_ranges(R1, R2)); + float_from_range(glb_ranges(R1, number_to_float_range(R2))); glb(#t_list{type=TypeA,terminator=TermA}, #t_list{type=TypeB,terminator=TermB}) -> %% A list is a union of `[type() | _]` and `[]`, so we're left with @@ -903,7 +909,7 @@ glb(#t_number{elements=R1}, #t_number{elements=R2}) -> glb(#t_number{elements=R1}, #t_integer{elements=R2}) -> integer_from_range(glb_ranges(R1, R2)); glb(#t_number{elements=R1}, #t_float{elements=R2}) -> - float_from_range(glb_ranges(R1, R2)); + float_from_range(glb_ranges(number_to_float_range(R1), R2)); glb(#t_map{super_key=SKeyA,super_value=SValueA}, #t_map{super_key=SKeyB,super_value=SValueB}) -> %% Note the use of meet/2; elements don't need to be normal types. @@ -1132,6 +1138,14 @@ lub_ranges({MinA,MaxA}, {MinB,MaxB}) -> lub_ranges(_, _) -> any. +%% Expands integer 0 to `-0.0 .. +0.0` +number_to_float_range({Min, 0}) -> + number_to_float_range({Min, +0.0}); +number_to_float_range({0, Max}) -> + number_to_float_range({-0.0, Max}); +number_to_float_range(Other) -> + Other. + lub_bs_matchable(UnitA, UnitB) -> #t_bs_matchable{tail_unit=gcd(UnitA, UnitB)}. @@ -1179,12 +1193,13 @@ float_from_range(none) -> none; float_from_range(any) -> #t_float{}; -float_from_range({Min0,Max0}) -> - case {safe_float(Min0),safe_float(Max0)} of +float_from_range({Min0, Max0}) -> + true = inf_le(Min0, Max0), %Assertion. + case {safe_float(Min0), safe_float(Max0)} of {'-inf','+inf'} -> #t_float{}; - {Min,Max} -> - #t_float{elements={Min,Max}} + {Min, Max} -> + #t_float{elements={Min, Max}} end. safe_float(N) when is_number(N) -> @@ -1218,21 +1233,48 @@ number_from_range(N) -> none end. -inf_le('-inf', _) -> true; -inf_le(A, B) -> A =< B. - -inf_ge(_, '-inf') -> true; -inf_ge('-inf', _) -> false; -inf_ge(A, B) -> A >= B. +inf_le('-inf', _) -> + true; +inf_le(A, B) when is_float(A), is_float(B) -> + %% When float ranges are compared to float ranges, the total ordering + %% function must be used to preserve `-0.0 =/= +0.0`. + total_compare(A, B, fun erlang:'=<'/2); +inf_le(A, B) -> + A =< B. + +inf_ge(_, '-inf') -> + true; +inf_ge('-inf', _) -> + false; +inf_ge(A, B) when is_float(A), is_float(B) -> + total_compare(A, B, fun erlang:'>='/2); +inf_ge(A, B) -> + A >= B. + +inf_min(A, B) when A =:= '-inf'; B =:= '-inf' -> + '-inf'; +inf_min(A, B) when is_float(A), is_float(B) -> + case total_compare(A, B, fun erlang:'=<'/2) of + true -> A; + false -> B + end; +inf_min(A, B) -> + min(A, B). -inf_min(A, B) when A =:= '-inf'; B =:= '-inf' -> '-inf'; -inf_min(A, B) when A =< B -> A; -inf_min(A, B) when A > B -> B. +inf_max('-inf', B) -> + B; +inf_max(A, '-inf') -> + A; +inf_max(A, B) when is_float(A), is_float(B) -> + case total_compare(A, B, fun erlang:'>='/2) of + true -> A; + false -> B + end; +inf_max(A, B) -> + max(A, B). -inf_max('-inf', B) -> B; -inf_max(A, '-inf') -> A; -inf_max(A, B) when A >= B -> A; -inf_max(A, B) when A < B -> B. +total_compare(A, B, Order) -> + Order(erts_internal:cmp_term(A, B), 0). %% diff --git a/lib/compiler/test/beam_type_SUITE.erl b/lib/compiler/test/beam_type_SUITE.erl index 49b5cb6e745c..69bfcbe7b448 100644 --- a/lib/compiler/test/beam_type_SUITE.erl +++ b/lib/compiler/test/beam_type_SUITE.erl @@ -31,7 +31,7 @@ switch_fail_inference/1,failures/1, cover_maps_functions/1,min_max_mixed_types/1, not_equal/1,infer_relops/1,binary_unit/1,premature_concretization/1, - funs/1,will_succeed/1]). + funs/1,will_succeed/1,float_confusion/1]). %% Force id/1 to return 'any'. -export([id/1]). @@ -76,7 +76,8 @@ groups() -> binary_unit, premature_concretization, funs, - will_succeed + will_succeed, + float_confusion ]}]. init_per_suite(Config) -> @@ -1505,6 +1506,47 @@ will_succeed_1(_V0, _V1) will_succeed_1(_, _) -> b. +%% GH-7901: Range operations did not honor the total order of floats. +float_confusion(_Config) -> + ok = float_confusion_1(catch (true = ok), -0.0), + ok = float_confusion_1(ok, 0.0), + {'EXIT', _} = catch float_confusion_2(), + {'EXIT', _} = catch float_confusion_3(id(0.0)), + ok = float_confusion_4(id(1)), + {'EXIT', _} = catch float_confusion_5(), + ok. + +float_confusion_1(_, _) -> + ok. + +float_confusion_2() -> + [ok || _ := _ <- ok, + float_confusion_crash(catch float_confusion_crash(ok, -1), -0.0)]. + +float_confusion_crash(_, 18446744073709551615) -> + ok. + +float_confusion_3(V) -> + -0.0 = abs(V), + ok. + +float_confusion_4(V) when -0.0 < floor(V band 1) -> + ok. + +float_confusion_5() -> + -0.0 = + case + fun() -> + ok + end + of + _V2 when (_V2 > ok) -> + 2147483647.0; + _ -> + -2147483648 + end * 0, + ok. + %%% %%% Common utilities. %%% diff --git a/lib/crypto/c_src/crypto_callback.c b/lib/crypto/c_src/crypto_callback.c index e62bc17d4072..4cfcf39b2d45 100644 --- a/lib/crypto/c_src/crypto_callback.c +++ b/lib/crypto/c_src/crypto_callback.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2014-2020. All Rights Reserved. + * Copyright Ericsson AB 2014-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/lib/crypto/c_src/dss.c b/lib/crypto/c_src/dss.c index 0756e9076d15..fdb9c6edd0bf 100644 --- a/lib/crypto/c_src/dss.c +++ b/lib/crypto/c_src/dss.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2010-2022. All Rights Reserved. + * Copyright Ericsson AB 2010-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/lib/crypto/doc/src/algorithm_details.xml b/lib/crypto/doc/src/algorithm_details.xml index 64cbbd3fc45b..a6b0d61aa67c 100644 --- a/lib/crypto/doc/src/algorithm_details.xml +++ b/lib/crypto/doc/src/algorithm_details.xml @@ -4,7 +4,7 @@

- 20142022 + 20142023 Ericsson AB. All Rights Reserved. diff --git a/lib/crypto/doc/src/crypto.xml b/lib/crypto/doc/src/crypto.xml index a7504a7440fe..a1d93faa3dea 100644 --- a/lib/crypto/doc/src/crypto.xml +++ b/lib/crypto/doc/src/crypto.xml @@ -3,7 +3,7 @@
- 19992022 + 19992023 Ericsson AB. All Rights Reserved. diff --git a/lib/crypto/doc/src/crypto_app.xml b/lib/crypto/doc/src/crypto_app.xml index ce527d60f486..249841d851fe 100644 --- a/lib/crypto/doc/src/crypto_app.xml +++ b/lib/crypto/doc/src/crypto_app.xml @@ -52,7 +52,7 @@

The crypto app is tested daily with at least one version of each of the OpenSSL 1.0.1, 1.0.2, 1.1.0, 1.1.1 and 3.0. FIPS mode is also tested for 1.0.1, 1.0.2 and 3.0.

-

Using OpenSSL 3.0 with Engines is supported since OTP @OTP-18832@.

+

Using OpenSSL 3.0 with Engines is supported since OTP 26.2.

Source releases of OpenSSL can be downloaded from the OpenSSL project home page, or mirror sites listed there. @@ -99,4 +99,3 @@

application(3)

- diff --git a/lib/crypto/doc/src/engine_keys.xml b/lib/crypto/doc/src/engine_keys.xml index 1951b97bf3cc..a7accb54a798 100644 --- a/lib/crypto/doc/src/engine_keys.xml +++ b/lib/crypto/doc/src/engine_keys.xml @@ -4,7 +4,7 @@
- 20172020 + 20172023 Ericsson AB. All Rights Reserved. diff --git a/lib/crypto/doc/src/engine_load.xml b/lib/crypto/doc/src/engine_load.xml index 303509ee9069..9680bab34d21 100644 --- a/lib/crypto/doc/src/engine_load.xml +++ b/lib/crypto/doc/src/engine_load.xml @@ -4,7 +4,7 @@
- 20172022 + 20172023 Ericsson AB. All Rights Reserved. diff --git a/lib/crypto/doc/src/fips.xml b/lib/crypto/doc/src/fips.xml index 81ea885f2ddd..a0086ae4e628 100644 --- a/lib/crypto/doc/src/fips.xml +++ b/lib/crypto/doc/src/fips.xml @@ -4,7 +4,7 @@
- 20142017 + 20142023 Ericsson AB. All Rights Reserved. diff --git a/lib/crypto/doc/src/notes.xml b/lib/crypto/doc/src/notes.xml index d4de68aa038c..4d9cd2dfefaf 100644 --- a/lib/crypto/doc/src/notes.xml +++ b/lib/crypto/doc/src/notes.xml @@ -31,6 +31,33 @@

This document describes the changes made to the Crypto application.

+
Crypto 5.4 + +
Fixed Bugs and Malfunctions + + +

+ Fixed some benign compile warnings on Windows.

+

+ Own Id: OTP-18895

+ + +
+ + +
Improvements and New Features + + +

+ Enable engine support for OpenSSL versions 3.

+

+ Own Id: OTP-18832 Aux Id: PR-7763

+ + +
+ +
+
Crypto 5.3
Fixed Bugs and Malfunctions diff --git a/lib/crypto/vsn.mk b/lib/crypto/vsn.mk index 23e59bf5ab6d..bd2ab35bc765 100644 --- a/lib/crypto/vsn.mk +++ b/lib/crypto/vsn.mk @@ -1 +1 @@ -CRYPTO_VSN = 5.3 +CRYPTO_VSN = 5.4 diff --git a/lib/debugger/doc/src/notes.xml b/lib/debugger/doc/src/notes.xml index 0416b8319177..50532508a519 100644 --- a/lib/debugger/doc/src/notes.xml +++ b/lib/debugger/doc/src/notes.xml @@ -33,6 +33,20 @@

This document describes the changes made to the Debugger application.

+
Debugger 5.3.3 + +
Fixed Bugs and Malfunctions + + +

Map comprehensions now work in the Debugger.

+

+ Own Id: OTP-18888 Aux Id: GH-7914

+ + +
+ +
+
Debugger 5.3.2
Fixed Bugs and Malfunctions diff --git a/lib/debugger/vsn.mk b/lib/debugger/vsn.mk index 6c183da437c5..5b8cefe2245f 100644 --- a/lib/debugger/vsn.mk +++ b/lib/debugger/vsn.mk @@ -1 +1 @@ -DEBUGGER_VSN = 5.3.2 +DEBUGGER_VSN = 5.3.3 diff --git a/lib/dialyzer/doc/src/notes.xml b/lib/dialyzer/doc/src/notes.xml index f7d3cc5b5c3b..951d76428fa1 100644 --- a/lib/dialyzer/doc/src/notes.xml +++ b/lib/dialyzer/doc/src/notes.xml @@ -32,6 +32,28 @@

This document describes the changes made to the Dialyzer application.

+
Dialyzer 5.1.2 + +
Fixed Bugs and Malfunctions + + +

+ Fix dialyzer --output flag to work. This option + was accidentally removed in OTP 26.0.

+

+ Own Id: OTP-18767 Aux Id: PR-7657

+ + +

Fixed a crash in contract checking relating to opaque + types.

+

+ Own Id: OTP-18772 Aux Id: GH-7676

+ + +
+ +
+
Dialyzer 5.1.1
Fixed Bugs and Malfunctions diff --git a/lib/dialyzer/vsn.mk b/lib/dialyzer/vsn.mk index b736a532c232..c9cff54439f0 100644 --- a/lib/dialyzer/vsn.mk +++ b/lib/dialyzer/vsn.mk @@ -1 +1 @@ -DIALYZER_VSN = 5.1.1 +DIALYZER_VSN = 5.1.2 diff --git a/lib/diameter/doc/src/notes.xml b/lib/diameter/doc/src/notes.xml index f94211a72ab6..eabcab67e419 100644 --- a/lib/diameter/doc/src/notes.xml +++ b/lib/diameter/doc/src/notes.xml @@ -43,6 +43,23 @@ first.

+
diameter 2.3.1 + +
Fixed Bugs and Malfunctions + + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +
+ +
+
diameter 2.3
Improvements and New Features diff --git a/lib/diameter/src/base/diameter_config_sup.erl b/lib/diameter/src/base/diameter_config_sup.erl index 00e4e3447d09..e10fa21b34ba 100644 --- a/lib/diameter/src/base/diameter_config_sup.erl +++ b/lib/diameter/src/base/diameter_config_sup.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2016. All Rights Reserved. +%% Copyright Ericsson AB 2016-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/diameter/vsn.mk b/lib/diameter/vsn.mk index 173f962b9d8f..59d473be9e55 100644 --- a/lib/diameter/vsn.mk +++ b/lib/diameter/vsn.mk @@ -17,5 +17,5 @@ # %CopyrightEnd% APPLICATION = diameter -DIAMETER_VSN = 2.3 +DIAMETER_VSN = 2.3.1 APP_VSN = $(APPLICATION)-$(DIAMETER_VSN)$(PRE_VSN) diff --git a/lib/edoc/doc/src/notes.xml b/lib/edoc/doc/src/notes.xml index a3a89e681037..2e3e7fb7305d 100644 --- a/lib/edoc/doc/src/notes.xml +++ b/lib/edoc/doc/src/notes.xml @@ -32,6 +32,21 @@

This document describes the changes made to the EDoc application.

+
Edoc 1.2.1 + +
Fixed Bugs and Malfunctions + + +

+ Emit <code> instead of <tt>.

+

+ Own Id: OTP-18782 Aux Id: PR-7643

+ + +
+ +
+
Edoc 1.2
Fixed Bugs and Malfunctions diff --git a/lib/edoc/vsn.mk b/lib/edoc/vsn.mk index f8feac1c374c..754954c97f0a 100644 --- a/lib/edoc/vsn.mk +++ b/lib/edoc/vsn.mk @@ -1 +1 @@ -EDOC_VSN = 1.2 +EDOC_VSN = 1.2.1 diff --git a/lib/eldap/doc/src/notes.xml b/lib/eldap/doc/src/notes.xml index 881c1215b407..6ec3cd9ea46b 100644 --- a/lib/eldap/doc/src/notes.xml +++ b/lib/eldap/doc/src/notes.xml @@ -31,6 +31,21 @@

This document describes the changes made to the Eldap application.

+
Eldap 1.2.12 + +
Fixed Bugs and Malfunctions + + +

+ Add missing dependency to asn1 application

+

+ Own Id: OTP-18810

+ + +
+ +
+
Eldap 1.2.11
Improvements and New Features diff --git a/lib/eldap/vsn.mk b/lib/eldap/vsn.mk index a25f97cb8998..f8a7c0e149b2 100644 --- a/lib/eldap/vsn.mk +++ b/lib/eldap/vsn.mk @@ -1 +1 @@ -ELDAP_VSN = 1.2.11 +ELDAP_VSN = 1.2.12 diff --git a/lib/erl_docgen/doc/src/notes.xml b/lib/erl_docgen/doc/src/notes.xml index 7bf93c4d6ba3..dc795fb909c1 100644 --- a/lib/erl_docgen/doc/src/notes.xml +++ b/lib/erl_docgen/doc/src/notes.xml @@ -31,7 +31,40 @@

This document describes the changes made to the erl_docgen application.

-
Erl_Docgen 1.5.1 +
Erl_Docgen 1.5.2 + +
Fixed Bugs and Malfunctions + + +

+ Fix erl_docgen dtd to only allow a single + datatype_title within a datatypes block.

+

+ Own Id: OTP-18775 Aux Id: PR-7663

+ + +

+ Fix so that EEP-48 doc chunks include the module summary + and generates equiv tags in the correct order.

+

+ The function/type group title is now also included in the + entry metadata.

+

+ Own Id: OTP-18776 Aux Id: PR-7663

+ + +

+ Update all <tt> html tags to be <code> + instead.

+

+ Own Id: OTP-18799 Aux Id: PR-7695

+ + +
+ +
+ +
Erl_Docgen 1.5.1
Fixed Bugs and Malfunctions diff --git a/lib/erl_docgen/priv/bin/xml_from_edoc.escript b/lib/erl_docgen/priv/bin/xml_from_edoc.escript index 4ef63e7fe2c7..42dfc745fc5e 100755 --- a/lib/erl_docgen/priv/bin/xml_from_edoc.escript +++ b/lib/erl_docgen/priv/bin/xml_from_edoc.escript @@ -2,7 +2,7 @@ %% -*- erlang -*- %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2018. All Rights Reserved. +%% Copyright Ericsson AB 2010-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/erl_docgen/src/docgen_xml_to_chunk.erl b/lib/erl_docgen/src/docgen_xml_to_chunk.erl index 32c500453ba8..348dd46b3361 100644 --- a/lib/erl_docgen/src/docgen_xml_to_chunk.erl +++ b/lib/erl_docgen/src/docgen_xml_to_chunk.erl @@ -35,7 +35,7 @@ main([_Application, FromBeam, _Escript, ToChunk]) -> Anno = erl_anno:set_file(Name, erl_anno:new(0)), - EmptyDocs = add_hidden_docs( + EmptyDocs = add_hidden_functions( Exports, #docs_v1{ anno = Anno, module_doc = hidden, @@ -284,10 +284,11 @@ docs(Application, OTPXml, FromBEAM)-> put(application, Application), put(module, filename:basename(filename:rootname(FromBEAM))), NewDom = transform(Dom, []), - Chunk = add_hidden_docs( - proplists:get_value(exports, Chunks), - to_chunk(NewDom, OTPXml, Module, - proplists:get_value(abstract_code, Chunks))), + Chunk = add_hidden_types( + add_hidden_functions( + proplists:get_value(exports, Chunks), + to_chunk(NewDom, OTPXml, Module, + proplists:get_value(abstract_code, Chunks)))), verify_chunk(Module, proplists:get_value(exports, Chunks), Chunk), Chunk; _Else -> @@ -299,7 +300,7 @@ docs(Application, OTPXml, FromBEAM)-> %% Create hidden function entries for any exported functions that %% does not have any documentation. -add_hidden_docs(Exports, #docs_v1{ anno = Anno, docs = Docs } = Chunk) -> +add_hidden_functions(Exports, #docs_v1{ anno = Anno, docs = Docs } = Chunk) -> HiddenFuncs = [{{function, F, A}, Anno, [iolist_to_binary(io_lib:format("~p/~p", [F, A]))], @@ -307,6 +308,15 @@ add_hidden_docs(Exports, #docs_v1{ anno = Anno, docs = Docs } = Chunk) -> lists:keysearch({function, F, A}, 1, Docs) == false ], Chunk#docs_v1{ docs = HiddenFuncs ++ Docs }. +add_hidden_types(#docs_v1{ anno = Anno, docs = Docs, metadata = Meta } = Chunk) -> + Types = maps:get(types, Meta, []), + HiddenTypes = + [{{type, F, A}, Anno, + [iolist_to_binary(io_lib:format("-type ~p/~p", [F, A]))], + hidden, #{}} || {F, A} := _ <- Types, + lists:keysearch({type, F, A}, 1, Docs) == false ], + Chunk#docs_v1{ docs = HiddenTypes ++ Docs }. + verify_chunk(M, Exports, #docs_v1{ docs = Docs } = Doc) -> %% Make sure that each documented function actually is exported diff --git a/lib/erl_docgen/vsn.mk b/lib/erl_docgen/vsn.mk index f24eb8ab8031..9b1d2879a6ab 100644 --- a/lib/erl_docgen/vsn.mk +++ b/lib/erl_docgen/vsn.mk @@ -1 +1 @@ -ERL_DOCGEN_VSN = 1.5.1 +ERL_DOCGEN_VSN = 1.5.2 diff --git a/lib/erl_interface/doc/src/ei.xml b/lib/erl_interface/doc/src/ei.xml index 305591bd2752..774e4c234460 100644 --- a/lib/erl_interface/doc/src/ei.xml +++ b/lib/erl_interface/doc/src/ei.xml @@ -4,7 +4,7 @@
- 20012020 + 20012023 Ericsson AB. All Rights Reserved. @@ -1278,7 +1278,7 @@ encodes the tuple {numbers,12,3.14159} without the initial version byte.

- Since OTP @OTP-18764@ maps can be encoded with syntax like + Since OTP 26.2 maps can be encoded with syntax like "#{k1 => v1, k2 => v2}".

diff --git a/lib/erl_interface/doc/src/notes.xml b/lib/erl_interface/doc/src/notes.xml index a221677caec9..ae92b9c3c517 100644 --- a/lib/erl_interface/doc/src/notes.xml +++ b/lib/erl_interface/doc/src/notes.xml @@ -31,6 +31,55 @@

This document describes the changes made to the Erl_interface application.

+
Erl_Interface 5.5 + +
Fixed Bugs and Malfunctions + + +

+ Fixed some benign compile warnings on Windows.

+

+ Own Id: OTP-18895

+ + +
+ + +
Improvements and New Features + + +

+ Add support to encode maps with ei_x_format.

+

+ Own Id: OTP-18764 Aux Id: PR-7602

+ + +

+ Replaced old md5 implementation with an implementation + from OpenSSL.

+

+ Own Id: OTP-18877

+ + +
+ + +
Known Bugs and Problems + + +

+ The ei API for decoding/encoding terms is not + fully 64-bit compatible since terms that have a + representation on the external term format larger than 2 + GB cannot be handled.

+

+ Own Id: OTP-16607 Aux Id: OTP-16608

+ + +
+ +
+
Erl_Interface 5.4
Improvements and New Features @@ -81,6 +130,22 @@
+
Erl_Interface 5.3.2.1 + +
Improvements and New Features + + +

+ Replaced old md5 implementation with an implementation + from OpenSSL.

+

+ Own Id: OTP-18877

+ + +
+ +
+
Erl_Interface 5.3.2
Fixed Bugs and Malfunctions @@ -199,6 +264,22 @@
+
Erl_Interface 5.2.2.1 + +
Improvements and New Features + + +

+ Replaced old md5 implementation with an implementation + from OpenSSL.

+

+ Own Id: OTP-18877

+ + +
+ +
+
Erl_Interface 5.2.2
Fixed Bugs and Malfunctions diff --git a/lib/erl_interface/src/Makefile.in b/lib/erl_interface/src/Makefile.in index 282250279826..fa1ea3cd394d 100644 --- a/lib/erl_interface/src/Makefile.in +++ b/lib/erl_interface/src/Makefile.in @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 1997-2022. All Rights Reserved. +# Copyright Ericsson AB 1997-2023. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/lib/erl_interface/src/misc/ei_format.c b/lib/erl_interface/src/misc/ei_format.c index 80feb6b933d3..6ae5521357b0 100644 --- a/lib/erl_interface/src/misc/ei_format.c +++ b/lib/erl_interface/src/misc/ei_format.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2001-2021. All Rights Reserved. + * Copyright Ericsson AB 2001-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/lib/erl_interface/test/ei_format_SUITE.erl b/lib/erl_interface/test/ei_format_SUITE.erl index 1d84111ec7c3..cabb9029736d 100644 --- a/lib/erl_interface/test/ei_format_SUITE.erl +++ b/lib/erl_interface/test/ei_format_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2001-2018. All Rights Reserved. +%% Copyright Ericsson AB 2001-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/erl_interface/test/ei_format_SUITE_data/ei_format_test.c b/lib/erl_interface/test/ei_format_SUITE_data/ei_format_test.c index f9b0f40c260d..f09961c87b9e 100644 --- a/lib/erl_interface/test/ei_format_SUITE_data/ei_format_test.c +++ b/lib/erl_interface/test/ei_format_SUITE_data/ei_format_test.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2001-2020. All Rights Reserved. + * Copyright Ericsson AB 2001-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/lib/erl_interface/vsn.mk b/lib/erl_interface/vsn.mk index b68159d9474b..fc72b67ecb61 100644 --- a/lib/erl_interface/vsn.mk +++ b/lib/erl_interface/vsn.mk @@ -1,2 +1,2 @@ -EI_VSN = 5.4 +EI_VSN = 5.5 ERL_INTERFACE_VSN = $(EI_VSN) diff --git a/lib/eunit/doc/src/notes.xml b/lib/eunit/doc/src/notes.xml index 1f1786159cb4..c584cf0bb758 100644 --- a/lib/eunit/doc/src/notes.xml +++ b/lib/eunit/doc/src/notes.xml @@ -33,6 +33,22 @@

This document describes the changes made to the EUnit application.

+
Eunit 2.9 + +
Improvements and New Features + + +

+ With this change, EUnit timetraps can be scaled with the + use of scale_timeouts option.

+

+ Own Id: OTP-18771 Aux Id: PR-7635

+ + +
+ +
+
Eunit 2.8.2
Improvements and New Features diff --git a/lib/eunit/test/eunit_SUITE.erl b/lib/eunit/test/eunit_SUITE.erl index 377dc97557c8..caf96a9075ca 100644 --- a/lib/eunit/test/eunit_SUITE.erl +++ b/lib/eunit/test/eunit_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2010-2022. All Rights Reserved. +%% Copyright Ericsson AB 2010-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/eunit/vsn.mk b/lib/eunit/vsn.mk index 81508f8ab4f2..3735d33bc8a9 100644 --- a/lib/eunit/vsn.mk +++ b/lib/eunit/vsn.mk @@ -1 +1 @@ -EUNIT_VSN = 2.8.2 +EUNIT_VSN = 2.9 diff --git a/lib/ftp/doc/src/introduction.xml b/lib/ftp/doc/src/introduction.xml index 4be13a71df44..194bc465f7c3 100644 --- a/lib/ftp/doc/src/introduction.xml +++ b/lib/ftp/doc/src/introduction.xml @@ -4,7 +4,7 @@
- 19972018 + 19972023 Ericsson AB. All Rights Reserved. diff --git a/lib/ftp/doc/src/notes.xml b/lib/ftp/doc/src/notes.xml index 2db120cc28af..1f2c040117f9 100644 --- a/lib/ftp/doc/src/notes.xml +++ b/lib/ftp/doc/src/notes.xml @@ -33,7 +33,24 @@ notes.xml
-
Ftp 1.2 +
Ftp 1.2.1 + +
Fixed Bugs and Malfunctions + + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +
+ +
+ +
Ftp 1.2
Fixed Bugs and Malfunctions diff --git a/lib/ftp/vsn.mk b/lib/ftp/vsn.mk index 0fd93d82a504..1bfc35dc6628 100644 --- a/lib/ftp/vsn.mk +++ b/lib/ftp/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = ftp -FTP_VSN = 1.2 +FTP_VSN = 1.2.1 PRE_VSN = APP_VSN = "$(APPLICATION)-$(FTP_VSN)$(PRE_VSN)" diff --git a/lib/inets/doc/src/httpd.xml b/lib/inets/doc/src/httpd.xml index 45d933f3a3d7..8ea35fa471fb 100644 --- a/lib/inets/doc/src/httpd.xml +++ b/lib/inets/doc/src/httpd.xml @@ -4,7 +4,7 @@
- 19972022 + 19972023 Ericsson AB. All Rights Reserved. diff --git a/lib/inets/doc/src/introduction.xml b/lib/inets/doc/src/introduction.xml index 04730e75c99e..5a36bf24586e 100644 --- a/lib/inets/doc/src/introduction.xml +++ b/lib/inets/doc/src/introduction.xml @@ -4,7 +4,7 @@
- 19972021 + 19972023 Ericsson AB. All Rights Reserved. diff --git a/lib/inets/doc/src/notes.xml b/lib/inets/doc/src/notes.xml index 2e9b55541eca..0add69c35168 100644 --- a/lib/inets/doc/src/notes.xml +++ b/lib/inets/doc/src/notes.xml @@ -33,7 +33,59 @@ notes.xml
-
Inets 9.0.2 +
Inets 9.1 + +
Fixed Bugs and Malfunctions + + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +

+ Correct IP protocol handling so that redirects always + uses correct IP-family options and not fails.

+

+ Own Id: OTP-18855

+ + +
+ + +
Improvements and New Features + + +

+ inets app starts ssl by default

+

+ Own Id: OTP-18735 Aux Id: PR-7596, GH-7580

+ + +

+ Avoid httpd returning 500 internal server error when + unable to open a file. 404 or 503 will be returned + instead.

+

+ Own Id: OTP-18882

+ + +

+ Properly handle documented option mime_type, for + backwards compatibility fallback to undocumented option + default_type if mime_type is not set.

+

+ Own Id: OTP-18891 Aux Id: PR-7843, GH-7827

+ + +
+ +
+ +
Inets 9.0.2
Fixed Bugs and Malfunctions diff --git a/lib/inets/src/http_client/httpc.erl b/lib/inets/src/http_client/httpc.erl index 74ad5a93b0e1..6ccbb63dae56 100644 --- a/lib/inets/src/http_client/httpc.erl +++ b/lib/inets/src/http_client/httpc.erl @@ -159,7 +159,7 @@ request(Url, Profile) -> | {ipv6_host_with_brackets, boolean()}, StreamTo :: none | self | {self, once} | file:name_all(), SocketOpt :: term(), - BodyFormat :: string() | binary() | atom(), + BodyFormat :: string | binary, Receiver :: pid() | fun((term()) -> term()) | { ReceiverModule::atom() @@ -220,7 +220,7 @@ request(Method, Request, HttpOptions, Options) -> | {receiver, Receiver} | {ipv6_host_with_brackets, boolean()}, StreamTo :: none | self | {self, once} | file:name_all(), - BodyFormat :: string() | binary() | atom(), + BodyFormat :: string | binary, SocketOpt :: term(), Receiver :: pid() | fun((term()) -> term()) @@ -903,7 +903,7 @@ maybe_format_body(BinBody, Options) -> | {socket_opts, [SocketOpt]} | {receiver, Receiver} | {ipv6_host_with_brackets, boolean()}, - BodyFormat :: string() | binary() | atom(), + BodyFormat :: string | binary, StreamTo :: none | self | {self, once} | file:name_all(), SocketOpt :: term(), Receiver :: pid() @@ -1150,7 +1150,7 @@ request_options([{Key, DefaultVal, Verify} | Defaults], Options, Acc) -> | {receiver, Receiver} | {ipv6_host_with_brackets, boolean()}, StreamTo :: none | self | {self, once} | file:name_all(), - BodyFormat :: string() | binary() | atom(), + BodyFormat :: string | binary, SocketOpt :: term(), Receiver :: pid() | fun((term()) -> term()) diff --git a/lib/inets/src/http_server/httpd_file.erl b/lib/inets/src/http_server/httpd_file.erl index f904f741c7e3..2684369ae20b 100644 --- a/lib/inets/src/http_server/httpd_file.erl +++ b/lib/inets/src/http_server/httpd_file.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2006-2022. All Rights Reserved. +%% Copyright Ericsson AB 2006-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/inets/src/inets_app/inets.erl b/lib/inets/src/inets_app/inets.erl index 35da4efc8eaf..67885fdeb33b 100644 --- a/lib/inets/src/inets_app/inets.erl +++ b/lib/inets/src/inets_app/inets.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2006-2022. All Rights Reserved. +%% Copyright Ericsson AB 2006-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/inets/test/http_test_lib.erl b/lib/inets/test/http_test_lib.erl index 5f58dbbe8b01..38dd2a202735 100644 --- a/lib/inets/test/http_test_lib.erl +++ b/lib/inets/test/http_test_lib.erl @@ -28,6 +28,7 @@ %% Note: This directive should only be used in test suites. -compile(export_all). -compile(nowarn_export_all). +-define(SOCKET_BACKLOG, 100). dummy_server(SocketType, Inet, Extra) -> dummy_server(self(), SocketType, Inet, Extra). @@ -42,7 +43,8 @@ dummy_server(Caller, SocketType, Inet, Extra) -> dummy_server_init(Caller, ip_comm, Inet, Extra) -> ContentCb = proplists:get_value(content_cb, Extra), - BaseOpts = [binary, {packet, 0}, {reuseaddr,true}, {active, false}, {nodelay, true}], + BaseOpts = [binary, {packet, 0}, {reuseaddr,true}, + {active, false}, {nodelay, true}, {backlog, ?SOCKET_BACKLOG}], Conf = proplists:get_value(conf, Extra), {ok, ListenSocket} = gen_tcp:listen(0, [Inet | BaseOpts]), {ok, Port} = inet:port(ListenSocket), @@ -60,8 +62,8 @@ dummy_server_init(Caller, unix_socket, Inet, Extra) -> ContentCb = proplists:get_value(content_cb, Extra), UnixSocket = proplists:get_value(unix_socket, Extra), SocketAddr = {local, UnixSocket}, - BaseOpts = [binary, {packet, 0}, {reuseaddr,true}, {active, false}, {nodelay, true}, - {ifaddr, SocketAddr}], + BaseOpts = [binary, {packet, 0}, {reuseaddr,true}, {active, false}, + {nodelay, true}, {ifaddr, SocketAddr}, {backlog, ?SOCKET_BACKLOG}], Conf = proplists:get_value(conf, Extra), {ok, ListenSocket} = gen_tcp:listen(0, [Inet | BaseOpts]), {ok, Port} = inet:port(ListenSocket), @@ -79,7 +81,8 @@ dummy_server_init(Caller, ssl, Inet, Extra) -> ContentCb = proplists:get_value(content_cb, Extra), SSLOptions = proplists:get_value(ssl, Extra), Conf = proplists:get_value(conf, Extra), - BaseOpts = [binary, {active, false}, {nodelay, true} | SSLOptions], + BaseOpts = [binary, {active, false}, {nodelay, true}, + {backlog, ?SOCKET_BACKLOG} | SSLOptions], dummy_ssl_server_init(Caller, BaseOpts, Inet, ContentCb, Conf). dummy_ssl_server_init(Caller, BaseOpts, Inet, ContentCb, Conf) -> diff --git a/lib/inets/test/httpd_serve_SUITE.erl b/lib/inets/test/httpd_serve_SUITE.erl index 0469498a54db..e7d545e0f3ea 100644 --- a/lib/inets/test/httpd_serve_SUITE.erl +++ b/lib/inets/test/httpd_serve_SUITE.erl @@ -105,12 +105,17 @@ run_server_assertions({ok, {Ip0, Port, Path}}, Assertions) when is_integer(Port) ok = verify_assertions(Assertions, ServerInfo), ct:comment("Ran ~w assertion(s).", [length(Assertions)]). -maybe_convert_to_localhost(Ip = {0, 0, 0, 0}) -> - case os:type() of - {win32, _} -> {127, 0, 0, 1}; - _ -> Ip - end; maybe_convert_to_localhost(Ip) -> + case os:type() of + {unix, linux} -> Ip; + _Other -> convert_to_localhost(Ip) + end. + +convert_to_localhost({0, 0, 0, 0}) -> + {127, 0, 0, 1}; +convert_to_localhost({0, 0, 0, 0, 0, 0, 0, 0}) -> + {0, 0, 0, 0, 0, 0, 0, 1}; +convert_to_localhost(Ip) -> Ip. %% @@ -120,24 +125,10 @@ maybe_convert_to_localhost(Ip) -> verify_200_at(Url) -> HttpcOpts = [{socket_opts, [{ipfamily, inet6fb4}]}], Request = {Url, []}, - Response = httpc:request(get, Request, [{autoredirect, false}], HttpcOpts), + Response = httpc:request(get, Request, [], HttpcOpts), case Response of {ok, {{_Version, 200, _}, _Headers, _Body}} -> Response; - {ok, {{_Version, 301, _}, Headers, _Body}} -> - % To be resilient against the case where the server is not - % reachable under its `server_name`, for instance, in test - % containers or other hosts with unexpected networking setups, - % replace the suggested hostname with the hostname we came from. - {_, SuggestedTarget} = proplists:lookup("location", Headers), - #{path := SuggestedPath} = uri_string:parse(SuggestedTarget), - OurUri = uri_string:parse(Url), - DecomposedTarget = maps:put(path, SuggestedPath, OurUri), - RedirectTarget = uri_string:recompose(DecomposedTarget), - RedirectedRequest = {RedirectTarget, []}, - RedirectedResponse = httpc:request(get, RedirectedRequest, [], HttpcOpts), - ct:log("Following redirect (rewritten from ~s to ~s)", [SuggestedTarget, RedirectTarget]), - {ok, {{_, 200, _}, _, _}} = RedirectedResponse; {error, {failed_connect, [{to_address, {"::", _Port}}, {inet6, [inet6], eaddrnotavail}, {inet, [inet], nxdomain}]}} -> diff --git a/lib/inets/vsn.mk b/lib/inets/vsn.mk index 2072aeaa0b31..eb20b7960389 100644 --- a/lib/inets/vsn.mk +++ b/lib/inets/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = inets -INETS_VSN = 9.0.2 +INETS_VSN = 9.1 PRE_VSN = APP_VSN = "$(APPLICATION)-$(INETS_VSN)$(PRE_VSN)" diff --git a/lib/kernel/doc/src/code.xml b/lib/kernel/doc/src/code.xml index bbd6e345ddc5..66090cc6fde2 100644 --- a/lib/kernel/doc/src/code.xml +++ b/lib/kernel/doc/src/code.xml @@ -233,7 +233,8 @@ zip:create("mnesia-4.4.7.ez", script can be strict or relaxed. It is particularly useful to set the flag to relaxed when elaborating with code loading from archives without editing the - boot script. The default is relaxed. See boot script. The default has changed to strict + in OTP 27 and the option is scheduled for removal in OTP 28. See erts:init(3).

diff --git a/lib/kernel/doc/src/notes.xml b/lib/kernel/doc/src/notes.xml index 0d2ab505cec0..9389d44a61d1 100644 --- a/lib/kernel/doc/src/notes.xml +++ b/lib/kernel/doc/src/notes.xml @@ -31,6 +31,98 @@

This document describes the changes made to the Kernel application.

+
Kernel 9.2 + +
Fixed Bugs and Malfunctions + + +

+ For inet_backend = socket, an unexpected receive + error such as etimedout caused the receiving state + machine server to crash. This bug has now been fixed.

+

+ Own Id: OTP-18749 Aux Id: GH-7608

+ + +

+ Fix bug where reading using file from a unicode + enabled standard_io, standard_error or any + other group backed device would result in + incorrect values being returned or a crash.

+

+ Now instead a no_translation error is returned to the + caller when unicode data is read using file. See + Using + Unicode in the STDLIB User's Guide for more + details on how to correctly read from standard_io.

+

+ Own Id: OTP-18800 Aux Id: PR-7714 GH-7591

+ + +

+ The native resolver interface module has gotten a rewrite + of its ETS table handling to minimize term copying, and + also to move the handling of client time-outs to the + clients, which helps the native resolver name server from + digging itself into a tar pit when heavily loaded.

+

+ Own Id: OTP-18812 Aux Id: ERIERL-997

+ + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +

+ Fix bug in pg if a client process both monitored a + group/scope and joined a group. The termination of such + process resulted in crash of the pg server + process.

+

+ Own Id: OTP-18833 Aux Id: GH-7625, PR-7659

+ + +

+ Fix crash when using file:consult and the + underlying file read returns an error while reading.

+

+ Own Id: OTP-18873 Aux Id: PR-7831

+ + +

+ Corrected gen_tcp_socket listen option handling.

+

+ Own Id: OTP-18883 Aux Id: #7764

+ + +
+ + +
Improvements and New Features + + +

+ Add Windows support for DGRAM socket connect.

+

+ Own Id: OTP-18762

+ + +

+ Document the, previously opaque, types select_tag() and + completion_tag().

+

+ Own Id: OTP-18818 Aux Id: #7337

+ + +
+ +
+
Kernel 9.1
Fixed Bugs and Malfunctions diff --git a/lib/kernel/src/Makefile b/lib/kernel/src/Makefile index 8ac476867407..162174f3bcc6 100644 --- a/lib/kernel/src/Makefile +++ b/lib/kernel/src/Makefile @@ -186,7 +186,6 @@ APPUP_TARGET= $(EBIN)/$(APPUP_FILE) ERL_COMPILE_FLAGS += -Werror ERL_COMPILE_FLAGS += -I../include -ERL_COMPILE_FLAGS += -pa ../ebin # ---------------------------------------------------- diff --git a/lib/kernel/src/code.erl b/lib/kernel/src/code.erl index b54e105ac3c3..304638e0a628 100644 --- a/lib/kernel/src/code.erl +++ b/lib/kernel/src/code.erl @@ -187,7 +187,7 @@ objfile_extension() -> load_file(Mod) when is_atom(Mod) -> case get_object_code(Mod) of error -> {error,nofile}; - {Mod,Binary,File} -> load_module(Mod, File, Binary, false, false) + {Mod,Binary,File} -> load_module(Mod, File, Binary, false) end. -spec ensure_loaded(Module) -> {module, Module} | {error, What} when @@ -200,7 +200,13 @@ ensure_loaded(Mod) when is_atom(Mod) -> case call({get_object_code_for_loading, Mod}) of {module, Mod} -> {module, Mod}; {error, What} -> {error, What}; - {Mod,Binary,File,Ref} -> load_module(Mod, File, Binary, false, Ref) + {Binary,File,Ref} -> + case erlang:prepare_loading(Mod, Binary) of + {error,_}=Error -> + call({load_error, Ref, Mod, Error}); + Prepared -> + call({load_module, Prepared, Mod, File, false, Ref}) + end end end. @@ -219,7 +225,7 @@ load_abs(File, M) when (is_list(File) orelse is_atom(File)), is_atom(M) -> FileName = code_server:absname(FileName0), case erl_prim_loader:get_file(FileName) of {ok,Bin,_} -> - load_module(M, FileName, Bin, false, false); + load_module(M, FileName, Bin, false); error -> {error, nofile} end; @@ -237,16 +243,16 @@ load_abs(File, M) when (is_list(File) orelse is_atom(File)), is_atom(M) -> load_binary(Mod, File, Bin) when is_atom(Mod), (is_list(File) orelse is_atom(File)), is_binary(Bin) -> case modp(File) of - true -> load_module(Mod, File, Bin, true, false); + true -> load_module(Mod, File, Bin, true); false -> {error,badarg} end. -load_module(Mod, File, Bin, Purge, EnsureLoaded) -> +load_module(Mod, File, Bin, Purge) -> case erlang:prepare_loading(Mod, Bin) of {error,_}=Error -> Error; Prepared -> - call({load_module, Prepared, Mod, File, Purge, EnsureLoaded}) + call({load_module, Prepared, Mod, File, Purge, false}) end. modp(Atom) when is_atom(Atom) -> true; diff --git a/lib/kernel/src/code_server.erl b/lib/kernel/src/code_server.erl index c5ba677c6e85..b96589fbf768 100644 --- a/lib/kernel/src/code_server.erl +++ b/lib/kernel/src/code_server.erl @@ -311,6 +311,9 @@ handle_call({load_module,PC,Mod,File,Purge,EnsureLoaded}, From, S) end, try_finish_module(File, Mod, PC, EnsureLoaded, From, S); +handle_call({load_error,Ref,Mod,Error}, _From, S) -> + reply_loading(Ref, Mod, Error, S); + handle_call({delete,Mod}, _From, St) when is_atom(Mod) -> case catch erlang:delete_module(Mod) of true -> @@ -339,11 +342,17 @@ handle_call({get_object_code,Mod}, _From, St0) when is_atom(Mod) -> handle_call({get_object_code_for_loading,Mod}, From, St0) when is_atom(Mod) -> case erlang:module_loaded(Mod) of true -> {reply, {module, Mod}, St0}; - false -> - case wait_loading(St0, Mod, From) of - {true, St1} -> {noreply, St1}; - false -> get_object_code_for_loading(St0, Mod, From) - end + false when St0#state.mode =:= interactive -> + %% Handles pending on_load events first. If the code is being + %% loaded, finish before adding more entries to the queue. + Action = fun(_, St1) -> + case erlang:module_loaded(Mod) of + true -> {reply, {module, Mod}, St1}; + false -> get_object_code_for_loading(St1, Mod, From) + end + end, + handle_pending_on_load(Action, Mod, From, St0); + false -> {reply, {error,embedded}, St0} end; handle_call(stop,_From, S) -> @@ -1122,45 +1131,34 @@ try_finish_module(File, Mod, PC, EnsureLoaded, From, St) -> _ -> fun(_, S0) -> case erlang:module_loaded(Mod) of - true when is_reference(EnsureLoaded) -> - S = done_loading(St, Mod, EnsureLoaded, {module, Mod}), - {reply,{module,Mod},S}; true -> - {reply,{module,Mod},S0}; + reply_loading(EnsureLoaded, Mod, {module, Mod}, St); false when S0#state.mode =:= interactive -> try_finish_module_1(File, Mod, PC, From, EnsureLoaded, S0); false -> - {reply,{error,embedded},S0} + reply_loading(EnsureLoaded, Mod, {error, embedded}, St) end end end, handle_pending_on_load(Action, Mod, From, St). -try_finish_module_1(File, Mod, PC, From, EnsureRef, #state{moddb=Db}=St) -> +try_finish_module_1(File, Mod, PC, From, EnsureLoaded, #state{moddb=Db}=St) -> case is_sticky(Mod, Db) of true -> %% Sticky file reject the load error_msg("Can't load module '~w' that resides in sticky dir\n",[Mod]), - {reply,{error,sticky_directory},St}; + reply_loading(EnsureLoaded, Mod, {error,sticky_directory}, St); false -> - try_finish_module_2(File, Mod, PC, From, EnsureRef, St) + try_finish_module_2(File, Mod, PC, From, EnsureLoaded, St) end. -try_finish_module_2(File, Mod, PC, From, EnsureRef, St0) -> - Action = fun(Result, #state{moddb=Db}=S0) -> - S = case is_reference(EnsureRef) of - true -> done_loading(S0, Mod, EnsureRef, Result); - false -> S0 - end, +try_finish_module_2(File, Mod, PC, From, EnsureLoaded, St0) -> + Action = fun(Result, #state{moddb=Db}=St1) -> case Result of - {module, _} = Module -> - ets:insert(Db, {Mod, File}), - {reply, Module, S}; - {error, on_load_failure} = Error -> - {reply, Error, S}; - {error, What} = Error -> - error_msg("Loading of ~ts failed: ~p\n", [File, What]), - {reply, Error, S} - end + {module, _} -> ets:insert(Db, {Mod, File}); + {error, on_load_failure} -> ok; + {error, What} -> error_msg("Loading of ~ts failed: ~p\n", [File, What]) + end, + reply_loading(EnsureLoaded, Mod, Result, St1) end, Res = case erlang:finish_loading([PC]) of ok -> @@ -1191,22 +1189,16 @@ get_object_code(#state{path=Path} = St, Mod) when is_atom(Mod) -> end. get_object_code_for_loading(St0, Mod, From) -> - case get_object_code(St0, Mod) of - {Bin, FName, St1} -> - {Ref, St2} = monitor_loader(St1, Mod, From, Bin, FName), - {reply, {Mod, Bin, FName, Ref}, St2}; - {error, St1} -> - %% Handles pending on_load events when we cannot find any - %% object code. It's possible that the user has loaded - %% a binary that has an on_load function, and in that case - %% we need to suspend them until the on_load function finishes. - handle_pending_on_load( - fun(_, St) -> - case erlang:module_loaded(Mod) of - true -> {reply, {module, Mod}, St}; - false -> {reply, {error, nofile}, St} - end - end, Mod, From, St1) + case wait_loading(St0, Mod, From) of + {true, St1} -> {noreply, St1}; + false -> + case get_object_code(St0, Mod) of + {Bin, FName, St1} -> + {Ref, St2} = monitor_loader(St1, Mod, From, Bin, FName), + {reply, {Bin, FName, Ref}, St2}; + {error, St1} -> + {reply, {error, nofile}, St1} + end end. monitor_loader(#state{loading = Loading0} = St, Mod, Pid, Bin, FName) -> @@ -1224,11 +1216,14 @@ wait_loading(#state{loading = Loading0} = St, Mod, Pid) -> false end. -done_loading(#state{loading = Loading0} = St, Mod, Ref, Res) -> +reply_loading(Ref, Mod, Reply, #state{loading = Loading0} = St) + when is_reference(Ref) -> {Waiting, Loading} = maps:take(Mod, Loading0), - _ = [reply(Pid, Res) || Pid <- Waiting], + _ = [reply(Pid, Reply) || Pid <- Waiting], erlang:demonitor(Ref, [flush]), - St#state{loading = Loading}. + {reply, Reply, St#state{loading = Loading}}; +reply_loading(Ref, _Mod, Reply, St) when is_boolean(Ref) -> + {reply, Reply, St}. loader_down(#state{loading = Loading0} = St, {Mod, Bin, FName}) -> case Loading0 of @@ -1236,7 +1231,7 @@ loader_down(#state{loading = Loading0} = St, {Mod, Bin, FName}) -> Tag = {'LOADER_DOWN', {Mod, Bin, FName}}, Ref = erlang:monitor(process, First, [{tag, Tag}]), Loading = Loading0#{Mod := Rest}, - _ = reply(First, {Mod, Bin, FName, Ref}), + _ = reply(First, {Bin, FName, Ref}), St#state{loading = Loading}; #{Mod := []} -> Loading = maps:remove(Mod, Loading0), @@ -1409,7 +1404,7 @@ handle_on_load(Res, Action, _, _, St) -> handle_pending_on_load(Action, Mod, From, #state{on_load=OnLoad0}=St) -> case lists:keyfind(Mod, 2, OnLoad0) of false -> - Action(ok, St); + Action({module, Mod}, St); {{From,_Ref},Mod,_Pids} -> %% The on_load function tried to make an external %% call to its own module. That would be a deadlock. diff --git a/lib/kernel/src/erl_erts_errors.erl b/lib/kernel/src/erl_erts_errors.erl index d21dc8267b20..9ec7c25d5b71 100644 --- a/lib/kernel/src/erl_erts_errors.erl +++ b/lib/kernel/src/erl_erts_errors.erl @@ -683,9 +683,9 @@ format_erlang_error(open_port, [Name, Settings], Cause) -> must_be_tuple(Name) end; format_erlang_error(phash, [_,N], _) -> - [must_be_pos_int(N)]; + [[], must_be_pos_int(N)]; format_erlang_error(phash2, [_,N], _) -> - [must_be_pos_int(N)]; + [[], must_be_pos_int(N)]; format_erlang_error(posixtime_to_universaltime, [_], _) -> [not_integer]; format_erlang_error(pid_to_list, [_], _) -> diff --git a/lib/kernel/src/erl_signal_handler.erl b/lib/kernel/src/erl_signal_handler.erl index a68291022714..5ea96a12f9fe 100644 --- a/lib/kernel/src/erl_signal_handler.erl +++ b/lib/kernel/src/erl_signal_handler.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2018. All Rights Reserved. +%% Copyright Ericsson AB 1996-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/kernel/src/group.erl b/lib/kernel/src/group.erl index 376480ad0406..4fb7cee29e4c 100644 --- a/lib/kernel/src/group.erl +++ b/lib/kernel/src/group.erl @@ -497,9 +497,7 @@ get_chars_loop(Pbs, M, F, Xa, Drv, Shell, Buf0, State, LineCont0, Encoding) -> true -> get_line(Buf0, Pbs, LineCont0, Drv, Shell, Encoding); false -> - %% get_line_echo_off only deals with lists, - %% so convert to list before calling it. - get_line_echo_off(cast(Buf0, list), Encoding, Pbs, Drv, Shell) + get_line_echo_off(Buf0, Encoding, Pbs, Drv, Shell) end, case Result of {done,LineCont1,Buf} -> @@ -604,7 +602,7 @@ get_line1({open_editor, _Cs, Cont, Rs}, Drv, Shell, Ls0, Encoding) -> get_line1(edlin:edit_line(Cs1, NewCont), Drv, Shell, Ls0, Encoding) end; %% Move Up, Down in History: Ctrl+P, Ctrl+N -get_line1({history_up,Cs,Cont,Rs}, Drv, Shell, Ls0, Encoding) -> +get_line1({history_up,Cs,{_,_,_,Mode0}=Cont,Rs}, Drv, Shell, Ls0, Encoding) -> send_drv_reqs(Drv, Rs), case up_stack(save_line(Ls0, edlin:current_line(Cont))) of {none,_Ls} -> @@ -612,7 +610,8 @@ get_line1({history_up,Cs,Cont,Rs}, Drv, Shell, Ls0, Encoding) -> get_line1(edlin:edit_line(Cs, Cont), Drv, Shell, Ls0, Encoding); {Lcs,Ls} -> send_drv_reqs(Drv, edlin:erase_line()), - {more_chars,Ncont,Nrs} = edlin:start(edlin:prompt(Cont)), + {more_chars,{A,B,C,_},Nrs} = edlin:start(edlin:prompt(Cont)), + Ncont = {A,B,C,Mode0}, send_drv_reqs(Drv, Nrs), get_line1( edlin:edit_line1( @@ -621,7 +620,7 @@ get_line1({history_up,Cs,Cont,Rs}, Drv, Shell, Ls0, Encoding) -> Ncont), Drv, Shell, Ls, Encoding) end; -get_line1({history_down,Cs,Cont,Rs}, Drv, Shell, Ls0, Encoding) -> +get_line1({history_down,Cs,{_,_,_,Mode0}=Cont,Rs}, Drv, Shell, Ls0, Encoding) -> send_drv_reqs(Drv, Rs), case down_stack(save_line(Ls0, edlin:current_line(Cont))) of {none,_Ls} -> @@ -629,7 +628,8 @@ get_line1({history_down,Cs,Cont,Rs}, Drv, Shell, Ls0, Encoding) -> get_line1(edlin:edit_line(Cs, Cont), Drv, Shell, Ls0, Encoding); {Lcs,Ls} -> send_drv_reqs(Drv, edlin:erase_line()), - {more_chars,Ncont,Nrs} = edlin:start(edlin:prompt(Cont)), + {more_chars,{A,B,C,_},Nrs} = edlin:start(edlin:prompt(Cont)), + Ncont = {A,B,C,Mode0}, send_drv_reqs(Drv, Nrs), get_line1(edlin:edit_line1(string:to_graphemes(lists:sublist(Lcs, 1, @@ -656,7 +656,44 @@ get_line1({search,Cs,Cont,Rs}, Drv, Shell, Ls, Encoding) -> put(search_quit_prompt, Cont), Pbs = prompt_bytes("\033[;1;4msearch:\033[0m ", Encoding), {more_chars,Ncont,_Nrs} = edlin:start(Pbs, {search,none}), + put(search, new_search), get_line1(edlin:edit_line1(Cs, Ncont), Drv, Shell, Ls, Encoding); +get_line1({help, Before, Cs0, Cont, Rs}, Drv, Shell, Ls0, Encoding) -> + send_drv_reqs(Drv, Rs), + {_,Word,_} = edlin:over_word(Before, [], 0), + {R,Docs} = case edlin_context:get_context(Before) of + {function, Mod} when Word =/= [] -> try + {ok, [{atom,_,Module}], _} = erl_scan:string(Mod), + {ok, [{atom,_,Word1}], _} = erl_scan:string(Word), + {function, c:h1(Module, Word1)} + catch _:_ -> + {ok, [{atom,_,Module1}], _} = erl_scan:string(Mod), + {module, c:h1(Module1)} + end; + {function, Mod} -> + {ok, [{atom,_,Module}], _} = erl_scan:string(Mod), + {module, c:h1(Module)}; + {function, Mod, Fun, _Args, _Unfinished, _Nesting} -> + {ok, [{atom,_,Module}], _} = erl_scan:string(Mod), + {ok, [{atom,_,Function}], _} = erl_scan:string(Fun), + {function, c:h1(Module, Function)}; + {term, _, {atom, Word1}}-> + {ok, [{atom,_,Module}], _} = erl_scan:string(Word1), + {module, c:h1(Module)}; + _ -> {error, {error, no_help}} + end, + case {R, Docs} of + {_, {error, _}} -> send_drv(Drv, beep); + {module, _} -> + Docs1 = " "++string:trim(lists:nthtail(3, Docs),both), + send_drv(Drv, {put_expand, unicode, + [unicode:characters_to_binary(Docs1)], 7}); + {function, _} -> + Docs1 = " "++string:trim(Docs,both), + send_drv(Drv, {put_expand, unicode, + [unicode:characters_to_binary(Docs1)], 7}) + end, + get_line1(edlin:edit_line(Cs0, Cont), Drv, Shell, Ls0, Encoding); get_line1({Expand, Before, Cs0, Cont,Rs}, Drv, Shell, Ls0, Encoding) when Expand =:= expand; Expand =:= expand_full -> send_drv_reqs(Drv, Rs), @@ -678,34 +715,14 @@ get_line1({Expand, Before, Cs0, Cont,Rs}, Drv, Shell, Ls0, Encoding) {Cs1, _} when Cs1 =/= [] -> Cs1; _ -> NlMatchStr = unicode:characters_to_binary("\n"++MatchStr), + NLines = case Expand of + expand -> 7; + expand_full -> 0 + end, case get(expand_below) of true -> - Lines = string:split(string:trim(MatchStr), "\n", all), - NoLines = length(Lines), - if NoLines > 5, Expand =:= expand -> - %% Only show 5 lines to start with - [L1,L2,L3,L4,L5|_] = Lines, - String = lists:join( - $\n, - [L1,L2,L3,L4,L5, - io_lib:format("Press tab to see all ~p expansions", - [edlin_expand:number_matches(Matches)])]), - send_drv(Drv, {put_expand, unicode, - unicode:characters_to_binary(String)}), - Cs1; - true -> - case get_tty_geometry(Drv) of - {_, Rows} when Rows > NoLines -> - %% If all lines fit on screen, we expand below - send_drv(Drv, {put_expand, unicode, NlMatchStr}), - Cs1; - _ -> - %% If there are more results than fit on - %% screen we expand above - send_drv_reqs(Drv, [{put_chars, unicode, NlMatchStr}]), - [$\e, $l | Cs1] - end - end; + send_drv(Drv, {put_expand, unicode, unicode:characters_to_binary(string:trim(MatchStr, trailing)), NLines}), + Cs1; false -> send_drv(Drv, {put_chars, unicode, NlMatchStr}), [$\e, $l | Cs1] @@ -753,42 +770,49 @@ get_line1({search_cancel,_Cs,_,Rs}, Drv, Shell, Ls, Encoding) -> send_drv_reqs(Drv, edlin:redraw_line(NCont)), get_line1({more_chars, NCont, []}, Drv, Shell, Ls, Encoding); %% Search mode is entered. -get_line1({What,{line,Prompt,{_,{RevCmd0,_},_},{search, none}},_Rs}, +get_line1({What,{line,Prompt,{_,{RevCmd0,_},_},{search, none}}=Cont0,Rs}, Drv, Shell, Ls0, Encoding) -> %% Figure out search direction. ^S and ^R are returned through edlin %% whenever we received a search while being already in search mode. + OldSearch = get(search), {Search, Ls1, RevCmd} = case RevCmd0 of [$\^S|RevCmd1] -> {fun search_down_stack/2, Ls0, RevCmd1}; [$\^R|RevCmd1] -> {fun search_up_stack/2, Ls0, RevCmd1}; - _ -> % new search, rewind stack for a proper search. - {fun search_up_stack/2, new_stack(get_lines(Ls0)), RevCmd0} + _ when RevCmd0 =/= OldSearch -> % new search, rewind stack for a proper search. + {fun search_up_stack/2, new_stack(get_lines(Ls0)), RevCmd0}; + _ -> + {skip, Ls0, RevCmd0} end, + put(search, RevCmd), Cmd = lists:reverse(RevCmd), - {Ls, NewStack} = case Search(Ls1, Cmd) of - {none, Ls2} -> - send_drv(Drv, beep), - put(search_result, []), - send_drv(Drv, delete_line), - send_drv(Drv, {insert_chars, unicode, unicode:characters_to_binary(Prompt++Cmd)}), - {Ls2, {[],{RevCmd, []},[]}}; - {Line, Ls2} -> % found. Complete the output edlin couldn't have done. - Lines = string:split(string:to_graphemes(Line), "\n", all), - Output = if length(Lines) > 5 -> - [A,B,C,D,E|_]=Lines, - (["\n " ++ Line1 || Line1 <- [A,B,C,D,E]] ++ - [io_lib:format("~n ... (~w lines omitted)",[length(Lines)-5])]); - true -> ["\n " ++ Line1 || Line1 <- Lines] - end, - put(search_result, Lines), - send_drv(Drv, delete_line), - send_drv(Drv, {insert_chars, unicode, unicode:characters_to_binary(Prompt++Cmd)}), - send_drv(Drv, {put_expand_no_trim, unicode, unicode:characters_to_binary(Output)}), - {Ls2, {[],{RevCmd, []},[]}} - end, - Cont = {line,Prompt,NewStack,{search, none}}, - more_data(What, Cont, Drv, Shell, Ls, Encoding); + if Search =:= skip -> + %% Move expand are the only valid requests to bypass search mode + %% Sending delete_chars, insert_chars, etc. will result in + %% expand area being cleared. + Rs1 = [R||{move_expand,_}=R<-Rs], + send_drv_reqs(Drv, Rs1), + more_data(What, Cont0, Drv, Shell, Ls0, Encoding); + true -> + {Ls, NewStack} = case Search(Ls1, Cmd) of + {none, Ls2} -> + send_drv(Drv, beep), + put(search_result, []), + send_drv(Drv, delete_line), + send_drv(Drv, {insert_chars, unicode, unicode:characters_to_binary(Prompt++Cmd)}), + {Ls2, {[],{RevCmd, []},[]}}; + {Line, Ls2} -> % found. Complete the output edlin couldn't have done. + Lines = string:split(string:to_graphemes(Line), "\n", all), + put(search_result, Lines), + send_drv(Drv, delete_line), + send_drv(Drv, {insert_chars, unicode, unicode:characters_to_binary(Prompt++Cmd)}), + send_drv(Drv, {put_expand, unicode, unicode:characters_to_binary(" "++lists:join("\n ",Lines)), 7}), + {Ls2, {[],{RevCmd, []},[]}} + end, + Cont = {line,Prompt,NewStack,{search, none}}, + more_data(What, Cont, Drv, Shell, Ls, Encoding) + end; get_line1({What,Cont0,Rs}, Drv, Shell, Ls, Encoding) -> send_drv_reqs(Drv, Rs), more_data(What, Cont0, Drv, Shell, Ls, Encoding). @@ -1076,6 +1100,8 @@ cast(eof, _, _) -> eof; cast(L, binary, ToEnc) -> unicode:characters_to_binary(L, utf8, ToEnc); +cast(L, list, _ToEnc) when is_list(L) -> + L; cast(L, list, _ToEnc) -> unicode:characters_to_list(L, utf8). @@ -1083,8 +1109,15 @@ append(eof, [], _) -> eof; append(eof, L, _) -> L; +append(L, [], _) when is_list(L) -> + %% When doing ++ all of L needs to be traversed to check if it is + %% a proper list. Since we know it is a proper list we just return + %% the list without checking. + L; +append(L, A, _) when is_list(L) -> + L ++ A; %% We know L is valid unicode, so we just append the two append(B, L, FromEnc) -> - unicode:characters_to_list(B, FromEnc) ++ L. + append(unicode:characters_to_list(B, FromEnc), L, FromEnc). check_encoding(eof, _) -> true; @@ -1099,4 +1132,3 @@ is_latin1([]) -> true; is_latin1(_) -> false. - diff --git a/lib/kernel/src/kernel.appup.src b/lib/kernel/src/kernel.appup.src index 59b095a764a4..08779b185b13 100644 --- a/lib/kernel/src/kernel.appup.src +++ b/lib/kernel/src/kernel.appup.src @@ -56,7 +56,9 @@ {<<"^9\\.0$">>,[restart_new_emulator]}, {<<"^9\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^9\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, - {<<"^9\\.0\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}], + {<<"^9\\.0\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, + {<<"^9\\.1$">>,[restart_new_emulator]}, + {<<"^9\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}], [{<<"^8\\.0$">>,[restart_new_emulator]}, {<<"^8\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^8\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, @@ -85,4 +87,6 @@ {<<"^9\\.0$">>,[restart_new_emulator]}, {<<"^9\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^9\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, - {<<"^9\\.0\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}]}. + {<<"^9\\.0\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, + {<<"^9\\.1$">>,[restart_new_emulator]}, + {<<"^9\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}]}. diff --git a/lib/kernel/src/prim_tty.erl b/lib/kernel/src/prim_tty.erl index 4197b00dffab..480b399d4be5 100644 --- a/lib/kernel/src/prim_tty.erl +++ b/lib/kernel/src/prim_tty.erl @@ -106,7 +106,7 @@ -export([init/1, reinit/2, isatty/1, handles/1, unicode/1, unicode/2, handle_signal/2, window_size/1, handle_request/2, write/2, write/3, npwcwidth/1, - ansi_regexp/0]). + ansi_regexp/0, ansi_color/2]). -export([reader_stop/1, disable_reader/1, enable_reader/1]). -nifs([isatty/1, tty_create/0, tty_init/3, tty_set/1, setlocale/1, @@ -143,6 +143,8 @@ buffer_before = [], %% Current line before cursor in reverse buffer_after = [], %% Current line after cursor not in reverse buffer_expand, %% Characters in expand buffer + buffer_expand_row = 1, + buffer_expand_limit = 0 :: non_neg_integer(), cols = 80, rows = 24, xn = false, @@ -170,8 +172,7 @@ -type request() :: {putc_raw, binary()} | {putc, unicode:unicode_binary()} | - {expand, unicode:unicode_binary()} | - {expand_with_trim, unicode:unicode_binary()} | + {expand, unicode:unicode_binary(), integer()} | {insert, unicode:unicode_binary()} | {insert_over, unicode:unicode_binary()} | {delete, integer()} | @@ -184,6 +185,7 @@ {move, integer()} | {move_line, integer()} | {move_combo, integer(), integer(), integer()} | + {move_expand, -32768..32767} | clear | beep. -type tty() :: reference(). @@ -194,6 +196,32 @@ ansi_regexp() -> ?ANSI_REGEXP. +ansi_fg_color(Color) -> + case Color of + black -> 30; + red -> 31; + green -> 32; + yellow -> 33; + blue -> 34; + magenta -> 35; + cyan -> 36; + white -> 37; + bright_black -> 90; + bright_red -> 91; + bright_green -> 92; + bright_yellow -> 93; + bright_blue -> 94; + bright_magenta -> 95; + bright_cyan -> 96; + bright_white -> 97 + end. +ansi_bg_color(Color) -> + ansi_fg_color(Color) + 10. + +-spec ansi_color(BgColor :: atom(), FgColor :: atom()) -> iolist(). +ansi_color(BgColor, FgColor) -> + io_lib:format("\e[~w;~wm", [ansi_bg_color(BgColor), ansi_fg_color(FgColor)]). + -spec on_load() -> ok. on_load() -> on_load(#{}). @@ -597,7 +625,7 @@ handle_request(State, redraw_prompt) -> {ClearLine, _} = handle_request(State, delete_line), {Redraw, NewState} = handle_request(State, redraw_prompt_pre_deleted), {[ClearLine, Redraw], NewState}; -handle_request(State = #state{unicode = U, cols = W}, redraw_prompt_pre_deleted) -> +handle_request(State = #state{unicode = U, cols = W, rows = R}, redraw_prompt_pre_deleted) -> {Movement, TextInView, EverythingFitsInView} = in_view(State), {_, NewPrompt} = handle_request(State, new_prompt), {Redraw, RedrawState} = insert_buf(NewPrompt, unicode:characters_to_binary(TextInView)), @@ -611,17 +639,65 @@ handle_request(State = #state{unicode = U, cols = W}, redraw_prompt_pre_deleted) true when Last =/= [] -> cols(Last, U); _ -> cols(State#state.buffer_before, U) + cols(State#state.buffer_after,U) end, - {ExpandBuffer, NewState} = insert_buf(RedrawState#state{ buffer_expand = [] }, iolist_to_binary(BufferExpand)), + ERow = State#state.buffer_expand_row, + BufferExpandLines = string:split(BufferExpand, "\n", all), + InputRows = (cols_multiline([State#state.buffer_before ++ State#state.buffer_after], W, U) div W), + ExpandRows = (cols_multiline(BufferExpandLines, W, U) div W), + ExpandRowsLimit = case State#state.buffer_expand_limit of + 0 -> + ExpandRows; + Limit -> + min(Limit, ExpandRows) + end, + ExpandRowsLimit1 = min(ExpandRowsLimit, R-1-InputRows), + BufferExpand1 = case ExpandRows > ExpandRowsLimit1 of + true -> + Color = lists:flatten(ansi_color(cyan, bright_white)), + StatusLine = io_lib:format(Color ++"\e[1m" ++ "rows ~w to ~w of ~w" ++ "\e[0m", + [ERow, (ERow-1) + ExpandRowsLimit1, ExpandRows]), + Cols1 = max(0,W*ExpandRowsLimit1), + Cols0 = max(0,W*(ERow-1)), + {_, _, BufferExpandLinesInViewStart, {_, BEStartIVHalf}} = split_cols_multiline(Cols0, BufferExpandLines, U, W), + {_, BufferExpandLinesInViewRev, _, {BEIVHalf, _}} = split_cols_multiline(Cols1, BufferExpandLinesInViewStart++[BEStartIVHalf], U, W), + BEIVHalf1 = case BEIVHalf of [] -> []; + _ -> [BEIVHalf] + end, + ExpandInView = lists:reverse(BEIVHalf1++BufferExpandLinesInViewRev), + ["\r\n",lists:join("\n", ExpandInView ++ [StatusLine])]; + false -> + ["\r\n",BufferExpand] + end, + {ExpandBuffer, NewState} = insert_buf(RedrawState#state{ buffer_expand = [] }, unicode:characters_to_binary(BufferExpand1)), BECols = cols(W, End, NewState#state.buffer_expand, U), MoveToEnd = move_cursor(RedrawState, BECols, End), {[encode(Redraw,U),encode(ExpandBuffer, U), MoveToEnd, Movement], RedrawState} end, {Output, State}; +handle_request(State = #state{ buffer_expand = Expand, buffer_expand_row = ERow, cols = W, rows = R, unicode = U}, {move_expand, N}) -> + %% Get number of Lines in terminal window + BufferExpandLines = case Expand of + undefined -> []; + _ -> string:split(Expand, "\n", all) + end, + ExpandRows = (cols_multiline(BufferExpandLines, W, U) div W), + InputRows = (cols_multiline([State#state.buffer_before ++ State#state.buffer_after], W, U) div W), + ExpandRowsLimit = case State#state.buffer_expand_limit of + 0 -> + ExpandRows; + Limit -> + min(Limit, ExpandRows) + end, + ExpandRowsLimit1 = min(ExpandRowsLimit, R-1-InputRows), + ERow1 = if ExpandRows > ExpandRowsLimit1 -> %% We need to page expand rows + min(ExpandRows-ExpandRowsLimit1+1,max(1,ERow + N)); + true -> 1 %% No need to page expand rows + end, + handle_request(State#state{buffer_expand_row = ERow1}, redraw_prompt); %% Clear the expand buffer after the cursor when we handle any request. handle_request(State = #state{ buffer_expand = Expand, unicode = U}, Request) when Expand =/= undefined -> - {Redraw, NoExpandState} = handle_request(State#state{ buffer_expand = undefined }, redraw_prompt), - {Output, NewState} = handle_request(NoExpandState#state{ buffer_expand = undefined }, Request), + {Redraw, NoExpandState} = handle_request(State#state{ buffer_expand = undefined, buffer_expand_row = 1 }, redraw_prompt), + {Output, NewState} = handle_request(NoExpandState#state{ buffer_expand = undefined, buffer_expand_row = 1 }, Request), {[encode(Redraw, U), encode(Output, U)], NewState}; handle_request(State, new_prompt) -> {"", State#state{buffer_before = [], @@ -629,11 +705,9 @@ handle_request(State, new_prompt) -> lines_before = [], lines_after = []}}; %% Print characters in the expandbuffer after the cursor -handle_request(State, {expand, Expand}) -> - handle_request(State#state{buffer_expand = Expand}, redraw_prompt); -handle_request(State, {expand_with_trim, Binary}) -> - handle_request(State, - {expand, iolist_to_binary(["\r\n",string:trim(Binary, both)])}); +handle_request(State, {expand, Expand, N}) -> + {_, NewState} = insert_buf(State#state{buffer_expand = []}, Expand), + handle_request(NewState#state{buffer_expand_limit = N}, redraw_prompt); %% putc prints Binary and overwrites any existing characters handle_request(State = #state{ unicode = U }, {putc, Binary}) -> %% Todo should handle invalid unicode? @@ -910,18 +984,23 @@ move(left, #state{ left = Left }, N) -> move(right, #state{ right = Right }, N) -> lists:duplicate(N, Right). -in_view(#state{lines_after = LinesAfter, buffer_before = Bef, buffer_after = Aft, lines_before = LinesBefore, rows=R, cols=W, unicode=U, buffer_expand = BufferExpand} = State) -> +in_view(#state{lines_after = LinesAfter, buffer_before = Bef, buffer_after = Aft, lines_before = LinesBefore, + rows=R, cols=W, unicode=U, buffer_expand = BufferExpand, buffer_expand_limit = BufferExpandLimit} = State) -> BufferExpandLines = case BufferExpand of undefined -> []; - _ -> string:split(erlang:binary_to_list(BufferExpand), "\r\n", all) + _ -> string:split(unicode:characters_to_list(BufferExpand), "\r\n", all) end, - ExpandRows = (cols_multiline(BufferExpandLines, W, U) div W), + ExpandLimit = case BufferExpandLimit of + 0 -> cols_multiline(BufferExpandLines, W, U) div W; + _ -> min(cols_multiline(BufferExpandLines, W, U) div W, BufferExpandLimit) + end, + ExpandRows = ExpandLimit, InputBeforeRows = (cols_multiline(LinesBefore, W, U) div W), InputRows = (cols_multiline([Bef ++ Aft], W, U) div W), InputAfterRows = (cols_multiline(LinesAfter, W, U) div W), %% Dont print lines after if we have expansion rows SumRows = InputBeforeRows+ InputRows + ExpandRows + InputAfterRows, - if SumRows > R -> + if SumRows > R -> RowsLeftAfterInputRows = R - InputRows, RowsLeftAfterExpandRows = RowsLeftAfterInputRows - ExpandRows, RowsLeftAfterInputBeforeRows = RowsLeftAfterExpandRows - InputBeforeRows, @@ -1005,7 +1084,7 @@ update_geometry(State) -> case tty_window_size(State#state.tty) of {ok, {Cols, Rows}} when Cols > 0 -> ?dbg({?FUNCTION_NAME, Cols}), - State#state{ cols = Cols, rows = Rows }; + State#state{ cols = Cols, rows = Rows}; _Error -> ?dbg({?FUNCTION_NAME, _Error}), State diff --git a/lib/kernel/src/user_drv.erl b/lib/kernel/src/user_drv.erl index 5017235d3f5a..0e48e3f5db19 100644 --- a/lib/kernel/src/user_drv.erl +++ b/lib/kernel/src/user_drv.erl @@ -52,8 +52,8 @@ %% guaranteed to have been written to the terminal {put_chars_sync, unicode, binary(), {From :: pid(), Reply :: term()}} | %% Put text in expansion area - {put_expand, unicode, binary()} | - {put_expand_no_trim, unicode, binary()} | + {put_expand, unicode, binary(), integer()} | + {move_expand, -32768..32767} | %% Move the cursor X characters left or right (negative is left) {move_rel, -32768..32767} | %% Move the cursor Y rows up or down (negative is up) @@ -789,10 +789,10 @@ io_request({put_chars_sync, unicode, Chars, Reply}, TTY) -> {Output, NewTTY} = prim_tty:handle_request(TTY, {putc, unicode:characters_to_binary(Chars)}), {ok, MonitorRef} = prim_tty:write(NewTTY, Output, self()), {Reply, MonitorRef, NewTTY}; -io_request({put_expand, unicode, Chars}, TTY) -> - write(prim_tty:handle_request(TTY, {expand_with_trim, unicode:characters_to_binary(Chars)})); -io_request({put_expand_no_trim, unicode, Chars}, TTY) -> - write(prim_tty:handle_request(TTY, {expand, unicode:characters_to_binary(Chars)})); +io_request({put_expand, unicode, Chars, N}, TTY) -> + write(prim_tty:handle_request(TTY, {expand, unicode:characters_to_binary(Chars), N})); +io_request({move_expand, N}, TTY) -> + write(prim_tty:handle_request(TTY, {move_expand, N})); io_request({move_rel, N}, TTY) -> write(prim_tty:handle_request(TTY, {move, N})); io_request({move_line, R}, TTY) -> diff --git a/lib/kernel/test/Makefile b/lib/kernel/test/Makefile index c00ae839d5a5..344850a63940 100644 --- a/lib/kernel/test/Makefile +++ b/lib/kernel/test/Makefile @@ -28,6 +28,7 @@ SOCKET_MODULES = \ socket_test_lib \ socket_test_logger \ socket_test_evaluator \ + socket_test_profile \ socket_test_ttest_lib \ socket_test_ttest_tcp_gen \ socket_test_ttest_tcp_gs \ diff --git a/lib/kernel/test/code_SUITE.erl b/lib/kernel/test/code_SUITE.erl index 82ddfbf97905..3284b2a8fc81 100644 --- a/lib/kernel/test/code_SUITE.erl +++ b/lib/kernel/test/code_SUITE.erl @@ -25,7 +25,8 @@ -export([all/0, suite/0,groups/0,init_per_group/2,end_per_group/2]). -export([set_path/1, get_path/1, add_path/1, add_paths/1, del_path/1, replace_path/1, load_file/1, load_abs/1, - ensure_loaded/1, ensure_loaded_many/1, ensure_loaded_many_kill/1, + ensure_loaded/1, ensure_loaded_many/1, ensure_loaded_many_kill/1, + ensure_loaded_bad_case/1, delete/1, purge/1, purge_many_exits/0, purge_many_exits/1, soft_purge/1, is_loaded/1, all_loaded/1, all_available/1, load_binary/1, dir_req/1, object_code/1, set_path_file/1, @@ -38,7 +39,7 @@ on_load_embedded/1, on_load_errors/1, on_load_update/1, on_load_trace_on_load/1, on_load_purge/1, on_load_self_call/1, on_load_pending/1, - on_load_deleted/1, + on_load_deleted/1, on_load_deadlock/1, big_boot_embedded/1, module_status/1, get_mode/1, code_path_cache/1, @@ -62,6 +63,7 @@ all() -> [set_path, get_path, add_path, add_paths, del_path, replace_path, load_file, load_abs, ensure_loaded, ensure_loaded_many, ensure_loaded_many_kill, + ensure_loaded_bad_case, delete, purge, purge_many_exits, soft_purge, is_loaded, all_loaded, all_available, load_binary, dir_req, object_code, set_path_file, upgrade, code_path_cache, @@ -72,7 +74,7 @@ all() -> on_load_binary, on_load_embedded, on_load_errors, {group, sequence}, on_load_purge, on_load_self_call, on_load_pending, - on_load_deleted, + on_load_deleted, on_load_deadlock, module_status, big_boot_embedded, get_mode, normalized_paths, mult_embedded_flags]. @@ -416,6 +418,13 @@ load_abs(Config) when is_list(Config) -> code:unstick_dir(TestDir), ok. +ensure_loaded_bad_case(Config) when is_list(Config) -> + %% Make sure loading an invalid .beam file does + %% not hold a permanent lock on the server. + {error, _} = code:ensure_loaded('STRING'), + {error, _} = code:ensure_loaded('STRING'), + ok. + ensure_loaded(Config) when is_list(Config) -> {module, lists} = code:ensure_loaded(lists), case init:get_argument(mode) of @@ -1901,6 +1910,31 @@ on_load_deleted(_Config) -> ok. +on_load_deadlock(Config) -> + Mod = ?FUNCTION_NAME, + register(Mod, self()), + Tree = ?Q(["-module('@Mod@').\n", + "-export([ext/0]).\n", + "-on_load(f/0).\n", + "f() ->\n", + " timer:sleep(1000),\n", + " '@Mod@':ext().\n", + "ext() -> ok.\n"]), + merl:print(Tree), + {ok,Mod,Code} = merl:compile(Tree), + + Dir = proplists:get_value(priv_dir, Config), + File = filename:join(Dir, atom_to_list(Mod) ++ ".beam"), + ok = file:write_file(File, Code), + code:add_path(Dir), + + spawn(fun() -> code:ensure_loaded(Mod) end), + {error,Reason} = code:ensure_loaded(Mod), + true = (Reason =:= deadlock) or (Reason =:= on_load_failure), + + code:del_path(Dir), + ok. + delete_before_reload(Mod, Reload) -> false = check_old_code(Mod), diff --git a/lib/kernel/test/esock_ttest/esock-ttest b/lib/kernel/test/esock_ttest/esock-ttest index 3126c34c55fd..01858072aedf 100755 --- a/lib/kernel/test/esock_ttest/esock-ttest +++ b/lib/kernel/test/esock_ttest/esock-ttest @@ -63,7 +63,7 @@ usage() -> "~n --async Asynchronous mode (Timeout = nowait)" "~n This option is only valid for transport = sock." "~n Also, its only used when active =/= false." - "~n --active boolean() | once." + "~n --active pos_integer() | boolean() | once." "~n Valid for both client and server." "~n Defaults to: false" "~n --transport Which transport to use: gen|sock[:plain|msg]" @@ -95,6 +95,7 @@ usage() -> "~n Only valid for client." "~n Mandatory." "~n Defaults to: 60 (seconds)" + "~n --profile Run profiling. " "~n" "~n" "~n", @@ -117,16 +118,20 @@ process_server_args(Args) -> domain => inet, async => false, active => false, - transport => {sock, plain}}, + transport => {sock, plain}, + profile => false}, process_server_args(Args, Defaults). process_server_args([], State) -> State; +process_server_args(["--profile"|Args], State) -> + process_server_args(Args, State#{profile => true}); + process_server_args(["--domain", Domain|Args], State) - when ((Domain =:= "local") orelse - (Domain =:= "inet") orelse - (Domain =:= "inet6")) -> + when (Domain =:= "local") orelse + (Domain =:= "inet") orelse + (Domain =:= "inet6") -> process_server_args(Args, State#{domain => list_to_atom(Domain)}); process_server_args(["--async"|Args], State) -> @@ -164,20 +169,38 @@ process_client_args(Args) -> msg_id => 1, %% Will be filled in based on msg_id if not provided max_outstanding => undefined, - runtime => ?SECS(60)}, + runtime => ?SECS(60), + profile => false}, process_client_args(Args, Defaults). process_client_args([], State) -> process_client_args_ensure_max_outstanding(State); +process_client_args(["--profile"|Args], State) -> + process_client_args(Args, State#{profile => true}); + process_client_args(["--async"|Args], State) -> process_client_args(Args, State#{async => true}); -process_client_args(["--active", Active|Args], State) - when (Active =:= "false") orelse - (Active =:= "once") orelse - (Active =:= "true") -> - process_client_args(Args, State#{active => list_to_atom(Active)}); +process_client_args(["--active", Active0|Args], State) -> + Active = + if + (Active0 =:= "false") orelse + (Active0 =:= "once") orelse + (Active0 =:= "true") -> + list_to_atom(Active0); + true -> + try list_to_integer(Active0) of + N when (N > 0) andalso (N =< 32767) -> + N; + _ -> + usage(f("Invalid Active: ~s", [Active0])) + catch + _:_:_ -> + usage(f("Invalid Active: ~s", [Active0])) + end + end, + process_client_args(Args, State#{active => Active}); process_client_args(["--transport", "gen" | Args], State) -> process_client_args(Args, State#{transport => gen}); @@ -280,53 +303,87 @@ process_client_args_ensure_max_outstanding( exec(#{role := server, domain := Domain, active := Active, - transport := gen}) + transport := gen, + profile := Profile}) when (Domain =:= inet) orelse (Domain =:= inet6) -> - case socket_test_ttest_tcp_server_gen:start(Domain, Active) of - {ok, {Pid, _}} -> - MRef = erlang:monitor(process, Pid), - receive - {'DOWN', MRef, process, Pid, Info} -> - Info - end; - {error, Reason} -> - eprint(f("Failed starting server: " - "~n ~p", [Reason])), - error + Fun = + fun() -> + case socket_test_ttest_tcp_server_gen:start(Domain, Active) of + {ok, {Pid, _}} -> + MRef = erlang:monitor(process, Pid), + receive + {'DOWN', MRef, process, Pid, Info} -> + Info + end; + {error, Reason} -> + eprint(f("Failed starting server: " + "~n ~p", [Reason])), + error + end + end, + case Profile of + true -> + socket_test_profile:profile( + f("server-gen-~w", [Active]), Fun); + false -> + Fun() end; exec(#{role := server, domain := Domain, active := Active, - transport := gs}) + transport := gs, + profile := Profile}) when (Domain =:= inet) orelse (Domain =:= inet6) -> - case socket_test_ttest_tcp_server_gs:start(Domain, Active) of - {ok, {Pid, _}} -> - MRef = erlang:monitor(process, Pid), - receive - {'DOWN', MRef, process, Pid, Info} -> + Fun = + fun() -> + case socket_test_ttest_tcp_server_gs:start(Domain, Active) of + {ok, {Pid, _}} -> + MRef = erlang:monitor(process, Pid), + receive + {'DOWN', MRef, process, Pid, Info} -> Info - end; - {error, Reason} -> - eprint(f("Failed starting server: " - "~n ~p", [Reason])), - error + end; + {error, Reason} -> + eprint(f("Failed starting server: " + "~n ~p", [Reason])), + error + end + end, + case Profile of + true -> + socket_test_profile:profile( + f("server-gs-~w", [Active]), Fun); + false -> + Fun() end; exec(#{role := server, domain := Domain, async := Async, active := Active, - transport := {sock, Method}}) -> - case socket_test_ttest_tcp_server_socket:start(Method, Domain, Async, Active) of - {ok, {Pid, _}} -> - MRef = erlang:monitor(process, Pid), - receive - {'DOWN', MRef, process, Pid, Info} -> - Info - end; - {error, Reason} -> - eprint(f("Failed starting server: " - "~n ~p", [Reason])), - error + transport := {sock, Method}, + profile := Profile}) -> + Fun = + fun() -> + case socket_test_ttest_tcp_server_socket:start(Method, Domain, + Async, Active) of + {ok, {Pid, _}} -> + MRef = erlang:monitor(process, Pid), + receive + {'DOWN', MRef, process, Pid, Info} -> + Info + end; + {error, Reason} -> + eprint(f("Failed starting server: " + "~n ~p", [Reason])), + error + end + end, + case Profile of + true -> + socket_test_profile:profile( + f("server-sock-~w-~w", [Method, Active]), Fun); + false -> + Fun() end; exec(#{role := client, @@ -338,22 +395,34 @@ exec(#{role := client, transport := gen, msg_id := MsgID, max_outstanding := MaxOutstanding, - runtime := RunTime}) -> - case socket_test_ttest_tcp_client_gen:start(true, - ServerInfo, - Active, - MsgID, MaxOutstanding, - RunTime) of - {ok, Pid} -> - MRef = erlang:monitor(process, Pid), - receive - {'DOWN', MRef, process, Pid, Info} -> - Info - end; - {error, Reason} -> - eprint(f("Failed starting server: " - "~n ~p", [Reason])), - error + runtime := RunTime, + profile := Profile}) -> + Fun = + fun() -> + case socket_test_ttest_tcp_client_gen:start(true, + ServerInfo, + Active, + MsgID, + MaxOutstanding, + RunTime) of + {ok, Pid} -> + MRef = erlang:monitor(process, Pid), + receive + {'DOWN', MRef, process, Pid, Info} -> + Info + end; + {error, Reason} -> + eprint(f("Failed starting server: " + "~n ~p", [Reason])), + error + end + end, + case Profile of + true -> + socket_test_profile:profile( + f("client-gen-~w", [Active]), Fun); + false -> + Fun() end; exec(#{role := client, server := {_Addr, _Port} = ServerInfo, @@ -361,22 +430,34 @@ exec(#{role := client, transport := gs, msg_id := MsgID, max_outstanding := MaxOutstanding, - runtime := RunTime}) -> - case socket_test_ttest_tcp_client_gs:start(true, - ServerInfo, - Active, - MsgID, MaxOutstanding, - RunTime) of - {ok, Pid} -> - MRef = erlang:monitor(process, Pid), - receive - {'DOWN', MRef, process, Pid, Info} -> - Info - end; - {error, Reason} -> - eprint(f("Failed starting server: " - "~n ~p", [Reason])), - error + runtime := RunTime, + profile := Profile}) -> + Fun = + fun() -> + case socket_test_ttest_tcp_client_gs:start(true, + ServerInfo, + Active, + MsgID, + MaxOutstanding, + RunTime) of + {ok, Pid} -> + MRef = erlang:monitor(process, Pid), + receive + {'DOWN', MRef, process, Pid, Info} -> + Info + end; + {error, Reason} -> + eprint(f("Failed starting server: " + "~n ~p", [Reason])), + error + end + end, + case Profile of + true -> + socket_test_profile:profile( + f("client-gs-~w", [Active]), Fun); + false -> + Fun() end; exec(#{role := client, server := ServerInfo, @@ -385,31 +466,42 @@ exec(#{role := client, transport := {sock, Method}, msg_id := MsgID, max_outstanding := MaxOutstanding, - runtime := RunTime}) -> - case socket_test_ttest_tcp_client_socket:start(true, - Async, - Active, - Method, - ServerInfo, - MsgID, MaxOutstanding, - RunTime) of - {ok, Pid} -> - MRef = erlang:monitor(process, Pid), - receive - {'DOWN', MRef, process, Pid, Info} -> - Info - end; - {error, Reason} -> - eprint(f("Failed starting server: " - "~n ~p", [Reason])), - error + runtime := RunTime, + profile := Profile}) -> + Fun = + fun() -> + case socket_test_ttest_tcp_client_socket:start(true, + Async, + Active, + Method, + ServerInfo, + MsgID, + MaxOutstanding, + RunTime) of + {ok, Pid} -> + MRef = erlang:monitor(process, Pid), + receive + {'DOWN', MRef, process, Pid, Info} -> + Info + end; + {error, Reason} -> + eprint(f("Failed starting server: " + "~n ~p", [Reason])), + error + end + end, + case Profile of + true -> + socket_test_profile:profile( + f("client-sock-~w-~w", [Method, Active]), Fun); + false -> + Fun() end; exec(_) -> usage("Unexpected option combo"), ok. - %% ========================================================================== f(F, A) -> diff --git a/lib/kernel/test/esock_ttest/esock-ttest-client b/lib/kernel/test/esock_ttest/esock-ttest-client index aff8e96fb1dc..e164f49771f3 100755 --- a/lib/kernel/test/esock_ttest/esock-ttest-client +++ b/lib/kernel/test/esock_ttest/esock-ttest-client @@ -26,7 +26,9 @@ # # Example: # Assuming your pwd is $ERL_TOP -# (cd lib/kernel/test && ./esock_ttest/esock-ttest-client 1 1.2.3.4 11111) +# (cd lib/kernel/test && ./esock_ttest/esock-ttest-client --msgid 1 --addr 1.2.3.4 --port 11111) + +program="`basename "$0"`" KERNEL=$ERL_TOP/lib/kernel KERNEL_TEST=$KERNEL/test @@ -35,41 +37,123 @@ ESOCK_TTEST=$KERNEL_TEST/esock_ttest RUNTIME=30 # RUNTIME=60 # RUNTIME=600 +# PROFILE="--profile" +MSGID=1 +ADDR=none +PORT=none +DOMAIN=inet + + +###################################################################### + +usage="\ +Usage: $program [options] + +This shell script is used to run the esock ttest client part. + +Options: + --help Display this help and exit. + --profile Perform a 'profile' run. + --msgid The type of message by the test. There are three: + 1: small + 2: medium + 3: large + Defaults to 1 + --runtime The length of the test in seconds. + Defaults to 30 + --addr
Address of the server. + Mandatory + --port Port number of the server. + Mandatory when +" + +###################################################################### + +now() { date '+%T'; } + +die () { + TIME=`now` + reason=$? + echo "ERROR [$TIME]: $@" >&2 + echo "$usage" + exit $reason +} + +while test $# != 0; do + # echo "arg $1" + case $1 in + --help) + echo "$usage" ; + exit 0 ;; + + --profile) + PROFILE="--profile" + shift ;; + + --msgid) + shift ; + MSGID=$1 + shift ;; + + --addr) + shift ; + ADDR=$1 + shift ;; + + --port) + shift ; + PORT=$1 + shift ;; + + --runtime) + shift ; + RUNTIME=$1 + shift ;; + + *) + die "Unexpected option $1"; + ;; + esac +done + + +if [ $ADDR = none ]; then + die "Mandatory option '--addr' missing"; +fi + +if [ $PORT = none ]; then + die "Mandatory option '--port' missing"; +fi + +SERVER_INFO=$ADDR:$PORT -if [ $# = 3 ]; then - MSGID=$1 - SERVER_INFO=$2:$3 - ITERATIONS="\ +# --------------------------------------------------------------------------- + +ITERATIONS="\ gen false $MSGID gen true $MSGID gen once $MSGID + gen 1 $MSGID + gen 5 $MSGID + gen 10 $MSGID + gen 20 $MSGID gs false $MSGID gs true $MSGID gs once $MSGID + gs 1 $MSGID + gs 5 $MSGID + gs 10 $MSGID + gs 20 $MSGID sock false $MSGID --async sock true $MSGID --async - sock once $MSGID --async" - -else - if [ $# = 2 ]; then - MSGID=$1 - SERVER_INFO=$2 - - ITERATIONS="\ - sock false $MSGID --async - sock true $MSGID --async - sock once $MSGID --async" - - else - echo "Invalid number of args" - exit 1; - fi -fi + sock once $MSGID --async + sock 1 $MSGID --async + sock 5 $MSGID --async + sock 10 $MSGID --async + sock 20 $MSGID --async" -# --------------------------------------------------------------------------- - # For when we have figured out how to configure local for gen_tcp... #ITERATIONS="\ @@ -90,10 +174,10 @@ echo "$ITERATIONS" | echo "" echo "=========== transport = $TRANSPORT, active = $ACTIVE, msg-id = $MSG_ID ===========" - # The /dev/null at the end is necessary because erlang "does things" with stdin - # and this case would cause the 'while read' to "fail" so that we only would - # loop one time - $ESOCK_TTEST/esock-ttest --client --transport $TRANSPORT $ASYNC --active $ACTIVE --msg-id $MSG_ID --scon $SERVER_INFO --runtime $RUNTIME &2 + echo "$usage" + exit $reason +} + +while test $# != 0; do + # echo "arg $1" + case $1 in + --help) + echo "$usage" ; + exit 0 ;; + + --active) + shift ; + TMP=$1 + if [ $TMP = once ]; then + ACTIVE=$TMP + elif [ $TMP = true ]; then + ACTIVE=$TMP + elif [ $TMP = false ]; then + ACTIVE=$TMP + else + die "Invalid 'active' value '$TMP'"; + fi + shift ;; + + --domain) + shift ; + TMP=$1 + if [ $TMP = inet ]; then + DOMAIN=$TMP + elif [ $TMP = inet6 ]; then + DOMAIN=$TMP + elif [ $TMP = inet6 ]; then + DOMAIN=$TMP + else + die "Invalid 'domain' value '$TMP'"; + fi + shift ;; + + *) + die "Unexpected option $1"; + ;; + esac +done + -$ESOCK_TTEST/esock-ttest --server --transport gen $ACTIVE +$ESOCK_TTEST/esock-ttest --server --domain $DOMAIN --transport gen --active $ACTIVE diff --git a/lib/kernel/test/esock_ttest/esock-ttest-server-gs b/lib/kernel/test/esock_ttest/esock-ttest-server-gs index 5f8fc283dcbd..376e38417c7a 100755 --- a/lib/kernel/test/esock_ttest/esock-ttest-server-gs +++ b/lib/kernel/test/esock_ttest/esock-ttest-server-gs @@ -20,13 +20,90 @@ # %CopyrightEnd% # +# Example: +# Assuming your pwd is $ERL_TOP +# (cd lib/kernel/test && ./esock_ttest/esock-ttest-server-gs [--active true --domain inet]) + +program="`basename "$0"`" + KERNEL=$ERL_TOP/lib/kernel KERNEL_TEST=$KERNEL/test ESOCK_TTEST=$KERNEL_TEST/esock_ttest -if [ $# = 1 ]; then - ACTIVE="--active $1" -fi +ACTIVE=true +DOMAIN=inet + +###################################################################### + +usage="\ +Usage: $program [options] + +This shell script is used to run the esock ttest server part with +'gs' (gen_tcp with 'socket' backend) as transport. + +Options: + --help Display this help and exit. + --active Active + once | boolean() + Defaults to true + --domain Domain of the transport + inet | inet6 | local + Defaults to 'inet'. +" + +###################################################################### + +now() { date '+%T'; } + +die () { + TIME=`now` + reason=$? + echo "ERROR [$TIME]: $@" >&2 + echo "$usage" + exit $reason +} + +while test $# != 0; do + # echo "arg $1" + case $1 in + --help) + echo "$usage" ; + exit 0 ;; + + --active) + shift ; + TMP=$1 + if [ $TMP = once ]; then + ACTIVE=$TMP + elif [ $TMP = true ]; then + ACTIVE=$TMP + elif [ $TMP = false ]; then + ACTIVE=$TMP + else + die "Invalid 'active' value '$TMP'"; + fi + shift ;; + + --domain) + shift ; + TMP=$1 + if [ $TMP = inet ]; then + DOMAIN=$TMP + elif [ $TMP = inet6 ]; then + DOMAIN=$TMP + elif [ $TMP = inet6 ]; then + DOMAIN=$TMP + else + die "Invalid 'domain' value '$TMP'"; + fi + shift ;; + + *) + die "Unexpected option $1"; + ;; + esac +done + -$ESOCK_TTEST/esock-ttest --server --transport gs $ACTIVE +$ESOCK_TTEST/esock-ttest --server --domain $DOMAIN --transport gs --active $ACTIVE diff --git a/lib/kernel/test/esock_ttest/esock-ttest-server-sock b/lib/kernel/test/esock_ttest/esock-ttest-server-sock index d21b812103e3..d08b226ba6eb 100755 --- a/lib/kernel/test/esock_ttest/esock-ttest-server-sock +++ b/lib/kernel/test/esock_ttest/esock-ttest-server-sock @@ -20,37 +20,98 @@ # %CopyrightEnd% # +# Example: +# Assuming your pwd is $ERL_TOP +# (cd lib/kernel/test && ./esock_ttest/esock-ttest-server-sock [--active true --domain inet]) + +program="`basename "$0"`" + KERNEL=$ERL_TOP/lib/kernel KERNEL_TEST=$KERNEL/test ESOCK_TTEST=$KERNEL_TEST/esock_ttest -# $1 - async - boolean() -# $2 - active - once | boolean() -# [$3 - domain - inet (default) | inet6 | local] -if [ $# -ge 2 ]; then +ASYNC= +ACTIVE=true +DOMAIN=inet +# PROFILE="--profile" + + +###################################################################### + +usage="\ +Usage: $program [options] + +This shell script is used to run the esock ttest server part with +'socket' as transport. + +Options: + --help Display this help and exit. + --async Asynchronous. + --active Active + once | boolean() + Defaults to true + --domain Domain of the transport + inet | inet6 | local + Defaults to 'inet'. +" + +###################################################################### + +now() { date '+%T'; } - async=$1 - active=$2 +die () { + TIME=`now` + reason=$? + echo "ERROR [$TIME]: $@" >&2 + echo "$usage" + exit $reason +} - if [ $async = true ]; then - ASYNC="--async" - else - ASYNC= - fi +while test $# != 0; do + # echo "arg $1" + case $1 in + --help) + echo "$usage" ; + exit 0 ;; - ACTIVE="--active $active" + --async) + ASYNC="--async" + shift ;; - if [ $# = 3 ]; then - DOMAIN="--domain $3" - fi + --active) + shift ; + TMP=$1 + if [ $TMP = once ]; then + ACTIVE=$TMP + elif [ $TMP = true ]; then + ACTIVE=$TMP + elif [ $TMP = false ]; then + ACTIVE=$TMP + else + die "Invalid 'active' value '$TMP'"; + fi + shift ;; + --domain) + shift ; + TMP=$1 + if [ $TMP = inet ]; then + DOMAIN=$TMP + elif [ $TMP = inet6 ]; then + DOMAIN=$TMP + elif [ $TMP = inet6 ]; then + DOMAIN=$TMP + else + die "Invalid 'domain' value '$TMP'"; + fi + shift ;; -else - echo " Missing args: async and active" - echo "" - exit 1 -fi + *) + die "Unexpected option $1"; + ;; + esac +done -$ESOCK_TTEST/esock-ttest --server $DOMAIN $ASYNC --transport sock $ACTIVE +$ESOCK_TTEST/esock-ttest --server --domain $DOMAIN $ASYNC --transport sock --active $ACTIVE diff --git a/lib/kernel/test/interactive_shell_SUITE.erl b/lib/kernel/test/interactive_shell_SUITE.erl index b15216317e4c..d6c688f1dffa 100644 --- a/lib/kernel/test/interactive_shell_SUITE.erl +++ b/lib/kernel/test/interactive_shell_SUITE.erl @@ -1128,20 +1128,48 @@ shell_expand_location_below(Config) -> {module, long_module} = code:load_binary(long_module, "long_module.beam", Bin) end), check_content(Term, "3>"), + tmux(["resize-window -t ",tty_name(Term)," -y 50"]), + timer:sleep(1000), %% Sleep to make sure window has resized + Result = 61, + Rows1 = 48, send_stdin(Term, "long_module:" ++ FunctionName), send_stdin(Term, "\t"), + check_content(Term, "3> long_module:" ++ FunctionName ++ "\nfunctions(\n|.)*a_long_function_name0\\("), + %% Check that correct text is printed below expansion - check_content(Term, io_lib:format("Press tab to see all ~p expansions", - [length(NumFunctions)])), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [1, 7, Result])), + send_stdin(Term, "\t"), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [1, Rows1, Result])), + send_tty(Term, "Down"), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [2, Rows1+1, Result])), + send_tty(Term, "PgDn"), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [7, Rows1+6, Result])), + send_tty(Term, "PgUp"), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [2, Rows1+1, Result])), + send_tty(Term, "PgUp"), + %% Overshoot up + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [1, Rows1, Result])), + %% Overshoot down + send_tty(Term, "PgDn"), + send_tty(Term, "PgDn"), + send_tty(Term, "PgDn"), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [14, Rows1+13, Result])), + check_content(Term, "\n.*a_long_function_name99\\("), + send_tty(Term, "Up"), + check_content(Term, io_lib:format("rows ~w to ~w of ~w", + [13, Rows1+12, Result])), + send_stdin(Term, "\t"), - %% The expansion does not fit on screen, verify that - %% expand above mode is used - check_content(fun() -> get_content(Term, "-S -5") end, - "\nfunctions\n"), - check_content(Term, "3> long_module:" ++ FunctionName ++ "$"), %% We resize the terminal to make everything fit and test that - %% expand below mode is used + %% expand below displays everything tmux(["resize-window -t ", tty_name(Term), " -y ", integer_to_list(Rows+10)]), timer:sleep(1000), %% Sleep to make sure window has resized send_stdin(Term, "\t\t"), @@ -1174,6 +1202,20 @@ shell_expand_location_above(Config) -> ok end. +shell_help(Config) -> + Term = start_tty(Config), + try + send_stdin(Term, "lists"), + send_stdin(Term, "\^[h"), + check_content(Term, "List processing functions."), + send_stdin(Term, ":all"), + send_stdin(Term, "\^[h"), + check_content(Term, "-spec all(Pred, List) -> boolean()"), + ok + after + stop_tty(Term), + ok + end. %% Test the we can handle invalid ansi escape chars. %% tmux cannot handle this... so we test this using to_erl shell_invalid_ansi(_Config) -> diff --git a/lib/kernel/test/prim_file_SUITE.erl b/lib/kernel/test/prim_file_SUITE.erl index 42d1b263ad18..aa96c334b938 100644 --- a/lib/kernel/test/prim_file_SUITE.erl +++ b/lib/kernel/test/prim_file_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2000-2022. All Rights Reserved. +%% Copyright Ericsson AB 2000-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/kernel/test/socket_test_profile.erl b/lib/kernel/test/socket_test_profile.erl new file mode 100644 index 000000000000..c6d343b1ed33 --- /dev/null +++ b/lib/kernel/test/socket_test_profile.erl @@ -0,0 +1,328 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2023-2023. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +%% +%%---------------------------------------------------------------------- +%% socket_test_profile: Utility module used for socket profiling +%%---------------------------------------------------------------------- + +-module(socket_test_profile). + +-include("kernel_test_lib.hrl"). + +-export([profile/2, prepare/2, analyse/2, + fprof_to_calltree/2, fprof_to_calltree/3]). + + +-define(BASE_FILENAME, "socket_test_profile_"). + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +%% Execute Fun and profile it with fprof. +profile(Slogan, Fun) when is_function(Fun, 0) -> + Pids = [self()], + {ok, TraceFile, DestFile, CallTreeFile} = prepare(Slogan, Pids), + Res = (catch Fun()), + ok = analyse(TraceFile, DestFile), + ok = file:delete(TraceFile), + ok = fprof_to_calltree(DestFile, CallTreeFile), + Res. + +fprof_trace_file(Slogan, FTS) -> + fprof_file(Slogan, FTS, "-fprof.trace"). + +fprof_dest_file(Slogan, FTS) -> + fprof_file(Slogan, FTS, ".fprof"). + +fprof_calltree_file(Slogan, FTS) -> + fprof_file(Slogan, FTS, ".calltree"). + +fprof_file(Slogan, FTS, Post) -> + lists:concat([?BASE_FILENAME, Slogan, "-", FTS, Post]). + + +%% Prepare for tracing +prepare(Slogan, Pids) -> + FTS = ?FTS(), + TraceFile = fprof_trace_file(Slogan, FTS), + DestFile = fprof_dest_file(Slogan, FTS), + CallTreeFile = fprof_calltree_file(Slogan, FTS), + {ok, _Pid} = fprof:start(), + erlang:garbage_collect(), + TraceOpts = [ + start, + {cpu_time, false}, + {procs, Pids}, + {file, TraceFile} + ], + ok = fprof:trace(TraceOpts), + {ok, TraceFile, DestFile, CallTreeFile}. + +%% Stop tracing and analyse it +analyse(TraceFile, DestFile) -> + fprof:trace(stop), + try + case fprof:profile([{file, TraceFile}]) of + ok -> + ok = fprof:analyse([{dest, DestFile}, {totals, false}]), + ok; + {error, Reason} -> + {error, Reason} + end + after + fprof:stop() + end. + +fprof_to_calltree(DestFile, CallTreeFile) -> + fprof_to_calltree(DestFile, CallTreeFile, 0). + +%% Create a calltree from an fprof file +fprof_to_calltree(FromFile, ToFile, MinPercent) -> + ReplyTo = self(), + Ref = make_ref(), + spawn_link(fun() -> + ReplyTo ! {Ref, do_fprof_to_calltree(FromFile, ToFile, MinPercent)} + end), + wait_for_reply(Ref). + +wait_for_reply(Ref) -> + receive + {Ref, Res} -> + Res; + {'EXIT', normal} -> + wait_for_reply(Ref); + {'EXIT', Reason} -> + exit(Reason) + end. + +do_fprof_to_calltree(FromFile, ToFile, MinPercent) -> + {ok, Fd} = file:open(ToFile, [write, raw]), + + {ok, ConsultedFromFile} = file:consult(FromFile), + [_AnalysisOpts, [_Totals] | Terms] = ConsultedFromFile, + Processes = split_processes(Terms, [], []), + Indent = "", + Summary = collapse_processes(Processes), + {_Label, _Cnt, Acc, _Own, _Roots, Details} = Summary, + %% log(Fd, Label, Indent, Acc, {Label, Cnt, Acc, Own}, [], 0), + gen_calltree(Fd, Indent, Acc, Summary, MinPercent), + Delim = io_lib:format("\n~80..=c\n\n", [$=]), + Write = + fun(P) -> + file:write(Fd, Delim), + gen_calltree(Fd, Indent, Acc, P, MinPercent) + end, + lists:foreach(Write, Processes), + file:write(Fd, Delim), + gen_details(Fd, Acc, Details), + file:close(Fd), + ok. + +%% Split all fprof terms into a list of processes +split_processes([H | T], ProcAcc, TotalAcc) -> + if + is_tuple(H) -> + split_processes(T, [H | ProcAcc], TotalAcc); + is_list(H), ProcAcc =:= [] -> + split_processes(T, [H], TotalAcc); + is_list(H) -> + ProcAcc2 = rearrange_process(lists:reverse(ProcAcc)), + split_processes(T, [H], [ProcAcc2 | TotalAcc]) + end; +split_processes([], [], TotalAcc) -> + lists:reverse(lists:keysort(3, TotalAcc)); +split_processes([], ProcAcc, TotalAcc) -> + ProcAcc2 = rearrange_process(lists:reverse(ProcAcc)), + lists:reverse(lists:keysort(3, [ProcAcc2 | TotalAcc])). + +%% Rearrange the raw process list into a more useful format +rearrange_process([[{Label, _Cnt, _Acc, _Own} | _ ] | Details]) -> + do_rearrange_process(Details, Details, Label, [], []). + +do_rearrange_process([{CalledBy, Current, _Calls} | T], Orig, Label, Roots, Undefs) -> + case [{undefined, Cnt, safe_max(Acc, Own), Own} || + {undefined, Cnt, Acc, Own} <- CalledBy] of + [] -> + do_rearrange_process(T, Orig, Label, Roots, Undefs); + NewUndefs -> + do_rearrange_process(T, Orig, Label, [Current | Roots], NewUndefs ++ Undefs) + end; +do_rearrange_process([], Details, Label, Roots, Undefs) -> + [{undefined, Cnt, Acc, Own}] = collapse_calls(Undefs, []), + Details2 = sort_details(3, Details), + {Label, Cnt, Acc, Own, lists:reverse(lists:keysort(3, Roots)), Details2}. + +%% Compute a summary of the rearranged process info +collapse_processes(Processes) -> + Headers = lists:map(fun({_L, C, A, O, _R, _D}) -> {"SUMMARY", C, A, O} end, + Processes), + [{Label, Cnt, Acc, Own}] = collapse_calls(Headers, []), + Details = lists:flatmap(fun({_L, _C, _A, _O, _R, D}) -> D end, Processes), + Details2 = do_collapse_processes(sort_details(1, Details), []), + Roots = lists:flatmap(fun({_L, _C, _A, _O, R, _D}) -> R end, Processes), + RootMFAs = lists:usort([MFA || {MFA, _, _, _} <- Roots]), + Roots2 = [R || RootMFA <- RootMFAs, + {_, {MFA, _, _, _} = R, _} <- Details2, + MFA =:= RootMFA], + Roots3 = collapse_calls(Roots2, []), + {Label, Cnt, Acc, Own, Roots3, Details2}. + +do_collapse_processes([{CalledBy1, {MFA, Cnt1, Acc1, Own1}, Calls1} | T1], + [{CalledBy2, {MFA, Cnt2, Acc2, Own2}, Calls2} | T2]) -> + Cnt = Cnt1 + Cnt2, + Acc = Acc1 + Acc2, + Own = Own1 + Own2, + Current = {MFA, Cnt, Acc, Own}, + CalledBy0 = CalledBy1 ++ CalledBy2, + Calls0 = Calls1 ++ Calls2, + CalledBy = collapse_calls(lists:keysort(3, CalledBy0), []), + Calls = collapse_calls(lists:keysort(3, Calls0), []), + do_collapse_processes(T1, [{CalledBy, Current, Calls} | T2]); +do_collapse_processes([{CalledBy, Current, Calls} | T1], + T2) -> + do_collapse_processes(T1, [{CalledBy, Current, Calls} | T2]); +do_collapse_processes([], + T2) -> + sort_details(3, T2). + +%% Reverse sort on acc field +sort_details(Pos, Details) -> + Pivot = fun({_CalledBy1, Current1, _Calls1}, + {_CalledBy2, Current2, _Calls2}) -> + element(Pos, Current1) =< element(Pos, Current2) + end, + lists:reverse(lists:sort(Pivot, Details)). + +%% Compute a summary from a list of call tuples +collapse_calls([{MFA, Cnt1, Acc1, Own1} | T1], + [{MFA, Cnt2, Acc2, Own2} | T2]) -> + Cnt = Cnt1 + Cnt2, + Acc = safe_sum(Acc1, Acc2), + Own = Own1 + Own2, + collapse_calls(T1, [{MFA, Cnt, Acc, Own} | T2]); +collapse_calls([{MFA, Cnt, Acc, Own} | T1], + T2) -> + collapse_calls(T1, [{MFA, Cnt, Acc, Own} | T2]); +collapse_calls([], + T2) -> + lists:reverse(lists:keysort(3, T2)). + +safe_sum(Int1, Int2) -> + if + Int1 =:= undefined -> Int2; + Int2 =:= undefined -> Int1; + true -> Int1 + Int2 + end. + +safe_max(Int1, Int2) -> + if + Int1 =:= undefined -> + io:format("111\n", []), + Int2; + Int2 =:= undefined -> + io:format("222\n", []), + Int1; + Int2 > Int1 -> Int2; + true -> Int1 + end. + +%% Compute a calltree and write it to file +gen_calltree(Fd, Indent, TotalAcc, {Label, Cnt, Acc, Own, Roots, Details}, MinPercent) -> + Header = {Label, Cnt, Acc, Own}, + MetaLabel = "Process", + Diff = length(Label) - length(MetaLabel), + IoList = io_lib:format("~s~s Lvl Pct Cnt Acc Own Calls => MFA\n", + [MetaLabel, lists:duplicate(Diff, $\ )]), + file:write(Fd, IoList), + log(Fd, Label, Indent, TotalAcc, Header, Roots, MinPercent), + NewIndent = " " ++ Indent, + Fun = fun({MFA, _C, _A, _O}) -> + [put_detail(Label, D) || D <- Details], + gen_calls(Fd, Label, NewIndent, TotalAcc, MFA, MinPercent) + end, + lists:foreach(Fun, Roots). + +gen_calls(Fd, Label, Indent, TotalAcc, MFA, MinPercent) -> + case get_detail(Label, MFA) of + {read, {_CalledBy, Current, _Calls}} -> + log(Fd, Label, Indent, TotalAcc, Current, -1, MinPercent); + {unread, {_CalledBy, Current, Calls}} -> + log(Fd, Label, Indent, TotalAcc, Current, Calls, MinPercent), + NewIndent = " " ++ Indent, + Fun = fun({NextMFA, _, _, _}) -> + gen_calls(Fd, Label, NewIndent, TotalAcc, + NextMFA, MinPercent) + end, + lists:foreach(Fun, Calls) + end. + +put_detail(Label, {_, {MFA, _, _, _}, _} = Detail) -> + put({Label, MFA}, {unread, Detail}). + +get_detail(Label, MFA) -> + Val = get({Label, MFA}), + case Val of + {unread, Detail} -> + put({Label, MFA}, {read, Detail}), + Val; + {read, _Detail} -> + Val + end. + +gen_details(Fd, Total, Details) -> + IoList = io_lib:format("Pct Cnt Acc Own MFA\n", []), + file:write(Fd, IoList), + do_gen_details(Fd, Total, Details). + +do_gen_details(Fd, Total, [{_CalledBy, {MFA, Cnt, Acc, Own}, _Calls} | Details]) -> + MFAStr = io_lib:format("~p", [MFA]), + {_, Percent} = calc_percent(Acc, Own, Total), + IoList = io_lib:format("~3.. B% ~10.3B ~10.3f ~10.3f => ~s\n", + [Percent, Cnt, Acc, Own, MFAStr]), + file:write(Fd, IoList), + do_gen_details(Fd, Total, Details); +do_gen_details(_Fd, _Total, []) -> + ok. + +log(Fd, Label, Indent, Acc, Current, Calls, MinPercent) when is_list(Calls) -> + log(Fd, Label, Indent, Acc, Current, length(Calls), MinPercent); +log(Fd, Label, Indent, Total, {MFA, Cnt, Acc, Own}, N, MinPercent) -> + {Max, Percent} = calc_percent(Acc, Own, Total), + if + Percent >= MinPercent -> + do_log(Fd, Label, Indent, Percent, MFA, Cnt, Max, Own, N); + true -> + ok + end. + +do_log(Fd, Label, Indent, Percent, MFA, Cnt, Acc, Own, N) -> + MFAStr = io_lib:format("~p", [MFA]), + CallsStr = io_lib:format(" ~5.. s ", [lists:concat([N])]), + IoList = io_lib:format("~s ~3.. B " + "~s~3.. B% ~10.. B ~10.. B ~10.. B ~s => ~s\n", + [Label, length(Indent) div 2, + Indent, Percent, Cnt, + round(Acc), round(Own), CallsStr, MFAStr]), + file:write(Fd, IoList). + +calc_percent(Acc, Own, Total) -> + Max = safe_max(Acc, Own), + {Max, round((Max * 100) / Total)}. diff --git a/lib/kernel/test/socket_test_ttest_tcp_client.erl b/lib/kernel/test/socket_test_ttest_tcp_client.erl index b062b82b2582..aeaa66ab831a 100644 --- a/lib/kernel/test/socket_test_ttest_tcp_client.erl +++ b/lib/kernel/test/socket_test_ttest_tcp_client.erl @@ -72,7 +72,7 @@ -define(T(), ?LIB:t()). -define(TDIFF(T1,T2), ?LIB:tdiff(T1, T2)). --type active() :: once | boolean(). +-type active() :: once | boolean() | 1..32767. -type msg_id() :: 1..3. -type max_outstanding() :: pos_integer(). -type runtime() :: pos_integer(). @@ -177,7 +177,7 @@ do_start(Quiet, is_pid(Parent) andalso is_function(Notify) andalso (is_atom(Transport) orelse is_tuple(Transport)) andalso - (is_boolean(Active) orelse (Active =:= once)) andalso + (is_boolean(Active) orelse (Active =:= once) orelse (is_integer(Active) andalso (Active > 0) andalso (Active =< 32767))) andalso (is_tuple(ServerInfo) orelse is_list(ServerInfo) orelse is_binary(ServerInfo)) andalso (is_integer(MsgID) andalso (MsgID >= 1) andalso (MsgID =< 3)) andalso (is_integer(MaxOutstanding) andalso (MaxOutstanding > 0)) andalso @@ -222,6 +222,7 @@ init(Quiet, Parent, Notify, Transport, Active, ServerInfo, MsgID, MaxOutstanding, RunTime) -> + put(activations, 0), if not Quiet -> ?I("init with" @@ -467,6 +468,10 @@ recv_reply(#{mod := Mod, ok -> State; + activate -> + activate(Mod, Sock, Active), + State; + {error, stop} -> ?I("receive [~w] -> stop", [Active]), %% This will have the effect that no more requests are sent... @@ -550,13 +555,21 @@ recv_reply_message3(_Mod, Sock, ID, Acc) -> {Tag, Sock, Msg} when (Tag =:= tcp) orelse (Tag =:= socket) -> - process_acc_data(ID, <>) + process_acc_data(ID, <>); + + {TagPassive, Sock} when (TagPassive =:= tcp_passive) orelse + (TagPassive =:= socket_passive) -> + ACnt = get(activations), + put(activations, ACnt+1), + activate after ?RECV_TIMEOUT -> ?I("timeout when" "~n ID: ~p" - "~n size(Acc): ~p", - [ID, size(Acc)]), + "~n size(Acc): ~p" + "~n MQ: ~p" + "~n ACnt: ~p", + [ID, size(Acc), mq(), get(activations)]), %% {error, timeout} recv_reply_message3(_Mod, Sock, ID, Acc) end. @@ -602,6 +615,11 @@ handle_message(#{quiet := Quiet, reply(Parent, Ref, ok), exit(normal); + {TagPassive, Sock} when (TagPassive =:= tcp_passive) orelse + (TagPassive =:= socket_passive) -> + activate(State), + State; + %% Only when active {TagClosed, Sock, Reason} when (TagClosed =:= tcp_closed) orelse (TagClosed =:= socket_closed) -> @@ -621,15 +639,24 @@ handle_message(#{quiet := Quiet, initial_activation(_Mod, _Sock, false = _Active) -> ok; initial_activation(Mod, Sock, Active) -> - Mod:active(Sock, Active). + activate(Mod, Sock, Active). maybe_activate(Mod, Sock, once = Active) -> - Mod:active(Sock, Active); + activate(Mod, Sock, Active); maybe_activate(_, _, _) -> ok. +activate(#{mod := Mod, + sock := Sock, + active := Active} = _State) -> + activate(Mod, Sock, Active). + +activate(Mod, Sock, Active) -> + ok = Mod:active(Sock, Active). + + %% ========================================================================== %% req(Pid, Req) -> @@ -660,6 +687,16 @@ next_id(_) -> 1. +%% ========================================================================== + +mq() -> + mq(self()). + +mq(Pid) when is_pid(Pid) -> + {messages, MQ} = process_info(Pid, messages), + MQ. + + %% ========================================================================== %% t() -> diff --git a/lib/kernel/test/socket_test_ttest_tcp_gen.erl b/lib/kernel/test/socket_test_ttest_tcp_gen.erl index 4c2322d35dae..e5cbeef42d22 100644 --- a/lib/kernel/test/socket_test_ttest_tcp_gen.erl +++ b/lib/kernel/test/socket_test_ttest_tcp_gen.erl @@ -18,6 +18,10 @@ %% %CopyrightEnd% %% +%% +%% Plain gen_tcp. +%% + -module(socket_test_ttest_tcp_gen). -export([ @@ -57,8 +61,10 @@ accept(Sock, Timeout) -> end. -active(Sock, NewActive) - when (is_boolean(NewActive) orelse (NewActive =:= once)) -> +%% active(Sock, NewActive) +%% when (is_boolean(NewActive) orelse (NewActive =:= once)) -> +%% inet:setopts(Sock, [{active, NewActive}]). +active(Sock, NewActive) -> inet:setopts(Sock, [{active, NewActive}]). diff --git a/lib/kernel/test/socket_test_ttest_tcp_gs.erl b/lib/kernel/test/socket_test_ttest_tcp_gs.erl index d6545c729e10..76c098fdde43 100644 --- a/lib/kernel/test/socket_test_ttest_tcp_gs.erl +++ b/lib/kernel/test/socket_test_ttest_tcp_gs.erl @@ -18,6 +18,11 @@ %% %CopyrightEnd% %% +%% +%% gen_tcp with inet_backend = socket. +%% That is, socket "via" gen_tcp. +%% + -module(socket_test_ttest_tcp_gs). -export([ @@ -38,6 +43,7 @@ -define(LIB, socket_test_lib). + %% ========================================================================== accept(Sock) -> @@ -57,8 +63,10 @@ accept(Sock, Timeout) -> end. -active(Sock, NewActive) - when (is_boolean(NewActive) orelse (NewActive =:= once)) -> +%% active(Sock, NewActive) +%% when (is_boolean(NewActive) orelse (NewActive =:= once)) -> +%% inet:setopts(Sock, [{active, NewActive}]). +active(Sock, NewActive) -> inet:setopts(Sock, [{active, NewActive}]). diff --git a/lib/kernel/test/socket_test_ttest_tcp_socket.erl b/lib/kernel/test/socket_test_ttest_tcp_socket.erl index b38056f92500..09eca0b954b9 100644 --- a/lib/kernel/test/socket_test_ttest_tcp_socket.erl +++ b/lib/kernel/test/socket_test_ttest_tcp_socket.erl @@ -18,6 +18,11 @@ %% %CopyrightEnd% %% +%% +%% socket with a basic wrapper that (basically) +%% provides active = false | true | once | n. +%% + -module(socket_test_ttest_tcp_socket). -export([ @@ -40,19 +45,22 @@ -define(READER_RECV_TIMEOUT, 1000). +-define(CSOCK(SOCK, METHOD), + ?CSOCK(SOCK, self(), METHOD)). +-define(CSOCK(SOCK, READER, METHOD), + #{sock => Sock, reader => READER, method => METHOD}). + -define(DATA_MSG(Sock, Method, Data), - {socket, - #{sock => Sock, reader => self(), method => Method}, - Data}). + {socket, ?CSOCK(Sock, Method), Data}). -define(CLOSED_MSG(Sock, Method), - {socket_closed, - #{sock => Sock, reader => self(), method => Method}}). + {socket_closed, ?CSOCK(Sock, Method)}). + +-define(PASSIVE_MSG(Sock, Method), + {socket_passive, ?CSOCK(Sock, Method)}). -define(ERROR_MSG(Sock, Method, Reason), - {socket_error, - #{sock => Sock, reader => self(), method => Method}, - Reason}). + {socket_error, ?CSOCK(Sock, Method), Reason}). -define(SELECT_INFO(TAG, REF), {select_info, TAG, REF}). @@ -88,7 +96,7 @@ accept(#{sock := LSock, opts := #{async := Async, reader_init(Self, Sock, Async, false, Method) end), maybe_start_stats_timer(Opts, Reader), - {ok, #{sock => Sock, reader => Reader, method => Method}}; + {ok, ?CSOCK(Sock, Reader, Method)}; {error, _} = ERROR -> ERROR end. @@ -104,7 +112,7 @@ accept(#{sock := LSock, opts := #{async := Async, reader_init(Self, Sock, Async, false, Method) end), maybe_start_stats_timer(Opts, Reader), - {ok, #{sock => Sock, reader => Reader, method => Method}}; + {ok, ?CSOCK(Sock, Reader, Method)}; {error, _} = ERROR -> ERROR end. @@ -113,7 +121,16 @@ accept(#{sock := LSock, opts := #{async := Async, active(#{reader := Pid}, NewActive) when (is_boolean(NewActive) orelse (NewActive =:= once)) -> Pid ! {?MODULE, active, NewActive}, - ok. + ok; +active(#{reader := Pid}, NewActive) + when (is_integer(NewActive) andalso + (-32768 =< NewActive) andalso (NewActive =< 32767)) -> + Ref = make_ref(), + Pid ! {?MODULE, self(), active, NewActive, Ref}, + receive + {?MODULE, Pid, active, Reply, Ref} -> + Reply + end. close(#{sock := Sock, reader := Pid}) -> @@ -373,7 +390,8 @@ sockname(#{sock := Sock}) -> reader_init(ControllingProcess, Sock, Async, Active, Method) when is_pid(ControllingProcess) andalso is_boolean(Async) andalso - (is_boolean(Active) orelse (Active =:= once)) andalso + (is_boolean(Active) orelse (Active =:= once) orelse + is_integer(Active)) andalso ((Method =:= plain) orelse (Method =:= msg)) -> put(verbose, false), MRef = erlang:monitor(process, ControllingProcess), @@ -406,6 +424,29 @@ reader_loop(#{active := false, {?MODULE, active, NewActive} -> reader_loop(State#{active => NewActive}); + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + {'DOWN', MRef, process, Pid, Reason} -> case maps:get(ctrl_proc_mref, State) of MRef -> @@ -441,6 +482,29 @@ reader_loop(#{active := once, {?MODULE, active, NewActive} -> reader_loop(State#{active => NewActive}); + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + {'DOWN', MRef, process, Pid, Reason} -> case maps:get(ctrl_proc_mref, State) of MRef -> @@ -487,7 +551,7 @@ reader_loop(#{active := once, reader_loop(#{active := once, async := true, asynch_info := AsynchInfo, - asynch_num := N, + asynch_num := ANum, sock := Sock, method := Method, ctrl_proc := Pid} = State) when (AsynchInfo =/= undefined) -> @@ -500,7 +564,7 @@ reader_loop(#{active := once, receive {?MODULE, stop} -> reader_exit(State, stop); - + {?MODULE, Pid, controlling_process, NewPid} -> OldMRef = maps:get(ctrl_proc_mref, State), erlang:demonitor(OldMRef, [flush]), @@ -508,10 +572,33 @@ reader_loop(#{active := once, Pid ! {?MODULE, self(), controlling_process}, reader_loop(State#{ctrl_proc => NewPid, ctrl_proc_mref => NewMRef}); - + {?MODULE, active, NewActive} -> reader_loop(State#{active => NewActive}); - + + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + {'DOWN', MRef, process, Pid, Reason} -> case maps:get(ctrl_proc_mref, State) of MRef -> @@ -526,7 +613,7 @@ reader_loop(#{active := once, Pid ! ?DATA_MSG(Sock, Method, Data), reader_loop(State#{active => false, asynch_info => undefined, - asynch_num => N+1}); + asynch_num => ANum+1}); {error, closed} = E1 -> Pid ! ?CLOSED_MSG(Sock, Method), @@ -545,12 +632,249 @@ reader_loop(#{active := once, Pid ! ?DATA_MSG(Sock, Method, Data), reader_loop(State#{active => false, asynch_info => undefined, - asynch_num => N+1}); + asynch_num => ANum+1}); {ok, #{iov := [Data]}} when is_binary(Data) -> Pid ! ?DATA_MSG(Sock, Method, Data), reader_loop(State#{active => false, asynch_info => undefined, - asynch_num => N+1}); + asynch_num => ANum+1}); + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end + end; + + + +%% Read *n* times and then change to false +reader_loop(#{active := N, + async := false, + sock := Sock, + method := Method, + ctrl_proc := Pid} = State) when is_integer(N) andalso (N > 0) -> + case do_recv(Method, Sock) of + {ok, Data} -> + Pid ! ?DATA_MSG(Sock, Method, Data), + N2 = + if + (N > 1) -> + N-1; + true -> + Pid ! ?PASSIVE_MSG(Sock, Method), + false + end, + reader_loop(State#{active => N2}); + {error, timeout} -> + receive + {?MODULE, stop} -> + reader_exit(State, stop); + + {?MODULE, Pid, controlling_process, NewPid} -> + OldMRef = maps:get(ctrl_proc_mref, State), + erlang:demonitor(OldMRef, [flush]), + NewMRef = erlang:monitor(process, NewPid), + Pid ! {?MODULE, self(), controlling_process}, + reader_loop(State#{ctrl_proc => NewPid, + ctrl_proc_mref => NewMRef}); + + {?MODULE, active, NewActive} -> + reader_loop(State#{active => NewActive}); + + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + + {'DOWN', MRef, process, Pid, Reason} -> + case maps:get(ctrl_proc_mref, State) of + MRef -> + reader_exit(State, {ctrl_exit, Reason}); + _ -> + reader_loop(State) + end + after 0 -> + reader_loop(State) + end; + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end; +reader_loop(#{active := N, + async := true, + asynch_info := undefined = _AsynchInfo, + sock := Sock, + method := Method, + ctrl_proc := Pid} = State) when is_integer(N) andalso (N > 0) -> + case do_recv(Method, Sock, nowait) of + {select, SelectInfo} -> + reader_loop(State#{asynch_info => SelectInfo}); + {completion, CompletionInfo} -> + reader_loop(State#{asynch_info => CompletionInfo}); + + {ok, Data} -> + Pid ! ?DATA_MSG(Sock, Method, Data), + N2 = + if + (N > 1) -> + N-1; + true -> + Pid ! ?PASSIVE_MSG(Sock, Method), + false + end, + reader_loop(State#{active => N2}); + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end; +reader_loop(#{active := N, + async := true, + asynch_info := AsynchInfo, + asynch_num := ANum, + sock := Sock, + method := Method, + ctrl_proc := Pid} = State) + when (AsynchInfo =/= undefined) andalso + is_integer(N) andalso (N > 0) -> + Ref = case AsynchInfo of + ?SELECT_INFO(_, SR) -> + SR; + ?COMPLETION_INFO(_, CR) -> + CR + end, + receive + {?MODULE, stop} -> + reader_exit(State, stop); + + {?MODULE, Pid, controlling_process, NewPid} -> + OldMRef = maps:get(ctrl_proc_mref, State), + erlang:demonitor(OldMRef, [flush]), + NewMRef = erlang:monitor(process, NewPid), + Pid ! {?MODULE, self(), controlling_process}, + reader_loop(State#{ctrl_proc => NewPid, + ctrl_proc_mref => NewMRef}); + + {?MODULE, active, NewActive} -> + reader_loop(State#{active => NewActive}); + + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N2 = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N2 -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N2 =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N2}) + end; + + {'DOWN', MRef, process, Pid, Reason} -> + case maps:get(ctrl_proc_mref, State) of + MRef -> + reader_exit(State, {ctrl_exit, Reason}); + _ -> + reader_loop(State) + end; + + ?SELECT_MSG(Sock, Ref) -> + case do_recv(Method, Sock, nowait) of + {ok, Data} when is_binary(Data) -> + Pid ! ?DATA_MSG(Sock, Method, Data), + N2 = + if + (N > 1) -> + N-1; + true -> + Pid ! ?PASSIVE_MSG(Sock, Method), + false + end, + reader_loop(State#{active => N2, + asynch_info => undefined, + asynch_num => ANum+1}); + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end; + + ?COMPLETION_MSG(Sock, Ref, Result) -> + %% Note that *Windows* does not support sendmsg/recvmsg + %% but we assume we can get it. Just to be future proof + case Result of + {ok, Data} when is_binary(Data) -> + Pid ! ?DATA_MSG(Sock, Method, Data), + N2 = + if + (N > 1) -> + N-1; + true -> + Pid ! ?PASSIVE_MSG(Sock, Method), + false + end, + reader_loop(State#{active => N2, + asynch_info => undefined, + asynch_num => ANum+1}); + {ok, #{iov := [Data]}} when is_binary(Data) -> + Pid ! ?DATA_MSG(Sock, Method, Data), + N2 = + if + (N > 1) -> + N-1; + true -> + Pid ! ?PASSIVE_MSG(Sock, Method), + false + end, + reader_loop(State#{active => N2, + asynch_info => undefined, + asynch_num => ANum+1}); {error, closed} = E1 -> Pid ! ?CLOSED_MSG(Sock, Method), @@ -562,6 +886,7 @@ reader_loop(#{active := once, end end; + %% Read and forward data continuously reader_loop(#{active := true, async := false, @@ -588,6 +913,29 @@ reader_loop(#{active := true, {?MODULE, active, NewActive} -> reader_loop(State#{active => NewActive}); + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + {'DOWN', MRef, process, Pid, Reason} -> case maps:get(ctrl_proc_mref, State) of MRef -> @@ -634,7 +982,7 @@ reader_loop(#{active := true, reader_loop(#{active := true, async := true, asynch_info := AsynchInfo, - asynch_num := N, + asynch_num := ANum, sock := Sock, method := Method, ctrl_proc := Pid} = State) when (AsynchInfo =/= undefined) -> @@ -659,6 +1007,29 @@ reader_loop(#{active := true, {?MODULE, active, NewActive} -> reader_loop(State#{active => NewActive}); + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + {'DOWN', MRef, process, Pid, Reason} -> case maps:get(ctrl_proc_mref, State) of MRef -> @@ -672,7 +1043,7 @@ reader_loop(#{active := true, {ok, Data} when is_binary(Data) -> Pid ! ?DATA_MSG(Sock, Method, Data), reader_loop(State#{asynch_info => undefined, - asynch_num => N+1}); + asynch_num => ANum+1}); {error, closed} = E1 -> Pid ! ?CLOSED_MSG(Sock, Method), @@ -691,12 +1062,199 @@ reader_loop(#{active := true, Pid ! ?DATA_MSG(Sock, Method, Data), reader_loop(State#{active => false, asynch_info => undefined, - asynch_num => N+1}); + asynch_num => ANum+1}); {ok, #{iov := [Data]}} when is_binary(Data) -> Pid ! ?DATA_MSG(Sock, Method, Data), reader_loop(State#{active => false, asynch_info => undefined, - asynch_num => N+1}); + asynch_num => ANum+1}); + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end + end; + +%% Read *once* and then change to false +reader_loop(#{active := once, + async := false, + sock := Sock, + method := Method, + ctrl_proc := Pid} = State) -> + case do_recv(Method, Sock) of + {ok, Data} -> + Pid ! ?DATA_MSG(Sock, Method, Data), + reader_loop(State#{active => false}); + {error, timeout} -> + receive + {?MODULE, stop} -> + reader_exit(State, stop); + + {?MODULE, Pid, controlling_process, NewPid} -> + OldMRef = maps:get(ctrl_proc_mref, State), + erlang:demonitor(OldMRef, [flush]), + NewMRef = erlang:monitor(process, NewPid), + Pid ! {?MODULE, self(), controlling_process}, + reader_loop(State#{ctrl_proc => NewPid, + ctrl_proc_mref => NewMRef}); + + {?MODULE, active, NewActive} -> + reader_loop(State#{active => NewActive}); + + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = OldActive + NewActive, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + + {'DOWN', MRef, process, Pid, Reason} -> + case maps:get(ctrl_proc_mref, State) of + MRef -> + reader_exit(State, {ctrl_exit, Reason}); + _ -> + reader_loop(State) + end + after 0 -> + reader_loop(State) + end; + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end; +reader_loop(#{active := once, + async := true, + asynch_info := undefined, + sock := Sock, + method := Method, + ctrl_proc := Pid} = State) -> + case do_recv(Method, Sock, nowait) of + {select, SelectInfo} -> + reader_loop(State#{asynch_info => SelectInfo}); + {completion, CompletionInfo} -> + reader_loop(State#{asynch_info => CompletionInfo}); + + {ok, Data} -> + Pid ! ?DATA_MSG(Sock, Method, Data), + reader_loop(State#{active => false}); + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end; +reader_loop(#{active := once, + async := true, + asynch_info := AsynchInfo, + asynch_num := ANum, + sock := Sock, + method := Method, + ctrl_proc := Pid} = State) when (AsynchInfo =/= undefined) -> + Ref = case AsynchInfo of + ?SELECT_INFO(_, SR) -> + SR; + ?COMPLETION_INFO(_, CR) -> + CR + end, + receive + {?MODULE, stop} -> + reader_exit(State, stop); + + {?MODULE, Pid, controlling_process, NewPid} -> + OldMRef = maps:get(ctrl_proc_mref, State), + erlang:demonitor(OldMRef, [flush]), + NewMRef = erlang:monitor(process, NewPid), + Pid ! {?MODULE, self(), controlling_process}, + reader_loop(State#{ctrl_proc => NewPid, + ctrl_proc_mref => NewMRef}); + + {?MODULE, active, NewActive} -> + reader_loop(State#{active => NewActive}); + + {?MODULE, From, active, NewActive, Ref} -> + OldActive = maps:get(active, State), + N = + if + is_integer(OldActive) -> + OldActive + NewActive; + true -> + NewActive + end, + if + 32767 < N -> + From ! {?MODULE, self(), active, {error, einval}, Ref}; + N =< 0 -> + From ! {?MODULE, self(), active, ok, Ref}, + Sock = maps:get(sock, State), + Method = maps:get(method, State), + Pid ! ?PASSIVE_MSG(Sock, Method), + reader_loop(State#{active => false}); + true -> + From ! {?MODULE, self(), active, ok, Ref}, + reader_loop(State#{active => N}) + end; + + {'DOWN', MRef, process, Pid, Reason} -> + case maps:get(ctrl_proc_mref, State) of + MRef -> + reader_exit(State, {ctrl_exit, Reason}); + _ -> + reader_loop(State) + end; + + ?SELECT_MSG(Sock, Ref) -> + case do_recv(Method, Sock, nowait) of + {ok, Data} when is_binary(Data) -> + Pid ! ?DATA_MSG(Sock, Method, Data), + reader_loop(State#{active => false, + asynch_info => undefined, + asynch_num => ANum+1}); + + {error, closed} = E1 -> + Pid ! ?CLOSED_MSG(Sock, Method), + reader_exit(State, E1); + + {error, Reason} = E2 -> + Pid ! ?ERROR_MSG(Sock, Method, Reason), + reader_exit(State, E2) + end; + + ?COMPLETION_MSG(Sock, Ref, Result) -> + %% Note that *Windows* does not support sendmsg/recvmsg + %% but we assume we can get it. Just to be future proof + case Result of + {ok, Data} when is_binary(Data) -> + Pid ! ?DATA_MSG(Sock, Method, Data), + reader_loop(State#{active => false, + asynch_info => undefined, + asynch_num => ANum+1}); + {ok, #{iov := [Data]}} when is_binary(Data) -> + Pid ! ?DATA_MSG(Sock, Method, Data), + reader_loop(State#{active => false, + asynch_info => undefined, + asynch_num => ANum+1}); {error, closed} = E1 -> Pid ! ?CLOSED_MSG(Sock, Method), @@ -709,6 +1267,7 @@ reader_loop(#{active := true, end. + do_recv(Method, Sock) -> do_recv(Method, Sock, ?READER_RECV_TIMEOUT). @@ -733,10 +1292,10 @@ reader_exit(#{async := false, active := Active}, stop) -> reader_exit(#{async := true, active := Active, asynch_info := AsynchInfo, - asynch_num := N}, stop) -> + asynch_num := ANum}, stop) -> vp("reader stopped when active: ~w" "~n Current asynch info: ~p" - "~n Number of asynch messages: ~p", [Active, AsynchInfo, N]), + "~n Number of asynch messages: ~p", [Active, AsynchInfo, ANum]), exit(normal); reader_exit(#{async := false, active := Active}, {ctrl_exit, normal}) -> vp("reader ctrl exit when active: ~w", [Active]), @@ -744,10 +1303,10 @@ reader_exit(#{async := false, active := Active}, {ctrl_exit, normal}) -> reader_exit(#{async := true, active := Active, asynch_info := AsynchInfo, - asynch_num := N}, {ctrl_exit, normal}) -> + asynch_num := ANum}, {ctrl_exit, normal}) -> vp("reader ctrl exit when active: ~w" "~n Current asynch info: ~p" - "~n Number of asynch messages: ~p", [Active, AsynchInfo, N]), + "~n Number of asynch messages: ~p", [Active, AsynchInfo, ANum]), exit(normal); reader_exit(#{async := false, active := Active}, {ctrl_exit, Reason}) -> vp("reader exit when ctrl crash when active: ~w", [Active]), @@ -755,10 +1314,10 @@ reader_exit(#{async := false, active := Active}, {ctrl_exit, Reason}) -> reader_exit(#{async := true, active := Active, asynch_info := AsynchInfo, - asynch_num := N}, {ctrl_exit, Reason}) -> + asynch_num := ANum}, {ctrl_exit, Reason}) -> vp("reader exit when ctrl crash when active: ~w" "~n Current asynch info: ~p" - "~n Number of asynch messages: ~p", [Active, AsynchInfo, N]), + "~n Number of asynch messages: ~p", [Active, AsynchInfo, ANum]), exit({controlling_process, Reason}); reader_exit(#{async := false, active := Active}, {error, closed}) -> vp("reader exit when socket closed when active: ~w", [Active]), @@ -766,10 +1325,10 @@ reader_exit(#{async := false, active := Active}, {error, closed}) -> reader_exit(#{async := true, active := Active, asynch_info := AsynchInfo, - asynch_num := N}, {error, closed}) -> + asynch_num := ANum}, {error, closed}) -> vp("reader exit when socket closed when active: ~w " "~n Current asynch info: ~p" - "~n Number of asynch messages: ~p", [Active, AsynchInfo, N]), + "~n Number of asynch messages: ~p", [Active, AsynchInfo, ANum]), exit(normal); reader_exit(#{async := false, active := Active}, {error, Reason}) -> vp("reader exit when socket error when active: ~w", [Active]), @@ -777,10 +1336,10 @@ reader_exit(#{async := false, active := Active}, {error, Reason}) -> reader_exit(#{async := true, active := Active, asynch_info := AsynchInfo, - asynch_num := N}, {error, Reason}) -> + asynch_num := ANum}, {error, Reason}) -> vp("reader exit when socket error when active: ~w: " "~n Current asynch info: ~p" - "~n Number of asynch messages: ~p", [Active, AsynchInfo, N]), + "~n Number of asynch messages: ~p", [Active, AsynchInfo, ANum]), exit(Reason). @@ -788,6 +1347,16 @@ reader_exit(#{async := true, +%% ========================================================================== + +mq() -> + mq(self()). + +mq(Pid) when is_pid(Pid) -> + {messages, MQ} = process_info(Pid, messages), + MQ. + + %% ========================================================================== vp(F, A) -> diff --git a/lib/kernel/test/zlib_SUITE.erl b/lib/kernel/test/zlib_SUITE.erl index 0b8af8c4c49c..55fa79e982a1 100644 --- a/lib/kernel/test/zlib_SUITE.erl +++ b/lib/kernel/test/zlib_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2005-2021. All Rights Reserved. +%% Copyright Ericsson AB 2005-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/kernel/vsn.mk b/lib/kernel/vsn.mk index 9d2fed5df517..b29edd8544d8 100644 --- a/lib/kernel/vsn.mk +++ b/lib/kernel/vsn.mk @@ -1 +1 @@ -KERNEL_VSN = 9.1 +KERNEL_VSN = 9.2 diff --git a/lib/megaco/test/megaco_mib_SUITE.erl b/lib/megaco/test/megaco_mib_SUITE.erl index f236825e44ec..ea0879e9a98a 100644 --- a/lib/megaco/test/megaco_mib_SUITE.erl +++ b/lib/megaco/test/megaco_mib_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2002-2022. All Rights Reserved. +%% Copyright Ericsson AB 2002-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/megaco/test/megaco_mreq_SUITE.erl b/lib/megaco/test/megaco_mreq_SUITE.erl index c304409f8fe6..10db9ef4bf3a 100644 --- a/lib/megaco/test/megaco_mreq_SUITE.erl +++ b/lib/megaco/test/megaco_mreq_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2003-2021. All Rights Reserved. +%% Copyright Ericsson AB 2003-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/mnesia/doc/src/notes.xml b/lib/mnesia/doc/src/notes.xml index 17095e36c906..af6599c4c68e 100644 --- a/lib/mnesia/doc/src/notes.xml +++ b/lib/mnesia/doc/src/notes.xml @@ -39,7 +39,42 @@ thus constitutes one section in this document. The title of each section is the version number of Mnesia.

-
Mnesia 4.22.1 +
Mnesia 4.23 + +
Fixed Bugs and Malfunctions + + +

+ Document mnesia:foldl/4 and mnesia:foldr/4.

+

+ Own Id: OTP-18798

+ + +

+ mnesia:add_table_copy/3 no longer fails with + reason system_limit when the node is starting.

+

+ Own Id: OTP-18850

+ + +
+ + +
Improvements and New Features + + +

+ Restore recreate of disc_only tables could crash if they + had an index.

+

+ Own Id: OTP-18843 Aux Id: GH-7766

+ + +
+ +
+ +
Mnesia 4.22.1
Fixed Bugs and Malfunctions @@ -76,6 +111,22 @@
+
+ +
Mnesia 4.21.4.2 + +
Fixed Bugs and Malfunctions + + +

+ mnesia:add_table_copy/3 no longer fails with + reason system_limit when the node is starting.

+

+ Own Id: OTP-18850

+ + +
+
Mnesia 4.21.4.1 @@ -205,6 +256,22 @@
+
+ +
Mnesia 4.20.4.4 + +
Fixed Bugs and Malfunctions + + +

+ mnesia:add_table_copy/3 no longer fails with + reason system_limit when the node is starting.

+

+ Own Id: OTP-18850

+ + +
+
Mnesia 4.20.4.3 diff --git a/lib/mnesia/src/mnesia.erl b/lib/mnesia/src/mnesia.erl index 00f696a1bcc2..6ddd75728e06 100644 --- a/lib/mnesia/src/mnesia.erl +++ b/lib/mnesia/src/mnesia.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2021. All Rights Reserved. +%% Copyright Ericsson AB 1996-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/mnesia/src/mnesia_dumper.erl b/lib/mnesia/src/mnesia_dumper.erl index 206ce3ce0611..cd712c34ade6 100644 --- a/lib/mnesia/src/mnesia_dumper.erl +++ b/lib/mnesia/src/mnesia_dumper.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2022. All Rights Reserved. +%% Copyright Ericsson AB 1996-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/mnesia/vsn.mk b/lib/mnesia/vsn.mk index 6de36caf8002..5bb651fdbd7a 100644 --- a/lib/mnesia/vsn.mk +++ b/lib/mnesia/vsn.mk @@ -1 +1 @@ -MNESIA_VSN = 4.22.1 +MNESIA_VSN = 4.23 diff --git a/lib/observer/src/crashdump_viewer.erl b/lib/observer/src/crashdump_viewer.erl index 2e9c79a6f33c..1243730773d7 100644 --- a/lib/observer/src/crashdump_viewer.erl +++ b/lib/observer/src/crashdump_viewer.erl @@ -2894,7 +2894,12 @@ parse_heap_term("Mh"++Line0, Addr, DecodeOpts, D0) -> %Head node in a hashmap. {Map,Line,D}; parse_heap_term("Mn"++Line0, Addr, DecodeOpts, D) -> %Interior node in a hashmap. {N,":"++Line} = get_hex(Line0), - parse_tuple(N, Line, Addr, DecodeOpts, D, []). + parse_tuple(N, Line, Addr, DecodeOpts, D, []); +parse_heap_term("Rf"++Line0, Addr, _DecodeOpts, D0) -> %Fun reference + {N,[]} = get_hex(Line0), + Term = {'#CDVFRef', N}, + D = gb_trees:insert(Addr, Term, D0), + {Term,[],D}. parse_tuple(0, Line, Addr, _, D0, Acc) -> Tuple = list_to_tuple(lists:reverse(Acc)), diff --git a/lib/os_mon/c_src/win32sysinfo.c b/lib/os_mon/c_src/win32sysinfo.c index bd51bf6777fb..23a766070917 100644 --- a/lib/os_mon/c_src/win32sysinfo.c +++ b/lib/os_mon/c_src/win32sysinfo.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 1997-2022. All Rights Reserved. + * Copyright Ericsson AB 1997-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/lib/os_mon/doc/src/notes.xml b/lib/os_mon/doc/src/notes.xml index 164566b2c3fb..a80ca81ce5dc 100644 --- a/lib/os_mon/doc/src/notes.xml +++ b/lib/os_mon/doc/src/notes.xml @@ -31,6 +31,21 @@

This document describes the changes made to the OS_Mon application.

+
Os_Mon 2.9.1 + +
Fixed Bugs and Malfunctions + + +

+ Fixed some benign compile warnings on Windows.

+

+ Own Id: OTP-18895

+ + +
+ +
+
Os_Mon 2.9
Fixed Bugs and Malfunctions diff --git a/lib/os_mon/vsn.mk b/lib/os_mon/vsn.mk index b109e7a0ab15..f570410a9a29 100644 --- a/lib/os_mon/vsn.mk +++ b/lib/os_mon/vsn.mk @@ -1 +1 @@ -OS_MON_VSN = 2.9 +OS_MON_VSN = 2.9.1 diff --git a/lib/public_key/asn1/Makefile b/lib/public_key/asn1/Makefile index ad3fd909dff9..f1a3972c89af 100644 --- a/lib/public_key/asn1/Makefile +++ b/lib/public_key/asn1/Makefile @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2008-2022. All Rights Reserved. +# Copyright Ericsson AB 2008-2023. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/lib/public_key/doc/src/notes.xml b/lib/public_key/doc/src/notes.xml index 502e1d84151f..f7b396e27aa6 100644 --- a/lib/public_key/doc/src/notes.xml +++ b/lib/public_key/doc/src/notes.xml @@ -35,6 +35,58 @@ notes.xml
+
Public_Key 1.15 + +
Fixed Bugs and Malfunctions + + +

+ ssl application will validate id-kp-serverAuth and + id-kp-clientAuth extended key usage only in end entity + certificates. public_key application will disallow + "anyExtendedKeyUsage" for CA certificates that includes + the extended key usage extension and marks it critical.

+

+ Own Id: OTP-18739

+ + +

+ Modernize ECC handling so that crypto FIPS support works + as expected.

+

+ Own Id: OTP-18854

+ + +
+ + +
Improvements and New Features + + +

+ Support certificate policies in path_validation - as + described by RFC 5280.

+

+ Own Id: OTP-17844 Aux Id: ERIERL-738

+ + +

+ Add more search paths for cacerts on Illumos.

+

+ Own Id: OTP-18814 Aux Id: PR-7435

+ + +

+ Make it possible to handle invalid date formats in the + verify_fun for pkix_path_validation/3

+

+ Own Id: OTP-18867 Aux Id: GH-7515

+ + +
+ +
+
Public_Key 1.14.1
Fixed Bugs and Malfunctions @@ -77,6 +129,25 @@
+
Public_Key 1.13.3.2 + +
Fixed Bugs and Malfunctions + + +

+ ssl application will validate id-kp-serverAuth and + id-kp-clientAuth extended key usage only in end entity + certificates. public_key application will disallow + "anyExtendedKeyUsage" for CA certificates that includes + the extended key usage extension and marks it critical.

+

+ Own Id: OTP-18739

+ + +
+ +
+
Public_Key 1.13.3.1
Fixed Bugs and Malfunctions diff --git a/lib/public_key/doc/src/public_key.xml b/lib/public_key/doc/src/public_key.xml index 5fc2420fab3c..3e02049bb216 100644 --- a/lib/public_key/doc/src/public_key.xml +++ b/lib/public_key/doc/src/public_key.xml @@ -5,7 +5,7 @@
2008 - 2022 + 2023 Ericsson AB, All Rights Reserved @@ -145,6 +145,14 @@ + + + +

Can be provided together with a custom private key, that specifies a key fun, + to provide additional options understood by the fun.

+ + + @@ -390,9 +398,14 @@ Public-key encryption using the private key. -

Public-key encryption using the private key. - See also crypto:private_encrypt/4.

+

Public-key encryption using the private key. See also crypto:private_encrypt/4. + The key, can besides a standard RSA key, be a map specifing the + key algorithm rsa and a fun to handle the encryption + operation. This may be used for customized the encryption + operation with for instance hardware security modules (HSM) or + trusted platform modules (TPM). +

@@ -1006,7 +1019,11 @@ end

Creates a digital signature.

The Msg is either the binary "plain text" data to be signed or it is the hashed value of "plain text", that is, the - digest.

+ digest. The key, can besides a standard key, be a map specifing + a key algorithm and a fun that should handle the signing. This may + be used for customized signing with for instance hardware security + modules (HSM) or trusted platform modules (TPM). +

diff --git a/lib/public_key/doc/src/public_key_app.xml b/lib/public_key/doc/src/public_key_app.xml index bc150b6ab0b0..4bf7fc0ebcd4 100644 --- a/lib/public_key/doc/src/public_key_app.xml +++ b/lib/public_key/doc/src/public_key_app.xml @@ -4,7 +4,7 @@
- 20162020 + 20162023 Ericsson AB. All Rights Reserved. diff --git a/lib/public_key/include/public_key.hrl b/lib/public_key/include/public_key.hrl index 1ebe541e3a51..6670e0e524bf 100644 --- a/lib/public_key/include/public_key.hrl +++ b/lib/public_key/include/public_key.hrl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2021. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/public_key/src/Makefile b/lib/public_key/src/Makefile index a2444c64d80b..52d93a9ff796 100644 --- a/lib/public_key/src/Makefile +++ b/lib/public_key/src/Makefile @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2008-2022. All Rights Reserved. +# Copyright Ericsson AB 2008-2023. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/lib/public_key/src/public_key.erl b/lib/public_key/src/public_key.erl index 858860e29c17..2cd045bd24b1 100644 --- a/lib/public_key/src/public_key.erl +++ b/lib/public_key/src/public_key.erl @@ -112,10 +112,22 @@ test_root_cert/0 ]). --type public_key() :: rsa_public_key() | rsa_pss_public_key() | dsa_public_key() | ec_public_key() | ed_public_key() . --type private_key() :: rsa_private_key() | rsa_pss_private_key() | dsa_private_key() | ec_private_key() | ed_private_key() . +-type public_key() :: rsa_public_key() | + rsa_pss_public_key() | + dsa_public_key() | + ec_public_key() | + ed_public_key() . +-type private_key() :: rsa_private_key() | + rsa_pss_private_key() | + dsa_private_key() | + ec_private_key() | + ed_private_key() | + #{algorithm := eddsa | rsa_pss_pss | ecdsa | rsa | dsa, + sign_fun => fun()} . +-type custom_key_opts() :: [term()]. -type rsa_public_key() :: #'RSAPublicKey'{}. --type rsa_private_key() :: #'RSAPrivateKey'{}. +-type rsa_private_key() :: #'RSAPrivateKey'{} | #{algorithm := rsa, + encrypt_fun => fun()}. -type dss_public_key() :: integer(). -type rsa_pss_public_key() :: {rsa_pss_public_key(), #'RSASSA-PSS-params'{}}. -type rsa_pss_private_key() :: { #'RSAPrivateKey'{}, #'RSASSA-PSS-params'{}}. @@ -635,16 +647,16 @@ encrypt_private(PlainText, Key) -> CipherText when PlainText :: binary(), Key :: rsa_private_key(), - Options :: crypto:pk_encrypt_decrypt_opts(), + Options :: crypto:pk_encrypt_decrypt_opts() | custom_key_opts(), CipherText :: binary() . -encrypt_private(PlainText, - #'RSAPrivateKey'{modulus = N, publicExponent = E, - privateExponent = D} = Key, - Options) +encrypt_private(PlainText, Key, Options) when is_binary(PlainText), - is_integer(N), is_integer(E), is_integer(D), is_list(Options) -> - crypto:private_encrypt(rsa, PlainText, format_rsa_private_key(Key), default_options(Options)). + Opts = default_options(Options), + case format_sign_key(Key) of + {extern, Fun} -> Fun(PlainText, Opts); + {rsa, CryptoKey} -> crypto:private_encrypt(rsa, PlainText, CryptoKey, Opts) + end. %%-------------------------------------------------------------------- %% Description: List available group sizes among the pre-computed dh groups @@ -831,7 +843,7 @@ sign(DigestOrPlainText, DigestType, Key) -> Signature when Msg :: binary() | {digest,binary()}, DigestType :: digest_type(), Key :: private_key(), - Options :: crypto:pk_sign_verify_opts(), + Options :: crypto:pk_sign_verify_opts() | custom_key_opts(), Signature :: binary() . sign(Digest, none, Key = #'DSAPrivateKey'{}, Options) when is_binary(Digest) -> %% Backwards compatible @@ -840,6 +852,8 @@ sign(DigestOrPlainText, DigestType, Key, Options) -> case format_sign_key(Key) of badarg -> erlang:error(badarg, [DigestOrPlainText, DigestType, Key, Options]); + {extern, Fun} when is_function(Fun) -> + Fun(DigestOrPlainText, DigestType, Options); {Algorithm, CryptoKey} -> try crypto:sign(Algorithm, DigestType, DigestOrPlainText, CryptoKey, Options) catch %% Compatible with old error schema @@ -1505,8 +1519,17 @@ format_pkix_sign_key({#'RSAPrivateKey'{} = Key, _}) -> Key; format_pkix_sign_key(Key) -> Key. + +format_sign_key(#{encrypt_fun := KeyFun}) -> + {extern, KeyFun}; +format_sign_key(#{sign_fun := KeyFun}) -> + {extern, KeyFun}; format_sign_key(Key = #'RSAPrivateKey'{}) -> {rsa, format_rsa_private_key(Key)}; +format_sign_key({#'RSAPrivateKey'{} = Key, _}) -> + %% Params are handled in options arg + %% provided by caller. + {rsa, format_rsa_private_key(Key)}; format_sign_key(#'DSAPrivateKey'{p = P, q = Q, g = G, x = X}) -> {dss, [P, Q, G, X]}; format_sign_key(#'ECPrivateKey'{privateKey = PrivKey, parameters = {namedCurve, Curve} = Param}) diff --git a/lib/public_key/test/Makefile b/lib/public_key/test/Makefile index 239e078193fe..79dcacb6ce09 100644 --- a/lib/public_key/test/Makefile +++ b/lib/public_key/test/Makefile @@ -1,7 +1,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2008-2022. All Rights Reserved. +# Copyright Ericsson AB 2008-2023. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/lib/public_key/test/erl_make_certs.erl b/lib/public_key/test/erl_make_certs.erl index 6a91b5ec117c..24b04ec2e1a2 100644 --- a/lib/public_key/test/erl_make_certs.erl +++ b/lib/public_key/test/erl_make_certs.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2011-2021. All Rights Reserved. +%% Copyright Ericsson AB 2011-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/public_key/test/public_key_SUITE.erl b/lib/public_key/test/public_key_SUITE.erl index 0c8cf07fb5a2..ee1ed7b56749 100644 --- a/lib/public_key/test/public_key_SUITE.erl +++ b/lib/public_key/test/public_key_SUITE.erl @@ -77,12 +77,16 @@ cert_pem/1, encrypt_decrypt/0, encrypt_decrypt/1, + encrypt_decrypt_sign_fun/0, + encrypt_decrypt_sign_fun/1, rsa_sign_verify/0, rsa_sign_verify/1, rsa_pss_sign_verify/0, rsa_pss_sign_verify/1, dsa_sign_verify/0, dsa_sign_verify/1, + custom_sign_fun_verify/0, + custom_sign_fun_verify/1, pkix/0, pkix/1, pkix_countryname/0, @@ -153,6 +157,7 @@ all() -> appup, {group, pem_decode_encode}, encrypt_decrypt, + encrypt_decrypt_sign_fun, {group, sign_verify}, pkix, pkix_countryname, @@ -190,7 +195,7 @@ groups() -> ec_pem_encode_generated, gen_ec_param_prime_field, gen_ec_param_char_2_field]}, {sign_verify, [], [rsa_sign_verify, rsa_pss_sign_verify, dsa_sign_verify, - eddsa_sign_verify_24_compat]} + eddsa_sign_verify_24_compat, custom_sign_fun_verify]} ]. %%------------------------------------------------------------------- init_per_suite(Config) -> @@ -655,6 +660,22 @@ encrypt_decrypt(Config) when is_list(Config) -> RsaEncrypted2 = public_key:encrypt_public(Msg, PublicKey), Msg = public_key:decrypt_private(RsaEncrypted2, PrivateKey), ok. + +%%-------------------------------------------------------------------- +encrypt_decrypt_sign_fun() -> + [{doc, "Test public_key:encrypt_private with user provided sign_fun"}]. +encrypt_decrypt_sign_fun(Config) when is_list(Config) -> + {PrivateKey, _DerKey} = erl_make_certs:gen_rsa(64), + #'RSAPrivateKey'{modulus=Mod, publicExponent=Exp} = PrivateKey, + EncryptFun = fun (PlainText, Options) -> + public_key:encrypt_private(PlainText, PrivateKey, Options) + end, + CustomPrivKey = #{encrypt_fun => EncryptFun}, + PublicKey = #'RSAPublicKey'{modulus=Mod, publicExponent=Exp}, + Msg = list_to_binary(lists:duplicate(5, "Foo bar 100")), + RsaEncrypted = public_key:encrypt_private(Msg, CustomPrivKey), + Msg = public_key:decrypt_public(RsaEncrypted, PublicKey), + ok. %%-------------------------------------------------------------------- rsa_sign_verify() -> @@ -731,6 +752,28 @@ dsa_sign_verify(Config) when is_list(Config) -> {DSAPublicKey, DSAParams}), false = public_key:verify(Digest, none, <<1:8, DigestSign/binary>>, {DSAPublicKey, DSAParams}). +%%-------------------------------------------------------------------- + +custom_sign_fun_verify() -> + [{doc, "Checks that public_key:sign correctly calls the `sign_fun`"}]. +custom_sign_fun_verify(Config) when is_list(Config) -> + {_, CaKey} = erl_make_certs:make_cert([{key, rsa}]), + PrivateRSA = public_key:pem_entry_decode(CaKey), + #'RSAPrivateKey'{modulus=Mod, publicExponent=Exp} = PrivateRSA, + PublicRSA = #'RSAPublicKey'{modulus=Mod, publicExponent=Exp}, + SignFun = fun (Msg, HashAlgo, Options) -> + public_key:sign(Msg, HashAlgo, PrivateRSA, Options) + end, + CustomKey = #{algorithm => rsa, sign_fun => SignFun}, + + Msg = list_to_binary(lists:duplicate(5, "Foo bar 100")), + RSASign = public_key:sign(Msg, sha, CustomKey), + true = public_key:verify(Msg, sha, RSASign, PublicRSA), + false = public_key:verify(<<1:8, Msg/binary>>, sha, RSASign, PublicRSA), + false = public_key:verify(Msg, sha, <<1:8, RSASign/binary>>, PublicRSA), + + RSASign1 = public_key:sign(Msg, md5, CustomKey), + true = public_key:verify(Msg, md5, RSASign1, PublicRSA). %%-------------------------------------------------------------------- pkix() -> diff --git a/lib/public_key/vsn.mk b/lib/public_key/vsn.mk index 90a474ba49dc..e35d04dffcb1 100644 --- a/lib/public_key/vsn.mk +++ b/lib/public_key/vsn.mk @@ -1 +1 @@ -PUBLIC_KEY_VSN = 1.14.1 +PUBLIC_KEY_VSN = 1.15 diff --git a/lib/reltool/doc/src/reltool_examples.xml b/lib/reltool/doc/src/reltool_examples.xml index a29403da7800..43ab8caadac7 100644 --- a/lib/reltool/doc/src/reltool_examples.xml +++ b/lib/reltool/doc/src/reltool_examples.xml @@ -420,7 +420,7 @@ Eshell V10.0 (abort with ^G) {write_file,"start.boot",<<131,104,3,119,...>>}]}, {copy_file,"Install"}, {create_dir,"misc", - [{copy_file,"makewhatis"},{copy_file,"format_man_pages"}]}, + [{copy_file,"format_man_pages"}]}, {create_dir,"usr", [{create_dir,"lib", [{copy_file,"liberl_interface_st.a"}, diff --git a/lib/runtime_tools/c_src/trace_ip_drv.c b/lib/runtime_tools/c_src/trace_ip_drv.c index 1774657d1001..61d6bff51bce 100644 --- a/lib/runtime_tools/c_src/trace_ip_drv.c +++ b/lib/runtime_tools/c_src/trace_ip_drv.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 1999-2022. All Rights Reserved. + * Copyright Ericsson AB 1999-2023. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/lib/runtime_tools/doc/src/notes.xml b/lib/runtime_tools/doc/src/notes.xml index 7e25c3c5cc4d..f0074291831d 100644 --- a/lib/runtime_tools/doc/src/notes.xml +++ b/lib/runtime_tools/doc/src/notes.xml @@ -32,6 +32,28 @@

This document describes the changes made to the Runtime_Tools application.

+
Runtime_Tools 2.0.1 + +
Fixed Bugs and Malfunctions + + +

+ Fixed issue with fetching port information for observer + could crash if port had died.

+

+ Own Id: OTP-18868 Aux Id: GH-7735

+ + +

+ Fixed some benign compile warnings on Windows.

+

+ Own Id: OTP-18895

+ + +
+ +
+
Runtime_Tools 2.0
Fixed Bugs and Malfunctions diff --git a/lib/runtime_tools/src/observer_backend.erl b/lib/runtime_tools/src/observer_backend.erl index 65440a09113e..7df0e41c357f 100644 --- a/lib/runtime_tools/src/observer_backend.erl +++ b/lib/runtime_tools/src/observer_backend.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2002-2022. All Rights Reserved. +%% Copyright Ericsson AB 2002-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/runtime_tools/src/runtime_tools.app.src b/lib/runtime_tools/src/runtime_tools.app.src index 10309fae0abf..d3be1a5efc29 100644 --- a/lib/runtime_tools/src/runtime_tools.app.src +++ b/lib/runtime_tools/src/runtime_tools.app.src @@ -30,4 +30,4 @@ {env, []}, {mod, {runtime_tools, []}}, {runtime_dependencies, ["stdlib-@OTP-18789@","mnesia-4.12","kernel-8.1", - "erts-@OTP-18765@"]}]}. + "erts-14.2"]}]}. diff --git a/lib/runtime_tools/vsn.mk b/lib/runtime_tools/vsn.mk index 6a374f92d477..170bce025d74 100644 --- a/lib/runtime_tools/vsn.mk +++ b/lib/runtime_tools/vsn.mk @@ -1 +1 @@ -RUNTIME_TOOLS_VSN = 2.0 +RUNTIME_TOOLS_VSN = 2.0.1 diff --git a/lib/ssh/doc/src/notes.xml b/lib/ssh/doc/src/notes.xml index 1e9ea82fb60b..533daa43cdc3 100644 --- a/lib/ssh/doc/src/notes.xml +++ b/lib/ssh/doc/src/notes.xml @@ -30,6 +30,80 @@ notes.xml
+
Ssh 5.1.1 + +
Fixed Bugs and Malfunctions + + +

+ With this change (being response to CVE-2023-48795), ssh + can negotiate "strict KEX" OpenSSH extension with peers + supporting it; also 'chacha20-poly1305@openssh.com' + algorithm becomes a less preferred cipher.

+

+ If strict KEX availability cannot be ensured on both + connection sides, affected encryption modes(CHACHA and + CBC) can be disabled with standard ssh configuration. + This will provide protection against vulnerability, but + at a cost of affecting interoperability. See Configuring algorithms in + SSH.

+

+ *** POTENTIAL INCOMPATIBILITY ***

+

+ Own Id: OTP-18897

+ + +
+ +
+ +
Ssh 5.1 + +
Fixed Bugs and Malfunctions + + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +

+ Avoid outputting ansi escape sequences to dumb ssh + clients.

+

+ Own Id: OTP-18861 Aux Id: PR-7627

+ + +

+ With this change, connection handler does not execute + socket operations until it becomes socket owner. + Previously errors could occur if connection handler tried + to work with socket whose owner exited.

+

+ Own Id: OTP-18869 Aux Id: PR-7849,GH-7571

+ + +
+ + +
Improvements and New Features + + +

+ With this change, reverse search works with ssh shell and + non dumb terminals.

+

+ Own Id: OTP-18730 Aux Id: PR-7499

+ + +
+ +
+
Ssh 5.0.1
Fixed Bugs and Malfunctions @@ -90,6 +164,43 @@
+
Ssh 4.15.3.1 + +
Fixed Bugs and Malfunctions + + +

+ With this change, connection handler does not execute + socket operations until it becomes socket owner. + Previously errors could occur if connection handler tried + to work with socket whose owner exited.

+

+ Own Id: OTP-18869 Aux Id: PR-7849,GH-7571

+ + +

+ With this change (being response to CVE-2023-48795), ssh + can negotiate "strict KEX" OpenSSH extension with peers + supporting it; also 'chacha20-poly1305@openssh.com' + algorithm becomes a less preferred cipher.

+

+ If strict KEX availability cannot be ensured on both + connection sides, affected encryption modes(CHACHA and + CBC) can be disabled with standard ssh configuration. + This will provide protection against vulnerability, but + at a cost of affecting interoperability. See Configuring algorithms in + SSH.

+

+ *** POTENTIAL INCOMPATIBILITY ***

+

+ Own Id: OTP-18897

+ + +
+ +
+
Ssh 4.15.3
Fixed Bugs and Malfunctions @@ -286,6 +397,43 @@
+
Ssh 4.13.2.4 + +
Fixed Bugs and Malfunctions + + +

+ With this change, connection handler does not execute + socket operations until it becomes socket owner. + Previously errors could occur if connection handler tried + to work with socket whose owner exited.

+

+ Own Id: OTP-18869 Aux Id: PR-7849,GH-7571

+ + +

+ With this change (being response to CVE-2023-48795), ssh + can negotiate "strict KEX" OpenSSH extension with peers + supporting it; also 'chacha20-poly1305@openssh.com' + algorithm becomes a less preferred cipher.

+

+ If strict KEX availability cannot be ensured on both + connection sides, affected encryption modes(CHACHA and + CBC) can be disabled with standard ssh configuration. + This will provide protection against vulnerability, but + at a cost of affecting interoperability. See Configuring algorithms in + SSH.

+

+ *** POTENTIAL INCOMPATIBILITY ***

+

+ Own Id: OTP-18897

+ + +
+ +
+
Ssh 4.13.2.3
Fixed Bugs and Malfunctions diff --git a/lib/ssh/doc/src/ssh.xml b/lib/ssh/doc/src/ssh.xml index bc96e13a8aee..7d3dfe2a5125 100644 --- a/lib/ssh/doc/src/ssh.xml +++ b/lib/ssh/doc/src/ssh.xml @@ -4,7 +4,7 @@
- 20042022 + 20042023 Ericsson AB. All Rights Reserved. diff --git a/lib/ssh/doc/src/ssh_connection.xml b/lib/ssh/doc/src/ssh_connection.xml index 4ecbfae02d09..677f8faad20e 100644 --- a/lib/ssh/doc/src/ssh_connection.xml +++ b/lib/ssh/doc/src/ssh_connection.xml @@ -5,7 +5,7 @@
2008 - 2022 + 2023 Ericsson AB, All Rights Reserved diff --git a/lib/ssh/doc/src/ssh_sftp.xml b/lib/ssh/doc/src/ssh_sftp.xml index 60752054dc08..08aeee67965a 100644 --- a/lib/ssh/doc/src/ssh_sftp.xml +++ b/lib/ssh/doc/src/ssh_sftp.xml @@ -4,7 +4,7 @@
- 20052022 + 20052023 Ericsson AB. All Rights Reserved. diff --git a/lib/ssh/src/ssh.hrl b/lib/ssh/src/ssh.hrl index aecce6c1efea..2cb9f3115cbe 100644 --- a/lib/ssh/src/ssh.hrl +++ b/lib/ssh/src/ssh.hrl @@ -443,6 +443,8 @@ send_ext_info, %% May send ext-info to peer recv_ext_info, %% Expect ext-info from peer + kex_strict_negotiated = false, + algorithms, %% #alg{} send_mac = none, %% send MAC algorithm @@ -514,7 +516,8 @@ c_lng, s_lng, send_ext_info, - recv_ext_info + recv_ext_info, + kex_strict_negotiated = false }). -record(ssh_pty, {c_version = "", % client version string, e.g "SSH-2.0-Erlang/4.10.5" diff --git a/lib/ssh/src/ssh_cli.erl b/lib/ssh/src/ssh_cli.erl index f3027b083af4..fbcec8e8c10b 100644 --- a/lib/ssh/src/ssh_cli.erl +++ b/lib/ssh/src/ssh_cli.erl @@ -431,8 +431,8 @@ io_request({put_chars, Cs}, Buf, Tty, _Group) -> put_chars(bin_to_list(Cs), Buf, Tty); io_request({put_chars, unicode, Cs}, Buf, Tty, _Group) -> put_chars(unicode:characters_to_list(Cs,unicode), Buf, Tty); -io_request({put_expand_no_trim, unicode, Expand}, Buf, Tty, _Group) -> - insert_chars(unicode:characters_to_list(Expand, unicode), Buf, Tty); +io_request({put_expand, unicode, Expand, _N}, Buf, Tty, _Group) -> + insert_chars(unicode:characters_to_list("\n"++Expand, unicode), Buf, Tty); io_request({insert_chars, Cs}, Buf, Tty, _Group) -> insert_chars(bin_to_list(Cs), Buf, Tty); io_request({insert_chars, unicode, Cs}, Buf, Tty, _Group) -> diff --git a/lib/ssh/src/ssh_connection_handler.erl b/lib/ssh/src/ssh_connection_handler.erl index 30719e56ae81..2648c5256bcc 100644 --- a/lib/ssh/src/ssh_connection_handler.erl +++ b/lib/ssh/src/ssh_connection_handler.erl @@ -706,6 +706,16 @@ handle_event(internal, #ssh_msg_disconnect{description=Desc} = Msg, StateName, D disconnect_fun("Received disconnect: "++Desc, D), {stop_and_reply, {shutdown,Desc}, Actions, D}; +handle_event(internal, #ssh_msg_ignore{}, {_StateName, _Role, init}, + #data{ssh_params = #ssh{kex_strict_negotiated = true, + send_sequence = SendSeq, + recv_sequence = RecvSeq}}) -> + ?DISCONNECT(?SSH_DISCONNECT_KEY_EXCHANGE_FAILED, + io_lib:format("strict KEX violation: unexpected SSH_MSG_IGNORE " + "send_sequence = ~p recv_sequence = ~p", + [SendSeq, RecvSeq]) + ); + handle_event(internal, #ssh_msg_ignore{}, _StateName, _) -> keep_state_and_data; diff --git a/lib/ssh/src/ssh_fsm_kexinit.erl b/lib/ssh/src/ssh_fsm_kexinit.erl index 6ac4ec798f21..05f7bdf22f16 100644 --- a/lib/ssh/src/ssh_fsm_kexinit.erl +++ b/lib/ssh/src/ssh_fsm_kexinit.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2021. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -58,7 +58,7 @@ callback_mode() -> handle_event(internal, {#ssh_msg_kexinit{}=Kex, Payload}, {kexinit,Role,ReNeg}, D = #data{key_exchange_init_msg = OwnKex}) -> Ssh1 = ssh_transport:key_init(peer_role(Role), D#data.ssh_params, Payload), - Ssh = case ssh_transport:handle_kexinit_msg(Kex, OwnKex, Ssh1) of + Ssh = case ssh_transport:handle_kexinit_msg(Kex, OwnKex, Ssh1, ReNeg) of {ok, NextKexMsg, Ssh2} when Role==client -> ssh_connection_handler:send_bytes(NextKexMsg, D), Ssh2; diff --git a/lib/ssh/src/ssh_system_sup.erl b/lib/ssh/src/ssh_system_sup.erl index 400f7c441f06..3e7c6144232b 100644 --- a/lib/ssh/src/ssh_system_sup.erl +++ b/lib/ssh/src/ssh_system_sup.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2022. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/ssh/src/ssh_transport.erl b/lib/ssh/src/ssh_transport.erl index c0b46b338bed..7adf8daa95a7 100644 --- a/lib/ssh/src/ssh_transport.erl +++ b/lib/ssh/src/ssh_transport.erl @@ -42,7 +42,7 @@ key_exchange_init_msg/1, key_init/3, new_keys_message/1, ext_info_message/1, - handle_kexinit_msg/3, handle_kexdh_init/2, + handle_kexinit_msg/4, handle_kexdh_init/2, handle_kex_dh_gex_group/2, handle_kex_dh_gex_init/2, handle_kex_dh_gex_reply/2, handle_new_keys/2, handle_kex_dh_gex_request/2, handle_kexdh_reply/2, @@ -236,7 +236,6 @@ supported_algorithms(cipher) -> same( select_crypto_supported( [ - {'chacha20-poly1305@openssh.com', [{ciphers,chacha20}, {macs,poly1305}]}, {'aes256-gcm@openssh.com', [{ciphers,aes_256_gcm}]}, {'aes256-ctr', [{ciphers,aes_256_ctr}]}, {'aes192-ctr', [{ciphers,aes_192_ctr}]}, @@ -244,6 +243,7 @@ supported_algorithms(cipher) -> {'aes128-ctr', [{ciphers,aes_128_ctr}]}, {'AEAD_AES_256_GCM', [{ciphers,aes_256_gcm}]}, {'AEAD_AES_128_GCM', [{ciphers,aes_128_gcm}]}, + {'chacha20-poly1305@openssh.com', [{ciphers,chacha20}, {macs,poly1305}]}, {'aes256-cbc', [{ciphers,aes_256_cbc}]}, {'aes192-cbc', [{ciphers,aes_192_cbc}]}, {'aes128-cbc', [{ciphers,aes_128_cbc}]}, @@ -359,7 +359,8 @@ kexinit_message(Role, Random, Algs, HostKeyAlgs, Opts) -> #ssh_msg_kexinit{ cookie = Random, kex_algorithms = to_strings( get_algs(kex,Algs) ) - ++ kex_ext_info(Role,Opts), + ++ kex_ext_info(Role,Opts) + ++ kex_strict_alg(Role), server_host_key_algorithms = HostKeyAlgs, encryption_algorithms_client_to_server = c2s(cipher,Algs), encryption_algorithms_server_to_client = s2c(cipher,Algs), @@ -388,10 +389,12 @@ new_keys_message(Ssh0) -> handle_kexinit_msg(#ssh_msg_kexinit{} = CounterPart, #ssh_msg_kexinit{} = Own, - #ssh{role = client} = Ssh) -> + #ssh{role = client} = Ssh, ReNeg) -> try - {ok, Algorithms} = select_algorithm(client, Own, CounterPart, Ssh#ssh.opts), + {ok, Algorithms} = + select_algorithm(client, Own, CounterPart, Ssh, ReNeg), true = verify_algorithm(Algorithms), + true = verify_kexinit_is_first_msg(Algorithms, Ssh, ReNeg), Algorithms of Algos -> @@ -404,10 +407,12 @@ handle_kexinit_msg(#ssh_msg_kexinit{} = CounterPart, #ssh_msg_kexinit{} = Own, end; handle_kexinit_msg(#ssh_msg_kexinit{} = CounterPart, #ssh_msg_kexinit{} = Own, - #ssh{role = server} = Ssh) -> + #ssh{role = server} = Ssh, ReNeg) -> try - {ok, Algorithms} = select_algorithm(server, CounterPart, Own, Ssh#ssh.opts), + {ok, Algorithms} = + select_algorithm(server, CounterPart, Own, Ssh, ReNeg), true = verify_algorithm(Algorithms), + true = verify_kexinit_is_first_msg(Algorithms, Ssh, ReNeg), Algorithms of Algos -> @@ -488,6 +493,21 @@ verify_algorithm(#alg{kex = Kex}) -> false -> {false, "kex"} end. +verify_kexinit_is_first_msg(#alg{kex_strict_negotiated = false}, _, _) -> + true; +verify_kexinit_is_first_msg(#alg{kex_strict_negotiated = true}, _, renegotiate) -> + true; +verify_kexinit_is_first_msg(#alg{kex_strict_negotiated = true}, + #ssh{send_sequence = 1, recv_sequence = 1}, + init) -> + true; +verify_kexinit_is_first_msg(#alg{kex_strict_negotiated = true}, + #ssh{send_sequence = SendSequence, + recv_sequence = RecvSequence}, init) -> + error_logger:warning_report( + lists:concat(["KEX strict violation (", SendSequence, ", ", RecvSequence, ")."])), + {false, "kex_strict"}. + %%%---------------------------------------------------------------- %%% %%% Key exchange initialization @@ -867,6 +887,9 @@ handle_new_keys(#ssh_msg_newkeys{}, Ssh0) -> ) end. +%%%---------------------------------------------------------------- +kex_strict_alg(client) -> [?kex_strict_c]; +kex_strict_alg(server) -> [?kex_strict_s]. %%%---------------------------------------------------------------- kex_ext_info(Role, Opts) -> @@ -1057,7 +1080,35 @@ known_host_key(#ssh{opts = Opts, peer = {PeerName,{IP,Port}}} = Ssh, %% %% The first algorithm in each list MUST be the preferred (guessed) %% algorithm. Each string MUST contain at least one algorithm name. -select_algorithm(Role, Client, Server, Opts) -> +select_algorithm(Role, Client, Server, + #ssh{opts = Opts, + kex_strict_negotiated = KexStrictNegotiated0}, + ReNeg) -> + KexStrictNegotiated = + case ReNeg of + %% KEX strict negotiated once per connection + init -> + Result = + case Role of + server -> + lists:member(?kex_strict_c, + Client#ssh_msg_kexinit.kex_algorithms); + client -> + lists:member(?kex_strict_s, + Server#ssh_msg_kexinit.kex_algorithms) + end, + case Result of + true -> + error_logger:info_report( + lists:concat([Role, " will use strict KEX ordering"])); + _ -> + ok + end, + Result; + _ -> + KexStrictNegotiated0 + end, + {Encrypt0, Decrypt0} = select_encrypt_decrypt(Role, Client, Server), {SendMac0, RecvMac0} = select_send_recv_mac(Role, Client, Server), @@ -1108,7 +1159,8 @@ select_algorithm(Role, Client, Server, Opts) -> c_lng = C_Lng, s_lng = S_Lng, send_ext_info = SendExtInfo, - recv_ext_info = RecvExtInfo + recv_ext_info = RecvExtInfo, + kex_strict_negotiated = KexStrictNegotiated }}. @@ -1206,7 +1258,8 @@ alg_setup(snd, SSH) -> c_lng = ALG#alg.c_lng, s_lng = ALG#alg.s_lng, send_ext_info = ALG#alg.send_ext_info, - recv_ext_info = ALG#alg.recv_ext_info + recv_ext_info = ALG#alg.recv_ext_info, + kex_strict_negotiated = ALG#alg.kex_strict_negotiated }; alg_setup(rcv, SSH) -> @@ -1218,22 +1271,23 @@ alg_setup(rcv, SSH) -> c_lng = ALG#alg.c_lng, s_lng = ALG#alg.s_lng, send_ext_info = ALG#alg.send_ext_info, - recv_ext_info = ALG#alg.recv_ext_info + recv_ext_info = ALG#alg.recv_ext_info, + kex_strict_negotiated = ALG#alg.kex_strict_negotiated }. - -alg_init(snd, SSH0) -> +alg_init(Dir = snd, SSH0) -> {ok,SSH1} = send_mac_init(SSH0), {ok,SSH2} = encrypt_init(SSH1), {ok,SSH3} = compress_init(SSH2), - SSH3; + {ok,SSH4} = maybe_reset_sequence(Dir, SSH3), + SSH4; -alg_init(rcv, SSH0) -> +alg_init(Dir = rcv, SSH0) -> {ok,SSH1} = recv_mac_init(SSH0), {ok,SSH2} = decrypt_init(SSH1), {ok,SSH3} = decompress_init(SSH2), - SSH3. - + {ok,SSH4} = maybe_reset_sequence(Dir, SSH3), + SSH4. alg_final(snd, SSH0) -> {ok,SSH1} = send_mac_final(SSH0), @@ -2198,6 +2252,14 @@ crypto_name_supported(Tag, CryptoName, Supported) -> same(Algs) -> [{client2server,Algs}, {server2client,Algs}]. +maybe_reset_sequence(snd, Ssh = #ssh{kex_strict_negotiated = true}) -> + {ok, Ssh#ssh{send_sequence = 0}}; +maybe_reset_sequence(rcv, Ssh = #ssh{kex_strict_negotiated = true}) -> + {ok, Ssh#ssh{recv_sequence = 0}}; +maybe_reset_sequence(_Dir, Ssh) -> + {ok, Ssh}. + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% %% Other utils @@ -2224,14 +2286,14 @@ ssh_dbg_flags(raw_messages) -> ssh_dbg_flags(hello); ssh_dbg_flags(ssh_messages) -> ssh_dbg_flags(hello). -ssh_dbg_on(alg) -> dbg:tpl(?MODULE,select_algorithm,4,x); +ssh_dbg_on(alg) -> dbg:tpl(?MODULE,select_algorithm,5,x); ssh_dbg_on(hello) -> dbg:tp(?MODULE,hello_version_msg,1,x), dbg:tp(?MODULE,handle_hello_version,1,x); ssh_dbg_on(raw_messages) -> ssh_dbg_on(hello); ssh_dbg_on(ssh_messages) -> ssh_dbg_on(hello). -ssh_dbg_off(alg) -> dbg:ctpl(?MODULE,select_algorithm,4); +ssh_dbg_off(alg) -> dbg:ctpl(?MODULE,select_algorithm,5); ssh_dbg_off(hello) -> dbg:ctpg(?MODULE,hello_version_msg,1), dbg:ctpg(?MODULE,handle_hello_version,1); ssh_dbg_off(raw_messages) -> ssh_dbg_off(hello); @@ -2254,9 +2316,9 @@ ssh_dbg_format(hello, {call,{?MODULE,handle_hello_version,[Hello]}}) -> ssh_dbg_format(hello, {return_from,{?MODULE,handle_hello_version,1},_Ret}) -> skip; -ssh_dbg_format(alg, {call,{?MODULE,select_algorithm,[_,_,_,_]}}) -> +ssh_dbg_format(alg, {call,{?MODULE,select_algorithm,[_,_,_,_,_]}}) -> skip; -ssh_dbg_format(alg, {return_from,{?MODULE,select_algorithm,4},{ok,Alg}}) -> +ssh_dbg_format(alg, {return_from,{?MODULE,select_algorithm,5},{ok,Alg}}) -> ["Negotiated algorithms:\n", wr_record(Alg) ]; diff --git a/lib/ssh/src/ssh_transport.hrl b/lib/ssh/src/ssh_transport.hrl index 009d85cb79e4..6fbf7d404c0a 100644 --- a/lib/ssh/src/ssh_transport.hrl +++ b/lib/ssh/src/ssh_transport.hrl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2022. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -266,5 +266,7 @@ -define(dh_group18, {2, 16#FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024E088A67CC74020BBEA63B139B22514A08798E3404DDEF9519B3CD3A431B302B0A6DF25F14374FE1356D6D51C245E485B576625E7EC6F44C42E9A637ED6B0BFF5CB6F406B7EDEE386BFB5A899FA5AE9F24117C4B1FE649286651ECE45B3DC2007CB8A163BF0598DA48361C55D39A69163FA8FD24CF5F83655D23DCA3AD961C62F356208552BB9ED529077096966D670C354E4ABC9804F1746C08CA18217C32905E462E36CE3BE39E772C180E86039B2783A2EC07A28FB5C55DF06F4C52C9DE2BCBF6955817183995497CEA956AE515D2261898FA051015728E5A8AAAC42DAD33170D04507A33A85521ABDF1CBA64ECFB850458DBEF0A8AEA71575D060C7DB3970F85A6E1E4C7ABF5AE8CDB0933D71E8C94E04A25619DCEE3D2261AD2EE6BF12FFA06D98A0864D87602733EC86A64521F2B18177B200CBBE117577A615D6C770988C0BAD946E208E24FA074E5AB3143DB5BFCE0FD108E4B82D120A92108011A723C12A787E6D788719A10BDBA5B2699C327186AF4E23C1A946834B6150BDA2583E9CA2AD44CE8DBBBC2DB04DE8EF92E8EFC141FBECAA6287C59474E6BC05D99B2964FA090C3A2233BA186515BE7ED1F612970CEE2D7AFB81BDD762170481CD0069127D5B05AA993B4EA988D8FDDC186FFB7DC90A6C08F4DF435C93402849236C3FAB4D27C7026C1D4DCB2602646DEC9751E763DBA37BDF8FF9406AD9E530EE5DB382F413001AEB06A53ED9027D831179727B0865A8918DA3EDBEBCF9B14ED44CE6CBACED4BB1BDB7F1447E6CC254B332051512BD7AF426FB8F401378CD2BF5983CA01C64B92ECF032EA15D1721D03F482D7CE6E74FEF6D55E702F46980C82B5A84031900B1C9E59E7C97FBEC7E8F323A97A7E36CC88BE0F1D45B7FF585AC54BD407B22B4154AACC8F6D7EBF48E1D814CC5ED20F8037E0A79715EEF29BE32806A1D58BB7C5DA76F550AA3D8A1FBFF0EB19CCB1A313D55CDA56C9EC2EF29632387FE8D76E3C0468043E8F663F4860EE12BF2D5B0B7474D6E694F91E6DBE115974A3926F12FEE5E438777CB6A932DF8CD8BEC4D073B931BA3BC832B68D9DD300741FA7BF8AFC47ED2576F6936BA424663AAB639C5AE4F5683423B4742BF1C978238F16CBE39D652DE3FDB8BEFC848AD922222E04A4037C0713EB57A81A23F0C73473FC646CEA306B4BCBC8862F8385DDFA9D4B7FA2C087E879683303ED5BDD3A062B3CF5B3A278A66D2A13F83F44F82DDF310EE074AB6A364597E899A0255DC164F31CC50846851DF9AB48195DED7EA1B1D510BD7EE74D73FAF36BC31ECFA268359046F4EB879F924009438B481C6CD7889A002ED5EE382BC9190DA6FC026E479558E4475677E9AA9E3050E2765694DFC81F56E880B96E7160C980DD98EDD3DFFFFFFFFFFFFFFFFF}). - +%%% OpenSSH KEX strict +-define(kex_strict_c, "kex-strict-c-v00@openssh.com"). +-define(kex_strict_s, "kex-strict-s-v00@openssh.com"). -endif. % -ifdef(ssh_transport). diff --git a/lib/ssh/test/ssh_connection_SUITE.erl b/lib/ssh/test/ssh_connection_SUITE.erl index 78123766240a..5e426cd8fcf3 100644 --- a/lib/ssh/test/ssh_connection_SUITE.erl +++ b/lib/ssh/test/ssh_connection_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2022. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/ssh/test/ssh_protocol_SUITE.erl b/lib/ssh/test/ssh_protocol_SUITE.erl index 94ba6c576efd..183a07c9d378 100644 --- a/lib/ssh/test/ssh_protocol_SUITE.erl +++ b/lib/ssh/test/ssh_protocol_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2022. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -54,6 +54,9 @@ empty_service_name/1, ext_info_c/1, ext_info_s/1, + kex_strict_negotiated/1, + kex_strict_msg_ignore/1, + kex_strict_msg_unknown/1, gex_client_init_option_groups/1, gex_client_init_option_groups_file/1, gex_client_init_option_groups_moduli_file/1, @@ -137,8 +140,10 @@ groups() -> gex_client_init_option_groups_moduli_file, gex_client_init_option_groups_file, gex_client_old_request_exact, - gex_client_old_request_noexact - ]}, + gex_client_old_request_noexact, + kex_strict_negotiated, + kex_strict_msg_ignore, + kex_strict_msg_unknown]}, {service_requests, [], [bad_service_name, bad_long_service_name, bad_very_long_service_name, @@ -165,17 +170,16 @@ groups() -> init_per_suite(Config) -> ?CHECK_CRYPTO(start_std_daemon( setup_dirs( start_apps(Config)))). - + end_per_suite(Config) -> stop_apps(Config). - - init_per_testcase(no_common_alg_server_disconnects, Config) -> start_std_daemon(Config, [{preferred_algorithms,[{public_key,['ssh-rsa']}, {cipher,?DEFAULT_CIPHERS} ]}]); - +init_per_testcase(kex_strict_negotiated, Config) -> + Config; init_per_testcase(TC, Config) when TC == gex_client_init_option_groups ; TC == gex_client_init_option_groups_moduli_file ; TC == gex_client_init_option_groups_file ; @@ -218,6 +222,8 @@ init_per_testcase(_TestCase, Config) -> end_per_testcase(no_common_alg_server_disconnects, Config) -> stop_std_daemon(Config); +end_per_testcase(kex_strict_negotiated, Config) -> + Config; end_per_testcase(TC, Config) when TC == gex_client_init_option_groups ; TC == gex_client_init_option_groups_moduli_file ; TC == gex_client_init_option_groups_file ; @@ -819,6 +825,80 @@ ext_info_c(Config) -> {result, Pid, Error} -> ct:fail("Error: ~p",[Error]) end. +%%%-------------------------------------------------------------------- +%%% +kex_strict_negotiated(Config0) -> + {ok,Pid} = ssh_test_lib:add_report_handler(), + Config = start_std_daemon(Config0, []), + {Server, Host, Port} = proplists:get_value(server, Config), + #{level := Level} = logger:get_primary_config(), + logger:set_primary_config(level, notice), + {ok, ConnRef} = std_connect({Host, Port}, Config, []), + {algorithms, A} = ssh:connection_info(ConnRef, algorithms), + ssh:stop_daemon(Server), + {ok, Reports} = ssh_test_lib:get_reports(Pid), + ct:log("Reports = ~p", [Reports]), + true = ssh_test_lib:kex_strict_negotiated(client, Reports), + true = ssh_test_lib:kex_strict_negotiated(server, Reports), + logger:set_primary_config(Level), + ok. + +%% Connect to an erlang server and inject unexpected SSH ignore +kex_strict_msg_ignore(Config) -> + ct:log("START: ~p~n=================================", [?FUNCTION_NAME]), + ExpectedReason = "strict KEX violation: unexpected SSH_MSG_IGNORE", + TestMessages = + [{send, ssh_msg_ignore}, + {match, #ssh_msg_kexdh_reply{_='_'}, receive_msg}, + {match, disconnect(?SSH_DISCONNECT_KEY_EXCHANGE_FAILED), receive_msg}], + kex_strict_helper(Config, TestMessages, ExpectedReason). + +%% Connect to an erlang server and inject unexpected non-SSH binary +kex_strict_msg_unknown(Config) -> + ct:log("START: ~p~n=================================", [?FUNCTION_NAME]), + ExpectedReason = "Bad packet: Size", + TestMessages = + [{send, ssh_msg_unknown}, + {match, #ssh_msg_kexdh_reply{_='_'}, receive_msg}, + {match, disconnect(?SSH_DISCONNECT_KEY_EXCHANGE_FAILED), receive_msg}], + kex_strict_helper(Config, TestMessages, ExpectedReason). + +kex_strict_helper(Config, TestMessages, ExpectedReason) -> + {ok,HandlerPid} = ssh_test_lib:add_report_handler(), + #{level := Level} = logger:get_primary_config(), + logger:set_primary_config(level, notice), + %% Connect and negotiate keys + {ok, InitialState} = ssh_trpt_test_lib:exec( + [{set_options, [print_ops, print_seqnums, print_messages]}] + ), + {ok, _AfterKexState} = + ssh_trpt_test_lib:exec( + [{connect, + server_host(Config),server_port(Config), + [{preferred_algorithms,[{kex,[?DEFAULT_KEX]}, + {cipher,?DEFAULT_CIPHERS} + ]}, + {silently_accept_hosts, true}, + {recv_ext_info, false}, + {user_dir, user_dir(Config)}, + {user_interaction, false} + | proplists:get_value(extra_options,Config,[]) + ]}, + receive_hello, + {send, hello}, + {send, ssh_msg_kexinit}, + {match, #ssh_msg_kexinit{_='_'}, receive_msg}, + {send, ssh_msg_kexdh_init}] ++ + TestMessages, + InitialState), + ct:sleep(100), + {ok, Reports} = ssh_test_lib:get_reports(HandlerPid), + ct:log("HandlerPid = ~p~nReports = ~p", [HandlerPid, Reports]), + true = ssh_test_lib:kex_strict_negotiated(client, Reports), + true = ssh_test_lib:kex_strict_negotiated(server, Reports), + true = ssh_test_lib:event_logged(server, Reports, ExpectedReason), + logger:set_primary_config(Level), + ok. %%%---------------------------------------------------------------- %%% @@ -840,7 +920,7 @@ modify_append(Config) -> Ciphers = filter_supported(cipher, ?CIPHERS), {ok,_} = chk_pref_algs(Config, - [?DEFAULT_KEX, ?EXTRA_KEX], + [?DEFAULT_KEX, ?EXTRA_KEX, list_to_atom(?kex_strict_s)], Ciphers, [{preferred_algorithms, [{kex,[?DEFAULT_KEX]}, {cipher,Ciphers} @@ -854,7 +934,7 @@ modify_prepend(Config) -> Ciphers = filter_supported(cipher, ?CIPHERS), {ok,_} = chk_pref_algs(Config, - [?EXTRA_KEX, ?DEFAULT_KEX], + [?EXTRA_KEX, ?DEFAULT_KEX, list_to_atom(?kex_strict_s)], Ciphers, [{preferred_algorithms, [{kex,[?DEFAULT_KEX]}, {cipher,Ciphers} @@ -868,7 +948,7 @@ modify_rm(Config) -> Ciphers = filter_supported(cipher, ?CIPHERS), {ok,_} = chk_pref_algs(Config, - [?DEFAULT_KEX], + [?DEFAULT_KEX, list_to_atom(?kex_strict_s)], tl(Ciphers), [{preferred_algorithms, [{kex,[?DEFAULT_KEX,?EXTRA_KEX]}, {cipher,Ciphers} @@ -887,7 +967,7 @@ modify_combo(Config) -> LastC = lists:last(Ciphers), {ok,_} = chk_pref_algs(Config, - [?DEFAULT_KEX], + [?DEFAULT_KEX, list_to_atom(?kex_strict_s)], [LastC] ++ (tl(Ciphers)--[LastC]) ++ [hd(Ciphers)], [{preferred_algorithms, [{kex,[?DEFAULT_KEX,?EXTRA_KEX]}, {cipher,Ciphers} diff --git a/lib/ssh/test/ssh_sup_SUITE.erl b/lib/ssh/test/ssh_sup_SUITE.erl index 6dc592b94251..ebbe10ab189c 100644 --- a/lib/ssh/test/ssh_sup_SUITE.erl +++ b/lib/ssh/test/ssh_sup_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2015-2021. All Rights Reserved. +%% Copyright Ericsson AB 2015-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/ssh/test/ssh_test_lib.erl b/lib/ssh/test/ssh_test_lib.erl index faa350423dca..f7355ba6a453 100644 --- a/lib/ssh/test/ssh_test_lib.erl +++ b/lib/ssh/test/ssh_test_lib.erl @@ -121,7 +121,11 @@ setup_host_key_create_dir/3, setup_host_key/3, setup_known_host/3, get_addr_str/0, -file_base_name/2 +file_base_name/2, +add_report_handler/0, +get_reports/1, +kex_strict_negotiated/2, +event_logged/3 ]). -include_lib("common_test/include/ct.hrl"). @@ -1267,3 +1271,49 @@ file_base_name(system_src, 'ecdsa-sha2-nistp521') -> "ssh_host_ecdsa_key521"; file_base_name(system_src, Alg) -> file_base_name(system, Alg). %%%---------------------------------------------------------------- +add_report_handler() -> + ssh_eqc_event_handler:add_report_handler(). + +get_reports(Pid) -> + ssh_eqc_event_handler:get_reports(Pid). + +-define(SEARCH_FUN(EXP), + begin + fun({info_report, _, {_, std_info, EXP}}) -> + true; + (_) -> + false + end + end). +-define(SEARCH_SUFFIX, " will use strict KEX ordering"). + +kex_strict_negotiated(client, Reports) -> + kex_strict_negotiated(?SEARCH_FUN("client" ++ ?SEARCH_SUFFIX), Reports); +kex_strict_negotiated(server, Reports) -> + kex_strict_negotiated(?SEARCH_FUN("server" ++ ?SEARCH_SUFFIX), Reports); +kex_strict_negotiated(SearchFun, Reports) when is_function(SearchFun) -> + case lists:search(SearchFun, Reports) of + {value, _} -> true; + _ -> false + end. + +event_logged(Role, Reports, Reason) -> + SearchF = + fun({info_msg, _, {_, _Format, Args}}) -> + AnyF = fun (E) when is_list(E) -> + case string:find(E, Reason) of + nomatch -> false; + _ -> true + end; + (_) -> + false + end, + lists:member(Role, Args) andalso + lists:any(AnyF, Args); + (_) -> + false + end, + case lists:search(SearchF, Reports) of + {value, _} -> true; + _ -> false + end. diff --git a/lib/ssh/test/ssh_to_openssh_SUITE.erl b/lib/ssh/test/ssh_to_openssh_SUITE.erl index c61907ec288b..5a8f4b31187d 100644 --- a/lib/ssh/test/ssh_to_openssh_SUITE.erl +++ b/lib/ssh/test/ssh_to_openssh_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2008-2021. All Rights Reserved. +%% Copyright Ericsson AB 2008-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -23,6 +23,7 @@ -include_lib("common_test/include/ct.hrl"). -include("ssh_test_lib.hrl"). +-include_lib("ssh/src/ssh_transport.hrl"). -export([ suite/0, @@ -38,7 +39,9 @@ -export([ erlang_server_openssh_client_renegotiate/1, + eserver_oclient_kex_strict/1, erlang_shell_client_openssh_server/1, + eclient_oserver_kex_strict/1, exec_direct_with_io_in_sshc/1, exec_with_io_in_sshc/1, tunnel_in_erlclient_erlserver/1, @@ -73,12 +76,14 @@ groups() -> [{erlang_client, [], [tunnel_in_erlclient_erlserver, tunnel_out_erlclient_erlserver, {group, tunnel_distro_server}, - erlang_shell_client_openssh_server + erlang_shell_client_openssh_server, + eclient_oserver_kex_strict ]}, {tunnel_distro_server, [], [tunnel_in_erlclient_openssh_server, tunnel_out_erlclient_openssh_server]}, {erlang_server, [], [{group, tunnel_distro_client}, erlang_server_openssh_client_renegotiate, + eserver_oclient_kex_strict, exec_with_io_in_sshc, exec_direct_with_io_in_sshc ] @@ -87,16 +92,15 @@ groups() -> tunnel_out_non_erlclient_erlserver]} ]. -init_per_suite(Config) -> +init_per_suite(Config0) -> ?CHECK_CRYPTO( - case gen_tcp:connect("localhost", ?SSH_DEFAULT_PORT, []) of + case gen_tcp:connect("localhost", ?SSH_DEFAULT_PORT, [{active, false}]) of {error,econnrefused} -> {skip,"No openssh daemon (econnrefused)"}; - _ -> + {ok, Sock} -> ssh_test_lib:openssh_sanity_check( - [{ptty_supported, ssh_test_lib:ptty_supported()} - | Config] - ) + [{ptty_supported, ssh_test_lib:ptty_supported()}, + {kex_strict, check_kex_strict(Sock)}| Config0]) end ). @@ -142,6 +146,25 @@ end_per_testcase(_TestCase, _Config) -> %% Test Cases -------------------------------------------------------- %%-------------------------------------------------------------------- erlang_shell_client_openssh_server(Config) when is_list(Config) -> + eclient_oserver_helper(Config). + +eclient_oserver_kex_strict(Config) when is_list(Config)-> + case proplists:get_value(kex_strict, Config) of + true -> + {ok, HandlerPid} = ssh_test_lib:add_report_handler(), + #{level := Level} = logger:get_primary_config(), + logger:set_primary_config(level, notice), + Result = eclient_oserver_helper(Config), + {ok, Reports} = ssh_test_lib:get_reports(HandlerPid), + ct:pal("Reports = ~p", [Reports]), + true = ssh_test_lib:kex_strict_negotiated(client, Reports), + logger:set_primary_config(Level), + Result; + _ -> + {skip, "KEX strict not support by local OpenSSH"} + end. + +eclient_oserver_helper(Config) -> process_flag(trap_exit, true), IO = ssh_test_lib:start_io_server(), Prev = lists:usort(supervisor:which_children(sshc_sup)), @@ -166,7 +189,6 @@ erlang_shell_client_openssh_server(Config) when is_list(Config) -> false end) end. - %%-------------------------------------------------------------------- %% Test that the server could redirect stdin and stdout from/to an %% OpensSSH client when handling an exec request @@ -231,6 +253,25 @@ exec_direct_with_io_in_sshc(Config) when is_list(Config) -> %%-------------------------------------------------------------------- %% Test that the Erlang/OTP server can renegotiate with openSSH erlang_server_openssh_client_renegotiate(Config) -> + eserver_oclient_renegotiate_helper(Config). + +eserver_oclient_kex_strict(Config) -> + case proplists:get_value(kex_strict, Config) of + true -> + {ok, HandlerPid} = ssh_test_lib:add_report_handler(), + #{level := Level} = logger:get_primary_config(), + logger:set_primary_config(level, notice), + Result = eserver_oclient_renegotiate_helper(Config), + {ok, Reports} = ssh_test_lib:get_reports(HandlerPid), + ct:log("Reports = ~p", [Reports]), + true = ssh_test_lib:kex_strict_negotiated(server, Reports), + logger:set_primary_config(Level), + Result; + _ -> + {skip, "KEX strict not support by local OpenSSH"} + end. + +eserver_oclient_renegotiate_helper(Config) -> _PubKeyAlg = ssh_rsa, SystemDir = proplists:get_value(data_dir, Config), PrivDir = proplists:get_value(priv_dir, Config), @@ -255,9 +296,9 @@ erlang_server_openssh_client_renegotiate(Config) -> OpenSsh = ssh_test_lib:open_port({spawn, Cmd++" < "++DataFile}), - Expect = fun({data,R}) -> + Expect = fun({data,R}) -> try - NonAlphaChars = [C || C<-lists:seq(1,255), + NonAlphaChars = [C || C<-lists:seq(1,255), not lists:member(C,lists:seq($a,$z)), not lists:member(C,lists:seq($A,$Z)) ], @@ -275,15 +316,14 @@ erlang_server_openssh_client_renegotiate(Config) -> (_) -> false end, - - try - ssh_test_lib:rcv_expected(Expect, OpenSsh, ?TIMEOUT) + try + ssh_test_lib:rcv_expected(Expect, OpenSsh, ?TIMEOUT) of - _ -> - %% Unfortunately we can't check that there has been a renegotiation, just trust OpenSSH. - ssh:stop_daemon(Pid) + _ -> + %% Unfortunately we can't check that there has been a renegotiation, just trust OpenSSH. + ssh:stop_daemon(Pid) catch - throw:{skip,R} -> {skip,R} + throw:{skip,R} -> {skip,R} end. %%-------------------------------------------------------------------- @@ -569,3 +609,17 @@ no_forwarding(Config) -> "---- The function no_forwarding() returns ~p", [Cmnd,TheText, FailRegExp, Result]), Result. + +check_kex_strict(Sock) -> + %% Send some version, in order to receive KEXINIT from server + ok = gen_tcp:send(Sock, "SSH-2.0-OpenSSH_9.5\r\n"), + ct:sleep(100), + {ok, Packet} = gen_tcp:recv(Sock, 0), + case string:find(Packet, ?kex_strict_s) of + nomatch -> + ct:log("KEX strict NOT supported by local OpenSSH"), + false; + _ -> + ct:log("KEX strict supported by local OpenSSH"), + true + end. diff --git a/lib/ssh/test/ssh_trpt_test_lib.erl b/lib/ssh/test/ssh_trpt_test_lib.erl index 4f037a36a4d3..b66280caf60a 100644 --- a/lib/ssh/test/ssh_trpt_test_lib.erl +++ b/lib/ssh/test/ssh_trpt_test_lib.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2004-2022. All Rights Reserved. +%% Copyright Ericsson AB 2004-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -74,7 +74,7 @@ exec(L, S) when is_list(L) -> lists:foldl(fun exec/2, S, L); exec(Op, S0=#s{}) -> S1 = init_op_traces(Op, S0), try seqnum_trace( - op(Op, S1)) + op(Op, S1), S1) of S = #s{} -> case proplists:get_value(silent,S#s.opts) of @@ -332,12 +332,20 @@ send(S0, ssh_msg_kexinit) -> {Msg, _Bytes, _C0} = ssh_transport:key_exchange_init_msg(S0#s.ssh), send(S0, Msg); +send(S0, ssh_msg_ignore) -> + Msg = #ssh_msg_ignore{data = "unexpected_ignore_message"}, + send(S0, Msg); + +send(S0, ssh_msg_unknown) -> + Msg = binary:encode_hex(<<"0000000C060900000000000000000000">>), + send(S0, Msg); + send(S0=#s{alg_neg={undefined,PeerMsg}}, Msg=#ssh_msg_kexinit{}) -> S1 = opt(print_messages, S0, fun(X) when X==true;X==detail -> {"Send~n~s~n",[format_msg(Msg)]} end), S2 = case PeerMsg of #ssh_msg_kexinit{} -> - try ssh_transport:handle_kexinit_msg(PeerMsg, Msg, S1#s.ssh) of + try ssh_transport:handle_kexinit_msg(PeerMsg, Msg, S1#s.ssh, init) of {ok,Cx} when ?role(S1) == server -> S1#s{alg = Cx#ssh.algorithms}; {ok,_NextKexMsgBin,Cx} when ?role(S1) == client -> @@ -359,7 +367,7 @@ send(S0=#s{alg_neg={undefined,PeerMsg}}, Msg=#ssh_msg_kexinit{}) -> send(S0, ssh_msg_kexdh_init) when ?role(S0) == client -> {OwnMsg, PeerMsg} = S0#s.alg_neg, {ok, NextKexMsgBin, C} = - try ssh_transport:handle_kexinit_msg(PeerMsg, OwnMsg, S0#s.ssh) + try ssh_transport:handle_kexinit_msg(PeerMsg, OwnMsg, S0#s.ssh, init) catch Class:Exc -> fail("Algorithm negotiation failed!", @@ -442,7 +450,7 @@ recv(S0 = #s{}) -> fail("2 kexint received!!", S); {OwnMsg, _} -> - try ssh_transport:handle_kexinit_msg(PeerMsg, OwnMsg, S#s.ssh) of + try ssh_transport:handle_kexinit_msg(PeerMsg, OwnMsg, S#s.ssh, init) of {ok,C} when ?role(S) == server -> S#s{alg_neg = {OwnMsg, PeerMsg}, alg = C#ssh.algorithms, @@ -726,23 +734,23 @@ report_trace(Class, Term, S) -> fun(true) -> {"~s ~p",[Class,Term]} end) ). -seqnum_trace(S) -> +seqnum_trace(S, S0) -> opt(print_seqnums, S, - fun(true) when S#s.ssh#ssh.send_sequence =/= S#s.ssh#ssh.send_sequence, - S#s.ssh#ssh.recv_sequence =/= S#s.ssh#ssh.recv_sequence -> + fun(true) when S0#s.ssh#ssh.send_sequence =/= S#s.ssh#ssh.send_sequence, + S0#s.ssh#ssh.recv_sequence =/= S#s.ssh#ssh.recv_sequence -> {"~p seq num: send ~p->~p, recv ~p->~p~n", [?role(S), - S#s.ssh#ssh.send_sequence, S#s.ssh#ssh.send_sequence, - S#s.ssh#ssh.recv_sequence, S#s.ssh#ssh.recv_sequence + S0#s.ssh#ssh.send_sequence, S#s.ssh#ssh.send_sequence, + S0#s.ssh#ssh.recv_sequence, S#s.ssh#ssh.recv_sequence ]}; - (true) when S#s.ssh#ssh.send_sequence =/= S#s.ssh#ssh.send_sequence -> + (true) when S0#s.ssh#ssh.send_sequence =/= S#s.ssh#ssh.send_sequence -> {"~p seq num: send ~p->~p~n", [?role(S), - S#s.ssh#ssh.send_sequence, S#s.ssh#ssh.send_sequence]}; - (true) when S#s.ssh#ssh.recv_sequence =/= S#s.ssh#ssh.recv_sequence -> + S0#s.ssh#ssh.send_sequence, S#s.ssh#ssh.send_sequence]}; + (true) when S0#s.ssh#ssh.recv_sequence =/= S#s.ssh#ssh.recv_sequence -> {"~p seq num: recv ~p->~p~n", [?role(S), - S#s.ssh#ssh.recv_sequence, S#s.ssh#ssh.recv_sequence]} + S0#s.ssh#ssh.recv_sequence, S#s.ssh#ssh.recv_sequence]} end). print_traces(S) when S#s.prints == [] -> S; diff --git a/lib/ssh/vsn.mk b/lib/ssh/vsn.mk index 50757ade5b5d..16e47bbf807c 100644 --- a/lib/ssh/vsn.mk +++ b/lib/ssh/vsn.mk @@ -1,4 +1,4 @@ #-*-makefile-*- ; force emacs to enter makefile-mode -SSH_VSN = 5.0.1 +SSH_VSN = 5.1.1 APP_VSN = "ssh-$(SSH_VSN)" diff --git a/lib/ssl/doc/src/notes.xml b/lib/ssl/doc/src/notes.xml index e22787552456..72fce239a08d 100644 --- a/lib/ssl/doc/src/notes.xml +++ b/lib/ssl/doc/src/notes.xml @@ -27,6 +27,72 @@

This document describes the changes made to the SSL application.

+
SSL 11.1 + +
Fixed Bugs and Malfunctions + + +

+ ssl application will validate id-kp-serverAuth and + id-kp-clientAuth extended key usage only in end entity + certificates. public_key application will disallow + "anyExtendedKeyUsage" for CA certificates that includes + the extended key usage extension and marks it critical.

+

+ Own Id: OTP-18739

+ + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +

+ Correct handling of TLS-1.3 legacy scheme names, could + cause interop failures for TLS-1.2 clients.

+

+ Own Id: OTP-18817

+ + +

+ Add missing export for connection_info() API type.

+

+ Own Id: OTP-18886

+ + +
+ + +
Improvements and New Features + + +

+ Fixed server name indication which was not handled + properly.

+

+ Own Id: OTP-18836 Aux Id: GH-7795

+ + +

+ Align documentation and implementation

+

+ Own Id: OTP-18853 Aux Id: PR-7841

+ + +

+ Improve connection setup by optimizing certificate + lookup.

+

+ Own Id: OTP-18893 Aux Id: PR-7920 PR-7921

+ + +
+ +
+
SSL 11.0.3
Fixed Bugs and Malfunctions @@ -345,6 +411,31 @@
+
SSL 10.9.1.3 + +
Fixed Bugs and Malfunctions + + +

+ ssl application will validate id-kp-serverAuth and + id-kp-clientAuth extended key usage only in end entity + certificates. public_key application will disallow + "anyExtendedKeyUsage" for CA certificates that includes + the extended key usage extension and marks it critical.

+

+ Own Id: OTP-18739

+ + +

+ Add missing export for connection_info() API type.

+

+ Own Id: OTP-18886

+ + +
+ +
+
SSL 10.9.1.2
Fixed Bugs and Malfunctions diff --git a/lib/ssl/doc/src/ssl.xml b/lib/ssl/doc/src/ssl.xml index f719c55c32f3..403872dff00f 100644 --- a/lib/ssl/doc/src/ssl.xml +++ b/lib/ssl/doc/src/ssl.xml @@ -343,11 +343,35 @@ -

The DER-encoded user's private key or a map referring to a crypto - engine and its key reference that optionally can be password protected, - see also crypto:engine_load/3 - and Crypto's Users Guide. If this option - is supplied, it overrides option keyfile.

+ +

The user's private key. Either the key can be provided + directly as DER encoded entity, or indirectly using a crypto + engine/provider (with key reference information) or an Erlang + fun (with possible custom options). The latter two options + can both be used for customized signing with for instance + hardware security modules (HSM) or trusted platform modules + (TPM).

+ + +

A DER encoded key will need to specify the ASN-1 type used to + create the encoding.

 + +

An engine/provider needs to specify specific + information to support this concept and can optionally be + password protected, see also crypto:engine_load/3 + and + Crypto's Users Guide.

+ +

A fun option should include a fun that mimics public_key:sign/4 + and possibly public_key:private_encrypt/4 + if legacy versions TLS-1.0 and TLS-1.1 should be supported.

+ + +

If this option is supplied, it overrides option keyfile. +

@@ -616,7 +640,7 @@ version. ROOT-CA, and so on. The default value is 10.

- + diff --git a/lib/ssl/src/ssl.app.src b/lib/ssl/src/ssl.app.src index b9f69af6a3bb..4708f7d611e5 100644 --- a/lib/ssl/src/ssl.app.src +++ b/lib/ssl/src/ssl.app.src @@ -88,6 +88,6 @@ {applications, [crypto, public_key, kernel, stdlib]}, {env, []}, {mod, {ssl_app, []}}, - {runtime_dependencies, ["stdlib-4.1","public_key-1.11.3","kernel-9.0", + {runtime_dependencies, ["stdlib-4.1","public_key-@OTP-18876@","kernel-9.0", "erts-14.0","crypto-5.0", "inets-5.10.7", "runtime_tools-1.15.1"]}]}. diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index f21a0fe9b3ed..43c5ae4ac3a9 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -84,7 +84,7 @@ signature_algs/2, eccs/0, eccs/1, - versions/0, + versions/0, groups/0, groups/1, format_error/1, @@ -111,9 +111,9 @@ -removed({cipher_suites, 1, "use cipher_suites/2,3 instead"}). -removed([{negotiated_next_protocol,1, - "use ssl:negotiated_protocol/1 instead"}]). + "use ssl:negotiated_protocol/1 instead"}]). -removed([{connection_info,1, - "use ssl:connection_information/[1,2] instead"}]). + "use ssl:connection_information/[1,2] instead"}]). -export_type([socket/0, sslsocket/0, @@ -193,7 +193,7 @@ -type legacy_hash() :: sha224 | sha | md5. --type sign_algo() :: rsa | dsa | ecdsa | eddsa. % exported +-type sign_algo() :: rsa | rsa_pss_pss | dsa | ecdsa | eddsa. % exported -type sign_schemes() :: [sign_scheme()]. @@ -233,8 +233,8 @@ }. -type old_cipher_suite() :: {kex_algo(), cipher(), hash()} % Pre TLS 1.2 - %% TLS 1.2, internally PRE TLS 1.2 will use default_prf - | {kex_algo(), cipher(), hash() | aead, hash()}. + %% TLS 1.2, internally PRE TLS 1.2 will use default_prf + | {kex_algo(), cipher(), hash() | aead, hash()}. -type named_curve() :: sect571r1 | sect571k1 | @@ -350,11 +350,17 @@ -type cert() :: public_key:der_encoded(). -type cert_pem() :: file:filename(). -type key() :: {'RSAPrivateKey'| 'DSAPrivateKey' | 'ECPrivateKey' |'PrivateKeyInfo', - public_key:der_encoded()} | - #{algorithm := rsa | dss | ecdsa, + public_key:der_encoded()} | + #{algorithm := sign_algo(), engine := crypto:engine_ref(), key_id := crypto:key_id(), - password => crypto:password()}. % exported + password => crypto:password()} | + #{algorithm := sign_algo(), + sign_fun := fun(), + sign_opts => list(), + encrypt_fun => fun(), %% Only TLS-1.0, TLS-1.1 and rsa-key + encrypt_opts => list() + }. % exported -type key_pem() :: file:filename(). -type key_pem_password() :: iodata() | fun(() -> iodata()). -type certs_keys() :: [cert_key_conf()]. @@ -367,12 +373,11 @@ -type ciphers() :: [erl_cipher_suite()] | string(). % (according to old API) exported -type cipher_filters() :: list({key_exchange | cipher | mac | prf, - algo_filter()}). % exported + algo_filter()}). % exported -type algo_filter() :: fun((kex_algo()|cipher()|hash()|aead|default_prf) -> true | false). -type keep_secrets() :: boolean(). -type secure_renegotiation() :: boolean(). -type allowed_cert_chain_length() :: integer(). - -type custom_verify() :: {Verifyfun :: fun(), InitialUserState :: any()}. -type crl_check() :: boolean() | peer | best_effort. -type crl_cache_opts() :: {Module :: atom(), @@ -430,9 +435,9 @@ {use_ticket, use_ticket()} | {early_data, client_early_data()} | {use_srtp, use_srtp()}. - %% {ocsp_stapling, ocsp_stapling()} | - %% {ocsp_responder_certs, ocsp_responder_certs()} | - %% {ocsp_nonce, ocsp_nonce()}. +%% {ocsp_stapling, ocsp_stapling()} | +%% {ocsp_responder_certs, ocsp_responder_certs()} | +%% {ocsp_nonce, ocsp_nonce()}. -type client_verify_type() :: verify_type(). -type client_reuse_session() :: session_id() | {session_id(), SessionData::binary()}. @@ -578,9 +583,9 @@ stop() -> %%-------------------------------------------------------------------- -spec connect(TCPSocket, TLSOptions) -> - {ok, sslsocket()} | - {error, reason()} | - {option_not_a_key_value_tuple, any()} when + {ok, sslsocket()} | + {error, reason()} | + {option_not_a_key_value_tuple, any()} when TCPSocket :: socket(), TLSOptions :: [tls_client_option()]. @@ -588,22 +593,22 @@ connect(Socket, SslOptions) -> connect(Socket, SslOptions, infinity). -spec connect(TCPSocket, TLSOptions, Timeout) -> - {ok, sslsocket()} | {error, reason()} when + {ok, sslsocket()} | {error, reason()} when TCPSocket :: socket(), TLSOptions :: [tls_client_option()], Timeout :: timeout(); (Host, Port, TLSOptions) -> - {ok, sslsocket()} | - {ok, sslsocket(),Ext :: protocol_extensions()} | - {error, reason()} | - {option_not_a_key_value_tuple, any()} when + {ok, sslsocket()} | + {ok, sslsocket(),Ext :: protocol_extensions()} | + {error, reason()} | + {option_not_a_key_value_tuple, any()} when Host :: host(), Port :: inet:port_number(), TLSOptions :: [tls_client_option()]. connect(Socket, SslOptions0, Timeout) when is_list(SslOptions0) andalso (is_integer(Timeout) andalso Timeout >= 0) or (Timeout == infinity) -> - + try CbInfo = handle_option_cb_info(SslOptions0, tls), Transport = element(1, CbInfo), @@ -617,10 +622,10 @@ connect(Host, Port, Options) -> connect(Host, Port, Options, infinity). -spec connect(Host, Port, TLSOptions, Timeout) -> - {ok, sslsocket()} | - {ok, sslsocket(),Ext :: protocol_extensions()} | - {error, reason()} | - {option_not_a_key_value_tuple, any()} when + {ok, sslsocket()} | + {ok, sslsocket(),Ext :: protocol_extensions()} | + {error, reason()} | + {option_not_a_key_value_tuple, any()} when Host :: host(), Port :: inet:port_number(), TLSOptions :: [tls_client_option()], @@ -664,7 +669,7 @@ listen(Port, Options0) -> %% Description: Performs transport accept on an ssl listen socket %%-------------------------------------------------------------------- -spec transport_accept(ListenSocket) -> {ok, SslSocket} | - {error, reason()} when + {error, reason()} when ListenSocket :: sslsocket(), SslSocket :: sslsocket(). @@ -672,7 +677,7 @@ transport_accept(ListenSocket) -> transport_accept(ListenSocket, infinity). -spec transport_accept(ListenSocket, Timeout) -> {ok, SslSocket} | - {error, reason()} when + {error, reason()} when ListenSocket :: sslsocket(), Timeout :: timeout(), SslSocket :: sslsocket(). @@ -686,7 +691,7 @@ transport_accept(#sslsocket{pid = {ListenSocket, dtls_gen_connection -> dtls_socket:accept(ListenSocket, Config, Timeout) end. - + %%-------------------------------------------------------------------- %% %% Description: Performs accept on an ssl listen socket. e.i. performs @@ -729,9 +734,9 @@ handshake(#sslsocket{} = Socket, Timeout) when (is_integer(Timeout) andalso Tim handshake(ListenSocket, SslOptions) -> handshake(ListenSocket, SslOptions, infinity). -spec handshake(Socket, Options, Timeout) -> - {ok, SslSocket} | - {ok, SslSocket, Ext} | - {error, Reason} when + {ok, SslSocket} | + {ok, SslSocket, Ext} | + {error, Reason} when Socket :: socket() | sslsocket(), SslSocket :: sslsocket(), Options :: [server_option()], @@ -782,7 +787,7 @@ handshake(Socket, SslOptions, Timeout) when (is_integer(Timeout) andalso Timeout %%-------------------------------------------------------------------- -spec handshake_continue(HsSocket, Options) -> - {ok, SslSocket} | {error, Reason} when + {ok, SslSocket} | {error, Reason} when HsSocket :: sslsocket(), Options :: [tls_client_option() | tls_server_option()], SslSocket :: sslsocket(), @@ -795,7 +800,7 @@ handshake_continue(Socket, SSLOptions) -> handshake_continue(Socket, SSLOptions, infinity). %%-------------------------------------------------------------------- -spec handshake_continue(HsSocket, Options, Timeout) -> - {ok, SslSocket} | {error, Reason} when + {ok, SslSocket} | {error, Reason} when HsSocket :: sslsocket(), Options :: [tls_client_option() | tls_server_option()], Timeout :: timeout(), @@ -850,7 +855,7 @@ close(#sslsocket{pid = [TLSPid|_]}, Other end; close(#sslsocket{pid = [TLSPid|_]}, Timeout) when is_pid(TLSPid), - (is_integer(Timeout) andalso Timeout >= 0) or (Timeout == infinity) -> + (is_integer(Timeout) andalso Timeout >= 0) or (Timeout == infinity) -> ssl_gen_statem:close(TLSPid, {close, Timeout}); close(#sslsocket{pid = {dtls = ListenSocket, #config{transport_info={Transport,_,_,_,_}}}}, _) -> dtls_socket:close(Transport, ListenSocket); @@ -963,7 +968,7 @@ connection_information(#sslsocket{pid = [Pid|_]}, Items) when is_pid(Pid) -> %%-------------------------------------------------------------------- -spec peername(SslSocket) -> {ok, {Address, Port}} | - {error, reason()} when + {error, reason()} when SslSocket :: sslsocket(), Address :: inet:ip_address(), Port :: inet:port_number(). @@ -1021,12 +1026,12 @@ negotiated_protocol(#sslsocket{pid = [Pid|_]}) when is_pid(Pid) -> %% TLS/DTLS version %%-------------------------------------------------------------------- cipher_suites(Description, Version) when Version == 'tlsv1.3'; - Version == 'tlsv1.2'; - Version == 'tlsv1.1'; - Version == tlsv1 -> + Version == 'tlsv1.2'; + Version == 'tlsv1.1'; + Version == tlsv1 -> cipher_suites(Description, tls_record:protocol_version_name(Version)); cipher_suites(Description, Version) when Version == 'dtlsv1.2'; - Version == 'dtlsv1'-> + Version == 'dtlsv1'-> cipher_suites(Description, dtls_record:protocol_version_name(Version)); cipher_suites(Description, Version) -> [ssl_cipher_format:suite_bin_to_map(Suite) || Suite <- supported_suites(Description, Version)]. @@ -1040,12 +1045,12 @@ cipher_suites(Description, Version) -> %% TLS/DTLS version %%-------------------------------------------------------------------- cipher_suites(Description, Version, StringType) when Version == 'tlsv1.3'; - Version == 'tlsv1.2'; - Version == 'tlsv1.1'; - Version == tlsv1 -> + Version == 'tlsv1.2'; + Version == 'tlsv1.1'; + Version == tlsv1 -> cipher_suites(Description, tls_record:protocol_version_name(Version), StringType); cipher_suites(Description, Version, StringType) when Version == 'dtlsv1.2'; - Version == 'dtlsv1'-> + Version == 'dtlsv1'-> cipher_suites(Description, dtls_record:protocol_version_name(Version), StringType); cipher_suites(Description, Version, rfc) -> [ssl_cipher_format:suite_map_to_str(ssl_cipher_format:suite_bin_to_map(Suite)) @@ -1125,8 +1130,8 @@ signature_algs(default, 'tlsv1.3') -> signature_algs(default, 'tlsv1.2') -> tls_v1:default_signature_algs([tls_record:protocol_version_name('tlsv1.2')]); signature_algs(all, 'tlsv1.3') -> - tls_v1:default_signature_algs([tls_record:protocol_version_name('tlsv1.3'), - tls_record:protocol_version_name('tlsv1.2')]) ++ + tls_v1:default_signature_algs([tls_record:protocol_version_name('tlsv1.3'), + tls_record:protocol_version_name('tlsv1.2')]) ++ tls_v1:legacy_signature_algs_pre_13(); signature_algs(all, 'tlsv1.2') -> tls_v1:default_signature_algs([tls_record:protocol_version_name('tlsv1.2')]) ++ @@ -1194,7 +1199,7 @@ groups(default) -> %%-------------------------------------------------------------------- -spec getopts(SslSocket, OptionNames) -> - {ok, [gen_tcp:option()]} | {error, reason()} when + {ok, [gen_tcp:option()]} | {error, reason()} when SslSocket :: sslsocket(), OptionNames :: [gen_tcp:option_name()]. %% @@ -1286,18 +1291,18 @@ setopts(#sslsocket{}, Options) -> %%--------------------------------------------------------------- -spec getstat(SslSocket) -> - {ok, OptionValues} | {error, inet:posix()} when + {ok, OptionValues} | {error, inet:posix()} when SslSocket :: sslsocket(), OptionValues :: [{inet:stat_option(), integer()}]. %% %% Description: Get all statistic options for a socket. %%-------------------------------------------------------------------- getstat(Socket) -> - getstat(Socket, inet:stats()). + getstat(Socket, inet:stats()). %%--------------------------------------------------------------- -spec getstat(SslSocket, Options) -> - {ok, OptionValues} | {error, inet:posix()} when + {ok, OptionValues} | {error, inet:posix()} when SslSocket :: sslsocket(), Options :: [inet:stat_option()], OptionValues :: [{inet:stat_option(), integer()}]. @@ -1350,7 +1355,7 @@ shutdown(#sslsocket{pid = [Pid|_]}, How) when is_pid(Pid) -> %%-------------------------------------------------------------------- -spec sockname(SslSocket) -> - {ok, {Address, Port}} | {error, reason()} when + {ok, {Address, Port}} | {error, reason()} when SslSocket :: sslsocket(), Address :: inet:ip_address(), Port :: inet:port_number(). @@ -1381,18 +1386,18 @@ versions() -> ImplementedTLSVsns = ?ALL_AVAILABLE_VERSIONS, ImplementedDTLSVsns = ?ALL_AVAILABLE_DATAGRAM_VERSIONS, - TLSCryptoSupported = fun(Vsn) -> - tls_record:sufficient_crypto_support(Vsn) + TLSCryptoSupported = fun(Vsn) -> + tls_record:sufficient_crypto_support(Vsn) + end, + DTLSCryptoSupported = fun(Vsn) -> + tls_record:sufficient_crypto_support(dtls_v1:corresponding_tls_version(Vsn)) end, - DTLSCryptoSupported = fun(Vsn) -> - tls_record:sufficient_crypto_support(dtls_v1:corresponding_tls_version(Vsn)) - end, SupportedTLSVsns = [tls_record:protocol_version(Vsn) || Vsn <- ConfTLSVsns, TLSCryptoSupported(Vsn)], SupportedDTLSVsns = [dtls_record:protocol_version(Vsn) || Vsn <- ConfDTLSVsns, DTLSCryptoSupported(Vsn)], AvailableTLSVsns = [Vsn || Vsn <- ImplementedTLSVsns, TLSCryptoSupported(tls_record:protocol_version_name(Vsn))], AvailableDTLSVsns = [Vsn || Vsn <- ImplementedDTLSVsns, DTLSCryptoSupported(dtls_record:protocol_version_name(Vsn))], - + [{ssl_app, ?VSN}, {supported, SupportedTLSVsns}, {supported_dtls, SupportedDTLSVsns}, @@ -1409,7 +1414,7 @@ versions() -> %% Description: Initiates a renegotiation. %%-------------------------------------------------------------------- renegotiate(#sslsocket{pid = [Pid, Sender |_]} = Socket) when is_pid(Pid), - is_pid(Sender) -> + is_pid(Sender) -> case ssl:connection_information(Socket, [protocol]) of {ok, [{protocol, 'tlsv1.3'}]} -> {error, notsup}; @@ -1451,7 +1456,7 @@ update_keys(_, Type) -> %%-------------------------------------------------------------------- -spec prf(SslSocket, Secret, Label, Seed, WantedLength) -> - {ok, binary()} | {error, reason()} when + {ok, binary()} | {error, reason()} when SslSocket :: sslsocket(), Secret :: binary() | 'master_secret', Label::binary(), @@ -1533,7 +1538,7 @@ str_to_suite(CipherSuiteName) -> _:_ -> {error, {not_recognized, CipherSuiteName}} end. - + %%%-------------------------------------------------------------- %%% Internal functions %%%-------------------------------------------------------------------- @@ -1623,7 +1628,8 @@ ssl_options() -> -spec update_options([any()], client | server, map()) -> map(). update_options(Opts, Role, InheritedSslOpts) when is_map(InheritedSslOpts) -> {UserSslOpts, _} = split_options(Opts, ssl_options()), - process_options(UserSslOpts, InheritedSslOpts, #{role => Role}). + Env = #{role => Role, validate_certs_or_anon_ciphers => Role == server}, + process_options(UserSslOpts, InheritedSslOpts, Env). process_options(UserSslOpts, SslOpts0, Env) -> %% Reverse option list so we get the last set option if set twice, @@ -1648,6 +1654,7 @@ process_options(UserSslOpts, SslOpts0, Env) -> SslOpts17 = opt_handshake(UserSslOptsMap, SslOpts16, Env), SslOpts18 = opt_use_srtp(UserSslOptsMap, SslOpts17, Env), SslOpts = opt_process(UserSslOptsMap, SslOpts18, Env), + validate_server_cert_opts(SslOpts, Env), SslOpts. -spec handle_options([any()], client | server, undefined|host()) -> {ok, #config{}}. @@ -1657,8 +1664,10 @@ handle_options(Opts, Role, Host) -> %% Handle all options in listen, connect and handshake handle_options(Transport, Socket, Opts0, Role, Host) -> {UserSslOptsList, SockOpts0} = split_options(Opts0, ssl_options()), - - Env = #{role => Role, host => Host}, + NeedValidate = not (Socket == undefined) andalso Role =:= server, %% handshake options + Env = #{role => Role, host => Host, + validate_certs_or_anon_ciphers => NeedValidate + }, SslOpts = process_options(UserSslOptsList, #{}, Env), %% Handle special options @@ -1810,21 +1819,21 @@ opt_verify_fun(UserOpts, Opts, _Env) -> Opts#{verify_fun => VerifyFun}. none_verify_fun() -> - fun(_, {bad_cert, _}, UserState) -> - {valid, UserState}; - (_, {extension, #'Extension'{critical = true}}, UserState) -> - %% This extension is marked as critical, so - %% certificate verification should fail if we don't - %% understand the extension. However, this is - %% `verify_none', so let's accept it anyway. - {valid, UserState}; - (_, {extension, _}, UserState) -> - {unknown, UserState}; - (_, valid, UserState) -> + fun(_, {bad_cert, _}, UserState) -> + {valid, UserState}; + (_, {extension, #'Extension'{critical = true}}, UserState) -> + %% This extension is marked as critical, so + %% certificate verification should fail if we don't + %% understand the extension. However, this is + %% `verify_none', so let's accept it anyway. {valid, UserState}; - (_, valid_peer, UserState) -> - {valid, UserState} - end. + (_, {extension, _}, UserState) -> + {unknown, UserState}; + (_, valid, UserState) -> + {valid, UserState}; + (_, valid_peer, UserState) -> + {valid, UserState} + end. convert_verify_fun() -> fun(_,{bad_cert, _} = Reason, OldFun) -> @@ -1840,7 +1849,7 @@ convert_verify_fun() -> {valid, UserState} end. -opt_certs(UserOpts, #{log_level := LogLevel} = Opts0, Env) -> +opt_certs(UserOpts, #{log_level := LogLevel, versions := Versions} = Opts0, Env) -> case get_opt_list(certs_keys, [], UserOpts, Opts0) of {Where, []} when Where =/= new -> opt_old_certs(UserOpts, #{}, Opts0, Env); @@ -1848,11 +1857,11 @@ opt_certs(UserOpts, #{log_level := LogLevel} = Opts0, Env) -> opt_old_certs(UserOpts, CertKey, Opts0, Env); {Where, CKs} when is_list(CKs) -> warn_override(Where, UserOpts, certs_keys, [cert,certfile,key,keyfile,password], LogLevel), - Opts0#{certs_keys => [check_cert_key(CK, #{}, LogLevel) || CK <- CKs]} + Opts0#{certs_keys => [check_cert_key(Versions, CK, #{}, LogLevel) || CK <- CKs]} end. -opt_old_certs(UserOpts, CertKeys, #{log_level := LogLevel}=SSLOpts, _Env) -> - CK = check_cert_key(UserOpts, CertKeys, LogLevel), +opt_old_certs(UserOpts, CertKeys, #{log_level := LogLevel, versions := Versions}=SSLOpts, _Env) -> + CK = check_cert_key(Versions, UserOpts, CertKeys, LogLevel), case maps:keys(CK) =:= [] of true -> SSLOpts#{certs_keys => []}; @@ -1860,7 +1869,7 @@ opt_old_certs(UserOpts, CertKeys, #{log_level := LogLevel}=SSLOpts, _Env) -> SSLOpts#{certs_keys => [CK]} end. -check_cert_key(UserOpts, CertKeys, LogLevel) -> +check_cert_key(Versions, UserOpts, CertKeys, LogLevel) -> CertKeys0 = case get_opt(cert, undefined, UserOpts, CertKeys) of {Where, Cert} when is_binary(Cert) -> warn_override(Where, UserOpts, cert, [certfile], LogLevel), @@ -1895,7 +1904,15 @@ check_cert_key(UserOpts, CertKeys, LogLevel) -> KF == 'RSAPrivateKey'; KF == 'DSAPrivateKey'; KF == 'ECPrivateKey'; KF == 'PrivateKeyInfo' -> CertKeys0#{key => Key}; - {_, #{engine := _, key_id := _, algorithm := _} = Key} -> + {_, #{engine := _, key_id := _, algorithm := Algo} = Key} -> + check_key_algo_version_dep(Versions, Algo), + CertKeys0#{key => Key}; + {_, #{sign_fun := _, algorithm := Algo} = Key} -> + check_key_algo_version_dep(Versions, Algo), + check_key_legacy_version_dep(Versions, Key, Algo), + CertKeys0#{key => Key}; + {_, #{encrypt_fun := _, algorithm := rsa} = Key} -> + check_key_legacy_version_dep(Versions, Key), CertKeys0#{key => Key}; {new, Err1} -> option_error(key, Err1) @@ -1912,6 +1929,29 @@ check_cert_key(UserOpts, CertKeys, LogLevel) -> end, CertKeys2. +check_key_algo_version_dep(Versions, eddsa) -> + assert_version_dep(key, Versions, ['tlsv1.3']); +check_key_algo_version_dep(Versions, rsa_pss_pss) -> + assert_version_dep(key, Versions, ['tlsv1.3', 'tlsv1.2']); +check_key_algo_version_dep(Versions, dsa) -> + assert_version_dep(key, Versions, ['tlsv1.2', 'tlsv1.1', 'tlsv1']); +check_key_algo_version_dep(_,_) -> + true. + +check_key_legacy_version_dep(Versions, Key, rsa) -> + check_key_legacy_version_dep(Versions, Key); +check_key_legacy_version_dep(_,_,_) -> + true. + +check_key_legacy_version_dep(Versions, Key) -> + EncryptFun = maps:get(encrypt_fun, Key, undefined), + case EncryptFun of + undefined -> + assert_version_dep(key, Versions, ['tlsv1.3', 'tlsv1.2']); + _ -> + assert_version_dep(key, Versions, ['tlsv1.1', 'tlsv1']) + end. + opt_cacerts(UserOpts, #{verify := Verify, log_level := LogLevel, versions := Versions} = Opts, #{role := Role}) -> {_, CaCerts} = get_opt_list(cacerts, undefined, UserOpts, Opts), @@ -2244,7 +2284,7 @@ opt_identity(UserOpts, #{versions := Versions} = Opts, _Env) -> PSKSize = byte_size(PSK1), assert_version_dep(psk_identity, Versions, ['tlsv1','tlsv1.1','tlsv1.2']), option_error(not (0 < PSKSize andalso PSKSize < 65536), - psk_identity, {psk_identity, PSK0}), + psk_identity, {psk_identity, PSK0}), PSK1; {_, PSK0} -> PSK0 @@ -2256,7 +2296,7 @@ opt_identity(UserOpts, #{versions := Versions} = Opts, _Env) -> UserSize = byte_size(User), assert_version_dep(srp_identity, Versions, ['tlsv1','tlsv1.1','tlsv1.2']), option_error(not (0 < UserSize andalso UserSize < 65536), - srp_identity, {srp_identity, PSK0}), + srp_identity, {srp_identity, PSK0}), {User, unicode:characters_to_binary(S2)}; {new, Err} -> option_error(srp_identity, Err); @@ -2616,6 +2656,36 @@ validate_filename([_|_] = FN, _Option) -> validate_filename(FN, Option) -> option_error(Option, FN). +validate_server_cert_opts(_Opts, #{validate_certs_or_anon_ciphers := false}) -> + ok; +validate_server_cert_opts(#{certs_keys := [_|_]=CertsKeys, ciphers := CPHS, versions := Versions}, _) -> + validate_certs_or_anon_ciphers(CertsKeys, CPHS, Versions); +validate_server_cert_opts(#{ciphers := CPHS, versions := Versions}, _) -> + validate_anon_ciphers(CPHS, Versions). + +validate_certs_or_anon_ciphers(CertsKeys, Ciphers, Versions) -> + CheckCertsAndKeys = + fun(Map) -> + (maps:is_key(cert, Map) orelse maps:is_key(certfile, Map)) + andalso (maps:is_key(key, Map) orelse maps:is_key(keyfile, Map)) + end, + case lists:any(CheckCertsAndKeys, CertsKeys) of + true -> ok; + false -> validate_anon_ciphers(Ciphers, Versions) + end. + +validate_anon_ciphers(Ciphers, Versions) -> + MakeSet = fun(Version, Acc) -> + Set = sets:from_list(ssl_cipher:anonymous_suites(Version), [{version, 2}]), + sets:union(Set, Acc) + end, + Anonymous = lists:foldl(MakeSet, sets:new([{version, 2}]), Versions), + CiphersSet = sets:from_list(Ciphers, [{version,2}]), + case sets:is_disjoint(Anonymous, CiphersSet) of + false -> ok; + true -> option_error(certs_keys, cert_and_key_required) + end. + %% Do not allow configuration of TLS 1.3 with a gap where TLS 1.2 is not supported %% as that configuration can trigger the built in version downgrade protection %% mechanism and the handshake can fail with an Illegal Parameter alert. @@ -2702,7 +2772,7 @@ all_suites([?TLS_1_3, Version1 |_]) -> ssl_cipher:all_suites(Version1) ++ ssl_cipher:anonymous_suites(Version1); all_suites([Version|_]) -> - ssl_cipher:all_suites(Version) ++ + ssl_cipher:all_suites(Version) ++ ssl_cipher:anonymous_suites(Version). tuple_to_map({Kex, Cipher, Mac}) -> @@ -2868,7 +2938,7 @@ connection_cb(tls) -> connection_cb(dtls) -> dtls_gen_connection; connection_cb(Opts) -> - connection_cb(proplists:get_value(protocol, Opts, tls)). + connection_cb(proplists:get_value(protocol, Opts, tls)). %% Assert that basic options are on the format {Key, Value} @@ -2952,4 +3022,4 @@ format_ocsp_params(Map) -> Nonce = maps:get(ocsp_nonce, Map, '?'), Certs = maps:get(ocsp_responder_certs, Map, '?'), io_lib:format("Stapling = ~W Nonce = ~W Certs = ~W", - [Stapling, 5, Nonce, 5, Certs, 5]). + [Stapling, 5, Nonce, 5, Certs, 5]). diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl index fb503259d589..7ab9487d5d83 100644 --- a/lib/ssl/src/ssl_cipher.erl +++ b/lib/ssl/src/ssl_cipher.erl @@ -344,15 +344,15 @@ tls_legacy_suites(Version) -> %%-------------------------------------------------------------------- anonymous_suites(Version) when ?TLS_1_X(Version) -> - SuitesToTest = anonymous_suite_to_test(Version), - lists:flatmap(fun tls_v1:exclusive_anonymous_suites/1, SuitesToTest); + Versions = versions_included(Version), + lists:flatmap(fun tls_v1:exclusive_anonymous_suites/1, Versions); anonymous_suites(Version) when ?DTLS_1_X(Version) -> dtls_v1:anonymous_suites(Version). -anonymous_suite_to_test(?TLS_1_0) -> [?TLS_1_0]; -anonymous_suite_to_test(?TLS_1_1) -> [?TLS_1_1, ?TLS_1_0]; -anonymous_suite_to_test(?TLS_1_2) -> [?TLS_1_2, ?TLS_1_1, ?TLS_1_0]; -anonymous_suite_to_test(?TLS_1_3) -> [?TLS_1_3]. +versions_included(?TLS_1_0) -> [?TLS_1_0]; +versions_included(?TLS_1_1) -> [?TLS_1_1, ?TLS_1_0]; +versions_included(?TLS_1_2) -> [?TLS_1_2, ?TLS_1_1, ?TLS_1_0]; +versions_included(?TLS_1_3) -> [?TLS_1_3]. %%-------------------------------------------------------------------- -spec filter(undefined | binary(), [ssl_cipher_format:cipher_suite()], diff --git a/lib/ssl/src/ssl_config.erl b/lib/ssl/src/ssl_config.erl index 761a4f431561..fc6a305c32c0 100644 --- a/lib/ssl/src/ssl_config.erl +++ b/lib/ssl/src/ssl_config.erl @@ -87,6 +87,9 @@ group_pairs([#{private_key := #'DSAPrivateKey'{}} = Pair | Rest], #{dsa := DSA} group_pairs([#{private_key := #{algorithm := dss, engine := _}} = Pair | Rest], Group) -> Pairs = maps:get(dsa, Group), group_pairs(Rest, Group#{dsa => [Pair | Pairs]}); +group_pairs([#{private_key := #{algorithm := Alg, sign_fun := _}} = Pair | Rest], Group) -> + Pairs = maps:get(Alg, Group), + group_pairs(Rest, Group#{Alg => [Pair | Pairs]}); group_pairs([#{private_key := #{algorithm := Alg, engine := _}} = Pair | Rest], Group) -> Pairs = maps:get(Alg, Group), group_pairs(Rest, Group#{Alg => [Pair | Pairs]}); @@ -107,16 +110,23 @@ prioritize_groups(#{eddsa := EDDSA, prio_eddsa(EDDSA) -> %% Engine not supported yet - using_curve({namedCurve, ?'id-Ed25519'}, EDDSA, []) ++ using_curve({namedCurve, ?'id-Ed448'}, EDDSA, []). + SignFunPairs = [Pair || Pair = #{private_key := #{sign_fun := _}} <- EDDSA], + SignFunPairs + ++ using_curve({namedCurve, ?'id-Ed25519'}, EDDSA, []) + ++ using_curve({namedCurve, ?'id-Ed448'}, EDDSA, []). prio_ecdsa(ECDSA) -> EnginePairs = [Pair || Pair = #{private_key := #{engine := _}} <- ECDSA], + SignFunPairs = [Pair || Pair = #{private_key := #{sign_fun := _}} <- ECDSA], Curves = tls_v1:ecc_curves(all), - EnginePairs ++ lists:foldr(fun(Curve, AccIn) -> - CurveOid = pubkey_cert_records:namedCurves(Curve), - Pairs = using_curve({namedCurve, CurveOid}, ECDSA -- EnginePairs, []), - Pairs ++ AccIn - end, [], Curves). + EnginePairs + ++ SignFunPairs + ++ lists:foldr( + fun(Curve, AccIn) -> + CurveOid = pubkey_cert_records:namedCurves(Curve), + Pairs = using_curve({namedCurve, CurveOid}, ECDSA -- EnginePairs -- SignFunPairs, []), + Pairs ++ AccIn + end, [], Curves). using_curve(_, [], Acc) -> lists:reverse(Acc); using_curve(Curve, [#{private_key := #'ECPrivateKey'{parameters = Curve}} = Pair | Rest], Acc) -> @@ -265,6 +275,8 @@ init_certificate_file(CertFile, PemCache, Role) -> file_error(CertFile, {certfile, Reason}) end. +init_private_key(#{algorithm := _, sign_fun := _SignFun} = Key, _, _) -> + Key; init_private_key(#{algorithm := Alg} = Key, _, _PemCache) when Alg =:= ecdsa; Alg =:= rsa; Alg =:= dss -> case maps:is_key(engine, Key) andalso maps:is_key(key_id, Key) of diff --git a/lib/ssl/src/ssl_connection.hrl b/lib/ssl/src/ssl_connection.hrl index 2546c1dc81c6..1b8efa0710e2 100644 --- a/lib/ssl/src/ssl_connection.hrl +++ b/lib/ssl/src/ssl_connection.hrl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2013-2022. All Rights Reserved. +%% Copyright Ericsson AB 2013-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/ssl/src/ssl_handshake.erl b/lib/ssl/src/ssl_handshake.erl index 918c55c7b27e..85c89178f5ad 100644 --- a/lib/ssl/src/ssl_handshake.erl +++ b/lib/ssl/src/ssl_handshake.erl @@ -386,15 +386,15 @@ certificate_verify(Signature, PublicKeyInfo, Version, %% Description: Checks that a public_key signature is valid. %%-------------------------------------------------------------------- verify_signature(_, Msg, {HashAlgo, SignAlgo}, Signature, - {_, PubKey, PubKeyParams}) when SignAlgo == rsa_pss_rsae; - SignAlgo == rsa_pss_pss -> - Options = verify_options(SignAlgo, HashAlgo, PubKeyParams), + {_, PubKey, _}) when SignAlgo == rsa_pss_rsae; + SignAlgo == rsa_pss_pss -> + Options = verify_options(SignAlgo, HashAlgo), public_key:verify(Msg, HashAlgo, Signature, PubKey, Options); -verify_signature(Version, Msg, {HashAlgo, SignAlgo}, Signature, {?rsaEncryption, PubKey, PubKeyParams}) +verify_signature(Version, Msg, {HashAlgo, SignAlgo}, Signature, {?rsaEncryption, PubKey, _}) when ?TLS_GTE(Version, ?TLS_1_2) -> - Options = verify_options(SignAlgo, HashAlgo, PubKeyParams), + Options = verify_options(SignAlgo, HashAlgo), public_key:verify(Msg, HashAlgo, Signature, PubKey, Options); -verify_signature(Version, {digest, Digest}, _HashAlgo, Signature, {?rsaEncryption, PubKey, _PubKeyParams}) +verify_signature(Version, {digest, Digest}, _HashAlgo, Signature, {?rsaEncryption, PubKey, _}) when ?TLS_LTE(Version, ?TLS_1_1) -> case public_key:decrypt_public(Signature, PubKey, [{rsa_pad, rsa_pkcs1_padding}]) of @@ -2011,12 +2011,7 @@ supported_cert_type_or_empty(Algo, Type) -> end. certificate_authorities_from_db(CertDbHandle, CertDbRef) when is_reference(CertDbRef) -> - ConnectionCerts = fun({{Ref, _, _}, Cert}, Acc) when Ref == CertDbRef -> - [Cert | Acc]; - (_, Acc) -> - Acc - end, - ssl_pkix_db:foldl(ConnectionCerts, [], CertDbHandle); + ssl_pkix_db:select_certs_by_ref(CertDbRef, CertDbHandle); certificate_authorities_from_db(_CertDbHandle, {extracted, CertDbData}) -> %% Cache disabled, Ref contains data lists:foldl(fun({decoded, {_Key,Cert}}, Acc) -> [Cert | Acc] end, @@ -2144,42 +2139,59 @@ do_digitally_signed(Version, Msg, HashAlgo, {#'RSAPrivateKey'{} = Key, #'RSASSA-PSS-params'{}}, SignAlgo) when ?TLS_GTE(Version, ?TLS_1_2) -> Options = signature_options(SignAlgo, HashAlgo), public_key:sign(Msg, HashAlgo, Key, Options); -do_digitally_signed(Version, {digest, Digest}, _HashAlgo, #'RSAPrivateKey'{} = Key, rsa) when ?TLS_LTE(Version, ?TLS_1_1) -> +do_digitally_signed(Version, {digest, Digest}, _HashAlgo, #'RSAPrivateKey'{} = Key, rsa) + when ?TLS_LTE(Version, ?TLS_1_1) -> public_key:encrypt_private(Digest, Key, [{rsa_pad, rsa_pkcs1_padding}]); +do_digitally_signed(Version, {digest, Digest}, _HashAlgo, #{algorithm := rsa, encrypt_fun := Fun} = Key0, rsa) + when ?TLS_LTE(Version, ?TLS_1_1) -> + CustomOpts = maps:get(encrypt_opts, Key0, []), + Key = #{algorithm => rsa, encrypt_fun => Fun}, + public_key:encrypt_private(Digest, Key, CustomOpts ++ [{rsa_pad, rsa_pkcs1_padding}]); do_digitally_signed(Version, {digest, Digest}, _, - #{algorithm := rsa} = Engine, rsa) when ?TLS_LTE(Version, ?TLS_1_1) -> + #{algorithm := rsa, engine := _} = Engine, rsa) when ?TLS_LTE(Version, ?TLS_1_1) -> crypto:private_encrypt(rsa, Digest, maps:remove(algorithm, Engine), rsa_pkcs1_padding); -do_digitally_signed(_, Msg, HashAlgo, #{algorithm := Alg} = Engine, SignAlgo) -> +do_digitally_signed(_, Msg, HashAlgo, #{algorithm := Alg, engine := _} = Engine, SignAlgo) -> Options = signature_options(SignAlgo, HashAlgo), crypto:sign(Alg, HashAlgo, Msg, maps:remove(algorithm, Engine), Options); do_digitally_signed(Version, {digest, _} = Msg , HashAlgo, Key, _) when ?TLS_LTE(Version,?TLS_1_1) -> public_key:sign(Msg, HashAlgo, Key); +do_digitally_signed(_, Msg, HashAlgo, #{algorithm := SignAlgo, sign_fun := Fun} = Key0, SignAlgo) -> + CustomOpts = maps:get(sign_opts, Key0, []), + Options = signature_options(SignAlgo, HashAlgo), + Key = #{algorithm => SignAlgo, sign_fun => Fun}, + public_key:sign(Msg, HashAlgo, Key, CustomOpts ++ Options); do_digitally_signed(_, Msg, HashAlgo, Key, SignAlgo) -> Options = signature_options(SignAlgo, HashAlgo), public_key:sign(Msg, HashAlgo, Key, Options). - -signature_options(SignAlgo, HashAlgo) when SignAlgo =:= rsa_pss_rsae orelse - SignAlgo =:= rsa_pss_pss -> - pss_options(HashAlgo); +signature_options(rsa_pss_rsae, HashAlgo) -> + pss_options(HashAlgo, hash_algo_byte_size(HashAlgo)); +signature_options(rsa_pss_pss, HashAlgo) -> + pss_options(HashAlgo, hash_algo_byte_size(HashAlgo)); signature_options(_, _) -> []. -verify_options(SignAlgo, HashAlgo, _KeyParams) - when SignAlgo =:= rsa_pss_rsae orelse - SignAlgo =:= rsa_pss_pss -> - pss_options(HashAlgo); -verify_options(_, _, _) -> +verify_options(rsa_pss_rsae, HashAlgo) -> + pss_options(HashAlgo, hash_algo_byte_size(HashAlgo)); +verify_options(rsa_pss_pss, HashAlgo) -> + pss_options(HashAlgo, hash_algo_byte_size(HashAlgo)); +verify_options(_, _) -> []. -pss_options(HashAlgo) -> - %% of the digest algorithm: rsa_pss_saltlen = -1 +pss_options(HashAlgo, SaltLen) -> [{rsa_padding, rsa_pkcs1_pss_padding}, - {rsa_pss_saltlen, -1}, + {rsa_pss_saltlen, SaltLen}, {rsa_mgf1_md, HashAlgo}]. +hash_algo_byte_size(sha256) -> + 32; +hash_algo_byte_size(sha384) -> + 48; +hash_algo_byte_size(sha512) -> + 64. + bad_key(#'DSAPrivateKey'{}) -> unacceptable_dsa_key; bad_key(#'RSAPrivateKey'{}) -> @@ -2188,6 +2200,10 @@ bad_key(#'ECPrivateKey'{}) -> unacceptable_ecdsa_key; bad_key(#{algorithm := rsa}) -> unacceptable_rsa_key; +bad_key(#{algorithm := rsa_pss_pss}) -> + unacceptable_rsa_pss_pss_key; +bad_key(#{algorithm := eddsa}) -> + unacceptable_eddsa_key; bad_key(#{algorithm := ecdsa}) -> unacceptable_ecdsa_key. diff --git a/lib/ssl/src/ssl_pkix_db.erl b/lib/ssl/src/ssl_pkix_db.erl index eac9e2a8b374..f869fb986753 100644 --- a/lib/ssl/src/ssl_pkix_db.erl +++ b/lib/ssl/src/ssl_pkix_db.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2007-2022. All Rights Reserved. +%% Copyright Ericsson AB 2007-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -36,7 +36,7 @@ extract_trusted_certs/1, remove_trusted_certs/2, insert/3, remove/2, clear/1, db_size/1, ref_count/3, lookup_trusted_cert/4, foldl/3, select_certentries_by_ref/2, - decode_pem_file/1, lookup/2]). + select_certs_by_ref/2, decode_pem_file/1, lookup/2]). %%==================================================================== %% Internal application API @@ -53,7 +53,7 @@ create(PEMCacheName) -> [%% Let connection process delete trusted certs %% that can only belong to one connection. (Supplied directly %% on DER format to ssl:connect/listen.) - ets:new(ssl_otp_cacertificate_db, [set, public]), + ets:new(ssl_otp_cacertificate_db, [set, public, {read_concurrency, true}]), %% Let connection processes call ref_count/3 directly {ets:new(ssl_otp_ca_file_ref, [set, public]), ets:new(ssl_otp_ca_ref_file_mapping, [set, protected]) @@ -251,6 +251,14 @@ foldl(Fun, Acc0, Cache) -> select_certentries_by_ref(Ref, Cache) -> ets:select(Cache, [{{{Ref,'_', '_'}, '_'},[],['$_']}]). +%%-------------------------------------------------------------------- +-spec select_certs_by_ref(reference(), db_handle()) -> term(). +%% +%% Description: Select certs originating from same source +%%-------------------------------------------------------------------- +select_certs_by_ref(Ref, Cache) -> + ets:select(Cache, [{{{Ref,'_','_'},'$1'},[],['$1']}]). + %%-------------------------------------------------------------------- -spec ref_count(term(), db_handle(), integer()) -> integer(). %% diff --git a/lib/ssl/test/property_test/ssl_eqc_cipher_format.erl b/lib/ssl/test/property_test/ssl_eqc_cipher_format.erl index 141ccfbe7f84..bcc6d156a9a5 100644 --- a/lib/ssl/test/property_test/ssl_eqc_cipher_format.erl +++ b/lib/ssl/test/property_test/ssl_eqc_cipher_format.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2019-2021. All Rights Reserved. +%% Copyright Ericsson AB 2019-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/ssl/test/ssl_api_SUITE.erl b/lib/ssl/test/ssl_api_SUITE.erl index aa93a021a02e..aed938eeaaf0 100644 --- a/lib/ssl/test/ssl_api_SUITE.erl +++ b/lib/ssl/test/ssl_api_SUITE.erl @@ -1684,9 +1684,10 @@ close_with_timeout(Config) when is_list(Config) -> close_in_error_state() -> [{doc,"Special case of closing socket in error state"}]. close_in_error_state(Config) when is_list(Config) -> - ServerOpts0 = ssl_test_lib:ssl_options(server_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_opts, Config), ServerOpts = [{cacertfile, "foo.pem"} | proplists:delete(cacertfile, ServerOpts0)], ClientOpts = ssl_test_lib:ssl_options(client_opts, Config), + _ = spawn(?MODULE, run_error_server_close, [[self() | ServerOpts]]), receive {_Pid, Port} -> @@ -1703,7 +1704,7 @@ close_in_error_state(Config) when is_list(Config) -> call_in_error_state() -> [{doc,"Special case of call error handling"}]. call_in_error_state(Config) when is_list(Config) -> - ServerOpts0 = ssl_test_lib:ssl_options(server_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_opts, Config), ClientOpts = ssl_test_lib:ssl_options(client_opts, Config), ServerOpts = [{cacertfile, "foo.pem"} | proplists:delete(cacertfile, ServerOpts0)], Pid = spawn(?MODULE, run_error_server, [[self() | ServerOpts]]), @@ -2187,27 +2188,44 @@ options_whitebox() -> customize_defaults(Opts, Role, Host) -> %% In many options test scenarios we do not care about verifcation options %% but the client now requiers verification options by default. - ClientIgnorDef = case proplists:get_value(verify, Opts, undefined) of - undefined when Role == client -> - [{verify, verify_none}]; - _ -> - [] - end, + DefOpts = case Role of + client -> + case proplists:get_value(verify, Opts, undefined) of + undefined -> [{verify, verify_none}]; + _ -> [] + end; + server -> + Ciphers = proplists:get_value(ciphers, Opts, undefined), + Cert = proplists:get_value(cert, Opts, undefined), + Key = proplists:get_value(key, Opts, undefined), + CertsKeys = proplists:get_value(certs_keys, Opts, undefined), + NoCertOrKeys = Cert == undefined orelse Key == undefined andalso + CertsKeys == undefined, + if Ciphers == undefined andalso NoCertOrKeys -> + [{certs_keys, [#{cert => <<>>, key => {rsa, <<>>}}]}]; + true -> + [] + end + end, + NoVerify = case Role of + client -> [{verify, verify_none}|DefOpts]; + server -> DefOpts + end, case proplists:get_value(protocol, Opts, tls) of dtls -> - {ok, #config{ssl=DOpts}} = ssl:handle_options([{verify, verify_none}, {protocol, dtls}], Role, Host), - {DOpts, ClientIgnorDef ++ Opts}; + {ok, #config{ssl=DOpts}} = ssl:handle_options([{protocol, dtls}|NoVerify], Role, Host), + {DOpts, DefOpts ++ Opts}; tls -> - {ok, #config{ssl=DOpts}} = ssl:handle_options([{verify, verify_none}], Role, Host), + {ok, #config{ssl=DOpts}} = ssl:handle_options(NoVerify, Role, Host), case proplists:get_value(versions, Opts) of undefined -> - {DOpts, ClientIgnorDef ++ [{versions, ['tlsv1.2','tlsv1.3']}|Opts]}; + {DOpts, DefOpts ++ [{versions, ['tlsv1.2','tlsv1.3']}|Opts]}; _ -> - {DOpts, ClientIgnorDef ++ Opts} + {DOpts, DefOpts ++ Opts} end; _ -> - {ok, #config{ssl=DOpts}} = ssl:handle_options(ClientIgnorDef, Role, Host), - {DOpts, ClientIgnorDef ++ Opts} + {ok, #config{ssl=DOpts}} = ssl:handle_options(NoVerify, Role, Host), + {DOpts, DefOpts ++ Opts} end. -define(OK(EXP, Opts, Role), ?OK(EXP,Opts, Role, [])). @@ -2279,6 +2297,41 @@ customize_defaults(Opts, Role, Host) -> end end()). +-define(ERR_UPD(EXP, Opts, Role), + fun() -> + Host = "dummy.host.org", + {__DefOpts, __Opts} = customize_defaults(Opts, Role, Host), + try ssl:handle_options(__Opts, Role, Host) of + {ok, #config{}} -> + ok; + Other -> + ?CT_PAL("ssl:handle_options(~0p,~0p,~0p).",[__Opts,Role,Host]), + error({unexpected, Other}) + catch + throw:{error,{options,{insufficient_crypto_support,{'tlsv1.3',_}}}} -> ignored; + C:Other:ST -> + ?CT_PAL("ssl:handle_options(~0p,~0p,~0p).",[__Opts,Role,Host]), + error({unexpected, C, Other,ST}) + end, + try ssl:update_options(__Opts, Role, __DefOpts) of + Other2 -> + ?CT_PAL("{ok,Cfg} = ssl:handle_options([],~p,~p)," + "ssl:update_options(~p,~p, element(2,Cfg)).", + [Role,Host,__Opts,Role]), + error({unexpected, Other2}) + catch + throw:{error,{options,{insufficient_crypto_support,{'tlsv1.3',_}}}} -> ignored; + throw:{error, {options, EXP}} -> ok; + throw:{error, EXP} -> ok; + C2:Other2:ST2 -> + ?CT_PAL("{ok,Cfg} = ssl:handle_options([],~p,~p)," + "ssl:update_options(~p,~p, element(2,Cfg)).", + [Role,Host,__Opts,Role]), + error({unexpected, C2, Other2,ST2}) + end + end()). + + options_whitebox(Config) when is_list(Config) -> Cert = proplists:get_value(cert, ssl_test_lib:ssl_options(server_rsa_der_opts, Config)), true = is_binary(Cert), @@ -2503,7 +2556,17 @@ options_cert(Config) -> %% cert[file] cert_keys keys password [{key, {rsa, <<>>}}], client, Old), ?OK(#{certs_keys := [#{key := #{}}]}, [{key, #{engine => foo, algorithm => foo, key_id => foo}}], client, Old), - + ?OK(#{certs_keys := [#{key := #{}}]}, + [{key, #{algorithm => eddsa, sign_fun => fun(_,_,_,_) -> << "dummy signature">> end}}, + {versions, ['tlsv1.3']}], client, Old), + ?OK(#{certs_keys := [#{key := #{}}]}, + [{key, #{algorithm => ecdsa, sign_fun => fun(_,_,_,_) -> << "dummy signature">> end}}], + client, Old), + ?OK(#{certs_keys := [#{key := #{}}]}, + [{key, #{algorithm => rsa, + sign_fun => fun(_,_,_,_) -> << "dummy signature">> end, + encrypt_fun => fun(_,_,_) -> << "dummy encrypt">> end}}, + {versions, ['tlsv1.3', 'tlsv1.2', 'tlsv1.1']}], client, Old), ?OK(#{certs_keys := [#{password := _}]}, [{password, "foobar"}], client, Old), ?OK(#{certs_keys := [#{password := _}]}, [{password, <<"foobar">>}], client, Old), Pwd = fun() -> "foobar" end, @@ -2517,9 +2580,18 @@ options_cert(Config) -> %% cert[file] cert_keys keys password client, Old), %% Errors + ?ERR({options,incompatible,[key,{versions,['tlsv1.2']}]}, + [{key, #{algorithm => eddsa, sign_fun => fun(_,_,_,_) -> << "dummy signature">> end}}, + {versions, ['tlsv1.2']}], client), + ?ERR({options,incompatible,[key,{versions,['tlsv1.3','tlsv1.2']}]}, + [{key, #{algorithm => rsa, + sign_fun => fun(_,_,_,_) -> << "dummy signature">> end, + encrypt_fun => fun(_,_,_) -> << "dummy encrypt">> end + }}], client), ?ERR({cert, #{}}, [{cert, #{}}], client), ?ERR({certfile, cert}, [{certfile, cert}], client), ?ERR({certs_keys, #{}}, [{certs_keys, #{}}], client), + ?ERR_UPD({certs_keys, cert_and_key_required}, [{certs_keys, []}], server), ?ERR({keyfile, #{}}, [{keyfile, #{}}], client), ?ERR({key, <<>>}, [{key, <<>>}], client), ?ERR({password, _}, [{password, fun(Arg) -> Arg end}], client), @@ -2548,7 +2620,11 @@ options_ciphers(_Config) -> ?OK(#{ciphers := [_|_]}, [{ciphers, "RC4-SHA:RC4-MD5"}], client), ?OK(#{ciphers := [_|_]}, [{ciphers, ["RC4-SHA", "RC4-MD5"]}], client), - %% FIXME extend this + ?OK(#{ciphers := [_|_]}, [{ciphers, ["TLS_DH_anon_WITH_AES_256_CBC_SHA256"]}], server), + %% Errors + ?ERR({ciphers, _}, [{ciphers, "foobar:RC4-MD5"}], client), + ?ERR({ciphers, _}, [{ciphers, ["RC4-SHA:RC4-MD5", "RC4-SHA:RC4-MD5"]}], client), + ?ERR_UPD({certs_keys, cert_and_key_required}, [{ciphers, "RC4-SHA:RC4-MD5"}], server), ok. options_client_renegotiation(_Config) -> diff --git a/lib/ssl/test/ssl_cert_SUITE.erl b/lib/ssl/test/ssl_cert_SUITE.erl index 28f8d06aa717..5f6007a1a05c 100644 --- a/lib/ssl/test/ssl_cert_SUITE.erl +++ b/lib/ssl/test/ssl_cert_SUITE.erl @@ -43,6 +43,8 @@ no_auth/1, auth/0, auth/1, + client_auth_custom_key/0, + client_auth_custom_key/1, client_auth_empty_cert_accepted/0, client_auth_empty_cert_accepted/1, client_auth_empty_cert_rejected/0, @@ -228,6 +230,7 @@ all_version_tests() -> [ no_auth, auth, + client_auth_custom_key, client_auth_empty_cert_accepted, client_auth_empty_cert_rejected, client_auth_use_partial_chain, @@ -459,6 +462,11 @@ auth() -> auth(Config) -> ssl_cert_tests:auth(Config). %%-------------------------------------------------------------------- +client_auth_custom_key() -> + ssl_cert_tests:client_auth_custom_key(). +client_auth_custom_key(Config) -> + ssl_cert_tests:client_auth_custom_key(Config). +%%-------------------------------------------------------------------- client_auth_empty_cert_accepted() -> ssl_cert_tests:client_auth_empty_cert_accepted(). client_auth_empty_cert_accepted(Config) -> diff --git a/lib/ssl/test/ssl_cert_tests.erl b/lib/ssl/test/ssl_cert_tests.erl index 2749d161943e..d2e5e76e27ad 100644 --- a/lib/ssl/test/ssl_cert_tests.erl +++ b/lib/ssl/test/ssl_cert_tests.erl @@ -29,6 +29,8 @@ no_auth/1, auth/0, auth/1, + client_auth_custom_key/0, + client_auth_custom_key/1, client_auth_empty_cert_accepted/0, client_auth_empty_cert_accepted/1, client_auth_empty_cert_rejected/0, @@ -87,18 +89,46 @@ auth() -> auth(Config) -> Version = proplists:get_value(version,Config), + CommonClientOpts = [{verify, verify_peer} | ssl_test_lib:ssl_options(extra_client, client_cert_opts, Config)], ClientOpts = case Version of 'tlsv1.3' -> - [{verify, verify_peer}, - {certificate_authorities, true} | - ssl_test_lib:ssl_options(extra_client, client_cert_opts, Config)]; + [{certificate_authorities, true} | CommonClientOpts]; _ -> - [{verify, verify_peer} | ssl_test_lib:ssl_options(extra_client, client_cert_opts, Config)] + CommonClientOpts end, ServerOpts = [{verify, verify_peer} | ssl_test_lib:ssl_options(extra_server, server_cert_opts, Config)], ssl_test_lib:basic_test(ClientOpts, ServerOpts, Config). %%-------------------------------------------------------------------- +client_auth_custom_key() -> + [{doc,"Test that client and server can connect using their own signature function"}]. + +client_auth_custom_key(Config) when is_list(Config) -> + Version = proplists:get_value(version,Config), + CommonClientOpts = [{verify, verify_peer} | ssl_test_lib:ssl_options(extra_client, client_cert_opts, Config)], + ClientOpts0 = case Version of + 'tlsv1.3' -> + [{certificate_authorities, true} | CommonClientOpts]; + _ -> + CommonClientOpts + end, + ClientKeyFilePath = proplists:get_value(keyfile, ClientOpts0), + [ClientKeyEntry] = ssl_test_lib:pem_to_der(ClientKeyFilePath), + ClientKey = ssl_test_lib:public_key(public_key:pem_entry_decode(ClientKeyEntry)), + ClientCustomKey = choose_custom_key(ClientKey, Version), + + ClientOpts = [ ClientCustomKey | proplists:delete(key, proplists:delete(keyfile, ClientOpts0))], + + ServerOpts0 = ssl_test_lib:ssl_options(extra_server, server_cert_opts, Config), + ServerKeyFilePath = proplists:get_value(keyfile, ServerOpts0), + [ServerKeyEntry] = ssl_test_lib:pem_to_der(ServerKeyFilePath), + ServerKey = ssl_test_lib:public_key(public_key:pem_entry_decode(ServerKeyEntry)), + ServerCustomKey = choose_custom_key(ServerKey, Version), + + ServerOpts = [ ServerCustomKey, {verify, verify_peer} | ServerOpts0], + + ssl_test_lib:basic_test(ClientOpts, ServerOpts, Config). +%%-------------------------------------------------------------------- client_auth_empty_cert_accepted() -> [{doc,"Client sends empty cert chain as no cert is configured and server allows it"}]. @@ -517,3 +547,30 @@ group_config(Config, ServerOpts, ClientOpts) -> {[{supported_groups, [x448, x25519]} | ServerOpts], [{groups,"P-256:X25519"} | ClientOpts]} end. + +choose_custom_key(#'RSAPrivateKey'{} = Key, Version) + when (Version == 'dtlsv1') or (Version == 'tlsv1') or (Version == 'tlsv1.1') -> + EFun = fun (PlainText, Options) -> + public_key:encrypt_private(PlainText, Key, Options) + end, + SFun = fun (Msg, HashAlgo, Options) -> + public_key:sign(Msg, HashAlgo, Key, Options) + end, + {key, #{algorithm => rsa, sign_fun => SFun, encrypt_fun => EFun}}; +choose_custom_key(Key, _) -> + Fun = fun (Msg, HashAlgo, Options) -> + public_key:sign(Msg, HashAlgo, Key, Options) + end, + {key, #{algorithm => alg_key(Key), sign_fun => Fun}}. + +alg_key(#'RSAPrivateKey'{}) -> + rsa; +alg_key({#'RSAPrivateKey'{}, #'RSASSA-PSS-params'{}}) -> + rsa_pss_pss; +alg_key(#'DSAPrivateKey'{}) -> + dsa; +alg_key(#'ECPrivateKey'{parameters = {namedCurve, CurveOId}}) when CurveOId == ?'id-Ed25519' orelse + CurveOId == ?'id-Ed448' -> + eddsa; +alg_key(#'ECPrivateKey'{}) -> + ecdsa. diff --git a/lib/ssl/test/ssl_eqc_SUITE.erl b/lib/ssl/test/ssl_eqc_SUITE.erl index e2f49a61b5c2..1c6f5ca7fd3d 100644 --- a/lib/ssl/test/ssl_eqc_SUITE.erl +++ b/lib/ssl/test/ssl_eqc_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2015-2022. All Rights Reserved. +%% Copyright Ericsson AB 2015-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/ssl/test/tls_api_SUITE.erl b/lib/ssl/test/tls_api_SUITE.erl index ea4cdf2800ac..11756bf2f73e 100644 --- a/lib/ssl/test/tls_api_SUITE.erl +++ b/lib/ssl/test/tls_api_SUITE.erl @@ -193,8 +193,9 @@ init_per_suite(Config0) -> try crypto:start() of ok -> ssl_test_lib:clean_start(), - ssl_test_lib:make_rsa_cert_with_protected_keyfile(Config0, - ?CORRECT_PASSWORD) + Config1 = ssl_test_lib:make_rsa_cert_with_protected_keyfile(Config0, + ?CORRECT_PASSWORD), + ssl_test_lib:make_dsa_cert(Config1) catch _:_ -> {skip, "Crypto did not start"} end. @@ -299,6 +300,7 @@ tls_upgrade_new_opts_with_sni_fun() -> tls_upgrade_new_opts_with_sni_fun(Config) when is_list(Config) -> ClientOpts = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), ServerOpts = ssl_test_lib:ssl_options(server_rsa_opts, Config), + ServerDsaOpts = ssl_test_lib:ssl_options(server_dsa_opts, Config), {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), TcpOpts = [binary, {reuseaddr, true}], Version = ssl_test_lib:protocol_version(Config), @@ -309,23 +311,26 @@ tls_upgrade_new_opts_with_sni_fun(Config) when is_list(Config) -> {ciphers, Ciphers}, {verify, verify_peer}], - Server = ssl_test_lib:start_upgrade_server([{node, ServerNode}, {port, 0}, - {from, self()}, - {mfa, {?MODULE, - upgrade_result, []}}, - {tcp_options, - [{active, false} | TcpOpts]}, - {ssl_options, [{versions, [Version |NewVersions]}, {sni_fun, fun(_SNI) -> ServerOpts ++ NewOpts end}]}]), + Server = ssl_test_lib:start_upgrade_server( + [{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {?MODULE, upgrade_result, []}}, + {tcp_options, + [{active, false} | TcpOpts]}, + {ssl_options, [{versions, [Version |NewVersions]}, + {sni_fun, fun(_SNI) -> ServerOpts ++ NewOpts end} + | ServerDsaOpts]}]), Port = ssl_test_lib:inet_port(Server), - Client = ssl_test_lib:start_upgrade_client([{node, ClientNode}, - {port, Port}, - {host, Hostname}, - {from, self()}, - {mfa, {?MODULE, upgrade_result, []}}, - {tcp_options, [binary]}, - {ssl_options, [{versions, [Version |NewVersions]}, - {ciphers, Ciphers}, - {server_name_indication, Hostname} | ClientOpts]}]), + Client = ssl_test_lib:start_upgrade_client( + [{node, ClientNode}, + {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {?MODULE, upgrade_result, []}}, + {tcp_options, [binary]}, + {ssl_options, [{versions, [Version |NewVersions]}, + {ciphers, Ciphers}, + {server_name_indication, Hostname} | ClientOpts]}]), ?CT_LOG("Client ~p Server ~p ~n", [Client, Server]), @@ -515,7 +520,7 @@ tls_client_closes_socket() -> [{doc,"Test what happens when client closes socket before handshake is completed"}]. tls_client_closes_socket(Config) when is_list(Config) -> - ServerOpts = ssl_test_lib:ssl_options(server_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_opts, Config), {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), TcpOpts = [binary, {reuseaddr, true}], diff --git a/lib/ssl/vsn.mk b/lib/ssl/vsn.mk index 81cd40bc17c2..a7cbe00f57e6 100644 --- a/lib/ssl/vsn.mk +++ b/lib/ssl/vsn.mk @@ -1 +1 @@ -SSL_VSN = 11.0.3 +SSL_VSN = 11.1 diff --git a/lib/stdlib/doc/src/argparse.xml b/lib/stdlib/doc/src/argparse.xml index 20e1f3a72139..136e174a8bea 100644 --- a/lib/stdlib/doc/src/argparse.xml +++ b/lib/stdlib/doc/src/argparse.xml @@ -692,9 +692,10 @@ argparse:run(["2", "-p", "3"], Cmd, #{}).

Generates help/usage information text for the command supplied, or any nested command when command - option is specified. Does not provide localisaton. - Expects progname to be set, otherwise defaults to - return value of init:get_argument(progname).

+ option is specified. Arguments are displayed in the same order as + specified in Command. Does not provide localisation. Expects + progname to be set, otherwise defaults to return value of + init:get_argument(progname).

 @@ -736,4 +737,3 @@ argparse:run(["2", "-p", "3"], Cmd, #{}). - diff --git a/lib/stdlib/doc/src/c.xml b/lib/stdlib/doc/src/c.xml index d924a31a27df..638e6ae38ec6 100644 --- a/lib/stdlib/doc/src/c.xml +++ b/lib/stdlib/doc/src/c.xml @@ -4,7 +4,7 @@
- 19962022 + 19962023 Ericsson AB. All Rights Reserved. diff --git a/lib/stdlib/doc/src/edlin.xml b/lib/stdlib/doc/src/edlin.xml index 56af730ef0c8..1c951138532c 100644 --- a/lib/stdlib/doc/src/edlin.xml +++ b/lib/stdlib/doc/src/edlin.xml @@ -142,6 +142,10 @@ $ ERL_FLAGS="-config $HOME/.erlang_keymap" erl

Move forward one word.

 + help + +

Display help for the module or function closest on the left of the cursor.

+ history_down

Move to the next item in the history.

@@ -158,6 +162,14 @@ $ ERL_FLAGS="-config $HOME/.erlang_keymap" erl

Delete the word under the cursor and save it in the kill buffer.

 + move_expand_down + +

Move down one line in the expand area e.g. help or tab completion pager.

+ + move_expand_up + +

Move up one line in the expand area e.g. help or tab completion pager.

+ new_line_finish

Add a newline at the end of the line and try to evaluate the current expression.

@@ -176,6 +188,14 @@ $ ERL_FLAGS="-config $HOME/.erlang_keymap" erl

Redraw the current line.

 + scroll_expand_down + +

Scroll down five lines in the expand area e.g. help or tab completion pager.

+ + scroll_expand_up + +

Scroll up five lines in the expand area e.g. help or tab completion pager.

+ search_cancel

Cancel the current search.

@@ -192,11 +212,11 @@ $ ERL_FLAGS="-config $HOME/.erlang_keymap" erl

Enter search mode, search the history.

 - search_down + skip_down

Skip to the next line in the history that matches the current search expression.

 - search_up + skip_up

Skip to the previous line in the history that matches the current search expression.

 diff --git a/lib/stdlib/doc/src/ets.xml b/lib/stdlib/doc/src/ets.xml index 855f38d2ae66..cc9eb6d4a98a 100644 --- a/lib/stdlib/doc/src/ets.xml +++ b/lib/stdlib/doc/src/ets.xml @@ -2294,13 +2294,16 @@ true + + Update the Pos:th element of the object with a specified key in an ETS table. +

This function provides an efficient way to update one or more elements within an object, without the trouble of having to look up, @@ -2324,12 +2327,19 @@ true ordered_set table (for details on the difference, see lookup/2 and new/2).

+

If a default object Default is specified, + it is used + as the object to be updated if the key is missing from the table. The + value in place of the key is ignored and replaced by the proper key + value.

The function fails with reason badarg in the following situations:

The table type is not set or ordered_set. Pos < 1. Pos > object arity. + The default object arity is smaller than + ]]>. The element to update is also the key. diff --git a/lib/stdlib/doc/src/notes.xml b/lib/stdlib/doc/src/notes.xml index d2410518cbef..2373fd775f73 100644 --- a/lib/stdlib/doc/src/notes.xml +++ b/lib/stdlib/doc/src/notes.xml @@ -31,6 +31,119 @@

This document describes the changes made to the STDLIB application.

+
STDLIB 5.2 + +
Fixed Bugs and Malfunctions + + +

+ Make shell_docs correctly trim the newline at the + end of code blocks.

+

+ Own Id: OTP-18777 Aux Id: PR-7663

+ + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +

+ Fixed a bug where autocompletion could crash the shell + when trying to expand a nested tuple.

+

+ Own Id: OTP-18822 Aux Id: PR-7796

+ + +

+ Removed auto closing feature, in autocompletion, for + function arguments, tuples, records and maps, since this + could interfere with autocompletion of atoms.

+

+ Own Id: OTP-18823

+ + +

+ Fixed a bug where autocompletion string formatting would + remove suggestions that had the same name but different + case.

+

+ Own Id: OTP-18824

+ + +

+ Fix so that ctrl+h, ctrl+backspace in the shell only + removes one character instead of a whole word.

+

+ Own Id: OTP-18826 Aux Id: PR-7797

+ + +

+ Fix so that its possible to override the default keyboard + shortcuts for the shell.

+

+ Own Id: OTP-18827 Aux Id: PR-7797

+ + +

+ Allow shell local func v(), in a restricted shell

+

+ Own Id: OTP-18828 Aux Id: PR-7799

+ + +

+ Report syntax error when writing an invalid attribute + like '1> -hej.'

+

+ Own Id: OTP-18829 Aux Id: PR-7799

+ + +

When attempting to match part of a record in the key + of a map generator, the entire record would be + matched.

+

+ Own Id: OTP-18866 Aux Id: GH-7875, PR-7878

+ + +
+ + +
Improvements and New Features + + +

+ The warning for accidental use of a future triple-quoted + string delimiter has been upgraded to instead warn for + adjacent strings without intervening white space, which + effectively is the same at a string start, but also + covers the same situation at a string end.

+

+ Own Id: OTP-18821 Aux Id: OTP-18746

+ + +

The removal of the deprecated slave module, + originally planned for OTP 27, has been postponed to OTP + 29.

+

+ Own Id: OTP-18840 Aux Id: PR-7629

+ + +

+ Guards have been added to gen_*:start* API + functions to catch bad arguments earlier. Before this + change, in some cases, a bad argument could tag along and + cause the server to fail later, right after start.

+

+ Own Id: OTP-18857 Aux Id: GH-7685

+ + +
+ +
+
STDLIB 5.1.1
Improvements and New Features diff --git a/lib/stdlib/doc/src/proc_lib.xml b/lib/stdlib/doc/src/proc_lib.xml index 3b5c3c6e1929..28c846b0429f 100644 --- a/lib/stdlib/doc/src/proc_lib.xml +++ b/lib/stdlib/doc/src/proc_lib.xml @@ -160,7 +160,7 @@ - + Returns the user-set process label.

Returns either undefined or the label for the process @@ -341,7 +341,7 @@ init(Parent) -> - + Set process label.

Set a label for the current process. diff --git a/lib/stdlib/doc/src/shell.xml b/lib/stdlib/doc/src/shell.xml index 3de6f34680b3..9b07d4c90028 100644 --- a/lib/stdlib/doc/src/shell.xml +++ b/lib/stdlib/doc/src/shell.xml @@ -282,6 +282,55 @@ To read all record definitions, use '_' as value of RecordNames.

+ lf() + +

Outputs locally defined function with function specs if they exist. +

+ + lt() + +

Outputs locally defined types. +

+ + lr() + +

Outputs locally defined records. +

+ + ff() + +

Forget locally defined functions (including function specs if they exist). +

+ + ff({FunName,Arity}) + +

Forget a locally defined function (including function spec if it exist). Where + FunName is the name of the function as an atom and Arity is an integer. +

+ + tf() + +

Forget locally defined types. +

+ + tf(Type) + +

Forget locally defined type where Type is the name of the type represented as an atom. +

+ + fl() + +

Forget locally defined functions, types and records. +

+ + save_module(FilePath) + +

Saves all locally defined functions, types and records to a module file, where FilePath should include both the + path to the file and the name of the module with .erl suffix. +

+

Example: src/my_module.erl +

+
diff --git a/lib/stdlib/doc/src/stdlib_app.xml b/lib/stdlib/doc/src/stdlib_app.xml index 6d455d09497d..35283c4029a7 100644 --- a/lib/stdlib/doc/src/stdlib_app.xml +++ b/lib/stdlib/doc/src/stdlib_app.xml @@ -60,7 +60,8 @@ shell_expand_location = above | below

Sets where the tab expansion text should appear in the shell. - The default is below.

+ The default is below. This will open a pager below the cursor that is scrollable + one line at a time with Up/Down arrow keys or 5 lines at a time with PgUp/PgDn.

 shell_history_length = integer() >= 0 diff --git a/lib/stdlib/doc/src/supervisor.xml b/lib/stdlib/doc/src/supervisor.xml index 71b1a6f3fdfd..d59aca0a2303 100644 --- a/lib/stdlib/doc/src/supervisor.xml +++ b/lib/stdlib/doc/src/supervisor.xml @@ -4,7 +4,7 @@
- 19962021 + 19962023 Ericsson AB. All Rights Reserved. diff --git a/lib/stdlib/src/argparse.erl b/lib/stdlib/src/argparse.erl index 69b734a126f5..8240f784dd90 100644 --- a/lib/stdlib/src/argparse.erl +++ b/lib/stdlib/src/argparse.erl @@ -1114,11 +1114,11 @@ format_help({ProgName, Root}, Format) -> %% collects options on the Path, and returns found Command collect_options(CmdName, Command, [], Args) -> - {CmdName, Command, maps:get(arguments, Command, []) ++ Args}; + {CmdName, Command, Args ++ maps:get(arguments, Command, [])}; collect_options(CmdName, Command, [Cmd|Tail], Args) -> Sub = maps:get(commands, Command), SubCmd = maps:get(Cmd, Sub), - collect_options(CmdName ++ " " ++ Cmd, SubCmd, Tail, maps:get(arguments, Command, []) ++ Args). + collect_options(CmdName ++ " " ++ Cmd, SubCmd, Tail, Args ++ maps:get(arguments, Command, [])). %% conditionally adds text and empty lines maybe_add(_ToAdd, [], _Element, Template) -> diff --git a/lib/stdlib/src/c.erl b/lib/stdlib/src/c.erl index f95852b89039..b65171a6a57b 100644 --- a/lib/stdlib/src/c.erl +++ b/lib/stdlib/src/c.erl @@ -30,7 +30,7 @@ lc_batch/0, lc_batch/1, i/3,pid/3,m/0,m/1,mm/0,lm/0, bt/1, q/0, - h/1,h/2,h/3,ht/1,ht/2,ht/3,hcb/1,hcb/2,hcb/3, + h/1,h/2,h/3,h1/1,h1/2,h1/3,ht/1,ht/2,ht/3,hcb/1,hcb/2,hcb/3, erlangrc/0,erlangrc/1,bi/1, flush/0, regs/0, uptime/0, nregs/0,pwd/0,ls/0,ls/1,cd/1,memory/1,memory/0, xm/1]). @@ -164,31 +164,33 @@ c(SrcFile, NewOpts, Filter, BeamFile, Info) -> -spec h(module()) -> h_return(). h(Module) -> - case code:get_doc(Module) of - {ok, #docs_v1{ format = Format } = Docs} when ?RENDERABLE_FORMAT(Format) -> - format_docs(shell_docs:render(Module, Docs)); - {ok, #docs_v1{ format = Enc }} -> - {error, {unknown_format, Enc}}; - Error -> - Error - end. + h2(Module, fun(Docs) -> format_docs(shell_docs:render(Module, Docs)) end). -spec h(module(),function()) -> hf_return(). h(Module,Function) -> - case code:get_doc(Module) of - {ok, #docs_v1{ format = Format } = Docs} when ?RENDERABLE_FORMAT(Format) -> - format_docs(shell_docs:render(Module, Function, Docs)); - {ok, #docs_v1{ format = Enc }} -> - {error, {unknown_format, Enc}}; - Error -> - Error - end. + h2(Module, fun(Docs) -> + format_docs(shell_docs:render(Module, Function, Docs)) + end). -spec h(module(),function(),arity()) -> hf_return(). h(Module,Function,Arity) -> + h2(Module, fun(Docs) -> + format_docs(shell_docs:render(Module, Function, Arity, Docs)) + end). + +h1(Module) -> + h2(Module, fun(Docs) -> shell_docs:render(Module, Docs) end). + +h1(Module,Function) -> + h2(Module, fun(Docs) -> shell_docs:render(Module, Function, Docs) end). + +h1(Module,Function,Arity) -> + h2(Module, fun(Docs) -> shell_docs:render(Module, Function, Arity, Docs) end). + +h2(Module, RenderFunction) -> case code:get_doc(Module) of {ok, #docs_v1{ format = Format } = Docs} when ?RENDERABLE_FORMAT(Format) -> - format_docs(shell_docs:render(Module, Function, Arity, Docs)); + RenderFunction(Docs); {ok, #docs_v1{ format = Enc }} -> {error, {unknown_format, Enc}}; Error -> @@ -333,22 +335,43 @@ find_beam_1(Module) -> %% -will try to find and examine the beam file if not in memory %% -will not cause a module to become loaded by accident compile_info(Module, Beam) when is_atom(Module) -> + case erlang:module_loaded(Module) of true -> %% getting the compile info for a loaded module should normally %% work, but return an empty info list if it fails - try erlang:get_module_info(Module, compile) - catch _:_ -> [] + try compile_info_add_cwd(Beam, erlang:get_module_info(Module, compile)) + catch _:_ -> compile_info_add_cwd(Beam, []) end; false -> case beam_lib:chunks(Beam, [compile_info]) of {ok, {_Module, [{compile_info, Info}]}} -> - Info; + compile_info_add_cwd(Beam, Info); Error -> Error end end. +compile_info_add_cwd(Beam, Info) -> + CwdOpts = + case beam_lib:chunks(Beam, [debug_info]) of + {ok, {_,[{debug_info,{debug_info_v1,erl_abstract_code,{_AST,Meta}}}]}} -> + case proplists:get_value(cwd, Meta) of + undefined -> + []; + Cwd -> + [{i, Cwd}] + end; + _ -> + [] + end, + case lists:keytake(options, 1, Info) of + false -> + [{options, CwdOpts}]; + {value, {options, Options}, InfoNoOpts} -> + [{options, Options ++ CwdOpts} | InfoNoOpts] + end. + %% compile module, backing up any existing target file and restoring the %% old version if compilation fails (this should only be used when we have %% an old beam file that we want to preserve) diff --git a/lib/stdlib/src/edlin.erl b/lib/stdlib/src/edlin.erl index 3aae25e7af58..7c512e71dbeb 100644 --- a/lib/stdlib/src/edlin.erl +++ b/lib/stdlib/src/edlin.erl @@ -114,15 +114,19 @@ edit(eof, _, {_,{Bef,Aft0},LA} = L, _, Rs) -> _ -> Aft0 end, {done,L,[],reverse(Rs, [{move_combo,-cp_len(Bef), length(LA), cp_len(Aft1)}])}; -edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> +edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode1, EscapePrefix}, Rs0) -> + {ShellMode, NextMode} = case ShellMode1 of + {_, _}=M -> M; + Mode -> {Mode, Mode} + end, case edlin_key:get_valid_escape_key(Buf, EscapePrefix) of {escape_prefix, EscapePrefix1} -> case ShellMode of tab_expand -> edit(Buf, P, MultiLine, {normal, none}, Rs0); - _ -> edit([], P, MultiLine, {ShellMode, EscapePrefix1}, Rs0) + _ -> edit([], P, MultiLine, {NextMode, EscapePrefix1}, Rs0) end; {invalid, _I, Rest} -> - edit(Rest, P, MultiLine, {ShellMode, none}, Rs0); + edit(Rest, P, MultiLine, {NextMode, none}, Rs0); {insert, C1, Cs2} -> %% If its a printable character %% we could in theory override it in the keymap, @@ -137,7 +141,8 @@ edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> normal -> {insert, C1}; search when $\s =< C1 ->{insert_search, C1}; search -> search_quit; - tab_expand -> tab_expand_quit + tab_expand -> tab_expand_quit; + help -> tab_expand_quit end, case Op of tab_expand_quit -> @@ -148,10 +153,10 @@ edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> case do_op(Op, MultiLine, Rs0) of {blink,N,MultiLine1,Rs} -> edit(Cs2, P, MultiLine1, {blink,N}, Rs); - {redraw, MultiLine1, Rs} -> - edit(Cs2, P, MultiLine1, {ShellMode, none}, redraw(P, MultiLine1, Rs)); - {MultiLine1,Rs} -> - edit(Cs2, P, MultiLine1, {ShellMode, none}, Rs) + {redraw, {_LB1, {_Bef1, _Aft1}, _LA1}=MultiLine1, Rs} -> + edit(Cs2, P, MultiLine1, {NextMode, none}, redraw(P, MultiLine1, Rs)); + {{_LB1, {_Bef1, _Aft1}, _LA1}=MultiLine1,Rs} -> + edit(Cs2, P, MultiLine1, {NextMode, none}, Rs) end end; {key, Key, Cs} -> @@ -165,18 +170,22 @@ edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> end; {ok, Value0} -> Value0 end, + Cont = {line,P,MultiLine,{NextMode, none}}, case Value of - none -> edit(Cs, P, MultiLine, {normal,none}, Rs0); - search -> {search,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; - search_found -> {search_found,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; - search_cancel -> {search_cancel,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; - search_quit -> {search_quit,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; - open_editor -> {open_editor,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; - history_up -> {history_up,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; - history_down -> {history_down,Cs,{line,P,MultiLine,{normal, none}},reverse(Rs0)}; + {mode, Mode1} -> + edit(Buf, P, MultiLine, {{Mode1, ShellMode}, none}, Rs0); + none -> edit(Cs, P, MultiLine, {NextMode,none}, Rs0); + search -> {search,Cs,Cont,reverse(Rs0)}; + search_found -> {search_found,Cs,Cont,reverse(Rs0)}; + search_cancel -> {search_cancel,Cs,Cont,reverse(Rs0)}; + search_quit -> {search_quit,Cs,Cont,reverse(Rs0)}; + format_expression -> {format_expression,Cs,Cont,reverse(Rs0)}; + open_editor -> {open_editor,Cs,Cont,reverse(Rs0)}; + history_up -> {history_up,Cs,Cont,reverse(Rs0)}; + history_down -> {history_down,Cs,Cont,reverse(Rs0)}; new_line -> MultiLine1 = {[lists:reverse(Bef)|LB],{[],Aft},LA}, - edit(Cs, P, MultiLine1, {normal, none}, reverse(redraw(P, MultiLine1, Rs0))); + edit(Cs, P, MultiLine1, {NextMode, none}, reverse(redraw(P, MultiLine1, Rs0))); new_line_finish -> % Move to end {{LB1,{Bef1,[]},[]}, Rs1} = do_op(end_of_expression, MultiLine, Rs0), @@ -184,10 +193,13 @@ edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> redraw_line -> Rs1 = erase_line(Rs0), Rs = redraw(P, MultiLine, Rs1), - edit(Cs, P, MultiLine, {normal, none}, Rs); + edit(Cs, P, MultiLine, {NextMode, none}, Rs); clear -> Rs = redraw(P, MultiLine, [clear|Rs0]), - edit(Cs, P, MultiLine, {normal, none}, Rs); + edit(Cs, P, MultiLine, {NextMode, none}, Rs); + help -> + {help, chars_before(MultiLine), Cs,{line, P, MultiLine, {help, none}}, + reverse(Rs0)}; tab_expand -> {expand, chars_before(MultiLine), Cs, {line, P, MultiLine, {tab_expand, none}}, @@ -198,7 +210,8 @@ edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> reverse(Rs0)}; tab_expand_quit -> %% When exiting tab expand mode, we want to evaluate the key in normal mode - edit(Buf, P, MultiLine, {normal,none}, Rs0); + %% we send a {move_rel, 0} event to make sure the paging area is cleared + edit(Buf, P, MultiLine, {normal,none}, [{move_rel, 0}|Rs0]); Op -> Op1 = case ShellMode of search -> @@ -209,14 +222,14 @@ edit(Buf, P, {LB, {Bef,Aft}, LA}=MultiLine, {ShellMode, EscapePrefix}, Rs0) -> {blink,N,MultiLine1,Rs} -> edit(Cs, P, MultiLine1, {blink,N}, Rs); {redraw, MultiLine1, Rs} -> - edit(Cs, P, MultiLine1, {normal, none}, redraw(P, MultiLine1, Rs)); + edit(Cs, P, MultiLine1, {NextMode, none}, redraw(P, MultiLine1, Rs)); {MultiLine1,Rs} -> - edit(Cs, P, MultiLine1, {ShellMode, none}, Rs) + edit(Cs, P, MultiLine1, {NextMode, none}, Rs) end end end. -%% do_op(Action, Before, After, Requests) +%% do_op(Action, {LinesBefore, {Before, After}, LinesAfter}, Requests) %% Before and After are of lists of type string:grapheme_cluster() do_op({insert,C}, {LB,{[],[]},LA}, Rs) -> {{LB,{[C],[]},LA},[{insert_chars, unicode,[C]}|Rs]}; @@ -246,6 +259,22 @@ do_op({insert,C}, {LB,{[Bef|Bef0], Aft},LA}, Rs) -> %% search: $TERMS %% $ResultLine1 %% $ResultLine2 +do_op(move_expand_up, Cont, Rs) -> + {Cont, [{move_expand, -1}|Rs]}; +do_op(move_expand_down, Cont, Rs) -> + {Cont, [{move_expand, 1}|Rs]}; +do_op({search,move_expand_up}, Cont, Rs) -> + {Cont, [{move_expand, -1}|Rs]}; +do_op({search,move_expand_down}, Cont, Rs) -> + {Cont, [{move_expand, 1}|Rs]}; +do_op(scroll_expand_up, Cont, Rs) -> + {Cont, [{move_expand, -5}|Rs]}; +do_op(scroll_expand_down, Cont, Rs) -> + {Cont, [{move_expand, 5}|Rs]}; +do_op({search,scroll_expand_up}, Cont, Rs) -> + {Cont, [{move_expand, -5}|Rs]}; +do_op({search,scroll_expand_down}, Cont, Rs) -> + {Cont, [{move_expand, 5}|Rs]}; do_op({insert_search, C}, {LB,{Bef, []},LA}, Rs) -> {{LB, {[C|Bef],[]}, LA}, [{insert_chars, unicode, [C]}, delete_after_cursor | Rs]}; @@ -256,9 +285,9 @@ do_op({insert_search, C}, {LB,{Bef, _Aft},LA}, Rs) -> do_op({search, backward_delete_char}, {LB,{[_|Bef], Aft},LA}, Rs) -> Offset= cp_len(Aft)+1, {{LB, {Bef,Aft}, LA}, - [{insert_chars, unicode, Aft}, {delete_chars,-Offset}|Rs]}; + [redraw, {insert_chars, unicode, Aft}, {delete_chars,-Offset}|Rs]}; do_op({search, backward_delete_char}, {LB,{[], Aft},LA}, Rs) -> - {{LB, {[],Aft}, LA}, [{insert_chars, unicode, Aft}, {delete_chars,-cp_len(Aft)}|Rs]}; + {redraw, {LB, {[],Aft}, LA}, [{insert_chars, unicode, Aft}, {delete_chars,-cp_len(Aft)}|Rs]}; do_op({search, skip_up}, {_,{Bef, Aft},_}, Rs) -> Offset= cp_len(Aft), {{[],{[$\^R|Bef],Aft},[]}, % we insert ^R as a flag to whoever called us @@ -286,7 +315,7 @@ do_op(backward_delete_char, {[PrevLine|LB],{[], Aft},LA}, Rs) -> NewLine = {LB, {lists:reverse(PrevLine), Aft}, LA}, {redraw, NewLine,Rs}; do_op(backward_delete_char, {LB,{[GC|Bef], Aft},LA}, Rs) -> - {{LB, {Bef,Aft}, LA},[{delete_chars,-gc_len(GC)}|Rs]}; + {redraw, {LB, {Bef,Aft}, LA},[{delete_chars,-gc_len(GC)}|Rs]}; do_op(forward_delete_word, {LB,{Bef, []},[NextLine|LA]}, Rs) -> NewLine = {LB, {Bef, NextLine}, LA}, {redraw, NewLine, Rs}; @@ -294,7 +323,7 @@ do_op(forward_delete_word, {LB,{Bef, Aft0},LA}, Rs) -> {Aft1,Kill0,N0} = over_non_word(Aft0, [], 0), {Aft,Kill,N} = over_word(Aft1, Kill0, N0), put(kill_buffer, reverse(Kill)), - {{LB, {Bef,Aft}, LA},[{delete_chars,N}|Rs]}; + {redraw, {LB, {Bef,Aft}, LA},[{delete_chars,N}|Rs]}; do_op(backward_delete_word, {[PrevLine|LB],{[], Aft},LA}, Rs) -> NewLine = {LB, {lists:reverse(PrevLine), Aft}, LA}, {redraw, NewLine,Rs}; @@ -339,20 +368,20 @@ do_op(transpose_word, {LB,{Bef0, Aft0},LA}, Rs) -> {Bef3,Word1,B2} = over_word(Bef2, [], B1), {Bef3, Word2B++reverse(Word2A)++NonWord++Word1, Aft1, B2} end, - {{LB, {reverse(TransposedWords)++Bef, Aft}, LA},[{insert_chars_over, unicode, TransposedWords}, {move_rel, -N}|Rs]}; + {redraw,{LB, {reverse(TransposedWords)++Bef, Aft}, LA},[{insert_chars_over, unicode, TransposedWords}, {move_rel, -N}|Rs]}; do_op(kill_word, {LB,{Bef, Aft0},LA}, Rs) -> {Aft1,Kill0,N0} = over_non_word(Aft0, [], 0), {Aft,Kill,N} = over_word(Aft1, Kill0, N0), put(kill_buffer, reverse(Kill)), - {{LB, {Bef,Aft}, LA},[{delete_chars,N}|Rs]}; + {redraw,{LB, {Bef,Aft}, LA},[{delete_chars,N}|Rs]}; do_op(backward_kill_word, {LB,{Bef0, Aft},LA}, Rs) -> {Bef1,Kill0,N0} = over_non_word(Bef0, [], 0), {Bef,Kill,N} = over_word(Bef1, Kill0, N0), put(kill_buffer, Kill), - {{LB,{Bef,Aft},LA},[{delete_chars,-N}|Rs]}; + {redraw,{LB,{Bef,Aft},LA},[{delete_chars,-N}|Rs]}; do_op(kill_line, {LB, {Bef, Aft}, LA}, Rs) -> put(kill_buffer, Aft), - {{LB, {Bef,[]}, LA},[{delete_chars,cp_len(Aft)}|Rs]}; + {redraw,{LB, {Bef,[]}, LA},[{delete_chars,cp_len(Aft)}|Rs]}; do_op(clear_line, _, Rs) -> {redraw, {[], {[],[]},[]}, Rs}; do_op(yank, {LB,{Bef, []},LA}, Rs) -> @@ -360,7 +389,7 @@ do_op(yank, {LB,{Bef, []},LA}, Rs) -> {{LB, {reverse(Kill, Bef),[]}, LA},[{insert_chars, unicode,Kill}|Rs]}; do_op(yank, {LB,{Bef, Aft},LA}, Rs) -> Kill = get(kill_buffer), - {{LB, {reverse(Kill, Bef),Aft}, LA},[{insert_chars, unicode,Kill}|Rs]}; + {redraw,{LB, {reverse(Kill, Bef),Aft}, LA},[{insert_chars, unicode,Kill}|Rs]}; do_op(forward_line, {_,_,[]} = MultiLine, Rs) -> {MultiLine, Rs}; do_op(forward_line, {LB,{Bef, Aft},[AL|LA]}, Rs) -> diff --git a/lib/stdlib/src/edlin_context.erl b/lib/stdlib/src/edlin_context.erl index 73e42cedf263..979d838a7cb6 100644 --- a/lib/stdlib/src/edlin_context.erl +++ b/lib/stdlib/src/edlin_context.erl @@ -65,6 +65,7 @@ | {fun_, Mod, Fun} %% cursor is in a fun mod:fun statement | {new_fun, Unfinished} | {function} + | {function, Mod} | {function, Mod, Fun, Args, Unfinished, Nesting} | {map, Binding, Keys} | {map_or_record} @@ -220,7 +221,8 @@ get_context([$:|Bef2], _) -> {Bef3, Mod} = edlin_expand:over_word(Bef2), case edlin_expand:over_word(Bef3) of {_, "fun"} -> {fun_, Mod}; - _ -> {function} + _ when Mod =:= [] -> {function}; + _ -> {function, Mod} end; get_context([$/|Bef1], _) -> {Bef2, Fun} = edlin_expand:over_word(Bef1), diff --git a/lib/stdlib/src/edlin_expand.erl b/lib/stdlib/src/edlin_expand.erl index 169e3cea86e9..37707348f2a1 100644 --- a/lib/stdlib/src/edlin_expand.erl +++ b/lib/stdlib/src/edlin_expand.erl @@ -89,6 +89,7 @@ expand(Bef0, Opts, #shell_state{bindings = Bs, records = RT, functions = FT}) -> {error, _Column} -> {no, [], []}; {function} -> expand_module_function(Bef0, FT); + {function, _Mod} -> expand_module_function(Bef0, FT); {fun_} -> expand_module_function(Bef0, FT); {fun_, Mod} -> expand_function_name(Mod, Word, "/", FT); @@ -633,18 +634,21 @@ expand_filepath(PathPrefix, Word) -> end. shell(Fun) -> - case shell:local_func(list_to_atom(Fun)) of + {ok, [{atom, _, Fun1}], _} = erl_scan:string(Fun), + case shell:local_func(Fun1) of true -> "shell"; false -> "user_defined" end. shell_default_or_bif(Fun) -> - case lists:member(list_to_atom(Fun), [E || {E,_}<-get_exports(shell_default)]) of + {ok, [{atom, _, Fun1}], _} = erl_scan:string(Fun), + case lists:member(Fun1, [E || {E,_}<-get_exports(shell_default)]) of true -> "shell_default"; _ -> bif(Fun) end. bif(Fun) -> - case lists:member(list_to_atom(Fun), [E || {E,A}<-get_exports(erlang), erl_internal:bif(E,A)]) of + {ok, [{atom, _, Fun1}], _} = erl_scan:string(Fun), + case lists:member(Fun1, [E || {E,A}<-get_exports(erlang), erl_internal:bif(E,A)]) of true -> "erlang"; _ -> shell(Fun) end. diff --git a/lib/stdlib/src/edlin_key.erl b/lib/stdlib/src/edlin_key.erl index 90b3d3e207cd..40bbadb9ed6c 100644 --- a/lib/stdlib/src/edlin_key.erl +++ b/lib/stdlib/src/edlin_key.erl @@ -113,7 +113,7 @@ get_valid_escape_key(Rest, Acc) -> {invalid, Acc, Rest}. merge(KeyMap) -> - merge(KeyMap, [normal, search, tab_expand], key_map()). + merge(KeyMap, [normal, search, tab_expand, help], key_map()). merge(_, [], KeyMap) -> KeyMap; merge(InputKeyMap, [Mode|ShellModes], KeyMap) -> @@ -153,6 +153,12 @@ merge(InputKeyMap, [Mode|ShellModes], KeyMap) -> key_map() -> #{ normal => normal_map(), search => #{ + "\^[OA" => move_expand_up, + "\^[[A" => move_expand_up, + "\^[OB" => move_expand_down, + "\^[[B" => move_expand_down, + "\^[[6~" => scroll_expand_down, + "\^[[5~" => scroll_expand_up, "\^R" => skip_up, "\^S" => skip_down, "\^[C" => search_cancel, @@ -165,8 +171,23 @@ key_map() -> #{ %% # everything else should exit search mode and edit the search result (search_quit), }, tab_expand => #{ + "\^[OA" => move_expand_up, + "\^[[A" => move_expand_up, + "\^[OB" => move_expand_down, + "\^[[B" => move_expand_down, + "\^[[6~" => scroll_expand_down, + "\^[[5~" => scroll_expand_up, "\t" => tab_expand_full, default => tab_expand_quit %% go to normal mode and evaluate key input again + }, + help => #{ + "\^[OA" => move_expand_up, + "\^[[A" => move_expand_up, + "\^[OB" => move_expand_down, + "\^[[B" => move_expand_down, + "\^[[6~" => scroll_expand_down, + "\^[[5~" => scroll_expand_up, + default => tab_expand_quit %% go to normal mode and evaluate key input again } }. @@ -215,6 +236,7 @@ normal_map() -> "\^[d" => kill_word, "\^[F" => forward_word, "\^[f" => forward_word, + "\^[h" => help, "\^[L" => redraw_line, "\^[l" => redraw_line, "\^[o" => open_editor, @@ -293,16 +315,21 @@ valid_functions() -> forward_delete_word, %% Delete the characters until the closest non-word character forward_line, %% Move forward one line forward_word, %% Move forward one word + help, %% Open up a pager with help for function or module closest to the cursor history_down, %% Move to the next item in the history history_up, %% Move to the previous item in the history %%jcl_menu, kill_line, %% Delete all characters from the cursor to the end of the line and save them in the kill buffer kill_word, %% Delete the word behind the cursor and save it in the kill buffer + move_expand_up, %% Move up one line in the expand area e.g. help or tab completion pager + move_expand_down, %% Move down one line in the expand area e.g. help or tab completion pager new_line_finish, %% Add a newline at the end of the line and try to evaluate the current expression new_line, %% Add a newline at the cursor position none, %% Do nothing open_editor, %% Open the current line in an editor i.e. EDITOR=code -w redraw_line, %% Redraw the current line + scroll_expand_up, %% Scroll up five lines in the expand area e.g. help or tab completion pager + scroll_expand_down, %% Scroll down five lines in the expand area e.g. help or tab completion pager search_cancel, %% Cancel the current search search_found, %% Accept the current search result and submit it search_quit, %% Accept the current search result, but edit it before submitting diff --git a/lib/stdlib/src/erl_stdlib_errors.erl b/lib/stdlib/src/erl_stdlib_errors.erl index 45005047bc1a..60996113c2d8 100644 --- a/lib/stdlib/src/erl_stdlib_errors.erl +++ b/lib/stdlib/src/erl_stdlib_errors.erl @@ -162,7 +162,7 @@ format_binary_error(split, [Subject, Pattern, _Options], _) -> format_binary_error(replace, [Subject, Pattern, Replacement], _) -> [must_be_binary(Subject), must_be_pattern(Pattern), - must_be_binary(Replacement)]; + must_be_binary_replacement(Replacement)]; format_binary_error(replace, [Subject, Pattern, Replacement, _Options], Cause) -> Errors = format_binary_error(replace, [Subject, Pattern, Replacement], Cause), case Cause of @@ -772,6 +772,8 @@ format_ets_error(update_element, [_,_,ElementSpec]=Args, Cause) -> case Cause of keypos -> [same_as_keypos]; + position -> + [update_op_range]; _ -> case is_element_spec_top(ElementSpec) of true -> @@ -785,6 +787,26 @@ format_ets_error(update_element, [_,_,ElementSpec]=Args, Cause) -> [<<"is not a valid element specification">>] end end]; +format_ets_error(update_element, [_, _, ElementSpec, Default]=Args, Cause) -> + TabCause = format_cause(Args, Cause), + ArgsCause = case Cause of + keypos -> + [same_as_keypos]; + position -> + [update_op_range]; + _ -> + case {is_element_spec_top(ElementSpec), format_tuple(Default)} of + {true, [""]} -> + [range]; + {true, TupleCause} -> + ["" | TupleCause]; + {false, [""]} -> + [<<"is not a valid element specification">>]; + {false, TupleCause} -> + ["" | TupleCause] + end + end, + [TabCause, "" | ArgsCause]; format_ets_error(whereis, _Args, _Cause) -> [bad_table_name]; format_ets_error(_, Args, Cause) -> @@ -1015,6 +1037,10 @@ must_be_pattern(P) -> bad_binary_pattern end. +must_be_binary_replacement(R) when is_binary(R) -> []; +must_be_binary_replacement(R) when is_function(R, 1) -> []; +must_be_binary_replacement(_R) -> bad_replacement. + must_be_position(Pos) when is_integer(Pos), Pos >= 0 -> []; must_be_position(Pos) when is_integer(Pos) -> range; must_be_position(_) -> not_integer. diff --git a/lib/stdlib/src/ets.erl b/lib/stdlib/src/ets.erl index 8628a7e29f61..54f195a40917 100644 --- a/lib/stdlib/src/ets.erl +++ b/lib/stdlib/src/ets.erl @@ -77,7 +77,7 @@ select_count/2, select_delete/2, select_replace/2, select_reverse/1, select_reverse/2, select_reverse/3, setopts/2, slot/2, take/2, - update_counter/3, update_counter/4, update_element/3, + update_counter/3, update_counter/4, update_element/3, update_element/4, whereis/1]). %% internal exports @@ -551,6 +551,22 @@ update_counter(_, _, _, _) -> update_element(_, _, _) -> erlang:nif_error(undef). +-spec update_element(Table, Key, ElementSpec :: {Pos, Value}, Default) -> true when + Table :: table(), + Key :: term(), + Pos :: pos_integer(), + Value :: term(), + Default :: tuple(); + (Table, Key, ElementSpec :: [{Pos, Value}], Default) -> true when + Table :: table(), + Key :: term(), + Pos :: pos_integer(), + Value :: term(), + Default :: tuple(). + +update_element(_, _, _, _) -> + erlang:nif_error(undef). + -spec whereis(TableName) -> tid() | undefined when TableName :: atom(). whereis(_) -> diff --git a/lib/stdlib/src/shell.erl b/lib/stdlib/src/shell.erl index 760889164c66..96b1f345cc75 100644 --- a/lib/stdlib/src/shell.erl +++ b/lib/stdlib/src/shell.erl @@ -164,7 +164,15 @@ server(StartSync) -> RT = ets:new(?RECORDS, [public,ordered_set]), _ = initiate_records(Bs, RT), process_flag(trap_exit, true), - %% Store function definitions and types in an ets table. + %% Store local definitions of functions, records and types in an ets table. + %% {function_def, {M,F,A}},string()} i.e. "func(X) -> X." + %% {function, {M,F,A}},fun()} i.e. rewrite of the function_def with the same MFA into a fun "fun(X)-> X end." + %% {function_type_def, {M,F,A}},string()} i.e. "-spec func(X :: integer()) -> X." + %% {{function_type, {M,F,A}}, attr_form()} i.e. The intermediate representation of the spec, used for type traversal in edlin_type_suggestion.erl + %% {type_def, {M,F,A}},string()} i.e. "-type foo() :: bar() | baz()" + %% {type, {M,F,A}}, attr_form()} i.e. The intermediate representation of the type, used for type traversal in edlin_type_suggestion.erl + %% {record_def, {M,F,A}},string()} i.e. "-record(foo, {bar :: baz()})." + %% The attr_form() of the record is saved in the RT table. FT = ets:new(user_functions, [public,ordered_set]), %% Check if we're in user restricted mode. @@ -308,11 +316,13 @@ get_command(Prompt, Eval, Bs, RT, FT, Ds) -> [{atom, _, FunName}, {'(', _}|_] -> case erl_parse:parse_form(Toks) of {ok, FunDef} -> - case {edlin_expand:shell_default_or_bif(atom_to_list(FunName)), shell:local_func(FunName)} of + FunName1 = lists:flatten(io_lib:fwrite("~tw",[FunName])), + case {edlin_expand:shell_default_or_bif(FunName1), shell:local_func(FunName1)} of {"user_defined", false} -> - FakeLine =reconstruct(FunDef, FunName), + FunDef1 = lists:flatten(escape_quotes(lists:flatten(erl_pp:form(FunDef)))), + FakeLine = reconstruct(FunDef, FunName), {done, {ok, FakeResult, _}, _} = erl_scan:tokens( - [], "fd("++ atom_to_list(FunName) ++ ", " ++ FakeLine ++ ").\n", + [], "fd("++ FunName1 ++ ", " ++ FakeLine ++ ", \"" ++ FunDef1 ++ "\").\n", {1,1}, [text,{reserved_word_fun,fun erl_scan:reserved_word/1}]), erl_eval:extended_parse_exprs(FakeResult); _ -> erl_eval:extended_parse_exprs(Toks) @@ -340,7 +350,26 @@ get_command(Prompt, Eval, Bs, RT, FT, Ds) -> end, Pid = spawn_link(Parse), get_command1(Pid, Eval, Bs, RT, FT, Ds). - +escape_quotes(String) -> escape_quotes(String, []). + +escape_quotes([], Acc) -> + % When we've processed all characters, reverse the accumulator + % because we've been prepending for efficiency reasons. + lists:reverse(Acc); + +escape_quotes([$\\, $\" | Rest], Acc) -> + % If we find an escaped quote (\"), + % we escape the backslash and the quote (\\\") and continue. + escape_quotes(Rest, [$\", $\\, $\\, $\\ | Acc]); + +escape_quotes([$\" | Rest], Acc) -> + % If we find a quote ("), + % we escape it (\\") and continue. + escape_quotes(Rest, [$\", $\\ | Acc]); + +escape_quotes([Char | Rest], Acc) -> + % In case of any other character, we keep it as is. + escape_quotes(Rest, [Char | Acc]). reconstruct(Fun, Name) -> lists:flatten(erl_pp:expr(reconstruct1(Fun, Name))). reconstruct1({function, Anno, Name, Arity, Clauses}, Name) -> @@ -1066,7 +1095,8 @@ init_dict([]) -> true. %% handled internally - it should return 'true' for all local functions %% handled in this module (i.e. those that are not eventually handled by %% non_builtin_local_func/3 (user_default/shell_default). -local_func() -> [v,h,b,f,fl,rd,rf,rl,rp,rr,history,results,catch_exception]. +%% fd, ft and td should not be exposed to the user +local_func() -> [v,h,b,f,ff,fl,lf,lr,lt,rd,rf,rl,rp,rr,tf,save_module,history,results,catch_exception]. local_func(Func) -> lists:member(Func, local_func()). local_func(v, [{integer,_,V}], Bs, Shell, _RT, _FT, _Lf, _Ef) -> @@ -1092,21 +1122,70 @@ local_func(f, [{var,_,Name}], Bs, _Shell, _RT, _FT, _Lf, _Ef) -> {value,ok,erl_eval:del_binding(Name, Bs)}; local_func(f, [_Other], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> erlang:raise(error, function_clause, [{shell,f,1}]); -local_func(fl, [], Bs, _Shell, _RT, FT, _Lf, _Ef) -> - {value, ets:tab2list(FT), Bs}; -local_func(fd, [{atom,_,FunName}, FunExpr], Bs, _Shell, _RT, FT, _Lf, _Ef) -> +%% Output local functions (with specs) +local_func(lf, [], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + Output = local_functions_and_specs(FT), + io:requests([{put_chars, unicode, Output++"\n"}, nl]), + {value, ok, Bs}; +%% Output local types +local_func(lt, [], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + Output = local_types(FT), + io:requests([{put_chars, unicode, Output}, nl]), + {value, ok, Bs}; +%% Output local records +local_func(lr, [], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + Output = local_records(FT), + io:requests([{put_chars, unicode, Output}, nl]), + {value, ok, Bs}; + +%% Undocumented function that outputs contents of FT to a module file +%% compiles it and loads it, its still in experimentation phase +%% In theory, you may want to be able to load a module in to local table +%% edit them, and then save it back to the file system. +%% You may also want to be able to save a test module. +local_func(save_module, [{string,_,PathToFile}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + [_Path, FileName] = string:split("/"++PathToFile, "/", trailing), + [Module, _] = string:split(FileName, ".", leading), + Module1 = io_lib:fwrite("~tw",[list_to_atom(Module)]), + Exports = [lists:flatten(io_lib:fwrite("~tw",[F]))++"/"++integer_to_list(A)||{F,A}<-local_defined_functions(FT)], + Output = ( + "-module("++Module1++").\n\n" ++ + "-export(["++lists:join(",",Exports)++"]).\n\n"++ + local_types(FT) ++ + local_records(FT) ++ + local_functions(FT) + ), + Ret = case filelib:is_file(PathToFile) of + false -> + write_and_compile_module(PathToFile, Output); + true -> + case io:get_line("File exists, do you want to overwrite it? (y/N)\n") of + "y\n" -> + write_and_compile_module(PathToFile, Output); + _ -> + io:format("Aborting save~n"), + ok + end + end, + {value, Ret, Bs}; +local_func(save_module, [_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> + erlang:raise(error, function_clause, [{shell,save_module,1}]); +local_func(fd, [{atom,_,FunName}, FunExpr, {string, _, FunDef}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> {value, Fun, []} = erl_eval:expr(FunExpr, []), {arity, Arity} = erlang:fun_info(Fun, arity), + %% unescape_quotes(FunDef) ets:insert(FT, [{{function, {shell_default, FunName, Arity}}, Fun}]), + ets:insert(FT, [{{function_def, {shell_default, FunName, Arity}}, FunDef}]), {value, ok, Bs}; -local_func(fd, [_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> - erlang:raise(error, function_clause, [{shell, fd, 1}]); +local_func(fd, [_,_,_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> + erlang:raise(error, function_clause, [{shell, fd, 3}]); local_func(ft, [{string, _, TypeDef}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> case erl_scan:tokens([], TypeDef, {1,1}, [text,{reserved_word_fun,fun erl_scan:reserved_word/1}]) of {done, {ok, Toks, _}, _} -> case erl_parse:parse_form(Toks) of {ok, {attribute,_,spec,{{FunName, Arity},_}}=AttrForm} -> ets:insert(FT, [{{function_type, {shell_default, FunName, Arity}}, AttrForm}]), + ets:insert(FT, [{{function_type_def, {shell_default, FunName, Arity}}, TypeDef}]), {value, ok, Bs}; {error,{_Location,M,ErrDesc}} -> ErrStr = io_lib:fwrite(<<"~ts">>, [M:format_error(ErrDesc)]), @@ -1118,12 +1197,33 @@ local_func(ft, [{string, _, TypeDef}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> end; local_func(ft, [_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> erlang:raise(error, function_clause, [{shell, ft, 1}]); +local_func(fl, [], Bs, _Shell, RT, FT, _Lf, _Ef) -> + LocalRecords = [R || {{record_def, R},_} <- ets:tab2list(FT)], + true = ets:delete_all_objects(FT), + lists:foreach(fun(Name) -> ets:delete(RT, Name) end, LocalRecords), + {value,ok,Bs}; +local_func(ff, [], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + ets:select_delete(FT, [{{{function_def, '_'},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{function, '_'},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{function_type_def, '_'},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{function_type, '_'},'_'}, [],[true]}]), + {value,ok,Bs}; +local_func(ff, [{atom, _, F}, {integer,_,A}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + M = shell_default, + ets:select_delete(FT, [{{{function_def, {M, F, A}},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{function, {M, F, A}},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{function_type_def, {M, F, A}},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{function_type, {M, F, A}},'_'}, [],[true]}]), + {value,ok,Bs}; +local_func(ff, [_,_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> + erlang:raise(error, function_clause, [{shell,ff,2}]); local_func(td, [{string, _, TypeDef}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> case erl_scan:tokens([], TypeDef, {1,1}, [text,{reserved_word_fun,fun erl_scan:reserved_word/1}]) of {done, {ok, Toks, _}, _} -> case erl_parse:parse_form(Toks) of {ok, {attribute,_,type,{TypeName, _, _}}=AttrForm} -> - ets:insert(FT, [{{type, TypeName}, AttrForm}]), + true = ets:insert(FT, [{{type, TypeName}, AttrForm}]), + true = ets:insert(FT, [{{type_def, TypeName}, TypeDef}]), {value, ok, Bs}; {error,{_Location,M,ErrDesc}} -> ErrStr = io_lib:fwrite(<<"~ts">>, [M:format_error(ErrDesc)]), @@ -1135,12 +1235,23 @@ local_func(td, [{string, _, TypeDef}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> end; local_func(td, [_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> erlang:raise(error, function_clause, [{shell, td, 1}]); -local_func(rd, [{string, _, TypeDef}], Bs, _Shell, RT, _FT, _Lf, _Ef) -> +local_func(tf, [], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + ets:select_delete(FT, [{{{type_def, '_'}, '_'}, [],[true]}]), + ets:select_delete(FT, [{{{type, '_'}, '_'}, [],[true]}]), + {value,ok,Bs}; +local_func(tf, [{atom,_,A}], Bs, _Shell, _RT, FT, _Lf, _Ef) -> + ets:select_delete(FT, [{{{type_def, A},'_'}, [],[true]}]), + ets:select_delete(FT, [{{{type, A},'_'}, [],[true]}]), + {value,ok,Bs}; +local_func(tf, [_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> + erlang:raise(error, function_clause, [{shell,tf,1}]); +local_func(rd, [{string, _, TypeDef}], Bs, _Shell, RT, FT, _Lf, _Ef) -> case erl_scan:tokens([], TypeDef, {1,1}, [text,{reserved_word_fun,fun erl_scan:reserved_word/1}]) of {done, {ok, Toks, _}, _} -> case erl_parse:parse_form(Toks) of - {ok,{attribute,_,_,_}=AttrForm} -> + {ok,{attribute,_,_,{TypeName,_}}=AttrForm} -> [_] = add_records([AttrForm], Bs, RT), + true = ets:insert(FT, [{{record_def, TypeName}, TypeDef}]), {value,ok,Bs}; {error,{_Location,M,ErrDesc}} -> ErrStr = io_lib:fwrite(<<"~ts">>, [M:format_error(ErrDesc)]), @@ -1152,15 +1263,16 @@ local_func(rd, [{string, _, TypeDef}], Bs, _Shell, RT, _FT, _Lf, _Ef) -> end; local_func(rd, [_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> erlang:raise(error, function_clause, [{shell, rd, 1}]); -local_func(rd, [{atom,_,RecName0},RecDef0], Bs, _Shell, RT, _FT, _Lf, _Ef) -> +local_func(rd, [{atom,_,RecName0},RecDef0], Bs, _Shell, RT, FT, _Lf, _Ef) -> RecDef = expand_value(RecDef0), RDs = lists:flatten(erl_pp:expr(RecDef)), RecName = io_lib:write_atom_as_latin1(RecName0), Attr = lists:concat(["-record(", RecName, ",", RDs, ")."]), {ok, Tokens, _} = erl_scan:string(Attr), case erl_parse:parse_form(Tokens) of - {ok,AttrForm} -> + {ok,{attribute,_,_,{TypeName,_}}=AttrForm} -> [RN] = add_records([AttrForm], Bs, RT), + true = ets:insert(FT, [{{record_def, TypeName}, RecDef}]), {value,RN,Bs}; {error,{_Location,M,ErrDesc}} -> ErrStr = io_lib:fwrite(<<"~ts">>, [M:format_error(ErrDesc)]), @@ -1168,15 +1280,19 @@ local_func(rd, [{atom,_,RecName0},RecDef0], Bs, _Shell, RT, _FT, _Lf, _Ef) -> end; local_func(rd, [_,_], _Bs, _Shell, _RT, _FT, _Lf, _Ef) -> erlang:raise(error, function_clause, [{shell,rd,2}]); -local_func(rf, [], Bs, _Shell, RT, _FT, _Lf, _Ef) -> +local_func(rf, [], Bs, _Shell, RT, FT, _Lf, _Ef) -> + ets:select_delete(FT, [{{{record_def, '_'},'_'}, [],[true]}]), true = ets:delete_all_objects(RT), {value,initiate_records(Bs, RT),Bs}; -local_func(rf, [A], Bs0, _Shell, RT, _FT, Lf, Ef) -> +local_func(rf, [A], Bs0, _Shell, RT, FT, Lf, Ef) -> {[Recs],Bs} = expr_list([A], Bs0, Lf, Ef), if '_' =:= Recs -> + ets:select_delete(FT, [{{{record_def, '_'},'_'}, [],[true]}]), true = ets:delete_all_objects(RT); true -> - lists:foreach(fun(Name) -> true = ets:delete(RT, Name) + lists:foreach(fun(Name) -> + true = ets:delete(RT, Name), + true = ets:delete(FT, {record_def, Name}) end, listify(Recs)) end, {value,ok,Bs}; @@ -1219,6 +1335,53 @@ local_func(F, As0, Bs0, _Shell, _RT, FT, Lf, Ef) when is_atom(F) -> {As,Bs} = expr_list(As0, Bs0, Lf, Ef), non_builtin_local_func(F,As,Bs, FT). +local_functions_and_specs(FT) -> + Output_functions = maps:from_list( + [{{FunNameAtom, Arity}, lists:flatten(FunDef)} ||{{function_def,{_, FunNameAtom, Arity}},FunDef} <- ets:tab2list(FT)]), + Output_function_specs = maps:from_list( + [{{FunNameAtom, Arity}, FunSpec}|| {{function_type_def, {_, FunNameAtom, Arity}}, FunSpec} <- ets:tab2list(FT)]), + Keys1 = maps:keys(Output_functions), + Keys2 = maps:keys(Output_function_specs), + Keys = lists:uniq(Keys1 ++ Keys2), + string:trim(lists:join($\n,lists:map(fun(Key) -> + Spec = maps:get(Key, Output_function_specs, nospec), + Def = maps:get(Key, Output_functions, nodef), + case {Spec, Def} of + {nospec, _} -> Def; + {_, nodef} -> + {FunName, Arity} = Key, + Spec ++ "%% " ++ lists:flatten(io_lib:fwrite("~tw",[FunName]))++ "/" ++ integer_to_list(Arity) ++ " not implemented"; + {_, _} -> Spec ++ Def + end + end, + Keys))). +local_defined_functions(FT) -> + [{F, A} ||{{function_def,{_, F, A}},_} <- ets:tab2list(FT)]. +local_functions(FT) -> + local_functions(local_defined_functions(FT), FT). +local_functions(Keys, FT) -> + lists:join($\n, + [begin + Spec = case ets:lookup(FT, {function_type_def, {shell_default, F, A}}) of + [{{function_type_def, {shell_default, F, A}}, Spec1}] -> Spec1; + [] -> "" + end, + [{{function_def, {shell_default, F, A}}, Def}] = ets:lookup(FT, {function_def, {shell_default, F, A}}), + Spec++Def + end || {F, A} <- Keys]). +%% Output local types +local_types(FT) -> + lists:join($\n, + [TypeDef||{{type_def, _},TypeDef} <- ets:tab2list(FT)]). +%% Output local records +local_records(FT) -> + lists:join($\n, + [RecDef||{{record_def, _},RecDef} <- ets:tab2list(FT)]). +write_and_compile_module(PathToFile, Output) -> + case file:write_file(PathToFile, unicode:characters_to_binary(Output)) of + ok -> c:c(PathToFile); + Error -> Error + end. non_builtin_local_func(F,As,Bs, FT) -> Arity = length(As), case erlang:function_exported(user_default, F, Arity) of diff --git a/lib/stdlib/src/shell_default.erl b/lib/stdlib/src/shell_default.erl index 669266d25504..fd11ecfae1c5 100644 --- a/lib/stdlib/src/shell_default.erl +++ b/lib/stdlib/src/shell_default.erl @@ -67,7 +67,15 @@ help() -> format("rr(File) -- read record information from File (wildcards allowed)\n"), format("rr(F,R) -- read selected record information from file(s)\n"), format("rr(F,R,O) -- read selected record information with options\n"), - format("** commands in module c **\n"), + format("lf() -- list locally defined functions\n"), + format("lt() -- list locally defined types\n"), + format("lr() -- list locally defined records\n"), + format("ff() -- forget all locally defined functions\n"), + format("ff({F,A}) -- forget locally defined function named as atom F and arity A\n"), + format("tf() -- forget all locally defined types\n"), + format("tf(T) -- forget locally defined type named as atom T\n"), + format("fl() -- forget all locally defined functions, types and records\n"), + format("save_module(FilePath) -- save all locally defined functions, types and records to a file\n"), c:help(), format("** commands in module i (interpreter interface) **\n"), format("ih() -- print help for the i module\n"), diff --git a/lib/stdlib/src/stdlib.app.src b/lib/stdlib/src/stdlib.app.src index 0f8925cc75ec..48a49ba4650d 100644 --- a/lib/stdlib/src/stdlib.app.src +++ b/lib/stdlib/src/stdlib.app.src @@ -116,6 +116,6 @@ dets]}, {applications, [kernel]}, {env, []}, - {runtime_dependencies, ["sasl-3.0","kernel-9.2","erts-@OTP-18765@","crypto-4.5", + {runtime_dependencies, ["sasl-3.0","kernel-9.2","erts-14.2","crypto-4.5", "compiler-5.0"]} ]}. diff --git a/lib/stdlib/src/stdlib.appup.src b/lib/stdlib/src/stdlib.appup.src index 6fc97ec312b0..cec6def4ce48 100644 --- a/lib/stdlib/src/stdlib.appup.src +++ b/lib/stdlib/src/stdlib.appup.src @@ -55,7 +55,8 @@ {<<"^5\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^5\\.0\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^5\\.1$">>,[restart_new_emulator]}, - {<<"^5\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}], + {<<"^5\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, + {<<"^5\\.1\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}], [{<<"^3\\.15$">>,[restart_new_emulator]}, {<<"^3\\.15\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.15\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, @@ -83,4 +84,5 @@ {<<"^5\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^5\\.0\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^5\\.1$">>,[restart_new_emulator]}, - {<<"^5\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}]}. + {<<"^5\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, + {<<"^5\\.1\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}]}. diff --git a/lib/stdlib/test/argparse_SUITE.erl b/lib/stdlib/test/argparse_SUITE.erl index a63b8867d25f..21a0b455de04 100644 --- a/lib/stdlib/test/argparse_SUITE.erl +++ b/lib/stdlib/test/argparse_SUITE.erl @@ -45,6 +45,7 @@ usage/0, usage/1, usage_required_args/0, usage_required_args/1, usage_template/0, usage_template/1, + usage_args_ordering/0, usage_args_ordering/1, parser_error_usage/0, parser_error_usage/1, command_usage/0, command_usage/1, usage_width/0, usage_width/1, @@ -52,7 +53,8 @@ validator_exception/0, validator_exception/1, validator_exception_format/0, validator_exception_format/1, - run_handle/0, run_handle/1 + run_handle/0, run_handle/1, + run_args_ordering/0, run_args_ordering/1 ]). -include_lib("stdlib/include/assert.hrl"). @@ -70,14 +72,14 @@ groups() -> very_short, multi_short, proxy_arguments ]}, {usage, [parallel], [ - usage, usage_required_args, usage_template, + usage, usage_required_args, usage_template, usage_args_ordering, parser_error_usage, command_usage, usage_width ]}, {validator, [parallel], [ validator_exception, validator_exception_format ]}, {run, [parallel], [ - run_handle + run_handle, run_args_ordering ]} ]. @@ -743,10 +745,12 @@ usage() -> usage(Config) when is_list(Config) -> Cmd = ubiq_cmd(), - Usage = "Usage:\n erl start {crawler|doze} [-lrfv] [-s ...] [-z ] [-m ] [-b ]\n" - " [-g ] [-t ] ---maybe-req -y --yyy [-u ] [-c ]\n" - " [-q ] [-w ] [--unsafe ] [--safe ] [-foobar ] [--force]\n" - " [-i ] [--req ] [--float ] []\n\n" + Usage = "Usage:\n" + " erl start {crawler|doze} [-rfvl] [--force] [-i ] [--req ]\n" + " [--float ] [-s ...] [-z ] [-m ] [-b ] [-g ]\n" + " [-t ] ---maybe-req -y --yyy [-u ] [-c ] [-q ]\n" + " [-w ] [--unsafe ] [--safe ] [-foobar ] []\n" + "\n" "Subcommands:\n" " crawler controls crawler behaviour\n" " doze dozes a bit\n\n" @@ -754,6 +758,12 @@ usage(Config) when is_list(Config) -> " server server to start\n" " optpos optional positional (int)\n\n" "Optional arguments:\n" + " -r recursive\n" + " -f, --force force\n" + " -v verbosity level\n" + " -i interval set (int >= 1)\n" + " --req required optional, right?\n" + " --float floating-point long form argument (float), default: 3.14\n" " -s initial shards (int)\n" " -z between (1 <= int <= 10)\n" " -l maybe lower (int <= 10)\n" @@ -769,13 +779,7 @@ usage(Config) when is_list(Config) -> " -w atom choice (choice: one, two)\n" " --unsafe unsafe atom (atom)\n" " --safe safe atom (existing atom)\n" - " -foobar foobaring option\n" - " -r recursive\n" - " -f, --force force\n" - " -v verbosity level\n" - " -i interval set (int >= 1)\n" - " --req required optional, right?\n" - " --float floating-point long form argument (float), default: 3.14\n", + " -foobar foobaring option\n", ?assertEqual(Usage, unicode:characters_to_list(argparse:help(Cmd, #{progname => "erl", command => ["start"]}))), FullCmd = "Usage:\n erl" @@ -793,13 +797,16 @@ usage(Config) when is_list(Config) -> " --float floating-point long form argument (float), default: 3.14\n", ?assertEqual(FullCmd, unicode:characters_to_list(argparse:help(Cmd, #{progname => erl}))), - CrawlerStatus = "Usage:\n erl status crawler [-rfv] [---extra ] [--force] [-i ]\n" - " [--req ] [--float ]\n\nOptional arguments:\n" - " ---extra extra option very deep\n -r recursive\n" - " -f, --force force\n -v verbosity level\n" + CrawlerStatus = "Usage:\n erl status crawler [-rfv] [--force] [-i ] [--req ]\n" + " [--float ] [---extra ]\n\n" + "Optional arguments:\n" + " -r recursive\n" + " -f, --force force\n" + " -v verbosity level\n" " -i interval set (int >= 1)\n" " --req required optional, right?\n" - " --float floating-point long form argument (float), default: 3.14\n", + " --float floating-point long form argument (float), default: 3.14\n" + " ---extra extra option very deep\n", ?assertEqual(CrawlerStatus, unicode:characters_to_list(argparse:help(Cmd, #{progname => "erl", command => ["status", "crawler"]}))), ok. @@ -856,6 +863,30 @@ usage_template(Config) when is_list(Config) -> unicode:characters_to_list(argparse:help(CmdISO, #{}))), ok. +usage_args_ordering() -> + [{doc, "Tests the ordering of arguments in usage text"}]. + +usage_args_ordering(Config) when is_list(Config) -> + Cmd = #{arguments => [ + #{name => first}, + #{name => second} + ], + commands => #{ + "cmd" => #{arguments => [ + #{name => third}, + #{name => fourth} + ]}} + }, + ?assertEqual("Usage:\n " ++ prog() ++ " cmd \n" + "\n" + "Arguments:\n" + " first first\n" + " second second\n" + " third third\n" + " fourth fourth\n", + unicode:characters_to_list(argparse:help(Cmd, #{command => ["cmd"]}))), + ok. + parser_error_usage() -> [{doc, "Tests that parser errors have corresponding usage text"}]. @@ -1060,4 +1091,18 @@ run_handle(Config) when is_list(Config) -> argparse:run(["map", "arg"], #{commands => #{"map" => #{ handler => {maps, to_list}, arguments => [#{name => arg}]}}}, - #{})). \ No newline at end of file + #{})). + +run_args_ordering() -> + [{doc, "Test that positional arguments are parsed in the correct order"}]. + +run_args_ordering(Config) when is_list(Config) -> + ?assertEqual([{first,"1"},{second,"2"},{third,"3"},{fourth,"4"}], + argparse:run(["cmd", "1", "2", "3", "4"], + #{arguments => [#{name => first}, #{name => second}], + commands => #{ + "cmd" => #{ + handler => {maps, to_list}, + arguments => [#{name => third}, #{name => fourth}]} + }}, + #{progname => example})). diff --git a/lib/stdlib/test/edlin_context_SUITE.erl b/lib/stdlib/test/edlin_context_SUITE.erl index 5e84a9019940..fe25293fbb80 100644 --- a/lib/stdlib/test/edlin_context_SUITE.erl +++ b/lib/stdlib/test/edlin_context_SUITE.erl @@ -42,8 +42,8 @@ get_context(_Config) -> {term} = edlin_context:get_context(lists:reverse("h(file,open), h(file")), {term} = edlin_context:get_context(lists:reverse("h(file,open), h(file,open")), {term, [{call, "h(file,open)"}], {call, "h(file,open)"}} = edlin_context:get_context(lists:reverse("h(file,open), h(file,open)")), - {function} = edlin_context:get_context(lists:reverse("file:")), - {function} = edlin_context:get_context(lists:reverse("file:open")), + {function, "file"} = edlin_context:get_context(lists:reverse("file:")), + {function, "file"} = edlin_context:get_context(lists:reverse("file:open")), {function, "file", "open", [], [], []} = edlin_context:get_context(lists:reverse("file:open(")), {string} = edlin_context:get_context(lists:reverse("file:open(\"")), {string} = edlin_context:get_context(lists:reverse("file:open(\"/")), diff --git a/lib/stdlib/test/ets_SUITE.erl b/lib/stdlib/test/ets_SUITE.erl index 73fd3df43a08..a4105a3c8176 100644 --- a/lib/stdlib/test/ets_SUITE.erl +++ b/lib/stdlib/test/ets_SUITE.erl @@ -62,7 +62,7 @@ -export([ordered/1, ordered_match/1, interface_equality/1, fixtable_next/1, fixtable_iter_bag/1, fixtable_insert/1, rename/1, rename_unnamed/1, evil_rename/1, - update_element/1, update_counter/1, evil_update_counter/1, partly_bound/1, match_heavy/1]). + update_element/1, update_element_default/1, update_counter/1, evil_update_counter/1, partly_bound/1, match_heavy/1]). -export([update_counter_with_default/1]). -export([update_counter_with_default_bad_pos/1]). -export([update_counter_table_growth/1]). @@ -147,7 +147,7 @@ all() -> {group, lookup_element}, {group, misc}, {group, files}, {group, heavy}, {group, insert_list}, ordered, ordered_match, interface_equality, fixtable_next, fixtable_iter_bag, fixtable_insert, - rename, rename_unnamed, evil_rename, update_element, + rename, rename_unnamed, evil_rename, update_element, update_element_default, update_counter, evil_update_counter, update_counter_with_default, update_counter_with_default_bad_pos, @@ -2552,22 +2552,26 @@ update_element_do(Tab,Tuple,Key,UpdPos) -> Length = tuple_size(Values), 29 = Length, - PosValArgF = fun(ToIx, ResList, [Pos | PosTail], Rand, MeF) -> + PosValArgF = fun MeF(ToIx, ResList, [Pos | PosTail], Rand) -> NextIx = (ToIx+Rand) rem Length, - MeF(NextIx, [{Pos,element(ToIx+1,Values)} | ResList], PosTail, Rand, MeF); + MeF(NextIx, [{Pos,element(ToIx+1,Values)} | ResList], PosTail, Rand); - (_ToIx, ResList, [], _Rand, _MeF) -> + MeF(_ToIx, ResList, [], _Rand) -> ResList; - (ToIx, [], Pos, _Rand, _MeF) -> + MeF(ToIx, [], Pos, _Rand) -> {Pos, element(ToIx+1,Values)} % single {pos,value} arg end, UpdateF = fun(ToIx,Rand) -> - PosValArg = PosValArgF(ToIx,[],UpdPos,Rand,PosValArgF), + PosValArg = PosValArgF(ToIx,[],UpdPos,Rand), %%io:format("update_element(~p)~n",[PosValArg]), ArgHash = erlang:phash2({Tab,Key,PosValArg}), true = ets:update_element(Tab, Key, PosValArg), + [DefaultObj] = ets:lookup(Tab, Key), + NewKey = make_ref(), + true = ets:update_element(Tab, NewKey, PosValArg, DefaultObj), + true = [update_tuple({ets:info(Tab, keypos), NewKey}, DefaultObj)] =:= ets:lookup(Tab, NewKey), ArgHash = erlang:phash2({Tab,Key,PosValArg}), NewTuple = update_tuple(PosValArg,Tuple), [NewTuple] = ets:lookup(Tab,Key), @@ -2578,27 +2582,27 @@ update_element_do(Tab,Tuple,Key,UpdPos) -> || I <- lists:seq(1, tuple_size(NewTuple))] end, - LoopF = fun(_FromIx, Incr, _Times, Checksum, _MeF) when Incr >= Length -> + LoopF = fun MeF(_FromIx, Incr, _Times, Checksum) when Incr >= Length -> Checksum; % done - (FromIx, Incr, 0, Checksum, MeF) -> - MeF(FromIx, Incr+1, Length, Checksum, MeF); + MeF(FromIx, Incr, 0, Checksum) -> + MeF(FromIx, Incr+1, Length, Checksum); - (FromIx, Incr, Times, Checksum, MeF) -> + MeF(FromIx, Incr, Times, Checksum) -> ToIx = (FromIx + Incr) rem Length, UpdateF(ToIx,Checksum), if Incr =:= 0 -> UpdateF(ToIx,Checksum); % extra update to same value true -> true end, - MeF(ToIx, Incr, Times-1, Checksum+ToIx+1, MeF) + MeF(ToIx, Incr, Times-1, Checksum+ToIx+1) end, FirstTuple = Tuple, true = ets:insert(Tab,FirstTuple), [FirstTuple] = ets:lookup(Tab,Key), - Checksum = LoopF(0, 1, Length, 0, LoopF), + Checksum = LoopF(0, 1, Length, 0), Checksum = (Length-1)*Length*(Length+1) div 2, % if Length is a prime ok. @@ -2616,6 +2620,7 @@ update_element_neg(Opts) -> update_element_neg_do(Set), ets:delete(Set), {'EXIT',{badarg,_}} = (catch ets:update_element(Set,key,{2,1})), + {'EXIT',{badarg,_}} = (catch ets:update_element(Set,key,{2,1},{a,b})), run_if_valid_opts( [ordered_set | Opts], @@ -2623,13 +2628,16 @@ update_element_neg(Opts) -> OrdSet = ets_new(ordered_set, OptsOrdSet), update_element_neg_do(OrdSet), ets:delete(OrdSet), - {'EXIT',{badarg,_}} = (catch ets:update_element(OrdSet,key,{2,1})) + {'EXIT',{badarg,_}} = (catch ets:update_element(OrdSet,key,{2,1})), + {'EXIT',{badarg,_}} = (catch ets:update_element(OrdSet,key2,{2,1},{a,b})) end), Bag = ets_new(bag,[bag | Opts]), DBag = ets_new(duplicate_bag,[duplicate_bag | Opts]), {'EXIT',{badarg,_}} = (catch ets:update_element(Bag,key,{2,1})), + {'EXIT',{badarg,_}} = (catch ets:update_element(Bag,key,{2,1},{key,0})), {'EXIT',{badarg,_}} = (catch ets:update_element(DBag,key,{2,1})), + {'EXIT',{badarg,_}} = (catch ets:update_element(DBag,key,{2,1},{key,0})), true = ets:delete(Bag), true = ets:delete(DBag), ok. @@ -2643,6 +2651,8 @@ update_element_neg_do(T) -> ArgHash = erlang:phash2({T,key,Arg3}), {'EXIT',{badarg,_}} = (catch ets:update_element(T,key,Arg3)), ArgHash = erlang:phash2({T,key,Arg3}), + {'EXIT',{badarg,_}} = (catch ets:update_element(T,key2,Arg3,Object)), + ArgHash = erlang:phash2({T,key,Arg3}), [Object] = ets:lookup(T,key) end, @@ -2667,6 +2677,32 @@ update_element_neg_do(T) -> ok. +update_element_default(Config) when is_list(Config) -> + EtsMem = etsmem(), + repeat_for_opts(fun update_element_default_opts/1), + verify_etsmem(EtsMem). + + +update_element_default_opts(Opts) -> + lists:foreach( + fun({Type, {Key, Pos}}) -> + run_if_valid_opts( + [Type, {keypos, Pos} | Opts], + fun(TabOpts) -> + Tab = ets_new(Type, TabOpts), + true = ets:update_element(Tab, Key, {3, b}, {key1, key2, a, x}), + [{key1, key2, b, x}] = ets:lookup(Tab, Key), + true = ets:update_element(Tab, Key, {3, c}, {key1, key2, a, y}), + [{key1, key2, c, x}] = ets:lookup(Tab, Key), + ets:delete(Tab) + end + ) + end, + [{Type, KeyPos} || Type <- [set, ordered_set], KeyPos <- [{key1, 1}, {key2, 2}]] + ), + ok. + + %% test various variants of update_counter. update_counter(Config) when is_list(Config) -> EtsMem = etsmem(), @@ -9515,9 +9551,20 @@ error_info(_Config) -> {update_element, ['$Tab', no_key, {2, new}], [no_fail]}, {update_element, [BagTab, no_key, {2, bagged}]}, {update_element, [OneKeyTab, one, not_tuple]}, - {update_element, [OneKeyTab, one, {0, new}]}, + {update_element, [OneKeyTab, one, {0, new}], [{error_term, position}]}, {update_element, [OneKeyTab, one, {1, new}], [{error_term,keypos}]}, - {update_element, [OneKeyTab, one, {4, new}]}, + {update_element, [OneKeyTab, one, {4, new}], [{error_term, position}]}, + + {update_element, ['$Tab', no_key, {2, new}, {no_key, old}], [no_fail]}, + {update_element, ['$Tab', no_key, {0, new}, {no_key, old}], [{error_term, position}]}, + {update_element, ['$Tab', no_key, {1, new}, {no_key, old}], [{error_term, keypos}]}, + {update_element, ['$Tab', no_key, {4, new}, {no_key, old}], [{error_term, position}]}, + {update_element, ['$Tab', no_key, {4, new}, not_tuple]}, + {update_element, [BagTab, no_key, {1, bagged}, {no_key, old}], []}, + {update_element, [OneKeyTab, no_key, {0, new}, {no_key, old}], [{error_term, position}]}, + {update_element, [OneKeyTab, no_key, {1, new}, {no_key, old}], [{error_term, keypos}]}, + {update_element, [OneKeyTab, no_key, {4, new}, {no_key, old}], [{error_term, position}]}, + {update_element, [OneKeyTab, no_key, {4, new}, not_tuple]}, {whereis, [{bad,name}], [no_table]} ], diff --git a/lib/stdlib/test/shell_SUITE.erl b/lib/stdlib/test/shell_SUITE.erl index d337077f1340..995eb48e5187 100644 --- a/lib/stdlib/test/shell_SUITE.erl +++ b/lib/stdlib/test/shell_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2004-2022. All Rights Reserved. +%% Copyright Ericsson AB 2004-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -25,7 +25,7 @@ bs_match_misc_SUITE/1, bs_match_int_SUITE/1, bs_match_tail_SUITE/1, bs_match_bin_SUITE/1, bs_construct_SUITE/1, - prompt_width/1, + prompt_width/1,local_definitions_save_to_module_and_forget/1, refman_bit_syntax/1, progex_bit_syntax/1, progex_records/1, progex_lc/1, progex_funs/1, @@ -78,7 +78,7 @@ suite() -> all() -> [forget, known_bugs, otp_5226, otp_5327, otp_5435, otp_5195, otp_5915, otp_5916, - prompt_width, + prompt_width,local_definitions_save_to_module_and_forget, start_interactive, whereis, {group, bits}, {group, refman}, {group, progex}, {group, tickets}, {group, restricted}, {group, records}, {group, definitions}]. @@ -189,13 +189,14 @@ comm_err(<<"ugly().">>), comm_err(<<"1 - 2.">>), %% Make sure we test all local shell functions in a restricted shell. LocalFuncs = shell:local_func(), -[] = lists:subtract(LocalFuncs, [v,h,b,f,fl,rd,rf,rl,rp,rr,history,results,catch_exception]), +[] = lists:subtract(LocalFuncs, [v,h,b,f,fl,ff,lf,lr,lt,rd,rf,rl,rp,rr,tf,save_module,history,results,catch_exception]), LocalFuncs2 = [ <<"A = 1.\nv(1).">>, <<"h().">>, <<"b().">>, <<"f().">>, <<"f(A).">>, - <<"fl()">>, <<"rd(foo,{bar}).">>, <<"rf().">>, <<"rf(foo).">>, <<"rl().">>, <<"rl(foo).">>, <<"rp([hej]).">>, - <<"rr(shell).">>, <<"rr(shell, shell_state).">>, <<"rr(shell,shell_state,[]).">>, - <<"history(20).">>, <<"results(20).">>, <<"catch_exception(0).">>], + <<"fl()">>, <<"ff()">>, <<"ff(my_func,1)">>, <<"lf()">>, <<"lr()">>, <<"lt()">>, + <<"rd(foo,{bar}).">>, <<"rf().">>, <<"rf(foo).">>, <<"rl().">>, <<"rl(foo).">>, <<"rp([hej]).">>, + <<"rr(shell).">>, <<"rr(shell, shell_state).">>, <<"rr(shell,shell_state,[]).">>, <<"tf()">>, <<"tf(hej)">>, + <<"save_module(\"src/my_module.erl\")">>, <<"history(20).">>, <<"results(20).">>, <<"catch_exception(0).">>], lists:foreach(fun(LocalFunc) -> try ("exception exit: restricted shell does not allow"++_Rest) = Error = local_func_error_t(LocalFunc), @@ -649,6 +650,92 @@ typed_records(Config) when is_list(Config) -> file:delete(Test), ok. +local_definitions_save_to_module_and_forget(Config) when is_list(Config) -> + %% extra dot on the empty line is a consequence of dotify + "ok.\n-type hej() :: integer().\n.\nok.\n" = t( + <<"-type hej() :: integer().\n" + "lt().">>), + "ok.\n-record(svej,{a}).\n.\nok.\n" = t( + <<"-record(svej, {a}).\n" + "lr().">>), + "ok.\nok.\n-spec my_func(X) -> X.\nmy_func(X) ->\n X.\n.\nok.\n" = t( + <<"-spec my_func(X) -> X.\n" + "my_func(X) -> X.\n" + "lf().">>), + %% Save local definitions to a module + U = unicode:characters_to_binary("😊"), + "ok.\nok.\nok.\nok.\nok.\nok.\n{ok,'MY_MODULE'}.\n" = t({ + <<"-type hej() :: integer().\n" + "-record(svej, {a :: hej()}).\n" + "my_func(#svej{a=A}) -> A.\n" + "-spec not_implemented(X) -> X.\n" + "-spec 'my_func",U/binary,"'(X) -> X.\n" + "'my_func",U/binary,"'(#svej{a=A}) -> A.\n" + "save_module(\"MY_MODULE.erl\").">>, unicode}), + %% Read back the newly created module + {ok,<<"-module('MY_MODULE').\n\n" + "-export([my_func/1,'my_func",240,159,152,138,"'/1]).\n\n" + "-type hej() :: integer().\n" + "-record(svej,{a :: hej()}).\n" + "my_func(#svej{a = A}) ->\n" + " A.\n\n" + "-spec 'my_func",240,159,152,138,"'(X) -> X.\n" + "'my_func",240,159,152,138,"'(#svej{a = A}) ->\n" + " A.\n">>} = file:read_file("MY_MODULE.erl"), + file:delete("MY_MODULE.erl"), + + %% Forget one locally defined type + "ok.\nok.\nok.\n-type svej() :: integer().\n.\nok.\n" = t( + <<"-type hej() :: integer().\n" + "-type svej() :: integer().\n" + "tf(hej).\n" + "lt().">>), + %% Forget one locally defined record + "ok.\nok.\nok.\n-record(hej,{a}).\n.\nok.\n" = t( + <<"-record(hej, {a}).\n" + "-record(svej, {a}).\n" + "rf(svej).\n" + "lr().">>), + %% Forget one locally defined function + "ok.\nok.\nok.\nok.\nok.\n-spec my_func2(X) -> X.\nmy_func2(X) ->\n X.\n.\nok.\n" = t( + <<"-spec my_func(X) -> X.\n" + "my_func(X) -> X.\n" + "-spec my_func2(X) -> X.\n" + "my_func2(X) -> X.\n" + "ff(my_func,1).\n" + "lf().">>), + %% Forget all locally defined types + "ok.\nok.\nok.\n.\nok.\n" = t( + <<"-type hej() :: integer().\n" + "-type svej() :: integer().\n" + "tf().\n" + "lt().">>), + %% Forget all locally defined records + "ok.\nok.\n[]\n.\nok.\n" = t( + <<"-record(hej, {a}).\n" + "-record(svej, {a}).\n" + "rf().\n" + "lr().">>), + %% Forget all locally defined functions + "ok.\nok.\nok.\nok.\nok.\n\n.\nok.\n" = t( + <<"-spec my_func(X) -> X.\n" + "my_func(X) -> X.\n" + "-spec my_func2(X) -> X.\n" + "my_func2(X) -> X.\n" + "ff().\n" + "lf().">>), + %% Forget all local definitions + "ok.\nok.\nok.\nok.\nok.\n\n.\nok.\n.\nok.\n.\nok.\n" = t( + <<"-type hej() :: integer().\n" + "-record(svej, {a}).\n " + "-spec my_func(X) -> X.\n" + "my_func(X) -> X.\n" + "fl().\n" + "lf().\n" + "lt().\n" + "lr().\n">>), + ok. + %% Known bugs. known_bugs(Config) when is_list(Config) -> %% erl_eval:merge_bindings/2 cannot handle _removal_ of bindings. diff --git a/lib/stdlib/vsn.mk b/lib/stdlib/vsn.mk index d4fc23fc153c..b6a1d37cb364 100644 --- a/lib/stdlib/vsn.mk +++ b/lib/stdlib/vsn.mk @@ -1 +1 @@ -STDLIB_VSN = 5.1.1 +STDLIB_VSN = 5.2 diff --git a/lib/tftp/doc/src/introduction.xml b/lib/tftp/doc/src/introduction.xml index 2ec4b58cee07..1c2b8d3acf49 100644 --- a/lib/tftp/doc/src/introduction.xml +++ b/lib/tftp/doc/src/introduction.xml @@ -4,7 +4,7 @@
- 19972018 + 19972023 Ericsson AB. All Rights Reserved. diff --git a/lib/tftp/doc/src/notes.xml b/lib/tftp/doc/src/notes.xml index 81fda3733652..5232e115927c 100644 --- a/lib/tftp/doc/src/notes.xml +++ b/lib/tftp/doc/src/notes.xml @@ -33,7 +33,24 @@ notes.xml
-
Tftp 1.1 +
Tftp 1.1.1 + +
Fixed Bugs and Malfunctions + + +

+ Replaced unintentional Erlang Public License 1.1 headers + in some files with the intended Apache License 2.0 + header.

+

+ Own Id: OTP-18815 Aux Id: PR-7780

+ + +
+ +
+ +
Tftp 1.1
Improvements and New Features diff --git a/lib/tftp/vsn.mk b/lib/tftp/vsn.mk index 8c81f083ccd6..86a4e8d6de36 100644 --- a/lib/tftp/vsn.mk +++ b/lib/tftp/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = tftp -TFTP_VSN = 1.1 +TFTP_VSN = 1.1.1 PRE_VSN = APP_VSN = "$(APPLICATION)-$(TFTP_VSN)$(PRE_VSN)" diff --git a/lib/wx/doc/src/notes.xml b/lib/wx/doc/src/notes.xml index 84954e7f59fb..62111551dec5 100644 --- a/lib/wx/doc/src/notes.xml +++ b/lib/wx/doc/src/notes.xml @@ -32,6 +32,24 @@

This document describes the changes made to the wxErlang application.

+
Wx 2.4 + +
Improvements and New Features + + +

+ Guards have been added to gen_*:start* API + functions to catch bad arguments earlier. Before this + change, in some cases, a bad argument could tag along and + cause the server to fail later, right after start.

+

+ Own Id: OTP-18857 Aux Id: GH-7685

+ + +
+ +
+
Wx 2.3.1
Fixed Bugs and Malfunctions @@ -70,6 +88,21 @@
+
Wx 2.2.2.1 + +
Fixed Bugs and Malfunctions + + +

The wx application would fail to build on macOS + with Xcode 15.

+

+ Own Id: OTP-18768 Aux Id: PR-7670

+ + +
+ +
+
Wx 2.2.2
Fixed Bugs and Malfunctions diff --git a/lib/wx/vsn.mk b/lib/wx/vsn.mk index 49636f859be2..441246c07238 100644 --- a/lib/wx/vsn.mk +++ b/lib/wx/vsn.mk @@ -1 +1 @@ -WX_VSN = 2.3.1 +WX_VSN = 2.4 diff --git a/lib/xmerl/doc/src/notes.xml b/lib/xmerl/doc/src/notes.xml index 6ba05c6a3663..886fd8230aea 100644 --- a/lib/xmerl/doc/src/notes.xml +++ b/lib/xmerl/doc/src/notes.xml @@ -32,6 +32,22 @@

This document describes the changes made to the Xmerl application.

+
Xmerl 1.3.34 + +
Fixed Bugs and Malfunctions + + +

+ Update all <tt> html tags to be <code> + instead.

+

+ Own Id: OTP-18799 Aux Id: PR-7695

+ + +
+ +
+
Xmerl 1.3.33
Fixed Bugs and Malfunctions diff --git a/lib/xmerl/src/xmerl.erl b/lib/xmerl/src/xmerl.erl index d3b968e32ef7..61a55002aebd 100644 --- a/lib/xmerl/src/xmerl.erl +++ b/lib/xmerl/src/xmerl.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2003-2016. All Rights Reserved. +%% Copyright Ericsson AB 2003-2023. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. diff --git a/lib/xmerl/vsn.mk b/lib/xmerl/vsn.mk index 39f3de400cfd..1c716b2073b2 100644 --- a/lib/xmerl/vsn.mk +++ b/lib/xmerl/vsn.mk @@ -1 +1 @@ -XMERL_VSN = 1.3.33 +XMERL_VSN = 1.3.34 diff --git a/make/install_dir_data.sh.in b/make/install_dir_data.sh.in index 8f050ed22b4b..5fb1392a9e75 100644 --- a/make/install_dir_data.sh.in +++ b/make/install_dir_data.sh.in @@ -2,7 +2,7 @@ # # %CopyrightBegin% # -# Copyright Ericsson AB 2019. All Rights Reserved. +# Copyright Ericsson AB 2019-2023. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/otp_versions.table b/otp_versions.table index 02b36c005bc6..91072bb17995 100644 --- a/otp_versions.table +++ b/otp_versions.table @@ -1,9 +1,12 @@ +OTP-26.2.1 : erts-14.2.1 ssh-5.1.1 # asn1-5.2.1 common_test-1.26 compiler-8.4.1 crypto-5.4 debugger-5.3.3 dialyzer-5.1.2 diameter-2.3.1 edoc-1.2.1 eldap-1.2.12 erl_docgen-1.5.2 erl_interface-5.5 et-1.7 eunit-2.9 ftp-1.2.1 inets-9.1 jinterface-1.14 kernel-9.2 megaco-4.5 mnesia-4.23 observer-2.15.1 odbc-2.14.1 os_mon-2.9.1 parsetools-2.5 public_key-1.15 reltool-1.0 runtime_tools-2.0.1 sasl-4.2.1 snmp-5.15 ssl-11.1 stdlib-5.2 syntax_tools-3.1 tftp-1.1.1 tools-3.6 wx-2.4 xmerl-1.3.34 : +OTP-26.2 : asn1-5.2.1 common_test-1.26 crypto-5.4 debugger-5.3.3 dialyzer-5.1.2 diameter-2.3.1 edoc-1.2.1 eldap-1.2.12 erl_docgen-1.5.2 erl_interface-5.5 erts-14.2 eunit-2.9 ftp-1.2.1 inets-9.1 kernel-9.2 mnesia-4.23 os_mon-2.9.1 public_key-1.15 runtime_tools-2.0.1 ssh-5.1 ssl-11.1 stdlib-5.2 tftp-1.1.1 wx-2.4 xmerl-1.3.34 # compiler-8.4.1 et-1.7 jinterface-1.14 megaco-4.5 observer-2.15.1 odbc-2.14.1 parsetools-2.5 reltool-1.0 sasl-4.2.1 snmp-5.15 syntax_tools-3.1 tools-3.6 : OTP-26.1.2 : erts-14.1.1 xmerl-1.3.33 # asn1-5.2 common_test-1.25.1 compiler-8.4.1 crypto-5.3 debugger-5.3.2 dialyzer-5.1.1 diameter-2.3 edoc-1.2 eldap-1.2.11 erl_docgen-1.5.1 erl_interface-5.4 et-1.7 eunit-2.8.2 ftp-1.2 inets-9.0.2 jinterface-1.14 kernel-9.1 megaco-4.5 mnesia-4.22.1 observer-2.15.1 odbc-2.14.1 os_mon-2.9 parsetools-2.5 public_key-1.14.1 reltool-1.0 runtime_tools-2.0 sasl-4.2.1 snmp-5.15 ssh-5.0.1 ssl-11.0.3 stdlib-5.1.1 syntax_tools-3.1 tftp-1.1 tools-3.6 wx-2.3.1 : OTP-26.1.1 : compiler-8.4.1 stdlib-5.1.1 wx-2.3.1 # asn1-5.2 common_test-1.25.1 crypto-5.3 debugger-5.3.2 dialyzer-5.1.1 diameter-2.3 edoc-1.2 eldap-1.2.11 erl_docgen-1.5.1 erl_interface-5.4 erts-14.1 et-1.7 eunit-2.8.2 ftp-1.2 inets-9.0.2 jinterface-1.14 kernel-9.1 megaco-4.5 mnesia-4.22.1 observer-2.15.1 odbc-2.14.1 os_mon-2.9 parsetools-2.5 public_key-1.14.1 reltool-1.0 runtime_tools-2.0 sasl-4.2.1 snmp-5.15 ssh-5.0.1 ssl-11.0.3 syntax_tools-3.1 tftp-1.1 tools-3.6 xmerl-1.3.32 : OTP-26.1 : asn1-5.2 common_test-1.25.1 compiler-8.4 crypto-5.3 debugger-5.3.2 dialyzer-5.1.1 erl_docgen-1.5.1 erts-14.1 inets-9.0.2 kernel-9.1 megaco-4.5 mnesia-4.22.1 observer-2.15.1 public_key-1.14.1 snmp-5.15 ssl-11.0.3 stdlib-5.1 # diameter-2.3 edoc-1.2 eldap-1.2.11 erl_interface-5.4 et-1.7 eunit-2.8.2 ftp-1.2 jinterface-1.14 odbc-2.14.1 os_mon-2.9 parsetools-2.5 reltool-1.0 runtime_tools-2.0 sasl-4.2.1 ssh-5.0.1 syntax_tools-3.1 tftp-1.1 tools-3.6 wx-2.3 xmerl-1.3.32 : OTP-26.0.2 : compiler-8.3.2 erts-14.0.2 kernel-9.0.2 ssh-5.0.1 ssl-11.0.2 stdlib-5.0.2 # asn1-5.1 common_test-1.25 crypto-5.2 debugger-5.3.1 dialyzer-5.1 diameter-2.3 edoc-1.2 eldap-1.2.11 erl_docgen-1.5 erl_interface-5.4 et-1.7 eunit-2.8.2 ftp-1.2 inets-9.0.1 jinterface-1.14 megaco-4.4.4 mnesia-4.22 observer-2.15 odbc-2.14.1 os_mon-2.9 parsetools-2.5 public_key-1.14 reltool-1.0 runtime_tools-2.0 sasl-4.2.1 snmp-5.14 syntax_tools-3.1 tftp-1.1 tools-3.6 wx-2.3 xmerl-1.3.32 : OTP-26.0.1 : compiler-8.3.1 erts-14.0.1 inets-9.0.1 kernel-9.0.1 ssl-11.0.1 stdlib-5.0.1 xmerl-1.3.32 # asn1-5.1 common_test-1.25 crypto-5.2 debugger-5.3.1 dialyzer-5.1 diameter-2.3 edoc-1.2 eldap-1.2.11 erl_docgen-1.5 erl_interface-5.4 et-1.7 eunit-2.8.2 ftp-1.2 jinterface-1.14 megaco-4.4.4 mnesia-4.22 observer-2.15 odbc-2.14.1 os_mon-2.9 parsetools-2.5 public_key-1.14 reltool-1.0 runtime_tools-2.0 sasl-4.2.1 snmp-5.14 ssh-5.0 syntax_tools-3.1 tftp-1.1 tools-3.6 wx-2.3 : OTP-26.0 : asn1-5.1 common_test-1.25 compiler-8.3 crypto-5.2 dialyzer-5.1 diameter-2.3 erl_docgen-1.5 erl_interface-5.4 erts-14.0 et-1.7 ftp-1.2 inets-9.0 jinterface-1.14 kernel-9.0 megaco-4.4.4 mnesia-4.22 observer-2.15 odbc-2.14.1 os_mon-2.9 parsetools-2.5 public_key-1.14 reltool-1.0 runtime_tools-2.0 sasl-4.2.1 snmp-5.14 ssh-5.0 ssl-11.0 stdlib-5.0 syntax_tools-3.1 tftp-1.1 tools-3.6 wx-2.3 # debugger-5.3.1 edoc-1.2 eldap-1.2.11 eunit-2.8.2 xmerl-1.3.31 : +OTP-25.3.2.8 : asn1-5.0.21.1 erl_interface-5.3.2.1 erts-13.2.2.5 mnesia-4.21.4.2 public_key-1.13.3.2 ssh-4.15.3.1 ssl-10.9.1.3 wx-2.2.2.1 # common_test-1.24.0.1 compiler-8.2.6.3 crypto-5.1.4.1 debugger-5.3.1.2 dialyzer-5.0.5 diameter-2.2.7 edoc-1.2 eldap-1.2.11 erl_docgen-1.4 et-1.6.5 eunit-2.8.2 ftp-1.1.4 inets-8.3.1.2 jinterface-1.13.2 kernel-8.5.4.2 megaco-4.4.3 observer-2.14 odbc-2.14 os_mon-2.8.2 parsetools-2.4.1 reltool-0.9.1 runtime_tools-1.19 sasl-4.2 snmp-5.13.5 stdlib-4.3.1.3 syntax_tools-3.0.1 tftp-1.0.4 tools-3.5.3 xmerl-1.3.31.1 : OTP-25.3.2.7 : erts-13.2.2.4 stdlib-4.3.1.3 # asn1-5.0.21 common_test-1.24.0.1 compiler-8.2.6.3 crypto-5.1.4.1 debugger-5.3.1.2 dialyzer-5.0.5 diameter-2.2.7 edoc-1.2 eldap-1.2.11 erl_docgen-1.4 erl_interface-5.3.2 et-1.6.5 eunit-2.8.2 ftp-1.1.4 inets-8.3.1.2 jinterface-1.13.2 kernel-8.5.4.2 megaco-4.4.3 mnesia-4.21.4.1 observer-2.14 odbc-2.14 os_mon-2.8.2 parsetools-2.4.1 public_key-1.13.3.1 reltool-0.9.1 runtime_tools-1.19 sasl-4.2 snmp-5.13.5 ssh-4.15.3 ssl-10.9.1.2 syntax_tools-3.0.1 tftp-1.0.4 tools-3.5.3 wx-2.2.2 xmerl-1.3.31.1 : OTP-25.3.2.6 : crypto-5.1.4.1 debugger-5.3.1.2 erts-13.2.2.3 inets-8.3.1.2 kernel-8.5.4.2 mnesia-4.21.4.1 public_key-1.13.3.1 ssl-10.9.1.2 # asn1-5.0.21 common_test-1.24.0.1 compiler-8.2.6.3 dialyzer-5.0.5 diameter-2.2.7 edoc-1.2 eldap-1.2.11 erl_docgen-1.4 erl_interface-5.3.2 et-1.6.5 eunit-2.8.2 ftp-1.1.4 jinterface-1.13.2 megaco-4.4.3 observer-2.14 odbc-2.14 os_mon-2.8.2 parsetools-2.4.1 reltool-0.9.1 runtime_tools-1.19 sasl-4.2 snmp-5.13.5 ssh-4.15.3 stdlib-4.3.1.2 syntax_tools-3.0.1 tftp-1.0.4 tools-3.5.3 wx-2.2.2 xmerl-1.3.31.1 : OTP-25.3.2.5 : inets-8.3.1.1 # asn1-5.0.21 common_test-1.24.0.1 compiler-8.2.6.3 crypto-5.1.4 debugger-5.3.1.1 dialyzer-5.0.5 diameter-2.2.7 edoc-1.2 eldap-1.2.11 erl_docgen-1.4 erl_interface-5.3.2 erts-13.2.2.2 et-1.6.5 eunit-2.8.2 ftp-1.1.4 jinterface-1.13.2 kernel-8.5.4.1 megaco-4.4.3 mnesia-4.21.4 observer-2.14 odbc-2.14 os_mon-2.8.2 parsetools-2.4.1 public_key-1.13.3 reltool-0.9.1 runtime_tools-1.19 sasl-4.2 snmp-5.13.5 ssh-4.15.3 ssl-10.9.1.1 stdlib-4.3.1.2 syntax_tools-3.0.1 tftp-1.0.4 tools-3.5.3 wx-2.2.2 xmerl-1.3.31.1 : @@ -27,6 +30,7 @@ OTP-25.0.3 : erts-13.0.3 ssl-10.8.3 # asn1-5.0.19 common_test-1.23 compiler-8.2 OTP-25.0.2 : erts-13.0.2 ssl-10.8.2 # asn1-5.0.19 common_test-1.23 compiler-8.2 crypto-5.1.1 debugger-5.3 dialyzer-5.0.1 diameter-2.2.6 edoc-1.2 eldap-1.2.10 erl_docgen-1.3 erl_interface-5.3 et-1.6.5 eunit-2.7.1 ftp-1.1.1 inets-8.0 jinterface-1.13 kernel-8.4.1 megaco-4.4 mnesia-4.21.1 observer-2.12 odbc-2.14 os_mon-2.7.1 parsetools-2.4 public_key-1.13 reltool-0.9 runtime_tools-1.19 sasl-4.2 snmp-5.13 ssh-4.14.1 stdlib-4.0.1 syntax_tools-3.0 tftp-1.0.3 tools-3.5.3 wx-2.2 xmerl-1.3.29 : OTP-25.0.1 : crypto-5.1.1 dialyzer-5.0.1 erts-13.0.1 kernel-8.4.1 mnesia-4.21.1 ssh-4.14.1 ssl-10.8.1 stdlib-4.0.1 # asn1-5.0.19 common_test-1.23 compiler-8.2 debugger-5.3 diameter-2.2.6 edoc-1.2 eldap-1.2.10 erl_docgen-1.3 erl_interface-5.3 et-1.6.5 eunit-2.7.1 ftp-1.1.1 inets-8.0 jinterface-1.13 megaco-4.4 observer-2.12 odbc-2.14 os_mon-2.7.1 parsetools-2.4 public_key-1.13 reltool-0.9 runtime_tools-1.19 sasl-4.2 snmp-5.13 syntax_tools-3.0 tftp-1.0.3 tools-3.5.3 wx-2.2 xmerl-1.3.29 : OTP-25.0 : asn1-5.0.19 common_test-1.23 compiler-8.2 crypto-5.1 debugger-5.3 dialyzer-5.0 diameter-2.2.6 edoc-1.2 erl_docgen-1.3 erl_interface-5.3 erts-13.0 eunit-2.7.1 inets-8.0 jinterface-1.13 kernel-8.4 megaco-4.4 mnesia-4.21 observer-2.12 odbc-2.14 parsetools-2.4 public_key-1.13 runtime_tools-1.19 sasl-4.2 snmp-5.13 ssh-4.14 ssl-10.8 stdlib-4.0 syntax_tools-3.0 tools-3.5.3 wx-2.2 xmerl-1.3.29 # eldap-1.2.10 et-1.6.5 ftp-1.1.1 os_mon-2.7.1 reltool-0.9 tftp-1.0.3 : +OTP-24.3.4.15 : asn1-5.0.18.2 erl_interface-5.2.2.1 erts-12.3.2.15 mnesia-4.20.4.4 ssh-4.13.2.4 # common_test-1.22.1.1 compiler-8.1.1.5 crypto-5.0.6.4 debugger-5.2.1.1 dialyzer-4.4.4.1 diameter-2.2.5 edoc-1.1 eldap-1.2.10 erl_docgen-1.2.1 et-1.6.5 eunit-2.7 ftp-1.1.1 inets-7.5.3.4 jinterface-1.12.2 kernel-8.3.2.4 megaco-4.3 observer-2.11.1 odbc-2.13.5 os_mon-2.7.1 parsetools-2.3.2 public_key-1.12.0.2 reltool-0.9 runtime_tools-1.18 sasl-4.1.2 snmp-5.12.0.3 ssl-10.7.3.9 stdlib-3.17.2.4 syntax_tools-2.6 tftp-1.0.3 tools-3.5.2 wx-2.1.4 xmerl-1.3.28.1 : OTP-24.3.4.14 : crypto-5.0.6.4 erts-12.3.2.14 kernel-8.3.2.4 mnesia-4.20.4.3 public_key-1.12.0.2 ssl-10.7.3.9 # asn1-5.0.18.1 common_test-1.22.1.1 compiler-8.1.1.5 debugger-5.2.1.1 dialyzer-4.4.4.1 diameter-2.2.5 edoc-1.1 eldap-1.2.10 erl_docgen-1.2.1 erl_interface-5.2.2 et-1.6.5 eunit-2.7 ftp-1.1.1 inets-7.5.3.4 jinterface-1.12.2 megaco-4.3 observer-2.11.1 odbc-2.13.5 os_mon-2.7.1 parsetools-2.3.2 reltool-0.9 runtime_tools-1.18 sasl-4.1.2 snmp-5.12.0.3 ssh-4.13.2.3 stdlib-3.17.2.4 syntax_tools-2.6 tftp-1.0.3 tools-3.5.2 wx-2.1.4 xmerl-1.3.28.1 : OTP-24.3.4.13 : compiler-8.1.1.5 debugger-5.2.1.1 erts-12.3.2.13 ssh-4.13.2.3 ssl-10.7.3.8 stdlib-3.17.2.4 # asn1-5.0.18.1 common_test-1.22.1.1 crypto-5.0.6.3 dialyzer-4.4.4.1 diameter-2.2.5 edoc-1.1 eldap-1.2.10 erl_docgen-1.2.1 erl_interface-5.2.2 et-1.6.5 eunit-2.7 ftp-1.1.1 inets-7.5.3.4 jinterface-1.12.2 kernel-8.3.2.3 megaco-4.3 mnesia-4.20.4.2 observer-2.11.1 odbc-2.13.5 os_mon-2.7.1 parsetools-2.3.2 public_key-1.12.0.1 reltool-0.9 runtime_tools-1.18 sasl-4.1.2 snmp-5.12.0.3 syntax_tools-2.6 tftp-1.0.3 tools-3.5.2 wx-2.1.4 xmerl-1.3.28.1 : OTP-24.3.4.12 : compiler-8.1.1.4 erts-12.3.2.12 stdlib-3.17.2.3 xmerl-1.3.28.1 # asn1-5.0.18.1 common_test-1.22.1.1 crypto-5.0.6.3 debugger-5.2.1 dialyzer-4.4.4.1 diameter-2.2.5 edoc-1.1 eldap-1.2.10 erl_docgen-1.2.1 erl_interface-5.2.2 et-1.6.5 eunit-2.7 ftp-1.1.1 inets-7.5.3.4 jinterface-1.12.2 kernel-8.3.2.3 megaco-4.3 mnesia-4.20.4.2 observer-2.11.1 odbc-2.13.5 os_mon-2.7.1 parsetools-2.3.2 public_key-1.12.0.1 reltool-0.9 runtime_tools-1.18 sasl-4.1.2 snmp-5.12.0.3 ssh-4.13.2.2 ssl-10.7.3.7 syntax_tools-2.6 tftp-1.0.3 tools-3.5.2 wx-2.1.4 : diff --git a/system/COPYRIGHT b/system/COPYRIGHT index 6a23a136a928..1dcbfd226017 100644 --- a/system/COPYRIGHT +++ b/system/COPYRIGHT @@ -3,9 +3,12 @@ This software is subject to the following Copyrights and Licenses: --------------------------------------------------------------------------- [Erlang/OTP except parts stated below] +* Info: + * SPDX-License-Identifier: Apache-2.0 + %CopyrightBegin% -Copyright Ericsson AB 1997-2021. All Rights Reserved. +Copyright Ericsson AB 1997-2023. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -22,19 +25,207 @@ limitations under the License. %CopyrightEnd% --------------------------------------------------------------------------- -[stdlib, compiler] +[configure scripts] + +* Info: + * SPDX-License-Identifier: FSFUL + * Tool: Autoconf + * Version: 2.71 + * Website: https://www.gnu.org/software/autoconf/ + * OTP Location: ./make/configure and + ./erts/configure and + ./lib/erl_interface/configure and + ./lib/crypto/configure and + ./lib/wx/configure and + ./lib/megaco/configure and + ./lib/snmp/configure and + ./lib/odbc/configure and + ./lib/common_test/test_server/configure and + ./lib/common_test/configure + +# Copyright (C) 1992-1996, 1998-2012, 2020-2021 Free Software Foundation, Inc. +# +# +# This configure script is free software; the Free Software Foundation +# gives unlimited permission to copy, distribute and modify it. + +--------------------------------------------------------------------------- +[config.guess and config.sub scripts] + +* Info: + * SPDX-License-Identifier: GPL-3.0-or-later WITH Autoconf-exception-generic-3.0 + * Tool: Autoconf + * Version: 2.71 + * Website: https://www.gnu.org/software/autoconf/ + * OTP Location: ./make/autoconf/config.{guess,sub} and + ./erts/autoconf/config.{guess,sub} and + ./lib/common_test/test_server/config.{guess,sub} + +# Copyright 1992-2021 Free Software Foundation, Inc. + +# This file is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . +# +# As a special exception to the GNU General Public License, if you +# distribute this file as part of a program that contains a +# configuration script generated by Autoconf, you may include it under +# the same distribution terms that you use for the rest of that +# program. This Exception is an additional permission under section 7 +# of the GNU General Public License, version 3 ("GPLv3"). +# +# Originally written by Per Bothner; maintained since 2000 by Ben Elliston. +# +# You can get the latest version of this script from: +# https://git.savannah.gnu.org/cgit/config.git/plain/config.guess +# +# Please send patches to . + +--------------------------------------------------------------------------- +[install-sh scripts] + +* Info: + * SPDX-License-Identifier: MIT + * Tool: Autoconf + * Version: 2.71 + * Website: https://www.gnu.org/software/autoconf/ + * OTP Location: ./make/autoconf/install-sh and + ./erts/autoconf/install-sh and + ./lib/common_test/test_server/install-sh + +# Copyright (C) 1994 X Consortium +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to +# deal in the Software without restriction, including without limitation the +# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +# sell copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN +# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC- +# TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +# +# Except as contained in this notice, the name of the X Consortium shall not +# be used in advertising or otherwise to promote the sale, use or other deal- +# ings in this Software without prior written authorization from the X Consor- +# tium. +# +# +# FSF changes to this file are in the public domain. + +--------------------------------------------------------------------------- +[erl_posix_str.c] + +* Info: + * SPDX-License-Identifier: TCL + * Modification of tclPosixStr.c from TCL + * Library: TCL + * Version 7.6 + * Website: https://www.tcl.tk/ + * OTP Location: ./erts/emulator/beam/erl_posix_str.c + +Copyright (c) 1991-1994 The Regents of the University of California. +Copyright (c) 1994-1996 Sun Microsystems, Inc. + +This software is copyrighted by the Regents of the University of +California, Sun Microsystems, Inc., and other parties. The following +terms apply to all files associated with the software unless explicitly +disclaimed in individual files. + +The authors hereby grant permission to use, copy, modify, distribute, +and license this software and its documentation for any purpose, provided +that existing copyright notices are retained in all copies and that this +notice is included verbatim in any distributions. No written agreement, +license, or royalty fee is required for any of the authorized uses. +Modifications to this software may be copyrighted by their authors +and need not follow the licensing terms described here, provided that +the new terms are clearly indicated on the first page of each file where +they apply. + +IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY +FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR ANY +DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. -* assert.hrl is Copyright (C) 2004-1016 Richard Carlsson, Mickaël Rémond -* array.erl is Copyright (C) 2006-2016 Richard Carlsson and Ericsson AB -* gb_trees.erl is Copyright (C) 1999-2001 Sven-Olof Nyström, Richard Carlsson -* gb_sets.erl is Copyright (C) 1999-2001 Richard Carlsson, Sven-Olof Nyström -* proplists.erl is Copyright (C) 2000-2003 Richard Carlsson -* cerl{_trees,_clauses}.erl are Copyright (C) 1999-2002 Richard Carlsson -* cerl_inline.erl is Copyright (C) 1999-2002 Richard Carlsson +THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES, +INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. THIS SOFTWARE +IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE +NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR +MODIFICATIONS. + +GOVERNMENT USE: If you are acquiring this software on behalf of the +U.S. government, the Government shall have only "Restricted Rights" +in the software and related documentation as defined in the Federal +Acquisition Regulations (FARs) in Clause 52.227.19 (c) (2). If you +are acquiring the software on behalf of the Department of Defense, the +software shall be classified as "Commercial Computer Software" and the +Government shall have only "Restricted Rights" as defined in Clause +252.227-7013 (c) (1) of DFARs. Notwithstanding the foregoing, the +authors grant the U.S. Government and others acting in its behalf +permission to use and distribute the software in accordance with the +terms specified in this license. + +--------------------------------------------------------------------------- +[various contributions] + +* Info: + * SPDX-License-Identifier: Apache-2.0 + * Various contributions made to Erlang/OTP under Apache 2.0 License + +* ./lib/stdlib/include/assert.hrl + and ./bootstrap/lib/stdlib/include/assert.hrl + * is Copyright (C) 2004-1016 Richard Carlsson, Mickaël Rémond +* ./lib/stdlib/src/array.erl + * Copyright (C) 2006-2016 Richard Carlsson and Ericsson AB +* ./lib/stdlib/src/gb_trees.erl + * Copyright (C) 1999-2001 Sven-Olof Nyström, Richard Carlsson +* ./lib/stdlib/src/gb_sets.erl + * Copyright (C) 1999-2001 Richard Carlsson, Sven-Olof Nyström +* ./lib/stdlib/src/proplists.erl + * Copyright (C) 2000-2003 Richard Carlsson +* ./lib/compiler/src/cerl_trees.erl and + ./lib/dialyzer/test/options1_SUITE_data/src/compiler/cerl_trees.erl + * Copyright (C) 1999-2002 Richard Carlsson +* ./lib/compiler/src/cerl_clauses.erl and + ./lib/dialyzer/test/options1_SUITE_data/src/compiler/cerl_clauses.erl + * Copyright (C) 1999-2002 Richard Carlsson +* ./lib/compiler/src/cerl_inline.erl and + ./lib/dialyzer/test/options1_SUITE_data/src/compiler/cerl_inline.erl + * Copyright (C) 1999-2002 Richard Carlsson +* ./lib/kernel/src/pg.erl and lib/kernel/doc/src/pg.xml and + ./lib/kernel/test/pg_SUITE.erl and ./lib/stdlib/src/peer.erl + * Copyright WhatsApp Inc. and its affiliates. All rights reserved. +* ./lib/kernel/doc/src/peer.xml + * Maxim Fedorov, WhatsApp Inc. --------------------------------------------------------------------------- [AsmJit] +* Info + * SPDX-License-Identifier: Zlib + * Library: AsmJit + * Git Repository: https://github.com/asmjit/asmjit + * Commit: 915186f6c5c2f5a4638e5cb97ccc23d741521a64 + * OTP Location: erts/emulator/asmjit + Copyright (c) 2008-2020 The AsmJit Authors This software is provided 'as-is', without any express or implied @@ -56,6 +247,13 @@ freely, subject to the following restrictions: --------------------------------------------------------------------------- [PCRE] +* Info: + * SPDX-License-Identifier: BSD-3-Clause + * Library: PCRE + * Version: 8.44 + * Website: https://pcre.org + * OTP Location: ./erts/emulator/pcre + PCRE LICENCE ------------ @@ -150,33 +348,20 @@ POSSIBILITY OF SUCH DAMAGE. End ---------------------------------------------------------------------------- -[Misc C library code] - - /* - * Copyright (c) 1985, 1988 Regents of the University of California. - * All rights reserved. - * - * Redistribution and use in source and binary forms are permitted - * provided that the above copyright notice and this paragraph are - * duplicated in all such forms and that any documentation, - * advertising materials, and other materials related to such - * distribution and use acknowledge that the software was developed - * by the University of California, Berkeley. The name of the - * University may not be used to endorse or promote products derived - * from this software without specific prior written permission. - * THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR - * IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED - * WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE. - */ - --------------------------------------------------------------------------- [zlib] +* Info: + * SPDX-License-Identifier: Zlib + * Library: Zlib + * Version: 1.2.13 + * Website: https://zlib.net/ + * OTP Location: ./erts/emulator/zlib + /* zlib.h -- interface of the 'zlib' general purpose compression library - version 1.2.3, July 18th, 2005 + version 1.2.13, October 13th, 2022 - Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler + Copyright (C) 1995-2022 Jean-loup Gailly and Mark Adler This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages @@ -199,13 +384,97 @@ End The data format used by the zlib library is described by RFCs (Request for - Comments) 1950 to 1952 in the files http://www.ietf.org/rfc/rfc1950.txt - (zlib format), rfc1951.txt (deflate format) and rfc1952.txt (gzip format). + Comments) 1950 to 1952 in the files http://tools.ietf.org/html/rfc1950 + (zlib format), rfc1951 (deflate format) and rfc1952 (gzip format). */ +--------------------------------------------------------------------------- +[ryu] + +* Info: + * SPDX-License-Identifier: Apache-2.0 OR BSL-1.0 + * Library: RYU + * Git repository: https://github.com/ulfjack/ryu + * Commit: 844864ac213bdbf1fb57e6f51c653b3d90af0937 + * OTP Location: ./erts/emulator/ryu + +// Copyright 2018 Ulf Adams +// +// The contents of this file may be used under the terms of the Apache License, +// Version 2.0. +// +// (See accompanying file LICENSE-Apache or copy at +// http://www.apache.org/licenses/LICENSE-2.0) +// +// Alternatively, the contents of this file may be used under the terms of +// the Boost Software License, Version 1.0. +// (See accompanying file LICENSE-Boost or copy at +// https://www.boost.org/LICENSE_1_0.txt) +// +// Unless required by applicable law or agreed to in writing, this software +// is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. + +--------------------------------------------------------------------------- +[to_chars() function in ryu] + +* Info: + * SPDX-License-Identifier: (Apache-2.0 WITH LLVM-exception) AND BSL-1.0 + * License Comment: The license information in the original file is not + clear on whether it should be AND or OR between + "Apache 2.0 with LLVM-exception" and "Boost Software + License 1.0". Therefore, just to be safe, an AND was + chosen in the SPDX license identifier expression + above. + * Library: STL + * Git repository: https://github.com/microsoft/STL + * Commit: e745bad3b1d05b5b19ec652d68abb37865ffa454 + * Original function: https://github.com/microsoft/STL/blob/e745bad3b1d05b5b19ec652d68abb37865ffa454/stl/inc/xcharconv_ryu.h#L1926 + * OTP Location: ./erts/emulator/ryu/d2s.c:to_chars() + +// xcharconv_ryu.h internal header + +// Copyright (c) Microsoft Corporation. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + + +// Copyright 2018 Ulf Adams +// Copyright (c) Microsoft Corporation. All rights reserved. + +// Boost Software License - Version 1.0 - August 17th, 2003 + +// Permission is hereby granted, free of charge, to any person or organization +// obtaining a copy of the software and accompanying documentation covered by +// this license (the "Software") to use, reproduce, display, distribute, +// execute, and transmit the Software, and to prepare derivative works of the +// Software, and to permit third-parties to whom the Software is furnished to +// do so, all subject to the following: + +// The copyright notices in the Software and this entire statement, including +// the above license grant, this restriction and the following disclaimer, +// must be included in all copies of the Software, in whole or in part, and +// all derivative works of the Software, unless such copies or derivative +// works are solely in the form of machine-executable object code generated by +// a source language processor. + +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT +// SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE +// FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, +// ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +// DEALINGS IN THE SOFTWARE. + --------------------------------------------------------------------------- [fp16] +* Info: + * SPDX-License-Identifier: MIT + * Library: FP16 + * Git Repository: https://github.com/Maratyszcza/FP16 + * Commit: 0a92994d729ff76a58f692d3028ca1b64b145d91 + * OTP Location: ./erts/emulator/beam/erl_bits_f16.h + The MIT License (MIT) Copyright (c) 2017 Facebook Inc. @@ -229,9 +498,35 @@ FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TOR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +--------------------------------------------------------------------------- +[OpenSSL MD5 implementation] + +* Info: + * SPDX-License-Identifier: Apache-2.0 + * Library: OpenSSL + * Version: 3.1.4 + * Git Repository: https://github.com/openssl/openssl + * Commit: 01d5e2318405362b4de5e670c90d9b40a351d053 + * OTP Location: ./erts/emulator/openssl and + ./lib/erl_interface/src/openssl + +/* + * Copyright 1995-2022 The OpenSSL Project Authors. All Rights Reserved. + * + * Licensed under the Apache License 2.0 (the "License"). You may not use + * this file except in compliance with the License. You can obtain a copy + * in the file LICENSE in the source distribution or at + * https://www.openssl.org/source/license.html + */ + --------------------------------------------------------------------------- [dialyzer] +* Info: + * SPDX-License-Identifier: Apache-2.0 + * Contribution made to Erlang/OTP + * OTP Location: ./lib/dialyzer + %% Copyright 1997-2016 Tobias Lindahl, Stavros Aronis, Kostis Sagonas, %% Richard Carlsson, et al. %% @@ -250,6 +545,11 @@ DEALINGS IN THE SOFTWARE. --------------------------------------------------------------------------- [edoc, syntax_tools] +* Info: + * SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later + * Contribution made to Erlang/OTP + * OTP Location: ./lib/edoc and ./lib/syntax_tools + %% Copyright 1997-2016 Richard Carlsson %% %% Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -275,6 +575,11 @@ DEALINGS IN THE SOFTWARE. --------------------------------------------------------------------------- [eunit] +* Info: + * SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later + * Contribution made to Erlang/OTP + * OTP Location: ./lib/eunit + %% Copyright 2004-2016 Richard Carlsson , %% Mickaël Rémond %% @@ -301,7 +606,13 @@ DEALINGS IN THE SOFTWARE. --------------------------------------------------------------------------- [leex] -%% Copyright (c) 2008 Robert Virding. All rights reserved. +* Info: + * SPDX-License-Identifier: BSD-2-Clause + * Contribution made to Erlang/OTP + * OTP Location: ./lib/parsetools/src/leex.erl and + ./lib/parsetools/doc/src/leex.xml + +%% Copyright (c) 2008,2009 Robert Virding. All rights reserved. %% %% Redistribution and use in source and binary forms, with or without %% modification, are permitted provided that the following conditions @@ -329,6 +640,13 @@ DEALINGS IN THE SOFTWARE. --------------------------------------------------------------------------- [eldap] +* Info: + * SPDX-License-Identifier: MIT + * Contribution made to Erlang/OTP + * Git Repository: https://github.com/etnt/eldap.git + * Commit: 36c595f56c12a44adbc942eef218df85137cbbee + * OTP Location: ./lib/eldap/src/eldap.erl + Copyright (c) 2010, Torbjorn Tornkvist Permission is hereby granted, free of charge, to any person obtaining a copy @@ -352,6 +670,13 @@ THE SOFTWARE. --------------------------------------------------------------------------- [wx documentation] +* Info: + * No SPDX License Identifier exist for this license. + * License: wxWindows Free Documentation Licence, Version 3 + * Git Repository: https://github.com/wxWidgets/wxWidgets + * Commit: dc585039bbd426829e3433002023a93f9bedd0c2 + * OTP Location: ./lib/wx/doc/src + wxWindows Free Documentation Licence, Version 3 =============================================== @@ -415,6 +740,9 @@ BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. --------------------------------------------------------------------------- [wx webview2 loader on windows] +* Info: + * SPDX-License-Identifier: BSD-3-Clause + Copyright (C) Microsoft Corporation. All rights reserved. Redistribution and use in source and binary forms, with or without diff --git a/system/doc/reference_manual/errors.xml b/system/doc/reference_manual/errors.xml index 0ce36c3f319c..6b47d5a94201 100644 --- a/system/doc/reference_manual/errors.xml +++ b/system/doc/reference_manual/errors.xml @@ -4,7 +4,7 @@
- 20032021 + 20032023 Ericsson AB. All Rights Reserved.