Merge branch 'main' of github.com:github/codeql into SharedDataflow_N…

…estedComprehensions

.devcontainer/devcontainer.json

@@ -4,6 +4,6 @@
     "slevesque.vscode-zipexplorer"
   ],
   "settings": {
-    "codeQL.experimentalBqrsParsing": true
+    "codeQL.runningQueries.memory": 2048
   }
 }

.github/workflows/labeler.yml

@@ -0,0 +1,11 @@
+name: "Pull Request Labeler"
+on:
+- pull_request_target
+
+jobs:
+  triage:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/labeler@v2
+      with:
+        repo-token: "${{ secrets.GITHUB_TOKEN }}"

README.md

@@ -9,7 +9,7 @@ You can use the [interactive query console](https://lgtm.com/help/lgtm/using-que
 
 ## Contributing
 
-We welcome contributions to our standard library and standard checks. Do you have an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request! Before you do, though, please take the time to read our [contributing guidelines](CONTRIBUTING.md). You can also consult our [style guides](https://github.com/github/codeql/tree/master/docs) to learn how to format your code for consistency and clarity, how to write query metadata, and how to write query help documentation for your query.
+We welcome contributions to our standard library and standard checks. Do you have an idea for a new check, or how to improve an existing query? Then please go ahead and open a pull request! Before you do, though, please take the time to read our [contributing guidelines](CONTRIBUTING.md). You can also consult our [style guides](https://github.com/github/codeql/tree/main/docs) to learn how to format your code for consistency and clarity, how to write query metadata, and how to write query help documentation for your query.
 
 ## License
 

change-notes/1.25/analysis-javascript.md

@@ -30,7 +30,7 @@
   - [yargs](https://www.npmjs.com/package/yargs)
   - [webpack-dev-server](https://www.npmjs.com/package/webpack-dev-server)
 
-* TypeScript 3.9 is now supported.
+* TypeScript 4.0 is now supported.
 
 * TypeScript code embedded in HTML and Vue files is now extracted and analyzed.
 

change-notes/1.26/analysis-cpp.md

@@ -13,13 +13,16 @@ The following changes in version 1.26 affect C/C++ analysis in all applications.
 
 | **Query**                  | **Expected impact**    | **Change**                                                       |
 |----------------------------|------------------------|------------------------------------------------------------------|
+| Declaration hides parameter (`cpp/declaration-hides-parameter`) | Fewer false positive results | False positives involving template functions have been fixed. |
 | Inconsistent direction of for loop (`cpp/inconsistent-loop-direction`) | Fewer false positive results | The query now accounts for intentional wrapping of an unsigned loop counter. |
 | Overflow in uncontrolled allocation size (`cpp/uncontrolled-allocation-size`) | | The precision of this query has been decreased from "high" to "medium". As a result, the query is still run but results are no longer displayed on LGTM by default. |
 | Comparison result is always the same (`cpp/constant-comparison`) | More correct results | Bounds on expressions involving multiplication can now be determined in more cases. |
 
 ## Changes to libraries
 
-* The models library now models some taint flows through `std::array`, `std::vector`, `std::deque`, `std::list` and `std::forward_list`.
+* The QL class `Block`, denoting the `{ ... }` statement, is renamed to `BlockStmt`.
+* The models library now models many taint flows through `std::array`, `std::vector`, `std::deque`, `std::list` and `std::forward_list`.
 * The models library now models many more taint flows through `std::string`.
+* The models library now models some taint flows through `std::ostream`.
 * The `SimpleRangeAnalysis` library now supports multiplications of the form
   `e1 * e2` and `x *= e2` when `e1` and `e2` are unsigned or constant.

change-notes/1.26/analysis-csharp.md

@@ -19,6 +19,12 @@ The following changes in version 1.26 affect C# analysis in all applications.
 ## Changes to code extraction
 
 * Partial method bodies are extracted. Previously, partial method bodies were skipped completely.
+* Inferring the lengths of implicitely sized arrays is fixed. Previously, multidimensional arrays were always extracted with the same length for
+each dimension. With the fix, the array sizes `2` and `1` are extracted for `new int[,]{{1},{2}}`. Previously `2` and `2` were extracted.
+* The extractor is now assembly-insensitive by default. This means that two entities with the same
+  fully-qualified name are now mapped to the same entity in the resulting database, regardless of
+  whether they belong to different assemblies. Assembly sensitivity can be reenabled by passing
+  `--assemblysensitivetrap` to the extractor.
 
 ## Changes to libraries
 

change-notes/1.26/analysis-java.md

@@ -0,0 +1,21 @@
+# Improvements to Java analysis
+
+The following changes in version 1.26 affect Java analysis in all applications.
+
+## General improvements
+
+## New queries
+
+| **Query**                   | **Tags**  | **Purpose**                                                        |
+|-----------------------------|-----------|--------------------------------------------------------------------|
+
+
+## Changes to existing queries
+
+| **Query**                    | **Expected impact**    | **Change**                        |
+|------------------------------|------------------------|-----------------------------------|
+
+
+## Changes to libraries
+
+* The QL class `Block`, denoting the `{ ... }` statement, is renamed to `BlockStmt`.

change-notes/1.26/analysis-javascript.md

@@ -26,7 +26,13 @@
 
 | **Query**                      | **Expected impact**          | **Change**                                                                |
 |--------------------------------|------------------------------|---------------------------------------------------------------------------|
+| Potentially unsafe external link (`js/unsafe-external-link`) | Fewer results | This query no longer flags URLs constructed using a template system where only the hash or query part of the URL is dynamic. |
 | Incomplete URL substring sanitization (`js/incomplete-url-substring-sanitization`) | More results | This query now recognizes additional URLs when the substring check is an inclusion check. |
+| Ambiguous HTML id attribute (`js/duplicate-html-id`) | Results no longer shown | Precision tag reduced to "low". The query is no longer run by default. |
+| Unused loop iteration variable (`js/unused-loop-variable`) | Fewer results | This query no longer flags variables in a destructuring array assignment that are not the last variable in the destructed array. |
+| Unsafe shell command constructed from library input (`js/shell-command-constructed-from-input`) | More results | This query now recognizes more commands where colon, dash, and underscore are used. |
+| Unsafe jQuery plugin (`js/unsafe-jquery-plugin`) | More results | This query now detects more unsafe uses of nested option properties. |
 
 
 ## Changes to libraries
+* The predicate `TypeAnnotation.hasQualifiedName` now works in more cases when the imported library was not present during extraction.

change-notes/1.26/analysis-python.md

@@ -0,0 +1,22 @@
+# Improvements to Python analysis
+
+The following changes in version 1.26 affect Python analysis in all applications.
+
+## General improvements
+
+
+## New queries
+
+| **Query**                   | **Tags**  | **Purpose**                                                        |
+|-----------------------------|-----------|--------------------------------------------------------------------|
+
+
+## Changes to existing queries
+
+| **Query**                  | **Expected impact**    | **Change**                                                       |
+|----------------------------|------------------------|------------------------------------------------------------------|
+
+
+## Changes to libraries
+
+* Added taint tracking support for string formatting through f-strings.

config/identical-files.json

@@ -325,11 +325,60 @@
     "csharp/ql/src/experimental/ir/implementation/raw/gvn/internal/ValueNumberingImports.qll",
     "csharp/ql/src/experimental/ir/implementation/unaliased_ssa/gvn/internal/ValueNumberingImports.qll"
   ],
+  "Inline Test Expectations": [
+    "cpp/ql/test/TestUtilities/InlineExpectationsTest.qll",
+    "python/ql/test/TestUtilities/InlineExpectationsTest.qll"
+  ],
   "XML": [
     "cpp/ql/src/semmle/code/cpp/XML.qll",
     "csharp/ql/src/semmle/code/csharp/XML.qll",
     "java/ql/src/semmle/code/xml/XML.qll",
     "javascript/ql/src/semmle/javascript/XML.qll",
     "python/ql/src/semmle/python/xml/XML.qll"
+  ],
+  "DuplicationProblems.qhelp": [
+    "cpp/ql/src/Metrics/Files/DuplicationProblems.qhelp",
+    "csharp/ql/src/Metrics/Files/DuplicationProblems.qhelp",
+    "javascript/ql/src/Metrics/DuplicationProblems.qhelp",
+    "python/ql/src/Metrics/DuplicationProblems.qhelp"
+  ],
+  "CommentedOutCodeQuery.qhelp": [
+    "cpp/ql/src/Documentation/CommentedOutCodeQuery.qhelp",
+    "python/ql/src/Lexical/CommentedOutCodeQuery.qhelp",
+    "csharp/ql/src/Bad Practices/Comments/CommentedOutCodeQuery.qhelp",
+    "java/ql/src/Violations of Best Practice/Comments/CommentedOutCodeQuery.qhelp",
+    "javascript/ql/src/Comments/CommentedOutCodeQuery.qhelp"
+  ],
+  "FLinesOfCodeReferences.qhelp": [
+    "java/ql/src/Metrics/Files/FLinesOfCodeReferences.qhelp",
+    "javascript/ql/src/Metrics/FLinesOfCodeReferences.qhelp"
+  ],
+  "FCommentRatioCommon.qhelp": [
+    "java/ql/src/Metrics/Files/FCommentRatioCommon.qhelp",
+    "javascript/ql/src/Metrics/FCommentRatioCommon.qhelp"
+  ],
+  "FLinesOfCodeOverview.qhelp": [
+    "java/ql/src/Metrics/Files/FLinesOfCodeOverview.qhelp",
+    "javascript/ql/src/Metrics/FLinesOfCodeOverview.qhelp"
+  ],
+  "CommentedOutCodeMetricOverview.qhelp": [
+    "cpp/ql/src/Metrics/Files/CommentedOutCodeMetricOverview.qhelp",
+    "csharp/ql/src/Metrics/Files/CommentedOutCodeMetricOverview.qhelp",
+    "java/ql/src/Metrics/Files/CommentedOutCodeMetricOverview.qhelp",
+    "javascript/ql/src/Comments/CommentedOutCodeMetricOverview.qhelp",
+    "python/ql/src/Lexical/CommentedOutCodeMetricOverview.qhelp"
+  ],
+  "FLinesOfDuplicatedCodeCommon.qhelp": [
+    "cpp/ql/src/Metrics/Files/FLinesOfDuplicatedCodeCommon.qhelp",
+    "java/ql/src/Metrics/Files/FLinesOfDuplicatedCodeCommon.qhelp",
+    "javascript/ql/src/Metrics/FLinesOfDuplicatedCodeCommon.qhelp",
+    "python/ql/src/Metrics/FLinesOfDuplicatedCodeCommon.qhelp"
+  ],
+  "CommentedOutCodeReferences.qhelp": [
+    "cpp/ql/src/Metrics/Files/CommentedOutCodeReferences.qhelp",
+    "csharp/ql/src/Metrics/Files/CommentedOutCodeReferences.qhelp",
+    "java/ql/src/Metrics/Files/CommentedOutCodeReferences.qhelp",
+    "javascript/ql/src/Comments/CommentedOutCodeReferences.qhelp",
+    "python/ql/src/Lexical/CommentedOutCodeReferences.qhelp"
   ]
 }

cpp/ql/examples/snippets/emptyblock.ql

@@ -9,6 +9,6 @@
 
 import cpp
 
-from Block blk
+from BlockStmt blk
 where blk.getNumStmt() = 0
 select blk

cpp/ql/examples/snippets/emptythen.ql

@@ -13,5 +13,5 @@
 import cpp
 
 from IfStmt i
-where i.getThen().(Block).getNumStmt() = 0
+where i.getThen().(BlockStmt).getNumStmt() = 0
 select i

cpp/ql/examples/snippets/singletonblock.ql

@@ -8,6 +8,6 @@
 
 import cpp
 
-from Block b
+from BlockStmt b
 where b.getNumStmt() = 1
 select b

cpp/ql/src/Best Practices/BlockWithTooManyStatements.ql

@@ -14,7 +14,7 @@ import cpp
 
 class ComplexStmt extends Stmt {
   ComplexStmt() {
-    exists(Block body |
+    exists(BlockStmt body |
       body = this.(Loop).getStmt() or
       body = this.(SwitchStmt).getStmt()
     |
@@ -24,7 +24,7 @@ class ComplexStmt extends Stmt {
   }
 }
 
-from Block b, int n, ComplexStmt complexStmt
+from BlockStmt b, int n, ComplexStmt complexStmt
 where
   n = strictcount(ComplexStmt s | s = b.getAStmt()) and
   n > 3 and

cpp/ql/src/Best Practices/Hiding/DeclarationHidesParameter.ql

@@ -11,20 +11,35 @@
 
 import cpp
 
+/**
+ * Gets the template that a function `f` is constructed from, or just `f` if it
+ * is not from a template instantiation.
+ */
+Function getConstructedFrom(Function f) {
+  f.isConstructedFrom(result)
+  or
+  not f.isConstructedFrom(_) and
+  result = f
+}
+
 /**
  * Gets the parameter of `f` with name `name`, which has to come from the
  * _definition_ of `f` and not a prototype declaration.
  * We also exclude names from functions that have multiple definitions.
  * This should not happen in a single application but since we
  * have a system wide view it is likely to happen for instance for
  * the main function.
+ *
+ * Note: we use `getConstructedFrom` to ensure that we look at template
+ * functions rather than their instantiations. We get better results this way
+ * as the instantiation is artificial and may have inherited parameter names
+ * from the declaration rather than the definition.
  */
 ParameterDeclarationEntry functionParameterNames(Function f, string name) {
   exists(FunctionDeclarationEntry fe |
     result.getFunctionDeclarationEntry() = fe and
-    fe.getFunction() = f and
+    getConstructedFrom(f).getDefinition() = fe and
     fe.getLocation() = f.getDefinitionLocation() and
-    result.getFile() = fe.getFile() and // Work around CPP-331
     strictcount(f.getDefinitionLocation()) = 1 and
     result.getName() = name
   )

cpp/ql/src/Best Practices/Hiding/DeclarationHidesVariable.ql

@@ -17,7 +17,7 @@ where
   shadowing(lv1, lv2) and
   not lv1.isCompilerGenerated() and
   not lv2.isCompilerGenerated() and
-  not lv1.getParentScope().(Block).isInMacroExpansion() and
-  not lv2.getParentScope().(Block).isInMacroExpansion()
+  not lv1.getParentScope().(BlockStmt).isInMacroExpansion() and
+  not lv2.getParentScope().(BlockStmt).isInMacroExpansion()
 select lv1, "Variable " + lv1.getName() + " hides another variable of the same name (on $@).", lv2,
   "line " + lv2.getLocation().getStartLine().toString()

cpp/ql/src/Best Practices/Likely Errors/EmptyBlock.ql

@@ -14,7 +14,7 @@
 
 import cpp
 
-predicate emptyBlock(ControlStructure s, Block b) {
+predicate emptyBlock(ControlStructure s, BlockStmt b) {
   b = s.getAChild() and
   not exists(b.getAChild()) and
   not b.isInMacroExpansion() and
@@ -23,7 +23,7 @@ predicate emptyBlock(ControlStructure s, Block b) {
 
 class AffectedFile extends File {
   AffectedFile() {
-    exists(Block b |
+    exists(BlockStmt b |
       emptyBlock(_, b) and
       this = b.getFile()
     )
@@ -37,7 +37,7 @@ class AffectedFile extends File {
 class BlockOrNonChild extends Element {
   BlockOrNonChild() {
     (
-      this instanceof Block
+      this instanceof BlockStmt
       or
       this instanceof Comment
       or
@@ -78,7 +78,7 @@ class BlockOrNonChild extends Element {
 /**
  * A block that contains a non-child element.
  */
-predicate emptyBlockContainsNonchild(Block b) {
+predicate emptyBlockContainsNonchild(BlockStmt b) {
   emptyBlock(_, b) and
   exists(BlockOrNonChild c, AffectedFile file |
     c.(BlockOrNonChild).getStartRankIn(file) = 1 + b.(BlockOrNonChild).getStartRankIn(file) and
@@ -91,7 +91,7 @@ predicate emptyBlockContainsNonchild(Block b) {
  * A block that is entirely on one line, which also contains a comment.  Chances
  * are the comment is intended to refer to the block.
  */
-predicate lineComment(Block b) {
+predicate lineComment(BlockStmt b) {
   emptyBlock(_, b) and
   exists(Location bLocation, File f, int line |
     bLocation = b.getLocation() and
@@ -106,7 +106,7 @@ predicate lineComment(Block b) {
   )
 }
 
-from ControlStructure s, Block eb
+from ControlStructure s, BlockStmt eb
 where
   emptyBlock(s, eb) and
   not emptyBlockContainsNonchild(eb) and

cpp/ql/src/Critical/DeadCodeGoto.ql

@@ -12,15 +12,15 @@
 import cpp
 import semmle.code.cpp.commons.Exclusions
 
-Stmt getNextRealStmt(Block b, int i) {
+Stmt getNextRealStmt(BlockStmt b, int i) {
   result = b.getStmt(i + 1) and
   not result instanceof EmptyStmt
   or
   b.getStmt(i + 1) instanceof EmptyStmt and
   result = getNextRealStmt(b, i + 1)
 }
 
-from JumpStmt js, Block b, int i, Stmt s
+from JumpStmt js, BlockStmt b, int i, Stmt s
 where
   b.getStmt(i) = js and
   s = getNextRealStmt(b, i) and

cpp/ql/src/Critical/aliasAnalysisWarning.qhelp

@@ -0,0 +1,11 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<fragment>
+<warning>
+This check is an approximation, so some results may not be actual defects in the program. 
+It is not possible in general to compute the exact value of the variable without running the program with all possible input data.
+</warning>
+</fragment>
+</qhelp>

cpp/ql/src/Critical/callGraphWarning.qhelp

@@ -0,0 +1,12 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<fragment>
+<warning>
+This check is an approximation, so some results may not be actual defects in the program. 
+It is not possible in general to compute which function is actually called in a virtual call,
+or a call through a pointer, without running the program with all possible input data.
+</warning>
+</fragment>
+</qhelp>

cpp/ql/src/Critical/dataFlowWarning.qhelp

@@ -0,0 +1,13 @@
+<!DOCTYPE qhelp PUBLIC
+  "-//Semmle//qhelp//EN"
+  "qhelp.dtd">
+<qhelp>
+<fragment>
+<warning>
+This check is an approximation, so some results may not be actual defects in the program. 
+It is not possible in general to compute the actual branch taken in conditional statements such
+as "if" without running the program with all possible input data. This means that it is not possible
+to determine if a particular statement is going to be executed.
+</warning>
+</fragment>
+</qhelp>