feat!: Stricter rule test validations (#17654)

* feat!: rule tester require suggestion matchers * feat!: rule tester require message or messageId * feat!: rule tester require output for suggestions * feat!: rule tester only allow desc or messageId in suggestion matchers * feat!: rule tester require suggestion output differs from original source code * feat!: rule tester typecheck only and filename test case properties * feat!: rule tester check whether code and output is equal * docs: update rule tester documentation to reflect additional assertions * fix: fix invalid test cases of builtin rules * fix: remove unnecessary space for the suggestion prefix * chore: fixing merge diversions * fix: remove incorrect claim that data is required if messageId with placeholders is used * chore: remove potentially confusing differentation between missing and failing fixer * fix: better explanation for missing suggestions * feat: support specifying an suggestion amount * fix: ecmaVersion should be a languageOptions property (not parserOptions) * chore: tweak message for omitting output if there is no autofix * chore: fixup using old error message for the output property and update its docs * fix: fixup reported error messages * fix: refine more assertion messages * docs: document suggestion can also be a number * chore: simplify suggestion existence checks * feat: check string and regexp error matchers for missing suggestion matchers * chore: cleanup message for missing suggestions when testing only the message
eslint · Feb 1, 2024 · 47e60f8 · 47e60f8
1 parent 1a94589
commit 47e60f8
Show file tree

Hide file tree

Showing 10 changed files with 404 additions and 181 deletions.
diff --git a/docs/src/integrate/nodejs-api.md b/docs/src/integrate/nodejs-api.md
@@ -735,19 +735,19 @@ A test case is an object with the following properties:
 
 In addition to the properties above, invalid test cases can also have the following properties:
 
-* `errors` (number or array, required): Asserts some properties of the errors that the rule is expected to produce when run on this code. If this is a number, asserts the number of errors produced. Otherwise, this should be a list of objects, each containing information about a single reported error. The following properties can be used for an error (all are optional):
-    * `message` (string/regexp): The message for the error
-    * `messageId` (string): The Id for the error. See [testing errors with messageId](#testing-errors-with-messageid) for details
+* `errors` (number or array, required): Asserts some properties of the errors that the rule is expected to produce when run on this code. If this is a number, asserts the number of errors produced. Otherwise, this should be a list of objects, each containing information about a single reported error. The following properties can be used for an error (all are optional unless otherwise noted):
+    * `message` (string/regexp): The message for the error. Must provide this or `messageId`
+    * `messageId` (string): The Id for the error. Must provide this or `message`. See [testing errors with messageId](#testing-errors-with-messageid) for details
     * `data` (object): Placeholder data which can be used in combination with `messageId`
     * `type` (string): The type of the reported AST node
     * `line` (number): The 1-based line number of the reported location
     * `column` (number): The 1-based column number of the reported location
     * `endLine` (number): The 1-based line number of the end of the reported location
     * `endColumn` (number): The 1-based column number of the end of the reported location
-    * `suggestions` (array): An array of objects with suggestion details to check. See [Testing Suggestions](#testing-suggestions) for details
+    * `suggestions` (array): An array of objects with suggestion details to check. Required if the rule produces suggestions. See [Testing Suggestions](#testing-suggestions) for details
 
     If a string is provided as an error instead of an object, the string is used to assert the `message` of the error.
-* `output` (string, required if the rule fixes code): Asserts the output that will be produced when using this rule for a single pass of autofixing (e.g. with the `--fix` command line flag). If this is `null`, asserts that none of the reported problems suggest autofixes.
+* `output` (string, required if the rule fixes code): Asserts the output that will be produced when using this rule for a single pass of autofixing (e.g. with the `--fix` command line flag). If this is `null` or omitted, asserts that none of the reported problems suggest autofixes.
 
 Any additional properties of a test case will be passed directly to the linter as config options. For example, a test case can have a `languageOptions` property to configure parser behavior:
 
@@ -784,12 +784,12 @@ Please note that `data` in a test case does not assert `data` passed to `context
 
 ### Testing Suggestions
 
-Suggestions can be tested by defining a `suggestions` key on an errors object. The options to check for the suggestions are the following (all are optional):
+Suggestions can be tested by defining a `suggestions` key on an errors object. If this is a number, it asserts the number of suggestions provided for the error. Otherwise, this should be an array of objects, each containing information about a single provided suggestion. The following properties can be used:
 
-* `desc` (string): The suggestion `desc` value
-* `messageId` (string): The suggestion `messageId` value for suggestions that use `messageId`s
-* `data` (object): Placeholder data which can be used in combination with `messageId`
-* `output` (string): A code string representing the result of applying the suggestion fix to the input code
+* `desc` (string): The suggestion `desc` value. Must provide this or `messageId`
+* `messageId` (string): The suggestion `messageId` value for suggestions that use `messageId`s. Must provide this or `desc`
+* `data` (object): Placeholder data which can be used in combination with `messageId`.
+* `output` (string, required): A code string representing the result of applying the suggestion fix to the input code
 
 Example:
 

diff --git a/lib/rule-tester/rule-tester.js b/lib/rule-tester/rule-tester.js
diff --git a/tests/fixtures/testers/rule-tester/suggestions.js b/tests/fixtures/testers/rule-tester/suggestions.js
@@ -164,3 +164,37 @@ module.exports.withoutHasSuggestionsProperty = {
         };
     }
 };
+
+module.exports.withFixerWithoutChanges = {
+    meta: { hasSuggestions: true },
+    create(context) {
+        return {
+            Identifier(node) {
+                if (node.name === "foo") {
+                    context.report({
+                        node,
+                        message: "Avoid using identifiers named 'foo'.",
+                        suggest: [{
+                            desc: "Rename identifier 'foo' to 'bar'",
+                            fix: fixer => fixer.replaceText(node, 'foo')
+                        }]
+                    });
+                }
+            }
+        };
+    }
+};
+
+module.exports.withFailingFixer = {
+    create(context) {
+        return {
+            Identifier(node) {
+                context.report({
+                    node,
+                    message: "some message",
+                    suggest: [{ desc: "some suggestion", fix: fixer => null }]
+                });
+            }
+        };
+    }
+};