Hacking Articles Tips Tricks Videos Tutorials

REST API fuzzer and negative testing tool. Run thousands of self-healing API tests within minutes with no coding effort!Comprehensive: tests are generated automatically based on a large number scenarios and cover every field and headerIntelligent: tests are generated based on data types and constraints; each Fuzzer have specific expectations depending on the scenario under testHighly Configurable: high amount of customization: you can exclude specific Fuzzers, HTTP response codes, provide business context and a lot moreSelf-Healing: as tests are generated, any OpenAPI spec change is picked up automaticallySimple to Learn: flat learning curve, with intuitive configuration and syntaxFast: automatic process for write, run and report tests which covers thousands of scenarios within minutes
OverviewBy using a simple and minimal syntax, with a flat learning curve, CATS (Contract Auto-generated Tests for Swagger) enables you to generate thousands of API tests within minutes with no coding effort. All tests are generated, run and reported automatically based on a pre-defined set of 89 Fuzzers. The Fuzzers cover a wide range of input data from fully random large Unicode values to well crafted, context dependant values based on the request data types and constraints. Even more, you can leverage the fact that CATS generates request payloads dynamically and write simple end-to-end functional tests.

___________________________
@hacking_Attack
@Hacking_Video

6 viewsedited 13:34

Please check the Slicing Strategies (https://github.com/Endava/cats#slicing-strategies-for-running-cats) section for making CATS run fast and comprehensive in the same time. Tutorials on how to use CATSThis is a list of articles with step-by-step guides on how to use CATS:Testing the GitHub API with CATS (https://ludovicianul.github.io/2020/10/05/github-api-testing/)How to write self-healing functional tests with no coding effort (https://ludovicianul.github.io/2020/09/09/cats/)Some bugs found by CATShashicorp/vault#13274 (https://github.com/hashicorp/vault/issues/13274) | hashicorp/vault#13273 (https://github.com/hashicorp/vault/issues/13273)hashicorp/vault#13225 (https://github.com/hashicorp/vault/issues/13225) | hashicorp/vault#13232 (https://github.com/hashicorp/vault/issues/13232)go-gitea/gitea#19397 (https://github.com/go-gitea/gitea/issues/19397) | go-gitea/gitea#19398 (https://github.com/go-gitea/gitea/issues/19398)go-gitea/gitea#19399 (https://github.com/go-gitea/gitea/issues/19399)InstallationHomebrew brew tap endava/tap > brew install cats">> brew tap endava/tap
> brew install catsManualCATS is bundled both as an executable JAR or a native binary. The native binaries do not need Java installed.After downloading your OS native binary, you can add it in classpath so that you can execute it as any other command line (https://www.kitploit.com/search/label/Command%20Line) tool:sudo cp cats /usr/local/bin/catsYou can also get autocomplete by downloading the cats_autocomplete (https://github.com/Endava/cats/blob/master/cats_autocomplete) script and do:source cats_autocompleteTo get persistent autocomplete, add the above line in ~/.zshrc or ./bashrc, but make sure you put the fully qualified path for the cats_autocomplete script.You can also check the cats_autocomplete source for alternative setup.There is no native binary for Windows, but you can use the uberjar version. This requires Java 11+ to be installed.You can run it as java -jar cats.jar.Head to the releases page to download the latest versions: https://github.com/Endava/cats/releases.BuildYou can build CATS from sources on you local box. You need Java 11+. Maven is already bundled.Before running the first build, please make sure you do a ./mvnw clean. CATS uses a fork ok OKHttpClient which will install locally under the 4.9.1-CATS version, so don't worry about overriding the official versions.You can use the following Maven command to build the project:./mvnw package -Dquarkus.package.type=uber-jarcp target/You will end up with a cats.jar in the target folder. You can run it wih java -jar cats.jar ....You can also build native images using a GraalVM Java version../mvnw package -PnativeNote: You will need to configure Maven with a Github PAT (https://docs.github.com/en/free-pro-team@latest/packages/guides/configuring-apache-maven-for-use-with-github-packages) with read-packages scope to get some dependencies for the build.Notes on Unit TestsYou may see some ERROR log messages while running the Unit Tests. Those are expected behaviour for testing the negative scenarios of the Fuzzers.Running CATSBlackbox modeBlackbox mode means that CATS doesn't need any specific context. You just need to provide the service URL, the OpenAPI spec and most probably authentication headers (https://github.com/Endava/cats#headers-file).> cats --contract=openapy.yaml --server=http://localhost:8080 --headers=headers.yml --blackboxIn blackbox mode CATS will only report ERRORs if the received HTTP response code is a 5XX. Any other mismatch between what the Fuzzer expects vs what the service returns (for example service returns 400 and service returns 200) will be ignored.The blackbox mode is similar to a smoke test. It will quickly tell you if the application has major bugs that must be addressed immediately.Context modeThe real power of CATS relies on running it in a non-blackbox mode also called context mode. Each Fuzzer has an expected HTTP response code based on the scenario under

___________________________
@hacking_Attack
@Hacking_Video

GitHub

GitHub - Endava/cats: CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates…

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-heal...

6 viewsedited 13:34

Hacking Articles Tips Tricks Videos Tutorials

test and will also check if the response is matching the schema defined in the OpenAPI spec specific to that response code. This will allow you to tweak either your OpenAPI spec or service behaviour in order to create good quality APIs and documentation and also to avoid possible serious bugs.Running CATS in context mode usually implies providing it a --refData (https://github.com/Endava/cats#reference-data-file) file with resource identifiers specific to the business logic. CATS cannot create data on its own (yet), so it's important that any request field or query param that requires pre-existence of those entities/resources to be created in advance and added to the reference data file.> cats --contract=openapy.yaml --server=http://localhost:8080 --headers=headers.yml --refData=referenceData.ymlNotes on skipped TestsYou may notice a significant number of tests marked as skipped. CATS will try to apply all Fuzzers to all fields, but this is not always possible. For example the BooleanFieldsFuzzer cannot be applied to String fields. This is why that test attempt will be marked as skipped. It was an intentional decision to also report the skipped tests in order to show that CATS actually tries all the Fuzzers on all the fields/paths/endpoints.Additionally, CATS support a lot more arguments (https://github.com/Endava/cats#available-arguments) that allows you to restrict the number of fuzzers, provide timeouts, limit the number of requests per minute and so on.Understanding how CATS works and reports resultsCATS generates tests based on configured Fuzzers. Each Fuzzer has a specific scenario and a specific expected result. The CATS engine will run the scenario, get the result from the service and match it with the Fuzzer expected result. Depending on the matching outcome, CATS will report as follows:INFO/SUCCESS is expected and documented behaviour. No need for action.WARN is expected but undocumented behaviour or some misalignment between the contract and the service. This will ideally be actioned.ERROR is abnormal/unexpected behaviour. This must be actioned.CATS will iterate through all endpoints, all HTTP methods and all the associated requests bodies and parameters (including multiple combinations when dealing with oneOf/anyOf elements) and fuzz their values considering their defined data type and constraints. The actual fuzzing depends on the specific Fuzzer executed. Please see the list of fuzzers and their behaviour. There are also differences on how the fuzzing works depending on the HTTP method:for methods with request bodies like POST, PUT the fuzzing will be applied at the request body data models levelfor methods without request bodies like GET, DELETE the fuzzing will be applied at the URL parameters levelThis means that for methods with request bodies (POST,PUT) that have also URL/path parameters, you need to supply the path parameters via urlParams or the referenceData file as failure to do so will result in Illegal character in path at index ... errors.Interpreting ResultsHTML_JSHTML_JS is the default report produced by CATS. The execution report in placed a folder called cats-report/TIMESTAMP or cats-report depending on the --timestampReports argument. The folder will be created inside the current folder (if it doesn't exist) and for each run a new subfolder will be created with the TIMESTAMP value when the run started. This allows you to have a history of the runs. The report itself is in the index.html file, where you can:filter test runs based on the result: All, Success, Warn and Errorfilter based on the Fuzzer so that you can only see the runs for that specific Fuzzersee summary with all the tests with their corresponding path against they were run, and the resulthave ability to click on any tests and get details about the Scenario being executed, Expected Result, Actual result as well as request/response detailsAlong with the summary from index.html each individual test will have a specific TestXXX.html

___________________________
@hacking_Attack
@Hacking_Video

GitHub

GitHub - Endava/cats: CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates…

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-heal...

7 viewsedited 13:34

Hacking Articles Tips Tricks Videos Tutorials

page with more details, as well as a json version of the test which can be latter replayed using > cats replay TestXXX.json.Understanding the Result Reason values:Unexpected Exception - reported as error; this might indicate a possible bug in the service or a corner case that is not handled correctly by CATSNot Matching Response Schema - reported as a warn; this indicates that the service returns an expected response code and a response body, but the response body does not match the schema defined in the contractUndocumented Response Code - reported as a warn; this indicates that the service returns an expected response code, but the response code is not documented in the contractUnexpected Response Code - reported as an error; this indicates a possible bug in the service - the response code is documented, but is not expected for this scenarioUnexpected Behaviour - reported as an error; this indicates a possible bug in the service - the response code is neither documented nor expected for this scenarioNot Found - reported as an error in order to force providing more context; this indicates that CATS needs additional business context in order to run successfully - you can do this using the --refData and/or --urlParams argumentsThis is the summary page:

___________________________
@hacking_Attack
@Hacking_Video

6 viewsedited 13:34

Hacking Articles Tips Tricks Videos Tutorials

6 views13:34

Hacking Articles Tips Tricks Videos Tutorials

And this is what you get when you click on a specific test:

___________________________
@hacking_Attack
@Hacking_Video

6 viewsedited 13:34

Hacking Articles Tips Tricks Videos Tutorials

6 views13:34

Hacking Articles Tips Tricks Videos Tutorials

6 views13:34

Hacking Articles Tips Tricks Videos Tutorials

- value2
- value3
oneOfSelection:
element#type: "Value"
expectedResponseCode: HTTP_CODE
httpMethod: HTTP_NETHODAnd a typical run will look like:> cats run functionalFuzzer.yml -c contract.yml -s http://localhost:8080This is a description of the elements within the functionalFuzzer.yml file:you can supply a description of the test. This will be set as the Scenario description. If you don't supply a description the testNumber will be used instead.you can have multiple tests under the same path: test1, test2, etc.expectedResponseCode is mandatory, otherwise the Fuzzer will ignore this test. The expectedResponseCode tells CATS what to expect from the service when sending this test.at most one of the properties can have multiple values. When this situation happens, that test will actually become a list of tests one for each of the values supplied. For example in the above example prop7 has 3 values. This will actually result in 3 tests, one for each value.test within the file are executed in the declared order. This is why you can have outputs from one test act as inputs for the next one(s) (see the next section for details).if the supplied httpMethod doesn't exist in the OpenAPI given path, a warning will be issued and no test will be executedif the supplied httpMethod is not a valid HTTP method, a warning will be issued and no test will be executedif the request payload uses a oneOf element to allow multiple request types, you can control which of the possible types the FunctionalFuzzer will apply to using the oneOfSelection keyword. The value of the oneOfSelection keyword must match the fully qualified name of the discriminator.if no oneOfSelection is supplied, and the request payload accepts multiple oneOf elements, than a custom test will be created for each type of payloadthe file uses Json path (https://github.com/json-path/JsonPath) syntax for all the properties you can supply; you can separate elements through # as in the example above instead of .Dealing with oneOf, anyOfWhen you have request payloads which can take multiple object types, you can use the oneOfSelection keyword to specify which of the possible object types is required by the FunctionalFuzzer. If you don't provide this element, all combinations will be considered. If you supply a value, this must be exactly the one used in the discriminator.Correlating TestsAs CATs mostly relies on generated data with small help from some reference data, testing complex business scenarios with the pre-defined Fuzzers is not possible. Suppose we have an endpoint that creates data (doing a POST), and we want to check its existence (via GET). We need a way to get some identifier from the POST call and send it to the GET call. This is now possible using the FunctionalFuzzer. The functionalFuzzerFile can have an output entry where you can state a variable name, and its fully qualified name from the response in order to set its value. You can then refer the variable using ${variable_name} from another test in order to use its value.Here is an example:/pet:
test_1:
description: Create a Pet
httpMethod: POST
name: "My Pet"
expectedResponseCode: 200
output:
petId: pet#id
/pet/{id}:
test_2:
description: Get a Pet
id: ${petId}
expectedResponseCode: 200Suppose the test_1 execution outputs:{
"pet":
{
"id" : 2
}
}When executing test_1 the value of the pet id will be stored in the petId variable (value 2). When executing test_2 the id parameter will be replaced with the petId variable (value 2) from the previous case.Please note: variables are visible across all custom tests; please be careful with the naming as they will get overridden.Verifying responsesThe FunctionalFuzzer can verify more than just the expectedResponseCode. This is achieved using the verify element. This is an extended version of the above functionalFuzzer.yml file./pet:
test_1:

___________________________
@hacking_Attack
@Hacking_Video

GitHub

GitHub - json-path/JsonPath: Java JsonPath implementation

Java JsonPath implementation. Contribute to json-path/JsonPath development by creating an account on GitHub.

7 viewsedited 13:34

Hacking Articles Tips Tricks Videos Tutorials

petId: pet#id
verify:
pet#name: "Baby"
pet#id: "[0-9]+"
/pet/{id}:
test_2:
description: Get a Pet
id: ${petId}
expectedResponseCode: 200Considering the above file:the FunctionalFuzzer will check if the response has the 2 elements pet#name and pet#idif the elements are found, it will check that the pet#name has the Baby value and that the pet#id is numericThe following json response will pass test_1:{
"pet":
{
"id" : 2,
"name": "Baby"
}
}But this one won't (pet#name is missing):{
"pet":
{
"id" : 2
}
}You can also refer to request fields in the verify section by using the ${request#..} qualifier. Using the above example, by having the following verify section:/pet:
test_1:
description: Create a Pet
httpMethod: POST
name: "My Pet"
expectedResponseCode: 200
output:
petId: pet#id
verify:
pet#name: "${request#name}"
pet#id: "[0-9]+"It will verify if the response contains a pet#name element and that its value equals My Pet as sent in the request.Some notes:verify parameters support Java regexes as valuesyou can supply more than one parameter to check (as seen above)if at least one of the parameters is not present in the response, CATs will report an errorif all parameters are found and have valid values, but the response code is not matched, CATs will report a warningif all the parameters are found and match their values, and the response code is as expected, CATs will report a successWorking with additionalProperties in FunctionalFuzzerYou can also set additionalProperties fields through the functionalFuzzerFile using the same syntax as for Setting additionalProperties in Reference Data (https://github.com/Endava/cats#setting-additionalproperties).FunctionalFuzzer Reserved keywordsThe following keywords are reserved in FunctionalFuzzer tests: output, expectedResponseCode, httpMethod, description, oneOfSelection, verify, additionalProperties, topElement and mapValues.Security FuzzerAlthough CATs is not a security testing tool, you can use it to test basic security scenarios by fuzzing specific fields with different sets of nasty strings (https://github.com/minimaxir/big-list-of-naughty-strings). The behaviour is similar to the FunctionalFuzzer. You can use the exact same elements for output variables, test correlation, verify responses and so forth, with the addition that you must also specify a targetFields and/or targetFieldTypes and a stringsList element. A typical securityFuzzerFile will look like this:/pet:
test_1:
description: Run XSS scenarios
name: "My Pet"
expectedResponseCode: 200
httpMethod: all
targetFields:
- pet#id
- pet#description
stringsFile: xss.txtAnd a typical run:> cats run securityFuzzerFile.yml -c contract.yml -s http://localhost:8080You can also supply output, httpMethod, oneOfSelection and/or verify (with the same behaviour as within the FunctionalFuzzer) if they are relevant to your case.The file uses Json path (https://github.com/json-path/JsonPath) syntax for all the properties you can supply; you can separate elements through # as in the example instead of ..This is what the SecurityFuzzer will do after parsing the above securityFuzzerFile:it will add the fixed value "My Pet" to all the request for the field namefor each field specified in the targetFields i.e. pet#id and pet#description it will create requests for each line from the xss.txt file and supply those values in each fieldif you consider the xss.txt sample file included in the CATs repo, this means that it will send 21 requests targeting pet#id and 21 requests targeting pet#description i.e. a total of 42 testsfor each of these 42 tests, the SecurityFuzzer will expect a 200 response code. If another response code is returned, then CATs will report the test as error.If you want the above logic to apply to all paths, you can use all as the path name:all:

___________________________
@hacking_Attack
@Hacking_Video

GitHub

GitHub - Endava/cats: CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates…

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-heal...

5 viewsedited 13:34

Hacking Articles Tips Tricks Videos Tutorials

stringsFile: xss.txtInstead of specifying the field names, you can broader to scope to target certain fields types. For example, if we want to test for XSS in all string fields, you can have the following securityFuzzerFile:all:
test_1:
description: Run XSS scenarios
name: "My Pet"
expectedResponseCode: 200
httpMethod: all
targetFieldTypes:
- string
stringsFile: xss.txtAs an idea on how to create security tests, you can split the nasty strings (https://github.com/minimaxir/big-list-of-naughty-strings) into multiple files of interest in your particular context. You can have a sql_injection.txt, a xss.txt, a command_injection.txt and so on. For each of these files, you can create a test entry in the securityFuzzerFile where you include the fields you think are meaningful for these types of tests. (It was a deliberate choice (for now) to not include all fields by default.) The expectedResponseCode should be tweaked according to your particular context. Your service might sanitize data before validation, so might be perfectly valid to expect a 200 or might validate the fields directly, so might be perfectly valid to expect a 400. A 500 will usually mean something was not handled properly and might signal a possible bug.Working with additionalProperties in SecurityFuzzerYou can also set additionalProperties fields through the functionalFuzzerFile using the same syntax as for Setting additionalProperties in Reference Data (https://github.com/Endava/cats#setting-additionalproperties).SecurityFuzzer Reserved keywordsThe following keywords are reserved in SecurityFuzzer tests: output, expectedResponseCode, httpMethod, description, verify, oneOfSelection, targetFields, targetFieldTypes, stringsFile, additionalProperties, topElement and mapValues.TemplateFuzzerThe TemplateFuzzer can be used to fuzz non-OpenAPI endpoints. If the target API does not have an OpenAPI spec available, you can use a request template to run a limited set of fuzzers. The syntax for running the TemplateFuzzer is as follows (very similar to curl:> cats fuzz -H header=value -X POST -d '{"field1":"value1","field2":"value2","field3":"value3"}' -t "field1,field2,header" -i "2XX,4XX" http://service-url The command will:send a POST request to http://service-urluse the {"field1":"value1","field2":"value2","field3":"value3"} as a templatereplace one by one field1,field2,header with fuzz data and send each request to the service endpointignore 2XX,4XX response codes and report an error when the received response code is not in this listIt was a deliberate choice to limit the fields for which the Fuzzer will run by supplying them using the -t argument. For nested objects, supply fully qualified names: field.subfield.Headers can also be fuzzed using the same mechanism as the fields.This Fuzzer will send the following type of data:null valuesempty valueszalgo textabugidas characterslarge random unicode datavery large strings (80k characters)single and multi code point emojisunicode control charactersunicode separatorsunicode whitespacesFor a full list of options run > cats fuzz -h.You can also supply your own dictionary of data using the -w file argument.HTTP methods with bodies will only be fuzzed at the request payload and headers level.HTTP methods without bodies will be fuzzed at path and query parameters and headers level. In this case you don't need to supply a -d argument.This is an example for a GET request:> cats fuzz -X GET -t "path1,query1" -i "2XX,4XX" http://service-url/paths1?query1=test&query2Reference Data FileThere are often cases where some fields need to contain relevant business values in order for a request to succeed. You can provide such values using a reference data file specified by the --refData argument. The reference data file is a YAML-format file that contains specific fixed values for different paths in the request document. The file structure is as follows:/path/0.1/auth:

___________________________
@hacking_Attack
@Hacking_Video

GitHub

GitHub - minimaxir/big-list-of-naughty-strings: The Big List of Naughty Strings is a list of strings which have a high probability…

The Big List of Naughty Strings is a list of strings which have a high probability of causing issues when used as user-input data. - minimaxir/big-list-of-naughty-strings

5 viewsedited 13:34

About

Blog

Apps

Platform