Releases · projectdiscovery/katana

@Ice3man543

What's Changed

Fixed higher usages of cpu and memory by katana by @Ice3man543 in #1107
Added content len field by @dogancanbakir in #1033
Added validations to handle bad URLs by @SecGus in #1090
⚠️ Added -tech-detect as optional feature (previously enabled as default) by @Ice3man543 in #1107

   -td, -tech-detect  enable technology detection (works with jsonl output)

Issue closed in this release - https://github.com/projectdiscovery/katana/milestone/10?closed=1

New Contributors

@SecGus made their first contribution in #1090

Full Changelog: v1.1.1...v1.1.2

@dogancanbakir

⚠️ Breaking Changes:

The Passive Crawling capability is being removed from katana and released as a separate project urlfinder by @dogancanbakir in #899

PASSIVE:
    -ps, -passive                   enable passive sources to discover target endpoints
    -pss, -passive-source string[]  passive source to use for url discovery (waybackarchive,commoncrawl,alienvault)

What's Changed

🎉 New Features

Added no-clobber flag to prevent katana from overwriting existing local files. by @dogancanbakir in #827

-ncb, -no-clobber                 do not overwrite output file

Added -store-field-dir flag to make per-host field directory optional by @dogancanbakir in #877

   -sfd, -store-field-dir string     store per-host field to custom directory

🐞 Bug Fixes

Fixed improper logging configuration by @dogancanbakir in #825
Fixed the issue with setting custom headers/cookie headers by @RamanaReddy0M in #813
Fixed header marshalling issue by preserving the header name casing by @dogancanbakir in #924
Fixed ignoring form data extraction with -form-extraction when action is "#" by @dogancanbakir in #1052
Fixed initial host scope checks to allow continued crawling when -crawl-scope is used by @dogancanbakir in #858
Fixed duplicate field output when multiple value is used with -field flag by @dogancanbakir in #1031
Fixed Dockerfile build errors by updating Golang by @dualfade in #1034
Fixed issue where Ctrl-c could not terminate the process in headless mode with -show-browser option by @zrquan in #972
Fixed the issue with setting custom headers/cookie headers in headless mode by @michael2to3 in #868
Fixed cookie handling in hybrid mode to include in output by @alban-stourbe-wmx in #936

Other Changes

The leakless switch is now enabled by default in headless mode, ensuring the browser is terminated when the Go process ends by @dogancanbakir in #831
Updated README for recently introduced flag -e by @zy9ard3 in #826
Corrected typos in comments by @eveneast in #851
Added directive to ensure compatibility with Windows OS or 386 architecture by @Mzack9999 in #841
Added deduplication for stored the fields on scan completion by @dogancanbakir in #885
Updated README for installation command by @dwisiswant0 in #1015
Added a response parser to extract endpoints from htmx attributes by @zrquan in #994
Updated chrome launcher logic to prioritize the specified Chrome path with -scp over default paths by @zrquan in #979
Added automatic form filling for select and textarea in forms by @alban-stourbe-wmx in #921
Disabled headless auto form filling by @dogancanbakir in #918
Added more JavaScript Libraries to improve parsing coverage by @geeknik in #900
Corrected path handling with filepath.Join to use OS-specific separators. @ShuBo6 in #883
Updated Dockerfile to recent Golang and Alpine images by @o6uoq in #886
Improved form extraction logic to use HTML placeholder if input value is missing by @fmuttis in #957

New Contributors

@zy9ard3 made their first contribution in #826
@eveneast made their first contribution in #851
@ShuBo6 made their first contribution in #883
@o6uoq made their first contribution in #886
@michael2to3 made their first contribution in #868
@alban-stourbe-wmx made their first contribution in #936
@zrquan made their first contribution in #972
@fmuttis made their first contribution in #957
@dualfade made their first contribution in #1034
@dwisiswant0 made their first contribution in #1015

Full Changelog: v1.1.0...v1.1.1

@dogancanbakir

What's Changed

🎉 New Features

Introduced passive crawling by @dogancanbakir in #781

   -ps, -passive  enable passive sources to discover target endpoints
   -pss, -passive-source string[]  passive source to use for url discovery (waybackarchive,commoncrawl,alienvault)

Example:

katana -u tesla.com -passive -f qurl
...
https://static-assets-pay.tesla.com/api/payment-schema/creditcard?countrycode=dk
https://static-assets-pay.tesla.com/api/paymenttypes/?clientlibrary=payment-website
https://tradepartnertickets.tesla.com/dist/main.css?029b26e9be3aef4fc82c
https://tradepartnertickets.tesla.com/dist/vendors.vendors.css?029b26e9be3aef4fc82c
https://workforce.tesla.com/auth/callback?code=na_921f11c72db1d416c2fb624ea94ab5e1bad5f803
...
...
...
[INF] Found 208114 endpoints for https://tesla.com in 2m11.65937825s (commoncrawl: 128, alienvault: 1615, waybackarchive: 206371)

Introduced option to exclude targets to crawl by @dogancanbakir in #731

   -e, -exclude string[]  exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)

🐞 Bug Fixes

Fixed issue with headless redirect by @Mzack9999 in #823
Fixed issue with filter and match functionality (SDK) by @dogancanbakir in #775
Fixed issue with -kf option to skip if the maximum depth is exceeded by @dogancanbakir in #799

Other Changes

Added conditional support to include go-tree-sitter in build process by @RamanaReddy0M in #814
Fixed nil pointer check by @dogancanbakir in #773
Fixed error check on hybrid request parsing by @niudaii in #804
Updated jsluice to new version with more results by @Ice3man543 in #695
Updated jsluice by @geeknik in #800

New Contributors

@beac0n5 made their first contribution in #732
@toufik-airane made their first contribution in #736
@niudaii made their first contribution in #804
@geeknik made their first contribution in #800

Full Changelog: v1.0.5...v1.1.0

@ErikOwen

What's Changed

🎉 New Features

Added option to disable redirects with -disable-redirects by @ErikOwen in #630

   -dr, -disable-redirects  disable following redirects (default false)

🐞 Bug Fixes

Fixed issue with header in output with XHR requests by @ErikOwen in #631
Fixed with -aff config option by @dogancanbakir in #676
Fixed issue with -kf option by @dogancanbakir in #655
Fixed issue with blank field with empty value by @dogancanbakir in #687

Full Changelog: v1.0.4...v1.0.5

@c3l3si4n

What's Changed

🎉 New Features

Added support for custom regex on field scope -fs option by @c3l3si4n in #571

katana -fs '(company-staging.io|company.com)' -u company.com

Added option to disable redirects by @WigzyDev in #588

   -dr, -disable-redirects  disable following redirects (default false)

Added -resume crawl option by @dogancanbakir in #538

   -resume string  resume crawl using resume.cfg

🐞 Bug Fixes

Fixed issue with -match-regex option by @dogancanbakir in #565
Fixed headless options parsing issue by @dogancanbakir in #564
Fixed missing ExtractRelativeEndpoints() in scriptJSFileRegexParser() by @0x123456789 in #587

Other Changes

Added build deps for go-tree-sitter -> tree-sitter by @dogancanbakir in #547
Added readme syntax highlight by @apriil15 in #553

New Contributors

@apriil15 made their first contribution in #553
@0x123456789 made their first contribution in #587
@c3l3si4n made their first contribution in #571
@WigzyDev made their first contribution in #588

Full Changelog: v1.0.3...v1.0.4

@dogancanbakir

What's Changed

🎉 New Features

Added DSL based advance response matcher and filter options by @dogancanbakir in #537

   -mdc, -match-condition string  match response with dsl based condition
   -fdc, -filter-condition string  filter response with dsl based condition

Added option for TLS connection randomization by @Mzack9999 in #489

   -tlsi, -tls-impersonate  enable experimental client hello (ja3) tls randomization

Added option for jsluice based javascript parsing by @Ice3man543 in #492

   -jsl, -jsluice  enable jsluice parsing in javascript file (memory intensive)

Added option to parse and include form, input, textarea & select elements in jsonl output by @aristosMiliaressis in #464

   -fx, -form-extraction  extract form, input, textarea & select elements in jsonl output

Added option to log XHR requests in jsonl output by @aristosMiliaressis in #481

   -xhr, -xhr-extraction  extract xhr request url,method in jsonl output

Added the stored response path information in json output when -j used with -sr option by @ErikOwen in #532

🐞 Bug Fixes

Fixed an issue wiht extension match / filter by @dogancanbakir in #491

🔨 Maintenance

Fixed release workflow to accommodate tree-sitter bindings by @Mzack9999 in #527

Other Changes

Added time unit support for -ct option by @RamanaReddy0M in #500

   -ct, -crawl-duration value  maximum duration to crawl the target for (s, m, h, d) (default s)

Added option to allow katana to use an existing chrome instance by @DanielIntruder in #490

   -cwu, -chrome-ws-url string  use chrome browser instance launched elsewhere with the debugger listening at this URL

Improved normalization of action & method attributes in form extraction feature by @aristosMiliaressis in #513

New Contributors

@aristosMiliaressis made their first contribution in #464
@DanielIntruder made their first contribution in #490
@jen140 made their first contribution in #511
@ErikOwen made their first contribution in #532

Full Changelog: v1.0.2...v1.0.3

@RamanaReddy0M

What's Changed

🎉 Features

Added support for custom header input from file by @RamanaReddy0M in #436

katana -H cookie.txt -u https://mail.google.com -headless

Added option to exclude raw request/response and body in jsonl output by @maik-s in #460

   -or, -omit-raw                    omit raw requests/responses from jsonl output
   -ob, -omit-body                   omit response body from jsonl output

🐞 Bugs

Fixed issue with regex parsing by @yuzhe-Mortal in #406
Fixed panic issue in headless mode by @RamanaReddy0M in #428
Fixed issue with custom header input support in headless mode by @tarunKoyalwar in #437

🔨 Maintenance

Added documentation for header / cookie input by @dogancanbakir in #461

Issues closed in this release - https://github.com/projectdiscovery/katana/milestone/6?closed=1

New Contributors

@maik-s made their first contribution in #460

Full Changelog: v1.0.1...v1.0.2

@RamanaReddy0M

What's Changed

Added support to ignore crawling same path with different query-param by @RamanaReddy0M in #371

   -iqp, -ignore-query-params  ignore crawling same path with different query-param values

Added support for adding url scheme with input by @iamargus95 in #399

katana -u hackerone.com

Added transparent embedding of common code by @Mzack9999 in #373
Updated using as lib example to reflect struct changes by @ShubhamRasal in #397
Updated using as lib example by @dogancanbakir in #390
Updated net/url with urlutil by @iamargus95 in #399
Fixed an issue with -do option by @Mzack9999 in #387

Issues closed in this release - https://github.com/projectdiscovery/katana/milestone/5?closed=1

New Contributors

@h4r5h1t made their first contribution in #378
@dogancanbakir made their first contribution in #390
@iamargus95 made their first contribution in #399

Full Changelog: v1.0.0...v1.0.1

@Mzack9999

What's Changed

Warning: breaking changes in json output structure.

previous format

{
  "timestamp": "2022-11-05T22:33:27.745815+05:30",
  "endpoint": "https://www.iana.org/domains/example",
  "source": "https://example.com",
  "tag": "a",
  "attribute": "href"
}

new format

{
  "timestamp": "2023-03-20T16:23:58.027559+05:30",
  "request": {
    "method": "GET",
    "endpoint": "https://example.com",
    "tag": "a",
    "attribute": "href",
    "source": "http://www.iana.org/domains/reserved",
    "raw": "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n"
  },
  "response": {
    "status_code": 200,
    "headers": {
      "accept_ranges": "bytes",
      "expires": "Mon, 27 Mar 2023 10:53:58 GMT",
      "last_modified": "Thu, 17 Oct 2019 07:18:26 GMT",
      "content_type": "text/html; charset=UTF-8",
      "server": "ECS (dcb/7EA3)",
      "vary": "Accept-Encoding",
      "etag": "\"3147526947\"",
      "cache_control": "max-age=604800",
      "x_cache": "HIT",
      "date": "Mon, 20 Mar 2023 10:53:58 GMT",
      "age": "331239"
    },
    "body": "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
    "technologies": [
      "Azure",
      "Amazon ECS",
      "Amazon Web Services",
      "Docker",
      "Azure CDN"
    ],
    "raw": "HTTP/1.1 200 OK\r\nContent-Length: 1256\r\nAccept-Ranges: bytes\r\nAge: 331239\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Mon, 20 Mar 2023 10:53:58 GMT\r\nEtag: \"3147526947\"\r\nExpires: Mon, 27 Mar 2023 10:53:58 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (dcb/7EA3)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\n\r\n<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
  }
}

Refactored katana core by @Mzack9999 in #312
- for synchronous correlation between http request/response
- added request/response data into json output
- added status code information in json output
- updated json output structure
Added technology detection of crawled endpoints in json output using wappalyzer by @Mzack9999 in #294

...
    "technologies": [
      "Azure",
      "Amazon ECS",
      "Amazon Web Services",
      "Docker",
      "Azure CDN"
    ],
...

Added match and filter support for discovered endpoint by @ShubhamRasal in #346

   -mr, -match-regex string[]  regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[]  regex or list of regex to filter on output url (cli, file)

Added custom resolver input support by @ShubhamRasal in #340

   -r, -resolvers string[]  list of custom resolver (file or comma separated)

Added self update option by @RamanaReddy0M in #356

   -up, -update  update katana to latest version

Added support for ufile field based filter by @Mzack9999 in #357

katana -f ufile -u https://www.tesla.com

Added -chrome-data-dir / -cdd flag to implement authenticated crawling #230 by @parthmalhotra in #231
Added examples using rdn, fqdn, etc. scope filter in readme by @olearycrew in #288
Fixed tls version error when using burp proxy by @Mzack9999 in #311
Fixed issues with katana consuming available cpu by @ShubhamRasal in #362
Fixed Dockerfile to avoid version mismatch by @olearycrew in #279
Updated regex to detect/extract more endpoints by @yuzhe-Mortal in #249
Updated debug logging by @tarunKoyalwar in #278

Issues closed in release - https://github.com/projectdiscovery/katana/milestone/4

New Contributors

@olearycrew made their first contribution in #279
@tarunKoyalwar made their first contribution in #278
@yuzhe-Mortal made their first contribution in #249
@RamanaReddy0M made their first contribution in #356

Full Changelog: v0.0.3...v1.0.0

@ShubhamRasal

What's Changed

Added custom field support for custom data extraction by @ShubhamRasal in #225
Added support to store http request/responses to file by @edoardottt in #185
Added option to specify local chrome binary path (-scp) by @parthmalhotra in #202
Added chrome requirements recommended setup for ubuntu by @Mzack9999 in #251
Added example to use katana as a library by @ShubhamRasal in #205
Added support for error logs (-elog) option by @Mzack9999 in #267
Added self-diagnostic functionality (-hc) option by @Mzack9999 in #260
Added prototype of cycle detection by @Mzack9999 in #250
Fixed a bug to use proxy in headless mode by @parthmalhotra in #169
Fixed a bug to load complete response in headless mode by @parthmalhotra in #195
Fixed a bug in the extension match/filer option by @ShubhamRasal in #184
Fixed a bug in the rate limit option by @ShubhamRasal in #214

Issues closed in release - https://github.com/projectdiscovery/katana/milestone/3?closed=1

New Contributors

@glaucocustodio made their first contribution in #173
@parthmalhotra made their first contribution in #169
@ShubhamRasal made their first contribution in #184
@0xflotus made their first contribution in #268

Full Changelog: v0.0.2...v0.0.3

Releases: projectdiscovery/katana

v1.1.2

What's Changed

New Contributors

Contributors

v1.1.1

⚠️ Breaking Changes:

What's Changed

🎉 New Features

🐞 Bug Fixes

Other Changes

New Contributors

Contributors

v1.1.0

What's Changed

🎉 New Features

Example:

🐞 Bug Fixes

Other Changes

New Contributors

Contributors

v1.0.5

What's Changed

🎉 New Features

🐞 Bug Fixes

Contributors

v1.0.4

What's Changed

🎉 New Features

🐞 Bug Fixes

Other Changes

New Contributors

Contributors

v1.0.3

What's Changed

🎉 New Features

🐞 Bug Fixes

🔨 Maintenance

Other Changes

New Contributors

Contributors

v1.0.2

What's Changed

🎉 Features

🐞 Bugs

🔨 Maintenance

New Contributors

Contributors

v1.0.1

What's Changed

New Contributors

Contributors

v1.0.0

What's Changed

New Contributors

Contributors

v0.0.3

What's Changed

New Contributors

Contributors