Problem Description

Sometimes, you need to use custom HTTP headers to bypass the authorization. One example is JWT tokens that are transferred via "Cookie" header.

Proposed Solution

You should be able to add custom HTTP headers to the crawl configuration.