Feature request: deduplicate URLs when checking

[scarf005]

related: #1452

• I'm using lychee to validate links in my blog's build artifacts.
• however, this means links get called multiple times due to duplicate elements such as headers or footers.

[_site/etc/hm-session-vars.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/kde/kde5-lockscreen.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/www/css.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/index.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/kde/kinoite-sddm.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/kde/kde6-lockscreen.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/www/index.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/www/relay-devtools.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/kde/index.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/etc/hangul.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/scarf005.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/etc/index.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/etc/binary-search.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

[_site/404.html]:
     [429] https://www.gnu.org/licenses/agpl-3.0.en.html | Network error: Too Many Requests

🔍 237 Total (in 8s) ✅ 222 OK 🚫 14 Errors 👻 1 Excluded

in this case, the link to license (https://www.gnu.org/licenses/agpl-3.0.en.html) in footer is duplicated to all pages in build artifact, and sending 15 or them gives 429 Too Many Requests error.

[wkaisertexas]

I am also looking for this feature. If the user interface will not change to declutter duplicate requests, at least a cache of each request made type would help speed things up

[wkaisertexas]

Also, by cache I do not mean the --cache which persists between runs but one which would always be enabled and would be an in-memory hashmap for the results