Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
f152b4d
fix: add support for name attributes in HTML fragment extraction
mre Sep 5, 2025
f165f37
feat: implement per-host rate limiting and statistics
mre Sep 7, 2025
c7480ff
fix: skip rate limiting tracking for file:// URLs
mre Sep 7, 2025
462ba32
feat: improve rate limiting logging and output formatting
mre Sep 7, 2025
7d25ea2
Fix lints
mre Sep 8, 2025
956e20a
Fix cookie jar sharing in per-host rate limiting
mre Sep 8, 2025
20844d2
Fix missing User-Agent header in per-host clients
mre Sep 8, 2025
9ecb5e9
Bring back global headers (e.g. for user-agent)
mre Sep 8, 2025
ded5bc1
Fix redirect handling in per-host clients
mre Sep 8, 2025
4577865
Pass missing args: max_redirects, timeout, allow_insecure
mre Sep 8, 2025
4e04271
Refactor host stats formatters to remove unused parameters and improv…
mre Sep 18, 2025
eed7576
remove confusing comment
mre Sep 18, 2025
ae104a0
Create `display_per_host_statistics` in separate file
mre Sep 18, 2025
8437068
Remove redundant check for `self.hosts`
mre Sep 22, 2025
92854c1
Import `std::collections::HashMap`
mre Sep 22, 2025
ada65ac
Use closures instead of if
mre Sep 22, 2025
d6f7836
Rename flags:
mre Sep 22, 2025
5960946
Fix `help` formatting
mre Sep 22, 2025
ea22e44
Reduce code duplication
thomas-zahner Oct 3, 2025
d33650d
Update documentation to reference hosts option
thomas-zahner Nov 21, 2025
dbada0d
clippy --fix
thomas-zahner Nov 27, 2025
3e0755c
Return HostPool instead of Client & code cleanup
thomas-zahner Nov 27, 2025
0939004
Move inner `Arc`s to the outside
thomas-zahner Nov 28, 2025
087ed56
Fix deadlock
thomas-zahner Nov 28, 2025
040971d
Update config options
thomas-zahner Nov 28, 2025
328ee22
Simplify host pool
thomas-zahner Dec 9, 2025
42dc072
Build host-specific reqwest clients again
thomas-zahner Dec 9, 2025
595e634
Remove max_concurrency and global_semaphore
thomas-zahner Dec 9, 2025
d374155
Update docs & reduce complexity
thomas-zahner Dec 10, 2025
3fdd992
Extract output functions
thomas-zahner Dec 10, 2025
7be4516
Replace Window with Vec
thomas-zahner Dec 10, 2025
420e822
Update RateLimitError
thomas-zahner Dec 11, 2025
00d0c13
Create RequestInterval
thomas-zahner Dec 11, 2025
08602e7
Remove RateLimitError
thomas-zahner Dec 12, 2025
75df7e8
Test and improve rate limit header handling
thomas-zahner Dec 15, 2025
3d0d4fa
Apply @mre's suggestions
thomas-zahner Dec 16, 2025
c748f7d
Apply suggestions from code review
thomas-zahner Dec 16, 2025
6297aa0
Fix tests
thomas-zahner Dec 17, 2025
a551fee
Minor improvements
thomas-zahner Dec 18, 2025
5704305
Remove reqwest_client from WebsiteChecker
thomas-zahner Dec 18, 2025
d5e8afe
Reference rate-limits crate as per @mre's suggestion
thomas-zahner Dec 18, 2025
0f21985
Update option names and the default interval value
thomas-zahner Dec 19, 2025
b815e61
Allow 0 to disable per-host rate limiting
thomas-zahner Dec 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix tests
  • Loading branch information
thomas-zahner committed Dec 21, 2025
commit 6297aa0567a1f51158be633ef9fcca487711977e
2 changes: 1 addition & 1 deletion lychee-bin/tests/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2361,7 +2361,7 @@ The config file should contain every possible key for documentation purposes."
#[tokio::test]
async fn test_retry_rate_limit_headers() {
const RETRY_DELAY: Duration = Duration::from_secs(1);
const TOLERANCE: Duration = Duration::from_millis(200);
const TOLERANCE: Duration = Duration::from_millis(500);
let server = wiremock::MockServer::start().await;

wiremock::Mock::given(wiremock::matchers::method("GET"))
Expand Down
5 changes: 4 additions & 1 deletion lychee-lib/src/checker/website.rs
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,10 @@ impl WebsiteChecker {
status
}
}
Err(e) => e.into(),
Err(e) => match e {
ErrorKind::NetworkRequest(error) => Status::from(error),
_ => e.into(),
},
}
}

Expand Down
3 changes: 1 addition & 2 deletions lychee-lib/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -545,8 +545,7 @@ impl Client {
}

let status = match uri.scheme() {
// We don't check tel: URIs
_ if uri.is_tel() => Status::Excluded,
_ if uri.is_tel() => Status::Excluded, // We don't check tel: URIs
_ if uri.is_file() => self.check_file(uri).await,
_ if uri.is_mail() => self.check_mail(uri).await,
_ => self.check_website(uri, credentials).await?,
Expand Down