fix(logger): rotate the log file instead of growing it without bound - #2346

Summary

AsyncWebCrawler.__init__ builds a default AsyncLogger pointed at <base_directory>/.crawl4ai/crawler.log whenever the caller passes no logger=. AsyncLogger._write_to_file appends every message to that file, even with verbose=False (which only silences the console), and nothing rotates or caps it. In one of our long-running containers the file reached 1.1 GB before anyone noticed. async_database.py does the same for crawler_db.log.

This PR makes AsyncLogger and AsyncFileLogger write through the standard library's RotatingFileHandler, capped at 10 MB with 3 backups. Every logger in a process that targets the same path shares one handler, so several crawlers rotate crawler.log once instead of renaming it out from under each other. Rotation across processes is not coordinated, same as RotatingFileHandler itself.

The line format written to the file is unchanged. AsyncWebCrawler and AsyncDatabaseManager need no changes; they pick up rotation through the default logger.

Behaviour change to be aware of: the default goes from unbounded growth to a 10 MB cap with 3 backups. Anyone who needs a complete crawler.log history, or a different policy, can pass their own AsyncLoggerBase via AsyncWebCrawler(logger=...), which already works today.

Related context: #264 confirms file logging was meant to stay on regardless of verbose. This PR keeps that and only bounds the size.

List of files changed and why

• crawl4ai/async_logger.py - _get_file_handler returns a process-wide RotatingFileHandler per path; _write_line sends a pre-formatted line through it. AsyncLogger and AsyncFileLogger use the handler instead of open(..., "a").
• tests/loggers/test_log_rotation.py - two loggers on one path share a handler; with the cap lowered to 2 KB they produce crawler.log plus three backups, every file under the cap.

How Has This Been Tested?

• pytest tests/loggers/test_log_rotation.py tests/test_async_logger_stderr.py passes.
• Built three real AsyncWebCrawler instances (no browser launched) with CRAWL4_AI_BASE_DIRECTORY pointed at a temp dir, lowered the cap to 5 KB, and logged 1200 url_status lines through them. Result: crawler.log plus three backups, each under the cap, all three crawlers sharing one handler.
• black reports no issues on the added lines. The rest of async_logger.py was left untouched since it is not black-formatted today and reformatting it would bury the change.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

🤖 Generated with Claude Code