Quick summary
Screaming Frog can crawl large sites beyond its default URL limits by switching to Database Storage mode and adjusting spider configuration settings. This interactive walkthrough shows you exactly how to configure System settings, raise the Max URL limit, and apply Advanced spider options to handle enterprise-scale crawls.
Steps
- Go to Screaming Frog and navigate to Settings > System to open the system configuration panel.
- Select Storage Mode to access the available storage options for the crawl.
- Select Database Storage mode, then click OK to confirm and save the storage setting.
- Go to Configuration > Spider > Limits to access the crawl limit controls.
- Raise or uncheck the Max URL limit to allow the spider to crawl beyond the default cap.
- Go to Advanced within the Spider configuration to access additional crawl options.
- Click the relevant boxes to configure Advanced settings, then click OK to apply all changes.
