The EDNBot Web Crawler — ExpiredDomains.network

What is EDNBot and Why is it Visiting Your Site?

EDNBot is an automated cloud crawler operated by ExpiredDomains.network. Our registered subscribers use the platform to analyze public authority websites in their niche to locate dead, broken, or expired outbound hyperlinks.

When you see EDNBot in your web server access logs, it means a digital marketer, SEO professional, or webmaster selected your public website as a target to identify broken links. EDNBot reads HTML pages specifically looking for outbound <a href> hyperlinks, and checks whether the external destination domains have expired.

                Zero Harassment & No Content Scraping: EDNBot does NOT download video/audio media, submit forms, crawl private areas behind login pages, bypass paywalls, or republish your copyrighted editorial text. It exclusively checks public link status.
            

Full User-Agent String:

Mozilla/5.0 (compatible; EDNBot/1.0; +https://expireddomains.network/bot.html)

How to Control or Block EDNBot

EDNBot strictly obeys webmaster instructions defined in your site's robots.txt file located at the root of your domain (e.g., https://yourdomain.com/robots.txt).

You have full control over whether EDNBot can visit your site, which directories it can access, and how quickly it requests pages.

Option 1: Block EDNBot from Your Entire Website

To completely disallow EDNBot from crawling any page on your server, place these two lines in your robots.txt file:

# Block EDNBot completely
User-agent: EDNBot
Disallow: /

Option 2: Slow Down EDNBot (Set a Crawl Delay)

If you want EDNBot to crawl more slowly to reduce server load, specify a Crawl-delay in seconds:

# Enforce a 10-second pause between requests
User-agent: EDNBot
Crawl-delay: 10

Option 3: Disallow Specific Directories Only

You can prevent EDNBot from visiting sensitive or resource-intensive sections of your site:

User-agent: EDNBot
Disallow: /admin/
Disallow: /cart/
Disallow: /private/

Webmaster FAQ

Does EDNBot cause high server load or bandwidth consumption?

No. EDNBot enforces conservative concurrency limits and automated polite delays between successive requests to the same host. We do not download heavy images, videos, or script bundles — only clean lightweight HTML pages.

Does EDNBot publish my content or steal my articles?

Never. EDNBot is an outbound link checking engine, not a content archive or scraper. We do not store, index, or republish your article texts, photos, or proprietary media. We only parse the destination hostnames of hyperlinks.

How often does EDNBot refresh robots.txt?

EDNBot checks your domain's robots.txt file before beginning any crawl run. Caches are refreshed every 24 hours. If you add a Disallow rule, EDNBot will honor it as soon as the cache expires.

Can I request a permanent block without editing robots.txt?

Yes. If you manage a domain and prefer not to edit your robots.txt file, email us at [email protected] with proof of domain ownership, and our engineering team will add your domain to our global blacklist within 24 hours.

Webmaster Inquiries & Assistance

If you have concerns about crawling frequency, need assistance configuring your robots.txt, or wish to report an issue, our technical team is here to help.

Contact Engineering