ProFindrBot

Last updated: June 2026

ProFindrBot is the web crawler operated by ProFindr. It visits public web pages to help build and maintain ProFindr, a search index of local home-service professionals across the United States. We are currently rolling the crawler out, so site owners may begin to see it in server logs.

How to identify ProFindrBot

ProFindrBot identifies itself with a User-Agent containing the token ProFindrBot and a link back to this page:

Mozilla/5.0 (compatible; ProFindrBot/1.0; +https://profindr.com/bot)

Please match on the token ProFindrBot rather than the full string. The version number may change over time.

How ProFindrBot behaves

It is designed for home-service provider pages, not broad unbounded crawling.
It follows only same-site links in bounded runs and limits how many pages it requests per site.
It skips cart, checkout, account, team, press, and other administrative pages that are not useful provider-service pages.
It waits between requests to avoid placing meaningful load on servers.
It requests HTML pages, skips large responses, and times out slow requests.
It records the HTTP status returned, including 403 and 429 responses.

We are actively expanding ProFindrBot to read and honor robots.txt directives, including Crawl-delay. Until that rollout is complete, the most reliable controls are the blocking methods below.

How to block or limit ProFindrBot

You can block ProFindrBot at your web server, CDN, or firewall by matching the ProFindrBot User-Agent token and returning a 403. For example, with Apache:

SetEnvIfNoCase User-Agent "ProFindrBot" block_bot
Deny from env=block_bot

You can also email support@profindr.com with “ProFindrBot” in the subject to request that your domain be excluded, report a problem, or ask us to slow down.

Crawler infrastructure

ProFindrBot crawls from dedicated infrastructure so site owners can confirm a request genuinely came from us rather than trusting the User-Agent alone. Our current crawler host is crawl-1.profindr.com, and its current source IP is 34.9.128.88.

To verify a request, use forward-confirmed reverse DNS: run a reverse-DNS/PTR lookup on the requesting IP, confirm it resolves to a hostname ending in .profindr.com, then run a forward DNS lookup on that hostname and confirm it returns the same IP. As we scale, we may add crawler hosts and IPs, so the .profindr.com reverse-DNS check is the better long-term signal than any single IP.

Contact

Questions or concerns about ProFindrBot? Email support@profindr.com or visit our contact page.