The findings

The real findings, finally. One line: “The crawler was told where not to go. I made sure of it.”

A crawler asks permission before it asks for pages. Whoever wrote the permissions for this site wrote them to keep one place out of Google. That place is where you need to be.