Robots.txt is a small file that gives web crawlers instructions about which areas of a website they may visit. It is like a sign at the entrance asking visitors to avoid certain rooms.
A site might ask a search crawler not to visit a private-looking test area. The file does not hide that area from people or protect it with a password.
Do not put secret information behind a robots.txt rule. Anyone can read the file, and crawlers may not follow every instruction. Use proper access controls for private material.