Redstoner/redstoner.com
Archived
0
Fork 0
Code Issues Pull Requests 2 Wiki Activity

robots.txt

Browse Source
This commit is contained in:
jomo
2014-06-25 02:18:03 +02:00
parent 194cd0d658
commit bfe68ce161
1 changed files with 59 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
public/robots.txt
+59 -1
View File
@@ -1 +1,59 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file #
# .-.
# ( )
# '-'
# J L
# | |
# J L
# | |
# J L
# .-'.___.'-.
# /___________\
# _.-""' `bmw._
# .' `.
# J `.
# F L
# J J
# J `
# | L
# | |
# | |
# | J
# | L
# | |
# | ,.___ ___....--._
# | ,' `""""""""' `-._
# | J _____________________`-.
# | F .-' `-88888-' `Y8888b.`.
# | | .' `P' `88888b \
# | | J # L # q8888b L
# | | | | )8888D )
# | J \ J d8888P P
# | L `. .b. ,88888P /
# | `. `-.___,o88888o.___,o88888P'.'
# | `-.__________________________..-'
# | |
# | .-----.........____________J
# | .' | | | |
# | J---|-----..|...___|_______|
# | | | | | |
# | Y---|-----..|...___|_______|
# | `. | | | |
# | `'-------:....__|______.J
# | |
# L___ |
# """----...______________....--'
#
#
User-agent: *
# Laws of Robotics
Disallow: /harming/humans
Disallow: /ignoring/human/orders
Disallow: /harm/to/self
# 10 seconds until next request
Crawl-delay: 10
# People don't like being stalked ;)
Disallow: /users