Bug Report #4421

Add default robots.txt

Added by Kemal Delalic 4 months ago. Updated 4 months ago.

Status:Closed Start date:01/29/2012
Priority:Normal Due date:
Assignee:- % Done:

0%
Category:Core
Target version:v3.3.0
Resolution:invalid Points:1

Description

With default routing, /robots.txt will return 500+. This doesn't seem like a big deal until Google removes few of your sites from the index completely. Default robots.txt file should be added instead, or served through php (to respect base_path directory).

Default robots.txt looks like this:

User-agent: *
Allow: /

History

#1
Updated by Jeremy Bush 4 months ago

  • Status changed from New to Closed
  • Resolution set to invalid

This isn't true at all:

http://kohanaframework.org/robots.txt
http://www.google.com/search?q=kohanaframework

The "default routing" will return a 404 response, in any case.

#2
Updated by Kemal Delalic 4 months ago

Yeah, these apps are 3.0, this has been changed since 3.1

Also available in: Atom PDF