[
Skip Navigation]
≡
↩️
🗣️
-
🏠
Help
:
Wiki
:
Allowed to Crawl Sites
≡
Welcome
Signin
Allowed to Crawl Sites@Help
View
Source
History
Discussion
Help Group
Create/Find Pages
Group Feed
My Groups
📖
Locale: en-US
Page: Allowed to Crawl Sites
⚙
🗄️
Page Type:
Standard
Page and Feedback
Page Alias
Media List
Presentation
Url Shortener
Share Wall
Alias Page To:
Page Border:
Solid
Dashed
None
Table of Contents:
Title:
Author:
Meta Robots:
Meta Description:
Meta Properties (such as Open Graph)
One line per property in format: name|content
Header Page Name:
Footer Page Name:
'''Allowed to Crawl Sites''' is a list of urls (one-per-line) and domains that the crawler is allowed to crawl. Only pages that are on sub-sites of the urls listed here will be crawled. <br /> This textarea is only used in determining by can be crawled if '''Restrict Sites By Url''' is checked. <br /> A line like: <pre> http://www.somewhere.com/foo/ </pre> would allow the url <pre> http://www.somewhere.com/foo/goo.jpg </pre> to be crawled. <br /> A line like: <pre> domain:foo.com </pre> would allow the url <pre> http://a.b.c.foo.com/blah/ </pre> to be crawled. It is also possible to allow a site using a regular expression: <pre> regex:/foo\d+/ </pre> would allow any url containing the string "foo" followed by 1 or more digits. <br />
X
(c) Hobby
GOOTII.COM
We use cookies to implement this site's user functionality, social media features, and traffic analytics.
Privacy Policy Details
.
Allow Cookies
