Question:

What if my site has some hash fragment URLs that should not be crawled by Google?

by Guest1948  |  12 years, 7 month(s) ago

0 LIKES UnLike

I do not want some of my content on AJAX Url to be crawled by Google. Need help.

 Tags: crawled, fragment, Google, Hash, site, URLs

   Report
Answer The Question I've Same Question Too

1 ANSWERS

Sort By: Date  |  Rating
  1. Guest9124
    When your site adopts the AJAX crawling scheme, the Google crawler will crawl every hash fragment URL it encounters. If you have hash fragment URLs that should not be crawled,  you should add a regular expression directive to your robots.txt file.

Sign In or Sign Up now to answser this question!
You're reading: What if my site has some hash fragment URLs that should not be crawled by Google?

Question Stats

Latest activity: 13 years, 1 month(s) ago.
This question has 1 answers.

BECOME A GUIDE

Share your knowledge and help people by answering questions.
Register Now