How to create a simple robots.txt file for your website

46 CommentsOn: How to create a simple robots.txt file for your website

How to create a simple robots.txt file for your website

robots.txt files tell search engines what they are and aren’t allowed to crawl and index on your Website, by using the robots.txt file properly we can control the way our websites are searched and indexed.

All major search engines check to see if a robots.txt file is present to adhere to any special rules you may have for them, spam bots and the like may ignore this file.

Benefits of using a robots.txt file

  • Can prevent duplicate content from being indexed
  • Saves bandwidth and keeps logs clean
  • Prevents server from continually having to send 404 file not found errors to search engines when they request the robots.txt file each visit
  • Prevent system folders and irrelevant files from showing up in Google etc
  • Can prevent confidential documents or images from being indexed
  • Increases the importance of pages that are allowed to be indexed
  • Can prevent users finding restricted pages of your website via search engines

How to create and upload a robots.txt file

  • Create a blank notepad/word document called robots and save as plain text (*.txt) (Most word processing programs should have the same functionality)
  • Write or copy in the code you want, and save.
  • Upload the robots.txt file to the root directory of your Website via FTP
    (Usually this is the public_html or www folder) and you’re done.

Allowing Robots Access to Everything

User-agent: *

This instructs search engines and robots that they can crawl and index all content.

Denying Robots Access to Everything

User-agent: *
Disallow: /

This instructs search engines and robots that they may not crawl or index any content.

Basic Template for a simple robots file

User-agent: *

Disallow: /cgi-bin/
Disallow: /*.js$
Disallow: /*.css$

# Google Image
User-agent: Googlebot-Image
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Allow: /*

How Robot file rules work

User-agent: *

User-agent is basically a synonym for "Search engines and robots called:"
and, * means anything, so in this case, we are targeting all User-agents.

Disallow: /cgi-bin/
Disallow: /*.js$
Disallow: /*.css$

By default, everything will be accessible to the robots that crawl your website, so we generally only need to specify where they don’t need to go. In most cases you don’t want CSS, JavaScript or system files indexed and showing up in Google.

The Disallow parameter is used to indicate locations that may not be accessed.

A Disallow with /on both ends/ means that folder and all its subfolders and url segments with that in it, may not be indexed.

Disallow /*.css$ means that any file of that type/extension may not be indexed, this could be used for any file type, for example you could prevent confidential pdf’s (.pdf) from being accessed by Google or images (.jpeg, .gif) etc.

# Google Image
User-agent: Googlebot-Image
Allow: /*

Using # at the start of a line is called commenting, this helps us humans make notes to remember what the hell we did, and tells robots to ignore this line.

Here we are targeting the Googlebot-Image User-agent specifically, to let it know that it may access and index all images with Allow: /* and saying that nothing is specifically forbidden by leaving Disallow blank, note that not all robots support Allow.

The reason we specifically target the image bot and adSense bot after we target all User-agents is to make sure that they are not being prevented by a previous rule, for example you could prevent everything in the /news/ section of your website from being indexed, text, files, images – but then you want images to still be indexed, and for adSense to still work.

Referencing a sitemap in your simple robots file


If you have a sitemap you can inform the search engines where it is here, it’s generally a good idea to keep your sitemap in the root directory of your website. [Create a sitemap]

Denying access to search result pages

Disallow: /*?*
Disallow: /*?

By preventing urls that contain question marks from being indexed we can stop search results from being indexed, and let the robots focus on your content/products.

Content Management Systems And Robots

If your running a recent version of WordPress, you can use these fantastic plugins to generate your robots and sitemap files and automate seo to an extent.

Other content management systems may also have plugins or addons that can help generate or maintain robots files, however because of the way each CMS is structured it’s not possible to make a robots file that can cater for everything.

Common Mistakes & Things To Remember

  • The robots file is case sensitive, and the file name is all lowercase: robots.txt
  • The file name is all lowercase: robots.txt
  • Keep to one rule per line
  • Old rules may be overwritten by newer rules declared after them
  • Anyone may view your robots.txt file, so be careful of what you include

Robots.txt Downloads

Blank robots.txt templateExample robots.txt file

Are there any other great robot.txt rules that you use on your websites?

Share or Save!

July 13, 2011 at 5:35 pm #1

i like your article very much.thanks for you share with us

August 1, 2011 at 1:54 am #2

It’s serious young girls lolita fantasy %-DD

March 29, 2012 at 11:25 am #3

Thankyou i realy impress that. Very nice :-)Bloger widget

Mayank Patel
September 15, 2012 at 9:17 am #4

Thanks for such a good information..!!!!!!1

Indian restaurant
March 22, 2013 at 8:09 am #5

Looking for this simple “robots.txt” file search for website. I was wondering for the “TXT” file creating, now i solve this file creation.. Thanks

June 26, 2013 at 8:25 am #6

Hi, I have created robots.txt file according to your suggestion, but I have a doubt, how i can place in the blog root directory. Kindly help me to place robots.txt file in root directory. I created blog in blogger site.


Resep masakan
October 2, 2013 at 3:31 am #7

I still haven’t figured out what should do with robot.txt because google said that any changes with robot.txt will not be saved. So, how to use this robot.txt if the hosting and platform are using google. Thanks

Projektowanie Ulotek
February 28, 2014 at 7:12 am #8

Hi there, all the time i used to check weblog posts here in the early hours in the daylight, as i love to
gain knowledge of more and more.

steroid abuse mental health
May 4, 2014 at 11:58 am #9

Attractive section of content. I just stumbled upon your site and
in accession capital to assert that I acquire actually enjoyed account your blog posts.
Any way I will be subscribing to your augment and even I achievement you access consistently quickly.

July 12, 2014 at 8:28 pm #10

It’s hard to find your blog in google. I found it on 17 spot, you should build
quality backlinks , it will help you to rank to google top 10.

I know how to help you, just search in google – k2 seo tips and tricks

temizlik şirketleri ankara
October 13, 2014 at 3:38 am #11

What’s up it’s me, I am also visiting this web page daily,
this web page is genuinely nice and the visitors are
genuinely sharing good thoughts.

How to create robots.txt file - Designers Mantra
November 2, 2014 at 9:28 pm #12

[...] Source Posted in: SEO ⋅ Tagged: google robots, robots.txt, robotstxt [...]

December 22, 2014 at 8:54 pm #13

Hi! I know thus is kinda off topic but I was wondering which blog platfrorm are you using for this website?
I’m getting sick and tired of WordPress because I’ve had issues with hackers and I’m looking
at alternatives for another platform. I would be great if you could point me in the direction of a good platform.

air huarache blanche
June 27, 2015 at 6:09 am #14

Would you mind if I quote a few of your blogposts as long as I provide credit and sources returning to your weblog: %BLOGURL%. I most certainly will aslo make certain to give you the proper anchor-text link using your webpage title: %BLOGTITLE%. Be sure to let me know if this is ok with you. Thankyou

July 3, 2015 at 5:09 am #15

Enable round hit several breaths into it to really make the end of the tube and
to cool somewhat.

my blog: hello kitty bong for sale ebay (Kandi)

binary options indicators
July 9, 2015 at 11:21 am #16

Thanks for every other informative website.
The place else may I am getting that type of information written in such a perfect method?
I have a project that I am simply now operating on, and I’ve
been on the look out for such info.

seo 69200 venissieux
March 14, 2016 at 10:38 am #17

This is realⅼy attention-grabbing, Уоu аrе
an overly skilled blogger. І Һave joined yоur rss
feed аnd lооk forward tߋ looking fоr extra օf үοur wonderful post.
Аlso, Ι’vе shared үour web site іn my social networks login
April 26, 2017 at 9:41 am #18

I pay a quick visit every day a few web pages and sites to
read articles, except this blog offers quality based writing.

тонометр механический microlife bp ag1-30 инструкция
November 2, 2017 at 11:37 pm #19

Thanks , I’ve recently been looking for information approximately this topic for ages and yours is the best I have discovered so
far. But, what concerning the bottom line?
Are you certain concerning the supply?

debbi edwards consulting
November 4, 2017 at 2:46 pm #20

I have read so many articles on the topic of the blogger lovers however this post is really a
good article, keep it up.

фенистил пенцивир цена инструкция
November 7, 2017 at 9:24 am #21

Very good article! We are linking to this particularly great post on our website.
Keep up the great writing.

доп.соглашение к трудовому договору о предоставлении дмс
November 9, 2017 at 12:59 am #22

I blog frequently and I seriously thank you for your information. This article has truly peaked my interest.
I’m going to bookmark your blog and keep checking for new details about once
a week. I subscribed to your RSS feed too.

барокамера инструкция по эксплуатации
November 9, 2017 at 6:16 am #23

Unquestionably consider that which you stated. Your favourite reason appeared to be at the net the simplest thing to
have in mind of. I say to you, I certainly get annoyed whilst folks consider worries that they plainly do not know about.
You controlled to hit the nail upon the top and defined out the
entire thing with no need side effect , folks can take
a signal. Will likely be back to get more. Thanks

типовая производственная инструкция машиниста компрессорных установок
November 13, 2017 at 7:24 pm #24

Thank you for some other wonderful post. The place else may just anybody
get that type of information in such an ideal method of writing?
I’ve a presentation next week, and I’m on the search
for such info.

seks fotoshop na android
November 19, 2017 at 6:00 pm #25

It’s actually a nice and useful piece of info. I am happy that you just shared this helpful info with us.
Please stay us up to date like this. Thanks for sharing.

бланк командировочного образец заполнения
November 22, 2017 at 8:41 am #26

Can I simply say what a relief to discover somebody who genuinely understands what
they’re discussing online. You definitely know how to bring an issue to light
and make it important. More and more people must look at this and understand this side of your story.
I was surprised that you’re not more popular because you definitely have the gift.

каротиноли м инструкция по применению
November 22, 2017 at 9:06 am #27

Thanks for sharing your thoughts about simple robots.txt.

должностная инструкция директор по качеству
November 22, 2017 at 6:23 pm #28

That is a good tip especially to those fresh to the blogosphere.
Short but very accurate information… Thank you for sharing this one.
A must read article!

Ms office 2013 для дома и учебы торрент
November 22, 2017 at 11:06 pm #29

Great site you have got here.. It’s difficult to find excellent writing like yours nowadays.
I honestly appreciate people like you! Take care!!

доверенность на управление автомобиль бланк 2016
November 23, 2017 at 10:05 am #30

Hi, i feel that i noticed you visited my weblog so i got here
to return the favor?.I am attempting to to find issues to
enhance my web site!I guess its adequate to use a few of
your concepts!!

скачать altercam
November 24, 2017 at 3:52 am #31

Hello, the whole thing is going nicely here and
ofcourse every one is sharing facts, that’s in fact excellent, keep up writing.

работа доставка документов в москве
November 24, 2017 at 6:26 pm #32

It’s genuinely very complicated in this active life to listen news on Television,
so I only use internet for that reason, and get the newest news.

срок выполнения обязательств поставщиком по договору поставки
November 26, 2017 at 6:15 am #33

This article will assist the internet viewers for building up new website or
even a weblog from start to end.

документы продажа доли в уставном капитале
November 26, 2017 at 7:31 am #34

Great web site you have got here.. It’s hard to find high quality writing
like yours these days. I seriously appreciate individuals like you!
Take care!!

Aman Akhar
July 28, 2018 at 9:47 am #35

hi, bro plz provide me best robot.txt file because some of my page blacked by google, so help me

Planet7 No Deposit
December 5, 2018 at 2:42 am #36

Hello there, just became aware of your blog through Google, and found that it’s really
informative. I’m going to watch out for brussels.
I will be grateful if you continue this in future.

Lots of people will be benefited from yyour writing.

December 5, 2018 at 7:36 pm #37

It’s not my first time to visit this site, i am browsing this website
dailly and get nice information from here everyday.

December 11, 2018 at 3:16 am #38

Hello, I enjoy reading all oof your post. I like to write a
little commen to support you.

December 20, 2018 at 1:38 pm #39

My brother recommended I may like this web site.

He used to be totally right. This submit truly made
my day. You can not consider simply how so much time I had spent
for this info! Thank you!

fifa 18 ultimate team hack
December 25, 2018 at 9:30 pm #40

Thank you for another informative website. Where else
may I get that kind of info written in such a perfect approach?
I have a project that I am just now operating on, and I have been on the glance out for such

mẫu áo đồng phục công ty
December 29, 2018 at 5:50 am #41

Bạn có thể yêu cầu mẫu áo đồng phục công ty could thực
tế (nếu muốn).

Bovada Casino Signup Bonus
December 29, 2018 at 7:54 pm #42

If some one desires expert view concerning blogging after thnat i propose him/her tto paay
a quick visit ths weblog, Keep up the pleasant work.

casino mobile
January 7, 2019 at 5:43 am #43

I am noo onger certain where you’re getting your info, but good topic.

I must spend a while finding out much more or working out more.
Thanks for magnificebt information I was looking for
this information forr my mission.

January 15, 2019 at 7:55 am #44

It’s truly a nice and helpful piece of info. I am glad that you shared this helpful info with us.

Please stay us up to date like this. Thank you for sharing.

January 16, 2019 at 2:53 pm #45

You’ll find available all sorts of helpful information on these community forums and be confident that most complaints is likely to be released.

January 26, 2019 at 2:19 pm #46

I do not know whether it’s just me or if everyone else encountering issues with your site.
It appears like some of the written text on your posts are running off the screen.
Can somebody else please comment and let me know
if this is happening to them as well? This could be a problem with my browser because I’ve had this happen before.

Appreciate it

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>