Robots and Visitors

Posted On: 2008-12-15

Well there's another weekend come and gone and if you wanted to spend it working then I hope it was a productive time for you. We achieved a lot over the weekend but hardly any of it had much to do with the Internet. Instead we've been busy on the first stage of getting the house in order for Christmas and we took some time off to go walking on the beach too.

The beach part was very important and no matter what the weather is like where you are, or how much work you want to get done in a short period of time, you should always take some time just to get out and enjoy some life in the real world.

Who looks at our sites?
For any Internet marketer ... whether they be in mainstream or adult ... the question of just who looks at your sites is very important. When you know who your audience is you can do so much more to target your sales pitch so that you can increase your chances of making sales.

All too often we presume to know who our target market is without being able to base our presumptions on fact. Of course free site builders and gallery builders really have little chance to learn who just who is looking at their sites because we get so little information about our visitors.

The billing companies and even the sponsors would have a much clearer idea of who is buying what we're selling because they see information about location, time of purchase etc. and that can go a long way to telling them a lot about the person who has just joined. Sadly though they don't release that information to us so we're basically flying blind and relying on what others may tell us without having any way of knowing if that information is genuine or not.

That's why a recent survey of guys here in Australia was interesting. Frequently we think of our audience as either being middle-aged guys who are furtively trying to view porn at work ... or college students who are looking for some entertainment after hours but, at least in Australia, that may not be type of person who views porn online at all.

The survey discovered that your average consumer of porn in Australia had a university education, was environmentally aware and perhaps even very keen to save the planet. They were also likely to vote for political parties that are not so mainstream; here in Australia that means that they probably voted for the Greens ... an environmental party that is really making it's presence felt on the political scene.

The survey also found something else that was very interesting. Of those in the survey who admitted to being Christians or attending church regularly 58 percent admitted that they liked to look at porn on the Net. Why does the word 'hypocrite' suddenly spring to mind?

Oh yes ... these are those wonderful people who like to look at what we provide but are always voting to have our sites banned from the Web.

Robots Exclusion Protocol
You know, we often talk of search engine spiders but the correct term for those small programs that scour our web pages is robots and search engines aren't the only places online that send out robots to go over the pages on the web. There are hundreds, if not thousands, of robots scouring the web every day of the week and some should be more than welcome on our sites while others are just ripping off our content.

So we use robots text files to try and control those robots. A robot like Googlebot is one that most of us would welcome but even then we might like to keep it out of some parts of our site so that not every page is indexed so we can exclude it using a robots text file.

Writing a robots text file is something that might seem quite simple until you actually sit down and start the process. If you've never tried it before ... or never had any success in writing a robots text file that actually works ... then an extensive article entitled 'Managing Robot's Access to Your Website' by Vanessa Fox over on janeandrobot.com will certainly give you a much better understand of what your robots text file needs to contain.

If you print it out you will find that it runs to about 15 pages but don't let that put you off. It's going to be an invaluable resource in helping you understand how to make a robots text file work for you.

And now I'm off to explore the rest of my day. On Thursday I'll be back to talk about some interesting moves that Google may be about to implement ... sadly I don't think that it's anything we should be all that happy about.