Building Your Own Search Engine

Posted On: 2007-08-16

Here's another brain doodle that's wandering around in the back of my brain at the moment. Actually it's been stuck back there for several weeks and first made an appearance when I suggested to Steve that we build our own search engine.

Back then my thoughts weren't very focused, I think I was verging on the delusional because I did have visions of developing something like Google. Fortunately Steve didn't quite laugh at me but he did give me a number of very valid reasons why what I was suggesting was just plain impractical.

At the same time he did make a few suggestions that got me thinking in some slightly straighter lines and the idea of building our own search engine has become a little more refined. In fact we've now actually built a very small search engine for something that's very relevant to the town we live in.

Now before you decide that even though we've built a very small search engine my thinking is still very delusional stop and consider this for a moment. There are a number of small developers out there who are looking seriously at building their own search engines. These guys are not trying to take on Google at their own game ... in fact they see that in the future Google may be it's own worst enemy. Instead they are looking at building what we might consider to be niche search engines.

These search engines aren't planning on indexing the whole world but are planning on indexing sites that are relevant to certain topics. For example, if you were looking for some words in a song you wouldn't go to Google, you'd go to a search engine that specialized in providing search engine results pages for song titles and lyrics. If you were looking for engineering information you would go to a search engine that specialized in engineering terms.

You would go to these search engines rather than monolithic Google because they provided far more relevant and spam free results than Google could provide. And the better the results they provided the more people would use them.

Ok, so that's the theory but how does that affect us here in adult? Well think about it for a moment, what would it take to provide more relevant search results for niche terms than Google provides? Think about how you could monetize something like that and whether or not you could actually generate a return on your investment.

Think outside of adult online too. Think about your own local area and what people look for in your local area. Even think outside the local area to big centers nearby. What do people search for in those areas and could you provide more relevant search results with a very localized search engine?

Steve and I certainly think that it's quite possible to do something like that and it's not very hard to get started. You might be thinking that it's not something you could ever do because you don't have the ability to write the software that would produce your own spider. But you don't have to ... it's already been done for you in a fairly basic way.

Have a look around on the Net and you'll find that there are very simple and easy to use software out there that will at least get you started. Steve and I found one that's ideal for the small projects that we have in mind. It not only searches your own sites but will also search any URL that you care to point it at. Once it's searched those sites it then compiles the information that it's found and there is your search engine ready to put up on the Web.

Now that's a very basic piece of software and it certainly does have it's limitations but at least it's a good starting point. From there the sky really is the limit and you really can expand from that small beginning.

Now don't forget what I said right at the beginning; even though we've been playing around with a local search engine this really is just a brain doodle at the moment. There are lots of negative aspects to trying to build a niche search engine and even if you do build it you then have to convince people to use it.

Obviously there are lots of challenges to be overcome if you really want to make something like this work but it's the guys who can improvise, adapt and overcome who are out in the lead making the money. If that's where you want to be then you have to start a few brain doodles of your own and see where they take you.