Top posts
Latest articles
Search Engines Poor Api Support
(No Ratings Yet)
Loading ...I’ve been trying to build some tooling to support the company I’m currently working for. It involves getting information from the search engines about rankings and competition. Now I’m not one to usually go against the search engines terms of services, but they are really pushing it.
First lets give you some access to the API’s that search engines have, just in case you want to play with them a bit:
- Google Api,
Really basic but well written API. Is pretty easy to use and supports a lot of features. (1000 search limit) - MSN Search API,
Very good Api if you’re writing for one of the Microsoft languages, otherwise I wouldn’t waste any time on them. Don’t even know if they support anything other then .Net - Yahoo Search Api,
One of the most stable Api’s around. It’s got a ton of features, but is a hell to implement in anything. I mean its got good XML powered results. But this does mean you need to build wrappers for everything to translate it into something useful for your program. (5000 search limit)
So what do these API’s offer you. You can use them to perform search operations without having to visit the website of the search engine. This can be useful if you’re developing some online / offline search feature. Or if you want some search engine tooling, eg. checking rankings. Actually it’s the only way as all of the search engine forbid the usage of automated queries that aren’t using their API.
So why am I so against these API’s?
Well let me first state: I like most of the service they offer, but….
They are just so fucking in accurate! If I perform a search to see how many pages of a website are in the index all three of the API’s return either 0 or the wrong number. When performing the same search in the search engine I get the real results. Why can’t the search engine give the same results using the API as they give to the rest of the world?
There is even one worse thing. The Google Api doesn’t work half of the time, and the other half the results are crap. Yahoo is just as bad and loves to make our live as developer as difficult as it can be with in-accurate results and dozens of ‘Bad Request‘ errors.
I’m sorry to say that this is one of the reasons that I had to build a tool that made a direct request to the search engine to analyze the results that way. I know it’s against the terms of services, but there’s no way around it at the moment. Let’s just hope the search engines fix some of these problems.
Yahoo’s strange estimations
Sometimes when you are surfing the web you just encounter something so strange that it makes you laugh. Today was one of those days. I regularly use Yahoo to search on the web and I’m expecting it to behave normally, but today I noticed something strange.
When you perform a search for games you get millions of results. (901 mil. to be exact) Now I’ve some time but that’s way to many. So I tried narrowing it down by looking for web games. So I refined my search to narrow down the amount of results. Which is also the normal behaviour of a search engine.
Here comes the fun part! My second search, which is supposed to contain less results, came back with 904 million matches. I now that search engines round up the numbers to save time, but 3 million more results that’s just ridiculous. Can someone explain to me how it is that a more precise search returns a more generic set!
I’m seriously considering to let Yahoo for what it is. An in accurate and imprecise search engines with semi decent results. Oh and just to make sure I checked multiple queries and found out it isn’t a fluke. I also checked the same searches in other search engines and they did narrow down the amount of results.
SQL the basics
Strange title isn’t. Is there such a thing as basic SQL(Structured Query Language). Lets try and find out. I will be covering some basics, thus creating a simple table. Making some basic queries and basics on T-SQL (Transact SQL).
Why use databases
Best to begin with a little story I suppose. Why would you want to use a database? Well it’s pretty easy. Databases give you the capability to store massive amounts of data in a orderly manner. With easy and structured access. I can hear those lovers of files shouting that you can do the same with files. I’ll grant them this, files can store information in the same way. But when it comes to quick access or easy manipulation then nothing beats databases.
Now I won’t bore you with the technical ways databases work. Meaning the locking mechanisms, reverting failures, tracking order and logging actions. (I could if I really wanted to, I mean we’ve all been taught how this works) Instead I will start out with the basics in table creation and querying.
Creating a table
The first thing you have to understand is the nature for SQL. Back when databases started there were many different companies creating them (still are), so there is a need for a standard. Having said this SQL is anything but standard. Every database supports different things, which makes SQL and database management in general very complicated. In this post I’ll be focusing on the MySQL variant.
We are going to create a table that will hold the information on personnel working for a small company. Now this means we need people’s home address information as well as the role they play and who is their boss.
Lets assume we want to know the following:
- Personel info,
- Home address
- Telephone number
- Birthday
- Social security number
- Employee info,
- Employee number
- Employee
- Boss
Did you already notice me breaking them up in two separate groups, these groups will also be my two tables. Most times this grouping happens naturally. You’ve also got the fields for the table. So here are the two creation SQL queries.
CREATE TABLE t_Personel (
p_ID INT(4),
p_Name VARCHAR(50) NOT NULL,
P_Address VARCHAR(150) NOT NULL,
p_Birth DATE,
p_SSN INT(10),
CONSTRAINT PRIMARY KEY(p_ID)
);
CREATE TABLE t_Employee (
p_ID INT(4),
p_Employee INT(4),
p_Boss INT(4),
CONSTRAINT PRIMARY KEY(p_ID),
CONSTRAINT FOREIGN KEY(p_Employee) REFERENCES t_Personel(p_ID) ON DELETE CASCADE,
CONSTRAINT FOREIGN KEY(p_Boss) REFERENCES t_Employee(p_ID) ON DELETE SET NULL
);
Time to explain the queries a bit. Most of it is pretty basic. You create a table with a view fields and tell what datatype they should have. This also means you need to indicate the size of the field, this is the part between the brackets. Something new is the creating of the PRIMARY KEY, which tells the database this field should be unique and indexed. Another constraint we add is a foreign key. This tells the database that the information in the field also has to exist in the referenced tables field.
Querying the database
Now that you have the database all set up (pretend it is already filled with useful information) you’re ready to start asking it for information. Lets presume that you want information on the boss of ‘F. MacBrull’. Looking at the tables you’ll have to first get the employee information of the guy, which means looking him up in the t_Personel. So lets do this part already.
SELECT FROM t_Personel p_ID WHERE p_Name = "F. MacBrull";
Now you have to find his employee information using the fetched information like this:
SELECT b.p_Boss FROM t_Employee AS b WHERE p_Employee = (SELECT a.p_ID FROM t_Personel AS a WHERE p_Name = "F. MacBrull");
Found that one a bit tougher already. We’re just beginning! As promised you’ve just selected the number of the boss that belongs to the already found information on ‘F. MacBrull’. So lets try and get his name, after all this is what we are really after.
SELECT c.p_Name FROM t_Personel WHERE c.p_ID = (SELECT b.p_Boss FROM t_Employee AS b WHERE p_Employee = (SELECT a.p_ID FROM t_Personel AS a WHERE p_Name = "F. MacBrull"));
Finally, we’ve got his bosses name. Some of you might ask: “Was all this really necessary?” NO! But it wouldn’t have been as much fun. And every time you query a database it takes time, so why not do it all at once.
Shared hosting lacking PHP 5
As some might know I’m a very active website developer / programmer. And herein also lays a problem, well more a difficulty. I love the new PHP5, but nobody seems to be willing to upgrade. In fact my current host is only supporting PHP 4.4.2, which is the most up-to-date version of 4.
So why do I believe that it’s important for webhosts to support PHP 5. Simple it allows much more complicated websites. Currently I’m working on the Movie & Serie database website, which has gotten way to complicated and way to big. But that aside it’s completely build up PHP 5. Why did I choose PHP 5 over 4?
Hmm, let me think. Oh yeah its the superb object support it has got. As my roots lay in normal system development (if you can call it that) I love objects and working object oriented. It gives you more control, makes everything reusable and cuts server costs down in half.
Why does it cut the costs on the server (usage in memory and speed)?
Ok so lets list a few things that are far better in the newest PHP version, I just love lists
- Better object support,
This means I, as webdeveloper, can create far more flexible and reliable website. Not only in performance but also in server abuse. (or for the techies less use of system resources) - Sharing information by reference,
Can you imagine having to copy this post every time you gave it to the next person. I wouldn’t be interested in that. Yet this is what the older PHP version do. And this is something the new version stopped doing. Instead of copying the post it just gives you the location. (I know this is not exactly true, but this blog is not just for techies) - A lot of new functionality,
Well PHP 5 contains a lot more functions, for stream control (accessing files), database and arrays.
This lack of PHP 5 support is actually the reason for me not to host my little project on any webhost at this time. (Yes I’m going old school) Currently its still hosted on my own little webserver, which also acts as the testing server. But this is far from optimal as the search engines are jumping on the site and downloading it like crazy. I’ve only got DSL so my connection can’t handle a lot.
Why don’t the webhosts switch then?
I do understand why the hosting providers are somewhat reluctant to make the step into the new version. I’ve had backward compatibility isues on some of my older website’s. So you can expect others to have the same problems. But is it enough to not do it. Try and get a few people to rebuild their website because you are doing an upgrade. It’s almost impossible!
I guess that’s the question that the providers must answer. But my expectation is that most hosting providers will upgrade in the near future. They just can’t afford to wait too long.
So far for my personal rant on the lack of PHP 5 support.
Google trend tracking
Just a quickie today.
I found a great marketing and SEO tool yesterday, just haven’t had time to post about it before. Google just launched a new service called Google Trends. It’s the first tool I’ve heard of that allows you to view the keyword usage over a prolonged period of time.
Just enter something in the little box and be amazed by the details it gives you. What’s even cooler is the fact that you’re able to select a region to see the data from. So why am I so exited, it’s pretty easy. Think about the research I’d normally have to do to get this type of information. I’d love to adjust marketing campaigns based on trends that occur yearly.
I’ve just done some checking based on the history of the keyword chocolate (hmm chocolate) in Canada. This showed me that people are searching for chocolat just before the ending of the year and just after the first quarter of the new year. Why is this important. Well easy, now I can adjust my PPC (Pay per click) advertising to display more ads during that time.
Something that amazed me a bit is Google trying to match the certain increase in searches to publishing of articles on the Internet. This is good to know as it helps in linking campaigns.
Just go and check it out, definately worth to spend some time with.
