SEM Development

I’ve recently realized what a pain it is to write regression tests for modules that access the Google AdWords and Yahoo DTCXML API directly.  I already hate writing tests so this is pain^2.  Here’s my setup:

The green box is Google’s AdWords API.  The blue box is my API object that uses SOAP::Lite to access the API.  The red box is supposed to represent a Criterion object.  I’ve talked about how I create an object representation of the API services before.  Of course there are more objects but they’re not important for this discussion.

I can take two approaches when writing tests for the API object:

1. Ask for login info and do live tests on the API.
2. Write canned SOAP envelopes and insert them into my module.

I originally took the first approach which works OK but dies if the service is down, the person enters the wrong login info, an on and on.  Now, I know I could write tests that make sure the service is up, or retry a call if a 500 is returned but these are just tests.  When I have to introduce exception handling into my tests scripts they’ve ceased to be tests.  Also, it uses up Google’s quota units and gets stuck on Yahoo’s rate limits without careful planning.

I got the idea for the second approach by looking over Mike Schilli’s CPAN contributions.  Specifically, I looked at Net::Amazon which is a perl interface for the Amazon shopping API.  He’s written canned files that hold various responses that should be expected from the Amazon service.  He then tests against these files rather than the actual service.

I think I’m going to try the second way out for the bulk of my tests and then throw a few live tests in for a sanity check.  I guess I could also keep local versions of the WSDLs and then do diffs against them to make sure they haven’t changed.  The problem is that my API modules are written to change with the WSDL (or schema for Yahoo! Search Marketing).  Taking that in account, I’m not going to keep local versions of the schema or WSDL … for testing anyway.  It’s a good idea to keep local versions to save loading them over the internet every time an object is instantiated.

I’ll talk about how I test the object layer modules in a later post.

The AdWords announcement says the change will happen on July 1st, 2006 and will include a $0.25/1000 units charge.  There’s been plenty of speculation about whether this was going to happen but now we finally have an answer.

This was certainly the best thing for Google.  They gave four reasons for the change:

• Flexibility and scalability
• Commercialization
• Standardization
• Efficiency incentive

I have to say flexibility is the biggest win for me.  My quota has run out before and led to some major headaches.  I’ve run a few numbers and the quota charge will be around 0.5% to 1% of the total revenue for a campaign.  This is negligible but I’d still like to see it fall.  This could be accomplished two ways:

1. Work harder at reducing quota units used
2. Work harder on increasing revenue

I think everyone would be better served by me working on #2.

For the past two months I’ve seen that when I add new keywords to Google AdWords they will start with a min bid of $5. This, of course, disables all of the newly added keywords and, since it would be daft to increase the bid to $5, they stay disabled. This ridiculously high min bid usually drops after about two days.

I’m not the only one to have noticed this Google AdWords trickery. I’ve always assumed that it was a sneaky way to introduce an editorial review process. By jacking up prices, Google makes it nearly impossible to traffic these keywords therefore giving them time to review the content of the destination page or creative. I’ve seen some other reasonable explanations for this phenomenom as well. The AdWords API group has a thread discussing this problem as well but I disagree that the min bid drops after adding creative. It still stays at $5 for around two days.

Whatever the reason, I wish Google would be a bit clearer about this policy.

I’ve found a good way to allay my previous fear of forking. Using the POE library. The website says POE stands for “Perl Object Environment” but that doesn’t really explain much about it. It’s an easy way to timeslice the execution of your program. It’s threading in only one thread. The explanation of the library can get pretty complex so I’ll leave it to the people who wrote the documentation site. There’s a comprehensive cookbook that has examples of many things that are likely similar to some process you’re trying to write.

How does this all relate to SEM? It allows me to run multiple HTTP requests to the same engine at the same time. This works incredibly well for my saveAll and getAll functions in my Net objects. Now I can just spray a bunch of keywords at these functions, they’ll split the keywords into their appropriate ad groups and then update them in parallel instead of in series. What used to take me 12 hours, now only takes me 2.

I’m not sure about anyone else but I hate using the raw fork() command in Perl. It may just be a psychological thing but even in programs I know will run well with good sychronization, I alway harbor a deep fear that the parent will start spawning zombie processes like an undead rabbit. Up until now, I’ve had to deal with that fear in my campaign updating processes. I fork both Google and YSM updates and it saves a ton of time.

Today, however, I was directed toward LWP::Parallel. It allows you to run HTTP calls in parallel. There’s also an LWP::Parallel::UserAgent. This is perfect. I currently inherit the LWP::UserAgent module and overload the request methods for both Google and YSM. Then I just use these modules as I would LWP::UserAgent. Now I’m going to add some logic to take advantage of this parallelization. I’ll keep you apprised of the details.

Maybe I’m the only one that sees it this way but I think you should be able to omit the destinationUrl when updating a Keyword or Criteria record. Currently, it is required, and thus omitting it results in a <destinationUrl xsi:nil="true"> being sent. This means your current destinationUrl is erased in favor of the ad group’s default URL. I just think that’s annoying. If I wanted to erase the destinationUrl I’d send a blank or NULL field. If I just don’t want to waste bandwidth by including it in the call, I’d like to be able to omit it. Apparently, that isn’t allowed.

YSM and Google’s account management APIs look incredibly different but do, essentially, the same thing. That is, allow you to manage your SEM campaigns through a web service. Ask Jeeves has a sponsored listings product with an API for larger advertisers and MSN will soon be releasing it sponsored search product. So, how are you going to keep a similar programming interface across all of these products? I’ll tell you how I’m doing it.

Essentially, these web services boil down to a CRUD (Create, Retrieve, Update, Destroy) service for your search advertising campaigns. So, why not create a class that does all of these things for each part of a campaign?

Firstly, I want to express the way of handling CRUD that is the most intuitive to me. I use four methods in all of my classes. There are:

new() or new(ID):
Instantiates an empty object or tries to instantiate a populated object if an ID is supplied. Dies if the ID does not exist.

exists(ID):
Instantiates an object with the ID supplied or returns false.

remove():
Deletes the object.

save():
Creates the object if none exists yet or updates it if it already exists.

For Google, I’ve written 5 classes:

• Account
• Campaign
• AdGroup
• Creative
• Keyword

For YSM, there are only 3 classes:

• Account
• Category
• Listing

This allows for code that looks kind of like this (in Perl, by the way):

#Get the ad group
my @ags = Google::AdGroup->getAll;
my $ag = pop @ags;



#Make the keyword
my $kw = new Google::Keyword;
$kw->maxCpc(40000); #microns, remember
$kw->text('somethin');
$kw->destinationUrl('http://www.example.com');



#Save the keyword in the ad group
$ag->addKeyword($kw);


More intuitive, don’t you think?

Google’s TrafficEstimatorService has been a joke for a while now. In theory, it’s a great idea. Put in a keyword, get back the estimated impressions, clicks, and even an estimated average rank. The problem lies in the way Google ranks ads. Here’s an explanation right from the horse’s mouth:

Your keyword-targeted ad is ranked on search results and content pages based on its maximum cost-per-click (CPC) - or maximum cost-per-impression (CPM) for site-targeted ads - and Quality Score. Having relevant ad text, a high CPC (or for site-targeted ads, a high CPM), and a strong CTR will result in a higher position for your ad. Because this ranking system uses well-targeted, relevant ads to help determine your ad’s position, your ad can’t be locked out of the top position based solely on price.

Since your rank, and the number of times your ad is shown, is based on your ad text there’s a large area of doubt when Google tries to estimate the number of impressions you’re going to get with a completely new keyword. This and the lackluster display of accuracy so far, I’m assuming, is why they’ve changed it. Here’s the lowdown (or if you prefer to read the whole thread):

Gone:

• impressions - The estimated number of impressions for a given
  keyword
• ctr - The estimated click-through-rate for a given keyword.
• notShownPerDay - The estimated number of times that the ad would not be shown, despite a keyword match

Added:

• clicksPerDay - The estimated number of clicks generated
  per day for a keyword in a given ad group

I have to say, if this makes the system more accurate, I like this change. It keeps the main reason I use the estimation service intact, that is, to forecast the amount of money it’ll take to traffic a new keyword (or a few million). What it doesn’t do is try to estimate the relevancy of a keyword before it’s added to the campaign. That’s fine with me. I like to determine the relevance of a keyword by the amount of money it’s profiting, not by its CTR.

I’ll get back to you on whether the new system is actually more accurate …

Session management made it’s appearance for the first time in Overture’s AWS DTC-XML v1.2.1 specification. It allows you to generate a snapshot of your account and keep it on Overture’s servers. There are two ways to create a session:

CreateSessionGetListings
This command creates a session containing all of your listings. It’s lifetime is 72 hours and this operation can be performed 4 times in a 24 hour period.

CreateSessionGetListingsUpdated
With this command you create a session only of listings that have been updated within an allotted timeframe. As with CreateSessionGetListings, this session’s lifetime is 72 hours. Happily, this operation can be performed up to 10 times in a 24 hour period.

After the session is created you can retrieve the listings by invoking the GetListingsBySession command. This command can be invoked up to 1200 times in a 24 hour period so it’s unlikely you’ll top your quota. You can choose one, some or all of the following fields:

• searchTerm
• url
• online
• title
• description
• categoryId
• clickIndex
• minBid
• advancedOptIn
• advancedClickIndex
• contentBid
• contentOnline
• contentOptIn
• bid
• market
• lastUpdated

Last, and definitely least, is the GetSessionInfo command. It just returns the time your session was created along with how many records are in it.

These session methods have been a lifesaver for me. It’s better than GetListings because I can assemble a session of listings that have been updated and operate on those, rather than downloading every listing and then figuring out which ones have changed. It also allows double the quota of GetListings which is fantastic for large campaigns.

Now, if they would just up the quota on AddListings…

According to Google’s quota allowances addKeyword consumes 50 quota units while updateKeyword only consumes 10 units. On many underperforming keywords I’ve begun to lower the maxCpc value to 10,000 microns (.01 cents) instead of altering the status to “Deleted”. In most cases this will cause Google to deactivate the keyword. If the keyword’s minCpc is low enough that even a .01 cent maxCpc doesn’t deactivate it and it’s still losing money, it’s unlikely it will ever turn a profit. In this boundary case, deleting it may not be a bad idea.

At some future date, if research determines that these deactivated keywords will be profitable at a higher maxCpc you can activate them for 1/5th of the quota that you would expend if you had deleted them.

A quick note: Even when a keyword is marked “Inactive” it will be active in the contextual search. To avoid this scenario, turn off contextual search.

Another note: Using the aggregate functions updateKeywordList or addKeywordList will not save your quota. Both use the same number of quota units as their singular brethren.