Controlling your Task Queues

Summary

One of the key features in any framework for executing tasks (serializable and persistents) is how you can control them when things go wrong. This posting discusses some of the control panel of the GAE task queue. Details of the process is described in another article.

Previous Work

On my previous entry I wrote about the plans of creating automated tasks to process 130,000 transactions.Today I write about how things did not go as expected.

Executing the Tasks

Everything went out of control and very quickly. There were some important limits that I needed to respect in the Google Maps API. The most important wass the 120,000 transactions a day limit. With that I had no trouble. However, there must be a limit of transactions per minute that is nowhere published. With the limits in the GAE tasks I had no trouble.

However, I realized that after processing around 800 transactions, the google Map API was giving me errors. When I looked at the log, I realized I had fullfilled the quota for the day. I initially expected the quota to be 2000 transactions a day. But whenever I started executing the tasks, I set the limit to 5 transactions a second. I quickly reached a total of 300 transactions a minute, thus fulfilling my quota.

Since I promised to my wife that I would have all the data processed by Thrusday, I started fearing that I had to sleep in the couch. The previous day I was bragging about all the computer power that I had available and today i had reached the maximum.

Controlling the Tasks

The console for the tasks in GAE is your best friend in this circumstances. Also make sure that during the different loads generated that you follow the best practices for loads.

The tasks api actually worked as expected but the Google Maps API was the first thing to break down. I had anticipated a processing of 200