I’ve been head-down here working on the Web crawler and haven’t had much occasion to sit down and write blog entries. It’s been a very busy but interesting and rewarding time. A high performance distributed Web crawler is a rather complex piece of work. I knew that it would be involved, but there’s a lot more to it than I originally thought. Managing hundreds of concurrent socket connections on a single machine and coordinating multiple machines gets complicated in a hurry.

It’s been a long time since I did any serious multithreaded programming, and I’m re-learning some lessons:

• When there are multiple threads, everything is subject to concurrency issues.
• Unless you can guarantee that a data structure cannot possibly be accessed by more than one thread of execution at a time, you will eventually experience a problem if you don’t protect it with some kind of synchronization lock.
• You’ll find that guarantee very difficult to make.
• It can take hours or days before your program crashes inexplicably.
• It can take just as long to find that one place where you forgot to protect a global data structure with a mutual exclusion lock.
• If you’re developing a multithreaded application on a single processor machine, you’ll invariably end up finding more bugs when you move to a dual processor machine where the computer really can do more than one thing at a time.
• Learn how to use trace logging and prepare to spend a lot of time examining the log files for problems.
• Race conditions and deadlocks are facts of life, difficult to predict, and painfully obvious in retrospect.

It’s really good to be programming again.