Thanks for the note. You might be right about the database. When I ‘dismissed’ databases, I did so in the context of performing a single query for each harvested link. I know from experience that such a thing does not perform well.

I hear you on the colors. I rather like the layout of this theme, but I definitely need to change the colors.

Odd about the “no comments” thing. When I looked at the entry I didn’t see the comments. Not until after I moderated them. But thanks for letting me know. I’m kinda new at this WordPress thing. . .

Roy

I think you dismissed using a database a bit too quickly. (I’m a database guy, of course I think that!) Suppose…..

URLs to be read are extracted from the db in batches, size to be determined by experimentation but at least hundreds and probably thousands of URLs at a time. As each web page is crawled the links found are written to a flat file. When the batch of URLs is done being processed the corresponding batch of linked URLs is closed and picked up by a background process that applies them to the database, inserting new rows. The insertion process can be written a variety of ways, but being a SQL guy I would import into a staging table using some bulk load utility, then use an INSERT… SELECT FROM STAGING WHERE NOT EXISTS () to apply the changes to the master table.

The database server would be a different machine from the crawler(s), spreading the workload. The crawler(s) would not have to pause for the sort/merge, as the database would be dealing with that in the background tasks. The process of coming up with a new batch of URLs for crawling and the process of updating with the results of a crawl need not have significant blocking.

Another thought. Harvested URLs fall into two categories - the same domain and a foreign domain. I suggest that the file of harvested URLs be two files, one each for local and foreign links. Local links would be applied to the datbase on a higher priority (if the priority is to pick a specific domain clean) or a lower priority (if the idea is to spread the crawl across the web as quickly as possible). In fact there could be multiple, tuneable rules for how the URLs are picked from the DB for processing, at least allowing for bias toward either the deep vs broad choice.

Not that any of the above is of any real use, but you got me thinking.

By the way, typing into a white box against a black background is a bit hard on the eyes.

Have fun!

Roy Harvey
Beacon Falls, CT