Multithreading is A Good Thing. Many solutions can be structured so that you can bring multiple processor cores to bear on the problem, and you can get very close to linear increases in throughput. That is, your quad-core processor will complete a job four times as fast as a single-core processor. Or very close to . . . → Read More: More threads makes the program slower?
|
|||||
|
This is the fourth in a series of posts about writing a Web crawler. Read the Introduction for background and a table of contents. The previous entry is Politeness. In the discussion of Crawling Models, I distinguished between two types of crawlers: “offline” crawlers that download documents and have a separate process create queue segments . . . → Read More: Writing a Web Crawler: Queue Management, Part 1 I haven’t given up on the carving. I’ve spent the last few weekends cutting up wood and making cutouts for those little birds. I enjoyed making those so much that I decided to send them to family and friends for Christmas. Last week I sent out a dozen, and I have more in the works. . . . → Read More: Short carving notes This is the third in a series of posts about writing a Web crawler. Read the Introduction for background and a table of contents. The previous entry is Crawling models. When you’re writing a Web crawler, it’s important for you to understand that your crawler is using others’ resources. Whenever you download a file from somebody . . . → Read More: Writing a Web Crawler: Politeness The big news story today was the death of Kim Jong-il, leader of North Korea. Of all the commentary I heard, one idea struck me as rather funny in a “you’re more right than you realize” kind of way. I don’t recall who it was that NPR interviewed, but his comments were to the effect . . . → Read More: The death of Kim Jong-il This is the second post in a series of posts about writing a Web crawler. Read the Introduction to get the background information. Expectations I failed to set expectations in the Introduction, which might have misled some readers to believe that I will be presenting a fully-coded, working Web crawler. That is not the case. . . . → Read More: Writing a Web Crawler: Crawling Models In C# versus the data structure, I described the skip list data structure and the difficulty I saw with implementing it in .NET. I published a C# implementation a couple of months ago, and it did indeed suffer from excessive memory usage. It performed quite well, but the memory requirement of 100 bytes per node . . . → Read More: Skip list revisited This is the first of a series of posts about writing a custom Web crawler. It assumes some knowledge of what a Web crawler is, and perhaps what crawlers are typically used for. I don’t know how many posts this subject will require, but it could be a rather long series. It turns out that . . . → Read More: Writing a Web Crawler: Introduction I’ve put together a desktop computer to use for development at home, and I need to get a new monitor for it. The old ViewSonic 15″ LCD that we paid $1,200 for 10 years ago is still in good shape, but its maximum resolution of 1280 x 1024 is not big enough to do serious . . . → Read More: Shopping for a monitor Sam and Ralph were among my favorite Looney Tunes characters. I thought of them this morning when I said hello to my co-worker as I came into the office. So I thought I’d look them up on YouTube. Google is great. I searched for [wolf sheepdog looney tunes], and got the Wikipedia article. All told, . . . → Read More: Sam Sheepdog and Ralph E. Wolf The latest issue (issue #57, Holiday 2011) of Woodcarving Illustrated Magazine has a pattern for a stylized bird that makes a great little tree ornament. I thought it would also be a good way to learn how to use my new Foredom power carver. So I spent some time with the bandsaw, cutting blanks from different . . . → Read More: Power carved tree ornaments Last time, I discussed existing text file viewers and why I find them inadequate. It’s disappointing that GNU less is the closest thing I’ve found to a good text file viewer for Windows. Functionally, less is almost perfect. For sure it blows away all of the other file viewers I’ve seen. Those other tools have . . . → Read More: Designing a better text file viewer I’ve evaluated a lot of text file viewers for Windows, looking for one that has a decent user interface, a small set of “must have” features, and that can handle very large files. One of the problems I have when looking for “large text file” viewers is that people have different definitions of “large.” Some . . . → Read More: Large text file viewers Stack Overflow is the best programmers’ resource to hit the Internet in quite some time. Online help forums for programmers are nothing new, but this one works better than anything else I’ve seen. I’m continually impressed by the variety and quality of the content there. I’m also amazed by some of the questions. For example, . . . → Read More: Dear Programmers: Google is your friend Paco is from a pattern created by a wood carver named Javo Sinta. (Alternate link, which might work better: Javo Sinta Woodcarving.) This carving is five inches tall, including the base. My version is carved from mesquite. I cut this out on the band saw a few months ago and started carving it with a . . . → Read More: Paco I finally finished the ornaments for the annual carving swap. This year I elected to participate in the smaller group of 12, rather than in the larger group (20 to 25). Here’s the whole batch: Clicking on the picture will give you a much better (larger) view. I think they’re all cute, but this one’s . . . → Read More: Christmas ornaments I’m setting up a computer that I can use to do some work from home. I’m still running Windows Server 2008 at the office because I’m slow to embrace change on my main development machine. But this new machine is going home and I want to explore what’s new in Windows. I figured I’d skip . . . → Read More: Installing Windows 8 Developer Preview A lot of interviewers will have a prospective developer do a code writing exercise in which the candidate is given a series of increasingly difficult small problems to solve in code. Often, this exercise is done on a white board, although with projectors and desktop sharing widely available now, many companies are moving to having . . . → Read More: The white board inquisition A recent question on Stack Overflow reads: Working with legacy code, I’ve found a lot of statements (more than 500) like this: bool isAEqualsB = (a == b) ? true : false; Does it make any sense to rewrite it like this? bool isAEqualsB = (a == b) Or will it be optimized? When you see . . . → Read More: First rule of optimization: Don’t A fairly common problem in programming is selecting the “best” items from a large group. For example, I have an application that makes video recommendations based on user input. The program takes the user’s input (typically the name of a song, an artist or a band), matches that up with our index of videos, and . . . → Read More: When theory meets practice |
|||||
|
Copyright © 2012 Jim's Random Notes - All Rights Reserved |
|||||
Comments are closed 