I’ve had many programmers tell me that they find the idea of multithreaded programming intriguing, but their problems just don’t lend themselves to being worked on by multiple threads. A little digging usually reveals that the person I’m talking to has exactly the wrong idea about what multithreading is all about.

Apparently, programmers who are unfamiliar with the use of multiple threads of execution believe that multithreading is about two things: massively parallel processing, and synchronization primitives like locks, semaphores, and interlocked operations. Whereas it’s true that parallel processing is possible and synchronization primitives are certainly useful, those things are irrelevant as far as most business applications are concerned. Correctly working with synchronization primitives is hard, especially in a complex program, and not many business application problems need the kind of massively parallel processing that seems to be all the rage in the literature. But business applications can make good use of multithreading.

I’ve developed applications for many types of businesses, and in every one of them there are applications that remind me of the COBOL reporting and transaction processing applications that I wrote in my first programming job. Those programs all take this basic form:

    open output file
    open input file
    while not end of input
        read input record
        process record
        write output
    end while
    close input file
    close output file

In many businesses, almost all of the applications take this form. It’s no wonder so many programmers think that multithreading has nothing to offer them. There is very little literature on how to use multiple threads for such programs. That’s unfortunate, because these input-process-output (IPO) programs lend themselves very well to multithreaded processing.

I find it useful to think of the traditional model above as a factory in which a worker grabs the raw materials from the delivery area, carries them to a workstation where he assembles them, and then carries the finished product to the shipping department before going back for the next batch of raw materials. The worker does everything himself; he’s a single thread of execution and he wastes a lot of potentially productive time walking back and forth between the delivery area, his workstation, and the shipping department.

Now imagine the same factory, but with the worker standing in one place. On his left is a conveyor belt that delivers raw materials. He pulls them off the belt, assembles the product, turns to his right and puts the finished product on a conveyor belt that carries it to the shipping department. Then he turns to the left and picks up the next batch of raw materials that’s already there waiting for him. The worker no longer wastes time carrying things around. In addition, he doesn’t have to wait for raw materials to be delivered, and finished products are whisked away as soon as he’s finished with them. The whole system is more productive because the individual parts can operate independently of each other.

It’s possible to make an IPO program work in a similar manner. One way is with asynchronous I/O operations. In the traditional model shown above, reading an input record consists of issuing a read request and waiting until the operating system can get the record. Sometimes that’s very quick because the data is already in memory and just has to be copied, but other times it can require many milliseconds to get data from the hard drive. Writing a record can be even more time consuming. To execute an asynchronous read, the program issues a request saying, in effect, “go get the next record, put it in this space, and I’ll come back for it later.” The program then goes on to do other things while the operating system is fulfilling the read request.

The IPO model modified for asynchronous I/O looks like this:

    open input file
    issue asynchronous request for record
    while not done
        wait for read to complete
        if end of file
            wait for pending write to complete
            break
        end if
        copy record to working area
        issue asynchronous request for next record
        process record
        wait for pending write to complete
        issue asynchronous write request
    end while

The first read, of course, is synchronous. The program can’t start processing until the first record is read. But subsequent reads take place while the program is processing the record. The same thing goes for the writes. They, too, happen while the program is processing records. Reading and writing are done concurrently, and both overlap with the processing time. The amount of time this saves can be substantial. The “wait for pending read” and “wait for pending write” are usually not waits at all: the operations completed while other processing was going on.

Although asynchronous I/O can give you a nice performance boost, it’s kind of difficult to use in .NET. Older versions of .NET supported the BeginRead / EndRead and BeginWrite / EndWrite asynchronous operations for binary streams. Those are effective, but hard to use. .NET 4.5 introduced ReadAsync and WriteAsync, which are much easier to use. Effective as they are, they do not provide a performance benefit when working with small records. If you have large records, or if your program is doing other processing that takes significant time, the asynchronous methods are well worth looking into.

Prior to .NET 4.5, there was no asynchronous support for reading and writing text files. .NET 4.5 introduced TextReader.ReadLineAsync and TextWriter.WriteLineAsync, but using them doesn’t provide any performance benefit when the processing doesn’t take much time. At least my tests didn’t show an improvement. I suspect that the task switching overhead swamps the actual input and output time and that if I did more processing of the line I’d see some improvement.

Fortunately, there’s another way to leverage multiple threads: a way that more closely simulates the more efficient factory that I described, and one that does provide significant performance improvement. How? By using three long-running tasks. One reads records from the input file and places them into a queue. One reads records from the input queue, processes them, and writes them to an output queue. The third reads the output queue and writes records to the output file. It’s the same concept as using asynchronous I/O, but there’s very little task switching overhead. Rather than start a pool thread for every line read and written, we start long-running threads that serve in-memory queues. It makes a huge difference.

Using queues, the program’s operation looks like this:

    Reader thread
        while not end of input
            read record
            add record to input queue
        end while
    end Reader thread

    Writer thread
        while not end of output
            get record from output queue
            write record to file
        end while
    end Writer thread

    Main thread
        initialize queues
        start Reader thread
        start Writer thread
        while not end of input queue
            get record from input queue
            process record
            put record onto output queue
        end while
        wait for Writer thread to complete
    end Main thread

Again, the idea here is to overlap the three sub tasks. While the reader is gathering records, the worker is processing and the writer is outputting records. Those threads can’t run at full speed all the time, of course, because the input and output queues have finite capacity. But the threads can service the queues fast enough that most of the time the worker doesn’t have to wait for input or output. The key is to have an effecient queue that supports concurrent readers and writers.

I went a little longer than I had planned here, so I’m going to stop for now. Next time we’ll look at some C# code, talk about the concurrent queue data structure, and run some tests to compare performance.