This is the third and last in my short series about using streams in C#. The other two entries are Part 1 and Part 2. In this final part, I want to talk a little about messaging protocols.

Many programmers get their introduction to non-file streams by trying to write a simple chat application: something that lets two computers connect and trade messages. It’s a good way to familiarize oneself with networking concepts and .NET objects like Socket, TcpClient, and TcpListener, and NetworkStream.

Very often, the programmer gets stumped because he expects the receiver to receive things exactly as the sender sent them.For example, he might write this sending code:

    TcpClient client = InitializeConnection();
    NetworkStream strm = client.GetStream();
    string msg = "Hello, world";
    byte[] msgBytes = Encoding.UTF8.GetBytes(msg);
    strm.Write(msgBytes, 0, msgBytes.Length);

And the receiver:

    TcpClient client = InitializeConnection();
    NetworkSTream strm = client.GetStream();
    byte[] buffer = new byte[1024]; // largest message is 1024 bytes
    int bytesRead = strm.Read(buffer, 0, buffer.Length);
    string msg = Encoding.UTF8.GetString(buffer);
    Console.WriteLine(msg);

I’m not going to worry here about how the TcpClient instances are initialized. The point here is how to use the NetworkStream. I’m also going to limit my discussion to NetworkStream because it’s simpler, it matches what I’ve been talking about, and you’ll likely end up using it if you use TcpClient. You really don’t need to delve into the mysterious world of sockets for most applications.

If you read Part 1 of this series, you’ll understand the first problem with the code abovecode: there’s no guarantee that the call to Read will return all of the bytes that the sender sent. It might only get the first five characters.

Part 1 also showed how to call Read in a loop until you’ve received all the data you expect, or until the end of the stream is reached. But that won’t work in this case. If I were to write the loop:

    int totalBytesRead = 0;
    while ((bytesRead = 
        strm.Read(buffer, totalBytesRead, buffer.Length-totalBytesRead)) != 0)
    {
        totalBytesRead += bytesRead;
    }
    string msg = Encoding.UTF8.GetString(buffer, 0, totalBytesRead);
    Console.WriteLine(msg);

That code will never terminate. Well, it will: when the connection is closed or when the sender has sent 1,024 bytes. The loop will read everything available and the block waiting for more. Remember, documentation for Stream.Read says:

The implementation will block until at least one byte of data can be read, in the event that no data is available. Read returns 0 only when there is no more data in the stream and no more is expected (such as a closed socket or end of file).

The documentation for NetworkStream is, I think, somewhat ambiguous on the issue:

This method reads data into the buffer parameter and returns the number of bytes successfully read. If no data is available for reading, the Read method returns 0. The Read operation reads as much data as is available, up to the number of bytes specified by the size parameter. If the remote host shuts down the connection, and all available data has been received, the Read method completes immediately and return zero bytes.

It should point out explicitly that the only time it will return 0 is when the underlying socket is closed and there are no more bytes available.

So we have what looks like a Catch-22: we can’t assume that a single read will give us what we need, and we can’t continue to read until there isn’t any more because we can’t tell where the end is. All we know is that we haven’t received any more yet.

How to resolve this situation?

What makes a text file different from other types of files? There’s no file system support for “text files.” It’s not like NTFS has a CreateTextFile function that you call to create a text file. As far as the file system is concerned, a text file is a database file is an image file is a music file, etc. They’re all just streams of bytes. A text file is just a stream of bytes to which we have added some order. Specifically, we’ve agreed that the file contains lines that are delimited by newline characters. The data describes its own format.

The simple solution to the problem above is to terminate each message with a newline. So the sender would send "Hello, world\n". The receiving end is a little more involved:

    int totalBytesRead = 0;
    while ((bytesRead = 
        strm.Read(buffer, totalBytesRead, buffer.Length-totalBytesRead)) != 0)
    {
        int start = totalBytesRead;
        totalBytesRead += bytesRead;
        int newlinePos = SearchBufferForNewline(buffer, start, totalBytesRead-start);
        if (newlinePos != -1)
        {
            // found a newline in the buffer.
            // get that message string and display it
            string msg = Encoding.UTF8.GetString(buffer, 0, newlinePos);
            Console.WriteLine(msg);
            // now copy remaining bytes to the front of the buffer,
            // and adjust totalBytesRead as appropriate
            totalBytesRead = totalBytesRead - newlinePos;
            Buffer.BlockCopy(
                buffer, newlinePos+1,
                buffer, 0,
                totalBytesRead);
         }
    }

Or you could go one better and wrap a StreamReader around the NetworkStream:

    var reader = new StreamReader(strm);
    string line;
    while ((line = reader.ReadLine()) != null)
    {
        Console.WriteLine(line);
    }

The point is that the stream itself isn’t going to tell you where the end of a message is.

Typically, there are three ways to know how much data you need to read:

1. Messages are a known fixed length in bytes, padded with spaces or zeros or some other character. For example, you might decide that all message packets between machines will be 128 bytes long. It’s wasteful, but it makes programming pretty easy.
2. Messages are delimited, as in the text example above. This works very well for text protocols. You can make it work with binary protocols, but it takes some special code because the delimiter might very well be part of the binary data. For example you might say that records are delimited by a byte with the value 0xFF. But what happens if 0xFF occurs as part of a record?
3. The first bytes of a message include a fixed-length header that says how many bytes follow. This is very effective. For example, say the first two bytes of the message header are a short integer that says how long the rest of the message is. Your receiver then has to read the first two bytes, convert that to a short integer, and then read that many following bytes. Using the loop, of course, to ensure that it doesn’t get a short packet.

There are variations on the above. For example the first byte of the message could be the message type, and you know that message type 1 is 27 bytes long, message type 2 is 18 bytes, etc. Again, the point is that the data, not the stream, knows how long it is. All the stream knows is when it’s reached its end. And that only because the sender closed the connection.

So that’s the short course on things that trip people up when they’re working with streams. It’s unfortunate that most tutorials leave a lot of that stuff out. Sometimes the tutorial contains the information but it’s not explored in detail. And very often programmers see the first part of a tutorial and figure, “I’ve got it.” We’re an impatient lot, sometimes. Whatever the case, I see lots of Stack Overflow questions describing problems that are caused by one or more of the misunderstandings I talked about in this series.

Happy coding.