C# Network Programming: Simple HTTP Client

Hi! :)

In yesterday's article, HTTP Requests in Wireshark, we used Wireshark to observe the messages sent and received by web browsers when downloading a webpage.

In today's article, we're going to do that ourselves, in code! :D More specifically, we will write a simple client that connects to a web server and downloads a webpage.

Although the HTTP requests sent by a web browser might seem a little complicated, HTTP Made Really Easy shows that it really takes a very short request to retrieve a web page. For example, the following simple request can download the homepage of Programmer's Ranch:


GET / HTTP/1.1
Host: www.programmersranch.com


Note that the above includes a double newline which is essential for the request to be interpreted by the server (refer to yesterday's article).

Start a new SharpDevelop project, and include the necessary libraries for I/O and network programming:

using System;
using System.IO;
using System.Net;
using System.Net.Sockets;

We first declare a String to contain the HTTP request, as follows:

            String request = @"GET / HTTP/1.1
Host: www.programmersranch.com
Connection: Close

";

This is a special kind of String. The @ before the starting quotes shows that it is a literal string. This means that newlines are included in the string, and we can use it to make multiline strings. I have also included a Connection: Close header field, so that the server will automatically close the connection once it has sent back all the data - this makes it easier for us to know when we have received everything. Finally, note the double-newline at the end of the request, which is important.

Now, this is all the code we need:

            using (TcpClient client = new TcpClient("programmersranch.com", 80))
            using (StreamWriter writer = new StreamWriter(client.GetStream()))
            using (StreamReader reader = new StreamReader(client.GetStream()))
            using (StreamWriter outputFile = File.CreateText("webpage.html"))
            {
                writer.Write(request);
                writer.Flush();
             
                String line = String.Empty;
                while ((line = reader.ReadLine()) != null)
                {
                    outputFile.WriteLine(line);
                }
             
                Console.WriteLine("Webpage has been written to webpage.html");
            }
         
            Console.Write("Press any key to continue . . . ");
            Console.ReadKey(true);

Here we're using TcpClient in order to connect to the website we want, and we are using port 80 since this is HTTP. We also declare a StreamWriter and StreamReader using the TcpClient's stream, so we can easily send data to and receive data from the server. Finally, we open a file called webpage.html to which we will write the received data. Since webpages tend to be quite long nowadays, this is better than writing it to the console window.

Note how the multiple using statements allow us to open and work with several resources, and they are automatically closed at the end.

In the body of the using statements, the first thing we do is send out the HTTP request, and remember to flush (remember the words of wisdom from an earlier article: streams and toilets must always be flushed) the stream to ensure that the request is actually sent.

Then, we receive the response from the server, line by line, and we write that line to the output file (webpage.html). When there is no more data to receive, reader.ReadLine() returns null, and the loop ends.

When you run this program...

...you will find the new file webpage.html in the folder where SharpDevelop puts your compiled executable (normally under bin\Debug in the folder where your source code is):

You can then open the file with your favourite text editor (Notepad++ is a good one) to view the full HTTP response:

You'll notice that the response includes the HTTP header (at the top) and the webpage's HTML, separated by a double-newline. As an exercise, try discarding the HTTP header, leaving only the HTML webpage.

Wonderful! :) In this article we have seen how easy it is to communicate with servers out there, and particularly how easy it is to download a webpage. If you want to learn about HTTP, HTTP Made Really Easy is a great place to start. You can also read an old blog post called "HTTP Communication: A Closer Look" which I had written about certain insights I observed while working on my BSc's Final Year Project. Finally, to learn about the HTTP protocol, there's no better place than RFC2616, which is the official standard.

We will do more network programming here in the future, so check back for more! :)

C# Network Programming: Echo Client/Server

Hola! :)

In today's article we're going to learn about network programming. That means you can have two (or more) machines talking to each other.

I have been doing network programming since 2007, and I can tell you it is awesome! :D This was one of my early projects:

This was a pacman game over a Google Maps setting when Google Android was still in its infancy. You can do a lot of cool stuff when computers interact with each other.

Today we'll write two small programs and have them communicate with each other. In order to do network programming, you will need to use the following libraries:

using System;
using System.IO;
using System.Net;
using System.Net.Sockets;

After adding the above in a new console application project, put in the following code:

            IPAddress ip = IPAddress.Any;
            int port = 18000;
            TcpListener server = new TcpListener(ip, port);
            server.Start();
            TcpClient client = server.AcceptTcpClient();

In network programming, you normally have a server, and any number of clients. The clients can connect to the server because they know its IP address and port. The IP address is a number identifying the machine (such as 192.168.5.185), and the port is a number used to connect to a particular server program (e.g. HTTP servers use port 80; SSH servers use port 22).

In the code above, we are simply starting a server and setting it to listen for connections on port 18000. The IP address is not important since it's the same as the machine running the program - so we set it to IPAddress.Any. A TCPListener is an actual server object: it allows us to accept connections from other machines and work with them. The TCPListener is started and then waits for a client to connect to it. When this happens, we obtain a TCPClient object. We can then talk to this client by obtaining its NetworkStream:

            NetworkStream stream = client.GetStream();

We can use this network stream the same way we did with files:

            using (StreamReader reader = new StreamReader(stream))
            using (StreamWriter writer = new StreamWriter(stream))
            {
                String line = reader.ReadLine();
                Console.WriteLine("Client said: {0}", line);
                writer.WriteLine(line);
            }

What we do here is wait for a line of text to arrive from the client that connected earlier, and store it in the line variable. After showing what we received, we use the StreamWriter to send back the same line of text.

If you press F5 now, all you get is a blank window: the program isn't doing anything while waiting for a connection.

Start a new console application for the client. Again, make sure you are using the correct libraries:

using System;
using System.IO;
using System.Net;
using System.Net.Sockets;

Now, add code to connect to the server:

            TcpClient client = new TcpClient("127.0.0.1", 18000);

The IP address 127.0.0.1 is special and means you are connecting on the same machine. If you are running the server on a different machine, you will need to change the IP address in the code above.

Next, we obtain the client's NetworkStream, as we did earlier for the server:

            NetworkStream stream = client.GetStream();

We can now use it to talk to the server:

            using (StreamReader reader = new StreamReader(stream))
            using (StreamWriter writer = new StreamWriter(stream))
            {
                Console.WriteLine("Write something to send to server:");
                String input = Console.ReadLine();
                writer.WriteLine(input);
                writer.Flush();
              
                String response = reader.ReadLine();
                Console.WriteLine("Server said: {0}", response);
            }
          
            Console.Write("Press any key to continue . . . ");
            Console.ReadKey(true);

When the user types something and presses ENTER, it is stored in the input variable. We then use writer.WriteLine(input) to send the input to the server.

The writer.Flush() is very important. If you leave it out, the program will be stuck and send nothing to the server. Like most input/output (I/O), network streams are buffered. That means they usually wait to have a certain amount of data before actually sending it out. The Flush() call forces the data to be sent.

Always remember: streams and toilets must always be flushed.

When the server sends back its response, we store it in the response variable, and show it in the console window. If you run this program now, you get the following exception:

Well duh, that's because the server is not running. So go back to your first (server) program and leave it running. Then, run the second (client) program:

Amazing! :D You have just manage to make two programs talk to each other. If you haven't already, try putting the server on one machine and the client on another (don't forget to change the IP address in the client).

What we have done here is an example of a simple protocol. A protocol consists of the rules by which computers talk to each other. In this case:

1. Client connects to server.
2. Client sends a line of text to server.
3. Server sends back that same line of text.

This is called an echo protocol, because the server echoes what the client says. Something of this sort is actually a standard echo protocol (RFC862) intended mostly for debugging.

Naturally, what we did here is very simple. Many standard protocols, such as IMAP (used for email), can get very complicated. Also, you'll notice that a new server must be run in order to handle each new client. We'll deal with this another time. Finally, if you have been following the above code carefully, you'll notice that I didn't Flush() the stream in the server program, even though I was writing data to it. That's because the stream is automatically closed because of the using statement. When that happens, any data in the stream is flushed, so we don't need to do that in code.

As you can see, it is very easy to write network programs in C# (not so much in other languages, such as C). Stick around, because there is much more to learn about network programming, and I will be writing several other articles on the topic that go into more detail and show you how to do certain things (e.g. download email or webpages).