Email: Protocols and Background

Hullo!

You sure as hell know what email is, but do you know how it actually works? In today's article I'll talk about how it works, and how best to learn about it in detail.

One of my previous articles, "HTTP Requests in Wireshark", showed that the World Wide Web was based on a standard protocol, HTTP. Web browsers and web servers could talk to each other using this protocol.

It turns out that email is no different. Email clients (e.g. Outlook, Thunderbird, etc) and servers (e.g. Microsoft Exchange, hMailServer, etc) can talk to each other using three standard protocols: IMAP, POP and SMTP. SMTP is used for sending email, while IMAP and POP are used for retrieving email from a server. IMAP is more powerful and complex than POP; POP doesn't even support folders. Each of these protocols uses a set of text commands similar to stuff you might have used in the command line, except that in this case you do it over the network (you'll see how in a minute). The protocols also define ports on which servers should listen, e.g. IMAP uses ports 143 and 993 by default.

You could study these protocols by just reading their respective RFCs, but you can only really learn them properly by seeing them in action. To do this, you will need some software.

First, install a mail server. hMailServer is decent and free, so unless you have a better alternative, go with it. Ideally install this on another computer, otherwise you won't be able to capture requests in Wireshark if your client and server are on the same PC. While installing hMailServer, choose SQL Compact as storage (to keep it simple) and give it an admin password.

When you're done installing hMailServer, you need to create an account. Run hMailServer and enter the admin password to access the Administrator console:

Click on the Add domain... button and invent a domain name. By convention it should end in .local, since this isn't a real domain name. Click the Save button to save it.

Next, find the Accounts folder somewhere under the Domains node.

Click the Add... button. Enter a username and password and click Save to create your account. Notice how the domain is used as part of the email address, for example in my case the email address is user@ranchtest.local.

Great. Now, we need to test this account using an email client. Thunderbird is a popular choice and it's free, although there are many others you can use if you want. Install and run it. Skip the shit that appears upon running:

Press Alt to show the program menu, since Thunderbird seem to have fallen into the horrible practice of hiding it. From the Tools menu, click on Account Settings...:

From the drop-down list, select Add Mail Account...:

Enter your basic account credentials, and then click the Manual config button when it appears:

Now you need tell Thunderbird how to connect to your mail server. Under Server hostname, type the name or IP address of the computer where hMailServer is installed. Using the PC name is better since the IP address might change regularly. If you installed hMailServer on the same machine (not recommended, see above) you can use 127.0.0.1 or localhost as the Server hostname. Other than that, use port 143 for IMAP and port 25 for SMTP; no SSL and Normal password for Authentication. It should look like this:

Click the Re-test button to allow Thunderburd to test those settings. If all is ok, the Done button is enabled, click it to save the settings and create the account. When you do this, a warning appears:

It's a security warning because the client and server will talk to each other on an unencrypted channel. Since you're working on a local test environment, you don't have to worry about it. We actually want the messages to be unencrypted, so that we can capture them with Wireshark.

Finally, install Wireshark so that we can take a look at the email protocols. Set it up for capturing; I explained how to do this in my article "HTTP Requests in Wireshark". After starting the capture session, set the Filter to "imap".

Using Thunderbird, send an email to yourself:

The email is sent to the server via SMTP, and then downloaded to the client via IMAP. After a second you should see some activity in Wireshark. From Thunderbird, click Get Mail to retrieve the new email in your inbox.

In Wireshark, you can see the IMAP commands as well as the email data. You can view this conveniently by right clicking on one of the packets and selecting Follow TCP Stream:

This might seem a bit cryptic, but it's quite easy to learn. In future articles I'll cover the email protocols bit by bit. So check back later for more about email technology! :)

HTTP Requests in Wireshark

Hi everyone! :)

In yesterday's article, Network Programming: Networking Theory, we discussed what happens when a message is sent over a network, and when it is received. Today, we're going to see a practical example of that, by observing the HTTP requests sent by a web browser.

The first thing you should do is download Wireshark. This program will allow you to monitor network traffic going into and out of your PC. After installing it, run it, and you will see the following main screen:

Click on "Capture Options" and tick the checkbox next to the network interface listed. The network interface is basically a network card or, more commonly, the networking hardware on your motherboard. Wireshark can monitor traffic passing through the Ethernet port.

Click the "Start" button to start capturing packets. Immediately, you will start seeing stuff going in and out of your PC. You will know whether it's incoming or outgoing depending on whether your PC's IP address is in the "Source" or "Destination" column (in the screenshot below, my IP address is hidden):

In the "Filter" field at the top, type "http" and press ENTER. This filter allows you to concentrate on a specific type of network traffic - in this case, we are focusing on HTTP traffic which is used by web browsers.

In the Capture menu, Restart capturing, since there is a lot of traffic that doesn't interest us. From a web browser, visit http://www.programmersranch.com/. Soon after, Stop capturing in Wireshark from the Capture menu.

You can now find various HTTP requests to various parts of the page at programmersranch.com, including the page itself and various images. The screenshot above shows the HTTP request for the main page.You can expand the sections towards the middle of the window to view more detail about various parts of the transmission. In this case, I've expanded the HTTP section, where you can see the whole HTTP request. You can do the same for TCP, IP, etc.

When you click on a particular section (such as HTTP), the relevant part of the hex view (at the bottom of the window) is highlighted. This is useful because it sometimes shows you things that you might otherwise miss. In particular, you'll notice that the last four characters are represented by hex values: 0d 0a 0d 0a. In decimal, this becomes 13 10 13 10, which map to the ASCII values of CR LF CR LF (carriage return, line feed, carriage return, line feed). In short, you have two blank lines at the end of the HTTP request. They are important because HTTP requests won't work without them.

You should also be able to find the HTTP response coming from the server, which contains the HTML arriving at your browser (shown above).

Finally, in Wireshark you can right click on a particular transmission and select "Follow TCP Stream":

This allows you to view all the relevant requests and responses on the same connection without having to find the packets one by one:

Be aware, however, that following a TCP stream like this will change the filter from http to something else. This means that you won't be seeing all incoming HTTP packets. Be sure to change the filter back in order to continue viewing HTTP traffic.

Very well. You now know how to use Wireshark to sniff packets going into and out of your PC. In code, you can create the same messages and send them out in a socket in order to achieve the same behaviour that browsers, email clients, etc. have. In tomorrow's article, we will be working with HTTP in code. So stick around. :)