IMAP: Working with Folders

Greetings!

In yesterday's article, "IMAP: Downloading emails", we saw how to log into an email account using IMAP, select a folder (in this case the INBOX), and download an email. The INBOX is the only folder that is standard in IMAP and always exists, even if empty.

In IMAP, however, the INBOX does not need to be the only folder. RFC3501, which to date defines the latest IMAP standard called IMAP4rev1 (which you no doubt noticed in CAPABILITY response in yesterday's article), provides several commands used for working with folders.

You can create folders using the CREATE command, which is used as follows:

CREATE foldername

Use this command to create a few folders we can work with:

You can now view a list of all folders using the LIST command, as follows:

02 LIST "" "*"

The parameters to LIST allow you to list a subset of folders rather than all of them, but that's not important at this stage. The command as above will give you the full list:

You'll notice the Trash and Sent folders which we didn't create. These folders were actually created by Thunderbird when I set up my account in it. The desktop clients create certain folders used to store sent emails, deleted emails, etc. These folders aren't standard, and in fact clients don't always use the same conventions (e.g. Thunderbird uses a Sent folder, while Outlook creates a Sent Items folder instead).

Aside from the actual folder name, each line of the LIST response gives you some additional information about the folder. In this case it's telling us that the folders don't have any children, and what the hierarchy delimiter is (in this case the dot). This is because folders can actually have subfolders, and the hierarchy delimiter separates each folder from its child (e.g. folder.subfolder if the dot is the hierarchy delimiter). There's actually a special version of the LIST command that is used specifically to retrieve the hierarchy delimiter:

C: 0006 LIST "" ""
S: * LIST (\Noselect) "." ""

Note that not all servers necessarily use the dot as the hierarchy delimiter. In hMailServer, for example, it is possible to change it:

Let's try creating a folder with a subfolder:

C: 0007 CREATE Code.PHP
S: 0007 OK CREATE Completed

If you open Thunderbird, you can see the folder hierarchy. First, though, you'll need to right click on the account to the left and select Subscribe...:

Yup, because in IMAP there's this thing called folder subscription. You might have hundreds of folders, but you might only be interested in a few of them. So aside from the LIST command, there's the LSUB command which only lists subscribed folders:

C: 0008 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: 0008 OK LSUB completed

Notice how this list is shorter than the one we got with the LIST command. We can subscribe folders using the SUBSCRIBE command (which is what happens if you tick those checkboxes in Thunderbird):

C: 0009 SUBSCRIBE Personal
S: 0009 OK Subscribe completed

If we repeat the LSUB command, the list is now updated:

C: 0010 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: * LSUB (\HasNoChildren) "." "Personal"
S: 0010 OK LSUB completed

To unsubscribe a folder, just use the UNSUBSCRIBE command:

C: 0011 UNSUBSCRIBE Personal
S: 0011 OK Unsubscribe completed

If you try the LSUB now, it will give you the same result as before.

You can delete a folder completely using the DELETE command:

C: 0013 DELETE Work
S: 0013 OK Delete completed

What are you looking at... everyone wants to delete work, no? :)

Finally, if you want to work with the emails in a folder (download them, flag them, mark them as unread, etc), then you'll have to select the folder. We've already seen the SELECT command in yesterday's article:

C: 0014 SELECT Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0014 OK [READ-WRITE] SELECT completed

We'll learn about the details of this response in another article. There is, however, another command that can be used to select folders: the EXAMINE command. Let's try it out:

C: 0018 EXAMINE Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS ()] limited
S: 0018 OK [READ-ONLY] EXAMINE completed

You'll notice that the response is mostly the same, except for the last two lines. The important thing to notice is that the SELECT command results in a READ-WRITE attribute in the last line, while the EXAMINE command gives you a READ-ONLY. That's precisely the difference between SELECT and EXAMINE: they essentially do the same thing, but EXAMINE selects the folder in read-only mode and doesn't let you make any changes.

Nice! Today we learned how to work with folders in IMAP. You can select a folder to work with using the SELECT (read-write) or EXAMINE (read-only) commands. Folders can be created with the CREATE command and deleted with the DELETE command (duh). Subfolders may be created by appending the hierarchy delimiter (e.g. ".") followed by the subfolder name to the parent folder name. All folders may be shown using the LIST command, while the LSUB command lists only subscribed folders. Folder subscription may be toggled using the SUBSCRIBE and UNSUBSCRIBE commands.

I hope this was interesting, and that you'll come back to learn more about IMAP in the upcoming articles! :)

IMAP: Downloading emails

Hello friends!

In yesterday's article, "Email: Protocols and Background", I explained how to set up an email client and server to study the email protocols, and also how to use Wireshark to observe how the clients and servers actually talk to each other using these protocols.

Today we'll begin learning IMAP, and use it to access our inbox and download emails.

As I explained in yesterday's article, IMAP is a protocol made up of text commands. You can open a socket (see "C# Network Programming: Simple HTTP Client"), send commands, and interpret the responses. Without needing to write any code, you can use the telnet program to open a raw connection and work using the command line. Connect to your hMailServer IMAP server using a command like this:

telnet serverIP 143

Obviously, replace serverIP with the actual IP address or hostname of the computer where hMailServer is installed. 143 is the port on which the IMAP server listens by default.

If telnet is not installed, you can install it on Windows 7 as follows. In the Start search box, find Programs and Features and open it. Select "Turn Windows features on or off" (it's a link on the left hand side). In the Windows Features window, tick the "Telnet Client" checkbox and click OK.

With telnet installed, connect to the IMAP server as described above. You should see the initial greeting from the hMailServer IMAP server:

* OK IMAPrev1

Now, type the following command and press ENTER:

01 capability

The response consists of a number of words that describe what functionality the IMAP server supports:

Next, login to the email account you created as part of yesterday's article. In my case the command looks like this:

02 login user@ranchtest.local pass

The response will tell you whether the login was successful or not.

In order to make this a little interesting, send a few emails to yourself via Thunderbird so you actually have something in your inbox. Next, access the inbox using the following command. The INBOX folder is standard in IMAP and always exists, even if you don't have anything in it.

03 select INBOX

You should be able to see a few things at this stage. First, whenever we send a command, we precede it with something like "01". This is called a tag, and can be any string (different clients use different formats ranging from numbers to random strings). When the server response to a command, the last line always starts with the same tag as the command - that way a client knows that the response for that command was received.

You'll probably also realise that telnet is a pain in the ass. If you make mistakes you can't backspace - that's because telnet sends everything you type, byte by byte. So each character is sent immediately and can't be undone. Last year I wrote a program called IMAPTalk which makes working with IMAP (and other protocols) much more convenient. Just download it and run it - no installation necessary. Just enter the hostname and connect:

If you turn on Auto-generate tags, you can actually leave out the tags and just type the commands - the tags will be filled in for you by IMAPTalk. If you repeat the commands we did above in IMAPTalk, it looks like this:

Better, no? If you look at the responses (in red), you'll notice there are a bunch of things I haven't explained yet. Don't worry about them just yet. The important thing you should take from the response to the SELECT command is this like:

* 3 EXISTS

That's telling us that there are 3 messages in the INBOX.

We can then retrieve them one by one using a FETCH command, while providing the message number:

04 fetch 1 BODY[]

The response is as follows:

You'll notice that there is a bunch of stuff in there, for such a short message. Part of it (the line starting with the * 1 FETCH) is IMAP, as are the last two lines. The stuff in between is the full email, known as the MIME. It consists of a header with a bunch of fields and values (you'll notice important stuff such as From, To, Date, etc), and after a double line break, the message itself. You can see this in Gmail by clicking on the arrow at the top-right of an opened email and clicking "Show original".

You can similarly retrieve the other messages in the inbox by replacing the 1 in the FETCH command with the message number. This is called the message sequence number, and can't be larger than the number of emails in the folder (in this case 3). If you provide a larger number you won't get an error, but you won't get any email data either.

So that's the easiest way to download emails manually using IMAP. We'll learn more about how messages are stored in IMAP folders, and more about the facilities offered by the IMAP protocol, in the coming articles. Come back again for more!