IMAP: Message and Folder Attributes

Hello!

In the last article, "IMAP: Working with Folders", you may have noticed that hMailServer returns some data about folders you select. For example:

C: 0002 SELECT INBOX
S: * 3 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376229395] current uidvalidity
S: * OK [UNSEEN 2] unseen messages
S: * OK [UIDNEXT 4] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0002 OK [READ-WRITE] SELECT completed

In this article we'll cover what these items mean, and learn how messages are organised in IMAP folders.

Let's start off with the UIDVALIDITY. The UIDVALIDITY is a 32-bit value associated with a folder when it is created. Each time a folder is selected, this UIDVALIDITY value is returned (as above). Normally, this value doesn't change. If it does, it means that the folder somehow got messed up on the server (it might also have been recreated with the same name). IMAP clients check this UIDVALIDITY, and if it changes, they are supposed to discard the messages they downloaded for that folder and download the entire folder again from scratch.

Email messages in a folder are identified by two different means: unique identifiers (UIDs) and sequence numbers. These are both 32-bit numbers but they work a bit differently. Let's say you have five messages in your INBOX folder:

Seq Number	1	2	3	4	5
UID	1	2	3	4	5

Initially, the sequence numbers and UIDs are the same. But look at what happens when the third message gets deleted:

Seq Number	1	2	3	4
UID	1	2	4	5

See, when a message gets deleted, the sequence numbers are reassigned to fill in the gap (sequence numbers above 3 are deducted by one in this case). So if there are n messages in a folder, sequence numbers run continuously from 1 to n without any gaps. On the other hand, message UIDs never change. If a message is deleted, its UID vanishes with it.

We have already seen the use of these identifiers in the FETCH command in the recent article "IMAP: Downloading emails". In a FETCH command like this...

0004 FETCH 1 BODY[]

...the '1' represents the sequence number of the message you want to retrieve. You can use UIDs instead, by using a UID FETCH command instead:

0005 UID FETCH 1 BODY[]

Back to the SELECT response at the beginning of this article, some things should now be clear. The EXISTS part tells you how many emails are in the folder - it's also the value of the highest sequence number available. UIDNEXT is a value higher than the highest UID in the folder, predicted but not required to be assigned when a new message is added in the folder. Section 2.3.1.1 of RFC3501 explains it pretty well:

   The next unique identifier value is the predicted value that will be
   assigned to a new message in the mailbox.  Unless the unique
   identifier validity also changes (see below), the next unique
   identifier value MUST have the following two characteristics.  First,
   the next unique identifier value MUST NOT change unless new messages
   are added to the mailbox; and second, the next unique identifier
   value MUST change whenever new messages are added to the mailbox,
   even if those new messages are subsequently expunged.

        Note: The next unique identifier value is intended to
        provide a means for a client to determine whether any
        messages have been delivered to the mailbox since the
        previous time it checked this value.  It is not intended to
        provide any guarantee that any message will have this
        unique identifier.  A client can only assume, at the time
        that it obtains the next unique identifier value, that
        messages arriving after that time will have a UID greater
        than or equal to that value.

Now, let's talk about flags. A message can be assigned a set of flags. On most servers, the flags are predefined and are the following:

• \Seen - if set, message is marked as read
• \Answered - if set, message is marked as replied to
• \Flagged - if set, message has a special status (e.g. flagged/starred/etc)
• \Deleted - more on this in a second
• \Draft - if set, message is marked as a draft; clients usually just create a folder for drafts and don't bother with this
• \Recent - messages that have been added to a folder are marked as recent; this status is removed once the folder is selected again later

You'll notice that some of the functionality you're used to, such as read/unread messages and messages being marked as answered in Outlook are supported in IMAP by flags. The \Deleted flag, however, deserves some special attention.

In IMAP, there is no Recycle Bin or Trash folder. Clients emulate Recycle Bin functionality by creating a Deleted Items folder (actual name varies between clients). When messages are deleted from another folder, they are moved to the Deleted Items folder. When they are deleted from the Deleted Items folder, they are deleted permanently.

Messages are deleted in IMAP in two stages. First, they are marked for deletion by setting the \Deleted flag. Then, an EXPUNGE command clears the entire folder of messages marked for deletion. Alternatively, a CLOSE command does the same as the EXPUNGE command but also deselects the folder.

Servers may optionally allow custom flags to be set. These are called keywords; they work like tags and don't start with a backslash (\). Servers that support keywords return a \* as part of the PERMANENTFLAGS line in the SELECT response - you can see this in Gmail:

S: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Flags permitted.

Now that I've explained flags, we can understand the remainder of the SELECT response. The RECENT count shows how many messages are marked with the \Recent flag - this may help to indicate any new messages in the folder, although using UIDNEXT is more reliable. The UNSEEN line (optional) gives the sequence number of the first message in the folder that is not marked with the \Seen flag.

The FLAGS line tells you what flags are supported for the selected folder, and the PERMANENTFLAGS tells you which flags can be modified. If any flags are in FLAGS but not in PERMANENTFLAGS, then they can only be modified temporarily; the old value is seen when a new session is initiated.

Any data about an email message can be retrieved using the FETCH command, and that includes UIDs and flags:

C: 0009 FETCH 1:* (UID FLAGS)
S: * 1 FETCH (UID 1 FLAGS (\Seen))
S: * 2 FETCH (UID 2 FLAGS ())
S: * 3 FETCH (UID 3 FLAGS ())
S: 0009 OK FETCH completed

This FETCH command is similar to the ones we used before. Instead of a single number, we specified 1:*, which means all messages in the range from 1 to *. The * unintuitively resolves to the highest sequence number in the folder, in this case 3.

The BODY[] field we were using earlier is substituted for (UID FLAGS) here. Although we could fetch UID or FLAGS individually (with or without brackets), we can specify several fields at once using an IMAP list - a bracketed space-delimited sequence of words. In the response lines, the numbers on the left (beside the asterisks) are the sequence numbers, and the items in the outer set of brackets are key-value pairs.

We can change flags using the STORE command:

C: 0010 STORE 1 +FLAGS (\Deleted)
S: * 1 FETCH (FLAGS (\Deleted \Seen) UID 1)
S: 0010 OK STORE completed

Like FETCH, the STORE command takes a sequence number, or can take a UID if preceded by the UID keyword. The second parameter (+FLAGS) describes the action to be taken. +FLAGS adds the flags in the last parameter; FLAGS replaces all flags with those in the last parameter; and -FLAGS removes the flags in the last parameter. For example:

C: 0011 STORE 2 FLAGS (\Draft \Flagged \Seen)
S: * 2 FETCH (FLAGS (\Flagged \Draft \Seen) UID 2)
S: 0011 OK STORE completed
C: 0012 STORE 1 -FLAGS (\Seen)
S: * 1 FETCH (FLAGS (\Deleted) UID 1)
S: 0012 OK STORE completed

So now the message flags look like this:

C: 0013 FETCH 1:* (UID FLAGS)
S: * 1 FETCH (UID 1 FLAGS (\Deleted))
S: * 2 FETCH (UID 2 FLAGS (\Draft \Flagged \Seen))
S: * 3 FETCH (UID 3 FLAGS ())
S: 0013 OK FETCH completed

When you want to delete messages marked with the \Deleted flag, just send an EXPUNGE or CLOSE command:

C: 0014 EXPUNGE
S: * 1 EXPUNGE
S: 0014 OK EXPUNGE Completed

The response gives you sequence numbers of deleted messages. These may include duplicates, because as messages are deleted, sequence numbers are decreased as explained earlier.

That's all for today! This article explained the various metadata associated with IMAP folders and email messages. It explained the difference between UIDs and sequence numbers, and how to work with flags among other things. I hope you'll come back to learn more! :)

IMAP: Working with Folders

Greetings!

In yesterday's article, "IMAP: Downloading emails", we saw how to log into an email account using IMAP, select a folder (in this case the INBOX), and download an email. The INBOX is the only folder that is standard in IMAP and always exists, even if empty.

In IMAP, however, the INBOX does not need to be the only folder. RFC3501, which to date defines the latest IMAP standard called IMAP4rev1 (which you no doubt noticed in CAPABILITY response in yesterday's article), provides several commands used for working with folders.

You can create folders using the CREATE command, which is used as follows:

CREATE foldername

Use this command to create a few folders we can work with:

You can now view a list of all folders using the LIST command, as follows:

02 LIST "" "*"

The parameters to LIST allow you to list a subset of folders rather than all of them, but that's not important at this stage. The command as above will give you the full list:

You'll notice the Trash and Sent folders which we didn't create. These folders were actually created by Thunderbird when I set up my account in it. The desktop clients create certain folders used to store sent emails, deleted emails, etc. These folders aren't standard, and in fact clients don't always use the same conventions (e.g. Thunderbird uses a Sent folder, while Outlook creates a Sent Items folder instead).

Aside from the actual folder name, each line of the LIST response gives you some additional information about the folder. In this case it's telling us that the folders don't have any children, and what the hierarchy delimiter is (in this case the dot). This is because folders can actually have subfolders, and the hierarchy delimiter separates each folder from its child (e.g. folder.subfolder if the dot is the hierarchy delimiter). There's actually a special version of the LIST command that is used specifically to retrieve the hierarchy delimiter:

C: 0006 LIST "" ""
S: * LIST (\Noselect) "." ""

Note that not all servers necessarily use the dot as the hierarchy delimiter. In hMailServer, for example, it is possible to change it:

Let's try creating a folder with a subfolder:

C: 0007 CREATE Code.PHP
S: 0007 OK CREATE Completed

If you open Thunderbird, you can see the folder hierarchy. First, though, you'll need to right click on the account to the left and select Subscribe...:

Yup, because in IMAP there's this thing called folder subscription. You might have hundreds of folders, but you might only be interested in a few of them. So aside from the LIST command, there's the LSUB command which only lists subscribed folders:

C: 0008 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: 0008 OK LSUB completed

Notice how this list is shorter than the one we got with the LIST command. We can subscribe folders using the SUBSCRIBE command (which is what happens if you tick those checkboxes in Thunderbird):

C: 0009 SUBSCRIBE Personal
S: 0009 OK Subscribe completed

If we repeat the LSUB command, the list is now updated:

C: 0010 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: * LSUB (\HasNoChildren) "." "Personal"
S: 0010 OK LSUB completed

To unsubscribe a folder, just use the UNSUBSCRIBE command:

C: 0011 UNSUBSCRIBE Personal
S: 0011 OK Unsubscribe completed

If you try the LSUB now, it will give you the same result as before.

You can delete a folder completely using the DELETE command:

C: 0013 DELETE Work
S: 0013 OK Delete completed

What are you looking at... everyone wants to delete work, no? :)

Finally, if you want to work with the emails in a folder (download them, flag them, mark them as unread, etc), then you'll have to select the folder. We've already seen the SELECT command in yesterday's article:

C: 0014 SELECT Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0014 OK [READ-WRITE] SELECT completed

We'll learn about the details of this response in another article. There is, however, another command that can be used to select folders: the EXAMINE command. Let's try it out:

C: 0018 EXAMINE Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS ()] limited
S: 0018 OK [READ-ONLY] EXAMINE completed

You'll notice that the response is mostly the same, except for the last two lines. The important thing to notice is that the SELECT command results in a READ-WRITE attribute in the last line, while the EXAMINE command gives you a READ-ONLY. That's precisely the difference between SELECT and EXAMINE: they essentially do the same thing, but EXAMINE selects the folder in read-only mode and doesn't let you make any changes.

Nice! Today we learned how to work with folders in IMAP. You can select a folder to work with using the SELECT (read-write) or EXAMINE (read-only) commands. Folders can be created with the CREATE command and deleted with the DELETE command (duh). Subfolders may be created by appending the hierarchy delimiter (e.g. ".") followed by the subfolder name to the parent folder name. All folders may be shown using the LIST command, while the LSUB command lists only subscribed folders. Folder subscription may be toggled using the SUBSCRIBE and UNSUBSCRIBE commands.

I hope this was interesting, and that you'll come back to learn more about IMAP in the upcoming articles! :)