On Mystery Meat Navigation and Unusability

Hello folks! :)

Today I'm going to talk about Mystery Meat Navigation (MMN). This term was invented 15 years ago by Vincent Flanders of Web Pages That Suck. It refers to a horrible practice of filling a website's navigation with meaningless icons. When a user moves his mouse over one of these icons, the icon changes or pops up some text, revealing what it really does.

User Experience Invention has a couple of fantastic examples of mystery meat navigation. First, the icons at the top are completely meaningless until you mouse over them to reveal things like "Buy our Stuff" and "Jobs". At the bottom is a road sign analogy: imagine you're driving and there's an empty sign, which transforms into an "Exit" sign just as you drive past it. Oops!

Even now, 15 years later, MMN is still used on the web. Even reputable web design companies here in Malta have fallen in the MMN trap. Alert eBusiness, for instance:

Right, so what do those icons mean? The last one seems pretty clear: a shopping cart. Mousing over it reveals it stands for Alert Payment Gateway, which is close enough. But what about the rest? The first one is a mouse, for instance. Would it ever cross your mind that it actually means "Web Design"?

Another example: Pcionix:

The home icon is pretty obvious, so that can be forgiven. But a pie chart that stands for SEO - seriously?

But this, from design.com.mt, is even worse:

This has got to be the worst of them all. Whereas you might be able to somehow guess what the icons in the other sites mean, the navigation here is hidden behind meaningless numbers that you again have to mouse over to understand.

It gets worse: there are videos on YouTube of sites with iconic navigation that actually floats around, so you actually have to find out where that "About Us" cube thingy moved to (examples: Mandarina Duck, Qualcomm).

So why is MMN bad? In case it isn't obvious, it is very annoying for users to have to click on stuff to figure out what the page offers. A website should give a clear indication of how it is structured, without the user needing to interact with it just to get an idea. Imagine you're driving and need to interact with a bunch of direction signs (such as these) one by one to get an idea of the places in the area. Then, after sifting through a dozen, you forget what you saw earlier and have to go back and interact with them again. Sorry, the "just a click away" idea is not an excuse when it comes to navigation, which is really a website's backbone.

Another great example comes from feedback that Vincent Flanders received, and illustrates how MMN would be if applied to a business's answering machine:

"You've reached XYZ Corporation. To find out what option #1 is, press 1. To find out what option #2 is, press 2. (Etc....) If you'd like to continue doing business with our company after we've slapped you around and wasted your valuable time, press 9"

MMN is a slap in the face of usability. It shows meaningless icons in the place of important navigational information. What could possibly worse?

The only thing worse than showing meaningless icons is not showing any icons at all! That's pretty much the direction taken by Windows 8's notorious alternate UI, formerly known as Metro. One if its design principles is "Do more with less" which includes "Put content before chrome". In this case the "chrome" refers to the stuff that makes the application - menus, the 'X' button at the top-right, toolbars, etc. So basically you end up with something like this:

That's the default PDF viewer on Windows 8 - one full screen Windows 8 Style (the new name for Metro) app with the PDF content and nothing else, not even an 'X' to close it. In fact Windows 8 users are somehow expected to know beforehand ("by osmosis", as this Windows 8.1 review puts it) that to close a Windows 8 Style app you have to grab it from the top and drag downwards with your mouse. Contrast this with the same PDF viewed on Windows 7:

Needless to say, everything that you can do with a PDF is immediately accessible either from the toolbars or via the menus. There is no hidden stuff, no needing to drag your mouse into a corner to open some Start Screen or Charms Bar. See, the program actually shows you what it can do, and for new users that's important. The "Content before Chrome" idea is wrong precisely because when you use a program, you want to do stuff, not just see stuff.

So it's no wonder that Microsoft seems to have made a U-turn on its Windows 8 Style design stuff. If MMN is an example of bad usability, this Windows 8 abomination is an example of... unusability.

C# WPF: Styling buttons in a window

Hi! :)

In this article we'll take a look at WPF styles, which allow us to manage the properties of many controls in one place. If you've used HTML and CSS in the past, this is a bit like how CSS allows you to customise the layout of HTML elements, leaving HTML to focus on the structure of the document.

To see how styling works, we'll create a Tic Tac Toe application (just the buttons... the logic is beyond the scope of the article). Start off by creating a new WPF application (I'm using Visual Studio 2012, but you may just as well use SharpDevelop or some other version of Visual Studio):

In MainWindow.xaml, you get some default XAML code. In VS2012, it looks like this:

<Window x:Class="CsWpfStyleButtons.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="525">
    <Grid>
       
    </Grid>
</Window>

A WPF UniformGrid gives us an easy way to make an nxn grid, and in Tic Tac Toe we want a 3x3 grid, so UniformGrid is perfect for the job. Change the XAML as follows (I've also edited some properties of the window):

<Window x:Class="CsWpfStyleButtons.MainWindow"

        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

        Title="Tic Tac Toe" Height="200" Width="200">

    <UniformGrid>

        <Button>X</Button>

        <Button></Button>

        <Button>O</Button>

        <Button></Button>

        <Button></Button>

        <Button>X</Button>

        <Button></Button>

        <Button>O</Button>

        <Button></Button>

    </UniformGrid>

</Window>

At this point, we get this:

Right, so what if we want to customise it? We might want a bigger font, for instance, but that would require setting the same attributes on each and every button. If we add a few more properties, the XAML quickly becomes a nightmare to maintain:

To make our life much easier, we can use styles. We define styles in a Window.Resources section, that goes just after the <Window> declaration and before the <UniformGrid>:

    <Window.Resources>

        <Style TargetType="{x:Type Button}">

            <Setter Property="FontSize" Value="24" />

            <Setter Property="Foreground" Value="DarkGreen" />

            <Setter Property="Margin" Value="3" />

            <Setter Property="FontFamily" Value="Verdana" />

            <Setter Property="BorderBrush" Value="AliceBlue" />

        </Style>

    </Window.Resources>

Here we define a style that works on elements of type Button. We can remove all the attributes from our buttons and into setters in this style, as above. If you run the application, you'll see that it just works:

Styles are quite powerful, and allow us to separate structure and style to a certain extent, much in the way we use HTML to control the structure of a webpage and CSS to style it. Styles allow us to apply the same properties to all controls of a particular type, as we have done here. They also allow us to affect particular elements of our choice, by setting an x:Key attribute instead of a TargetType, much in the manner of CSS classes.

SDL2: Empty Window

Hi all! :)

[Update 2015-11-14: This article is out of date. Check out the latest version at Gigi Labs.]

Yesterday's article dealt with setting up SDL2 in Visual Studio. Today we're going to continue what we did there by showing an empty window and allowing the user to exit by pressing the X at the top-right of the window. This is very similar to the "SDL Quickstart for Linux: Empty Window" article I wrote almost three years ago on my other blog; however this article is for Windows and deals with SDL2, rather than SDL1.2.x.

It takes very little to show an empty window. Use the following code:

#include <SDL2/SDL.h>

int main(int argc, char ** argv)
{
  SDL_Init(SDL_INIT_VIDEO);

SDL_Window * screen = SDL_CreateWindow("My SDL Empty Window", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, 640, 480, 0);

  SDL_Quit();

  return 0;
}

We use SDL_Init() to initialise SDL, and tell it which subsystems we need - in this case video is enough. At the end, we use SDL_Quit() to clean up. It is possible to set up SDL_Quit with atexit(), as the SDL_Quit() documentation shows.

We create a window using SDL_CreateWindow(). This is quite different from how we used to do it in SDL 1.2.x. We pass it the window caption, initial coordinates where to put the window (not important in our case), window width and height, and flags (e.g. fullscreen).

If you try and run the code, it will work, but the window will flash for half a second and then disappear. You can put a call to SDL_Delay() to make it persist for a certain number of milliseconds:

  SDL_Delay(3000);

Now, let's make the window actually remain until it is closed. Use the following code:

#include <SDL2/SDL.h>

int main(int argc, char ** argv)
{
  bool quit = false;
  SDL_Event event;

  SDL_Init(SDL_INIT_VIDEO);

  SDL_Window * screen = SDL_CreateWindow("My SDL Empty Window", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, 640, 480, 0);

  while (!quit)
  {
  SDL_WaitEvent(&event);

  switch(event.type)
  {
  case SDL_QUIT:
  quit = true;
  break;
  }
  }

  SDL_Quit();

  return 0;
}

The while (!quit) part is very typical in games and is in fact called a game loop. We basically loop forever, until the conditions necessary for quitting occur.

We use SDL_WaitEvent() to wait for an event (e.g. keypress) to happen, and we pass a reference to an SDL_Event structure. Another possibility is to use SDL_PollEvent(), which checks continuously for events and consumes a lot of CPU cycles (SDL_WaitEvent() basically just sleeps until an event occurs, so it's much more lightweight).

The event type gives you an idea of what happened. It could be a key press, mouse wheel movement, touch interaction, etc. In our case we're interested in the SDL_QUIT event type, which means the user clicked the window's top-right X button to close it.

We can now run this code, and the window remains until you close it:

Wasn't that easy? You can use this as a starting point to start drawing stuff in your window. Have fun, and come back again for more tutorials! :)

SDL2: Setting up SDL2 in Visual Studio 2010

Hi folks!

[Update 2015-11-14: This article is out of date. Check out the latest version at Gigi Labs.]

Yesterday I realised that earlier this month, SDL2 has been released. SDL is a fantastic library for cross-platform game development, and I had used SDL 1.2.x for several projects ranging from my Picaxo Image Viewer to early iterations of Ultima 1 Revenge, which I mentioned in yesterday's article. I had also written an article on setting up SDL 1.2.x in Linux on my other blog. SDL2 apparently brings about many improvements we were yearning for, including support for multiple windows.

This article is a tutorial on how to set up a Visual Studio 2010 project to work with SDL2. SDL2 is a C library, so SDL2 projects are normally written in C/C++, although it is possible to use other languages. This article is just about setting things up, so you don't really need to know anything. But for future SDL2 tutorials, some C/C++ knowledge will be expected.

Before we begin, you'll need to grab some files from the SDL2 download page. You'll need the Windows development libraries for Visual C++, and you'll also need the Windows runtime binaries (we'll use the x86 ones by default, but you might need the x64 ones if you eventually want to compile 64-bit versions of your game).

Extract the contents of the development libraries. You should have an include folder, a lib folder, and a few loose files. Locate the following directory, where Visual Studio keeps its header files and libraries for C++:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A

In C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include, create a folder called SDL2 and extract the contents of the development libraries' include folder there:

Next, copy the .lib files from the development libraries' lib\x86 folder into C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Lib:

...and copy the .lib files from the development libraries' lib\x64 folder into C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Lib\x64:

This will allow Visual Studio to find the header foles and .lib's without any additional configuration. Alternatively, you can put them somewhere else and configure the paths accordingly (this article explains how), but I find the above method easier.

Next, create a project in Visual Studio. You need to select Empty Project, from Visual C++ -> General:

In Solution Explorer, right click on Source Files and add a new item. Select C++ File and name it main.cpp:

Still in Solution Explorer, right click on the project (not the solution) and select Properties.

In the treeview to the left, go to Configuration Properties -> Linker -> Input. In the field called Additional Dependencies, replace all the default crap with the following:

SDL2.lib;SDL2main.lib

Next, in Linker -> System, change SubSystem to Windows:

Now, all we need is some minimal code to compile and test. I used the following, based on TwinklebearDev's tutorial (though note the different include):

#include <SDL2/SDL.h>

int main(int argc, char ** argv)
{
    SDL_Init(SDL_INIT_EVERYTHING);
    SDL_Quit();

    return 0;
}

The project should compile, but when you try to run it, you get this:

That's because SDL2 executables need to have the SDL2.dll file in the same folder. Remember the runtime binaries you downloaded earlier? This is where they come in. Grab the SDL2.dll from the x86 runtime binaries package and put it in your project's Debug folder, where the executable is produced. If you try running it now, it should work.

Great, so it compiles and runs, but doesn't do anything. Given that there are no errors, it works, and you can use this as a starting point for further development. TwinklebearDev's tutorial even describes how to export a template from this project, so you don't need to do the same configuration every time.

I hope you found this useful, and check back for more articles in future! :)

Ultima 1 Reverse Engineering: Decoding Savegame Files

Hey all! :)

It's no secret that I'm a long-time fan of the Ultima series of games. My most significant contribution to the Ultima fan community was running Dino's Ultima Page, a news and information hub for the community, for almost ten years.

Ultima insipired such a great following that hundreds of fan-made remakes, tools and other projects appeared over the years. Many remakes were intended to be mods of games such as Dungeon Siege or Neverwinter Nights. Among other things, talented developers found ways to understand the data files of the games (including the graphics, savegames, maps, etc). This resulted in several useful tools (savegame editors, map viewers, etc), but more importantly, it allowed people to create new game engines using the original data files, with the intention of allowing the games to be played on modern operating systems. The first of these was Exult (an Ultima 7 engine), but over the years, projects appeared for most Ultimas - such as Pentagram for Ultima 8, or Nuvie for Ultima 6.

This article describes the process of reverse engineering, i.e. how to try to make sense of data files given the data files and nothing else. We will be looking at the savegame format of Ultima 1. Although Ultima 1 is not free, you can get the entire first Ultima trilogy from Good Old Games for a few bucks. Ultima 1 is a great choice to start reverse engineering because it's relatively simple - the savegame file is only 820 bytes long and is uncompressed. This made sense for me as when I started this project I was still a budding programmer, and it also made sense because there was very little knowledge about the U1 formats around, whereas the later games had been studied in depth. Reverse engineering is a bit of an advanced topic so feel free to skip it if you feel lost, but it's also a very interesting topic.

So first thing you need to do is install Ultima 1. If you got it off Good Old Games, it conveniently comes with DOSBox, allowing you to play it under modern operating systems:

After launching the game, you will find yourself in the main menu.

Press 'a' to go through the character creation process.

Once you have selected your attributes and saved your character, you find yourself back in the main menu. In the Ultima 1 folder, you should notice a new file called PLAYER1.U1:

That's the savegame file we'll be messing around with. You can use a hex editor to take a glimpse of its contents. Personally I like XVI32 because it's pretty lightweight and even allows you to edit hex entries.

This might look like gibberish, but you can already notice a few things. The first 14 bytes in the savegame file are reserved for the player's name. Knowing the stats you chose during character creation (strength 32, agility 32, stamina 15, charisma 12, wisdom 11 and intelligence 13), you can also spot them in hex on the second line (20, 20, 0F, 0C, 0B, 0D). Windows Calculator has a Programmer mode that is quite useful for switching between decimal and hex:

A good way of decoding more parts of the savegame file is interacting with the game itself. Start the game. The world view looks like this:

Keep a copy of your savegame file at this point. If you move around a bit and save the game, you should at the very least observe changes in your character's X- and Y-coordinates:

You can manually check which bytes in the savegame file changed. I found it more useful to write a hex diff tool that actually highlights the bytes that changed:

As you can see, it's not so simple: there are many things that might change, including food. However, you can choose your moves carefully (e.g. 2 steps west, 3 steps north) so that you can then spot which bytes have changed that much and determine which are the world coordinates.

Another way of learning more about the savegame file is by actually tampering with it. Using XVI32, I changed the byte after '96' from 00 to 13:

After running the game, note how the Hits shot up from 150 to 5014:

That makes a bit of sense: the 96 (hex) we saw earlier corresponds to 150 in decimal - the original value of Hits. But why did it become 5014 when we tweaked the byte after it?

It's because DOS games like this stored values as 16-bit integers in little endian format (i.e. the bigger byte is the second one). So if we have the value we tweaked, i.e. 96 13, that's actually (13 * 100) + 96 (all hex), which results in 5014 (decimal).

Isn't that neat? Reverse engineering requires a lot of time and patience, but it's a bit like fitting together the pieces of a jigsaw puzzle. After a while you might end up understanding a good chunk of the data files:

Once you understand the data files (which also includes map and graphics files), you can then proceed to write all sorts of tools and stuff. I had called this project U1Revenge (Ultima 1 Reverse Engineering Effort) and wrote a map viewer and was working on an engine for it. Although I stopped working on it last year, I did release a couple of demos, the latest of which you can grab from the project page.

Reverse engineering is certainly not a new art. The book Masters of Doom describes how fans of DOOM would hack the game's map files to create their own level editors. Many games have similarly been studied, and a wealth of knowledge is available today. Reverse engineering is not just an achievement; it is a glimpse of history, and helps to understand how games were created even before we were born. The following links provide further reading:

• Dino's Guide to Ultima 1 - additional technical info on Ultima 1
• Ultima IV Technical Info - technical info on Ultima 4; helps understand DOS graphics
• DOS Game Modding Wiki - a wealth of technical info about many DOS games
• Reverse engineering guide - basics on reverse engineering, hex and stuff

IMAP: Message and Folder Attributes

Hello!

In the last article, "IMAP: Working with Folders", you may have noticed that hMailServer returns some data about folders you select. For example:

C: 0002 SELECT INBOX
S: * 3 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376229395] current uidvalidity
S: * OK [UNSEEN 2] unseen messages
S: * OK [UIDNEXT 4] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0002 OK [READ-WRITE] SELECT completed

In this article we'll cover what these items mean, and learn how messages are organised in IMAP folders.

Let's start off with the UIDVALIDITY. The UIDVALIDITY is a 32-bit value associated with a folder when it is created. Each time a folder is selected, this UIDVALIDITY value is returned (as above). Normally, this value doesn't change. If it does, it means that the folder somehow got messed up on the server (it might also have been recreated with the same name). IMAP clients check this UIDVALIDITY, and if it changes, they are supposed to discard the messages they downloaded for that folder and download the entire folder again from scratch.

Email messages in a folder are identified by two different means: unique identifiers (UIDs) and sequence numbers. These are both 32-bit numbers but they work a bit differently. Let's say you have five messages in your INBOX folder:

Seq Number	1	2	3	4	5
UID	1	2	3	4	5

Initially, the sequence numbers and UIDs are the same. But look at what happens when the third message gets deleted:

Seq Number	1	2	3	4
UID	1	2	4	5

See, when a message gets deleted, the sequence numbers are reassigned to fill in the gap (sequence numbers above 3 are deducted by one in this case). So if there are n messages in a folder, sequence numbers run continuously from 1 to n without any gaps. On the other hand, message UIDs never change. If a message is deleted, its UID vanishes with it.

We have already seen the use of these identifiers in the FETCH command in the recent article "IMAP: Downloading emails". In a FETCH command like this...

0004 FETCH 1 BODY[]

...the '1' represents the sequence number of the message you want to retrieve. You can use UIDs instead, by using a UID FETCH command instead:

0005 UID FETCH 1 BODY[]

Back to the SELECT response at the beginning of this article, some things should now be clear. The EXISTS part tells you how many emails are in the folder - it's also the value of the highest sequence number available. UIDNEXT is a value higher than the highest UID in the folder, predicted but not required to be assigned when a new message is added in the folder. Section 2.3.1.1 of RFC3501 explains it pretty well:

   The next unique identifier value is the predicted value that will be
   assigned to a new message in the mailbox.  Unless the unique
   identifier validity also changes (see below), the next unique
   identifier value MUST have the following two characteristics.  First,
   the next unique identifier value MUST NOT change unless new messages
   are added to the mailbox; and second, the next unique identifier
   value MUST change whenever new messages are added to the mailbox,
   even if those new messages are subsequently expunged.

        Note: The next unique identifier value is intended to
        provide a means for a client to determine whether any
        messages have been delivered to the mailbox since the
        previous time it checked this value.  It is not intended to
        provide any guarantee that any message will have this
        unique identifier.  A client can only assume, at the time
        that it obtains the next unique identifier value, that
        messages arriving after that time will have a UID greater
        than or equal to that value.

Now, let's talk about flags. A message can be assigned a set of flags. On most servers, the flags are predefined and are the following:

• \Seen - if set, message is marked as read
• \Answered - if set, message is marked as replied to
• \Flagged - if set, message has a special status (e.g. flagged/starred/etc)
• \Deleted - more on this in a second
• \Draft - if set, message is marked as a draft; clients usually just create a folder for drafts and don't bother with this
• \Recent - messages that have been added to a folder are marked as recent; this status is removed once the folder is selected again later

You'll notice that some of the functionality you're used to, such as read/unread messages and messages being marked as answered in Outlook are supported in IMAP by flags. The \Deleted flag, however, deserves some special attention.

In IMAP, there is no Recycle Bin or Trash folder. Clients emulate Recycle Bin functionality by creating a Deleted Items folder (actual name varies between clients). When messages are deleted from another folder, they are moved to the Deleted Items folder. When they are deleted from the Deleted Items folder, they are deleted permanently.

Messages are deleted in IMAP in two stages. First, they are marked for deletion by setting the \Deleted flag. Then, an EXPUNGE command clears the entire folder of messages marked for deletion. Alternatively, a CLOSE command does the same as the EXPUNGE command but also deselects the folder.

Servers may optionally allow custom flags to be set. These are called keywords; they work like tags and don't start with a backslash (\). Servers that support keywords return a \* as part of the PERMANENTFLAGS line in the SELECT response - you can see this in Gmail:

S: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Flags permitted.

Now that I've explained flags, we can understand the remainder of the SELECT response. The RECENT count shows how many messages are marked with the \Recent flag - this may help to indicate any new messages in the folder, although using UIDNEXT is more reliable. The UNSEEN line (optional) gives the sequence number of the first message in the folder that is not marked with the \Seen flag.

The FLAGS line tells you what flags are supported for the selected folder, and the PERMANENTFLAGS tells you which flags can be modified. If any flags are in FLAGS but not in PERMANENTFLAGS, then they can only be modified temporarily; the old value is seen when a new session is initiated.

Any data about an email message can be retrieved using the FETCH command, and that includes UIDs and flags:

C: 0009 FETCH 1:* (UID FLAGS)
S: * 1 FETCH (UID 1 FLAGS (\Seen))
S: * 2 FETCH (UID 2 FLAGS ())
S: * 3 FETCH (UID 3 FLAGS ())
S: 0009 OK FETCH completed

This FETCH command is similar to the ones we used before. Instead of a single number, we specified 1:*, which means all messages in the range from 1 to *. The * unintuitively resolves to the highest sequence number in the folder, in this case 3.

The BODY[] field we were using earlier is substituted for (UID FLAGS) here. Although we could fetch UID or FLAGS individually (with or without brackets), we can specify several fields at once using an IMAP list - a bracketed space-delimited sequence of words. In the response lines, the numbers on the left (beside the asterisks) are the sequence numbers, and the items in the outer set of brackets are key-value pairs.

We can change flags using the STORE command:

C: 0010 STORE 1 +FLAGS (\Deleted)
S: * 1 FETCH (FLAGS (\Deleted \Seen) UID 1)
S: 0010 OK STORE completed

Like FETCH, the STORE command takes a sequence number, or can take a UID if preceded by the UID keyword. The second parameter (+FLAGS) describes the action to be taken. +FLAGS adds the flags in the last parameter; FLAGS replaces all flags with those in the last parameter; and -FLAGS removes the flags in the last parameter. For example:

C: 0011 STORE 2 FLAGS (\Draft \Flagged \Seen)
S: * 2 FETCH (FLAGS (\Flagged \Draft \Seen) UID 2)
S: 0011 OK STORE completed
C: 0012 STORE 1 -FLAGS (\Seen)
S: * 1 FETCH (FLAGS (\Deleted) UID 1)
S: 0012 OK STORE completed

So now the message flags look like this:

C: 0013 FETCH 1:* (UID FLAGS)
S: * 1 FETCH (UID 1 FLAGS (\Deleted))
S: * 2 FETCH (UID 2 FLAGS (\Draft \Flagged \Seen))
S: * 3 FETCH (UID 3 FLAGS ())
S: 0013 OK FETCH completed

When you want to delete messages marked with the \Deleted flag, just send an EXPUNGE or CLOSE command:

C: 0014 EXPUNGE
S: * 1 EXPUNGE
S: 0014 OK EXPUNGE Completed

The response gives you sequence numbers of deleted messages. These may include duplicates, because as messages are deleted, sequence numbers are decreased as explained earlier.

That's all for today! This article explained the various metadata associated with IMAP folders and email messages. It explained the difference between UIDs and sequence numbers, and how to work with flags among other things. I hope you'll come back to learn more! :)

IMAP: Working with Folders

Greetings!

In yesterday's article, "IMAP: Downloading emails", we saw how to log into an email account using IMAP, select a folder (in this case the INBOX), and download an email. The INBOX is the only folder that is standard in IMAP and always exists, even if empty.

In IMAP, however, the INBOX does not need to be the only folder. RFC3501, which to date defines the latest IMAP standard called IMAP4rev1 (which you no doubt noticed in CAPABILITY response in yesterday's article), provides several commands used for working with folders.

You can create folders using the CREATE command, which is used as follows:

CREATE foldername

Use this command to create a few folders we can work with:

You can now view a list of all folders using the LIST command, as follows:

02 LIST "" "*"

The parameters to LIST allow you to list a subset of folders rather than all of them, but that's not important at this stage. The command as above will give you the full list:

You'll notice the Trash and Sent folders which we didn't create. These folders were actually created by Thunderbird when I set up my account in it. The desktop clients create certain folders used to store sent emails, deleted emails, etc. These folders aren't standard, and in fact clients don't always use the same conventions (e.g. Thunderbird uses a Sent folder, while Outlook creates a Sent Items folder instead).

Aside from the actual folder name, each line of the LIST response gives you some additional information about the folder. In this case it's telling us that the folders don't have any children, and what the hierarchy delimiter is (in this case the dot). This is because folders can actually have subfolders, and the hierarchy delimiter separates each folder from its child (e.g. folder.subfolder if the dot is the hierarchy delimiter). There's actually a special version of the LIST command that is used specifically to retrieve the hierarchy delimiter:

C: 0006 LIST "" ""
S: * LIST (\Noselect) "." ""

Note that not all servers necessarily use the dot as the hierarchy delimiter. In hMailServer, for example, it is possible to change it:

Let's try creating a folder with a subfolder:

C: 0007 CREATE Code.PHP
S: 0007 OK CREATE Completed

If you open Thunderbird, you can see the folder hierarchy. First, though, you'll need to right click on the account to the left and select Subscribe...:

Yup, because in IMAP there's this thing called folder subscription. You might have hundreds of folders, but you might only be interested in a few of them. So aside from the LIST command, there's the LSUB command which only lists subscribed folders:

C: 0008 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: 0008 OK LSUB completed

Notice how this list is shorter than the one we got with the LIST command. We can subscribe folders using the SUBSCRIBE command (which is what happens if you tick those checkboxes in Thunderbird):

C: 0009 SUBSCRIBE Personal
S: 0009 OK Subscribe completed

If we repeat the LSUB command, the list is now updated:

C: 0010 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: * LSUB (\HasNoChildren) "." "Personal"
S: 0010 OK LSUB completed

To unsubscribe a folder, just use the UNSUBSCRIBE command:

C: 0011 UNSUBSCRIBE Personal
S: 0011 OK Unsubscribe completed

If you try the LSUB now, it will give you the same result as before.

You can delete a folder completely using the DELETE command:

C: 0013 DELETE Work
S: 0013 OK Delete completed

What are you looking at... everyone wants to delete work, no? :)

Finally, if you want to work with the emails in a folder (download them, flag them, mark them as unread, etc), then you'll have to select the folder. We've already seen the SELECT command in yesterday's article:

C: 0014 SELECT Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0014 OK [READ-WRITE] SELECT completed

We'll learn about the details of this response in another article. There is, however, another command that can be used to select folders: the EXAMINE command. Let's try it out:

C: 0018 EXAMINE Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS ()] limited
S: 0018 OK [READ-ONLY] EXAMINE completed

You'll notice that the response is mostly the same, except for the last two lines. The important thing to notice is that the SELECT command results in a READ-WRITE attribute in the last line, while the EXAMINE command gives you a READ-ONLY. That's precisely the difference between SELECT and EXAMINE: they essentially do the same thing, but EXAMINE selects the folder in read-only mode and doesn't let you make any changes.

Nice! Today we learned how to work with folders in IMAP. You can select a folder to work with using the SELECT (read-write) or EXAMINE (read-only) commands. Folders can be created with the CREATE command and deleted with the DELETE command (duh). Subfolders may be created by appending the hierarchy delimiter (e.g. ".") followed by the subfolder name to the parent folder name. All folders may be shown using the LIST command, while the LSUB command lists only subscribed folders. Folder subscription may be toggled using the SUBSCRIBE and UNSUBSCRIBE commands.

I hope this was interesting, and that you'll come back to learn more about IMAP in the upcoming articles! :)