Gigi Labs

Please follow Gigi Labs for the latest articles.

Saturday, August 31, 2013

SDL2: Empty Window

Hi all! :)

[Update 2015-11-14: This article is out of date. Check out the latest version at Gigi Labs.]

Yesterday's article dealt with setting up SDL2 in Visual Studio. Today we're going to continue what we did there by showing an empty window and allowing the user to exit by pressing the X at the top-right of the window. This is very similar to the "SDL Quickstart for Linux: Empty Window" article I wrote almost three years ago on my other blog; however this article is for Windows and deals with SDL2, rather than SDL1.2.x.

It takes very little to show an empty window. Use the following code:

#include <SDL2/SDL.h>

int main(int argc, char ** argv)
{
  SDL_Init(SDL_INIT_VIDEO);

SDL_Window * screen = SDL_CreateWindow("My SDL Empty Window", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, 640, 480, 0);

  SDL_Quit();

  return 0;
}

We use SDL_Init() to initialise SDL, and tell it which subsystems we need - in this case video is enough. At the end, we use SDL_Quit() to clean up. It is possible to set up SDL_Quit with atexit(), as the SDL_Quit() documentation shows.

We create a window using SDL_CreateWindow(). This is quite different from how we used to do it in SDL 1.2.x. We pass it the window caption, initial coordinates where to put the window (not important in our case), window width and height, and flags (e.g. fullscreen).

If you try and run the code, it will work, but the window will flash for half a second and then disappear. You can put a call to SDL_Delay() to make it persist for a certain number of milliseconds:

  SDL_Delay(3000);

Now, let's make the window actually remain until it is closed. Use the following code:

#include <SDL2/SDL.h>

int main(int argc, char ** argv)
{
  bool quit = false;
  SDL_Event event;

  SDL_Init(SDL_INIT_VIDEO);

  SDL_Window * screen = SDL_CreateWindow("My SDL Empty Window", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, 640, 480, 0);

  while (!quit)
  {
  SDL_WaitEvent(&event);

  switch(event.type)
  {
  case SDL_QUIT:
  quit = true;
  break;
  }
  }

  SDL_Quit();

  return 0;
}

The while (!quit) part is very typical in games and is in fact called a game loop. We basically loop forever, until the conditions necessary for quitting occur.

We use SDL_WaitEvent() to wait for an event (e.g. keypress) to happen, and we pass a reference to an SDL_Event structure. Another possibility is to use SDL_PollEvent(), which checks continuously for events and consumes a lot of CPU cycles (SDL_WaitEvent() basically just sleeps until an event occurs, so it's much more lightweight).

The event type gives you an idea of what happened. It could be a key press, mouse wheel movement, touch interaction, etc. In our case we're interested in the SDL_QUIT event type, which means the user clicked the window's top-right X button to close it.

We can now run this code, and the window remains until you close it:


Wasn't that easy? You can use this as a starting point to start drawing stuff in your window. Have fun, and come back again for more tutorials! :)

Friday, August 30, 2013

SDL2: Setting up SDL2 in Visual Studio 2010

Hi folks!

[Update 2015-11-14: This article is out of date. Check out the latest version at Gigi Labs.]

Yesterday I realised that earlier this month, SDL2 has been released. SDL is a fantastic library for cross-platform game development, and I had used SDL 1.2.x for several projects ranging from my Picaxo Image Viewer to early iterations of Ultima 1 Revenge, which I mentioned in yesterday's article. I had also written an article on setting up SDL 1.2.x in Linux on my other blog. SDL2 apparently brings about many improvements we were yearning for, including support for multiple windows.

This article is a tutorial on how to set up a Visual Studio 2010 project to work with SDL2. SDL2 is a C library, so SDL2 projects are normally written in C/C++, although it is possible to use other languages. This article is just about setting things up, so you don't really need to know anything. But for future SDL2 tutorials, some C/C++ knowledge will be expected.

Before we begin, you'll need to grab some files from the SDL2 download page. You'll need the Windows development libraries for Visual C++, and you'll also need the Windows runtime binaries (we'll use the x86 ones by default, but you might need the x64 ones if you eventually want to compile 64-bit versions of your game).

Extract the contents of the development libraries. You should have an include folder, a lib folder, and a few loose files. Locate the following directory, where Visual Studio keeps its header files and libraries for C++:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A

In C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include, create a folder called SDL2 and extract the contents of the development libraries' include folder there:


Next, copy the .lib files from the development libraries' lib\x86 folder into C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Lib:


...and copy the .lib files from the development libraries' lib\x64 folder into C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Lib\x64:


This will allow Visual Studio to find the header foles and .lib's without any additional configuration. Alternatively, you can put them somewhere else and configure the paths accordingly (this article explains how), but I find the above method easier.

Next, create a project in Visual Studio. You need to select Empty Project, from Visual C++ -> General:


In Solution Explorer, right click on Source Files and add a new item. Select C++ File and name it main.cpp:



Still in Solution Explorer, right click on the project (not the solution) and select Properties.

In the treeview to the left, go to Configuration Properties -> Linker -> Input. In the field called Additional Dependencies, replace all the default crap with the following:

SDL2.lib;SDL2main.lib


Next, in Linker -> System, change SubSystem to Windows:


Now, all we need is some minimal code to compile and test. I used the following, based on TwinklebearDev's tutorial (though note the different include):

#include <SDL2/SDL.h>

int main(int argc, char ** argv)
{
    SDL_Init(SDL_INIT_EVERYTHING);
    SDL_Quit();

    return 0;
}

The project should compile, but when you try to run it, you get this:


That's because SDL2 executables need to have the SDL2.dll file in the same folder. Remember the runtime binaries you downloaded earlier? This is where they come in. Grab the SDL2.dll from the x86 runtime binaries package and put it in your project's Debug folder, where the executable is produced. If you try running it now, it should work.

Great, so it compiles and runs, but doesn't do anything. Given that there are no errors, it works, and you can use this as a starting point for further development. TwinklebearDev's tutorial even describes how to export a template from this project, so you don't need to do the same configuration every time.

I hope you found this useful, and check back for more articles in future! :)

Thursday, August 29, 2013

Ultima 1 Reverse Engineering: Decoding Savegame Files

Hey all! :)

It's no secret that I'm a long-time fan of the Ultima series of games. My most significant contribution to the Ultima fan community was running Dino's Ultima Page, a news and information hub for the community, for almost ten years.

Ultima insipired such a great following that hundreds of fan-made remakes, tools and other projects appeared over the years. Many remakes were intended to be mods of games such as Dungeon Siege or Neverwinter Nights. Among other things, talented developers found ways to understand the data files of the games (including the graphics, savegames, maps, etc). This resulted in several useful tools (savegame editors, map viewers, etc), but more importantly, it allowed people to create new game engines using the original data files, with the intention of allowing the games to be played on modern operating systems. The first of these was Exult (an Ultima 7 engine), but over the years, projects appeared for most Ultimas - such as Pentagram for Ultima 8, or Nuvie for Ultima 6.

This article describes the process of reverse engineering, i.e. how to try to make sense of data files given the data files and nothing else. We will be looking at the savegame format of Ultima 1. Although Ultima 1 is not free, you can get the entire first Ultima trilogy from Good Old Games for a few bucks. Ultima 1 is a great choice to start reverse engineering because it's relatively simple - the savegame file is only 820 bytes long and is uncompressed. This made sense for me as when I started this project I was still a budding programmer, and it also made sense because there was very little knowledge about the U1 formats around, whereas the later games had been studied in depth. Reverse engineering is a bit of an advanced topic so feel free to skip it if you feel lost, but it's also a very interesting topic.

So first thing you need to do is install Ultima 1. If you got it off Good Old Games, it conveniently comes with DOSBox, allowing you to play it under modern operating systems:


After launching the game, you will find yourself in the main menu.


Press 'a' to go through the character creation process.


Once you have selected your attributes and saved your character, you find yourself back in the main menu. In the Ultima 1 folder, you should notice a new file called PLAYER1.U1:


That's the savegame file we'll be messing around with. You can use a hex editor to take a glimpse of its contents. Personally I like XVI32 because it's pretty lightweight and even allows you to edit hex entries.


This might look like gibberish, but you can already notice a few things. The first 14 bytes in the savegame file are reserved for the player's name. Knowing the stats you chose during character creation (strength 32, agility 32, stamina 15, charisma 12, wisdom 11 and intelligence 13), you can also spot them in hex on the second line (20, 20, 0F, 0C, 0B, 0D). Windows Calculator has a Programmer mode that is quite useful for switching between decimal and hex:



A good way of decoding more parts of the savegame file is interacting with the game itself. Start the game. The world view looks like this:


Keep a copy of your savegame file at this point. If you move around a bit and save the game, you should at the very least observe changes in your character's X- and Y-coordinates:


You can manually check which bytes in the savegame file changed. I found it more useful to write a hex diff tool that actually highlights the bytes that changed:


As you can see, it's not so simple: there are many things that might change, including food. However, you can choose your moves carefully (e.g. 2 steps west, 3 steps north) so that you can then spot which bytes have changed that much and determine which are the world coordinates.

Another way of learning more about the savegame file is by actually tampering with it. Using XVI32, I changed the byte after '96' from 00 to 13:


After running the game, note how the Hits shot up from 150 to 5014:


That makes a bit of sense: the 96 (hex) we saw earlier corresponds to 150 in decimal - the original value of Hits. But why did it become 5014 when we tweaked the byte after it?

It's because DOS games like this stored values as 16-bit integers in little endian format (i.e. the bigger byte is the second one). So if we have the value we tweaked, i.e. 96 13, that's actually (13 * 100) + 96 (all hex), which results in 5014 (decimal).

Isn't that neat? Reverse engineering requires a lot of time and patience, but it's a bit like fitting together the pieces of a jigsaw puzzle. After a while you might end up understanding a good chunk of the data files:


Once you understand the data files (which also includes map and graphics files), you can then proceed to write all sorts of tools and stuff. I had called this project U1Revenge (Ultima 1 Reverse Engineering Effort) and wrote a map viewer and was working on an engine for it. Although I stopped working on it last year, I did release a couple of demos, the latest of which you can grab from the project page.


Reverse engineering is certainly not a new art. The book Masters of Doom describes how fans of DOOM would hack the game's map files to create their own level editors. Many games have similarly been studied, and a wealth of knowledge is available today. Reverse engineering is not just an achievement; it is a glimpse of history, and helps to understand how games were created even before we were born. The following links provide further reading:

Monday, August 19, 2013

IMAP: Message and Folder Attributes

Hello!

In the last article, "IMAP: Working with Folders", you may have noticed that hMailServer returns some data about folders you select. For example:

C: 0002 SELECT INBOX
S: * 3 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376229395] current uidvalidity
S: * OK [UNSEEN 2] unseen messages
S: * OK [UIDNEXT 4] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0002 OK [READ-WRITE] SELECT completed

In this article we'll cover what these items mean, and learn how messages are organised in IMAP folders.

Let's start off with the UIDVALIDITY. The UIDVALIDITY is a 32-bit value associated with a folder when it is created. Each time a folder is selected, this UIDVALIDITY value is returned (as above). Normally, this value doesn't change. If it does, it means that the folder somehow got messed up on the server (it might also have been recreated with the same name). IMAP clients check this UIDVALIDITY, and if it changes, they are supposed to discard the messages they downloaded for that folder and download the entire folder again from scratch.

Email messages in a folder are identified by two different means: unique identifiers (UIDs) and sequence numbers. These are both 32-bit numbers but they work a bit differently. Let's say you have five messages in your INBOX folder:

Seq Number
1
2
3
4
5
UID
1
2
3
4
5

Initially, the sequence numbers and UIDs are the same. But look at what happens when the third message gets deleted:

Seq Number
1
2
3
4
UID
1
2
4
5

See, when a message gets deleted, the sequence numbers are reassigned to fill in the gap (sequence numbers above 3 are deducted by one in this case). So if there are n messages in a folder, sequence numbers run continuously from 1 to n without any gaps. On the other hand, message UIDs never change. If a message is deleted, its UID vanishes with it.

We have already seen the use of these identifiers in the FETCH command in the recent article "IMAP: Downloading emails". In a FETCH command like this...

0004 FETCH 1 BODY[]

...the '1' represents the sequence number of the message you want to retrieve. You can use UIDs instead, by using a UID FETCH command instead:

0005 UID FETCH 1 BODY[]

Back to the SELECT response at the beginning of this article, some things should now be clear. The EXISTS part tells you how many emails are in the folder - it's also the value of the highest sequence number available. UIDNEXT is a value higher than the highest UID in the folder, predicted but not required to be assigned when a new message is added in the folder. Section 2.3.1.1 of RFC3501 explains it pretty well:

   The next unique identifier value is the predicted value that will be
   assigned to a new message in the mailbox.  Unless the unique
   identifier validity also changes (see below), the next unique
   identifier value MUST have the following two characteristics.  First,
   the next unique identifier value MUST NOT change unless new messages
   are added to the mailbox; and second, the next unique identifier
   value MUST change whenever new messages are added to the mailbox,
   even if those new messages are subsequently expunged.

        Note: The next unique identifier value is intended to
        provide a means for a client to determine whether any
        messages have been delivered to the mailbox since the
        previous time it checked this value.  It is not intended to
        provide any guarantee that any message will have this
        unique identifier.  A client can only assume, at the time
        that it obtains the next unique identifier value, that
        messages arriving after that time will have a UID greater
        than or equal to that value.

Now, let's talk about flags. A message can be assigned a set of flags. On most servers, the flags are predefined and are the following:

  • \Seen - if set, message is marked as read
  • \Answered - if set, message is marked as replied to
  • \Flagged - if set, message has a special status (e.g. flagged/starred/etc)
  • \Deleted - more on this in a second
  • \Draft - if set, message is marked as a draft; clients usually just create a folder for drafts and don't bother with this
  • \Recent - messages that have been added to a folder are marked as recent; this status is removed once the folder is selected again later
You'll notice that some of the functionality you're used to, such as read/unread messages and messages being marked as answered in Outlook are supported in IMAP by flags. The \Deleted flag, however, deserves some special attention.

In IMAP, there is no Recycle Bin or Trash folder. Clients emulate Recycle Bin functionality by creating a Deleted Items folder (actual name varies between clients). When messages are deleted from another folder, they are moved to the Deleted Items folder. When they are deleted from the Deleted Items folder, they are deleted permanently.

Messages are deleted in IMAP in two stages. First, they are marked for deletion by setting the \Deleted flag. Then, an EXPUNGE command clears the entire folder of messages marked for deletion. Alternatively, a CLOSE command does the same as the EXPUNGE command but also deselects the folder.


Servers may optionally allow custom flags to be set. These are called keywords; they work like tags and don't start with a backslash (\). Servers that support keywords return a \* as part of the PERMANENTFLAGS line in the SELECT response - you can see this in Gmail:

S: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Flags permitted.

Now that I've explained flags, we can understand the remainder of the SELECT response. The RECENT count shows how many messages are marked with the \Recent flag - this may help to indicate any new messages in the folder, although using UIDNEXT is more reliable. The UNSEEN line (optional) gives the sequence number of the first message in the folder that is not marked with the \Seen flag.

The FLAGS line tells you what flags are supported for the selected folder, and the PERMANENTFLAGS tells you which flags can be modified. If any flags are in FLAGS but not in PERMANENTFLAGS, then they can only be modified temporarily; the old value is seen when a new session is initiated.

Any data about an email message can be retrieved using the FETCH command, and that includes UIDs and flags:

C: 0009 FETCH 1:* (UID FLAGS)
S: * 1 FETCH (UID 1 FLAGS (\Seen))
S: * 2 FETCH (UID 2 FLAGS ())
S: * 3 FETCH (UID 3 FLAGS ())
S: 0009 OK FETCH completed

This FETCH command is similar to the ones we used before. Instead of a single number, we specified 1:*, which means all messages in the range from 1 to *. The * unintuitively resolves to the highest sequence number in the folder, in this case 3.

The BODY[] field we were using earlier is substituted for (UID FLAGS) here. Although we could fetch UID or FLAGS individually (with or without brackets), we can specify several fields at once using an IMAP list - a bracketed space-delimited sequence of words. In the response lines, the numbers on the left (beside the asterisks) are the sequence numbers, and the items in the outer set of brackets are key-value pairs.

We can change flags using the STORE command:

C: 0010 STORE 1 +FLAGS (\Deleted)
S: * 1 FETCH (FLAGS (\Deleted \Seen) UID 1)
S: 0010 OK STORE completed

Like FETCH, the STORE command takes a sequence number, or can take a UID if preceded by the UID keyword. The second parameter (+FLAGS) describes the action to be taken. +FLAGS adds the flags in the last parameter; FLAGS replaces all flags with those in the last parameter; and -FLAGS removes the flags in the last parameter. For example:

C: 0011 STORE 2 FLAGS (\Draft \Flagged \Seen)
S: * 2 FETCH (FLAGS (\Flagged \Draft \Seen) UID 2)
S: 0011 OK STORE completed
C: 0012 STORE 1 -FLAGS (\Seen)
S: * 1 FETCH (FLAGS (\Deleted) UID 1)
S: 0012 OK STORE completed

So now the message flags look like this:

C: 0013 FETCH 1:* (UID FLAGS)
S: * 1 FETCH (UID 1 FLAGS (\Deleted))
S: * 2 FETCH (UID 2 FLAGS (\Draft \Flagged \Seen))
S: * 3 FETCH (UID 3 FLAGS ())
S: 0013 OK FETCH completed

When you want to delete messages marked with the \Deleted flag, just send an EXPUNGE or CLOSE command:

C: 0014 EXPUNGE
S: * 1 EXPUNGE
S: 0014 OK EXPUNGE Completed

The response gives you sequence numbers of deleted messages. These may include duplicates, because as messages are deleted, sequence numbers are decreased as explained earlier.

That's all for today! This article explained the various metadata associated with IMAP folders and email messages. It explained the difference between UIDs and sequence numbers, and how to work with flags among other things. I hope you'll come back to learn more! :)

Tuesday, August 13, 2013

IMAP: Working with Folders

Greetings!

In yesterday's article, "IMAP: Downloading emails", we saw how to log into an email account using IMAP, select a folder (in this case the INBOX), and download an email. The INBOX is the only folder that is standard in IMAP and always exists, even if empty.

In IMAP, however, the INBOX does not need to be the only folder. RFC3501, which to date defines the latest IMAP standard called IMAP4rev1 (which you no doubt noticed in CAPABILITY response in yesterday's article), provides several commands used for working with folders.

You can create folders using the CREATE command, which is used as follows:

CREATE foldername

Use this command to create a few folders we can work with:


You can now view a list of all folders using the LIST command, as follows:

02 LIST "" "*"

The parameters to LIST allow you to list a subset of folders rather than all of them, but that's not important at this stage. The command as above will give you the full list:


You'll notice the Trash and Sent folders which we didn't create. These folders were actually created by Thunderbird when I set up my account in it. The desktop clients create certain folders used to store sent emails, deleted emails, etc. These folders aren't standard, and in fact clients don't always use the same conventions (e.g. Thunderbird uses a Sent folder, while Outlook creates a Sent Items folder instead).

Aside from the actual folder name, each line of the LIST response gives you some additional information about the folder. In this case it's telling us that the folders don't have any children, and what the hierarchy delimiter is (in this case the dot). This is because folders can actually have subfolders, and the hierarchy delimiter separates each folder from its child (e.g. folder.subfolder if the dot is the hierarchy delimiter). There's actually a special version of the LIST command that is used specifically to retrieve the hierarchy delimiter:

C: 0006 LIST "" ""
S: * LIST (\Noselect) "." ""

Note that not all servers necessarily use the dot as the hierarchy delimiter. In hMailServer, for example, it is possible to change it:


Let's try creating a folder with a subfolder:

C: 0007 CREATE Code.PHP
S: 0007 OK CREATE Completed

If you open Thunderbird, you can see the folder hierarchy. First, though, you'll need to right click on the account to the left and select Subscribe...:


Yup, because in IMAP there's this thing called folder subscription. You might have hundreds of folders, but you might only be interested in a few of them. So aside from the LIST command, there's the LSUB command which only lists subscribed folders:

C: 0008 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: 0008 OK LSUB completed

Notice how this list is shorter than the one we got with the LIST command. We can subscribe folders using the SUBSCRIBE command (which is what happens if you tick those checkboxes in Thunderbird):

C: 0009 SUBSCRIBE Personal
S: 0009 OK Subscribe completed

If we repeat the LSUB command, the list is now updated:

C: 0010 LSUB "" "*"
S: * LSUB (\HasNoChildren) "." "INBOX"
S: * LSUB (\HasNoChildren) "." "Trash"
S: * LSUB (\HasNoChildren) "." "Sent"
S: * LSUB (\HasNoChildren) "." "Personal"
S: 0010 OK LSUB completed

To unsubscribe a folder, just use the UNSUBSCRIBE command:

C: 0011 UNSUBSCRIBE Personal
S: 0011 OK Unsubscribe completed

If you try the LSUB now, it will give you the same result as before.

You can delete a folder completely using the DELETE command:

C: 0013 DELETE Work
S: 0013 OK Delete completed

What are you looking at... everyone wants to delete work, no? :)

Finally, if you want to work with the emails in a folder (download them, flag them, mark them as unread, etc), then you'll have to select the folder. We've already seen the SELECT command in yesterday's article:

C: 0014 SELECT Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS (\Deleted \Seen \Draft \Answered \Flagged)] limited
S: 0014 OK [READ-WRITE] SELECT completed

We'll learn about the details of this response in another article. There is, however, another command that can be used to select folders: the EXAMINE command. Let's try it out:

C: 0018 EXAMINE Personal
S: * 0 EXISTS
S: * 0 RECENT
S: * FLAGS (\Deleted \Seen \Draft \Answered \Flagged)
S: * OK [UIDVALIDITY 1376393234] current uidvalidity
S: * OK [UIDNEXT 1] next uid
S: * OK [PERMANENTFLAGS ()] limited
S: 0018 OK [READ-ONLY] EXAMINE completed

You'll notice that the response is mostly the same, except for the last two lines. The important thing to notice is that the SELECT command results in a READ-WRITE attribute in the last line, while the EXAMINE command gives you a READ-ONLY. That's precisely the difference between SELECT and EXAMINE: they essentially do the same thing, but EXAMINE selects the folder in read-only mode and doesn't let you make any changes.

Nice! Today we learned how to work with folders in IMAP. You can select a folder to work with using the SELECT (read-write) or EXAMINE (read-only) commands. Folders can be created with the CREATE command and deleted with the DELETE command (duh). Subfolders may be created by appending the hierarchy delimiter (e.g. ".") followed by the subfolder name to the parent folder name. All folders may be shown using the LIST command, while the LSUB command lists only subscribed folders. Folder subscription may be toggled using the SUBSCRIBE and UNSUBSCRIBE commands.

I hope this was interesting, and that you'll come back to learn more about IMAP in the upcoming articles! :)

Monday, August 12, 2013

IMAP: Downloading emails

Hello friends!

In yesterday's article, "Email: Protocols and Background", I explained how to set up an email client and server to study the email protocols, and also how to use Wireshark to observe how the clients and servers actually talk to each other using these protocols.

Today we'll begin learning IMAP, and use it to access our inbox and download emails.

As I explained in yesterday's article, IMAP is a protocol made up of text commands. You can open a socket (see "C# Network Programming: Simple HTTP Client"), send commands, and interpret the responses. Without needing to write any code, you can use the telnet program to open a raw connection and work using the command line. Connect to your hMailServer IMAP server using a command like this:

telnet serverIP 143

Obviously, replace serverIP with the actual IP address or hostname of the computer where hMailServer is installed. 143 is the port on which the IMAP server listens by default.

If telnet is not installed, you can install it on Windows 7 as follows. In the Start search box, find Programs and Features and open it. Select "Turn Windows features on or off" (it's a link on the left hand side). In the Windows Features window, tick the "Telnet Client" checkbox and click OK.


With telnet installed, connect to the IMAP server as described above. You should see the initial greeting from the hMailServer IMAP server:

* OK IMAPrev1

Now, type the following command and press ENTER:

01 capability

The response consists of a number of words that describe what functionality the IMAP server supports:


Next, login to the email account you created as part of yesterday's article. In my case the command looks like this:

02 login user@ranchtest.local pass

The response will tell you whether the login was successful or not.

In order to make this a little interesting, send a few emails to yourself via Thunderbird so you actually have something in your inbox. Next, access the inbox using the following command. The INBOX folder is standard in IMAP and always exists, even if you don't have anything in it.

03 select INBOX


You should be able to see a few things at this stage. First, whenever we send a command, we precede it with something like "01". This is called a tag, and can be any string (different clients use different formats ranging from numbers to random strings). When the server response to a command, the last line always starts with the same tag as the command - that way a client knows that the response for that command was received.

You'll probably also realise that telnet is a pain in the ass. If you make mistakes you can't backspace - that's because telnet sends everything you type, byte by byte. So each character is sent immediately and can't be undone. Last year I wrote a program called IMAPTalk which makes working with IMAP (and other protocols) much more convenient. Just download it and run it - no installation necessary. Just enter the hostname and connect:


If you turn on Auto-generate tags, you can actually leave out the tags and just type the commands - the tags will be filled in for you by IMAPTalk. If you repeat the commands we did above in IMAPTalk, it looks like this:


Better, no? If you look at the responses (in red), you'll notice there are a bunch of things I haven't explained yet. Don't worry about them just yet. The important thing you should take from the response to the SELECT command is this like:

* 3 EXISTS

That's telling us that there are 3 messages in the INBOX.

We can then retrieve them one by one using a FETCH command, while providing the message number:

04 fetch 1 BODY[]

The response is as follows:


You'll notice that there is a bunch of stuff in there, for such a short message. Part of it (the line starting with the * 1 FETCH) is IMAP, as are the last two lines. The stuff in between is the full email, known as the MIME. It consists of a header with a bunch of fields and values (you'll notice important stuff such as From, To, Date, etc), and after a double line break, the message itself. You can see this in Gmail by clicking on the arrow at the top-right of an opened email and clicking "Show original".

You can similarly retrieve the other messages in the inbox by replacing the 1 in the FETCH command with the message number. This is called the message sequence number, and can't be larger than the number of emails in the folder (in this case 3). If you provide a larger number you won't get an error, but you won't get any email data either.

So that's the easiest way to download emails manually using IMAP. We'll learn more about how messages are stored in IMAP folders, and more about the facilities offered by the IMAP protocol, in the coming articles. Come back again for more!

Sunday, August 11, 2013

Email: Protocols and Background

Hullo!

You sure as hell know what email is, but do you know how it actually works? In today's article I'll talk about how it works, and how best to learn about it in detail.

One of my previous articles, "HTTP Requests in Wireshark", showed that the World Wide Web was based on a standard protocol, HTTP. Web browsers and web servers could talk to each other using this protocol.

It turns out that email is no different. Email clients (e.g. Outlook, Thunderbird, etc) and servers (e.g. Microsoft Exchange, hMailServer, etc) can talk to each other using three standard protocols: IMAP, POP and SMTP. SMTP is used for sending email, while IMAP and POP are used for retrieving email from a server. IMAP is more powerful and complex than POP; POP doesn't even support folders. Each of these protocols uses a set of text commands similar to stuff you might have used in the command line, except that in this case you do it over the network (you'll see how in a minute). The protocols also define ports on which servers should listen, e.g. IMAP uses ports 143 and 993 by default.

You could study these protocols by just reading their respective RFCs, but you can only really learn them properly by seeing them in action. To do this, you will need some software.

First, install a mail server. hMailServer is decent and free, so unless you have a better alternative, go with it. Ideally install this on another computer, otherwise you won't be able to capture requests in Wireshark if your client and server are on the same PC. While installing hMailServer, choose SQL Compact as storage (to keep it simple) and give it an admin password.

When you're done installing hMailServer, you need to create an account. Run hMailServer and enter the admin password to access the Administrator console:


Click on the Add domain... button and invent a domain name. By convention it should end in .local, since this isn't a real domain name. Click the Save button to save it.


Next, find the Accounts folder somewhere under the Domains node.


Click the Add... button. Enter a username and password and click Save to create your account. Notice how the domain is used as part of the email address, for example in my case the email address is user@ranchtest.local.


Great. Now, we need to test this account using an email client. Thunderbird is a popular choice and it's free, although there are many others you can use if you want. Install and run it. Skip the shit that appears upon running:


Press Alt to show the program menu, since Thunderbird seem to have fallen into the horrible practice of hiding it. From the Tools menu, click on Account Settings...:


From the drop-down list, select Add Mail Account...:


Enter your basic account credentials, and then click the Manual config button when it appears:


Now you need tell Thunderbird how to connect to your mail server. Under Server hostname, type the name or IP address of the computer where hMailServer is installed. Using the PC name is better since the IP address might change regularly. If you installed hMailServer on the same machine (not recommended, see above) you can use 127.0.0.1 or localhost as the Server hostname. Other than that, use port 143 for IMAP and port 25 for SMTP; no SSL and Normal password for Authentication. It should look like this:


Click the Re-test button to allow Thunderburd to test those settings. If all is ok, the Done button is enabled, click it to save the settings and create the account. When you do this, a warning appears:


It's a security warning because the client and server will talk to each other on an unencrypted channel. Since you're working on a local test environment, you don't have to worry about it. We actually want the messages to be unencrypted, so that we can capture them with Wireshark.

Finally, install Wireshark so that we can take a look at the email protocols. Set it up for capturing; I explained how to do this in my article "HTTP Requests in Wireshark". After starting the capture session, set the Filter to "imap".

Using Thunderbird, send an email to yourself:


The email is sent to the server via SMTP, and then downloaded to the client via IMAP. After a second you should see some activity in Wireshark. From Thunderbird, click Get Mail to retrieve the new email in your inbox.


In Wireshark, you can see the IMAP commands as well as the email data. You can view this conveniently by right clicking on one of the packets and selecting Follow TCP Stream:


This might seem a bit cryptic, but it's quite easy to learn. In future articles I'll cover the email protocols bit by bit. So check back later for more about email technology! :)

Friday, August 9, 2013

C# OOP: Encapsulation and Properties

Hi everyone!

Some recent articles have discussed different aspects of Object Oriented Programming. Today we'll talk about controlling access to members (variables or methods) of our classes.

Let's begin with a scenario. We have an accounting program that keeps track of a person's money. First, we need to use the following library:

using System.Collections.Generic;

Then we declare a Transaction class to contain the details of each transaction (deposits, withdrawals... basically anything that affects the amount of money in the account):

    class Transaction
    {
        public DateTime date;
        public String detail;
        public float amount;
       
        public Transaction(DateTime date, String detail, float amount)
        {
            this.date = date;
            this.detail = detail;
            this.amount = amount;
        }
    }

We also need an Account class to keep the overall account information, as well as a list of all the transactions in the account:

    class Account
    {
        public String name;
        public float balance;
        public List<Transaction> transactions;
       
        public Account(String name)
        {
            this.name = name;
            this.balance = 0.0f;
            this.transactions = new List<Transaction>();
        }
       
        public void Show()
        {
            Console.WriteLine("Account for {0}", name);
            Console.WriteLine();
           
            foreach (Transaction transaction in transactions)
            {
                Console.WriteLine("{0:yyyy-MM-dd}  {1, -30}  {2}",
                        transaction.date, transaction.detail, transaction.amount.ToString("0.00").PadLeft(8' '));
            }
           
            Console.WriteLine();
            Console.WriteLine("Balance: {0:0.00}"this.balance);
        }
    }

We can now see how this works with some sample code in Main():

            Console.Title = "C# Encapsulation with Accounting";
           
            Transaction trans1 = new Transaction(new DateTime(20130807), "Deposit: My first salary"20.0f);
            Transaction trans2 = new Transaction(new DateTime(20130808), "Withdrawal... need cash", -15.0f);
            Transaction trans3 = new Transaction(new DateTime(20130809), "Deposit: Birthday present"10.0f);
           
            Account account = new Account("Bill Gates");
           
            account.transactions.Add(trans1);
            account.balance += trans1.amount;
            account.transactions.Add(trans2);
            account.balance += trans2.amount;
            account.transactions.Add(trans3);
            account.balance += trans3.amount;

            account.Show();
           
            Console.ReadKey(true);

So here we've created an account for Bill Gates and added three transactions. The Show() method gives us a picture of the account status and history:


That's great. But what if the guy writing the code in Main() forgets to update the account balance? The sum of the transactions won't agree with the value in the balance member variable. We could provide a method to take care of this in Account:

        public void AddTransaction(Transaction transaction)
        {
            transactions.Add(transaction);
            this.balance += transaction.amount;
        }

...and while this is certainly useful, it doesn't really solve the problem. Because while a programmer might use this method, there is nothing to keep him from updating transactions and balance separately, the old way, as illustrated earlier. Even worse, someone might actually tamper with them, potentially removing transactions or modifying the balance into something different.

The one thing we've been doing wrong all along is declaring all member variables and methods in our classes as public. When they're public, it means anyone can touch them. We can restrict access to members by using a protected or private access modifier instead of public (there are others, but they're not important at this stage).

Once we switch the member variables in the Account class to private:

        private String name;
        private float balance;
        private List<Transaction> transactions;

...then the program won't compile:


That's because we're accessing these private member variables directly from within Main(). When a variable is private, it means that only code within the same class can access it. So it's ok for the AddTransaction() method we added earlier to use the transactions member variable, but the code in Main() can't. Instead, the code in Main() must be refactored to use AddTransaction():

            account.AddTransaction(trans1);
            account.AddTransaction(trans2);
            account.AddTransaction(trans3);

With this change, it compiles just as well. So we've just seen how we can prevent users of a class from tampering with its internal state, and yet still do useful stuff with it via the methods it does expose as public. This is called encapsulation, and it effectively means "data hiding". In OOP it is usually the case that our classes provide a public interface by which other objects can use them, but they hide all the rest from public view.

Naturally, it is often the case that other objects want to access member variables for legitimate reasons. In that case, instead of making the member variable public, we add methods allowing access to them:

        public String GetName()
        {
            return this.name;
        }
       
        public void SetName(String value)
        {
            this.name = value;
        }
       
        public float GetBalance()
        {
            return this.balance;
        }

These are called getter and setter methods, because they allow you to retrieve and modify the variable. A setter can be omitted (as with the balance variable) if that variable is meant to be read-only.

Although using getter and setter methods is a great way of using encapsulation in any OOP language, C# provides the use of properties, which do the same thing but are arguably more elegant:

        public String Name
        {
            get
            {
                return this.name;
            }
            set
            {
                this.name = value;
            }
        }
       
        public float Balance
        {
            get
            {
                return this.balance;
            }
        }

Properties are actually converted to getter and setter methods during compilation, but this process is transparent to you (the programmer). Using them in code is pretty easy:

Console.WriteLine("{0} has {1} Euros in his account.", account.Name, account.Balance);

Right, so what about the protected access modifier? It's like private, but also allows subclasses to access a class's members. You should use it sparingly, or not at all. Scott Meyers explains why in his excellent book "Effective C++: 55 Specific Ways to Improve Your Programs and Designs". When you make something public, there are potentially infinitely many places from where it can be accessed and modified. So if you decide to change that variable (e.g. remove it and replace it with a computation), you have to refactor all those places. If something is protected, it can be accessed from any subclass, so there are still porentially infinitely many places from where it can be modified. If you want a subclass to access a variable, your best bet is to encapsulate it using a getter/setter pair or a property, and have the subclass access that.

To see this idea in action, let's remove the Account class's balance variable and replace it with a sum of the transaction amounts:

         public float Balance
        {
            get
            {
                float bal = 0.0f;
               
                foreach (Transaction transaction in this.transactions)
                {
                    bal += transaction.amount;
                }
               
                return bal;
            }
        }

So like this, any other classes accessing the Balance property may continue to do so without needing to refactor anything. In our case we only need to make some small changes in the Account class itself (no need to maintain a separate balance variable any more in the constructor and in AddTransaction(), and use the property instead of the variable in Show()). But if we had to refactor a hundred other classes to remove the balance variable, it would have been a very expensive change.

Cool. So in this article we learned why we should hide certain internal class data from outside code that might tamper with it - a practice called encapsulation. Encapsulation is so important that, together with inheritance and polymorphism, it is known as one of the Three Pillars of OOP. We also learned about access modifiers (public, protected, private), getter/setter methods, as well as properties - something unique to C#.

From these OOP articles you might begin to see what OOP is really about: the complex relationships between objects make designing OOP software quite a challenge. It's also quite fun. :)

I hope you enjoyed this article, and check back for more! :)