Email servers are probably about the most complicated servers to set up because not only is there a huge number of security implications to running your own mail server but they also tend to be split up into many small parts taht all do different, highly specialized, things. This page aims to give you an overview of how and email system works. I don't pretend to be an expert on these things but this should be a good starting point.
At the pointy end of the system is the MTA which stands for mail transport agent. This is the bit that does the grunt work of moving email around on the Internet. Probably the most popular MTA is sendmail but there are plenty of others to choose from including courier-mta, postfix and qmail as well as pay for offerings. The MTA probably has the most security issues as it is world facing. There used to be problems (and to some extent there still is) with open relays. These are badly configured MTAs that allow anyone to connect and send email through them. Many modern MTAs make it deliberately quite difficult to configure an open realy.
The next step down is the MDA or mail delivery agent. This has the job of deliverying the email to the user. Typically this means placing the email in a mbox or maildir. A popular MDA is maildrop but, as with MTAs, plenty of other exist (for instance procmail). As well as deliverying the mail the MDA often filters the mail as well. This can be just a simple set of rules for putting certain email in certain folders or it can be as complex as intergrating multiple external mail scanners that check to spam and viruses.
Depending on the set up the next part of the puzzle is optional but almost always present and is either an IMAP or POP server. If you are setting up your own system I strongly recommend an IMAP server as it means that you will be able to easily check your mail from anywhere in the world. Again there are loads of IMAP servers but they vary in quality. I use the Courier IMAP server and have found it to be quite reliable if a little slow. The IMAP server essentailly owns your mailbox (I use mailbox here not to refere to the mbox file format but a generic place that holds you mail) and provides a view of it to the next layer of software, the MUA.
The MUA is the bottom rung of the ladder and stands for mail user agent. This is the application that you actually read you email with. The reason that the layer above is optional is because there are a few MUAs that can directly read mbox format files (there might be some that can read maildirs as well). These are rare though and I wouldn't recommend using them. It is much better to get a decent MUA (such as Thunderbird, KMail, Evolution, etc, etc) that can interface with and IMAP server.