At least once a week, a Remedy ticket pops into my queue from a user frantically requesting a restore of missing messages from their mailbox.  Sometimes this is warrented, as both Entourage and Outlook operating in cached mode are somewhat notorious for eating messages. However, a restore is usually overkill, as the messages are usually not as “missing” as the end-user might believe at first blush.  This is an important point, as restoring the contents of an individual mailbox from backup is a non-trivial, time-consuming task.

So how is one to know if a restore is actually warranted?  There are several steps that an Exchange end-user can take to on their end (or that their desktop support personnel could walk them through) to try to recover their missing e-mail before ever having to get their friendly, neighborhood Exchange administrator involved. Here is a handy-dandy checklist which might help.

Deleted Item Recovery

When a message is deleted in Exchange, it does not go away immediately.  First, the message goes to the “Deleted Items” folder, where it remains until such time as that folder is emptied. (Client software can be configured to automatically empty Deleted Items upon logging out.)  If the missing items are still there, great. Just drag them back where they belong. But if the Deleted Items folder has been emptied, all is not lost. When the Deleted Items folder is emptied, the messages within it actually get shuffled into another, hidden folder referred to as a dumpster. Items are retained in the dumpster for a period specified by the Exchange administrators on a database by database basis. (In the case of AEMS, deleted item retention is set for 14 days.)

During this retention window, items can be retrieved from the dumpster using the “Recover Deleted Items” feature found in Outlook and OWA Premium. (Alas, this leaves Mac and Linux users out in the cold until Exchange 2010 is deployed with its lovely cross-browser support for OWA Premium.  Until then, seek a coworker’s PC, or use a Windows Terminal Server or a VM. I know, it isn’t fair, but relief is at least in sight.) In Outlook, highlight the Deleted Items folder, then go to the Tools menu and select “Recover Deleted Items….”  In OWA Premium, click Options, then click “Deleted Items.”

It is important to note that EACH FOLDER in an Exchange mailbox has its own hidden dumpster folder.  The procedure above only works for recovering items that were deleted by way of the Deleted Items folder.  In cases where the user has shift-deleted a message, that message will have bypassed the Deleted Items folder and would be sent directly to the dumpster associated with whatever folder the message was in.  (This also holds true for messages deleted via IMAP or POP, as those protocols generally bypass the Deleted Items folder.)  In that case, it is necessary to use Outlook to access the dumpster for the appropriate folder by selecting the folder in question, then choosing “Recover Deleted Items….” from the Tools menu.  (OWA Premium only exposes the dumpster associated with the Deleted Items folder.)

Make Sure the Messages Are Actually Missing

If the preceding step doesn’t do the trick, it doesn’t hurt to actually verify that messages are really missing.  They may merely be misplaced by a long-forgotten rule or a sloppy click-drag.  The following steps are designed to check for these issues.

Try Different Clients

Try viewing the mailbox multiple ways to rule out a problem with the client.  Especially try using Outlook Web Access (OWA), as this will present the most authoritative look into the mailbox, free from problems introduced by client software.

• If the messages show up in OWA, but do not show up with, for example, Outlook or Entourage, then the messages are actually in the mailbox, but the client software is not displaying them properly, perhaps due to a corrupted local cache. Rebuild your client profile (preferably not in cached mode in the case of Outlook), and all should be right in the world.  Another reason that Outlook might not show the mailbox as it really is might be that you’ve been tinkering with custom views (View -> Current View), which might restrict what messages get shown.
• If messages do not show up in OWA, but do show up on the client, the messages are not actually in the mailbox.  The most common cause for this is that the Outlook profile has been accidentally configured to deliver the mail to a local PST file rather than to a mailbox.  In this case, you are in luck. Simply fix this profile setting, then drag your mail from the PST back up to the server.

Search, Search, Search….

You would be suprised how often it turns out that a user has accidentally dragged their messages off into a different folder without realizing it.  If you know the subject of one of your missing messages, nose around for it using the search feature of your client.  Search the entire mailbox, not just where you think the messages should be.  If you having any PST files (personal folders), don’t forget to check there.

Check Rules and Auto-Archiving

Sometimes a mailbox rule will intercept and shuffle around messages in ways that were not intended. Double check your rules for one which might be doing something unanticipated.  Auto-archiving can also sneak up on you.  If all of your messages from prior to a certain date have suddenly gone AWOL, check to see if Auto-Archive might have tucked them away into an archive file.

Check PST Files

Several ways in which messages might get shoved into PST files have already been mentioned.  Leave no stone unturned.  See if your wayward messages might be lurking locally.  Remember that PST files are local to your specific computer. If you use Outlook on multiple computers, you’ll need to check for PST files on all of them. (And, for the record, Microsoft does not officially support accessing PSTs via a network share. That is bad, bad, bad.)

Okay, Maybe DO You Need a Restore

If you’ve worked your way through all of these scenarios and are still unable to find your missing e-mail, then perhaps a restore is in order.  When you submit your restore request, be sure to provide your best estimate of a date and approximate time when the missing items were actually known to be in your mailbox.  Your Exchange admins will need that info in order to determine from which backup the restore needs to be taken. The dates that you received the messages don’t help here (“All of my mail older than six months is gone!”).  We need to know when the messages were there.  Also keep in mind that backups are generally only kept for a finite amount of time.  They take up storage space.  For AEMS, backups are only retained for two weeks.  If you come to us with a restore request a month after the messages went missing, we can’t help you.

(This is a repeat from my personnal blog, but relevant enough to my AEMS customers that I felt it appropriate to repost here.)

This MS Exchange Team Blog article (as well as this follow-up) go into great detail about the history of this notorious error, but erroneously claims that it only occurs in muli-domain environments.  We have the issue here, and we have a single-domain forest.

The short version is that Exchange Server 2003 SP2 introduced changes in the security model for the delegate-setting process which, combined with updates on the client end, result in the following error when a user attempts to modify delegates:

This error does not occur for all mailboxes, and we’ve found neither rhyme nor reason for when it does, but there is a fix.

Over the years, some Exchange users have learned the hard way that there is a limit to how many mailbox rules they can define.  The limit is based upon the cumulative size of all the rules associated with a mailbox, rather than the number of rules.  Under Exchange 2003, mailbox rules were capped at 32KB.  With Exchange 2007, that cap has doubled to 64KB, which generally translates to about 100-150 rules, depending upon how big they are (even the size of the rule name counts towards this).  You know you’ve hit that cap when Outlook gives you an error such as the following:

One or more rules could not be uploaded to Exchange server and have been deactivated. This could be because some of the parameters are not supported or there is insufficient space to store all your rules.

So, what is an Exchange user to do when confronted by this? There are two things to try:

1. First, try to reduce the total size of your rules. This Microsoft KnowledgeBase article provides excellent tips on how to do so.
2. If that doesn’t do the trick, contact your friendly neighborhood Exchange administrator (for AEMS users, that means going through the Help Desk) and ask him/her to bump up your rules quota. That’s right, Exchange 2007 added the ability for Exchange administrators to modify this quota on a per-user basis*.

“So,” I hear you cry, “given the availability of the second option, why should I bother with the first?”

That is a perfectly valid question, and one for which I can offer an equally valid response: efficiency.  The more mailbox rules you have, the more processing each message in your mailbox must endure. The more rules you have, the more processing and memory overhead is incurred.  Individually, it doesn’t amount to much, but aggregated over thousands of users, it can mean a noticable impact on performance.  Besides, if you have enough rules that you bump into your rules quota, odds are that there is a way to accomplish what you are trying to do more efficiently.

While we are on the subject of server performance, there are two other things that I should point out that can impact performance, not just for you, but for all users on your server:

Ever since I first learned a few months ago that Exchange 2010 will not support Single Copy Clusters (a high-availability model which has served us quite well to date), I’ve been scheming to figure out ways to transition to 2010 in a way that impacts end-users as little as possible, a task made more difficult by the lack of information about upgrade paths which has come out of Microsoft to date. I had even come up with an elaborate transition model built around Standby Continuous Replication (SCR) which would have made the transition almost transparent to end-users, provided that Exchange 2010 supports in-place upgrades from 2007, or that an Exchange 2010 mailbox server can serve as an SCR target for Exchange 2007. Alas, a new webcast from Microsoft has dashed those hopes, as both of those options are explicitly excluded from officially supported upgrade paths.

The Exchange engineers really dropped the ball on this one. Allowing an Exchange 2010 mailbox server to be an SCR target for 2007 would make transitioning quite painless.  Alas, it is not meant to be. (Of course, I understand WHY they couldn’t do this.  Exchange 2010 marks the first time that the ESE database schema for mailbox storage has been changed substantially since the introduction of Exchange. Thus, the 2007 and 2010 mailbox stores are incompatible, so it would do no good to replicate a 2007 database to a 2010 server unless some sort of database format conversion could be done on the fly.)

The supported transition path involves building new Exchange 2010 servers and performing move-mailbox operations on each mailbox, which, fortunately for us, is an easily scripted operation with PowerShell. There is one bright side to this.  Exchange 2010 introduces the concept of “Online Move Mailbox.” In previous versions of Exchange, the move-mailbox operation rendered a mailbox unavailable for the duration of the move operation.  With this enhancement, the mailbox remains continuously available except for a brief interruption when access is handed off from the source server to the destination server.  This online move mailbox feature is supported for moves from Exchange 2007 SP2 (a forthcoming service pack) to Exchange 2010.

One nice feature of Exchange 2007 is the introduction of specific mailbox types, especially room resource mailboxes, which we create manually on AEMS at the request of departments outside of the EID system (since the associated Active Directory account is disabled for resource mailboxes).  Resource mailboxes can be configured to automatically accept meeting requests, and, if configured with the “AllowConflicts” parameter set to “$true”, can automatically accept conflicting appointments.  This can be handy for group out-of-office or vacation calendars.

Unfortunately, when a recurring item is submitted to such a calendar, specific instances which conflict with other items are declined, despite “AllowConflicts” being enabled.  According to Microsoft, this bug is by design. The only workaround is to create discrete one-time items, or to create a single event which spans multiple days.

With the combination of Outlook 2007 and Exchange 2007, several new services have been introduced which are designed to simplify the end-user experience, although there are “gotchas” to be aware of. These features are called AutoDiscover, AutoConfigure, and the Availability Service.  The latter two are built around the first.

AutoDiscover is an XML-based service used by the client to locate and connect to other web-based Exchange services, including AutoConfigure, Availability Service, and Exchange Web Services (a new API for programmatically accessing Exchange).  The client (which can be Outlook 2007, Windows Mobile, the upcoming Entourage EWS, or an EWS-based program or script) passes the user’s e-mail address and credentials to the AutoDiscover service as a SOAP query (after first checking to see if the desired information is available in Active Directory). If the credentials are valid, and depending upon the type of request, AutoDiscover will pass back a SOAP response with the requested information, such as what server the client should be configured to point to, the URL for downloading the Offline Address Book, the URL for Exchange Web Services, or the URL for the Availability Service.

AutoConfiguration is an Outlook 2007 feature which uses the AutoDiscover service to configure the Outlook profile (if the user chooses not to configure their profile manually).  There are several caveats to be aware of:

1. By default, AutoConfiguration will set the Outlook profile to cached mode, which we don’t generally recommend for our clients.  Cached mode and been known to cause major problems such as inconsistent views of shared calendars and mailbox data loss due to corruption of the OST.
2. The upcoming Entourage EWS (which is currently in beta) adds autoconfiguration functionality, but returns the name of a specific domain controller for the directory server (rather than the directory.austin.utexas.edu NetScaler alias).  This is problematic for off-campus users, as the DCs are not routed off-campus.
3. When attempting to perform an autoconfiguration, the client will parse the e-mail address provided by the user and use that to construct an address to query for the AutoDiscover server (assuming that it cannot pull this info from the Service Connection Point in Active Directory).  For example, for username@mydomain.com, the client will look for the AutoDiscover service at autodiscover.mydomain.com. However, suppose that the mydomain.com Exchange server is hosting another SMTP address space, such as for an acquired company.  In this example, the user’s e-mail address is username@otherdomain.com.  The client will look for the AutoDiscover info at autodiscover.otherdomain.com.  For this scenario, either the user should  specify a “mydomain.com” address, or the network admins should add a CNAME entry to DNS pointing autodiscover.otherdomain.com to autodiscover.mydomain.com.

The Availability Service is a SOAP/XML mechanism by which Outlook 2007 retrieves free/busy information from the Exchange server.  The client uses the AutoDiscover service to locate the SOAP URL for the Availability Service, then uses SOAP queries to pull the relevant free/busy information.  This is all well and good except for one fundamental design flaw in Outlook 2007.  For authenticating against the Availability Service, Outlook uses the credentials that were used to login to the workstation, not those provided for mailbox access.  This is problematic for client systems which are not joined to the domain.  The only workaround is to join the client to the domain.  For off-site usage, this generally requires using a VPN service to join the system to the domain and to authenticate, even though the Availability Service itself simply uses web protocols.

Background: About a month ago, we ran into a peculiar situation where store.exe on one of our mailbox servers was repeatedly crashing, with Event ID 9673 being generated in the Application Event Log from MSExchangeIS.  Per KnowledgeBase article 959135, the appropriate fix for this was to apply the recently-released Rollup 7 for Exchange 2007 SP1.  That didn’t fix the issue, but instead the crash rate increased.  After a marathon consultation with Microsoft Premier Support, we discovered that, in addition to the installation of the rollup, it was necessary to create a registry key called “Search Folder Nesting Level” in HKLM\System\CCS\Services\MSExchangeIS\ParamtersSystem and set it to a value of 10 (down from its default value of 20). At the time of the incident, the need for this registry setting was not yet mentioned in the KB article.

Continue reading →