Paving the Way to Exchange 2010 (Pay Attention, Entourage Users)

When will AEMS be upgraded to Exchange 2010?

That is a very good question, and somewhat in flux given current efforts to migrate ITS servers to the new data center. Tentatively, we are looking at sometime in 2011, depending upon when we can purchase the needed hardware.

How will this transition be done?

We’ve already taken the first steps by upgrading our environment to Exchange 2007 SP2, which applies the relevant AD Schema modifications which are required for Exchange 2010. An additional benefit of this update, is that it will enable the use of “online mailbox moves” when we migrate users from 2007 mailbox servers to 2010 mailbox servers.

Exchange cannot be upgraded “in-place” to 2010. What will be necessary for us to do will be to install new Exchange 2010 servers in parallel with our existing Exchange 2007 servers. We will then move user mailboxes over from the 2007 servers to the 2010 servers.  Due to the new “online mailbox move” feature, connectivity to the mailbox from Outlook will be maintained throughout the move until the very end of the move process, at which time connectivity will be interrupted for about 30 seconds while the connection is switched over to the new server location. (Outlook 2003 users will likely have to restart Outlook.)

That sounds all very well and good for Outlook users. What about other clients?

Unfortunately, online mailbox moves only work for Outlook users since it is a feature tied to the MAPI protocol. Anyone using an IMAP client, Entourage, Outlook:mac, or Snow Leopard’s Exchange-enabled applications will find their mailbox access interrupted for the duration of the move of their mailbox (just as it was for all users during the transition to Exchange 2007). Such clients will likely have to be restarted after the mailbox move is complete. The duration for any given mailbox move will, of course, depend upon the size of the mailbox and how busy the servers are at the time of the move.

Important note for Entourage users!

If you are using a version of Entourage prior to the Web Services Edition, you will need to upgrade to Entourage 2008 Web Services Edition (or to Outlook:mac 2011) prior to having your mailbox moved to Exchange 2010. The new version of Exchange drops support for WebDAV, the protocol used by older Entourage versions for Exchange connectivity, in favor of Exchange Web Services (EWS). Upgrading to the Web Services Edition is accomplished by applying a downloadable patch to an existing Entourage 2008 installation. The latest version of this patch , released just this week, also adds synchronization of Tasks, Notes, and Categories. Alternatively, upgrade to Outlook:mac 2011, which is available now from ITS’ Site Licensed disk share as part of the Office 2011 suite. Outlook:mac 2011, which also makes use of EWS for its Exchange connectivity (as opposed to the MAPI protocol used by the Windows version of Outlook), is essentially a from-the-ground-up rewrite of Entourage using the Cocoa API’s and incorporates substantial improvements over Entourage, most notably a change in the architecture of the local cache database to make it more Spotlight-friendly.

Important note for Snow Leopard users!

With the release of the Snow Leopard edition of Mac OS X, Apple added native Exchange connectivity (via EWS) to Mail, iCal, and Address Book. For the most part, this works fairly well. However, based upon reports from other institutions which have already made the jump to Exchange 2010, these applications exhibit some problematic behavior during the transition period.

Earlier, I mentioned that transitioning to Exchange 2010 involves setting up Exchange 2010 servers in parallel with the existing 2007 environment. What is supposed to happen with EWS connections (and all protocol connections, for that matter) is the following:

  1. The client connects to the 2010 Client Access Server (CAS) and authenticates.
  2. If the user’s mailbox is on a 2010 Mailbox (MBX) server, the 2010 CAS mediates the connection normally.
  3. If the user’s mailbox is still on a 2007 MBX server, the 2010 CAS redirects the client (complete with authentication tokens) to the appropriate 2007 CAS, which then mediates communication between the client and mailbox.

Unfortunately, it seems that the Snow Leopard apps do not handle the redirection mentioned in step 3 properly. Other institutions have had to instruct the users of those applications to temporarily configure their software to use an alternative client configuration (pointing specifically to the legacy CAS infrastructure) until such time as the mailboxes are moved.  We will of course be testing to find ways to mitigate the impact of this flaw. Hopefully, by the time we are ready to roll out 2010, Apple will have issued an update to correct this issue.

This all sounds grand. But what are the benefits of upgrading to Exchange 2010?

  • An improved high-availability model for reducing downtimes.
  • An improved fine-grained security model based upon Role-Based Access Control (RBAC) , which will allow us to delegate some support functions to Help Desk staff and/or departmental Technical Support Contacts.
  • An overhauled database schema, allowing for improved performance and better scalability as we grow.
  • A vastly improved OWA experience. OWA, now called Outlook Web App, has many new features, including allowing users to perform their own searches of the message tracking log, support for sharing Calendars and viewing shared Calendars, external Calendar sharing, Conversation View, and cross-platform OWA Premium support in Firefox and Safari! (Mac and Linux users rejoice! Although not officially tested by Microsoft, Chrome works as well.)
  • Support for MailTips when used with Outlook 2010.

So, what’s up with MAIL02?

AEMS users whose mailboxes reside on mailbox server MAIL02 will have undoubtedly noticed that their particular mailbox server has been woefully unstable of late, much to everyone’s consternation. The problem is that the store.exe processes, which handles interactions with the databases containing the mailboxes, has been experiencing sporadic crashes. Through analysis of crash dumps, Microsoft Support has determined that problem is a known issue, addressed in Exchange Server 2007, Service Pack 2, Rollup 3.

We had been planning on installing Service Pack 2 and the latest rollup at some point this fall since it is a prerequisite for an eventual transition to Exchange 2010. However, this issue has forced our hand, and we are now planning to install Service Pack 2 and Rollup 4 during the maintenance window this coming weekend (or sooner if the Powers-That-Be™ authorize it). I have been knee-deep in testing the new updates to move them through our change management process.

Microsoft has not published details on the nature of the bug, and for very good reason. A nefarious individual could use the bug to launch a denial of service attack on Exchange servers which are not patched up to at least SP RU 3.  Suffice it to say, that it involves store.exe not being able to properly digest certain strings of data coming from mobile devices.

We apologize for the inconvenience and ask that everyone bear with us until we get this fixed.

UPDATE [Aug. 24, 2:30PM]: Microsoft’s analysis of the last two crash dumps that we sent in revealed that the user responsible for them was the same one found in the first crash dump analysis, despite the fact that we had disabled ActiveSync access for that user. It turns out that all previous cases that the folks at Microsoft had encountered of this problem were due to mobile devices, but the crash analysis only reveals that the method of access being used was non-MAPI. The user in question happens to be an Entourage user.  We’ve cut off all non-MAPI access for the user in question and provided instructions on accessing his mailbox via Outlook on vDesk.

That having been said, although we haven’t had a store crash since 11:11am, CPU usage on MAIL02 is still being pegged. The store crashes have clearly left the server in a degraded state performance-wise. We will likely have to perform a failover and reboot at some point in order to clear it up, possibly after 5:00pm.

UPDATE [Aug. 25, 9:25AM]: Mid-afternoon yesterday, I decided to go ahead and perform an emergency failover of MAIL02 since its performance was in such a degraded condition. Performance has been solid since then, and there have been no further store.exe crashes.

We are preceding with plans to apply updates this weekend. In fact, the change board meeting to gain approval for the change takes place in about an hour.

Addressing iOS 4 Exchange Sync Issues

After upgrading their iPhones to iOS 4, some individuals have encountered issues with poor synchronization performance. (My own iPhone 3G seemed like it was attempting to sync constantly.) The folks at Apple have published a KB article discussing this issue, and the article includes a download link for a configuration profile designed to address this issue. The downloadable file is simply a snippet of XML which sets the value of the default EAS task timeout to 240.

To install the fix, simply point your iPhone’s web browser to the link above, then select the download link. From there, follow your nose.  Please note that installing this fix requires a reboot of the iPhone. While the installer does provide notification of this early on in the process, there is no prompt at the end to perform the reboot.

For those of you in the UT community who user our iPhone profile generator (EID authentication required), I have spoken to the maintainer of that page about including the relevant XML in the downloadable profiles produced by that page. He is looking into it.

Update: Our Microsoft TAM has passed along some info which refers to this issue as well as a few other known issues involving iOS 4 and Exchange. Microsoft is working with Apple to remedy the issues:

* iPhone 4 users reportedly cannot send email via 3G, but can via WiFi, when connecting to an Exchange 2010 environment. (Thankfully, we are still on 2007.)
* Exchange 2003 issue: the new iPhone is reportedly deleting all meetings in a recurring meeting if an exception is generated for a day in the series. (Again, not applicable in our environment.)
* There have also been reports of issues with iOS 4 synchronizing with Google apps via ActiveSync, but this was apparently an issue with Google’s servers.

Behold the power of Get-MailboxFolderStatistics

Not too long ago, I received a ticket from a user who was having quota issues and could not account for what was using up their Exchange quota. With a simple Exchange Management Shell one-liner, I was able to provide a list of every folder in that user’s mailbox, the number of items in each folder, and size of the folder contents:









[PS] E:\working\scripts>Get-MailboxFolderStatistics jane.user | sort-object item
sinfolder -descending | ft Name, FolderPath, ItemsInFolder, FolderSize -auto

Name                     FolderPath     ItemsInFolder FolderSize
----                     ----------     ------------- ----------
Inbox                    /Inbox                    33 103574B
Calendar                 /Calendar                 25 64012B
Deleted Items            /Deleted Items             7 9101B
Sent Items               /Sent Items                6 7690B
Notes                    /Notes                     0 0B
Outbox                   /Outbox                    0 0B
Tasks                    /Tasks                     0 0B
RSS Feeds                /RSS Feeds                 0 0B
Contacts                 /Contacts                  0 0B
Top of Information Store /                          0 0B
Drafts                   /Drafts                    0 0B
Junk E-Mail              /Junk E-Mail               0 0B
Journal                  /Journal                   0 0B

But wait, there’s more. As mentioned in my previous article, I’ve encountered issues with folders which have leading or trailing whitespace in their name. Sure enough, I was able to crank out a simple one-liner to list all  folders possessing this defect:

[PS] E:\working\scripts>get-mailbox -resultsize unlimited | Get-MailboxFolderSta
tistics | where {($_.Name -like "* ") -or ($_.Name -like " *")} | fl Name,Identi
ty,FolderPath
[PS] E:\working\scripts>get-mailbox -resultsize unlimited | Get-MailboxFolderSta tistics | where {($_.Name -like "* ") -or ($_.Name -like " *")} | fl Name,Identi ty,FolderPath

When Bugs Collide: A Caution for Entourage Users

Entourage users: please DO NOT create folders with leading or trailing spaces in their names!

When creating a folder with a leading or trailing space in the name, Entourage will quietly generate an error as follows:

Error

Could not synchronize record: <foldername> to Exchange server: <identity name>

Explanation

The folder save operation failed due to invalid property values.

Error: -19918

However, Entourage will then proceed to behave as if the folder creation actually worked, display the misnamed folder in the folder tree, and allowing messages to be dropped into it.  Looking at the same mailbox via Outlook, the misnamed folder never appears, and the “moved” messages remain where they originally are. (Verified with testing using Entourage 2008 Web Services Edition, version 13.0.4 (100208).)

I note that Outlook will strip leading or trailing whitespace from folder names when creating them, as does Apple’s Mail.app.

I’m not the first to notice this issue: http://blogs.technet.com/b/pawan/archive/2010/03/02/issue-with-synchronizing-folders-in-entourage-ews-which-are-created-with-trailing-or-ending-spaces.aspx

On a potentially related note, it seems that one of our users actually managed to create a folder with a trailing space on our Exchange 2007 SP1 system (I assume that this was done using an older, pre-EWS version of Entourage), which created a world of woes when trying to merge his restored mailbox contents from a Recovery Storage Group. I need to perform some more testing to verify that the trailing whitespace was the source of the problem with restore-mailbox, which might be tricky if I can’t figure out how to create a test folder with trailing whitespace in the name to begin with.  I’ll follow up on that later once I have had a chance to do some testing.

Calendar Sharing Permissions Bug in Entourage Web Services

I’ve long been aware that Entourage does not know how to properly handle the granularity of control over free/busy information introduced by Exchange 2007, but I’ve now become aware of an even more serious bug on this front in Entourage 2008 Web Services Edition. It seems that anytime that version of Entourage is used to modify Calendar sharing permissions on an Exchange 2007 server (and presumably an Exchange 2010 server), the user or group to whom the permissions are being granted looses the ability to see the free/busy information for that Calendar, regardless of the permissions level granted.

Here are the steps that I have taken to replicate this issue in our environment.  We have Exchange  2007 SP1 Rollup 8. In the testing which I describe below, I am using Outlook 2007 and Entourage 2008 Web Services Edition, Version 13.0.03  (091002). Steps to reproduce the issue are as follows:

  • – In Outlook, grant sharing permission to another user on your Calendar. For this example, we’ll grant the user “Free/Busy time, subject, location” permissions. Verify that on the permissions tab, with that user selected, in the Read section, the radio button  for “Free/Busy time, subject, location is checked. After clicking “Apply”, verify that the user can in fact see your free/busy. We have now set a baseline for expected functionality.
  • – Now, in Entourage, check the sharing permissions on your Calendar. For the user to whom you have granted permissions, the permissions level will show as “None.”  Now let us use Entourage to modify the sharing permissions. Set it to “Reviewer” for example, and click “OK.”
  • – At this point, the user to whom permissions have been granted will be unable to see your free/busy information. (This is true regardless of what permission  level is set in Entourage.)  Looking at the Calendar sharing permissions in Outlook, none of the Read permission items are checked.

This is a serious bug in Entourage, and the only way that I know of to repair the damage is to properly set the permission with an Outlook client.

UPDATE (Sept. 15): We received a note from Cornell indicating that they found that applying Service Pack 3 fixed this issue. Thanks for the heads-up. We are at SP2 RU4, and the problem still manifests at that patch level.

Dealing with “Orphaned” Delegations

Every once in a while, we get a ticket complaining about meeting invites generating an NDR claiming that a mailbox (not belonging to any of the invitees) does not exist. Invariably, this turns out to be due to one of the invitees having a delegate whose mailbox has been deleted, and this can be a pesky problem to resolve.

Here is a summary of how the delegation process works under the hood, and how to deal with “dangling” delegations when mailboxes are deleted.

Suppose you have a mailbox for user A who wishes to make user B a delegate.  When the delegation is made the following takes place:

  1. Appropriate sharing permissions are placed on the relevant folders in user A’s mailbox, depending upon the permissions set with the delegation tool.
  2. If the checkbox for forwarding meeting requests is set, a special hidden forwarding rule is created in user A’s mailbox.
  3. User B is added to user A’s publicDelegates attribute (the send-on-behalf-of field), and user A is added to the publicDelegatesBL attribute of user B. This does not always happen in recent versions of Outlook, especially when the person trying to set the delegation is not actually the owner of mailbox A. See this earlier posting for more information on this issue.

    Unfortunately, if the mailbox for user B is deleted, their publicDelegatesBL entry in AD gets cleared out, so there is no direct way to track down who they might have been a delegate for. Even more problematic, when user B’s mailbox is deleted, the hidden forwarding rule in mailbox A does not get updated, which is why it is important to always remove someone as a delegate prior to their mailbox being deleted.

    Assuming that we can narrow down whose mailbox has the orphaned delegation, we can manually fix this by using the MFCMAPI tool to delete the hidden forwarding rule (the instructions being contained in the README file accompanying MFCMAPI). Unfortunately, if there are other delegates to that mailbox, the forwarding for them would break as well, so all of those delegates would need to be removed and re-added, which requires logging directly into the mailbox in question.

    If the owner of the problematic mailbox is okay with this, the Exchange administrators can go ahead and nuke that hidden rule from their mailboxe, but the mailbox owner will need to go in afterwards and redo their delegations, or the Exchange administrators could do that with their permission.

    Still More Exchange 2010 Goodies…

    Microsoft Forefront Protection 2010 for Exchange Server is now available. Capacity planning guidance for this product has also been published.

    Also of note is that Forefront support for the AhnLab, CA, and Sophos engines is being retired on Dec. 1, 2009. The new set of available scan engines will be Authentium, Kasperky, Microsoft, Norman, and VirusBuster.

    The only piece now missing is a 2010-friendly version of DPM for backing up Exchange.

    The Latest Exchange 2010 Goodies

    Exchange 2010 RTM bits have been made available for download. Weee!

    Also, the Exchange 2010 Mailbox Server Role Requirements Calculator is out and I’m reading through the documentation on it now. (Now for a comparable tool for scaling CAS and HUB deployments and we’ll be all set.)

    But wait, there is more. The Exchange Server 2010 Deployment Assistant has also been released. This is basically a wizard which asks questions about your environment, and creates deployment checklists. Very helpful, far more so than trying to wade through the mountains of TechNet docs to create such checklists (which I remember vividly doing for our transition to Exchange 2007). On the downside, the current release is only for transitions from Exchange 2003 to 2010. A version which includes transitioning from 2007 is slated for release in early 2010.

    Of more immediate interest to the UT community, after I (along with our storage guru, Alex Barth) had spent most of last week cobbling together a preliminary plan for transitioning AEMS to Exchange 2010, we met with a Dell consultant for a whiteboard session to go over options for such a transition. Although there was not much depth to the session, the consultant did bring to our attention some ideas we had not really considered, so now we have more food for thought and testing to perform. My plan may very well undergo significant rewrites, but that’s okay. Actual deployment is a long way off, which gives us ample time to refine our plan so that, hopefully, it will go off with a minimum of problems.