Issues with iCal Saving Calendar Items to Exchange in Snow Leopard

Recently, once of our Exchange users encountered an odd error while trying to use iCal to save a calendar event to our Exchange 2010 server. When trying to save a new event, the following error would crop up:

iCal can’t save the event “whatever the event name is” to the Exchange server.

The account *****@*****.**** currently can’t be modified. To discard your changes and continue using the version of your calendars that’s on the server, click Rever to Server. To save your changes on your computer until the problem is resolved, click Go Offline.

(Go Offline)     (Revert to Server) (Try Again)

Weird, eh?

After doing a bit of digging, we quickly found the issue documented here and there on the web. It seems that the problem stems from the fact that iCal does not by default use the system settings for Time Zones, and the above error pops up if there is a mismatch.

To fix the issue, it is necessary to go to the Advanced tab of the iCal preferences (iCal -> Prefererences -> Advanced) and enable time zone support. Also, check the system’s time zone settings (System Preferences -> Date & Time -> Time Zone) and make sure that the correct time zone is set there.

Well, that could have gone better….

So here is the skinny on what went awry with the transition of AEMS to Exchange 2010 Client Access Servers.

Although most users did not encounter any difficulties, a sizable subset encountered difficulties which can be broken down into the following categories, difficulties which were serious enough that it was felt that a backdown of our change was warrented:

IMAP4 issues: we’ve already identified and corrected one source of issues for IMAP4 users, but are still trying to characterize a remaining issue (a process made more difficult by the fact that we have reverted our change).

Droid and iPhones running pre-IOS4 software: In a 2010/2007 coexistence scenario, when an ActiveSync devices connects to the 2010 CAS, Exchange looks up the account, notices that the user still has their mailbox on Exchange 2007, and sends out an HTTP 451 error, saying “Hey, you can’t talk to that mailbox here, but if you go over this legacy address, you should be able to connect.” This does not seem to work with older iPhones or with any Android phones. We are currently trying to figure out a solution that doesn’t involve having each user of these devices manually modify their device settings to point to legacy.austin.utexas.edu, then reconfiguring to point the device back to wmail.austin.utexas.edu once their mailbox is moved, a dicey prospect at best. Unfortunately, the Microsoft engineer with which I was speaking this afternoon was not optimistic about us finding a good alternative solution for that.

Users with custom-hosted e-mail domains using non-Outlook clients: We have noted widespread problems for customers who are not using Outlook and have their client software (mobile or desktop) configured to use an email address other than their @austin.utexas.edu addy. This includes custom hosting customers, and folks who have their primary SMTP address set to use their @mail.utexas.edu address. The upshot for this is that autodiscover is broken for those address spaces, resulting in their connection not getting properly redirected to legacy.austin.utexas.edu. We MAY have come up with an interim solution which would involve DNS changes in those hosted domains. Alternatively, impacted users can configure their clients with their @austin.utexas.edu. (Outgoing mail would still APPEAR to come from their other address.)

I should point out that any user that had reconfigured their client to point to legacy.austin.utexas.edu in order to make it work need not make any changes at this point, but they will have to revert that change once we finally migrate their mailbox to 2010.

I had said in my presentation last week that problems inevitably crop up with a transition as big as this. I REALLY would not have minded being proven wrong….

Update (June 14):

For the three main issues, here is where we currently stand on trying to come up with a fix:

1) IMAP4 with NTLM – support for this seems to have been dropped from Exchange 2010 RTM, but quietly restored in SP1. Our MS support engineer is double-checking this as I type, but I suspect that, at worst, we will be able to do on this will be to tell IMAP4 users to make sure their clients are set to use Password auth. At best, we’ll figure out server settings to get this working.

2) Autodiscover failing with custom hosted domain email addresses – this will be relatively simple to fix. It requires a DNS change for every domain for which AEMS does custom address hosting.

3) Android failing to redirect – this is due to a bug in the Android code, which Google will likely not be fixing until the end of the year.  Having every Android user reconfigure their client – twice, once for the wmail switchover and once for their mailbox move – is obviously not an acceptable solution. We may be able to address this issue the same way in which we are handling Snow Leopard’s inability to handle the same redirect – setting up an iRule to on the F5 load balancers to route all traffic from these devices to legacy.austin.utexas.edu to handle the transition of wmail to Exchange 2010, identify those users by parsing our IIS logs, then move those users to Ex2010 mailbox servers in a single batch while removing the iRule. Both setting up the iRule and scouring the IIS logs are somewhat complicated by the fact that Android devices are bizarrely not consistent in how they populate the user-agent string in the HTTP headers, frequently filling it with manufacturer and device model (which is rather useless data for protocol logging), rather than something consistent, useful and simple to key on like “Android v#.# EAS”.

Accessing Exchange from Linux

Once upon a time, there was an Exchange Connector for the Linux email client Evolution, and it was fairly good. Basically an OWA screen-scraper, it enabled Evolution to function as a Linux equivalent to Outlook, and (based upon my admittedly limited tinkering with it) it seemed to get the job done.

Then along came Exchange 2007, which changed how OWA worked, thus breaking the Exchange Connector. Linux users needing to connect to Exchange were faced with being restricted to POP3 or IMAP4 (assuming their Exchange admins had those legacy protocols turned on), using OWA (which is pretty limited in a non-IE environment), or running Outlook in a Windows VM.

But there are other options these days. In addition to the dramatically improved cross-browser OWA found in Exchange 2010, there is a MAPI plugin for Revolution! I’ve not yet tinkered with it, but this opens up quite a few options.

The Perils of Running Antiquated Operating Systems (I’m looking at you, XP users!)

It is easy to forget that Windows XP will turn a decade old this fall. That is long run for an OS, and technology has continued to march on, yet many people still cling to XP. It is easy to see why. It was one of the more nimble and stable desktop OS releases that Microsoft has ever had. Being based upon the NT kernal, rather than DOS, it stood heads and shoulders above the old Windows 95, 98, and ME releases. Basically Windows 2000 Workstation with added MaxOS X inspired eye-candy, XP was a solid OS. When its successor, Windows Vista, was released in the fall of 2006, it was less than a resounding success. Sure, it was shiny and modern, but it was a resource hog with steep hardware requirements, and less stable than its predecessors. (To be fair, the latter issue was resolved with subsequent Service Pack releases.) Microsoft’s current desktop OS is Windows 7, which is essentially a souped-up Vista Service Pack, tuned to address many of the performance issues associated with Vista. It is a very nice OS, and Windows folk really should be using it, but a lot still aren’t.

I mention all of this because I ran into an issue last week caused by people clinging to XP. A user of my Exchange system (a Mac user running Outlook:mac 2011 – and yes, I have plenty of criticisms for that product as well) had found that some of the recipients of his digitally-signed messages could not read those messages. It turned out that the common factor amongst those recipients was that they were XP users. By default, when someone digitally signs a message using a personal cert on Outlook 2011, it uses SHA512 (a subset of SHA2) for its signing algorithm. But, as it turns out, the signing and encryption libraries in XP SP2 or earlier can only deal with messages signed using the SHA1 signing algorithm. XP SP3 added SOME SHA2 support, but it is quite limited.

http://blogs.technet.com/b/pki/archive/2010/09/30/sha2-and-windows.aspx

So, if you are an XP user and can’t read a signed message, you’ll have to ask the sender resend the message either unsigned or signed with SHA1. And consider upgrading. Please.

Paving the Way to Exchange 2010 (Pay Attention, Entourage Users)

When will AEMS be upgraded to Exchange 2010?

That is a very good question, and somewhat in flux given current efforts to migrate ITS servers to the new data center. Tentatively, we are looking at sometime in 2011, depending upon when we can purchase the needed hardware.

How will this transition be done?

We’ve already taken the first steps by upgrading our environment to Exchange 2007 SP2, which applies the relevant AD Schema modifications which are required for Exchange 2010. An additional benefit of this update, is that it will enable the use of “online mailbox moves” when we migrate users from 2007 mailbox servers to 2010 mailbox servers.

Exchange cannot be upgraded “in-place” to 2010. What will be necessary for us to do will be to install new Exchange 2010 servers in parallel with our existing Exchange 2007 servers. We will then move user mailboxes over from the 2007 servers to the 2010 servers.  Due to the new “online mailbox move” feature, connectivity to the mailbox from Outlook will be maintained throughout the move until the very end of the move process, at which time connectivity will be interrupted for about 30 seconds while the connection is switched over to the new server location. (Outlook 2003 users will likely have to restart Outlook.)

That sounds all very well and good for Outlook users. What about other clients?

Unfortunately, online mailbox moves only work for Outlook users since it is a feature tied to the MAPI protocol. Anyone using an IMAP client, Entourage, Outlook:mac, or Snow Leopard’s Exchange-enabled applications will find their mailbox access interrupted for the duration of the move of their mailbox (just as it was for all users during the transition to Exchange 2007). Such clients will likely have to be restarted after the mailbox move is complete. The duration for any given mailbox move will, of course, depend upon the size of the mailbox and how busy the servers are at the time of the move.

Important note for Entourage users!

If you are using a version of Entourage prior to the Web Services Edition, you will need to upgrade to Entourage 2008 Web Services Edition (or to Outlook:mac 2011) prior to having your mailbox moved to Exchange 2010. The new version of Exchange drops support for WebDAV, the protocol used by older Entourage versions for Exchange connectivity, in favor of Exchange Web Services (EWS). Upgrading to the Web Services Edition is accomplished by applying a downloadable patch to an existing Entourage 2008 installation. The latest version of this patch , released just this week, also adds synchronization of Tasks, Notes, and Categories. Alternatively, upgrade to Outlook:mac 2011, which is available now from ITS’ Site Licensed disk share as part of the Office 2011 suite. Outlook:mac 2011, which also makes use of EWS for its Exchange connectivity (as opposed to the MAPI protocol used by the Windows version of Outlook), is essentially a from-the-ground-up rewrite of Entourage using the Cocoa API’s and incorporates substantial improvements over Entourage, most notably a change in the architecture of the local cache database to make it more Spotlight-friendly.

Important note for Snow Leopard users!

With the release of the Snow Leopard edition of Mac OS X, Apple added native Exchange connectivity (via EWS) to Mail, iCal, and Address Book. For the most part, this works fairly well. However, based upon reports from other institutions which have already made the jump to Exchange 2010, these applications exhibit some problematic behavior during the transition period.

Earlier, I mentioned that transitioning to Exchange 2010 involves setting up Exchange 2010 servers in parallel with the existing 2007 environment. What is supposed to happen with EWS connections (and all protocol connections, for that matter) is the following:

  1. The client connects to the 2010 Client Access Server (CAS) and authenticates.
  2. If the user’s mailbox is on a 2010 Mailbox (MBX) server, the 2010 CAS mediates the connection normally.
  3. If the user’s mailbox is still on a 2007 MBX server, the 2010 CAS redirects the client (complete with authentication tokens) to the appropriate 2007 CAS, which then mediates communication between the client and mailbox.

Unfortunately, it seems that the Snow Leopard apps do not handle the redirection mentioned in step 3 properly. Other institutions have had to instruct the users of those applications to temporarily configure their software to use an alternative client configuration (pointing specifically to the legacy CAS infrastructure) until such time as the mailboxes are moved.  We will of course be testing to find ways to mitigate the impact of this flaw. Hopefully, by the time we are ready to roll out 2010, Apple will have issued an update to correct this issue.

This all sounds grand. But what are the benefits of upgrading to Exchange 2010?

  • An improved high-availability model for reducing downtimes.
  • An improved fine-grained security model based upon Role-Based Access Control (RBAC) , which will allow us to delegate some support functions to Help Desk staff and/or departmental Technical Support Contacts.
  • An overhauled database schema, allowing for improved performance and better scalability as we grow.
  • A vastly improved OWA experience. OWA, now called Outlook Web App, has many new features, including allowing users to perform their own searches of the message tracking log, support for sharing Calendars and viewing shared Calendars, external Calendar sharing, Conversation View, and cross-platform OWA Premium support in Firefox and Safari! (Mac and Linux users rejoice! Although not officially tested by Microsoft, Chrome works as well.)
  • Support for MailTips when used with Outlook 2010.

So, what’s up with MAIL02?

AEMS users whose mailboxes reside on mailbox server MAIL02 will have undoubtedly noticed that their particular mailbox server has been woefully unstable of late, much to everyone’s consternation. The problem is that the store.exe processes, which handles interactions with the databases containing the mailboxes, has been experiencing sporadic crashes. Through analysis of crash dumps, Microsoft Support has determined that problem is a known issue, addressed in Exchange Server 2007, Service Pack 2, Rollup 3.

We had been planning on installing Service Pack 2 and the latest rollup at some point this fall since it is a prerequisite for an eventual transition to Exchange 2010. However, this issue has forced our hand, and we are now planning to install Service Pack 2 and Rollup 4 during the maintenance window this coming weekend (or sooner if the Powers-That-Be™ authorize it). I have been knee-deep in testing the new updates to move them through our change management process.

Microsoft has not published details on the nature of the bug, and for very good reason. A nefarious individual could use the bug to launch a denial of service attack on Exchange servers which are not patched up to at least SP RU 3.  Suffice it to say, that it involves store.exe not being able to properly digest certain strings of data coming from mobile devices.

We apologize for the inconvenience and ask that everyone bear with us until we get this fixed.

UPDATE [Aug. 24, 2:30PM]: Microsoft’s analysis of the last two crash dumps that we sent in revealed that the user responsible for them was the same one found in the first crash dump analysis, despite the fact that we had disabled ActiveSync access for that user. It turns out that all previous cases that the folks at Microsoft had encountered of this problem were due to mobile devices, but the crash analysis only reveals that the method of access being used was non-MAPI. The user in question happens to be an Entourage user.  We’ve cut off all non-MAPI access for the user in question and provided instructions on accessing his mailbox via Outlook on vDesk.

That having been said, although we haven’t had a store crash since 11:11am, CPU usage on MAIL02 is still being pegged. The store crashes have clearly left the server in a degraded state performance-wise. We will likely have to perform a failover and reboot at some point in order to clear it up, possibly after 5:00pm.

UPDATE [Aug. 25, 9:25AM]: Mid-afternoon yesterday, I decided to go ahead and perform an emergency failover of MAIL02 since its performance was in such a degraded condition. Performance has been solid since then, and there have been no further store.exe crashes.

We are preceding with plans to apply updates this weekend. In fact, the change board meeting to gain approval for the change takes place in about an hour.

Addressing iOS 4 Exchange Sync Issues

After upgrading their iPhones to iOS 4, some individuals have encountered issues with poor synchronization performance. (My own iPhone 3G seemed like it was attempting to sync constantly.) The folks at Apple have published a KB article discussing this issue, and the article includes a download link for a configuration profile designed to address this issue. The downloadable file is simply a snippet of XML which sets the value of the default EAS task timeout to 240.

To install the fix, simply point your iPhone’s web browser to the link above, then select the download link. From there, follow your nose.  Please note that installing this fix requires a reboot of the iPhone. While the installer does provide notification of this early on in the process, there is no prompt at the end to perform the reboot.

For those of you in the UT community who user our iPhone profile generator (EID authentication required), I have spoken to the maintainer of that page about including the relevant XML in the downloadable profiles produced by that page. He is looking into it.

Update: Our Microsoft TAM has passed along some info which refers to this issue as well as a few other known issues involving iOS 4 and Exchange. Microsoft is working with Apple to remedy the issues:

* iPhone 4 users reportedly cannot send email via 3G, but can via WiFi, when connecting to an Exchange 2010 environment. (Thankfully, we are still on 2007.)
* Exchange 2003 issue: the new iPhone is reportedly deleting all meetings in a recurring meeting if an exception is generated for a day in the series. (Again, not applicable in our environment.)
* There have also been reports of issues with iOS 4 synchronizing with Google apps via ActiveSync, but this was apparently an issue with Google’s servers.

Microsoft Unveils RDCMan 2.2

Previously an internal-only tool, Microsoft’s Remote Desktop Connection Manager is now available for general use. And considering that I tend to have about a dozen RDC sessions open on my desktop at any given time, I have now doubt that I will find this utility useful.

All the gory details are available here.

Behold the power of Get-MailboxFolderStatistics

Not too long ago, I received a ticket from a user who was having quota issues and could not account for what was using up their Exchange quota. With a simple Exchange Management Shell one-liner, I was able to provide a list of every folder in that user’s mailbox, the number of items in each folder, and size of the folder contents:









[PS] E:\working\scripts>Get-MailboxFolderStatistics jane.user | sort-object item
sinfolder -descending | ft Name, FolderPath, ItemsInFolder, FolderSize -auto

Name                     FolderPath     ItemsInFolder FolderSize
----                     ----------     ------------- ----------
Inbox                    /Inbox                    33 103574B
Calendar                 /Calendar                 25 64012B
Deleted Items            /Deleted Items             7 9101B
Sent Items               /Sent Items                6 7690B
Notes                    /Notes                     0 0B
Outbox                   /Outbox                    0 0B
Tasks                    /Tasks                     0 0B
RSS Feeds                /RSS Feeds                 0 0B
Contacts                 /Contacts                  0 0B
Top of Information Store /                          0 0B
Drafts                   /Drafts                    0 0B
Junk E-Mail              /Junk E-Mail               0 0B
Journal                  /Journal                   0 0B

But wait, there’s more. As mentioned in my previous article, I’ve encountered issues with folders which have leading or trailing whitespace in their name. Sure enough, I was able to crank out a simple one-liner to list all  folders possessing this defect:

[PS] E:\working\scripts>get-mailbox -resultsize unlimited | Get-MailboxFolderSta
tistics | where {($_.Name -like "* ") -or ($_.Name -like " *")} | fl Name,Identi
ty,FolderPath
[PS] E:\working\scripts>get-mailbox -resultsize unlimited | Get-MailboxFolderSta tistics | where {($_.Name -like "* ") -or ($_.Name -like " *")} | fl Name,Identi ty,FolderPath

When Bugs Collide: A Caution for Entourage Users

Entourage users: please DO NOT create folders with leading or trailing spaces in their names!

When creating a folder with a leading or trailing space in the name, Entourage will quietly generate an error as follows:

Error

Could not synchronize record: <foldername> to Exchange server: <identity name>

Explanation

The folder save operation failed due to invalid property values.

Error: -19918

However, Entourage will then proceed to behave as if the folder creation actually worked, display the misnamed folder in the folder tree, and allowing messages to be dropped into it.  Looking at the same mailbox via Outlook, the misnamed folder never appears, and the “moved” messages remain where they originally are. (Verified with testing using Entourage 2008 Web Services Edition, version 13.0.4 (100208).)

I note that Outlook will strip leading or trailing whitespace from folder names when creating them, as does Apple’s Mail.app.

I’m not the first to notice this issue: http://blogs.technet.com/b/pawan/archive/2010/03/02/issue-with-synchronizing-folders-in-entourage-ews-which-are-created-with-trailing-or-ending-spaces.aspx

On a potentially related note, it seems that one of our users actually managed to create a folder with a trailing space on our Exchange 2007 SP1 system (I assume that this was done using an older, pre-EWS version of Entourage), which created a world of woes when trying to merge his restored mailbox contents from a Recovery Storage Group. I need to perform some more testing to verify that the trailing whitespace was the source of the problem with restore-mailbox, which might be tricky if I can’t figure out how to create a test folder with trailing whitespace in the name to begin with.  I’ll follow up on that later once I have had a chance to do some testing.