Exchange Server 2010 : Monitoring and Troubleshooting Unified Messaging

A number of tools are built in to the Exchange Server 2010 unified messaging platform to support the troubleshooting and monitoring of the services.

First and foremost, it is highly recommended that administrators deploy Microsoft System Center Operations Manager (SCOM) 2007 to monitor the Exchange Server 2010 infrastructure. There is a management pack specific to the Exchange Server 2010 platform with a wealth of knowledge built in.

That said, it is still important for the Exchange Server administrator to have a good knowledge and familiarity with the tools that are available to monitor Exchange Server 2010 unified messaging. These tools include the following:

• Exchange Management Shell test cmdlets

• Performance Monitor objects and counters

• Event log messages

• Removing the first UM server

These tools and techniques are covered in the next sections.

Active Calls

The system can provide information on active calls, which is very useful for monitoring and troubleshooting the unified messaging system. The Get-UMActiveCalls cmdlet returns information about the calls that are active and being processed by the Unified Messaging (UM) server. The syntax for the cmdlet is given in Table 1.

Table 1. Get-UMActiveCalls cmdlets

Syntax
Get-UMActiveCalls [-Server <ServerIdParameter>] Get-UMActiveCalls -InstanceServer <UMServer> Get-UMActiveCalls -DialPlan <UMDialPlanIdParameter> Get-UMActiveCalls -IPGateway <UMIPGatewayIdParameter>

Figure 1 shows two instances of the command. In the first execution of the command, it shows an active call with the DialedNumber of 333, indicating that it has come in via the subscriber access line configured in the example installation. In the second execution of the command, it shows that the caller has dialed extension 102.

Figure 1. Active calls.

Connectivity

Connectivity to the IP/VoIP gateway can be one of the most troublesome aspects of the deployment and support of a unified messaging system. The Test-UMConnectivity cmdlet can be used to test the operation of a computer that has the Unified Messaging server role installed. The syntax for the cmdlet is given in Table 2.

Table 2. Test-UMConnectivity cmdlet

Syntax
test-UMConnectivity [-Fax <$true | $false>] [-ListenPort <Int32>] [-MonitoringContext <$true | $false>] [-Secured <$true | $false>] [-Timeout <Int32>] Test-UMConnectivity -IPGateway <UMIPGatewayIdParameter> -Phone <String> [-Fax <$true | $false>] [-ListenPort <Int32>] [-MonitoringContext <$true | $false>] [-Secured <$true | $false>] [-Timeout <Int32>]

Performance Monitors

Unlike many applications, the Exchange Server 2010 unified messaging application is very well instrumented.

Tables of the counters for each of the monitored objects are noted in the balance of this section.

General Performance Counters for Unified Messaging

The counters listed in Table 3 are under the MSExchangeUMGeneral performance object and are useful for monitoring and troubleshooting general problems with the Exchange 2010 UM server.

Table 3. Counters for the MSExchangeUMGeneral Object

Performance Counter	Description
Total Calls	The number of calls since the service was started.
Total Calls per Second	The number of new calls that arrived in the last second.
Calls Disconnected by User Failure	The total number of calls disconnected after too many user entry failures.
Calls Rejected	The total number of new call invitations that have been rejected.
Calls Rejected per Second	The number of new call invitations that have been rejected in the last second.
Current Calls	The number of calls currently connected to the UM server.
Current Voice Calls	The number of voice calls currently connected to the UM server.
Current Fax Calls	The number of fax calls currently connected to the UM server. Voice calls become fax calls after a fax tone is detected.
Current Auto Attendant Calls	The number of Auto Attendant calls that are currently connected to the UM server.
Current Play on Phone Calls	The number of outbound calls initiated to play back messages.
Current Unauthenticated Pilot Number Calls	The number of voice calls to the pilot number that have not yet been authenticated.
Total Play to Phone Calls	The total number of Play to Phone calls that were initiated since the service was started.
Average Call Duration	The average duration, in seconds, of calls since the service was started.
Average Recent Call Duration	The average duration, in seconds, of the last 50 calls.
User Response Latency	The average response time, in milliseconds, for the system to respond to a user request. This average is calculated over the last 25 calls. This counter is limited to calls that require significant processing.
Delayed Calls	The number of calls that experienced one or more delays longer than 2 seconds.
Call Duration Exceeded	The number of calls that were disconnected because they exceeded the UM maximum call length. This number includes all types of calls, including fax calls.
Current Prompt Editing Calls	The number of logged on users who are editing custom prompts.
Current Subscriber Access Calls	The number of logged on subscribers who are currently connected to the UM server.
OCS User Event Notifications	The number of OCS User Event notifications that have occurred since the service was started.
Current CAS Connections	The number of connections that are currently open between the UM server and client access servers.

Call Answering Performance Counters for Unified Messaging

The counters listed in Table 4 are under the MSExchangeUMCallAnswering performance object and are useful for monitoring and troubleshooting call answering problems with the Exchange Server 2010 UM server.

Table 4. Counters for the MSExchangeUMCallAnswering Object

Performance Counter	Description
Call Answering Calls	The number of diverted calls that were answered on behalf of subscribers.
Call Answering Voice Messages	The total number of messages that were submitted because the calls were answered on behalf of subscribers.
Call Answering Voice Messages per Second	The number of messages that were submitted because the calls were answered on behalf of subscribers.
Call Answering Missed Calls	The number of times a diverted call was dropped without a message being left.
Call Answering Escapes	The number of times a caller pressed the * key to connect to another user rather than leaving a message.
Average Voice Message Size	The average size, in seconds, of voice messages left for subscribers.
Average Recent Message Size	The average size, in seconds, of the last 50 voice messages left for subscribers.
Average Greeting Size	The average size, in seconds, of recorded greetings that have been retrieved by the UM server.
Calls Without Personal Greetings	The number of diverted calls received for subscribers who did not have recorded greeting messages.
Fetch Greeting Timed Out	The number of diverted calls for which the subscriber’s personal greeting could not be retrieved within the time allowed.
Calls Disconnected by Callers During UM Audio Hourglass	The number of calls during which the caller disconnected while Unified Messaging was playing the audio hourglass tones.
Calls Disconnected by UM on Irrecoverable External Error	The number of calls that have been disconnected after an irrecoverable external error occurred.
Call Failed Because Transport Unavailable	The number of calls that failed because the transport service is not available to submit the message.
Diverted Extension Not Provisioned	The number of calls received for which the diverted extension supplied with the call is not a UM subscriber extension.

Subscriber Access Performance Counters for Unified Messaging

The counters listed in Table 5 are under the MSExchangeUMSubscriberAccess performance object and are useful for monitoring and troubleshooting subscriber access problems with the Exchange Server 2010 UM server.

Table 5. Counters for the MSExchangeUMSubscriberAccess Object

Performance Counter	Description
Subscriber Authentication Failures	The number of authentication failures that have occurred since the service was started. This number is incremented once for every failed authentication. It’s possible that a single phone call could generate several authentication failures.
Subscriber Logons	The number of UM subscribers who have successfully authenticated since the service was started.
Subscriber Logon Failures	The number of authentication failures since the service was started. This number is incremented once when all three per-call logon attempts fail.
Average Subscriber Call Duration	The average duration, in seconds, that subscribers spent logged on to the system. This timer starts when logon completes.
Average Recent Subscriber Call Duration	The average length of time, in seconds, that subscribers spent logged on to the system for the last 50 subscriber calls.
Voice Message Queue Accessed	The number of times subscribers accessed their voice message queues using the telephone user interface.
Voice Messages Heard	The number of voice messages played to subscribers. This count is incremented as soon as playback starts. The subscriber does not need to listen to the entire message.
Voice Messages Sent	The number of voice messages sent by authenticated UM subscribers.
Average Sent Voice Message Size	The average size, in seconds, of voice messages that are sent. This size does not include any attachment data.
Average Recent Sent Voice Message Size	The average size, in seconds, of the last 50 voice messages that were sent.
Voice Messages Deleted	The number of voice messages that were deleted by authenticated subscribers.
Reply Messages Sent	The number of replies sent by authenticated subscribers.
Forward Messages Sent	The number of messages forwarded by authenticated subscribers.
Email Message Queue Accessed	The number of times subscribers accessed their email message queue using the telephone user interface.
Email Messages Heard	The number of email messages heard by authenticated subscribers.
Email Messages Deleted	The number of email messages deleted by authenticated subscribers.
Calendar Accessed	The number of times subscribers accessed their calendars using the telephone user interface.
Calendar Items Heard	The number of calendar items heard by authenticated subscribers.
Calendar Late Attendance	The number of messages sent to inform the organizer of a meeting that the subscriber will be late.
Calendar Items Details Requested	The number of times a subscriber requested additional details for a calendar item.
Meetings Declined	The number of Meeting Declined messages sent by subscribers.
Meetings Accepted	The number of Meeting Accepted messages sent by subscribers.
Called Meeting Organizer	The number of times subscribers called the meeting organizer.
Replied to Organizer	The number of times subscribers sent reply messages to meeting organizers.
Contacts Accessed	The number of times subscribers accessed the Main Menu Contacts option using the telephone user interface.
Contact Items Heard	The number of times authenticated subscribers listened to directory details.
Launched Calls	The number of subscriber calls that resulted in an outbound call being placed.
Calls Disconnected by Callers During UM Audio Hourglass	The number of times callers disconnected while Unified Messaging was playing the audio hourglass tones.
Calls Disconnected by UM on Irrecoverable External Error	The number of subscriber calls that have been disconnected after an irrecoverable external error occurred.
Directory Accessed	The number of times subscribers accessed the Main Menu Directory option using the telephone user interface.
Directory Accessed by Extension	The number of directory access operations in which the user supplied the extension number.
Directory Accessed by Dial by Name	The number of directory access operations where the subscriber used the Dial by Name feature.
Directory Accessed Successfully by Dial by Name	The number of dial by name directory access operations that completed successfully on behalf of subscribers.
Directory Accessed by Spoken Name	The number of directory access operations in which the subscriber spoke a recipient name.
Directory Accessed Successfully by Spoken Name	The number of speech-recognition directory access operations that completed successfully on behalf of subscribers.

The variety of counters in the subscriber access area is impressive and can really aid in the understanding of the behavior of the subscribers.

Unified Messaging Auto Attendant Performance Counters

The counters listed in Table 6 are under the MSExchangeUMAutoAttendant performance object and are useful for monitoring and troubleshooting Auto Attendant problems with the Exchange Server 2010 UM server.

Table 6. Counters for the MSExchangeUMAutoAttendant Object

Performance Counter	Description
Total Calls	The number of calls that have been processed by this Auto Attendant.
Business Hours Calls	The number of calls processed by this Auto Attendant during business hours.
Out of Hours Calls	The number of calls processed by this Auto Attendant outside of business hours.
Disconnected Without Input	The number of calls that were dropped without any input being offered to the Auto Attendant prompts.
Transferred Count	The number of calls that were transferred by this Auto Attendant. This number does not include calls that were transferred by the operator.
Directory Accessed	The number of directory access operations performed by this Auto Attendant.
Directory Accessed by Extension	The number of directory access operations in which the user supplied the extension number.
Directory Accessed by Dial by Name	The number of directory access operations in which the subscriber used the Dial by Name feature.
Directory Accessed Successfully by Dial by Name	The number of successful directory access operations in which the caller used the Dial by Name feature.
Directory Accessed by Spoken Name	The number of directory access operations in which the subscriber spoke a recipient name.
Directory Accessed Successfully by Spoken Name	The number of successful directory access operations in which the caller used the Spoken Name feature.
Operator Transfers	The number of calls that were transferred to the operator.
Menu Option 1 Used	The number of times that a caller has chosen option 1 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 2 Used	The number of times that a caller has chosen option 2 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 3 Used	The number of times that a caller has chosen option 3 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 4 Used	The number of times that a caller has chosen option 4 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 5 Used	The number of times that a caller has chosen option 5 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 6 Used	The number of times that a caller has chosen option 6 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 7 Used	The number of times that a caller has chosen option 7 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 8 Used	The number of times that a caller has chosen option 8 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option 9 Used	The number of times that a caller has chosen option 9 from the custom menu. This value is always zero if no menu or option is defined.
Menu Option Timed Out	The number of times that the system has timed out waiting for a caller to select an option from the custom menu. This value is always zero if no menu is defined.
Average Call Time	The average length of time that callers interacted with the Auto Attendant.
Calls with DTMF Fallback	The total number of times a caller has been passed the DTMF fallback auto attendant. This happens only for speech-enabled auto attendants.
% Successful Calls	% Successful Calls calculates the success rate of the auto attendant.
Calls Disconnected by UM on Irrecoverable External Error	The total number of calls disconnected after an irrecoverable external error occurred.
Average Recent Call Time	The average length of time, in seconds, of the last 50 Auto Attendant calls.
Calls with Speech Input	The total number of calls during which the caller is determined to have spoken at least once.
Calls with Sent Message	The total number of calls in which the caller has sent a voice message.
Calls with Spoken Name	The total number of calls to this Auto Attendant in which a caller has spoken a name at least once.
Custom Menu Options	The total number of times callers have selected custom menu options.
Disallowed Transfers	The number of times a caller was transferred to the operator because the user identified was configured to only accept calls from logged on users.
Operator Transfers Requested by user	The number of times a caller to this Auto Attendant has asked to be transferred to an operator.
Operator Transfers Requested by User from Opening Menu	The number of times a caller to this Auto Attendant has asked to be transferred to an operator while at the opening menu.
Sent to Auto Attendant	The number of times a caller has used the custom menu to go to an Auto Attendant.
Ambiguous Name Transfers	The number of times that a caller was transferred to the operator because the name that they spelled or spoke was too common in the search results.

The variety of counters in the Auto Attendant area is impressive and can really aid in the understanding of the behavior of the callers, the menu choices they make, how long they stay in the system, and their preferred method of access to the menus.

System Resources and Availability Counters for Unified Messaging

The counters listed in Table 7 are under the MSExchangeAvailability performance object and are useful for monitoring and troubleshooting system resource and availability problems with the Exchange Server 2010 UM server.

Table 7. Counters for the MSExchangeAvailability Object

Performance Counter	Description
Directory Access Failures	The number of times that attempts to access Active Directory failed.
Hub Transport Access Completed	The number of times that the Hub Transport server was accessed successfully.
Hub Transport Access Failures	The number of times that attempts to access a Hub Transport server failed. This number is only incremented if all Hub Transport servers were unavailable.
Mailbox Server Access Failures	The number of times the system failed to access a mailbox server.
Maximum Calls Allowed	The length of time, in seconds, that the server was concurrently processing the maximum number of calls allowed.
Worker Process Recycled	The number of times a new UM worker process has been started.
Failed to Redirect Call	The number of times the unified messaging service failed to redirect calls to a UM worker process.
Total Worker Process Call Count	The total number of calls handled by this UM worker process.
Unhandled Exceptions per Second	The number of calls that encountered an unhandled exception in the last second.
Incomplete Signaling Information	The number of calls for which the signaling information was missing or incomplete.
Calls Disconnected by UM on Irrecoverable External Error	The number of calls disconnected after an irrecoverable external error occurred.
Calls Disconnected by UM on Irrecoverable External Error/sec	The number of calls disconnected after an irrecoverable external error occurred in the last second.
Calls Disconnected on Irrecoverable Internal Error	The number of calls disconnected after an internal system error occurred.
Call Answer Queued Messages	The number of messages created and not yet submitted for delivery.
Spoken Name Accessed	The number of times the system retrieved the recorded name of a user.
Name TTSed	The number of times the system used text-to-speech to create an audio version of the display name of a subscriber.
Queued OCS User Event Notifications	The number of notifications that have been created and not yet submitted for delivery.

Unified Messaging Performance Monitoring Counters

The counters listed in Table 8 are under the MSExchangeUMPerformance performance object and are useful for monitoring and troubleshooting server latency problems with the Exchange Server 2010 UM server. These counters measure the time in number of seconds that server operations took. This is an important measure of the time that callers are waiting for the UM server to complete a task.

Table 8. Counters for the MSExchangeUMPerformance Object

Performance Counter	Description
Operations over Two Seconds	The number of all UM operations that took between 2 and 3 seconds to complete. This is the time during which a caller was waiting for UM to respond.
Operations over Three Seconds	The number of all UM operations that took between 3 and 4 seconds to complete. This is the time during which a caller was waiting for UM to respond.
Operations over Four Seconds	The number of all UM operations that took between 4 and 5 seconds to complete. This is the time during which a caller was waiting for UM to respond.
Operations over Five Seconds	The number of all UM operations that took between 5 and 6 seconds to complete. This is the time during which a caller was waiting for UM to respond.
Operations over Six Seconds	The number of all UM operations that took more than 6 seconds to complete. This is the time during which a caller was waiting for UM to respond.
Operations under Two Seconds	The number of all UM operations that took less than 2 seconds to complete. This is the time during which a caller was waiting for Unified Messaging to respond.

Event Logs

Event logs are important for troubleshooting Microsoft Exchange Server unified messaging. Each of the different aspects of a Unified Messaging server generate their own set of error messages. The tables of events show the errors and events that could be generated for each of the following categories:

• Call answering

• Call transfer

• Subscriber access

• Auto Attendant

• Active Directory

• Prompt Publishing

• Outdialing

• Administrative

• Speech grammar

• System

• Performance

The event log messages are very detailed and specific to many conditions, making it very easy to understand, audit, troubleshoot, and instrument the Unified Messaging server.

Although there are hundreds of event IDs related to unified messaging, a few common events are reviewed here for help in troubleshooting.

Call Answering

Unified messaging generates call answering events to help troubleshoot call answering features and issues. A good example is when a call is directed to a user that is not enabled for unified messaging. Exchange Server UM logs an 1169 event in Event Viewer with the extension number of the affected user. To resolve this issue, enable the user for Exchange Server unified messaging.

Call Transfer

Unified messaging generates call transfer events for actions related to call transfers in unified messaging. Event IDs 1025 and 1136 are often found together and relate to a failure in Exchange Server UM to transfer a call. Relevant information such as call ID and destination number are included in the body of the event logged in Event Viewer.

Subscriber Access

Exchange Server 2010 unified messaging generates events based on subscriber access behavior. This relates mostly to user induced events such as event ID 1012 that indicates a user has entered his PIN incorrectly more times than is allowed by the policy and has been locked out. Reset the user’s PIN to allow him to log in again.

Another common event is ID 1080 that indicates the UM service could not process a message for a user because her mailbox is over quota. In this situation the administrator can either raise the quota, or the user must delete messages from her mailbox to get under her allowed quota limit.

Auto Attendant

The Exchange Server 2010 Unified Messaging Auto Attendant is one of the most important functions within UM. It is almost always a caller’s entryway to the system. As such, any warnings or errors encountered should be taken seriously. Event ID 1128 indicates that an operator extension has not been defined. The UM service cannot forward a call to an operator. This means calls will simply be disconnected if they are not routed through the Auto Attendant. To resolve this issue, assign an operator extension using the UM portion of the Exchange Management Console.

Active Directory

Every part of Exchange Server 2010 is tightly coupled with Active Directory, and Exchange Server 2010 Unified Messaging is no exception. If the UM service cannot contact Active Directory for lookup or authentication data, a warning or error is logged. If Active Directory is unavailable on start up of the UM service, the service fails to start.

Prompt Publishing

Exchange Server 2010 Unified Messaging enables a number of custom prompts to be configured. Although this enables a lot of flexibility, it can also be a source of errors. The UM service logs alerts and errors for events related to custom prompts. Any update of custom prompts is logged. For example, Event ID 1100 indicates a successful publishing whereas Event ID 1099 indicates that a failure occurred and provides information on where the failure happened.

Outdialing

Exchange Server 2010 UM enables users to dial out from the system dependent on policy. This includes the Play on Phone functionality. Most of the events in this section focus on the dialing rules set in Exchange Server UM. The dialing rules might be different depending if the next hop is a PBX or a media gateway. An example of a warning is Event ID 1076 that indicates the UM service cannot complete a Play on Phone request due to improperly configured dialing rules. To resolve the issue, review and make the appropriate changes to your dialing rules.

Administrative

The administrative events logged for Exchange Server 2010 unified messaging are all administrative. They indicate actions such as enabling or disabling a user for Exchange Server UM or the changing of a PIN. They are often good to check when you make an administrative change if you suspect that change was not successful.

Speech Grammar

Exchange Server 2010 unified messaging has a built-in speech grammar file for understanding Automatic Speech Recognition (ASR) commands. If this file is not present or is malformed, a warning or error is logged. Event ID 1086, for example, indicates that the speech grammar files cannot be found, and ASR will be disabled. To resolve this issue, place a valid speech grammar file in the default location and restart the service.

System

The large majority of events logged for Exchange Server 2010 unified messaging fall under the System category. The events range from informational such as when the UM worker process successfully starts to significant failures such as TLS certificate failures. Other events that are important are those related to the UM IP gateway such as Event IDs 1124 and 1165 that indicate no IP gateways were found. To resolve this issue, ensure the UM server is configured to the correct IP gateway and the gateway is operational.

Performance

There are only two performance-related events for Exchange Server unified messaging. Event ID 1054 is informational and indicates the UM worker process was terminated because the startup time exceeded the maximum. Event ID 1089 is a warning and indicates the IP gateway did not respond promptly to a SIP request issued by the UM server.

Removing the First UM Server in a Dial Plan

The unified messaging prompt publishing point for a dial plan is automatically set at the time that the first Unified Messaging server joins the dial plan. Before you can remove the first Unified Messaging server, you need to migrate all of the custom prompts and then set the new server as the prompt publishing point.

For each dial plan that has a prompt publishing point on the server to be removed, run the following Exchange Management Shell command:

Set-UMDialPlan -Identity <Dial_Plan_Name> -PromptPublishingPoint <UNC_Path_to_PromptPublishingPoint_Share_On_New_Server>

					  

Then copy all of the contents of the old Universal Naming Convention (UNC) path to the new UNC path. The first Unified Messaging server can now be removed safely.