Programming the iPhone : Progressive Enhancement - Audio Support

Many applications provide support for ambient or event-based audio in the form of background music or sound effects. The use of sound is particularly strong in games— both immersive 3D experiences and tile- or sprite-based 2D games. Audio support makes a lot of sense for Cocoa Touch applications. After all, the iPhone is more a descendant of the iPod than a cellular phone, and the iPod Touch is positioned primarily as a media player.

Users often wear headphones when working with the devices, making the barrier to experience quite low. The media player role of the devices can dissuade developers from adding audio support to their applications—after all, when users are wearing headphones, they are likely listening to music or making a phone call and are thus opening an application for some secondary purpose.

There are two approaches to handling audio with Cocoa Touch. The first approach is to interrupt audio playback by other applications. For example, if you develop an application that records sound using the microphone, you may want to stop playback of music running in the iPod application.

The second approach is to mix audio from your application with audio from other running applications, such as the built-in iPod application. This approach treats audio more as an enhancement than a requirement, and is preferable for applications that don’t focus primarily on audio. Using the sound recorder example, you might allow the iPod to continue to play music so that users can record themselves singing along to a song.

You will notice a recurring theme in deciding how to handle various user experience concerns. User expectation should be a particularly significant consideration. Important questions to ask yourself are:

• How should my application behave if the phone rings?

• How should my application treat music that is playing when the application is launched?

• How should my application treat phone calls occurring when the application is launched?

• What is the best default behavior where sound effects are concerned?

• What controls should play sound?

• Should the application respect global settings regarding sound effects?

• Does the core function of the application depend on users hearing all audio?

• Are only some sounds essential for the proper use of the application?

The simplest way to play short audio on the iPhone is to use System Sound Services. All sounds played through System Sound Services will mix with audio from other applications and will obey the standard behavior around the hardware Ring/Silent switch and volume rocker. System Sound Services should be used for playing non-essential audio effects, such as startup or alert sounds.

There are two functions for playing sounds through System Sound Services. They are almost identical in their use and operation. The difference is that one function will trigger a vibration in lieu of playing audio if the phone is silenced, whereas the other will not. When choosing these functions, developers should take care to examine user expectations:

void AudioServicesPlaySystemSound (SystemSoundID inSystemSoundID)

Play a sound resource as identified by a SystemSoundID. If the Ring/Silent switch is set to silence the ringer, the sound will not play. This option is preferable for all short, non-essential sounds, such as sound effects for controls or game feedback sounds.

void AudioServicesPlayAlertSound (SystemSoundID inSystemSoundID)

Play a sound resource as identified by a SystemSoundID. If the Ring/Silent switch is set to silence the ringer, the phone will briefly vibrate. This function should be used only when it’s vital to communicate to the user that an event has occurred, because vibration is a disruptive form of feedback that can’t be ignored. If a user has silenced their phone, they likely want to avoid interruptions. Interruptive errors are a good use of this function.

The following example plays an audio response to a network operation. If there is an error, an alert sound—or vibration, if the device is silenced—is played. In a real application, you would probably cache the SystemSoundID variables in your instance:

// Assumes a declaration of instance variables:
// SystemSoundID successSound, errorSound
- (void)performAudioNotificationForStatus:(BOOL)success
{
	NSString *successPath = [[NSBundle mainBundle]
		pathForResource:@"success" ofType:@"caf"];
	NSString *errorPath = [[NSBundle mainBundle]pathForResource:@"failure"
		ofType:@"caf"];
	OSStatus error;
	if(success){
		if(successSound == nil){
			error = AudioServicesCreateSystemSoundID(
				(CFURLRef)[NSURL fileURLWithPath:errorPath],
				&successSound
			);
		}
		AudioServicesPlaySystemSound(successSound);
	}else{
		if(errorSound == nil){
			error = AudioServicesCreateSystemSoundID(
				(CFURLRef)[NSURL fileURLWithPath:successPath],
				&errorSound
			);
		}
		AudioServicesPlayAlertSound(errorSound);
	}
}

- (void)dealloc
{
	AudioServicesDisposeSystemSoundID(successSound);
	AudioServicesDisposeSystemSoundID(errorSound);
	[super dealloc];
}

					  

If an application needs to play sounds longer than 30 seconds or play non-trivial audio, you can take advantage of several more advanced audio playback frameworks and tools:

OpenAL

Used especially for spatial sound programming. OpenAL is popular in game programming or in applications used for playing audio with spatial variation. The OpenAL FAQ for iPhone OS, bundled with the iPhone SDK documentation, is a good starting point for learning about OpenAL.

Audio Queue Services

Used as a straightforward way to record or play audio. The Audio Queue Services Programming Guide, bundled with the iPhone SDK documentation, is an excellent resource for learning more about Audio Queue Services.

AVAudioPlayer class

The AVAudioPlayer class, bundled with a delegate protocol, was added to iPhone OS 2.2 as part of the AVFoundation framework. The AVAudioPlayer class provides robust functionality around playing audio files. Developers can use the class to play multiple sounds, play sounds with distinct levels, loop sounds, seek through files during playback, and gather data that can be used in visualizing the power levels of audio channels. The AVAudioPlayerDelegate protocol defines methods used for handling interruptions to playback, such as from an incoming phone call, along with expected error, start, and stop events.

Audio units

Audio units are audio processing plug-ins. You can read about audio units in the Audio Unit Component Services Reference in the iPhone SDK documentation.

When using any of these more complex audio programming resources, you can control the way your sounds mix with other applications using audio sessions. The Audio Session Services Reference documentation, included with the iPhone SDK, describes the use of audio sessions. Audio sessions are used to define the conditions of operation for audio handling on the iPhone OS. Each application is given an audio session object that acts as an intermediary between the application and the operating system. Developers set properties of the audio session for their application to define how the iPhone OS should make decisions about audio playback in response to the myriad conditions that affect the mobile user experience. Typical events with consequences for audio playback are:

• Incoming phone calls

• Terminating phone calls

• Playing music with the iPod application

• User interaction with the Ring/Silence switch

• Users switching between the headphone jack output and the built-in speaker

• Users interacting with the Sleep/Wake button

• The auto-sleep timer expires and the device goes to sleep

There are lots of ways to customize audio sessions, and Apple has provided several preset collections of settings—called categories—that solve the most common use cases. Though only one category can be assigned to an audio session at any given time, the category is mutable and can be changed for different circumstances. The audio session categories and the corresponding UInt32 constant used to set the category are:

UserInterfaceSoundEffects

Used for applications that provide simple, short, user-initiated sound effects, such as keyboard taps or startup sounds. Sounds will mix with audio from other applications and will obey the Ring/Silent switch. Identified by kAudioSessionCate⁠gory_UserInterfaceSoundEffects.

AmbientSound

Used for slightly longer sounds that can safely obey the Ring/Silent switch without hindering the core functionality of the application. Sounds will mix with audio from other applications and will obey the Ring/Silent switch. Identified by kAudioSessionCategory_AmbientSound.

SoloAmbientSound

Used similarly to the AmbientSound category, except that the audio does not mix with audio from other applications. Identified by kAudioSessionCategory_SoloAmbientSound.

MediaPlayback

Used for audio or audio/video playback. All other applications are silenced. MediaPlayback sounds do not obey the Ring/Silent switch. Identified by kAudioSessionCategory_MediaPlayback.

LiveAudio

Used for applications that simulate music instruments, audio generators, or signal processors. All other applications are silenced. LiveAudio sounds do not obey the Ring/Silent switch. Identified by kAudioSessionCategory_LiveAudio.

RecordAudio

Used for applications that record audio. All other applications are silenced. RecordAudio sounds do not obey the Ring/Silent switch. Identified by kAudioSessionCate⁠gory_RecordAudio.

PlayAndRecord

Used for applications that both play and record audio. All other applications are silenced. PlayAndRecord sounds do not obey the Ring/Silent switch. Identified by kAudioSessionCategory_PlayAndRecord.

Setting a category isn’t difficult. This example sets a session to the AmbientSound category:

- (void)prepareAmbientAudio
{
	OSStatus result = AudioSessionInitialize(NULL, NULL, NULL, self);
	if(result){
		NSLog(@"Ambient audio could not be prepared. Consider this suspicious.");
	}else {
		UInt32 category = kAudioSessionCategory_AmbientSound;
		result = AudioSessionSetProperty(
			kAudioSessionProperty_AudioCategory,
			sizeof(category),
			&category
		);
		if(result){
			NSLog(@"Could not set audio session category. This is no good. Fix it!");
		}else {
			result = AudioSessionSetActive(true);
			if(result){
				NSLog(@"Could not set audio session to active status. Fix it!");
			}
		}
	}
}


					  

Audio programming possibilities on the iPhone are shockingly robust for a mobile device. Of course, with great power comes great responsibility—so be considerate of user expectations when adding audio to applications, but don’t hesitate to create innovative applications. Let users decide how and when audio should contribute to their experience.

NOTE

Like sound, vibration can be either an excellent enhancement or an annoying deal breaker for users of an application. The mobile HIG describes user expectations as they relate to vibration, and Apple advises developers to use vibration only when appropriate. You can trigger vibration by calling the AudioServicesPlaySystemSound function and passing a constant SystemSoundID called kSystemSoundID_Vibrate:

- (void)vibrate
{
    AudioServicesPlaySystemSound(kSystemSoundID_Vibrate);
}