Windows Azure : Messaging with the queue - Understanding message visibility

2/17/2011 9:15:13 AM
A key aspect of a queue is how it manages its messages and their visibility. This is how the queue implements the message durability developers are looking for. The goal is to protect against a consumer getting a message, and then failing to process and delete that message. If that happened, the message would be lost, and this isn’t good news for any processing system.

Visibility timeouts and idempotency are the two best tools for making sure your messages are never lost. Understanding how these concepts relate to the queue and understanding the lifecycle of a message are important to the success of your code.

1. About message visibility and invisibility

Every message has a visibility timeout property. When a message is pulled from the queue, it isn’t really deleted; it’s just temporarily marked as invisible. The consumer is also given a receipt (called the pop receipt) that’s unique to that GetMessage() operation. The duration of invisibility can be controlled by the consumer, and it can be as long as 2 hours. If not explicitly set, it will default to 30 seconds. While a message is invisible, it won’t be given out in response to new GetMessage() operations.

As an example, let’s say a producer has placed four messages in a queue, as shown in figure 1, and we have two consumers that will be reading messages out of the queue.

Figure 1. Two consumers getting messages from a queue. A message is marked invisible when a GetMessage() operation is performed—just like a cloak of invisibility. Note that this effect times out after a while.

Consumer 1 gets a message (msg 1), and that message is marked invisible . Seconds later, consumer 2 performs a get operation as well. Because the first message (msg 1) is invisible, the queue responds with the next message (msg 2) .

Not long thereafter, consumer 1 finishes processing msg 1 and performs a delete operation on the message. As part of the delete operation, the queue checks the pop receipt consumer 1 provides when it passes in the message. This is to make sure consumer 1 is the most recent reader of the message in question. The receipt matches in this case, and the message is deleted.

Consumer 1 then does an immediate read, and gets msg 3. Consumer 1 fails to complete processing within the invisibility time window and fails to delete msg 3 in time. It becomes visible again.

Just at that time, consumer 2 deletes msg 2 and does a get. The queue responds with msg 3, because it’s visible again. While consumer 2 is processing msg 3, consumer 1 does finally finish processing msg 3 and tries to delete it. This time, the pop receipt, which consumer 1 has, doesn’t match the most recently provided pop receipt, which was given to consumer 2 when msg 3 was handed out for a second time. Because the pop receipt doesn’t match, an error is thrown, and the message isn’t deleted. You’ll likely see a 400 (Bad Request) error when this happens. The inner exception details will explain that there aren’t any messages with a matching pop receipt available in the queue to be deleted.

2. Setting visibility timeout

You can set the length of the visibility timeout when you get the message. This lets you determine the proper length of the timeout for each individual message.

When you specify the visibility timeout, you want to balance the expected processing time and how long it will take for the system to recover from an error in processing. If the timeout is too long, it will take a long time for the system to recover a lost message. If the timeout is too short, too many of your messages will be repeatedly reprocessed.

This leads us to an important aspect of queues in general, but specifically the Azure queue system.

3. Planning on failure

The Queue service guarantee is worded as promising that every message will be processed, at least once. You can see this “at least once” business in the previous scenario. Because consumer 1 failed to delete the message in time, the queue handed it out to another consumer. The queue has to assume the original consumer has failed in some way.

This is very useful because it provides a way for your system to take a hit (a server going down) and keep on going. Cloud architecture plans on failure and makes that central to the structure of the system. Queues provide that capability quite nicely.

The downside is that it’s possible that a consumer doesn’t crash but just takes longer to process the message than intended. In this case, you need to make sure that your processing code is either idempotent, or that it checks before processing each message to make sure it isn’t a duplicate copy. Because the message being reprocessed is actually the same message, its ID property will be the same. This makes it easy to check a history table or perhaps the status of a related order before processing starts.

This little bit of complexity might make you think about deleting a message as soon as you receive it—before you process it. Doing so is dangerous and unwise because there will be failure along the way, and when that happens, the message would be lost forever.

4. Use idempotent processing code

The goal of a messaging system is to make sure you never lose a message. No matter how small or large, you never want to lose an order, or a set of instructions, or anything else you might be processing.

To avoid complexity, it’s best to make sure your processing code is idempotent. Idempotent means that the process can be executed several times and the system will result in the same state. Suppose you’re working with a piece of software that tracks dog food delivery. When the food is delivered to the physical address, the handheld computer sends a message to your queue in the cloud. The software uploads the physical signature of the recipient to BLOB storage and submits an order-delivered message to the queue. The message contains the time of delivery and the order number, which happens to also be the filename of the signature in BLOB storage.

When this message is processed, the consumer copies the signature image to permanent storage, with the proper filename, and marks the order as delivered in the package-tracking database.

If this message were to be processed several times, there would be no detriment to the system. The signature file would be overwritten with the same file. The order status is already set to delivered, so we’d just be setting its status to delivered again. Using the same delivery time doesn’t change the overall state of the system.

This is the best way to handle the processing of queue messages, but it isn’t always possible. The next section will discuss some common queue-processing patterns, and some of them deal with working around this issue.

  •  Windows Azure : Messaging with the queue - Working with messages
  •  Windows 7 : Maintaining Your System Configuration (part 4) - Configuring Remote Access
  •  Windows 7 : Maintaining Your System Configuration (part 3) - Configuring User Profiles, Environment Variables, and Startup and Recovery
  •  Windows 7 : Maintaining Your System Configuration (part 2) - Creating or Joining a Homegroup & Viewing Hardware Settings
  •  Windows 7 : Maintaining Your System Configuration (part 1) - Configuring the Computer Name and Membership
  •  Windows Server 2008 : DHCP/WINS/Domain Controllers - Exploring DHCP Changes in Windows Server 2008 R2
  •  Windows Server 2008 : DHCP/WINS/Domain Controllers - Exploring the Dynamic Host Configuration Protocol (DHCP)
  •  Windows Server 2008 : DHCP/WINS/Domain Controllers - Understanding the Key Components of an Enterprise Network
  •  Windows Azure : Messaging with the queue - Working with basic queue operations
  •  Windows Azure : Messaging with the queue - Decoupling your system with messaging
    PS4 game trailer XBox One game trailer
    WiiU game trailer 3ds game trailer
    Top 10 Video Game
    -   Minecraft Mods - MAD PACK #10 'NETHER DOOM!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Minecraft Mods - MAD PACK #9 'KING SLIME!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Minecraft Mods - MAD PACK #2 'LAVA LOBBERS!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Minecraft Mods - MAD PACK #3 'OBSIDIAN LONGSWORD!' with Vikkstar & Pete (Minecraft Mod - Mad Pack 2)
    -   Total War: Warhammer [PC] Demigryph Trailer
    -   Minecraft | MINIONS MOVIE MOD! (Despicable Me, Minions Movie)
    -   Minecraft | Crazy Craft 3.0 - Ep 3! "TITANS ATTACK"
    -   Minecraft | Crazy Craft 3.0 - Ep 2! "THIEVING FROM THE CRAZIES"
    -   Minecraft | MORPH HIDE AND SEEK - Minions Despicable Me Mod
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 92 "IS JOE DEAD?!"
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 93 "JEDI STRIKE BACK"
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 94 "TATOOINE PLANET DESTRUCTION"
    -   Minecraft | Dream Craft - Star Wars Modded Survival Ep 95 "TATOOINE CAPTIVES"
    -   Hitman [PS4/XOne/PC] Alpha Gameplay Trailer
    -   Satellite Reign [PC] Release Date Trailer