BizTalk 2006 : Building a Resequencing Aggregator

1/2/2012 11:16:48 AM
So now that you know how to solve the problem of receiving messages out of order, you canproperly receive messages, resequence them, and send them in the correct order to their destination. But what about batch scenarios—what happens in the case where you need to combine the received messages, order them, and then aggregate them into a batch? This need is quite common in the EDI world where batching has existed for years. Also, in many legacy system integration projects, large aggregated data files are needed to load downstream systems with data. For example, let's look at the company ABC example. Assume that orders received from the FTP bulk upload are compressed and need to be sent as a batch to the fulfillment system, which only understands flat text files. Luckily you have all the pieces you need to solve this except one. To recap, you can
  1. Use the Unzip component to decompress the files.

  2. Use the resequencing pipeline and pipeline component from the resequencing example to add the sequence information.

  3. Use the resequencing orchestration to order them properly.

However, you still have no way to A) combine the messages into one message, and B)turn the sorted list of the messages received into a flat file. Luckily, with BizTalk 2006 this becomes a trivial task. There are, however, two possible solutions to this problem depending on how the messages are received. If the messages are not received as described, but arereceived as independent Interchanges, you need to use the Resequencer pattern described previously. If they are indeed received as a batch, a more elegant solution is available.

Solution 1: Status Quo—Messages Received As Independent Interchanges

If the messages are received independently, a simple change is required to the resequencingorchestration to allow the pattern to work. All you need to do is not send the messages as they arrive and wait to receive all messages before sending them. Once all the messages in the batch are received, you can use the new Call Pipeline from Orchestration code that ships with BizTalk 2006 to send the SortedList to a send pipeline that contains a flat-file Assembler component. So long as the messages are defined with a schema that uses the flat-file extensions of BizTalk, the pipeline will return a text message with all the messages inside aggregated in their proper order. The orchestration will look something like Figure 6-9.

The code for the Call Send Pipeline Expression shape will look like this:

//Initialize pipeline output
AggregatedOutMessage = null;
//Add the messages to the list via a component
PipelineHelper.AddSortedMessages(SortedList, SendPipeMsg)

You will need to use a Construct shape, since the shape is creating new messages via thecomponent. The SendPipeMsg is of type Microsoft.XLANGs.Pipeline.SendPipelineInputMessages. The reason why you need to add these messages to the SendPipeMsg via a component is because within the component you will need to loop through each of the messages stored in the sorted list, and add them one at a time to the SendPipeMsg to ensure that the order is properly maintained. Alternatively, you could use a Loop shape in the orchestration, but this is much easier done in code. Once this is done, the send pipeline is called, the messages are aggregated, and the complete message is sent to the send port.

Aggregating resequencing orchestration

Solution 2: Not So Status Quo—Messages Received As a Batch

In the resequencing aggregator outlined earlier, the messages are received as a completeInterchange. They may be out of order when stored inside the ZIP file, but they are completely batched as a whole. In this case, you actually don't need a resequencing orchestration and can implement this solution entirely within a pipeline and use content-based routing.

Let's reexamine our list of pipeline components from previous examples:

  • Unzip component: The Unzip component extracts the ZIP file contents and createsa multipart message based on the files included.

  • Resequencing Decoder: The component examines a message and determines whereit fits within a unique sequence.

What if you modified the Resequencing Decoder to handle multipart messages? In thisscenario, the component would have additional smarts to know that multipart messages can also be resequenced. In this case, the component would simply loop through each of the message parts, examine its data, determine what number it is in the sequence, and promote that property along with the unique sequence ID and the last message in the sequence property. Also in this case, you need to use another message part feature—they have a property bag. So instead of promoting the values to the message context of the message, you simply write these values as key/value pairs to the parts property bag. Once all the message parts are resequenced properly, the message can flow through BizTalk like any other message. The send port hosting the FlatFileAssembling component would be subscribed for messages of this type, parse the incoming messages, and output one aggregated message, which is sent to the downstream system.

While the preceding example is technically easier to implement, it has some drawbacks. First, as stated earlier, since pipeline execution is stateful, should the pipeline fail for whatever reason, processing stops. You will manually need to have some error-handling logic and subscribe to the failed message for processing. This is explored in the next section. Also, care should be taken to properly performance test the solution. If the ZIP file is large and contains a high number of individual transactions, the CPU on this machine could be affected. Make sure that you test this scenario properly before implementing it.

  •  Windows System Programming : Listing Registry Keys and Contents
  •  Windows System Programming : Registry Management
  •  .NET Debugging : PowerDbg (part 2) - Send-PowerDbgCommand & Extending PowerDbg
  •  .NET Debugging : PowerDbg (part 1) - Installing PowerDbg & Analyze-PowerDbgThreads
  •  Sharepoint 2010 : Business Connectivity Services Deployment Types (part 3) - Configure Indexing & Performing a Search
  •  Sharepoint 2010 : Business Connectivity Services Deployment Types (part 2) - Creating a Profile Page to Display BCS Results
  •  Sharepoint 2010 : Business Connectivity Services Deployment Types (part 1) - Code-Based Solutions
  •  Sharepoint 2010 : BCS Architecture - Presentation & Core Components
  •  Collaborating via Web-Based Communication Tools : Evaluating Instant Messaging Services
  •  Collaborating via Web-Based Communication Tools : Evaluating Web Mail Services
  •  Developing the SAP Data Center : Data Center Physical Requirements
  •  Developing the SAP Data Center : Introducing the SAP Data Center
  •  Inventory of Broadband Phone Services
  •  Parallel Programming : Task Relationships (part 2) - Parent and Child Tasks
  •  Parallel Programming : Task Relationships (part 1) - Continuation Tasks
  •  BizTalk 2006 : Handling Ordered Delivery
  •  BizTalk 2006 : Implementing Dynamic Parallel Orchestrations
  •  Windows System Programming : The Registry
  •  Windows System Programming : File Locking
  •  SharePoint 2010 : Security - Secure Store Service & Using SSS with BCS
    Top 10
    Nikon 1 J2 With Stylish Design And Dependable Image And Video Quality
    Canon Powershot D20 - Super-Durable Waterproof Camera
    Fujifilm Finepix F800EXR – Another Excellent EXR
    Sony NEX-6 – The Best Compact Camera
    Teufel Cubycon 2 – An Excellent All-In-One For Films
    Dell S2740L - A Beautifully Crafted 27-inch IPS Monitor
    Philips 55PFL6007T With Fantastic Picture Quality
    Philips Gioco 278G4 – An Excellent 27-inch Screen
    Sony VPL-HW50ES – Sony’s Best Home Cinema Projector
    Windows Vista : Installing and Running Applications - Launching Applications
    Most View
    Bamboo Splash - Powerful Specs And Friendly Interface
    Powered By Windows (Part 2) - Toshiba Satellite U840 Series, Philips E248C3 MODA Lightframe Monitor & HP Envy Spectre 14
    MSI X79A-GD65 8D - Power without the Cost
    Canon EOS M With Wonderful Touchscreen Interface (Part 1)
    Windows Server 2003 : Building an Active Directory Structure (part 1) - The First Domain
    Personalize Your iPhone Case
    Speed ​​up browsing with a faster DNS
    Using and Configuring Public Folder Sharing
    Extending the Real-Time Communications Functionality of Exchange Server 2007 : Installing OCS 2007 (part 1)
    Google, privacy & you (Part 1)
    iPhone Application Development : Making Multivalue Choices with Pickers - Understanding Pickers
    Microsoft Surface With Windows RT - Truly A Unique Tablet
    Network Configuration & Troubleshooting (Part 1)
    Panasonic Lumix GH3 – The Fastest Touchscreen-Camera (Part 2)
    Programming Microsoft SQL Server 2005 : FOR XML Commands (part 3) - OPENXML Enhancements in SQL Server 2005
    Exchange Server 2010 : Track Exchange Performance (part 2) - Test the Performance Limitations in a Lab
    Extra Network Hardware Round-Up (Part 2) - NAS Drives, Media Center Extenders & Games Consoles
    Windows Server 2003 : Planning a Host Name Resolution Strategy - Understanding Name Resolution Requirements
    Google’s Data Liberation Front (Part 2)
    Datacolor SpyderLensCal (Part 1)