Simulating SOA : Interpreting the Results of a Simulation

In this section, we study the results of running a simulation of the scenarios discussed above. The following table summarizes the results for the first scenario:

Test	Settings	Results
-Sequential Sync	Duration-10 minutes Arrival Rate-2 seconds Time Per Service-500 milliseconds fixed + 500 milliseconds exponential BPELIN Poll-100 milliseconds	BPELIN depth rises to 111 by the end. There are 297 arrivals, but only about 2/3 (185) make it out of the queue to create process instances. Average cycle time is 3.1 seconds, all of which is burst time.
-Parallel Sync	Duration-10 minutes Arrival Rate-2 seconds Time Per Service-500 milliseconds fixed + 500 milliseconds exponential BPELIN Poll-100 milliseconds	Similar behavior. BPELIN depth rises to 125. Only 177 instances started out of 304 arrivals. Average cycle time of 3.3 seconds is all burst.
-Async	Duration-10 minutes Starter Arrival Rate-2 seconds Async Reply Arrival Rate-500 milliseconds fixed + 500 milliseconds exponential BPELIN Poll-100 milliseconds BPELOUT Poll-300 milliseconds.	Better than 1a and 1b in all respects: BPELIN depth maxes at six; BPELOUT has average depth of 3.1; 288 instances started out of 289 arrivals; average cycle time is 1.9 seconds, only 320 milliseconds of which is burst.

Scenario 1, which explores the benefits of asynchronous invocations, has three variants: Process 1a invokes synchronously three expensive services in sequence; Process 1b invokes these services synchronously in parallel; Process 1c invokes them asynchronously and waits for their replies in parallel. The simulated duration for each run is ten minutes, and new processes are started with an exponential mean of two seconds. The duration of a service call is 500 fixed milliseconds plus additional time that is exponential with a mean of 500 milliseconds; thus, the average duration of a service call is one second, and the average duration of three service calls is three seconds.

The results confirm that time-consuming bursts cause the inbound queue to back-up and, therefore, slow the rate at which processes can be instantiated or receive intermediate events. Processes 1a and 1b average about three seconds of cycle time (or start-to-finish execution), which is not surprising given that these processes invoke in a single burst three services with an average duration of one second each. Even worse, after only ten minutes of simulated time, the BPELIN queue

accumulates over 100 pending requests (111 for 1a, 125 for 1b). Over the time, the backlog worsens, and if the simulation is run long enough, the queue maxes out.

Process 1c, by moving to an asynchronous model, is able to keep up with arrivals. 1c's average cycle time is just 1.9 seconds (low enough to keep BPELIN's queue depth to a handful), and it spends most of that time in an idle state, to be awakened only when one of its services sends back its result. The 320 milliseconds of burst time (which consists mostly of node traversal) is the only real work the process performs. The process spends the other 1.6 seconds waiting, and during this time, it does not tie up the inbound queue.

Here are the results for scenario 2.

Test	Settings	Results
-Two Services (Sync)	Duration-10 minutes Arrival Rate-2 seconds Time Per Service-1.2 second exponential BPELIN Poll-100 milliseconds	Behavior is similar to that of 1a. BPELIN queue depth climbs to 126 by the end. Of 283 arrivals, only 219 instances are started. Average cycle time is 2.6 seconds, and it's all burst.
-One Combined Service (Sync)	Duration-10 minutes Arrival Rate-2 seconds Time Per Service-2.4 seconds exponential BPELIN Poll-100 milliseconds	Similar to 1b: BPELIN depth rises to 112; only 199 instances created from 312 arrivals; average cycle time 2.9 seconds, all of which is burst.
-One Async Invoke, Two Responses	Duration-10 minutes Starter Arrival Rate-2 seconds Async Reply Arrival Rate-1.2 seconds exponential BPELIN Poll-100 milliseconds BPELOUT Poll-300 milliseconds	Much healthier: queue depths are negligible; 302 instances created from 303 arrivals; average cycle time is 2.2 seconds, with an average burst of 340 milliseconds.
-One Async Invoke, One Response	Duration-10 minutes Starter Arrival Rate-2 seconds Async Reply Arrival Rate-2.4 seconds exponential BPELIN Poll-100 milliseconds BPELOUT Poll-300 milliseconds	Similar to 2c. Average cycle time is 2.5 seconds; average burst is 140 milliseconds.

Process 2a in scenario 2 faces a problem similar to that of 1a: a new instance is created every two seconds, but the work that instance performs, in its one and only burst, averages over 2.4 seconds. As a result, the BPELIN queue fills up quickly and is unable to reduce its backlog. That work is to invoke two services in a sequence. The designer, faced with 2a's results, might consider reducing granularity and combining the two services into one. Process 1b is a refactoring of process 1a, designed to call this new, consolidated service synchronously. Let's assume the worst that the average service time for the new service (2.4 seconds) is the same, calling the original two services in sequence (1.2 seconds each). With these parameters, Process 2b performs as poorly in simulation as 2a.

As in scenario 1, the fix, implemented in Processes 2c and 2d, is to request the service asynchronously and sit idle until the response arrives. To keep the design coarse-grained, the request is still a request to perform two actions: the actions of the two services in Process 2a. Processes 2c and 2d differ on how they receive the response. 2c gets one response for each of the two actions, waiting for them together in a flow activity. Process 2d gets the two responses together, in a single step. As the simulation results indicate, both processes perform well, on par with Process 1c's results: low queue depths, and low burst times. As 2c has a more complex graph, its average burst time is higher than 2d's (340 milliseconds versus 140 milliseconds). On the other hand, 2c is faster to get a result, thus, getting a head start with follow-on processing.

The results for scenario 3 are the following.

Test	Settings	Results
-Huge Async Response	Duration-1 hour. Starter Arrival Rate-1 second Async Reply Arrival Rate-500 milliseconds fixed + 800 milliseconds exponential Reply Message Size-10,000 KB BPELIN Poll-200 milliseconds BPELOUT Poll-250 milliseconds BPELIN Max Size-50,000 KB Message Put Cost-5 milliseconds/KB	BPELIN queue twice fills to its maximum size of 50,000 KB, and there are two balks. The average cycle time is 51.1 seconds, but only 120 milliseconds of this is burst time. The average cost of putting messages in a queue is 25 seconds!
-Smaller Chunked Async Responses	Duration-1 hour. Starter Arrival Rate-1 second Async Reply Arrival Rate-500 milliseconds fixed + 500 milliseconds exponential Reply Message Size-2500 KB BPELIN Poll-200 milliseconds BPELOUT Poll-250 milliseconds BPELIN Max Size-50,000 KB Number of Chunks-4 Message Put Cost-5 milliseconds/KB	Much better. The BPELIN queue reaches a maximum size of 17,535 KB, and there are no balks. The average cycle time is 14.6 seconds, 360 milliseconds of which is burst. The average queue put cost is 6.3 seconds.

Scenario 3 deals with the problem of message size and considers the benefits of chunking. Process 3a makes an asynchronous service request and gets back, in a single step, a huge result. Process 3b makes the same request, but gets back the results in smaller chunks, using a while loop to get one after the next. The simulated duration for each process is one hour. In each case, the arrival events are exponential with a mean of one second. The size of the response message in 3a is 10,000 KB but 2,500 KB in 3b. In 3b, there are four chunks, the total size of which is equal to the size of 3a's single response message. The capacity of the BPELIN queue, on which these responses are delivered to the process, is 50,000 KB. The time required to put a message on a queue in both cases is five milliseconds per KB; a 10,000 KB message thus requires 50 seconds to queue!

The results confirm that chunking not only reduces the load on the BPELIN queue, but also results in faster cycle times. In 3a's simulation, huge messages twice fill the BPELIN queue to capacity, and hence, two messages fail to queue. For 3a, the average cost of putting a message on the queues is 25 seconds (it's lower than 50 seconds because several smaller messages are also queued along the way). As a result, the average cycle time is appallingly high: 51.1 seconds. In 3b's simulation, the average cycle time is much smaller-14.6 seconds-and BPELIN's maximum size is a more manageable 17,545 KB.