Bridging the Gap: Replicating MuleSoft's Scatter-Gather in Workato

Introduction

When migrating from MuleSoft to Workato, one of the primary challenges is replicating MuleSoft's Scatter-Gather, a robust parallel (or multi-threaded) invocation feature. Scatter-Gather enables simultaneous processing of multiple requests, which can significantly improve performance in scenarios requiring high efficiency.

Although Scatter-Gather is not a built-in feature in Workato, this functionality is often essential during migration, especially for enterprise integration teams. The feature may be used to execute database select calls or multiple GET calls on an API to retrieve large record sets efficiently. Additionally, it is important for synchronous APIs with substantial records from various sources to have a quick response time, making the scatter-gather or parallel invocation pattern essential to meet the SLA/expectations of the API consumers. 

This article outlines methods to replicate MuleSoft's Scatter-Gather in Workato through custom scripts or connector development, and additional architectural design considerations to enable parallel processing.

Understanding the Scatter-Gather of MuleSoft

Scatter-Gather is a powerful parallel processing technique used in integration platforms like MuleSoft, enabling concurrent execution of multiple tasks. This feature improves both performance and scalability.

Here’s how it works:

1. The Scatter-Gather component receives a Mule event and distributes it to each processing route.
2. Each route begins execution in parallel, with each processor returning a Mule event after completing its task ( which could be the same Mule event without modifications or a new Mule event created by the processors in the route as a result of the modifications applied).
3. Once all routes have finished processing, the Scatter-Gather component aggregates the results into a new Mule event, passing it to the next component.

Challenges in Workato

While Workato does not have a native Scatter-Gather feature, similar functionality can be implemented through custom connectors or JavaScript. However, before implementing any solution, it's essential to analyze your use case and answer key questions:

• Is parallel invocation truly necessary?
• Will parallel execution significantly improve performance, or is sequential processing sufficient?

Understanding the Use Case

Before diving into implementation, it's crucial to analyze your use case. Consider factors like:

• Response Time Requirements: Are there strict latency constraints?
• Data Volume: What data load will each call handle??
• API Frequency: How often is the API triggered?

For example, consider an integration project where Scatter-Gather was used to optimize the performance of a synchronous API. This API made three GET calls: one for employee asset entries and two for enriching the data. After analyzing the response times (one call took 2000ms while others took 150ms and 200ms), a design decision was made to switch to sequential execution, as there was no significant gain from parallel execution.

For asynchronous tasks, such as scheduled jobs or data syncs, the benefits of parallel execution might not justify the complexity it introduces. For synchronous APIs, where response times are critical, parallel execution might be more beneficial.

Replacing Scatter-Gather in Workato

If a parallel invocation is deemed necessary, here are the main approaches for replacing Scatter-Gather in Workato:

1. Bulk or Batch Operations: If the target API supports bulk or batch operations, this can replace the need for parallel calls. For instance, a ServiceNow API used Scatter-Gather in MuleSoft to fetch tickets of different types (interactions, incidents, service requests). In Workato, a single Batch API request to ServiceNow, handled multiple ticket types in one go, thereby simplifying the integration.
   
   
2. JavaScript Promise() for Parallel HTTP Calls: For straightforward use cases, JavaScript’s Promise() can handle parallel HTTP calls. This allows Workato to fetch data in parallel and aggregate the results. Here’s an example of the code:
   
   

   // Example JavaScript code for parallel invocation
   const fetchAPI1 = fetch('API_URL_1');
   const fetchAPI2 = fetch('API_URL_2');
   const fetchAPI3 = fetch('API_URL_3');
   
   Promise.all([fetchAPI1, fetchAPI2, fetchAPI3])    
   .then((responses) => Promise.all(responses.map((res) => res.json())))    
   .then((data) => {        
       // Combine and process data from all API responses        
       const combinedData = {            
           dataFromAPI1: data[0],            
           dataFromAPI2: data[1],            
           dataFromAPI3: data[2],        
       };        
       console.log(combinedData);    
   });

3. Connector SDK for Multi-Threaded Operations: For more complex scenarios, you can build a custom connector using Workato's Connector SDK. This approach allows you to create multi-threaded actions to send HTTP requests in parallel across multiple threads. Additionally, you can create a service facade to orchestrate and route requests to various databases or applications.
   Example: In MuleSoft, a /current-user/tasks endpoint aggregated tasks across multiple databases and applications (e.g., ServiceNow, Jira). In Workato, this was replicated by creating recipe functions for each task source, exposing a service facade API, and using the Connector SDK for parallel invocation.
   
   Mule scatter-gather flow:
   
   

   MuleSoft Scatter-Gather Flow

   
   Workato multi-threaded action architecture:

Here’s a table summarizing the comparison between JavaScript (Promise) for Parallel HTTP and Connector SDK for multi-threaded actions:

Feature	JavaScript (Promise) for Parallel HTTP	Connector SDK: Multi-threaded Action
Implementation	Requires writing custom JavaScript code using Promise()	Built-in support for multi-threading via the Connector SDK
Custom Development	Custom coding is required for everything (error handling, etc.)	SDK provides more structure but requires custom connector development
Debugging	Limited debugging capabilities	Enhanced debugging tools via the Connector console debugger
Error Handling	Manual error handling needed in JavaScript	Advanced error handling is built into the SDK with retry mechanisms
Throttling & Thread Control	No built-in controls for throttling	Fine-grained control over threads, throttling, and execution limits (using an additional service facade API collection)
Scalability	Not ideal for large-scale, complex integrations	Scalable with control over the maximum number of threads and throttling by using the service facade collection
Setup Complexity	Simple to set up for basic parallel execution	Requires more setup and architectural planning for complex workflows
Flexibility	Lightweight and flexible for simple use cases	More suited for complex, multi-service integrations
Suitability	Best for small, simple parallel HTTP calls	Ideal for large, complex, and high-performance integrations

This table highlights the pros and cons of each approach, helping to determine which is more suitable for a given scenario. In addition to these comparisons, there are a couple of important considerations:

• Licensing Implications: Replicating Scatter-Gather through API recipes may result in additional licensing costs, especially when exposing service facade endpoints.
• API Concurrency and Sizing: High-volume transactions require close attention to API concurrency limits. Ensure that your concurrency settings can handle the transaction volume to avoid performance bottlenecks.

By addressing these considerations, you can ensure that your implementation is both scalable and cost-effective.

Conclusion

Migrating Scatter-Gather functionality from MuleSoft to Workato requires careful evaluation of the use case and performance requirements. Although Workato lacks a native Scatter-Gather feature, it can be replicated through bulk operations, JavaScript Promise(), or the Connector SDK. The key is to analyze whether parallel invocation is essential and select the most suitable method to implement it in Workato.

Ready to optimize your integration performance?

At Twenty20 Systems, we specialize in helping businesses migrate and optimize their integration workflows. Whether you're transitioning from MuleSoft to Workato or need support with advanced parallel processing, our team can help tailor a solution to meet your enterprise needs.

Contact us or visit our website to learn more about our migration solutions and how we can help you streamline your integration processes.