Integration One: November 2014

Ross Mason on Wednesday, August 10, 2011

Real-time Web with the new Salesforce streaming API

Its about time the Web became more event-driven. We have had AJAX for many years enabling events between server and browser, but on the backend we are still polling data. With the explosion of public APIs from SaaS, Social Media and Infrastructure Apps, more and more applications are written by composing web APIs. Developers often need to call a API to get data updates, only to find that nothing has changed. Streaming APIs provide a more elegant solution to polling allowing developers to subscribe to changes they are interested in.

Salesforce recently announced their new Streaming API that given a SOQL query can associate it with a topic that applications can subscribe to. This means any application subscribed on that topic will receive updates as they happen in realtime.

Streaming Schreaming. So What?

Realtime data over the web sounds good but how much improvement is there over polling? Well for starters polling isn’t very elegant, asking an application for changes is hugely inefficient. Secondly, polling APIs mean that you have rate limit restrictions, these restrictions exist to reduce load on the APIs infrastructure, and rate limiting on SaaS applications is often restricted to the number of calls a day which means you usually cannot even get close to real time without paying a lot more. Finally, the REST paradigm (the one used by 90% of public APIs) is not well suited for realtime, Streaming APIs work around the limitations to allow data to be pushed to the client in the same way AJAX does i.e. CometD and Web Sockets.
If you think back to days when AJAX term was coined, many people didn’t see the true value of what the technology offered until the first Web 2.0 web sites started to get attention. To the less informed, AJAX seemed to just remove the need for the user to refresh their page in the browser. I believe streaming APIs is a major step in moving towards an event-driven Web.

The SalesForce Streaming API

Salesforce is not the first company to announce a streaming API, Twitter, Facebook and others started to emerge almost a year ago with streaming APIs. What’s different about Salesforce is that they are the shining light for most other SaaS companies (after all they pretty much defined the category) and they’ve started the ball rolling and others will follow suit. Getting our application data at realtime isn’t something that we’ve had before, it will change the dynamics of the applications we build.
Of course, there is a big win for Salesforce with their streaming API, with over 50% of all traffic going through their APIs, that’s a lot of applications nagging their infrastructure asking for updated information. The number of requests that return no data could be as high as 60%. That’s a lot of inefficient processing. The Salesforce platform can be greatly optimised if people start using the streaming APIs since a single query result set could be filtered to serve hundreds or thousands of customers.

Streaming APIs and Mule

Mule is the first integration platform to support the SalesForce streaming API, and to test it out we create a simple demo application with Salesforce Chatter and Twilio running on our cloud platform; Mule iON.

The Demo is pretty simple, when you post to someone’s chatter wall hashtags can be used to perform actions. In this case if you post to the wall and include the #now hashtag the recipient will will be notified of your message through SMS.

To make this possible we created a Salesforce custom object (SMSNotification) that includes the user’s mobile number to send SMS messages on. We created a trigger to update this custom object with FeedItems posted on the users Chatter wall. We then created a topic called ‘/SMSNotificationStream‘ that selects everything in the SMSNotification object. So when a new post on a wall is made the SMSNotification object gets updated with the item and has a mobile number associated with it that is used to send SMSes.
One of the nice things about this demo is that is super easy, there is only a few lines of Mule XML to make it work:

      
<!-- configure the cloud connectors -->

<sfdc:config name="mySF" username="${sf.user}" password="${sf.password}" securityToken="${sf.securityToken}"/>

<twilio:config name="myTwilio" accountSid="${twilio.sid}" authToken="${twilio.authToken}"/>

<flow name="sendSmsNotifications">

    <sfdc:subscribe-topic config-ref="mySF" topic="/SMSNotificationStream"/>

    <regex-filter pattern="\#now"/>

    <twilio:send-sms-message config-ref="myTwilio"

                             from="+14155555555"

                             to="#[map-payload:To__c]"

                             body="#[map-payload:Text__c]"/>

</flow>

            view raw
            gistfile1.xml
            hosted with ❤ by GitHub

You’ll see that the flow starts with ‘sfdc:subscribe-topic‘ this initiates a streaming connection to Salesforce and will trigger the flow when new data arrives.
The filter is used to detect hashtags. Here we only look for #now, but we could look different ones and react to each separately
The Twilio ‘send-sms-message‘ command extracts the number and message to SMS from the payload received from Salesforce
This configuration can be dropped in to Mule iON and happily sit there listening and reacting to Chatter; like a message switchboard in the cloud
This all happens in realtime!

The screenshot below shows what a wall message looks like in Salesforce Chatter and the SMS message being received via Google Voice.

Let me at it!

Right now the Salesforce Streaming API is in beta and the streaming connector for Mule will not be officially released until Mule 3.2 in a few weeks. If you want to try it out before that please let me know (tweet or email ross at mulesoft).
Follow: @rossmason, @mulejockey

Source : Mule Blog

Ross Mason on Wednesday, November 28, 2012

The Era of Open APIs Demands a New Type of Application

With the rapidly increasing adoption of SaaS, integration Platform as a Service (iPaaS) has become the preferred way to connect SaaS applications. However, with the explosion of Open APIs on the Web connecting APIs together is becoming the norm for application development. However, typical application containers and even application PaaS offerings don’t help in this new era where applications compose APis together from many different sources.

In the iPaaS world composing APIs together is modus operandi. An application built on an iPaaS is focused on connecting 2 or more systems together via APIs in order to synchronizing data between them. However, we’ve taken this new breed of applications much further with the concept of Integration Apps.
Integration Apps exist in a world where everything needs to connect. Open APIs define thousands of new endpoints for exchanging data and leveraging new functionality and Integration Apps are optimized for dealing with working with many different data sources, focusing on composition rather than just coding.
To explain Integration Apps lets take a look at the anatomy of a traditional Web application.

This will be familiar with any developer; it’s the traditional 3-tier application model that defines how most applications to date have been built. There are a few problems with this architecture in today’s API-centric world:

The database has traditionally been the source of truth, but now applications work with many data sources. Increasingly apps need to read and write from APIs from different 3rd party providers.
The App Server is just an HTTP container. It doesn’t provide much in the way of capabilities other than hosting code and mapping HTTP requests to an application.
Custom logic is a big bucket where the application logic resides. This is where data access and application code is hosted. Developers often use open source frameworks such as Ruby on Rails and the Spring Framework to make creating applications easier. But these frameworks don’t provide much for dealing with connecting to lots of data sources or working with different data formats.
Traditional web apps are user-focused, however new applications need to cater for machines too. Increasingly, more people think about building applications API-first. That is creating an API for the application that can be consumed by JavaScript, native mobile applications and other applications.

When people move their traditional applications to the cloud, hosting them on Platform as a Service offerings such Heroku, CloudBees or Open Stack the picture looks very similar.

Even when running on PaaS, applications don’t change. This is deliberate since PaaS has focused on getting existing applications into the cloud. However, this does nothing for dealing with the explosion of Open APIs.

Introducing Integration Apps

In contrast Integration Apps embrace the need to connect to APIs and work with multiple data sources, data formats and mediating between different applications. The iPaaS platform provides completely new types services for dealing with interactions for other remote systems including monitoring, tracking, governance and mediation. This is needed because Integration Apps take an message-driven approach to connecting APIs. This means rather than making only synchronous calls in code, interactions are defined through messages being passed between components. This will be familiar to developers that understand newer languages such as Node.js or Scala were messages are passed to listeners or between Actors.

With Integration Apps there are more capabilities built in so the developer doesn’t have to do heavy lifting.

Connectivity is focused on working with lots of different APIs, this layer manages the security, session management and monitoring of connections.

The ‘Custom Logic’ is joined by new capabilities to deal with composing different APIs together using Orchestration. Data Mapping is needed since the data exchanged via APIs comes in differentformats, so being able to work with XML, JSON, RSS, ATOM, CSV and legacy formats is really important. There is also more focus on Error handling since interactions between different systems needs to be clear and visible.

iPaaS offers the same services as PaaS such as database, storage. But because these applications are message-driven there is a whole new set of platform services that help you track information between systems, set up alerts and error handling with message replay.

Integration Apps don’t only connect with Open APIs on the Web, often connecting with on-premise applications and data sources is needed so the notion of a data gateway is important.

One element missing from the above picture is the User Interface. This is because increasingly, applications are being built to serve machines not people, making UI optional. Synchronizing data between a few applications doesn’t need human interaction; nor do many automated business processes. However, Integration Apps natively support publishing ReST APIs so that other applications and mobile devices can interact with an Integration Apps.

Integration Apps in the wild

MuleSoft already offer public Integration Apps that synchronise data between SaaS applications or between SaaS and on premise applications like SAP. The most well known Integration App out in the wild is dataloader.io. This is an Integration Apps that allows users to upload data into SalesForce – a very common task. The dataloader.io iApp is the only cloud-based solution for Salesforce and has become very popular taking the number 1 spot on the Salesforce AppExchange. This Integration App looks very much like the diagram above: it has a JavaScript UI that talks to the app via its ReST API, the Integration Apps uses a mix of Orchestration, Data Mapping and custom logic to allow users to create jobs for loading data into Salesforce.
Without iPaaS and Integration Apps capabilities this application would have taken months to build from the ground up with all the monitoring, management, error handling and connectivity, instead we built it in 4 weeks. And now it serves 1,000s of Salesforce users per day.

Faster, more Productive

The combination of iPaaS and Integration Apps is very powerful, and enables a new type of application that responds to changes in real-time. Pushing more services and functionality into the platform drives a configuration over coding approach which means developers can focus on composing their application rather than coding everything from the ground up. If necessary the developer can insert custom logic into their App but for most scenarios it isn’t necessary, with orchestration and data mapper providing the tools to work with APIs and different data formats.
The applications that take advantage of these new capabilities will create richer, more engaging applications. This new breed of applications will further fuel the Open API explosion adding new APIs that can be consumed people and machines. Open APIs power a world where everything needs to connect, its time for a new platform that embraces this.
Follow: @CloudHub, @MuleSoft, @rossmason

Source: Mulesoft Blog