Wednesday, July 12, 2017

Design-first with Ballerina

Download this technical paper as a PDF.

The Need for Design-First

For almost as long as engineers have been integrating disparate/distributed systems together, there have existed the need for a common language, A common vocabulary that can dictate and specify the boundaries of the interfaces, allowing engineers to focus on the internal processing logic while mitigating interoperability concerns.

Though other technologies existed preceding it, an argument can be made that it was with the emergence of SOA and WSDL that an industry wide effort was first made to standardise the design of the interfaces that systems can expose for integration purposes. WSDL is an interface definition language which provides a means of specifying an interface’s operations, messaging format and data types in a manner that is implementation independent, this is achieved through the language agnostic quality of XML, the base language upon which WSDL was built.
Interface definition languages are tightly coupled with the technologies they are intended to be used with and built upon. With the fallout of XML seen in the last decade, and the rise of new architectural paradigms such as REST and ROA, a void was created in the IDL space. Swagger(Open API) was one such specification written to address this very need.

Apart from acting as a contract, definitions be it WSDL or Swagger provide an alternative way of approaching interface creation through the act of defining the contract prior to implementation. This approach referred to as “Design-first, “Contract-First” or “API-First”(a more vendor specific incarnation) allows organizations better segregation between the high level business requirements and lower functional/non-functional requirements.


Choosing Design-First for API

In their 2017 research on the consequences of API adoption on organizational performance, Benzell, Lagarda and Alstyne classifies API as either closed or open. Closed API are meant for internal consumption, whereas Open API are meant for external consumption, be it B2B or B2C. The researchers find a value increase of 10% and a profit increase of 5% for API adopters, the increase in valuation was driven mainly by Open API[1].


When designing Open API, those API that are expected to be consumed by outsiders, be it independent developers or strategic partners, it is important to provide a good developer experience for the consumer, as it plays an important role in forming opinions on the organization and the consequent success or failure of the API. As a means of addressing these concerns organizations may turn to “Design-First” and Swagger. Vasudevan identifies the following three situations as being ideal for this approach,


When Developer Experience Matters - a well designed API is easy to understand and use. As such they can be integrated into a developer’s applications faster, this leads to better adoption.


When Delivering Mission Critical APIs - In a B2B interaction, as the API acts as a channel of communication between the organization and its partners, the API consumption experience plays a role in forming opinions about the organization and such a thought out API design is a must.


When Ensuring Good Communication - At design and implementation time, having a central human readable contract allows for development teams to be in sync and identify design issue at a much earlier stage in the design lifecycle.    


According to Vasudevan though “Design-First” approach is best suited for the situations mentioned above code-first approach is likely to produce a better time-to-market. [3]

Moreover, “Design-first” provides the organization a means of addressing the developer experience through the separation of implementation from design. Similar to the the requirement specification endeavours of the monolithic software systems that required engineers specialized in requirement analysis and elicitation, the responsibility of the design can be handed over to those who specialize in the area before the design skeleton be handed over to those who specialize in implementation. The end result of this separation of concerns being the creation of a polished interface that is more in line with the business requirements and is of higher quality. [4]

In its short period of existence, Swagger(Open API Initiative) has amassed a healthy community reflected by its repository activity and usage in an estimated 10,000 production deployments. [2] In his comparison of Swagger with other API design tools Stowe finds it to have greater community support, language support and API building tools.[5] Using its many language support and API building tools/projects users may quickly proof their designs or generate implementation specific skeleton code.

Design First flow with Swagger

Design-first for Integration

Organizational information systems have moved away from being monolithic and disparate in favour of architectures that provide better agility such as Service Oriented Architectures and Microservices based Architectures. With the adoption of these architectures and the realization of the value creation of Open API, the number of API an organization needs to design and work with has seen an exponential increase. It is no longer enough to ensure the developer experience and quality of singular API, Organizations must work to increase these attributes in groups of API and API interactions. The parallelism posed by API interaction is a requirement that is beyond the capabilities of Swagger and is better addressed by another definition mechanism.             

In one post, Dr. Sanjiva Weerawarana, the founder of WSO2 speaks about how the integration space has become more parallel, he goes on to make an analogy of how the increased dependency on external network services in systems is similar to the way in which developers used to have to depend on external libraries to provide different capabilities to their programs.[6] It was with this intent of addressing the lack of proper definition mechanism for interaction/integration mechanism that the graphical syntax of Ballerina was birthed. Ballerina is a general purpose, concurrent and strongly typed programming language that is optimized for integration. Its graphical syntax is designed to resemble UML sequence diagrams, as such it can effortlessly capture the parallelism of integration scenarios as well as providing the missing element in an end-to-end “Design-First” adoption for organizations.

Ballerina Design-First Example  

With Ballerina, organizations can now define the contracts of singular API using Swagger and continue on to define the interaction between these API using the same tool. As Swagger is tightly integrated into the ballerina’s design tool, the composer, Swagger definitions seamlessly get translated into Ballerina code and diagrams.

Design-First flow with Swagger and Ballerina


Note: proceeding example is built around the service chaining sample shipped with the ballerina distribution. The ballerina code segments are compatible with the runtime version 0.89 and may not be compatible with other versions.


This example demonstrates how a simple service chaining scenario consisting three API may be constructed with a “Design-First” approach. ABC Bank API; is to act as the entry point for the user, it will orchestrate the interaction between the two other API, Branch Locator API; responsible for locating a branch based on a zip code provided and Bank Info API; responsible for providing information based on a branch code. Once the orchestration is complete, ABC Bank API responds back to the user.   

As the first step of constructing this scenario, the swagger definitions for the three API can be created. This step will allow the developer experience to be defined for each singular API, consultants may utilize the Swagger editor provided with the Ballerina Composer for this task or use any other editor they are familiar with. The resultant Swagger definitions will look similar to what’s found below,




At this time the ballerina composer requires the swagger definitions to be in JSON format in order for them to be imported. If the definitions were created using a third party editor they may be at once saved as JSON files. If this is not the case the definitions can be converted using free YAML to JSON converters available online.


The JSON conversions for the YAML definitions found above would be,








Next, the interaction between the three API should be designed. If the consultants used the Ballerina Composer for the previous step they may switch to design view and continue on to define the orchestration behavior using the Ballerina Graphical syntax. When designing this interaction consultants should utilize Function Constructs to abstract implementation level logic blocks in order to preserve design fidelity. The resultant diagram will look similar to,


In the above diagram the logic needed to prepare the payload for the Branch Info API and Bank Info API have been abstracted into functions prepareLocationServicePayload and prepareInfoServicePayload.

After this stage, the consultants may verify and proof the design before handing over the definitions to implementation specialists who may fill-in the design to construct the final ballerina deployable artifact(s). The resultant ballerina implementation for the ABC Bank API may take the shape of,



The implementation specialists may pack the services into a single deployable archive, such as the one found below,



Find out more about ballerina here: http://ballerinalang.org/



Thursday, June 1, 2017

Creating a Google Calendar Reminder with Ballerina

Ballerina makes it easy to use and tape together different API to create mashup applications and services. This example explores how ballerina can be used to invoke the Google Calendar API to create events that can work as a simple task reminder.

In order access Google API, you must create an application in the developer console[1], subscribe to the API you wish to use(in this case the calendar API) and generate Oauth credentials which can then be utilized to create an API key with the scopes needed for the API. Conversely, you could use the google playground application[2] to generate an API key to try out the example. To generate an API key this way,

  1. Select the calendar API with “https://www.googleapis.com/auth/calendar” (this scope is needed to create new calendar events) from the drop down found in step 1.
  2. Authorize API to generate an Authorization code along with an access token and a refresh token.

Execute the program passing in the following information in the order specified.
  • Access token
  • Event end date time in rfc3339[3] format(look to the example execution found below)
  • Event start date time in rfc3339 format.
  • Description for the event
  • Event title.


Example,
ballerina run main calendarAPI/sample/CalendarEventCreator.bal "ya29.XXXXXXXXOEyIODzSIvcnH4xkGCZBjPaRNJTzobPnLOo8a-5fzOnUILeFryHvbIVr8PX_DCFksmZf7ek3fFbRxvBVrB3K9wEQIwGbwXCD9_yJYy8w9EHdXXXXXXXX" "2017-06-02T12:30:00+07:00" "2017-06-02T12:10:00+07:00" "yet another google calendar API test" "test one two"

A new google calendar event will get created based on the values passed in and a pop up reminder will be displayed 15 mins prior to the starting time.

You may follow a similar formula to utilize any other google API with ballerina. In a future post we will discuss how the google calendar API invocation logic may be abstracted into its own connector so that it can be conveniently utilized in other ballerina programs.


Note: this example was written for ballerina 0.86 and it may not work with other versions.

Monday, May 29, 2017

Ballerina for simple server side Web Socket programs

WebSockets is one of those protocols that was invented to solve the transmission barrier posted by HTTP’s stateless request-response model. One of the key differentiators of it is its ability piggy back on HTTP infrastructure using the HTTP upgrade mechanism[1], this ability when coupled with the inbuilt support provided by HTML5 for client-side WebSockets programming, allowed it to emerge as a strong contender to win the full-duplex protocol war.

As with other transport/application protocols supported by Ballerina, a consistent programming interface is provided to put together WebSocket based server side programs with ease. The example found below demonstrates this capability.








To try out the example, create the folder structure <BALLERINA_HOME>/socketproxy/sample and copy the content into the two appropriately named bal files. Make an archive[2] and make a websocket client call to “ws://localhost:9090/socketproxy/ws”, find a sample python client attached(it requires python-websocket package to work).


Look to the websocket samples found in the ballerina 0.87 distribution to learn more about websocket programming with ballerina.

Note: the example above was constructed to demonstrate a simple use case involving websockets, a real world implementation should be built after careful analysis of the requirements.

Wednesday, May 24, 2017

Ballerina in the Cloud

As discussed in other posts, Ballerina provides integration capabilities in a lean, mobile distribution. Its graphical and textual syntax parity makes it ideal for developers coming in from different backgrounds. Someone who wishes to avoid the textual syntax can do so without losing any programming mobility. Regardless of the syntax chosen, the end result of a designed ballerina program would be a file with a “.bal” extension or a combination of such files archived into files with either “.bsz” or “.bmz” extensions.

WSO2 Integration Cloud is a SaaS imagination of WSO2’s industry trusted enterprise integration solution, apart from handling WSO2 EI projects it is now capable of providing a cost effective means of serving your ballerina projects to the world. This post is meant to help you get started with serving your ballerina services from the cloud,

Creating Ballerina Services


Ballerina services that are meant for the integration cloud maybe created the same manner as any other ballerina service. However, they must be packed into archives prior to uploading. Look to the ballerina packaging documentation[1] if you’re not familiar with this task.

For the moment it is advisable to limit your services to the HTTP transport and to a single archive(if it's a service spanning multiple ballerina files).

Uploading the Services to the Cloud  

Integration Cloud takes care of the administration and configuration tasks associated with serving ballerina integration projects, from containerization of the archives to scaling on demand.

Uploading a ballerina archive to the integration cloud is as easy as logging in, specifying an application name, version and uploading the archive from your local file system. Users may also select instance specifications and pass in environmental variables if required. It is advisable to select a container specification with moderate amount of ram(think 512).

Example


The example below shows the ease at which ballerina integration projects maybe created and served from the cloud. Note that this example was written for Ballerina 0.87.

Prerequisites

Integration Cloud Access: WSO2 integration cloud provides a 2 week trial program, you may avail this offer to try out the example.

Geoname Username: Geoname is a geolocation service made available free of charge for educational purposes. Create an account with Geonames and activate free web service usage through ManageAccounts.

Walk-through


  1. Create the folder structure <BALLERINA_HOME>/geochain/sample
  2. Copy the contents of the gist found below into <BALLERINA_HOME>/geochain/sample/GeoChainService.bal
  3. Navigate to <BALLERINA_HOME> and execute the package command on the service with ./bin/ballerina build service geochain/sample -o GeoChainService.bsz
  4. Login to the integration cloud with your credentials and start creating a new ballerina application (new application -> ballerina -> local file system)
  5. Put in “GeoChain” as the application name, “1.0.0” as the version and upload the newly created archive found at <BALLERINA_HOME>.
  6. Expand the advanced settings section and select a container specification which is more powerful than the one selected by default.
  7. Complete the upload by clicking “Create”.
  8. Give the integration cloud a moment to deploy the service and navigate to the application list to find the GeoChain service deployed. Click on it to get to its overview page and find the service host. It will look like,
    1. http://dumiduh-geochain.wso2apps.com
    2. The service will be available at, <your service host>/geochain



Make a GET to the service with a JSON payload that contains the country code and your geoname username. Example payload and invocation,

curl -X GET -d @payload http://dumiduh-geochain.wso2apps.com/geochain
{"country":"LK","username":"username1102"} 


[1] - http://ballerinalang.org/docs/user-guide/0.8/tutorials/create-archive/



Tuesday, May 23, 2017

GeoChain: API Scraping with Ballerina

Ballerina makes it easy to scrape data from other endpoints. The example below uses some of the string manipulating capabilities discussed in String Manipulation with Ballerina to scrap a piece of data from one geonames API and using it calls another.




To try out the example, first create an account with Geonames and activate free web service usage through ManageAccounts. Then run the ballerina with the command line parameters shown below,

ballerina <Geonames Username> <Country Code> <Number of Wiki hits to retrieve> 
example, 
ballerina testaccount1101 LK 20


Similarly, ballerina can be used for service chaining scenarios. Look to the Service Chaining example shipped with the distribution for more information. Find out more about Ballerina here: http://ballerinalang.org/


Note: this example was written for ballerina 0.87 and it may not work with other versions.

Monday, May 22, 2017

Using the ballerina file transport to access a local file

Ballerina provides support for multiple transports such as HTTP, JMS, Web Sockets, File, FTP and SFTP. A service file is bound to a single transport, however there are no restrictions on packaging services of different transports together, this allows programmers to craft interfaces and glue applications that span multiple varied transports.  Further, the mechanism in which content is accessed and manipulated is constant regardless of the protocol being used, meaning the manipulation logic written for one transport say HTTP can be used with another with little to no alterations. This uniformity provides for a smoother programming experience and is sure to increase your programming efficiency.

The example below uses the file transport to access a local file’s content, it then creates a simple xml payload using the content(if the given condition is met) before sending it out to a processing endpoint over HTTP.




Note: to use some of the transports mentioned above you will need to put in a few dependency jars into ballerina libs. Look to the respective transport’s documentation for further instructions.

Note: the example above s is meant to be a crude demonstration of the uniform, almost transport independent programming interface provided by ballerina.

Find out more about ballerina here: http://ballerinalang.org/


Note: this example was written for ballerina 0.87 and may not work with other versions.

Thursday, May 18, 2017

Use ballerina to extend your bash scripts capabilities

Ballerina provides a comprehensive collection of connectors that can be used to work with popular services such as twitter, gmail and facebook. They are easy to use and comes bundled in a lean distribution(what is 15mb these days?) When coupled with its integration excellence it can solve many integration challenges in a way that was not possible before.

Ballerina can also become a useful tool for DevOps engineers who welcome the challenge of learning a new language. The example below shows how a ballerina script that uses the twitter API can be embedded into a shell script. Similarly you can avail any one of the ballerina connectors listed here[1].



Execute the script as follows,

 ./script.sh <consumerKey> <consumerSecret> <accessToken> <accessTokenSecret>


Look to the ballerina 0.8X twitter connector sample readme for further instructions. Find out more about ballerina here: http://ballerinalang.org


[1] - http://ballerinalang.org/docs/api/0.8/

Note: this example was written for ballerina 0.86 and may not work with other versions.