Wednesday, October 22, 2014

Managed APIs with WSO2 API Manager

In this post we'll take a look at what Managed API's are and how to expose your API's as managed API's using the WSO2 API Manager.

APIs

An API is a business capability delivered over the Internet to internal or external consumers. API's provide a means for vendors to provide their software as a service (SaaS) remotely to other parties. API's hide the underlying implementations and provide consumers with a standard programming interface to access functions.

For example there are many calendar apps that can access your google calendar and display events that are on it. This is made possible by the Google Calendar API. Many websites have YouTube videos embedded in them, this is done by using YouTube APIs.

Most service providers expose their services as API's. Yahoo, Amazone, Facebook, TripAdvisor and eBay to name a few.

eBay gives a good crisp intro as to what their APIs allow you to do: https://go.developer.ebay.com/developers/ebay/products/what-ebay-api

Managed API's

So what are managed API's ?

Say you operate a service that provide customers with reccomendations to purchase/sell stocks based on current market trends and predictions. You want to monetize this service and charge for API usage.

You want to expose your API's to both stock brokers and individual clients who wish to do trading on their own. You intend on having different account registrations for individual clients e.g: Basic accounts will have a lesser monthly subscription fee but with a limited number of API calls allowed, Silver/Gold subscriptions will have a higher number of permitted calls while Premium accounts will allow unlimited API calls. Furthermore, stock brokers will be charged differently based on the number of calls they make.

This type of service requires Managed API's. You should be able to authenticate and authorize users, limit access (throttle) and collect statistics in order to monetize the API's.

With Managed API's you can do the following:

Actively advertise API's and allow subscriptions to them
Can Secure API's and carry out authentication and authorization
Monitore and monetize with analytics
Throttle access
Enforce Service Level Agreements (SLA's)

The WSO2 API Manager is a product that allows organizations to expose their business functionality as Managed API's.




The API-M is made up of 4 components, namely the API Publisher, API Store, API Gateway and Key Manager.

API Publisher: allows developers to publish API's, manage API versions, govern the API life-cycles, apply security policies, attach related documentation etc.

API Store: is a store where API's are advertised. Subscribers can discover and subscribe to API's using the API Store. There is a public user store as well as tenant based user stores.

API Gateway: the API Gateway is where API's are deployed. It intercepts API calls and enforces SLA's, performs security checks (using key manager), handles throttling and passes it to the actual back end service. The gateway also publishes API data for analysis.

Key Manager: handles security key related operations and token validations.

In this example we'll use an existing weather API in order to demonstrate how to create a Managed API using it.

OpenWeatherMap provides free weather data based on City. The following service call returns a JSON request which gives you the current weather in London:
http://api.openweathermap.org/data/2.5/weather?q=London

We'll create a managed API using this service as the back end.

Download and extract WSO2 API-M from here.

Start the server by going to <PRODUCT_HOME>/bin and by running wso2server.bat (Windows) or wso2server.bat (Unix).

Once the server starts you will see 3 url's displayed. By default they are:

http://your_ip:9443/carbon - Admin console url
http://your_ip:9763/publisher - Publisher url
http://your_ip:9763/store - Store url

These ports can be changed by incrementing the <offset> value in <PRODUCT_HOME>/repository/conf/carbon.xml

WSO2 products adhere to a role based permission model. Users are granted permissions based on the roles assigned to them. Permissions are assigned to roles, not the users.

Create 3 roles that allow creating API's, publishing and subscribing. Log in to the Admin Console using the default username/password : admin/admin and follow the instructions in the following documentation to create user roles: User Roles in the API Manager

Now add 2 users 'provider1' with create and publish roles assigned and 'subscriber1' with subscriber role assigned. Follow the steps listed in: Adding Users

Now log into the Publisher (http://your_ip:9763/publisher) using the 'provider1' user


The above screenshot shows some already existing API's. If there are no API's created yet this page will be blank.

Create API

Now Click on Add in the left panel.

Fill in the details as shown below:



Context will be the URI context path of the API
You can have many versions of the same API; define a version number
you can add a Thumbnail to your API as well

You can add Resources by defining a URL pattern and selecting the HTTP verbs that are associated with it. If you don't do this you will be prompted to add a wildcard resource (/*) with all 5 methods associated with it.

Click on Save.
Then click Implement


give the production endpoint as: http://api.openweathermap.org/data/2.5/weather

you can enter a Sanbox endpoint as well if needed for testing.

Now click Manage


Tier Availability means what throttling tiers would be available for subscription.

by default the API-M ships with 4 throttling tiers: bronze, silver, gold and unlimited. You can add more tiers based on your requirement.

Access to API could be throttled in a fine grained manner at Resource/verb level:



Click Save & Publish and the API will be Published. Optionally you can just save the API and publish later. If the user you are logged in with does not have Publish permission you will not be able to publish.


Notice the Copy button below. That allows you to easily copy the API and create a new version.

You can manage the Lifecycle by going to the Lifecycle tab.



API Store

Now lets take a look at the API Store - http://your_ip:9763/store


You will be taken to the public store. API's will be listed in thumbnail views similar to a mobile app store.

Log in using the subscriber credentials we created: subscriber1/subscriber1

Now you can Subscribe to an API. Click on the Weather API to subscribe to it.


You will see 2 drop-downs. One to select the Application and the other for the Tier.

Application - Applications allow API's to be grouped together. A single application may have one or many API's. Access keys are issued on an application basis, not on an API basis.

By default the application selected would be 'DefaultApplication'. Click on the dropdown and select New Application.

This will direct you to create a new application.


Notice the application allows you to define a Throttling Tier as well. This will throttle out all API access at Application level once the quota is exceeded even if individual API's have not reached their throttle limit. Lets select Gold for the application.

Once the application is added select Silver for the throttling tier of the API and click Subscribe.

API Manager uses OAuth 2.0 for authentication. Go to My Subscriptions and select the weatherApp. Click on Generate under Production and Sandbox (if needed). This will generate aConsumer Key, Consumer secret and also an Access Token that could be used for testing.


A token validity period could be set for the Access token and you can Re-generate it when ever required.


Testing the API

The API Store ships with a RestClient that helps you test your API's Restfully.

Under tools select RestClient.

Use one of the production URL's from the WeatherService API page (click on WeatherService API to go to the page).

Copy the Access token which was generated in our previous step.

Add url with the query parameter: q=London
Pass the token by specifying it as: Authorization :Bearer 36bd42e345c1bb027ce952255918f4
Note: B in Bearer should be Capital


Now you'll get the weather in London as a JSON response;



Throttling:

Remember we set the throttling tier to Silver for the Weather API. Now send more than 5 requests within a minute using the rest client and see what happens.

You will get a Message Throttled Out response with a code.

* Notice the response is in JSON. If you want to convert this to XML you can do so by editing the synapse configuration and setting the 'messageType' property in the outgoing message to XML. However, the more recommended way of doing this is by adhering to the API Façade pattern. Refer to my previous blog post: Format Conversion using API Façade Pattern 



Friday, September 5, 2014

Protocol Conversion from REST to SOAP using Messaging Bridge Enterprise Integration Pattern (EIP) and Content Type Negotiation using WSO2 ESB

Say you have a SOAP service. You need to expose it as a REST service. Furthermore, you may need other mediations such as XML to JSON conversion to happen.

This could be done by implementing the Messaging Bridge Enterprise Integration Pattern (EIP).
Ref: https://docs.wso2.com/display/IntegrationPatterns/Messaging+Bridge

Furthermore, with the WSO2 ESB Content type negotiations (conversions) such as XML to JSON is just a simple configuration that would take only a few seconds.

For the back-end service we'll use the ZooDataService we built in a previous example using WSO2 DSS to expose the data as SOAP services: Expose Data as a Service using WSO2 Data Services Server (DSS)

Now that we've got a service we'll see how to do the SOAP to REST conversion, and the XML to JSON conversion.

1. First lets take a look at the SOAP to REST Conversion:

For the SOAP to REST conversion we will adhere to the Messaging Bridge Integration pattern as mentioned above.

We will create an API in the WSO2 ESB for this purpose and expose the SOAP service as a REST service via the said API.

Below is the architecture for our system:




Now lets look at how to Create the ESB API:

Go to the ESB management console and add an API

Switch to Source View and enter the following configuration:

<api xmlns="http://ws.apache.org/ns/synapse" name="ZooData" context="/zoodata">
   <resource methods="GET" uri-template="/{animalID}">
      <inSequence>
         <payloadFactory media-type="xml">
            <format>
               <p:getAnimalByID xmlns:p="http://ws.wso2.org/dataservice">
                  <a:AnimalID xmlns:a="http://ws.wso2.org/dataservice">$1</a:AnimalID>
               </p:getAnimalByID>
            </format>
            <args>
               <arg evaluator="xml" expression="get-property('uri.var.animalID')"></arg>
            </args>
         </payloadFactory>
         <log level="full"></log>
         <call>
            <endpoint>
               <address uri="http://10.100.5.132:9769/services/ZooDataService/select_with_key_animal_operation" format="soap12"></address>
            </endpoint>
         </call>
         <property name="messageType" value="application/json" scope="axis2"></property>
         <respond></respond>
      </inSequence>
   </resource>
</api>
                        
                        

Now you can invoke the service Restully using the following GET request:

http://localhost:8281/zoodata/003

you will get the following XML output:

<animalCollection>
<animal>
<AnimalID>003</AnimalID>
<Species>Penguin</Species>
<Name>Rico</Name>
<Country>Madagaskar</Country>
<Speciality>Demolitions</Speciality>
</animal>
</animalCollection>


2. Next we'll look at how to convert the content types (Content Type Negotiation):

To convert the content type from XML to JSON  in the outgoing response simple change the value attribute of the 'messageType' property to 'application/json'. This works vice versa as well.

<property name="messageType" value="application/json" scope="axis2"></property>

now you'll get the output in JSON format:

{"animalCollection":{"animal":{"AnimalID":"003","Species":"Penguin","Name":"Rico","Country":"Madagaskar","Speciality":"Demolitions"}}}




Furthermore:
Now since you have achieved protocol & content type conversion you can use the WSO2 API Manager to expose this as a Managed API if you wish. You will be using the API Façade pattern in doing so.


For further information on API Façade pattern refer:
http://wso2.com/blogs/architecture/2013/05/a-pragmatic-approach-to-the-api-faade-pattern/
 


Expose Data as a Service using WSO2 Data Services Server (DSS)

Scenario:
You run a Zoo... yeah that's right a Zoo. You have a set of animals and each of them are renowned for something. You have their information which include an id for each animal, their country of origin, the name you gave them and what they are known for.

You need to expose this animal information as a web service.

So what do you do.

  • You open up eclipse
  • Create a web services project
  • Add the db drivers
  • Create the DAO layer
  • Create the DTO ....
WHoA WHOaaa... STOP ... 

That's ancient science... in the time u did all that you could have had your services exposed and be already testing them. 

How ?

Well this amazing little product called the WSO2 Data Services Server (DSS for short). It'll make your life so much more easy when it comes to exposing your data as services.

In this example we'll look at how to expose data as a SOAP service in a few minutes with just a few clicks, without writing a single line of code.

Yep .. your heard me.. without a single line of code .. 

What you need:
  1. MySQL Database and JDBC driver (or any other db + driver of your choice)
  2. WSO2 DSS 

Step 1:

Create a database called CUSTOMER_DB
create database Zoo_db

Create table CUSTOMER

CREATE TABLE `animal` (
  `AnimalID` varchar(90) NOT NULL PRIMARY KEY,
  `Species` varchar(200) DEFAULT NULL,
  `Name` varchar(200) DEFAULT NULL,
  `Country` varchar(200) DEFAULT NULL,
  `Speciality` varchar(200) DEFAULT NULL
);

Insert the animal data:

insert into animal values ("001", "Penguin", "Skipper", "Madagaskar", "Leading a pack of Penguins");
insert into animal values ("002", "Penguin", "Kowalsky", "Madagaskar", "Research & Development");
insert into animal values ("003", "Penguin", "Rico", "Madagaskar", "Demolitions");
insert into animal values ("004", "Penguin", "Private", "Madagaskar", "hmmm.. saying random feminine stuff");
insert into animal values ("005", "Lemur", "Julian", "Madagaskar", "Dance Unit");
insert into animal values ("006", "Lemur", "Maurice", "Madagaskar", "Maintenance");
insert into animal values ("007", "Lemur", "Mort", "Madagaskar", "Feet Hugging");

Step 2:
  • Download and Unzip WSO2 DSS
  • Add the MySQL connector JAR to the <DSS_HOME>\repository\components\dropins directory
  • Start the server by running the wso2server.bat (Windows) or wso2server.sh (Linux) in <DSS_HOME>/bin directory
  • The management console will be accessible on https://localhost:9443/carbon/ - 9443 is the default port. If you want to change this you can do it by offsetting the port in <DSS_HOME>\repository\conf\carbon.xml - Increment the <Offset>0</Offset> value. All ports used by DSS will be incremented accordingly.
  • Login with default admin username/password admin:admin
  • Using the controls on the left go to Configure -> Data Sources -> Add New Data Source


  • Choose RDBMS, Give a name, Enter Driver String and URL, username/password as shown


Test and save the DataSource

Next go to Main, Under Services -> DataServices click Generate:




Select the created ZooDS from the dropdown and give the DB name we created (Zoo_db)
And then select the table for which you want the services generated for. In our case the animal DB



click next and you'll get the option to select whether to create separate services for the different operations (select, insert, delete etc.) or bundle them to one service. We'll select to expose all in one service and give it the name "ZooDataService"



Click next and then Finish. This will create a set of services that expose all the operations such as Select, Insert, Delete etc. as soap servicesNow your service will be listed under Services -> list

Now you can try out the different operations using the 'Try This Service' option which you'll see




And there you go ... Who said exposing data required hours of coding ?

So to sum it up WSO2 DSS is like Rico from Penguins of Madagaskar... you'll know what I mean if u've watched it... Tools at your fingertip.. Fast and Easy .. Just like that ;)

Monday, June 9, 2014

HectorException

Hector Exception ?? ... wonder where Achilles was :p

The following exception is related to port offsets.

TID: [0] [BAM] [2014-06-06 14:51:06,015] ERROR {org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber} -  Error processing event.  {org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber}
com.google.common.util.concurrent.UncheckedExecutionException: me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
at org.wso2.carbon.databridge.persistence.cassandra.datastore.ClusterFactory.getCluster(ClusterFactory.java:95)
at org.wso2.carbon.databridge.datasink.cassandra.subscriber.BAMEventSubscriber.receive(BAMEventSubscriber.java:69)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:390)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:244)
at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspace(AbstractCluster.java:199)
at org.wso2.carbon.databridge.persistence.cassandra.datastore.CassandraConnector.createKeySpaceIfNotExisting(CassandraConnector.java:456)
at org.wso2.carbon.databridge.persistence.cassandra.datastore.ClusterFactory.initCassandraKeySpaces(ClusterFactory.java:86)
at org.wso2.carbon.databridge.persistence.cassandra.datastore.ClusterFactory$1.load(ClusterFactory.java:74)
at org.wso2.carbon.databridge.persistence.cassandra.datastore.ClusterFactory$1.load(ClusterFactory.java:54)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
... 12 more


If you get this in BAM that means you are running BAM with a port offset and have not offset accordingly in the <BAM_HOME>/repository/conf/etc/hector-config.xml file. 

Scheduled Tasks with WSO2 ESB

This article demonstrates how to create a simple scheduled task using the WSO2 ESB.

This is an example that you can complete withing 5 minutes and will give you a basic understanding of how easy it is to deploy a scheduled task using the WSO2 ESB.

Steps:
1. Create Task class
2. Export task class as a jar
3. Add to WSO2 ESB classpath
4. Create Scheduled task

Step 1:
First of all you need to create a Task Class. For this example we will create a simple class that prints out "HELLO" and a timestamp.

First download the synapse-core-1.2.jar from either the Synapse site or from the Maven Repository

Create a Java Project in eclipse (or any IDE you preffer) and create the following class:

package wso2.demo.taskt;

import java.io.BufferedWriter;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Date;

import org.apache.synapse.ManagedLifecycle;

import org.apache.synapse.core.SynapseEnvironment;
import org.apache.synapse.startup.Task;

public class SayHello implements Task, ManagedLifecycle {


private SynapseEnvironment synapseEnvironment;

@Override

public void destroy() {
// TODO Auto-generated method stub

}


@Override

public void init(SynapseEnvironment arg0) {
this.synapseEnvironment = synapseEnvironment;
}

@Override

public void execute() {

try {
File file = new File("E:\\sayhello.txt");
 
// if file doesnt exists, then create it
if (!file.exists()) {
file.createNewFile();
}

FileWriter fw = new FileWriter(file.getAbsoluteFile(), true);

BufferedWriter bw = new BufferedWriter(fw);
bw.write("---------------- HELLO ----------------- "+ (new Date()).toString() + "\n");
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}

}



Step 2:
Export the above class as a Jar file. Say with the name SayHello.jar

Step 3:
Add the above class to the ESB Class path by placing it in the <ESB_HOME>/repository/components/lib directory.


Step 4:

Now you can create the Scheduled task in your ESB.

1. Login to the ESB

2. Select "Scheduled Tasks" under the "Main" tab

























3. Add a new task



Replace the Task Implementation field with the qualified name of the class you created.

In the next section the Trigger type would be what type of scheduled task you want it to be. By simply defining the count and interval, or by providing a cron expression. Lets select "simple" for this example.

Count - defines how many times your job needs to run
Interval - specifies time gap between task executions

Click Schedule to save the task.

Aaaand That's it ...

Your output file would show the following:

---------------- HELLO ----------------- Mon Jun 09 20:11:37 IST 2014
---------------- HELLO ----------------- Mon Jun 09 20:11:42 IST 2014
---------------- HELLO ----------------- Mon Jun 09 20:12:46 IST 2014
---------------- HELLO ----------------- Mon Jun 09 20:12:51 IST 2014
---------------- HELLO ----------------- Mon Jun 09 20:12:56 IST 2014
---------------- HELLO ----------------- Mon Jun 09 20:13:01 IST 2014


For more options in creating scheduled tasks visit the following page:
Adding and Scheduling tasks in WSO2 ESB