Flink-ESB Message Gateway Tutorials
Use case: using gateway as reverse proxy
Original situation: application server running on backend network offers multiple different services over HTTP on URLs: URL1, URL2, ... URLn.
Those services can be called using different HTTP methods: GET, POST, PUT, DELETE
Objective: add possibility to call all those services to external clients, which cannot call services on corporate backend network directly
Flink-ESB Message Gateway should be used as reverse proxy. External clients should be able to call internal services using address of message gateway, which in turn generates another request to application server, gets a reply, and passes that reply back to the callers
Where needed, authentication layer should be added to the services
Estimated costs:
Programming efforts: 0
Administrator efforts: 15 minutes
In this tutorial elasticsearch services are used, which are installed together with Flink-ESB or Flink-ESB Message Gateway.
Elasticsearch provides a huge number of different services out of the box for HTTP GET, POST, PUT and DELETE.
Of course, you can use any other existing application server for this tutorial, just make sure to adjust destinations in step 2 accordingly.
Step 1: defining 4 façade entries for each HTTP method GET, POST, PUT and DELETE in Flink-ESB Message Gateway Config Console
Open Flink-ESB Message Gateway Config Console in the browser. Default URL: https://localhost:8082. Login with default username/password: admin/admin
Click on "Gateway Config" tab if you are not there already, and press "Add Entry" button.
Enter data as on the picture below: "Config Id" : 1001 (or any unique Id), "Method" : GET, "URI" : /es/.*. Leave all other fields with their default values
Click on added row to hide details panel. Click "Add Entry" button again and specify details for the second façade entry: "Config Id" : 1002 (or any unique Id), "Method" : POST, "URI" : /es/.*.
Hide details panel. Click "Add Entry" button and specify details for the third façade entry: "Config Id" : 1003 (or any unique Id), "Method" : PUT, "URI" : /es/.*.
Hide details panel. Click "Add Entry" button and specify details for the fourth façade entry: "Config Id" : 1004 (or any unique Id), "Method" : DELETE, "URI" : /es/.*.
Hide details panel of the fourth façade entry. Check now the data of all four destinations. It should look like on picture below:
If you found an error on some row, click on that row to unhide details panel and fix the data. Now click "Save Modifications" to save new façade entries
Step 2: defining 4 destinations for each HTTP method GET, POST, PUT and DELETE in Flink-ESB Message Gateway Config Console
Click on "Destinations" tab and click "Add Entry" button. Type data as on picture below: "Destination Id" : 1001 (or any unique Id), "Protocol" : HTTP, "Method" : GET, "Destination Expression" : 'http://192.168.2.116:9200/'+property['http.url'].substring(4)+(property['http.queryString']!='' ? '?'+property['http.queryString'] : '') (use IP of your machine, where message gateway is installed)
Click on added row to hide details panel. Click "Add Entry" button again and specify details for the second destination: "Destination Id" : 1002 (or any unique Id), "Protocol" : HTTP, "Method" : POST, "Destination Expression" : 'http://192.168.2.116:9200/'+property['http.url'].substring(4)+(property['http.queryString']!='' ? '?'+property['http.queryString'] : '')
Hide details panel. Click "Add Entry" button and specify details for the third destination: "Destination Id" : 1003 (or any unique Id), "Protocol" : HTTP, "Method" : PUT, "Destination Expression" : 'http://192.168.2.116:9200/'+property['http.url'].substring(4)+(property['http.queryString']!='' ? '?'+property['http.queryString'] : '')
Hide details panel. Click "Add Entry" button and specify details for the fourth destination: "Destination Id" : 1004 (or any unique Id), "Protocol" : HTTP, "Method" : DELETE, "Destination Expression" : 'http://192.168.2.116:9200/'+property['http.url'].substring(4)+(property['http.queryString']!='' ? '?'+property['http.queryString'] : '')
Hide details panel of the fourth destination. Check now the data of all four destinations. It should look like on picture below:
Let's explain the meaning of that long expression used for the destinations.
'http://192.168.2.116:9200/' + property['http.url'].substring(4) + (property['http.queryString'] != '' ? '?' + property['http.queryString'] : '' )
1) 'http://192.168.2.116:9200/' - is a literal value for HTTP address of elasticsearch. Make sure to ajust IP. Port is 9200 per default after Flink-ESB Message Gateway installation
2) property['http.url'].substring(4) is the URI received, with first 4 characters removed. For example, when you hit URL https://localhost:8081/es/test/some/function, Flink-ESB adds a property with name "http.url" and value "/es/test/some/function" to the request message for every incoming HTTP request. substring(4) would remove "/es/" from URI. Why "/es/"? Because in step 1 all façade entries we configured on URI "/es/.*"
3) (property['http.queryString'] != '' ? '?' + property['http.queryString'] : '' ) checks whether property with name "http.queryString" is not empty string, and if it is not: it concatenates '?' and value of property "http.queryString", otherwise this expression outputs empty string ''. Where does this property "http.queryString" come from? When you hit URL http://localhost:8081/es/test/some/function?param1=value1¶m2=value2, the part after '?' is called "query string" and it is set by Flink-ESB as the value of property "http.queryString" for every incoming HTTP request. So this expression is used to grab the value of query string and pass it unchanged to the end destination
The other way around, if you want to get output from elasticsearch service available on URL http://192.168.2.116:9200/_cat/nodes over Flink-ESB Message Gateway, you should hit URL https://localhost:8081/es/_cat/nodes
Now click "Save Modifications" to save destinations
Step 3: assigning destinations to façade entries in Flink-ESB Message Gateway Config Console
Click on "Gateway Config" tab and click on the row with "Config Id" 1001 (or whatever Id you've assigned to GET façade entry) to unhide details panel.
Click on "Edit Destination Rules" link, select "Show all rules" checkbox and select a checkbox on the left side of destination with Id 1001 (or whatever Id you've assigned to GET destination). Destination should change a color from grey to black as on picture below:
Click on façade entry row to hide details. Now repeat the same procedure for the other 3 façade entries. Make sure to assign POST destination to POST entry, PUT to PUT, and DELETE to DELETE. Verify again the data in config console and make sure it looks as on picture below:
If errors found, go ahead and fix the data, then click "Save Modifications" to save modifications
Step 4: posting configuration to the running instances of Flink-ESB Message Gateway
Click on "Gateway Config" tab if not there already, and click "Trigger Instances" button. Confirm a message box by pressing "OK"
Step 5: verifying functionality using browser, curl and Flink-ESB Admin Console
Open the browser and type URL http://localhost:8081/es/_cat (or whatever IP and port you have for message gateway). You should get output as on the picture below:
Go ahead and try other URIs from the output on the picture above. Just do not forget to prepend /es/ to URIs
Open Flink-ESB Admin Console in the browser. Default URL: https://localhost:8443/console/start. Login using your account/password (default after installation: admin/admin)
Click on "Applications Overview" to expand it. Now you should see an instance name. Click on triangle on the left side of the instance name to expand it. You should see now applications installed on that instance. Click on the triangle beside "Message Gateway" application and click on "Routes"
Now you should see the routes. Click on "from-HTTP-1001" route, where "1001" is Id of façade entry. Now you should see following:
Click on "Events" tab on the right panel right below the graphical presentation of the route.
Copy "msgId" of the latest event found as picture below shows:
Now you can try calling elasticsearch service with some long query string. Type following URL in the browser: https://localhost:8081/es/flink_message_*/_search?q=msgid:5d1a97d3-866e-4aaa-b4b3-6750e994fee4&size=100&sort=starttime:desc&pretty
This should output all processing events stored by Flink-ESB in elasticsearch for the selected message id, sorted by the time in reverse order:
Now try to run the same search query using elasticsearch REST api with HTTP POST. You can use either curl, or some browser plugin able to send HTTP POST requests, or SoapUI, or any other appropriate tool.
The command in curl looks like this:
curl -k -XPOST 'https://localhost:8081/es/flink_*/_search?pretty' -d '{ "size" : 100, "from" : 0, "query" : { "filtered" : { "filter" : { "bool" : { "must" : [ { "term" : {"appid" : "message-gateway"}} ,{ "term" : {"msgid" : "5d1a97d3-866e-4aaa-b4b3-6750e994fee4"}} ] } } } }, "sort" : [ {"endtime" : {"order" : "desc"}} ], "_source": { "include": [ "*" ] } }'
Using brwoser plugin, or SoapUI, or any other tools, just make sure to use JSON message (selected blue) as the body of HTTP POST request, and as URL use URL shown in brown color
Step 6: adding authentication layer to some services in Flink-ESB Message Gateway Config Console
Go to "Gateway Config" tab in config console. Click on façade entry with Id 1002 (HTTP / POST) to unhide details panel. In "Roles" textbox type admin,user as on picture below:
Click "Save Modifications" and then click "Trigger Instances"
Now if you repeat HTTP POST call you should get "Error 401 Unauthorized". To get it working again, you must provide username/password of any user, who belongs to "admin" or "user" role. Per default after Flink-ESB Message Gateway installation a file based login is enabled, with all the users, passwords and roles are stored in the file called etc/realm.properties within your Flink-ESB Message Gateway installation path
Use default username/password admin/admin. Modify curl command as below:
curl -k -XPOST --digest --user admin:admin 'https://localhost:8081/es/flink_*/_search?pretty' -d '{ "size" : 100, "from" : 0, "query" : { "filtered" : { "filter" : { "bool" : { "must" : [ { "term" : {"appid" : "message-gateway"}} ,{ "term" : {"msgid" : "5d1a97d3-866e-4aaa-b4b3-6750e994fee4"}} ] } } } }, "sort" : [ {"endtime" : {"order" : "desc"}} ], "_source": { "include": [ "*" ] } }'
Now you should get an output of the secured service after being authenticated by Message Gateway
If you want to protect only certain services, for example not all HTTP GET, but just part of HTTP GET, then you must create another façade entry for that URI (or URI regular expression), specify "Roles" and assign a destination to it:
Click "Save Modifications" and then click "Trigger Instances"
Now all HTTP GET services should still be available without authentication, except /es/_cat/health. To get a response from that service you must authenticate yourself