Flink-ESB Tutorials

Use case: create HTTP(S) file repository to store / retrieve files of any size

Short description: Create file repository offering storage and retrieval of files over HTTP(S). File repository should be able to handle very big files (>100 GB) without consuming of much RAM (working in stream mode).

Add security layer to manage read-only and read-write access using roles.

Example of usage: Application A produces files on hourly basis, for example report files containing customer data. Application B imports those files and processes them.

Problem: how to deliver files from Application A to Application B if there is no direct connectivity between server A and server B, or where server A and server B do now allow accessing their local folders from remote machines?

Solution: create a file repository. Application A would transfer files using HTTP(S) to the file repository. Application B would fetch the files from the file repository using HTTP(S). If needed, add security layer to administer access to the files and folders.

Following Flink-ESB components are used in this part of tutorial:

First level components:

HTTP Server : represents an instance of an empty HTTP server container. Add one or multiple "HTTP-Connectors" to create HTTP ports on that server instance.

HTTP Connector : used always together with "HTTP Server". Represents one port, where HTTP server accepts HTTP or HTTPS requests.

Resource Handler : used always together with "HTTP Server". Allows to access server's files and folders via HTTP.

Context : used always together with "HTTP Server". Used to configure server's security layer.

File Login Service : used always together with "Context" component. Used to set up security layer, where usernames/passwords/roles are stored in the file.

Step 1: creating a new Flink-ESB apllication in Flink-ESB Editor

Open Flink-ESB Editor and select "File -> New -> Project" in the main menu. Double click "Flink-ESB" folder to expand it, select "Create New Flink-ESB Project" and press "Next"

This opens a "New Flink-ESB Project" wizard. Enter data as on picture below: "Project Name" : file-repository, unselect "Use default location" checkbox and select a folder where a project should be created:

Click "Next" and type some name in "Provider" field, after that click "Finish". This should create a new Flink-ESB project and open a file called "project.esbbp" in the main editor pane. This is where we will create all the components of the application.

Step 2: creating HTTP Server in Flink-ESB Editor

Click on "HTTP Adapter" drawer on the components palette to the right of the main editor pain. Click on "HTTP Server" component, now move mouse pointer over editor canvas and click with the left mouse to put "HTTP Server" component onto canvas. Provide properties of the server as on picture below:

Click "Save Changes" to save Server's properties. Now select "HTTP Connector" within "HTTP Adapter" palette, and drop it onto editor canvas near "HTTP Server" component. Enter properties as on picture below:

Click "Save Changes" to save Connector's properties.

If you want this port to handle SSL (TLS) traffic, click on "SSL Parameters" tab in the properties window and provide needed properties.

Step 3: adding Resource Handler to HTTP Server in Flink-ESB Editor

Click on "HTTP Adapter" palette drawer to expand it, and place a "Resource Handler" component on editor canvas (scroll down within "HTTP Adapter" palette if you cannot see it). Enter properties as on picture below:

Click "Save Changes" to save Resource Handler's properties.

Assuming a local folder d:\repository on your windows machine looks like on picture below:

Now try to access URL http://localhost:4848/repo. You should get a response as on picture below:

Now let's copy several files into folder d:\repository and refresh http://localhost:4848/repo in the browser. You should see now the files you've just copied to the folder:

Clcik on one of the folder's links to go into that folder and to view its content:

Go ahead and try reaching repository from another machine. Just make sure to use repository's IP instead of "localhost" in the URL. You should be able to view all the files and folders in the repository:

Step 4: enabling repository modification in Flink-ESB Editor

Select "Resource Handler" component in Flink-ESB Editor and adjust its properties as on picture below:

Click "Save Changes" and then save the project file. Restart the application.

Now go to the browser and refresh the page:

Now you should see 4 buttons on the top of the page to upload files, delete files, rename file and create folder. Also there is a checkbox on the left side of every file and folder.

Click "Create Folder" button in the browser. Enter a name of new folder, for example "test":

Click "Save" button and confirm a question in message box. A new folder must appear now. Check the local folder on repository machine to make sure the folder is created there.

Click on "test" folder to go into it. It must be empty.

Let us now upload some files in there. Click "Upload File(s)" button and select one or several files on some folder on your machine. Press "Upload" button. Now you should see all those files in the repository's folder:

Go to repository machine and verify that all those files exist in "test" subfolder within repository location.

Now let us rename one of the files. Select a checkbox beside one of the files. "Rename File" and "Delete File(s)" buttons should now be enabled. Click on "Rename File" button and modify a file name in the textbox:

Press "Save" button and confirm question in the message box. Now the name of the file must be modified:

Now try to delete files. Select checkbox for one or several files and press "Delete File(s)" button. Confirm a question in the message box. This should delete selected files:

Step 5: adding security layer to the file repository

Right now everybody who has access to your repository server via HTTP URL of your file repository can read, write and delete files and folders on the repository machine. Let us add security layer, so that only authenticated users will be able to access file repository.

Click on "HTTP Adapter" palette drawer to expand it, and place a "File Login Service" component on editor canvas (scroll down within "HTTP Adapter" palette if you cannot see it). Enter properties as on picture below:

"File" property points to a file, containing users data (username, password, roles). This file will be used to authenticate users trying to access HTTP server, that is defined in this application.

To create such file, open a new file in text editor and paste following content into it:

# The format is
# username: password[,role]
admin: admin,admin,user
other: other,other
username: password,user

Save this file with any name (for example: realm.properties), but make sure its name matches "File" property of "File Login Service" component.

Flink-ESB provides command to encrypt/obfuscate/hash the password. Connect to SSH console of any Flink-ESB instance and run command: esb:app:password -u username password:

So for user "admin" configure obfuscated password instead of password in clear text: OBF:1u2a1toa1w8v1tok1u30

Instead of having all passwords in clear text, you can paste following content into "realm.properties" file:

# The format is
# username: password[,role]
admin: OBF:1u2a1toa1w8v1tok1u30,admin,user
other: OBF:1xmk1w261u9r1w1c1xmq,other
username: OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v,user

Add "Context" component from "HTTP Adapter" palette to the editor canvas. Set its properties as on picture below:

Click "Save Changes" and then click on "Security" tab in the properties window. Enter the properties as on picture below:

Click "Save Changes" and then click on "Security Constraints" tab in the properties window. Click "Add Constraint" button and enter properties as on picture below:

Click "Save Changes" to save Context's properties. Now select "HTTP Server" component in the editor and click on "Contexts" tab in the properties window. You should see one row with the context ID "repo-context". Select a checkbox beside that row:

Click "Save Changes" and save the project file. Now restart the application in the editor.

Now when you go back to the browser and try to reach file repository, you are asked to enter username and password to authenticate:

Enter username as username and password as password. Now you should be able to see repository content:

Go ahead and try to login using admin as username and admin as password. Login should work as well.

Try to access file repository using other as username and other as password. You should get HTTP 403 error:

Now try some non-existing (in your realm.properties file) credentials, for example test as username and test as password. You should also get HTTP 403 error.

Step 6: adding a new role for repository modifications

Now let us create one additional level of security: READ-ONLY roles, and READ-WRITE roles. It means if you would login with the user, who is a member of READ-ONLY role, you should be able only to view repository's content. If you login with the user, who is member of READ-WRITE role, you should be able to view and modify the content of repository.

Select "Context" component in Flink-ESB Editor, click "Security Constraints" tab in the properties window and modify properties as on picture below:

Click "Save Changes". With that modification you define, that all users, that are members of either "user" or "admin" roles will be authorized to access file repository.

Now select "Resource Handler" component in the editor and modify its properties as on picture below:

Click "Save Changes". With that modification you define, that only members of "admin" group will be allowed to modify content of repository.

Save project file and restart the application.

Now login into file repository using username as username, and password as password. You should get a response as on picture below:

As you can see, 4 buttons to modify repository are not visible, and also checkboxes on the left side of every file and folder have disappeared. So you are able to view the content, but not to modify it.

Now login using admin as username, and admin as password. You should get a response as on picture below:

Compare this picture with the previous one. With admin user you can view and modify the content of file repository.

Step 7: working with very big files

As you have seen in the previous steps of this tutorial, with "Resource Handler" component you can access files and folders on the server over HTTP. You can upload new files onto the file repository using browser.

But what if you need to upload very big files? Using browser for that activity is not recommended, or even dangerous. You can end up will all your memory consumed by the browser. For such tasks use scripts, not the browser.

"Resource Handler" component offers out of the box API for such tasks. You can use any command which is able to send a content of file to the server with HTTP POST. Let us do that with curl command.

Create a big file on remote machine (if it does not exist yet). On Linux machine you can use this command to create a file of 10 GB size:

fallocate -l 10G test.img

To upload this file to remote file repository we can use following command:

curl -i -vvv --digest --user "admin:admin" -XPOST http://192.168.2.105:4848/repo/test/test.img --data-binary @test.img

This way you can upload files of any size to the remote file repository. Flink-ESB processes data in stream mode, which means, it does not consume much RAM, even if the file is huge. The only limitation is file space available on your file repository server.

Transferring 10 GB of data to remote server can take a while. After curl command is completed check the content of "test" subfolder within file repositoty. It should look like on picture below:

This proves, that the data transfer was successfully completed

Step 8: working with non-completed transfers

When you transfer big files, there could be some network outages, or the script can be aborted manually. In this case only a part of the file is transferred to the file repository. Can we recognize such situations?

It is actually quite simple. Flink-ESB appends "__IN_TRANSFER__" to the file being transferred. Only after transfer is completed, it renames the file.

Let us verify this functionality. Create another big file with this command:

fallocate -l 5G test2.img

This creates a file with name "test2.img" and size of 5 GB. Now let us upload this file to remote file repository:

curl -i -vvv --digest --user "admin:admin" -XPOST http://192.168.2.105:4848/repo/test/test2.img --data-binary @test2.img

Run curl command, wait 1 or 2 minutes (transfer should be in progress), now press Ctrl+C to abort curl command execution.

Go to the browser and check the content of "test" subfolder within file repositoty. It should look like on picture below:

This gives us an indication, that there is either a transfer in progress, or the transfer was not completed.

So in order to get rid of incompleted files, all is needed, is to search for the older files containing "__IN_TRANSFER__" in their names, and delete them.