In the aftermath of my Robot Framework workshop at the Xebia 2015 TestWorks Conf, I received several e-mails from people who had attended the workshop. They were asking questions and describing (smaller and larger) problems surrounding various aspects of their test automation efforts with the Robot Framework. Some of these questions and problems are identical to those that, as a consultant, I encounter in the field. Since the involved topics may thus be of interest to a broader public, I decided to dedicate a series of blog posts to them. Better (very) late than never, right?
The first of these posts will show you how to use the Java Database Library while running RF on Python and also elaborate on why you may want to do so. As an extra, we will be putting the library into actual use as well, by connecting to an IBM DB2 database and, subsequently, running some keywords.
Please note that I will use these treatments also as an opportunity to shed some extra light on various aspects of the RF that we will encounter and that I feel may be of interest to those that would like a somewhat better understanding of RF’s internals. So, for some readers this will feel like a mild and acceptable (and maybe even welcome) digression, while for the practically inclined it may constitute an inexcusable transgression. You can’t win ’em all, I guess. 🙂

There are currently two database libraries available for the RF: one is implemented in Python and supporting ODBC (amongst others), while the other is implemented in Java and supporting JDBC.
The further differences between these libraries (i.e. other than the implementation languages and supported database APIs) and the question of when to use the one and when the other, will be covered by another post. In this post, I will merely be explaining the usage of the Java Database Library in a special context, namely using it through the so-called ‘Remote Library Interface’ (RLI). Through the RLI, we will be able to use this Java test library with a RF instance that is running in the CPython interpreter.
But first, let’s shed some light on the problem that we mean to solve through this construction as well as on some of the technical concepts surrounding both that problem and its solution.

Using non-Python RF test libraries

It all starts promising

The Robot Framework is written in Python. However, since alongside the reference CPython interpreter implementation (which you get when you install Python) there is a Java port (Jython) as well as a .NET port (IronPython) of Python available (amongst others), we can run the RF on these platforms (i.e. Java and .NET) as well. We can do this by simply executing the RF with the relevant interpreter (assuming that interpreter to have been installed properly). The RF is thus fully compatible with Jython and IronPython and, in that sense, natively supports the usage of these interpreters: it requires neither C-extension modules nor Python libraries that have no implementation in Jython or IronPython.
The beauty of it is, that this provides us also with the possibility to employ RF test libraries that have been written in Java or .NET, by running the RF on/with the corresponding interpreter.  Consequently, we are able to use non-Python libraries such as the Java JDBC Database Library by running RF on Jython. The Jython interpreter not only compiles the RF Python code into Java bytecode for execution within a JVM, but also allows for the importing of Java classes (such as those of the Database Library) as (if they were) Python modules.

And then there’s a catch, of course

However, the usage of any interpreter will prevent us from running test libraries that are incompatible with that interpreter. This is because, unlike the RF, not all test libraries natively support all interpreters.
Firstly, libraries that have been expressly written for usage with a specific non-CInterpreter in mind, such as Jython or IronPython, will inevitably (and obviously) be limited to that interpreter.
For instance, running the RF on Python will prevent running .NET test libraries as well as Java test libraries. For example, our Java Database Library: only when running on Jython, we can import the relevant Java classes from the DB Library and run them alongside the RF in the JVM. The same goes for any .NET libraries, which we can use only when running the RF on IronPython. Similarly, running the RF on Jython prevents one from using .NET libraries. And running the RF on IronPython … well, you get the picture.
But, secondly (and maybe surprisingly), even test libraries written in Python, might not run on one or both of the other interpreters.
For instance, let’s consider Jython. For one, Jython does not have as many of the build-in libraries as Python. That is, not all native Python build-in modules have a Jython implementation counterpart. Moreover, certain Python libraries themselves depend on modules written in C, which Jython cannot handle/interpret. So, a RF test library, although written in pure Python, may depend on a C extension module (or a Python module which in turn depends on a C extension module), which is no problem for the CPython interpreter, but IS a problem for the Jython interpreter. Similarly, a RF test library written in Python may depend on a CPython build-in library that is not yet re-implemented for Jython. All of this goes for IronPython, as well.
Therefore, since (the) most (used) RF test libraries have been (and will be) written in Python and/or depend on C-extensions or on modules not implemented in other interpreters, running the RF on Python is preferred. For instance, running the RF on Jython because of the JDBC Database Library, would prevent us from using such pivotal Python test libraries as the two HTTP libraries and the SUDS (SOAP) library, since all of these depend on modules that are not compatible with Jython.
So, if we were to run the Java Database Library with Jython, we would effectively lose several essential test libraries. For me, this disadvantage is a show stopper in a lot of situations.

But it wouldn’t be Robot Framework without a happy ending

Thankfully there is an easy way to use Java or .NET test libraries while still running the RF on Python, thus avoiding the disadvantage of running the RF on one of the ports. As a matter of fact, you could even write test libraries in other languages (for which Python has no ports) and still run them on Python! For instance, test libraries written in Ruby, nodeJS, PHP, Perl or other languages.
This is truly great stuff: using test libraries developed in different languages in one and the same test set, test case or even in the same statement! You can use any test library that is out there, no matter the language it is implemented in. Additionally, your own team members can provide their test libraries written in the language that they are most skilled with, knowledgeable of and comfortable with, thus lowering the threshold for such a contribution. Of course, you wouldn’t want to involve a myriad of different languages into your automation projects and it is debatable whether using even just two different languages is desirable at all, given certain risks and drawbacks that such an approach would entail. But even when you stick to one language, the fact that you can choose between so many of them, is a unique and powerful trait of this test automation framework. And, of course, the most important use case is always relevant, namely to be able to use any third-party libraries that exist within RF’s ecosystem, regardless the language they have been written in: see a Ruby library you could use? Just grab it and run it alongside your Python, Java and other libraries!
That mechanism that achieves all of this is the Remote Library Interface (RLI).

The Remote Library Interface

I will describe neither the internals nor the usage of the RLI in great detail here, partly because certain information is already available (e.g. through the RLI project page as well as through the RF user guide), partly because both topics will be covered in another post.
As can be gathered from its GitHub project page, the RLI allows for a test library to run as an external process, i.e. on a host system (or, at least, a process) that is different from the one that the RF itself is running on (in). As we will see, this also enables a powerful distributed testing setup. But it also allows for using test libraries written in languages that are incompatible with the interpreter you use during your RF test runs. The latter is what we are (mainly) interested in.
Let’s see how all of this is accomplished then.
At the highest level, there are three components involved: a client component, a (remote) server component and the actual test library (or libraries) that is (are) to be made available to the platform. Strictly speaking, then, the RLI consists of the first and second of these components. Therefore, let’s have a closer look at both of them.

The client component

The first component is called ‘Remote’ library and it is one of the standard test libraries that come bundled with the RF (click an image to enlarge):

Consequently, we import the Remote library like we would any other test library. For example in RIDE (one of many possible editors for writing RF test code):

(As we will learn later on, when actually setting up the RLI mechanism, the importing of the Remote library is actually the last step to perform.)
The imported Remote library acts as a client to the second component (the server component, see the next section for details) and is, as such, the link between the RF on the one hand and that second component on the other hand. The argument in the above screen shot (the aptly named ‘Args’) directs the Remote library to the server counterpart it is supposed to connect with. That is, the url and port designate the host system that the remote server part is residing on and the port it’s listening in on. And the /JMSLibrary part specifies the name of the target test library that is being served and exposed by that remote component. The test library is what actually implements the keywords and that, as such, is the component that the remote server should pass any keyword calls, as received from the Remote client library, to.
We could therefore have another Remote library import statement connecting to the same remote server instance, but addressing (targeting) another, different test library running in that instance. As a matter of fact, we can have as many such import statements as we would like, to the same or other remote server instances running on the same or different host systems! This makes for powerful distributed test execution as well. Of course, a remote server’s host system is not necessarily an external system, but may very well be the system that the RF is running on (localhost). Accordingly, the following screen shot shows a setup with three (not four) remote server instances: one running on localhost and serving two libraries (HelloWorld and StackTest), another one running also on localhost, but serving just one library (JMSLibrary), and, finally, one running on an external system, also serving one library (SomeLibrary).

So, we can connect to multiple remote servers (each serving multiple test libraries) on each of a set of multiple hosts systems. Of course, when running multiple remote servers on one and the same host system, a unique port number must be assigned to each remote server instance.
It will be clear by now, that the difference with other test libraries is, that the Remote client library does not contain any keywords itself, but that through it the keywords contained in the remote test library are made available to the RF. Like all other RF test libraries, the Remote library has been written against the RF library API. But instead of exposing its own keywords through that API to the platform, it merely acts as a proxy, doing nothing more than ‘patching through’ (or forwarding) keyword calls from our RF test code to the remote server component and ‘patching through’ (to the RF and, ultimately, our test code) the possible return values that the remote server received from an executed test library keyword and that it then communicated back to the client.

The server component

The second component in the RLI mechanism is the so-called ‘Remote server’. This is the central component, the ‘heart’ of the RLI, where all of the magic happens.

Adding (even more) flexibility and versatility: ‘porting’ test libraries

As already described in the previous section, it is a ‘loosely coupled’ component, that is able to run remotely and that serves one or more test libraries that implement the actual keywords that we need to have available in the framework. Such libraries have expressly been written for the RLI, which enables a remote server to expose the keywords implemented in those libraries to the Remote client library. The client, in turn, makes the keywords available to the RF platform.
Libraries that may thus be exposed can be libraries that you have written yourself or existing, shared, third-party libraries within the RF ecosystem.
Such test libraries can be written in Java, Python, Ruby, PHP or several other supported languages, depending on your needs. For each supported language, a remote server component implementation has been made available. From the RLI GitHub project page:

A remote server implementation can be created for (in) any language that supports xml-rpc, such as the languages in the table. A Remote (client) library instance and a Remote server communicate using a simple, proprietary ‘remote protocol‘ on top of xml-rpc. Xml-rpc is a protocol that uses xml as the format for the messages that you want to have transported and http as the actual transport mechanism. Hence the urls in the import statements: client and server call each other (and thus connect) via http.
Since every remote server, regardless the language that it is implemented in, thus connects to the framework through an instance of the same Remote (client) library that is implemented in Python, we can connect to any non-Python test library without having to run on anything else than our trusted CPython interpreter! For instance, the jrobotremoteserver can serve test libraries written in Java and since we can connect to that remote server through the Remote library, we can use the Java libraries even when running the RF on Python:

In the above setup, we can connect from the RF running on Python to the Java Database library running in the JVM (on the same system). For example, in order to be able to validate that the correct database inserts have been made by the SUT in response to a REST call that was made through a RF REST test library that we need to run with the CPython interpreter.
Using the RLI, in particular through deploying the relevant remote server, we are thus able to ‘port’ a test library. With ‘porting’, I mean the process of making available to an interpreter (such as the CPython interpreter) a library that is in or of itself incompatible with said interpreter (such as a test library written in Java). The RLI provides such a process or mechanism of porting over such libraries to an interpreter.
How the different remote servers are distributed and need to be deployed on the host systems and how they integrate and interact with the actual test libraries, depends on the corresponding implementation language as well as on the remote server design. The jrobotremoteserver, for instance, is distributed as an executable jar file and uses a Jetty servlet container to run the test libraries. See below for detailed information on the deployment of the jrobotremoteserer.

Adding (even more) power: remote execution & distributed execution

Besides porting test libraries, other important use cases for the RLI are remote execution and distributed testing.
Remote test execution entails that the test library (or libraries) runs (run) on a different host system than the RF itself.
A remote setup is necessary when it is not possible for a SUT (system under test) to be called remotely. A web-based application, for instance, sporting a REST API or being exposed by SOAP, can always be called remotely, using RF’s HTTP or SOAP library: no remote library interface required. The same goes for testing this web app at the GUI level using Selenium Webdriver, since the remotely hosted web content is loaded into our local browser through http (and with the Selenium Grid we can even use browser instances on remote systems).
But we might have a stand-alone Java application, that we would like to test at the API-level. To be able to do so, we must create our own Java test libraries with ‘glue code’: little wrapper functions that hold the intelligence to correctly call our API functions. Through this test library, these wrapper functions are made available to the RF as keywords and through them we are then able to call API functions in our test code. Hence the term ‘glue code’ (more on this in a follow-up post).
Now, typically, the SUT will be hosted on a system in our staging environment (as will be other components, such as a database), while our automated testing framework will be residing on a separate server, usually tightly integrated into our Continuous Integration infrastructure/setup. But since the SUT requires the caller to be located on the same system, we cannot simply run our created test libraries on the tooling server. We therefore deploy (the Java implementation of) the remote server on the SUT system and have it serve our Java test libraries, which enables us to run and call these test libraries remotely:

The test libraries now run in the Remote server on the remote system that also hosts the SUT. (Note that this is a simplification: a follow-up post will elaborate on some of the involved complexities when testing against a Java API.)
As will be clear (and therefore not further explicated here), the possibility to run one or more remote servers on one or more local and/or remote systems, with each remote server in turn running one or more test libraries, opens up endless possibilities with regard to a distributed/parallel/concurrent testing setup.

Fully transparent and non-intrusive

Please note that the described diversity/multitude of available remote servers (as illustrated in the table) is completely hidden from the users of the RF, even if several or all of these servers are in use. As a matter of fact, the entire RLI mechanism is completely transparent to the users of the RF.
When using one or more (different) remote servers, first the relevant servers need to be deployed on the relevant (local and/or remote) hosts systems (more on how to actually do that in the next sections). Then for each test library running in any of the remote servers, a single import statement must be added to the RF automation code. This statement imports the Remote (client) library, as already shown in the screen shot above. Through the url argument the remote client is directed to the remote server to which to connect as well as directed to the target test library that is to be made available through that connection. Through the ‘Alias’ argument, we can give a logical name to this resource (i.e. to the remote test library).
Within the RF the resource is, from then on, known under that logical name and can consequently also be addressed within the automation code under that name. Both the fact that the library is a remote library and that it is written in a different language (than the RF and/or other test libraries), is of no concern to the user and the RF therefore hides these facts completely from the user. As far as the RF and its users are concerned, there is simply yet another test library available to the platform. Note that even the import statement says nothing about whether a library is in Ruby, Java, Python, etc. (although it does say something about it being external, even if only in the sense of running in another process).
The following screen shot illustrates the complete and utter transparency of the mechanism. It shows  the tooltip help and the auto-complete (again in RIDE) for the Java Database Library, which has been imported as a remote test library under the logical name of ‘DBLibrary’:

Running the Java Database Library

As already mentioned, our motivation for using the RLI to provide test libraries to the RF, might be that we need to run them remotely and/or in distributed fashion or that we need to run libraries that have been written in languages that are incompatible with the interpreter we need to execute our RF test runs with. The latter motivation is what drives us here, since we are interested in using the Java Database Library while still running the RF (and hence our tests) on Python.
Therefore, let’s now put above theory into practice by running the Java DB Library using the RLI.
Just as an example, we will connect to an IBM DB2 database. Of course, this is not necessary: we can also use the Python Database Library to connect to DB2 via ODBC. As mentioned before, the question of which database test library is to be preferred in what situations (and why), will be the dealt with in a follow-up post.

About the library

Being a JDBC library, the Database Library is written in Java. It is distributed as an executable jar file. As such it can only execute in a JVM and, consequently, cannot be used by a RF instance running in the CPython interpreter.
Nevertheless, we would like to use the library while running the RF on Python. To this end, we can use the jrobotremoteserver, which is the remote server implementation for Java.
The jrobotremoteserver is distributed as an executable jar file and uses a bundled Jetty servlet container to run/serve test libraries written in Java. These test libraries are Java class files (possibly packaged in a jar file). We can load one or more of such test libraries by starting the jrobotremoteserver from the command line (probably using a batch file) with a ‘–library’ switch for each test library we would like it to serve. For instance (assuming all involved Java resources to be on the classpath):
Java org.robotframework.remoteserver.RemoteServer --library JMSLibrary:/someAlias --port 8270
This will start the jrobotremoteserver and have it serve the JMSLibrary. A separate post will show an example of creating an actual Java test library from scratch and serving it like this in the jrobotremoteserver.
To our benefit, the author of the Java Database Library has created a second distribution that packages/bundles and reuses the jrobotremoteserver, so as to provide a remote database library that can be used out of the box. After downloading the Database Library Github project (see next section), we can see the two distributions:

Let’s now see how to get this up and running!

Taking the library into use

Assuming Windows as operating system, Python, Robot Framework, IBM DB2 (e.g. the free ‘DB2 Express-C‘) and Java JRE/JDK (6 or higher) to be installed thereon and RIDE (Robot Framework Integrated Development Environment) as our editor:

Making the library available to the platform

• Go to: https://github.com/ThomasJaspers/robotframework-dblibrary
• Click the green ‘Clone or download’ button and subsequently the ‘Download ZIP’ button and save the ‘robotframework-dblibrary-master.zip’ file to your local hard drive.
• Go to the download location and extract (to some target folder) the ‘dblibrary-2.0-server.jar’ that can be found in the ‘robotframework-dblibrary-master\sample’ folder of the downloaded ‘robotframework-dblibrary-master.zip’ file. (The adjacent dblibrary-2.0.jar file is the ‘regular’ Database Library, that must be run in a JVM and, hence, can only be used when RF is running on Jython.)
• Note: In the folder ‘robotframework-dblibrary-master’ of the zip, you can find the keyword documentation: ‘DatabaseLibrary_v20.html’. You may want to extract it, too.
• Before you can use the database library within Robot Framework, you need to run the extracted jar (see above for a description of how this works). See the next couple of steps.
• Navigate to the target folder (into which you extracted the jar file)
• Create a batch file in this folder, with the following contents:

@echo off

set CLASSPATH=%CLASSPATH%;%~dp0\dblibrary-2.0-server.jar;%~dp0\db2jcc.jar;%~dp0\db2jcc_license_cu.jar

java org.robot.database.server.RemoteServer --port 8271

Some remarks on this:

In line 3, Java needs to be able to find and run the RemoteServer class, which ultimately will lead to the remote server, Jetty and the actual database test library being loaded. To this end we need to add the dblibrary-2.0-server.jar file (which contains said class) to the classpath.

Therefore, in line 2 we extend the Windows ‘classpath’ system variable (assuming one to exist) by adding the path to the jar. However, the database library itself will require the relevant jdbc driver for database access: without it we cannot connect to the database. A database vendor, such as Oracle or IBM will provide their own jdbc drivers. Since in our case we would like to access IBM’s DB2, we will also have to include the relevant driver to the classpath. However, before the actual jdbc connection will be established between the jdbc driver and the database, the jdbc driver will check whether a connection to the server is actually licensed (allowed for). To this end, we need to have the proper license file on the classpath as well. In this example the driver is contained within the ‘db2jcc.jar’ and the license is contained within ‘db2jcc_license_cu.jar’. To determine the correct driver and license in your own context, please consult your db admin or consult this overview of driver versions (you can download the drivers from there as well). General info on DB2 jdbc drivers (e.g. on the differences in types) here, here and here. Info on the license file: here.

Further, please note that you can also add the paths to the three jars to an existing Windows classpath system variable in the ‘Environment variables’ dialog instead of in this batch file.

Also note that on line 2 I am using ‘%~dp0’ to designate the folder where the batch file itself resides. This assumes that (and consequently only works if) all of the three jar files are in the same folder as the batch file. You can also use other designations, such as a fully qualified path name to each of the resources (i.e. jar files).

And, finally, please note that you could also run the batch file (and thus start the Remote DB Library) from within your RF test code, e.g. through a setup function.

• Run the batch file (with a double-click or within a command line window).
• In the test code, create an import statement for the test suite or keyword resource where you would like to use Database Library keywords:

• With these steps, we have made available the Database Library to the RF. We are now ready to use the keywords provided in the library to access our DB2 database.

Connect to a database

• The first step will be to connect to the database. This is typically done in a setup. Similarly, the disconnect is done by way of teardown. It is beyond the scope of this post to go into detail on setup/teardown functions (when/how/why to use them, typical pitfalls in using them, etc.): this, also, will be the topic of another intended post.

Remarks:

The first argument is the fully qualified package name to the IBM jdbc driver class as contained in the driver jar (as mentioned above).

The second argument is the jdbc url. For information on both of these, see for instance this site. Note that my jdbc url points to the SAMPLE database as shipped with DB2 Express-C. The few lines in my sample ‘test case’ (see next bullet) will be using this SAMPLE db.

The third and fourth parameters are the credentials required to connect.

Use the Database Library keywords in your test code

•  Now that we have added a proper connect statement, we can start querying or manipulating the database. The following lines are just random examples of keywords that we are now able to use in our reusable test functions (i.e. keywords) and test designs (test cases) that we build from the test functions. So please note that the following lines are by no means whatsoever to be taken as an example of a meaningful, effective or well-structured test case! The ‘test case’ is merely a random set of examples of keyword calls. You need to create (layers of) domain functions (and probably other types of reusable resources) and create your test designs from those domain functions (see some of my other blog posts).

Note that I do not even bother here, to assign the output (return) of the ‘Execute Sql’ keyword to a variable.

Finally, please note that in the example the keyword names are prefixed by ‘DBLibrary’. As I have explained, ‘DBLibrary’ is the logical name or alias for the jdbc database library. I am prefixing that alias to the keyword names here, because I have also installed the other RF database library (the Python one, see above). Since these two libraries have certain keyword names in common, by default I prefix the alias to a keyword name, so as to avoid confusion and possible conflicts. If you have only one database library in use, the prefixing is not necessary.

• Run the test and inspect the results:

• Mission accomplished!

Summary

Running the RF on any given interpreter, potentially limits the availability of test libraries that we need to have available.
The RLI is a simple mechanism that greatly enhances the portability and scalability of the RF and enables us to stick with the interpreter of our choice without losing those test libraries that are in or of themselves incompatible with that interpreter. Running on Python we can use non-Python libraries (written e.g. in Java, Ruby, PHP, etc.), running on Jython we can use non-Java libraries (written e.g. in Python with C-extensions, Ruby, PHP, etc.) and running on IronPython we can use non-.Net libraries (written e.g. in Java, Perl, PHP, etc.). Through it we can also execute tests on remote systems that do not allow for remote calling structures or set up a distributed test infrastructure.
The Java Database library is an example of a library that can be used through the RLI. In our example, we have used this Java library to successfully connect to an IBM DB2 database while running the RF on Python.