Docker is one of the new hot things out there. With a different set of technologies and ideas compared to traditional virtual machines, it implements something similar and at the same time different, with the idea of containers: almost all VMs power but much faster and with very interesting additional goodies.
In this article I assume you already know something about Docker and know how to interact with it. If it's not the case I can suggest you these links to start with:
My personal contribution to the topic is to show you a possible workflow that allows you to start and stop Docker containers from within a Maven job.
The reason why I have investigated in this functionality is to help with tests and integration tests in Java projects built with Maven.
The problem is well known: your code interacts with external systems and services. Depending on what you are really writing this could mean Databases, Message Brokers, Web Services and so on.
The usual strategies to test these interactions are:
In memory servers; implemented in java that are usually very fast but too often their limit is that they are not the real thing
A layer of stubbed services, that you implement to offers the interfaces that you need.
Real external processes, sometimes remote, to test real interactions.
Those strategies work but they often require a lot of effort to be put in place.
And the most complete one, that is the one that uses proper external services, poses problems for what concerns isolation: imagine that you are interacting with a database and that you perform read/write operations just while someone else was accessing the same resources. Again, you may find the correct workflows that invovle creating separate schemas and so on, but, again, this is extra work and very often a not very straight forward activity.
Wouldn't it be great if we could have the same opportunities that these external systems offers, but in totaly isolation? And what do you think if I also add speed to the offer?
Docker is a tool that offers us this opportunity.
You can start a set of Docker container with all the services that you need, at the beginning of the testing suite, and tear it down at the end of it. And your Maven job can be the only consumer of these services, with all the isolation that it needs. And you can all of this easily scripted with the help of Dockerfiles, that are, at the end, not much more than a sequential set of command line invocations.
Let see how to enable all of this.
The first prerequisite is obviously to have Docker installed on your system. As you may already know Docker technology depends on the capabilities of the Linux Kernel, so you have to be on Linux OR you need the help of a traditional VM to host the Docker server process.
This is the official documentation guide that shows you how to install under different Linux distros:
Once you are ready and you have Docker installed, you need to apply a specific configuration.
Docker, in recents versions, exposes its remote API, by default, only over Unix Sockets. Despite we could interact with them with the right code, I find much easier to interact with the API over HTTP.
To obtain this, you have to pass a specific flag to the Docker daemon to tell it to listen also on HTTP.
I am using Fedora, and the configuration file to modify is /usr/lib/systemd/system/docker.service.
The only modification compared to the defaults it's been adding -H tcp://127.0.0.1:4243.
Now, after I have reloaded systemd scripts and restarted the service I have a Docker daemon that exposes me a nice REST API I can poke with curl.
sudo systemctl daemon-reload
sudo systemctl restart docker
curl http://127.0.0.1:4243/images/json # returns a json in output
You probably also want this configuration to survive future Docker rpm updates. To achieve that you have to copy the file you have just modified to a location that survives rpm updates. The correct way to achieve this in systemd is with:
Now we have all we need to interact easily with Docker.
You may at this point expect me to describe you how to use the Maven Docker plugin. Unluckily that's not the case. There is no such plugin yet, or at least I am not aware of it. I am considering writing one but for the moment being I have solved my problems quickly with the help of GMaven plugin, a little bit of Groovy code and the help of the java library Rest-assured.
Here is the code to startup Docker containers
import com.jayway.restassured.RestAssured
import static com.jayway.restassured.RestAssured.*
import static com.jayway.restassured.matcher.RestAssuredMatchers.*
import com.jayway.restassured.path.json.JsonPath
import com.jayway.restassured.response.Response
RestAssured.baseURI = "http://127.0.0.1"
RestAssured.port = 4243
// here you can specify advance docker params, but the mandatory one is the name of the Image you want to use
def dockerImageConf = '{"Image":"${docker.image}"}'
def dockerImageName = JsonPath.from(dockerImageConf).get("Image")
log.info "Creating new Docker container from image $dockerImageName"
def response = with().body(dockerImageConf).post("/containers/create")
if( 404 == response.statusCode ) {
log.info "Docker image not found in local repo. Trying to dowload image '$dockerImageName' from remote repos"
response = with().parameter("fromImage", dockerImageName).post("/images/create")
def message = response.asString()
//odd: rest api always returns 200 and doesn't return proper json. I have to grep
if( message.contains("HTTP code: 404") ) fail("Image $dockerImageName NOT FOUND remotely. Abort. $message}")
log.info "Image downloaded"
// retry to create the container
response = with().body(dockerImageConf).post("/containers/create")
if( 404 == response.statusCode ) fail("Unable to create container with conf $dockerImageConf: ${response.asString()}")
}
def containerId = response.jsonPath().get("Id")
log.info "Container created with id $containerId"
// set the containerId to be retrieved later during the stop phase
project.properties.setProperty("containerId", "$containerId")
log.info "Starting container $containerId"
with().post("/containers/$containerId/start").asString()
def ip = with().get("/containers/$containerId/json").path("NetworkSettings.IPAddress")
log.info "Container started with ip: $ip"
System.setProperty("MONGODB_HOSTNAME", "$ip")
System.setProperty("MONGODB_PORT", "27017")
Rest-assured fluent API should suggest what is happening, and the inline comment should clarify it but let me add a couple of comments.
The code to start a container is my implementation of the functionality of docker run as described in the official API documentation here:
The specific problem I had to solve was how to propagate the id of my Docker container from a Maven Phase to another one. I have achieved the functionality thanks to the line:
// set the containerId to be retrieved later during the stop phase
project.properties.setProperty("containerId", "$containerId")
I have also exposed a couple of Maven properties that can be useful to interact with the API:
docker.image - The name of the image you want to spin
docker.remove.container - If set to false, tells Maven to not remove the stopped container from filesystem (useful to inspect your docker container after the job has finished)
You may find here a full working example. I have been told that sometimes my syntax colorizer script eats some keyword or change the case of words, so if you want to copy and paste it may be a better idea cropping from Github.
This is a portion of the output while running the Maven build with the command mvn verify :
...
[INFO] --- gmaven-plugin:1.4:execute (start-docker-images) @ gmaven-docker ---
[INFO] Creating new Docker container from image {"Image":"pantinor/centos-mongodb"}
log4j:WARN No appenders could be found for logger (org.apache.http.impl.conn.BasicClientConnectionManager).
log4j:WARN Please initialize the log4j system properly.
[INFO] Container created with id 5283d970dc16bd7d64ec08744b5ecec09b57d9a81162826e847666b8fb421dbc
[INFO] Starting container 5283d970dc16bd7d64ec08744b5ecec09b57d9a81162826e847666b8fb421dbc
[INFO] Container started with ip: 172.17.0.2
...
[INFO] --- gmaven-plugin:1.4:execute (stop-docker-images) @ gmaven-docker ---
[INFO] Stopping Docker container 5283d970dc16bd7d64ec08744b5ecec09b57d9a81162826e847666b8fb421dbc
[INFO] Docker container stopped
[INFO] Docker container deleted
...
If you have any question or suggestion please feel free to let me know!
Let's assume that we have to execute a bunch of acceptance tests with a BDD framework like Cucumber as part of a Maven build.
Using Maven Failsafe Plugin is not complex. But it has an implicit requirement: The container that hosts the implementation we are about to test needs to be already running.
Many containers like Jetty or JBoss provide their own Maven plugins, to allow to start the server as part of a Maven job. And there is also the good generic Maven Cargo plguin that offers an implementation of the same behavior for many different container.
These plugins allow for instance, to start the server at the beginning of a Maven job, deploy the implementation that you want to test, fire your tests and stop the server at the end.
All the mechanisms that I have described work and they are usually very useful for the various testing approaches.
Unluckily, I cannot apply this solution if my container is not a supported container. Unless obviuosly, I decide to write a custom plugin or add the support to my specific container to Maven Cargo.
In my specific case I had to find a way to use Red Hat's JBoss Fuse, a Karaf based container.
I decided to try keeping it easy and to not write a full featured Maven plugin and eventually to rely to GMaven plugin, or how I have recently read on the internet the "Poor Man's Gradle".
GMaven is basically a plugin to add Groovy support to you Maven job, allowing you to execute snippets of Groovy as part of your job. I like it because it allows me to inline scripts directly in the pom.xml.
It permits you also to define your script in a separate file and execute it, but that is exactly the same behaviour you could achieve with plain java and Maven Exec Plugin; a solution that I do not like much because hides the implementation and makes harder to imagine what the full build is trying to achieve.
Obviously this approach make sense if the script you are about to write are simple enough to be autodescriptive.
I will describe my solution starting with sharing with you my trial and errors and references to various articles and posts I have found:
At first I have considered to use Maven Exec Plugin to directly launch my container. Something like what was suggested here
That plugin invocation, as part of a Maven job, actually allows me to start the container, but it has a huge drawback: he Maven lifecycle stops until the external process terminates or is manually stopped.
This is because the external process execution is "synchronous" and Maven doesn't consider the command execution finished, so, it never goes on with the rest of the build instructions.
This is not what I needed, so I have looked for something different.
At first I have found this suggestion to start a background process to allow Maven not to block:
This approach actually works. My Maven build doesn't stop and the next lifecycle steps are executed.
But I have a new problem now. My next steps are immediately executed.
I have no way to trigger the continuation only after my container is up and running.
Browsing a little more I have found this nice article:
The article, very well written, seems to describe exactly my scenario. It's also applied to my exact context, trying to start a flavour of Karaf.
It uses a different approach to start the process in background, the Antrun Maven plugin. I gave it a try and unluckily I am in the exact same situation as before. The integration tests are executed immediately, after the request to start the container but before the container is ready.
Convinced that I couldn't find any ready solution I decided to hack the current one with the help of some imperative code.
I thought that I could insert a "wait script", after the start request but before integration test are fired, that could check for a condition that assures me that the container is available.
So, if the container is started during this phase:
pre-integration-test
and my acceptance tests are started during the very next
integration-test
I can insert some logic in pre-integration-test that keeps polling my container and that returns only after the container is "considered" available.
import static com.jayway.restassured.RestAssured.*;
println("Wait for FUSE to be available")
for(int i = 0; i < 30; i++) {
try{
def response = with().get("http://localhost:8383/hawtio")
def status = response.getStatusLine()
println(status)
} catch(Exception e){
Thread.sleep(1000)
continue
}finally{
print(".")
}
if( !(status ==~ /.*OK.*/) )
Thread.sleep(1000)
}
And is executed by this GMaven instance:
org.codehaus.gmavengmaven-plugin1.8########### wait for FUSE to be available ############pre-integration-testexecute<![CDATA[
import static com.jayway.restassured.RestAssured.*;
...
}
]]>
My (ugly) script, uses Rest-assured and an exception based logic to check for 30 seconds if a web resource, that I know my container is deploying will be available.
This check is not as robust as I'd like to, since it checks for a specific resource but it's not necessary a confirmation that the whole deploy process has finished. Eventually, a better solution would be use some management API that could be able to check the state of the container, but honestly I do not know if they are exposed by Karaf and my simple check was enough for my limited use case.
With the GMaven invocation, now my maven build is behaving like I wanted.
This post showed a way to enrich your Maven script with some programmatic logic without the need of writing a full featured Maven plugin. Since you have full access to the Groovy context, you can think to perform any kind of task that you could find helpful. For instance you could also start background threads that will allow the Maven lifecycle to progress while keep executing your logic.
My last suggestion is to try keeping the logic in your scripts simple and to not turn them in long and complex programs. Readability was the reason I decided to use rest-assured instead of direct access to Apache HttpClient.
This is a sample full pom.xml
4.0.0${groupId}.${artifactId}xxxxxxxesb1.0.0-SNAPSHOTacceptance/data/software/RedHat/FUSE/fuse_full/jboss-fuse-6.0.0.redhat-024/bin/maven-failsafe-plugin2.12.2integration-testverifyorg.apache.maven.pluginsmaven-surefire-plugin**/*Test*.javaintegration-testtestintegration-testnone**/RunCucumberTests.javamaven-antrun-plugin1.6############## start-fuse ################pre-integration-testrunmaven-antrun-plugin1.6############## stop-fuse ################post-integration-testrunorg.codehaus.gmavengmaven-plugin1.8########### wait for FUSE to be available ############pre-integration-testexecute<![CDATA[
import static com.jayway.restassured.RestAssured.*;
println("Wait for FUSE to be available")
for(int i = 0; i < 30; i++) {
try{
def response = with().get("http://localhost:8383/hawtio")
def status = response.getStatusLine()
println(status)
} catch(Exception e){
Thread.sleep(1000)
continue
}finally{
print(".")
}
if( !(status ==~ /.*OK.*/) )
Thread.sleep(1000)
}
]]>
info.cukescucumber-java${cucumber.version}testinfo.cukescucumber-picocontainer${cucumber.version}testinfo.cukescucumber-junit${cucumber.version}testjunitjunit4.11testorg.apache.httpcomponentshttpclient4.2.5com.jayway.restassuredrest-assured1.8.1
In a previous post Rest Invocation with Maven I've seen how to invoke a REST service from a Maven Pom file, using the Maven Groovy plugin.
In this post I will show how to upload a file to a webpage, still using some Groovy code.
We will do this in 2 different ways: using plain Apache Http Client and using Rest-assured, the library already described here in this previous post, Rest Assured or Rest-very-Easy
Apache Http Client
Groovy Script
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.client.methods.HttpPost
import org.apache.http.entity.mime.MultipartEntity
import org.apache.http.entity.mime.content.FileBody
import org.apache.http.auth.AuthScope
import org.apache.http.auth.UsernamePasswordCredentials
def name = "${input_file}"
log.info( "Archive file: $name" )
def f = new File(name)
// The execution:
DefaultHttpClient httpclient = new DefaultHttpClient()
httpclient.getCredentialsProvider().setCredentials(
new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT),
new UsernamePasswordCredentials( "${username}",
"${password}" )
)
def post = new HttpPost("${form_endpoint}")
def entity = new MultipartEntity()
def fileBody = new FileBody(f)
entity.addPart("file", fileBody)
post.setEntity(entity)
def response = httpclient.execute(post)
def status = response.getStatusLine()
if( !(status ==~ /.*OK.*/) )
fail("Unable to deploy. Return status code: $status" )
else
log.info("Deployment Successful: Result status code $status")
The solution based on HttpClient works fine but it's a little clumsy. We have already agreed that having a script embedded in a pom.xml file is handy but not as clean as writing a full featured Maven Plugin.
But even without a complete Maven Plugin we can improve the readability of the script thanks to Rest-assured and the fluent style that allows us to write a much cleared script.
And at the same time we can reduce the number of the direct dependencies in our pom.xml, since we are delegating the rest-assured itself to declare what it needs, that by the way, is again Apache Http Client, since Rest-assured is based on it.
Notice that my script distinguish between .zip and non zip input files, but this distinction it's only dued to the fact that my endpoint was "confused" when I was passing an .zip file without specifying any mimetype. The default mimetype for rest-assured, when you use an overloaded version of .multipart() is application/octet-stream
Groovy Script
import static com.jayway.restassured.RestAssured.*;
def name = "${input_file}"
log.info("Uploading Archive file: $name")
//we have to determine the mimetype to correctly support both zip and xml
def mimeType = name.endsWith("zip") ? "application/zip" : "application/xml"
def f = new File(name)
def response = with()
.auth().basic("${username}", "${password}")
.multiPart("file", f, mimeType)
.post("${form_endpoint")
def status = response.getStatusLine()
if( !(status ==~ /.*OK.*/) )
fail("Unable to deploy. Return status code: $status" )
else
log.info("Deployment Successful: Result status code $status")
Recently I had to write some Java code to consume REST services over HTTP.
I've decided to use the Client libraries of RestEasy, the framework I use most of the time to expose REST services in Java, since it also implements the official JAX-RS specification.
I am very satisfied with the annotation driven approach that the specification defines and it makes exposing REST services a very pleasant task.
But unluckily I cannot say that I like the client API the same way.
If you are lucky enough to be able to build a proxy client based on the interface implemented by the service, well, that's not bad:
import org.jboss.resteasy.client.ProxyFactory;
...
// this initialization only needs to be done once per VM
RegisterBuiltin.register(ResteasyProviderFactory.getInstance());
SimpleClient client = ProxyFactory.create(MyRestServiceInterface.class, "http://localhost:8081");
client.myBusinessMethod("hello world");
Having a Proxy client similar to a JAX-WS one is good, I do agree.
But most of the time, when we are consuming REST web service we do not have a Java interface to import.
All those Twitter, Google or whatever public rest services available out there are just HTTP endpoints.
The way to go with RestEasy in these cases is to rely on the RestEasy Manual ClientRequest API:
ClientRequest request = new ClientRequest("http://localhost:8080/some/path");
request.header("custom-header", "value");
// We're posting XML and a JAXB object
request.body("application/xml", someJaxb);
// we're expecting a String back
ClientResponse<String> response = request.post(String.class);
if (response.getStatus() == 200) // OK!
{
String str = response.getEntity();
}
That is in my opinion a very verbose way to fetch what is most of the time, just a bunch of strings from the web.
And it gets even worse if you need to include Authentication informations:
// Configure HttpClient to authenticate preemptively
// by prepopulating the authentication data cache.
// 1. Create AuthCache instance
AuthCache authCache = new BasicAuthCache();
// 2. Generate BASIC scheme object and add it to the local auth cache
BasicScheme basicAuth = new BasicScheme();
authCache.put("com.bluemonkeydiamond.sippycups", basicAuth);
// 3. Add AuthCache to the execution context
BasicHttpContext localContext = new BasicHttpContext();
localContext.setAttribute(ClientContext.AUTH_CACHE, authCache);
// 4. Create client executor and proxy
httpClient = new DefaultHttpClient();
ApacheHttpClient4Executor executor = new ApacheHttpClient4Executor(httpClient, localContext);
client = ProxyFactory.create(BookStoreService.class, url, executor);
{"id":27016395,"id_str":"27016395","name":"Resteasy","screen_name":"resteasy","location":"","url":null,"description":"jboss.org\/resteasy\n\nJBoss\/Red Hat REST project","protected":false,"followers_count":244,"friends_count":1,"listed_count":21,"created_at":"Fri Mar 27 14:39:52 +0000 2009","favourites_count":0,"utc_offset":null,"time_zone":null,"geo_enabled":false,"verified":false,"statuses_count":8,"lang":"en","status":{"created_at":"Tue Mar 23 14:48:51 +0000 2010","id":10928528312,"id_str":"10928528312","text":"Doing free webinar tomorrow on REST, JAX-RS, RESTEasy, and REST-*. Only 40 min, so its brief. http:\/\/tinyurl.com\/yz6xwek","source":"web","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":0,"favorited":false,"retweeted":false},"contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_image_url":"http:\/\/a0.twimg.com\/sticky\/default_profile_images\/default_profile_0_normal.png","profile_image_url_https":"https:\/\/si0.twimg.com\/sticky\/default_profile_images\/default_profile_0_normal.png","profile_link_color":"0084B4","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"default_profile":true,"default_profile_image":true,"following":null,"follow_request_sent":null,"notifications":null}
And the same interface undestands JSON object navigation. Note that the navigation expression does not include "user" since it was not there in the full json response:
And a Groovy snippet that can be pasted and executed directly in groovyConsole thanks to Grapes fetches the dependencies and add them automatically to the classpath, that shows you JAXB support:
This is just a brief list of the features of the library, just you an idea of how easy it is to work with it.
For a further examples I suggest you to read the official pages here:
