My Blog

A diversion from the day job : Docker and R Clients

by admin on 18th February 2015 1 comment

In the last article I wrote I discussed an Oracle R Enterprise (ORE) project we were building for a customer. Following getting the server side stuff setup we realised we had a bunch of clients to deploy to and I decided to go with a Docker solution given that a) it would allow us to deploy to a larger number of users really quickly and b) it gave me the opportunity to in effect play with what’s (in my mind at least) going to be a really powerful tool in a lot of peoples kit bags in the near term.

Now this article isn’t designed to be a primer on docker, nor is it designed to discuss and debate the architecture, if you’re looking for that there are tons of better placed people than me! Why I chose to look at Docker was the fact that we could package a small image (in the end about 1.4GB) and deploy just this to the clients needing the work. No more building a whole bunch of VMs, no requirements to run install scripts on a load of client machines and worry about the inevitable breakages.

So how did I go about it? Well I run Mac as my desktop of choice and as Docker is based on Linux Container (Lxc) I needed to us eh nice handy Boot2Docker tool (http://boot2docker.io/)

Install is very straight forward on both Mac and Windows, and only takes a few minutes, once it’s installed run the boot2Docker application and you’ll be presented with a bash shell. So first just test it’s working by running the obligatory “hello world” application. In the output below you can see how it pulls the image down automatically and run it, and the “Hello from Docker” in the output shows its working.

 

bash-3.2$ docker run hello-world
Unable to find image ‘hello-world:latest’ locally
511136ea3c5a: Pull complete
31cbccb51277: Pull complete
e45a5af57b00: Pull complete
hello-world:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
(Assuming it was not already locally available.)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bashFor more examples and ideas, visit:
http://docs.docker.com/userguide/
You might be asking where the image was pulled from? That’ll be the Docker repository of publicly available images, and quite handily for me I queried this and found an Oracle Linux image, you can simply use the “docker pull” syntax to grab this image locally..
bash-3.2$ docker pull oracle/oraclelinux
Pulling repository oracle/oraclelinux
1d675f101368: Pulling dependent layers
511136ea3c5a: Download complete
9083863f17ce: Download complete
7765c87c37bb: Downloading 8.616 MB/98.43 MB 57s
And once it’s finished you’ll see the output below. I now have an Oracle Linux image.
bash-3.2$ docker pull oracle/oraclelinux
Pulling repository oracle/oraclelinux
1d675f101368: Download complete
511136ea3c5a: Download complete
9083863f17ce: Download complete
7765c87c37bb: Download complete
Status: Downloaded newer image for oracle/oraclelinux:latest
Let’s run that image and see what we get. For this I use the run -i -t syntax specifying that I want a bash shell to be executed from the image itself.
bash-3.2$ docker run -i -t oracle/oraclelinux /bin/bash
[root@b809f34547f4 /]# pwd
/
So now I’m in let’s have a look at the output of a uname and oracle-release
[root@b809f34547f4 /]# uname -a
Linux b809f34547f4 3.18.5-tinycore64 #1 SMP Sun Feb 1 06:02:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@b809f34547f4 /]# cat /etc/oracle-release
Oracle Linux Server release 7.0
The image we downloaded, probably as expected, in an Oracle Linux 7 image. My task was to create a small image with the R client on it we could distribute to developers. Let’s get on an add the R stuff into the image. The output below is truncated for brevity
First add the R rpms (the Oracle image comes handily setup with the public yum repository)
[root@b809f34547f4 client]# yum install R.x86_64
ol7_UEKR3                                                | 1.2 kB     00:00
ol7_addons                                               | 1.2 kB     00:00
ol7_latest                                               | 1.4 kB     00:00
ol7_optional_latest                                      | 1.2 kB     00:00
Now download and unzip the Oracle instant client  (i’ve removed the wget output that I used to get the zip files from otn.oracle.com to make the output easier to read)
[root@b809f34547f4 /]# mkdir /oreclient
[root@b809f34547f4 oreclient]# unzip instantclient-sdk-linux.x64-11.2.0.4.0.zip
Archive:  instantclient-sdk-linux.x64-11.2.0.4.0.zip
creating: instantclient_11_2/sdk/
creating: instantclient_11_2/sdk/include/
inflating: instantclient_11_2/sdk/include/occi.h
inflating: instantclient_11_2/sdk/include/occiCommon.h
inflating: instantclient_11_2/sdk/include/occiControl.h
inflating: instantclient_11_2/sdk/include/occiData.h
[root@b809f34547f4 oreclient]# unzip instantclient-sdk-linux.x64-11.2.0.4.0.zip
Archive:  instantclient-sdk-linux.x64-11.2.0.4.0.zip
replace instantclient_11_2/sdk/include/occi.h? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
inflating: instantclient_11_2/sdk/include/occi.h
inflating: instantclient_11_2/sdk/include/occiCommon.h
inflating: instantclient_11_2/sdk/include/occiControl.h
inflating: instantclient_11_2/sdk/include/occiData.h
inflating: instantclient_11_2/sdk/include/occiObjects.h
Now unzip the ORE client zip file
[root@b809f34547f4 oreclient]# unzip ore-client-linux-x86-64-1.4.1.zip
Archive:  ore-client-linux-x86-64-1.4.1.zip
inflating: client/ORE_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
  inflating: client/OREbase_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
Change to that directory and run the R install command on the libraries provided.
[root@b809f34547f4 oreclient]# cd client/
[root@b809f34547f4 client]# pwd
/oreclient/client
[root@b809f34547f4 client]# R CMD INSTALL ORE_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘ORE’ …
* DONE (ORE)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREbase_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREbase’ …
* DONE (OREbase)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREcommon_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREcommon’ …
* DONE (OREcommon)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREdm_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREdm’ …
* DONE (OREdm)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREeda_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREeda’ …
* DONE (OREeda)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREembed_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREembed’ …
* DONE (OREembed)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREgraphics_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREgraphics’ …
* DONE (OREgraphics)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREmodels_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREmodels’ …
* DONE (OREmodels)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREpredict_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREpredict’ …
* DONE (OREpredict)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL OREstats_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘OREstats’ …
* DONE (OREstats)
Making ‘packages.html’ … done
[root@b809f34547f4 client]# R CMD INSTALL ORExml_1.4.1_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘ORExml’ …
* DONE (ORExml)
Making ‘packages.html’ … done
Now do the same for the supporting libraries
[root@b809f34547f4 oreclient]# unzip ore-supporting-linux-x86-64-1.4.1.zip
Archive:  ore-supporting-linux-x86-64-1.4.1.zip
inflating: supporting/Cairo_1.5-5_R_x86_64-unknown-linux-gnu.tar.gz
inflating: supporting/DBI_0.2-7_R_x86_64-unknown-linux-gnu.tar.gz
inflating: supporting/ROracle_1.1-12_R_x86_64-unknown-linux-gnu.tar.gz
inflating: supporting/arules_1.1-3_R_x86_64-unknown-linux-gnu.tar.gz
inflating: supporting/png_0.1-7_R_x86_64-unknown-linux-gnu.tar.gz
  inflating: supporting/statmod_1.4.20_R_x86_64-unknown-linux-gnu.tar.gz
[root@b809f34547f4 supporting]# R CMD INSTALL ROracle_1.1-12_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘ROracle’ …
* DONE (ROracle)
Making ‘packages.html’ … done
[root@b809f34547f4 supporting]# R CMD INSTALL DBI_0.2-7_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘DBI’ …
* DONE (DBI)
Making ‘packages.html’ … done
[root@b809f34547f4 supporting]# R CMD INSTALL png_0.1-7_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘png’ …
* DONE (png)
Making ‘packages.html’ … done
[root@b809f34547f4 supporting]# R CMD INSTALL Cairo_1.5-5_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘Cairo’ …
* DONE (Cairo)
Making ‘packages.html’ … done
[root@b809f34547f4 supporting]# R CMD INSTALL arules_1.1-3_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘arules’ …
* DONE (arules)
Making ‘packages.html’ … done
[root@b809f34547f4 supporting]# R CMD INSTALL statmod_1.4.20_R_x86_64-unknown-linux-gnu.tar.gz
* installing to library ‘/usr/lib64/R/library’
* installing *binary* package ‘statmod’ …
* DONE (statmod)
Making ‘packages.html’ … done
Right that’s it! Nice an easy.
The important thing to remember at this point with Docker is that the base image I started hasn’t changed. A copy-on-change filesystem is implemented and my changes are recorded there, but if I stop this and restart that same image all my changes will be lost.
To preserve my changes I need to create a new image from the running container. From another shell I can run the “docker ps” command to see the running images.
bash-3.2$ docker ps
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS               NAMES
b809f34547f4        oracle/oraclelinux:latest   “/bin/bash”         28 minutes ago      Up 28 minutes                           lonely_kirch
Notice in this that the container id (b809f34547f4) is the “hostname” that has been showing on the command line output I’ve been pasting. If you’d been wondering where that comes from, you now know.
Now I’m going to use the docker commit syntax to create an image from that running container.
bash-3.2$ docker commit -m=”Test R Distribution” -a=”James Anthony” b809f34547f4 jamescanthony/r_distribution:v1
e3b548c3e7062d1ed7904dc3c613d02c1657cbec05b197847543b03b4f2c5d8d
And I can use “docker images” to see my newly created image, that it has a size of just over 1.3GB
bash-3.2$ docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
jamescanthony/r_distribution   v1                  e3b548c3e706        About a minute ago   1.345 GB
oraclelinux                    latest              3200155fe586        12 days ago          195.1 MB
oracle/oraclelinux             latest              1d675f101368        2 weeks ago          265.2 MB
ubuntu                         14.04               5ba9dab47459        2 weeks ago          188.3 MB
ubuntu                         14.04.1             5ba9dab47459        2 weeks ago          188.3 MB
ubuntu                         latest              5ba9dab47459        2 weeks ago          188.3 MB
ubuntu                         trusty              5ba9dab47459        2 weeks ago          188.3 MB
hello-world                    latest              e45a5af57b00        6 weeks ago          910 B
bash-3.2$
Without closing my still running container (creating the image didn’t reboot or shutdown and was very fast) I’m going to fire up the new image and make sure it’s all working.
bash-3.2$ docker  run -t -i jamescanthony/r_distribution:v1 /bin/bash
[root@03d7bb2cd2f2 /]#
Notice I’ve got a new Image ID in there. Let’s launch R  and see what we get…
[root@03d7bb2cd2f2 /]# R
Oracle Distribution of R version 3.1.1  (–) — “Sock it to Me”
Copyright (C)  The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type ‘license()’ or ‘licence()’ for distribution details.R is a collaborative project with many contributors.
Type ‘contributors()’ for more information and
‘citation()’ on how to cite R or R packages in publications.Type ‘demo()’ for some demos, ‘help()’ for on-line help, or
‘help.start()’ for an HTML browser interface to help.
Type ‘q()’ to quit R.You are using Oracle’s distribution of R. Please contact
Oracle Support for any problems you encounter with this
distribution.
So far so good, let’s load the ORE libraries
> library(ORE)
Loading required package: OREbaseAttaching package: ‘OREbase’The following objects are masked from ‘package:base’:cbind, data.frame, eval, interaction, order, paste, pmax, pmin,
rbind, tableLoading required package: OREembed
Error in dyn.load(file, DLLpath = DLLpath, …) :
unable to load shared object ‘/usr/lib64/R/library/png/libs/png.so’:
libpng12.so.0: cannot open shared object file: No such file or directory
Error: package ‘OREembed’ could not be loaded
> quit()
Save workspace image? [y/n/c]: n
A problem! Well I sort of knew this was going to be an issue, but it illustrates another great point. I’ve cloned my running image to test it works, it doesn’t and I know what the problem is. In this case the libpng12 library isn’t installed. So back in my original shell I simply use yum to install this.
[root@b809f34547f4 client]# yum install libpng12
Resolving Dependencies
–> Running transaction check
—> Package libpng12.x86_64 0:1.2.50-6.el7 will be installed
–> Finished Dependency Resolution…
Installed:
libpng12.x86_64 0:1.2.50-6.el7Complete!
Now I simply take another commit, this time giving it a tag of v2
bash-3.2$ docker commit -m=”Test R Distribution” -a=”James Anthony” b809f34547f4 jamescanthony/r_distribution:v2
112c79ddee63fee8f542a0eddd103dff3bab072be3733244867c1b4fa76e1a51
And now I can fire this up and test…
bash-3.2$ docker  run -t -i jamescanthony/r_distribution:v2 /bin/bash
[root@e0120571a7be /]# ROracle Distribution of R version 3.1.1  (–) — “Sock it to Me”
Copyright (C)  The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type ‘license()’ or ‘licence()’ for distribution details.R is a collaborative project with many contributors.
Type ‘contributors()’ for more information and
‘citation()’ on how to cite R or R packages in publications.Type ‘demo()’ for some demos, ‘help()’ for on-line help, or
‘help.start()’ for an HTML browser interface to help.
Type ‘q()’ to quit R.You are using Oracle’s distribution of R. Please contact
Oracle Support for any problems you encounter with this
distribution.> library(ORE)
Loading required package: OREbaseAttaching package: ‘OREbase’The following objects are masked from ‘package:base’:cbind, data.frame, eval, interaction, order, paste, pmax, pmin,
rbind, tableLoading required package: OREembed
Loading required package: OREstats
Loading required package: MASS
Loading required package: OREgraphics
Loading required package: OREeda
Loading required package: OREmodels
Loading required package: OREdm
Loading required package: lattice
Loading required package: OREpredict
Loading required package: ORExml
Success! At this stage all that is left is to commit this to the docker repository and then my users can go away and download it.
Anyone who’s had developers, sys admins or anyone else requiring a pre-packaged set of tools, and has created build scripts, documentation etc can see the power of this. Imagine wanting 500 users all to just upgrade to the latest version of something. With Docker they just upgrade their images at a few hundred MB, or maybe like in this case a small 1.5GB download (the copy-on-change part). And it’s all isolated from their existing desktop stuff or other images, so no conflict, no problems. Docker really does have the potential to simplify a lot of what we do.
I’ll be exploring the isolation and resource constraint facilities (especially the IO side) in the next few weeks for some web, app and DB side work. If you’re interested let me know.

adminA diversion from the day job : Docker and R Clients

Related Posts

Take a look at these posts

1 comment

Join the conversation
  • Jakob - 6th October 2015 reply

    Hello there! This post couldn’t be written any better! Looking through this article reminds me of my previous roommate! He continually kept preaching about this. I am going to send this information to him. Fairly certain he’ll have a great read. I appreciate you for sharing!

Join the conversation