Deploy and configure a local Docker caching proxy

Recently I was looking into caching for Docker layers downloading for the Fabric8 development environment, to allow me to trash the vms where my Docker daemon was running and still avoiding me to re-download basic images each single time I recreate my vm.

As usual, I tried to hit Google first, and was pointed to these couple of pages:

http://unpoucode.blogspot.it/2014/11/use-proxy-for-speeding-up-docker-images.html

and

https://blog.docker.com/2015/10/registry-proxy-cache-docker-open-source/

Since I use quite often Jerome Petazzoni’s approach for a transparent Squid + iptables to cache and sniff simple http traffic (usually http invocation from java programs), I’ve found that the first solution based made sense, so I have tried that first.

It turned out that the second link was what I was looking for; but I have still spent some good learning hours with the no longer working suggestions from the first one, learning the hard way that Squid doesn’t play that nice with Amazon’s Cloudfront CDN, used by Docker Hub. But I have to admit that it’s been fun.
Now I know how to forward calls to Squid to hit an intermediate interceptors that mangles with query params, headers and everything else.
I couldn’t find a working combination for Cloudfront but I am now probably able to reproduce the infamous Cats Proxy Prank. =)

Anyhow, as I was saying, what I was really looking for is that second link that shows you how to setup an intermediate Docker proxy, that your Docker daemon will try to hit, before defaulting to the usual Docker Hub public servers.

Almost everything that I needed was in that page, but I have found the information to be a little more cryptic that needed.

The main reason for that is because that example assumes I need security (TLS), which is not really my case since the proxy is completely local.

Additionally, it shows how to configure you Docker Registry using YAML configuration. Again, not really complex, but indeed more than needed.

Yes, because what you really need to bring up the simplest local (not secured) Docker proxy is this oneliner:

docker run -p 5000:5000 -d --restart=always --name registry   \
  -e REGISTRY_PROXY_REMOTEURL=http://registry-1.docker.io \
  registry:2

The interesting part here is that registry image, supports a smart alternative way to forward configuration to it, that saves you from passing it a YAML confguration file.

The idea, described here, is that if you follow a naming convention for the environment variables, that reflects the hierarchy of the YAML tree, you can turn something like:

...
proxy:
  remoteurl: http://registry-1.docker.io
...

That you should write in a .yaml file and pass to the process in this way:

docker run -d -p 5000:5000 --restart=always --name registry \
  -v `pwd`/config.yml:/etc/docker/registry/config.yml \
  registry:2

Into the much more conventient -e REGISTRY_PROXY_REMOTEURL=http://registry-1.docker.io runtime environment variable!

Let’s improve the example a little, so that we also pass our Docker proxy a non-volatile storage location for the cached layers, so that we are not losing them between invocations:

docker run -p 5000:5000 -d --restart=always --name registry   \
  -e REGISTRY_PROXY_REMOTEURL=http://registry-1.docker.io \
  -v /opt/shared/docker_registry_cache:/var/lib/registry \
  registry:2

Now we have everything we needed to save a good share of bandwidth, each time we need to get some Docker image that had already passed through our local proxy.

The only remaining bit is to tell our Docker daemon to be aware of the new proxy:

# update your docker daemon config, according to your distro
# content of my `/etc/sysconfig/docker` in Fedora 23
OPTIONS=" --registry-mirror=http://localhost:5000"

Reload (or restart) your Docker daemon and you are done! Just be aware that if you restart the daemon you might need also to re-start the Registry container, if you ware working on a single node.

An interesting discovery, it’s been learning that Docker daemon doesn’t break if it cannot find the specified registry-mirror. So you can add the configuration and forget about it, knowing that your interaction with Docker Hub will just benefit of possible hits your caching proxy, assuming it’s running.

You can see it working with the following tests:

docker logs -f registry

will log all the outgoing download requests, and once the set of requrests that compose a single pull image operation will be completed, you will also be able to check that the image is now completely served by your proxy with this invocation:

curl http://localhost:5000/v2/_catalog
# sample output
{"repositories":["drifting/true"]}

The article would be finished, but since I feel bad to show how to disable security on the internet, here’s also a very short and fully working and tested example of how to implement the same with TLS enabled:

# generate a self signed certificate; accept default for every value a part from Common Name where you have to put your box hostname
mkdir -p certs && openssl req  -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key  -x509 -days 365 -out certs/domain.crt

# copy to the locally trusted ones, steps for Fedora/Centos/RHEL
sudo cp certs/domain.crt /etc/pki/ca-trust/source/anchors/

# load the newly added certificate
sudo update-ca-trust enable

# run the registy using those keys that you have generated, mounting the files inside the container
docker run -p 5000:5000 --restart=always --name registry \
  -v `pwd`/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
  registry:2

# now you just need to remember that you are working in https, so you need to use that protocol in your docker daemon configuration, instead of plain http; also use that when you interact with the API in curl

Debugging tip: How to simulate a slow hardisk

As a Software Engineer there are times when you’d like to have a slower system.

It doesn’t happen really often actually: usually it’s when someone reports a bug on your software that you have never seen before and that you can not reproduce.

The majority of time, the reason of those ghost bugs are race conditions.

Race conditions, are issues you might face with multithreaded programming. Imagine that your software does multiple things at the same time. Despite most of the time those things happen in the expected and intuitive order, sometimes, they don’t; leading to unexpected state of your program.

They are indeed a bug. It’s dev’s fault. But the dev doesn’t have much way to protect himself from them. There are programming styles and technologies that push you to avoid the risk altogether, but I think that in general, they are a condition that each developer has to be familiar with.

So why a slower system helps?

Because most of the times, race conditions are masked by the fact the operations still happen “reasonably quickly”. This ambiguos “reasonably quickly” is the main issue. There is no clear limit or number that tells you how quickly. You just have higher chances to see them if things are slow enough to show they are not happening in the correct order or they are not waiting for the correct checkpoints.

In my experience with java applications, the main performance related aspect, while reproducing race condition is disk access speed. More thatn cpu speed or the amount of RAM, I have noticed that disk speed is the biggest differentiation between similar systems.

In this post I will show how to simulate a slow hardisk on Linux to increase your chances to reproduce race conditions.

The solution will be based on nbd and trickle and it will use the network layer to regulate the i/o throughput for your virtual hardisk.

I’d like to start adding that this isn’t anything new and that I’m not suggesting any particularly revolutionary approach. There are many blogpost out there that describe how to achieve this. But for multiple reasons, none of those that I have read worked out of the box on my Fedora 22 or Centos 6 installations.
That is the main reason that pushed me to give back to the internet, adding what might be, just another page on the argument.

Let’s start with the idea of using nbd or Network Block Device to simulate our hardisk.

As far as I understand there aren’t official ways, exposed by the linux Kernel to regulate the I/O speeds of generic block devices.

Over the internet you may find many suggestions, spanning from disabling read and write cache to geneate real load that could make your system busy.

QoS can be enforced on the network layer though. So the idea is to emulate a block device via network.

Despite this might sound very complicated (and maybe it is), the problem has been already been solved by the Linux Kernel with the nbd module.

Since on my Fedora 22 that module is not enabled automatically by default, we have to install it first, and then enable it:

# install nbd module
sudo yum install nbd

# load nbd module
sudo modprobe nbd

# check nbd module is really loaded
lsmod | grep nbd
nbd                    20480  0

Now that nbd is installed and the module loaded we create a configuration file for its daemon:

# run this command as root
"cat > /etc/nbd-server/config" <<EOF
[generic]
[test]
    exportname = /home/pantinor/test_nbd
    copyonwrite = false
EOF

Where exportname is a path to a file that will represent your slow virtual hardisk.

You can create the file with this command:

# create an empty file, and reserve it 1GB of space
dd if=/dev/zero of=/home/pantinor/test_nbd bs=1G count=1

Now that config and the destination files are in place, you can start the nbd-server using daemon:

# start ndb-server daemon
sudo systemctl start nbd-server.service

# monitor the daaemon start up with:
journalctl -f --unit nbd-server.service

At this point you have a server network process, listening on port 10809 that any client over your network can connect to , to mount it as a network block device.

We can mount it with this this command:

# "test" corresponds to the configuration section in daemon  config file
sudo nbd-client -N test  127.0.0.1 10809  /dev/nbd0
# my Centos 6 version of nbd-client needs a slightly different synatx:
#    sudo nbd-client -N test  127.0.0.1   /dev/nbd0

Now we have created a virtual block device, called /dev/nbd0. Now we can format it like it was a normal one:

# format device
sudo mkfs /dev/nbd0 

# create folder for mounting
sudo mkdir /mnt/nbd

# mount device, sync option is important to not allow the kernel to cheat!
sudo mount -o sync /dev/nbd0 /mnt/nbd

# add write permissions to everyone
sudo chmod a+rwx /mnt/nbd

Not that we have passed to mount command the flag -o sync. This command has an important function: to disable an enhancement in the linux Kernel that delays the completion of write operations to the devices. Without that all the write operations will look like instantaneous, and the kernel will actually complete the write requests in background. With this flag instead, all the operation will wait until the operation has really completed.

You can check that now you are able to read and write on the mount point /mnt/nbd.

Let’s now temporarily unmount and disconnect from nbd-server:

sudo umount /mnt/nbd

sudo nbd-client -d /dev/nbd0

And let’s introduce trickle.

Trickle is a software you can use to wrap other processes and to limit their networking bandwidth.

You can use it to limit any other program. A simple test you can perform with it is to use it with curl:

# download a sample file and limits download speed to 50 KB/s
trickle -d 50 -u 50  curl -O  http://download.thinkbroadband.com/5MB.zip

Now, as you can expect, we just need to join trickle and nbd-server behavior, to obtain the desired behavior.

Let’s start stopping current nbd-server daemon to free up its default port:

sudo systemctl stop nbd-server.service

And let’s start it via trickle:

# start nbd-server limiting its network throughput
trickle -d 20 -u 20 -v nbd-server -d

-d attaches the server process to the console, so the console will be blocked and it will be freed only one you close the process or when a client disconnects.
Ignore the error message: trickle: Could not reach trickled, working independently: No such file or directory

Now you can re-issue the commands to connect to nbd-server and re mount it:

sudo nbd-client -N test  127.0.0.1 10809  /dev/nbd0

sudo mount -o sync /dev/nbd0 /mnt/nbd

And you are done! Now you have a slow hardisk mounted on /dev/nbd0.

You can verify the slow behavior in this way:

sudo dd if=/dev/nbd0 of=/dev/null bs=65536 skip=100 count=10
10+0 records in
10+0 records out
655360 bytes (655 kB) copied, 18.8038 s, 34.9 kB/s

# when run against an nbd-server that doesn't use trickle the output is:
# 655360 bytes (655 kB) copied, 0.000723881 s, 905 MB/s

Now that you have a slow partition, you can just put the files of your sw there to simulate a slow i/o.

All the above steps can be converted to helper scripts that will make the process much simpler like those described here: http://philtortoise.blogspot.it/2013/09/simulating-slow-drive.html.

Share your Bash history across terminals in Terminator

As many developers I spent lots of time working with a command line shell. My operating system of choice is Fedora (currently 18) and I am using the excellent Terminator as a more powerful replacement for the basic default shell that comes with Gnome 3.

The typical workflow with Terminator is to use its very intuitive key shortcuts to split the current working window, maximize it, start working, resize when finish and jump back to any other window of interest.

It's a very fast workflow and I cannot see myself going back to anything different.

But this high productive usage has a drawback:

the commands history is not shared across the different windows!

This is often annoying since one of the reasons I am spawning a new shell is to do some "temporary" work without moving away from my current position. And this usually involves redoing some already executed step, so the lack of history is quite annoying.

I will describe here a way to address this problem.

The first thing to know is that the command history is not present because by default bash history is flushed to .bash_history only when you terminate a session.

A way to force a more frequent flush is to play with the environment variable PROMPT_COMMAND

If we modify one of our bash configuration files, .bashrc for instance.

#save history after every command
#use 'history -r' to reload history
PROMPT_COMMAND="history -a ; $PROMPT_COMMAND" 

What we are saying here is to invoke history -a with every new command prompt line. This flush the history to file every time we see a new command prompt. You can verify this if you monitory you .bash_history file.

Have we reached what we were hoping for?

Not yet.

Even if the history is now persisted, you will notice that you running shell do not see this history. This is because the history is loaded only at the beginning of a new session.

The only thing left is to manually force a reload of the history with the command

history -r

Done. We have now access to other shells history.

The last question:

why don't we add the reload of the history directly in the PROMPT_COMMAND variable?

Because you probably don't want that. Having all your shells always sharing a global history will break the most obvious behavior of the shell that is to show you the previous command you typed in that very shell.

Fedora 17 - Google Calendar Integration

I have just found this useful post about how to integrate your Google Calendar with Gnome3 Calendar:

http://www.webupd8.org/2011/09/google-calendar-gnome-shell-integration.html


I have just tried and it's working great!

( I cannot take a snapshot since the key is not working when the Calendar Overlay is open =P )

Gnome-shell: Show contact list

I have recently switched from RHEL 6 to Fedora 17.

I found myself less productive during the first days and I had the opportunity to agree with people that identify RHEL as a good distribution for enterprises and for people that need to be productive from the beginning, without losing time to adapt to a new environment, even if this could be winning in the long run.

But after this thoughts, that I still believe are true, I started to enjoy the experience of Fedora and Gnome3. I think they are able to satisfy you if you are the kind of person that likes to tweak his system and likes to experiment with it.

 In Fedora 17, one of the biggest area in which you can play with settings is Gnome3.

 Like other people I do believe that some of the decision of the developers have been to radical and that it's not giving out of the box the best experience that it could. A typical example is the fact that they've hidden the shutdown functionalities and that you have to press ALT to see them, something that everyone could agree is not intuitive at all. 

But maybe the idea behind all this radical decisions is that they want to offer a minimal customization, to allow people to change everything that they do not like and to extend the behaviour, a la Firefox.


And talking specifically about this customization, one extension that I've found useful is Empathy Contact List in UserMenu https://extensions.gnome.org/extension/396/empathy-contact-list-in-usermenu/


"Shows an Empathy Contact List launcher in UserMenu"


Something that is really useful in my opinion because otherwise, if you are not using external sw like Pidgin, you have to perform a long set of action to open a message window to one of your contact:

- you have to go in expose mode
- you have to type the name of the contact to open his  panel
- you have to click on the message button

Only after you have done all this you can have a message window.
Like others said, I do not think this is very handy.

The linked extension allows you to have easy access to the contact list. Stop.

But there is a problem. It doesn't work in Fedora 17.

The reason for this is that the code of the extension is trying to execute empathy.desktop, the program integrated in Gnome3 to provide instant messaging functionality, but in Fedora 17, the shortcut file, to execute the program has a slightly different name: fedora-empathy.desktop .

To correct this and fix the problem this is what you have to do:

1) open the source code of the extension here

vi ~/.local/share/gnome-shell/extensions/empathy-contacts-in-usermenu@rtt.uy/extension.js

2) edit line 19 changing from 'empathy.desktop' to 'fedora-empathy.desktop' and save

3) restart Gnome3 with ALT+F2 followed by the command r

Enjoy your functionality!


The last thing that I'd like to add is the way to access useful information when you try to understand what is happening with an extension that is not behaving properly. There is a log file that gives you lots of useful information.

The approach I suggest you is to execute this command in a shell before starting to use the problematic extension:

tail -f ~/.xsession-errors

In this way you can clearly see what is wrong. This was my log file, that led me to the fix:

    JS ERROR: !!!   Exception in callback for signal: activate
    JS ERROR: !!!     message = '"Shell.AppSystem.get_default().lookup_app("empathy.desktop") is null"'
    JS ERROR: !!!     lineNumber = '19'
    JS ERROR: !!!     fileName = '"/home/pantinor/.local/share/gnome-shell/extensions/empathy-contacts-in-usermenu@rtt.uy/extension.js"'
    JS ERROR: !!!     stack = '"([object Object],[object _private_Clutter_Event])@/home/pantinor/.local/share/gnome-shell/extensions/empathy-contacts-in-usermenu@rtt.uy/extension.js:19

Gnome Extensions Installation Silent Error

I was one of those unlucky ones not able to install Gnome Extensions via the website, on my Fedora 17.

 I've found out I wasn't the only one but we were not a big number and there was not a clear solution to our problem.

 Apparently I have found that is somewhat related to SSL and the time set in you laptop. Yes. I do not why, but it was definitely my case.  

This is the error I was receiving:

JS ERROR: !!!   Exception was: TypeError: info is null
JS ERROR: !!!     lineNumber = '83'JS ERROR: !!!     fileName = '"/usr/share/gnome-shell/js/ui/extensionSystem.js"'
JS ERROR: !!!     stack = '"([object _private_Soup_SessionAsync],[object _private_Soup_Message])@/usr/share/gnome-shell/js/ui/extensionSystem.js:83"'
JS ERROR: !!!     message = '"info is null"'

If you are also unable to join your Google account to your Gnome 3 profile, ( Check with the Google one since the other options were working fine ) you will probably have my same problem.

For some reason that I cannot explain it seems that the clock or the timezone set on you Linux are involved in the problem.

 Try to change them switching form either network updated to something else like New York timezone and repeat the process.
It's enough to check if you can join the Google account.

As soon as you see you are able, also the extension installation is fixed.

Open Shift on Feodra 17

I have just moved from RHEL 6 to Fedora 17. Not completely seamless but probably I had part in the mess. Anyway it's been quite simple and still feeling more a Windows user than a Linux one, seeing that to restore my previous programs and configurations it is just a matter of moving files, like I used to do on Windows choosing Portable Application make me feel happy (and save a lot of time)

I decided I needed to blog here since I have found a problem with Red Hat Open Shift installation procedure.

If you follow the official steps on Open Shift site you'll receive a 404 error, very disappointing.

But the bug has already been tracked:

https://bugzilla.redhat.com/show_bug.cgi?id=826808 

Anyway it seems that Fedora already includes the right packages, they are just with a slightly different name:

yum search rhc

#will lead you to: 

yum install rubygem-rhc

that will provide you the rhc command!