A recent encounter with a customer resulted in a couple good questions regarding the workflow that one would use to deploy apps to Cloud Foundry in order to try for a 100% up-time.  Based on that, I would share the questions and answers here.

Question #1 - How do you push updates to your application without downtime?

Currently when you push, or restart for that matter, an application running on Cloud Foundry, the change is applied in a series of steps that go roughly like this.

• New app bits are uploaded
• The current version of the app is stopped
• Staging for the new app occurs (i.e. the build pack runs)
• The new app is started

What’s important to understand about this process is that when the app is stopped all instances of your application are stopped. Thus there will be some small amount of downtime, while your new app stages and is started.

The typical suggestion for working around this is to do what are called blue / green deployments, which work by running both the current and new version of application at the same time. Since both apps are running, you can switch to the new app in a controlled fashion by simply manipulating the routes, something that happens instantly and does not require downtime.

Question #2 - How do you push updates to your application if it’s not taking web requests but still needs to maintain high availability?

If you have an application that is not taking HTTP requests, like a background worker, the typical blue / green deployment scenario may or may not work for you. If you’re running a background worker and need to keep it highly available, here are some things to consider.

1. If you have a background worker style task, it may be as simple as starting a second instance of the application that is running the new code and then stopping the old instance. The key to making this work on CF is to use different application names (both bound to the same service, if the worker is using services). This will enable both to run at the same time and allow you to shutdown the old worker instance when you’re satisfied that the new code is working properly.
   
   Before doing this though, please keep in mind that there will be a window of time where there are two versions of your application running. This means that before adopting this approach, you should consider what will happen if there are two versions of your worker running at the same time. Will they play nice together or will they compete for the work, and will they both be compatible (i.e. did the database schema change, did message formats change, etc..).
   
2. Another solution to this problem is to simply ignore it. Depending on the architecture of your application you may be able to just push your new changes and ignore the fact that the application will be down for a small window of time. This will generally be the case for background worker tasks that are simply pulling jobs from a queue or database. This flexibility comes from the fact that by their nature the database or queue will hold the jobs while your application is not running. Given this, all you need to do is push the new change and wait for the app to catch up on it’s work.
   
   Before going with this approach, there are some important things that you should consider. First, you should have a good understanding of how long it will take for your new version of the application to stage, start up and begin doing work. This is critical and leads us to the second point. You need to have a good estimate as to how much work will be queued up while the application is restarting and if your service is capable of storing that much data. This is key to not losing any work while the new version of your application is starting up.
   
   Lastly, you want to consider how long it will take your application to recover from being down.  While the app is down, jobs will be queuing up on the database or messaging system. You'll want to consider how long it will take for the new application to catch up with the queue jobs.  If the time it takes for you to recover from being down is too long you may want to look at temporarily increasing the number of instances of your application.  If your application supports this, it will allow you to catch up more quickly.  Then after things are caught up, you can scale down with cf scale to your usual level.
   
3. With a blue / green deployment you have the luxury of running two versions of the application at once, but your end-users are only using one at any given time. This is accomplished by manipulating the application mappings such that your users get directed to the version that you want them to see. With a background task, there is no such external control or switch. As soon as you start the second version of your application, it’ll begin working.
   
   One way around the lack of an external switch would be to build an internal one into your application. This could be something like an “admin” console (or REST endpoint) that allows you to enable or disable processing, flipping a record in a database or even sending a special message to control the application.   Exactly how it’s implemented will largely depend on the application and what fits best for it’s workflow, but in the end what you have is an internal switch to turn on or off processing for the application.
   
   This switch can then be used in conjunction with the first or second approaches listed above to give some additional control over the application and your deployment workflow.

As I mentioned in a previous post, I was lucky enough to be selected to speak at SpringOne2GX 2014. I co-presented at the event with Stuart Williams on our talk, Fastest Servlets in the West talk. 

As you might expect, the session talked about performance of Servlet based applications running in Apache Tomcat. We'll also talked about load testing, tuning the container and presented some tips for squeezing every bit of performance out of your app when it's running on Apache Tomcat.

I'm happy to say that our presentation went quite well that the recording is now up on InfoQ.  If you're interested in watching the video, here's the link.

Wordpress on CloudFoundry

If you're looking for a guide on how to run Wordpress on CloudFoundry, I've written a blog post for work which walks through the process.

Here's a quick overview of the article. It shows you how to...

• Obtain an account with the Cloud Foundry provider of your choice
• Install the cf client on your PC
• Setup persistent storage for your WordPress assets
• Create a MySQL Service
• Configure WordPress
• Deploy to Cloud Foundry
• Optionally scale the application to meet your performance and redundancy requirements

There's also short video that shows the process.

Debugging Java Applications on CloudFoundry

In many cases, it’s possible to run applications locally and when doing so it’s easy enough to debug them using Eclipse or your favorite Java IDE. In some cases, perhaps due to required third party services or maybe just because you want to see how it runs in a different environment, you want to run the application in CloudFoundry.

Because of the way that CloudFoundry deploys your applications and isolates them, it’s not possible to just connect to your application with the remote Java debugger. Instead you need to flip things around and instruct the application to connect to you.

Here are the instructions for setting up remote debugging when using bosh-lite or a CloudFoundry installation that can connect directly to your PC or laptop.

1. Open Eclipse and your project.
2. Right click on your project, go to Debug as and pick Debug Configurations.
3. Create a new Remote Java Application.
4. Make sure your project is selected, pick Standard (Socket Listen) from the Connection Type drop down and set a port (make sure this is open if you’re running a firewall).
5. Press the Debug button.

The debugger should now be running. If you switch to the Debug perspective, you should see your application listed in the Debug panel and it should say “Waiting for vm to connect at port ”.

The next step is to push your application to CloudFoundry and instruct it to connect to the debugger running on your local machine. Here are the steps to accomplish that (note I’m using the Spring Music application for this demo).

1. Edit your manifest.yml file. Set the instances count to 1. If you were to set this greater than one, you’d end up with multiple applications trying to connect to your debugger.
2. Also in manifest.yml, add the env section and setup a variable called JAVA_OPTS. In that, you’ll want to add the remote debugger configuration which looks like this -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>.
3. Save the manifest.yml file.
4. Run cf push.

When that completes, your application has started successfully and you see that the application is now connected to the debugger running in your IDE you are all set. From here, you can add breakpoints and interrogate the application just as you would if it were running locally.

If this does not work, try running cf logs --recent to investigate what happened when the application started. If there was a connection problem, you’ll see it in the output.

Firewalls & NAT

Previously I mentioned that these instructions would only work if the remote debugger could connect directly to your PC or laptop. If you’re like most people and behind a corporate or personal firewall and NAT, debugging from a public cloud offering like PWS is not going to work. It simply won’t be able to connect to your machine. Fortunately, there are a some approaches to working around this.

Port Forwarding

One options is port forwarding. If you’re behind a home router, this may be an option for you. Most home routers (enterprise ones do too, but your admin probably won’t want to do this) support the option for forward a request to the router on a designated port to a machine on the local network. As you can probably guess, if we set up port forwarding on our home router for the debug port, we can then redirect those requests to our PC or laptop.

The basic steps for this would look like this.

1. Pick an incoming port and configure the router to forward this to your PC or laptop. Your router may support forwarding this to a different port on your local machine. For simplicity, I’d suggest using the same port for both.
2. Start the debugger and listen on the port that the router is going to forward requests to on your machine.
3. Edit your manifest.yml file. Set JAVA_OPTS to -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>.
4. Run cf push.

If port forwarding works correctly, you should see the same results as above with the remote application connecting to your IDE’s debugger. If not, you may need to look at the application logs or even your router’s logs to troubleshoot further.

SSH Tunnel

Another option, if you have a publicly accessible server somewhere, is to tunnel the connection to your machine over SSH. With this option, you simply connect to your public machine with SSH and setup a reverse tunnel. The application will connect to the tunnel, which is listening on your public machine and it’s traffic will be forwarded to your local machine over SSH.

The basic steps for this would look like this.

1. Obtain a public server.
2. Install SSHD. Edit /etc/ssh/sshd_config, add or set GatewayPorts to yes. Restart SSHD.
3. On your local machine run ssh -f -N -T -R 0.0.0.0:<public-port>:127.0.0.1:<debugger-port> <user>@<public-server-ip> (Windows users can use cygwin or possibly Putty, although the command will be different). This will instruct SSH to connect to the remote host, setup a reverse tunnel and go into the background. The reverse tunnel will listen on your public server on the port you specify (i.e. public-port) and forward traffic to the debugger port on your local machine. You can use different port numbers, but it is easiest if you just use the same port.
4. Start the debugger and listen on the same port (i.e. debugger-port) that you used in the SSH command.
5. Edit your manifest.yml file. Set JAVA_OPTS to -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>.
6. Run cf push.

If the SSH reverse tunnel works correctly, you should see the same results as above. If not, you’ll need to check the application logs or public server logs for more details.

Concerns / Risks

When using an SSH tunnel in this manner, there are a few things to keep in mind.

1. By routing the debugger through an SSH tunnel, you’re going to add latency to the connection, especially if it goes over the public Internet. This latency will be noticed through the debugger and can slow down the startup of an application. To compensate for this, you may need to increase the application startup timeout with the -t argument to cf push.
2. If there is a firewall on your public server, it may may block the connections from your application to the public server. In other words, it will likely prevent the tunnel from listening on the . You’ll need to adjust your firewall rules to allow this.
3. You must set the GatewayPorts setting in your sshd_config file. This enables remote hosts (i.e. your application’s debugger) to connect to the <public-ip>:<public-port>. Without this, SSHD will only bind to the localhost.