Friday, November 12, 2021

Bootiful Podcast Appearance

I recently got the opportunity to talk with the amazing Josh Long on his podcast, Bootiful Podcast. It was a great time and a fun conversation. We started from the beginning and talked about all things containers, Docker, Buildpacks, Paketo, and of course, Spring & Native Images.


If you're new to buildpacks or looking to give them a test run, this is a great place to start. We covered a lot of the concepts and questions people frequently ask. It was also great to talk about some of the improvements that we've recently added into the Java buildpack. For example, we recently added support for UPX compression of native images and the ability to use Tilt with buildpacks for microservice development.


Please check it out! You won't be disappointed. Josh is always entertaining, and hopefully you'll come away having learned something new.

Friday, September 03, 2021

Recent Happenings

I have over the years done a lot of work with buildpacks. Both on Cloud Foundry and now with Cloud Native Buildpacks. This year I've been fortunate enough to start a job at VMware working on them full time, which I'm pretty excited about.

I am primarily working to develop the Java-related Paketo buildpacks, as well as contribute to the Buildpacks project & to also maintain the Java Cloud Foundry buildpack.

To support this effort, I'm going to try and start writing more about these efforts. What's new, what's cool and more about how to use buildpacks.

To kick this off, I have a post I wrote up for work: Container Images the Easy Way with Cloud Native Buildpacks. If you're new or wondering about buildpacks, this is a good place to start. It talks about how buildpacks can help you and gives some comparison to using Dockerfiles to build your container images, something many people are doing at the moment.

I also had the opportunity to be on Tanzu.tv and discuss similar topics, so if you'd rather watch a video than read, you can find some of similar info about buildpacks there.

Friday, April 24, 2020

PHP Cloud Native Buildpack Updates

It's been a little while since I've posted an update on the PHP Cloud Native Buildpacks. The good news is that lots of progress has been made. We've basically achieved feature parity with the old PHP buildpack and I believe the PHP CNB's should be working for most apps now!

If you're coming from the old PHP buildpack, there are some differences. This is basically a major version bump, so it was an opportunity to make a few breaking changes that we believe will generally improve the user experience. Check out the migration documentation for details on what's changed.

Here's a quick demo of what you can do.
  1. Start out by cloning the Symfony Demo app.
  2. Add a file to the root called buildpack.yml. In that file, add the following:

    ---
    php:
      webdirectory: public


    This will tell the buildpack the location of the files that we want it to serve.
  3. Create the folder .php.ini.d and in it put the file symfony.ini. Inside that file, add the line: variables_order = "EGPCS". This tells PHP that we want the $_ENV superglobal. I'm not a Symfony expert, but it seems to be required for Symfony to configure itself from actual environment variables, which we'll be doing.
These are the only changes we need to make for the demo app to work.

Next up, install prerequisites. You only need to do this once.
  1. If you don't have the pack cli install, go do that now.
  2. If you don't have docker installed, do that.
Now, from the root of our application directory we just run pack build -b paketo-buildpacks/php -e APP_ENV=prod symfony-5-demo to build an image.

Breaking it down:
  • pack build is the command to build an image from Cloud Native Buildpacks
  • the -b flag indicates which Cloud Native Buildpacks to use. In this case, we point it to the meta buildpack for PHP. Meta buildpacks are collections of Cloud Native Buildpacks.
  • the -e flag set an environment variable picked up by Symfony during build. It tells it that we're doing a production build.
  • the last bit is the image name we want to create
Lastly, we can run our image with docker run -e PORT=8080 -p 8080:8080 -e APP_ENV=prod symfony-5-demo.

Breaking it down:
  • docker run will run your image
  • -e sets the PORT environment variable which tells the buildpack what port it should have the app listen to.
  • -p tells Docker that we want to expose port 8080. This needs to match the value of PORT so that the port on which the app is listening matches the port on which docker is allowing traffic.
  • the second -e again tells Symfony that we're running in production mode
  • the last bit is the image name to run, this should match the image name we used with pack build.
At this point, you can go to http://localhost:8080/ in your browser and access the Symfony Demo App that's been built with Cloud Native Buildpacks and is running inside your Docker container.

Some final notes:
  • By default, it's going to run with PHP's built-in web server. That's not very "production", so we can add Nginx or Apache Web Server by adding the line webserver: httpd or webserver: nginx to buildpack.yml (add it under the php: block). That's it. No server configuration files required. The PHP Cloud Native Buildpacks handle it all for you.
  • The first time you run pack build, it will be slow. It needs to download resources like the build & run images, plus of course it needs to download PHP itself. After the first time, things will speed up dramatically. The images will exist locally and PHP downloads are smartly cached. With downloads cached, a full build takes about 10 seconds on my laptop.
That's it. Hope you all enjoy. Feel free to raise any feedback on Github.

Sunday, September 08, 2019

PHP Cloud Native Buildpacks Now in the Official Builder

In my previous post, I talked about how to use the PHP Cloud Native buildpacks. It was not super tricky but required some manual work to set up. This is because the PHP CNBs were not, at the time, part of an official builder.

What's a builder? It's basically an image containing a bunch of CNBs, all ready for your use. See this link for more details.

If you are to run `pack suggest-builders`, then you will see the list of official builders. At the time of writing, that is Heroku, Cloud Foundry (bionic) and Cloud Foundry (cflinuxfs3).

$ pack suggest-builders
Suggested builders:
    Heroku:            heroku/buildpacks:18            heroku-18 base image with buildpacks for Ruby, Java, Node.js, Python, Golang, & PHP
    Cloud Foundry:     cloudfoundry/cnb:bionic         Ubuntu bionic base image with buildpacks for Java, NodeJS and Golang
    Cloud Foundry:     cloudfoundry/cnb:cflinuxfs3     cflinuxfs3 base image with buildpacks for Java, NodeJS, Python, Golang, PHP, HTTPD and NGINX

Tip: Learn more about a specific builder with:
    pack inspect-builder [builder image]


Notice how PHP is now included in the Cloud Foundry cflinuxfs3 builder.

So now, instead of all the work I had you doing in my last post, you can just run `pack build --builder "cloudfoundry/cnb:cflinuxfs3"`. The builder will automatically select the PHP CNBs, because it sees you're trying to build a PHP app. The result will be an image you can run with `docker run -it -e PORT=8080 -p 8080:8080 image-name`, just like in my previous post.

That's it! If you'd like to understand the output a bit more, keep reading. Otherwise go and enjoy!



If you're curious about the output, the interesting new bit is the DETECTING phase. In the previous post, we manually specified just the PHP CNBs, so that's all you'd see in the output. Now, we see many CNBs running, but only the PHP CNBs will be selected and actually run at build time.

===> DETECTING
[detector] ======== Results ========
[detector] skip: org.cloudfoundry.archiveexpanding@1.0.0-RC03
[detector] pass: org.cloudfoundry.openjdk@1.0.0-RC03
[detector] skip: org.cloudfoundry.buildsystem@1.0.0-RC03
[detector] pass: org.cloudfoundry.jvmapplication@1.0.0-RC03
[detector] pass: org.cloudfoundry.tomcat@1.0.0-RC03
[detector] pass: org.cloudfoundry.springboot@1.0.0-RC03
[detector] pass: org.cloudfoundry.distzip@1.0.0-RC03
[detector] skip: org.cloudfoundry.procfile@1.0.0-RC03
[detector] skip: org.cloudfoundry.azureapplicationinsights@1.0.0-RC03
[detector] skip: org.cloudfoundry.debug@1.0.0-RC03
[detector] skip: org.cloudfoundry.googlestackdriver@1.0.0-RC03
[detector] skip: org.cloudfoundry.jdbc@1.0.0-RC03
[detector] skip: org.cloudfoundry.jmx@1.0.0-RC03
[detector] pass: org.cloudfoundry.springautoreconfiguration@1.0.0-RC03
[detector] Resolving plan... (try #1)
[detector] fail: org.cloudfoundry.jvmapplication@1.0.0-RC03 requires jvm-application
[detector] Resolving plan... (try #2)
[detector] fail: org.cloudfoundry.jvmapplication@1.0.0-RC03 requires openjdk-jre
[detector] Resolving plan... (try #3)
[detector] fail: org.cloudfoundry.jvmapplication@1.0.0-RC03 requires jvm-application
[detector] ======== Output: org.cloudfoundry.yarn@0.0.24 ========
[detector] no "yarn.lock" found at: /workspace/yarn.lock
[detector] ======== Results ========
[detector] pass: org.cloudfoundry.node-engine@0.0.46
[detector] fail: org.cloudfoundry.yarn@0.0.24
[detector] ======== Results ========
[detector] pass: org.cloudfoundry.node-engine@0.0.46
[detector] fail: org.cloudfoundry.npm@0.0.29
[detector] ======== Output: org.cloudfoundry.pipenv@0.0.14 ========
[detector] no Pipfile found
[detector] ======== Results ========
[detector] pass: org.cloudfoundry.python@0.0.20
[detector] skip: org.cloudfoundry.pipenv@0.0.14
[detector] pass: org.cloudfoundry.pip@0.0.20
[detector] Resolving plan... (try #1)
[detector] fail: org.cloudfoundry.pip@0.0.20 requires requirements
[detector] ======== Results ========
[detector] fail: org.cloudfoundry.conda@0.0.13
[detector] ======== Output: org.cloudfoundry.dotnet-core-conf@0.0.20 ========
[detector] *.runtimeconfig.json file not found
[detector] ======== Results ========
[detector] fail: org.cloudfoundry.dotnet-core-conf@0.0.20
[detector] ======== Output: org.cloudfoundry.go-mod@0.0.18 ========
[detector] no "go.mod" found at: /workspace/go.mod
[detector] ======== Results ========
[detector] pass: org.cloudfoundry.go-compiler@0.0.20
[detector] fail: org.cloudfoundry.go-mod@0.0.18
[detector] ======== Output: org.cloudfoundry.dep@0.0.17 ========
[detector] failed detection: no Gopkg.toml found at root level
[detector] ======== Results ========
[detector] pass: org.cloudfoundry.go-compiler@0.0.20
[detector] fail: org.cloudfoundry.dep@0.0.17
[detector] ======== Output: org.cloudfoundry.php-composer@0.0.16 ========
[detector] no "composer.json" found in the following locations: [/workspace/composer.json /workspace/htdocs/composer.json]
[detector] ======== Results ========
[detector] pass: org.cloudfoundry.php-dist@0.0.28
[detector] skip: org.cloudfoundry.php-composer@0.0.16
[detector] pass: org.cloudfoundry.httpd@0.0.18
[detector] pass: org.cloudfoundry.php-web@0.0.18
[detector] Resolving plan... (try #1)
[detector] skip: org.cloudfoundry.httpd@0.0.18 provides unused httpd
[detector] Success! (2)


See how they all fail, until you get down to the PHP CNBs. Then you see Success!.

Then just like the last time, we see the PHP CNBs running during the BUILD phase.

===> BUILDING
[builder]
[builder] PHP Buildpack 0.0.28
[builder]   PHP 7.2.22: Contributing to layer
[builder]     Downloading from https://buildpacks.cloudfoundry.org/dependencies/php/php7-7.2.22-linux-x64-cflinuxfs3-c9315fce.tgz
[builder]     Verifying checksum
[builder]        Expanding to /layers/org.cloudfoundry.php-dist/php-binary
[builder]     Writing PATH to shared
[builder]     Writing MIBDIRS to shared
[builder]     Writing PHP_HOME to shared
[builder]     Writing PHP_EXTENSION_DIR to shared
[builder]     Writing PHP_API to shared
[builder]
[builder] PHP Web Buildpack 0.0.18
[builder]   Configuring PHP Script
[builder]   PHP Web 173b6da1f1b7b7a59af0c2ebcfe1c4ea4b183063a90b92f1f2ae6d8a9ebb0bfd: Contributing to layer
[builder]     Writing PHPRC to shared
[builder]     Writing PHP_INI_SCAN_DIR to shared
[builder] WARNING: `app.php` start script not found. App will not start unless you specify a custom start command.
[builder]   Process types:
[builder]     task: php /workspace/app.php
[builder]     web:  php /workspace/app.php


That's all there is too it.

Thursday, July 04, 2019

PHP Cloud Native Buildpacks

At work, I've been helping to rewrite the PHP buildpack as a set of Cloud Native Buildpacks. The PHP CNBs are coming together, current quality is alpha, but I think they're ready enough for people to try them out and report how they work for you. This post has instructions and a demo to use the PHP CNBs.

But first, a slight digression.

A little about the architecture of the PHP CNBs. The previous PHP buildpack has been decomposed into a set of five PHP CNBs, two of which are optional. There are php-cnb, php-composer-cnb, httpd-cnb, nginx-cnb and php-web-cnb.

Here's a rough description of each:
  • php-cnb provides PHP binaries, that's it.
  • php-composer-cnb provides all the functionality related to Composer. It installs and runs Composer.
  • httpd-cnb provides Apache Web Server. It is optional.
  • nginx-cnb provides Nginx Web Server. It is optional.
  • php-web-cnb ties everything together. It generates the configuration for PHP, PHP-FPM, HTTPD and Nginx. It also contains the logic to generate start commands for various types of PHP apps. It can run PHP cli scripts, PHP's bundled web server, PHP-FPM plus HTTPD and PHP-FPM plus Nginx.
What's also super cool about these CNBs is that some of them like httpd-cnb and nginx-cnb can work all on their own. Want to stand up a proxy or a static site, httpd-cnb or nginx-cnb can be given an httpd.conf or nginx.conf file and they'll run their respective server with that config for you. Similarly, you can mix and match CNBs. Don't want to use the PHP binaries provided by php-cnb, you could substitute another CNB that provides PHP. Pick the parts you like, replace the ones you don't with other things.

On to the show.

If you want to get started there's a little setup that you need to perform.

First,  `git clone` these repos and optionally check out a release branch.
  • https://github.com/cloudfoundry/php-cnb
  • https://github.com/cloudfoundry/php-composer-cnb
  • https://github.com/cloudfoundry/httpd-cnb
  • https://github.com/cloudfoundry/nginx-cnb
  • https://github.com/cloudfoundry/php-web-cnb

Second, install the latest version of the pack cli. That is v0.2.1 at time of writing.

Third, install Docker if you don't have it already. Make sure it's running too.

Fourth, package up each buildpack that you'd like to use. In each folder that you cloned, you can run `./scripts/package.sh` and it will build the CNB (this requires Golang to be installed). Note the path at which the packaged CNB can be found. You'll need to pass this to the pack cli so it can find the buildpacks. To do that, run `export BUILDPACKS='--buildpack /path/to/buildpack1 --buildpack /path/to/buildpack2 ...'` and paste in the path to each buildpack you packaged. Order is important, use the order listed in the bullet list above.

If you'd like to build them all at once, you can run the following command from the directory where you cloned all of the repos:

for buildpack in ./php-cnb ./php-composer-cnb ./httpd-cnb ./nginx-cnb ./php-web-cnb; do pushd "$buildpack" && ./scripts/package.sh && popd; done | grep "Creating package in" | awk '{printf "--buildpack %s ", $5}'

This will run the package script for each CNB & then print out a list of the locations for each package. Simply copy the output, then run `export BUILDPACKS='paste output from command here'`. We set this as a convenience to make using the pack cli easier and the commands shorter.

At this point, you're all set to build some images. To do this, you run `pack build $BUILDPACKS -p /path/to/php/files`.  This will build an image using the buildpacks that we've specified in our environment variable and it will use the PHP code that you've pointed to with the `-p` argument (you can skip `-p` if the files are in the current directory.

As `pack build` runs, you'll see each build pack run. If there are any errors the buildpack will fail and tell you what happened. If it succeeds, you'll end up with a docker image that you can run using the command `docker run`. For example, if you have a web app, you can `docker run -it -e PORT=8080 -p 8080:8080 `.  The PORT env variable tells the web server which port it should listen to. That should match with the port that you expose using the `-p` flag.

Time for a full example.

Let's say you want to run PHP MyAdmin. The following is a demo of how you could do that. For simplicity sake, it spins up a Percona DB as well. That allows you to have something to connect to within PHP MyAdmin. If you already have a MySQL DB, you can skip that part and point `htdocs/config.inc.php` to your existing server (you could also skip the docker network bits, that just makes it easy to connect to the deployed Percona DB).

Download and run the Gist below. This will set everything up for you.

Here are the highlights:
  • Line #10 runs Percona
  • Lines #15 - #21 download PHP MyAdmin & configure `htdocs/config.inc.php`. If you want to adjust PHP MyAdmin's config, you can do so at this point.
  • Lines #24 - #36 adds a php.ini snippet that enables the PHP extensions needed by PHP MyAdmin
  • Line #45 runs `pack build`
  • Line #48 runs the image that we build with Docker.
At this point, you can go to http://localhost:8080 in your browser and you should be able to access PHP MyAdmin. Enter `root` and `hacKm3` as the credentials and you can access the database. If you edited the config to point to your own MySQL server, use the credentials for that server instead.

Last notes:
  • If you want to adjust the PHP MyAdmin config. Run `docker stop php-myadmin`. Edit `htdocs/config.inc.php` then run lines #45 & #48 again.
  • If you want to clean up & remove everything run `docker stop php-myadmin test-db` followed by `docker system prune --volumes`. The latter will clean up a bunch of things for you, be careful when running that if you are running other things with Docker.
Please provide any feedback about the PHP CNBs to the respective CNB Github page. Open an issue with your comments/questions. Thanks & hope you enjoy!

Sunday, February 17, 2019

Cloud Native Buildpacks

In the past, I've worked with buildpacks through my time using Cloud Foundry. Cloud Foundry has first class support for buildpacks, which allows you to push code and let the buildpack handle the messy parts of actually running your code. Things like installing a language runtime, installing servers, etc...

Recently the buildpacks world has expanded with the CNCF's acceptance of the Cloud Native Buildpacks project into the CNCF sandbox (sometimes called v3 buildpacks). In addition to an excellent and easily readable spec, this work brings us the `pack` CLI tool, which allows you to run Cloud Native Buildpacks on your local PC and easily deploy the output, which is an OCI image, to Docker or anywhere else you can run an OCI image.

In this post, I'm going to walk through some basics and show you how to get started with `pack`, build some image and run them.

Getting Started

To get started you need to install Docker. The Community Edition works fine. Follow the previous link to get that installed, if you don't have it already.

Then install the `pack` CLI.  You can download `pack` from its Github project here. At the time of writing, I'm using the 0.0.9 release. To download the tar or zip, extract the `pack` binary and put it somewhere on your PATH. On Mac/Linux, `/usr/local/bin` is a good place. Once installed, you should be able to run `pack version` and see `v0.0.9 (git sha: a1a1a0eef63bd09136ab76663bdbc3b0ab3a4931)`.

Hello World

To get a basic app going, we need to do one more thing first. Obtain some buildpacks to use. So run `git clone https://github.com/buildpack/samples`, which is a repo that has a couple very basic sample buildpacks.

Sidebar. At the time of writing, the sample buildpack we're using has an error with it's metadata. You may not need to do this in the future. Edit `samples/hello-world-buildpack/buildpack.toml` and put in the following:

[buildpack]
id = "io.buildpacks.samples.buildpack.hello-world"
version = "0.0.1"
name = "Hello World Buildpack"

[[stacks]]
id = "io.buildpacks.stacks.bionic"

Now that we have buildpacks, we need an app to run. We'll create that now. Run `mkdir hello-world` and then `cd hello-world`. In that folder create `app.sh` and put the following in that file.

#!/bin/bash

while [ 1 -eq 1 ]; do
  echo "Hello World!"
  sleep 5
done

Last step, run `chmod 755 app.sh` to make it executable.

At this point, we now have a buildpack to use and our application code. It's time to run `pack` and make an image.

From our application directory run `pack build --buildpack $(cd ..; pwd)/samples/hello-world-buildpack/ hello-world-app` or replace `$(cd ..; pwd)/samples/hello-world-buildpack/` with the full path to the sample repo you cloned above. This will create an image called `hello-world-app` using the `hello-world-buildpack`, which does nothing (it's a no-op). The output should look something like this.

$ pack build --buildpack $(cd ..; pwd)/samples/hello-world-buildpack/ hello-world-app
Defaulting app directory to current working directory /Users/dmikusa/Downloads/hello-world (use --path to override)
Using default builder image packs/samples:v3alpha2
Pulling builder image packs/samples:v3alpha2 (use --no-pull flag to skip this step)
Selected run image packs/run:v3alpha2 from stack io.buildpacks.stacks.bionic
Pulling run image packs/run:v3alpha2 (use --no-pull flag to skip this step)
Using cache volume pack-cache-153f385b25c48f5d30ee0544d75bee63
===> DETECTING
Using manually-provided group
[detector] 2019/02/17 21:09:40 Trying group of 1...
[detector] 2019/02/17 21:09:41 ======== Results ========
[detector] 2019/02/17 21:09:41 Hello World Buildpack: pass
===> ANALYZING
Reading information from previous image for possible re-use
[analyzer] 2019/02/17 21:09:42 WARNING: image 'hello-world-app' not found or requires authentication to access
[analyzer] 2019/02/17 21:09:42 removing cached layers for buildpack 'config' not in group
===> BUILDING
[builder] ---> Hello World buildpack
[builder]      env_dir: /platform/env
[builder]      plan_path: /tmp/plan.333599924/io.buildpacks.samples.buildpack.hello-world/plan.toml
[builder]      layers_dir: /workspace/io.buildpacks.samples.buildpack.hello-world
[builder] ---> Done
===> EXPORTING
[exporter] 2019/02/17 21:09:48 adding layer 'app' with diffID 'sha256:361cdaf2662ea41f08da0204a4c0393beb629cff85df2b3d650ed7423dc188f2'
[exporter] 2019/02/17 21:09:48 adding layer 'config' with diffID 'sha256:ab046f0bf0b24db6ae8f59e437cc570925e451cb60ae714fcb125ab4095dd9bb'
[exporter] 2019/02/17 21:09:49 adding layer 'launcher' with diffID 'sha256:d77dc7ed6207d6bb9c389aa5f087ea7fffea9238e2de84b03f8b3c1152e1e58f'
[exporter] 2019/02/17 21:09:49 setting metadata label 'io.buildpacks.lifecycle.metadata'
[exporter] 2019/02/17 21:09:49 setting env var 'PACK_LAYERS_DIR=/workspace'
[exporter] 2019/02/17 21:09:49 setting env var 'PACK_APP_DIR=/workspace/app'
[exporter] 2019/02/17 21:09:49 setting entrypoint '/lifecycle/launcher'
[exporter] 2019/02/17 21:09:49 setting empty cmd
[exporter] 2019/02/17 21:09:49 writing image
[exporter] 2019/02/17 21:09:49
[exporter] *** Image: hello-world-app@9b265f861002fa1018d48577fb2f78c4e32b58f72f81c8ef9f6692d2040b4d60
Successfully built image hello-world-app

The interesting bits for now are DETECTING, where the buildpack's detection script runs. This buildpack doesn't do anything but we can see it's marked as "pass" which means the buildpack's build script will get a chance to run. Down below you can see that happening under BUILDING. This again does nothing, but echo a few directories where files reside during build. Legit buildpacks would use detect to determine when they should/shouldn't run and build to install things like runtimes, servers and all the stuff necessary to run your apps.

 The output from above is a image that you can run. If you execute `docker images`, you'll see `hello-world-app` listed.

$ docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
hello-world-app                            latest              9b265f861002        6 minutes ago       164MB

You can then run it with `docker run -it --name=hello hello-world-app bash app.sh`. The app will run forever printing "Hello World!". Run `docker stop hello` to stop the app.

Hello World++

To spice things up just a little bit and show what it's like to deploy changes to our app, let's edit our `app.sh` script. Set it to this.

#!/bin/bash

while [ 1 -eq 1 ]; do
  if [ "$NAME" == "" ]; then
    echo "Hello World!"
  else
    echo "Hello $NAME!"
  fi
  sleep 5
done

This will allow us to provide a name to print. Run `pack build --buildpack $(cd ..; pwd)/samples/hello-world-buildpack/ hello-world-app` again. This will create a new image with our updated app.

Side note, if you run `docker images` you'll see that the old image is no longer used and can be removed at your leisure.

$ docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
hello-world-app                            latest              793b5e60b5ad        9 seconds ago       164MB
                                                   9b265f861002        20 minutes ago      164MB

To run the updated app image, you can use the same command `docker run -it --name=hello hello-world-app bash app.sh` and you'll see the same output. However, if you run `docker run -it -e NAME=Daniel --name=hello hello-world-app bash app.sh` you'll see our enhancement.

$ docker run -it -e NAME=Daniel --name=hello hello-world-app bash app.sh
Hello Daniel!
... 

We use Docker's ability to set environment variables to inject some data into our application. More importantly though, you can see that pushing updates and changes is the same process as you used before which makes integrating into build systems and CI/CD systems simple.

Summary

I hope you find getting started is easy. Once you get Docker & pack installed it's one command to stamp out an image using a buildpack and our application. Right now, that buildpack isn't doing anything, so it's not the best demonstration of why you'd want use buildpacks or the full power of them, but I hope this is enough to get you thinking about how this can integrate into your build flows, maybe your CI/CD system and how it can work for you.

My next post will be more practical. It'll dig into some actual buildpacks and show how you can use them to make images for actual applications, and I hope this will better showcase why you would want to use buildpacks.

Friday, May 18, 2018

WordPress Running on Cloud Foundry

I'd previously written an article on deploying WordPress on Cloud Foundry.  The process was a little clunky and has since broken, because of updates & changes to Cloud Foundry.  To remedy this, I wrote a new post which was published today on the Cloud Foundry Foundation Blog.

Here's the link -> https://www.cloudfoundry.org/blog/install-scale-wordpress-cloud-foundry-2018/


Sunday, January 08, 2017

Freenas: Migrating from VirtualBox to Bhyve

I, like many people, use the VirtualBox template jail on Freenas.  I've been using this for about a year to run Crashplan.  It's generally worked good, but the last couple of 9.10 maintenance updates have caused some problems (ex: The virtual machine 'xxxx' has terminated unexpectedly during startup with exit code 1).  See here for more on how 9.10.2 broke the VirtualBox template.

From what I've been able to read on the Freenas forums, this isn't going to get fixed and you have two options.  Don't upgrade and keep using VirtualBox or upgrade and use Bhyve.  Since I only have one VM and it's pretty simple, I opted for the latter solution.  This post is going to cover the steps that I had to do to convert from using VirtualBox to using Bhyve and iohyve without reinstalling the VM OS.

Convert the Image

The first step is to convert your old VirtualBox image to a raw disk image.  There's no direct way to import the VDI or directly use it with iohyve, but you can convert the VDI to a raw disk image and we can use that.  To convert the disk, do the following.
  1. SSH to your VirtualBox jail.
  2. Install the qemu-devel pkg with the command `pkg install qemu-devel`.
  3. Run `qemu-img convert -f vdi -O raw /path/to/virtualbox/vm.vdi /path/to/raw/image.img`
That's it.  You now have a raw image file of your old VirtualBox VM.

Setup Iohyve

There is some setup that's needed before you can use iohyve on your Freenas.  This only needs to be done once, so if you've used iohyve before on your Freenas you can skip these steps.
  1. Initialize iohyve by running `iohyve setup pool=main kmod=1 net=igb0`.  Note that you need to replace `main` with the name of your ZFS pool and `igb0` with the name of your network device.
  2. Run `ln -s /mnt/iohyve /iohyve`.  I don't know exactly why you need this symlink, but just make sure the symlink exists so that you comply with the requirement from the iohyve README.
Create a New VM

The next step is to create our new VM using iohyve.  If you're not familiar with it, iohyve is a convenient wrapper script around Bhyve which makes managing your VMs easier.  In addition to it being convenient, it's also installed out of the box on Freenas 9.10.2, which makes it a logical choice to use.

To create a new VM with iohyve, run the following commands:
  1. Create the VM, run `iohyve create crashplan 8G`.  The `8G` should be the maximum size of your disk.
  2. Configure the VM, run `iohyve set crashplan loader=grub-bhyve os=ubuntu ram=512M cpu=1`.  We set `loader=grub-bhyve` because this is a Linux VM being booted by Grub.  Set the OS to whatever you're using, my VM is using Ubuntu (if you're using LVM with Debian or Ubuntu, use `d8lvm` instead).  Then set the RAM and CPU limits, I picked 512M of RAM and one VCPU.
  3. You can now run `iohyve getall crashplan` to confirm all the settings of your VM.
Your VM is now created.  At this point, you would normally proceed to install an OS, however we've done that already in VirtualBox and so we don't want to do it again.

Import the Raw Image

The next step is to import the raw image that we exported previously.  This is simply done with `dd`.  When using iohyve, you'll see that you have a ZFS dataset for each VM.  For example, my ZFS pool is named `main` and my VM is named `crashplan`, so I have a disk at `main/iohyve/crashplan`.  In addition to that, there is a dataset for the disk.  In my case there's only one disk, so it's `disk0`.  To import the disk, we just need to `dd` the raw disk image to `disk0`.  That can be done with the following command.
dd if=/mnt/main/jails/virtualbox_jail/home/vbox/VirtualBox\ VMs/CrashPl
anBackup/CrashPlanBackup.img of=/dev/zvol/main/iohyve/crashplan/disk0
Make sure you replace `if=...` with the path to your raw image file and replace `of=...` to the path to your iohyve VM disk.  Once `dd` finishes, the VM should be ready to run.

Run Your VM

To start your VM run `iohyve start crashplan`.  Then run `iohyve console crashplan` to connect the console to the VM.  The console will display the output from the VM as it starts up.  When done you'll even be able to login to the VM through the console, but you'll probably want to connect via SSH instead.

Start VM at Boot

At this point your VM should be up and running, however if you were to restart your Freenas the VM would not be running on restart.  Fortunately it's pretty easy to make the VM start at boot.  The following instructions will make your VM start at boot.

  • In the Freenas UI, go to System -> Tunables.
  • Add a tunable.  Variable is `iohyve_enable`, value is `YES`, type is `rc.conf`. Make sure enabled is checked.
  • Add a tunable.  Variable is `iohyve_flags`, value is `kmod=1 net=igb0` (where `igb0` is the name of your network device), type is `rc.conf`.  Also make sure enabled is checked.
These settings will enable iohyve and get it started.  From there, we need to tell iohyve to start our VM.  That can be done with the following command.

iohyve set crashplan boot=1
As with all the commands I've listed, replace `crashplan` with the name of your VM.

Clean Up

While not strictly necessary, you can now clean up the old VirtualBox template jail.  Simply delete the jail a that should be it.

Summary

With any luck you should now have your VirtualBox VM functional and running on Bhyve instead.  More work may be necessary if you've got a more complicated VirtualBox VM.  For example, if you have multiple disks, multiple NICs, a GUI or were using VirtualBox specific features like shared folders.  In that case, you may want to stay on VirtualBox until official support for Bhyve is ready in Freenas 10.

Wednesday, July 15, 2015

2015 MacAdmins Conference at Penn State


The 2015 MacAdmins Conference at Penn State was last week.  It was a fantastic conference and I had the good fortune to be able to both attend and speak at the event.  If you were in one of my sessions and are looking for slides or code samples see the info below.  I believe video should be available shortly and as soon as it is I'll update this post with the relevant links.

Basic App Development with Swift



The Sys Admin's Guide to Python

If you missed the conference this year, put next year's date on your calendar now.  The 2016 MacAdmins Conference at Penn State is scheduled for June 27-30, 2016.  See you then!

Friday, March 13, 2015

A recent encounter with a customer resulted in a couple good questions regarding the workflow that one would use to deploy apps to Cloud Foundry in order to try for a 100% up-time.  Based on that, I would share the questions and answers here.

Question #1 - How do you push updates to your application without downtime?

Currently when you push, or restart for that matter, an application running on Cloud Foundry, the change is applied in a series of steps that go roughly like this.
  • New app bits are uploaded
  • The current version of the app is stopped
  • Staging for the new app occurs (i.e. the build pack runs)
  • The new app is started
What’s important to understand about this process is that when the app is stopped all instances of your application are stopped. Thus there will be some small amount of downtime, while your new app stages and is started.
The typical suggestion for working around this is to do what are called blue / green deployments, which work by running both the current and new version of application at the same time. Since both apps are running, you can switch to the new app in a controlled fashion by simply manipulating the routes, something that happens instantly and does not require downtime.

Question #2 - How do you push updates to your application if it’s not taking web requests but still needs to maintain high availability?

If you have an application that is not taking HTTP requests, like a background worker, the typical blue / green deployment scenario may or may not work for you. If you’re running a background worker and need to keep it highly available, here are some things to consider.
  1. If you have a background worker style task, it may be as simple as starting a second instance of the application that is running the new code and then stopping the old instance. The key to making this work on CF is to use different application names (both bound to the same service, if the worker is using services). This will enable both to run at the same time and allow you to shutdown the old worker instance when you’re satisfied that the new code is working properly.

    Before doing this though, please keep in mind that there will be a window of time where there are two versions of your application running. This means that before adopting this approach, you should consider what will happen if there are two versions of your worker running at the same time. Will they play nice together or will they compete for the work, and will they both be compatible (i.e. did the database schema change, did message formats change, etc..).
  2. Another solution to this problem is to simply ignore it. Depending on the architecture of your application you may be able to just push your new changes and ignore the fact that the application will be down for a small window of time. This will generally be the case for background worker tasks that are simply pulling jobs from a queue or database. This flexibility comes from the fact that by their nature the database or queue will hold the jobs while your application is not running. Given this, all you need to do is push the new change and wait for the app to catch up on it’s work.

    Before going with this approach, there are some important things that you should consider. First, you should have a good understanding of how long it will take for your new version of the application to stage, start up and begin doing work. This is critical and leads us to the second point. You need to have a good estimate as to how much work will be queued up while the application is restarting and if your service is capable of storing that much data. This is key to not losing any work while the new version of your application is starting up.

    Lastly, you want to consider how long it will take your application to recover from being down.  While the app is down, jobs will be queuing up on the database or messaging system. You'll want to consider how long it will take for the new application to catch up with the queue jobs.  If the time it takes for you to recover from being down is too long you may want to look at temporarily increasing the number of instances of your application.  If your application supports this, it will allow you to catch up more quickly.  Then after things are caught up, you can scale down with cf scale to your usual level.
  3. With a blue / green deployment you have the luxury of running two versions of the application at once, but your end-users are only using one at any given time. This is accomplished by manipulating the application mappings such that your users get directed to the version that you want them to see. With a background task, there is no such external control or switch. As soon as you start the second version of your application, it’ll begin working.

    One way around the lack of an external switch would be to build an internal one into your application. This could be something like an “admin” console (or REST endpoint) that allows you to enable or disable processing, flipping a record in a database or even sending a special message to control the application.   Exactly how it’s implemented will largely depend on the application and what fits best for it’s workflow, but in the end what you have is an internal switch to turn on or off processing for the application.

    This switch can then be used in conjunction with the first or second approaches listed above to give some additional control over the application and your deployment workflow.

Tuesday, January 27, 2015

As I mentioned in a previous post, I was lucky enough to be selected to speak at SpringOne2GX 2014. I co-presented at the event with Stuart Williams on our talk, Fastest Servlets in the West talk. 

As you might expect, the session talked about performance of Servlet based applications running in Apache Tomcat. We'll also talked about load testing, tuning the container and presented some tips for squeezing every bit of performance out of your app when it's running on Apache Tomcat.

I'm happy to say that our presentation went quite well that the recording is now up on InfoQ.  If you're interested in watching the video, here's the link.


Thursday, December 18, 2014

Wordpress on CloudFoundry

If you're looking for a guide on how to run Wordpress on CloudFoundry, I've written a blog post for work which walks through the process.

Here's a quick overview of the article. It shows you how to...

  • Obtain an account with the Cloud Foundry provider of your choice
  • Install the cf client on your PC
  • Setup persistent storage for your WordPress assets
  • Create a MySQL Service
  • Configure WordPress
  • Deploy to Cloud Foundry
  • Optionally scale the application to meet your performance and redundancy requirements
There's also short video that shows the process.

Friday, August 22, 2014

Debugging Java Applications on CloudFoundry

In many cases, it’s possible to run applications locally and when doing so it’s easy enough to debug them using Eclipse or your favorite Java IDE. In some cases, perhaps due to required third party services or maybe just because you want to see how it runs in a different environment, you want to run the application in CloudFoundry.

Because of the way that CloudFoundry deploys your applications and isolates them, it’s not possible to just connect to your application with the remote Java debugger. Instead you need to flip things around and instruct the application to connect to you.

Here are the instructions for setting up remote debugging when using bosh-lite or a CloudFoundry installation that can connect directly to your PC or laptop.

  1. Open Eclipse and your project.
  2. Right click on your project, go to Debug as and pick Debug Configurations.
  3. Create a new Remote Java Application.
  4. Make sure your project is selected, pick Standard (Socket Listen) from the Connection Type drop down and set a port (make sure this is open if you’re running a firewall).
  5. Press the Debug button.

The debugger should now be running. If you switch to the Debug perspective, you should see your application listed in the Debug panel and it should say “Waiting for vm to connect at port ”.

The next step is to push your application to CloudFoundry and instruct it to connect to the debugger running on your local machine. Here are the steps to accomplish that (note I’m using the Spring Music application for this demo).

  1. Edit your manifest.yml file. Set the instances count to 1. If you were to set this greater than one, you’d end up with multiple applications trying to connect to your debugger.
  2. Also in manifest.yml, add the env section and setup a variable called JAVA_OPTS. In that, you’ll want to add the remote debugger configuration which looks like this -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>.
  3. Save the manifest.yml file.
  4. Run cf push.

When that completes, your application has started successfully and you see that the application is now connected to the debugger running in your IDE you are all set. From here, you can add breakpoints and interrogate the application just as you would if it were running locally.

If this does not work, try running cf logs --recent to investigate what happened when the application started. If there was a connection problem, you’ll see it in the output.

Firewalls & NAT


Previously I mentioned that these instructions would only work if the remote debugger could connect directly to your PC or laptop. If you’re like most people and behind a corporate or personal firewall and NAT, debugging from a public cloud offering like PWS is not going to work. It simply won’t be able to connect to your machine. Fortunately, there are a some approaches to working around this.

Port Forwarding


One options is port forwarding. If you’re behind a home router, this may be an option for you. Most home routers (enterprise ones do too, but your admin probably won’t want to do this) support the option for forward a request to the router on a designated port to a machine on the local network. As you can probably guess, if we set up port forwarding on our home router for the debug port, we can then redirect those requests to our PC or laptop.

The basic steps for this would look like this.

  1. Pick an incoming port and configure the router to forward this to your PC or laptop. Your router may support forwarding this to a different port on your local machine. For simplicity, I’d suggest using the same port for both.
  2. Start the debugger and listen on the port that the router is going to forward requests to on your machine.
  3. Edit your manifest.yml file. Set JAVA_OPTS to -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>.
  4. Run cf push.

If port forwarding works correctly, you should see the same results as above with the remote application connecting to your IDE’s debugger. If not, you may need to look at the application logs or even your router’s logs to troubleshoot further.

SSH Tunnel


Another option, if you have a publicly accessible server somewhere, is to tunnel the connection to your machine over SSH. With this option, you simply connect to your public machine with SSH and setup a reverse tunnel. The application will connect to the tunnel, which is listening on your public machine and it’s traffic will be forwarded to your local machine over SSH.

The basic steps for this would look like this.

  1. Obtain a public server.
  2. Install SSHD. Edit /etc/ssh/sshd_config, add or set GatewayPorts to yes. Restart SSHD.
  3. On your local machine run ssh -f -N -T -R 0.0.0.0:<public-port>:127.0.0.1:<debugger-port> <user>@<public-server-ip> (Windows users can use cygwin or possibly Putty, although the command will be different). This will instruct SSH to connect to the remote host, setup a reverse tunnel and go into the background. The reverse tunnel will listen on your public server on the port you specify (i.e. public-port) and forward traffic to the debugger port on your local machine. You can use different port numbers, but it is easiest if you just use the same port.
  4. Start the debugger and listen on the same port (i.e. debugger-port) that you used in the SSH command.
  5. Edit your manifest.yml file. Set JAVA_OPTS to -agentlib:jdwp=transport=dt_socket,address=<your-ip>:<your-port>.
  6. Run cf push.

If the SSH reverse tunnel works correctly, you should see the same results as above. If not, you’ll need to check the application logs or public server logs for more details.

Concerns / Risks


When using an SSH tunnel in this manner, there are a few things to keep in mind.

  1. By routing the debugger through an SSH tunnel, you’re going to add latency to the connection, especially if it goes over the public Internet. This latency will be noticed through the debugger and can slow down the startup of an application. To compensate for this, you may need to increase the application startup timeout with the -t argument to cf push.
  2. If there is a firewall on your public server, it may may block the connections from your application to the public server. In other words, it will likely prevent the tunnel from listening on the . You’ll need to adjust your firewall rules to allow this.
  3. You must set the GatewayPorts setting in your sshd_config file. This enables remote hosts (i.e. your application’s debugger) to connect to the <public-ip>:<public-port>. Without this, SSHD will only bind to the localhost.