Announcing Faster Container Builds Powered by Kubernetes

October 5, 2016 · By Joseph Schorr

CoreOS Tectonic enables everyone to operate their server systems like Google does; we call this GIFEE (Google’s Infrastructure For Everyone). Built on Kubernetes, the production-grade container orchestration system, Tectonic enables you to manage complex containerized application infrastructure from application source code to live load balancer.

A key piece of Tectonic is Quay: the container registry by CoreOS. It is the cornerstone for building, analyzing and distributing container images and is used by thousands of companies, large and small, as part of their containerized infrastructure. Quay is even used to build and deploy itself, demonstrating the power of the engineering tools we develop at CoreOS.

Quay's build system transforms Dockerfiles and source code into container images automatically. It is an important piece of infrastructure for our customers (and ourselves) and we are happy to announce that, starting today, it has gotten a major upgrade: All builds on Quay are now executed on a Tectonic cluster, running on Packet bare metal servers.

Put simply: with this change builds on Quay just got a lot better for everyone, as we have taken advantage of the speed and automatic scaling of Kubernetes.

Quay builder KVM diagram
Each build in Quay is assigned to a builder, which executes within a KVM instance users should see significant and immediate improvements:

  • Builds now have a much faster startup time, roughly an 80% improvement. Instead of waiting for an AWS EC2 instance to boot (which can take upwards of three minutes), build start times are closer to 15 seconds

  • Leveraging Kubernetes results in much more efficient use of compute resources, which allows us to offer more resources for each individual build

Quay EC2 Timing Diagram
Container builds with Kubernetes start 80% more quickly than the previous system

For Quay Enterprise customers (support coming soon!), there will be additional benefits:

  • The build cluster can automatically scale based on demand. No more manual scaling necessary.

  • By leveraging Kubernetes, no other setup is needed: Simply point Quay Enterprise to a valid Kubernetes cluster and it just works as expected.

  • Quay Enterprise customers will get the same hardened security associated with builds, with each build running in its own micro-virtual machine, fully isolated from the machine conducting the build.

With builds powered on a baremetal provider like Packet, Quay users experience faster build times without needing to do anything.

“As container based stacks become more commonplace and complex, services like Quay play an increasingly critical role in everything from developer velocity to production stability and security," said Zachary Smith, CEO, Packet. “Like most cloud native applications, Quay runs best on 'Google style' infrastructure: dedicated, bare metal servers with advanced hardware security features like Trusted Computing and a lightning fast network. We're thrilled to have Quay running on Packet, and can't wait to see what users all over the world do with faster, more secure builds."

How it works

Container builds typically begin on Quay via a webhook called from a code repository, such as GitHub. The webhook call contains the commit to build, along with metadata such as the commit message and author, which will be displayed in the Quay UI.

Once a build has been triggered, it is immediately placed into a queue to be picked up for processing:

Build Triggers Diagram
New code commits can automatically trigger a new container build

On each instance in the Quay cluster is a build manager, which monitors the build queue and periodically checks for new builds that have been triggered. Whenever possible, a build is retrieved by the build manager from the queue and scheduled onto the builders using the execution API. Previously, this scheduling was done via the Amazon EC2 API. With Kubernetes, the Quay build managers instead make use of the Kubernetes API to schedule a Job for each build. Each Job consists of metadata for the job (such as a one-time use token for authorization), as well a single Pod executing a container that runs a virtual machine containing the Quay build agent.

Build Queue Diagram
One KVM instance is run per build

It’s important that each build run in a virtual machine due to the requirements of docker build. Docker builds must run as root, and while solutions do exist to enforce security for root containers (such as user namespacing), they also introduce incompatibilities that are not present on most developer’s laptops when performing builds. To ensure the same experience, the Quay build system runs all builds as root at all times, wrapping them in a virtual machine to ensure security of the cluster while maintaining compatibility.

Build Instance Diagram
Containers are built inside an ephemeral KVM instance and stored in Quay

Things get more interesting within the virtual machine. Inside the virtual machine is our containerized build agent, which connects to the build manager, requests the details of the build (using the token mentioned above), performs the build and push, and streams all logs back to the build manager. The build agent is never given any information unrelated to the build itself, and is never trusted to run a second build. Once a build has completed, the Job is removed via the Kubernetes API, destroying its Pod, the VM and all data found inside.

Interested in learning more? Stay in touch with CoreOS

The CoreOS team is at LinuxCon and ContainerCon in Berlin this week, so please stop by the booth area to learn more.

Get started with Quay hosted today.

If you are interested in learning more and staying in touch with CoreOS about Quay, Quay Enterprise, or Tectonic, please fill out your information.

Quay container registry by CoreOS celebrates two years

August 18, 2016 · By Jake Moshenko and Joey Schorr

Today we celebrate two illustrious years of Quay at CoreOS. Quay, the secure container registry, enables developers and DevOps to build, distribute and deploy their Docker and rkt containers.

Quay container registry by CoreOS celebrates two years

A look back at the container registry of choice

We launched Quay soon after the emergence of the Docker ecosystem.

In using containers to develop our own applications and provide isolated environments for our existing customers, we found there were no tools available to store or manage our private container images. As a result, we decided to build a container repository ourselves.

Our team of two started the initial implementation of Quay on September 20, 2013, resulting in a working prototype. Building on our proof of concept, we continued to develop and formally released the first version of Quay on October 3, 2013. We made our grand announcement at the Docker Meetup in New York City that evening, where we, fortuitously, also met Alex Polvi, CEO of a then-new startup called CoreOS. This chance meeting established a strong professional relationship, one which would eventually see Quay become part of the CoreOS family.

“Secure hosting for private Docker containers” was our original slogan. To continue the shipping analogy associated with containers, we decided to name the product “Quay,” as a quay is the part of the pier where containers are stored while being unloaded from a ship. Likewise, Quay is the platform from which you can build, store, and manage your software containers.

Our initial features included setting up permissions, and creating and deleting the repository. As our initial customer base grew rapidly, it became clear we were meeting a market need. Since we were providing such a needed service, our early customers spread the word and drove rapidly expanding adoption. Quay became our main focus, and over the next few months we designed a stronger UI and added build support, robot accounts, and organizations to provide a more robust container management experience.

Quay was a popular tool in the container community at this time, but there was also a need for an enterprise-level product. As a result, we started working on Quay Enterprise in early April 2014, simultaneously expanding Quay’s features, while working to ensure a shared consistent, code base between our hosted product and our budding enterprise solution.

During this time, primarily through our continued relationship with Alex Polvi, we realized we had a growing symbiotic relationship with CoreOS. Quay Enterprise customers were deploying their containers on CoreOS because they wanted to run their containers on a trusted OS, and CoreOS provided the security, features, and performance they were looking for. CoreOS customers, in turn, needed a container registry to manage their images: it was a great match.

Realizing the benefits to our customers, in August of 2014, Quay joined the CoreOS family and we’ve been growing the product ever since.

Quay and Quay Enterprise by CoreOS today

In the last two years, we’ve continued to build Quay as the premier container registry. Customers are choosing Quay and Quay Enterprise for its advanced functionality and features:

  • Quay security scanning with Clair scans your container images layer by layer to inform you of any known vulnerabilities and notifies you if your images are susceptible to a newly discovered exploit

  • Georeplication provides the peace of mind that your content across the globe is consistent and highly available

  • Fine-grained Access Control Lists and audit logging ensure that you can control and audit who is reading and writing your images

  • Multiple means of distribution, from squashed images to torrent distribution, make it faster and easier to deploy container-based applications

  • Quay’s automated garbage collection ensures you’re always using resources efficiently for active objects without any downtime of your critical registry infrastructure

  • Automated Docker Builds allow you to streamline your CI/CD pipeline for an even quicker application development cycle

Have you used Quay? Share your favorite features and feedback and tag @quayio on Twitter or contact us to share your story and we will enter you in a drawing for a Quay gift!

Try Quay out

Need a better way to store, build, and manage your containers? Try the Quay container registry today for top-of-the-line container management.

A Torrent of Pulls

May 9, 2016 · By Joseph Schorr

Today at CoreOS Fest 2016, Brandon Philips, CTO of CoreOS, highlighted news that we are bringing together BitTorrent and Quay for improved efficiency. This is a preview feature and tool that enables support for pulling appc and Docker container images using BitTorrent, with the new quayctl tool. Quayctl integrates with both the rkt and Docker container engines and we want you to try it out.

Improving container images in production

Containerization technologies such as rkt and the docker engine have provided major benefits for the deployment of software such as reproducibility, ease of development and ease of deployment. One major painpoint developers encounter when starting to use containers in production is the sheer size of the container images produced. Distributing large container images today to large numbers of machines can be a headache when bandwidth is limited, registries are located geographically far away, or the number and size of image layers are quite large.

At CoreOS, we’ve been giving a great deal of thought to how to mitigate some of the headaches associated with distributing of container images. Last year, we launched our squashed image support in Quay for downloading container images without their intermediate layers. Squashed images have been a great benefit to our customers and ourselves, with reported download times as much as 50% shorter versus a standard docker pull. However, squashed images must be fully pulled from registries every time, putting load on network gateways and not using local resources. Machines and services are rarely started alone; what if there were a way to share images within a cluster?

Using BitTorrent with containers and Quay

Enter BitTorrent. BitTorrent provides an easy way for multiple machines within a cluster to share pieces of large binary files peer-to-peer. Many organizations, such as Twitter and Facebook, have adopted BitTorrent for deployment with significant results. With BitTorrent, companies have seen drastic reductions in download and deployment time, as well as increased stability from having multiple machines serving their binary data.

Recognizing the benefits of using BitTorrent with containers, we are excited today to announce support for pulling appc and Docker images using BitTorrent via the new quayctl tool. Developers will save time on deployment of their containers, as well as benefit from increased efficiency thanks to the ability to use BitTorrent and Quay.

A torrent of images

Introducing BitTorrent-based pulling of Quay images into your deployment process is quite straightforward using the quayctl tool.

First, download the binary version of the tool from the GitHub releases page via curl or wget.

Once installed on a machine, quayctl can be used in place of rkt fetch or docker pull to pull the image via torrent:

$ quayctl rkt torrent pull
Discovering image
Downloading torrent for image Completed
Loading image
Successfully pulled image

$ quayctl docker torrent pull
Downloading manifest for image
Downloaded manifest for image
sha256:3eecc807f63d Completed
sha256:48341aa39e69 Completed
sha256:f0a98344d604 Completed
Connecting to docker
Pulling image
f0a98344d604 [===========================] 100.00 % 0  Pull complete
3eecc807f63d [===========================] 100.00 % 0  Pull complete
48341aa39e69 [===========================] 100.00 % 0  Pull complete
Successfully pulled image

For each layer of an image, quayctl will pull the layer’s contents via torrent, using local seeds where possible and falling back to a Quay-provided webseed when image data is not yet in the cluster. During the download process, each client will seed to others in the swarm.

Once all layers have been downloaded, quayctl will then call the container engine to load the layers into the image store, allowing the image to be used as if pulled via a normal call:

$ rkt image list
ID                     NAME

Ensuring privacy and security

Private repositories and container images provided a unique challenge for implementing BitTorrent support.

For pull credentials, quayctl will automatically read from the local rkt or Docker credentials store, allowing existing flows that make use of rkt credential config or docker login to continue working without change.

When sharing pieces, Quay makes use of the Chihaya open source BitTorrent tracker to find peers. As part of this project, we implemented a custom JWT auth solution, allowing only those with the necessary signed token to access the peer list for private blocks. This allows private clusters to load peers for private images, without risk of leaking data or the IP addresses of the peers.

Seeding the cluster

To ensure maximum sharing within a cluster, it is recommended that all machines using BitTorrent seed their data once a container image has been downloaded. The quayctl tool provides a dedicated seed command which allows seeding of an image, either indefinitely or for a specified period of time:

$ quayctl rkt torrent seed --duration=10m

Squashed images

In addition to using BitTorrent for pulling Docker images layer-by-layer, quayctl can also be used to BitTorrent pull squashed Docker images from Quay (rkt images are already squashed).

Squashed images can be pulled by adding the --squashed argument to the torrent pull command

NOTE: In order to torrent pull a squashed image, the image must have been downloaded at least once via a normal HTTP request

$ quayctl docker torrent pull --squashed
Starting download of squashed image Completed
Importing squashed image
Successfully pulled squashed image

Once pulled, the image will be available with the tag tag.squash

Benefits of torrenting

In our initial experiments, we have found that torrenting of images yielded reduced download times when performed on clusters behind single gateways to the external internet, or on clusters geographically remote from US East (where Quay stores its data). In both cases, the ability for machines locally within the cluster to share image data helped reduce download times compared to pulling data from Quay’s storage provider across the gateway or the world.

In our next experiments, changing to torrenting of ACI or squashed images resulted in even larger download time reductions (over 30% reduction from a normal pull), as the single images were significantly smaller than the sum of all the layers. We also saw far less variation in download times, as most of the pieces were pulled from within the local cluster.

We believe that torrenting will result in even further reductions of download times when used in environments with very large numbers of machines starting at once (50+), very large container images (1GB+) or on providers without direct networking access to Amazon S3.

Get started with BitTorrent and Quay by CoreOS

Using BitTorrent and Quay to store and manage application containers saves developers time with the ability to develop and distribute large container images quickly, safely and efficiently within a cluster. Get started with the Quay container registry to get the benefits of faster development and deployment.

Quay Enterprise Security Scanner Now Available

May 6, 2016 · By Jimmy Zelinskie

We are happy to announce that Quay Security Scanner is now available in the latest release (v1.16.0) of Quay Enterprise, the on-premises version of the Quay container registry by CoreOS. This release marks the Quay Security Scanner feature as enterprise ready. When this feature is enabled in Quay Enterprise, all container images in the registry are indexed and cross-referenced against public vulnerability databases. Enterprises have long invested in their security auditing process in a pre-container world, and tools like Quay Security Scanner aim to extend that process into container cluster infrastructure.

Your build process: A ticking time bomb

With the popularity of large container base images, the majority of containers passively include an additional operating system full of software — and software vulnerabilities — along with their applications. Worse, this layer of system software is often cached when rebuilding a container, leaving new container images with an updated target application but old libraries and system utilities. As these base-images continue to stagnate, more and more vulnerabilities are discovered and until now developers have had no convenient tooling available to audit their containers.

Quay Security Scanner in a nutshell

Driven by the open source Clair container image scanning engine, Quay Security Scanner is the first vulnerability detection system to be fully integrated with an on-premises container registry. All container images stored in Quay have an associated Vulnerabilities page that shows developers the software installed in each container, the known vulnerabilities afflicting that software, and whatever fixes are available for those security issues. Unlike other solutions, images need only be scanned once, while the vulnerability data is always update to date. The Security Scanner is smart enough to update previously indexed packages when new vulnerabilities appear, without scanning the container image again.

Quay Security Scanning Overview screen

Quay’s notification system includes an additional event type that users can configure to select what severity levels will trigger notifications of vulnerabilities that are initially detected in an image, as well as if vulnerabilities are later reclassified as more dangerous. Quay notifications can be sent to users via Quay itself, e-mail, webhook, Flowdock, HipChat, or Slack.

Improve your container security with Quay Enterprise

Current customers can enable Quay Security Scanner by following the instructions found in the Quay Enterprise documentation. More information on the latest release of Quay Enterprise can be found in the release notes. If you don’t already have Quay Enterprise, all plans start with a free trial.

Learn more and get any questions answered with our expert team at CoreOS Fest Berlin, May 9-10, and San Francisco, May 9.

Quay Security Scanner now powered by Clair 1.0

March 25, 2016 · By Jake Moshenko

You may have heard that the open source project Clair by CoreOS recently released version 1.0. If you’ve been following along, you may also know that Quay’s Security Scanner, a container registry feature that analyzes container images for known vulnerabilities, is based on Clair. Quay Security Scanner now has an entirely new interface atop the Clair 1.0 APIs and PostgreSQL backend. In this post, we discuss some of the details of the new Quay user experience that make essential security information more accessible, reliable, and actionable.

We welcome you to try out Quay with Security Scanning to examine your container images and take action to repair discovered vulnerabilities.

Quay Security Scanning Overview screen

Clair 1.0 delivers improved performance in the Quay Security Scanner

When we launched the early preview of the Quay Security Scanner, we received two consistent patterns of feedback:

  1. Provide guidance to take action: Scanning results should offer more guidance on corrective actions. When users received the bad news that their container images were plagued with vulnerabilities, they needed to figure out the fixes for themselves. Where were the vulnerabilities coming from? What could be done to mediate them?
  2. Improve performance: Users asked for better performance from the analyzer to provide results faster for larger numbers of more complex container images.

The latest update to Quay improves and expands the remediation information presented to administrators when vulnerabilities are detected, and includes the locations of upstream fixes and updates. Clair now uses a well-typed SQL database structure that allows it to comprehensively track the packages that introduce or fix a vulnerability. This empowers container image developers to take action to repair the detected security issues.

To improve performance, we leveraged PostgreSQL recursive queries to collate all of this information directly within the database, without any round trips between the data store and the application logic. The overall performance improvement in Quay Security Scanning amounts to a measured reduction of 99.9% in the time required to scan and report on deeply layered container images with large numbers of packages.

With a significantly quicker and richer data model, the Clair API and the Quay Security Scanning GUI built atop it have matured to expose scan information more easily and completely. Check out the Clair 1.0 release announcement for more background on the API changes supporting these dashboard improvements.

Distilling Clair data into compelling insights

New Quay Security Scanning sports a remodeled dashboard for vulnerability data. It has new high-level views to visually summarize security status across your entire container repository, as well as presents analysis detail and suggested fixes and updates through a series of powerful drill-downs.

The high-level CVE view now summarizes vulnerability data in a glanceable, prioritized way. It breaks down the total set of vulnerabilities to which an image is susceptible — and what patches are ready to be applied to affected packages within the container image.

Quay Security Scanning CVE view

Each discovered vulnerability is now described in greater detail on the vulnerability drill-down pages. These panels enumerate the components that determine the severity of a vulnerability according to the Common Vulnerability Scoring System (CVSS). The screenshot below shows the categories on which an example vulnerability is scored, allowing administrators to prioritize remediation actions for the most significant threats.

Quay Security Scanning drilldown

Even the best analysis of individual vulnerabilities can be of limited utility. For example, you may manually check an image for famous vulnerabilities such as Heartbleed, Shellshock, and GHOST. These are only three of the more than 18,000 vulnerabilities tracked in the Clair database today.

Less notorious vulnerabilities are just as damaging to security. Regular tracking and updating of these issues is essential. To assist developers in this process, the latest Quay Security Scanner introduces a new package view, which summarizes critical flaws and maps them to the packages they affect.

Quay Security Scanning Container Packages overview

With great data comes great responsibility

A major added value in the new Quay Security Scanner dashboard is the ability to take action on vulnerability data in an obvious way. The common way to resolve a vulnerability is to either upgrade or remove an affected package. Therefore it is very important to know how changing or removing a package affects the total attack surface of all identified vulnerabilities.

An analysis of the entire Quay dataset indicated that 80% of all high- or critical-level vulnerabilities can be fixed through a simple upgrade. In order to encourage developers to upgrade these packages, we’ve begun to measure the “upgrade impact” and use that weighted score to provide customizable sorting of the packages in container images. In the example shown below, we could eliminate 11 out of 23 vulnerabilities just by upgrading four packages to their latest versions.

Quay Security Scanning impact ranking

Quay also shows the original command that introduced the package to the container image, helping developers immediately identify exactly where in the build process a package upgrade is needed.

When looking for candidates for removal, users can sort by vulnerability count to quickly identify those with a very low upgrade impact score. In this way you have the largest impact on your vulnerability surface for items that don’t currently have patches or fixes available. In the screenshot below, by removing libv8, we could remove 32 vulnerabilities that were otherwise unfixable. If this dependency was introduced by a build or test tool, this is a very worthwhile removal to consider.

Quay Security Scanning

Polish throughout the Quay UI makes it as easy as possible to take action to repair problems found by Security Scanning, such as adding a toggle to show only fixable vulnerabilities in the vulnerability view. Combining this actionable information with the existing vulnerability notification system, we and the developer community can dramatically reduce the number of fixable vulnerabilities in container images stored on Quay.

Greater security through continuous improvement

All of these great features are ready to use today in the hosted version of Quay. We welcome you to try it out for free. We are also committed to shipping these features for on-premises use in Quay Enterprise, and Security Scanning will be available in the next release.

We encourage you to check your container images hosted on Quay for vulnerabilities, and use the detailed action information in the new Security Scanner dashboard to eliminate them. Together, we can get the fixable vulnerability number down to zero! CoreOS is on a mission to secure the infrastructure that powers the Internet, and Quay and Clair help drive that goal forward with tools to address in-container vulnerabilities.

If you have any feedback about Clair, Security Scanning, or anything else Quay, we would be happy to hear from you via Twitter, IRC, or in an email to