In Part 1 of Prodvana's Architecture, we highlighted the need for intent-based deployment systems and an overview of the three components that make up Prodvana. In this part, we will examine the first component of Dynamic Delivery: The Prodvana Compiler.

Teams are moving towards decentralized ownership. The person who develops the application is not the same person who owns the infrastructure, environments, or release processes.

In a decentralized world, having deployment configurations be a series of steps in a pipeline requires all configuration stakeholders to understand all pipeline steps in equal depth. The cognitive overhead is extraordinarily high, preventing individual teams from moving at their maximum velocity.

With intent-based deployments, each stakeholder is responsible only for defining their requirements. The deployment system compiles the requirements into a release or deployment.

In their Managed Delivery blog post, Netflix explains this as:

"For Netflix to evolve delivery and infrastructure details for application owners, Spinnaker needs to understand their application delivery requirements, not just a list of individual steps." - Managed Delivery: Evolving Continuous Delivery at Netflix

Most deployment systems are intent-based only on what kind of workload to run (e.g., Kubernetes configurations) and fall back to a static set of steps to deploy the workload.

This is why we built the Prodvana Compiler, a desired state compiler.

A desired state captures what workloads should be run, where they should run, and the requirements/constraints for running them. In other words, a desired state captures the user intent for deploying a system.

The Desired State Spec

To capture the deployment intent, a desired state must contain the following information:

• A list of environments to run the workload in.

• For each environment:

  • The runtime configuration for the workload, e.g., the Kubernetes configuration.

  • Where (i.e., which cluster) the workload should run.

  • The list of conditions that must be true before the workload can be deployed to this environment.

User Configurations

The Compiler splits configurations between Release Channels (environments) and Services (workloads).

Release-Channel Configurations: The Environments

Release Channel requirements constitute the environment that Services can run in.

Here is an example of an Application with three Release Channels: staging, production-us, and production-eu, where the two production Release Channels are configured to deploy after staging.

application:
  name: my-application
  releaseChannels:
  - name: staging
    runtimes:
    - runtime: staging
  - name: production-us
    runtimes:
    - runtime: production-us
    preconditions:
    - releaseChannelStable:
        releaseChannel: staging
    - manualApproval: {}
  - name: production-eu
    runtimes:
    - runtime: production-eu
    preconditions:
    - releaseChannelStable:
        releaseChannel: staging
    - manualApproval

Notice the following:

• Unlike a pipeline definition, there are no individual steps. Release ordering and manual approvals are defined as intent-based requirements.

• Each Release Channel is responsible for defining its definition of stability. For example, if staging needed to receive traffic for 60 minutes to ensure no alerts were firing, only staging would need to be updated. The two production Release Channels would then automatically wait for staging to be stable. This is especially useful if staging has a different owner from production (e.g., QA teams). An updated version of the staging the configuration would be:

- name: staging
  runtimes: ...
  protections:
  - ref:
      name: no-alerts-staging
    lifecycle:
    - postDeployment:
        checkDuration

• Each Release Channel defines its infrastructure requirements (i.e., what Runtime to run in), but without knowing the infrastructure-level details about that Runtime (e.g., what Kubernetes version the Runtime is on).

• There is no Service-level information, e.g., Kubernetes configurations. 

A common theme emerges: intent-based requirements allow for separation of responsibility. For example, one does not have to know which Services are running to define Release Channel requirements.

Service Configurations: The Workloads

Service requirements describe the workload. We minimize modifying existing configurations, like Kubernetes or ECS definitions, by referencing existing configurations only. Here is an example:

service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    local:
      path

The Prodvana Service configuration file references the existing Kubernetes configuration file, making it trivial to import a Service on Prodvana. The Prodvana Service configuration file is then submitted to Prodvana with our CLI, pvnctl. pvnctl resolves any local file references and inlines the content of the files into the configuration file itself.

service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    inlined: |
      apiVersion: apps/v1
      kind: Deployment
      ... # the rest of Kubernetes configuration here

The same pattern is used for other types of backends. Here are examples of how Prodvana works with Helm and ECS:

Helm

service:
  name: my-service
  application: my-application
  helm:
    remote:
      repo: https://kubecost.github.io/cost-analyzer/
      chart: kubecost/cost-analyzer
      chartVersion: "1.102.2"

ECS

service:
  name: my-service
  application: my-application  
  awsEcs:
    taskDefinition:
      local:
        path: task-definition.json  # ECS standard task definition JSON
    serviceDefinition:
      local:
        path: service-definition.json  # ECS standard service definition JSON

Parameterization - Practical Gitops

Checking a Service's configuration into a repository enables a GitOps workflow with Prodvana. Changes to any Service must go through a configuration file change, which can trigger any necessary code review or testing.

However, we have found that this workflow can become onerous as organizations scale up and have bottlenecks in code review, CI, and other areas. A classic example here is updating the image being deployed and, often, updating the image automatically on staging as soon as tests pass.

ArgoCD, a fully GitOps-based deployment system, works around the GitOps commit limitation by having an external service make these changes and commit them back to the repo. Solutions that circumvent the GitOps limitation only add to the problem. Now, the git repository is in the critical path of deployment. GitOps users end up with many automated commits, making it necessary to split out configuration repos from code repos.

We do not believe GitOps is a one-size-fits-all solution. Prodvana provides flexibility through parameters as a first-class primitive. This delivers the benefits of GitOps while allowing teams to denote changes that bypass configuration file changes where policy allows.

Parameters are predefined values in the Prodvana Service configuration file that can be referenced from anywhere in the configuration hierarchy, including referenced external configuration files (e.g., Kubernetes configuration).

Here is how you parameterize the image.

service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    local:
      path: deployment.yaml
  parameters:
    - name: image
      required: true
      dockerImage:
        imageRegistryInfo:
          containerRegistry: my-registry-name
          imageRepository

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
    spec:
      containers:
        - name: my-container
          image: "{{.Params.image}}"

At deployment time, Prodvana will prompt for the values for the parameters in the web UI or be specified as part of a command line operation.

Parameters allow for significant changes to undergo code review while enabling swift implementation of minor, predefined adjustments. Parameters are optional, so they can be turned off for teams that want to adopt a strict GitOps workflow.

Working with Existing Configuration Files

As we talked to various teams, we encountered an increasing permutation of how configurations are written. In the Kubernetes ecosystem alone, we have seen Kustomize, jsonnet, starlark, and fully custom config generation scripts.

We had a choice: support all of these configuration methods by building support directly into the Prodvana Compiler or have the configuration happen and compile down to plain Kubernetes before entering our systems.

We settled on the latter because building support into the Compiler meant Prodvana would be on the critical path for new user onboarding whenever a new configuration method is added.

For example, users of Kustomize would first run kubectl kustomize to produce the denormalized Kubernetes configuration before running pvnctl to submit the file to Prodvana (side note: for Kustomize, we did build in first-class support that wraps this process, but the idea is the same). Similarly, users with custom config generation can keep their custom generation and produce a final Kubernetes configuration file used with Prodvana.

The order of compilation does have one downside. Prodvana parameters are evaluated in the Prodvana Compiler after any user configuration generation. This makes parameterizing non-string values like replica count difficult in some configuration generation methods. For example, this would not be accepted as valid YAML by Kustomize:

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: {{.Params.replicas}}  # not valid yaml syntax

Yet, this would produce the wrong output with replicas being a string:

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: "{{.Params.replicas}}"  # incorrect type for replicas

We worked around this by allowing Kubernetes configuration patches inside the Prodvana Compiler.

service:
  kubernetesConfig:
    type: KUSTOMIZE
    local:
      path: deployment.yaml  # replicas in here doesn't matter, gets replaced below
    patches:
      - replace:
          path: /spec/replicas
          intAsString: "{{.Params.replicas}}"

A Rejected Alternative: Abstracting Away the Backend

A solution that would eliminate the need to interact with various configurations is to abstract them all away. There are even configuration languages built to be backend-agnostic, like Score. We experimented with this approach and rejected it during the design phase.

While the promises of backend-agnostic configurations are great, enforcing them at the deployment system was impractical. Esoteric, backend-specific settings would leak in a way that did not make sense for other backends, and onboarding would be fragile as users would have to modify their existing configurations. This means making two significant changes simultaneously as part of onboarding: a workload configuration migration and a deployment system move. This increases risk and is cumbersome to accomplish simultaneously.

Handling Variations

Now that the base case of repeatable Services is handled, we must consider variation by Release Channel. For example, a Service may have a different replica count for each Release Channel or a different Kubernetes configuration file for each Release Channel. Unfortunately, there are no accepted standards across various systems or organizations.

We understand asking users to change how they structure their configuration is not always feasible (e.g., the configuration may be part of a different overall hierarchy or generated using a tool that cannot be modified easily). Therefore, we had to make our configuration language flexible while achieving simplicity for new services.

Here are the different variations of configurations we have seen and how we handle them:

• simple configuration variation per Release Channel.

• complete configuration redefinition per Release Channel.

• arbitrary definitions across the code base.

Simple Variation per Release Channel: Template Variables

In many cases, Release Channel differences are minimal and can be captured via a few template variables. We often see this with customers who started with a single Release Channel and want to expand to more on Prodvana.

service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    local:
      path: deployment.yaml
  perReleaseChannel:
  - releaseChannel: production
    constants:
    - name: replicas
      string:
        value: "10"
  - releaseChannel: staging
    constants:
    - name: replicas
      string:
        value: "1"

These template variables are termed constants because, unlike parameters, their values remain fixed. They can also be defined globally within the Release Channel as part of the Application configuration.

application:
  name: my-app
  releaseChannels:
  - name: production
    constants:
    - name: env
      string:
        value: production
  - name: staging
    constants:
    - name: env
      string:
        value

The variables can be used in any part of the Service configuration and referenced configuration files like Kubernetes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
  annotations:
    env: "{{.Constants.env}}"
spec:
  replicas

The template variable option is simple and is the recommended default for new Services.

Complete Configuration Redefinition: Configurations with a Predictable Directory Structure

Another typical pattern is to have entirely different configuration files for each Release Channel. This is most common for users of Kubernetes templating libraries, like jsonnet or Kustomize.

If the files are predictably organized, they can be templated. For example, if the directory structure is:

$service/staging/
$service/production

Then the Prodvana Service configuration looks like this:

# this file should live in $service/
service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    local:
      path: .
      subPath: "{{.Builtins.ReleaseChannel.Name}}"

The Service configuration supports relative paths enabling support for directory structures of the form:

staging/$service/
production/$service

# assuming this files live under prodvana/, which should be a sibling of staging/ and production/
service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    local:
      path: ..  # note this change, meaning go up one directory then join with `subPath`
      subPath: "{{.Builtins.ReleaseChannel.Name}}/{{.Builtins.Service.Name}}"

We have also seen configurations organized by properties of the underlying infrastructure. For example:

aws/staging/
aws/production/
gcp/staging/
gcp/production

For such cases, Prodvana supports Labels at the Runtime level, which can then be plumbed through in templating”

service:
  name: web
  application: cross-cloud-provider
  kubernetesConfig:
    type: KUBERNETES
    local:
      path: .
      subPath: '{{.Builtins.Runtime.Labels.cloudProvider}}/{{.Builtins.Runtime.Labels.lifecycle}}'

Arbitrary Definitions: Manual Definition

If the configurations do not follow any predictable patterns, they can be specified manually.

service:
  name: my-service
  perReleaseChannel:
  - releaseChannel: production
    kubernetesConfig:
      type: KUBERNETES
      local:
        path: my-custom-path-for-production/
  - releaseChannel: staging
    kubernetesConfig:
      type: KUBERNETES
      local:
        path

For example, the following configuration uses templating for most directories but with an override for a single Release Channel:

service:
  name: my-service
  kubernetesConfig:
    type: KUBERNETES
    local:
      path: .
      subPath: "{{.Builtins.ReleaseChannel.Name}}"
  perReleaseChannel:
  - releaseChannel: production
    kubernetesConfig:
      type: KUBERNETES
      local:
        path

We can easily onboard workloads by supporting all these arbitrary directory structures without having our users modify directory structures.

Putting It All Together

The Prodvana Compiler takes the Service and Release Channel configurations above and produces these artifacts in the following order:

1. A compiled Service configuration per Releases Channel.

2. An identifier that bundles the compiled Service configurations into one artifact.

3. A desired state that captures the intent and requirements.

An Example Compiled Service Configuration

service: my-service
application: my-application
releaseChannel: staging
kubernetesConfig:
  type: KUBERNETES
  local:
    inlined: "..."  # full content of Kubernetes YAML
runtime:
  runtime

Notice that the compiled configuration is fully self-contained and has all the information needed to run the Service. It is now the full workload specification, not unconnected requirements.

Once all the configurations across all release channels are generated, they are combined into a single bundle and given an identifier - a Service bundle version.

An Example of a Desired State

The Prodvana Compiler then produces a desired state for the service, capturing the release intent and requirements.

service:
  application: my-application
  service: my-service
  releaseChannels:
    - releaseChannel: staging
      versions:
        - version: bundle-version
    - releaseChannel: production-us
      versions:
        - version: bundle-version
      preconditions:
      - releaseChannelStable:
          releaseChannel: staging
          version: bundle-version
    - releaseChannel

Results

• Users typically onboard their first service onto Prodvana in under 20 minutes, including the time it takes to sign up for Prodvana and link a Runtime. 

• Users have been able to adapt their configurations to Prodvana with minimal changes to their configuration or codebase. 

• A significant reason users come to Prodvana is to support regional expansion or complex tenancy. With separate Release Channel and Service configurations, users can seamlessly spin up new Release Channels with minimal Service changes. Our users have, on average, over 5 Release Channels per Application.

Next Step: Convergence

Now that a fully compiled configuration and desired state have been created, the Prodvana Convergence Engine can begin converging to the desired state. To learn more about how the Convergence Engine works, check out Part 3 of Prodvana's Architecture.