Continuous development using skaffold

Skaffold is by and large a developer tool to build software, test it and deploy it to targeted environment as we will see in this blog. There is plethora of option that can be made use of in the controlling file, skaffold.yaml , once developer have generated it. As the name suggest it becomes the basis of your infrastructure and also of your software build and deployment. What is of peculiar interest for an infrastructure person, is to see, how this tool glues the development to operations, as it also creates the infrastructure and deploys the software in that infrastructure. 

Skaffold essentially requires developer to generate the controlling file or the pipeline, skaffold,yaml, that has sections for:

• Building the the software from build context
• Generate the artifact and store it in repo
• Tag the artifact
• Deploy the Artifacts to your kubernetes cluster, using the depolyer of your choice.

In Dev mode, as we see in this article, it can continuously watch for the changes in source files, in build context and automatically push new changes to deployed infrastructure for local development loop and continuous testing.

Skaffold pipeline can be seen to have following stages, and as one can imagine these can be placed as workflow tasks in CI/CD tools such as GitHub Actions, for an evolved experience with skaffold:

# skaffold init :     

                Seed and generate a controlling file, skaffold.yaml, for whole build, test, deploy process.

# skaffold build :     

                Build images with different builders available.

# skaffold test :     

                run test with testers.

# skaffold deploy :     

                deploy with deployers, kubectl, helm or kustomize.

# skaffold verify :     

                verify the deployments with specified test containers.

# skaffold render :     

                render the manifests with different renderers (rawYaml, helm, kustomize, kpt).

# skaffold run :     

                to build and deploy once.

# skaffold dev :     

                to enter continuous development loop for local development and testing.

# skaffold debug :     

                run the pipeline in debug mode.

# skaffold apply :     

                to apply hydrated manifests to a cluster.

For complete references of CLI, click here.

Lets’ just get started with build, deployment of one of the developer project in angular by cloning this repo. Thanks to Dan Wahlin for code base.

So, looking into the build context, one can see a docker-compose.yaml file in the code base. Yes, it is multi-container app !! This file specifies nginx service with its docker file location, build context and other details and also same set of information for second service, a node-js service. 

Both these services comes provisioned under network, app-network using bridge virtual device for networking amongst the services in our software that would be deployed in container ecosystem. Run following command from README file to generate the skaffold pipeline file, there is little modification in the command line for first service build context in value of payload. 

Below command uses docker-compose.yml file to convert it to skaffold constuct. Skaffold internally uses Kompose to convert the compose files to skaffold specs.

# skaffold init --compose-file docker-compose.yml -f skaffoldtest.yaml \

> -a '{"builder":"Docker","payload":{"path":".docker/nginx.dockerfile"},"image":"nginx-angular-jumpstart"}' \

 > -a '{"builder":"Docker","payload":{"path":".docker/node.dockerfile"},"image":"node-service-jumpstart"}'

Output of above commands is stored in skaffoldtest.yaml, as shown below:

This will generate rawYaml that represents the deployment / service file spec in kubernetes for our services. In build stages, we have changed the build context of angular image to current working directory and also to reflect the right build context and location of Dockerfile for node-js service.

Since we already have the kubernetes deployement and service files in hidden ./.k8s directory, we will remove the redundant rawYaml files from our current directory context. Reflect the same in skaffoldtest.yaml file as below, under deploy stanza, besides updating the context directory for both images,

apiVersion: skaffold/v4beta9
kind: Config
metadata:
  name: angular-jumpstart
build:
  artifacts:
    - image: nginx-angular-jumpstart
      context: .
      docker:
        dockerfile: .docker/nginx.dockerfile
    - image: node-service-jumpstart
      context: .
      docker:
        dockerfile: .docker/node.dockerfile
manifests:
  rawYaml:
  - .k8s/*.yml  
deploy:
  kubectl: {}

Now, we are ready to check our developer work in a local development loop, run 

# skaffold dev 

that generates the tags for images, image artifacts for services, deploys them using k8s deployment and service definitions for both nginx and node-js. 

Generating tags...

- nginx-angular-jumpstart -> nginx-angular-jumpstart:12fa4e4-dirty

- node-service-jumpstart -> node-service-jumpstart:12fa4e4-dirty

Checking cache...

- nginx-angular-jumpstart: Not found. Building

- node-service-jumpstart: Not found. Building

Starting build...

Found [custom] context, using local docker daemon.

Building [node-service-jumpstart]...

Target platforms: [linux/amd64]

Sending build context to Docker daemon 1.332MB

Step 1/11 : FROM node:16.17.0-alpine

16.17.0-alpine: Pulling from library/node

213ec9aee27d: Pulling fs layer

864b973d1bf1: Pulling fs layer

80fe61ad56f5: Pulling fs layer

e3887ab559e6: Pulling fs layer

e3887ab559e6: Waiting

213ec9aee27d: Verifying Checksum

213ec9aee27d: Download complete

.

.

.

.

.

<clipped for brevity>

Starting deploy...                                           <<<<Deploys your K8s manifests

- deployment.apps/nginx created

- service/nginx created

- configmap/env-vars created

- deployment.apps/node created

- service/node created

Waiting for deployments to stabilize...        <<<<< Monitors for services to be ready

- deployment/node is ready. [1/2 deployment(s) still pending]

- deployment/nginx is ready.

Deployments stabilized in 2.133 seconds

Listing files to watch...

- nginx-angular-jumpstart

- node-service-jumpstart

Press Ctrl+C to exit

Watching for changes  <<<< Actively monitors for changes in context dir for rebuild

                                                   and Redploy

[node-service-jumpstart] Express listening on port 8080

At this time, remember you are making use of minikube and images are stored in

minikube environment and services are deployed in minikube under profile 

named custom, check this blog for creating custom cluster using minikube:

# minikube image ls -p custom | grep -i nginx

Set docker variables to minikube, by running

# eval $(minikube -p custom docker-env)

# env | grep -i docker 

output of above command: 

DOCKER_TLS_VERIFY=1

MINIKUBE_ACTIVE_DOCKERD=custom

DOCKER_CERT_PATH=/home/xyz/.minikube/certs

DOCKER_HOST=tcp://192.168.49.2:2376

 

This ensures that skaffold is aware of the local cluster running out of minikube.

Run kubectl commands to get the deployment information on our services:

# kubectl describe deployment.apps/nginx

# kubectl describe deployment.apps/node

Also note the services kubernetes object for nginx would not have been exposed 

for either node port or external access via Load balancer port type. This becomes 

clear when you observe EXTERNAL-IP field of service/nginx, as seen below

# kubectl get all -l app=node

These services will be exposed for external access when we create tunnel route , it inserts a routing entry in the route table like below: 

# minikube tunnel -p custom  <Run this in another terminal with sudo password requirement>

# ip route 

10.96.0.0/12 via 192.168.49.2 dev br-ce631cd494b3

Note that minikube sits on below IP addy, and hence tunnel is made as above route entry,

# minikube ip -p custom

192.168.49.2 

 

Now we are all set to check the continuous development of skaffold. As show above skaffold dev is not waiting for changes, if there are any, it is going to rebuild images and redeploy the application.

 

In another terminal, go to directory Angular-Jumpstart/src/spp and modify the file,  app.component.html , that looks like this:

 

 <main class="container">
    <cm-navbar></cm-navbar>
    <router-outlet></router-outlet>
    <cm-growler position="top-right" [timeout]="2000"></cm-growler>
    <cm-modal></cm-modal>
    <cm-overlay [delay]="300">
        <span class="spinner"></span>&nbsp;&nbsp;&nbsp;Loading
    </cm-overlay>
</main>
<br /><br />

TO

 

<main class="container">

          HELLO WORLD

          <cm-navbar></cm-navbar>

          <router-outlet></router-outlet>

          <cm-growler position="top-right" [timeout]="2000"></cm-growler>

          <cm-modal></cm-modal>

          <cm-overlay [delay]="300">

                    <span class="spinner"></span>&nbsp;&nbsp;&nbsp;Loading

           </cm-overlay>

</main>

<br /><br />         

           

 

And voila, check your terminal where skaffold dev is running, it immediately rebuilds 

image and redeploys kubernetes manifests.