Jenkins X: Monitor Tekton Pipelines deployments with Kibana

If something goes wrong, the first thing you do is look at the logs. For serverless Jenkins X it is a little problematic if some time has passed -  build pods have already died safely and jx get build logs only gives you:

error: no Tekton pipelines have been triggered which match the current filter

But what if you want to find the cause of some error or compare the current build log with yesterday?

There is a dashboard for Tekton which looks nice but persistent logs unfortunately are not supported yet (I hope this feature will appear in the future).

Happily there is an Elastic Stack. I installed the Kibana following the instructions from tekton-pipelines-elastic-tutorials.

Although there are an warning that it was tested on a Mac only, everything started and works fine on Ubuntu, for example.

Kibana allows not only to monitor Tekton Tasks or Pipelines (and not only) in real time but also to check one or another log for yesterday or any other day.

How To Install Jenkins X On Ubuntu VM

I hope these steps will be useful for those who want to try Jenkins X on a local VM without external IP (NAT). In my case this is VMware ESXi VM, Ubuntu 18.04.2 LTS, 10GB RAM.

For the Kubernetes installation, just follow the good instructions from here. 

My installation was different in that the to fix coredns Pending state, I’ve installed weave-net pod network instead of CALICO:

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\ n')"

Few preparations were necessary before jx install: I killed much time getting ADDRESS column not empty in kubectl get ing output, using ClusterIP, NodePort and LoadBalancer in rotation, so I chose separate ingress install:

$ helm upgrade --install ingress --namespace=ingress --set \

  rbac.create=true,controller.kind=DaemonSet,controller.service.type=ClusterIP,controller.hostNetwork=true,controller.extraArgs.report-node-internal-ip-address=true stable/nginx-ingress

If you see:

Error: could not find tiller 

use these commands:

$ kubectl —-namespace kube-system create serviceaccount tiller

$ kubectl create clusterrolebinding tiller-cluster-rule —-clusterrole=cluster-admin —-serviceaccount=kube-system:tiller

$ kubectl —-namespace kube-system patch deploy tiller-deploy -p ‘{“spec”:{“template”:{“spec”:{“serviceAccount”:”tiller”}}}}’ 

$ kubectl taint nodes —all node-role.kubernetes.io/master- 

   Then I used nfs provider to get persistent volumes (nfs server is required):

$ helm install stable/nfs-client-provisioner --set nfs.server=NFS_SERVER_IP --set nfs.path=NFS_SHARE_PATH 

       

Now it's ready for install:

$ jx install --provider=kubernetes --external-ip=YOUR_VM_IP --ingress-service=ingress-nginx-ingress-controller --ingress-deployment=ingress-nginx-ingress-controller --ingress-namespace=ingress --on-premise 

Answer "no" when jx will ask "No existing ingress controller found in the kube-system namespace, shall we install one?"

After install, check that ADDRESS column is not empty in kubectl get ing output:

$ kubectl get ing
NAME                HOSTS                                    ADDRESS      PORTS   AGE
chartmuseum         chartmuseum.jx.10.2.20.46.nip.io         10.2.20.46   80      42h
deck                deck.jx.10.2.20.46.nip.io                10.2.20.46   80      42h
docker-registry     docker-registry.jx.10.2.20.46.nip.io     10.2.20.46   80      42h
hook                hook.jx.10.2.20.46.nip.io                10.2.20.46   80      42h
jx-jx-app-jenkins   jx-jx-app-jenkins.jx.10.2.20.46.nip.io   10.2.20.46   80      41h
monocular           monocular.jx.10.2.20.46.nip.io           10.2.20.46   80      42h
nexus               nexus.jx.10.2.20.46.nip.io               10.2.20.46   80      42h
tide                tide.jx.10.2.20.46.nip.io                10.2.20.46   80      42h

  Check the local hook ingress connectivity:

$ curl -S hook.jx.10.2.20.46.nip.io

Now you can import your local project to Jenkins X with jx import or add some predefined projects with jx create.  
Since VM does not have an external IP,  webhooks that Jenkins prescribed on a Github will not work. There are several solutions here - for example ultrahook.  Replace Jenkins X webhooks URL for Staging, Production and your Dev repositories on Github with ultrahook URL and run with an ingress hook URL:

$ ultrahook github http://hook.jx.10.2.20.46.nip.io/hook

To get Jenkins X console working (it only makes sense if you use Jenkinsfile in some project),  needs to install Jenkins App first (choose all required plugins):

$ jx add app jx-app-jenkins

and run it:

$ jx console

  Jenkins Console: http://jx-jx-app-jenkins.jx.10.2.20.46.nip.io

After opening this URL in browser, you'll see usual Jenkins login page. Surprisingly, the admin password you wrote down after installation is not suitable here. Maybe I missed something, but I had to run the following commands to find out the password:

$ go get github.com/mfuentesg/ksd

and then look for the jenkins-admin-password field:

$ kubectl get secret jx-jx-app-jenkins -o yaml | ksd

And you should see the familiar interface after login: