Welcome to the second installment of the Cloud Lab series brought to you by Mr. Ken Lomax. If you are still new to Docker and Kubernetes, work through the first CloudLab beforehand to get up to speed. At the end of this Cloud Lab you’ll have a basic experience of a typical tool suite Cloud Engineers need in today’s cloud landscape.
This will include
• Prometheus – to gather monitoring data
• Grafana – to show monitoring data
• Jaeger – to trace the calls
• FluentD – to monitor the logs of your Cloud App
• Canary deployments
Create a project in Google Cloud and enable Kubernetes
Create a new project in your Google Cloud account and take a note of the Project ID.
Enable the Kubernetes API Engine in your Google Cloud Platform.
Launch your cloud shell (more..)
All the following linux commands should be typed in your cloud shell.
Microservices and those pesky Cross Cutting Concerns
Images from (https://www.youtube.com/watch?v=LYeAgVaJrm0)
Istio is an open source Service Mesh, that layers on top of Kubernetes. It gives you
• Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic.
• Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault injection.
• A pluggable policy layer and configuration API supporting access controls, rate limits and quotas.
• Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress.
• Secure service-to-service communication in a cluster with strong identity-based authentication and authorization.
Deploying a microservice-based application in an Istio service mesh allows one to externally control service monitoring and tracing, request (version) routing, resiliency testing, security and policy enforcement, etc., in a consistent way across the services, for the application as a whole.
Close up of proxy (a.k.a. Sidecar/Envoy)
This tour will show some typical Istio use cases in action including:
• canary trials
• Jaeger for tracing,
• Prometheus for monitoring,
• Servicegraph to visualize your microservices, and
• Grafana for viewing metrics
• FluentD for logging
Download Istio 1.0.2.
This includes Istio and example applications.
This includes Istio itself and some example kubernetes applications including BookInfo.
Deploy the Book Info Example without Istio
Create a kubernetes cluster on google cloud
Before deploying as a Kubernetes cluster with Istio added in, let’s deploy BookInfo as a normal Kubernetes cluster.
Add the line type: LoadBalancer to the productpage service in samples/bookinfo/platform/kube/bookinfowithoutistio.yaml
Deploy this (Istio-free) BookInfo application into your Kubernetes cluster:
List the pods that are now running in the kubernetes cluster
Note that there is one of each. (This will change when we install Istio)
Wait for the external-IP to change from “<pendinG>”, to find where you can hit the product page.
Open the BookInfo application
- the product page is forwarding to the 3 different Review services in a round-robin fashion.
- the components are writtien in a variety of languages: java, node, ruby, python
- kubernetes is taking care of the deployment status
- we have limited insight and control into the run-time behaviour of this cluster.
- It would be very cool if we could get much more insight and control of the run-time behaviour of this cluster, without making any changes to the java, ruby, python, node code. This is where Istio can help
Create an Istio-enabled Kubernetes Cluster
Delete the previous Cluster that you deployed without Istio..
With an Istio-enabled cluster, we will want Istio to take care of all traffic entering and exiting the cluster. For that reason we should not include the line type: LoadBalancer in the deployment file we added earlier.
Note that an Istio-enabled cluster needs some extra permissions.
Deploy Istio into this cluster
Add the cluster-admin ClusterRole to your user account, to enable all the added functionality that Istio will introduce.
Deploy Istio into this cluster, in its own namespace “istio-system”.
This will deploy the Istio services and control plane into your Kubernetes Cluster. Istio will create its own Kubernetes Namespace and a bunch of services and deployments. In addition, this command will install helper services. Jaeger for tracing, Prometheus for monitoring, Servicegraph to visualize your microservices, and Grafana for viewing metrics.
Find the istio pods that are running in your Kubernetes Cluster:
The Istio-Sidecar-injector will automatically inject Envoy containers into your application pods, if the pods are in namespaces labeled with istio-injection=enabled, so add that label to your namespace:
Deploy the Book Info Example with Istio
Deploy the BookInfo application into your Kubernetes cluster:
• we used the flag “istio-injection=enabled” when creating our cluster earlier.
• this means that any Pods that Kubernetes creates in the default namespace will automatically get an Istio sidecar proxy attached to it. There is an option to do manual sidecar injection also.
Examine the new Cluster
Find the services and pods that are now running in Kubernetes.
the Istio Pods are not listed. This is because they are in their own namspace “istio-system”
there are now two pods per container – the extra pod in each container is an Istio Sidecar, that Istio has automatically deployed.
Wait for all of your pods to reach a “Running” status before continuing.
Access your BookInfo Application via an Istio Gateway
Note that none of the BookInfo services have an external IP
This is because Istio is managing all traffic. To allow incoming traffic, Istio uses Ingress Gateways.
Right now, the gateway is not set-up, so Istio is dropping all traffic at the edge of the cluster. Set a Gateway up:
This file contains two objects. The first object is a Gateway, which will allow us to bind to the “istio-ingressgateway” that exists in the cluster. The second object is a VirtualService, which let’s us apply routing rules.
Confirm that your gateway is created
Note again that the review section of the BookInfo site rotates between the three review versions: red stars, black stars and no stars.
Start the monitoring services
Open a second window and create tunnels into your Kubernetes cluster for Jaeger , Servicegraph , and Grafana .
This command will not exit as it keeps the connections open.
Return to the first cloud shell and hit your Bookinfo application with some requests using curl:
and let’s try some in parallel
Explore the Tracing metrics
Open Jaeger @ https://ssh.cloud.google.com/devshell/proxy?authuser=1&port=16686
You can also reach this via the web preview button just above your google cloud shell.
Find traces for istio-policy and for productpage
Note: To do this, each service needs to collect and propagate the following headers from the incoming request to any outgoing requests:
Explore the Graphing metrics
Open the Istio ServiceGraph @ https://ssh.cloud.google.com/devshell/proxy?authuser=1&port=8088
This will give you a 404. Just adjust the end of the resulting path to dotviz to see the graph..
(something like https://8088-dot-4500286-dot-devshell.appspot.com/dotviz )
Explore Prometheus Metrics
Open Prometheus @ https://ssh.cloud.google.com/devshell/proxy?authuser=1&port=9090
Use the query:
Other queries to try:
Total count of all requests to the productpage service:
Total count of all requests to v3 of the reviews service:
Rate of requests over the past 5 minutes to all instances of the productpage service:
Explore the Performance Metrics
Open Grafana @ https://ssh.cloud.google.com/devshell/proxy?authuser=1&port=3000
Explore the various Dashboards available.
Modify the Istio Routing
With Istio it is easy to delcaritively define the run-time routing between the services.
D efine the versions, called subsets , to which we can route requests.
With Istio, you can control where traffic goes using a VirtualService.
Let’s send all requests to review v3
Note that all the reviews now show red stars (v3).
Or delete that, and send all to v2 and v3 split 50% each
Note that the reviews now alternate between red and black stars.
Or, even better, have the user jason go to v2, and everyone else to v3..
Logging with FluentD and Kibana
To collect, store and view the logs, we use a FluentD, ElasticSearch, Kibana Stack:
- FluentD – an open source log collector
- ElasticSearch to store the logs
- Kibana to view the logs
Use your BookInfo deployment and google shell to follow the tutorial at https://istio.io/docs/tasks/telemetry/fluentd/
And that is it for the second Cloud Labs session! Please share any comments, Feedback or questions you might have on this or the first Cloud labs article with us and we shall respond!
Keep an eye on
Cloud Labs: an initiative for “Pimping our skills – keeping up with the cloud”
for more CloudLabs in the next weeks