In this post I’m gonna talking about how to secure your micro-services in service mesh using istio. Prefer if the reader have a foundation knowledge of kubernetes but it’s not mandatory at all.

So What is Istio ?

Istio is a service mesh that allows organizations to connect, secure and observe microservices. Since its inception three years ago, it’s risen to become one of Google’s most prominent open source projects, a top-three keyword at KubeCon, and a mature, production-ready offering supported by a robust community.

Solutions built on a microservices-based architecture help solve the problem of breaking down complex tasks into manageable portions that can be built and deployed independently. However, managing a growing set of microservices across different business units within organizations can quickly become a huge headache, if not done well. A service mesh framework like Istio provides the following advanced features to help improve availability and resiliency:

  • Observability through rich tracing, monitoring, and logging of services.
  • Traffic management through timeouts, retries.
  • Paths for safe upgrades through canary rollouts and A/B testing.
  • Automatically secured services by enabling end user and transport authentication, authorization, and audit capabilities.

However, microservices also have particular security needs:

  • To defend against man-in-the-middle attacks, they need traffic encryption.
  • To provide flexible service access control, they need mutual TLS and fine-grained access policies.
  • To determine who did what at what time, they need auditing tools

The goals of Istio security are:

  • Security by default: no changes needed to application code and infrastructure
  • Defense in depth: integrate with existing security systems to provide multiple layers of defense
  • Zero-trust network: build security solutions on distrusted networks

One of the main challenges of managing microservices-based solutions is how to properly secure not just the microservices themselves but also the communication between them.

In this tutorial we focus on how Istio manages security within a service mesh, specifically on how to use mutual transport layer security (TLS) to secure communication between services.

Although our microservices work in internal environments that we generally accept as secure, making the communication between microservices encrypted will be to our advantage for all kinds of security issues.

Well, in the context of security, istio supports two different authentication methods.

  • Transport authentication (mTLS) for service-to-service communication.
  • End-user authentication with JTW for client-to-service communication.

Securing Service-to-Service Communication with Mutual TLS.

Istio uses the Envoy’s sidecar proxy for kind of this operations and to intercept the network as we can see the diagram below.

diagram.jpg

Mutual TLS authentication makes the traffic secure and reliable in both client and server directions.

Understanding how mutual TLS works with Istio.

TLS, a protocol designed to provide secure communication between apps, supports many algorithms to exchange keys and verify message integrity, and various ciphers to encrypt messages. As the number of services scales across multiple deployments, securing them properly can be a daunting task.

Istio completely shifts the burden of configuring security for each individual service away from developers as you don’t have to change any part of application code when it’s comes to develop a security feature to protect your services. Istio’s Citadel component (and other components like Envoy sidecar proxies, Pilot and Mixer) manages all the parts and pieces of securing the services in a service mesh. Istio supports mutual TLS, which validates the identify of both the client and the server services.

The Citadel component in Istio manages the lifecycle of keys and certificates issued for services. When Istio establishes mutual TLS authentication, it uses these keys and certificates to exchange the identities of services. To establish a mutual TLS connection between two services, the envoy proxy on the client side establishes a mutual TLS handshake with the envoy proxy on the server side during which the client side envoy proxy verifies the identity of the server side and whether it is authorized to run the target service. When the identities of the services are verified, the mutual TLS connection is established and the client service sends communication through the client side proxy to the server side proxy and finally to the target service.

Mutual TLS authentication

Istio tunnels service-to-service communication through the client- and server-side PEPs, which are implemented as Envoy proxies. When a workload sends a request to another workload using mutual TLS authentication, the request is handled as follows:

  1. Istio re-routes the outbound traffic from a client to the client’s local sidecar Envoy.

  2. The client side Envoy starts a mutual TLS handshake with the server side Envoy. During the handshake, the client side Envoy also does a secure naming check to verify that the service account presented in the server certificate is authorized to run the target service.

  3. The client side Envoy and the server side Envoy establish a mutual TLS connection, and Istio forwards the traffic from the client side Envoy to the server side Envoy.

  4. After authorization, the server side Envoy forwards the traffic to the server service through local TCP connections.

First of all what it is kubernetes ?

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

The name Kubernetes originates from Greek, meaning helmsman or pilot. Google open-sourced the Kubernetes project in 2014. > Kubernetes combines over 15 years of Google’s experience running production workloads at scale with best-of-breed ideas and > practices from the community. Source — kubernetes.io

In this scenario we have an infrastructure deployed with two services in namespace called deprecated without side-car proxy (Enovy) injected and the network traffic between them is in fully plain HTTP.

cluster1.png

Automatic sidecar injection

we need to make default namespace enabled with automated enovy side-car injection so the deployed services within its can leverage istio capabilities.

Sidecars can be automatically added to applicable Kubernetes pods using a mutating webhook admission controller provided by Istio.When we set the istio-injection=enabled label on a namespace and the injection webhook is enabled, any new pods that are created in that namespace will automatically have a sidecar added to them.

kubectl label namespace default istio-injection=enabled
kubectl get namespace -L istio-injection

NAME              STATUS   AGE     ISTIO-INJECTION
default           Active   9d         enabled
deprecated        Active   6h20m   
istio-system      Active   9d        disabled
kube-node-lease   Active   9d      
kube-public       Active   9d      
kube-system       Active   9d

then we deploy the httpbin , sleep services in default namespace.

##################################################################################################
# httpbin service
##################################################################################################
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

kubectl apply -f httpbin/httpbin.yaml  
serviceaccount/httpbin created
service/httpbin created
deployment.apps/httpbin created

##################################################################################################
# Sleep service
##################################################################################################
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: governmentpaas/curl-ssl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

kubectl apply -f sleep/sleep.yaml

serviceaccount/sleep created
service/sleep created
deployment.apps/sleep created

cluster2.png

we can notice the difference between services between namespace deprecated and default , that is the deploying services in default namespace has been injected with container enovy-proxy side-car.

cluster2.png

we can verify setup by sending an HTTP request with curl from any sleep pod in the namespace default or deprecated to either httpbin.default, httpbin.deprecated. All requests should succeed with HTTP code 200.

For example, here is a command to check sleep.default to httpbin.deprecated reachability:

kubectl exec "$(kubectl get pod -l app=sleep -n default -o jsonpath={.items..metadata.name})" -c sleep -n default -- curl http://httpbin.deprecated:8000/ip -s -o /dev/null -w "%{http_code}\n"

200

one-liner command conveniently iterates through all reachability combinations:

for from in "deprecated" "default"; do for to in "deprecated" "default"; do kubectl exec "$(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name})" -c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; done; done

sleep.deprecated to httpbin.deprecated: 200
sleep.deprecated to httpbin.default: 200
sleep.default to httpbin.deprecated: 200
sleep.default to httpbin.default: 200

Services use authentication policies to define the kind of requests that a service receives, whether it is encrypted using mutual TLS or plain text. Istio uses these authentication policies, along with service identities and service name checks, to establish mutual TLS connection between services. The authentication policies and secure naming information is distributed to the Envoy proxies by the Pilot component. The Mixer component handles the authorization and auditing part of Istio security.

The process of enabling mutual TLS connections between services in Istio. We need to define a Policy object and DestinationRule object. we use a Policy object (also called an authentication policy) to define what kind of requests a service receives. A DestinationRule object applies to the traffic that is destined for a target service. It tells the client services whether to send encrypted traffic to the target service or to send plain-text requests.

To enable a mutual TLS connection between services, we need to define a Policy object and a DestinationRule object. However in the Istio 1.4+, a new automatic mutual TLS feature was added. If you turn on this setting, services are automatically enabled with mutual TLS, and you only need to specify a Policy object (a DestinationRule object is not needed). However, this post details how to define the Policy object and a DestinationRule object to enable mutual TLS between services.

Authentication policies (Peer Authentication)

Authentication policies apply to requests that a service receives. To specify client-side authentication rules in mutual TLS.

Peer authentication policies specify the mutual TLS mode Istio enforces on target workloads.The following modes are supported:

  • PERMISSIVE: Workloads accept both mutual TLS and plain text traffic. This mode is most useful during migrations when workloads without sidecar cannot use mutual TLS. Once workloads are migrated with sidecar injection, you should switch the mode to STRICT.
  • STRICT: Workloads only accept mutual TLS traffic.
  • DISABLE: Mutual TLS is disabled. From a security perspective, you shouldn’t use this mode unless you provide your own security solution. Verify there is no peer authentication policy in the system with the following command:
kubectl get peerauthentication --all-namespaces

No resources found.

Verify that there are no destination rules that apply on the example services. we can do this by checking the host: value of existing destination rules and make sure they do not match. For example:

kubectl get destinationrules.networking.istio.io --all-namespaces -o yaml | grep "host:"

Scopes

In Istio, there are three levels of granularity through which we can define our mTLS settings. For each service, Istio applies the narrowest matching policy. The order is: workload-specific, namespace-wide, mesh-wide.

  • Mesh-wide policy: A policy specified for the root namespace (Istio-namespace) without or with an empty selector field.
  • Namespace-wide policy: A policy specified for a non-root namespace without or with an empty selector field.
  • Workload-specific policy: a policy defined in the regular namespace, with non-empty selector field.

Globally enabling Istio mutual TLS in STRICT mode

While Istio automatically upgrades all traffic between the proxies and the workloads to mutual TLS, workloads can still receive plain text traffic. To prevent non-mutual TLS traffic for the whole mesh, set a mesh-wide peer authentication policy with the mutual TLS mode set to STRICT. The mesh-wide peer authentication policy should not have a selector and must be applied in the root namespace, for example:

kubectl apply -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF

peerauthentication.security.istio.io/default created

This peer authentication policy configures workloads to only accept requests encrypted with TLS. Since it doesn’t specify a value for the selector field, the policy applies to all workloads in the mesh.

Let’s try initiate communication between between sleep service that exist in deprecated name-space to httpbin service that exist in default name-space.

for from in "deprecated"; do for to in "default"; 
do kubectl exec "$(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name})" 
-c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; done; 
done

sleep.deprecated to httpbin.default: 000
command terminated with exit code 56
You now seeing the request has been failed due to the nature of services that haven't enovy-proxy side-car (sleep.deprecated) , can't communicate with with proxies services inside the mesh. (httpbin.default)

This is expected because mutual TLS is now strictly required and only workload accepts secure communication within https, but the workload without sidecar cannot comply.

on contrary if sleep.default initiate communication with httpbin.default , the connection will be successed.

for from in "default"; do for to in "default"; do kubectl exec "$(kubectl get pod -l 
app=sleep -n ${from} -o jsonpath={.items..metadata.name})" 
-c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; 
done; done

sleep.default to httpbin.default: 200
Remove global authentication policy and destination rules added in the session.

kubectl delete peerauthentication -n istio-system default

Ensure sniffing attack is mitigated

After migrating all clients traffic to be encrypted to Istio and injecting the Envoy sidecar in default name-space, we can initiate connections between services-workloads and ensure in the default namespace to only accept mutual TLS traffic.

for from in "deprecated" "default"; do for to in "default" "deprecated"; do kubectl 
exec "$(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name})" 
-c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; 
done; done

sleep.deprecated to httpbin.default: 000
command terminated with exit code 56
sleep.deprecated to httpbin.deprecated: 200
sleep.default to httpbin.default: 200
sleep.default to httpbin.deprecated: 200

Suppose attacker was compromised a workload inside default namespace and trying to sniffing on network packets between microservices network interface hoping for compromising critical data.

kubectl exec -n default "$(kubectl get pod -n default -lapp=httpbin -ojsonpath={.items..metadata.name})" -c istio-proxy -- sudo tcpdump dst port 80  -A

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 2324 bytes

You will see plain text and encrypted text in the output when requests are sent from sleep.deprecated.

Enable mutual TLS per namespace or workload

Namespace-wide policy

To change mutual TLS for all workloads within a particular namespace, use a namespace-wide policy. The specification of the policy is the same as for a mesh-wide policy, but you specify the namespace it applies to under metadata. For example, the following peer authentication policy enables strict mutual TLS for the default namespace:

kubectl apply -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "peer-authentication-policy-namespace-wide"
  namespace: "default"
spec:
  mtls:
    mode: STRICT
EOF
peerauthentication.security.istio.io/peer-authentication-policy-namespace-wide

As this policy is applied on workloads in namespace default only, you should see only request from client-without-sidecar (sleep.deprecated) to httpbin.default start to fail.

for from in "deprecated" "default"; do for to in "deprecated" "default"; do 
kubectl exec "$(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name})" 
-c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; 
done; done

sleep.deprecated to httpbin.deprecated: 200
sleep.deprecated to httpbin.default: 000
command terminated with exit code 56
sleep.default to httpbin.deprecated: 200
sleep.default to httpbin.default: 200

Mutual TLS per workload policy

To set a peer authentication policy for a specific workload, you must configure the selector section and specify the labels that match the desired workload. However, Istio cannot aggregate workload-level policies for outbound mutual TLS traffic to a service. Configure a destination rule to manage that behavior.

For example, the following peer authentication policy and destination rule enable strict mutual TLS for the httpbin.default workload:

kubectl apply  -f - <<EOF   
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "peer-authentication-workload-wide-httpbin"
  namespace: "default"
spec:
  selector:
    matchLabels:
      app: httpbin
  mtls:
    mode: STRICT
EOF
peerauthentication.security.istio.io/peer-authentication-workload-wide-httpbin created

And a destination rule:

cat <<EOF | kubectl apply  -f -
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "httpbin"
spec:
  host: "httpbin.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF
destinationrule.networking.istio.io/httpbin created

Again, run the probing command. As expected, request from sleep.deprecated to httpbin.default starts failing with the same reasons.

for from in "deprecated"; do for to in "default"; do kubectl exec "$(kubectl get pod -l app=sleep 
-n ${from} -o jsonpath={.items..metadata.name})" -c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s 
-o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; 
done; done

sleep.deprecated to httpbin.default: 000
command terminated with exit code 56

Summary:

This tutorial discussed how mutual TLS authentication works in Istio for service-to-service authentication. To enable mutual TLS in Istio, you need to define authentication policies for services at a workload-specific level, namespace level, or mesh-wide scope. An peer authentication policy defines what kind of traffic a service receives.

You also need to define destination rules. A DestinationRule object is an important part of Istio’s traffic routing functionality. It defines what happens to the traffic destined for a target service (of which defining the TLS settings for authentication is one part of it).

In addition to mutual TLS or transport authentication, Istio also supports origin authentication (also known as end-user authentication), where Istio authenticates the clients that make requests. Currently, Istio only supports origin authentication using JSON Web Tokens to authenticate client requests.