Accessing a service in a different namespace from a single ingress in Kubernetes

IPv4. Its rare when its public, and annoying when its private. So we try and conserve this precious resource. One of the things that makes it complex is Kubernetes namespaces. A Kubernetes Ingress controller is not namespace aware (you can’t have a shared Ingress that has services in multiple namespaces). Or can you?

What if I told you you could install a single Ingress (and cert-manager etc) and then have a service in each namespace served by it? Would you rejoice over saving a few $100/mo on public IP rental in ‘the cloud’?

Lets dig in. Imagine we have 3 namespaces with interesting services. ‘foo’, ‘bar’ and ‘kube-system’ (which has our dashboard).

Lets assume we have ‘kibana’ running in kube-system. We want to expose this to the ‘public internet’. Likely we would also use oauth2 proxy here to sign in, but I’ll ignore auth for now.  We are going to use a new service (synthetic) which lives in the default namespace alongside our Ingress controller as ‘glue’. Its kind of like a DNS CNAME.

First we install a single global ingress. Lets use helm:

helm install stable/nginx-ingress --name ingress --set controller.service.externalTrafficPolicy=Local --set rbac.create=true

Wait for the LoadBalancer to get a public IP, register it in DNS. You can either use a wildcard (*.something.MYDOMAIN.CA) or register each service, your call. All will use the same IP.

(To avoid complicating this, I’ll show the cert-manager etc at the end, but its optional, we just need the next step with the Ingress + Service)

Once we have installed the below yaml we can now browse https://kibana.MYDOMAIN.CA/ and we are there. Repeat for the other services. Done! We have a single public IP

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: 'true'
    nginx.ingress.kubernetes.io/tls-acme: 'true'
spec:
  tls:
  - hosts:
    - kibana.MYDOMAIN.CA
    secretName: tls-secret-kibana
  rules:
  - host: kibana.MYDOMAIN.CA
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601
---
kind: Service
apiVersion: v1
metadata:
  name: kibana
  namespace: default
spec:
  type: ExternalName
  externalName: kibana.kube-system.svc.cluster.local
  ports:
  - port: 5601

Now, to complete this and show with SSL certificates. You don’t need this, above is all you need to expose the service, but why not do it on TLS at the same time? Its free!.

helm install stable/cert-manager --namespace kube-system --set ingressShim.defaultIssuerName=letsencrypt --set ingressShim.defaultIssuerKind=ClusterIssuer --name cert

cat << EOF | kubectl -n kube-system apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: don@agilicus.com
    privateKeySecretRef:
      name: letsencrypt
    http01: {}
EOF

Posted

in

by

Comments

13 Responses to “Accessing a service in a different namespace from a single ingress in Kubernetes”

  1. Holms

    I don’t understand, you have service in default namespace, but you wrote in the beginning that it should be in kube-system namespace. Your ExternalName says it’s kube-system, and namespace of the service is default..? How that can even work? Currently I have my service in mynamespace, and ingress in default, and it doesn’t see each other.

    1. db

      the service acts like a dns name inside kubernetes. And, in this case, uses an externalname to reference.
      it uses the fully-namespaced version.

      kibana.kube-system.svc.cluster.local

      will resolve from all namespaces.

  2. Javier Palacios

    Sorry, but it is very clear that you’re accessing a service in the ingress namespace:
    “`
    kind: Service
    metadata:
    namespace: default
    spec:
    externalName: kibana.kube-system.svc.cluster.local
    “`
    You are just proxying to an external service that by chance happens to be in another namespace in the same cluster. Sad that the misleading title showed me first in google results.

    1. db

      no.
      the services are in namespace A, B
      the ingress is namespace C. This technique allows you to access, from a single ingress controller, services in other namespaces.
      this is what i was trying to achieve.

      that is a `bridge` service, it exists solely to make a DNS entry to find the ClusterIP in the destination namespace… it has no code or pod behind it.
      its just like an entry in /etc/hosts

  3. Andrew Pickin

    This ‘bridge service’ – we call in ‘external service’, but yours is probably a better name – is extremely useful when using multiple namespaces.
    It’s surprising how badly documented this feature is.

  4. Timothy

    Just a pity that the service cannot be referenced directly in a “normal for k8s” way of `..svc` in the ingress namespace rather than having to create a service which simply points to a service in a different namespace

    1. db

      you can if you use istio instead of nginx

  5. Thanks for that, it helped a lot! Just a remark, native GKE ingress, doesn’t work with ExternalName, so you need to use nginx.

    1. db

      good to know!
      you can also use istio which simplifies this (but is overall more complex for sure).

  6. Andy

    I don’t get this. It does not work for me.
    And the main thing, you did not specify the namespace of Ingress in manifest. Why? What namespace you created it in? If in default then i did the same and it does not work for me as supposed to be, i’m getting eror 503 and it’s fixable only i create Ingress in the same namespace as my service in.

    1. db

      its been a while, perhaps something has changed.

      but yes, the key here, is 1 in default, shared by all in other namespace.

  7. Mario Stefanutti

    We are also interested in the final solution, if there was one

    1. db

      so the method i posted does work.
      you create a service w/ an external in one of the namespaces.

      we are using something similar for istio[-system] now that we’ve switched from nginx ingress.

Leave a Reply to Wojciech Sznapka Cancel reply

Your email address will not be published. Required fields are marked *