Enabling HTTPS on your on-premise kubeflow (with paid cert and letsencrypt)

You can enable HTTPS on your kubeflow server using Let’s encrypt with this walkthrough.

Test Environment

• Provisioning machine - Ubuntu 18.04
• Node machines - Ubuntu 20.04
• deepops - 22.08
• kubernetes - 1.23.7
• kubeflow - 1.6.1
• MetalLB - 0.13.7

Walkthrough

0. Prerequisition

• Kubeflow installed on Kubernetes

1. Setup HTTPS

We should access kubeflow with HTTPS because many of our kubeflow apps use Secure Cookies. You should expose an external IP, point your domain name to IP, then configure HTTPS for the domain.

First, Install the MetalLB for exposing your service with external IP.

# install MetalLB (https://metallb.universe.tf/installation/)
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl apply -f - -n kube-system
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml

# MetalLb layer2 configuration
kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250 # Change to yours
EOF
kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system
spec:
  ipAddressPools:
  - first-pool
EOF

# change the type of istio-ingressgateway service.(from NodePort to LoadBalancer)
kubectl edit svc -n istio-system istio-ingressgateway
'''
type: LoadBalancer # from ClusterIP
'''

Second, you have to point a domain name to your external IP address(or forwarded public IP).

Then, you can enable HTTPS on your kubeflow server using letsencrypt. If you have your own paid certificate, skip this step.

# add cluster issuer
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-istio
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: example@example.com #change your email
    privateKeySecretRef:
      name: letsencrypt-prod-istio
    solvers:
    - http01:
        ingress:
          class: istio
EOF

# add /.well-known to SKIP_AUTH_URI for acme challenge and rebuild the auth service
vi common/oidc-authservice/base/params.env
'''
SKIP_AUTH_URI=/dex /.well-known
'''
kustomize build common/oidc-authservice/base | kubectl apply -f -

# create certificate
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  commonName: www.example.com # Change to yours
  dnsNames:
    - www.example.com # Change to yours
  issuerRef:
    name: letsencrypt-prod-istio
    kind: ClusterIssuer
EOF

With paid certificate, you may have the key file and cert file. You can make k8s secret object with them. If you are using letsencrypt, the Certificate object will be used as this secret. So you don’t need to make another secret object.

kubectl create -n istio-system secret tls kubeflow-ingressgateway-certs --key=server.key --cert=cert.pem

Finally, you can enable HTTPS on your server by adding configuration to your ingressgateway

# set force redirection from http to https
kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway
'''
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs
'''

99. FAQ

• I have only pfx file of my paid certificate.

You can get key and cert from your pfx file.

# Export the private key from the pfx file
openssl pkcs12 -in myCert.pfx -nocerts -out key.pem
# You should type your password you created when exporting the cert

# Remove the password and Format  the key to RSA
openssl rsa -in key.pem -out server.key
# This will prompt you for a pem passphrase. This would be the passphrase you used above.

# Export the certificate file from the pfx file
openssl pkcs12 -in myCert.pfx -clcerts -nokeys -out cert.pem
# Enter the password you created when exporting the cert
'''