Install Single-Node for Test

nix

# server
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && sudo install minikube-linux-amd64 /usr/local/bin/minikube

minikube start

# client, via snap
sudo snap install kubectl --classic
# or, client via apt
sudo apt-get update && sudo apt-get install -y apt-transport-https && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update && \
sudo apt-get install -y kubectl

[ref]

win

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')) # install choco

choco install minikube -y  # "server", which can be replaced by k8s in Docker-Desktop.
choco install kubernetes-cli  -y  # "client" / "controller".

minikube start --vm-driver hyperv --hyperv-virtual-switch "Default Switch"  # needs admin. it will use virtualbox by default, we want to use hyperv. 

// Can also create a new V-switch ref.

Note: Docker-Desktop is easier to use (just by one click), but then we do not have minikube command.

config

There is a config file on the kube master, in home root dir as /home/some_username/config . Copy and put into local machine (client) as /home/$USER/.kube/config.

checks & tests

Tip: most docker ... cmds can be changed to kubectl ....

kubectl config get-contexts
kubectl get po
kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.10 --port=8080
kubectl get po
kubectl expose deployment hello-minikube --type=NodePort

web ui

Run anywhere which has kubectl available:

kubectl proxy

Open url:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
Note 1: ssh tunnel may be needed.
Note 2: changing the running machine of "kubectl proxy" may require re-auth by the config file (or token).

Spark on K8s

container images

(for spark 2.2, see FAQ below.)
First, build or find a suitable container image (run the cmd in spark distribution folder top level.)

Tip: binding (py/r) versions are good to tag, as the workers should use the same version with the driver.

Scala/Java

./bin/docker-image-tool.sh -t spark-scala build

Py:

./bin/docker-image-tool.sh -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile -t spark-py368 build

R:

./bin/docker-image-tool.sh -p ./kubernetes/dockerfiles/spark/bindings/R/Dockerfile -t spark-r351 build

I have problems to build images using docker-image-tool.sh, but used the spark Dockerfile to build some images here:

# py
sunnybingome/spark8s:pyspark240py368

# r
sunnybingome/spark8s:official-spark240-r351

# jar 
# jar can run using either the py or r image above.

k8s master api address

kubectl cluster-info | grep 'Kubernetes master'
# out e.g.: Kubernetes master is running at https://192.1.1.1:443

RBAC (Rule-Based Access Control)

FAQ of Spark on K8s:

Problem: Error: forbidden: User "system:serviceaccount:serv-account-1:default" cannot get resource "pods" in API group "" in the namespace "ns-1".
Reason: RBAC does not allow driver containers to creat workers yet (cluster mode).
Solution:

# config RBAC
kubectl create serviceaccount <service-account-name> # service-account itself (e.g. myspark, spark1) is across spacenames.
kubectl create clusterrolebinding <role-name> --clusterrole=edit --serviceaccount=<namespace>:<service-account-name> --namespace=<namespace>

# then run spark-submit cluster mode with the RBAC config:
--conf spark.kubernetes.authenticate.driver.serviceAccountName=<service-account-name>

Ref: official doc

Problem: Exception: Python in worker has different version X.x than that in driver X.x, PySpark cannot run with different minor versions
Reason: Spark checks the major version number and the first minor version number to match the driver with its workers. When using spark-on-k8s, the mis-match can only happen when using client-mode (see also spark.
Solution: Use cluster mode, or use pipenv --python=3.x; in driver to ensure the same py version on the driver and the workers.

Problem: Spark version 2.2 is old and it is hard to find container images.
Solution: See Spark 2.2 doc (py 2.7.13). (since Spark 2.3, only Dockerfiles are provided).

Problem: having problems to run the official spark integration test with k8s.
Reason: (not sure).
Solution: I use my own simple script. (TODO: github url)

Problem: Spark 2.2 cannot recognize k8s as the master. (ERROR: ... must be yarn ... etc.)
Reason: Spark 2.2 default distribution does not include k8s support.
Solution: download the k8s-supported Spark 2.2 distribuion, or wget directly from https://is.gd/OVewKa. Then the official example can be run directly by copy-paste (w/ correct k8s IP & corrected ns).