Spark & Jupyter on K8S — Tech Story

Introduction:

First let talk about us…

So with that context — let’s start:

Jupyter & Spark Architecture — Connecting the dots
profileList:
- display_name: "Default Small Image"
kubespawner_override:
cpu_limit: .5
cpu_guarantee: .5
mem_limit: "2G"
mem_guarantee: "512M"
node_selector:
cloud.google.com/gke-nodepool: small-users-pool
- display_name: "Strong Compute Image"
kubespawner_override:
cpu_limit: 8
cpu_guarantee: 8
mem_limit: "16G"
mem_guarantee: "512M"
node_selector:
cloud.google.com/gke-nodepool: users-strong-pool-8c-16
- display_name: "Super Strong Image"
kubespawner_override:
cpu_limit: 8
cpu_guarantee: 8
mem_limit: "32G"
mem_guarantee: "512M"
node_selector:
cloud.google.com/gke-nodepool: users-strong-pool-8c-32g
"spark.dynamicAllocation.executorIdleTimeout":"60s",
"spark.dynamicAllocation.executorAllocationRatio": "0.2",
"spark.dynamicAllocation.schedulerBacklogTimeout":"1s",
"spark.dynamicAllocation.shuffleTracking.timeout": "10s" ,
"spark.dynamicAllocation.shuffleTracking.enabled": "true",
"spark.dynamicAllocation.minExecutors": "2",
"spark.dynamicAllocation.maxExecutors": "1000"
RUN  apt-get -y -o Acquire::Check-Valid-Until=false update && apt-get install --yes --no-install-recommends \ 
ca-certificates \
curl \
gnupg \
&& echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" \
| tee /etc/apt/sources.list.d/gcsfuse.list \
&& curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - \
&& apt-get -y -o Acquire::Check-Valid-Until=false update \
&& apt-get install --yes gcsfuse \
&& apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
gcsfuse -o nonempty --key-file /path/to/service/account/secret.json [bucket name] /etc/google/
"spark.driver.extraClassPath": "/etc/google/jars/*",
"spark.executor.extraClassPath": "/etc/google/jars/*"

Next things needs to be handled:

DaemonSet — can help you receive faster loading of images

Bugs always exists between the keyboard to the screen :)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store