Working for a while with Jupyter, both for ML processes and Data Engineering ETL’s, we came up with interactive solution to make our ETL’s stable, reliable and maintainable — read more about how we build our interactive platform in the following story:

https://medium.com/@benm_23166/tech-story-spark-jupyter-on-k8s-64b9d43a37ba

After having all infrastructure ready to work - we started writing our ETL pipelines in Jupyter and schedule it in Google Cloud Composer. In our ETL’s we came across multiple problems — we wanted to resolved them with Jupyter Magics infrastructure. Our goal was to create more modular, maintainable and flexible source of code. …

Ben Mizrahi

Bugs always exists between the keyboard to the screen :)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store