The Devops Case For Jupyter Note(Run)books

 With all the excitement watching NASA successfully land perseverance on Mars I thought if we can land a rover on Mars why can't we go to Jupyter.....Notebooks that is. The Jupyter project has been around since 2014 and used by Data Scientists and the scientific community for quite some time. Before we can dive into how we can use these notebooks for DevOps we first need an understanding of what they are and can provide.

What is a Jupyter Notebook?

A Jupyter Notebook in a nutshell is a special json file (*.ipjnb) that when rendered by a Jupyter Server renders as a web application that allows the user to blend content, data, equations, and executable code to create interactive notebooks that can be saved, shared, and exported into a number of formats. While most notebooks use the python programming language, Jupyter supports over 100 languages through its kernels.

Each notebook is made up of one or more cells. Each cell can hold markdown, code or raw content. Code cells are cells that are going to perform a task, using cell magic this could be to render html, write a file to disk or execute code written in any language you have a kernel installed for.

Code cells can even be used to install additional libraries and/or plugins to add additional functionality to your Jupyter notebooks and server.

When the example code cell below is run the html output is included on the notebook below the cell. Sometimes it may not make sense to show both the code cell (input) and the output, for this you can add the meta tag 'hide-input' which will collapse the code cell but still render its output.

Pretty cool stuff right. But I'm not a data scientist and don't really imagine myself rendering complex equations to study gravitational theories, so how does this all apply to a DevOps use case? 

DevOps Runbooks

Runbooks have been around in one form or another for a long time, Systems Admins have been using them for years as a way to share information between engineers in  the admin group and even other IT personnel. The format and medium for runbooks has changed over the years and can vary depending on your project and clients needs.

A standard runbook will contain at a minimum the following information:

• Systems architecture and design diagrams
• Technical Requirements for the system 
• List of key personnel supporting the system
• Troubleshooting workflows
• List of common issues
• Who to contact during outages and emergencies
• List of all changes to the system

These runbooks are typically found on internal wiki or systems like Confluence the format and content may vary depending on who created them or updated them last.

Jupyter supports markdown as well so using a Jupyter notebook for static information is definitely possible and will give us the rich text output we need.  Jupyter notebooks go even further than just static content they allow us to have markdown and executable code on the same page. 

What this means is instead of having a wiki page that gives you step-by-step commands, SQL queries, etc that you hope are up-to-date and were copied correctly you can execute these directly from the notebook and evaluate the output. No more fumbling for credentials, ssh sessions, dealing with incomplete or incorrect commands.

So lets pick apart a typical wiki runbook and see how we could replace each bit. The first requirement is these notebooks need to be shareable. So far all we have talked about is jupyter notebooks which are typically single-user and can be run locally, what we need is a central server for running and sharing.

JupyterHub & JupyterLab

JupyterHub gives us a central hub to spawn single-user notebook servers on-demand so we don't need to tie up our local machines with running processes. The Hub also makes it easy for us to share these notebooks so we don't have to email them back-and-forth to each other in order to share them. 

The hub provides an authentication mechanism and proxy server so we can add additional users and share our notebooks between users. 

By itself, JupyterHub solves some of the core issues but still leaves off some functionality, such as how do we best manage our secrets. We can run JupyterHub and JupyterLab together to lay the groundwork for providing the functionality that we will need.

JupyterLab is the new enhanced web interface for jupyter notebooks, it provides a better interface where we can browse, create, and view multiple notebooks in a tabbed environment, open a terminal for quick access to a command line, install additional plugins and language kernels, etc.

In this post I won't go in to the how to install JupyterHub and JupyterLab, but if you are interested in setting up a quick POC or just want to try it out there are a few different options:

• Grab a Docker version of their stack
• Setup your own server using "The Littlest JupyterHub" approach. 
• Checkout a running demo version using Binder

Shh! It's a Secret

A secret can be a password, token, identity, or any other piece of data that should not be known by everyone except those that need to know and should not be checked in to code repositories. 

In terms of DevOps, this could be a token used to deploy code to a client's production servers. There may be only one token that the team supporting the project should use. So the use of the token should be shared among the team.

Typical jupyter notebooks use environment variables to store and use secrets; while this prevents those secrets from being checked-in to a repo it does have some drawbacks. Environment variables are not the best use if we are to share our notebooks with other members of the team. The two major drawbacks to this approach are:

• Each team member would have to create their own environment variables to hold the data.
• Any time an environment value changed we would need to restart our jupyter server.

If your organization uses Vault, AWS Secrets Manager or any other secrets manager implementation you can easily implement a python method to retrieve the secrets from your secrets manager and use them in your notebooks. 

If your not lucky enough to have access to a secrets manager another option would be to use IPython Secrets. The IPython Secrets library makes it easy to store and retrieve secrets from within your notebook. The library uses the python keyring project which utilizes the system's keystore or credential locker or a number of different backends to encrypt and store them. 

The nice thing about this approach is you can start using the system's default keystore and later migrate the backend to a 3rd party provided backend or a custom one.

Below is an example of how this could be done using IPython Secrets:

    from ipython_secrets import *
    import requests
    
    deploy_token = get_secret('DEPLOY_TOKEN')
    
    headers = {
     'PRIVATE-TOKEN': deploy_token
    }
    
    response = requests.request("GET",
      "https://gitlab.yourdomain.com/api/v4/projects/16/deployments",
       headers=headers)
  

The first call to 'get_secret()' will prompt for a password to the keyring, then if the key is not in use will ask us for the secret to store after entering the secret it will be removed from the front-end view. If the secret we are requesting already exists, after asking us for the keyring password, the secret is decrypted and assigned to our variable for use.

Head Stuck In the Clouds

One of the many DevOps tasks is managing your cloud instances whether they are running on GCP, AWS, Azure, or an on-prem Kubernetes cluster.  Jupyter Notebooks give us flexibility to run shell scripts, access the command line and run python code. 

With this type of flexibility we can install, configure and run the aws cli from a notebook or use the AWS SDK (boto3) to access AWS resources. In the example below we use a notebook to perform each command line entry that you would normally perform to download and install the AWS Cli. 

  The awscli is installed to the system running the notebook so we only need to perform those steps once then we can use it in our notebook code cells just like we would from any command line or script.

While the examples above are using awscli this could have easily been terraform, ansible or any other utility or SDK you need access to.

Descriptive Runbooks

So far we have seen examples of using different notebook cells for markdown and code and you may be thinking pretty cool but how does this make for a better runbook?

Below is an example notebook that shows using both markdown and code together to describe the solution and the code that will perform the task. 

In the above notebook there are four cells, two that contain markdown and two that contain shell scripts.

In this example we use the markdown cells to describe what each script will do and if there are any special considerations the engineer should be aware of prior to running the script.

The DevOps engineer can adjust the variables in the scripts in the notebook if needed or run it as-is right from the notebook.

After the engineer has run the script the output is displayed under the code cell and is saved with the notebook by default. While you can always clear the output(s) from the notebook, this does allow the engineer or senior engineer to go back and examine the output to determine if there are further issues.

Some sections of a runbook may contain sql or api calls to gather data to create reports or dashboards.

Dashboards

Some dashboards may be created and managed by the devops team but the main consumer of that dashboard may not be as technical so we don't always want the code cells to be available, some dashboards should only contain output.

Voila allows us to create interactive dashboards from notebooks but removes the code cells from the users.  Voila can be installed as a standalone app or as part of our Jupyter server. 

One example for this could be a report that displays all commits to a particular release branch so we can better track what is being deployed. Release notes or reports are an important part of every release so we can track what is changing or if issues arise later we can use the dashboard to see what all changed to the code and which developers committed that code.

Below is an example of this where we gather the commits from a particular branch for the date/time period of the sprint and display the output in an html table.

The output of the code cell displays all commits, who committed them, when, the url to that commit, etc. While it may be a bit much for release notes where Jira story IDs and short descriptions may be more suited for, this gives a better picture of everything that changed and following the links we can examine that code a bit more.

Conclusion

After reading through some of these examples it is easy to see the power and flexibility that Jupyter can bring to our runbooks. While these are just a few examples there is much more that could be done and many more extensions available or we could write our own if we can't find what we need.