The impact of a chatbot working as an assistant in a course for supporting student learning and engagement

What Is Jupyter Notebook? Data Analysis Made Easier

Jupyter Notebook combines live code, graphics, visualizations, and text in shareable notebooks that run in a web browser

At some point, we all need to show our work. Most programming work is shared either as raw source code or as a compiled executable. The source code provides complete information, but in a way that's more "tell" than "show." The executable shows us what the software does, but even when shipped with the source code it can be difficult to grasp exactly how it works.

Imagine being able to view the code and execute it in the same UI, so that you could make changes to the code and view the results of those changes instantly, in real time? That's just what Jupyter Notebook offers. 

Jupyter Notebook was created to make it easier to show one's programming work, and to let others join in. Jupyter Notebook allows you to combine code, comments, multimedia, and visualizations in an interactive document — called a notebook, naturally — that can be shared, re-used, and re-worked. 

And because Jupyter Notebook runs via a web browser, the notebook itself could be hosted on your local machine or on a remote server. 

IDG

A Jupyter Notebook with text, code, and a visualization. Altering the code and re-running it live in the browser will change the output, and the new results can be saved to a new notebook.

Jupyter Notebook benefits

Originally developed for data science applications written in Python, R, and Julia, Jupyter Notebook is useful in all kinds of ways for all kinds of projects:

Data visualizations. Most people have their first exposure to Jupyter Notebook by way of a data visualization, a shared notebook that includes a rendering of some data set as a graphic. Jupyter Notebook lets you author visualizations, but also share them and allow interactive changes to the shared code and data set.

Code sharing. Cloud services like GitHub and Pastebin provide ways to share code, but they're largely non-interactive. With a Jupyter Notebook, you can view code, execute it, and display the results directly in your web browser.

Live interactions with code. Jupyter Notebook code isn't static; it can be edited and re-run incrementally in real time, with feedback provided directly in the browser. Notebooks can also embed user controls (e.G., sliders or text input fields) that can be used as input sources for code.

Documenting code samples. If you have a piece of code and you want to explain line-by-line how it works, with live feedback all along the way, you could embed it in a Jupyter Notebook. Best of all, the code will remain fully functional—you can add interactivity along with the explanation, showing and telling at the same time.

Jupyter Notebook components

Jupyter Notebooks can include several kinds of ingredients, each organized into discrete blocks:

Text and HTML. Plain text, or text annotated in the Markdown syntax to generate HTML, can be inserted into the document at any point. CSS styling can also be included inline or added to the template used to generate the notebook.

Code and output. The code in Jupyter Notebook notebooks is typically Python code, although you may add support in your Jupyter environment for other languages such as R or Julia. The results of executed code appear immediately after the code blocks, and the code blocks can be executed and re-executed in any order you like, as often as you like. 

Visualizations. Graphics and charts can be generated from code, by way of modules like Matplotlib, Plotly, or Bokeh. Like output, these visualizations appear inline next to the code that generates them. However, code can also be configured to write them out to external files if needed.

Multimedia. Because Jupyter Notebook is built on web technology, it can display all the types of multimedia supported in a web page. You can include them in a notebook as HTML elements, or you can generate them programmatically by way of the IPython.Display module.

Data. Data can be provided in a separate file alongside the .Ipynb file that constitutes a Jupyter Notebook notebook, or it can be imported programmatically—for instance, by including code in the notebook to download the data from a public Internet repository or to access it via a database connection.

Jupyter Notebook use cases

The most common use cases for Jupyter Notebook are data science, mathematics, and other research projects that involve visualizations of data or formulas. Apart from those, though, there are plenty of other use cases:

Sharing a visualization, with or without interactivity. People often share the results of a data visualization as a static image, but that's useful only up to a point. By sharing a Jupyter notebook, you allow your target audience to dive in and play around. They can gain a thorough understanding of the data, interactively.

Documenting a process with code. Many programmers who blog about their programming experiences write up their posts in a Jupyter notebook. Others can download their notebook and recreate the exercise.

Live documentation for a library or module. Most documentation for Python modules is static; a Jupyter notebook can be used as an interactive sandbox for learning how a module works. Any Python module that runs well in a notebook interface (essentially, anything that writes to stdout as part of its behavior) is a good candidate for this.

Sharing code and data generally. All you need to do to share a Jupyter notebook and its associated data files is pack it up into an archive.

JupyterLab

A next-generation user interface for Jupyter Notebook, called JupyterLab, is now available and considered ready for production use.

As explained in the blog post announcing general availability, JupyterLab is more malleable than a conventional Jupyter Notebook, allowing users to drag-and-drop cells within and between notebooks and to arrange the workspace into separate tabs and subsections. Code can run directly from text files as well as Jupyter Notebook files, and many common file formats for both code and data can be rendered with live previews.

JupyterLab can also be customized with extensions to add support for new file formats, to enrich the interface, or to provide any number of other additional features, enabling a much broader range of notebook applications than Jupyter Notebook. The long-term plan is to replace the current Jupyter Notebook interface with JupyterLab, but only after JupyterLab has proved sufficiently stable and reliable.

IDG

JupyterLab is a totally re-engineered front-end for Jupyter Notebook. In addition to providing greater flexibility in arranging the workspace, JupyterLab is customizable through extensions. 

Jupyter Notebook limitations

As powerful and useful as Jupyter Notebook can be, it does have some limitations that need to be taken into account.

Notebooks aren't self-contained. This is the single biggest drawback of using Jupyter Notebook: Notebooks require the Jupyter runtime, along with any libraries you plan on using. A few strategies exist for creating self-contained Jupyter Notebooks, but none of them is officially supported. You're best off distributing notebooks to people who already have infrastructure in place to run them, or don't mind the setup (by way of Anaconda, for instance).

Session state cannot be saved easily. The state of any code running in a Jupyter notebook cannot be preserved and restored with Jupyter Notebook's default toolset. Every time you load the notebook, you will need to re-run the code in it to restore its state.

No interactive debugging or other IDE features. Jupyter Notebook is not a full-blown development environment for Python. Many of the features you would expect to find in an IDE—e.G., interactive debugging, code completion, and module management—aren't available there.

Data Science Notebooks Get Real: JupyterLab Releases To Users

These days, the industry would have you believe that data and analytics is all being done in the service of AI. And, given that, there's a lot of orientation toward data scientists' seemingly favorite tool for working with data: the notebook. The biggies in this space include Apache Zeppelin, and more prominently, Jupyter (formerly iPython). If you're on the Databricks platform, you'll end up using their own brand of notebook, but even those are Jupyter-compatible.

One way to think of a notebook is a place to put executable code and annotate it. It might be more accurate, though, to say that notebooks are a place to put a bunch of annotations and decorate them with code. But regardless of how snarky you may want to be, notebooks let you intersperse rich text in Markdown format with code, in any number of languages, run it in place, and view its output, in textual, tabular or graphical format. If you finesse things enough, you can even cobble together a poor man's dashboard with these assets.

Beyond presentationThe problem with notebooks is that they're much better for experimental data science work than they are for production data engineering work. That's my own opinion, of course. But I stand by it. Notebooks are more about presentation than they are about development, and they lack many of the niceties of a real integrated development environment (IDE) like Eclipse, PyCharm, Visual Studio or, for that matter, RStudio.

But it doesn't have to be that way. Jupyter notebooks started out as a tool for Python code exclusively and Anaconda, which produces the leading Python distribution, has been hard at work enhancing the Jupyter notebook facility into something much more IDE-like, with JupyterLab. Earlier this week, the long-touted tool was released in a form its developers saw as "ready for users." Given that modest but significant milestone, I decided to download JupyterLab, fire it up and check it out. In general, I am impressed.

JupyterLab, with an R language-based notebook and several of its visualizations, displayed in a single layout.

Coding productivityJupyterLab is a superset of Jupyter itself. As such, anything you can do in a Jupyter notebook, you can do in JupyterLab. But you can do a bunch more.

To begin with, syntax completion using the Tab key, and object tool tip inspection, using Shift-Tab, are present, just as they are in the conventional notebook experience. But JupyterLab enhances these facilities compared to their standalone Jupyter notebook implementations, with additional information about the types of matched items in tab completion and additional information about objects in the tool tips. Having that sort of contextual assistance and help reduces the amount of context switching developers have to do to figure things out.

Developers can also work in a more imperative programming mode, by running their code interactively inside of "consoles" and not just notebooks. Consoles are live sessions with the Jupyter "kernels" (the language interpreters that actually execute code, behind the notebooks ) so they can run their code in a more programmer-oriented environment before inserting it into the primarily text- and visualization-based document manifested in a notebook.

I spy more than iPy(nb)JupyterLab goes well beyond notebooks though, allowing developers to open up files in a variety of formats, containing data or other assets they may use or produce from their code. This includes source code files in supported languages, as well as files in plain text, CSV and other delimited text formats, JSON, various image, and even in PDF formats. The viewers and editors that help out here include a full-featured text editor, an image viewer, a tabular data viewer, a tree-view JSON viewer, and viewers for Vega, Vega-Lite and VDOM files.

Sometimes, multiple viewers are applicable to certain files. For example, JSON and CSV files can be opened up as plain text. But a a tree viewer can also be used in the case of the former, and the table viewer can be used in the case of the latter. Jupyter allows such files to be opened up in multiple editors simultaneously, and keeps the multiple views synchronized, so that edits made in one are rendered in the other(s).

Double visionThis multi-view paradigm works for notebooks as well. For example, by right-clicking on a rendered visual inside a notebook and selecting "Create New View for Output" from the resulting context menu, the visuals can be displayed in their own view, side-by-side with other such visuals and the conventional view of the notebook that produced them.

This is done with just a little drag and drop finesse, allowing multiple notebook, output and file viewers to be displayed in horizontally- or vertically-proximate regions, separated by splitter bars, with each region accommodating potentially multiple tabbed documents, and each being individually scrollable. Once you get good at creating such layouts, you can even set things up in a dashboard-like layout, as shown in the screen capture above.

Single document; multiple langaugesRecognizing that all this eye candy can sometimes be distracting, JupyterLab allows users to toggle between such tiled layouts and a single document view, wherein the active document takes over the entire editing region of the JupyterLab browser tab or window.

JupyterLab is supported in the Chrome, Firefox and Safari browsers. In my admittedly limited testing, it also worked quite well in Windows 10's Edge browser, with the exception of the single document view feature, which caused serious rendering anomalies -- and which I confirmed did not exist in Chrome.

As with Jupyter itself, JupyterLab accommodates notebooks in multiple languages, provided the kernels for those languages are installed. A Launcher feature (shown above) then allows users to select a language for the creation of a new notebook or console. The Python kernel is installed by default and, in my own installation, I managed to install kernels for R and Node/JavaScript as well. The figure below shows a Node-based notebook along with a separate view of one of its rendered visuals.

A Node.Js-based Jupyter notebook and one of its visuals (a map of us airport locations)

Adapted from Mike Bostock's block (https://gist.Github.Com/mbostock/4408297).

We don't need no stinkin' installers...Or do we?Installation of JupyterLab can be complex, owing mostly to its cobbling together of multiple technologies. Installing Anaconda Python will get the core of Jupyter on your system, and a subsequent install step will install support for the R kernel as well. From there you can download JupyterLab itself, and then separately install other languages and the Jupyter kernels as well. That's what I did to get the Node.Js kernel installed, and it took a couple of steps to get it right. Bringing down sample notebooks took additional searching and effort as did the installation of modules necessary to support the code within them.

This can be difficult but it's not rocket science (or even data science): I am a former developer and still pretty hands-on with tools, so I have some instinct for these things. But my dev skills are lapsed, and I'm no tools ninja. If I can do it, then likely so can anyone with the prerequisite skills and interest to use JupyterLab in the first place. Still, patience and some spare time are required, and a master installer would remove a lot of friction.

Cloud adoption dividendBut for many Jupyter users, installation has been a non-issue, as the facility is embedded in numerous cloud services, including Microsoft HDInsight, Amazon SageMaker, and Google Cloud Datalab. Will the various services and products that embed Jupyter now also embed JupyterLab? I hope so, as this will raise the bar in the data engineering experience on all these platforms.

Web browser-based developer tools are innovative, but they can also be limiting. Jupyter has been a good exemplar of this conundrum. But JupyterLab helps transcend the limitations, while retaining the innovation and convenience. Let's hope its adoption ion the ecosystem is brisk.

Drops Of Jupyter Notebooks: How To Keep Notes In The ... - Hackaday

Our digital world is so much more interactive than the paper one it has been replacing. That becomes very obvious in the features of Jupyter Notebooks. The point is to make your data beautiful, organized, interactive, and shareable. And you can do all of this with just a bit of simple coding.

We already leveraged computer power by moving from paper spreadsheets to digital spreadsheets, but they are limited. One thing I've seen over and over again — and occasionally been guilty of myself — is spreadsheet abuse. That is, using a spreadsheet program to do something I probably ought to write a program to do. For those times that you want something quick but want something more than a spreadsheet, you should check out Jupyter Notebooks. The system is most commonly associated with Python, but it isn't Python-specific. There are over 100 languages supported — many community-developed. You can even install a C++ interpreter backend for it. Because of the client/server architecture, it is very simple to share notebooks with other users.

You can — in theory — use Jupyter for anything you could use Python for. In practice, it seems to get a lot of workout with people analyzing large data sets, doing machine learning, and similar tasks.

The Good: Simple, Powerful, Extensible

The idea is simple. Think of a Markdown-enabled web page that can connect to a backend (a kernel, in Jupyter-speak). The backend can run on your machine or remotely and will support some kind of language — often Python. The document has cells that line up vertically (like a single wide spreadsheet column). For example, here's a simple notebook I created to explain how a bunch of sine waves add up to a square wave:

You can try it live in your browser or download it from GitHub. You can see that you can get "live" graphical output, along with text and other media. In fact, I'm not taking good advantage of the formatting, but you can do anything you can do with Markdown in the text cells.

The code is pretty standard Python. For example, here's one of the cells:

a0=amplitude*np.Sin(time); plot.Plot(time,a0); plot.Title("Fundamental"); plot.Xlabel("Time"); plot.Grid(True,which="both");

Further down in the document you'll see that you can also deploy widgets. For example, using a slider to set parameters. We'll come back to that topic in a bit. In addition to widgets, you can get extensions that let you layout cells in a grid. These are often used to create dashboards like the one below, for example. In fact, there are lots of extensions, for lots of different purposes.

The Bad: Support for Non-Python Languages

Non-Python languages are tricky to use with Jupyter. I tried using the C++ interpreter and found it a bit hard to get going. Some of that is because C++ isn't happy with being run incrementally — Redefining things, for example, makes it unhappy. If you want C++ or Fortran or any of the other myriad options, they may or may not work well. They may or may not be able to use libraries that a lot of Python notebooks will employ. Don't get me wrong. I haven't found any that don't work at all, but sometimes it is inconvenient or difficult compared to using the Python kernel.

The other thing that strikes me as odd is that the tasks notebooks seem best for is not always what they are most used for. If you think about it, the notebooks are really an exercise in literate programming. However, it seems to me that most of the notebooks are just sent around as quick web applications. You can share a static image of a page, of course. You can also share read-only versions. GitHub, for example, will render a notebook on display. There's also Binder which will let you share an interactive version.

Joel Bennet does what he calls literate devops — which is similar to literate programming using Jupyter and — of all things — Powershell in the video below.

Jupyter is not magic. It facilitates rapidly building little Python applications that have a very particular web interface. There are probably projects it isn't suitable for. Not every job requires a hammer. You can save yourself some grief, though, by doing a little research on best practices before you start anything substantial. But you should constantly be asking yourself if any tool is the right tool for a given job and not just using the same thing for everything.

The Ugly: Python Package Management

I find it taxing that the system relies on Python. I don't have much against the language itself (although my personal preference is for whitespace to not be meaningful). However, ensuring Python has everything it needs for a given notebook tends to be super painful. If you plan on distributing, this becomes another layer of issues in ensuring everyone has the right packages. On Binder, you can provide a requirements.Txt file that tells it what things you need to import, so that's workable but an extra step.

The bulletproof way to install the program locally is with Anaconda which — of course — creates a totally different Python environment than your normal Python environment. Yes, I know about virtualenv. And pip. Of course, my Linux system has a package manager, too, and it has versions of Jupyter and all the Python libraries. But everyone wants their own package manager to rule my system and I have no idea what to do about your system.

Once you get it installed, it is fine. And if you get it working on Binder, you should be good since it builds each user a new Docker container. However, if you really plan on distributing complex notebooks, the installation across multiple platforms and Python versions could pose a risk.

Widgets Make It Interactive

If you look at the last two code cells of my example document (from above), you'll see that I use a slider widget to let you interactively adjust the equations and the graph. That's just one of the various widgets available.

If you aren't picky, the system will build widgets for a function for you. You don't always get the control of things like ranges and steps, but for many functions, you can get a reasonable UI by just making a simple call to interact, or including it with the function:

@interact(x=True, y=1.0) def g(x, y): return (x, y)

That will produce a checkbox for x and a slider for y. You'll get default values, but in many cases, that'll be acceptable.

The Latest

I've been talking about traditional notebooks, but the next generation interface rolled out last year. Known as JupyterLab, it allows you to use other tools like editors in a tabbed-interface. Binder supports the new interface if you want to give it a quick spin.

You can continue working with traditional notebooks using the new interface, so we expect to see increased adoption of JupyterLab over time.

So…

Should you use Jupyter? That's like asking if you should use a saw. If you are cutting wood, yes! If you are trying to join two pieces of plastic, no. Jupyter definitely fits a niche — and a niche that many of us writing math- and data-intensive software work within. The fact that you can distribute it easily and even interface with hardware makes it attractive for projects where you want something quick but powerful.

Although some of the languages other than Python are second-class citizens, there are many choices and you can work around any limitations. So even if you aren't a Python guru, you'll still want to add this power notebook system to your toolbox.

---