Sphinx documentation build fails on Windows 10 when using jupyter-sphinx extension

[niketagrawal]

Description

Sphinx documentation fails to build on Windows 10 when jupyter-sphinx extension is used to embed python code in the documentation.

$ sphinx-build . _build
Running Sphinx v3.3.1
WARNING: while setting up extension jupyter_sphinx.execute: `jupyter-sphinx` was initialized with the `jupyter_sphinx.execute` sub-module. Replace this with `jupyter_sphinx`. Initializing with `jupyter_sphinx.execute` will be removed in version 0.3
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
executing index... [100%] index

Extension error:
Notebook execution failed (exception: )
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\nbclient\client.py", line 365, in _async_cleanup_kernel
    await ensure_async(self.kc.stop_channels())
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\jupyter_client\client.py", line 125, in stop_channels
    if self.shell_channel.is_alive():
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\jupyter_client\client.py", line 151, in shell_channel
    socket = self.connect_shell(identity=self.session.bsession)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\jupyter_client\connect.py", line 561, in connect_shell
    return self._create_connected_socket('shell', identity=identity)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\jupyter_client\connect.py", line 545, in _create_connected_socket
    sock = self.context.socket(socket_type)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\sugar\context.py", line 226, in socket
    s = self._socket_class(self, socket_type, **kwargs)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\_future.py", line 144, in __init__
    self._init_io_state()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\asyncio.py", line 56, in _init_io_state
    self.io_loop.add_reader(self._fd, lambda: self._handle_events(0, 0))
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\asyncio\events.py", line 501, in add_reader
    raise NotImplementedError
NotImplementedError
Exception ignored in: <function Socket.__del__ at 0x000001ADCB24BAF0>
Traceback (most recent call last):
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\sugar\socket.py", line 68, in __del__
    self.close()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\_future.py", line 160, in close
    self._clear_io_state()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\asyncio.py", line 63, in _clear_io_state
    self.io_loop.remove_reader(self._fd)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\asyncio\events.py", line 504, in remove_reader
    raise NotImplementedError
NotImplementedError:
Exception ignored in: <function Socket.__del__ at 0x000001ADCB24BAF0>
Traceback (most recent call last):
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\sugar\socket.py", line 68, in __del__
    self.close()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\_future.py", line 160, in close
    self._clear_io_state()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\zmq\asyncio.py", line 63, in _clear_io_state
    self.io_loop.remove_reader(self._fd)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\asyncio\events.py", line 504, in remove_reader
    raise NotImplementedError
NotImplementedError:

Setup:

Windows 10
conda 4.8.3
Python 3.8.6
sphinx 3.3.1
jupyter-sphinx 0.3.2

Steps to reproduce

• Create a conda virtual environment, activate it and install sphinx and jupyter-sphinx

$ conda create --name=jupyter-sphinx-test python=3.8
$ conda activate jupyter-sphinx-test
$ conda install -c anaconda sphinx
$ conda install jupyter_sphinx

• Create a new directory named docs and generate the basic sphinx documentation template inside it.

$ mkdir docs
$ cd docs
$ sphinx-quickstart

• Append the index.rst file generated from the above step with the below python code segment using the jupyter-execute directive.

.. jupyter-execute::

  print('hello ')

• Enable the jupyter-sphinx extension in conf.py

extensions = [
    'jupyter_sphinx.execute',
]

Note: The build log pasted above throws a warning that suggests to replace jupyter_sphinx.execute with jupyter_sphinx. Documentation was rebuilt after making the change but the issue still exists.

• Build the documentation.

$ sphinx-build . _build

[akhmerov]

The linter warning is about registering a directive with docuitls, and I don't think it's related to the build error, can you please open a separate issue for that one?

[niketagrawal]

Ok, will do.

[akhmerov]

The first problem seems to be due to limitations of nbclient on windows. Can you please install the latest nbconvert, and try running nbconvert --execute test.ipynb (where test.ipynb is any notebook with a code cell)?

If you observe the same failure, then please open an issue in the https://github.com/jupyter/nbclient repository.

I'm unfortunately unable to do that myself due to the lack of a windows environment.

[niketagrawal]

It appears nbconvert is already installed to latest version in my case

$ conda install nbconvert
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda

# All requested packages already installed.

Created a test.ipynb with a print statement in it. The command nbconvert --execute test.ipynb wasn't recognized. Is the command correct?

nbconvert --execute test.ipynb
bash: nbconvert: command not found

[akhmerov]

Sorry, it should be jupyter nbconvert --execute test.ipynb.

[niketagrawal]

Ok, I did think of jupyter as the probable prefix to the command but wanted to be sure.
The command fails to execute with the below error.
An output format needs to be specified as pointed out.

$ jupyter nbconvert --execute test.ipynb .
Traceback (most recent call last):
  File "~\Anaconda3\envs\jupyter-sphinx-test\Scripts\jupyter-nbconvert-script.py", line 10, in <module>
    sys.exit(main())
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\jupyter_core\application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\traitlets\config\application.py", line 845, in launch_instance
    app.start()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\nbconvert\nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "~\Anaconda3\envs\jupyter-sphinx-test\lib\site-packages\nbconvert\nbconvertapp.py", line 512, in convert_notebooks
    raise ValueError(
ValueError: Please specify an output format with '--to <format>'.
The following formats are available: ['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'script', 'slides', 'webpdf']

[akhmerov]

Sorry again (should really not write these from memory) jupyter nbconvert --execute test.ipynb --to ipynb

[niketagrawal]

No problem :)
This time it does succeed. As a result, a new notebook with the same content as test.ipynb is created.

$ jupyter nbconvert --execute test.ipynb --to ipynb
[NbConvertApp] Converting notebook test.ipynb to ipynb
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] Writing 946 bytes to test.nbconvert.ipynb

[stephprobst]

This seems to be related to this bug.

The following workaround worked for me. Just put in the top of your conf.py file.

import sys
import asyncio

if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())