Connection to local LLM

[seb999]

I have a NVIDIA GPU H100 that run locally a LLM model. With that config we keep total control of our data.
Can jupyter-ai be interfaced with this LLM model, using the URL of the engine and the API key ?

[srdas]

@seb999 See if this documentation helps: https://jupyter-ai.readthedocs.io/en/v2/users/openrouter.html

Or, if it;s running a model supported by Ollama, see: https://jupyter-ai.readthedocs.io/en/v2/users/index.html#ollama-usage

[kentwait]

Is there a way to preconfigure the language and completion model without having to use the GUI?
I am using Jupyterhub and Docker to spawn lab instances and I would like to know I can use a config file or environment variables?
I read the configuration section https://jupyter-ai.readthedocs.io/en/latest/users/index.html#configuration but I am still prompted to configure instead of being able to see the chat interface in the sidebar.

[srdas]

@kentwait If you use the configuration options from the CLI, it should write to the config.json file in this location: ~/Library/Jupyter/jupyter-ai/ -- if this file exists then chat should not ask you to configure. It is possible that you need to enter at least one chat model to create the config.json file to start off and after that you can use the CLI to write to the config.json file.

[kentwait]

Thank you @srdas for suggesting the CLI instead of a config file. I can't quite make the config file work but the CLI worked well.

If anyone has the same problem as I did, I solved it by adding the following to my jupyterhub_config.py. Note that I am using DockerSpawner as the spawner for Jupyterlab user servers.

c.DockerSpawner.args = [
    "--AiExtension.default_language_model=ollama:deepseek-r1:8b",
]

c.DockerSpawner.environment = {
    "OLLAMA_HOST": "http://192.168.0.1:11434",  # change to your own Ollama location
}

[kentwait]

P.S. maybe don't use reasoning/thinking models like deepseek-r1 in Ollama with JupyterAI. It seems that chat and generate doesn't distinguish the "thinking" response to the actual response, breaking JupyterAI on out-of-the-box settings.

[srdas]

@kentwait Glad this worked, if you'd like, please submit a PR to update the user documentation in the configuration section with this additional functionality.

[dlqqq]

@seb999 It depends on the API provided by your LLM.

• If it's providing an OpenAI-like API, it can be used with the openai-chat provider while specifying a custom Base API URL.
• If it's providing the service through Ollama, it can be used with the ollama provider using a custom URL as well.

If you share the API you are using, we can provide more help. Very curious to see if somebody can use Jupyter AI w/ a H100. 👀