Introduction

Last week I met with a couple of good friends I haven’t seen in ages. They both work in IT and we worked together in the past at different companies. We were talking about the impact the AI and LLM is having in our work life and the discussion moved into how many enterprises, including theirs, are blocking access to ChatGPT, Copilot and friends.

Why?

In a few words: data privacy and security. It is probably harmless to access your favourite chat platform and ask for help with a shell script or more information on an error log. However, things can be more difficult to contain once you start using agents that can read your code (think of Copilot, Roo-Code, etc), for example, and share it with the LLM models for analysis.

Then there are the accidents, like copying and pasting credentials or God forbid, customers’ data.

Remember, anything you send to the AI models is used to train the next models. So if you want everyone to know your password is asdf1234, be careful not to type it in!

What’s the solution?

Most if not all of the large platforms provide an “enterprise” edition designed for enterprises that want to take advantage of AI without compromising their data. If you’re adopting one of these solutions, be very careful and read the small print to ensure your data is not going to be used for training and that the models do not learn from your chat inputs.

The second option is to run your own in-house LLM. If you haven’t tried it this is a lot easier than you probably thought. There are some caveats:

• The responses are not as fast as with commercial models
• The best models in terms of quality of responses remain commercial
• You will need to host the AI service in servers with GPU, and this can be expensive

How?

The solution below is based on using Ollama. It’s very easy to install and run. Get to the Ollama website and download the server for your platform.

Next, create a service to ensure ollama runs in the background. For example, you can add this to /etc/systemd/system/ollama.service

[Unit]
Description=Ollama Service
After=network.target

[Service]
Type=simple
Environment=OLLAMA_HOST=127.0.0.1:11434
ExecStart=/opt/ollama/bin/ollama serve
Restart=always
RestartSec=10

# Optional security measures
PrivateTmp=true
ProtectSystem=full
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

If you prefer to run Ollama in docker, ensure you have the nvidia toolkit installed and run it with

docker run --rm -d \  -v ollama:/root/.ollama -p 11434:11434 \  --gpus=all \  --name ollama ollama/ollama

You will need to download a LLM model to use. Try different ones and see which ones work better for you. I’m currently using qwen2.5:32b-instruct

ollama pull qwen2.5:32b-instruct

Unless you’re happy with the command line, you probably want a web UI for the chat interactions. Open WebUI is brilliant, I especially adore their code formatting and syntax highlighting.

It is simple to run in a container:

docker run -d -p 3000:8080 \  --add-host=host.docker.internal:host-gateway \  -v open-webui:/app/backend/data --name open-webui \  --restart always -e WEBUI_AUTH=true \  ghcr.io/open-webui/open-webui:main

Final words

We all love cool tools, and AI is revolutionising the way we work. However, it’s crucial not to lose sight of what’s important: security and your data. Be very careful with what you share. If you have concerns, consider using an enterprise edition that provides isolation and ensures your data won’t be used to train future LLMs. If this option isn’t feasible for you, try running the chat models locally, as demonstrated in this example.

Contact our team for a free consultation to discuss how we can tailor our approach to your specific needs and challenges.

I, for one, welcome our new robot overlords

‍