Running your own AI models: when and how?

You choose private AI per task. For a lot of work an outside AI provider is fine. For sensitive data, work that runs day and night or experiments where you ask the same question a thousand times, a model on your own machines often fits better.

We are Studio Northstar, an agentic product studio in The Hague. We build products with ARCS, our agentic platform, and run part of our work on our own servers. Below are the options, the tools and what you give up, with Northstar One as an example.

What is private AI?

Private AI means running AI models on machines and networks you control, so prompts and company data do not go to an outside AI provider. That can be a laptop, a server in your office or hardware in a data centre that runs only for you.

A few related terms mean something else. A local LLM is a language model on your own device, the smallest form of private AI. A self-hosted LLM or on-premise LLM runs on a server you operate. European AI says where a provider is based or where its servers are; your data still goes to that provider. Sovereign AI is about who has control over models, hardware and data. There is no fixed definition of it.

Your own model does not exempt you from the rules. The Dutch Data Protection Authority⁠↗ writes that the GDPR is technology-neutral and that you establish in advance whether you can comply, even for a pilot. That applies on your own server just as much. The AI Act⁠↗ sorts systems by risk, and that risk depends on what you use AI for, not on where the model runs.

When does data have to stay in-house?

Data has to stay in-house when a law, a contract or your own policy says it may not go to a third party. You decide that per type of data.

• Agreements with clients. A non-disclosure agreement or data processing agreement can forbid data from going to another party, including an AI provider.
• Sensitive personal data. Medical data or information about employees calls for extra care.
• Trade secrets. Source code, pricing models, plans nobody is supposed to see yet.
• Your own policy. Some organisations don’t want internal documents processed outside the building, even where that would be allowed.

There is also a reason that has nothing to do with privacy: cost. For tasks that run continuously, a per-token rate can add up fast. An experiment where you ask a question a thousand times is only feasible if costs stay under control. On your own hardware you pay for the machine and its upkeep, not per question.

Local, own server or private cloud: the options

You have three options: a model on a laptop or workstation, your own server in the office or a data centre (on-premise), or a private cloud, where you rent hardware in a shielded environment that runs only for you. An outside AI provider is listed for comparison.

A model on your laptop is a good way to find out what a local model can do. For a team you need a server that serves several people or systems at once, with access control and logging.

Which models and tools?

To run an LLM locally or on your own server you need two things: an open model whose weights you may download, and software that loads the model and makes it reachable. Ollama, LM Studio and vLLM are three well-known tools. We name them to show what exists, not as advice for your situation.

Ollama

Ollama⁠↗ runs models on macOS, Windows and Linux, also as a Docker image, and is open source under the MIT licence. It has a REST API, so your own software can call the model. The project page names gpt-oss, Qwen, Gemma and DeepSeek, among others. According to the Ollama FAQ⁠↗, it only accepts connections from the machine itself by default (127.0.0.1, port 11434).

Note that Ollama also offers cloud models that run on Ollama’s servers. According to Ollama, prompts are then processed but not stored, logged or used for training. For private AI you want that switched off. The FAQ names the setting OLLAMA_NO_CLOUD=1 for it.

LM Studio

LM Studio⁠↗ is a desktop app for downloading and running models such as Qwen, Mistral, Gemma and gpt-oss. According to its documentation, LM Studio can work entirely offline once you have the model files. It is mainly useful for testing on a workstation what a model can do.

vLLM

vLLM⁠↗ is a library for running models and serving them, aimed at servers. It has an OpenAI-compatible API server, so software that already calls such an API can talk to your own server the same way. vLLM supports, among others, NVIDIA and AMD GPUs and several kinds of CPU.

What hardware do you need?

That mostly depends on the size of the model, how much text you send at once and how many people or systems use it at the same time. The first question is always whether the model fits in memory.

The tools say something about it themselves. LM Studio⁠↗ requires an Apple Silicon chip on a Mac, recommends 16 GB of RAM or more and notes that language models can consume a lot of RAM. On an 8 GB Mac, LM Studio says to stick to smaller models and modest context sizes. On Windows it recommends at least 16 GB of RAM and 4 GB of dedicated VRAM. Ollama⁠↗ loads a model onto a single GPU if it fits and otherwise spreads it across several GPUs. With ollama ps you can see whether a model runs in GPU memory, in regular RAM or split across both.

Those are requirements for using an app on one workstation. For a server that serves a team, you choose the hardware after a test with your own tasks: which model reaches the quality you need, and how many requests does it have to handle at once? Start with the task and the model, then work out what the machine has to do.

What do you give up?

You usually give something up in three areas: quality, maintenance and the way you pay.

• Quality. The models you can run yourself are not the same as the models that are only available through an outside provider. For classifying, summarising or searching your own documents an open model can be enough, for other tasks it isn’t. You only know once you test it on your own work.
• Maintenance. Updates, security, monitoring and what happens when a machine fails are on you or on whoever runs it for you. New models don’t arrive by themselves.
• Cost. With a provider you pay per use. With your own hardware you pay up front for the machines and then for power, space and upkeep, even when they sit idle. That can pay off for work that runs a lot and for long. For the occasional question it usually doesn’t.

How do you choose where each task runs?

Choose per task based on three things: how sensitive the data is, what quality you need and how often the task runs. Record that choice, so you can show later why something ran where it did.

That is how we work too. Our frequently asked questions say: “We choose per task where the work runs and record that choice. That can be on our own machines or with a model from an outside AI provider.” In practice we choose between three places: on the device itself, on our own hardware or at a large AI provider.

An example from the other side: Jev by TypeSafe is a decision model you call through OpenRouter, so at an outside provider. For short choices without sensitive data that can fit well. For a case file with medical data it doesn’t.

How we do it: Northstar One

Northstar One is a small number of our own servers on which we run AI models. We use it for sensitive information and long-running tasks, and for experiments that would get too expensive at an outside AI provider charging per token.

The machines can only be reached through a private network and are not publicly accessible from the internet. Right now we use them for internal work: classification, embeddings, evaluation and long-running tasks for our products. Before an improvement goes into a product, we test it on fixed test sets and have models review each other’s work.

Within an engagement we can also use Northstar One for client work, with a record of where the work runs. The next step is private installations for organisations that have to process their information within an agreed environment. More is on the Northstar One page.

Not sure whether a task can go to a provider or has to stay in-house? Get in touch and describe the task and the data. We will work out together where the work is best run. How to get started with agents is in AI agents for business.