Local AI:what really stays in-houseand what does not

Local AI means running your AI application so that control over data and systems stays inside your company rather than being handed entirely to a cloud service. In practice, the label covers two very different set-ups. In the first, the language model itself runs entirely on your server and no text leaves your network. In the second, the application, your files and your documents sit with you, while the model that writes the answers is called as a service from an AI provider. Both have their place. Which one fits depends on the data you handle, the quality you need and who will look after the technology.

Nikolai Schöbel and Jeremias Burger

Co-founders of Scalableloops GmbH. Nikolai Schöbel leads online marketing and AI strategy, Jeremias Burger the AI architecture. Both build AI systems and train teams on them in their own agency work.

How do you run an AI model entirely on your own server?

You need a so-called open-weight model, one whose weights are published and which you are allowed to run on your own hardware. The choice has grown considerably. On 5 August 2025, OpenAI released two open-weight models, gpt-oss-120b and gpt-oss-20b, under the Apache 2.0 licence; according to TechCrunch, they are the company’s first open language models since GPT-2. Open-weight models are also available from DeepSeek and from Alibaba with its Qwen series, among others.

Hardware requirements depend on model size. OpenAI states that the larger model fits on a single data-centre GPU with 80 GB of memory, while the smaller one runs within 16 GB of memory. That puts the smaller model within reach of a well-equipped workstation; the larger one needs server hardware. If a whole team uses the model at the same time, you need extra headroom for parallel requests.

MIT Technology Review points out that organisations such as hospitals, law firms and governments may need models they can run locally for data security reasons. The benefit is clear: prompts never leave your network, no external provider processes your data, and you are not dependent on the pricing or product decisions of a cloud service.

How does AI on your own server work with a cloud model?

The second option separates data storage from computing power. Application, knowledge base, templates and histories live on your server. Only when a task requires it does the relevant snippet go to a provider’s language model, and the answer comes back. You keep control of your data store while still using the most capable models.

What matters is the terms under which the provider processes the data it receives, and consumer and business offerings differ considerably. One example: according to its documentation, OpenAI does not use data sent through its API to train its models unless the customer explicitly opts in, a rule that has applied since 1 March 2023. Since February 2025, OpenAI business customers can also choose processing in Europe for eligible API endpoints.

Such commitments do not replace your own review. But they explain why option B is the pragmatic middle ground for many companies: the knowledge of the business sits centrally in-house, the model can be swapped, and you decide task by task which data goes where. Tasks involving particularly sensitive data can additionally be routed to a local model.

Is local AI automatically GDPR-compliant?

No, but it makes many things easier. On 6 May 2024, the Datenschutzkonferenz, the body of Germany’s data protection authorities, published guidance on AI and data protection. It distinguishes closed systems, where processing takes place in a contained environment and users keep control of inputs and outputs, from open systems that a provider runs as a cloud service, for example. From a data protection perspective, it considers technically closed systems preferable.

If you use an AI application from an external provider, the guidance says there is often a processor relationship under Art. 28 GDPR, which requires a corresponding agreement with the provider. You should also check whether inputs and outputs are used for training and exclude this where possible. If data goes to countries outside the EU, the rules in Chapter V of the GDPR apply.

For the USA, an adequacy decision by the European Commission has been in place since 10 July 2023: the EU-US Data Privacy Framework. It permits transfers to recipients certified under the framework. The EU General Court upheld the decision on 3 September 2025, and an appeal was lodged with the Court of Justice on 31 October 2025. If you rely on transfers to the USA, keep an eye on the case; option A and processing in Europe make you less dependent on it.

Some duties apply wherever the model runs: a legal basis for processing, informing the people concerned, honouring their rights and, where a high risk is likely, a data protection impact assessment under Art. 35 GDPR. The German authorities state that this will frequently be the case when AI applications are used. On top of that, the AI Act’s AI literacy obligation has applied since 2 February 2025, which we explain in our article on the AI literacy obligation under Article 4.

How do we bring local AI into your business?

Our AI agent system follows option B and keeps option A open. The system and your files sit on your server. You decide which language model does the work: a high-quality cloud model, a local open-weight model or both, depending on the task. That means you are not tied to any single AI provider.

The system gets to know your business, works with the software you already use and prepares work that a person reviews and approves before anything goes out. Which data may go to which model is something we clarify with you and your data protection officer before we set anything up.

To help your team use the new capabilities safely, we accompany the rollout with AI training tailored to the tasks in your business. If you use AI in customer contact, for example as a chatbot, also read our article on AI disclosure for chatbots.