Last Wednesday we held our latest AI Tooling and Applications workshop at the Ballarat Hackerspace.

The speaker was Chris Berry from [Zai Node](https://www.zainode.com), who provide services for private AI hosting.

Many industries have strong privacy requirements and privately running your own AI gives assurance that your data stays where you expect it to stay. At the moment, the data centres for many big AI providers are overseas, often obscured by tooling, and it can be hard to get guarantees that limit where your data is sent.

Another reason for running AI locally is simple: it allows more experimentation, and you decide the limits. You can max out the server, run as many requests as you'd like, and experiment freely.

Chris first demonstrated something really interesting. To get started running your own LLMs, you just need your current computer or laptop. CPU-only is fine — any reasonably modern one (last five or so years) will work, and you can run smaller models and get some output. Practically there are some hiccups: the output is slower, the results are not as good, and you are limited to smaller models. However, it does work, and is useful in limited, easy scenarios where the final output quality does not need to be high.

This prompted lots of discussion around the limitations of models at this scale, and what hardware is needed to run bigger ones.

Next, Chris moved on to the purpose-built NVIDIA DGX Spark. This is a machine, approximately $7,000 AUD, built specifically to run LLMs locally. It is quite small — about the size of a small NUC (think about half an office desktop machine). However it is quite powerful, with 128GB of unified RAM, which can run very capable models using tools like Ollama, OpenCode, and OpenClaw.

This opens up much more capable models, about a generation behind the state-of-the-art commercial models. The results are fast, and if you give good validation criteria, the output can be quite good.

Chris showed two demos: the first a basic "categorise my invoices for me" example, and the second analysing a complex, obfuscated codebase to fix a bug that had been annoying him.

Thanks to Chris for his talk — he did a great job, and has lots of experience working with businesses to integrate AI into their workflows.

To hear about upcoming workshops, join up with us at BRAIN to receive notifications about new posts, events, and more. Also, if you are working with AI in Ballarat, or would like to be, we want to hear from you!

The link has been copied!