Anapoly AI Labs

Tag: local-LLM

  • Momentum is gathering

    Transperency label: human only

    When Randolph Lalonde released the first “broadcast” of his Spinward Fringe space opera in 2008, AI was entirely in the realm of science fiction. Lalonde imagined AIs as autonomous agents capable of making complex decisions, sometimes with unintended consequences. He portrayed AIs both as companions (personal assistants worn on the wrist, shipboard intelligences) and as adversaries (corporate-controlled AI enforcers). His AI characters often formed deep, personal bonds with humans, raising questions about trust, dependency, and emotional attachment to machines. Fast forward to where we are today, and these AI themes are remarkably relevant to current debates.

    The pace of AI development is accelerating, and we are on the cusp of seeing autonomous agents as an everyday reality. Alongside that, a capable, open source AI can now run on a desktop computer. If I set up my own personal or business AI on an Apple Mac Mini, newly-emerging software tools will let me access it securely from almost anywhere through my laptop, tablet or smartphone, and thus probably from the smartwatch on my wrist as well.

    Tailscale is one such tool. It is like a private, secure internet just for you or your business, allowing you or your team to access files, apps, or AI tools from anywhere, without exposing sensitive data online. It does this by creating a secure, peer-to-peer mesh network in which devices connect directly to each other rather than through a centralised server. This network operates over but is entirely private from the internet.

    If you are running an AI locally on your own server, you can use Tailscale to connect to it securely from your laptop, tablet or phone. No data leaves your control, and you can use the AI without having to put your data at risk. The direction of travel is clear; momentum is gathering and it will be impossible to stop. The best society can do is try to steer it.

  • Testing a local AI

    I am using Obsidian to build not a second brain, but a workspace for my brain: a space in which to think. The workspace is intended to become an ideas factory, designed as a knowledge network: a network of ideas at different stages of development.

    There is scope for AI technology to enhance my thinking in that space. For example, ideas in the knowledge network can be embedded into a vector database, roughly similar to how an AI organises knowledge. This allows similarities among widely scattered notes — the semantic connections — to be highlighted, and lets me search not just for specific words but also for ideas with related meaning. The Smart Connections plugin implements these useful capabilities.

    I can take this a step further by inviting an AI into my thinking space. Ollama lets me run a large language model on my laptop and connect it to my Obsidian vault through the Smart Connections plugin, so the AI can query, summarise, and respond directly from my notes. I downloaded the Mistral 7B Q4_K_M model for this purpose. Put simply, it’s a compressed version of the 7B model that runs faster and uses less memory, at the cost of some accuracy.

    I ran two tests. First, I pointed it to a short daily note and asked, “What is Claire’s new job?” After a pause of about half a minute, it gave me the correct answer.

    For the second test, I pointed it to a more substantial document and asked, “What is the purpose of Goal-Directed Context Management?” The reply took eight minutes, but the answer was very good:

    The purpose of Goal-Directed Context Management, as outlined in the provided document, is to manage a large language model (such as ChatGPT) effectively when it’s being used as a collaborator during the production or editing of knowledge-based products.

    This framework aims to configure and adapt the behavior of the AI to suit different roles over the course of work, while also providing scaffolding in the form of project and program management techniques to keep the AI focused and effective over the lifecycle of an information-based product.

    By aligning the AI’s evolving context with the purpose and structure of a project or program, Goal-Directed Context Management ensures that the AI always has the right information to contribute effectively to the current stage of work.

    The test confirmed three things: I could run an AI locally to keep my data private, the model gave accurate responses, and, although performance was slow on my three-year-old Dell XPS, it still worked. Watching my own laptop pull a thoughtful answer from my notes was oddly satisfying, even if I had time to make tea while it thought.