DEV Community: MrDoe

ClippyAI - Developing a Local AI Agent

MrDoe — Tue, 02 Jul 2024 22:50:48 +0000

Introduction

As a developer, I’ve always been passionate about creating tools that solve real-world problems. But there was one issue that consistently irked me: the never-ending stream of repetitive emails. Whether it was customer inquiries, tech support requests, or project updates, my inbox overflowed with similar questions day in and day out. It was like Groundhog Day, but with email threads.

The Annoyance Factor

Picture this: You’re sipping your morning coffee, already diving into some exciting coding challenges, and suddenly, ping! —another email lands in your inbox. It’s the same query you’ve answered a hundred times before. You sigh, type in your well-crafted response, and hit send. Rinse and repeat. It’s not just time-consuming; it’s soul-draining, because you totally lose focus of your coding work.

The Eureka Moment

One fateful afternoon, as I stared at my screen, contemplating the meaning of life (and another email), it hit me: Why not build an AI agent that assists you on these repetitive tasks?
Of course could I just use ChatGPT, but that's notowed at my company due to data privacy reasons. It's also very distracting and time-consuming to navigate to the website and copy and paste the source email, ask ChatGPT to write an answer and copy and paste the reply back to your email application.

The data protection part is easily solvable: With today's modern CPUs and GPUs it is possible to use Ollama and host the interference of an AI model of your choice locally at reasonable speed.

But how to integrate an agent for Ollama into your OS?

The DeepL Windows desktop app came into mind, where you just hit Ctrl+C twice and the app instantly translates the text you selected.

So my idea was to create a daemon who watches the clipboard for changes and then sends the content along with a task description to Ollama.

ClippyAI wouldn’t just suggest — I wanted it to take action. When I hit reply, it would automatically type out the response. Imagine the joy of watching ClippyAI do the grunt work while I sipped my coffee.

I chose the name ClippyAI as a mixture of "Clipboard" and "AI" and it was also inspired by the nostalgic Microsoft Office paperclip, which everyone hated these days.

Because I'm mostly a .NET developer, I used .NET 8 as foundation. I wanted to create a multi-platform application, because I'm using Windows at work and Linux at home, so I chose the Avalonia framework for this project.

After I realized, that the main idea was working well, I extended the ClippyAI's tasks to not just answering emails, but to also explain or translate the copied text or even to do some custom user defined tasks with it.

Key Features

Clipboard Integration: ClippyAI monitors your clipboard activity in real-time. Whenever you copy text, URLs, or other content, it automatically sends it to the Ollama AI model for analysis.
Context-Aware Responses: Modern AI models such as Llama3 or Gemma2 are able to consider the context of your task, if you give them enough input. Whether you’re drafting an email, writing code, or composing a document, ClippyAI can provide relevant and accurate responses.
Workflow Enhancement: By automating repetitive typing tasks, ClippyAI frees up your time and mental energy. Say goodbye to monotonous copy-paste routines!

Getting Started

Install Ollama from https://ollama.com.
Run ollama pull gemma2 on the command line to install the gemma2 AI model on your PC.
Clone the ClippyAI repository from https://github.com/MrDoe/ClippyAI and build and run it in Visual Studio or VS Code. If you are using VS Code, you need to build it twice, because during the first build the necessary resources files will be generated.
Check the configuration in App.config. The standard values should work fine for most installations.

Early Development Phase

While ClippyAI shows some promise, it’s essential to note that it’s still in its early development phase. As with any cutting-edge technology, there are risks involved. Here’s what you should be aware of:

Use it at your own risk: ClippyAI is experimental. It may occasionally produce unexpected results or errors. Always double-check the generated content before finalizing it.
Document safety: ClippyAI may unintentionally delete or overwrite your existing documents, if you are using the keyboard output and Auto-Mode. So be careful where to place your cursor!
Known issues: German umlauts and other special characters are currently not typed in keyboard mode under Linux/X11.
Developers Wanted: ClippyAI is an open-source project that is open for contributions from developers like you. If you want to join the development, clone the repo and submit a pull request!

Conclusion

ClippyAI is still a work in progress. It won’t win any Turing Awards yet, but it’s a little side project I want to extend further. So, the next time you receive a prompt reply from me, know that ClippyAI is doing its thing. And if it ever goes rogue, blame the coffee.

Disclaimer: ClippyAI may occasionally channel its inner HAL 9000. Use at your own risk.

Hosting Your Own AI Chatbot on Android Devices

MrDoe — Sat, 06 Apr 2024 23:27:50 +0000

Are you tired of handing over your personal data to big tech companies every time you interact with an AI assistant? Well, I've got good news - there's a way to run powerful language models right on your Android smartphone or tablet, and it all starts with llama.cpp.

In this in-depth tutorial, I'll walk you through the process of setting up llama.cpp on your Android device, so you can experience the freedom and customizability of local AI processing. No more relying on distant servers or worrying about your data being compromised. It's time to take back control and unlock the full potential of modern machine learning technology.

The Advantages of Running a Large Language Model (LLM) Locally

Before we dive into the technical details, let's explore what's the reason for running AI models locally on Android devices.

Firstly, it gives you complete control over your data. When you engage with a cloud-based AI assistant, your conversations, queries, and even personal information are sent to remote servers, where you have little to no visibility or control over how it's used or even sold to third party companies.

With llama.cpp, everything happens right on your device. Your interactions with the AI never leave your smartphone or tablet, ensuring your privacy remains intact. Plus, you can even use these local AI models in places where you don't have an internet connection or aren't allowed to access cloud-based AI services, like some workplaces.

But the benefits don't stop there. By running a local Ai, you also have the power to customize it. Instead of being limited to the pre-built models offered by big tech companies, you can hand-pick AI models that are tailored to your specific needs and interests. Or, if you own the right hardware and are experienced with AI models, you can even fine-tune the models yourself to create a truly personalized AI experience.

Getting Started with llama.cpp on Android

Alright, let's dive into setting up llama.cpp on your Android device.

Prerequisites

Before we begin, make sure your Android device meets the following requirements:

Android 8.0 or later
At least 6-8GB of RAM for optimal performance
A modern Snapdragon or Mediatek CPU with at least 4 cores
Enough storage space for the application and language model files (typically 1-8GB)

Step 1: Install F-Droid and Termux

First, you'll need to install the F-Droid app repository on your Android device. F-Droid is a great source for open-source software, and it's where we'll be getting the Termux terminal emulator.

Head over to the F-Droid website and follow the instructions to install the app. Once that's done, open F-Droid and search for Termux and install the latest version.
Please don't use Google Play Store to install Termux, as the version there is very outdated.

Setup Termux Repositories (optional)

If you change the termux repository server to one in your country you can gain faster download speeds when installing packages:

termux-change-repo

If you need help, check the Termux Wiki site.

Step 2: Set up the llama.cpp Environment

With Termux installed, it's time to get the llama.cpp project up and running. Start by opening the Termux app and install the following packages, which we'll need later for compiling llama.cpp:

pkg i clang wget git cmake

Now clone the llama.cpp git repository to your phone:

git clone https://github.com/ggerganov/llama.cpp.git

Next, we need to set up the Android NDK (Native Development Kit) to compile the llama.cpp project. Visit the Termux-NDK repository and download the latest NDK release. Extract the ZIP file, then set the NDK path in Termux:

unzip [NDK_ZIP_FILE].zip
export NDK=~/[EXTRACTED_NDK_PATH]

Step 3.1: Compile llama.cpp with Android NDK

With the NDK set up, you can now compile llama.cpp for your Android device. There are two options: with or without GPU acceleration. I recommend starting with the non-GPU version, as it's a bit simpler to set up.

mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-24 -DCMAKE_C_FLAGS=-march=native ..
make

If everything goes well, you should now have working llama.cpp binaries in the build folder of the project. You can now continue with downloading a model file (Step 4).

Step 3.2 Build llama.cpp with GPU Acceleration (optional)

Building llama.cpp with OpenCL and CLBlast support can increase the overall performance, but requires some additional steps:

Download necessary packages:

apt install ocl-icd opencl-headers opencl-clhpp clinfo libopenblas

Download CLBlast, compile it and copy clblast.h into the llama.cpp folder:

git clone https://github.com/CNugteren/CLBlast.git
cd CLBlast
cmake .
cmake --build . --config Release
mkdir install
cmake --install . --prefix ~/CLBlast/install
cp libclblast.so* $PREFIX/lib
cp ./include/clblast.h ../llama.cpp

Copy OpenBLAS files to llama.cpp:

cp /data/data/com.termux/files/usr/include/openblas/cblas.h .
cp /data/data/com.termux/files/usr/include/openblas/openblas_config.h .

Build llama.cpp with CLBlast:

cd ~/llama.cpp
mkdir build
cd build
cmake -DLLAMA_CLBLAST=ON -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-24 -DCMAKE_C_FLAGS=-march=native -DCLBlast_DIR=~/CLBlast/install/lib/cmake/CLBlast ..
cd ..
make

Add LD_LIBRARY_PATH under ~/.bashrc（Run program directly on physical GPU）：

echo "export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH:$PREFIX" >> ~/.bashrc

Check if GPU is available for OpenCL:

clinfo -l

If everything is working fine, e.g. for a Qualcomm Snapdragon SoC, it will display:

Platform #0: QUALCOMM Snapdragon(TM)
 `-- Device #0: QUALCOMM Adreno(TM)

Step 4: Download and Copy a Language Model

Finally, you'll need to download a compatible language model and copy it to the ~/llama.cpp/models directory. Head over to Hugging Face and search for a GGUF-formatted model that fits within your device's available RAM. I'd recommend starting with TinyLlama-1.1B.

Once you've downloaded the model file, use the

termux-setup-storage

command in Termux to grant access to your device's shared storage. Then, move the model file to the llama.cpp models directory:

mv ~/storage/downloads/model_name.gguf ~/llama.cpp/models

Step 5: Running llama.cpp

With the llama.cpp environment set up and a language model in place, you're ready to start interacting with your very own local AI assistant. I recommend to run the llama.cpp web server:

cd llama.cpp
./server -m models/[YourModelName].gguf -t [#threads]

Replace #threads with the number of cores of your Android device minus 1, otherwise it may become unresponsive.

And then access the AI chatbot locally by opening http://localhost:8080 in your mobile browser.

Alternatively, you can run the llama.cpp chat directly in Termux:

./main -m models/[YourModelName].gguf --color -inst

Conclusion

While performance will vary based on your device's hardware capabilities, even mid-range phones should be able to run llama.cpp reasonably well as long as you choose small enough models that fit into your device's memory. High-end devices will, of course, be able to take fuller advantage of the model's capabilities.