I've been working on my own fork of a set of Rust language wrappers around the famous llama.cpp project. This blog post covers my motivations, what I've added and a sample project to get readers started.

I've finally released the AI chat software I've been working on. I'll try to write a blog post about it at some point, but until then, you can find more information at the github repo.

Sentient_Core is a local llm AI application so all of the text inference is done on your machine. Optionally, it can support calling out to koboldcpp through it's API so it is possible to do the heavy compute on a different machine - or use more up to date model formats because rustformer's llm project is still trying to merge a current version of ggml after being behind for months.

If you have any questions or comments, feel free to ask away!

Okay! I finally got a 'killer feature' prototyped in my terminal based AI chat application ... vector embeddings! I got it generating them from the chat log and searching for semantic relevance appropriately. Took a bit to sort this out because of the technology stack I'm using... But now it works.

Life's gonna be rough for the next few days, but late next week I hope to actually write some more useful blog posts containing the stuff I've learned. I can't wait!

🤩

I finally finished writing this one up. My previous post here was about 1 minute after I got the finetuned model working for the first time, so it's taken me until now to put it all together. I'm still not super happy with the writing in this one, but I'm kinda 'over' the writing process right now ...

If you spot any errors, or have any questions, feel free to reply.

Holy crap! I've finally done it. I've generated a dataset of conversations (all locally), cleaned them up and then finetuned it on open-llama-7b (just to test) and IT WORKED! AHHHHH! happy noises

Okay I gotta go to sleep now. I have to get up for work in less than five hours. I'll be cleaning up everything, commiting code and doing the write up this week.

I've been trying to make my own qlora finetunes for a little while now, but progress has been slow. For a long while I couldn't get anything to come out right besides the examples, so here's a blog post on the progress I made.

I wrote some notes down into a blog entry after getting Lemmy and Caddy to play with each other from separate docker-compose files. Nothing advanced here, but if you're like me and Docker only pops in and out of projects occasionally, this might be a helpful guide for what changes to expect.

This is my step-by-step guide on how to replicate fine tuning of the example datasets using axolotl.

Last I checked, the bitsandbytes library copy was still needed and open-llama-3b was still problematic for quantizing, but hopefully those issues are solved at some point.

What I didn't know when I first wrote the post was that it was possible to load the finetuned LoRA file in a frontend like text-generation-webui. I have since updated the text to account for that. There are performance side-effects of just loading the qlora adapter in the webui besides just the penalty to load time. This should show how fast text inference was with little context in tokens/p while using the transformers library and source model in f16 or quantized 8-bit & 4-bit and how fast I can run a merged q4_0 quantization.

![](https://lemmy.animal-machine.com/pictrs/image/301cc1b7-2dd3-4fcd-ad24-69a

I did it again and started a blog. Even further, I spun up a Lemmy instance so that I could control my online account in the fediverse. While doing so it dawned on me that it'd be the perfect way to host discussions on my blog posts!

While I'm pulling out of a variety of online spaces, I'm moving to solidify my identity in the fediverse spaces. So I'm @[email protected] here ... and currently @tbogdala over at mastodon.gamedev.place. The problem is that I have a lot of interests and I just feel weird tooting about all of them on a 'gamedev' instance. Sigh... I might host my own too, later, I don't know. All of this is a little stream-of-consciousness but that should cover the intro bit.