72b version consumes about 6gb of ram on my laptop, cant play games with it loaded because it takes forever to reply. The lower versions work better, but then they arent as smart. Seems its mostly GPU power and memory that are important.

Damn it would have been nice to have something that can just pull chat from twitch/youtube and then read it out or respond..But it looks like the new llama models can do this off even smartphones or raspberry pi's. So I might end up hosting my own on vps or from raspberry pi then connecting it to chat. Having this app be able to be used on low sys requirements and work with twitch or youtube would be a gread addon and increase sales im sure.

Originally posted by Riftwind:

72b version consumes about 6gb of ram on my laptop, cant play games with it loaded because it takes forever to reply. The lower versions work better, but then they arent as smart. Seems its mostly GPU power and memory that are important.

you can use online api to slove this problem

Ive heard you can, but theres little to no documentation or instructions on how, where and what to use to get this api sort of thing working. this program works out of the box, requires hardly any setup and doesnt require connecting to an external site to use, which i really like.

Some are 6GB Vram

https://steamhost.cn/steamcommunity_com/workshop/browse/?appid=2331610&requiredtags[]=chat_service

In my mind ChatWaifu performance for the LLM is comparatively the same to when i optimize KoboldCPP to run locally on my machine. I can get more out of KoboldCPP but at the cost of worse performance. What ChatWaifu seems to have done is optimize for performance.

As it is AI compute it is normal for it to take 100% of your CPU and GPU resources to finish as soon as it can.