In my previous post, I wrote about upgrading my computer for local AI and exploring what my new NVIDIA RTX 4080 SUPER with 16 GB of VRAM could handle.
After doing a new search and comparing the available Qwen models more carefully, I have now selected the model I plan to use with my Hermes AI agent:
Qwen3.6-27B-GGUF — UD-IQ3_XXS
This may look like a complicated name, but the basic idea is simple: it is a capable AI model that has been compressed so it can run on a powerful home computer without needing an expensive professional graphics card.
Why I Chose Qwen3.6-27B
Qwen3.6-27B is a model with 27 billion parameters. Parameters are part of what an AI model learns during its training. A larger number does not automatically make every model better, but it usually gives the model more capacity for understanding instructions, writing, reasoning and solving complicated tasks.
This model appears especially suitable for:
General writing and conversation
Creative writing
Coding and technical assistance
Following detailed instructions
Working with AI agents and tools
Planning and completing multi-step tasks
Reasoning through difficult questions
The full model would normally need much more memory than my graphics card provides. That is why I selected a compressed version called UD-IQ3_XXS.
This version is approximately 12 GB, allowing it to fit within my limited internet download allowance while also being practical for my RTX 4080 SUPER.
The Exact Model File
The complete filename I selected is:
Qwen3.6-27B-UD-IQ3_XXS.gguf
Its listed download size is:
12 GB
The model is provided by Unsloth on Hugging Face.
It is important to choose the exact filename because the same page contains many versions. Some are over 15, 20 or even 30 GB. Downloading the wrong one would waste a considerable amount of internet data.
The letters GGUF mean that the model has been prepared in a format commonly used by local AI applications such as llama.cpp, LM Studio and Ollama.
The term IQ3_XXS describes how strongly the model has been compressed. This compression causes some loss in quality compared with the much larger versions, but it makes the model practical for my hardware and internet situation.
What About the Newer Qwen3.8?
Qwen3.8 is the newest generation and is normally the model I would recommend investigating first. It improves areas such as coding, professional work, research, long tasks, tool use and autonomous agent planning.
The smaller open version of Qwen3.8 is also a 27-billion-parameter model, but it remains demanding for a graphics card with 16 GB of VRAM.
More importantly, Qwen3.8 is still very new. At least one recent Hermes Agent issue reported that it could sometimes reach a maximum-token error during longer goal-based tasks, while Qwen3.6 successfully completed the same request.
This does not mean Qwen3.8 is a bad model. It may eventually become the better choice after its support becomes more mature. However, for my first Hermes setup, Qwen3.6 currently appears to offer a safer balance between capability, stability, file size and hardware requirements.
You can read about the newer generation on the official Qwen3.8 page.
Why an Offline AI Model Is Helpful
Cloud AI services are often more powerful, but they depend on a stable internet connection. That is not always convenient when living somewhere with slow, expensive or limited internet access.
An offline model is downloaded once and then runs directly on the computer. After the initial download, ordinary conversations with it do not consume internet data.
This provides several advantages:
It continues working when the internet is slow or unavailable.
There are no usage limits for ordinary local conversations.
It does not require paying for every message or token.
Private drafts and documents can remain on the local computer.
Response speed is not controlled by internet quality.
The owner has more control over the model and its settings.
It can be connected to local files and tools through an agent such as Hermes.
An offline model is not a complete replacement for cloud AI. The largest cloud models will generally remain more capable. However, a strong local model can become a dependable everyday assistant, especially when internet access is limited.
Using It to Write Stories and Novels
One of the main reasons I am interested in local AI is creative writing.
A model such as Qwen3.6-27B can help a writer develop:
Story concepts and themes
Fantasy worlds and their histories
Character profiles and relationships
Chapter plans and story outlines
Locations, cultures and magical systems
Dialogue ideas
Scene descriptions
Alternative versions of difficult passages
Consistency checks between chapters
Editing and rewriting suggestions
Summaries of characters, places and important objects
It can also act as a creative partner when a writer feels stuck. Instead of asking it to write the entire novel automatically, the writer can use it to explore possibilities, test ideas and improve existing work.
For long novels, it is still important to keep organized reference documents. These may include a story bible, character sheets, a timeline, a glossary and chapter summaries. A local model cannot remember an unlimited manuscript all at once, but an AI agent can be given the relevant reference material as it works through the story chapter by chapter.
The writer should always remain the creative director. AI can suggest, organize and revise, but the heart of a novel still comes from the person imagining its world.
My Planned Starting Settings
I plan to begin with a context length of approximately 8,192 tokens. In simple terms, context length controls how much information the model can actively consider during one conversation.
Although Qwen3.6 supports much larger context sizes, increasing the context also consumes more graphics memory. Starting with 8K should provide a more stable experience on my RTX 4080 SUPER.
My initial setup will therefore be:
Model: Qwen3.6-27B
File: Qwen3.6-27B-UD-IQ3_XXS.gguf
Download size: 12 GB
Format: GGUF
Graphics card: RTX 4080 SUPER with 16 GB VRAM
Starting context: 8K
AI agent: Hermes Agent
Main uses: Writing, research assistance, creative development and local agent tasks
I will test thinking mode for difficult work and use the quicker non-thinking mode for ordinary conversations and simpler writing tasks.
Final Thoughts
Selecting a local AI model is not simply about downloading the newest or largest model. The model must match the computer, available graphics memory, system memory, storage space and internet limitations.
For my present setup, Qwen3.6-27B-UD-IQ3_XXS appears to be the most sensible starting point. It gives me access to a capable 27-billion-parameter model in a file of approximately 12 GB, while remaining suitable for my RTX 4080 SUPER.
Qwen3.8 may become my next upgrade once it has matured further and its Hermes compatibility is more stable. For now, Qwen3.6 provides the balance I need: capable enough for serious creative work, small enough to download, and practical enough to run locally.
My next step will be connecting the model to Hermes Agent and discovering how well a completely local AI assistant can help me organize ideas, build fictional worlds and write stories without depending continuously on the internet.
No comments:
Post a Comment