cmaven

Worknotes

Local LLM Coding Agent series ① Local LLM Core Concepts · ② Ollama Local LLM Setup Guide (this post) · ③ Ollama Multi-GPU Operations & Troubleshooting · ④ AI Code Review with Two Local LLMs

:bulb: This post documents a setup that runs OpenCode as a Korean-language coding agent using only two NVIDIA A30 24GB GPUs — no paid API. The conclusion is simple: load a single Qwen3.5 35B-A3B Q4_K_M on Ollama, split it across both GPUs, and secure a 64K context. No NVLink required.

The final architecture first:



1
2
3
4
5
6
7
8
9
10
11
12



User
  │ instructions in Korean
  ▼
OpenCode
  │ read/edit files, run commands, test
  ▼
Ollama's OpenAI-compatible API (127.0.0.1:11434)
  │ one model auto-split across two A30s
  ▼
Qwen3.5 35B-A3B Q4_K_M (~24GB)
  ├─ A30 #0 24GB
  └─ A30 #1 24GB
  • Cost: no per-token API fees. Code never leaves the server, so it works on air-gapped networks
  • Model: Qwen3.5 35B-A3B Q4_K_M — a balance of Korean, coding, tool calling, and long context
  • Context: 65,536 tokens. Ollama’s official docs recommend 64K+ for coding-agent workloads
  • No NVLink: the goal isn’t 2× speed — it’s fitting a model + context that won’t fit on one card across two

If VRAM, quantization, KV cache, or MoE are new terms, read ① Local LLM Core Concepts first — it’s quicker that way. This post assumes those concepts and covers only the install and configuration.

[01] Model Selection — Korean Fluency Alone Isn’t Enough

As Core Concepts 2-9 showed, an OpenCode model needs tool calling stability as much as natural Korean. The selection priorities:



1
2
3
4
5



1. OpenCode tool calling stability
2. Coding and repository-scale task ability
3. Understanding Korean instructions, writing Korean output
4. Feasibility on two A30s
5. Installation and maintenance effort

Comparing the candidates against those criteria:

Model Strengths Weaknesses Best For
Qwen3.5 35B-A3B Q4 Balanced Korean/coding/tool-calling/long context, official Ollama release PCIe communication overhead when split across two GPUs Default main model
Qwen3-Coder 30B-A3B Q4 Strong at coding-agent and repository work General Korean prose may trail Korean-specialized models Coding quality first
EXAONE 4.0 32B Q4 Strong Korean domain knowledge and expression, official GGUF 19.3GB weights make long context tight on a single A30 Korean planning/docs/review
Kanana-2 30B-A3B Korean tokenizer, emphasizes Korean agent performance 32K default context, no simple official Ollama path Advanced comparison experiments

Why Qwen3.5 35B-A3B is the default:

  • Supports 201 languages and dialects including Korean, with strong multilingual understanding
  • A general chat model that also supports coding, tool calling, and agent work
  • Official Ollama Q4_K_M release at about 24GB, Apache 2.0 licensed
  • One A30 is a tight fit for weights plus long context, but two cards can target a 64K context
  • Connecting a single model is the simplest thing a beginner can operate

The same reasoning explains why EXAONE and Kanana aren’t the default. They’re strong Korean candidates, but the bottleneck in this setup isn’t Korean — it’s tool calling and long context. It’s also hard to objectively crown one “best Korean model”: naturalness, domain knowledge, coding, tool calling, and post-quantization quality are all different axes. Stabilize the whole environment on Qwen3.5 first, then compare against EXAONE and Kanana using 10–20 real work tasks (see the final section).

[02] Hardware Preparation

Item Value
GPU NVIDIA A30 24GB × 2 (NVLink optional)
System RAM 64GB minimum, 128GB+ recommended
Storage 100GB+ free, 200GB+ if comparing models
Power/cooling Up to 165W × 2 for the GPUs alone; passive cooling demands real chassis airflow
OS Ubuntu 22.04 family

Why system RAM matters beyond the GPUs: model file loading, OpenCode itself, Git/build/test, container and Kubernetes tooling, and any CPU offload under VRAM pressure all use main memory. CPU offload can keep things running but slows them down badly — avoid it in a healthy setup.

2-1. Verify Both GPUs



1
2
3
4
5
6



nvidia-smi -L
# GPU 0: NVIDIA A30 (...)
# GPU 1: NVIDIA A30 (...)

nvidia-smi          # overall status
nvidia-smi topo -m  # without NVLink the GPUs show a PCIe path — not an error

Even if nvidia-smi works, check that Driver Version is 550 or newer. Recent Ollama CUDA runners require 550+, and anything below that runs on CPU no matter how healthy the GPUs are (see Ops & Troubleshooting 01). On a server with no driver, or one below that version:



1
2
3
4



sudo apt update
ubuntu-drivers devices
sudo ubuntu-drivers install
sudo reboot

On a Kubernetes node, or where the NVIDIA GPU Operator manages drivers, check for conflicts with the existing configuration before installing anything at the OS level.

2-2. Disable MIG

The A30 supports MIG (slicing one GPU into partitions), but this setup uses the full 24GB of both GPUs.



1
2
3
4
5
6



nvidia-smi -q | grep -A 2 "MIG Mode"

# if enabled, during a maintenance window
sudo nvidia-smi -i 0 -mig 0
sudo nvidia-smi -i 1 -mig 0
sudo reboot

MIG changes can fail while the GPU is in use. Stop processes, containers, and Kubernetes pods using the GPU first, and on a production server confirm the blast radius before proceeding.

[03] Installing and Tuning Ollama



1
2
3



curl -fsSL https://ollama.com/install.sh | sh
ollama -v
sudo systemctl status ollama

The systemd override is the heart of the setup. Open it with sudo systemctl edit ollama and add:



1
2
3
4
5
6
7
8
9



[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_CONTEXT_LENGTH=65536"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_MODELS=/data/ollama/models"





1
2



sudo systemctl daemon-reload
sudo systemctl restart ollama
Setting Meaning
OLLAMA_HOST=127.0.0.1:11434 API reachable only locally (see security section)
OLLAMA_CONTEXT_LENGTH=65536 64K default context (see Core Concepts 2-6)
OLLAMA_NUM_PARALLEL=1 one concurrent request to save VRAM
OLLAMA_MAX_LOADED_MODELS=1 one model resident at a time
OLLAMA_FLASH_ATTENTION=1 reduces long-context memory use
OLLAMA_KV_CACHE_TYPE=q8_0 8-bit KV cache — roughly half of f16 (see Core Concepts 2-7)
OLLAMA_KEEP_ALIVE=-1 keep the model resident on the GPUs
OLLAMA_MODELS=/data/ollama/models store models on the data disk instead of root (see Ops & Troubleshooting 02)

OLLAMA_KEEP_ALIVE=-1 may not suit a server that shares GPUs with other workloads. Change it to something like 30m if needed.

Leaving out CUDA_VISIBLE_DEVICES is deliberate. With only two GPUs, 0,1 restricts nothing while getting in the way of Ollama’s own discovery, and it leaves a warning in the log.



1
2



WARN msg="user overrode visible devices" CUDA_VISIBLE_DEVICES=0,1
WARN msg="if GPUs are not correctly discovered, unset and try again"

Set it only on a server where Ollama should get a subset of the GPUs.

[04] Pulling the Model and Verifying the Two-GPU Split



1
2
3



ollama pull qwen3.5:35b-a3b   # ~24GB — takes a while depending on network and disk
ollama list                   # confirm installation
ollama run qwen3.5:35b-a3b

A Korean smoke test:



1
2
3



앞으로 모든 설명은 자연스러운 한국어로 작성해줘.
코드, 함수명, 명령어, API 이름은 원문을 유지해줘.
Kubernetes Controller가 무엇인지 초보자에게 설명해줘.

(Exit with /bye.) While it generates, watch both GPUs from another terminal.



1
2



watch -n 1 nvidia-smi   # VRAM and utilization should rise on GPU 0 and 1
ollama ps

What to check in ollama ps:

  • PROCESSOR shows GPU usage
  • CONTEXT is near 65536
  • No excessive CPU/GPU mixed offload

If only one GPU is used, Ollama may have decided the model plus context fits on one card — which can be normal. The real problem is VRAM exhaustion or CPU offload at 64K. In that case work through: ① stop other GPU processes → ② restart Ollama → ③ confirm the inference compute log lines report library=cuda (see Ops & Troubleshooting 01) → ④ check journalctl -e -u ollama → ⑤ re-check nvidia-smi during generation.

If it reads 100% CPU instead, this is not the section you want — go to Ops & Troubleshooting 01, it’s the driver.

Finally, verify the OpenAI-compatible API.



1
2
3
4
5
6
7
8



curl http://127.0.0.1:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.5:35b-a3b",
    "messages": [
      { "role": "user", "content": "한국어로 한 문장만 답해줘. 로컬 LLM 연결 시험이야." }
    ]
  }'

A JSON response with a Korean answer means you’re ready to connect OpenCode.

[05] Installing and Connecting OpenCode



1
2
3



curl -fsSL https://opencode.ai/install | bash
source ~/.bashrc
opencode --version

One config file completes the connection.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28



mkdir -p ~/.config/opencode
cat > ~/.config/opencode/opencode.jsonc <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",

  "model": "ollama/qwen3.5:35b-a3b",
  "small_model": "ollama/qwen3.5:35b-a3b",

  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local - A30 x2",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1"
      },
      "models": {
        "qwen3.5:35b-a3b": {
          "name": "Qwen3.5 35B-A3B Q4 - Local A30 x2",
          "limit": {
            "context": 65536,
            "output": 8192
          }
        }
      }
    }
  }
}
EOF
  • No API key. Requests only go to the local Ollama at 127.0.0.1
  • small_model is the slot OpenCode uses for lightweight jobs like session titles. It’s pinned to the same local model so no external provider is ever touched
  • The config schema can change between OpenCode versions. On errors, check the schema at https://opencode.ai/config.json first

Run it inside the Git project you’ll work on:



1
2



cd /path/to/your/project
opencode





1
2



/models   # confirm "Ollama Local - A30 x2 / Qwen3.5 35B-A3B Q4" is selected
/init     # analyze the project and generate AGENTS.md

[06] AGENTS.md: Locking In Korean Output Quality

AGENTS.md holds the working rules the AI agent follows in that project: which language to answer in, build and test commands, files it must not touch, approval rules for dangerous commands, and the report format. The key benefit: you stop repeating “answer in Korean” in every conversation.

Place this at the project root:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25



# Project AI Working Rules

## Response language
- Write explanations, plans, progress, and reports in Korean.
- Keep code, variable/function names, API names, and CLI commands in their original form.
- Keep technical terms that lose meaning in translation in English, explaining them in Korean on first appearance.

## Working style
- On any request, inspect the relevant files and current implementation first.
- Present a short plan of what will change before changing it.
- Don't modify an unnecessarily large number of files at once.
- Never invent APIs from guesswork; check the repository's actual code and docs.

## Verification
- After edits, run build, lint, and unit tests where possible.
- On failure, explain the cause in Korean, fix, and rerun.
- If tests couldn't run, state why and which commands are needed.

## Safety
- Never auto-run dangerous commands: rm -rf, disk formatting, git push --force.
- Ask for approval before operationally impactful Kubernetes commands: delete, drain, cordon.
- Never print or commit .env files, certificates, private keys, or tokens.

## Reporting
- Summarize in Korean: files changed / key changes / tests run and results / remaining risks.

If answers start mixing Korean and English, check this file’s language rules and start a fresh session. Some English remaining while the model reads English library docs or error logs is normal.

[07] Operations — Plan First, Build Later

OpenCode has two working modes. Plan investigates the project, finds relevant files, and writes a change plan — file edits are restricted. Build does the actual edits, code generation, shell commands, builds, and tests. Local models make more mistakes than large commercial ones, so this ordering matters even more here.



1



Investigate with Plan → human reviews the plan → implement with Build → test → human reviews git diff

How you phrase requests also drives the success rate. Bad versus good:



1
2
3
4
5
6



Bad:  Fix this.

Good: This repository is a Kubernetes Operator project.
      Check the current status.conditions update logic.
      Find the relevant files first, explain the current behavior in Korean,
      then write only an edit plan. Don't change any files yet.

Not handing over huge tasks at once follows the same logic. Not “build the whole system” but “step 1: review only the CRD type definitions → step 2: implement only the Validate logic → step 3: add unit tests.” Local models succeed far more often when work is sliced into small steps.

And always pair it with Git. Branch before working — git switch -c ai/test-local-opencode — so even if OpenCode edits something badly, git diff shows it and you can roll back.

[08] Security

  • API binding: keep OLLAMA_HOST=127.0.0.1:11434. Binding 0.0.0.0:11434 lets anyone on the network reach the model API without authentication
  • Remote use: if a laptop’s OpenCode needs the server’s Ollama, don’t open the port — tunnel it: ssh -L 11434:127.0.0.1:11434 user@server, then connect the laptop to http://127.0.0.1:11434/v1
  • Run as: OpenCode executes shell commands and edits files, so run it as a regular user, never root
  • Kubernetes permissions: give OpenCode a least-privilege kubeconfig — dev namespaces only, mostly read-only, no delete. Never the production cluster-admin kubeconfig
  • Secrets: block printing/committing .env, kubeconfig, SSH/TLS private keys, API tokens, cloud credentials, and DB passwords via AGENTS.md rules

[09] Common Problems

Symptom Check
Model missing in OpenCode Model ID in opencode.jsonc must exactly match the name from curl http://127.0.0.1:11434/api/tags (qwen3.5:35b-a3b)
Answers but can’t call file-edit tools Latest OpenCode/Ollama, official model tag, Build agent selected, context not too small, AGENTS.md not over-restricting tool use
Quality drops in long conversations New session per feature. Rules live in AGENTS.md, not chat. Commit finished work and hand over a short status to the next session
Answers mix Korean and English Check the language rules in AGENTS.md and start a new session. Some English remaining when reading English docs or error logs is normal

Ollama-side symptoms — 100% CPU, Connection refused, OOM, models vanishing after reboot, CPU offload — are gathered in ③ Ollama Multi-GPU Operations & Troubleshooting.

[10] Next Steps — Model Comparison and Per-GPU Split

Once the environment is stable, comparing Korean quality on real work is worth the time. EXAONE 4.0 32B’s official GGUF Q4_K_M is about 19.3GB.



1
2



ollama stop qwen3.5:35b-a3b
ollama run hf.co/LGAI-EXAONE/EXAONE-4.0-32B-GGUF:Q4_K_M

Evaluate with actual tasks, not model-card scores:

Axis Test
Korean comprehension Does it drop conditions from long requirements?
Korean expression Awkward translationese or unnecessary English mixing?
Code editing Multi-file edits succeeding in a real repository?
Tool calling Success rate on file/shell/test tool calls
Error recovery Does it fix itself from failing test logs?
Speed/memory Time to first token; GPU load behavior at 32K/48K/64K

Once comfortable, a per-GPU dual-model setup is also possible:



1
2



A30 #0 └─ EXAONE 4.0 32B Q4      → Korean requirements analysis, docs, review
A30 #1 └─ Qwen3-Coder 30B-A3B Q4 → code implementation, tests

The upside: the two models are independent, so there’s almost no inter-GPU traffic, and the Korean model and coding model split roles cleanly. The costs: running two Ollama servers, fitting model + KV cache into 24GB per GPU (start contexts at 24K–32K), and more complex provider config. Start with the baseline — one Qwen3.5 split across both cards — and try this afterward.

Pairing that layout with OpenCode subagents lets you separate the implementing model from the reviewing one. The follow-up post, AI Code Review with Two Local LLMs, covers per-model context limits, restricting the reviewer’s permissions, and capping the review rounds.

Eight operating principles to close:

  1. Never blindly trust a local model’s output
  2. Always review the commands OpenCode ran and the files it changed
  3. Never hand it production cluster permissions
  4. Slice features into small tasks
  5. Manage project rules in AGENTS.md
  6. Split overly long sessions into new ones
  7. Evaluate model performance on real work tasks
  8. Without NVLink, two GPUs buy memory, not speed

References

Environment: Ubuntu 22.04 / NVIDIA A30 24GB × 2 (no NVLink) / Ollama / OpenCode. Docs and model tags are as of 2026-07-30; re-verify the OpenCode config schema and Ollama’s Hugging Face GGUF support against the versions at install time.

You may also enjoy