Inkling 2 Mini Hands-On: Download the Model and Run Local Inference

The weights of Inkling 2 Mini have been made public, which means you can run it on your own computer instead of only using it through the cloud API. This tutorial guides you through the entire process: starting with environment checks, pulling the model with Ollama and generating your first response, then finally calling it from your programs using curl. The entire walkthrough is based on Apple Silicon Macs, and every step tells you how to determine whether it succeeded.

Environment Check: First Confirm Your Machine Can Run It

Before downloading anything, confirm three conditions. Open Terminal and run the following commands in order:

uname -m
sysctl hw.memsize
df -h /

If the first command outputs arm64, it indicates Apple Silicon; if it outputs x86_64, you will need to find an alternative solution on an Intel Mac. The second command outputs the memory size in bytes; 34359738368 corresponds to 32 GB, which we recommend as the minimum capacity. The third command checks disk space; the model file is about 20 GB, and adding runtime and system cache, please reserve extra space for cache and running.

The official model card lists that Apple Silicon environments require 32 GB or more of unified memory. The actual context length you can configure also depends on runtime and cache overhead.

Install Ollama: Choose the Version Based on the Model Page

We follow the Ollama route because it wraps model format, quantization, and caching for you, so you don't need to worry about underlying details. First go to the Ollama download page, install the desktop app, then launch it; you should see a small icon in the menu bar. Then return to the terminal to verify the version:

ollama --version

If you get command not found, first confirm that the desktop app has been launched. When the Ollama desktop app starts, it runs a service in the background; follow the app prompts to install the command-line tool. Seeing the version number output means the environment is ready.

There is a pitfall regarding versions: the Ollama model page for Inkling 2 Mini requires a certain version or newer of Ollama to support its architecture. If you use an older version, ollama pull will report errors like unsupported model architecture. If you encounter such a message, install the version that supports this model according to the model page instructions.

Pull the Model and Verify the Download

Now pull the model, about 20 GB, depending on your network speed:

ollama pull inkling-2-mini

The download process shows a progress bar and speed. The success marker for this step is no error messages, and the last line shows success. Next, confirm it is indeed local:

ollama list

The output should include a line inkling-2-mini, with the SIZE column showing about 20 GB. The tag and size are subject to the model repository and your local list.

While downloading, you can observe the ~/.ollama/models directory; the file size will keep growing. However, we don't need to intervene manually; this is just to confirm that the data is actually written to disk.

Run Your First Response

This is the moment to witness the result. Execute:

ollama run inkling-2-mini "Explain in three sentences what KV cache is"

The first run requires loading the model; after the load completes, the terminal will display the model-generated response. Seeing a response indicates that the model has been successfully loaded and inference has completed on your machine.

If you get memory error or it crashes during loading, refer to the troubleshooting section below. First resolve the loading error, then test the API.

Calling via the API: How to Use It in Programs

Programs can call the same model through Ollama's local HTTP API. After Ollama starts, it listens on 127.0.0.1:11434 by default; let's test directly with curl:

curl http://127.0.0.1:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "inkling-2-mini",
    "messages": [{"role": "user", "content": "What is sparse attention?"}],
    "stream": false
  }'

The response is JSON, and the answer is in the message.content field. You can extract it using a pipe:

curl -s http://127.0.0.1:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"inkling-2-mini","messages":[{"role":"user","content":"What is sparse attention?"}],"stream":false}' \
  | python3 -c "import sys, json; print(json.load(sys.stdin)['message']['content'])"

If you get connection refused, it means the Ollama service is not running; go back to the desktop app and check if the icon is in the menu bar. If you get a JSON response but it lacks the message field, the request or inference may have failed; first check the HTTP status and the error field in the JSON, then verify the model name with ollama list.

Another Route: Hugging Face Raw Weights

Ollama is good for quick start, but if you want to use vLLM or other inference frameworks, you need the raw weights. Go to the Hugging Face repository, which totals about 20.2 GB, including safetensors, configuration files, tokenizers, etc.

First install the official command-line tool:

python3 -m pip install -U huggingface_hub

Then download to a specified directory:

hf download thinkingmachines/Inkling-2-Mini --local-dir ./Inkling-2-Mini

Note that this step only downloads files; it does not include inference capability. After the download completes, you will see model.safetensors.index.json and the main weight files, which are the model parameters. To actually run it, you need an inference framework that loads these weights—vLLM, SGLang, etc.—each with its own installation and configuration documentation. Ollama essentially does something similar internally, just wrapped up.

The division of labor between these two routes: Ollama manages weights and runtime for you, while the Hugging Face route gives you full control, at the cost of handling framework compatibility yourself. The mixed-precision format of the weights (NVFP4/MXFP8) has requirements for inference frameworks; pay attention to the compatibility list on the official model card when choosing.

Common Issues and Troubleshooting

Model pull interrupted or verification failed
First check disk space; run df -h / to verify whether there is room for the model and cache. Then re-run ollama pull inkling-2-mini. If interruptions recur, it may be a network issue; try switching network environments.

ollama --version reports command not found
The desktop app is not running, or the installation path is not in PATH. First click the Ollama icon in the menu bar to confirm the service is running, then open a new terminal window and try again.

API returns connection refused
The Ollama service is not listening on port 11434. On macOS, the most common scenario is that the desktop app was quit; restart it. Check method: curl http://127.0.0.1:11434, if it connects, it will return Ollama is running.

Insufficient memory when loading the model
The official minimum requirement is 32 GB unified memory; below this, do not try the full model. If your machine has 32 GB but loading still fails, first add "options": {"num_ctx": 2048} to the API request JSON above, verify with a short prompt, then gradually increase context based on actual memory and task.

ollama run reports unsupported model architecture
The Ollama version is too old. Check the model page's current installation instructions and verify the local version.

Hugging Face download speed is slow
First confirm that you can access Hugging Face, and check the free space on the target disk. After the network recovers, re-run the same download command.

Further Reading: Explore More

The official model card contains architecture details and benchmark results, worth a detailed read. The Ollama API documentation has more parameters (like temperature, top-p), which you can adjust to your application needs. If you want to run this model with vLLM, search its documentation for Inkling 2 compatibility notes.

The model's middle 12 layers reuse the same set of weights, executed three times. The approximately 20 GB file size also relies on NVFP4 and MXFP8 mixed precision. 1M tokens is the model's maximum supported limit; the local context should be configured according to available memory.