Unsloth on a 24 GB Mac: Polished App, Out of Memory Before the First Picture

The installer is a DMG of 24 MB, read by the tester as 'the smallest file I've ever seen'.

First window: 'Getting things ready... This won't take long.' with a link 'Show setup details'. The tester: 'Already feels much more polished than the other apps... a clear communication.'

The detail log reads: 'Updating existing Unsloth install', 'frontend bundled (Tauri)', 'node v24.13.0 | npm 11.6.2 (system)', 'unsloth 2026.8.18 -> 2026.10.3 available, updating', then a 13-step dependency bar (pip bootstrap, MLX stack for Apple Silicon, base packages, unsloth extras, codecs, overrides, studio deps, data designer, local plugin, diffusers pin, finalizing). Notes: 'torch 2.10.0 detected -- installing torchao==0.16.0'; 'transformers 5.3.0, 5.5.0, 5.10.2 pre-installed'; 'gpu Apple Silicon (Metal, unified memory)'; 'installing prebuilt llama.cpp'.

The tester says it ran for about four or five minutes, then the app opened. The log shows no time estimate and no disk space estimate.

Four to five minutes for the setup. Not timed.

The small DMG probably downloads most of the software during this first step. Not checked.

'It gives me the details about what it does -- great -- but it doesn't tell me anything about estimated time, volumes, how much disk space it takes up.'

The main window opens with 'How can I help?' and a sidebar: New chat, Model hub, Library, Images, Train, More. The logo carries the label BETA.

Started from an existing install ('same as before'), so this is an update, not a first-run test. Onboarding conclusions need a clean install to confirm.

The model picker lists Recommended models with figures such as 27.3B and 271.0M. These are parameter counts, not gigabytes. Two entries are greyed out (176.9B and 320.8B).

Greyed entries are probably too large for this machine. The app view captured does not say so.

'There were lots of models listed with the size. Very good.' The numbers shown are parameter counts, so this does not tell the reader how much memory a model needs.

Instead of choosing a model, the tester typed a request for a calculation. The app started loading models by itself. A Qwen3.8-27B MLX 4-bit model started in under a minute, per the notes; a second toast showed Mistral-7B-Instruct-v0.3-4bit loading (screenshot withheld, same type as Figure 5).

The loading toast shows the path /Users/hanskuepper/.lmstudio/models/lmstudio-community/Qwen3.8-27B-MLX-4bit. The model was read from LM Studio's folder on the disk.

He suspected it came from the hard drive, because 'that's a big model that doesn't go over the internet in one minute'. The path supports this. Why Unsloth chose this model and the Mistral one is not recorded.

The question 'what is 5 times 866' was answered '5 x 866 = 4,330' (correct), after 'Worked for 10s', at 11.0 tok/s. The header shows 'Local' and a context meter '120 / 262.1k'.

The tester then asked the model where it got the model from. The captured part of the answer talks about MLX weights and Apple Silicon unified memory; the full answer is not captured. The notes say the answer was that the app does not search the drive.

The chat model's own account of the app is not reliable evidence. One answer calls the model 'Qwen 3 27B MLX 4-bit', while the app names it Qwen3.8-27B-MLX-4bit.

The On Device tab lists models by source. Under CUSTOM FOLDERS two paths appear (.lmstudio/models and .ollama/models) with Ollama models gemma4:latest, gemma3:1b and qwen3-coder:30b. Under LM STUDIO the Qwen3.8-27B model is marked 'Loaded'.

Either the tester added the two folders or the app found them on its own. Not recorded. Earlier use of the app may have set them up.

Switching to qwen3-coder:30b (30.5B, Q4_K_M, GGUF) shows 'Starting model... Switching models. Loading cached model into memory.' with a Cancel button. The context meter reads '1.3k / 8.2k', much smaller than the 262.1k of the first model.

'I like how easy to change models.' 'Some Ollama models do not work inside Unsloth, but great that it tried at least.'

Per the notes, an older model worked and a newer one crashed. Which models were meant is not recorded.

He says it is 'great that it tried' to use his Ollama models, and that reading models already on the disk avoids another large download.

The hub shows the device: '0 GiB VRAM + 24 GiB shared', '24 GiB RAM', '12/12 CPU', plus filters (format, capabilities, newest). Cards show likes, downloads, age and license (embeddinggemma-2-GGUF: apache-2.0, 1.5 GB).

'Yes, great model hub.' The hub lists, among others, Qwen-Image-2.1-GGUF (850.0K downloads, 16 days old), GLM-5.3-GGUF and Qwen3.8-Flash-Next-GGUF.

Per the notes, clicking a file that is too big for the Mac gave several warnings ('this file is 30 to 40 gigabytes total'); the download is still allowed. No screenshot.

The download panel for unsloth/Qwen3.8-27B-GGUF UD-Q5_K_S reads 1.2 GB of 20 GB at 1.4 MB/s, 3 h 47 m left; later 7.1 GB of 20 GB at 983 KB/s, 3 h 31 m left.

'The professionalism of this app is visible. It has a timer attached. It tells you the download speed.'

The speeds suggest a slow connection at the test location; the tester does not say so. Whether the download finished is not recorded.

The Z-Image (BF16) dialog lists model files 12.3 GB plus required encoder and decoder 8.2 GB, total 20.5 GB. This explains the 'difference between the gigabytes that the model has and then something else', which the tester mentions elsewhere.

The notes say 'the picture LLM was downloaded'. Which one is not recorded; the screenshots show picture models FLUX.2-klein-4B and its GGUF variant on the device (Figure 18).

The tester pasted a PromptQuorum page URL and asked 'from a seo perspective will this help me?'. The chat shows 'Using tool: Reading promptquorum.com' and 'Waiting for approval: web_search' with the buttons Allow, Always allow and Deny. Timestamp 5:13 PM; context '648 / 34.4k'.

Per the notes, the first try said it could not read the page; after clicking the Search function it worked the second time. 'Maybe the first time should have had a warning.'

The first failure may have come from the missing web-search approval or the Search toggle being off. Not verified.

The run ended with 'Ran out of memory. Try a smaller model or shorter input.' after 168.10 s, with a Retry button.

Voice input is built in. Using it asks to download the 'Qwen3-ASR 0.6B' speech model, about 1.0 GB, which it says is fetched once into the Hugging Face cache. Whether the tester downloaded it is not recorded.

A local event search ('What is happening tonight in Heidelberg, Germany?') ran two rounds of tool calls and ended with 'Ran out of memory' after 110.65 s. The header shows a red note: 'Failed to load model: No safetensors found in /Users/.../.cache/huggin...'.

The load error and the out-of-memory message may be linked (a model being swapped while memory ran short). The screenshot does not show a link.

'But yes, local AI, always some issues.'

The tester's goal was pictures. The sidebar has an 'Images' item. The notes show a long New York skyline prompt (Brooklyn view, Brooklyn Bridge, golden hour, 8K detail) with composition and avoid lists. No result, error or settings are recorded for it.

Loading the picture model flux.2-klein was refused: 'needs about 17 GB of memory for its weights, but only about 2 GB is usable on this device (of the 7 GB currently free, after reserving room for the operating system)'. It adds that the Mac has one memory pool, offloading to the CPU frees nothing, and suggests a smaller model, a lower quant or closing other apps. It names UNSLOTH_DIFFUSION_ALLOW_OVERSIZED_LOAD=1 as an override.

After a clean reboot the message reads: about 17 GB needed, about 6 GB usable, of 11 GB currently free.

The two messages differ by 5 GB in both pairs (7 - 2 and 11 - 6), so the app seems to hold back about 5 GB for the operating system. Two data points only.

The device list shows FLUX.2-klein-4B-GGUF (7.8 GB), FLUX.2-klein-4B (16 GB) and mistral-7b-v0.3-bnb-4bit (4.1 GB). The tester: 'This should be small enough.' The tester says the smallest picture model on the hub crashes his 24 GB machine, and that the models he clicked all said they were too big because only 'seven gigabytes' were available.

The tester wonders why only seven gigabytes are available ('I have much more'). The dialog states that 7 GB were free at that moment and that the other apps were running; after the reboot it states 11 GB. The notes place the '7 GB' remark next to the Z-Image dialog; the 7 GB figure is in the flux.2-klein message.

'I'm lost... I have a 24-gigabyte machine and I cannot find an LLM that can serve me and can do the job.' He thinks 24 GB is more than 80 percent of his readers have (his estimate, not a measured figure).

'More than 80% of my readers' have 24 GB or less. Source not given.

The tester concludes that local models and daily work compete for the same memory, and that Chrome and other apps take 5 to 8 GB. He wrote 'VRAM' throughout; Apple Silicon has one memory pool, as the hub says ('0 GiB VRAM + 24 GiB shared').

Activity Monitor, one capture: memory used 12.92 GB, cached files 10.95 GB, app memory 9.81 GB, wired 1.83 GB, swap 0.

Activity Monitor, another capture: memory used 9.67 GB, cached files 8.14 GB, app memory 6.97 GB, wired 1.71 GB, swap 0. The short history graph fits a recent start.

The 11 GB 'currently free' in Figure 15 is consistent with 24.00 - 12.92 = 11.08 GB from Figure 16. That the two were taken at the same time is not recorded.

'Even after reboot, already nine gigabytes gone for general things.' 'Either I shut the computer down, reboot, stop all the processes... or I cannot run the LLM.'

Two possible answers: models must get smaller and better, or computers need 48 or 64 GB. 'All of this talk about this works on your computer is a little bit fake.'

That 48 or 64 GB would 'confidently' fix this is untested.

Starting a model shows a toast 'Starting model... Loading cached model into memory.' with a spinner and Cancel. No percentage, size or time.

The wait ran longer than two or three minutes (not timed).

'The worst way you can treat a client is to do the work and not update the client.' The tester names this a user interface issue.

Header meters show tokens used against the context (for example '648 / 34.4k') and tokens per second (11.0 tok/s).

The notes say 'no token counter'. The screenshots show a context meter and a tokens-per-second figure. A per-message token count may be what he meant (assumption).

The notes call a local chat model 'the most stupid idea' because it 'does not research the internet'. The same notes and screenshots show a Search function, a web-search approval prompt and tool calls reading a page.

He does not want to train models; the chat function is of little use to him; he wants a picture and video tool and 'is not seeing it yet'. Training and the 'Train' tab were not tried.

'Maybe it's on my side, but no, not for me. I'll check again in three or six months, but currently not on a M3 24 GB Pro MacBook.'