For larger sizes you still can, it just becomes slower and slower. For a simple classification task (small input, tiny output, and you can constrain output to a couple tokens) you could even run something like a 4B or 8B model on the CPU

GPU and VRAM (or fast unified RAM) is generally the option that is both available and performant, but especially really small models also run quite well on CPU and system RAM.

The advantage is mainly in memory bandwidth. External GPUs' internal memory is slightly faster than DDR attached to your CPU.

Other types of "AI" models do make use of the extra compute in GPUs but not LLMs.

    providers:
      - # llama-server
        id: openai:chat:qwen
        config:
          apiBaseUrl: http://localhost:7876
          apiKey: "..."
          passthrough:
            chat_template_kwargs:
              enable_thinking: false