#llm

8B~27B 모델을 서빙하려고 합니다. throughput이 중요하면 vLLM, VRAM이 빠듯하면 llama.cpp라는 얘기가 있는데 실제 운영하시는 분들 경험 공유 부탁드려요.

Updated list of open LLMs you can use commercially. Handy when picking a model for a client project.

Ollama on a laptop is great for tinkering, but once you need reliability, hosted APIs win on latency and uptime. My experience so far.

가성비 미쳤습니다. 4bit로 12GB 램 노트북에서 돌아간다고 하니 로컬 실험용으로 딱이네요.

L4 한 장으로 Qwen2.5-7B 서빙. 처리량 만족스럽습니다. 양자화 없이도 충분하네요.

동일 7B 모델로 비교했는데 p50은 비슷하고 p99는 TRT-LLM이 더 안정적이었습니다. 수치 표로 정리했어요.

Prefill pipelining finally landed. Our 128k-context serving costs should drop meaningfully.

One endpoint for all providers, consistent interface, and the config is plain YAML. We swapped out three vendor SDKs in an afternoon. If you're multi-provider, just use it.

Woke up to the leaderboard shuffle. Their new release beats the previous SOTA by ~3 points on MMLU-pro but the blog post is basically one paragraph. Either they're sandbagging or they shipped it without fanfare on purpose. Either way the pricing page is still the old one, which is the real story.

vLLM vs TGI vs llama.cpp server. For a small team serving one 8B model, what's the least ops burden?

Downloaded the 27B variant last night. Quantized to Q4 it fits in 16GB RAM and generates at a very usable speed. Not Claude-level reasoning, but for a local model it's a big step. Google is winning the 'open weights you can actually run' game right now.

로컬에서 돌릴 수 있는 새 오픈웨이트 모델 소식. 7B인데 성능이 무섭다.

System prompt first, stable ordering, minimal drift. Any other tricks to keep cache hits high?

Long context is expensive and my tests show the model forgets the middle anyway. Chunking into a RAG loop works but feels like giving up. Do you just pay for big context or actually engineer around it?

Building evals feels like a second product. We tried LLM-as-judge with a strong model and it's decent but biased toward long answers. Anyone using lightweight checks + sampling? What's your minimal viable eval?

We're on vLLM with continuous batching and it's fine, but TGI has better tool-calling support out of the box. Am I missing something? Also considering SGLang. Real-world throughput numbers welcome.

Been running a 70B Q4 on one 80GB card for a side project. Key tricks: prefix caching on, max_num_seqs tuned, and a tiny health check for the load balancer. Sharing my config because I wish someone had shared theirs.

12B model on the family PC, a simple web UI. It explains math problems step by step instead of just giving answers. No accounts, no data leaving the house, and no monthly subscription. The kids actually use it.

Wrote our own gateway instead of buying one. Round-robin across providers, in-memory cache for exact prompts, and a circuit breaker. It's not fancy but it cut our API bill 35% and we control everything.

It categorizes spending, flags weird charges, and answers questions like 'how much did I spend on coffee in March'. Accuracy is 90%+ on categories. Caveat: you have to trust it with your data, so it runs fully local. Worth it for the anxiety reduction alone.

I ran Llama 4.5 8B Q4 on my M3 Pro and it's... fine? Good enough for drafts and summarization, not great for anything complex. Battery drains fast though. For $0 in API costs, the tradeoff is getting more tempting.