질문답변

코드 문서는 300토큰, 일반 문서는 500~800토큰으로 쓰고 있는데요. 법률 문서처럼 구조화된 건 섹션 단위로 자르는 게 낫다는 의견도 있더라고요. 다들 어떻게 하세요?

8B~27B 모델을 서빙하려고 합니다. throughput이 중요하면 vLLM, VRAM이 빠듯하면 llama.cpp라는 얘기가 있는데 실제 운영하시는 분들 경험 공유 부탁드려요.

도메인 지식이 필요하면 RAG, 스타일/포맷 고정이면 FT라는 기본 공식이 있는데, 실제 프로젝트에서는 어떤 기준으로 결정하시나요?

bge-m3에서 최신 모델로 바꾸려고 하는데 벡터 차원이 달라서 재임베딩이 필요할 것 같습니다. 100만 건 정도인데 시간이 얼마나 걸릴지, 꿀팁 있으면 알려주세요.

프롬프트 캐시 쓰면 50%는 아낀다고 들었는데, 실제로 얼마나 효과 보셨나요? 컨텍스트 압축이나 요약 전략도 궁금합니다.

한국어 문서 검색인데 bge-m3 vs multilingual-e5 고민입니다. 경험 공유 부탁해요.

LLM이 스트리밍으로 출력할 때 JSON을 점진적으로 파싱하는 방법 질문입니다. partial JSON 파서 쓰시는 분 계신가요?

도메인 용어가 많은데 파인튜닝이 나을지 RAG가 나을지. 둘 다 해보신 분들 의견 궁금합니다.

현재 Chroma 쓰다가 Qdrant로 옮길지 고민 중. 데이터 500만 건 정도인데 경험 있으신 분?

웹에서 문서를 읽어와서 요약하는 기능인데, 문서에 악성 지시문이 있으면 대응이 어렵네요. 방어 전략 공유 부탁드려요.

We use RAGAS but the scores don't correlate with human judgment. What do you actually use in production?

Chroma vs Qdrant vs LanceDB for a small self-hosted RAG service. Disk and RAM matter — running on a free-tier VM.

Long agent sessions blow up tokens. Do you use summarization, sliding windows, or something else?

bge-m3에서 다른 모델로 바꾸려는데, 기존 문서 전부 다시 인덱싱해야 하는 게 맞는지요?

DeepSeek 캐시 히트율 66% 나오는데, 프롬프트 구조 어떻게 짜면 캐시가 잘 맞나요?

Model cites chunks that don't actually support the answer. Any post-processing tricks that work?

vLLM vs TGI vs llama.cpp server. For a small team serving one 8B model, what's the least ops burden?

임베딩 + 소형 LLM만으로 API 없이 운영 가능한지 궁금합니다. 트래픽은 하루 1천 건 수준이에요.

System prompt first, stable ordering, minimal drift. Any other tricks to keep cache hits high?

We need to ground answers in internal docs that update weekly. RAG feels obvious but retrieval quality is killing us. Fine-tuning is a one-time cost but the docs change. Anyone run a hybrid? What's your split?

Long context is expensive and my tests show the model forgets the middle anyway. Chunking into a RAG loop works but feels like giving up. Do you just pay for big context or actually engineer around it?

Building evals feels like a second product. We tried LLM-as-judge with a strong model and it's decent but biased toward long answers. Anyone using lightweight checks + sampling? What's your minimal viable eval?

We're on vLLM with continuous batching and it's fine, but TGI has better tool-calling support out of the box. Am I missing something? Also considering SGLang. Real-world throughput numbers welcome.

JSON mode is simpler but occasionally returns garbage. Function calling is more reliable but couples us to the provider's schema. We're on DeepSeek and both work 'okay'. What's the pragmatic choice for a production pipeline?

I keep seeing MCP everywhere but every real deployment I've touched still uses plain REST under the hood. Is anyone running MCP servers in production with multiple clients? What's the actual win?

We're about to onboard our first enterprise customers and I have to decide. Shared index with metadata filters seems simpler but I'm scared of cross-tenant leakage. Per-tenant collections mean more ops. What do you run?

We're paying a lot for repeated system prompts across a long agent conversation. Their caching docs promise 90% savings on reads. Anyone measured real-world savings? Does it break with streaming?

QLoRA with 4-bit got me through a few epochs but training was slow and the output quality was meh. Should I just rent an 80GB card for a weekend instead? What's the realistic minimum for decent results?

Picked a model, built a few retrieval test sets, but I'm not sure my eval is meaningful. Same-document hits feel like cheating. What's your go-to method for measuring retrieval quality early?