semcode RAG System — Documentation

semcode is an MCP server that provides hybrid semantic search over code from GitHub repositories. It parses source files into code symbols (classes, methods, functions) using Tree-sitter, indexes them with both dense and sparse embedding vectors, and retrieves them using Reciprocal Rank Fusion (RRF).

What it indexes: code symbols from any configured GitHub repository. Each symbol (class, method, function, interface) is stored as a Qdrant point with its source text, metadata, and two embedding vectors.

How it serves queries: AI clients connect via MCP and call tools like search_code, find_symbol, and find_usages. Each search runs a hybrid query — dense semantic vectors find conceptually similar code; BM25 sparse vectors find exact and partial identifier matches. RRF merges both rankings into a single ordered result list.

---

Documentation Index

Document	What it covers
ingestion.md	End-to-end indexing pipeline: GitHub file discovery, incremental change detection, parsing, embedding, upsert, and stale cleanup
dense-vectors.md	Dense embedding providers (Jina, Voyage, OpenAI, Ollama), the embedding text strategy, and provider selection
sparse-vectors.md	BM25 sparse embeddings, the code identifier tokenizer, and the sparse vector format
retrieval-rrf.md	Hybrid search architecture, RRF fusion, name lookup, and the four MCP tool entry points
configuration.md	All environment variables, config.yaml structure, dynamic service registration, and startup validation
deployment.md	Docker Compose setup, embedding paths, curated config.yaml services, GitHub token at scale, and MCP transports
http-api.md	POST /reindex and POST /reindex-history, inline repo registration, and NDJSON frame shapes
qdrant-collections.md	The code_symbols, git_commits and service_registry collections: vectors, HNSW settings, and payload fields

---

Quick Start

1. Configure services — either copy config.example.yaml to config.yaml and add your GitHub repositories, or skip config.yaml entirely and register repos on the fly via POST /reindex (see the GitHub Actions example) — the two can also be combined. See configuration.md for all fields and how the two interact.
2. Set environment variables — copy .env.example to .env and set at minimum GITHUB_TOKEN. The default embedding provider (jina) requires a locally running TEI container; for a hosted alternative, set EMBEDDINGS_PROVIDER=voyage and VOYAGE_API_KEY=....
3. Start Qdrant and the server — make docker-up-jina (local Jina) or make docker-up (hosted provider or Ollama), then connect your MCP client to http://localhost:8090/mcp. See deployment.md.

For full setup instructions, see the root README.