Desearch

<h3 align="center">
  <a name="readme-top"></a>
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="./docs/assets/desearch-logo.png">
    <img src="./docs/assets/desearch-logo-black.png" alt="Desearch" width="360">
  </picture>
</h3>

<div align="center">
  <a href="./LICENSE"><img src="https://img.shields.io/github/license/Desearch-ai/subnet-22" alt="License"></a>
  <a href="https://github.com/Desearch-ai/subnet-22/graphs/contributors"><img src="https://img.shields.io/github/contributors/Desearch-ai/subnet-22.svg" alt="Contributors"></a>
  <a href="https://www.desearch.ai/network"><img src="https://img.shields.io/badge/Bittensor-Subnet%2022-blue" alt="Bittensor Subnet 22"></a>
  <a href="https://desearch.ai"><img src="https://img.shields.io/badge/Visit-desearch.ai-orange" alt="Visit desearch.ai"></a>
</div>

<p align="center">
  <a href="https://x.com/desearch_ai"><img src="https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white" alt="Follow on X"></a>
  <a href="https://www.linkedin.com/company/desearch-ai/"><img src="https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" alt="Follow on LinkedIn"></a>
  <a href="https://discord.com/invite/eb6DTZNMF5"><img src="https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join our Discord"></a>
</p>

---

# Desearch

**A web index for AI, built through open competition on Bittensor.**

AI applications need relevant, current sources and text they can use, soon enough to finish their
task. Desearch collects and refreshes useful pages from the public web, prepares them for retrieval
and develops the models that help AI find the right sources. Search is the first application: fast
search for AI agents, and deeper research for questions that take longer.

The work happens in the open on Bittensor Subnet 22. Miners build and refresh the collection,
validators check their work under published rules, and the Desearch team integrates accepted work
into a tested index and the search API.

<p align="center">
  <a href="./docs/miner.md">⛏️ Mine</a> ·
  <a href="./docs/validator.md">🛡️ Validate</a> ·
  <a href="./docs/architecture.md">⚙️ How it works</a> ·
  <a href="./docs/desearch-2.0/README.md">🧭 Desearch 2.0</a> ·
  <a href="https://console.desearch.ai">🔑 Search API</a>
</p>

---

## Why an index

In Desearch 1.0, miners answered search requests and validators judged the answers. It worked, but
good answers could arrive too late, some depended on outside search providers, and little of the
work left anything that could be kept and improved.

Desearch 2.0 prepares information before a query arrives. Pages are collected, cleaned,
deduplicated, organized and prepared for retrieval in the background, so a query searches a
maintained collection instead of starting from scratch. That gives:

- **Faster retrieval**: expensive preparation moves out of the query path
- **More control over quality**: coverage, freshness and supporting text can be inspected, and models
  improved against the same collection
- **Work that lasts**: every accepted page serves many later searches, not one answer

The collection covers the public web: company websites, news, articles, blogs and documentation. It
starts with selected sources and grows as results prove useful.

## What the network builds

| Work | Who does it | Status |
| --- | --- | --- |
| **Crawl** and refresh pages, extract their text | Miners fetch assigned pages; validators re-check a sample of their tasks | Running on SN22 |
| **Embed** accepted text with a selected model | Miners run the model on their GPUs; validators recompute a sample | Running on SN22, with Desearch's embedding model |
| Train retrieval models, improve ranking, extraction and coverage | Model builders and data operators | Later programs, announced before they open |
| Build, test and serve the index | The Desearch team | Team-operated; queries never wait on miner work |

## Mining and validating

**Miners** crawl and embed: they take tasks, fetch the assigned pages and upload the extracted text,
and turn that text into vectors on their own GPU. They are paid by their share of accepted work, and
can test their miner locally before registering.
[What a miner does →](./docs/miner.md#-what-a-miner-does) · [Miner setup →](./docs/miner.md#-setup) ·
[Test locally →](./docs/test-your-miner.md) · [Emission →](./docs/emission.md)

**Validators** check miners' work, the pages they crawl and the vectors they embed, and set weights
from the work that holds up.
[What a validator does →](./docs/validator.md#-what-a-validator-does) ·
[Validator setup →](./docs/validator.md#-setup)

The rules are published and every result is [public](https://api-22.desearch.ai/v1/tasks).
[How it works →](./docs/architecture.md)

## Use Desearch

The search API is available today for developers building AI agents and research tools. Get a key in
the [console](https://console.desearch.ai) and start with the
[documentation](https://www.desearch.ai/docs/guide/introduction/desearch-ai).

| SDK | Install |
| --- | --- |
| [Python](https://github.com/Desearch-ai/desearch.py) | `pip install desearch-py` |
| [JavaScript / TypeScript](https://github.com/Desearch-ai/desearch.js) | `npm install desearch-js` |
| [MCP server](https://github.com/Desearch-ai/mcp-desearch) | `npm install -g desearch-mcp-server` |

## Documentation

| Guide | What's inside |
| --- | --- |
| ⛏️ [Miner](./docs/miner.md) | [What a miner does](./docs/miner.md#-what-a-miner-does) and how it earns; [setup](./docs/miner.md#-setup): install, register, configure, run, monitor |
| 🧪 [Test your miner locally](./docs/test-your-miner.md) | Run your miner against a local task API and validator before registering |
| 🛡️ [Validator](./docs/validator.md) | [What a validator does](./docs/validator.md#-what-a-validator-does): checks and weights; [setup](./d