Feval

# Feval

Feval is a Bittensor subnet for evaluating LoRA improvements to a fixed base
model. Miners publish a model and its rollouts, validators verify the work and
independently set weights from the same code-pinned protocol rules.

## Install

Use Linux or WSL with an NVIDIA GPU, a working CUDA driver, Python 3.12, and a
Bittensor wallet.

```bash
git clone <repository-url> feval
cd feval
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e .
feval --help
```

Copy the environment template only when credentials or reporting settings are
needed:

```bash
cp .env.example .env
```

The CLI reads `.env` from the current working directory before running a
command. Run it from the directory containing your `.env`; values already
exported in the shell take precedence over that file.

Set `HF_TOKEN` on miner hosts that upload to Hugging Face. Validators do not
need a Hugging Face token when model and rollout repositories are public.
Never commit `.env`, wallet files, private keys, tokens, or validator state.

Each twelve-hour window draws 100,000 rows from six pinned public Hugging Face
datasets whose raw candidate union contains well over one million rows. Sources
are never downloaded whole: parquet footers give a row-group map for free, and
a seed revealed at the window boundary picks which row groups (or, for a JSONL
source, which byte blocks) to read, fetching only the columns Feval grades on.
The first cycle for a window reports per-source scan progress; the
unauthenticated-request warning is harmless for public repositories.

## Repository layout

```text
.
+-- src/feval/
|   +-- cli/          # Command-line interface
|   +-- core/         # Network config and protocol constants
|   +-- protocol/     # Deterministic seeds, Merkle roots, and submissions
|   +-- datasets/     # Source sampling, task normalisation, and verifiers
|   +-- models/       # LoRA artifacts, rollout generation, and inference
|   +-- chain/        # Bittensor commitments, axon serving, and weights
|   +-- nodes/        # Miner and validator runtime loops
|   `-- utils/        # Small shared helpers
+-- docs/assets/      # Documentation assets
+-- network.json      # Reviewed local overrides for code-pinned defaults
+-- SECURITY.md       # Trust model and validator hardening notes
+-- THIRD_PARTY_NOTICES.md # Dataset attribution and license summary
`-- pyproject.toml    # Package metadata and dependencies
```

## Miner

A miner trains a PEFT LoRA for the base model pinned in `network.json`. The
adapter directory must contain:

```text
adapter_config.json
adapter_model.safetensors
```

Create one Hugging Face model repository and one Hugging Face dataset
repository for rollouts. Publish the adapter and commit its immutable revision
on chain:

```bash
feval miner publish-model \
  --config network.json \
  --adapter-dir output/my-lora \
  --model-repo <hf-account>/<model-repo> \
  --rollout-repo <hf-account>/<rollout-repo> \
  -w <wallet-name> -H <hotkey-name> -n finney
```

Keep the current evaluation-window rollouts updated:

```bash
feval miner watch-rollouts \
  --config network.json \
  --adapter-dir output/my-lora \
  --work-dir miner-work \
  --max-new-tokens 32768 \
  --batch-size 4 \
  --poll-seconds 60 \
  -w <wallet-name> -H <hotkey-name> -n finney
```

The watcher uploads a new rollout when the evaluation window or committed
model changes. Publish the model again before generating rollouts whenever the
adapter bytes change. Use `feval miner status --help` and
`feval miner leaderboard --help` to inspect public validator results.

## Validator

A validator reads miner commitments, pins immutable Hugging Face revisions,
scores each complete rollout set, audits sampled token traces with the
committed LoRA, and submits weights.

The `feval196-feval/feval-valid` W&B project is Open, so validators do not need
an invitation or team membership. Each validator authenticates with its own
W&B account by setting `WANDB_API_KEY`, and Feval automatically publishes its
public result summary to this fixed destination. Reporting failures are visible
in validator status but never change evaluation or emitted weights.

The published leaderboard table includes each miner's `model_repo`,
`rollout_repo`, and `invalid_reason`. The invalid reason is empty while a miner
is still auditing or an operational retry is in progress. It contains only a
sanitized audit-gate outcome; internal system errors are not published.

Set `FEVAL_REPORT_WANDB=0` in `.env` to disable automatic reporting. The public
destination is `WANDB_ENTITY=feval196-feval` and `WANDB_PROJECT=feval-valid`;
validator results always use that shared destination.

Run continuously:

```bash
feval validator run \
  --config network.json \
  --work-dir validator-work \
  --state validator-state.json \
  -w <wallet-name> -H <hotkey-name> -n finney
```

Feval automatically starts vLLM workers with `spawn` and uses the native sampler
to avoid fork-related startup hangs and FlashInfer sampler warmup failures.
No environment exports are needed for these defaults. Explicit
`VLLM_WORKER_MULTIPROC_METHOD` and `VLLM_USE_FLASHINFER_SAMPLER` settings still
override them.

Keep `validator-state.json` and its backup between restarts. To verify a setup
without submitting weights, run one cycle with:

```bash
feval validator run \
  --config network.json \
  --work-dir validator-work \
  --state validator-state.json \
  --once --dry-run-weights \
  -w <wallet-name> -H <hotkey-name> -n <network>
```

Check liveness with `feval health --state validator-state.json`. See
`feval validator export-results --help` to publish a sanitized result summary.

## Evaluation protocol

Every 3,600 finalized blocks (approximately twelve hours), miners and validators
derive the same 100,000-row evaluation set from immutable, code-pinned dataset
revisions. A model must be committed before the first block of the window. The
hash of that finalized boundary block reveals the evaluation seed, so a