Quick Start

XFaiss provides a ./xfaiss.sh script for building, data preparation, and benchmarking with predefined configurations:

Step Command Description
Build ./xfaiss.sh build Build XFaiss library including tutorials and benchmarks
Prepare Data ./xfaiss.sh prepare-data-tui Prepare dataset and index files (TUI)
  ./xfaiss.sh prepare-data [options] Prepare dataset and index files (CLI)
Benchmark ./xfaiss.sh bench-tui Run search benchmark (TUI)
  ./xfaiss.sh bench <preset> [options] Run search benchmark (CLI, see Benchmark Reference)

Prepare Data

prepare-data downloads public benchmark datasets, converts formats, and builds Faiss indexes. prepare-data-tui provides the same functionality with an interactive TUI.

Datasets

Dataset Dimension Vectors Metric Download Size
sift1m 128 1M L2 ~500MB
gist1m 960 1M L2 ~4GB
sift100m 128 100M L2 ~49GB
deep100m 96 100M L2 ~358GB
7m-d1024-l2 1024 7M L2 generated
7m-d1024-ip 1024 7M IP generated

Pipeline

  1. Download — fetch raw datasets from public sources into {data_dir}/downloads/
  2. Prepare vectors — convert base/query vectors to .fvecs and prepare (or compute) ground truth under {data_dir}/indexes/
  3. Build indexes — create IVF-Flat, IVF-RaBitQ, and (for SIFT-1M / GIST-1M) HNSW-Flat (experimental) indexes

Output (per dataset, under {data_dir}/indexes/{dataset}/):

{data_dir}/indexes/{dataset}/
  ├── base.fvecs
  ├── query.fvecs
  ├── groundtruth.ivecs
  ├── {dataset}.ivf_flat.{metric}.nlist4096.faiss
  ├── {dataset}.ivf_rabitq.{metric}.nlist4096.faiss
  └── {dataset}.hnsw_flat.{metric}.M32.faiss   # SIFT-1M / GIST-1M only

{metric} is l2 or ip. IVF indexes use nlist=1000 for the 7M-d1024 datasets.

CLI Options 

./xfaiss.sh prepare-data --data-dir <dir> [options...]
Option Description Default
--data-dir <dir> Root data directory /var/opt/xvector-data
--datasets <list> Comma-separated dataset names all

TUI

prepare-data-tui opens a TUI to select datasets for download and index generation:

$ ./xfaiss.sh prepare-data-tui
Select datasets (space to toggle, enter to confirm):
  ✓ sift1m        SIFT 1M (128-dim, ~500MB download)
  • gist1m        GIST 1M (960-dim, ~4GB download)
  • sift100m      SIFT 100M / BigANN (128-dim, 1B raw ~49GB download)
  • deep100m      Deep 100M (96-dim, 1B raw ~358GB download)
  ✓ 7m-d1024-l2   Synthetic 7M (1024-dim, L2, generated)
> ✓ 7m-d1024-ip   Synthetic 7M (1024-dim, IP, generated)

Benchmark

bench-tui opens a TUI to select a benchmark preset and configure parameters:

1. Select benchmark preset:

$ ./xfaiss.sh bench-tui
Select benchmark preset:
> ivf-flat-sift1m                     IVF-Flat on SIFT-1M
  ivf-flat-gist1m                     IVF-Flat on GIST-1M
  ivf-flat-sift100m                   IVF-Flat on SIFT-100M
  ivf-flat-deep100m                   IVF-Flat on Deep-100M
  ivf-flat-7m-d1024-l2-query-fixed    IVF-Flat on 7M-d1024 (L2, Query-Fixed)
  ivf-flat-7m-d1024-l2-cluster-fixed  IVF-Flat on 7M-d1024 (L2, Cluster-Fixed) [experimental]
  ivf-flat-7m-d1024-ip-query-fixed    IVF-Flat on 7M-d1024 (IP, Query-Fixed)
  ivf-flat-7m-d1024-ip-cluster-fixed  IVF-Flat on 7M-d1024 (IP, Cluster-Fixed) [experimental]
  ivf-rabitq-sift1m                   IVF-RaBitQ on SIFT-1M
  ivf-rabitq-gist1m                   IVF-RaBitQ on GIST-1M
  ivf-rabitq-sift100m                 IVF-RaBitQ on SIFT-100M
  ivf-rabitq-deep100m                 IVF-RaBitQ on Deep-100M
  ivf-flat-fullscan-sift1m            IVF-Flat full scan (nprobe=nlist) on SIFT-1M
  ivf-flat-fullscan-gist1m            IVF-Flat full scan (nprobe=nlist) on GIST-1M
  knn-exact-sift1m-l2                 KNN Exact (MuIndexFlat, L2) on SIFT-1M
  knn-exact-sift1m-ip                 KNN Exact (MuIndexFlat, IP) on SIFT-1M
  knn-exact-gist1m-l2                 KNN Exact (MuIndexFlat, L2) on GIST-1M
  knn-exact-gist1m-ip                 KNN Exact (MuIndexFlat, IP) on GIST-1M
  hnsw-flat-sift1m                    HNSW-Flat on SIFT-1M
  hnsw-flat-gist1m                    HNSW-Flat on GIST-1M

2. Review configurations and run:

  Preset: ivf-flat-sift1m
> ── Data ────────────────────────
    DATA_DIR           /var/opt/xvector-data
  ── Search ──────────────────────
    NPROBE             128
    TOPK               100
    SKIP_CPU           no
    MAX_QUERIES        (all)
    RANDOM_QUERIES     (preset default)
    SEARCH_ITERATIONS_ON_MU (1)
    SEARCH_ITERATIONS_ON_CPU (1)
  ── Search (IVF-Flat) ───────────
    FLAT_PARALLEL_MODE 1 (QUERY_FIXED)
  ── Device ──────────────────────
    DEVICEID           0
    NUMSUB             24
    TASKCOUNT          192
    LOCALITY_MODE      1 (SPREAD)
  ── CPU / Host ──────────────────
    NUMA_CPUBIND           0
    NUMA_MEMBIND           0
    OMP_NUM_THREADS        64
    OPENBLAS_NUM_THREADS   1
    GDB                    no
  ────────────────────────────────
    >>> Run benchmark
    <<< Change preset

Next Steps