H.A.R.B.O.R
Hierarchical Analysis for Reference-Guided Biodiversity Organism Recognition
AI-Driven eDNA Taxonomy Inference | Bioinformatics
Overview
HARBOR is an open-set eDNA taxonomy inference system built for large-scale biodiversity recognition from environmental DNA sequences. Traditional taxonomy systems force every sequence into an existing classification – even when the sequence belongs to an unknown organism. HARBOR solves this by combining fast deterministic similarity checks, deep CNN-based sequence embeddings, and statistical novelty detection to produce three honest output types: high-confidence known assignments, explicit NOVEL flags, and explainability metrics. It is designed to scale to multi-marker reference databases (~2.8GB) in alignment-free mode without sacrificing scientific defensibility.
Key Features
Open-Set / Novelty-Aware Classification
Distinguishes known sequences from out-of-distribution ones. Novel sequences receive explicit NOVEL flags.
Multi-Stage Hybrid Inference
Three-stage pipeline: Deterministic similarity → Embedding-based ANN search → Statistical novelty detection.
Marker-Aware Representation Learning
Learns biologically meaningful representations per genetic marker, preventing cross-marker interference.
High-Throughput Similarity Retrieval
FAISS-based similarity indices enabling low-latency nearest-neighbor search over millions of sequences.
Scientific Traceability + MLOps
Full dataset versioning, experiment tracking via DVC and Weights & Biases.
Risk-Controlled System Engineering
Designed to mitigate: reproducibility loss, dependency drift, data corruption, model–index mismatch.
Technology Stack
Algorithms & Methods
Deterministic Feature Engineering
Sliding-window k-mer encoding with hash-based conversion, alignment-free.
Deep Representation Learning (CNN)
Marker-specific CNN: Embedding → 1D Conv → Global pooling → Projection + normalization.
Contrastive Learning
Unsupervised instance-level contrastive learning, no taxonomy label dependency.
FAISS ANN Retrieval
Per-marker FAISS indices for low-latency Top-K nearest neighbor search.
Statistical Novelty Detection
Distance-based and density-based novelty checks with explicit NOVEL flags.
System Architecture
Data Ingestion + Marker Separation
Curated reference FASTA ingestion, marker-wise separation, taxonomy annotation
Deterministic Preprocessing
Sliding-window feature extraction, hash encoding, invalid nucleotide filtering
Deterministic Similarity Resolver
High-confidence known match detection (fast path)
Marker-Specific Embedding Model
CNN embedding generation per marker, normalized vector output
FAISS Index Layer
Per-marker ANN indices, low-latency Top-K retrieval
Statistical Novelty Detection Layer
Distance/density novelty logic, unsafe assignment rejection
MLOps Layer
DVC versioning, W&B tracking, cron automation, Docker runtime
Results & Outcomes
- Alignment-free inference pipeline operational at multi-marker scale (~2.8GB reference databases)
- Open-set taxonomy classification with honest NOVEL detection
- Explainable, auditable outputs with confidence and similarity evidence
- MLOps infrastructure for reproducibility and drift monitoring