Enuxia-Public/enuxia-vacuum
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update documentation to reflect VastuSLAM rename and add evaluation module

Browse Source 
- Rename DhruvaSLAM references to VastuSLAM throughout CLAUDE.md
- Update module organization to reflect current vastu-slam structure
- Add evaluation module to README.md module table
- Remove obsolete .gitignore and BENCHMARK_LOG.md files
- Minor code formatting fixes in benchmarks
This commit is contained in:
codetiger
2026-01-01 14:54:32 +05:30
parent 95ed452773
commit e66268db0f
7 changed files with 301 additions and 314 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+21 -20
View File
@@ -7,7 +7,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
VacuumTiger is an open-source robotic vacuum firmware project for the CRL-200S hardware platform. The project consists of three main components:
1. **SangamIO**: A daemon providing hardware abstraction and real-time control via TCP/UDP streaming
2. **DhruvaSLAM**: A multi-threaded 2D SLAM daemon for localization and mapping
2. **VastuSLAM**: A high-performance 2D SLAM library for robot vacuum cleaners
3. **Drishti**: Python visualization and control client for monitoring and commanding the robot
**Target Platform**: Embedded Linux (ARM) - Allwinner A33 running Tina Linux
@@ -126,7 +126,7 @@ All messages use length-prefixed framing with Protobuf payloads:
### Proto Schema
See `sangam-io/proto/sangamio.proto` for SangamIO messages (sensors, commands) and `dhruva-slam/proto/dhruva.proto` for SLAM output messages (odometry, map, scan).
See `sangam-io/proto/sangamio.proto` for SangamIO messages (sensors, commands).
### Topics (UDP Streaming)
@@ -256,20 +256,22 @@ VacuumTiger/
│ ├── sangamio.toml # Hardware configuration
│ └── mock.toml # Simulation configuration
│
├── dhruva-slam/ # 2D SLAM daemon
├── vastu-slam/ # High-performance 2D SLAM library
│ ├── src/
│ │ ├── main.rs # Entry point, multi-threaded daemon
│ │ ├── core/ # Foundation types (Pose2D, PointCloud2D)
│ │ ├── sensors/ # Odometry fusion, lidar preprocessing
│ │ ├── algorithms/ # Matching, mapping, planning, descriptors
│ │ ├── engine/ # SLAM orchestration, pose graph
│ │ ├── io/ # SangamIO client, streaming, bags
│ │ ├── state/ # Multi-threaded state management
│ │ ├── threads/ # SLAM, publisher, exploration, navigation
│ │ ├── exploration/ # Frontier-based exploration
│ │ └── navigation/ # Path planning and execution
│ ├── proto/ # Protobuf schema (dhruva.proto)
│ └── dhruva-slam.toml # Runtime configuration
│ │ ├── lib.rs # Library entry point
│ │ ├── core/ # Foundation types (Pose2D, WorldPoint, GridCoord, sensors)
│ │ ├── grid/ # Occupancy grid storage (SoA layout, log-odds updates)
│ │ ├── matching/ # Scan matching (correlative, branch-bound, loop closure)
│ │ ├── submap/ # Submap manager, pose graph, multi-submap matching
│ │ ├── config/ # YAML configuration loading
│ │ ├── io/ # Persistence (.vastu, .pgm, .svg formats)
│ │ ├── modes/ # Operation modes (localization-only)
│ │ └── evaluation/ # Cartographer-style evaluation metrics
│ ├── configs/ # YAML configuration files
│ ├── maps/ # Test maps (PGM format)
│ ├── scenarios/ # Test scenarios (YAML)
│ ├── benches/ # Performance benchmarks
│ └── tests/ # Integration tests
│
└── drishti/ # Python visualization client
├── drishti.py # Console client
@@ -410,19 +412,18 @@ python drishti.py --robot 192.168.68.101 --verbose
- **v0.2.0**: Daemon architecture, performance optimizations
- **v0.1.0**: Initial release with basic functionality
### DhruvaSLAM
- **v0.1.0**: Multi-threaded SLAM daemon with exploration and navigation
### VastuSLAM
- **v0.1.0**: High-performance 2D SLAM library with scan matching, submap architecture, and loop closure
## Documentation References
- **sangam-io/README.md**: SangamIO daemon documentation
- **sangam-io/COMMANDS.md**: GD32 command reference with TCP API
- **sangam-io/SENSORSTATUS.md**: Sensor packet byte layout
- **dhruva-slam/README.md**: SLAM daemon documentation
- **dhruva-slam/src/*/README.md**: Module-specific documentation (core, sensors, algorithms, engine, io)
- **vastu-slam/README.md**: VastuSLAM library documentation
- **vastu-slam/src/*/README.md**: Module-specific documentation (core, grid, matching, submap, config, io, modes, evaluation)
- **drishti/README.md**: Python client documentation
- **protocol-mitm/README.md**: MITM reverse engineering tools
- **slam-test/README.md**: SLAM integration test framework
## Future Roadmap
vastu-slam/.gitignore
-1
View File
@@ -1 +0,0 @@
multithread-proposal.md
vastu-slam/README.md
+1
View File
@@ -196,6 +196,7 @@ Priority (highest to lowest):
| [`config`](src/config/) | YAML configuration loading with sensible defaults | [README](src/config/README.md) |
| [`io`](src/io/) | Persistence: native .vastu format, ROS .pgm export, SVG visualization | [README](src/io/README.md) |
| [`modes`](src/modes/) | Operation modes: localization-only (no map updates) | [README](src/modes/README.md) |
| [`evaluation`](src/evaluation/) | Cartographer-style evaluation: ATE, RPE, pose relations metrics | [README](src/evaluation/README.md) |
## Coordinate System
vastu-slam/benches/evaluation.rs
+4 -2
View File
@@ -1,13 +1,13 @@
//! Benchmark evaluation metrics performance.
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
use vastu_slam::LidarScan;
use vastu_slam::Pose2D;
use vastu_slam::evaluation::{
AbsoluteTrajectoryError, GroundTruthConfig, GroundTruthRelations, RelationsMetrics,
RelativePoseError,
};
use vastu_slam::matching::loop_closure::LidarIris;
use vastu_slam::LidarScan;
fn create_trajectory(n: usize, spacing: f32) -> Vec<Pose2D> {
(0..n)
@@ -115,7 +115,9 @@ fn bench_rpe(c: &mut Criterion) {
fn create_test_scan(n: usize) -> LidarScan {
use std::f32::consts::PI;
let angles: Vec<f32> = (0..n).map(|i| -PI + i as f32 * 2.0 * PI / n as f32).collect();
let angles: Vec<f32> = (0..n)
.map(|i| -PI + i as f32 * 2.0 * PI / n as f32)
.collect();
let ranges: Vec<f32> = angles.iter().map(|a| 2.0 + (a * 3.0).sin() * 0.5).collect();
LidarScan::new(ranges, angles, 0.15, 8.0)
}
vastu-slam/benches/grid_operations.rs
+1 -1
View File
@@ -2,8 +2,8 @@
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
use std::f32::consts::PI;
use vastu_slam::{LidarScan, MapConfig, OccupancyGridMap, Pose2D};
use vastu_slam::matching::PrecomputedGrids;
use vastu_slam::{LidarScan, MapConfig, OccupancyGridMap, Pose2D};
/// Create a room scan for benchmarking.
fn room_scan(
vastu-slam/benchmarks/BENCHMARK_LOG.md
-290
View File
@@ -1,290 +0,0 @@
# VastuSLAM Benchmark Log
This file tracks performance, accuracy, and quality metrics across commits for optimization reference.
## Standard Metrics Framework
### 1. Performance Metrics (Latency)
| Metric | Description | Target | Unit |
|--------|-------------|--------|------|
| `scan_matching_180` | Scan matching with 180 points | <1.5ms | µs |
| `scan_matching_360` | Scan matching with 360 points (typical) | <3.0ms | µs |
| `scan_matching_720` | Scan matching with 720 points (high-res) | <10ms | µs |
| `grid_update_180` | Grid update with 180 points | <200µs | µs |
| `grid_update_360` | Grid update with 360 points | <400µs | µs |
| `grid_update_720` | Grid update with 720 points | <800µs | µs |
| `coverage_stats` | Coverage computation (800x800 grid) | <200µs | µs |
| `ate_100` | ATE computation (100 poses) | <5µs | µs |
| `rpe_100` | RPE computation (100 poses) | <5µs | µs |
| `relations_metrics_100` | Relations metrics (100 poses) | <100µs | µs |
| `ground_truth_gen_100` | Ground truth generation (100 poses) | <50µs | µs |
### 2. Accuracy Metrics
| Metric | Description | Target | Unit |
|--------|-------------|--------|------|
| `ate_trans_mean` | ATE translation mean error | <0.01 | m |
| `ate_rot_mean` | ATE rotation mean error | <0.5 | deg |
| `rpe_trans_mean` | RPE translation mean error | <0.005 | m |
| `rpe_rot_mean` | RPE rotation mean error | <0.3 | deg |
| `relations_trans_mean` | Relations translation error | <0.01 | m |
| `relations_rot_mean` | Relations rotation error | <0.5 | deg |
### 3. Quality Metrics
| Metric | Description | Target | Unit |
|--------|-------------|--------|------|
| `loop_closure_precision` | Loop closure true positive rate | >0.9 | ratio |
| `loop_closure_recall` | Loop closure detection rate | >0.8 | ratio |
| `map_wall_count` | Detected wall cells in test room | >100 | cells |
| `map_floor_count` | Detected floor cells in test room | >1000 | cells |
### 4. Test Coverage
| Metric | Description |
|--------|-------------|
| `unit_tests_passed` | Number of unit tests passing |
| `integration_tests_passed` | Number of integration tests passing |
| `all_tests_pass` | Boolean: all tests pass |
---
## Benchmark History
### Entry: 2026-01-01 (Grid Pyramid Parallelization)
**Commit:** `db285a3`
**Message:** Parallelize grid pyramid construction with rayon
**Platform:** macOS Darwin 25.2.0 (Apple Silicon, 10 cores)
**Rust:** nightly (portable_simd feature)
#### Grid Pyramid Build Performance
| Metric | Sequential | Parallel | Speedup |
|--------|------------|----------|---------|
| grid_pyramid_863x860 | 1230 µs | 670.43 µs | **1.84x** |
#### Implementation Details
| Component | Cells | Parallelization |
|-----------|-------|-----------------|
| Level 0 | 742K | `par_iter` over flat index |
| Level 1 | 185K | `par_iter` within level |
| Level 2 | 46K | `par_iter` within level |
| Level 3 | 11K | `par_iter` within level |
#### Notes
- Parallelized Level 0 construction (742K cells with Gaussian exp())
- Parallelized coarse levels within each level (max of 2x2 regions)
- Levels must be built sequentially (level i depends on level i-1)
- Also parallelized `update_from_storage()` for incremental updates
- 1.84x speedup (45% improvement) on 10-core Apple Silicon
---
### Entry: 2026-01-01 (Rayon Parallelization)
**Commit:** `a1e3d28e6d5c77f0107a89ee95251ba8b59c1ad1`
**Message:** Add rayon parallelization for scan matching
**Platform:** macOS Darwin 25.2.0 (Apple Silicon, 10 cores)
**Rust:** nightly (portable_simd feature)
#### Parallel vs Sequential Performance
| Metric | Sequential | Parallel | Speedup |
|--------|------------|----------|---------|
| scan_matching_360 | 1716.1 µs | 448.75 µs | **3.82x** |
#### Configuration
| Setting | Value |
|---------|-------|
| Rayon version | 1.10 |
| Default threads | auto (CPU cores) |
| Parallelization | Outer x-loop in search_poses |
#### Test Results
| Category | Passed | Total | Status |
|----------|--------|-------|--------|
| Unit tests | 195 | 195 | ✅ |
| Accuracy tests | 10 | 10 | ✅ |
| Performance tests | 6 | 6 | ✅ |
| Quality tests | 10 | 10 | ✅ |
| **Total** | **221** | **221** | ✅ |
#### Changes Made
- Added `rayon = "1.10"` dependency to Cargo.toml
- Implemented `search_poses_parallel()` in `matching/matcher/core.rs`
- Added `dispatch_search_poses()` to choose between sequential/parallel
- Added `use_parallel: bool` config flag in `matching/config.rs`
- Added `configure_thread_pool(threads, stack_kb)` API in `lib.rs`
- Updated benchmarks to compare sequential vs parallel
#### API Usage
```rust
// Configure thread pool at startup (optional, for embedded)
vastu_slam::configure_thread_pool(3, 256); // 3 threads, 256KB stack
// Enable parallel matching
let config = CorrelativeMatcherConfig {
use_parallel: true,
..Default::default()
};
```
#### Notes
- 3.82x speedup on 10-core Apple Silicon
- Expected 2-3x speedup on 4-core ARM (Allwinner A33)
- Default: parallel disabled (use_parallel: false) for backwards compatibility
- Thread pool defaults to CPU core count if not configured
---
### Entry: 2025-12-31
**Commit:** `1321540e164d6955c07759ce8a200a7313bcb57a`
**Message:** Simplify vastu-slam configuration and increase default grid size
**Platform:** macOS Darwin 25.2.0 (Apple Silicon)
**Rust:** nightly (portable_simd feature)
#### Performance Results
| Metric | Value | Change | Status |
|--------|-------|--------|--------|
| scan_matching_180 | 964.96 µs | -3.1% | ✅ improved |
| scan_matching_360 | 1793.7 µs | -2.9% | ✅ improved |
| scan_matching_720 | 6546.6 µs | -3.3% | ✅ improved |
| grid_update_180 | 104.96 µs | +0.3% | ⚪ stable |
| grid_update_360 | 209.88 µs | +0.4% | ⚪ stable |
| grid_update_720 | 418.05 µs | -0.5% | ⚪ stable |
| coverage_stats | 125.15 µs | -4.8% | ✅ improved |
| ate_100 | 1.51 µs | +0.1% | ⚪ stable |
| rpe_100 | 3.04 µs | -0.3% | ⚪ stable |
| relations_metrics_100 | 58.32 µs | +0.6% | ⚪ stable |
| ground_truth_gen_100 | 19.96 µs | -0.1% | ⚪ stable |
#### Throughput Analysis
| Operation | Rate | Real-time Factor |
|-----------|------|------------------|
| Scan matching (360pt) | 557 scans/sec | 111x @ 5Hz lidar |
| Grid update (360pt) | 4765 updates/sec | 953x @ 5Hz lidar |
| Combined (match + update) | 499 cycles/sec | 100x @ 5Hz lidar |
#### Test Results
| Category | Passed | Total | Status |
|----------|--------|-------|--------|
| Unit tests | 195 | 195 | ✅ |
| Accuracy tests | 10 | 10 | ✅ |
| Performance tests | 6 | 6 | ✅ |
| Quality tests | 10 | 10 | ✅ |
| **Total** | **221** | **221** | ✅ |
#### Notes
- First baseline entry after config simplification
- Scan matching improved ~3% (likely due to reduced config overhead)
- Coverage stats improved ~5% (SIMD optimization benefit)
- All accuracy/quality thresholds met
---
## Benchmark Procedures
### Running Benchmarks
```bash
# Full benchmark suite
cd vastu-slam
cargo +nightly bench
# Specific benchmark group
cargo +nightly bench --bench scan_matching
cargo +nightly bench --bench grid_operations
cargo +nightly bench --bench evaluation
# Save results to file
cargo +nightly bench 2>&1 | tee benchmark_output.txt
```
### Running Tests
```bash
# All tests (debug mode)
cargo test
# Release mode tests (faster)
cargo test --release
# Specific test category
cargo test --test accuracy
cargo test --test performance
cargo test --test quality
```
### Adding New Benchmark Entry
1. Run full benchmark suite: `cargo +nightly bench`
2. Run all tests: `cargo test`
3. Get commit info: `git rev-parse HEAD && git log -1 --format='%s'`
4. Add new entry section below with results
5. Calculate percentage changes from previous entry
6. Note any significant changes or regressions
### Regression Detection
A regression is flagged when:
- Scan matching latency increases >10%
- Grid update latency increases >20%
- Any accuracy test fails
- Any metric exceeds its target threshold
---
## Target Platform Performance
### ARM Cortex-A7 @ 1GHz (Allwinner A33)
Expected performance scaling from Apple Silicon:
- Scan matching: ~5-15ms (10x slower)
- Grid update: ~1-2ms (10x slower)
- Real-time factor: ~10-20x @ 5Hz lidar (still real-time capable)
### Memory Footprint
| Component | Size |
|-----------|------|
| Binary (stripped, musl) | ~350 KB |
| Grid storage (800x800) | ~2.5 MB |
| Submap (200x200 + 60 scans) | ~324 KB |
| Total runtime (typical) | <10 MB |
---
## Historical Trends
```
scan_matching_360 (µs)
├─ 2026-01-01: 448.75 µs (parallel, 3.82x speedup)
├─ 2026-01-01: 1716.1 µs (sequential, inlined)
├─ 2025-12-31: 1793.7 µs (baseline)
│
grid_pyramid_863x860 (µs)
├─ 2026-01-01: 670.43 µs (parallel, 1.84x speedup)
├─ 2026-01-01: 1230 µs (sequential)
│
grid_update_360 (µs)
├─ 2025-12-31: 209.88 µs (baseline)
│
coverage_stats (µs)
├─ 2025-12-31: 125.15 µs (baseline)
```
*Add new entries above as benchmarks are run*
vastu-slam/src/evaluation/README.md
+274
View File
@@ -0,0 +1,274 @@
# Evaluation Module
Cartographer-style evaluation framework for VastuSLAM accuracy, performance, and quality metrics.
## Overview
The evaluation framework is built on three pillars:
1. **Accuracy** - Pose relations metrics, ATE (Absolute Trajectory Error), RPE (Relative Pose Error)
2. **Performance** - Latency benchmarks, resource usage tracking
3. **Quality** - Map consistency, loop closure precision/recall
## Cartographer-Style Evaluation
The core idea is to evaluate using **relative poses** between trajectory nodes, which works without external ground truth (GPS, motion capture).
### Two-Step Process
1. **Generate ground truth**: Extract pose relations from an optimized trajectory (with loop closures applied)
2. **Evaluate**: Compare a test trajectory against these ground truth relations
```rust
use vastu_slam::evaluation::{GroundTruthRelations, RelationsMetrics, GroundTruthConfig};
// Step 1: Generate ground truth from optimized trajectory
let config = GroundTruthConfig::default();
let ground_truth = GroundTruthRelations::generate(
&optimized_poses,
&loop_closures,
config,
);
// Step 2: Evaluate test trajectory
let metrics = RelationsMetrics::compute(&test_poses, &ground_truth);
metrics.print();
// Output:
// Abs translational error 0.00832 +/- 0.00421 m
// Abs rotational error 0.15234 +/- 0.08123 deg
```
## Module Structure
```
evaluation/
├── mod.rs # Module exports
├── ground_truth.rs # GroundTruthRelations, PoseRelation
├── relations_metrics.rs # RelationsMetrics (Cartographer format)
├── accuracy.rs # ATE, RPE (evo-compatible metrics)
├── performance.rs # PerformanceTracker, timing utilities
└── quality.rs # Map quality, loop closure precision/recall
```
## Key Types
### Ground Truth Generation
| Type | Purpose |
|------|---------|
| `GroundTruthConfig` | Configuration for relation extraction (min distance, outlier thresholds) |
| `PoseRelation` | Relative transform between two trajectory nodes |
| `GroundTruthRelations` | Collection of relations from an optimized trajectory |
### Accuracy Metrics
| Type | Purpose |
|------|---------|
| `RelationsMetrics` | Cartographer-style pose relations evaluation |
| `AbsoluteTrajectoryError` | ATE after SE(2) alignment (evo `evo_ape` compatible) |
| `RelativePoseError` | RPE over fixed distances (evo `evo_rpe` compatible) |
| `AccuracyMetrics` | Container for ATE + RPE |
| `TrajectoryError` | Statistics (RMSE, mean, std, min, max, median) |
### Performance Metrics
| Type | Purpose |
|------|---------|
| `PerformanceTracker` | Record operation timings during SLAM |
| `PerformanceMetrics` | Aggregated timing statistics |
| `OperationTiming` | Per-operation statistics (mean, std, min, max) |
| `TimingGuard` | RAII guard for automatic timing |
### Quality Metrics
| Type | Purpose |
|------|---------|
| `LoopClosurePR` | Precision/Recall for loop closure detection |
| `LoopClosureGroundTruth` | Ground truth loop closure pairs |
| `MapQualityMetrics` | Coverage, wall sharpness, cell distribution |
## Usage Examples
### Pose Relations Evaluation (Cartographer-style)
```rust
use vastu_slam::evaluation::{GroundTruthRelations, RelationsMetrics, GroundTruthConfig};
// Generate ground truth from optimized trajectory
let config = GroundTruthConfig {
min_covered_distance: 0.5, // 50cm minimum between nodes
outlier_threshold_meters: 0.05, // 5cm outlier threshold
outlier_threshold_radians: 0.035, // ~2 degrees
node_sampling_interval: 1,
};
let ground_truth = GroundTruthRelations::generate(
&optimized_trajectory,
&loop_closure_pairs,
config,
);
// Evaluate test trajectory
let metrics = RelationsMetrics::compute_with_threshold(
&test_trajectory,
&ground_truth,
0.01, // 1cm pass/fail threshold
);
println!("{}", metrics.summary());
// Output: trans: 0.0083±0.0042m, rot: 0.15±0.08°, 45 relations, PASS
```
### ATE/RPE Evaluation (evo-compatible)
```rust
use vastu_slam::evaluation::AccuracyMetrics;
// Compute both ATE and RPE
let metrics = AccuracyMetrics::compute(
&estimated_trajectory,
&ground_truth_trajectory,
1.0, // 1m delta for RPE
);
metrics.print();
// === Absolute Trajectory Error (ATE) ===
// Alignment: SE(2)
// Translation RMSE: 0.012345 m (mean: 0.010234, std: 0.006789)
// Rotation RMSE: 0.034567 rad (mean: 0.028901, std: 0.019012)
//
// === Relative Pose Error (RPE) ===
// Delta: 1 m, Pairs: 42
// Translation RMSE: 0.008765 m (mean: 0.007654, std: 0.004321)
// Rotation RMSE: 0.023456 rad (mean: 0.019876, std: 0.012345)
```
### Performance Tracking
```rust
use vastu_slam::evaluation::PerformanceTracker;
use std::time::Duration;
let mut tracker = PerformanceTracker::with_scan_interval(Duration::from_millis(200));
tracker.start();
// Manual timing
let start = std::time::Instant::now();
// ... perform scan matching ...
tracker.record("scan_matching", start.elapsed());
// RAII timing guard
{
let _guard = tracker.time("map_update");
// ... update occupancy grid ...
} // Duration recorded automatically on drop
tracker.record_scan();
// Get metrics
let metrics = tracker.metrics();
metrics.print();
// === Performance Metrics ===
// Scans processed: 100
// Scans per second: 12.50
// Real-time factor: 2.50x
// Total time: 8000.00 ms
//
// Operation timings:
// scan_matching: 5.23 ± 1.12 ms (min: 3.45, max: 8.90, n=100)
// map_update: 2.34 ± 0.56 ms (min: 1.23, max: 4.56, n=100)
```
### Loop Closure Quality
```rust
use vastu_slam::evaluation::{LoopClosurePR, LoopClosureGroundTruth};
// Create ground truth from trajectory
let ground_truth = LoopClosureGroundTruth::from_trajectory(
&trajectory,
50, // Min 50 nodes separation
0.5, // Max 0.5m distance for loop closure
);
// Evaluate detections (from_node, to_node, score)
let detections = vec![
(0, 100, 0.95),
(25, 125, 0.87),
(50, 150, 0.82),
];
let pr = LoopClosurePR::compute(&detections, &ground_truth, 1.0);
pr.print();
// === Loop Closure Quality ===
// True Positives: 3
// False Positives: 0
// False Negatives: 1
// Precision: 100.00%
// Recall: 75.00%
// F1 Score: 85.71%
// Max Recall @100% Precision: 75.00%
```
### Map Quality Metrics
```rust
use vastu_slam::evaluation::MapQualityMetrics;
let quality = MapQualityMetrics::compute(&grid);
quality.print();
// === Map Quality Metrics ===
// Coverage: 45.23 m²
// Known cells: 78.5%
// Wall sharpness: 0.87
// Avg wall thickness: 2.34 cells
//
// Cell distribution:
// Floor: 125000
// Wall: 8500
// Cliff: 120
// Bump: 45
// Unknown: 34335
```
## Evaluation Strategy
### When External Ground Truth is Unavailable
1. Run full SLAM with loop closure optimization
2. Extract ground truth relations from the optimized trajectory
3. Re-run SLAM (or use odometry-only) and compare against relations
This tests the **self-consistency** of the SLAM system.
### When External Ground Truth is Available
Use ATE/RPE for absolute accuracy comparison:
- **ATE**: Best for evaluating global accuracy after alignment
- **RPE**: Best for evaluating local drift/odometry quality
## Threshold Checking
```rust
use std::collections::HashMap;
let mut thresholds = HashMap::new();
thresholds.insert("scan_matching".to_string(), 15.0); // 15ms max
thresholds.insert("map_update".to_string(), 5.0); // 5ms max
let violations = metrics.check_thresholds(&thresholds);
if !violations.is_empty() {
for v in violations {
eprintln!("Performance regression: {}", v);
}
}
```
## Key Design Decisions
1. **Cartographer-style relations**: Works without external ground truth
2. **evo compatibility**: ATE/RPE metrics match evo tool output format
3. **RTAB-Map style loop closure**: Precision/recall at 100% precision threshold
4. **RAII timing guards**: Automatic duration recording for cleaner code
5. **Serde support**: All metrics can be serialized for reporting