Enhancing Infrastructure Monitoring with Calibrated Vision Language Model Ensembles: A Graffiti Detection Case Study

Abstract

Urban transit authorities face significant challenges in efficiently monitoring distributed infrastructure assets. This dissertation presents a novel three-stage pipeline for automated infrastructure condition assessment: (1) GPS-based geofence creation that automatically defines inspection boundaries, (2) YOLOv11-powered detection optimized for transit infrastructure, and (3) damage assessment using a specialized Vision Language Model (VLM) ensemble. Our ensemble approach employs probability calibration and weight optimization, outperforming individual models with an 84.4% F1 score. To address extreme data scarcity, we developed a synthetic graffiti generation methodology, expanding from just 3 real examples to a comprehensive evaluation dataset with balanced color representation. Experiments utilized data collected along 227.58 miles of bus routes covering 778 unique bus stops across the Reno metro area. The system operates efficiently on standard hardware with geofence-triggered processing, making advanced VLM technology accessible for practical transit authority deployment. Key contributions include: (1) a pipeline architecture that reduces computational requirements by 95% compared to continuous processing; (2) ensemble techniques using isotonic regression calibration and differential evolution optimization that significantly improve detection accuracy; (3) a parametric graffiti generation methodology creating realistic synthetic data with controllable characteristics; and (4) a color-specific analysis framework revealing critical insights about model performance across different visual characteristics. Ablation studies demonstrate that while traditional CNN models catastrophically fail on unseen variations, VLMs maintain more consistent behavior across varied visual characteristics, highlighting their superior generalization capabilities. The system transforms infrastructure monitoring from a manual, subjective process that burdens drivers into a data-driven approach supporting proactive maintenance and strategic planning.