Background The capillary refill test is widely used in pediatric emergency medicine and critical care although its validity and reliability are debated. Naked eye estimation is the recommended method for capillary refill time (CR time) assessment. The goal of this study was to compare naked eye estimations of the CR time in pediatric patients to quantified capillary refill time (qCR time) using polarized reflectance imaging as an objective reference, and to investigate interobserver and intra-observer consistency of naked eye assessments of CR time. Method A film sequence comprising videos of capillary refill tests from 15 emergency pediatric patients was shown under standardized conditions to 62 observers (pediatricians, nurses, assistant nurses, and medical secretaries). The observers’ estimations of CR time in seconds and in descriptive categorizations were compared to objectively derived qCR time. Three tests were shown twice without the observers’ knowledge. Results There was poor interobserver agreement in all professions, with limits of agreement ranging from 1.17 s (assistant nurses) to 2.00 s (secretaries). Intra-observer agreement for estimations of both time and descriptive categorizations was limited. The correlation between naked eye assessments and qCR time was weak. Conclusion This study shows that naked eye assessment of CR time in children is highly subjective with poor reproducibility in pediatric nurses and pediatricians, as well as in comparison to a quantitative method. Based on the lack of both inter- and intra-observer consistency in the assessments, these findings suggest that CR time assessed by naked eye should be questioned as a routine test in pediatric emergencies.