{"schema_version":"2026-09-15.story-v1","story":{"slug":"judging-factory-ai","title":"What counts as evidence that factory AI works?","description":"Compare inspection measurements, benchmark limits, and maintenance adoption without collapsing them into one success rate.","type":"reading-guide","editorial_question":"What does each reported result actually measure, and how far can it travel beyond the original setting?","limits":"Inspection error rates, benchmark scores, and maintenance adoption describe different things. This guide does not combine them into an industry-wide effectiveness estimate or a causal explanation.","canonical_url":"https://www.manufacturing.ai/stories/judging-factory-ai"},"policy":{"editorial_prompts_are_not_evidence":true,"publication_is_not_human_verification":true,"preserve_assertion_assessments":true,"evidence_entries_may_share_a_source":true,"linked_documents_do_not_add_verification":true,"do_not_treat_a_connection_as_causation":true,"full_topic_records":"https://www.manufacturing.ai/api/v1/topics/{slug}"},"sections":[{"title":"Look for a defined measurement","editorial_prompt":"Read the task, dataset, and error definitions attached to the result. An individual study is not a general accuracy figure for an industry.","topic_slug":"machine-vision","concept_slug":"machine-vision","topic_url":"https://www.manufacturing.ai/topics/machine-vision","evidence_url":"https://www.manufacturing.ai/topics/machine-vision#machine-vision","assertions":[{"id":"31aa4d69-3b3c-4bd2-bd64-1dc605ad6e71","statement":"On 2,042 real metal-box images captured in an unconstrained industrial environment, the best method in a peer-reviewed study achieved 10.6% false positives and 5.41% false negatives on defect localisation, against 13.02% and 8.6% for a fine-tuned VGG-16 baseline.","assessment":"reported","editor_reviewed":false,"reference_only_count":0,"evidence":[{"id":"ca669509-bc8d-4c5a-b840-fe4a2e8adeb0","stance":"supports","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Automatic detection and classification of manufacturing defects in metal boxes using deep neural networks","publisher":"PLoS ONE (Essid, Laga, Samir)","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC6226149/","sourceType":"academic","retrievalStatus":"fetched","inspected":true,"publishedAt":"2018-11-09","content_inspected":true,"documents":[]}},{"id":"9616a72a-69bf-4333-9145-04efabf915fc","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Automatic detection and classification of manufacturing defects in metal boxes using deep neural networks","publisher":"PLoS ONE (Essid, Laga, Samir)","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC6226149/","sourceType":"academic","retrievalStatus":"fetched","inspected":true,"publishedAt":"2018-11-09","content_inspected":true,"documents":[]}}]},{"id":"8b12a0c3-5be8-4fba-8c14-506316bc6e18","statement":"Machine vision is the industrial application of imaging — a combination of sensors, lenses, lighting and software that inspects, measures or identifies a part and hands a decision to an automation system — and its vendors distinguish it from the broader term computer vision, sorting cameras into line-scan, 2D area-scan and 3D categories.","assessment":"reported","editor_reviewed":false,"reference_only_count":0,"evidence":[{"id":"a2d67f1b-00e7-4457-be7a-5dd5ca14a308","stance":"supports","is_primary_for_assertion":true,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Machine vision basics: definitions, uses, and benefits","publisher":"Cognex","url":"https://www.cognex.com/what-is/machine-vision","sourceType":"company_website","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-11-06","content_inspected":true,"documents":[]}},{"id":"c7b68315-c963-45da-9d2d-e7b92d4673ae","stance":"supports","is_primary_for_assertion":true,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Categories of vision systems and the applications they perform","publisher":"Cognex","url":"https://www.cognex.com/en/tools-and-resources/resource-center/machine-vision/types-of-machine-vision-systems","sourceType":"company_website","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-11-06","content_inspected":true,"documents":[]}},{"id":"29ba9363-a379-496d-9fba-ba750e250a32","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Machine vision basics: definitions, uses, and benefits","publisher":"Cognex","url":"https://www.cognex.com/what-is/machine-vision","sourceType":"company_website","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-11-06","content_inspected":true,"documents":[]}}]}]},{"title":"Check the evaluation setting","editorial_prompt":"Compare the benchmark conditions with the claimed production use. Preserve the distinction between a preprint's argument and independently validated results.","topic_slug":"machine-vision","concept_slug":"visual-anomaly-detection","topic_url":"https://www.manufacturing.ai/topics/machine-vision","evidence_url":"https://www.manufacturing.ai/topics/machine-vision#visual-anomaly-detection","assertions":[{"id":"91ef178e-4052-4fc5-9875-16016a0d91ce","statement":"Across nine datasets, eleven state-of-the-art models and seven metrics, a benchmark study found that models reaching 99.9% image-level AUROC on MVTecAD degrade significantly on real-world data; its own benchmark uses six datasets with real-world defects against three with laboratory-produced defects, and excludes MVTecAD.","assessment":"reported","editor_reviewed":false,"reference_only_count":0,"evidence":[{"id":"41516371-0ce6-4a98-8c65-3e881bdb31ad","stance":"supports","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Beyond Academic Benchmarks: Critical Analysis and Best Practices for Visual Industrial Anomaly Detection","publisher":"arXiv (Baitieva, Bouaouni, Briot, Ameln, Khalfaoui, Akcay)","url":"https://arxiv.org/html/2503.23451v1","sourceType":"academic","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-03-30","content_inspected":true,"documents":[]}},{"id":"38a8f422-bb4f-4669-bf3f-f3fa8e275c79","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Beyond Academic Benchmarks: Critical Analysis and Best Practices for Visual Industrial Anomaly Detection","publisher":"arXiv (Baitieva, Bouaouni, Briot, Ameln, Khalfaoui, Akcay)","url":"https://arxiv.org/html/2503.23451v1","sourceType":"academic","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-03-30","content_inspected":true,"documents":[]}}]},{"id":"9e1b78b8-0ed5-42f3-aada-204ec25c2d9f","statement":"The benchmark study argues that image-level AUROC is not accountable for the relative importance of errors, that in production a missed defective part costs significantly more than a false positive, and that test-set-based early stopping, best-epoch reporting and centre-crop augmentation inflate published results — a cost-per-error view that sits alongside, rather than against, the practitioner account that repeated false alarms are what end deployments.","assessment":"reported","editor_reviewed":false,"reference_only_count":0,"evidence":[{"id":"521fd5b7-9097-4d6e-9224-488a80aedac5","stance":"supports","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Beyond Academic Benchmarks: Critical Analysis and Best Practices for Visual Industrial Anomaly Detection","publisher":"arXiv (Baitieva, Bouaouni, Briot, Ameln, Khalfaoui, Akcay)","url":"https://arxiv.org/html/2503.23451v1","sourceType":"academic","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-03-30","content_inspected":true,"documents":[]}},{"id":"f6fe80fe-2a36-4a50-bdd6-e0355c59e888","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"AI vision finds its footing on the factory floor","publisher":"Automotive Manufacturing Solutions","url":"https://www.automotivemanufacturingsolutions.com/smart-factory/ai-vision-finds-its-footing-on-the-factory-floor/2644253","sourceType":"journalism","retrievalStatus":"fetched","inspected":true,"publishedAt":"2026-04-10","content_inspected":true,"documents":[]}},{"id":"36346267-b539-4b5e-be63-4136102d3e5d","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Beyond Academic Benchmarks: Critical Analysis and Best Practices for Visual Industrial Anomaly Detection","publisher":"arXiv (Baitieva, Bouaouni, Briot, Ameln, Khalfaoui, Akcay)","url":"https://arxiv.org/html/2503.23451v1","sourceType":"academic","retrievalStatus":"fetched","inspected":true,"publishedAt":"2025-03-30","content_inspected":true,"documents":[]}}]}]},{"title":"Distinguish adoption from effectiveness","editorial_prompt":"Ask whether the source measures use, intention, or outcomes, and whether the population and question match the other evidence.","topic_slug":"predictive-maintenance","concept_slug":"predictive-maintenance","topic_url":"https://www.manufacturing.ai/topics/predictive-maintenance","evidence_url":"https://www.manufacturing.ai/topics/predictive-maintenance#predictive-maintenance","assertions":[{"id":"8ed6afbe-b614-4904-888b-1cc3e7511acc","statement":"Predictive maintenance uses equipment condition data to service a machine shortly before it fails, as distinct from schedule-based preventive and after-failure reactive maintenance; Deloitte estimates it can cut maintenance costs by up to 25% and raise uptime by 10-20%.","assessment":"reported","editor_reviewed":false,"reference_only_count":0,"evidence":[{"id":"355deead-b5df-441a-8ee1-70a8c019549b","stance":"supports","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"25 Maintenance Stats, Trends, And Insights","publisher":"MaintainX","url":"https://www.getmaintainx.com/blog/maintenance-stats-trends-and-insights","sourceType":"other","retrievalStatus":"fetched","inspected":true,"publishedAt":null,"content_inspected":true,"documents":[]}},{"id":"9b0aacc8-6438-498d-b1b8-863556b14357","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"25 Maintenance Stats, Trends, And Insights","publisher":"MaintainX","url":"https://www.getmaintainx.com/blog/maintenance-stats-trends-and-insights","sourceType":"other","retrievalStatus":"fetched","inspected":true,"publishedAt":null,"content_inspected":true,"documents":[]}}]},{"id":"a56ad64b-8144-4c21-bf0f-445c8e718329","statement":"Surveys disagree on the direction of predictive maintenance adoption: Fluke's 2026 Censuswide survey of 600+ decision-makers in the US, UK and Germany reports a rise from 9% to 18%, while the 2025 State of Industrial Maintenance Report reports a fall from 30% to 27% — bases of 9% and 30% cannot describe the same threshold.","assessment":"disputed","editor_reviewed":false,"reference_only_count":0,"evidence":[{"id":"0e62ac11-d2e2-4ee3-98d2-dbe436f7a793","stance":"supports","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Fluke Survey Finds Predictive Maintenance Adoption Doubles as Manufacturers Boost Digital Investment","publisher":"The Manila Times (reproducing Fluke's GlobeNewswire release)","url":"https://www.manilatimes.net/2026/05/07/tmt-newswire/globenewswire/fluke-survey-finds-predictive-maintenance-adoption-doubles-as-manufacturers-boost-digital-investment/2338228","sourceType":"press_release","retrievalStatus":"fetched","inspected":true,"publishedAt":"2026-05-07","content_inspected":true,"documents":[]}},{"id":"c0fa7ffd-c889-4860-a225-3d8c8cba8122","stance":"supports","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Fluke survey shows growth in predictive maintenance adoption as skills shortages persist","publisher":"MRO Magazine","url":"https://www.mromagazine.com/2026/05/10/fluke-survey-shows-growth-in-predictive-maintenance-adoption-as-skills-shortages-persist/","sourceType":"journalism","retrievalStatus":"fetched","inspected":true,"publishedAt":"2026-05-10","content_inspected":true,"documents":[]}},{"id":"b1ccf1cf-e1c6-4579-8df0-52020ceaf880","stance":"qualifies","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"Fluke Survey Finds Predictive Maintenance Adoption Doubles as Manufacturers Boost Digital Investment","publisher":"The Manila Times (reproducing Fluke's GlobeNewswire release)","url":"https://www.manilatimes.net/2026/05/07/tmt-newswire/globenewswire/fluke-survey-finds-predictive-maintenance-adoption-doubles-as-manufacturers-boost-digital-investment/2338228","sourceType":"press_release","retrievalStatus":"fetched","inspected":true,"publishedAt":"2026-05-07","content_inspected":true,"documents":[]}},{"id":"ca8e8df5-f7f8-4b8f-8552-2e951673ed01","stance":"challenges","is_primary_for_assertion":false,"origin_independence":"unknown","note":null,"quote":null,"source":{"title":"25 Maintenance Stats, Trends, And Insights","publisher":"MaintainX","url":"https://www.getmaintainx.com/blog/maintenance-stats-trends-and-insights","sourceType":"other","retrievalStatus":"fetched","inspected":true,"publishedAt":null,"content_inspected":true,"documents":[]}}]}]}]}